Learned Robust PCA: A Scalable Deep Unfolding Approach for High-Dimensional Outlier Detection

1 Introduction . Over the last decade , robust principal component analysis ( RPCA ) , one of the fundamental dimension reduction techniques , has received intensive investigations from theoretical and empirical perspectives [ 1–13 ] . RPCA also plays a key role in a wide range of machine learning tasks , such as video background subtraction [ 14 ] , singing-voice separation [ 15 ] , face modeling [ 16 ] , image alignment [ 17 ] , feature identification [ 18 ] , community detection [ 19 ] , fault detection [ 20 ] , and NMR spectrum recovery [ 21 ] . While the standard principal component analysis ( PCA ) is known for its high sensitivity to outliers , RPCA is designed to enhance the robustness of PCA when outliers are present . In this paper , we consider the following RPCA setting : given an observed corrupted data matrix Y = X ? + S ? ∈ Rn1×n2 , ( 1 ) where X ? is a rank-r data matrix and S ? is a sparse outlier matrix , reconstruct X ? and S ? simultaneously from Y . One of the main challenges of designing an RPCA algorithm is to avoid high computational costs . Inspired by deep unfolded sparse coding [ 22 ] , some recent works [ 23–25 ] have successfully extended deep unfolding techniques to RPCA and achieved noticeable accelerations in certain applications . ∗Authors are listed in alphabetic order . Correspondence shall be addressed to H.Q . Cai . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Specifically , by parameterizing a classic RPCA algorithm and unfolding it as a feedforward neural network ( FNN ) , one can improve its performance by learning the parameters of the FNN through backpropagation . Since RPCA problems of a specific application often share similar key properties ( e.g. , rank , incoherence , and amount of outliers ) , the parameters learned from training examples that share the properties can lead to superior performance . Nevertheless , all existing learning-based approaches call an expensive step of singular value thresholding ( SVT ) [ 26 ] at every iteration during both training and inference . SVT involves a full or truncated singular value decomposition ( SVD ) , which costs from O ( n3 ) to O ( n2r ) flops2 with some large hidden constant in front3 . Thus , these approaches are not scalable to high-dimensional RPCA problems . Another issue is that the existing approaches only learn the parameters for a finite number of iterations . If a user targets at a specific accuracy of recovery , then the prior knowledge of the unfolded algorithm is required for choosing the number of unfolded layers . Moreover , if the user desires better accuracy later , one may have to restart the learning process to obtain parameters of the extra iterations . We must ask the questions : “ Can one design a highly efficient and easy-to-learn method for highdimensional RPCA problems ? ” and “ Do we have to restrict ourselves to finite-iteration unfolding ( or a fixed-layer neural network ) ? ” In this paper , we aim to answer these questions by proposing a scalable and learnable approach for high-dimensional RPCA problems , which has a flexible feedforwardrecurrent-mixed neural network model for potentially infinite-iteration unfolding . Consequently , our approach can satisfy arbitrary accuracy without relearning the parameters . Related work . The earlier works [ 1–3 ] for RPCA are based on the convex model : minimize X , S ‖X‖∗ + λ‖S‖1 , subject to Y = X + S. ( 2 ) It has been shown that ( 2 ) guarantees exact recovery , provided α . O ( 1/µr ) , a condition with an optimal order [ 4 , 5 ] . Therein , α is the sparsity parameter of S ? and µ is the incoherence parameter of L ? , which will be formally defined later in Assumptions 2 and 1 , respectively . Problem ( 2 ) can be transformed into a semidefinite program and has a variety of dedicated methods , their per-iteration complexities are at least O ( n3 ) flops and some of them guarantee only sublinear convergence . Later , various non-convex methods with linear convergence were proposed for RPCA : [ 7 ] uses alternating minimization , and it can tolerate outliers up to a fraction of only α . O ( 1/µ2/3r2/3n ) . [ 6 ] develops an alternating projections method ( AltProj ) that alternatively projects Y − S onto the space of low-rank matrices and Y −X onto the space of sparse matrices . AltProj costs O ( n2r2 ) flops and tolerates outliers up to α . O ( 1/µr ) . An accelerated version of AltProj ( AccAltProj ) was proposed in [ 10 ] , which improves the computational complexity to O ( n2r ) with sacrifices on the tolerance to outliers , which are allowed up to α . O ( 1/max { µr2κ3 , µ3/2r2κ , µ2r2 } ) , where κ is the condition number of X ? . Note that the key technique used for outlier detection in AltProj and AccAltProj is employing adaptive hard-thresholding parameters . Another line of non-convex RPCA algorithms reformulate the low-rank component as X = LR > , where L ∈ Rn1×r and R ∈ Rn2×r , and then performs gradient descend on L and R separately . Since the low-rank constraint of X is automatically satisfied under this reformulation , the costly step of SVD is avoided ( except for one SVD during initialization ) . Specifically speaking , [ 8 ] uses a complicated objective function , which includes a practically unimportant balance regularization ‖L > L−R > R‖F . While GD costs onlyO ( n2r ) flops per iteration , its convergence rate relies highly on κ . Recently , ScaledGD [ 12 ] introduces a scaling term in gradient descend steps to remove the dependence on κ in the convergence rate of GD . ScaledGD also drops the balance regularization and targets on a more elegant objective : minimize L∈Rn1×r , R∈Rn2×r , S∈Rn1×n2 1 2 ‖LR > + S − Y ‖2F , subject to S is α-sparse , ( 3 ) where α-sparsity means no more than a fraction α of non-zeros on every row and every column of S. To enforce the sparsity constraint of S , ScaledGD ( and also GD ) employs a sparsification operator : [ Tα̃ ( M ) ] i , j = { [ M ] i , j , if | [ M ] i , j | ≥ | [ M ] ( α̃n2 ) i , : | and | [ M ] i , j | ≥ | [ M ] ( α̃n1 ) : ,j | 0 , otherwise , ( 4 ) 2For ease of presentation , we take n : = n1 = n2 when discuss computational complexities . 3The hidden constant is often hundreds and sometimes thousands . where [ · ] ( k ) i , : and [ · ] ( k ) : ,j denote the k-th largest element in magnitude on the i-th row and in the j-th column , respectively . In other words , Tα̃ keeps only the largest α̃ fraction of the entries on every row and in every column . The per-iteration computational complexity of ScaledGD remains O ( n2r ) and it converges faster on ill-conditioned problems . The tolerance to outliers of GD and ScaledGD are α . O ( 1/max { µr3/2κ3/2 , µrκ2 } ) and α . O ( 1/µr3/2κ ) , respectively . It is worth mentioning there exist some other RPCA settings that high-dimensional problems can be efficiently solved . For example , if the columns of X ? are independently sampled over a zero-mean multivariate Gaussian distribution , an efficient approach is using Grassmann averages [ 27 , 28 ] . Deep unfolding is a technique that dates back to 2010 from a fast approximate method for LASSO : LISTA [ 22 ] parameterizes the classic Iterative Shrinkage-Thresholding Algorithm ( ISTA ) as a fullyconnected feedforward neural network and demonstrates that the trained neural network generalizes well to unseen samples from the distribution used for training . It achieves the same accuracy within one or two order-of-magnitude fewer iterations than the original ISTA . Later works [ 29–36 ] extend this approach to different problems and network architectures and have good performance . Another recently developed technique “ learning to learn ” [ 37–39 ] parameterizes iterative algorithms as recurrent neural networks and shows great potential on machine learning tasks . Applying deep unfolding on RPCA appeared recently . CORONA [ 23 , 24 ] uses convolutional layers in deep unfolding and focuses on the application of ultrasound imaging . It uses SVT for low-rank approximation and mixed ` 1,2 thresholding for outlier detection since the outliers in ultrasound imaging problems are usually structured . refRPCA [ 25 ] focuses on video foreground-background separation and also uses SVT for low-rank approximation . However , refRPCA employs a sequence of constructed reference frames to reflex correlation between consecutive frames in the video , which leads to a complicated yet learnable proximal operator for outlier detection . Given the high computational cost of SVT , both CORONA and refRPCA are not scalable . In addition , Denise [ 40 ] studies a deep unfolding RPCA method for the special case of positive semidefinite low-rank matrices . Denise achieves a remarkable speedup from baselines ; however , such RPCA applications are limited . Contribution . In this work , we propose a novel learning-based method , which we call Learned Robust PCA ( LRPCA ) , for solving high-dimensional RPCA problems . Our main contributions are : 1 . The proposed algorithm , LRPCA , is scalable and learnable . It uses a simple formula and differentiable operators to avoid the costly SVD and partial sorting ( see Algorithm 1 ) . LRPCA costs O ( n2r ) flops with a small constant4 while all its parameters can be optimized by training . 2 . An exact recovery guarantee is established for LRPCA under some mild conditions ( see Theorem 1 ) . In particular , LRPCA can tolerate outliers up to a fraction of α . O ( 1/µr3/2κ ) . Moreover , the theorem confirms that there exist a set of parameters for LRPCA to outperform the baseline algorithm ScaledGD . 3 . We proposed a novel feedforward-recurrent-mixed neural network ( FRMNN ) model to solve RPCA . The new model first unfolds finite iterations of LRPCA and individually learns the parameters at the significant iteration ; then , it learns the rules of parameter updating for subsequent iterations . Therefore , FRMNN learns the parameters for infinite iterations of LRPCA while we ensure no performance reduction from classic deep unfolding . 4 . The numerical experiments confirm the advantages of LRPCA on both synthetic and realworld datasets . In particular , we successfully apply LRPCA to large video background subtraction tasks where the problem dimensions exceed the capacities of the existing learning-based RPCA approaches . Notation . For any matrix M , [ M ] i , j denotes the ( i , j ) -th entry , σi ( M ) denotes the i-th singular value , ‖M‖1 : = ∑ i , j | [ M ] i , j | denotes the entrywise ` 1 norm , ‖M‖F : = ( ∑ i , j [ M ] 2 i , j ) 1/2 denotes the Frobenius norm , ‖M‖∗ : = ∑ i σi ( M ) denotes the nuclear norm , ‖M‖∞ : = maxi , j | [ M ] i , j | denotes the largest magnitude , ‖M‖2 , ∞ : = maxi ( ∑ j [ M ] 2 i , j ) 1/2 denotes the largest row-wise ` 2 norm , and M > denotes the transpose . We use κ : = σ1 ( X ? ) σr ( X ? ) to denote the condition number of X ? . 4More precisely , LRPCA costs as low as 3n2r + 3n2 +O ( nr2 ) flops per iteration , which is much smaller than the hidden constant of truncated SVD .

Summary: The authors study the problem of robust PCA inspired  by a LISTA type approach. In particular, the authors consider  a hybrid model of feedforward, and recurrent NN architectures and "learn the parameters of (scaled) GD" that is used to solve RPCA. I am not extremely familiar with the literature, but essentially the hybrid model allows for a non-apriori (in terms of the architecture) number of iterations that the algorithm can run for. This seems particularly interesting to me. The authors show that under certain conditions of the parameters, the algorithm enjoys linear convergence. The authors also show synthetic data experiments and a use-case of background subtraction to validate their results.