Learned Robust PCA: A Scalable Deep Unfolding Approach for High-Dimensional Outlier Detection

1 Introduction . Over the last decade , robust principal component analysis ( RPCA ) , one of the fundamental dimension reduction techniques , has received intensive investigations from theoretical and empirical perspectives [ 1–13 ] . RPCA also plays a key role in a wide range of machine learning tasks , such as video background subtraction [ 14 ] , singing-voice separation [ 15 ] , face modeling [ 16 ] , image alignment [ 17 ] , feature identification [ 18 ] , community detection [ 19 ] , fault detection [ 20 ] , and NMR spectrum recovery [ 21 ] . While the standard principal component analysis ( PCA ) is known for its high sensitivity to outliers , RPCA is designed to enhance the robustness of PCA when outliers are present . In this paper , we consider the following RPCA setting : given an observed corrupted data matrix Y = X ? + S ? ∈ Rn1×n2 , ( 1 ) where X ? is a rank-r data matrix and S ? is a sparse outlier matrix , reconstruct X ? and S ? simultaneously from Y . One of the main challenges of designing an RPCA algorithm is to avoid high computational costs . Inspired by deep unfolded sparse coding [ 22 ] , some recent works [ 23–25 ] have successfully extended deep unfolding techniques to RPCA and achieved noticeable accelerations in certain applications . ∗Authors are listed in alphabetic order . Correspondence shall be addressed to H.Q . Cai . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Specifically , by parameterizing a classic RPCA algorithm and unfolding it as a feedforward neural network ( FNN ) , one can improve its performance by learning the parameters of the FNN through backpropagation . Since RPCA problems of a specific application often share similar key properties ( e.g. , rank , incoherence , and amount of outliers ) , the parameters learned from training examples that share the properties can lead to superior performance . Nevertheless , all existing learning-based approaches call an expensive step of singular value thresholding ( SVT ) [ 26 ] at every iteration during both training and inference . SVT involves a full or truncated singular value decomposition ( SVD ) , which costs from O ( n3 ) to O ( n2r ) flops2 with some large hidden constant in front3 . Thus , these approaches are not scalable to high-dimensional RPCA problems . Another issue is that the existing approaches only learn the parameters for a finite number of iterations . If a user targets at a specific accuracy of recovery , then the prior knowledge of the unfolded algorithm is required for choosing the number of unfolded layers . Moreover , if the user desires better accuracy later , one may have to restart the learning process to obtain parameters of the extra iterations . We must ask the questions : “ Can one design a highly efficient and easy-to-learn method for highdimensional RPCA problems ? ” and “ Do we have to restrict ourselves to finite-iteration unfolding ( or a fixed-layer neural network ) ? ” In this paper , we aim to answer these questions by proposing a scalable and learnable approach for high-dimensional RPCA problems , which has a flexible feedforwardrecurrent-mixed neural network model for potentially infinite-iteration unfolding . Consequently , our approach can satisfy arbitrary accuracy without relearning the parameters . Related work . The earlier works [ 1–3 ] for RPCA are based on the convex model : minimize X , S ‖X‖∗ + λ‖S‖1 , subject to Y = X + S. ( 2 ) It has been shown that ( 2 ) guarantees exact recovery , provided α . O ( 1/µr ) , a condition with an optimal order [ 4 , 5 ] . Therein , α is the sparsity parameter of S ? and µ is the incoherence parameter of L ? , which will be formally defined later in Assumptions 2 and 1 , respectively . Problem ( 2 ) can be transformed into a semidefinite program and has a variety of dedicated methods , their per-iteration complexities are at least O ( n3 ) flops and some of them guarantee only sublinear convergence . Later , various non-convex methods with linear convergence were proposed for RPCA : [ 7 ] uses alternating minimization , and it can tolerate outliers up to a fraction of only α . O ( 1/µ2/3r2/3n ) . [ 6 ] develops an alternating projections method ( AltProj ) that alternatively projects Y − S onto the space of low-rank matrices and Y −X onto the space of sparse matrices . AltProj costs O ( n2r2 ) flops and tolerates outliers up to α . O ( 1/µr ) . An accelerated version of AltProj ( AccAltProj ) was proposed in [ 10 ] , which improves the computational complexity to O ( n2r ) with sacrifices on the tolerance to outliers , which are allowed up to α . O ( 1/max { µr2κ3 , µ3/2r2κ , µ2r2 } ) , where κ is the condition number of X ? . Note that the key technique used for outlier detection in AltProj and AccAltProj is employing adaptive hard-thresholding parameters . Another line of non-convex RPCA algorithms reformulate the low-rank component as X = LR > , where L ∈ Rn1×r and R ∈ Rn2×r , and then performs gradient descend on L and R separately . Since the low-rank constraint of X is automatically satisfied under this reformulation , the costly step of SVD is avoided ( except for one SVD during initialization ) . Specifically speaking , [ 8 ] uses a complicated objective function , which includes a practically unimportant balance regularization ‖L > L−R > R‖F . While GD costs onlyO ( n2r ) flops per iteration , its convergence rate relies highly on κ . Recently , ScaledGD [ 12 ] introduces a scaling term in gradient descend steps to remove the dependence on κ in the convergence rate of GD . ScaledGD also drops the balance regularization and targets on a more elegant objective : minimize L∈Rn1×r , R∈Rn2×r , S∈Rn1×n2 1 2 ‖LR > + S − Y ‖2F , subject to S is α-sparse , ( 3 ) where α-sparsity means no more than a fraction α of non-zeros on every row and every column of S. To enforce the sparsity constraint of S , ScaledGD ( and also GD ) employs a sparsification operator : [ Tα̃ ( M ) ] i , j = { [ M ] i , j , if | [ M ] i , j | ≥ | [ M ] ( α̃n2 ) i , : | and | [ M ] i , j | ≥ | [ M ] ( α̃n1 ) : ,j | 0 , otherwise , ( 4 ) 2For ease of presentation , we take n : = n1 = n2 when discuss computational complexities . 3The hidden constant is often hundreds and sometimes thousands . where [ · ] ( k ) i , : and [ · ] ( k ) : ,j denote the k-th largest element in magnitude on the i-th row and in the j-th column , respectively . In other words , Tα̃ keeps only the largest α̃ fraction of the entries on every row and in every column . The per-iteration computational complexity of ScaledGD remains O ( n2r ) and it converges faster on ill-conditioned problems . The tolerance to outliers of GD and ScaledGD are α . O ( 1/max { µr3/2κ3/2 , µrκ2 } ) and α . O ( 1/µr3/2κ ) , respectively . It is worth mentioning there exist some other RPCA settings that high-dimensional problems can be efficiently solved . For example , if the columns of X ? are independently sampled over a zero-mean multivariate Gaussian distribution , an efficient approach is using Grassmann averages [ 27 , 28 ] . Deep unfolding is a technique that dates back to 2010 from a fast approximate method for LASSO : LISTA [ 22 ] parameterizes the classic Iterative Shrinkage-Thresholding Algorithm ( ISTA ) as a fullyconnected feedforward neural network and demonstrates that the trained neural network generalizes well to unseen samples from the distribution used for training . It achieves the same accuracy within one or two order-of-magnitude fewer iterations than the original ISTA . Later works [ 29–36 ] extend this approach to different problems and network architectures and have good performance . Another recently developed technique “ learning to learn ” [ 37–39 ] parameterizes iterative algorithms as recurrent neural networks and shows great potential on machine learning tasks . Applying deep unfolding on RPCA appeared recently . CORONA [ 23 , 24 ] uses convolutional layers in deep unfolding and focuses on the application of ultrasound imaging . It uses SVT for low-rank approximation and mixed ` 1,2 thresholding for outlier detection since the outliers in ultrasound imaging problems are usually structured . refRPCA [ 25 ] focuses on video foreground-background separation and also uses SVT for low-rank approximation . However , refRPCA employs a sequence of constructed reference frames to reflex correlation between consecutive frames in the video , which leads to a complicated yet learnable proximal operator for outlier detection . Given the high computational cost of SVT , both CORONA and refRPCA are not scalable . In addition , Denise [ 40 ] studies a deep unfolding RPCA method for the special case of positive semidefinite low-rank matrices . Denise achieves a remarkable speedup from baselines ; however , such RPCA applications are limited . Contribution . In this work , we propose a novel learning-based method , which we call Learned Robust PCA ( LRPCA ) , for solving high-dimensional RPCA problems . Our main contributions are : 1 . The proposed algorithm , LRPCA , is scalable and learnable . It uses a simple formula and differentiable operators to avoid the costly SVD and partial sorting ( see Algorithm 1 ) . LRPCA costs O ( n2r ) flops with a small constant4 while all its parameters can be optimized by training . 2 . An exact recovery guarantee is established for LRPCA under some mild conditions ( see Theorem 1 ) . In particular , LRPCA can tolerate outliers up to a fraction of α . O ( 1/µr3/2κ ) . Moreover , the theorem confirms that there exist a set of parameters for LRPCA to outperform the baseline algorithm ScaledGD . 3 . We proposed a novel feedforward-recurrent-mixed neural network ( FRMNN ) model to solve RPCA . The new model first unfolds finite iterations of LRPCA and individually learns the parameters at the significant iteration ; then , it learns the rules of parameter updating for subsequent iterations . Therefore , FRMNN learns the parameters for infinite iterations of LRPCA while we ensure no performance reduction from classic deep unfolding . 4 . The numerical experiments confirm the advantages of LRPCA on both synthetic and realworld datasets . In particular , we successfully apply LRPCA to large video background subtraction tasks where the problem dimensions exceed the capacities of the existing learning-based RPCA approaches . Notation . For any matrix M , [ M ] i , j denotes the ( i , j ) -th entry , σi ( M ) denotes the i-th singular value , ‖M‖1 : = ∑ i , j | [ M ] i , j | denotes the entrywise ` 1 norm , ‖M‖F : = ( ∑ i , j [ M ] 2 i , j ) 1/2 denotes the Frobenius norm , ‖M‖∗ : = ∑ i σi ( M ) denotes the nuclear norm , ‖M‖∞ : = maxi , j | [ M ] i , j | denotes the largest magnitude , ‖M‖2 , ∞ : = maxi ( ∑ j [ M ] 2 i , j ) 1/2 denotes the largest row-wise ` 2 norm , and M > denotes the transpose . We use κ : = σ1 ( X ? ) σr ( X ? ) to denote the condition number of X ? . 4More precisely , LRPCA costs as low as 3n2r + 3n2 +O ( nr2 ) flops per iteration , which is much smaller than the hidden constant of truncated SVD .

Summary: The paper propose a new technique for robust PCA that can be realised as a trainable neural network. The technique provides similar theoretical guarantees to existing methods, yet scale better to larger problems. The paper is well-written and mostly easy to follow.