Square Root Principal Component Pursuit: Tuning-Free Noisy Robust Matrix Recovery

1 Introduction . The problem of recovering a low-rank matrix from unreliable observations arises in a wide range of engineering applications , including collaborative filtering [ 1 ] , latent semantic indexing [ 2 ] , image and video analysis [ 3 , 4 , 5 ] and so on . This problem can be formalized in terms of the following observation model : given an observationD which is a superposition D = L0 low-rank + S0 sparse + Z0 noise . ( 1.1 ) 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . of an unknown low-rank matrix L0 , sparse corruptions S0 and dense noise Z0 , our goal is to accurately estimate both L0 and S0 . This model has been intensely studied , leading to algorithmic theory for methods based on both convex and nonconvex optimization [ 6 , 7 , 8 ] . One virtue of the convex approach is that in the noise-free setting ( Z0 = 0 ) , it is possible to exactly recover a broad range of low-rank and sparse pairs ( L0 , S0 ) , with a universal choice of regularization parameters , which does not depend on either the rank or sparsity . This makes it possible to deploy this method in a “ hands-free ” manner , provided the dataset of interest indeed has low-rank and sparse structure . In the presence of noise , however , the situation becomes more complicated : all efficient , guaranteed estimators require knowledge of the noise level ( or the rank and sparsity ) [ 8 , 9 , 10 ] . This is problematic , since in most applications the noise level is not known ahead of time . In standard convex formulations , the appropriate regularization parameter depends on the noise standard deviation , leaving the user with a painful and time-consuming task of tuning these parameters on a per-dataset basis . Motivated by this issue , we revisit this classical matrix recovery problem . The main contribution of this paper is the proposal and analysis of a new formulation for robust matrix recovery , which stably recovers L0 and S0 without requiring prior knowledge of the rank , sparsity , or noise level . In particular , our approach admits a single , universal choice of regularization parameters , which under standard hypotheses on L0 and S0 , yields an estimation error proportional to the noise standard deviation σ . To our knowledge , our method and analysis are the first to achieve this . Our approach is based on a combination of two natural ideas . For matrix recovery , we draw on the stable principal component pursuit [ 9 ] , a natural convex relaxation , which minimizes a combination of the nuclear norm of L , the ` 1 norm of S and the squared Frobenius norm ‖Z‖2F of the noise . This is a principled approach to handling both the structured components L0 , S0 and the noise : ‖Z‖2F can bemotivated naturally from the negative log-likelihood of the gaussian distribution . Moreover , under mild assumptions on the rank and singular vectors of L0 and the sparsity pattern of S0 , the reconstruction error of stable PCP is O ( ‖Z0‖F ) [ 9 ] . On the other hand , optimally balancing these terms requires knowledge of the standard deviation σ of the true noise distribution . To address this issue , we draw inspiration from the square root Lasso [ 11 ] . The square root Lasso is a sparse estimator which achieves minimax optimal estimation with a universal choice of parameters , which does not depend on the noise level . The core idea is very simple : instead of penalizing the squared Frobenius norm ‖Z‖2F of the noise , one penalizes its square root , ‖Z‖F . We call the resulting formulation square root principal component pursuit ( √ PCP ) . Our new formulation has the benefit that with a noise-independent universal choice of regularization parameters , essentially the same level of reconstruction error can be achieved . This makes √ PCP a more practical approach to low-rank recovery in unknown noise . Due to the square root term , the objective function is no longer smooth or differentiable , and so we can not apply algorithms such as the proximal gradient method1 . Nevertheless , our new formulation remains convex and separable , i.e . the objective is the sum of functions of different variables , making Alternating Direction Method of Multipliers ( ADMM ) a suitable solver [ 12 ] . We test our new formulation with ADMM on both simulated data and real data in image processing . The experimental results show the effectiveness of our proposed new formulation in recovering the low-rank and the sparse matrix , and suggest that √ PCP has better performance than anticipated by our loose upper bound of the reconstruction error . 1.1 Notations and Assumptions . We use ‖A‖ , ‖A‖F , and ‖A‖∗ to denote the spectral , Frobenius , and nuclear norm of the matrixA , andA∗ its ( conjugate ) transpose . For convenience , we letX0 = ( L0 , S0 ) be the concatenation ofL0 andS0 . We assume thatL0 is a low-rankmatrix of rank rwhose compact 1which requires the objective to be the sum of a smooth and a non-smooth function SVD is L0 = UΣV ∗ , U ∈ Rn1×r , V ∈ Rn2×r , and without loss of generality , n1 ≥ n2 . Let T = { UQ∗ + RV ∗|Q ∈ Rn2×r , R ∈ Rn1×r } denote the tangent space of rank r matrices at L0 . In addition , we assume that S0 is sparse with support in Ω . Since it is impossible to disentangle L0 and S0 if the low-rank matrix L0 is sparse , or if the sparse matrix S0 is low-rank , we make the following two assumptions : Assumption 1.1 The low-rank matrix L0 satisfies the incoherence property with parameter ν2 , i.e . max i ‖U∗ei‖2 ≤ νr n1 , max i ‖V ∗ei‖2 ≤ νr n2 , ‖UV ∗‖∞ ≤ √ νr n1n2 . Assumption 1.2 The support Ω is chosen uniformly among all sets of cardinalitym , and the signs of supports are random , i.e . P [ ( S0 ) i , j > 0| ( i , j ) ∈ Ω ] = P [ ( S0 ) i , j < 0| ( i , j ) ∈ Ω ] = 0.5 . These assumptions follow [ 6 ] ; indeed , our proof makes use of a dual certificate constructed for noiseless low-rank and sparse recovery in that paper . 1.2 Problem Formulation and Main Results . Inspired by square root Lasso , we propose to solve the robust noisy matrix recovery problem through the following optimization problem : √ PCP : min L , S ‖L‖∗ + λ‖S‖1 + µ ‖L + S −D‖F . ( 1.2 ) The parameter λ that balances the low-rank and the sparse regularizers is studied in [ 6 ] , where it is shown thatλ = 1/√n1 gives exact recoverywhenZ0 = 0 and theµ ‖L + S −D‖F penalty term in ( 1.2 ) is replaced with the constraint L + S = D. In this work , we build on this result and focus on the parameter µ . Our main result is that under the aforementioned ( standard ) hypotheses on L0 and S0 , using a single , universal choice µ = √ n2/2 , √ PCP recovers L0 and S0 , with an estimation error that is proportional to the norm of the noise : Theorem 1.1 Under Assumptions 1.1 and 1.2 , provided that r , m satisfies r ≤ ρrn2ν−1 ( log n1 ) −2 , m ≤ ρsn1n2 , ( 1.3 ) where ρr ≤ 1/10 , ρs are some positive constants . Then there is a numerical constant c such that with probability at least 1− cn−101 , the √ PCP problem ( 1.2 ) with λ = 1/√n1 and µ = √ n2/2 produces a solution X̂ = ( L̂ , Ŝ ) such that ‖X̂ −X0‖F ≤ 560 √ n1n2‖Z0‖F . ( 1.4 ) Why is it possible to achieve accurate estimation with a single choice of µ ? We draw intuition from a connection to the stable principal component pursuit formulations studied in [ 9 ] . This work studies both constrained and unconstrained formulations : StablePCPc : min L , S ‖L‖∗ + λ‖S‖1 s.t . ‖L + S −D‖F ≤ δ . ( 1.5 ) StablePCPu : min L , S ‖L‖∗ + λ‖S‖1 + µ̄ 2 ‖L + S −D‖2F . ( 1.6 ) These formulations are equivalent , and equivalent to √ PCP in the following sense : for each problem instance , there is a calibration of parameters δ ↔ µ̄ ↔ µ such that √ PCP , StablePCPu and StablePCPc have exactly the same set of optimal solutions . However , ( 1.5 ) - ( 1.6 ) require that the parameters δ and µ̄ be determined on an instance-by-instance basis , based on the noise level . Choosing these parameters correctly is essential : in ( 1.5 ) , δ should be chosen to be larger than ‖Z0‖F . For square ( n1 = n2 = n ) matrices , in a stochastic setting in which ( Z0 ) ij are iid N ( 0 , σ2 ) , following [ 9 ] , µ̄ can be chosen as 12σ√n . This follows from 2ν ≥ 1 since ‖U‖2F = r , and somaxi ‖U∗ei‖2 ≥ rn1 . the fact that in this setting n−1/2‖Z0‖ → 2σ almost surely ; setting µ̄ in this fashion ensures that the singular value shrinkage induced by the nuclear norm regularizer ‖ · ‖∗ is greater than the largest singular value of Z0 . In contrast to this σ-dependent penalty parameter , fixing S = S0 , √ PCP formulation ( 1.2 ) requires that 0 ∈ ∂ ( ‖L̂‖∗+µ‖L̂−Z0−L0‖F ) , which translates into−µ L̂−Z0−L0‖L̂−Z0−L0‖F ∈ ∂‖L̂‖∗ . With the hope that L̂ ≈ L0 , and by the subdifferential formulation3 , we have µ ‖Z0‖‖Z0‖F ≈ 1 . The concentration stated above then gives an intuitive choice for µ ≈ nσ 2σ √ n , or µ = c0 √ n for some c0 > 0 . The magic of √ PCP is that by using the Frobenius norm instead of its square , the objective function becomes homogeneous , i.e . the gradient of the penalty term at the ground truthX0 becomes σ independent , making a universal penalty parameter possible . 1.3 Relationship to the Literature . The problem of low-rank matrix recovery from gross sparse corruption can be considered a form of robust PCA , [ 6 , 13 ] , and has been studied extensively in the literature . Algorithmic theory has been developed for both convex [ 6 , 13 , 14 , 15 , 16 ] , and nonconvex optimization methods [ 7 , 8 , 17 , 18 , 19 ] . While many of the aforementioned works pertain to noiseless data , a line of work has studied extensions to noisy data . [ 9 ] studied the problem of robust matrix recovery with bounded noise under the incoherence assumption , and proved a bound on the recovery error , with linear dependence on the noise level but suboptimal dependence on the matrix size . [ 20 ] studied the problem with a weaker assumption about spikiness using decomposable regularizers and restricted strong convexity ( RSC ) , and obtained essentially optimal bounds on the reconstruction error when the noise level is large . These weaker assumptions are not sufficient to ensure exact recovery , and so when the noise standard deviation σ is small , this approach does not yield a reconstruction error proportional to the noise level . [ 10 ] formulated robust PCA as a semidefinite programming problem , which requires strong assumptions about square matrices and positive semidefiniteness of the lowrank matrix . Some other works [ 21 , 22 ] further assumed partial observation of the matrix , and also derived tighter bounds on the recovery error . The recent work of [ 8 ] achieves optimal error bounds for both large and small noise , using a novel analysis that leverages an auxiliary nonconvex program . Taken together , these results give efficient and provably effective methods , whose statistical performance is nearly optimal . Compared to e.g. , [ 8 , 20 ] , the stability guarantees provided by our theory are worse by a dimension-dependent factor . Nevertheless , all of the above works regarding robust PCA with noise , the optimization involves parameters that must be set based on the noise level distribution , and therefore challenging in actual applications . On the other hand , there has been existingwork in the literature on structured signal recovery without needing to know the noise level . Our proposed square root PCP is directly inspired by the square root Lasso [ 11 ] , which proposed the idea of replacing the squared loss with its “ square root ” version . This allows for a choice of the parameter independent of noise level , while maintaining near-oracle performance . Later works have extended this idea to other scenarios , such as group lasso [ 23 ] , SLOPE variable selection [ 24 ] , elastic net [ 25 ] and matrix completion [ 26 ] , etc . Notably , the work of [ 26 ] studied the matrix completion where one aims to recover a low-rank matrix from noisy linear observations , aka matrix completion . Compared with that paper , this work aims to solve a different problem where the observation also contains a sparse outlier matrix . To the best of our knowledge , this paper is the first to propose a provable algorithm for Robust PCA with noisy observation that does not require knowledge of the noise level beforehand . On the algorithmic side , interior point method and first order method are used to solve the square root Lasso in [ 11 ] , while later works apply ADMM to the problem [ 27 ] [ 28 ] . In our problem , the objective function can be transformed into a separable form , making ADMM a reasonable choice . We note that for large σ the error bound established in this paper is suboptimal compared with [ 11 ] and [ 26 ] . The problem of square root lasso enjoys benign properties ( lower bounds on restricted eigenvalues ) which we do not have in square root PCP . The paper [ 26 ] on matrix completion makes a spikiness assumption , and proves that a square root 3The subdifferential of a norm satisfies ∂‖x‖ = { z | 〈x , z〉 = ‖x‖ , ‖z‖∗ ≤ 1 } . lasso-inspired formulation achieves essentially optimal estimation when the noise is large . As with robust matrix recovery , the spikiness assumption is not strong enough to imply exact recovery in the noiseless case . Compared to these works , the principal differences in this paper are ( i ) the problem formulation : we consider robust PCA with sparse errors , ( ii ) the analysis , which proceeds down different lines , and ( iii ) that our bounds are linear in the noise level , for both large and small noise . However , in contrast to [ 26 ] , our analysis does not yield minimax optimal estimation errors ; it is worse by a dimension-dependent factor . Improving this dependence is an important direction for future work .

Summary: This work combines stable Principal Component Pursuit and Square Root Lasso and proposes Square Root Principal Component Pursuit for low-rank + sparse recovery problem. Compared to stable PCP, Square Root PCP has the advantage that the choice of the regularization parameter is noise-independent, so it is easier to be deployed in applications where the noise level is unknown. Theoretical analysis and experiments confirm the effectiveness of the proposed method.