Implicit Bias of Projected Subgradient Method Gives Provable Robust Recovery of Subspaces of Unknown Codimension

Robust subspace recovery ( RSR ) is a fundamental problem in robust representation learning . Here we focus on a recently proposed RSR method termed Dual Principal Component Pursuit ( DPCP ) approach , which aims to recover a basis of the orthogonal complement of the subspace and is amenable to handling subspaces of high relative dimension . Prior work has shown that DPCP can provably recover the correct subspace in the presence of outliers , as long as the true dimension of the subspace is known . We show that DPCP can provably solve RSR problems in the unknown subspace dimension regime , as long as orthogonality constraints -adopted in previous DPCP formulations- are relaxed and random initialization is used instead of spectral one . Namely , we propose a very simple algorithm based on running multiple instances of a projected sub-gradient descent method ( PSGM ) , with each problem instance seeking to find one vector in the null space of the subspace . We theoretically prove that under mild conditions this approach will succeed with high probability . In particular , we show that 1 ) all of the problem instances will converge to a vector in the nullspace of the subspace and 2 ) the ensemble of problem instance solutions will be sufficiently diverse to fully span the nullspace of the subspace thus also revealing its true unknown codimension.We provide empirical results that corroborate our theoretical results and showcase the remarkable implicit rank regularization behavior of PSGM algorithm that allows us to perform RSR without being aware of the subspace dimension . 1 INTRODUCTION . Robust subspace recovery ( RSR ) refers to methods designed to identify an underlying linear subspace ( with dimension less than the ambient data dimension ) in a dataset which is potentially corrupted with outliers ( i.e. , points that do not lie in the linear subspace ) . Many methods for RSR have been proposed in the literature over the past several years Xu et al . ( 2012 ) ; You et al . ( 2017a ) ; Lerman & Maunu ( 2018 ) . Formulations based on convex relaxations and decompositions of the data matrices into a low-rank matrices plus sparse corruptions – either uniformly at random as in Robust PCA ( RPCA ) Candès et al . ( 2011 ) or column-sparse corruptions as in Xu et al . ( 2012 ) ; McCoy & Tropp ( 2011 ) 1 – can , in certain situations , be shown to provably recover the true subspace when the dimension is unknown . However , these theoretical guarantees often require the dimension of the subspace , d , to be significantly less than the ambient dimension of the data , D , and these methods are not suitable to the more challenging high relative subspace dimension regime ( i.e. , when dD ≈ 1 ) . The Dual Principal Component Pursuit approach for RSR . Recently , progress has been made towards solving the RSR problem in the high relative dimension regime by a formulation termed Dual Principal Component Pursuit ( DPCP ) , which is provably robust in recovering subspaces of high relative dimension , Tsakiris & Vidal ( 2018 ) . As implied by its name , DPCP follows a dual perspective of RSR aiming to recover a basis for the orthogonal complement of the inliers ’ subspace . However , a key limitations of DPCP is that it requires a priori knowledge of the true subspace dimension . 1Note that RPCA is closely related but distinct from RSR . In RPCA the corruptions are assumed to be entry-wise in a data matrix , while in RSR the corruptions apply to an entire datapoint ( i.e. , column-wise ) . DPCP for c = 1 . Let X̃ ∈ RD× ( N+M ) denote the data matrix defined as X̃ = [ X O ] Γ where X ∈ RD×N is a matrix containing N inliers as its columns , O ∈ RD×M is a matrix containing M outliers , and Γ is an unknown permutation matrix . DPCP was first formulated in Tsakiris & Vidal ( 2018 ) for handling subspaces of codimension c = D−d equal to 1 ( i.e. , the subspace is a hyperplane with dimension d = D − 1 ) formulated as the following optimization problem : min b∈RD ‖X̃ > b‖1 s.t . ‖b‖2 = 1 . ( 1 ) Note that problem ( 1 ) is nonconvex due to the spherical constraint imposed on vector b ∈ SD−1 which is a normal vector of the D−1 dimensional hyperplane . In Tsakiris & Vidal ( 2018 ) the authors showed that the global minimizer of ( 1 ) will give a normal vector of the underlying true hyperplane as long as both inliers and outliers are well-distributed or the ratio between the number of inliers and number of outliers is sufficiently small . Following a probabilistic point of view , the authors in Zhu et al . ( 2018 ) presented an improved theoretical analysis of DPCP giving further insights on the remarkable robustness of DPCP in recovering the true underlying subspaces even in datasets heavily corrupted by outliers . Moreover , the authors introduced a projected subgradient method which converges to a normal vector of the true subspace at a linear rate . Recursive DPCP for known c > 1 . The authors of Zhu et al . ( 2018 ) also proposed an extension to DPCP which allows for subspaces with codimension c > 1 via a projected subgradient algorithm which attempted to learn c normal vectors to the subspace in a recursive manner . Specifically , after convergence to a normal vector , the projected subgradient algorithm is initialized with a vector orthogonal to the previously estimated normal vector . However , for that approach to be successful the knowledge of the true subspace codimension c becomes critical . Specifically , if an underestimate of the true codimension c is assumed the recovered basis for the null space , B̂ , will fail to span the whole null space , S⊥ . On the other hand , an overestimate of c will lead to columns of B̂ corresponding to vectors that lie in S. Orthogonal DPCP for known c. In Zhu et al . ( 2019 ) , an alternative to ( 1 ) was proposed which attempts to solve for c normal vectors to the subspace at once via the formulation min B∈RD×c ‖X̃ > B‖1,2 s.t . B > B = I . ( 2 ) The authors also propose an optimization algorithm based on the projected Riemannian subgradient method ( RSGM ) , which builds on similar ideas as the projected subgradient method of Zhu et al . ( 2018 ) and enjoys a linear converge rate when the step size is selected based on a geometrically diminishing rule . In Ding et al . ( 2021 ) a theoretical analysis is provided on the geometric properties of 2 showing the merits of DPCP in handling datasets a ) highly contaminated by outliers ( in the order of M = O ( N2 ) ) and b ) subspaces of high relative dimension . However again , it is critical to note that a key shortcoming of this approach is the fact that because all minimizers of problem ( 2 ) will be orthogonal matrices , a prerequisite for recovering the correct orthogonal complement of the inliers subspace is the a priori knowledge of the true codimension c ( see Fig . 1 ) . Contributions . In this work , we address this key limitation by proposing a framework that allows us to perform robust subspace recovery in high relative subspace dimension regime without requiring a priori knowledge of the true subspace dimension . In particular , our proposed approach will be based on the simple approach of solving multiple , parallel instances of the DPCP formulation for solving for a single normal vector to the subspace , min B∈RD×c′ c′∑ i=1 ‖X̃ > bi‖1 s.t . ‖bi‖2 = 1 , i = 1 , 2 , . . . , c′ ( 3 ) where c′ is assumed to be a safe overestimate of c i.e. , c′ ≥ c. Contrary to ( 2 ) , the objective function in ( 3 ) decouples over the columns bi of matrixB = [ b1 b2 · · · bc′ ] and thus can be can be solved in a parallel manner by independently applying a projected subgradient algorithm ( referred to as PSGM ) from c′ different random initialization . Moreover , we observe that with random initialization we can get vectors sufficiently spread on the sphere that lead PSGM ( initialized with those vectors ) to return normal vectors of S. These are all linearly independent when c′ ≤ c and thus can span S⊥ when c′ = c. If c′ > c then PSGM will return c′ − c redundant vectors that will still lie in S⊥ yet they will be linearly dependent ( see Figure 1 ) . That being said , we show that this simple strategy permits us to robustly recover the true subspace even without knowledge of the true codimension c. As is detailed in Sections 3 and 4 , this remarkable behavior of PSGM originates from the implicit bias that is induced in the optimization problem due to a ) the relaxation of orthogonality constraints in ( 3 ) and b ) the random initialization scheme that is adopted . Our specific contributions are as follows : 1 . First , we focus on a continuous version of ( 3 ) i.e. , in the case that inliers and outliers are distributed under continuous measures which induces a benign landscape in DPCP and hence is easier to be analyzed . We prove that DPCP problem formulated as in ( 3 ) can be solved via a projected subgradient algorithm that implicitly biases solutions towards low-rank matrices B̂ ∈ RD×c′ whose columns are the projections of the randomly initialized columns of B0 onto S⊥ . As a result , B̂ almost surely spans S⊥ as long as it is randomly initialized with c′ ≥ c. 2 . Second , we analyze the discrete version which is more challenging , yet of more practical interest , showing that iterates of DPCP-PSGM converge to a scaled and perturbed version of the initial matrixB0 . This compelling feature of DPCP-PSGM allows to derive a sufficient condition and a probabilistic bound guaranteeing when the matrix B̂ ∈ RD×c′ spans S⊥ . 3 . We provide empirical results both on simulated and a real datasets , corroborating our theory and showing the robustness of our approach even without knowledge of the true subspace codimension . 2 RELATED WORK . Subspace Recovery . Learning underlying low-dimensional subspace representations of data has been a central topic of interest in machine learning research . Principal Component Analysis ( PCA ) has been the most celebrated method of this kind and is based on the minimization of the perpendicular distances of the data points from the estimates linear subspace , Jolliffe & Cadima ( 2016 ) . Albeit , it is originally formulated as nonconvex optimization problem , PCA can be easily solved in closed form using a singular value decomposition ( SVD ) operation ( see e.g . Vidal et al . ( 2016 ) ) . Despite its great success , PCA is prone to failure when handling datasets that contain outliers i.e. , data points whose deviation from the inliers ’ subspace is “ large ” in the ` 2 norm sense . Robust Subspace Recovery ( RSR ) . To remedy this weakness of PCA robust subspace recovery ( RSR ) methods attempt to identify the outliers in the dataset an recover the true underlying lowdimensional subspace of the inliers Lerman & Maunu ( 2018 ) ; Maunu et al . ( 2019 ) . A classical approach to this problem is RANSAC Fischler & Bolles ( 1981 ) , which is given a time budget and randomly chooses per iteration d points and then fits a d-dimensional subspace to those points and checks how the proposed subspace fits the remaining data points . RANSAC then outputs the subspace that agrees with the largest number of points . However , RANSAC ’ s reliance on randomly sampling points to propose subspaces can be highly inefficient when the number of outliers is high ( as well as the fact that RANSAC also needs knowledge of the true subspace dimension , d ) . The need to tackle inherent shortcomings of RANSAC pertaining to computational complexity issues inspired alternative convex formulations of RSR , Xu et al . ( 2012 ) ; You et al . ( 2017b ) ; Rahmani & Atia ( 2017 ) ; Zhang & Lerman ( 2014 ) . In Xu et al . ( 2012 ) the authors decompose the data matrix as a sum of a low-rank and a column-sparse component . However , theoretical guarantees obtained for convex formulations only hold for subspaces of relatively low-dimensional subspaces i.e. , for d D where d and D denote the subspace and the ambient dimension , respectively . To the best of our knowledge , existing RSR algorithms rely heavily on one of two key assumptions . 1 ) The subspace is very low-dimensional relative to the ambient dimension ( d D ) or 2 ) The subspace dimension is a priori known . Undoubtedly , the second hypothesis is rather strong in real-world applications , and many applications also do not satisfy the first assumption . Moreover , heuristic strategies for selecting the dimension of the subspace are hard to be applied in the RSR setting since they incur computationally prohibitive procedures , Lerman & Maunu ( 2018 ) . Relation to Orthogonal Dictionary Learning ( ODL ) . Note that objective functions in the form of ( 3 ) show up beyond RSR problems i.e. , in orthogonal dictionary learning ( ODL ) , sparse blind deconvolution , etc. , Qu et al . ( 2020 ) . Specifically , based on a similar formulation the authors in Bai et al . ( 2019 ) proved that c′ = O ( c log c ) independent random initial vectors suffice in order to recover with high probability a dictionary of size D × c with high accuracy . In this paper we aim to recover a basis of the orthogonal complement of a subspace of unknown dimension instead of accurately estimating a dictionary hence our goal differs from that in Bai et al . ( 2019 ) . Implicit bias in Robust Recovery Problems . The notions of implicit bias and implicit regularization have been used interchangeably in the nonconvex optimization literature for describing the tendency of optimization algorithms to converge to global minima of minimal complexity with favorable generalization properties in overparameterized models , Gunasekar et al . ( 2018 ) . In the context of robust recovery , the authors in You et al . ( 2020 ) showed that Robust PCA can be suitably reparametrized in such a way to favor low-rank and sparse solutions without using any explicit regularization . In this work , we use the term implicit bias for describing the convergence of DPCPPSGM to low-rank solutions , which are not necessarily global minimizers , that span the orthogonal complement of the subspace when a ) orthogonality constraints in DPCP formulation are relaxed b ) DPCP is overparameterized i.e. , c′ ≥ c and c ) PSGM randomly initialized . To the best of our knowledge our work is the first that explores implicit bias in projected subgradient algorithms that minimize objective functions with sphere/orthogonality constraints .

Summary: This paper discusses solving robust subspace recovery by solving parallel versions of $\min_b$ $\|X^Tb\|_1$ st $\| b\|_2=1$, for random initializations of $b$, using projected subgradient method. The argument is that using this  method, in the limit where the number of inliers greatly outnumber the outliers, that if the inlier dimension is c, then c random initializations will converge on identifying this dimension with probability 1. Experimental results and proofs are given to prove this idea.