Implicit Regularization in Matrix Sensing via Mirror Descent

1 Introduction . Matrix sensing represents a paradigm in modern statistics [ 8 , 26 , 27 ] , with applications ranging from image compression [ 2 ] to collaborative filtering [ 18 ] and dimensionality reduction [ 34 ] , for instance . The goal is to recover a rank-r matrix X ? ∈ Rn×n′ from a set of linear measurements yi = 〈Ai , X ? 〉 , i = 1 , . . . , m , where the sensing matrices Ai ∈ Rn×n ′ are observed . This formulation includes the problem of matrix completion , where a subset of the entries of the matrix X ? is observed . Most of the literature on matrix sensing is based on some form of explicit regularization or rank constraint to encourage or enforce low-rankness of the estimated matrix . A popular approach is based on minimizing the nuclear norm or on using explicit regularization techniques based on the nuclear norm , e.g . [ 5 , 15 , 17 , 23 , 26 , 27 , 30 ] . Another popular approach is based on non-convex optimization and the low-rank factorization X = UV > with matrices U ∈ Rn×r , V ∈ Rn′×r , where the factorization itself enforces the low-rankness of X , e.g . [ 7 , 8 , 16 , 21 , 22 , 29 , 31 , 39 ] . The literature on implicit regularization for matrix sensing is more recent and less well developed . When X ? is assumed to be a positive semidefinite matrix , it was first empirically observed in [ 14 ] that mininizing the unregularized empirical risk using vanilla gradient descent with parametrization X = UU > and random full-rank initialization close to zero yields a low nuclear norm solution , even when U ∈ Rn×n , i.e . when no constraint on the rank of X is enforced . It was later proved that , with this parametrization , gradient descent minimizes the nuclear norm under the assumption that the sensing matrices Ai ’ s commute [ 14 ] , or when the sensing matrices satisfy a restricted isometry property [ 19 ] . Gradient descent with low-rank initialization within a specific “ capture neighborhood ” has been studied in [ 9 ] , which ensures that the iterates of the algorithm stay low-rank . When X ? is a general rectangular matrix , implicit regularization in matrix sensing has been studied through 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . the lenses of deep matrix factorization in [ 3 , 12 , 20 , 25 ] , and empirical and theoretical evidence is provided which suggests that a notion of rank-minimization is involved in the implicit bias of gradient descent . However , these works do not establish an explicit characterization of the limiting point of the optimization algorithm , e.g . in the form of a quantity that is minimized among all minimizers of the empirical risk . In the special case of full-observation matrix sensing , gradient flow has been shown to learn solutions with gradually increasing rank [ 11 ] . Most theoretical results on implicit regularization in matrix sensing consider the continuous-time dynamics ( i.e . gradient flow ) , e.g . [ 3 , 9 , 12 , 14 , 20 , 25 ] , or assume an infinitesimally small initialization , e.g . [ 14 , 20 ] . Notable exceptions are [ 11 ] , which makes a commutativity assumption on the data matrices in discrete time , and [ 19 ] , which assumes a restricted isometry property that leads to a suboptimal sample complexity when the sensing matrices belong to a general class of random matrices [ 27 ] . In the context of linear neural networks , the dependence of the implicit bias of gradient descent on the initialization has been studied in [ 4 , 24 , 35 , 38 ] . For instance , linear diagonal networks with shared weights were considered in [ 35 ] , where it was shown that gradient descent minimizes a quantity which corresponds to the ` 1-norm in the limit α → 0 and to the ( weighted ) ` 2-norm in the limit α→∞ , where α denotes the initialization size . This result was later generalized in [ 38 ] using a tensor formulation , which allows for architectures including linear diagonal networks and linear full-length convolutional networks . However , these results focus on vector-based notions of norm-like functions that do not capture matrix-based quantities typically of interest in matrix sensing . 1.1 Our contributions . We study the implicit bias of discrete-time mirror descent in matrix sensing in both the general case of rectangular matrices and the particular case of positive semidefinite matrices . Under the only assumption on the sensing matrices Ai ’ s that there exists a matrix achieving zero training error , we characterize the limiting point as the matrix that minimizes a quantity which interpolates between the nuclear norm and Frobenius norm , parametrized by the mirror map parameter , in the rectangular case ; and which is a linear combination of the nuclear norm and the negative von Neumann entropy , parametrized by the initialization size , in the positive semidefinite case . Compared to results on implicit regularization for gradient descent , our framework for mirror descent is simple , and the same analysis yields results for both the case of general rectangular and positive semidefinite matrices . In the general case of rectangular matrices , we show that mirror descent , initialized at zero and equipped with the spectral hypentropy mirror map [ 10 ] parametrized by β > 0 , among all global minimizers of the empirical risk , converges to a matrix that minimizes a quantity interpolating between the nuclear norm in the limit β → 0 and the Frobenius norm in the limit β → ∞ . As a consequence , our result implies that , for β → 0 , mirror descent can recover a rank-r matrix X ? under the same set of assumptions that is sufficient for nuclear norm minimization to be successful , namely when the sensing matrices Ai ’ s satisfy the restricted isometry property with restricted isometry constant smaller than some absolute constant [ 27 ] , or if X ? satisfies an incoherence condition and r ( n + n′ ) ( modulo constants and logarithmic term ) random entries of X ? are observed [ 26 ] . To the best of our knowledge , this is the first recovery guarantee for an implicit regularization-based algorithm that does not explicitly enforce low-rankness in general rectangular matrix sensing . In the particular case of positive semidefinite matrices , we can alternatively consider the spectral entropy mirror map and show that mirror descent , initialized at αI for any α > 0 , converges to a positive semidefinite matrix that minimizes a linear combination of the nuclear norm and the negative von Neumann entropy , where the relative weights are controlled by the initialization size α . While the limit α→ 0 corresponds to minimizing the nuclear norm as in the case of rectangular matrices , the limit α → ∞ corresponds to maximizing the nuclear norm . This also translates into guaranteed recovery of a low-rank matrix X ? for α → 0 under the same assumptions that are sufficient for nuclear norm minimization [ 26 , 27 ] . A comparable result for gradient descent with full-rank factorized parametrization has been established in [ 19 ] under a stronger assumption on the restricted isometry constant , which is assumed to be smaller than a quantity depending on both the rank and the condition number of the matrix X ? and translates into a sub-optimal sample complexity . We establish our results using a potential-based analysis for mirror descent in terms of the Bregman divergence , which provides an alternative proof technique to characterize the limiting point of mirror descent compared to the analysis based on KKT optimality conditions used in [ 13 ] . As a by-product , our proof of Theorem 1 yields an alternative proof of Theorem 1 in [ 13 ] . The advantage of our approach is that convergence of mirror descent does not need to be assumed a-priori and can instead be established using convexity of the empirical risk . The analysis in terms of the Bregman divergence is not limited to convex settings and has been applied to non-convex problems , e.g . [ 36 , 37 , 40 ] , and hence can be of more general interest to investigate the phenomenon of implicit bias . In the case of square matrices , we show , assuming that the sensing matrices Ai ’ s are symmetric and commute , that gradient descent with full-rank parametrization X = UU > −VV > , U , V ∈ Rn×n , is a first-order approximation to mirror descent equipped with the spectral hypentropy , where the initialization size corresponds to the mirror map parameter β . A similar connection between mirror descent and reparametrized gradient descent has been established in the vector-case [ 1 , 10 , 33 , 36 ] . Similarly , we show that gradient descent with full-rank parametrization X = UU > , U ∈ Rn×n , is a first-order approximation to mirror descent equipped with the spectral entropy when the sensing matrices are symmetric and commute , thus recovering a generalization of Theorem 1 in [ 14 ] which holds for any positive initialization size α > 0 ( rather than in the limit α→ 0 ) . We present numerical simulations which suggest that , in some regimes , our results on the dependence on the initialization size α for mirror descent might be indicative for the behavior of gradient descent , even when the sensing matrices Ai ’ s do not commute . More precisely , the final estimates of gradient descent and mirror descent closely track each other when the number of measurements is sufficiently large for nuclear norm minimization to recover a planted low-rank matrix , while gradient descent seems to put more emphasis on lowering the effective rank [ 28 ] at the expense of a higher nuclear norm when fewer measurements are available , which supports the empirical observations in [ 3 , 20 ] . 2 Background . We begin by introducing some notation used throughout this paper . We use boldface uppercase letters to denote matrices and boldface lowercase letters to denote vectors . We write ‖ · ‖∗ , ‖ · ‖F and ‖ · ‖2 for the nuclear , Frobenius and spectral norm , respectively , and denote by 〈X , Y〉 = tr ( X > Y ) the standard Frobenius inner product . We write Sn , Sn+ ⊆ Rn×n for the set of symmetric and positive semidefinite matrices , respectively . Without loss of generality , we will always assume n ≤ n′ . We first give a brief overview of unconstrained mirror descent ( with matrix arguments ) . Let D ⊆ Rn×n′ be an open convex set . We say that Φ : D → R is a mirror map if it is strictly convex , differentiable , and its gradient takes all possible values , i.e . { ∇Φ ( X ) : X ∈ D } = Rn×n′ . Given a mirror map Φ , the associated Bregman divergence is defined as DΦ ( X , Y ) = Φ ( X ) − Φ ( Y ) − 〈∇Φ ( Y ) , X−Y〉 . ( 1 ) Then , the mirror descent algorithm to minimize a function f : D → R is defined by the update ∇Φ ( Xt+1 ) = ∇Φ ( Xt ) − ηt∇f ( Xt ) , ( 2 ) where ηt > 0 is a sequence of step sizes . We approach the problem of matrix sensing , where we are given measurements { Ai , yi } mi=1 , by minimizing the unregularized empirical risk f ( X ) = 1 2m m∑ i=1 ( 〈Ai , X〉 − yi ) 2 ( 3 ) using mirror descent equipped with the spectral hypentropy mirror map , which is defined as [ 10 ] Φβ ( X ) = n∑ i=1 σi arcsinh ( σi β ) − √ σ2i + β 2 , ( 4 ) for some β > 0 , where { σi } ni=1 denote the singular values of X . We provide expressions for the gradient∇Φβ and discuss the per-iteration computational cost of the corresponding mirror descent algorithm in Appendix A . If the optimization is restricted to positive semidefinite matrices X ∈ Rn×n , we can replace Rn×n′ with Sn in above definitions and consider the spectral entropy ( using the convention 0 log 0 = 0 ) Φ ( X ) = tr ( X log X−X ) , ( 5 ) which is well-defined on the set of positive semidefinite matrices Sn+ .

Summary: In this manuscript, the authors study the trajectory of mirror descent to optimize the unregularized empirical risk functional to solve low-rank matrix estimation problems. They characterize the trajectory of mirror descent with different mirror maps and showcase an implicit bias towards certain structures in solutions. In particular, when equipped with either the spectral hypentropy map or spectral entropy map, they show that mirror descent converges to a global minimizer that minimizes a particular quantity depending on the singular values of the resulting matrix. This quantity interpolates between the nuclear norm and Frobenius norm for the spectral hypentropy map and is a linear combination of the nuclear norm and von Neumann entropy for the spectral entropy map. They also show algorithmic guarantees for mirror descent to solve matrix sensing and matrix completion problems that operate with sample complexities on par with traditional nuclear norm minimization approaches, without explicit regularization. Finally, a connection between gradient descent over the parameterization $UU^T - VV^T$ and mirror descent is shown and toy experiments show mirror descent and gradient descent behave similarly.