Implicit Regularization in Matrix Sensing via Mirror Descent

Abstract: We study discrete-time mirror descent applied to the unregularized empirical risk in matrix sensing. In both the general case of rectangular matrices and the particular case of positive semidefinite matrices, a simple potential-based analysis in terms of the Bregman divergence allows us to establish convergence of mirror descent---with different choices of the mirror maps---to a matrix that, among all global minimizers of the empirical risk, minimizes a quantity explicitly related to the nuclear norm, the Frobenius norm, and the von Neumann entropy. In both cases, this characterization implies that mirror descent, a first-order algorithm minimizing the unregularized empirical risk, recovers low-rank matrices under the same set of assumptions that are sufficient to guarantee recovery for nuclear-norm minimization. When the sensing matrices are symmetric and commute, we show that gradient descent with full-rank factorized parametrization is a first-order approximation to mirror descent, in which case we obtain an explicit characterization of the implicit bias of gradient flow as a by-product.


Reviews and author responses:

 Dear authors,

thank you for your responses. I have now raised my score to 7. 



The authors analyze the convergence of  mirror descent  in matrix sensing for specific selection of mirror maps. Specifically, they prove that when the mirror map is the spectral hypentropy or spectral entropy then MD converges to global minimizers of the empirical objective minimize quantities related to the nuclear, the Frobenius or the Von Neumann entropy.   
Strengths
- The main contribution of the paper is the derivation of Theorems 1 and 2 which establish convergence of mirror descent (MD) to global minimizers of the empirical loss of the matrix sensing problem that minimize certain quantities related to nuclear norm, Frobenius norm and von Neumann entropy. These results are interesting and provide an insight into the interplay between the geometries induced in mirror descent and the implicit regularization phenomena that are brought up. 
- Theorems 3 and 4 present recovery guarantees for the proposed mirror descent algorithm, the first of this kind in the framework of implicit regularization.

Weaknesses
- The contribution of these results to already existing work is somewhat limited given Theorem 1 of [19], which proves that MD converges to minimizers of the Bregman divergence. That being said, though the authors provide an alternative proof and focus on specific mirror maps, the added knowledge seems not to be  that significant.  The authors should make more clear the advantage of their proof as compared to the one followed in Theorem 1 of [19]. 
- From a practical point of view, the selection of the mirror maps gives rise to computationally expensive SVD steps per iteration of the algorithm. That being said, it is so clear what are the advantages of an implicitly regularized algorithm compared to an explicitly regularized one.

Minor comments 

- The first 3 paragraphs of Section 1.1  present the contribution without explicitly referring to the added knowledge to the existing literature. The authors should rewrite that part and clarify what point is new and what is not.
- Typos: 
 - line 105 : numerical
- line 109 : large
- line 146: identity
-line 164: descent
-line 345: regularization Yes

 Thank you for your response. After reading the other reviewers' assessments, I would like to keep my score.


In this manuscript, the authors study the trajectory of mirror descent to optimize the unregularized empirical risk functional to solve low-rank matrix estimation problems. They characterize the trajectory of mirror descent with different mirror maps and showcase an implicit bias towards certain structures in solutions. In particular, when equipped with either the spectral hypentropy map or spectral entropy map, they show that mirror descent converges to a global minimizer that minimizes a particular quantity depending on the singular values of the resulting matrix. This quantity interpolates between the nuclear norm and Frobenius norm for the spectral hypentropy map and is a linear combination of the nuclear norm and von Neumann entropy for the spectral entropy map. They also show algorithmic guarantees for mirror descent to solve matrix sensing and matrix completion problems that operate with sample complexities on par with traditional nuclear norm minimization approaches, without explicit regularization. Finally, a connection between gradient descent over the parameterization $UU^T - VV^T$ and mirror descent is shown and toy experiments show mirror descent and gradient descent behave similarly.  # Originality

There has been a flurry of interest in understanding implicit bias in matrix factorization problems, and this work provides an interesting perspective by studying the solutions chosen by mirror descent methods for various mirror maps. Similar results have been reported in [13], but the proof techniques in this work (direct analysis of mirror descent trajectory) and [13] (KKT conditions) differ from one another. The precise formulae presented on which minimizers are chosen appear to be novel as well. The shown equivalence between exponentiated gradient and mirror descent is nice, but not entirely novel as it appears in [35] for the vector-valued case.

# Quality

- Strengths: - The paper is well-written. - To the reviewers knowledge, applying mirror descent to solve matrix factorization problems and analyzing its implicit bias towards low-rank solutions appears novel. Moreover, the characterization of their preferred solutions holds in fairly general settings (e.g., no assumptions on the measurement matrices $A_i$ aside from the existence of a $0$ loss matrix). - The results on recovery guarantees for matrix sensing/completion hold with sample complexity on par with convex nuclear norm minimization techniques. - The connection between exponentiated gradient and mirror descent is interesting and extends previously known results to show the convergence of mirror descent to small nuclear norm solutions in the small initialization scale regime. 

- Weaknesses: - Mirror descent would not be a practical algorithm to use due to computational complexity and seems to perform essentially the same as standard gradient descent - No convergence rates are provided in the theoretical analysis of applying mirror descent to matrix sensing/completion. The results only establish convergence to a neighborhood of the underlying low-rank matrix. Moreover, these neighborhoods depend logarithmically on the parameters $\beta$ (for hypentropy) and $\alpha$, the initialization size, for the entropy map so that to achieve $\epsilon$ precision, one would need $\beta$ or $\alpha$ to behave like $\exp(-1/\epsilon)$. In addition, in the matrix completion setting, a term depending on the sample complexity appears in the estimation bound, which would require $m = \Omega(nn’\log^2(n’))$ to make small, but the theorem operates in the $\Omega(r (n + n’)\log^2(n’))$ regime. Hence, while the sample complexity in the theorem does scale in the same way as nuclear norm approaches, the resulting error bounds could potentially be quite large even for exponentially small $\beta$ or $\alpha$. Could the authors comment on this point and, perhaps, why the error bound scales this way? My main concern is that the bound may be vacuous when operating in the traditional nuclear norm sample complexity regime.

# Clarity

Overall, the paper is well-written and easy to follow. Related work seems to be cited well and the authors do a good job of discussing their results in relation to others on implicit bias in matrix factorization. Here are a couple of typos I found:
- pg 1 line 33 “when then” -> “when the”, pg 3 line 105 “numercal” -> “numerical”, pg 3 line 109 “largeg” -> “large”

# Significance

This work shows that mirror descent in matrix estimation problems exhibits an implicit bias towards certain structured solutions, and precisely characterizes this structure. I think that this does inspire research into further analyzing the role of mirror maps in other estimation problems and in analyzing the potential implicit bias of other first order algorithms. There are no clear societal impacts of this work. The authors discuss some of the downsides and limitations of their approach (namely, the computational cost of applying mirror descent). Another limitation that should also be discussed is the potential looseness of the error bounds.

 Dear authors,

Thank you for the response. After reading the other reviews and author responses, I have decided to raise my score. I think that the paper provides a nice contribution in showcasing further evidence of mirror descent’s implicit bias, and has the potential to inspire future work.

 We thank the reviewer for their feedback. The primary focus of our paper was indeed to understand how the implicit bias of mirror descent can be leveraged to solve matrix recovery problems rather than to argue that mirror descent is a computationally competitive algorithm for this problem. Typically, algorithms based on implicit regularization (i.e. mirror descent or gradient descent with full-rank factorized parametrization) cannot match the computational efficiency of gradient descent with low-rank factorized parametrization (see lines 354-356). Please find our responses to the questions below.

1. We thank the reviewer for pointing this out and we have made the dependence on $\beta$ more explicit in Theorem 1. We remark that in Theorems 3 and 4, where an upper bounds on $\beta$ is assumed, the step size $\eta$ can be chosen independently from $\beta$.

2. Indeed, a smaller value for $\beta$ leads to a slower convergence of mirror descent (initially). This can be perhaps most directly seen from Proposition 5, which shows that $\beta$ corresponds to the initialization size in the exponentiated gradient algorithm defined in (15) (under the assumption that the sensing matrices are symmetric and commute). Hence, a small value for $\beta$ means that the algorithm defined in (15) does not move much initially, as both $\mathbf{U}_t$ and $\mathbf{V}_t$ start from small initial values and are updated multiplicatively. While a rigorous analysis of the convergence speed of mirror descent was outside the scope of our paper, we remark that precise convergence speed guarantees showing a similar behaviour have previously been established for mirror descent (in the vector-case) for the problem of sparse phase retrieval in [35].

3. We believe that there is indeed a tradeoff between computational complexity and statistical error controlled by $\beta$, similar to the tradeoff established in the context of sparse phase retrieval in [35]. We point out that the logarithmic dependence on $\beta$ and $\alpha$ is likely not sharp as we observed in our experiments (we observed similar results for the dependence on $\beta$, and we omitted those experiments given the similarity to the experiments in Section 6 and Appendix C).

4. The analysis of the evolution of the Bregman divergence is a general tool that can be employed to study the trajectory of mirror descent. Beyond characterizing the implicit bias of mirror descent, it has been previously used to establish the convergence of mirror descent for a general class of non-convex optimization problems [39], and, more specifically, the potential-based analysis of mirror descent has been used to establish minimax-optimal rates of convergence in noisy sparse phase retrieval [36], for instance. We remark that in previous related works that establish convergence speed guarantees, e.g. [19, 35, 36], a sample complexity that scales quadratically in the respective notions of sparsity in matrix sensing and sparse phase retrieval is needed, while our results only require a linear sample complexity, but do not establish any convergence speed guarantees for the estimation error. We will add a discussion on this point in the main paper, and we hope that our work can inspire future research in this direction, possibly bridging this gap and establishing rigorous convergence speed guarantees when only a sample complexity that scales linearly in the respective notion of sparsity is assumed.

 We thank the reviewer for their feedback. Please find our responses below.

1. We believe that the reviewer is referring to Theorem 1 of [13]. The main advantages of our proof technique over the KKT optimality conditions-based proof of Theorem 1 in [13] are 1) the fact that convergence of mirror descent does not need to be assumed, and 2) that this type of potential-based analysis of mirror descent is also applicable to more general non-convex problems (e.g. [39]) and can, in principle, be used to establish convergence speed guarantees (e.g. [35, 36]), see also lines 88-95. We remark that in previous related works that establish convergence speed guarantees, e.g. [19, 35, 36], a sample complexity that scales quadratically in the respective notions of sparsity in matrix sensing and sparse phase retrieval is required, while our results only require a linear sample complexity, but do not establish any convergence speed guarantees for the estimation error. We will add a discussion about this in the main paper, and we hope that our work can inspire future research in this direction, possibly bridging this gap and establishing rigorous convergence speed guarantees when only a sample complexity that scales linearly in the respective notion of sparsity is assumed. 

2. Indeed, mirror descent is computationally more expensive than gradient descent due to an SVD in each iteration of the algorithm. We point out that, as elaborated in lines 344-356 and Section A of the Appendix, the SVD can be avoided in the positive semidefinite case, where we only need to compute matrix exponentials, which, though more expensive than matrix multiplication, are typically cheaper to compute than an SVD.

 We thank the reviewer for their feedback. We fully agree with the reviewer and believe that both points are very interesting directions for future research.

 We thank the reviewer for their feedback.

Regarding the logarithmic dependence of the error bounds in Theorems 3 and 4 on $\beta$ and $\alpha$: even though the error bounds in equations (13) and (14) involve a term of the form $\frac{nn’\log^2n’}{m}$, both bounds tend to zero for any $n,n’,m$ in the limit $\beta,\alpha\rightarrow 0$. While this would require an impractically small $\beta$ or $\alpha$, our error bounds demonstrate that mirror descent can recover a planted matrix $\mathbf{X}^\star$ from $\Omega(r(n+n’)\log^2(n’))$ samples (exactly in the limit $\beta,\alpha \rightarrow 0$, which is a setting often considered in the literature, e.g. [14, 20, 34]).

We obtain the logarithmic dependence on $\beta$ and $\alpha$ since our error bounds are established by 1) bounding the nuclear norm of the limiting point of mirror descent using Theorems 1 and 2, and 2) using this bound on the nuclear norm to control the estimation error $||\mathbf{X}_t-\mathbf{X}^\star||_F$. In step 1), the upper bound on the nuclear norm inherits the logarithmic dependence on $\beta$ and $\alpha$ from the characterization of the limiting point of mirror descent in Theorems 1 and 2 (which is exact).

As we point out in lines 246-252, error bounds with a polynomial dependence on the initialization size $\alpha$ as well as convergence speed guarantees have been established using additional assumptions for gradient descent in [19]. We expect that the analysis of [19] can be adapted to mirror descent, since the operations $\exp$ and $\log$ leave the eigenspaces of symmetric matrices invariant, and we expect a similar adaptive decomposition of $\mathbb{S}^n_+$ into a rank-$r$ subspace, to which $\mathbf{X}_t$ is approximately confined, and a rank-$(n-r)$ subspace where $\mathbf{X}_t$ remains small, to be possible. In the present paper, our focus was on understanding how the implicit bias of mirror descent can be leveraged to solve matrix sensing and completion, under assumptions matching those required by nuclear norm minimization (in particular assuming a sample complexity that scales linearly in the rank $r$ of $\mathbf{X}^\star$ rather than quadratically as in [19]). We will add a discussion about this in the main paper, and we leave an analysis that establishes error bounds that do not depend polynomially on $\beta$ and $\alpha$, and convergence guarantees, desirably without additional assumptions compared to Theorems 3 and 4 (in particular with a linear sample complexity), to future work. 

The paper provides the convergence analysis of mirror descent for matrix sensing to particular minimum norm solutions. The updates that are studied are induced by the hypentropy and von Neumann divergences. The authors analyze multiple interesting cases including matrix completion and the case where the instances satisfy the RIP. They also provide some experimental evaluations to validate their findings.  The paper studies an interesting problem and greatly generalizes the results in [13] and [14]. There are a few remarks about the analysis:

1) Although hypentropy is an interesting divergence, it is only an approximation of the squared Euclidean and KL divergence in the limit cases. A more natural extension is the tempered KL divergence which was partially analyzed in [1]. I believe most of the analysis may carry over.

2) The construction in Eq. (15) is interesting, but not exactly related to the reparameterization of the Exponentiated Gradient Unnormalized (EGU) algorithm. The exact reparameterization for the vector case was discussed in (Amid and Warmuth, 2020) along with regret bounds for the discrete case. It would be interesting to extend the analysis to the matrix case where matrix U is updated using gradient descent.

Reference:
Ehsan Amid and Manfred K. Warmuth. "Winnowing with Gradient Descent". COLT, 2020. Yes

This paper studies the implicit regularization of a particular algorithm, mirror descent, for matrix recovery problems.   I think the paper presents an interesting phenomenon that is also present in mirror descent.  I believe the paper only attempts to explain some phenomenon in mirror descent, rather than arguing that the mirror descent method is the correct or a competitive method in solving matrix recovery problems.   

I have a few questions that I would like to hear the author's responses. 

1. In theorem 1, does the stepsize eta depends on the parameter beta as well? 

2. I wonder how beta affects the algorithmic performance, i.e., the convergence speed of the algorithm? Can the author comment on the relationship between the magnitude of beta and the speed of the mirror descent method?

3. I notice that the bias/difference between X_infty and X* in Theorem 3 and 4 depends logarithmically on the parameter beta. So one wants to choose a very smaller beta to reduce the bias. It seems in Theorem 4, one needs beta to be exp(-n^(1/2)) so the RHS of (13) can be smaller than ||X*||_*. Is there a tradeoff between computational complexity, and statistical error, i.e., the difference between  X_infty and X*,  in choosing beta? 

4. I think characterizing the implicit regularization phenomenon is interesting by itself. But I wonder whether there are any broader applications/impacts of the results or techniques used in the paper.  A few limitations are discussed in the end. 

Meta-review: All reviewers are satisfied by the authors' response and agree that this is well-written paper with solid contributions, though some of the results could be considered incremental given the existing work in [13,35]. The authors are encouraged to provide additional discussion on the points raised by the reviewers.

Decision: accept