reviews: 
Comments: This paper studies the optimization landscape of regularized stochastic gradient descent applied to matrix completion with linear networks problems (which is equivalent to matrix completion with a $2/L$ Shatten norm regularizer). Several properties of the optimization landscape are proved, including the fact that the only critical points of the optimization problem over the factor matrices (minimizing $\mathcal{L}_{\lambda}(\theta)$ must be local minima of the optimization problem over the full matrix $A$, unless they are strict saddle points in the original problem.

 In addition, it is shown that if gradient flow converges to a global minimum, then a version of gradient flow with a sufficiently small regularization parameter will converge to a minimum with a larger rank than the ground truth. Arguably the most significant final result is that stochastic gradient descent jumps from high rank local minima to lower rank local minima, with the jumps being one directional: one cannot return to a higher rank region after entering a lower rank region. Here, the lower rank regions should be understood as defined on page 5, in an approximate sense. Throughout the proofs, the fact that the local minima of the optimization problem over $\mathcal{L}_\lambda(\theta)$ must be balanced (cf. Proposition A.1). An approximate version of this condition is also present in the definition of the low absorbing low rank spaces in the main results of the paper.
Strengths: The main paper is well-written and the results appear generally sound. The results are of great importance to the field and interest to the community. **This is highly non trivial and important work**.
Weaknesses: Although the main paper is well written, the **proofs are not reader-friendly** at all. 

The writing of the proofs is very terse and laconic, omitting many details. Although this is reminiscent of some great pure mathematics papers that were ahead of their time and I enjoyed the challenge some of the time, I strongly believe this style should only be considered acceptable if there is absolutely zero tolerance for any errors or inaccuracies whatsoever. I don't think the proofs actually stand up to this amount of scrutiny: there are **at least a few typos, minor errors and imprecisions** in the subset of the proofs I was able to look at, and since a lot of information is left out for the reader to figure out, the additional presence of even a small number of actual errors dramatically expands the "search space" from the point of view of the reader.  I would really like to see a substantial revision of the paper with more detailed and careful proofs (and maintaining my score is conditional on that). 

For instance, in page 12, point "(0)", the definition of the $U_i, V_i$ is not really consistent: the index under the $U,V,S$ is used both to mean the iteration step in the sequence and the position in the product $W^L...W^1$. 

In addition, in page 13, consider the following statement the authors make " as $\lambda\rightarrow 0$, the critical points of the loss move continuously. Consider a continuous path of critical points, as $\lambda\rightarrow 0$, it converges to..."
Although the argument makes sense intuitively, filling in the gaps with rigorous proofs is definitely beyond the scope of what can be expected of the reader to do. At least some citations are a minimum. I doubt that simple continuity is enough to guarantee convergence (even if a subsequence converges, the path could oscillate widely), probably the only way to rigorously prove the statement is to use a quantitative version of the statement relying on calculus of variations. 

This is not the only example. In point (1) in page 14, the authors say "the singular value ..... must converge to a non zero eigenvalue".  It is not clear **why this is the case**, or why the the *singular value* turns into an *eigenvalue* after convergence. Far more details are required. 

In the middle of page 14, it is hard to imagine that the equation $U_{\ell,i}(\lambda)U_{\ell-1,i}(\lambda)$ can be correct without **at least a transpose missing**. Of course, the lack of a rigorous and consistent definition of $U_{\ell,i}$ does not help here. 

At the bottom of page 14: the line starting with "other directions" ends with " $L-1,)$" and a few lines below we have the equation $U_\ell^\top dU_{\ell}+ = -dU_\ell^\top U_\ell$. What does "+=" mean here? The same issue is present in many other parts of the paper, including in the third line of text on page 15.

Towards the end of Appendix A in page 17, the term "saddle to saddle" is mentioned with absolutely no explanation or citation. 

In the middle of page 13, the authors use the fact that "a matrix cannot be approached with matrices of strictly lower rank", which is true but should probably warrant a citation since the equivalent statement is not true for tensors. 


The proof of Proposition A4 is very hard to make sense of without further information: the first sentence is ""let A(\lambda) be path of global minima restricted to the set of matrices of rank $r^*$ or less." how do you construct the path? Even for $L=2$, there can be a continuous set of global minima of local intrinsic dimension higher than 2, how do you use the axiom of choice to construct a "path"? 
Sentences such as "going along directions that increase the rank of $A(\lambda)$, the regularization term increases at a rate of $d^{2/L}$ for $d$ the distance" definitely need more mathematical details. 

Similarly, the statement about $\phi$ being differentiable in the directions which do not change the rank should be made more precise (although I agree with it, probably at least a citation to [1] is a minimum) 

For proposition A.5, the proof starts with the following sentence "We know that L2 regularized GF $\theta_\lambda(t)$ converges to unregularized GF $\theta(t)$ as $\lambda\rightarrow 0$". There are two parameters here, $\lambda$ and $t$, is the convergence uniform over all $t$?













========more minor points:=====

Many apologies if I am being picky but as a relative outsider to optimization literature, even the statement that the point 0 is a critical point was not immediately  obvious to me (perhaps either a calculation of the gradient or a mention of the fact that $L>1$ would help). 

In the bottom of page 13, the equation before equation (1) is presumably the end of a sentence, thus the next line should be rewritten. Below, that "no such thing happen" should be "no such thing happens"

Some citation for Fact C.4 (Ky Fan?) would be nice. 


In page 19, just before the beginning of Section D.1. Do the authors mean $G_{\theta,ij}$ instead of $G_{\theta,j}$?


Just above equation (6), $\|W_\ell|^2$ should be $\|W_\ell\|^2$ and the sentence is missing a period. 








[1] Characterization of the subdifferential of some matrix norms, G.A. Watson. 1992, linear algebra and its applications.
Recommendation: 8: accept, good paper
Reviewer confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness/correctness: 3 good
Clarity: 3 good