Kati8/ICLR_2023_2025 huggingface dataset:
authors: Zihan Wang
authors: Arthur Jacot
reviews: 
Comments: This papers considers the problem of minimizing parameters of a deep linear network with the matrix completion loss. Specifically the authors considered an $\ell_2$ regularized version of this problem and show that with certain conditions on the learning rate and the regularization parameter, SGD can jump from a high-rank local min to a low-rank local min, while it cannot jump from a low-rank local min to a high-rank local min.
Strengths: I think the theoretical results of this paper are interesting because it gives a nice characterization of the implicit bias of GD/SGD for deep linear networks. The proof that SGD can "jump" from high-rank to low-rank local minima is new to me and I think its a good step towards understand the training dynamics for deep neural networks.
Weaknesses: I think the main weakness of this paper is that both the theoretical results and the experiments require specific conditions on the $\ell_2$ regularization parameter $\lambda$ and the learning rate $\eta$, which seem to be a bit artificial. For example, the requirement that $\lambda$ is large in Theorem 3.2 seem to artificially cause $\|\theta\|$ to decay more quickly, thus biasing it towards low numerical rank. In the numerical experiments, a similar annealing technique is used run SGD with a large $\lambda$ and $\eta$, before switching to smaller parameters. 

My main point is that the implicit bias observed in this paper could be a result of a deliberate choice of parameters, instead of a natural property of SGD and GD. I hope the authors can clarify this point.
Recommendation: 8: accept, good paper
Reviewer confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness/correctness: 3 good
Clarity: 2 fair
reviews: 
Comments: This paper analyzes the matrix completion task with deep linear neural networks. It shows that the critical points that are not local minima can be avoided with a small ridge. And it shows that GD cannot avoid overestimating minima but  SGD can jump from any minimum to a lower rank minimum.
Strengths: I think analyzing the training dynamics of the deep linear neural networks on the matrix completion task is a very interesting problem. This paper provides insights on the advantages of using SGD to get a low-rank minima and provides experimental results.
Weaknesses: 1. I feel the statement of theorems is not very clear. It uses a lot of ''small enough'', "large enough".  I think the statement should be more rigorous. 

2. This paper claims that it shows GD can avoid rank-underestimating minima by taking a small enough ridge $\lambda$. But Proposition 3.2 is for Gradient Flow with a very strong assumption. I believe there is a gap. 

3. For the function $f_\alpha$, it takes a very specific form. The authors claim that changing $f_\alpha$ with similar properties should not affect the results. I don't see the reason why the condition cannot be extended to more general functions. I believe it could improve the results.

4. A small suggestion is that the proof in the appendix is not easy to read. I think it can be more organized and add more explanation.
Recommendation: 5: marginally below the acceptance threshold
Reviewer confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness/correctness: 2 fair
Clarity: 3 good
reviews: 
Comments: This work shows that when applied to matrix completion with deep linear networks, SGD can transition from local minima with higher ranks to solutions with lower ranks, while transitions in the opposite direction are zero. Crucially, this results depends on the gradient distribution of SGD which leads to drastically different outcomes than what common SDE-based models for SGD exhibit. The authors further provide numerical experiments that exhibit the predicted transitions in practice.
Strengths: This work provides an interesting theoretical insight on the distinction between stochastic and deterministic gradient descent. I find it especially exciting that it provides a concrete example how the gradient distribution of SGD enables phenomena that are not apparent from SDE-based models.
Weaknesses: I can not think of a weakness. It's a very nice paper in my opinion.
Recommendation: 10: strong accept, should be highlighted at the conference
Reviewer confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness/correctness: 4 excellent
Clarity: 4 excellent