Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper studies the convergence of a deep linear network trained with a Bures-Wasserstein loss. 

The authors first give a description of the critical point of the Bures-Wasserstein loss (and its perturbation) when restricted to the set of rank $k$ matrices $\mathcal{M}_k$. If I understand correctly, the critical points of the loss as a function of the parameters of the network are not studied.

Convergence of gradient flow and gradient descent on the parameters of the network (with the perturbative Bures-Wasserstein loss) are then showed for initialization that are balanced and have a uniform deficiency margin.
strength_and_weaknesses: The proof of convergence is strong, but I have issues with its novelty (see next section) and with the assumptions:
 - Though the balanced assumption is common due to its power, I find it quite restrictive. The main reason why I am interested in the convergence of deep linear networks is as a stepping stone to later prove convergence of nonlinear networks. And these kind of balanced arguments break down in the non-linear case. I think it is important to focus on non-balanced linear networks.
 - The uniform deficiency margin assumption is very strong and the discussion of this assumption is misleading: (Arora et al., 202) show that a random network at initialization will have a positive deficiency margin with prob. close to 0.5 only when when the input and outputs have dimension 1. In Remark 5.4 they cite the above paper but omit the dimension assumption which is very strong (balanced dimension 1 network are equivalent to scalar networks whose convergence with prob. 0.5 is obvious). In Remark 5.4 the authors write "in our setting, the uniform deficiency margin can also be satisfied using a similar result" but I couldn't find the corresponding argument anywhere in the appendix.

clarity,_quality,_novelty_and_reproducibility: The paper is well written and clear.

The novelty of this paper only lies in adapting the proof of convergence of (Arora et al. 2020) to the perturbative Bures-Wasserstein loss. The result of this paper requires the same two very strong assumptions required for the proof of (Arora et al. 2020), suggesting that there is no significant improvement to the proof techniques. 

I do not understand why the paper focuses on such a specific loss as the Bures-Wasserstein loss. Furthermore it is unclear why there would be an advantage in optimizing the Bures-Wasserstein loss with a deep linear network instead of with a matrix directly, since both seem to converge to the same global minimum. My impression is that deep linear networks are mainly useful in settings such as matrix completion where their implicit bias towards low-rank solution help generalization. Is there such an advantage in this setting?
summary_of_the_review: The paper adapts the proof of convergence of (Arora et al. 2020) to the Bures-Wasserstein loss. The same strong assumptions are required for their convergence proof.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper studies the convergence of deep linear networks under the so-called Bures-Wasserstein loss, a loss function on the space of positive semi-definite matrices. Such losses arise as the 2-Wasserstein distance between gaussian distributions, and hence can be considered as approximating a gaussian target in a generative setting. 

The authors characterized the critical points of the loss. Since the Bures-Wasserstein loss is non-smooth, the authors introduced a smooth version of the loss and prove the convergence rates of the gradient flow under an initial condition and a uniform margin condition. Extention to gradient descent with small step-sizes was considered.
strength_and_weaknesses: Strength:
- Many properties of the Bures-Wasserstein loss in the context of deep linear networks are proved, which might be interesting for future works.


Weaknesses:
- The motivation for using the Bures-Wasserstein loss is not clear. In other words, while I find the math interesting, I do not understand why we should study the problem presented in the paper. Can the authors provide further arguments other than "the Bures-Wasserstein distance has also attracted much interest"? Please note that each of the cited literature in the introduction, unlike this work, has its own motivation for the Bures-Wasserstein distance.

- The characterization of the critical points of (non-smooth) Bures-Wasserstein losses is quite disconnected from the rest of the paper.

- Why does the gradient descent (Theorem 5.7) converge faster than the gradient flow, ie exponential versus $O(t^{-1})$? Moreover, the proof of Theorem 5.7 relies on a contraction argument. Why does this not apply to GF, as an "infinitesimal step-size GD"?


- The bound in Lemma 3.2 does not decrease to 0. In fact, quite unintuitively, it grows with $r$, the rank constraint. These facts suggest that the relation between the smooth and non-smooth Bures-Wasserstein losses is not captured correctly.

- The uniform margin condition in Definition 5.2 is exceedingly strong as it is required to hold for *all* unitary matrix $U$. This is different from Arora et al. 2019a where the "over all $U$" part is not present. For instance, let $\Sigma_\theta = \Sigma_0 = I_n$. Then the $\max_U$ over the left-hand side of (7) would grow with $n$, which is not a property of the deficiency margin condition in (Arora et al. 2019a).
In this light, Remark 5.4 seems quite misleading as it suggests that the uniform deficiency margin condition can be easily met with high probability by choosing a specific initialization as in Arora et al. 2019a. 


- The authors claimed to have proved $O(t^{-1})$ convergence for GF. However, inspecting Theorem 5.6, there is an $\Delta_t^*$ term in (10) which might increase over $t$ (at least the authors did not provide a rigorous proof that $\Delta_t^*$ remains bounded). The proof therefore seems incomplete to me.
clarity,_quality,_novelty_and_reproducibility: In my opinion, if the authors can address my concerns above, then the novelty of the math part is significant enough to meet the bar for acceptance. What is lacking, however, is a sound motivation for the considered setting.
summary_of_the_review: This paper studies a mathematically interesting problem and provides a detailed analysis. My two major concerns are:

1. The various gaps in the derivation, and 
2. Motivation.

I suggest the authors to improve the paper based in these two directions.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper studies the minimization of the Bures-Wassertein loss on positive symmetric matrices. Seen as a function over matrices of constant ranks, they give a characterisation of its critical points and of its gradient flow when added an overparametrized multiplicative structure.
strength_and_weaknesses: ### **Strength**

The paper tries to derive results on what can be rebaptized the Bures-Wasserstein loss that may appear for generative purposes.

-----------------------------------------------


### **Weaknesses**

It is hard to really understand the purpose of the paper: 
- Is it to derive properties of the Bures-Wasserstein distance function restricted to manifolds of constant rank as a theoretical object? In this perspective, not only ICLR may not be a target for such a paper, but also, the results do not seem very conclusive with respect to this precise literature. 
- Is this to theoretically ground generative models? Here again, the overall motivation is not clearly depicted and furthermore, even if this motivation appears in the introduction and the title, there is no comment and/or simulation of this. Which, once again, is clearly out of target when considering this is a ICLR submission.
- Finally, the mathematical writing as well as the story behind the paper is really hard to follow. Notations are very messy even-though all the reparametrisations used necessitate a careful writing. As a striking example, the beginning of the Section 3  concerning the notations is really hard to follow: $\pi(W)$ is introduced but is in fact $\Sigma$, $\mu$ appears for no reel reason sometimes, and there is no consistency from one line to the other. I advise the authors to ***carefully and explicitly*** write any parameter dependence on the notations used, otherwise it is impossible to follow. Examples: $\Sigma \rightarrow \Sigma(W)$,  $W^* \rightarrow W^*_{J_k} $, etc...
- The theorems, the assumptions and the Lemma are not commented. The setup of the Theorems are not recalled. For example, Theorem 5.6 is on the gradient flow in terms of $W$ or $\Sigma$? Furthermore Theorem 5.6 cannot be really considered as a convergence result as $\|\Delta_t^*\|$ appears in the right term. Finally why do we have a Theorem 5.7 as a linear convergence result whereas Therem 5.6 is polynomial ?
clarity,_quality,_novelty_and_reproducibility: As said before, clarity and quality are low.

That being said, I truly believe that, once clarified, the paper can be strong in terms of results. I strongly encourage the authors to take a serious second look at the clarity of exposition and the correctness of the statements.
summary_of_the_review: Under this state, the paper is inappropriate for publication.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper shows the convergence of specific first-order gradient-based algorithms on deep linear networks with Bures-Wasserstein loss, under the assumption that the initial weights are balanced. The author studies both the original non-smooth Bures-Wasserstein loss and a smooth perturbative one. They characterize the critical points of those two losses over the rank-constrained matrices and they show the convergence of GF on smooth perturbative BW loss, and GD on BW loss, under the assumption that the initial weights are balanced, and has a uniform deficiency margin.


strength_and_weaknesses: Strength: Although this paper considers a deep linear network, the problem setting is relatively new as previous work primarily studies square loss or logistic/exponential loss. The writings are mostly clear.

Weakness: I have several concerns about the quality and significance of the results. Specifically:
1. The discussion of critical points of BW loss and its smooth version is less relevant to the rest of the paper because the results are shown for the space of $W$, the end-to-end matrix. If one is concerned about the landscape of deep linear networks, he should study the critical points of BW loss in the weight space $W_1, \cdots, W_N$, as those in [1]. If this paper is mainly about convergence, I don't know why Section 4 is presented.
2. The convergence requires a uniform deficiency margin at initialization. Although this is very related to the condition proposed in [2], this UDM assumption is much more restrictive than the one in [2]. In fact, it is highly nontrivial to see whether this UDM assumption can be satisfied, but the UDM assumption is essential to the convergence proof in the paper. The author needs to justify the UDM assumption formally by showing there exists an initial $W(0)$ that satisfies UDM.
3. Regarding the contribution, the new thing is the BW loss. Except for the loss, the convergence analysis relies on a balanced assumption together with a deficiency margin (with minor modification), which is mostly based on the analysis in [2]. Moreover, even the balanced assumption is restrictive and recent work has shown convergence of GF under non-balanced initialization [3,4]. Unless there is a significant difference when analyzing the convergence under the BW loss, the contribution of this paper is minor, in my opinion.

Minor comments:
1. It's unclear from Theorem 5.6 whether the loss has $\mathcal{O}(t^{-1})$ convergence or not because $||\Delta_t^* ||$ depends on time $t$

Reference:

[1] Kenji Kawaguchi. Deep learning without poor local minima, Advances in Neural Information Processing Systems, volume 29. Curran Associates, Inc., 2016.

[2] Sanjeev Arora, Nadav Cohen, Noah Golowich, and Wei Hu. A convergence analysis of gradient descent for deep linear neural networks. ICLR, 2018

[3] Chulhee Yun, Shankar Krishnan, and Hossein Mobahi. A unifying view on implicit bias in training linear neural networks. ICLR, 2020.

[4] Hancheng Min, Salma Tarmoun, René Vidal, and Enrique Mallada. On the explicit role of initialization on the convergence and implicit bias of overparametrized linear networks. ICML, 2021
clarity,_quality,_novelty_and_reproducibility: The paper is mostly clear. One assumption is not justified. The results are not very significant compared to prior work.
summary_of_the_review: One assumption is not justified in the paper; 

The results are not very significant compared to prior work.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The authors consider a deep matrix factorization model of covariance matrices trained with the Bures-Wasserstein distance. Unlike most recent publication that focus on discriminative settings and the square loss, the authors' model considers another interesting type of loss that connects with the generative setting. The authors characterize the critical points and minimizers of the Bures-Wasserstein distance over the space of rank-bounded matrices. The authors establish convergence results for gradient flow using a smooth perturbative version of the loss and convergence results for finite step size gradient descent under certain assumptions on the initial weights.  The reviewers thought the convergence proof is interesting. The reviewers however found the balanced assumption too restrictive raised due to lack of generalization to nonlinear cases and also thought the uniform deficiency margin is also restrictive and somewhat misleading, lack of novelty in the proof compared to Arora 2022, and lack of clarity on focusing on Bures-Wasserstein loss. The authors provided a thorough response but their response did not fully assuage reviewer concerns. I agree with the reviewer assessments, while the paper has some interesting ideas more work is required to relax the stringent assumptions. I recommend submission to a future ML venue after a thorough revision but can not recommend acceptance of the paper in its current version. 


justification_for_why_not_higher_score: the reviews do not merit a higher score
Decision: Reject