SlowGuess/NeurIPS_2023_OCR huggingface dataset:
R. Parhi and R. D. Nowak. What kinds of functions do deep neural networks learn? insights from variational spline theory. SIAM Journal on Mathematics of Data Science, 2022.
J. Pennington, S. Schoenholz, and S. Ganguli. The emergence of spectral universality in deep networks. In International Conference on Artificial Intelligence and Statistics, pages 1924-1932. PMLR, 2018.
S. Pesme and N. Flammarion. Saddle-to-saddle dynamics in diagonal linear networks, 2023.
S. Pesme, L. Pillaud-Vivien, and N. Flammarion. Implicit bias of sgd for diagonal linear networks: a provable benefit of stochasticity. Advances in Neural Information Processing Systems, 34: 29218-29230, 2021.
L. Pillaud-Vivien, J. Reygner, and N. Flammarion. Label noise (stochastic) gradient descent implicitly solves the lasso for quadratic parametrisation. In Conference on Learning Theory, pages 2127-2159. PMLR, 2022.
A. M. Saxe, J. L. McClelland, and S. Ganguli. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. In 2nd International Conference on Learning Representations, ICLR 2014, 2014.
A. M. Saxe, J. L. McClelland, and S. Ganguli. A mathematical theory of semantic development in deep neural networks. Proceedings of the National Academy of Sciences, 116(23):11537-11546, 2019.
M. Soltanolkotabi, A. Javanmard, and J. Lee. Theoretical insights into the optimization landscape of over-parameterized shallow neural networks. IEEE Transactions on Information Theory, 65: 742-769, 2017.
S. Tarmoun, G. Franca, B. D. Haeffele, and R. Vidal. Understanding the dynamics of gradient flow in overparameterized linear models. In M. Meila and T. Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 10153-10161. PMLR, 18-24 Jul 2021.
N. Timor, G. Vardi, and O. Shamir. Implicit regularization towards rank minimization in relu networks. In Proceedings of The 34th International Conference on Algorithmic Learning Theory, Proceedings of Machine Learning Research, 2023.
G. Vardi. On the implicit bias in deep-learning algorithms. arXiv preprint arXiv:2208.12591, 2022.
B. Woodworth, S. Gunasekar, J. D. Lee, E. Moroshko, P. Savarese, I. Golan, D. Soudry, and N. Srebro. Kernel and rich regimes in overparametrized models. In J. Abernethy and S. Agarwal, editors, Proceedings of Thirty Third Conference on Learning Theory, volume 125 of Proceedings of Machine Learning Research, pages 3635-3673. PMLR, 09-12 Jul 2020.
C. Yun, S. Krishnan, and H. Mobahi. A unifying view on implicit bias in training linear neural networks. In International Conference on Learning Representations, 2021. URL https://openreview.net/forum?id=ZsZM-4iMQkH.
C. Zhang, S. Bengio, M. Hardt, B. Recht, and O. Vinyals. Understanding deep learning requires rethinking generalization. arXiv preprint arXiv:1611.03530, 2016.
The appendix is organized as follows,
- In section A, we present the experiment details.
- In section B, we present the proof of Theorems 3.1, 3.2 and Corollary 3.1.
- section C contains the proofs of the supporting lemmas.
- In the final section D, we discuss our choice on the noise model and present the proof of Proposition 4.1.
- For the results referenced in the main section, the convergence rate of mirror flow can be found at C.8, the time evolution of singular values and their limiting jump process is available at C.12, the stationarity of singular vectors in the orthogonal case at C.11, the discussion on the noise model at D.
Notations. For matrices of appropriate dimensions, we use  $[A,B]$  to denote  $AB - BA$ .
Experiments. We discretize the SDE (4.1) with a step-size  $\sim 1 / \sqrt{t}$ . We simulate three parallel runs and track the evolution of singular values and the evolution of alignment using the commutator of the row  $\boldsymbol{w}_i$ , a, i.e.,  $[\boldsymbol{w}_i,\mathbf{a}]\coloneqq (\boldsymbol{w}_i\mathbf{a}^\top -\mathbf{a}\boldsymbol{w}_i^\top)$ . We consider the SDE for dimension  $d = 2$ . The evolution is initialized at  $W(0) = \mathbf{I}_2$  and  $\mathbf{a}(0) = 0$ . As seen in Figure 3a, the noise shape inherently induces a low-rank structure where it intensifies a singular value and significantly diminishes the other singular value. As predicted by our proposition (4.1), figure 3b shows that the rows of  $\boldsymbol{W}$  align with a, thus giving a rank 1 structure. The experiments were run on a 16-GB RAM Apple M1 mac with OS Ventura 13.3.1.
(b)
Figure 3: Three parallel runs of the noise dynamics Eq. (4.1) for  $d = 2$ . (a) The evolution of singular values with  $\sigma_0$  increasing and  $\sigma_1$  decaying. (b) Measuring the norm of the commutator again as predicted by Proposition 4.1.
B Main Proofs
In this section, we present the proofs of the theorem discussed in Section 3.
$$
\lim _ {t \to \infty} (\boldsymbol {W} _ {1} (t), \boldsymbol {W} _ {2} (t)) = (\boldsymbol {W} _ {1} ^ {\infty}, \boldsymbol {W} _ {2} ^ {\infty}) \in \mathcal {I} _ {\boldsymbol {\theta}}.
$$
$$
\lim  _ {t \to \infty} \boldsymbol {\beta} (t) = \operatorname * {a r g m i n} _ {X   \boldsymbol {\beta} = Y} \left\| \boldsymbol {\beta} \right\| _ {F} \stackrel {{\mathrm {d e f}}} {{=}} \boldsymbol {\beta} _ {*}.
$$
Proof. We initialize such that  $\mathbf{W}_1 = \sqrt{2\gamma} P, \mathbf{W}_2 = 0$ . Lemma 3.1 states that the dynamics of gradient flow is restricted to a subspace and can be equivalently described by,
$$
\dot {\boldsymbol {Z}} _ {1} = \boldsymbol {R} \boldsymbol {Z} _ {2} ^ {\top}, \quad \dot {\boldsymbol {Z}} _ {2} = \boldsymbol {Z} _ {1} ^ {\top} \boldsymbol {R}. \tag {B.1}
$$
where  $\pmb{R} \coloneqq X^{\top}(Y - X\pmb{Z}_1\pmb{Z}_2)$ .
(i) To show the convergence, we track the evolution of  $\operatorname{tr}\left(\boldsymbol{R}^{\top}\boldsymbol{R}\right)$  and use the following descent inequality to show that it converges to 0. With  $\lambda_{\min}(X^T X)$  denoting the smallest eigenvalue of the  $X^{\top}X$ , the descent inequality (C.13) is as follows,
$$
\widehat {\operatorname {t r} \left(\boldsymbol {R} ^ {\top} \boldsymbol {R}\right)} \leq - 2 \gamma \lambda_ {\min } (X ^ {\top} X) \operatorname {t r} \left(\boldsymbol {R} ^ {\top} \boldsymbol {R}\right).
$$
Refer to Lemma C.6 for the detailed proof.
(ii) To show that  $\beta \rightarrow \beta_{*}$  in the limit, we show that  $\beta \in \operatorname{span}\big(X^{\top}\big)$ , i.e.,  $\beta = X^{\top} \lambda$ , for some  $\lambda$ . This satisfies the KKT conditions required for the following minimization problem.
$$
\boldsymbol {\beta} _ {\infty} \in \operatorname * {a r g m i n} _ {X \boldsymbol {\beta} = Y} \frac {1}{2} \big \| \boldsymbol {\beta} \big \| ^ {2} = \boldsymbol {\beta} _ {*}.
$$
The complete proof can be found at Lemma C.6.
(iii) For the limit of the projected dynamics  $(Z_1^\infty, Z_2^\infty) := \lim_{t \to \infty} (Z_1(t), Z_2(t))$ , Lemma C.4 shows the following,
$$
(\boldsymbol {Z} _ {1} ^ {\infty}, \boldsymbol {Z} _ {2} ^ {\infty}) \in \operatorname * {a r g m i n} _ {X \boldsymbol {Z} _ {1} \boldsymbol {Z} _ {2} = y} \frac {1}{2} \big \| \boldsymbol {Z} _ {2} \big \| _ {F} ^ {2} + \frac {1}{2} \big \| \boldsymbol {Z} _ {1} \big \| _ {F} ^ {2} - \gamma \log \left(\det  \left(\boldsymbol {Z} _ {1} \boldsymbol {Z} _ {1} ^ {\top}\right)\right).
$$
Using the transformation from Eq. C.7, we have,
$$
\boldsymbol {W} _ {1} = \boldsymbol {Z} _ {1} P, \quad \boldsymbol {W} _ {2} = P ^ {\top} \boldsymbol {Z} _ {2}.
$$
$$
\left\| \boldsymbol {W} _ {1} \right\| _ {F} = \left\| \boldsymbol {Z} _ {1} \right\| _ {F}, \quad \left\| \boldsymbol {W} _ {2} \right\| _ {F} = \left\| \boldsymbol {Z} _ {2} \right\| _ {F},
$$
$$
\boldsymbol {Z} _ {1} \boldsymbol {Z} _ {1} ^ {\top} = \boldsymbol {W} _ {1} \boldsymbol {W} _ {1} ^ {\top}, \quad \boldsymbol {Z} _ {1} \boldsymbol {Z} _ {2} = \boldsymbol {W} _ {1} \boldsymbol {W} _ {2}.
$$
$$
\left(\boldsymbol {W} _ {1} ^ {\infty}, \boldsymbol {W} _ {2} ^ {\infty}\right) \in \operatorname * {a r g m i n} _ {X \boldsymbol {W} _ {1} \boldsymbol {W} _ {2} = Y} \frac {1}{2} \left\| \boldsymbol {W} _ {2} \right\| _ {F} ^ {2} + \frac {1}{2} \left\| \boldsymbol {W} _ {1} \right\| _ {F} ^ {2} - \gamma \log \left(\det  \left(\boldsymbol {W} _ {1} \boldsymbol {W} _ {1} ^ {\top}\right)\right).
$$
This hold on the set  $\{(\pmb{W}_1, \pmb{W}_2): \pmb{W}_1 P_\perp^\top = 0, P_\perp \pmb{W}_2 = 0\}$  which is ensured from gradient flow from Lemma 3.1.
Corollary 3.1. [Singular values at the limit] Using the same quantities as in Theorem 3.1, denote  $(\sigma_{1}(\pmb{W}_{1}^{\infty}),\dots,\sigma_{d}(\pmb{W}_{1}^{\infty}))$  and  $(\sigma_{1}(\pmb{\beta}_{*}),\dots,\sigma_{k}(\pmb{\beta}_{*}))$  the singular values of  $\pmb{W}_1^\infty ,\pmb{\beta}_*$  are
Proof. Using the transformation  $\mathbf{W}_1 = \mathbf{Z}_1\mathbf{P}$  from Lemma 3.1, we obtain,
$$
\boldsymbol {W} _ {1} \boldsymbol {W} _ {1} ^ {\top} = \boldsymbol {Z} _ {1} \boldsymbol {Z} _ {1} ^ {\top}. \tag {B.2}
$$
Thus,  $W_{1}$  and  $Z_{1}$  share the same singular values, similarly  $W_{2}$  and  $Z_{2}$  share the same singular values. The expressions for the singular values of  $Z_{1}$  (similarly  $Z_{2}$ ) can be found at Lemma C.10. This along with the fact that  $\beta \rightarrow \beta_{*}$  proves the Corollary 3.1
where the potential writes  $\psi_{\gamma}(\beta) \coloneqq \frac{2}{3}\left[\sqrt{\|\beta\|^2 + \gamma^2} + \gamma\right]^{3/2} - 2\gamma\left[\sqrt{\|\beta\|^2 + \gamma^2} + \gamma\right]^{1/2}$ .
Proof. The equivalence with mirror flow for scalar regression is shown in Lemma C.7. The continuous time mirror descent for singular values of  $\beta$  is derived in Lemma C.13.
This section contains all the technical lemmas and definitions used in the proofs in the section before.
$$
\dot {\boldsymbol {Z}} _ {1} = \boldsymbol {R} \boldsymbol {Z} _ {2} ^ {\top}, \quad \dot {\boldsymbol {Z}} _ {2} = \boldsymbol {Z} _ {1} ^ {\top} \boldsymbol {R}. \tag {C.1}
$$
Proof. We choose  $P_{\perp} \in \mathbb{R}^{(l - d) \times l}$  such that  $P_{\perp} P^{\top} = 0$ . Using the fact that  $P, P_{\perp}$  orthogonal and span the entire  $\mathbb{R}^{l \times l}$ ,
$$
P ^ {\top} P + P _ {\perp} ^ {\top} P _ {\perp} = \mathbf {I} _ {l}.
$$
Denoting  $(\mathbf{Z}_1)_{\perp} \coloneqq \mathbf{W}_1 P_{\perp}^{\top}, (\mathbf{Z}_2)_{\perp} \coloneqq P_{\perp} \mathbf{W}_2$ , we have,
$$
\boldsymbol {W} _ {1} = \boldsymbol {W} _ {1} \left[ P ^ {\top} P + P _ {\perp} ^ {\top} P _ {\perp} \right] = \boldsymbol {W} _ {1} P ^ {\top} P + \boldsymbol {W} _ {1} P _ {\perp} ^ {\top} P _ {\perp} = \boldsymbol {Z} _ {1} P + \left(\boldsymbol {Z} _ {1}\right) _ {\perp} P _ {\perp}. \tag {C.2}
$$
$$
\boldsymbol {W} _ {2} = \left[ P ^ {\top} P + P _ {\perp} ^ {\top} P _ {\perp} \right] \boldsymbol {W} _ {2} = P ^ {\top} P \boldsymbol {W} _ {2} + P _ {\perp} ^ {\top} P _ {\perp} \boldsymbol {W} _ {2} = P ^ {\top} \boldsymbol {Z} _ {2} + P _ {\perp} ^ {\top} \left(\boldsymbol {Z} _ {2}\right) _ {\perp}. \tag {C.3}
$$
Recalling the evolution of gradient flow (2.4) on the loss
$$
\dot {\boldsymbol {W}} _ {1} = X ^ {\top} \left(Y - X \boldsymbol {W} _ {1} \boldsymbol {W} _ {2}\right) \boldsymbol {W} _ {2} ^ {\top},
$$
$$
\dot {\boldsymbol {W}} _ {2} = \boldsymbol {W} _ {1} ^ {\top} X ^ {\top} \left(Y - X \boldsymbol {W} _ {1} \boldsymbol {W} _ {2}\right).
$$
Multiplying the gradient flow updates with  $P^{\top}$ ,  $P$  from the right, left (resp.) for the above equations
$$
\dot {\boldsymbol {W}} _ {1} P ^ {\top} = X ^ {\top} \left(Y - X \boldsymbol {W} _ {1} \boldsymbol {W} _ {2}\right) \boldsymbol {W} _ {2} ^ {\top} P ^ {\top}, \quad P \dot {\boldsymbol {W}} _ {2} = P \boldsymbol {W} _ {1} ^ {\top} X ^ {\top} \left(Y - X \boldsymbol {W} _ {1} \boldsymbol {W} _ {2}\right).
$$
Similarly multiplying with  $P_{\perp}^{\top}, P_{\perp}$ , we have,
$$
\dot {\boldsymbol {W}} _ {1} P _ {\perp} ^ {\top} = X ^ {\top} \left(Y - X \boldsymbol {W} _ {1} \boldsymbol {W} _ {2}\right) \boldsymbol {W} _ {2} ^ {\top} P _ {\perp} ^ {\top}, \quad P _ {\perp} \dot {\boldsymbol {W}} _ {2} = P _ {\perp} \boldsymbol {W} _ {1} ^ {\top} X ^ {\top} \left(Y - X \boldsymbol {W} _ {1} \boldsymbol {W} _ {2}\right).
$$
Using the above, we have,
$$
\begin{array}{l} \boldsymbol {W} _ {1} \boldsymbol {W} _ {2} = \boldsymbol {W} _ {1} \left[ P ^ {\top} P + P _ {\perp} ^ {\top} P _ {\perp} \right] \boldsymbol {W} _ {2}, \\ = \boldsymbol {W} _ {1} P ^ {\top} P \boldsymbol {W} _ {2} + \boldsymbol {W} _ {1} P _ {\perp} ^ {\top} P _ {\perp} \boldsymbol {W} _ {2}. \\ \end{array}
$$
$$
\boldsymbol {W} _ {1} \boldsymbol {W} _ {2} = \boldsymbol {Z} _ {1} \boldsymbol {Z} _ {2} + \left(\boldsymbol {Z} _ {1}\right) _ {\perp} \left(\boldsymbol {Z} _ {2}\right) _ {\perp}. \tag {C.5}
$$
Rewriting the evolution in terms of  $Z_{1}, Z_{2}, (Z_{1})_{\perp}, (Z_{2})_{\perp}$ ,
$$
\dot {\boldsymbol {Z}} _ {1} = X ^ {\top} \left(Y - X \left[ \boldsymbol {Z} _ {1} \boldsymbol {Z} _ {2} + (\boldsymbol {Z} _ {1}) _ {\perp} (\boldsymbol {Z} _ {2}) _ {\perp} \right]\right) \boldsymbol {Z} _ {2} ^ {\top}, \quad \dot {\boldsymbol {Z}} _ {2} = \boldsymbol {Z} _ {1} ^ {\top} X ^ {\top} \left(Y - X \left[ \boldsymbol {Z} _ {1} \boldsymbol {Z} _ {2} + (\boldsymbol {Z} _ {1}) _ {\perp} (\boldsymbol {Z} _ {2}) _ {\perp} \right]\right).
$$
$$
(\dot {\boldsymbol {Z}} _ {1}) _ {\perp} = X ^ {\top} \left(Y - X \left[ \boldsymbol {Z} _ {1} \boldsymbol {Z} _ {2} + (\boldsymbol {Z} _ {1}) _ {\perp} (\boldsymbol {Z} _ {2}) _ {\perp} ]\right) (\boldsymbol {Z} _ {2}) _ {\perp} ^ {\top}, \right. (\dot {\boldsymbol {Z}} _ {2}) _ {\perp} = (\boldsymbol {Z} _ {1}) _ {\perp} ^ {\top} X ^ {\top} \left(Y - X \left[ \boldsymbol {Z} _ {1} \boldsymbol {Z} _ {2} + (\boldsymbol {Z} _ {1}) _ {\perp} (\boldsymbol {Z} _ {2}) _ {\perp} ]\right)\right).
$$
This is exactly equivalent to the gradient flow under the linear transformation which maps  $W_{1}, W_{2}$  to  $(Z_{1}, (Z_{1})_{\perp}), (Z_{2}, (Z_{2})_{\perp})$ . Now taking into consideration the evolution of  $(Z_{1})_{\perp}, (Z_{2})_{\perp}$ , we have that  $(0, 0)$  is a equilibrium point for the dynamics. From our initialization  $I_{\gamma}$ ,
$$
\begin{array}{l} \left(\mathbf {Z} _ {1}\right) _ {\perp} \mid_ {t = 0} = \left. \mathbf {W} _ {1} \right| _ {t = 0} P _ {\perp} ^ {\top} = \sqrt {2 \gamma} P P _ {\perp} ^ {\top} = 0, \\ \left(\boldsymbol {Z} _ {2}\right) _ {\perp} \big | _ {t = 0} = P _ {\perp} \boldsymbol {W} _ {2} \big | _ {t = 0} = 0. \\ \end{array}
$$
As we initialized at the equilibrium of the dynamics we have  $((\mathbf{Z}_1)_{\perp},(\mathbf{Z}_2)_{\perp}) = (0,0)$  for any time  $t$ . From Eq. (C.5), we have,
$$
\boldsymbol {W} _ {1} \boldsymbol {W} _ {2} = \boldsymbol {Z} _ {1} \boldsymbol {Z} _ {2}. \tag {C.6}
$$
The gradient flow (2.4) is equivalent to
$$
\dot {\boldsymbol {Z}} _ {1} = X ^ {\top} \left(Y - X \boldsymbol {Z} _ {1} \boldsymbol {Z} _ {2}\right) \boldsymbol {Z} _ {2} ^ {\top},
$$
$$
\dot {\boldsymbol {Z}} _ {2} = \boldsymbol {Z} _ {1} ^ {\top} X ^ {\top} \left(Y - X \boldsymbol {Z} _ {1} \boldsymbol {Z} _ {2}\right).
$$
where  $\mathbf{Z}_1(0) = \sqrt{2\gamma} PP^\top = \mathbf{I}_d$ ,  $\mathbf{Z}_2(0) = 0$ . Furthermore, from Eq. C.2, C.3, we have the following,
$$
\boldsymbol {W} _ {1} = \boldsymbol {Z} _ {1} P, \quad \boldsymbol {W} _ {2} = P ^ {\top} \boldsymbol {Z} _ {2}. \tag {C.7}
$$
This finishes the proof of the lemma.
Lemma C.1. For the projected matrices given in (C.1), we have the following invariant,
$$
\mathbf {Z} _ {1} ^ {\top} \mathbf {Z} _ {1} - \mathbf {Z} _ {2} \mathbf {Z} _ {2} ^ {\top} = 2 \gamma \mathbf {I}. \tag {C.8}
$$
Proof. Recalling the dynamics (C.1)
$$
\dot {\boldsymbol {Z}} _ {1} = \boldsymbol {R} \boldsymbol {Z} _ {2} ^ {\top} \quad , \quad \dot {\boldsymbol {Z}} _ {2} = \boldsymbol {Z} _ {1} ^ {\top} \boldsymbol {R}.
$$
$$
\widehat {\boldsymbol {Z} _ {1} ^ {\top} \boldsymbol {Z} _ {1}} = (\dot {\boldsymbol {Z}} _ {1}) ^ {\top} \boldsymbol {Z} _ {1} + \boldsymbol {Z} _ {1} ^ {\top} (\dot {\boldsymbol {Z}} _ {1}), = \boldsymbol {Z} _ {2} \boldsymbol {R} ^ {\top} \boldsymbol {Z} _ {1} + \boldsymbol {Z} _ {1} ^ {\top} \boldsymbol {R} \boldsymbol {Z} _ {2} ^ {\top}.
$$
$$
\widehat {\boldsymbol {Z} _ {2} \boldsymbol {Z} _ {2} ^ {\top}} = \boldsymbol {Z} _ {2} \boldsymbol {R} ^ {\top} \boldsymbol {Z} _ {1} + \boldsymbol {Z} _ {1} ^ {\top} \boldsymbol {R} \boldsymbol {Z} _ {2} ^ {\top}.
$$
Hence,  $\widehat{\pmb{Z}_1^\top\pmb{Z}_1 - \pmb{Z}_2\pmb{Z}_2^\top} = 0$  . This implies,
$$
\left. \boldsymbol {Z} _ {1} ^ {\top} \boldsymbol {Z} _ {1} - \boldsymbol {Z} _ {2} \boldsymbol {Z} _ {2} ^ {\top} = \left[ \boldsymbol {Z} _ {1} ^ {\top} \boldsymbol {Z} _ {1} - \boldsymbol {Z} _ {2} \boldsymbol {Z} _ {2} ^ {\top} \right] \right| _ {t = 0} = 2 \gamma P P ^ {\top} = 2 \gamma \mathbf {I} _ {d}.
$$
Lemma C.2. Let  $\alpha \coloneqq Z_1^{-\top}Z_2$ , we have the following time evolution of parameters:
$$
\dot {\boldsymbol {\alpha}} = \boldsymbol {R} - \boldsymbol {\alpha} \boldsymbol {R} ^ {\top} \boldsymbol {\alpha}, \quad a n d \quad \beta = 2 \gamma \left(\mathbf {I} - \boldsymbol {\alpha} \boldsymbol {\alpha} ^ {\top}\right) ^ {- 1} \boldsymbol {\alpha}.
$$
Proof. Taking the time derivative of  $\alpha$ ,
$$
\begin{array}{l} \dot {\boldsymbol {\alpha}} = \dot {\boldsymbol {Z}} _ {1} ^ {- 1} \boldsymbol {Z} _ {2} + \boldsymbol {Z} _ {1} ^ {- 1} \dot {\boldsymbol {Z}} _ {2} = - \boldsymbol {Z} _ {1} ^ {- 1} \dot {\boldsymbol {Z}} _ {1} \boldsymbol {Z} _ {1} ^ {- 1} \boldsymbol {Z} _ {2} + \boldsymbol {Z} _ {1} ^ {- 1} \boldsymbol {Z} _ {1} \boldsymbol {R}, \\ = - \boldsymbol {Z} _ {1} ^ {- 1} \boldsymbol {Z} _ {2} \boldsymbol {R} ^ {\top} \boldsymbol {Z} _ {1} ^ {- 1} \boldsymbol {Z} _ {2} + \boldsymbol {R}, \\ = - \alpha \boldsymbol {R} ^ {\top} \boldsymbol {\alpha} + \boldsymbol {R}. \\ \end{array}
$$
The evolution of  $Z_{1}Z_{1}^{\top}$ ,  $\alpha \alpha^{\top}$
$$
\widehat {\left(\boldsymbol {Z} _ {1} \boldsymbol {Z} _ {1} ^ {\top}\right)} = \boldsymbol {Z} _ {1} \boldsymbol {Z} _ {1} ^ {\top} \boldsymbol {\alpha} \boldsymbol {R} ^ {\top} + \boldsymbol {R} \boldsymbol {\alpha} ^ {\top} \boldsymbol {Z} _ {1} \boldsymbol {Z} _ {1} ^ {\top}, \tag {C.9}
$$
$$
\begin{array}{l} \widehat {\boldsymbol {\alpha} \boldsymbol {\alpha} ^ {\top}} = \left(\boldsymbol {R} - \boldsymbol {\alpha} \boldsymbol {R} ^ {\top} \boldsymbol {\alpha}\right) \boldsymbol {\alpha} ^ {\top} + \boldsymbol {\alpha} \left(\boldsymbol {R} - \boldsymbol {\alpha} \boldsymbol {R} ^ {\top} \boldsymbol {\alpha}\right) ^ {\top}, \\ = \boldsymbol {R} \alpha^ {\top} - \alpha \boldsymbol {R} ^ {\top} \alpha \alpha^ {\top} + \alpha \boldsymbol {R} ^ {\top} - \alpha \alpha^ {\top} \boldsymbol {R} \alpha^ {\top}, \\ = \left(I - \alpha \boldsymbol {\alpha} ^ {\top}\right) \boldsymbol {R} \boldsymbol {\alpha} ^ {\top} + \alpha \boldsymbol {R} ^ {\top} \left(I - \alpha \boldsymbol {\alpha} ^ {\top}\right). \\ \end{array}
$$
Computing the evolution of  $(\mathbf{I} - \alpha \pmb{\alpha}^{\top})^{-1}$
$$
\begin{array}{l} \left(\mathbf {I} - \overset {\cdot} {\boldsymbol {\alpha}} \boldsymbol {\alpha} ^ {\top}\right) ^ {- 1} = \left(\mathbf {I} - \boldsymbol {\alpha} \boldsymbol {\alpha} ^ {\top}\right) ^ {- 1} \left[ \widehat {\boldsymbol {\alpha}} \boldsymbol {\alpha} ^ {\top} \right] \left(\mathbf {I} - \boldsymbol {\alpha} \boldsymbol {\alpha} ^ {\top}\right) ^ {- 1}, (C.10) \\ = \boldsymbol {R} \alpha^ {\top} \left(\mathbf {I} - \boldsymbol {\alpha} \boldsymbol {\alpha} ^ {\top}\right) ^ {- 1} + \left(\mathbf {I} - \boldsymbol {\alpha} \boldsymbol {\alpha} ^ {\top}\right) ^ {- 1} \boldsymbol {\alpha} \boldsymbol {R} ^ {\top}. (C.11) \\ \end{array}
$$
Let  $C_{\alpha} \coloneqq 2\gamma (\mathbf{I} - \alpha \pmb{\alpha}^{\top})^{-1}$  and  $C_Z \coloneqq Z_1Z_1^\top$ , so we have,
$$
\dot {C} _ {\alpha} = C _ {\alpha} \alpha R ^ {\top} + R \alpha^ {\top} C _ {\alpha},
$$
$$
\dot {C} _ {Z} = C _ {Z} \alpha R ^ {\top} + R \alpha^ {\top} C _ {Z}.
$$
Since, at initialization,  $C_{\alpha}(0) = C_{Z}(0)$ , we have  $C_{\alpha} = C_{Z}$ , for any time  $t$ . Therefore, we have,
$$
Z _ {1} Z _ {1} ^ {\top} = 2 \gamma (\mathbf {I} - \boldsymbol {\alpha} \boldsymbol {\alpha} ^ {\top}) ^ {- 1}. \tag {C.12}
$$
Using  $\beta = Z_{1}Z_{2} = Z_{1}Z_{1}^{\top}\alpha$  and the above invariant, we obtain  $\beta = 2\gamma (\mathbf{I} - \pmb {\alpha}\pmb{\alpha}^{\top})^{-1}\pmb{\alpha}$
Lemma C.3. The following property holds for  $\alpha$ :
$$
\boldsymbol {\alpha} _ {\infty} = \lim  _ {t \rightarrow \infty} \boldsymbol {\alpha} (t) \in s p a n (\boldsymbol {X} ^ {T}).
$$
Proof. From the evolution of  $\alpha$ , we have
$$
\dot {\boldsymbol {\alpha}} = \boldsymbol {R} - \boldsymbol {\alpha} \boldsymbol {R} ^ {\top} \boldsymbol {\alpha}.
$$
Let  $U \in \mathbb{R}^{d \times d}$  be the matrix projection each column of  $\alpha$  on the column span of  $X^{\top}$ , i.e.,  $\operatorname{span}\{x_1, \ldots, x_n\}$ ,  $U_{\perp} \in \mathbb{R}^{d \times d}$  be the matrix projection on the orthogonal space, i.e.,  $\operatorname{Ker}(X^{\top})$ . So  $\alpha = U\alpha + U_{\perp}\alpha$ . Note that  $R = X^{\top}(Y - XZ_1Z_1^{\top}\alpha)$ . So  $U_{\perp}R = 0$ , since  $U_{\perp}X^{\top} = 0$ . The evolution of  $U_{\perp}\alpha$  is that
$$
U _ {\perp} ^ {\bullet} \boldsymbol {\alpha} = - U _ {\perp} \boldsymbol {\alpha} R ^ {\top} (U \boldsymbol {\alpha} + U _ {\perp} \boldsymbol {\alpha}).
$$
Again,  $U_{\perp} \alpha = 0$  is the equilibrium point and our initialization  $\alpha = 0$  ensures that it stays at this equilibrium. This proves the lemma.
Lemma C.4. Let  $(\mathbf{Z}_1^\infty, \mathbf{Z}_2^\infty) \stackrel{\mathrm{def}}{=} \lim_{t \to \infty} (\mathbf{Z}_1(t), \mathbf{Z}_2(t))$  the limit of the gradient flow dynamics. Then,
$$
(\boldsymbol {Z} _ {1} ^ {\infty}, \boldsymbol {Z} _ {2} ^ {\infty}) \in \operatorname * {a r g m i n} _ {X \boldsymbol {Z} _ {1} \boldsymbol {Z} _ {2} = Y} \frac {1}{2} \left\| \boldsymbol {Z} _ {2} \right\| ^ {2} + \frac {1}{2} \left\| \boldsymbol {Z} _ {1} \right\| _ {F} ^ {2} - \gamma \log \big (\det  \big (\boldsymbol {Z} _ {1} \boldsymbol {Z} _ {1} ^ {\top} \big) \big).
$$
Proof. From Lemma C.3, we have that the  $(Z_1^\infty)^{-\top}Z_2^\infty \in \operatorname{span}(X)$ , so the condition (P2) from Proposition C.5 holds. Note that from (P2) of Proposition C.5, we have,
$$
\begin{array}{l} \nabla \Psi_ {1} \left(\boldsymbol {Z} _ {1} ^ {\infty}\right) = \left(\boldsymbol {Z} _ {1} ^ {\infty}\right) ^ {- \top} \boldsymbol {Z} _ {2} ^ {\infty} \left(\boldsymbol {Z} _ {2} ^ {\infty}\right) ^ {\top}, \\ = \left(\boldsymbol {Z} _ {1} ^ {\infty}\right) ^ {- \top} \left(\left(\boldsymbol {Z} _ {1} ^ {\infty}\right) ^ {\top} \boldsymbol {Z} _ {1} ^ {\infty} - 2 \gamma \mathbf {I}\right), \\ = \boldsymbol {Z} _ {1} ^ {\infty} - 2 \gamma (\boldsymbol {Z} _ {1} ^ {\infty}) ^ {- \top}. \\ \end{array}
$$
which is satisfied by the potential
$$
\Psi_ {1} \left(\boldsymbol {Z} _ {1}\right) = \frac {1}{2} \left\| \boldsymbol {Z} _ {1} \right\| _ {F} ^ {2} - \gamma \log \left(\det  \left(\boldsymbol {Z} _ {1} \boldsymbol {Z} _ {1} ^ {\top}\right)\right).
$$
When the imbalance is not isotropic, i.e.,  $Z_1^\top Z_1 - Z_2Z_2^\top = D$ , where  $D$  is some diagonal matrix ( $\neq c\mathbf{I}$ , for any constant  $c$ ). It this case,
$$
\nabla \Psi_ {1} \left(\boldsymbol {Z} _ {1} ^ {\infty}\right) = \boldsymbol {Z} _ {1} ^ {\infty} - D \left(\boldsymbol {Z} _ {1} ^ {\infty}\right) ^ {- \top},
$$
and there exists no such function  $\Psi_{1}$  and the proof breaks.
Proposition C.5. Let  $(Z_1^*, Z_2^*)$  satisfy the following minimization problem
$$
(\boldsymbol {Z} _ {1} ^ {*}, \boldsymbol {Z} _ {2} ^ {*}) = \operatorname * {a r g m i n} _ {X   \boldsymbol {Z} _ {1}   \boldsymbol {Z} _ {2} = y} \Psi_ {1} \left(\boldsymbol {Z} _ {1}\right) + \Psi_ {2} \left(\boldsymbol {Z} _ {2}\right),
$$
for some non-negative potential functions  $\Psi_{1}(\mathbf{Z}_{1}),\Psi_{2}(\mathbf{Z}_{2})$  . Then,  $(Z_1^*,Z_2^*)$  satisfies
$(\pmb {P}\pmb {I})(\pmb{Z}_1^*)^{-\top}\nabla \Psi_2(\pmb{Z}_2^*)\in \operatorname {span}(X),$
$(P2)$ $\nabla \Psi_{1}(\mathbf{Z}_{1}^{*}) = (\mathbf{Z}_{1}^{*})^{-\top}\nabla \Psi_{2}(\mathbf{Z}_{2}^{*})(\mathbf{Z}_{2}^{*})^{\top}.$
Proof. The Lagrangian for the minimization problem above is,