SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\mathcal {L} \left(\boldsymbol {Z} _ {1}, \boldsymbol {Z} _ {2}, \lambda\right) = \Psi_ {1} \left(\boldsymbol {Z} _ {1}\right) + \Psi_ {2} \left(\boldsymbol {Z} _ {2}\right) + \langle \lambda , X \boldsymbol {Z} _ {1} \boldsymbol {Z} _ {2} - y \rangle .
$$
Taking derivatives w.r.t. to  $Z_{1}, Z_{2}$ , we get,
$$
\nabla_ {\boldsymbol {Z} _ {1}} \mathcal {L} \left(\boldsymbol {Z} _ {1}, \boldsymbol {Z} _ {2}, \lambda\right) = \nabla \Psi_ {1} \left(\boldsymbol {Z} _ {1}\right) + \left(X ^ {\top} \lambda\right) \boldsymbol {Z} _ {2} ^ {\top}.
$$
$$
\nabla_ {\boldsymbol {Z} _ {2}} \mathcal {L} (\boldsymbol {Z} _ {1}, \boldsymbol {Z} _ {2}, \lambda) = \nabla \Psi_ {2} (\boldsymbol {Z} _ {2}) + \boldsymbol {Z} _ {1} ^ {\top} (X ^ {\top} \lambda).
$$
As  $Z_1^*, Z_2^*$  should satisfy  $\nabla \mathcal{L} = 0$ .
$$
\left(\boldsymbol {Z} _ {1} ^ {*}\right) ^ {- \top} \nabla \Psi_ {2} \left(\boldsymbol {Z} _ {2} ^ {*}\right) = - X ^ {\top} \lambda ,
$$
$$
\nabla \Psi_ {1} \left(\boldsymbol {Z} _ {1} ^ {*}\right) = \left(\boldsymbol {Z} _ {1} ^ {*}\right) ^ {- \top} \nabla \Psi_ {2} \left(\boldsymbol {Z} _ {2} ^ {*}\right) \left(\boldsymbol {Z} _ {2} ^ {*}\right) ^ {\top}
$$
Lemma C.6. Let  $(Z_{1}, Z_{2})$  be the process that follows the GF equations (3.1), initialized according to condition  $I_{\gamma}$ , for some  $\gamma > 0$ . Then
$$
\lim _ {t \to \infty} Y - X \mathbf {Z} _ {1} (t) \mathbf {Z} _ {2} (t) = 0.
$$
$$
\lim_{t\to \infty}\boldsymbol {\beta}(t) = \operatorname *{argmin}_{X\boldsymbol {\beta} = Y}\left\| \boldsymbol {\beta}\right\|_{2}\stackrel {\mathrm{def}}{=}\boldsymbol{\beta}_{*}.
$$
Proof. The evolution of  $\pmb{R}$  writes,
$$
\dot {\boldsymbol {R}} = - \left(X ^ {\top} X\right) \left[ \boldsymbol {Z} _ {1} \boldsymbol {Z} _ {1} ^ {\top} \boldsymbol {R} + \boldsymbol {R} \boldsymbol {Z} _ {2} ^ {\top} \boldsymbol {Z} _ {2} \right],
$$
$$
\widehat {\operatorname {t r} \left(\boldsymbol {R} ^ {\top} \boldsymbol {R}\right)} = - 2 \operatorname {t r} \left(\boldsymbol {R} ^ {\top} \left(X ^ {\top} X\right) \boldsymbol {Z} _ {1} \boldsymbol {Z} _ {1} ^ {\top} \boldsymbol {R}\right) - 2 \operatorname {t r} \left(\boldsymbol {R} ^ {\top} \left(X ^ {\top} X\right) \boldsymbol {R} \boldsymbol {Z} _ {2} ^ {\top} \boldsymbol {Z} _ {2}\right).
$$
Note that for any three PSD matrices  $ABC$ . Using Lasserre [1995], (see Lemma 7, Min et al., 2021])
$$
\operatorname {t r} (A B C) \geq \lambda_ {\min } (A) \lambda_ {\min } (B) \operatorname {t r} C.
$$
From the invariance, we have,
$$
\boldsymbol {Z} _ {1} ^ {\top} \boldsymbol {Z} _ {1} = 2 \gamma \mathbf {I} _ {d} + \boldsymbol {Z} _ {2} ^ {\top} \boldsymbol {Z} _ {2},
$$
$$
\boldsymbol {Z} _ {1} ^ {\top} \boldsymbol {Z} _ {1} \succcurlyeq 2 \gamma \mathbf {I} _ {d}.
$$
Since  $Z_1^\top Z_1$  and  $Z_1Z_1^\top$ , share the same eigenvalues,  $Z_1Z_1^\top \geq 2\gamma \mathbf{I}_d$ . Therefore  $\lambda_{min}(Z_1Z_1^\top) \geq 2\gamma$ . Let  $\lambda_{min}(X^\top X)$  be the smallest non-zero eigenvalue of  $X^\top X$ . Although  $(X^\top X)$  can have zero eigenvalues, we can always restrict the evolution to the span  $(X^\top)$ , which  $R$  belongs to and without loss of generality assume that all the eigenvalues are non-zero. Using this for the first term, we have,
$$
\begin{array}{l} \operatorname {t r} \left(\boldsymbol {R} ^ {\top} \left(X ^ {\top} X\right) \boldsymbol {Z} _ {1} \boldsymbol {Z} _ {1} ^ {\top} \boldsymbol {R}\right) = \operatorname {t r} \left(\left(X ^ {\top} X\right) \boldsymbol {Z} _ {1} \boldsymbol {Z} _ {1} ^ {\top} \boldsymbol {R R} ^ {\top}\right), \\ \geq \lambda_ {\min } (X ^ {\top} X) \lambda_ {\min } (\mathbf {Z} _ {1} \mathbf {Z} _ {1} ^ {\top}) \operatorname {t r} \left(\boldsymbol {R} \boldsymbol {R} ^ {\top}\right) \geq 2 \lambda_ {\min } (X ^ {\top} X) \gamma \operatorname {t r} \left(\boldsymbol {R} \boldsymbol {R} ^ {\top}\right). \\ \end{array}
$$
For the second term,  $Z_2^\top Z_2 \succcurlyeq 0$  we have,
$$
\begin{array}{l} \operatorname {t r} \left(\boldsymbol {R} ^ {\top} (X ^ {\top} X) \boldsymbol {R} \boldsymbol {Z} _ {2} ^ {\top} \boldsymbol {Z} _ {2}\right) = \operatorname {t r} \left((X ^ {\top} X) \boldsymbol {R} \boldsymbol {Z} _ {2} ^ {\top} \boldsymbol {Z} _ {2} \boldsymbol {R} ^ {\top})\right), \\ \geq \lambda_ {\min } (X ^ {\top} X) \operatorname {t r} \left(\boldsymbol {R} \boldsymbol {Z} _ {2} ^ {\top} \boldsymbol {Z} _ {2} \boldsymbol {R} ^ {\top}\right), \\ \geq 0. \\ \end{array}
$$
Combining both,
$$
\widehat {\operatorname {t r} \left(\boldsymbol {R} ^ {\top} \boldsymbol {R}\right)} \leq - 2 \lambda_ {\min } \left(X ^ {\top} X\right) \gamma \operatorname {t r} \left(\boldsymbol {R} \boldsymbol {R} ^ {\top}\right). \tag {C.13}
$$
Thus, using Gronwall we can show that  $\left\| \boldsymbol{R} \right\|_F^2$  decays exponentially to zero, thus we have  $X\beta - Y \to 0$ . The last step is due to overparameterization, i.e.,  $d > n$  and existence of a interpolating solution.
Implicit bias of  $\beta$ . We know that  $\alpha \in \operatorname{span}(X^{\top})$  and  $\beta = (\mathbf{I} - \alpha \pmb{\alpha}^{\top})^{-1}\pmb{\alpha}$ . Using, Woodbury matrix identity,
$$
\left(\mathbf {I} - \alpha \boldsymbol {\alpha} ^ {\top}\right) ^ {- 1} = \mathbf {I} - \alpha \left(\mathbf {I} - \boldsymbol {\alpha} ^ {\top} \boldsymbol {\alpha}\right) ^ {- 1} \boldsymbol {\alpha} ^ {\top},
$$
$$
\left(\mathbf {I} - \alpha \alpha^ {\top}\right) ^ {- 1} \alpha = \alpha - \alpha \left(\mathbf {I} - \alpha^ {\top} \alpha\right) ^ {- 1} \alpha^ {\top} \alpha .
$$
Therefore  $\beta \in \operatorname{span}(X^{\top})$ . Hence, this satisfies the KKT conditions for
$$
\boldsymbol {\beta} _ {\infty} \in \operatorname * {a r g m i n} _ {X \boldsymbol {\beta} = Y} \frac {1}{2} \left\| \boldsymbol {\beta} \right\| ^ {2}.
$$
Lemma C.7 (Mirror flow  $(\mathbf{k} = 1)$ .) For  $k = 1$ , the dynamics of  $\beta$  can be characterized as a mirror flow
$$
\mathrm {d} \nabla \psi_ {\gamma} (\boldsymbol {\beta}) = - \left[ \gamma + \sqrt {\| \boldsymbol {\beta} \| ^ {2} + \gamma^ {2}} \right] ^ {1 / 2} \nabla \mathcal {L} (\boldsymbol {\beta}) \mathrm {d} t, \tag {C.14}
$$
where the potential writes
$$
\psi_ {\gamma} (\boldsymbol {\beta}) := \frac {2}{3} \left[ \sqrt {\| \boldsymbol {\beta} \| ^ {2} + \gamma^ {2}} + \gamma \right] ^ {3 / 2} - 2 \gamma \left[ \sqrt {\| \boldsymbol {\beta} \| ^ {2} + \gamma^ {2}} + \gamma \right] ^ {1 / 2}.
$$
Proof. Here we consider the case  $k = 1$ , let  $r \coloneqq X^{\top}(Y - X\beta)$ . It is denoted lowercase since  $r$  here is a vector. The gradient flow C.1 now in  $Z_{1}$ ,  $\alpha$  can be now can be written as,
$$
\dot {\boldsymbol {\alpha}} = \left(\mathbf {I} - \boldsymbol {\alpha} \boldsymbol {\alpha} ^ {\top}\right) \boldsymbol {r}, \quad \dot {\boldsymbol {Z}} _ {1} = \boldsymbol {Z} _ {1} \boldsymbol {\alpha} \boldsymbol {r} ^ {\top},
$$
$$
\dot {\boldsymbol {\beta}} = \left(\boldsymbol {Z} _ {1} ^ {\top} \dot {\boldsymbol {Z}} _ {1} \boldsymbol {\alpha}\right) = \left(\boldsymbol {Z} _ {1} ^ {\top} \boldsymbol {Z} _ {1} + \boldsymbol {\alpha} ^ {\top} \boldsymbol {Z} _ {1} ^ {\top} \boldsymbol {Z} _ {1} \boldsymbol {\alpha}\right) \boldsymbol {r}, \quad \dot {\boldsymbol {r}} = - \left(X ^ {\top} X\right) \left(\boldsymbol {Z} _ {1} ^ {\top} \boldsymbol {Z} _ {1} + \boldsymbol {\alpha} ^ {\top} \boldsymbol {Z} _ {1} ^ {\top} \boldsymbol {Z} _ {1} \boldsymbol {\alpha}\right) \boldsymbol {r}.
$$
Recall from Lemma 3.3 that
$$
\boldsymbol {\beta} = 2 \gamma (\mathbf {I} - \boldsymbol {\alpha} \boldsymbol {\alpha} ^ {\top}) ^ {- 1} \boldsymbol {\alpha}.
$$
Using Sherman-Morrison,
$$
\left(\mathbf {I} - \boldsymbol {\alpha} \boldsymbol {\alpha} ^ {\top}\right) ^ {- 1} = \mathbf {I} + \frac {\boldsymbol {\alpha} \boldsymbol {\alpha} ^ {\top}}{1 - \left\| \boldsymbol {\alpha} \right\| ^ {2}}.
$$
$$
\boldsymbol {\beta} = 2 \gamma \left[ (\mathbf {I} - \boldsymbol {\alpha} \boldsymbol {\alpha} ^ {\top}) ^ {- 1} \right] \boldsymbol {\alpha} = 2 \gamma \left[ \mathbf {I} + \frac {\boldsymbol {\alpha} \boldsymbol {\alpha} ^ {\top}}{1 - \| \boldsymbol {\alpha} \| ^ {2}} \right] \boldsymbol {\alpha} = 2 \gamma \frac {\boldsymbol {\alpha}}{1 - \| \boldsymbol {\alpha} \| ^ {2}}.
$$
Taking the norms, we get,
$$
\left\| \boldsymbol {\beta} \right\| = 2 \gamma \frac {\left\| \boldsymbol {\alpha} \right\|}{1 - \left\| \boldsymbol {\alpha} \right\| ^ {2}}, \quad \left\| \boldsymbol {\alpha} \right\| ^ {2} + \frac {2 \gamma}{\left\| \boldsymbol {\beta} \right\|} \left\| \boldsymbol {\alpha} \right\| - 1 = 0.
$$
$$
\left\| \boldsymbol {\alpha} \right\| = - \frac {\gamma}{\left\| \boldsymbol {\beta} \right\|} + \sqrt {\left(\frac {\gamma}{\left\| \boldsymbol {\beta} \right\|}\right) ^ {2} + 1}.
$$
$$
\begin{array}{l} 1 - \left\| \boldsymbol {\alpha} \right\| ^ {2} = 1 - \left[ - \frac {\gamma}{\| \boldsymbol {\beta} \|} + \sqrt {\left(\frac {\gamma}{\| \boldsymbol {\beta} \|}\right) ^ {2} + 1} \right] ^ {2}, \\ = 2 \frac {\gamma}{\| \boldsymbol {\beta} \|} \sqrt {\left(\frac {\gamma}{\| \boldsymbol {\beta} \|}\right) ^ {2} + 1} - 2 \left(\frac {\gamma}{\| \boldsymbol {\beta} \|}\right) ^ {2}, \\ = \frac {2 \gamma}{\sqrt {\gamma^ {2} + \left\| \boldsymbol {\beta} \right\| ^ {2}} + \gamma}. \\ \end{array}
$$
Now consider the evolution of  $\alpha$
$$
\dot {\boldsymbol {\alpha}} = \left[ \mathbf {I} - \boldsymbol {\alpha} \boldsymbol {\alpha} ^ {\top} \right] \boldsymbol {r},
$$
$$
\left[ \mathbf {I} - \alpha \boldsymbol {\alpha} ^ {\top} \right] ^ {- 1} \dot {\boldsymbol {\alpha}} = \boldsymbol {r},
$$
$$
\left[ \mathbf {I} + \frac {\boldsymbol {\alpha} \boldsymbol {\alpha} ^ {\top}}{1 - \| \boldsymbol {\alpha} \| ^ {2}} \right] \dot {\boldsymbol {\alpha}} = \boldsymbol {r}.
$$
One cannot write  $\left[\mathbf{I} + \frac{\boldsymbol{\alpha}\boldsymbol{\alpha}^{\top}}{1 - \left\|\boldsymbol{\alpha}\right\|^{2}}\right]$  as a Hessian of any function of  $\pmb{\alpha}$ . However, multiplying on both sides with  $(1 - \left\| \boldsymbol {\alpha}\right\|^{2})^{-1 / 2}$ , we obtain,
$$
\left[ \frac {\mathbf {I}}{\left(1 - \left\| \boldsymbol {\alpha} \right\| ^ {2}\right) ^ {1 / 2}} + \frac {\boldsymbol {\alpha} \boldsymbol {\alpha} ^ {\top}}{\left(1 - \left\| \boldsymbol {\alpha} \right\| ^ {2}\right) ^ {3 / 2}} \right] \dot {\boldsymbol {\alpha}} = \frac {\boldsymbol {r}}{\left(1 - \left\| \boldsymbol {\alpha} \right\| ^ {2}\right) ^ {1 / 2}},
$$
$$
\frac {\mathrm {d}}{\mathrm {d} t} \left[ \frac {\boldsymbol {\alpha}}{\left(1 - \| \boldsymbol {\alpha} \| ^ {2}\right) ^ {1 / 2}} \right] = \frac {\boldsymbol {r}}{\left(1 - \| \boldsymbol {\alpha} \| ^ {2}\right) ^ {1 / 2}}.
$$
We tackle the left hand side as follows,
$$
\boldsymbol {\beta} = \frac {2 \gamma \boldsymbol {\alpha}}{1 - \left\| \boldsymbol {\alpha} \right\| ^ {2}},
$$
$$
\begin{array}{l} \frac {\boldsymbol {\alpha}}{\left(1 - \left\| \boldsymbol {\alpha} \right\| ^ {2}\right) ^ {1 / 2}} = \frac {\boldsymbol {\beta}}{2 \gamma} \left(1 - \left\| \boldsymbol {\alpha} \right\| ^ {2}\right) ^ {1 / 2} = \frac {\boldsymbol {\beta}}{2 \gamma} \left(\frac {2 \gamma}{\sqrt {\gamma^ {2} + \left\| \boldsymbol {\beta} \right\| ^ {2}} + \gamma}\right) ^ {1 / 2}, \\ = \frac {\beta}{\sqrt {2 \gamma} \left(\sqrt {\gamma^ {2} + \left\| \boldsymbol {\beta} \right\| ^ {2}} + \gamma\right) ^ {1 / 2}}. \\ \end{array}
$$
Substituting the above expression and also  $\left(1 - \left\| \boldsymbol{\alpha} \right\|^2\right)^{1/2}$  on the RHS gives us,
$$
\frac {\mathrm {d}}{\mathrm {d} t} \left[ \frac {\beta}{\left(\sqrt {\gamma^ {2} + \left\| \boldsymbol {\beta} \right\| ^ {2}} + \gamma\right) ^ {1 / 2}} \right] = \left(\sqrt {\gamma^ {2} + \left\| \boldsymbol {\beta} \right\| ^ {2}} + \gamma\right) ^ {1 / 2} \boldsymbol {r}.
$$
Define the mirror potential,
$$
\psi_ {\gamma} (\beta) \stackrel {\text {d e f}} {=} \frac {2}{3} \left[ \sqrt {\| \beta \| ^ {2} + \gamma^ {2}} + \gamma \right] ^ {3 / 2} - 2 \gamma \left[ \sqrt {\| \beta \| ^ {2} + \gamma^ {2}} + \gamma \right] ^ {1 / 2}. \tag {C.15}
$$
Using the fact that,
$$
\frac {\mathrm {d}}{\mathrm {d} x} \left[ \frac {2}{3} \left[ \sqrt {x ^ {2} + p ^ {2}} + q \right] ^ {3 / 2} - 2 q \left[ \sqrt {x ^ {2} + p ^ {2}} + q \right] ^ {1 / 2} \right] = \left[ \sqrt {x ^ {2} + p ^ {2}} + q \right] ^ {- 1 / 2} x.
$$
$$
\nabla \psi_ {\gamma} (\boldsymbol {\beta}) = \left(\gamma + \sqrt {\| \boldsymbol {\beta} \| ^ {2} + \gamma^ {2}}\right) ^ {- 1 / 2} \boldsymbol {\beta}.
$$
Thus, we can write it as a continuous-time mirror descent.
$$
\widehat {\nabla \psi_ {\gamma} (\boldsymbol {\beta})} = - \left(\sqrt {\gamma^ {2} + \left\| \boldsymbol {\beta} \right\| ^ {2}} + \gamma\right) ^ {1 / 2} \nabla \mathcal {L} (\boldsymbol {\beta}).
$$
Lemma C.8 (Convergence rate of mirror flow). For the dynamics of the mirror flow given by Eq. (C.14), we have the following rate of convergence,
$$
\mathcal {L} \left(\boldsymbol {\beta} \left(\mathbf {t}\right)\right) \leq \frac {D _ {\psi_ {\gamma}} \left(\boldsymbol {\beta} _ {*} , \boldsymbol {\beta} _ {0}\right)}{\sqrt {2 \gamma} \mathbf {t}}.
$$
Proof. Note that  $\psi_{\gamma}$  is convex. Let  $D_{\psi_{\gamma}}(.,.)$  be the Bregman divergence defined with the potential  $\psi_{\gamma}(.)$ . Taking the time derivative of the Bregman divergence, we get,
$$
\begin{array}{l} \frac {\mathrm {d}}{\mathrm {d} t} D _ {\psi_ {\gamma}} (\beta_ {*}, \beta) = \left\langle \widehat {\nabla \psi_ {\gamma} (\beta)}, \beta - \beta * \right\rangle , \\ = - \left(\sqrt {\gamma^ {2} + \left\| \boldsymbol {\beta} \right\| ^ {2}} + \gamma\right) ^ {1 / 2} \langle \nabla \mathcal {L} (\boldsymbol {\beta}), \boldsymbol {\beta} - \boldsymbol {\beta} * \rangle . \\ \end{array}
$$
$$
- \left(\sqrt {\gamma^ {2} + \left\| \boldsymbol {\beta} \right\| ^ {2}} + \gamma\right) ^ {1 / 2} \leq - \sqrt {2 \gamma}
$$
Using the convexity of  $\mathcal{L}(.)$  in  $\beta$ , we have
$$
\langle \nabla \mathcal {L} (\beta), \beta - \beta * \rangle \leq \mathcal {L} (\beta) - \mathcal {L} (\beta_ {*}) = \mathcal {L} (\beta).
$$
The last step is using the existence of an interpolating solution. Substituting the above two inequalities, we get,
$$
\frac {\mathrm {d}}{\mathrm {d} t} D _ {\psi_ {\gamma}} \left(\boldsymbol {\beta} _ {*}, \boldsymbol {\beta}\right) \leq - \sqrt {2 \gamma} \mathcal {L} (\boldsymbol {\beta}).
$$
Integrating, we get,
$$
\int_ {0} ^ {t} \mathcal {L} (\boldsymbol {\beta} (s)) d s \leq \frac {1}{\sqrt {2 \gamma}} \left[ D _ {\psi_ {\gamma}} \left(\boldsymbol {\beta} _ {*}, \boldsymbol {\beta} _ {0}\right) - D _ {\psi_ {\gamma}} \left(\boldsymbol {\beta} _ {*}, \boldsymbol {\beta} _ {t}\right) \right] \leq \frac {D _ {\psi_ {\gamma}} \left(\boldsymbol {\beta} _ {*} , \boldsymbol {\beta} _ {0}\right)}{\sqrt {2 \gamma}}.
$$
The last step is using the fact that Bregman divergence is positive for convex functions. We will now show that the loss is decreasing along the trajectory,
$$
\mathcal {L} \dot {\left(\boldsymbol {\beta}\right)} = \left\langle \nabla \mathcal {L} (\boldsymbol {\beta}), \dot {\boldsymbol {\beta}} \right\rangle = - \frac {1}{\left(\sqrt {\gamma^ {2} + \left\| \boldsymbol {\beta} \right\| ^ {2}} + \gamma\right) ^ {1 / 2}} \left\langle \widehat {\overline {{\nabla \psi_ {\gamma} (\boldsymbol {\beta})}}}, \dot {\boldsymbol {\beta}} \right\rangle .
$$
$$
\widehat {\dot {\nabla \psi_ {\gamma} (\boldsymbol {\beta})}} = \nabla^ {2} \psi_ {\gamma} (\boldsymbol {\beta}) \dot {\boldsymbol {\beta}}, \quad \left\langle \widehat {\dot {\nabla \psi_ {\gamma} (\boldsymbol {\beta})}, \dot {\boldsymbol {\beta}}} \right\rangle = \left\langle \nabla^ {2} \psi_ {\gamma} (\boldsymbol {\beta}) \dot {\boldsymbol {\beta}}, \dot {\boldsymbol {\beta}} \right\rangle \geq 0, \quad (\text {U s i n g c o n v e x i t y}).
$$
Thus  $\mathcal{L}(\beta) \leq 0$ , using this,
$$
t \mathcal {L} (\boldsymbol {\beta} (t)) \leq \int_ {0} ^ {t} \mathcal {L} (\boldsymbol {\beta} (s)) d s \leq \frac {D _ {\psi_ {\gamma}} (\boldsymbol {\beta} _ {*} , \boldsymbol {\beta} _ {0})}{\sqrt {2 \gamma}}.
$$
Definition C.9. [Singular value decomposition and notation.] The singular value decomposition of a given matrix  $A \in \mathbb{R}^{d \times k}$  be denoted as  $U_A D_A V_A^\top$  where  $U_A \in \mathbb{R}^{d \times d}$ ,  $D_A \in \mathbb{R}^{d \times k}$ ,  $V_A \in \mathbb{R}^{k \times k}$ .
Let  $k < d$  and consider the products  $AA^{\top}$  and  $A^{\top}A$ . We will refer to the diagonal matrices  $D_{A^{\top}A} \in \mathbb{R}^{k \times k}$  and  $D_{AA^{\top}} \in \mathbb{R}^{d \times d}$  as  $D_A^2$  and  $\tilde{D}_A^2$ , in order to ease notation. Note that  $\tilde{D}_A^2$  is a block-diagonal matrix containing  $D_A^2$  as the first diagonal block and  $\mathbf{0}$  as the second diagonal block.
Finally, if  $k = d$  then  $D_A^2 = \tilde{D}_A^2$ .
Lemma C.10 (Singular values of  $Z_{1}$  and  $\beta$ ). Using the notations in Definition C.9, the singular values of  $Z_{1}, Z_{2}$  and  $\beta$  are related as the following,
$$
\mathbf {D} _ {\mathbf {Z} _ {2}} ^ {2} = - \gamma I _ {k} + \sqrt {\gamma^ {2} I _ {k} + \mathbf {D} _ {\boldsymbol {\beta}} ^ {2}} \quad a n d \quad \tilde {\mathbf {D}} _ {\mathbf {Z} _ {1}} ^ {2} = \gamma I _ {d} + \sqrt {\gamma^ {2} I _ {d} + \tilde {\mathbf {D}} _ {\boldsymbol {\beta}} ^ {2}},
$$
Proof. From the invariance (C.8) we further deduce (by appropriately multiplying left and right with  $\mathbf{Z}_1$ ,  $\mathbf{Z}_2$  and their transposes):
$$
\boldsymbol {\beta} ^ {\top} \boldsymbol {\beta} - (\mathbf {Z} _ {2} ^ {\top} \mathbf {Z} _ {2}) ^ {2} = 2 \gamma \mathbf {Z} _ {2} ^ {\top} \mathbf {Z} _ {2} \quad \text {a n d} \quad (\mathbf {Z} _ {1} \mathbf {Z} _ {1} ^ {\top}) ^ {2} - \boldsymbol {\beta} \boldsymbol {\beta} ^ {\top} = 2 \gamma \mathbf {Z} _ {1} \mathbf {Z} _ {1} ^ {\top},
$$
which implies that  $[\beta^{\top}\beta, \mathbf{Z}_2^{\top}\mathbf{Z}_2] = \mathbf{0}$  and  $[\beta\beta^{\top}, \mathbf{Z}_1\mathbf{Z}_1^{\top}] = \mathbf{0}$ . Hence  $\beta^{\top}\beta$  and  $\mathbf{Z}_2^{\top}\mathbf{Z}_2$  commute and can be simultaneously diagonalizable, same is the case with  $\beta\beta^{\top}, \mathbf{Z}_1\mathbf{Z}_1^{\top}$ . Therefore, the following relation holds (elementwise):
$$
\mathbf {D} _ {\mathbf {Z} _ {2}} ^ {2} = - \gamma I _ {k} + \sqrt {\gamma^ {2} I _ {k} + \mathbf {D} _ {\boldsymbol {\beta}} ^ {2}} \quad \text {a n d} \quad \tilde {\mathbf {D}} _ {\mathbf {Z} _ {1}} ^ {2} = \gamma I _ {d} + \sqrt {\gamma^ {2} I _ {d} + \tilde {\mathbf {D}} _ {\boldsymbol {\beta}} ^ {2}}, \tag {C.16}
$$
where the appropriate dimensionality of the diagonal matrices is evident from the indexing of the identity matrices.
Theorem C.11 (Singular vectors are static under orthogonal data). Let  $\alpha \coloneqq \mathbf{Z}_1^{-\top}\mathbf{Z}_2$ . Then, it holds that
$$
\mathbf {U} _ {\boldsymbol {\beta}} = \mathbf {U} _ {\mathbf {Z} _ {1}} = \mathbf {U} _ {\boldsymbol {\alpha}},
$$
$$
\mathbf {V} _ {\beta} = \mathbf {V} _ {\mathbf {Z} _ {2}} = \mathbf {V} _ {\alpha}.
$$
Furthermore, if  $X^{\top}X = I_{d}$ , it additionally holds that
$$
\mathbf {U} _ {\boldsymbol {\alpha}} = \mathbf {U} _ {\mathbf {R}} = U _ {\boldsymbol {\beta} ^ {*}},
$$
$$
\mathbf {V} _ {\boldsymbol {\alpha}} = \mathbf {V} _ {\mathbf {R}} = V _ {\beta^ {*}}.
$$
Proof. From invariance (C.8), by multiplying left and right with  $\mathbf{V}_{\mathbf{Z}_1}^\top$  and  $\mathbf{V}_{\mathbf{Z}_1}$ , respectively, we get that:
$$
\mathbf {D} _ {\mathbf {Z} _ {1}} ^ {2} - \mathbf {V} _ {\mathbf {Z} _ {1}} ^ {\top} \mathbf {Z} _ {2} \mathbf {Z} _ {2} ^ {\top} \mathbf {V} _ {\mathbf {Z} _ {1}} = 2 \gamma I _ {d},
$$
which implies that  $\mathbf{V}_{\mathbf{Z}_1}$  also diagonalizes  $\mathbf{Z}_2\mathbf{Z}_2^\top$  (i.e., the left singular vectors of  $\mathbf{Z}_1$  are the same as the right singular vectors of  $\mathbf{Z}_2$  for all  $t > 0$ ). As a result,  $\beta = \mathbf{U}_{\mathbf{Z}_1}\mathbf{D}_\beta \mathbf{V}_{\mathbf{Z}_2}^\top$  and thus we have shown that  $\mathbf{U}_{\beta} = \mathbf{U}_{\mathbf{Z}_1}$  and  $\mathbf{V}_{\beta} = \mathbf{V}_{\mathbf{Z}_2}$ .
Next, by definition  $\alpha = \mathbf{Z}_1^{-\top}\mathbf{Z}_2$ , and  $\mathbf{Z}_1$  and  $\mathbf{Z}_1^{-T}$  have the same left and right singular vectors (since  $\mathbf{Z}_1^{-1} := \mathbf{V}_{\mathbf{Z}_1}\mathbf{D}_{\mathbf{Z}_1}^{-1}\mathbf{U}_{\mathbf{Z}_1}^{\top}$ ). This, in conjunction with the above proves  $\mathbf{U}_{\mathbf{Z}_1} = \mathbf{U}_{\alpha}$  and  $\mathbf{V}_{\mathbf{Z}_2} = \mathbf{V}_{\alpha}$ .
To show that  $\mathbf{U}_{\alpha} = \mathbf{U}_{\mathbf{R}}$  and  $\mathbf{V}_{\alpha} = \mathbf{V}_{\mathbf{R}}$ , we need to do a bit more work. We will show that:
$$
\boldsymbol {\alpha} ^ {\top} \mathbf {R} = \mathbf {R} ^ {\top} \boldsymbol {\alpha}, \text {a n d} \tag {C.17}
$$
$$
\boldsymbol {\alpha} \mathbf {R} ^ {\top} = \mathbf {R} \boldsymbol {\alpha} ^ {\top}. \tag {C.18}
$$
Once we prove (C.17) and (C.18), it directly follows that
$$
\left(\boldsymbol {\alpha} \boldsymbol {\alpha} ^ {\top}\right) \left(\mathbf {R R} ^ {\top}\right) = \left(\mathbf {R R} ^ {\top}\right) \left(\boldsymbol {\alpha} \boldsymbol {\alpha} ^ {\top}\right) \quad \text {a n d} \quad \left(\boldsymbol {\alpha} ^ {\top} \boldsymbol {\alpha}\right) \left(\mathbf {R} ^ {\top} \mathbf {R}\right) = \left(\mathbf {R} ^ {\top} \mathbf {R}\right) \left(\boldsymbol {\alpha} ^ {\top} \boldsymbol {\alpha}\right), \tag {C.19}
$$
To prove (C.17) we will show that  $\mathbf{R}^{\top}\pmb {\alpha} - \pmb{\alpha}^{\top}\mathbf{R} = 0,\forall t > 0$ . First, we recall some relations that will be needed. Under orthogonal data, it holds that
$$
\mathbf {R} = \left(\boldsymbol {\beta} ^ {*} - \boldsymbol {\beta}\right) \quad \text {a n d} \quad \dot {\mathbf {R}} = - \dot {\boldsymbol {\beta}}. \tag {C.20}
$$
Furthermore, using the reparametrization in terms of  $\alpha$  which induces identity (C.12) and the invariance (C.8), the original dynamics of  $\beta$  given by  $\dot{\boldsymbol{\beta}} = R\mathbf{Z}_2^T\mathbf{Z}_2 + \mathbf{Z}_1\mathbf{Z}_1^\top R$  rewrites as
$$
\dot {\boldsymbol {\beta}} = 2 \gamma \left[ \mathbf {R} \boldsymbol {\alpha} ^ {\top} \left(I _ {d} - \boldsymbol {\alpha} \boldsymbol {\alpha} ^ {\top}\right) ^ {- 1} \boldsymbol {\alpha} + \left(I _ {d} - \boldsymbol {\alpha} \boldsymbol {\alpha} ^ {\top}\right) ^ {- 1} \mathbf {R} \right]. \tag {C.21}
$$
Now, using (C.21), it holds that
$$
\begin{array}{l} \widehat {\mathbf {R} ^ {\top} \boldsymbol {\alpha}} = \mathbf {R} ^ {\top} \mathbf {R} - (\mathbf {R} ^ {\top} \boldsymbol {\alpha}) ^ {2} - 2 \gamma \mathbf {R} ^ {\top} (I _ {d} - \boldsymbol {\alpha} \boldsymbol {\alpha} ^ {\top}) ^ {- 1} \boldsymbol {\alpha} - 2 \gamma \boldsymbol {\alpha} ^ {\top} (I _ {d} - \boldsymbol {\alpha} \boldsymbol {\alpha} ^ {\top}) ^ {- 1} \boldsymbol {\alpha} \mathbf {R} ^ {\top} \boldsymbol {\alpha} \\ = \mathbf {R} ^ {\top} \mathbf {R} - \left(\mathbf {R} ^ {\top} \boldsymbol {\alpha}\right) ^ {2} - 2 \gamma \mathbf {R} ^ {\top} \boldsymbol {\alpha} \left(I _ {k} - \boldsymbol {\alpha} ^ {\top} \boldsymbol {\alpha}\right) ^ {- 1} - 2 \gamma \boldsymbol {\alpha} ^ {\top} \boldsymbol {\alpha} \left(I _ {k} - \boldsymbol {\alpha} ^ {\top} \boldsymbol {\alpha}\right) ^ {- 1} \mathbf {R} ^ {\top} \boldsymbol {\alpha} \\ = \mathbf {R} ^ {\top} \mathbf {R} - (\mathbf {R} ^ {\top} \boldsymbol {\alpha}) ^ {2} - 2 \gamma \mathbf {R} ^ {\top} \boldsymbol {\alpha} (I _ {k} - \boldsymbol {\alpha} ^ {\top} \boldsymbol {\alpha}) ^ {- 1} - 2 \gamma (I _ {k} - \boldsymbol {\alpha} ^ {\top} \boldsymbol {\alpha}) ^ {- 1} \mathbf {R} ^ {\top} \boldsymbol {\alpha} + \mathbf {R} ^ {\top} \boldsymbol {\alpha}, \\ \end{array}
$$
were we used the push-through identity  $(I + \mathbf{U}\mathbf{V})^{-1}\mathbf{U} = \mathbf{U}(I + \mathbf{V}\mathbf{U})^{-1}$
Denoting  $\mathbf{C} \coloneqq \mathbf{R}^{\top}\alpha$ , computing the transpose version of the above ODE and using the fact that  $\mathbf{C}^{2} - (\mathbf{C}^{\top})^{2} = \frac{1}{2}\left(\mathbf{C} + \mathbf{C}^{\top}\right)\left(\mathbf{C} - \mathbf{C}^{\top}\right) + \frac{1}{2}\left(\mathbf{C} - \mathbf{C}^{\top}\right)\left(\mathbf{C} + \mathbf{C}^{\top}\right)$ , the following holds: