SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} f (\boldsymbol {U} _ {k + 1}, \boldsymbol {V} _ {k}) \leq f (\boldsymbol {U} _ {k}, \boldsymbol {V} _ {k}) - (\eta - \frac {\eta^ {2}}{2}) \left\langle (\boldsymbol {U} _ {k} \boldsymbol {V} _ {k} ^ {\top} - \boldsymbol {M}) \mathcal {V} _ {k} \mathcal {V} _ {k} ^ {\top}, (\boldsymbol {U} _ {k} \boldsymbol {V} _ {k} ^ {\top} - \boldsymbol {M}) \right\rangle \\ \leq f \left(\boldsymbol {U} _ {k}, \boldsymbol {V} _ {k}\right) - \left(\eta - \frac {\eta^ {2}}{2}\right) \sigma_ {r} \left(\mathcal {V} _ {E} ^ {\top} \mathcal {V} _ {k}\right) f \left(\boldsymbol {U} _ {k}, \boldsymbol {V} _ {k}\right). \tag {13} \\ = \left(1 - (\eta - \frac {\eta^ {2}}{2}) \sigma_ {r} \left(\mathcal {V} _ {E} ^ {\top} \mathcal {V} _ {k}\right)\right) f (\boldsymbol {U} _ {k}, \boldsymbol {V} _ {k}) \\ \end{array}
$$
$$
\begin{array}{l} f (\boldsymbol {U} _ {k + 1}, \boldsymbol {V} _ {k + 1}) \leq f (\boldsymbol {U} _ {k + 1}, \boldsymbol {V} _ {k}) - \left(\eta - \frac {\eta^ {2}}{2}\right) \left\langle \left(\boldsymbol {V} _ {k} \boldsymbol {U} _ {k + 1} ^ {\top} - \boldsymbol {M} ^ {\top}\right) \mathcal {U} _ {k + 1} \mathcal {U} _ {k + 1} ^ {\top}, \left(\boldsymbol {U} _ {k + 1} \boldsymbol {V} _ {k} ^ {\top} - \boldsymbol {M}\right) \right\rangle \\ \leq f \left(\boldsymbol {U} _ {k + 1}, \boldsymbol {V} _ {k}\right) - \left(\eta - \frac {\eta^ {2}}{2}\right) \sigma_ {r} \left(\mathcal {U} _ {E} ^ {\top} \mathcal {U} _ {k + 1}\right) f \left(\boldsymbol {U} _ {k + 1}, \boldsymbol {V} _ {k}\right) \\ = \left(1 - \left(\eta - \frac {\eta^ {2}}{2}\right) \sigma_ {r} \left(\mathcal {U} _ {E} ^ {\top} \mathcal {U} _ {k + 1}\right)\right) f \left(\boldsymbol {U} _ {k + 1}, \boldsymbol {V} _ {k}\right) \tag {14} \\ \end{array}
$$
$$
f \left(\boldsymbol {U} _ {k + 1}, \boldsymbol {V} _ {k + 1}\right) \leq \left(1 - \left(\eta - \frac {\eta^ {2}}{2}\right) \sigma_ {r} \left(\mathcal {V} _ {E} ^ {\top} \mathcal {V} _ {k}\right)\right) \left(1 - \left(\eta - \frac {\eta^ {2}}{2}\right) \sigma_ {r} \left(\mathcal {U} _ {E} ^ {\top} \mathcal {U} _ {k + 1}\right)\right) f \left(\boldsymbol {U} _ {k}, \boldsymbol {V} _ {k}\right). \tag {15}
$$
To guarantee the linear convergence, we only need  $0 < \eta < 2$ .  $\sigma_r\left(\mathcal{U}_E^\top \mathcal{U}_{k + 1}\right)$  as well as  $\sigma_r\left(\mathcal{V}_E^\top \mathcal{V}_k\right)$  is strictly larger than  $0$  ( $\mathcal{V}_E$  is the orthogonal row subspace of  $\pmb{U}_k\pmb{V}_k^\top -\pmb{M}$  and  $\nu_{k}$  is the orthogonal subspace of  $\pmb{V}_k$ ), which indicates that the linear convergence rate is independent of the condition number of matrix  $M$ .
We show in Fig. 2 that the convergence of ScaledGD and AltScaledGD are independent of the condition number  $\kappa$  of the matrix  $M$  with general random initialization and small initialization. In Fig. 2, we set the rank of the matrix  $M$  as 5, with condition number  $\kappa$  ranging from 10, 100, 200. It can be seen from the left subfigure of Fig. 2 that for general random initialization, the error curves of ScaledGD with different  $\kappa$  are the exactly the same, and the error curves of ScaledGD also coincide with that of the AltScaledGD. These results are also true for small initialization as shown in the right subfigure of Fig. 2. These observations certificate that preconditioning in Eq. (2) and Eq. (4) indeed help accelerating the convergence such that the convergence rate is independent of the condition number of the matrix  $M$ .
<sup>2</sup>Since  $\mathbf{U}_k\mathbf{V}_k^\top \to \mathbf{M}$ ,  $k \to \infty$  and the same analysis is applied on Eq. (10) with respect to  $\mathbf{V}$ , we know that  $\eta \leq c\kappa$ .
4 Theoretical analysis - proof sketch
In this section, we provide the proof sketch of our results in Section 3. For simplicity, we present the theoretical analysis for rank one matrix factorization where  $\mathbf{U} \in \mathbb{R}^{m \times 1}$  and  $\mathbf{V} \in \mathbb{R}^{n \times 1}$ . More detailed proofs for the main theorems are provided in the supplementary material.
4.1 Convergence of ScaledGD
It can be deduced that the objective function of problem (1) is upper bounded by four terms as
$$
\begin{array}{l} \| \boldsymbol {U} _ {k + 1} \boldsymbol {V} _ {k + 1} ^ {\top} - \boldsymbol {M} \| _ {F} \leq \underbrace {(1 - \eta) ^ {2} \| \boldsymbol {U} _ {k} \boldsymbol {V} _ {k} ^ {\top} - \boldsymbol {M} \| _ {F}} _ {\text {\textcircled {1}}} + (1 - \eta) \eta \underbrace {\| \boldsymbol {M} \| _ {F} \| \mathcal {V} _ {* \perp} ^ {\top} \mathcal {V} _ {k} \| _ {2}} _ {\text {\textcircled {2}}} \\ + \eta (1 - \eta) \underbrace {\| \boldsymbol {M} \| _ {F} \| \mathcal {U} _ {*} ^ {\top} \mathcal {U} _ {k} \| _ {2}} _ {③} + \eta^ {2} \underbrace {\| \boldsymbol {M} \| _ {F} \left| 1 - \frac {\| \boldsymbol {M} \| _ {F} \cos \boldsymbol {\theta} _ {k} ^ {u} \cos \boldsymbol {\theta} _ {k} ^ {v}}{\| \boldsymbol {U} _ {k} \boldsymbol {V} _ {k} ^ {\top} \| _ {F}} \right|} _ {④} \tag {16} \\ \end{array}
$$
where  $\mathcal{U}_k$  and  $\mathcal{V}_k$  correspond to the orthogonal basis of the column space of  $U_k$  and  $V_k$ ,  $\mathcal{U}_{*\perp}$  and  $\mathcal{V}_{*\perp}$  are the orthogonal complements of the left and right singular vector matrices of  $M$  (i.e.  $U_{*}, V_{*}$ ), and  $\cos \theta_k^u$  is cosine value of the angle between the vectors  $U_k$  and  $U_*$ ,  $\cos \theta_k^v$  is cosine value of the angle between the vectors  $V_k$  and  $V_*$ . The upper-bound depicts the differences between  $U_{k+1}V_{k+1}^\top$  and  $M$  in two aspects
The angle between the subspace of  $U_{k}V_{k}^{\top}$  and  $M$ :  $\| \mathcal{U}_{* \perp}^{\top} \mathcal{U}_{k} \|_{2}, \| \mathcal{V}_{* \perp}^{\top} \mathcal{V}_{k} \|_{2}$ ;
The difference of the length (norm) between  $\mathbf{U}_k\mathbf{V}_k^\top$  and  $M$ :  $\| M\| _F - \| U_kV_k^\top \| _F$ .
The term ② and ③ are related to the angle between the subspace of  $M$  and  $\pmb{U}_k\pmb{V}_k^\top$ , the term ④ is related to the difference between the norm of  $M$  and  $\pmb{U}_k\pmb{V}_k^\top$ , as given by the following lemma.
Lemma 1. If  $\langle M, U_k V_k^\top \rangle \geq \| U_k V_k^\top \|_F^2$ , then there is constant  $C_u \geq 0$  such that
$$
\left| 1 - \frac {\left\| \boldsymbol {M} \right\| _ {F} \cos \boldsymbol {\theta} _ {k} ^ {u} \cos \boldsymbol {\theta} _ {k} ^ {v}}{\left\| \boldsymbol {U} _ {k} \boldsymbol {V} _ {k} ^ {\top} \right\| _ {F}} \right| \leq C _ {u} \left(\left\| \boldsymbol {M} \right\| _ {F} - \left\| \boldsymbol {U} _ {k} \boldsymbol {V} _ {k} ^ {\top} \right\| _ {F}\right) \tag {17}
$$
According to Eq. (16), we know that the decrease of the objective function in problem (1) is decided by the decrease of the distance between the subspace  $(2)$  and  $(3)$  and the difference between the norm of  $M$  and  $U_{k}V_{k}^{\top}$ $(4)$ . The following lemma further reveals that the distance between the subspace of  $U_{k}V_{k}^{\top}$  and  $M$  decreases.
Lemma 2. (Convergence of the distance between subspaces) For the ScaledGD (2), if  $\| M\| _F\cos \pmb {\theta}_k^u\cos \pmb {\theta}_k^v\geq \| \pmb {U}_k\pmb {V}_k^\top \| _F$  , then the following holds
$$
\left\| \mathcal {U} _ {* \perp} ^ {\top} \mathcal {U} _ {k + 1} \right\| _ {2} \leq (1 - \eta) \left\| \mathcal {U} _ {* \perp} ^ {\top} \mathcal {U} _ {k} \right\| _ {2}, \left\| \mathcal {V} _ {* \perp} ^ {\top} \mathcal {V} _ {k + 1} \right\| _ {2} \leq (1 - \eta) \left\| \mathcal {V} _ {* \perp} ^ {\top} \mathcal {V} _ {k} \right\| _ {2} \tag {18}
$$
The Lemma 2 indicates that the term (2) and (3) in Eq. (16) decrease linearly if the norm of  $\|U_k V_k^\top\|_F$  is smaller than norm of the projection of  $M$  onto the column and row spaces of  $U_k V_k^\top$ . At the mean time, the condition  $\|M\|_F \cos \theta_k^u \cos \theta_k^v \geq \|U_k V_k^\top\|_F$  also guarantees the linear convergence of the differences between the norm of  $U_k V_k^\top$  and  $M$ .
Theorem 5. (Convergence of the matrix norm) For the ScaledGD (2), if  $\| M\| _F\cos \pmb {\theta}_k^u\cos \pmb {\theta}_k^v\geq$ $\| \pmb {U}_k\pmb {V}_k^\top \| _F$  for all  $k\geq 0$  , then we have
$$
\left\| \boldsymbol {M} \right\| _ {F} - \left\| \boldsymbol {U} _ {k + 1} \boldsymbol {V} _ {k + 1} ^ {\top} \right\| _ {F} \leq (1 - \eta) ^ {2 k} k C _ {\alpha} \tag {19}
$$
where  $C_{\alpha}$  is a constant and  $\eta$  is the step length  $0 \leq \eta < 1$ .
Both Lemma 2 and Theorem 5 are built on the condition that  $\| M\| _F\cos \pmb {\theta}_k^u\cos \pmb {\theta}_k^v\geq \| \pmb {U}_k\pmb {V}_k^\top \| _F$  for all  $k\geq 0$  , while this is not a trivial condition for ScaledGD. The following lemma guarantees that the condition can be satisfied if the step length  $\eta$  is smaller than a constant.
Lemma 3. Let  $\eta \leq c_{\eta} < 1$  with  $c_{\eta}$  a small constant, if  $\| M \|_F \cos \theta_0^u \cos \theta_0^v \geq \| U_0 V_0^\top \|_F$  then the following is true
$$
\| \boldsymbol {M} \| _ {F} \cos \boldsymbol {\theta} _ {k} ^ {u} \cos \boldsymbol {\theta} _ {k} ^ {v} \geq \| \boldsymbol {U} _ {k} \boldsymbol {V} _ {k} ^ {\top} \| _ {F}, \forall k > 0. \tag {20}
$$
The above results guarantee the local linear convergence of the term ②, ③ and ④ on the condition that
$$
\left\| \boldsymbol {M} \right\| _ {F} \cos \boldsymbol {\theta} _ {0} ^ {u} \cos \boldsymbol {\theta} _ {0} ^ {v} \geq \left\| \boldsymbol {U} _ {0} \boldsymbol {V} _ {0} ^ {\top} \right\| _ {F} \tag {21}
$$
which is critical for our analysis on random Gaussian initialization and small initialization.
4.1.1 Small initialization
In practice, the condition  $\| M\| _F\cos \theta_0^u\cos \theta_0^v\geq \| U_0V_0^\top \| _F$  can be easily satisfied by very small (near zero) initialization. According to the random matrix theory Theorem 2.7.5 in Tao [2012], for Gaussian initialization there exists  $\nu >0$  such that with high probability  $\cos \theta_0^u$  and  $\cos \theta_0^v$  is lower bounded by constant  $1 / \nu$  , therefore one can simply set the norm of  $U_{0}$  and  $V_{0}$  to be sufficiently small such that the inequality (21) holds. In consequence small initialization can guarantee the global linear conver
gence of ScaledGD, as shown in Fig. 3. While small initialization is very special, it can not helps us fully understand the global convergence property of ScaledGD from arbitrary initialization for the non-convex objective (1), even though small initialization has been widely used in the global convergence analysis of gradient descent algorithms Stöger and Soltanolkotabi [2021], Ye and Du [2021], Ma and Fattahi [2022] and ScaledGD for symmetric low rank matrix recovery problems Xu et al. [2023], Zhang et al. [2021].
Figure 3: Global convergence of small initialization and general random initialization.
4.1.2 General random initialization
In order to understand the optimization path of ScaledGD for the non-convex objective (1), we present the theoretical analysis of ScaledGD from random Gaussian initialization that may not satisfy the condition in Eq. (21). As shown in Fig. 3, when initialized with  $\sigma = 1$  ScaledGD iterations are also attracted by the saddle point thus enter the saddle region (zoomed region marked by red rectangle), while it can escape saddle region very fast. To rigorously characterize the saddle avoid phase, we first show and prove that the norm of matrices  $U_{k}$  and  $V_{k}$  decrease if  $\| M\| _F\max \{\cos \pmb {\theta}_k^u,\cos \pmb {\theta}_k^v\} < \| U_kV_k^\top \| _F$  as given by the following lemma and shown in Fig. 4.
Lemma 4. If the condition  $\| M\| _F\max \{\cos \pmb {\theta}_k^u,\cos \pmb {\theta}_k^v\} <  \| U_kV_k^\top \| _F$  is satisfied then we have
$$
\left\| \boldsymbol {U} _ {k + 1} \right\| _ {F} <   \left\| \boldsymbol {U} _ {k} \right\| _ {F} \quad \text {a n d} \quad \left\| \boldsymbol {V} _ {k + 1} \right\| _ {F} <   \left\| \boldsymbol {V} _ {k} \right\| _ {F}. \tag {22}
$$
Furthermore, if the condition  $\| M \|_F \cos \theta_k^u \cos \theta_k^v \geq \| U_k V_k^\top \|_F$  is satisfied then we have
$$
\left\| \boldsymbol {U} _ {k + 1} \right\| _ {F} \geq \left\| \boldsymbol {U} _ {k} \right\| _ {F} \quad \text {a n d} \quad \left\| \boldsymbol {V} _ {k + 1} \right\| _ {F} \geq \left\| \boldsymbol {V} _ {k} \right\| _ {F}. \tag {23}
$$
In general, if we initialize the matrices  $\mathbf{U}$  and  $\mathbf{V}$  as  $\mathcal{N}(0,\sigma)$  with large  $\sigma$ , then in the initial phase, with high probability we have  $\| M\| _F\max \{\cos \theta_0^u,\cos \theta_0^v\} < \| U_0V_0^\top \| _F$ . According to Lemma 3 and Lemma 4, we know that the norm of the matrices  $\mathbf{U}_k$  and  $\mathbf{V}_k$  decreases with the increase of  $k$  until it reaches the condition  $\| M\| _F\cos \theta_k^u\cos \theta_k^v\geq \| U_kV_k^\top \| _F$ , which is also illustrated in Fig. 4. In Fig. 4, we plot the changes of the norm of matrices  $\mathbf{U}$  and  $\mathbf{V}$ , the nested subfigure illustrates the the matrix norm in log scale. It is very interesting to study the changes of the matrix norm with respect to the optimization path. Generally, if  $\| U_0\| _F$  and  $\| V_0\| _F$  is initialized very large, then the decrease of the norm will decrease the objective function (1). Meanwhile,  $\mathbf{U} = \mathbf{0}$  and  $\mathbf{V} = \mathbf{0}$  is a saddle point of the objective function (1), the results in Lemma 4 thus indicate that the matrices  $\mathbf{U}_k$  and  $\mathbf{V}_k$  are updated toward the saddle point zero. While interestingly, as shown in Fig. 4 the matrix norm decreases to a magnitude which is strictly larger than zero, then the matrix norm begins to increase. These observation indicates that ScaledGD can escape from the saddle point zero, the saddle avoid phase is also illustrated in Fig. 5.
Analysis on the entire iteration process. It can be easily deduced from Eq. (16) that
$$
\left\| \boldsymbol {U} _ {k} \boldsymbol {V} _ {k} ^ {\top} - \boldsymbol {M} \right\| _ {F} <   (1 - \eta) ^ {2 k} \left\| \boldsymbol {U} _ {0} \boldsymbol {V} _ {0} ^ {\top} - \boldsymbol {M} \right\| _ {F} + \| \boldsymbol {M} \| _ {F}, \tag {24}
$$
Figure 4: Illustration of the norms of matrices  $\mathbf{U}$  and  $\mathbf{V}$ .
Figure 5: Illustration of the saddle avoid phase.
therefore after  $T_{1} = O\left(\ln \frac{1}{\delta}\right)$  iterations (for sufficiently small  $\delta$ )³, we have
$$
\left\| \boldsymbol {U} _ {k} \boldsymbol {V} _ {k} ^ {\top} - \boldsymbol {M} \right\| _ {F} \leq \left\| \boldsymbol {M} \right\| _ {F}, \forall k \geq T _ {1}, \tag {25}
$$
which indicates that  $\| M\| _F\cos \pmb {\theta}_k^u\cos \pmb {\theta}_k^v\geq \frac{1}{2}\| U_kV_k^\top \| _F$ . We term this period of time the initial phase. The following lemma tells that after  $T_{1}$  iterations the term ②, ③ and ④ decrease linearly.
Lemma 5. After  $T_{1}$  iterations of ScaledGD, the following inequalities hold  $\forall k \geq T_{1}$
$$
\left\| \mathcal {U} _ {* \perp} ^ {\top} \mathcal {U} _ {k + 1} \right\| _ {2} \leq \left(1 - \chi_ {k}\right) \left\| \mathcal {U} _ {* \perp} ^ {\top} \mathcal {U} _ {k} \right\| _ {2} \tag {26}
$$
$$
\left\| \mathcal {V} _ {* \perp} ^ {\top} \mathcal {V} _ {k + 1} \right\| _ {2} \leq \left(1 - \chi_ {k}\right) \left\| \mathcal {V} _ {* \perp} ^ {\top} \mathcal {V} _ {k} \right\| _ {2} \tag {27}
$$
$$
1 - \cos \boldsymbol {\theta} _ {k + 1} ^ {u} \cos \boldsymbol {\theta} _ {k + 1} ^ {v} \leq \left(1 - \chi_ {k}\right) ^ {2} \left(1 - \cos \boldsymbol {\theta} _ {k} ^ {u} \cos \boldsymbol {\theta} _ {k} ^ {v}\right) \tag {28}
$$
where  $\begin{array}{r}\tau_{k} = \frac{\eta\tau_{k}}{1 - \eta(1 - \tau_{k})} < 1 \end{array}$  and  $\begin{array}{r}\tau_{k} = \frac{\|M\|_{F}\cos\pmb{\theta}_{k}^{u}\cos\pmb{\theta}_{k}^{v}}{\|\pmb{U}_{k}\pmb{V}_{k}^{\top}\|_{F}}\in [1 / 2,1]. \end{array}$
If  $\frac{1}{2}\|U_kV_k^\top\|_F \leq \|M\|_F\cos\theta_k^u\cos\theta_k^v \leq \|U_kV_k^\top\|_F$  and  $\|M\|_F \geq \|U_kV_k^\top\|_F$ , we have that the term (4) in Eq. (16) is upper bounded by  $1 - \cos\theta_k^u\cos\theta_k^v$ . Thus the above Lemma 5 indicates that after  $T_1$  iterations, the objective function decreases at rate  $1 - \chi_k$ . Meanwhile,  $\cos\theta_k^u$  and  $\cos\theta_k^v$  are increasing, and  $\|U_kV_k^\top\|_F$  continues to decrease until  $\|M\|_F\cos\theta_k^u\cos\theta_k^v \geq \|U_kV_k^\top\|_F$ , which means the value  $\tau_k = \frac{\|M\|_F\cos\theta_k^u\cos\theta_k^v}{\|U_kV_k^\top\|_F}$  is monotonically increasing with the increase of  $k$  until up to 1. In consequence, the  $\chi_k$  is monotonically increasing from  $\frac{\eta/2}{1-\eta/2}$  to  $\eta$ . We name the period in which  $\chi_k$  increases from  $\frac{\eta/2}{1-\eta/2}$  to  $\eta$  the saddle avoid phase as shown in Fig. 5. The Lemma 5 also indicates that the ScaledGD escapes saddle points exponentially fast. After the saddle avoid phase, the ScaledGD converges to the global minima at rate  $1 - \eta$  according to the analysis in Section 4.1.1, since  $\|M\|_F\cos\theta_k^u\cos\theta_k^v \geq \|U_kV_k^\top\|_F$ , we name this period the linear convergence phase as shown in Fig. 5.
4.2 Convergence of AltScaledGD
The convergence analysis of the AltScaledGD is similar to that of ScaledGD, while different to ScaledGD, the objective function (1) is upper-bounded by three terms in AltScaledGD as
$$
\left\| \boldsymbol {U} _ {k + 1} \boldsymbol {V} _ {k + 1} ^ {\top} - \boldsymbol {M} \right\| _ {F} \leq \underbrace {(1 - \eta) ^ {2} \| \boldsymbol {U} _ {k} \boldsymbol {V} _ {k} ^ {\top} - \boldsymbol {M} \| _ {F}} _ {\text {①}} + (\eta - \eta^ {2}) \underbrace {\left\| \boldsymbol {M} \right\| _ {F} \left\| \mathcal {V} _ {*} ^ {\top} \mathcal {V} _ {k} \right\| _ {2}} _ {\text {②}} + \eta \underbrace {\left\| \boldsymbol {M} \right\| _ {F} \left\| \mathcal {U} _ {*} ^ {\top} \mathcal {U} _ {k + 1} \right\| _ {2}} _ {\text {③}} \tag {29}
$$
3 Please refer to the supplementary results for more detailed analysis.
<sup>4</sup>Since in this period of time, the norm of the matrices  $\mathbf{U}_k$  and  $\mathbf{V}_k$  decrease, while once  $\chi_k = \eta$  (equivalently  $\tau_k = 1$ ), according to Lemma 4 and Lemma 3, the norm of the matrices  $\mathbf{U}_k$  and  $\mathbf{V}_k$  begin to increase, which indicates that the matrices  $\mathbf{U}_k$  and  $\mathbf{V}_k$  are escaping from the saddle point zero.
therefore, the analysis of AltScaledGD for (1) is much easier than that of the ScaledGD in Eq. (16). Specifically, we only need to guarantee that the distance between subspaces of  $U_{k}$  and  $U_{*}$  ( $\| \mathcal{U}_{* \perp}^{\top} \mathcal{U}_{k} \|_{2}$ ),  $V_{k}$  and  $V_{*}$  ( $\| \mathcal{V}_{* \perp}^{\top} \mathcal{V}_{k} \|_{2}$ ) decrease linearly. The Lemma 2 also holds for AltScaledGD as
Lemma 6. (Convergence of the distance between subspaces) For AltScaledGD (4), if  $\| M\| _F\cos \pmb {\theta}_k^u\cos \pmb {\theta}_k^v\geq \| \pmb {U}_k\pmb {V}_k^\top \| _F$  and  $0 <   \eta \leq 1$  , then the following holds
$$
\left\| \mathcal {U} _ {* \perp} ^ {\top} \mathcal {U} _ {k + 1} \right\| _ {2} \leq (1 - \eta) \left\| \mathcal {U} _ {* \perp} ^ {\top} \mathcal {U} _ {k} \right\| _ {2}, \left\| \mathcal {V} _ {* \perp} ^ {\top} \mathcal {V} _ {k + 1} \right\| _ {2} \leq (1 - \eta) \left\| \mathcal {V} _ {* \perp} ^ {\top} \mathcal {V} _ {k} \right\| _ {2}. \tag {30}
$$
The condition in Lemma 6 can be satisfied  $\forall k$  if  $\| M\| _F\cos \pmb {\theta}_0^u\cos \pmb {\theta}_0^v\geq \| \pmb {U}_0\pmb {V}_0^\top \| _F$  and  $0 <   \eta \leq 1$  as specified by the following lemma, the condition is mild compared to the condition in Lemma 3.
Lemma 7. For AltScaledGD (4), if  $\| M\| _F\cos \pmb {\theta}_0^u\cos \pmb {\theta}_0^v\geq \| \pmb {U}_0\pmb {V}_0^\top \| _F$  and  $0 <   \eta \leq 1$  , then the following is true
$$
\left\| \boldsymbol {M} \right\| _ {F} \cos \boldsymbol {\theta} _ {k} ^ {u} \cos \boldsymbol {\theta} _ {k} ^ {v} \geq \left\| \boldsymbol {U} _ {k} \boldsymbol {V} _ {k} ^ {\top} \right\| _ {F}, \forall k > 0. \tag {31}
$$
The convergence analysis of AltScaledGD is the same as that of the ScaledGD in Section 4.1 with small initialization and general Gaussian initialization (the three phases convergence). The main difference between ScaledGD and AltScaledGD is that  $\eta$  in ScaledGD should be small such that  $\eta \leq c_{\eta} < 1$ , while for AltScaledGD  $\eta$  can be as large as 1. It can be seen from Eq. (29) that if  $\eta = 1$ , the the AltScaledGD Eq. (4) converges to the global minima in just one iteration. We also illustrate the convergence of ScaledGD Eq. (2) and AltScaledGD Eq. (4) with respect to the learning rate  $\eta$  in Fig. 6. It can be seen from Fig. 6 that for small learning rate  $\eta = 0.1$ , the convergence property (the loss curve) of AltScaledGD is almost exactly the same as ScaledGD, while for large learning rate  $\eta = 0.8$ , the
AltScaledGD converges very fast, in contrast the ScaledGD does not converge as the condition  $\eta \leq c_{\eta}$  is not satisfied according to Lemma 3. These results certify the superiority of AltScaledGD over ScaledGD, since both ScaledGD and AltScaledGD converges fast with large  $\eta$ , while the learning rate  $\eta$  is upper-bounded by a small constant  $c_{\eta}$  in ScaledGD.
Figure 6: Illustration of the effect of learning rate  $\eta$  for the convergence.
In this work, we are the first to rigorously prove the global convergence of ScaledGD and AltScaledGD for the non-convex low rank matrix factorization problem and show that thanks to the preconditioning matrices the global convergence rate of ScaledGD and AltScaledGD are independent of the condition number of the matrix  $M$ , thus they converge faster than gradient descent algorithm for ill-conditioned problem. We further prove that ScaledGD and AltScaledGD converges linearly from both small initialization and general random initialization, which is in contrast to the existing global convergence analysis that are only applicable to small initialization. Meanwhile, we show that compared to ScaledGD, AltScaledGD is more practical as it enables larger learning rate thus converges fast.
Limitations. This paper concerns low-rank matrix factorization which is the population loss of the more general low-rank matrix recovery problem, such as matrix completion and matrix sensing. While the empirical loss is different to the population loss in that the number of the samples is limited, therefore our results can not directly applied to general low-rank matrix recovery. Our further work is to study the empirical loss with the help of RIP condition for matrix sensing and the sampling lower-bound for matrix completion.
This research was supported by the National Key R&D Program of China (2020YFA0713900), the China NSFC projects under contracts 62372359, 61721002, 12226004, the Macao Science and Technology Development Fund under Grant 061/2020/A2, and the Fundamental Research Funds for the Central University under Grant ZYTS23056.