SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Xixi Jia $^{1}$  Hailin Wang $^{2}$  Jiangjun Peng $^{2}$  Xiangchu Feng $^{1*}$  Deyu Meng $^{2,3}$
$^{1}$ School of Mathematics and Statistics, Xidian University
$^{2}$ School of Mathematics and Statistics, Xi'an Jiaotong University
<sup>3</sup> Macao Institute of Systems Engineering, Macau University of Science and Technology {hsijiaxidian, andrew.pengjj}@gmail.com; wanghailin97@163.com xcfeng@mail.xidian.edu.cn; dymeng@xjtu.edu.cn
Low-rank matrix factorization (LRMF) is a canonical problem in non-convex optimization, the objective function to be minimized is non-convex and even non-smooth, which makes the global convergence guarantee of gradient-based algorithm quite challenging. Recent work made a breakthrough on proving that standard gradient descent converges to the  $\varepsilon$ -global minima after  $O\left(\frac{d\kappa^2}{\tau^2}\ln \frac{d\sigma_d}{\tau} + \frac{d\kappa^2}{\tau^2}\ln \frac{\sigma_d}{\varepsilon}\right)$  iterations from small initialization with a very small learning rate (both are related to the small constant  $\tau$ ). While the dependence of the convergence on the condition number  $\kappa$  and small learning rate makes it not practical especially for ill-conditioned LRMF problem.
In this paper, we show that precondition helps in accelerating the convergence and prove that the scaled gradient descent (ScaledGD) and its variant, alternating scaled gradient descent (AltScaledGD) converge to an  $\varepsilon$ -global minima after  $O(\ln \frac{d}{\delta} + \ln \frac{d}{\varepsilon})$  iterations from general random initialization. Meanwhile, for small initialization as in gradient descent, both ScaledGD and AltScaledGD converge to  $\varepsilon$ -global minima after only  $O(\ln \frac{d}{\varepsilon})$  iterations. Furthermore, we prove that as a proximity to the alternating minimization, AltScaledGD converges faster than ScaledGD, its global convergence does not rely on small learning rate and small initialization, which certificates the advantages of AltScaledGD in LRMF.
Low-rank matrix factorization aims to approximate a given rank  $d$  matrix  $M \in \mathbb{R}^{m \times n}$  by the product of two factor matrices  $U \in \mathbb{R}^{m \times d}$ ,  $V \in \mathbb{R}^{n \times d}$ , which plays a fundamental and essential role in low-rank matrix recovery such as matrix completion Jain et al. [2013], Ge et al. [2016], Sun and Luo [2016], matrix sensing Chi et al. [2019], Zhao et al. [2015], Charisopoulos et al. [2021], robust principal component analysis Candès et al. [2011], Cai et al. [2021], and the theoretical analysis of deep neural network Du et al. [2018]. Meanwhile, low-rank matrix factorization is also viewed as a canonical problem in non-convex optimization as the objective function to be minimized is non-convex and even non-smooth. Mathematically, we are to solve
$$
\min  _ {\boldsymbol {U} \in \mathbb {R} ^ {m \times d}, \boldsymbol {V} \in \mathbb {R} ^ {n \times d}} f (\boldsymbol {U}, \boldsymbol {V}) := \frac {1}{2} \| \boldsymbol {U} \boldsymbol {V} ^ {\top} - \boldsymbol {M} \| _ {F} ^ {2}, \tag {1}
$$
where  $d \ll \min(m, n)$ . Though problem (1) is not difficult to solve, the study of this problem has great significance to the gradient-based algorithm for low-rank matrix recovery. Hou et al. [2020], Li et al. [2019b], Chen et al. [2019], Tong et al. [2021], as it is exactly the population loss of low-rank matrix recovery models. Meanwhile, from the perspective of non-convex optimization, problem (1) is an ideal test bed for the theoretical analysis of the asymptotic convergence of gradient-based algorithm for non-convex optimization.
The theoretical guarantee for the global convergence of gradient-based algorithm for problem (1) is challenging, which is due to the following reasons: 1) the problem is non-convex with respect to the variables  $\mathbf{U}$  and  $\mathbf{V}$ , and there are infinitely many local minima and saddle points. Specifically, if  $\mathbf{U}^*$  and  $\mathbf{V}^*$  is an optimal solution of problem (1), then  $\mathbf{U}^*\mathbf{Q}$  and  $\mathbf{V}^*\mathbf{Q}^{-\top}$  is also an optimal solution for any invertible matrix  $\mathbf{Q}$ ; 2) the problem is non-smooth with respect to the variables  $\mathbf{U}$  and  $\mathbf{V}$  and is not coercive due to  $f(\alpha \mathbf{U}\frac{1}{\alpha}\mathbf{V}^{\top}) = f(\mathbf{U}\mathbf{V}^{\top})$  where the scalar  $\alpha$  can be arbitrarily large or small. In theory, gradient-based algorithm is only able to find critical points, while practically, gradient descent algorithm has been verified to converge to the global minima of problem (1) efficiently.
To close the gap between theory and practice, Li et al. [2019a], Ge et al. [2017], Chi et al. [2019] proved that even though the loss in Eq. (1) is non-convex its loss landscape has some nice property: all local minima are global optima and all the saddle points are strict saddles. Therefore gradient descent algorithms can be guarantee to converge to the global minima. To help escape the strict saddles, Jin et al. [2017] proposed perturbed gradient descent by adding isotropic noise to the gradient at each iteration, they prove that with high probability, perturbed gradient descent converges to the global minima from random initialization at a linear rate. While as analyzed in Ye and Du [2021] and verified by experiments that the gradient perturbation is not really necessary for problem (1).
In contrast to the perturbed gradient descent, Du et al. [2018] studied the naive gradient descent for problem (1), they exploited the balancedness of the two factors  $\|U\|_F^2 - \|V\|_F^2$  maintained by gradient flow and proved polynomial convergence rate of gradient descent for problem (1) when  $d = 1$ . Furthermore, Ye and Du [2021] improved the results of Du et al. [2018] to rank  $d$  case, they proved that gradient descent converges to the  $\varepsilon$ -global minima of problem (1) after  $O\left(\frac{d\kappa^2}{\tau^2}\ln \frac{d\sigma_d}{\tau} + \frac{d\kappa^2}{\tau^2}\ln \frac{\sigma_d}{\varepsilon}\right)$  iterations. Ye and Du [2021] divided the convergence process into two stages: warm-up phase which takes  $O\left(\frac{d\kappa^2}{\tau^2}\ln \frac{d\sigma_d}{\tau}\right)$  iterations and local convergence phase which takes  $O\left(\frac{d\kappa^2}{\tau^2}\ln \frac{\sigma_d}{\varepsilon}\right)$  iterations. The warm-up phase in Ye and Du [2021] is actually the saddle avoid phase after which the gradient descent escapes all the saddle regions as shown in Fig. 1.
Figure 1: Illustration of the global convergence of GD and AltScaledGD from random initialization.
It can be seen from Fig. 1 and proved by Ye and Du [2021] that both the saddle avoid phase and local convergence
phase of gradient descent highly relies on the condition number  $\kappa$  of the matrix  $M$ . If the condition number  $\kappa$  is large, then gradient descent takes long time to escape the saddle regions and also converges slowly. It is therefore very important to know can we improve the gradient-based algorithm such that the global convergence is independent of the condition number? Besides, the global convergence of both Du et al. [2018] and Ye and Du [2021] require very small learning rate (related to the small constant  $\tau$ ), which seriously limits the application of gradient descent algorithm for ill-conditioned LRMF problem.
Recently, the scaled gradient descent algorithm (ScaledGD) Apuroop [2012], Mishra and Sepulchre [2016], Tanner and Wei [2016], has been proved by Tong et al. [2021], Tong [2022] to converge very fast from specialized initialization (spectral initialization) to the global minima of problem (1) and the convergence rate is independent of the condition number (Theorem 5 in Tong et al. [2021]). Yet the convergence result provided by Tong et al. [2021] is only local, whether the scaled gradient algorithm can escape saddle regions efficiently for the non-convex problem (1) is still not clear.
In this paper, we are the first to prove that ScaledGD as well as AltScaledGD converge to the global minima of problem (1) from general random Gaussian initialization, and the convergence rate is independent of the condition number of the matrix  $M$ . Moreover, we show that the global convergence
results of ScaledGD and AltScaledGD do not rely on small initialization, the global convergence of AltScaledGD does not even require a small learning rate, which significantly improves the result of Ye and Du [2021]. To sum up, the contributions of this paper are three-fold:
1. We provide a very simple proof framework for the convergence of ScaledGD and AltScaledGD from general random initialization. Specifically, we divide the optimization process into three phases: initial phase, saddle avoid phase and linear convergence phase. We prove that the loss decreases at rate  $(1 - \eta)^{2k}$  in the initial phase, and further decreases linearly as  $(1 - \chi_k)^k$  in the saddle avoid phase and the linear convergence phase, where  $\eta$  and  $\chi_k$  are independent of the condition number  $\kappa$  and  $\chi_k$  is monotonically increasing from  $\frac{\eta^2}{(2 - \eta)^2}$  to  $\eta$ ;
2. We prove that if the scale of the random initialization is smaller than a given constant (small initialization), then the loss decrease linearly as  $(1 - \eta)^k$  from such small initialization to the global minima for both ScaledGD and AltScaledGD.
3. We show that AltScaledGD is a significant improvement of the ScaledGD in that it converges fast with large learning rate up to 1. While in contrast the learning rate of ScaledGD should be smaller than a constant  $c_{\eta}$  that is much smaller than 1.
The organization of this paper is as follow. In Section 2, we introduce the related works on ScaledGD and AltScaledGD. In Section 3, we present our main results, then we give more detailed theoretical analysis on the proof sketch in Section 4. Finally, we conclude this work in Section 5.
In this section, we introduce the ScaledGD and the AltScaledGD as specified in Apuroop [2012], Mishra and Sepulchre [2016], Tanner and Wei [2016], Tong et al. [2021]. We show that our work is a significant improvement to these existing works on the global convergence analysis.
2.1 Scaled gradient descent
Different to the gradient descent algorithm which takes the negative gradient direction as the descent direction, scaled gradient descent is designed to accelerate the convergence process by scaled the gradient with a preconditioning matrix. Specifically, the ScaledGD updates the variables  $\mathbf{U}_k$ ,  $\mathbf{V}_k$  as
$$
\left\{ \begin{array}{l} \boldsymbol {U} _ {k + 1} = \boldsymbol {U} _ {k} - \eta \nabla_ {\boldsymbol {U} _ {k}} f (\boldsymbol {U} _ {k}, \boldsymbol {V} _ {k}) \left(\boldsymbol {V} _ {k} ^ {\top} \boldsymbol {V} _ {k}\right) ^ {- 1} \\ \boldsymbol {V} _ {k + 1} = \boldsymbol {V} _ {k} - \eta \nabla_ {\boldsymbol {V} _ {k}} f (\boldsymbol {U} _ {k}, \boldsymbol {V} _ {k}) \left(\boldsymbol {U} _ {k} ^ {\top} \boldsymbol {U} _ {k}\right) ^ {- 1} \end{array} \right. \tag {2}
$$
where  $\eta$  is the learning rate,  $U_{k}^{\top}U_{k}$  and  $V_{k}^{\top}V_{k}$  are matrices of  $d\times d$ , and  $d\ll \min (m,n)$  therefore the computation of ScaledGD is comparable to that of gradient descent. The inverse matrices  $(V_{k}^{\top}V_{k})^{-1}$  and  $(U_{k}^{\top}U_{k})^{-1}$  is the preconditioning for the gradient descent. If we denote  $\boldsymbol {X} = (\boldsymbol {U},\boldsymbol {V})$ , then Eq. (2) corresponds to
$$
\boldsymbol {X} _ {k + 1} = \boldsymbol {X} _ {k} - \eta \nabla f _ {\boldsymbol {X} _ {k}} (\boldsymbol {X} _ {k}) \boldsymbol {H} _ {k} \tag {3}
$$
where  $\pmb{H}_k = \begin{bmatrix} (\pmb{V}_k^\top \pmb{V}_k)^{-1} & 0 \\ 0 & (\pmb{U}_k^\top \pmb{U}_k)^{-1} \end{bmatrix}$ .
Apuroop [2012], Mishra and Sepulchre [2016] proved that ScaledGD is derived by imposing a new metric on the tangent space of the Riemannian manifold. They verified empirically that ScaledGD converges much faster than gradient descent while there is no rigorous convergence rate analysis. Recently, Tong et al. [2021] is the first to prove the linear convergence property of Eq. (2) for problem (1), while their proof relies on specialized initialization that  $\mathrm{dist}(\mathbf{X}_0,\mathbf{X}_*)\leq 0.1\sigma_d(M)$  where  $X^{*} = (U^{*},V^{*})$  and  $U^{*}V^{*^{\top}} = M$  (Theorem 5 Tong et al. [2021]). The local convergence guarantee is far from satisfactory to understand the convergence of ScaledGD for the non-convex optimization problem (1).
2.2 Alternating scaled gradient descent
The Gaussian-Seidel version of ScaledGD is the following alternating scaled gradient descent which writes
$$
\left\{ \begin{array}{l} \boldsymbol {U} _ {k + 1} = \boldsymbol {U} _ {k} - \eta \nabla_ {\boldsymbol {U} _ {k}} f (\boldsymbol {U} _ {k}, \boldsymbol {V} _ {k}) \left(\boldsymbol {V} _ {k} ^ {\top} \boldsymbol {V} _ {k}\right) ^ {- 1} \\ \boldsymbol {V} _ {k + 1} = \boldsymbol {V} _ {k} - \eta \nabla_ {\boldsymbol {V} _ {k}} f (\boldsymbol {U} _ {k + 1}, \boldsymbol {V} _ {k}) \left(\boldsymbol {U} _ {k + 1} ^ {\top} \boldsymbol {U} _ {k + 1}\right) ^ {- 1} \end{array} \right. \tag {4}
$$
Eq. (4) was studied as scaled alternating steepest descent algorithm in Tanner and Wei [2016], and it is also closely related to the alternating minimization algorithm for the minimization problem (1) when  $\eta = 1$ , and other matrix recovery problem as Wen et al. [2012], Jain et al. [2013], Chandrasekher et al. [2022]. Different to alternating minimization, the AltScaledGD presented in Eq. (4) can be broadly used in lots of low-rank matrix recovery problem where alternating minimization is computationally prohibitive, such as matrix completion Zilber and Nadler [2022], Sun and Luo [2016], matrix sensing Ma et al. [2021]. Existing works for Eq. (4) only proved convergence to critical point as Wen et al. [2012], Tanner and Wei [2016], yet global convergence analysis of AltScaledGD is still vague.
In this paper, we provide rigorous proofs for the global convergence of ScaledGD Eq. (2) and AltScaledGD Eq. (4), and we show that both ScaledGD and AltScaledGD converge linearly for random Gaussian initialization after saddle avoidance. Meanwhile, we show that AltScaledGD is robust to the learning rate  $\eta$  which can be set as large as 1, while large  $\eta$  can seriously deteriorates the convergence property of ScaledGD as illustrated by Fig. 6, which sheds light on the superiority of AltScaledGD Eq. (4) over ScaledGD Eq. (2) on problem (1) as well as more low-rank matrix recovery problem.
In this section, we present our main theorems on the convergence of ScaledGD and AltScaledGD for two different random initialization: general random initialization and small initialization. These two initializations are both random Gaussian initialization with zero mean but different variances. Small initialization is widely used in the convergence analysis of low rank matrix factorization problem Stöger and Soltanolkotabi [2021], Ye and Du [2021], Ma and Fattahi [2022], while small initialization is skin to spectral initialization which does not help us fully understand the global convergence of the non-convex problem. In this paper, we provide both the global convergence analysis of general random initialization and small initialization.
3.1 Global convergence of ScaledGD
If the matrix  $M$  is rank one, i.e.,  $d$  in Eq. (1) is 1, then Eq. (2) is exactly gradient descent with adaptive step-size. We show that such specialized gradient descent for  $d \geq 1$  converges linearly to the global minima after an initial decreasing phase and the convergence rate is independent of the singular value of  $M$ .
Theorem 1 (General random initialization). Let  $\mathbf{U}_0 \in \mathbb{R}^{m \times d}$  and  $V_0 \in \mathbb{R}^{n \times d}$  be random Gaussian that follows  $\mathcal{N}(0, \sigma)$  for  $\sigma > c_{\mathrm{init}}$  ( $c_{\mathrm{init}}$  is a positive constant), and  $\mathbf{U}_k$ ,  $\mathbf{V}_k$  are updated by Eq. (2). If  $\eta \leq c_{\eta} < 1$  for small constant  $c_{\eta}$ , we have that the objective function of problem (1) decreases linearly after  $T_1 = O\left(\ln \frac{d}{\delta}\right)$  iterations, namely
$$
\left\| \boldsymbol {U} _ {k + T _ {1}} \boldsymbol {V} _ {k + T _ {1}} ^ {\top} - \boldsymbol {M} \right\| _ {F} \leq \alpha_ {1} \left(1 - \chi_ {k + T _ {1}}\right) ^ {k} \| \boldsymbol {M} \| _ {F}, \forall k \geq 0 \tag {5}
$$
where  $\chi_{k + T_1}$  is monotonically increasing from  $\frac{\eta^2}{(2 - \eta)^2}$  to  $\eta$ ,  $\delta$  is a sufficiently small constant,  $\alpha_{1}$  is a constant.
The Theorem 1 indicates that the global convergence of ScaledGD can be divided into three phases: the initial phase that lasts  $T_{1}$  iterations, the saddle avoid phase in which  $\chi_{k + T_1}$  increases from  $\frac{\eta / 2}{1 - \eta / 2}$  to  $\eta$  and the final linear convergence phase with convergence rate  $1 - \eta$ . While if the scale of the initialization  $U_{0}$  and  $V_{0}$  are very small (with small  $\sigma$ ), then the following theorem shows that the ScaledGD converges linearly without entering the saddle regions.
Theorem 2 (Small initialization). Let  $\mathbf{U}_0 \in \mathbb{R}^{m \times d}$  and  $\mathbf{V}_0 \in \mathbb{R}^{n \times d}$  be random Gaussian that follows  $\mathcal{N}(0, \sigma)$ , with  $\sigma \leq c_{\mathrm{init}}$  and  $\mathbf{U}_k$ ,  $\mathbf{V}_k$  are updated by Eq. (2). If  $\eta \leq c_{\eta} < 1$  for small constant  $c_{\eta}$ , we have that the objective function of problem (1) decreases linearly, namely
$$
\left\| \boldsymbol {U} _ {k} \boldsymbol {V} _ {k} ^ {\top} - \boldsymbol {M} \right\| _ {F} \leq \alpha_ {2} (1 - \eta) ^ {k} \| \boldsymbol {M} \| _ {F} \tag {6}
$$
where  $c_{\mathrm{init}}$  is a small constant and  $\alpha_{2}$  is a constant.
3.2 Global convergence of AltScaledGD
We now present the main convergence results of AltScaledGD.
Theorem 3 (General random initialization). Let  $\mathbf{U}_0 \in \mathbb{R}^{m \times d}$  and  $\mathbf{V}_0 \in \mathbb{R}^{n \times d}$  be random Gaussian that follows  $\mathcal{N}(0, \sigma)$  for  $\sigma > c_{\mathrm{init}}$ ,  $\mathbf{U}_k$ ,  $\mathbf{V}_k$  are updated by Eq. (4), we have that the objective function of problem (1) decreases linearly after  $T_1 = O(\ln \frac{d}{\delta})$  iterations, namely
$$
\left\| \boldsymbol {U} _ {k + T _ {1}} \boldsymbol {V} _ {k + T _ {1}} ^ {\top} - \boldsymbol {M} \right\| _ {F} \leq \alpha_ {1} \left(1 - \chi_ {k + T _ {1}}\right) ^ {k} \| \boldsymbol {M} \| _ {F} \tag {7}
$$
where  $\chi_{k + T_1}$  is monotonically increasing from  $\frac{\eta^2}{(2 - \eta)^2}$  to  $\eta$ ,  $0 < \eta \leq 1$  and  $\alpha_{1}$  is a constant.
Theorem 4 (Small initialization). Let  $\mathbf{U}_0 \in \mathbb{R}^{m \times d}$  and  $\mathbf{V}_0 \in \mathbb{R}^{n \times d}$  be random Gaussian that follows  $\mathcal{N}(0, \rho)$ , with  $\sigma \leq c_{\mathrm{init}}$ ,  $\mathbf{U}_k$ ,  $\mathbf{V}_k$  are updated by Eq. (4) then we have that the objective function of problem (1) decreases linearly, namely
$$
\left\| \boldsymbol {U} _ {k} \boldsymbol {V} _ {k} ^ {\top} - \boldsymbol {M} \right\| _ {F} \leq \alpha_ {2} (1 - \eta) ^ {k} \| \boldsymbol {M} \| _ {F} \tag {8}
$$
where  $0 < \eta \leq 1$  is the step size,  $\alpha_{2}$  is a constant and  $c_{\mathrm{init}}$  is a small constant.
The convergence results of ScaledGD and AltScaledGD are almost the same with differences in that for ScaledGD the learning  $\eta$  should be smaller than a constant  $c_{\eta}$  which is much less than 1. The small constant  $c_{\eta}$  greatly restricts the convergence rate of ScaledGD. While for AltScaledGD the learning rate  $\eta$  can be as large as 1, which indicates the superiority of AltScaledGD over ScaledGD in convergence as  $\eta$  is crucial to the convergence rate. From the above theorems, it can be easily deduced that both ScaledGD and AltScaledGD converge to an  $\varepsilon$ -global minima after  $O(\ln \frac{d}{\delta} + \ln \frac{d}{\varepsilon})$  iterations from general random initialization. While for small initialization, these two algorithms only need  $O(\ln \frac{d}{\varepsilon})$  iterations to converge to an  $\varepsilon$ -global minima. More detailed analysis and proofs are provided in Section 4 the proof sketch part and the supplementary materials.
3.3 Why does preconditioning help?
In previous works, preconditioning has been used to improve the condition of the optimization problem Saad [2003], Zhang et al. [2021]. In this paper, we analyze how does the preconditioning help improving the convergence by analyzing the effect of condition number  $\kappa$  on the learning rate  $\eta$  as the convergence rate highly depends on the learning rate. For simplicity, we take one step of the AltScaledGD as example to analyze the effect of the preconditioning. Since  $f(\pmb{U},\pmb{V})$  is quadratic on  $\pmb{U}$ , then it can be verified
$$
f \left(\boldsymbol {U} _ {k + 1}, \boldsymbol {V} _ {k}\right) \leq f \left(\boldsymbol {U} _ {k}, \boldsymbol {V} _ {k}\right) + \langle \nabla f _ {\boldsymbol {U}}, \Delta \rangle + \frac {1}{2} \| \Delta \| _ {\mathcal {A} _ {k}} ^ {2} \tag {9}
$$
where  $\Delta = \mathbf{U}_{k + 1} - \mathbf{U}_k$  and  $\| \cdot \|_{\mathcal{A}_k}$  is a local norm defined by  $\| \Delta \|_{\mathcal{A}_k} = \langle \Delta V_k^\top V_k,\Delta \rangle$
For gradient descent, we take  $\Delta = -\eta \nabla f_{\mathbf{U}}(\mathbf{U}_k,\mathbf{V}_k)$ , then Eq. (9) becomes
$$
\begin{array}{l} f (\boldsymbol {U} _ {k + 1}, \boldsymbol {V} _ {k}) \leq f (\boldsymbol {U} _ {k}, \boldsymbol {V} _ {k}) - \eta \| \nabla f _ {\boldsymbol {U}} (\boldsymbol {U} _ {k}, \boldsymbol {V} _ {k}) \| _ {F} ^ {2} + \frac {\eta^ {2} \sigma_ {1} ^ {2} (\boldsymbol {V} _ {k})}{2} \| \nabla f _ {\boldsymbol {U}} (\boldsymbol {U} _ {k}, \boldsymbol {V} _ {k}) \| _ {F} ^ {2} \\ \leq f \left(\boldsymbol {U} _ {k}, \boldsymbol {V} _ {k}\right) - 2 \eta \sigma_ {d} ^ {2} \left(\boldsymbol {V} _ {k}\right) f \left(\boldsymbol {U} _ {k}, \boldsymbol {V} _ {k}\right) + \eta^ {2} \sigma_ {1} ^ {4} \left(\boldsymbol {V} _ {k}\right) f \left(\boldsymbol {U} _ {k}, \boldsymbol {V} _ {k}\right) \tag {10} \\ \leq \left(1 - \left(2 \eta \sigma_ {d} ^ {2} (\boldsymbol {V} _ {k}) - \eta^ {2} \sigma_ {1} ^ {4} (\boldsymbol {V} _ {k})\right)\right) f (\boldsymbol {U} _ {k}, \boldsymbol {V} _ {k}) \\ \end{array}
$$
Similarly, it holds
$$
f \left(\boldsymbol {U} _ {k + 1}, \boldsymbol {V} _ {k + 1}\right) \leq \left(1 - \left(2 \eta \sigma_ {d} ^ {2} \left(\boldsymbol {U} _ {k + 1}\right) - \eta^ {2} \sigma_ {1} ^ {4} \left(\boldsymbol {U} _ {k + 1}\right)\right)\right) f \left(\boldsymbol {U} _ {k + 1}, \boldsymbol {V} _ {k}\right) \tag {11}
$$
$$
f \left(\boldsymbol {U} _ {k + 1}, \boldsymbol {V} _ {k + 1}\right) \leq \left(1 - \left(2 \eta \sigma_ {d} ^ {2} \left(\boldsymbol {U} _ {k + 1}\right) - \eta^ {2} \sigma_ {1} ^ {4} \left(\boldsymbol {U} _ {k + 1}\right)\right)\right) \left(1 - \left(2 \eta \sigma_ {d} ^ {2} \left(\boldsymbol {V} _ {k}\right) - \eta^ {2} \sigma_ {1} ^ {4} \left(\boldsymbol {V} _ {k}\right)\right)\right) f \left(\boldsymbol {U} _ {k}, \boldsymbol {V} _ {k}\right) \tag {12}
$$
Figure 2: Illustration of convergence of ScaledGD and AltScaledGD under different condition  $\kappa$  and different initialization.
To guarantee the linear convergence of gradient descent, it is required that  $2\eta \sigma_r^2(\mathbf{V}_k) \geq \eta^2 \sigma_1^4(\mathbf{V}_k)$  and  $2\eta \sigma_r^2(\mathbf{U}_{k+1}) \geq \eta^2 \sigma_1^4(\mathbf{U}_{k+1})$  which implies  $\eta \leq \min \left\{\frac{2}{\sigma_1^2(\mathbf{V}_k)} \kappa(\mathbf{V}_k)^2, \frac{2}{\sigma_1^2(\mathbf{U}_{k+1})} \kappa(\mathbf{U}_{k+1})^2\right\}^2$ . In contrast, if we take  $\Delta = -\eta \nabla f_U(\mathbf{U}_k, \mathbf{V}_k)(\mathbf{V}_k^\top \mathbf{V}_k)^{-1}$ , then