SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Daesung Kim
Samsung Electronics dskim95phd@gmail.com
Hye Won Chung
hwchung@kaist.ac.kr
The nonconvex formulation of the matrix completion problem has received significant attention in recent years due to its affordable complexity compared to the convex formulation. Gradient Descent (GD) is a simple yet efficient baseline algorithm for solving nonconvex optimization problems. The success of GD has been witnessed in many different problems in both theory and practice when it is combined with random initialization. However, previous works on matrix completion require either careful initialization or regularizers to prove the convergence of GD. In this paper, we study the rank-1 symmetric matrix completion and prove that GD converges to the ground truth when small random initialization is used. We show that in a logarithmic number of iterations, the trajectory enters the region where local convergence occurs. We provide an upper bound on the initialization size that is sufficient to guarantee the convergence, and show that a larger initialization can be used as more samples are available. We observe that the implicit regularization effect of GD plays a critical role in the analysis, and for the entire trajectory, it prevents each entry from becoming much larger than the others.
Recovering a low-rank matrix from a set of linear measurements is at the heart of many statistical learning problems. Depending on the structure of the matrix and the linear measurements, it reduces to various problems such as phase retrieval [1], blind deconvolution [2], and matrix sensing [3]. Matrix completion [4] is also one such type of problem where each measurement provides an entry of the matrix, and the goal is to recover the low-rank matrix from a partial, usually very sparse, observation of the entries. One of the most notable applications of matrix completion is collaborative filtering [5], which aims to predict user preferences for items based on a highly incomplete observation of user-item ratings. There are also a number of different applications, such as principal component analysis [6] and image reconstruction [7], just to name a few.
Extensive amount of work has been dedicated to provide an efficient recovery algorithm for matrix completion with theoretical guarantees [8]. The convex relaxation based nuclear norm minimization [4, 9] was the first algorithm proven to recover the matrix with near optimal sample complexity. Despite its theoretical success, the convex algorithm was found hard to be used in practical scenarios due to its unaffordable computational complexity and memory size. Therefore, the nonconvex formulation of matrix completion with quadratic loss has received significant attention in recent years. Many different algorithms have been proposed for the nonconvex problem, and their convergence toward the ground truth has been analyzed. Examples include optimization on Grassmann manifolds [10], alternating minimization [11], projected gradient descent [12], gradient descent with regularizer [13], and (vanilla) gradient descent [14, 15].
Gradient descent (GD) has served as a baseline algorithm for solving nonconvex optimization problems. However, the convergence of GD to global minimizers is not guaranteed, and it can take exponential time to escape saddle points [16]. Nevertheless, GD with random initialization has been shown to successfully recover the global minimum in many different problems such as phase retrieval [1], matrix sensing [17], matrix factorization [18], and neural network training [19]. Previous work on matrix completion [14, 15] proved the convergence of GD under the spectral initialization, which locates the initial point in the local region of the minima. However, the role of random initialization in solving matrix completion with GD is not fully understood yet, although its success is observed in practice. Therefore, we aim to answer the following question:
Can  $GD$  with random initialization solve the nonconvex matrix completion problem?
We answer this question affirmatively and show that GD with small random initialization successfully converges to the ground truth for rank-1 symmetric matrix completion. In the analysis, we use vanilla GD, which does not incorporate any modifications, such as regularization or truncation, into the GD algorithm. We also characterize the entire trajectory that GD follows by showing that the trajectory is well approximated by the fully observed case. The small initialization plays a critical role in analyzing the trajectory of the early stages, where the randomly initialized vector is nearly orthogonal to the first eigenvector of the ground truth matrix. We provide a bound on the required initialization size for the algorithm to converge, and our bound suggests that one can use a larger initialization to improve the convergence speed as more samples are provided. However, in any case, GD with a small random initialization takes only logarithmic amount of time (with respect to the matrix dimension) to reach the point where local convergence can begin. To the best of our knowledge, this is the first result on matrix completion that proves the convergence of vanilla GD without a carefully designed initialization.
Although our result is restricted to the rank-1 case, we believe that this work provides an important evidence for understanding the more general rank- $r$  case. At the end of this paper, we will discuss some technical difficulties that the rank- $r$  case naturally has, and provide some empirical results related to them. However, studying the rank-1 matrix completion problem is not only motivated by theoretical interest, but the problem itself also appears in some practical problems such as crowdsourcing [20, 21].
Related Works This work is motivated by the recent success of small initialization in matrix factorization and matrix sensing. It was first conjectured in [22] that sufficiently small step sizes and initialization lead GD to converge to the minimum nuclear norm solution of a full-dimensional matrix sensing problem. The conjecture was proved in [17] for the fully overparameterized matrix sensing under the standard restricted isometry property (RIP). A recent study by [23] provided more general results by showing that the early iterations of GD with small initialization have spectral bias. Many other works such as [24, 25, 26] have also studied how GD or gradient flow with small initialization implicitly forces the recovered matrix to be low-rank. However, the recovery guarantee for matrix completion has not been provided by any work.
For the matrix sensing where RIP holds, the loss function has global benign geometry in that it does not contain any spurious local minima or non-strict saddle points [27]. In the case of matrix completion, a similar result was obtained but with a regularizer that penalizes the matrices with large rows [28]. Controlling the norm of each row (absolute value of each entry in the case of rank-1) is the biggest hurdle in the analysis of matrix completion. In the local convergence analysis of [14], it was proved that GD implicitly regularizes the largest  $\ell_2$ -norm of the rows of error matrices, showing that explicit regularization is unnecessary. In this paper, we also prove that such an implicit regularization is induced by GD when it starts from a point of small size. We show that the trajectory is close to the fully observed case in both  $\ell_2$  and  $\ell_{\infty}$  norms. Thus, the trajectory is confined to the region where it has benign geometry, and GD can converge without an explicit regularizer.
Notations We denote vectors with lowercase bold letters and matrices with uppercase bold letters. The components or entries of them are written without bold. We use  $\| \cdot \| _2$  and  $\| \cdot \|_{\infty}$  to denote  $\ell_2$  and  $\ell_{\infty}$ -norm of vectors, respectively, and  $\| \cdot \|_{\mathrm{F}}$  is used for Frobenius norm of matrices. For any norm  $\| \cdot \|$  and two vectors  $x,y$ , we let  $\| x\pm y\| = \min \{\| x + y\| ,\| x - y\|\}$ . Asymptotic dependencies with respect to the matrix dimension are denoted with the standard big  $O$  notations, or with the symbols,  $\lesssim ,\asymp$  and  $\gtrsim$
The matrix completion problem aims to reconstruct a low-rank matrix from partially observed entries. In this paper, we focus on the case where the ground truth matrix, denoted by  $M^{\star} \in \mathbb{R}^{n \times n}$ , is a rank-1 positive semidefinite matrix. Thus, the ground truth matrix is decomposed as  $M^{\star} = \lambda^{\star} \pmb{u}^{\star} \pmb{u}^{\star \top}$  with  $\lambda^{\star} > 0$  and a unit vector  $\pmb{u}^{\star}$ . We define  $\pmb{x}^{\star} = \sqrt{\lambda^{\star}} \pmb{u}^{\star}$  so that  $M^{\star} = \pmb{x}^{\star} \pmb{x}^{\top}$ . To follow the standard incoherence assumption, we let  $\| \pmb{u}^{\star} \|_{\infty} = \sqrt{\frac{\mu}{n}}$  and allow  $\mu$  to be as large as  $\text{poly}(\log n)$ . We consider a random sampling model that is also symmetric as  $M^{\star}$ . Each entry in the diagonal and the upper (or lower) triangular part of  $M^{\star}$  is independently revealed with probability  $0 < p \leq 1$ . We consider the noisy case where Gaussian noise is added to each observation. Formally, we get as an observation the matrix  $M^{\circ}$  whose  $(i,j)$ th entry is  $\frac{1}{p} \delta_{ij}(M_{ij}^{\star} + E_{ij})$ , where  $[\delta_{ij}]_{1 \leq i \leq j \leq n}$  are independent Bernoulli random variables with expectation  $p$  and  $[E_{ij}]_{1 \leq i \leq j \leq n}$  are independent Gaussian random variables with the distribution  $\mathcal{N}(0, \sigma^2)$ . They are both symmetric in the sense that  $\delta_{ij} = \delta_{ji}$  and  $E_{ij} = E_{ji}$  for all  $1 \leq i \leq j \leq n$ . We use  $E$  to denote the symmetric matrix whose entries are  $E_{ij}$ . We denote the set of observed entries as  $\Omega := \{(i,j) | \delta_{ij} = 1\}$ , and define an operator  $\mathcal{P}_{\Omega}$  on matrices that sets the entries not contained in  $\Omega$  to zero. (e.g.  $M^{\circ} = \frac{1}{p} \mathcal{P}_{\Omega}(M^{\star} + E)$ )
To recover the matrix  $M^{\star}$ , we find  $\pmb{x} \in \mathbb{R}^n$  that minimizes the nonconvex loss function  $f(\pmb{x})$ , which is the sum of the squared differences on the observed entries. It is explicitly written as  $f(\pmb{x}) \coloneqq \frac{1}{4p} \sum_{(i,j) \in \Omega} (x_i x_j - x_i^\star x_j^\star - E_{ij})^2$ . We apply vanilla GD to solve the optimization problem starting from a small randomly initialized vector  $\pmb{x}^{(0)}$ . Each entry of  $\pmb{x}^{(0)}$  is sampled independently from the Gaussian distribution  $\mathcal{N}\left(0, \frac{1}{n} \beta_0^2\right)$ , so that the squared norm of  $\pmb{x}^{(0)}$  is expected to be  $\beta_0^2$ . The update rule of GD is written as
$$
\boldsymbol {x} ^ {(t + 1)} = \boldsymbol {x} ^ {(t)} - \eta \nabla f (\boldsymbol {x} ^ {(t)}) = \boldsymbol {x} ^ {(t)} - \frac {\eta}{p} \mathcal {P} _ {\Omega} (\boldsymbol {x} ^ {(t)} \boldsymbol {x} ^ {(t) \top}) \boldsymbol {x} ^ {(t)} + \eta \boldsymbol {M} ^ {\circ} \boldsymbol {x} ^ {(t)}, \tag {1}
$$
where  $\eta > 0$  is the step size.
We define  $F$  as the loss function  $f$  when all entries of  $M^{\star}$  are observed without noise, i.e.,  $F(\pmb {x}):= \frac{1}{4}\left\| \pmb {x}\pmb{x}^{\top} - M^{\star}\right\|_{\mathrm{F}}^{2}$ . We also define  $\widetilde{\pmb{x}}^{(t)}$  as the trajectory of GD when it is applied to  $F$  with the same initial point  $\pmb{x}^{(0)}$ , i.e.,  $\widetilde{\pmb{x}}^{(t)}$  is the trajectory of the fully observed case. Specifically, it evolves with
$$
\widetilde {\boldsymbol {x}} ^ {(t + 1)} = \widetilde {\boldsymbol {x}} ^ {(t)} - \eta \nabla F (\widetilde {\boldsymbol {x}} ^ {(t)}) = \widetilde {\boldsymbol {x}} ^ {(t)} - \eta \left\| \widetilde {\boldsymbol {x}} ^ {(t)} \right\| _ {2} ^ {2} \widetilde {\boldsymbol {x}} ^ {(t)} + \eta M ^ {\star} \widetilde {\boldsymbol {x}} ^ {(t)} \tag {2}
$$
from the same starting point  $\widetilde{\pmb{x}}^{(0)} = \pmb{x}^{(0)}$
Lastly, we introduce the so-called leave-one-out sequences. These were the main ingredient in controlling the  $\ell_{\infty}$ -norm of trajectory in [14]. We use them for a similar purpose. For each  $l \in [n]$ , we define an operator  $\mathcal{P}_{\Omega}^{(l)}$  such that  $\mathcal{P}_{\Omega}^{(l)}(\pmb{X})$  is equal to  $\pmb{X}$  on the  $l$ th row and column, and equal to  $\frac{1}{p}\mathcal{P}_{\Omega}(\pmb{X})$  otherwise. The  $l$ th leave-one-out sequence,  $\pmb{x}^{(t,l)}$ , evolves with
$$
\boldsymbol {x} ^ {(t + 1, l)} = \boldsymbol {x} ^ {(t, l)} - \eta \mathcal {P} _ {\Omega} ^ {(l)} \left(\boldsymbol {x} ^ {(t, l)} \boldsymbol {x} ^ {(t, l) \top}\right) \boldsymbol {x} ^ {(t, l)} + \eta \boldsymbol {M} ^ {(l)} \boldsymbol {x} ^ {(t, l)}, \tag {3}
$$
for  $\pmb{x}^{(0,l)} = \pmb{x}^{(0)}$ , where  $M^{(l)} = \mathcal{P}_{\Omega}^{(l)}(M^{\star}) + E^{(l)}$ , and  $E^{(l)}$  is obtained by zeroing out the  $l$ th row and column of  $\frac{1}{p}\mathcal{P}_{\Omega}(E)$ .
In this section, we present our main results. The first main result concerns the global convergence of GD with small random initialization.
Theorem 3.1. Let us consider a rank-1 matrix completion problem that recovers the matrix  $M^{\star} = \pmb{x}^{\star}\pmb{x}^{\star\top} \in \mathbb{R}^{n\times n}$  such that  $\| \pmb{x}^{\star}\|_{2} = \sqrt{\lambda^{\star}}$  and  $\| \pmb{x}^{\star}\|_{\infty} = \sqrt{\frac{\mu}{n}}\|\pmb{x}^{\star}\|_{2}$ , where  $\mu = O(\mathrm{poly}(\log n))$ . Let the initial point  $\pmb{x}^{(0)} \in \mathbb{R}^{n}$  be sampled from the Gaussian distribution  $\mathcal{N}(\mathbf{0},\frac{1}{n}\beta_0^2\pmb{I})$  and  $\pmb{x}^{(t)}$  be updated with (1). Suppose that a small step size with  $\eta \lambda^{\star} < 0.1$  is used and the sample complexity satisfies  $n^2 p \gtrsim \mu^5 n\log^{22}n$ . Then, there exists  $T^{\star} = (1 + o(1))\frac{1}{\eta\lambda^{\star}}\log \frac{\sqrt{\lambda^{\star}n}}{\beta_0}$  such that
$$
\left\| \boldsymbol {x} ^ {(t)} \pm \boldsymbol {x} ^ {\star} \right\| _ {2} \lesssim \frac {1}{\sqrt {\log n}} \| \boldsymbol {x} ^ {\star} \| _ {2}, \tag {4}
$$
$$
\left. \max  _ {1 \leq l \leq n} \left\| \boldsymbol {x} ^ {(t)} - \boldsymbol {x} ^ {(t, l)} \right\| _ {2} \lesssim \frac {1}{\sqrt {\log n}} \| \boldsymbol {x} ^ {\star} \| _ {\infty}, \right. \tag {6}
$$
$$
\left\| \boldsymbol {x} ^ {(t)} \pm \boldsymbol {x} ^ {\star} \right\| _ {\infty} \lesssim \frac {1}{\sqrt {\log n}} \| \boldsymbol {x} ^ {\star} \| _ {\infty}, \tag {5}
$$
$$
\max  _ {1 \leq l \leq n} \left| \left(\boldsymbol {x} ^ {(t, l)} - \boldsymbol {x} ^ {\star}\right) _ {l} \right| \lesssim \frac {1}{\sqrt {\log n}} \| \boldsymbol {x} ^ {\star} \| _ {\infty} \tag {7}
$$
hold at  $t = T^{\star}$  with probability at least  $1 - o(1 / \sqrt{\log n})$ , if a sufficiently small initialization with
$$
\sqrt {\lambda^ {\star}} n ^ {- 1 0} \lesssim \beta_ {0} \lesssim \sqrt {\lambda^ {\star}} \sqrt [ 4 ]{\frac {n p}{\mu^ {5} \log^ {2 6} n}} \frac {1}{\sqrt [ 4 ]{n}} \tag {8}
$$
is used and the noise satisfies  $\sigma \lesssim \frac{\lambda^{\star}\mu}{n}\sqrt{\log n}$ .
Theorem 3.1 proves that, starting from a small random initialization, the trajectory of GD eventually enters the local region of the global minimizers  $\pm x^{\star}$  in terms of both  $\ell_{2}$  and  $\ell_{\infty}$  norms. Combined with the result of [14], GD starts to converge linearly to either  $x^{\star}$  or  $-x^{\star}$  after  $t = T^{\star}$ , as stated in the corollary below.
Corollary 3.2. Suppose that the conditions in Theorem 3.1 are satisfied, and let  $\rho$  be a constant such that  $1 - \frac{\eta}{10} \leq \rho < 1$ . Then, with probability at least  $1 - o(1 / \sqrt{\log n})$ , we have
$$
\left\| \boldsymbol {x} ^ {(t)} \pm \boldsymbol {x} ^ {\star} \right\| _ {2} \lesssim \left(\frac {1}{\sqrt {\log n}} \rho^ {t - T ^ {\star}} + \frac {\sigma}{\lambda^ {\star}} \sqrt {\frac {n}{p}}\right) \| \boldsymbol {x} ^ {\star} \| _ {2}, \tag {9}
$$
$$
\left\| \boldsymbol {x} ^ {(t)} \pm \boldsymbol {x} ^ {\star} \right\| _ {\infty} \lesssim \left(\frac {1}{\sqrt {\log n}} \rho^ {t - T ^ {\star}} + \frac {\sigma}{\lambda^ {\star}} \sqrt {\frac {n}{p}}\right) \| \boldsymbol {x} ^ {\star} \| _ {\infty}, \tag {10}
$$
for all  $T^{\star}\leq t\leq T = O(n^{5})$
The desired global convergence result is provided by Corollary 3.2. Several remarks about Theorem 3.1 and Corollary 3.2 are in order.
Matrix Recovery Suppose  $\boldsymbol{x}^{(t)}$  converges to a global minimum  $\boldsymbol{y}^{\star}$  of the function  $f$ , which is different from  $\pm \boldsymbol{x}^{\star}$ . In such a case, despite achieving global convergence, the reconstructed matrix  $\boldsymbol{y}^{\star}\boldsymbol{y}^{\star\top}$  deviates from the ground truth matrix  $M^{\star}$ . However, Theorem 3.1 establishes that  $\boldsymbol{x}^{(t)}$  converges exclusively to the correct global minima  $\pm \boldsymbol{x}^{\star}$ , so that the matrix  $M^{\star}$  is recovered with high probability.
Leave-one-out Sequence To apply the local convergence result of [14], in addition to (4) and (5), the existence of leave-one-out sequences  $\{\pmb{x}^{(t,l)}\}_{l\in [n]}$  satisfying (6) and (7) is required. Leave-one-out sequences also play a critical role and appear naturally in the proof of Theorem 3.1.
Sample Complexity The required sample complexity for Theorem 3.1 to hold is optimal up to a logarithmic factor compared to the statistical lower bound of  $\Omega(n\log n)$ . We have not done our best to optimize the log factors, and about half of them can be reduced with more delicate analysis. We will discuss this briefly in Section 6.
Convergence Time Considering that  $\beta_0^{-1}$  is at most polynomial in  $n$  (due to the lower bound of (8)), only  $O(\log n)$  iterations are required for GD to enter the local region. It takes  $O(\log (\frac{1}{\epsilon}))$  more iterations to achieve  $\epsilon$ -accuracy in the local region, so the total iteration complexity is given by  $O(\log n) + O(\log (\frac{1}{\epsilon}))$ .
Initialization Size Although small initialization provides a good geometry to GD, a larger initialization is preferred because the convergence time,  $T^{\star}$ , is inversely proportional to  $\beta_0$ . When the sample complexity is optimal, i.e.,  $n^2 p \asymp n$  poly(log  $n$ ), an upper bound on the initialization size given by Theorem 3.1 is  $n^{-\frac{1}{4}}$ , ignoring the log factors. However, as more samples are provided, we are allowed to use a larger initialization to reduce the convergence time. When the sample complexity satisfies  $n^2 p \asymp n^{1 + a}$ , the bound is  $n^{-\frac{1}{4}(1 - a)}$  ignoring the log factors. The bound becomes nearly constant as  $a$  approaches 1, namely the fully observed case, and this is consistent with the previous result that small initialization is unnecessary for the fully observed case [18]. We also note that the lower bound of (8) is necessary in the proof of Theorem 3.1, since we derive probabilistic bounds for all iterations, and the lower bound limits the maximum number of iterations. However, we can further reduce the lower bound  $n^{-10}$  to  $n^{-c}$  for any constant  $c > 10$  by tuning some constant factors during the proof.
Noise Size From the incoherence assumption, the maximum absolute value of entries of  $M^{\star}$  is bounded by  $\frac{\lambda^{\star}\mu}{n}$ . The condition  $\sigma \lesssim \frac{\lambda^{\star}\mu}{n}\sqrt{\log n}$  in Theorem 3.1 allows the standard deviation of the Gaussian noise to be much larger than the maximum entry. It also implies  $\frac{\sigma}{\lambda^{\star}}\sqrt{\frac{n}{p}} \lesssim \mu \sqrt{\frac{\log n}{np}}$ , so that the upper bounds in Corollary 3.2 are dominated by the first terms at  $t = T^{\star}$  and they eventually converge to the second terms as  $t$  increases.
Estimation Error The current estimation bounds (4) to (7) are all proportional to  $\frac{1}{\sqrt{\log n}}$  times the norms of  $x^{\star}$ . However, if we do not allow the initialization size to grow with the sample complexity, we are able to obtain tighter bounds; if we use the fixed initialization size  $n^{-\frac{1}{4}}$  regardless of the sample complexity, in Theorem 3.1, the factor  $\frac{1}{\sqrt{\log n}}$  is improved to  $\frac{1}{\sqrt{np}} + \frac{\sigma}{\lambda^{\star}} \sqrt{\frac{n}{p}}$ , and the upper bound on noise size is also improved to  $\sigma \lesssim \frac{\lambda^{\star} \mu}{n} \sqrt{np}$  (not being precise on the factors of  $\mu$  and  $\log n$  here). Then, the estimation error in Corollary 3.2 is improved to  $\frac{1}{\sqrt{np}} \rho^{t} + \frac{\sigma}{\lambda^{\star}} \sqrt{\frac{n}{p}}$  to match the result of [14] which uses spectral initialization. Thus, we have a tradeoff between estimation error and initialization size.
The next main result concerns the trajectory of GD before it enters the local region. The theorem states that for all  $t \leq T^{\star}$ ,  $\pmb{x}^{(t)}$  stays close to the fully observed case  $\widetilde{\pmb{x}}^{(t)}$  in both  $\ell_{2}$  and  $\ell_{\infty}$ -norm.
Theorem 3.3. Suppose that the conditions of Theorem 3.1 hold, and  $T^{\star}$  is defined as in Theorem 3.1. Then, for all  $t \leq T^{\star}$ , we have
$$
\left\| \boldsymbol {x} ^ {(t)} - \widetilde {\boldsymbol {x}} ^ {(t)} \right\| _ {2} \lesssim \frac {1}{\sqrt {\log n}} \left\| \widetilde {\boldsymbol {x}} ^ {(t)} \right\| _ {2}, \tag {12}
$$
with probability at least  $1 - o(1 / \sqrt{\log n})$ .
Trajectory of GD The sequence  $\widetilde{\pmb{x}}^{(t)}$  is a linear combination of  $\pmb{x}^{(0)}$  and  $\pmb{u}^{\star}$  (see (C.1) in the appendix), and it is easy to analyze how  $\widetilde{\pmb{x}}^{(t)}$  evolves. By showing that  $\pmb{x}^{(t)}$  stays close to  $\widetilde{\pmb{x}}^{(t)}$  for all iterations, we not only show the convergence of GD with small initialization as in Theorem 3.1, but also characterize the exact trajectory that GD follows by Theorem 3.3.
Implicit Regularization One can prove that  $\widetilde{\pmb{x}}^{(t)}$  is incoherent up to some log factors over all iterations, and from (11) and (12), the incoherence of  $\pmb{x}^{(t)}$  is bounded by that of  $\widetilde{\pmb{x}}^{(t)}$ . Thus, Theorem 3.3 shows that the incoherence of  $\pmb{x}^{(t)}$  is implicitly controlled by GD without any regularizer. This is an improvement over the previous result on the global convergence of GD for matrix completion [28], where an explicit regularizer was used to control the  $\ell_{\infty}$ -norm of  $\pmb{x}^{(t)}$ , although no small initialization was used in that work.
4 Fully Observed Case and Proof Sketch
Before we explain the proof of Theorems 3.1 and 3.3, we describe the trajectory of the fully observed case. We characterize  $\widetilde{\pmb{x}}^{(t)}$  with three variables:  $\widetilde{\alpha}_t = |\pmb{u}^{\star \top}\widetilde{\pmb{x}}^{(t)}|$ ,  $\widetilde{\beta}_t = \| \widetilde{\pmb{x}}^{(t)}\|_2$ , and  $\widetilde{\gamma}_t = \| \widetilde{\pmb{x}}_{\perp}^{(t)}\|_2$ , where  $\widetilde{\pmb{x}}_{\perp}^{(t)} = \widetilde{\pmb{x}}^{(t)} - \pmb{u}^{\star}\pmb{u}^{\star \top}\widetilde{\pmb{x}}^{(t)}$ . According to (2), the three variables are updated with
$$
\begin{array}{l} \widetilde {\alpha} _ {t + 1} = \left(1 - \eta \widetilde {\beta} _ {t} ^ {2} + \eta \lambda^ {\star}\right) \widetilde {\alpha} _ {t}; \quad \widetilde {\gamma} _ {t + 1} = \left(1 - \eta \widetilde {\beta} _ {t} ^ {2}\right) \widetilde {\gamma} _ {t}; \\ \widetilde {\beta} _ {t} ^ {2} = \widetilde {\alpha} _ {t} ^ {2} + \widetilde {\gamma} _ {t} ^ {2}. \\ \end{array}
$$