SlowGuess/NeurIPS_2023_OCR huggingface dataset:
At  $t = 0$ , due to random initialization, the initial vector is nearly orthogonal to  $\pmb{u}^{\star}$ , and we have  $\widetilde{\alpha}_0 \approx \frac{1}{\sqrt{n}} \beta_0$  and  $\widetilde{\gamma}_0 \approx \widetilde{\beta}_0 = \beta_0$ . Also, due to the small initialization, the term  $\eta \widetilde{\beta}_t^2$  is ignorable until  $\widetilde{\beta}_t$  becomes sufficiently large, so  $\widetilde{\alpha}_t$  grows exponentially at the rate of  $1 + \eta \lambda^{\star}$ , while  $\widetilde{\gamma}_t$  remains still. Thus, in the early iterations where  $(1 + \eta \lambda^{\star})^{t}$  is still much less than  $\sqrt{n}$ ,  $\widetilde{\beta}_t$  is kept close to its initial
Figure 1: Evolution of the quantities  $\widetilde{\alpha}_t$ ,  $\widetilde{\beta}_t$ , and  $\widetilde{\gamma}_t$  simulated with  $\widetilde{\alpha}_0 = \frac{1}{\sqrt{n}}\beta_0$ ,  $\beta_0 = \frac{1}{n}$ ,  $\lambda^{\star} = 1$ , and  $n = 1000$ .
value  $\beta_0$  while the trajectory becomes more parallel to  $\pmb{u}^{\star}$  as  $\widetilde{\alpha}_t$  increases. When  $(1 + \eta \lambda^{\star})^{t}$  becomes much larger than  $\sqrt{n}$ , the trajectory becomes almost parallel to  $\pmb{u}^{\star}$  in that  $\widetilde{\beta}_t \approx \widetilde{\alpha}_t \gg \widetilde{\gamma}_t$ . Until  $\widetilde{\beta}_t$  (asymptotically) reaches  $\frac{\sqrt{\lambda^*}}{\sqrt{\log n}}$ , we can consider  $\widetilde{\alpha}_t$  as increasing at a rate of  $(1 + \eta \lambda^{\star})$ , and it takes about  $\frac{1}{\log(1 + \eta\lambda^{\star})} \log \frac{\sqrt{\lambda^n}}{\beta_0}$  steps to reach this point. After that, we can no longer ignore the term  $\eta \widetilde{\beta}_t^2$ , and  $\widetilde{\alpha}_t$  increases at a slower rate as  $\widetilde{\beta}_t$  increases. We can show that  $\widetilde{\beta}_t^2$  becomes sufficiently close to  $\lambda^{\star}$  within  $O(\log \log n)$  additional iterations, as stated in the following lemma.
Lemma 4.1. Let  $T_2'$  be the largest  $t$  such that  $\widetilde{\beta}_t^2 \leq \frac{\lambda^\star}{64\log n}$ . At  $t = T_2' + \frac{6\log\log n}{\log(1 + \eta\lambda^\star)}$ , we have  $\widetilde{\beta}_t^2 \geq \lambda^\star \left(1 - \frac{1}{\log n}\right)$ .
Finally, local convergence to  $\pmb{u}^{\star}$  occurs in that  $\widetilde{\alpha}_t$  approaches  $\lambda^{\star}$  and  $\widetilde{\gamma}_t$  decreases exponentially with the rate  $(1 - \eta \lambda^{\star})$ . The actual behavior of quantities  $\widetilde{\alpha}_t, \widetilde{\beta}_t, \widetilde{\gamma}_t$  are plotted in Figure 1.
We define the iterates before  $(1 + \eta \lambda^{\star})^{t}$  reaches  $\frac{1}{\sqrt{n p}}\sqrt{n}$ , within some logarithmic factors, as Phase I, and the next iterates before  $\widetilde{\beta}_{t}^{2}$  reaches  $\lambda^{\star}\left(1 - \frac{1}{\log n}\right)$  as Phase II. Different techniques are used for each phase to prove that  $\pmb{x}^{(t)}$  stays close to  $\widetilde{\pmb{x}}^{(t)}$ . At the end of Phase I,  $\widetilde{\alpha}_{t}$  is increased to  $\frac{1}{\sqrt{n p}}\beta_{0}$  from its initial scale  $\frac{1}{\sqrt{n}}\beta_{0}$ , but it is still not dominant over  $\beta_{0}$ . Therefore, the magnitudes of both  $\pmb{x}^{(t)}$  and  $\widetilde{\pmb{x}}^{(t)}$  are kept close to  $\beta_{0}$  throughout Phase I, and we take advantage of the small random initialization to show that the deviation of  $\pmb{x}^{(t)}$  from  $\widetilde{\pmb{x}}^{(t)}$  does not increase much, and is kept at  $\sqrt{\frac{1}{n p}}$  times the norms of  $\pmb{x}^{(t)}$ . In Phase II, we show that  $\pmb{x}^{(t)} - \widetilde{\pmb{x}}^{(t)}$  expands at a rate of at most  $(1 + \eta \lambda^{\star})$ . Since the norms of  $\pmb{x}^{(t)}$  also grows at a rate of  $(1 + \eta \lambda^{\star})$  during most of Phase II, the norms of  $\pmb{x}^{(t)} - \widetilde{\pmb{x}}^{(t)}$  remain negligible compared to those of  $\pmb{x}^{(t)}$ . The next two sections give the main lemmas of Phase I and II, respectively, which are used to prove Theorems 3.1 and 3.3. For a visual representation of the results in the following two sections, please refer to Figure 2.
5 Phase I: Finding Direction
We provide detailed results and proof ideas for Phase I. Our main goal is to analyze the deviation of  $\pmb{x}^{(t)}$  from  $\widetilde{\pmb{x}}^{(t)}$ . First, if we look at the update equations (1) and (2), the second term is proportional to the third power of  $\| \pmb{x}^{(t)}\|_2$ , while the other terms depend linearly on  $\| \pmb{x}^{(t)}\|_2$ . Thus, the second term is almost negligible due to the small initialization. Without the second terms, the difference between  $\pmb{x}^{(t)}$  and  $\widetilde{\pmb{x}}^{(t)}$  at  $t = 1$  is  $\eta(M^\circ - M^\star)\pmb{x}^{(0)}$ . From concentration inequalities, one can see that the  $\ell_2$  and  $\ell_\infty$  norms of  $\eta(M^\circ - M^\star)\pmb{x}^{(0)}$  are about  $\frac{1}{\sqrt{np}}$  times smaller than those of  $\widetilde{\pmb{x}}^{(1)}$ .
Due to the third terms of (1) and (2), the norms of  $\pmb{x}^{(t)} - \widetilde{\pmb{x}}^{(t)}$  can grow exponentially at a rate of  $(1 + \eta \lambda^{\star})$  in the worst case where  $\pmb{x}^{(t)} - \widetilde{\pmb{x}}^{(t)}$  is parallel to  $\pmb{u}^{\star}$ . In such a case, the norms of  $\pmb{x}^{(t)} - \widetilde{\pmb{x}}^{(t)}$  would be larger than those of  $\widetilde{\pmb{x}}^{(t)}$  at the end of Phase I, since those of  $\widetilde{\pmb{x}}^{(t)}$  remain still in Phase I. However, we overcome this problem by proving that the bounds grow at most polynomially with respect to  $t$ , and since  $t$  is at most  $O(\log n)$ , the bounds remain  $\frac{1}{\sqrt{np}}$  times smaller than the norms of  $\widetilde{\pmb{x}}^{(t)}$  up to logarithmic factors throughout Phase I.
Lemma 5.1. Let  $T_{1}$  be the largest  $t$  such that  $(1 + \eta \lambda^{\star})^{t} \leq \sqrt{\frac{\mu^{4} \log^{21} n}{np}} \sqrt{n}$ . Under the conditions of Theorem 3.1, with probability at least  $1 - o(1 / \sqrt{\log n})$ , for all  $t \leq T_{1}$ , we have
$$
\left\| \boldsymbol {x} ^ {(t)} - \widetilde {\boldsymbol {x}} ^ {(t)} \right\| _ {2} \lesssim \mu \sqrt {\frac {\log n}{n p}} \beta_ {0} t, \quad (1 3) \quad \left\| \boldsymbol {x} ^ {(t)} - \widetilde {\boldsymbol {x}} ^ {(t)} \right\| _ {\infty} \lesssim \sqrt {\frac {\mu^ {3} \log^ {2} n}{n p}} \frac {\beta_ {0}}{\sqrt {n}} t ^ {2}. \tag {14}
$$
$T_{1}$  is defined to be the end of Phase I. Lemma 5.1 proves Theorem 3.3 for Phase I.
Proof of (13) We will first demonstrate how to obtain the  $\ell_2$ -norm bound of Lemma 5.1. Let us define a sequence  $\widehat{\pmb{x}}^{(t)}$  that is updated as
$$
\widehat {\boldsymbol {x}} ^ {(t + 1)} = \widehat {\boldsymbol {x}} ^ {(t)} - \eta \left\| \widetilde {\boldsymbol {x}} ^ {(t)} \right\| _ {2} ^ {2} \widehat {\boldsymbol {x}} ^ {(t)} + \eta M ^ {\circ} \widehat {\boldsymbol {x}} ^ {(t)}; \quad \widehat {\boldsymbol {x}} ^ {(0)} = \boldsymbol {x} ^ {(0)}. \tag {15}
$$
(c)
Figure 2: An illustrative description of trajectory of various quantities compared to the norms of  $\pmb{x}^{(t)}$  on logarithmic scales. Arrows between lines represent the ratio between them. The quantities depicted are not precise, and only the key factors are shown for simplicity. (a) In Phase I,  $\| \widehat{\pmb{x}}^{(t)} - \widetilde{\pmb{x}}^{(t)} \|_2$  increases linearly and  $\| \pmb{x}^{(t)} - \widehat{\pmb{x}}^{(t)} \|_2$  increases exponentially with the rate of  $(1 + \eta \lambda^{\star})$ . They have the same scale at the end of Phase I. (b) In Phase I, even if the  $\pmb{u}^{\star}$  component of  $\pmb{x}^{(t)} - \pmb{x}^{(t,l)}$  grows exponentially, it remains almost orthogonal to  $\pmb{u}^{\star}$  throughout the phase. (c) Phase II is divided into three parts according to the growth speed of  $\pmb{x}^{(t)}$  and  $\pmb{x}^{(t)} - \widetilde{\pmb{x}}^{(t)}$ . The ratio between them at the start and the end of each part is described, and it is at most  $\frac{1}{\sqrt{\log n}}$  in Phase II.
Note that the norm of  $\widetilde{\pmb{x}}^{(t)}$  is used in the second term of (15). The update equation of  $\widehat{\pmb{x}}^{(t)}$  differs from  $\widetilde{\pmb{x}}^{(t)}$  in the third term and from  $\pmb{x}^{(t)}$  in the second term. We use  $\widehat{\pmb{x}}^{(t)}$  as a proxy for bounding  $\left\| \pmb{x}^{(t)} - \widetilde{\pmb{x}}^{(t)}\right\|_2$ . We first show that  $\left\| \widehat{\pmb{x}}^{(t)} - \widetilde{\pmb{x}}^{(t)}\right\|_2$  grows at most linearly with respect to  $t$ .
Lemma 5.2. With probability at least  $1 - o(1 / \sqrt{\log n})$ , for all  $t \leq T_1$ , we have
$$
\left\| \widehat {\boldsymbol {x}} ^ {(t)} - \widetilde {\boldsymbol {x}} ^ {(t)} \right\| _ {2} \lesssim \mu \sqrt {\frac {\log n}{n p}} \beta_ {0} t. \tag {16}
$$
The proof of this lemma is based on the fact that  $\widehat{\pmb{x}}^{(t)}$  is a product of  $\pmb{x}^{(0)}$  and a matrix polynomial of  $\pmb{I}$  and  $M^{\circ}$ , while  $\widetilde{\pmb{x}}^{(t)}$  is a product between  $\pmb{x}^{(0)}$  and a matrix polynomial of  $\pmb{I}$  and  $M^{\star}$ . We prove the lemma by comparing the two matrix polynomials. We remark that Lemma 5.2 holds regardless of the small initialization, but it relies on the randomness of  $\pmb{x}^{(0)}$ .
Since  $\pmb{x}^{(t)}$  and  $\widehat{\pmb{x}}^{(t)}$  differ only in the second term, their initial difference is proportional to  $\beta_0^3$ . More precisely, it is  $\frac{1}{\sqrt{np}}\beta_0^3$ . We show that the difference grows exponentially at a rate of  $(1 + \eta \lambda^{\star})$ .
Lemma 5.3. If (14) holds for all  $t \leq T_1$ , we have
$$
\left\| \boldsymbol {x} ^ {(t)} - \widehat {\boldsymbol {x}} ^ {(t)} \right\| _ {2} \lesssim \frac {1}{\lambda^ {\star}} \sqrt {\frac {\mu^ {3} \log^ {3} n}{n p}} (1 + \eta \lambda^ {\star}) ^ {t} \beta_ {0} ^ {3} \tag {17}
$$
for all  $t \leq T_1$  with probability at least  $1 - o(1 / \sqrt{\log n})$ .
The upper bound in (17) becomes smaller than that of (16) if  $(1 + \eta \lambda^{\star})^{t}\beta_{0}^{2}\leq \lambda^{\star}\sqrt{\frac{1}{\mu\log^{2}n}}$ . One can check that this condition is satisfied from the definition of  $T_{1}$  given in Lemma 5.1 and the bound on the initialization size (8). Thus, (13) is proved by (16) and (17).
Proof of (14) We control the  $l$ th component of  $\pmb{x}^{(t)} - \widetilde{\pmb{x}}^{(t)}$  using the  $l$ th leave-one-out sequence. Leave-one-out sequences have two important properties. First, because they are defined without only one row/column, they are extremely close to  $\pmb{x}^{(t)}$ , and at  $t = 1$ ,  $\left\| \pmb{x}^{(t)} - \pmb{x}^{(t,l)} \right\|_2$  is about  $\frac{1}{\sqrt{n p}} \frac{\beta_0}{\sqrt{n}}$ . Second, the  $l$ th component of the  $l$ th leave-one-out sequence evolves similarly to that of  $\widetilde{\pmb{x}}^{(t)}$  and is easy to analyze. With these two properties, we bound the  $l$ th component of  $\pmb{x}^{(t)} - \widetilde{\pmb{x}}^{(t)}$  as
$$
\left| \left(\boldsymbol {x} ^ {(t)} - \widetilde {\boldsymbol {x}} ^ {(t)}\right) _ {l} \right| \leq \left\| \boldsymbol {x} ^ {(t)} - \boldsymbol {x} ^ {(t, l)} \right\| _ {2} + \left| \left(\boldsymbol {x} ^ {(t, l)} - \widetilde {\boldsymbol {x}} ^ {(t)}\right) _ {l} \right|. \tag {18}
$$
We claim that both  $\left\| \boldsymbol{x}^{(t)} - \boldsymbol{x}^{(t,l)} \right\|_2$  and  $\left| (\boldsymbol{x}^{(t,l)} - \widetilde{\boldsymbol{x}}^{(t)})_l \right|$  increase at most polynomially with respect to  $t$  from the initial scale  $\frac{1}{\sqrt{np}} \frac{\beta_0}{\sqrt{n}}$ .
Lemma 5.4. With probability at least  $1 - o(1 / \sqrt{\log n})$ , for all  $t \leq T_1$ , we have
$$
\left\| \boldsymbol {x} ^ {(t)} - \boldsymbol {x} ^ {(t, l)} \right\| _ {2} \lesssim \mu \sqrt {\frac {\log^ {2} n}{n p}} \frac {\beta_ {0}}{\sqrt {n}} t, \quad (1 9) \quad \left| \left(\boldsymbol {x} ^ {(t, l)} - \widetilde {\boldsymbol {x}} ^ {(t)}\right) _ {l} \right| \lesssim \sqrt {\frac {\mu^ {3} \log^ {2} n}{n p}} \frac {\beta_ {0}}{\sqrt {n}} t ^ {2}. \tag {20}
$$
As explained for  $\pmb{x}^{(t)} - \widetilde{\pmb{x}}^{(t)}$ , due to the third terms of (1) and (3),  $\pmb{x}^{(t)} - \pmb{x}^{(t,l)}$  can also grow exponentially at the rate of  $(1 + \eta \lambda^{\star})$  in the worst case where  $\pmb{x}^{(t)} - \pmb{x}^{(t,l)}$  is parallel to  $\pmb{u}^{\star}$ . This contradicts our result (19) that  $\left\| \pmb{x}^{(t)} - \pmb{x}^{(t,l)} \right\|_2$  grows only linearly. We show that  $\pmb{x}^{(t)} - \pmb{x}^{(t,l)}$  remains nearly orthogonal to  $\pmb{u}^{\star}$  in Phase I, and thus the worst case does not occur.
Lemma 5.5. For all  $l \in [n]$  and  $t \leq T_1$ , we have
$$
\left| \boldsymbol {u} ^ {(l) \top} (\boldsymbol {x} ^ {(t)} - \boldsymbol {x} ^ {(t, l)}) \right| \lesssim \sqrt {\frac {\mu^ {3} \log^ {2} n}{n p}} (1 + \eta \lambda^ {\star}) ^ {t} \frac {\beta_ {0}}{n}
$$
with probability at least  $1 - o(1 / \sqrt{\log n})$ , where  $\pmb{u}^{(l)}$  is the first eigenvector of  $\pmb{M}^{(l)}$ .
Note that  $\pmb{u}^{(l)}$  is almost parallel to  $\pmb{u}^{\star}$  (see Lemma A.5 in the appendix). The  $\pmb{u}^{(l)}$  component of  $\pmb{x}^{(t)} - \pmb{x}^{(t,l)}$  is initialized to the order of  $\frac{1}{\sqrt{n p}}\frac{\beta_0}{n}$ , which is  $\frac{1}{\sqrt{n}}$  times smaller than  $\left\| \pmb{x}^{(t)} - \pmb{x}^{(t,l)}\right\|_2$ . Although it is increased exponentially, from the definition of  $T_{1}$ , the  $\pmb{u}^{(l)}$  component remains much smaller than  $\left\| \pmb{x}^{(t)} - \pmb{x}^{(t,l)}\right\|_2$  in Phase I.
One can see that  $\left|\left(\boldsymbol{x}^{(t,l)} - \widetilde{\boldsymbol{x}}^{(t)}\right)_l\right|$  increases by  $\left\| \boldsymbol{x}^{(t)} - \widetilde{\boldsymbol{x}}^{(t)}\right\|_2\left\| \boldsymbol{u}^\star \right\|_\infty$  at each step, and summing the bound (13) up to  $t$  gives (20). Finally, (14) is obtained by putting (19) and (20) into (18).
6 Phase II: Expansion
In the next phase, we show that the bounds obtained in Phase I are increased at a rate of  $(1 + \eta \lambda^{\star})$ .
Lemma 6.1. Let  $T_{2}$  be the largest  $t$  such that  $\widetilde{\beta}_{t}^{2} \leq \lambda^{\star}\left(1 - \frac{1}{\log n}\right)$ . Then, for all  $T_{1} < t \leq T_{2}$ , we have
$$
\left\| \boldsymbol {x} ^ {(t)} - \widetilde {\boldsymbol {x}} ^ {(t)} \right\| _ {2} \lesssim \mu \sqrt {\frac {\log^ {3} n}{n p}} \beta_ {0} (1 + \eta \lambda^ {\star}) ^ {t - T _ {1}}, \tag {21}
$$
$$
\left\| \boldsymbol {x} ^ {(t)} - \boldsymbol {x} ^ {(t, l)} \right\| _ {2} \lesssim \mu \sqrt {\frac {\log^ {5} n}{n p}} \frac {\beta_ {0}}{\sqrt {n}} (1 + \eta \lambda^ {\star}) ^ {t - T _ {1}}, \tag {22}
$$
$$
\left\| \boldsymbol {x} ^ {(t)} - \widetilde {\boldsymbol {x}} ^ {(t)} \right\| _ {\infty} \lesssim \sqrt {\frac {\mu^ {3} \log^ {8} n}{n p}} \frac {\beta_ {0}}{\sqrt {n}} (1 + \eta \lambda^ {\star}) ^ {t - T _ {1}}, \tag {23}
$$
$$
\left| \left(\boldsymbol {x} ^ {(t, l)} - \widetilde {\boldsymbol {x}} ^ {(t)}\right) _ {l} \right| \lesssim \sqrt {\frac {\mu^ {3} \log^ {8} n}{n p}} \frac {\beta_ {0}}{\sqrt {n}} (1 + \eta \lambda^ {\star}) ^ {t - T _ {1}}, \tag {24}
$$
$T_{2}$  is defined as the end of Phase II. We will explain how Lemma 6.1 leads to Theorem 3.3 in Phase II. Let us first focus on (21) and (11). We can divide Phase II into three parts according to the behavior of  $\left\| \widetilde{\boldsymbol{x}}^{(t)}\right\| _2$ . First,  $\left\| \widetilde{\boldsymbol{x}}^{(t)}\right\| _2$  is kept close to  $\beta_0$  until  $(1 + \eta \lambda^{\star})^{t}$  becomes  $\sqrt{n}$ , or  $(1 + \eta \lambda^{\star})^{t - T_1}$  becomes  $\sqrt{np}$ . In this part, although the bounds increase exponentially with the rate of  $(1 + \eta \lambda^{\star})$ , the factor  $\frac{1}{\sqrt{np}}$ , which was already present in (13) of Phase I, compensates for this increase. At the end of the first part,  $\left\| \boldsymbol{x}^{(t)} - \widetilde{\boldsymbol{x}}^{(t)}\right\| _2$  is smaller than  $\left\| \widetilde{\boldsymbol{x}}^{(t)}\right\| _2$  by some log factors. Next,  $\left\| \widetilde{\boldsymbol{x}}^{(t)}\right\| _2$  grows at the rate of  $(1 + \eta \lambda^{\star})$  until it reaches  $\frac{\sqrt{\lambda^{\star}}}{8\sqrt{\log n}}$ . Since both  $\left\| \boldsymbol{x}^{(t)} - \widetilde{\boldsymbol{x}}^{(t)}\right\| _2$  and  $\left\| \widetilde{\boldsymbol{x}}^{(t)}\right\| _2$  increase with  $(1 + \eta \lambda^{\star})$ ,
(b)
Figure 3: (a) Evolution of the quantities  $|\pmb{u}^{\star \top}\pmb{x}^{(t)}|$  and  $\| \pmb{x}^{(t)}\|_2$ , which behave similarly to  $\widetilde{\alpha}_t$  and  $\widetilde{\beta}_t$ , respectively, and  $\| \pmb{x}^{(t)} \pm \pmb{x}^\star \|_2$ , which shows local convergence. (b) Comparison between the norms of  $\pmb{x}^{(t)}$  and  $\pmb{x}^{(t)} - \widetilde{\pmb{x}}^{(t)}$ . (c) Convergence of GD with respect to the initialization size and sampling probability.  $\| \pmb{x}^{(t)} \pm \pmb{x}^\star \|_2$  was measured at  $t = \frac{1}{\log(1 + \eta\lambda^\star)}\log \frac{\sqrt{\lambda^\star n}}{\beta_0} + 100$  and averaged over 1000 trials.
the ratio between them is maintained in the second part. Finally, in the remaining iterations,  $\left\| \widetilde{\boldsymbol{x}}^{(t)}\right\|_2$  increases with  $(1 - \eta \widetilde{\beta}_t^2 + \eta \lambda^\star)$  at each step, and the increment becomes smaller as it converges to  $\sqrt{\lambda^\star}$ . Thus, as in the first part,  $\left\| \boldsymbol{x}^{(t)} - \widetilde{\boldsymbol{x}}^{(t)}\right\|_2$  increases faster than  $\left\| \widetilde{\boldsymbol{x}}^{(t)}\right\|_2$ . However, from Lemma 4.1, the length of this part is  $O(\log \log n)$ , and the ratio between  $\left\| \boldsymbol{x}^{(t)} - \widetilde{\boldsymbol{x}}^{(t)}\right\|_2$  and  $\left\| \widetilde{\boldsymbol{x}}^{(t)}\right\|_2$  increases only by  $\log^6 n$ . We prove that the log factors already present at the end of the second part compensate this, and finally (11) holds for all  $t$  in Phase II. A more delicate analysis may prove that  $\left\| \boldsymbol{x}^{(t)} - \widetilde{\boldsymbol{x}}^{(t)}\right\|_2$  grows at the same rate as  $\left\| \widetilde{\boldsymbol{x}}^{(t)}\right\|_2$  in the third part, and this will reduce the required sample complexity by at most  $\log^{12}n$ . A similar argument can be used to prove that the bounds for  $\left\| \boldsymbol{x}^{(t)} - \widetilde{\boldsymbol{x}}^{(t)}\right\|_\infty$ ,  $\left\| \boldsymbol{x}^{(t)} - \boldsymbol{x}^{(t,l)}\right\|_2$ , and  $\left|\left(\boldsymbol{x}^{(t,l)} - \widetilde{\boldsymbol{x}}^{(t)}\right)_l\right|$  are smaller than  $\left\| \widetilde{\boldsymbol{x}}^{(t)}\right\|_\infty$  by some log factors throughout Phase II.
At the end of Phase II,  $\widetilde{\pmb{x}}^{(t)}$  is very close to  $\pm \pmb{x}^{\star}$  in both  $\ell_2$  and  $\ell_{\infty}$  norms (see Corollary C.3 in the appendix), so one can replace  $\widetilde{\pmb{x}}^{(t)}$  of Lemma 6.1 with  $\pm \pmb{x}^{\star}$  to prove (4) to (7) of Theorem 3.1. Hence, we can let  $T^{\star} = T_{2}$ , and as explained in Section 4,  $T_{2}$  is approximately given by  $\frac{1}{\log(1 + \eta\lambda^{\star})}\log \frac{\sqrt{\lambda^{\star}n}}{\beta_0} +O(\log \log n)$ .
7 Simulation
In this section, we present some simulation results that support our theoretical findings.
Trajectory of GD With the dimension  $n = 5000$ , we constructed the ground truth vector  $\pmb{u}^{\star}$  by sampling it from the Gaussian distribution  $\mathcal{N}(0, \frac{1}{n}\pmb{I})$  and normalizing it to have unit norm. We let  $\lambda^{\star} = 1$  so that the matrix  $M^{\star}$  is given by  $\pmb{u}^{\star}\pmb{u}^{\star \top}$ , and we randomly sampled the matrix symmetrically with a sampling rate of  $p = 0.1$  and Gaussian noise of  $\sigma = \frac{0.1}{n}$ . The initialization size was set to  $\beta_0 = \frac{1}{n}$  and a step size of 0.1 was used for GD. Figure 3 (a) and (b) represent one trial of the experiment, but similar graphs were obtained in each repetition of the experiment. The evolution of some important quantities such as  $\left\| \pmb{x}^{(t)}\right\|_{2}$  and  $\left|\pmb{u}^{\star \top}\pmb{x}^{(t)}\right|$  is shown in Figure 3(a). As in the fully observed case, the signal component  $\left|\pmb{u}^{\star \top}\pmb{x}^{(t)}\right|$  increases at the rate of  $(1 + \eta \lambda^{\star})$  until it approaches  $\sqrt{\lambda^{\star}}$ , and a local convergence to  $\pmb{x}^{\star}$  occurs, where  $\left\| \pmb{x}^{(t)} - \pmb{x}^{\star}\right\|_{2}$  decreases exponentially and saturates at the level determined by the noise size  $\sigma$ . In Figure 3(b), we describe the deviation of  $\pmb{x}^{(t)}$  from  $\widetilde{\pmb{x}}^{(t)}$  in both  $\ell_{2}$  and  $\ell_{\infty}$  norms. The solid lines represent the norms of  $\pmb{x}^{(t)}$  and the dotted lines represent those of  $\pmb{x}^{(t)} - \widetilde{\pmb{x}}^{(t)}$ . We can see that there is a gap between the solid and the dotted lines during the whole iterations. Thus,  $\pmb{x}^{(t)}$  stays close to the trajectory of the fully observed case, as we proved in Theorem 3.3.
Small Initialization In the next experiment, we investigated the importance of a small initialization for the convergence of GD. We used the same conditions as in the previous experiment except
$n = 500$ . We measured  $\left\| \pmb{x}^{(t)} \pm \pmb{x}^{\star}\right\|_{2}$  at  $t = \frac{1}{\log(1 + \eta^{\lambda^{\star}})} \log \frac{\sqrt{\lambda^{\star} n}}{\beta_{0}} + 100$  and averaged it over 1000 trials. We repeated the experiment while changing the initialization size from  $10^{0}$  to  $10^{-9}$  and the sampling probability from 0.01 to 0.04. The result is summarized in Figure 3(c). For all sampling probabilities, the small initialization improves the convergence of GD. Also, the performance starts to saturate at much larger initialization sizes as the sampling probability increases, and this is consistent with our finding (8) that a larger initialization is possible as more samples are available.
In this paper, we showed that for rank-1 symmetric matrix completion with  $\ell_2$  loss, GD can converge to the ground truth starting from a small random initialization. Ignoring log factors, the bound on the initialization size is  $n^{-\frac{1}{4}}$  when the optimal  $n$  poly  $(\log n)$  samples are provided, and the bound becomes larger as more samples are provided. The result is interesting because the loss function does not have global benign geometry if no regularizer is applied. Our result does not use any explicit regularizer and relies only on the implicit regularizing effect of GD.
The most important future work is an extension to the rank-  $r$  case. Suppose that  $M^{\star}$  is a rank-  $r$  matrix and its eigendecomposition is given by  $\boldsymbol{U}^{\star}\boldsymbol{\Sigma}^{\star}\boldsymbol{U}^{\star\top} = \boldsymbol{X}^{\star}\boldsymbol{X}^{\star\top}$ , where  $\boldsymbol{\Sigma}^{\star} = \mathrm{diag}(\lambda_1^{\star}, \dots, \lambda_r^{\star})$  and  $\boldsymbol{X}^{\star} = \boldsymbol{U}^{\star}\boldsymbol{\Sigma}^{\star\frac{1}{2}}$ . Then, the trajectory of GD becomes an  $n \times r$  matrix  $\boldsymbol{X}^{(t)}$ , which is updated as
$$
\boldsymbol {X} ^ {(t + 1)} = \boldsymbol {X} ^ {(t)} - \frac {\eta}{p} \mathcal {P} _ {\Omega} \left(\boldsymbol {X} ^ {(t)} \boldsymbol {X} ^ {(t) \top}\right) \boldsymbol {X} ^ {(t)} + \eta \boldsymbol {M} ^ {\circ} \boldsymbol {X} ^ {(t)}.
$$
Figure 4: Trajectory of GD obtained for a rank-3 matrix with non-zero eigenvalues 1,0.75,0.5. The same parameters were used as in Figure 3.
Each entry of  $\mathbf{X}^{(0)}$  is sampled independently from the Gaussian distribution  $\mathcal{N}\left(0,\frac{1}{n}\beta_0^2\right)$  as in the rank-1 case.