SlowGuess/NeurIPS_2023_OCR huggingface dataset:
(E.1) at  $(t + 1)$  We decompose  $x^{(t + 1)} - \widetilde{x}^{(t + 1)}$  as
$$
\begin{array}{l} \pmb {x} ^ {(t + 1)} - \widetilde {\pmb {x}} ^ {(t + 1)} \\ = (\boldsymbol {x} - \eta \nabla f (\boldsymbol {x})) - (\widetilde {\boldsymbol {x}} - \eta \nabla F (\widetilde {\boldsymbol {x}})) \\ = (\boldsymbol {x} - \eta \nabla g (\boldsymbol {x})) - (\widetilde {\boldsymbol {x}} - \eta \nabla G (\widetilde {\boldsymbol {x}})) + \eta \left(M ^ {\circ} \boldsymbol {x} - M ^ {\star} \widetilde {\boldsymbol {x}}\right) \\ = (\boldsymbol {x} - \eta \nabla g (\boldsymbol {x})) - (\widetilde {\boldsymbol {x}} - \eta \nabla g (\widetilde {\boldsymbol {x}})) - \eta (\nabla g (\widetilde {\boldsymbol {x}}) - \nabla G (\widetilde {\boldsymbol {x}})) + \eta \boldsymbol {M} ^ {\star} (\boldsymbol {x} - \widetilde {\boldsymbol {x}}) + \eta (\boldsymbol {M} ^ {\circ} - \boldsymbol {M} ^ {\star}) \boldsymbol {x} \\ = \int_ {0} ^ {1} (\boldsymbol {I} - \eta \nabla^ {2} g (\boldsymbol {x} (\tau)) (\boldsymbol {x} - \widetilde {\boldsymbol {x}}) \mathrm {d} \tau - \eta \| \widetilde {\boldsymbol {x}} \| _ {2} ^ {2} (\boldsymbol {I} _ {\widetilde {\boldsymbol {x}}} - \boldsymbol {I}) \widetilde {\boldsymbol {x}} + \eta \boldsymbol {M} ^ {\star} (\boldsymbol {x} - \widetilde {\boldsymbol {x}}) + \eta (\boldsymbol {M} ^ {\circ} - \boldsymbol {M} ^ {\star}) \boldsymbol {x} \\ = \underbrace {\left((1 - \eta \| \widetilde {\boldsymbol {x}} \| _ {2} ^ {2}) \boldsymbol {I} - 2 \eta \widetilde {\boldsymbol {x}} \widetilde {\boldsymbol {x}} ^ {\top} + \eta \boldsymbol {M} ^ {\star}\right) (\boldsymbol {x} - \widetilde {\boldsymbol {x}})} _ {①} - \eta \underbrace {\int_ {0} ^ {1} \left(\nabla^ {2} g (\boldsymbol {x} (\tau)) - \left(\| \widetilde {\boldsymbol {x}} \| _ {2} ^ {2} \boldsymbol {I} + 2 \widetilde {\boldsymbol {x}} \widetilde {\boldsymbol {x}} ^ {\top}\right)\right) (\boldsymbol {x} - \widetilde {\boldsymbol {x}}) \mathrm {d} \tau} _ {②} \\ - \eta \underbrace {\left\| \widetilde {\boldsymbol {x}} \right\| _ {2} ^ {2} \left(I _ {\widetilde {\boldsymbol {x}}} - I\right) \widetilde {\boldsymbol {x}}} _ {\text {③}} + \eta \underbrace {(M ^ {\circ} - M ^ {\star}) \boldsymbol {x}} _ {\text {④}}, \\ \end{array}
$$
where  $\pmb {x}(\tau) = \widetilde{\pmb{x}} +\tau (\pmb {x} - \widetilde{\pmb{x}})$ . For the term ①, we require a bound on
$$
\left\| (1 - \eta \| \widetilde {\boldsymbol {x}} \| _ {2} ^ {2}) \boldsymbol {I} - 2 \eta \widetilde {\boldsymbol {x}} \widetilde {\boldsymbol {x}} ^ {\top} + \eta \boldsymbol {M} ^ {\star} \right\|.
$$
If we write  $\widetilde{\pmb{x}}$  as  $\widetilde{\alpha}_t\pmb{u}^\star +\widetilde{\pmb{x}}_\perp$ , we have  $\widetilde{\alpha}_t^2\leq \lambda^\star$  and  $\| \widetilde{\pmb{x}}_{\perp}\|_{2}\lesssim \beta_{0}$ . Then, we have
$$
\begin{array}{l} \left\| \left(1 - \eta \| \widetilde {\boldsymbol {x}} \| _ {2} ^ {2}\right) \boldsymbol {I} - 2 \eta \widetilde {\boldsymbol {x}} \widetilde {\boldsymbol {x}} ^ {\top} + \eta \boldsymbol {M} ^ {\star} \right\| \\ = \left\| \left(1 - \eta \| \widetilde {\boldsymbol {x}} \| _ {2} ^ {2}\right) \boldsymbol {I} + \eta \left(\lambda^ {\star} - 2 \widetilde {\alpha} _ {t} ^ {2}\right) \boldsymbol {u} ^ {\star} \boldsymbol {u} ^ {\star \top} - 2 \eta \left(\widetilde {\boldsymbol {x}} \widetilde {\boldsymbol {x}} ^ {\top} - \widetilde {\alpha} _ {t} ^ {2} \boldsymbol {u} ^ {\star} \boldsymbol {u} ^ {\star \top}\right) \right\| \\ \leq \left\| (1 - \eta \| \widetilde {\boldsymbol {x}} \| _ {2} ^ {2}) \boldsymbol {I} + \eta (\lambda^ {\star} - 2 \widetilde {\alpha} _ {t} ^ {2}) \boldsymbol {u} ^ {\star} \boldsymbol {u} ^ {\star \top} \right\| + 2 \eta \left\| \widetilde {\boldsymbol {x}} \widetilde {\boldsymbol {x}} ^ {\top} - \widetilde {\alpha} _ {t} ^ {2} \boldsymbol {u} ^ {\star} \boldsymbol {u} ^ {\star \top} \right\| \\ \leq \left(1 + \eta \lambda^ {\star}\right) + 2 \eta \left(2 \alpha_ {t} \| \widetilde {\boldsymbol {x}} _ {\perp} \| _ {2} + \| \widetilde {\boldsymbol {x}} _ {\perp} \| _ {2} ^ {2}\right) \\ \leq 1 + \eta \lambda^ {\star} + \frac {c _ {8}}{\log^ {2} n} \\ \end{array}
$$
for some universal constant  $c_8 > 0$ . This implies the desired bound
$$
\left\| \text {①} \right\| _ {2} \leq \left(1 + \eta \lambda^ {\star} + \frac {c _ {8}}{\log^ {2} n}\right) \| \boldsymbol {x} - \widetilde {\boldsymbol {x}} \| _ {2}.
$$
For all  $0 \leq \tau \leq 1$ , we have  $\| \pmb{x}(\tau) - \widetilde{\pmb{x}} \|_{\infty} \leq \| \pmb{x} - \widetilde{\pmb{x}} \|_{\infty}$ , and the induction hypothesis (E.2) gives
$$
\| \boldsymbol {x} - \widetilde {\boldsymbol {x}} \| _ {\infty} \lesssim \sqrt {\frac {1}{\mu \log^ {1 3} n}} (1 + \eta \lambda^ {\star}) ^ {T _ {2}} \frac {\beta_ {0}}{n}.
$$
Hence, by Lemma G.10, we have
$$
\begin{array}{l} \left\| \nabla^ {2} g (\boldsymbol {x} (\tau)) - \left(\| \widetilde {\boldsymbol {x}} \| _ {2} ^ {2} \boldsymbol {I} + 2 \widetilde {\boldsymbol {x}} \widetilde {\boldsymbol {x}} ^ {\top}\right) \right\| \lesssim n \| \boldsymbol {x} - \widetilde {\boldsymbol {x}} \| _ {\infty} (\| \boldsymbol {x} \| _ {\infty} + \| \widetilde {\boldsymbol {x}} \| _ {\infty}) + \sqrt {\frac {n \log n}{p}} \| \widetilde {\boldsymbol {x}} \| _ {\infty} ^ {2} \\ \lesssim \left(\sqrt {\frac {1}{\log^ {1 2} n}} + \sqrt {\frac {\mu^ {2} \log^ {3} n}{n p}}\right) (1 + \eta \lambda^ {\star}) ^ {2 T _ {2}} \frac {\beta_ {0} ^ {2}}{n} \\ \lesssim \lambda^ {\star} \sqrt {\frac {1}{\log^ {1 2} n}} \\ \end{array}
$$
if  $n^2 p \gtrsim \mu^2 n\log^{15}n$  because  $\| \widetilde{\boldsymbol{x}} \|_{\infty} \lesssim \sqrt{\mu\log n}(1 + \eta\lambda^{\star})^{T_2}\frac{\beta_0}{n}$  and  $(1 + \eta\lambda^{\star})^{T_2} \lesssim \sqrt{\lambda^{\star}}\frac{\sqrt{n}}{\beta_0}$ . This gives
$$
\left\| \text {②} \right\| _ {2} \lesssim \lambda^ {\star} \sqrt {\frac {1}{\log^ {1 2} n}} \| \boldsymbol {x} - \widetilde {\boldsymbol {x}} \| _ {2}. \tag {E.6}
$$
For the term ③, we use Lemma G.8 to obtain
$$
\| \textcircled {③} \| _ {2} \lesssim \lambda^ {\star} \sqrt {\frac {\mu \log n}{n p}} \| \widetilde {\boldsymbol {x}} \| _ {2}.
$$
Lastly, the term ④ is bounded with
$$
\left\| ④ \right\| _ {2} \lesssim \| M ^ {\circ} - M ^ {\star} \| \| \boldsymbol {x} \| _ {2} \lesssim \lambda^ {\star} \mu \sqrt {\frac {\log n}{n p}} \| \widetilde {\boldsymbol {x}} \| _ {2}.
$$
Combining all, there exists a universal constant  $c_9 > 0$  such that
$$
\left\| \text {(1)} \right\| _ {2} + \left\| \text {(2)} \right\| _ {2} \leq \left(1 + \eta \lambda^ {\star} + \frac {c _ {9}}{\log^ {2} n}\right) \| \boldsymbol {x} - \widetilde {\boldsymbol {x}} \| _ {2},
$$
$$
\eta \left(\| \text {③} \| _ {2} + \| \text {④} \| _ {2}\right) \leq c _ {9} \mu \sqrt {\frac {\log n}{n p}} \| \widetilde {\boldsymbol {x}} \| _ {2}.
$$
Because  $\left\| \pmb{x}^{(T_1)}\right\| _2\lesssim \sqrt{\log n}\beta_0$  by (C.3) and  $\left\| \widetilde{\pmb{x}}^{(t)}\right\| _2$  can grow at a rate at most  $(1 + \eta \lambda^{\star})$ , there exists a universal constant  $c_{10} > 0$  such that
$$
c _ {8} \mu \sqrt {\frac {\log n}{n p}} \left\| \widetilde {\boldsymbol {x}} ^ {(t)} \right\| _ {2} \leq c _ {1 0} \mu \sqrt {\frac {\log^ {2} n}{n p}} (1 + \eta \lambda^ {\star}) ^ {t - T _ {1}} \beta_ {0}. \tag {E.7}
$$
Hence, for all  $T_{1} \leq s \leq t$ , we have
$$
\left\| \boldsymbol {x} ^ {(s + 1)} - \widetilde {\boldsymbol {x}} ^ {(s + 1)} \right\| _ {2} \leq \left(1 + \eta \lambda^ {\star} + \frac {c _ {9}}{\log^ {2} n}\right) \left\| \boldsymbol {x} ^ {(s)} - \widetilde {\boldsymbol {x}} ^ {(s)} \right\| _ {2} + c _ {1 0} \mu \sqrt {\frac {\log^ {2} n}{n p}} (1 + \eta \lambda^ {\star}) ^ {t - T _ {1}} \beta_ {0}.
$$
$$
x _ {s + 1} = \left(1 + \eta \lambda^ {\star} + \frac {c _ {9}}{\log^ {2} n}\right) x _ {s} + c _ {1 0} \mu \sqrt {\frac {\log^ {2} n}{n p}} (1 + \eta \lambda^ {\star}) ^ {t - T _ {1}} \beta_ {0}, \quad x _ {T _ {1}} = c _ {7} \mu \sqrt {\frac {\log^ {3} n}{n p}} \beta_ {0}
$$
$$
\left\| \boldsymbol {x} ^ {(t + 1)} - \widetilde {\boldsymbol {x}} ^ {(t + 1)} \right\| _ {2} \leq 2 c _ {7} \mu \sqrt {\frac {\log^ {3} n}{n p}} \beta_ {0} (1 + \eta \lambda^ {\star}) ^ {t + 1 - T _ {1}}.
$$
(E.3) at  $(t + 1)$  Similar to the proof of (D.8), we have the decomposition
$$
\begin{array}{l} \boldsymbol {x} ^ {(t + 1)} - \boldsymbol {x} ^ {(t + 1, l)} = \underbrace {(1 - \eta \| \widetilde {\boldsymbol {x}} \| _ {2} ^ {2} - 2 \eta \widetilde {\boldsymbol {x}} \widetilde {\boldsymbol {x}} ^ {\top}) (\boldsymbol {x} - \boldsymbol {x} ^ {(l)})} _ {①} \\ - \eta \underbrace {\int_ {0} ^ {1} \left(\nabla^ {2} g (\boldsymbol {x} ^ {(l)} (\tau)) - \left(\| \widetilde {\boldsymbol {x}} \| _ {2} ^ {2} \boldsymbol {I} + 2 \widetilde {\boldsymbol {x}} \widetilde {\boldsymbol {x}} ^ {\top}\right)\right) (\boldsymbol {x} - \boldsymbol {x} ^ {(l)}) \mathrm {d} \tau} _ {②} \\ - \eta \underbrace {\left(\frac {1}{p} \mathcal {P} _ {\Omega_ {l}} \left(\boldsymbol {x} ^ {(l)} \boldsymbol {x} ^ {(l) \top}\right) - \mathcal {P} _ {l} \left(\boldsymbol {x} ^ {(l)} \boldsymbol {x} ^ {(l) \top}\right)\right) \boldsymbol {x} ^ {(l)}} _ {③} \\ + \eta M ^ {\star} (\boldsymbol {x} - \boldsymbol {x} ^ {(l)}) + \eta \underbrace {(M ^ {\circ} - M ^ {\star}) (\boldsymbol {x} - \boldsymbol {x} ^ {(l)})} _ {④} \\ + \eta \underbrace {\left(\frac {1}{p} \mathcal {P} _ {\Omega} (\boldsymbol {M} ^ {\star}) - \mathcal {P} _ {\Omega} ^ {(l)} (\boldsymbol {M} ^ {\star})\right) \boldsymbol {x} ^ {(l)}} _ {⑤} + \eta \underbrace {\left(\frac {1}{p} \mathcal {P} _ {\Omega} (\boldsymbol {E}) - \boldsymbol {E} ^ {(l)}\right) \boldsymbol {x} ^ {(l)}} _ {⑥}, \\ \end{array}
$$
where  $\pmb{x}^{(l)}(\tau) = \pmb{x}^{(l)} + \tau (\pmb{x} - \pmb{x}^{(l)})$ . Both of the terms ① and ② can be bounded similar to ① and ② of  $\pmb{x}^{(t + 1)} - \widetilde{\pmb{x}}^{(t + 1)}$  as
$$
\begin{array}{l} \left\| \mathbf {①} \right\| _ {2} \leq \left(1 + \eta \lambda^ {\star} + \frac {c _ {1 1}}{\log^ {2} n}\right) \left\| \boldsymbol {x} - \boldsymbol {x} ^ {(l)} \right\| _ {2}. \\ \left\| \text {②} \right\| _ {2} \lesssim \lambda^ {\star} \sqrt {\frac {1}{\log^ {1 2} n}} \left\| \boldsymbol {x} - \boldsymbol {x} ^ {(l)} \right\| _ {2} \\ \end{array}
$$
for some universal constant  $c_{11} > 0$ . For the terms (3) and (5), we use Lemma G.11 to obtain
$$
\begin{array}{l} \| \textcircled {③} \| _ {2} \lesssim \sqrt {\frac {\log n}{p}} \left\| \boldsymbol {x} ^ {(l)} \right\| _ {2} \left\| \boldsymbol {x} ^ {(l)} \right\| _ {\infty} ^ {2} \lesssim \lambda^ {\star} \mu \sqrt {\frac {\log n}{n p}} \frac {1}{\sqrt {n}} \| \widetilde {\boldsymbol {x}} \| _ {2}, \\ \left\| \widehat {⑤} \right\| _ {2} \lesssim \lambda^ {\star} \mu \sqrt {\frac {\log n}{n p}} \frac {1}{\sqrt {n}} \| \widetilde {\boldsymbol {x}} \| _ {2}, \\ \end{array}
$$
and use Lemma G.13 to obtain
$$
\left\| \text {⑥} \right\| _ {2} \lesssim \lambda^ {\star} \mu \sqrt {\frac {\log^ {2} n}{n p}} \frac {1}{\sqrt {n}} \| \widetilde {\boldsymbol {x}} \| _ {2}.
$$
From Lemmas A.1 and A.3, the term ④ is bounded as
$$
\left\| \text {④} \right\| _ {2} \leq \left\| M ^ {\circ} - M ^ {\star} \right\| \left\| \boldsymbol {x} - \boldsymbol {x} ^ {(l)} \right\| _ {2} \lesssim \lambda^ {\star} \mu \sqrt {\frac {\log n}{n p}} \left\| \boldsymbol {x} - \boldsymbol {x} ^ {(l)} \right\| _ {2}.
$$
Combining all with (E.7), there exists a universal constant  $c_{12} > 0$  such that
$$
\begin{array}{l} \left\| \text {①} \right\| _ {2} + \left\| \text {②} \right\| _ {2} + \left\| \text {④} \right\| _ {2} \leq \left(1 + \eta \lambda^ {\star} + \frac {c _ {1 2}}{\log^ {2} n}\right) \| \boldsymbol {x} - \widetilde {\boldsymbol {x}} \| _ {2}, \\ \eta \left(\left\| \text {③} \right\| _ {2} + \left\| \text {⑤} \right\| _ {2} + \left\| \text {⑥} \right\| _ {2}\right) \leq c _ {1 2} \mu \sqrt {\frac {\log^ {3} n}{n p}} \frac {\beta_ {0}}{\sqrt {n}} (1 + \eta \lambda^ {\star}) ^ {t - T _ {1}}. \\ \end{array}
$$
$$
\left\| \boldsymbol {x} ^ {(s + 1)} - \boldsymbol {x} ^ {(s + 1, l)} \right\| _ {2} \leq \left(1 + \eta \lambda^ {\star} + \frac {c _ {1 2}}{\log^ {2} n}\right) \left\| \boldsymbol {x} ^ {(s)} - \boldsymbol {x} ^ {(s, l)} \right\| _ {2} + c _ {1 2} \mu \sqrt {\frac {\log^ {3} n}{n p}} \frac {\beta_ {0}}{\sqrt {n}} (1 + \eta \lambda^ {\star}) ^ {t - T _ {1}},
$$
for all  $T_1 \leq s \leq t$ . An analysis on the recursive equation
$$
x _ {s + 1} = \left(1 + \eta \lambda^ {\star} + \frac {c _ {1 2}}{\log^ {2} n}\right) x _ {s} + c _ {1 2} \mu \sqrt {\frac {\log^ {3} n}{n p}} \frac {\beta_ {0}}{\sqrt {n}} (1 + \eta \lambda^ {\star}) ^ {t - T _ {1}}, \quad x _ {T _ {1}} = c _ {7} \mu \sqrt {\frac {\log^ {4} n}{n p}} \frac {\beta_ {0}}{\sqrt {n}}
$$
$$
\begin{array}{l} \left\| \boldsymbol {x} ^ {(t + 1)} - \boldsymbol {x} ^ {(t + 1, l)} \right\| _ {2} \leq 2 \left(c _ {1 2} \mu \sqrt {\frac {\log^ {3} n}{n p}} (t + 1 - T _ {1}) + c _ {7} \mu \sqrt {\frac {\log^ {4} n}{n p}}\right) \frac {\beta_ {0}}{\sqrt {n}} (1 + \eta \lambda^ {\star}) ^ {t + 1 - T _ {1}} \\ \leq c _ {1 3} \mu \sqrt {\frac {\log^ {5} n}{n p}} \frac {\beta_ {0}}{\sqrt {n}} (1 + \eta \lambda^ {\star}) ^ {t + 1 - T _ {1}} \\ \end{array}
$$
holds for some universal constant  $c_{13} > 0$  because  $T_{2} - T_{1}\lesssim \log n$
(E.4) at  $(t + 1)$  We use the same bound
$$
\left| \left(\boldsymbol {x} ^ {(t + 1, l)} - \widetilde {\boldsymbol {x}} ^ {(t + 1)}\right) _ {l} \right| \leq \left(1 - \eta \| \widetilde {\boldsymbol {x}} \| _ {2} ^ {2}\right) \left| \left(\boldsymbol {x} ^ {(l)} - \widetilde {\boldsymbol {x}}\right) _ {l} \right| + \eta \left(\lambda^ {\star} \| \boldsymbol {u} ^ {\star} \| _ {\infty} + \| \widetilde {\boldsymbol {x}} \| _ {2} \| \widetilde {\boldsymbol {x}} \| _ {\infty}\right) \left\| \boldsymbol {x} ^ {(l)} - \widetilde {\boldsymbol {x}} \right\| _ {2}.
$$
that was used in the proof of (D.10). From (E.1) and (E.3), we have
$$
\left\| \boldsymbol {x} ^ {(l)} - \widetilde {\boldsymbol {x}} \right\| _ {2} \leq \left\| \boldsymbol {x} ^ {(l)} - \boldsymbol {x} \right\| _ {2} + \| \boldsymbol {x} - \widetilde {\boldsymbol {x}} \| _ {2} \leq 3 c _ {7} \mu \sqrt {\frac {\log^ {5} n}{n p}} \beta_ {0} (1 + \eta \lambda^ {\star}) ^ {t - T _ {1}}.
$$
Combined with the fact that  $\lambda^{\star}\| \pmb{u}^{\star}\|_{\infty} + \| \widetilde{\pmb{x}}\|_{2}\| \widetilde{\pmb{x}}\|_{\infty}\leq 3\lambda^{\star}\sqrt{\frac{\mu}{n}}$ , there exists a universal constant  $c_{14} > 0$  such that
$$
\eta (\lambda^ {\star} \| \boldsymbol {u} ^ {\star} \| _ {\infty} + \| \widetilde {\boldsymbol {x}} \| _ {2} \| \widetilde {\boldsymbol {x}} \| _ {\infty}) \Big \| \boldsymbol {x} ^ {(l)} - \widetilde {\boldsymbol {x}} \Big \| _ {2} \leq c _ {1 4} \sqrt {\frac {\mu^ {3} \log^ {5} n}{n p}} \frac {\beta_ {0}}{\sqrt {n}} (1 + \eta \lambda^ {\star}) ^ {t - T _ {1}}.
$$
Hence, for all  $T_1 \leq s \leq t$ , we have
$$
\left| \left(\boldsymbol {x} ^ {(s + 1, l)} - \widetilde {\boldsymbol {x}} ^ {(s + 1)}\right) _ {l} \right| \leq \left| \left(\boldsymbol {x} ^ {(s, l)} - \widetilde {\boldsymbol {x}} ^ {(s)}\right) _ {l} \right| + c _ {1 4} \sqrt {\frac {\mu^ {3} \log^ {5} n}{n p}} \frac {\beta_ {0}}{\sqrt {n}} (1 + \eta \lambda^ {\star}) ^ {t - T _ {1}},
$$
$$
\begin{array}{l} \left| (\boldsymbol {x} ^ {(t + 1, l)} - \widetilde {\boldsymbol {x}} ^ {(t + 1)}) _ {l} \right| \leq c _ {1 4} \sqrt {\frac {\mu^ {3} \log^ {5} n}{n p}} \frac {\beta_ {0}}{\sqrt {n}} \sum_ {s = T _ {1}} ^ {t} (1 + \eta \lambda^ {\star}) ^ {s - T _ {1}} + c _ {7} \sqrt {\frac {\mu^ {3} \log^ {8} n}{n p}} \frac {\beta_ {0}}{\sqrt {n}} \\ \leq 2 c _ {7} \sqrt {\frac {\mu^ {3} \log^ {8} n}{n p}} \frac {\beta_ {0}}{\sqrt {n}} (1 + \eta \lambda^ {\star}) ^ {t + 1 - T _ {1}}. \\ \end{array}
$$
(E.2) at  $(t + 1)$  The  $l$ th component of  $\pmb{x}^{(t + 1)} - \widetilde{\pmb{x}}^{(t + 1)}$  is bounded by
$$
\begin{array}{l} \left| \left(\boldsymbol {x} ^ {(t + 1)} - \widetilde {\boldsymbol {x}} ^ {(t + 1)}\right) _ {l} \right| \leq \left\| \boldsymbol {x} ^ {(t + 1)} - \boldsymbol {x} ^ {(t + 1, l)} \right\| _ {\infty} + \left| \left(\boldsymbol {x} ^ {(t + 1, l)} - \widetilde {\boldsymbol {x}} ^ {(t + 1)}\right) _ {l} \right| \\ \leq \left\| \boldsymbol {x} ^ {(t + 1)} - \boldsymbol {x} ^ {(t + 1, l)} \right\| _ {2} + \left| (\boldsymbol {x} ^ {(t + 1, l)} - \widetilde {\boldsymbol {x}} ^ {(t + 1)}) _ {l} \right| \\ \leq c _ {1 3} \mu \sqrt {\frac {\log^ {5} n}{n p}} \frac {\beta_ {0}}{\sqrt {n}} (1 + \eta \lambda^ {\star}) ^ {t + 1 - T _ {1}} + 2 c _ {7} \sqrt {\frac {\mu^ {3} \log^ {8} n}{n p}} \frac {\beta_ {0}}{\sqrt {n}} (1 + \eta \lambda^ {\star}) ^ {t + 1 - T _ {1}} \\ \leq 3 c _ {7} \sqrt {\frac {\mu^ {3} \log^ {8} n}{n p}} \frac {\beta_ {0}}{\sqrt {n}} (1 + \eta \lambda^ {\star}) ^ {t + 1 - T _ {1}}. \\ \end{array}
$$
F Fixed Initialization Size
In Section 3, we claimed that the estimation error is improved to  $\frac{1}{\sqrt{n}p} + \frac{\sigma}{\lambda^{\star}}\sqrt{\frac{n}{p}}$  if the initialization size is fixed to  $n^{-1/4}$  regardless of the sample complexity. We briefly discuss how the proofs should change in such a case. For clear presentation,  $\mu$  and  $\log n$  factors are ignored in this section.
For every bound of Phase I (Lemmas 5.1 to 5.5),  $\frac{1}{\sqrt{np}}$  is changed to  $\frac{1}{\sqrt{np}} + \frac{\sigma}{\lambda^{\star}} \sqrt{\frac{n}{p}}$ , while allowing  $\sigma$  to be as large as  $\frac{\lambda^{\star} \mu}{n} \sqrt{np}$ . More importantly, the definition of Phase I is changed to be the largest  $t$  such that  $(1 + \eta \lambda^{\star})^{t} \leq \sqrt{n}$ , so it is lengthened by  $\sqrt{np}$  times than before. In the original proof, the estimation error of  $\frac{1}{\sqrt{np}}$  obtained at the end of Phase I was increased to  $\frac{1}{\mathrm{poly}(\log n)}$  during the first part of Phase II. However, if  $(1 + \eta \lambda^{\star})^{t}$  equals  $\sqrt{n}$  at the end of Phase I, we do not have such a part in Phase II, and the estimation error obtained at the end of Phase I is maintained through Phase II.
We introduce some technical lemmas in this section. Most of them are the results of classical concentration inequalities.
Theorem G.1 (Matrix Bernstein Inequality). Let  $\{\pmb{X}_i\}$  be  $n\times n$  independent symmetric random matrices. Assume that each random matrix satisfies  $\mathbb{E}X_{i} = 0$  and  $\| X_i\| \leq L$  almost surely. Then, for all  $\tau \geq 0$ , we have
$$
\mathbb {P} \left[ \left\| \sum_ {i} \boldsymbol {X} _ {i} \right\| \geq \tau \right] \leq n \exp \left(\frac {- \tau^ {2} / 2}{V + L \tau / 3}\right),
$$
where  $V = \left\| \sum_{i}\mathbb{E}(X_{i}^{2})\right\|$
Corollary G.2 (Matrix Bernstein Inequality). Let  $\{X_i\}$  be  $n \times n$  independent symmetric random matrices. Assume that each random matrix satisfies  $\mathbb{E} X_i = 0$  and  $\| X_i \| \leq L$  almost surely. Then, with high probability, we have
$$
\left\| \sum_ {i} \boldsymbol {X} _ {i} \right\| \lesssim \sqrt {V \log n} + L \log n,
$$
where  $V = \left\| \sum_{i}\mathbb{E}(\boldsymbol{X}_{i}^{2})\right\|$
Lemma G.3. For any fixed matrix  $M \in \mathbb{R}^{n \times n}$ , we have
$$
\left\| \frac {1}{p} \mathcal {P} _ {\Omega} (M) - M \right\| \lesssim \sqrt {\frac {n \log n}{p}} \| M \| _ {\infty} + \frac {\log n}{p} \| M \| _ {\infty}
$$
Proof. We decompose the matrix into the sum of independent symmetric matrices.
$$
\frac {1}{p} \mathcal {P} _ {\Omega} (\boldsymbol {M}) - \boldsymbol {M} = \sum_ {i <   j} \left(\frac {\delta_ {i j}}{p} - 1\right) M _ {i j} \left(\boldsymbol {e} _ {i} \boldsymbol {e} _ {j} ^ {\top} + \boldsymbol {e} _ {j} \boldsymbol {e} _ {i} ^ {\top}\right) + \sum_ {i} \left(\frac {\delta_ {i i}}{p} - 1\right) M _ {i i} \boldsymbol {e} _ {i} \boldsymbol {e} _ {i} ^ {\top}
$$
We calculate  $L$  and  $V$  of Corollary G.2. We have  $L \leq \frac{1}{p} \| M \|_{\infty}$  because
$$
\begin{array}{l} \left\| \left(\frac {\delta_ {i j}}{p} - 1\right) M _ {i j} \left(\boldsymbol {e} _ {i} \boldsymbol {e} _ {j} ^ {\top} + \boldsymbol {e} _ {j} \boldsymbol {e} _ {i} ^ {\top}\right) \right\| \leq \frac {1}{p} \| \boldsymbol {M} \| _ {\infty}, \\ \left\| \left(\frac {\delta_ {i i}}{p} - 1\right) M _ {i i} e _ {i} e _ {i} ^ {\top} \right\| \leq \frac {1}{p} \| \boldsymbol {M} \| _ {\infty}. \\ \end{array}
$$
We also have the following bound on  $V$ .
$$
V = \frac {1 - p}{p} \left\| \sum_ {i, j} M _ {i j} ^ {\star 2} \boldsymbol {e} _ {i} \boldsymbol {e} _ {i} ^ {\top} \right\| \leq \frac {n}{p} \| \boldsymbol {M} \| _ {\infty} ^ {2}
$$
Hence, Corollary G.2 implies the desired result.
We can prove Lemma A.1 by applying Lemma G.3 to  $M^{\star}$  and using  $\| M^{\star}\|_{\infty} = \lambda^{\frac{\mu}{n}}$
We introduce classical Bernstein inequality and the results obtained from it.
Theorem G.4 (Bernstein Inequality). Let  $\{X_i\}$  be independent random variables. Assume that each random variable satisfies  $\mathbb{E}X_{i} = 0$  and  $|X_{i}|\leq L$  almost surely. Then, for all  $\tau \geq 0$ , we have
$$
\mathbb {P} \left[ \left| \sum_ {i} X _ {i} \right| \geq \tau \right] \leq 2 \exp \left(\frac {- \tau^ {2} / 2}{V + L \tau / 3}\right),
$$
where  $V = \sum_{i}\mathbb{E}\big[X_i^2\big]$
Corollary G.5 (Bernstein Inequality). Let  $\{X_i\}$  be independent random variables. Assume that each random variable satisfies  $\mathbb{E}X_{i} = 0$  and  $|X_{i}|\leq L$  almost surely. Then, with high probability, we have
$$
\left| \sum_ {i} X _ {i} \right| \lesssim \sqrt {V \log n} + L \log n,
$$
Lemma G.6. Let and  $\{X_{i}\}$  be independent Bernoulli random variables with expectation  $p$ . Then, for any fixed vector  $\mathbf{a}$ , we have
$$
\left| \sum_ {i} \left(\frac {X _ {i}}{p} - 1\right) a _ {i} \right| \lesssim \sqrt {\frac {\log n}{p}} \| \boldsymbol {a} \| _ {2} + \frac {\log n}{p} \| \boldsymbol {a} \| _ {\infty}
$$
Proof. We can apply Corollary G.5 with  $L = \frac{1}{p}\| \boldsymbol{a}\|_{\infty}$  and  $V = \frac{1 - p}{p}\| \boldsymbol{a}\|_2^2$ .
Lemma G.7. If  $n^2 p \gtrsim \mu n \log n$ , we have
$$
\left\| \frac {1}{p} \mathcal {P} _ {\Omega} \left(\boldsymbol {M} ^ {\star}\right) \right\| _ {2, \infty} \lesssim \lambda^ {\star} \sqrt {\frac {\mu}{n p}}
$$
Proof. Let us consider  $\ell_2$ -norm of the  $i$ th row of  $M^\circ$ .
$$
\begin{array}{l} \left\| \left(\frac {1}{p} \mathcal {P} _ {\Omega} (\boldsymbol {M} ^ {\star})\right) _ {i *} \right\| _ {2} ^ {2} = \lambda^ {\star 2} u _ {i} ^ {\star 2} \sum_ {j} \frac {1}{p ^ {2}} \delta_ {i j} u _ {j} ^ {\star 2} \\ \leq \frac {1}{p} \lambda^ {\star 2} \| \boldsymbol {u} ^ {\star} \| _ {\infty} ^ {2} \left(\| \boldsymbol {u} ^ {\star} \| _ {2} ^ {2} + \left(\sum_ {j} \frac {1}{p} \delta_ {i j} u _ {j} ^ {\star 2} - \| \boldsymbol {u} ^ {\star} \| _ {2} ^ {2}\right)\right) \\ \lesssim \frac {\lambda^ {\star 2} \mu}{n p} \left(1 + \sqrt {\frac {\log n}{n p}}\right) \lesssim \frac {\lambda^ {\star 2} \mu}{n p} \\ \end{array}
$$
The third line follows from Lemma G.6.
Proof of Lemma A.2. The spectral norm of a symmetric matrix that has nonzero entries only on the  $l$ th row/column is bounded by twice of the norm of its  $l$ th row. Hence,
$$
\begin{array}{l} \left\| \frac {1}{p} \mathcal {P} _ {\Omega} (M ^ {\star}) - \mathcal {P} _ {\Omega} ^ {(l)} (M ^ {\star}) \right\| \leq 2 \left\| \left(\frac {1}{p} \mathcal {P} _ {\Omega} (M ^ {\star}) - \mathcal {P} _ {\Omega} ^ {(l)} (M ^ {\star})\right) _ {l *} \right\| _ {2} = 2 \left\| \left(\frac {1}{p} \mathcal {P} _ {\Omega} (M ^ {\star}) - M ^ {\star}\right) _ {l *} \right\| _ {2} \\ \lesssim \left\| \frac {1}{p} \mathcal {P} _ {\Omega} (M ^ {\star}) \right\| _ {2, \infty} + \| M ^ {\star} \| _ {2, \infty} \lesssim \lambda^ {\star} \sqrt {\frac {\mu}{n p}}, \\ \end{array}
$$
where the last inequality follows from Lemma G.7.
Lemma G.8. Let  $\mathbf{y}$  be a vector that is independent from the sampling. Then, if  $n^2 p \gtrsim n \log n$ , we have
$$
\max _ {i \in [ n ]} \left| \| \boldsymbol {x} \| _ {2, i} ^ {2} - \| \boldsymbol {y} \| _ {2} ^ {2} \right| \lesssim n \| \boldsymbol {x} - \boldsymbol {y} \| _ {\infty} (\| \boldsymbol {x} \| _ {\infty} + \| \boldsymbol {y} \| _ {\infty}) + \sqrt {\frac {\log n}{p}} \| \boldsymbol {y} \| _ {2} \| \boldsymbol {y} \| _ {\infty} + \frac {\log n}{p} \| \boldsymbol {y} \| _ {\infty} ^ {2}
$$
with very high probability.
Proof. Let us fix  $i$  and decompose the difference as