SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In this subsection, we provide a proof to the following lemma, which is a formal statement of Lemma 5.2.
Lemma D.1. With high probability, there exists a universal constant  $c_0 > 0$  such that
$$
\left\| \widehat {\boldsymbol {x}} ^ {(t)} - \widetilde {\boldsymbol {x}} ^ {(t)} \right\| _ {2} \leq c _ {0} \mu \sqrt {\frac {\log n}{n p}} \beta_ {0} t \tag {D.1}
$$
for all  $t \leq T_1$ , if  $n^2 p \gtrsim \mu^4 n \log^{21} n$  and the initialization point  $\pmb{x}^{(0)}$  satisfies (B.1) to (B.6).
Proof. Let us rewrite the update equations (2) and (15) as
$$
\widetilde {\boldsymbol {x}} ^ {(t + 1)} = \left(\boldsymbol {I} - \eta \widetilde {\beta} _ {t} ^ {2} + \eta \boldsymbol {M} ^ {\star}\right) \widetilde {\boldsymbol {x}} ^ {(t)},
$$
$$
\widehat {\boldsymbol {x}} ^ {(t + 1)} = \left(\boldsymbol {I} - \eta \widetilde {\beta} _ {t} ^ {2} + \eta \boldsymbol {M} ^ {\circ}\right) \widehat {\boldsymbol {x}} ^ {(t)},
$$
where  $\widetilde{\beta}_t = \left\| \widetilde{\pmb{x}}^{(t)}\right\| _2^2$ . Then,  $\widehat{\pmb{x}}^{(t)} - \widetilde{\pmb{x}}^{(t)}$  is a product between  $\pmb{x}^{(0)}$  and  $P^{(t)}(I,M^{\star},H)$ , which is a matrix polynomial of  $\pmb{I}$ ,  $M^{\star}$ ,  $\pmb{H}$ , where  $\pmb{H} = \pmb{M}^{\circ} - \pmb{M}^{\star}$ .
$$
P ^ {(t)} (\boldsymbol {I}, \boldsymbol {M} ^ {\star}, \boldsymbol {H}) := \left(\prod_ {s = 1} ^ {t} \left((1 - \eta \widetilde {\beta} _ {s} ^ {2}) \boldsymbol {I} + \eta \boldsymbol {M} ^ {\star} + \eta \boldsymbol {H}\right) - \prod_ {s = 1} ^ {t} \left((1 - \eta \widetilde {\beta} _ {s} ^ {2}) \boldsymbol {I} + \eta \boldsymbol {M} ^ {\star}\right)\right) \tag {D.2}
$$
$$
\widehat {\boldsymbol {x}} ^ {(t)} - \widetilde {\boldsymbol {x}} ^ {(t)} = P ^ {(t)} (\boldsymbol {I}, \boldsymbol {M} ^ {\star}, \boldsymbol {H}) \boldsymbol {x} ^ {(0)} \tag {D.3}
$$
We classify the terms that appear after expanding the matrix polynomial  $P^{(t)}(\mathbf{I}, \mathbf{M}^{\star}, \mathbf{H})$  into two types; 1) the terms that contain  $\mathbf{H}$  but not  $\mathbf{M}^{\star}$ , 2) the terms that contain both  $\mathbf{H}$  and  $\mathbf{M}^{\star}$ . We define  $P_{1}^{(t)}(\mathbf{I}, \mathbf{H})$  to be a matrix polynomial of  $\mathbf{I}$  and  $\mathbf{H}$ , which is equal to summation of the first type, and it is explicitly written as
$$
P _ {1} ^ {(t)} (\pmb {I}, \pmb {H}) = \prod_ {s = 1} ^ {t} \left((1 - \eta \widetilde {\beta} _ {s} ^ {2}) \pmb {I} + \eta \pmb {H}\right) - \prod_ {s = 1} ^ {t} (1 - \eta \widetilde {\beta} _ {s} ^ {2}) \pmb {I}.
$$
We correspondingly define  $P_2^{(t)}(\pmb {I},\pmb{M}^\star ,\pmb {H})$  to be summation of the second type, and it is equal to
$$
P _ {2} ^ {(t)} (\boldsymbol {I}, \boldsymbol {M} ^ {\star}, \boldsymbol {H}) = P ^ {(t)} (\boldsymbol {I}, \boldsymbol {M} ^ {\star}, \boldsymbol {H}) - P _ {1} ^ {(t)} (\boldsymbol {I}, \boldsymbol {H}).
$$
For  $x, y \in \mathbb{R}$ , we define  $P_1^{(t)}(x,y)$  as the value that is obtained by substituting  $x,y$  instead of  $\pmb{I}$ ,  $\pmb{H}$ , respectively. For example,  $P_1^{(t)}(1,2) = \prod_{s=1}^t (1 - \eta \widetilde{\beta}_s^2 + 2\eta) - \prod_{s=1}^t (1 - \eta \widetilde{\beta}_s^2)$ . For  $x,y,z \in \mathbb{R}$ ,  $P_2^{(t)}(x,y,z)$  is defined in a similar manner.
We bound the contribution of each type separately because the triangle inequality gives
$$
\left\| \widehat {\boldsymbol {x}} ^ {(t)} - \widetilde {\boldsymbol {x}} ^ {(t)} \right\| _ {2} \leq \left\| P _ {1} ^ {(t)} (\boldsymbol {I}, \boldsymbol {H}) \boldsymbol {x} ^ {(0)} \right\| _ {2} + \left\| P _ {2} ^ {(t)} (\boldsymbol {I}, \boldsymbol {M} ^ {\star}, \boldsymbol {H}) \boldsymbol {x} ^ {(0)} \right\| _ {2}.
$$
Every term in  $P_1^{(t)}(\pmb {I},\pmb {H})$  is  $\pmb{H}^s$  times a constant. We have  $\left\| H^{s}\pmb{x}^{(0)}\right\|_{2}\leq \left\| H\right\|^{s}\left\| \pmb{x}^{(0)}\right\|_{2}$ , and hence with triangle inequality
$$
\left\| P _ {1} ^ {(t)} (\boldsymbol {I}, \boldsymbol {H}) \boldsymbol {x} ^ {(0)} \right\| _ {2} \leq P _ {1} ^ {(t)} (1, \| \boldsymbol {H} \|) \beta_ {0}.
$$
If  $n^2 p \gtrsim \mu^2 n\log^3 n$ , we can further bound  $P_1^{(t)}(1,\| H\|)$  as
$$
\begin{array}{l} P _ {1} ^ {(t)} (1, \| \boldsymbol {H} \|) = \prod_ {s = 1} ^ {t} (1 - \eta \widetilde {\beta} _ {s} ^ {2} + \eta \| \boldsymbol {H} \|) - \prod_ {s = 1} ^ {t} (1 - \eta \widetilde {\beta} _ {s} ^ {2}) \\ = \prod_ {s = 1} ^ {t} (1 - \eta \widetilde {\beta} _ {s} ^ {2}) \left(\prod_ {s = 1} ^ {t} \left(1 + \frac {\eta \| \boldsymbol {H} \|}{1 - \eta \widetilde {\beta} _ {s} ^ {2}}\right) - 1\right) \\ \leq \left(1 + \frac {\eta}{1 - \eta \lambda^ {\star}} \| \boldsymbol {H} \|\right) ^ {t} - 1 \\ \leq \left(\exp \left(\frac {\eta}{1 - \eta \lambda^ {\star}} \| \boldsymbol {H} \| t\right) - 1\right) \\ \leq \frac {2 \eta}{1 - \eta \lambda^ {\star}} \| \boldsymbol {H} \| t \\ \end{array}
$$
The third line uses the fact that  $\widetilde{\beta}_t^2 \leq \lambda^\star$  for all  $t \leq T_1$ . The fourth and fifth lines are derived from an elementary inequality  $1 + x \leq e^x \leq 1 + 2x$ , which holds for small  $x > 0$ . Note that  $\eta \| H \| t \lesssim \eta \lambda^\star \mu \sqrt{\frac{\log n}{np}} \ll 1$  from Lemmas A.1 and A.3, and the fact that  $T_1 \lesssim \log n$ .
We can decompose every term of second type as a product of  $\eta$ ,  $\lambda^{\star}$ ,  $\pmb{u}^{\star}$ ,  $\pmb{H}^{s}\pmb{u}^{\star}$ ,  $\pmb{u}^{\star \top}\pmb{H}^{s}\pmb{x}^{(0)}$ ,  $\pmb{u}^{\star \top}\pmb{H}^{s}\pmb{u}^{\star}$ , and  $\pmb{u}^{\star \top}\pmb{x}^{(0)}$ . We describe this with some examples.
$$
(\eta \boldsymbol {H}) ^ {s _ {1}} (\eta \boldsymbol {M} ^ {\star}) (\eta \boldsymbol {H}) ^ {s _ {2}} \boldsymbol {x} ^ {(0)} = \eta^ {s _ {1} + s _ {2} + 1} \lambda^ {\star} \left(\boldsymbol {H} ^ {s _ {1}} \boldsymbol {u} ^ {\star}\right) \left(\boldsymbol {u} ^ {\star \top} \boldsymbol {H} ^ {s _ {2}} \boldsymbol {x} ^ {(0)}\right)
$$
$$
(\eta \boldsymbol {H}) ^ {s} (\eta \boldsymbol {M} ^ {\star}) \boldsymbol {x} ^ {(0)} = \eta^ {s + 1} \left(\boldsymbol {H} ^ {s} \boldsymbol {u} ^ {\star}\right) \left(\boldsymbol {u} ^ {\star \top} \boldsymbol {x} ^ {(0)}\right)
$$
$$
(\eta \boldsymbol {M} ^ {\star}) (\eta \boldsymbol {H}) ^ {s} (\eta \boldsymbol {M} ^ {\star}) \boldsymbol {x} ^ {(0)} = \eta^ {s + 2} \lambda^ {\star 2} \boldsymbol {u} ^ {\star} (\boldsymbol {u} ^ {\star \top} \boldsymbol {H} \boldsymbol {u} ^ {\star}) (\boldsymbol {u} ^ {\star \top} \boldsymbol {x} ^ {(0)})
$$
$$
(\eta \boldsymbol {M} ^ {\star}) (\eta \boldsymbol {H}) ^ {s} \boldsymbol {x} ^ {(0)} = \eta^ {s + 1} \lambda^ {\star} \boldsymbol {u} ^ {\star} (\boldsymbol {u} ^ {\top} \boldsymbol {H} ^ {s} \boldsymbol {x} ^ {(0)})
$$
The terms  $H^{s}u^{\star}$  and  $u^{\star \top}H^{s}u^{\star}$  are bounded with
$$
\left\| \boldsymbol {H} ^ {s} \boldsymbol {u} ^ {\star} \right\| _ {2} \leq \left\| \boldsymbol {H} \right\| ^ {s}, \quad \left| \boldsymbol {u} ^ {\star \top} \boldsymbol {H} ^ {s} \boldsymbol {u} ^ {\star} \right| \leq \left\| \boldsymbol {H} \right\| ^ {s}, \tag {D.4}
$$
and the terms that contain  $\pmb{x}^{(0)}$  are bounded with (B.3). For every term of second type that includes  $s_1$  times of  $\eta M^{\star}$  and  $s_2$  times of  $\eta H$ , the bounds (D.4) and (B.3) imply that  $\ell_2$ -norm of the term multiplied by  $\pmb{x}^{(0)}$  is at most
$$
\left(\eta \lambda^ {\star}\right) ^ {s _ {1}} \left(\eta \| \boldsymbol {H} \|\right) ^ {s _ {2}} 2 \sqrt {\frac {\log n}{n}} \beta_ {0}.
$$
Hence, similar to the first type, we have
$$
\left\| P _ {2} ^ {(t)} (\boldsymbol {I}, \boldsymbol {M} ^ {\star}, \boldsymbol {H}) \boldsymbol {x} ^ {(0)} \right\| _ {2} \leq P _ {2} ^ {(t)} (1, \lambda^ {\star}, \| \boldsymbol {H} \|) 2 \sqrt {\frac {\log n}{n}} \beta_ {0}.
$$
If  $n^2 p \gtrsim \mu^2 n\log^3 n$ , we can further bound  $P_2^{(t)}(1,\lambda^\star, \| H\|)$  as
$$
\begin{array}{l} P _ {2} ^ {(t)} (1, \lambda^ {\star}, \| \boldsymbol {H} \|) = \prod_ {s = 1} ^ {t} (1 - \eta \beta_ {s} ^ {2} + \eta \lambda^ {\star} + \eta \| \boldsymbol {H} \|) - \prod_ {s = 1} ^ {t} \left(1 - \eta \beta_ {s} ^ {2} + \eta \lambda^ {\star}\right) - P _ {1} ^ {(t)} (1, \| \boldsymbol {H} \|) \\ \leq \prod_ {s = 1} ^ {t} \left(1 - \eta \beta_ {s} ^ {2} + \eta \lambda^ {\star} + \eta \| \boldsymbol {H} \|\right) - \prod_ {s = 1} ^ {t} \left(1 - \eta \beta_ {s} ^ {2} + \eta \lambda^ {\star}\right) \\ \leq \left(\prod_ {s = 1} ^ {t} \left(1 - \eta \beta_ {s} ^ {2} + \eta \lambda^ {\star}\right)\right) \left(\prod_ {s = 1} ^ {t} \left(1 + \frac {\eta}{1 - \eta \beta_ {s} ^ {2} + \eta \lambda^ {\star}} \| \boldsymbol {H} \|\right) - 1\right) \\ \leq \left(1 + \eta \lambda^ {\star}\right) ^ {t} \left(\left(1 + \eta \| \boldsymbol {H} \|\right) ^ {t} - 1\right) \\ \leq 2 \eta \| \boldsymbol {H} \| t (1 + \eta \lambda^ {\star}) ^ {t}. \\ \end{array}
$$
Combining all, we have
$$
\begin{array}{l} \left\| \widehat {\boldsymbol {x}} ^ {(t)} - \widetilde {\boldsymbol {x}} ^ {(t)} \right\| _ {2} \leq 4 \eta \| \boldsymbol {H} \| t \left(1 + \sqrt {\frac {\log n}{n}} (1 + \eta \lambda^ {\star}) ^ {t}\right) \beta_ {0} \\ \leq 4 \eta \| \boldsymbol {H} \| t \left(1 + \sqrt {\frac {\log n}{n}} (1 + \eta \lambda^ {\star}) ^ {T _ {1}}\right) \beta_ {0} \\ \leq 4 \eta \| \boldsymbol {H} \| t \left(1 + \sqrt {\frac {\mu^ {4} \log^ {2 2} n}{n p}}\right) \beta_ {0} \\ \leq c _ {0} \mu \sqrt {\frac {\log n}{n p}} \beta_ {0} t \\ \end{array}
$$
for all  $t \leq T_1$  for some constant  $c_0 > 0$  if  $n^2 p \gtrsim \mu^4 n \log^{22} n$ .
D.2 Proof of Lemmas 5.3 to 5.5
We prove Lemmas 5.3 to 5.5 all together in an inductive manner.
Lemma D.2. Suppose that the initialization point  $\pmb{x}^{(0)}$  satisfies (B.1) to (B.6). If  $n^2 p \gtrsim \mu^5 n\log^{22}n$ , for all  $t \leq T_1$ , we have
$$
\left\| \boldsymbol {x} ^ {(t)} - \widetilde {\boldsymbol {x}} ^ {(t)} \right\| _ {2} \leq 2 c _ {0} \mu \sqrt {\frac {\log n}{n p}} \beta_ {0} t, \tag {D.5}
$$
$$
\left\| \boldsymbol {x} ^ {(t)} - \widetilde {\boldsymbol {x}} ^ {(t)} \right\| _ {\infty} \leq \left(3 c _ {0} + c _ {5}\right) \sqrt {\frac {\mu^ {3} \log^ {2} n}{n p}} \frac {\beta_ {0}}{\sqrt {n}} t ^ {2}, \tag {D.6}
$$
$$
\left\| \boldsymbol {x} ^ {(t)} - \widehat {\boldsymbol {x}} ^ {(t)} \right\| _ {2} \leq c _ {2} \left(3 c _ {0} + c _ {5} + 1\right) \frac {1}{\lambda^ {\star}} \sqrt {\frac {\mu^ {3} \log^ {3} n}{n p}} (1 + \eta \lambda^ {\star}) ^ {t} \beta_ {0} ^ {3}, \tag {D.7}
$$
$$
\left\| \boldsymbol {x} ^ {(t)} - \boldsymbol {x} ^ {(t, l)} \right\| _ {2} \leq c _ {5} \mu \sqrt {\frac {\log^ {2} n}{n p}} \frac {\beta_ {0}}{\sqrt {n}} t, \tag {D.8}
$$
$$
\left| \boldsymbol {u} ^ {(l) \top} \left(\boldsymbol {x} ^ {(t)} - \boldsymbol {x} ^ {(t, l)}\right) \right| \leq c _ {6} \sqrt {\frac {\mu^ {3} \log^ {2} n}{n p}} (1 + \eta \lambda^ {\star}) ^ {t} \frac {\beta_ {0}}{n}, \tag {D.9}
$$
$$
\left| \left(\boldsymbol {x} ^ {(t, l)} - \widetilde {\boldsymbol {x}} ^ {(t)}\right) _ {l} \right| \leq 3 c _ {0} \sqrt {\frac {\mu^ {3} \log^ {2} n}{n p}} \frac {\beta_ {0}}{\sqrt {n}} t ^ {2}, \tag {D.10}
$$
with high probability, where  $c_{2}, c_{5}, c_{6}$  are positive constants.
Before we start the proof, we introduce some notations. For  $\pmb{x} \in \mathbb{R}^n$ , let us define
$$
\| \boldsymbol {x} \| _ {2, i} = \sqrt {\frac {1}{p} \sum_ {j = 1} ^ {n} \delta_ {i j} x _ {j} ^ {2}}, \quad \boldsymbol {I} _ {\boldsymbol {x}} = \frac {1}{\| \boldsymbol {x} \| _ {2} ^ {2}} \operatorname {d i a g} (\| \boldsymbol {x} \| _ {2, 1} ^ {2}, \dots , \| \boldsymbol {x} \| _ {2, n} ^ {2}).
$$
$\| \pmb{x}\|_{2,i}$  is the  $\ell_2$ -norm of  $\pmb{x}$  estimated with sampling of the  $i$ th row. With this notation, we can write the gradient of  $f$  as
$$
\nabla f (\boldsymbol {x}) = \left\| \boldsymbol {x} \right\| _ {2} ^ {2} I _ {\boldsymbol {x}} \boldsymbol {x} - M ^ {\circ} \boldsymbol {x}.
$$
The function  $g$  is defined as
$$
g (\boldsymbol {x}) = \frac {1}{4 p} \left\| \mathcal {P} _ {\Omega} \left(\boldsymbol {x} \boldsymbol {x} ^ {\top}\right) \right\| _ {\mathrm {F}} ^ {2},
$$
and its gradient satisfies
$$
\nabla f (\boldsymbol {x}) = \nabla g (\boldsymbol {x}) - M ^ {\circ} \boldsymbol {x}.
$$
The Hessian of  $g(\pmb {x})$  is equal to
$$
\nabla^ {2} g (\boldsymbol {x}) = \| \boldsymbol {x} \| _ {2} ^ {2} \boldsymbol {I} _ {\boldsymbol {x}} + \frac {2}{p} \mathcal {P} _ {\Omega} \left(\boldsymbol {x x} ^ {\top}\right).
$$
The base case  $t = 0$  for induction hypotheses (D.5) to (D.10) trivially hold because all three sequences  $\pmb{x}^{(t)}, \widehat{\pmb{x}}^{(t)}, \widetilde{\pmb{x}}^{(t)}$  start from the same point. Now, we assume that the hypotheses hold up to the  $t$ th iteration and show that they hold at the  $(t + 1)$ st iteration. For brevity, we drop the superscript  $(t)$  from  $\pmb{x}^{(t)}, \pmb{x}^{(t,l)}, \widehat{\pmb{x}}^{(t)}, \widetilde{\pmb{x}}^{(t)}$  and denote them as  $\pmb{x}, \pmb{x}^{(l)}, \widehat{\pmb{x}}, \widetilde{\pmb{x}}$ , respectively. Also, recall that  $T_{1}$  is defined to be the last  $t$  such that  $(1 + \eta \lambda^{\star})^{t} \leq \sqrt{\frac{\mu^{4} \log^{21} n}{np}} \sqrt{n}$ , and the magnitude of initialization satisfies  $\beta_{0}^{2} \lesssim \lambda^{\star} \sqrt{\frac{np}{\mu^{5} \log^{26} n}} \frac{1}{\sqrt{n}}$  so that there exists a constant  $c_{1} > 0$  such that  $(1 + \eta \lambda^{\star})^{t} \beta_{0}^{2} \leq c_{1} \frac{\lambda^{\star}}{\sqrt{\mu \log^{5} n}}$ .
(D.7) at  $(t + 1)$  We first decompose  $\pmb{x}^{(t + 1)} - \widehat{\pmb{x}}^{(t + 1)}$  as
$$
\begin{array}{l} \boldsymbol {x} ^ {(t + 1)} - \widehat {\boldsymbol {x}} ^ {(t + 1)} = (\boldsymbol {I} + \eta \boldsymbol {M} ^ {\circ}) (\boldsymbol {x} - \widehat {\boldsymbol {x}}) - \eta \| \boldsymbol {x} \| _ {2} ^ {2} \boldsymbol {I} _ {\boldsymbol {x}} \boldsymbol {x} + \eta \| \widetilde {\boldsymbol {x}} \| _ {2} ^ {2} \widehat {\boldsymbol {x}} \\ = \left(\boldsymbol {I} - \eta \| \widetilde {\boldsymbol {x}} \| _ {2} ^ {2} \boldsymbol {I} + \eta M ^ {\circ}\right) (\boldsymbol {x} - \widehat {\boldsymbol {x}}) - \eta \left(\| \boldsymbol {x} \| _ {2} ^ {2} \boldsymbol {I} _ {\boldsymbol {x}} - \| \widetilde {\boldsymbol {x}} \| _ {2} ^ {2} \boldsymbol {I}\right) \boldsymbol {x}. \tag {D.11} \\ \end{array}
$$
With the help of Lemma G.8, we bound the maximum entry of a diagonal matrix  $\| \pmb {x}\| _2^2\pmb {I}_\pmb{x} - \| \widetilde{\pmb{x}}\| _2^2\pmb {I}$  We have
$$
\begin{array}{l} \max  _ {i \in [ n ]} \left| \| \boldsymbol {x} \| _ {2, i} ^ {2} - \| \widetilde {\boldsymbol {x}} \| _ {2} ^ {2} \right| \lesssim n \| \boldsymbol {x} - \widetilde {\boldsymbol {x}} \| _ {\infty} \| \boldsymbol {x} + \widetilde {\boldsymbol {x}} \| _ {\infty} + \sqrt {\frac {\log n}{p}} \| \widetilde {\boldsymbol {x}} \| _ {2} \| \widetilde {\boldsymbol {x}} \| _ {\infty} + \frac {\log n}{p} \| \widetilde {\boldsymbol {x}} \| _ {\infty} ^ {2} \\ \lesssim (3 c _ {0} + c _ {5}) \sqrt {\frac {\mu^ {3} \log^ {3} n}{n p}} \beta_ {0} ^ {2} t ^ {2} + \sqrt {\frac {\log^ {2} n}{n p}} \beta_ {0} ^ {2} + \frac {\log^ {2} n}{n p} \beta_ {0} ^ {2} \\ \lesssim \sqrt {\frac {\mu^ {3} \log^ {3} n}{n p}} \beta_ {0} ^ {2} ((3 c _ {0} + c _ {5}) t ^ {2} + 1) \\ \end{array}
$$
if  $n^2 p \gtrsim n\log^2 n$ . Hence, there exists a universal constant  $c_2 > 0$  that is independent of  $t$  such that
$$
\left(\max  _ {i \in [ n ]} \left| \left\| \boldsymbol {x} ^ {(t)} \right\| _ {2, i} ^ {2} - \left\| \widetilde {\boldsymbol {x}} ^ {(t)} \right\| _ {2} ^ {2} \right|\right) \left\| \boldsymbol {x} ^ {(t)} \right\| _ {2} \leq \frac {1}{2} c _ {2} \sqrt {\frac {\mu^ {3} \log^ {3} n}{n p}} \beta_ {0} ^ {3} ((3 c _ {0} + c _ {5}) t ^ {2} + 1)
$$
for all  $t \leq T_1$ . With the decomposition (D.11), we have
$$
\begin{array}{l} \left\| \boldsymbol {x} ^ {(t + 1)} - \widehat {\boldsymbol {x}} ^ {(t + 1)} \right\| _ {2} \leq \left(1 - \eta \| \widetilde {\boldsymbol {x}} \| _ {2} ^ {2} + \eta \lambda^ {\circ}\right) \| \boldsymbol {x} - \widehat {\boldsymbol {x}} \| _ {2} + \eta \left(\max  _ {i \in [ n ]} \left\| \boldsymbol {x} \| _ {2, i} ^ {2} - \| \widetilde {\boldsymbol {x}} \| _ {2} ^ {2} \right|\right) \| \boldsymbol {x} \| _ {2} \\ \leq \left(1 + \eta \lambda^ {\circ}\right) \| \boldsymbol {x} - \widehat {\boldsymbol {x}} \| _ {2} + \frac {1}{2} c _ {2} \left(\eta \lambda^ {\star}\right) ^ {3} \frac {1}{\lambda^ {\star}} \sqrt {\frac {\mu^ {3} \log^ {3} n}{n p}} \beta_ {0} ^ {3} \left(\left(3 c _ {0} + c _ {5}\right) t ^ {2} + 1\right). \\ \end{array}
$$
From (A.1), there exists a universal constant  $c_3 > 0$  such that  $\eta \lambda^\circ \leq \eta \lambda^\star + \frac{c_3}{\log^2 n}$  if  $n^2 p \gtrsim \mu^2 n \log^5 n$ . Combining all, for all  $s \leq t$ , we have
$$
\begin{array}{l} \left\| \boldsymbol {x} ^ {(s + 1)} - \widehat {\boldsymbol {x}} ^ {(s + 1)} \right\| _ {2} \leq \left(1 + \eta \lambda^ {\star} + \frac {c _ {3}}{\log^ {2} n}\right) \left\| \boldsymbol {x} ^ {(s)} - \widehat {\boldsymbol {x}} ^ {(s)} \right\| _ {2} \\ + \frac {1}{2} c _ {2} (\eta \lambda^ {\star}) ^ {3} \frac {1}{\lambda^ {\star}} \sqrt {\frac {\mu^ {3} \log^ {3} n}{n p}} \beta_ {0} ^ {3} ((3 c _ {0} + c _ {5}) s ^ {2} + 1). \\ \end{array}
$$
An analysis on the recursive equation
$$
x _ {s + 1} = \left(1 + \eta \lambda^ {\star} + \frac {c _ {3}}{\log^ {2} n}\right) x _ {s} + \frac {1}{2} c _ {2} (\eta \lambda^ {\star}) ^ {3} \frac {1}{\lambda^ {\star}} \sqrt {\frac {\mu^ {3} \log^ {3} n}{n p}} \beta_ {0} ^ {3} ((3 c _ {0} + c _ {5}) s ^ {2} + 1), \quad x _ {0} = 0,
$$
proves that
$$
\left\| \pmb {x} ^ {(t + 1)} - \widehat {\pmb {x}} ^ {(t + 1)} \right\| _ {2} \leq c _ {2} (3 c _ {0} + c _ {5} + 1) \frac {1}{\lambda^ {\star}} \sqrt {\frac {\mu^ {3} \log^ {3} n}{n p}} (1 + \eta \lambda^ {\star}) ^ {t + 1} \beta_ {0} ^ {3}.
$$
(D.8) at  $(t + 1)$  We decompose  $x^{(t + 1)} - x^{(t + 1,l)}$  as
$$
\begin{array}{l} \boldsymbol {x} ^ {(t + 1)} - \boldsymbol {x} ^ {(t + 1, l)} = (1 - \eta \| \widetilde {\boldsymbol {x}} \| _ {2} ^ {2}) (\boldsymbol {x} - \boldsymbol {x} ^ {(l)}) - 2 \eta \underbrace {\widetilde {\boldsymbol {x}} \widetilde {\boldsymbol {x}} ^ {\top} (\boldsymbol {x} - \boldsymbol {x} ^ {(l)})} _ {①} \\ - \eta \underbrace {\int_ {0} ^ {1} \left(\nabla^ {2} g (\boldsymbol {x} (\tau)) - \left(\| \widetilde {\boldsymbol {x}} \| _ {2} ^ {2} \boldsymbol {I} + 2 \widetilde {\boldsymbol {x}} \widetilde {\boldsymbol {x}} ^ {\top}\right)\right) (\boldsymbol {x} - \boldsymbol {x} ^ {(l)}) \mathrm {d} \tau} _ {②} \\ - \eta \underbrace {\left(\frac {1}{p} \mathcal {P} _ {\Omega_ {l}} \left(\boldsymbol {x} ^ {(t , l)} \boldsymbol {x} ^ {(t , l) \top}\right) - \mathcal {P} _ {l} \left(\boldsymbol {x} ^ {(t , l)} \boldsymbol {x} ^ {(t , l) \top}\right)\right) \boldsymbol {x} ^ {(t , l)}} _ {③} \\ + \eta \underbrace {\lambda^ {(l)} \boldsymbol {u} ^ {(l)} \boldsymbol {u} ^ {(l) \top} \left(\boldsymbol {x} ^ {(t)} - \boldsymbol {x} ^ {(t , l)}\right)} _ {④} + \eta \underbrace {\left(M ^ {\circ} - \lambda^ {(l)} \boldsymbol {u} ^ {(l)} \boldsymbol {u} ^ {(l) \top}\right) \left(\boldsymbol {x} ^ {(t)} - \boldsymbol {x} ^ {(t , l)}\right)} _ {⑤} \\ + \eta \underbrace {\left(\frac {1}{p} \mathcal {P} _ {\Omega} (\boldsymbol {M} ^ {\star}) - \mathcal {P} _ {\Omega} ^ {(l)} (\boldsymbol {M} ^ {\star})\right) \boldsymbol {x} ^ {(t , l)}} _ {⑥} + \eta \underbrace {\left(\frac {1}{p} \mathcal {P} _ {\Omega} (\boldsymbol {E}) - \boldsymbol {E} ^ {(l)}\right) \boldsymbol {x} ^ {(t , l)}} _ {⑦}, \\ \end{array}
$$
where  $\pmb{x}^{(l)}(\tau) = \pmb{x}^{(l)} + \tau (\pmb{x} - \pmb{x}^{(l)})$ . ① is easily bounded by
$$
\left\| \widehat {①} \right\| _ {2} \leq \left\| \widetilde {\boldsymbol {x}} \right\| _ {2} ^ {2} \left\| \boldsymbol {x} - \boldsymbol {x} ^ {(l)} \right\| _ {2} \lesssim \beta_ {0} ^ {2} \left\| \boldsymbol {x} - \boldsymbol {x} ^ {(l)} \right\| _ {2} \lesssim \lambda^ {\star} \sqrt {\frac {1}{\mu^ {5} \log^ {2 6} n}} \sqrt {\frac {n p}{n}} \left\| \boldsymbol {x} - \boldsymbol {x} ^ {(l)} \right\| _ {2}.
$$
From Lemma G.10 and (C.4), for all  $0 \leq \tau \leq 1$ , we have
$$
\left\| \nabla^ {2} g \left(\boldsymbol {x} ^ {(l)} (\tau)\right) - \left(\left\| \widetilde {\boldsymbol {x}} \right\| _ {2} ^ {2} \boldsymbol {I} + 2 \widetilde {\boldsymbol {x}} \widetilde {\boldsymbol {x}} ^ {\top}\right) \right\| \lesssim n \| \boldsymbol {x} (\tau) - \widetilde {\boldsymbol {x}} \| _ {\infty} \left(\left\| \boldsymbol {x} (\tau) \right\| _ {\infty} + \left\| \widetilde {\boldsymbol {x}} \right\| _ {\infty}\right) + \sqrt {\frac {\log^ {3} n}{n p}} \beta_ {0} ^ {2}. \tag {D.12}
$$
From the definition of  $\pmb{x}(\tau)$ , we have
$$
\left\| \boldsymbol {x} ^ {(l)} (\tau) - \widetilde {\boldsymbol {x}} \right\| _ {\infty} \leq (1 - \tau) \left\| \boldsymbol {x} ^ {(l)} - \boldsymbol {x} \right\| _ {2} + \| \boldsymbol {x} - \widetilde {\boldsymbol {x}} \| _ {\infty} \lesssim \sqrt {\frac {\mu^ {3} \log^ {6} n}{n p}} \frac {\beta_ {0}}{\sqrt {n}},
$$
where the last inequality is from the induction hypotheses (D.6), (D.8), and the fact that  $t \leq T_1 \lesssim \log n$ . Inserting this bound back to (D.12), we get
$$
\left\| \nabla^ {2} g \left(\boldsymbol {x} ^ {(l)} (\tau)\right) - \left(\left\| \widetilde {\boldsymbol {x}} \right\| _ {2} ^ {2} \boldsymbol {I} + 2 \widetilde {\boldsymbol {x}} \widetilde {\boldsymbol {x}} ^ {\top}\right) \right\| \lesssim \sqrt {\frac {\mu^ {3} \log^ {7} n}{n p}} \beta_ {0} ^ {2}, \tag {D.13}
$$
$$
\left\| \textcircled {2} \right\| _ {2} \lesssim \sqrt {\frac {\mu^ {3} \log^ {7} n}{n p}} \beta_ {0} ^ {2} \left\| \boldsymbol {x} - \boldsymbol {x} ^ {(l)} \right\| _ {2} \lesssim \lambda^ {\star} \sqrt {\frac {1}{\mu^ {2} n \log^ {1 9} n}} \left\| \boldsymbol {x} - \boldsymbol {x} ^ {(l)} \right\| _ {2}.
$$
It is implied from Lemma G.11 that
$$
\| \mathbf {\Phi} 3 \| _ {2} \leq \left\| \boldsymbol {x} ^ {(l)} \right\| _ {\infty} ^ {2} \sqrt {\frac {\log n}{p}} \left\| \boldsymbol {x} ^ {(l)} \right\| _ {2} \lesssim \sqrt {\frac {\log^ {4} n}{n p}} \frac {\beta_ {0} ^ {3}}{\sqrt {n}} \lesssim \lambda^ {\star} \sqrt {\frac {1}{\mu^ {5} n \log^ {2 2} n}} \frac {\beta_ {0}}{\sqrt {n}}.
$$
A bound on ④ follows from the induction hypothesis (D.9) and the spectral bound (A.2).
$$
\left\| \bigcirc 4 \right\| _ {2} \leq \lambda^ {(l)} \left| \boldsymbol {u} ^ {(l) \top} (\boldsymbol {x} ^ {(t)} - \boldsymbol {x} ^ {(t, l)}) \right| \lesssim \lambda^ {\star} \sqrt {\frac {\mu^ {3} \log^ {2} n}{n p}} \frac {\beta_ {0}}{n} (1 + \eta \lambda^ {\star}) ^ {t} \lesssim \lambda^ {\star} \frac {\sqrt {\mu^ {7} \log^ {2 3} n}}{n p} \frac {\beta_ {0}}{\sqrt {n}}.
$$
The second largest eigenvalue of  $M^{(l)}$  is at most  $\left\| M^{(l)} - M^{\star} \right\|$  by Weyl's Theorem, and from Lemmas A.1 and A.2, we have
$$
\left\| \boldsymbol {M} ^ {(l)} - \boldsymbol {M} ^ {\star} \right\| \lesssim \left\| \boldsymbol {M} ^ {(l)} - \boldsymbol {M} ^ {\circ} \right\| + \| \boldsymbol {M} ^ {\circ} - \boldsymbol {M} ^ {\star} \| \lesssim \lambda^ {\star} \mu \sqrt {\frac {\log n}{n p}}.
$$
$$
\begin{array}{l} \left\| \text {⑤} \right\| _ {2} \leq \left(\left\| \boldsymbol {M} ^ {\circ} - \boldsymbol {M} ^ {(l)} \right\| + \left\| \boldsymbol {M} ^ {(l)} - \lambda^ {(l)} \boldsymbol {u} ^ {(l)} \boldsymbol {u} ^ {(l) \top} \right\|\right) \left\| \boldsymbol {x} - \boldsymbol {x} ^ {(l)} \right\| _ {2} \\ \lesssim \lambda^ {\star} \mu \sqrt {\frac {\log n}{n p}} \left\| \boldsymbol {x} - \boldsymbol {x} ^ {(l)} \right\| _ {2}. \\ \end{array}
$$
Lastly, we apply Lemmas G.11 and G.13 to get
$$
\| \widehat {⑥} \| _ {2} \lesssim \lambda^ {\star} \mu \sqrt {\frac {\log n}{n p}} \frac {\beta_ {0}}{\sqrt {n}}, \quad \| \widehat {⑦} \| _ {2} \lesssim \lambda^ {\star} \mu \sqrt {\frac {\log^ {2} n}{n p}} \frac {\beta_ {0}}{\sqrt {n}},
$$
There exists a universal constant  $c_4 > 0$  such that
$$
\eta \left(2 \big \| \big | ① \big | \big | _ {2} + \big \| \big | ② \big | \big | _ {2} + \big \| \big | ⑤ \big | \big | _ {2}\right) \leq \frac {c _ {4}}{\log^ {2} n} \Big \| \boldsymbol {x} - \boldsymbol {x} ^ {(l)} \Big \| _ {2}
$$
if  $n^2 p \gtrsim \mu^2 n\log^5 n$ , and there exists a universal constant  $c_5 > 0$  such that