SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \left\| \mathbf {u} _ {t + 1} ^ {j} (r) - \mathbf {u} _ {0} ^ {j} (r) \right\| \\ = \left\| \sum_ {t ^ {\prime} = 0} ^ {t} \eta \mathbf {g} _ {t ^ {\prime}} ^ {j} (r) \right\| = \eta \left\| \sum_ {t ^ {\prime} = 0} ^ {t} \frac {2}{n} \sum_ {i = 1} ^ {n} a _ {j} (r) \mathbb {I} \left\{\mathbf {u} _ {t} ^ {j} (r) ^ {\top} \boldsymbol {\alpha} _ {i} \right\} \boldsymbol {\alpha} _ {i} \left(h _ {t} ^ {j} (\boldsymbol {\alpha} _ {i}) - w _ {t} ^ {i} (j)\right) \right\| \\ \leq \frac {2 \eta}{\sqrt {m}} \sum_ {t ^ {\prime} = 0} ^ {t} \frac {1}{n} \sum_ {i = 1} ^ {n} \left(\left| h _ {t} ^ {j} (\boldsymbol {\alpha} _ {i}) - w _ {*} ^ {i} (j) \right| + \left| w _ {*} ^ {i} (j) - w _ {t} ^ {i} (j) \right|\right) \\ \leq \frac {2 \eta}{\sqrt {m}} \sum_ {t ^ {\prime} = 0} ^ {t} \left(\sqrt {\frac {1}{n} \sum_ {i = 1} ^ {n} \left(h _ {t} ^ {j} (\boldsymbol {\alpha} _ {i}) - w _ {*} ^ {i} (j)\right) ^ {2}} + \sqrt {\frac {1}{n} \sum_ {i = 1} ^ {n} \left(w _ {*} ^ {i} (j) - w _ {t} ^ {i} (j)\right) ^ {2}}\right) \\ \leq \frac {2 \eta}{\sqrt {m}} \sum_ {t ^ {\prime} = 0} ^ {t} \left(\sqrt {\hat {\mathcal {R}} ^ {j} \left(\boldsymbol {\theta} _ {t ^ {\prime}} ^ {j}\right)} + \sqrt {(1 - \mu \gamma) ^ {K t ^ {\prime}} D}\right) \\ \leq 2 \eta \sum_ {t ^ {\prime} = 0} ^ {t} \left(\frac {\sqrt {\left(1 - \frac {\eta \lambda_ {0}}{4 n}\right) ^ {t ^ {\prime} - 1} \widehat {\mathcal {R}} ^ {j} \left(\boldsymbol {\theta} _ {0} ^ {j}\right) + C \frac {4 n (1 - \mu \gamma) ^ {K} D}{\eta \lambda_ {0}}}}{\sqrt {m}} + \frac {1}{\sqrt {m}} \sqrt {(1 - \mu \gamma) ^ {K t ^ {\prime}} D}\right) \\ \leq 2 \eta \left(\frac {\frac {8 n}{\eta \lambda_ {0}} \sqrt {\widehat {\mathcal {R}} (\boldsymbol {\theta} _ {0})} + t \sqrt {C \frac {4 n (1 - \mu \gamma) ^ {K} D}{\eta \lambda_ {0}}}}{\sqrt {m}} + \frac {1}{\sqrt {m} \mu \gamma} \sqrt {(1 - \mu \gamma) ^ {K} D}\right), \\ \end{array}
$$
where at last step we use the fact  $\sqrt{1 - a} \leq 1 - \frac{a}{2}$  for  $a \in (0,1)$ , and triangle inequality to split terms inside square root. Since we choose  $\gamma = \frac{1}{L}$ , we have
$$
\left\| \mathbf {u} _ {t + 1} (r) - \mathbf {u} _ {t} (r) \right\| \leq 2 \eta \left(\frac {8 n B _ {y}}{\eta \lambda_ {0} \sqrt {m}} + t \sqrt {C \frac {4 n \left(1 - \frac {1}{\kappa}\right) ^ {K} D}{\eta \lambda_ {0} m}} + \frac {2 \kappa \sqrt {\left(1 - \frac {1}{\kappa}\right) ^ {K} D}}{\sqrt {m}}\right)
$$
According to our choice of  $K$ :
$$
K \geq \max  \left\{\kappa \log \left(\frac {\eta \lambda_ {0} n T ^ {2} C D}{1 6 B _ {y} ^ {2} n}\right), 2 \kappa \log \left(\frac {\eta \lambda_ {0} \kappa \sqrt {D}}{4 B _ {y} n}\right) \right\},
$$
$$
t \sqrt {C \frac {4 n \left(1 - \frac {1}{\kappa}\right) ^ {K} D}{\eta \lambda_ {0} m}} \leq \frac {8 n B _ {y}}{\eta \lambda_ {0} \sqrt {m}},
$$
$$
\frac {2 \kappa \sqrt {(1 - \frac {1}{\kappa}) ^ {K} D}}{\sqrt {m}} \leq \frac {8 n B _ {y}}{\eta \lambda_ {0} \sqrt {m}}.
$$
Hence we conclude that:
$$
\left\| \mathbf {u} _ {t + 1} (r) - \mathbf {u} _ {t} (r) \right\| \leq \frac {4 8 n B _ {y}}{\lambda_ {0} \sqrt {m}}.
$$
Now we switch to proving hypothesis (I). According to updating rule, we have
$$
\begin{array}{l} \boldsymbol {\theta} _ {t + 1} ^ {j} = \boldsymbol {\theta} _ {t} ^ {j} - \eta \mathbf {g} _ {t} ^ {j} \\ = \boldsymbol {\theta} _ {t} ^ {j} - \eta \frac {2}{n} \boldsymbol {\Phi} _ {t} ^ {j} \left(\mathbf {h} _ {t} ^ {j} - \mathbf {w} ^ {t} (j)\right) \\ \end{array}
$$
where  $\mathbf{w}^t (j) = [w_1^t (j),\dots ,w_n^t (j)]$  . Hence
$$
\begin{array}{l} \widehat {\mathcal {R}} ^ {j} \left(\boldsymbol {\theta} _ {t + 1} ^ {j}\right) = \frac {1}{n} \left\| \boldsymbol {\Phi} _ {t + 1} ^ {j} ^ {\top} \boldsymbol {\theta} _ {t + 1} ^ {j} - \mathbf {w} ^ {*} (j) \right\| ^ {2} \\ = \frac {1}{n} \left\| \boldsymbol {\Phi} _ {t + 1} ^ {j} ^ {\top} \boldsymbol {\theta} _ {t + 1} ^ {j} - \boldsymbol {\Phi} _ {t} ^ {j} ^ {\top} \boldsymbol {\theta} _ {t + 1} ^ {j} + \boldsymbol {\Phi} _ {t} ^ {j} ^ {\top} \boldsymbol {\theta} _ {t + 1} ^ {j} - \mathbf {w} ^ {*} (j) \right\| ^ {2} \\ \leq \inf  _ {p > 0} \left(1 + \frac {1}{p}\right) \underbrace {\frac {1}{n} \left\| \boldsymbol {\Phi} _ {t + 1} ^ {j} {} ^ {\top} \boldsymbol {\theta} _ {t + 1} ^ {j} - \boldsymbol {\Phi} _ {t} ^ {j} {} ^ {\top} \boldsymbol {\theta} _ {t + 1} ^ {j} \right\| ^ {2}} _ {T _ {1}} + (1 + p) \underbrace {\frac {1}{n} \left\| \boldsymbol {\Phi} _ {t} ^ {j} {} ^ {\top} \boldsymbol {\theta} _ {t + 1} ^ {j} - \mathbf {w} ^ {*} (j) \right\| ^ {2}} _ {T _ {2}} \tag {27} \\ \end{array}
$$
where  $\mathbf{w}^{*}(j) = [w^{*}(\pmb{\alpha}_{1})(j),\dots,w^{*}(\pmb{\alpha}_{n})(j)]$
For  $T_{1}$ :
$$
T _ {1} = \frac {1}{n} \left\| \boldsymbol {\Phi} _ {t + 1} ^ {j} ^ {\top} \boldsymbol {\theta} _ {t + 1} ^ {j} - \boldsymbol {\Phi} _ {t} ^ {j} ^ {\top} \boldsymbol {\theta} _ {t + 1} ^ {j} \right\| ^ {2} = \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| \sum_ {r = 1} ^ {m} a _ {j} (r) (\mathbb {I} \{\mathbf {u} _ {t + 1} ^ {j} ^ {\top} (r) \boldsymbol {\alpha} _ {i} \} - \mathbb {I} \{\mathbf {u} _ {t} ^ {j} ^ {\top} (r) \boldsymbol {\alpha} _ {i} \}) \mathbf {u} _ {t} ^ {j} ^ {\top} (r) \boldsymbol {\alpha} _ {i} \right\| ^ {2}
$$
Notice that:
$$
\begin{array}{l} \left| \sum_ {r = 1} ^ {m} a _ {j} (r) \left(\mathbb {I} \left\{\mathbf {u} _ {t + 1} ^ {j} (r) ^ {\top} \boldsymbol {\alpha} _ {i} \right\} - \mathbb {I} \left\{\mathbf {u} _ {t} ^ {j} (r) ^ {\top} \boldsymbol {\alpha} _ {i} \right\}\right) \mathbf {u} _ {t} ^ {\top} (r) \boldsymbol {\alpha} _ {i} \right| \\ \leq \frac {1}{\sqrt {m}} \sum_ {r = 1} ^ {m} \left| \left(\mathbb {I} \left\{\mathbf {u} _ {t + 1} ^ {j} (r) ^ {\top} \boldsymbol {\alpha} _ {i} \right\} - \mathbb {I} \left\{\mathbf {u} _ {t} ^ {j} (r) ^ {\top} \boldsymbol {\alpha} _ {i} \right\}\right) \mathbf {u} _ {t} ^ {\top} (r) \boldsymbol {\alpha} _ {i} \right| \\ \leq \frac {1}{\sqrt {m}} \sum_ {r = 1} ^ {m} \left| \left(\mathbb {I} \left\{\mathbf {u} _ {t + 1} ^ {j} (r) ^ {\top} \boldsymbol {\alpha} _ {i} \right\} - \mathbb {I} \left\{\mathbf {u} _ {t} ^ {j} (r) ^ {\top} \boldsymbol {\alpha} _ {i} \right\}\right) \right| \left\| \mathbf {u} _ {t} ^ {j} ^ {\top} (r) \right\| \\ \leq \frac {1}{\sqrt {m}} \left(\left| P \left(\boldsymbol {\theta} _ {t + 1}, \boldsymbol {\alpha} _ {i}\right) \right| + \left| P \left(\boldsymbol {\theta} _ {t}, \boldsymbol {\alpha} _ {i}\right) \right|\right) \left\| \mathbf {u} _ {t} ^ {\top} (r) - \mathbf {u} _ {t + 1} ^ {\top} (r) \right\|. \\ \end{array}
$$
According to updating rule, we have:
$$
\begin{array}{l} \left\| \mathbf {u} _ {t} ^ {j} ^ {\top} (r) - \mathbf {u} _ {t + 1} ^ {j} ^ {\top} (r) \right\| = \left\| \eta \mathbf {g} _ {t} ^ {j} (r) \right\| \\ \leq \eta \left\| \frac {2}{n} \sum_ {i = 1} ^ {n} \mathbb {I} \left\{\mathbf {u} _ {t} ^ {j} ^ {\top} (r) \boldsymbol {\alpha} _ {i} \right\} \boldsymbol {\alpha} _ {i} ^ {\top} a _ {j} (r) \left(h _ {t} ^ {j} (\boldsymbol {\alpha} _ {i}) - w _ {i} ^ {t} (j)\right) \right\| \\ + \eta \left\| \frac {2}{n} \sum_ {i = 1} ^ {n} \mathbb {I} \left\{\mathbf {u} _ {t} ^ {j} (r) ^ {\top} \boldsymbol {\alpha} _ {i} \right\} \boldsymbol {\alpha} _ {i} ^ {\top} a _ {j} (r) \left(w ^ {*} (\boldsymbol {\alpha} _ {i}) (j) - w _ {i} ^ {t} (j)\right) \right\| \\ \leq \frac {2 \eta \sqrt {\hat {\mathcal {R}} ^ {j} \left(\boldsymbol {\theta} _ {t} ^ {j}\right)}}{\sqrt {m}} + \frac {2 \eta}{\sqrt {m}} \sqrt {(1 - \mu \gamma) ^ {K t} D} \\ \end{array}
$$
Since we prove that  $\max_{r\in [m]}\left\| \mathbf{u}_{t + 1}^j (r) - \mathbf{u}_t^j (r)\right\| \leq \rho$  , we have:
$$
\left| P \left(\boldsymbol {\theta} ^ {t + 1}, \boldsymbol {\alpha} _ {i}\right) \right| \leq m \rho + \sqrt {m \nu}.
$$
Putting pieces together yields:
$$
\begin{array}{l} T _ {1} \leq \frac {4}{m} \left(m \rho + \sqrt {m \nu}\right) ^ {2} \left(\frac {2 \eta \sqrt {\widehat {\mathcal {R}} ^ {j} \left(\boldsymbol {\theta} _ {t} ^ {j}\right)}}{\sqrt {m}} + \frac {2 \eta}{\sqrt {m}} \sqrt {(1 - \mu \gamma) ^ {K t} D}\right) ^ {2} \\ \leq 4 \left(\sqrt {m} \rho + \sqrt {\nu}\right) ^ {2} \left(\frac {4 \eta^ {2} \widehat {\mathcal {R}} (\boldsymbol {\theta} ^ {t})}{m} + \frac {4 \eta^ {2}}{m} (1 - \mu \gamma) ^ {K t} D\right). \tag {28} \\ \end{array}
$$
For  $T_{2}$
$$
\begin{array}{l} \frac {1}{n} \left\| \boldsymbol {\Phi} _ {t} ^ {j} ^ {\top} \boldsymbol {\theta} _ {t + 1} ^ {j} - \mathbf {w} ^ {*} (j) \right\| ^ {2} \\ = \frac {1}{n} \left\| \boldsymbol {\Phi} _ {t} ^ {j} ^ {\top} (\boldsymbol {\theta} _ {t} ^ {j} - \eta \mathbf {g} _ {t} ^ {j}) - \mathbf {w} ^ {*} (j) \right\| ^ {2} \\ = \frac {1}{n} \left\| \boldsymbol {\Phi} _ {t} ^ {j} ^ {\top} (\boldsymbol {\theta} _ {t} ^ {j} - \frac {2}{n} \eta \boldsymbol {\Phi} _ {t} ^ {j} (\mathbf {h} _ {t} ^ {j} - \mathbf {w} ^ {*} (j))) - \mathbf {w} ^ {*} (j) - \boldsymbol {\Phi} _ {t} ^ {j} ^ {\top} (\frac {2}{n} \eta \boldsymbol {\Phi} _ {t} ^ {j} (\mathbf {w} ^ {t} (j) - \mathbf {w} ^ {*} (j))) \right\| ^ {2} \\ \leq \inf  _ {q > 0} \left(1 + \frac {1}{q}\right) \frac {1}{n} \left\| \left(\mathbf {I} - \frac {2}{n} \eta \boldsymbol {\Phi} _ {t} ^ {j} ^ {\top} \boldsymbol {\Phi} _ {t}\right) \left(\mathbf {h} _ {t} ^ {j} - \mathbf {w} ^ {*} (j)\right) \right\| ^ {2} + (1 + q) \eta^ {2} \frac {1}{n} \left\| \frac {2}{n} \boldsymbol {\Phi} _ {t} ^ {j} ^ {\top} \boldsymbol {\Phi} _ {t} \left(\mathbf {w} ^ {t} (j) - \mathbf {w} ^ {*} (j)\right) \right\| ^ {2} \\ \leq \inf  _ {q > 0} \left(1 + \frac {1}{q}\right) \left(1 - \frac {2 \eta \lambda_ {\operatorname* {m i n}} (\hat {\mathbf {K}} _ {t})}{n}\right) ^ {2} \widehat {\mathcal {R}} ^ {j} (\boldsymbol {\theta} _ {t} ^ {j}) + (1 + q) \frac {4 \eta^ {2}}{n ^ {3}} \left\| \boldsymbol {\Phi} _ {t} ^ {j} ^ {\top} \boldsymbol {\Phi} _ {t} ^ {j} (\mathbf {w} ^ {t} (j) - \mathbf {w} ^ {*} (j)) \right\| ^ {2} \\ \end{array}
$$
We choose  $q = \frac{n}{2\eta\lambda_{\min}(\hat{\mathbf{K}}_t)}$ , and the fact that  $(1 + a)(1 - a)^2 \leq (1 - a)$ :
$$
\begin{array}{l} T _ {2} = \frac {1}{n} \left\| \boldsymbol {\Phi} _ {t} ^ {j} ^ {\top} \boldsymbol {\theta} _ {t + 1} ^ {j} - \mathbf {w} ^ {*} (j) \right\| ^ {2} \\ \leq \left(1 - \frac {2 \eta \lambda_ {\operatorname* {m i n}} (\hat {\mathbf {K}} _ {t})}{n}\right) \widehat {\mathcal {R}} ^ {j} (\boldsymbol {\theta} _ {t} ^ {j}) + \left(1 + \frac {n}{2 \eta \lambda_ {\operatorname* {m i n}} (\hat {\mathbf {K}} _ {t})}\right) \frac {4 \eta^ {2}}{n ^ {3}} \left\| \boldsymbol {\Phi} _ {t} ^ {j} ^ {\top} \boldsymbol {\Phi} _ {t} ^ {j} (\mathbf {w} ^ {t} (j) - \mathbf {w} ^ {*} (j)) \right\| ^ {2} \\ \leq \left(1 - \frac {2 \eta \lambda_ {\operatorname* {m i n}} (\hat {\mathbf {K}} _ {t})}{n}\right) \widehat {\mathcal {R}} ^ {j} \left(\boldsymbol {\theta} _ {t} ^ {j}\right) \\ + \left(1 + \frac {n}{2 \eta \lambda_ {\mathrm {m i n}} (\hat {\mathbf {K}} _ {t})}\right) \frac {4 \eta^ {2}}{n ^ {3}} \left(\left\| \hat {\mathbf {K}} _ {0} \right\| + 2 n ^ {2} (\rho + \sqrt {\frac {\nu}{m}})\right) ^ {2} (1 - \mu \gamma) ^ {K t} D \\ \end{array}
$$
Since we choose  $\eta \leq \frac{n}{2\lambda_{\min}(\hat{\mathbf{K}}_t)}$ , we can re-write the above inequality as:
$$
\begin{array}{l} T _ {2} \leq \left(1 - \frac {2 \eta \lambda_ {\operatorname* {m i n}} (\hat {\mathbf {K}} _ {t} ^ {j})}{n}\right) \widehat {\mathcal {R}} ^ {j} (\boldsymbol {\theta} _ {t} ^ {j}) + \frac {4 \eta}{n ^ {2} \lambda_ {\operatorname* {m i n}} (\hat {\mathbf {K}} _ {t})} \left(\left\| \hat {\mathbf {K}} _ {0} \right\| + 2 n ^ {2} (\rho + \sqrt {\frac {\nu}{m}})\right) ^ {2} (1 - \mu \gamma) ^ {K t} D \\ \leq \left(1 - \frac {2 \eta \lambda_ {\operatorname* {m i n}} (\hat {\mathbf {K}} _ {0})}{n} + 4 n \eta \left(\rho + \sqrt {\frac {\nu}{m}}\right)\right) \widehat {\mathcal {R}} ^ {j} \left(\boldsymbol {\theta} _ {t} ^ {j}\right) + \frac {4 \eta \left(\left\| \hat {\mathbf {K}} _ {0} \right\| + 2 n ^ {2} (\rho + \sqrt {\frac {\nu}{m}})\right) ^ {2} (1 - \mu \gamma) ^ {K t} D}{n ^ {2} \left(\lambda_ {\operatorname* {m i n}} (\hat {\mathbf {K}} _ {0}) - 2 n ^ {2} (\rho + \sqrt {\frac {\nu}{m}})\right)} \\ \end{array}
$$
Plugging  $T_{1}$  and  $T_{2}$  back to (27), and using the eigenvalue lower bound from Lemma 10 (d) yields:
$$
\begin{array}{l} \widehat {\mathcal {R}} ^ {j} \left(\boldsymbol {\theta} _ {t + 1} ^ {j}\right) \leq \inf  _ {p > 0} (1 + p) 4 (\sqrt {m} \rho + \sqrt {\nu}) ^ {2} \left(\frac {4 \eta^ {2} \widehat {\mathcal {R}} \left(\boldsymbol {\theta} ^ {t}\right)}{m} + \frac {4 \eta^ {2}}{m} (1 - \mu \gamma) ^ {K t} D\right) \\ + \left(1 + \frac {1}{p}\right) \left[ \left(1 - \frac {\eta \lambda_ {0}}{n} + 4 n \eta \left(\rho + \sqrt {\frac {\nu}{m}}\right)\right) \widehat {\mathcal {R}} ^ {j} \left(\boldsymbol {\theta} _ {t} ^ {j}\right) \right. \\ \left. + \frac {4 \eta \left(\left\| \hat {\mathbf {K}} _ {0} \right\| + 2 n ^ {2} (\rho + \sqrt {\frac {\nu}{m}})\right) ^ {2} (1 - \mu \gamma) ^ {K t} D}{n ^ {2} \left(\frac {1}{2} \lambda_ {0} - 2 n ^ {2} (\rho + \sqrt {\frac {\nu}{m}})\right)} \right]. \\ \end{array}
$$
Now we plug in  $\rho = \frac{48nB_y}{\lambda_0\sqrt{m}}$
$$
\begin{array}{l} \widehat {\mathcal {R}} ^ {j} (\pmb {\theta} _ {t + 1} ^ {j}) \leq \inf _ {p > 0} (1 + p) 4 \left(\sqrt {m} \frac {4 8 n B _ {y}}{\lambda_ {0} \sqrt {m}} + \sqrt {\nu}\right) ^ {2} \left(\frac {4 \eta^ {2} \widehat {\mathcal {R}} ^ {j} (\pmb {\theta} _ {t} ^ {j})}{m} + \frac {4 \eta^ {2}}{m} (1 - \mu \gamma) ^ {K t} D\right) \\ + \left(1 + \frac {1}{p}\right) \left[ \left(1 - \frac {\eta \lambda_ {0}}{n} + 4 n \eta \left(\frac {4 8 n B _ {y}}{\lambda_ {0} \sqrt {m}} + \sqrt {\frac {\nu}{m}}\right)\right) \widehat {\mathcal {R}} (\boldsymbol {\theta} ^ {t}) \right. \\ \left. + \frac {4 \eta \left(\left\| \hat {\mathbf {K}} _ {0} \right\| + 2 n ^ {2} \left(\frac {4 8 n B _ {y}}{\lambda_ {0} \sqrt {m}} + \sqrt {\frac {\nu}{m}}\right)\right) ^ {2} (1 - \mu \gamma) ^ {K t} D}{n ^ {2} \left(\frac {1}{2} \lambda_ {0} - 2 n ^ {2} \left(\frac {4 8 n B _ {y}}{\lambda_ {0} \sqrt {m}} + \sqrt {\frac {\nu}{m}}\right)\right)} \right]. \tag {29} \\ \end{array}
$$
According to our choice of  $m$ :
$$
m \geq \left(\frac {4 8 n B _ {y}}{\lambda_ {0}} + \sqrt {\nu}\right) ^ {2} \frac {2 5 6 n ^ {4}}{\lambda_ {0} ^ {2}}
$$
we know that
$$
4 n \eta \left(\frac {4 8 n B _ {y}}{\lambda_ {0} \sqrt {m}} + \sqrt {\frac {\nu}{m}}\right) \leq \frac {\eta \lambda_ {0}}{4 n},
$$
$$
4 \left(\sqrt {m} \frac {4 8 n B _ {y}}{\lambda_ {0} \sqrt {m}} + \sqrt {\nu}\right) ^ {2} \frac {4 \eta^ {2} \widehat {\mathcal {R}} (\boldsymbol {\theta} ^ {t})}{m} \leq \frac {\eta^ {2} \lambda_ {0} ^ {2}}{1 6 n ^ {4}} \widehat {\mathcal {R}} (\boldsymbol {\theta} ^ {t}),
$$
$$
2 n ^ {2} \left(\frac {4 8 n B _ {y}}{\lambda_ {0} \sqrt {m}} + \sqrt {\frac {\nu}{m}}\right) \leq \frac {\lambda_ {0}}{8}.
$$
Plugging above inequality back to Eq.(29) yields:
$$
\begin{array}{l} \widehat {\mathcal {R}} \left(\boldsymbol {\theta} ^ {t + 1}\right) \leq \inf  _ {p > 0} (1 + p) \left(4 \left(\sqrt {m} \frac {4 8 n B _ {y}}{\lambda_ {0} \sqrt {m}} + \sqrt {\nu}\right) ^ {2} \left(\frac {4 \eta^ {2}}{m} (1 - \mu \gamma) ^ {K t} D\right) + \frac {\eta^ {2} \lambda_ {0} ^ {2}}{1 6 n ^ {4}} \widehat {\mathcal {R}} \left(\boldsymbol {\theta} ^ {t}\right)\right) \\ + \left(1 + \frac {1}{p}\right) \left(1 - \frac {3 \eta \lambda_ {0}}{4 n}\right) \widehat {\mathcal {R}} (\boldsymbol {\theta} ^ {t}) + \left(1 + \frac {1}{p}\right) \frac {4 \eta \left(\left\| \hat {\mathbf {K}} _ {0} \right\| + \frac {\lambda_ {0}}{8}\right) ^ {2} (1 - \mu \gamma) ^ {K t} D}{\frac {3}{8} \lambda_ {0} n ^ {2}}. \\ \end{array}
$$
According to the fact that:
$$
\left(1 - \frac {1}{a}\right) \left(1 + \frac {1}{2 a - 1}\right) = \frac {2 a - 2}{2 a - 1} \leq 1 - \frac {1}{2 a}
$$
we choose  $p = \frac{8n}{3\eta\lambda_0} - 1$ , and it yields:
$$
\begin{array}{l} \widehat {\mathcal {R}} \left(\boldsymbol {\theta} ^ {t + 1}\right) \leq \frac {8 n}{3 \eta \lambda_ {0}} \left(4 \left(\frac {4 8 n B _ {y}}{\lambda_ {0}} + \sqrt {\nu}\right) ^ {2} \left(\frac {4 \eta^ {2}}{m} (1 - \mu \gamma) ^ {K t} D\right) + \frac {\eta^ {2} \lambda_ {0} ^ {2}}{1 6 n ^ {4}} \widehat {\mathcal {R}} \left(\boldsymbol {\theta} ^ {t}\right)\right) \\ + \left(1 - \frac {3 \eta \lambda_ {0}}{8 n}\right) \widehat {\mathcal {R}} (\pmb {\theta} ^ {t}) + \left(1 + \frac {1}{\frac {8 n}{3 \eta \lambda_ {0}} - 1}\right) \frac {4 \eta \left(\left\| \hat {\mathbf {K}} _ {0} \right\| + \frac {\lambda_ {0}}{8}\right) ^ {2} (1 - \mu \gamma) ^ {K t} D}{\frac {3}{8} \lambda_ {0} n ^ {2}}. \\ \end{array}
$$
Using the fact  $\eta \leq \frac{n}{\lambda_0}$  yields:
$$
\begin{array}{l} \widehat {\mathcal {R}} (\pmb {\theta} ^ {t + 1}) \leq \frac {8 n}{3 \eta \lambda_ {0}} \left(4 \left(\frac {4 8 d n B _ {y}}{\lambda_ {0}} + \sqrt {\nu}\right) ^ {2} \left(\frac {4 \eta^ {2}}{m} (1 - \mu \gamma) ^ {K t} D\right)\right) \\ + \left(1 - \frac {3 \eta \lambda_ {0}}{8 n} + \frac {\eta \lambda_ {0}}{6 n ^ {3}}\right) \widehat {\mathcal {R}} (\pmb {\theta} ^ {t}) + \frac {8 \eta \left(\left\| \hat {\mathbf {K}} _ {0} \right\| + \frac {\lambda_ {0}}{8}\right) ^ {2} (1 - \mu \gamma) ^ {K t} D}{\frac {3}{8} \lambda_ {0} n ^ {2}} \\ \leq \left(\frac {4 8 n B _ {y}}{\lambda_ {0}} + \sqrt {\nu}\right) ^ {2} \left(\frac {1 2 8 n \eta}{3 \lambda_ {0} m} (1 - \mu \gamma) ^ {K t} D\right) \\ + \left(1 - \frac {3 \eta \lambda_ {0}}{8 n} + \frac {\eta \lambda_ {0}}{6 n ^ {3}}\right) \widehat {\mathcal {R}} (\boldsymbol {\theta} ^ {t}) + \frac {8 \eta \left(\left\| \hat {\mathbf {K}} _ {0} \right\| + \frac {\lambda_ {0}}{8}\right) ^ {2} (1 - \mu \gamma) ^ {K t} D}{\frac {3}{8} \lambda_ {0} n ^ {2}}. \\ \end{array}
$$
Since  $n^2 \geq \frac{4}{3}$ , we know  $\frac{\eta\lambda_0}{6n^3} \leq \frac{\eta\lambda_0}{8n}$ . Hence we have:
$$
\begin{array}{l} \widehat {\mathcal {R}} \left(\boldsymbol {\theta} ^ {t + 1}\right) \\ \leq \left(\frac {4 8 d n B _ {y}}{\lambda_ {0}} + \sqrt {\nu}\right) ^ {2} \left(\frac {1 2 8 n \eta}{3 \lambda_ {0} m} (1 - \mu \gamma) ^ {K t} D\right) \\ + \left(1 - \frac {\eta \lambda_ {0}}{4 n}\right) \widehat {\mathcal {R}} (\boldsymbol {\theta} ^ {t}) + \frac {8 \eta \left(\frac {1}{2} \| \mathbf {K} \| + \frac {\lambda_ {0}}{8}\right) ^ {2} (1 - \mu \gamma) ^ {K t} D}{\frac {3}{8} \lambda_ {0} n ^ {2}} \\ = \left(1 - \frac {\eta \lambda_ {0}}{4 n}\right) \widehat {\mathcal {R}} (\boldsymbol {\theta} ^ {t}) + \frac {4 n}{\eta \lambda_ {0}} \left(\left(\frac {4 8 n B _ {y}}{\lambda_ {0}} + \sqrt {\nu}\right) ^ {2} \frac {3 2 \eta^ {2}}{3 m} + \frac {2 \eta^ {2} \left(\frac {1}{2} \| \mathbf {K} \| + \frac {\lambda_ {0}}{8}\right) ^ {2}}{\frac {3}{8} n ^ {2}}\right) (1 - \mu \gamma) ^ {K t} D. \\ \end{array}
$$
Plugging lower bound of  $m \geq \left(\frac{48nB_y}{\lambda_0} + \sqrt{\nu}\right)^2 \frac{256n^4}{\lambda_0^2}$  yields:
$$
\widehat {\mathcal {R}} (\pmb {\theta} ^ {t + 1}) \leq \left(1 - \frac {\eta \lambda_ {0}}{4 n}\right) \widehat {\mathcal {R}} (\pmb {\theta} ^ {t}) + \frac {4 n}{\eta \lambda_ {0}} \left(\frac {\lambda_ {0} ^ {2} \eta^ {2}}{2 4 n ^ {4}} + \frac {2 \eta^ {2} \left(\frac {1}{2} \| \mathbf {K} \| + \frac {\lambda_ {0}}{8}\right) ^ {2}}{\frac {3}{8} n ^ {2}}\right) (1 - \mu \gamma) ^ {K t} D.
$$
Finally, according to inductive hypothesis (I), we plug in the convergence rate for  $\widehat{\mathcal{R}} (\pmb {\theta}^t)$  and get
$$
\begin{array}{l} \widehat {\mathcal {R}} (\boldsymbol {\theta} ^ {t + 1}) \leq \left(1 - \frac {\eta \lambda_ {0}}{4 n}\right) ^ {t} \widehat {\mathcal {R}} (\boldsymbol {\theta} _ {0}) + \left(1 - \frac {\eta \lambda_ {0}}{4 n}\right) \sum_ {s = 0} ^ {t - 1} \left(1 - \frac {\eta \lambda_ {0}}{4 n}\right) ^ {t - 1 - s} (1 - \mu \gamma) ^ {s} C (1 - \mu \gamma) ^ {K} D \\ + \frac {4 n}{\eta \lambda_ {0}} \left(\frac {\lambda_ {0} ^ {2} \eta^ {2}}{2 4 n ^ {4}} + \frac {2 \eta^ {2} \left(\frac {1}{2} \| \mathbf {K} \| + \frac {\lambda_ {0}}{8}\right) ^ {2}}{\frac {3}{8} n ^ {2}}\right) (1 - \mu \gamma) ^ {K t} D \\ \leq \left(1 - \frac {\eta \lambda_ {0}}{4 n}\right) ^ {t} \widehat {\mathcal {R}} (\boldsymbol {\theta} _ {0}) + C \sum_ {s = 0} ^ {t - 1} \left(1 - \frac {\eta \lambda_ {0}}{4 n}\right) ^ {t - s} (1 - \mu \gamma) ^ {s} (1 - \mu \gamma) ^ {K} D \\ + \frac {4 n}{\eta \lambda_ {0}} \left(\frac {\lambda_ {0} ^ {2} \eta^ {2}}{2 4 n ^ {4}} + \frac {2 \eta^ {2} \left(\frac {1}{2} \| \mathbf {K} \| + \frac {\lambda_ {0}}{8}\right) ^ {2}}{\frac {3}{8} n ^ {2}}\right) \left(1 - \frac {\eta \lambda_ {0}}{4 n}\right) ^ {t - t} (1 - \mu \gamma) ^ {t} (1 - \mu \gamma) ^ {K} D \\ \end{array}
$$
Now we just need to ensure that
$$
\frac {\lambda_ {0} ^ {2} \eta^ {2}}{2 4 n ^ {4}} + \frac {2 \eta^ {2} \left(\frac {1}{2} \left\| \mathbf {K} \right\| + \frac {\lambda_ {0}}{8}\right) ^ {2}}{\frac {3}{8} n ^ {2}} \leq C.
$$
We hence choose  $C$  as:
$$
C = \frac {\lambda_ {0} ^ {2} \eta^ {2}}{2 4 n ^ {4}} + \frac {4 \eta^ {2} \left(\frac {1}{2} \left\| \mathbf {K} \right\| + \frac {\lambda_ {0}}{8}\right) ^ {2}}{\frac {3}{8} n ^ {2}}
$$
Hence we proved that:
$$
\begin{array}{l} \widehat {\mathcal {R}} \left(\boldsymbol {\theta} ^ {t + 1}\right) \leq \left(1 - \frac {\eta \lambda_ {0}}{4 n}\right) ^ {t} \widehat {\mathcal {R}} \left(\boldsymbol {\theta} _ {0}\right) + C \sum_ {s = 0} ^ {t} \left(1 - \frac {\eta \lambda_ {0}}{4 n}\right) ^ {t - s} (1 - \mu \gamma) ^ {s} (1 - \mu \gamma) ^ {K} D \\ \leq \left(1 - \frac {\eta \lambda_ {0}}{4 n}\right) ^ {t} \widehat {\mathcal {R}} (\boldsymbol {\theta} _ {0}) + \frac {4 n C (1 - \mu \gamma) ^ {K} D}{\eta \lambda_ {0}}. \\ \end{array}
$$
$$
C = \frac {\lambda_ {0} ^ {2} \eta^ {2}}{2 4 n ^ {4}} + \frac {4 \eta^ {2} \left(\frac {1}{2} \| \mathbf {K} \| + \frac {\lambda_ {0}}{8}\right) ^ {2}}{\frac {3}{8} n ^ {2}}.
$$
B.4 Relations Between Different Predictors
In this section, we are going to relate the predictions of neural network, random feature predictor and kernel OLS predictor.
Lemma 12 (Neural Network Predictor and Random Feature Predictor Coupling). Let  $\tilde{n} := \left(\frac{48nB_y}{\lambda_0\sqrt{m}} + \sqrt{\frac{\nu}{m}}\right)$ . For Algorithm 2, the following statement holds for any  $j \in [d]$  and  $t \in [T]$ :
$$
\begin{array}{l} \left\| \mathbf {h} _ {t + 1} ^ {j} - \mathbf {h} _ {t + 1} ^ {\mathrm {r f}, j} \right\| \\ \leq \left(1 - \frac {\eta \lambda_ {0}}{4 n}\right) ^ {\frac {t}{2}} \frac {1 6}{\lambda_ {0}} \left(\left(\frac {3}{2} \| \mathbf {K} \| + 2 n ^ {2} \tilde {n} + 2 \sqrt {n} n \tilde {n}\right) \sqrt {(1 - \mu \gamma) ^ {K} D} + \left(2 \sqrt {n} n ^ {2} \tilde {n} + 2 \sqrt {n} n \tilde {n}\right) \sqrt {\hat {\mathcal {R}} (\boldsymbol {\theta} _ {0})}\right) \\ + \left(2 \sqrt {n} \tilde {n} + 4 \sqrt {n} n \tilde {n}\right) \frac {8 n}{\lambda_ {0}} \sqrt {\frac {4 n C (1 - \mu \gamma) ^ {K} D}{\eta \lambda_ {0}}}. \\ \end{array}
$$
Proof. Notice the following canonical decomposition: