SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Neural Tangent Feature (NTF) predictor For parameter vector  $\bar{\theta}_t^j\in \mathbb{R}^{Nd}$  and  $\bar{\theta}_t = (\bar{\theta}_t^1,\dots ,\bar{\theta}_t^d)$  (the procedure for obtaining them is discussed later), we have the following NTF predictor defined as
$$
\mathbf {h} _ {t} ^ {\mathrm {r f}} = \left[ \begin{array}{c} (\phi_ {0} ^ {1} (\boldsymbol {\alpha})) ^ {\top} \bar {\boldsymbol {\theta}} _ {t} ^ {1} \\ \vdots \\ (\phi_ {0} ^ {d} (\boldsymbol {\alpha})) ^ {\top} \bar {\boldsymbol {\theta}} _ {t} ^ {d} \end{array} \right]  .
$$
We also use  $h_t^{\mathrm{rf},j}(\alpha)$  to indicate the  $j$ th coordinate of  $\mathbf{h}_t^{\mathrm{rf}}(\alpha)$ . Then, the objective of the NTF predictor is
$$
\widehat {\mathcal {R}} ^ {\mathrm {r f}} (\pmb {\theta} _ {t}) = \frac {1}{n} \sum_ {i = 1} ^ {n} \left(\mathbf {h} _ {t} ^ {\mathrm {r f}} (\pmb {\alpha} _ {i}) - \mathbf {w} ^ {*} (\pmb {\alpha} _ {i})\right) ^ {2} = \frac {1}{n} \sum_ {i = 1} ^ {n} \sum_ {j = 1} ^ {d} \left(\phi_ {0} ^ {j} ^ {\top} (\pmb {\alpha} _ {i}) \bar {\pmb {\theta}} _ {t} ^ {j} - w ^ {*} (\pmb {\alpha} _ {i}) (j)\right) ^ {2},
$$
its pointwise counterpart is
$$
\widehat {\mathcal {R}} ^ {\mathrm {r f}, j} (\boldsymbol {\theta} _ {t} ^ {j}) = \frac {1}{n} \sum_ {i = 1} ^ {n} \left(h _ {t} ^ {\mathrm {r f}, j} (\boldsymbol {\alpha} _ {i}) - w ^ {*} (\boldsymbol {\alpha} _ {i}) (j)\right) ^ {2} = \frac {1}{n} \sum_ {i = 1} ^ {n} \left(\phi_ {0} ^ {j} ^ {\top} (\boldsymbol {\alpha} _ {i}) \bar {\boldsymbol {\theta}} _ {t} ^ {j} - w ^ {*} (\boldsymbol {\alpha} _ {i}) (j)\right) ^ {2},
$$
and GD update rule is defined as
$$
\bar {\boldsymbol {\theta}} _ {t + 1} = \bar {\boldsymbol {\theta}} _ {t} - \eta \nabla \widehat {\mathcal {R}} ^ {\mathrm {r f}} \left(\boldsymbol {\theta} _ {t}\right).
$$
Kernel Least Squares (KLS) predictor In the following we will work with a kernel function:
Proposition 3 (Kernel induced by Rectified Linear Unit (ReLU) activation). The following kernel function is called the NTK function induced by the ReLU activation:
$$
\kappa (\boldsymbol {\alpha}, \boldsymbol {\alpha} ^ {\prime}) = (\boldsymbol {\alpha} ^ {\top} \boldsymbol {\alpha} ^ {\prime}) \int_ {\mathbb {R} ^ {d}} \mathbb {I} \left\{\mathbf {w} ^ {\top} \boldsymbol {\alpha} > 0 \right\} \mathbb {I} \left\{\mathbf {w} ^ {\top} \boldsymbol {\alpha} ^ {\prime} > 0 \right\} \mathcal {N} (\mathrm {d} \mathbf {w} \mid \mathbf {0}, \mathbf {I} _ {d}) \quad (\boldsymbol {\alpha}, \boldsymbol {\alpha} ^ {\prime} \in \mathbb {S} ^ {d - 1}).
$$
The following holds for  $\kappa$ :
- It is a reproducing kernel and has analytic form  $\kappa(\alpha, \alpha') = (\alpha^\top \alpha')(\pi - \arccos(\alpha^\top \alpha'))$  [15].
$\sup_{\pmb {\alpha},\pmb{\alpha}^{\prime}\in \mathbb{S}^{d - 1}}\kappa (\pmb {\alpha},\pmb{\alpha}^{\prime})\leq 1.$
- Eigenvalues of  $\kappa$  satisfy  $\mu_{k} \leq C k^{-\frac{d}{2}}$  for  $k \in \mathbb{N}$  [2].
- The NTK matrix is a symmetric matrix  $\mathbf{K} \in \mathbb{R}^{n \times n}$  with entries  $(\mathbf{K})_{i,j} = \kappa (\pmb{\alpha}_i, \pmb{\alpha}_j)$ .
Throughout,  $\mathcal{H}$  is the RKHS induced by  $\kappa$ .
Having established the kernel function, we assume the following about matrix  $\mathbf{K}$ :
Assumption 2 (Smallest eigenvalue of the NTK matrix). Assume that there exists fixed  $\lambda_0 > 0$  such that  $\mathbb{P}(\lambda_{\mathrm{min}}(\mathbf{K})\geq \lambda_0)\geq 1 - \delta_{\lambda_0}$  and  $\delta_{\lambda_0}\in [0,1]$
Assumption 2 is fairly standard and often satisfied with sample-dependent lower bounds:
- In particular, [9] shows that  $\lambda_0 > 0$  whenever no two distinct inputs are parallel.
- In a random design setting, [4, Lemma 5.2] show that when inputs are sampled from isotropic Gaussian and  $n \leq d^{C}$  for some activation function-dependent  $C$ ,  $\lambda_0 = \Omega(d)$  with probability at least  $1 - e^{-\mathcal{O}(n)}$ .
- In a random design setting, [29] show that (for a certain well-behaved family of input distributions),  $\lambda_{\min}(\mathbf{K}) = \Theta_{\mathbb{P}}(d)$  with high probability. More precisely, their Theorem 3.2 implies that we have  $\lambda_{\min}(\mathbf{K}) \geq C \log(n, d)d$  with probability at least  $1 - n^2 e^{-\mathcal{O}(\sqrt{d})}$ .
In addition to the NTK matrix, we define its empirical counterpart for the  $j$ th component of the hidden layer, namely
$$
\hat {\mathbf {K}} ^ {j} = \left(\boldsymbol {\Phi} _ {0} ^ {j}\right) ^ {\top} \boldsymbol {\Phi} _ {0} ^ {j} \in \mathbb {R} ^ {n \times n}.
$$
Then, it is clear that  $\mathbf{K} = \mathbb{E}[\hat{\mathbf{K}}^j\mid \mathrm{data}]$
Now, for some vector  $\mathbf{c} \in \mathbb{R}^d$ , we define the KLS predictor as
$$
\mathbf {h} _ {t} ^ {\mathrm {k l s}} (\boldsymbol {\alpha}) = \left[ \begin{array}{c} \sum_ {i = 1} ^ {n} k (\boldsymbol {\alpha} _ {i}, \boldsymbol {\alpha}) c _ {i} ^ {1} \\ \vdots \\ \sum_ {i = 1} ^ {n} k (\boldsymbol {\alpha} _ {i}, \boldsymbol {\alpha}) c _ {i} ^ {d} \end{array} \right],
$$
We also use  $h_t^{\mathrm{kl},j}(\alpha)$  to indicate the  $j$ th coordinate of  $\mathbf{h}_t^{\mathrm{kl}}(\alpha)$ . The objective for KLS predictor is defined as:
$$
\widehat {\mathcal {R}} ^ {\mathrm {k l s}} \left(\mathbf {c} _ {t}\right) = \frac {1}{n} \sum_ {i = 1} ^ {n} \left(\mathbf {h} _ {t} ^ {\mathrm {k l s}} \left(\boldsymbol {\alpha} _ {i}\right) - \mathbf {w} ^ {*} \left(\boldsymbol {\alpha} _ {i}\right)\right) ^ {2}
$$
for the pointwise loss function as:
$$
\widehat {\mathcal {R}} ^ {\mathrm {k l s}, j} \left(\mathbf {c} _ {t} ^ {j}\right) = \frac {1}{n} \sum_ {i = 1} ^ {n} \left(\mathbf {h} _ {t} ^ {\mathrm {k l s}, j} \left(\boldsymbol {\alpha} _ {i}\right) - w ^ {*} \left(\boldsymbol {\alpha} _ {i}\right) (j)\right) ^ {2}
$$
and finally GD update rule is defined as:
$$
\mathbf {c} _ {t + 1} = \mathbf {c} _ {t} - \eta \nabla \widehat {\mathcal {R}} ^ {\mathrm {k l s}} (\mathbf {c} _ {t}).
$$
Proposition 4. Let  $\mathbf{w}_i^t$  be iterates generated by Algorithm 2. Then the following statement holds true:
$$
\left\| \mathbf {w} _ {i} ^ {t} - \mathbf {w} ^ {*} (\boldsymbol {\alpha} _ {i}) \right\| ^ {2} \leq (1 - \mu \gamma) ^ {t} D
$$
B.3 Properties of ReLU Networks
In this section we include standard lemmata for analysis of shallow neural networks, and where appropriate we account for the fact that we are working with vector-valued predictors.
Proposition 5 (Activation patterns [21]). Assume that initial parameters of a neural network are chosen as in section 3.1. Consider the set of indices of neurons that changed their activation patterns on input  $\alpha$ , when  $\theta_0$  is replaced by some parameters  $\tilde{\pmb{\theta}} = (\tilde{\mathbf{u}}_1, \dots, \tilde{\mathbf{u}}_m)$ :
$$
P (\tilde {\boldsymbol {\theta}}, \boldsymbol {\alpha}) = \left\{k \in [ m ] | \mathbb {I} \{\tilde {\mathbf {u}} ^ {\top} (k) \boldsymbol {\alpha} > 0 \} - \mathbb {I} \{\mathbf {u} _ {0} ^ {\top} (k) \boldsymbol {\alpha} > 0 \} \neq 0 \right\} \quad (\tilde {\boldsymbol {\theta}} \in \mathbb {R} ^ {N m}, \boldsymbol {\alpha} \in \Delta^ {N}).
$$
Then, for  $\tilde{\theta}$  whose components satisfy  $\max_k\| \tilde{\mathbf{u}}_k - \mathbf{u}_{0,k}\| \leq \rho$  for some fixed  $\rho \geq 0$ , and any  $\alpha \in \Delta^{N}$ , the following facts hold:
(a) For all  $k \in P(\tilde{\pmb{\theta}}, \pmb{\alpha})$ ,  $|\mathbf{u}_{0,k}^{\top} \pmb{\alpha}| \leq \| \tilde{\mathbf{u}}_k - \mathbf{u}_{0,k} \|$ .
(b)  $\mathbb{E}|P(\tilde{\boldsymbol{\theta}},\boldsymbol {\alpha})|\leq m\rho$
(c) With probability at least  $1 - 2e^{-\nu}$  for any  $\nu >0$ $|P(\tilde{\pmb{\theta}},\pmb {\alpha})|\leq m\rho +\sqrt{m\nu}$
(d)  $\left\| \phi_{\tilde{\theta}} - \phi_0\right\| \leq \rho +\sqrt{\frac{\nu}{m}}.$
(e)  $\left\| (\phi_{\tilde{\pmb{\theta}}} - \phi_0)^\top \tilde{\pmb{\theta}}\right\| \leq \rho (\sqrt{m}\rho +\sqrt{\nu}).$
Proposition 6 (Bounded gradient). The gradients of neural network are bounded:
$$
\left\| \nabla_ {\boldsymbol {\theta}} \mathbf {h} _ {\boldsymbol {\theta}} (\boldsymbol {\alpha}) \right\| ^ {2} \leq d, \quad \left\| \nabla \widehat {\mathcal {R}} (\boldsymbol {\theta}) \right\| ^ {2} \leq 4 d \widehat {\mathcal {R}} (\boldsymbol {\theta}).
$$
Proof. The proof simply follows by definition:
$$
\left\| \nabla_ {\boldsymbol {\theta}} \mathbf {h} _ {\boldsymbol {\theta}} (\boldsymbol {\alpha}) \right\| ^ {2} = \sum_ {j = 1} ^ {d} \left\| \phi^ {j} (\boldsymbol {\alpha}) \right\| ^ {2} = \sum_ {j = 1} ^ {d} \sum_ {r = 1} ^ {m} \frac {1}{m} \mathbb {I} \left\{\mathbf {u} _ {r} ^ {\top} \boldsymbol {\alpha} \right\} \| \boldsymbol {\alpha} \| ^ {2} \leq d
$$
$$
\begin{array}{l} \left\| \nabla \widehat {\mathcal {R}} (\boldsymbol {\theta}) \right\| ^ {2} = \left\| \frac {2}{n} \sum_ {i = 1} ^ {n} \nabla_ {\boldsymbol {\theta}} \mathbf {h} _ {\boldsymbol {\theta}} (\boldsymbol {\alpha} _ {i}) (\mathbf {h} _ {\boldsymbol {\theta}} (\boldsymbol {\alpha} _ {i}) - \mathbf {w} ^ {*} (\boldsymbol {\alpha} _ {i})) \right\| ^ {2} \\ \leq 4 \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| \nabla_ {\boldsymbol {\theta}} \mathbf {h} _ {\boldsymbol {\theta}} (\boldsymbol {\alpha} _ {i}) \right\| ^ {2} \left\| \mathbf {h} _ {\boldsymbol {\theta}} (\boldsymbol {\alpha} _ {i}) - \mathbf {w} ^ {*} (\boldsymbol {\alpha} _ {i})\right) \| ^ {2} \leq 4 d \widehat {\mathcal {R}} (\boldsymbol {\theta}). \\ \end{array}
$$
Lemma 10 (Spectral Property of Empirical Kernel Matrix). Given a set of parameter  $\theta_t = \{\mathbf{u}_t(1),\dots,\mathbf{u}_t(m)\}$ , if we assume  $\max_{r\in [m]}\| \mathbf{u}_t(r) - \mathbf{u}_0(r)\| \leq \rho$ , then the following statements about the spectral property of its random feature and empirical gram matrix hold true:
(a)  $\| \phi_t(\pmb {\alpha}_i)\| \leq 1,\| \pmb {\Phi}_t\| \leq \sqrt{n},\| \pmb {\Phi}_t - \pmb {\Phi}_0\| \leq \sqrt{n(\rho + \sqrt{\frac{\nu}{m}})}$
(b)  $\left\| \hat{\mathbf{K}}_t\right\| \leq \left\| \hat{\mathbf{K}}_0\right\| +2n^2 (\rho +\sqrt{\frac{\nu}{m}}),$
(c)  $\lambda_{\min}(\hat{\mathbf{K}}_t)\geq \lambda_{\min}(\hat{\mathbf{K}}_0) - 2n^2 (\rho +\sqrt{\frac{\nu}{m}}),$
(d) If we assume  $m \geq \frac{64n^2\log(n / \nu')}{\lambda_0^2}$ , then  $\mathbb{P}\left(\lambda_{\min}(\hat{\mathbf{K}}_0) \geq \frac{1}{2}\lambda_0\right) \geq 1 - \nu'$ , and  $\mathbb{P}\left(\left\| \hat{\mathbf{K}}_0\right\| \leq \frac{1}{2}\|\mathbf{K}_0\|\right) \geq 1 - \nu'$ .
Proof. We begin with proving  $(a)$ , the spectral norm  $\phi_t(\alpha_i)$
$$
\left\| \phi_ {t} (\boldsymbol {\alpha} _ {i}) \right\| ^ {2} \leq \sum_ {r = 1} ^ {m} \left\| a (r) \mathbb {I} \left\{\mathbf {u} _ {t} ^ {\top} (r) \boldsymbol {\alpha} _ {i} \right\} \boldsymbol {\alpha} _ {i} \right\| ^ {2} \leq 1
$$
the spectral bound for  $\Phi_t$ .
$$
\left\| \boldsymbol {\Phi} _ {t} \right\| ^ {2} = \left\| \left(\phi_ {t} (\boldsymbol {\alpha} _ {1}), \dots , \phi_ {t} (\boldsymbol {\alpha} _ {n})\right) \right\| ^ {2} \leq \sum_ {i = 1} ^ {n} \left\| \phi_ {t} (\boldsymbol {\alpha} _ {i}) \right\| _ {F} ^ {2} \leq n,
$$
so we know  $\| \Phi_t\| \leq \sqrt{n}$
For  $\| \Phi_t - \Phi_0\|$ , by definition we have:
$$
\begin{array}{l} \left\| \Phi_ {t} - \Phi_ {0} \right\| ^ {2} \leq \left\| \Phi_ {t} - \Phi_ {0} \right\| _ {F} ^ {2} \\ \leq \sum_ {i = 1} ^ {n} \left\| \phi_ {t} (\boldsymbol {\alpha} _ {i}) - \phi_ {0} (\boldsymbol {\alpha} _ {i}) \right\| ^ {2} \\ \leq \sum_ {i = 1} ^ {n} \left\| \phi_ {t} (\boldsymbol {\alpha} _ {i}) - \phi_ {0} (\boldsymbol {\alpha} _ {i}) \right\| ^ {2} \\ \leq \sum_ {i = 1} ^ {n} \sum_ {r = 1} ^ {m} | a (r) \mathbb {I} \left\{\mathbf {u} _ {t} ^ {\top} (r) (\boldsymbol {\alpha} _ {i}) \right\} - a (r) \mathbb {I} \left\{\mathbf {u} _ {0} ^ {\top} (r) (\boldsymbol {\alpha} _ {i}) \right\} | ^ {2} \\ \leq \frac {1}{m} \sum_ {i = 1} ^ {n} | P (\boldsymbol {\theta} _ {t}, \boldsymbol {\alpha} _ {i}) | \\ \leq n (\rho + \sqrt {\frac {\nu}{m}}). \\ \end{array}
$$
Now we show the bound on the norm of  $\hat{\mathbf{K}}_t$ . We first examine  $\left\| \phi_t^\top (\pmb {\alpha}_i)\phi_t(\pmb {\alpha}_j) - \phi_0^\top (\pmb {\alpha}_i)\phi_0(\pmb {\alpha}_j)\right\| _F$ . Consider
$$
\begin{array}{l} \left| \phi_ {t} ^ {\top} (\boldsymbol {\alpha} _ {i}) \phi_ {t} (\boldsymbol {\alpha} _ {j}) - \phi_ {0} ^ {\top} (\boldsymbol {\alpha} _ {i}) \phi_ {0} (\boldsymbol {\alpha} _ {j}) \right| \\ = \left| \sum_ {r = 1} ^ {m} a (r) \mathbb {I} \left\{\mathbf {u} _ {t} ^ {\top} (r) \boldsymbol {\alpha} _ {i} \right\} \cdot a (r) \mathbb {I} \left\{\mathbf {u} _ {t} ^ {\top} (r) \boldsymbol {\alpha} _ {j} \right\} \boldsymbol {\alpha} _ {i} ^ {\top} \boldsymbol {\alpha} _ {j} - \sum_ {r = 1} ^ {m} a (r) \mathbb {I} \left\{\mathbf {u} _ {0} ^ {\top} (r) \boldsymbol {\alpha} _ {i} \right\} \cdot a (r) \mathbb {I} \left\{\mathbf {u} _ {0} ^ {\top} (r) \boldsymbol {\alpha} _ {j} \right\} \boldsymbol {\alpha} _ {i} ^ {\top} \boldsymbol {\alpha} _ {j} \right| \\ \leq \frac {1}{m} \sum_ {r = 1} ^ {m} \left| \mathbb {I} \left\{\mathbf {u} _ {t} ^ {\top} (r) \boldsymbol {\alpha} _ {i} \right\} \cdot \mathbb {I} \left\{\mathbf {u} _ {t} ^ {\top} (r) \boldsymbol {\alpha} _ {j} \right\} - \mathbb {I} \left\{\mathbf {u} _ {0} ^ {\top} (r) \boldsymbol {\alpha} _ {i} \right\} \cdot \mathbb {I} \left\{\mathbf {u} _ {0} ^ {\top} (r) \boldsymbol {\alpha} _ {j} \right\} \right| | \boldsymbol {\alpha} _ {i} ^ {\top} \boldsymbol {\alpha} _ {j} | \\ = \frac {1}{m} \sum_ {r = 1} ^ {m} \left| \left(\mathbb {I} \left\{\mathbf {u} _ {t} ^ {\top} (r) \boldsymbol {\alpha} _ {i} \right\} - \mathbb {I} \left\{\mathbf {u} _ {0} ^ {\top} (r) \boldsymbol {\alpha} _ {i} \right\}\right) \cdot \mathbb {I} \left\{\mathbf {u} _ {t} ^ {\top} (r) \boldsymbol {\alpha} _ {j} \right\} - \mathbb {I} \left\{\mathbf {u} _ {0} ^ {\top} (r) \boldsymbol {\alpha} _ {i} \right\} \right. \\ \times \left(\mathbb {I} \left\{\mathbf {u} _ {0} ^ {\top} (r) \boldsymbol {\alpha} _ {j} \right\} - \mathbb {I} \left\{\mathbf {u} _ {t} ^ {\top} (r) \boldsymbol {\alpha} _ {j} \right\}\right) | | \boldsymbol {\alpha} _ {i} ^ {\top} \boldsymbol {\alpha} _ {j} | \\ = \frac {1}{m} \left(\left| P \left(\boldsymbol {\theta} _ {t}, \boldsymbol {\alpha} _ {i}\right) \right| + \left| P \left(\boldsymbol {\theta} _ {t}, \boldsymbol {\alpha} _ {j}\right) \right|\right) \\ \leq 2 (\rho + \sqrt {\frac {\nu}{m}}). \\ \end{array}
$$
Notice that due to triangle inequality we have  $\left\| \hat{\mathbf{K}}_t\right\| \leq \left\| \hat{\mathbf{K}}_0\right\| +\left\| \hat{\mathbf{K}}_t - \hat{\mathbf{K}}_0\right\|$ . Now we examine  $\left\| \hat{\mathbf{K}}_t - \hat{\mathbf{K}}_0\right\|$ :
$$
\begin{array}{l} \left\| \hat {\mathbf {K}} _ {t} - \hat {\mathbf {K}} _ {0} \right\| \leq \left\| \hat {\mathbf {K}} _ {t} - \hat {\mathbf {K}} _ {0} \right\| _ {F} (22) \\ = \sqrt {\sum_ {i , j \in [ n ]} \left| \phi_ {t} ^ {\top} \left(\boldsymbol {\alpha} _ {i}\right) \phi_ {t} \left(\boldsymbol {\alpha} _ {j}\right) - \phi_ {0} ^ {\top} \left(\boldsymbol {\alpha} _ {i}\right) \phi_ {0} \left(\boldsymbol {\alpha} _ {j}\right) \right| ^ {2}} (23) \\ \leq \sum_ {i, j \in [ n ]} \left| \phi_ {t} ^ {\top} \left(\boldsymbol {\alpha} _ {i}\right) \phi_ {t} \left(\boldsymbol {\alpha} _ {j}\right) - \phi_ {0} ^ {\top} \left(\boldsymbol {\alpha} _ {i}\right) \phi_ {0} \left(\boldsymbol {\alpha} _ {j}\right) \right| (24) \\ = 2 n ^ {2} \left(\rho + \sqrt {\frac {\nu}{m}}\right). (25) \\ \end{array}
$$
At last, we prove statement (d). According to Weyl's inequality:
$$
\lambda_ {\min } \left(\hat {\mathbf {K}} _ {0}\right) \geq \lambda_ {\min } (\mathbf {K}) - \left\| \hat {\mathbf {K}} _ {0} - \mathbf {K} \right\|.
$$
According to basic concentration inequality we know that the following statement holds with probability at least  $1 - \nu$ :
$$
| \hat {\mathbf {K}} _ {0} (i, j) - \mathbf {K} (i, j) | \leq \frac {2 \sqrt {\log (1 / \nu^ {\prime})}}{\sqrt {m}}
$$
Taking union bound over all  $n^2$  entries and summing over all entries yields:
$$
\left\| \hat {\mathbf {K}} _ {0} - \mathbf {K} \right\| \leq \left\| \hat {\mathbf {K}} _ {0} - \mathbf {K} \right\| _ {F} \leq \sum_ {i, j \in [ n ]} | \hat {\mathbf {K}} _ {0} (i, j) - \mathbf {K} (i, j) | \leq \frac {4 n \sqrt {\log (n / \nu^ {\prime})}}{\sqrt {m}}
$$
To ensure the RHS less than  $\frac{1}{2}\lambda_0$ , we need
$$
m \geq \frac {6 4 n ^ {2} \log (n / \nu^ {\prime})}{\lambda_ {0} ^ {2}}.
$$
Last, since  $\left\| \hat{\mathbf{K}}_0 - \mathbf{K}\right\| \leq \frac{1}{2}\lambda_0$ , we know that it is also true  $\left\| \hat{\mathbf{K}}_0 - \mathbf{K}\right\| \leq \frac{1}{2}\sigma_{\max} = \frac{1}{2}\|\mathbf{K}\|$ .
Lemma 11 (Deviation of Gradient Computed on Inexact Labels). For Algorithm 2, the following statements hold true:
$$
\begin{array}{l} \left\| \nabla_ {\pmb {\theta} _ {t} ^ {j}} \widehat {\mathcal {R}} ^ {j} (\pmb {\theta} _ {t} ^ {j}) - \mathbf {g} _ {t} ^ {j} \right\| ^ {2} \leq 4 (1 - \mu \gamma) ^ {K t} D, \\ \left\| \boldsymbol {\Phi} _ {t} ^ {j} ^ {\top} (\nabla \widehat {\mathcal {R}} ^ {j} (\boldsymbol {\theta} _ {t} ^ {j}) - \mathbf {g} _ {t} ^ {j}) \right\| ^ {2} \leq \frac {4}{n} \left(\frac {3}{2} \| \mathbf {K} \| + 2 n ^ {2} \left(\rho + \sqrt {\frac {\nu}{m}}\right)\right) ^ {2} (1 - \mu \gamma) ^ {K t} D. \\ \end{array}
$$
Proof. Recall the definition of  $\nabla_{\pmb{\theta}_t}\widehat{\mathcal{R}} (\pmb {\theta}_t)$  and  $\mathbf{g}_t$
$$
\begin{array}{l} \nabla_ {\boldsymbol {\theta} _ {t}} \widehat {\mathcal {R}} (\boldsymbol {\theta} _ {t}) = \left[ \begin{array}{c} \frac {2}{n} \sum_ {i = 1} ^ {n} \phi_ {t} ^ {1} (h _ {t} ^ {1} (\boldsymbol {\alpha} _ {i}) - \mathbf {w} ^ {*} (\boldsymbol {\alpha} _ {i}) (1)) \\ \vdots \\ \frac {2}{n} \sum_ {i = 1} ^ {n} \phi_ {t} ^ {d} (h _ {t} ^ {d} (\boldsymbol {\alpha} _ {i}) - \mathbf {w} ^ {*} (\boldsymbol {\alpha} _ {i}) (d)) \end{array} \right] = \left[ \begin{array}{c} \frac {2}{n} \boldsymbol {\Phi} _ {t} ^ {1} (\mathbf {h} _ {t} ^ {1} - \mathbf {w} ^ {*} (1)) \\ \vdots \\ \frac {2}{n} \boldsymbol {\Phi} _ {t} ^ {d} (\mathbf {h} _ {t} ^ {d} - \mathbf {w} ^ {*} (d)) \end{array} \right], \\ \mathbf {g} _ {t} = \left[ \begin{array}{c} \frac {2}{n} \sum_ {i = 1} ^ {n} \phi_ {t} ^ {1} (h _ {t} ^ {1} (\boldsymbol {\alpha} _ {i}) - \mathbf {w} _ {i} ^ {t} (1)) \\ \vdots \\ \frac {2}{n} \sum_ {i = 1} ^ {n} \phi_ {t} ^ {d} (h _ {t} ^ {d} (\boldsymbol {\alpha} _ {i}) - \mathbf {w} _ {i} ^ {t} (d)) \end{array} \right] = \left[ \begin{array}{c} \frac {2}{n} \boldsymbol {\Phi} _ {t} ^ {1} (\mathbf {h} _ {t} ^ {1} - \mathbf {w} _ {i} ^ {t} (1)) \\ \vdots \\ \frac {2}{n} \boldsymbol {\Phi} _ {t} ^ {d} (\mathbf {h} _ {t} ^ {d} - \mathbf {w} _ {i} ^ {t} (d)) \end{array} \right] \\ \end{array}
$$
where  $\mathbf{w}^t (j) = [w_1^t (j),\dots,w_n^t (j)]^\top \in \mathbb{R}^n$  and  $\mathbf{w}^{*}(j) = [w^{*}(\pmb{\alpha}_{1})(j),\dots,w^{*}(\pmb{\alpha}_{n})(j)]^\top \in \mathbb{R}^n$ . Hence
$$
\begin{array}{l} \left\| \nabla_ {\boldsymbol {\theta} _ {t} ^ {j}} \widehat {\mathcal {R}} ^ {j} \left(\boldsymbol {\theta} _ {t} ^ {j}\right) - \mathbf {g} _ {t} ^ {j} \right\| ^ {2} = \left\| \frac {2}{n} \boldsymbol {\Phi} _ {t} ^ {j} \left(\mathbf {h} _ {t} ^ {j} - \mathbf {w} ^ {*} (j)\right) - \frac {2}{n} \boldsymbol {\Phi} _ {t} ^ {j} \left(\mathbf {h} _ {t} ^ {j} - \mathbf {w} ^ {t} (j)\right) \right\| ^ {2} \\ = \left\| \frac {2}{n} \boldsymbol {\Phi} _ {t} ^ {j} \left(\mathbf {w} ^ {t} (j) - \mathbf {w} ^ {*} (j)\right) \right\| ^ {2} \\ \leq \frac {4}{n ^ {2}} \max  _ {j \in [ d ]} \left\| \boldsymbol {\Phi} _ {t} ^ {j} \right\| ^ {2} \sum_ {i = 1} ^ {n} \left\| \mathbf {w} _ {i} ^ {t} - \mathbf {w} ^ {*} (\boldsymbol {\alpha} _ {i}) \right\| ^ {2} \\ \leq 4 (1 - \mu \gamma) ^ {K t} D. \\ \end{array}
$$
For the second result we have
$$
\begin{array}{l} \left\| \boldsymbol {\Phi} _ {t} ^ {j} ^ {\top} (\nabla \widehat {\mathcal {R}} ^ {j} (\boldsymbol {\theta} _ {t} ^ {j}) - \mathbf {g} _ {t} ^ {j}) \right\| ^ {2} = \frac {4}{n ^ {2}} \left\| \boldsymbol {\Phi} _ {t} ^ {\top} \boldsymbol {\Phi} _ {t} (\mathbf {h} _ {t} ^ {j} - \mathbf {w} ^ {*} (j)) - \boldsymbol {\Phi} _ {t} ^ {j} ^ {\top} \boldsymbol {\Phi} _ {t} ^ {j} (\mathbf {h} _ {t} ^ {j} - \mathbf {w} ^ {t} (j)) \right\| ^ {2} \\ = \frac {4}{n ^ {2}} \left\| \boldsymbol {\Phi} _ {t} ^ {j} ^ {\top} \boldsymbol {\Phi} _ {t} ^ {j} (\mathbf {w} ^ {t} (j) - \mathbf {w} ^ {*} (j)) \right\| ^ {2} \\ \leq \frac {4}{n ^ {2}} \left\| \Phi_ {t} ^ {j} ^ {\top} \Phi_ {t} ^ {j} \right\| ^ {2} n (1 - \mu \gamma) ^ {K t} D \\ \leq \frac {4}{n} \left(\left\| \hat {\mathbf {K}} _ {0} \right\| + 2 n ^ {2} \left(\rho + \sqrt {\frac {\nu}{m}}\right)\right) ^ {2} (1 - \mu \gamma) ^ {K t} D \\ \leq \frac {4}{n} \left(\frac {3}{2} \| \mathbf {K} \| + 2 n ^ {2} (\rho + \sqrt {\frac {\nu}{m}})\right) ^ {2} (1 - \mu \gamma) ^ {K t} D. \\ \end{array}
$$
The following theorem shows that Algorithm 2 can converge on empirical risk defined in (4).
Theorem 5 (Convergence of Algorithm). For Algorithm 2, if we choose
$$
m \geq \left(\frac {4 8 n B _ {y}}{\lambda_ {0}} + \sqrt {\nu}\right) ^ {2} \frac {2 5 6 n ^ {4}}{\lambda_ {0} ^ {2}},
$$
$$
\eta \leq \min  \left\{\frac {3 n ^ {3}}{\lambda_ {0}} \left(\frac {4 8 n B _ {y}}{\lambda_ {0}} + \sqrt {\nu}\right), \frac {3 \lambda_ {0} n ^ {2}}{2 5 6 \left(\frac {1}{2} \| \mathbf {K} \| + \frac {\lambda_ {0}}{8}\right) ^ {2}} \right\},
$$
and assume the inner iteration number of Algorithm 1 satisfying
$$
K \geq \max  \left\{\kappa \log \left(\frac {\eta \lambda_ {0} n T ^ {2} D}{1 6 B _ {y} ^ {2} n}\right), 2 \kappa \log \left(\frac {\eta \lambda_ {0} \kappa \sqrt {D}}{4 B _ {y} n}\right) \right\}, \tag {26}
$$
then the following statement holds for any  $t \in [T]$  and  $j \in [d]$ :
$$
\widehat {\mathcal {R}} ^ {j} \left(\boldsymbol {\theta} _ {t + 1} ^ {j}\right) \leq \left(1 - \frac {\eta \lambda_ {0}}{4 n}\right) ^ {t} \widehat {\mathcal {R}} ^ {j} \left(\boldsymbol {\theta} _ {0} ^ {j}\right) + \frac {4 n C (1 - \mu \gamma) ^ {K} D}{\eta \lambda_ {0}},
$$
$$
C = \frac {\lambda_ {0} ^ {2} \eta^ {2}}{2 4 n ^ {4}} + \frac {4 \eta^ {2} \left(\frac {1}{2} \left\| \mathbf {K} \right\| + \frac {\lambda_ {0}}{8}\right) ^ {2}}{\frac {3}{8} n ^ {2}}.
$$
Proof. We prove by induction. We make the following inductive hypotheses:
(I)  $\widehat{\mathcal{R}}^j (\pmb {\theta}_t^j)\leq \left(1 - \frac{\eta\lambda_0}{4n}\right)^{t - 1}\widehat{\mathcal{R}}^j (\pmb {\theta}_0^j) + \sum_{s = 0}^{t - 1}\left(1 - \frac{\eta\lambda_0}{4n}\right)^{t - 1 - s}(1 - \mu \gamma)^s C(1 - \mu \gamma)^K D,$
(II)  $\max_{r\in [m]}\left\| \mathbf{u}_t^j (r) - \mathbf{u}_t^j (r)\right\| \leq \rho \coloneqq \frac{48nB_y}{\lambda_0\sqrt{m}}.$
where  $C$  is the constant to be determined later. First we are going to show hypothesis (II) holding for  $t + 1$ . First, by our choice of  $\eta$ , we know  $(1 - \mu \gamma) \leq \left(1 - \frac{\eta \lambda_0}{4n}\right)$ . Hence
$$
\sum_ {s = 0} ^ {t - 1} \left(1 - \frac {\eta \lambda_ {0}}{4 n}\right) ^ {t - 1 - s} (1 - \mu \gamma) ^ {s} \leq \frac {4 n}{\eta \lambda_ {0}}.
$$
Now we are going to bound maximal neuron drifting from initialization: for any  $j \in [d]$