SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\underset {(\hat {u}, \bar {u}) \sim \Gamma_ {t}} {\mathbb {E}} \left| B _ {t} ^ {\prime \prime} \right| \lesssim \left(\hat {\sigma} _ {2, d} ^ {2} + \hat {\sigma} _ {4, d} ^ {2}\right) \cdot \left(\frac {d (\log d) ^ {O (1)}}{\sqrt {m}} + \overline {{\Delta}} _ {t}\right) \cdot \overline {{\Delta}} _ {t} ^ {2} \tag {E.156}
$$
which is at most the bound that we have for  $B_{t}^{\prime}$ , and thus,
$$
\underset {(\hat {u}, \bar {u}) \sim \Gamma_ {t}} {\mathbb {E}} \left[ B _ {t} \right] \lesssim \frac {\left(\hat {\sigma} _ {2 , d} ^ {2} + \hat {\sigma} _ {4 , d} ^ {2}\right) d (\log d) ^ {O (1)}}{\sqrt {m}} \overline {{\Delta}} _ {t} + \left(\hat {\sigma} _ {2, d} ^ {2} + \hat {\sigma} _ {4, d} ^ {2}\right) \overline {{\Delta}} _ {t} ^ {3} \tag {E.157}
$$
E.3 Upper Bound on  $C_t$
$$
C _ {t} (\chi) := - 2 \left\langle \operatorname {g r a d} _ {\hat {u}} \widehat {L} \left(\hat {\rho} _ {t}\right) - \operatorname {g r a d} _ {\hat {u}} L \left(\hat {\rho} _ {t}\right), \hat {u} _ {t} (\chi) - \bar {u} _ {t} (\chi) \right\rangle . \tag {E.158}
$$
This quantity is essentially the part of the growth of  $\| \delta_t\| _2^2$  which is due to the difference between the gradient using finite samples and the gradient using infinitely many samples. When there is no risk of confusion, we simply use  $C_t$ ,  $\hat{u}_t$ , and  $\bar{u}_t$ , omitting the  $\chi$  dependency. We bound this term with the following lemma:
Lemma E.11. Suppose we are in the setting of Theorem 3.4. Assume that the number of samples  $x_{i}$  is  $n \leq d^{C}$ , for any universal constant  $C > 0$ . Assume that the width  $m \leq d^{C}$  for any universal constant  $C > 0$ . Let  $T > 0$ . Assume for all  $t \in [0, T]$  we have a bound  $B_{1}$  on  $\overline{\Delta}_{t}$  with  $B_{1} \leq \frac{1}{\sqrt{d}}$ . Additionally, assume for all  $t \in [0, T]$  that we have a bound  $B_{2}$  on  $\Delta_{\max,t}$  such that  $B_{2} \geq \frac{1}{\sqrt{d}}$ . Then, with probability  $1 - \frac{1}{d^{\Omega(\log d)}}$ , for all  $t \in [0, T]$  and  $\chi \in \mathbb{S}^{d-1}$ , we have
$$
\left| C _ {t} (\chi) \right| \lesssim \hat {\sigma} _ {\max } ^ {2} (\log d) ^ {O (1)} \cdot \left(\sqrt {\frac {d}{n}} \| \delta_ {t} \| _ {2} + \frac {d ^ {2}}{n} B _ {2} ^ {2} \overline {{\Delta}} \| \delta_ {t} \| _ {2} ^ {2} + \frac {d ^ {2 . 5}}{n} \overline {{\Delta}} ^ {2} B _ {2} ^ {2} \| \delta_ {t} \| _ {2} + \frac {d ^ {4}}{n} B _ {2} ^ {4} \overline {{\Delta}} ^ {2} \| \delta_ {t} \| _ {2} ^ {2}\right). \tag {E.159}
$$
where  $\delta_t = \hat{u}_t(\chi) - \bar{u}_t(\chi)$ .
Proof. We bound  $C_t$  by expanding  $\mathrm{grad}_{\hat{u}} L(\hat{\rho}_t)$  as the sum of several monomials, as well as expanding  $\mathrm{grad}_{\hat{u}} \widehat{L}(\hat{\rho}_t)$ , and then applying Lemma I.7 to each term from  $\mathrm{grad}_{\hat{u}} L(\hat{\rho}_t)$  and the corresponding term
from  $\operatorname{grad}_{\hat{u}} \widehat{L}(\hat{\rho}_t)$ . For convenience, we use  $\ell^{(x)}(\hat{\rho})$  to denote the loss of  $f_{\hat{\rho}}$  on a single data point  $x \in \mathbb{S}^{d-1}$ , and  $\operatorname{grad}_{\hat{u}} \ell^{(x)}(\hat{\rho})$  to denote the projected gradient of  $\ell^{(x)}(\hat{\rho})$  with respect to  $\hat{u}$ . We have
$$
\begin{array}{l} \left\langle \operatorname {g r a d} _ {\hat {u}} \ell^ {(x)} \left(\hat {\rho} _ {t}\right), \hat {u} _ {t} - \bar {u} _ {t} \right\rangle = \left\langle \left(f _ {\hat {\rho}} (x) - y (x)\right) \sigma^ {\prime} \left(\left\langle \hat {u} _ {t}, x \right\rangle\right) x, \left(I - \hat {u} _ {t} \hat {u} _ {t} ^ {\top}\right) \left(\hat {u} _ {t} - \bar {u} _ {t}\right) \right\rangle (E.160) \\ = \left\langle \left(f _ {\hat {\rho}} (x) - y (x)\right) \sigma^ {\prime} \left(\left\langle \hat {u} _ {t}, x \right\rangle\right) x, \hat {u} _ {t} - \bar {u} _ {t} \right\rangle (E.161) \\ - \left\langle \left(f _ {\hat {\rho}} (x) - y (x)\right) \sigma^ {\prime} \left(\left\langle \hat {u} _ {t}, x \right\rangle\right) x, \hat {u} _ {t} \right\rangle \cdot \left(1 - \left\langle \hat {u} _ {t}, \bar {u} _ {t} \right\rangle\right) (E.162) \\ = \left\langle \left(f _ {\hat {\rho}} (x) - y (x)\right) \sigma^ {\prime} \left(\left\langle \hat {u} _ {t}, x \right\rangle\right) x, \hat {u} _ {t} - \bar {u} _ {t} \right\rangle (E.163) \\ - \left\langle \left(f _ {\hat {\rho}} (x) - y (x)\right) \sigma^ {\prime} \left(\left\langle \hat {u} _ {t}, x \right\rangle\right) x, \hat {u} _ {t} \right\rangle \cdot \frac {\left\| \hat {u} _ {t} - \bar {u} _ {t} \right\| _ {2} ^ {2}}{2} (E.164) \\ \end{array}
$$
For convenience, define  $\delta_t \coloneqq \hat{u}_t - \bar{u}_t$ . Then we can rewrite the above as
$$
\begin{array}{l} \langle \operatorname {g r a d} _ {\hat {u}} \ell^ {(x)} (\hat {\rho} _ {t}), \delta_ {t} \rangle = \left\langle \left(f _ {\hat {\rho}} (x) - y (x)\right) \sigma^ {\prime} \left(\left\langle \hat {u} _ {t}, x \right\rangle\right) \langle x, \delta_ {t} \right\rangle (E.165) \\ - \left\langle \left(f _ {\hat {\rho}} (x) - y (x)\right) \sigma^ {\prime} \left(\left\langle \hat {u} _ {t}, x \right\rangle\right) x, \hat {u} _ {t} \right\rangle \cdot \frac {\left\| \delta_ {t} \right\| _ {2} ^ {2}}{2} (E.166) \\ \end{array}
$$
For illustration, define  $\mathcal{C}(g)$  as
$$
\mathcal {C} (g) := \left| \frac {1}{n} \sum_ {i = 1} ^ {n} g \left(x _ {i}\right) - \underset {x \sim \mathbb {S} ^ {d - 1}} {\mathbb {E}} [ g (x) ] \right| \tag {E.167}
$$
i.e. how close the empirical mean of the  $g(x_{i})$  is to its expected value. We will show that  $\mathcal{C}(g)$  is small when  $g$  corresponds to each of the terms on the right-hand side of Eq. (E.165), even when we take the supremum over the  $\hat{u}_t$ , which may depend on the  $x_{i}$ .
Concentration for Mean of First Term in Eq. (E.165). As a first step, we list all of the terms we obtain when expanding the first term on the right-hand side of Eq. (E.165). The  $\sigma^{\prime}(\langle \hat{u}_t,x\rangle)$  factor consists of the following terms: (1) a  $\hat{\sigma}_{2,d}\sqrt{N_{2,d}}\langle \hat{u}_t,x\rangle$  term, (2) a  $\hat{\sigma}_{4,d}\sqrt{N_{4,d}}\langle \hat{u}_t,x\rangle^3$  term, and (3) a  $\frac{\hat{\sigma}_{4,d}\sqrt{N_{4,d}}}{d}\langle \hat{u}_t,x\rangle$  term — here, we have stated these terms up to absolute constant factors for convenience. In the  $f_{\hat{\rho}}(x) - y(x)$  factor,  $f_{\hat{\rho}}(x)$  contributes (1) a  $\hat{\sigma}_{2,d}\sqrt{N_{2,d}}\langle \hat{u}_t',x\rangle^2$  term, (2) a  $\hat{\sigma}_{4,d}\sqrt{N_{4,d}}\langle \hat{u}_t',x\rangle^4$  term, and (3) a  $\frac{\hat{\sigma}_{4,d}\sqrt{N_{4,d}}}{d}\langle \hat{u}_t',x\rangle^2$  term. Here note that  $\hat{u}_t'$  is not a single vector, but ranges over the entire support of  $\hat{\rho}_t$ . Thus we implicitly perform a union bound over the  $m$  vectors in the support of  $\hat{\rho}_t$  — this does not affect the proof since the failure probability of Lemma I.7 is  $\frac{1}{d^{\Omega(\log d)}}$ , while we assume the width  $m$  is at most  $d^{C}$  where  $C$  can be any absolute constant. Additionally,  $y(x)$  consists of the following terms: (1) a  $\hat{\sigma}_{2,d}\gamma_2\sqrt{N_{2,d}}\langle e_1,x\rangle^2$  term, (2) a  $\hat{\sigma}_{4,d}\gamma_4\sqrt{N_{4,d}}\langle e_1,x\rangle^4$  term, and (3) a  $\frac{\hat{\sigma}_{4,d}\gamma_4\sqrt{N_{4,d}}}{d}\langle e_1,x\rangle^2$  term. Finally, there is a factor of  $\langle \delta_t,x\rangle$ .
For convenience, throughout this proof, let  $\bar{\delta}_t = \frac{\delta_t}{\|\delta_t\|_2}$ . In each of the above factors, for every occurrence of  $\langle \hat{u}_t,x\rangle$ , we substitute  $\hat{u}_t = \bar{u}_t + \delta_t$  and further expand. This is useful since originally  $\hat{u}_t$  depends on the  $x_i$  and thus, when we applied Lemma I.7 we would have to take the supremum over  $\hat{u}_t$  — however, now  $\bar{u}_t$  can be considered a fixed vector with respect to the  $x_i$ , and while we need to take the supremum over  $\delta_t$ , in each term where  $\delta_t$  occurs, we obtain an additional factor of  $\|\delta_t\|_2$ , which reduces the error from uniform convergence. We now completely expand the first term on the right-hand side of Eq. (E.165). First we consider the contribution to  $\mathcal{C}(g)$  of  $f_{\hat{\rho}}(x)$  and later study  $y(x)$ . Up to constant factors, it suffices to obtain an upper bound on the concentration of
$$
\sum_ {\substack {p + r = 1, 3 \\ q + s = 2, 4}} \hat {\sigma} _ {\max } ^ {2} d ^ {\frac {p + 1 + q + r + s}{2}} \left\langle \delta_ {t}, x \right\rangle^ {p + 1} \left\langle \delta_ {t} ^ {\prime}, x \right\rangle^ {q} \left\langle \bar {u} _ {t}, x \right\rangle^ {r} \left\langle \bar {u} _ {t} ^ {\prime}, x \right\rangle^ {s} \tag{E.168}
$$
where we have let  $\hat{\sigma}_{\mathrm{max}} = \max (|\hat{\sigma}_{2,d}|,|\hat{\sigma}_{4,d}|)$ . Here, the sum ranges over  $p + q = 1,3$  and  $q + s = 2,4$  since the  $\delta_t$  and  $\bar{u}_t$  terms are due to the  $\sigma^{\prime}(\langle \hat{u}_t,x\rangle)$  factor, while the  $\delta_t^\prime$  and  $\bar{u}_t^\prime$  terms are due to the  $f_{\hat{\rho}}(x)$  factors.
We now obtain concentration bounds for each of the terms in Eq. (E.168). First, we consider the case where  $p + q \geq 1$ , as in this case, Lemma I.7 can be applied (since we can choose  $p_1, p_2, p_3, p_4$
in Lemma I.7 so that  $u_{1} = \bar{\delta}_{t}$  and  $p_{1} = p + 1$ , and  $u_{2} = \bar{\delta}_{t}'$  and  $p_{2} = q$ , and  $p_{3} = p_{4} = 0$  — the hypotheses are then satisfied since  $p_{1} + p_{2} + p_{3} + p_{4} \geq 2$ . From Lemma I.7, we obtain the bound
$$
\begin{array}{l} \hat {\sigma} _ {\max } ^ {2} d ^ {\frac {p + q + r + s + 1}{2}} \left| \frac {1}{n} \sum_ {i = 1} ^ {n} \langle \delta_ {t}, x \rangle^ {p + 1} \langle \delta_ {t} ^ {\prime}, x \rangle^ {q} \langle \bar {u} _ {t}, x \rangle^ {r} \langle \bar {u} _ {t} ^ {\prime}, x \rangle^ {s} - \underset {x \sim \mathbb {S} ^ {d - 1}} {\mathbb {E}} [ \langle \delta_ {t}, x \rangle^ {p + 1} \langle \delta_ {t} ^ {\prime}, x \rangle^ {q} \langle \bar {u} _ {t}, x \rangle^ {r} \langle \bar {u} _ {t} ^ {\prime}, x \rangle^ {s} ] \right| (E.169) \\ \lesssim \hat {\sigma} _ {\max } ^ {2} (\log d) ^ {O (1)} \left(\sqrt {\frac {d}{n}} + \frac {d ^ {\frac {p + 1 + q}{2}}}{n}\right) \| \delta_ {t} \| _ {2} ^ {p + 1} \| \delta_ {t} ^ {\prime} \| _ {2} ^ {q} (E.170) \\ \end{array}
$$
Here in applying Lemma I.7, we have assumed that the number of samples  $n$  is at most  $d^{C}$  for some absolute constant  $C$ , meaning that the  $\frac{1}{d^{\Omega(\log d)}}$  term in the conclusion is a lower-order term. Lastly, we consider the case where  $p + q = 0$ . To apply Lemma I.7, we can let  $u_{1} = \bar{u}_{t}$ ,  $p_{1} = 1$ ,  $u_{2} = \frac{\delta_{t}}{\|\delta_{t}\|_{2}}$ ,  $p_{2} = 1$ , and let  $p_{3} = p_{4} = 0$ , with  $w_{1} = \bar{u}_{t}$ ,  $q_{1} = r - 1$  and  $w_{2} = \bar{u}_{t}'$ ,  $q_{2} = s$  since  $p_{1} + p_{2} = 2$ , the hypotheses of Lemma I.7 are still satisfied. This leads to us obtaining only one power of  $\| \delta_t\| _2$  in the bound for this term: if  $p = q = 0$ , then we have the bound
$$
\begin{array}{l} \hat {\sigma} _ {\max } ^ {2} d ^ {\frac {p + q + r + s + 1}{2}} \left| \frac {1}{n} \sum_ {i = 1} ^ {n} \langle \delta_ {t}, x \rangle^ {p + 1} \langle \delta_ {t} ^ {\prime}, x \rangle^ {q} \langle \bar {u} _ {t}, x \rangle^ {r} \langle \bar {u} _ {t} ^ {\prime}, x \rangle^ {s} - \underset {x \sim \mathbb {S} ^ {d - 1}} {\mathbb {E}} [ \langle \delta_ {t}, x \rangle^ {p + 1} \langle \delta_ {t} ^ {\prime}, x \rangle^ {q} \langle \bar {u} _ {t}, x \rangle^ {r} \langle \bar {u} _ {t} ^ {\prime}, x \rangle \right. (E.171) \\ \lesssim \hat {\sigma} _ {\max } ^ {2} (\log d) ^ {O (1)} \left(\sqrt {\frac {d}{n}} + \frac {d}{n}\right) \| \delta_ {t} \| _ {2} (E.172) \\ \end{array}
$$
Summing over the possible values of  $p, q, r, s$ , we find that the overall contribution to the concentration error of the term with  $f_{\hat{\rho}}$  is up to logarithmic factors and a  $\hat{\sigma}_{\mathrm{max}}^2$  factor,
$$
\begin{array}{l} \underset {\delta^ {\prime} \sim \hat {\rho}} {\mathbb {E}} \sum_ {p, q, r, s} \left(\sqrt {\frac {d}{n}} \| \delta_ {t} \| _ {2} ^ {p + 1} \| \delta_ {t} ^ {\prime} \| _ {2} ^ {q} + \frac {d ^ {\frac {p + 1 + q}{2}}}{n} \| \delta_ {t} \| _ {2} ^ {p + 1} \| \delta_ {t} ^ {\prime} \| _ {2} ^ {q}\right) (E.173) \\ \lesssim \underset {\delta^ {\prime} \sim \hat {\rho}} {\mathbb {E}} \sum_ {p \leq 3, q \leq 4} \left(\sqrt {\frac {d}{n}} \| \delta_ {t} \| _ {2} ^ {p + 1} \| \delta_ {t} ^ {\prime} \| _ {2} ^ {q} + \frac {d ^ {\frac {p + 1 + q}{2}}}{n} \| \delta_ {t} \| _ {2} ^ {p + 1} \| \delta_ {t} ^ {\prime} \| _ {2} ^ {q}\right) (E.174) \\ \lesssim \sqrt {\frac {d}{n}} \| \delta_ {t} \| _ {2} + \underset {\delta^ {\prime} \sim \hat {\rho}} {\mathbb {E}} \sum_ {p \leq 3, q \leq 4} \frac {d ^ {\frac {p + q + 1}{2}}}{n} \| \delta_ {t} \| _ {2} ^ {p + 1} \| \delta_ {t} ^ {\prime} \| _ {2} ^ {q} (E.175) \\ \lesssim \sqrt {\frac {d}{n}} \| \delta_ {t} \| _ {2} + \underset {\delta^ {\prime} \sim \hat {\rho}} {\mathbb {E}} \sum_ {p \leq 3, q \leq 4} \frac {d ^ {\frac {p + q + 1}{2}}}{n} \| \delta_ {t} \| _ {2} ^ {p + 1} \| \delta_ {t} ^ {\prime} \| _ {2} ^ {q} (E.176) \\ \lesssim \sqrt {\frac {d}{n}} \| \delta_ {t} \| _ {2} + \sum_ {p \leq 3, q \leq 4} \frac {d ^ {\frac {p + q + 1}{2}}}{n} \| \delta_ {t} \| _ {2} ^ {p + 1} \underset {\delta^ {\prime} \sim \hat {\rho}} {\mathbb {E}} \| \delta_ {t} ^ {\prime} \| _ {2} ^ {q} (E.177) \\ \end{array}
$$
Finally, in order to simplify this bound, we isolate the dominant terms among the terms of the form  $\frac{d^{\frac{p + q + 1}{2}}}{n}\| \delta_t\| _2^{p + 1}\mathbb{E}_{\delta '\sim \hat{\rho}}\| \delta_t'\| _2^q$ , with the following cases. First, consider all the terms where  $p = 0$ . For  $q\leq 2$  we can bound these terms by  $\frac{d^{\frac{p + q + 1}{2}}}{n}\overline{\Delta}^q\| \delta_t\| _2$ , and for  $q > 2$  we can bound these terms by  $\frac{d^{\frac{p + q + 1}{2}}}{n}\cdot \overline{\Delta}^2\cdot B_2^{q - 2}\| \delta_t\| _2$ . Thus, out of these terms, the term with  $q = 4$  is dominant, and it contributes
$$
\frac {d ^ {2 . 5}}{n} \cdot \overline {{\Delta}} ^ {2} \cdot B _ {2} ^ {2} \| \delta_ {t} \| _ {2} \tag {E.178}
$$
Next, consider the case where  $p \geq 1$ . In this case, since the bound  $B_{2}$  on  $\Delta_{\max,t}$  is greater than  $\frac{1}{d^{1/2}}$ , the term with  $p = 3$  and  $q = 4$  dominates, and it contributes
$$
\frac {d ^ {4}}{n} \cdot B _ {2} ^ {2} \overline {{\Delta}} ^ {2} \cdot B _ {2} ^ {2} \| \delta_ {t} \| _ {2} ^ {2} = \frac {d ^ {4}}{n} B _ {2} ^ {4} \overline {{\Delta}} ^ {2} \| \delta_ {t} \| _ {2} ^ {2} \tag {E.179}
$$
However, we must lastly consider the case where  $p \geq 1$  and  $q < 2$  — the bound on the concentration error may be larger for  $q = 1$  than  $q = 2$  since  $\overline{\Delta} < \frac{1}{\sqrt{d}}$ . In this case, since  $B_2 > \frac{1}{\sqrt{d}}$ , we must
consider the  $p = 3, q = 1$  and  $p = 3, q = 0$  terms, which respectively contribute
$$
\frac {d ^ {2 . 5}}{n} \bar {\Delta} B _ {2} ^ {2} \| \delta_ {t} \| _ {2} ^ {2} \tag {E.180}
$$
$$
\frac {d ^ {2}}{n} B _ {2} ^ {2} \| \delta_ {t} \| _ {2} ^ {2} \tag {E.181}
$$
Note that the  $p = 3, q = 0$  term dominates since our bound on  $\overline{\Delta}$  is less than  $\frac{1}{\sqrt{d}}$ , meaning we can ignore the  $p = 3, q = 1$  term. Thus our overall bound on the concentration error originating from the first term in Eq. (E.165) is
$$
\hat {\sigma} _ {\max } ^ {2} (\log d) ^ {O (1)} \cdot \left(\sqrt {\frac {d}{n}} \| \delta_ {t} \| _ {2} + \frac {d ^ {2}}{n} B _ {2} ^ {2} \bar {\Delta} \| \delta_ {t} \| _ {2} ^ {2} + \frac {d ^ {2 . 5}}{n} \bar {\Delta} ^ {2} B _ {2} ^ {2} \| \delta_ {t} \| _ {2} + \frac {d ^ {4}}{n} B _ {2} ^ {4} \bar {\Delta} ^ {2} \| \delta_ {t} \| _ {2} ^ {2}\right) \tag {E.182}
$$
To deal with the terms originating from  $y(x)$ , we observe that we can simply treat  $\langle e_1, x \rangle$  in the same way as we deal with  $\langle \bar{u}_t, x \rangle$ , and thus the terms originating from  $y(x)$  do not make any new contributions to the concentration error.
Concentration for Mean of Second Term in Eq. (E.165). Finally, to deal with the second term on the right-hand side of Eq. (E.165), we note that we can follow the same proof, but instead obtaining an upper bound on the concentration of
$$
\| \delta_ {t} \| _ {2} ^ {2} \sum_ {\substack {p + r = 2, 4 \\ q + s = 2, 4}} \hat {\sigma} _ {\max } ^ {2} d ^ {\frac {p + q + r + s}{2}} \langle \delta_ {t}, x \rangle^ {p} \langle \delta_ {t} ^ {\prime}, x \rangle^ {q} \langle \bar {u} _ {t}, x \rangle^ {r} \langle \bar {u} _ {t} ^ {\prime}, x \rangle^ {s} \tag{E.183}
$$
and as before, the contribution of the terms originating from  $y(x)$  is at most the contribution of the terms in Eq. (E.183). Due to the additional  $\left\| \delta_t \right\|_2^2$  factor, we can in fact ignore the contribution of the terms in Eq. (E.183), as the dominant terms will be less than the dominant terms from Eq. (E.168).
Concluding to Bound  $C_t$ . In summary, with probability at least  $1 - \frac{1}{d^{\Omega(\log d)}}$ , for all choices of  $\delta_t, \delta_t'$  we have
$$
| \mathcal {C} (g) | \lesssim \hat {\sigma} _ {\max } ^ {2} (\log d) ^ {O (1)} \cdot \left(\sqrt {\frac {d}{n}} \| \delta_ {t} \| _ {2} + \frac {d ^ {2}}{n} B _ {2} ^ {2} \overline {{\Delta}} \| \delta_ {t} \| _ {2} ^ {2} + \frac {d ^ {2 . 5}}{n} \overline {{\Delta}} ^ {2} B _ {2} ^ {2} \| \delta_ {t} \| _ {2} + \frac {d ^ {4}}{n} B _ {2} ^ {4} \overline {{\Delta}} ^ {2} \| \delta_ {t} \| _ {2} ^ {2}\right) \tag {E.184}
$$
where  $g(x) = \langle \mathrm{grad}_{\tilde{u}}\ell^{(x)}(\hat{\rho}_t),\delta_t\rangle$  . Since  $C_t$  is exactly equal to
$$
\frac {1}{n} \sum_ {i = 1} ^ {n} g \left(x _ {i}\right) - \underset {x \sim \mathbb {S} ^ {d - 1}} {\mathbb {E}} [ g (x) ] \tag {E.185}
$$
up to signs and constant factors, we therefore have
$$
\left| C _ {t} \right| \lesssim \hat {\sigma} _ {\max } ^ {2} (\log d) ^ {O (1)} \cdot \left(\sqrt {\frac {d}{n}} \| \delta_ {t} \| _ {2} + \frac {d ^ {2}}{n} B _ {2} ^ {2} \bar {\Delta} \| \delta_ {t} \| _ {2} ^ {2} + \frac {d ^ {2 . 5}}{n} \bar {\Delta} ^ {2} B _ {2} ^ {2} \| \delta_ {t} \| _ {2} + \frac {d ^ {4}}{n} B _ {2} ^ {4} \bar {\Delta} ^ {2} \| \delta_ {t} \| _ {2} ^ {2}\right). \tag {E.186}
$$
E.4 Additional Lemmas
Lemma E.12 (Uniform Bound on  $\sigma$  and  $\sigma'$ ). For any  $t \in [-1, 1]$ , we have  $|\sigma(t)| \leq |\hat{\sigma}_{2,d}| \sqrt{N_{2,d}} + |\hat{\sigma}_{4,d}| \sqrt{N_{4,d}}$ , and additionally,  $|\sigma'(t)| \lesssim |\hat{\sigma}_{2,d}| \sqrt{N_{2,d}} + |\hat{\sigma}_{4,d}| \sqrt{N_{4,d}}$ . Finally, we have  $|\sigma''(t)| \lesssim |\hat{\sigma}_{2,d}| \sqrt{N_{2,d}} + |\hat{\sigma}_{4,d}| \sqrt{N_{4,d}}$ .
Proof of Lemma E.12. The first statement is by the definition of  $\sigma$ , and because  $|P_{2,d}(t)|, |P_{4,d}(t)| \leq 1$  for any  $t \in [-1,1]$ , by Atkinson and Han [12], Eq. (2.116). The second and third statements are by the definition of  $\sigma$  and using Eq. (C.4).
Lemma E.13 (Error from Finite Width). Let  $\rho$  be a distribution on  $\mathbb{S}^{d - 1}$ , and let  $\bar{\rho}$  be a distribution on  $\mathbb{S}^{d - 1}$  of support at most  $m$ , where the elements of the support of  $\bar{\rho}$  are sampled i.i.d. from  $\rho$ . Additionally, let  $\delta > 0$ . Then, for some absolute constant  $C > 0$ ,
$$
\underset {x \sim \mathbb {S} ^ {d - 1}} {\mathbb {E}} [ (f _ {\rho} (x) - f _ {\bar {\rho}} (x)) ^ {2} ] \lesssim \frac {(\hat {\sigma} _ {2 , d} ^ {2} + \hat {\sigma} _ {4 , d} ^ {2}) (d ^ {2} + \log \frac {1}{\delta}) (\log d) ^ {C}}{m} + \frac {\hat {\sigma} _ {2 , d} ^ {2} + \hat {\sigma} _ {4 , d} ^ {2}}{d ^ {\Omega (\log d)}} \tag {E.187}
$$
with probability at least  $1 - \delta$ , where  $f_{\rho}$  and  $f_{\bar{\rho}}$  are defined as in Eq. (3.1).
Proof of Lemma E.13. We can write  $f_{\rho}(x) = \mathbb{E}_{u\sim \rho}[\sigma (\langle u,x\rangle)]$ , and  $f_{\bar{\rho}}(x) = \frac{1}{m}\sum_{i = 1}^{m}\sigma (\langle u_i,x\rangle)$ . Now, let  $B\in (0,1)$ , which we will choose appropriately later, and define the function  $\varphi_B:[-1,1]\to [-1,1]$  by
$$
\varphi_ {B} (t) = \left\{ \begin{array}{l l} t & t \in [ - B, B ] \\ B & t > B \\ - B & t <   - B \end{array} \right. \tag {E.188}
$$
Additionally, define  $\tilde{f}_{\rho}(x) = \mathbb{E}_{u\sim \rho}[\sigma (\varphi_B(\langle u,x\rangle))]$ , and define  $\tilde{f}_{\bar{\rho}}(x) = \frac{1}{m}\sum_{i = 1}^{m}\sigma (\varphi_B(\langle u_i,x\rangle))$ . Let us analyze the error incurred from replacing  $f_{\rho}$  be  $\tilde{f}_{\rho}$  and  $f_{\bar{\rho}}$  by  $\tilde{f}_{\bar{\rho}}$ . Observe that
$$
\underset {x \sim \mathbb {S} ^ {d - 1}} {\mathbb {E}} \left[ \left(f _ {\rho} (x) - \tilde {f} _ {\rho} (x)\right) ^ {2} \right] = \underset {x \sim \mathbb {S} ^ {d - 1}} {\mathbb {E}} \left(f _ {\rho} (x)\right) ^ {2} - 2 \underset {x \sim \mathbb {S} ^ {d - 1}} {\mathbb {E}} f _ {\rho} (x) \tilde {f} _ {\rho} (x) + \underset {x \sim \mathbb {S} ^ {d - 1}} {\mathbb {E}} \left(\tilde {f} _ {\rho} (x)\right) ^ {2} \tag {E.189}
$$
We can simplify the second term on the right-hand side by expanding  $f_{\rho}$  and  $\tilde{f}_{\rho}$ :
$$
\begin{array}{l} \underset {x \sim \mathbb {S} ^ {d - 1}} {\mathbb {E}} f _ {\rho} (x) \tilde {f} _ {\rho} (x) = \underset {x \sim \mathbb {S} ^ {d - 1}} {\mathbb {E}} \underset {u _ {1}, u _ {2} \sim \rho} {\mathbb {E}} \sigma (\langle u _ {1}, x \rangle) \sigma (\varphi_ {B} (\langle u _ {2}, x \rangle)) (E.190) \\ = \underset {u _ {1}, u _ {2} \sim \rho} {\mathbb {E}} \underset {x \sim \mathbb {S} ^ {d - 1}} {\mathbb {E}} \sigma \left(\langle u _ {1}, x \rangle\right) \sigma \left(\varphi_ {B} \left(\langle u _ {2}, x \rangle\right)\right) (E.191) \\ \end{array}
$$
To simplify the inner expectation, we use Proposition I.5 to find that  $\varphi_B(\langle u_2,x\rangle) = \langle u_2,x\rangle$  with probability at least  $1 - 2\exp \left(-\frac{B^2d}{2}\right)$ . Thus,
$$
\left| \underset {x \sim \mathbb {S} ^ {d - 1}} {\mathbb {E}} \left(f _ {\rho} (x)\right) ^ {2} - \underset {x \sim \mathbb {S} ^ {d - 1}} {\mathbb {E}} f _ {\rho} (x) \tilde {f} _ {\rho} (x) \right| \tag {E.192}
$$
$$
\begin{array}{l} \leq \mathbb {E} _ {u _ {1}, u _ {2} \sim \rho} \mathbb {E} _ {x \sim \mathbb {S} ^ {d - 1}} | \sigma (\langle u _ {1}, x \rangle) | \left| \sigma \left(\varphi_ {B} (\langle u _ {2}, x \rangle)\right) - \sigma (\langle u _ {2}, x \rangle) \right| (E.193) \\ \leq \left(| \hat {\sigma} _ {2, d} | \sqrt {N _ {2 , d}} + | \hat {\sigma} _ {4, d} | \sqrt {N _ {4 , d}}\right) \underset {u _ {2} \sim \rho} {\mathbb {E}} \underset {x \sim \mathbb {S} ^ {d - 1}} {\mathbb {E}} \left| \sigma \left(\varphi_ {B} (\langle u _ {2}, x \rangle)\right) - \sigma (\langle u _ {2}, x \rangle) \right| (By Lemma E.12) \\ \end{array}
$$
$$
\lesssim \exp \left(- \frac {B ^ {2} d}{2}\right) \cdot \left(\hat {\sigma} _ {2, d} ^ {2} N _ {2, d} + \hat {\sigma} _ {4, d} ^ {2} N _ {4, d}\right) \quad (\text {B y})
$$
$$
\left| \underset {x \sim \mathbb {S} ^ {d - 1}} {\mathbb {E}} \left(f _ {\rho} (x)\right) ^ {2} - \underset {x \sim \mathbb {S} ^ {d - 1}} {\mathbb {E}} \left(\tilde {f} _ {\rho} (x)\right) ^ {2} \right| \lesssim \exp \left(- \frac {B ^ {2} d}{2}\right) \cdot \left(\hat {\sigma} _ {2, d} ^ {2} N _ {2, d} + \hat {\sigma} _ {4, d} ^ {2} N _ {4, d}\right) \tag {E.194}
$$
$$
\underset {x \sim \mathbb {S} ^ {d - 1}} {\mathbb {E}} \left[ \left(f _ {\rho} (x) - \tilde {f} _ {\rho} (x)\right) ^ {2} \right] \lesssim \exp \left(- \frac {B ^ {2} d}{2}\right) \cdot \left(\hat {\sigma} _ {2, d} ^ {2} N _ {2, d} + \hat {\sigma} _ {4, d} ^ {2} N _ {4, d}\right) \tag {E.195}
$$
By the same argument, we can show that
$$
\underset {x \sim \mathbb {S} ^ {d - 1}} {\mathbb {E}} \left[ \left(f _ {\bar {\rho}} (x) - \tilde {f} _ {\bar {\rho}} (x)\right) ^ {2} \right] \lesssim \exp \left(- \frac {B ^ {2} d}{2}\right) \cdot \left(\hat {\sigma} _ {2, d} ^ {2} N _ {2, d} + \hat {\sigma} _ {4, d} ^ {2} N _ {4, d}\right) \tag {E.196}
$$