SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \left(\operatorname {g r a d} _ {\tilde {u} _ {t}} \widehat {L} \left(\tilde {\rho} _ {t}\right) - \operatorname {g r a d} _ {\hat {u} _ {\eta_ {t}}} \widehat {L} \left(\hat {\rho} _ {t}\right)\right) \cdot \tilde {\delta} _ {t} (F.19) \\ = \frac {1}{n} \sum_ {i = 1} ^ {n} \left(\left(f _ {\tilde {\rho} _ {t}} \left(x _ {i}\right) - y \left(x _ {i}\right)\right) \sigma^ {\prime} \left(\tilde {u} _ {t} \cdot x _ {i}\right) - \left(f _ {\hat {\rho} _ {\eta t}} \left(x _ {i}\right) - y \left(x _ {i}\right)\right) \sigma^ {\prime} \left(\hat {u} _ {\eta t} \cdot x _ {i}\right)\right) x _ {i} \cdot \tilde {\delta} _ {t}. (F.20) \\ \end{array}
$$
We proceed by first showing using Lemma I.7 that we do not incur much error when we replace the empirical average over the  $x_{i}$  by an expectation over  $x\sim \mathbb{S}^{d - 1}$ . We can separate the right hand side of Eq. (F.19) into two parts:
$$
S _ {1} = \frac {1}{n} \sum_ {i = 1} ^ {n} \left(f _ {\tilde {\rho} _ {t}} \left(x _ {i}\right) \sigma^ {\prime} \left(\tilde {u} _ {t} \cdot x _ {i}\right) - f _ {\hat {\rho} _ {\eta t}} \left(x _ {i}\right) \sigma^ {\prime} \left(\hat {u} _ {\eta t} \cdot x _ {i}\right)\right) (\tilde {\delta} _ {t} \cdot x _ {i}) \tag {F.21}
$$
$$
S _ {2} = \frac {1}{n} \sum_ {i = 1} ^ {n} \left(y \left(x _ {i}\right) \sigma^ {\prime} \left(\tilde {u} _ {t} \cdot x _ {i}\right) - y \left(x _ {i}\right) \sigma^ {\prime} \left(\hat {u} _ {\eta t} \cdot x _ {i}\right)\right) \left(\tilde {\delta} _ {t} \cdot x _ {i}\right). \tag {F.22}
$$
We wish to show that  $S_{1}$  and  $S_{2}$  are very close to
$$
M _ {1} = \mathbb {E} _ {x \sim \mathbb {S} ^ {d - 1}} \left[ \left(f _ {\tilde {\rho} _ {t}} (x) \sigma^ {\prime} (\tilde {u} _ {t} \cdot x) - f _ {\hat {\rho} _ {\eta t}} (x) \sigma^ {\prime} (\hat {u} _ {\eta t} \cdot x)\right) (\tilde {\delta} _ {t} \cdot x) \right] \tag {F.23}
$$
$$
M _ {2} = \underset {x \sim \mathbb {S} ^ {d - 1}} {\mathbb {E}} \left[ \left(y \left(x _ {i}\right) \sigma^ {\prime} \left(\tilde {u} _ {t} \cdot x _ {i}\right) - y \left(x _ {i}\right) \sigma^ {\prime} \left(\hat {u} _ {\eta t} \cdot x _ {i}\right)\right) \left(\tilde {\delta} _ {t} \cdot x _ {i}\right) \right] \tag {F.24}
$$
respectively, with high probability — then, we will obtain bounds on  $M_1$  and  $M_2$ . We will show that  $S_1$  and  $M_1$  are close — the proof that  $S_2$  and  $M_2$  are close is similar.
Expanding into Monomials Our proof strategy is to expand the following quantity, for  $x \in \mathbb{S}^{d-1}$ :
$$
G (x) = \left(f _ {\tilde {\rho} _ {t}} (x) \sigma^ {\prime} (\tilde {u} _ {t} \cdot x) - f _ {\hat {\rho} _ {\eta t}} (x) \sigma^ {\prime} (\hat {u} _ {\eta t} \cdot x)\right) (\tilde {\delta} _ {t} \cdot x) \tag {F.25}
$$
and obtain concentration bounds for each of the terms. In this proof, we let  $\Gamma$  denote the natural coupling between  $\bar{\rho},\hat{\rho},\tilde{\rho}$  which is the distribution over triples  $(\bar{u}_{\eta t}(\chi),\hat{u}_{\eta t}(\chi),\tilde{u}_{\eta t}(\chi))$ , where  $\chi$  is sampled from  $\{\chi_1,\ldots ,\chi_m\}$ . Then, we can write  $G(x)$  as
$$
G (x) = \underset {\tilde {u} _ {\eta t} ^ {\prime}, \hat {u} _ {\eta t} ^ {\prime}, \tilde {u} _ {t} ^ {\prime} \sim \Gamma} {\mathbb {E}} \left[ \left(\sigma \left(\tilde {u} _ {t} ^ {\prime} \cdot x\right) \sigma^ {\prime} \left(\tilde {u} _ {t} \cdot x\right) - \sigma \left(\hat {u} _ {\eta t} ^ {\prime} \cdot x\right) \sigma^ {\prime} \left(\hat {u} _ {\eta t} \cdot x\right)\right) \cdot (\tilde {\delta} _ {t} \cdot x) \right]. \tag {F.26}
$$
For convenience, let us define
$$
g (x) = \left(\sigma \left(\tilde {u} _ {t} ^ {\prime} \cdot x\right) \sigma^ {\prime} \left(\tilde {u} _ {t} \cdot x\right) - \sigma \left(\hat {u} _ {\eta t} ^ {\prime} \cdot x\right) \sigma^ {\prime} \left(\hat {u} _ {\eta t} \cdot x\right)\right) \cdot (\tilde {\delta} _ {t} \cdot x). \tag {F.27}
$$
Clearly we have  $G(x) = \mathbb{E}_{\bar{u}_{\eta t}^{\prime},\hat{u}_{\eta t}^{\prime},\tilde{u}_{t}^{\prime}\sim \Gamma}[g(x)]$ . Now, we consider the effect of expanding  $g(x)$  by expanding  $\sigma$  and  $\sigma^\prime$  into monomials that involve  $(\tilde{u}_t^\prime \cdot x)$ ,  $(\tilde{u}_t\cdot x)$ ,  $(\hat{u}_{\eta t}^{\prime}\cdot x)$  and  $(\hat{u}_{\eta t}\cdot x)$ . Since  $\sigma$
is an even polynomial of degree 4 and  $\sigma'$  is an odd polynomial of degree 3, we can write  $g(x)$  as a linear combination of terms of the following form:
$$
\left(\left(\tilde {u} _ {t} ^ {\prime} \cdot x\right) ^ {i _ {1}} \left(\tilde {u} _ {t} \cdot x\right) ^ {i _ {2}} - \left(\hat {u} _ {\eta t} ^ {\prime} \cdot x\right) ^ {i _ {1}} \left(\hat {u} _ {\eta t} \cdot x\right) ^ {i _ {2}}\right) (\tilde {\delta} _ {t} \cdot x) \tag {F.28}
$$
where  $i_1 = 0,2,4$  and  $i_2 = 1,3$  — each such term also has a constant factor of  $O(\hat{\sigma}_{i_1,d}\hat{\sigma}_{i_2,d}\sqrt{N_{i_1,d}}\sqrt{N_{i_2,d}})$ . We now consider the contribution of Eq. (F.28) for each value of  $i_1, i_2$ . In the case where  $i_1 = 0$ , we can rewrite Eq. (F.28) as  $(\tilde{\delta}_t \cdot x)^2$  when  $i_2 = 1$ , and
$$
\left. \left((\tilde {u} _ {t} \cdot x) ^ {2} + (\tilde {u} _ {t} \cdot x) (\hat {u} _ {\eta t} \cdot x) + (\hat {u} _ {\eta t} \cdot x) ^ {2}\right) (\tilde {\delta} _ {t} \cdot x) ^ {2} \right. \tag {F.29}
$$
when  $i_2 = 3$ . In the case where  $i_1$  and  $i_2$  are both nonzero, letting  $\tilde{\delta}_t' = \tilde{u}_t' - \hat{u}_{\eta t}'$  we can write Eq. (F.28) as
$$
\begin{array}{l} \left(\left(\tilde {u} _ {t} ^ {\prime} \cdot x\right) ^ {i _ {1}} \left(\tilde {u} _ {t} \cdot x\right) ^ {i _ {2}} - \left(\hat {u} _ {\eta t} ^ {\prime} \cdot x\right) ^ {i _ {1}} \left(\hat {u} _ {\eta t} \cdot x\right) ^ {i _ {2}}\right) (\tilde {\delta} _ {t} \cdot x) (F.30) \\ = \left(\left(\tilde {u} _ {t} ^ {\prime} \cdot x\right) ^ {i _ {1}} - \left(\hat {u} _ {\eta t} ^ {\prime} \cdot x\right) ^ {i _ {1}}\right) \left(\tilde {u} _ {t} \cdot x\right) ^ {i _ {2}} \left(\tilde {\delta} _ {t} \cdot x\right) (F.31) \\ + \left(\hat {u} _ {\eta t} ^ {\prime} \cdot x\right) ^ {i _ {1}} \left(\left(\tilde {u} _ {t} \cdot x\right) ^ {i _ {2}} - \left(\hat {u} _ {\eta t} \cdot x\right) ^ {i _ {2}}\right) \left(\tilde {\delta} _ {t} \cdot x\right) (F.32) \\ = \sum_ {\substack {i _ {3} + i _ {4} = i _ {1} - 1 \\ i _ {3}, i _ {4} \geq 0}} \left(\tilde {u} _ {t} ^ {\prime} \cdot x\right) ^ {i _ {3}} \left(\hat {u} _ {\eta t} ^ {\prime} \cdot x\right) ^ {i _ {4}} \left(\tilde {u} _ {t} ^ {\prime} \cdot x - \hat {u} _ {\eta t} ^ {\prime} \cdot x\right) \left(\tilde {u} _ {t} \cdot x\right) ^ {i _ {2}} \left(\tilde {\delta} _ {t} \cdot x\right) (F.33) \\ + \sum_ {\substack {i _ {3} + i _ {4} = i _ {2} - 1 \\ i _ {3}, i _ {4} \geq 0}} \left(\hat {u} _ {\eta t} ^ {\prime} \cdot x\right) ^ {i _ {1}} \left(\tilde {u} _ {t} \cdot x\right) ^ {i _ {3}} \left(\hat {u} _ {\eta t} \cdot x\right) ^ {i _ {4}} \left(\tilde {u} _ {t} \cdot x - \hat {u} _ {\eta t} \cdot x\right) \left(\tilde {\delta} _ {t} \cdot x\right) (F.34) \\ = \sum_ {\substack {i _ {3} + i _ {4} = i _ {1} - 1 \\ i _ {3}, i _ {4} \geq 0}} \left(\tilde {u} _ {t} ^ {\prime} \cdot x\right) ^ {i _ {3}} \left(\tilde {u} _ {\eta t} ^ {\prime} \cdot x\right) ^ {i _ {4}} \left(\tilde {u} _ {t} \cdot x\right) ^ {i _ {2}} \left(\tilde {\delta} _ {t} \cdot x\right) \left(\tilde {\delta} _ {t} ^ {\prime} \cdot x\right) (F.35) \\ + \sum_ {i _ {3} + i _ {4} = i _ {2} - 1 \atop i _ {3}, i _ {4} \geq 0} \left(\hat {u} _ {\eta t} ^ {\prime} \cdot x\right) ^ {i _ {1}} \left(\tilde {u} _ {t} \cdot x\right) ^ {i _ {3}} \left(\hat {u} _ {\eta t} \cdot x\right) ^ {i _ {4}} \left(\tilde {\delta} _ {t} \cdot x\right) ^ {2}. (F.36) \\ \end{array}
$$
In summary, Eq. (F.28) can be written as a linear combination of terms of the form
$$
\left(\tilde {u} _ {t} ^ {\prime} \cdot x\right) ^ {i _ {3}} \left(\hat {u} _ {\eta t} \cdot x\right) ^ {i _ {4}} \left(\tilde {u} _ {t} \cdot x\right) ^ {i _ {2}} \left(\tilde {\delta} _ {t} \cdot x\right) \left(\tilde {\delta} _ {t} ^ {\prime} \cdot x\right) \tag {F.37}
$$
where  $i_3 + i_4 = 1,3$  and  $i_2 = 1,3$ , and terms of the form
$$
\left(\hat {u} _ {\eta t} ^ {\prime} \cdot x\right) ^ {i _ {1}} \left(\tilde {u} _ {t} \cdot x\right) ^ {i _ {3}} \left(\hat {u} _ {\eta t} \cdot x\right) ^ {i _ {4}} \left(\tilde {\delta} _ {t} \cdot x\right) ^ {2} \tag {F.38}
$$
where  $i_1 = 0,2,4$  and  $i_3 + i_4 = 0,2$ .
Contribution of Eq. (F.38) to Concentration Error In Eq. (F.38), we can write  $\tilde{u}_t = \hat{u}_{\eta t} + \tilde{\delta}_t$ , and thus write Eq. (F.38) as a linear combination of terms of the form
$$
\left(\hat {u} _ {\eta t} ^ {\prime} \cdot x\right) ^ {i _ {1}} \left(\hat {u} _ {\eta t} \cdot x\right) ^ {i _ {2}} \left(\tilde {\delta} _ {t} \cdot x\right) ^ {2 + i _ {3}} \tag {F.39}
$$
where  $i_1 = 0, 2, 4$  and  $i_2 + i_3 = 0, 2$ . Next, in order to use Lemma I.7, we write  $\hat{u}_{\eta t} = \bar{u}_{\eta t} + \delta_t$  and expand (recalling the definition of  $\delta_t$  from Lemma 5.4). Thus, Eq. (F.38) is a linear combination of terms of the form
$$
\left(\bar {u} _ {\eta t} ^ {\prime} \cdot x\right) ^ {i _ {1}} \left(\delta_ {\eta t} ^ {\prime} \cdot x\right) ^ {i _ {2}} \left(\bar {u} _ {\eta t} \cdot x\right) ^ {i _ {3}} \left(\delta_ {\eta t} \cdot x\right) ^ {i _ {4}} \left(\tilde {\delta} _ {t} \cdot x\right) ^ {2 + i _ {5}} \tag {F.40}
$$
where  $i_1 + i_2 = 0, 2, 4$ , and  $i_3 + i_4 + i_5 = 0, 2$ . Finally, we obtain a concentration bound for Eq. (F.40) using Lemma I.7:
$$
\begin{array}{l} \left| \frac {1}{n} \sum_ {i = 1} ^ {n} \left(\bar {u} _ {\eta t} ^ {\prime} \cdot x _ {i}\right) ^ {i _ {1}} \left(\delta_ {\eta t} ^ {\prime} \cdot x _ {i}\right) ^ {i _ {2}} \left(\bar {u} _ {\eta t} \cdot x _ {i}\right) ^ {i _ {3}} \left(\delta_ {\eta t} \cdot x _ {i}\right) ^ {i _ {4}} \left(\tilde {\delta} _ {t} \cdot x _ {i}\right) ^ {2 + i _ {5}} \right. (F.41) \\ - \underset {x \sim \mathbb {S} ^ {d - 1}} {\mathbb {E}} \left[ \left(\bar {u} _ {\eta t} ^ {\prime} \cdot x\right) ^ {i _ {1}} \left(\delta_ {\eta t} ^ {\prime} \cdot x\right) ^ {i _ {2}} \left(\bar {u} _ {\eta t} \cdot x\right) ^ {i _ {3}} \left(\delta_ {\eta t} \cdot x\right) ^ {i _ {4}} \left(\tilde {\delta} _ {t} \cdot x\right) ^ {2 + i _ {5}} \right] \Bigg | (F.42) \\ \lesssim \left(\log d\right) ^ {O (1)} \frac {\left\| \delta_ {\eta t} ^ {\prime} \right\| _ {2} ^ {i _ {2}} \left\| \delta_ {\eta t} \right\| _ {2} ^ {i _ {4}} \left\| \tilde {\delta} _ {t} \right\| _ {2} ^ {2 + i _ {5}}}{d ^ {(i _ {1} + i _ {2} + i _ {3} + i _ {4} + i _ {5}) / 2 + 1}} \left(\sqrt {\frac {d}{n}} + \frac {d ^ {(i _ {2} + i _ {4} + i _ {5}) / 2 + 1}}{n} + \frac {1}{d ^ {\Omega (\log d)}}\right). (F.43) \\ \end{array}
$$
Here, when applying Lemma I.7,  $\bar{u}_{\eta t}$  and  $\bar{u}_{\eta t}^{\prime}$  take the role of  $w_{1}$  and  $w_{2}$ , while  $\delta_{\eta t}^{\prime} / \| \delta_{\eta t}^{\prime}\|_{2}$ ,  $\delta_{\eta t} / \| \delta_{\eta t}\|_{2}$  and  $\tilde{\delta}_t / \| \tilde{\delta}_t\|_2$  take the role of  $u_{1}, u_{2}, u_{3}$  and  $u_{4}$  (we let  $u_{3} = u_{4} = \tilde{\delta}_{t} / \| \tilde{\delta}_{t}\|_{2}$  and choose  $p_3$  and  $p_4$  so that  $p_3 + p_4 = 2 + i_5$ ). Note that we implicitly perform a union bound over  $\bar{u}_{\eta t}^{\prime}$  and  $\delta_{\eta t}^{\prime}$  drawn from the support of  $f_{\hat{\rho}_{\eta t}}$  — this does not affect the failure probability, since the failure probability for the result of Lemma I.7 is  $\frac{1}{d^{\Omega(\log d)}}$ , and it is assumed in Theorem 3.5 that  $m \leq d^{C}$  for some universal constant  $C$ . When applying Lemma I.7 at later points in this proof, we implicitly perform this union bound.
In order to compute the final contribution of each such term to the concentration error between  $S_{1}$  and  $M_{1}$ , we must first (1) incorporate the coefficients of  $\sigma$  and  $\sigma'$  with respect to the Legendre polynomials  $P_{2,d}$  and  $P_{4,d}$ , (2) take the average of  $\| \delta_{\eta t}' \|_2^b$  with respect to  $\chi$  (which is implicit throughout this proof), and (3) use the fact that  $n \gtrsim d^3 (\log d)^{O(1)}$ .
Recall that  $\sigma = \hat{\sigma}_{2,d}\sqrt{N_{2,d}} P_{2,d} + \hat{\sigma}_{4,d}\sqrt{N_{4,d}} P_{4,d}$ . Using Eq. (C.4) and the fact that  $N_{2,d} \asymp d^2$  and  $N_{4,d} \asymp d^4$  (by Equation (2.10) of Atkinson and Han [12]), we find that, up to constant factors, the coefficient of the  $0^{\text{th}}$ -order term in  $\sigma(t)$  can be bounded above by  $\max(\hat{\sigma}_{2,d}, \hat{\sigma}_{4,d})$ , the coefficient of the  $2^{\text{nd}}$ -order term in  $\sigma(t)$  can be bounded above by  $\max(\hat{\sigma}_{2,d}, \hat{\sigma}_{4,d})d$ , and the coefficient of the  $4^{\text{th}}$  order term can be bounded above by  $\hat{\sigma}_{4,d}d^2$ . Similarly, up to constant factors, the coefficient of the  $1^{\text{st}}$ -order term in  $\sigma'(t)$  can be bounded above by  $\max(\hat{\sigma}_{2,d}, \hat{\sigma}_{4,d})d$  and the coefficient of the  $3^{\text{rd}}$ -order term in  $\sigma'(t)$  can be bounded above by  $\max(\hat{\sigma}_{2,d}, \hat{\sigma}_{4,d})d^2$ . Thus, each term of degree  $k$  originating from  $\sigma$  contributes a factor of  $\max(\hat{\sigma}_{2,d}, \hat{\sigma}_{4,d})d^{k/2}$  to the coefficient of Eq. (F.38), and each term of degree  $k-1$  originating from  $\sigma'$  contributes a factor of  $\max(\hat{\sigma}_{2,d}, \hat{\sigma}_{4,d})d^{k/2}$  to the coefficient of Eq. (F.38). Finally, for each term in Eq. (F.40) of total degree  $i_1 + i_2 + i_3 + i_4 + i_5 + 2$ , note that  $\sigma$  and  $\sigma'$  together contribute a factor of degree  $i_1 + i_2 + i_3 + i_4 + i_5 + 1$ , since in Eq. (F.27), aside from the factors contributed by  $\sigma$  and  $\sigma'$ , only a single factor of  $\tilde{\delta}_t \cdot x$  is present.
Thus, to obtain the final contribution of Eq. (F.41) to the overall error between  $M_{1}$  and  $S_{1}$ , we multiply by an additional coefficient of  $\max (\hat{\sigma}_{2,d}^2,\hat{\sigma}_{4,d}^2)d^{(i_1 + i_2 + i_3 + i_4 + i_5) / 2 + 1}$ . Taking this into account, the contribution of Eq. (F.41) is
$$
\left(\hat {\sigma} _ {2, d} ^ {2} + \hat {\sigma} _ {4, d} ^ {2}\right) \left(\log d\right) ^ {O (1)} \| \delta_ {\eta t} ^ {\prime} \| _ {2} ^ {i _ {2}} \| \delta_ {\eta t} \| _ {2} ^ {i _ {4}} \| \tilde {\delta} _ {t} \| _ {2} ^ {2 + i _ {5}} \left(\sqrt {\frac {d}{n}} + \frac {d ^ {(i _ {2} + i _ {4} + i _ {5}) / 2 + 1}}{n} + \frac {1}{d ^ {\Omega (\log d)}}\right). \tag {F.44}
$$
Next, we simplify this using casework on  $i_2$ . For the case  $i_2 \leq 1$ , since  $i_4 + i_5 \leq 2$ , the bound is at most
$$
\left(\hat {\sigma} _ {2, d} ^ {2} + \hat {\sigma} _ {4, d} ^ {2}\right) \| \delta_ {\eta t} ^ {\prime} \| _ {2} ^ {i _ {2}} \| \delta_ {\eta t} \| _ {2} ^ {i _ {4}} \| \tilde {\delta} _ {t} \| _ {2} ^ {2 + i _ {5}} \lesssim \left(\hat {\sigma} _ {2, d} ^ {2} + \hat {\sigma} _ {4, d} ^ {2}\right) \| \tilde {\delta} _ {t} \| _ {2} ^ {2} \tag {F.45}
$$
since  $d^{(i_2 + i_4 + i_5) / 2 + 1} \leq d^{2.5}$ , and because  $n \gtrsim d^3 (\log d)^{O(1)}$ , and we eliminate a factor of  $\| \delta_{\eta t}' \|_2^{i_2} \| \delta_{\eta t} \|_2^{i_4} \| \tilde{\delta}_t \|_2^{i_5}$  because all of the particles have  $\ell_2$  norm at most 1. Additionally, for the case  $i_2 \geq 2$ , after taking the average of  $\| \delta_{\eta t}' \|_2^{i_2} \lesssim \| \delta_{\eta t}' \|_2^2$  across all  $\chi$ , the bound is at most
$$
\begin{array}{l} \left. \left(\hat {\sigma} _ {2, d} ^ {2} + \hat {\sigma} _ {4, d} ^ {2}\right) (\log d) ^ {O (1)} \bar {\Delta} _ {\eta t} ^ {2} \| \tilde {\delta} _ {t} \| _ {2} ^ {2} \left(\sqrt {\frac {d}{n}} + \frac {d ^ {(i _ {2} + i _ {4} + i _ {5}) / 2 + 1}}{n} + \frac {1}{d ^ {\Omega (\log d)}}\right) \right. \tag {F.46} \\ \lesssim \left(\hat {\sigma} _ {2, d} ^ {2} + \hat {\sigma} _ {4, d} ^ {2}\right) \overline {{\Delta}} _ {\eta t} ^ {2} \| \tilde {\delta} _ {t} \| _ {2} ^ {2} \left(d ^ {i _ {2} / 2 - 1} + O \left(\frac {1}{d}\right)\right) \\ (\mathrm {B . c .} n \gtrsim d ^ {3} (\log d) ^ {O (1)} \text {a n d} i _ {4} + i _ {5} \leq 2) \\ \lesssim \left(\hat {\sigma} _ {2, d} ^ {2} + \hat {\sigma} _ {4, d} ^ {2}\right) \| \tilde {\delta} _ {t} \| _ {2} ^ {2} \quad (\mathrm {B . c .} \overline {{\Delta}} _ {\eta t} \leq \frac {1}{\sqrt {d}} \text {a n d} i _ {2} \leq 4) \\ \end{array}
$$
where we have  $\overline{\Delta}_{\eta t} \leq \frac{1}{\sqrt{d}}$  because by Lemma E.3, we have that Assumption 5.3 holds as long as  $\eta t \leq T_{*,\epsilon}$ . In summary, we have shown that terms of the form Eq. (F.38) contribute at most  $(\hat{\sigma}_{2,d}^2 + \hat{\sigma}_{4,d}^2) \| \tilde{\delta}_t \|_2^2$ , up to constant factors, to the concentration error between  $S_1$  and  $M_2$ .
Contribution of Eq. (F.37) to Concentration Error We expand Eq. (F.37) by using  $\tilde{u}_t = \tilde{\delta}_t + \delta_{\eta t} + \bar{u}_{\eta t}, \hat{u}_{\eta t} = \delta_{\eta t} + \bar{u}_{\eta t}$  and  $\tilde{u}_t' = \tilde{\delta}_t' + \delta_{\eta t}' + \bar{u}_{\eta t}'$ . Thus, we can write Eq. (F.37) as a linear combination of terms of the form
$$
\left(\tilde {\delta} _ {t} \cdot x\right) ^ {1 + i _ {1}} \left(\tilde {\delta} _ {t} ^ {\prime} \cdot x\right) ^ {1 + i _ {2}} \left(\delta_ {\eta t} \cdot x\right) ^ {i _ {3}} \left(\delta_ {\eta t} ^ {\prime} \cdot x\right) ^ {i _ {4}} \left(\bar {u} _ {\eta t} \cdot x\right) ^ {i _ {5}} \left(\bar {u} _ {\eta t} ^ {\prime} \cdot x\right) ^ {i _ {6}} \tag {F.47}
$$
where  $i_1 + i_2 + i_3 + i_4 + i_5 + i_6 = 6$  (since the degrees of each of the monomials must be 8) and  $i_3 \leq 3$  (since each of these terms originates from  $(\tilde{u}_t' \cdot x)^{i_3} (\hat{u}_{\eta t}' \cdot x)^{i_4} (\tilde{u}_t \cdot x)^{i_2} (\tilde{\delta}_t \cdot x) (\tilde{\delta}_t' \cdot x)$  where  $i_2 \leq 3$ ). Using Lemma I.7, we obtain a concentration bound for Eq. (F.47):
$$
\begin{array}{l} \left| \frac {1}{n} \sum_ {i = 1} ^ {n} \left(\tilde {\delta} _ {t} \cdot x _ {i}\right) ^ {1 + i _ {1}} \left(\tilde {\delta} _ {t} ^ {\prime} \cdot x _ {i}\right) ^ {1 + i _ {2}} \left(\delta_ {\eta t} \cdot x _ {i}\right) ^ {i _ {3}} \left(\delta_ {\eta t} ^ {\prime} \cdot x _ {i}\right) ^ {i _ {4}} \left(\bar {u} _ {\eta t} \cdot x _ {i}\right) ^ {i _ {5}} \left(\bar {u} _ {\eta t} ^ {\prime} \cdot x _ {i}\right) ^ {i _ {6}} \right. (F.48) \\ - \underset {x \sim \mathbb {S} ^ {d - 1}} {\mathbb {E}} \left[ \left(\tilde {\delta} _ {t} \cdot x\right) ^ {1 + i _ {1}} \left(\tilde {\delta} _ {t} ^ {\prime} \cdot x\right) ^ {1 + i _ {2}} \left(\delta_ {\eta t} \cdot x\right) ^ {i _ {3}} \left(\delta_ {\eta t} ^ {\prime} \cdot x\right) ^ {i _ {4}} \left(\bar {u} _ {\eta t} \cdot x\right) ^ {i _ {5}} \left(\bar {u} _ {\eta t} ^ {\prime} \cdot x\right) ^ {i _ {6}} \right] \Bigg | (F.49) \\ \lesssim (\log d) ^ {O (1)} \frac {\| \tilde {\delta} _ {t} \| _ {2} ^ {1 + i _ {1}} \| \tilde {\delta} _ {t} ^ {\prime} \| _ {2} ^ {1 + i _ {2}} \| \delta_ {\eta t} \| _ {2} ^ {i _ {3}} \| \delta_ {\eta t} ^ {\prime} \| _ {2} ^ {i _ {4}}}{d ^ {1 + (i _ {1} + i _ {2} + i _ {3} + i _ {4} + i _ {5} + i _ {6}) / 2}} \left(\sqrt {\frac {d}{n}} + \frac {d ^ {1 + (i _ {1} + i _ {2} + i _ {3} + i _ {4}) / 2}}{n} + \frac {1}{d ^ {\Omega (\log d)}}\right). (F.50) \\ \end{array}
$$
As discussed in the previous case, after we consider the coefficients of  $\sigma(t)$  and  $\sigma'(t)$ , we find that the term in Eq. (F.47) contributes
$$
\left. \left(\hat {\sigma} _ {2, d} ^ {2} + \hat {\sigma} _ {4, d} ^ {2}\right) \left(\log d\right) ^ {O (1)} \| \tilde {\delta} _ {t} \| _ {2} ^ {1 + i _ {1}} \| \tilde {\delta} _ {t} ^ {\prime} \| _ {2} ^ {1 + i _ {2}} \| \delta_ {\eta t} \| _ {2} ^ {i _ {3}} \| \delta_ {\eta t} ^ {\prime} \| _ {2} ^ {i _ {4}} \left(\sqrt {\frac {d}{n}} + \frac {d ^ {1 + \left(i _ {1} + i _ {2} + i _ {3} + i _ {4}\right) / 2}}{n} + \frac {1}{d ^ {\Omega (\log d)}}\right). \right. \tag {F.51}
$$
Now, suppose  $i_5 + i_6 \geq 2$ . Then,  $i_1 + i_2 + i_3 + i_4 \leq 4$ , meaning the contribution to the concentration error is at most
$$
\left. \left(\hat {\sigma} _ {2, d} ^ {2} + \hat {\sigma} _ {4, d} ^ {2}\right) \| \tilde {\delta} _ {t} \| _ {2} \| \tilde {\delta} _ {t} ^ {\prime} \| _ {2} \leq \left(\hat {\sigma} _ {2, d} ^ {2} + \hat {\sigma} _ {4, d} ^ {2}\right) \max  _ {\chi} \| \tilde {\delta} _ {t} (\chi) \| _ {2} ^ {2} \right. \tag {F.52}
$$
using the fact that  $n \gtrsim d^3 (\log d)^C$  for a sufficiently large constant  $C$ , and  $\| \tilde{\delta}_t \|_2 \lesssim 1$  and  $\| \delta_{\eta t} \|_2 \lesssim 1$ . On the other hand, if  $i_5 + i_6 \leq 1$ , then  $i_1 + i_2 + i_3 + i_4 \geq 5$ , and since  $i_3 \leq 3$ , this implies that  $i_1 + i_2 + i_4 \geq 2$ . In this case, the bound is at most
$$
\begin{array}{l} \left. \left(\hat {\sigma} _ {2, d} ^ {2} + \hat {\sigma} _ {4, d} ^ {2}\right) \left(\log d\right) ^ {O (1)} \| \tilde {\delta} _ {t} \| _ {2} ^ {1 + i _ {1}} \| \tilde {\delta} _ {t} ^ {\prime} \| _ {2} ^ {1 + i _ {2}} \| \delta_ {\eta t} \| _ {2} ^ {i _ {3}} \| \delta_ {\eta t} ^ {\prime} \| _ {2} ^ {i _ {4}} \left(\sqrt {\frac {d}{n}} + \frac {d ^ {1 + \left(i _ {1} + i _ {2} + i _ {3} + i _ {4}\right) / 2}}{n} + \frac {1}{d ^ {\Omega (\log d)}}\right) \right. \tag {F.53} \\ \lesssim \left(\hat {\sigma} _ {2, d} ^ {2} + \hat {\sigma} _ {4, d} ^ {2}\right) \| \tilde {\delta} _ {t} \| _ {2} ^ {1 + i _ {1}} \| \tilde {\delta} _ {t} ^ {\prime} \| _ {2} ^ {1 + i _ {2}} \| \delta_ {\eta t} \| _ {2} ^ {i _ {3}} \| \delta_ {\eta t} ^ {\prime} \| _ {2} ^ {i _ {4}} d \\ (\mathrm {B . c .} i _ {1} + i _ {2} + i _ {3} + i _ {4} \leq 6 \text {a n d} n \gtrsim d ^ {3} (\log d) ^ {C}) \\ \lesssim \left(\hat {\sigma} _ {2, d} ^ {2} + \hat {\sigma} _ {4, d} ^ {2}\right) \max  _ {\chi} \| \tilde {\delta} _ {t} (\chi) \| _ {2} ^ {2} \cdot \frac {1}{d ^ {(i _ {1} + i _ {2}) / 2}} \| \delta_ {\eta t} ^ {\prime} \| _ {2} ^ {i _ {4}} d. \\ \left(\text {B . c .} \max  _ {\chi} \| \tilde {\delta} _ {t} (\chi) \| _ {2} \leq \frac {1}{\sqrt {d}} \text {a n d} \| \delta_ {\eta t} \| _ {2} \lesssim 1\right) \\ \end{array}
$$
Thus, if  $i_1 + i_2 \geq 2$ , then the contribution of this term to the overall concentration error is at most  $(\hat{\sigma}_{2,d}^2 + \hat{\sigma}_{4,d}^2) \max_{\chi} \| \tilde{\delta}_t(\chi) \|_2^2$ . On the other hand, if  $i_4 = 1$ , then we can take the average of  $\| \delta_{\eta t}' \|_2$  across  $\chi$ , and by Lemma E.3, we have that Assumption 5.3 holds for  $\eta t \leq T_{*,\epsilon}$ , meaning that  $\mathbb{E}_{\chi} \| \delta_{\eta t}' \|_2 \leq \overline{\Delta}_{\eta t} \leq \frac{1}{\sqrt{d}}$ . In this case, the contribution of this term to the overall concentration error is at most  $(\hat{\sigma}_{2,d}^2 + \hat{\sigma}_{4,d}^2) \max_{\chi} \| \tilde{\delta}_t(\chi) \|_2^2 \cdot \frac{1}{d^{(i_1 + i_2 + i_4)/2}} d = (\hat{\sigma}_{2,d}^2 + \hat{\sigma}_{4,d}^2) \max_{\chi} \| \tilde{\delta}_t(\chi) \|_2^2$ . Finally, when  $i_4 \geq 2$ , we can take the average of  $\| \delta_{\eta t}' \|_2^2$  over  $\chi$  — we have  $\mathbb{E}_{\chi} \| \delta_{\eta t}' \|_2^2 \leq \frac{1}{d}$ , and therefore the contribution of this term to the overall concentration error is at most
$$
\left(\hat {\sigma} _ {2, d} ^ {2} + \hat {\sigma} _ {4, d} ^ {2}\right) \max  _ {\chi} \| \tilde {\delta} _ {t} (\chi) \| _ {2} ^ {2} \cdot \frac {1}{d ^ {1 + \left(i _ {1} + i _ {2}\right) / 2}} d \leq \left(\hat {\sigma} _ {2, d} ^ {2} + \hat {\sigma} _ {4, d} ^ {2}\right) \max  _ {\chi} \| \tilde {\delta} _ {t} (\chi) \| _ {2} ^ {2}. \tag {F.54}
$$
In all cases, the contribution of Eq. (F.37) to the overall concentration error between  $S_{1}$  and  $M_{1}$  is at most
$$
\left(\hat {\sigma} _ {2, d} ^ {2} + \hat {\sigma} _ {4, d} ^ {2}\right) \max  _ {\chi} \| \tilde {\delta} _ {t} (\chi) \| _ {2} ^ {2}. \tag {F.55}
$$
Overall Concentration Error
In summary, the overall concentration error between  $S_{1}$  and  $M_{1}$  is that of  $G(x)$ . We have expanded  $G(x)$  into various monomials of dot products involving  $x$ , and shown that each monomial contributes at most  $(\hat{\sigma}_{2,d}^{2} + \hat{\sigma}_{4,d}^{2})\max_{\chi}\|\tilde{\delta}_{t}(\chi)\|_{2}^{2}$  to the concentration error. Thus, with high probability,