SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\tau^ {2} \leq \frac {1 + \log^ {k / 2} (4 q (c d) ^ {k ^ {\star} / 2})}{(c d) ^ {k ^ {\star} / 2}} \lesssim \frac {\log^ {k ^ {\star} / 2} (q d)}{d ^ {k ^ {\star} / 2}}.
$$
Lemma 20 (Rosenthal-Burkholder-Pinelis Inequality [44]). Let  $\{Y_i\}_{i=0}^n$  be a martingale with martingale difference sequence  $\{X_i\}_{i=1}^n$  where  $X_i = Y_i - Y_{i-1}$ . Let
$$
\langle Y \rangle = \sum_ {i = 1} ^ {n} \mathbb {E} [ \| X _ {i} \| ^ {2} | \mathcal {F} _ {i - 1} ]
$$
denote the predictable quadratic variation. Then there exists an absolute constant  $C$  such that for all  $p$ ,
$$
\| Y _ {n} \| _ {p} \leq C \Bigg [ \sqrt {p \| \langle Y \rangle \| _ {p / 2}} + p \left\| \max _ {i} \| X _ {i} \| \right\| _ {p} \Bigg ].
$$
The above inequality is found in Pinelis [44, Theorem 4.1]. It is often combined with the following simple lemma:
Lemma 21. For any random variables  $X_{1},\ldots ,X_{n}$
$$
\left\| \max _ {i} \| X _ {i} \| \right\| _ {p} \leq \left(\sum_ {i = 1} ^ {n} \| X _ {i} \| _ {p} ^ {p}\right) ^ {1 / p}.
$$
This has the immediate corollary:
Lemma 22. Let  $\{Y_i\}_{i=0}^n$  be a martingale with martingale difference sequence  $\{X_i\}_{i=1}^n$  where  $X_i = Y_i - Y_{i-1}$ . Let  $\langle Y \rangle = \sum_{i=1}^n \mathbb{E}[\|X_i\|^2 | \mathcal{F}_{i-1}]$  denote the predictable quadratic variation. Then there exists an absolute constant  $C$  such that for all  $p$ ,
$$
\left\| Y _ {n} \right\| _ {p} \leq C \left[ \sqrt {p \| \langle Y \rangle \| _ {p / 2}} + p n ^ {1 / p} \max  _ {i} \left\| X _ {i} \right\| _ {p} \right].
$$
We will often use the following corollary of Holder's inequality to bound the operator norm of a product of two random variables when one has polynomial tails:
Lemma 23. Let  $X, Y$  be random variables with  $\| Y \|_p \leq \sigma_Y p^C$ . Then,
$$
\mathbb {E} [ X Y ] \leq \| X \| _ {1} \cdot \sigma_ {Y} \cdot (2 e) ^ {C} \cdot \max \left(1, \frac {1}{C} \log \left(\frac {\| X \| _ {2}}{\| X \| _ {1}}\right)\right) ^ {C}.
$$
Proof. Fix  $\epsilon \in [0,1]$ . Then using Holder's inequality with  $1 = 1 - \epsilon + \frac{\epsilon}{2} + \frac{\epsilon}{2}$  gives:
$$
\mathbb {E} [ X Y ] = \mathbb {E} [ X ^ {1 - \epsilon} X ^ {\epsilon} Y ] \leq \| X \| _ {1} ^ {1 - \epsilon} \| X \| _ {2} ^ {\epsilon} \| Y \| _ {2 / \epsilon}.
$$
Using the fact that  $X, Y$  have polynomial tails we can bound this by
$$
\mathbb {E} [ X Y ] = \mathbb {E} \left[ X ^ {1 - \epsilon} X ^ {\epsilon} Y \right] \leq \| X \| _ {1} ^ {1 - \epsilon} \| X \| _ {2} ^ {\epsilon} \sigma_ {Y} (2 / \epsilon) ^ {C}.
$$
First, if  $\| X\| _2\geq e^C\| X\| _1$  , we can set  $\epsilon = \frac{C}{\log\left(\frac{\|X\|_2}{\|X\|_1}\right)}$  which gives
$$
\mathbb {E} [ X Y ] \leq \| X \| _ {1} \cdot \sigma_ {Y} \cdot \left(\frac {2 e}{C} \log \left(\frac {\| X \| _ {2}}{\| X \| _ {1}}\right)\right) ^ {C}.
$$
Next, if  $\| X\| _2\leq e^C\| X\| _1$  we can set  $\epsilon = 1$  which gives
$$
\mathbb {E} [ X Y ] \leq \| X \| _ {2} \| Y \| _ {2} \leq \| X \| _ {1} \sigma_ {Y} (2 e) ^ {C}
$$
Finally, the following basic lemma will allow us to easily convert between  $p$ -norm bounds and concentration inequalities:
Lemma 24. Let  $\delta \geq 0$  and let  $X$  be a mean zero random variable satisfying
$$
\mathbb {E} [ | X | ^ {p} ] ^ {1 / p} \leq \sigma_ {X} p ^ {C} \quad f o r \quad p = \frac {\log (1 / \delta)}{C}
$$
for some  $C$ . Then with probability at least  $1 - \delta$ ,  $|X| \leq \sigma_X(ep)^C$ .
Proof. Let  $\epsilon = \sigma_X(ep)^C$ . Then,
$$
\begin{array}{l} \mathbb {P} [ | X | \geq \epsilon ] = \mathbb {P} [ | X | ^ {p} \geq \epsilon^ {p} ] \\ \leq \frac {\mathbb {E} [ | X | ^ {p} ]}{\epsilon^ {p}} \\ \leq \frac {(\sigma_ {X}) ^ {p} p ^ {p C}}{\epsilon^ {p}} \\ = e ^ {- C p} \\ = \delta . \\ \end{array}
$$
D Additional Technical Lemmas
The following lemma extends Steins's lemma  $(\mathbb{E}_{x\sim N(0,1)}[xg(x)] = \mathbb{E}_{x\sim N(0,1)}[g'(x)])$  to the ultraspherical distribution  $\mu^{(d)}$  where  $\mu^{(d)}$  is the distribution of  $z_{1}$  when  $z\sim \mathrm{Unif}(S^{d - 1})$ :
Lemma 25 (Spherical Stein's Lemma). For any  $g \in L^2(\mu^{(d)})$ ,
$$
\mathbb {E} _ {z \sim \operatorname {U n i f} (S ^ {d - 1})} [ z _ {1} g (z _ {1}) ] = \frac {\mathbb {E} _ {z \sim \operatorname {U n i f} (S ^ {d + 1})} [ g ^ {\prime} (z _ {1}) ]}{d}
$$
where  $z_{1}$  denotes the first coordinate of  $z$ .
Proof. Recall that the density of  $z_{1}$  is equal to
$$
\frac {\left(1 - x ^ {2}\right) ^ {\frac {d - 3}{2}}}{C (d)} \quad \text {w h e r e} \quad C (d) := \frac {\sqrt {\pi} \cdot \Gamma (\frac {d - 1}{2})}{\Gamma (\frac {d}{2})}.
$$
$$
\mathbb {E} _ {z \sim \mathrm {U n i f} (S ^ {d - 1})} [ z _ {1} g (z _ {1}) ] = \frac {1}{C (d)} \int_ {- 1} ^ {1} z _ {1} g (z _ {1}) (1 - z _ {1} ^ {2}) ^ {\frac {d - 3}{2}} d z _ {1}.
$$
Now we can integrate by parts to get
$$
\begin{array}{l} \mathbb {E} _ {z \sim \operatorname {U n i f} \left(S ^ {d - 1}\right)} \left[ z _ {1} g \left(z _ {1}\right) \right] = \frac {1}{C (d)} \int_ {- 1} ^ {1} \frac {g ^ {\prime} \left(z _ {1}\right) \left(1 - z ^ {2}\right) ^ {\frac {d - 1}{2}}}{d - 1} d z _ {1} \\ = \frac {C (d + 2)}{C (d) (d - 1)} \mathbb {E} _ {z \sim \operatorname {U n i f} \left(S ^ {d + 1}\right)} \left[ g ^ {\prime} \left(z _ {1}\right) \right] \\ = \frac {1}{d} \mathbb {E} _ {z \sim \operatorname {U n i f} \left(S ^ {d + 1}\right)} \left[ g ^ {\prime} \left(z _ {1}\right) \right]. \\ \end{array}
$$
Lemma 26. For  $j \leq d / 4$ ,
$$
\mathbb {E} _ {z \sim \operatorname {U n i f} (S ^ {d - 1})} \left(\frac {z _ {1} ^ {k}}{(1 - z _ {1} ^ {2}) ^ {j}}\right) \lesssim \mathbb {E} _ {z \sim \operatorname {U n i f} (S ^ {d - 2 j - 1})} \left(z _ {1} ^ {k}\right).
$$
Proof. Recall that the PDF of  $\mu^{(d)}$  is
Using this we have that:
$$
\begin{array}{l} \mathbb {E} _ {z \sim \operatorname {U n i f} (S ^ {d - 1})} \left[ \frac {z _ {1} ^ {k}}{(1 - z _ {1} ^ {2}) ^ {j}} \right] = \frac {1}{C (d)} \int_ {- 1} ^ {1} \frac {x ^ {k}}{(1 - x ^ {2}) ^ {j}} (1 - x ^ {2}) ^ {\frac {d - 3}{2}} d x \\ = \frac {1}{C (d)} \int_ {- 1} ^ {1} x ^ {k} \left(1 - x ^ {2}\right) ^ {\frac {d - 2 j - 3}{2}} d x \\ = \frac {C (d - 2 j)}{C (d)} \mathbb {E} _ {z _ {1} \sim \mu^ {(d - 2 j)}} \left[ z _ {1} ^ {k} \right] \\ = \frac {\Gamma (\frac {d}{2}) \Gamma (\frac {d - 2 j - 1}{2})}{\Gamma (\frac {d - 1}{2}) \Gamma (\frac {d - 2 j}{2})} \mathbb {E} _ {z \sim \operatorname {U n i f} (S ^ {d - 2 j - 1})} \left[ z _ {1} ^ {k} \right] \\ \lesssim \mathbb {E} _ {z \sim \operatorname {U n i f} (S ^ {d - 2 j - 1})} \left[ z _ {1} ^ {k} \right]. \\ \end{array}
$$
We have the following generalization of Lemma 8:
Corollary 3. For any  $k, j \geq 0$  with  $d \geq 2j + 1$  and  $\alpha \geq C^{-1/4}d^{1/2}$ , there exist  $c(j,k), C(j,k)$  such that
$$
\mathcal {L} _ {\lambda} \left(\frac {\alpha^ {k}}{(1 - \alpha) ^ {j}}\right) \leq C (j, k) s _ {k} (\alpha ; \lambda).
$$
$$
\mathcal {L} _ {\lambda} \left(\frac {\alpha^ {k}}{(1 - \alpha) ^ {j}}\right) = \mathbb {E} _ {z \sim \mathrm {U n i f} (S ^ {d - 2})} \left[ \frac {\left(\frac {\alpha + \lambda z _ {1} \sqrt {1 - \alpha^ {2}}}{\sqrt {1 + \lambda^ {2}}}\right) ^ {k}}{\left(1 - \frac {\alpha + \lambda z _ {1} \sqrt {1 - \alpha^ {2}}}{\sqrt {1 + \lambda^ {2}}}\right) ^ {j}} \right].
$$
Now let  $X = \frac{\lambda \sqrt{1 - \alpha^2}}{\sqrt{1 + \lambda^2}} \cdot \left(1 - \frac{\alpha}{\sqrt{1 + \lambda^2}}\right)^{-1}$  and note that by Cauchy-Schwarz,  $X \leq 1$ . Then,
$$
\begin{array}{l} \mathcal {L} _ {\lambda} \left(\frac {\alpha^ {k}}{(1 - \alpha) ^ {j}}\right) = \frac {1}{\left(1 - \frac {\alpha}{\sqrt {1 + \lambda^ {2}}}\right) ^ {j}} \mathbb {E} _ {z \sim \operatorname {U n i f} (S ^ {d - 2})} \left[ \frac {\left(\frac {\alpha + \lambda z \sqrt {1 - \alpha^ {2}}}{\sqrt {1 + \lambda^ {2}}}\right) ^ {k}}{(1 - X z _ {1}) ^ {j}} \right] \\ \asymp \mathbb {E} _ {z \sim \mathrm {U n i f} (S ^ {d - 2})} \left[ \frac {(1 + X z _ {1}) ^ {j} \left(\frac {\alpha + \lambda z \sqrt {1 - \alpha^ {2}}}{\sqrt {1 + \lambda^ {2}}}\right) ^ {k}}{(1 - X ^ {2} z _ {1} ^ {2}) ^ {j}} \right]. \\ \end{array}
$$
Now we can use the binomial theorem to expand this. Ignoring constants only depending on  $j, k$ :
$$
\begin{array}{l} \mathcal {L} _ {\lambda} \left(\frac {\alpha^ {k}}{(1 - \alpha) ^ {j}}\right) = \frac {1}{\left(1 - \frac {\alpha}{\sqrt {1 + \lambda^ {2}}}\right) ^ {j}} \mathbb {E} _ {z \sim \operatorname {U n i f} (S ^ {d - 2})} \left[ \frac {\left(\frac {\alpha + \lambda z _ {1} \sqrt {1 - \alpha^ {2}}}{\sqrt {1 + \lambda^ {2}}}\right) ^ {k}}{(1 - X z _ {1}) ^ {j}} \right] \\ \asymp \lambda^ {- k} \sum_ {i = 0} ^ {k} \alpha^ {k - i} \lambda^ {i} (1 - \alpha^ {2}) ^ {i / 2} \mathbb {E} _ {z \sim \operatorname {U n i f} \left(S ^ {d - 2}\right)} \left[ \frac {(1 + X z _ {1}) ^ {j} z _ {1} ^ {i}}{(1 - X ^ {2} z _ {1} ^ {2}) ^ {j}} \right] \\ \leq \lambda^ {- k} \sum_ {i = 0} ^ {k} \alpha^ {k - i} \lambda^ {i} (1 - \alpha^ {2}) ^ {i / 2} \mathbb {E} _ {z \sim \operatorname {U n i f} \left(S ^ {d - 2}\right)} \left[ \frac {(1 + X z _ {1}) ^ {j} z _ {1} ^ {i}}{(1 - z _ {1} ^ {2}) ^ {j}} \right]. \\ \end{array}
$$
By Lemma 26, the  $z_{1}$  term is bounded by  $d^{-\frac{i}{2}}$  when  $i$  is even and  $Xd^{-\frac{i + 1}{2}}$  when  $i$  is odd. Therefore this expression is bounded by
$$
\begin{array}{l} \left(\frac {\alpha}{\lambda}\right) ^ {k} \sum_ {i = 0} ^ {\lfloor \frac {k}{2} \rfloor} \left(\frac {\lambda^ {2} (1 - \alpha^ {2})}{\alpha^ {2} d}\right) ^ {i} + \left(\frac {\alpha}{\lambda}\right) ^ {k - 1} \sum_ {i = 0} ^ {\lfloor \frac {k - 1}{2} \rfloor} \alpha^ {- 2 i} \lambda^ {2 i} (1 - \alpha^ {2}) ^ {i} d ^ {- (i + 1)} \\ \succ s _ {k} (\alpha ; \lambda) + \frac {1}{d} s _ {k - 1} (\alpha ; \lambda). \\ \end{array}
$$
$$
\frac {\frac {1}{d} s _ {k - 1} (\alpha ; \lambda)}{s _ {k} (\alpha ; \lambda)} = \left\{ \begin{array}{l l} \frac {\lambda}{d \alpha} & \alpha^ {2} \geq \frac {\lambda^ {2}}{d} \\ \frac {\alpha}{\lambda} & \alpha^ {2} \leq \frac {\lambda^ {2}}{d} \text {a n d} k ^ {\star} \text {i s e v e n} \leq C ^ {- 1 / 4}. \\ \frac {\lambda}{d \alpha} & \alpha^ {2} \leq \frac {\lambda^ {2}}{d} \text {a n d} k ^ {\star} \text {i s o d d} \end{array} \right.
$$
Therefore,  $s_k(\alpha; \lambda)$  is the dominant term which completes the proof.
Lemma 27 (Adapted from Abbe et al. [4]). Let  $\eta, a_0 \geq 0$  be positive constants, and let  $u_t$  be a sequence satisfying
$$
u _ {t} \geq a _ {0} + \eta \sum_ {s = 0} ^ {t - 1} u _ {s} ^ {k}.
$$
Then, if  $\max_{0\leq s\leq t - 1}\eta u_s^{k - 1}\leq \frac{\log 2}{k}$ , we have the lower bound
$$
u _ {t} \geq \left(a _ {0} ^ {- (k - 1)} - \frac {1}{2} \eta (k - 1) t\right) ^ {- \frac {1}{k - 1}}.
$$
Proof. Consider the auxiliary sequence  $w_{t} = a_{0} + \eta \sum_{s=0}^{t-1} w_{s}^{k}$ . By induction,  $u_{t} \geq w_{t}$ . To lower bound  $w_{t}$ , we have that
$$
\begin{array}{l} \eta = \frac {w _ {t} - w _ {t - 1}}{w _ {t - 1} ^ {k}} = \frac {w _ {t} - w _ {t - 1}}{w _ {t} ^ {k}} \cdot \frac {w _ {t} ^ {k}}{w _ {t - 1} ^ {k}} \\ \leq \frac {w _ {t} ^ {k}}{w _ {t - 1} ^ {k}} \int_ {w _ {t - 1}} ^ {w _ {t}} \frac {1}{x ^ {k}} d x \\ = \frac {w _ {t} ^ {k}}{w _ {t - 1} ^ {k} (k - 1)} \left(w _ {t - 1} ^ {- (k - 1)} - w _ {t} ^ {- (k - 1)}\right) \\ \leq \frac {\left(1 + \eta w _ {t - 1} ^ {k - 1}\right) ^ {k}}{(k - 1)} \left(w _ {t - 1} ^ {- (k - 1)} - w _ {t} ^ {- (k - 1)}\right) \\ \leq \frac {\left(1 + \frac {\log 2}{k}\right) ^ {k}}{(k - 1)} \left(w _ {t - 1} ^ {- (k - 1)} - w _ {t} ^ {- (k - 1)}\right) \\ \leq \frac {2}{k - 1} \left(w _ {t - 1} ^ {- (k - 1)} - w _ {t} ^ {- (k - 1)}\right). \\ \end{array}
$$
$$
w _ {t} ^ {- (k - 1)} \leq w _ {t - 1} ^ {- (k - 1)} - \frac {1}{2} \eta (k - 1).
$$
Altogether, we get
$$
w _ {t} ^ {- (k - 1)} \leq a _ {0} ^ {- (k - 1)} - \frac {1}{2} \eta (k - 1) t,
$$
$$
u _ {t} \geq w _ {t} \geq \left(a _ {0} ^ {- (k - 1)} - \frac {1}{2} \eta (k - 1) t\right) ^ {- \frac {1}{k - 1}},
$$
To compute the smoothed loss  $L_{\lambda}(w;x;y)$  we used the closed form for  $\mathcal{L}_{\lambda}(He_k(w\cdot x))$  (see Appendix B.3). Experiments were run on 8 NVIDIA A6000 GPUs. Our code is written in JAX [45] and we used Weights and Biases [46] for experiment tracking.