SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\left| h _ {k + 1} (\beta) - h _ {k} (\beta) \right| \leqslant 8 L _ {2} \gamma_ {k} ^ {2} \mathcal {L} _ {\mathcal {B} _ {k}} (\beta_ {k}) \| \beta \| _ {\infty}.
$$
Proof. We have  $\alpha_{+,k + 1}^2 = \alpha_{+,k}^2 e^{-\delta_{+,k}}$  and  $\alpha_{-,k + 1}^2 = \alpha_{-,k}^2 e^{-\delta_{-,k}}$ , for  $\delta_{+,k} = \tilde{q} (\gamma_k\nabla \mathcal{L}_{\mathcal{B}_k}(\beta_k))$  and  $\delta_{-,k} = \tilde{q} (-\gamma_k\nabla \mathcal{L}_{\mathcal{B}_k}(\beta_k))$ . And  $\alpha_{k + 1} = \alpha_k\exp (-\delta_k)$  where  $\delta_{k}\coloneqq \delta_{+,k} + \delta_{-,k} = q(\gamma_{k}\nabla \mathcal{L}_{\mathcal{B}_{k}}(\beta_{k}))$
To prove the result we will use that for  $\beta \in \mathbb{R}^d$ , we have  $|(h_{k+1} - h_k)(\beta)| \leqslant \sum_{i=1}^{d} \int_0^{|\beta_i|} |\nabla_i h_{k+1}(x) - \nabla_i h_k(x)| \, \mathrm{d}x$ .
First, using that  $|\operatorname{arcsinh}(a) - \operatorname{arcsinh}(b)| \leqslant |\ln(a / b)|$  for  $ab > 0$ . We have that
$$
\begin{array}{l} \left| \operatorname {a r c s i n h} \left(\frac {x}{\alpha_ {k + 1} ^ {2}}\right) - \operatorname {a r c s i n h} \left(\frac {x}{\alpha_ {k} ^ {2}}\right) \right| \leqslant \ln \left(\frac {\alpha_ {k} ^ {2}}{\alpha_ {k + 1} ^ {2}}\right) \\ \mathbf {\delta} = \delta_ {k} \mathbf {\Psi}, \\ \end{array}
$$
since  $\delta_k \geqslant 0$  due to our stepsize condition.
We now prove that  $|\phi_{k + 1} - \phi_k| \leqslant \frac{|\delta_{+,k} - \delta_{-,k}|}{2}$ . We have  $\phi_k = \arcsinh\left(\frac{\alpha_{+,k}^2 - \alpha_{-,k}^2}{2\alpha_{+,k}\alpha_{-,k}}\right)$  and hence,
$$
\left| \phi_ {k + 1} - \phi_ {k} \right| = \left| \operatorname {a r c s i n h} \left(\frac {\alpha_ {+, k} ^ {2} - \alpha_ {- , k} ^ {2}}{2 \alpha_ {+, k} \alpha_ {- , k}}\right) - \operatorname {a r c s i n h} \left(\frac {\alpha_ {+, k + 1} ^ {2} - \alpha_ {- , k + 1} ^ {2}}{2 \alpha_ {+, k + 1} \alpha_ {- , k + 1}}\right) \right|.
$$
Then, assuming that  $\alpha_{+,k,i} \geqslant \alpha_{-,k,i}$ , we have:
$$
\begin{array}{l} \frac {\alpha_ {+, k + 1 , i} ^ {2} - \alpha_ {- , k + 1 , i} ^ {2}}{2 \alpha_ {+, k + 1 , i} \alpha_ {- , k + 1 , i}} = e ^ {\delta_ {k, i} / 2} \frac {\alpha_ {+, k , i} ^ {2} e ^ {- \delta_ {+ , k , i}} - \alpha_ {- , k , i} ^ {2} e ^ {- \delta_ {- , k , i}}}{2 \alpha_ {+, k , i} \alpha_ {- , k , i}} \\ \left\{ \begin{array}{l l} \leqslant \left\{ \begin{array}{l l} e ^ {\frac {\delta_ {+, k , i - \delta_ {- , k , i}}}{2}} \frac {\alpha_ {+, k , i} ^ {2} - \alpha_ {- , k , i} ^ {2}}{2 \alpha_ {+, k , i} \alpha_ {- , k , i}} & \text {i f} \quad \delta_ {+, k, i} \geqslant \delta_ {-, k, i} \\ e ^ {\frac {\delta_ {- , k , i - \delta_ {+ , k , i}}}{2}} \frac {\alpha_ {+, k , i} ^ {2} - \alpha_ {- , k , i} ^ {2}}{2 \alpha_ {+, k , i} \alpha_ {- , k , i}} & \text {i f} \quad \delta_ {-, k, i} \geqslant \delta_ {+, k, i} \end{array} \right. \\ \geqslant \left\{ \begin{array}{l l} e ^ {- \frac {\delta_ {+, k , i - \delta_ {- , k , i}}}{2}} \frac {\alpha_ {+, k , i} ^ {2} - \alpha_ {- , k , i} ^ {2}}{2 \alpha_ {+, k , i} \alpha_ {- , k , i}} & \text {i f} \quad \delta_ {+, k, i} \geqslant \delta_ {-, K, i} \\ e ^ {- \frac {\delta_ {- , k , i - \delta_ {+ , k , i}}}{2}} \frac {\alpha_ {+, k , i} ^ {2} - \alpha_ {- , k , i} ^ {2}}{2 \alpha_ {+, k , i} \alpha_ {- , k , i}} & \text {i f} \quad \delta_ {-, k, i} \geqslant \delta_ {+, k, i} \end{array} \right. \end{array} \right. \\ \end{array}
$$
We thus have  $\frac{\alpha_{+,k+1,i}^2 - \alpha_{-,k+1,i}^2}{2\alpha_{+,k+1,i}\alpha_{-,k+1,i}} \in \left[e^{-\frac{\left|\delta_{+,k,i} - \delta_{-,k,i}\right|}{2}}, e^{\frac{\left|\delta_{+,k,i} - \delta_{-,k,i}\right|}{2}}\right] \times \frac{\alpha_{+,k,i}^2 - \alpha_{-,k,i}^2}{2\alpha_{+,k,i}\alpha_{-,k,i}}$ , and this holds similarly if  $\alpha_{+,k,i} \leqslant \alpha_{-,k,i}$ . Then, using  $|\operatorname{arcsinh}(a) - \operatorname{arcsinh}(b)| \leqslant |\ln(a / b)|$  we obtain that:
$$
\begin{array}{l} \left| \phi_ {k + 1} - \phi_ {k} \right| = \left| \operatorname {a r c s i n h} \left(\frac {\alpha_ {+, k} ^ {2} - \alpha_ {- , k} ^ {2}}{2 \alpha_ {+, k} \alpha_ {- , k}}\right) - \operatorname {a r c s i n h} \left(\frac {\alpha_ {+, k + 1} ^ {2} - \alpha_ {- , k + 1} ^ {2}}{2 \alpha_ {+, k + 1} \alpha_ {- , k + 1}}\right) \right| \\ \leqslant \frac {\left| \delta_ {+, k} - \delta_ {- , k} \right|}{2}. \\ \end{array}
$$
Wrapping things up, we have:
$$
\left| \nabla h _ {k} (\beta) - \nabla h _ {k + 1} (\beta) \right| \leqslant \delta_ {k} + \frac {\left| \delta_ {+, k} - \delta_ {- , k} \right|}{2} \leqslant 2 \delta_ {k},
$$
This leads to the following bound:
$$
\begin{array}{l} \left| h _ {k + 1} (\beta) - h _ {k} (\beta) \right| \leqslant \langle | 2 \delta_ {k} |, | \beta | \rangle \\ \leqslant 2 \| \delta_ {k} \| _ {1} \| \beta \| _ {\infty}. \\ \end{array}
$$
Recall that  $\delta_{k} = q(\gamma_{k}\nabla \mathcal{L}_{\mathcal{B}_{k}}(\beta_{k})$ , hence from Lemma 5 if  $\gamma_k\| \nabla \mathcal{L}_{\mathcal{B}_k}(\beta_k)\|_\infty \leqslant 1 / 2$ , we get that
$$
\left\| \delta_ {k} \right\| _ {1} \leqslant 2 \gamma_ {k} ^ {2} \left\| \nabla \mathcal {L} _ {\mathcal {B} _ {k}} (\beta_ {k}) \right\| _ {2} ^ {2} \leqslant 4 L _ {2} \gamma_ {k} ^ {2} \mathcal {L} _ {\mathcal {B} _ {k}} (\beta_ {k}).
$$
Putting things together we obtain that
$$
\begin{array}{l} \left| h _ {k + 1} (\beta) - h _ {k} (\beta) \right| \leqslant \langle | 2 \delta_ {k} |, | \beta | \rangle \\ \leqslant 8 L _ {2} \gamma_ {k} ^ {2} \mathcal {L} _ {\mathcal {B} _ {k}} (\beta_ {k}) \| \beta \| _ {\infty}. \\ \end{array}
$$
I Concentration inequalities for matrices
In this last section of the appendix, we provide and prove several concentration bounds for random vectors and matrices, with (possibly uncentered) isotropic gaussian inputs. These inequalities can easily be generalized to subgaussian random variables via more refined concentration bounds, and to non-isotropic subgaussian random variables [19], leading to a dependence on an effective dimension and on the subgaussian matrix  $\Sigma$ . We present these lemmas before proving them in a row.
The next two lemmas closely resemble the RIP assumption, for centered and then for uncentered.   
gaussians.
Lemma 8. Let  $x_{1},\ldots ,x_{n}\in \mathbb{R}^{d}$  be i.i.d. random variables of law  $\mathcal{N}(0,I_d)$  and  $H = \frac{1}{n}\sum_{i = 1}^{n}x_{i}x_{i}^{\top}$ . Then, denoting by  $\mathcal{C}$  the set of all  $s$ -sparse vector  $\beta \in \mathbb{R}^d$  satisfying  $\| \beta \| _2\leqslant 1$ , there exist  $C_4,C_5 > 0$  such that for any  $\varepsilon >0$ , if  $n\geqslant C_4s\ln (d)\varepsilon^{-2}$ ,
$$
\mathbb{P}\left(\sup_{\beta \in \mathcal{S}}\| H\beta -\beta \|_{\infty}\geqslant \varepsilon\right)\leqslant e^{-C_{5}n}.
$$
Lemma 9. Let  $x_{1},\ldots ,x_{n}\in \mathbb{R}^{d}$  be i.i.d. random variables of law  $\mathcal{N}(\mu ,\sigma^2 I_d)$  and  $H = \frac{1}{n}\sum_{i = 1}^{n}x_{i}x_{i}^{\top}$ . Then, denoting by  $\mathcal{C}$  the set of all s-sparse vector  $\beta \in \mathbb{R}^d$  satisfying  $\| \beta \| _2\leqslant 1$ , there exist  $C_4,C_5 > 0$  such that for any  $\varepsilon >0$ , if  $n\geqslant C_4s\ln (d)\varepsilon^{-2}$
$$
\mathbb {P} \left(\sup  _ {\beta \in \mathcal {S}} \left\| H \beta - \mu \langle \mu , \beta \rangle - \sigma^ {2} \beta \right\| _ {\infty} \geqslant \varepsilon\right) \leqslant e ^ {- C _ {5} n}.
$$
We then provide two lemmas that estimate the mean Hessian of SGD.
Lemma 10. Let  $x_{1},\ldots ,x_{n}$  be i.i.d. random variables of law  $\mathcal{N}(0,I_d)$ . Then, there exist  $c_{1},c_{2} > 0$  such that with probability  $1 - \frac{1}{d^2}$  and if  $n = \Omega (s^{5 / 4}\ln (d))$ , we have for all s-sparse vectors  $\beta$ :
$$
c _ {1} \| \beta \| _ {2} ^ {2} \mathbf {1} \leqslant \frac {1}{n} \sum_ {i = 1} ^ {n} x _ {i} ^ {2} \left\langle x _ {i}, \beta \right\rangle^ {2} \leqslant c _ {2} \| \beta \| _ {2} ^ {2} \mathbf {1},
$$
where the inequality is meant component-wise.
Lemma 11. Let  $x_{1},\ldots ,x_{n}$  be i.i.d. random variables of law  $\mathcal{N}(\mu ,\sigma^2 I_d)$ . Then, there exist  $c_{0},c_{1},c_{2} > 0$  such that with probability  $1 - \frac{c_0}{d^2} -\frac{1}{nd}$  and if  $n = \Omega (s^{5 / 4}\ln (d))$  and  $\mu \geqslant 4\sigma \sqrt{\ln(d)}\mathbf{1}$ , we have for all  $s$ -sparse vectors  $\beta$ :
$$
\frac {\mu^ {2}}{2} \left(\langle \mu , \beta \rangle^ {2} + \frac {1}{2} \sigma^ {2} \| \beta \| _ {2} ^ {2}\right) \leqslant \frac {1}{n} \sum_ {i} x _ {i} ^ {2} \langle x _ {i}, \beta \rangle^ {2} \leqslant 4 \mu^ {2} \left(\langle \mu , \beta \rangle^ {2} + 2 \sigma^ {2} \| \beta \| _ {2} ^ {2}\right).
$$
Finally, next two lemmas are used to estimate  $\lambda_{b},\Lambda_{b}$  in our paper.
Lemma 12. Let  $x_{1},\ldots ,x_{n}\in \mathbb{R}^{d}$  be i.i.d. random variables of law  $\mathcal{N}(\mu \mathbf{1},\sigma^2 I_d)$ . Let  $H = \frac{1}{n}\sum_{i = 1}^{n}x_{i}x_{i}^{\top}$  and  $\tilde{H} = \frac{1}{n}\sum_{i = 1}^{n}\| x_i\|^2 x_ix_i^\top$ . There exist numerical constants  $C_2,C_3 > 0$  such that
$$
\mathbb {P} \left(C _ {2} \left(\mu^ {2} + \sigma^ {2}\right) d H \preceq \tilde {H} \preceq C _ {3} \left(\mu^ {2} + \sigma^ {2}\right) d H\right) \geqslant 1 - 2 n e ^ {- d / 1 6}.
$$
Lemma 13. Let  $x_{1},\ldots ,x_{n}\in \mathbb{R}^{d}$  be i.i.d. random variables of law  $\mathcal{N}(\mu \mathbf{1},\sigma^2 I_d)$  for some  $\mu \in \mathbb{R}$ . Let  $H = \frac{1}{n}\sum_{i = 1}^{n}x_{i}x_{i}^{\top}$  and for  $1\leqslant b\leqslant n$  let  $\tilde{H}_b = \mathbb{E}_{\mathcal{B}}\left[\left(\frac{1}{b}\sum_{i\in \mathcal{B}}x_{i}x_{i}^{\top}\right)^{2}\right]$  where  $\mathcal{B}\subset [n]$  is sampled uniformly at random in  $\{\mathcal{B}\subset [n]$  s.t.  $|\mathcal{B}| = b\}$ . With probability  $1 - 2ne^{-d / 16} - 3 / n^2$ , we have, for some numerical constants  $c_{1},c_{2},c_{3},C > 0$ :
$$
\left(c _ {1} \frac {d (\mu^ {2} + \sigma^ {2})}{b} - c _ {2} \frac {(\sigma^ {2} + \mu^ {2}) \ln (n)}{\sqrt {d}} - c _ {3} \frac {\mu^ {2} d}{n}\right) H \preceq \tilde {H} _ {b} \preceq C \left(\frac {d (\mu^ {2} + \sigma^ {2})}{b} + \frac {(\sigma^ {2} + \mu^ {2}) \ln (n)}{\sqrt {d}} + \mu^ {2} d\right)
$$
Proof of Lemma 8. For  $j \in [d]$ , we have:
$$
\begin{array}{l} (H \beta) _ {j} = \frac {1}{n} \sum_ {i = 1} ^ {n} x _ {i j} \langle x _ {i}, \beta \rangle \\ = \frac {1}{n} \sum_ {i = 1} ^ {n} \sum_ {j ^ {\prime} = 1} ^ {d} x _ {i j} x _ {i j ^ {\prime}} \beta_ {j ^ {\prime}} \\ = \frac {1}{n} \sum_ {i = 1} ^ {n} x _ {i j} ^ {2} \beta_ {j} + \frac {1}{n} \sum_ {i = 1} ^ {n} \sum_ {j ^ {\prime} \neq j} x _ {i j} x _ {i j ^ {\prime}} \beta_ {j ^ {\prime}} \\ = \frac {\beta_ {j}}{n} \sum_ {i = 1} ^ {n} x _ {i j} ^ {2} + \frac {1}{n} \sum_ {i = 1} ^ {n} x _ {i j} \sum_ {j ^ {\prime} \neq j} x _ {i j ^ {\prime}} \beta_ {j ^ {\prime}}. \\ \end{array}
$$
We thus notice that  $\mathbb{E}[H\beta] = \beta$ , and
$$
(H \beta) _ {j} = \beta_ {j} + \frac {\beta_ {j}}{n} \sum_ {i = 1} ^ {n} \left(x _ {i j} ^ {2} - 1\right) + \frac {1}{n} \sum_ {i = 1} ^ {n} z _ {i},
$$
where  $z_{i} = x_{ij}\sum_{j^{\prime}\neq j}x_{ij^{\prime}}\beta_{j^{\prime}}$ , and  $\sum_{j^{\prime}\neq j}x_{ij^{\prime}}\beta_{j^{\prime}}\sim \mathcal{N}(0,\| \beta \| ^2 -\beta_j^2)$  and  $\| \beta \| ^2 -\beta_j^2\leqslant 1$ . Hence,  $z_{j} + x_{ij}^{2} - 1$  is a centered subexponential random variables (with a subexponential parameter of order 1). Thus, for  $t\leqslant 1$ :
$$
\mathbb {P} \left(\left| \frac {\beta_ {j}}{n} \sum_ {i = 1} ^ {n} (x _ {i j} ^ {2} - 1) + \frac {1}{n} \sum_ {i = 1} ^ {n} z _ {i} \right| \geqslant t\right) \leqslant 2 e ^ {- c n t ^ {2}}.
$$
Hence, using an  $\varepsilon$ -net of  $\mathcal{C} = \{\beta \in \mathbb{R}^d : \| \beta \|_2 \leqslant 1, \| \beta \|_0\}$  (of cardinality less than  $d^s \times (C / \varepsilon)^s$ , and for  $\varepsilon$  of order 1), we have, using the classical  $\varepsilon$ -net trick explained in [Chapt. 9, [58] or [App. C, Even and Massoulie [19]]:
$$
\mathbb {P} \left(\sup  _ {\beta \in \mathcal {C}, j \in [ d ]} | (H \beta) _ {j} - \beta_ {j} | \geqslant t\right) \leqslant d \times d ^ {s} (C / \varepsilon) ^ {s} \times 2 e ^ {- c n t ^ {2}} = \exp \left(- c \ln (2) n t ^ {2} + (s + 1) \ln (d) + s \ln (C / \varepsilon)\right).
$$
Consequently, for  $t = \varepsilon$  and if  $n \geqslant C_4 s \ln(d) / \varepsilon^2$ , we have:
$$
\mathbb {P} \left(\sup  _ {\beta \in \mathcal {C}, j \in [ d ]} | (H \beta) _ {j} - \beta_ {j} | \geqslant t\right) \leqslant \exp \left(- C _ {5} n t ^ {2}\right).
$$
Proof of Lemma 9. We write  $x_{i} = \sigma z_{i} + \mu$  where  $z_{i}\sim \mathcal{N}(0,I_{d})$ . We have:
$$
\begin{array}{l} X ^ {\top} X \beta = \frac {1}{n} \sum_ {i = 1} ^ {n} (\mu + \sigma z _ {i}) \langle \mu + \sigma z _ {i}, \beta \rangle \\ = \mu \langle \mu , \beta \rangle + \frac {\sigma^ {2}}{n} \sum_ {i = 1} ^ {n} z _ {i} \langle z _ {i}, \beta \rangle + \frac {\sigma}{n} \sum_ {i = 1} ^ {n} \mu \langle z _ {i}, \beta \rangle + \frac {\sigma}{n} \sum_ {i = 1} ^ {n} z _ {i} \langle \mu , \beta \rangle \\ = \mu \langle \mu , \beta \rangle + \frac {\sigma^ {2}}{n} \sum_ {i = 1} ^ {n} z _ {i} \langle z _ {i}, \beta \rangle + \sigma \mu \left\langle \frac {1}{n} \sum_ {i = 1} ^ {n} z _ {i}, \beta \right\rangle + \frac {\sigma \langle \mu , \beta \rangle}{n} \sum_ {i = 1} ^ {n} z _ {i}. \\ \end{array}
$$
The first term is deterministic and is to be kept. The second one is of order  $\sigma^2\beta$  whp using Lemma 8. Then,  $\frac{1}{n}\sum_{i = 1}^{n}z_{i}\sim \mathcal{N}(0,I_{d} / n)$ , so that
$$
\mathbb {P} \left(\left| \left\langle \frac {1}{n} \sum_ {i = 1} ^ {n} z _ {i}, \beta \right\rangle \right| \geqslant t\right) \leqslant 2 e ^ {- n t ^ {2} / \left(2 \| \beta \| _ {2} ^ {2}\right)},
$$
$$
\mathbb {P} \left(\left| \frac {1}{n} \sum_ {i = 1} ^ {n} z _ {i j} \right| \geqslant t\right) \leqslant 2 e ^ {- n t ^ {2} / 2}.
$$
$$
\mathbb {P} \left(\sup  _ {\beta \in \mathcal {C}} \left\| \frac {1}{n} \sum_ {i = 1} ^ {n} z _ {i j} \right\| _ {\infty} \geqslant t, \sup  _ {\beta \in \mathcal {C}} \left| \langle \frac {1}{n} \sum_ {i = 1} ^ {n} z _ {i}, \beta \rangle \right| \geqslant t\right) \leqslant 4 e ^ {c s \ln (d)} e ^ {- n t ^ {2} / 2}.
$$
Thus, with probability  $1 - Ce^{-n\varepsilon^2}$  and under the assumptions of Lemma 8, we have  $\left\| X^\top X\beta - \mu \langle \mu, \beta \rangle - \sigma^2\beta \right\|_\infty \leqslant \varepsilon$
Proof of Lemma 10. To ease notations, we assume that  $\sigma = 1$ . We remind (O'Donnell [46], Chapter 9 and Tao [54]) that for i.i.d. real random variables  $a_1, \ldots, a_n$  that satisfy a tail inequality of the form
$$
\mathbb {P} \left(\left| a _ {1} - \mathbb {E} a _ {1} \right| \geqslant t\right) \leqslant C e ^ {- c t ^ {p}}, \tag {27}
$$
for  $p < 1$ , then for all  $\varepsilon >0$  there exists  $C^\prime ,c^\prime$  such that for all  $t$
$$
\mathbb {P} \left(\left| \frac {1}{n} \sum_ {i = 1} ^ {n} a _ {i} - \mathbb {E} a _ {1} \right| \geqslant t\right) \leqslant C ^ {\prime} e ^ {- c ^ {\prime} n t ^ {p - \varepsilon}}.
$$
We now expand  $\frac{1}{n}\sum_{i = 1}^{n}x_i^2\langle x_i,\beta \rangle^2$
$$
\begin{array}{l} \frac {1}{n} \sum_ {i = 1} ^ {n} x _ {i} ^ {2} \langle x _ {i}, \beta \rangle^ {2} = \frac {1}{n} \sum_ {i \in [ n ], k, \ell \in [ d ]} x _ {i} ^ {2} x _ {i k} x _ {i \ell} \beta_ {k} \beta_ {\ell} \\ = \frac {1}{n} \sum_ {i \in [ n ], k \in [ d ]} x _ {i} ^ {2} x _ {i k} ^ {2} \beta_ {k} ^ {2} + \frac {1}{n} \sum_ {i \in [ n ], k \neq \ell \in [ d ]} x _ {i} ^ {2} x _ {i k} x _ {i \ell} \beta_ {k} \beta_ {\ell}. \\ \end{array}
$$
Thus, for  $j\in [d]$
$$
\left(\frac {1}{n} \sum_ {i = 1} ^ {n} x _ {i} ^ {2} \langle x _ {i}, \beta \rangle^ {2}\right) _ {j} = \sum_ {k \in [ d ]} \frac {\beta_ {k} ^ {2}}{n} \sum_ {i \in [ n ]} x _ {i j} ^ {2} x _ {i k} ^ {2} + \sum_ {k \neq \ell \in [ d ]} \frac {\beta_ {k} \beta_ {\ell}}{n} \sum_ {i \in [ n ]} x _ {i j} ^ {2} x _ {i k} x _ {i \ell}.
$$
We notice that for all indices, all  $x_{ij}^{2}x_{ik}x_{il}$  and  $x_{ij}^{2}x_{ik}^{2}$  satisfy the tail inequality Eq. (27) for  $C = 8$ ,  $c = 1/2$  and  $p = 1/2$ , so that for  $\varepsilon = 1/4$ :
$$
\mathbb {P} \big (\left| \frac {1}{n} \sum_ {i = 1} ^ {n} x _ {i j} ^ {2} x _ {i k} x _ {i \ell} \right| \geqslant t \big) \leqslant C ^ {\prime} e ^ {- c ^ {\prime} n t ^ {1 / 4}} \quad , \quad \mathbb {P} \big (\left| \frac {1}{n} \sum_ {i = 1} ^ {n} x _ {i j} ^ {2} x _ {i k} ^ {2} - \mathbb {E} \left[ x _ {i j} ^ {2} x _ {i k} ^ {2} \right] \right| \geqslant t \big) \leqslant C ^ {\prime} e ^ {- c ^ {\prime} n t ^ {1 / 4}}.
$$
For  $j \neq k$ , we have  $\mathbb{E}\left[x_{ij}^{2}x_{ik}^{2}\right] = 1$  while for  $j = k$ , we have  $\mathbb{E}\left[x_{ij}^{2}x_{ik}^{2}\right] = \mathbb{E}\left[x_{ij}^{4}\right] = 3$ . Hence,
$$
\mathbb {P} \left(\exists j, k \neq \ell , | \frac {1}{n} \sum_ {i = 1} ^ {n} x _ {i j} ^ {2} x _ {i k} x _ {i \ell} | \geqslant t, | \frac {1}{n} \sum_ {i = 1} ^ {n} x _ {i j} ^ {2} x _ {i k} ^ {2} - \mathbb {E} \left[ x _ {i j} ^ {2} x _ {i k} ^ {2} \right] | \geqslant t\right) \leqslant C ^ {\prime} d ^ {2} e ^ {- c ^ {\prime} n t ^ {1 / 4}}.
$$
Thus, with probability  $1 - C^{\prime}d^{2}e^{-c^{\prime}nt^{1 / 4}}$ , for all  $j\in [d]$
$$
\left| \left(\frac {1}{n} \sum_ {i = 1} ^ {n} x _ {i} ^ {2} \langle x _ {i}, \beta \rangle^ {2}\right) _ {j} - 2 \beta_ {j} ^ {2} - \| \beta \| _ {2} ^ {2} \right| \leqslant t \sum_ {k, \ell} | \beta_ {k} | | \beta_ {\ell} | = t \| \beta \| _ {1} ^ {2}.
$$
Using the classical technique of Baraniuk et al. [4], to make a union bound on all  $s$ -sparse vectors, we consider an  $\varepsilon$ -net of the set of  $s$ -sparse vectors of  $\ell^2$ -norm smaller than 1. This  $\varepsilon$ -net is of cardinality less than  $(C_0 / \varepsilon)^s d^s$ , and we only need to take  $\varepsilon$  of order 1 to obtain the result for all  $s$ -sparse vectors. This leads to:
$$
\begin{array}{l} \mathbb {P} \left(\exists \beta \in \mathbb {R} ^ {d} s \text {- s p a r s e a n d} \| \beta \| _ {2} \leqslant 1, \exists j \in \mathbb {R} ^ {d}, \quad \left| \left(\frac {1}{n} \sum_ {i = 1} ^ {n} x _ {i} ^ {2} \langle x _ {i}, \beta \rangle^ {2}\right) _ {j} - 2 \beta_ {j} ^ {2} - \| \beta \| _ {2} ^ {2} \right| \geqslant t \| \beta \| _ {1} ^ {2}\right) \\ \leqslant C ^ {\prime} d ^ {2} e ^ {c _ {1} s + s \ln (d)} e ^ {- c ^ {\prime} n t ^ {1 / 4}}. \\ \end{array}
$$
This probability is equal to  $C' / d^2$  for  $t = \left(\frac{(s + 4)\ln(d) + c_1s}{c'n}\right)^4$ . We conclude that with probability  $1 - C' / d^2$ , all  $s$ -sparse vectors  $\beta$  satisfy:
$$
\left| \left(\frac {1}{n} \sum_ {i = 1} ^ {n} x _ {i} ^ {2} \langle x _ {i}, \beta \rangle^ {2}\right) _ {j} - 2 \beta_ {j} ^ {2} - \| \beta \| _ {2} ^ {2} \right| \leqslant \left(\frac {(s + 4) \ln (d) + c _ {1} s}{c ^ {\prime} n}\right) ^ {4} \| \beta \| _ {1} ^ {2} \leqslant \left(\frac {(s + 4) \ln (d) + c _ {1} s}{c ^ {\prime} n}\right) ^ {4} s \| \beta \| _ {2} ^ {2},
$$
and the RHS is smaller than  $\| \beta \| _2^2 /2$  for  $n\geqslant \Omega (s^{5 / 4}\ln (d))$
Proof of Lemma 11. We write  $x_{i} = \mu + \sigma z_{i}$  where  $x_{i} \sim \mathcal{N}(0,1)$ . We have:
$$
\mathbb {P} \big (\forall i \in [ n ], \forall j \in [ d ], | z _ {i j} | \geqslant t \big) \leqslant e ^ {\ln (n d) - t ^ {2} / 2} = \frac {1}{n d},
$$
for  $t = 2\sqrt{\ln(nd)}$ . Thus, if  $\mu \geqslant 4\sigma \sqrt{\ln(nd)}$  we have  $\frac{\mu}{2} \leqslant x_i \leqslant 2\mu$ , so that
$$
\frac {\mu^ {2}}{2 n} \sum_ {i} \langle x _ {i}, \beta \rangle^ {2} \leqslant \frac {1}{n} \sum_ {i} x _ {i} ^ {2} \langle x _ {i}, \beta \rangle^ {2} \leqslant \frac {4 \mu^ {2}}{n} \sum_ {i} \langle x _ {i}, \beta \rangle^ {2}.
$$
Then,  $\langle x_i,\beta \rangle \sim \mathcal{N}(\langle \mu ,\beta \rangle ,\sigma^2\| \beta \| _2^2)$ . For now, we assume that  $\| \beta \| _2 = 1$ . We have  $\mathbb{P}(|\langle x_i,\beta \rangle^2 -$ $\langle \mu ,\beta \rangle^2 -\sigma^2\| \beta \| _2^2 |\geqslant t)\leqslant Ce^{-ct / \sigma^2}$ , and for  $t\leqslant 1$ , using concentration of subexponential random variables [58]:
$$
\mathbb {P} \left(\left| \frac {1}{n} \sum_ {i} \langle x _ {i}, \beta \rangle^ {2} - \langle \mu , \beta \rangle^ {2} - \sigma^ {2} \| \beta \| _ {2} ^ {2} \right| \geqslant t\right) \leqslant C ^ {\prime} e ^ {- n c ^ {\prime} t ^ {2} / \sigma^ {4}},
$$
and using the  $\varepsilon$ -net trick of Baraniuk et al. [4],
$$
\mathbb {P} \left(\sup  _ {\beta \in \mathcal {C}} \left| \frac {1}{n} \sum_ {i} \langle x _ {i}, \beta \rangle^ {2} - \langle \mu , \beta \rangle^ {2} - \sigma^ {2} \| \beta \| _ {2} ^ {2} \right| \geqslant t\right) \leqslant C ^ {\prime} e ^ {s \ln (d) - n c ^ {\prime} t ^ {2} / \sigma^ {4}} = \frac {C ^ {\prime}}{d ^ {2}},
$$
for  $t = \sigma^2\| \beta \| _2^2\sqrt{\frac{2(cs + 2)\ln(d)}{n}}$ . Consequently, we have, with probability  $1 - \frac{C'}{d^2} -\frac{1}{nd}$ :
Proof of Lemma 12. First, we write  $x_{i} = \mu \mathbf{1} + \sigma z_{i}$ , where  $z_{i}\sim \mathcal{N}(0,I)$ , leading to:
$$
\frac {1}{n} \sum_ {i \in [ n ]} \| x _ {i} \| _ {2} ^ {2} x _ {i} x _ {i} ^ {\top} = \frac {1}{n} \sum_ {i \in [ n ]} \left(\sigma^ {2} \| z _ {i} \| _ {2} ^ {2} + d \mu^ {2} + 2 \sigma \mu \langle \mathbf {1}, z _ {i} \rangle\right) x _ {i} x _ {i} ^ {\top}
$$
We use concentration of  $\chi_d^2$  random variables around  $d$ :
$$
\mathbb {P} \left(\chi_ {d} ^ {2} > d + 2 t + 2 \sqrt {d t}\right) \geqslant t) \leqslant e ^ {- t} \quad \text {a n d} \quad \mathbb {P} \left(\chi_ {d} ^ {2} > d - 2 \sqrt {d t}\right) \leqslant t) \leqslant e ^ {- t},
$$
so that for all  $i \in [n]$ ,
$$
\mathbb {P} \left(\| z _ {i} \| _ {2} ^ {2} \notin [ d - 2 \sqrt {d t}, d + 2 t + 2 \sqrt {d t} ]\right) \leqslant 2 e ^ {- t}.
$$
$$
\mathbb {P} (\forall i \in [ n ], \| z _ {i} \| _ {2} ^ {2} \in [ d - 2 \sqrt {d t}, d + 2 t + 2 \sqrt {d t} ]) \geqslant 1 - 2 n e ^ {- t}.
$$
Taking  $t = d / 16$
$$
\mathbb {P} (\forall i \in [ n ], \| z _ {i} \| _ {2} ^ {2} \in [ \frac {d}{2}, 1 3 d / 8 ]) \geqslant 1 - 2 n e ^ {- d / 1 6}.
$$
Then, for all  $i$ ,  $\langle \mathbf{1}, z_i \rangle$  is of law  $\mathcal{N}(0, d)$ , so that  $\mathbb{P}(|\langle \mathbf{1}, z_i \rangle| \geqslant t) \leqslant 2e^{-t^2/(2d)}$  and
$$
\mathbb {P} \left(\forall i \in [ n ], | \langle \mathbf {1}, z _ {i} \rangle | \geqslant t\right) \leqslant 2 n e ^ {- \frac {t ^ {2}}{2 d}}.
$$
Taking  $t = \sqrt{2} d^{3 / 4}$
$$
\mathbb {P} \left(\forall i \in [ n ], | \langle \mathbf {1}, z _ {i} \rangle | \geqslant d ^ {3 / 4}\right) \leqslant 2 n e ^ {- d ^ {1 / 2}}.
$$
Thus, with probability  $1 - 2n(e^{-d / 16} + e^{-\sqrt{d}}$ , we have  $\forall i \in [n]$ ,  $|\langle \mathbf{1}, z_i \rangle| \geqslant d^{3/4}$  and  $\| z_i \|_2^2 \in [\frac{d}{2}, 13d / 8]$ , so that
$$
\left(\frac {d}{2} \sigma^ {2} + d \mu^ {2} - 2 \mu \sigma d ^ {3 / 4}\right) H \preceq \tilde {H} \preceq \left(\frac {1 3 d}{8} \sigma^ {2} + d \mu^ {2} + 2 \mu \sigma d ^ {3 / 4}\right) H,
$$
leading to the desired result.
Proof of Lemma 13. We have:
$$
\begin{array}{l} \tilde {H} _ {b} = \mathbb {E} \left[ \frac {1}{b ^ {2}} \sum_ {i, j \in \mathcal {B}} \langle x _ {i}, x _ {j} \rangle x _ {i} x _ {j} ^ {\top} \right] \\ = \mathbb {E} \left[ \frac {1}{b ^ {2}} \sum_ {i \in \mathcal {B}} \| x _ {i} \| _ {2} ^ {2} x _ {i} x _ {i} ^ {\top} + \frac {1}{b ^ {2}} \sum_ {i, j \in \mathcal {B}, i \neq j} \langle x _ {i}, x _ {j} \rangle x _ {i} x _ {j} ^ {\top} \right] \\ = \frac {1}{b ^ {2}} \sum_ {i \in [ n ]} \mathbb {P} (i \in \mathcal {B}) \| x _ {i} \| _ {2} ^ {2} x _ {i} x _ {i} ^ {\top} + \frac {1}{b ^ {2}} \sum_ {i \neq j} \mathbb {P} (i, j \in \mathcal {B}) \langle x _ {i}, x _ {j} \rangle x _ {i} x _ {j} ^ {\top}. \\ \end{array}
$$
Then, since  $\mathbb{P}(i\in \mathcal{B}) = \frac{b}{n}$  and  $\mathbb{P}(i,j\in \mathcal{B}) = \frac{b(b - 1)}{n(n - 1)}$  for  $i\neq j$ , we get that:
$$
\tilde {H} _ {b} = \frac {1}{b n} \sum_ {i \in [ n ]} \| x _ {i} \| _ {2} ^ {2} x _ {i} x _ {i} ^ {\top} + \frac {(b - 1)}{b n (n - 1)} \sum_ {i \neq j} \langle x _ {i}, x _ {j} \rangle x _ {i} x _ {j} ^ {\top}.
$$
Using Lemma 12, the first term satisfies: