SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Hence, since  $\left\| \Sigma_f - \mathbb{E}f(\boldsymbol{\eta}^*)f(\boldsymbol{\eta}^*)^\top \right\|$  is at least a sufficiently large multiple of  $\varepsilon \log (1 / \varepsilon) / \alpha^3$  and  $\frac{\varepsilon}{\alpha^2}$  is a sufficiently small constant, it follows that the first two terms are at most  $0.0001\left\| \Sigma_f - \mathbb{E}f(\boldsymbol{\eta}^*)f(\boldsymbol{\eta}^*)^\top \right\|$ . The second term is at most
$$
O \left(\sqrt {\frac {\varepsilon}{\alpha^ {3}} \left\| \Sigma_ {f} - \mathbb {E} f (\boldsymbol {\eta} ^ {*}) f (\boldsymbol {\eta} ^ {*}) ^ {\top} \right\|}\right).
$$
We claim that this is at most  $0.0001\left\| \Sigma_f - \mathbb{E}f(\boldsymbol{\eta}^*)f(\boldsymbol{\eta}^*)^\top \right\|$  as well. Indeed, if it were larger, we would have that  $\left\| \Sigma_f - \mathbb{E}f(\boldsymbol{\eta}^*)f(\boldsymbol{\eta}^*)^\top \right\| = O\left(\frac{\varepsilon}{\alpha^3}\right)$ , a contradiction. Thus, we get that
$$
\sum_ {i \in S _ {b}} w _ {i} \tau_ {i} \geqslant 0. 9 9 \left\| \Sigma_ {f} - \mathbb {E} f (\boldsymbol {\eta} ^ {*}) f (\boldsymbol {\eta} ^ {*}) ^ {\top} \right\|.
$$
It remains to upper bound the contribution of indices  $i$  in  $S_{b} \setminus T$ . Note that
$$
\sum_ {i \in S _ {b} \cap T} w _ {i} \tau_ {i} = \sum_ {i \in S _ {b}} w _ {i} \tau_ {i} - \sum_ {i \in S _ {b} \backslash T} w _ {i} \tau_ {i}.
$$
We claim that the scores for  $i \notin T$  cannot be too large. Indeed, since  $\sum_{i \in T} w_i > 2\varepsilon$  and  $\sum_{i \in S_b \cap T} w_i \leqslant \varepsilon$  it follows that  $\sum_{i \in S_g \cap T} w_i \geqslant \varepsilon$ . Thus, since we assume the  $\tau_i$  are sorted in decreasing order it follows that for  $i \notin T$  it holds that
$$
\tau_ {i} \leqslant \frac {1}{\sum_ {i \in S _ {g} \cap T} w _ {i}} \sum_ {i \in S _ {g} \cap T} w _ {i} \tau_ {i} \leqslant O \left(\log (1 / \varepsilon) + \frac {\varepsilon}{\alpha^ {3}} \cdot \left\| \Sigma_ {f} - \mathbb {E} f (\pmb {\eta} ^ {*}) f (\pmb {\eta} ^ {*}) ^ {\top} \right\|\right).
$$
$$
\sum_ {i \in S _ {b} \backslash T} w _ {i} \tau_ {i} \leqslant O \left(\varepsilon \log (1 / \varepsilon) + \frac {\varepsilon^ {2}}{\alpha^ {3}} \cdot \left\| \Sigma_ {f} - \mathbb {E} f (\boldsymbol {\eta} ^ {*}) f (\boldsymbol {\eta} ^ {*}) ^ {\top} \right\|\right)
$$
Putting everything together we have shown that
$$
\sum_ {i \in S _ {b} \cap T} w _ {i} \tau_ {i} \geqslant 0. 9 \left\| \Sigma_ {f} - \mathbb {E} f (\boldsymbol {\eta} ^ {*}) f (\boldsymbol {\eta} ^ {*}) ^ {\top} \right\|
$$
Finally, it is easy to see that if we can only compute an estimation  $\hat{v}$  of  $v$  such that
$$
\hat {v} ^ {\top} \left(\Sigma_ {f} - \mathbb {E} f (\boldsymbol {\eta} ^ {*}) f (\boldsymbol {\eta} ^ {*}) ^ {\top}\right) \hat {v} = \left\| \Sigma_ {f} - \mathbb {E} f (\boldsymbol {\eta} ^ {*}) f (\boldsymbol {\eta} ^ {*}) ^ {\top} \right\| \pm O (\varepsilon),
$$
the proof still works, so we can use the  $O(\varepsilon)$ -close estimation  $\hat{\Sigma}$  of  $\mathbb{E}f(\boldsymbol{\eta}^{*})f(\boldsymbol{\eta}^{*})^{\top}$  and compute the top eigenvector of  $\Sigma_f - \hat{\Sigma}$ .
J Concentration Bounds
Throughout this section we will use the following versions of the Matrix Bernstein Inequality. A proof of a slightly more general version can be found in [Tro15, Corollary 6.2.1].
Fact J.1. Let  $L, B > 0$ ,  $M \in \mathbb{R}^{d \times d}$  be a symmetric random matrix and  $M_1, \ldots, M_n$  i.i.d. copies of  $M$ . Suppose that  $\| M \| \leqslant L$ . Further, assume that  $\left\| \mathbb{E}M^2 \right\| \leqslant B$ . Then the estimator  $M = \frac{1}{n}\sum_{i=1}^{n} M_i$  satisfies for all  $t > 0$
$$
\mathbb {P} \left(\left\| \bar {\boldsymbol {M}} - \mathbb {E} \boldsymbol {M} \right\| \geqslant t\right) \leqslant 2 d \cdot \exp \left(- \frac {t ^ {2} n}{2 B + 2 L t}\right).
$$
Fact J.2. Let  $x_1, \ldots, x_n$  be iid  $d$ -dimensional random vectors such that  $\| x_1 \| \leqslant L$  with probability 1. Then
$$
\mathbb {P} \left(\| \bar {\boldsymbol {v}} - \mathbb {E} \boldsymbol {v} \| \geqslant t\right) \leqslant 2 d \cdot \exp \left(- \frac {t ^ {2} n}{2 L ^ {2} + 2 L t}\right).
$$
Fact J.3. Let  $L > 0$ ,  $\pmb{v} \in \mathbb{R}^d$  be a random vector and  $\pmb{v}_1, \dots, \pmb{v}_n$  i.i.d. copies of  $\pmb{v}$ . Suppose that  $\| \pmb{v} \| \leqslant L$ . Then the estimator  $\bar{\Sigma} = \frac{1}{n} \sum_{i=1}^{n} \pmb{x}_i \pmb{x}_i^\top$  satisfies for all  $t > 0$
$$
\mathbb {P} \left(\left\| \bar {\boldsymbol {\Sigma}} - \mathbb {E} \boldsymbol {x} _ {1} x _ {1} ^ {\top} \right\| \geqslant t\right) \leqslant 2 d \cdot \exp \left(- \frac {t ^ {2} n}{2 L \| \boldsymbol {\Sigma} \| + 2 L t}\right).
$$
We will use this to prove that sampling preserves bounded  $f$ -moments.
Lemma J.4. Let  $k, \ell, n, d \in \mathbb{N}_{\geqslant 1}$ ,  $\sigma, \rho > 0$ ,  $\delta \in (0,1)$ . Let  $\nu$  be a distribution with  $(2k, \ell)$ -certifiable  $\sigma$ -bounded  $f$ -moments, and  $\eta_1, \ldots, \eta_n \sim_{i.i.d.} \nu$ . Suppose that  $n \geqslant 10 \cdot d^k \cdot \log(d / \delta)$  and that  $\forall x \in \mathbb{R}^d$ ,  $\|f(x)\| \leqslant \rho \sqrt{d}$ .
Then with probability at least  $1 - \delta$ , uniform distribution over the set  $\{\pmb{\eta}_1, \dots, \pmb{\eta}_n\}$  has  $(2k, \ell)$ -certifiable  $(\sigma + \rho)$ -bounded  $f$ -moments. In other words, with probability at least  $1 - \delta$ ,
$$
\left| \right. \frac {v}{\ell} \left(v ^ {\otimes k}\right) ^ {\top} \left(\frac {1}{n} \sum_ {i = 1} ^ {n} f \left(\boldsymbol {\eta} _ {i}\right) ^ {\otimes k} \left(f \left(\boldsymbol {\eta} _ {i}\right) ^ {\otimes k}\right) ^ {\top}\right) v ^ {\otimes k} \leqslant \left(\left(\sigma + \rho\right) \cdot \| v \|\right) ^ {2 k}.
$$
If  $\pmb{\eta}_1, \dots, \pmb{\eta}_n$  follow an elliptical distribution with scatter matrix  $\Sigma$  and  $\sigma = O(\sqrt{k\|\Sigma\|})$ , we only need  $n \gtrsim (\frac{\mathrm{r}(\Sigma)}{k})^k \cdot \log(d/\delta)$ . Further, the uniform distribution has 2σ-bounded f-moments.
Proof. Let  $T_{k} = \mathbb{E}f(\pmb {\eta})^{\otimes k}\left(f(\pmb {\eta})^{\otimes k}\right)^{\top}$ . Note that  $\{\pmb {\eta}\}$  having  $(2k,\ell)$ -certifiable  $\sigma$ -bounded  $f$ -moments is equivalent to the fact that the following SoS proof exists
$$
\left| \frac {v}{\ell} (v ^ {\otimes k}) ^ {\top} T _ {k} v ^ {\otimes k} \leqslant (\sigma \cdot \| v \|) ^ {2 k} \right..
$$
Denote  $\bar{T}_k = \frac{1}{n}\sum_{i=1}^{n}f(\pmb{\eta}_i)^{\otimes k}\left(f(\pmb{\eta}_i)^{\otimes k}\right)^\top$ . Notice that by Fact K.3
$$
\left| \right. \frac {v}{\ell} \left(v ^ {\otimes k}\right) ^ {\top} \bar {T} _ {k} v ^ {\otimes k} = \left(v ^ {\otimes k}\right) ^ {\top} \left[ T _ {k} + \bar {T} _ {k} - T _ {k} \right] v ^ {\otimes k} \leqslant \| v \| ^ {2 k} \cdot \left(\sigma^ {2 k} + \left\| \bar {T} _ {k} - T _ {k} \right\|\right).
$$
Hence, it is enough to bound  $\left\| \bar{T}_k - T_k \right\|$ . Our strategy will be to use Fact J.1. To this end, let  $M = f(\pmb{\eta})^{\otimes k} \left(f(\pmb{\eta})^{\otimes k}\right)^{\top}$  and for  $i \in [n]$ , let  $M_i = f(\pmb{\eta}_i)^{\otimes k} \left(f(\pmb{\eta}_i)^{\otimes k}\right)^{\top}$ . This implies that  $T_k = \mathbb{E}M$  and  $\bar{T}_k = \frac{1}{n} \sum_{i=1}^{n} M_i$ . Note that since the entries of  $f$  are bounded by  $h$  in magnitude it follows that for each  $i$
$$
\left\| M _ {i} \right\| = \left\| f \left(\boldsymbol {\eta} _ {i}\right) \right\| ^ {2 k} \leqslant \rho^ {2 k} d ^ {k}.
$$
$$
\boldsymbol {M} ^ {2} = \left\| f (\boldsymbol {\eta}) \right\| ^ {2 k} \cdot f (\boldsymbol {\eta}) ^ {\otimes k} \left(f (\boldsymbol {\eta}) ^ {\otimes k}\right) ^ {\top} \preceq \rho^ {2 k} d ^ {k} \cdot f (\boldsymbol {\eta}) ^ {\otimes k} \left(f (\boldsymbol {\eta}) ^ {\otimes k}\right) ^ {\top}.
$$
Since  $\left\| \mathbb{E}f(\pmb {\eta})^{\otimes k}\left(f(\pmb {\eta})^{\otimes k}\right)^{\top}\right\| \leqslant \sigma^{2k}$ , it holds that
$$
\left\| \mathbb {E} M ^ {2} \right\| \leqslant \rho^ {2 k} d ^ {k} \cdot \left\| \mathbb {E} f (\boldsymbol {\eta}) ^ {\otimes k} \left(f (\boldsymbol {\eta}) ^ {\otimes k}\right) ^ {\top} \right\| \leqslant (\rho \sigma) ^ {2 k} d ^ {k}.
$$
10Here we use the parametrization from Appendix G, where  $\mathrm{Tr}\Sigma = d$
By Fact J.1,
$$
\mathbb {P} \left(\left\| \bar {\boldsymbol {T}} _ {k} - T _ {k} \right\| \geqslant \frac {1 0 \rho^ {2 k} d ^ {k} \cdot \log (d / \delta)}{n} + \sigma^ {k} \sqrt {\frac {1 0 \rho^ {2 k} d ^ {k} \cdot \log (d / \delta)}{n}}\right) \leqslant \delta . \tag {J.1}
$$
Hence with probability at least  $1 - \delta$
$$
\left| \frac {v}{\ell} (v ^ {\otimes k}) ^ {\top} \bar {\boldsymbol {T}} _ {k} v ^ {\otimes k} \leqslant \| v \| ^ {2 k} \cdot (\sigma^ {2 k} + \rho^ {2 k} + \rho^ {k} \sigma^ {k}) \leqslant \| v \| ^ {2 k} \cdot (\sigma + \rho) ^ {2 k} \right.
$$
To see the improved sample complexity for elliptical distributions, note that since  $\mathrm{r}(\Sigma) = \frac{\mathrm{Tr}(\Sigma)}{\|\Sigma\|} = \frac{d}{\|\Sigma\|}$  and  $\sigma = O(\rho \sqrt{k\|\Sigma\|})$ , it follows that
$$
\rho^ {2 k} d ^ {k} = \rho^ {2 k} \| \Sigma \| ^ {k} \mathrm {r} (\Sigma) ^ {k} \cdot \frac {k ^ {k}}{k ^ {k}} = \sigma^ {2 k} \cdot \left(\frac {\mathrm {r} (\Sigma)}{k}\right) ^ {k}.
$$
Thus, if  $n \gtrsim \left(\frac{\mathrm{r}(\Sigma)}{k}\right)^k \cdot \log(d / \delta)$ , then  $\left\| \bar{T}_k - T_k \right\|$  in Eq. (J.1) is at most  $\sigma^{2k}$  with probability at least  $1 - \delta$ .
Fact J.5. [DTV16] Let  $\pmb{\eta}$  be an elliptically distributed vector with location 0 and scatter matrix  $\Sigma$ . Let  $s(\eta) = \frac{\sqrt{\mathrm{Tr}(\Sigma)}}{\|\eta\|}\pmb{\eta}$ . Then
$$
\left\| \mathbb {E} s (\boldsymbol {\eta}) s (\boldsymbol {\eta}) ^ {\top} \right\| \leqslant \| \Sigma \| .
$$
Fact J.6. [Bec09] Let  $\pmb{x} \sim N(0, \Sigma)$ . Then for all  $t > 0$ ,
$$
\mathbb {P} \left[ \| \boldsymbol {x} \| ^ {2} \leqslant \operatorname {T r} (\Sigma) - \sqrt {2 t} \| \Sigma \| _ {\mathrm {F}} \right] \leqslant \exp (- t)
$$
$$
\mathbb {P} \left[ \| \boldsymbol {x} \| ^ {2} \geqslant \operatorname {T r} (\Sigma) + \sqrt {2 t} \| \Sigma \| _ {\mathrm {F}} + t \| \Sigma \| \right] \leqslant \exp (- t).
$$
Fact J.7. [DK22] Let  $\xi_1, \ldots, \xi_n$  be independent zero mean 1-sub-Gaussian  $d$ -dimensional vectors. Let  $\varepsilon, \delta \in (0,1)$ , and suppose that  $n \geqslant d + \log(1/\delta)$ . Let  $\mathcal{W}_{\varepsilon} = \{w \in \mathbb{R}^d \mid 0 \leqslant w_i \leqslant 1/n, \sum_{i=1}^{n} |w_i - 1/n| \leqslant \varepsilon\}$ . Then, with probability  $1 - \delta$ , for every  $w \in \mathcal{W}_{\varepsilon}$
$$
\left\| \sum_ {i = 1} ^ {n} w _ {i} \boldsymbol {\xi} _ {i} \right\| \leqslant O \left(\varepsilon \sqrt {\log (1 / \varepsilon)} + \sqrt {\frac {d + \log (1 / \delta)}{n}}\right),
$$
$$
\left\| \sum_ {i = 1} ^ {n} w _ {i} \boldsymbol {\xi} _ {i} \boldsymbol {\xi} _ {i} ^ {\top} - \frac {1}{n} \sum_ {i = 1} ^ {n} \mathbb {E} \boldsymbol {\xi} \boldsymbol {\xi} ^ {\top} \right\| \leqslant O \left(\varepsilon \log (1 / \varepsilon) + \sqrt {\frac {d + \log (1 / \delta)}{n}}\right).
$$
Similarly, with probability  $1 - \delta$  it holds that for every set  $T\subseteq [n]$  with  $|T|\leqslant \varepsilon n$  and every  $w\in \mathcal{W}_e$  it holds that
$$
\left\| \sum_ {i \in T} w _ {i} \xi_ {i} \xi_ {i} ^ {\top} \right\| \leqslant O \left(\varepsilon \log (1 / \varepsilon) + \sqrt {\frac {d + \log (1 / \delta)}{n}}\right).
$$
Lemma J.8. Let  $\varepsilon, \delta \in (0,1)$  and suppose that  $n \geqslant d\log d + \log(1 / \delta)$ . Let  $\xi_1, \ldots, \xi_n$  be independent zero mean 1-sub-Gaussian  $d$ -dimensional vectors, and let  $a_1, \ldots, a_n \in \mathbb{R}^d$  be fixed (non-random) vectors such that  $\|a_i\| \leqslant 100\varepsilon \sqrt{d} / \alpha$  for some  $\alpha \in (0,1)$ . Suppose in addition that  $\|\xi_i\| \leqslant 10\sqrt{d}$  with probability 1. Let  $\mathcal{W}_{\varepsilon} = \{w \in \mathbb{R}^d \mid 0 \leqslant w_i \leqslant 1 / n, \sum_{i=1}^{n} |w_i - 1 / n| \leqslant \varepsilon\}$ . Then, with probability  $1 - \delta$ , for every  $w \in \mathcal{W}_{\varepsilon}$ ,
$$
\sum_ {i = 1} ^ {n} w _ {i} \left(\pmb {\xi} _ {i} + a _ {i}\right) \left(\pmb {\xi} _ {i} + a _ {i}\right) ^ {\top} \succeq \frac {1}{n} \sum_ {i = 1} ^ {n} \mathbb {E} \pmb {\xi} \pmb {\xi} ^ {\top} - O \left(\varepsilon \log (1 / \varepsilon) + \varepsilon / \alpha + \sqrt {\frac {d \log (d) + \log (1 / \delta)}{n}}\right).
$$
$$
\sum_ {i = 1} ^ {n} w _ {i} \left(\boldsymbol {\xi} _ {i} + a _ {i}\right) \left(\boldsymbol {\xi} _ {i} + a _ {i}\right) ^ {\top} = \sum_ {i = 1} ^ {n} w _ {i} \boldsymbol {\xi} _ {i} \boldsymbol {\xi} _ {i} ^ {\top} + \sum_ {i = 1} ^ {n} w _ {i} \left(\boldsymbol {\xi} _ {i} a _ {i} ^ {\top} + a _ {i} \boldsymbol {\xi} _ {i} ^ {\top} + a _ {i} a _ {i} ^ {\top}\right).
$$
Let us bound  $u \sum_{i=1}^{n} w_i (\pmb{\xi}_i a_i^\top + a_i \pmb{\xi}_i^\top + a_i a_i^\top) u^\top$  for all unit vectors  $u \in \mathbb{R}^d$ . Note that since  $\mathcal{W}_{\varepsilon}$  is a polytope, it is enough to bound this value only for  $w$  that are its vertices, that is, the indicators of sets of size at least  $(1 - \varepsilon)n$ , normalized by  $1/n$ . Let  $S \subset [n]$  be an arbitrary fixed (non-random) set, and let  $u \in \mathbb{R}^d$  be an arbitrary fixed (non-random) unit vector. Then
$$
u ^ {\top} \sum_ {i \in S} \frac {1}{n} \left(\boldsymbol {\xi} _ {i} a _ {i} ^ {\top} + a _ {i} \boldsymbol {\xi} _ {i} ^ {\top} + a _ {i} a _ {i} ^ {\top}\right) u = \frac {1}{n} \sum_ {i \in S} 2 \langle \boldsymbol {\xi} _ {i}, u \rangle \langle \boldsymbol {a} _ {i}, u \rangle + \langle \boldsymbol {a} _ {i}, u \rangle^ {2}.
$$
By Hoeffding's inequality, with probability at least  $1 - \delta$
$$
\sum_ {i \in S} 2 \langle \boldsymbol {\xi} _ {i}, u \rangle \langle \boldsymbol {a} _ {i}, u \rangle + \langle \boldsymbol {a} _ {i}, u \rangle^ {2} \geqslant - O \left(\| A _ {S} u \| \sqrt {\log (1 / \delta)}\right) + \| A _ {S} u \| ^ {2} \geqslant - O \left(\log (1 / \delta)\right).
$$
where  $A_{S}\in \mathbb{R}^{|S|\times d}$  is the matrix with rows  $a_{i}$  for  $i\in S$
By union bound over all sets of size at least  $(1 - \varepsilon)n$ , with probability  $1 - \delta$ , for every set  $S$  of size at least  $(1 - \varepsilon)n$ ,
$$
\sum_ {i \in S} 2 \langle \boldsymbol {\xi} _ {i}, u \rangle \langle \boldsymbol {a} _ {i}, u \rangle + \langle \boldsymbol {a} _ {i}, u \rangle^ {2} \geqslant - O (\varepsilon n \log (1 / \varepsilon) + \log (1 / \delta)).
$$
Let  $\mathcal{N}$  be an  $(0.01 / d)$ -net in the  $d$ -dimensional unit ball of size  $|\mathcal{N}| \leqslant (300d)^d$ . Then by union bound over  $\mathcal{N}$ , we get that with probability  $1 - \delta$ , for every set  $S$  of size at least  $(1 - \varepsilon)n$ , and for every  $u \in \mathcal{N}$ ,
$$
\sum_ {i \in S} 2 \langle \boldsymbol {\xi} _ {i}, u \rangle \langle \boldsymbol {a} _ {i}, u \rangle + \langle \boldsymbol {a} _ {i}, u \rangle^ {2} \geqslant - O (\varepsilon n \log (1 / \varepsilon) + d \log (d) + \log (1 / \delta)).
$$
Now, if some unit  $u$  is  $(0.01 / d)$ -close to  $u' \in \mathcal{N}$ , we get
$$
\begin{array}{l} \frac {1}{n} \sum_ {i \in S} 2 \langle \boldsymbol {\xi} _ {i}, u \rangle \langle \boldsymbol {a} _ {i}, u \rangle \geqslant \frac {1}{n} \sum_ {i \in S} 2 \langle \boldsymbol {\xi} _ {i}, u ^ {\prime} \rangle \langle \boldsymbol {a} _ {i}, u ^ {\prime} \rangle - O \left(\max  _ {i} \| \boldsymbol {\xi} _ {i} \| \cdot r \cdot \| u - u ^ {\prime} \|\right) \\ \geqslant - O \left(\varepsilon \log (1 / \varepsilon) + \varepsilon / \alpha + \frac {d \log (d) + \log (1 / \delta)}{n}\right). \\ \end{array}
$$
Using the concentration of  $\sum_{i=1}^{n} w_i \pmb{\xi}_i \pmb{\xi}_i^\top$  from Fact J.7, we get the desired bound.
K Sum-of-Squares Toolkit
The following fact can be found in [KS17a, Lemma A.2]
Fact K.1. For all  $k\in \mathbb{N}_{\geqslant 1}$  it holds that
$$
\left| \frac {X , Y}{2 k} (X + Y) ^ {2 k} \leqslant 2 ^ {2 k - 1} \cdot \left(X ^ {2 k} + Y ^ {2 K}\right). \right.
$$
The next fact shows that sum-of-squares captures the Cauchy-Schwarz Inequality, a proof can be found in [MSS16, Lemma A.1]
Fact K.2. For all  $n \in \mathbb{N}_{\geqslant 1}$  it holds that
$$
\left| \frac {X _ {1} , Y _ {1} , \dots , X _ {n} , Y _ {n}}{2} \left(\sum_ {i = 1} ^ {n} X _ {i} Y _ {i}\right) ^ {2} \leqslant \left(\sum_ {i = 1} ^ {n} X _ {i} ^ {2}\right) \left(\sum_ {j = 1} ^ {n} Y _ {i} ^ {2}\right) \right.
$$
Fact K.3. Let  $M \in \mathbb{R}^{n \times n}$  be a symmetric matrix and  $X$  be an  $n$ -vector of formal variables. Then it holds that
$$
\left| \frac {X}{2} \langle X, M X \rangle \leqslant \| M \| \| X \| ^ {2} \right..
$$
Proof. We can rewrite the inequality as  $X^{\top} \left( \| M \| \cdot I_{n} - M \right) X \geqslant 0$ . Since  $\| M \| \cdot I_{n} - M$  is positive semi-definite, it follows that there exists a matrix  $L \in \mathbb{R}^{n \times n}$  such that  $\| M \| \cdot I_{n} - M = LL^{\top}$ . Hence,
$$
X ^ {\top} \left(\| M \| \cdot I _ {n} - M\right) X = \| L X \| ^ {2},
$$
which is a sum of squares in  $X$ .
Lemma K.4. Let  $k$  be a positive integer, and let  $\boldsymbol{\eta}$  be elliptical  $d$ -dimensional vector with location 0 and scatter matrix  $\Sigma$  that satisfies
Let  $f: \mathbb{R}^d \to \mathbb{R}^d$  be a projection onto the Euclidean ball of radius  $R$  centered at 0.
Then  $\pmb{\eta}$  has  $(2k, 2k)$ -certifiable  $q$ -bounded  $f$ -moments, where
$$
q = 2 R \sqrt {\frac {k \| \Sigma \|}{\operatorname {T r} (\Sigma)}}.
$$
Proof. Let  $v_{1}, \ldots, v_{d}$  be variables and consider the polynomial  $\mathbb{E}\left\langle f(\boldsymbol{\eta}), v \right\rangle^{2k}$ . Let  $s(\boldsymbol{\eta}) = \frac{R}{\|\boldsymbol{\eta}\|} \boldsymbol{\eta} = \frac{R}{\|f(\boldsymbol{\eta})\|} f(\boldsymbol{\eta})$ . Since spherical projection of elliptical distributions depends only on  $\Sigma$  (see Theorem 35 in [Fra04]),  $s(\boldsymbol{\eta})$  has the same distribution as  $s(\boldsymbol{w})$ , where  $\boldsymbol{w} \sim N(0, \Sigma)$ . Hence
$$
\begin{array}{l} \left| \frac {v}{2 k} \langle f (\boldsymbol {\eta}), v \rangle^ {2 k} \leqslant \langle s (\boldsymbol {\eta}), v \rangle^ {2 k} \right. \\ = R ^ {2 k} \left\langle \frac {\boldsymbol {w}}{\| \boldsymbol {w} \|}, v \right\rangle^ {2 k} \\ = R ^ {2 k} \cdot \mathbf {1} _ {\left[ \| \boldsymbol {w} \| ^ {2} > \frac {\operatorname {T r} (\Sigma)}{2} \right]} \left\langle \frac {\boldsymbol {w}}{\| \boldsymbol {w} \|}, v \right\rangle^ {2 k} + R ^ {2 k} \cdot \mathbf {1} _ {\left[ \| \boldsymbol {w} \| ^ {2} \leqslant \frac {\operatorname {T r} (\Sigma)}{2} \right]} \left\langle \frac {\boldsymbol {w}}{\| \boldsymbol {w} \|}, v \right\rangle^ {2 k} \\ \leqslant \left(\frac {2 R ^ {2}}{\operatorname {T r} (\Sigma)}\right) ^ {k} \langle \boldsymbol {w}, v \rangle^ {2 k} + R ^ {2 k} \cdot \mathbf {1} _ {\left[ \| \boldsymbol {w} \| ^ {2} \leqslant \frac {\operatorname {T r} (\Sigma)}{2} \right]} \cdot \| v \| ^ {2 k}. \\ \end{array}
$$
By Fact J.6,
$$
\mathbb {P} \left[ \| \boldsymbol {w} \| ^ {2} \leqslant \operatorname {T r} (\Sigma) / 2 \right] \leqslant d ^ {- 2 k}.
$$
$$
\left| \frac {v}{2 k} \mathbb {E} \langle f (\boldsymbol {\eta}), v \rangle^ {2 k} \leqslant \left(\frac {2 R ^ {2}}{\operatorname {T r} (\Sigma)}\right) ^ {k} \mathbb {E} \langle \boldsymbol {w}, v \rangle^ {2 k} + (R / d) ^ {2 k} \| v \| ^ {2 k}. \right.
$$
Since  $\pmb{w}$  is  $(2k, 2k)$ -certifiably 1-subgaussian (see Lemma 5.1 in [KS17b]), and since  $\|\Sigma\| / \operatorname{Tr}(\Sigma) \geqslant 1/d$ ,
$$
\left| \frac {v}{2 k} \mathbb {E} \langle f (\boldsymbol {\eta}), v \rangle^ {2 k} \leqslant 2 \cdot \left(\frac {2 R ^ {2} k \| \Sigma \|}{\operatorname {T r} (\Sigma)}\right) ^ {k} \| v \| ^ {2 k}. \right.
$$