SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \leqslant \varepsilon^ {1 - \frac {1}{2 k}} \left(\sum_ {i = 1} ^ {n} w _ {i} \langle f (\boldsymbol {\eta} _ {i} ^ {*}), u \rangle^ {2 k}\right) ^ {\frac {1}{2 k}} + \varepsilon^ {1 - \frac {1}{2 k}} \left(\sum_ {i = 1} ^ {n} w _ {i} \langle f (\hat {\mu} - y _ {i}), u \rangle^ {2 k}\right) ^ {\frac {1}{2 k}} \\ + \varepsilon^ {1 - \frac {1}{2 k}} \left(\sum_ {i = 1} ^ {n} w _ {i} \left\langle f \left(\mu^ {*} - y _ {i}\right) - f \left(\hat {\mu} - y _ {i}\right), u \right\rangle^ {2 k}\right) ^ {\frac {1}{2 k}}. \\ \end{array}
$$
We bound the three terms in the square bracket one by one.
The first term can be bounded as follows
$$
\sum_ {i = 1} ^ {n} w _ {i} \left\langle f \left(\boldsymbol {\eta} _ {i} ^ {*}\right), u \right\rangle^ {2 k} \leqslant \sum_ {i = 1} ^ {n} \frac {1}{n} \left\langle f \left(\boldsymbol {\eta} _ {i} ^ {*}\right), u \right\rangle^ {2 k} \leqslant \bar {\mathbf {m}} _ {2 k} ^ {2 k} \| u \| ^ {2 k}.
$$
The second one is at most  $\bar{\mathfrak{m}}_{2k}\| u\|$  by definition of  $\hat{\mathfrak{m}}_{2k}^w$ . For the third, we use that  $f$  is 1-Lipschitz to bound
$$
\left(\sum_ {i = 1} ^ {n} w _ {i} \cdot \langle f (\mu^ {*} - y _ {i}) - f (\hat {\mu} - y _ {i}), u \rangle^ {2 k}\right) ^ {\frac {1}{2 k}} \leqslant \left(\sum_ {i = 1} ^ {n} w _ {i} \cdot \| f (\mu^ {*} - y _ {i}) - f (\hat {\mu} - y _ {i}) \| ^ {2 k} \| u \| ^ {2 k}\right) ^ {\frac {1}{2 k}} \leqslant \| u \| ^ {2}.
$$
$$
\sum_ {i = 1} ^ {n} \mathbf {1} \left\{i \in S _ {b} \right\} \cdot w _ {i} \cdot \langle f (\mu^ {*} - y _ {i}) - f (\mu^ {*} - \boldsymbol {y} _ {i} ^ {*}), u \rangle \leqslant \varepsilon^ {1 - \frac {1}{2 k}} \| u \| \cdot \left(\hat {\mathfrak {m}} _ {2 k} ^ {w} + \bar {\mathfrak {m}} _ {2 k} + \| u \|\right).
$$
Putting all the above together, we obtain the desired bound.
D Filtering Algorithm for Bounded Clipped Covariance
In this section, we show that the filtering algorithm for robust mean estimation under moment constraints (cf. for example [Li19, DK22]) can be adapted to also yield an efficient algorithm for robustly estimating the location parameter of symmetric distributions. For now, we focus on the case that  $k = 2$ , we discuss the extension to higher moments in Appendix E. Again, we will focus on the abstract setting and show that we can apply these results to semi-product and elliptical distributions. We will closely follow the exposition in [Li19, DK22].
Recall that we denote  $\mathcal{L}^w (\mu) = \sum_{i = 1}^n w_iF\left(\mu -y_i\right)$ .
Assumption D.1 (Goodness Condition). Let  $F \colon \mathbb{R}^d \to \mathbb{R}$ ,  $f = \nabla F$  be such that  $f$  is 1-Lipschitz. Let  $\mu^{*} \in \mathbb{R}^{d}, 0 < \sigma, 0 < \alpha < 1$  and  $y_{1}, \ldots, y_{n} \in \mathbb{R}^{d}$ . Let  $D$  be a distribution over  $\mathbb{R}^d$  and  $\pmb{\eta}_1^*, \dots, \pmb{\eta}_n^*$  be  $n$  i.i.d_samples from  $D$ . We say that the goodness condition holds, if
1.  $\alpha \| \hat{\mu}^w -\mu^*\| ^2\leqslant 10\langle \nabla \mathcal{L}^w (\mu^*),\mu^* -\hat{\mu}^w\rangle$
2.  $\left\| \frac{1}{n}\sum_{i = 1}^{n}f(\pmb{\eta}_i^*)\right\| \leqslant \frac{\alpha}{100}\sigma$
3.  $\bar{\mathbf{m}}_2 = \max_{v\in \mathbb{S}^{d - 1}}\sqrt{\frac{1}{n}\sum_{i = 1}^{n}\left\langle f\left(\pmb{\eta}_i^*\right),v\right\rangle^2}\leqslant \sigma .$
Under this condition we can show the following theorem.
Theorem D.2. Let  $\mu^{*} \in \mathbb{R}^{d}, 0 < \sigma, 0 < \alpha < 1, \varepsilon > 0$ , and  $\pmb{\eta}_{1}^{*}, \dots, \pmb{\eta}_{n}^{*}$  be  $n$  i.i.d.samples from some distribution  $D$  over  $\mathbb{R}^d$ . Assume that  $\sqrt{\varepsilon} \leqslant \frac{\alpha}{1000}$ . Let  $y_{1}, \ldots, y_{n}$  be an  $\varepsilon$ -corruption of  $\pmb{y}_{1}^{*} = \mu^{*} + \pmb{\eta}_{1}^{*}, \ldots, \pmb{y}_{n}^{*} = \mu^{*} + \pmb{\eta}_{n}^{*}$ . Assume Assumption D.1 (the goodness condition) holds, then Algorithm D.3, given  $y_{1}, \ldots, y_{n}$  and  $\sigma$ , terminates after at most  $2\varepsilon n + 1$  iterations and computes  $\hat{\mu}$  such that
$$
\| \hat {\mu} ^ {w} - \mu^ {*} \| \leqslant O \left(\frac {\left\| \frac {1}{n} \sum_ {i = 1} ^ {n} f \left(\boldsymbol {\eta} _ {i} ^ {*}\right) \right\|}{\alpha} + \frac {\sqrt {\varepsilon}}{\alpha} \cdot \sigma\right).
$$
Moreover each iteration can be implemented in time  $n^{O(1)}d^{O(1)}$ .
To describe the algorithm, let  $C > 0$  be some universal constant and for  $\mu \in \mathbb{R}^d$ ,  $w \in \mathcal{W}_{\varepsilon}$  let
$$
\hat {\mu} (w) = \min  _ {\mu \in \mathbb {R} ^ {d}} \mathcal {L} ^ {w} (\mu),
$$
$$
\Sigma_ {f} (w) = \sum_ {i = 1} ^ {n} w _ {i} f (\hat {\mu} ^ {w} - y _ {i}) f (\hat {\mu} ^ {w} - y _ {i}) ^ {\top}.
$$
We will use the following Filtering Algorithm:
Algorithm D.3 (Filtering Algorithm).
Input:  $\varepsilon$ -corrupted sample  $y_{1},\ldots ,y_{n}$  and  $\sigma >0$
Output: Location estimate  $\hat{\mu}$
Let  $w^{(0)} = \frac{1}{n}\mathbb{1}_n$
- Compute  $\hat{\mu}^{(0)} = \min_{\mu \in \mathbb{R}^d}\mathcal{L}^{w^{(0)}}(\mu)$  and  $\Sigma_f^{(0)} = \Sigma_f(w^{(0)})$
- Let  $t = 0$ .
- while  $\left\| \Sigma_f^{(t)}\right\| >100\cdot \sigma$  do
- Compute  $v^{(t)}$  the top eigenvector of  $\Sigma_f^{(t)}$ .
- For  $i \in [n]$ , compute  $\tau_i^{(t)} = \left\langle v^{(t)}, f(\hat{\mu}^{(t)} - y_i) \right\rangle^2$ .
- For  $i \in [n]$ , set  $w_i^{(t+1)} = \left( 1 - \frac{\tau_i}{\tau_{\max}} \right) w_i^{(t)}$ , where  $\tau_{\max} = \max_{i \in [n]} \tau_i$ .
- Compute  $\hat{\mu}^{(t + 1)} = \min_{\mu \in \mathbb{R}^d}f^{w^{(t + 1)}}(\mu)$  and  $\Sigma_f^{(t + 1)} = \Sigma_f(w^{(t + 1)})$
-  $t\gets t + 1$
Output  $\hat{\mu}^{(t)}$
The proof of Theorem D.2 is very similar to the one in [Li19]. We will use the following lemma whose proof we will give at the end of this section
Lemma D.4. Assume that Theorem D.2  $\| \Sigma_f^{(t)}\| >100\cdot \sigma$  and
$$
\sum_ {i \in S _ {g}} \left(\frac {1}{n} - w _ {i} ^ {(t)}\right) <   \sum_ {i \in S _ {b}} \left(\frac {1}{n} - w _ {i} ^ {(t)}\right).
$$
$$
\sum_ {i \in S _ {g}} \left(\frac {1}{n} - w _ {i} ^ {(t + 1)}\right) <   \sum_ {i \in S _ {b}} \left(\frac {1}{n} - w _ {i} ^ {(t + 1)}\right).
$$
With this in hand, we will prove Theorem D.2.
Proof of Theorem D.2. First note, that every iteration can clearly be implanted to run in time  $n^{O(1)}d^{O(1)}$ . First, we will show that the algorithm terminates after at most  $\lceil 2\varepsilon n\rceil$  iterations. We will prove it by contradiction. Suppose that the algorithm does not terminate after  $T = \lceil 2\varepsilon n\rceil$  iterations. Note that the number of zero entries of  $w^{(t)}$  increases by at least 1 in every iteration. Hence, after  $T$  iterations we have set at least  $\varepsilon n$  entries of  $w$  to zero whose index lies in  $S_{g}$ . By assumption that the algorithm didn't terminate and Lemma D.4 it holds that
$$
\varepsilon \leqslant \sum_ {i \in S _ {g}} \left(\frac {1}{n} - w _ {i} ^ {(T)}\right) <   \sum_ {i \in S _ {b}} \left(\frac {1}{n} - w _ {i} ^ {(T)}\right) \leqslant \frac {| S _ {b} |}{n} \leqslant \varepsilon .
$$
Next, we prove the correctness of the algorithm. Let  $T$  be the index of the last iteration of the algorithm before termination. Note that by our invariant
$$
\left\| \frac {1}{n} - w ^ {(T)} \right\| _ {1} = \sum_ {i \in S _ {g}} \frac {1}{n} - w _ {i} ^ {(T)} + \sum_ {i \in S _ {b}} \frac {1}{n} - w _ {i} ^ {(T)} <   2 \sum_ {i \in S _ {b}} \frac {1}{n} - w _ {i} ^ {(T)} \leqslant 2 \varepsilon .
$$
Since also  $0 \leqslant w^{(T)} \leqslant \frac{1}{n}$ , it follows that  $w^{(T)} \in \mathcal{W}_{2\varepsilon}$ . By Theorem C.1 and since  $\left\| \Sigma_f^{(T)} \right\| \leqslant 100 \cdot \sigma$  it follows that
Lastly, we will prove Lemma D.4.
Proof of Lemma D.4. For simplicity, let  $w = w^{(t)}$  and  $w' = w^{(t+1)}$ . Note, that it is enough to show that
$$
\sum_ {i \in S _ {g}} w _ {i} - w _ {i} ^ {\prime} <   \sum_ {i \in S _ {b}} w _ {i} - w _ {i} ^ {\prime}.
$$
Further, recall that  $w_{i}^{\prime} = \left(1 - \frac{\tau_{i}}{\tau_{\max}}\right)w_{i}$ , so for all  $i\in [n]$ ,  $w_{i} - w_{i}^{\prime} = \frac{1}{\tau_{\max}}\tau_{i}w_{i}$ . Hence is enough to show that
$$
\sum_ {i \in S _ {g}} \tau_ {i} w _ {i} <   \sum_ {i \in S _ {b}} \tau_ {i} w _ {i}.
$$
Since  $S_{g}$  and  $S_{b}$  partition  $[n]$  and
$$
\sum_ {i = 1} ^ {n} w _ {i} \tau_ {i} = \sum_ {i = 1} ^ {n} w _ {i} \left\langle v ^ {(t)}, f \left(\hat {\mu} ^ {(t)} - y _ {i}\right) \right\rangle^ {2} = \left(v ^ {(t)}\right) ^ {\top} \Sigma_ {f} ^ {(t)} \left(v ^ {(t)}\right) = \left\| \Sigma_ {f} ^ {(t)} \right\|,
$$
we can prove  $\sum_{i\in S_g}\tau_iw_i <   \sum_{i\in S_b}\tau_iw_i$  by showing that
$$
\sum_ {i \in S _ {g}} \tau_ {i} w _ {i} <   \frac {\| \Sigma_ {f} ^ {(t)} \|}{2}.
$$
To this end, note that
$$
\begin{array}{l} \sum_ {i \in S _ {g}} \tau_ {i} w _ {i} = \sum_ {i \in S _ {g}} w _ {i} \left<   v ^ {(t)}, f \left(\hat {\mu} ^ {(t)} - y _ {i}\right)\right> ^ {2} \\ \leqslant \frac {2}{n} \sum_ {i \in [ n ]} \left\langle v ^ {(t)}, f \left(\mu^ {*} - y _ {i}\right) \right\rangle^ {2} + \frac {2}{| S _ {g} |} \sum_ {i \in S _ {g}} \left\langle v ^ {(t)}, f \left(\hat {\mu} ^ {(t)} - y _ {i}\right) - f \left(\mu^ {*} - y _ {i}\right) \right\rangle^ {2}. \\ \end{array}
$$
The first term is at most  $2\sigma^2$ . For the second term we observe that since  $f$  is 1-Lipschitz, for all  $i \in [n]$  it holds that
$$
\left\langle v ^ {(t)}, f \left(\hat {\mu} ^ {(t)} - y _ {i}\right) - f \left(\mu^ {*} - y _ {i}\right) \right\rangle^ {2} \leqslant \left\| f \left(\hat {\mu} ^ {(t)} - y _ {i}\right) - f \left(\mu^ {*} - y _ {i}\right) \right\| ^ {2} \leqslant \left\| \hat {\mu} ^ {(t)} - \mu^ {*} \right\| ^ {2}.
$$
Hence, by Theorem C.1,
$$
\sum_ {i \in S _ {g}} \tau_ {i} w _ {i} \leqslant 2 \sigma^ {2} + \left\| \hat {\mu} ^ {(t)} - \beta^ {*} \right\| ^ {2} \leqslant 2 \sigma^ {2} + \frac {1 0 0 ^ {2}}{\alpha^ {2}} \left(\left\| \frac {1}{n} \sum_ {i = 1} ^ {n} f (\boldsymbol {\eta} _ {i} ^ {*}) \right\| + \sqrt {\varepsilon} \cdot \Sigma_ {f} ^ {(t)}\right) ^ {2} <   \frac {\Sigma_ {f} ^ {(t)}}{2},
$$
where we used  $\varepsilon \lesssim \alpha^2$  and  $\left\| \frac{1}{n}\sum_{i = 1}^{n}f(\pmb{\eta}_i^*)\right\| \lesssim \alpha \sigma$ .
E Filtering Algorithm for Bounded Higher-Order Clipped Moments
In this section, we show how to adapt the filtering algorithm to incorporate higher-order moments of the transformed noise. We follow closely the exposition in [DK22, Chapter 6.5]. In particular, consider the following definition similar to the by now standard notion of cerifiably bounded moments or certifiable subgaussianity [KSS18, HL18]. Let  $f\colon \mathbb{R}^d\to \mathbb{R}^d$  be some function.
Definition E.1. Let  $k, \ell \in \mathbb{N}$  and  $\sigma > 0$ . We say a distribution  $D$  over  $\mathbb{R}^d$  has  $(2k, \ell)$ -certifiably  $\sigma$ -bounded  $f$ -moments if there is a degree- $\ell$  SoS proof (cf. Appendix B), in formal variables  $v$ , that
$$
\underset {\boldsymbol {\eta} \sim D} {\mathbb {E}} \langle f (\boldsymbol {\eta}), v \rangle^ {2 k} \leqslant (\sigma \cdot \| v \|) ^ {2 k}.
$$
We again define a goodness condition analogous to Assumption D.1. Specifically, we assume
Assumption E.2 (Higher-Order Goodness Condition). Let  $F\colon \mathbb{R}^d\to \mathbb{R}$ ,  $f = \nabla F$  be such that  $f$  is 1-Lipschitz. Let  $\mu^{*}\in \mathbb{R}^{d},0 < \sigma ,0 < \alpha < 1,k\in \mathbb{N}$  and  $y_{1},\ldots ,y_{n}\in \mathbb{R}^{d}$ . Let  $D$  be a distribution over  $\mathbb{R}^d$  and  $\pmb{\eta}_1^*,\dots,\pmb{\eta}_n^*$  be  $n$  i.i.d.samples from  $D$ . We say that the higher-order goodness condition holds, if
1.  $\alpha \| \hat{\mu}^w -\mu^*\| ^2\leqslant 10\langle \nabla \mathcal{L}^w (\mu^*),\mu^* -\hat{\mu}^w\rangle$
2.  $\left\| \frac{1}{n}\sum_{i = 1}^{n}f(\pmb{\eta}_i^*)\right\| \leqslant \frac{\alpha}{100}\sigma$
3. the uniform distribution over the set  $\{\pmb{\eta}_1^*,\dots,\pmb{\eta}_n^*\}$  has  $(2k,\ell)$ -certifiably  $\sigma$ -bounded  $f$ -moments. In other words,
$$
\left| \frac {v}{\ell} \right. \frac {1}{n} \sum_ {i = 1} ^ {n} \langle f (\boldsymbol {\eta} _ {i} ^ {*}), v \rangle^ {2 k} \leqslant (\sigma \cdot \| v \|) ^ {2 k}.
$$
Specifically, we will show
Theorem E.3. Let  $\mu^{*} \in \mathbb{R}^{d}, 0 < \sigma, 0 < \alpha < 1, k \in \mathbb{N}$ , and  $\pmb{\eta}_{1}^{*}, \dots, \pmb{\eta}_{1}^{*}$  be  $n$  i.i.d.samples from some distribution  $D$  over  $\mathbb{R}^d$ . Assume that  $\varepsilon^{1 - \frac{1}{2k}} \leqslant \frac{\alpha}{1000}$ . Let  $y_{1}, \ldots, y_{n}$  be an  $\varepsilon$ -corruption of  $\pmb{y}_{1}^{*} = \mu^{*} + \pmb{\eta}_{1}^{*}, \ldots, \pmb{y}_{n}^{*} = \mu^{*} + \pmb{\eta}_{n}^{*}$ . Assume Assumption E.2 (the higher-order goodness condition) holds, then Algorithm E.4, given  $y_{1}, \ldots, y_{n}$  and  $\sigma$ , terminates after at most  $2\varepsilon n + 1$  iterations and computes  $\hat{\mu}$  such that
$$
\| \hat {\mu} - \mu^ {*} \| \leqslant O \left(\frac {\left\| \frac {1}{n} \sum_ {i = 1} ^ {n} f \left(\boldsymbol {\eta} _ {i} ^ {*}\right) \right\|}{\alpha} + \frac {\varepsilon^ {1 - \frac {1}{2 k}}}{\alpha} \cdot \sigma\right).
$$
Moreover each iteration can be implemented in time  $n^{O(1)} \cdot d^{O(\ell)}$ .
Additionally to the notation introduced in Appendix D, consider the following polynomial in formal variables  $v$
$$
p ^ {w} (v) = \sum_ {i = 1} ^ {n} w _ {i} \langle v, f (y _ {i} - \hat {\mu}) \rangle^ {2 k}.
$$
In what follows we will write  $p^{w(t)}$  as just  $p^{(t)}$ .
Algorithm E.4 (Filtering Algorithm).
Let  $w^{(0)} = \frac{1}{n}$
- Compute  $\hat{\mu}^{(0)} = \min_{\mu \in \mathbb{R}^d}\mathcal{L}^{w^{(0)}}(\mu)$ .
- Let  $t = 0$ .
- while there is no degree- $\ell$  SoS proof (in variables  $v$ ) that  $p^{(t)}(v) \leqslant (100 \cdot \sigma \cdot \|v\|)^{2k}$  do
- Find a degree- $\ell$  pseudo-expectation that satisfies the constraint  $\| v\|^2 = 1$  such that  $\tilde{\mathbb{E}} p^t(v) > (99 \cdot \sigma)^{2k}$ .
- For  $i \in [n]$ , compute  $\tau_i^{(t)} = \tilde{\mathbb{E}}\left\langle v, f(\hat{\mu}^{(t)} - y_i)\right\rangle^{2k}$ .
- For  $i \in [n]$ , set  $w_i^{(t+1)} = \left( 1 - \frac{\tau_i}{\tau_{\max}} \right) w_i^{(t)}$ , where  $\tau_{\max} = \max_{i \in [n]} \tau_i$ .
- Compute  $\hat{\mu}^{(t + 1)} = \min_{\mu \in \mathbb{R}^d}f^{w^{(t + 1)}}(\mu)$
-  $t\gets t + 1$
The proof of Theorem E.3 is very similar to the one of Theorem D.2. Again, we will use the following lemma whose proof we will give at the end of this section
Lemma E.5. Assume that
$$
\sum_ {i \in S _ {g}} \frac {1}{n} - w _ {i} ^ {(t)} <   \sum_ {i \in S _ {b}} \frac {1}{n} - w _ {i} ^ {(t)}
$$
and  $p^t(v) \leqslant \left(100 \cdot \sigma \cdot \|v\|^2\right)^k$  does not have a degree- $\ell$  SoS proof. Then also
$$
\sum_ {i \in S _ {g}} \frac {1}{n} - w _ {i} ^ {(t + 1)} <   \sum_ {i \in S _ {b}} \frac {1}{n} - w _ {i} ^ {(t + 1)}.
$$
With this in hand, we will prove Theorem E.3.
Proof of Theorem E.3. First, note that by the facts in Appendix B we can compute the pseudoexpectation in each iteration in time  $n^{O(1)}d^{O(l)}$  and hence, every iteration can be implemented to run in this time. Exactly as in the proof of Theorem D.2 we can conclude that the algorithm terminates after  $T\leqslant 2\varepsilon n + 1$  iterations and that  $w^{T}\in \mathcal{W}_{2\varepsilon}$
$$
\hat {\mathfrak {m}} _ {2 k} ^ {w} = \max  _ {v \in \mathbb {S} ^ {d - 1}} \left[ \sum_ {i = 1} ^ {n} w _ {i} \left\langle f \left(\hat {\mu} - y _ {i}\right), v \right\rangle^ {2 k} \right] ^ {\frac {1}{2 k}} = \max  _ {v \in \mathbb {S} ^ {d - 1}} \left[ p ^ {w} (v) \right] ^ {\frac {1}{2 k}}.
$$
By Theorem C.1 it follows that
$$
\left\| \hat {\mu} ^ {(T)} - \mu^ {*} \right\| \leqslant O \left(\frac {\left\| \frac {1}{n} \sum_ {i = 1} ^ {n} f \left(\boldsymbol {\eta} _ {i} ^ {*}\right) \right\|}{\alpha} + \frac {\varepsilon^ {1 - 1 / (2 k)} \cdot \left(\hat {\mathfrak {m}} _ {2 k} ^ {w ^ {T}} + \bar {\mathfrak {m}} _ {2 k}\right)}{\alpha}\right).
$$
Note that since the uniform distribution over  $\{\pmb{\eta}_1^*,\dots,\pmb{\eta}_n^*\}$  has  $(2k,\ell)$ -certifiably  $\sigma$ -bounded  $f$ -moments, it holds that  $\bar{\mathfrak{m}}_{2k} \leqslant \sigma$ . Since the algorithm terminates after  $T$  iterations it holds that  $\hat{\mathfrak{m}}_{2k}^{w^T} \leqslant 100\sigma$  which completes the proof.
Next, we will prove Lemma E.5.
Proof of Lemma E.5. Again, for simplicity, let  $w = w^{(t)}$  and  $w' = w^{(t+1)}$ . As in the proof of Lemma D.4, it is enough to show that
$$
\sum_ {i = 1} ^ {n} w _ {i} \tau_ {i} = \sum_ {i = 1} ^ {n} w _ {i} \tilde {\mathbb {E}} \left\langle v, f \left(\hat {\mu} ^ {(t)} - y _ {i}\right) \right\rangle^ {k} = \tilde {\mathbb {E}} p ^ {t} (v) \geqslant (9 9 \cdot \sigma) ^ {2 k}.
$$
Using Fact K.1, we continue to bound
$$
\begin{array}{l} \sum_ {i \in S _ {g}} \tau_ {i} w _ {i} = \sum_ {i \in S _ {g}} w _ {i} \tilde {\mathbb {E}} \left<   v, f (\hat {\mu} ^ {(t)} - y _ {i}) \right> ^ {2 k} \\ \leqslant 4 ^ {k} \cdot \left[ \frac {1}{n} \sum_ {i \in [ n ]} \tilde {\mathbb {E}} \left\langle v, f \left(\boldsymbol {\eta} _ {i} ^ {*}\right) \right\rangle^ {2 k} + \frac {1}{| S _ {g} |} \sum_ {i \in S _ {g}} \tilde {\mathbb {E}} \left\langle v, f \left(\hat {\mu} ^ {(t)} - y _ {i}\right) - f \left(\mu^ {*} - y _ {i}\right) \right\rangle^ {2 k} \right]. \\ \end{array}
$$
The first term is at most  $\sigma^{2k}$ . For the second term, notice that by Fact K.2
$$
\left| \right. \frac {v}{2 k} \left\langle v, f \left(\hat {\mu} ^ {(t)} - y _ {i}\right) - f \left(\mu^ {*} - y _ {i}\right)\right\rangle^ {2 k} \leqslant \| v \| ^ {2 k} \left\| f \left(\hat {\mu} ^ {(t)} - y _ {i}\right) - f \left(\mu^ {*} - y _ {i}\right)\right\| ^ {2 k} \leqslant \| v \| ^ {2 k} \left\| \hat {\mu} ^ {(t)} - \mu^ {*} \right\| ^ {2 k}.
$$
Hence, the second term is at most  $\left\| \hat{\mu}^{(t)} - \mu^{*}\right\|^{2k}$ . By Theorem C.1 it follows that
$$
\left\| \hat {\mu} ^ {(T)} - \mu^ {*} \right\| \leqslant 1 0 0 \cdot \left(\frac {\left\| \frac {1}{n} \sum_ {i = 1} ^ {n} f \left(\boldsymbol {\eta} _ {i} ^ {*}\right) \right\|}{\alpha} + \frac {\varepsilon^ {1 - 1 / (2 k)} \cdot \left(\hat {\mathfrak {m}} _ {2 k} ^ {w ^ {t}} + \bar {\mathfrak {m}} _ {2 k}\right)}{\alpha}\right).
$$
Recall that  $\bar{\mathfrak{m}}_{2k} \leqslant \sigma$ . Since
$$
\hat {\mathfrak {m}} _ {2 k} ^ {w} = \max  _ {v \in \mathbb {S} ^ {d - 1}} \left[ \sum_ {i = 1} ^ {n} w _ {i} \left\langle f \left(\hat {\mu} - y _ {i}\right), v \right\rangle^ {2 k} \right] ^ {\frac {1}{2 k}} = \max  _ {v \in \mathbb {S} ^ {d - 1}} \left[ p ^ {w} (v) \right] ^ {\frac {1}{2 k}},
$$
$\varepsilon^{1 - \frac{1}{2k}} \leqslant \frac{\alpha}{1000}$  and  $\left\| \frac{1}{n} \sum_{i=1}^{n} f(\boldsymbol{\eta}_i^*) \right\| \leqslant \frac{\alpha}{100} \sigma$ , we get
$$
\sum_ {i \in S _ {g}} \tau_ {i} w _ {i} <   0. 2 \cdot \tilde {\mathbb {E}} p ^ {t} (v) + 0. 1 \cdot (9 9 \cdot \sigma) ^ {2 k} <   \tilde {\mathbb {E}} p ^ {t} (v) / 2.
$$
F Product Distributions
In this section we prove that  $(\alpha, \rho)$ -semi-product distributions satisfy Assumption D.1 with  $\sigma = O(\rho)$  and with  $F$  equals to the entry-wise Huber loss function with parameter  $2\rho$ :
$$
F (x) = \sum_ {j = 1} ^ {d} \Phi_ {2 \rho} \left(x _ {j}\right)
$$
with probability  $1 - \delta$  as long as  $n \gtrsim \frac{d + \log(1 / \delta)}{\alpha^4}$ . Moreover, if in addition  $n \gtrsim \frac{1}{\alpha^4} \cdot d^k \cdot \log(d / \delta)$ , then Assumption E.2 is also satisfied. We can use this to prove Theorem A.3 as follows: Let  $\pmb{\eta}_1^*$ , ...,  $\pmb{\eta}_n^*$  follow an  $(\alpha, \rho)$ -semi-product distribution. We will see in Lemma F.3 that for any  $n$ , it holds with probability at least  $1 - \delta$  that  $\left\| \frac{1}{n} \sum_{i=1}^{n} f(\pmb{\eta}_i^*) \right\| \leqslant \rho \cdot \sqrt{\frac{d + \log(1 / \delta)}{n}}$ . Hence, by Theorem E.3 (or Algorithm D.3 for  $k = 1$ ) it follows that as long as  $n \gtrsim \frac{1}{\alpha^4} \cdot d^k \log(d / \delta)$  (or  $n \gtrsim \frac{d + \log(1 / \delta)}{\alpha^4}$  for  $k = 1$ ) our estimator achieves error
$$
O \left(\frac {\left\| \frac {1}{n} \sum_ {i = 1} ^ {n} f \left(\boldsymbol {\eta} _ {i} ^ {*}\right) \right\|}{\alpha} + \frac {\varepsilon^ {1 - \frac {1}{2 k}}}{\alpha} \cdot \sigma\right) = O \left(\rho \cdot \left[ \sqrt {\frac {d + \log (1 / \delta)}{\alpha^ {4} n}} + \frac {\varepsilon^ {1 - \frac {1}{2 k}}}{\alpha} \right]\right).
$$
Letting  $k = O(\log(1/\varepsilon))$  proves Theorem A.3. Note that similarly to Theorem 1.6, for  $1 \leqslant k \leqslant O(\log(1/\varepsilon))$  we get a sequence of algorithms interpolating between error  $\sqrt{\varepsilon}$  and  $\varepsilon$  (using roughly  $d^k$  samples).
Note that  $f = \nabla F$  is indeed 1-Lipschitz since the derivative of the Huber loss is Lipschitz. In the next to lemmas we prove the first assumption from Assumption D.1.
Lemma F.1. Let  $w \in \mathcal{W}_{2\varepsilon}$  for  $\varepsilon \lesssim \alpha$ . Suppose that  $n \gtrsim \log(d)/\alpha$ . Then with probability  $1 - \exp(-\Omega(\alpha n))$  for all  $u \in \mathbb{R}^d$  such that  $\|u\|_{\max} \leqslant \rho$ ,
$$
\mathcal {L} ^ {w} \left(\mu^ {*} + u\right) - \mathcal {L} ^ {w} \left(\mu^ {*}\right) - \left\langle \nabla \mathcal {L} ^ {w} \left(\mu^ {*}\right), u \right\rangle \geqslant \frac {\alpha}{4} \cdot \| u \| ^ {2}.
$$
Proof. Let  $j \in [d]$  and  $\zeta_i = \mu - y_i$ .
$$
\mathcal {L} _ {j} ^ {w} \left(\mu_ {j} ^ {*} + u _ {j}\right) - \mathcal {L} _ {j} ^ {w} \left(\mu_ {j} ^ {*}\right) - \left(\mathcal {L} _ {j} ^ {w}\right) ^ {\prime} \left(\mu_ {j} ^ {*}\right) \cdot u _ {j} = \sum_ {i = 1} ^ {n} w _ {i} \left[ \Phi \left(\zeta_ {i j} + u _ {j}\right) - \Phi \left(\zeta_ {i j}\right) - u _ {j} \cdot \phi \left(\zeta_ {i j}\right) \right].
$$
Let  $M$  be the set of uncorrupted samples such that  $|\zeta_{ij}| = |\pmb{\eta}_{ij}|\leqslant h / 2$ . Note that for  $i\in M$ ,  $\Phi (\zeta_{ij} + u_j) = \frac{1}{2} |\zeta_{ij} + u_j|^2$  and  $\Phi (\zeta_{ij} + u_j) = \frac{1}{2} |\zeta_{ij}|^2$ . Hence
$$
\sum_ {i \in M} w _ {i} \left[ \Phi \left(\zeta_ {i j} + u _ {j}\right) - \Phi \left(\zeta_ {i j}\right) - u _ {j} \cdot \phi \left(\zeta_ {i j}\right) \right] = \frac {1}{2} \sum_ {i \in M} w _ {i} u _ {j} ^ {2}.
$$
And by convexity,
$$
\sum_ {i \notin M} w _ {i} \left[ \Phi \left(\| \zeta_ {i} + u \|\right) - \Phi \left(\| \zeta_ {i} \|\right) - \langle f \left(\zeta_ {i}\right), u \rangle \right] \geqslant 0.
$$
Since  $w\in \mathcal{W}_{2\varepsilon}$
$$
\sum_ {i \in M} \left| w _ {i} - \frac {1}{n} \right| \leqslant 2 \varepsilon .
$$
$$
\sum_ {i \in M} w _ {i} \geqslant \frac {| M |}{n} - 2 \varepsilon .
$$
By a Chernoff bound,
$$
\left| M \right| \geqslant 0. 9 \alpha - \varepsilon \geqslant \alpha / 2
$$
with probability  $1 - \exp(-\alpha n)$ . The result follows from a union bound over all  $j \in [d]$ .
Lemma F.2. Let  $w \in \mathcal{W}_{2\varepsilon}$  for  $\varepsilon \lesssim \alpha$ , and suppose that  $n \gtrsim \log(d)/\alpha^2$ . Then with probability  $1 - \exp(-\Omega(\alpha^2 n))$ ,
$$
\langle \nabla \mathcal {L} ^ {w} (\mu^ {*}), \mu^ {*} - \hat {\mu} ^ {w} \rangle \geqslant \frac {\alpha}{4} \| \hat {\mu} ^ {w} - \mu^ {*} \| ^ {2}.
$$
Proof. Fix  $j \in d$  and let  $u_{j} = \mu_{j}^{*} - \hat{\mu}_{j}(w)$ . Let  $u_{j}'$  and  $t$  be such that
$$
\left| u _ {j} ^ {\prime} \right| \leqslant \rho ,
$$
$$
\hat {\mu} _ {j} (w) = \mu_ {j} ^ {*} + t \cdot u _ {j} ^ {\prime}.
$$
It follows that  $t = \max \{1, |u_j|\}$ . Clearly,  $\hat{\mu}_j(w)$  is the (unique) minimizer of  $\mathcal{L}_j^w$ . Let  $\hat{\mu}_j' = \mu_j^* + u_j'$ , by convexity of  $\mathcal{L}_j^w$  it holds that  $\mathcal{L}_j^w(\hat{\mu}_j') \leqslant \mathcal{L}_j^w(\mu_j^*)$ . Since  $|u_j'| \leqslant \rho$  it follows by Lemma F.1 that
$$
\mathcal {L} _ {j} ^ {w} \left(\mu_ {j} ^ {*}\right) \geqslant \mathcal {L} _ {j} ^ {w} \left(\hat {\mu} _ {j} ^ {\prime}\right) \geqslant \mathcal {L} _ {j} ^ {w} \left(\mu_ {j} ^ {*}\right) + \left(\mathcal {L} _ {j} ^ {w}\right) ^ {\prime} \left(\mu_ {j} ^ {*}\right) \cdot \left(- u _ {j} ^ {\prime}\right) + \frac {\alpha}{4} \left(u _ {j} ^ {\prime}\right) ^ {2}
$$