SlowGuess/NeurIPS_2023_OCR huggingface dataset:
with probability at least  $1 - \exp (-\Omega (\alpha n))$ . Rearranging yields
$$
\left| u _ {j} ^ {\prime} \right| \leqslant \frac {4}{\alpha} \cdot \left(\mathcal {L} _ {j} ^ {w}\right) ^ {\prime} \left(\mu_ {j} ^ {*}\right).
$$
We next examine
$$
\begin{array}{l} \left(\mathcal {L} _ {j} ^ {w}\right) ^ {\prime} \left(\mu_ {j} ^ {*}\right) = \sum_ {i = 1} ^ {n} w _ {i} \phi \left(\left(y _ {i}\right) _ {j} - \mu_ {j} ^ {*}\right) \\ = \sum_ {i = 1} ^ {n} w _ {i} \phi (\boldsymbol {\eta} _ {i j}) + \sum_ {i \in S _ {b}} ^ {n} w _ {i} \left[ \phi ((y _ {i}) _ {j} - \mu_ {j} ^ {*}) - \phi ((\boldsymbol {y} _ {i} ^ {*}) _ {j} - \mu_ {j} ^ {*}) \right]. \\ \end{array}
$$
ByHoeffding'sinequality,
$$
\left| \sum_ {i = 1} ^ {n} w _ {i} \phi (\boldsymbol {\eta} _ {i j}) \right| \leqslant \left| \frac {1}{n} \sum_ {i = 1} ^ {n} \phi (\boldsymbol {\eta} _ {i j}) \right| + 2 \rho \varepsilon \leqslant 2 \rho \tau / \sqrt {n} + 4 \rho \varepsilon
$$
with probability at least  $1 - \exp (-\tau^2 /2)$
The second term can be bounded as follows:
$$
\sum_ {i \in S _ {b}} w _ {i} \cdot \left[ \phi \left(\mu_ {j} ^ {*} - \left(y _ {i}\right) _ {j}\right) - \phi \left(\mu_ {j} ^ {*} - \left(\boldsymbol {y} _ {i} ^ {*}\right) _ {j}\right) \right] \leqslant 4 \rho \varepsilon ,
$$
where we used that for all  $x, y \in \mathbb{R}$  it holds that  $|\phi(x) - \phi(y)| \leqslant 2\rho$  and  $\sum_{i \in S_b} w_i \leqslant 2\varepsilon$ . Putting everything together we obtain
$$
| u _ {j} ^ {\prime} | \leqslant \rho \frac {2 \tau}{\alpha \sqrt {n}} + \frac {4 \rho \varepsilon}{\alpha}
$$
with probability  $1 - \exp (-\tau^2 /2)$ . For  $\tau = \alpha \sqrt{n} /10$ ,  $|u_j^{\prime}| < \rho$ . Hence,  $t = 1$  since otherwise
$$
\left| u _ {j} \right| = \left| \hat {\mu} _ {j} (w) - \mu_ {j} ^ {*} \right| = t \cdot \left| u _ {j} ^ {\prime} \right| <   \left| u _ {j} \right|.
$$
It follows that  $u_{j}^{\prime} = u_{j}$  and thus  $|\hat{\mu}_j(w) - \mu_j^*| < \rho$ .
By Lemma F.1
$$
\mathcal {L} ^ {w} \left(\mu^ {*}\right) \geqslant \mathcal {L} ^ {w} \left(\hat {\mu}\right) \geqslant \mathcal {L} ^ {w} \left(\mu^ {*}\right) + \left\langle \nabla \mathcal {L} ^ {w} \left(\mu^ {*}\right), - u \right\rangle + \frac {\alpha}{4} \left\| u \right\| ^ {2},
$$
$$
\left\langle \nabla \mathcal {L} ^ {w} \left(\mu^ {*}\right), u \right\rangle \geqslant \frac {\alpha}{4} \| u \| ^ {2}.
$$
In the following lemma we show that Assumption 2 of Assumption D.1 is satisfied for  $\sigma = O(\rho)$  with probability  $1 - \delta$  as long as  $n \gtrsim \frac{d + \log(1 / \delta)}{\alpha^2}$ .
Lemma F.3. With probability at least  $1 - \delta$
$$
\left\| \frac {1}{n} \sum_ {i = 1} ^ {n} f (\boldsymbol {\eta} _ {i}) \right\| \leqslant 1 0 \rho \sqrt {\frac {d + \log (1 / \delta)}{n}}.
$$
Proof. Let  $u \in \mathbb{R}^d$  be some fixed vector. Since  $\operatorname{sign}(\pmb{\eta})$  is independent of  $\operatorname{abs}(\pmb{\eta})$ , random variables  $\tilde{\pmb{u}}_{ij} = u_j \cdot |\pmb{\eta}_{ij}|$  are independent of  $\operatorname{sign}(\pmb{\eta})$ . By Hoeffding's inequality
$$
\left| \left\langle \frac {1}{n} \sum_ {i = 1} ^ {n} f (\boldsymbol {\eta} _ {i}), u \right\rangle \right| = \left| \frac {1}{n} \sum_ {i = 1} ^ {n} \sum_ {j = 1} ^ {d} \operatorname {s i g n} (\boldsymbol {\eta} _ {i j}) \tilde {\boldsymbol {u}} _ {i j} \right| \leqslant 2 \rho \cdot \| u \| \cdot \tau / \sqrt {n}
$$
with probability at least  $1 - \exp(-\tau^2 / 2)$ . Let  $\mathcal{N}_{1/2}$  be a  $1/2$ -net of size  $6^d$  in the Euclidean unit  $d$ -dimensional ball. Then
$$
\sup  _ {\| u \| \leqslant 1} \left| \left\langle \frac {1}{n} \sum_ {i = 1} ^ {n} f \left(\boldsymbol {\eta} _ {i}\right), u \right\rangle \right| \leqslant \sup  _ {\| u \| \leqslant 1 / 2} \left| \left\langle \frac {1}{n} \sum_ {i = 1} ^ {n} f \left(\boldsymbol {\eta} _ {i}\right), u \right\rangle \right| + \sup  _ {u \in \mathcal {N} _ {1 / 2} 1 / 2} \left| \left\langle \frac {1}{n} \sum_ {i = 1} ^ {n} f \left(\boldsymbol {\eta} _ {i}\right), u \right\rangle \right|.
$$
Hence by union bound,
$$
\left\| \frac {1}{n} \sum_ {i = 1} ^ {n} f \left(\boldsymbol {\eta} _ {i}\right) \right\| = \sup  _ {\| u \| \leqslant 1} \left| \left\langle \frac {1}{n} \sum_ {i = 1} ^ {n} f \left(\boldsymbol {\eta} _ {i}\right), u \right\rangle \right| \leqslant 2 \sup  _ {u \in \mathcal {N} _ {1 / 2}} \left| \left\langle \frac {1}{n} \sum_ {i = 1} ^ {n} f \left(\boldsymbol {\eta} _ {i}\right), u \right\rangle \right| \leqslant 4 \rho \sqrt {\frac {2 \ln (6) \cdot d + \log (1 / \delta)}{n}}.
$$
In the following lemma we show that Assumption 3 of Assumption D.1 is satisfied for  $\sigma = O(\rho)$  with probability  $1 - \delta$  as long as  $n \gtrsim d + \log(1 / \delta)$ .
Lemma F.4. Let  $\delta \in (0,1)$  and suppose that  $n\geqslant d + \log (1 / \delta)$ . Then
$$
\left\| \frac {1}{n} \sum_ {i = 1} ^ {n} f (\boldsymbol {\eta} _ {i}) (f (\boldsymbol {\eta} _ {i})) ^ {\top} \right\| \leqslant O (\rho^ {2})
$$
Proof. Let  $u \in \mathbb{R}^d$  be a fixed vector. Since the random vector sign  $(\eta)$  is independent of  $\mathrm{abs}(\eta)$ , the random vector  $\mathrm{abs}(f(\eta)) \circ u$  with entries  $u_j \cdot |\eta(j)|$  is independent of  $\mathrm{sign}(\eta)$ . Since the entries of  $\mathrm{sign}(\eta)$  are iid Rademacher, for every positive integer  $k$ ,
$$
\begin{array}{l} \sup  _ {\| u \| \leqslant 1} \mathbb {E} \left\langle f (\boldsymbol {\eta}), u \right\rangle^ {2 k} \leqslant \sup  _ {\| u \| \leqslant 1} \mathbb {E} \left\langle \operatorname {s i g n} (\boldsymbol {\eta}), \operatorname {a b s} (f (\boldsymbol {\eta})) \circ u \right\rangle^ {2 k} \\ \leqslant \sup  _ {\| u \| \leqslant 2 \rho} \mathbb {E} \left\langle \operatorname {s i g n} (\boldsymbol {\eta}), u \right\rangle^ {2 k} \\ \leqslant (2 \rho) ^ {2 k} \sup  _ {\| u \| \leqslant 1} \mathbb {E} \left\langle \operatorname {s i g n} (\boldsymbol {\eta}), u \right\rangle^ {2 k} \\ \leqslant (2 \rho) ^ {2 k}. \\ \end{array}
$$
Hence  $f(\eta)$  is  $2\rho$ -subgaussian and the result follows from concentration of empirical covariance for subgaussian distributions (see Theorem 6.5 in [Wai19]).
In the following lemma we show that Assumption 3 of Assumption E.2 is satisfied for  $\sigma = O(\rho)$  with probability  $1 - \delta$  as long as  $n \gtrsim d^k \cdot \log (d / \delta)$ .
Lemma F.5. Let  $k$  be a positive integer and suppose that  $n \geqslant 10 \cdot d^{k} \cdot \log (d / \delta)$ . Then with probability at least  $1 - \delta$ , uniform distribution over the set  $\{\pmb{\eta}_1, \dots, \pmb{\eta}_n\}$  has  $(2k, 2k)$ -certifiable  $4\rho$ -bounded  $f$ -moments.
Proof. First we show that the random vector  $\pmb{\eta}$  has  $(2k, 2k)$ -certifiable  $2\rho$ -bounded  $f$ -moments. Let's call a multi-index  $\beta$  even if every element in  $\beta$  appears even number of times. Note that  $\beta$  is even iff  $\beta = 2\beta'$  for a multi-index  $\beta'$  of size  $k$ . Denote by  $\mathcal{E}$  the set of all even multi-indices of size  $2k$ . Let  $v_1, \ldots, v_d$  be variables, and consider the polynomial  $\mathbb{E}\langle f(\pmb{\eta}), v \rangle^{2k}$ . It follows that
$$
\begin{array}{l} \left| \frac {v}{2 k} \mathbb {E} \langle f (\boldsymbol {\eta}), v \rangle^ {2 k} = \mathbb {E} \mathbb {E} \left[ \langle \operatorname {s i g n} (\boldsymbol {\eta}), \operatorname {a b s} (f (\boldsymbol {\eta})) \circ v \rangle^ {2 k} \mid \operatorname {a b s} (f (\boldsymbol {\eta})) \right] \right. \\ = \mathbb {E} \sum_ {\beta \in \mathcal {E}} \mathbb {E} \left[ \operatorname {s i g n} (\boldsymbol {\eta}) ^ {\beta} \left(\operatorname {a b s} (f (\boldsymbol {\eta})) \circ v\right) ^ {\beta} \mid \operatorname {a b s} (f (\boldsymbol {\eta})) \right] \\ = \mathbb {E} \sum_ {\beta \in \mathcal {E}} (\operatorname {a b s} (f (\boldsymbol {\eta})) \circ v) ^ {\beta} \\ \leqslant (2 \rho) ^ {2 k} \sum_ {\beta \in \mathcal {E}} v ^ {\beta} \\ = (2 \rho) ^ {2 k} \sum_ {| \beta^ {\prime} | = k} v ^ {2 \beta^ {\prime}} \\ = (2 \rho) ^ {2 k} \| v \| ^ {2 k}. \\ \end{array}
$$
Hence  $\eta$  has  $(2k, 2k)$ -certifiable  $2\rho$ -bounded  $f$ -moments. Since  $\forall x \in \mathbb{R}^d$ ,  $\|f(x)\| \leqslant 2\rho \sqrt{d}$ , the result follows from Lemma J.4.
G Elliptical Distributions
In this section we use a different normalization than the one described in the introduction. Concretely, we assume that  $\mathrm{Tr}(\Sigma) = d$  and that for some  $\rho > 0$  and  $\alpha \in (0,1)$ ,  $\mathbb{P}\left[\boldsymbol{R} \leqslant \rho \sqrt{d}\right] \geqslant \alpha$ . For  $\alpha = 1/2$ , this is the same model as in the Introduction, since we can multiply  $\Sigma^{1/2}$  by  $s = \sqrt{d / \operatorname{Tr}(\Sigma)}$  and divide  $\boldsymbol{R}$  by  $s$ . Hence the new parameter  $\rho$  written in terms of the parameters defined in the Introduction is equal to  $\sqrt{2}/s = \sqrt{2 \operatorname{Tr}(\Sigma)/d}$ . Note that  $\rho \| \Sigma\|$  in new notation is equal to  $\sqrt{2} \| \Sigma\|$  in the notation used in the introduction. Note that the effective rank,  $\mathrm{r}(\Sigma) = \frac{\operatorname{Tr} \Sigma}{\|\Sigma\|}$ , is the same in both parametrizations.
We assume that  $\rho$  is known. In the case of unknown  $\rho$  and  $\alpha \geqslant \Omega(1)$ , we can estimate  $\rho$  using the corrupted samples and obtain the same guarantees as for known  $\rho$  (up to a constant factor). We describe how to achieve this in Appendix G.1.
We prove that elliptical distributions satisfy Assumption D.1 and with,  $\sigma = O(\rho \sqrt{\|\Sigma\|})$  and  $F: \mathbb{R}^d \to \mathbb{R}$  defined as
$$
F (x) = \Phi_ {2 0 \rho \sqrt {d}} \left(\| x \|\right),
$$
with probability  $1 - \delta$  as long as  $n \gtrsim \frac{\mathrm{r}(\Sigma) \log(d / \delta)}{\alpha^4}$ . Moreover, if in addition  $n \gtrsim \frac{1}{\alpha^4} \left( \frac{\mathrm{r}(\Sigma)}{k} \right)^k \cdot \log(d / \delta)$ , then Assumption E.2 is also satisfied with  $\sigma = O(\rho \sqrt{k \| \Sigma \|})$ .
Going back to the parametrization of the introduction and letting  $\alpha = \frac{1}{2}$ , this means the following: For  $n \gtrsim \frac{\mathrm{r}(\Sigma) \log(d / \delta)}{\alpha^4}$ , elliptical distributions satisfy Assumption D.1 with  $\sigma = O(\sqrt{\|\Sigma\|})$ . Further, if  $n \gtrsim \frac{1}{\alpha^4} (\frac{\mathrm{r}(\Sigma)}{k})^k \cdot \log(d / \delta)$ , they satisfy Assumption E.2 with  $\sigma = O(\sqrt{k\|\Sigma\|})$ . Further, for  $\pmb{\eta}_1^*$ , ...,  $\pmb{\eta}_n^*$  following an elliptical distribution with the above parameters, we will in Lemma G.4 show that for  $n \geqslant \log(1 / \delta)$  it holds with probability at least  $1 - \delta$  that  $\| \frac{1}{n} \sum_{i=1}^{n} f(\pmb{\eta}_i^*) \| \leqslant O(\sqrt{\frac{\operatorname{Tr}(\Sigma) \cdot \log(d / \delta)}{n}})$ .
Hence, by Theorem E.3 (or Theorem D.2 for  $k = 1$ ) our estimator achieves error
$$
O \left(\left\| \frac {1}{n} \sum_ {i = 1} ^ {n} f \left(\boldsymbol {\eta} _ {i} ^ {*}\right) \right\| + \varepsilon^ {1 - \frac {1}{2 k}} \cdot \sigma\right) = O \left(\sqrt {\| \Sigma \|} \cdot \left[ \sqrt {\frac {\operatorname {r} (\Sigma) \log (d / \delta)}{n}} + \sqrt {k} \varepsilon^ {1 - \frac {1}{2 k}} \right]\right).
$$
This gives the proof of Theorem 1.6.
Note that  $f = \nabla F$  sends  $x$  to  $\phi_{20\rho \sqrt{d}}(\|x\|) \cdot \frac{x}{\|\boldsymbol{x}\|}$  and hence is indeed 1-Lipschitz since it is the projection of  $x$  onto the  $\ell_2$ -ball of radius  $20\rho \sqrt{d}$ . In the next three lemmas we prove the first assumption from Assumption D.1.
Lemma G.1. Suppose that  $d \geqslant 100$ . Then
$$
\mathbb {P} \left[ \left\| \boldsymbol {R} \Sigma^ {1 / 2} \boldsymbol {u} \right\| \leqslant h / 2 \right] \geqslant 0. 8 \alpha ,
$$
where  $h = 20\rho \sqrt{d}$
Proof. Let  $\pmb{g} \sim N(0, 4\rho^2\Sigma)$  be a  $d$ -dimensional Gaussian vector such that  $\pmb{g} = \pmb{R}'\Sigma^{1/2}\pmb{u}$ , where  $\pmb{R}'$  is independent of  $\pmb{R}$  and has distribution of the norm of Gaussian vector  $N(0, 2\rho^2\mathrm{Id})$ . Note that  $\operatorname{Tr}\left(\mathbb{E}\pmb{gg}^\top\right) = 4\rho^2d$ . Applying Fact J.6 with  $t = 2$ , we get
$$
\mathbb {P} \left[ \| \boldsymbol {g} \| \geqslant 1 0 \rho \sqrt {d} \right] \leqslant \exp (- 2) \leqslant 0. 1 5.
$$
$$
\mathbb {P} \left[ \boldsymbol {R} ^ {\prime} \geqslant \rho \sqrt {d} \right] \geqslant 0. 9 9,
$$
$$
\mathbb {P} \left[ \| \boldsymbol {g} \| \leqslant 1 0 \rho \sqrt {d}, \boldsymbol {R} ^ {\prime} \geqslant \rho \sqrt {d} \right] \geqslant 0. 8.
$$
Since  $\pmb{R}$  is independent of  $\pmb{g}$  and  $\pmb{R}^{\prime}$ , we get
$$
\begin{array}{l} \mathbb {P} \left[ \left\| \boldsymbol {R} \Sigma^ {1 / 2} \boldsymbol {u} \right\| \leqslant 1 0 \sqrt {d} \right] \geqslant \mathbb {P} \left[ \left\| \boldsymbol {R} \Sigma^ {1 / 2} \boldsymbol {u} \right\| \leqslant \| \boldsymbol {g} \|, \| \boldsymbol {g} \| \leqslant 1 0 \rho \sqrt {d} \right] \\ = \mathbb {P} \left[ \boldsymbol {R} \leqslant \boldsymbol {R} ^ {\prime}, \| \boldsymbol {g} \| \leqslant 1 0 \rho \sqrt {d} \right] \\ \geqslant \mathbb {P} \left[ \boldsymbol {R} \leqslant \rho \sqrt {d}, \boldsymbol {R} ^ {\prime} \geqslant \rho \sqrt {d}, \| \boldsymbol {g} \| \leqslant 1 0 \rho \sqrt {d} \right] \\ \geqslant 0. 8 \alpha . \\ \end{array}
$$
Lemma G.2. Let  $w \in \mathcal{W}_{2\varepsilon}$  for  $\varepsilon \lesssim \alpha$ . Then with probability  $1 - \exp(-\Omega(\alpha n))$  for all  $u \in \mathbb{R}^d$  such that  $\|u\| \leqslant \rho \sqrt{d}$ ,
Proof. Let  $\zeta_{i} = \mu -y_{i}$
$$
\mathcal {L} ^ {w} (\mu^ {*} + u) - \mathcal {L} ^ {w} (\mu^ {*}) - \langle \mathcal {L} ^ {w} (\mu^ {*}), u \rangle = \sum_ {i = 1} ^ {n} w _ {i} [ f (\| \zeta_ {i} + u \|) - f (\| \zeta_ {i} \|) - \langle f (\zeta_ {i}), u \rangle ]
$$
Let  $M$  be the set of uncorrupted samples such that  $\| \zeta_i\| = \| \pmb {\eta}_i\| \leqslant h / 2$ . Note that for  $i\in M$ ,  $f(\| \zeta_i + u\|) = \frac{1}{2}\| \zeta_i + u\|^2$  and  $f(\| \zeta_i\|) = \frac{1}{2}\| \zeta_i\|^2$ . Hence
$$
\sum_ {i \in M} w _ {i} \left[ f \left(\| \zeta_ {i} + u \|\right) - f \left(\| \zeta_ {i} \|\right) - \langle f (\zeta_ {i}), u \rangle \right] = \frac {1}{2} \sum_ {i \in M} w _ {i} \| u \| ^ {2}.
$$
$$
\sum_ {i \notin M} w _ {i} \left[ f \left(\| \zeta_ {i} + u \|\right) - f \left(\| \zeta_ {i} \|\right) - \langle f \left(\zeta_ {i}\right), u \rangle \right] \geqslant 0.
$$
By Lemma G.1 and a Chernoff bound,
$$
\left| M \right| \geqslant 0. 6 \alpha - \varepsilon \geqslant \alpha / 2
$$
with probability  $1 - \exp (-\alpha n)$
Lemma G.3. Let  $w \in \mathcal{W}_{2\varepsilon}$  for  $\varepsilon \lesssim \alpha$ , and suppose that  $n \gtrsim \log(d)/\alpha^2$ . Then with probability  $1 - \exp(-\Omega(\alpha^2 n))$ ,
$$
\left\langle \nabla \mathcal {L} ^ {w} \left(\mu^ {*}\right), \mu^ {*} - \hat {\mu} ^ {w} \right\rangle \geqslant \frac {\alpha}{4} \left\| \hat {\mu} ^ {w} - \mu^ {*} \right\| ^ {2}.
$$
Proof. Let  $u = \mu^{*} - \hat{\mu}(w)$ . Let  $u'$  and  $t$  be such that
$$
\| u ^ {\prime} \| \leqslant \rho \sqrt {d},
$$
$$
\hat {\mu} (w) = \mu^ {*} + t \cdot u ^ {\prime}.
$$
It follows that  $t = \max \{1, \| u\| \}$ . Clearly,  $\hat{\mu}(w)$  is the (unique) minimizer of  $\mathcal{L}^w$ . Let  $\hat{\mu}' = \mu^* + u'$ , by convexity of  $\mathcal{L}^w$  it holds that  $\mathcal{L}^w(\hat{\mu}') \leqslant \mathcal{L}^w(\mu^*)$ . Since  $\| u' \| \leqslant \rho \sqrt{d}$  it follows by Lemma G.2 that
$$
\mathcal {L} ^ {w} \left(\mu^ {*}\right) \geqslant \mathcal {L} ^ {w} \left(\hat {\mu} ^ {\prime}\right) \geqslant \mathcal {L} ^ {w} \left(\mu^ {*}\right) - \left\langle \nabla \mathcal {L} ^ {w} \left(\mu^ {*}\right), u \right\rangle + \frac {\alpha}{4} \| u ^ {\prime} \| ^ {2}
$$
$$
\left\| u ^ {\prime} \right\| \leqslant \frac {4}{\alpha} \cdot \left\| \nabla \mathcal {L} ^ {w} \left(\mu^ {*}\right) \right\|.
$$
$$
\begin{array}{l} \nabla \mathcal {L} ^ {w} (\mu^ {*}) = \sum_ {i = 1} ^ {n} w _ {i} f ((y _ {i}) - \mu^ {*}) \\ = \sum_ {i = 1} ^ {n} w _ {i} f (\boldsymbol {\eta} _ {i}) + \sum_ {i \in S _ {b}} ^ {n} w _ {i} \left[ f \left(\left(y _ {i}\right) - \mu^ {*}\right) - f \left(\left(\boldsymbol {y} _ {i} ^ {*}\right) - \mu^ {*}\right) \right]. \\ \end{array}
$$
By vector Bernstein inequality Fact J.2,
$$
\left\| \sum_ {i = 1} ^ {n} w _ {i} f (\boldsymbol {\eta} _ {i}) \right\| \leqslant \left\| \frac {1}{n} \sum_ {i = 1} ^ {n} f (\boldsymbol {\eta} _ {i}) \right\| + 4 0 \varepsilon \rho \sqrt {d} \leqslant 2 0 0 \rho \sqrt {d} \cdot \left(\tau / \sqrt {n} + \tau^ {2} / n\right) + 4 0 \varepsilon \rho \sqrt {d}
$$
$$
\sum_ {i \in S _ {b}} w _ {i} \cdot \left[ f \left(\mu^ {*} - \left(y _ {i}\right)\right) - f \left(\mu^ {*} - \left(\boldsymbol {y} _ {i} ^ {*}\right)\right) \right] \leqslant 4 0 \varepsilon \rho \sqrt {d},
$$
where we used that for all  $x, y \in \mathbb{R}$  it holds that  $|f(x) - f(y)| \leqslant 20\rho \sqrt{d}$  and  $\sum_{i \in S_b} w_i \leqslant 2\varepsilon$ . Putting everything together we obtain
$$
\| u ^ {\prime} \| \leqslant 2 0 0 \frac {\rho \sqrt {d}}{\alpha} \cdot (\tau / \sqrt {n} + \tau^ {2} / n) + \frac {4 0 \varepsilon \rho \sqrt {d}}{\alpha}
$$
with probability  $1 - \exp (-\tau^2 /2)$ . For  $\tau = \alpha \sqrt{n} /1000, |u'| < \rho$ . Hence,  $t = 1$  since otherwise
$$
\| u \| = \left\| \hat {\mu} (w) - \mu^ {*} \right\| = t \cdot \left\| u ^ {\prime} \right\| <   \| u \|.
$$
It follows that  $u^{\prime} = u$  and thus  $\| \hat{\mu} (w) - \mu^{*}\| <  \rho \sqrt{d}$
By Lemma G.2
$$
\mathcal {L} ^ {w} \left(\mu^ {*}\right) \geqslant \mathcal {L} ^ {w} \left(\hat {\mu}\right) \geqslant \mathcal {L} ^ {w} \left(\mu^ {*}\right) + \left\langle \nabla \mathcal {L} ^ {w} \left(\mu^ {*}\right), - u \right\rangle + \frac {\alpha}{4} \| u \| ^ {2},
$$
In the following lemma we show that Assumption 2 of Assumption D.1 is satisfied for  $\sigma = O(\rho \| \Sigma \|)$  with probability  $1 - \delta$  as long as  $n \gtrsim \frac{d\log(d / \delta)}{\alpha^2}$ .
Lemma G.4. Let  $\delta \in (0,1)$  and suppose that  $n\geqslant \log (1 / \delta)$ . Then with probability at least  $1 - \delta$
$$
\left\| \frac {1}{n} \sum_ {i = 1} ^ {n} f (\boldsymbol {\eta} _ {i}) \right\| \leqslant O \left(\rho \sqrt {\frac {d \log (d / \delta)}{n}}\right).
$$
Proof. The lemma is a direct consequence of vector Bernstein inequality Fact J.2.  $\square$
In the following lemma we show that Assumption 3 of Assumption D.1 is satisfied for  $\sigma = O(\rho)$  with probability  $1 - \delta$  as long as  $n \gtrsim d\log(d / \delta)$ .
Lemma G.5. Let  $\delta \in (0,1)$  and suppose that  $n\geqslant \mathrm{r}(\Sigma)\log (d / \delta)$ . Then
$$
\left\| \frac {1}{n} \sum_ {i = 1} ^ {n} f (\boldsymbol {\eta} _ {i}) (f (\boldsymbol {\eta} _ {i})) ^ {\top} \right\| \leqslant O (\rho^ {2} \| \Sigma \|)
$$
Proof. Let  $s(\pmb{\eta}) = \frac{h}{\|\pmb{\eta}\|} \pmb{\eta} = \frac{h}{\|f(\pmb{\eta})\|} f(\pmb{\eta})$ , where  $h = 20\rho \sqrt{d}$ . Note that  $\|s(\pmb{\eta})\| \geqslant \|f(\pmb{\eta})\|$ . By Fact J.5 and since by our parametrization  $\operatorname{Tr} \Sigma = d$ , it follows that
$$
\sup  _ {\| u \| \leqslant 1} \mathbb {E} \left\langle f (\boldsymbol {\eta}), u \right\rangle^ {2} \leqslant \sup  _ {\| u \| \leqslant 1} \mathbb {E} \left\langle s (\boldsymbol {\eta}), u \right\rangle^ {2} \leqslant (2 0 \rho) ^ {2} \| \Sigma \| .
$$
Let  $t = \rho^2\| \Sigma \| \cdot \sqrt{\frac{\mathrm{r}(\Sigma)\log(d / \delta)}{n}}\leqslant \rho^2\| \Sigma \|$ . It follows by Fact J.3 that
$$
\left\| \frac {1}{n} \sum_ {i = 1} ^ {n} f (\boldsymbol {\eta} _ {i}) (f (\boldsymbol {\eta} _ {i})) ^ {\top} \right\| \leqslant O (\rho^ {2} \| \Sigma \| + t) \leqslant O (\rho^ {2} \| \Sigma \|),
$$
with probability at least  $1 - \delta$ , where we also used that  $t = \rho^2\|\Sigma\| \cdot \sqrt{\frac{\mathrm{r}(\Sigma)\log(d / \delta)}{n}} \leqslant \rho^2\|\Sigma\|$  when applying Fact J.3.
In the following lemma we show that Assumption 3 of Assumption E.2 is satisfied for  $\sigma = O(\rho \| \Sigma \|)$  with probability  $1 - \delta$  as long as  $n \gtrsim r(\Sigma)^k \log(d / \delta)$ .
Lemma G.6. Let  $k$  be a positive integer and suppose that  $n \geqslant 10 \cdot \mathrm{r}(\Sigma)^k \cdot \log (d / \delta)$  and
$$
\left\| \Sigma \right\| _ {F} \leqslant \frac {\operatorname {T r} (\Sigma)}{1 0 \sqrt {k \log d}}.
$$
Then with probability at least  $1 - \delta$ , uniform distribution over the set  $\{\pmb{\eta}_1, \dots, \pmb{\eta}_n\}$  has  $(2k, 2k)$ -certifiable  $O\left(\rho \sqrt{k\|\Sigma\|}\right)$ -bounded  $f$ -moments.
Proof. By Lemma K.4,  $\pmb{\eta}$  has  $(2k, 2k)$ -certifiable  $O\left(\rho \sqrt{k\|\Sigma\|}\right)$ -bounded  $f$ -moments. Since  $\forall x \in \mathbb{R}^d$ ,  $\|f(x)\| \leqslant 20\rho \sqrt{d}$  the result follows from Lemma J.4.
G.1 Unknown  $\rho$
If  $\rho$  is not known and if  $\alpha = 0.01$  (or arbitrary constant), we can first divide the  $n$  samples into 2 subsamples of size at least  $n' = \lfloor n / 2 \rfloor$ , and use the first half to learn  $\rho$ , and the second half for mean estimation.
To do this, let us take the first  $n'$  samples and again divide them into two subsamples of size at least  $n'' = \lfloor n'/2 \rfloor$ , and subtract the second half from the first half (if  $n'$  is odd, let us ignore the last sample). Then we get  $n''$  ( $\varepsilon$ -corrupted) iid copies  $\xi_1, \ldots, \xi_{n''}$  of an elliptically distributed vector with location 0 and the scatter matrix  $\Sigma$  as the original distribution. Moreover,  $R$  for this distribution satisfies  $\mathbb{P}\left[R \leqslant 2\rho\sqrt{d}\right] \geqslant \alpha^2$ . Let us compute and sort the norms of the samples, and let us take the maximal value  $b$  among the smallest  $\alpha^2/2$  norms. If  $\varepsilon \leqslant 0.01\alpha$ , then by Chernoff bound with probability at least  $1 - \exp(-\Omega(\alpha n))$ ,  $b \leqslant 2\rho\sqrt{d}$  and  $\mathbb{P}[R \leqslant b \mid b] \geqslant \alpha^2/4$ .
Now, we take the second  $n'$  samples, divide them into two subsamples of size at least  $n'' = \lfloor n'/2 \rfloor$ , and add the second half to the first half (if  $n'$  is odd, let us ignore the last sample). Since the initial noise distribution is symmetric about zero, we get ( $\varepsilon$ -corrupted) iid samples  $2\mu + \zeta_1, \ldots, 2\mu + \zeta_{n''}$ , where  $\zeta_i$  have the same distribution as  $\xi_i$  described above. Hence we can use  $h = 20b$  in our proofs and get the same guarantees as if we used  $h = 20\rho \sqrt{d}$  (up to a constant factor  $O(1 / \alpha)$ ).
H Proof of Identifiability for Near-Optimal Error
In this section, we will prove the statement of identifiability underlying Theorem 1.4. We first introduce some notation. We use similar notation as in [Li19]. Let  $D$  be an  $(\alpha, \rho)$ -semi-product distribution with location  $\mu^{*} \in \mathbb{R}^{d}$ . Let  $\pmb{y}_{1}^{*}, \ldots, \pmb{y}_{n}^{*}$  be  $n$  i.i.d.samples from  $D$  and let  $y_{1}, \ldots, y_{n}$  be an  $\varepsilon$  corruption thereof. Further, let  $S_{g} = \{i \mid y_{i} = \pmb{y}_{i}^{*}\}$  be the set of uncorrupted indices and  $S_{b} = [n] \setminus S_{g}$  be the set of corrupted ones. Let
$$
\mathcal {W} _ {\varepsilon} = \left\{w \in \mathbb {R} ^ {n} \mid 0 \leqslant w \leqslant \frac {1}{n} \mathbb {1}, \| w - \frac {1}{n} \mathbb {1} \| _ {1} \leqslant \varepsilon \right\},
$$
where  $\mathbb{1}$  is the all-ones vector and the inequalities between vectors are entry-wise. Throughout the note we assume that  $\varepsilon \lesssim \alpha$ . Note, that otherwise faithfully recovering  $\mu^{*}$  is impossible: For each coordinate, with high probability there is only an  $\Theta (\alpha)$ -fraction of the samples where the noise is small. If  $\varepsilon \gtrsim \alpha$ , the adversary could corrupt precisely this fraction and erase all information about this coordinate.
Further, for  $h > 0$ , let  $F_{h} \colon \mathbb{R}^{d} \to \mathbb{R}$  be defined as  $x \mapsto F_{h}(x) = \sum_{j=1}^{d} \Phi_{h}(x_{j})$ , where  $\Phi_{h}$  is the Huber loss