SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\Phi (t) = \left\{ \begin{array}{l l} \frac {1}{2} t ^ {2}  , & \quad \text {i f}   | t | \leqslant h  , \\ h \cdot (| x | - h)  , & \quad \text {o t h e r w i s e}. \end{array} \right.
$$
and denote by  $f = \nabla F$  its gradient. Note that in our case  $f$  is the derivative of the Huberloss applied entry-wise. The derivative of the Huber loss is  $\phi \colon \mathbb{R} \to \mathbb{R}$ ,
$$
x \mapsto \left\{ \begin{array}{l l} t  , & \quad \text {i f}   | t | \leqslant h  , \\ h \cdot \operatorname {s i g n} (t)  , & \quad \text {o t h e r w i s e}. \end{array} \right.
$$
For  $w\in \mathbb{R}^n$  with non-negative entries, we define  $\mathcal{L}^w\colon \mathbb{R}^d\to \mathbb{R}$  as
It follows that  $\nabla \mathcal{L}^w (\mu) = \sum_{i = 1}^n w_if(\mu -y_i)$ . We let
$$
\hat{\mu}^{w} = \operatorname *{arg  min}_{\mu \in \mathbb{R}^{d}}\mathcal{L}^{w}(\mu).
$$
and  $\hat{\Sigma}_f^w = \sum_{i=1}^{n} w_i f(y_i - \hat{\mu}^w) f(y_i - \hat{\mu}^w)^\top$ .
Similar as in Appendix G.1, by looking at differences of pairs of samples, we can assume that we have access to the  $(\varepsilon$ -corrupted) samples from the noise distribution, by replacing  $\alpha$  by  $\alpha^2$  in the definition of the semi-product distribution. This allows us to compute the following relevant
parameters: Note that  $\mathbb{E}_{\eta \sim D}f(\pmb{\eta}^{*})f(\pmb{\eta}^{*})^{\top}$  is diagonal. We can estimate each diagonal entry  $\sigma_{j}^{2}$  up to a factor  $(1\pm O(\varepsilon +\sqrt{\log(1 / \delta) / n}))$  with probability  $1 / \delta$  by using 1-dimensional robust mean estimation. We will assume that  $\mathbb{E}_{\eta \sim D}f(\pmb{\eta}^{*})f(\pmb{\eta}^{*})^{\top}$  is known, the error  $O(\varepsilon)$  does not affect our results, since our error bounds will be larger than this. Similarly, when  $\alpha$  is known, we can also compute  $\rho$  as in Appendix G.1.
Identifiability Our aim will be to prove the following theorem
Theorem H.1. Let  $w \in \mathcal{W}_{\varepsilon}$ ,  $\delta \in (0,1)$ , and assume that  $\varepsilon \leqslant \alpha / 100$ ,  $\rho = 1$  and  $n \geqslant \frac{d\log(d)}{\alpha^4} + \log(1/\delta)$ . In the above let  $h = 2$ . Then with probability at least  $1 - \delta$  it holds that
$$
\left\| \hat {\mu} ^ {w} - \mu^ {*} \right\| \leqslant O \left(\sqrt {\frac {d \log (d) + \log (1 / \delta)}{\alpha^ {2} n}} + \frac {1}{\alpha^ {3 / 2}} \cdot \sqrt {\varepsilon \left(\left\| \Sigma_ {f} ^ {w} - \mathbb {E} f (\boldsymbol {\eta} ^ {*}) f (\boldsymbol {\eta} ^ {*}) ^ {\top} \right\| + \varepsilon \log (1 / \varepsilon)\right)}\right).
$$
In our proof, we will use the following properties analogous to stability-conditions used before in the literature (cf. for example [DK22]).
Lemma H.2. Suppose that the conditions of Theorem H.1 hold. Then with probability at least  $1 - \delta$  the following events hold
1.  $\| \sum_{i = 1}^{n}w_{i}f\left(\pmb{\eta}_{i}^{*}\right)\| \leqslant O\left(\varepsilon \sqrt{\log(1 / \varepsilon)} +\sqrt{\frac{d + \log(1 / \delta)}{n}}\right),$
2.  $\left\| \sum_{i\in S_g}w_if(\pmb {\eta}^*)f(\pmb {\eta}^*)^\top -\mathbb{E}f(\pmb {\eta}_i^*)f(\pmb {\eta}_i^*)^\top \right\| \leqslant O\left(\varepsilon \log (1 / \varepsilon) + \sqrt{\frac{d + \log(1 / \delta)}{n}}\right),$
3.  $\begin{array}{rlr}\sum_{i\in S_g}w_if(\hat{\mu}^w -\pmb {y}_i^*)f(\hat{\mu}^w -\pmb {y}_i^*)^\top & \succeq & \mathbb{E}f(\pmb {\eta}^*)f(\pmb {\eta}^*)^\top \\ O\left(\varepsilon \log (1 / \varepsilon) / \alpha -\sqrt{\frac{d\log d + \log(1 / \delta)}{n}}\right)\cdot \mathrm{Id}_d. \end{array}$
Proof. By Lemma F.4,  $f(\pmb{\eta}^{*})$  is  $O(1)$ -sub-Gaussian. Hence the first two inequalities follow from Fact J.7.
Let us prove the third one. First, let  $\Delta \in [-10\varepsilon /\alpha ,10\varepsilon /\alpha ]^d$  be an arbitrary fixed (non-random) vector. Let  $A = \{|\pmb{\eta}_{ij}|  | i\in [n], j\in [d]\}$ . Consider the random matrix
$$
\boldsymbol {M} (i) = f \left(\boldsymbol {\eta} _ {i} ^ {*} + \Delta\right) f \left(\boldsymbol {\eta} _ {i} ^ {*} + \Delta\right) ^ {\top} - \mathbb {E} \left[ f \left(\boldsymbol {\eta} _ {i} ^ {*} + \Delta\right) \mid \boldsymbol {A} \right] \mathbb {E} \left[ f \left(\boldsymbol {\eta} _ {i} ^ {*} + \Delta\right) \mid \boldsymbol {A} \right] ^ {\top}
$$
Since for  $l, j \in [d]$  such that  $l \neq j$ ,  $\mathrm{sign}(\pmb{\eta}_{ij})$  and  $\mathrm{sign}(\pmb{\eta}_{il})$  are independent given  $A$ ,  $\mathbb{E}[M(i)_{lj} \mid A] = 0$ . Note that for all  $j \in [d]$ ,  $\mathbb{E}[M(i)_{jj} \mid A]$  is equal to
$$
\mathbb {E} \left[ f \left(\left(\boldsymbol {\eta} _ {i} ^ {*}\right) _ {j}\right) ^ {2} - f \left(\left(\boldsymbol {\eta} _ {i} ^ {*}\right) _ {j}\right) \left[ f \left(\left(\boldsymbol {\eta} ^ {*}\right) _ {j} + \Delta_ {j}\right) - f \left(\left(\boldsymbol {\eta} _ {i} ^ {*}\right) _ {j}\right) \right] + \left[ f \left(\left(\boldsymbol {\eta} ^ {*}\right) _ {j} + \Delta_ {j}\right) - f \left(\left(\boldsymbol {\eta} _ {i} ^ {*}\right) _ {j}\right) \right] ^ {2} \mid \boldsymbol {A} \right].
$$
Since  $f$  is 1-Lipschitz and is bounded by 2,
$$
\left| \mathbb {E} \left[ f \left(\left(\boldsymbol {\eta} _ {i} ^ {*}\right) _ {j}\right) \left(f \left(\left(\boldsymbol {\eta} _ {i} ^ {*}\right) _ {j} + \Delta_ {j}\right) - f \left(\left(\boldsymbol {\eta} _ {i} ^ {*}\right) _ {j}\right)\right) \mid \boldsymbol {A} \right] \right| \leqslant 2 | \Delta_ {j} | \leqslant 2 0 \varepsilon / \alpha .
$$
$$
\mathbb {E} [ \boldsymbol {M} (i) \mid \boldsymbol {A} ] \succeq \mathbb {E} \left[ f \left(\boldsymbol {\eta} _ {i} ^ {*}\right) f \left(\boldsymbol {\eta} _ {i} ^ {*}\right) ^ {\top} \mid \boldsymbol {A} \right] - O (\varepsilon / \alpha) \cdot \operatorname {I d} _ {d}.
$$
Denote  $D = \frac{1}{n}\sum_{i = 1}^{n}\mathbb{E}\left[f(\pmb{\eta}_i^*)f(\pmb{\eta}_i^*)^\top \Big|A\right]$ . Since the entries of  $\operatorname{sign}(\pmb{\eta}^*)$  are mutually independent given  $\pmb{A}$ , the entries of  $f(\pmb{\eta}^* + \Delta)$  are also mutually independent given  $\pmb{A}$ . Since the entries are bounded by  $O(1)$ , by Lemma J.8, for all  $\delta \in (0,1)$ ,
$$
\mathbb {P} \left[ \sum_ {i = 1} ^ {n} w _ {i} ^ {\prime} f \left(\boldsymbol {\eta} _ {i} ^ {*} + \Delta\right) f \left(\boldsymbol {\eta} _ {i} ^ {*} + \Delta\right) ^ {\top} \succeq \boldsymbol {D} - O \left(\frac {\varepsilon}{\alpha} + \varepsilon \log (1 / \varepsilon) + \sqrt {\frac {d + \log \left(\frac {1}{\delta}\right)}{n}}\right) \cdot \operatorname {I d} _ {d} \mid \boldsymbol {A} \right] \geqslant 1 - \delta / 1 0,
$$
where  $w_{i}^{\prime} = w_{i}\mathbf{1}_{[i\in S_{g}]}$ . Taking expectations from both sides, we get
$$
\mathbb {P} \left[ \sum_ {i = 1} ^ {n} w _ {i} ^ {\prime} f \left(\boldsymbol {\eta} _ {i} ^ {*} + \Delta\right) f \left(\boldsymbol {\eta} _ {i} ^ {*} + \Delta\right) ^ {\top} \succeq \boldsymbol {D} - O \left(\frac {\varepsilon}{\alpha} + \varepsilon \log (1 / \varepsilon) + \sqrt {\frac {d + \log \left(\frac {1}{\delta}\right)}{n}}\right) \cdot \operatorname {I d} _ {d} \right] \geqslant 1 - \delta / 1 0.
$$
Let  $\mathcal{N}_{\varepsilon'}$  be an  $\varepsilon'$ -net in  $[-10\varepsilon/\alpha, 10\varepsilon/\alpha]^d$  of size at most  $\left(\frac{20\varepsilon}{\varepsilon' \alpha}\right)^d$ . Since  $f$  is 1-Lipschitz and is bounded by 1,
$$
\left\| f \left(\boldsymbol {\eta} _ {i} ^ {*} + \Delta\right) w _ {i} f \left(\boldsymbol {\eta} _ {i} ^ {*} + \Delta\right) ^ {\top} - f \left(\boldsymbol {\eta} _ {i} ^ {*} + \Delta^ {\prime}\right) f \left(\boldsymbol {\eta} _ {i} ^ {*} + \Delta^ {\prime}\right) ^ {\top} \right\| _ {\max } \leqslant 2 \| \Delta - \Delta^ {\prime} \| _ {\max }.
$$
Since the spectral norm can be only by at most  $\sqrt{d}$  factor larger than the infinity norm, by taking  $\varepsilon' = \varepsilon \sqrt{d} / (100\alpha)$  and using a union bound, we get that with probability  $1 - \delta / 10$ , for all  $\Delta \in [-10\varepsilon / \alpha, 10\varepsilon / \alpha]^d$  (including  $\hat{\mu}^w$ ),
$$
\sum_ {i = 1} ^ {n} w _ {i} ^ {\prime} f \left(\boldsymbol {\eta} _ {i} ^ {*} + \Delta\right) f \left(\boldsymbol {\eta} _ {i} ^ {*} + \Delta\right) ^ {\top} \succeq \boldsymbol {D} - O \left(\frac {\varepsilon}{\alpha} + \sqrt {\frac {d \log d + \log \left(\frac {1}{\delta}\right)}{n}} + \frac {d \log d + \log \left(\frac {1}{\delta}\right)}{n}\right) \cdot \mathrm {I d} _ {d}.
$$
Note that  $D$  is a diagonal matrix, and its diagonal entries are
$$
\boldsymbol {D} _ {j j} = \frac {1}{n} \sum_ {i = 1} ^ {n} f \left(\boldsymbol {\eta} _ {i j} ^ {*}\right) ^ {2}.
$$
Since  $f(\pmb{\eta}_j^*)$  are  $O(1)$ -sub-Gaussian, with probability at least  $1 - \delta / 10$ ,
$$
\left\| \boldsymbol {D} - \mathbb {E} f \left(\boldsymbol {\eta} ^ {*}\right) f \left(\boldsymbol {\eta} ^ {*}\right) ^ {\top} \right\| \leqslant O \left(\sqrt {\frac {\log (d / \delta)}{n}}\right).
$$
Hence we get the desired bound.
Proof of Theorem H.1. By Lemma F.1,  $\frac{\alpha}{4}\left\| \hat{\mu}^w - \mu^*\right\|^2 \leqslant \langle \nabla \mathcal{L}^w (\mu^*), \mu^* - \hat{\mu}^w \rangle$ . Hence, it is enough to bound  $\langle \nabla \mathcal{L}^w (\mu^*), \mu^* - \hat{\mu}^w \rangle$ . Let  $u = \hat{\mu}^w - \mu^*$ . Observe that using the Cauchy-Schwarz Inequality, it follows that
$$
\begin{array}{l} \langle \nabla \mathcal {L} ^ {w} (\mu^ {*}), u \rangle = \sum_ {i = 1} ^ {n} w _ {i} \left\langle f \left(\mu^ {*} - y _ {i}\right), u \right\rangle \\ = \sum_ {i = 1} ^ {n} w _ {i} \left\langle f \left(\mu^ {*} - \boldsymbol {y} _ {i} ^ {*}\right), u \right\rangle + \sum_ {i \in S _ {b}} w _ {i} \left\langle f \left(\mu^ {*} - y _ {i}\right) - f \left(\mu^ {*} - \boldsymbol {y} _ {i} ^ {*}\right), u \right\rangle \\ \leqslant \left\| \sum_ {i = 1} ^ {n} w _ {i} f \left(\mu^ {*} - \boldsymbol {y} _ {i} ^ {*}\right) \right\| \cdot \| u \| + \sqrt {\varepsilon} \cdot \sqrt {\sum_ {i \in S _ {b}} w _ {i} \left\langle f \left(\mu^ {*} - y _ {i}\right) - f \left(\mu^ {*} - \boldsymbol {y} _ {i} ^ {*}\right) , u \right\rangle^ {2}}. \\ \end{array}
$$
Note that by the first point of Lemma H.2 it holds that  $\| \sum_{i = 1}^{n}w_{i}f\left(\mu^{*} - \boldsymbol{y}_{i}^{*}\right)\| \leqslant O(\varepsilon \sqrt{\log(1 / \varepsilon)} +$ $\sqrt{\frac{d + \log(1 / \delta)}{n}})$ . For the second term, we can bound
$$
\begin{array}{l} \sum_ {i \in S _ {b}} w _ {i} \left\langle f \left(\mu^ {*} - y _ {i}\right) - f \left(\mu^ {*} - \boldsymbol {y} _ {i} ^ {*}\right), u \right\rangle^ {2} \\ \lesssim \underbrace{\sum_{i\in S_{b}}w_{i}\left\langle f\left(\mu^{*} - \boldsymbol{y}_{i}^{*}\right),u\right\rangle^{2}}_{\text{Term A}} + \underbrace{\sum_{i\in S_{b}}w_{i}\left\langle f\left(\hat{\mu}^{w} - y_{i}\right),u\right\rangle^{2}}_{\text{Term B}} + \underbrace{\sum_{i\in S_{b}}w_{i}\left\langle f\left(\mu^{*} - y_{i}\right) - f\left(\hat{\mu}^{w} - y_{i}\right),u\right\rangle^{2}}_{\text{Term C}}. \\ \end{array}
$$
Term A is at most  $O\left(\varepsilon \log (1 / \varepsilon) + \sqrt{\frac{d + \log(1 / \delta)}{n}}\right) \cdot \| u\| ^2$  by combining the second point of Lemma H.2 with Fact J.7. To bound Term C, we use that  $f$  is 1-Lipschitz to observe
$$
\sum_ {i \in S _ {b}} w _ {i} \left<   f \left(\mu^ {*} - y _ {i}\right) - f \left(\hat {\mu} ^ {w} - y _ {i}\right), u \right> ^ {2}
$$
$$
\leqslant \| u \| ^ {2} \cdot \sum_ {i \in S _ {b}} w _ {i} \| f (\mu^ {*} - y _ {i}) - f (\hat {\mu} ^ {w} - y _ {i} ^ {*}) \| ^ {2} \leqslant \| u \| ^ {2} \cdot \sum_ {i \in S _ {b}} w _ {i} \| \mu^ {*} - \hat {\mu} ^ {w} \| ^ {2} \leqslant \varepsilon \cdot \| u \| ^ {4}.
$$
It remains to bound Term B. To this end, we use the third point of Lemma H.2 and obtain
$$
\begin{array}{l} \sum_ {i \in S _ {b}} w _ {i} \left\langle f \left(\hat {\mu} ^ {w} - y _ {i}\right), u \right\rangle^ {2} = \sum_ {i = 1} ^ {n} w _ {i} \left\langle f \left(\hat {\mu} ^ {w} - y _ {i}\right), u \right\rangle^ {2} - \sum_ {i \in S _ {g}} w _ {i} \left\langle f \left(\hat {\mu} ^ {w} - \boldsymbol {y} _ {i} ^ {*}\right), u \right\rangle^ {2} \\ \leqslant \left(\left\| \hat {\Sigma} _ {f} - \mathbb {E} f \left(\boldsymbol {\eta} ^ {*}\right) f \left(\boldsymbol {\eta} ^ {*}\right) ^ {\top} \right\| + O (\varepsilon \log (1 / \varepsilon) / \alpha) + \sqrt {\frac {d \log d + \log \left(\frac {1}{\delta}\right)}{n}}\right) \cdot \| u \| ^ {2}. \\ \end{array}
$$
Putting the above bounds together, we have shown that
$$
\left\langle \nabla \mathcal {L} ^ {w} \left(\mu^ {*}\right), u \right\rangle \leqslant \| u \| \cdot O \left(\frac {\varepsilon \sqrt {\log (1 / \varepsilon)}}{\sqrt {\alpha}} + \sqrt {\frac {d \log (d) + \log (1 / \delta)}{n}} + \varepsilon \| u \| + \sqrt {\varepsilon \left\| \hat {\Sigma} _ {f} - \mathbb {E} f \left(\boldsymbol {\eta} ^ {*}\right) f \left(\boldsymbol {\eta} ^ {*}\right) ^ {\top} \right\|}\right).
$$
(Note that we also use  $\sqrt{\varepsilon} \cdot \sqrt{\varepsilon \log(1 / \varepsilon) + \sqrt{\frac{d + \log(1 / \delta)}{n}}} \leqslant \varepsilon \sqrt{\log(1 / \varepsilon)} + \sqrt{\frac{d + \log(1 / \delta)}{n}}$ .) Since  $\langle \nabla \mathcal{L}^w(\mu^*), u \rangle \geqslant \frac{\alpha}{4} \|u\|^2$  and  $\varepsilon \lesssim \alpha$ , rearranging and dividing by  $\|u\|$  yields that
$$
\| u \| \leqslant O \left(\sqrt {\frac {d \log (d) + \log (1 / \delta)}{\alpha^ {2} n}} + \frac {1}{\alpha^ {3 / 2}} \cdot \sqrt {\varepsilon \left(\left\| \hat {\Sigma} _ {f} - \mathbb {E} f (\boldsymbol {\eta} ^ {*}) f (\boldsymbol {\eta} ^ {*}) ^ {\top} \right\| + \varepsilon \log (1 / \varepsilon)\right)}\right).
$$
I Filtering Algorithm for Near-Optimal Error
In this section, we will use the proof of identifiability from Appendix H to obtain an algorithm which efficiently recovers  $\mu^{*}$  up to (nearly) optimal error. In particular, we show that
Theorem I.1. Let  $C > 0$  be a large enough absolute constant. Let  $\varepsilon, \alpha > 0$  be such that  $\sqrt[3]{\varepsilon} \leqslant \frac{\alpha}{C}$ . Let  $D$  be an  $(\alpha, \rho)$ -product distribution with location  $\mu^{*} \in \mathbb{R}^{d}$ . Let  $n \geqslant C \cdot \frac{d\log(d) + \log(1 / \delta)}{\alpha^{6}\varepsilon^{2}\log(1 / \varepsilon)}$ . There exists an algorithm running in time  $n^{O(1)}d^{O(1)}$ , such that given an  $\varepsilon$ -corrupted sample of  $D$  of size  $n$  and  $\rho^{9}$ , with probability at least  $1 - \delta$  it outputs  $\hat{\mu}$  satisfying
$$
\left\| \mu^ {*} - \hat {\mu} \right\| \leqslant O \left(\rho \cdot \left[ \sqrt {\frac {d \log (d) + \log (1 / \delta)}{\alpha^ {4} n}} + \frac {\varepsilon \sqrt {\log (1 / \varepsilon)}}{\alpha^ {3}} \right]\right).
$$
The algorithm we propose is based on the by-now-standard filtering approach. We closely follow the exposition in [Li19]. To describe the algorithm, let  $C > 0$  be some universal constant and for  $\mu \in \mathbb{R}^d$ ,  $w \in \mathcal{W}_{\varepsilon}$  let
$$
\hat {\mu} \left(w\right) = \min  _ {\mu \in \mathbb {R} ^ {d}} \mathcal {L} ^ {w} \left(\mu\right), \qquad \qquad \Sigma_ {f} \left(w\right) = \sum_ {i = 1} ^ {n} w _ {i} f \left(\hat {\mu} \left(w\right) - y _ {i}\right) f \left(\hat {\mu} \left(w\right) - y _ {i}\right) ^ {\top}.
$$
Note that since we assume  $\rho$  to be known, we can without loss of generality assume that  $\rho = 1$  by scaling. Our algorithm is the following.
<sup>9</sup>We remark that in the case of unknown  $\rho$  but known  $\alpha$ , we can first estimate  $\rho$  from the corrupted samples similar as in Appendix G.1, at the cost of a worse dependence on  $\alpha$ .
Algorithm I.2 (Filtering Algorithm).
Input:  $\varepsilon$ -corrupted sample  $y_{1},\ldots ,y_{n}$  and  $\rho ,\gamma >0$
Output: Location estimate  $\hat{\mu}$ .
Let  $w^{(0)} = \frac{1}{n}\mathbb{1}_n$
- Compute  $\hat{\mu}^{(0)} = \min_{\mu \in \mathbb{R}^d}\mathcal{L}^{w^{(0)}}(\mu)$  and  $\Sigma_f^{(0)} = \Sigma_f(w^{(0)})$
- Let  $t = 0$ .
- while  $\| \Sigma_f^{(t)} - \mathbb{E}f(\pmb{\eta}^*)f(\pmb{\eta}^*)^\top\| >C\varepsilon \log (1 / \varepsilon) / \alpha^3$  do
- Compute  $v^{(t)}$  the top eigenvector of  $\Sigma_f^{(t)} - \mathbb{E}f(\pmb{\eta}^*)f(\pmb{\eta}^*)^\top$ .
- For  $i \in [n]$ , compute  $\tau_i^{(t)} = \left\langle v^{(t)}, f(\hat{\mu}^{(t)} - y_i) \right\rangle^2$ .
- Sort the  $\tau_{i}$ . Assume that  $\tau_{1} \geqslant \ldots \geqslant \tau_{n}$ .
- Let  $N$  be the smallest index such that  $\sum_{i\leqslant N}w_i^{(t)} > 2\varepsilon$  and let  $\tau_{\mathrm{max}} = \max_{i\in [n]}\tau_{i}$ .
- For  $1 \leqslant i \leqslant N$ , set  $w_{i}^{(t + 1)} = (1 - \frac{\tau_{i}}{\tau_{\max}}) \cdot w_{i}^{(t)}$  and for  $N < i \leqslant n$ , set  $w_{i}^{(t + 1)} = w_{i}^{(t)}$ .
- Compute  $\hat{\mu}^{(t + 1)} = \min_{\mu \in \mathbb{R}^d}f^{w^{(t + 1)}}(\mu)$  and  $\Sigma_f^{(t + 1)} = \Sigma_f(w^{(t + 1)})$
-  $t\gets t + 1$
We will use the following lemma whose proof we will give at the end of this section.
Lemma I.3. Assume that  $\| \Sigma_f^{(t)} - \mathbb{E}f(\pmb {\eta}^*)f(\pmb {\eta}^*)^\top \| >C\varepsilon \log (1 / \varepsilon) / \alpha^3$  and
With this in hand, we will prove Theorem I.1.
Proof of Theorem I.1. First note, that every iteration can clearly be implanted to run in time  $n^{O(1)}d^{O(1)}$ . We will show that the algorithm terminates after at most  $\lceil 2\varepsilon n\rceil$  iterations. Assume towards a contradiction that the algorithm does not terminate after  $T = \lceil 2\varepsilon n\rceil$  iterations. Note that the number entries of  $w^{(t)}$  that are equal to 0 increases by at least 1 in every iteration. Hence, after  $T$  iterations we have set at least  $\varepsilon n$  entries of  $w$  to zero whose index lies in  $S_{g}$ . By assumption that the algorithm didn't terminate and Lemma I.3 it holds that
Since also  $0 \leqslant w^{(T)} \leqslant \frac{1}{n}$ , it follows that  $w^{(T)} \in \mathcal{W}_{2\varepsilon}$ . By Theorem H.1 and since  $\| \Sigma_f^{(T)} - \mathbb{E}f(\boldsymbol{\eta}^*)f(\boldsymbol{\eta}^*)^\top\| \leqslant C\varepsilon \log (1 / \varepsilon) / \alpha^3$  it follows that
$$
\left\| \hat {\mu} ^ {T} - \mu^ {*} \right\| \leqslant O \left(\rho \cdot \left[ \sqrt {\frac {d \log (d) + \log (1 / \delta)}{n}} + \frac {\varepsilon \sqrt {\log (1 / \varepsilon)}}{\alpha^ {3}} \right]\right).
$$
Lastly, we will prove Lemma I.3.
Proof of Lemma I.3. On a high level, we will show that each iteration of the filtering algorithm removes relatively more weight from corrupted samples than from uncorrupted ones. For simplicity, let  $w = w^{(t)}$  and  $w' = w^{(t+1)}$ . Also, let  $\hat{\mu} = \hat{\mu}^{(t)}, \Sigma_f = \Sigma_f^{(t)}$ , and  $v = v^{(t)}$ . Note, that it is enough to show that
Let  $N$  be the smallest index such that  $\sum_{i\leqslant N}w_i > \varepsilon$  and denote by  $T = \{1,\ldots ,N\}$  Further, recall that  $w_{i}^{\prime} = \left(1 - \frac{\tau_{i}}{\tau_{\max}}\right)w_{i}$  if  $i\in T$  and  $w_{i}^{\prime} = w_{i}$  otherwise. Thus, for  $i\in T$ , it holds that  $w_{i} - w_{i}^{\prime} = \frac{1}{\tau_{\max}}\tau_{i}w_{i}$ , while for  $i\notin T$  it holds that  $w_{i} - w_{i}^{\prime} = 0$ . Hence, the above condition is equivalent to
$$
\sum_ {i \in S _ {g} \cap T} w _ {i} \tau_ {i} <   \sum_ {i \in S _ {b} \cap T} w _ {i} \tau_ {i}.
$$
We will show that the above is indeed true in two steps. We will show that
1.  $\sum_{i\in S_g\cap T}w_i\tau_i\leqslant O\left(\varepsilon \log (1 / \varepsilon) + \frac{\varepsilon^2}{\alpha^3}\cdot \left\| \Sigma_f - \mathbb{E}f(\pmb {\eta}^*)f(\pmb {\eta}^*)^\top \right\|\right)$
2.  $\sum_{i\in S_b\cap T}w_i\tau_i\geqslant \Omega \left(\| \Sigma_f - \mathbb{E}f(\boldsymbol {\eta}^*)f(\boldsymbol {\eta}^*)^\top \| \right)$
This implies our claim since by assumption  $\| \Sigma_f - \mathbb{E}f(\pmb{\eta}^*)f(\pmb{\eta}^*)^\top \| > C\varepsilon \log (1 / \varepsilon) / \alpha^3 > C\varepsilon \log (1 / \varepsilon)$  and  $\frac{\varepsilon^2}{\alpha^3} \leqslant \frac{\varepsilon}{\alpha^3}$  is sufficiently small.
Upper Bounding the Contribution of  $S_{g} \cap T$  Recall that  $n \geqslant \frac{d\log(d) + \log(1 / \delta)}{\alpha^6\varepsilon^2\log(1 / \varepsilon)}$ . Thus, by definition of  $\tau_{i}$  and since  $f$  is 1-Lipschitz it follows that
$$
\begin{array}{l} \sum_ {i \in S _ {g} \cap T} w _ {i} \tau_ {i} = \sum_ {i \in S _ {g} \cap T} w _ {i} \langle f (\hat {\mu} - \boldsymbol {y} _ {i} ^ {*}), v \rangle^ {2} \\ \leqslant O \left(\sum_ {i \in S _ {g} \cap T} w _ {i} \langle f (\mu^ {*} - \boldsymbol {y} _ {i} ^ {*}), v \rangle^ {2} + \sum_ {i \in S _ {g} \cap T} w _ {i} \langle f (\hat {\mu} - \boldsymbol {y} _ {i} ^ {*}) - f (\mu^ {*} - \boldsymbol {y} _ {i} ^ {*}), v \rangle^ {2}\right) \\ \leqslant O \left(\varepsilon \log (1 / \varepsilon) + \sum_ {i \in S _ {g} \cap T} w _ {i} \| \hat {\mu} - \mu^ {*} \| ^ {2}\right) \\ \leqslant O \left(\varepsilon \log (1 / \varepsilon) + \varepsilon \cdot \left(\frac {1}{\alpha^ {3}} \cdot \left[ \varepsilon \left\| \Sigma_ {f} - \mathbb {E} f (\boldsymbol {\eta} ^ {*}) f (\boldsymbol {\eta} ^ {*}) ^ {\top} \right\| + \varepsilon^ {2} \log (1 / \varepsilon) \right]\right)\right) \\ \leqslant O \left(\varepsilon \log (1 / \varepsilon) + \frac {\varepsilon^ {2}}{\alpha^ {3}} \cdot \left\| \Sigma_ {f} - \mathbb {E} f (\boldsymbol {\eta} ^ {*}) f (\boldsymbol {\eta} ^ {*}) ^ {\top} \right\|\right), \\ \end{array}
$$
where in the second inequality we used the last property of Fact J.7 and in the third inequality we used Theorem H.1.
Lower Bounding the Contribution of  $S_{b} \cap T$  Our strategy will be to first show that the contribution of all points in  $S_{b}$ , also those not necessarily in  $T$ , must be large and then show that the contribution of points in  $S_{b} \setminus T$  is in fact small. Together they will yield our claim. Note that by definition of  $v$ , it holds that
$$
\sum_ {i = 1} ^ {n} w _ {i} \tau_ {i} = v ^ {\top} \left(\Sigma_ {f} - \mathbb {E} f (\boldsymbol {\eta} ^ {*}) f (\boldsymbol {\eta} ^ {*}) ^ {\top}\right) v + \mathbb {E} \left\langle f (\boldsymbol {\eta} ^ {*}), v \right\rangle^ {2} = \mathbb {E} \left\langle f (\boldsymbol {\eta} ^ {*}), v \right\rangle^ {2} + \left\| \Sigma_ {f} - \mathbb {E} f (\boldsymbol {\eta} ^ {*}) f (\boldsymbol {\eta} ^ {*}) ^ {\top} \right\|.
$$
Let  $\gamma \coloneqq \mathbb{E}\left\langle f\left(\eta^{*}\right),v\right\rangle^{2}$ . Note that  $\gamma = O(1)$ . By the second point in Lemma H.2, Cauchy-Schwarz inequality and Theorem H.1,
$$
\begin{array}{l} \left| \sum_ {i \in S _ {g}} w _ {i} \tau_ {i} - \gamma \right| = \left| \sum_ {i \in S _ {g}} w _ {i} \langle f (\hat {\mu} - y _ {i}), v \rangle^ {2} - \gamma \right| \\ \leqslant \left| \sum_ {i \in S _ {g}} w _ {i} \langle f (\mu^ {*} - \boldsymbol {y} _ {i} ^ {*}), v \rangle^ {2} - \gamma \right| + \left| \sum_ {i \in S _ {g}} w _ {i} \langle f (\hat {\mu} - \boldsymbol {y} _ {i} ^ {*}) - f (\mu^ {*} - \boldsymbol {y} _ {i} ^ {*}), v \rangle^ {2} \right| \\ + 2 \left| \sum_ {i \in S _ {g}} w _ {i} \langle f \left(\mu^ {*} - \boldsymbol {y} _ {i} ^ {*}\right), v \rangle \left(\langle f \left(\hat {\mu} - \boldsymbol {y} _ {i} ^ {*}\right), v \rangle - \langle f \left(\mu^ {*} - \boldsymbol {y} _ {i} ^ {*}\right), v \rangle\right) \right| \\ \leqslant O \left(\varepsilon \log (1 / \varepsilon) + \| \hat {\mu} - \mu^ {*} \| ^ {2} + \sqrt {\gamma} \cdot \| \hat {\mu} - \mu^ {*} \|\right) \\ \leqslant O \left(\varepsilon \log (1 / \varepsilon) + \frac {\varepsilon}{\alpha^ {3}} \left\| \Sigma_ {f} - \mathbb {E} f (\boldsymbol {\eta} ^ {*}) f (\boldsymbol {\eta} ^ {*}) ^ {\top} \right\| + \frac {\varepsilon^ {2} \log (1 / \varepsilon)}{\alpha^ {3}}\right) \\ \leqslant O \left(\frac {\varepsilon}{\alpha^ {3}} \left\| \Sigma_ {f} - \mathbb {E} f (\boldsymbol {\eta} ^ {*}) f (\boldsymbol {\eta} ^ {*}) ^ {\top} \right\| + \frac {\varepsilon^ {2} \log (1 / \varepsilon)}{\alpha^ {3}} + \| \hat {\mu} - \mu^ {*} \|\right). \\ \end{array}
$$