SlowGuess/NeurIPS_2023_OCR huggingface dataset:
error  $O(\sqrt{\varepsilon})$ . Similarly, for  $\alpha \in [0,1]$ , [CTBJ22] shows how to achieve error  $O(\varepsilon^{\alpha/(1+\alpha)})$  when in every direction the  $1+\alpha$  moments are bounded. Note that in constraint to this, our algorithm can handle distributions without a first moment and achieves error  $O(\varepsilon \sqrt{\log(1/\varepsilon)})$ . An interesting question is whether assuming bounded higher-order moments can lead to improved error guarantees. Interestingly, [HL19] shows that this alone is likely to be not enough in order to go beyond error  $\sqrt{\varepsilon}$ . So far, there have been two ways of adding additional assumptions: First, assuming that  $\mathcal{D}$  has  $k$ -th moments bounded by  $\sigma_k$  in every direction, and assuming that it is certifiable in sum-of-squares, it is possible to achieve error  $O(\sigma_k\varepsilon^{1-1/(2k)})$  using  $d^{O(k)}$  samples and time [HL18, KSS18, ST21]. Second, if the  $k$ -th moments are bounded by  $\sigma_k$ , not necessarily certifiably in sum-of-squares, and the covariance matrix is a multiple of the identity (or known), [DKP20] achieves error  $O(\sigma_k\varepsilon^{1-1/(2k)})$ .
Other Results On Symmetric Noise Distributions and the Filtering Techniques Huber-loss minimization has been used to design computationally efficient estimators for a variety of problems: Linear regression in the heavy-tailed/symmetric setting [TJSO14, PJL20, SZF20, dNS21,  $\mathrm{dLN^{+}21}$ ], as well as the robust setting [PJL20]. Other works developed algorithms for PCA under symmetric noise [CLMW11,  $\mathrm{dLN^{+}21}$ , dNNS23].
The filtering technique [DKK+19, DKK+17] (or versions thereof) have been successfully applied to robust and heavy-tailed (when the covariance exists) mean estimation [DHL19, DKP20, HLZ20, DK22, DKLP22] as well as a pre-processing step in robust regression [PJL20].
We will first describe the general theme of our techniques. These ideas will yield algorithms achieving (nearly) optimal guarantees using quasi-polynomially many samples. Note however, that using these, we can already achieve error  $o(\sqrt{\varepsilon})$  using only polynomially many samples. At the end of this section we show how to improve this to nearly linearly many (in the dimension) samples for semi-product distributions.
One-Dimensional Robust Estimation via Huber-Loss Minimization Current algorithms for robust mean estimation are only known to work for distributions with bounded moments. In many cases this is assumption is reasonable, and in many cases it is also necessary [KSS18, CTBJ22]. However, there are many other distributions that do not necessarily have any moments at all, but which one would expect to behave nicely. Current approaches are inapplicable in these settings. Of particular interest is the class of symmetric distributions, which might not even have a first moment - in this case we want to estimate the location parameter of the distribution, which always exists and coincides with the mean in case it exists. In non-robust statistics (when there are no corruptions), symmetric distributions in some sense behave as nicely as the Gaussian distribution: The minimizer of the entry-wise Huber-loss function achieves the same guarantees for such distributions as the sample mean in the Gaussian case. The entry-wise Huber-loss is the function  $\mathcal{L}_H:\mathbb{R}^d\to \mathbb{R}$  defined as  $\mathcal{L}_H(v)\coloneqq \sum_{j = 1}^d\Phi (v_j)$ , where  $\Phi \colon \mathbb{R}\to \mathbb{R}$  is a Huber penalty:  $\Phi (x) = x^{2} / 2$ , if  $|x|\leqslant 1$  and  $\Phi (x) = |x| - 1 / 2$  otherwise.  $\Phi$  is convex, and has many appealing properties, in particular with respect to symmetric distributions, since its derivative is a bounded odd Lipschitz function:  $\phi (x)\coloneqq \Phi '(x) = x$ , if  $|x|\leqslant 1$ , and  $\phi (x) = \mathrm{sign}(x)$  otherwise.
Since the Gaussian distribution is symmetric itself, the Huber-loss estimator has the same guarantees as the sample mean in this setting. Beyond that, the Huber-loss estimator enjoys some robustness guarantees that the sample mean does not: Let us consider the one-dimensional case, where the Huber-loss behaves similarly to the median but has additional properties that will be useful to us, e.g., it is differentiable everywhere. In this setting, the sample mean has unbounded error even if there is only a single corrupted sample, while the Huber-loss achieves the information theoretically optimal error  $O(\varepsilon)$  when an arbitrary  $\varepsilon$ -fraction of the samples is corrupted. Moreover, these guarantees extend to arbitrary one-dimensional symmetric distributions that satisfy mild scaling assumptions. Unfortunately, this does not directly yield a good robust estimator for the high-dimensional setting. If we naively apply the one-dimensional estimator entry-wise, we can only guarantee an error bound of  $O(\varepsilon \sqrt{d})$ , which is far away from the  $O(\varepsilon)$  error that is statistically possible for the Gaussian distribution and symmetric distributions. Moreover, it is also inferior to error rates obtained by efficient estimators that use assumptions about bounded moments: Such estimators achieve error
that does not depend on  $d$ . In this work we show that there exist more sophisticated estimators based on the Huber-loss<sup>3</sup> that achieve dimension-independent error, often matching what is possible for the Gaussian distribution, for estimating the location of symmetric distributions even in the high-dimensional robust setting.
Proofs of identifiability and the filtering technique Before describing how we use the Huber-loss in the high-dimensional setting, it will be instructive to recall how the classical approach for distributions with bounded moments works – for simplicity, we focus on bounded second moments and sketch how this extends to higher-order moments. Later, we will see how to modify these ideas, using the Huber-loss, to work also in the symmetric setting without moment assumptions. In particular the version for Gaussian-like error for semi-product distribution requires several new technical ideas compared to the Gaussian setting.
At the heart lie identifiability proofs which can be made algorithmic using either sum-of-squares [KSS18, HL18] or the filtering technique  $\mathrm{[DKK^{+}19,DKK^{+}17,DHL19]}$ . For bounded covariance distributions these take the following form: If two distributions  $D_{1}$  and  $D_{2}$  with means  $\mu_{1}$ , respectively  $\mu_{2}$ , and covariance matrices  $\Sigma_1\preceq \operatorname {Id}_d$ , respectively  $\Sigma_{2}\preceq \operatorname {Id}_{d}$ , are  $\varepsilon$ -close in statistical distance, then  $\| \mu_1 - \mu_2\| \leqslant O\left(\sqrt{\varepsilon}\right)$ . In what follows, for simplicity of the discussion, we will sometimes switch between empirical and true distributions. In robust mean estimation, we assume that  $D_{1}$  is the empirical distribution over the corrupted samples we observe, while  $D_{2}$  is the empirical distribution of the uncorrupted samples. Hence, by assumption,  $\Sigma_{2}\preceq \operatorname {Id}_{d}$  and we wish to estimate  $\mu_{2}$ . The above statement of identifiability asserts that as long as  $\Sigma_{1}\preceq \operatorname {Id}_{d}$ , the empirical mean of the corrupted samples, i.e.,  $\mu_{1}$ , is  $O(\sqrt{\varepsilon})$ -close to  $\mu_{2}$ . Of course,  $\Sigma_{1}\preceq \operatorname {Id}_{d}$  might not hold, since the outliers could potentially introduce a large eigenvalue in the empirical covariance matrix. However, since we know that these large eigenvalues must have been introduced by outliers, this gives rise to a win-win analysis: We check if  $\Sigma_{1}\preceq \operatorname {Id}_{d}$  holds. If it does,  $\mu_{1}$  is  $O(\sqrt{\varepsilon})$ -close to  $\mu_{2}$  already, if not, we compute its top eigenvector and remove samples that have large correlation with this eigenvector. It turns out that this procedure will always remove more corrupted samples than uncorrupted ones. Thus, we can iterate this procedure and are guaranteed that it terminates after  $O(\varepsilon n)$  iterations. After its termination, we are left with a distribution  $D_1'$ , with mean  $\mu_1^\prime$ , that is  $O(\varepsilon)$ -close in total variation distance to  $D_{1}$  and  $D_{2}$ , and has small covariance  $\Sigma_1^{\prime}\preceq \operatorname {Id}_d$ , hence  $\| \mu_1' - \mu_2\| \leqslant O(\sqrt{\varepsilon})$
Of course, for arbitrary symmetric distributions the covariance might not exist. So neither the above proof of identifiability nor the filtering technique apply to this setting. However, one of our main observations is that we can obtain a similar statement of identifiability also in this case. In addition, we will later show how to adjust the filtering technique to work with our identifiability proof - this requires non-trivial modifications of the original approach. First, observe that the identifiability statement in the bounded moment setting, can be phrased as follows: For  $j \in \{1, 2\}$ , the mean  $\mu_j$  of a distribution  $D_j$  corresponds to the minimizer of the quadratic loss:
$$
\mu_ {j} = \operatorname * {a r g   m i n} _ {a \in \mathbb {R} ^ {d}} \underset {\boldsymbol {y} \sim D _ {j}} {\mathbb {E}} \| \boldsymbol {y} - a \| ^ {2}  .
$$
Hence, if  $D_{1}$ , with covariance  $\Sigma_{1} \preceq \mathrm{Id}_{d}$ , and  $D_{2}$  with covariance  $\Sigma_{2} \preceq \mathrm{Id}_{d}$ , are  $\varepsilon$ -close in statistical distance, then the minimizers,  $\mu_{1}$  and  $\mu_{2}$ , of the quadratic loss are  $O(\sqrt{\varepsilon})$ -close to each other. A crucial observation is that, if a distribution  $D$  is symmetric, then its location  $\mu$  is the entry-wise Huber-loss minimizer<sup>4</sup>
$$
\mu = \operatorname *{arg  min}_{a\in \mathbb{R}^{d}}\underset {\boldsymbol {y}\sim D}{\mathbb{E}}\mathcal{L}_{H}\left(\boldsymbol {y} - a\right).
$$
Hence, we would like to obtain a similar statement about minimizers of the Huber-loss for two  $\varepsilon$ -close distributions  $D_{1}$  and  $D_{2}$ . This will be useful since we will need to learn the location  $\mu_{2}$  of a symmetric distribution  $D_{2}$  from  $D_{1}$ .
Abstract proof of identifiability In order to formalize the above, consider the following abstract setting: Let  $\mathcal{L}$  be some loss function that is differentiable,  $\Omega(1)$ -strongly convex and smooth (i.e. its gradient  $\nabla \mathcal{L}$  is 1-Lipschitz). Let  $D_{1}$  and  $D_{2}$  be two distributions that are  $\varepsilon$ -close in the statistical distance, Then, if for  $j \in \{1,2\}$  the minimizers  $\hat{\mu}_{j} := \arg \min_{a \in \mathbb{R}^{d}} \mathbb{E}_{\boldsymbol{y} \sim D_{j}} \mathcal{L}(\boldsymbol{y} - a)$  satisfy
<sup>3</sup>We use entry-wise Huber-loss for semi-product distributions. For elliptical distributions we use another loss function, that is also related to the Huber penalty.
Assuming that each entry of  $y - \mu$  has nonzero probability mass in the interval  $[-1, 1]$ .
$\mathbb{E}_{\pmb{y}\sim D_j}\langle \nabla \mathcal{L}(\pmb {y} - \hat{\mu}_j),u\rangle^2\leqslant 1$  for all unit vectors  $u$ , then  $\| \hat{\mu}_1 - \hat{\mu}_2\| \leqslant O(\sqrt{\varepsilon})$ . Indeed, by strong convexity and the definition of  $\hat{\mu}_1$ ,
$$
\begin{array}{l} \Omega \left(\| \hat {\mu} _ {1} - \hat {\mu} _ {2} \| ^ {2}\right) \leqslant \underset {\boldsymbol {y} \sim D _ {1}} {\mathbb {E}} \mathcal {L} (\boldsymbol {y} - \hat {\mu} _ {1}) - \underset {\boldsymbol {y} \sim D _ {1}} {\mathbb {E}} \mathcal {L} (\boldsymbol {y} - \hat {\mu} _ {2}) - \underset {\boldsymbol {y} \sim D _ {1}} {\mathbb {E}} \langle \nabla \mathcal {L} (\boldsymbol {y} - \hat {\mu} _ {2}), \hat {\mu} _ {1} - \hat {\mu} _ {2} \rangle \\ \leqslant \underset {\boldsymbol {y} \sim D _ {1}} {\mathbb {E}} \left\langle \nabla \mathcal {L} \left(\boldsymbol {y} - \hat {\mu} _ {2}\right), \hat {\mu} _ {1} - \hat {\mu} _ {2} \right\rangle . \\ \end{array}
$$
Notice that there is a coupling  $\omega$  of  $D_{1}$  and  $D_{2}$  such that  $\mathbb{P}_{(\pmb{y}_1,\pmb{y}_2)\sim \omega}(\pmb{y}_1\neq \pmb{y}_2)\leqslant \varepsilon$ . Since  $\mathbb{E}_{\pmb{y}\sim D_2}\nabla \mathcal{L}(\pmb{y} - \hat{\mu}_2) = 0$ , we obtain using the Cauchy-Schwarz Inequality
$$
\begin{array}{l} \Omega \left(\| \hat {\mu} _ {1} - \hat {\mu} _ {2} \| ^ {2}\right) \leqslant \underset {\left(\boldsymbol {y} _ {1}, \boldsymbol {y} _ {2}\right) \sim \omega} {\mathbb {E}} \left\langle \nabla \mathcal {L} \left(\boldsymbol {y} _ {1} - \hat {\mu} _ {2}\right) - \nabla \mathcal {L} \left(\boldsymbol {y} _ {2} - \hat {\mu} _ {2}\right), \hat {\mu} _ {1} - \hat {\mu} _ {2} \right\rangle \\ = \underset {\left(\mathbf {y} _ {1}, \mathbf {y} _ {2}\right) \sim \omega} {\mathbb {E}} \mathbf {1} _ {\left[ \mathbf {y} _ {1} \neq \mathbf {y} _ {2} \right]} \left\langle \nabla \mathcal {L} \left(\mathbf {y} _ {1} - \hat {\mu} _ {2}\right) - \nabla \mathcal {L} \left(\mathbf {y} _ {2} - \hat {\mu} _ {2}\right), \hat {\mu} _ {1} - \hat {\mu} _ {2} \right\rangle \\ \leqslant \sqrt {\varepsilon_ {(\boldsymbol {y} _ {1} , \boldsymbol {y} _ {2}) \sim \omega} \left\langle \nabla \mathcal {L} (\boldsymbol {y} _ {1} - \hat {\mu} _ {2}) - \nabla \mathcal {L} (\boldsymbol {y} _ {2} - \hat {\mu} _ {2}) , \hat {\mu} _ {1} - \hat {\mu} _ {2} \right\rangle^ {2}}. \\ \end{array}
$$
Since  $\nabla \mathcal{L}$  is 1-Lipschitz, using the bounds on  $\max_{||u|| = 1}\mathbb{E}_{\pmb {y}\sim D_j}\left\langle \nabla \mathcal{L}(\pmb {y} - \hat{\mu}_j),u\right\rangle^2$  yields
$$
\begin{array}{l} \underset {(\boldsymbol {y} _ {1}, \boldsymbol {y} _ {2}) \sim \omega} {\mathbb {E}} \left\langle \nabla \mathcal {L} \left(\boldsymbol {y} _ {1} - \hat {\mu} _ {2}\right) - \nabla \mathcal {L} \left(\boldsymbol {y} _ {2} - \hat {\mu} _ {2}\right), \hat {\mu} _ {1} - \hat {\mu} _ {2} \right\rangle^ {2} \\ = \underset {\boldsymbol {y} _ {j} \sim D _ {j}} {\mathbb {E}} \left\langle \nabla \mathcal {L} \left(\boldsymbol {y} _ {1} - \hat {\mu} _ {2}\right) - \nabla \mathcal {L} \left(\boldsymbol {y} _ {1} - \hat {\mu} _ {1}\right) + \nabla \mathcal {L} \left(\boldsymbol {y} _ {1} - \hat {\mu} _ {1}\right) - \nabla \mathcal {L} \left(\boldsymbol {y} _ {2} - \hat {\mu} _ {2}\right), \hat {\mu} _ {1} - \hat {\mu} _ {2} \right\rangle^ {2} \\ \leqslant O \left(\underset {\boldsymbol {y} _ {j} \sim D _ {j}} {\mathbb {E}} \left\langle \nabla \mathcal {L} \left(\boldsymbol {y} _ {1} - \hat {\mu} _ {2}\right) - \nabla \mathcal {L} \left(\boldsymbol {y} _ {1} - \hat {\mu} _ {1}\right), \hat {\mu} _ {1} - \hat {\mu} _ {2} \right\rangle^ {2} + \| \hat {\mu} _ {1} - \hat {\mu} _ {2} \| ^ {2}\right) \\ \leqslant O \left(\| \hat {\mu} _ {1} - \hat {\mu} _ {2} \| ^ {4} + \| \hat {\mu} _ {1} - \hat {\mu} _ {2} \| ^ {2}\right). \\ \end{array}
$$
Hence,  $\| \hat{\mu}_1 - \hat{\mu}_2\|^2 \leqslant O(\sqrt{\varepsilon} \cdot \| \hat{\mu}_1 - \hat{\mu}_2\|^2 + \sqrt{\varepsilon} \cdot \| \hat{\mu}_1 - \hat{\mu}_2\|)$ . If  $\varepsilon$  is small enough (smaller than some constant that depends on the strong convexity parameter), we obtain that  $\| \hat{\mu}_1 - \hat{\mu}_2\| \leqslant O(\sqrt{\varepsilon})$ .
Similar to the bounded moment setting, we can generalize this observation to the case when for some integer  $k > 1$  and some  $m_{2k} > 0$ ,  $\max_{\| u \| = 1} \mathbb{E}_{\boldsymbol{y}_j \sim D_j} \langle \nabla \mathcal{L}(\boldsymbol{y} - \hat{\mu}_j), u \rangle^{2k} \leqslant m_{2k}^{2k}$ . In this case we can use Hölder's inequality to obtain the bound  $\| \hat{\mu}_1 - \hat{\mu}_2 \| \leqslant O(m_{2k} \cdot \varepsilon^{1 - 1/(2k)})$ . Note that in order to combine higher-order moment bounds with the filtering technique, we need an efficient procedure to certify these bounds. Thus, previous works only applied to distributions for which such moment certificates, in sum-of-squares, exist. We remark that in the symmetric setting we are able to obtain the necessary certificates for filtering without making any sum-of-squares related assumption on the distribution.
Filtering in the symmetric setting We will show how to adopt the filtering technique to make the above proof of identifiability algorithmic for symmetric distributions. Indeed, let  $D_{2}$  be a symmetric distribution with location  $\mu^{*}$ . Assume we can choose a loss function  $\mathcal{L}$  such that  $\hat{\mu}_{2} = \mu^{*}$  and  $\mathbb{E}_{\boldsymbol{y}\sim D_2}\nabla \mathcal{L}(\boldsymbol {y} - \hat{\mu}_2)\nabla \mathcal{L}(\boldsymbol {y} - \hat{\mu}_2)^{\top}\preceq \mathrm{Id}_d$ . Then, if  $D_{1}$  is an  $\varepsilon$ -corruption of  $D_{2}$ , we can iteratively compute  $\mathbb{E}_{\boldsymbol{y}\sim D_1}\nabla \mathcal{L}(\boldsymbol {y} - \hat{\mu}_1)\nabla \mathcal{L}(\boldsymbol {y} - \hat{\mu}_1)^{\top}$  and check its top eigenvalue. If it is smaller than 1, we know that  $\hat{\mu}_{1}$  must be  $O(\sqrt{\varepsilon})$ -close to  $\hat{\mu}_{2} = \mu^{*}$ , if not, we can remove points which are aligned with the top eigenvector. Similar to the classical filtering setting, we can argue that this removes more corrupted points than uncorrupted points. Thus, after at most  $O(\varepsilon n)$  iterations, we must have that  $\hat{\mu}_{1}$  is indeed  $O(\sqrt{\varepsilon})$ -close to  $\mu^{*}$ . A similar argument works for higher-order moments.
It remains to verify the assumptions we used on the loss function  $\mathcal{L}$  and its interaction with our distribution. First, we required that  $\mathcal{L}$  is smooth and strongly convex, which is very restrictive. Fortunately, we can relax the second assumption: For our argument it is enough if  $\mathcal{L}$  is locally strongly convex around the minimizer (and globally smooth). Second, we assumed that  $\mathbb{E}_{\boldsymbol{y} \sim D_2} \nabla \mathcal{L}(\boldsymbol{y} - \hat{\mu}_2) \nabla \mathcal{L}(\boldsymbol{y} - \hat{\mu}_j)^{\top} \preceq \mathrm{Id}_d$ .  $\nabla \mathcal{L}(\boldsymbol{y} - \hat{\mu}_2)$  can be seen as some transformation of the distribution, and our condition requires that the covariance of the transformed distribution is bounded. In general, this might not be satisfied even if the transformations are well-behaved (e.g., bounded
In this discussion we assume that the operators  $\mathbb{E}$  and  $\nabla$  commute. In the analyses of the algorithms we use empirical means, and in this case they commute by linearity of the gradient.
and Lipschitz). However, we show that for appropriate  $\mathcal{L}$ , it is indeed satisfied for elliptical and semi-product distributions. For the sake of exposition, we focus here only on elliptical distributions. For such distributions, we use the loss function  $\mathcal{L}_E(v) \coloneqq r^2 \cdot \Phi\left(\|v\| / r\right)$  for some  $r > 0$ . It is not hard to see that it is globally smooth and locally strongly convex (in some neighborhood of the minimizer). Further,  $\nabla \mathcal{L}_E(v)$  is a projection of  $v$  onto the ball of radius  $r$  (with center at zero). The covariance of such a projection is bounded by the covariance of the projection onto the sphere of radius  $r$ . Fortunately, spherical projections of elliptical distributions are well-behaved. In particular, they only depend on the scatter matrix  $\Sigma$ , and hence we can without loss of generality assume that the initial elliptical distribution was Gaussian. For which we can obtain a bound on the covariance of  $\nabla \mathcal{L}_E(\boldsymbol{y} - \hat{\mu}_2)$ .
Similarly, we show that under mild assumptions on the scatter matrix (that in particular are satisfied for scatter matrices with condition number  $O(1)$ ), we can certify tight bounds on the moments of  $\nabla \mathcal{L}_E(\pmb{y} - \hat{\mu}_2)$  in sum-of-squares, which leads to polynomial-time algorithms that obtain error  $o\left(\sqrt{\varepsilon}\right)$ . We remark that prior to this work, in the unknown covariance case, robust mean estimation with error  $o\left(\sqrt{\varepsilon}\right)$  was only possible under the assumption that higher-order moments of  $D_1$  are certifiably bounded in sum-of-squares. We show that error  $o\left(\sqrt{\varepsilon}\right)$  is possible for arbitrary elliptical distributions that might not even have a first moment. Moreover, we show that by exploiting  $O(\log(1/\varepsilon))$  many bounded moments of the transformed distributions, we achieve the near optimal error of  $O(\varepsilon \sqrt{\log(1/\varepsilon)})$  using quasi-polynomially many samples and time polynomial in the input.
The above approach also works for semi-product distributions by choosing  $\mathcal{L}$  to be the entrywise Huber-loss. This works both for the setting when  $\nabla \mathcal{L}(\pmb{y} - \hat{\mu}_2)$  has bounded covariance and certifiably bounded higher-order moments. We remark that in this case we can achieve the information theoretically optimal error  $O(\varepsilon)$  with quasi-polynomial number of samples in polynomial time (in the number of samples) – this again follows by exploiting  $O(\log(1 / \varepsilon))$  many moments.
2.1 Nearly optimal error for semi-product distributions using polynomially many samples
Note that even in the standard Gaussian case, using the standard filtering approach yields (nearly) optimal algorithms only when using quasi-polynomially many samples. Note that this seems somewhat inherent to the approach since it only uses low-degree moment information. To reduce this to polynomially many samples, we can use a stronger identifiability statement that exists for the standard Gaussian distribution. In particular, let  $D_{2} = N(\mu_{2},\mathrm{Id}_{d})$  and  $D_{1}$  be an  $\varepsilon$ -corruption of  $D_{1}$  with mean  $\mu_{1}$  and covariance  $\Sigma_{1}$ . Then, it holds that [DKK  $^{+}19$ , DKK  $^{+}17$ ] (see also [Li19])
$$
\left\| \mu_ {1} - \mu_ {2} \right\| \leqslant O \left(\varepsilon \sqrt {\log (1 / \varepsilon)} + \sqrt {\varepsilon \left(\left\| \Sigma_ {1} - \operatorname {I d} _ {d} \right\| + \varepsilon \log (1 / \varepsilon)\right)}\right).
$$
Thus, we can hope to achieve error  $O(\varepsilon \sqrt{\log(1 / \varepsilon)})$  by checking if  $\| \Sigma_1 - \mathrm{Id}_d\| \leqslant O(\varepsilon \sqrt{\log(1 / \varepsilon)})$  and iteratively removing points aligned with the top eigenvector of  $\Sigma_{1} - \mathrm{Id}_{d}$ . Indeed, a procedure very similar to this works (see e.g. [Li19]). However, it is crucial that we only remove points among the top  $\varepsilon$ -fraction of points correlated with the top eigenvector, since otherwise we cannot ensure that we remove more corrupted than uncorrupted points. The proof of the above uses stability conditions of the Gaussian distribution (cf. [DK22]). That is, the proof uses that for i.i.d.samplest  $\pmb{\eta}_1^*,\dots ,\pmb{\eta}_n^*\sim N(\mu^*,\mathrm{Id}_d)$  with  $n$  sufficiently large, it holds that for all subsets  $T$  of size  $(1 - 10\varepsilon)n$  we have
$$
\left\| \frac {1}{| T |} \sum_ {i \in T} \boldsymbol {\eta} ^ {*} \right\| \leqslant O \left(\varepsilon \sqrt {\log (1 / \varepsilon)}\right) \quad \text {a n d} \quad \left\| \frac {1}{| T |} \sum_ {i \in T} \boldsymbol {\eta} ^ {*} \boldsymbol {\eta} ^ {* ^ {\top}} - \operatorname {I d} _ {d} \right\| \leqslant O (\varepsilon \log (1 / \varepsilon)).
$$
We show that the above approach can be adapted to the semi-product setting. Indeed, let  $D_{2}$  be a semi-product distribution with location  $\mu^{*}$  and  $\Sigma_2^{\mathcal{L}} = \mathbb{E}_{\boldsymbol{y}\sim D_2}\nabla \mathcal{L}(\boldsymbol {y} - \mu^*)(\nabla \mathcal{L}(\boldsymbol {y} - \mu^*))^\top$ , where  $\mathcal{L}$  is the entry-wise Huber-loss. For simplicity assume that  $\Sigma_2^{\mathcal{L}} = \gamma \cdot \mathrm{Id}_d$  for some known  $\gamma$  (our approach extends to more general diagonal matrices and unknown  $\gamma$  as well - we show how to estimate all relevant parameters from the corrupted sample). Then, if  $D_{1}$  is an  $\varepsilon$ -corruption of  $D_{2}$ , we show that for  $\hat{\mu}_1\coloneqq \arg \min_{a\in \mathbb{R}^d}\mathbb{E}_{\boldsymbol {y}\sim D_1}\mathcal{L}(\boldsymbol {y} - a)$  it holds that