SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Finally, it is critical to ensure that the fraction of outliers in the simulated samples from  $G_{a}$  remains  $O(\epsilon)$ . In fact, this may not be true depending on the choice of the interval  $I$ . For example, if the adversary knows  $I$ , then outliers could all happen to have their labels inside  $I$ , which would cause the contamination rate to blow up. To overcome this, we randomize the selection of  $I$ : Since the distributions of outliers is independent of  $I$ , choosing the center of  $I$  to be random means that the bad event of the previous sentence is now a small probability event.
4.1 Proof Sketch of Theorem 1.4
The algorithm is given in Algorithm 2. We now give the proof sketch of Theorem 1.4: We claim that the following three events hold simultaneously with probability at least  $0.9(\widehat{\beta}_0,\widehat{\beta}_1$  and  $\widehat{\sigma}_y^2$  are defined in Algorithm 2):
(i)  $|\widehat{\sigma}_y^2 -\sigma_y^2 |\lesssim \sigma_y^2\epsilon \log (1 / \epsilon)$
(ii)  $|a| > 0.0001\widehat{\sigma}_y$
(iii)  $\| \widehat{\beta}_I - \mu_{G_I}\| _2\lesssim \epsilon$  (1)
Given the above, we first show that  $\widehat{\beta}$  is  $O(\sigma \epsilon)$ -close to  $\beta$ . For simplicity, let us assume momentarily that  $\widehat{\sigma}_y^2 = \sigma_y^2$ . Then by triangle inequality  $\| \widehat{\beta} - \beta \|_2 \leq \frac{\sigma_y^2}{|a|} (\| \widehat{\beta}_I - \mu_{G_I} \|_2 + \| \mu_{G_I} - \frac{a}{\sigma_y^2} \beta \|_2) \lesssim \sigma_y \epsilon + \sigma_y \| \mu_{G_I} - \frac{a}{\sigma_y^2} \beta \|_2$ . The second term arises from the fact that we use  $G_I$  instead of  $G_a$  in the algorithm (since simulating samples from  $G_a$  is algorithmically impossible). Using the expression for  $G_I$  from Claim 4.1, we can upper bound this term by  $\sigma_y \| \mu_{G_I} - \frac{a}{\sigma_y^2} \beta \|_2 \leq \frac{\ell \| \beta \|_2}{\sigma_y}$ . Since  $\| \beta \|_2 \lesssim \sigma \epsilon \log(1 / \epsilon)$  and  $\ell := \sigma_y / \log(1 / \epsilon)$ , this expression is overall  $O(\sigma \epsilon)$ .
Now, we show why the events in (1) above hold. The first is simply the guarantee of the trimmed mean on a subexponential distribution (cf. Appendix B) and the second holds because  $a \sim \mathcal{U}([- \hat{\sigma}_y, \hat{\sigma}_y])$ .
The third is significantly more involved. We first claim that Line 4 approximately preserves (within log factors) the size of the dataset and maintains the ratio of inliers to outliers (within constants). Recall that we use  $G$  for the inlier distribution and  $B$  for the distribution of outliers.
Lemma 4.2. Consider the context of Theorem 1.4 and Algorithm 2. First, for every possible choice of  $I$  that can be made in Line 3,  $\mathbf{Pr}_{(X,y)\sim G}[y\in I]\gtrsim \ell /\sigma_y$ . Second,  $\mathbf{E}_I\left[\mathbf{Pr}_{(X,y)\sim B}[y\in I|I]\right]\lesssim \ell /\sigma_y$ , where the outer expectation is taken with respect to the random choice of the center of  $I$ .
Proof Sketch. For inliers,  $y \sim \mathcal{N}(0, \sigma_y^2)$  and thus the fraction of inliers in  $S$  that belong to  $S'$  is at least  $\Omega(\ell / \sigma_y)$  since  $I \subset [-2\sigma_y, 2\sigma_y]$  with length  $\ell$ . For outliers, we may assume without loss of generality that  $y \in [-\sigma_y - \ell, \sigma_y + \ell]$  always (since otherwise  $y \notin I$ ). Since  $I$  is independent of  $(X, y)$ , we may treat  $(X, y)$  as fixed and only  $I$  as random. Thus, the probability of  $y \in I$  is at most the ratio of the length of  $I$  to the length of  $[- \sigma_y - \ell, \sigma_y + \ell]$ , which is at most  $O(\ell / \sigma_y)$  as claimed.
Observe that the set  $S' = \{(x,y) : (x,y) \in S, y \in I\}$  in Line 4 consists of i.i.d. points from  $(1 - \epsilon_I)G_I + \epsilon_IB_I$ , where  $G_I$  is the conditional distribution from Claim 4.1,  $B_I$  is the conditional distribution of the outliers, and  $\epsilon_I := \frac{\epsilon\operatorname*{Pr}_{(X,y)}\sim_B[y \in I]}{(1 - \epsilon)\operatorname*{Pr}_{(X,y)}\sim_D[y \in I] + \epsilon\operatorname*{Pr}_{(X,y)}\sim_B[y \in I} \leq \frac{\epsilon\operatorname*{Pr}_{(X,y)}\sim_B[y \in I]}{(1 - \epsilon)\operatorname*{Pr}_{(X,y)}\sim_D[y \in I]}$ . By Lemma 4.2,  $\mathbf{E}_I[\epsilon_I] = O(\epsilon)$  and thus by Markov's inequality, with high constant probability we will have that  $\epsilon_I = O(\epsilon)$ . Finally, it remains to show that  $S'$  satisfies the goodness conditions as dictated by Theorem 1.3. This requires technical effort and is deferred to Appendix D.2.
In this paper, we provided fast algorithms for mean estimation and linear regression that achieve optimal error under Huber contamination model for isotropic Gaussian inlier distributions. Several open problems and avenues for improvement remain. First, the sample complexity of our linear regression algorithm is multiplicative in  $\log(1/\delta)$ , where  $\delta$  is the failure probability, as opposed to additive in the information-theoretic sample complexity [CGR16]. More broadly, it is an open problem to design algorithms that achieve similar optimal guarantees for other fundamental tasks: robust principal component analysis, sparse mean estimation, and covariance estimation. In particular, the best known algorithm for the covariance estimation (or mean estimation with unknown covariance) achieving the optimal error runs in quasi-polynomial time [DKKLMS18].
Ilias Diakonikolas is supported by NSF Medium Award CCF-2107079, NSF Award CCF-1652862 (CAREER), and a DARPA Learning with Less Labels (LwLL) grant. Daniel Kane is supported by NSF Medium Award CCF-2107547 and NSF Award CCF-1553288 (CAREER). Ankit Pensia was supported by NSF Awards CCF-1652862, and CCF-1841190, and CCF-2011255; The majority of this work was done while Ankit Pensia was at UW-Madison. Thanasis Pittas is supported by NSF Medium Award CCF-2107079 and NSF Award DMS-2023239 (TRIPODS
[ABHHRT72] D. F. Andrews, P. J. Bickel, F. R. Hampel, P. J. Huber, W. H. Rogers, and J. W. Tukey. *Robust Estimates of Location: Survey and Advances*. Princeton, NJ, USA: Princeton University Press, 1972.
[BBHLT20] M. Brennan, G. Bresler, S. B. Hopkins, J. Li, and T.Schramm. "Statistical Query Algorithms and Low-Degree Tests Are Almost Equivalent". In: CoRR abs/2009.06107 (2020).
[BDLS17] S. Balakrishnan, S. S. Du, J. Li, and A. Singh. "Computationally Efficient Robust Sparse Estimation in High Dimensions". In: Proc. 30th Annual Conference on Learning Theory (COLT). Vol. 65. 2017, pp. 1-44.
[Ber18] D. S. Bernstein. Scalar, Vector, and Matrix Mathematics: Theory, Facts, and Formulas. Revised and expanded edition. Princeton: Princeton University Press, 2018.
[Bha13] R. Bhatia. Matrix analysis. Vol. 169. Springer Science & Business Media, 2013.
[BLM13] S. Boucheron, G. Lugosi, and P. Massart. Concentration Inequalities: A Nonasymptotic Theory of Independence. Oxford University Press, 2013.
[BP21] A. Bakshi and A. Prasad. "Robust Linear Regression: Optimal Rates in Polynomial Time". In: Proc. 53rd Annual ACM Symposium on Theory of Computing (STOC). ACM Press, 2021, pp. 102-115.
[CATJFB20] Y. Cherapanamjeri, E. Aras, N. Tripuraneni, M. I. Jordan, N. Flammarion, and P. L. Bartlett. "Optimal Robust Linear Regression in Nearly Linear Time". In: abs/2007.08137 (2020).
[CDG19] Y. Cheng, I. Diakonikolas, and R. Ge. "High-Dimensional Robust Mean Estimation in Nearly-Linear Time". In: Proc. 30th Annual Symposium on Discrete Algorithms (SODA). SIAM, 2019, pp. 2755-2771.
[CDGS20] Y. Cheng, I. Diakonikolas, R. Ge, and M. Soltanolkotabi. "High-Dimensional Robust Mean Estimation via Gradient Descent". In: Proc. 37th International Conference on Machine Learning (ICML). 2020.
[CDGW19] Y. Cheng, I. Diakonikolas, R. Ge, and D. P. Woodruff. "Faster Algorithms for High-Dimensional Robust Covariance Estimation". In: Proc. 32nd Annual Conference on Learning Theory (COLT). 2019.
[CDKGGS22] Y. Cheng, I. Diakonikolas, D. M. Kane, R. Ge, S. Gupta, and M. Soltanolkotabi. "Outlier-Robust Sparse Estimation via Non-Convex Optimization". In: Advances in Neural Information Processing Systems 35 (NeurIPS). 2022.
[CFB19] Y. Cherapanamjeri, N. Flammarion, and P. L. Bartlett. "Fast Mean Estimation with Sub-Gaussian Rates". In: Proc. 32nd Annual Conference on Learning Theory (COLT). 2019.
[CGR16] M. Chen, C. Gao, and Z. Ren. “A General Decision Theory for Huber's  $\mathbb{S}$ epsilon $\mathbb{S}$ -Contamination Model”. In: Electronic Journal of Statistics 10.2 (2016), pp. 3752-3774.
[CGR18] M. Chen, C. Gao, and Z. Ren. "Robust Covariance and Scatter Matrix Estimation under Huber's Contamination Model". In: The Annals of Statistics 46 (2018), pp. 1932-1960.
[CMY20] Y. Cherapanamjeri, S. Mohanty, and M. Yau. "List decodable mean estimation in nearly linear time". In: Proc. 61st IEEE Symposium on Foundations of Computer Science (FOCS). 2020.
A Additional Related Work 14
B Additional Preliminaries 15
B.1 Linear Algebraic Facts 15
B.2 Concentration of Measure Facts 15
B.3 Existing Robust Algorithms: Trimmed Mean and [DKPP22] 16
B.4 Goodness Condition and Its Sample Complexity 17
B.5 Filtering Procedure 22
B.6 Algorithm from [DKKLMS18] 23
C Omitted Proofs from Section 3 25
C.1 Deterministic Conditions for Algorithm 1 26
C.2 Case 1 26
C.3 Case 2 28
C.4 Combining Everything Together 31
D Omitted Proofs from Section 4 32
D.1 Conditional Distribution of Covariates 32
D.2 Robustness of Goodness Conditions to Spectral Noise 33
D.3 Proof of Theorem 1.4 34
E Adaptation of Algorithm 1 to the Streaming Setting 36
A systematic study of robust statistics was initiated in the 1960s ([Tuk60; Hub64; ABHHRT72]). We refer the reader to the book [HR09] for a classical treatment of this topic. The theoretical investigation in these studies often focused on the asymptotics of sample size going to infinity. A non-asymptotic analysis of the sample complexity can be found in, for example, [CGR18; CGR16; LM21].
Algorithmic high-dimensional robust statistics has been an active area of research since the publication of [DKKLMS16; LRV16]. Various high-dimensional inference tasks have been investigated in the literature: (i) Unsupervised learning tasks include mean estimation [DKKLMS16; LRV16; KSS18; CFB19; DKP20; HLZ20; CTBJ22; HR21], sparse estimation [BDLS17; DKKPP22; DKLP22; DKKPS19], covariance estimation [CDGW19; LY20], and PCA [XCM13; KSKO20; JLT20], and (ii) Supervised learning tasks include linear regression [DKS19; KKM18; CATJFB20; PJL20; Dep20; BP21], and general stochastic optimization tasks [PBR19; DKKLSS19].
Within algorithmic robust statistics, our work is related to a line of work that develops novel algorithmic approaches to robust inference. For example, approaches based on gradient-descent and matrix multiplicative weights update [CDGS20; ZJS22; HLZ20; CDKGGS22]. In particular, several recent works have developed fast algorithms, often nearly linear time as opposed to simply polynomial time algorithms, for robust inference [CDG19; CDGW19; LY20; ZJS22; CDKGGS22; DHL19; CMY20; DKKLT21; DKKLT22].
B Additional Preliminaries
In this section, we state the results that will be used in the proofs later. Appendices B.1 and B.2 contain facts pertaining to linear algebra and concentration of measure, respectively. Appendix B.3 records some results from both folklore and prior works in robust statistics that are needed in this paper. Appendix B.4 focuses on the goodness condition and proves the samples complexity and key properties of good sets. Appendix B.5 formally states the guarantee of the multi-directional filtering procedure. Finally, Appendix B.6 presents the improved runtime guarantee of the algorithm in [DKKLMS18].
B.1 Linear Algebraic Facts
Fact B.1. Let  $\mathbf{A}$  be a PSD matrix. Then for any unit vector  $x$  and  $m\geq 1$ ,  $x^{\top}\mathbf{A}^{m}x\geq (x^{\top}\mathbf{A}x)^{m}$ .
Fact B.2. If  $\mathbf{A} \in \mathbb{R}^{d \times d}$  is symmetric and  $p \geq 1$ , the Schatten norms of  $\mathbf{A}$  satisfy the following:  $\| \mathbf{A} \|_{p+1} \leq \| \mathbf{A} \|_p \leq \| \mathbf{A} \|_{p+1} d^{\frac{1}{p(p+1)}}$ .
In the following we let  $\mathbb{C}$  denote the set of complex numbers, though throughout the paper we will only work with real matrices.
Definition B.3 (Gershgorin Discs). For any complex  $n \times n$  matrix  $\mathbf{A}$ , for  $i \in [n]$ , let  $R_i'(\mathbf{A}) = \sum_{j \neq i} |a_{ij}|$  and let  $G(\mathbf{A}) = \bigcup_{i=1}^{n} \{z \in \mathbb{C} : |z - a_{ii}| \leq R_i'(\mathbf{A})\}$ . Each disc  $\{z \in \mathbb{C} : |z - a_{ii}| \leq R_i'(\mathbf{A})\}$  is called Gershgorin disc and their union  $G(\mathbf{A})$  is called the Gershgorin domain.
Fact B.4 (Gershgorin's Disc Theorem). For any complex  $n \times n$  matrix  $\mathbf{A}$ , all the eigenvalues of  $\mathbf{A}$  belong to the Gershgorin domain  $G(\mathbf{A})$ .
Fact B.5. If  $\mathbf{A},\mathbf{B},\mathbf{C}$  are symmetric  $d\times d$  matrices with  $\mathbf{A}\succeq 0$  and  $\mathbf{B}\preceq \mathbf{C}$  then  $\operatorname {tr}(\mathbf{AB})\leq \operatorname {tr}(\mathbf{AC})$
The following two linear algebra results follow from the Lieb-Thirring inequality.
Fact B.6 (Lemma 7 in [JLT20]). Let  $\mathbf{A},\mathbf{B}$  be positive semidefinite matrices with  $\mathbf{B} \preceq \mathbf{A}$  and  $p \in \mathbb{N}$ . Then,  $\operatorname{tr}(\mathbf{B}^p) \leq \operatorname{tr}(\mathbf{A}^{p-1}\mathbf{B})$ .
Fact B.7 (Lieb-Thirring Inequality; see, e.g., Problem IX.8.1 in [Bha13]). Let  $\mathbf{A},\mathbf{B}$  be positive semidefinite matrices. For all  $k\in \mathbb{N}$ ,  $\mathrm{tr}((\mathbf{AB})^k)\leq \mathrm{tr}(\mathbf{A}^k\mathbf{B}^k)$ .
Fact B.8. Let  $\mathbf{A}$  be a PSD matrix. Then,  $\| \mathbf{A}\|_{\mathrm{F}}\leq \sqrt{\|\mathbf{A}\|_{\mathrm{op}}\|\mathbf{A}\|_{1}}$  . In particular, if  $\| \mathbf{A}\|_{\mathrm{op}}\leq \| \mathbf{A}\| _1 / t,$  then  $\| \mathbf{A}\|_{\mathrm{F}}^{2}\leq \| \mathbf{A}\|_{1}^{2} / t.$
Fact B.9 (See, e.g., [Ber18, Proposition 11.10.34]). Let  $\mathbf{R}$  and  $\mathbf{M}$  be two square matrices, then  $\| \mathbf{RM} \|_{\mathrm{F}} \leq \min \left( \left( \| \mathbf{R} \|_{\mathrm{F}} \| \mathbf{M} \|_{\mathrm{op}}, \| \mathbf{R} \|_{\mathrm{op}} \| \mathbf{M} \|_{\mathrm{F}} \right) \right.$  and  $\| \mathbf{RM} \|_{\mathrm{op}} \leq \| \mathbf{R} \|_{\mathrm{op}} \| \mathbf{M} \|_{\mathrm{op}}$ .
B.2 Concentration of Measure Facts
Fact B.10 (Power iteration). For any positive semidefinite matrix  $\mathbf{A} \in \mathbb{R}^{d \times d}$  and  $\eta, \delta \in (0,1)$ , if  $p > \frac{C}{\eta} \log(d / (\eta \delta))$  for a sufficiently large constant  $C$ , and  $u := \mathbf{A}^p z$  for  $z \sim \mathcal{N}(0,\mathbf{I})$ , then
$$
\Pr \left[ u ^ {\top} \mathbf {A} u / \| u \| _ {2} ^ {2} \geq (1 - \eta) \| \mathbf {A} \| _ {\mathrm {o p}} \right] \geq 1 - \delta .
$$
Definition B.11 (Sub-Gaussian and Sub-exponential Random Variables). A one-dimensional random variable  $Y$  is sub-Gaussian if  $\| Y\|_{\psi_2} \coloneqq \sup_{p\geq 1}p^{-1 / 2}\mathbf{E}[|Y|^p]$  is finite. We say that  $\| Y\|_{\psi_2}$  is the sub-Gaussian norm of  $Y$ . A random vector  $X$  in  $\mathbb{R}^d$  is sub-Gaussian if for every  $v \in S^{d - 1}$ ,  $\left\| v^\top X\right\|_{\psi_2}$  is finite. The sub-Gaussian norm of the vector is defined to be
$$
\left\| X \right\| _ {\psi_ {2}} := \sup  _ {v \in \mathcal {S} ^ {d - 1}} \left\| v ^ {\top} X \right\| _ {\psi_ {2}}.
$$
We call a centered one-dimensional random variable  $Y$  a  $(\nu, \alpha)_+$  sub-exponential if  $\mathbf{E}[\exp(\lambda Y)] \leq \nu^2 \lambda^2 / 2$  for all  $\lambda \leq 1 / \alpha$ . We call  $\|Y\|_{\psi_1} := \sup_{p \geq 1} p^{-1} \mathbf{E}[|Y|^p]$  the sub-exponential norm of  $Y$ . We extend our definition to vectors as before.
Lemma B.12 (Properties of Sub-exponential Random Variables [Wai19; BLM13]). The class of sub-exponential random variables satisfy the following:
1. [Wai19, Proposition 2.9] If  $Y$  is a centered  $(\nu, \alpha)_+$  sub-exponential random variable, then with probability  $1 - \delta$ ,  $Y \lesssim \nu \sqrt{\log(1 / \delta)} + \alpha \log(1 / \delta)$ .
2. [BLM13, Theorem 2.3] If  $Y$  is a centered random variable satisfying that for all  $\delta \in (0,1)$ ,  $Y \leq \nu \sqrt{\log(1 / \delta)} + \alpha \log(1 / \delta)$ , then  $Y$  is  $(\nu', \alpha')_+$  sub-exponential with  $\nu' \lesssim \nu + \alpha$  and  $\alpha' \lesssim \alpha$ .
3. [Wai19, Section 2.1.3] Let  $Y_{1}, \ldots, Y_{k}$  be  $k$  centered independent  $(\nu, \alpha)_{+}$  sub-exponential random variables. Then  $\sum_{i=1}^{k} Y_{i}$  is a  $(\nu \sqrt{k}, \alpha)_{+}$  sub-exponential random variable.
Fact B.13 (Hanson-Wright Inequality). Let  $X \sim \mathcal{N}(0, \mathbf{I})$  Then, for every  $t \geq 0$ :
$$
\mathbf {P r} [ | X ^ {\top} \mathbf {A} X - \mathbf {E} [ X ^ {\top} \mathbf {A} X ] | > t ] \leq 2 \exp \left(- 0. 1 \min \left(\frac {t ^ {2}}{K ^ {4} \| \mathbf {A} \| _ {\mathrm {F}} ^ {2}}, \frac {t}{K ^ {2} \| \mathbf {A} \| _ {\mathrm {o p}}}\right)\right).
$$
Fact B.14 (Johnson-Lindenstrauss Sketch). Fix a set of  $n$  points  $x_{1},\ldots ,x_{n}\in \mathbb{R}^{d}$ . Let  $g(x)\coloneqq \|\mathbf{A}(x - b)\|_2^2$  be a polynomial for some  $\mathbf{A}\in \mathbb{R}^{d\times d}$  and  $b\in \mathbb{R}^d$ , and let  $\mathbf{U}$  be the (random) matrix having as rows the vectors  $u_{i} = \mathbf{A}z_{i} / \sqrt{k}$  for  $z_{i}\sim \mathcal{N}(0,\mathbf{I}_{d})$ ,  $i = 1,\dots ,k$ . Define  $\tilde{g} (x)\coloneqq \|\mathbf{U}(x - b)\|_2^2$ . If  $C$  is a sufficiently large constant and  $k > C\log ((n + d) / \delta)$ , then with probability at least  $1 - \delta$ , the following holds:
1.  $0.8g(x_{i})\leq \tilde{g} (x_{i})\leq 1.2g(x_{i})$  for every  $i\in [n]$
2.  $0.8\| \mathbf{A}\|_{\mathrm{F}}^{2}\leq \| \mathbf{U}\|_{\mathrm{F}}^{2}\leq 1.2\| \mathbf{A}\|_{\mathrm{F}}^{2}.$
Fact B.15. For any  $d \times d$  symmetric matrix  $\mathbf{A}$ , we have that  $\mathrm{Var}_{z,z' \sim \mathcal{N}(0,\mathbf{I}_d)}[\langle \mathbf{A}z, \mathbf{A}z' \rangle^2] = \| \mathbf{A}^2\|_{\mathrm{F}}^2$ . If  $\mathbf{A}$  is a PSD matrix, then for any  $\beta > 0$ , it holds  $\mathbf{Pr}_{z \sim \mathcal{N}(0,\mathbf{I})}[z^\top \mathbf{A}z \geq \beta \mathrm{tr}(\mathbf{A})] \geq 1 - \sqrt{e\beta}$ .
Fact B.16 (VC inequality). Let  $\mathcal{F}$  be a class of Boolean functions with finite VC dimension  $\mathrm{VC}(\mathcal{F})$  and let a probability distribution  $D$  over the domain of these functions. For a set  $S$  of  $n$  independent samples from  $D$
$$
\sup  _ {f \in \mathcal {F}} \left| \mathbf {P r} _ {X \sim S} [ f (X) ] - \mathbf {P r} _ {X \sim D} [ f (X) ] \right| \lesssim \sqrt {\frac {\operatorname {V C} (\mathcal {F})}{n}} + \sqrt {\frac {\log (1 / \tau)}{n}},
$$
with probability at least  $1 - \tau$ .
B.3 Existing Robust Algorithms: Trimmed Mean and [DKPP22]
In this section, we state the guarantees of the trimmed mean and the fast robust mean estimation algorithm from [DKPP22].
We begin with the guarantee of the trimmed mean. The following holds even in the strong contamination model where an adversary can edit  $\epsilon$ -fraction of the samples arbitrarily. We will need it to estimate the (unknown) variance of the labels in our linear regression algorithm.
Fact B.17 (Univariate Trimmed Mean, see, e.g., [LM21]). Let  $\epsilon_0$  be a sufficiently small absolute constant. There is an algorithm (trimmed mean) that, for every  $\epsilon \in (0,\epsilon_0)$  and a univariate distribution  $D$  that has  $\psi_{1}$ -norm at most  $\sigma^2$  (cf. Definition B.11), given a set of  $n\gg \log (1 / \delta) / (\epsilon \log^2 (1 / \epsilon))$  samples from  $D$  with corruption at rate  $\epsilon$ , outputs a  $\widehat{\mu}$  such that  $|\widehat{\mu} -\mathbf{E}_{X\sim D}[X]|\lesssim \sigma^2\epsilon \log (1 / \epsilon)$  with probability at least  $1 - \delta$ .
The following result is implicit in [DKPP22]. It will serve as a preprocessing step to ensure small operator norm of the covariance matrix in our robust mean estimation algorithm.
Lemma B.18. There exists an algorithm for which the following is true. If  $S$  is a set of  $n$  points in  $\mathbb{R}^d$  containing a  $G \subset S$  such that the uniform distribution on  $G$  satisfies conditions (2.a) and (2.b) of Definition 2.1 with  $\alpha = \epsilon / \log(1/\epsilon)$ , then the algorithm runs on input  $S$  and  $\epsilon$ , outputs weights  $w(x) \in [0,1]$  for each  $x \in S$  such that the following hold with probability at least 0.999:
1.  $\frac{1}{|G|}\sum_{x\in G}w(x)\geq 1 - \frac{\epsilon}{\log(1 / \epsilon)}.$
2. Let  $Q$  be the discrete distribution on  $S$  that assigns probability mass  $w(x) / \sum_{x \in S} w(x)$  to each  $x \in S$ . Then, the top eigenvalue of the covariance matrix of  $Q$  is at most  $1 + O(\epsilon \log^2(1/\epsilon))$ .
Proof. The algorithm that achieves the guarantee is a slightly modified version of Algorithm 1 in [DKPP22] where: (i) instead of outputting an estimate of the mean, the algorithm outputs the underlying weights that it has been updating and (ii) the "DownweightingFilter" procedure is replaced by Algorithm 3 with  $\beta = \log(1/\epsilon)$ , i.e., the only difference is on the threshold that is used to decide when to stop filtering. Algorithm 3 increases the threshold by the factor  $\beta = \log(1/\epsilon)$ , which as we show in Lemma B.27 results in removing  $\log(1/\epsilon)$  times more mass from the outliers than inliers. The increase in the filtering threshold translates to having an extra  $\log(1/\epsilon)$  factor in Item 2 (without this increased threshold, the variance along every direction would have been  $\epsilon \log(1/\epsilon)$ ), and the fact that this filter now removes  $\log(1/\epsilon)$  times more mass from the outliers than inliers translates to having  $\epsilon$  divided by  $\log(1/\epsilon)$  in Item 1.
B.4 Goodness Condition and Its Sample Complexity
We first restate the definition of the conditions. Recall that for a weight function  $w: \mathbb{R}^d \to [0,1]$  we let  $G_w$  denote weighted by  $w$  version of the distribution  $G$ . The matrix  $\overline{\Sigma}_{G_w}$  below denotes  $\mathbf{E}_{X \sim G_w}[(X - \mu)(X - \mu)^\top]$  (as opposed to the covariance matrix  $\Sigma_{G_w} := \mathbf{E}_{X \sim G_w}[(X - \mu_{G_w})(X - \mu_{G_w})^\top]$ ).
Definition 2.1 ( $(\epsilon, \alpha, k)$ -Goodness). We say that a distribution  $G$  on  $\mathbb{R}^d$  is  $(\epsilon, \alpha, k)$ -good with respect to  $\mu \in \mathbb{R}^d$ , if the following conditions are satisfied:
(1) (Median) For all  $v \in S^{d-1}$ ,  $\mathbf{Pr}_{X \sim G}[|v^\top(X - \mu)| \gtrsim \epsilon] < 1/2$ .
(2) For every weight function  $w$  with  $\mathbf{E}_{X\sim G}[w(X)]\geq 1 - \alpha$  , the following hold:
Remark B.19. We note that once the conditions of Definition 2.1 are established, then they also hold for projections of the data on a subspace, something that we will need later on in our proofs. Concretely, let  $\mathcal{V}$  be any subspace of  $\mathbb{R}^d$ . Then Conditions (1), (2.a) and (2.b) continue to hold for the projected points  $\mathrm{Proj}_{\mathcal{V}}(x)$ . This is because all of these conditions concern properties of one-dimensional projections of the data. Regarding the third condition, if the rows of  $\mathbf{U}$  belong in the subspace  $\mathcal{V}$  (which will be the case in our analysis later on), the last condition also continues to hold for the points  $\mathrm{Proj}_{\mathcal{V}}(x)$  since the  $\| \mathbf{U} \mathrm{Proj}_{\mathcal{V}}(x) \|_2^2 = \| \mathbf{U} \Pi_{\mathcal{V}} x \|_2^2 = \| \mathbf{U} x \|_2^2$ , where we used that  $\mathbf{U} \Pi_{\mathcal{V}} = \mathbf{U}$  because the rows of  $\mathbf{U}$  already live in the space  $\mathcal{V}$ .
B.4.1 Sample Complexity
In our results, we use the goodness condition from Definition 2.1 with parameters  $\alpha = \epsilon / \log(1/\epsilon)$  and  $k = \mathrm{polylog}(d/\epsilon)$ . The following result implies that a set of  $\frac{(d + \log(1/\tau))\mathrm{polylog}(d/\epsilon)}{\epsilon^2}$  i.i.d. samples from the inliers satisfies the desired stability condition.