SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The condition on  $\Sigma_{P_w}$  in Setting 2.3 amounts to a "warm start" for our algorithm, which can be obtained in nearly-linear time by running the algorithm from [DKPP22] first (see Appendix B).
The following result gives a sufficient condition for the (weighted) sample mean using (2.a) and (2.b):
Lemma 2.4 (Certificate Lemma). In Setting 2.3, if  $\| \Sigma_{P_w}\|_{\mathrm{op}}\leq 1 + \lambda$  then  $\| \mu_{P_w} - \mu \| _2\lesssim \epsilon +\sqrt{\lambda\epsilon}$
In light of the certificate lemma, our goal will be to downweigh outliers until the top eigenvalue (in the subspace of interest) is at most  $1 + O(\epsilon)$ . That would make the bound above  $O(\epsilon)$ .
We now explain the filtering procedure that we use. A filtering algorithm takes weights  $w(x)$  for each point and scores  $\tau(x)$  (that capture how much of an outlier the point is) the procedure updates the weights to  $w'(x)$  such that: (i) it removes much more mass from outliers than inliers, and (ii)  $\epsilon \mathbf{E}_{X \sim B}[w'(x)\tau(x)]$  (the contribution to the weighted average of scores by the outliers) is small after filtering. By considering appropriate scores  $\tau(x)$  that use multi-directional projections of the form of Condition (2.c) of Definition 2.1, we show the following in Appendix B.
Lemma 2.5 (Multi-Directional Filtering). Consider Setting 2.3. Given a nearly-orthogonal matrix  $\mathbf{U} \in \mathbb{R}^{k \times d}$  satisfying  $\| \mathbf{U}^{\top} \mathbf{U} \|_{\mathrm{op}} \leq 2 \operatorname{tr}(\mathbf{U}^{\top} \mathbf{U}) / \log(1 / \epsilon)$ , there is an algorithm that reads  $\epsilon$ , the  $n$  points, their weights  $w(x)$ , and returns weights  $w'$  in time  $ndk + \mathrm{polylog}(d / \epsilon, \| \mathbf{U} \|_{\mathrm{F}}^{2})$  such that
(i)  $\mathbf{E}_{X\sim G}[w(X) - w'(X)] < (\epsilon / \log(1/\epsilon)) \mathbf{E}_{X\sim B}[w(X) - w'(X)]$ .
(ii)  $\epsilon \mathbf{E}_{X\sim B}[w^{\prime}(X)\| \mathbf{U}(X - \mu_{P_w})\| _2^2 ]\lesssim \epsilon \mathrm{tr}(\mathbf{U}^\top \mathbf{U}).$
2.2 Polynomial Time Algorithm
We now record a (refined) guarantee of the algorithm from [DKKLMS18], which would be useful later on. This algorithm uses the multi-directional filter from Lemma B.27 with  $\mathbf{U}$  set to be the top- $k$  eigenvectors of the covariance matrix until the top- $k$  eigenvalues are  $1 + O(\epsilon)$  for  $k = \Theta (\log (1 / \epsilon))$ . Lemma 2.4 then guarantees that the empirical mean in the subspace of the  $(d - k)$  last eigenvalues is  $O(\epsilon)$ -close to  $\mu$ . Finally, the algorithm employs a brute force procedure that uses median and (Condition (1)) to estimate the mean in the remaining  $k$ -dimensional subspace.
Lemma 2.6 (Adapted From [DKKLMS18]). Let  $T$  be a set of  $n$  i.i.d. samples from an  $\epsilon$ -corrupted version of  $\mathcal{N}(0, \mathbf{I}_d)$  and let  $r, \delta \in (0, 1)$  be parameters. If  $n \gg (d + \log(1/\delta)) / \epsilon^2)$  polylog  $(d / \epsilon)$ , then there is an algorithm that, when having as input  $T$ ,  $\epsilon$ , and  $r$ , after time  $\tilde{O}(nd^2 + \frac{1}{\epsilon^{2+r}})$ , it outputs a vector  $\hat{\mu} \in \mathbb{R}^d$  such that  $\|\hat{\mu} - \mu\|_2 \lesssim \epsilon / r$  with probability  $1 - \delta$ .
Since the refined runtime above does not appear in [DKKLMS18], we provide a proof in Appendix B. In particular, it is important that the runtime is  $nd^2 + \frac{1}{\epsilon^{2 + \tau}}$ . Even though this is super-linear in the size of the input,  $nd$ , our main algorithm in the next section will apply Lemma 2.6 only after projecting the entire dataset to a subspace of a much smaller dimension (polylog(d/ε) in place of  $d$  before).
3 Almost-Linear Time Algorithm for Robust Mean Estimation
In this section, we describe the main ideas of the almost-linear time algorithm for Gaussian robust mean estimation that achieves Theorem 1.3.
As described earlier, our algorithm runs in two stages, both of which will run in almost-linear time. In the first stage, our goal is to find a low-dimensional subspace  $\mathcal{V}$  (of dimension at most  $\mathrm{polylog}(d / \epsilon)$ ) and a vector that is  $O(\epsilon)$  close to  $\mu$  when projected in subspace  $\mathcal{V}^{\perp}$ . In the second stage, we deploy the basic  $O(\epsilon)$  algorithm from [DKKLMS18] on the input data after projecting it onto  $\mathcal{V}$  and estimate  $\mu$  in that subspace. By a refined analysis of their argument (Lemma 2.6), the second stage runs fast because  $\dim(V) = O(\mathrm{polylog}(d / \epsilon))$ . By combining these two estimates we get an approximation of  $\mu$  in the whole  $\mathbb{R}^d$ . Since the second stage algorithm's analysis closely follows [DKKLMS18], in the reminder we focus on the first stage, below: (the probability of success can be boosted by standard arguments)
Theorem 3.1 (First Stage in Almost-Linear Time). Given a set of  $n$  samples, the uniform distribution on which has the form  $(1 - \epsilon)G + \epsilon B$  for some  $G$  satisfying Conditions (2.a) to (2.c) of Definition 2.1 with respect to  $\mu \in \mathbb{R}^d$  and parameters  $\alpha = \epsilon / \log(1/\epsilon)$  and  $k = \Theta(\log^2(n + d))$ . Algorithm 1 takes as input the  $n$  points and  $\epsilon$ , and with probability 0.99, returns a vector  $\hat{\mu}$  and a subspace  $\mathcal{V}$  such that  $\| \mathrm{Proj}_{\mathcal{V}^\perp}(\hat{\mu} - \mu) \|_2 \lesssim \epsilon$  and  $\dim(\mathcal{V}) = \mathrm{polylog}(d/\epsilon)$ . The algorithm runs in time  $O(nd\mathrm{polylog}(d/\epsilon))$ .
Theorem 3.1 is realized by Algorithm 1. Without loss of generality, we assume Setting 2.3 holds at the beginning of the algorithm. Recall the high-level strategy that was outlined in Section 1.2: We want to iteratively downweigh points and add directions to  $\mathcal{V}$  so that (i) the weight removed
from inliers is at most  $O(\epsilon / \log(1/\epsilon))$ , (ii) the downweighted dataset along every direction in  $\mathcal{V}^{\perp}$  has variance at most  $1 + O(\epsilon)$ , (iii) and  $\dim(\mathcal{V}) \leq \mathrm{polylog}(d/\epsilon)$ . Having the first two, the certificate lemma (Lemma 2.4) implies that the empirical (weighted) mean is  $O(\epsilon)$ -close to  $\mu$  along  $\mathcal{V}^{\perp}$ . The way to achieve (i) and (ii) in [DKKLMS18] was by using the matrix  $\mathbf{U}$  to be the top-  $k$  eigenvectors of the covariance matrix (also, the algorithm of [DKKLMS18] does not add directions to  $\mathcal{V}$  until the very end; see Section 2.2)). As mentioned earlier, this approach runs in quadratic time. Our main technical insight is to (a) randomize the choice of  $\mathbf{U}$  when safe to do so, and (b) when it is not safe, allow the algorithm to remove a direction by adding it to  $\mathcal{V}$  at any time (as opposed to waiting until the end). We first describe the notation below for a more detailed overview.
Notation for Algorithm 1. For each round  $t \in [t_{\mathrm{max}}]$  of our algorithm, we maintain weights  $w_t: \mathbb{R}^d \to [0,1]$  over the dataset, capturing the confidence in the points being inliers, i.e.,  $w(x) = 0$  represents outliers and  $w(x) = 1$  represents inliers. We also maintain a (low-dimensional) subspace  $\mathcal{V}_t$  with the goal of making the covariance of the projections of the data on  $\mathcal{V}_t^\perp$  be small at the end. Let  $\mu_t^\perp, \Sigma_t^\perp$  be the sample mean and covariance of the data after projected on  $\mathcal{V}_t^\perp$  and weighted by  $w_t$ , and define  $\mathbf{B}_t^\perp \approx \boldsymbol{\Sigma}_t^\perp - \boldsymbol{\Pi}_{\mathcal{V}_t^\perp}$  (see Lines 5 to 7 and 4 for precise definitions). We use the potential function  $\phi_t := \operatorname{tr}((\mathbf{M}_t^\perp)^\top (\mathbf{M}_t^\perp)) = \| \mathbf{M}_t^\perp \|_F^2$  to track the progress of our algorithm, where  $\mathbf{M}_t^\perp = (\mathbf{B}_t^\perp)^p$  for  $p = \log d$ . Observe that the potential function  $\phi_t$  ignores the contribution from the directions in  $\mathcal{V}_t$ .
Algorithm 1 Robust Mean Estimation Under Huber Contamination (Stage 1)
Input: Parameter  $\epsilon \in (0,1 / 2)$ , uniform distribution over  $n$  points that can be written as  $P = (1 - \epsilon)G + \epsilon B$  where  $G$  satisfies Definition 2.1 with appropriate parameters. Output: An approximation of the mean in a subspace  $\mathcal{V}^{\perp}$  and the orthogonal subspace  $\mathcal{V}$ .  
1: Let  $C$  be a sufficiently large constant,  $k = C\log^{2}(n + d)$ ,  $t_{\max} = (\log (d / \epsilon))^{C}$ .  
2: Initialize  $V_{1} \gets \emptyset$  and  $w_{1}(x) = 1$  for all  $x \in \mathbb{R}^{d}$ .  
3: for  $t = 1, \ldots, t_{\max}$  do  
4: Let  $\mathcal{V}_{t}$  be the subspace spanned by the vectors in  $V_{t}$ , and  $\mathcal{V}_{t}^{\perp}$  be the perpendicular subspace.  
5: Let  $P_{t}$  be the distribution  $P$  re-weighted by  $w_{t}$ , i.e.,  $P_{t}(x) = w_{t}(x)P(x) / \mathbf{E}_{X \sim P}[w(X)]$ .  
6: Let  $\mu_{t}^{\perp}, \Sigma_{t}^{\perp}$  be the mean and covariance of  $\mathrm{Proj}_{\mathcal{V}_{t}}(X)$  when  $X \sim P_{t}$   
7: Define  $\mathbf{B}_{t}^{\perp} = (\mathbf{E}_{X \sim P}[w(X)])^{2}\boldsymbol{\Sigma}_{t}^{\perp} - (1 - C_{1}\epsilon)\boldsymbol{\Pi}_{\mathcal{V}_{t}^{\perp}}$ , where  $\boldsymbol{\Pi}_{\mathcal{V}_{t}^{\perp}}$  is the orthogonal projection matrix for  $\mathcal{V}_{t}^{\perp}$ , and  $\mathbf{M}_{t}^{\perp} := (\mathbf{B}_{t}^{\perp})^{p}$  for  $p = \log(d)$ .  
8: Calculate  $\widehat{\lambda}_{t}$  such that  $\widehat{\lambda}_{t} / \| \mathbf{B}_{t}^{\perp} \|_{\mathrm{op}} \in [0.1 / 10]$  using power iteration. ▷ cf. Appendix B  
9: If  $\widehat{\lambda}_{t} \leq C\epsilon$  then return  $\mu_{t}$  and  $V_{t}$ .  
10: Let  $q_{t} := \mathbf{Pr}_{z,z' \sim \mathcal{N}(0,\mathbf{I})}[\|\mathbf{M}_{t}^{\perp}z, \mathbf{M}_{t}^{\perp}z'\|] > \| \mathbf{M}_{t}^{\perp}z\|_{2}\|\mathbf{M}_{t}^{\perp}z'\|_{2}/k^{2}]$ .  
11: Calculate an estimate  $\widehat{q}_{t}$  such that  $|\widehat{q}_{t} - q_{t}| \leq \frac{1}{10(k^{2}t_{\max})}$ .  
12: if  $\widehat{q}_{t} \leq 1/(k^{2}t_{\max})$  then ▷ Case 1 (cf. Section 3.1)  
13: for  $j \in [k]$  do  
14:  $v_{t,j} \gets \mathbf{M}_{t}^{\perp}z_{t,j}$  for  $z_{t,j} \sim \mathcal{N}(0,\mathbf{I})$ .  
15:  $\mathbf{U}_{t} \gets [v_{t,1},\dots,v_{t,k}]^{\top}$  i.e., the matrix with rows  $v_{t,j}$  for  $j \in [k]$ .  
16:  $w_{t+1} \gets \text{MULTI-DIRECTIONALFILTER}(P,w,\epsilon,\mathbf{U}_{t})$  ▷ cf. Lemma B.27  
17: else ▷ Case 2 (cf. Section 3.2)  
18:  $u_{t} \gets (\mathbf{B}_{t}^{\perp})^{p'}z / \|(\mathbf{B}_{t}^{\perp})^{p'}z\|_{2}$  for  $p' := C\log^{2}(dt_{\max}), z \sim \mathcal{N}(0,\mathbf{I})$ . ▷ Power iteration  
19:  $V_{t+1} \gets V_{t} \cup \{u_{t}\}$ .  
20: Let  $\mu_{\mathcal{V}_t} = \mathbf{E}_X \sim P_t[\text{Proj}_{\mathcal{V}_t}(X)]$  be the mean of  $P_t$  after projection to  $\mathcal{V}_t$ .  
21: return  $\mu_{\mathcal{V}_t}$  and  $V_{t}$ .
We will show that in every iteration, the potential function decreases multiplicatively, i.e.,  $\phi_{t + 1}\leq (1 - \mathrm{polylog}(\epsilon /d))\phi_t$  while removing at most  $O(\epsilon /\log (1 / \epsilon))$  fraction of inliers throughout the algorithm (so that we do not fall outside of Setting 2.3). Since  $\phi_t$  at  $t = 0$  is at most poly  $((d / \epsilon)^p)$  and the algorithm necessarily terminates when it reaches below  $\epsilon^p$  (because this implies that  $\| \mathbf{B}_t^\perp \|_{\mathrm{op}}\leq (\phi_t)^{1 / p} = O(\epsilon)$  which would cause Line 9 to activate), then after  $t_\mathrm{max} = \mathrm{polylog}(d / \epsilon)$  iterations the algorithm yields a  $\mu_{P_w}$  such that  $\| \mathrm{Proj}_{\mathcal{V}^{\perp}}(\mu_{P_w} - \mu)\| _2 = O(\epsilon)$  by Lemma 2.4. Since each iteration is implementable in  $\tilde{O} (nd)$  time, the whole algorithm terminates in  $\tilde{O} (nd)$  time.
In the next two subsections, we explain how the algorithm decides whether to expand the subspace  $\mathcal{V}_t$  or to remove outliers. This decision is based on Line 12, which checks if two random  $\mathbf{M}_t^\perp z$ ,  $\mathbf{M}_t^\perp z'$  for  $z, z' \sim \mathcal{N}(0, \mathbf{I})$  are nearly-orthogonal with reasonable probability.
3.1 Case 1: Many Large Eigenvalues
By construction, "Case 1" corresponds to the case where the rows of  $\mathbf{U}_t$  are nearly-orthogonal (with high probability); see Line 12. Thus,  $\mathbf{U}_t$  will be nearly-orthogonal, permitting the use of the multi-directional filtering algorithm from Lemma B.27.
The explanation above ensures that the multi-directional filtering procedure with random  $v_{i}$ 's is correct. The reason that it is significantly faster than [DKKLMS18] is that the  $v_{i}$ 's are now randomized along the top eigenvalues of  $\mathbf{B}_t^\perp$  because they are of the form  $\mathbf{M}_t z / \| \mathbf{M}_t z \|$ ; In contrast, [DKKLMS18] sets  $v_{i}$ 's deterministically equal to the top-  $k$  eigenvalues of  $\mathbf{B}_t^\perp$ . As outlined in the introduction, the random choice of  $v_{i}$ 's prevents the "adversary" to place the outliers in such a way that the outliers in the orthogonal subspace are unaffected during filtering. In particular, filtering with the random  $v_{i}$ 's reduces the contribution of outliers, not only along the exact top-  $k$  eigenvectors of  $\mathbf{B}_t^\perp$ , but also along all directions with variance comparable to the top eigenvector. We use the technical insights from [DKPP22] to show that the potential function decreases multiplicatively:
Claim 3.2. With high constant probability, for every round  $t$  that Line 12 succeeds,  $\phi_{t+1} \leq 0.99\phi_t$ . Moreover,  $\mathbf{E}_{X \sim G}[w_t(X)] \geq 1 - \epsilon / \log(1/\epsilon)$  throughout the algorithm's execution.
Proof Sketch. We first sketch how  $\mathbf{U}_t$  is valid for the multi-directional filter, Lemma B.27. Since the check of Line 12 succeeds, with high probability, we have that for each  $t\in [t_{\mathrm{max}}]$  rounds, the angles of every pair of rows of  $\mathbf{U}_t$  formed in line 15 have cosine at most  $1 / k^2$ . Also, Hanson-Wright inequality implies that with high probability,  $\| v_{t,j}\| _2^2\lesssim \mathrm{tr}(\mathbf{U}_t^\top \mathbf{U}_t)\log (k)$ . Combining both of these with Gershgorin Discs Theorem and the choice of  $k$ , we have that  $\| \mathbf{U}_t^\top \mathbf{U}_t\|_{\mathrm{op}}\leq 2\mathrm{tr}(\mathbf{U}_t^\top \mathbf{U}_t) / \log (1 / \epsilon)$ , satisfying the requirements in Lemma B.27 and ensuring  $\mathbf{E}_{X\sim G}[w_{t + 1}(X)]\geq 1 - \epsilon /\log (1 / \epsilon)$ .
To show that  $\phi_{t + 1}$  reduces, we first use the following linear-algebraic result to relate  $\phi_{t + 1}$  with  $\mathrm{tr}(\mathbf{M}_t^\perp \mathbf{B}_{t + 1}^\perp \mathbf{M}_t^\perp)$  from [DKPP22]:  $\phi_{t + 1} = \mathrm{tr}((\mathbf{M}_{t + 1}^\perp)^2)\leq d^{1 / 2p}\left(\mathrm{tr}(\mathbf{M}_t^\perp \mathbf{B}_{t + 1}^\perp \mathbf{M}_t^\perp)\right)^{\frac{2p}{2p + 1}}$ . By the definition of  $\mathbf{B}_t^\perp$ , we have  $\mathrm{tr}(\mathbf{M}_t^\perp \mathbf{B}_{t + 1}^\perp \mathbf{M}_t^\perp)\approx \mathbf{E}_{P_{t + 1}}[\| \mathbf{M}_t^\perp (x - \mu)\| _2^2 -\mathrm{tr}((\mathbf{M}_t^\perp)^2)]$ . To upper bound this, we will use the guarantees of the multi-dimensional filter along with the goodness conditions and the fact that  $\| \mathbf{M}_t^\perp (x - \mu)\| _2^2\approx \| \mathbf{U}_t(x - \mu)\| _2^2$  (since  $\mathbf{U}_t$  is a Johnson-Lindenstrauss sketch of  $\mathbf{M}_t^\perp$  with  $k\gtrsim \log (n)$ ; see Appendix B) as follows: we show in the appendix that the contribution from inliers is small by Definition 2.1 and the contribution from outliers,  $\epsilon \mathbf{E}_{B_{t + 1}}[\| \mathbf{U}_t(x - \mu)\| _2^2,$  is controlled by Lemma B.27. Combining the two aforementioned arguments with some algebraic manipulations we can formally show that:
Lemma 3.3 (Filtering Implication). After the filtering,  $\mathrm{tr}(\mathbf{M}_t^\perp \mathbf{B}_{t + 1}^\perp \mathbf{M}_t^\perp)\leq 0.1\| \mathbf{B}_t^\perp \|_{\mathrm{op}}\mathrm{tr}((\mathbf{M}_t^\perp)^2)$  Combining Lemma C.4 with  $\phi_{t + 1}\leq d^{1 / 2p}\left(\mathrm{tr}(\mathbf{M}_t^\perp \mathbf{B}_{t + 1}^\perp \mathbf{M}_t^\perp)\right)^{2p + 1}$  from earlier, and noting that  $d^{1 / 2p} = d^{1 / 2\log d}\leq 3$  and  $\| \mathbf{B}_t^\perp \|_{\mathrm{op}}^{2p}\leq \mathrm{tr}((\mathbf{M}_t^\perp)^2) = \phi_t$  , we have  $\phi_{t + 1}\leq 0.99\phi_t$  , as desired.
3.2 Case 2: A Few Large Eigenvalues
Consider now the alternate case where  $q_{t}$  from Line 10 is large. Then the approach from the previous case is inapplicable because  $\mathbf{M}_t^\perp z$ 's will be highly correlated vectors. The following result formally proves that this happens only when the top eigenvalue of  $\mathbf{M}_t^\perp$  contributes significantly to its spectrum.
Claim 3.4. If  $\mathbf{Pr}_{z,z^{\prime}\sim \mathcal{N}(0,\mathbf{I})}\left[\frac{|\langle\mathbf{M}_t^\perp z,\mathbf{M}_t^\perp z^{\prime}\rangle|}{\|\mathbf{M}_t^\perp z\|_2\|\mathbf{M}_t^\perp z^{\prime}\|_2} >\gamma \right]\geq \alpha$  then  $\frac{\|\mathbf{M}_t^\perp\|_{\mathrm{op}}^2}{\mathrm{tr}((\mathbf{M}_t^\perp)^2)}\geq \mathrm{poly}(\alpha ,\gamma)$
Proof Sketch. Let  $q_{t}$  be the probability from the statement. Pretend for simplicity that  $\| \mathbf{M}_t^\perp z\| _2^2$  is equal to its expectation  $\mathrm{tr}((\mathbf{M}_t^\perp)^2)$ . Applying Markov's inequality, we see that  $\alpha < q_{t}\leq \gamma^{-2}\mathbf{E}[\langle \mathbf{M}_t^\perp z,\mathbf{M}_t^\perp z'\rangle^2 ] / \mathrm{tr}((\mathbf{M}_t^\perp)^2) = \gamma^{-2}\| (\mathbf{M}_t^\perp)^2\| _F^2 /\mathrm{tr}((\mathbf{M}_t^\perp)^2)$ . Further using the standard fact that  $\| \mathbf{A}\|_{\mathrm{F}}^{2}\leq \mathrm{tr}(\mathbf{A})\| \mathbf{A}\|_{\mathrm{op}}$  for a psd matrix  $\mathbf{A}$  applied to  $\mathbf{M}_t^\perp$  completes the proof.
Since  $\alpha, \gamma$  are  $\mathrm{polylog}(d / \epsilon)$  in our setting (Line 12), Claim 3.4 reveals that the top eigenvalue of  $(\mathbf{M}_t^\perp)^2$  is at least  $\phi_t / \mathrm{polylog}(d / \epsilon)$ . Thus, our algorithm can take a top eigenvector  $u$  of  $\mathbf{M}_t^\perp$  and add it
to the subspace of directions to ignore,  $\mathcal{V}_t$ , i.e., project all data on the subspace perpendicular to  $u$  in the future iterations. Formally, we show that the potential decreases if  $u$  has large projection along  $\mathbf{M}_t^\perp$ :
Claim 3.5. Let  $V_{t + 1} = V_t \cup \{u\}$  for some unit vector  $u \in V_t^\perp$ , then  $\phi_{t + 1} \leq \phi_t - u^\top (\mathbf{M}_t^\perp)^2 u$ .
Proof Sketch. Going from step  $t$  to step  $(t + 1)$ , the effect of adding  $u$  in the subspace  $V_{t + 1}$  is that  $\mathbf{B}_{t + 1}^{\perp} = \Delta \mathbf{B}_t^\perp \Delta$  for the projection matrix  $\Delta = \mathbf{I} - uu^{\top}$ . Then by properties of the trace operator and Lieb-Thirring inequality:  $\phi_{t + 1} = \mathrm{tr}((\mathbf{B}_{t + 1}^{\perp})^{2p}) = \mathrm{tr}((\Delta \mathbf{B}_t^\perp \Delta)^{2p}) = \mathrm{tr}((\mathbf{B}_t^\perp \Delta)^{2p}) \leq \mathrm{tr}((\mathbf{B}_t^\perp)^{2p}\Delta^{2p}) = \mathrm{tr}((\mathbf{B}_t^\perp)^{2p}\Delta) = \mathrm{tr}((\mathbf{B}_t^\perp)^{2p}) - u^\top (\mathbf{B}_t^\perp)^{2p}u = \phi_t - u^\top (\mathbf{M}_t^\perp)^2 u.$
Claims 3.4 and 3.5 imply that if we use  $u$  equal to the top eigenvector of  $\mathbf{M}_t^\perp$ , then our proof will be completed. However, the algorithm cannot compute exact eigenvectors in almost-linear time. Fortunately, power iteration in Line 20 computes a fine enough approximation such that  $u^\top (\mathbf{M}_t^\perp)^2 u$  is a constant fraction of  $\| \mathbf{M}_t^\perp \|_{\mathrm{op}}^2$ . Thus, we obtain  $\phi_{t + 1} \leq (1 - 1 / \mathrm{polylog}(d / \epsilon))\phi_t$  as desired.
4 Robust Linear Regression: Optimal Error In Almost-Linear Time
In this section, we focus our attention to linear regression in Huber contamination model and sketch the proof of Theorem 1.4. Recall that each inlier sample  $(X,y)$  is distributed according to the Gaussian linear regression model of Definition 1.2.
Algorithm 2 Robust Linear Regression Under Huber Contamination
Input:  $\epsilon >0$  , multiset  $S$  of  $\mathbb{R}^{d + 1}$  containing pairs of the form  $(x,y)$  with  $x\in \mathbb{R}^d$ $y\in \mathbb{R}$  Output: A vector in  $\mathbb{R}^d$    
1: Set  $\widehat{\sigma}_y^2\gets \mathrm{TRIMMEDMEAN}(\{y:(x,y)\in S\})$  ▷  $\widehat{\sigma}_y^2 = \sigma_y^2 (1\pm O(\epsilon \log (1 / \epsilon)))$    
2: Draw  $a\in \mathbb{R}$  uniformly at random from  $[- \widehat{\sigma}_y,\widehat{\sigma}_y]$    
3: Define the interval  $I\coloneqq [a - \ell ,a + \ell ]$  for  $\ell \coloneqq \widehat{\sigma}_y / \log (1 / \epsilon)$  ▷ Random choice of I   
4:  $S^{\prime}\gets \{x:(x,y)\in S,y\in I\}$  ▷ Simulating  $G_{I}$    
5:  $\widehat{\beta}_{I}\gets \mathrm{ROBUSTMEAN}(S^{\prime},10\epsilon)$  ▷ Algorithm from Theorem 1.3   
6: return  $\beta = (\widehat{\sigma_y^2 / a})\widehat{\beta}_I$  ▷ Rescaling
Our main technical insight is a novel reduction to robust mean estimation. Existing reductions in the literature (see, e.g., [BDLS17]) rely on the fact that  $\mathbf{E}[yX] = \mathbf{E}[XX^\top\beta + Xz] = \beta$ . However, it is unclear how to estimate the mean of  $yX$  up to error  $O(\epsilon)$  under Huber contamination because  $yX$  is very far from Gaussian; e.g.,  $yX$  is subexponential as opposed to (sub)-Gaussian, and it is not even symmetric around  $\beta$ , implying median might not even work along a direction. Thus, existing approaches using this methodology have error  $\Omega(\sigma\epsilon\log(1/\epsilon))$ . In contrast, our reduction reduces to robust (almost)-Gaussian mean estimation by using the conditional distribution of  $X$  given  $y = a$  under Definition 1.2.
Claim 4.1 (Conditional Distribution). Let  $(X,y) \in \mathbb{R}^{d+1}$  follow Definition 1.2. Given  $a \in \mathbb{R}$ , denote by  $G_{a}$  the distribution of  $X$  given  $y = a$ . Similarly, given an interval  $I \subset \mathbb{R}$ , let  $G_{I}$  represent the conditional distribution of  $X$  given  $y \in I$ . Define  $\sigma_{y}^{2} \coloneqq \sigma^{2} + \| \beta \|_{2}^{2}$ . Then,  $G_{a} = \mathcal{N}\left(\frac{a}{\sigma_{y}^{2}}\beta, \mathbf{I}_{d} - \frac{1}{\sigma_{y}^{2}}\beta\beta^{\top}\right)$  and  $G_{I} = \frac{1}{\mathbf{Pr}[y \in I]}\int_{I}\phi(a';0,\sigma_{y}^{2})\mathcal{N}\left(\frac{a'}{\sigma_{y}^{2}}\beta, I_{d} - \frac{1}{\sigma_{y}^{2}}\beta\beta^{\top}\right)\mathrm{d}a'$ , where  $\phi(z;0,\nu^{2})$  denotes the pdf of the  $\mathcal{N}(0,\nu^{2})$  at  $z \in \mathbb{R}$ .
Since the conditional distribution  $G_{a}$  above is just a Gaussian whose mean is scaled version of  $\beta$  (and roughly isotropic covariance), one would ideally like to get  $(O(\epsilon)\text{-corrupted})$  samples from  $G_{a}$ , then use the robust mean estimator from Theorem 1.3, and finally scale the result back appropriately.
Obviously, it is impossible to simulate  $G_{a}$  using  $G$  since there is zero probability over the inliers that  $y$  is exactly equal to  $a$  in our dataset. Instead, we can simulate the conditional distribution given  $y \in I$  for some interval  $I$  centered around  $a$ . The length  $\ell$  of  $I$  needs to be carefully selected. On the one hand,  $I$  should be sufficiently narrow to ensure that the mean of  $G_{I}$  closely approximates the mean of  $G_{a}$ . On the other hand, it should not be excessively narrow to avoid rejecting a significant portion of our samples. As we show later, an interval of length  $\Theta(\sigma / \log(1/\epsilon))$  meets both of these criteria as long as  $\| \beta \|_2 \lesssim \sigma \epsilon \log(1/\epsilon)$ . Also, a set of i.i.d. samples from  $G_{I}$  for such a small interval satisfy the goodness condition with the correct parameters because  $G_{I}$  will be roughly isotropic.