SlowGuess/NeurIPS_2023_OCR huggingface dataset:
1.  $(1 - \epsilon)\mathbf{E}_{X\sim G}[w(X) - w'(X)] <   \frac{\epsilon}{\beta - 1}\mathbf{E}_{X\sim B}[w(X) - w'(X)]$
2.  $\mathbf{E}_{X\sim P}[w(X)\tilde{\tau} (X)]\leq T\beta$
The algorithm can be implemented in  $O(n \log (r / T)))$  time.
Proof. The weight removed from inliers is:
$$
(1 - \epsilon) \underset {X \sim G} {\mathbf {E}} [ w (X) - w ^ {\prime} (X) ] = \frac {1 - \epsilon}{r} \underset {X \sim G} {\mathbf {E}} [ w (X) \tilde {\tau} (X) ] \leq T / r
$$
The weight removed from outliers is:
$$
\begin{array}{l} \epsilon_ {X \sim B} \mathbf {E} [ w (X) - w ^ {\prime} (X) ] = \frac {\epsilon}{r} _ {X \sim B} \mathbf {E} [ w (X) \tilde {\tau} (X) ] \\ = \frac {1}{r} \left(\mathbf {E} _ {X \sim P} [ w (X) \tilde {\tau} (X) ] - (1 - \epsilon) \mathbf {E} _ {X \sim G} [ w (X) \tilde {\tau} (X) ]\right) \\ \geq \frac {T (\beta - 1)}{r} \\ \end{array}
$$
Regarding runtime, for any  $\ell > r / (eT)$  we have that  $\mathbf{E}_{X\sim P}[w_{\ell}(X)\tilde{\tau} (X)]\leq$ $\mathbf{E}_{X\sim P}[w_{\ell}(X)\tilde{\tau} (X)\exp (-\ell \tilde{\tau} (X) / r)]\leq \frac{r}{e\ell}\leq T,$  where we used the inequality  $xe^{-\alpha x}\leq 1 / (e\cdot \alpha).$
(i)  $\mathbf{E}_{X\sim G}[w(X) - w'(X)] < (\epsilon / \log(1/\epsilon)) \mathbf{E}_{X\sim B}[w(X) - w'(X)]$ .
(ii)  $\epsilon \mathbf{E}_{X\sim B}[w'(X)\| \mathbf{U}(X - \mu_{P_w})\| _2^2 ]\lesssim \epsilon \mathrm{tr}(\mathbf{U}^\top \mathbf{U}).$
Proof. Let  $g(x) \coloneqq \| \mathbf{U}(x - \mu_{P_w})\| _2^2$  and  $\tau (x)\coloneqq g(x)\mathbb{1}(p(x) > 100\mathrm{tr}(\mathbf{U}^\top \mathbf{U}))$ . Denote  $\mathbf{B}_{P_w}\coloneqq \Sigma_{P_w} - \mathbf{I}$ . Using the goodness conditions and the certificate lemma:
$$
\begin{array}{l} \underset {X \sim G} {\mathbf {E}} [ w (X) \tau (X) ] \leq \frac {2 \epsilon}{\log (1 / \epsilon)} \| \mathbf {U} \| _ {\mathrm {F}} ^ {2} + \| \mathbf {U} (\mu - \mu_ {P _ {w}}) \| _ {2} ^ {2} (12) \\ \leq \left(\frac {2 \epsilon}{\log (1 / \epsilon)} + \| \mu - \mu_ {P _ {w}} \| _ {2} ^ {2}\right) \| \mathbf {U} \| _ {\mathrm {F}} ^ {2} \quad (byLemmaB.25) \\ \leq \left(\frac {2 \epsilon}{\log (1 / \epsilon)} + \epsilon^ {2} + O \left(\| \mathbf {B} _ {P _ {w}} \| _ {\mathrm {o p}} \epsilon\right)\right) \| \mathbf {U} \| _ {\mathrm {F}} ^ {2} (13) \\ \lesssim \left(\frac {\epsilon}{\log (1 / \epsilon)} + \epsilon \| \mathbf {B} _ {P _ {w}} \| _ {\mathrm {o p}}\right) \| \mathbf {U} \| _ {\mathrm {F}} ^ {2} (14) \\ \lesssim \left(\frac {\epsilon}{\log (1 / \epsilon)} + \epsilon^ {2} \operatorname {p o l y} \log (1 / \epsilon)\right) \| \mathbf {U} \| _ {\mathrm {F}} ^ {2} \quad (\text {B y}) \\ \lesssim \left(\frac {\epsilon}{\log (1 / \epsilon)}\right) \| \mathbf {U} \| _ {\mathrm {F}} ^ {2}, (15) \\ \end{array}
$$
where the first inequality uses Lemma B.25 (this uses that  $\| \mathbf{U}(\mu -\mu_{P_w})\| _2^2\leq \mathrm{tr}(\mathbf{U}^\top \mathbf{U})$  which follows by the certificate lemma and that  $\| \mathbf{B}_{P_w}\| < 1$ ), the second inequality uses  $\| \mathbf{U}(x - \mu)\| _2^2 = \mathrm{tr}(\mathbf{U}^\top \mathbf{U}(x - \mu)(x - \mu)^\top)$  and then the fact  $\mathrm{tr}(\mathbf{AB})\leq \mathrm{tr}(\mathbf{A})\| \mathbf{B}\|_{\mathrm{op}}$  for  $\mathbf{A} = \mathbf{U}^{\top}\mathbf{U},\mathbf{B} = (x - \mu)(x - \mu)^{\top}$ , the third line uses the certificate lemma (Lemma 2.4).
We can thus apply the filtering lemma (Lemma B.27) with  $T$  equal to the right hand side of Equation (15), and  $\beta = \log(1 / \epsilon)$  to obtain that  $\mathbf{E}_{X \sim P}[w'(X)\tau(X)] \leq T\beta \lesssim \epsilon \| \mathbf{U}\|_{\mathrm{F}}^2$ . In more detail,
$$
\begin{array}{l} \epsilon_ {X \sim B} \left[ w ^ {\prime} (X) g _ {t} (X) \right] \leq \epsilon \left(\mathbf {E} _ {X \sim B} \left[ w ^ {\prime} (X) \tau_ {t} (X) \right] + 1 0 0 \operatorname {t r} \left(\mathbf {U} ^ {\top} \mathbf {U}\right)\right) \\ \lesssim \epsilon \underset {X \sim B} {\mathbf {E}} [ w ^ {\prime} (X) \tau_ {t} (X) ] + \epsilon \mathrm {t r} (\mathbf {U} ^ {\top} \mathbf {U}) \\ \lesssim T \log (1 / \epsilon) + \epsilon \operatorname {t r} \left(\mathbf {U} ^ {\top} \mathbf {U}\right) \quad (\text {b y}) \\ \lesssim \epsilon \operatorname {t r} (\mathbf {U} ^ {\top} \mathbf {U}) \\ \end{array}
$$
This completes the proof of part (ii). Part (i) follows directly from Lemma B.27.
B.6 Algorithm from [DKKLMS18]
We now describe (a simplified version of) the algorithm from [DKKLMS18] and state its improved runtime.
Algorithm 4 Optimal Error Robust Mean Estimation Under Hub Contamination
1: Input: Parameters  $\epsilon, r$ , and sets  $S_{1}, S_{2} \subset \mathbb{R}^{d}$ , where for each  $i = 1, 2$  the uniform distribution on  $S_{i}$  is of the form  $(1 - \epsilon)G_{i} + \epsilon B_{i}$  for some  $G_{i}$  satisfying Definition 2.1 with appropriate parameters.  
2: Output: A vector in  $\mathbb{R}^d$ .  
3: Let  $k := r\log(1/\epsilon)$ , and  $P$  be the uniform distribution on  $S_{1}$ .  
4: Calculate naive estimate  $\widehat{\mu}$  s.t.  $\|\widehat{\mu} - \mu\|_2 = R$  for  $R = O(\sqrt{d\log(1/\epsilon)})$ . ▷ e.g., geometric mean  
5: Initialize  $t \gets 0$  and  $w_{t}(x) = \mathbb{1}(\|x - \widehat{\mu}\|_2 \leq 2R)$  for all  $x \in \mathbb{R}^d$   
6: Denote by  $P_t$  the weighted by  $w$  version of  $P$ , by  $\Sigma_t$  the covariance of  $P_t$ .  
7: Let  $v_{t,1}, \ldots, v_{t,d}$  be the eigenvector decomposition of  $\Sigma_t$  and  $\lambda_{t,1} \geq \ldots \geq \lambda_{t,d}$  the corresponding eigenvalues.  
8: while  $\sum_{i=1}^{k} \frac{1}{k} \lambda_{t,k} \geq 1 + \lambda$  for  $\lambda = \frac{C}{r} \epsilon$  do  
9: Define the polynomial  $g_t(x) = \|U_t(x - \mu_t)\|_2^2$ , where  $U_t = [v_{t,1}/\sqrt{k}, \dots, v_{t,k}/\sqrt{k}]^\top$ .  
10: Define the score function  $\tau_t(x) = g_t(x) \mathbb{1}(g_t(x) > 100/r)$   
11:  $w_{t+1} \gets \text{Filter}(P, \epsilon, w_t, \tau_t)$  ▷ Algorithm 3  
12:  $t \gets t+1$ , and update vectors  $v_{t,1}, \ldots, v_{t,k}$ .  
13: Let  $\mathcal{V} = \text{span}\{v_1, \dots, v_k\}$  and  $\mathcal{V}^\perp = \text{span}\{v_{k+1}, \dots, v_d\}$ .  
14:  $\mu_1 \gets \text{Proj}_{\mathcal{V}^\perp}(\mu_t)$ .  
15:  $\mu_2 \gets \text{BruteForce}(S_2, \epsilon, \mathcal{V})$ .  
16: return  $\mu_1 + \mu_2$ .
Lemma 2.6 (Adapted From [DKKLMS18]). Let  $T$  be a set of  $n$  i.i.d. samples from an  $\epsilon$ -corrupted version of  $\mathcal{N}(0, \mathbf{I}_d)$  and let  $r, \delta \in (0, 1)$  be parameters. If  $n \gg (d + \log(1/\delta)) / \epsilon^2)$  polylog  $(d / \epsilon)$ , then there is an algorithm that, when having as input  $T$ ,  $\epsilon$ , and  $r$ , after time  $\tilde{O}(nd^2 + \frac{1}{\epsilon^{2+r}})$ , it outputs a vector  $\widehat{\mu} \in \mathbb{R}^d$  such that  $\|\widehat{\mu} - \mu\|_2 \lesssim \epsilon / r$  with probability  $1 - \delta$ .
Proof Sketch. The algorithm consists of two parts. The first part (while loop of Line 8) finds weights for the points  $w_{t}(x)$  and subspace  $\mathcal{V} = \operatorname{span}\{v_{1},\dots,v_{k}\}$  such that the weighted empirical mean approximates the true mean in that subspace within  $\ell_2$ -distance  $O(\epsilon /r)$ , and the second part runs a brute force procedure (Lemma B.28) to approximate the true mean in the remaining  $k$ -dimensional subspace. The brute force procedure runs in time exponential to the dimension of that subspace, but since  $k = r\log (1 / \epsilon)$ , it becomes polynomial in  $1 / \epsilon$ . Concretely, the runtime of the brute force step is given in Lemma B.28. The dataset  $S_{2}$  that we use for this brute force step is of smaller cardinality: since the dimension of the subspace is  $k$ ,  $n^{\prime} = O(k / \epsilon^{2})$  samples in  $S_{2}$  are enough to satisfy the goodness conditions. Thus, the runtime from Lemma B.28 with  $n$  replaced by  $n^{\prime}$  is  $O((d + \epsilon^{-2 - r})\mathrm{polylog}(d / \epsilon))$ .
In the remainder, we sketch the runtime bound for the first stage of the algorithm, by arguing that the number of iterations is  $\tilde{O}(d)$ . If we let our algorithm compute exact eigenvalues and eigenvectors, each iteration needs  $nd^3$  time, resulting in a total of  $\tilde{O}(nd^4)$  for the runtime of the while loop of Line 8. This can improved to  $\tilde{O}(nd^2)$  by calculating approximate top  $k$ -eigenvectors (see, e.g., [MM15]).
To show that the number of iterations is  $\tilde{O}(d)$ , we will show that every time that the filtering procedure of Line 11 is used, it removes at least  $\tilde{\Omega}(\epsilon / d)$  mass from the points' weights. Then, given that the outliers themselves have mass at most  $O(\epsilon)$ , the algorithm would stop after  $\tilde{O}(d)$  iterations. The mass removed per iteration is (by definition of the filtering procedure)
$$
\mathbf {E} _ {X \sim P} [ w _ {t} (X) - w _ {t + 1} (X) ] = \frac {\mathbf {E} _ {X \sim P} [ w _ {t} (X) \tau_ {t} (X) ]}{\max _ {x : w _ {t} (x) > 0} \tau_ {t} (x)}.
$$
The denominator is upper bounded by  $\max_{x:w_t(x) > 0}\tau_t(x)\leq \tilde{O} (d)$  by the prepossessing of Line 3.10 For the numerator,
$$
\underset {X \sim P} {\mathbf {E}} [ w _ {t} (X) \tau_ {t} (X) ] = \underset {X \sim P} {\mathbf {E}} [ w _ {t} (X) g _ {t} (X) ] - \underset {X \sim P} {\mathbf {E}} [ w _ {t} (X) g _ {t} (X) \mathbb {1} (g _ {t} (X) \leq 1 0 0 / r) ].
$$
<sup>10</sup> Since the norm of the samples from the Gaussian distribution is tightly concentrated, the filtering procedure removes  $o(\epsilon / \log(1/\epsilon))$  fraction of inliers.
The first term is at least  $(1 + \lambda)$  because of the condition in Line 8. We also claim that the second term is at most  $(1 + \lambda /2)$ . These two would imply that  $\mathbf{E}_{X\sim P}[w_t(X) - w_{t + 1}(X)] = \Omega (\lambda) = \Omega (\epsilon)$ , which implies that each iteration removes  $\tilde{\Omega} (\epsilon /d)$  fraction of points, completing the proof. The remaining claim for bounding  $\mathbf{E}_{X\sim P}[w_t(X)g_t(X)\mathbb{1}(g_t(X)\leq 100 / r)]$  is similar to [DKPP23, Lemma 2.14 (iv)]: We first note that  $\mathbf{E}_{X\sim G}[w_t(X)\mathbb{1}(g_t(X)\leq 100 / r)] = 1 - O(\epsilon /\log (1 / \epsilon))$ , because at most  $O(\epsilon /\log (1 / \epsilon))$ -fraction of inliers have  $g_{t}(x) > 100 / r$  (cf. goodness condition (2.c) in Definition 2.1). Then,
$$
\begin{array}{l} \left| \mathbf {E} _ {X \sim P} [ w _ {t} (X) \mathbb {1} (g _ {t} (X) \leq 1 0 0 / r) ] - 1 \right| \leq (1 - \epsilon) \left| \mathbf {E} _ {X \sim G} [ w _ {t} (X) g _ {t} (X) \mathbb {1} (g _ {t} (X) \leq 1 0 0 / r) ] - 1 \right| \\ + \epsilon \left| _ {X \sim B} ^ {\mathbf {E}} \left[ w _ {t} (X) g _ {t} (X) \mathbb {1} \left(g _ {t} (X) \leq 1 0 0 / r\right) \right] - 1 \right|. \\ \end{array}
$$
The first term is upper bounded by  $\lambda /4$  by the goodness condition (2.b) along with the certificate lemma (Lemma B.23). For the second term we can use the bound
$$
\epsilon \left| _ {X \sim B} \left[ w _ {t} (X) g _ {t} (X) 1 \left(g _ {t} (X) \leq 1 0 0 / r\right) \right] - 1 \right| \leq \epsilon \left(1 0 0 / r + 1\right) = O (\epsilon / r) <   \lambda / 4.
$$
We will use the following brute force algorithm to estimate the mean in a low-dimensional subspace.
Lemma B.28. There exists an algorithm, BRUTEFORCE, that given a set of  $n$  samples in  $\mathbb{R}^d$  from an  $\epsilon$ -corrupted version of a distribution  $G$  satisfying the first condition in Definition 2.1 and a subspace  $\mathcal{V}$  of dimension  $k$ , runs in time  $O\left(dnk + (n\log n + \mathrm{poly}(k))2^{O(k)}\right)$  and finds a vector  $\tilde{\mu} \in \mathbb{R}^d$  such that  $\| \mathrm{Proj}_{\mathcal{V}}(\mu - \tilde{\mu}) \|_2 = O(\epsilon)$ .
Proof. The algorithm first projects the points to the  $k$ -dimensional subspace. It then computes an  $\eta$ -cover  $\mathcal{C}$  of the  $k$ -dimensional unit sphere, which by Fact B.22 consists of at most  $(4 / r)^k$  vectors, and for every vector  $u \in \mathcal{C}$ , it finds an estimate  $\mu_u$  for the  $u$ -projection of the mean satisfying  $|u^\top \mu - \mu_u| \leq \gamma$  for  $\gamma = O(\epsilon)$  being the same as in the right hand side of the first goodness condition in Definition 2.1 (because of that goodness condition, letting  $\mu_u$  be just the median in the  $u$ -direction achieves  $\gamma$ -error). It then solves the linear program
$$
\text {f i n d} x \text {s . t .} | u ^ {\top} x - \mu_ {u} | \leq 2 \gamma \forall u \in \mathcal {C}.
$$
Let  $v \in S^{d-1}$  be in the direction of  $x - \mu$  and  $u = \arg \min_{y \in \mathcal{C}} \| v - y \|_2$ . Then,
$$
\begin{array}{l} \left\| x - \mu \right\| _ {2} = \left| v ^ {\top} (x - \mu) \right| \leq \left| u ^ {\top} (x - \mu) \right| + \left| (v - u) ^ {\top} (x - \mu) \right| \\ \leq | u ^ {\top} (x - \mu_ {u}) | + | u ^ {\top} (\mu_ {u} - \mu) | + \| v - u \| _ {2} \| x - \mu \| _ {2} \\ \leq 4 \gamma + \eta \| x - \mu \| _ {2}, \tag {16} \\ \end{array}
$$
which establishes that  $\| x - \mu \|_2 \leq 4\gamma / (1 - \eta)$ .
Regarding runtime, the projection of the data takes  $O(dnk)$  time, solving the median in each direction takes  $O((4/\eta)^k n \log n)$  time. For the LP, the separation oracle that checks all the constraints exhaustively needs time  $(4/\eta)^k k$  and the LP can be solved by  $O(\mathrm{poly}(k) \log(R/r))$  calls to that separation oracle, where  $R$ ,  $r$  are bounds on the volume of the feasible set  $\eta \leq \mathrm{vol}(\{x \in \mathbb{R}^k : |u^\top x - \mu_u| \leq \gamma_1 \forall u \in \mathcal{C}\}) \leq R$ . For the lower bound we have that the volume of the feasible set is at least the volume of a  $k$ -dimensional ball of radius  $\gamma$  because  $\mu$  is a feasible solution and  $\mu + \gamma v$  remains feasible for any unit vector  $v$ . For the upper bound, we have that the feasible set is included in a  $k$ -dimensional ball of radius  $4\gamma/(1-\eta)$  by Equation (16). Thus the runtime for the LP is  $O(\mathrm{poly}(k) \log(R/r)) = O(\mathrm{poly}(k))$ . Finally, choosing  $\eta = 1/4$ , simplifies the total runtime to  $O\left(dnk + (n \log n + \mathrm{poly}(k)) 2^{O(k)}\right)$ .
C Omitted Proofs from Section 3
We begin by stating additional conditions that must hold throughout the algorithm's execution. Subsequently, in Appendices C.2 and C.3, we present a comprehensive analysis of the two cases in our main algorithm. Finally, in Appendix C.4, we combine all the components to conclude the proof of Theorem 1.3.
C.1 Deterministic Conditions for Algorithm 1
Condition C.1 (Deterministic Conditions for Algorithm 1). Consider the notation in Algorithm 1. For all  $t \in [t_{\max}]$ , the following hold:
(i) Spectral norm of  $\mathbf{B}_t^\perp$ :  $\widehat{\lambda}_t \in [0.1\|\mathbf{B}_t^\perp\|_{\mathrm{op}}, 10\|\mathbf{B}_t^\perp\|_{\mathrm{op}}]$ .
(ii) Frobenius norm  $\| \mathbf{U}_t\|_{\mathrm{F}}^2\in [0.8k\| \mathbf{M}_t^\perp \|_{\mathrm{F}}^2,1.2k\| \mathbf{M}_t^\perp \|_{\mathrm{F}}^2 ]$
(iii) Scores: For all  $x \in \mathrm{support}(P)$ ,  $\| \mathbf{U}_t(x - \mu_t)\|_2^2 / \| \mathbf{M}_t^\perp (x - \mu_t)\|_2^2 \in [0.8k, 1.2k]$ .
(iv) Maximum norm:  $\max_{j\in [k]}\| v_{t,j}\| _2^2\leq \frac{\log k}{k}\mathrm{tr}(\mathbf{U}_t^\top \mathbf{U}_t)$
(v) Probability estimate:  $|\widehat{q}_t - q_t| \leq 0.1 / (k^2 t_{\max})$ .
(vi)  $\mathbf{U}_t$  almost orthogonal: For every pair  $u_i, u_j$  of distinct rows of  $\mathbf{U}_t$  it holds  $|\langle u_i, u_j \rangle| \leq \| u_i \|_2 \| u_j \|_2 / k^2$ .
(vii) Approximate eigenvector:  $u_{t}$  from Line 19 satisfies  $u_{t}^{\perp}\mathbf{B}_{t}^{\perp}u_{t}\geq (1 - 1 / p)\| \mathbf{B}_{t}^{\perp}\|_{\mathrm{op}}$ .
Lemma C.2. Condition C.1 is satisfied with probability at least 0.9.
Proof. The spectral norm condition holds by using Fact B.10 with  $\eta = 0.9$ ,  $\delta = 0.001 / t_{\mathrm{max}}$ , and a union bound over the  $t_{\mathrm{max}}$  iterations of the algorithm.
The next two conditions hold with probability 0.999 by using Fact B.14 with  $\mathbf{A} = \mathbf{M}_t^\perp \sqrt{k}$  and a similar union bound.
Regarding condition (iv), with probability 0.999 we have the following regarding the vectors  $v_{t,j}$  from line 15
$$
\begin{array}{l} \| v _ {t, j} \| _ {2} ^ {2} \lesssim \sqrt {\log (k t _ {\max })} \| (\mathbf {M} _ {t} ^ {\perp}) ^ {2} \| _ {\mathrm {F}} + \log (k t _ {\max }) \| (\mathbf {M} _ {t} ^ {\perp}) ^ {2} \| _ {\mathrm {o p}} \\ \lesssim \log (k t _ {\max }) \| \mathbf {M} _ {t} ^ {\perp} \| _ {\mathrm {F}} ^ {2} \lesssim \frac {\log (k t _ {\max })}{k} \| \mathbf {U} _ {t} ^ {\perp} \| _ {\mathrm {F}} ^ {2} \lesssim \frac {\log (k)}{k} \| \mathbf {U} _ {t} ^ {\perp} \| _ {\mathrm {F}} ^ {2}, \\ \end{array}
$$
where the first step uses Hanson-Wright inequality with probability of failure  $1 / (1000kt_{\mathrm{max}})$ , the second step uses the fact  $\| \mathbf{A}\|_{\mathrm{F}}^2\leq \operatorname {tr}(\mathbf{A})\| \mathbf{A}\|_{\mathrm{op}}\leq \operatorname {tr}(\mathbf{A})^2$  applied with  $\mathbf{A} = (\mathbf{M}_t^\perp)^2$ , the third step is due to condition (ii), and the last one is because  $t_\mathrm{max}\lesssim \mathrm{poly}(k)$  for the choice of values that we have made for  $t_\mathrm{max}$  and  $k$ .
For Condition (v), if  $\widehat{q}_t$  is the empirical average of  $\mathbb{1}(|\langle \mathbf{M}_t^\perp z, \mathbf{M}_t^\perp z' \rangle| > \| \mathbf{M}_t^\perp z \|_2 \| \mathbf{M}_t^\perp z' \|_2 / k^2)$  over  $\Theta(k^4 t_{\max}^2)$  instantiations of  $z, z' \sim \mathcal{N}(0, \mathbf{I})$ , then the condition holds by a standard Chernoff bound.
Condition (vi) holds by the fact that the algorithm uses  $\mathbf{U}_t$  only in the case that  $\widehat{q_t} \leq 1 / (k^2 t_{\mathrm{max}})$ . By condition (v), in that case we have that  $\mathbf{Pr}_{z,z' \sim \mathcal{N}(0,\mathbf{I})}[\|\mathbf{M}_t^\perp z, \mathbf{M}_t^\perp z'\rangle| > \|\mathbf{M}_t^\perp z\|_2\|\mathbf{M}_t^\perp z'\|_2 / k^2] \geq 0.9 / (k^2 t_{\mathrm{max}})$ . Thus, by a union bound, with high constant probability, for every iteration  $t$ , all pairs of rows of  $\mathbf{U}_t$  will have angle with cosine in  $[-1 / k^2, 1 / k^2]$ .
Regarding Condition (vii): By Fact B.10 with  $\eta = 1 / p = 1 / \log(d)$  and  $\delta = 1 / t_{\max}$  (so that we can do a union bound over all iterations of the algorithm), we have that if  $p' \gg \log^2(dt_{\max})$  then  $u_t^\top \mathbf{B}_t^\perp u_t \geq (1 - 1/p) \| \mathbf{B}_t^\perp \|_{\mathrm{op}}$  with high constant probability throughout the algorithm.
C.2 Case 1
This section contains the details missing from Section 3.1. We restate and prove the following:
Claim C.3 (Claim 3.2 restated). Assuming that Condition C.1 is satisfied, for every iteration  $t$  that the condition of Line 12 is satisfied,  $\phi_{t + 1} \leq 0.99\phi_t$ . Moreover,  $\mathbf{E}_{X \sim G}[w_t(X)] \geq 1 - \epsilon / \log(1/\epsilon)$  and  $\mathbf{B}_t^\perp \succeq 0$  throughout the algorithm's execution.
The reminder of the subsection is dedicated to the proof of the claim. In particular, we focus on the claim  $\phi_{t + 1}\leq 0.99\phi_t$  , since the invariant  $\mathbf{E}_{X\sim G}[w_t(X)]\geq 1 - \epsilon /\log (1 / \epsilon)$  follows by the guarantee of the filter (Lemma 2.5) rather directly: First, we note that by Lemma B.26, the deterministic
conditions (vi) and (iv) from Condition C.1 imply that the goodness condition (2.c) is applicable to the polynomial  $\| \mathbf{U}_t(x - \mu)\| _2^2$  where  $\mathbf{U}_t$  is the matrix used in Line 15; thus, Lemma 2.5 is applicable. Second, by that lemma, we have that every time that  $w_{t}(x)$  changes to  $w_{t + 1}(x)$  it holds that  $\mathbf{E}_{X\sim G}[w_t(X) - w_{t + 1}(X)] < (\epsilon /\log (1 / \epsilon))\mathbf{E}_{X\sim B}[w_t(X) - w_{t + 1}(X)]$ . If  $t^{*}$  denotes the final iteration, this means that  $\mathbf{E}_{X\sim G}[w_0(X) - w_{t^*}(X)] < (\epsilon /\log (1 / \epsilon))\mathbf{E}_{X\sim B}[w_0(X) - w_{t^*}(X)]\leq \epsilon /\log (1 / \epsilon)$ , thus  $\mathbf{E}_{X\sim G}[w_t(X)]\geq 1 - \epsilon /\log (1 / \epsilon)$  throughout the execution.
We also note, that  $\mathbf{E}_{X\sim G}[w_t(X)]\geq 1 - \epsilon /\log (1 / \epsilon)$  implies that  $\mathbf{B}_t\succeq 0$  , and  $\mathbf{B}_t^\perp \succeq 0$  . This is because, by condition (2.b) of the goodness condition, it holds  $\Sigma_{t}\succeq (1 - O(\epsilon))\mathbf{I}$  . Also, we have already shown that  $\mathbf{E}_{X\sim G}[w_t(X)]\geq 1 - \epsilon /\log (1 / \epsilon)$  . Thus  $\mathbf{B}_t\coloneqq (\mathbf{E}_{X\sim G}[w_t(X)])^2\Sigma_t - (1-$ $C_1\epsilon)\mathbf{I}\succeq (1 - O(\epsilon))^2 (1 - O(\epsilon))\mathbf{I} - (1 - C_1\epsilon)\mathbf{I}\succeq 0,$  if  $C_1$  is large enough.
Regarding the potential function  $\phi_t$ , we will establish the following series of inequalities, for  $c$  being some constant of the form  $c = O(1 / C)$ , where  $C$  is the constant used in Algorithm 1 (we can assume  $c < 0.0001$  by selecting  $C$  appropriately large in the algorithm):
$$
\begin{array}{l} \phi_ {t + 1} = \mathrm {t r} ((\mathbf {M} _ {t + 1} ^ {\perp}) ^ {2}) = \| \mathbf {M} _ {t + 1} ^ {\perp} \| _ {2 p} ^ {2 p} \\ \leq \left(d ^ {\frac {1}{2 p (2 p + 1)}} \| \mathbf {M} _ {t + 1} ^ {\perp} \| _ {2 p + 1}\right) ^ {2 p} \quad (\text {b y}) \\ = d ^ {\frac {1}{2 p + 1}} \left(\| \mathbf {M} _ {t + 1} ^ {\perp} \| _ {2 p + 1} ^ {2 p + 1}\right) ^ {\frac {2 p}{2 p + 1}} \\ = d ^ {\frac {1}{2 p + 1}} \left(\operatorname {t r} \left(\left(\mathbf {B} _ {t + 1} ^ {\perp}\right) ^ {2 p + 1}\right)\right) ^ {\frac {2 p}{2 p + 1}} \\ \leq d ^ {\frac {1}{2 p}} \left(\operatorname {t r} \left(\mathbf {M} _ {t} ^ {\perp} \mathbf {B} _ {t + 1} ^ {\perp} \mathbf {M} _ {t} ^ {\perp}\right)\right) ^ {\frac {2 p}{2 p + 1}} \quad (\text {b y}) \\ \leq d ^ {\frac {1}{2 p + 1}} \left(c \| \mathbf {B} _ {t} ^ {\perp} \| _ {\mathrm {o p}} \| \mathbf {B} _ {t} ^ {\perp} \| _ {2 p} ^ {2 p}\right) ^ {\frac {2 p}{2 p + 1}} \quad (\text {b y}) \\ \leq d ^ {\frac {1}{2 p}} c ^ {\frac {2 p}{2 p + 1}} \left(\| \mathbf {B} _ {t} ^ {\perp} \| _ {2 p} \| \mathbf {B} _ {t} ^ {\perp} \| _ {2 p} ^ {2 p}\right) ^ {\frac {2 p}{2 p + 1}} \quad (\text {b y}) \\ \leq 0. 9 9 \left\| \mathbf {B} _ {t} ^ {\perp} \right\| _ {2 p} ^ {2 p} = 0. 9 9 \phi_ {t}. \quad (\text {s i n c e} p \gg \log (d) \text {a n d} c <   0. 0 0 0 1) \\ \end{array}
$$
Where the application of Fact B.6 uses that  $\mathbf{B}_{t + 1} \preceq \mathbf{B}_t$ : this is by definition of the covariance matrix as  $\frac{1}{2(\mathbf{E}_{X \sim P}[w_t(X)])^2} \mathbf{E}_{X,Y \sim P}[w_t(X)w_t(Y)(X - Y)(X - Y)^\top]$  which shows that downweighting points can only make  $\mathbf{B}_t$  the matrix smaller in PSD order (note that  $\mathbf{B}_t$  has the covariance matrix multiplied by  $(\mathbf{E}_{X \sim P}[w_t(X)])^2$ ).
It remains to establish the main inequality that we used above:
Lemma C.4 (Filtering Implication (Lemma C.4 restated)). In the context of Algorithm 1, and assuming that Condition C.1 holds, for every round  $t \in [t_{\max}]$  that the algorithm enters Line 13, it holds  $\mathrm{tr}(\mathbf{M}_t^\perp \mathbf{B}_{t + 1}^\perp \mathbf{M}_t^\perp) \leq c' \| \mathbf{B}_t^\perp \|_{\mathrm{op}} \| \mathbf{M}_t^\perp \|_{\mathrm{F}}^2$  for some  $c' < O(1 / C)$  (where  $C$  is the constant appearing in the algorithm's pseudocode).
Proof. First, we note that