SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Lemma B.20. Let  $\epsilon_0$  be a small enough positive constant, let  $\epsilon \in (0,\epsilon_0)$ , and let  $\tau \in (0,1)$ . Let  $S\subset \mathbb{R}^d$  be a set of  $n$  independent samples from  $\mathcal{N}(\mu ,\mathbf{I}_d)$  for  $n\gg \frac{k^3(d\log(d / \epsilon) + \log(1 / \tau))}{\min(\alpha^2,\epsilon^2 / \log^2(1 / \epsilon))}$ . If  $G$  denotes the uniform distribution over  $S$ , then with probability at least  $1 - \tau$ ,  $G$  is  $(\epsilon ,\alpha ,k)$ -good.
Proof. We prove the results for each condition separately.
Condition (1) As the VC dimension of Linear Threshold Functions (LTFs) is  $d$ , the VC inequality (Theorem B.16) implies that there exists a sufficiently large universal constant  $C$  such that for any unit vector  $v$  and  $\gamma$ ,  $|\mathbf{Pr}_{X \sim S}[v^{\top}(X - \mu) > \gamma] - \mathbf{Pr}_{X \sim \mathcal{N}(0,1)}[X > \gamma]| \leq C(\sqrt{d / n} + \sqrt{\log(1 / \tau) / n})$  with probability  $1 - \tau$ .
We next apply Taylor's Theorem to the complementary cumulative function of the Gaussian,  $\bar{\Phi}(t) := \mathbf{Pr}_{X \sim \mathcal{N}(0,1)}[X > t] = (1/\sqrt{2\pi}) \int_t^\infty e^{-x^2/2} \mathrm{d}x$ . The first two derivatives are  $\bar{\Phi}'(t) =$
Moreover, the algorithm runs in time  $O(nd\mathrm{polylog}(d / \epsilon))$ .
$-e^{-t^2 /2} / \sqrt{2\pi}$  and  $\bar{\Phi}''(t) = te^{-t^2 /2} / \sqrt{2\pi}$  and thus there exists a  $\xi$  between 0 and  $t$  such that  $\bar{\Phi}(t) = 1/2 - t/\sqrt{2\pi} + t^2\xi e^{-\xi^2/2} / (2\sqrt{2\pi}) \leq 1/2 - t/\sqrt{2\pi} + t^3/(2\sqrt{2\pi})$ . Combining with the VC inequality, with probability  $1 - \tau$ , we have that
$$
\operatorname * {P r} _ {X \sim G} [ v ^ {\top} (X - \mu) > \gamma ] \leq 1 / 2 - \gamma / \sqrt {2 \pi} + \gamma^ {3} / (2 \sqrt {2 \pi}) + C (\sqrt {d / n} + \sqrt {\log (1 / \tau) / n}). \tag {2}
$$
By our assumption that  $\sqrt{d / n} +\sqrt{\log(1 / \tau) / n}$  is  $O(\epsilon)$  and thus taking  $\gamma = \Theta (\epsilon)$ , we obtain that the right hand side is less than  $1 / 2$
Conditions (2.a) and (2.b) Let  $G$  be the uniform distribution over a set of  $n$  samples drawn from  $\mathcal{N}(0, \mathbf{I})$ . We recall the following result from the recent robust statistics literature:
Lemma B.21 ([DK19; DKKLMS16]). Let  $\alpha \in (0, \epsilon_0)$  for a small enough positive constant  $\epsilon_0$ . Let  $P$  be an  $O(1)$ -subgaussian distribution with mean  $\mu$  and covariance  $\pmb{\Sigma}$ , where  $\|\pmb{\Sigma} - \mathbf{I}\|_{\mathrm{op}} \leq \alpha \log(1 / \alpha)$ . The uniform distribution over a set of  $n \gg (d + \log(1 / \delta)) / \alpha^2$  points drawn i.i.d. from  $P$  satisfies Conditions (2.a) and (2.b) from Definition 2.1 with probability  $1 - \delta$ .
Condition (2.c) First we argue that  $\mathbf{E}_{X\sim \mathcal{N}(\mu ,\mathbf{I})}[\mathbb{1}(p(X) > 100\mathrm{tr}(\mathbf{U}^{\top}\mathbf{U}))] < \epsilon /\log (1 / \epsilon)$  for all the polynomials  $p(x)$  described in Condition (2.c). Fix such a  $p(x)$  and observe that  $\mathbf{E}[p(x)] = \mathrm{tr}(\mathbf{U}^{\top}\mathbf{U})$ . Under the Gaussian distribution, by Hanson-Wright inequality (Fact B.13), we have that
$$
\mathbf {P r} _ {X \sim \mathcal {N} (\mu , \mathbf {I})} \left[ \left| (X - \mu) ^ {\top} \mathbf {U} ^ {\top} \mathbf {U} (X - \mu) - \operatorname {t r} (\mathbf {U} ^ {\top} \mathbf {U}) \right| > t \right] \leq 2 \exp \left(- 0. 1 \min  \left(\frac {t ^ {2}}{\| \mathbf {U} ^ {\top} \mathbf {U} \| _ {F} ^ {2}}, \frac {t}{\| \mathbf {U} ^ {\top} \mathbf {U} \| _ {\mathrm {o p}}}\right)\right).
$$
Let  $k' := \log(k / \epsilon) / 2$ . Using our assumption  $\| \mathbf{U}^\top \mathbf{U} \|_{\mathrm{op}} \leq \operatorname{tr}(\mathbf{U}^\top \mathbf{U}) / k'$ , Fact B.8 implies that  $\| \mathbf{U}^\top \mathbf{U} \|_{\mathrm{F}}^2 \leq \operatorname{tr}(\mathbf{U}^\top \mathbf{U})^2 / k'$ . Plugging in  $t = 99 \operatorname{tr}(\mathbf{U}^\top \mathbf{U})$  above, we obtain that
$$
\operatorname * {P r} _ {X \sim \mathcal {N} (\mu , \mathbf {I})} \left[ | (X - \mu) ^ {\top} \mathbf {U} ^ {\top} \mathbf {U} (X - \mu) \geq 1 0 0 \mathrm {t r} (\mathbf {U} ^ {\top} \mathbf {U}) | \right] \leq 2 \exp (- 4 8 k ^ {\prime}) = 2 \epsilon^ {2 4} / k ^ {2 4}, \tag {4}
$$
where we use the definition of  $k'$ . Moreover, integrating the tail inequality Equation (3) implies that  $\mathbf{E}[p^2(X)] \lesssim (\mathbf{E}[p(X)])^2 + \| \mathbf{U}^\top \mathbf{U} \|_F^2 \lesssim \mathrm{tr}(\mathbf{U}^\top \mathbf{U})^2$ . Alternatively, Equation (3) implies that with probability  $1 - \delta$ ,
$$
\begin{array}{l} p _ {\mathbf {U}} (x) \mathbb {1} \left(p _ {\mathbf {U}} (x) \geq 2 \mathbf {E} \left[ p _ {\mathbf {U}} (X) \right]\right) \leq 2 \left(p _ {\mathbf {U}} (x) - \mathbf {E} \left[ p _ {\mathbf {U}} (X) \right]\right) \mathbb {1} \left(p _ {\mathbf {U}} (x) \geq 2 \mathbf {E} \left[ p _ {\mathbf {U}} (X) \right]\right) \\ \lesssim \sqrt {k ^ {2} / k ^ {\prime}} \sqrt {\log (1 / \delta)} + (k / k ^ {\prime}) \log (1 / \delta) \\ \end{array}
$$
The covering argument is based on the following discretization of the unit sphere.
Fact B.22 (Cover of the sphere). Let  $r > 0$ . There exists a set  $\mathcal{C}$  of unit vectors of  $\mathbb{R}^d$ , such that  $|\mathcal{C}| \leq (1 + 2 / r)^d$  and for every  $v \in S^{d - 1}$  we have that  $\min_{y \in \mathcal{C}} \| y - v \|_2 \leq r$ .
Let  $\mathcal{V}$  be the set of all nearly orthogonal matrices:
$$
\mathcal {V} := \left\{\mathbf {U} \in \mathbb {R} ^ {k \times d}: \| \mathbf {U} ^ {\top} \mathbf {U} \| _ {\mathrm {o p}} \leq 2 \operatorname {t r} (\mathbf {U} ^ {\top} \mathbf {U}) / k ^ {\prime}, \operatorname {t r} (\mathbf {U} ^ {\top} \mathbf {U}) = k \right\}.
$$
Let  $\mathcal{V}_{\eta} \subset \mathcal{V}$  be the  $\eta$ -cover of  $\mathcal{V}$  for some  $\eta \in (0,1)$  such that for any  $\mathbf{U} \in \mathcal{V}$ , we have a matrix  $\mathbf{U}'$  such that  $\| \mathbf{U} - \mathbf{U}' \|_{\mathrm{F}} \leq \eta$ . We now bound the cardinality of  $\eta$ -cover  $\mathcal{V}_{\eta}$ . Looking it as a vector in  $\mathbb{R}^{dk}$  as flattened vector, there exists an  $\eta$ -cover of size  $2(3/\eta)^{dk}$ , where we also use that we can ensure  $\mathcal{V}_{\eta} \subset \mathcal{V}$  using [Ver18, Exercise 4.2.9].
For any  $\mathbf{U} \in \mathcal{V}$ , define  $p_{\mathbf{U}}(x) = \| \mathbf{U}x\|_2^2$  and  $\tau_{\mathbf{U}}(x) = p_{\mathbf{U}}(x)\mathbb{1}(p_{\mathbf{U}}(x) \geq 100\mathrm{tr}(\mathbf{U}^\top \mathbf{U}))$ . Let  $X_1, \ldots, X_n$  be  $n$  i.i.d. random variables from  $\mathcal{N}(0,\mathbf{I})$ . We first claim that for  $X \sim \mathcal{N}(0,\mathbf{I})$ ,  $\tau_{\mathbf{U}}(X) - \mathbf{E}[\tau_{\mathbf{U}}(X)]$  is a  $(O(k^2 / k'), O(k / k'))$ -subexponential random variable and  $\mathbf{E}[\tau_{\mathbf{U}}(X)] \lesssim \epsilon^3$ . For the latter, the Cauchy-Schwarz inequality along with Equation (3) applies that
$$
\begin{array}{l} \mathbf {E} [ \tau_ {\mathbf {U}} (X) ] = \mathbf {E} [ p _ {\mathbf {U}} (X) \mathbb {1} (p _ {\mathbf {U}} (x) \geq 1 0 0 \mathrm {t r} (\mathbf {U} ^ {\top} \mathbf {U})) ] \leq \sqrt {\mathbf {E} [ p _ {\mathbf {U}} ^ {2} (X) ]} \sqrt {\operatorname * {P r} (p _ {\mathbf {U}} (X) \geq 1 0 0 \mathrm {t r} (\mathbf {U} ^ {\top} \mathbf {U}))} ] \\ = O (k) O \left(\epsilon^ {1 2} / k ^ {1 2}\right), \\ \end{array}
$$
which is less than  $\epsilon^3$ . For the former, we have that with probability  $1 - \delta$ .
$$
\tau_ {\mathbf {U}} (x) - \mathbf {E} [ \tau_ {\mathbf {U}} (X) ] \leq p _ {\mathbf {U}} (x) \mathbb {1} (p _ {\mathbf {U}} (x) \geq 2 \mathbf {E} [ p _ {\mathbf {U}} (X) ]) \lesssim \sqrt {k ^ {2} / k ^ {\prime}} \log (1 / \delta) + (k / k ^ {\prime}) \log (1 / \delta).
$$
Thus, Lemma B.12 implies that  $\tau_{\mathbf{U}}(x) - \mathbf{E}[\tau_{\mathbf{U}}(X)]$  is a  $(O(k^2 /k'),O(k / k'))$  subexponential random variable. Applying the concentration properties of the mean of independent subexponential random variables (Lemma B.12), we obtain that for any fixed  $\mathbf{U}\in \mathcal{V}_{\eta}$ , with probability  $1 - \tau$
$$
\sum_ {i = 1} ^ {n} \frac {1}{n} \tau_ {\mathbf {U}} \left(x _ {i}\right) \lesssim \epsilon^ {3} + \sqrt {\frac {k ^ {2} \log (1 / \tau)}{k ^ {\prime} n}} + \frac {k \log (1 / \tau)}{k ^ {\prime} n}. \tag {5}
$$
Taking a union bound over every  $\mathbf{U} \in \mathcal{V}_{\eta}$ , we obtain the following: with probability  $1 - \tau$
$$
\forall U ^ {\prime} \in \mathcal {V} _ {\eta}: \quad \sum_ {i = 1} ^ {n} \frac {1}{n} \tau_ {\mathbf {U} ^ {\prime}} (x _ {i}) \lesssim \epsilon^ {3} + \sqrt {\frac {k ^ {2} \log (1 / \tau)}{k ^ {\prime} n}} + \frac {k \log (1 / \tau)}{k ^ {\prime} n} + \sqrt {\frac {d k ^ {3} \log (1 / \eta)}{k ^ {\prime} n}} + \frac {d k ^ {2} \log (1 / \eta)}{k ^ {\prime} n}. \tag {6}
$$
Let  $\mathbf{U} \in \mathcal{V}$  and  $\mathbf{U}' \in \mathcal{V}_{\eta}$  be arbitrary such that  $\| \mathbf{U} - \mathbf{U}' \|_{\mathrm{F}} \leq \eta$ . The goal now is to show that  $\sum_{i=1}^{n} \frac{1}{n} \tau_{\mathbf{U}}(x_i)$  and  $\sum_{i=1}^{n} \frac{1}{n} \tau_{\mathbf{U}'}(x_i)$  are close to each other and thus the former is bounded similarly to (6). Define the difference of the polynomials  $\Delta p(x) \coloneqq p_{\mathbf{U}}(x) - p_{\mathbf{U}'}(x)$ . We will also use the notation  $i \sim [n]$  to denote the averaging over  $[n]$  samples. First, we have that
$$
\begin{array}{l} \underset {i \sim [ n ]} {\mathbf {E}} \left[ \tau_ {\mathbf {U}} \left(x _ {i}\right) \right] = \underset {i \sim [ n ]} {\mathbf {E}} \left[ p _ {\mathbf {U}} \left(x _ {i}\right) \mathbb {1} \left(p _ {\mathbf {U}} \left(x _ {i}\right) > 1 0 0 k\right) \right] \\ = \mathbf {E} _ {i \sim [ n ]} \left[ p _ {\mathbf {U} ^ {\prime}} \left(x _ {i}\right) \mathbb {1} \left(p _ {\mathbf {U} ^ {\prime}} \left(x _ {i}\right) > 1 0 0 k - \Delta p \left(x _ {i}\right)\right) \right] + \mathbf {E} _ {i \sim [ n ]} \left[ \Delta p \left(x _ {i}\right) \mathbb {1} \left(p _ {\mathbf {U}} \left(x _ {i}\right) > 1 0 0 k\right) \right] \\ = \mathbf {E} _ {i \sim [ n ]} \left[ p _ {\mathbf {U} ^ {\prime}} \left(x _ {i}\right) \mathbb {1} \left(p _ {\mathbf {U} ^ {\prime}} \left(x _ {i}\right) > 1 0 0 k - \Delta p \left(x _ {i}\right)\right) \right] + \mathbf {E} _ {i \sim [ n ]} \left[ | \Delta p \left(x _ {i}\right) | \right]. \tag {7} \\ \end{array}
$$
We start with the second term of (7).
$$
\begin{array}{l} \underset {i \sim [ n ]} {\mathbf {E}} [ | \Delta p (x _ {i}) | ] = \underset {i \sim [ n ]} {\mathbf {E}} [ | \left<   \mathbf {U} ^ {\top} \mathbf {U} - \mathbf {U} ^ {\prime \top} \mathbf {U} ^ {\prime}, x _ {i} x _ {i} ^ {\top} \right> | ] \\ \leq \| \mathbf {U} ^ {\top} \mathbf {U} - \mathbf {U} ^ {\prime \top} \mathbf {U} ^ {\prime} \| _ {\mathrm {F}} \underset {i \sim [ n ]} {\mathbf {E}} [ \| x _ {i} \| _ {2} ^ {2} ] \\ \lesssim d ^ {2} \eta , \\ \end{array}
$$
where we used two things: First that by Gaussian norm concentration (e.g., Hanson-Wright inequality with  $A = I$  combined with Lemma B.12):
$$
\frac {1}{n} \sum_ {i = 1} ^ {n} \| x _ {i} \| _ {2} ^ {2} \leq d + O \left(\sqrt {d} \sqrt {\frac {\log (1 / \delta)}{n}} + \frac {\log (1 / \delta)}{n}\right) = O (d), \tag {8}
$$
if  $n\gg \log (1 / \delta)$  and, second, we used that
$$
\begin{array}{l} \left\| \mathbf {U} ^ {\top} \mathbf {U} - \mathbf {U} ^ {\prime \top} \mathbf {U} ^ {\prime} \right\| _ {\mathrm {F}} = \left\| \mathbf {U} ^ {\top} \left(\mathbf {U} - \mathbf {U} ^ {\prime}\right) + \left(\mathbf {U} - \mathbf {U} ^ {\prime}\right) ^ {\top} \mathbf {U} ^ {\prime} \right\| _ {\mathrm {F}} \\ \leq 2 \max  \left(\| \mathbf {U} \| _ {\mathrm {o p}}, \| \mathbf {U} ^ {\prime} \| _ {\mathrm {o p}}\right) \| \mathbf {U} - \mathbf {U} ^ {\prime} \| _ {\mathrm {F}} \leq 2 d \eta . \\ \end{array}
$$
We now bound the first term in (7),
$$
\begin{array}{l} \underset {i \sim [ n ]} {\mathbf {E}} \left[ p _ {\mathbf {U} ^ {\prime}} (x _ {i}) \mathbb {1} \left(p _ {\mathbf {U} ^ {\prime}} (x _ {i}) > 1 0 0 k - \Delta p (x _ {i})\right) \right] \\ = \underset {i \sim [ n ]} {\mathbf {E}} \left[ p _ {\mathbf {U} ^ {\prime}} (x _ {i}) \mathbb {1} (p _ {\mathbf {U} ^ {\prime}} (x _ {i}) > 1 0 0 k - \Delta p (x _ {i})) \mathbb {1} (\Delta p (x _ {i}) \leq k) \right] \\ + \underset {i \sim [ n ]} {\mathbf {E}} \left[ p _ {\mathbf {U} ^ {\prime}} \left(x _ {i}\right) \mathbb {1} \left(p _ {\mathbf {U} ^ {\prime}} \left(x _ {i}\right) > 1 0 0 k - \Delta p \left(x _ {i}\right)\right) \mathbb {1} \left(\Delta p \left(x _ {i}\right) > k\right) \right] \\ \leq \mathbf {E} _ {i \sim [ n ]} \left[ p _ {\mathbf {U} ^ {\prime}} \left(x _ {i}\right) \mathbb {1} \left(p _ {\mathbf {U} ^ {\prime}} \left(x _ {i}\right) > 9 9 k\right) \right] + \mathbf {E} _ {i \sim [ n ]} \left[ | p _ {\mathbf {U} ^ {\prime}} \left(x _ {i}\right) | \mathbb {1} \left(\Delta p \left(x _ {i}\right) > k\right) \right] \tag {9} \\ \end{array}
$$
The first term is bounded by  $(6)^{8}$
There is a small difference in the constant in front of  $k$ , but the same proof gives similar quantitative bounds.
Choose  $\eta = 0.001\epsilon^3 /d^2$ . Then second term in (9),  $\mathbf{E}_{i\sim [n]}[|p_{\mathbf{U}'}(x_i)|\mathbb{1}(\Delta p(x_i) > k)]$  is zero unless there exists an  $x_{i}$  in the sample set with  $\| x_{i}\| >10\sqrt{d}$ . This relies on the fact that  $\Delta p(x)\leq \| x\| _2^2\| \mathbf{U}^\top \mathbf{U} - \mathbf{U}'^\top \mathbf{U}\|_{\mathrm{F}}\leq 2d\eta \| x\| _2^2$  which becomes less than  $k$  if  $\| x\| _2^2\leq 100d$  and  $\eta = 0.001\epsilon^3 /d^2$ . But the probability of the event  $\| x_{i}\| >10\sqrt{d}$  is exponentially small:
$$
\Pr_ {X _ {1}, \dots , X _ {n} \sim \mathcal {N} (0, I)} [ \exists \in S: \| X _ {i} \| _ {2} > 1 0 \sqrt {d} ] \leq n e ^ {- d / 1 0 0}, \tag {10}
$$
by Gaussian norm concentration and a union bound.
Therefore taking  $\eta = 0.001\epsilon^3 /d^2$ , we have that with high probability the scores from  $\mathcal{V}$  and  $\mathcal{V}_{\eta}$  are close up to  $\epsilon^3$ . Combining this with Equation (6) and the union bound over  $\mathcal{E}$  when  $n\gg d + \log (1 / \delta)$ , we obtain that with probability at least  $1 - \delta /2 - ne^{-d / 100}$ , for all  $\mathbf{U}\in \mathcal{V}$ :
$$
\sum_ {i = 1} ^ {n} \frac {1}{n} \tau_ {\mathbf {U}} (x) \lesssim \epsilon^ {3} + \sqrt {\frac {k ^ {2} \log (1 / \tau)}{k ^ {\prime} n}} + \frac {k \log (1 / \tau)}{k ^ {\prime} n} + \sqrt {\frac {k ^ {3} d \log (d / \epsilon)}{k ^ {\prime} n}} + \frac {d k ^ {2} \log (d / \epsilon)}{k ^ {\prime} n}.
$$
Thus, as soon as  $n = C \frac{k^3 (d \log(d / \epsilon) + \log(1 / \tau))}{k' \epsilon^2 / \log^2(1 / \epsilon)}$  for a sufficiently constant  $C$ , and  $\epsilon \in (0, \epsilon_0)$  a sufficiently small constant  $\epsilon_0$ , all of the terms above become less than  $\epsilon / \log(1 / \epsilon)$ . We also need  $ne^{-d / 100}$  to be at most  $\delta / 2$ . Using the same  $n$  as before, the quantity  $ne^{-d / 100}$  is at most  $\delta / 2$  if  $d \gg \text{poly}(\epsilon^{-1} \delta^{-1})$ . We can assume without loss of generality that the last condition is always true as we can artificially augment the dimension by padding the data with Gaussian coordinates until the resulting dimension becomes  $d' \gg \text{poly}(\epsilon^{-1} \delta^{-1})$  and it is easy to see that once the goodness conditions get established for the augmented data, they continue to hold for the original data.
B.4.2 Helper Lemmata Related to Goodness Condition
We now record implications of the goodness conditions that we will use in our analysis of the main algorithm. The first lemma below provides a certification that when the operator norm of the empirical covariance is suitably bounded, the empirical mean closely approximates the true mean. This allows us to halt the outlier filtering process and return the resulting vector.
Lemma B.23 (Certificate Lemma (restated)). Let  $0 < \alpha < \epsilon < 1/4$  and  $\delta \in (0,1)$ . Let  $P = (1 - \epsilon)G + \epsilon B$  be a mixture of distributions, where  $G$  satisfies Conditions (2.a) and (2.b) of Definition 2.1 with respect to  $\mu \in \mathbb{R}^d$ . Let  $w(x)$  be such that  $\mathbf{E}_{X \sim G}[w(X)] > 1 - \alpha$ . Assume that the top eigenvalue of  $\Sigma_{P_w}$  is less than  $1 + \lambda$ . Then,
$$
\left\| \mu_ {P _ {w}} - \mu \right\| _ {2} \lesssim \alpha \sqrt {\log (1 / \alpha)} + \sqrt {\lambda \epsilon} + \epsilon + \sqrt {\alpha \epsilon \log (1 / \alpha)}.
$$
Proof. Let  $\rho = \epsilon \mathbf{E}_{X\sim B}[w(X)] / \mathbf{E}_{X\sim P}[w(X)]$ . Denote  $P_{w}(x) = w(x)P(x) / \mathbf{E}_{X\sim P}[w(x)]$ ,  $B_{w}(x) = w(x)B(x) / \mathbf{E}_{X\sim B}[w(X)]$ ,  $G_{w}(x) = w(x)G(x) / \mathbf{E}_{X\sim G}[w(X)]$  the weighted by  $w(x)$  versions of the distributions  $P, B, G$  and denote by  $\Sigma_{P_w}, \Sigma_{B_w}, \Sigma_{G_w}$  their covariance matrices. We can write:
$$
\boldsymbol {\Sigma} _ {P _ {w}} = \rho \boldsymbol {\Sigma} _ {B _ {w}} + (1 - \rho) \boldsymbol {\Sigma} _ {G _ {w}} + \rho (1 - \rho) \left(\mu_ {G _ {w}} - \mu_ {B _ {w}}\right) \left(\mu_ {G _ {w}} - \mu_ {B _ {w}}\right) ^ {\top}. \tag {11}
$$
Let  $v$  be the vector in the direction of  $\mu_{G_w} - \mu_{B_w}$ . Since the top eigenvalue of  $\Sigma_{P_w}$  is less than  $1 + \lambda$ , we obtain the following:
$$
\begin{array}{l} 1 + \lambda \geq v ^ {\top} \Sigma_ {P _ {w}} v \geq (1 - \rho) v ^ {\top} \Sigma_ {G _ {w}} v + \rho (1 - \rho) \left(v ^ {\top} \left(\mu_ {B _ {w}} - \mu_ {G _ {w}}\right)\right) ^ {2} \\ \geq (1 - \rho) (1 - \alpha \log (1 / \alpha)) + \rho (1 - \rho) \left(v ^ {\top} \left(\mu_ {B _ {w}} - \mu_ {G _ {w}}\right)\right) _ {2} ^ {2}, \\ \end{array}
$$
where the second step uses Equation (11) and the last step uses that  $G$  satisfies Conditions (2.a) and (2.b) of Definition 2.1. The expression above implies the following:
$$
(v ^ {\top} \left(\mu_ {B _ {w}} - \mu_ {G _ {w}}\right)) ^ {2} \leq \frac {\lambda + \rho + \alpha \log (1 / \alpha)}{\rho (1 - \rho)}.
$$
We can now bound the error  $\| \mu_{P_w} - \mu \|_2$  as follows:
$$
\left\| \mu_ {P _ {w}} - \mu \right\| _ {2} = \left| v ^ {\top} \left(\mu_ {G _ {w}} - \mu\right) + \rho \left(\mu_ {B _ {w}} - \mu_ {G _ {w}}\right) \right|
$$
$$
\begin{array}{l} \leq | v ^ {\top} \left(\mu_ {G _ {w}} - \mu\right) | + \rho | v ^ {\top} \left(\mu_ {B _ {w}} - \mu_ {G _ {w}}\right) | \\ \leq \left\| \mu_ {G _ {w}} - \mu \right\| _ {2} + \rho | v ^ {\top} \left(\mu_ {B _ {w}} - \mu_ {G _ {w}}\right) | \\ \leq \alpha \sqrt {\log (1 / \alpha)} + \sqrt {\rho} \sqrt {\frac {\lambda + \rho + \alpha \log (1 / \alpha)}{1 - \rho}}, \\ \end{array}
$$
where the last inequality uses that  $G$  satisfies Conditions (2.a) and (2.b). We now use bounds on  $\rho$  to simplify the terms. Recall that  $\rho = \epsilon \mathbf{E}_{X\sim B}[w(X)] / \mathbf{E}_{X\sim P}[w(X)] \leq \epsilon /(1 - \alpha)$ . As  $\alpha < 1 / 2$ , we get that  $\rho < 2\epsilon$ . In addition, note that  $\rho < 1 / 2$
Recall the goodness conditions are phrased in terms of the deviation from the true mean. However, the true mean is unknown to the algorithm and the algorithm will use the empirical mean as an approximation, which introduces additional errors in our bounds. To analyze these errors, we require the following lemmata.
Lemma B.24 (see, e.g., Lemma 2.13 in [DKPP22]). Let  $w: \mathbb{R}^d \to [0,1]$  such that  $\mathbf{E}_{X \sim G}[w(X)] \geq 1 - \alpha$  and let  $G$  be distribution satisfying Conditions (2.a) and (2.b) of Definition 2.1 with respect to  $\mu \in \mathbb{R}^d$  with parameters  $(\epsilon, \alpha, k)$ . For any matrix  $\mathbf{U} \in \mathbb{R}^{m \times d}$  and any vector  $b \in \mathbb{R}^d$ , we have that
$$
\underset {X \sim G _ {w}} {\mathbf {E}} \left[ \| \mathbf {U} (X - b) \| _ {2} ^ {2} \right] = \| \mathbf {U} \| _ {\mathrm {F}} ^ {2} (1 \pm \alpha \log (1 / \alpha)) + \| \mathbf {U} (\mu - b) \| _ {2} ^ {2} \pm 2 \alpha \sqrt {\log (1 / \alpha)} \| \mathbf {U} \| _ {\mathrm {F}} ^ {2} \| \mu - b \| _ {2}.
$$
Lemma B.25. Let  $w: \mathbb{R}^d \to [0,1]$  such that  $\mathbf{E}_{X \sim G}[w(X)] \geq 1 - \alpha$  and let  $G$  be an  $(\epsilon, \alpha, k)$ -good distribution with respect to  $\mu \in \mathbb{R}^d$  as in Definition 2.1. For any matrix  $\mathbf{U} \in \mathbb{R}^{k \times d}$ , if we define the polynomial  $\tau(x) := \| \mathbf{U}(x - b) \|_2^2 \mathbb{1}(\| \mathbf{U}(x - b) \|_2^2 > 100 \mathrm{tr}(\mathbf{U}^\top \mathbf{U}))$  and assume that  $\| \mathbf{U}(\mu - b) \|_2^2 \leq \mathrm{tr}(\mathbf{U}^\top \mathbf{U})$ , then it holds
$$
\underset {X \sim G _ {w}} {\mathbf {E}} [ \tau (X) ] \leq \frac {2 \epsilon}{\log (1 / \epsilon)} \operatorname {t r} (\mathbf {U} ^ {\top} \mathbf {U}) + 2 \| \mathbf {U} (\mu - b) \| _ {2} ^ {2}.
$$
$$
\begin{array}{l} \| \mathbf {U} (x - b) \| _ {2} ^ {2} \leq 2 \| \mathbf {U} (x - \mu) \| _ {2} ^ {2} + 2 \| \mathbf {U} (\mu - b) \| _ {2} ^ {2} \\ \leq 2 \| \mathbf {U} (x - \mu) \| _ {2} ^ {2} + 2 \operatorname {t r} \left(\mathbf {U} ^ {\top} \mathbf {U}\right), \\ \end{array}
$$
where the first step uses the triangle inequality combined with the inequality  $(a + b)^2 \leq 2a^2 + 2b^2$ . Using the above, we can write
$$
\begin{array}{l} \underset {X \sim G _ {w}} {\mathbf {E}} [ \tau (X) ] \leq 2 \underset {X \sim G _ {w}} {\mathbf {E}} [ \| \mathbf {U} (x - \mu) \| _ {2} ^ {2} \mathbb {1} (\| \mathbf {U} (x - b) \| _ {2} ^ {2} > 4 9 \operatorname {t r} (\mathbf {U} ^ {\top} \mathbf {U}) ] + 2 \| \mathbf {U} (\mu - b) \| _ {2} ^ {2} \\ \leq 2 \epsilon / \log (1 / \epsilon) + 2 \| \mathbf {U} (\mu - b) \| _ {2} ^ {2}, \\ \end{array}
$$
where the last line uses the goodness condition for  $G$ .
Lastly, the following lemma shows that if the matrix  $\mathbf{U}$  used in the last goodness condition consists of nearly orthogonal vectors, then the matrix satisfies the near-orthogonality condition mentioned there.
Lemma B.26. Let  $u_{i}$  for  $i \in [k]$  be vectors in  $\mathbb{R}^d$  and define  $\mathbf{U} = [u_1, \ldots, u_k]$ . If for every  $i, j \in [k]$  with  $i \neq j$ , it holds  $|\langle u_i, u_j \rangle| \leq \| u_i \|_2 \| u_j \|_2 / k^2$ , and  $\max_i \| u_i \|_2^2 \leq \frac{\log k}{k} \operatorname{tr}(\mathbf{U}^\top \mathbf{U})$  then  $\| \mathbf{U}^\top \mathbf{U} \|_{\mathrm{op}} \leq 2 \operatorname{tr}(\mathbf{U}^\top \mathbf{U}) / (k / \log k)$ .
Proof. We note that by the Gershgorin circle theorem (Fact B.4),
$$
\left\| \mathbf {U} ^ {\top} \mathbf {U} \right\| _ {\mathrm {o p}} \leq \max  _ {i \in [ k ]} \left\| u _ {i} \right\| _ {2} ^ {2} + \sum_ {i \neq j} | \langle u _ {i}, u _ {j} \rangle |
$$
<sup>9</sup>This lemma does not use the last goodness condition; thus the parameter  $k$  does not appear in the conclusion.
$$
\begin{array}{l} \leq \max _ {i \in [ k ]} \| u _ {i} \| _ {2} ^ {2} + k ^ {2} \frac {\| u _ {i} \| _ {2} \| u _ {j} \| _ {2}}{k ^ {2}} \\ \leq 2 \max _ {i \in [ k ]} \| u _ {i} \| _ {2} ^ {2} \leq 2 \frac {\operatorname {t r} (\mathbf {U} ^ {\top} \mathbf {U})}{k / \log (k)}. \\ \end{array}
$$
B.5 Filtering Procedure
In this section, we formally describe the filtering procedure (Algorithm 3) and prove its formal guarantee (Lemma 2.5).
Algorithm 3 Down-weighting Filter
1: Input: Distribution  $P$  on  $n$  points, weight  $w(x)$  for each point  $x$ , score  $\tilde{\tau}(x)$  for every point  $x$  with the guarantee  $\tilde{\tau}(x) < r$ , threshold  $T > 0$ , parameters  $\beta > 0$ ,  $\ell_{\max} \in \mathbb{Z}_+$ .
2: Output: New weights  $w'(x)$ .
3: Initialize  $w'(x) = w(x)$ .
4:  $\ell_{\mathrm{max}}\gets r / (eT)$
5: for  $i = 1,\dots ,\ell_{\mathrm{max}}$  do  $\triangleright$  Can be implemented in  $O(\log \ell_{\mathrm{max}})$  with Binary Search
6: if  $\mathbf{E}_{X\sim P}[w'(X)\tilde{\tau} (X)] > T\beta$  then
7:  $w^{\prime}(x)\gets w(x)(1 - \tilde{\tau} (x) / r).$
8: else return  $w'$ .
9: return  $w'$ .
Lemma B.27 (Filtering Guarantee). Let  $P = (1 - \epsilon)G + \epsilon B$  be a mixture of distributions supported on  $n$  points and  $\beta > 1$ . If  $(1 - \epsilon)\mathbf{E}_{X\sim G}[w(X)\tilde{\tau} (X)] < T$ ,  $\| \tilde{\tau}\|_{\infty}\leq r$  and  $\ell_{\mathrm{max}} > r / T$  then the new weights  $w^{\prime}(x)$  satisfy: