SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \boldsymbol {\Sigma} _ {t + 1} ^ {\perp} = \mathbf {E} _ {X \sim P _ {t + 1}} \left[ \left(\operatorname {P r o j} _ {\mathcal {V} _ {t + 1} ^ {\perp}} (X) - \mu_ {t + 1} ^ {\perp}\right) \left(\operatorname {P r o j} _ {\mathcal {V} _ {t + 1} ^ {\perp}} (X) - \mu_ {t + 1} ^ {\perp}\right) ^ {\top} \right] \\ \preceq_ {X \sim P _ {t + 1}} \mathbf {E} \left[ \left(\operatorname {P r o j} _ {\mathcal {V} _ {t + 1} ^ {\perp}} (X) - \mu_ {t} ^ {\perp}\right) \left(\operatorname {P r o j} _ {\mathcal {V} _ {t + 1} ^ {\perp}} (X) - \mu_ {t} ^ {\perp}\right) ^ {\top} \right] \\ = \frac {1}{\mathbf {E} _ {X \sim P} \left[ w _ {t + 1} (X) \right]} \underset {X \sim P} {\mathbf {E}} \left[ w _ {t + 1} (X) \left(\operatorname {P r o j} _ {\mathcal {V} _ {t + 1} ^ {\perp}} (X) - \mu_ {t} ^ {\perp}\right) \left(\operatorname {P r o j} _ {\mathcal {V} _ {t + 1} ^ {\perp}} (X) - \mu_ {t} ^ {\perp}\right) ^ {\top} \right]. \tag {17} \\ \end{array}
$$
To avoid the repetitive notation of  $\mathrm{Proj}_{\mathcal{V}_{t + 1}^{\perp}}(X)$  in the remainder of the proof, we introduce the shorthand notation  $X^{\perp}$  to denote the projection of the random variable  $X$  to  $\mathcal{V}_{t + 1}^{\perp}$ . To avoid confusion for later on, we also note that since we are in the case that the algorithm enters Line 13,  $\mathcal{V}_{t + 1}^{\perp} = \mathcal{V}_t^{\perp}$ , i.e., no change is done to the subspace and in this subsection it does not matter which of the two subspaces we use in our notation.
We decompose  $\mathrm{tr}(\mathbf{M}_t^\perp \mathbf{B}_{t + 1}^\perp \mathbf{M}_t^\perp)$  into the contribution from inliers and outliers: Using the definition of  $\mathbf{B}_{t + 1}^{\perp} = \mathbf{E}_{X\sim P}[w_{t + 1}(X)]^{2}\boldsymbol{\Sigma}_{t + 1}^{\perp} - (1 - C_1\epsilon)\boldsymbol{\Pi}_{\mathcal{V}_t^\perp}$  and Equation (17), we have that
$$
\begin{array}{l} \operatorname {t r} \left(\mathbf {M} _ {t} ^ {\perp} \mathbf {B} _ {t + 1} ^ {\perp} \mathbf {M} _ {t} ^ {\perp}\right) (18) \\ \leq \mathbf {E} _ {X \sim P} [ w _ {t + 1} (X) ] \mathbf {E} _ {X \sim P} [ w _ {t + 1} (X) \| \mathbf {M} _ {t} ^ {\perp} (X ^ {\perp} - \mu_ {t} ^ {\perp}) \| _ {2} ^ {2} ] - (1 - C _ {1} \epsilon) \operatorname {t r} \left(\mathbf {M} _ {t} ^ {\perp} \boldsymbol {\Pi} _ {\mathcal {V} _ {t} ^ {\perp}} \mathbf {M} _ {t} ^ {\perp}\right) (19) \\ \leq (1 - \epsilon) \underset {X \sim G} {\mathbf {E}} [ w _ {t + 1} (X) \| \mathbf {M} _ {t} ^ {\perp} (X ^ {\perp} - \mu_ {t} ^ {\perp}) \| _ {2} ^ {2} ] + \epsilon \underset {X \sim B} {\mathbf {E}} [ w _ {t + 1} (X) \| \mathbf {M} _ {t} ^ {\perp} (X ^ {\perp} - \mu_ {t} ^ {\perp}) \| _ {2} ^ {2} ] (20) \\ - \left(1 - C _ {1} \epsilon\right) \operatorname {t r} \left(\left. \mathbf {M} _ {t} ^ {\perp}\right) ^ {2}\right) \\ \leq \mathbf {E} _ {X \sim G} \left[ w _ {t + 1} (X) \| \mathbf {U} _ {t} \left(X ^ {\perp} - \mu_ {t} ^ {\perp}\right) \| _ {2} ^ {2} \right] + \frac {1 . 2 5 \epsilon}{k} \mathbf {E} _ {X \sim B} \left[ w _ {t + 1} (X) \| \mathbf {U} _ {t} \left(X ^ {\perp} - \mu_ {t} ^ {\perp}\right) \| _ {2} ^ {2} \right] (21) \\ - \left(1 - C _ {1} \epsilon\right) \| \mathbf {M} _ {t} ^ {\perp} \| _ {\mathrm {F}} ^ {2}, (22) \\ \end{array}
$$
where the first inequality uses the definition of  $\mathbf{B}_t$ , Fact B.5 and Equation (17), for the last term in Equation (19) we used that  $\Pi_{\mathcal{V}_t^\perp}\mathbf{M}_t^\perp = \mathbf{M}_t^\perp$  which can be shown as follows: recall that using our definitions  $\mathbf{M}_t^\perp = (\mathbf{B}_t^\perp)^p = \left(\Pi_{\mathcal{V}_t^\perp}\mathbf{B}_t\Pi_{\mathcal{V}_t^\perp}\right)^p$ , then since  $\Pi_{\mathcal{V}_t^\perp}$  is idempotent, multiplying that expression from any side does not change it. For Equation (21) we used  $1 - \epsilon < 1$  and the Johnson-Lindenstrauss sketch guarantee (condition (iii) of Condition C.1).
We start by upper bounding the combined contribution of the first and last term. We will need our assumption that  $G$  (the uniform distribution on inlier samples) satisfies conditions (2.b) and (2.a) from Definition 2.1 with  $\alpha = \epsilon / \log(1/\epsilon)$ , in order to apply Lemma B.24 with  $\mathbf{U} = \mathbf{M}_t^\perp$  and  $b = \mu_t^\perp$ :
$$
\begin{array}{l} \underset {X \sim G} {\mathbf {E}} [ w _ {t + 1} (X) \| \mathbf {M} _ {t} ^ {\perp} \left(X ^ {\perp} - \mu_ {t} ^ {\perp}\right) \| _ {2} ^ {2} ] - (1 - C _ {1} \epsilon) \| \mathbf {M} _ {t} ^ {\perp} \| _ {\mathrm {F}} ^ {2} (23) \\ \leq \left(1 + O (\epsilon)\right) \| \mathbf {M} _ {t} ^ {\perp} \| _ {\mathrm {F}} ^ {2} + \| \mathbf {M} _ {t} ^ {\perp} \left(\mu^ {\perp} - \mu_ {t} ^ {\perp}\right) \| _ {2} ^ {2} + O (\epsilon) \| \mathbf {M} _ {t} ^ {\perp} \| _ {\mathrm {F}} ^ {2} \| \mu^ {\perp} - \mu_ {t} ^ {\perp} \| _ {2} - (1 - C _ {1} \epsilon) \| \mathbf {M} _ {t} ^ {\perp} \| _ {\mathrm {F}} ^ {2} (24) \\ \leq \left(1 + O (\epsilon)\right) \| \mathbf {M} _ {t} \| _ {\mathrm {F}} ^ {2} + \| \mathbf {M} _ {t} \| _ {\mathrm {F}} ^ {2} O \left(\| \mathbf {B} _ {t} ^ {\perp} \| _ {\mathrm {o p}} \epsilon\right) + O (\epsilon) \| \mathbf {M} _ {t} \| _ {\mathrm {F}} ^ {2} O \left(\sqrt {\| \mathbf {B} _ {t} ^ {\perp} \| _ {\mathrm {o p}} \epsilon}\right) - (1 - C _ {1} \epsilon) \| \mathbf {M} _ {t} ^ {\perp} \| _ {\mathrm {F}} ^ {2} (25) \\ \leq O (1 / C) \left\| \mathbf {B} _ {t} ^ {\perp} \right\| _ {\mathrm {o p}} \left\| \mathbf {M} _ {t} \right\| _ {\mathrm {F}} ^ {2}, (26) \\ \end{array}
$$
where Equation (24) uses Lemma B.24, Equation (25) uses  $\| \mathbf{M}_t(x^\perp -\mu_t^\perp)\| _2^2 = \mathrm{tr}(\mathbf{M}_t^\top \mathbf{M}_t(x^\perp -$ $\mu_t^\perp)(x - \mu_t^\perp)^\top)$  and then the fact  $\operatorname {tr}(\mathbf{AB})\leq \operatorname {tr}(\mathbf{A})\| \mathbf{B}\|_{\mathrm{op}}$  for  $\mathbf{A} = \mathbf{M}_t^\top \mathbf{M}_t$ $\mathbf{B} = (x^{\perp} - \mu_{t}^{\perp})(x-$ $\mu_t^\perp)^\top$  as well as the certificate lemma (Lemma 2.4) which gives a bound where it can be checked that the dominant term is  $O(\sqrt{\epsilon\|\mathbf{B}_t^\perp\|_{\mathrm{op}}})$ , and Equation (26) uses that  $\epsilon < O(1 / C)\| \mathbf{B}_t^\perp \|_{\mathrm{op}}$  due to Line 9 of Algorithm 1 and condition (i) of Condition C.1.
We now focus on upper bounding the middle term of Equation (21), where we will use the guarantee of filtering, in particular the second part of Lemma 2.5. By Lemma B.26, the deterministic conditions (vi) (pairwise angles close to 90 degrees) and (iv) from Condition C.1 imply that the goodness condition (2.c) is applicable to the polynomial  $\| \mathbf{U}_t(x - \mu)\| _2^2$  where  $\mathbf{U}_t$  is the matrix used in Line 15. In other words, the requirement of Lemma 2.5 is satisfied and the lemma is applicable to our case. We thus obtain that
$$
\frac {\epsilon}{k} \underset {X \sim_ {B}} {\mathbf {E}} [ w _ {t + 1} (X) \| \mathbf {U} _ {t} (X ^ {\perp} - \mu^ {\perp}) \| _ {2} ^ {2} ] \lesssim \frac {\epsilon}{k} \| \mathbf {U} _ {t} \| _ {\mathrm {F}} ^ {2} \lesssim \epsilon \| \mathbf {M} _ {t} \| _ {\mathrm {F}} ^ {2} \lesssim \frac {1}{C} \| \mathbf {B} _ {t} ^ {\perp} \| _ {\mathrm {o p}} \| \mathbf {M} _ {t} \| _ {\mathrm {F}} ^ {2},
$$
where the second inequality uses the Johnson-Lindenstrauss guarantee (condition (iii) of Condition C.1) and the last inequality uses that  $\epsilon < O(1 / C)\| \mathbf{B}_t^\perp \|_{\mathrm{op}}$ , which is due to Line 9 of Algorithm 1 and Condition (i) of Condition C.1.
C.3 Case 2
This section contains the details missing from Section 3.2. We begin by proving the formal version of Claim 3.4 below:
<sup>11</sup>Since we are working with the projection of the data to  $\mathcal{V}_t^\perp$ , this lemma requires that the goodness conditions hold for the projected data, which as we have noted before holds (see Remark B.19).
Claim C.5 (Formal version of Claim 3.4). If  $\mathbf{Pr}_{z,z^{\prime}\sim \mathcal{N}(0,\mathbf{I})}\left[\frac{|\langle\mathbf{M}_t^\perp z,\mathbf{M}_t^\perp z^{\prime}\rangle|}{\|\mathbf{M}_t^\perp z\|_2\|\mathbf{M}_t^\perp z^{\prime}\|_2} >\gamma \right]\geq \alpha$  then  $\frac{\|\mathbf{M}_t^\perp\|_{\mathrm{op}}^2}{\mathrm{tr}((\mathbf{M}_t^\perp)^2)}\gtrsim \gamma^2\alpha^5.$
Proof. For some parameter  $\delta > 0$ , define the following events for random variables  $z, z' \colon \mathcal{E}_1 = \{z : \| \mathbf{M}_t^\perp z\|_2^2 > \mathrm{tr}((\mathbf{M}_t^\perp)^2)\delta\}$ , and  $\mathcal{E}_2 = \{z : \| \mathbf{M}_t^\perp z' \|_2^2 > \mathrm{tr}((\mathbf{M}_t^\perp)^2)\delta\}$ . We have the following series of inequalities:
$$
\begin{array}{l} \alpha \leq \mathbf {P r} _ {z, z ^ {\prime} \sim \mathcal {N} (0, \mathbf {I})} \left[ \left| \frac {\langle \mathbf {M} _ {t} ^ {\perp} z , \mathbf {M} _ {t} ^ {\perp} z ^ {\prime} \rangle}{\| \mathbf {M} _ {t} ^ {\perp} z \| _ {2} \| \mathbf {M} _ {t} ^ {\perp} z ^ {\prime} \| _ {2}} \right| > \gamma \right] \\ \leq \mathbf {P r} _ {z, z ^ {\prime} \sim \mathcal {N} (0, \mathbf {I})} \left[ \left| \frac {\langle \mathbf {M} _ {t} ^ {\perp} z , \mathbf {M} _ {t} ^ {\perp} z ^ {\prime} \rangle}{\| \mathbf {M} _ {t} ^ {\perp} z \| _ {2} \| \mathbf {M} _ {t} ^ {\perp} z ^ {\prime} \| _ {2}} \right| > \gamma \wedge \mathcal {E} _ {1} \wedge \mathcal {E} _ {2} \right] + \mathbf {P r} [ \overline {{\mathcal {E} _ {1}}} \vee \overline {{\mathcal {E} _ {2}}} ] \\ \leq \mathbf {P r} _ {z, z ^ {\prime} \sim \mathcal {N} (0, \mathbf {I})} \left[ \left| \frac {\left. \langle \mathbf {M} _ {t} ^ {\perp} z , \mathbf {M} _ {t} ^ {\perp} z ^ {\prime} \right\rangle}{\operatorname {t r} \left(\left(\mathbf {M} _ {t} ^ {\perp}\right) ^ {2}\right) \delta} \right| > \gamma \wedge \mathcal {E} _ {1} \wedge \mathcal {E} _ {2} \right] + 2 \sqrt {e \delta} (27) \\ \leq \mathbf {P r} _ {z, z ^ {\prime} \sim \mathcal {N} (0, \mathbf {I})} \left[ \left| \frac {\langle \mathbf {M} _ {t} ^ {\perp} z , \mathbf {M} _ {t} ^ {\perp} z ^ {\prime} \rangle}{\operatorname {t r} ((\mathbf {M} _ {t} ^ {\perp}) ^ {2}) \delta} \right| > \gamma \right] + 2 \sqrt {e \delta} \\ \leq \frac {1}{\gamma^ {2}} \frac {\mathbf {E} _ {z , z ^ {\prime} \sim \mathcal {N} (0 , \mathbf {I})} [ \langle \mathbf {M} _ {t} ^ {\perp} z , \mathbf {M} _ {t} ^ {\perp} z ^ {\prime} \rangle^ {2} ]}{\operatorname {t r} ((\mathbf {M} _ {t} ^ {\perp}) ^ {2}) ^ {2} \delta^ {2}} + 2 \sqrt {e \delta} (28) \\ \leq \frac {1}{\gamma^ {2}} \frac {\left\| \left(\mathbf {M} _ {t} ^ {\perp}\right) ^ {2} \right\| _ {\mathrm {F}} ^ {2}}{\operatorname {t r} \left(\left(\mathbf {M} _ {t} ^ {\perp}\right) ^ {2}\right) ^ {2} \delta^ {2}} + 2 \sqrt {e \delta}, (29) \\ \end{array}
$$
where Equation (27) is obtained by a union bound along with the second part of Fact B.15, Equation (28) is obtained by squaring and using Markov's inequality, and Equation (29) follows by using the first part of Fact B.15.
Picking the value  $\delta = \alpha^2 / 1000$  and reorganizing Equation (29) implies the lower bound  $\| (\mathbf{M}_t^\perp)^2\|_{\mathrm{F}} / \mathrm{tr}((\mathbf{M}_t^\perp)^2) \gtrsim \alpha^5\gamma^2$ . Combining that with the upper bound  $\|\mathbf{A}\|_{\mathrm{F}}^2 \leq \mathrm{tr}(\mathbf{A})\|\mathbf{A}\|_{\mathrm{op}}$  for any PSD matrix  $\mathbf{A}$  (Fact B.8), we obtain:
$$
\alpha^ {5} \gamma^ {2} \lesssim \frac {\| (\mathbf {M} _ {t} ^ {\perp}) ^ {2} \| _ {\mathrm {F}}}{\operatorname {t r} \left((\mathbf {M} _ {t} ^ {\perp}) ^ {2}\right)} \leq \sqrt {\frac {\| (\mathbf {M} _ {t} ^ {\perp}) ^ {2} \| _ {\mathrm {o p}}}{\operatorname {t r} \left((\mathbf {M} _ {t} ^ {\perp}) ^ {2}\right)}}. \tag {30}
$$
We now turn to the second claim, using the same notation as in the pseudocode. The requirement that  $\mathbf{B}_t$  is positive semi-definite (PSD) has already been established. As shown in Claim C.3, whenever the weights are changed, we have proven that  $\mathbf{B}_t$  remains PSD.
Claim C.6 (Claim 3.5 restated). Assume that  $\mathbf{B}_t$  is PSD. Let  $V_{t + 1} = V_t\cup \{u_t\}$  for some unit vector  $u_{t}\in V_{t}^{\perp}$ , then  $\phi_{t + 1}\leq \phi_t - u_t^\top (\mathbf{M}_t^\perp)^2 u_t$ .
Proof. We first examine the effect that projecting everything to the space perpendicular to  $u_{t}$  has on the matrix  $\mathbf{B}_t^\perp$ .
Claim C.7. Define  $\Delta_t \coloneqq \mathbf{I} - u_t u_t^\top$ . We have the following:
(i)  $\Pi_{t + 1}^{\perp} = \Delta_{t}\Pi_{t}^{\perp}\Delta_{t}$
(ii)  $\pmb{\Sigma}_{t + 1}^{\perp} = \pmb{\Delta}_t\pmb{\Sigma}_t^{\perp}\pmb{\Delta}_t$
(iii)  $\mathbf{B}_{t + 1}^{\perp} = \pmb {\Delta}_t\mathbf{B}_t^{\perp}\pmb {\Delta}_t$
Proof. We prove each of these separately.
Proof of (i) We first recall the definition of the orthogonal projection matrix: Given any basis  $v_{t,1}, \ldots, v_{t,r} \in \mathbb{R}^d$  of the subspace  $\mathcal{V}_t^\perp$  ( $r$  is the dimension of the subspace), if  $\mathbf{A}_t$  is the  $d \times r$  matrix that has the  $v_{t,i}$ 's as its columns, then the matrix that performs the orthogonal projection to  $\mathcal{V}_t^\perp$  is  $\boldsymbol{\Pi}_t^\perp := \mathbf{A}_t(\mathbf{A}_t^\top \mathbf{A}_t)^{-1} \mathbf{A}_t^\top$ . Consider the basis that is orthonormal and starts with  $v_{t,1} = u_t$  (the vector mentioned in the claim's statement; such a basis exists since  $u_t \in \mathcal{V}_t^\perp$ ). Then,  $\boldsymbol{\Pi}_t^\perp = \mathbf{A}_t \mathbf{A}_t^\top = u_t u_t^\top + \sum_{i=2}^{r} v_{t,i} v_{t,i}^\top$ . Now, let  $\boldsymbol{\Pi}_{t+1}^\perp$  be the matrix that projects to the subspace  $\mathcal{V}_{t+1}^\perp := \{x \in \mathcal{V}_t^\perp : x \perp u_t\}$ . Since  $u_t$  itself belongs to  $\mathcal{V}_t^\perp$ , an orthonormal basis of  $\mathcal{V}_{t+1}^\perp$  is  $v_{t,2}, \ldots, v_{t,r}$ , and thus if we define  $\mathbf{A}_{t+1} := [v_{t,2}, \ldots, v_{t,r}]$  then the orthogonal projection matrix for  $\mathcal{V}_{t+1}^\perp$  is  $\boldsymbol{\Pi}_{t+1}^\perp = \mathbf{A}_{t+1} \mathbf{A}_{t+1}^\top = \sum_{i=2}^{r} v_{t,i} v_{t,i}^\top$ . We claim that this is the same as  $\boldsymbol{\Delta}_t \boldsymbol{\Pi}_t^\perp \boldsymbol{\Delta}_t$ : Indeed,  $\boldsymbol{\Delta}_t \boldsymbol{\Pi}_t^\perp \boldsymbol{\Delta}_t = (\boldsymbol{\Delta}_t \mathbf{A}_t)(\boldsymbol{\Delta}_t \mathbf{A}_t)^\top$ , and  $\boldsymbol{\Delta}_t \mathbf{A}_t = (\mathbf{I} - u_t u_t^\top) \mathbf{A}_t$  is the matrix with zeroes in the first column and  $v_{t,2}, \ldots, v_{t,r}$  in the rest of them, thus  $(\boldsymbol{\Delta}_t \mathbf{A}_t)(\boldsymbol{\Delta}_t \mathbf{A}_t)^\top = \sum_{i=2}^{r} v_{t,i} v_{t,i}^\top$ .
Proof of (ii) Let  $\mathbf{A}_t$  be exactly as in the previous paragraph. For the specific orthonormal basis  $v_{t,1},\ldots ,v_{t,r}\in \mathbb{R}^d$  of  $\mathcal{V}_t^\perp$  (where  $r$  is the dimension of that subspace) where the first vector has been chosen to be  $v_{t,1} = u_t$ , we define the  $d\times r$  matrix  $\mathbf{A}_t = [v_{t,1},v_{t,2},\dots,v_{t,r}] = [u_t,v_{t,2},\dots,v_{t,r}]$ . Also, let  $\mu_t$  be the mean of  $P_{t}$ . Then,
$$
\begin{array}{l} \boldsymbol {\Delta} _ {t} \boldsymbol {\Sigma} _ {t} ^ {\perp} \boldsymbol {\Delta} _ {t} = \underset {X \sim P _ {t}} {\mathbf {E}} \left[ \boldsymbol {\Delta} _ {t} \left(\operatorname {P r o j} _ {\mathcal {V} _ {t} ^ {\perp}} (X - \mu_ {t})\right) \left(\operatorname {P r o j} _ {\mathcal {V} _ {t} ^ {\perp}} (X - \mu_ {t})\right) ^ {\top} \boldsymbol {\Delta} _ {t} \right] \\ = \mathbf {E} _ {X \sim P _ {t}} \left[ \boldsymbol {\Delta} _ {t} \boldsymbol {\Pi} _ {t} ^ {\perp} (X - \mu_ {t}) (X - \mu_ {t}) ^ {\top} \boldsymbol {\Pi} _ {t} ^ {\perp} \boldsymbol {\Delta} _ {t} \right] \\ = \mathbf {E} _ {X \sim P _ {t}} \left[ \boldsymbol {\Delta} _ {t} \mathbf {A} _ {t} \mathbf {A} _ {t} ^ {\top} \left(X - \mu_ {t}\right) \left(X - \mu_ {t}\right) ^ {\top} \mathbf {A} _ {t} \mathbf {A} _ {t} ^ {\top} \boldsymbol {\Delta} _ {t} \right] (31) \\ = \underset {X \sim P _ {t}} {\mathbf {E}} \left[ \mathbf {A} _ {t + 1} \mathbf {A} _ {t + 1} ^ {\top} (X - \mu_ {t}) (X - \mu_ {t}) ^ {\top} \mathbf {A} _ {t + 1} \mathbf {A} _ {t + 1} ^ {\top} \right] \\ = \mathbf {E} _ {X \sim P _ {t}} \left[ \left(\operatorname {P r o j} _ {\mathcal {V} _ {t + 1} ^ {\perp}} (X - \mu_ {t})\right) \left(\operatorname {P r o j} _ {\mathcal {V} _ {t + 1} ^ {\perp}} (X - \mu_ {t})\right) ^ {\top} \right] (32) \\ \end{array}
$$
for the last line we used the expressions for  $\mathbf{A}_t, \mathbf{A}_{t+1}$  that we gave in the previous paragraph to conclude that  $\mathbf{A}_{t+1} \mathbf{A}_{t+1}^\top = \boldsymbol{\Pi}_{\mathcal{V}_{t+1}^\perp}$ .
Proof of (iii) The proof follows by the definition of  $\mathbf{B}_{t + 1}^{\perp} = (\mathbf{E}_{X\sim P}[w_t(X)])^2\Sigma_{t + 1}^{\perp} - (1 - C_1\epsilon)\Pi_{\nu_{t + 1}^{\perp}}$ , the previous two claims, and linearity.
We now show how to complete the proof given Claim C.7.
$$
\begin{array}{l} \phi_ {t + 1} := \operatorname {t r} \left(\left(\mathbf {B} _ {t + 1} ^ {\perp}\right) ^ {2 p}\right) \\ = \operatorname {t r} \left(\left(\boldsymbol {\Delta} _ {t} \mathbf {B} _ {t} ^ {\perp} \boldsymbol {\Delta} _ {t}\right) ^ {2 p}\right) \quad (\text {b y C l a i m C . 7}) \\ = \operatorname {t r} \left(\left(\mathbf {B} _ {t} ^ {\perp} \boldsymbol {\Delta} _ {t}\right) ^ {2 p}\right) \quad (\text {b y}) \\ \leq \operatorname {t r} \left(\left(\mathbf {B} _ {t} ^ {\perp}\right) ^ {2 p} \boldsymbol {\Delta} _ {t} ^ {2 p}\right) \quad (\text {b y}) \\ = \operatorname {t r} \left(\left(\mathbf {B} _ {t} ^ {\perp}\right) ^ {2 p} \boldsymbol {\Delta} _ {t}\right) \quad (\boldsymbol {\Delta} _ {t} := \mathbf {I} - u _ {t} u _ {t} ^ {\top} \text {i s i d e m p o t e n t}) \\ = \operatorname {t r} \left(\left(\mathbf {B} _ {t} ^ {\perp}\right) ^ {2 p} \left(\mathbf {I} - u _ {t} u _ {t} ^ {\top}\right)\right) \quad (\text {b y}) \\ = \operatorname {t r} \left(\left(\mathbf {B} _ {t} ^ {\perp}\right) ^ {2 p}\right) - \operatorname {t r} \left(\left(\mathbf {B} _ {t} ^ {\perp}\right) ^ {2 p} u _ {t} u _ {t} ^ {\top}\right) \\ = \operatorname {t r} \left(\left(\mathbf {B} _ {t} ^ {\perp}\right) ^ {2 p}\right) - u _ {t} ^ {\top} \left(\mathbf {B} _ {t} ^ {\perp}\right) ^ {2 p} u _ {t}, \quad (\text {b y}) \\ \end{array}
$$
where for the third step one can expand out  $(\pmb{\Delta}_t\mathbf{B}_t^\perp \pmb{\Delta}_t)^{2p} = \pmb{\Delta}_t\mathbf{B}_t^\perp \pmb{\Delta}_t^2\mathbf{B}_t^\perp \pmb{\Delta}_t^2\cdots \pmb{\Delta}_t^2\mathbf{B}_t^\perp \pmb{\Delta}_t$  and observe that: (i)  $\pmb{\Delta}_t^2 = \pmb{\Delta}_t$  since  $\pmb{\Delta}_t \coloneqq \mathbf{I} - u_t u_t^\top$ , and (ii) using the cyclic property of trace, after applying trace to both sides we can move the first  $\pmb{\Delta}_t$  to the end.
Finally, we show how the above two claims imply that  $\phi_{t + 1} \leq (1 - \mathrm{polylog}(\epsilon / d))\phi_t$ , for every iteration  $t$  that Line 19 of the algorithm is executed:
$$
\begin{array}{l} \phi_ {t + 1} - \phi_ {t} \geq u _ {t} ^ {\top} \left(\mathbf {B} _ {t} ^ {\perp}\right) ^ {2 p} u _ {t} \tag {byClaimC.6} \\ \geq \left(u _ {t} ^ {\top} \mathbf {B} _ {t} ^ {\perp} u _ {t}\right) ^ {2 p} \quad (\text {b y}) \\ = \left(\left(1 - \frac {1}{p}\right) \| \mathbf {B} _ {t} ^ {\perp} \| _ {\mathrm {o p}}\right) ^ {2 p} \quad (\text {b y}) \\ \gtrsim \left\| \mathbf {B} _ {t} ^ {\perp} \right\| _ {\mathrm {o p}} ^ {2 p} = \left\| \mathbf {M} _ {t} ^ {\perp} \right\| _ {\mathrm {o p}} ^ {2} \\ \gtrsim \frac {\mathrm {t r} ((\mathbf {M} _ {t} ^ {\perp}) ^ {2})}{\mathrm {p o l y l o g} (d / \epsilon)} = \frac {\phi_ {t}}{\mathrm {p o l y l o g} (d / \epsilon)}. \qquad (\text {b y C l a i m 3 . 4 w i t h} \alpha , \gamma = \mathrm {p o l y l o g} (d / \epsilon)) \\ \end{array}
$$
C.4 Combining Everything Together
We now describe in more detail the algorithm that achieves Theorem 1.3 and put together the previous parts to derive the guarantee and runtime of the main theorem.
The final algorithm is outlined in Algorithm 5.
Algorithm 5 Full Algorithm
1: Input:  $\epsilon \in (0,1/2)$ , sets  $S_1, S_2 \subset \mathbb{R}^d$  datasets, where for each  $i = 1,2$  the uniform distribution on  $S_i$  is of the form  $(1 - \epsilon)G_i + \epsilon B_i$  for some  $G_i$  satisfying Definition 2.1 with appropriate parameters.
2: Output: A vector in  $\mathbb{R}^d$ .
3: Run the algorithm from Lemma B.18 on  $S_{1}$  and let  $w(x)$  for  $x \in S_{1}$  be the weights that it outputs. Let  $P$  be the distribution assigning mass  $w(x) / \sum_{x \in S_{1}} w(x)$  to every  $x \in S_{1}$ .
4: Run Algorithm 1 on input  $P$ . Let  $\mathcal{V}_1$  and  $\mu_1$  be the subspace and the vector that it outputs.
5: Project all points from  $S_{2}$  to  $\mathcal{V}_1$ , i.e.,  $S_{2} \gets \{\operatorname{Proj}_{\mathcal{V}_{1}}(x): x \in S_{2}\}$ .
6: Run Algorithm 4 on  $S_{2}$ . Let  $\mu_{2}$  be the vector that it outputs.
7: return  $\mu_{1} + \mu_{2}$ .
Proof of Theorem 1.3. The set  $S_{1}$  is comprised of i.i.d. points from the  $\epsilon$ -corrupted version of  $\mathcal{N}(\mu, \mathbf{I})$  and its size is chosen to be a large enough multiple of  $\epsilon^{-2}(d + \log(1/\delta))\mathrm{poly}\log(d/\epsilon)$ , so that by Lemma B.20 the uniform distribution on  $S_{1}$  is  $(\epsilon, \alpha, k)$ -good with  $\alpha = \epsilon/\log(1/\epsilon)$  and  $k = \mathrm{poly}\log(d/\epsilon)$ . The preprocessing step of Line 3 ensures that the eigenvalues of the covariance matrix are at most  $1 + \epsilon\mathrm{poly}\log(1/\epsilon)$ , which is part of Setting 2.3 and utilized in the analysis of the next steps. The preprocessing step of Line 3 runs in nearly-linear time  $O(nd\mathrm{poly}\log(d/\epsilon))$ .
The step of Line 4 has been analyzed in Section 3 and Appendix C using the potential argument. We have demonstrated that in each iteration of the algorithm, the potential function  $\phi_t \coloneqq \mathrm{tr}((\mathbf{M}_t^\perp)^2)$  undergoes multiplicative reduction:  $\phi_{t+1} \leq (1 - \mathrm{polylog}(\epsilon / d))\phi_t$ . Initially, the potential is naively bounded by  $\phi_0 = \mathrm{poly}(d / \epsilon)^{\log d}$ . With just  $\mathrm{polylog}(d / \epsilon)$  iterations, the potential decreases to  $\phi_t \leq \epsilon^{\log d}$ , resulting in  $\| \mathbf{B}_t^\perp \|_{\mathrm{op}} = O(\epsilon)$  and leading to the termination of Algorithm 1 with  $\| \mathrm{Proj}_{\mathcal{V}_1^\perp}(\mu_1 - \mu) \|_2 = O(\epsilon)$ .
Each step of the algorithm can be implemented in  $O(nd\mathrm{polylog}(d / \epsilon))$  time, for example, computing  $(\mathbf{B}^{\perp})^{\log d}z$  by iteratively multiplying  $z$  by  $\mathbf{B}^{\perp}$ . Utilizing the special form of covariance matrices enables each multiplication to be calculated efficiently in  $O(nd)$  time.
Since Algorithm 1 only adds one direction per iteration to the subspace  $\mathcal{V}_1$ , the dimension of  $\mathcal{V}_1$  will be  $d' = O(\mathrm{polylog}(d / \epsilon))$ . We now use a smaller sized second dataset  $S_2$  that consists of  $\epsilon^{-2}(d' + \log(1 / \delta))\mathrm{polylog}(d / \epsilon)$  samples and run Algorithm 4 on  $S_2$ . The runtime of that algorithm is given in Lemma 2.6, and since we are using  $d' = O(\mathrm{polylog}(d / \epsilon))$  for the dimension, the runtime of that algorithm is  $O(\epsilon^{-2 - r}\mathrm{polylog}(d / \epsilon))$ . Finally, since the output  $\mu_2$  approximates  $\mu$  in the subspace  $\mathcal{V}_1$  within error  $O(\epsilon)$ , combining  $\mu_1$  and  $\mu_2$  yields an estimate with  $\| (\mu_1 + \mu_2) - \mu \|_2 \leq \| \mathrm{Proj}_{\mathcal{V}_1^\perp}(\mu_1 - \mu) \|_2 + \| \mathrm{Proj}_{\mathcal{V}_1}(\mu_2 - \mu) \|_2 = O(\epsilon)$ .
D Omitted Proofs from Section 4
In this section, we provide the details omitted from Section 4. Appendix D.1 proves the properties of the conditional distribution of the covariates conditioned on the responses being in an interval. Appendix D.2 then uses these properties to show that the samples from this distribution satisfy the goodness condition. Finally, we provide the details missing from the proof sketch in the main body in Appendix D.3.
D.1 Conditional Distribution of Covariates
We first begin by proving Claim 4.1.