SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Proof. Let  $u_{1}, \ldots, u_{d}$  be the orthonormal eigenvectors of  $A'$  with eigenvalues  $\lambda_{1}, \lambda_{2}, \ldots, \lambda_{d}$  such that  $|\lambda_i| = 1$  for  $i \leq \operatorname{rank}(A')$  and  $\lambda_i = 0$  otherwise. Without loss of generality, we assume  $\operatorname{rank}(B') \leq \operatorname{rank}(A')$ , otherwise the result is trivial. Let  $\mathcal{I}$  be the set of  $i$ 's such that  $i \leq \operatorname{rank}(A')$  and  $B'u_i = 0$ .
$$
\begin{array}{l} \| B ^ {\prime} - A ^ {\prime} \| _ {\mathrm {F}} ^ {2} = \sum_ {i = 1} ^ {d} \| B ^ {\prime} u _ {i} - A ^ {\prime} u _ {i} \| _ {2} ^ {2} \geq \sum_ {i = 1} ^ {\operatorname {r a n k} \left(A ^ {\prime}\right)} \| B ^ {\prime} u _ {i} - A ^ {\prime} u _ {i} \| _ {2} ^ {2} \\ = \sum_ {i = 1} ^ {\operatorname {r a n k} \left(A ^ {\prime}\right)} \| B ^ {\prime} u _ {i} - \lambda_ {i} u _ {i} \| _ {2} ^ {2}. \geq \sum_ {i \in \mathcal {I}} \| \lambda_ {i} u _ {i} \| _ {2} ^ {2} = | \mathcal {I} |. \\ \end{array}
$$
Since the rank of  $B'$  is less than the rank of  $A'$ , it follows that  $B'u_i$  is nonzero for at most  $\operatorname{rank}(B')$  many  $u_i$ 's in  $\{u_i : i \leq \operatorname{rank}(A')\}$ . Thus  $|\mathcal{I}|$  is at least  $\operatorname{rank}(A') - \operatorname{rank}(B')$ .
By Claim C.4, we have that  $\mathrm{rank}(\mathbb{P}_B) - \mathrm{rank}(ABA)\leq \| \mathbb{P}_B - ABA\|_{\mathbb{F}}^2$ . Thus, it implies the desired inequality,  $|d_B - d_A|\leq \| ABA - \mathbb{P}_A\|_{\mathbb{F}}^2$
We will now establish the first inequality in (5). Observe that both  $\mathbb{P}_A$  and  $\mathbb{P}_B$  are PSD matrices, whose eigenvalues are either 1 or 0. The following result upper bounds the Frobenius norm of  $\mathbb{P}_A - \mathbb{P}_B$  in terms of the difference of their ranks.
Claim C.5. If  $A$  and  $B$  are two PSD matrices such that  $\mathbb{P}_B \preceq \mathbb{P}_A$ , then  $\| \mathbb{P}_A - \mathbb{P}_B \|_F^2 = \mathrm{rank}(A) - \mathrm{rank}(B)$ .
Proof. Let  $d_A$  be the rank of  $A$  and  $d_B$  be the rank of  $B$ . We have that  $d_B = \mathrm{rank}(\mathbb{P}_B) \leq \mathrm{rank}(\mathbb{P}_A) = d_A$ . Consider any orthonormal basis  $v_1, \ldots, v_d$  in  $\mathbb{R}^d$ , such that  $v_1, \ldots, v_{d_A}$  spans the column space of  $\mathbb{P}_A$ , and  $v_1, \ldots, v_{d_B}$  spans the column space of  $B$ . Since the column space of  $\mathbb{P}_B$  is a subspace of that of  $\mathbb{P}_B$ , such an orthonormal basis always exists.
Moreover, since the eigenvalues of  $\mathbb{P}_A$  are either 0 or 1, we have that  $\mathbb{P}_Av_i = v_i$  for  $i\leq d_A$  and 0 otherwise. Similarly,  $\mathbb{P}_Bv_i = v_i$  for  $i\leq d_B$  and 0 otherwise.
We will also use the following basic fact: for any square matrix  $J$  and any set of orthonormal basis vectors  $u_{1},\ldots ,u_{d}$ , we have that  $\| J\|_{\mathrm{F}}^{2} = \sum_{i = 1}^{d}\| Ju_{i}\|_{2}^{2}$ . Applying this fact to  $\mathbb{P}_A - \mathbb{P}_B$  and the orthonormal basis  $v_{1},\ldots ,v_{d}$  above, we get the following:
$$
\begin{array}{l} \| \mathbb {P} _ {A} - \mathbb {P} _ {B} \| _ {\mathrm {F}} ^ {2} = \sum_ {i = 1} ^ {d} \| \mathbb {P} _ {A} v _ {i} - \mathbb {P} _ {B} v _ {i} \| _ {2} ^ {2} = \sum_ {i = 1} ^ {d _ {B}} \| \mathbb {P} _ {A} v _ {i} - \mathbb {P} _ {B} v _ {i} \| _ {2} ^ {2} + \sum_ {d _ {B} + 1} ^ {d _ {A}} \| \mathbb {P} _ {A} v _ {i} - \mathbb {P} _ {B} v _ {i} \| _ {2} ^ {2} = \sum_ {d _ {B} + 1} ^ {d _ {A}} \| v _ {i} \| _ {2} ^ {2} \\ = d _ {A} - d _ {B}, \\ \end{array}
$$
where the second equality uses that  $\mathbb{P}_A v_i = \mathbb{P}_B v_i = 0$  for  $i \geq d_A$ , and the third equality uses that  $\mathbb{P}_A v_i = \mathbb{P}_B v_i = v_i$  for  $i \leq d_B$  and  $\mathbb{P}_B v_i = 0$  and  $\mathbb{P}_A v_i = v_i$  for  $i \in \{d_B + 1, \dots, d_A\}$ .
By Claim C.5, we have that  $\| \mathbb{P}_A - \mathbb{P}_B\|_{\mathrm{F}}^2 = |\operatorname {rank}(A) - \operatorname {rank}(B)|$
D Stability: Sample Complexity and Its Consequences
In this section, we prove results related to stability. Appendix D.1 proves the sample complexity for the stability condition to hold for Gaussians and Appendix D.2 focuses on some consequences of stability.
Remark D.1. We note that the deterministic conditions of Definition 2.1 are not specific to the Gaussian distribution but hold with polynomial sample complexity  $\mathrm{poly}(d / \eta)$  for broader classes of distributions, roughly speaking, distributions with light tails for degree-4 polynomials.
- Condition (L.1) is satisfied with polynomial sample complexity by distributions that have bounded second moment.
- Condition (L.3) at the population level is a direct implication of hypercontractivity of degree-2 polynomials. The  $\log(1/\eta)$  factor in the condition as stated currently is tailored to the Gaussian distribution but it is not crucial for the algorithm and could be relaxed to some polynomial of  $1/\eta$ ; this would translate to incurring some  $\mathrm{poly}(1/\eta)$  to the final error guarantee of the main theorem. After establishing the condition at the population level, it can be transferred to sample level with polynomial sample complexity.
- Regarding Condition (L.2), every entry of the Frobenius norm is a degree 2 polynomial, thus the property holds for distributions with light tails for these polynomials, e.g., hypercontractive degree-2 distributions.
- Condition (L.4) can also be derived by hypercontractivity of degree 4 polynomials similarly.
- Condition (L.5) holds almost surely for continuous distributions.
D.1 Sample Complexity of Stability Conditions
Proof. Let  $A$  be a set of i.i.d. samples from  $\mathcal{N}(\mu, \Sigma)$ . We check the four conditions from Definition 2.1 separately. Since  $A$  is a set of i.i.d. samples from a Gaussian, it satisfies Condition (L.1) with high probability (see, e.g., [Li18, Corollary 2.1.13]).
We start with Condition (L.2). Let  $X \sim \mathcal{N}(\mu, \Sigma)$ , then  $X - \mu \sim \mathcal{N}(0, \Sigma)$ . Equivalently,  $X - \mu = \Sigma^{1/2} Z$  with  $Z \sim \mathcal{N}(0, I)$ . Let  $A$  be a set of  $m$  i.i.d. samples from  $\mathcal{N}(\mu, \Sigma)$  and like before, for every  $x \in A$ , write  $x - \mu = \Sigma^{1/2} z$ , where  $z$  are corresponding i.i.d. samples from  $\mathcal{N}(0, I)$ . Let a subset  $A' \subset A$  with  $|A'| \geq |A|(1 - \epsilon)$ . Then, we have the following
$$
\left. \operatorname {t r} \left(\left(\frac {1}{n} \sum_ {x \in A ^ {\prime}} (x - \mu) (x - \mu) ^ {\top} - \Sigma\right) U\right) = \operatorname {t r} \left(\Sigma^ {1 / 2} \left(\frac {1}{n} \sum_ {z} z z ^ {\top} - I\right) \Sigma^ {1 / 2} U\right) \right.
$$
$$
\begin{array}{l} = \operatorname {t r} \left(\left(\frac {1}{n} \sum_ {z} z z ^ {\top} - I\right) \Sigma^ {1 / 2} U \Sigma^ {1 / 2}\right) \\ \leq C \epsilon \log (1 / \epsilon) \| \Sigma^ {1 / 2} U \Sigma^ {1 / 2} \| _ {F}, \\ \end{array}
$$
where the first line uses the rewriting that we introduced before, the second line uses the cyclic property of the trace operator, and the last line uses [Li18, Corollary 2.1.13]. As stated in that corollary, the inequality holds with probability  $1 - \delta /4$  as long as the sample complexity is a sufficiently large multiple of  $(d^{2} / \epsilon^{2})\log (4 / \delta)$ . Fixing  $\epsilon = \epsilon_0$  a sufficiently small positive absolute constant can make the right hand side of the above inequality  $0.1\| \Sigma^{1 / 2}U\Sigma^{1 / 2}\|_{\mathrm{F}}$ .
Condition Condition (L.1) can be shown identically using [Li18, Corollary 2.1.9].
We now turn our attention to Condition (L.3). [DKKLMS16, Lemma 5.17] implies that with the stated number of samples, with high probability, it holds that
$$
\mathop{\mathrm{Pr}}\limits_{X\sim A}\left[|p(x)| > \sqrt{\underset{Y\sim\mathcal{N}(\mu,\Sigma)}{\mathbf{Var}}[p(Y)]} 10\ln \left(\frac{2}{\eta}\right)\right]\leq \frac{\eta}{2}.
$$
Since  $|A'| \geq |A|(1 - \epsilon_0) \geq |A|/2$ , Condition (L.3) holds. Finally, Condition (L.4) follows by noting that  $\mathbf{Var}_{X \sim A'}[p(X)] \leq 2\mathbf{Var}_{X \sim A}[p(X)]$  and using [DKKLMS16, Lemma 5.17] again with  $\epsilon = \epsilon_0$ . In order for the conclusion of that lemma to hold with probability  $1 - \delta/4$ , the number of samples of the Gaussian component is a sufficiently large multiple of  $d^2\log^5(4d / (\eta \delta)) / \eta^2$ . A union bound over the events corresponding to each of the four conditions concludes the proof.
Lastly, we show Condition (L.5). We will in fact show a stronger statement: suppose the rank of  $\Sigma$  is  $d_{\Sigma}$ . Then, if we take  $m \geq d_{\Sigma} + 1$  i.i.d. samples  $\{\mu + z_1, \mu + z_2, \dots, \mu + z_m\}$  from  $\mathcal{N}(\mu, \Sigma)$ , it must be the case that with probability 1, for all  $A' \subseteq A$  with  $|A'| \geq d_{\Sigma} + 1$ , we have  $\sum_{x \in A'} (v^\top x)^2 = 0$  (or equivalently,  $v$  being orthogonal to all vectors  $x \in A'$ ) implying  $v^\top \Sigma v = 0$  for all vectors  $v \in \mathbb{R}^d$ .
We will need the following straightforward claim: if  $A' = \{\mu + z_0', \dots, \mu + z_{d_\Sigma}'\}$  (of size  $d_\Sigma + 1$ ) is such that 1)  $\{z_1' - z_0', \dots, z_{d_\Sigma}' - z_0'\}$  are linearly independent and 2)  $z_i'$  all lie within the column space of  $\Sigma$ , then for any vector  $v \in \mathbb{R}^d$ ,  $v$  being orthogonal to all of  $A'$  implies that  $v^\top \Sigma v = 0$ . To see why this is true, observe that  $\{z_1' - z_0', \dots, z_{d_\Sigma}' - z_0'\}$ , which are linearly independent vectors that all lie in the column space of  $\Sigma$ , must span the entirety of the column space of  $\Sigma$  by dimension counting. Thus, if  $v$  is orthogonal to all the points in  $A'$ , then  $v$  is orthogonal to all of  $\{z_1' - z_0', \dots, z_{d_\Sigma}' - z_0'\}$ , which implies  $v^\top \Sigma v = 0$ .
We can now show the following inductive claim on the size of  $A$ , which combined with the above claim implies the strengthened version of Condition (L.5). Specifically, we show inductively that, with probability 1 over the sampling of  $A$ , for any subset  $A' \subset A$  of size at most  $d_{\Sigma} + 1$ , the claim conditions hold for  $A'$ . Namely, if  $A' = \{\mu + z_0', \dots, \mu + z_j'\}$  for some  $j \leq d_{\Sigma}$ , for some arbitrary ordering of the elements in  $A'$ , then 1) the set  $\{z_1' - z_0', \dots, z_j' - z_0'\}$  is linearly independent and 2)  $z_i'$  all lie in the column space of  $\Sigma$ .
The base case of  $|A| = 1$  is trivial. Now suppose the above statement is true for  $|A_{\ell}| = \ell$ , and consider sampling a new point  $\mu + z_{\ell}$  and adding it to  $A_{\ell}$  to form  $A_{\ell+1}$ . Take any subset  $A'$  of  $A_{\ell+1}$  of size at most  $d_{\Sigma} + 1$ . If  $A' \subset A_{\ell}$  we are already done by the induction hypothesis. Otherwise,  $A' = \{\mu + z_{\ell}\} \cup A_{\ell}'$ , where  $A_{\ell}' = \{\mu + z_0', \dots, \mu + z_j'\}$  has size at most  $d_{\Sigma}$  and satisfies the conditions in the induction by the inductive hypothesis. The space of  $\mu + z$  such that  $z - z_0'$  lies in the span of  $\{z_1' - z_0', \dots, z_j' - z_0'\}$  has measure 0 under  $\mathcal{N}(\mu, \Sigma)$ , given there are strictly fewer than  $d_{\Sigma}$  many linearly independent vectors in  $\{z_1' - z_0', \dots, z_j' - z_0'\}$ , all of which lie in the column space of  $\Sigma$ . Furthermore, there is only a finite number of such  $A_{\ell}'$ . Thus, with probability 1, the new point  $\mu + z_{\ell}$  will be sampled such that for all such  $A_{\ell}'$ ,  $\{\mu + z_{\ell}\} \cup A_{\ell}'$  remains linearly independent. Additionally, also with probability 1,  $z_{\ell}$  will lie in the column space of  $\Sigma$ . This completes the proof of the inductive claim, which as we have shown implies Condition (L.5).
D.2 Facts Regarding Stability
The following corollaries will be useful later on. In particular, since our algorithm will perform transformations of the dataset, we will need Corollary D.2 to argue about the stability properties of these sets. Moreover, the next lemma (Lemma D.3) will be used to extract the final Frobenius norm guarantee of our theorem given the first bullet of that theorem. The proofs are deferred to Appendix D.4.
Corollary D.2 (Stability of Transformed Sets). Let  $D$  be a distribution over  $\mathbb{R}^d$  with mean  $\mu$  and covariance  $\Sigma$ . Let  $A$  be a set of points in  $\mathbb{R}^d$  that satisfies Conditions (L.1) and (L.2) of Definition 2.1 of  $(\eta, \epsilon)$ -stability with respect to  $D$ . Then we have that for any symmetric transformation  $P \in \mathbb{R}^{d \times d}$  and every  $A' \subset A$  with  $|A'| \geq (1 - \epsilon)|A|$ , the following holds:
1.  $\left\| P\left(\frac{1}{|A^{\prime}|}\sum_{x\in A^{\prime}}x - \mu\right)\right\|_{2}\leq 0.1\sqrt{\|P\Sigma P\|_{\mathrm{op}}}$
2.  $\left\| P\left(\frac{1}{|A'|}\sum_{x\in A'}(x - \mu)(x - \mu)^\top -\Sigma\right)P\right\|_{\mathrm{F}}\leq 0.1\| P\Sigma P\|_{\mathrm{op}}.$
The following lemma (proved in Appendix D.4) will be needed to prove our certificate lemma to transfer the final guarantees of our algorithm from the empirical covariance of the sample set to the actual covariance of the Gaussians.
Lemma D.3. Let  $A$  be a set of points in  $\mathbb{R}^d$  satisfying Conditions (L.1) and (L.2) of  $(\eta, \epsilon)$ -stability (Definition 2.1) with respect to  $\mu, \Sigma$ . Let  $A' \subseteq A$  such that  $|A'| \geq (1 - \epsilon)|A|$ . If  $P, L \in \mathbb{R}^{d \times d}$  are symmetric matrices with the property that  $\| \mathbf{Cov}_{X \sim A'}[PX] - L \|_{\mathrm{F}} \leq r$ , then
$$
\left\| P \Sigma P - L \right\| _ {\mathrm {F}} = O (r + \left\| L \right\| _ {\mathrm {o p}}).
$$
D.3 Certificate Lemma
The following result shows that if the covariance matrix of  $X^{\otimes 2}$  over the corrupted set of points is bounded in operator norm, then the empirical second moment matrix of  $T$  is a good estimate of covariance of inliers in Frobenius norm. We prove a more general version below that allows for arbitrary linear transformation matrix  $P$  and since we are interested in guarantees for relative Frobenius norm, our algorithm will chose  $P$  so that  $\mathbf{E}_{X\sim T}[PXX^\top P]$  is equal to identity.
Lemma D.4 (Bounded fourth-moment and second moment imply closeness of covariance). Let  $T$  be a set of  $n$  points in  $\mathbb{R}^d$ . Let  $P$  be an arbitrary symmetric matrix. Let  $w \in \mathbb{R}^{d^2}$  be the leading eigenvector of the  $\mathbf{Cov}_{X \sim T}[(PX)^{\otimes 2}]$  with  $\|w\|_2 = 1$ , and let  $A \in \mathbb{R}^{d \times d}$  be  $w^\sharp$ . Note that  $\|w\|_2 = 1$  implies  $\|A\|_{\mathrm{F}} = 1$ . For every subset  $T' \subseteq T$  with  $|T'| \geq (\alpha/2)|T|$ , we have that
$$
\begin{array}{l} \left\| \mathbf {E} _ {X \sim T ^ {\prime}} [ P X X ^ {\top} P ] - \mathbf {E} _ {x \sim T} [ P X X ^ {\top} P ] \right\| _ {F} ^ {2} \leq \frac {2}{\alpha} \operatorname {V a r} _ {X \sim T} [ X ^ {\top} P A P X ] \quad a n d \\ \left\| \mathbf {C o v} _ {X \sim T ^ {\prime}} [ P X ] - \underset {x \sim T} {\mathbf {E}} [ P X X ^ {\top} P ] \right\| _ {\mathrm {F}} ^ {2} \leq \frac {4}{\alpha} \mathbf {V a r} _ {X \sim T} [ X ^ {\top} P A P X ] + \frac {8}{\alpha^ {2}} \left\| \underset {X \sim T} {\mathbf {E}} [ P X X ^ {\top} P ] \right\| _ {\mathrm {o p}} ^ {2}. \\ \end{array}
$$
In particular, suppose that there exists a set  $S$  that satisfies Conditions (L.1) and (L.2) of  $(\eta ,\epsilon)$ -stability (Definition 2.1) with respect to  $\mu$  and  $\Sigma$  for some  $\eta$  and  $\epsilon$ , and suppose that  $|S\cap T^{\prime}|\geq \min (\alpha |T| / 2,(1 - \epsilon)|S|)$ . Then
$$
\left\| P \Sigma P - \underset {X \sim T ^ {\prime}} {\mathbf {E}} [ P X X ^ {\top} P ] \right\| _ {\mathrm {F}} \lesssim \frac {1}{\alpha} \operatorname {V a r} [ X ^ {\top} P A P X ] + \frac {1}{\alpha^ {2}} \left\| \underset {X \sim T} {\mathbf {E}} [ P X X ^ {\top} P ] \right\| _ {\mathrm {o p}} ^ {2}.
$$
Proof. Let  $\widetilde{T}$  define the set  $\{PX:X\in T\}$ . In this notation,  $w$  is the leading eigenvector of  $\mathbf{Cov}_{\widetilde{x}\sim \widetilde{T}}[\widetilde{X}^{\otimes 2}]$  with  $\| w\| _2 = 1$ , and  $A = w^{\sharp}$ . Then, for any  $\widetilde{T}'\subseteq \widetilde{T}$  with  $|\widetilde{T} '|\geq \alpha |\widetilde{T} '| / 2$ , we have the following
$$
\begin{array}{l} \underset {\widetilde {X} \sim \widetilde {T}} {\mathbf {V a r}} [ \widetilde {X} ^ {\top} A \widetilde {X} ] = \sup  _ {v \in \mathbb {R} ^ {d ^ {2}}, \| v \| _ {2} ^ {2} = 1} \underset {\widetilde {X} \sim \widetilde {T}} {\mathbf {E}} \left[ \left\langle \widetilde {X} ^ {\otimes 2} - \underset {\widetilde {Y} \sim \widetilde {T}} {\mathbf {E}} [ \widetilde {Y} ^ {\otimes 2} ], v \right\rangle^ {2} \right] \\ = \sup  _ {v \in \mathbb {R} ^ {d ^ {2}}, \| v \| _ {2} ^ {2} = 1} \frac {| \widetilde {T} ^ {\prime} |}{| \widetilde {T} |} \underset {\widetilde {X} \sim \widetilde {T} ^ {\prime}} {\mathbf {E}} \left[ \left\langle \widetilde {X} ^ {\otimes 2} - \underset {\widetilde {Y} \sim \widetilde {T}} {\mathbf {E}} [ \widetilde {Y} ^ {\otimes 2} ], v \right\rangle^ {2} \right] \\ \end{array}
$$
$$
\begin{array}{l} + \frac {| \widetilde {T} \setminus \widetilde {T} ^ {\prime} |}{| \widetilde {T} |} _ {\widetilde {X} \sim \widetilde {T} \setminus \widetilde {T} ^ {\prime}} \left[ \left\langle \widetilde {X} ^ {\otimes 2} - \mathbf {E} _ {\widetilde {Y} \sim \widetilde {T}} [ \widetilde {Y} ^ {\otimes 2} ], v \right\rangle^ {2} \right] \\ \geq \frac {\alpha}{2} \sup  _ {v \in \mathbb {R} ^ {d ^ {2}}, \| v \| _ {2} ^ {2} = 1} \underset {\widetilde {X} \sim \widetilde {T} ^ {\prime}} {\mathbf {E}} \left[ \left\langle \widetilde {X} ^ {\otimes 2} - \underset {\widetilde {Y} \sim \widetilde {T}} {\mathbf {E}} [ \widetilde {Y} ^ {\otimes 2} ], v \right\rangle^ {2} \right] \quad (\text {s i n c e} | \widetilde {T} ^ {\prime} | \geq (\alpha / 2) | \widetilde {T} |) \\ \geq \frac {\alpha}{2} \sup  _ {v \in \mathbb {R} ^ {d ^ {2}}, \| v \| _ {2} ^ {2} = 1} \left\langle \underset {\tilde {X} \sim \tilde {T} ^ {\prime}} {\mathbf {E}} [ \widetilde {X} ^ {\otimes 2} ] - \underset {\tilde {Y} \sim \tilde {T}} {\mathbf {E}} [ \widetilde {Y} ^ {\otimes 2} ], v \right\rangle^ {2} (\mathbf {E} [ Y ^ {2} ] \geq (\mathbf {E} [ Y ]) ^ {2} \text {f o r a n y} Y) \\ = \frac {\alpha}{2} \sup  _ {J \in \mathbb {R} ^ {d \times d}, \| J \| _ {\mathrm {F}} ^ {2} = 1} \left[ \left\langle \mathbf {E} _ {\widetilde {X} \sim \widetilde {T} ^ {\prime}} [ \widetilde {X} \widetilde {X} ^ {\top} ] - \mathbf {E} _ {\widetilde {Y} \sim \widetilde {T}} [ \widetilde {Y} \widetilde {Y} ^ {\top} ], J \right\rangle^ {2} \right] \\ = \frac {\alpha}{2} \left\| \mathbf {E} _ {\widetilde {X} \sim \widetilde {T} ^ {\prime}} \left[ \widetilde {X} \widetilde {X} ^ {\top} \right] - \mathbf {E} _ {\widetilde {X} \sim \widetilde {T}} \left[ \widetilde {X} \widetilde {X} ^ {\top} \right] \right\| _ {\mathrm {F}} ^ {2}, \\ \end{array}
$$
where the penultimate step above holds by lifting the flattened matrix inner products in the previous line to the matrix inner product, and the last line uses the variation characterization of the Frobenius norm. This implies the first inequality since  $\mathbf{E}_{\widetilde{X}\sim \widetilde{T}}[\widetilde{X}\widetilde{X}^{\top}] = \mathbf{E}_{X\sim T}[PXX^{\top}P]$ .
We will now establish the second inequality. Using the relation between the second moment matrix and the covariance, we have the following:
$$
\begin{array}{l} \left\| \mathbf {C o v} _ {\widetilde {X} \sim \widetilde {T} ^ {\prime}} [ \widetilde {X} ] - \mathbf {E} _ {\widetilde {X} \sim \widetilde {T}} [ \widetilde {X} \widetilde {X} ^ {\top} ] \right\| _ {\mathrm {F}} ^ {2} = \left\| \mathbf {E} _ {\widetilde {X} \sim \widetilde {T} ^ {\prime}} [ \widetilde {X} \widetilde {X} ^ {\top} ] - \mathbf {E} _ {\widetilde {X} \sim \widetilde {T} ^ {\prime}} [ \widetilde {X} ] \mathbf {E} _ {\widetilde {X} \sim \widetilde {T} ^ {\prime}} [ \widetilde {X} ^ {\top} ] - \mathbf {E} _ {\widetilde {X} \sim \widetilde {T}} [ \widetilde {X} \widetilde {X} ^ {\top} ] \right\| _ {\mathrm {F}} ^ {2} \\ \leq 2 \left\| \underset {\widetilde {X} \sim \widetilde {T} ^ {\prime}} {\mathbf {E}} [ \widetilde {X} \widetilde {X} ^ {\top} ] - \underset {\widetilde {X} \sim \widetilde {T}} {\mathbf {E}} [ \widetilde {X} \widetilde {X} ^ {\top} ] \right\| _ {\mathrm {F}} ^ {2} + 2 \left\| \underset {\widetilde {X} \sim \widetilde {T} ^ {\prime}} {\mathbf {E}} [ \widetilde {X} ] \underset {\widetilde {X} \sim \widetilde {T} ^ {\prime}} {\mathbf {E}} [ \widetilde {X} ^ {\top} ] \right\| _ {\mathrm {F}} ^ {2} \\ \leq \frac {4}{\alpha} \mathbf {V a r} _ {\widetilde {X} \sim \widetilde {T}} [ \widetilde {X} ^ {\top} A \widetilde {X} ] + 2 \left\| \mathbf {E} _ {\widetilde {X} \sim \widetilde {T} ^ {\prime}} [ \widetilde {X} ] \mathbf {E} _ {\widetilde {X} \sim \widetilde {T} ^ {\prime}} [ \widetilde {X} ^ {\top} ] \right\| _ {\mathrm {o p}} ^ {2}, \\ \end{array}
$$
where the last step uses that  $\mathbf{E}_{\widetilde{X}\sim \widetilde{T}^{\prime}}[\widetilde{X} ]\mathbf{E}_{\widetilde{X}\sim \widetilde{T}^{\prime}}[\widetilde{X}^{\top}]$  has rank one and thus its Frobenius and operator norm match. Thus to establish the desired inequality, it suffices to prove that  $\mathbf{E}_{\widetilde{X}\sim \widetilde{T}^{\prime}}[\widetilde{X} ]\mathbf{E}_{\widetilde{X}\sim \widetilde{T}^{\prime}}[\widetilde{X}^{\top}]\preceq (2 / \alpha)\cdot \mathbf{E}_{\widetilde{X}\sim \widetilde{T}}[\widetilde{X}\widetilde{X}^{\top}]$ . Indeed, we have that
$$
\begin{array}{l} \underset {\widetilde {X} \sim \widetilde {T} ^ {\prime}} {\mathbf {E}} [ \widetilde {X} ] \underset {\widetilde {X} \sim \widetilde {T} ^ {\prime}} {\mathbf {E}} [ \widetilde {X} ^ {\top} ] \preceq \underset {\widetilde {X} \sim \widetilde {T} ^ {\prime}} {\mathbf {E}} [ \widetilde {X} \widetilde {X} ^ {\top} ] = \frac {1}{| \widetilde {T} ^ {\prime} |} \sum_ {\widetilde {X} \in \widetilde {T} ^ {\prime}} \widetilde {X} \widetilde {X} ^ {\top} = \frac {| \widetilde {T} |}{| \widetilde {T} ^ {\prime} |} \frac {1}{| \widetilde {T} |} \sum_ {\widetilde {X} \in \widetilde {T} ^ {\prime}} \widetilde {X} \widetilde {X} ^ {\top} \\ \preceq \frac {2}{\alpha} \frac {1}{| \widetilde {T} |} \sum_ {\widetilde {X} \in \widetilde {T} ^ {\prime}} \widetilde {X} \widetilde {X} ^ {\top} \preceq \frac {2}{\alpha} \frac {1}{| \widetilde {T} |} \sum_ {\widetilde {X} \in \widetilde {T}} \widetilde {X} \widetilde {X} ^ {\top}, \\ \end{array}
$$
where the last expression is in fact equal to  $(2 / \alpha)\cdot \mathbf{E}_{\widetilde{X}\sim \widetilde{T}}[\widetilde{X}\widetilde{X}^{\top}]$ . This completes the proof for the second statement.
We now consider the case when  $T$  contains a large stable subset. Let  $T' = T \cap S$ , which satisfies that  $|T'| \geq (\alpha/2)|T|$  and  $|T'| \geq (1 - \epsilon)|S|$ . By applying Lemma D.3 with  $L = \mathbf{E}_{X \sim T}[PXX^\top P]$  to the second inequality in the statement with  $T'$ , we obtain the desired result.
D.4 Proofs of Corollary D.2 and lemma D.3
1.  $\left\| P\left(\frac{1}{|A'|}\sum_{x\in A'}x - \mu\right)\right\| _2\leq 0.1\sqrt{\|P\Sigma P\|_{\mathrm{op}}}$
2.  $\left\| P\left(\frac{1}{|A^{\prime}|}\sum_{x\in A^{\prime}}(x - \mu)(x - \mu)^{\top} - \Sigma\right)P\right\|_{\mathrm{F}}\leq 0.1\| P\Sigma P\|_{\mathrm{op}}.$
Proof. We explain each condition separately. We begin with the first condition as follows: For any unit vector  $u$  in  $\mathbb{R}^d$ , we have that
$$
\left| u ^ {\top} P \left(\frac {1}{| A ^ {\prime} |} \sum_ {x \in A ^ {\prime}} x - \mu\right) \right| \leq 0. 1 \sqrt {u ^ {\top} P \Sigma P u} \leq \sqrt {\| P \Sigma P \| _ {\mathrm {o p}}},
$$
where the first inequality uses Condition (L.1) of Definition 2.1 for  $v = Pu$ .
We now focus our attention to the second condition. Denote  $\hat{\Sigma} := \frac{1}{|A'|} \sum_{x \in A'} (x - \mu)(x - \mu)^\top$  for saving space. Let  $Q$  be any symmetric matrix  $Q \in \mathbb{R}^{d \times d}$  with  $\|Q\|_{\mathrm{F}} \leq 1$ . Using the cyclical property of trace, we obtain