SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Proof. The last step of the algorithm must succeed by an averaging argument. Consider the midpoint  $t$  of the returned subinterval  $I'$ , which is at least  $q_{m_1}$  and at most  $q_{|T|-m_1}$ . Since  $T_1$  contains all points at most  $q_{m_1}$ , and  $T_2$  contains all points at most  $q_{|T|-m_1}$ , we must have  $\min(|T_1|, |T_2|) \geq m_1$ . Lastly, we verify the second desideratum, which holds if  $t \notin I$ . For the sake of contradiction, if  $t \in I$ , then since  $I$  has length  $r$  and  $I'$  has length at least  $R/(2m/n') \geq 5r$ , then  $I \subseteq I'$ . However, since  $|I' \cap T| \leq n'/2$ , we know that  $I$  cannot be strictly contained in  $I'$ , reaching the desired contradiction.
4 High-Level Proof Sketch of Theorem 1.2
We discuss the proof strategy of Theorem 1.2 at a high level. See Appendix F for the complete proof.
Proof Sketch Recall that Algorithm 1 is a recursive algorithm: each call repeatedly filters out samples before either terminating or splitting the dataset into two and recursively calling itself. The execution of Algorithm 1 can thus be viewed as a binary tree, with each node being a recursive call.
The high-level idea for proving Theorem 1.2 is straightforward, though involving technical calculations to implement. Consider the subtree grown from the root recursive call, up to and including a certain level  $j$ . We proceed by induction on the height  $j$  of such subtree, and claim there must exist a leaf node in this subtree such that most of the inliers remain in the input dataset of the leaf node.
Concretely, let  $\mathcal{T}_j$  be the subtree of height  $j$  grown from the root node. We claim that there must be a leaf in this subtree, whose input set  $T$  satisfies
$$
\alpha^ {3} \left(\epsilon_ {0} / 4 0\right) | T | + | S \setminus T | \leq (j + 1) \alpha^ {3} \left(\epsilon_ {0} / 2 0\right) m + \ell , \tag {1}
$$
recalling that  $\alpha$  is the proportion of inliers,  $\epsilon_0$  is the maximum fraction of inliers removed by the adversary,  $\ell$  is the actual (unknown) number of inliers removed and  $m$  is the size of the original dataset returned by the adversary. The left hand side keeps track of both a) how many inliers we have accidentally removed, through the  $|S\setminus T|$  term, and b) the relative proportions of the outliers we have removed versus the inliers we have removed, by comparing both terms on the left hand side.
For the induction step, we need to analyze the execution of a single recursive call. We show that (1) implies Assumption 3.1, and so the inductive step can follow the case analysis outlined in Section 3—either we terminate, or we decide to either filter or bipartition the sample set. To convert this into a high-probability statement, we use a standard (sub-)martingale argument in Appendix F.1.  $\square$
We thank the anonymous reviewers for insightful comments and suggestions on this work. Ilias Diakonikolas is supported by NSF Medium Award CCF-2107079, NSF Award CCF-1652862 (CAREER), and a DARPA Learning with Less Labels (LwLL) grant. Daniel M. Kane is supported by NSF Medium Award CCF-2107547 and NSF Award CCF-1553288 (CAREER). Jasper C.H. Lee is supported in part by the generous funding of a Croucher Fellowship for Postdoctoral Research, NSF award DMS-2023239, NSF Medium Award CCF-2107079 and NSF AiTF Award CCF-2006206. Ankit Pensia is supported by NSF Awards CCF-1652862, and CCF-1841190, and CCF-2011255; The majority of this work was done while Ankit Pensia was at UW-Madison. Thanasis Pittas is supported by NSF Medium Award CCF-2107079 and NSF Award DMS-2023239 (TRIPODS).
[AK05] S. Arora and R. Kannan. "Learning mixtures of separated nonspherical Gaussians". In: The Annals of Applied Probability 15.1A (2005), pp. 69-92.
[BBHLT20] M. Brennan, G. Bresler, S. B. Hopkins, J. Li, and T.Schramm. "Statistical Query Algorithms and Low-Degree Tests Are Almost Equivalent". In: CoRR abs/2009.06107 (2020).
Gaussian Mixture Models Gaussian mixture models (GMM) have a long history in statistics and theoretical computer science. In theoretical computer science, a system investigation of efficient algorithms for GMM was initiated by [Das99], and then improved upon in a series of works, see, for example, [VW04; AK05; DS07; KK10]. However, these algorithms imposed additional assumptions on the parameters of GMM, for example, separation between the components. Later, [MV10; BS10] gave algorithms without any assumptions on the components but these algorithms were sensitive to outliers. The field of algorithmic robust statistics has developed in the last decade to develop outlier-robust algorithms. A recent line of work [BDHKKK20; LM21; LM22; BDJKKV22] has developed efficient algorithms for learning GMM robustly.
List-decodable Learning The model of list-decodable learning was first introduced in [BBV08] and was first used in the context of high-dimensional estimation in [CSV17]. Regarding mean estimation in particular, a body of work presented algorithms with increasingly better guarantees [DKS18; KS17; RY20a; CMY20; DKK20a; DKKLT21; DKKLT22; DKKPP22; ZS22].
B Omitted Preliminaries
B.1 Notation
We use  $\mathbb{Z}_{+}$  to denote positive integers. For  $n\in \mathbb{Z}_+$ , we denote  $[n]\stackrel {\mathrm{def}}{=}\{1,\ldots ,n\}$  and use  $\mathcal{S}^{d - 1}$  for the  $d$  -dimensional unit sphere. For a vector  $v$ , we let  $\| v\| _2$  denote its  $\ell_2$  -norm. We use  $I_{d}$  to denote the  $d\times d$  identity matrix. We will drop the subscript when it is clear from the context. For a matrix  $A$ , we use  $\| A\|_{\mathrm{F}}$  and  $\| A\|_{\mathrm{op}}$  to denote the Frobenius and spectral (or operator) norms respectively. We say that a symmetric  $d\times d$  matrix  $A$  is PSD (positive semidefinite) and write  $A\succeq 0$  if for all vectors  $x\in \mathbb{R}^d$  we have that  $x^{\top}Ax\geq 0$ . We denote  $\lambda_{\max}(A)\coloneqq \max_{x\in \mathcal{S}^{d - 1}}x^{\top}Ax$ . We write  $A\succeq B$  when  $B - A$  is PSD. For a matrix  $A\in \mathbb{R}^{d\times d}$ ,  $\operatorname {tr}(A)$  denotes the trace of the matrix A. We denote by  $\langle v,u\rangle$ , the standard inner product between the vectors  $u,v$ . For matrices  $U,V\in \mathbb{R}^{d\times d}$ , we generalize the inner product as follows:  $\langle U,V\rangle = \sum_{ij}U_{ij}V_{ij}$ . For a matrix  $A\in \mathbb{R}^{d\times d}$ , we use  $A^{\flat}$  to denote the flattened vector in  $\mathbb{R}^{d^2}$ , and for a  $v\in \mathbb{R}^{d^2}$ , we use  $v^{\sharp}$  to denote the unique matrix  $A$  such that  $A^{\flat} = v^{\sharp}$ . For a matrix  $A$ , we let  $A^{\dagger}$  denote its pseudo-inverse. We use  $\otimes$  to denote the Kronecker product. For a matrix  $A$ , we use  $\ker (A)$  for the null space of  $A$  and use  $\mathbb{P}_A$  to denote the projection onto the range of  $A$ , i.e.,  $\mathbb{P}_A = AA^\dagger$  [Ber18, Proposition 8.1.7 (xii)].
We use the notation  $X \sim D$  to denote that a random variable  $X$  is distributed according to the distribution  $D$ . For a random variable  $X$ , we use  $\mathbf{E}[X]$  for its expectation. We use  $\mathcal{N}(\mu, \Sigma)$  to denote the Gaussian distribution with mean  $\mu$  and covariance matrix  $\Sigma$ . For a set  $S$ , we use  $\mathcal{U}(S)$  to denote the uniform distribution on  $S$  and use  $X \sim S$  as a shortcut for  $X \sim \mathcal{U}(S)$ .
We use  $a \lesssim b$  to denote that there exists an absolute universal constant  $C > 0$  (independent of the variables or parameters on which  $a$  and  $b$  depend) such that  $a \leq C b$ .
B.2 Miscellaneous Facts
In this section, we state standard facts that will be used in our analysis.
Fact B.1. For any matrix  $A \in \mathbb{R}^{d \times d}$ ,  $\mathbf{Var}_{X \sim \mathcal{N}(\mu, \Sigma)}[X^{\top}AX] \leq 4\|\Sigma^{1/2}A\Sigma^{1/2}\|_{\mathrm{F}}^{2} + 8\|\Sigma^{1/2}A\mu\|_{2}^{2}$ .
Proof. We write  $X \sim \mathcal{N}(\mu, \Sigma)$  as  $X = \Sigma^{1/2} Z + \mu$  for  $Z \sim \mathcal{N}(0, I)$ . We have that
$$
\begin{array}{l} \mathop{\bf V}\limits_{X\sim \mathcal{N}(\mu ,\Sigma)}[X^{\top}AX] = \mathop{\bf V}\limits_{Z\sim \mathcal{N}(0,I)}[(\Sigma^{1 / 2}Z + \mu)^{\top}A(\Sigma^{1 / 2}Z + \mu)] \\ = \mathbf {V a r} _ {Z \sim \mathcal {N} (0, I)} \left[ Z ^ {\top} \Sigma^ {1 / 2} A \Sigma^ {1 / 2} Z + 2 \mu^ {\top} A \Sigma^ {1 / 2} Z + \mu^ {\top} A \mu \right] \\ \leq 2 \left(\mathbf {V a r} _ {Z \sim \mathcal {N} (0, I)} \left[ Z ^ {\top} \Sigma^ {1 / 2} A \Sigma^ {1 / 2} Z \right] + 4 \underset {Z \sim \mathcal {N} (0, I)} {\mathbf {V a r}} \left[ \mu^ {\top} A \Sigma^ {1 / 2} Z \right]\right) \\ \leq 4 \| \Sigma^ {1 / 2} A \Sigma^ {1 / 2} \| _ {\mathrm {F}} ^ {2} + 8 \| \Sigma^ {1 / 2} A \mu \| _ {2} ^ {2}, \tag {2} \\ \end{array}
$$
where the first inequality line uses that  $\mathbf{Var}[A + B] = \mathbf{Var}[A] + \mathbf{Var}[B] + 2\mathbf{Cov}[A,B]\leq$ $\mathbf{Var}[A] + \mathbf{Var}[B] + 2\sqrt{\mathbf{Var}[A]\mathbf{Var}[B]} = (\sqrt{\mathbf{Var}[A]} +\sqrt{\mathbf{Var}[B])}^2\leq 2(\mathbf{Var}[A] + \mathbf{Var}[B])$  , and the last inequality uses the following: The first term uses the fact that for a symmetric matrix  $B$  and a vector  $v,\mathbf{Var}_{Z\sim \mathcal{N}(0,I)}[Z^{\top}BZ] = 2\| B\|_{\mathrm{F}}^{2}$  and  $\mathbf{Var}_{Z\sim \mathcal{N}(0,I)}[v^{\top}Z] = \| v\|_{2}^{2}$  , with  $B\coloneqq \Sigma^{1 / 2}A\Sigma^{1 / 2}$  and  $v\coloneqq \Sigma^{1 / 2}A\mu$
Fact B.2 (Azuma-Hoeffding Inequality). Let  $\{X_{n}\}_{n\in \mathbb{N}}$  be a submartingale or supermartingale with respect to a sequence  $\{Y_{n}\}_{n\in \mathbb{N}}$ . If for all  $n = 1,2,\ldots$  it holds  $|X_{n} - X_{n - 1}|\leq c_{n}$  almost surely, then for any  $\epsilon >0$  and  $n\in \mathbb{N}$
$$
\Pr \left[ X _ {n} - X _ {t} > \epsilon \right] \leq \exp \left(- \frac {2 \epsilon^ {2}}{\sum_ {t = 1} ^ {\infty} c _ {t}}\right).
$$
Fact B.3 (See, e.g., [Ber18, Proposition 11.10.34]). Let  $R$  and  $M$  be two square matrices, then  $\| RM\|_{\mathrm{F}}\leq \min \left(\left(\| R\|_{\mathrm{F}}\| M\right\|_{\mathrm{op}},\| R\|_{\mathrm{op}}\| M\|_{\mathrm{F}}\right)$ .
Fact B.4 (Frobenius norm and projection matrices). Let  $A$  be a symmetric matrix. If  $L_{1}$  and  $L_{2}$  are two PSD projection matrices satisfying  $L_{1} \preceq L_{2}$ , then  $\| L_{1}A L_{1}^{\top}\|_{\mathrm{F}} \leq \| L_{2}A L_{2}^{\top}\|_{\mathrm{F}}$ .
Fact B.5 (Pseudo-inverse of a matrix). Let  $G$  be a square matrix and define  $R = G^{\dagger}$ . Let the SVD of  $G$  be  $U\Lambda V^{\top}$ . Then  $R = V\Lambda^{\dagger}U^{\top}$ ,  $\mathbb{P}_G = U\Lambda \Lambda^{\dagger}U^{\top}$ , and  $R = R\mathbb{P}_G$ . Moreover, let  $\Delta$  be any arbitrary diagonal matrix such that if  $\Lambda_{i,i} = 0$ , then  $\Delta_{i,i} = 0$ , and define  $L = V\Delta V^{\top}$ . Then  $L = LRG$ .
Proof. The first follows from [Ber18, Chapter 8], the second follows from [Ber18, Proposition 8.1.7 (xii)], and the third follows from the first two and the fact that  $\Lambda^{\dagger}\Lambda \Lambda^{\dagger} = \Lambda$ . The final claim follows from the following series of equalities:
$$
\begin{array}{l} L R G = V \Delta V ^ {\top} V \Lambda^ {\dagger} U ^ {\top} U \Lambda V ^ {\top} \\ = V \Delta V ^ {\top} V \Lambda^ {\dagger} U ^ {\top} U \Lambda V ^ {\top} \\ = V \Delta \Lambda^ {\dagger} \Lambda V ^ {\top} \\ = V \Delta V ^ {\top} = L, \\ \end{array}
$$
where we use that  $\Delta$  is diagonal and for each  $i$  such that  $\Delta_{i,i} \neq 0$ ,  $\Lambda_{i,i}^{\dagger}\Lambda_{i,i} = 1$ .
Fact B.6 ([Ber18, Fact 8.4.16]). Let  $A$  and  $B$  be two square matrices and suppose that  $A$  is full rank. Then  $\mathbb{P}_{BA} = \mathbb{P}_B$ .
C Linear Algebraic Results
In this section, we prove the proofs of certain linear algebraic results that we need. In particular, we provide the proofs of Lemmata C.1 and C.3 that will be used later.
C.1 Relative Frobenius Norm to Frobenius Norm: Proof of Lemma C.1
In this section, we provide the proof of Lemma C.1.
Lemma C.1. Consider two arbitrary full rank, positive definite matrices  $\Sigma_{1}$  and  $\Sigma_{2}$ , and suppose there exists a full rank, positive definite matrix  $H$  such that (i)  $\|I - H^{\dagger/2} \Sigma_{1} H^{\dagger/2}\|_{\mathrm{F}} \leq \rho$  and (ii)  $\|I - H^{\dagger/2} \Sigma_{2} H^{\dagger/2}\|_{\mathrm{F}} \leq \rho$ . Then, for an arbitrary full rank, positive definite matrix  $\Sigma$ , we have
$$
\| \Sigma^ {- 1 / 2} \Sigma_ {1} \Sigma^ {- 1 / 2} - \Sigma^ {- 1 / 2} \Sigma_ {2} \Sigma^ {- 1 / 2} \| _ {\mathrm {F}} \leq 5 \rho \max  (\| \Sigma^ {- 1 / 2} \Sigma_ {1} \Sigma^ {- 1 / 2} \| _ {\mathrm {o p}}, \| \Sigma^ {- 1 / 2} \Sigma_ {2} \Sigma^ {- 1 / 2} \| _ {\mathrm {o p}}).
$$
Proof of Lemma C.1. Let  $G = H^{-1/2}\Sigma^{1/2}$ . Let  $\widetilde{\Sigma}_1 = \Sigma^{-1/2}\Sigma_1\Sigma^{-1/2}$  and  $\widetilde{\Sigma}_2 = \Sigma^{-1/2}\Sigma_2\Sigma^{-1/2}$ , both of which are positive definite matrices. Since  $\Sigma$  is full rank, we obtain that
$$
\begin{array}{l} \Sigma_ {1} = \Sigma^ {1 / 2} \Sigma^ {- 1 / 2} \Sigma_ {1} \Sigma^ {- 1 / 2} \Sigma^ {1 / 2} = \Sigma^ {1 / 2} \widetilde {\Sigma} _ {1} \Sigma^ {1 / 2} \\ \Sigma_ {2} = \Sigma^ {1 / 2} \Sigma^ {- 1 / 2} \Sigma_ {2} \Sigma^ {- 1 / 2} \Sigma^ {1 / 2} = \Sigma^ {1 / 2} \widetilde {\Sigma} _ {2} \Sigma^ {1 / 2}. \\ \end{array}
$$
This directly gives us that  $G\widetilde{\Sigma}_1G^\top = I - H\Sigma_1H$  and  $G\widetilde{\Sigma}_2G^\top = I - H\Sigma_2H$ . Overall, we have obtained the following:
$$
\| I - G \widetilde {\Sigma} _ {1} G ^ {\top} \| _ {\mathrm {F}} \leq \rho \quad \text {a n d} \quad \| I - G \widetilde {\Sigma} _ {2} G ^ {\top} \| _ {\mathrm {F}} \leq \rho .
$$
We first state a result, proved in the end of this section, that transfers closeness in relative Frobenius norm to Frobenius norm of the difference (along most of the directions).
Lemma C.2. Let  $G$  be a  $d \times d$  matrix and let  $B$  be an arbitrary symmetric matrix of the same dimension. Recall that  $\mathbb{P}_G$  denotes the projection matrix onto range of  $G$ . Suppose that
$$
\left\| \mathbb {P} _ {G} - G B G ^ {\top} \right\| _ {\mathrm {F}} \leq \rho
$$
for some  $\rho \geq 1$ . Let the SVD of  $G$  be  $G = U\Lambda V^{\top}$  for some orthonormal matrices  $U$  and  $V$  and a diagonal matrix  $\Lambda$ .
Then there exists a  $d \times d$  matrix  $L$  such that
1. (L captures directions of  $G$ )  $L = V\Delta_{B}V^{\top}$  for some binary diagonal matrix  $\Delta_B \preceq \Lambda \Lambda^\dagger$ . That is, the diagonal entries of  $\Delta_B$  are all in  $\{0,1\}$ , and furthermore, all the 1 diagonal entries of  $\Delta_B$  are also non-zero diagonal entries in  $\Lambda$ .
2. (rank of  $L$ ) The rank of  $L$  satisfies  $\| \Lambda - \Delta_B \|_{\mathrm{F}} \leq 2\rho$  and
3. (closeness of  $G^{\dagger}$  and  $B$  along  $L$ )  $\| L(RR^{\top} - B)L^{\top}\|_{\mathrm{F}}\leq 2\rho \| B\|_{\mathrm{op}}$ , where  $R = G^{\dagger}$ .
We will now apply Lemma C.2 twice following the same notation as in its statement. In particular, the SVD decomposition of  $G = U\Delta V^{\top}$  and  $R = G^{\dagger}$ . Lemma C.2 implies that there exist two binary diagonal matrices  $\Delta_1$  and  $\Delta_2$ , where  $\Delta_1$  is for  $\widetilde{\Sigma}_1$  and  $\Delta_2$  is for  $\widetilde{\Sigma}_2$  such that (i) both  $\| I - \Delta_1\|_{\mathrm{F}}$  and  $\| I - \Delta_1\|_{\mathrm{F}}$  are at  $2\rho$ , and (ii)  $V\Delta_1V^\top (RR^\top -\widetilde{\Sigma}_1)V\Delta_1V^\top$  and  $V\Delta_2V^\top (RR^\top -\widetilde{\Sigma}_2)V\Delta_2V^\top$  have Frobenius norms at most  $2\rho \| \widetilde{\Sigma}_1\|_{\mathrm{op}}$  and  $2\rho \| \widetilde{\Sigma}_2\|_{\mathrm{op}}$ , respectively. Let  $\Delta = \Delta_1\times \Delta_2$  and define  $L = V\Delta V^{\top}$ . By (i), we have that  $\| I - L\|_{\mathrm{F}} = \| I - \Delta \|_{\mathrm{F}}\leq 4\rho$ . Since  $L\spreceq V\Delta_1V^\top$  and  $L$  is a projection matrix, Fact B.4 along with (ii) above implies that
$$
\left\| L \left(R R ^ {\top} - \widetilde {\Sigma} _ {1}\right) L ^ {\top} \right\| _ {F} \leq 2 \rho \| \widetilde {\Sigma} _ {1} \| _ {\mathrm {o p}}.
$$
Similarly for  $\widetilde{\Sigma}_2$ . By triangle inequality, we obtain that
$$
\left\| L \left(\widetilde {\Sigma} _ {1} - \widetilde {\Sigma} _ {2}\right) L ^ {\top} \right\| _ {\mathrm {F}} \leq 4 \rho \max  \left(\left\| \widetilde {\Sigma} _ {1} \right\| _ {\mathrm {o p}}, \left\| \widetilde {\Sigma} _ {2} \right\| _ {\mathrm {o p}}\right).
$$
Since  $L$  is a projection matrix, triangle inequality implies that
$$
\begin{array}{l} \left\| \left(\widetilde {\Sigma} _ {1} - \widetilde {\Sigma} _ {2}\right) \right\| _ {\mathrm {F}} \leq \left\| L \left(\widetilde {\Sigma} _ {1} - \widetilde {\Sigma} _ {2}\right) L ^ {\top} \right\| _ {\mathrm {F}} + \left\| (I - L) \left(\widetilde {\Sigma} _ {1} - \widetilde {\Sigma} _ {2}\right) (I - L) ^ {\top} \right\| _ {\mathrm {F}} \\ \lesssim \left\| L \left(\widetilde {\Sigma} _ {1} - \widetilde {\Sigma} _ {2}\right) L ^ {\top} \right\| _ {F} + \| I - L \| _ {F} \cdot \left\| \left(\widetilde {\Sigma} _ {1} - \widetilde {\Sigma} _ {2}\right) \right\| _ {\text {o p}} \\ \lesssim \rho \cdot \max  \left(\| \widetilde {\Sigma} _ {1} \| _ {\mathrm {o p}}, \| \widetilde {\Sigma} _ {2} \| _ {\mathrm {o p}}\right). \\ \end{array}
$$
We now provide the proof of Lemma C.2.
$$
\| \mathbb {P} _ {G} - G B G ^ {\top} \| _ {\mathrm {F}} \leq \rho
$$
1. (L captures directions of  $G$ )  $L = V\Delta_{B}V^{\top}$  for some binary diagonal matrix  $\Delta_B \preceq \Lambda \Lambda^\dagger$ . That is, the diagonal entries of  $\Delta_B$  are all in  $\{0,1\}$ , and furthermore, all the 1 diagonal entries of  $\Delta_B$  are also non-zero diagonal entries in  $\Lambda$ .
2. (rank of  $L$ ) The rank of  $L$  satisfies  $\| \Lambda - \Delta_B \|_{\mathbb{F}} \leq 2\rho$  and
3. (closeness of  $G^{\dagger}$  and  $B$  along  $L$ )  $\| L(RR^{\top} - B)L^{\top}\|_{\mathrm{F}}\leq 2\rho \| B\|_{\mathrm{op}}$ , where  $R = G^{\dagger}$ .
Proof. Let  $u_{1}, u_{2}, \ldots, u_{d}$  be the columns of  $U$  and let  $v_{1}, v_{2}, \ldots, v_{d}$  be the columns of  $V$ . Let the entries of  $\Lambda$  be  $\lambda_{1}, \lambda_{2}, \ldots, \lambda_{d}$ , which are non-negative by SVD property. Let  $\mathcal{J}$  be the set of  $i$ 's in  $[d]$  such that  $\lambda_{i} > 0$ .
Define the matrices  $F\coloneqq U^{\top}GBG^{\top}U$  and  $B^{\prime}\coloneqq V^{\top}BV$ . Then since  $G^{\top} = V\Lambda U^{\top}$ , we have
$$
F = U ^ {\top} G B G ^ {\top} U = U ^ {\top} U \Lambda V ^ {\top} B V \Lambda U ^ {\top} U = \Lambda V ^ {\top} B V \Lambda = \Lambda B ^ {\prime} \Lambda .
$$
Thus the  $(i,j)$ -th entry of  $F$  is  $B_{i,j}^{\prime}\lambda_{i}\lambda_{j}$ .
Now define the matrix  $M := U^{\top} \mathbb{P}_G U$ . From Fact B.5, we have that  $\mathbb{P}_G = U \Lambda \Lambda^\dagger U^\top$ . And thus,  $M = \Lambda \Lambda^\dagger$ , which is the diagonal matrix with entries  $M_{ii} = 1$  for each  $i \in \mathcal{I}$ , and  $M_{ii} = 0$  otherwise.
Using the invariance of Frobenius norm under orthogonal transform, we obtain the following series of equalities:
$$
\begin{array}{l} \| \mathbb {P} _ {G} - G B G ^ {\top} \| _ {\mathrm {F}} ^ {2} \\ = \left\| U ^ {\top} \left(\mathbb {P} _ {G} - G B G ^ {\top}\right) U \right\| _ {\mathrm {F}} ^ {2} \\ = \| M - F \| _ {\mathrm {F}} ^ {2} \\ \geq \sum_ {i \in \mathcal {I}} (M _ {i, i} - F _ {i, i}) ^ {2} \\ = \sum_ {i \in \mathcal {J}} \left(1 - B _ {i, i} ^ {\prime} \cdot \lambda_ {i} ^ {2}\right) ^ {2} \tag {3} \\ \end{array}
$$
Let  $\mathcal{I} \subseteq \mathcal{J}$  be the set of  $i$ 's in  $\mathcal{J}$  such that  $\lambda_i^2 B_{i,i}' < (1/2)$ . Then  $\sum_{i \in \mathcal{I}} \left(1 - B_{i,i}' \cdot \lambda_i^2\right)^2 > |\mathcal{I}|/4$ . Thus if  $|\mathcal{I}|$  is larger than  $4\rho^2$ , then the Frobenius norm squared is larger than  $r^2$ , which is a contradiction.
If  $i\in \mathcal{J}\setminus \mathcal{I}$  , then  $\lambda_i^2\geq 1 / (2B_{i,i}^{\prime})$  and not just  $\lambda_{i} > 0$
Define the matrix  $L$  to be  $L \coloneqq \sum_{i: i \in \mathcal{I} \setminus \mathcal{I}} v_i v_i^\top$ , which is equal to  $V \Delta_B V^\top$  where  $\Delta_B$  is a diagonal matrix with the  $i$ -th diagonal entry equal to 1 if  $i \in \mathcal{I} \setminus \mathcal{I}$  and 0 otherwise. By definition,  $L$  has rank  $|\mathcal{J}| - |\mathcal{I}|$  and furthermore, the 1 diagonal entries of  $\Delta_B$  are always non-zero diagonal entries in  $\Lambda$ .
Let  $R = G^{\dagger}$ , and by Fact B.5, we have that  $R = V\Lambda^{\dagger}U^{\top}$ . Thus  $LR = V\Delta_{B}V^{\top}V\Lambda^{\dagger}U^{\top} = V\Delta_{B}\Lambda^{\dagger}U^{\top}$ , and thus the singular values of  $LR$  satisfy that they are at most
$$
\frac {1}{\min _ {i \in \mathcal {I} \setminus \mathcal {I}} \lambda_ {i}} \leq \max _ {i} \sqrt {2 B _ {i , i} ^ {\prime}} \leq \sqrt {2 \| B ^ {\prime} \| _ {\mathrm {o p}}} = \sqrt {2 \| V ^ {\top} B V \| _ {\mathrm {o p}}} = \sqrt {2 \| B \| _ {\mathrm {o p}}}.
$$
We first need two more observations:  $L = LRG$  and  $LRR^{\top}L^{\top} = LR\mathbb{P}_{G}R^{\top}L^{\top}$ , both of which hold by Fact B.5. Finally, we show that  $RR^{\top}$  and  $B$  are close along the directions in  $L$ .
$$
\begin{array}{l} \| L (R R ^ {\top} - B) L ^ {\top} \| _ {F} \\ = \left\| L R R ^ {\top} L ^ {\top} - L B L ^ {\top} \right\| _ {F} \\ = \left\| L R \mathbb {P} _ {G} R ^ {\top} L ^ {\top} - L R G B G ^ {\top} R ^ {\top} L ^ {\top} \right\| _ {F} \\ = \left\| L R \left(\mathbb {P} _ {G} - G B G ^ {\top}\right) R ^ {\top} L ^ {\top} \right\| _ {F} \\ \leq \| L R \| _ {\mathrm {o p}} \| \mathbb {P} _ {G} - G B G ^ {\top} \| _ {\mathrm {F}} \| R ^ {\top} L ^ {\top} \| _ {\mathrm {o p}} \\ = \| L R \| _ {\mathrm {o p}} ^ {2} \| \mathbb {P} _ {G} - G B G ^ {\top} \| _ {\mathrm {F}} \\ \leq 2 \| B \| _ {\mathrm {o p}} \cdot \rho . \tag {4} \\ \end{array}
$$
C.2 Proof of Lemma C.3
We now restate and prove Lemma C.3.
Lemma C.3. Let  $A$  and  $B$  be two PSD matrices with  $\mathbb{P}_B \preceq \mathbb{P}_A$  or equivalently  $\ker(A) \subseteq \ker(B)$ . Then  $\|ABA - \mathbb{P}_B\|_{\mathbb{F}} \leq 2\|ABA - \mathbb{P}_A\|_{\mathbb{F}}$ .
Proof. Applying triangle inequality, we have that
$$
\| A B A - \mathbb {P} _ {B} \| _ {\mathrm {F}} \leq \| A B A - \mathbb {P} _ {A} \| _ {\mathrm {F}} + \| \mathbb {P} _ {A} - \mathbb {P} _ {B} \| _ {\mathrm {F}}.
$$
It thus suffices to show that the second term,  $\| \mathbb{P}_B - \mathbb{P}_A\|_{\mathrm{F}}$ , is less than  $\| ABA - \mathbb{P}_A\|_{\mathrm{F}}$ . Let  $d_A$  be the rank of  $A$  and  $d_B$  be the rank of  $B$ . We will show the following two results, which imply the desired bound:
$$
\left\| \mathbb {P} _ {B} - \mathbb {P} _ {A} \right\| _ {\mathrm {F}} ^ {2} = \left| d _ {B} - d _ {A} \right| \leq \left\| A B A - \mathbb {P} _ {B} \right\| _ {\mathrm {F}} ^ {2} \tag {5}
$$
In the rest of the proof, we will establish the above two inequalities.
We begin with the second inequality in (5). By the lemma assumptions, we have that the null space of  $A$  being a subspace of the null space of  $B$ . This is equivalent to saying that the column space of  $B$  being a subspace of the column space of  $A$ . In particular, this also implies that  $d_B \leq d_A$ . It is easy to see that  $\operatorname{rank}(ABA) \leq \operatorname{rank}(B) \leq \operatorname{rank}(A) = \operatorname{rank}(\mathbb{P}_A)$ , and the eigenvalues of  $\mathbb{P}_A$  are either 0 or 1. We now state a simple result lower bounding the Frobenius norm by the difference of the ranks for such matrices:
Claim C.4 (Bound on rank difference). Let  $A'$  be a symmetric matrix such that the eigenvalues of  $A'$  belong to  $\{0,1,-1\}$ . Let  $B'$  be an arbitrary matrix. Then  $\| B' - A' \|_{\mathrm{F}}^2 \geq \operatorname{rank}(A') - \operatorname{rank}(B')$ .