SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \alpha^ {3} \left(\epsilon_ {0} / 4 0\right) | T ^ {\prime} | + | S \setminus T ^ {\prime} | \leq \alpha^ {3} \left(\epsilon_ {0} / 4 0\right) | T _ {\left(t _ {i ^ {\prime}} ^ {*}\right)} ^ {\left(i ^ {\prime}, j\right)} | + | S \setminus T ^ {\prime} | \quad \text {(s i n c e} T ^ {\prime} \subset T _ {\left(t _ {i ^ {\prime}} ^ {*}\right)} ^ {\left(i ^ {\prime}, j\right)} \\ \leq \alpha^ {3} \left(\epsilon_ {0} / 4 0\right) \left| T _ {\left(t _ {i ^ {\prime}} ^ {*}\right)} ^ {\left(i ^ {\prime}, j\right)} \right| + \left| S \setminus S _ {\text {c o r e}} ^ {\left(i ^ {\prime}, j\right)} \right| \quad (\text {b y}) \\ \leq \alpha^ {3} (\epsilon_ {0} / 4 0) | T _ {(t _ {i ^ {\prime}} ^ {*})} ^ {(i ^ {\prime}, j)} | + | S \setminus T _ {(t _ {i ^ {\prime}} ^ {*})} ^ {(i ^ {\prime}, j)} | + | S \setminus S _ {H, A} | \\ \end{array}
$$
(by the definition of  $S_{\mathrm{core}}^{(i',j)}$  and a union bound)
$$
\begin{array}{l} \leq \alpha^ {3} \left(\epsilon_ {0} / 4 0\right) \left| T _ {\left(t _ {i ^ {\prime}} ^ {*}\right)} ^ {\left(i ^ {\prime}, j\right)} \right| + \left| S \setminus T _ {\left(t _ {i ^ {\prime}} ^ {*}\right)} ^ {\left(i ^ {\prime}, j\right)} \right| + \eta n \quad (\text {b y}) \\ \leq (j + 1) \alpha^ {3} \left(\epsilon_ {0} / 2 0\right) m + \ell + \left(\epsilon_ {0} / 4 0\right) \alpha^ {3} m + \eta n \quad \text {(b y} \\ \leq (j + 1) \alpha^ {3} (\epsilon_ {0} / 2 0) m + \ell + (\epsilon_ {0} / 2 0) \alpha^ {3} m \quad \text {(s i n c e} \eta = (\epsilon_ {0} / 4 0) \alpha^ {3} \text {a n d} n \leq m) \\ \end{array}
$$
$$
= (j + 2) \alpha^ {3} \left(\epsilon_ {0} / 2 0\right) m + \ell . \tag {16}
$$
Thus Condition 3 is satisfied by  $T'$  for case  $j + 1$ , completing the inductive proof of this lemma.
We now use the lemma to conclude the proof of Theorem F.1.
By Lemma F.4, we know that with probability at least  $1 - 0.09\delta$ , the recursion tree generated up to level  $9 / \alpha$ , namely  $\mathcal{T}_{9 / \alpha}$ , satisfies the lemma guarantees. In particular, since (i) by Line 19 in Algorithm 1, the input to any recursive call must have size at least  $m_{1} = \alpha m / 9$ , and (ii) Condition 1 of Lemma F.4 applied to  $j = m / m_{1} = 9 / \alpha$  yields an input size upper bound of 0, we can conclude that the execution of the algorithm must have completely terminated by level  $9 / \alpha$ . We know by Lemma F.4 that there exists a leaf node  $i$  in the recursion tree whose input satisfies Condition 3 of the third part of the lemma statement. That is, letting  $j$  denote the level of that leaf, we have that
$$
\alpha^ {3} \left(\epsilon_ {0} / 4 0\right) \left| T _ {(0)} ^ {(i, j)} \right| + \left| S \setminus T _ {(0)} ^ {(i, j)} \right| \leq (j + 1) \alpha^ {3} \left(\epsilon_ {0} / 2 0\right) m + \ell . \tag {17}
$$
The recursion node  $\mathcal{T}_{i,j}$  starts with that input set  $T_{(0)}^{(i,j)}$  and, before terminating, it may perform some filtering steps. As in the proof of Lemma F.4, we will use Lemma F.6 to analyze the "working set" right before the recursive call terminates. If we denote by  $t^*$  the number of filtering steps before termination, Lemma F.6 yields that except with probability  $0.01\alpha \delta$ , we have
$$
\alpha^ {3} \left(\epsilon_ {0} / 4 0\right) \left| T _ {\left(t ^ {*}\right)} ^ {(i, j)} \right| + \left| S \setminus T _ {\left(t ^ {*}\right)} ^ {(i, j)} \right| \leq (j + 1) \alpha^ {3} \left(\epsilon_ {0} / 2 0\right) m + \ell + \left(\epsilon_ {0} / 4 0\right) \alpha^ {3} m. \tag {18}
$$
Note that by Fact F.2, the total failure probability is upper bounded by  $0.09\delta + 0.01\alpha \delta$  which is less than  $\delta$ .
We are now ready to prove the first bullet of Theorem F.1. The above inequality implies that
$$
\begin{array}{l} | T _ {(t ^ {*})} ^ {(i, j)} \cap S | = | S | - | S \setminus T _ {(t ^ {*})} ^ {(i, j)} | \\ \geq n - \ell - \left(\epsilon_ {0} / 2 0\right) (j + 1) \alpha^ {3} m - \left(\epsilon_ {0} / 4 0\right) \alpha^ {3} m (by(18)) \\ \geq n - \ell - (\epsilon_ {0} / 2 0) (j + 2) \alpha^ {3} m \\ \geq n - \ell - \left(\epsilon_ {0} / 2 0\right) (j + 2) \alpha^ {2} n \quad (n \geq \alpha m) \\ \geq n - \ell - (\epsilon_ {0} / 2) \alpha n \quad (j \leq 9 / \alpha \text {a n d} \alpha <   1 / 2) \\ \geq n - \ell - \epsilon_ {0} (1 - \epsilon_ {0}) \alpha n \quad (1 / 2 \leq 1 - \epsilon_ {0}) \\ \geq n - \ell - \epsilon_ {0} \alpha (n - \ell) \quad (\ell \leq \epsilon_ {0} n) \\ = (n - \ell) (1 - \epsilon_ {0} \alpha). (19) \\ \end{array}
$$
(19) further implies that  $|T_{(t^{*})}^{(i,j)} \cap S| \geq (n - \ell)(1 - \epsilon_0\alpha) \geq n(1 - \epsilon_0)(1 - \epsilon_0\alpha) \geq (\alpha / 2)m$  (since  $n \geq \alpha m$  and  $\epsilon_0$  and  $\alpha$  are small), meaning that the set  $T_{(t^{*})}^{(i,j)}$  will indeed be returned. This completes the proof of the first bullet of Theorem F.1.
We now move to the second bullet. Defining  $H = \mathbf{E}_{X \sim T_{(t^*)}^{(i,j)}}[XX^\top]$ , we need to show that  $\| H^{\dagger/2}\Sigma H^{\dagger/2} - HH^\dagger \|_{\mathrm{F}} \lesssim (1/\alpha^4)\log^2(1/\alpha)$ . We will apply Lemma E.2 to show the bounds in the second bullet, which requires checking that Assumption 3.1 is satisfied by  $T_{(t^*)}^{(i,j)}$ . Earlier, right below (19), we have already checked that  $|T_{(t^*)}^{(i,j)} \cap S| \geq (\alpha/2)|T_{(t^*)}^{(i,j)}|$ . The only remaining condition to check is  $|T_{(t^*)}^{(i,j)}| \leq |S| / \alpha$ , but this is trivially true since  $T_{(t^*)}^{(i,j)} \subset T_{(0)}^{(0,0)}$  (the original input set to the recursive algorithm) and  $|T_{(0)}^{(0,0)}| \leq |S| / \alpha$  by Definition 1.1. Thus, Lemma E.2 yields that
$$
\begin{array}{l} \left\| H^{\dagger /2}\Sigma H^{\dagger /2} - \mathbb{P}_{H}\right\|_{F}^{2}\lesssim \frac{1}{\alpha}\underset {\widetilde{X}\sim T_{(t^{*})}^{(i,j)}}{\operatorname{Var}}[\widetilde{X}^{\top}A\widetilde{X} ] + \frac{1}{\alpha^{2}} \\ \leq \frac {1}{\alpha} \underset {\widetilde {X} \sim T _ {(t ^ {*})} ^ {(i, j)}} {\mathbf {E}} [ f (\widetilde {X}) ] + \frac {1}{\alpha^ {2}} \quad (\operatorname {V a r} (Y) \leq \mathbf {E} [ (Y - c) ^ {2} ] \text {f o r a n y} c \in \mathbb {R}) \\ \end{array}
$$
$$
\begin{array}{l} \lesssim \frac {R ^ {2}}{\alpha^ {4}} \quad \left(\text {c . f . L i n e 1 2 o f A l g o r i t h m 1}\right) \\ \lesssim \frac {1}{\alpha^ {8}} \log^ {2} \left(\frac {1}{\alpha}\right), \quad (R = \Theta ((1 / \alpha^ {2}) \log (1 / \alpha)), \text {n o t i n g} \epsilon_ {0} = 0. 0 1) \\ \end{array}
$$
where the penultimate inequality uses the fact that, if the algorithm terminated, it must be the case that  $\mathbf{E}_{\widetilde{X}\sim T_{(t^{*})}^{(i,j)}}[f(\widetilde{X})]\leq C'R^2 /\alpha^3$  by Line 12 in Algorithm 1. Taking a square root on both sides implies that
$$
\left\| H ^ {\dagger / 2} \Sigma H ^ {\dagger / 2} - \mathbb {P} _ {H} \right\| _ {\mathrm {F}} \lesssim \frac {1}{\alpha^ {4}} \log \left(\frac {1}{\alpha}\right)
$$
The same guarantee holds for  $\left\| H^{\dagger /2}\Sigma H^{\dagger /2} - \mathbb{P}_{\Sigma}\right\|_{\mathrm{F}}$ , also following from Lemma E.2.
Lastly, we check that we only return  $O(1 / \alpha)$  sets in the output list. This is true by construction: (i) Line 13 only allows sets to be output if they have size at least  $\Omega (\alpha m)$ , (ii) there were only  $m$  points to begin with and (iii) all the leaves have disjoint input sets by Lemma F.4.
F.1 Proof of Lemma F.6
We now prove that filtering does not remove too many inliers using a standard martingale argument.
Lemma F.6. Recall the notations and assumptions from Assumption 3.1. Consider an execution of COVLISTDECODING  $(T_0)$  with  $T_0 = T$  (with  $T$  satisfying the assumptions from Assumption 3.1), and suppose that  $|T_0| \geq C' / (\alpha^6 \epsilon_0^2) \log (1 / (\alpha \delta))$ . Further assume that  $|S \setminus T_0| \leq 1.5 \epsilon_0 |S|$  (which is a strengthening over Assumption 3.1). Moreover, denote by  $T_t$  the dataset through the loop iterations of Algorithm 1. Then, with probability at least  $1 - 0.01 \alpha \delta$ , it holds that
$$
\alpha^ {3} (\epsilon_ {0} / 4 0) | T _ {t} | + | S \setminus T _ {t} | \leq \alpha^ {3} (\epsilon_ {0} / 4 0) | T _ {0} | + | S \setminus T _ {0} | + \alpha^ {3} (\epsilon_ {0} / 4 0) | T _ {0} |,
$$
Proof. We will use the notation of Assumption 3.1 for a single call of Algorithm 1. Denote by  $t$  the iteration count. Also define the stopping time  $t_\mathrm{end}$  to be the first iteration when  $\alpha^3 (\epsilon_0 / 40)|T_{t_\mathrm{end}}| + |S\setminus T_{t_\mathrm{end}}| > 2\epsilon_0|S|$  or when the iteration goes into Line 12 or Line 19 instead of Line 16 (that is, when  $\mathbf{E}_X\sim T_{t_\mathrm{end}}[f(H_t^\dagger /2X)]\leq C'R^2 /\alpha^3$  or when  $q_{\mathrm{right}} - q_{\mathrm{left}} > R$  for the iteration  $t_\mathrm{end})$ . Now, define  $\Delta_t = \alpha^3 (\epsilon_0 / 40)|T_{\min \{t,t_\mathrm{end}\}}| + |S\setminus T_{\min \{t,t_\mathrm{end}\}}|$ .
In order to prove the lemma, we will show that at the first  $t^*$  (if one exists) such that  $\alpha^3 (\epsilon_0 / 40)|T_{t^*}| + |S\setminus T_{t^*}| > \alpha^3 (\epsilon_0 / 40)|T_0| + |S\setminus T_0| + \alpha^3 (\epsilon_0 / 40)|T_0|$ , then  $\Delta_{t^*} > \alpha^3 (\epsilon_0 / 40)|T_0| + |S\setminus T_0| + \alpha^3 (\epsilon_0 / 40)|T_0|$  as well. Afterwards, we will show that  $\Delta_t$  is a sub-martingale and use sub-martingale tail bounds to show that the sequence  $\Delta_t$  remains small over an entire trajectory of  $|T_0|$  steps with high probability.
The first step is easy to show, since the threshold of  $\alpha^3 (\epsilon_0 / 40)|T_0| + |S\setminus T_0| + \alpha^3 (\epsilon_0 / 40)|T_0| < \alpha^3 (\epsilon_0 / 20)|T_0| + 1.5\epsilon_0|S|\leq \alpha^2 (\epsilon_0 / 20)|S| + 1.5\epsilon_0|S|\leq 2\epsilon_0|S|$ , where the first inequality is by the lemma assumption. Therefore,  $t^{*}\leq t_{\mathrm{end}}$  if it exists, meaning that  $\Delta_{t^{*}} = \alpha^{3}(\epsilon_{0} / 40)|T_{t^{*}}| + |S\setminus T_{t^{*}}|$ .
Now we need to show that  $\Delta_t$  is a sub-martingale with respect to the sequence  $T_{\min\{t, t_{\mathrm{end}}\}}$ . If  $t \geq t_{\mathrm{end}}$ , then  $\Delta_{t+1}$  is by definition equal to  $\Delta_t$  and thus  $\mathbf{E}[\Delta_{t+1} \mid T_{\min\{t, t_{\mathrm{end}}\}}] = \Delta_t$ . Otherwise, we have that  $\mathbf{E}[\Delta_{t+1} \mid T_t]$  is equal to  $\Delta_t$  plus the expected number of inliers removed minus  $\alpha^3 (\epsilon_0 / 40)$  times the expected number of all points removed by our filter. Since the stopping condition is not satisfied, we have  $|S \setminus T_t| \leq 2\epsilon_0 |S|$ , meaning that Assumption 3.1 continues to hold for  $T_t$ , as well as that the other conditions for Lemma E.4 hold. We can therefore apply Lemma E.4 to obtain
$$
\mathbf {E} [ \Delta_ {t + 1} \mid T _ {t} ] = \Delta_ {t} + \sum_ {\widetilde {x} \in \widetilde {S} \cap \widetilde {T} _ {t}} f (\widetilde {x}) - \alpha^ {3} \frac {\epsilon_ {0}}{4 0} \sum_ {\widetilde {x} \in \widetilde {T} _ {t}} f (\widetilde {x}) \leq \Delta_ {t},
$$
Summarizing, the above case analysis shows that  $\{\Delta_t\}_{t\in \mathbb{N}}$  is a sub-martingale with respect to  $T_{\min \{t,t_{\mathrm{end}}\}}$ .
We note also that for every  $t$ ,  $|\Delta_t - \Delta_{t + 1}| \leq 1$  with probability 1. Thus, using the standard Azuma-Hoeffding inequality for sub-martingales (Fact B.2), we have that for every  $t \leq |T_0|$
$$
\mathbf {P r} \left[ \Delta_ {t} - \Delta_ {0} > (\epsilon_ {0} / 4 0) \alpha^ {3} | T _ {0} | \right] \leq e ^ {- (2 / 4 0 ^ {2}) \cdot \epsilon_ {0} ^ {2} \alpha^ {6} | T _ {0} |}.
$$
By a union bound over all  $t \in [|T_0|]$ , we have
$$
\Pr \left[ \exists t \in [ m ]: \Delta_ {t} > \Delta_ {0} + \alpha^ {3} \left(\epsilon_ {0} / 4 0\right) \mid T _ {0} \mid \right] \leq \left| T _ {0} \right| e ^ {- \left(2 / 4 0 ^ {2}\right) \cdot \epsilon_ {0} ^ {2} \alpha^ {6} \left| T _ {0} \right|} \leq 0. 0 1 \alpha \delta , \tag {20}
$$
where the last inequality uses that  $|T_0| > \frac{C'}{\alpha^6\epsilon_0^2}\log \left(\frac{1}{\alpha\delta}\right)$  for a sufficiently large absolute constant  $C'$ .
G Applications to Learning Gaussian Mixture Models
In this section, we prove the applications of our result to robustly learning Gaussian mixture models.
We show the first two bullets as a direct corollary of Theorem 1.2 applied to each Gaussian component  $D = \mathcal{N}(\mu_p, \Sigma_p)$  as the inlier distribution. The proof of the final bullet involves some more involved linear algebraic lemmas. Here we give only the high-level proof of the final bullet, and defer the proof of these individual lemmas to Appendix C.
Proof. First observe that, with a sufficiently large  $\mathrm{poly}(d,\frac{1}{\alpha},\log \frac{1}{\delta})$  samples, with probability at least  $1 - \delta /4$ , the number of inliers  $n_p$  from component  $p$  is at least  $0.99\alpha m$  for all components  $p$ . We will condition on this event.
Furthermore, noting that in the corruption model defined in Definition 1.3, at most  $\epsilon \alpha m$  points can be corrupted total, this implies that  $\ell_p \leq \epsilon \alpha m \leq (\epsilon / 0.99) n_p$  (since  $n_p \geq 0.99 \alpha m$ ).
Therefore, we apply Theorem 1.2 to all the  $k$  Gaussian components, using parameter  $0.99\alpha$  in place of  $\alpha$ ,  $\epsilon / 0.99$  in place of  $\epsilon$ , and failure probability  $\delta / (4k)$ . This implies that, with a union bound, using  $m = \mathrm{poly}(d, 1/\alpha, \log k/\delta) = \mathrm{poly}(d, 1/\alpha, \log 1/\delta)$  samples (since  $k \lesssim 1/\alpha$ , for otherwise the Gaussian mixture will have weight greater than 1), with probability at least  $1 - \delta/4$ , the two bullet points in the theorem hold for all the Gaussian components, directly guaranteed by Theorem 1.2.
It remains to show that the algorithm will return at most  $k$  subsets in its output list. Noting that, across the subsets output by the algorithm for each of the components, there is already a total of  $(1 - 0.01\alpha)\sum_{p = 1}^{k}(n_p - \ell_p)$  points being output. Recall that  $\sum_{p}n_{p} = m$  and  $\sum_{p}\ell_{p}\leq \epsilon \alpha m$ . Thus, in
the output subsets corresponding to the components, there are a total of at least  $m - (\epsilon + 0.01)\alpha m \geq m - 0.4\alpha m$  points (where we used that  $\epsilon$  is bounded by sufficiently small constant. Since (i) there were only  $m$  points in the input and (ii) all the output subsets are disjoint subsets of the input, this means that there are at most  $0.4\alpha m$  points that can be output in any additional subsets not corresponding to any Gaussian component. However, by Theorem 1.2, any output subset must have size at least  $0.5(0.99\alpha)m$ , and so there cannot be any extra subset output in the list that does not correspond to any Gaussian component. This shows that there can be at most  $k$  subsets in the output list.
We turn to the final bullet of the theorem. We will show that any two components  $p$  and  $p'$  that belong to the same set  $T_{i}$  in the output of the algorithm satisfy that  $\| \Sigma^{-1/2}\Sigma_{p}\Sigma^{-1/2} - \Sigma^{-1/2}\Sigma_{p'}\Sigma^{-1/2}\|_{\mathrm{F}} \lesssim \frac{1}{\alpha^{5}}\log(1/\alpha)$ . Taking the contrapositive yields the final theorem bullet.
Let  $H$  be the second moment matrix of this set, that is,  $H = \mathbf{E}_{X\sim T_i}[XX^\top]$ . Then  $H$  satisfies that  $\| I - H^{\dagger /2}\Sigma_pH^{\dagger /2}\|_{\mathrm{F}}\leq r$  and  $\| I - H^{\dagger /2}\Sigma_{p'}H^{\dagger /2}\|_{\mathrm{F}}\leq r$  for  $r\lesssim (1 / \alpha^4)\log (1 / \alpha)$ , by the second bullet point of the theorem. Furthermore,  $H$  is full rank because  $H$  is the second moment matrix of a large subset of empirical points of component  $p$  and thus Condition (L.5) implies that  $\ker (H)$  is contained in  $\ker (\Sigma_p)$ , which is empty since  $\Sigma_p$  is full rank by the theorem assumption.
We will now apply the following result (Lemma C.1) to infer that  $\Sigma_{p}$  and  $\Sigma_{p'}$  must be close, in the sense of the final theorem bullet point. We defer the proof of this lemma to Appendix C.
We will now apply Lemma C.1 to  $\Sigma_1 = \Sigma_p$  and  $\Sigma_2 = \Sigma_{p'}$ , and upper bound the right hand side of the lemma to give the final theorem bullet point.
We claim that the operator norm of  $\Sigma^{-1/2}\Sigma_p\Sigma^{-1/2}$  is bounded by  $1/\alpha$ , due to the fact that component  $p$  has mass at least  $\alpha$ . Indeed, we have that
$$
\Sigma_ {p} \preceq \mathbf {E} _ {X \sim \mathcal {N} (\mu_ {p}, \Sigma_ {p})} [ (X - \mu) (X - \mu) ^ {\top} ] \preceq \frac {1}{\alpha_ {p}} \cdot \mathbf {E} _ {X \sim \sum_ {i} \alpha_ {i} \mathcal {N} (\mu_ {i}, \Sigma_ {i})} [ (X - \mu) (X - \mu) ^ {\top} ] = \frac {1}{\alpha_ {p}} \cdot \Sigma .
$$
This in particular implies that  $\Sigma^{-1/2}\Sigma_p\Sigma^{-1/2} \preceq \frac{1}{\alpha_p}I \preceq \frac{1}{\alpha}I$ , and hence  $\|\Sigma^{-1/2}\Sigma_p\Sigma^{-1/2}\|_{\mathrm{op}} \leq \frac{1}{\alpha}$ . The same conclusion holds for the component  $p'$ . Recalling that  $r \lesssim (1/\alpha^4)\log(1/\alpha)$ , the right hand side of Lemma C.1 can be upper bounded by  $O((1/\alpha^5)\log(1/\alpha))$ , thus yielding the final bullet in the theorem, as discussed above.
We end this section with a brief discussion on the proof of Lemma C.1. The lemma states that, if there exists a single matrix  $H$  that is an approximate square root to both  $\Sigma_{1}$  and  $\Sigma_{2}$ , in the sense that  $\|I - H^{\dagger/2}\Sigma_{1}H^{\dagger/2}\|_{\mathrm{F}}$  and  $\|I - H^{\dagger/2}\Sigma_{1}H^{\dagger/2}\|_{\mathrm{F}}$  are both upper bounded, then  $\Sigma_{1}$  and  $\Sigma_{2}$  cannot be too far from each other, in that  $\Sigma_{1} - \Sigma_{2}$  must also have bounded Frobenius norm. In fact, this can be generalized to the normalized version  $\Sigma^{-1/2}(\Sigma_{1} - \Sigma_{2})\Sigma^{-1/2}$  for any positive definite matrix  $\Sigma$ .
For simplicity of the discussion, let us assume for now that  $\Sigma = I$ . To show Lemma C.1, a natural first idea is to show that, if  $\|I - H^{\dagger/2}\Sigma_1H^{\dagger/2}\|_{\mathrm{F}}$  is bounded, say by some quantity  $\rho$ , then so must be  $\|HH^{\top} - \Sigma_1\|_{\mathrm{F}}$  by some quantity related to  $O(\rho)$ , times the operator norm of  $\Sigma_1$  for appropriate scaling. Unfortunately, that is not true: Even for  $\Sigma_1 = I$ , we can have  $H$  being infinity in the first diagonal element and 1s in the other diagonal elements, and  $\|I - H^{\dagger/2}\Sigma_1H^{\dagger/2}\|_{\mathrm{F}}$  would only be 1 yet  $HH^{\top} - \Sigma_1$  is infinite in the first dimension. The crucial observation, then, is that a slightly weaker statement is true: On an orthonormal basis that depends only on  $H$  and not on  $\Sigma_1$ ,  $HH^{\top}$  approximates  $\Sigma_1$  in most dimensions, namely  $d - O(\rho^2)$  many dimensions. This statement is formally captured by Lemma C.2. Thus, in these dimensions, we can bound the difference between  $\Sigma_1$  and  $\Sigma_2$  (in Frobenius norm). In the rest of the dimensions where  $HH^{\top}$  does not approximate either  $\Sigma_1$  or  $\Sigma_2$  well, we can bound these dimensions' contribution to the Frobenius norm of  $\Sigma_1 - \Sigma_2$  simply by  $\max(\|\Sigma_1\|_{\mathrm{op}}, \|\Sigma_2\|_{\mathrm{op}})$  times the number of such dimensions, which is small. Combining these bounds yields Lemma C.1 as desired.