SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\mathbb {P} \left(\boldsymbol {a} ^ {\top} \boldsymbol {\Sigma} ^ {\star} \boldsymbol {a} \leq c _ {3} \left(\operatorname {t r} \left(\boldsymbol {\Sigma} ^ {\star}\right) + \| \boldsymbol {\Sigma} ^ {\star} \| _ {F} \sqrt {t} + \| \boldsymbol {\Sigma} ^ {\star} \| _ {\mathrm {o p}} t\right)\right) \geq 1 - 2 \exp \left(- c _ {4} t\right).
$$
We set  $t = -\frac{1}{c_4}\log \left(\frac{c_2}{4}\right)$  so that  $2\exp \left(-c_{4}t\right) = \frac{c_{2}}{2}$ . Since  $\pmb{\Sigma}^{\star}$  is symmetric positive semidefinite, we have
$$
\| \boldsymbol {\Sigma} ^ {\star} \| _ {F} \leq \operatorname {t r} \left(\boldsymbol {\Sigma} ^ {\star}\right)
$$
$$
\text {a n d} \left\| \boldsymbol {\Sigma} ^ {\star} \right\| _ {\mathrm {o p}} \leq \operatorname {t r} \left(\boldsymbol {\Sigma} ^ {\star}\right)
$$
As a result, we have that there exists some constant  $c_5 > 0$  such that
$$
\mathbb {P} \left(\boldsymbol {a} ^ {\top} \boldsymbol {\Sigma} ^ {\star} \boldsymbol {a} \leq c _ {5} \operatorname {t r} \left(\boldsymbol {\Sigma} ^ {\star}\right)\right) \geq 1 - \frac {c _ {2}}{2}. \tag {51}
$$
Substituting the two bounds back to (47). By a union bound of (50) and (51), we have
$$
\begin{array}{l} \mathbb {P} \left(\left(\frac {\mu_ {y}}{\boldsymbol {a} ^ {\top} \boldsymbol {\Sigma} ^ {\star} \boldsymbol {a}} \wedge \tau^ {\prime}\right) \cdot | \langle \boldsymbol {a} \boldsymbol {a} ^ {\top}, \boldsymbol {U} \rangle | \geq \frac {1}{2} \left(\frac {\mu_ {y}}{c _ {5} \operatorname {t r} (\boldsymbol {\Sigma} ^ {\star})} \wedge \tau^ {\prime}\right)\right) \\ \geq \mathbb {P} \left(\frac {\mu_ {y}}{\boldsymbol {a} ^ {\top} \boldsymbol {\Sigma} ^ {\star} \boldsymbol {a}} \wedge \tau^ {\prime} \geq \frac {\mu_ {y}}{c _ {5} \operatorname {t r} (\boldsymbol {\Sigma} ^ {\star})} \wedge \tau^ {\prime}\right) + \mathbb {P} \left(\left| \langle \boldsymbol {a} \boldsymbol {a} ^ {\top}, \boldsymbol {U} \rangle \right| \geq \frac {1}{2}\right) - 1 \\ \geq \mathbb {P} \left(\frac {\mu_ {y}}{\boldsymbol {a} ^ {\top} \boldsymbol {\Sigma} ^ {\star} \boldsymbol {a}} \geq \frac {\mu_ {y}}{c _ {5} \operatorname {t r} \left(\boldsymbol {\Sigma} ^ {\star}\right)}\right) + \mathbb {P} \left(\left| \langle \boldsymbol {a} \boldsymbol {a} ^ {\top}, \boldsymbol {U} \rangle \right| \geq \frac {1}{2}\right) - 1 \geq \frac {c _ {2}}{2} \tag {52} \\ \end{array}
$$
Combining (52) and (47), and redefining constant  $c_{2}$  appropriately, we have
$$
\mathbb {P} \left(\left| \langle \widetilde {\boldsymbol {A}} ^ {\tau^ {\prime}}, \boldsymbol {U} \rangle \right| \geq \frac {1}{2} \left(\frac {\mu_ {y}}{\operatorname {t r} (\boldsymbol {\Sigma} ^ {\star})} \wedge \tau^ {\prime}\right)\right) \geq c _ {2},
$$
E.2.2 Proof of Lemma 13
We begin by noting that for any matrix  $\mathbf{U} \in E$ ,
$$
\begin{array}{l} \mathbb {E} \left[ \sup  _ {\boldsymbol {U} \in E} \frac {1}{n} \sum_ {i = 1} ^ {n} \varepsilon_ {i} \langle \widetilde {\boldsymbol {A}} _ {i} ^ {\tau^ {\prime}}, \boldsymbol {U} \rangle \right] \stackrel {(i)} {\leq} \mathbb {E} \left[ \right. \sup  _ {\boldsymbol {U} \in E} \left.\left\| \frac {1}{n} \sum_ {i = 1} ^ {n} \varepsilon_ {i} \widetilde {\boldsymbol {A}} _ {i} ^ {\tau^ {\prime}} \right\| _ {\mathrm {o p}} \cdot \| \boldsymbol {U} \| _ {*} \right] \\ \stackrel {(\mathrm {i i})} {\leq} 4 \sqrt {2 r} \cdot \mathbb {E} \left. \left\| \frac {1}{n} \sum_ {i = 1} ^ {n} \varepsilon_ {i} \widetilde {\boldsymbol {A}} _ {i} ^ {\tau^ {\prime}} \right\| _ {\text {o p}}, \right. \tag {53} \\ \end{array}
$$
where step (i) follows from Hölder's inequality, and step (ii) follows from the fact that  $\| U \|_* \leq 4\sqrt{2r}$  from the definition of the set  $E$ . It remains to bound the expectation of the operator norm in (53). We follow the standard covering arguments in [52, Section 5.4.1], [50, Section 8.6], [32, Section 4.1], with a slight modification to accommodate the bounded term  $\left(\frac{y + \bar{\eta}_i}{a_i^\top \Sigma^* a_i} \wedge \tau'\right)$  that appears in each of the matrices  $\widetilde{A}_i^{\tau'}$ . As a result, there exist universal constants  $c_1$  and  $c_2$  such that if  $n$  satisfies  $n \geq c_1 d$ , then we have
$$
\mathbb {E} \left[ \left\| \frac {1}{n} \sum_ {i = 1} ^ {n} \varepsilon_ {i} \widetilde {\boldsymbol {A}} _ {i} ^ {\tau^ {\prime}} \right\| _ {\mathrm {o p}} \right] \leq c _ {2} \tau^ {\prime} \sqrt {\frac {d}{n}}.
$$
We conclude by re-defining  $c_2$  appropriately.
F Proof of Corollary 1
We proceed by considering two cases. For each case, the proof consists of two steps. We first verify that the choices of the averaging parameter  $m$  and truncation threshold  $\tau$ ,
$$
m = \left\lceil \left[\left(\frac {\nu_ {\eta} ^ {2}}{(y ^ {\uparrow}) ^ {2}}\right) ^ {2} \frac {N}{d} \right] ^ {1 / 3} \right\rceil \quad \text {a n d} \quad \tau = \frac {y ^ {\uparrow}}{\sigma_ {r} r} \sqrt {\frac {N}{m d}}, \tag {54}
$$
satisfy the assumptions of Theorem 1, namely  $n \geq C_2 r d$  and  $\tau \geq \frac{\mu_y}{\operatorname{tr}(\Sigma^\star)}$ . We then invoke Theorem 1.
F.1 Case 1: high-noise regime
In this case, we have  $\frac{\nu_n^2}{(y^\dagger)^2} >\sqrt{\frac{d}{N}}$ , which means by setting  $m$  according to Equation (54), we have  $m\geq 2$ . As a result, the bound
$$
\left[ \left[ \left(\frac {\nu_ {\eta} ^ {2}}{(y ^ {\uparrow}) ^ {2}}\right) ^ {2} \frac {N}{d} \right] ^ {1 / 3} \right] \leq 2 \left[ \left(\frac {\nu_ {\eta} ^ {2}}{(y ^ {\uparrow}) ^ {2}}\right) ^ {2} \frac {N}{d} \right] ^ {1 / 3} \tag {55}
$$
holds in the high-noise regime.
Verifying the condition on  $n$ . Recall that  $n = \frac{N}{m}$ . We have
$$
\begin{array}{l} n = \frac {N}{m} \stackrel {\text {(i)}} {\geq} \frac {N}{2} \left(\left(\frac {\nu_ {\eta} ^ {2}}{(y ^ {\uparrow}) ^ {2}}\right) ^ {2} \frac {N}{d}\right) ^ {- 1 / 3} \\ = \frac {1}{2} \left(N ^ {2} d \left(\frac {(y ^ {\uparrow}) ^ {2}}{\nu_ {\eta} ^ {2}}\right) ^ {2}\right) ^ {1 / 3} \\ \geq \binom {\mathrm {i i})} {2} \left(C _ {2} ^ {3} \left(\frac {(y ^ {\uparrow}) ^ {2}}{\nu_ {\eta} ^ {2}}\right) ^ {2} \left(\frac {\nu_ {\eta} ^ {2}}{(y ^ {\uparrow}) ^ {2}}\right) ^ {2} r ^ {3} d ^ {3}\right) ^ {1 / 3} \\ = C _ {2} r d, \\ \end{array}
$$
where step (i) is true by plugging in the choice of  $m$  from (54) and applying the bound (55), and step (ii) is true by substituting in the assumption  $N \geq 2C_2^{3/2}\left(\frac{\nu_n^2}{(y^\dagger)^2}\right)^2r^{3/2}d$ . Thus the condition  $n \geq C_2rd$  of Theorem 1 is satisfied.
Verifying the condition on  $\tau$ . For the term  $\sqrt{\frac{N}{dm}}$  in the expression of  $\tau$  in (54), note that, by the previous point,  $\frac{N}{m} = n \gtrsim rd$  (with a constant that is greater than 1). Thus  $\sqrt{\frac{N}{dm}} \geq \sqrt{r} > 1$ . Therefore, to verify the condition  $\tau \geq \frac{\mu_y}{\operatorname{tr}(\Sigma^{\star})}$ , it suffices to verify that
$$
\frac {y ^ {\uparrow}}{\sigma_ {r} r} \geq \frac {\mu_ {y}}{\operatorname {t r} \left(\boldsymbol {\Sigma} ^ {\star}\right)}. \tag {56}
$$
By definition, we have  $y^\uparrow \geq \mu_y$ . Furthermore, since  $\Sigma^{\star}$  is symmetric positive semidefinite, its eigenvalues are all non-negative and are identical to its singular values, and hence  $\operatorname{tr}(\Sigma^{\star}) \geq \sigma_r r$ , verifying the condition (56).
**Invoking Theorem 1.** By setting  $\lambda_{n}$  to its lower bound in (9) and substituting in  $n = \frac{N}{m}$  and our choice of  $\tau$  from (54), we have
$$
\lambda_ {n} = C _ {1} \left(3 \frac {\left(y ^ {\uparrow}\right) ^ {2}}{\sigma_ {r} r} \sqrt {\frac {m d}{N}} + \frac {\nu_ {\eta} ^ {2}}{m}\right) \tag {57}
$$
Substituting this expression of  $\lambda_{n}$  to the error bound (10), then substituting in our choice of  $m$  from (54) to (57) and defining  $C' = 3C \cdot C_1$ , we have
$$
\| \widehat {\boldsymbol {\Sigma}} - \boldsymbol {\Sigma} ^ {\star} \| _ {F} \leq C ^ {\prime} \left(\frac {\operatorname {t r} \left(\boldsymbol {\Sigma} ^ {\star}\right) ^ {2}}{\sigma_ {r} r}\right) \frac {(y ^ {\uparrow}) ^ {4 / 3} (\nu_ {\eta} ^ {2}) ^ {1 / 3}}{\mu_ {y} ^ {2}} \sqrt {r} \left(\frac {d}{N}\right) ^ {1 / 3}.
$$
Using the fact that  $\operatorname{tr}(\pmb{\Sigma}^{\star}) \leq \sigma_1 r$ , we have
$$
\| \widehat {\pmb {\Sigma}} - \pmb {\Sigma} ^ {\star} \| _ {F} \leq C ^ {\prime} \frac {\sigma_ {1} ^ {2}}{\sigma_ {r}} \frac {(y ^ {\uparrow}) ^ {4 / 3} (\nu_ {\eta} ^ {2}) ^ {1 / 3}}{\mu_ {y} ^ {2}} r ^ {3 / 2} \left(\frac {d}{N}\right) ^ {1 / 3}
$$
F.2 Case 2: low-noise regime
In this case, we have  $\frac{\nu_{\eta}^{2}}{(y^{\uparrow})^{2}} \leq \sqrt{\frac{d}{N}}$ , which means by setting  $m$  according to Equation (54), we have  $m = 1$ . As a result, no averaging occurs.
Verifying the condition on  $n$ . Because  $m = 1$  in this case, we have that  $n = N$ . By assumption, we have that  $N \geq C_2rd$ , satisfying the condition  $n \geq C_2rd$  in Theorem 1.
Verifying the condition on  $\tau$ . By the same analysis as in Case 1, we have that the condition  $\tau \geq \frac{\mu_y}{tr\pmb{\Sigma}^*}$  in Theorem 1.
**Invoking Theorem 1.** By setting  $\lambda_{n}$  to its lower bound in (9), substituting in our choice of  $\tau$  from (54) and noting  $m = 1$ , we have
$$
\lambda_ {n} = C _ {1} \left(3 \frac {\left(y ^ {\uparrow}\right) ^ {2}}{\sigma_ {r} r} \sqrt {\frac {d}{n}} + \frac {1}{\sigma_ {r} r} \nu_ {\eta} ^ {2}\right). \tag {58}
$$
We define  $C' = 3C \cdot C_1$  and note that  $n = N$  under Case 2. Substituting this expression of  $\lambda_n$  in (58) to the error bound (10), then using the fact that under Case 2, the bound  $\nu_\eta^2 \leq (y^\uparrow)^2 \sqrt{\frac{d}{N}}$  holds, we have
$$
\| \widehat {\pmb {\Sigma}} - \pmb {\Sigma} ^ {\star} \| _ {F} \leq C ^ {\prime} \left(\frac {\mathrm {t r} (\pmb {\Sigma} ^ {\star}) ^ {2}}{\sigma_ {r} r}\right) \left(\frac {y ^ {\uparrow}}{\mu_ {y}}\right) ^ {2} \sqrt {\frac {r d}{N}}.
$$
$$
\| \widehat {\pmb {\Sigma}} - \pmb {\Sigma} ^ {\star} \| _ {F} \leq C ^ {\prime} \frac {\sigma_ {1} ^ {2}}{\sigma_ {r}} \left(\frac {y ^ {\uparrow}}{\mu_ {y}}\right) ^ {2} \sqrt {\frac {r ^ {3} d}{N}},
$$