SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We fix any  $\pmb{v} \in S^{d-1}$ , and bound  $\pmb{v}^{\top} \left( \mathbb{E}\left[\bar{\pmb{A}}\right] - \mathbb{E}\left[\widetilde{\pmb{A}}\right] \right) \pmb{v}$ . Similar to the proof of Lemma 6, we note that  $\pmb{v}^{\top} \pmb{a} \sim \mathcal{N}(0,1)$  and denote the random variable  $G \sim \mathcal{N}(0,1)$ . Substituting in the expression for sensing matrices  $\bar{\pmb{A}}$  and  $\widetilde{\pmb{A}}$ , we have
$$
\begin{array}{l} \left| \boldsymbol {v} ^ {\top} \left(\mathbb {E} [ \bar {\boldsymbol {A}} ] - \mathbb {E} [ \tilde {\boldsymbol {A}} ]\right) \boldsymbol {v} \right| = \left| \boldsymbol {v} ^ {\top} \mathbb {E} [ \bar {\gamma} ^ {2} \boldsymbol {a} \boldsymbol {a} ^ {\top} - \tilde {\gamma} ^ {2} \boldsymbol {a} \boldsymbol {a} ^ {\top} ] \boldsymbol {v} \right| \\ \stackrel {\mathrm {(i)}} {=} \mathbb {E} \left[ \left(\bar {\gamma} ^ {2} - \widetilde {\gamma} ^ {2}\right) G ^ {2} \right] \\ \stackrel {\mathrm {(i i)}} {=} \mathbb {E} \left[ \left(\bar {\gamma} ^ {2} - \widetilde {\gamma} ^ {2}\right) G ^ {2} \cdot \mathbb {1} \{\bar {\gamma} ^ {2} \geq \tau \} \right] \\ \leq \mathbb {E} \left[ \bar {\gamma} ^ {2} G ^ {2} \cdot \mathbb {1} \{\bar {\gamma} ^ {2} \geq \tau \} \right] \\ \stackrel {(i i i)} {\leq} \left(\mathbb {E} \left[ (\bar {\gamma} ^ {2} G ^ {2}) ^ {2} \right] \cdot \mathbb {E} \left[ \mathbb {1} \{\bar {\gamma} ^ {2} \geq \tau \} \right]\right) ^ {1 / 2} \\ \stackrel {\text {(i v)}} {\leq} \left(\mathbb {E} \left[ | \bar {\gamma} ^ {2} | ^ {4} \right] \cdot \mathbb {E} \left[ | G ^ {2} | ^ {4} \right]\right) ^ {1 / 4} \left(\mathbb {P} \left(\bar {\gamma} ^ {2} \geq \tau\right)\right) ^ {1 / 2}, \tag {36} \\ \end{array}
$$
where where step (i) is true because  $\bar{\gamma}^2 \geq \widetilde{\gamma}^2$  from to (TP2), step (ii) is true due to (TP3), and steps (iii) and (iv) follow from Cauchy-Schwarz inequality. We proceed by bounding each of the terms in (36) separately. First, we can upper bound the fourth moment  $\mathbb{E}\left[|\bar{\gamma}^2|^4\right]$  by the quantity  $M$  from Equation (29). Second,  $G^2$  is a sub-exponential random variable. By Lemma 2 in Appendix D.2, we have that  $\mathbb{E}\left[|G^2|^4\right]^{1/4} \leq c$  for some constant  $c$ . It remains to bound the term  $\left(\mathbb{P}\left(\bar{\gamma}^2 \geq \tau\right)\right)^{1/2}$ . We have
$$
\begin{array}{l} \mathbb {P} \left(\bar {\gamma} ^ {2} \geq \tau\right) \stackrel {(i)} {\leq} \frac {\mathbb {E} \left| \bar {\gamma} ^ {2} \right| ^ {2}}{\tau^ {2}} \\ \stackrel {\mathrm {(i i)}} {\leq} \frac {\left(\mathbb {E} | \bar {\gamma} ^ {2} | ^ {4}\right) ^ {1 / 2}}{\tau^ {2}} \\ \stackrel {\mathrm {(i i i)}} {\leq} \frac {M ^ {1 / 2}}{\tau^ {2}}, \\ \end{array}
$$
where step (i) follows from Markov's inequality, step (ii) follows from Cauchy-Schwarz inequality, and step (iii) follows from the fourth moment bound on the averaged scaling  $\bar{\gamma}^2$ . Putting everything together back to (36), we have
$$
\left| \boldsymbol {v} ^ {\top} \left(\mathbb {E} [ \bar {\boldsymbol {A}} ] - \mathbb {E} [ \tilde {\boldsymbol {A}} ]\right) \boldsymbol {v} \right| \lesssim \frac {M ^ {1 / 2}}{\tau}
$$
for any vector  $\pmb{v} \in S^{d-1}$ . Therefore,
$$
\left\| \mathbb {E} \left[ \tilde {\boldsymbol {A}} \right] - \mathbb {E} \left[ \bar {\boldsymbol {A}} \right] \right\| _ {\mathrm {o p}} \lesssim \frac {M ^ {1 / 2}}{\tau},
$$
E.1.3 Proof of Lemma 8
Substituting in the definitions  $\bar{A} = \bar{\gamma}^2 aa^\top$  and  $\tilde{A} = \tilde{\gamma}^2 aa^\top$ , we have
$$
\langle \bar {A}, \boldsymbol {\Sigma} ^ {\star} \rangle \bar {A} - \langle \widetilde {A}, \boldsymbol {\Sigma} ^ {\star} \rangle \widetilde {A} = (\bar {\gamma} ^ {4} - \widetilde {\gamma} ^ {4}) (\boldsymbol {a} ^ {\top} \boldsymbol {\Sigma} ^ {\star} \boldsymbol {a}) \boldsymbol {a} \boldsymbol {a} ^ {\top}.
$$
Therefore, our goal is to bound the operator norm
$$
\left\| \left(\bar {\gamma} ^ {4} - \widetilde {\gamma} ^ {4}\right) \left(\boldsymbol {a} ^ {\top} \boldsymbol {\Sigma} ^ {\star} \boldsymbol {a}\right) \boldsymbol {a} \boldsymbol {a} ^ {\top} \right\| _ {\mathrm {o p}} = \sup  _ {\boldsymbol {v} \in \mathcal {S} ^ {d - 1}} \left| \boldsymbol {v} ^ {T} \left(\bar {\gamma} ^ {4} - \widetilde {\gamma} ^ {4}\right) \left(\boldsymbol {a} ^ {\top} \boldsymbol {\Sigma} ^ {\star} \boldsymbol {a}\right) \boldsymbol {a} \boldsymbol {a} ^ {\top} \boldsymbol {v} \right|.
$$
Similar to the proof of Lemma 7, we fix any vector  $\pmb{v} \in S^{d-1}$ . Again, note that  $\pmb{v}^{\top} \pmb{a} \sim \mathcal{N}(0,1)$  and denote  $G \sim \mathcal{N}(0,1)$ . We have
$$
\begin{array}{l} \left| \boldsymbol {v} ^ {\top} \mathbb {E} \left[ \left(\bar {\gamma} ^ {4} - \widetilde {\gamma} ^ {4}\right) \left(\boldsymbol {a} ^ {\top} \boldsymbol {\Sigma} ^ {\star} \boldsymbol {a}\right) \boldsymbol {a} \boldsymbol {a} ^ {\top} \right] \boldsymbol {v} \right| \stackrel {{\text {(i)}}} {=} \mathbb {E} \left[ \left(\bar {\gamma} ^ {4} - \widetilde {\gamma} ^ {4}\right) \left(\boldsymbol {a} ^ {\top} \boldsymbol {\Sigma} ^ {\star} \boldsymbol {a}\right) G ^ {2} \right] \\ = \mathbb {E} \left[ \left(\bar {\boldsymbol {\gamma}} ^ {2} + \tilde {\boldsymbol {\gamma}} ^ {2}\right) \left(\bar {\boldsymbol {\gamma}} ^ {2} - \tilde {\boldsymbol {\gamma}} ^ {2}\right) \left(\boldsymbol {a} ^ {\top} \boldsymbol {\Sigma} ^ {\star} \boldsymbol {a}\right) G ^ {2} \right] \\ \stackrel {(\mathrm {i i})} {\leq} \mathbb {E} \left[ 2 \bar {\gamma} ^ {2} (\bar {\gamma} ^ {2} - \tilde {\gamma} ^ {2}) (a ^ {\top} \Sigma^ {\star} a) G ^ {2} \right] \\ \stackrel {\text {(i i i)}} {=} 2 \mathbb {E} \left[ (y + \bar {\eta}) (\bar {\gamma} ^ {2} - \widetilde {\gamma} ^ {2}) G ^ {2} \right] \\ \stackrel {\text {(i v)}} {\leq} 2 (y + \eta^ {\uparrow}) \mathbb {E} \left[ \left(\bar {\gamma} ^ {2} - \widetilde {\gamma} ^ {2}\right) G ^ {2} \mathbf {1} \{\gamma^ {2} \geq \tau \} \right] \\ \end{array}
$$
where steps (i) and (ii) are true because  $\bar{\gamma}^2 \geq \widetilde{\gamma}^2$  from (TP2), step (iii) follows from the definition  $\bar{\gamma}^2 = \frac{y + \bar{\eta}}{a^\top \Sigma^\star a}$ , and step (iv) follows from (TP3) and the definition of  $\eta^\uparrow$  as the upper bound on the noise  $\eta$ .
The rest of the proof follows the exact steps of the proof of Lemma 7 in Appendix E.1.2. Therefore, we have the bound
$$
\left\| \mathbb {E} \left[ \left(\bar {\gamma} ^ {4} - \widetilde {\gamma} ^ {4}\right) \left(\pmb {a} ^ {\top} \pmb {\Sigma} ^ {\star} \pmb {a}\right) \pmb {a} \pmb {a} ^ {\top} \right] \right\| _ {\mathrm {o p}} \lesssim \frac {y ^ {\uparrow} M ^ {1 / 2}}{\tau},
$$
E.1.4 Proof of Lemma 9
The proof follows the steps as in the proof of Lemma 6, and we describe the difference of the two proofs. We again apply Bernstein's inequality.
Proving a Bernstein condition. We prove a Bernstein condition with  $u_{1} = c_{1}(y + \eta^{\uparrow})^{2}$  and  $u_{2} = c_{2}(y + \eta^{\uparrow})\tau$ . Namely, for every integer  $p \geq 2$ , we have (cf. (31) in Lemma 6)
$$
\mathbb {E} \left[ \left| \boldsymbol {v} ^ {\top} \langle \widetilde {\boldsymbol {A}}, \boldsymbol {\Sigma} ^ {\star} \rangle \widetilde {\boldsymbol {A}} \boldsymbol {v} \right| ^ {p} \right] \leq \frac {p !}{2} u _ {1} u _ {2} ^ {p - 2}. \tag {37}
$$
To show (37), we plug in  $\widetilde{A} = \widetilde{\gamma}^2\pmb{a}\pmb{a}^\top$  and have
$$
\begin{array}{l} \mathbb {E} \left| \boldsymbol {v} ^ {\top} \langle \widetilde {\boldsymbol {A}}, \boldsymbol {\Sigma} ^ {\star} \rangle \widetilde {\boldsymbol {A}} \boldsymbol {v} \right| ^ {p} = \mathbb {E} \left(\widetilde {\gamma} ^ {2} \boldsymbol {a} ^ {\top} \boldsymbol {\Sigma} ^ {\star} \boldsymbol {a}\right) ^ {p} \cdot \left| \boldsymbol {v} ^ {\top} \widetilde {\boldsymbol {A}} \boldsymbol {v} \right| ^ {p} \\ \stackrel {(i)} {\leq} \mathbb {E} \left(\bar {\gamma} ^ {2} \boldsymbol {a} ^ {\top} \boldsymbol {\Sigma} ^ {\star} \boldsymbol {a}\right) ^ {p} \cdot \left| \boldsymbol {v} ^ {\top} \widetilde {\boldsymbol {A}} \boldsymbol {v} \right| ^ {p} \\ \stackrel {\text {(i i)}} {=} \mathbb {E} (y + \bar {\eta}) ^ {p} \cdot \left| \boldsymbol {v} ^ {\top} \widetilde {\boldsymbol {A}} \boldsymbol {v} \right| ^ {p} \\ \stackrel {(i i i)} {\leq} (y + \eta^ {\uparrow}) ^ {p} \cdot \mathbb {E} \left| \boldsymbol {v} ^ {\top} \widetilde {\boldsymbol {A}} \boldsymbol {v} \right| ^ {p}, \tag {38} \\ \end{array}
$$
where step (i) follows from (TP2), step (ii) follows from the definition  $\bar{\gamma}^2 = \frac{y + \bar{\eta}}{a^\top \Sigma^\star a}$ , and step (iii) follows from the definition of  $\eta^\uparrow$  as the upper bound on the noise  $\eta$ . Substituting in (31) from Lemma 6 to bound the term  $\mathbb{E}\left|\boldsymbol{v}^\top \widetilde{\boldsymbol{A}}\boldsymbol{v}\right|^p$  in (38) completes the proof of the Bernstein condition (37).
Applying Bernstein's inequality. The rest of the proof follows in the same manner as the proof of Lemma 6 in Appendix E.1.1, with an additional factor of  $(y + \eta^{\uparrow})$ . We have
$$
\left\| \mathbb {E} \left[ \langle \widetilde {\boldsymbol {A}}, \boldsymbol {\Sigma} ^ {\star} \rangle \widetilde {\boldsymbol {A}} \right] - \frac {1}{n} \sum_ {i = 1} ^ {n} \langle \widetilde {\boldsymbol {A}} _ {i}, \boldsymbol {\Sigma} ^ {\star} \rangle \widetilde {\boldsymbol {A}} _ {i} \right\| _ {\mathrm {o p}} \lesssim y ^ {\uparrow} \left(\sqrt {\frac {M ^ {1 / 2} t}{n}} + \frac {\tau t}{n}\right)
$$
with probability at least  $1 - 2 \cdot 9^d \cdot \exp(-t)$ , as desired.
E.1.5 Proof of Lemma 10
Recall that by definition  $\bar{A} = \bar{\gamma}^2\pmb {a}\pmb{a}^\top = \frac{y + \bar{\eta}}{\pmb{a}^\top\pmb{\Sigma}^\star\pmb{a}}\pmb {a}\pmb{a}^\top$  . We have
$$
\begin{array}{l} \left\| \mathbb {E} \left[ \bar {\eta} \bar {\boldsymbol {A}} \right] \right\| _ {\mathrm {o p}} = \left\| \mathbb {E} \left[ \bar {\eta} (y + \bar {\eta}) \frac {\boldsymbol {a} \boldsymbol {a} ^ {\top}}{\boldsymbol {a} ^ {\top} \boldsymbol {\Sigma} ^ {\star} \boldsymbol {a}} \right] \right\| _ {\mathrm {o p}} \\ = \left\| \mathbb {E} [ \bar {\eta} (y + \bar {\eta}) ] \cdot \mathbb {E} \left[ \frac {\boldsymbol {a} \boldsymbol {a} ^ {\top}}{\boldsymbol {a} ^ {\top} \boldsymbol {\Sigma} ^ {\star} \boldsymbol {a}} \right] \right\| _ {\mathrm {o p}} \\ = \frac {\sigma_ {\eta} ^ {2}}{m} \cdot \left\| \mathbb {E} \left[ \frac {\boldsymbol {a} \boldsymbol {a} ^ {\top}}{\boldsymbol {a} ^ {\top} \boldsymbol {\Sigma} ^ {\star} \boldsymbol {a}} \right] \right\| _ {\text {o p}}. \tag {39} \\ \end{array}
$$
To bound the operator norm term in (39), we apply Lemma 4(b) in Appendix D.4. For any matrix  $\mathbf{U}$ , we have
$$
\mathbb {E} \left[ \frac {\boldsymbol {a} ^ {\top} \boldsymbol {U} \boldsymbol {a}}{\boldsymbol {a} ^ {\top} \boldsymbol {\Sigma} ^ {\star} \boldsymbol {a}} \right] \lesssim \frac {1}{\sigma_ {r} r} \| \boldsymbol {U} \| _ {*}. \tag {40}
$$
Note that  $\frac{aa^{\top}}{a^{\top}\Sigma^{\star}a}$  is symmetric positive semidefinite, so we have
$$
\begin{array}{l} \left\| \mathbb {E} \left[ \frac {\boldsymbol {a} \boldsymbol {a} ^ {\top}}{\boldsymbol {a} ^ {\top} \boldsymbol {\Sigma} ^ {\star} \boldsymbol {a}} \right] \right\| _ {\text {o p}} = \sup  _ {\boldsymbol {v} \in S ^ {d - 1}} \left| \boldsymbol {v} ^ {\top} \mathbb {E} \left[ \frac {\boldsymbol {a} \boldsymbol {a} ^ {\top}}{\boldsymbol {a} ^ {\top} \boldsymbol {\Sigma} ^ {\star} \boldsymbol {a}} \right] \boldsymbol {v} \right| \\ = \sup  _ {\boldsymbol {v} \in \mathcal {S} ^ {d - 1}} \mathbb {E} \left[ \frac {\boldsymbol {a} ^ {\top} (\boldsymbol {v} \boldsymbol {v} ^ {\top}) \boldsymbol {a}}{\boldsymbol {a} ^ {\top} \boldsymbol {\Sigma} ^ {\star} \boldsymbol {a}} \right] \\ \stackrel {\text {(i)}} {\lesssim} \frac {1}{\sigma_ {r} r} \sup  _ {\boldsymbol {v} \in \mathcal {S} ^ {d - 1}} \| \boldsymbol {v} \boldsymbol {v} ^ {\top} \| _ {*} \\ \stackrel {\text {(i i)}} {=} \frac {1}{\sigma_ {r} r}, \tag {41} \\ \end{array}
$$
where step (i) is true by substituting in (40) with  $\pmb{U} = \pmb{v}\pmb{v}^T$ , and step (ii) is true because  $\pmb{v}$  is unit norm, and hence  $\| \pmb{v}\pmb{v}^\top \|_* = 1$ . Substituting (41) back to (39), we have
$$
\left\| \mathbb {E} \left[ \bar {\eta} \bar {\boldsymbol {A}} \right] \right\| _ {\mathrm {o p}} \lesssim \frac {1}{\sigma_ {r} r} \cdot \frac {\nu_ {\eta} ^ {2}}{m},
$$
E.2 Proof of Proposition 3
We analyze the term  $\frac{1}{n}\sum_{i = 1}^{n}\langle \widetilde{\boldsymbol{A}}_i,\boldsymbol {U}\rangle^2$  from (26). Recall from the definition of  $\widetilde{\boldsymbol{A}}$  that for any  $i = 1,\ldots ,n$
$$
\widetilde {\boldsymbol {A}} _ {i} = \widetilde {\gamma} _ {i} ^ {2} \boldsymbol {a} _ {i} \boldsymbol {a} _ {i} ^ {\top} = \left(\frac {y + \bar {\eta} _ {i}}{\boldsymbol {a} _ {i} ^ {\top} \boldsymbol {\Sigma} ^ {\star} \boldsymbol {a} _ {i}} \wedge \tau\right) \boldsymbol {a} _ {i} \boldsymbol {a} _ {i} ^ {\top},
$$
$$
\langle \widetilde {\boldsymbol {A}} _ {i}, \boldsymbol {U} \rangle^ {2} = \left(\frac {y + \bar {\eta} _ {i}}{\boldsymbol {a} _ {i} ^ {\top} \boldsymbol {\Sigma} ^ {\star} \boldsymbol {a} _ {i}} \wedge \tau\right) ^ {2} \left(\boldsymbol {a} _ {i} ^ {\top} \boldsymbol {U} \boldsymbol {a} _ {i}\right) ^ {2}. \tag {42}
$$
From (42), we have that for any matrix  $\mathbf{U}$ , the term  $\sum_{i=1}^{n} \langle \widetilde{\mathbf{A}}_i, \mathbf{U} \rangle^2$  is nondecreasing in  $\tau$  when  $\tau > 0$ . Defining a random matrix
$$
\widetilde {\boldsymbol {A}} ^ {\tau^ {\prime}} := \left(\frac {y + \bar {\eta}}{\boldsymbol {a} ^ {\top} \boldsymbol {\Sigma} ^ {\star} \boldsymbol {a}} \wedge \tau^ {\prime}\right) \boldsymbol {a} \boldsymbol {a} ^ {\top}, \tag {43}
$$
for any  $\tau' \in (0, \tau]$ , we have
$$
\frac {1}{n} \sum_ {i = 1} ^ {n} \langle \widetilde {\boldsymbol {A}} _ {i}, \boldsymbol {U} \rangle^ {2} \geq \frac {1}{n} \sum_ {i = 1} ^ {n} \langle \widetilde {\boldsymbol {A}} _ {i} ^ {\tau^ {\prime}}, \boldsymbol {U} \rangle^ {2}, \tag {44}
$$
where for every  $i = 1, \dots, n$ , matrix  $\widetilde{A}_i^{\tau'}$  is formed with the same realizations of random quantities  $\mathbf{a}_i$  and  $\bar{\eta}_i$  as  $\widetilde{A}_i$ . The two matrices only differ in choice of truncation threshold:  $\tau'$  instead of  $\tau$ . As a result, for the rest of the proof, we lower bound  $\frac{1}{n} \sum_{i=1}^{n} \langle \widetilde{A}_i^{\tau'}, \mathbf{U} \rangle^2$  for an appropriate choice of  $\tau'$  to be specified later. To proceed, we use a small-ball argument [49, 50] based on the following lemma.
Lemma 11 ([50, Proposition 5.1], adapted to our notation). Let  $X_{1},\ldots ,X_{n}\in \mathbb{R}^{d\times d}$  be i.i.d. copies of a random matrix  $\pmb {X}\in \mathbb{R}^{d\times d}$ . Let  $E\subset \mathbb{R}^{d\times d}$  be a subset of matrices. Let  $\xi >0$  and  $Q > 0$  be real values such that for every matrix  $\pmb {U}\in E$ , the marginal tail condition holds:
$$
\mathbb {P} \left(| \langle \boldsymbol {X}, \boldsymbol {U} \rangle | \geq 2 \xi\right) \geq Q. \tag {45}
$$
Define the Rademacher width as
$$
W := \mathbb {E} \left[ \sup  _ {\boldsymbol {U} \in E} \frac {1}{n} \sum_ {i = 1} ^ {n} \varepsilon_ {i} \langle \boldsymbol {X} _ {i}, \boldsymbol {U} \rangle \right],
$$
where  $\varepsilon_1, \ldots, \varepsilon_n$  are i.i.d. Rademacher random variables independent of  $\{\mathbf{X}_i\}_{i \in [n]}$ . Then for any  $t > 0$ , we have
$$
\inf  _ {\boldsymbol {U} \in E} \left(\frac {1}{n} \sum_ {i = 1} ^ {n} \langle \boldsymbol {X} _ {i}, \boldsymbol {U} \rangle^ {2}\right) ^ {1 / 2} \geq \xi (Q - t) - 2 W.
$$
with probability at least 1 -  $\exp \left(-\frac{nt^2}{2}\right)$ .
Recall the error set  $\mathcal{E}$  defined in Equation (22). Because the claim (26) is invariant to scaling, it suffices to prove it for  $\| \boldsymbol{U} \|_F = 1$ . Correspondingly, we define the set  $E$  as
$$
\begin{array}{l} E = \mathcal {E} \cap \left\{\boldsymbol {U} \in \mathbb {R} ^ {d \times d}: \| \boldsymbol {U} \| _ {F} = 1 \right\} \\ = \left\{\boldsymbol {U} \in \mathbb {S} ^ {d \times d}: \| \boldsymbol {U} \| _ {F} = 1, \| \boldsymbol {U} \| _ {*} \leq 4 \sqrt {2 r} \right\}. \tag {46} \\ \end{array}
$$
We invoke Lemma 11 with set  $E$  defined above,  $\pmb{X}_i = \widetilde{\pmb{A}}_i^{\tau'}$ ,  $\xi = \frac{c_1}{2}\left(\frac{\mu_y}{\mathrm{tr}(\pmb{\Sigma}^*)}\wedge \tau'\right)$ , and  $Q = c_{2}$  where  $\mu_y$  is the median of  $\bar{\eta}$  and  $c_{1}$  and  $c_{2}$ , are constants to be specified later. The rest of the proof is comprised of two steps. We first verify that our choices for  $\xi$  and  $Q$  are valid for establishing the marginal tail condition (45). We then bound the Rademacher width  $W$  above. The following lemma verifies our choices for  $\xi$  and  $Q$ .
Lemma 12. Consider any  $\tau' \in (0, \tau]$ . There exist absolute constants  $c_1, c_2 > 0$  such that for every  $U \in E$ , we have
$$
\mathbb {P} \left(\left| \langle \widetilde {\boldsymbol {A}} ^ {\tau^ {\prime}}, \boldsymbol {U} \rangle \right| \geq c _ {1} \left(\frac {\mu_ {y}}{\operatorname {t r} \left(\boldsymbol {\Sigma} ^ {\star}\right)} \wedge \tau^ {\prime}\right)\right) \geq c _ {2}.
$$
The proof of Lemma 12 is presented in Appendix E.2.1. We now turn to the second step of the proof, which is bounding the Rademacher width  $W$ . The next lemma characterizes this width.
Lemma 13. Consider any  $\tau' \in (0, \tau]$ . Let  $\widetilde{A}_1^{\tau'}$ , ...,  $\widetilde{A}_n^{\tau'} \in \mathbb{R}^{d \times d}$  be i.i.d. copies of the random matrix  $\widetilde{A}^{\tau'} \in \mathbb{R}^{d \times d}$  defined in Equation (43). Let  $E$  be the set defined in Equation (46). Then, there exists some absolute constants  $c_1$  and  $c_2$  such that if  $n \geq c_1 d$ , then we have
$$
\mathbb {E} \left[ \sup  _ {\boldsymbol {U} \in E} \frac {1}{n} \sum_ {i = 1} ^ {n} \varepsilon_ {i} \langle \widetilde {\boldsymbol {A}} _ {i} ^ {\tau^ {\prime}}, \boldsymbol {U} \rangle \right] \leq c _ {2} \tau^ {\prime} \sqrt {\frac {r d}{n}}.
$$
The proof of Lemma 13 is presented in Appendix E.2.2. Lemma 12 establishes the marginal tail condition for Lemma 11, and Lemma 13 upper bounds the Rademacher width. We now invoke Lemma 11 and substitute in the upper bound for the Rademacher width  $W$ . For some constant  $c_4$ , if  $n \geq c_4 d$ , we have that with probability at least  $1 - \exp \left(-\frac{nt^2}{2}\right)$ ,
$$
\begin{array}{l} \inf  _ {\boldsymbol {U} \in E} \left(\frac {1}{n} \sum_ {i = 1} ^ {n} \langle \widetilde {\boldsymbol {A}} _ {i}, \boldsymbol {U} \rangle^ {2}\right) ^ {1 / 2} \geq \inf  _ {\boldsymbol {U} \in E} \left(\frac {1}{n} \sum_ {i = 1} ^ {n} \langle \widetilde {\boldsymbol {A}} _ {i} ^ {\tau^ {\prime}}, \boldsymbol {U} \rangle^ {2}\right) ^ {1 / 2} \\ \geq \frac {c _ {1}}{2} \left(\frac {\mu_ {y}}{\operatorname {t r} (\boldsymbol {\Sigma} ^ {\star})} \wedge \tau^ {\prime}\right) (c _ {2} - t) - c _ {3} \tau^ {\prime} \sqrt {\frac {r d}{n}}, \\ \end{array}
$$
where step (i) is true due to the monotonicity property (44). We set  $\tau' = \frac{\mu_y}{\mathrm{tr}(\pmb{\Sigma}^*)}$ , where recall that  $\mu_y$  is the median of the random quantity  $y + \bar{\eta}$ . By the assumption  $\tau \geq \frac{\mu_y}{\mathrm{tr}(\pmb{\Sigma}^*)}$ , this choice of  $\tau'$  satisfies  $\tau' \leq \tau$ . Setting  $t = \frac{c_2}{2}$ , we have that with probability at least  $1 - \exp \left(-\frac{c_2^2n}{8}\right)$ ,
$$
\inf  _ {\boldsymbol {U} \in E} \frac {1}{n} \left(\sum_ {i = 1} ^ {n} \langle \widetilde {\boldsymbol {A}} _ {i} ^ {\tau^ {\prime}}, \boldsymbol {U} \rangle^ {2}\right) ^ {1 / 2} \geq \frac {c _ {1} c _ {2}}{4} \frac {\mu_ {y}}{\operatorname {t r} (\boldsymbol {\Sigma} ^ {\star})} - c _ {3} \frac {\mu_ {y}}{\operatorname {t r} (\boldsymbol {\Sigma} ^ {\star})} \sqrt {\frac {r d}{n}}.
$$
Recall from the definition of  $E$  (46) that  $\| \boldsymbol{U} \|_F = 1$ . As a result, if  $n \geq \max \left\{\left(\frac{4c_3}{c_1c_2}\right)^2, c_4\right\} rd$ , we have
$$
\inf _ {\boldsymbol {U} \in \mathcal {E}} \frac {1}{n} \sum_ {i = 1} ^ {n} \langle \widetilde {\boldsymbol {A}} _ {i}, \boldsymbol {U} \rangle^ {2} \geq \left(\frac {c _ {1} c _ {2}}{4} \frac {\mu_ {y}}{\operatorname {t r} (\boldsymbol {\Sigma} ^ {\star})}\right) ^ {2} \| \boldsymbol {U} \| _ {F} ^ {2}
$$
with probability at least  $1 - \exp \left(-\frac{c_2^2n}{8}\right)$ . We conclude by setting  $\kappa_{\mathcal{L}} = \left(\frac{c_1c_2}{4}\right)^2$ ,  $c = \frac{c_2^2}{8}$ , and  $C = \max \left\{\left(\frac{4c_3}{c_1c_2}\right)^2, c_4\right\}$  in Proposition 3.
E.2.1 Proof of Lemma 12
We fix any  $U \in E$ . Recall that  $\mu_y$  denotes the median of  $y + \bar{\eta}$ . Let  $\mathcal{G}$  be the event that  $y + \bar{\eta} \geq \mu_y$ , which occurs with probability  $\frac{1}{2}$ . For any  $\xi > 0$ , because the averaged noise  $\bar{\eta}$  and sensing vector  $\mathbf{a}$  are independent, we have
$$
\begin{array}{l} \mathbb {P} \left(\left| \langle \widetilde {\boldsymbol {A}} ^ {\tau^ {\prime}}, \boldsymbol {U} \rangle \right| \geq \xi\right) \stackrel {\text {(i)}} {=} \mathbb {P} \left(\left(\frac {y + \bar {\eta}}{\boldsymbol {a} ^ {\top} \boldsymbol {\Sigma} ^ {\star} \boldsymbol {a}} \wedge \tau^ {\prime}\right) \cdot \left| \langle \boldsymbol {a} \boldsymbol {a} ^ {\top}, \boldsymbol {U} \rangle \right| \geq \xi\right) \\ = \mathbb {P} \left(\left(\frac {y + \bar {\eta}}{\boldsymbol {a} ^ {\top} \boldsymbol {\Sigma} ^ {\star} \boldsymbol {a}} \wedge \tau^ {\prime}\right) \cdot | \langle \boldsymbol {a} \boldsymbol {a} ^ {\top}, \boldsymbol {U} \rangle | \geq \xi \mid \mathcal {G}\right) \cdot \mathbb {P} (\mathcal {G}) \\ = \frac {1}{2} \mathbb {P} \left(\left(\frac {y + \bar {\eta}}{\boldsymbol {a} ^ {\top} \boldsymbol {\Sigma} ^ {\star} \boldsymbol {a}} \wedge \tau^ {\prime}\right) \cdot | \langle \boldsymbol {a} \boldsymbol {a} ^ {\top}, \boldsymbol {U} \rangle | \geq \xi \mid \mathcal {G}\right) \\ \geq \frac {1}{2} \mathbb {P} \left(\left(\frac {\mu_ {y}}{\boldsymbol {a} ^ {\top} \boldsymbol {\Sigma} ^ {\star} \boldsymbol {a}} \wedge \tau^ {\prime}\right) \cdot | \langle \boldsymbol {a} \boldsymbol {a} ^ {\top}, \boldsymbol {U} \rangle | \geq \xi\right), \tag {47} \\ \end{array}
$$
where step (i) is true by plugging in the definition of  $\widetilde{A}^{\tau'}$ , and step (ii) is true by the definition of the event  $\mathcal{G}$ . We proceed by bounding the terms in (47) separately.
Lower bound on  $\left|\langle a a^{\top}, U \rangle \right|$ . We use the approach from [32, Section 4.1]. By Paley-Zygmund inequality,
$$
\mathbb {P} \left(\left| \langle \boldsymbol {a} \boldsymbol {a} ^ {\top}, \boldsymbol {U} \rangle \right| ^ {2} \geq \frac {1}{2} \mathbb {E} \left[ \left| \langle \boldsymbol {a} \boldsymbol {a} ^ {\top}, \boldsymbol {U} \rangle \right| ^ {2} \right]\right) \geq \frac {1}{4} \frac {\left(\mathbb {E} \left[ \left| \langle \boldsymbol {a} \boldsymbol {a} ^ {\top} , \boldsymbol {U} \rangle \right| ^ {2} \right]\right) ^ {2}}{\mathbb {E} \left[ \left| \langle \boldsymbol {a} \boldsymbol {a} ^ {\top} , \boldsymbol {U} \rangle \right| ^ {4} \right]} \tag {48}
$$
We now analyze the terms in (48). As noted in [32, Section 4.1], there exists some constant  $c_{1} > 0$  such that for any matrix  $U$  with  $\| U \|_{F} = 1$ ,
$$
\mathbb {E} \left[ \left| \langle \boldsymbol {a} \boldsymbol {a} ^ {\top}, \boldsymbol {U} \rangle \right| ^ {2} \right] \geq 1 \quad \text {a n d} \quad \mathbb {E} \left[ \left| \langle \boldsymbol {a} \boldsymbol {a} ^ {\top}, \boldsymbol {U} \rangle \right| ^ {4} \right] \leq c _ {1} \left(\mathbb {E} \left[ \left| \langle \boldsymbol {a} \boldsymbol {a} ^ {\top}, \boldsymbol {U} \rangle \right| ^ {2} \right]\right) ^ {2}. \tag {49}
$$
Note that by the definition of the set  $E$ , every matrix  $U \in E$  satisfies  $\| U \|_F = 1$ . Utilizing inequalities (48) and (49), there exists positive constant  $c_2 > 0$  such that
$$
\mathbb {P} \left(\left| \langle \boldsymbol {a} \boldsymbol {a} ^ {\top}, \boldsymbol {U} \rangle \right| \geq \frac {1}{2}\right) \geq c _ {2}. \tag {50}
$$
Upper bound on  $a^\top \Sigma^\star a$ . By Hanson-Wright inequality [51, Theorem 1.1], there exist some positive absolute constants  $c_3$  and  $c_4$  such that for any  $t > 0$ , we have