SlowGuess/NeurIPS_2023_OCR huggingface dataset:
By the assumption that the noise is upper bounded by  $\eta^{\uparrow}$ , we have  $y + \bar{\eta} \leq y + \eta^{\uparrow}$ . Therefore, we have
$$
\begin{array}{l} \mathbb {E} \left(\bar {\gamma} ^ {2}\right) ^ {4} = \mathbb {E} \left(\frac {y + \bar {\eta}}{\boldsymbol {a} ^ {\top} \boldsymbol {\Sigma} ^ {\star} \boldsymbol {a}}\right) ^ {4} \\ \leq (y + \eta^ {\uparrow}) ^ {4} \cdot \mathbb {E} \left(\frac {1}{\boldsymbol {a} ^ {\top} \boldsymbol {\Sigma} ^ {\star} \boldsymbol {a}}\right) ^ {4} \\ \stackrel {\mathrm {(i)}} {\lesssim} \left(\frac {1}{\sigma_ {r} r}\right) ^ {4}, \\ \end{array}
$$
where step (i) applies part (a) of Lemma 4.
Recall that we assume we collect  $N$  measurements under the inverted measurements sensing model presented in Algorithm 1 with standard Gaussian sensing vectors and bounded noise, mean-zero noise (Assumption 1).
We first introduce a restricted strong convexity (RSC) condition that our proof relies on. Since the matrix  $\pmb{\Sigma}^{\star}$  is assumed to be symmetric positive semidefinite matrices and of rank  $r$ , we follow [24] and consider a restricted set on which we analyze the behavior of the sensing matrices  $\widetilde{A}_i$ . We call this set the "error set", defined by:
$$
\mathcal {E} = \left\{\boldsymbol {U} \in \mathbb {S} ^ {d \times d}: \| \boldsymbol {U} \| _ {*} \leq 4 \sqrt {2 r} \| \boldsymbol {U} \| _ {F} \right\}, \tag {22}
$$
where recall that  $\mathbb{S}^{d\times d}$  denotes the set of symmetric  $d\times d$  matrices. We say that our shrunken sensing matrices  $\{\widetilde{\pmb{A}}_i\}_{i = 1}^n$  satisfy a restricted strong convexity (RSC) condition over the error set  $\mathcal{E}$ , if there exists some positive constant  $\kappa >0$  such that
$$
\frac {1}{n} \sum_ {i = 1} ^ {n} \langle \widetilde {\boldsymbol {A}} _ {i}, \boldsymbol {U} \rangle^ {2} \geq \kappa \| \boldsymbol {U} \| _ {F} ^ {2} \quad \text {f o r a l l} \boldsymbol {U} \in \mathcal {E}. \tag {23}
$$
The following proposition shows that the estimation error, when the sensing matrices satisfy the RSC condition and the regularization parameter is sufficiently large.
Proposition 1 ([39, Theorem 1] with  $q = 0$ ). Suppose that  $\Sigma^{\star}$  has rank  $r$  and the shrunken sensing matrices satisfy the restricted strong convexity condition (23) with positive constant  $\kappa > 0$ . Then if the regularization parameter satisfies
$$
\lambda_ {n} \geq 2 \left\| \frac {1}{n} \sum_ {i = 1} ^ {n} y \widetilde {\boldsymbol {A}} _ {i} - \frac {1}{n} \sum_ {i = 1} ^ {n} \langle \widetilde {\boldsymbol {A}} _ {i}, \boldsymbol {\Sigma} ^ {\star} \rangle \widetilde {\boldsymbol {A}} _ {i} \right\| _ {\mathrm {o p}}, \tag {24}
$$
any optimal solution  $\widehat{\Sigma}$  of the optimization program (8) satisfies
$$
\left\| \widehat {\boldsymbol {\Sigma}} - \boldsymbol {\Sigma} ^ {\star} \right\| _ {F} \leq \frac {3 2 \sqrt {r} \lambda_ {n}}{\kappa}.
$$
This theorem is a special case of Theorem 1 in [39], which is in turn adapted from Theorem 1 in [24] (see [24] or [39] for the proof). Proposition 1 is a deterministic and nonasymptotic result and provides a roadmap for proving our desired upper bound. First, we show that the operator norm (24) can be upper bounded with high probability, allowing us to set the regularization parameter  $\lambda_{n}$  accordingly. Second, we show that the RSC condition (23) is satisfied with high probability. We begin by bounding the operator norm (24) in the following proposition.
Proposition 2. Let  $y^{\uparrow} = y + \eta^{\uparrow}$ . Suppose that  $\Sigma^{\star}$  has rank  $r$ , with  $r > 8$ . Then there exists a positive absolute constant  $C_1$  such that
$$
\left\| \frac {1}{n} \sum_ {i = 1} ^ {n} y \widetilde {\boldsymbol {A}} _ {i} - \frac {1}{n} \sum_ {i = 1} ^ {n} \langle \widetilde {\boldsymbol {A}} _ {i}, \boldsymbol {\Sigma} ^ {\star} \rangle \widetilde {\boldsymbol {A}} _ {i} \right\| _ {\mathrm {o p}} \leq C _ {1} \left[ y ^ {\uparrow} \left(\frac {y ^ {\uparrow}}{\sigma_ {r} r} \sqrt {\frac {d}{n}} + \frac {d}{n} \tau + \left(\frac {y ^ {\uparrow}}{\sigma_ {r} r}\right) ^ {2} \frac {1}{\tau}\right) + \frac {1}{\sigma_ {r} r} \frac {\nu_ {\eta} ^ {2}}{m} \right] \tag {25}
$$
with probability at least  $1 - 4\exp (-d)$
The proof of Proposition 2 is provided in Appendix E.1. Next, we show that the RSC condition (23) is satisfied with high probability, as is done in the following proposition.
Proposition 3. Let  $\mu_y$  be the median of  $y + \bar{\eta}$ . Suppose that the truncation threshold  $\tau$  satisfies  $\tau \geq \frac{\mu_y}{\mathrm{tr}(\pmb{\Sigma}^\star)}$ . Then there exist positive absolute constants  $\kappa_{\mathcal{L}}$ ,  $c$ , and  $C$  such that if the number of sensing vectors satisfy
$$
n \geq C r d
$$
$$
\frac {1}{n} \sum_ {i = 1} ^ {n} \langle \widetilde {\boldsymbol {A}} _ {i}, \boldsymbol {U} \rangle^ {2} \geq \kappa_ {\mathcal {L}} \left(\frac {\mu_ {y}}{\operatorname {t r} \left(\boldsymbol {\Sigma} ^ {\star}\right)}\right) ^ {2} \| \boldsymbol {U} \| _ {F} ^ {2} \tag {26}
$$
simultaneously for all matrices  $\mathbf{U} \in \mathcal{E}$  with probability at least  $1 - \exp(-cn)$ , where  $\mathcal{E}$  is the error set defined in Equation (22).
The proof of Proposition 3 is provided in Appendix E.2. We now utilize the results of Propositions 1, 2 and 3 to derive our final error bound. By Proposition 2, the operator norm (24) can be upper bounded with high probability. We set the regularization parameter  $\lambda_{n}$  to satisfy
$$
\lambda_ {n} \geq C _ {1} \left[ y ^ {\uparrow} \left(\frac {y ^ {\uparrow}}{\sigma_ {r} r} \sqrt {\frac {d}{n}} + \frac {d}{n} \tau + \left(\frac {y ^ {\uparrow}}{\sigma_ {r} r}\right) ^ {2} \frac {1}{\tau}\right) + \frac {1}{\sigma_ {r} r} \frac {\nu_ {\eta} ^ {2}}{m} \right],
$$
where  $C_1$  is the constant in Proposition 2. Furthermore, by Proposition 3, we have that there exist universal constants  $C_2 > 0$  such that if the number of sensing vectors satisfies  $n \geq C_2 r d$ , the RSC condition (23) holds for constant  $\kappa = \kappa_{\mathcal{L}} \left( \frac{\mu_y}{\mathrm{tr}(\Sigma^*)} \right)^2$  with high probability. Taking a union bound, we have that Proposition 2 and Proposition 3 hold simultaneously with probability at least  $1 - 4 \exp(-d) - \exp(-cn)$ . Invoking Proposition 1, we have
$$
\begin{array}{l} \left\| \widehat {\boldsymbol {\Sigma}} - \boldsymbol {\Sigma} ^ {\star} \right\| _ {F} \leq 3 2 \sqrt {r} \cdot \frac {\lambda_ {n}}{\kappa_ {\mathcal {L}} \left(\frac {\mu_ {y}}{\operatorname {t r} (\boldsymbol {\Sigma} ^ {\star})}\right) ^ {2}} \\ \lesssim \left(\frac {\mathrm {t r} (\pmb {\Sigma} ^ {\star})}{\mu_ {y}}\right) ^ {2} \sqrt {r} \lambda_ {n} \\ \end{array}
$$
with probability at least  $1 - 4\exp (-d) - \exp (-cn)$ , as desired.
E.1 Proof of Proposition 2
In the proof, we decompose the operator norm  $\left\| \frac{1}{n} \sum_{i=1}^{n} y \widetilde{\boldsymbol{A}}_i - \frac{1}{n} \sum_{i=1}^{n} \langle \widetilde{\boldsymbol{A}}_i, \boldsymbol{\Sigma}^\star \rangle \widetilde{\boldsymbol{A}}_i \right\|_{\mathrm{op}}$  from (25) into individual terms and bound them separately. Recall the definitions of random matrices  $\bar{\boldsymbol{A}}$  from Equation 15 and  $\widetilde{\boldsymbol{A}}$  from Equation 16.
Step 1: decompose the error into five terms. We begin by adding and subtracting multiple quantities as follows:
$$
\begin{array}{l} \frac {1}{n} \sum_ {i = 1} ^ {n} y \widetilde {\boldsymbol {A}} _ {i} - \frac {1}{n} \sum_ {i = 1} ^ {n} \langle \widetilde {\boldsymbol {A}} _ {i}, \boldsymbol {\Sigma} ^ {\star} \rangle \widetilde {\boldsymbol {A}} _ {i} = \frac {1}{n} \sum_ {i = 1} ^ {n} y \widetilde {\boldsymbol {A}} _ {i} - \mathbb {E} [ y \widetilde {\boldsymbol {A}} ] + \mathbb {E} [ y \widetilde {\boldsymbol {A}} ] - \mathbb {E} [ y \bar {\boldsymbol {A}} ] \\ + \mathbb {E} [ y \bar {\boldsymbol {A}} ] - \mathbb {E} [ \langle \widetilde {\boldsymbol {A}}, \boldsymbol {\Sigma} ^ {\star} \rangle \widetilde {\boldsymbol {A}} ] + \mathbb {E} [ \langle \widetilde {\boldsymbol {A}}, \boldsymbol {\Sigma} ^ {\star} \rangle \widetilde {\boldsymbol {A}} ] - \frac {1}{n} \sum_ {i = 1} ^ {n} \langle \widetilde {\boldsymbol {A}} _ {i}, \boldsymbol {\Sigma} ^ {\star} \rangle \widetilde {\boldsymbol {A}} _ {i} \\ \stackrel {(i)} {=} \frac {1}{n} \sum_ {i = 1} ^ {n} y \widetilde {\boldsymbol {A}} _ {i} - \mathbb {E} \left[ y \widetilde {\boldsymbol {A}} \right] + \mathbb {E} \left[ y \widetilde {\boldsymbol {A}} \right] - \mathbb {E} \left[ y \bar {\boldsymbol {A}} \right] \\ + \mathbb {E} \left[ \langle \bar {\boldsymbol {A}}, \boldsymbol {\Sigma} ^ {\star} \rangle \bar {\boldsymbol {A}} \right] - \mathbb {E} \left[ \langle \widetilde {\bar {\boldsymbol {A}}}, \boldsymbol {\Sigma} ^ {\star} \rangle \widetilde {\bar {\boldsymbol {A}}} \right] - \mathbb {E} \left[ \bar {\eta} \bar {\boldsymbol {A}} \right] \\ + \mathbb {E} \left[ \langle \widetilde {\boldsymbol {A}}, \boldsymbol {\Sigma} ^ {\star} \rangle \widetilde {\boldsymbol {A}} \right] - \frac {1}{n} \sum_ {i = 1} ^ {n} \langle \widetilde {\boldsymbol {A}} _ {i}, \boldsymbol {\Sigma} ^ {\star} \rangle \widetilde {\boldsymbol {A}} _ {i}, \tag {27} \\ \end{array}
$$
where step (i) is true by substituting  $y = \langle \bar{A}, \Sigma^{\star} \rangle - \bar{\eta}$  to the term of  $\mathbb{E}[y\bar{A}]$ , and the fact that the noise term  $\bar{\eta}$  is zero-mean. By triangle inequality, we group the terms in (27) and bound the operator norm by
$$
\begin{array}{l} \left\| \frac {1}{n} \sum_ {i = 1} ^ {n} y \widetilde {\boldsymbol {A}} _ {i} - \frac {1}{n} \sum_ {i = 1} ^ {n} \langle \widetilde {\boldsymbol {A}} _ {i}, \boldsymbol {\Sigma} ^ {\star} \rangle \widetilde {\boldsymbol {A}} _ {i} \right\| _ {\text {o p}} \leq \underbrace {y \left\| \frac {1}{n} \sum_ {i = 1} ^ {n} \widetilde {\boldsymbol {A}} _ {i} - \mathbb {E} [ \widetilde {\boldsymbol {A}} ] \right\| _ {\text {o p}}} _ {\text {T e r m 1}} \\ + \underbrace {y \left\| \mathbb {E} [ \widetilde {\boldsymbol {A}} ] - \mathbb {E} [ \bar {\boldsymbol {A}} ] \right\| _ {\mathrm {o p}}} _ {\text {T e r m 2}} \\ + \underbrace {\left\| \mathbb {E} \left[ \langle \bar {A} , \boldsymbol {\Sigma} ^ {\star} \rangle \bar {A} \right] - \mathbb {E} \left[ \langle \tilde {\bar {A}} , \boldsymbol {\Sigma} ^ {\star} \rangle \tilde {\bar {A}} \right] \right\| _ {\mathrm {o p}}} _ {\text {T e r m 3}} \\ + \underbrace {\left\| \mathbb {E} \left[ \langle \widetilde {\boldsymbol {A}} , \boldsymbol {\Sigma} ^ {\star} \rangle \widetilde {\boldsymbol {A}} \right] - \frac {1}{n} \sum_ {i = 1} ^ {n} \langle \widetilde {\boldsymbol {A}} _ {i} , \boldsymbol {\Sigma} ^ {\star} \rangle \widetilde {\boldsymbol {A}} _ {i} \right\| _ {\mathrm {o p}}} _ {\text {T e r m 4}} \\ + \underbrace {\left\| \mathbb {E} [ \bar {\eta} \bar {\boldsymbol {A}} ] \right\| _ {\mathrm {o p}}} _ {\text {T e r m 5}}. \tag {28} \\ \end{array}
$$
In the remaining proof, we bound the five terms in (28) individually. We first discuss the nature of these five terms.
- Terms 1 and 4: These two terms characterize the difference between the empirical mean of quantities involving  $\widetilde{A}$  and their true expectation (see Lemma 6 and Lemma 9). In the proof, we show that the empirical mean concentrates around the expectation with high probability, as a function of the number of sensing vectors  $n$ .
- Terms 2 and 3: These two terms characterize the difference in expectation introduced by truncating  $\bar{A}$  to  $\tilde{A}$  (see Lemma 7 and Lemma 8). Hence, these two terms characterize biases that arise from truncation. They diminish as  $\tau \to \infty$ , because setting  $\tau$  to  $\infty$  is equivalent to no thresholding, and hence  $\tilde{A}$  becomes identical to  $\bar{A}$ . Since expectations are considered, these two terms depend on the threshold  $\tau$ , but not the number of sensing vectors  $n$  or the averaging parameter  $m$ .
- Term 5: Term 5 is a bias term that arises from the fact that the mean of the noise  $\eta$  conditioned on sensing matrix  $\bar{A}$  is non-zero. We show that this bias scales like  $\frac{1}{m}$  (see Lemma 10) in terms of the averaging parameter  $m$ .
Putting these terms together, Terms 1 and 4 depend on  $n$ , Terms 2 and 3 depend on  $\tau$ , and Term 5 depends on  $m$ . In Corollary 1, we set the values of  $\tau, n$  and  $m$  to balance these terms.
Step 2: bound the five terms individually. In what follows, we provide five lemmas to bound each of the five terms individually. In the proofs of the five lemmas, we rely on an upper bound on the fourth moment of the  $m$ -sample averaged measurements  $\bar{\gamma}^2$ . As shown in Lemma 5 in Appendix D.5, for some absolute constant  $c$ , this fourth moment can be upper bounded by a quantity that we denote  $M$ :
$$
\mathbb {E} \left[ \left(\bar {\gamma} ^ {2}\right) ^ {4} \right] \leq M = c \left(\frac {y ^ {\uparrow}}{\sigma_ {r} r}\right) ^ {4}. \tag {29}
$$
We also rely heavily on the following truncation properties relating the  $m$ -sample averaged measurements  $\bar{\gamma}^2$  and truncated measurements  $\tilde{\gamma}^2$ :
$$
\tilde {\gamma} _ {i} ^ {2} \leq \tau \tag {TP1}
$$
$$
\tilde {\gamma} _ {i} ^ {2} \leq \bar {\gamma} _ {i} ^ {2} \tag {TP2}
$$
$$
\bar {\gamma} _ {i} ^ {2} - \tilde {\gamma} _ {i} ^ {2} = \left(\tilde {\gamma} _ {i} ^ {2} - \tilde {\gamma} _ {i} ^ {2}\right) \cdot \mathbf {1} \left\{\bar {\gamma} _ {i} ^ {2} \geq \tau \right\}. \tag {TP3}
$$
The following lemma provides a bound for Term 1.
Lemma 6. Let  $\widetilde{A}_1, \ldots, \widetilde{A}_n$  be i.i.d copies of a random matrix  $\widetilde{A}$  as defined in Equation (16). There exists an absolute constant  $c > 0$  such that for any  $t > 0$ , we have
$$
\left\| \frac {1}{n} \sum_ {i = 1} ^ {n} \widetilde {\boldsymbol {A}} _ {i} - \mathbb {E} \left[ \widetilde {\boldsymbol {A}} _ {i} \right] \right\| _ {\mathrm {o p}} \leq c \left(\sqrt {\frac {M ^ {1 / 2} t}{n}} + \frac {\tau t}{n}\right)
$$
with probability at least  $1 - 2 \cdot 9^d \cdot \exp(-t)$ .
The proof of Lemma 6 is provided in Appendix E.1.1. The next lemma provides an upper bound for Term 2.
Lemma 7. Let  $\tilde{A}$  and  $\tilde{A}$  be the random matrices defined in Equation (15) and Equation (16), respectively. Then there exists an absolute constant  $c > 0$  such that
$$
\left\| \mathbb {E} \left[ \widetilde {\boldsymbol {A}} \right] - \mathbb {E} \left[ \bar {\boldsymbol {A}} \right] \right\| _ {\mathrm {o p}} \leq \frac {c M ^ {1 / 2}}{\tau}.
$$
The proof of Lemma 7 is provided in Appendix E.1.2. The following lemma provides an upper bound for Term 3. Recall that the quantity  $y^{\uparrow}$  denotes  $y + \eta^{\uparrow}$ .
Lemma 8. Let  $\tilde{A}$  and  $\tilde{A}$  be the random matrices defined in Equation (15) and Equation (16), respectively. Then there exists an absolute constant  $c > 0$  such that
$$
\left\| \mathbb {E} \left[ \langle \bar {\boldsymbol {A}}, \boldsymbol {\Sigma} ^ {\star} \rangle \bar {\boldsymbol {A}} \right] - \mathbb {E} \left[ \langle \widetilde {\bar {\boldsymbol {A}}}, \boldsymbol {\Sigma} ^ {\star} \rangle \widetilde {\bar {\boldsymbol {A}}} \right] \right\| _ {\mathrm {o p}} \leq \frac {c y ^ {\uparrow} M ^ {1 / 2}}{\tau}.
$$
The proof of Lemma 8 is provided in Appendix E.1.3. The following lemma provides an upper bound for Term 4.
Lemma 9. Let  $\widetilde{A}_1, \ldots, \widetilde{A}_n$  be i.i.d copies of a random matrix  $\widetilde{A}$  defined in Equation (16). There exists an absolute constant  $c > 0$  such that for any  $t > 0$ , we have
$$
\left\| \mathbb {E} \left[ \langle \widetilde {\boldsymbol {A}}, \boldsymbol {\Sigma} ^ {\star} \rangle \widetilde {\boldsymbol {A}} \right] - \frac {1}{n} \sum_ {i = 1} ^ {n} \langle \widetilde {\boldsymbol {A}} _ {i}, \boldsymbol {\Sigma} ^ {\star} \rangle \widetilde {\boldsymbol {A}} _ {i} \right\| _ {\mathrm {o p}} \leq c y ^ {\uparrow} \left(\sqrt {\frac {M ^ {1 / 2} t}{n}} + \frac {\tau t}{n}\right)
$$
The proof of Lemma 9 is provided in Appendix E.1.4. We note that Terms 2 and 3 are bias that result from shrinkage, but crucially are inversely dependent on the shrinkage threshold  $\tau$ . This fact allows us to set  $\tau$  so that the order of Terms 2 and 3 match the order of Terms 1 and 4.
The final lemma bounds Term 5, which is a bias that arises from the dependence of the sensing matrix  $\vec{A}$  on the noise  $\eta$ .
Lemma 10. Let  $\bar{A}$  be the random matrix defined in Equation (15). Suppose that  $\Sigma^{\star}$  has rank  $r$  with  $r > 2$ . Then there exists an absolute constant  $c > 0$  such that
$$
\mathbb {E} \left[ \left\| \bar {\eta} \bar {\boldsymbol {A}} \right\| _ {\mathrm {o p}} \right] \leq \frac {c}{\sigma_ {r} r} \frac {\nu_ {\eta} ^ {2}}{m}.
$$
The proof of Lemma 10 is provided in Appendix E.1.5. We note that the bias scales with the variance of the  $m$ -sample averaged noise  $\bar{\eta}$ , which scales inversely with  $m$ .
Step 3: combine the five terms. We set  $t = (\log 9 + 1)d$ . Substituting the bounds from Lemmas 6-10 back to (28) and taking a union bound, we have that with probability at least  $1 - 4\exp (-d)$ ,
$$
\begin{array}{l} \left\| \frac {1}{n} \sum_ {i = 1} ^ {n} y \widetilde {\boldsymbol {A}} _ {i} - \frac {1}{n} \sum_ {i = 1} ^ {n} \langle \widetilde {\boldsymbol {A}} _ {i}, \boldsymbol {\Sigma} ^ {\star} \rangle \widetilde {\boldsymbol {A}} _ {i} \right\| _ {\text {o p}} \lesssim (y ^ {\uparrow} + 1) \left(\sqrt {\frac {M ^ {1 / 2} d}{n}} + \frac {d}{n} \tau + \frac {M ^ {1 / 2}}{\tau}\right) + \frac {1}{\sigma_ {r} r} \frac {\nu_ {\eta} ^ {2}}{m} \\ \stackrel {(i)} {\lesssim} y ^ {\uparrow} \left(\frac {y ^ {\uparrow}}{\sigma_ {r} r} \sqrt {\frac {d}{n}} + \frac {d}{n} \tau + \left(\frac {y ^ {\uparrow}}{\sigma_ {r} r}\right) ^ {2} \frac {1}{\tau}\right) + \frac {1}{\sigma_ {r} r} \frac {\nu_ {\eta} ^ {2}}{m}, \\ \end{array}
$$
where step (i) is true by substituting in the expression (29) for  $M$ .
E.1.1 Proof of Lemma 6.
Let  $\mathcal{A}_{\frac{1}{4}} \subseteq S^{d-1}$  be a  $\frac{1}{4}$ -covering of the  $d$ -dimensional unit sphere  $\mathcal{S}^{d-1} := \{x \in \mathbb{R}^d : \|x\|_2 = 1\}$ . By a covering argument [46, Exercise 4.4.3], for any symmetric matrix  $\pmb{U} \in \mathbb{S}^{d \times d}$ , its operator norm is bounded by  $\|\pmb{U}\|_{\mathrm{op}} \leq 2\sup_{\pmb{v} \in \mathcal{A}_{\frac{1}{4}}} |\pmb{v}^\top \pmb{U}\pmb{v}|$ . Hence, we have
$$
\begin{array}{l} \left\| \frac {1}{n} \sum_ {i = 1} ^ {n} \widetilde {\boldsymbol {A}} _ {i} - \mathbb {E} \left[ \widetilde {\boldsymbol {A}} \right] \right\| _ {\text {o p}} \leq 2 \sup  _ {\boldsymbol {v} \in \mathcal {A} _ {\frac {1}{4}}} \left| \boldsymbol {v} ^ {\top} \left(\frac {1}{n} \sum_ {i = 1} ^ {n} \widetilde {\boldsymbol {A}} _ {i} - \mathbb {E} \left[ \widetilde {\boldsymbol {A}} \right]\right) \boldsymbol {v} \right| \\ = 2 \sup  _ {\boldsymbol {v} \in \mathcal {A} _ {\frac {1}{4}}} \left| \frac {1}{n} \sum_ {i = 1} ^ {n} \boldsymbol {v} ^ {\top} \widetilde {\boldsymbol {A}} _ {i} \boldsymbol {v} - \mathbb {E} \left[ \boldsymbol {v} ^ {\top} \widetilde {\boldsymbol {A}} \boldsymbol {v} \right] \right|. \tag {30} \\ \end{array}
$$
We invoke Bernstein's inequality. We first show that the Bernstein condition holds. Namely, we show that for each integer  $p \geq 2$ , we have that for any unit vector  $\pmb{v} \in \mathbb{R}^d$
$$
\mathbb {E} \left| \boldsymbol {v} ^ {\top} \widetilde {\boldsymbol {A}} \boldsymbol {v} \right| ^ {p} \leq \frac {p !}{2} u _ {1} u _ {2} ^ {p - 2}, \tag {31}
$$
where  $u_{1} = c_{1}M^{\frac{1}{2}}$  and  $u_{2} = c_{2}\tau$  for some universal positive constants  $c_{1}$  and  $c_{2}$ . Given the Bernstein condition (31), we then apply Bernstein's inequality to bound (30).
Proving the Bernstein condition (31). We fix any unit vector  $\pmb{v} \in \mathbb{R}^d$ . Since  $\widetilde{\pmb{A}} = \widetilde{\gamma}^2 \pmb{a} \pmb{a}^\top$ , we have  $\pmb{v}^\top \widetilde{\pmb{A}} \pmb{v} = \widetilde{\gamma}^2 (\pmb{v}^\top \pmb{a})^2$ . Recall that the random vector  $\pmb{a}$  is distributed as  $\pmb{a} \sim \mathcal{N}(\pmb{0}, \pmb{I}_d)$ . Since  $\pmb{v}$  is a unit vector, it follows that  $\pmb{v}^\top \pmb{a} \sim \mathcal{N}(0, 1)$ . Denote by  $G \sim \mathcal{N}(0, 1)$  a standard normal random variable. For any integer  $p \geq 2$ , we have
$$
\begin{array}{l} \mathbb {E} \left| \boldsymbol {v} ^ {\top} \widetilde {\boldsymbol {A}} \boldsymbol {v} \right| ^ {p} = \mathbb {E} \left(\widetilde {\gamma} ^ {2} G ^ {2}\right) ^ {p} \leq^ {(\mathrm {i})} \tau^ {p - 2} \mathbb {E} \left[ \left(\widetilde {\gamma} ^ {2}\right) ^ {2} G ^ {2 p} \right] \\ \stackrel {\text {(i i)}} {\leq} \tau^ {p - 2} \cdot \mathbb {E} \left[ \left(\bar {\gamma} ^ {2}\right) ^ {2} G ^ {2 p} \right] \\ \stackrel {(i i i)} {\leq} \tau^ {p - 2} \left(\mathbb {E} \left[ (\bar {\gamma} ^ {2}) ^ {4} \right] \cdot \mathbb {E} \left[ G ^ {4 p} \right]\right) ^ {1 / 2} \\ \stackrel {\text {(i v)}} {\leq} \tau^ {p - 2} \left(M \cdot \mathbb {E} \left[ G ^ {4 p} \right]\right) ^ {1 / 2}, \tag {32} \\ \end{array}
$$
where steps (i) and (ii) follow from (TP1) and (TP2), respectively; step (iii) follows from Cauchy-Schwarz inequality; and step (iv) follows upper bounding the fourth moment of  $\bar{\gamma}^2$  with the quantity  $M$  from Equation (29).
Note that since  $G$  is standard normal, by definition  $G^2$  follows a Chi-Square distribution with 1 degree of freedom, and hence sub-exponential. By Lemma 2 in Appendix D.2, there exists some constant  $c > 0$  such that we have  $\left(\mathbb{E}\left[(G^2)^p\right]\right)^{1/p} \leq cp$  for all  $p \geq 1$ . Hence, we have  $\left(\mathbb{E}\left[G^{4p}\right]\right)^{1/2p} \leq 2cp$  and
$$
\left(\mathbb {E} \left[ G ^ {4 p} \right]\right) ^ {1 / 2} \leq (2 c p) ^ {p} = \left(\frac {p}{e}\right) ^ {p} \cdot (2 e c) ^ {p}
$$
$$
<   p! \cdot (2 e c) ^ {p} \tag {33}
$$
where step (i) is true by Stirling's inequality that for all  $p \geq 1$ ,
$$
p! > \sqrt {2 \pi p} \left(\frac {p}{e}\right) ^ {p} e ^ {\frac {1}{1 2 p + 1}} > \left(\frac {p}{e}\right) ^ {p}.
$$
Substituting (33) back to (32) and rearranging terms completes the proof of the Bernstein condition (31).
Applying Bernstein's inequality to bound (30). By Bernstein's inequality (see Lemma 3), given condition (31), we have that for any unit vector  $\pmb{v} \in \mathbb{R}^d$  and any  $t > 0$ ,
$$
\mathbb {P} \left(\left| \frac {1}{n} \sum_ {i = 1} ^ {n} \boldsymbol {v} ^ {\top} \widetilde {\boldsymbol {A}} _ {i} \boldsymbol {v} - \mathbb {E} \left[ \boldsymbol {v} ^ {\top} \widetilde {\boldsymbol {A}} \boldsymbol {v} \right] \right| \geq 2 \left(\sqrt {\frac {c _ {1} M ^ {1 / 2} t}{n}} + \frac {c _ {2} \tau t}{n}\right)\right) \leq 2 \exp (- t). \tag {34}
$$
By [46, Corollary 4.2.13], the cardinality of the covering set  $\mathcal{A}_{\frac{1}{4}}$  is bounded above by  $9^d$ . Therefore, taking a union bound on (34), we have
$$
\mathbb {P} \left(\sup  _ {\boldsymbol {v} \in \mathcal {A} _ {\frac {1}{4}}} \left| \frac {1}{n} \sum_ {i = 1} ^ {n} \boldsymbol {v} ^ {\top} \widetilde {\boldsymbol {A}} _ {i} \boldsymbol {v} - \mathbb {E} \left[ \boldsymbol {v} ^ {\top} \widetilde {\boldsymbol {A}} \boldsymbol {v} \right] \right| \geq 2 \left(\sqrt {\frac {c _ {1} M ^ {1 / 2} t}{n}} + \frac {c _ {2} \tau t}{n}\right)\right) \leq 2 \cdot 9 ^ {d} \cdot \exp (- t). \tag {35}
$$
Substituting in (30) to (35), for any  $t > 0$ , we have
$$
\mathbb {P} \left(\left\| \frac {1}{n} \sum_ {i = 1} ^ {n} \widetilde {\boldsymbol {A}} _ {i} - \mathbb {E} \left[ \widetilde {\boldsymbol {A}} \right] \right\| _ {\mathrm {o p}} \lesssim \sqrt {\frac {M ^ {1 / 2} t}{n}} + \frac {\tau t}{n}\right) \geq 1 - 2 \cdot 9 ^ {d} \cdot \exp (- t),
$$
E.1.2 Proof of Lemma 7
By definition of the operator norm, we have
$$
\left\| \mathbb {E} \left[ \tilde {\boldsymbol {A}} \right] - \mathbb {E} \left[ \bar {\boldsymbol {A}} \right] \right\| _ {\mathrm {o p}} = \sup  _ {\boldsymbol {v} \in \mathcal {S} ^ {d - 1}} \left| \boldsymbol {v} ^ {\top} \left(\mathbb {E} \left[ \bar {\boldsymbol {A}} \right] - \mathbb {E} \left[ \tilde {\boldsymbol {A}} \right]\right) \boldsymbol {v} \right|.
$$