SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \operatorname *{arg  min}_{\mathbf{V}\in \mathcal{O}(n,d)}\widehat{R}^{2}(\mathbf{V}) = \operatorname *{arg  max}_{\mathbf{V}\in \mathcal{O}(n,d)}\int_{0}^{T}\sum_{i = 1}^{n}\left\| \mathbf{V}^{\top}\widehat{\Lambda}_{i}(t)\right\|_{2}^{2} \\ = \underset {\mathbf {V} \in \mathcal {O} (n, d)} {\arg \max } \int_ {0} ^ {T} \left\| \widehat {\boldsymbol {\Lambda}} (t) \mathbf {V} \right\| _ {\mathrm {F}} ^ {2} \\ = \underset {\mathbf {V} \in \mathcal {O} (n, d)} {\arg \max } \int_ {0} ^ {T} \operatorname {t r} \left\{\mathbf {V} ^ {\top} \widehat {\boldsymbol {\Lambda}} ^ {2} (t) \mathbf {V} \right\} \mathrm {d} t \\ = \underset {\mathbf {V} \in \mathcal {O} (n, d)} {\arg \max } \operatorname {t r} \left\{\mathbf {V} ^ {\top} \left(\int_ {0} ^ {T} \widehat {\boldsymbol {\Lambda}} ^ {2} (t)   \mathrm {d} t\right) \mathbf {V} \right\} \\ = \underset {\mathbf {V} \in \mathcal {O} (n, d)} {\arg \max } \operatorname {t r} \left\{\mathbf {V} ^ {\top} \hat {\boldsymbol {\Sigma}} \mathbf {V} \right\} \\ = \hat {\mathbf {U}} \\ \end{array}
$$
where the final equality follows from the Courant-Fisher min-max theorem. This concludes the proof.
F.1 Prerequisites
F.1.1 Additional notation
In this proof, we use the notation  $a_{n} \lesssim b_{n}$  to mean  $a_{n} \lesssim b_{n}$  with overwhelming probability.
F.1.2 Symmetric dilation with change of basis "trick"
Symmetric dilation is a proof technique which allows statements about the eigenvectors of a symmetric matrix to be easily extended to hold for the singular vectors of a (potentially rectangular) asymmetric
matrix. Let  $\mathbf{M}$  be an  $n_1\times n_2$  matrix with non-zero singular values  $\{\sigma_i\}_{i = 1}^r$  and corresponding orthonormal left singular vectors  $\{u_i\}_{i = 1}^r$  and right singular vectors  $\{v_{i}\}_{i = 1}^{r}$ . Its symmetric dilation is the  $n\times n$  matrix (with  $n = n_{1} + n_{2}$ ) constructed as
$$
\mathcal {D} (\mathbf {M}) = \left( \begin{array}{c c} \mathbf {0} & \mathbf {M} \\ \mathbf {M} ^ {\top} & \mathbf {0} \end{array} \right).
$$
One can easily verify that  $\mathcal{D}(\mathbf{M})$  has eigenvalues  $\{\pm \sigma_i\}_{i = 1}^r$  and eigenvectors  $\{(u_i^\top ,\pm v_i^\top)^\top \}_{i = 1}^r$ . We stack the first  $d$  left and right singular vectors into matrices  $\mathbf{U}\in \mathbb{R}^{n_1\times d}$  and  $\mathbf{V}\in \mathbb{R}^{n_2\times d}$ , and stack the first  $2d$  eigenvectors of  $\mathcal{D}(\mathbf{M})$  into a matrix
$$
\bar {\mathbf {U}} = \frac {1}{\sqrt {2}} \left( \begin{array}{c c} \mathbf {U} & \mathbf {U} \\ \mathbf {V} & - \mathbf {V} \end{array} \right).
$$
$$
\| \mathbf {U} \| _ {2, \infty} \vee \| \mathbf {V} \| _ {2, \infty} = \left\| \bar {\mathbf {U}} \right\| _ {2, \infty}, \qquad \text {a n d} \qquad \| \mathbf {M} \| _ {2} = \| \mathcal {D} (\mathbf {M}) \| _ {2}.
$$
While this standard construction is very useful when  $n_1 \asymp n_2$ , it can lead to suboptimal bounds when  $n_2 \gg n_1$ , or  $n_1 \gg n_2$ , due to an issue about incoherence, which was first raised in [13]. The incoherence of a subspace  $\mathcal{U}_0$  spanned by the orthonormal columns of a matrix  $\mathbf{U}_0 \in \mathbb{R}^{n_0 \times d}$  is
$$
\mu \left(\mathbf {U} _ {0}\right) = \sqrt {\frac {n _ {0}}{d}} \left\| \mathbf {U} _ {0} \right\| _ {2, \infty}.
$$
To obtain a good entrywise eigenvector bound under a signal-plus-noise matrix model it is typically necessary that  $\mu (\bar{\mathbf{U}})\asymp 1$ . Observe that
$$
\mu (\bar {\mathbf {U}}) = \sqrt {\frac {n _ {1} + n _ {2}}{2 d}} \left\| \bar {\mathbf {U}} \right\| _ {2, \infty} = \sqrt {\frac {n _ {1} + n _ {2}}{2 d}} \left(\| \mathbf {U} \| _ {2, \infty} \vee \| \mathbf {V} \| _ {2, \infty}\right) = \sqrt {\frac {n _ {1} + n _ {2}}{2 n _ {1}}} \mu (\mathbf {U}) + \sqrt {\frac {n _ {1} + n _ {2}}{2 n _ {2}}} \mu (\mathbf {V}).
$$
If  $\mu (\mathbf{U}),\mu (\mathbf{V})\asymp 1$  and  $n_1\succ n_2$ , then  $\mu (\bar{\mathbf{U}})\asymp 1$  and it is typically possible to obtain good bounds. However, when  $n_2\gg n_1$ , we have  $\mu (\bar{\mathbf{U}})\gg 1$ , and a good bound can typically not be obtained. The imbalance of  $n_1$  and  $n_2$  can cause similar issues when obtaining spectral norm bounds.
This issue can be overcome by changing to a basis which balances the contribution from its first  $n_1$  and second  $n_2$  elements of each column. Specifically, let  $\pi_1 = \sqrt{2n_1 / (n_1 + n_2)}$  and  $\pi_2 = \sqrt{2n_2 / (n_1 + n_2)}$ , and consider the basis  $\tilde{e}_1, \ldots, \tilde{e}_{n_1 + n_2}$ , such that
$$
e _ {i} = \left\{ \begin{array}{l l} \pi_ {1} \tilde {e} _ {i} & \text {i f} i \in \{1, \ldots , n _ {1} \} \\ \pi_ {2} \tilde {e} _ {i} & \text {i f} i \in \{n _ {1} + 1, \ldots , n _ {1} + n _ {2} \}, \end{array} \right.
$$
where  $\{e_i\}_{i=1}^{n_1+n_2}$  are the standard basis vectors in  $\mathbb{R}^{n_0}$ . Let  $\|\cdot\|_{\eta}$  denote a norm with respect to the column basis  $\{\tilde{e}_i\}_{i=1}^{n_1+n_2}$ , then one can verify that
$$
\| \mathcal {D} (\mathbf {M}) \| _ {2} = \| \mathbf {M} \|, \quad \text {a n d} \quad \left\| \left\| \bar {\mathbf {U}} \right\| \right\| _ {2, \infty} = \pi_ {1} \| \mathbf {U} \| _ {2, \infty} \vee \pi_ {2} \| \mathbf {V} \| _ {2, \infty}.
$$
As a result, if  $\tilde{\mu} (\mathbf{U}_0) = \sqrt{n_0 / d}\| \mathbf{U}_0\|_{2,\infty}$ , then
$$
\tilde {\mu} (\bar {\mathbf {U}}) = \mu (\mathbf {U}) \vee \mu (\mathbf {V}),
$$
regardless of the relative sizes of  $n_1$  and  $n_2$ . We use this symmetric dilation with change-of-basis "trick" to apply some existing theorems for symmetric matrices to our setting.
F.1.3 Concentration inequalities
In this section, we state a collection of lemmas which we will make use of throughout the proof. We begin with a tail bound for a Poisson random variable.
Lemma 3. Let  $X \sim \mathrm{Poisson}(\lambda)$ . Then
$$
\mathbb {P} \left(| X - \lambda | \geq t\right) \leq 2 \exp \left(- \frac {t ^ {2}}{2 (\lambda + t / 3)}\right).
$$
For  $t\geq \lambda$
$$
\mathbb {P} (| X - \lambda | \geq t) \leq 2 e ^ {- 3 t / 8}.
$$
The bound can be established by approximating the Poisson distribution with mean  $\lambda$  as the sum of  $k$  Bernoulli random variables with mean  $\lambda / k$ , applying Bernstein's inequality, and taking  $k \to 0$ .
Our next result is a concentration bound which adapts Lemma A.1 of [66] and can be proved using a vector version of the Bernstein inequality (Corollary 4.1 in [67]).
Lemma 4. Let  $X_{i} \sim \mathrm{Poisson}(\lambda_{i})$  independently for all  $i = 1, \dots, n$ , and suppose  $\mathbf{Q} \in \mathbb{R}^{n \times d}$  is a deterministic matrix whose rows we denote  $Q_{i}$ . Let  $\lambda_{\max} := \max_{i \in [n]} \lambda_{i}$ , then with probability  $1 - 28n^{-3}$
$$
\left\| \sum_ {i = 1} ^ {n} (X _ {i} - \lambda_ {i}) Q _ {i} \right\| _ {2} \leq 3 \log^ {2} n \| \mathbf {Q} \| _ {2, \infty} + \sqrt {6 \lambda_ {\max} \log n} \| \mathbf {Q} \| _ {\mathrm {F}}.
$$
Next, we state a concentration bound for the spectral norm of random matrices with independent entries which appears as Corollary 3.12 in [68]. The original statement of this lemma is for symmetric random matrices, although we general it to arbitrary random matrices using the symmetric dilation with change-of-basis trick described in Section F.1.2.
Lemma 5 (Corollary 3.12 of [68]). Let  $\mathbf{X}$  be an  $n_1\times n_2$  matrix whose entries  $x_{ij}$  are independent random variables which obey
$$
\mathbb {E} \left(x _ {i j}\right) = 0, \quad a n d \quad \left| x _ {i j} \right| \leq B, \quad i \in [ n _ {1} ], j \in [ n _ {2} ].
$$
Then there exists a universal constant  $c > 0$  such that for any  $t \geq 0$
$$
\mathbb {P} \left\{\| \mathbf {X} \| \geq 4 \sqrt {\nu} + t \right\} \leq n \exp \left(- \frac {t ^ {2}}{c B ^ {2}}\right).
$$
$$
\nu := \max  \left\{\pi_ {1} \max  _ {i \in [ n _ {1} ]} \sum_ {j = 1} ^ {n _ {2}} \mathbb {E} \left(x _ {i j} ^ {2}\right), \pi_ {2} \max  _ {i \in [ n _ {2} ]} \sum_ {j = 1} ^ {n _ {1}} \mathbb {E} \left(x _ {j i} ^ {2}\right) \right\}
$$
and  $\pi_k = 2n_k / (n_1 + n_2)$
F.1.4 Weyl's inequality and Wedin's  $\sin \Theta$  theorem
The next two lemmas are classical results matrix perturbation theory. Weyl's inequality shows that the singular values of a matrix are stable with respect to small perturbations.
Lemma 6 (Weyl's inequality). Let  $\mathbf{M}, \mathbf{E}$  be  $n_1 \times n_2$  real-valued matrices. Then for every  $1 \leq i \leq (n_1 \wedge n_2)$ , the  $i$ th largest singular value of  $\mathbf{M}$  and  $\mathbf{M} + \mathbf{E}$  obey
$$
\left| \sigma_ {i} (\mathbf {M} + \mathbf {E}) - \sigma_ {i} (\mathbf {M}) \right| \leq \left\| \mathbf {E} \right\| _ {2}.
$$
One way to measure the distance between two subspaces  $\mathcal{U}$  and  $\widehat{\mathcal{U}}$  is via principal angles. Let  $\mathbf{U}$ ,  $\widehat{\mathbf{U}}$  be matrices whose orthonormal columns span  $\mathcal{U}$  and  $\widehat{\mathcal{U}}$  respectively, and let  $\{\xi_i\}_{i=1}^d$  denote the singular values of  $\mathbf{U}^\top \widehat{\mathbf{U}}$ . Then the principal angles  $\{\theta_i\}_{i=1}^d$  between  $\mathcal{U}$  and  $\widehat{\mathcal{U}}$  are defined by  $\xi_i = \cos(\theta_i)$ . Let  $\sin \Theta(\mathbf{U}, \widehat{\mathbf{U}}) := \mathrm{diag}(\sin \theta_1, \ldots, \sin \theta_d)$ . Another way to measure the distance between  $\mathcal{U}$  and  $\widehat{\mathcal{U}}$  is via the difference between the projection operators  $\mathbf{UU}^\top$  and  $\widehat{\mathbf{U}} \widehat{\mathbf{U}}^\top$ , and in fact, these two characterisations are equivalent. Specifically,
$$
\left\| \sin \Theta (\mathbf {U}, \widehat {\mathbf {U}}) \right\| _ {2} \equiv \left\| \mathbf {U} \mathbf {U} ^ {\top} - \widehat {\mathbf {U}} \widehat {\mathbf {U}} ^ {\top} \right\| _ {2}.
$$
We will use this equivalence without mention throughout the proof. Wedin's  $\sin \Theta$  theorem shows that the singular vectors of a matrix are stable with respect to small perturbations.
Lemma 7. Let  $\mathbf{M}$  and  $\widehat{\mathbf{M}} = \mathbf{M} + \mathbf{E}$  be two  $n_1 \times n_2$  real-valued matrices, and denote by  $\mathbf{U}$ ,  $\widehat{\mathbf{U}}$  (respectively  $\mathbf{V}$ ,  $\widehat{\mathbf{V}}$ ) the matrices whose columns contain  $d$  orthonormal left (respectively, right) singular vectors, corresponding to the  $d$  largest singular values of  $\mathbf{M}$  and  $\widehat{\mathbf{M}}$ . Let  $\delta = \sigma_d(\mathbf{M}) - \sigma_{d+1}(\mathbf{M})$  and suppose that  $\| \mathbf{E} \| < (1 - 1/\sqrt{2})\delta$ , then
$$
\left\| \sin \Theta (\mathbf {U}, \widehat {\mathbf {U}}) \right\| _ {2} \vee \left\| \sin \Theta (\mathbf {V}, \widehat {\mathbf {V}}) \right\| _ {2} \leq \frac {2 (\| \mathbf {E} ^ {\top} \mathbf {U} \| _ {2} \vee \| \mathbf {E V} \| _ {2})}{\delta} \leq \frac {2 \| \mathbf {E} \|}{\delta}.
$$
See [69] for a proof.
F.2 Implications of Assumptions 1-4
We state here some inequalities involving the parameters of our problem which follow from Assumptions 1-4, and elementary linear algebra. We will use these facts throughout the proof without mention.
$$
\sqrt {n \lambda_ {\max }} \lesssim \delta \leq n \lambda_ {\max }; \tag {6}
$$
$$
\delta \log n \gtrsim \kappa n \lambda_ {\max }; \tag {7}
$$
$$
\kappa \lesssim \log n. \tag {8}
$$
The inequality (6) holds since  $\delta \leq \sigma_1^{1/2}(\pmb{\Sigma}) \leq \sqrt{n}\|\pmb{\Sigma}\|_{\max}^{1/2} \leq n\lambda_{\max}$ , and
$$
\delta \gtrsim \frac {\kappa n \lambda_ {\max}}{\log (\delta / \sqrt {n \lambda_ {m a x}})} \gtrsim \frac {n \lambda_ {\max}}{\log n} \gtrsim \sqrt {n \lambda_ {\max} \log n} \gtrsim \sqrt {n \lambda_ {\max}}
$$
where we invoked Assumption 4. (7) holds by noting that the previous bound implies  $\log (\delta /\sqrt{n\lambda_{\mathrm{max}}})\lesssim \log n$  and invoking Assumption 3. (8) follows from (6) since  $\kappa \lesssim \delta \log n / n\lambda_{\mathrm{max}}\lesssim \log n$
F.3 Setup
We begin by defining  $M$  equally spaced bins in  $(0,1]$
$$
B _ {1} := \left(0, \frac {1}{M} \right], \quad B _ {2} := \left(\frac {1}{M}, \frac {2}{M} \right], \dots , B _ {M} := \left(\frac {M - 1}{M}, 1 \right],
$$
and define the piecewise approximation of  $\lambda_{i}(t)$
$$
\bar {\lambda} _ {i} (t) = M \int_ {B _ {m}} \lambda_ {i} (t) \mathrm {d} t, \quad t \in B _ {m}, m \in [ M ].
$$
We then define  $t_1, \ldots, t_m \in (0, 1]$  such that  $\bar{\lambda}_{ij}(t) = \lambda_{ij}(t_m)$  for all  $t \in B_m$ , which exist by the continuity of  $\lambda_{ij}(t)$ , and define the piecewise constant approximation of  $Y_i(t)$  as  $\bar{Y}_i(t) = \mathbf{U}^\top \bar{\Lambda}_i(t)$ . Our strategy to obtain the bound in Theorem 1 is to decompose it into bias and variance terms:
$$
\max  _ {i, j \in [ n ]} \sup  _ {t \in \mathcal {T}} \left\| \mathbf {W} _ {1} \widehat {Y} _ {i} (t) - Y _ {i} (t) \right\| _ {2} = \underbrace {\max  _ {i , j \in [ n ]} \sup  _ {t \in \mathcal {T}} \left\| \mathbf {W} _ {1} \widehat {Y} _ {i} (t) - \bar {Y} _ {i} (t) \right\| _ {2}} _ {\text {v a r i a n c e}} + \underbrace {\max  _ {i , j \in [ n ]} \sup  _ {t \in \mathcal {T}} \left\| \mathbf {W} _ {2} \bar {Y} _ {i} (t) - Y _ {i} (t) \right\| _ {2}} _ {\text {b i a s}}.
$$
Section F.6 is dedicated to bounding the bias term, and the rest of this section is dedicated to bounding the variance term. Define the unfolding matrices  $\widehat{\Lambda}$  and  $\Lambda$  (without arguments) and their (thin) singular value decompositions as
$$
\widehat {\boldsymbol {\Lambda}} := \left(\widehat {\boldsymbol {\Lambda}} (t _ {1}) \dots \widehat {\boldsymbol {\Lambda}} (t _ {M})\right) = \widehat {\mathbf {U}} \widehat {\mathbf {S}} \widehat {\mathbf {V}} ^ {\top} + \widehat {\mathbf {U}} _ {\perp} \widehat {\mathbf {S}} _ {\perp} \widehat {\mathbf {V}} _ {\perp} ^ {\top},
$$
$$
\bar {\boldsymbol {\Lambda}} := \left(\boldsymbol {\Lambda} \left(t _ {1}\right) \dots \boldsymbol {\Lambda} \left(t _ {M}\right)\right) = \bar {\mathbf {U}} \bar {\mathbf {S}} \bar {\mathbf {V}} ^ {\top} + \bar {\mathbf {U}} _ {\perp} \bar {\mathbf {S}} _ {\perp} \bar {\mathbf {V}} _ {\perp} ^ {\top}.
$$
Then one has that for  $t \in B_m$ ,  $m \in [M]$
$$
\widehat {\mathbf {Y}} (t) := \widehat {\boldsymbol {\Lambda}} (t _ {m}) \widehat {\mathbf {U}} = \widehat {\mathbf {V}} _ {m} \widehat {\mathbf {S}}, \qquad \bar {\mathbf {Y}} (t) := \bar {\boldsymbol {\Lambda}} (t _ {m}) \bar {\mathbf {U}} = \bar {\mathbf {V}} _ {m} \widehat {\mathbf {S}}
$$
where  $\widehat{\mathbf{V}}_m, \bar{\mathbf{V}}_m$  denote the  $m$ th blocks of  $\widehat{\mathbf{V}}$  and  $\bar{\mathbf{V}}$  respectively. Therefore it follows that,
$$
\max  _ {i, j \in [ n ]} \sup  _ {t \in \mathcal {T}} \left\| \mathbf {W} _ {1} \widehat {Y} _ {i} (t) - \bar {Y} _ {i} (t) \right\| _ {2} = \left\| \widehat {\mathbf {V}} \widehat {\mathbf {S}} \mathbf {W} _ {1} ^ {\top} - \bar {\mathbf {V}} \bar {\mathbf {S}} \right\| _ {2, \infty}.
$$
For ease of exposition, we drop the subscript 1 on  $\mathbf{W}_1$  in this section. Our bound is based on the following decomposition of  $\widehat{\mathbf{V}}\widehat{\mathbf{S}} -\bar{\mathbf{V}}\bar{\mathbf{S}}\mathbf{W}$
Proposition 1. We have the decomposition
$$
\begin{array}{l} \widehat {\mathbf {V}} \widehat {\mathbf {S}} - \bar {\mathbf {V}} \bar {\mathbf {S}} \mathbf {W} = \bar {\mathbf {V}} \left(\bar {\mathbf {V}} ^ {\top} \widehat {\mathbf {V}} \widehat {\mathbf {S}} - \bar {\mathbf {S}} \mathbf {W}\right) (9) \\ + \left(\mathbf {I} - \bar {\mathbf {V}} \bar {\mathbf {V}} ^ {\top}\right) \bar {\mathbf {A}} ^ {\top} \left(\widehat {\mathbf {U}} - \bar {\mathbf {U}} \mathbf {W}\right) (10) \\ + \left(\mathbf {I} - \bar {\mathbf {V}} \bar {\mathbf {V}} ^ {\top}\right) \left(\hat {\mathbf {A}} - \bar {\mathbf {A}}\right) ^ {\top} \bar {\mathbf {U}} \mathbf {W} (11) \\ + \left(\mathbf {I} - \bar {\mathbf {V}} \bar {\mathbf {V}} ^ {\top}\right) \left(\hat {\boldsymbol {\Lambda}} - \bar {\boldsymbol {\Lambda}}\right) ^ {\top} \left(\hat {\mathbf {U}} - \bar {\mathbf {U}} \mathbf {W}\right). (12) \\ \end{array}
$$
Proof of Proposition 1. We begin by adding and subtracting terms to obtain
$$
\widehat {\mathbf {V}} \widehat {\mathbf {S}} - \bar {\mathbf {V}} \bar {\mathbf {S}} \mathbf {W} = \widehat {\mathbf {V}} \widehat {\mathbf {S}} - \bar {\mathbf {V}} \bar {\mathbf {V}} ^ {\top} \widehat {\mathbf {V}} \widehat {\mathbf {S}} + \underbrace {\bar {\mathbf {V}} (\bar {\mathbf {V}} ^ {\top} \widehat {\mathbf {V}} \widehat {\mathbf {S}} - \bar {\mathbf {S}} \mathbf {W})}.
$$
Then, noting that  $\widehat{\mathbf{V}}\widehat{\mathbf{S}} = \widehat{\mathbf{\Lambda}}^{\top}\widehat{\mathbf{U}}$  and  $(\mathbf{I} - \bar{\mathbf{V}}\bar{\mathbf{V}}^{\top})\bar{\mathbf{\Lambda}}^{\top}\bar{\mathbf{U}} = \mathbf{0}$ , we have
$$
\begin{array}{l} \widehat {\mathbf {V}} \widehat {\mathbf {S}} - \bar {\mathbf {V}} \bar {\mathbf {V}} ^ {\top} \widehat {\mathbf {V}} \widehat {\mathbf {S}} = \widehat {\mathbf {A}} ^ {\top} \widehat {\mathbf {U}} - \bar {\mathbf {V}} \bar {\mathbf {V}} ^ {\top} \widehat {\mathbf {A}} ^ {\top} \widehat {\mathbf {U}} \\ = (\mathbf {I} - \bar {\mathbf {V}} \bar {\mathbf {V}} ^ {\top}) \hat {\mathbf {A}} ^ {\top} \hat {\mathbf {U}} \\ = (\mathbf {I} - \bar {\mathbf {V}} \bar {\mathbf {V}} ^ {\top}) (\hat {\mathbf {A}} - \bar {\mathbf {A}}) ^ {\top} \hat {\mathbf {U}} - (\mathbf {I} - \bar {\mathbf {V}} \bar {\mathbf {V}} ^ {\top}) \bar {\mathbf {A}} ^ {\top} \hat {\mathbf {U}} \\ = (\mathbf {I} - \bar {\mathbf {V}} \bar {\mathbf {V}} ^ {\top}) (\hat {\boldsymbol {\Lambda}} - \bar {\boldsymbol {\Lambda}}) ^ {\top} \hat {\mathbf {U}} - \underbrace {(\mathbf {I} - \bar {\mathbf {V}} \bar {\mathbf {V}} ^ {\top}) \bar {\boldsymbol {\Lambda}} ^ {\top} (\hat {\mathbf {U}} - \bar {\mathbf {U}} \mathbf {W})} _ {(1 0)}. \\ \end{array}
$$
Next, we decompose  $(\mathbf{I} - \bar{\mathbf{V}}\bar{\mathbf{V}}^{\top})(\hat{\boldsymbol{\Lambda}} -\bar{\boldsymbol{\Lambda}})^{\top}\hat{\mathbf{U}}$  by adding and subtracting terms to obtain
$$
(\mathbf {I} - \bar {\mathbf {V}} \bar {\mathbf {V}} ^ {\top}) (\hat {\mathbf {\Lambda}} - \bar {\mathbf {\Lambda}}) ^ {\top} \hat {\mathbf {U}} = \underbrace {(\mathbf {I} - \bar {\mathbf {V}} \bar {\mathbf {V}} ^ {\top}) (\hat {\mathbf {\Lambda}} - \bar {\mathbf {\Lambda}}) ^ {\top} \bar {\mathbf {U}} \mathbf {W}} _ {(1 1)} + \underbrace {(\mathbf {I} - \bar {\mathbf {V}} \bar {\mathbf {V}} ^ {\top}) (\hat {\mathbf {\Lambda}} - \bar {\mathbf {\Lambda}}) ^ {\top} (\hat {\mathbf {U}} - \mathbf {U} \mathbf {W})} _ {(1 2)}.
$$
F.4 Technical propositions
We now outline a series of technical propositions which we require to bound terms (9)-(12) which we prove in Section G.
Our first proposition is a 1-norm and spectral norm bound for  $\widehat{\Lambda}$ .
Proposition 2. The bounds
$$
\left\| \widehat {\boldsymbol {\Lambda}} - \bar {\boldsymbol {\Lambda}} \right\| _ {1} \lesssim \sqrt {M n \lambda_ {\mathrm {m a x}} \log n}, \quad \left\| \widehat {\boldsymbol {\Lambda}} - \bar {\boldsymbol {\Lambda}} \right\| _ {2} \lesssim \sqrt {M n \lambda_ {\mathrm {m a x}}}
$$
hold with overwhelming probability.
The spectral norm bound is obtained using Lemma 5, and the 1-norm bound is obtained via an application of the classical Bernstein inequality. The next proposition provides control on the singular values of  $\widehat{\Lambda}$ .
Proposition 3. Let  $\sigma_{i}(\cdot)$  denote the  $i$ th ordered singular value of a matrix. The singular values of  $\widehat{\Lambda}$  satisfy
$$
\sqrt {M \sigma_ {d} (\pmb {\Sigma})} \lesssim \sigma_ {d} (\widehat {\pmb {\Lambda}}) \leq \sigma_ {1} (\widehat {\pmb {\Lambda}}) \lesssim \sqrt {M \sigma_ {1} (\pmb {\Sigma})}.
$$
The result is obtained using Weyl's inequality. The next proposition provides control of the spectral norm of  $\mathbf{Q}^{\top}(\hat{\boldsymbol{\Lambda}} -\bar{\boldsymbol{\Lambda}})\mathbf{R}$ , where  $\mathbf{Q},\mathbf{R}$  are conformable, deterministic unit-norm matrices.
Proposition 4. For conformable, deterministic unit-norm matrices  $\mathbf{Q},\mathbf{R}$ , the bound
$$
\left\| \mathbf {Q} ^ {\top} (\widehat {\boldsymbol {\Lambda}} - \bar {\boldsymbol {\Lambda}}) \mathbf {R} \right\| _ {2} \lesssim M \log^ {3 / 2} n \tag {13}
$$
holds with overwhelming probability.
The proof of Proposition 4 employs a classical  $\varepsilon$ -net argument to the spectral norm of an appropriately constructed symmetric dilation matrix.
The next proposition states that both the matrices  $\bar{\mathbf{U}}^{\top}\hat{\mathbf{U}}$  and  $\bar{\mathbf{V}}^{\top}\hat{\mathbf{V}}$  are well approximated by a common orthogonal matrix.
Proposition 5. There exists an orthogonal matrix  $\mathbf{W}$  such that
$$
\left\| \bar {\mathbf {U}} ^ {\top} \hat {\mathbf {U}} - \mathbf {W} \right\| _ {2} \lesssim \frac {\sqrt {n \lambda_ {\max}}}{\delta}, \quad \left\| \bar {\mathbf {V}} ^ {\top} \hat {\mathbf {V}} - \mathbf {W} \right\| _ {2} \lesssim \frac {\sqrt {n \lambda_ {\max}}}{\delta}
$$
To prove Proposition 5, we employ the Wedin sinTheta theorem to obtain a bound on  $\| \bar{\mathbf{U}}^{\top}\hat{\mathbf{U}} -\mathbf{W}\|_{2}$ . We then obtain a bound on  $\| \bar{\mathbf{U}}^{\top}\hat{\mathbf{U}} -\bar{\mathbf{V}}^{\top}\hat{\mathbf{V}}\|_{2}$ , and combine these bounds to establish the proposition.
The next technical tool we require is the ability to "swap"  $\mathbf{W}$ ,  $\bar{\mathbf{S}}$  and  $\widehat{\mathbf{S}}$ .
Proposition 6. The bound
$$
\left\| \mathbf {W} \widehat {\mathbf {S}} - \bar {\mathbf {S}} \mathbf {W} \right\| _ {2} \lesssim M \log^ {3 / 2} n
$$
This result follows by applying the previous propositions to an appropriately constructed decomposition.
Part of the challenge of obtaining a good bound on the term (12) is that  $(\tilde{\mathbf{A}} - \tilde{\mathbf{A}})$  and  $(\widehat{\mathbf{U}} - \bar{\mathbf{U}}\mathbf{W})$  are dependent, and this dependence must be decoupled in order to apply the standard suite of matrix perturbation tools. For  $m = 1,\dots ,n$ , let
$$
\mathcal {N} _ {m} = \{(i, j): i = m \text {o r} j \in \{m + (\ell - 1) n, \ell \in [ M ] \} \}
$$
and construct the auxiliary matrices  $\widehat{\Lambda}^{(1)},\ldots ,\widehat{\Lambda}^{(n)}$  defined by
$$
\widehat {\boldsymbol {\Lambda}} _ {i j} ^ {(m)} = \left\{ \begin{array}{l l} \widehat {\boldsymbol {\Lambda}} _ {i j} & \text {i f} (i, j) \notin \mathcal {N} _ {m}, \\ \bar {\boldsymbol {\Lambda}} _ {i j} & \text {i f} (i, j) \in \mathcal {N} _ {m}. \end{array} \right. \tag {14}
$$
In words,  $\widehat{\pmb{\Lambda}}_{ij}^{(m)}$  is the matrix obtained by replacing the  $m$ th row and columns of each of its blocks with its expectation. In this way, the  $m$ th row of  $(\widehat{\pmb{\Lambda}} - \bar{\pmb{\Lambda}})$  and  $\widehat{\pmb{\Lambda}}^{(m)}$  are independent. Let  $\widehat{\mathbf{U}}^{(m)}$  denote the matrix of leading left singular values of  $\widehat{\pmb{\Lambda}}^{(m)}$ .
We apply a result due to [11], which provides  $\ell_{2,\infty}$  control of  $\| \widehat{\mathbf{U}} \|_{2,\infty},\| \widehat{\mathbf{U}}^{(m)}\|_{2,\infty}$ , and  $\| \widehat{\mathbf{U}}^{(m)}\mathbf{W}^{(m)} - \mathbf{U}\|_{2,\infty}$ .
Proposition 7. The bounds
$$
\left\| \widehat {\mathbf {U}} \right\| _ {2, \infty}, \left\| \widehat {\mathbf {U}} ^ {(m)} \right\| _ {2, \infty}, \left\| \widehat {\mathbf {U}} ^ {(m)} \mathbf {W} ^ {(m)} - \mathbf {U} \right\| _ {2, \infty} \lesssim \frac {\mu \lambda_ {\max } \sqrt {d n} \log n}{\delta}
$$
In addition, we require control on the spectral norm difference between the projection matrices  $\widehat{\mathbf{U}}^{(m)}(\widehat{\mathbf{U}}^{(m)})^{\top}$  and the projection matrices  $\bar{\mathbf{U}}\bar{\mathbf{U}}^{\top}$  and  $\widehat{\mathbf{U}}\widehat{\mathbf{U}}^{\top}$ , which is provided in the following proposition.
Proposition 8. The bounds
$$
\left\| \widehat {\mathbf {U}} ^ {(m)} \left(\widehat {\mathbf {U}} ^ {(m)}\right) ^ {\top} - \bar {\mathbf {U}} \bar {\mathbf {U}} ^ {\top} \right\| _ {2} \lesssim \frac {n \lambda_ {\max }}{\delta}, \tag {15}
$$