SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Memory constraint. For streaming PCA without fairness constraints, the main objective could be written as  $\mathrm{tr}(\mathbf{V}^{\intercal}\mathbf{X}^{\intercal}\mathbf{X}\mathbf{V}) = \sum_{i=1}^{N}\mathrm{tr}(\mathbf{V}^{\intercal}\mathbf{x}_i\mathbf{x}_i^{\intercal}\mathbf{V})$ , which is easily amenable to memory-limited algorithm such as noisy power method (Mitliagkas et al., 2013) or stochastic optimization (Oja, 1982). Both approaches utilize the fact that instead of storing  $d\times d$  matrices, one only needs to store matrix-vector product of size  $d\times 1$ , e.g.,  $\mathbf{V}^{\intercal}\mathbf{x}_i$ . This is not the case for the approach of Kleindessner et al. (2023). To see this, consider Eqn. (14) without the covariance constraint, i.e., fair PCA with only the mean constraint. Even here, although the objective can be written as  $\sum_{i=1}^{N}\mathrm{tr}\left(\mathbf{U}^{\intercal}\mathbf{U}_{f}^{\intercal}\mathbf{x}_{i}\mathbf{x}_{i}^{\intercal}\mathbf{U}_{f}\mathbf{U}\right)$ ,
we must know  $U_{f}$  in order to proceed further. Moreover, as  $U_{f}$  is of size  $\mathcal{O}(d^{2})$ , it cannot be stored nor estimated explicitly. When the covariance constraint is also taken into account, although the matrix in question  $M_{f,S'}$  is of dimension  $(d - 1) \times l$ , which is within the memory constraint if  $l = \mathcal{O}(1)$ , the computation of  $M_{f,S'}$  requires the knowledge of  $U_{f}$ . This is because Kleindessner et al. (2023) dealt with the two constraints sequentially (mean first, then covariance), which forced the reparametrization to be done twice and, more importantly, coupling the memory requirement for the computation of  $U_{f}$  and  $M_{f,S'}$ .
Statistical consideration. The statistical guarantee (global convergence, sample complexity) of streaming PCA algorithms is often obtained by appropriately bounding the error term and using proper matrix concentration inequalities (Tropp, 2015). For example, for the sample complexity guarantee of noisy power method (Balcan et al., 2016; Hardt and Price, 2014), as the learner performs power method on the empirical covariance  $\sum_{i} \boldsymbol{x}_{i} \boldsymbol{x}_{i}^{\top}$  instead of the true covariance  $\boldsymbol{\Sigma}$ , the proof proceeds by first showing that the empirical covariance is close enough to the true covariance (i.e., the norm of their error term is sufficiently bounded), which then implies that the variance of the estimation isn't too high. Thus to analyze the error bound of the final iterates  $\mathbf{V}$  when the approach of Kleindessner et al. (2023) is extended to a streaming setting, one would have to bound the estimation error of  $M_{f,S'}^{\top} U_f^{\top} \Sigma U_f M_{f,S'}$ . There are three sources of estimation error:  $M_{f,S'}, U_f^{\top}, \Sigma$ . Recalling that  $M_{f,S'}$  consists of the eigenvectors of  $U_f^{\top} S' U_f$ , one can see that the estimation error of  $M_{f,S'}$  is actually nontrivially dependent on the estimation quality of both  $U$  and  $S'$ . Here, we say nontrivially because the error isn't simply bounded in a linear sense via the usual triangle inequality; it requires rather intricate techniques involving eigenvector perturbation theory (Davis and Kahan, 1969; Golub and Loan, 2013), which may require additional assumptions on eigenvalue gaps. As one can see later in the proof, our approach considers both constraints simultaneously and thus allows for a quite simple theoretical analysis.
E Proofs of Lemma 2, 3, and Theorem 1
E.1 Notations and Assumptions
We first recall the data generation process of our interest. At every time step, a pair of a sensitive attribute and a data point is sampled as  $a \sim \text{Bernoulli}(p)$  and  $\boldsymbol{x}|a \sim \mathcal{D}_a$ , respectively, where  $\mathcal{D}_a$  has mean  $\mu_a$  and covariance  $\Sigma_a$ . This can be written more compactly as  $\boldsymbol{x} \sim \mathcal{D} := p_0\mathcal{D}_0 + p_1\mathcal{D}_1$ , where we denote  $p_0 := 1 - p$  and  $p_1 := p$ .
We recall a set of notation needed for the proof. The mean difference is denoted by  $\pmb{f} = \pmb{\mu}_1 - \pmb{\mu}_0$ , while the second moment difference is  $S = \mathbb{E}[\pmb{x}\pmb{x}^{\top}|a = 1] - E[\pmb{x}\pmb{x}^{\top}|a = 0] = \pmb{\Sigma}_1 - \pmb{\Sigma}_0 + \pmb{\mu}_1\pmb{\mu}_1^{\top} - \pmb{\mu}_0\pmb{\mu}_0^{\top}$ . Let  $P_{m} \in \mathbb{R}^{d\times m}$  be the matrix whose columns are top  $m$  eigenvectors of  $S$  in magnitude of eigenvalues. Note that  $t \in [T]$  is the time variable for Algorithm 3, which we often omit if the context is clear. Moreover,  $b$  (without any sub-/superscript) is the block size of Algorithm 3. The remaining notation is listed below; most of them came from the pseudocode of our algorithms.
- Counts of data points for each sensitive attribute  $a$ :
$$
b _ {t} ^ {(a)} = \sum_ {i = (t - 1) b + 1} ^ {t b} \mathbb {1} _ {[ a _ {i} = a ]}, \qquad B ^ {(a)} = \sum_ {t = 1} ^ {T} b _ {t} ^ {(a)} = \sum_ {i = 1} ^ {T b} \mathbb {1} _ {[ a _ {i} = a ]}
$$
Estimates of group-conditional means:
$$
\overline {{\boldsymbol {m}}} ^ {(a)} = \left\{ \begin{array}{l l} \frac {1}{T B ^ {(a)}} \sum_ {i = 1} ^ {T b} \mathbb {1} _ {[ a _ {i} = a ]} \boldsymbol {x} _ {i} & \text {i f} B ^ {(a)} > 0, \\ \mathbf {0} & \text {i f} B ^ {(a)} = 0 \end{array} \right.
$$
Estimate of the group-conditional mean gap:  $\widehat{f} = \overline{m}^{(1)} - \overline{m}^{(0)}$
- Estimate of the group-conditional second moments:
$$
\boldsymbol {A} _ {t} ^ {(a)} = \left\{ \begin{array}{l l} \frac {1}{b _ {t} ^ {(a)}} \sum_ {i = (t - 1) b + 1} ^ {t b} \mathbb {1} _ {[ a _ {i} = a ]} \boldsymbol {x} _ {i} \boldsymbol {x} _ {i} ^ {\intercal} & \text {i f} b _ {t} ^ {(a)} > 0, \\ \boldsymbol {0} & \text {i f} b _ {t} ^ {(a)} = 0 \end{array} \right.
$$
Estimate of the group-conditional second moment gap:  $\widehat{S}_t = A_t^{(1)} - A_t^{(0)}$
-  $\mathbf{W}_t \in \mathrm{St}(d, n)$  (decision variable of Algorithm 1 and 3)
$C_t^{(a)} = A_t^{(a)}W_{t - 1}$
-  $F_{t} = C_{t}^{(1)} - C_{t}^{(0)} = \widehat{S}_{t}W_{t - 1}$
-  $W_{t} = \mathbb{QR}(F_{t})$  (i.e.,  $F_{t}\stackrel {\mathbb{QR}}{=}W_{t}R_{t},R_{t}\in \mathbb{R}^{m\times m}$  is a lower triangular matrix)
Recall the assumptions on the data distribution, which we will assume throughout the proof:
Assumption 1. Consider our data generation process  $a \sim \text{Bernoulli}(p)$  and  $\boldsymbol{x} \mid a \sim \mathcal{D}_a$ . There exists  $\sigma, V, \mathcal{M}, \mathcal{V}, g_{\min} > 0$ ,  $f_{\max} \in (g_{\min}, \infty)$ , and  $p_{\min} \in (0, 0.5]$  such that the followings hold for any  $(\mathcal{D}_0, \mathcal{D}_1, p) \in \mathcal{F}_d$ : for  $a \in \{0, 1\}$ ,  $\mathcal{D}_a \in \mathrm{nSG}(\sigma)$ ,
$$
\left\| \boldsymbol {g} \right\| _ {2} = \left\| \boldsymbol {\Pi} _ {\boldsymbol {P} _ {m}} ^ {\perp} \boldsymbol {f} \right\| _ {2} \in \{0 \} \cup \left[ g _ {\min }, f _ {\max } \right], \left\| \boldsymbol {f} \right\| _ {2} \leq f _ {\max }, a n d p \in \left[ p _ {\min }, 1 - p _ {\min } \right].
$$
where  $\nu_{1} \geq \dots \geq \nu_{d} \geq 0$  and  $\kappa_{1} \geq \dots \geq \kappa_{d} \geq 0$  are the singular values of  $S$  and  $\Pi_{U}^{\perp}\Sigma\Pi_{U}^{\perp}$ , respectively.
$^4$ $\pmb{y}$  is norm-sub-Gaussian, denoted as  $\mathrm{nSG}(\sigma)$ , if  $\mathbb{P}[\| \pmb {y} - \mathbb{E}\pmb {y}\| \geq t]\leq 2e^{-\frac{t^2}{2\sigma^2}}$ . We refer interested readers to Jin et al. (2019, Section 2) for more discussions on norm-sub-Gaussianity.
We provide some intuition on the assumptions that we impose here. Assumption 1 consists of three parts. The first part, which involves  $\sigma, V, \mathcal{M}$  and  $\nu$ , ensures that the maximum deviation in mean and covariance of each  $\mathcal{D}_a$  are well bounded; this is critical in allowing for us to use proper matrix concentration inequalities (to be described in the next subsection) and has been used in various streaming PCA literature (Bienstock et al., 2022; Huang et al., 2021; Jain et al., 2016). The second part, which involves  $g_{\mathrm{min}}$  and  $f_{\mathrm{max}}$ , imposes a bound on the maximum mean separation,  $\pmb{f} = \pmb{\mu}_1 - \pmb{\mu}_0$ ,  $\ell_2$ -wise and angle-wise, respectively. If the mean difference can be arbitrarily large, then the  $\ell_2$ -estimation error that one has to achieve becomes arbitrarily small; precisely speaking,  $\| \pmb{f} \|_2$  acts as a Lipschitz constant. On the other hand, if the mean difference can be arbitrarily small, then the angle-wise estimation error becomes arbitrarily large. The last part, which involves  $p_{\mathrm{min}}$ , ensures that both groups are selected with some positive, nonvanishing probability. Assumption 2 is standard in streaming PCA literature to ensure convergence; indeed, if the singular value gap is zero, a definitive convergence result can never be obtained, as the ground-truth solution becomes vague.
E.2 Matrix/Vector Concentration Inequalities
Before moving on to our proof, we review some useful concentration inequalities for our theoretical analysis.
Definition 4 (Variance of random matrix). For a zero-mean random matrix  $Z$ , its variance is defined as
$$
\operatorname {V a r} (\boldsymbol {Z}) = \max  \left(\| \mathbb {E} \left[ \boldsymbol {Z} \boldsymbol {Z} ^ {\intercal} \right] \| _ {2}, \| \mathbb {E} \left[ \boldsymbol {Z} ^ {\intercal} \boldsymbol {Z} \right] \| _ {2}\right).
$$
In general,  $\operatorname{Var}(\mathbf{Z}) = \operatorname{Var}(\mathbf{Z} - \mathbb{E}[\mathbf{Z}])$ .
Proposition 1 (Matrix Bernstein inequality (Theorem 6.6.1 of Tropp (2015))). Consider a finite collection  $\{\mathbf{Y}_j\}_{j=1}^b$  of independent matrices with the same size ( $d_1 \times d_2$ ). Suppose they are zero mean and they have uniformly bounded singular values, i.e.,
$$
\mathbb {E} \left[ \boldsymbol {Y} _ {j} \right] = \boldsymbol {0} \quad a n d \quad \mathbb {P} \left(\left\| \boldsymbol {Y} _ {j} \right\| \leq \mathcal {M}\right) = 1 \quad f o r e a c h j \in [ b ].
$$
Let  $\mathcal{V}$  be an upper bound of matrix variance,  $\mathcal{V} \geq \operatorname{Var}(\mathbf{Y}_j)$  for all  $j \in [b]$ . Then, for all  $x \geq 0$ ,
$$
\mathbb {P} \left(\left\| \frac {1}{b} \sum_ {j = 1} ^ {b} \mathbf {Y} _ {j} \right\| \geq x\right) \leq \left(d _ {1} + d _ {2}\right) \exp \left(\frac {- b x ^ {2}}{2 (\mathcal {V} + \mathcal {M} x / 3)}\right).
$$
In particular, if  $0 \leq x \leq \frac{3\mathcal{V}}{\mathcal{M}}$ ,
$$
\mathbb {P} \left(\left\| \frac {1}{b} \sum_ {j = 1} ^ {b} \boldsymbol {Y} _ {j} \right\| \geq x\right) \leq \left(d _ {1} + d _ {2}\right) \exp \left(\frac {- b x ^ {2}}{4 \mathcal {V}}\right).
$$
Proposition 2 (Vector Hoeffding inequality (Corollary 7 of Jin et al. (2019))). There exists an absolute constant  $c$  such that if  $\pmb{y}_1, \dots, \pmb{y}_b$  be independent random vectors with common dimension  $d$ , and assume the following:
$$
\mathbb {E} [ \boldsymbol {y} _ {i} ] = 0, \quad \boldsymbol {y} _ {i} \in \mathrm {n S G} (\sigma).
$$
Then, for any  $x\geq 0$
$$
\mathbb {P} \left(\left\| \frac {1}{b} \sum_ {j = 1} ^ {b} \boldsymbol {y} _ {j} \right\| _ {2} \geq x\right) \leq 2 d \exp \left(- \frac {b x ^ {2}}{\mathfrak {c} ^ {2} \sigma^ {2}}\right).
$$
Remark 2. With an additional assumption that the random variables are bounded, we can also consider using vector Bernstein inequality (Lemma 18 of Kohler and Lucchi (2017)), which removes the factor of dimension  $d$  from the RHS of the concentration inequality.
E.3 Proof of Lemma 2 - Bounding Error in Second Moment Gap
Recall that Algorithm 3 performs NPM on the second moment difference matrix  $S$  so that an iteration can be written as
$$
\boldsymbol {W} _ {t} = \mathrm {Q R} (\widehat {\boldsymbol {S}} _ {t} \boldsymbol {W} _ {t - 1}) = \mathrm {Q R} (\boldsymbol {S} \boldsymbol {W} _ {t - 1} + \boldsymbol {Z} _ {t, 1}),
$$
where  $\mathbf{Z}_{t,1} = (\widehat{\mathbf{S}}_t - \mathbf{S})\mathbf{W}_{t - 1}$  is the noise matrix.
The lemma below provides a general statement on sufficient size  $b$  of blocks to bound the second moment difference error  $\widehat{S}_t - S$  multiplied by some fixed matrices.
Lemma 4. Let any  $\delta >0$  and matrices  $M\in \mathbb{R}^{m\times d}$  and  $N\in \mathbb{R}^{d\times n}$  be given. If we choose  $b\geq \frac{4}{p_{\min}}\max \left(1,\frac{8\mathcal{M}^2}{9\mathcal{V}}\right)\log \frac{2(m + n)}{\delta}$ , then with probability at least  $1 - \delta$ ,
$$
\left\| \boldsymbol {M} (\widehat {\boldsymbol {S}} _ {t} - \boldsymbol {S}) \boldsymbol {N} \right\| _ {2} \leq \mathcal {E} _ {m + n} ^ {(\mathrm {S})} \triangleq \sqrt {\frac {3 2 \mathcal {V}}{b p _ {\mathrm {m i n}}} \log \frac {2 (m + n)}{\delta}}.
$$
We now proceed the proof of Lemma 2. To this end, recall that  $\mathbf{Z}_{t,1} = (\widehat{\mathbf{S}}_t - \mathbf{S})\mathbf{W}_{t - 1}$ . Firstly, applying Lemma 4 with  $M = I_d$  and  $N = W_{t - 1}\in \mathbb{R}^{d\times m}$ , a sufficient condition for having  $5\| Z_{t,1}\| _2\leq \epsilon \Delta_{m,\nu}$  with probability at least  $1 - \delta$  is that
$$
b \geq \frac {4 0 0 \mathcal {V}}{\epsilon^ {2} \Delta_ {m , \nu} ^ {2} p _ {\mathrm {m i n}}} \log \frac {2 (d + m)}{\delta} + \frac {4}{p _ {\mathrm {m i n}}} \max \left(1, \frac {8 \mathcal {M} ^ {2}}{9 \mathcal {V}}\right) \log \frac {2 (d + m)}{\delta}.
$$
Secondly, applying Lemma 4 with  $M = P_{m}^{\intercal} \in \mathbb{R}^{m \times d}$  and  $N = W_{t-1} \in \mathbb{R}^{d \times m}$ , a sufficient condition for having  $5\|P_{m}^{\intercal}\boldsymbol{Z}_{t,1}\|_{2} \leq \frac{\delta\Delta_{m,\nu}}{2\sqrt{dm}}$  with probability at least  $1 - \delta$  is that
$$
b \geq \frac {3 2 0 0 d m \mathcal {V}}{\delta^ {2} \Delta_ {m , \nu} ^ {2} p _ {\mathrm {m i n}}} \log \frac {4 m}{\delta} + \frac {4}{p _ {\mathrm {m i n}}} \max \left(1, \frac {8 \mathcal {M} ^ {2}}{9 \mathcal {V}}\right) \log \frac {4 m}{\delta}.
$$
Combining all bounds for  $b$ , we conclude that a sufficient block size  $b$  is
$$
b = \Omega \left(\frac {\mathcal {V}}{\Delta_ {m , \nu} ^ {2} p _ {\mathrm {m i n}}} \left(\frac {d m}{\delta^ {2}} \log \frac {m}{\delta} + \frac {1}{\epsilon^ {2}} \log \frac {d}{\delta}\right) + \frac {\mathcal {M} ^ {2}}{\mathcal {V} p _ {\mathrm {m i n}}} \log \frac {d}{\delta}\right).
$$
Proof of Lemma 4. Note that  $(a_{i},\pmb{x}_{i})$  's are i.i.d. samples  $(i\in \{(t - 1)b + 1,\dots ,tb\})$  . Consider independent random matrices
$$
\boldsymbol {Y} _ {i} ^ {(a)} = \boldsymbol {M} \left(\boldsymbol {x} _ {i} \boldsymbol {x} _ {i} ^ {\intercal} - \left(\boldsymbol {\Sigma} _ {a} + \boldsymbol {\mu} _ {a} \boldsymbol {\mu} _ {a} ^ {\intercal}\right)\right) \boldsymbol {N}.
$$
For each set  $A \subseteq [b]$ , define an event  $E_A := \{a_i = \mathbb{1}_{[i \in A]} \forall i \in [b]\}$ . Note that  $\mathbb{P}[E_A] = p_0^{b - |A|} p_1^{|A|}$ . To exploit this, we first apply a peeling argument as follows: for any  $c_0, c_1 \in (0, 1)$  with  $c_0 + c_1 = 1$ ,
$$
\begin{array}{l} \mathbb {P} \left(\left\| \boldsymbol {M} (\widehat {\boldsymbol {S}} _ {t} - \boldsymbol {S}) \boldsymbol {N} \right\| _ {2} \geq x\right) \\ = \sum_ {r = 0} ^ {b} \sum_ {A \in \binom {[ b ]} {r}} \mathbb {P} \left(\left\| M (\widehat {\boldsymbol {S}} _ {t} - \boldsymbol {S}) \boldsymbol {N} \right\| _ {2} \geq x \mid E _ {A}\right) \mathbb {P} (E _ {A}) \\ = \sum_ {r = 1} ^ {b - 1} \sum_ {A \in \left(\begin{array}{c}[ b ]\\r\end{array}\right)} \mathbb {P} \left( \right.\left\| \frac {1}{r} \sum_ {i \in A} \mathbf {Y} _ {i} ^ {(1)} - \frac {1}{b - r} \sum_ {i \in A ^ {c}} \mathbf {Y} _ {i} ^ {(0)} \right\| _ {2} \geq x \left. \right\rvert   E _ {A}\left. \right) \mathbb {P} \left(E _ {A}\right) \\ + \mathbb {P} \left(\left\| \frac {1}{b} \sum_ {i = 1} ^ {b} \boldsymbol {Y} _ {i} ^ {(0)} \right\| _ {2} \geq x \mid E _ {\emptyset}\right) \mathbb {P} (E _ {\emptyset}) + \mathbb {P} \left(\left\| \frac {1}{b} \sum_ {i = 1} ^ {b} \boldsymbol {Y} _ {i} ^ {(1)} \right\| _ {2} \geq x \mid E _ {[ b ]}\right) \mathbb {P} (E _ {[ b ]}) \\ \leq \sum_{r = 1}^{b - 1}\sum_{A\in \binom {[b]}{r}}\left\{\mathbb{P}\left(\left\| \frac{1}{b - r}\sum_{i\in A^{\mathfrak{c}}} \boldsymbol{Y}_{i}^{(0)}\right\|_{2}\geq c_{0}x\bigg|E_{A}\right) + \mathbb{P}\left(\left\| \frac{1}{r}\sum_{i\in A}\boldsymbol{Y}_{i}^{(1)}\right\|_{2}\geq c_{1}x\bigg|E_{A}\right)\right\} p_{0}^{b - r}p_{1}^{r} \\ + \mathbb {P} \left(\left\| \frac {1}{b} \sum_ {i = 1} ^ {b} \boldsymbol {Y} _ {i} ^ {(0)} \right\| _ {2} \geq c _ {0} x \Bigg | E _ {\emptyset}\right) p _ {0} ^ {b} + \mathbb {P} \left(\left\| \frac {1}{b} \sum_ {i = 1} ^ {b} \boldsymbol {Y} _ {i} ^ {(1)} \right\| _ {2} \geq c _ {1} x \Bigg | E _ {[ b ]}\right) p _ {1} ^ {b}, \\ \end{array}
$$
where the last inequality holds for the following fact: if  $\| \pmb{a} - \pmb{b}\| \geq x$  then  $\| \pmb{a}\| \geq \lambda x$  or  $\| \pmb{b}\| \geq (1 - \lambda)x$  for any  $\lambda \in (0,1)$ .
Now, we make the crucial observation that conditioned on the event  $E_{A}$ , both  $\left\{\mathbf{Y}_i^{(1)}\right\}_{i\in A}$  and  $\left\{\mathbf{Y}_i^{(0)}\right\}_{i\in A^c}$  are sets of i.i.d. random zero-mean  $m\times n$  matrices. In addition, with Assumption 1, we have that for each  $i\in A$
$$
\mathbb {P} \left[ \left| \left| \mathbf {Y} _ {i} ^ {(1)} \right| \right| _ {2} \leq \mathcal {M} \Big | i \in A \right] = 1, \quad \operatorname {V a r} \left(\mathbf {Y} _ {i} ^ {(1)} \Big | i \in A\right) \leq \mathcal {V}.
$$
(and analogously for each  $i \in A^{\mathsf{c}}$ ). Thus, applying the matrix Bernstein inequality<sup>5</sup> (Proposition 1), we obtain the following: when  $0 \leq x \leq \frac{3\mathcal{V}}{\mathcal{M}}$ ,
$$
\begin{array}{l} \mathbb {P} \left(\left\| (\widehat {\boldsymbol {S}} _ {t} - \boldsymbol {S}) \boldsymbol {W} _ {t - 1} \right\| _ {2} \geq x\right) \\ \leq (m + n)\sum_{r = 1}^{b - 1}p_{0}^{b - r}p_{1}^{r}\sum_{A\in \binom {[b]}{r}}\left\{\exp \left(-\frac{(b - r)c_{0}^{2}x^{2}}{4\mathcal{V}}\right) + \exp \left(-\frac{rc_{1}^{2}x^{2}}{4\mathcal{V}}\right)\right\} \\ + (m + n) p _ {0} ^ {b} \exp \left(- \frac {b c _ {0} ^ {2} x ^ {2}}{4 \mathcal {V}}\right) + (m + n) p _ {1} ^ {b} \exp \left(- \frac {b c _ {1} ^ {2} x ^ {2}}{4 \mathcal {V}}\right) \\ \leq (m + n)\sum_{r = 0}^{b}\binom {b}{r}p_{0}^{b - r}p_{1}^{r}\left\{\exp \left(-\frac{(b - r)c_{0}^{2}x^{2}}{4\mathcal{V}}\right) + \exp \left(-\frac{rc_{1}^{2}x^{2}}{4\mathcal{V}}\right)\right\} \\ = (m + n) \left\{\left(p _ {0} + p _ {1} \exp \left(- \frac {c _ {1} ^ {2} x ^ {2}}{4 \mathcal {V}}\right)\right) ^ {b} + \left(p _ {1} + p _ {0} \exp \left(- \frac {c _ {0} ^ {2} x ^ {2}}{4 \mathcal {V}}\right)\right) ^ {b} \right\} <   \delta . \\ \end{array}
$$
For this to occur, it suffices for both terms to be bounded by  $\frac{\delta}{2}$ . Let us consider only the first term, as the second term follows from symmetry. Taking the log, we have
$$
\log \left(\left(1 - p _ {1}\right) + p _ {1} \exp \left(- \frac {c _ {1} ^ {2} x ^ {2}}{4 \mathcal {V}}\right)\right) \leq \frac {1}{b} \log \frac {\delta}{2 (m + n)}.
$$
Using  $\log (1 + y)\leq y$  , it suffices to have
$$
\frac {1}{b} \log \frac {2 (m + n)}{\delta} \leq p _ {1} \left(1 - \exp \left(- \frac {c _ {1} ^ {2} x ^ {2}}{4 \mathcal {V}}\right)\right).
$$
Using  $y \leq 1 - e^{-2y}$  for  $y \in [0,1/2]$ , it now suffices to have
$$
\frac {8 \mathcal {V}}{b p _ {1} c _ {1} ^ {2}} \log \frac {2 (m + n)}{\delta} \leq x ^ {2} \leq \min  \left(\frac {2 \mathcal {V}}{c _ {1} ^ {2}}, \frac {9 \mathcal {V} ^ {2}}{\mathcal {M} ^ {2}}\right).
$$
Combining this with the other term and, for simplicity choosing  $c_{0} = c_{1} = 1 / 2$ , we have our desired statement.
E.4 Proof of Lemma 3 - Bounding the Final Error
Recall that the iterates are  $V_{\tau +1} = \mathbb{QR}(\Pi_{\widehat{U}}^{\perp}\widehat{\Sigma}_{\tau}\Pi_{\widehat{U}}^{\perp}V_{\tau})$  , where  $\widehat{U}$  is the output of Algorithm 3,  $\Pi_{\widehat{U}}^{\perp}:= I_d - \widehat{U}\widehat{U}^{\intercal}$  , and  $\widehat{\Sigma}_{\tau}:= \frac{1}{B}\sum_{j = (\tau -1)\mathcal{B} + 1}^{\tau \mathcal{B}}\tilde{x}_j\tilde{x}_j^\intercal$  is the sample covariance at time step  $\tau$  of Algorithm 4. This can be rewritten as
$$
\boldsymbol {V} _ {\tau + 1} = \mathbb {Q R} \left(\boldsymbol {\Pi} _ {\boldsymbol {U}} ^ {\perp} \boldsymbol {\Sigma} \boldsymbol {\Pi} _ {\boldsymbol {U}} ^ {\perp} \boldsymbol {V} _ {\tau} + \boldsymbol {Z} _ {\tau , 2}\right),
$$
where  $Z_{\tau,2} = (\Pi_{\widehat{U}}^{\perp}\widehat{\Sigma}_{\tau}\Pi_{\widehat{U}}^{\perp} - \Pi_{\widehat{U}}^{\perp}\Sigma\Pi_{\widehat{U}}^{\perp})V_{\tau}$  is the noise matrix and  $\boldsymbol{\Sigma} = \sum_{a\in \{0,1\}}p_a(\boldsymbol{\Sigma}_a + \boldsymbol{\mu}_a\boldsymbol{\mu}_a^\intercal)$ . By Assumption 1, we have that  $\| \boldsymbol{\Sigma}\|_2 \leq V$ .
The lemma below provides a general statement on a sufficient block size  $\mathcal{B}$  to bound the covariance error  $\widehat{\Sigma} - \Sigma$  multiplied by some fixed matrices.
Lemma 5. Let any  $\delta >0$  and matrices  $M\in \mathbb{R}^{m\times d}$  and  $N\in \mathbb{R}^{d\times n}$  be given. If we choose  $\mathcal{B}\geq \frac{4}{9}\frac{\mathcal{M}^2}{\mathcal{V} + V^2}\log \frac{4(m + n)}{\delta}$ , then the following holds with probability at least  $1 - \delta$ :
$$
\left\| M (\widehat {\boldsymbol {\Sigma}} - \boldsymbol {\Sigma}) \boldsymbol {N} \right\| _ {2} \leq \mathcal {E} _ {m + n} ^ {(\Sigma)} \triangleq \sqrt {\frac {4 (\mathcal {V} + V ^ {2})}{\mathcal {B}} \log \frac {4 (m + n)}{\delta}}.
$$
Precisely, we use its conditional version where the means and the variances are replaced with their conditional counterparts.
One could try to optimize for  $c_{0}, c_{1}$  to obtain an "optimal" sample complexities. However, from some preliminary computations, this seems to be not worth pursuing, and we conjecture that it would yield the same asymptotic dependency as when  $c_{0} = c_{1} = 1/2$ .
We now proceed the proof of Lemma 3. Fix some  $V \in \mathrm{St}(d, k)$ , and from the design of our Algorithm 4,  $\widehat{U}$  is also fixed. By our given assumption on upper bound of  $\left\| \Pi_{\widehat{U}} - \Pi_U \right\|_2$ , and the fact that  $\| V \|_2 = \| Q_k \|_2 = 1$ , we have
$$
\begin{array}{l} \| \boldsymbol {Z} _ {\tau , 2} \| _ {2} \leq \left\| \boldsymbol {\Pi} _ {\hat {\boldsymbol {U}}} ^ {\perp} (\hat {\boldsymbol {\Sigma}} - \boldsymbol {\Sigma}) \boldsymbol {\Pi} _ {\hat {\boldsymbol {U}}} ^ {\perp} \boldsymbol {V} \right\| _ {2} + 2 \| \boldsymbol {\Sigma} \| _ {2} \| \boldsymbol {\Pi} _ {\hat {\boldsymbol {U}}} - \boldsymbol {\Pi} _ {\boldsymbol {U}} \| _ {2} \\ \leq \left\| \boldsymbol {\Pi} _ {\hat {U}} ^ {\perp} (\hat {\boldsymbol {\Sigma}} - \boldsymbol {\Sigma}) \boldsymbol {\Pi} _ {\hat {U}} ^ {\perp} \boldsymbol {V} \right\| _ {2} + \frac {\epsilon \Delta_ {k , \kappa}}{1 0} \\ \end{array}
$$
and, with a similar logic,
$$
\left\| \boldsymbol {Q} _ {k} ^ {\intercal} \boldsymbol {Z} _ {\tau , 2} \right\| _ {2} \leq \left\| \boldsymbol {Q} _ {k} ^ {\intercal} \boldsymbol {\Pi} _ {\widehat {\boldsymbol {U}}} ^ {\perp} (\widehat {\boldsymbol {\Sigma}} - \boldsymbol {\Sigma}) \boldsymbol {\Pi} _ {\widehat {\boldsymbol {U}}} ^ {\perp} \boldsymbol {V} \right\| _ {2} + \frac {\delta \Delta_ {k , \kappa}}{2 0 \sqrt {d k}}.
$$
Applying Lemma 5 with  $M = \Pi_{\widehat{U}}^{\perp} \in \mathbb{R}^{d \times d}$  and  $N = \Pi_{\widehat{U}}^{\perp} V \in \mathbb{R}^{d \times k}$ , a sufficient condition for having  $\left\| \Pi_{\widehat{U}}^{\perp} (\widehat{\Sigma} - \Sigma) \Pi_{\widehat{U}}^{\perp} V \right\|_2 \leq \frac{\epsilon \Delta_{k,\kappa}}{10}$  with probability at least  $1 - \delta$  is that
$$
\mathcal {B} \geq \frac {4 0 0 (\mathcal {V} + V ^ {2})}{\epsilon^ {2} \Delta_ {k , \kappa} ^ {2}} \log \frac {4 (d + k)}{\delta} + \frac {4}{9} \frac {\mathcal {M} ^ {2}}{\mathcal {V} + V ^ {2}} \log \frac {4 (d + k)}{\delta}.
$$