SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Description of the algorithms. The pseudocode of our algorithm is shown in Algorithms 1 and 2. The goal of Algorithm 1 is to estimate  $U = \left[ P_{m} \mid \frac{g}{\|g\|_{2}} \right]$  in Eqn. (2) as accurately as possible. Lines 5-13 do the estimation of  $P_{m}$  and  $g = \Pi_{P_{m}}^{\perp} f$ ; line 11 is the NPM to find  $P_{m}$ , lines 12 and 13 are the estimation of  $f$  and  $g$  respectively, and line 17 is the concatenation of the estimates of  $P_{m}$  and  $g / \| g \|_{2}$ . Especially at line 17, the algorithm determines whether to incorporate mean difference by checking  $\hat{g}$ , which can be proved to be correct, i.e.,  $g = 0$  if and only if  $\hat{g} = 0$  with high probability. With the estimated  $U$  from Algorithm 1, Algorithm 2 performs the usual NPM on  $\Pi_{U}^{\perp} \Sigma \Pi_{U}^{\perp}$ , as in Eqn. (4). The memory complexity of Algorithm 2 is  $O(d \max(m, k))$ , since we do not have to store all  $b$  or  $B$  data points at each time, and all the operations used can be implemented in a manner that conforms to the memory limitation; the full pseudocodes are provided in Appendix B.
At time step  $t$  of Algorithm 1, for each  $a \in \{0,1\}$ ,  $b_{t}^{(a)}$  is the number of data points  $x_{i}$ 's such that  $a_{i} = a$ ;  $m_{t}^{(a)}$  is the term used for estimation of the group-wise sample mean of  $x_{i}$ 's;  $C_{t}^{(a)}$  is used for the group-wise sample second moment. Their forms are as follows and can be computed
incrementally in the streaming setup:
$$
b _ {t} ^ {(a)} = \sum_ {i = b (t - 1) + 1} ^ {b t} \mathbb {1} _ {[ a _ {i} = a ]}, \quad \boldsymbol {m} _ {t} ^ {(a)} = \sum_ {i = b (t - 1) + 1} ^ {b t} \frac {\mathbb {1} _ {[ a _ {i} = a ]}}{b _ {t} ^ {(a)}} \boldsymbol {x} _ {i}, \quad C _ {t} ^ {(a)} = \sum_ {i = b (t - 1) + 1} ^ {b t} \frac {\mathbb {1} _ {[ a _ {i} = a ]}}{b _ {t} ^ {(a)}} \boldsymbol {x} _ {i} \boldsymbol {x} _ {i} ^ {\intercal} \boldsymbol {W} _ {t - 1}, \tag {6}
$$
where we set the last two quantities to  $\mathbf{0}$  when  $b_{t}^{(a)} = 0$ .
Note that as  $b_{t}^{(a)}$  itself is random, this presents some technical challenges in the proofs of the theoretical guarantees. For instance, the above estimators for the mean and covariance are biased. Still, by properly using peeling argument and matrix concentration inequalities as well as perturbation theories (Golub and Loan, 2013; Tropp, 2015), we could sufficiently bound their errors. Informally speaking, we show that line 11 corresponds to the noisy power method for the matrix  $S$ , which incurs the memory complexity  $\mathcal{O}(dm)$ .
5.2 Previous Approaches are not Suitable for Streaming Setup
All the existing approaches to fair PCA require the full knowledge of  $\pmb{\mu}_1 - \pmb{\mu}_0$  and  $\pmb{\Sigma}_1 - \pmb{\Sigma}_0$ , or even the full data matrix  $\pmb{X}$ . Olfat and Aswani (2019) need  $\pmb{f}$  and  $\pmb{S}$  to formulate the convex matrix constraints for their semi-positive definite programming (SDP), which is then solved with commercial SDP solver; Lee et al. (2022) need  $\pmb{X}$  to compute the derivative of their maximum mean discrepancy (MMD) penalty term, which requires  $\mathcal{O}(d^2)$  memory to compute kernel Gram matrices. One may hope that the PCA-type approach taken by Kleindessner et al. (2023) may be easily extendable to our streaming setting by just using the standard techniques (e.g., using matrix-vector products) from streaming PCA (Mitliagkas et al., 2013). Indeed, they also propose a similar relaxation of the covariance constraint, leading to a closed-form solution. However, their formulation is not memory-efficient and, more importantly, is not as applicable to streaming settings as our formulation; see Appendix D for more discussions.
5.3 Extension to Multiple Sensitive Groups/Attributes
So far, we have considered a single and binary-sensitive attribute for simplicity. Here, we briefly discuss extending our framework to multiple non-binary sensitive attributes. Suppose that there are  $\ell$  different attributes (e.g., gender and race), and for each  $r\in [\ell ]$ , the  $r$ -th sensitive attribute has  $g_{r}$  groups (e.g., male, female, and non-binary:  $g_{\mathrm{gender}} = 3$  in this case). We first describe the new data generation process: at each time  $t$ , we obtain  $((a_{1,t},\dots ,a_{\ell ,t}),\boldsymbol {x}_t)$ , where for each  $r$ ,  $a_{r,t}\sim \mathrm{Categorical}(p_{r,1},\dots ,p_{r,g_r})$  with  $\sum_{a = 1}^{g_r}p_{r,a} = 1$ . Then, our formulation of fair PCA easily extends via a one-versus-all comparison approach. That is to say, for each  $r\in [\ell ]$ , define
$$
\boldsymbol {f} _ {r, a} = \mathbb {E} [ \boldsymbol {x} \mid a _ {r} = a ] - \mathbb {E} [ \boldsymbol {x} \mid a _ {r} \neq a ], \quad \boldsymbol {S} _ {r, a} = \mathbb {E} [ \boldsymbol {x x ^ {\intercal}} \mid a _ {r} = a ] - \mathbb {E} [ \boldsymbol {x x ^ {\intercal}} \mid a _ {r} \neq a ].
$$
Denote the top- $m_r$  column basis of eigenspace of  $\pmb{S}_{r,a}$  by  $P_{m_r}(S_{r,a})$ . Then our notion of fair PCA can be extended as follows: where  $d,k,m_1,\dots ,m_n$  are given as  $d > k + \ell +\sum_{r}m_{r}$
$$
\max  _ {\boldsymbol {V} \in \operatorname {S t} (d, k)} \operatorname {t r} \left(\boldsymbol {V} ^ {\intercal} \boldsymbol {\Sigma} \boldsymbol {V}\right), \quad \text {s u b j e c t t o} \operatorname {c o l} (\boldsymbol {V}) \subset \bigcap_ {r \in [ \ell ]} \bigcap_ {a \in [ g _ {r} ]} \operatorname {c o l} \left(\left[ \boldsymbol {P} _ {m _ {r}} \left(\boldsymbol {S} _ {r, a}\right) \mid \boldsymbol {f} _ {r, a} \right]\right) ^ {\perp}. \tag {7}
$$
Our algorithm can also be naturally adapted to this scenario via one-versus-all manner. In Appendix C, we provide the full pseudocode for the general case of multiple and non-binary sensitive attributes.
6 FNPM is a PAFO-Learning Algorithm
We now show that our proposed memory-efficient algorithm, FNPM, is actually a PAFO-learning algorithm in that with a certain sample complexity, it satisfies the definition of PAFO-learnability. The proofs of all the theoretical results stated here are deferred to Appendix E.
To use proper matrix concentration inequalities for our error term analysis, various streaming PCA literature adopt some assumption on the underlying data distribution, e.g., sub-Gaussianity (Bienstock et al., 2022; Jain et al., 2016; Yang et al., 2018). Throughout the paper and for our theoretical discussions, we consider a collection of distributions  $\mathcal{F}_d\subset \mathcal{P}_d\times \mathcal{P}_d\times (0,1)$  satisfying the following assumptions to our data generation process in terms of the data point conditioned on the sensitive attribute and its eigenspectrum:
Assumption 1. Consider our data generation process  $a \sim \text{Bernoulli}(p)$  and  $\mathbf{x} \mid a \sim \mathcal{D}_a$ . There exists  $\sigma, V, \mathcal{M}, \mathcal{V}, g_{\min} > 0$ ,  $f_{\max} \in (g_{\min}, \infty)$ , and  $p_{\min} \in (0, 0.5]$  such that the followings hold for any  $(\mathcal{D}_0, \mathcal{D}_1, p) \in \mathcal{F}_d$ : for  $a \in \{0, 1\}$ ,  $\mathcal{D}_a \in \mathrm{nSG}(\sigma)$ ,
$$
\mathbb {P} \left[ \| \boldsymbol {x x} ^ {\intercal} - \left(\boldsymbol {\Sigma} _ {a} + \boldsymbol {\mu} _ {a} \boldsymbol {\mu} _ {a} ^ {\intercal}\right) \| _ {2} \leq \mathcal {M} \mid a \right] = 1, \| \boldsymbol {\Sigma} _ {a} + \boldsymbol {\mu} _ {a} \boldsymbol {\mu} _ {a} ^ {\intercal} \| _ {2} \leq V, \operatorname {V a r} \left(\boldsymbol {x x} ^ {\intercal} \mid a\right) \leq \mathcal {V},
$$
$$
\| \boldsymbol {g} \| _ {2} = \left\| \Pi_ {\boldsymbol {P} _ {m}} ^ {\perp} \boldsymbol {f} \right\| _ {2} \in \{0 \} \cup \left[ g _ {\min }, f _ {\max } \right], \| \boldsymbol {f} \| _ {2} \leq f _ {\max }, a n d p \in \left[ p _ {\min }, 1 - p _ {\min } \right].
$$
Assumption 2. Fix  $m, k \in \mathbb{N}$ . There exist  $\Delta_{m,\nu}, \Delta_{k,\kappa}, K_{m,\nu}, K_{k,\kappa} \in (0,\infty)$  such that for any  $(\mathcal{D}_0, \mathcal{D}_1, \cdot) \in \mathcal{F}_d$ , the followings hold:
$$
\nu_ {m} - \nu_ {m + 1} \geq \Delta_ {m, \nu}, \kappa_ {k} - \kappa_ {k + 1} \geq \Delta_ {k, \kappa}, \nu_ {m} \leq K _ {m, \nu}, a n d \kappa_ {k} \leq K _ {k, \kappa},
$$
where  $\nu_{1} \geq \dots \geq \nu_{d} \geq 0$  and  $\kappa_{1} \geq \dots \geq \kappa_{d} \geq 0$  are the singular values of  $S$  and  $\Pi_{\overline{U}}^{\perp}\Sigma\Pi_{\overline{U}}^{\perp}$ , respectively.
We start by establishing the sample complexity bounds of Algorithms 1 and 2 based on the convergence bound for NPM by Hardt and Price (2014). Recall that NPM is an algorithm for finding top- $r$  eigenspace (in magnitude) of a symmetric (but not necessarily PSD) matrix  $\mathbf{A}$  under a random noise  $Z \in \mathbb{R}^{d \times k}$ , by update  $\mathbf{V}_t \gets \mathbb{QR}(\mathbf{A}\mathbf{V}_{t-1} + \mathbf{Z}_t)$ . We start by recalling their meta-sample complexity result for NPM, which we have slightly reformulated for our convenience:
Lemma 1 (Corollary 1.1 of Hardt and Price (2014)). Let  $1 \leq r < d$ ,  $\epsilon \in (0,1/2)$  and  $\delta \in (0,2e^{-cd})$  where  $c$  is an absolute constant. Let  $\mathbf{L}_r$  be the  $d \times r$  matrix, whose columns correspond to the top- $r$  eigenvectors (in magnitude) of the symmetric (not necessarily PSD) matrix  $\mathbf{A} \in \mathbb{R}^{d \times d}$ , and let  $\xi_1 \geq \dots \geq \xi_d \geq 0$  be the singular values of  $\mathbf{A}$ . Assume that the noise matrices  $\mathbf{Z}_t \in \mathbb{R}^{d \times r}$  satisfy
$$
5 \left\| \boldsymbol {Z} _ {t} \right\| _ {2} \leq \epsilon \left(\xi_ {r} - \xi_ {r + 1}\right) \quad a n d \quad 5 \left\| \boldsymbol {L} _ {r} ^ {\top} \boldsymbol {Z} _ {t} \right\| _ {2} \leq \frac {\delta \left(\xi_ {r} - \xi_ {r + 1}\right)}{2 \sqrt {d r}}, \quad \forall t \geq 1. \tag {8}
$$
Then, after  $T = \Theta\left(\frac{\xi_r}{\xi_r - \xi_{r + 1}}\log \left(\frac{d}{\epsilon\delta}\right)\right)$  steps of NPM,  $\left\| \Pi_{V_T}^\perp L_r\right\| _2\leq \epsilon$  with probability at least  $1 - \delta$
First, we prove that the  $W_{T}$  resulting from Algorithm 1 (NPM for the second moment gap) converges to the true value. We can view Algorithm 1's updates as  $\boldsymbol{W}_{t} \gets \mathbb{QR}(\boldsymbol{S}\boldsymbol{W}_{t-1} + \boldsymbol{Z}_{t,1})$ , where the noise matrix in this case is  $\boldsymbol{Z}_{t,1} := (\boldsymbol{C}_{t}^{(1)} - \boldsymbol{C}_{t}^{(0)}) - \boldsymbol{S}\boldsymbol{W}_{t-1}$  and  $\boldsymbol{C}_{t}^{(a)}$  is as defined in Eqn. (6). The following lemma asserts that with large enough block size  $b$ , the error matrices are sufficiently bounded such that the NPM iterates converge:
Lemma 2. Let  $\epsilon, \delta \in (0,1)$ . It is sufficient to choose the block size  $b$  in Algorithm 1 as
$$
b = \Omega \left(\frac {\mathcal {V}}{\Delta_ {m , \nu} ^ {2} p _ {\operatorname* {m i n}}} \left(\frac {d m}{\delta^ {2}} \log \frac {m}{\delta} + \frac {1}{\epsilon^ {2}} \log \frac {d}{\delta}\right) + \frac {\mathcal {M} ^ {2}}{\mathcal {V} p _ {\operatorname* {m i n}}} \log \frac {d}{\delta}\right) \tag {9}
$$
to make the following hold with probability at least  $1 - \delta$ :
$$
5 \left\| \boldsymbol {Z} _ {t, 1} \right\| _ {2} \leq \epsilon \Delta_ {m, \nu} \quad a n d \quad 5 \left\| \boldsymbol {P} _ {m} ^ {\intercal} \boldsymbol {Z} _ {t, 1} \right\| _ {2} \leq \frac {\delta \Delta_ {m , \nu}}{2 \sqrt {d m}}, \quad \forall t \geq 1, \tag {10}
$$
where we recall that the columns of  $\pmb{P}_m$  are the top- $m$  (in magnitude) orthonormal eigenvectors of  $\pmb{S}$ .
Let  $\widehat{\pmb{U}}$  be the final estimate of the true  $\pmb{U}$  outputted by Algorithm 1. For Algorithm 2, the noise matrix is  $Z_{\tau,2} \coloneqq \left( \Pi_{\widehat{\pmb{U}}}^{\perp} \widehat{\Sigma}_{\tau} \Pi_{\widehat{\pmb{U}}}^{\perp} - \Pi_{\pmb{U}}^{\perp} \Sigma \Pi_{\pmb{U}}^{\perp} \right) V_{\tau-1}$ , where  $\widehat{\Sigma}_{\tau} \coloneqq \frac{1}{\mathcal{B}} \sum_{j} x_j x_j^{\mathsf{T}}$  is the sample covariance at time step  $\tau$  of Algorithm 2. Similarly, with large enough block size  $\mathcal{B}$ , we have the following lemma:
$^2 y$  is norm-sub-Gaussian, denoted as nSG  $(\sigma)$ , if  $\mathbb{P}[\| \pmb {y} - \mathbb{E}\pmb {y}\| \geq t]\leq 2e^{-\frac{t^2}{2\sigma^2}}$ . We refer interested readers to Jin et al. (2019, Section 2) for more discussions on norm-sub-Gaussianity.
<sup>3</sup>It depends polynomially only in the sub-Gaussian moment of the data distribution  $\mathcal{D}$ ; see Theorem 1.1 of Rudelson and Vershynin (2009).
Lemma 3. Let  $\epsilon, \delta \in (0,1)$ . Suppose that  $\left\| \Pi_{\hat{U}} - \Pi_U \right\|_2 \leq \frac{\Delta_{k,\kappa}}{20V} \min \left( \epsilon, \frac{\delta}{2\sqrt{dk}} \right)$ . Then, it is sufficient to choose the block size  $\mathcal{B}$  in Algorithm 2 as
$$
\mathcal {B} = \Omega \left(\frac {\mathcal {V} + V ^ {2}}{\Delta_ {k , \kappa} ^ {2}} \left(\frac {d k}{\delta^ {2}} \log \frac {k}{\delta} + \frac {1}{\epsilon^ {2}} \log \frac {d}{\delta}\right) + \frac {\mathcal {M} ^ {2}}{\mathcal {V} + V ^ {2}} \log \frac {d}{\delta}\right), \tag {11}
$$
$$
5 \left\| Z _ {\tau , 2} \right\| _ {2} \leq \epsilon \Delta_ {k, \kappa} \quad a n d \quad 5 \left\| Q _ {k} ^ {\top} Z _ {\tau , 2} \right\| _ {2} \leq \frac {\delta \Delta_ {k , \kappa}}{2 \sqrt {d k}}, \quad \forall \tau \geq 1, \tag {12}
$$
where the columns of  $Q_k$  is the top- $k$  (in magnitude) orthonormal eigenvectors of  $\Pi_U^\perp \Sigma \Pi_U^\perp$ . (We note that  $Q_k$  is a solution for Eqn. (4).)
Remark 1. We emphasize that, regardless of the block size requirement in the above lemmas, all the operations can be implemented within  $o(d^2)$  space requirement.
Combining the convergence results, we can prove the following PAFO-learnability guarantee in the memory-limited, streaming setting:
Theorem 1. Let  $d, m, k \in \mathbb{N}$  be fixed. Consider a collection  $\mathcal{F}_d \subset \mathcal{P}_d \times \mathcal{P}_d \times (0,1)$  satisfying Assumptions 1 and 2. Then,  $\mathcal{F}_d$  is PAFO-learnable for streaming PCA with FNPM, where the sufficient number of samples is given as  $N_{\mathcal{F}_d}(\varepsilon_{\mathrm{o}}, \varepsilon_{\mathrm{f}}, \delta) = N_1 + N_2$ , with
$$
N _ {1} = \tilde {\Omega} \left(\frac {K _ {m , \nu}}{p _ {\min }} \left\{\frac {\mathcal {V}}{\Delta_ {m , \nu} ^ {3}} \left(\frac {d m}{\delta^ {2}} + \frac {\alpha}{\eta_ {k} ^ {2}}\right) + \frac {\mathcal {M} ^ {2}}{\mathcal {V} \Delta_ {m , \nu}} \right\} + \frac {\mathbb {1} [ \boldsymbol {g} \neq \boldsymbol {0} ] \sigma^ {2}}{p _ {\min } g _ {\min } ^ {2} \eta_ {k} ^ {2}}\right), \tag {Algothm 1}
$$
$$
N _ {2} = \tilde {\Omega} \left(\frac {K _ {k , \kappa} (\mathcal {V} + V ^ {2})}{\Delta_ {k , \kappa} ^ {3}} \left(\frac {d k}{\delta^ {2}} + \frac {k ^ {2} V ^ {2}}{\varepsilon_ {\mathrm {o}} ^ {2}}\right) + \frac {K _ {k , \kappa} \mathcal {M} ^ {2}}{\Delta_ {k , \kappa} (\mathcal {V} + V ^ {2})}\right), \tag {Algothm 2}
$$
where  $\eta_{k} = \Theta$  (min  $\{\varepsilon_{\mathrm{f}},\frac{\Delta_{k,\kappa}}{kV^{2}}\varepsilon_{\mathrm{o}},\frac{\Delta_{k,\kappa}}{V\sqrt{dk}}\delta \}$  and  $\alpha = 1 + \frac{\mathbb{1}[g\neq0]f_{\max}^2}{g_{\min}^2}$
Let us take a moment to digest the sample complexity in Theorem 1. The second term  $N_{2}$  is determined by the  $\varepsilon_{\mathrm{o}}$ -optimality requirement in our PAFO-learnability. Note that  $N_{2}$  has no dependencies on fairness-related quantities such as  $\varepsilon_{\mathrm{f}}$ ,  $p_{\mathrm{min}}$ , and  $\Delta_{m,\nu}$ . On the other hand, the first term  $N_{1}$  is not only determined by the  $\varepsilon_{\mathrm{f}}$ -fairness requirement, but also the  $\varepsilon_{\mathrm{o}}$ -optimality as well. This is the "price" of pursuing fairness;  $U$ , which encodes the unfair subspace needed to be nullified, is required to be accurately estimated as it impacts not only the level of fairness but also the resulting solution's optimality. This is clear in our formulation of fair PCA; in Eqn. (4), note how the optimal solution depends heavily on  $\Pi_U$ .
We further elaborate on the dependencies of  $N_{1}$  on fairness-related quantities, namely  $p_{\mathrm{min}}$  and  $g$ . First, if  $p_{\mathrm{min}} \to 0$ , i.e., if one of the two groups is never sampled, then the sample complexity tends to infinity, and the learnability does not hold; this aligns with our intuition, as we need samples from both of the sensitive groups. Its dependency is also quite natural, as the minimum expected number of samples from either group depends linearly on  $\frac{1}{p_{\mathrm{min}}}$ . Also, when  $g \neq 0$ ,  $N_{1}$  has an additional additive term scaling linearly with  $\frac{1}{g_{\mathrm{min}}^2}$ . This is because when  $g \neq 0$ , we must explicitly account for the approximation error of  $\frac{g}{\|g\|_2}$  due to the QR decomposition at the end of Algorithm 1.
7 Experiments on Real-World Datasets
The code for all experiments is available at github.com/HanseulJo/fair-streaming-pca.
7.1 CelebA Dataset
We evaluate the efficacy of our proposed FNPM on the CelebA dataset (Liu et al., 2015b). It has been considered by Kleindessner et al. (2023) to show the superior efficiency of their fair PCA algorithm compared to previous approaches (Lee et al., 2022; Olfat and Aswani, 2019; Ravfogel et al., 2022a). However, even in Kleindessner et al. (2023), the images were resized and grey-scaled,
Fair NPM Original
Figure 1: Experimental results on full-resolution CelebA dataset. Original image v.s. FNPM output. (Sensitive attribute: "Eyeglasses")
reducing the dimension from the original  $218 \times 178 = 38,804$  to  $80 \times 80 = 6,400$ . Indeed, for a modest-sized computer, it is impossible to load all 162,770 original images in training set to the memory at once, while they require a full dataset to run each step of their algorithm. Here, we use the original resolution, full-color CelebA dataset. We implement our FNPM using Python JAX NumPy Module (Bradbury et al., 2023; Harris et al., 2020) and Pytorch (Paszke et al., 2017). All experiments were performed on Apple 2020 Mac mini M1 with 16GB RAM.
Although CelebA is not streaming in nature, we intend to show that transforming it to one and using our memory-efficient approach allows us to scale up fair PCA. Since there are three channels of color, we run FNPM channel-wise but in parallel as usual in vision tasks (Priorov et al., 2013). For each channel of colors, we project the data onto a  $k = 1000$ -dimensional subspace while nullifying  $m = 2$  leading eigenvectors of covariance difference.
The resulting images are displayed in Figure 1. Here, we consider 'Eyeglasses' as a sensitive attribute to divide groups. We adopt the predefined train-validation split and run our algorithm only on the training set for 5 iterations with block sizes of  $b = \mathcal{B} = 32,000$ . Then, using the output  $V$  of FNPM, we project images selected from the validation set. We observe that we have images of faces wearing colorful glasses by nullifying some of the leading eigenvectors of covariance difference. For the images originally with sunglasses, their glasses get blurred, and "virtual" eyes are added to them. We provide more results on the other attributes and ablation studies on varying  $m$  and  $b$  in Appendix F.2.
7.2 UCI Datasets
For the sake of completeness, we conduct a quantitative evaluation of our algorithm on UCI datasets (Adult Income, COMPAS, German Credit) and compare it with six previous works (Kleindessner et al., 2023; Lee et al., 2022; Olfat and Aswani, 2019; Ravfogel et al., 2020, 2022a; Samadi et al., 2018). The results for the Adult Income dataset are shown in Table 1. We assess several variants of our methods: in Table 1, "mean" is when we match the means and not second moments, "FNPM" is when we run our Algorithm 2 with a block size of full-batch, and "offline" is when we directly solve our "Null It Out" formulation of fair PCA (Eqn. (4)) via offline eigen-decomposition. In the table, we report the explained variance (\% Var), representation fairness measured by maximum mean discrepancy  $(\mathrm{MMD}^2)$ , downstream task accuracy (\% Acc), and downstream task fairness in demographic parity (\%  $\Delta_{\mathrm{DP}}$ ). The result showcases that our method yields competitive quantitative performance even for the common tabular datasets while being much more memory efficient. We defer the full results for other UCI datasets to Appendix F.3.
8 Other Related Works
Fairness in ML. There are largely two directions in the literature of algorithmic fairness. One direction is to propose a suitable and meaningful fairness definition (Dwork et al., 2012; Feldman et al., 2015; Hardt et al., 2016). The other direction is to develop efficient fair algorithms, although often the fairness constraint forces the algorithm to be much more inefficient than its unfair counterpart, or it calls for a need for a completely new algorithmic approach. There are also different ways of
Table 1: Dataset = Adult Income [feature dim=102, #(train data)=31,655],  $k = 2$
Method	%Var(↑)	MMD2(↓)	%Acc(↑) kernel	ΔDP(↓) SVM
PCA	6.88(0.14)	0.374(0.006)	82.4(0.2)	0.19(0.01)
Olfat and Aswani (2019) (0.1)	Memory Out
Olfat and Aswani (2019) (0.0)	Memory Out
Lee et al. (2022) (1e-3)	5.68(0.11)	0.0(0.0)	80.34(0.24)	0.05(0.01)
Lee et al. (2022) (1e-6)	5.42(0.11)	0.0(0.0)	79.41(0.23)	0.02(0.01)
Kleindessner et al. (2023) (mean)	5.74(0.11)	0.002(0.0)	80.6(0.2)	0.07(0.01)
Kleindessner et al. (2023) (0.85)	4.09(0.17)	0.001(0.001)	75.52(0.21)	0.0(0.0)
Kleindessner et al. (2023) (0.5)	2.63(0.07)	0.0(0.0)	75.38(0.18)	0.0(0.0)
Kleindessner et al. (2023) (kernel)	Takes too long time
Ravfogel et al. (2020)	1.91(0.08)	0.001(0.001)	75.67(0.31)	0.0(0.0)
Ravfogel et al. (2022a)	1.91(0.09)	0.006(0.011)	75.59(0.34)	0.0(0.0)
Samadi et al. (2018)	N/A	N/A	82.63(0.18)	0.15(0.01)
Ours (offline, mean)	5.74(0.11)	0.002(0.0)	80.6(0.2)	0.07(0.01)
Ours (FNPM, mean)	5.74(0.11)	0.002(0.0)	80.6(0.2)	0.07(0.01)
Ours (offline, m=15)	4.04(0.14)	0.001(0.001)	75.51(0.23)	0.0(0.0)
Ours (FNPM, m=15)	4.07(0.13)	0.001(0.0)	75.54(0.25)	0.0(0.0)
Ours (offline, m=50)	2.63(0.07)	0.0(0.0)	75.38(0.18)	0.0(0.0)
Ours (FNPM, m=50)	2.64(0.06)	0.0(0.0)	75.44(0.21)	0.0(0.0)
imposing fairness in an ML pipeline, such as learning fair pre-processing (Biswas and Rajan, 2021), fair in-processing (Roh et al., 2021; Wan et al., 2023; Zafar et al., 2019), and more. The reader is encouraged to check Barocas et al. (2019) for a more comprehensive treatment of this subject.
Fair online/Streaming Learning. Bechavod et al. (2020); Gillen et al. (2018) study individual fairness in online learning in a learning theoretic framework, even when the underlying metric is unavailable. Stemming from the concept of fair clustering as proposed in Chierichetti et al. (2017), Bera et al. (2022); Schmidt et al. (2020) study imposing demographic parity on clustering in the streaming setting. Such fairness has been considered in various other streaming problems such as online selection (Correa et al., 2021), streaming submodular optimization (El Halabi et al., 2020), and diversity maximization (Wang et al., 2022). Quite surprisingly, demographic parity (or any other concept of fairness) has never been considered in the setting of streaming PCA.