SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Streaming PCA. Without the fairness constraint, streaming PCA has been studied much from statistics and the machine learning community. Two prominent algorithms have been studied; the noisy power method (Mitliagkas et al., 2013) and Oja's method (Oja, 1982). Much work has been done in improving the theoretical guarantees of streaming PCA (Balcan et al., 2016; Hardt and Price, 2014; Jain et al., 2016; Liang, 2023), improving the algorithm itself (Xu, 2023; Yun, 2018), or extending the guarantees to various different settings (Balzano et al., 2018; Bienstock et al., 2022; Kumar and Sarkar, 2023). Memory-limited, streaming versions of somewhat related problems, such as community detection (Yun et al., 2014) and low-rank matrix completion (Yun et al., 2015), have been tackled as well using similar spectral techniques as PCA (e.g., power method). However, to the best of our knowledge, fairness (regardless of the definition) has never been considered in this context of streaming PCA, which we tackle in this work and which we believe is of great importance.
In this work, we tackled the two outstanding problems of the existing fair PCA literature. From the theoretical side, we illustrated a new formulation of fair PCA based on the "Null It Out" approach and then provided a novel statistical framework called PAFO-learnability of fair PCA. From the practical side, we addressed the memory-limited scenarios by proposing a new problem setting called fair streaming PCA and a memory-efficient two-stage algorithm called FNPM. Based on these, we established a statistical guarantee that our algorithm achieves the PAFO-learnability for fair streaming PCA. Lastly, we ran experiments on the CelebA and UCI datasets to certify the scalability of our method.
We thank the anonymous reviewers for their helpful comments and suggestions. We also thank Gwangsu Kim (Jeonbuk National University) for the helpful discussions in the initial phase of the research. This work was supported by the Institute of Information & Communications Technology Planning & Evaluation (IITP) grants funded by the Korean government(MSIT) (No.2019-0-00075, Artificial Intelligence Graduate School Program(KAIST); No. 2022-0-00184, Development and Study of AI Technologies to Inexpensively Conform to Evolving Policy on Ethics).
Hervé Abdi and Lynne J. Williams. Principal component analysis. WIREs Computational Statistics, 2(4):433-459, 2010.
Nivasini Ananthakrishnan, Shai Ben-David, Tosca Lechner, and Ruth Urner. Identifying regions of trusted predictions. In Proceedings of the Thirty-Seventh Conference on Uncertainty in Artificial Intelligence, volume 161 of Proceedings of Machine Learning Research, pages 2125-2134. PMLR, 27-30 Jul 2021.
Maria-Florina Balcan, Simon Shaolei Du, Yining Wang, and Adams Wei Yu. An Improved Gap-Dependency Analysis of the Noisy Power Method. In 29th Annual Conference on Learning Theory, volume 49 of Proceedings of Machine Learning Research, pages 284-309, Columbia University, New York, New York, USA, 23-26 Jun 2016. PMLR.
Laura Balzano, Yuejie Chi, and Yue M. Lu. Streaming PCA and Subspace Tracking: The Missing Data Case. Proceedings of the IEEE, 106(8):1293-1310, 2018.
Solon Barocas and Andrew D. Selfst. Big Data's Disparate Impact. 104 California Law Review 671, 2016.
Solon Barocas, Moritz Hardt, and Arvind Narayanan. *Fairness and Machine Learning: Limitations and Opportunities*. fairmlbook.org, 2019.
Yahav Bechavod, Christopher Jung, and Steven Z. Wu. Metric-Free Individual Fairness in Online Learning. In Advances in Neural Information Processing Systems, volume 33, pages 11214-11225. Curran Associates, Inc., 2020.
Suman K. Bera, Syamantak Das, Sainyam Galhotra, and Sagar Sudhir Kale. Fair K-Center Clustering in MapReduce and Streaming Settings. In Proceedings of the ACM Web Conference 2022, WWW '22, page 1414-1422, New York, NY, USA, 2022. Association for Computing Machinery.
Daniel Bienstock, Minchan Jeong, Apurv Shukla, and Se-Young Yun. Robust Streaming PCA. In Advances in Neural Information Processing Systems, volume 35, pages 4231-4243. Curran Associates, Inc., 2022.
Sumon Biswas and Hridesh Rajan. Fair Preprocessing: Towards Understanding Compositional Fairness of Data Transformers in Machine Learning Pipeline. In Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2021, page 981-993, New York, NY, USA, 2021. Association for Computing Machinery.
James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne, and Qiao Zhang. JAX: composable transformations of Python+NumPy programs, 2023. URL http://github.com/google/jax.
Luiz Chamon and Alejandro Ribeiro. Probably Approximately Correct Constrained Learning. In Advances in Neural Information Processing Systems, volume 33, pages 16722-16735. Curran Associates, Inc., 2020.
Flavio Chierichetti, Ravi Kumar, Silvio Lattanzi, and Sergei Vassilvitskii. Fair Clustering Through Fairlets. In Advances in Neural Information Processing Systems, volume 30, pages 5036-5044. Curran Associates, Inc., 2017.
Se-Young Yun. Noisy Power Method with Grassmann Average. In 2018 IEEE International Conference on Big Data and Smart Computing (BigComp), pages 709-712, 2018.
Se-Young Yun, Marc Lelarge, and Alexandre Proutiere. Streaming, Memory Limited Algorithms for Community Detection. In Advances in Neural Information Processing Systems, volume 27, pages 3167-3175. Curran Associates, Inc., 2014.
Se-Young Yun, Marc Lelarge, and Alexandre Proutiere. Fast and Memory Optimal Low-Rank Matrix Approximation. In Advances in Neural Information Processing Systems, volume 28, pages 3166-3185. Curran Associates, Inc., 2015.
Muhammad Bilal Zafar, Isabel Valera, Manuel Gomez-Rodriguez, and Krishna P. Gummadi. Fairness Constraints: A Flexible Approach for Fair Classification. Journal of Machine Learning Research, 20(75):1-42, 2019.
Rich Zemel, Yu Wu, Kevin Swersky, Toni Pitassi, and Cynthia Dwork. Learning Fair Representations. In Proceedings of the 30th International Conference on Machine Learning, volume 28 of Proceedings of Machine Learning Research, pages 325-333, Atlanta, Georgia, USA, 17-19 Jun 2013. PMLR.
Tianxiang Zhao, Enyan Dai, Kai Shu, and Suhang Wang. Towards Fair Classifiers Without Sensitive Attributes: Exploring Biases in Related Features. In Proceedings of the Fifteenth ACM International Conference on Web Search and Data Mining, WSDM '22, page 1433-1442, New York, NY, USA, 2022. Association for Computing Machinery.
1 Introduction 1
2 Preliminaries 2
3 An Alternative Approach to Fair PCA 3
3.1 "Null It Out" Formulation of Fair PCA 3
3.2 An Explicit Characterization for Solution of Fair PCA 3
3.3 Comparison to the Existing Covariance Matching Constraint 3
4 Statistical Viewpoint: PAFO-Learnability of PCA 4
5 Algorithmic Viewpoint: Fair Streaming PCA 4
5.1 Our Algorithm: Fair Noisy Power Method (FNPM) 5
5.2 Previous Approaches are not Suitable for Streaming Setup 6
5.3 Extension to Multiple Sensitive Groups/Attributes 6
6 FNPM is a PAFO-Learning Algorithm 6
7 Experiments on Real-World Datasets 8
7.1 CelebA Dataset 8
7.2 UCI Datasets 9
8 Other Related Works 9
9 Conclusion 10
A Broader Impacts, Limitations, and Future Directions 19
A.1 Broader Impacts 19
A.2 Limitations and Future Directions 19
B Full Pseudocodes of Algorithms 1 and 2 for Fair "Streaming" PCA 21
C Extension of Algorithms 3 to Multiple and Non-binary Attributes 22
D More Detailed Comparison to Kleindessner et al. (2023) 23
D.1 Their Approach 23
D.2 Unsuitability for the Streaming Setting 23
E Proofs of Lemma 2, 3, and Theorem 1 25
E.1 Notations and Assumptions 25
E.2 Matrix/Vector Concentration Inequalities 26
E.3 Proof of Lemma 2-Bounding Error in Second Moment Gap 26
E.4 Proof of Lemma 3 - Bounding the Final Error 28
E.5 Bounding the Estimation Error of Mean Difference 30
E.6 Proof of Theorem 1 - Sample Complexity for PAFO-learnability 31
F More Experiments 36
F.1 Synthetic Example 36
F.2 Additional Results on CelebA Dataset 36
F.3 Full Results on UCI Dataset 36
A Broader Impacts, Limitations, and Future Directions
A.1 Broader Impacts
This work proposes a dimensionality reduction method that addresses memory efficiency and fairness while providing a statistical guarantee. By identifying and nullifying the "unfair direction" inherent in the data distribution, we offer an alternative approach to fair PCA. We anticipate that our approach will motivate researchers to explore other dimensionality reduction techniques (e.g., auto-encoder) with fairness constraints. Significantly, our contribution includes a rigorous theoretical framework, PAFO learnability, which enables sample complexity analysis of fair PCA. We envision the potential for our theory to be further generalized to broader contexts, including alternative definitions of fairness and optimality of different algorithms for fair machine learning.
On the application side, the memory efficiency of our method can facilitate the scalability of fair PCA, making it viable for processing high-dimensional datasets, even in scenarios where data points arrive in a streaming fashion. One possible application of our approach is data pre-/post-processing to alleviate unfairness, such as generating fair word embeddings by eliminating sensitive attribute information through orthogonal projection. For more detailed discussions on potential future directions, please refer to below.
A.2 Limitations and Future Directions
Here, we list some of our work's limitations and possible extensions/future directions.
Individual Fairness for PCA. Our definition of fairness in PCA only covers group fairness via fair presentation learning, which was also the case for the prior works (Kleindessner et al., 2023; Lee et al., 2022; Olfat and Aswani, 2019). None of the works, including ours, have yet to consider the notion of individual fairness (Dwork et al., 2012) in the context of PCA, for which we do not have a definitive answer.
Knowledge of sensitive attribute. Our framework requires the full knowledge of the sensitive attribute  $s$  for all data points  $x$ 's, which was also the case for all the previous works (Kleindessner et al., 2023; Lee et al., 2022; Olfat and Aswani, 2019). But, the assumption of such knowledge may not be feasible in the real world due to privacy or legal reasons (Lahoti et al., 2020; Zhao et al., 2022), or even just due to some extrinsic noises. Considering the case where the dataset may lack sensitive attributes for some or all of the data points is an important future direction.
Making the algorithm anytime. In our formulation of fair PCA, our algorithm is two-phase, with the first phase as a "burn-in" period for estimating the unfair subspace. Thus, it is not an anytime algorithm in that if the algorithm stops whilst in the first phase, then the resulting  $\mathbf{V}$  is random and completely uninformative. Designing an anytime variant of our algorithm is an important future direction. One possible way to achieve that is to consider other streaming PCA algorithms such as Oja's method (Oja and Karhunen, 1985) or accelerated NPM (Xu, 2023; Xu and Li, 2022).
Improving gap dependence. Our algorithm's current sample complexity analysis relies on the analysis of NPM by Hardt and Price (2014), which relies on the immediate singular value gap,  $\sigma_{k} - \sigma_{k + 1}$ . Balcan et al. (2016) showed that considering greater iteration rank  $q \geq k$ , i.e., by considering optimization variable of greater size, leads to a better gap dependency: from  $\sigma_{k} - \sigma_{k + 1}$  to  $\sigma_{k} - \sigma_{q + 1}$ . However, this is incompatible with our current definition of PAFO-learnability (Definition 2) because with greater iteration rank, the solution  $\mathbf{V}^{\star}$  to which the explained variance should be compared against is not clear. The problem is that the sample complexity guarantee of Balcan et al. (2016); Hardt and Price (2014) is derived in terms of the sine angle between the top  $k$ -eigenspace of the true covariance and the estimated  $q$ -dimensional subspace. Clearing this up would allow for a better theoretical guarantee in sample complexity and thus an important future direction.
Kernelizing our framework Kernel PCA (Schölkopf et al., 1998) is a cornerstone in modern machine learning that has lent itself to be an inspiration to many applications, both theory and practice-wise. Unlike previous fair PCA works (Kleindessner et al., 2023; Olfat and Aswani, 2019), in which the authors provided a kernelized version of their fair PCA algorithm, both our statistical framework and memory-efficient algorithm for streaming setting do not readily extend to the kernelized version. Algorithmically, to tackle the streaming setting, one may take inspiration from streaming kernel PCA (Ghashami et al., 2016; Ullah et al., 2018), which was in turn inspired by matrix sketching (Liberty, 2013).
Extending to other settings. In a similar spirit, extending our formulation of fair PCA to other streaming settings such as sparse (Wang and Lu, 2016; Yang and Xu, 2015), nonstationary (Bienstock et al., 2022), or even distributionally robust settings (Vu et al., 2022) would also be interesting.
More experiments Last but not least, we've only performed experiments on synthetic, CelebA, and UCI datasets. It would be interesting to try our framework (either the statistical formulation, the streaming setting, or both) on datasets from other domains, such as NLP and graphs. GloVe vectors (Pennington et al., 2014) has been used as a benchmark in a similar task called "concept erasure" (Ravfogel et al., 2020, 2022b); group fairness in spectral clustering of graphs has been studied as well (Kleindessner et al., 2019; Wang et al., 2023).
B Full Pseudocodes of Algorithms 1 and 2 for Fair “Streaming” PCA
Throughout the appendices, from hereon, we will refer to our algorithms by Algorithms 3 and 4 instead of Algorithms 1 and 2.
Algorithm 3: UnfairSubspace (= Algorithm 1)  
1 Input:  $d, m$ , Block size  $b$ , Number of iterations  $T$ ;  
2 Output: A matrix  $\widehat{U}$  with orthonormal columns;  
3  $W_0 \gets \mathbb{QR}(\mathcal{N}(0,1)^{d \times m})$ ;  
4  $(\overline{\boldsymbol{m}}^{(0)}, \overline{\boldsymbol{m}}^{(1)}, B^{(0)}, B^{(1)}) \gets (\boldsymbol{0}_d, \boldsymbol{0}_d, 0, 0)$ ;  
5 for  $t \in [T]$  do  
6  $(b_t^{(0)}, b_t^{(1)}) \gets (0, 0)$ ;  
7  $(\boldsymbol{m}_t^{(0)}, \boldsymbol{m}_t^{(1)}, C_t^{(0)}, C_t^{(1)}) \gets (\boldsymbol{0}_d, \boldsymbol{0}_d, \boldsymbol{0}_{d \times m}, \boldsymbol{0}_{d \times m})$ ;  
8 for  $i \in [b]$  do  
9Receive  $(a, x) = (a_{(t-1)b+i}, x_{(t-1)b+i})$ ;  
10  $b_t^{(a)} \gets b_t^{(a)} + 1$ ;  
11  $m_t^{(a)} \gets m_t^{(a)} + x$ ;  
12  $C_t^{(a)} \gets C_t^{(a)} + xx^T W_{t-1}$ ;  
13 foreach  $a \in \{0, 1\}$  do  
14 if  $b_t^{(a)} > 0$  then  
15  $m_t^{(a)} \gets \frac{1}{b_t^{(a)}} m_t^{(a)}$ ;  
16  $C_t^{(a)} \gets \frac{1}{b_t^{(a)}} C_t^{(a)}$ ;  
17  $\overline{m}^{(a)} \gets \frac{B^{(a)}}{B^{(a)} + b_t^{(a)}} \overline{m}^{(a)} + \frac{b_t^{(a)}}{B^{(a)} + b_t^{(a)}} m_t^{(a)}$ ;  
18  $B^{(a)} \gets B^{(a)} + b_t^{(a)}$ ;  
19  $W_t = QR(C^{(1)} - C^{(0)})$ ;  
20  $\widehat{f} \gets \overline{m}^{(1)} - \overline{m}^{(0)}$ ;  
21  $\widetilde{g} \gets \widehat{f} - W_T W_T^\top f$ ;  
22 if  $\|\widehat{g}\|_2 = 0$  then  
23  $\widehat{U} = W_T$   
24 else  
25  $\widehat{U} = [W_T | \frac{\widehat{g}}{\|\widehat{g}\|_2}]$   
26 return  $\widehat{U}$   
Algorithm 4: Fair NPM (= Algorithm 2)  
1 Input:  $d, k$ , Block sizes  $B, b$ , Numbers of iterations  $T, T$ ;  
2 Output:  $V_T \in \mathrm{St}(d, k)$ ;  
3  $\widehat{U} \gets \text{UnfairSubspace}(b, T)$ ;  
4  $V_0 \gets QR(\mathcal{N}(0,1)^{d \times k})$ ;  
5 for  $\tau \in [\mathcal{T}]$  do  
6  $V_\tau \gets V_{\tau-1} - \widehat{U} \widehat{U}^\top V_{\tau-1}$ ;  
7  $C \gets 0_{d \times k}$ ;  
8 for  $j \in [\mathcal{B}]$  do  
9Receive  $(*, \tilde{x}) = (*, \tilde{x}_{(\tau-1)\mathcal{B}+j})$ ;  
10  $C \gets C + \frac{1}{B} \tilde{x} \tilde{x}^\top V_\tau$ ;  
11  $V_\tau \gets QR(C - \widehat{U} \widehat{U}^\top C)$ ;  
12 return  $V_T$
C Extension of Algorithms 3 to Multiple and Non-binary Attributes
We remark that Algorithm 3 is a special case of the pseudocode below. By applying Algorithm 4 after applying this algorithm, we can handle the multiple non-binary attribute case and approximately and memory-efficiently solve the extended notion of fair PCA elaborated in Section 5.3.
Algorithm 5:UnfairSubspace_Extended
1 Input:  $d, m_1, \dots, m_\ell$ , Block size  $b$ , Number of iterations  $T$ ;  
2 Output: A matrix  $\hat{U}$  with orthonormal columns;  
3 for  $r \in [\ell]$  do  
4 for  $a \in [g_r]$  do  
5  $\begin{array}{rl} & W_{0}^{(r,a)} = \mathsf{QR}(\mathcal{N}(0,1)^{d\times m_r}); \\ & (\overline{\boldsymbol{m}}^{(r,a,\mathrm{in})}, \overline{\boldsymbol{m}}^{(r,a,\mathrm{out})}, B^{(r,a)}) \leftarrow (\boldsymbol{0}_d, \boldsymbol{0}_d, 0) \end{array}$   
6  
7 for  $t \in [T]$  do  
8 Receive  $\{(a_{1,i}, \dots, a_{\ell,i}), x_i)\}_{i=b(t-1)+1}$ ;  
9 for  $r \in [\ell]$  do  
10 for  $a \in [g_r]$  do  
11  $b_t^{(r,a)} = \sum_{i=b(t-1)+1}^{bt} \mathbb{1}_{[a_r,i=a]}$ ;  
12  $B^{(r,a)} \leftarrow B^{(r,a)} + b_t^{(r,a)}$ ;  
13 if  $b_t^{(a)} > 0$  then  
14  $\boldsymbol{m}_t^{(r,a,\mathrm{in})} = \frac{1}{b_t^{(a)}} \sum_{i=b(t-1)+1}^{bt} \mathbb{1}_{[a_r,i=a]} x_i$ ;  
15  $C_t^{(r,a,\mathrm{in})} = \frac{1}{b_t^{(a)}} \sum_{i=b(t-1)+1}^{bt} \mathbb{1}_{[a_r,i=a]} x_i x_i^\intercal W_{t-1}^{(r,a)}$ ;  
16 else  
17  $(\boldsymbol{m}_t^{(r,a,\mathrm{in})}, C_t^{(r,a,\mathrm{in})}) \leftarrow (\boldsymbol{0}_d, \boldsymbol{0}_d \times m_r)$   
18 if  $b_t^{(a)} < b$  then  
19  $\boldsymbol{m}_t^{(r,a,\mathrm{out})} = \frac{1}{b - b_t^{(a)}} \sum_{i=b(t-1)+1}^{bt} \mathbb{1}_{[a_r,i≠a]} x_i$ ;  
20  $C_t^{(r,a,\mathrm{out})} = \frac{1}{b - b_t^{(a)}} \sum_{i=b(t-1)+1}^{bt} \mathbb{1}_{[a_r,i≠a]} x_i x_i^\intercal W_{t-1}^{(r,a)}$ ;  
21 else  
22  $(\boldsymbol{m}_t^{(r,a,\mathrm{out})}, C_t^{(r,a,\mathrm{out})}) \leftarrow (\boldsymbol{0}_d, \boldsymbol{0}_d \times m_r)$   
23  $\overline{\boldsymbol{m}}^{(r,a,\mathrm{in})} \leftarrow \frac{B^{(r,a)} - b_t^{(r,a)}}{B^{(a)}} \overline{\boldsymbol{m}}^{(r,a,\mathrm{in})} + \frac{b_t^{(a)}}{B^{(a)}} \boldsymbol{m}_t^{(r,a,\mathrm{in})}$ ;  
24  $\overline{\boldsymbol{m}}^{(r,a,\mathrm{out})} \leftarrow \frac{(t-1)b - B^{(r,a)} + b_t^{(r,a)}}{tb - B^{(a)}} \overline{\boldsymbol{m}}^{(r,a,\mathrm{out})} + \frac{b - b_t^{(a)}}{tb - B^{(a)}} \boldsymbol{m}_t^{(r,a,\mathrm{out})}$ ;  
25  $W_t^{(r,a)} = \mathsf{QR}\left(C_t^{(r,a,\mathrm{in})} - C_t^{(r,a,\mathrm{out})}\right)$ ;  
26 for  $r \in [\ell]$  do  
27 for  $a \in [g_r]$  do  
28  $\widehat{\boldsymbol{f}}^{(r,a)} = \overline{\boldsymbol{m}}_t^{(r,a,\mathrm{in})} - \overline{\boldsymbol{m}}_t^{(r,a,\mathrm{out})}$ ;  
29  $\hat{\boldsymbol{U}} = \mathsf{QR}\left(\left[\boldsymbol{W}_t^{(1,1)}\right| \dots\right|\left.\boldsymbol{W}_t^{\left(\ell,g_\ell\right)}\right| \widehat{\boldsymbol{f}}^{(1,1)}\right| \dots\left|\widehat{\boldsymbol{f}}^{\left(\ell,g_\ell\right)}\right|$ ;  
30 return  $\hat{\boldsymbol{U}}$
D More Detailed Comparison to Kleindessner et al. (2023)
D.1 Their Approach
Kleindessner et al. (2023) considered the following formulation of fair PCA:
$$
\max  _ {\boldsymbol {V} \in \operatorname {S t} (d, k)} \operatorname {t r} \left(\boldsymbol {V} ^ {\mathsf {T}} \boldsymbol {X} ^ {\mathsf {T}} \boldsymbol {X} \boldsymbol {V}\right), \quad \text {s u b j e c t t o} \boldsymbol {f} ^ {\mathsf {T}} \boldsymbol {V} = \boldsymbol {0} \wedge \boldsymbol {V} ^ {\mathsf {T}} \left(\boldsymbol {\Sigma} _ {1} - \boldsymbol {\Sigma} _ {0}\right) \boldsymbol {V} = \boldsymbol {0}, \tag {13}
$$
and proposed a reasonable approximation of the covariance constraint, which we briefly describe here. Let us write  $S' = \Sigma_1 - \Sigma_0$  for brevity. As implicitly assumed in Kleindessner et al. (2023), let us assume that  $f \neq 0$ . The mean constraint is dealt with first. Denoting  $U_f \in \mathrm{St}(d, d-1)$  to be the matrix whose columns form a basis  $(d-1)$ -dimensional nullspace of  $f$ , the mean constraint is satisfied if and only if  $V$  is of the form  $U_f U$  with  $U \in \mathrm{St}(d-1, k)$  as the intermediate optimization variable. With this first reparametrization, the optimization now becomes
$$
\max  _ {\boldsymbol {U} \in \operatorname {S t} (d - 1, k)} \operatorname {t r} \left(\boldsymbol {U} ^ {\top} \boldsymbol {U} _ {\boldsymbol {f}} ^ {\top} \boldsymbol {X} ^ {\top} \boldsymbol {X} \boldsymbol {U} _ {\boldsymbol {f}} \boldsymbol {U}\right), \quad \text {s u b j e c t} \boldsymbol {U} ^ {\top} \boldsymbol {U} _ {\boldsymbol {f}} ^ {\top} \boldsymbol {S} ^ {\prime} \boldsymbol {U} _ {\boldsymbol {f}} \boldsymbol {U} = \mathbf {0}, \tag {14}
$$
which now has only the covariance constraint.
To deal with the possible infeasibility of the covariance constraint, Kleindessner et al. (2023) proposed the following approach: letting  $M_{\pmb{f},\pmb{S}'} \in \mathrm{St}(d - 1,l)$  be the matrix whose columns are the  $l$  smallest eigenvectors of  $\pmb{U}_{\pmb{f}}^{\top}\pmb{S}'\pmb{U}_{\pmb{f}}$  (in magnitude),  $\pmb{U}$  only needs to nullify the eigenspace spanned by the remaining  $d - 1 - l$  eigenvectors. To see which terms are ignored with the relaxation, let  $\sum_{i = 1}^{d}q_{i}^{\prime}\pmb{v}^{\prime}\pmb{v}_{i}^{\prime \top}$  be the eigenvalue decomposition of  $\pmb{U}_{\pmb{f}}^{\top}\pmb{S}'\pmb{U}_{\pmb{f}}$  with  $|q_1^\prime |\geq |q_2^\prime |\geq \dots |q_d^\prime |\geq 0$ . This approach essentially ignores the constraint  $\pmb{U}^{\top}\pmb{E}_l\pmb{U} = \mathbf{0}$ , where
$$
\boldsymbol {E} _ {l} = \boldsymbol {M} _ {\boldsymbol {f}, \boldsymbol {S} ^ {\prime}} ^ {\intercal} \boldsymbol {U} _ {\boldsymbol {f}} ^ {\intercal} \boldsymbol {S} ^ {\prime} \boldsymbol {U} _ {\boldsymbol {f}} \boldsymbol {M} _ {\boldsymbol {f}, \boldsymbol {S} ^ {\prime}} = \sum_ {i = d - l} ^ {d - 1} q _ {i} ^ {\prime} \boldsymbol {v} ^ {\prime} \boldsymbol {v} _ {i} ^ {\prime \intercal}. \tag {15}
$$
The number  $l \in \{k, \dots, d - 1\}$  controls how much the group-conditional covariances will be equalized; a smaller  $l$  means that the covariance constraint is enforced more stringently and vice versa. Ultimately, the learner can control  $l$  as a hyperparameter to create a trade-off between fairness and the explained variance. Moreover, if  $|q_i'|$ 's are negligible for all  $i \geq d - 1 - l$ , then  $M_{f,S'}^{\top} U_f^{\top} S' U_f M_{f,S'} \approx 0$ , and the relaxation becomes tighter.
As any  $U$  of the form  $M_{f,S^{\prime}}\Lambda$  satisfies the relaxed covariance constraint, a second reparameterization in terms of the new optimization variable  $\Lambda \in \mathrm{St}(l,k)$  gives us
$$
\max  _ {\Lambda \in \operatorname {S t} (l, k)} \operatorname {t r} \left(\Lambda^ {\top} M _ {f, S ^ {\prime}} ^ {\top} U _ {f} ^ {\top} X ^ {\top} X U _ {f} M _ {f, S ^ {\prime}} \Lambda\right), \tag {16}
$$
which can be solved via the standard SVD-based approach for vanilla PCA. Then, the final solution is obtained as  $V^{*} = M_{f,S^{\prime}}U_{f}\Lambda^{*}$ .
D.2 Unsuitability for the Streaming Setting
In Section 5.2, we provided a rough overview of why existing approaches to fair PCA (Kleindessner et al., 2023; Lee et al., 2022; Olfat and Aswani, 2019) are not amenable to our streaming setting. Especially as the approach of Kleindessner et al. (2023) (described above) is almost like a PCA, one may wonder if standard techniques used in streaming PCA (Mitliagkas et al., 2013) can be used. Here, we argue in detail why that is not the case, which is in sharp contrast to our approach and our reformulation of fair PCA that led to a memory-efficient fair streaming PCA algorithm.