Title: Fair Streaming Principal Component Analysis: Statistical and Algorithmic Viewpoint
Venue: NeurIPS 2023
Accepted: yes
Keywords: streaming

PCA

memory-limited

fair representation

online learning
Abstract: Fair Principal Component Analysis (PCA) is a problem setting where we aim to perform PCA while making the resulting representation fair in that the projected distributions, conditional on the sensitive attributes, match one another. However, existing approaches to fair PCA have two main problems: theoretically, there has been no statistical foundation of fair PCA in terms of learnability; practically, limited memory prevents us from using existing approaches, as they explicitly rely on full access to the entire data. On the theoretical side, we rigorously formulate fair PCA using a new notion called probably approximately fair and optimal (PAFO) learnability. On the practical side, motivated by recent advances in streaming algorithms for addressing memory limitation, we propose a new setting called fair streaming PCA along with a memory-efficient algorithm, fair noisy power method (FNPM). We then provide its statistical guarantee in terms of PAFO-learnability, which is the first of its kind in fair PCA literature. We verify our algorithm in the CelebA dataset without any pre-processing; while the existing approaches are inapplicable due to memory limitations, by turning it into a streaming setting, we show that our algorithm performs fair PCA efficiently and effectively.

Fair Streaming Principal Component Analysis: Statistical and Algorithmic Viewpoint

Abstract

Fair Principal Component Analysis (PCA) is a problem setting where we aim to perform PCA while making the resulting representation fair in that the projected distributions, conditional on the sensitive attributes, match one another. However, existing approaches to fair PCA have two main problems: theoretically, there has been no statistical foundation of fair PCA in terms of learnability; practically, limited memory prevents us from using existing approaches, as they explicitly rely on full access to the entire data. On the theoretical side, we rigorously formulate fair PCA using a new notion called probably approximately fair and optimal (PAFO) learnability. On the practical side, motivated by recent advances in streaming algorithms for addressing memory limitation, we propose a new setting called fair streaming PCA along with a memory-efficient algorithm, fair noisy power method (FNPM). We then provide its statistical guarantee in terms of PAFO-learnability, which is the first of its kind in fair PCA literature. Lastly, we verify the efficacy and memory efficiency of our algorithm on real-world datasets.

1 Introduction

Algorithmic fairness ensures that machine learning algorithms do not propagate nor exacerbate bias, which may lead to discriminatory decision-making (Barocas and Selbst, 2016) and thus has been a very active area of research. This has direct implications in our everyday life, including but not limited to criminal justice (Kirchner et al., 2016), education (Kizilcec and Lee, 2021), and more. See Mehrabi et al. (2021) for a comprehensive survey of bias and fairness in machine learning.

Often, one needs to consider fairness for a large number of high-dimensional data points. One of the standard tools for dealing with such high-dimensional data is PCA (Hotelling, 1933; Pearson, 1901), a classical yet still one of the most popular algorithms for performing interpretable dimensionality reduction. It has been adapted as a baseline and/or standard tool in exploratory data analysis, whose application ranges from natural sciences, engineering (Abdi and Williams, 2010; Jolliffe and Cadima, 2016), and even explainable AI (Li et al., 2023; Tjoa and Guan, 2021). Due to its ubiquity and wide applicability, several works study defining fairness in PCA and developing a fair variant of it. A recent line of research (Kleindessner et al., 2023; Lee et al., 2022; Olfat and Aswani, 2019) defines PCA fairness in the context of fair representation (Zemel et al., 2013) in that the projected group conditional distributions should match.

However, existing fair PCA approaches suffer from two problems. Theoretically, they provide no statistical foundation of fair PCA or guarantees for their algorithms. By statistical foundation, we mean the usual PAC-learnability (Shalev-Schwartz and Ben-David, 2014) guarantees, e.g., sample complexity for ensuring optimality in explained variance and fairness constraint with high probability.

On top of that, the second problem arises from a practical viewpoint: memory limitation. All the aforementioned fair PCA algorithms assume that the learner can store the entire data points and incurs memory complexity of order at least O ( d max( N,d )) , where d is the dimensionality of the data and N is the number of data points. As memory limitation is often a critical bottleneck in deploying machine learning algorithms (Mitliagkas et al., 2013), as much as fairness is important, it is also paramount that imposing fairness to PCA does not incur too much memory overhead. A popular approach to mitigate such memory limitation for PCA is to consider the one-pass, streaming setting. In this setting, each data point is revealed to the learner sequentially, each point is irretrievably gone unless she explicitly stores it, and she can use only O ( dk ) memory, with k being the target dimension of projection. Indeed, without the fairness constraint, streaming PCA has been studied extensively; see Balzano et al. (2018) and references therein.

In this work, we address both problems in a principled manner. Our contributions are as follows:

We provide an alternative formulation of fair PCA based on the 'Null It Out' approach (Section 3). Based on the new formulation, we introduce the concept of probably approximately fair and optimal (PAFO)-learnability to formalize the problem of fair PCA (Section 4).

To address the memory limitation, we propose a new problem setting called fair streaming PCA , as well as fair noisy power method (FNPM), a simple yet memory-efficient algorithm based on the noisy power method. (Section 5). We note that our algorithm incurs a much lower memory complexity even compared to the most efficient variant of fair PCA proposed by Kleindessner et al. (2023).

We then prove that our algorithm achieves the PAFO-learnability for fair streaming PCA (Section 6). Such statistical guarantee is the first of its kind in fair PCA literature.

Lastly, we empirically validate our algorithm on CelebA and UCI datasets. Notably, we run FNPM on the original full-resolution CelebA dataset on which existing fair PCA algorithms fail due to high memory requirements. It shows turning such a non-streaming setting into a streaming setting and applying our algorithm allows one to bypass the memory limitation (Section 7).

2 Preliminaries

Notation. For ℓ ≥ 1 , let I ℓ be the identity matrix of size ℓ × ℓ . For k < d , we bring the Stiefel manifold St( d, k ) = { A ∈ R d × k : A ⊺ A = I k } , which is the collection of all rankk semi-orthogonal matrices. We denote an orthonormal column basis of a (full column rank) matrix M ∈ R d × k obtained by QR decomposition as QR ( M ) ∈ St( d, k ) and denote its column space by col( M ) . Also, for A ∈ St( d, k ) , we denote the orthogonal projection matrix to col( A ) ⊥ = null( A ⊺ ) as Π ⊥ A = I d -AA ⊺ = I d -Π A . Moreover, we denote the collection of all possible d -dimensional probability distributions as P d . For a zero-mean random matrix Z , its (scalar-valued) variance is defined as Var( Z ) = max( ∥ E [ ZZ ⊺ ] ∥ 2 , ∥ E [ Z ⊺ Z ] ∥ 2 ) . In general, Var( Z ) = Var( Z -E [ Z ]) . Lastly, we use the usual O , Ω , and Θ notations for asymptotic analyses, where tildes ( ˜ O , ˜ Ω , and ˜ Θ , resp.) are used for hiding logarithmic factors.

Setup. Assume that the sensitive attribute variable, which we will be imposing fairness, is binary, denoted by a ∈ { 0 , 1 } . For each group a , let D a be a d -dimensional distribution of mean µ a and covariance Σ a , both of which are assumed to be well-defined. We often call them group-conditional mean and covariance, respectively. With a fixed, unknown mixture parameter p ∈ (0 , 1) , let us denote the total data distribution as D := p D 0 +(1 -p ) D 1 . Equivalently, the sensitive attribute follows a ∼ Bernoulli( p ) , and the conditional random variable x | a is sampled from D a . In that case, µ a = E [ x | a ] and Σ a = E [ xx ⊺ | a ] -µ a µ ⊺ a . We often write p 0 = 1 -p and p 1 = p for brevity. We also define the mean difference f := µ 1 -µ 0 and the second moment difference S := E [ xx ⊺ | a = 1] -E [ xx ⊺ | a = 0] = Σ 1 -Σ 0 + µ 1 µ ⊺ 1 -µ 0 µ ⊺ 0 . Accordingly, denote the true mean and covariance of D as µ and Σ , respectively. For simplicity, let us assume that D is centered, i.e., µ = 0 ; note that this does not mean that the group conditional distributions D a 's are centered.

PCA. In the offline setting, the full covariance matrix Σ is given which is often a sample covariance matrix 1 n ∑ n i =1 x i x ⊤ i for n data points x 1 , . . . , x n . The goal of vanilla (offline) PCA (Hotelling, 1933; Pearson, 1901) is to compute the loading matrix V ∈ R d × k that preserves as much variance as possible after projecting Σ via V , i.e. , maximize tr( Π V Σ ) = tr( V ⊺ Σ V ) . Here, k < d is the

target dimension to which the data's dimensionality d is to be reduced and is chosen by the learner. We additionally consider the constraint V ∈ St( d, k ) to ensure that the resulting coordinate after the transformation is orthogonal and thus amenable to various statistical interpretations (Johnson and Wichern, 2008). Without any fairness constraint, Eckart-Young theorem (Eckart and Young, 1936) implies that the solution is characterized as a matrix whose columns are the topk eigenvectors of Σ .

Fair PCA. Recently, it has been suggested that performing vanilla PCA on real-world datasets may exhibit bias, making the final outputted projection 'unfair'. As is often the case, there can be multiple definitions of fairness in PCA, but the following two are the most popular: equalizing reconstruction losses (Kamani et al., 2022; Samadi et al., 2018; Tantipongpipat et al., 2019; Vu et al., 2022), or equalizing the projected distributions (Kleindessner et al., 2023; Lee et al., 2022; Olfat and Aswani, 2019) from the perspective of fair representation (Zemel et al., 2013); we focus on the latter one.

3 An Alternative Approach to Fair PCA

3.1 'Null It Out' Formulation of Fair PCA

In this work, we consider fair PCA as learning fair representation (Zemel et al., 2013). The goal is to preserve as much variance as possible while obfuscating any information regarding the sensitive attribute. To this end, we take the 'Null It Out' approach as proposed in Ravfogel et al. (2020). Intuitively, we want to nullify the directions in which the sensitive attribute a can be inferred, and in this work, we consider two such unfair directions : mean difference f and eigenvectors of second moment difference S . To give the learner flexibility in choosing the trade-off between fairness and performance (measured in explained variance), let m ≥ 1 be the number of top eigenvectors of S to nullify. Thus, the learner is nullifying at most ( m +1) -dimensional subspace that is unfair with respect to a , which we refer to as the unfair subspace . Precisely, we formulate our fair PCA as follows:

where d is the data dimensionality, k is the target dimension, and the columns of P m ∈ St( d, m ) is topm orthonormal eigenvectors of S .

3.2 An Explicit Characterization for Solution of Fair PCA

To first construct the unfair subspace that is spanned by f as well as P m , let us define U ∈ St( d, m ′ ) to be the orthogonal matrix whose columns form a basis of col([ P m | f ]) . Then, U has a closed form as follows: m ′ = m if f ∈ col( P m ) and m ′ = m +1 otherwise, and

where g = Π ⊥ P m f ∈ col( P m ) ⊥ . Note that g is a vector in a direction that f is projected onto col( P m ) ⊥ = null( P ⊺ m ) . For this U , our constraint in (1) can be interpreted as an equivalent nullity constraint Π U V = 0 :

The above is equivalent to the following problem without any constraint other than semi-orthgonality:

which is basically the vanilla k -PCA problem of a matrix Π ⊥ U ΣΠ ⊥ U . Therefore, a topk orthonormal column basis of this matrix is indeed a solution of our problem (4).

3.3 Comparison to the Existing Covariance Matching Constraint

Previous works on fair PCA (Kleindessner et al., 2023; Olfat and Aswani, 2019) consider an exact covariance-matching constraint (namely, V ⊺ ( Σ 1 -Σ 0 ) V = 0 ). In fact, this is equivalent to the condition V ⊺ SV = 0 under the mean-matching constraint f ⊺ V = 0 , which can be derived as

One immediate problem with this is that the constraint may be infeasible depending on the choice of Σ 0 , Σ 1 , or S ; e.g., when Σ 1 -Σ 0 is positive definite. For this reason, Kleindessner et al. (2023); Olfat and Aswani (2019) propose relaxations of the fairness constraints but provide no further discussions on its impact on statistical guarantees. On the contrary, our formulation is always feasible without any need for relaxation, allowing us to consider a rigorous definition of fair PCA (Definition 2) for the first time in fair PCA literature.

4 Statistical Viewpoint: PAFO-Learnability of PCA

As all the distribution statistics ( Σ , p, · · · ) are unknown, the learner, given some finite number of samples, must learn all of them and solve fair PCA. In supervised learning, such a problem is often formalized in a PAC-learnability framework (Shalev-Schwartz and Ben-David, 2014). In the context of PAC-learnability for unsupervised learning settings, TV-learning, which is the task of learning distribution, has been mainly considered so far (Ananthakrishnan et al., 2021; Hopkins et al., 2023). However, unlike TV-learning, it is unnecessary to learn the whole distribution in fair PCA; moreover, fair PCA has the fairness constraint Π U V = 0 to be satisfied. Inspired by the unsupervised PAC-learnability as well as constrained PAC-learnability (Chamon and Ribeiro, 2020), we propose a new notion of learnability for fair PCA, called PAFO (Probably Approximately Fair and Optimal) learnability , as follows:

Definition 1 (Projection Learner) . A projection learner is a function that takes k ≥ 1 and d -dimensional samples as input and outputs a loading matrix V ∈ St( d, k ) .

Definition 2 (PAFO-Learnability of PCA) . Let d, k, m be integers such that 1 ≤ k < d and m<d . We say that F d ⊂ P d ×P d × (0 , 1) is PAFO-learnable for PCA if there exists a function N F d : (0 , 1) 3 → N and a projection learner A satisfying the following:

For every ε o , ε f , δ ∈ (0 , 1) and ( D 0 , D 1 , p ) ∈ F d , when running A on N ≥ N F d ( ε o , ε f , δ ) i.i.d. samples from D := p D 1 +(1 -p ) D 0 of the form ( a, x ) , A returns V s.t., with probability at least 1 -δ (over the draws of the N samples),

where U is as defined in Eqn. (2) and V ⋆ is any solution to Eqn. (4) (with prescribed k and m ).

Like in the usual PAC-learnability, N F d is referred to as the sample complexity of fair PCA. Observe how the optimality is measured w.r.t. the optimal solution of fair PCA, not the vanilla PCA. Also, the two conditions are not overlapping: vanilla PCA (overly) satisfies ε o -optimality in explained variance but does not satisfy ε f -optimality in fairness, and vice versa for a poorly chosen V ∈ St ( d, k ) with col( V ) ⊆ col([ P m | f ]) ⊥ .

5 Algorithmic Viewpoint: Fair Streaming PCA

We now introduce a new problem setting, fair streaming PCA . In this setting, the learner receives a stream of pairs ( a t , x t ) ∈ { 0 , 1 } × R d sequentially. Note that the sensitive attribute information a t is also available at each time-step; this is commonly assumed when considering fairness in streaming setting (Bera et al., 2022; El Halabi et al., 2020). Precisely, we assume the following model of the data generation process: at each time-step t , a sensitive attribute is chosen as a t ∼ Bernoulli( p ) , then the data is sampled from the corresponding sensitive group's conditional distribution x t | a t ∼ D s t . Importantly, as done in previous streaming PCA literature (Mitliagkas et al., 2013), we assume that the learner has only O ( dk ) memory, where d is the data dimension and k is the target dimension. We can formally define the PAFO-learnability in this streaming setting:

Definition 3. We say that F d ⊆ P d ×P d × (0 , 1) is PAFO-learnable for streaming PCA if the projection learner A for which Definition 2 holds uses only O ( dk ) memory for streaming data.

5.1 Our Algorithm: Fair Noisy Power Method (FNPM)

One only needs to estimate U to use the off-the-shelf streaming PCA algorithm. As U is of size d × m , storing its estimate is no problem for the memory constraint as long as m = O ( k ) . Naturally, we proceed via a two-stage approach; first, estimate U sufficiently well, then with the fixed estimate of U , apply the noisy power method (Hardt and Price, 2014; Mitliagkas et al., 2013) for V .

For estimating U , one needs to estimate f and P m . Estimating f can be done using the usual cumulative averaging. As for P m , we can consider the two main approaches for streaming PCA: Oja's method (Huang et al., 2021; Oja, 1982; Oja and Karhunen, 1985) and noisy power method (NPM) (Hardt and Price, 2014; Mitliagkas et al., 2013). We first show that Oja's method is inapplicable for our purpose, as it may ignore some eigenvectors corresponding to negative (but large in magnitude) eigenvalues of S . For instance, if S = -2 e 1 e ⊺ 1 + e 2 e ⊺ 2 +4 e 3 e ⊺ 3 with e i being the standard basis vectors, then Oja's method with m = 2 would yield [ e 2 | e 3 ] when we actually want [ e 1 | e 3 ] . For the same reason, simply shifting the eigenvalue spectrum by considering S + ∥ S ∥ 2 I does not work. Thus we apply NPM for estimating P m in our case, which is known to converge as long as the singular value gap of P m is large enough and norms of the noise matrices at each iterate are properly bounded (Balcan et al., 2016; Hardt and Price, 2014).

Algorithm 1: UnfairSubspace

Description of the algorithms. The pseudocode of our algorithm is shown in Algorithms 1 and 2. The goal of Algorithm 1 is to estimate U = [ P m | g ∥ g ∥ 2 ] in Eqn. (2) as accurately as possible. Lines 5-13 do the estimation of P m and g = Π ⊥ P m f ; line 11 is the NPM to find P m , lines 12 and 13 are the estimation of f and g respectively, and line 17 is the concatenation of the estimates of P m and g / ∥ g ∥ 2 . Especially at line 17, the algorithm determines whether to incorporate mean difference by checking ˆ g , which can be proved to be correct, i.e. , g = 0 if and only if ˆ g = 0 with high probability. With the estimated U from Algorithm 1, Algorithm 2 performs the usual NPM on Π ⊥ U ΣΠ ⊥ U , as in Eqn. (4). The memory complexity of Algorithm 2 is O ( d max( m,k )) , since we do not have to store all b or B data points at each time, and all the operations used can be implemented in a manner that conforms to the memory limitation; the full pseudocodes are provided in Appendix B.

At time step t of Algorithm 1, for each a ∈ { 0 , 1 } , b ( a ) t is the number of data points x i 's such that a i = a ; m ( a ) t is the term used for estimation of the group-wise sample mean of x i 's; C ( a ) t is used for the group-wise sample second moment. Their forms are as follows and can be computed

incrementally in the streaming setup:

where we set the last two quantities to 0 when b ( a ) t = 0 .

Note that as b ( a ) t itself is random, this presents some technical challenges in the proofs of the theoretical guarantees. For instance, the above estimators for the mean and covariance are biased. Still, by properly using peeling argument and matrix concentration inequalities as well as perturbation theories (Golub and Loan, 2013; Tropp, 2015), we could sufficiently bound their errors. Informally speaking, we show that line 11 corresponds to the noisy power method for the matrix S , which incurs the memory complexity O ( dm ) .

5.2 Previous Approaches are not Suitable for Streaming Setup

All the existing approaches to fair PCA require the full knowledge of µ 1 -µ 0 and Σ 1 -Σ 0 , or even the full data matrix X . Olfat and Aswani (2019) need f and S to formulate the convex matrix constraints for their semi-positive definite programming (SDP), which is then solved with commercial SDP solver; Lee et al. (2022) need X to compute the derivative of their maximum mean discrepancy (MMD) penalty term, which requires O ( d 2 ) memory to compute kernel Gram matrices. One may hope that the PCA-type approach taken by Kleindessner et al. (2023) may be easily extendable to our streaming setting by just using the standard techniques (e.g., using matrix-vector products) from streaming PCA (Mitliagkas et al., 2013). Indeed, they also propose a similar relaxation of the covariance constraint, leading to a closed-form solution. However, their formulation is not memory-efficient and, more importantly, is not as applicable to streaming settings as our formulation; see Appendix D for more discussions.

5.3 Extension to Multiple Sensitive Groups/Attributes

So far, we have considered a single and binary-sensitive attribute for simplicity. Here, we briefly discuss extending our framework to multiple non-binary sensitive attributes. Suppose that there are ℓ different attributes (e.g., gender and race), and for each r ∈ [ ℓ ] , the r -th sensitive attribute has g r groups (e.g., male, female, and non-binary: g gender = 3 in this case). We first describe the new data generation process: at each time t , we obtain (( a 1 ,t , · · · , a ℓ,t ) , x t ) , where for each r , a r,t ∼ Categorical( p r, 1 , · · · , p r,g r ) with ∑ g r a =1 p r,a = 1 . Then, our formulation of fair PCA easily extends via a one-versus-all comparison approach. That is to say, for each r ∈ [ ℓ ] , define

̸

̸

Denote the topm r column basis of eigenspace of S r,a by P m r ( S r,a ) . Then our notion of fair PCA can be extended as follows: where d, k, m 1 , · · · , m n are given as d > k + ℓ + ∑ r m r ,

Our algorithm can also be naturally adapted to this scenario via one-versus-all manner. In Appendix C, we provide the full pseudocode for the general case of multiple and non-binary sensitive attributes.

6 FNPM is a PAFO-Learning Algorithm

We now show that our proposed memory-efficient algorithm, FNPM, is actually a PAFO-learning algorithm in that with a certain sample complexity, it satisfies the definition of PAFO-learnability. The proofs of all the theoretical results stated here are deferred to Appendix E.

To use proper matrix concentration inequalities for our error term analysis, various streaming PCA literature adopt some assumption on the underlying data distribution, e.g., sub-Gaussianity (Bienstock et al., 2022; Jain et al., 2016; Yang et al., 2018). Throughout the paper and for our theoretical discussions, we consider a collection of distributions F d ⊂ P d ×P d × (0 , 1) satisfying the following assumptions to our data generation process in terms of the data point conditioned on the sensitive attribute and its eigenspectrum:

Assumption 1. Consider our data generation process a ∼ Bernoulli( p ) and x | a ∼ D a . There exists σ, V, M , V , g min > 0 , f max ∈ ( g min , ∞ ) , and p min ∈ (0 , 0 . 5] such that the followings hold for any ( D 0 , D 1 , p ) ∈ F d : for a ∈ { 0 , 1 } , D a ∈ nSG( σ ) , 2

and

Assumption 2. Fix m,k ∈ N . There exist ∆ m,ν , ∆ k,κ , K m,ν , K k,κ ∈ (0 , ∞ ) such that for any ( D 0 , D 1 , · ) ∈ F d , the followings hold:

where ν 1 ≥ · · · ≥ ν d ≥ 0 and κ 1 ≥ · · · ≥ κ d ≥ 0 are the singular values of S and Π ⊥ U ΣΠ ⊥ U , respectively.

We start by establishing the sample complexity bounds of Algorithms 1 and 2 based on the convergence bound for NPM by Hardt and Price (2014). Recall that NPM is an algorithm for finding topr eigenspace (in magnitude) of a symmetric (but not necessarily PSD ) matrix A under a random noise Z ∈ R d × k , by update V t ← QR ( AV t -1 + Z t ) . We start by recalling their meta-sample complexity result for NPM, which we have slightly reformulated for our convenience:

Lemma 1 (Corollary 1.1 of Hardt and Price (2014)) . Let 1 ≤ r < d , ϵ ∈ (0 , 1 / 2) and δ ∈ (0 , 2 e -cd ) , where c is an absolute constant. 3 Let L r be the d × r matrix, whose columns correspond to the topr eigenvectors (in magnitude) of the symmetric (not necessarily PSD) matrix A ∈ R d × d , and let ξ 1 ≥ · · · ≥ ξ d ≥ 0 be the singular values of A . Assume that the noise matrices Z t ∈ R d × r satisfy

Then, after T = Θ ( ξ r ξ r -ξ r +1 log ( d ϵδ ) ) steps of NPM, ∥ ∥ Π ⊥ V T L r ∥ ∥ 2 ≤ ϵ with probability at least 1 -δ .

First, we prove that the W T resulting from Algorithm 1 (NPM for the second moment gap) converges to the true value. We can view Algorithm 1's updates as W t ← QR ( SW t -1 + Z t, 1 ) , where the noise matrix in this case is Z t, 1 := ( C (1) t -C (0) t ) -SW t -1 and C ( a ) t is as defined in Eqn. (6). The following lemma asserts that with large enough block size b , the error matrices are sufficiently bounded such that the NPM iterates converge:

Lemma 2. Let ϵ, δ ∈ (0 , 1) . It is sufficient to choose the block size b in Algorithm 1 as

to make the following hold with probability at least 1 -δ :

where we recall that the columns of P m are the topm (in magnitude) orthonormal eigenvectors of S .

Let ̂ U be the final estimate of the true U outputted by Algorithm 1. For Algorithm 2, the noise matrix is Z τ, 2 := ( Π ⊥ ̂ U ̂ Σ τ Π ⊥ ̂ U -Π ⊥ U ΣΠ ⊥ U ) V τ -1 , where ̂ Σ τ := 1 B ∑ j x j x ⊺ j is the sample covariance at time step τ of Algorithm 2. Similarly, with large enough block size B , we have the following lemma:

2

Lemma 3. Let ϵ, δ ∈ (0 , 1) . Suppose that ∥ ∥ Π ̂ U -Π U ∥ ∥ 2 ≤ ∆ k,κ 20 V min ( ϵ, δ 2 √ dk ) . Then, it is sufficient to choose the block size B in Algorithm 2 as

to make the following hold with probability at least 1 -δ :

where the columns of Q k is the topk (in magnitude) orthonormal eigenvectors of Π ⊥ U ΣΠ ⊥ U . (We note that Q k is a solution for Eqn. (4).)

Remark 1. We emphasize that, regardless of the block size requirement in the above lemmas, all the operations can be implemented within o ( d 2 ) space requirement.

Combining the convergence results, we can prove the following PAFO-learnability guarantee in the memory-limited, streaming setting:

Theorem 1. Let d, m, k ∈ N be fixed. Consider a collection F d ⊂ P d × P d × (0 , 1) satisfying Assumptions 1 and 2. Then, F d is PAFO-learnable for streaming PCA with FNPM, where the sufficient number of samples is given as N F d ( ε o , ε f , δ ) = N 1 + N 2 , with

̸

̸

Let us take a moment to digest the sample complexity in Theorem 1. The second term N 2 is determined by the ε o -optimality requirement in our PAFO-learnability. Note that N 2 has no dependencies on fairness-related quantities such as ε f , p min , and ∆ m,ν . On the other hand, the first term N 1 is not only determined by the ε f -fairness requirement, but also the ε o -optimality as well. This is the 'price' of pursuing fairness; U , which encodes the unfair subspace needed to be nullified, is required to be accurately estimated as it impacts not only the level of fairness but also the resulting solution's optimality. This is clear in our formulation of fair PCA; in Eqn. (4), note how the optimal solution depends heavily on Π U .

̸

̸

We further elaborate on the dependencies of N 1 on fairness-related quantities, namely p min and g . First, if p min → 0 , i.e., if one of the two groups is never sampled, then the sample complexity tends to infinity, and the learnability does not hold; this aligns with our intuition, as we need samples from both of the sensitive groups. Its dependency is also quite natural, as the minimum expected number of samples from either group depends linearly on 1 p min . Also, when g = 0 , N 1 has an additional additive term scaling linearly with 1 g 2 min . This is because when g = 0 , we must explicitly account for the approximation error of g ∥ g ∥ 2 due to the QR decomposition at the end of Algorithm 1.

7 Experiments on Real-World Datasets

The code for all experiments is available at github.com/HanseulJo/fair-streaming-pca .

7.1 CelebA Dataset

We evaluate the efficacy of our proposed FNPM on the CelebA dataset (Liu et al., 2015b). It has been considered by Kleindessner et al. (2023) to show the superior efficiency of their fair PCA algorithm compared to previous approaches (Lee et al., 2022; Olfat and Aswani, 2019; Ravfogel et al., 2022a). However, even in Kleindessner et al. (2023), the images were resized and grey-scaled,

Original

Fair NPM

BJoRK

reducing the dimension from the original 218 × 178 = 38,804 to 80 × 80 = 6,400. Indeed, for a modestsized computer, it is impossible to load all 162,770 original images in training set to the memory at once, while they require a full dataset to run each step of their algorithm. Here, we use the original resolution, full-color CelebA dataset. We implement our FNPM using Python JAX NumPy Module (Bradbury et al., 2023; Harris et al., 2020) and Pytorch (Paszke et al., 2017). All experiments were performed on Apple 2020 Mac mini M1 with 16GB RAM.

Although CelebA is not streaming in nature, we intend to show that transforming it to one and using our memory-efficient approach allow us to scale up fair PCA. Since there are three channels of color, we run FNPM channel-wise but in parallel as usual in vision tasks (Priorov et al., 2013). For each channel of colors, we project the data onto a k = 1000 -dimensional subspace while nullifying m = 2 leading eigenvectors of covariance difference.

The resulting images are displayed in Figure 1. Here, we consider 'Eyeglasses' as a sensitive attribute to divide groups. We adopt the predefined train-validation split and run our algorithm only on the training set for 5 iterations with block sizes of b = B = 32 , 000 . Then, using the output V of FNPM, we project images selected from the validation set. We observe that we have images of faces wearing colorful glasses by nullifying some of the leading eigenvectors of covariance difference. For the images originally with sunglasses, their glasses get blurred, and 'virtual' eyes are added to them. We provide more results on the other attributes and ablation studies on varying m and b in Appendix F.2.

7.2 UCI Datasets

For the sake of completeness, we conduct a quantitative evaluation of our algorithm on UCI datasets (Adult Income, COMPAS, German Credit) and compare it with six previous works (Kleindessner et al., 2023; Lee et al., 2022; Olfat and Aswani, 2019; Ravfogel et al., 2020, 2022a; Samadi et al., 2018). The results for the Adult Income dataset are shown in Table 1. We assess several variants of our methods: in Table 1, 'mean' is when we match the means and not second moments, 'FNPM' is when we run our Algorithm 2 with a block size of full-batch, and 'offline' is when we directly solve our 'Null It Out' formulation of fair PCA (Eqn. (4)) via offline eigen-decomposition. In the table, we report the explained variance (%Var), representation fairness measured by maximum mean discrepancy (MMD 2 ), downstream task accuracy (%Acc), and downstream task fairness in demographic parity (% ∆ DP). The result showcases that our method yields competitive quantitative performance even for the common tabular datasets while being much more memory efficient. We defer the full results for other UCI datasets to Appendix F.3.

8 Other Related Works

Fairness in ML. There are largely two directions in the literature of algorithmic fairness. One direction is to propose a suitable and meaningful fairness definition (Dwork et al., 2012; Feldman et al., 2015; Hardt et al., 2016). The other direction is to develop efficient fair algorithms, although often the fairness constraint forces the algorithm to be much more inefficient than its unfair counterpart, or it calls for a need for a completely new algorithmic approach. There are also different ways of

imposing fairness in an ML pipeline, such as learning fair pre-processing (Biswas and Rajan, 2021), fair in-processing (Roh et al., 2021; Wan et al., 2023; Zafar et al., 2019), and more. The reader is encouraged to check Barocas et al. (2019) for a more comprehensive treatment of this subject.

Fair online/streaming Learning. Bechavod et al. (2020); Gillen et al. (2018) study individual fairness in online learning in a learning theoretic framework, even when the underlying metric is unavailable. Stemming from the concept of fair clustering as proposed in Chierichetti et al. (2017), Bera et al. (2022); Schmidt et al. (2020) study imposing demographic parity on clustering in the streaming setting. Such fairness has been considered in various other streaming problems such as online selection (Correa et al., 2021), streaming submodular optimization (El Halabi et al., 2020), and diversity maximization (Wang et al., 2022). Quite surprisingly, demographic parity (or any other concept of fairness) has never been considered in the setting of streaming PCA.

Streaming PCA. Without the fairness constraint, streaming PCA has been studied much from statistics and the machine learning community. Two prominent algorithms have been studied; the noisy power method (Mitliagkas et al., 2013) and Oja's method (Oja, 1982). Much work has been done in improving the theoretical guarantees of streaming PCA (Balcan et al., 2016; Hardt and Price, 2014; Jain et al., 2016; Liang, 2023), improving the algorithm itself (Xu, 2023; Yun, 2018), or extending the guarantees to various different settings (Balzano et al., 2018; Bienstock et al., 2022; Kumar and Sarkar, 2023). Memory-limited, streaming versions of somewhat related problems, such as community detection (Yun et al., 2014) and low-rank matrix completion (Yun et al., 2015), have been tackled as well using similar spectral techniques as PCA (e.g., power method). However, to the best of our knowledge, fairness (regardless of the definition) has never been considered in this context of streaming PCA, which we tackle in this work and which we believe is of great importance.

9 Conclusion

In this work, we tackled the two outstanding problems of the existing fair PCA literature. From the theoretical side, we illustrated a new formulation of fair PCA based on the 'Null It Out' approach and then provided a novel statistical framework called PAFO-learnability of fair PCA. From the practical side, we addressed the memory-limited scenarios by proposing a new problem setting called fair streaming PCA and a memory-efficient two-stage algorithm called FNPM. Based on these, we established a statistical guarantee that our algorithm achieves the PAFO-learnability for fair streaming PCA. Lastly, we ran experiments on the CelebA and UCI datasets to certify the scalability of our method.

Acknowledgments and Disclosure of Funding

We thank the anonymous reviewers for their helpful comments and suggestions. We also thank Gwangsu Kim (Jeonbuk National University) for the helpful discussions in the initial phase of the research. This work was supported by the Institute of Information & Communications Technology Planning & Evaluation (IITP) grants funded by the Korean government(MSIT) (No.2019-0-00075, Artificial Intelligence Graduate School Program(KAIST); No. 2022-0-00184, Development and Study of AI Technologies to Inexpensively Conform to Evolving Policy on Ethics).

References

Hervé Abdi and Lynne J. Williams. Principal component analysis. WIREs Computational Statistics , 2(4):433-459, 2010.

Nivasini Ananthakrishnan, Shai Ben-David, Tosca Lechner, and Ruth Urner. Identifying regions of trusted predictions. In Proceedings of the Thirty-Seventh Conference on Uncertainty in Artificial Intelligence , volume 161 of Proceedings of Machine Learning Research , pages 2125-2134. PMLR, 27-30 Jul 2021.

Maria-Florina Balcan, Simon Shaolei Du, Yining Wang, and Adams Wei Yu. An Improved GapDependency Analysis of the Noisy Power Method. In 29th Annual Conference on Learning Theory , volume 49 of Proceedings of Machine Learning Research , pages 284-309, Columbia University, New York, New York, USA, 23-26 Jun 2016. PMLR.

Laura Balzano, Yuejie Chi, and Yue M. Lu. Streaming PCA and Subspace Tracking: The Missing Data Case. Proceedings of the IEEE , 106(8):1293-1310, 2018.

Solon Barocas and Andrew D. Selbst. Big Data's Disparate Impact. 104 California Law Review 671 , 2016.

Solon Barocas, Moritz Hardt, and Arvind Narayanan. Fairness and Machine Learning: Limitations and Opportunities . fairmlbook.org, 2019.

Yahav Bechavod, Christopher Jung, and Steven Z. Wu. Metric-Free Individual Fairness in Online Learning. In Advances in Neural Information Processing Systems , volume 33, pages 11214-11225. Curran Associates, Inc., 2020.

Suman K. Bera, Syamantak Das, Sainyam Galhotra, and Sagar Sudhir Kale. Fair K-Center Clustering in MapReduce and Streaming Settings. In Proceedings of the ACM Web Conference 2022 ,WWW '22, page 1414-1422, New York, NY, USA, 2022. Association for Computing Machinery.

Daniel Bienstock, Minchan Jeong, Apurv Shukla, and Se-Young Yun. Robust Streaming PCA. In Advances in Neural Information Processing Systems , volume 35, pages 4231-4243. Curran Associates, Inc., 2022.

Sumon Biswas and Hridesh Rajan. Fair Preprocessing: Towards Understanding Compositional Fairness of Data Transformers in Machine Learning Pipeline. In Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering , ESEC/FSE 2021, page 981-993, New York, NY, USA, 2021. Association for Computing Machinery.

James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne, and Qiao Zhang. JAX: composable transformations of Python+NumPy programs, 2023. URL http://github.com/google/jax .

Luiz Chamon and Alejandro Ribeiro. Probably Approximately Correct Constrained Learning. In Advances in Neural Information Processing Systems , volume 33, pages 16722-16735. Curran Associates, Inc., 2020.

Flavio Chierichetti, Ravi Kumar, Silvio Lattanzi, and Sergei Vassilvitskii. Fair Clustering Through Fairlets. In Advances in Neural Information Processing Systems , volume 30, pages 5036-5044. Curran Associates, Inc., 2017.

Jose Correa, Andres Cristi, Paul Duetting, and Ashkan Norouzi-Fard. Fairness and Bias in Online Selection. In Proceedings of the 38th International Conference on Machine Learning , volume 139 of Proceedings of Machine Learning Research , pages 2112-2121. PMLR, 18-24 Jul 2021.

Chandler Davis and W. M. Kahan. Some new bounds on perturbation of subspaces. Bulletin of the American Mathematical Society , 75(4):863 - 868, 1969.

Cynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Richard Zemel. Fairness through Awareness. In Proceedings of the 3rd Innovations in Theoretical Computer Science Conference , ITCS '12, page 214-226, New York, NY, USA, 2012. Association for Computing Machinery.

Carl Eckart and Gale Young. The approximation of one matrix by another of lower rank. Psychometrika , 1:211-218, 1936.

Marwa El Halabi, Slobodan Mitrovi´ c, Ashkan Norouzi-Fard, Jakab Tardos, and Jakub M Tarnawski. Fairness in Streaming Submodular Maximization: Algorithms and Hardness. In Advances in Neural Information Processing Systems , volume 33, pages 13609-13622. Curran Associates, Inc., 2020.

Michael Feldman, Sorelle A. Friedler, John Moeller, Carlos Scheidegger, and Suresh Venkatasubramanian. Certifying and Removing Disparate Impact. In Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , KDD '15, page 259-268, New York, NY, USA, 2015. Association for Computing Machinery.

Mina Ghashami, Daniel J. Perry, and Jeff Phillips. Streaming Kernel Principal Component Analysis. In Proceedings of the 19th International Conference on Artificial Intelligence and Statistics , volume 51 of Proceedings of Machine Learning Research , pages 1365-1374, Cadiz, Spain, 09-11 May 2016. PMLR.

Stephen Gillen, Christopher Jung, Michael Kearns, and Aaron Roth. Online Learning with an Unknown Fairness Metric. In Advances in Neural Information Processing Systems , volume 31, pages 2605-2614. Curran Associates, Inc., 2018.

Gene H. Golub and Charles F. Van Loan. Matrix Computations . Johns Hopkins Studies in the Mathematical Sciences. The Johns Hopkins University Press, 4 edition, 2013.

Moritz Hardt and Eric Price. The Noisy Power Method: A Meta Algorithm with Applications. In Advances in Neural Information Processing Systems , volume 27, pages 2861-2869. Curran Associates, Inc., 2014.

Moritz Hardt, Eric Price, and Nati Srebro. Equality of Opportunity in Supervised Learning. In Advances in Neural Information Processing Systems , volume 29, pages 3323-3331. Curran Associates, Inc., 2016.

Charles R. Harris, K. Jarrod Millman, Stéfan J. van der Walt, Ralf Gommers, Pauli Virtanen, David Cournapeau, Eric Wieser, Julian Taylor, Sebastian Berg, Nathaniel J. Smith, Robert Kern, Matti Picus, Stephan Hoyer, Marten H. van Kerkwijk, Matthew Brett, Allan Haldane, Jaime Fernández del Río, Mark Wiebe, Pearu Peterson, Pierre Gérard-Marchant, Kevin Sheppard, Tyler Reddy, Warren Weckesser, Hameer Abbasi, Christoph Gohlke, and Travis E. Oliphant. Array programming with NumPy. Nature , 585(7825):357-362, September 2020.

Max Hopkins, Daniel M. Kane, Shachar Lovett, and Gaurav Mahajan. Do PAC-Learners Learn the Marginal Distribution? arXiv preprint arXiv:2302.06285 , 2023.

Harold Hotelling. Analysis of a complex of statistical variables into principal components. Journal of Educational Psychology , 24(6):417-441, 1933.

De Huang, Jonathan Niles-Weed, and Rachel Ward. Streaming k-PCA: Efficient guarantees for Oja's algorithm, beyond rank-one updates. In Proceedings of Thirty Fourth Conference on Learning Theory , volume 134 of Proceedings of Machine Learning Research , pages 2463-2498. PMLR, 15-19 Aug 2021.

Prateek Jain, Chi Jin, Sham M. Kakade, Praneeth Netrapalli, and Aaron Sidford. Streaming PCA: Matching Matrix Bernstein and Near-Optimal Finite Sample Guarantees for Oja's Algorithm. In 29th Annual Conference on Learning Theory , volume 49 of Proceedings of Machine Learning Research , pages 1147-1164, Columbia University, New York, New York, USA, 23-26 Jun 2016. PMLR.

Chi Jin, Praneeth Netrapalli, Rong Ge, Sham M. Kakade, and Michael I. Jordan. A Short Note on Concentration Inequalities for Random Vectors with SubGaussian Norm. arXiv preprint arXiv:1902.03736 , 2019.

Richard A. Johnson and Dean W. Wichern. Applied Multivariate Statistical Analysis . Pearson, 6 edition, 2008.

Ian T. Jolliffe and Jorge Cadima. Principal component analysis: a review and recent developments. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences , 374(2065), 2016.

Mohammad Mahdi Kamani, Farzin Haddadpour, Rana Forsati, and Mehrdad Mahdavi. Efficient fair principal component analysis. Machine Learning , 111(10):3671-3702, 2022.

L Kirchner, J. Larson, S. Mattu, and J. Angwin. Machine bias. ProPublica, 2016.

René F. Kizilcec and Hansol Lee. Algorithmic Fairness in Education. arXiv preprint arXiv:2007.05443 , 2021.

Matthäus Kleindessner, Samira Samadi, Pranjal Awasthi, and Jamie Morgenstern. Guarantees for Spectral Clustering with Fairness Constraints. In Proceedings of the 36th International Conference on Machine Learning , volume 97 of Proceedings of Machine Learning Research , pages 3458-3467. PMLR, 09-15 Jun 2019.

Matthäus Kleindessner, Michele Donini, Chris Russell, and Muhammad Bilal Zafar. Efficient fair PCA for fair representation learning. In Proceedings of The 26th International Conference on Artificial Intelligence and Statistics , volume 206 of Proceedings of Machine Learning Research , pages 5250-5270. PMLR, 25-27 Apr 2023.

Jonas Moritz Kohler and Aurelien Lucchi. Sub-sampled Cubic Regularization for Non-convex Optimization. In Proceedings of the 34th International Conference on Machine Learning , volume 70 of Proceedings of Machine Learning Research , pages 1895-1904. PMLR, 06-11 Aug 2017.

Syamantak Kumar and Purnamrita Sarkar. Streaming PCA for Markovian Data. arXiv preprint arXiv:2305.02456 , 2023.

Preethi Lahoti, Alex Beutel, Jilin Chen, Kang Lee, Flavien Prost, Nithum Thain, Xuezhi Wang, and Ed Chi. Fairness without Demographics through Adversarially Reweighted Learning. In Advances in Neural Information Processing Systems , volume 33, pages 728-740. Curran Associates, Inc., 2020.

Junghyun Lee, Gwangsu Kim, Mahbod Olfat, Mark Hasegawa-Johnson, and Chang D. Yoo. Fast and Efficient MMD-Based Fair PCA via Optimization over Stiefel Manifold. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 36, pages 7363-7371, Jun. 2022.

Xun Li, Dongsheng Chen, Weipan Xu, Haohui Chen, Junjun Li, and Fan Mo. Explainable dimensionality reduction (XDR) to unbox AI 'black box' models: A study of AI perspectives on the ethnic styles of village dwellings. Humanities and Social Sciences Communications , 10(1):35, Jan 2023.

Xin Liang. On the optimality of the Oja's algorithm for online PCA. Statistics and Computing , 33(3): 62, Mar 2023.

Edo Liberty. Simple and Deterministic Matrix Sketching. In Proceedings of the 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , KDD '13, page 581-588, New York, NY, USA, 2013. Association for Computing Machinery.

Ziqi Liu, Yu-Xiang Wang, and Alexander Smola. Fast Differentially Private Matrix Factorization. In Proceedings of the 9th ACM Conference on Recommender Systems , RecSys '15, page 171-178, New York, NY, USA, 2015a. Association for Computing Machinery.

Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep Learning Face Attributes in the Wild. In 2015 IEEE International Conference on Computer Vision (ICCV) , pages 3730-3738, 2015b.

Albert W. Marshall, Ingram Olkin, and Barry C. Arnold. Inequalities: Theory of Majorization and Its Applications . Springer Series in Statistics. Springer New York, 2 edition, 2011.

Ninareh Mehrabi, Fred Morstatter, Nripsuta Saxena, Kristina Lerman, and Aram Galstyan. A Survey on Bias and Fairness in Machine Learning. ACM Computing Surveys , 54(6), jul 2021.

Mirsky. A trace inequality of John von Neumann. Monatshefte für Mathematik , 79(4):303-306, Dec 1975.

Ioannis Mitliagkas, Constantine Caramanis, and Prateek Jain. Memory Limited, Streaming PCA. In Advances in Neural Information Processing Systems , volume 26, pages 2886-2894. Curran Associates, Inc., 2013.

Erkki Oja. Simplified neuron model as a principal component analyzer. Journal of Mathematical Biology , 15(3):267-273, Nov 1982.

Erkki Oja and Juha Karhunen. On Stochastic Approximation of the Eigenvectors and Eigenvalues of the Expectation of a Random Matrix. Journal of Mathematical Analysis and Applications , 106: 69-84, 1985.

Matt Olfat and Anil Aswani. Convex Formulations for Fair Principal Component Analysis. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 33, pages 663-670, Jul. 2019.

Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in PyTorch. In NIPS 2017 Workshop Autodiff , 2017.

Karl Pearson. Liii. on lines and planes of closest fit to systems of points in space. The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science , 2(11):559-572, 1901.

Jeffrey Pennington, Richard Socher, and Christopher Manning. GloVe: Global Vectors for Word Representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages 1532-1543, Doha, Qatar, October 2014. Association for Computational Linguistics.

Andrey Priorov, Kirill Tumanov, Vladimir Volokhov, Evgeny Sergeev, and Ivan Mochalov. Applications of image filtration based on principal component analysis and nonlocal image processing. IAENG International Journal of Computer Science , 40(2):62-80, 2013.

Shauli Ravfogel, Yanai Elazar, Hila Gonen, Michael Twiton, and Yoav Goldberg. Null It Out: Guarding Protected Attributes by Iterative Nullspace Projection. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , pages 7237-7256, Online, July 2020. Association for Computational Linguistics.

Shauli Ravfogel, Michael Twiton, Yoav Goldberg, and Ryan D Cotterell. Linear Adversarial Concept Erasure. In Proceedings of the 39th International Conference on Machine Learning , volume 162 of Proceedings of Machine Learning Research , pages 18400-18421. PMLR, 17-23 Jul 2022a.

Shauli Ravfogel, Francisco Vargas, Yoav Goldberg, and Ryan Cotterell. Adversarial Concept Erasure in Kernel Space. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing , pages 6034-6055, Abu Dhabi, United Arab Emirates, December 2022b. Association for Computational Linguistics.

Yuji Roh, Kangwook Lee, Steven Euijong Whang, and Changho Suh. FairBatch: Batch Selection for Model Fairness. In International Conference on Learning Representations , 2021.

Mark Rudelson and Roman Vershynin. Smallest singular value of a random rectangular matrix. Communications on Pure and Applied Mathematics , 62(12):1707-1739, 2009.

Samira Samadi, Uthaipon Tantipongpipat, Jamie H Morgenstern, Mohit Singh, and Santosh Vempala. The Price of Fair PCA: One Extra dimension. In Advances in Neural Information Processing Systems , volume 31, pages 10999-11010. Curran Associates, Inc., 2018.

Melanie Schmidt, Chris Schwiegelshohn, and Christian Sohler. Fair coresets and streaming algorithms for fair k-means. In Approximation and Online Algorithms , pages 232-251, Cham, 2020. Springer International Publishing.

Bernhard Schölkopf, Alexander Smola, and Klaus-Robert Müller. Nonlinear Component Analysis as a Kernel Eigenvalue Problem. Neural Computation , 10(5):1299-1319, 07 1998.

Shalev-Schwartz and S. Ben-David. Understanding Machine Learning: From Theory to Algorithms . Cambridge University Press, 2014.

Uthaipon Tantipongpipat, Samira Samadi, Mohit Singh, Jamie H Morgenstern, and Santosh Vempala. Multi-Criteria Dimensionality Reduction with Applications to Fairness. In Advances in Neural Information Processing Systems , volume 32, pages 15161-15171. Curran Associates, Inc., 2019.

Erico Tjoa and Cuntai Guan. A Survey on Explainable Artificial Intelligence (XAI): Toward Medical XAI. IEEE Transactions on Neural Networks and Learning Systems , 32(11):4793-4813, 2021.

Joel A. Tropp. An Introduction to Matrix Concentration Inequalities. Foundations and Trends® in Machine Learning , 8(1-2):1-230, 2015.

Enayat Ullah, Poorya Mianjy, Teodor Vanislavov Marinov, and Raman Arora. Streaming Kernel PCA with ˜ O ( √ n ) Random Features. In Advances in Neural Information Processing Systems , volume 31. Curran Associates, Inc., 2018.

John von Neumann. Some Matrix-Inequalities and Metrization of Matrix-Space. Tomsk University Review , 1:286-300, 1937.

Hieu Vu, Toan Tran, Man-Chung Yue, and Viet Anh Nguyen. Distributionally Robust Fair Principal Components via Geodesic Descents. In International Conference on Learning Representations , 2022.

Mingyang Wan, Daochen Zha, Ninghao Liu, and Na Zou. In-Processing Modeling Techniques for Machine Learning Fairness: A Survey. ACM Transactions on Knowledge Discovery from Data , 17 (3), mar 2023.

Chuang Wang and Yue M. Lu. Online learning for sparse PCA in high dimensions: Exact dynamics and phase transitions. In 2016 IEEE Information Theory Workshop (ITW) , pages 186-190, 2016.

Ji Wang, Ding Lu, Ian Davidson, and Zhaojun Bai. Scalable Spectral Clustering with Group Fairness Constraints. In Proceedings of The 26th International Conference on Artificial Intelligence and Statistics , volume 206 of Proceedings of Machine Learning Research , pages 6613-6629. PMLR, 25-27 Apr 2023.

Yanhao Wang, Francesco Fabbri, and Michael Mathioudakis. Streaming Algorithms for Diversity Maximization with Fairness Constraints. In 2022 IEEE 38th International Conference on Data Engineering (ICDE) , pages 41-53, Los Alamitos, CA, USA, 2022. IEEE Computer Society.

Zhiqiang Xu. On the Accelerated Noise-Tolerant Power Method. In Proceedings of The 26th International Conference on Artificial Intelligence and Statistics , volume 206 of Proceedings of Machine Learning Research , pages 7147-7175. PMLR, 25-27 Apr 2023.

Zhiqiang Xu and Ping Li. Faster Noisy Power Method. In Proceedings of The 33rd International Conference on Algorithmic Learning Theory , volume 167 of Proceedings of Machine Learning Research , pages 1138-1164. PMLR, 29 Mar-01 Apr 2022.

Puyudi Yang, Cho-Jui Hsieh, and Jane-Ling Wang. History PCA: A new algorithm for streaming PCA. arXiv preprint arXiv:1802.05447 , 2018.

Wenzhuo Yang and Huan Xu. Streaming Sparse Principal Component Analysis. In Proceedings of the 32nd International Conference on Machine Learning , volume 37 of Proceedings of Machine Learning Research , pages 494-503, Lille, France, 07-09 Jul 2015. PMLR.

Se-Young Yun. Noisy Power Method with Grassmann Average. In 2018 IEEE International Conference on Big Data and Smart Computing (BigComp) , pages 709-712, 2018.

Se-Young Yun, Marc Lelarge, and Alexandre Proutière. Streaming, Memory Limited Algorithms for Community Detection. In Advances in Neural Information Processing Systems , volume 27, pages 3167-3175. Curran Associates, Inc., 2014.

Se-Young Yun, Marc Lelarge, and Alexandre Proutière. Fast and Memory Optimal Low-Rank Matrix Approximation. In Advances in Neural Information Processing Systems , volume 28, pages 3166-3185. Curran Associates, Inc., 2015.

Muhammad Bilal Zafar, Isabel Valera, Manuel Gomez-Rodriguez, and Krishna P. Gummadi. Fairness Constraints: A Flexible Approach for Fair Classification . Journal of Machine Learning Research , 20(75):1-42, 2019.

Rich Zemel, Yu Wu, Kevin Swersky, Toni Pitassi, and Cynthia Dwork. Learning Fair Representations. In Proceedings of the 30th International Conference on Machine Learning , volume 28 of Proceedings of Machine Learning Research , pages 325-333, Atlanta, Georgia, USA, 17-19 Jun 2013. PMLR.

Tianxiang Zhao, Enyan Dai, Kai Shu, and Suhang Wang. Towards Fair Classifiers Without Sensitive Attributes: Exploring Biases in Related Features. In Proceedings of the Fifteenth ACM International Conference on Web Search and Data Mining , WSDM'22, page 1433-1442, New York, NY, USA, 2022. Association for Computing Machinery.

Contents

Appendix

A Broader Impacts, Limitations, and Future Directions

A.1 Broader Impacts

This work proposes a dimensionality reduction method that addresses memory efficiency and fairness while providing a statistical guarantee. By identifying and nullifying the 'unfair direction' inherent in the data distribution, we offer an alternative approach to fair PCA. We anticipate that our approach will motivate researchers to explore other dimensionality reduction techniques ( e.g., auto-encoder) with fairness constraints. Significantly, our contribution includes a rigorous theoretical framework, PAFO learnability, which enables sample complexity analysis of fair PCA. We envision the potential for our theory to be further generalized to broader contexts, including alternative definitions of fairness and optimality of different algorithms for fair machine learning.

On the application side, the memory efficiency of our method can facilitate the scalability of fair PCA, making it viable for processing high-dimensional datasets, even in scenarios where data points arrive in a streaming fashion. One possible application of our approach is data pre-/postprocessing to alleviate unfairness, such as generating fair word embeddings by eliminating sensitive attribute information through orthogonal projection. For more detailed discussions on potential future directions, please refer to below.

A.2 Limitations and Future Directions

Here, we list some of our work's limitations and possible extensions/future directions.

Individual Fairness for PCA. Our definition of fairness in PCA only covers group fairness via fair presentation learning, which was also the case for the prior works (Kleindessner et al., 2023; Lee et al., 2022; Olfat and Aswani, 2019). None of the works, including ours, have yet to consider the notion of individual fairness (Dwork et al., 2012) in the context of PCA, for which we do not have a definitive answer.

Knowledge of sensitive attribute. Our framework requires the full knowledge of the sensitive attribute s for all data points x 's, which was also the case for all the previous works (Kleindessner et al., 2023; Lee et al., 2022; Olfat and Aswani, 2019). But, the assumption of such knowledge may not be feasible in the real world due to privacy or legal reasons (Lahoti et al., 2020; Zhao et al., 2022), or even just due to some extrinsic noises. Considering the case where the dataset may lack sensitive attributes for some or all of the data points is an important future direction.

Making the algorithm anytime. In our formulation of fair PCA, our algorithm is two-phase, with the first phase as a 'burn-in' period for estimating the unfair subspace. Thus, it is not an anytime algorithm in that if the algorithm stops whilst in the first phase, then the resulting V is random and completely uninformative. Designing an anytime variant of our algorithm is an important future direction. One possible way to achieve that is to consider other streaming PCA algorithms such as Oja's method (Oja and Karhunen, 1985) or accelerated NPM (Xu, 2023; Xu and Li, 2022).

Improving gap dependence. Our algorithm's current sample complexity analysis relies on the analysis of NPM by Hardt and Price (2014), which relies on the immediate singular value gap, σ k -σ k +1 . Balcan et al. (2016) showed that considering greater iteration rank q ≥ k , i.e., by considering optimization variable of greater size, leads to a better gap dependency: from σ k -σ k +1 to σ k -σ q +1 . However, this is incompatible with our current definition of PAFO-learnability (Definition 2) because with greater iteration rank, the solution V ⋆ to which the explained variance should be compared against is not clear. The problem is that the sample complexity guarantee of Balcan et al. (2016); Hardt and Price (2014) is derived in terms of the sine angle between the top k -eigenspace of the true covariance and the estimated q -dimensional subspace. Clearing this up would allow for a better theoretical guarantee in sample complexity and thus an important future direction.

Kernelizing our framework Kernel PCA (Schölkopf et al., 1998) is a cornerstone in modern machine learning that has lent itself to be an inspiration to many applications, both theory and practice-wise. Unlike previous fair PCA works (Kleindessner et al., 2023; Olfat and Aswani, 2019), in which the authors provided a kernelized version of their fair PCA algorithm, both our statistical framework and memory-efficient algorithm for streaming setting do not readily extend to the kernelized version. Algorithmically, to tackle the streaming setting, one may take inspiration from streaming kernel PCA (Ghashami et al., 2016; Ullah et al., 2018), which was in turn inspired by matrix sketching (Liberty, 2013).

Extending to other settings. In a similar spirit, extending our formulation of fair PCA to other streaming settings such as sparse (Wang and Lu, 2016; Yang and Xu, 2015), nonstationary (Bienstock et al., 2022), or even distributionally robust settings (Vu et al., 2022) would also be interesting.

More experiments Last but not least, we've only performed experiments on synthetic, CelebA, and UCI datasets. It would be interesting to try our framework (either the statistical formulation, the streaming setting, or both) on datasets from other domains, such as NLP and graphs. GloVe vectors (Pennington et al., 2014) has been used as a benchmark in a similar task called 'concept erasure' (Ravfogel et al., 2020, 2022b); group fairness in spectral clustering of graphs has been studied as well (Kleindessner et al., 2019; Wang et al., 2023).

B Full Pseudocodes of Algorithms 1 and 2 for Fair 'Streaming' PCA

Throughout the appendices, from hereon, we will refer to our algorithms by Algorithms 3 and 4 instead of Algorithms 1 and 2.

C Extension of Algorithms 3 to Multiple and Non-binary Attributes

We remark that Algorithm 3 is a special case of the pseudocode below. By applying Algorithm 4 after applying this algorithm, we can handle the multiple non-binary attribute case and approximately and memory-efficiently solve the extended notion of fair PCA elaborated in Section 5.3.

Algorithm 5: UnfairSubspace_Extended

̸

̸

D More Detailed Comparison to Kleindessner et al. (2023)

D.1 Their Approach

Kleindessner et al. (2023) considered the following formulation of fair PCA:

̸

and proposed a reasonable approximation of the covariance constraint, which we briefly describe here. Let us write S ′ = Σ 1 -Σ 0 for brevity. As implicitly assumed in Kleindessner et al. (2023), let us assume that f = 0 . The mean constraint is dealt with first. Denoting U f ∈ St( d, d -1) to be the matrix whose columns form a basis ( d -1) -dimensional nullspace of f , the mean constraint is satisfied if and only if V is of the form U f U with U ∈ St( d -1 , k ) as the intermediate optimization variable. With this first reparametrization, the optimization now becomes

which now has only the covariance constraint.

To deal with the possible infeasibility of the covariance constraint, Kleindessner et al. (2023) proposed the following approach: letting M f , S ′ ∈ St( d -1 , l ) be the matrix whose columns are the l smallest eigenvectors of U ⊺ f S ′ U f (in magnitude), U only needs to nullify the eigenspace spanned by the remaining d -1 -l eigenvectors. To see which terms are ignored with the relaxation, let ∑ d i =1 q ′ i v ′ v ′ i ⊺ be the eigenvalue decomposition of U ⊺ f S ′ U f with | q ′ 1 | ≥ | q ′ 2 | ≥ ·· · | q ′ d | ≥ 0 . This approach essentially ignores the constraint U ⊺ E l U = 0 , where

The number l ∈ { k, · · · , d -1 } controls how much the group-conditional covariances will be equalized; a smaller l means that the covariance constraint is enforced more stringently and vice versa. Ultimately, the learner can control l as a hyperparameter to create a trade-off between fairness and the explained variance. Moreover, if | q ′ i | 's are negligible for all i ≥ d -1 -l , then M ⊺ f , S ′ U ⊺ f S ′ U f M f , S ′ ≈ 0 , and the relaxation becomes tighter.

As any U of the form M f , S ′ Λ satisfies the relaxed covariance constraint, a second reparameterization in terms of the new optimization variable Λ ∈ St( l, k ) gives us

which can be solved via the standard SVD-based approach for vanilla PCA. Then, the final solution is obtained as V ∗ = M f , S ′ U f Λ ∗ .

D.2 Unsuitability for the Streaming Setting

In Section 5.2, we provided a rough overview of why existing approaches to fair PCA (Kleindessner et al., 2023; Lee et al., 2022; Olfat and Aswani, 2019) are not amenable to our streaming setting. Especially as the approach of Kleindessner et al. (2023) (described above) is almost like a PCA, one may wonder if standard techniques used in streaming PCA (Mitliagkas et al., 2013) can be used. Here, we argue in detail why that is not the case, which is in sharp contrast to our approach and our reformulation of fair PCA that led to a memory-efficient fair streaming PCA algorithm.

Memory constraint. For streaming PCA without fairness constraints, the main objective could be written as tr( V ⊺ X ⊺ XV ) = ∑ N i =1 tr ( V ⊺ x i x ⊺ i V ) , which is easily amenable to memory-limited algorithm such as noisy power method (Mitliagkas et al., 2013) or stochastic optimization (Oja, 1982). Both approaches utilize the fact that instead of storing d × d matrices, one only needs to store matrixvector product of size d × 1 , e.g., V ⊺ x i . This is not the case for the approach of Kleindessner et al. (2023). To see this, consider Eqn. (14) without the covariance constraint, i.e., fair PCA with only the mean constraint. Even here, although the objective can be written as ∑ N i =1 tr ( U ⊺ U ⊺ f x i x ⊺ i U f U ) ,

we must know U f in order to proceed further. Moreover, as U f is of size O ( d 2 ) , it cannot be stored nor estimated explicitly. When the covariance constraint is also taken into account, although the matrix in question M f , S ′ is of dimension ( d -1) × l , which is within the memory constraint if l = O (1) , the computation of M f , S ′ requires the knowledge of U f . This is because Kleindessner et al. (2023) dealt with the two constraints sequentially (mean first, then covariance), which forced the reparametrization to be done twice and, more importantly, coupling the memory requirement for the computation of U f and M f , S ′ .

Statistical consideration. The statistical guarantee (global convergence, sample complexity) of streaming PCA algorithms is often obtained by appropriately bounding the error term and using proper matrix concentration inequalities (Tropp, 2015). For example, for the sample complexity guarantee of noisy power method (Balcan et al., 2016; Hardt and Price, 2014), as the learner performs power method on the empirical covariance ∑ i x i x ⊺ i instead of the true covariance Σ , the proof proceeds by first showing that the empirical covariance is close enough to the true covariance (i.e., the norm of their error term is sufficiently bounded), which then implies that the variance of the estimation isn't too high. Thus to analyze the error bound of the final iterates V when the approach of Kleindessner et al. (2023) is extended to a streaming setting, one would have to bound the estimation error of M ⊺ f , S ′ U ⊺ f Σ U f M f , S ′ . There are three sources of estimation error: M f , S ′ , U ⊺ f , Σ . Recalling that M f , S ′ consists of the eigenvectors of U ⊺ f S ′ U f , one can see that the estimation error of M f , S ′ is actually nontrivially dependent on the estimation quality of both U and S ′ . Here, we say nontrivially because the error isn't simply bounded in a linear sense via the usual triangle inequality; it requires rather intricate techniques involving eigenvector perturbation theory (Davis and Kahan, 1969; Golub and Loan, 2013), which may require additional assumptions on eigenvalue gaps. As one can see later in the proof, our approach considers both constraints simultaneously and thus allows for a quite simple theoretical analysis.

E Proofs of Lemma 2, 3, and Theorem 1

E.1 Notations and Assumptions

We first recall the data generation process of our interest. At every time step, a pair of a sensitive attribute and a data point is sampled as a ∼ Bernoulli( p ) and x | a ∼ D a , respectively, where D a has mean µ a and covariance Σ a . This can be written more compactly as x ∼ D := p 0 D 0 + p 1 D 1 , where we denote p 0 := 1 -p and p 1 := p .

Werecall a set of notation needed for the proof. The mean difference is denoted by f = µ 1 -µ 0 , while the second moment difference is S = E [ xx ⊺ | a = 1] -E [ xx ⊺ | a = 0] = Σ 1 -Σ 0 + µ 1 µ ⊺ 1 -µ 0 µ ⊺ 0 . Let P m ∈ R d × m be the matrix whose columns are top m eigenvectors of S in magnitude of eigenvalues. Note that t ∈ [ T ] is the time variable for Algorithm 3, which we often omit if the context is clear. Moreover, b (without any sub-/superscript) is the block size of Algorithm 3. The remaining notation is listed below; most of them came from the pseudocode of our algorithms.

Counts of data points for each sensitive attribute a :

Estimates of group-conditional means:

Estimate of the group-conditional mean gap: ̂ f = m (1) -m (0)

Estimate of the group-conditional second moments:

Estimate of the group-conditional second moment gap: ̂ S t = A (1) t -A (0) t

W t ∈ St( d, n ) (decision variable of Algorithm 1 and 3)

C ( a ) t = A ( a ) t W t -1

F t = C (1) t -C (0) t = ̂ S t W t -1

W t = QR ( F t ) ( i.e. , F t QR = W t R t , R t ∈ R m × m is a lower triangular matrix)

Recall the assumptions on the data distribution, which we will assume throughout the proof:

Assumption 1. Consider our data generation process a ∼ Bernoulli( p ) and x | a ∼ D a . There exists σ, V, M , V , g min > 0 , f max ∈ ( g min , ∞ ) , and p min ∈ (0 , 0 . 5] such that the followings hold for any ( D 0 , D 1 , p ) ∈ F d : for a ∈ { 0 , 1 } , D a ∈ nSG( σ ) , 4

and

Assumption 2. Fix m,k ∈ N . There exist ∆ m,ν , ∆ k,κ , K m,ν , K k,κ ∈ (0 , ∞ ) such that for any ( D 0 , D 1 , · ) ∈ F d , the followings hold:

where ν 1 ≥ · · · ≥ ν d ≥ 0 and κ 1 ≥ · · · ≥ κ d ≥ 0 are the singular values of S and Π ⊥ U ΣΠ ⊥ U , respectively.

We provide some intuition on the assumptions that we impose here. Assumption 1 consists of three parts. The first part, which involves σ, V, M and V , ensures that the maximum deviation in mean and covariance of each D a are well bounded; this is critical in allowing for us to use proper matrix concentration inequalities (to be described in the next subsection) and has been used in various streaming PCA literature (Bienstock et al., 2022; Huang et al., 2021; Jain et al., 2016). The second part, which involves g min and f max , imposes a bound on the maximum mean separation, f = µ 1 -µ 0 , ℓ 2 -wise and angle-wise, respectively. If the mean difference can be arbitrarily large, then the ℓ 2 -estimation error that one has to achieve becomes arbitrarily small; precisely speaking, ∥ f ∥ 2 acts as a Lipschitz constant. On the other hand, if the mean difference can be arbitrarily small, then the angle-wise estimation error becomes arbitrarily large. The last part, which involves p min , ensures that both groups are selected with some positive, nonvanishing probability. Assumption 2 is standard in streaming PCA literature to ensure convergence; indeed, if the singular value gap is zero, a definitive convergence result can never be obtained, as the ground-truth solution becomes vague.

E.2 Matrix/Vector Concentration Inequalities

Before moving on to our proof, we review some useful concentration inequalities for our theoretical analysis.

Definition 4 (Variance of random matrix) . For a zero-mean random matrix Z , its variance is defined as

In general, Var( Z ) = Var( Z -E [ Z ])

Proposition 1 (Matrix Bernstein inequality (Theorem 6.6.1 of Tropp (2015))) . Consider a finite collection { Y j } b j =1 of independent matrices with the same size ( d 1 × d 2 ). Suppose they are zero mean and they have uniformly bounded singular values, i.e.,

Let V be an upper bound of matrix variance, V ≥ Var( Y j ) for all j ∈ [ b ] . Then, for all x ≥ 0 ,

In particular, if 0 ≤ x ≤ 3 V M ,

Proposition 2 (Vector Hoeffding inequality (Corollary 7 of Jin et al. (2019))) . There exists an absolute constant c such that if y 1 , · · · , y b be independent random vectors with common dimension d , and assume the following:

Remark 2. With an additional assumption that the random variables are bounded, we can also consider using vector Bernstein inequality (Lemma 18 of Kohler and Lucchi (2017)), which removes the factor of dimension d from the RHS of the concentration inequality.

E.3 Proof of Lemma 2 - Bounding Error in Second Moment Gap

Recall that Algorithm 3 performs NPM on the second moment difference matrix S so that an iteration can be written as

where Z t, 1 = ( ̂ S t -S ) W t -1 is the noise matrix.

The lemma below provides a general statement on sufficient size b of blocks to bound the second moment difference error ̂ S t -S multiplied by some fixed matrices.

Then, for any x ≥ 0 ,

Lemma 4. Let any δ > 0 and matrices M ∈ R m × d and N ∈ R d × n be given. If we choose b ≥ 4 p min max ( 1 , 8 M 2 9 V ) log 2( m + n ) δ , then with probability at least 1 -δ ,

We now proceed the proof of Lemma 2. To this end, recall that Z t, 1 = ( ̂ S t -S ) W t -1 . Firstly, applying Lemma 4 with M = I d and N = W t -1 ∈ R d × m , a sufficient condition for having 5 ∥ Z t, 1 ∥ 2 ≤ ϵ ∆ m,ν with probability at least 1 -δ is that

Secondly, applying Lemma 4 with M = P ⊺ m ∈ R m × d and N = W t -1 ∈ R d × m , a sufficient condition for having 5 ∥ P ⊺ m Z t, 1 ∥ 2 ≤ δ ∆ m,ν 2 √ dm with probability at least 1 -δ is that

Combining all bounds for b , we conclude that a sufficient block size b is

Proof of Lemma 4. Note that ( a i , x i ) 's are i.i.d. samples ( i ∈ { ( t -1) b + 1 , . . . , tb } ). Consider independent random matrices

For each set A ⊆ [ b ] , define an event E A := { a i = ✶ [ i ∈ A ] ∀ i ∈ [ b ] } . Note that P [ E A ] = p b -| A | 0 p | A | 1 . To exploit this, we first apply a peeling argument as follows: for any c 0 , c 1 ∈ (0 , 1) with c 0 + c 1 = 1 ,

where the last inequality holds for the following fact: if ∥ a -b ∥ ≥ x then ∥ a ∥ ≥ λx or ∥ b ∥ ≥ (1 -λ ) x for any λ ∈ (0 , 1) .

Now, we make the crucial observation that conditioned on the event E A , both { Y (1) i } i ∈ A and { Y (0) i } i ∈ A c are sets of i.i.d. random zero-mean m × n matrices. In addition, with Assumption 1, we have that for each i ∈ A ,

(and analogously for each i ∈ A c ). Thus, applying the matrix Bernstein inequality 5 (Proposition 1), we obtain the following: when 0 ≤ x ≤ 3 V M ,

For this to occur, it suffices for both terms to be bounded by δ 2 . Let us consider only the first term, as the second term follows from symmetry. Taking the log, we have

Using log(1 + y ) ≤ y , it suffices to have

Using y ≤ 1 -e -2 y for y ∈ [0 , 1 / 2] , it now suffices to have

Combining this with the other term and, for simplicity 6 choosing c 0 = c 1 = 1 / 2 , we have our desired statement.

E.4 Proof of Lemma 3 - Bounding the Final Error

Recall that the iterates are V τ +1 = QR ( Π ⊥ ̂ U ̂ Σ τ Π ⊥ ̂ U V τ ) , where ̂ U is the output of Algorithm 3, Π ⊥ ̂ U := I d -̂ U ̂ U ⊺ , and ̂ Σ τ := 1 B ∑ τ B j =( τ -1) B +1 ˜ x j ˜ x ⊺ j is the sample covariance at time step τ of Algorithm 4. This can be rewritten as

where Z τ, 2 = ( Π ⊥ ̂ U ̂ Σ τ Π ⊥ ̂ U -Π ⊥ U ΣΠ ⊥ U ) V τ is the noise matrix and Σ = ∑ a ∈{ 0 , 1 } p a ( Σ a + µ a µ ⊺ a ) . By Assumption 1, we have that ∥ Σ ∥ 2 ≤ V .

The lemma below provides a general statement on a sufficient block size B to bound the covariance error ̂ Σ -Σ multiplied by some fixed matrices.

Lemma 5. Let any δ > 0 and matrices M ∈ R m × d and N ∈ R d × n be given. If we choose B ≥ 4 9 M 2 V + V 2 log 4( m + n ) δ , then the following holds with probability at least 1 -δ :

We now proceed the proof of Lemma 3. Fix some V ∈ St( d, k ) , and from the design of our Algorithm 4, ̂ U is also fixed. By our given assumption on upper bound of ∥ ∥ Π ̂ U -Π U ∥ ∥ 2 , and the fact that ∥ V ∥ 2 = ∥ Q k ∥ 2 = 1 , we have

and, with a similar logic,

Applying Lemma 5 with M = Π ⊥ ̂ U ∈ R d × d and N = Π ⊥ ̂ U V ∈ R d × k , a sufficient condition for having ∥ ∥ ∥ Π ⊥ ̂ U ( ̂ Σ -Σ ) Π ⊥ ̂ U V ∥ ∥ ∥ 2 ≤ ϵ ∆ k,κ 10 with probability at least 1 -δ is that

Furthermore, applying Lemma 5 with M = Q ⊺ k Π ⊥ ̂ U ∈ R k × d and N = Π ⊥ ̂ U V ∈ R d × k , a sufficient condition for having ∥ ∥ ∥ Q ⊺ k Π ⊥ ̂ U ( ̂ Σ -Σ ) Π ⊥ ̂ U V ∥ ∥ ∥ 2 ≤ δ ∆ k,κ 20 √ dk with probability at least 1 -δ is that

Combining all bounds for B , we conclude that a sufficient block size B to have our desired statement is

Proof of Lemma 5. Note that ˜ x j 's are i.i.d. samples from D ( j ∈ { ( τ -1) B +1 , . . . , τ B} ) . Let Y j = M ( ˜ x j ˜ x ⊺ j -Σ ) N .

Then, Y i are i.i.d. random zero-mean d × k matrices across j . In addition, with Assumption 1,

and

Applying the matrix Bernstein inequality (Proposition 1), we obtain the following: when 0 ≤ x ≤ 3( V + V 2 ) M ,

Solving for x , we have

from which the statement naturally follows.

E.5 Bounding the Estimation Error of Mean Difference

Lemma 6. For any δ > 0 , choose b ≥ 2 Tp min log 4 d δ . Then, the following holds with probability at least 1 -δ :

where c

Proof. Again, note that ( a i , x i ) 's are i.i.d. samples ( i ∈ [ Tb ] ). Consider independent random vectors

For each A ⊆ [ Tb ] , E A = { a i = ✶ [ i ∈ A ] ∀ i ∈ [ b ] } is an event satisfying P ( E A ) = p Tb -| A | 0 p | A | 1 . To exploit this, we now apply the peeling argument as follows: for any c 0 , c 1 ∈ (0 , 1) with c 0 + c 1 = 1 ,

Observe that conditioned on the event E A , both { y (1) i } i ∈ A and { y (0) i } i ∈ A c are sets of i.i.d. random zero-mean d -dimensional vectors. In addition, with Assumption 1, we have that for each i ∈ A , y (1) i ∈ nSG( σ 1 ) (and analogously for each i ∈ A c ). Thus, applying the vector Bernstein inequality (Proposition 2), we obtain the following: for x > 0 ,

It suffices for each of both terms to be bounded by δ 2 . Let us consider only the first term, as the second term follows from symmetry. Taking the log, we have

Using log(1 + y ) ≤ y , it suffices to have

Using y ≤ 1 -e -2 y for y ≤ 1 / 2 , it now suffices to have

Combining this with the other term and, for simplicity (see the footnote on pg. 25), choosing c 0 = c 1 = 1 / 2 , we have our desired statement.

E.6 Proof of Theorem 1 - Sample Complexity for PAFO-learnability

Let us fix ε f , ε o , δ ∈ (0 , 1) . From the definition of PAFO-learnability (Definition 2), with a large enough sample size, we must guarantee the following with probability at least 1 -δ :

Let ̂ U be the final estimate of the unfair subspace U from Algorithm 1, and let V be the final estimate of Q k = V ⋆ , whose columns are top k eigenvectors of Π ⊥ U ΣΠ ⊥ U .

We first recall the von Neumann trace inequality (Mirsky, 1975; von Neumann, 1937):

Proposition 3 (Theorem H.1.g of Marshall et al. (2011)) . If A , B are two n × n Hermitian matrices, then

where λ i ( · ) is the i -th smallest eigenvalue.

Also, the following lemma implies that the sine angle between subspaces (with the same dimension) and the corresponding Grassmannian (projection) distance are the same.

Lemma 7. Let A 1 ∈ R d × k and B 1 ∈ R d × ℓ be semi-orthogonal matrices: A ⊺ 1 A 1 = I k and B ⊺ 1 B 1 = I ℓ . Define

Then, (i) if k = ℓ ,

where σ min ( M ) is the minimum singular value of M . On the other hand, (ii) if k < ℓ ,

Now we can write the optimality (the first inequality) as follows:

Here, ( a ) follows from the von Neumann trace inequality, ( b ) follows from the fact that Q k Q ⊺ k -V V ⊺ is a symmetric matrix of rank at most 2 k , and ( c ) follows from Lemma 7. Thus, for the optimality, it suffices to ensure that ∥ Π ⊥ V Q k ∥ 2 ≤ ε f 2 kV .

Similarly, for the fairness (the second inequality), it suffices to ensure that ∥ Π U Π ⊥ ̂ U ∥ 2 ≤ ε o or ∥ Π ̂ U -Π U ∥ 2 ≤ ε o , as

where ( ∗ ) follows from V = Π ⊥ ̂ U V (due to the design of Algorithm 4) and ( ⋆ ) follows from Lemma 7.

From Lemma 1 (letting T = Θ ( K k,κ ∆ k,κ log d ϵδ ) ) and 3 (substituting ϵ ↦→ ε f 2 kV ), given that

a total of N 2 = T B samples are sufficient in Algorithm 4 for ensuring ε f -optimality with probability at least 1 -δ 2 , where

Wenowfocus on obtaining the sample complexity for Algorithm 3 to satisfy Eqn. (17) with probability at least 1 -δ 2 . Then combining those with a union bound gives us the desired statement.

From hereon and forth, we write

̸

Also, we introduce the following lemma that provides a general bound of sine angle between a pair of vectors with a small ℓ 2 -distance.

Lemma 8. Consider two vectors a , b ∈ R d . If we denote the (acute) angle between a and b as θ ( a , b ) ,

Consider any ϵ ∈ ( 0 , ∥ a ∥ 2 2 ] and suppose ∥ a -b ∥ 2 ≤ ϵ . Then,

Case 1. Assume g = 0 . In this case, U = [ P m | h ] , ∥ g ∥ 2 ≥ g min , and ∥ f ∥ 2 ≤ f max by Assumption 1. We want to upper-bound the following probability to be < δ/ 2 :

̸

̸

Let us obtain a sample complexity to bound the first probability term to be < δ 4 . Note that

Firstly, to yield

with probability at least 1 -δ 12 , it is sufficient that

due to application of Lemma 1 and 2.

Secondly, we aim to make the second term ∥ ∥ ∥ hh ⊺ -̂ h ̂ h ⊺ ∥ ∥ ∥ 2 small with high probability. We first claim that, if ∥ g -̂ g ∥ 2 ≤ g min 2 min { η k √ 2 , 1 } , then ∥ ∥ ∥ hh ⊺ -̂ h ̂ h ⊺ ∥ ∥ ∥ 2 ≤ η k 2 . Here is the proof of the claim: Since ∥ g -̂ g ∥ 2 ≤ ∥ g ∥ 2 2 , we have

Thus, since we have

To have

̸

with probability at least 1 -δ 12 , it is sufficient that

Moreover, to have

̸

̸

with probability at least 1 -δ 12 , it is sufficient that

Next, we obtain a sample complexity to bound the second probability term P [ ̂ g = 0 ] < δ 4 . Because of Eqn. (21),

̸

To have ∥ Π P m -Π W T ∥ 2 < g min 2 f max with probability at least 1 -δ 8 , it suffices to have a sample complexity in Eqn. (22) and (23). Also, to have ∥ ∥ ∥ f -̂ f ∥ ∥ ∥ 2 < g min 2 with probability at least 1 -δ 8 , it suffices to have a sample complexity in Eqn. (24).

Combining the bounds in the Case 1 with a union bound, we obtain a full sample complexity for Algorithm 3 as follows: a total of N (1) 1 = Tb samples are sufficient for ensuring ( ε f , ε o ) -PAFOlearnability with probability at least 1 -δ , where

Case 2. Assume g = 0 . In this case, U = P m . We want to upper-bound the follow probability to be < δ/ 2 :

̸

̸

The first probability term is bounded by δ 4 given that a sufficient number of samples just like Eqn. (19) and (20). To bound the second one, note that ̂ h is orthogonal to every column of W T ( i.e., Π W T ̂ h = 0 ). Thus,

which can be bounded by < η k with probability at least 1 -δ 4 given that a sufficient number of samples just like Eqn. (19) and (20) again. Therefore, a total of N (2) 1 = Tb samples are sufficient for ensuring ( ε f , ε o ) -PAFO-learnability with probability at least 1 -δ , where

̸

Combining Case 1 (Eqn. (25)) and Case 2 (Eqn. (26)) as N 1 = ✶ [ g = 0 ] N (1) 1 + ✶ [ g = 0 ] N (2) 1 , we have our desired statement. Lastly, we provide the missing proof for our lemmas:

Proof of Lemma 7. Let

be d × d orthogonal matrices. We first show that sin (col( A 1 ) , col( B 1 )) = ∥ A ⊺ 2 B 1 ∥ 2 . It can be proven as

Thus, from now on, we will work with ∥ A ⊺ 2 B 1 ∥ 2 instead of sin (col( A 1 ) , col( B 1 )) .

The rest of the proof borrows the idea from the proof of Golub and Loan (2013, Theorem 2.5.1). Observe that

Note that A ⊺ 2 B 1 and A ⊺ 2 B 1 are submatrices of a d × d orthogonal matrix Q defined as

Readers might notice that A ⊺ 1 B 1 ∈ R k × ℓ . For any unit vector (in 2-norm) x ∈ R k ,

Thus,

Likewise, working with Q ⊺ , one can deduce that

Therefore, our desired statement naturally follows.

Proof of Lemma 8. To prove the first claim, let u and v be unit ( ℓ 2 norm) vectors in direction of a and b , respectively. Then, by Lemma 7,

Now, observe that ∥ b ∥ 2 ≥ ∥ a ∥ 2 -ϵ ≥ ∥ a ∥ 2 / 2 and

Thus,

F More Experiments

F.1 Synthetic Example

5

0

5

7.5

5.0

2.5

0.0

2.5

5.0

7.5

Vanilla NPM

Group 0

Group 1

5

0

5

7.5

5.0

2.5

0.0

2.5

5.0

7.5

Mean-matched NPM

Group 0

Group 1

5

0

5

7.5

5.0

2.5

0.0

2.5

5.0

7.5

Fair NPM

Group 0

Group 1

We randomly generated two different group-conditional distributions as 10-dimensional multivariate Gaussians with different mean vectors µ 0 and µ 1 , satisfying µ = (1 -p ) µ 0 + p µ 1 = 0 , and different covariance matrices Σ 0 and Σ 1 . We choose the sampling probability parameter p as 0.2, which induces asymmetric sampling between two sensitive attributes. The covariance matrices are designed so that both of their eigenvalue spectra, { σ 1 , . . . , σ d } , have power-law decay as many practical datasets do (Liu et al., 2015a), i.e., σ j = Θ( j -α ) for some decay parameter α ≥ 1 .

We first run and compare three different algorithms: vanilla NPM (without any fairness constraint), mean-matched NPM (with only constraint V ⊺ f = 0 ), and FNPM with m = 3 . To ease the visualization, we project the sampled distributions onto a 2-dimensional subspace ( i.e., running 2-PCA). After running three algorithms for ten iterations and with a block size of b = B = 1000 , we randomly sample 1000 data points and visualize the results of projecting the data points in Figure 2. In particular, for FNPM, we run 50 iterations for unfair subspace estimation (Algorithm 3) and run the other 50 iterations for PCA (Algorithm 4). We observe that FNPM does indeed enforce both mean-matching and (partial) covariance-matching, despite the setting being streaming and having asymmetric sampling probability.

F.2 Additional Results on CelebA Dataset

In Figures 3-5, we provide additional experimental results on the CelebA dataset (Liu et al., 2015b). We consider three sensitive attributes in the CelebA dataset: 'Eyeglasses', 'Mouth Slightly Open', and 'Goattee'. In all the figures, False is when the sensitive attribute is absent; True is otherwise.

Figure 3 presents the main results for the new attributes. The first row shows the original images, the second row shows the images after projecting them to col( V ⋆ ) , and the third row shows the images after projecting them to the estimated unfair subspace, col( ̂ U ) . Here, both ̂ U and V ⋆ are obtained from our FNPM; specifically speaking, they are obtained from Algorithm 3 and Algorithm 4, respectively. While we've used m = 2 for Figure 1 in the main text, we use m = 5 here.

We then perform the two ablation studies as described in the main text, one on the dimension m of unfair subspace and another on the block size b of Algorithm 3, for the additional sensitive attributes considered. In Figure 4, we vary m ∈ { 1 , 2 , 5 , 10 } : as m increases, more features of images are 'erased', making the images from the two sensitive groups less distinguishable. At the same time, more semantically meaningful features are erased as well, resulting in rather 'alien-like' images. In Figure 5, we vary b ∈ { 32000 , 8000 , 3200 , 1600 } : as b increases, we have a more accurate estimation of unfair subspace col( U ) , resulting in more indistinguishable images (in sensitive attributes). As soon as the batch size b exceeds a certain threshold (e.g., 8000 for 'Goattee'), col( U ) is estimated very well, and the unfair features are cleanly recovered, as it can be seen in the bottom rows.

F.3 Full Results on UCI Dataset

The full experiment results of fair PCA and downstream tasks on UCI datasets are provided. Again, our method is competitive to the existing fair PCA algorithms across the considered UCI datasets.

Original

Fair NPM

(Unfair subsp.)

Original

Fair NPM

(Unfair subsp.)

Original

Fair NPM

(Unfair subsp.)

BJoRK

Span

Vanilla

mean & Cov (1)

mean & Cov (2)

mean & Cov (5)

mean & Cov (10)

(original)

Orthogonal

Span

Vanilla

mean & Cov (1)

mean & Cov (2)

mean & Cov (5)

mean & Cov (10)

(original)

Orthogonal

Span

Vanilla

mean & Cov (1)

mean & Cov (2)

mean & Cov (5)

mean & Cov (10)

(original)

Orthogonal

Span

Vanilla

mean & Cov (1)

mean & Cov (2)

mean & Cov (5)

mean & Cov (10)

(original)

Orthogonal

Span

Vanilla

mean & Cov (1)

mean & Cov (2)

mean & Cov (5)

mean & Cov (10)

(original)

Orthogonal

Span

Vanilla

mean & Cov (1)

mean & Cov (2)

mean & Cov (5)

mean & Cov (10)

(original)

Orthogonal

Span

Vanilla

mean & Cov (1)

mean & Cov (2)

mean & Cov (5)

mean & Cov (10)

(original)

Orthogonal

Span

Vanilla

mean & Cov (1)

mean & Cov (2)

mean & Cov (5)

mean & Cov (10)

(original)

Orthogonal

Span

Vanilla

mean & Cov (1)

mean & Cov (2)

mean & Cov (5)

mean & Cov (10)

(original)

Orthogonal

Span

Vanilla

mean & Cov (1)

mean & Cov (2)

mean & Cov (5)

mean & Cov (10)

(original)

Orthogonal

Span

Vanilla

mean & Cov (1)

mean & Cov (2)

mean & Cov (5)

mean & Cov (10)

Orthogonal

Span

Vanilla

mean & Cov (1)

mean & Cov (2)

mean & Cov (5)

mean & Cov (10)

Orthogonal

(original)

(c) Attribute: 'Goatee' (Left: False, Right: True)

(original)

Span

Vanilla

b=32000

b=8000

b=3200

b=1600

(original)

Orthogonal

Span

Vanilla

b=32000

b=8000

b=3200

b=1600

(original)

Orthogonal

Span

Vanilla

b=32000

b=8000

b=3200

b=1600

(original)

Orthogonal

Span

Vanilla

b=32000

b=8000

b=3200

b=1600

(original)

Orthogonal

Span

Vanilla

b=32000

b=8000

b=3200

b=1600

(original)

Orthogonal

Span

Vanilla

b=32000

b=8000

b=3200

b=1600

(original)

Orthogonal

Span

Vanilla

b=32000

b=8000

b=3200

b=1600

(original)

Orthogonal

Span

Vanilla

b=32000

b=8000

b=3200

b=1600

(original)

Orthogonal

Span

Vanilla

b=32000

b=8000

b=3200

b=1600

(original)

Orthogonal

Span

Vanilla

b=32000

b=8000

b=3200

b=1600

(original)

Orthogonal

Span

Vanilla

b=32000

b=8000

b=3200

b=1600

(original)

Orthogonal

Span

Vanilla

b=32000

b=8000

b=3200

b=1600

(original)

Orthogonal

(c) Attribute: 'Goatee' (Left: False, Right: True)

Peer reviews:
comments: SUMMARY:
This paper studies fair PCA problem. The authors provide a novel formulation of fair PCA based on the "Null it Out" approach and propose the corresponding criterion called PAFO-learnability. The authors also present a streaming algorithm for fair PCA, which has low memory complexity. Experimental results verify the scalability of the proposed method.

STRENGTHS:
1. The paper is well-written and mostly clear.
2. The studied problem is interesting and important. Streaming algorithms are very useful in the limited-memory setting.

WEAKNESSES:
1. Assumption 6.1 is a bit strong to me. The paper may show that when D_s is a common distribution such as sub-Gaussian distribution, the generated data satisfies assumption 6.1.
2. I'm worried about the tightness of theorems in section 6. The theoretical results show that Alg 1 and Alg 2 may require very large block sizes, which reduces the contirbution of saving memory.
3. The experiments part may present some quantitative results rather than just show the images. The expriments should verify that proposed FNPM algorithm is PAFO-learnability.

QUESTIONS:
1. Why this paper proposes a new formulation of fair PCA rather than use a existing one? What's the advantage of "Null it out" formulation?
2. Do existing fair PCA algorithms satisfy PAFO-Learnability?

LIMITATIONS:
The paper has no potential negative societal impact.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new approach for Fair PCA algorithms that is scalable and fair at the same time. The main contributions of the paper are as follows:\n\n- A new formulation of fair PCA based on the \"Null It Out\" approach. The goal is to maximize explained variance while nullifying the subspace spanned by the mean difference and leading eigenvectors of the covariance difference between groups. This formulation leads to a closed-form solution and avoids infeasibility issues in previous covariance matching approaches. Their approach removes the unfair subspaces using a noisy power method.\n\n- A new notion of learnability for fair PCA called Probably Approximately Fair and Optimal (PAFO)-learnability. This provides a statistical framework to analyze some of the fair PCA algorithms.\n\n- A new setting called fair streaming PCA, which addresses practical memory limitations. The authors propose an algorithm called Fair Noisy Power Method (FNPM) which only requires O(dk) memory, where d is data dimension and k is the target PCA dimension.\n\n- The empirical study of this method using a vision task is intuitive.\n\nSTRENGTHS:\nI believe the strengths of this paper can be summarized in three main key points:\n\n- Theoretical rigor. The paper provides the first statistical framework for analyzing fair PCA in terms of PAFO-learnability. This gives theoretical guarantees on the solution quality of algorithms like FNPM. Previous works mainly lacked such a framework.\n\n- They propose  FNPM for this problem, which is quite simple to implement, building on standard tools like cumulative averaging and the noisy power method. This makes it easy to apply in practice.\n\n- Their approach is scalable and intuitive. They have validated that with a vision task to demonstrate the scalability. Also, the formulation based on nullifying the \"unfair\" subspace gives flexibility in how much fairness to impose by choosing m. This can be tuned based on the use case. The analysis provides insights into how properties like the singular value gaps and mean difference norm affect the sample complexity and solution quality.\n\nWEAKNESSES:\nThere are some main concerns I have regarding this paper:\n\n- The results are limited to binary sensitive attributes and two groups. Extending the approach to handle more complex, multi-group scenarios with sensitive feature interactions would strengthen the paper. Some of the main previous approaches can easily handle this problem as well [A,B].\n\n- Certain assumptions made, like those in Assumptions 6.1 and 6.2, are quite strong and restrictive. Relaxing these assumptions, or testing how sensitive the approach is to their violation would improve the robustness.\n\n- The experiments focus on how much FNPM removes features visually related to sensitive attributes. Evaluating the fairness of solutions in a more quantitative, metric-based fashion would provide a more objective assessment. This quantity is defined differently in various approaches.\n\n- Lack of comparison with the other branch of fair PCA methods. Although the goal of equalizing losses is different than what presented here, it would be more beneficial to better understand how different methods would compare.\n\n- Comparing the effects of this fair PCA on downstream tasks like classification can be beneficial to better understand the effects of a fair PCA approach.\n\n\n\n[A] Morgenstern, Jamie, et al. \"Fair dimensionality reduction and iterative rounding for sdps.\" arXiv 2019 (2019).\n\n[B] Kamani, Mohammad Mahdi, et al. \"Efficient fair principal component analysis.\" Machine Learning (2022): 1-32.\n\nQUESTIONS:\nSee the previous part\n\nLIMITATIONS:\nSome of the limitations I discussed in the weakness section are not clearly discussed in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis research paper focuses on the fair principal component analysis (PCA) problem using streaming data while requiring low memory. The authors introduce a new formulation for fair PCA, which involves optimizing the vanilla PCA objective with a linear \"fair\" constraint. In the oracle setting, where the true parameter is known, the problem has a closed-form solution. The authors define the concept of \"PAFO-learnable\" to quantify the sample complexity of learning a semi-orthogonal matrix V, which is an approximately optimal solution to the oracle problem. They present streaming algorithms and prove that such an algorithm scheme has finite sample complexity according to the proposed PAFO-learnable notion. The main idea is to utilize the noisy power method framework to estimate the unfair subspace (the linear constraint) and subsequently employ this estimation in the NPM (Noisy Power Method) to estimate fair PCA. The effectiveness of the proposed method is evaluated using real-world data.\n\nSTRENGTHS:\nOverall, the contribution of the paper is well-motivated and aligns with the ongoing development of methods for problems involving fair constraints. The paper is well-written, and the investigation is quite extensive. However, I must admit that I am not familiar with recent developments in fairness in machine learning and cannot provide an assessment of the significance of the new formulation Equation (1) and the concept of learnability (Definition 4.2), although they appear reasonable and interesting.\n\nWEAKNESSES:\nI have a few minor comments:\n\n* L68: It might be more appropriate to use the term \"semi-orthogonal matrix\" instead of \"orthogonal\" to distinguish between O(d) and St(d,k).\n* L69, L116, L171: QR decomposition typically yields two outputs, namely the (semi-)orthogonal part and the upper triangular part.\n* L164: Regarding F_d, it differs from the one defined in Definition 4.2, where learnability is defined for a different quantity denoted as F_d.\n* L262: Once again, F_{d,m,k} is inconsistent with the definition provided in Definition 4.2.\n\nQUESTIONS:\nSee above.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-10"}

comments: SUMMARY:
The paper defines a new notion for fair pca. It is assumed that the data comes from  mixture of (two) distributions and the goal is to find a subspace for it such that the solution subspace is perpendicular to 

1) the difference in the difference vector of the mean
2) the top m eigen vectors of the difference of the variance

This problem can simply be computed by computing PCA on a projected space. This paper looks for an algorithm that is 

1) with probability 1-delta, reports an approximate solution.
2) Works in the streaming setting where once gets samples from the mixture of the distributions and uses space which depends on O(kd).

The algorithm is to use the samples to estimate the parameters of the two distributions and thus approximating the orthogonal constraint and then run the standard SVD algorithms.

STRENGTHS:
- The paper defines a new notion for fair PCA.

WEAKNESSES:
- The paper does not discuss why this particular notion captures fairness.
- The amount of technical novelty in the paper is limited.

QUESTIONS:
NA

LIMITATIONS:
NA
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-10