Distributionally Robust Fair Principal Components via Geodesic Descents

1 INTRODUCTION . Machine learning models are ubiquitous in our daily lives and supporting the decision-making process in diverse domains . With their flourishing applications , there also surface numerous concerns regarding the fairness of the models ’ outputs ( Mehrabi et al. , 2021 ) . Indeed , these models are prone to biases due to various reasons ( Barocas et al. , 2018 ) . First , the collected training data is likely to include some demographic disparities due to the bias in the data acquisition process ( e.g. , conducting surveys on a specific region instead of uniformly distributed places ) , or the imbalance of observed events at a specific period of time . Second , because machine learning methods only care about data statistics and are objective driven , groups that are under-represented in the data can be neglected in exchange for a better objective value . Finally , even human feedback to the predictive models can also be biased , e.g. , click counts are human feedback to recommendation systems but they are highly correlated with the menu list suggested previously by a potentially biased system . Real-world examples of machine learning models that amplify biases and hence potentially cause unfairness are commonplace , ranging from recidivism prediction giving higher false positive rates for African-American1 to facial recognition systems having large error rate for women2 . To tackle the issue , various fairness criteria for supervised learning have been proposed in the machine learning literature , which encourage the ( conditional ) independence of the model ’ s predictions on a particular sensitive attribute ( Dwork et al. , 2012 ; Hardt et al. , 2016b ; Kusner et al. , 2017 ; Chouldechova , 2017 ; Verma & Rubin , 2018 ; Berk et al. , 2021 ) . Strategies to mitigate algorithmic bias are also investigated for all stages of the machine learning pipelines ( Berk et al. , 2021 ) . For the pre-processing steps , ( Kamiran & Calders , 2012 ) proposed reweighting or resampling techniques to achieve statistical parity between subgroups ; or in the training steps , fairness can be encouraged by adding constraints ( Donini et al. , 2018 ) or regularizing the original objective function ( Kamishima et al. , 2012 ; Zemel et al. , 2013 ) ; and in the post-processing steps , adjusting classification threshold based on examining black-box models over a holdout dataset can be used ( Hardt et al. , 2016b ; Wei et al. , 2019 ) . Since biases may already exist in the raw data , it is reasonable to demand the machine learning pipeline to combat biases as early as possible . We focus in this paper on the Principal Component 1 https : //www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing 2 https : //news.mit.edu/2018/study-finds-gender-skin-type-bias-artificial-intelligence-systems-0212 Analysis ( PCA ) , which is a fundamental dimensionality reduction technique in the early stage of the pipelines ( Pearson , 1901 ; Hotelling , 1933 ) . PCA finds a linear transformation that embeds the original data into a lower-dimensional subspace that maximizes the variance of the projected data . Thus , PCA is prone to amplify biases if the data variability is different between the majority and the minority subgroups , see a toy example in Figure 1 . A naive approach to promote fairness is to train one independent transformation for each subgroup . However , this requires knowing the sensitive attribute of each sample at test time , which would raise disparity concerns . On the contrary , using a single transformation for all subgroups is “ group-blinded ” and faces no discrimination problem ( Lipton et al. , 2018 ) . Learning a fair PCA has attracted attention from many fields from machine learning , statistics to signal processing . Samadi et al . ( 2018 ) and Zalcberg & Wiesel ( 2021 ) propose to find the principal components that minimize the maximum subgroup reconstruction error ; the min-max formulations can be relaxed and solved as semidefinite programs . Olfat & Aswani ( 2019 ) propose to learn a transformation that minimizes the possibility of predicting the sensitive attribute from the projected data . Apart from being a dimensionality reduction technique , PCA can also be thought of as a representation learning toolkit . Viewed in this way , we can also consider a more general family of fair representation learning methods that can be applied before any further analysis steps . There are a number of works develop towards this idea ( Kamiran & Calders , 2012 ; Zemel et al. , 2013 ; Calmon et al. , 2017 ; Feldman et al. , 2015 ; Beutel et al. , 2017 ; Madras et al. , 2018 ; Zhang et al. , 2018 ; Tantipongpipat et al. , 2019 ) , which apply a multitude of fairness criteria . In addition , we also focus on the robustness criteria for the linear transformation . Recently , it has been observed that machine learning models are susceptible to small perturbations of the data ( Goodfellow et al. , 2014 ; Madry et al. , 2017 ; Carlini & Wagner , 2017 ) . These observations have fuelled many defenses using adversarial training ( Akhtar & Mian , 2018 ; Chakraborty et al. , 2018 ) and distributionally robust optimization ( Rahimian & Mehrotra , 2019 ; Kuhn et al. , 2019 ) . Contributions . This paper blends the ideas from the field of fairness in artifical intelligence and distributionally robust optimization . Our contributions can be described as follows . • We propose the fair principal components which balance between the total reconstruction error and the absolute gap of reconstruction error between subgroups . Moreover , we also add a layer of robustness to the principal components by considering a min-max formulation that hedges against all perturbations of the empirical distribution in a moment-based ambiguity set . • We provide the reformulation of the distributionally robust fair PCA problem as a finitedimensional optimization problem over the Stiefel manifold . We provide a Riemannian gradient descent algorithm and show that it has a sub-linear convergence rate . Figure 1 illustrates the qualitative comparison between ( fair ) PCA methods and our proposed method on a 2-dimensional toy example . The majority group ( blue dots ) spreads on the horizontal axis , while the minority group ( yellow triangles ) spreads on the slanted vertical axis . The nominal PCA ( red ) captures the majority direction to minimize the total error , while the fair PCA of Samadi et al . ( 2018 ) returns the diagonal direction to minimize the maximum subgroup error . Our fair PCA can probe the full spectrum in between these two extremes by sweeping through our penalization parameters appropriately . If we do not penalize the error gap between subgroups , we recover the PCA method ; if we penalize heavily , we recover the fair PCA of Samadi et al . ( 2018 ) . Extensive numerical results on real datasets are provided in Section 5 . Proofs are relegated to the appendix . 2 FAIR PRINCIPAL COMPONENT ANALYSIS . 2.1 PRINCIPAL COMPONENT ANALYSIS . We first briefly revisit the classical PCA . Suppose that we are given a collection of N i.i.d . samples { x̂i } Ni=1 generated by some underlying distribution P. For simplicity , we assume that both the empirical and population mean are zero vectors . The goal of PCA is to find a k-dimensional linear subspace of Rd that explains as much variance contained in the data { x̂i } Ni=1 as possible , where k < d is a given integer . More precisely , we parametrize k-dimensional linear subspaces by orthonormal matrices , i.e. , matrices whose columns are orthogonal and have unit Euclidean norm . Given any such matrix V , the associated k-dimensional subspace is the one spanned by the columns of V . The projection matrix onto the subspace is V V > , and hence the variance of the projected data is given by tr ( V V > ΞΞ > ) , where Ξ = [ x̂1 , · · · , x̂N ] ∈ Rd×N is the data matrix . By a slight abuse of terminology , sometimes we refer to V as the projection matrix . The problem of PCA then reads max V ∈Rd×k , V > V=Ik tr ( V V > ΞΞ > ) . ( 1 ) For any vector X ∈ Rd and orthonormal matrix V , denote by ` ( V , X ) the reconstruction error , i.e. , ` ( V , X ) = ‖X − V V > X‖22 = X > ( Id − V V > ) X . The problem of PCA can alternatively be formulated as a stochastic optimization problem min V ∈Rd×k , V > V=Ik EP̂ [ ` ( V , X ) ] , ( 2 ) where P̂ is the empirical distribution associated with the samples { x̂i } Ni=1 and X ∼ P̂ . It is wellknown that PCA admits an analytical solution . In particular , the optimal solution to problem ( 2 ) ( and also problem ( 1 ) ) is given by any orthonormal matrix whose columns are the eigenvectors associated with the k largest eigenvalues of the sample covariance matrix ΞΞ > . 2.2 FAIR PRINCIPAL COMPONENT ANALYSIS . In the fair PCA setting , we are also given a discrete sensitive attributeA ∈ A , whereAmay represent features such as race , gender or education . We consider binary attribute A and let A = { 0 , 1 } . A straightforward idea to define fairness is to require the ( strict ) balance of a certain objective between the two groups . For example , this is the strategy in Hardt et al . ( 2016a ) for developing fair supervised learning algorithms . A natural objective to balance in the PCA context is the reconstruction error . It is therefore tempted to adopt the following definition . Definition 2.1 ( Fair projection ) . Let Q be an arbitrary distribution of ( X , A ) . A projection matrix V ∈ Rd×k is fair relative to Q if the conditional expected reconstruction error is equal between subgroups , i.e. , EQ [ ` ( V , X ) |A = a ] = EQ [ ` ( V , X ) |A = a′ ] ∀ ( a , a′ ) ∈ A×A . Unfortunately , Definition 2.1 is too stringent : for a general probability distribution Q , it is possible that there exists no fair projection matrix V . Proposition 2.2 ( Impossibility result ) . For any distribution Q on X × A , let S = EQ [ XX > |A = 0 ] − EQ [ XX > |A = 1 ] . Then , there exists a fair projection matrix V ∈ Rd×k relative to Q if and only if rank ( S ) ≤ k. One way to circumvent the impossibility result is to relax the requirement of strict balance to approximate balance . In other words , an inequality constraint of the following form is imposed : |EQ [ ` ( V , X ) |A = a ] − EQ [ ` ( V , X ) |A = a′ ] | ≤ ∀ ( a , a′ ) ∈ A×A , where > 0 is some prescribed fairness threshold . This approach has been adopted in other fair machine learning settings , see Donini et al . ( 2018 ) and Agarwal et al . ( 2019 ) for example . In this paper , instead of imposing the fairness requirement as a constraint , we penalize the unfairness in the objective function . Specifically , for any projection matrix V , we define the unfairness as the absolute difference between the conditional loss between two subgroups : U ( V , Q ) , |EQ [ ` ( V , X ) |A = 0 ] − EQ [ ` ( V , X ) |A = 1 ] | . We thus consider the following fairness-aware PCA problem min V ∈Rd×k , V > V=Ik EP̂ [ ` ( V , X ) ] + λU ( V , P̂ ) , ( 3 ) where λ ≥ 0 is a penalty parameter to encourage fairness . Note that for fair PCA , the dataset is { ( x̂i , âi ) } Ni=1 and hence the empirical distribution P̂ is given by P̂ = 1N ∑N i=1 δ ( x̂i , âi ) .

Summary: This paper aims to improve fairness and distributional robustness in dimensionality reduction techniques. More specifically, it proposes a regularization term to principal compoenent analysis (PCA) such that the expected reconstruction errors for each groups (conditioned on a binary sensitive variable) are similar, and relaxes constraint on the empirical probability distribution by maximizing over a Wasserstein-type ambiguity set. This is optimized with subgradient descent on the Stiefel manifold, and numerical experiments comparing the proposed method to previous fair PCA are conducted.