Distributionally Robust Fair Principal Components via Geodesic Descents
Authors: Hieu Vu
Toan Tran
Man-Chung Yue
Viet Anh Nguyen
Venue: ICLR 2022 Poster (2022)
Keywords: fair principal component analysis
distributionally robust optimization
manifold optimization

Abstract: Principal component analysis is a simple yet useful dimensionality reduction technique in modern machine learning pipelines. In consequential domains such as college admission, healthcare and credit approval, it is imperative to take into account emerging criteria such as the fairness and the robustness of the learned projection. In this paper, we propose a distributionally robust optimization problem for principal component analysis which internalizes a fairness criterion in the objective function. The learned projection thus balances the trade-off between the total reconstruction error and the reconstruction error gap between subgroups, taken in the min-max sense over all distributions in a moment-based ambiguity set. The resulting optimization problem over the Stiefel manifold can be efficiently solved by a Riemannian subgradient descent algorithm with a sub-linear convergence rate. Our experimental results on real-world datasets show the merits of our proposed method over state-of-the-art baselines. 

Published as a conference paper at ICLR 2022
DISTRIBUTIONALLY ROBUST FAIR PRINCIPAL COMPONENTS VIA GEODESIC DESCENTS
Hieu Vu, Toan Tran
Man-Chung Yue Hong Kong Polytechnic University
Viet Anh Nguyen VinAI Research, Vietnam
VinAI Research, Vietnam
ABSTRACT
Principal component analysis is simple yet useful dimensionality reduction tech- nique in modern machine learning pipelines. In consequential domains such as college admission, healthcare and credit approval, it is imperative to take into account emerging criteria such as the fairness and the robustness of the learned projection. In this paper, we propose distributionally robust optimization prob- lem for principal component analysis which internalizes a fairness criterion in the objective function. The learned projection thus balances the trade-off between the total reconstruction error and the reconstruction error gap between subgroups. taken in the min-max sense over all distributions in a moment-basee ambiguity set. The resulting optimization problem over the Stiefel manifold can be effi- ciently solved by a Riemannian subgradient descent algorithm with a sub-linear convergence rate. Our experimental results on real-world datasets show the merits of our proposed method over state-of-the-arr baselines.
arXiv: MEXiv:2202.03071V [cs.LG] 7 Feb 2022
INTRODUCTION
Machine learning models are ubiquitous in our daily lives and supporting the decision- pro- cess in diverse domains. With their flourishing applications, there also surface numerous concerns regarding the fairness of the models' outputs (Mehrabi et al.] 2021]. Indeed, these models are prone to biases due to various reasons (Barocas et al. 2018). First, the collected training data is likely to include some demographic disparities due to the bias in the data acquisition process (c.g., con- ducting surveys on a specific region instead uniformly distributed places), or the imbalance of observed events at a specific period of time. Second, because machine learning methods only care about data statistics and are objective driven, groups that are under-represented in the data can be neglected in exchange for better objective value. Finally, even human feedback the predictive models can also be biased, e.g., click counts are human feedhack to recommendation systems but they are highly correlated with the menu list suggested previously by a potentially biased system. Real-world examples of machine learning models that amplify biases and hence potentially cause unfairness are commonplace, ranging from recidivism prediction giving higher false positive rates for Africcn-Americat to facial recognition systems having large error rate for womer
To tackle the issue, various fairness criteria for supervised learning have been proposed in the lit- erature, which encourage the (conditional) independence of the model's 's predictions on particular sensitive attribute (Dwork et | 2012 Hardt et 2016b) Kusner et al. 2017 [Couldechova] 2017] Verma & Rubin] 2018) Berk et al. 2021). Strategies to mitigate algorithmic bias are also investi- gated for all stages of the machine learning pipelines (Berk et al.) 2021). For the pre-processing steps, (Kamiran & Calders 2012) proposed reweighting or resampling techniques to achieve sta- tistical parity between subgroups; in the training steps, fairness can be encouraged by adding con- straints (Donini al.) 2018) or regularizing the original objective function (Kamishima et al. 2012] Zemel et al. 2013); and in the post-processing steps, adjusting classification threshold by examining black-box models over holdout dataset can be used (Hardt al. 2016b Wei et al. 2019).
Since biases may already exist in the raw data, it is reasonable to demand machine learning pipelines to combat biases as early as possible. We focus in this paper on the Principal Component Analy- sis (PCA), which is a fundamental dimensionality reduction technique in the early stage of the
2 http://wwwwpuuuuiiaaattttetmeets

---

Published as conference paper at ICLR 2022
pipelines (Pearson 1901 Hotelling 1933). PCA finds linear transformation that embeds the origi- nal data into a dimee-dimeniional subspace that maximizes the variance of the projected data. Thus, PCA may amplify biases if the data variabilityy is different between the majority and the minority subgroups, see an example Figure [ A naive approach to promote fairness is to train one inde- pendent transformation for each subgroup. However, this requires knowing the sensitive attribute of each sample, which would raise disparity concerns. On the contrary, using single transformation for all subgroups is "group-blinded" and faces no discrimination problem (Lipton al. 2018
Learning a fair PCA has attracted attention from many fields from machine learning, statistics to signal processing. Samadi al. (2018) and Zalcberg & Wiesel (202I) propose to find the principal components that minimize the maximum subgroup reconstruction error; the min-max formulations can relaxed and solved as semidefinite programs. Olfftt & Aswani (2019) propose to learn transformation that minimizes the possibility of predicting the sensitive attribute from the projected data. Apart from being a dimensionality reduction technique, PCA can also be thought of as representation learning toolkit. Viewed in this way, can also consider more general family of fair representation learning methods that can be applied before any further analysis steps. There number of works develop towards this idea (Kamiran & Calders 2012) Zemel et al. 2013
Calmon et al.] 2017; Feldman et al.) 2013; Beutel et 2017 Madras et al. 2018 Zhang et al, 2018 Tattipongppppt et al. 2019), which apply multitude of fairness criteria.
In addition, we also focus on the robustness criteria for the linear transformation. Recently, it has been observed that machine learning models susceptible to small perturbations of the data (Good- fellow et al., (2014] Madry et al. [2017] Carlini & Wagner 2017).
These observations have fuelled
many defenses using adversarial training (Akhtar & Mian
2018
Chakraborty et al. (0018) and dis-
tributionally robust optimization (Rahimian & Mehrotra 2019
Kuhn et al.) 2019 Blanchet et al.
2021.
Contributions. This paper blends the ideas from the field of fairness in artifical intelligence and distributionally robust optimization. Our contributions can be described as follows.
We propose the fair principal components which balance between the total reconstruction error and the absolute gap of reconstruction error between subgroups. Moreover, we also add a layer of robustness to the principal components by considering min-max formulation that hedges against all perturbations of the empirical distribution moment-based ambiguity set.
We provide the reformulation of the distributionally robust fair PCA problem as finite- dimensional optimization problem over the Stiefel manifold. We provide a Riemannian gradient descent algorithm and show that has a sub-linear convergence rate.
Figure illustrates the qualitative comparison between (fair) PCA methods and our proposed method on a 2-dimensional toy example. The majority group (blue dots) spreads the hor- izontal axis, while the minority group (yellow triangles) spreads the slanted vertical axis. The nominal PCA (red) captures the majority direction to minimize the total error, while the fair PCA of [Samadi et al. (2018) returns the diagonal direction to minimize the maximum subgroup error. Our fair PCA can probe the full spectrum between these two extremes by sweeping through our penalization parameters appropriately. If we do not penalize the error gap between subgroups, method; if we penalize heavily, we recover the fair PCA Samadi et al. (2018). Extensive nu- merical results on real datasets are provided Proofs relegated to the appendix.
-4-
Group0
Group1
Samadi et al. (2018)
recover the PCA
Figure 1: Nominal PCA (red arrow), fair PCA by Samadi et al. (2018) (green arrow), and our spec- trum of fair PCA (shorter arrows). Arrows show directions and are not normalized unit length.
Section 5
2

---

Published as a conference paper at ICLR 2022
2
FAIR PRINCIPAL COMPONENT ANALYSIS
2.1 PRINCIPAL. COMPONENT ANALYSIS
We first briefly revisit the classical PCA. Suppose that we given collection of N ii.d. samples { generated by some underlying distribution P.. For simplicity, we assume that both the empirical and population mean zero vectors. The goal of PCA is to find a k-dimensional linear subspace of I2 that explains as much variance contained in the data { f==1 as possible, where k < d is given integer. More precisely, we parametrize k-dimensional linear subspaces by orthonormal matrices, i.e., matrices whose columns are orthogonal and have unit Euclidean norm. Given any such matrix V, the associated k-dimensional subspace is the one spanned by the columns of V. The projection matrix onto the subspace is VV T and hence the variance of the projected data is given by VVVT==T). where [i,,... .#N] E RdxN is the data matrix. By a slight abuse of terminology, sometimes we refer to V as the projection matrix. The problem of PCA then reads
tr (VVT=ET)
max VeReXR.VVVaS
For any vector E Rd and orthonormal matrix V, denote by {(V, X) the reconstruction error, ie., XV, = ||X VV x13 = x - -VV ).X
The problem of PCA can alternatively be formulated stochastic optimization problem
min VeRi Ep]((V,
where I is the empirical distribution associated with the samples {á; | and X ~ P. It is well- known that PCA admits an analytical solution. In particular, the optimal solution to problem [2] (and also problem(1) is given by any orthonormal matrix whose columns are the eigenvectors associated with the & largest eigenvalues of the sample covariance matrix =
2.2 FAIR PRINCIPAL COMPONENT ANALYSIS
In the fair PCA setting, we are also given discrete sensitive attribute A E A, where A may represent features such race, gender education. We consider binary attribute and let A (0, A straightforward idea to define fairness is to require the (strict) balance of a certain objective between the two groups. For example, this is the strategy in Hardt et al. (2016a) for developing fair supervised learning algorithms. A natural objective to balance in the PCA context is the reconstruction error. Definition 2.1 (Fair projection). Let Q be an arbitrary distribution of (X,A), A projection matrix V E Badk is fair relative to Q if the conditional expected reconstruction error is equal between subgroups, Le., [V[ X) a) Fq(e(V, X) a') for any aa,a")= E A A.
Unfortunately, Definition 2.1 is too stringent: for general probability distribution Q, it is possible that there exists no fair projection matrix V.
Proposition 2.2 (Impossibility result). For any distribution Q on X' x A, there exists a fair projection
matrix V E kd relative to Q if and only if raan((EgXX.. IA = 0] Eg [XX I = 1J) s k.
One way to circumvent the impossibility result is to relax the requirement of strict balance to ap- proximate balance. In other words, an inequality constraint of the following form is imposed:
[V X) a) - FV X)|4= [1]4=aa a']l Se V(a, a') E A x A.
where e some prescribed fairness threshold. This approach has been adopted in other fair machine learning settings, see Donini et al. (2018) and Agaawal et al. (2019) for example.
In this paper, instead of imposing the fairness requirement as a constraint, we penalize the unfairness in the objective function. Specifically, for any projection matrix V, we define the unfaimeess as the absolute difference between the conditional loss between two subgroups:
U(V,Q) & [FV[e(V, X)J4 0] (V( x) A 1]|
We thus consider the following fairness PCA problem
min VERaX VTv==
Ep( + AU(V,
(3)
where À > 0 is penalty parameter encourage fairness. Note that for fair PCA, the dataset is ((i., N and hence the empirical distribution Po is given by To N Σm 6(e,)))-
3

---

Published as a conference paper at ICLR 2022
3 DISTRIBUTIONALLY ROBUST FAIR PCA
The weakness of empirical Distribution-based stochastic optimization has been well-documented, see (Smith Winkler 2006 Homem-de Mello & Bayraksan, (014). particular, due to overfit- ting. the out-of-sample performance of the decision, prediction, or estimation obtained from such stochastic optimization model unsatisfactory, especially the low sample size regime. Ideally, we could improve the performance by using the underlying distribution P instead of the empirical distribution P.. But the underlying distribution I' is unavailable in most practical situations, if not all. Distributional robustification is an emerging approach to handle this issue and has been shown to deliver promising out-of-samp oampea performance in many applications (Delage & Ye! 2010 Namkoong & Duchi 2017 Kuhn et al., 2019 Rahimian & Mehrotra 2019). Motivated by the success of distri- butional robustification, especially in machine learning Nguyen al. (2019); Taskesen et al. (2021
we propose robustified version of model [ called the distributionally robust fairness-aware PCA:
min VERAXE VT V=lk QEB(P)
sup
Eg[[(V, + AU(V,
(4)
where (IP) is a set of probability distributions similar to the empirical distribution in a certain sense, called the ambiguity set. The empirical distribution IP is also called the nominal distribu- tion. Many different ambiguity sets have been developed and studied in the optimization literature, see Rahimian & Mchrotra (2019) for extensive overview.
3.1 THE WASSERSTEIN TYPE AMBIGUITY SET
To present our ambiguity set and main results, need introduce some definitions and notations. Definition 3.1 (Wasserstein-tyype divergence). The divergence W between two probability distribu- tions Qi (f1. En) E TRd $4 and Q2 (12, Σ2) E I2d x S defined as
W(Q1
|1 pell + tr Σ1 + Eg 2(1)
The divergence W coincides with the squared type-2 Wasserstein distance between two Gaus- sian distributions A (11, Σ1) and ((2,, Σ2) (Givens & Shorti 1984.. One can readily show that W is non-negative, and it vanishes if and only if (#1, En) and Q2 have the same first- and second-moments. Recently, distributional robustification with Wasserstein-type ambiguity sets has been applied widely to various problems including domain adaption (Taskesen et al., 2021). risk measurement (Nguyen et al. 2021b) and statistical estima- tion (Nguyen al.) (2221a). The Wasserstein type divergence in Definition 3. is also related the theory of optimal transport with its applications in robust decision making (Mohajerin Esfahani & Kuhn 2018 Blanchet & Murthy 20199] Yue al. (2021) and potential applications in fair machine
(122 E2), which implies that Qi
learning (" (Tassesen et al. 2020 Si et al. 2021 Wang et al. 2021).
Recall that the nominal distribution is P bution given A = a is given by
DM 6(2,,a.). For any E A, its conditional distri-
P
E2.1 where
Ig a (1:
Ia
TET
We also use (Pa, Eg) to denote the empirical mean vector and covariance matrix of X given A = a:
Pa = Eg_ [X] Ep|-=Ep/------ a) and E. + Pali E2 [XX Fx[XX A - a).
For any a E A, the empirical marginal distribution of A is denoted by p.. IZa//N.
Finally, for any set S, we use P(S) to denote the set of all probability distributions supported on S. For any integer k. the k-bb-- identity matrix is denoted 4.- We then define our ambiguity set as
JQa E P(X') such that: Q( x {a)) = (X) VX Rd. aEA )
B(P)= QEP(X
A):
(5)
W(Q., Pa) & Ea Va
4

---

Published as a conference paper at ICLR 2022
where Qa is the conditional distribution of X A Intuitively, each Q B(P) is joint distribu- tion of the random vector (X; formed by taking a mixture of conditional distributions Qa with mixture weight Pa- Each conditional distribution Q. is constrained in an E, meiiihhboroood of the nominal conditional distribution IP, with respect the W divergence. Because the loss function & a quadratic function of X, the (conditional) expected losses only involve the first two moments of X, and thus prescribing the ambiguity set using W would suffice for the purpose of robustification.
3.2 REFORMULATION
We now present the reformulation of problem (4) under the ambiguity set (F)..
Theorem 3.2 (Reformulation). Suppose that for any a E A, either of the following two conditions holds:
(i) Marginal probability bounds: 0 S X Pa.
iii Eigenvalue bounds: the empirical second moment matrix M, = No ΣiEz, &G satisfies E (M(a)) 2 Ca, where ay(Ma, is the j-th smallest eigenvalues of Ma.
Then problem
[4] is equivalent to
min VERax VVT==
max (Jo(V), (VV)),
(6a)
where for each (a, a') E 1), the function Ja is defined as Ja(V) Ka 0a v (La VVT M.) + tn / (La VVT .M.)
(Ia- VV VVT.CaV .Ca).
(6b)
and the parameters E R, 1 E E E R and C E 84 are defined as
Ka (pa + A)Ea + (far A)Ea'
0a = 21pa + AlVea, Da = 2/par AlvEa'
(6c)
Ca (fa + X) + (for - A)Mar
We now briefly explain the steps that lead the results Theorem 3.2 Letting
Jo(V) sup (0o + ()) 0| + (6) A) X)IA = 1],
QE
(1( = sup (po \V, 0J + (6) + (VV 1],
QEB(P)
then by expanding the term U(V, ! using its definition, problem
(4)
becomes
min max JI VEBAXI.VVV-- V-1.
Leveraging the definition the ambiguity set B(P). for any pair a') e ((0,1), (1,0)), we can decompose Ja into two separate supremum problems as follows
Ja(V)= sup
(pa + A)E [)( +
sup
(por - X) (((V,)).
Q. F.
Q
The next proposition asserts that each individual supremum in the above expression admits lytical expression.
Proposition 3.3 (Reformulation). Fix a E A. For any v E I, Ea E &R,, it holds that
sup
E... [[[V,)]]
Q. Q.W.Q......se..
(Ia (la- VVT, ,M) VVT, + VEa
if 2 0,
V (Ia VVT,
MM,) VVT,No)
Vea
if 0 and (Ia VVT ,Ma) 2 Ear
if and (Ia VVT (V,Ma < Ea-
5

---

Published as a conference paper at ICLR 2022
The proof of Theorem 3.2 now follows by applying Proposition 3.3 each term in Ja and balance
the parameters to obtain (
(c) A detailed proof is relegated to the appendix. In the next section, we
study an efficient algorithm to solve problem [[a]]
Remark 3,4 (Recovery of the nominal PCA). If = 0 and = 0 Va E A. our formulation M In this case, our robust fair principal components reduce to the standard principal components. On the contrary, existing fair PCA methods such as (Samadi
becomes the standard PCA problem !2
ct al. 2018) and Olfat & Aswani (2019) cannot recover the standard principal components.
RIEMANNIAN GRADIENT DESCENT ALGORITHM
The distributionally robust fairness-aware PCA problem is originally an dimesie-dimensional min-max problem. Indeed, the inner maximization problem in (4) optimizes over the space of prob- ability measures. Thanks to Theorem 3.2 is reduced to the simpler finiteddimensional minimax problem [6ai] where the inner problem is only maximization over two points. Problem (6a) is, however, still challenging as is a non-convex optimization problem over a non-convex feasible re- gion defined by the orthogonality constraint VTV Ia. The purpose of this section is to devise an efficient algorithm for solving problem (6a) to local optimality based on Riemannian optimization.
4.1 REPARAMETRIZATTIO
As mentioned above, the non-convexity of problem [6a] comes from both the objective function and the feasible region. It turns out that we can get rid of the non-convexity of the objective function via simple change of variables, To see that, we let U E [dx((dk) be an orthonormal matrix complement V. that is, U and V satisfy UU + VV La. Thus, can express the objective function J via
J(V) F(U) A max(Fo ((U)), where for (a, a') E 1),,1), (1, the function F. is defined
Fa(U) 2 Ka + 0.y (UUT, ,Ma) + day (UUT, Ma)) + (UUT, ,Ca). Moreover, letting M é {U E (dx(d-)) U Ia-k). we can re-express problem 66 as
min F(U). UEM
The set M of problem (7) is Riemannian manifold, called the Stiefel manifold (bssil al. 2007
Section 3.3.2). It is then natural to solve [7] using Riemannian optimization algorithms (Absil 2007). In fact, problem [6a] itself (before the change of variables) can also be cast as Riemannian optimization problem over another Stiefel manifold. 'he change of variables above might seem unnecessary. Nonetheless, the upshot of problem (7] is that the objective function F convex (in the traditional sense). This faciliates the application of the theoretical and algorithmic framework developed in Li al. (202I) for (weakly) convex optimization over the Stiefel manifolds.
4.2 THE RIEMANNIAN SUBGRADIENT
Note that the objective function F is non-smooth since it is defined as the maximum of two func- tions Fo and Fi. To apply the framework in Li et al. (2021), we need to compute the Riemannian subgradient of the objective function F. Since the Stiefel manifold M an embedded manifold in Euclidean space, the Riemannian subgradient of F at any point U E M is given by the orthogonal projection of the usual Euclidean subgradient onto the tangent space of the manifold M at the point U. see Absil et al. (2007) Section 3.6.1) for example.
Lemma 4.1. For any point U E M, let ay E arg (0,1 UU) and d' = 1 - av. Then, a the point U is given by
Riemannian subgradient of the functiono
function F
Oay
Daly
grad = -Ia-UUT U
May U +
Mai U 2C U
VUUTT Mav)
(UUU, Maa)
It is possible that the maximizer is unique. In that case, chonsing oz he either O would work.

---

Published as a conference paper at ICLR 2022
4.3 RETRACTIONS
Another important instrument required by the framework in Li et al. (2021) is retraction of the Stiefel manifold M. At each iteration, the point U > obtained by moving from the current iterate U the opposite direction of the Riemannian gradient A may not lie on the manifold in general, where Y concept of retraction. Given a point U E M on the manifold, the Riemanmian gradient A E T&M (which must lie in the tangent space M) and a stepsize ?, the retraction map Rtr defines point
the stepsize.
Riemannian optimization, this is cireumvented by the
which is guaranteed to lie on the manifold M. Roughly speaking, the retraction Rire(.) approximates the geodesic curve through U along the input tangential direction. For a formal definition of retractions, we refer the readers to (Absil et al., 2007 Section 4.1). In this paper, we focus on the following two commonly used retractions for Stiefel manifolds. The first one the QR decomposition booooossition-bbeeed retraction using the Q-factor qf(: in the QR decomposition:
Ru ()) qf(U + UE M,, ToM.
The second one the polar decomposition--aasee retraction
Rttl polar (A) (U + +) AT)
UEMAATTU.
(8)
4.4 ALGORITHM AND CONVERGENCE GUARANTEES
Associated with any choice of retraction Rtr is a concrete instantiation of the Riemannian subgradi- ent descent algorithm for problem 77] which is presented in Algorithm I wwih specific choice of
the stepsizes 7/2 motivated by the theoretical results of (Li et al. 2021).
Algorithm Riemannian Subgradient Descent for (7)
1: Input: An initial point Un. a number of iterations 7 and a retraction Rtr (U,A) (A).
2: for = 0, 1, ......,7 1, do
3: Find a A arg 40.1 (U.)).
4: Compute the Riemannian subgradient A. gradF(U,) using the formula
On,
0a Ma U2 + 2C
A = -1
Ma Up
VULUT Ma)
5: Set Ut+1 =
where the step-size 7t V++I is constant.
6: end for
7: Output: U,.
We now study the convergence guarantee of Algorithm T objective function F is Lipschitz continuous (with respect to the Riemannian metric on the Stiefel manifold M) with explicit Lipschitz constant L.
The following lemma shows that the
Lemma 4.2 (Lipschitz continuity). The function F is L-Lipschitz continuous on M, where given by
o max (Mo)
0, mmax (Mdx)
90 o mmax (Max) V Omin (Mz)
0 muax (M))
L A max
00
0,
mmin (M))
V mmin (Mo)
V Omin (M))
(9)
2vd ko (Co), 2 ko max (Gi)
We now proceed to show that Algorithm D enjoys a sub-linear convergence rate. To state the result, define the Moreau envelope
(U) A min UEM
F(U") 1U" -IU
24

---

Published as a conference paper at ICLR 2022
where || || denotes the Frobenius norm of a matrix. Also, measure the progress of the algorithm, we need to introduce the proximal mapping on the Stiefel manifold (Li al. 2021)
prox, (U) E arg min D'EM
F(U')+
110 UI
2p
From Li ct al. 2021 Equation (22)), we have that
[gradF(U))ll> prrxx |prrx
(U).
Therefore, the number gap, (U) is good candidate quantify the progress of optimization algo- rithms for solving problem (7)
Theorem 4.3 (Convergence guarantee). Let (U.)t=1..... be the sequence of iterates generated by
Algorithm Suppose that μ 1/4L, where L is the Lipschitz constant of / in (9 Then, we have
min gap, (U:) FuF((U)) ming Fa(U) + 2L 1)
(T + 1)1/4
NUMERICAL EXPERIMENTS
We compare our proposed method, denoted RFPCA, against two of methods for fair PCA: 1) Fai
FairPCA rPCA Samadi et al. [ and 2) CFPCA Olfat & Aswani (2019 with both cases: only mean constraint, and both mean and covariance constraints. We consider wide variety of datasets with ranging sample sizes and number of features. Further details about the datatasets can be found in Appendix C The code for all experiments is available in supplementary materials. We include here some details about the pppre-errrmetes that we search in the cross-valle coos-validation steps.
RFPCA. We notice that the neighborhood size should be inversely proportional to the size of subgroup a. Indeed, a subgroup with large sample size likely to have more reliable estimate of the moment information. Then we parameterize the neighborhood size by a common scalar a, and we have Ea = a/ a/v /Na where No is the number of samples in group a. We search E {0.05, 0.1,0.15) and À {0., 0.5, 1., 1.5, 2.0, 2.5). For better convergence quality, we set the number of iteration for our subgradient descent algorithm to T = 1000 and also repeat the Riemannian descent for 20 randomly generated initial point Up-
Fai rPCA. According to Samadi et al. (2018), we only need tens of iterations for the multiplica- tive weight algorithm to provide good-quality solution; however, to ensure fair comparison, we set the number of iterations to 1000 for the convergence guarantee. We search the learning rate 77 of the algorithm from set of 17 values evenly spaced in [0.25, 4.25) and {0.1).
CFPCA. Following Olfat & Aswani 2019), for the mean-constrained version of CFPCA, we search 8 from {0., 0.1, 0.3, 0.5, 0.6, 0.7, 0.9), and for both the mean and covariance con- strained version, we fix
while searching y in (0.0001, 0.001, 0.01, 0.05, 0.5].
Trade-offs. First, we examine the trade-off between the total reconstruction error and the gap be- tween the subgroup error. In this experiment, we only compare model with FairPCA and CFPCA mean-constraint version. We plot pareto curve for each of them over the two criteria with different hyper-parameters (hyper-parameters test range mentioned above). The whole datasets are used for training and evaluation. The results averaged over runs are shown in Figure 2
In testing methods with different principal components, we first split each dataset into training set and test set with equal size (50% each), the projection matrix of each method is learned from training set and tested over both sets. In this case, we only compare our method with traditional PCA and FairPCA method. We fix one set paremeparameters for each method. For PairPCA we set 77 = 0.1 and for RFPCA we set a 0.15, = 0.5, others hyper-parameters are kept as discussed before. The results are averaged over 5 different splits. Figure 3 shows the consistence of our method performing fair projections over different values of k. Our method (cross) exhibits smaller gap of subgroup errors. More results and discussions on the effect of e can be found in Appendix D.2
http://gttthhi.oommmamimmmmm/m/a http:///gthuu.ccom//oofffffffffe
8

---

Published as a conference paper at ICLR 2022
RFPCA FairPCA
PCA Groupo
Group1
1.4
CFPCA
FairPCA Groupo
FairPCA Group1
Groupo
Average Subgraup Reconstratiion
RFPCA Group1
1.2
20
Absolute differnce reconstratiiiooo
1.0
20-
20:
Figure 2: Pareto curves on Default Credit dataset (all data) with 3 principal components
Figure 3: Subgroup average error with different
on Biodeg dataset (Out-of-sample,
Cross-validationn Next, we report the performance of all methods based on three criteria: absolute difference between average reconstruction еπоо of all data (ARE.), and the fairness criterion defined by Olfat & Aswani (2019) with respect a linear SVM's classifier family (AFFin) Due the space constraint, we only include the first two criteria in the main text, see Appendix 4 for full results. To emphasize the generalization capacity of each algorithm, we split each dataset into training set and a test set with ratio of 30% 70% respectively, and only extract top three principal components from the training set. We find the best hyper-parameters by 3-fold cross validation, and prioritize the one giving minimum value of the summation (ABDiff. ARE.). The results are averaged over 10 different training-testinng splits. We report the performance on both training set (In-sample data) and test set (Out-of tattoff-ampll data). The details results for Out-of-sampo sample data is given in Table I more details about settings and performance can be found at Appendix D
between groups (ABDift.), average reconstruction
Results. Our proposed RFPCA method outperforms 11 out of 15 datasets in terms of the subgroup error gap ABDiff, and 9 out of 15 with the totall error ARE. criterion. There are datasets that RFPCA gives the best results for both criteria, and for the remaining datasets, RFPCA has small performance gaps compared with the best method.
Table 1: Out-of-sample errors on real datasets. Bold indicates the lowest error for each dataset.
RFPCA
FairPCA
OPPCA-Mear Con.
CFPCA ABDift.
Both Con.
Dataset
ABDiff.
ARE.
ABDiff.
ARE.
ABDiff.
ARE.
ARE.
Default Credit
0.9483
10.3995
1.4401
10.4439
0.9367
10.9451
3.3359
22.0310
Biodeg
23.0066
33.8571
27.5159
34.6184
29.1728
37.6052
37.9533
50.7090
E. Coli
1.1500
1.7210
1.5280
2.4799
1.1005
2.9466
5.1275
5.6674
Energy
0.0125
0.2238
0.0138
0.2225
0.1229
2.7318
0.1001
7.9511
German Credit
2.0588
43.9032
1.3670
44.0064
1.7845
43.96488
1.4955
49.5014
Image
0.7522
6.0199
1.6129
10.2616
1.1499
14.3725
4.7013
19.3356
Letter
0.1712
7.4176
1,2489
7.4470
0.4427
8.7445
0.5743
15.1779
Magic
1.8314
3.9094
2.9405
3.3815
5.5790
4.2105
8.7810
9.0064
Parkinsons
0.3273
5.0597
0.8678
4.9044
3,3804
5.7260
18.3312
19.7001
SkillCraft
0.7669
8.2828
0.7771
8.2494
1.0283
9.9484
1.2849
15.9751
Statlog
0.0838
3.0998
0.3356
7.9734
0.4476
10.8263
13.8437
35.8268
Steel
1.1472
12.5944
1.2208
12.3096
4.8710
16.4015
3.8084
25.8953
Taiwan Credit
0.5523
10.9845
0.5710
10.9415
0.5744
13.0437
0.9535
21.8963
Wine Quality
0.6359
4.2801
0.3046
6.0936
1.5020
6.1118
3.0451
10.1001
LFW
0.4463
7.6229
0.5340
7.6361
fail to converge
The code estimate this quantity is provided the author's repository

---

Published as a conference paper at ICLR 2022
REFERENCES
Pierre-Antoine Absil, Robert Mahony, and Rodolphe Sepulchre. Optimization Algorithms on Matrix Manifolds. Princeton University Press, 2007.
Alekh Agarwal, Miroslav Dudik, and Zhiwei Steven Wu. Fair regression: Quantitative definitions and reduction-based algorithms. In International Conference on Machine Learning, pp. 120-129. PMLR, 2019.
Naveed Akhtar and Ajmal Mian. Threat of adversarial attacks on deep learning computer vision: A survey. leee Access, 6: 6:14410-14430, 2018.
Solon Barocas, Moritz Hardt, and Arvind Narayanan. Faimess and machine learning. fairmlbook. org, 2019, 2018.
Richard Berk, Hoda Heidari, Shahin Jabbari, Michael Kearns, and Aaron Roth. Fairness in criminal justice risk assessments: The state of the art. Sociological Methods & Research, 50(1):3-44, 2021.
Alex Beutel, Jilin Chen, Zhe Zhao, and Ed H Chi. Data decisions and theoretical implications when adversarially learning fair representations. arXiv preprint arXiv: 2017.
Jose Blanchet and Karthyek Murthy, Quantifying distributional model risk via optimal transport. Mathematics of Operations Research, 44(2):565-600, 2019.
Jose Blanchet, Karthyek Murthy, and Viet Anh Nguyen. Statistical analysis of Wasserstein distribu- tionally robust estimators. INFORMS TutORials in Operations Research, 2021.
Flavio P Calmon, Dennis Wei, Bhanukiran Vinzamuri, Karthikeyan Natesan Ramamurthy, and Kush R Varshney, Optimized pre-processing for discrimination prevention. In Proceedings of the 31st International Conference on Neural Information Processing Systems, pp. 1999540044 2017.
Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In 2017
IEEE Symposium on Security and Privacy (SP), pp. 39-57. IEEE, 2017.
Anirban Chakraborty, Manaar Alam, Vishal Dey, Anupam Chattopadhyay, and Debdeep Mukhopad- hyay. Adversarial attacks and defences: A survey, arXiv preprint w:Xh::1810000065 2018.
Alexandra Chouldechova. Fair prediction with disparate impact: A study of bias in recidivism
prediction instruments. Big Data, 5(2):153-163, 2017.
Erick Delage and Yinyu Ye. Distributionally robust optimization under moment uncertainty with application to data-driven problems. Operations Research, 58(3):595-611 2010.
Michele Donini, Luca Oneto, Shai Ben-David, John Shawe-Taylor, and Massimiliano Pontil. Em- pirical risk minimization under fairness constraints. In Advances in Neural Information Process- ing Systems, pp- 2791-2801, 2018.
Cynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Richard Zemel. Fairness through awareness. In Proceedings the ence, pp. 214-226, 2012.
the 3rd innovations in theoretical computer science confer-
Michael Feldman, Sorelle A Friedler, John Moeller, Carlos Scheidegger, and Suresh Venkatasubra- manian. Certifying and removing disparate impact. In Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 259-268, 2015.
C.R. Givens and R.M. Shortt. class Wasserstein metrics for probability distributions. The Michigan Mathematical Journal, 31(2):231 1984.
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint 141.6214 2014.
Moritz Hardt, Eric Price, Eric Price, and Nati Srebro. Equality of opportunity in supervised learning.
In Advances in Neural Information Processing Systems 29, pp. 3315-3323, 2016a.
10

---

Published as a conference paper at ICLR 2022
Moritz Hardt, Eric Price, and Nati Srebro. Equality of opportunity in supervised learning. Advances in neural information processing systems, 29:3315-2333 -3233, 2016b.
Tito Homem-de Mello and Güzin Bayraksan. Monte Carlo sampling- based methods for stochastic optimization. Surveys in Operations Research and Management Science, 19(1):56-85, 2014.
Harold Hotelling. Analysis of a complex of statistical variables into principal components. Journal of educational psychology, 24(6):417, 1933.
Faisal Kamiran and Toon Calders. Data preprocessing techniques for classification without discrim- ination. Knowledge and Information Systems, 33(1):1-33) 2012.
Toshihiro Kamishima, Shotaro Akaho, Hideki Asoh, and Jun Sakuma. Fairness-aware classifier with prejudice remover regularizer. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pp. 35-50. Springer, 2012.
Daniel Kuhn, Peyman Mohajerin Esfahani, Viet Anh Nguyen, and Soroosh Shafieczadeh-Abadee Wasserstein distributionally robust optimization: Theory and applications in machine learning. In Operations Research & Management Science in the Age of Analytics, pp. 130-166. INFORMS, 2019.
Matt J Kusner, Joshua Loftus, Chris Russell, and Ricardo Silva. Counterfactual fairness. arXiv preprint arXiv: 06856, 2017.
Xiao Li, Shixiang Chen, Zengde Deng, Qing Qu, Zhihui Zhu, and Anthony Man Cho So. Weakly convex optimization over Stiefel manifold using Riemannian subgradient-type methods. SIAM Journal on Optimization, 33(3):1605-1 1634, 2021.
Zachary Lipton, Julian McAuley, and Alexandra Chouldechova. Does mitigating ML's impact dis- parity require treatment disparity? In Advances Neural Information Processing Systems, pp- 8125-8135, 2018.
David Madras, Elliot Creager, Toniann Pitassi, and Richard Zemel. Learning adversarially fair and transferable representations. In International Conference on Machine Learning, pp. 3384-3393. PMLR, 2018.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706.0608. 2017.
Ninareh Mchrabi, Fred Morstatter, Nripsuta Saxena, Kristina Lerman, and Aram Galstyan. A survey on bias and fairness in machine learning. ACM Computing Surveys (CSUR), 54(6):1-35, 2021.
Peyman Mohajerin Esfahani and Daniel Kuhn. Data-driven distributionally robust optimization us- ing the Wasserstein metric: Performance guarantees and tractable reformulations. Mathematical Programming. (1(1122)111-116 2018.
Hongseok Namkoong and John C Duchi. Variance-based regularization with convex objectives. In Advances in Neural Information Processing Systems 30, pp. 2971-2980, 2017.
Viet Anh Nguyen. Adversarial Analytics. PhD thesis, Ecole Polytechnique Fédérale de Lausanne, 2019.
Viet Anh Nguyen. Soroosh Shafieezadeh Abadeh, Man Man-Chung Yue, Daniel Kuhn, and Wolfram Wiesemann. Calculating optimistic likelihoods using (geodesically) convex optimization. In Advances in Neural Information Processing Systems, pp. 13942-13953 2019.
Viet Anh Nguyen, Shafiezzadeh-Ahadeh, D. Kuhn, and P. Mohajerin Esfahani. Bridging Bayesian and minimax mean square error estimation via Wasserstein distributionally robust optimization. Mathematics of Operations Research, 2021a.
Viet Anh Nguyen, Soroosh Shaficezadeh-Abiaee Damir Filipović, and Daniel Kuhn. Mcan- covariance robust risk measurement. arXiv preprint arXiv:2112.0000 2021b.
11

---

Published as a conference paper at ICLR 2022
Matt Olfat and Anil Aswani. Convex formulations for fair principal component analysis. In Pro- ceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 663-670,20 2019.
Karl Pearson. Liii On lines and planes of closest fit to systems of points in space. The London, Edinburgh, and Dublin philosophical magazine and journal of science, 2(11):559 -72, 1901.
Hamed Rahimian and Sanjay Mehrotra. Distributionally robust optimization: A review. arXiv 2019.
preprint
Samira Samadi, Uthaipon Tantipongpipat, Jamie H Morgenstern Mohit Singh, and Santosh Vem- pala. The price of fair PCA: One extra dimension. In Advances in Neural Information Processing Systems, pp. 10976-10987, 2018.
Nian Si, Karthyck Murthy, Jose Blanchet, and Viet Anh Nguyen. Testing group fairness via optimal transport projections. In Proceedings of the 38th International Conference Machine Learning, 2021.
James E Smith and Robert L Winkler, The optimizer's ' curse: Skepticism and postdecision surprise in decision analysis. Management Science, 52(3):311-322, 2006.
Uthaipon Tantipongpipat, Samira Samadi, Mohit Singh, Jamie Morgenstern, and Santosh Vem- pala. Multi-criteria dimensionality reduction with applications to fairness. arXiv preprint arXiv: / 1281, 2019.
Bahar Taskesen, Viet Anh Nguyen, Daniel Kuhn, and Jose Blanchet. A distributionally robust approach fair classification. arXiv preprint urXiv:2007.09530 2020.
Bahar Taskesen, Man-Chung Yue, Jose Blanchet, Daniel Kuhn, and Viet Anh Nguyen. Sequential domain adaptation by synthesizing distributionally robust experts. In Proceedings of the 38th International Conference on Machine Learning, 2021.
Sahil Verma and Julia Rubin. Fairness definitions explained. In 2018 IEEE/ACM International Workshop on Software Fairness (fairware), pp. 1-7. IEEE, 2018
Yijie Wang, Viet Anh Nguyen, and Grani Hanasusanto. Wasserstein robust support vector machines with fairness constraints. arXiv preprint arXiv:2103.06822 2021.
Dennis Wei, Karthikeyan Natesan Ramamurthy, and Flavio du Pin Calmon. Optimized score trans- formation for fair classification, arXiv preprint
2019.
Man-Chung Yue, Daniel Kuhn, and Wolfram Wiesemann. On linear optimization over Wasserstein balls. Mathematical Programming, 2021.
Gad Zalcberg and Ami Wiesel. Fair principal component analysis and filter design. IEEE Transac- tions on Signal Processing, 69:4835 2021.
Rich Zemel, Yu Wu, Kevin Swerssy, Toni Pitassi, and Cynthia Dwork. Learning fair representations. In International Conference on Machine Learning, pp. 325-333. PMLR, 2013.
Brian Hu Zhang, Blake Lemoine, and Margaret Mitchell. Mitigating unwanted biases with adver- sarial learning, In Proceedings of the 2018 AAAI/ACM Conference on A1, Ethics, and Society, pp. 335-340, 2018
A PROOFS
A.1 PROOFS SECTION 2
T AA = 1]. We first prove the "only if" direction. Suppose that there exists a fair projection matrix V MM relative to Q. Let
Proof of Proposition 2.2 Let S = Ex[XX IA = 0J - -
U E Ma- & be complement matrix of V. Then, Definition 2.1 can be rewritten as
(UUT S) = 0,
12

---

Published as a conference paper at ICLR 2022
which implies that the null space of S has dimension at least d - k. By the rank- duality, we have rank(S) I k.
Next, we prove the "if" direction. Suppose that rank(S) < k. Then, the matrix S has at least d (repeated) zero eigenvalues. Let U' E Maa be an orthonormal matrix whose columns are any d eigenvectors corresponding to the zero eigenvalues of 5 and V MM be complement matrix of U. Then,
-V- .S) (UUT, S) =.
Therefore, V is a fair projection matrix relative to Q. This completes the proof.
A.2 PROOF OF SECTION
Proofs of Proposition 3.3
By exploiting the definition of the loss function l, we find
sup
URa. [VV
C..W(Q..Pa)s..
Ir (u([ VV VV + (alla)
sup
l/,a - -Mal3 + tr Iga + Eg -2 I 52 +1) S sa
s.t.
inf
7(Ea - tr (Sia) )) 7 tr ((7 v(1 - VVT)) +
s.t.
TT - u(I - VVT)
7/
= 0, 71 v(1 - VVT), 72 0,
7ida T
7 Al +T
where the last equality follows from Nguyen (2019 Lemma 3.22). By the Woodbury matrix inver- sion, have
- - - 7(0 (リ- (1 VVT).
Moreover, using the Schur complement, the semidefinite constraint is equivalent to
r1lial12 + 2 P2dd (7d -VV-VV Pa.
which implies that at optimality, we have
7 ター 11 (T -V /a.
u7
-=
At the same time, the constraint 71 > v(l is equivalent to 7 > v. Combining all previous
equations, we have
YU [Ia -T-VVT. .Ma) Yea + o -- 1 (0
sup &... [[(V,))] QQ. P.).....ss.. Scu
inf
The dual optimal solution y* given by
(L-VVVT,bt Ma)
v
1
if
ca
(I--VVT,M Me)
v
1
if 0 and (Ia - VVT, ,,ae) 2 Ear
Ea
0
if 0 and (Ia VVT, ,M)) < Ea-
VVT,Ma
Note that y* 2 max(0, v} in all the cases. Therefore, we have
sup Q...(Q.....ee.
EEE [(V,
2 if 2 O,
v VEa V (Is (lo--VT, MMa)
2
U /Ea (Ta- VTVVVV, VVT,Ma)
if < 0 and (Ta VVT, VVTMaa 2 Ear
if < 0 and (Is VVTMM. < Ea.
This completes the proof.
13

---

Published as a conference paper at ICLR 2022
We now ready to prove Theorem 3.2
Proof of Theorem 3.2 By expanding the absolute value, problem (4) is equivalent to min V (V)
where for each (a, a') E [(0, (1,0)), we can re-express Ju as
(V) sup @. C..W(......ee.
(fa + Q). [K(V, X)] +
sup (.
(Pe X)Ee, (e(V, X)]
a
Using Proposition 3.3 reformulate the two individual supremum problems, have
(V) = (fa- A)ea + 21 + Aly Ea(Ta - VVT, .Ma) + (fa + A) VV .Ma)
+ (far A)Ea 2/p NIVE (Ta VVT Ma) + (fe' >) VV Ma.).
By defining the necessary parameters k, 0, 8 and C' as in the statement of the theorem, we arrive at the postulated result.
A.3 PROOFS OF SECTION
Proof of Lemma 4.1 Let ag arg mag Pa(U) and d/ ag. Then, an Euclidean subgradient of F is given by
Bac
VF(U)- VUUT Mar)
Mao U + V(UUT, Maae)
Mat U 2Ca U E Rdx(d-))
The tangent space of the Stiefel manifold M at U is given by
ToM = (A Rd
AU+UTA= 0}:
whose orthogonal projection (Absil et 2007
Example 3.6.2) can be computed explicitly via
UUU D 2-- D-D U),,, DE De
ProjtuM(() (Ta -U--UU1 )
Therefore, Riemannian subgradient of F at any point U .M is given by
gU)) (rooit.M(F((U))
(TUOT, Mar)
Oug
dal
Mai U + 2C )
(Ia -Ia-UUT UUT)
M U
((UUT,M
In the last line, we have used the fact that, if D = SU for some symmetric matrix S, then
This completes the proof.
The proof of Lemma 4.2 relies on the following preliminary result.
Lemma A.1. Let M (dd- be a positive definite matrix. Then,
KUUT, - (U'TT, .M) < 22d (0mma(MMM (M)llU -U'IFF VU, e M,,
kamax
(10)
and
V(UUT .M) (UTTT M)
T (M)
U - UU VU,U" U E
(11)
V Vanin (M)
where maxx (M) and amin (M) denote the maximum and minimum eigenvalues of the matrix M.
14

---

Published as a conference paper at ICLR 2022
Proof of Lemma For inequality (10)
KUUT, ,M) (U'U", .M
KUUT .M) - (UUT + KUUT, M) (U'UT,,))
- (U
- |
2v wodd-kmmar max |U - UF
For inequality and that
we first note that the function I + ii / Lipsccit on [Tmin +00)
(UUT, ,M) 2 (d (ij
VU E.M.
Therefore.
V.OUT, (U'UTT,
KUUT, - (U'U'
$ 2 2/ (d min (M)
max (M)
UIU Un|p:
/ (M)
where the last inequality follows from
10p This completes the proof.
We now ready prove Lemma 4.2
Proof of Lemma 4.2
Let U
U E M be two arbitrary points. We have
IP(U) - F(U")
|Imax (((( max (FF(U"),FP(""))
s max (E(1,1)
|Fa(U) FF(U')
0u Tmax (Ma)
maxx (M -o) 2 ko mox (Ca) I0 U'
S max max (0,11
01-a
Tmin (Ma) V (omin(M (M11aa
where the last inequality follows from the definition of Fa and Lemma A.1 This completes the proof.
Proof of Theorem 4.3 The proof follows from the fact that F is convex on the Euclidean space
Rdx (d-k) RR, Lemma 4.2 and Li et al. (2021 Theorem 2) (and the remarks following it).
B EXTENSION TO NON-BINARY SENSITIVE ATTRIBUTES
The main paper focuses on case of binary sensitive attribute with A = {0, 1). In this appendix, extend approach to the when the sensitive attribute is non-binary. Concretely, sup- pose that the sensitive attribute A can take on any of the m possible values from to m. other words, the attribute space now becomes A m1.....m..
Definition B.1 (Generalized unfairness measure). The generalized unfairness measure is defined as maximum pairwise unfairness measure, that is,
max max (V, (aAA
[gle(V, a] Egl((V, a'l
Notice that if A {0, 1}, then U max U recovers the unfairness measure for binary sensitive
attribute defined in Section 2.2 We now consider the following generalized fairness-aware PCA problem
min
sup Eq(e(V.X)) + AU max (V, V).
(12)
VER4xE.1 V Vnle CCB(P)
Here we recall that the ambiguity set B(P) is defined in mulation of (2)
The next theorem provides the refor-
15

---

Published as a conference paper at ICLR 2022
Theorem B.2 (Reformulation of non-binary faimess- PCA). Suppose that for any A, either of the following two conditions holds:
(i) Marginal probability bounds: 0 S A S Pa.
iii) Eigenvalue bounds: the empirical second moment matrix M, = No Evez L satisfies
d- (AMa) 2 Su, where (Maa) is the j-th smallest eigenvalues of Ma.
Then problem 12
equivalent to
Σ
2c o EulTa VVT, MM) A(Ia VV VVT,M - (aa)) + Mes - Eai)
min VERUXI.V
max
V astaa
bea
where the parameter Ca.a', b admits values
Pa + X
if a,
Ca,a'
Pa. - Al
if d', otherwise.
Pb
Proof of Theorem B.2
For simplicity, we let E(V, Q. = Eg(e(V, = b]. Then, the objective
function problem 12
can be re-written as
sup
Bg(e(V, AU max (V, Q)
QE3(F))
sup QE3(F)
Pb E(V, Q,6) + X max (E(V,Q,a) E(V,Q,a)) azd
PEA
Σ baa.a' W.Qo.
sup
(V((VQ.,))
sup W(Gu.
(ia + A)F(V (V;
sup W(Q..P.))ee
(par -V ()
Σ Do (V(Ta-VT, ([Ta-VV,, VVT, Mb) + VEO + (pa + 2)
(2))(11) (Ia VVT Vea
adkn'
bya,al
2
+ (Pa- 2) (V Mar) + sga(fa - A) A)V
Σ po (Ia-VVT VVT ,M)) + +0b) + Σ
= max a'fia
BEA
BEA
+ (X M, - Mar) + t - Ear)
where the first equality follows from the definition of U max(( (V, and E(V, Q, 6). the second from the definition (5) of the ambiguity set B(P)), the third from Proposition 3.3 and the fourth from the
definition of Ca.a'.b b. Noting that the first sum in the above maximization is independent of a and a', the proof is completed.
Theorem 12 indicates that the sensitive attribute admits finite values, then the distributionally robust fairness- PCA problem using Umax unfaimesss measure can be reformulated as an optimization problem over the Stiefel manifold, where the objective function is a pointwise maxi- mization of finite number of individual functions. It is also easy can be reparametrized using U, and the Riemannian gradient descent algorithm in Section 4 can be adapted to solve for the optimal solution. The details the algorithm omitted.
that each individual function
16

---

Published as a conference paper at ICLR 2022
C INFORMATION ON DATASETS
We summarize here the number of observations, dimensions, and the sensitive attribute of the data sets. For further information about the data sets and pre-processing steps, please refer to Samadi et al. (218)) for Default Credit and Labeled Faces in the Wild (LFW) data sets, Olfat & Aswani 2019 for others. For each data set, we further remove columns with too small standard deviation Ie 5) as they do not significantly affect the results, and ones with too large standard deviation ( 1000) which we consider as unreliable features.
Table 2: Number of observations N, dimensions d, and sensitive attribute A of datasets used in this
paper. (y - yes, n no)
Default Credit
Biodeg
E. Coli 333 7 isCytoplasm
Energy
German Credit
30000
1055
768
1000
d
22
410
48
Education (high/low)
Ready Biodegradable
Orientation
A13 2 200DM
A
(y/n)
(y/n)
(yn)
(y/)
Image
Letter
Magic
Parkinsons
SkillCraft
N
660
20000
19020
5875
3337
d
18
16
10
20
17
class (path/grass)
Vowel (y/n)
classIsGamma (y/n)
Sex (male/female)
Age 20 (yn)
Statlog
Steel
Taiwan Credit
Wine Quality
I.FW
N
3071
1941
29623
6497
4000
36
24
22
11
576
RedSoil
A
(vsgrey/dampprey)
FaultOther (y/n)
Sex (male/female)
White (y/n)
Sex (male/female)
D ADDITIONAL RESULTS
D.1 DETAIL PERFORMANCES
Table the performances of four examined methods with two criteria ABDiff. and ARE. is clear that our method achieves the best results over all 14 datasets w.r.t. ABDiff,, and datasets on ARE., which is equal to the number of datasets Fa rPCA out-perform others.
compeeeet Table the main text, from which we can see that two versions of CFPCA out-perform others over all datasets w.r.t. L which is the criteria they optimize for.
Table 3: In-sample performance over two criteria
RFPCA
FairPCA
CFPCA- Meaa Con.
CFPCA
Both Con.
Dataset
ABDiff.
ARE.
ABDiff.
ARE.
ABDiff.
ARE.
ABDiff.
ARE.
Default Credit
0.9457
9.9072
1.5821
9.9049
0.9949
10.5164
3.2827
21.4523
Biodeg
9.4093
23.1555
14.2587
23.8227
15.5545
26.6540
24.8706
39.8737
E. Coli
0.5678
1.4804
0.9191
2.0840
0.9539
2.8360
4.5225
5.2155
Energy
0.0094
0.2295
0.0153
0.2273
0.2658
2.7893
0.2136
7.8768
German Credit
1.6265
40.1512
2.9824
40.3393
2.6109
40.1860
2.8741
47.1006
Image
0.1320
5.0924
0.7941
9.0437
0.6910
13.1491
3.0118
18.0000
Letter
0.1121
7.4088
1.2560
7.4375
0.4572
8.7764
0.5301
15.2234
Magic
1.7405
3.8766
2.8679
3.3500
5.5-105
4.1938
8. 8.7963
8.9695
Parkinsons
0.1238
5.0471
0.6702
4.8760
3.9470
5.9379
17.8122
19.9788
SkillCraft
0.4231
8,1569
0.5576
8.1096
0.7156
9.7755
0.9334
15.8245
Statlog
0.1972
3.0588
0.3315
7,9980
0.3857
10.9358
13.0725
35. 35.9214
Steel
0.6943
11.0396
1.8015
10.7653
2,8933
14.5680
1.9322
23,9906
Taiwan Credit
1.1516
10.5136
1.3362
10.4478
1.3158
12.5867
2.2720
21 21.4365
Wine Quality
0.1125
4.1491
0.1705
5.8999
1.1359
5.9117
2.5852
9.8959
LFW
0.4147
7.5137
0.5300
7.5127
fail converge
A Adjustment for the LFW dataset. To demonstrate the efficacy of our method on high- high-dimensional data sets, also do experiments subset of 2000 faces for each of male and female group (4000
17

---

Published as a conference paper at ICLR 2022
Table Out-of-sample performance measured using the Lin criterion.
RFPCA
FairPCA
CHPCA-Mean Con.
CHPCA Both Con.
Default Credit
0.1596
0.2236
0.0574
0.0413
Biodeg
0.4892
0.4759
0.2014
0.1371
E. Coli
0.8556
0.7444
0.4455
0.2532
Energy 0.0580
0.0554
0.0502
0.0736
German Credit
0.1997
0.1737
0.1408
0.1093
Image
0.9996
0.9498
0.1874
0.2013
Letter 0.0954
0.0942
0.0556
0.0455
Magic
0.2195
0.2531
0.1561
0.0882
Parkinson's
0.1459
0.1061
0.1805
0.0480
SkillCraft
0.1126
0.1141
0.0721
0.0742
Statlog
0.9804
0.6309
0.1359
0.0669
Steel
0.2288
0.2240
0.1418
0,0875
Taiwan Credit
0.0604
0.0535
0.0391
0.0370
Wine Quality 0.9699
0.4639
0.2192
0.0817
in total) from LFW dataset all images are rescaled to resolution 24 x 24 (dimensions d = 576). The experiment follows the same procedure in Section [ with reducing the number of iterations to 500 for both REPCA and FairPCA and 2-fold cross validation, the results are averaged over 10 train-test simulations. Due the high dimension of the input, the implementation of Olfat & Aswani (2019) fails to return any result.
18

---

Published as a conference paper at ICLR 2022
D.2 VISUALIZATION
D.2.1 EFFECTS OF THE AMBIGUITY SET RADIUS
We examine the change of the model's performance with respect to the change of the radius of the ambiguity sets. To generate the toy data (also used for Figure we use two 2-dimensional Gaussian distributions to represent two groups of sensitive attribute, A 0 and A 1. or groups and simplicity. The two distributions both have the mean (0, 0) and covariance matrices for group 0 and 1 are
4.0 (10
70.2 0.4 0.4
and
0.2
3.0)
respectively. For the test set, the number of samples is 8000 for group 0 and 4000 for group 1, while for the training set, we have 200 for group and 100 for group We average the results over 100 simulations, for each simulation, the test data is fixed, the training data is randomly generated with the number of samples mentioned above. The projections are learned on training data and measured test data by the summation of ARE, and ABDiff. We fixed À 0.1, which is not too small for achieving fair projections, and not too large to clearly observe the effects ε, and we also fixed Eo for better visualization. Note that we still compute E a/ VNi in which, is tested with 100 values evenly spaced in [0, 10].
The experiment results visualized in Figure The result suggests that increasing the ambiguity set radius can improve the overall model's performance. This justifies the benefit of adding distribu- tional robustness the fairness-awar PCA model. After saturation point, a too large radius can lessen the role of empirical data, and the model prioritizes a more extreme distribution that is far from the target distribution, which causes the reduction the model's performance on target data.
5.0
4.5
ADE A600T
4.0
Figure 4: Performance changes w.r.. the ambiguity set's radius. The solid line is the average over 100 simulations, and the shade represent the 1-standard deviation range.
19

---

Published as a conference paper at ICLR 2022
D.2.2 PARETO CURVES
Figures andd the Pareto frontier for two datasets (Biodeg and German Credit) with principal components, One can observe that RF RFPCA produces points that dominate other methods based on the trade-off between ARE. and ABDiff.
Pareto plot with 3 features InSample (Blodeg 40-features)
Pareto plot with 3 features InSample (German Credit 48-features)
Absolute difference of reconstruction errer between groups
RFPCA
Absolute difference of recorstructon erter betwern groups
2.00
RFPCA
FairPCA
FairPCA
CFPCA
1.75
CFPCA
20-
1.50
18
1.25
1.00
0.75
0.50
0.25
10-
23.5
26.5
27.0
Reconstruction data)
Reconstruction emror tall data)
Figure 5: Pareto curves on Biodeg
Figure 6: Pareto curves on German Credit
dataset (all data) with 3 principal components
dataset (all data) with 3 principal components
20

---

Published as a conference paper at ICLR 2022
D.2.3 PERFORMANCE WITH DIFFERENT PRINCIPAL COMPONENTS
We collect here the reconstruction errors for different numbers of principal components.
Average reconstruction error On Default Credit (22-dim) in-sample
Average reconstruction error on Default Credit (22-dim) Out-of-sample
12
PCA- Groppr
PCA Groupo
PCA Group1
PCA Groupl
FairPCA Groupo
10
FairPCA Groupo
Average Subgroup Reconstruction Errer
FairPCA Groupl
Averago Subgraup Rcconstruction Error
FairPCA Groupl
RFPCA GroupD
RFPCA Groupo
RFPCA Groupl
8
RFPCA Groupl
2
10
12
14
18
20
2
10
12
14
16
18
20
Figure 7: Subgroup average error with different & on Default Credit dataset
Figure 8: Subgroup average error with different & on Default Credit dataset
Average reconstruction error on E. Coli (7-dim) In-sample
Average reconstruction error on E. Coli (7-dim) Out-of-sample
4.0
PCA GroupO
PCA GroupO
PCA Groupl
PCA Groupl
FairPCA Group0
FairPCA Group0
Average Sibgroup Reconsr pranssruction Frror
FairPCA Group1
Average Subgroup Reconstruuction Error
FairPCA Groupl
3.0
RFPCA Groupo
RFPCA - Groupo
RFPCA Group1
RFPCA Groupl
2.5
2
1.5
1.0
0.5
2
6
Figure 9: Subgroup average error with different i on E. Coli dataset
Figure 10: Subgroup average error
with different on E. Coli dataset
21

---

Published as a conference paper at ICLR 2022
Average reconstruction error on Magic (10-dim) In-saeple
Average reconstruction error
Magic (10-dim) outple
PCA Groupo
PCA Groupo
PCA Groupl
PCA Groupl
FairPCA Group0
FairPCA Group0
Average subgroup Reconstruction Enrrr
6
FairPCA Graupl
Average Subgroup Reconstruction Emrr
FairPCA - Groupl
RFPCA GroupO
RFPCA Groupo
St
RFPCA Group1
RFPCA Group1
-
2
Figure 11: Subgroup average error with different on Magic dataset
Figure 12: Subgroup average error with different on Magic dataset
Average reconstruction error
Average reconstruction error on Steel (24-dim) Out-of-sample
on Steel (24-dim) in-sample
PCA Groupō
PCA Groupō
ICA-Groupl
PCA Groupl
FairPCA Group0
14
FairPCA Group0
Average Subgroup Reconstuction Enrrr
FairPCA Groupl
Avorage Subgroup Rcconstruction Enrr
FairPCA Groupl
RFPCA Groupo
RFPCA Groupo
RFPCA Groupl
RFPCA Groupl
20-
4
2
10
12
14
16
18
20
2
10
12
14
16
18
20
Figure 13: Subgroup average error with different & on Steel dataset
Figure 14: Subgroup average error with different & on Steel dataset
22

---

Published as a conference paper at ICLR 2022
Average reconstruction error on Wine Quality (11-dim) In-sample
Average reconstruction error Wine Quality (1l-dim) Out-of-sample
8-
PCA GroupO
8
PCA GroupO
PCA Groupl
PCA Groupl
FairPCA Groupo
FairPCA GroupD
Average Subgroup Reconstruction Error
FairPCA Groupl
Average Subgroup Reconstruction Error
FairPCA Groupl
RFPCA - Group0
RFPCA - Groupo
RFPCA Group1
RFPCA Groupl
5.
4
2
2
2
6
10
2
8
10
Figure 15: Subgroup average error
Figure 16: Subgroup average error
with different & on Wine Quality dataset
with different on Wine Quality dataset
23


Reviews:
[{"review_id": "q5AbtHKzXl6", "reviewer": "Reviewer_wgFH", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper tackles an interesting problem, and is mostly well-written and easy to follow. The problem formulation and re-formulation steps are not trivial.\n\nHowever, there are places in this paper that needs more explanation and justification. \n\n- There needs to be a discussion on the assumptions for Theorem 3.2. How restrictive are they? Were they satisfied in the experiments? For example, (i) implies that the regularization parameter is less than 0.5, which will restrict the flexibility shown in Figure 1. \n- The ambiguity set B(P) is a central part of the proposed method. What is the justification for using Wasserstein-type divergence as opposed to others? Also, the second line of equation (5) needs to be explained more carefully.\n- In defining $gap_\\mu(U)$, is $||U'-U||_F$ the best choice to measure the difference between two orthonormal matrices? One can think of many examples with rotations where $U$ and $U'$ span the same subspace but will have very large frobenius norm difference. A more typical choice in the literature would be $||UU^T - U' U' ^T||$. \n- Is the RHS of Theorem 4.3. necessarily decreasing with $t$? It's not obvious to me why this guarantees convergence.\n- Experiments and discussions on the robustness of the proposed method are lacking, compared to the fairness aspect. Are there any comparisons to other robust PCA methods, or simulation experiments?\n- In the experiments, how were the sensitive attributes chosen? Were they based on what are considered sensitive attributes in the real world, e.g. race, gender, or were they chosen randomly?\n- This paper is lacking a conclusion section. It would have made it stronger if the authors mentioned how they would extend this work to non-binary sensitive attributes.\n- On a very high level, it's not *as* obvious why reconstruction error should be balanced compared to, for example, in supervised learning. The toy example in Figure 1 is helpful but I'm wondering if the authors can describe a real-world example where bias in reconstruction error will be harmful - where PCA is the end result rather than a preprocessing step for supervised learning problem.\n\nOther minor problems that can be fixed with careful editing, such as:\n- Page 2: \"signal process\" -> signal processing\n- Page 5: \"By the definition the ambiguity...\"\n- Page 7: \"At each iteration, the point...\" sentence is hard to parse.\n- Page 9: \"In Each dataset, ...\" \"other features is...\" \"the the generalization...\"", "summary_of_the_paper": "This paper aims to improve fairness and distributional robustness in dimensionality reduction techniques. More specifically, it proposes a regularization term to principal compoenent analysis (PCA) such that the expected reconstruction errors for each groups (conditioned on a binary sensitive variable) are similar, and relaxes constraint on the empirical probability distribution by maximizing over a Wasserstein-type ambiguity set. This is optimized with subgradient descent on the Stiefel manifold, and numerical experiments comparing the proposed method to previous fair PCA are conducted.", "summary_of_the_review": "It's a good paper, but is missing some interpretation of its theoretical results and justifications. It is likely that I will change my recommendation after the discussion period, if the authors provide more details.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "p5rEK2lAq4k", "reviewer": "Reviewer_oUxn", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength: + Robustness criterion for linear transformation of fairness can be formulated as a single optimization problem as long as the ambiguity set is defined using a single normal distribution.\n+ A smooth reformulation is presented which can be solved using standard riemannian gradient descent algorithm.\n\nWeakness: - Generalization to multiclass sensitive attribute, and other fairness metrics is missing.\n- Figure 1 illustrating different approaches is nice, although it fails to capture the distributional robustness aspect of the formulation presented in the paper.\n- It is hard to follow the paper since the notations are not standard.\n- Experimental results are preliminary and conclusions are not clear.\n", "summary_of_the_paper": "The paper presents an optimization based formulation of fair principal components analysis problem with a particular focus on the robustness of the optimal models. In the case when the desired robustness is specified by Gaussian distributions, the authors show how to adapt a manifold subgradient descent to solve a non-convex reformulation of the proposed fair pca formulation.\n", "summary_of_the_review": "Justification:  The technical content is interesting, although in its current form, it is hard to follow. For example, why is problem (4)  infinite dimensional? As far as I can see, the decision variables in optimization problem (4) is a d by k matrix which is finite dimensional, although the zeroth order and first order oracles maybe noisy. This is evident in their convergence analysis, where they can only bound the gradient norm of the reformulated variable U (which in its original formulation is V). The paper contains some technical insights in the formulation as a SDP which happens to have a nice connection to Riemannian geometry as illustrated. However, at times, the paper reads as if one has to refer multiple textbooks just to make sure that paper indeed is technically correct. For example, many quantities appear without definitions (\\hat{p}_a is not defined), and sometimes with somewhat unclear notations (Q(dx\\times da). To me, the biggest technical question that the paper fails to answer is the generalization to nonbinary features. This is important in the context of the paper's contribution because we know that standard SDP solvers do not scale to settings with big or high dimensional data. The convergence rate in this setting is also not clear since a naive extension would indicate at least polynomial slowdown. The experiments are also inconclusive since the paper focuses exclusively on the binary sensitive attribute setting, and with just one metric. While the approach here may not be directly extendable to other metrics, the paper completely misses on discussing when the chosen formulation is appropriate for practitioners. \n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "XrcL6hpyMRL", "reviewer": "Reviewer_Hi3C", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n\n1. The paper is nicely written and easy to follow. The motivation is clear and the problem formulation is novel.\n\n2. Results on reformulation and riemannian gradient descent render tractable algorithms for solving the proposed problem. I do not find major technical errors.\n\n3. Numerical experiments are based on real data set and are comprehensive.\n\nWeakness:\n\n1. The formulation simply combines distributioanlly robust PCA with a fairness penalty. Although this formulation is new, it seems a natural extension of existing frameworks.\n\n2. The theoretical results are relatively easy to derive based on existing results on distributionally robust optimization and riemannian optimization. Hence, the originality is not methodological, but just applying well-established tools to a new formulation.\n\n3. The dimensions of variables in UCI datasets are still relatively low (<=40 according to Table 2). The results can be strengthened by testing on datasets of larger dimensions.\n", "summary_of_the_paper": "This paper studies the formulation, reformulation and algorithm for distributionally robust fairness-aware PCA. The reformulation exploits techniques from distributionally robust optimization, and the algorithm is based on reimannian sub-gradient descent. The theory is tested on UCI datasets.", "summary_of_the_review": "This paper proposes a new formulation for PCA and derives a complete set of results with regard to the computational issues of the new results. My only reservation is that most theory are relatively straightforward compared to existing literature on distributionally robust optimization and Riemannian optimization.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "ExdywrgBjg0", "reviewer": "Reviewer_8Bm5", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The strength of the paper is the rigorous optimization framework to solve the proposed fair PCA, with a nice reformulation and theoretical property. \n\nThe weakness of this paper is in the following folds. First, it is not very clear on the significance of the novelty. The authors cast the PCA problem as the minimization of the penalized reconstructed error. Such a formulation is widely common in the machine learning community. Because of the L2-type loss function, the use of the distributionally robust optimization is not that surprising with the known complication on the penalty part. Second, the proposed method involves the selection of tuning parameter. While the authors seem to avoid the discussion tuning parameter selection. Moreover, the numerical experiments just consider the use of fixed value of these tuning parameters, which makes reader hard to judge the quality of the numerical comparison. Third, due the use of the penalty to engage the fairness, it is not how good the proposed method enables the fairness, especially there is no criterion on the selection of the tuning parameter \\lambda. Fourth, under the context of fairness, the authors also avoid the discussion on how the choice of k, the number of PCs, would affect the fairness.  ", "summary_of_the_paper": "This paper considers the fair PCA with distribution ally robust optimization algorithm. The problem is of great interest. The key idea of the proposed method is to consider a penalized loss function with the penalty on the fairness criterion. The distributionally robust optimization and its reformulation is developed for parameter estimation. ", "summary_of_the_review": "\nBased on the aforementioned strengths and weakness, I would consider the proposed method contain interest merits, with the conservation of the sufficiency on the methodology novelty, framework rigor, and the applicability in practice.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper considers the problem of distributionally robust fair PCA for binary sensitive variables. The main modeling contribution of the paper is the consideration of fairness and robustness of the PCA simultaneously, and the main technical contribution of the paper is the provision of a Riemannian subgradient descent algorithm for this problem and proof that it reaches local optima of this non-convex optimization problem. The results will be of interest to those working at the intersection of fair and robust learning.
Decision: Accept (Poster)