How Robust is Your Fairness? Evaluating and Sustaining Fairness under Unseen Distribution Shifts
Authors: Haotao Wang
Junyuan Hong
Jiayu Zhou
Zhangyang Wang
Venue: Accepted by TMLR ()

Abstract: Increasing concerns have been raised on deep learning fairness in recent years. Existing fairness-aware machine learning methods mainly focus on the fairness of in-distribution data. However, in real-world applications, it is common to have distribution shift between the training and test data. In this paper, we first show that the fairness achieved by existing methods can be easily broken by slight distribution shifts. To solve this problem, we propose a novel fairness learning method termed CUrvature MAtching (CUMA), which can achieve robust fairness generalizable to unseen domains with unknown distributional shifts. Specifically, CUMA enforces the model to have similar generalization ability on the majority and minority groups, by matching the loss curvature distributions of the two groups. We evaluate our method on three popular fairness datasets. Compared with existing methods, CUMA achieves superior fairness under unseen distribution shifts, without sacrificing either the overall accuracy or the in-distribution fairness.

Published in Transactions on Machine Learning Research (03/2023)
How Robust is Your Fairness? Evaluating and Sustaining Fairness under Unseen Distribution Shifts
Haotao Wang University of Terps at Austin
htuong edu
Junyuan Hong
hongju 2 edu
Michigan State University
Jiayu Zhou Michigon State University
RayuzSmsu.ede edu
Zhangyang Wang
ollswang@utezass edu
University of Teras at Austin
Reviewed on OpenReview: https: //tps://on // operreveew..eee... net/ forum? id- 11pGlecTs2
Abstract
Increasing concerns have been raised on deep learning fairness in recent years. Existing fairness-aware machine learning methods mainly focus on the fairness of in-distribution data. However, in real-world applications, it is common to have distribution shift between the training and test data. In this paper, we first show that the fairness achieved by existing methods can be easily broken by slight distribution shifts. To solve this problem, we propose a novel fairness learning method termed CUrvature MAtching (CUMA), which can achieve robust fairness generalizable to unseen domains with unknown distributional shifts, Specifically, CUMA enforces the model to have similar generalization ability the majority and minority groups, by matching the loss curvature distributions of the two groups. We evaluate our method on three popular fairness datasets. Compared with existing methods, CUMA achieves superior fairness under unseen distribution shifts, without sacrificing either the overall accuracy or the in-distribution fairness.
1
Introduction
With the wide deployment of deep learning in modern business applications concerning individual lives and privacy, there naturally emerge concerns machine learning fairness (Podesta et al., 2014 Muñoz al.
2016 Smuha 2019 Although research efforts on various fairness-aware learning algorithms have been carried out [Ewwards & Storkey 2016 Hardt et al. 2016 Du et al. 2020), most of them focus only on equalizing model performance across different groups on in-distribution data.
Unfortunately, in real-world applications, one commonly encounters data with unforeseeable distribution shifts from model training. Tt has been shown that deep learning models have drastically degraded performance (Hendrycks & Dietterich 2019 Hendrycks behaviors (Qiu et al. 2019 Yan et al.) 2021] under unseen distribution shifts. Most previous fairness learning in-distribution data. However, those algorithms do not take into considcration the stability or "robustness" of their found fairness-aware minima. Taking object detection in self-driving cars for example, it might have been calibrated over high-quality clear images to be "fair" with different pedestrian skin colors; however such fairness may break down when applied to data collected in adverse visual conditions, such as inclement weather, poor lighting, or other digital artifacts. Our experiments also find that previous state-of-the-art fairness algorithms would be jeopardized if distributional shifts are present in test data. The above findings beg the following question:
al., 2020 2021 Taori al. 2020) and show unpredictable
algorithms aim to achieve fairness

---

Published in Transactions on Machine Learning Research (03/2023)
Lxx)
(x)
L(x)
X
X
X
Xi
Xa X
X1 x
Xa X
X1
X
(c) Our robust fairness learning (In-distribution & robust fairness) Figure 1: Illustrating the achieved fairness of normal training, traditional fair training and our proposed robust fair training algorithms. Horizontal and vertical axes represent input and corresponding loss value C(x), respectively. Solid blue curves show the loss landscapes. Circles denote majority data points (%a and r',,, while triangles denote minority data points (I; and x(). Green points (I, and x;) are in-distribution data while red ones (x', and zy) are sampled from test sets with distribution shifts, (a) Normal training results in unfair models: minority group has worse performance (i.e., larger loss values). (b) Traditional fairness learning algorithms can achieve in-distribution fairness but not in robust way: small distribution shift can break the fairness due to loss curvature biases across different groups. In fact, such learned fair models can have almost the same large bias as the normally trained models when facing distribution shifts. (c) Our robust fairness learning algorithm can simultancously achieve fairness both on in-distribution data and at distribution shifts, by matching both loss values and loss curvatures across different groups.
(a) Normal learning
(b) Traditional fairness learning
(Unfair)
(In-distribution fairness)
Using the currently available training set, how to achieve robust fairness that is generalizable to unseen domains with unpredictable distribution shifts?
To solve this problem, we decompose it into the following two-step objectives and achieve them one by one:
1. The minority and majority groups should have similar prediction true positive rates and false positive rates on in-distribution data. This is usually attained by traditional in-distribution fairness goals.
2. The minority and majority groups should have similar robustness under unseen distributional shifts. In this context, the "robustness" refers to model performance gap between in-distribution and unseen uut-of-distributie data: the larger gap the weaker.
The first objective well studied and can be achieved by existing fairness learning methods such as adversarial training Edwards & Storkey (2016); Hardt al. (2016); Du et al.) (2020). In this paper, we focus our efforts addressing the second objective, which has been much less studied. We present empirical evidence that the fairness achieved by existing in-distribution oriented methods can be easily compromised by even slight. distribution shifts.
Next, to mitigate this fragility, we note that model robustness against distributional shift is often found to be highly correlated with the loss curvature (Bartlett al., 2017 Weng et al., 2018). Our experiments further observed that, the local loss curvature of minority group often much larger than that of majority group, which explains their discrepancy of robustness. Motivated by the above, we propose new fairness learning algorithm, termed Curvature Matching (CUMA), to robustify the fairness. Specifically, CUMA enforces the model to have similar robustness on the majority and minority groups, by matching the loss curvature distributions of the two groups. As a plug-and-play modular, CUMA can be flexibly combined with existing in-distribution fairness learning methods, such as adversarial training, to fulfil our overall goal robust fairness. We illustrate the core idca of CUMA and its difference compared with traditional in-distribution fairness methods in Figure [ We evaluate our method on three popular fairness datasets: Adult, C&C,, and CelebA. Experimental results show that CUMA achieves significantly more robust fairness against unseen
2

---

Published in Transactions on Machine Learning Research (03/2023)
distribution shifts, without sacrificing either overall accuracy or the in-distribution fairness, compared to traditional fairness learning methods.
2 Preliminaries
2.1 Machine Learning Fairness
Wide Existence of Unfairness in Machine Learning With the wide application of machine learning in real-world applications, increasing concerns have been raised on the potential unfairness in machine learning model and algorithus. For instance, [Buolamwini Gebru (2018) shows current commercial gender-recognition systems have substantial accuracy disparities among groups with different genders and skin colors. [Wilson ct al. 2019) observe that state-of-tho-art segmentation models achieve better performance on pedestrians with lighter skin colors. In (Shankar et al. 2017: de Vries el al. 2019), it is found that the common geographical bias in public image databases can lead to strong performance disparities among images from locales with different income levels. Nagpal et al.] (2019) reveal that the focus region of face-classfficttion models depends on people's ages or races, which may explain the source of age- and race-biases of classifiers. On the awareness of the unfairness, many efforts have been devoted to mitigate such biases in computer vision tasks.
Problem Setting and Metrics Machine learning fairness can be generally categorized into individual fairness and group fairness (Du et al. 2020). Individual fairness requires similar imputs to have similar predictions (Doork et al. 2012). This paper focuses group fairness, whose definition is provided below. Given input data X E R with sensitive attributes A E {0, 1} and their corresponding ground truth labels E [0, 1}, group fairness requires a learned binary classifier ((;00) R - {0, 1} parameterized by 0 to give equally accurate predictions (denoted as Ý = f())) on the two groups with A = and A 1. Multiple fairness critcria. have been defined in this context. Demographic parity (DP) (Edwards & Storkey 2016)
requires identical ratio of positive predictions between two groups: P(Ý = 1/4 0) P(Y 1|4 1). Equalized Odds (EO) (Hardt et al. 2016) requires identical false positive rates (FPRs) and false negative rates (FNRs) between the two groups: p( + Y|A= = O, y) p(ř + YIA 1,Y = y), Vy E {0, 1].
Based on these fairness criteria, quantified metrics are defined to measure faimesss. For example, the EO and
EOpp distances (Madras et al. 2018) are defined as follows:
AgO: = Σ IP(Ý YYIA= 0, Y,Y = v) - P( +YIA = 1,Y y)l
(1)
(E(0,1}
Δ = P( # Y 0,Y 1) 1) P( # Y = 1, =)]-
(2)
Mary et al. (2019) proposed to generalize the fairness mcasurements from binary variables to multi-variato or continuous variables by using the Renyi maximum correlation coefficient.
Bias Mitigation Methods Many methods have been proposed mitigate model bias. Data pre- processicg methods such as re-weighting (Kamiran & Calders 2012) and datas-transfommation (Calmon et al. 2017 have been used to reduce discrinination before model Training. In contrast, Hardt et al. (2016) and Zhao et al. (2017) propose post-processing methods to calibrate model predictions towards a desired fair distribution after model training. Instead of pre- or post-processing, researchers have explored to enhance fairness during training. For example, Madras et al. (0018) uses a adversarial training technique and shows the learned fair representations can transfer unseen target tasks. The key technique, adversarial training (Edwards & Storkey| 20166,, was designed for feature disentanglement on hidden representations such that sensitive (Edwards & Storkey 2016) or domain-specific information (Ganin et al. 2016) will be removed while keeping other useful information for the target task. The hidden representations
typically the output of intermediate layers of neural networks (Ganin et al., 2016 Edwards & Storkey 2016 Madras et al. 2018). Instead, methods, like adversarial debiasing (Zhang el al. 2018) and its simplified version (Wadsworth et al. 2018), directly apply the adversary on the output layer of the classifier, which also promotes the model fairness. Observing the unfairness due to ignoring the worst learning risk of specific samples, Hashimoto ett al. (2018) proposes use distributionally robust optimization which provably bounds the worst-case risk over groups Creager et al. (2019) proposes flexible fair representation learning framework based VAE
3

---

Published in Transactions on Machine Learning Research (03/2023)
Kingma & Welling 2013], that can be easily adapted for different sensitive attribute settings during run-time. Sarhan et al. [2020) uses orthogonality constraints as a proxy for independence to disentangles the utility and sensitive representations. Martinez et al. (2020) formulates group fairness with multiple sensitive attributes as a multi-objective learning problem and proposes a simple optimization algorithm to find the Pareto optimality.
Another line of research focuses on learning unbiased representations from biased ones (Bahng et al. 2020;
Nam et a 2020). Bahng et al. (2020) proposes a novel framework to learn unbiased representations by explicitly en enforcing them to be different from set of pre-delined biased representations. [am al. (2020) observes that data bias can be either benign or malicious, and removing malicious bias along can achieve fairness. Li & Vasconcelos (019) jointly learns a data re-sampling weight distribution that penalizes easy samples and network parameters. Li et al. (2019) scaled by higher-order power to re-emphasize the loss of minorily samples (or nodes) in distributed learning. [Agaraal el al. (2018) formulates a fairness-const optimization to train a randomized classifier which is provably accurate and fair. Quadrianto et al. (0019) casts the scnsitive information removal problem as a data-to-data translation problem with unknown target
domain. Wang al. (2019) shows the elfectiveness of adversarial debiasing technique (Zhang et al.. 2018 in
fair image classification and activity recognition tasks. Beyond the supervised learning, FairFaceGAN (Wang
et al. 2020) is proposed to prevent undesired sensitive feature translation during image editing. Similar ideas
have also been successfully applied to visual question answering (Park et al. 2020). Szabó et al. (2021)
proposed tilted cross-entropy to achieve fairness in semantic segmentation. Prost et al. (2019) proposed to
improve the trade-off between performance and fairness by kernel- based distribution matching. Baharlouei et al. 2019) used Renyi correlation as a regularization term in training to impose group fairness. Cho et al.
(2020) used kernel density estimation to quantify fairness measurements in a differentiable way, which is used in training to optimize the trade-off between accuracy and fairness. Recently, Lowy et al. (2022) developed the first stochastic in-processing fairness algorithm with guaranteed convergence in stochastic optimization with any batch size. The proposed method, named FERMI, is capable of achieving fairness on multiple (non-bing binary) sensitive attributes and non-binary targets, performing well even with minibatch size as small as onc. FERMI empirically achieves stato-of-tho-art performance across multiple different problem scttings, oulperforming previous methods by a significant margin. For more related works on machine learning fairness, please refer to the recent survery (Hort et al. 2022).
Fairness under distributional shift Recently, several papers have investigated the fairness learning problem under distributional shift (Mandal et al. 2020 Zhang et al., 2021 Rezaei et al. 2021 Singh et al., 2021 Dai & Brown 2020). Although these works are relevant with ours, there are significant differences in the problem settings. Zhang et al. (202T) studied the problem of enforcing fairness in online learning, where the training distribution constantly shifts. The authors proposed to adapt the model to be fair on the current known data distribution. In contrast, our work aims to generalize fairness learned on current distribution to unknown and unseen target distributions. In our setting, the algorithm can not access any training data from the unknown target distributions. [Reaci et al.] (2021) studied to preserve fairness under covariate shift. However, their method requires unlabeled data from the target distribution. In other words, they assume the target distribution to be known. In contrast, our method is more general and works on unknown target distributions. (Singh et al.] (2021) also studied to preserve fairness under covariate shift. However, their method is based on model adaptation and requires the existence of a joint causal graph to represent the data distribution for all domains. Our method, however, does not require such and generally works on any unseen target distributions, Dai & Brown (2020) studies fairness under label distributional shift, while we focus covariate shift and temporal shift. Mchrabi et al. (2021) proposed data poisoning attacks targeting fairness.
2.2 Model Robustness and Smoothness
Model generalization and robustness have been shown to be highly correlated with model smoothness Moooavi-Dezzooli et al. 2019: Weng et al. 2018). Weng et al. (2018) and Guo et al. (2018) use the local
Lipschilz constant to estimate model robustness against small perturbations ions on inputs within a hyler-ball.
Moosavii-ezzoon et al. (2019) proposes to improve model robustness by adding a curvature constraint to encourage model smoothness. Miyato et al. (2018) approximates model local smoothness by the spectral norm of Hessian matrix, and improves model robustness against adversarial attacks by regularizing model smoothness.
4

---

Published in Transactions on Machine Learning Research (03/2023)
3 The Challenge of Robust Fairness
In this section, show that the current statc-of-tho-art in-distribution fairness learning methods suffer significant performance drop under unseen distribution shifts. Specifically, we train the model using normal training, AdvDebias (Zhang et al.
2018) and LAFTR (Madras et al. 2018) on Adult (Kohavi 1996) dataset (i.e., US Census data before 1996). We evaluate the ĐEO on the original Adult test set and the 2015 subset of Folktables datase [Ding et al.) 2021 (i.e., US Census data in 2015) respectively, in order to check whether the fairness achieved on in-distribution data is preserved under the temporal distribution shift. The results are shown in Figure] 3
As we can see, LAFTR and AdvDebias successfully improve the in-distribution fairness compared with normal training. However, both methods suffer significant performance drop in terms of Ago and go under the temporal distribution shift. Morcover, under the distribution shift, the Ago and Agopp achieved by LAFTR and AdvDebias are almost, the same with or even worse than that of normal training. In other words, the models trained by LAFTR and AdvDebias are almost as unfair as a normally trained model under this naturally occurring distribution shift.
4
Curvature Matching: Towards Robust Fairness under Unseen Distributional Shifts
In this section, present our proposed solution for the robust fairness challenge described in Section 3
4.1 Loss Curvature as the Measure for Robustness
Before introducing our robust fairness learning method, we need to first define the measure for model robustness under unseen distributional shifts.
Consider a binary classifier ()) trained on two groups of data X1 and X2. Our goal is to define a metric to measure the gap of model robustness between the two groups. Previous research (Guo et al. 2018 Weng et al. 2018 has shown both theoretically and empirically that deep model robustness scales with its model smoothness Motivated by the above, use the spectral norm of Hessian matrix to approximate local smoothness as mcasure of model robustness. Specifically, given an input x, the Hessian matrix (xx) is defined as the second-order gradient of L(x) with respect to model weights 0: H(x) Vzl(x). The approximated local curvature C(x) at point æ is thus defined as:
C(x) o(H(x),,
(3)
where (H) is the spectral norm (SN) of H: a (H) supe:llela=1 Hellz. Intuitively, C(x) measures the maximal directional curvature or change rate of the loss function æ. Thus, smaller C(x) indicates better local smoothness around x.
Practical Curvature Approximation It is inefficient to directly optimize the loss curvature through Eq. (3), since it involves high order gradients 2 To solve this problem, we use a one-shot power iteration method (PIM) for practical approximation of C(x) during training. First we rewrite C(x) with the following form: C(x) (x) = [I where v is the dominant eigenvector with the maximal eigenvalue, which can be calculated by power iteration method. In practice, we estimate the dominant eigenvector sign(s) [(ign[[]]|
v, where g Vol(x). This is because previous works have observed a large similarity between the dominant eigenvector and the gradient direction (Miyato el al. 2018 Moosavi-Dezioon et al. 2019). We further approximate Hessian matrix by finite differentiation on gradients:
by the gradient direction: : =
'ee mainly focus covariate shift since the curvature smoothness is related to robustness under covariate shift. Our experiments on CelebA and C&C datasets verified the effectiveness of our method under covariste shift. For example, on CelebA dataset, the source (training) distribution contains young faxes and the target (test) distribution contains old faces, which is covariate shift setting. Our method is also empirically effective against temporal shift, verified in experiments on Adult datasct.
2The Hessian matrix itself involves second order gradients, and backpropagation through Eq. requires even higher order gradient on top of the Hessian matrix SN.
5

---

Published in Transactions on Machine Learning Research (03/2023)
H(x) Vollothe)-Volle where h is small constant. As result, the final approximation of curvature smoothness is
C(x) VoL(o - h8) voC(x)]
(4)
[1
4.2 Curvature Matching
Equipped with the practical curvature approximation, now we can match the curvature distribution of the two groups by minimizing their maximum-mean-disonn mnean-isccrpaney (MMD) (Gretton et al., 2012) distance. Suppose
c(X,) Q1 and C(X2) Q2, where C(-) is defined in Eq. define the curvature matching loss functions
Con = MMD* 22).
55)
The MMD distance, which widely usod to measure the distance between two high-dimensional distributions Bitikowski et al. 2018), is defined
in deep learning (Li et al. 2015 2017:
MMD Q) -Ep
2EEp exk( Y)J EE))(XY)
(6)
where ~ P,Y Q and -(---) is the kernel function. In practice, we use finite samples from P and Q to statistically estimate their MMD distance:
M
M k(x,,
(MD)(
xe)
M
M
N
N
k(2,, U5)
k(y;,Up)
(7)
MN
N
where {x; ~ PIM, {ug ~ Q1 1= and we use the mixed RBF kernel function as k(x, v). Selecting a proper
kernel width is important for RBF kernel (Sutherland et al. 2016 Liu et al. 2020). We follow Li et al. 2015 Lie v1 20
2017) to a series of different kernel width simultaneously: k(,, y) Eoes e During training, we use the whole batch of training samples to estimate MMD, where the batch size is 64. In other words, M + N = 64 in equation 7
with $ {1, 2, 4, 8, 16).
As a side note, MMD has been previously used in fairness learning. Quadrianto & Sharmanska (017 defines a more general fairness metric using MMD distance, and shows DP and EO to be the special cases of their unified metric. Their paper, however, still focuses on the in-distribution fairness. In contrast, our CUMA minimizes the MMD distance on the curvature distributions to achieve robust. fairness.
Back to our method. After defining Lem, add it to the traditional adversarially fair training Ganii et al. 2016 Madras et al. 2018) loss function as a regularizer, in order to attain both in-distribution fairness and robust fairness. As illustrated in Figure 2 our model follows the same "two-head" structure as traditional adversarial lcarning frameworks (Ganin et al., 2016 Madras et al., 2018), where h2 the utility head for the target task, hr is the adversarial head predict sensitive attributes, and fs is the shared backbone, Suppose for each sample 24, the sensitive attribute a, and the corresponding target label is yi, then our overall optimization problem can be written
min max L min max(Caf CLaad Lem)
(8)
02.
3Thus the binary classifier ((:0) he(Ss(:0.):;p 0e), with J0

---

Published in Transactions on Machine Learning Research (03/2023)
where
N
Laf =
0:):
(9)
N
N
1
Lade
((h ((a((f((mr
(10)
N
((..) is the cross-entropy loss function, a and y are trade-off hyperparameters, and N is the number of training samples.
L Lem
C(.)
Q2
f(:;0) = he (fs (:; 0s);0r)
x
fs(::0)
hc(:;,,)
Lclf
ha (; 0g)
Ladv
Figure 2: The overall framework of CUMA. is the input sample. h, is the utility head for the target task. ha is the adversarial head to predict sensitive attributes. Sa is the shared backbone. C(.) is the curvature estimation function, as defined in Eq. (3 Q and Q2 are local curvature distributions of majority and
minority groups, respectively. Lem, Las and Ladv are three loss terms as defined in Eq. 5 99 and (10p respectively.
5
Experiments
5.1
Experimental Setup
Datascts and pro-processing Experiments conducted three datasets widely used to evaluate machine learning fairness: Adult [Kohavi 1996). and CelebA (Liu el al. 2015), and Communities and Crime (C&C) (Redmond & Baveja 2002 Adult dataset has 48,842 samples with basic personal information such as education and occupation, where 30,000 are used for training and the rest for evaluation. The target task is to predict the person's annual income, and we use "gender" (male or female) as the sensitive attribute. The features in Adult dataset are of either continuous (e-g., age) or categorical (e.g. sex) values. We use one-hot. encoding on the categorical features and then concatenate them with the continuous ones. We use data whitening the concatenated features. CelebA has over 200,000 images of celebrity faces, with 40 attribute annotations. The larget lask is to predict gender (male or female) and the sensitive altribute to protect is "chubby". We randomly select 10, 000 as training samples and 1, 000 as testing samples. All images are center-cropped and resized to 64 x 64, and pixel values are scaled to [0, 1]. C&C dataset has 1,994 samples with neighborhood population statistics, where 1,500 are used for training and the rest for evaluation. The target task is predict violent crime per capita, and we use TRacePetBlack (percentage of black population in the neighborhood) sensitive attributes. All features in C&C dataset are of continous values in [0, 1]. To
Traditional image classification datasets (e.g. ImageNet) directly applicable since they lack fairness attribute labels.

---

Published in Transactions on Machine Learning Research (03/2023)
fit in the fairness problem setting, we binarilize the target and sensitive attributes with the top-30% largest value as the threshold (As result P[A- 0] = 30% and Ply = 0] = =0%).. We also do data-whitening on C&C.
Models For C&C and Adult datasets, we use two-layer MLPs for Say he and ha- Specifically, suppose the input feature dimension is d, then the dimensions of hidden layers in fa and hy are d d 100 + 64 and 64 + 32 + 2, respectively. ha has identical model structure with h4 For all three sub-networks, ReLU activation function and dropout layer with 0.25 dropout ratio are applied between the two fully connected layers. For CelebA dataset, we use 18 as backbone, where the first three stages are used as fa and the last stage (logether with the fully connected classification layer) is used as ht. The auxiliary adversarial head ha has the same structure as he
Baseline methods We compare CUMA with the following state-of-the-art in-distribution fairness algo- rithms. Adversarial debiasing (AdvDebias) (Zhang et al. 2018) is one of the most popular fair training algorithm based on adversarial training (Ganin et al., 2016). Madras et al. (2018) proposes a similar framework termed Learned Adversarially Fair and Transferable Representations (LAFTR), by replacing the cross-entropy loss used in (Zhang et al. 2018) with a group-normalized & loss, which is shown to work better on highly unbalanced datasets. We also include normal (fairnes--inoorant) training as a baseline. We also add the "Oracle" bascline mothod: We train with AdyDebias on the target distribution and tested the model on the target distribution, to see how hard it is to directly achieve fairness on the target data distribution. Lastly, we also add RobustFair Mandal et al. 2020). previous method to pursue fairness beyond the training data, as baseline.
Evaluation metric We report the overall accuracy on all test samples in the original test sets. To measure in-distribution fairness, use AFO and AEOp on the original test sets. To measure robust fairness under distribution shifts, we use Ago and AgOpp on test sets with distribution shifts. See the following paragraph for the details in constructing distribution shifts. Since there is inherent trade-off between fairness and accuracy, we report the trade-offs between fairness and accuracy by sctting the loss function weights (e.g., a and 7) to different values. For example, the larger a, the better fairness and the worse accuracy. We show the results by visualizing the trade-off curves between fairness and accuracy of different methods. The closer the curve to the top-left corner (i..., with larger accuracy and smaller Ago or AEOP ), the better Pareto frontier is achieved.
Distribution shifts Adult datasct contains US Census data collected before 1996. We use the 2014 and 2015 subset of Polktables dataset (Ding et al. 2021), which contain US Census data collected in 2014 and 2015 respectively, as the test sets with distribution shifts. This simulates the real-world temporal distributional shifts.
On CelebA dataset, we train the model on 10, 000 "young" face images. We use another 1, 000 "young" face images in-distribution test set and 1, 0,000 "not young" face images as the test set with distribution shifts. This simulates the real-world scenario when the model is trained and used on people with different age groups We also construct another test set with a different type of distribution shift, by applying strong JPRG compression on the original 1, 000 "young" test images, following Hendrycks & Dietterich (2019). This simulates the scenario when the model is trained on good quality images while the test images has poor visual quality.
For C&C dataset, we construct two artificial distribution shifts by adding random Gaussian and uniform noises, respectively, to the test data. Specifically, the categorical features in C&C dataset are first onc-hot encoded and then whitened into float-value vectors, where noises are added. Both types of noises have mean p = 0 and has standard derivation o = 0.03
Implementation details Unless further specified, we set the loss trade-off parameter a to 1 in all experiments by default. We use Adam optimizer (Kingma & Ba. 2014) with initial learning rate 10 and weight docay 10-5 The learning rate is gradually decreased to by cosine annealing learning rate scheduler Loshchilov & Hutter 2016). On both Adult and C&C datasets, we train for 50 epochs from scratch for all methods. On CelebA dataser, we first normally train model for 100 epochs, and then finetune it for 20
8

---

Published in Transactions on Machine Learning Research (03/2023)
0.89
CUMA
CUMA
0.80
CUMA
0.80
AdvDebias
AdvDebias
AdvDebias
0.88
0.75
LAFTR
LAFTR
LAFTR
0.75
Accuracy
0.87
Robustfair
Normal training
Accuracy
0.70
Oracle
RobustFair
Accuracy
0.70
RobustFair
Oracle
Normal training
0.65
Normal training
0.86
0.65
0.60
0.85
0.60
0.55
0.55
0.84
0.05
0.10
0.15
0.10
0.15
0.10
0.15
0.20
In distribution Ago
Αço under temporal shift (2014)
Дςо under temporal shift (2015)
0.89
CUMA
CUMA
0.80
CUMA
0.80
AdvDebias
AdvDebias
AdvDebias
0.88
LAFTR
LAFTR
0.75
LAFTR
0.75
Accuracy
0.87
RobustFair
Accuracy
RobustFair
Accuracy
0.70
RobustFair
Normal training
0.70
Oracle
Oracle
Normal training
0.65
Normal training
0.86
0.65
0.60
0.85
0.60
0.55
0.55
0.20 0.25 0.30
0.20
0.25
0.30
0.35
0.20
0.25
0.30 0.35
In distribution А9орp
Деорр under temporal shift (2014)
Δеорр under temporal shift (2015)
Figure 3: Trade-off curves between fairness and accuracy of different methods. Results are reported Adult dataset with "Sex" as the sensitive attribute.
epochs using CUMA. For fair comparison, we train for 120 epochs on CelebA for all baseline methods. The constant in Eq. 4 set to by default.
5.2 Main Results
Experimental results on three datasets with different sensitive attributes are shown in Figures 3 we compare CUMA with the bascline methods on different metrics as discussed in Section 5.1 "Nommal" means standard training without any fairness regularization. All numberr are shown as percentages. Many intriguing findings can be concluded from the results.
5 where
First, that previous statc-of-the-art fairness learning algorithms would be jeopardized if distributional shifts are present in test data. For example, on Adult dataset (Figure 3 LAFTR achieves Ago 11.96% on in-distribution test set, while that number is increased to 13.95% on the 2014 test set and 14.80% the 2015 test set, which is almost as unfair as the normally trained model. Similarly, on CelenA dataset with "Chubby" as the sensitive attribute (Figure 4 LAFTR achieves Ago = 31.02% on the original CelebA test set, while that number increased to 35.88% and 37.46% under distribution shifts of user age and image quality, respectively.
that CUMA achieves the best robust fairness under distribution shifts under all evaluated settings, while maintaining similar in-distribution fairness and overall accuracy. For example, Adult dataset (Figure 3. CUMA achieves 1.92% and 1.96% less Ago than the socond-best performer (AdvDebias) on the 2014 and 2015 Census dataset, respectively. On CelebA dataset (Figure 4 with "Chubby" as the sensitive attribute, CUMA achieves 2.62% and 2.74% less Ago than the second-ber best performers under distribution shifts of user age and image quality, respectively. Moreover, still in Figure 4 CUMA and LAFTR achieve almost identical in-distribution fairness (the difference between their Ago on original test 0.5%), CUMA keeps the fairness under distribution shifts (with only around 1.6% increase in A Ao), while the fairness
Second,

---

Published in Transactions on Machine Learning Research (03/2023)
0.84
0.82
0.82
CUMA
CUMA
0.82
AdvDebias
AdvDebias
0.81
0.80
LAFTR
0.80
LAFTR
Accuracy
Accuracy
RobustFair
Accuracy
RobustFair
0.78
0.78
0.80
Oracle
Oracle
0.76
CUMA
0.76
0.79
AdvDebias
0.74
LAFTR
0.74
0.72
0.78
RobustFair
0.72
0.70
0.35
0.40
0.35
0.40
0.45
0.35
0.40
0.45
In distribution AEO
Σεо under Age Shift
Aςо under Visual Quality Shif
0.84
0.82
0.82
CUMA
CUMA
0.82
AdvDebias
AdvDebias
0.81
0.80
LAFTR
0.80
LAFTR
Accuracy
Accuracy
RobustFair Oracle
Accuracy
RobustFair
0.78
0.78
0.80
Oracle
0.76
CUMA
0.76
0.79
AdvDebias
0.74
LAFTR
0.74
0.72
0.78
RobustFair
0.72
0.70
0.25
0.30
0.25
0.30
0.35
0.30 Деорр under Visual Quality Sh
0.25
In distribution ΔеорР
Деорр under Age Shift
Figure 4: Trade-off curves between fairness and accuracy of different methods. Results are reported on CelebA dataset with "Chubby" as the sensitive attribute.
achieved by LAFTR is significantly worse, espocially under the image quality distribution shift, where the AoO increased by 6.44%.
5.3 Ablation Study
In this section, we check the sensitivity of CUMA with respect its hyper-parameten the loss trade-off parameters and 7 in Eq. (8) and h in Eq. 4 Results are shown in Table When fixing a the best trado-off between overall accuracy and robust fairness is achieved at round y 1, which we use the default
7. Varying the value of h hardly affects the performance of CUMA.
Table 1: Ablation study results on the loss trade-off parameters and y in the CUMA algorithm. Results are reported on C&C dataset with "RacePetBlackk the sensitive attribute.
0
0.1
1
10
0
0.1
10
0.1
1
Accuracy
89.21
86.94
85.40
83.75
84.79
85.19
85.40
84.79
85.32
85.40
Ao
64.27
66.51
28.74
33.16
39.84
38.85
28.74
27.95
30.56
28.74
with Gaussian noise
6 Conclusion
In this paper, we first observe the challenge robust fairness: Existing statc-of-the-art in-distribution fairness learning methods suffer significant performance drop under unseen distribution shifts. To solve this problem, propose novel robust fairness learning algorithm, termed Curvature Matching (CUMA), to simultaneously
10

---

Published in Transactions on Machine Learning Research (03/2023)
0.87
CUMA
CUMA
0.855
AdvDebias
0.86
AdvDebias
0.86
LAFTR
LAFTR
0.850
Accuracy
Accuracy
0.85
RobustFair
Accuracy
0.85
RobustFair
0.845
Oracle
Oracle
0.84
0.84
CUMA
0.840
0.83
AdvDebias
0.83
0.835
LAFTR
0.82
0.830
RobustFair
0.81
0.82
0.30
0.35
0.40
0.25
0.30
0.35
0.40
0.20
0.25
0.30
0.35
In distribution AEO
ΣEO under Gaussain Noise
ΔoΟ under Uniform Noise
0.87
CUMA
CUMA
0.855
AdvDebias
0.86
AdvDebias
0.86
LAFTR
LAFTR
0.850
Accuracy
Accuracy
0.85
RobustFair
Accuracy
0.85
RobustFair
0.845
Oracle
Oracle
0.84
0.84
0.840
CUMA
0.83
AdvDebias
0.83
0.835
LAFTR
0.82
0.830
RobustFair
0.81
0.82
0.15
0.20
0.25
0.1
0.2
0.15
0.20
0.25
In distribution ΔoорP
Деорр under Gaussain Noise
Деорр under Uniform Noise
Figure 5: Trade-off curves between fairness and accuracy of different methods. Results are reported on C&C dataset with "RlccPctBlackk as the sensitive attribute.
achieve both traditional in-distribution fairness and robust fairness. Experiments show CUMA achieves more robust fairness under unseen distribution shifts, without more sacrifice on either overall accuracies or the in-distribution fairness compared with traditional in-distribution fairness learning methods.
7
Broader Impact
With the wide application of machine learning models in many real-world applications, new requirements have been raised guarantee machine learning models make ethical and fair decisions. This work takes one step forward in this direction, by generalizing the fairness in-distribution data to out-of-distributioo data.
Limitations and Future Works Following previous fairness learning works (Zhang al. 2018 Madras et al. 2018), this paper focuses on the binary classification problem with binary privacy attributes. The proposed method potentially be generalized to the multiclass classification continuous prediction problems following similar ideas as in (Mary et al.] 2019 Denis et al. 2202)), which we leave as the future work. We also empirically find that the proposed method do not directly benefit model accuracy under distributional shifts. would be a good future direction to investigate how to simultaneously achieve better model accuracy and fairness under distributional shifts, Another topic we leave as future work is to study the thcorctical guarantocs on the generalization bounds for fairness.
The work of J. Zhou in part supported by the National Science Foundation under Grant ITS-2212174, IIS-1749940, Office of Naval Research -00012420-1-2882, and National Institute on Aging (NIA) RF1AG072449. The work of Z. Wang is in part supported by the National Science Foundation under Grant ITS-2212176.
Acknowleddeement
11

---

Published in Transactions on Machine Learning Research (03/2023)
References
Alekh Agarwal, Alina Beygelzimer, Miroslav Dudik, John Langford, and Hanna Wallach. A reductions approach to fair classification. In International Conference on Machine Learning, pp. 60-69, 2018.
Sina Baharlouci, Maher Nouiched, Ahmad Beirami, and Mcisam Razaviyayn. R 'eyyi fair inference. arXiv preprint ar (rX:::1006.1 12005, 2019.
Hyojin Bahng, Sanghyuk Chun,, Sangdoo Yun, Jaegul Choo, and Seong Joon Oh. Learning de- representations with biased representations. In International Conference on Machine Learning, pp. 2020.
Peter Bartlett, Dylan Foster, and Matus Telgarsky. Spectrally-normallzzed margin bounds for neural networks. In Advances in Neural Information Processing Systems, 2017.
Mikolaj Binkowski, Dougal J Sutherland, Michael Arbel, and Arthur Gretton. Demystifying MMD GANs, International Conference on Learning Representations, 2018.
Joy Buolamwini and Timnit Gebru. Gender shades: Intersectional accuracy disparities in commercial gender classification. In Conference on Fairness, Accountability and Transpareney, pp. 77-91, 2018.
Flavio P Calmon, Dennis Wei, Bhanukiran Vinzamuri, Karthikeyan Natesan Ranamurthy, and Kush R. Varshney. Optimized pre-processing for discrimination prevention. In Advances in Neural Information Processing Systems, pp. 3995-4004, 2017.
Jacwoong Cho, Gycongjo Hwang, and Changho Suh. A fair classifier using kernel donsity cstimation. Advances in neural information processing systems, 33:15088 15099, 2020.
Elliot Creager, David Madras, Jörn-Henrik Jacobsen, Marissa Weis, Kevin Swersky, Toniann Pitassi, and Richard Zemel. Flexibly fair representation learning by disentanglement. In International Conference Machine Learning, pp. 1436 1445, 2019.
Jessica Dai and Sarah M Brown. Label bias, label shift: Fair machine learning with unreliable labels. In Advances in Neural Information Processing Systems Workshop, 2020.
Terrance de Vries, Ishan Misra, Changhan Wang, and Laurens van der Maaten. Does object recognition work for everyone? In IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 52-59, 2019.
Christophe Denis, Romuald Flie, Mohamed Hebiri, and François Hu. Fairness guarantee in multi-class classification. arXiv preprint arXiv:2109.180 13642, 2021.
Frances Ding, Moritz Hardt, John Miller, and Ludwig Schmidt. Retiring Adult: New datasets for fair machine learning. arXio preprint arxiv:200 10800484, 2021.
Mengnan Du, Fan Yang, Na Zou, and Xia Hu. Fairness in deep learning: A computational perspective. IEEE Intelligent. Systems, 2020.
Cynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Richard Zemel. Fairness through awareness. In Innovations in Theoretical Computer Science Conference, pp. 214 11-2226, 2012.
Harrison Edwards and Amos Storkey. Censoring representations with an adversary. In International
Conference on Learning Representations, 2016.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversaria training of neural networks. Journal of Machine Learning Research, 17(1):2096 2030, 2016.
Arthur Gretton, Karsten M Borgwardt, Malte J Rasch, Bernhard Schölkopf, and Alexander Smola. kernel two-sample test.. Journal of Machine Learning Research, 13(1):723 773, 2012.
12

---

Published in Transactions on Machine Learning Research (03/2023)
Yiwen Guo, Chao Zhang, Changshui Zhang, and Yurong Chen. Sparse DNNs with improved adversarial robustness. In Advances in Newral Information. Processing Systems, 2018.
Moritz Hardt, Eric Price, and Nathan Srebro. Equality of opportunity in supervised learning. In Advonces in Neural Information Processing Systems, 2016.
Tatsunori Hashimoto, Megha Srivastava, Hongseok Namkoong, and Percy Liang. Fairness without demo- graphics in repeated loss minimization. In International Conference on Machine Learning, pp. 1929 1938, 2018.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness common corruptions and perturbations. International Conference on Learning Representations, 2019.
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, and Justin Gilmer. The many faces of robustness: A critical analysis of out-of-distribution generalization. arXiv preprint 1rXic:2006.16241, 2020.
Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. In IEEE Conference on Computer Vision and Pattern Recognition, 2021.
Max Hort, Zhenpeng Chen, Jie M Zhang, Federica Sarro, and Mark Harman. Bia mitigation for machine learning classifiers: A comprehensive survey. arXiv preprint arXiv:2207.0000 2022.
Sunhee Hwang, Sungho Park, Dohyung Kim, Mirae Do, and Hyeran Byun. Fairfacegan: Fairness-aweer aware facial imago-to-image translation. In British Machine Vision Conference, 2020.
Faisal Kamiran and Toon Calders. Data preprocessing techniques for classification without discrimination. Knowledge and Information Systems, 33(1):1-33, 2012.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1rXiv:1412.660 12.0990,..2.. 2014.
Diederik P Kingma and Max Welling, Auto-encoding variational bayes. arXiv preprint dr1v:1331.6611, 2013.
Ron Kohavi. Scaling up the accuracy of naive-Bayes classifiers: decision-tree hybrid. International Conference on Knowledge Discovery and Data Mining, pp. 202-207, 1996.
Chun-Liang Li, Wei-Cheng Chang, Yu Cheng, Yiming Yang, and Barnabás Póczos. MMD GAN: Towards deeper understanding of moment matching network. In International Conference on Machine Learning, 2017.
Tian Li, Maziar Sanjabi, Ahmad Beirami, and Virginia Smith. Fair resource allocation in federated learning. In International Conference on Learning Representations, 2019.
Yi Li and Nuno Vasconcelos. REPAIR: Removing representation bias by dataset resampling. In IEEE Conference on Computer Vision and Pottern Recognition, pp. 9572-9581, 2019.
Yujia Li, Kevin Swersky, and Rich Zemel. Generative moment matching networks. In International Conference on Machine Learning, pp. 1718-1727, 2015.
Feng Liu, Wenkai Xu, Jie Lu, Guangquan Zhang, Arthur Gretton, and Danica J Sutherland. Learning deep kernels for non-paramttric two-sample tests. In International conference on machine learning, pp- 6316-6326. PMLR, 2020.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In IEEE International Conference on Computer Vision, 2015.
Ilya Loshchilov and Frank Hutter. SGDR: Stochastic gradient descent with warm restarts. arXiv preprint ar 1ix:1608.000000 2016.
13

---

Published in Transactions on Machine Learning Research (03/2023)
Andrew Lowy, Sina Baharlouci, Rakesh Pavan, Meisam Razaviyayn, and Ahmad Beirami. A stochastic optimization framework for fair risk minimization. Transactions of Machine Learving Research, 2022.
David Madras, Elliot Creager, Toniann Pitassi, and Richard Zemel. Learning adversarially fair and transferable representations. In International Conference Machine Learning, pp. 3384-30 1439-,33,, 2018.
Debmalya Mandal, Samuel Deng, Suman Jana, and Daniel Hsu. Ensuring fairness beyond the training data. In Advances in Neural Information Processing Systems, 2020.
Natalia Martinez, Martin Bertran, and Guillermo Sapiro. Minimax Pareto fairness: A multi objective perspective. In International Conference on Machine Learning, pp. 6755-6764, 2020.
Jérémie Mary, Clément Calauzenes, and Noureddine El Karoui. Fairness-aware learning for continuous attributes and treatments. In International Conference on Machine Learning, pp. 4382 4391. PMLR, 2019.
Ninareh Mchrabi, Muhammad Naveed, Fred Morstatter, and Aram Galstyan. Exacerbating algorithmic bias through fairness attacks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 8930-8938, 2021.
Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, and Shin Ishii. Virtual adversarial training: a regulariza- (ion method for supervised and semi-supervised learning. IEEE Transactions on Pallern Analysis and Machine Intelligence, 41 (8):11979 1993, 2018.
Seyed-Mohsen Mocsavi-Dezfooli, Alhussein Fawzi, Jonathan Uesato, and Pascal Frossard. Robustness via curvature regularization, and vice versa. In JEEE Conference on Computer Vision and Pattern Recognition, pp. 9078-9086, 2019.
Cecilia Muñoz, Megan Smith, and DJ Patil. Big data: A report on algorithmic systems, opportunity, and civil rights. United States Exccutive Office of the President, 2016.
Shruti Nagpal, Maneet Singh, Richa Singh, and Mayank Vatsa. Deep learning for face rocognition: Pride prejudiced? arXiv preprinl ar Xrv: 09111,111,,,,,, 2019.
Junhyun Nam, Hyuntak Cha, Sung-Soo Ahn, Jaeho Lee, and Jinwoo Shin. Learning from failure: De-biasing classifier from biased classifier. In Advances in Neural Information Processing Systems, 2020.
Sungho Park, Sunhee Hwang, Jongkwang Hong, and Hyeran Byun. Fair-VQA: Pairness-aware visual question answering through sensitive attribute prediction. IEEE Access, 5:215091-215099 2020.
John Podesta, Penny Pritzker, Ernest J. Moniz, John Holdren, and Jeffery Zients. Big data: Seizing opportunilies and preserving values. United States Executive Office of the President, 2014.
Flavien Prost, Hai Qian, Qiuwen Chen, Ed H Chi, Jilin Chen, and Alex Beutel. Toward a better trade-off between performance and fairness with kernel-based distribution matching. arxiv preprind Xiv: 1910.1117777 1779, 2019.
Yuxian Qiu, Jingwen Leng, Cong Guo, Quan Chen, Chao Li, Minyi Guo, and Yuhao Zhu. Adversarial defense through network profiling based path extraction. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 4777 4786, 2019.
Novi Quadrianto and Viktoriia Sharmanska. Recycling privileged learning and distribution matching for fairness. In Advances in Neural Information Processing Systems, 2017.
Novi Quadrianto, Viktoriía Sharmanska, and Oliver Thomas. Discovering fair representations in the data domain. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 9227-8236, 2019.
Michael Redmond and Alok Baveja. A data-driven software tool for enabling cooperative information sharing among police departments. European Journal of Operational Research, 141(3):660-66) 678, 2002.
Ashkan Rezaci, Anqi Liu, Omid Memarrast, and Brian Ziebart. Robust fairness under covariate shift. In A AAII pp. 9419 9419-9427, 2021.
14

---

Published in Transactions on Machine Learning Research (03/2023)
Mhd Hasan Sarhan, Nassir Navab, Abouzar Eslami, and Shadi Albarqouni. Fairness by learning orthogonal disentangled representations. In European Conference on Computer Vision, pp. 746-761 2020.
Shreya Shankar, Yoni Halpern, Eric Breck, James Atwood, Jimbo Wilson, and D Sculley. No classification without representation: Assessing geodiversity issues in open data sets for the developing world. In Advances in Neural Information Processing Systems Workshop, 2017.
Harvineet Singh, Rina Singh, Vishwali Mhasawade, and Rumi Chunara. Fairness violations and mitigation under covariate shift. In FAceT, pp. 3-13, 2021.
Nathalie A Smuha. The EU approach to ethics guidelines for trustworthy artificial intelligence. Computer Law Review International, 20(4):97- 106, 2019.
Danica J Sutherland, Hsiao-Yi Yu Tung, Heiko Strathmann, Soumyajit De, Aaditya Ramdas, Alex Smola, and Arthur Gretton. Generative models and model criticism via optimized maximum mean discrepancy. aiv preprint arXie:1611.04458 2016.
Attila Szabó, Hadi Jamali- Rad, and Siva-Datta Mannava. Tilted cross-entropy (tce): Promoting fairness in semantic segmentation. In JEEE Conference On Computer Vision And Pottern Recognition, pp. 2305-2310, 2021.
Rohan Taori, Achal Dave, Vaishaal Shankar, Nicholas Carlini, Benjamin Recht, and Ludwig Schmidt. Measuring robustness to natural distribution shifts in image classification. In Advances in Neural Information Processing Systems, 2020.
Christina Wadsworth, Francesca Vera, and Chris Piech. Achieving fairness through adversarial learning: an application to recidivism prediction. arXiv preprint AAt:11877.0019 2018.
Tianlu Wang, Jieyu Zhao, Mark Yatskar, Kai-Wei Chang, and Vicente Ordonez. Balanced datasets are not enough: Estimating and mitigating gender bias in deep image representations. In IEEE International Conference on Computer Vision, pp. 5310-5319, 2019.
Tsui- Wei Weng, Huan Zhang, Pu Chen, Jinfeng Yi, Dong Su, Yupeng Gao, Cho-Jui Hsich, and Luca Daniel. Evaluating the robustness of neural net works::s: An extreme value theory approach. In International Conference on Learning Representations, 2018.
Benjamin Wilson, Judy Hoffman, and Jamie Morgenstern. Predictive inequity in object detection. arXiv preprint XrXi::1902. 1097, 2019.
Hanshu Yan, Jingfeng Zhang, Gang Niu, Jiashi Feng, Vincent YF Tan, and Masashi Sugiyama. CIFS: Improving adversarial robustness of cnns via channel-wise importance based feature selection. arXiv preprinl arXiv:2102.055 05311, 2021.
Brian Hu Zhang, Blake Lemoine, and Margarel Mitchell. Mitigating unwanted biases with adversarial learning. In AAAI Conference on AI, Ethics, and Society, pp. 335 335-340 340, 2018
Wenbin Zhang, Albert Bifet, Xiangliang Zhang, Jeremy C Weiss, and Wolfgang Nejdl. FARF: A fair and adaptive random forests classifier. In PACKDD, pp. 245-256, 2021.
Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai- Wei Chang, Men also like shopping: Reducing gender bias amplification using corpus-level constraints. arXio preprint arXiv: 1717007777953 2017.
15


Reviews:
[{"review_id": "EE95nv4wcw", "reviewer": "Reviewer_4fit", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n- The problem that is studied in this paper is extremely important to the community.\n\n- The paper constructs distribution shift datasets on three popular fairness datasets. These datasets could be of independent interest to the community.\n\n- The proposed method is simple and based on intuitive principle that the distribution of the curvatures of losses should be similar on both subgroups. The experiments also show effectiveness of the method.\n\nWeaknesses\n\n- The cited literature in the paper is outdated and only covers papers published through 2020. I encourage the authors to update the literature survey and cover the papers published in the past two years as well. You may find this survey helpful to this end:\n\nHort, Max, et al. \"Bias mitigation for machine learning classifiers: A comprehensive survey.\" arXiv preprint arXiv:2207.07068 (2022).\n\n- There are many more recent baselines for EO notion of fairness. For example, see the list below:\n\nMary, Jérémie, et al. \"Fairness-aware learning for continuous attributes and treatments.\" ICML 2019.\n\nProst, Flavien, et al. \"Toward a better trade-off between performance and fairness with kernel-based distribution matching.\" NeurIPS Workshops 2019.\n\nBaharlouei, Sina, et al. \"Rényi fair inference.\" ICLR 2020.\n\nCho, Jaewoong, et al. \"A fair classifier using kernel density estimation.\" NeurIPS 2020.\n\nLowy, Andrew, et al. \"A stochastic optimization framework for fair risk minimization.\" TMLR 2022.\n\n- There is significant inconsistency between the notions of fairness considered in different parts of the paper. It is claimed in Section 1 that fairness is achieved if the accuracy is similar on different subgroups. This is *representation parity* notion of fairness, which is different from (and may even be incompatible with) the *equalized odds* notion that is considered subsequently.\n\n- As it stands, no guarantees have been made about the generalization of fairness mitigation out-of-distribution. It would be great if the authors can better justify the choice of curvature for this problem (e.g., using theoretical arguments on toy problems).\n\n- As far as I understand, the curvature of the loss landscape (which is the metric of interest for robustness in this paper) has been shown to be related to (adversarial) robustness in a small ball. This could be very different from natural distribution shifts with arbitrary changes in the distributions of interest.\n\n- The comparisons as currently reported are not apples-to-apples. There is an inherent tradeoff between *fairness violation* and *accuracy*. Most SOTA methods, including all of the ones considered in this paper are capable of producing a Pareto front in the tradeoff curve between these two metrics by increasing the weight of the loss term associated with fairness. Therefore, the goal should be to produce a tradeoff curve whose Pareto front dominates the Pareto front of a different method. Currently the authors are reporting single points on this front which do not necessarily dominate each other. Hence, unfortunately the comparisons, as it stands, do not reveal much about the effectiveness of the method.\n\n- *method*: The method's effectiveness has not been demonstrated on other fairness mitigation techniques other than adversarial training. As it stands, the applicability of the method seems narrow unless the authors can bring empirical evidence to the contrary.\n\n- *fairness notions*: The method's effectiveness has not been demonstrated on other fairness metrics other than equalized odds. For example, the motivating factor in this paper was based on *representation parity.* Other popular notions are *demographic parity* and *equal opportunity*.\n\n- *scaling*: The method's effectiveness has not been tested beyond binary classification with binary groups.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies a very important problem: How well does fairness mitigation generalize out-of-distribution. The authors build on the heuristic that a smoother loss landscape generalizes better out-of-distribution and add a regularizer to improve the generalization of the adversarial training method for fairness. The authors also curate three datasets to extend existing popular fairness datasets out of distribution. It is shown that for equalized odds notion of fairness, the proposed method results in a better tradeoff curve between fairness violation and accuracy on these binary classification tasks with binary sensitive attributes.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Overall, while the paper is studying a very important problem, I think the contributions of the paper as it stands falls short of a fully fledged paper. I am proposing some axes for improving the paper for it to become a valuable contribution to the literature.\n\n- Given the datasets are an important contribution of this paper, they warrant more scrutiny and description. For example, what if you trained on the new datasets and tested on it as an Oracle method to determine how hard these benchmarks are? Also, would it have been possible to develop other benchmarks based on the well-known distribution shift datasets like WILDS, Imagenet-C, etc?\n\n- SOTA in-processing fairness methods are capable of producing fairness violation / accuracy tradeoff curves rather than a single point. Please turn all results in comparing tradeoff curves, which will be much more informative, and in particular the goal should be to achieve a tradeoff curve that dominates the baselines. What has been reported in Table 1 misses the point completely because it doesn't even consider the performance. Please turn all these comparisons to tradeoff curves.\n\n- Please discuss how the kernel width has been selected for the MMD loss. Also, please discuss how many samples are needed for the MMD loss to be representative of the distribution mismatch between the curvatures.\n\n- Please expand the scope of the effectiveness of the proposed method along the *methods* and *fairness notion* and *scaling* axes. As it stands the experiments only consider adversarial training for equalized odds in binary classification with binary sensitive attributes which makes the applicability of the scope of this paper too narrow.\n\n- Please add a broader impact section that clearly discusses the limitations of the methods used in this paper, given the methods might be used in socially consequential applications.\n\nminor issues and typos:\n\n- The specific subsection on fairness in CV seems totally out of place because I didn't find anything in the rest of the paper to warrant that distinction. Please blend it in with the rest of the subsections. Also, please refer to [Szabó et al., 2021] on fairness in CV.\n\nSzabó, Attila, Hadi Jamali-Rad, and Siva-Datta Mannava. \"Tilted cross-entropy (TCE): Promoting fairness in semantic segmentation.\" In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2305-2310. 2021.\n\n- page 1, 2nd paragraph\n> ... commonly encounter data ...\n\n, -> \"... commonly encounters data ...\"\n\n- page 3, Section 2\n> Compared with individual fairness, group fairness is a more popular setting and thus the focus of our paper\"\n\nThe notion of fairness that is used is determined by the application at hand; some applications require individual fairness while others may warrant group fairness. It is okay to focus on group fairness in this paper, but I would avoid calling it a more popular notion of fairness.\n\n- page 4, Section 3\n> and LAFTR Madras et al. (2018) on Adult Kohavi (1996) dataset\n\nPlease make sure to use \\cite, \\citet, \\citep properly. This is a repeated issue throughout the paper.\n\n- page 6, Section 4.2\n>  DP and EO to be the spatial cases of their\n\nspecial", "broader_impact_concerns": "The paper is missing a broader impact section that clearly discusses the limitations. ", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "rjz6C7aSVk", "reviewer": "Reviewer_cMfQ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros:\n\n1. In the view of significance to the community/society, this paper considers an interesting problem: maintaining fairness in unseen domains. To the best of my knowledge, this is new and novel. More importantly, this problem is worthy to investigate because that distribution shift happens all the time. It is glad to see a study regarding the distribution shift in the fairness field.\n\n2. In the view of technical contributions, this paper studies this novel problem from Loss Curvature, which was found to be useful in robustness by Guo et al. (2018) and Weng et al. (2018). The motivation is clear and interesting. However, we cannot directly use this idea. To this end, this paper considers Curvature Matching which is new to the field.\n\n3. This paper is easy to follow and understand.\n\nCons:\n\n1. Experiment setting should be clarified. I am not sure where only two groups are considered in this paper. If yes, more motivations or explanations should be given.\n\n2. Bandwidth selection of MMD is very important [r1, r2], more discussion should be added.\n\n3. Eq. (3) is not clear. Parameters should be linked to each loss function here.\n\n[r1] Generative models and model criticism via optimized maximum mean discrepancy, ICLR.\n\n[r2] Learning Deep Kernels for Non-Parametric Two-Sample Tests, ICML.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper is novel from problem setting and technical solutions. \n\n1. In the view of significance to the community/society, this paper considers an interesting problem: maintaining fairness in unseen domains. To the best of my knowledge, this is new and novel. More importantly, this problem is worthy to investigate because that distribution shift happens all the time. It is glad to see a study regarding the distribution shift in the fairness field.\n\n2. In the view of technical contributions, this paper studies this novel problem from Loss Curvature, which was found to be useful in robustness by Guo et al. (2018) and Weng et al. (2018). The motivation is clear and interesting. However, we cannot directly use this idea. To this end, this paper considers Curvature Matching which is new to the field.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Cons:\n\n1. Experiment setting should be clarified. I am not sure where only two groups are considered in this paper. If yes, more motivations or explanations should be given.\n\n2. Bandwidth selection of MMD is very important [r1, r2], more discussion should be added.\n\n3. Eq. (3) is not clear. Parameters should be linked to each loss function here.\n\n[r1] Generative models and model criticism via optimized maximum mean discrepancy, ICLR.\n\n[r2] Learning Deep Kernels for Non-Parametric Two-Sample Tests, ICML.", "broader_impact_concerns": "There are no concerns regarding broader impact.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "mITurGzgNo", "reviewer": "Reviewer_QVkJ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The research problem, robust fairness issues under the distribution shift, is quite practical and interesting to study. \n2. The proposed method does not need the target labeled/unlabeled data to have the robustness, which is claimed as one of the important differences from prior works. \n3. The writings of the paper is quite easy to follow and understand.\n\nWeaknesses:\n1. The novelty of the proposed method may not be very large. The proposed method is largely based on prior ideas of curvature matching and adversarial training. \n2. I'm wondering which kind of distribution shift could the proposed method handle? The authors only slightly mentioned that they focus on covariate shift in Sec 2.1. If it is true, why does the covariate shift assumption necessary in this work? How to ensure this assumption in the experiments? I think at least the authors should explain more clearly about their assumption on distribution shift in the paper.  \n3. The proposed method is compared with the baselines that only considers the in-distribution fairness. It makes more sense to compare it with some robust fairness methods under distribution shift, for example, Mandal et al. Ensuring fairness beyond the training data. In NeurIPS 2020.\n4. Format of the references is not consistent. Take NeurIPS papers in the references as an example:\n- Debmalya Mandal, Samuel Deng, Suman Jana, and Daniel Hsu. Ensuring fairness beyond the training data. In NeurIPS, 2020.\n- Yiwen Guo, Chao Zhang, Changshui Zhang, and Yurong Chen. Sparse DNNs with improved adversarial robustness. In Advances in Neural Information Processing Systems, 2018.\n- Flavio P Calmon, Dennis Wei, Bhanukiran Vinzamuri, Karthikeyan Natesan Ramamurthy, and Kush R Varshney. Optimized pre-processing for discrimination prevention. In International Conference on Neural Information Processing Systems, pp. 3995–4004, 2017.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this paper, the authors considered the robust fairness problem under the unseen distribution shift and proposed a method for this problem by matching the loss curvature distributions of the majority and minority groups to let the model have the same generalization ability for the two groups. The method is evaluated on three datasets under the fairness settings. The main contributions of the paper include the proposed fairness robustness method and the related empirical evaluations. ", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. I may suggest the authors explain more about their assumptions on the distribution shift (weakness point 2). \n2. It is better if the proposed method can be compared with baselines that consider distribution shift if possible (weakness point 3).\n3. The authors should carefully check the format of the references to keep them consistent (weakness point 4). \n4. In the ablation study, it is better to check for example the cases when 1) both $\\alpha$ and $\\gamma$ are zeros 2) only $\\alpha=0$ 3) only $\\gamma=0$ , to see the contributions of the different loss terms to the final results. ", "broader_impact_concerns": "The paper does not include a section for describing the broader impact but personally I do not have big concerns regarding the broader impact for this work. \n", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Zka1Hk1mDU", "reviewer": "Reviewer_F7Tn", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n- The research problem is interesting and realistic. The example of the unknown distribution shifts in self-driving is clear.\n- The objective that the minority and majority groups should have similar robustness is reasonable. The technique is sound and easy to follow.\n- The design of the experiments for producing distribution shifts is reasonable and practical.\n\n**Weaknesses**\n- The paper claims that the fairness achieved by existing methods can be easily broken by slight distribution shifts, while only two adversarial-training-based fairness learning methods (LAFTR and AdvDebias) are used to verify and compared.  I think more different and recent fairness learning methods should be compared to support this claim. Besides,  since multiple fairness criteria have been defined as denoted in the paper, it would be better to also report some other fairness criteria in the experiments except  Equalized Odds to support the claim.\n- It would be better to give some experimental evidence about the effectiveness of practical curvature approximation since this effectiveness might influence the learning objective a lot.\n- Can the curvature matching perform as a plug-and-play module added to other recent fairness learning methods?  If so, I think this will arise a lot of interest for some readers.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies the robustness problem of fairness under unknown distribution shifts. It shows some adversarial-training-based fairness learning methods are non-robust to distribution shifts.  To address the problem, by introducing the loss curvature as the measure for robustness,  the paper proposes a curvature matching method to make the robustness of the two data groups similar. Experiments verify the effectiveness of the proposed method compared with two fairness learning methods.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See the above weaknesses. ", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This submission studied the robustness of fairness against distribution shifts. It demonstrated that the fairness achieved by previous methods (based on adversarial training) can no longer be guaranteed under even small distribution shifts. Then it advocated to use the loss curvature as the measure for robustness, and proposed a novel "curvature matching" solution following the idea that the loss curvatures of different data groups should be similar before and after a distribution shift happens. By this way, the proposed method can successfully transfer the fairness obtained on the training distribution to the test distribution.

The submission received 4 reviews from 4 experts in different research areas --- fairness, robustness, distribution shift, and domain adaptation. There were some concerns about the experiment design, and fortunately the authors addressed the issues. In the end, all 4 reviewers voted for acceptance, and thus we should definitely accept the paper for publication! Note that there are "some further minor comments that need attention" from Reviewer 4fit (who like the paper most among the reviewers), so please take them into account (if possible) in the final version.
Decision: Accept (poster)