Equalized Robustness: Towards Sustainable Fairness Under Distributional Shifts
Authors: Haotao Wang
Junyuan Hong
Jiayu Zhou
Zhangyang Wang
Venue: ICLR 2022 Submitted (2022)

Abstract: Increasing concerns have been raised on deep learning fairness in recent years. Existing fairness metrics and algorithms mainly focus on the discrimination of model performance across different groups on in-distribution data. It remains unclear whether the fairness achieved on in-distribution data can be generalized to data with unseen distribution shifts, which are commonly encountered in real-world applications. In this paper, we first propose a new fairness goal, termed Equalized Robustness (ER), to impose fair model robustness against unseen distribution shifts across majority and minority groups. ER measures robustness disparity by the maximum mean discrepancy (MMD) distance between the loss curvature distributions of two groups of data. We show that previous fairness learning algorithms designed for in-distribution fairness fail to meet the new robust fairness goal. We further propose a novel fairness learning algorithm, termed Curvature Matching (CUMA), to simultaneously achieve both traditional in-distribution fairness and our new robust fairness. CUMA efficiently debiases the model robustness by minimizing the MMD distance between loss curvature distributions of two groups. Experiments on three popular datasets show CUMA achieves superior fairness in robustness against distribution shifts, without more sacrifice on either overall accuracies or the in-distribution fairness.

Under review as a conference paper at ICLR 2022
EQUALIZED ROBUSTNESS: TOWARDS SUSTAINABLE FAIRNESS UNDER DISTRIBUTIONAL SHIFTS
Anonymous authors Paper under double-blind review
ABSTRACT
Increasing concerns have been raised on deep learning fairness in recent years. Ex- isting faireess metrics and algorithms mainly focus on the discrimination of model performance across different groups on in-distribution data. It remains unclear whether the faimess achieved on in-distribution data can be generalized to data with unseen distribution shifts, which are commonly encountered in real-world applications. In this paper, we first propose new fairness goal, termed Equal- ized Robustness (ER), to impose fair model robustness against unseen distribution shifts across majority and minority groups. ER measures robustness disparity by the maximum mean discrepancy (MMD) distance between the loss curvature distributions of two groups of data. We show that previous faimess learning al- gorithms designed for in-distribution faimess fail to meet the new robust fairesss goal. We further propose novel faimess learning algorithm, termed Curvature Matching (CUMA), to simultancously achieve both traditional in-distribution fair- ness and our new robust fairness. CUMA debiases the model robustness by min- imizing the MMD distance between loss curvature distributions of two groups. Experiments on three popular datasets show CUMA achieves superior faimess in robustness against distribution shifts, without more sacrifice on either overall accuracies or the in-distribution fairness.
INTRODUCTION
With the wide deployment of deep learning in modern business applications concerning individual lives and privacy, there naturally emerge concerns on machine learning fairness (Podesta et al., 2014; Muñoz et al., 2016; Smuha, 2019). Research efforts on various faimess evaluation metrics and corresponding enforcing methods have been carried out (Edwards & Storkey, 2016; Hardt al., 2016; Du et al., 2020). Specifically, many such metrics require some form of "equalized model performance" across different groups on in-distribution data. Examples include Demographic parity (DP) (Edwards & Storkey, 2016), Equalized Opportunity (EOpp). and Equalized Odds (EO) (Hardt ct al., 2016).
Unfortunately, when deployed for real-world applications, deep models commonly encounter data with unforeseeable distribution shifts (Hendrycks & Dietterich, 2019; Hendrycks et al., 2020; 2021). ) It has been shown that deep learning models can have drastically degraded performance (Hendrycks & Dietterich, 2019; Hendrycks et al., 2020; 2021; Taori et al., 2020) and show unreliable behaviors (Qiu al., 2019; Yan et al., 2021) under unseen distribution shifts. Intuitively speaking, previous fairness learning algorithms aim to optimize the model to a local minimum where data from majority and minority groups have similar average loss values (and thus similar in-distribution performance). However, those algorithms do not take into consideration the the stability or "robustness" of their found fairness-aware minima. Taking object detection in self-driving car for example, it might have been calibrated over high-quality clear images to be "fair" with different skin colors; however such fairness may severely break down when applied to data collected in adverse visual conditions, such as inclement weather, poor lighting, or other digital artifacts. Our experiments also find that previous sfte-of-the-ar fairness algorithms would be jeopardized if distributional shifts are present in test data, illustrated in Figure (b). The above findings beg the following question:
How to ochieve practically sustainable fairness, e.g., even under unseen distribution shifts?

---

Under review as a conference paper at ICLR 2022
((x)
Lx))
(x))
X
Xa
Xa 2/2
Xj x
X2 X
X X
(a) Normal training (Unfair)
(b) Traditional fair training
(c) Robust fair training (In-distribution & robust faimess)
(In-distribution fairness)
Figure Illustrating the achieved fairness of normal training, traditional fair training and our proposed robust fair training algorithms. Horizontal and vertical axes represent input and corre- sponding loss value [(x), respectively. Solid blue curves show the loss landscapes. Circles denote majority data points (z, and z²,), while triangles denote minority data points (z; and xt). Green points (ra and x;)) are in-distribution data while red ones (x', and z are sampled from test sets with distribution shifts. (a) Normal training results in unfair models: minority group has worse perfor- mance (i.e., larger loss values). (b) Traditional fair training algorithms can achieve in-distribution fairness but not in a robust way: small distribution shift can break the fairesss due to loss cur- vature biases across different groups. In fact, such learned fair models can have almost the same large bias as the normally trained models when facing distribution shifts. (c) Our robust fair training algorithm can simultancously achieve faimess both on in in-distribution data and at distribution shifts, by matching both loss values and loss curvatures across different groups.
To answer that, we first propose a new fairness objective, termed Equalized Robustness (ER), which aims to impose "equalized robustness" against unseen distribution shifts across the majority and minority groups, so that the learned fairness can sustain even with test data perturbed. ER explicitly considers a dimension of fairness that is practically significant yet so far largely overlooked. In other words, ER assesses fairness on distibtiisstiibuiionn.. Therefore it works as complement instead of replacement for previous fairness metrics, which focus on assessing the "in-distribution" fairness.
Previous research has shown that model robustness against input perturbation is highly correlated with loss curvature smoothness (Bartlett et al., 2017; Moosavi-Dezfooli et al., 2019; Weng et al., 2018). Our experiments also observed that, the local loss curvature of minority group often larger than that of majority group, leading to the two group's 's robustness discrepancy against distribution shifts. To this end, we propose empirically quantify the robustness discrepancy as the maximum mean discrepancy (MMD) (Gretton et al., 2012) distance between the local model smoothness dis- tributions, for data samples from the majority and minority groups. We experimentally demonstrate that our new metric aligns well with model performance under real-world distribution shifts. On top of that, we further propose a new fair learning algorithm, termed Curvature Matching (CUMA), to simultaneously achieve both traditional in-distribution fairness and ER. CUMA matches the local curvature distribution between data points from the two different groups, as illustrated in Figure (c), by adding eurvature- matihiing regularizer that can be efficiently computed via one-shot power iteration method. Our codes will be released upon acceptance.
Our contributions can be summarized as bellow:
We propose Equalized Robustmess (ER), new faimess objective for machine learning models, to impose equalized model robustness against unforeseeable distributions shifts across majority and minority groups
We further propose a new fairness learning algorithm dubbed Curvature Matching (CUMA), which enforces ER during training by utilizing a one-shot power iteration method.
Experiments show that CUMA achieves much more robust faimess against distribution shifts, without more sacrifice on either overall accuracies or the in-distribution fairness, compared with traditional in-distribution fair leamnng methods.
2

---

Under review as a conference paper at ICLR 2022
2 PRELIMINARIES
2.1 MACHINE LEARNING FAIRNESS
Problem Setting and Metries Machine learning fairness be generally categorized into indi- vidual faimess and group faimess (Du et al., 2020). Individual faimess requires similar inputs to have similar predictions (Dwork et al., 2012). Compared with individual fairness, group fairness is a more popular setting and thus the focus of our paper. Given input data X E RR" with sensitive at- tributes A E {0, } and their corresponding ground truth labels Y {00 1), group fairness requires a learned binary classifier [(;;0) I + {0,1] } parameterized by 0 to give equally accurate predic- tions (denoted as Y == f(X)) on the two groups with A = 0 and A = 1. Multiple fairness criteria have been defined in this context. Demographic parity (DP) (Edwards & Storkey, 2016) requires identical ratio of positive predictions between two groups: P(Ý = 1| = 0) = P(Ý = 1|A = 1). Equalized Odds (EO) (Hardt et al., 2016) requires identical false positive rates (FPRs) and false negative rates (FNRs) between the two groups: p(Ý * Y| 0, Y y) P(Ý * Y|A
1,Y = y). Vy E {0, 1}. Equalized Opportunity (EOpp) (Hardt et al., 2016) requires only equal FNRs between the groups: P(Ý * YIA O,Y = 0) P( * Y = 1, = 0). Based on these fairness criteria, quantified metrics are defined to measure fairness. Specifically, DP, EO and EOpp distances (Madras et al., 2018) are defined as follows:
APP = IP( = 1|A = 0) - P( = 1|4=1111 1)]
(1)
AEO
Σ VEf0,1)
IP
O.Y y) P(Ý # Y|A 1|A=1YY = 4)
(2)
= A80m == P O) -YA== MMD has been previously used to define faimess metric in (Quadrianto & Sharmanska, 2017) de- fines a more general fairness metric using MMD distance, and shows ADP. Ago and A to be spatial cases of their unified metric. All these metrics consider the in-distribution fairness, while our Equalized Generalizibility is the first fairness metric explicitly aware of robust generalization ability on unseen distributions.
P(
(3)
Bias Mitigation Methods Many methods have been proposed to mitigate model bias. Data pre- processing methods such re-weighting (Kamiran & Calders, 2012) and data taatatttaasoomatiinn (Cal- mon et al., 2017) have been used to reduce discrimination before model training. In contrast, Hardt al. (2016) and Zhao et al. (2017) propose post-processing methods calibrate model predic- tions towards a desired fair distribution after model training. Instead of pre- or post-processigg researchers have explored enhance fairness during training. For example, Madras al. (2018) uses a adversarial training technique and shows the learned fair representations can transfer to unseen target tasks. The key technique, adversarial training (Edwards & Storkey, 2016), was designed for feature disentanglement on hidden representations such that sensitive (Edwards & Storkey, 2016) or domain-specific information (Ganin et al., 2016) will removed while keeping other useful infor- mation for the target task. The hidden representations are typically the output of intermediate layers of neural networks (Ganin et al., 2016; Edwards & Storkey, 2016; Madras et al., 2018). Instead, methods, like adversarial debiasing (Zhang et al., 2018) and its simplified version (Wadsworth et al., 2018), directly apply the adversary on the output layer of the classifier, which also promotes the model fairness. Observing the unfairness due ignoring the worst learning risk of specific samples, Hashimoto et al. (2018) proposes to use distributionally robust optimization which provably bounds the worst-case risk over groups. Creager et al. (2019) proposes a flexible fair representation learning framework based on VAË (Kingma & Welling, 2013), that can be easily adapted for different sensi- tive attribute settings during run-time. Sarhan et al. (2020) uses orthogonality constraints as a proxy for independence to disentangles the utility and sensitive representations. Martinez et al. (2020) for- mulates group fairness with multiple sensitive attributes as multi-objective learning problem and proposes simple optimization algorithm to find the Pareto optimality. Another line of research fo- cuses on learning unbiased representations from biased ones (Bahng et al., 2020; Nam et al., 2020). Bahng et al. (2020) proposes novel framework to learn unbiased representations by explicitly en- forcing them to be different from a set of pre-defined biased representations. Nam et al. (2020) observes that data bias can be either benign malicious, and removing malicious bias along can achieve fairness. Li & Vasconcelos (2019) jointly learns a data re- re-sampling weight distribution that penalizes easy samples and network parameters.
3

---

Under review as a conference paper at ICLR 2022
Applications in Computer Vision When many fairness metrics and debiasing algorithms de- signed for general learning problems as aforementioned, there are a line of research and applications computer vision tasks. For instance, Buolamwini et al. (Buo- lamwini & Gebru, 2018) shows current commercial gender-recognition systems have substantial ac- euracy disparities among groups with different genders and skin colors. Wilson et al. (2019) observe that state-of-the-an segmentation models achieve better performance on pedestrians with lighter skin colors. In (Shankar et al., 2017; de Vries al., 2019), it found that the common geographical bias in public image databases can lead to strong performance disparities among images from locales with different income levels. Nagpal et al. (2019) reveal that the focus region of clss-classiiication models depends on people's ages or races, which may explain the source of age- and race-biases of classifiers. On the awareness of the unfairness, many efforts have been devoted to mitigate such biases in computer vision tasks. Wang et al. (2019) shows the effectiveness of adversarial debiasing technique (Zhang ct al., 2018) in fair image classification and activity recognition tasks. Beyond the supervised learning, FairFaceGAN (Hwang et al., 2020) is proposed to prevent undesired sensitive feature translation during image editing. Similar ideas have also been successfully applied to visual question answering (Park et al., 2020).
focusing
2.2 MODEL ROBUSTNESS AND SMOOTINESS
Model generalization ability and robustness has been shown to be highly correlated with model smoothness Giboosvvi-ezfool et al., 2019; Weng et al., 2018). Weng et al. (2018) and Guo et al. (2018) use local Lipschitz constant to estimate model robustness against small perturbations on inputs within a haper-ball. Moosavi-Dezfodd et al. (2019) proposes to improve model robustness by adding a curvature constraint to encourage model smoothness. Miyato et al. (2018) approximates model local smoothness by the spectral norm of Hessian matrix, and improves model robustness against adversarial attacks by regularizing model smoothness.
3 EQUALIZED ROBUSTNESS: A NEW METRIC FOR FAIR GENERALIZATION AND ROBUSTNESS
Consider a binary classifier f(: trained on two groups of data X, and X2 respectively. Our goal is to define metric to measure the gap of model robustness between the two groups Formulating such a metric is highly non-trivial, with difficulties from mainly two aspects.
The first challenge is that we need to ensure fair generalization against multiple unseen distribution shifts that may encounter in real world applications. A trivial solution would be selecting a set of predefined distribution shifts and measuring the average performance gap (c.g.. Agо) against them. However, this approach requires engineering overhead in handcrafting the predefined distribution shifts, and the predefined distribution shifts may not be representative enough to cover all unseen cases. Previous research (Miyato et al., 2018; Moosavi-Dezfooli et al., 2019; Guo et al., 2018; Weng et al., 2018) has shown both theoretically and empirically that deep model robustness scales with its model smoothness. Following (Miyato et al., 2018; Moosavi-Dezfool et al., 2019), we use the spectral norm of Hessian matrix approximate local smoothness as an indicator of model robustness. Specifically, given an input x, the Hessian matrix H Hxx) is defined as the second-ord -rddrr gradient of L(x) with respect to input æ: H(x) = (zr) The approximated local curvature C(x) at point x is thus defined as:
C(x) = o H (x)),
(4)
where (H) is the spectral norm of H: o (H) sup Hell Intuitively, C(x) measures the maximal directional curvature change rate x. Thus, smaller C(x) indicates better local smoothness around x (Miyato et al., 2018; -oosavi-Dezfooli et al., 2019).
For the second difficulty, unlike previous faimess metrics where the target random variable follows a Bernoulli distribution, the local curvature used in ER is a continuous random variable without a simple underlying distribution. The unknown distribution form makes difficult to directly measure the difference between the curvature distributions by a parametric statistic test (e.g.. t-test or KL divergence). To tackle this problem, we utilize maximum mean discrepancy (MMD) (Gretton al.,
Such Y in DP and Y # V in EO and EOpp. (See Section 2.1.)
4

---

Under review as a conference paper at ICLR 2022
2012) do two-sample test C(X1) and C(X2). MMD is distribution distance measure, agnostic to the exact distribution formulation and only based on the mean difference. Formally, our new fairesss metric for equalized robustness is defined as follows:
Our new fairness metric AER
Consider a machine learning model f trained on two groups of data 1 and X2 respectively. Suppose ((Xi) Pi and C(X2) P2, then the model's A ERR is defined as the squared maximum mean Seea--iicrrepacyy (MMD) distance between C(X1) and C(X2):
A FR = (M.)((1.PP))
(5)
MMD widely used measure the distance between two nigh-dimensional distributions in deep learning (Li et al., 2015; 2017; Binkkowsi et al., 2018), The MMD distance between two distribu- tions and Q is defined as
MMD (P, Q) = 1ppe - po pell = Ex],(X,))] 2Ep, elk(, Exll((X))))
(6)
where P. ~ @ and k)- -) is the kernel function. In practice, we use finite samples from P and Q to statistically estimate their MMD distance:
M
N Σ k(),,
MMD
1
k(
2
Σ
Σ
k(y5. Uy') (7)
M
MN
N2
==
where for ~ PI {U ~ U and we use the mixed RBF kernel function k(2,8) v) =
Eoege e ! with hyperparameter = {1, 2. 4. 16). Ablation studies on values are con-
ducted in Section 5.3.
4 CURVATURE MATCHING: FAIR MACHINE LEARNING TOWARDS EQUALIZED ROBUSTNESS
4.1 PRACTICAL. CURVATURE APPROXIMATION
In order to achieve equalized robustness, intuitive solution is add A ER (Eq. (5)) regu- larization term in the loss function during training phase. However, is non-practical to precisely calculate the spectral norm (which is equal to the absolute value of dominant eigenvalue) of Hes- sian matrix in AgR- To solve this problem, we use one-shot power iteration method (PIM) for practical approximation of C(x) during training. First we rewrite C(x) with the following form: C(x) = a (H (z)) || (x)vll where is the dominant eigenvector with the maximal eigenvalue, which can be calculated by power iteration method. practice, to increase training efficiency, we use one-shot power iteration method. Specifically, we estimate the dominant eigenvector by the gradient direction:
sign[g]]l = v. where g v.((x). This is because previous works have observed a large similarity between the dominant eigenvector and the gradient direction (Miyato et al., 2018; Moosavi-Dezoooli et al,, 2019). We further approximate Hessian matrix by finite differ- entiation gradients: Hxx) ze v.L(zztho-VV,tt. where is small constant. As result, the
sign(g)
h
final approximation of curvature smoothness is
C(x) ~e C(x) IV.[(r hō) v.C(x)I
4.2 CURVATURE MATCHING
With the practical curvature approximation, now match the curvature distribution of the two groups by minimizing the MMD distance. Suppose C(Xi) Q and C(X2) Q2. we define the curvature matching loss functions as:
Cam = MMD2 22)
(9)
We add L.m to the traditional adversarially fair training (Ganin et al., 2016; Madras et al., 2018) loss function regularizer, in order attain both in-distribution fairness and fair robustness. As
5

---

Under review as a conference paper at ICLR 2022
Lem
c(-)
The overall framework of CUMA. is the input sample. he is the utility head for the tar- get task. h2 the adversarial head to predict sitive attributes. f. is the shared backbone. C(-) is the curvature estimation function, defined in Eq. (4). Q and 22 are local curvature distri- butions of majority and minority groups, respec- tively. Lem. Ley and Lade are three loss terms as defined in Eq. (9) and (11).
Figure 2:
Q2
f(;00) h, (.,((((;,,)))))
fz(:0,)
he 0,)
Lll
ha (:; 0a)
Ladv
illustrated Figure 2, our model follows the same "two-head" structure as traditional adversarial learning frameworks (Ganin al., 2016; Madras al., 2018), where hy is the utility head for the target task, h, is the adversarial head to predict sensitive attributes, and fa is the shared backbone." Suppose for each sample x;, the sensitive attribute is ag and the corresponding target label is y, then our overall optimization problem can be written as:
min max L = min max(Las OxLadd + 7oom) 0,,0z
(10)
8..6,
where
Laff
Σ
v:), Ladv
Σ
(11)
N
N
((::) )) is the cross-entropy loss function, a and 7 are trade-off hyperparameters, and N is the number of training samples.
EXPERIMENTS
5.1 EXPERIMENTAL SETUP
Datasets and pre-processing Experiments are conducted on three datasets widely used to evalu- machine learning fairness: Communities and Crime (C&C) (Redmond & Baveja, 2002), Adult (Kohavi, 1996), and CelebA (Liu et al., 2015).² C&C dataset has 1,994 samples with neighborhood population statistics, where 1,500 are used for training and the for evaluation. The target task is to predict violent crime per capita, and we use "RacePetBlack" (percentage of black population in the neighborhood) and "FemalePctDiv" (divorce ratio of female in the neighborhood)
sensi- tive attributes. All features in C&C dataset are of continous values in [0, 1]. To fit in the fairness problem setting, we binarilize the target and sensitive attributes with the top-30% largest value as the threshold. We also do data-whitenin on C&C. Adult dataset has 48,842 samples with basic personal information such as education and occupation, where 30,000 are used for training and the rest for evaluation. The target task is predict the person's 's annual income, and (male or female) as the sensitive attribute. The features in Adult dataset are cither continuous (e.g., age) categorical (c.g. sex) values. We use one-hot encoding then concatenate them with the continuous ones. We use data whitening on the concatenated fea- tures. CelebA has over 200,000 images of celebrity faces, with 40 attribute annotations. The target task is to predict the "attractiveness" attribute and the sensitive attributes to protect are "chubby" and "cyeglasses". We randomly select 45, 000 training samples and 5, 000 as testing samples. All images are center-cropped and resized to 128 x 128, and pixel values are scaled to [0, 1].
"gender"
the categorical features and
(=: 0.); 0c), with 0 0, U 'Traditional image classification datasets (c.g., ImageNet) are not directly applicable since they lack fairesss attribute labels.
Thus the binary classifier f(: he(fs(::
As result P 30% and PlY 30%.

---

Under review as a conference paper at ICLR 2022
Results C&C dataset with "RacePetBBack" as the sensitive attribute. The best and second- best metrics are shown in bold and underlined, respectively.
Table 1:
Original Test Set
With Gaussion Noise
With Unitorm Noise
Mothod
Accuracy (t)
ARog!(J
Aso ())
ABn (L)
Asorell>
AED (1)
A0
ARO (L)
In-destrbution Jairness
Robusi Jairness under distribution shifis
Normal
89,05
38.52
63.22
46.16
35.43
60.13
39.51
64.21
AdvDebias
84,79
26.68
39.84
21.77
26.68
39.84
23.65
36.81
LAFTR
85.80
13.32
28.83
16.98
13.53
29.04
16.69
32.20
CUMA
85.20=1170
12.71=1-17
28.17=1.770
7.59=0.19
10.17:0..9
28.69:1.92
12.85=2.98
27.11=0.82
Table 2: Results C&C dataset with "FemalePctDiv" as the sensitive attribute. second- best metrics are shown in bold and underlined, respectively.
The best and
Original Test Set
With Gaussian Noise
With Uniform Noise
Method
Accuracy (1)
Ago ())
Agg (U)
Agoupl) Robust under distribution shiffs
Ago (1)
A
[ (L)
Ago (
In-distribution foirness
Normal 85.60)
17.28
54.74
67.69
17.63
56.41
18.77
$4.60
AdvDehias
83.57
12,80
38.73
37.17
12,80
38.73
11.38
37.15
LAFTR
83.16
11.73
27.83
28.15
11.73
29.30
11.38
30.11
CUMA
83.09:#1.01
8.65##..59
27.57=0.74
27.70:11004
8.7120.88
27.70::1.01
9.63::1.37
28.35::1.73
Models For C&C and Adult datasets, we use two-layer MLPs for fa. h, and hn- For CelebA dataset, we use R backbone, where the first three stages are used as Ss and the last stage (together with the fully connected classification layer) is used as h4- The auxiliary adversarial head hg has the same structure as ht. Detailed model structures are described in Appx. A.
Baseline Methods We compare CUMA with the following state-of-the-an art in in-distribution fairness algorithms. Adversarial debiasing (AdyDebias) (Zhang et al., 2018) is one of the most popular fair training algorithm based on adversarial training (Ganin al., 2016). Madras et al. (2018) proposes similar framework termed Learned Adversarially Fair and Transferable Representations (LAFTR), by replacing the cross-entropy loss used in (Zhang al., 2018) with a group-normalized & loss, which is shown to work better on highly unbalanced datasets. We also include normal (fairness- ignorant) training as a baseline.
We use three different groups of evaluation metrics: the overall accuracy, in-distribution faimess metrics, and robust faimess metrics. We report the overall accuracy on all test samples in the original test sets. To measure in-distribution fairesss, we use AEOp and A on the original test sets. To measure robust fairness under distribution shifts, we use our newly R on the original test sets, and also A gopp and A go on set of pre-defined real-world distribution shifts. We intend to show that AgR calculated on the original test sets aligns well with robust fairness under real-world distribution shifts. See the following paragraph for the details in constructing distributional shifts.
Evaluation Metric
proposed AER
Distributional shifts On Adult and C&C datasets, we construct two distribution shifts by adding random Gaussian and uniform noises. respectively, to the test data. Specifically, following (Madras et al., 2018; Zhang et al., 2018), the categorical features in Adult and C&C datasets are first one-hot encoded and then whitened into float-value vectors, where noises are added. Both types of noises have mean μ = 0 and has standard derivation = 0.03 On CelebA dataset, following (Hendrycks & Dietterich, 2019), we construct two distribution shifts by adding random Gaussian (with mean 0 and standard derivation a 0.08) and impulse noise (with ratio 0.03), respectively. We report the fairness in robustness against other settings distribution shifts in Appx. C.
Implementation Details Unless further specified, we set the loss trade-off parameter a to in all experiments by default. We use Adam optimizer (Kingma & Ba, 2014) with initial learning rate 10-3 and weight decay 10-5, The learning rate is gradually decreased to by cosine annealing learning rate scheduler (Loshchilov & Hutter, 2016). On both Adult and C&C datasets, train for 50 epochs from scratch for all methods. On CelebA dataser, we first normally train a model for 100 epochs, and then finetune it for 20 epochs using CUMA. For fair comparison, epochs on CelebA for all baseline methods. The constant h in Eq. (8) is set to by default. For more implementation details, please check Appx. A.
train for 120
7

---

Under review as a conference paper at ICLR 2022
Table 3: Results Adult dataset with "Sex" the sensitive attribute. The best and second-best
metrics are shown in bold and underlined, respectively.
Original Test Set
With Gaussian Noise
With Uniform Noise
Method
[
Sgo (
AER
Ago (J)
A (
Ago (J)
Accuracy (t)
In-distributton Jairness
Robast jairness under distriiation shifts
Normal
86.11
6.65
15.45
34.25
6,66
15.011
6.87
15.72
AdvDebias
85.17
5.12
5.92
16.78
5.10
5.95
5.77
7.29
LAFTR
85.97
6,28
11.96
25.38
6.22
12,08
6.45
12.06
CUMA
85.30=1073
4.83=0.24
4.77=0.34
5.59-0.28
4.74=0.32
4.81
5.4
6.87=0.31
Table Results on CelebA dataset with "Chubby" as the sensitive attribute. The best and second- best metrics are shown in bold and underlined, respectively.
Original
With Caussian Noise
With Impulse
Method
Accuracy (t)
A
Ago (1)
Ag
I Ago (1)
A
Ago (
In-distribation Jairesss
Robust Jairness under distribution shifis
Normal
91.25
38.45
42.56
59.34
39.16
43,90
39.76
44.51
AdvDebias
90.48
26.41
29.73
42.65
28.95
35.46
29.73
36.48
LAFTR
89.92
26.54
29.10
39.16
27.94
34.60
28.96
35.
CUMA
89.97=0.38
27.19=0.75
30.26::0.95
23.23=+0.39
27.62:+0.85
31.49:1.28
27.97=0.48
31.74=1.11
5.2
MAIN RESULTS
Experimental results on three datasets with different sensitive attributes shown in Tables 3-5, where we compare CUMA with the baseline methods on three different groups of metrics as dis- cussed in Section 5.1. "Normal" means standard training without any fairness regularization. All numbers are shown as percentages. Many intriguing findings can be concluded from the results.
First, we see that previous state-of-the-aar art fairness learning algorithms would be jeopardized if dis- tributional shifts are present in test data. For example, on C&C dataset with "RacePotBlack" as sensitive attribute (Table 1). LAFTR achieves Ago 28.88% 23% on in-distribution test set, while that number is increased to 32.20% 20% on the test set perturbed with uniform random noise. Similarly, for AdvDebias, achieves FO = 29.73% on the original CelebA test set with "chubby" as the sensi- tive attribute (Table 4), while that number is increased to 35.46% and 36.48% on test sets perturbed with Gaussian and impulse noises, respectively.
Second, we see that CUMA achieves the best robust fairness under distribution shifts on all three benchmark datasets with different sensitive attribute settings, while maintaining similar in- distribution fairness and overall accuracy. For example, C&C dataset with "RacePetBlack" as the sensitive attribute (Table 1). CUMA achieves 2.73% and 4.82% less Ago than the second- performer (LAFTR) under distribution shifts by additive Gaussain and uniform noises, respectively. Moreover, for the same experiment setting, although CUMA and LAFTR achieve almost identi- cal in-distribution fairness (the difference between their A EO original test set is within 0.5%), CUMA keeps (and even increases) the fairness under distribution shifts (c.g.. 1.33% smaller Ágo under uniform noises), while the fairness achieved by LAFTR jeopardized under both types of distribution shifts (e.g., 3.37% larger Ago under uniform noises). Similarly, on CelebA dataset with "Chubby" as the sensitive attribute, LAFTR has even slightly better in-distribution fairness than CUMA. However, when the test sets have distribution shifts, the fairness achieved by LAFTR jeopardized (with 5.50% and 6. 6.02% more A EO under Gaussian and uniform noises, respectively), while CUMA keeps its fairness and achieves better fairness under distribution shifts (e.g., 2.50% and 3.17% less Ago compared with LAFTR.).
Third, for all three datasets, the AER calculated on the original test set highly correlates with tradi- tional faimness metrics (c.g., Ao Δgо) calculated on the perturbed test sets: the smaller A ER on the in-distribution test set, the smaller A go on perturbed test sets. This shows that our new metrie AgR aligns well with robust fairness under real-world distribution shifts, and validates the rationality of using it indicator of model robustness discrimination.
More experimental results are shown in Appx. B (trade-off curves between fairness and accuracy) and Appx. C (results other settings of distributional shifts).
8

---

Under review as a conference paper at ICLR 2022
Table 5: Results CelebA dataset with "Fyeglasses" as the sensitive attribute. The best and second- best metrics are shown in bold and underlined, respectively.
Original Test Set
With Gaussian Noise
With Impulse Noise
Method
Agoير() Ago (1)
Ags ())
Agau(e) ())
Ago (L)
A ggo (L)
Ago (U)
Accuracy (1)
In-distribution fairness
Robust fairnsss under distribution shufts
Normal
90.52
36.40
43.96
54.38
35.62
42.91
37.92
AdvDebias
88.65
23.15
32.56
41.06
25,70
36.41
23.92
33.46
LAFTR
89.72
24.90
35.48
42.93
26.12
37.94
24.52
34.10
CUMA
89.10 0.13
24.16=0.44
33.39-40.22
32.56-+0.41
25.76++0.50
34.77+-0.47
22.61+0006
5.3 ABLATION STUDY
Ablation Study on AER In this section, we study how well can the AEn predict the ro- for in the
bust fairness and the sensitivity of AER with respect to S (the sampling mixed RBF kernel function, as described in Section 3). A small a will make the AEr more sensitive the difference between the two sample set, which could be caused by either the true discrepancy of distributions or the different noise introduced by sampling. In contrast, larger ones may under-estimate the discrepancy. Thus, a proper & should in- clude a wide range of a to avoid the domination of either large or small values. In this paper, we choose a geometric sequence with 2 as the base, i.e., S = {1, 2, 4, 8, 16). Furthermore, we compare A gR values under three different sets: S {0.25, 0.5, 1, 4). S2 = (1,2,4, 16) (the default S as defined in Section 3), and 83 {4,8, 16, 32, 64). Results are shown in Table 6. As in Section 5.2, we empirically evaluate the robust fairness by AgO on the test set cor- rupted by uniform noise. From the results, we observe that with all three different S settings, AER aligns well with the model fairness under distribution shifts (Ago under uniform noise).
Table 6: AER values with different mixed RBF kernel scale parameter set S. Results are reported on C&C dataset with "RacePetBlackk" as the sensi- tive attribute. Models are trained by CUMA with different 7 values.
Ang on Original Test Set
Aso Test Set with Uniform Noise
8 Si
S == 82
8= Es
7
12.72
13.52
11.06
31.09 27.02
7
8.56
7.61
4.22
3= 10
8.40
7.24
4.02
26.98
Ablation Study on CUMA In this section, we check the sensitivity of CUMA with respect to its hyper- parameters: the loss trade-off pa- rameters and in Eq. (10) and h in Eq. (8). Results are shown in Ta- ble 7. When fixing 7 1, AER peeks at around a = 1, so we use it the default value. When fixing 1, the best trade-off between overall accuracy and robust fairness achieved at round y = 1, which we use as the default 7. Varying the value of h hardly affects the performance of CUMA.
Table 7: Ablation study results on the loss trade-off parame- ters and y in the CUMA algorithm. Results are reported on C&C dataset with "RacePetBlack" as the sensitive attribute.
0.1
10
0.1
10
0.1
Accuracy
86.93
85.40
83.75
85.19
85.40
85.32
85.40
ARO
59.74
28.35
32,68
38.85
28.35
27.99
29.15
28.35
Age
42.50
7.61
18.56
13.52
7.61
7.24
7.53
7.61
CONCLUSION
In this paper, we first propose a new fairesss goal, termed Equalized Robustness (ER), to impose fair model robustness against unseen distribution shifts across different data groups. We further pro- pose a novel faimess learning algorithm, termed Curvature Matching (CUMA), to simultancously achieve both traditional in-distribution fairness and our new robust fairness. Experiments show CUMA achieves superior faimess in robustness against distribution shifts, without more sacrifice on either overall accuracies or the in-distribution fairness compared with traditional in-distribution fair learning methods. As pioneer work, the new concept of ER proposed in this paper aims to measure a new dimension of fairness that is practically significant yet so far largely overlooked: ER assesses "fut-off disttribution" faimeess while previous metrics focus ER works as a complement instead a replacement for previous fairness metrics. We hope our work can open up more discussions on how to evaluate model fairness in more complete spectrum.
"in-distribution" faimess. Therefor,

---

Under review as a conference paper at ICLR 2022
REFERENCES
Hyojin Bahng, Sanghyuk Chun, Sangdoo Yun, Jaegul Choo, and Seong Joon Oh, Learning de-biased representations with biased representations. In International Conference on Machine Learning, pp. 528-539, 2020.
Peter Bartlett, Dylan J Foster, and Matus Telgarsky. Spectrally-normaiized margin bounds for neural networks. In Advances in Neural Information Processing Systems, 2017.
Mikolaj Binkkowki,, Dougal Sutherland, Michael Arbel, and Arthur Gretton. Demystifying MMD GANs. In International Conference on Learning Representations, 2018.
Joy Buolamwini and Timnit Gebru. Gender shades: Intersectional accuracy disparities in com- mercial gender classification. Conference on Fairness, Accountability and Transparency, pp. 77-91, 2018.
Flavio P Calmon, Dennis Wei, Bhanukiran Vinzamuri, Karthikeyan Natesan Ramamurthy, and Kush R Varshney. Optimized pre-processing for discrimination prevention. In International Conference on Neural Information Processing Systems, pp. 3995-99044 2017.
Elliot Creager, David Madras, Jörn-Henrik Jacobsen, Marissa Weis, Kevin Swersky, Toniann Pitassi, and Richard Zemel. Flexibly fair representation learning by disentanglement. In International Conference on Machine Learning, pp. 1436-1445, 2019.
Terrance de Vries, Ishan Misra, Changhan Wang, and Laurens van der Maaten. Does object recog- nition work for everyone? Workshops, pp. 52-59, 2019.
IEEE Conference on Computer Vision and Puttern Recognition
Mengnan Du, Fan Yang, Na Zou, and Xia Hu. Fairness in deep learning: A computational perspec- tive. IEEE Intelligent Systems, 2020.
Cynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Richard Zemel. Fairness through awareness. In Innovations in Theoretical Computer Science Conference, pp. 214-226, 2012.
Harrison Edwards and Amos Storkey. Censoring representations with an adversary. In International Conference Learning Representations, 2016.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversarial training of neural net- works. Journal of Machine Learning Research, 17(1):2096-200 2016.
Arthur Gretton, Karsten M Borgwardt, Malte J Rasch, Bernhard Schölkopf. and Alexander Smola.
A kernel two-sample test. Journal of Machine Learning Research, 13(1):723-773, 2012.
Yiwen Guo, Chao Zhang, Changshui Zhang, and Yurong Chen. Sparse DNNs with improved adver- sarial robustness. In Advances in Neural Information Processing Systems, 2018.
Moritz Hardt, Eric Price, and Nathan Srebro. Equality of opportunity in supervised learning. In Advances in Neural Information Processing Systems, 2016.
Tatsunori Hashimoto, Megha Srivastava, Hongseok Namkoong, and Percy Liang, Fairness without demographics in repeated loss minimization. In International Conference on Machine Learning, pp. 1929-1938, 2018.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common cor- ruptions and perturbations. International Conference Learning Representations, 2019.
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang. Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, and Justin Gilmer. The many faces of robustness: A critical analysis of duttrf-distributtoon generalization. arXiv preprint arxiv:2006. 16241, 2020.
Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. In IEEE Conference on Computer Vision and Pattern Recognition, 2021.
10

---

Under review as a conference paper at ICLR 2022
Sunhee Hwang, Sungho Park, Dohyung Kim, Mirae Do, and Hyeran Byun. Fairfacegan: Fairness- aware facial image-to to to-image translation. In British Machine Vision Conference, 2020.
Faisal Kamiran and Toon Calders. Data preprocessing techniques for classification without discrim- ination. Knowledge and Information Systems, 33(1):1-33, 2012.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint
1rXiv:1442.6680 2014.
Diederik Kingma and Max Welling. Auto-encoding variational bayes. arXiv:1312.6114, 2013.
arXiv preprint
Ron Kohavi. Scaling up the accuracy Bayes classifiers: A decision-tree hybrid. In Interna- lional Conference on Knowledge Discovery and Data Mining, pp. 202-207, 1996.
Chun-Liang Li, Wei-Cheng Chang, Yu Cheng, Yiming Yang, and Barnabás Póezos. MMD GAN: Towards deeper understanding of moment matching network. In International Conference on Machine Learning. 2017.
Yi Li and Nuno Vasconcelos. REPAIR:
Removing representation bias by dataset resampling. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 9572-9581, 2019.
Yujia Li, Kevin Swersky, and Rich Zemel. Generative moment matching networks. In International Conference on Machine Learning, pp. 1718-1727, 2015.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In IEEE International Conference on Computer Vision, 2015.
lly Loshchilov and Frank Hutter. SGDR: Stochastic gradient descent with warm restarts. arXiv preprint arXiv:1608.03900 2016.
David Madras, Elliot Creager, Toniann Pitassi, and Richard Zemel. Learning adversarially fair and transferable representations. In International Conference on Machine Learning, pp. 3384-3393, 2018.
Natalia Martinez, Martin Bertran, and Guillermo Sapiro. Minimax Pareto fairness: A multi objective perspective. International Conference on Machine Learning, pp. 6755-6764, 2020.
Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, and Shin Ishii. Virtual adversarial training: regularization method for supervised and cemi-supervised learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, (1(8):1979 1993, 2018.
Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, Jonathan Uesato, and Pascal Frossard. Robust- ness via curvature regularization, and vice versa. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 9078-9086, 2019.
Cecilia Muñoz, Megan Smith, and DJ Patil. Big data: A report on algorithmic systems, opportunity, and civil rights. United States Executive Office of the President, 2016.
Shruti Nagpal, Maneet Singh, Richa Singh, and Mayank Vatsa. Deep learning for face recognition: Pride or prejudiced? arXiv preprint 1r0iv:/904.0./ 2019.
Junhyun Nam, Hyuntak Cha, Sung-Soo Ahn, Jaeho Lee, and Jinwoo Shin. Learning from fail- ure: De-biasing classifier from biased classifier. In Advances in Neural Information Processing Systems, 2020.
Sungho Park, Sunhee Hwang, Jongkwang Hong, and Hyeran Byun. Fair-VQA: Fairness-aware visual question answering through sensitive attribute prediction. IEEE Access, 8:215091-215099, 2020.
John Podesta, Penny Pritzker, Ernest Moniz, John Holdren, and Jeffery Zients. Big data: Seizing opportunities and preserving values. United States Executive Office of the President, 2014.

---

Under review as a conference paper at ICLR 2022
Yuxian Qiu, Jingwen Leng, Cong Guo, Quan Chen, Chao Li, Minyi Guo, and Yuhao Zhu, Adver- sarial defense through network profiling based path extraction. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 4777-4786, 2019.
Novi Quadrianto and Viktoriia Sharmanska. Recycling privileged learning and distribution matching for fairness. In Advances in Neural Information Processing Systems, 2017.
Michael Redmond and Alok Baveja. A data-driven software tool for enabling cooperative infor- mation sharing among police departments. European Journal of Operational Research, 141(3): 660-678, 2002.
Mhd Hasan Sarhan, Nassir Navab, Abouzar Eslami, and Shadi Albarqouni. Faireess by learning orthogonal disentangled representations. In European Conference on Computer Vision, pp. 746- 761, 2020.
Shreya Shankar, Yoni Halpern, Eric Breck, James Atwood, Jimbo Wilson, and Sculley. No classi- fication without representation: Assessing geodiversity issues in open data sets for the developing world. Advances in Neural Information Processing Systems Workshop, 2017.
Nathalie Smuha. The EU approach to ethics guidelines for trustworthy artificial intelligence. Computer Law Review International, 20(4):97-106, 2019.
Rohan Taori, Achal Dave, Vaishaal Shankar, Nicholas Carlini, Benjamin Recht, and Ludwig
Schmidt. Measuring robustness to natural distribution shifts in image classification. In Advances in Neural Information Processing Systems, 2020.
Christina Wadsworth, Francesca Vera, and Chris Piech. Achieving fairness through adversarial learning: an application to recidivism prediction. arXiv preprint arXiv:1807.00199 2018.
Tianlu Wang, Jieyu Zhao, Mark Yatskar, Kai-Wei Chang, and Vicente Ordonez. Balanced datasets are not enough: Estimating and mitigating gender bias deep image representations. In IEEE International Conference Computer Vision, pp. 5310-52 (0195319,200 2019.
Tsui-We Wei Weng, Huan Zhang. Pin- Yu Chen, Jinfeng Yi, Dong Su, Yupeng Gao, Cho-Jui Hsieh, and Luca Daniel. Evaluating the robustness of neural networks: An extreme value theory approach. In International Conference on Learning Representations, 2018.
Benjamin Wilson, Judy Hoffman, and Jamie Morgenstern. Predictive inequity in object detection. arXiv preprint arXiv: /097, 2019.
Hanshu Yan, Jingfeng Zhang, Gang Niu, Jiashi Feng, Vincent YF Tan, and Masashi Sugiyama. CIFS: Improving adversarial robustness of cnns via channel-wise importance-base feature se- lection. arXiv preprint 2rXiv:2102.053:1 2021.
Brian Hu Zhang, Blake Lemoine, and Margaret Mitchell. Mitigating unwanted biases with adver- sarial learning. In AAAI Conference on AI, Ethics, and Society. pp. 335-340, 2018.
Jieyu Zhao Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai- Wei Chang. Men also like shopping: Reducing gender bias amplification using corpus-level constraints. arXiv preprint arXiv 1707.09477 2017.
12

---

Under review as a conference paper at ICLR 2022
A MORE IMPLEMENTATION DETAILS
On C&C and Adult datsets, suppose the imput feature dimension d, then the dimensions hidden layers in fa and f, are d + 100 64 and 64 32 2. respectively. ho has identical model structure with hy. For all three sub-networks, ReLU activation function and dropout layer with 0.25 dropout ratio applied between the two fully connected layers. On CelebA dataset, ResNet18 as backbone. The input feature size of h2 and ha 8 x 8 256 (with channel-last layout).
the
B TRADE-OFF CURVES BETWEEN FAIRNESS AND ACCURACY
For CUMA and both baseline methods, we can obtain different trade-off's between fairness and accuracy by setting the loss function weights (e.g., a and 7) to different values. For example, the larger a, the better fairness and the worse aceuracy. Such trade-off curves between fairness and accuracy of different methods are shown in Figure 3. The closer the curve to the top-left comner (i.e., with larger accuracy and smaller Ao)) the better Pareto frontier achieved. As we can see, our method achieves the best Pareto frontiers for both in-distribution fairness (left panel) and robust fairness under distribution shifts (middle and right panel).
CUMA
CUMA
AdvDebias
AdvDebias
LAFTR
LAFTR
Accuracy
0.850
Accuracy
0.850
Accuracy
0.850
0.845
0.845
0.845
0.840
0.840
0.840
CUMA
AdvDebias
0.835
LAFTR
0.30
0.35
0.4C
0.25
0.30
0.35
0.40
0.25
0.30
0.35
distribution Aςо
Σoo under Gaussain Noise
AEO under Uniform Noise
Figure 3: Trade-off curves between faimesss and accuracy of different methods. Results are reported on C&C dataset with "RacePetBBlack" the sensitive attribute.
C RESULTS ON OTHER SETTINGS OF DISTRIBUTIONAL. SHIFTS
Table 8: Results on C&C dataset with "RacePetBlack" as the sensitive attribute. second- best metries are shown in bold and underlined, respectively.
The best and
Original Test Set
With Gaussian Noise
With Uniform Noise
Method
())
Ago (
Agn ())
Agonp(L) Aço (L)
Aoo Ago (.)
Accuracy (1)
br-distributton fairness
Robust Jairness under distribution shifts
Normal 89.05
38.52
63.22
46.16
36.71
61.54
40.22
63.17
AdvDebias
84.79
26.68
39.84
21.77
28.61
37.02
22.84
37.41
LAFTR
85,80
13.32
28.83
16.98
13.96
31.25
16.58
33.42
CUMA
85.40
12.52
28.35
7.61
11.76
27.15
12,80
27.41
Table 9: Results C&C dataset with "FemalePccDivv" as the sensitive attribute. second-bes metrics are shown in bold and underlined, respectively.
The best and
Orginaa Test Set
With Gaussian Noise
With Uniform Noise
Method
A (
Ago (1)
ARR (L)
A (1)
AgO (
())
ARO (
Accuracy (t)
In-distribution fairness
Robust fairness under distribution shifts
Normal 85,60
17.28
54.74
67.69
18.52
57.64
20.25
55.52
AdyDebias
83,57
12.80
38.73
14,90
39.60
12.58
35.26
LAFTR
83,16
11.73
27.83
28.15
13.12
30.21
12.41
31,52
CUMA 83.37
8.90
27.79
23.13
9.12
28.74
9.96
29.23
In this section, we show that the conclusions drawn in Section 5.2 hold under different settings of distributional shifts. Specifically, consider noise setting with mean μ 0 and standard
13

---

Under review as a conference paper at ICLR 2022
derivation σ 0.06 (other than the μ 0 and standard derivation 0.03 evaluated in the main text) for both random Gaussian and uniform noises. The results under these new distributional shifts on C&C dataset shown in Tables 8 and 9.
14


Reviews:
[{"review_id": "on8aBGWWBtO", "reviewer": "Reviewer_ebvu", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Comments: \n* Although the ideal model would indeed be robust against any unforeseen distributional shift, in certain applications (e.g., demographic changes in a student population applying for college), these changes are less dramatic over time (altough not necessarily predefined as described on page 4, Section 3). Furthermore, the decision-maker (e.g., school) might have some external information about the type of distributional shift. How can we get better guarantees in this case? Would the algorithm still be applicable with some modification to take into account this knowledge (and how)?\n* In terms of writing, the paper is well written and precise but remains less accessible to readers who are familiar with the fairness literature but not other related parts (e.g., robustness and smoothness). \n* In particular, this makes it harder for me to evaluate the novelty of the methodological contribution. Does the algorithm bring new ideas (of potential general interest) or just adapt existing robustness techniques to this particular fairness-related application? If it is the latter, what potential difficulties and special conditions one needs to address (that make the problem challenging)?\n* I found the experiments extensive enough. However, a major question I have is how the distributional shift is modeled in those static datasets? On page 7, the paper mentions that Gaussian and uniform noise is used. First, I think that these shifts are not necessarily representable of the true shifts in a population (if not, are there some real cases that justify this assumption?). Second, this approach essentially introduces a “synthetic” distributional shift. It think it would make more sense to perform the same experiments on a dataset with observations over multiple years or after a policy change. The idea is to test how a model trained on old data performs on the most recent ones. I suspect that the changes might be not be very dramatic (since we are talking mostly about demographic changes that are slower).\n* Given the focus on fairness, the choice of the CelebA dataset (with sensitive features “eyeglasses” and “chubby”) is odd and this case study could potentially be omitted.\n\nMinor comments: \n* the bright green color in links is difficult for reading\n* I am not very familiar with the ICLR format, but Section 2 reads like a mix of math preliminaries and related literature. I think it’s better to shorten and remove unnecessary mathematical definitions.\n* Typo: “the the stability”, p.1\n\n", "summary_of_the_paper": "The paper is motivated by a common problem in real world applications of deep models, distributional shifts, which can cause unreliable behavior in the deployed models. In particular,  state-of-the-art fairness algorithms would be affected by such distributional shifts in the test data. This poses the following question studied by the paper: how can one achieve fairness when there exist unseen distributional shifts?\n\nTowards this end, the paper proposes a new objective: Equalized Robustness (ER), that imposes equalized model robustness against distributions shifts across majority and minority. Fruthermore, the paper develops a new algorithm called Curvature Matching (CUMA) that imposes ER during training, and tests it through experiments.\n\n", "summary_of_the_review": "The paper tackles an emerging problem (fairness under distributional shifts) and proposes a new objective and algorithm. While both the metric and the algorithm seem novel, I am not entirely convinced about the methodological contribution overall, given that I am not familiar with the general literature on robustness against distributional shifts. I also have concerns about the value of the experimental evaluation as explained in my comments above.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "_nP9InE1KQN", "reviewer": "Reviewer_xUAG", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "+ The paper is very well-written and well-motivated. The paper provides a clear motivating example in the context of self-driving cars: a model may be fair with respect to skin tone for optimal light conditions, but disparities may arise when there is rain, poor lighting, etc. Figure 1 also provides a nice visual illustration of the key concepts.\n+ The paper is very accessible. The notation is clear and the authors provide intuition about the key concepts.\n\n- Does matching curvatures only guarantee that the groups will have similar level robustness which may be low or high robustness? Or does it specifically encourage high robustness?\n- Would be helpful to be more specific about distributional shifts. It appears these are covariate shifts—i.e. shifts in the distribution of features/predictors, but assuming there is no shift in p(y |x).\n- Do we expect the simulated distribution shifts (Gaussian and uniform noise) to be representative of real-world distribution shifts? Is it possible to empirically analyze a real world distribution shift?\n- It is important to acknowledge the problems of a task like predicting \"attractiveness\".\n\nMinor\n- Would be useful to discuss the choice of kernel for MMD and what that may impact\n- Empirical results should include uncertainty estimates\n- I found the explanation of how hyper parameter S impacts the MMD to be confusing. I did not follow the sentences starting with: “A small sigma will make ….” Does small refer to the value of the spectral norm or to the number of sigmas in the sampled set?\n- Would be helpful to include dimensions in introduction of the setting in section 3. E.g. x is a p-dimensional, Hessian is p x p matrix, etc\n\n", "summary_of_the_paper": "This paper proposes a new fairness metric, Equalized Robustness (ER), to assess model robustness for sensitive subgroups. ER measures the maximum mean discrepancy (MMD) between the loss curvature of the two sensitive groups. The paper proposes using gradient direction to approximate the spectral norm and using finite differences to approximate the Hessian. The MMD is calculated using the RBF kernel.\n\n The paper shows that existing in-distribution fairness promoting methods do not achieve parity with respect to the new metric. A new method, curvature matching (CUMA), is proposed to achieve both in-distribution and robust fairness. CUMA adds the MMD distance as a regularizer to the standard adversarially fair learning approach (which includes a cross entropy loss term and an adversarially fair loss term for in-distribution fairness).\n\nThe paper performs ablation studies for the hyper parameter used in the RBF used to compute MMD.\n\n\n\n\n", "summary_of_the_review": "The ability for fairness properties to generalize is an important problem tackled by this paper. This paper makes a couple novel contributions: 1) a fairness metric that captures robustness to distributional shifts; 2)  learning method to achieve this notion of “out-of-distribution” fairness as well as standard in-distribution fairness. The main technical areas of improvement are 1) include analysis on how well this method works for real-world (as opposed to simulated) distributional shifts and 2) provide error bars to support the empirical claims. I would also encourage the authors to exercise more care/caution around a task like predicting \"attractiveness\".", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "P2qBKmI2mK_", "reviewer": "Reviewer_cgsp", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength\n  - The idea is new and interesting.\n  - The paper is well written.\n\nWeakness\n  - I do not understand the fundamental claim that the similarity of the distributions of local curvatures\n  ensure the fairness robustness. Even there is no explicit definition of fairness robust to distribution shift.\n\n- I agree that local curvatures affects the robustness of a prediction model but I do not understand why and how  local curvatures affect the degree of fairness robustness. \n\n- Numerical results do not  seem to support the main claim. For standard fair learning algorithms (not\nrobust to distribution shift) including AdvDebias and LAFTR, the degree of unfairness (e.g. $\\Delta E_{OPP}$) even decreases as noises are added in the test data.   \n\n- Adding a small noise would not be sufficient to see the effect of robustness. As  (Ensuring Fairness Beyond the Training Data, NIPS2020) is done, the worst case distribution shift could be considered for numerical comparison.", "summary_of_the_paper": "The paper proposed a new penalty term which is expected to make the final prediction model\nrobust to distribution shift of test data. The proposed penalty is based on the\ndistributions of local curvatures of two sensitive groups and enforces these two distributions to be similar.\nThe proposed idea is motivated by robust learning methods where the prediction model is robust to distributional shift. ", "summary_of_the_review": "I do not understand why making the prediction model robust results in fairness robustness.\nFor this reason, I recommend \"rejection\". However, it the authors explain successively why and how the two concepts - prediction robustness and fairness robustness, are related and how the proposed method utilizes this relation. Instead of intuitive explanations, some equations with rigorous definitions\nwould be helpful for understanding the value of the proposed method.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "1QbooNdByc_", "reviewer": "Reviewer_bgyC", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Points in favor\n+ Fairness is a hot and timely topic.\n+ Selection of problem domain, i.e., fairness under distribution shifts. \n\n\nPoints against\n- The authors overstate several of their contributions while the actual contribution is unknown. This paper does not propose a new fairness goal and the first fairness metric under distribution shifts. Rather it extends equalized model performance to distribution shift settings. Similarity, the 'new' learning algorithm is basically a combination of existing techniques such as Hessian matrix and adversarial learning frameworks. The authors should tone down these claims. It is also a stretch to argue that the proposed method does not achieve \"much more robust fairness\" and is insignificant instead. The statistic test is therefore suggested to verify the importance of these results. \n\n- The paper does not discuss and compare proper state of the art. The related work discusses pre-, in-, and post-processing approaches while studies focus on fairness under distribution shifts are not cited and compared. To give concrete examples, fairness drift are considered in \"FARF: A Fair and Adaptive Random Forests Classifier\" as well as reference therein. I urge the authors to compare their method with recent works that consider fairness drift and discuss at least incomparability otherwise.   \n\n- Format violations such as the deceased margins before and after equations.    ", "summary_of_the_paper": "This paper studies fairness under distribution shifts. The model claims to be fair in robustness against distribution shifts with several experiments being conducted. ", "summary_of_the_review": "Limited contribution, inappropriate related work and format violations. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper considers learning a fair classifier under distribution shift. The proposal involves an additional MMD penalty between the model curvatures on the data subgroups defined by the sensitive attribute. Reviewers generally found the problem setting to be well motivated, and the paper to have interesting ideas. Some concerns were raised in the initial set of reviews:

(1) _Relation between local curvatures and fairness robustness_. The concern was that the paper does not make sufficiently clear how similarity of the distributions of local curvatures ensures fairness robustness, and that there is no explicit definition of fairness robust to distribution shift.

(2) _Comparison to related work_. The concern was that works such as FARF as also considering the issue of distribution shift.

(3) _Technical novelty_. The concern was that technical depth of the proposal may be limited, as it builds on existing ideas (e.g., adversarial learning, Hessian to measure curvature).

(4) _Significance of results_. The concern was that the improvements of the proposed method are not significant, statistically and/or practically.

For point (1), the response clarified that the proposal is to ensure that the local curvature (and hence robustness) across data subgroups is similar. The relevant reviewer was still unclear as to whether this ensures what one might intuitively consider "robust fairness". On my review of the paper, I do concur that from the Introduction, and the para preceding Eqn 4, it appears that one natural notion is

$ \sup_{\mathbb{Q} \in \mathcal{U}( \mathbb{P} )} \Delta( \mathbb{Q}( \hat{Y}, Y \mid A = 0 ), \mathbb{Q}( \hat{Y}, Y \mid A = 1 ) ) $

where $\mathbb{P}$ is the observed data distribution, $\mathcal{U}$ is some uncertainty set, and $\Delta$ is some fairness measure (e.g., DP). Assuming this is indeed the ideal, it would be useful to mathematically contrast it to the proposal adopted in the present paper. The para preceding Eqn 4 correctly notes that the above notion would require specifying $\mathcal{U}$. This may be challenging, but an apparently reasonable strategy that follows the distributionally-robust optimization literature would be to use a specific ball around the training distribution (e.g., all distributions with bounded KL divergence against $\mathbb{P}$). Further, it is of interest to ask whether the proposed objective in any way approximate this one; put another way, is there any implicit assumption made as to which class of distributions one is likely to encounter?

Further discussion would also be useful on the following alternative to the objective presented in the paper: rather than match the curvatures for the subgroups, simply minimise their unweighted average. This ought also to ensure robustness under the two different distributions; page 2 hints that this might not work owing to the different scales of these terms (i.e., the minority subgroup being much less robust), but the point does not seem to be discussed very explicitly subsequently.

For point (2), the response noted that FARF is designed for online learning, whereas the present paper involves a single, static training set drawn iid from a single distribution. In the present paper, the drift happens at test time, and the learner has no access to samples from this distribution. The authors argued that FARF can be applied as-is to this setting. From my reading of this and the FARF paper, I agree that while the latter should be cited, it is not clearly applicable to the present setting.

This said, the present paper primarily focusses on the covariate shift setting, for which there have been some relevant recent works; see:

Singh et al., "Fairness Violations and Mitigation under Covariate Shift", FAccT '21.

Rezaei et al., "Robust Fairness under Covariate Shift", AAAI '21.

The former uses tools from joint causal graphs, while the latter assumes access to an unlabelled sample for the target distribution. The present work is certainly different in technical details, but at a minimum it seems prudent to acknowledge that there are relevant works on ensuring fairness outside the observed training distribution, and thus tone down statements such as "As a pioneer work...". There also seems scope to compare against the latter, e.g., to see how valuable having a few samples from the target domain are.

Another work relevant to the spirit of ensuring fairness beyond the observed data is

Mandal et al., "Ensuring Fairness Beyond the Training Data", NeurIPS 2020.

This is in line with the distributionally-robust objective suggested in point (1), where one considers test distributions that can be arbitrary re-weightings of the training distribution.

For point (3), from my reading, the technical content is reasonable. I would however have liked more mathematical discussions on point (1) above, which is important as it is the foundation of the strategy followed.

For point (4), the response asserts their improvements are significant practically and statistically. From my reading, I am inclined to agree with this claim. I would however note that another reviewer raised the question of whether Gaussian and uniform noise are reflective of real-world distribution shifts. I concur with this concern; this part of the paper seems a little disappointing. The response mentioned results on a new setting with more realistic shift, which we suggest is incorporated into future versions of the paper.

Overall, the paper has some interesting ideas for a topical and important problem. At the same time, there is scope for tightening the work per the comments above, particularly on points (1) and (2), and to some extent (4). We believe that addressing these would help properly situate the work, and thus increase its clarity and potential impact. We thus encourage the authors to consider incorporating these for a future submission.
Decision: Reject