Chasing Fairness Under Distribution Shift: A Model Weight Perturbation Approach
Authors: Zhimeng Jiang
Xiaotian Han
Hongye Jin
Guanchu Wang
Rui Chen
Na Zou
Xia Hu
Venue: NeurIPS 2023 poster (2023)
Keywords: Model Weight Perturbation
fairness
distribution shift

Abstract: Fairness in machine learning has attracted increasing attention in recent years. The fairness methods improving algorithmic fairness for in-distribution data may not perform well under distribution shifts. In this paper, we first theoretically demonstrate the inherent connection between distribution shift,  data perturbation, and model weight perturbation.
Subsequently, we analyze the sufficient conditions to guarantee fairness (i.e., low demographic parity) for the target dataset, including fairness for the source dataset, and low prediction difference between the source and target datasets for each sensitive attribute group. Motivated by these sufficient conditions, we propose robust fairness regularization (RFR) by considering the worst case within the model weight perturbation ball for each sensitive attribute group. We evaluate the effectiveness of our proposed RFR algorithm on synthetic and real distribution shifts across various datasets. Experimental results demonstrate that RFR achieves better fairness-accuracy trade-off performance compared with several baselines. The source code is available at \url{https://github.com/zhimengj0326/RFR_NeurIPS23}.

Chasing Fairness Under Distribution Shift: A Model Weight Perturbation Approach
Zhimeng Jiang' Xiaotian Han'*, Hongye Jin', Guanchu Wang?, Rui Chen Na Zou', Xia Hu² Teexs A&M University, "iice University, "Samsung Electronics America
Abstract
Fairness in machine learning has attracted increasing attention in recent years. The fairness methods improving algorithmic fairness for in-distribution data may not perform well under distribution shifts. In this paper, we first theoretically demonstrate the inherent connection between distribution shift, data perturbation, and model weight perturbation. Subsequently, we analyze the sufficient condi- tions to guarantee fairness (i.e., low demographic parity) for the target dataset, including fairness for the source dataset, and low prediction difference between the source and target datasets for each sensitive attribute group. Motivated by these sufficient conditions, we propose robust fairness regularization (RFR) by considering the worst case within the model weight perturbation ball for each sensitive attribute group. We evaluate the effecctiveness of our proposed RFR algorithm on synthetic and real distribution shifts across various datasets. Exper- imental results demonstrate that RFR achieves better fairness- accuracy trade-off performance compared with several baselines. The source code available at https: git..b..... com homm/hhimnggiiii 0326 RFR Ner_NNuurIP233
Introduction
Previous research 11 has shown that classifier trained specific source distribution will perform worse when testing on a different target distribution, due to distribution shift. Recently, many studies have focused investigating the impact of distribution shift machine learning models, where fairness performance degradation is even more significant than that of prediction distribution shift challenges machine learning models in high stake applications, such as criminal justice G healthcare 7 and job marketing [ Thus the transferability of the faimeess performance under distribution shift crucial consideration for real-world applications.
performance [5]. The sensitivity of fairness
To achieve the fairness of the model (already achieved fairness on the source dataset) ) on the target dataset, we first reveal that distribution shift equivalent to model weight perturbation, and then seek to achieve fairness under distribution shift via model weight perturbation. Specifically, i) we reveal the inherent connection between distribution shift, data perturbation, and model weight perturbation. We theoretically demonstrate that any distribution shift can be equivalent data perturbation and model weight perturbation in terms of loss value. In other words, the effect of distribution shift on model training be attributed to data model perturbation. ii) Given the established connection between the distribution shift and model weight perturbation, we next tackle fairness under the distribution shift problem. We first investigate demographic parity relation between source and target datasets. Achieving fair prediction (c.g.. low demographic parity) in the source dataset is insufficient for fair prediction the target dataset. More importantly, the average prediction difference between source and target datasets with the same sensitive attribute also matters for achieving faimesss in target datasets.
"qqual contribution.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

DaIs
Equivvalncee
Weight Fotudttioon
Source
Distriuttion
(a)
The overview distribution shift understanding. The left part demonstrates distribution shift can be transformed as data perturbation, while the right part shows that data perturbation and model weight perturbation are equivalent.
Figure 1:
Motivated by the established equivalence connection, we propose robust faimnsss regularization (RFR) to enforce average prediction robustness over model weight. this way, the well-trained model can tackle the distribution shift problem in terms of demographic parity. Considering the expensive computation for the inner maximization problem in RFR, we accelerate RFR by obtaining the approximate closed-form model weight perturbation using first-order Taylor expansion. In turn, an efficient RFR algorithm, trading the inner maximization problem into two forward and backward propagations for each model update, is proposed to achieve robust fairess under distribution shift. Our contributions are highlighted as follows:
We theoretically reveal the inherent connection between distribution shift, data perturbation, and model weight perturbation. In other words, distribution shift and model perturbation equivalent in terms of loss value for any model architecture and loss function.
We first analyze the sufficient conditions to guarantee fairness transferability under distribution shift. Based the established connection, we propose RFR explicitly pursuing sufficient conditions for robust fairness by considering the worst case of model perturbation.
We evaluate the effectiveness RFR on various real-world datasets with synthetic and real distribution shifts. Experiments results demonstrate that RFR mostly achieve better faimeess- accuracy tradeoff with both synthetic and real distribution shifts.
2
Understanding Distribution Shift
In this section, first provide the notations used in this paper. And then, theoretically understand the relations between distribution shift, data perturbation, and model weight perturbation, as shown in Figure T
2.1 Notations
We consider source dataset Dş and target dataset Dy, defined probability distribution Ps and Pr for samples E and T E T, respectively. Each sample defines values for three random variables: features X with arbitrary domain X. binary sensitive attribute
A {0. 1}, and label Y arbitrary domain V i.c., S = T = X' x A x V. We denote ó as data perturbation on source domain S. where òx (XX and dY) data perturbation of features and labels. Using P()) denote the space of probability distributions over some domain, we denote the space of distributions over examples as P(S). We use ||-I1p Ilp as Lp norm. Let fo (a: be the output of the neural networks parameterized with to approximate the true label We define I(fo(z), y) as the loss function for neural network training. and the optimal model parameters 6* training on source data given by 0" arg min Rs. where
9 (X), X)] Since the target distribution Py maybe be different with source
Rs = E (X)) distribution Ps. the well-trained model fo- (-) trained on soure dataset typically does not perform well in target dataset T.

---

2.2 Distribution Shift is Data Perturbation
Deep neural networks are immensely challenged by data quality or dynamic environments The well-trained deep neural network model may not perform well if existing feature/label noise in training data or distribution shifts among training and target environments. In this subsection, we reveal the inherent equivalence between distribution shift and data perturbation for any neural networks (a()), source distribution Ps. and target distribution Py using optimal transport. We first provide the formal definition of optimal transport as follows:
Definition (Optimal Transport [I]], Considering two distributions with probability distribution Ps and PT. and cost function moving from to t c(s, t), optimal transport between probability distribution Ps and PT is given by
ipf....///f If c(s,s) t)dsdtt
(s,t)) arg inf (EI(Ps.PT)
where the distribution set ((Ps, is the collection of all possible transportation plans and given by I(Ps, Pr) ()),,) > 0, 7s, t)dt Ps(s), (s, PT(().. In other words, the distribution set consists of all possible joint distributions with margin distribution Ps and PT.
Based on the definition of optimal transport, we demonstrate that distribution shift equivalent to data perturbation in the following theorem:
Theorem 2.1. For any two different distributions with probability distribution Ps and PT, adding data perturbation 8 &on source data S can make perturbated source data and target data with the same distribution, where the distribution of data perturbation 8 is given by
P(6) 11
> (s, 50ds.
(2)
Additionally, for any positive p O. data perturbation 8 with minimal power [[[[8]]]] is given by
Eq.12) 2 if optimal transport plan 7 ((5)) ) is calculated based on Eq. 7 with cost function c(s, ()
||8
Proof sketch. Given two different distributions, there many possible perturbations to move one distribution to another. Optimal transport can select the optimal feasible perturbations or transportation plan in terms of specific objectives (e.g., perturbations power). Given the optimal transportation plan, we can derive the distribution of perturbations based on basic probability theory.
Theorem T demonstrates the equivalence between distribution shift and data perturbation, where data perturbation distribution is dependent on optimal transport between source and target distribution. The intuition is that such distribution shift can be achieved via optimal transportation (i.e., data perturbation), Based on Theorem 2.1 we have the following Corollary 2.2 on the equivalent neural network behavior for source and target data:
Corollary 2.2. Given source and target datasets with probability distribution Ps and Pr. there exists data perturbation
that the training loss of any neural network fo(-) target distribution equals that for source distribution with data perturbation 5, i.e.
E (x (X), Y)] = Exx (x))x))))_x)))xx)))))>))))) BxX, [[o(X + 6x(X)), &
(3)
In other words, for the model trained with loss minimization source data, the deteriorated performance on the target dataset stems from the perturbation of features and labels. The proof sketch based Theorem 2 since adding a perturbation on the source dataset can be consistent with the target dataset distribution. Therefore, the conclusion can hold for any loss function that is only dependent on the dataset.
2.3 Data Perturbation Equals Model Weight Perturbation
Although we understand that the distribution shift can be attributed to the data perturbation of features and labels in source data, it is still unclear how to tackle the distribution shift issue. A natural solution
Data perturbation includes perturbation for features 6x (X) labela dy (Y).
3

---

is to adopt adversarial training to force the well-trained model to be robust over data perturbation. However, it is complicated to generate data perturbation on features and labels simultaneously. and many well-developed adversarial training methods are mainly designed for adversarial feature perturbation. Fortunately, we show that model weight perturbation equivalent to data perturbation by Theorem 2.3
Theorem 2.3. Considering the source dataset with distribution Ps. suppose the source dataset is perturbed with data perturbation 6, and the neural network is given by fo(-). for general case, there exists model weight perturbation 20 so that the training loss on perturbed source dataset is the same with that for model weight perturbation A on source distribution:
Exx (1) [[fo(X + 5x(X)), Y + 8Y = E(x.)) (fo+so(X),))) (4)
Proof sketch. The loss under data perturbation and weight perturbation can be analyzed using first-order Tayler expansion. The critical step is to find the condition of the equivalence for the first order term of data perturbation and weight perturbation. Fortunately, the existence of such conditions can be easily proved using linear algebra.
Theorem demonstrates that the training loss on perturbed data distribution (but fixed model weight) equals the training loss on perturbed model weight (but original data distribution). In other words, the training loss fluctuation from data perturbation can equal model weight perturbation. Furthermore, conclude that chasing robust model over data perturbation can be achieved via model weight perturbation, i.e., finding a "flattened" local minimum in terms of the target objective is sufficient for robustness.
3 Methodology
In this section, we first analyze the sufficient condition to achieve robust faimess over distribution shift in terms of demographic parity. Based on the analysis, we propose a simple yet effective robust fairness regularization via explicit adopting group model weight perturbation. Note that the proposed robust fairness regularization involves computation-expensiv maximization problem, we further accelerate model training via first-order Taylor expansion.
3.1 Robust Fairness Analysis
We consider binary sensitive attribute A {0, 1} and demographic parity fairness metric, i.e., the average prediction gap of model fo(.) for different sensitive attribute groups in target dataset ADP, T [
E [fx()]] where To and Ti represent the target datasets for sensitive attribute A = 0 and A = 1, respectively. However, only source dataset is available for neural network training. In other words, even though demographic parity on source dataset ADPs [Es, - Es, [ is low, demographic parity on target dataset PD may not be guaranteed to be low due to distribution shift.
To investigate robust fairness over distribution shift, we try to reveal the connection between demo- graphic parity for source and target datasets. We bound the demographic parity difference for source and target datasets as follows:
(a) DPT S
DPs + [Ta]fo(x)] - En, - Es. [f(())] Es,
DPs + (f) - E [xx())] + [Es, [x)] - E7, [fo
(5)
where inequality (a) and (b) hold due to a a- b < |a bl and ||a- la'- -b < la-aa a'| + 8' respectively, for any a, a', b, b'. In other words, in order to minimize demographic parity for target dataset DP,, the objective of DPs minimization is insufficient. The minimization of prediction difference for source and target datasets given sensitive attribute groups A = 0 and A = 1, defined as
Au Es, [x)] E,, fo( and A Es,, [x)] - Ey, [fel (x)]l, are also beneficial to achieve robust fairness over distribution shift in terms of demographic parity.
The bound in Eq.
5 is tight when both condition (a) DPs < DPT and (b) maximum and minimum
of value set min! E.s [fo)),, To [o(]]]] min [Es, [x)()), ET [x){ both are from source or target distribution. Even though conditions (a) and (b) may not hold for the neural network model, we

---

would like to that our goal is not to obtain a tight bound for demographic parity on target distribution. Instead, we aim to find sufficient conditions to guarantee low demographic parity and such low demographic parity can be achieved in model training without any target distribution information. The proposed upper bound Eq. actually reveals sufficient conditions, which can be achieved by our proposed RFR algorithm.
3.2 Robust Fairness Regularization
Motivated by Section 3.1 and distribution shift understanding in Section
2 we develop robust
fairness regularization to achieve robust fairness over distribution shift. Section 3.1 demonstrates that fair prediction on the target dataset requires fair prediction on the source dataset and low prediction difference between the source and target dataset for each sensitive attribute, i.e., low
Au = Es [x]] Er, (x)]] and A = [,s, [fo(x)] E,, [x()]]]. Based on Theorem 2.3 there
exists co that the following equality holds:
Ao [fo()) E.,, (Sote (x)]l-
(6)
Note that the distribution shift is unknown, consider the worst case for model weight perturbation Eo within Ly-norm perturbation ball with radius as follows:
A = Es. [x)( - E.o Es, [foteo (x)] S max [ss, (foteo (x)] Es,, (fa()]]]
(7)
||oolle
where || Ilp represents Lp norm, and p are hyperparameters. Note that the feasible region of model weight perturbation co symmetric, and the neural network prediction locally linear around parameter 0, we have Ess. (x) - [f,
[fo. (x)] Es, [fox))) due to the
local linearity.In other words, the absolute operation can be removed we consider the maximation
problem in symmetric feasible region since there always non-negative value for the pair perturbation and -Eg Therefore, we can further bound Do as
Ao < max ESo (x)] Es
||roleSS
2 max Es, lSotro (x)] [o(x)] 2 LRFR,SD
(8)
Spooll
Similarly, can bound the prediction difference for source and target distribution with sensitive group A = as follows:
A < max Es, [Jo+a (x)] - E.s, [fo(x)] 2 LRENSS. [| Sp
Therefore, demographic parity for source and target distribution relation is given by DP < DPs CRFR,S, + CRFR.S, we propose robust fairness regualarization (RFR) to achieve robust fairness as
LRFR = CRFRRS, + CRFRRS.
(10)
is worth noting that our proposed RFR agnostic to the training loss function and model architec- tures. Additionally, we follow [12 to accelerate model training using sharpness-aware minimization via trading maximization problem can be simplified as two forward and two backward propagations.
More details on training acceleration are in Appendix
3.3 The Proposed Method
In this subsection, we introduce how to use the proposed RFR to achieve robust fairness, i.c., the fair model (low ADPs) trained the source dataset will also be fair the target dataset (low AD Py). Considering binary sensitive attribute A E {0, 1} and binary classification problem Y E {0, 1). the classification loss is denoted
LeLF Es -Y (1-
(11)
To achieve fairness, consider demographic parity as fairness regularization, i.e.,
LDp [ss. Es, [fo(x)]].
(12)
5

---

Table 1: Performance Comparison with Baselines Synthetic Dataset. (a, 3) control distribution shift intensity, and (0, 1) represents no distribution shift. The best/second results are highlighted in boldface/underline, respectively.
(
Methooh
Ace DEDIEEIIAS
Jop 05)
Adut
Aso
Avc (83
A
AUS-1
Ane (50)
ACS-U
Sen 0S)
10.00
(100.00)
0.66
0.8300.20
75.22-0.63
0.5+0.30
FCR
70 89.00
2000 12205
7
09770022
TOUFO
RIK
78.8610.09
0444000
74.15100.11
1.84
1000
50.05
10.00
0.7110.10
0.0610.11
MLP
52 05.05.0.05
15 15.1010.09
14.3110.009
TT.S5.0..2 25
50.4210.10
1.148.007
11010.07
REG
50 10.6410000
3.74.0
3.20.10
T7.87
0.724...0
0
(.03
ADV
107+0.87
70.91
76.19
0.7240.04
0.71#0.0G
MLP
$2.07.0.005
15 20.20.1011
14.500015 15
T7 77.8810.55 25
3.35. 0.36
10..0.
1.1710.04
11316.04
RBG
60.6210.07
3.7210.05
32.10.05
T8 191.0.000 12
1.00
90.0610.09
0.7010.09
0.610 11
(3.0.80 6.0
ADV
78.97+0.49 40
1.28+0.72 74
100+0.1 10
05.71=0.88
2.20
70 16
L.340.14
1I
PCR
70 02100.0111
10.00:0::05
10.10.05
76
2.97
7h 101.022
10610..1
1.140.
00.15
1:7680155
00:28.00.12
0.7600.44
0.5158.04
Adult
ACS-I
ACS-E
20)) '00)
14-
12-
Adv
2.5-
3.5
3.0
Adv
Reg
1.0
1.2
Reg
Adv
FCR
2.0
FCR
FCR
0.6
RFR
0.2
0.0
76
79
80
81
82
55
60
75
55
70
75
80
Acc
Acc
Acc
(1.5.3.0)
Reg
3.5
Reg
1.2
Reg
Adv
2.5
3.0
Adv
0.8
Adv
FCR
2.0
FCR
FCR
1.5
0.4
0.5
0.0
76
77
50
55
65
75
65
80
Acc
Acc
Acc
Figure 2: The fairness (DP) and prediction (Acc) trade-off performance three datasets with different synthetic distribution shifts. The units for x- and y-axis are percentages (%).
Additionally, we also adopt LRFR as robust fairness regularization. The overall objective is given as follows:
(13) where is hyperparameter to balance the model performance and faimess. LCLF is the loss function for the downstream task. LDP is ensure the demographic parity on the source dataset, and LRF is to ensure the transferability of faimess from the source dataset to the target dataset.
Lau LCLF + X (Lor + LRFR),
In the proposed RFR algorithm, we directly apply approximation to accelerate model training. Note that the optimal model weight perturbation is dependent on the current model weight 0, two forward and backward propagations required to calculate the final gradient.
Experiments
In this section, conduct experiments evaluate the effectiveness of proposed RFR, aiming answer the three research questions. Q1: How effective is RFR to achieve faimess under distribution shift for synthetic distribution shift? Q2: How effective is RFR for real spatial and temporal distribution shift? Q3: How sensitive is RFR to the key hyperparameter X?
4.1 Experimental Setting
In this subsection, we present the experimental setting, including datasets, evaluation metrics, baseline methods, distribution shift generation, and implementation details.
Datasets. adopt the following datasets in our experiments. UCI Adult KK] dataset contains information about 45, 222 individuals with 15 attributes from the 1994 US Census. We consider

---

gender as the sensitive attribute and the task is to predict whether the income of the person is higher than $50k or not. ACS-Income [4 extracted from the American Community Survey (ACS) Public Use Microdata Sample (PUMS) with 3, 236, 107 samples. We choose gender the sensitive attribute. Similar to the task in UCT Adult, the task is to predict whether the individual income above $50k. ACS-Employmeet sensitive attribute. The task is to predict whether an individual employed.
[14] also derives from ACS PUMS, and we also use gender as the
Evaluation Metrics. We use accuracy to evaluate the prediction performance for the downstream task. For fairness metrics, we adopt common-used quantitative group fairness metrics to measure
the prediction bias [T5] [1], ie.. demographic parity A DP PP 1/A 0) P( 1 1)] and equal opportunity Ago = |P(Y 1|A 0, Y = 1) - P(Y = 1|A 1, = 1)), where represents sensitive attribute, Y and represent the ground-truth label and predicted label, respectively.
Baselines. our experiments, we consider vanilla multi-layer perceptron (MLP) and two widely adopted in-processing debiasing methods, including fairness regularization (REG), adversarial debi- asing (ADV), and fair consistency regularization (FCR). MLP directly uses vanilla 3-layer MIP with 50 hidden unit and ReLU activation function [17] minimize cross-entropy loss with source dataset. In the experiments, we adopt the same model architecture for all other methods (i.e., REG and ADV). REG adds fairness-related metric a regularization term in the objective function mitigate the prediction bias [8 19] Specifically, we directly adopt demographic parity as regularization term, and the objective function LCLF ALDp, where LCLF and LDP are defined in Eqs. and (12). ADV (0 employs a two-player game to mitigate bias, in which classification network trained to predict labels based on input features, and an adversarial network takes the output of the classification network as input and aims to identify which sensitive attribute group the sample belongs to. FCR [1] aims minimize and balance consistency loss across groups.
ACS-I
ACS-E
1.2
3.5
Reg
1.0
Reg
2016
3.0
Adv
0.8
Adv
2.0
FCR
0.6
FCR
1.5
RFR
0.4
RFR
2018
0.5
0.0
0.0
60.062.565.067
www.36666667007777777 2.7.072.575.077.
Acc
Acc
1.0
3.5
Reg
Reg
MI
3.0
0.8
Adv
Adv
2.5
0.6
0
2.0
FCR
RFR
0.4
RFR
CA
0.2
0.0
0.0
60.0 62.5
65.0
67.5 70.0 Acc
72.5
75.0
75
Acc
Figure 3: DP and Ace trade-off performance on three real-world datasets with temporal (Top) and spatial (Bottom) distribution shift. The trade-off trade-off performance. The units for x- and y-axis are percentages (%).
close the right bottom corner means better
Synthetic and Real Distribution Shift. synthetic distribution shift, we follow work [22
We adopt synthetic and real distribution shifts. For 23 to generate distribution shift via biased sampling. Specifically, adopt applying principal component analysis (PCA) 24 to retrieve the first principal component C from input attributes. Subsequently, we estimate the mean (C()) and standard deviation (C), and set Gaussian distribution (C)) to randomly sampling of target dataset. As for source dataset, we choose different parameters a and 3 to generate distribution shift using another Gaussian distribution (u(((C) a, =()) for randomly sampling. The source and target datasets constructed by sampling without replacement. For real distribution shift, we adopt the sampling

---

and pre-processing approaches following Folktables 14 to generate spatial and temporal distribution shift via data partition based on different US states and year from 2014 to 2018
Implementation Details. We run the experiments 5 times and report the average performance for each method. We adopt Adam optimizer with 10-3 learning rate and 0.01 weight decay for all models. For baseline ADV, we alternatively train classification and adversarial networks with and 30 epochs, respectively. The hyperparameters for ADV are set {0.0, 1.0, 10.0, 100.0, 500.0). For adding regularization, we adopt the hyperparameters set {0.0, 0.5, 1.0, 10.0, 30.0, 50.03.
4.2 Experimental Results on Synthetic Distribution Shift
In this experiment, we evaluate the effectiveness of our proposed Robust Fairness Regularization (RFR) method with synthetic distribution shifts via biased sampling with different parameters (a, (). We compare the performance of RFR with several other baseline methods, including standard training, adversarial training, and fairness regularization methods. The results of performance value are presented in Table D and the results of fairness-accuracy tradeoff are presented in Figure 2 We have the following observations:
The results in Table I demonstrate that RFR consistently outperforms the basclines in terms of fairness for small distribution shifts, achieving better balance between faimess and accuracy. For example, RFR achieves an improvement of 47.0% on metric A DP and 84.8% on metric A EO compared to the second-best method in the Adult dataset with (1.0, -.0)-syythetic distribution shift. Furthermore, we observed that the outperforms of RFR compared with baselines decrease as the distribution shift intensity increases. The reason is that the approximation RFR involved Taylor expansion over model perturbation and is effective with mild distribution shifts.
The results in Figure I show that RFR achieved a better fairness-accuracy tradcoff compared to the baseline methods for mild distribution shift, We observed that our proposed method achieved a better Pareto frontier compared to the existing methods, and the bias can be mitigated with tiny accuracy drop.
The experimental results show that our method can effectively address the fairness problem under mild distribution shift while maintaining high accuracy, outperforming the existing state-of-the-art baseline methods.
4.3 Experimental Results on Real Distribution Shift
In this experiment, we evaluate the performance of RFR multiple real-world datasets with real distribution shifts. We use ACS dataset in the experiment. The distribution of this dataset varies across different time periods geographic locations, which also causes fairness performance degradation under the distribution shift. The results are presented in Table 2 and the results of the fairness-accures tradeoff are presented in Figure The results show that our method consistently outperforms the baselines across all datasets, achieving a better balance between fairness and accuracy. Specifically, we have the following observations:
Table demonstrates that RFR consistently outperforms the baselines across all datasets in terms fairness- aacness-accuracy tradeoff. This suggests that our method effective in achieving robust faimesss under real temporal and spatial distribution shifts. For example, RFR achieved improvement of 34.9% on metric A DP and 34.4% on metric A EOO compared to the second-best method in ACS-I dataset with temporal distribution shift.
Figure 3 shows that RFR can achieve better fairness-accuracy tradeoff than baselines, i.c., RFR is particularly effective in addressing the faimess problem on the ACS dataset with source/target data varying across different time periods or geographic locations. This is important for many practical applications, where faimess is critical requirement, such as in credit scoring, and loan approval.
The variance for spatial shift on ACS-I is higher than that of temporal shift on methods, which indicates neural networks easily converge to different local minima for spatial distribution shift. The proposed methods can achieve better fairness results for most cases and the tradeoff results clearly demonstrate the effectiveness.
8

---

80
3.5
3.0
ACS-1:2016 2018
75
ACS-I:MI - CA
2.5
ACC
70
2.0
ACS-E:2016 2018
ACS-E:N SIMS-EMII - CA
65
ACS-12016-2218
1.5
ACS-LMMI CA
1.0
60
ACS-E:2016 2018
0.5
55
ACS CA
0.0
8 10
4
10
Hyperpameter A Figure 4: The hyperparameters study for X. The Left subfigure and Right subfigure show the results of accuracy and fairness performance, respectively.
Hyperpameter À
Overall, experimental results on temporal and spatial distribution shifts further support the effectiveness of our proposed method RFR and its potential for practical applications in diverse settings.
Table 2: Performance comparison with baselines on real temporal (the year 2016 to the year 2018) and spatial (Michigan State California State) distribution shift. The best and second-best results are highlighted with hold and underline, respectively.
Real
Methods
ACS-1
ACS-E
Acc (%)
App (%)
Ago (7)
Acc (%)
App (%)
Aro (%)
MLP
77.55:0111
3,202:033 38
3.48=0.11
80.6620.05
1.0750.10
10020.10
REG
77.74+0.62
2.09+-021
2.27=0.24
80.37+0.12
0.77+0008
0.74+0.08
2016 2018
ADV
75.94+0.10
2.41+0-19
2.53=0.55
79.62:0014
1.17#0.11
1.10 14
FCR
76.40+0.45
2.81+0.30
2.96=0.30
79.59++0.38
0.95+0042
0.91+0.34
RFR
77.4940.32
1.36:0.11
1.49=0.17
80.36:0.05
0.61#00111
0.58
MLP
75.62.10.80
5.22.10.86
3.60
79.02
0.73.10.07
0.94.10.05
REG
75.52=0..8
2.88:10.44
2.17=0.22
75.34=1.111
0.42=0009
0.61::011
MI + CA
ADV 73.38
1.04.10.88
0.54
17.46
0.61
0.80.10.13
FCR
74,284+0.35
5.06+40.62
3.67
77.96--0.22
0.44+-0.14
0.67-40.38
RFR
74.63.10.45
1.3510.39
1.30
78.8410.21
0.44..0.09
0.65
4.4 Hyperparameter Study
In this experiment, we investigate the sensitivity of the hyperparameter À in Equation Call LCLF + X(Crr + Lurn) for spatial and temporal distribution shift across different datasets. Specifically, we the hyperparameter as À (0.0, 0.1, 0.5, 1.0, 3.0, 5.0, 10.0). From the results in Figure it is seen that the accuracy and demographic parity are both sensitive to hyperparameter X, which implies the capability accuracy-faimess control. With the increase of X, accuracy decreases while demographic parity also decreases. When À is smaller than 5, accuracy drops slowly while demographic parity drops faster. Such observation represents that an appropriate hyperparameter can mitigate prediction bias while preserving comparable prediction performance.
5
Related Work
In this section, we present two lines of related work, including fairness in machine learning and distribution shift.
Fairness in Machine Learning. Fairness 255
is legal requirement for machine learning
models for various high-stake real-world predictions, such as healtheare 77 32 33 education [34]- 36), and job market [37 38 Achieving fairness, either from data or model perspective [39.42] in machine learning is challenging problem. As such, there has been an increasing interest in both the industrial and research community develop algorithms to mitigate these issues and ensure that machine learning models make fair and unbiased decisions. Extensive efforts led to the development of various techniques and metrics for fairness and proposed various definitions of fairness, such as group fairness 4 40 individual fairness [3)53). and counterfactual fairness (54)56) In this
paper, we focus on group fairness, and the widely used methods to achieve group fairness are fair regularization and adversarial debias method. 1 proposed to add a fairness regularization term

---

to the objective function to achieve group fairness, and [20 proposes to jointly train a classification network and an adversarial network to mitigate the bias for different demographic groups achieve group fairness. Overall, ensuring fairness in machine learning is a critical and ongoing research area that will continue to an important focus for the development of responsible machine learning.
Distribution Shift. Previous work 4 57 reveals that classifier trained on source distribution will perform worse on given target distribution because of the distribution shift, and recently extensive works have explored the influence of distribution shift in model prediction. Moreover, distribution shift can significantly affect the fairness performance of machine learning models. The sensitivity of fairness the distribution shift is notorious for the legal requirement. The degraded performance fair models under a distribution shift would trigger new bias and discrimination issues. There some works
60 that solve fairness under various distribution shifts. For example, [23] explore the fairness under covariate shift, where the inputs change with the in-distribution label. 60 proposes Shifty algorithms to hold fairness guarantees when the dataset in the deployment environment is out-of-de of-distribution of the training datasets (distribution shift). Our work is different from those prior works from model weight perturbation perspective.
X
58
21
59
Conclusion
This paper aims the solve the fairness problem under the distribution shifts from the model weight perturbation perspective. We first establish theoretical connection between distribution shift, data perturbation, and model weight perturbation, which allowed us to conclude that distribution shift and model perturbation are equivalent. We then propose sufficient condition for ensuring fairness transference under distribution shift. To explicitly chase such sufficient conditions, we introduce Robust Fairness Regularization (RFR) method based on the established understanding, to achieve robust fairness. Our experiments on both synthetic and real distribution shifts demonstrate the effectiveness of RFR in achieving a better Tairress-accuracy tradeoff compared to existing baselines. We believe that our understanding of distribution shift is valuable and intriguing to the development of robust machine learning models, and the proposed RFR approach can be of great practical value to build fair and robust machine learning models in real-world applications.
7 Acknowledgement
The authors thank the anonymous reviewers for their helpful comments. The work is in part supported by NSF grants NSF IIS-2224843, I1S-1939716, and 199-990, The views and conclusions contained in this paper are those of the authors and should not be interpreted as representing any funding agencies.
10

---

References
[1] Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. arXiv preprint arXiv: 1.:00.12200 /2261, 2019.
[2] Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: critical analysis of out-of-distribute generalization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8340-8348 2021.
[3] Rohan Taori, Achal Dave, Vaishaal Shankar, Nicholas Carlini, Benjamin Recht, and Ludwig Schmidt. Measuring robustness to natural distribution shifts image classification. Advances in Neural Information Processing Systems, 33:18583-18599 2020.
[4] Steffen Schneider, Evgenia Rusak, Luisa Eck, Oliver Bringmann, Wieland Brendel, and Matthias Bethge. Improving robustness against common corruptions by covariate shift adaptation. Advances Neural Information Processing Systems, 33:1 1539-11551, 2020.
[5] Elisa Celis, Anay Mehrotra, and Nisheeth Vishnoi. Fair classification with adversarial perturbations. Advances in Neural Information Processing Systems, 34:8158-81711 2021.
[6] Songül Tolan, Marius Miron, Emilia Gómez, and Carlos Castillo. Why machine learning may lead to unfairness: Evidence from risk assessment for juvenile justice in catalonia. In Proceedings of the Seventeenth International Conference on Artificial Intelligence and Law, pages 83 83-92, 2019.
[7] Muhammad Aurangzeb Ahmad, Arpit Patel, Carly Eckert, Vikas Kumar, and Ankur Teredesai. Fairness in machine learning for healthcare. Proceedings of the 26th ACM SIGKDD, pages 3529-3530, -530,, 2020.
[8] Jeff Johnson, Donald M Truxillo, Berrin Erdogan, Talya N Bauer, and Leslie Hammer. Percep- tions of overall faimess: are effects on job performance moderated by leader-member exchange? Human Performance, 22(5), 2009.
[9] Ainhize Barrainkua, Paula Gordaliza, Jose A Lozano, and Novi Quadrianto. A survey on preserving fairness guarantees in changing environments. arXiv preprint arXiv:2211.07530, 2022.
[10] Olivia Wiles, Sven Gowal, Florian Stimberg, Sylvestre-Alvise Rebuffi, Ira Ktena, Krishna- murthy Dj Dvijotham, and Ali Taylan Cemgil. A fine-grained analysis on distribution shift. In International Conference on Learning Representations, 2022.
[11] Cédric Villani al. Optimal transport: old and new, volume 338. Springer, 2009.
[12] Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-aware mini- mization for efficiently improving generalization. arXiv preprint arXiv:20110.01412, 2020.
[13] Dheeru Dua and Casey Graff. UCI machine learning repository, 2017.
[14] Frances Ding, Moritz Hardt, John Miller, and Ludwig Schmidt. Retiring adult: New datasets for fair machine learning. NeurlPS, 2021.
[15] Christos Louizos, Kevin Swersky, Yujia Li, Max Welling, and Richard Zemel. The variational fair autoencoder. arXiv preprint 10XX:::5111008.00 2015.
[16] Alex Beutel, Jilin Chen, Zhe Zhao, and Ed Chi. Data decisions and theoretical implications when adversarially learning fair representations. arXiv preprint arXiv:1707.00075 2017.
[17] Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted boltzmann machines. In ICML, 2010.
[18] Toshihiro Kamishima, Shotaro Akaho, Hideki Asoh, and Jun Sakuma. Fairness-awae classifier with prejudice remover regularizer. Joint European conference on machine learning and knowledge discovery in databases. Springer, 2012.

---

[19] Ching Yao Chuang and Youssef Mroueh. Fair mixup: Fairnsss via interpolation. In ICLR, 2020.
[20] Gilles Louppe. Michael Kagan, and Kyle Cranmer. Learning to pivot with adversarial networks. NeurlPS, 30, 2017.
[21] Bang An, Zora Che, Mucong Ding, and Furong Huang. Transferring fairesss under distribution shifts via fair consistency regularization. arXiv preprint arXiv:2206. 12796, 2022.
[22] Arthur Gretton, Alex Smola, Jiayuan Huang, Marcel Schmittfull, Karsten Borgwardt, and Bernhard Schölkopf. Covariate shift by kernel mean matching. Dataset shift in machine learning, 3(4):5, 2009.
[23] Ashkan Rezaci, Anqi Liu, Omid Memarrast, and Brian D Ziebart. Robust faimess under covariate shift. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, 2021.
pages
[24] Hervé Abdi and Lynne J Williams. Principal component analysis. Wiley interdisciplinary reviews: computational statistics, 2(4))433 2010.
[25] Ninareh Mehrabi, Fred Morstatter, Nripsuta Saxena, Kristina Lerman, and Aram Galstyan. A survey on bias and faimess in machine learning. ACM Computing Surveys (CSUR), 14(6):1-35, 2021.
[26] Dana Pessach and Erez Shmueli. A review on fairness in machine learning. ACM Computing Surveys (CSUR), 55(3):1 2022.
[27] Michael Madaio, Lisa Egede, Hariharan Subramonyam, Jennifer Wortman Vaughan, and Hanna Wallach. Assessing the fairness of ai systems: Ai practitioners' processes, challenges, and needs for support. Proceedings of the ACM on Human- Compute Interaction, 6(CSCW1):1-22 2022.
[28] Dexun Li and Pradeep Varakantham. Efficient resource allocation with fairness constraints in restless multi-armed bandits. In James Cussens and Kun Zhang, editors, Uncertainty in Artificial Intelligence, Proceedings of the Thirty-Fighth Conference on Uncertainty in Artifi- cial Intelligence, UAI 2022, 1-5 August 2022, Eindhoven, The Netherlands, volume 180 of Proceedings of Machine Learning Research, pages 1158-1167. PMLR, 2022.
[29] Kirtan Padh, Diego Antognini. Emma Lejal Glaude, Boi Faltings, and Claudiu Musat. Ad- dressing fairness in classification with a model-agnostic multi-objective algorithm. In Cassio P. de Campos, Marloes H. Maathuis, and Erik Quaeghebeur, editors, Proceedings of the Thirty- Seventh Conference on Uncertainty in Artificial Intelligence, UAI 2021, Virtual Event, 27-30 July 2021, volume 161 of Proceedings of Machine Learning Research, pages 800-609. AUAI Press, 2021.
[30] Zhimeng Jiang, Xiaotian Han, Chao Fan, Zirui Liu, Na Zou, Ali Mostafavi, and Xia Hu. Fmp: Toward fair graph message passing against topology bias. arXiv preprint arXiv:2202.044: 0877, 2022.
[31] Xiaotian Han, Zhimeng Jiang, Hongye Jin, Zirui Liu, Na Zou, Qitan Wang, and Xia Hu. Retiring ADP: New distribution-level metrics for demographic parity. Transactions on Machine Learning Research, 2023.
[32] Margrét Vilborg Bjarnadöttir and David Anderson. Machine learning in healtheare: Fairness, issues, and challenges. In Pushing the Boundaries: Frontiers in Impactful OR/OM Research, pages 64-83, INFORMS, 2020.
[33] Thomas Grote and Geoff Keeling. Enabling fairness in healthcare through machine learning. Ethics and Information Technology, 24(3):39, 2022.
[34] Steinar Bøyum. Fairness in education-a normative analysis of aecd policy documents. Journal of Education Policy, 29(6):8668770 2014.
[35] Paolo Brunori, Vito Peragine, and I Laura Serlenga. Fairness in education: The italian university before and after the reform. Economics of Education Review, 31(5):764-777, 2012.
12

---

[36] René F Kizilcec and Hansol Lee. Algorithmic faimess in education. In The ethics of artificial intelligence education, pages 702. Routledge, 2022.
[37] Lily Hu and Yiling Chen. short-term intervention for long-term fairesss in the labor market. In Proceedings of the 2018 World Wide Web Conference, pages 1389-1399 2018.
[38] G Stoney Alder and Joseph Gilbert. Achieving ethics and fairness in hiring: Going beyond the law, Journal of Business Ethics, 68:449-464, 2006.
[39] Daochen Zha, Zaid Pervaiz Bhat, Kwei-Herng Lai, Fan Yang, Zhimeng Jiang, Shaochen Zhong, and Xia Hu. Data-centric artificial intelligence: A survey. arXiv preprint arXiv:2303. 10158, 2023.
[40] Qizhang Feng, Zhimeng Jiang, Ruiquan Li, Yicheng Wang, Na Zou, Jiang Bian, and Xia Hu. Fair graph distillation. 2023.
[41] Chia- Yuan Chang, Yu-Neng Chuang. Zhimeng Jiang, Kwei-Herng Lai, Anxiao Jiang, and Na Zou. Coda: Temporal domain generalization via concept drift simulator. arXiv preprint arXiv:231000150,, 2023.
[42] Chia- Yuan Chang, Yu-Neng Chuang, Kwei-Herng Lai, Xiaotian Han, Xia Hu, and Na Zou. Towards assumption frree bias mitigation. arXiv preprint arXiv:2307.04105, 2023.
[43] Moritz Hardt, Eric Price, and Nati Srebro. Equality of opportunity in supervised learning. NeurIPS, 29, 2016.
[44] Sahil Verma and Julia Rubin. Fairness definitions explained. In 2018 ieeelacm international workshop on software fairness (fairware), pages 1-7. IEEE, 2018.
[45] Peizhao Li. Yifei Wang, Han Zhao, Pengyu Hong. and Hongfu Liu. On dyadic faimess: Exploring and mitigating bias in graph connections. In International Conference on Learning Representations, 2020.
[46] Gaurush Hiranandani, Jatin Mathur, Harikrishna Narasimhan, and Oluwasanmi Koyejo. Quadratic metric elicitation for fairess and beyond. In James Cussens and Kun Zhang, editors, Uncertainty in Artificial Intelligence, Proceedings of the Thirty-Eightl Conference on Uncertainty Artificial Intelligence, UAI 2022, 1-5 August 2022, Eindhoven, The Netherlands, volume 180 of Proceedings of Machine Learning Research, pages 811-821. PMLR, 2022.
[47] Zhimeng Jiang, Xiaotian Han, Chao Fan, Fan Yang, Ali Mostafavi, and Xia Hu. Generalized demographic parity for group fairness. In International Conference on Learning Representations, 2022.
[48] Mikhail Yurochkin, Amanda Bower, and Yuekai Sun. Training individually fair ml models with sensitive subspace robustness. International Conference on Learning Representations, 2020.
[49] Debarghya Mukherjee, Mikhail Yurochkin, Moulinath Banerjee, and Yuekai Sun. Two simple ways to learn individual faimnsss metrics from data. In ICMI., 2020.
[50] Mikhail Yurochkin and Yuekai Sun. Sensei: Sensitive set invariance for enforcing individual fairness. In International Conference on Learning Representations, 2021.
[51] Jian Kang, Jingrui He, Ross Maciejewski. and Hanghang Tong. Inform: Individual fairness on graph mining. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 379-389, 2020.
[52] Debarghya Mukherjee, Felix Petersen, Mikhail Yurochkin, and Yuekai Sun. Domain adaptation meets individual fairness. and they get along. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information P Processing Systems, 2022.
[53] Shantanu Das, Swapnil Dhamal, Ganesh Ghalme, Shweta Jain, and Sujit Gujar. Individual fairness in feature-based pricing for monopoly markets. In James Cussens and Kun Zhang, editors, Uncertainty in Artificial Intelligence, Proceedings of the Tiihth-Eighth Conference on Uncertainty Artificial Intelligence, UAI 2022, 1-5 August 2022, Findhoven, The Netherlands, volume 180 of Proceedings of Machine Learning Research, pages 486-495. PMLR, 2022.
13

---

[54] Matt Kusner, Joshua Loftus, Chris Russell, and Ricardo Silva. Counterfactual faimesss. NeurIPS, 30, 2017.
[55] Chirag Agarwal, Himabindu Lakkaraju, and Marinka Zitnik. Towards a unified framework for fair and stable graph representation learning. In UAI 2021: Uncertainly in Artificial Intelligence, 2021.
[56] Aogi Zuo, Susan Wei, Tongliang Liu, Bo Han, Kun Zhang, and Mingming Gong. Counterfactual fairmess with partially known causal graph. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.
[57] Chia- Yuan Chang, Yu-Neng Chuang, Guanchu Wang, Mengnan Du, and Zou Na. Dispel: Domain generalization via domain-specific liberating. arXiv preprint urXiv:2307.07188 2023.
[58] Jessica Schrouff, Natalie Harris, Oluwasanmi Koyejo, Ibrahim Alabdulmohsin, Eva Schnider, Krista Opsahl-Ong, Alex Brown, Subhrajit Roy, Diana Mincu, Christina Chen, et al. Maintaining fairness across distribution shift: do we have viable solutions for real-world applications? arXiv preprint arXiv:2202.01054, 2022.
[59] Harvineet Singh, Rina Singh, Vishwali Mhasawade, and Rumi Chunara. Faimnsss violations and mitigation under covariate shift. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, pages 3-13, 2021.
[60] Stephen Giguere, Blossom Metevier, Bruno Castro da Silva, Yuriy Brun, Philip Thomas, and Scott Niekum. Fairness guarantees under demographic shift. In International Conference on Learning Representations, 2022.
[61] Jiongli Zhu, Sainyam Galhotra, Nazanin Sabri, and Babak Salimi. Consistent range approxima- tion for fair predictive modeling. Proceedings of the VLDB Endowment, 16 (((11))2225-9999 22988,2938, 2023.
[62] Jiawei Du, Hanshu Yan, Jiashi Feng, Joey Tianyi Zhou, Liangl Zhen, Rick Siow Mong Goh, and Vincent YF Tan. Efficient sharpness-awant minimization for improved training of neural networks. International Conference on Learning Representations, 2022.
[63] Maksym Andriushchenko and Nicolas Flammarion. Towards understanding sharpness-aware minimization. In International Conference on Machine Learning, pages 639-668. PMLR, 2022.
[64] Yuzhe Lu, Zhenlin Wang, Runtian Zhai, Soheil Kolouri, Joseph Campbell, and Katia Sycara. Predicting out-of-distribution error with confidence optimal transport. arXiv preprint arXiv:2302.05011, 2023.
[65] Yaodong Yu, Zitong Yang, Alexander Wei, Yi Ma, and Jacob Steinhardt. Predicting out-of- distribution error with the projection norm. In International Conference on Machine Learning, pages 25721-25746. PMLR, 2022.
14


Reviews:
[{"review_id": "sJb6bBfctK", "reviewer": "Reviewer_646Z", "summary": "The paper studies the problem of training a fair ML model in the presence of distribution shift. The proposed method aims to bound the biases on the target distribution and use it as a regularizer.\n", "questions": "Why does RFR not have the best fairness for spatial distribution shift? Is it because lambda is small?\n", "limitations": "The paper does not discuss the limitations of the approach. I think the loose bound of fairness estimates can adversely affect model accuracy. Authors should discuss the practical implications of the presented approach.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "S1 The problem of distribution shift is quite fundamental.\nS2 The idea of bounding the biases and using it as a regularization term is underexplored.\n", "weaknesses": "W1 The approach of using the worst-case bound as a regularizer has been explored before for selection bias [1]. Please discuss the differences with this approach.\n\nW2 The experiments do not explain the observations in detail. Specifically, the variance is quite high for spatial shift. Are the results staistically significant? Why are the baselines doing worse?\n\nIt would be worth adding an experiment where source and target are exact same. I would expect the baselines to perform much better in this case. More generally, only when the source and target are very different, I expect RFR to perform better than the rest. The current experiments do not seem to clarify any such aspect.\n\nW3 Please show how loose are the estimated bounds. Are the bounds tight?\n\nW4 If possible, consider adding proofs or sketch of theorems. \n[1] https://arxiv.org/pdf/2212.10839.pdf\n\n\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive comments. \n\n**[W1] The approach of using the worst-case bound as a regularizer has been explored before for selection bias [1]. Please discuss the differences with this approach.**\n\n**Ans:** We will add the discussion on the difference with [1]. The difference is two-fold: (1) The problem setting. [1] considers fairness problem with selection bias, where the selection bias is described with available auxiliary information. Our paper mainly focuses on fairness problem under distribution shift without any information on target distribution, which can be but is not necessarily caused by selection bias. (2) Methodology. [1] mainly focus on Consistent Range Approximation (CRA) of a fairness query using probability information in data collection. Our paper mainly focuses on DP metric difference in source and target distribution, and then further derives a model perturbation approach to achieve fairness under distribution. \n\n[1] https://arxiv.org/pdf/2212.10839.pdf\n\n\n**[W2] The experiments do not explain the observations in detail. Specifically, the variance is quite high for spatial shift. Are the results staistically significant? Why are the baselines doing worse? It would be worth adding an experiment where source and target are exact same. I would expect the baselines to perform much better in this case. More generally, only when the source and target are very different, I expect RFR to perform better than the rest. The current experiments do not seem to clarify any such aspect.**\n\n**Ans:** (1) The variance for spatial shift on ACS-I is higher than that of temporal shift on all methods, which indicates neural networks easily converge to different local minima for spatial distribution shift. We run each experiment five times and compare the average results. The proposed methods can achieve better fairness results for most cases and the tradeoff results clearly demonstrate the effectiveness. (2) For the setting without distribution shift, we add the experimental results in Figure. 3 and Table. 1. It is seen that adding regularization plays a strong baseline and achieves comparable performance with our proposed method. We will add the experimental results and more discussion in the revised version. \n\n**[W3] Please show how loose are the estimated bounds. Are the bounds tight?**\n\n**Ans:** The bound in Eq. (5) is tight when both condition (a) $DP_{\\mathcal{S}}\\leq DP_{\\mathcal{T}}$ and (b) maximum and minimum of value set $\\{ \\mathbb{E}_{\\mathcal{S}_0}[f_{\\theta}(\\mathbf{x})], \\mathbb{E}_{\\mathcal{S}_1}[f_{\\theta}(\\mathbf{x})], \\mathbb{E}_{\\mathcal{T}_0}[f_{\\theta}(\\mathbf{x})], \\mathbb{E}_{\\mathcal{T}_1}[f_{\\theta}(\\mathbf{x})] \\}$ both are from source or target distribution. Even though conditions (a) and (b) may not hold for neural network model, we would like to that our goal is not to obtain a tight bound for demographic parity on target distribution. Instead, we aim to find **sufficient conditions** to guarantee low demographic parity and such low demographic parity can be achieved in model training without any target distribution information. The proposed upper bound Eq. (5) actually reveals sufficient conditions, which can be achieved by our proposed RFR algorithm.\n \n**[W4] If possible, consider adding proofs or sketch of theorems.**\n\n**Ans:** We will add the sketch of theorems in the revised version. Due to the main text page limitation, we will leave proofs in Appendices.\n\n**[Q1] Why does RFR not have the best fairness for spatial distribution shift? Is it because lambda is small?**\n\n**Ans:** As shown in the hyperparameter study (Figure. 4), hyperparameter $\\lambda$ is critical to achieving a \"good\" trade-off between accuracy and demographic parity. When selecting a larger hyperparameter $\\lambda$, RFR can achieve better fairness but worse accuracy. In this paper, we consider the worst case to tackle fairness problem under distribution shift without access to target distribution. Therefore, when the (unknown) target distribution closes (deviates) to the worst case, the fairness performance could be (not be) the best compared with several baselines. For spatial distribution shift, our proposed method can still achieve comparable or even better than ADV in terms of tradeoff performance, as shown in Figure 3. \n\n\n**[Q2]  I think the loose bound of fairness estimates can adversely affect model accuracy. The authors should discuss the practical implications of the presented approach.**\n\n**Ans:** The bound of fairness estimation in Eq. (5) is unrelated to accuracy performance. For the algorithm implementation, the accuracy and fairness tradeoff can be controlled by hyperparameter $\\lambda$ as shown in Eq. (13). Please see Appendix G for practical implementation."}, {"review_id": "Ht2lgStUvf", "reviewer": "Reviewer_iSrC", "summary": "In this work, the authors seek to learn neural networks that are fair not just w.r.t. the observed distribution of individuals, but also under shifts to this distribution. In order to achieve this, they propose robust fairness regularization (RFR) which uses adversarial weight-space perturbations at train time to create more robustly fair models (w.r.t. distribution shifts). RFR being a reliable regularizer for distribution shifts relies on a correspondence between distribution and point-wise data perturbations as well as a correspondence between point-wise data perturbations and model weight perturbations. Finally, the paper proposes that for each group, the difference between the predictive performances on the source and target distribution should be the same. By enforcing this over all groups, the authors have the RFR loss that they propose.", "questions": "Please address my key point on how one picks $\\rho$ in a principled way. ", "limitations": "See weaknesses.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 4, "contribution": 2, "strengths": "The paper focuses on a very important topic for trustworthy ML. \n\nThe paper is well-written and the notation is clear and easy to follow.\n\nThe experimental results that are reported are strong and at the core evaluate the method well, though I have some suggestions below in the weaknesses for improvements here. ", "weaknesses": "Major weakness: The major weakness that I see with this paper is the underlying theoretical motivation. In particular Theorem 2.3 appears to have a critical flaw that is never addressed in the paper (as far as I can tell). I elaborate further at the bottom of this section, but the key point is this, though there exists weight perturbation that give us the same loss as any arbitrary input perturbations, there is no given relationship between the magnitude of the perturbations in either domain. Therefore, even given that you know the Wasserstein distance is bounded by some value $\\epsilon$ there is no way to select the value of $\\rho$ before you have observed the target distribution, rendering the method difficult to use in practice. Moreover, I find it troubling that this is (1) not addressed in the paper, and (2) the authors do not explain how they select $\\rho$ either theoretically or experimentally. It is obvious to me that given access to the target distribution, one can optimize $\\rho$ to give good results, but this is the same as training on your test distribution and claiming good results. Without a clear and theoretically grounded argument for selecting $\\rho$ and an open source implementation showing that this works well across datasets, I do not think I can vote to accept this paper. \n\n\nAnother weakness I see, is that the authors do not compare their weight perturbation method to the corresponding input perturbation method (as far as I can see). The ADV method used in the experiments refers to methods like LAFTR which are adversarial, but not in the same spirit as crafting worst-case perturbations.\n\n\nElaborated major weakness: The  first claim (Corollary 2.2)  is that there exists a perturbation map that makes the classification loss between source, $P_S$, and target $P_T$ the same. This statement seems clear and intuitive; however, we do not, a priori know how the magnitude by which we will have to perturb each data point. Let us assume, for sake of argument, that we know a priori that the Wasserstein distance between $P_S$ and $P_T$ is bounded by a real value $\\epsilon > 0$. Then for an empirical distribution consisting of $n$ atoms, we know that we won’t perturb any given point by more than $n\\epsilon$, so this gives us some practical guidance on how to set the data perturbation value, assuming we know a bound on the Wasserstein distance. The second claim made by the paper, Theorem 2.3, is that there exists a weight perturbation such that the model has the same loss on the source distribution (after its weights have been perturbed) as it does on the target distribution. However, though we may have a bound on how large the input perturbation may be, there is absolutely no correspondence between the magnitude of input perturbations, $n\\epsilon$ and the magnitude we should set for the weight space perturbations $\\rho$. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive comments. \n\n**[Q1] The major weakness that I see with this paper is the underlying theoretical motivation. Theorem 2.3 appears to have a critical flaw. (1) There is no given relationship between the magnitude of the perturbations in either domain and thus there is no way to select the value before you have observed the target distribution. (2) The authors do not explain how they select either theoretically or experimentally.**\n\n**Ans:** We thank the reviewer for this constructive comment. \n\n(1) We agree with the reviewer that there is no given relation between the magnitude of the perturbations $\\rho$ in either domain. However, we would like to respectfully clarify that it is not necessary to consider such a relation since there is no accessible information on target distribution. As shown in Table 1 of our paper, a heuristic $\\rho=0.0005$ contributes to a better accuracy-fairness tradeoff than baseline methods. \n\n(2) We would like to respectfully clarify that it is unnecessary to theoretically or experimentally optimize $\\rho$. According to the result of the hyperparameter study on $\\rho$ given $\\lambda=0.1$ in Figure~4 of our paper, the accuracy and DP are stable for $1\\times 10^{-5} \\leq \\rho\\leq1\\times 10^{-3}$, which indicates the performance is rarely affected by the selection of $\\rho$. A heuristic $\\rho=0.0005$ in our experiment is sufficient for mitigating the unfairness under distribution shift. \n\nAs Reviewer #3kzb noted, *more discussion of the hyper-parameter $\\rho$ is not necessary for this work*, given the extensive experimentation in this paper. This indicates our analysis and experiment sufficiently demonstrate the effectiveness of RFR.\n\n\n**[Q2] The authors do not compare their weight perturbation method to the corresponding input perturbation method (as far as I can see). The ADV method used in the experiments refers to methods like LAFTR which are adversarial, but not in the same spirit as crafting worst-case perturbations.**\n\n**Ans:** Thanks for the insightful comments. We agree that ADV is an adversarial method like LAFTR and different from the input perturbation method. However, the input perturbation method is complicated since the input perturbation is dependent on input samples. Meanwhile, the proposed model weight perturbation is simple and the weight perturbation for all samples is the same. Considering the worst-case scenario, the input adversarial method is similar to adversarial attack while the goal of the attack is different. The method of perturbation/attack is highly related to feature type (numerical, categorial, or mixed). We leave the investigation of the input perturbation method for future work."}, {"review_id": "z3Uhes9YTO", "reviewer": "Reviewer_Viye", "summary": "theoretical proof: prove that distribution shifting can be written as data perturbation, and the model trained with the perturbed data can be written as the perturbed model trained with original source data.\nmethodology: motivated by the connection between distribution shifting and data weight perturbation, the author proposed RFR(robust fairness regularization), which is an upper bound of the difference of demographic parity between source and target.", "questions": "- I cannot understand the proof of theorem 2.3, lines 499-501; is that sufficient? Please give more details.\n- I think it would be better if the graph of EO was also included in appendix.\n", "limitations": "refer weakness and questions.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "well written; easy to follow. \nwell-structured and coherent framework that facilitates easy comprehension.", "weaknesses": "- I think that the objective function which the author suggested did not reflect the true effect of $L_{RFR}$. Please add the experiments without $L_{DP}$\n- It would be better if the notation $\\delta_X, \\delta_Y$ were changed. The author writes $\\delta_X, \\delta_Y$ to denote the perturbation for feature and label respectively, however, in equation (3), it seems like they depend on each random variable X and Y. It can make confusion for readers.\n- some typos: for example, line 495(appendix), $E_{\\delta_Y}[\\delta_X]$ → $E_{\\delta_X}[\\delta_X]$, line 152 $\\|\\cdot\\|_2$ →$\\| \\cdot\\|_p$", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive comments.\n\n**[W1] I think that the objective function which the author suggested did not reflect the true effect of $L_{RFR}$. Please add the experiments without $L_{DP}$.**\n\n**Ans:** We thank the reviewer for this constructive comment. As required by the reviewer, we add ablation studies to reflect the true effort of $L_{RFR}$ in Figure 2 of the one-page rebuttal. \n\n**[W2] It would be better if the notation $\\delta_X$, $\\delta_Y$ were changed. The author writes $\\delta_X$, $\\delta_Y$ to denote the perturbation for feature and label respectively, however, in equation (3), it seems like they depend on each random variable X and Y. It can make confusion for readers.**\n \n**Ans:** We thank the reviewer for this thoughtful comment. As required by the reviewer, we will revise the notation $\\delta_X$, $\\delta_Y$ into $\\delta_X(X)$, $\\delta_Y(Y)$ for the first appearance and explicitly mentioned using $\\delta_X$, $\\delta_Y$ for simplicity for the remaining part of the paper.\n\n**[W3] Some typos: for example, line 495(appendix), \n $E_{\\delta_Y}[\\delta_X] \\rightarrow E_{\\delta_{X}}[\\delta_{X}]$, line 152 $|\\cdot|_2 \\rightarrow |\\cdot|_p$.**\n \n **Ans:** We will revise these typos in the revised version.\n\n**[Q1] I cannot understand the proof of theorem 2.3, lines 499-501; is that sufficient? Please give more details.**\n\n**Ans:** Eqs. (18) and (20) provide the loss value analysis for data and model weight perturbation. When chasing Eq. (18) equals Eq. (20), the model weight perturbation is treated as multivariate but with only one equation. It is easy to see that there exists a model weight perturbation solution to guarantee that Eq. (18) equals Eq. (20).\n\n**[Q2] I think it would be better if the graph of EO was also included in appendix.**\n\n**Ans:** We have added the tradeoff results between EO and accuracy in Figure 4 of the one-page rebuttal. We will add the results in Appendix for the revised version."}, {"review_id": "MGkpBkY6v8", "reviewer": "Reviewer_bBot", "summary": "The paper is in the context of fairness measures under distribution shift. \nThere is first a theoretical connection between distribution shift, data perturbation and model perturbations.\nThe proposed method Robust Fairness Regularizatin (RFR), proposes a regularization using 3 terms L_CLF-- classifier loss function, L_DP ensuring DP on source data and L_RFR= L_RFR_S0 +  L_RFR_S1 which is the fairness transferability from training set.\nThe experiments are in  synthetic (PCA transformations) and real data (Adult and Folk)", "questions": "The work states that \"any distribution shift can be equivalent to data perturbations.\" Does this approach (Sec2.2) considers pure concept shift? When $P(X_S)=P(X_T)$ and $P(Y_S)=P(Y_T)$, but the only thing that changes is the relationship between covariate and target? $X\\Rightarrow Y$.\n\nIs the method strictly limited to NN? If so (since its tabular data) it may be worth stating it earlier and clearer in the paper. \nI wonder if the theoretical results from section 2 can be aligned with the previous empirical findings of https://arxiv.org/pdf/2302.05018.pdf ", "limitations": "Fairness is just one metric. I wonder if this limitation should be stated early on the paper or even affect the title. \n\nThe work does not acknowledge the difference between algorithmic fairness and fairness in broader socio-technical systems.\n\n", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper is extremely well-written and clear. Enjoyable lecture.\n\nThe method proposed is simple. \n\nThe work seems extensive and thorough. \n\nExperiments are well-redacted and easy to understand.\n\nThe work is one of the few (to the best of my knowledge) that are tackling distribution shift and fairness, without strong assumptions like causal graph availability.", "weaknesses": "The method is designed/tested in Neural Network architectures in tabular datasets. In tabular data algorithms like gradient boosting decision trees achieve SOTA[1] and are restricted to Linear Regression in regulated environments. \n\n\n\nThe paper calls fairness to just one metric --Demographic Parity-- Fairness is not just one metric. L_RFR is not performance based but group disparate treatment based. \n\nThe work does not acknowledge the difference between algorithmic fairness and fairness in broader socio-technical systems.\n\n\n\n## Minor\n\nL72 Citation [9] is a specific \"fairness distribution shift\" citation, not a general one about distribution shift, which is what is about the sentence. \n\nI wonder how different are actually Adult and ACS Income.\n\nThere seem to be many citations coming from Arxiv. Perhaps it could be good to check if there are more recent proceedings versions. \n\n\n[1] https://proceedings.neurips.cc/paper_files/paper/2022/file/0378c7692da36807bdec87ab043cdadc-Paper-Datasets_and_Benchmarks.pdf", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive comments.\n\n**[W1] The method is designed/tested in Neural Network architectures in tabular datasets. In tabular data algorithms like gradient boosting decision trees achieve SOTA[1] and are restricted to Linear Regression in regulated environments.**\n\n**Ans:** In this paper, we only consider the classification task and we leave the regression task in future work. We will add more discussion on the benchmark paper [1] in our revised version.\n\n\n**[W2] The paper calls fairness to just one metric --Demographic Parity-- Fairness is not just one metric. L_RFR is not performance-based but group disparate treatment based.**\n\n**Ans:** $L_{RFR}$ is specifically designed for demographic parity. There are many other fairness metrics, such as equal opportunities, and individual fairness, that could be specifically designed under distribution shift. We leave these parts in the future work. We will add the discussion in Appendix~H.\n\n**[W3] The work does not acknowledge the difference between algorithmic fairness and fairness in broader socio-technical systems.**\n\n**Ans:** We will add acknowledge the difference between algorithmic fairness and fairness in broader socio-technical systems. Algorithmic Fairness focuses on ensuring fairness and lack of bias in the decisions made by algorithms or machine learning models. Fairness in socio-technical systems goes beyond algorithmic fairness, considering broader societal impacts, ethical considerations, data biases, and interdisciplinary collaboration. It focuses on human-centered design, policy changes, and ongoing assessment to ensure technology aligns with societal values and promotes equitable outcomes.\n\n**[Q1] L72 Citation [9] is a specific \"fairness distribution shift\" citation, not a general one about distribution shift, which is what is about the sentence.**\n\n**Ans:** We thank the reviewer for this comment. We would like to clarify that our work targets the fairness problem under distribution shift. We believe the initial reference [9] can provide comprehensive background information and motivation for supporting the necessity of our work. However, from the perceptive of the reviewer, we concur that it is better to cite another work about general distribution shift here. Therefore, we will also cite existing work [2] on the general distribution shift here in the camera-ready version.\n\n[2] Wiles, Olivia, et al. \"A Fine-Grained Analysis on Distribution Shift.\" International Conference on Learning Representations. 2022.\n\n**[Q2] I wonder how different are actually Adult and ACS Income.**\n\n**Ans:** ACS Income dataset is coming from paper [3]. [3] releases a more comprehensive Adult dataset, which is a superset of the UCI Adult data from available US Census sources. The data span multiple years and all states of the United States, allowing researchers to study temporal and spatial distribution shifts.\n\n[3] Ding, Frances, et al. \"Retiring adult: New datasets for fair machine learning.\" Advances in neural information processing systems 34 (2021): 6478-6490.\n\n**[Q3] There seem to be many citations coming from Arxiv. Perhaps it could be good to check if there are more recent proceedings versions.[1] https://proceedings.neurips.cc/paper_files/paper/2022/file/0378c7692da36807bdec87ab043cdadc-Paper-Datasets_and_Benchmarks.pdf**\n\n**Ans:** We thank the reviewer for this thoughtful comment. As the reviewers mentioned, we will revise the Arxiv reference into proceeding versions in the camera-ready version.\n\n\n**[Q4] The work states that \"any distribution shift can be equivalent to data perturbations.\" Does this approach (Sec2.2) considers pure concept shift? When $P(\\mathbf{X}_s)=P(\\mathbf{X}_T)$ and $P(\\mathbf{Y}_s)=P(\\mathbf{Y}_T)$\n, but the only thing that changes is the relationship between covariate and target?  $X \\Rightarrow Y$.**\n\n**Ans:** We consider a general distribution shift case. We would like to highlight that the data perturbation $\\delta_X$ and $\\delta_Y$ are related to $X$ and $Y$. The pure concept shift can be included in data perturbation. For example, considering two examples $(X_1, Y_1)$ and $(X_2, Y_2)$, we can only add feature data perturbation to exchange features as $(X_2, Y_1)$ and $(X_1, Y_2)$. In this way, the data marginal distribution keeps the same but the relation between features and labels could be changed.\n\n**[Q5] Is the method strictly limited to NN? If so (since it's tabular data) it may be worth stating it earlier and clearer in the paper. I wonder if the theoretical results from section 2 can be aligned with the previous empirical findings of [4] https://arxiv.org/pdf/2302.05018.pdf.**\n\n**Ans:** We will mention that our paper is designed for NN and mentioned earlier in the paper. The extension for other models, such as decision tree, would be an interesting direction in the future, especially for SOTA performance of decision tree in regression tasks. As for comparison with paper [4], [4] adopts optimal transport to predict model performance while we use optimal transport to understand distribution shift problem and derive the model perturbation problem."}, {"review_id": "nmSsD9XtLK", "reviewer": "Reviewer_3kzb", "summary": "This paper tackles the issue of fairness in situations where there are distribution shifts, viewing it from the angle of model weight perturbation. The authors establish a theoretical link between distribution shift, data perturbation, and model weight perturbation, leading to the conclusion that distribution shift and model perturbation are essentially the same. They propose a sufficient condition to ensure fairness transference under distribution shift and introduce the Robust Fairness Regularization (RFR) method to meet these conditions and achieve robust fairness. Experimental results on both synthetic and real distribution shifts show the RFR method outperforms existing baselines in terms of fairness-accuracy tradeoff. ", "questions": "The hyperparameter tuning seems important. Do you record the experimental setting variables and preferably provide means to reimplement the result?\n\n----------\nI have seen other reviews and am a little confused and quite frankly appalled why we need to assume zero access to data from the target distribution. Is it even possible if we do not even know data from the target distribution (or the distribution variation) to do transfer learning? Although it would be interesting to know a good $\\rho$ a priori by either prior knowledge or theoretical bounds, many papers out there also do not discuss hyperparameters like $\\rho$, and they are all fine. I fail to see why this could be a MAJOR problem. In my view, it is not even a weakness, but something that would simply be good to know and good to ask the authors about. I would like to raise my confidence to 4 and advocate the paper if needed.", "limitations": "The authors do not talk about limitations of this paper.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The fairness under covariate shift is an important problem. The authors also manage to show that Data Perturbation equals Model Weight Perturbation and use this motivation to construct the following methodology descriptions and theoretical analysis. The motivation and writing is strong.   \n2. The method proposed by this work is novel about the insights from optimal transport to optimize the worst case within the model weight perturbation ball and Data Perturbation equals Model Weight Perturbation. This paper points out the limitations of previous works well.  \n3. The experiments on fairness accuracy tradeoff and hyperparameter study are sufficient. Although in the table it does not explicitly outperforms baselines, the tradeoff is clearly better.", "weaknesses": "I do not find important weaknesses on this paper. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for spending time and effort reviewing our paper. We appreciate your constructive comments and support to our paper. This feedback truly encouraged us to redouble our efforts in conducting quality and impactful research.\n\n\n**[Q1] The hyperparameter tuning seems important. Do you record the experimental setting variables and preferably provide means to reimplement the result?**\n\n**Ans:** We thank the reviewer for this constructive comment. The experiment result of tuning the key hyper-parameter $\\rho$ of RFR has been given in Figure 1 of the one-page rebuttal. It is observed that the accuracy-fairness trade-off of RFR is stable for $1\\times 10^{-5} \\leq \\rho\\leq 1\\times 10^{-3}$. This significantly indicates RFR is simple and easy to implement. Other hyper-parameter settings of the experiment have been given in Section 4.1.5 *Implementation Details* for the readers to reproduce our results.\n\n**[Q2] I have seen other reviews and am a little confused and quite frankly appalled why we need to assume zero access to data from the target distribution. Is it even possible if we do not even know data from the target distribution (or the distribution variation) to do transfer learning?**\n\n**Ans:** Yes. We target the fairness problem without accessing target distribution in this paper. Our proposed method RFR can mitigate the unfairness under distribution shift without the knowledge of target distribution. As shown in Table 1 of our paper, the experiment on three datasets comprehensively demonstrates the effectiveness of RFR, where RFR achieves a better accuracy-fairness tradeoff than the four baseline methods. Moreover, the results of a hyper-parameter study in Figure 4 of our paper show that RFR is stable towards the value of $\\rho$, indicating that RFR is easy to implement and re-produce in practical scenarios.\n"}]

Author rebuttal:
We thank all reviewers for the constructive comments and the positive evaluation for most reviewers. We provide the summary for the key revision.
* Add experiments on hyperparameter study on perturbation $\rho$.
* Add comparison results (Table. 1 and Figure 1 in one-page rebuttal) for the setting without distribution shift.
* Add more details on hyperparameter $\rho$ selection. We choose $\rho=0.0005$ in the experiments and it already works well. We also add discussion on several follow-up works to further improve our performance via better hyperparameter $\rho$ selection via auxiliary information or AutoML with target performance predictor.
* Add discussion on the difference from several existing works [1,2,3].
* Provide more justifications for experimental observation.
* More clarification on the bound Eq. (5).
* add more details on the proof of theorem 2.3, lines 499-501.
* Revise several notation typos.

[1] Zhu, Jiongli, et al. "Consistent Range Approximation for Fair Predictive Modeling" arXiv preprint arXiv:2212.10839.

[2] Wiles, Olivia, et al. “A Fine-Grained Analysis on Distribution Shift.” International Conference on Learning Representations. 2022.

[3] Lu, Yuzhe, et al. "Predicting Out-of-Distribution Error with Confidence Optimal Transport." arXiv preprint arXiv:2302.05018 (2023).

Meta-review: This paper has an overall positive opinions after a couple of score revisions after rebuttal.  

Authors propose a novel method to address fairness issues under covariate shift when target samples are not available. The problem is important. Authors did a number of additional experiments that include ablations, hyper parameter sensitivity and comparisons with baselines during the rebuttal that seems to have addressed reviewer concerns adequately.

I encourage the authors to add these new experimental results to the camera ready.
Decision: Accept (poster)