FR-GAN: Fair and Robust Training
Authors: Yuji Roh
Kangwook Lee
Gyeong Jo Hwang
Steven Euijong Whang
Changho Suh
Venue:  (2020)
Keywords: generative adversarial networks
model fairness
model robustness

Abstract: We consider the problem of fair and robust model training in the presence of data poisoning. Ensuring fairness usually involves a tradeoff against accuracy, so if the data poisoning is mistakenly viewed as additional bias to be fixed, the accuracy will be sacrificed even more. We demonstrate that this phenomenon indeed holds for state-of-the-art model fairness techniques. We then propose FR-GAN, which holistically performs fair and robust model training using generative adversarial networks (GANs). We first use a generator that attempts to classify examples as accurately as possible. In addition, we deploy two discriminators: (1) a fairness discriminator that predicts the sensitive attribute from classification results and (2) a robustness discriminator that distinguishes examples and predictions from a clean validation set. Our framework respects all the prominent fairness measures: disparate impact, equalized odds, and equal opportunity. Also, FR-GAN optimizes fairness without requiring the knowledge of prior statistics of the sensitive attributes. In our experiments, FR-GAN shows almost no decrease in fairness and accuracy in the presence of data poisoning unlike other state-of-the-art fairness methods, which are vulnerable. In addition, FR-GAN can be adjusted using parameters to maintain reasonable accuracy and fairness even if the validation set is too small or unavailable.

Under review as a conference paper at ICLR 2020
FR-GAN: FAIR AND ROBUST TRAINING
Anonymous authors Paper under double-blind review
ABSTRACT
We consider the problem of fair and robust model training in the presence of data poisoning. Ensuring fairness usually involves a tradeoff against accuracy, so if the data poisoning is mistakenly viewed as additional bias to be fixed, the accuracy will be sacrificed even more. We demonstrate that this phenomenon indeed holds for otate-of-the-art model fairness techniques. We then propose FR- FR-GAN, which holistically performs fair and robust model training using generative adversarial networks (GANs). We first use generator that attempts to classify examples as accurately possible. In addition, we deploy two discriminators: (1) a faimess discriminator that predicts the sensitive attribute from classification results and (2) robustness discriminator that distinguishes examples and predictions from a clean validation set. Our framework respects all the prominent fairness measures: disparate impact, equalized odds, and equal opportunity. Also, FR-GAN opti- mizes fairness without requiring the knowledge of prior statistics of the sensitive attributes. In our experiments, FR-GAN shows almost no decrease in fairness and accuracy in the presence of data poisoning unlike other statc-of-the-ar fair- ness methods, which are vulnerable. In addition, FR-GAN can be adjusted using parameters to maintain reasonable accuracy and faimess even if the validation set is too small or unavailable.
INTRODUCTION
As machine learning becomes widespread in the Software 2.0 era (Karpathy) 2017). there is an ur- gent need to address the issues of faimess and robustness. For sensitive applications like healthcare, finance, and self-driving cars, a trained model must not discriminate customers based on sensitive attributes including age, sex, or religion. In addition, as applications often rely on external data sets for their training data, the model training must be resilient against noisy or even adversarial data.
Roth 2018 Verma & Rubin 2018) has focused on developing metrics such as disparate impact (Feldman al., 2013). equalized odds (Hardt et al.) 2016), and equal opportunity (Hardt et al., 2016). which capture various notions of discrimination. More recently, there has been surge in unfairness mitigattion techniques (Bellamy et al.) 2018), which improve the model fairness by either fixing the training data, training process, trained model. Unfairness mitigation usually involves some tradeoff be- tween the model's accuracy and faimess. Most recently, generative adversarial networks (GANs) are being adapted to a fairness setting (Zhang et al.) 2018). The architecture of GANs is suitable because it can be used to compete the training between two objectives; accuracy and fairness.
Traditionally, model fairness research (Venkatasubrammaninn 2019 Chouldechova
Robust model training also important and needs to be concurrently taken into consideration. There has been a proliferation of algorithms that make model training resilient against noisy and even adversarial data (Natarajan et al., 2013] Biggio al., 201T, Frénay & Verleysen 2014). The latter scenario may occur when the source of the training data is external, and we cannot control the data that is being added. For example, any public dataset on the Web may contain intentionally- false information. Moreover, one must have robust training algorithm if the data set comes from diverse users' inputs that cannot be fully trusted (Konečný et al., (0166). Recently, data poisoning attacks have become increasingly sophisticated where it is becoming difficult to sanitize the data against all of them (Koh et al., 2018). In addition, dataset searching is becoming mainstream as demonstrated by Google Dataset Search (Goods) (Halevy et al.) 2016) and its public version for searching scientific datasets (Noy et al. 2019). While data lakes within companies may consist of refined datasets, datasets in the public are easy to poison. And anyone can poison public datasets

---

Under review as a conference paper at ICLR 2020
using attacks in the literature and share them. We thus believe that it is essential to address both bias and poisoning as a preventive measure.
Solving model faimess without addressing data poisoning may lead to worse tradcoff' between accuracy and fairness. For example, consider a banking system that is giving out loans where there two sensitive groups: men and women. According disparate impact, the two groups must have similar ratios of positive predictions. A perfect disparate impact would have a value of 1. Suppose the clean training data has no bias and perfect (or high) disparate impact score. Now suppose this data is poisoned where half of the positive labels of men are flipped to negative labels. If the model fitted so as to make more negative predictions on men than women, then disparate impact would worsen. To address this problem, one may suggest using data sanitization techniques prior to the model training. but it is challenging to do so without any knowledge of the model. Even though many data sanitization methods are proposed, there have been newer attacks that can easily evade such defenses (Koh al. 2018).
Our main contribution is the development of a new framework called FR-GAN, which trains ac- curate models that are also fair and robust to poisoning. FR-GAN consists of a generator used for classification, a discriminator that distinguishes predictions from one sensitive group against oth- ers, and a second discriminator that distinguishes (examples, predictions} of the training data from {examples, labels} of separate and clean validation set. The first discriminator ensures that the prediction ý is independent of the sensitive attribute z. We show that variations of this approach can be used to maximize any of the following prominent fairness measures: disparate impact, equalized odds, and equal opportunity. The second discriminator ensures that the model predictions on the training data are "consistent" with labels on clean data. We theoretically show that FR-G FR-GAN opti- mizes fairness without requiring the knowledge of prior statistics of sensitive attributes. In addition, the parameters of FR- RR-G can be adjusted to maintain reasonable accuracy and fairness even if the validation set is too small or unavailable. In the following sections. we present the related work, demonstrate the weaknesses of current fairness methods, and propose FR-GAN.
RELATED WORK
Model fairness The notion of discrimination has many definitions and usually comes from certain social goals that one wants to guarantee. Fairness research has been focused on defining fairness
measures that capture a variety of notions of fairness (Verma & Rubin 2018). There are largely two types of fairness measures: (1) group fairness (Barocas & Selbst 2016 Hardt et al. 20166). which ensures similar statistics between two sensitive groups; (2) individual fairness (Dwork et al. 2012). which guarantees similar prediction results across nearby examples. Group fairness measures are widely studied in general applications, even though these cannot be simultaneously satisfied in cer- tain circumstances (Kleinberg et al., (017). In addition, there are also recent significant works that focus on individual fairness (Garg et al. 2018 Yurochkin et al. 2019 Kearns et al.] 2019: Jung et al. 2.19). Among the two types of fairness with different aspects, we focus on group fairness as our initial research. Recently, there has also been a surge of research on unfairness mitigation tech- niques (Bellamy et al. 2018).. Depending
where fix occurs, there are mainly three approaches: (1) pre-processing techniques (Kamiran & Calders) 2011 du Pin Calmon et al., 2017 Zemel et al.)
2013
Feldman al., (015) that fix the training data; (2) in-processing techniques (Zafar et al.
2017 Jiang & Nachum 2019 Zhang et al. 20188 Kamishima et al. 2012 Cotter et al. 2019 2018
Agarwal et al. (018)) that address the issue during model training; and (3) post-processing tech-
niques (Hardt et al.) 20166, Pleiss et al. 2017 Kamiran et al. 2012) that manipulate predictions while maintaining the model.
Among the three, the in ir-prccesiing techniques have the advantages that one can work with any data and that there is more control on model training (Venkatasuubramaninn 2019). FAIRNESS CON- STRAINTS (Zafar et al., (2017) incorporates regularization term that reflects fairness constraints in the context of convex margin-based classifiers such as logistic regression and support vector ma- chines (SVMs). The main hyperparameter is the À term, which serves to balance the accuracy and fairness objectives. LABEL BIAS CORRECTION (Jiang & Nachum 2019) assumes the existence of true yet possibly biased labels. A reweighting algorithm is proposed with theoretical guarantees that training on the resulting loss corresponds to training on the true unbiased labels, which yields a fair model. Here the hyperparameters include the number of training iterations. Finally, ADVERSARIAL.
2

---

Under review as a conference paper at ICLR 2020
(pos) (meg)
Disparats Impact
0.8-
5.3
(nee) polemned data
0.7
0.3
0.6
0.5
-3.3
FC clean data
0.4
PC polsoned data
-10.0
-5.0
0:0
5.0
10.0
0.50
0.63
1.70
0.75
0,80
0.85
0.90
a
Accuracy
(a) Synthetic data with z-flipped poisoning
(b) faimmes tradeoff for FAIRNESS CON-
STRAINTS Zafar et al. 2017
The left figure shows a synthetie dataset with a certain poisoning. Examples are divided into = 0 (marked with circles) and (crosses) per sensitive attribute The blue points indicate positive labels while the red points denote negative ones. For data poisoning, we flipped values that reduce the model accuracy the most (similar to label flipping (Paudice et al. 2018). but specialized for fairness). The right figure shows that poisoning significantly worsens the accuracy- fairness tradeoff (i.e., shifts the left) of the FAIRNESS CONSTRAINTS method (Zafar al.) 2017).
Figure| 1:
DEBIASING (Zhang et al. 2018) trains classifier that adversary cannot predict the sensitive attributes based on the output of the classifier. Hence, the classifier achieves fairesss when its pre- dictions do not have any correlation with sensitive attributes. This idea of adversarial training is similar to GANs (Goodfellow et al., 2014). ADVERSARIAL DEBIASING provides insights of using adversarial training minimize mutual information between the prediction and sensitive attributes. In this paper, we strengthen the theoretical results of ADVERSARIAL DEBIASING using informa- tion theory, and this motivates us provide systematic methodology for various faireess metrics (Section
As we demonstrate in Section 3 the existing techniques are not tailored for robust training, so are vulnerable to data poisoning attacks. In comparison, FR-GAN addresses both model faimess and robust training within the same model training process.
Robust training There is a heavy literature on how to make the model training robust against noisy even adversarial data (Natarajan al.) 2013 Biggio al.) 2011 Frénay & Verleysen 2014 Kurakin et al. 2017). A major challenge is that there can be a wide range of data poisoning attacks that keep on evolving. One defense approach to sanitize the training data and thus fix the root cause. A fundamental limitation is that we do not know in general what kind of poisoning will occur, so a universal defense against all possible attacks does not seem feasible as demonstrated by Koh let al. (2018). A more recent trend is to develop general defense algorithms for any attack during model training using meta learning (Veit et al.] 2017: Li et al., 2017 Xiao et al. 2015 Hendryykks et al. 2018). Ren et al. (2018) propose a meta-learning framework that employs clean validation set for the purpose of preventing the model training from being influenced by poisoned data. By using the validation loss as meta objective, it intends to defend against any poisoning attack. Our FR-GAN framework is inspired by this approach, but employs a GAN-based model to support fair and robust training without using meta learning. In particular, the design of FR-GAN's robustness discriminator is based on natuallifformation.
theoretical insights (Section 4.2) Another line
of research is defending against adversarial attacks during test time (Biggio al. 2013 Goodfellow
et al. 2015 Wong & Kolter 2018). In comparison, our focus is on defending against data poisoning on the training data.
3 VULNERABILITY OF FAIRNESS METHODS
We perform experiments to demonstrate that state-of-the-art fairness methods are indeed vulnerable even to simple poisoning attacks. We generate a synthetic dataset as shown in Figure Ia| (see the generation details in Section 5.1). There are two non-sensitive attributes a and b, which are reflected in the x-axis and y-axis, respectively. The examples are further divided into two classes based on the sensitive attribute 2. To generate poisoned data, measure accuracy degradation for flipping of each 2. We then choose the top- 10% of the a values with the highest degradation and flip them. This approach is similar to the existing label flipping method (Paudice al. 2018), except that the
3

---

Under review as a conference paper at ICLR 2020
Softmax
Di(2)
Z, Y
X, Z
Generator (Classifier)
Discriminator for Faimess
D( =
D121(8)
X, 2,
Discriminater for Robustness
Xvals Zoul Yeol
Figure 2: The architecture of FR-GAN.
value is being flipped instead of the label that the model fairness is also affected. To make validation set, we randomly select clean examples that amount to 10% of the entire training data.
We use disparate impact (Feldman et al. 2015) a fairness measure and evaluate one fairness method, called FAIRNESS CONSTRAINTS Zafar et al. 2017). As this method involves a regulariza- tion factor À that balances the accuracy and fairness objectives, we can obtain tradeoff curve by adjusting its value. Figure shows two accuracy- fairres-fairmess tradeoff curves obtained with the clean and poisoned synthetic datasets. Notice that adding data poisoning clearly shifts the curve to the left, which means both accuracy and fairness decrease. This coincides with our intuition. The poisoning confuses the model so that there are more biased examples to fix, which in turn makes overreact and thus sacrifice more accuracy. In the supplementary,
show similar trends when the data is poisoned via label flipping. Section 5 we will show how data poisoning affects LABEL BIAS CORRECTION (Jiang & Nachum 2019) and ADVERSARIAL DEBIASING (Zhang et al. 2018).
4 FR-GAN
We now describe FR-G FR-GAN, which is shown in Figure 2 Unlike traditional GANs, the generator is a classifier that receives an example x E X and returns prediction ÿ. The other two discriminators optimize fairness and robustness describe in the following sections.
4.1 FAIRNESS
We denote by D., the training data set. Suppose Drr has m examples {(x((1, 2(0,, ()) where (+)) contains the non-sensitive attributes, z(() contains the sensitive attributes, and (1)) is the label. Both the sensitive attribute and label can be multi-class, i.e., they can have one of multiple values. For notational simplicity, we assume there is sensitive attribute, which merged result multiple sensitive attributes with a larger alphabet size. For illustrative purposes, focus disparate impact, leaving in the supplementary our formulation for equalized odds and equal opportunity.
be viewed
The first discriminator in FR-GAN distinguishes predictions w.r.t. sensitive group from those in the others. Disparate impact intends the sensitive attribute to be independent the model's prediction, i.e.,
I(2; Ý) -
We explain how FR-GAN enforce the above constraint. Let P2(z) be the distribution of Z where z Z and Z is the set of possible sensitive attribute values. Let YIZ (+) and
Y (t) Then Py(() Σ.ez Pz
We show that mutual information is equivalent to the following expression where the optimal dis- criminator (g) (6.2(0.2) and E.ez (1) EV.
(o)
Theorem 1. T(Z; max E (2()) ELez Pz(=)Eppi, [og (ogD.()) + H(Z).
While deferring the detailed proof to the supplemental materials, we provide brief overview of the proof. As the optimization problem in the RHS is convex, we find the optimal dicriminator by solving the KKT conditions. We then show that the maximum value attained by the optimal discriminator equal to the mutual information by using the properties mutual information and the generalized Jensen-Shannon divergence (Lin 1991).
4

---

Under review as a conference paper at ICLR 2020
What is more involved than showing the above equality is designing right optimization prob- lem. One needs to carefully handcraft a plausible optimization problem so that its unique solution matches the desired quantity. Here, design the above optimization problem via 'guess- -ccesk approach aided by the structural insights across the KI. divergences that appear in an alternative ex- pression of mutual information.
We now discuss how to implement the above expression. Since we do not know (+) exactly, we compute the following empirical version:
Σ SEZ
Pz(e)
Σ
-
log D2 (1(0)) (())).
max Da (.) E. D. (1) - I49
m
boff)
Now for sufficiently large m, the number m, of examples with z = z approximately the same as (z)m.. Therefore, the above expression becomes;
Σ Σ
1 log + (Z)).
max D. E. 1.)
ZEZ (aa
Interestingly, this formulation is exactly the same that in the original GAN (Goodfellow al. 2014) when 2 = 2. We also remark that our formulation does not require a prior knowledge on Pz(=).
4.2 ROBUSTNESS
We utilize clean yet small validation set to ensure robust training. In particular, add second discriminator that distinguishes the training data with predictions {(x((), 200), )( from the validation set {I 2val yvall maal
(6) ) Intuitively, if the classifier is confused by data poisoning in
the training data, then its predictions will not be consistent with the labels of the clean data, and the discriminator would able to detect that difference. While using a validation set is inspired by the meta learning approach by Ren al. (2018), take adversarial learning approach. A key motivation behind this choice is that our GAN approach introduces a knob that controls the emphasis of robust training. We find that this knob enables FR-GAN be more robust to the validation set size. See details in Section 5.1
Our formulation is follows. Let V Bern(3) denote whether an example comes from training data (V = 1) or from the validation set (V = 0). We then want to ensure that V is independent of the training data and predictions, i.e., I(V; X, Z, Y) O, which means the predictions on the training data are indistinguishable from labels of the validation set. This way we can mimic the clean data set while expecting an indirect sanitization effect. We optimize this independence using formulation similar to that of fairness. Another possible formulation is to use the hard decision value from D(Y), say Z, instead of Z. However, this discriminator can only trained after the fairness discriminator is trained. Instead, we take approach that allows us to train the two discriminators at the same time.
4.3 ARCHITECTURE
Figure illustrates the architecture of FR-GAN, For the loss function the generator, we employ the cross entropy loss:
L
( log (() - (1 -1 (0) log(1 g()))
1=1
As per Theorem I we set the loss function w.r.t. the fairness discriminator as:
La max
Σ Σ
1 m
log D. (f) + H(Z)
DC)
ZEZ koff)=s
where D(.) == (Di()..... D|j ())). The condition Exe Di( = can be enforced by adding softmax layer the discriminator.
5

---

Under review as a conference paper at ICLR 2020
Table 1: Accuracy and fairness performances on the clean and poisoned synthetic datasets w.r.t
disparate impact. Two types of methods are compared: (1) non-fairness methods: logistic regres-
sion (LR), the meta learning by Ren et al. (2018) (ML), and R-GAN (FR-GAN with A 0);
(2) fairness-only methods: FAIRNESS CONSTRAINTS (Zafar et al., (017)) (FC), LABEL BIAS COR-
RECTION (Jiang & Nachum 2019) (LBC), and ADVERSARIAL DEBIASING (Zhang et al. 2018) (AD). Also "+ LD" denotes loss defense (Koh et al. 2018) (a data sanitization technique that is applied prior to model training). For FR-GAN, the validation set is 10% of Dr. For each fairness or accuracy result of the poisoned data, we make comparison with the clean data result and show the percentage increase or decrease.
Type
Method
Clean data
Poisoned data
D. impact
Acc.
D. impact
Acc.
LR
0.409
0,885
0.436 (6.60% f)
0.876 (1.02% J)
Non-fair
ML
0.429
0.883
0.386 (10.0% J)
0.884 (0.11% f)
R-GAN
0.414
0.883
0.338 (18.4% J)
0.884 (0.11 % f)
FC
0.822
0.806
0.756 (8.03% J)
0.655 (18.7% J)
LBC
0.819
0.760
0.816 (0.37% L)
0.738 (2.89% J)
AD
0.807
0.811
0.711 (11.9% J)
0.812 (0.12% f)
Fair-only
FC+LD
0.822
0.806
0.763 (7.18% L)
0.651 (19.2% J)
LBC+LD
0.819
0.760
0.513 (37.4% L)
0.859 (13.0% f)
AD+LD
0.807
0.811
0.730 (9.54% ()
0.818 (0.86% f)
Both
FR-GAN
0.813
0.809
0.830 (2.09% 1)
0.808 (0. 12% J)
Finally, implementing L(V; X, 2, 8) as advised by Theorem robustness discriminator
!
we set the loss function w.r.t the
L3
Σ Ep(i==0
- log D" (x 00) g())
Σ
Log(1
D' 20 (()) H(V).
D*(1)
The final objective function is to minimize the weighted sum of these value functions:
min L1 + \1 A2L3.
G(-)
Here x and X2 are tuning knobs that play roles to emphasize fair and robust training, respectively.
5 EXPERIMENTS
We provide experimental results for disparate impact, while leaving in the supplementary those for equalized odds and equal opportunity, All of our results are on a separate test set. More implemen- tation details are also in the supplementary.
5.1 SYNTHETIC DATA RESULTS
For the synthetic data, we generate 2,000 examples with two non-sensitive attributes and b, sensitive attribute z, and label y, using a method similar the algorithm proposed by Zatar al. (017). Both and y are binary and generated uniformly at random. For the four possible (z. y) combinations, we generate different normal distributions for a and b. Finally for each example, the and b values are sampled as per the normal distribution associated with the (z, y) pair. For data poisoning, we flip the values of examples so as to lower the model's accuracy the most as described in Section
To make fair comparison in the context of robust training, we also improve the existing fairness methods with data sanitization and compare them with FR-GAN. Since FR-GAN is one of the first works to address both fairness and robustness, there are few baselines to compare with. Hence, employed one reasonable baseline, which first sanitizes the poisoned data using well-known sanitization technique and then performs each fairness algorithm. There are several sanitization techniques surveyed in Koh al. (2018), and we choose the loss defense method. Here examples that

---

Under review as a conference paper at ICLR 2020
Dipparate Iuppoet
Disparate Iunpoet
Disprrate
clean data
clean data
EILGAN an case cile
polsoned
polsoned data
FR-GAN on palumed data (nd 18R1
(L:t Accuracy
USS
Accuracy
Accuracy
(a) LABEL BIAS CORRECTION
(b) ADVERSARIAL DEBIASING
(c) FR-GAN (Val. set size = 10%)
Disprrate Impact
Daparite Imppuct
Diperrate Imppoct
FR-GAN chen lkda
FR-GAN on chon
FR-GAN thes date
dats Ival 2NC
promord (eal (dK))
poinceal (.5
6.00
Accuracy
Accuracy
Accuracy
(d) FR-GAN
(e) FR-GAN
(I)
FR-GAN
(Val. set size 2%; λ2 0.2)
(Val. set size = 0.5%; A = 0.2)
(Val. set size = 0.5%; 0.1)
Figure securacy-fairnes tradeoff curves.
are not well fit by the model and result in high loss are considered poisoned and are subsequently discarded. Loss defense is the only method that utilizes the trained model and thus has the best chance of defending poisoning attacks.
Accuracy and fairness We compare FR-GAN with FAIRNESS CONSTRAINTS Zafarr et al., 2017). LABEL BIAS CORRECTION (Jiang & Nachum 2019), ADVERSARIAL DEBIASING (Zhang al. 2018). and a plain non- non-fairness method, logistic regression. We also compare with the meta learning approach by Ren et al. (2018), which focuses on robust training. For this comparison, we (no fairness constraint), and name R-GAN. For FR-GAN, we use validation set that amounts to 10% of Der When setting λ and X2, we usually fix A2 to some value and then adjust λ1. For the clean data, we apply proper hyperparameters so that the disparate impacts are similar (around 0.8) across all distinct methods. We do the same for the poisoned data. There is no such hyperparameter tuning for logistic regression and the meta learning by Ren et al. (0018), the frameworks have no knob for adjusting faimesss. We find that this leads to poor disparate impacts. For the three fairness methods, data poisoning aggravates performances in accuracy, fairness or both. For example, the disparate impact and accuracy of FAIRNESS CON- S STRAINTS fall by 8.03% and 18.7%, respectively. On the other hand, the performance changes for FR-GAN are negligible: disparate impact increases by 2.09%, and accuracy decreases by 0.12%. Table I also shows that combining the fairness methods with loss defense (rows 6-8) does not al- ways yleld better accuracy and fairness. In fact, using the loss defense may lower the accuracy or fairness (e.g., LBC+LD has a disparate impact of 0.513 on poisoned data while LBC has 0.8 0.816). Perhaps this is because the data sanitization sometimes mistakenly discards clean data as well. The results demonstrate that even data sanitization technique may not help much. In the supplementary, we also perform experiments using equalized odds and show that FR-GAN has similar benefits.
tailor our framework to setting where A
We observe how accuracy trades off with fairness on clean and poisoned datasets. The results for FAIRNESS CONSTRAINTS are shown in Figure [b]] For LABEL BIAS CORRECTION, we employ the number of training iterations tradeoff curve shifts to the left, which demonstrates a clear tradeoff degradation. For ADVERSARIAL DEBIASING, employ the parameter (Zhang et al., 2018) analogous to X1 as knob to trade accuracy off fairness. We see in Figure 3b that the tradeoff curve again shifts to the left.
knob to trade accuracy off fairness. As shown in Figure Bal the
Validation set size Figures 3c to 3e show how the validation set size affects the robustness of FR- GAN. In particular, we compare the cccuracy-fairness tradeoff of FR-GAN on clean data and that on poisoned data while varying the size of the validation set. When running
poisoned data, we fixed \ = 0.2 and varied Å1- We see that even 2% validation set (Figure 3d) is sufficient to maintain the accuracy and fairness obtained on the clean data. When using 0.5% (Figure Beb, the validation set is too small and has an adverse effect on the training. However, our framework includes a tuning knob X2, so by decreasing X2 down to 0, could de-emphasize robust training, thereby avoiding
7

---

Under review as a conference paper at ICLR 2020
the adverse effect (Figure 31). This is in contrast to the meta learning approach, which suffers from a non-negligible performance degradation for a very small validation set. See details in the supplementary. The results show that only holistic framework like FR-GAN achieve both excellent model fairness and training robustness. In comparison, other methods tailored for only one of these objectives lose either accuracy, fairness both.
Ablation study (Without 'R') For small value of X2, we observe in Figure 3i that the accuracy- fairness tradeoff curve shifts to the left. just like other fairness-only methods. To make the compari- son clearer, we also plot them together in the supplementary. (Without 'F") When X = 0. FR-GAN expected to behave similarly to the other non-fair algorithms. The performance of FR-GAN with One can observe that R-GAN achieves
A = 0, which we dub as R-GAN, is reported in Table []] poor fairness performance just like the other non-fair algorithms.
5.2 REAL DATA RESULTS
We use the following benchmark datasets: ProPublica COMPAS (Angwin et al. 2016) and Adult Census [Kobavi] 1996). For sensitive attributes, we use SEX for both datasets. We do not employ the German Credit data set (Dua & Graff 2017) its highly biased towards positive labels. With such dataset, it is nearly impossible to observe any meaningful tradeoff.
For data poisoning, we use the same method employed on synthetic data: flipping the values so as to maximize the accuracy performance degradation. While there may be stronger attacks, this poisoning attack is effective enough to fool the three fairness methods. In the supplementary, we also evaluate FR-GAN against label flipping attacks and show similar results. We use the same constructions for the validation set and poisoning as described Section
Accuracy and fairness See Table 2 As in Table I we apply proper hyperparameters so that disparate impacts are similar across all distinct methods, both for the clean and poisoned datasets. The results are similar to Table T logistic regression and meta learning exhibit poor disparate im- pacts; the three fairness methods have worse disparate impact and accuracy due to data poisoning; and FR-GAN again shows little degradation both in fairness and accuracy. Table also shows that combining the fairness methods with loss defense (rows 6-8 and 15-17) does not always yield better accuracy and fairness and may even lower them, which is consistent to the results on synthetic data.
DISCUSSION
In this section, we discuss possible ways to extend FR-GAN.
Subjective labels FR-GAN can be generalized to a setting where labels are subjective instead of being clean or poisoned. Here the dataset without the undesirable human biases becomes the clean validation set. In general, given two datasets with different distributions where one of them is desirable, FR-GAN can train robustly against the other distribution.
Constructing real datasets We propose a general methodology for constructing real dataset that has both bias and poisoning, as well as a "clean" validation set. Suppose that we are performing evaluation tasks (say loan decisions) where there is high chance of human bias (i.e., poisoning) due to the high workload. We can construct clean validation set by selecting a few tasks, assigning more evaluators, and taking a majority vote of the evaluations to minimize the bias. While employing more evaluators can be expensive, constructing a small validation set is sufficient for robust training, which makes this method practical.
Generalizing to all attacks An interesting question is whether FR-GAN can defend against all poisoning attacks. If we know all the possible attacks, we can construct a training set containing these attacks. In the more challenging case where we do not know which attacks even exist, there seems to be a fundamental limitation in protecting against the attacks. Generalization a universal problem in machine learning where model trained on one dataset is not guaranteed to perform well in another dataset with a different distribution. Although the generalization is a critical issue to address, it is currently beyond the scope of this paper.
8

---

Under review as a conference paper at ICLR 2020
Table Accuracy and fairness performances on real data for disparate impact. Two types of meth- ods are compared: (1) non-fairness methods: logistic regression (LR), the meta learning by Ren et al. (2018) (ML), and R-GA R-GAN (FR-GAM -RA with À 0); (2) faimess-only methods: FAIRNESS CON- STRAINTS Zatar et al.) (017) (FC). LABEL BIAS CORRECTION (Jiang & Nachum 2019) (LBC), and A ADVERSARIAL DEBIASING (Zhang et al. 2018) (AD). Also "+ LD" denotes loss defense (Koh) et al. 2018) (a data sanitization technique that is applied prior to model training). For FR-GAN, the validation set 10% D... For each fairness or accuracy result of the poisoned data, we compare with the clean data result and show the percentage increase decrease.
Dataset
Type
Method
Clean data
Poisoned data
D. impact
Acc.
D. impact
Acc.
LR
0.465
0.674
0.431 (7.31%
0.667 (1.04% 4)
Non-fair
ML
0,493
0.680
0.164 (66.7% J)
0,661 (2.79% !)
R-GAN
0.574
0.683
0.393 (31.5%
0.650 (4.83% J)
FC 0.777
0.682
0.790 (1.67% 1)
0.671 (1.61% 4)
LBC
0,866
0.671
0.807 (6.81%
0.615 (8.35% J)
AD
0.846
0.680
0.822 (2.84% !)
0.655 (3.68% L)
COMPAS
Fair-only
FC LD
0.777
0.682
0.790 (1.67%
0.668 (2.05%
LBC + LD 0,866
0.671
0.822 (5.08% !)
0.628 (6.41%
AD + LD
0.846
0.680
0.836 (1.18%
0.620 (8.82% ))
Both
FR-GAN
0.936
0.667
0.947 (1.18% 1)
0.660 (1.05% 4)
LR
0.328
0.847
0.230
(29.9%
0.848 (0.12% f)
Non-fair
ML
0.346
0.844
0.215 (37.9% J)
0.843 (0.12% !)
R-GAN
0.335
0.847
0.285 (14.9%
0.834 (1.53% 4)
FC
0.825
0.826
0.816 (1.09% !)
0.828 (0.24% 1)
LBC
0,825
0.825
0,825 (0.00% -)
0.795 (3.64% !)
AD
0.850
0.767
0.679 (20.1% 4)
0.754 (1.69% J)
Adult
Fair-only
FC + LD
0.825
0.826
0.798 (3.27%
0.807 (2.30% J)
LBC + LD 0.825
0.825
0.815 (1.21% .)
0.811 (1.70% !)
AD LD
0.850
0.767
0.821 (3.41%
0.728 (5.08% 4)
Both
FR-GAN
0.870
0.819
0.866 (0.46% J)
0.827 (0.98% 1)
CONCLUSION
We proposed FR-GAN, which is holistic framework that enables both unfairness mitigation and robust training. Our key contribution proposing a novel GAN architecture that enjoys the syner- gistic effect of combining two approaches: (1) employing fairness discriminator that distinguishes predictions w.r.t. one sensitive group from others and (2) employing robust discriminator that dis- tinguishes training data with predictions from clean yet small validation set. We showed that existing fairness methods are vulnerable to data poisoning, even when combined with data samitiza- tion techniques. In comparison, FR GAN is robust to the poisoning and can be adjusted to maintain reasonable accuracy and fairness even if the validation set is too small or unavailable. We showed that FR- FR-GAN is guaranteed to maximize fairness and supports all the prominent fairness measures: disparate impact, equalized odds, and equal opportunity.
REFERENCES
Alekh Agarwal, Alina Beygelzimer, Miroslav Dudík, John Langford, and Hanna M. Wallach. A reductions approach to fair classification. In ICML, pp. 60-69, 2018.
J. Angwin, J. Larson, S. Mattu, and L. Kirchner. Machine bias: There's software used across the country to predict future criminals. And its biased against blacks., 2016.
Solon Barocas and Selbst. Big Data's Disparate Impact. California Law Review, 2016.

---

Under review as a conference paper at ICLR 2020
Rachel K. E. Bellamy, Kuntal Dey, Michael Hind, Samuel C. Hoffman, Stephanie Houde, Kalapriya Kannan, Pranay Lohia, Jacquelyn Martino, Sameep Mehta, Aleksandra Mojsilovic, Seema Na- gar, Karthikeyan Natesan Ramamurthy, John T. Richards, Diptikalyan Saha, Prasanna Sattigeri, Moninder Singh, Kush R. Varshney, and Yunfeng Zhang. Al faimess 360: An extensible toolkit for detecting, understanding, and mitigating unwanted algorithmic bias. CoRR, abs//810.011 2018.
Battista Biggio, Blaine Nelson, and Pavel Laskov. Support vector machines under adversarial label noise. In ACMI,, pp. 97-112, 2011.
Battista Biggio, Igino Corona, Davide Maiorca, Blaine Nelson, Nedim Smdic, Pavel Laskov, Gior- gio Giacinto, and Fabio Roli. Evasion attacks against machine learning at test time. In ECML PKDD, pp. 387-402, 2013.
Alexandra Chouldechova and Aaron Roth. The frontiers of faimess in machine learning. CORR, abs/1810.08810, 2018.
Andrew Cotter, Maya R. Gupta, Heinrich Jiang, Nathan Srebro, Karthik Sridharan, Serena Wang, Blake E. Woodworth, and Seungil You. Training well-generalizing classifiers for fairness metrics and other data-dependent constraints. CoRR, abs/ 000080, 2018.
Andrew Cotter, Heinrich Jiang, and Karthik Sridharan. Two-player games for efficient non-convex constrained optimization. In ALT, pp. 300-332, 2019.
Flávio du Pin Calmon, Dennis Wei, Bhanukiran Vinzamuri, Karthikeyan Natesan Ramamurthy, and Kush R. Varshney. Optimized pre-processing for discrimination prevention. In NewrlPS, pp. 3995-4004, 2017.
Dheeru Dua and Casey Graff. UCI machine learning repository, 2017.
Cynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Richard Zemel. Fairness through awareness. In Proceedings of the 3rd Innovations in Theoretical Computer Science Con- ference, ITCS 12. pp. 214-226, New York, NY, USA, 2012. ACM. ISBN 978-1-4503-1115-1. doi: 10.1145/2090236.000000 2090255.
Michael Feldman, Sorelle A. Friedler, John Moeller, Carlos Scheidegger, and Suresh Venkatasubra- manian. Certifying and removing disparate impact. In KDD, pp. 259-268, 2015.
Benoît Frénay and Michel Verleysen. Classification in the presence of label noise: A survey. IEEE Trans. Neural Netw. Learning Syst., 25(5):845-869, 2014.
Sahaj Garg, Vincent Perot, Nicole Limtiaco, Ankur Taly, Ed Huai hsin Chi, and Alex Beutel. Coun- terfactual fairness in text classification through robustness. In AIES, 2018.
lan J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio. Generative adversarial nets. In NeurlPS, pp. 2672-2680,
2014.
Ian Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In ICLR, 2015.
Alon Halevy, Flip Korn, Natalya F. Noy, Christopher Olston, Neoklis Polyzotis, Sudip Roy, and Steven Euijong Whang. Goods: Organizing google's datasets. SIGMOD, 2016.
Moritz Hardt, Eric Price, and Nati Srebro. Equality of opportunity in supervised learning. In NewrlPS, pp. 3315-3323, 2016.
Dan Hendrycks, Mantas Mazeika, Duncan Wilson, and Kevin Gimpel. Using trusted data to train deep networks labels corrupted by severe noise. In NearIPS, pp. 10477-10486, 2018.
Heinrich Jiang and Ofir Nachum. Identifying and correcting label bias in machine learning. CORR, abs/1901.00 004966, 2019.
Christopher Jung, Michael J. Kearns, Seth Neel, Aaron Roth, Logan Stapleton, and Zhiwei Steven Wu. Eliciting and enforcing subjective individual faimess. CoRR, abs/ (0s/1905.100 10660, 2019.
10

---

Under review as a conference paper at ICLR 2020
Faisal Kamiran and Toon Calders. Data preprocessing techniques for classification without discrim- ination. Knowl. Inf. Syst., 33(1):1-33, 2011.
Faisal Kamiran, Asim Karim, and Xiangliang Zhang. Decision theory for discrimination-awe classification. In ICDM, pp. 924 929, 2012.
Toshihiro Kamishima, Shotaro Akaho, Hideki Asoh, and Jun Sakuma. Fairness-aware classifier with prejudice remover regularizer. In ECML PKDD, pp- 35-50, 2012.
Andrej Karpathy.
Software 2.0.
https://neddummcommeBeee
software- 2- a64 5413644553b33333 2017.
Michael Kearns, Aaron Roth, and Saeed Sharif--Malvaerdii. Average individual fairness: Algo- rithms, generalization and experiments, 2019.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 1412.6980, 2014.
Jon M. Kleinberg, Sendhil Mullainathan, and Manish Raghavan. Inherent trade-offs in the fair determination of risk scores. In I7CS, pp. 43:1-43:23, 2017.
Pang Wei Koh, Jacob Steinhardt, and Perey Liang. Stronger data poisoning attacks break data sanitization defenses. CoRR, abs/1811 111111101741, 2018.
Ron Kohavi. Scaling up the accuracy of naive-bayes classifiers: decision-tree hybrid. In KDD, pp. 202-207, 1996.
Jakub Konečný, H Brendan McMahan, Felix X Yu, Peter Richtárik, Ananda Theertha Suresh, and Dave Bacon. Federated learning: Strategies for improving communication efficiency. arXiv preprint arXiv:1610.05492 2016.
Alexey Kurakin, Ian J. Goodfellow, and Samy Bengio. Adversarial machine learning scale. In ICLR, 2017.
Yuncheng Li, Jianchao Yang, Yale Song, Liangliang Cao, Jiebo Luo, and Li-Jia Li. Learning from noisy labels with distillation. In ICCV, pp. 1928-1936, 2017.
Jianhua Lin. Divergence measures based on the Shannon entropy. IEEE Transactions on Information theory, 37(1):145-151, 1991.
Nagarajan Natarajan, Inderjit S. Dhillon, Pradeep Ravikumar, and Ambuj Tewari. Learning with noisy labels. In NeurlPS, pp. 1196-1204, 2013.
Natasha Noy, Matthew Burgess, and Dan Brickley, Google dataset search: Building a search engine for datasets in an open web ecosystem. In 28th Web Conference (WebConf 2019), 2019.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang. Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in PyTorch. In NIPS Autodiff Workshop, 2017.
Andrea Paudice, Luis Muñoz-González, and Emil C. Lupu. Label sanitization against label flipping poisoning attacks. In ECML PKDD, pp. 5- 5-15, 2018.
Geoff Pleiss, Manish Raghavan, Felix Wu, Jon M. Kleinberg, and Kilian Q. Weinberger. On fairness and calibration. In NeurlPS, pp. 5684-5693, 2017.
Mengye Ren, Wenyuan Zeng, Bin Yang, and Raquel Urtasun. Learning to reweight examples for robust deep learning. In JCML, pp. 4331-4340, 2018.
Andreas Veit, Neil Alldrin, Gal Chechik, Ivan Krasin, Abhinav Gupta, and Serge J. Belongic. Learn- ing from noisy large-scale datasets with minimal supervision. In CVPR, pp. 6575-6583, 2017.
Suresh Venkatasubramamiann Algorithmic faimess: Measures, methods and representations. In PODS. pp. 481, 2019.
11

---

Under review as a conference paper at ICLR 2020
Sahil Verma and Julia Rubin. Faimesss definitions explained. In SeirWare@ICSE. pp. 1-7,
Eric Wong and J. Zico Kolter. Provable defenses against adversarial examples via the convex outer adversarial polytope. In ICML, pp. 5283-5292, 2018.
Tong Xiao, Tian Xia, Yi Yang, Chang Huang, and Xiaogang Wang. Learning from massive noisy labeled data for image classification. In CVPR, pp. 2691-2699, 2015.
Mikhail Yurochkin, Amanda Bower, and Yuekai Sun. Learning fair predictors with sensitive sub- space robustness. ArXiv, abs/1907 197000007000,0 2019.
Muhammad Bilal Zafar, Isabel Valera, Manuel Gomez-Rodrigue and Krishna P. Gummadi. Fair- ness constraints: Mechanisms for fair classification. In AISTATS, pp. 962-970, 2017.
Richard S. Zemel, Yu Wu, Kevin Swersky, Toniann Pitassi, and Cynthia Dwork. Learning fair representations. ICML, pp. 325-333, 2013.
Brian Hu Zhang, Blake Lemoine, and Margaret Mitchell. Mitigating unwanted biases with adver- sarial learning. In AIES, pp. 335-340, 2018.
12

---

Under review as a conference paper at ICLR 2020
A
APPENDIX
A. PROOF FOR THEOREM 1
Before we present the proof of the main theorem, we first recall our notation. Let (2(2) be the distribution of Z where 2 E Z and Z is the set of possible sensitive attribute values. Let YIZ
2 (-) and Y ~ Py() Then Py() = ExEZ Pz(x) Also, let Y Py(-).
For convenience, let us repeat the statement of Theorem 1 There:
I(Z;Y)= P.
max
Σ
(2(2)) [ogD.(()) + (Z).
=E2
We now prove the theorem.
Proof. Denote by D the collection of (2) for all possible values of z and 9. and by v the collec- tion of V2 for all values of g. We can construct the Lagrangian function as follows:
L(D,V) = Σ Pr
[og Dx + H(2) + Σ Vg
1- Σ D.(0)) (--D.(0)
:E2
DEY
2E2
We use the following KKT conditions:
VL(D,V) P2(x)
(0)
1/3 = =0,
v(0.2) E y x 2,
(T)
(0)
1 Σ P-())=0
VO E.
Solving the equations, obtain 0z= = Ps (0) for all Thus,
(1) = Pz(2) (o(z))PP1(00 pr
Py (0)
Putting this the above optimization.
P2(3)Pp11(8) (8)
Σ Pz (2(2))Pr1=
log Pg())
H())
2E2
(2())py1=())
1
(2(2))Pp1=
log Pp(
Σ
P2(2) log Pz (z)
ZEZ
REZ
log (9)
Pz (2(2))pp11
2EZ
P2(2) (P Pp(s)
sEZ
JSP2 (P
)= ) 1) -())))
Here, the second last equality is due to the definition of the generalized JS divergence, and the last equality is due to its equivalence to the mutual information (Lin 1991).
A.2 EXTENSIONS TO OTHER FAIRNESS MEASURES
The following theorem relates the conditional mutual information I(Z: to the solution of an optimization problem.
Theorem 2.
T(Z; rIY) (ain())-E D2 (0)=
ΣΣ
Pr.z (v. 2) [og (2)p()) H2
yEy SEZ
13

---

Under review as a conference paper at ICLR 2020
Recall that this conditional mutual information term be used to capture equalized odds, which is another important faimesss metric. We also note that the following theorem can be modified in straightforward manner so that can handle T(Z = 1), which can be used to capture equal opportunity.
We now prove the theorem.
Proof. Denote by D the collection of D2lu(0) for all possible values of (z, D. and y) and by the collection of Vygú for all values of and ÿ. We can construct the Lagrangian function as follows:
log P2)N(()) + H(ZIV) + ΣΣ Vu. (1
(1- ---Payi Polu(()
=(D,v)= ΣΣ Py,z(y, 2) E
(1-
ZE2
BEL vEy
2E2
We use the following KKT conditions:
@L(DDV) -y, 2)
Priv.(0)
Vyy. O,
v(O,0,2) x
0D:1p(0)
(9)
1 Σ
11 (0) = 0,
v(g,9) x)
#EZ
Solving the two equations, we obtain 2.8 = (y: for all (9.9) E x y Thus,
= (2) (0)
D (0) Priy(() Vy.DEyx
Putting this to the above optimization.
log PpI (r)
ΣΣ
Pr.z PP
H(((Y)
2E
Pe
ΣΣ VEY #E2
Pr.z(y: EPrin.
log (8)
Σ Σ
Pv.z (y) 1 P2))(=)
yEy 2E
log PYIN.((M)
Σ VEY #EZ
Pr.z(U: 2) Epr...
(P))())
log Priv.
Σ
Py (2) E
Pr)
wEY 2E2
log Prip.= (2)
Pr(v) Σ Pz N.
Pviu (V)
VEY
=E2
Er((v) Σ (ziv()) (Prin (P (K)
VEY
SEZ
Σ Py( JS (Pylero""..
Primei
VEY
Σ Pr(yM = v) ((2;
The third last equality is due the definition of the generalized JS divergence; the second last equality is due to its equivalence to the mutual information (Lin) 1991); and the last equality is due the definition of conditional mutual information.
We now discuss how to actually compute the mutual information. We compute the following empir- ical version using the example ((,(0))))) 209, ())
ΣΣ Pr:
Σ
log (2))(((0)) + (Z
Pap, Pabo (0)=
1:40
My.s
vEy 262
(i(y()) (()()))
14

---

Under review as a conference paper at ICLR 2020
Din(0)
Softmax
2, Y
X, Z
Generator (Classifier)
y
Discriminator for Fairness
D(Y)
Diamll))
Softmax (3]]
Di Dij)(())
Z. Y
Discriminator
D(X,
X v Zuaty Yoal
for Robustness
Figure 4: The architecture of FR-GAN for equalized odds.
Disparate Impact
Disparate Impact
clean data
poisoned data
0.5
0.8-
0.5
0.5
an
clean
poisoned data
Accuracy
Accuracy
(a) FAIRNESS CONSTRAINTS
(b) LABEL BIAS CORRECTION
Disparate Impact
1.0
Disparate Impact
0.8
0.7
0.6
0.7
clean data
0.5
clean data
poisoned data
poisoned data
0.76
0.78
0.50
0.82
0.84
0.86
8.80
0.82
U.8E
0.85
0.88
Accuracy
Accuracy
(c) ADVERSARIAL DEBIASING
(d) FR-GAN (Val. set size 10%: 0.15)
Figure 5: Accuracy-fairness tradeoff curves for poisoned synthetic data with label flipping (7%).
Now for sufficiently large value of m, my.s (y, Therefore, the above expression is approximated as:
ΣΣ Σ
log Dzly (7 HZIY).
max Da (W): (1)
ZEZ nig +0)))))))))))))
Hence, we can set L2 (i.e., the loss w.r.t. the fairness discriminator) to the above expression. The rest of the objective function is the Figure 4 shows the resulting FR-GAN architecture.
shows
A.3 ADDITIONAL EXPERIMENTS
A.3.1 SYNTHETIC DATA
We continue our experiments from Sections 3 and 5.1
Label flipping for data poisoning Figure 5a to 5d show the experimental results when the data is poisoned via label flipping. We used the same synthetic data, as illustrated in Figure la and flipped 7% of labels from the clean data to make a poisoned data. Figures 5a 5b and 5c show the accuracy-fairne fairness tradeoff curves of FAIRNESS CONSTRAINTS, LABEL BIAS CORRECTION, and ADVERSARIAL DEBIASING, respectively. While the tradeoff curves in Fig cooosiidrral shift to the left, Figure 5d shows that FR-GAN is noticeably robust against label flipping.
15

---

Under review as a conference paper at ICLR 2020
Table Accuracy and fairness performances of the meta learning method by Ren al. (2018) on the clean and poisoned synthetic datasets for different validation set sizes. We used the same 2- flipping poisoning attack described in Section 3
Data
Validation size
Disparate impact
Accuracy
Clean Data
10%
0.429
0.883
10%
0.386
0.884
%
0.401
0,886
Poisoned Data
1%
0.297
0.845
0.5 %
0.269
0.829
0.1 %
0.348
0.585
Disparate Impact
0.8
0.7
0.6
FR-GAN clean data
FR-GAN on poisoned data (val 0.5%)
AD on clean data
0.4
AD on poisoned data
0.70
0.75
0.50
0.85
Accuracy
Figure 6: Comparison between ADVERSARIAL DEBIASING (AD) and FR-GAN with small valida- tion set (0.5%) on synthetic data.
Meta learning with different validation set sizes Table 3able3sshows shows the accuracy and fairness results for the meta learning method by Ren et al. (2018) (ML) for different validation set sizes. We observe a drastic decrease of accuracy when the validation set size is 0.1 % of the training data.
Ablation study for fairness Figure shows the accuracy- fairness tradeoff curves for ADVER- SARIAL DEBIASING and FR-GAN on the synthetic dataset. ADVERSARIAL DEBIASING tradeoff curves are from Figure 3b, and the FR-GAN curve comes from Figure 3i We observe in Figure 6 that the accuracy-fairness tradeoff of FR-GAN using small value of X2 on the poisoned data is similar the results of fairness-only methods.
A.3.2 REAL DATA
We continue our experiments from Section 5.2
Label flipping for data poisoning Table uses the same experimental setting as in Table except that data poisoning label flipping (not w.r.t. sensitive attributes). We observe similar trends as in Table 2 For the non-fairness methods, the fairness results are significantly worse than other methods. For the fairness-only methods, both fairness and accuracy tend to be high on the clean data, but least one of them decreases significantly FR-GAN yields high fairness and accuracy on the clean data and is also the most resilient against poisoned data. The results emphasize our holistic technique that takes both fairness and accuracy into consideration.
the poisoned data. On the other hand,
A.3.3 FR-GAN USING OTHER FAIRNESS MEASURES
FR FR-GAN respects equalized odds and equal opportunity. Table 5 shows the experimental results the synthetic and real datasets for equalized odds. We that FR-GAN significantly improves equalized odds with reasonable accuracy. The results w.r.t. equal opportunity similar and thus shown here.
As we showed in Appendix A.2
16

---

Under review as a conference paper at ICLR 2020
Table 4: Accuracy and fairness performances on real data for disparate impact where the data is poisoned using label flipping attack (Paudice et al. 2018). Two types of methods are compared: (1) non fainnairness methods: logistic regression (LR), the meta learning by Ren et al. (2018) (ML), and R-GAN (FR-GAN with A 0); (2) fairness-only methods: FAIRNESS CONSTRAINTS ZZfaar et al. 2017) (FC), LABEL BIAS CORRECTION (Jiang & Nachum 2019 (LBC), and ADVERSARIAL DEBIASING (Zhang et al. 2018) (AD). Also "+ LD" denotes loss defense (Koh et al. 2018) (a data sanitization technique that is applied prior to model training). For FR-GAN, the validation set is 10% of ir For each fairness or accuracy result of the poisoned data, we make comparison with the clean data result and show the percentage increase or decrease.
Dataset
Type
Method
Clean data
Poisoned data
D. impact
Acc.
D. impact
Acc.
LR
0.465
0.674
0.856 (84. 1% f)
0.487 (27.7% J)
Non-fair
ML
0.493
0.680
0.608 (23.3% f)
0.629 (7.50% J)
R-GAN
0.574
0.683
0.919 (60.1%
0.621 (9.08% .)
FC
0.777
0.682
0.931 (19.8% f)
0.490 (28.2% J)
LBC
0.866
0,671
0.945 (9.12% f)
0.488 (27.3% ()
AD
0.846
0.680
0.941 (11.2% f)
0.615 (9.56% J)
COMPAS
Fair-only
FC LD
0.777
0.682
0.820 (5.53% f)
0.544 (20.2% ()
LBC + LD
0.866
0.671
0.928 (7.16% f)
0.550 (18.0%
AD + LD
0.846
0.680
0.822 (2.84% L)
0.614 (9.71% %
Both
FR-GAN
0.936
0.667
0.964 (2.99% f)
0.621 (6.90% J)
LR
0.328
0.847
0.316 (3.66% L)
0.824 (2.72% J)
Non-fair
ML
0.346
0.844
0.413 (19.4% f)
0.826 (2.13%
R-GAN
0.335
0.847
0.342 (2.09% f)
0.833 (2.13% .)
FC
0,825
0,826
0.814 (1.33% L)
0.802 (2.91% J)
LBC
0,825
0,825
0.734 (11.0% L)
0.819 (0.73% ()
AD
0.850
0.767
0.652 (23.3% L)
0.749 (2.35%
Adult
Fair-only
FC LD
0.825
0.826
0.762 (7.64% L)
0.831 (0.61% f)
LBC + LD
0.825
0.825
0.720 (12.7% L)
0.827 (0.24%
AD + LD
0.850
0.767
0.761 (10.5% ()
0.803 (4.69% 1)
Both
FR-GAN
0.870
0.819
0.859 (1.26% L)
0.789 (3. 33.66% J)
Table 5: Accuracy and fairness performances on synthetic and real datasets w.r.t equalized odds.
Two algorithms compared: (1) logistic regression (LR, non-fairness method) and (2) FR-GAN.
Dataset
Method
Equalized odds
Accuracy
Y 0
Y
LR
0.351
0.804
0.885
Synthetic Data
FR-GAN
0.888
0.936
0.865
LR 0.427
0.557
0.674
Compas
FR-GAN
0.718
0.959
0.628
LR 0.286
0.909
0.848
AdultCensus
FR-GAN
0.503
0.917
0.842
A.4 TRAINING METHODOLOGY
We implement all algorithms in PyTorch (Paszke et al. 2017 and all experiments performed on a server with Intel i7-6850 CPUs.
The generator G is neural network with zero one hidden layer. The discriminator D is single layer neural network, and the discriminator D is a neural network with one hidden layer. We used or 16 nodes in the hidden layers. We set Adam optimizer (Kingma & Ba 2014) for the generator,
17

---

Under review as a conference paper at ICLR 2020
Table Confusion matrix on poisoned AdultCensus dataset w.r.t. disparate impact. Two algorithms are compared: (1) AD and (2) FR-GAN. For a fair comparison, we use data sanitization before running AD.
Female
Male
Method
Prediction =0
Prediction
Prediction
Prediction =
True 0
2,502
566
4,330
583
AD LD
True
191
194
1,551
714
True
2,765
303
4,611
302
FR-GAN
True =
95
290
1,144
1,121
Table 7: Confusion matrix poisoned AdultCensus dataset w.r.t. equalized odds. Two algorithms are compared: (1) AD and (2) FR-GAN. For a fair comparison, we use data sanitization before running AD.
Female
Male
Method
Prediction =0
Prediction
Prediction 0
Prediction
True True
2,935
133
4.629
284
AD LD
249
136
1,590
675
True 0
2,859
209
4,480
443
FR-GAN
True = 1
116
269
960
1,305
and stochastic gradient descent (SGD) optimizer for each discriminator. We empirically observe that one can stabilize the training procedure by freezing the parameters of the fairness discriminator DP for the initial phase of training. Thus, we choose freeze the parameters of the fairness dis- criminator D for the first few epochs until the generator achieves a certain accuracy. We use the generator/diserimnnaao update ratio of 1:1 for the first 300 epochs and use the update ratio of 1:3 (or 1:5) for the rest of training.
Also, we use the following details for choosing the values of d₁ and Ag. For clean data, we set A2 as small value (e.g., 0.01) and vary A from 0 to 0.95. For poisoned data, we set X2 as 0.2, 0.3, or 0.4, and vary A from 0 to 0.95 Az. Given the values of X₁ and A2, we also normalize I, (the generator loss) by multiplying it with
A1 22).
A.5 ADDITIONAL EXPERIMENTS ON ADULT DATASET
We compare FR-GAN and AD using the AdultCensus dataset and the same poisoning in Section
3. For fair comparison, we use loss defense (LD) as data sanitization prior running AD. We generate confusion matrices both for disparate impact (DI) and equalized odds (EO), Overall, FR- GAN performs better than AD because of its robustness discriminator. The robustness discriminator successfully ignores the poisoned distribution in the training data, and PR-GAN'S TPR is higher than AD with sanitization. T confusion matrices as results of experiment. The results are reported when AD achieves (Acc, DI) (0.728, 0.821), and FR-GAN achieves (Acc, DI) = (0.827, 0.866). addition, Table T represents confusion matrices as results of equalized-odds-foceeee focused experiment. The results are reported when AD achieves (Ace, EO when y 1, EO when = 0) = (0.788, 0.844, 0.750), and FR-GAN achieves (Ace, EO when = 1, EO when y 0) (0.838, 0.824, 0.773).
18


Reviews:
[{"review_id": "ryg38nhhtS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces a new method for training a classifier that simultaneously optimizes for a fairness criterion and robustness to data poisoning. The method is shown to increase measures of fairness and reduce inaccuracy on poisoned data relative to classifiers that only consider accuracy or fairness. Extensive results are shown for both synthetic and real benchmark data sets.\n\nI would lean to reject for the following reasons: 1) the problem is not well-motivated. I would like a more clear example of some problem with sensitive attributes in which the data is publicly available and the providers of the data are motivated to falsify it. 2) the contribution is very simple and the individual pieces do not seem to be significant contributions. In particular, the use of GANs for fairness is previously done, and the use of the GAN for robustness here seems too simple to be broadly useful 3) the results are less convincing than they might otherwise be because none of the competing methods tested make use of a clean validation set 4) the paper is somewhat unpolished. I find the results difficult to read, although the arrows are helpful, and it is not clear to me whether these results are on a test set or the training set.\n\nLack of convincing tests for robustness: It is disappointing that FR-GAN does not offer any promises to be robust in general. Despite access to a clean validation set, the classifier is trained only to ignore the type of data poisoning that exists in the training set. If the test set were out of distribution in a different way relative to the training set, I see no reason to believe FR-GAN would protect against this. Furthermore, because it is not stated that these are test set results, I am not certain that they are not training set results, in which case some performance may be due to overfitting.\n\nMinor notes:\n\nIt would be nice for comparison if the charts had the same axes throughout.\n\nWhat are the numbers of nodes used in the hidden layers?", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}, {"review_id": "rklyVs7S5B", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes to use a GAN style approach for training a classifier that is robust to data poisoning and can achieve a pre-specified notion of group fairness.\n\nThe contribution of this paper is incremental in the context of prior Adversarial Debasing (AD) approach using essentially same GAN for group fairness and prior work presenting ideas of utilizing clean validation data to defend against data poisoning. This paper is proposing to add an additional discriminator to the AD approach that distinguishes training data and clean validation data. If the training data is poisoned, such distinguishment may be possible and maximizing loss of this discriminator can help to robustify against poisoned samples.\n\nExperimental results are insufficient to argue improvement over the AD. There are no AD results in the equalized odds Adult experiment in the supplement. I recommend more detailed comparison against the AD method (including results showing confusion matrices). Also note that AD, as presented in the original paper, is optimizing for demographic parity, but can also be adjusted to other group fairness metrics. Finally, in the context of Adult dataset, it is important to also report performance metrics such as balanced TPR since the labels are quite imbalanced.\n\nAre there any real data examples where poisoning does not need to be introduced artificially and the proposed method helps to improve the fairness properties? I think an interesting direction could be to consider data where labels are subjective. For example, a dataset on loan decisions can be naturally \"poisoned\" with human biases, i.e. information that someone did not receive the loan may be due to an error or bias of a human in charge of the decision making.\n\nLastly I think that the discussion of the prior related work on fairness is incomplete. This paper exclusively covers group fairness, which indeed has been shown to have some disadvantages. For example, prior work [1] has shown that some group fairness notions can not be satisfied simultaneously in certain cases. In this regard it is also important to report multiple group fairness metrics simultaneously in the experiments. The other fairness definition, that has not been mentioned in this paper, is individual fairness [2]. It has legal and intuitive interpretations. Multiple recent papers have explored the direction of individual fairness [3,4,5,6].\n\n[1] Kleinberg, J., Mullainathan, S., & Raghavan, M. (2016). Inherent trade-offs in the fair determination of risk scores.\n[2] Dwork, C., Hardt, M., Pitassi, T., Reingold, O., & Zemel, R. (2012, January). Fairness through awareness.\n[3] Garg, S., Perot, V., Limtiaco, N., Taly, A., Chi, E. H., & Beutel, A. (2019, January). Counterfactual fairness in text classification through robustness.\n[4] Yurochkin, M., Bower, A., & Sun, Y. (2019). Learning fair predictors with Sensitive Subspace Robustness.\n[5] Kearns, M., Roth, A., & Sharifi-Malvajerdi, S. (2019). Average Individual Fairness: Algorithms, Generalization and Experiments.\n[6] Jung, C., Kearns, M., Neel, S., Roth, A., Stapleton, L., & Wu, Z. S. (2019). Eliciting and Enforcing Subjective Individual Fairness.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}, {"review_id": "SJe-mj1AFH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper combines adversarial fair training with adversarial robust training. The basic idea is that a classifier is combined with two adversaries: one tries to predict the sensitive attribute $Z$ from the output of the classifier (essentially the approach by Edwards&Storkey 2015) and the other adversary tries to recognize if a label was predicted or is from a clean hold-out dataset. The latter is intended to harden the classifier against data-poisoning of the training set.\n\n---\n\nThe paper is clearly written and technically sound.\n\nThe fairness aspect of the proposed method is fairly standard and not very novel (going back to 2015). The robustness aspect is an interesting addition and seems novel, but I'm not sure if it's enough to get the paper accepted. If I understand it correctly, FR-GAN without the \"R\" part should just be equivalent to Adversarial Debiasing (Zhang et al., 2018). And if there is no data-poisoning, then the \"R\" part doesn't have any effect.\n\nThe fact that this is the first fairness-related method that additionally deals with robustness, makes it also difficult to judge the performance of the method. I would wish for a more appropriate baseline; one that makes use of the clean validation set somehow.\n\nThere might be synergistic effects where the robustness aspects helps the fairness aspect but this comes at the cost of needing a clean validation set (and it only matters with poisoned data).\n\nWhat is also missing is a motivating real-world example. When would you encounter flipped labels in the training set, but also have access to a clean validation set?\n\nMinor comments:\n\n- I did not understand what was meant by the phrase \"so that the model accuracy is reduced the most.\" in the first paragraph of section 3", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This manuscript proposes an approach for fair and robust training of predictive modeling -- both of which are implemented using adversarial methods, i.e., an adversarial loss for fairness and an adversarial loss for robustness. The resulting model is evaluated empirically and shown to improve fairness and robustness performance.

The reviewers and AC agree that the problem studied is timely and interesting, as there is limited work on joint fairness and robustness. However, the reviewers were unconvinced about the novelty and clarity of the conceptual and empirical results. In reviews and discussion, the reviewers also noted insufficient motivation for the approach. 
Decision: Reject