Title: FR-GAN: FAIR AND ROBUST TRAINING

Abstract: We consider the problem of fair and robust model training in the presence of data poisoning. Ensuring fairness usually involves a tradeoff against accuracy, so if the data poisoning is mistakenly viewed as additional bias to be fixed, the accuracy will be sacrificed even more. We demonstrate that this phenomenon indeed holds for state-of-the-art model fairness techniques. We then propose FR-GAN, which holistically performs fair and robust model training using generative adversarial networks (GANs). We first use a generator that attempts to classify examples as accurately as possible. In addition, we deploy two discriminators: (1) a fairness discriminator that predicts the sensitive attribute from classification results and (2) a robustness discriminator that distinguishes examples and predictions from a clean validation set. Our framework respects all the prominent fairness measures: disparate impact, equalized odds, and equal opportunity. Also, FR-GAN optimizes fairness without requiring the knowledge of prior statistics of the sensitive attributes. In our experiments, FR-GAN shows almost no decrease in fairness and accuracy in the presence of data poisoning unlike other state-of-the-art fairness methods, which are vulnerable. In addition, FR-GAN can be adjusted using parameters to maintain reasonable accuracy and fairness even if the validation set is too small or unavailable.

1. INTRODUCTION:

As machine learning becomes widespread in the Software 2.0 era (Karpathy, 2017) , there is an urgent need to address the issues of fairness and robustness. For sensitive applications like healthcare, finance, and self-driving cars, a trained model must not discriminate customers based on sensitive attributes including age, sex, or religion. In addition, as applications often rely on external data sets for their training data, the model training must be resilient against noisy or even adversarial data.

Traditionally, model fairness research (Venkatasubramanian, 2019; Chouldechova & Roth, 2018; Verma & Rubin, 2018) has focused on developing metrics such as disparate impact (Feldman et al., 2015) , equalized odds (Hardt et al., 2016) , and equal opportunity (Hardt et al., 2016) , which capture various notions of discrimination. More recently, there has been a surge in unfairness mitigation techniques (Bellamy et al., 2018) , which improve the model fairness by either fixing the training data, training process, or trained model. Unfairness mitigation usually involves some tradeoff between the model's accuracy and fairness. Most recently, generative adversarial networks (GANs) are being adapted to a fairness setting (Zhang et al., 2018) . The architecture of GANs is suitable because it can be used to compete the training between two objectives: accuracy and fairness.

Robust model training is also important and needs to be concurrently taken into consideration. There has been a proliferation of algorithms that make model training resilient against noisy and even adversarial data (Natarajan et al., 2013; Biggio et al., 2011; Frénay & Verleysen, 2014) . The latter scenario may occur when the source of the training data is external, and we cannot control the data that is being added. For example, any public dataset on the Web may contain intentionallyfalse information. Moreover, one must have a robust training algorithm if the data set comes from diverse users' inputs that cannot be fully trusted (Konečnỳ et al., 2016) . Recently, data poisoning attacks have become increasingly sophisticated where it is becoming difficult to sanitize the data against all of them ( Koh et al., 2018) . In addition, dataset searching is becoming mainstream as demonstrated by Google Dataset Search (Goods) (Halevy et al., 2016) and its public version for searching scientific datasets (Noy et al., 2019) . While data lakes within companies may consist of refined datasets, datasets in the public are easy to poison. And anyone can poison public datasets using attacks in the literature and share them. We thus believe that it is essential to address both bias and poisoning as a preventive measure.

Solving model fairness without addressing data poisoning may lead to a worse tradeoff between accuracy and fairness. For example, consider a banking system that is giving out loans where there are two sensitive groups: men and women. According to disparate impact, the two groups must have similar ratios of positive predictions. A perfect disparate impact would have a value of 1. Suppose the clean training data has no bias and a perfect (or high) disparate impact score. Now suppose this data is poisoned where half of the positive labels of men are flipped to negative labels. If the model is fitted so as to make more negative predictions on men than women, then disparate impact would worsen. To address this problem, one may suggest using data sanitization techniques prior to the model training, but it is challenging to do so without any knowledge of the model. Even though many data sanitization methods are proposed, there have been newer attacks that can easily evade such defenses (Koh et al., 2018) .

Our main contribution is the development of a new framework called FR-GAN, which trains accurate models that are also fair and robust to poisoning. FR-GAN consists of a generator used for classification, a discriminator that distinguishes predictions from one sensitive group against others, and a second discriminator that distinguishes {examples, predictions} of the training data from {examples, labels} of a separate and clean validation set. The first discriminator ensures that the prediction ŷ is independent of the sensitive attribute z. We show that variations of this approach can be used to maximize any of the following prominent fairness measures: disparate impact, equalized odds, and equal opportunity. The second discriminator ensures that the model predictions on the training data are "consistent" with labels on clean data. We theoretically show that FR-GAN optimizes fairness without requiring the knowledge of prior statistics of sensitive attributes. In addition, the parameters of FR-GAN can be adjusted to maintain reasonable accuracy and fairness even if the validation set is too small or unavailable. In the following sections, we present the related work, demonstrate the weaknesses of current fairness methods, and propose FR-GAN.

2. RELATED WORK:



Model fairness:

The notion of discrimination has many definitions and usually comes from certain social goals that one wants to guarantee. Fairness research has been focused on defining fairness measures that capture a variety of notions of fairness (Verma & Rubin, 2018) . There are largely two types of fairness measures: (1) group fairness (Barocas & Selbst, 2016; Hardt et al., 2016) , which ensures similar statistics between two sensitive groups; (2) individual fairness (Dwork et al., 2012) , which guarantees similar prediction results across nearby examples. Group fairness measures are widely studied in general applications, even though these cannot be simultaneously satisfied in certain circumstances (Kleinberg et al., 2017) . In addition, there are also recent significant works that focus on individual fairness (Garg et al., 2018; Yurochkin et al., 2019; Kearns et al., 2019; Jung et al., 2019) . Among the two types of fairness with different aspects, we focus on group fairness as our initial research. Recently, there has also been a surge of research on unfairness mitigation techniques (Bellamy et al., 2018) . Depending on where a fix occurs, there are mainly three approaches:

(1) pre-processing techniques (Kamiran & Calders, 2011; du Pin Calmon et al., 2017; Zemel et al., 2013; Feldman et al., 2015) that fix the training data; (2) in-processing techniques (Zafar et al., 2017; Jiang & Nachum, 2019; Zhang et al., 2018; Kamishima et al., 2012; Cotter et al., 2019; 2018; Agarwal et al., 2018) that address the issue during model training; and (3) post-processing techniques (Hardt et al., 2016; Pleiss et al., 2017; Kamiran et al., 2012) that manipulate predictions while maintaining the model. Among the three, the in-processing techniques have the advantages that one can work with any data and that there is more control on model training (Venkatasubramanian, 2019) . FAIRNESS CON-STRAINTS (Zafar et al., 2017) incorporates a regularization term that reflects fairness constraints in the context of convex margin-based classifiers such as logistic regression and support vector machines (SVMs). The main hyperparameter is the λ term, which serves to balance the accuracy and fairness objectives. LABEL BIAS CORRECTION (Jiang & Nachum, 2019) DEBIASING (Zhang et al., 2018) trains a classifier so that an adversary cannot predict the sensitive attributes based on the output of the classifier. Hence, the classifier achieves fairness when its predictions do not have any correlation with sensitive attributes. This idea of adversarial training is similar to GANs (Goodfellow et al., 2014) . ADVERSARIAL DEBIASING provides insights of using adversarial training to minimize mutual information between the prediction and sensitive attributes.

In this paper, we strengthen the theoretical results of ADVERSARIAL DEBIASING using information theory, and this motivates us to provide systematic methodology for various fairness metrics (Section 4.1).

As we demonstrate in Section 3, the existing techniques are not tailored for robust training, so are vulnerable to data poisoning attacks. In comparison, FR-GAN addresses both model fairness and robust training within the same model training process.

Robust training There is a heavy literature on how to make the model training robust against noisy or even adversarial data (Natarajan et al., 2013; Biggio et al., 2011; Frénay & Verleysen, 2014; Kurakin et al., 2017) . A major challenge is that there can be a wide range of data poisoning attacks that keep on evolving. One defense approach is to sanitize the training data and thus fix the root cause. A fundamental limitation is that we do not know in general what kind of poisoning will occur, so a universal defense against all possible attacks does not seem feasible as demonstrated by Koh et al. (2018) . A more recent trend is to develop general defense algorithms for any attack during model training using meta learning (Veit et al., 2017; Li et al., 2017; Xiao et al., 2015; Hendrycks et al., 2018) . Ren et al. (2018) propose a meta-learning framework that employs a clean validation set for the purpose of preventing the model training from being influenced by poisoned data. By using the validation loss as a meta objective, it intends to defend against any poisoning attack. Our FR-GAN framework is inspired by this approach, but employs a GAN-based model to support fair and robust training without using meta learning. In particular, the design of FR-GAN's robustness discriminator is based on mutual-information-based theoretical insights (Section 4.2). Another line of research is defending against adversarial attacks during test time (Biggio et al., 2013; Goodfellow et al., 2015; Wong & Kolter, 2018) . In comparison, our focus is on defending against data poisoning on the training data.

3. VULNERABILITY OF FAIRNESS METHODS:

We perform experiments to demonstrate that state-of-the-art fairness methods are indeed vulnerable even to simple poisoning attacks. We generate a synthetic dataset as shown in Figure 1a (see the generation details in Section 5.1). There are two non-sensitive attributes a and b, which are reflected in the x-axis and y-axis, respectively. The examples are further divided into two classes based on the sensitive attribute z. To generate poisoned data, we measure accuracy degradation for a flipping of each z. We then choose the top-10% of the z values with the highest degradation and flip them. This approach is similar to the existing label flipping method (Paudice et al., 2018) , except that the z value is being flipped instead of the label so that the model fairness is also affected. To make a validation set, we randomly select clean examples that amount to 10% of the entire training data.

We use disparate impact (Feldman et al., 2015) as a fairness measure and evaluate one fairness method, called FAIRNESS CONSTRAINTS (Zafar et al., 2017) . As this method involves a regularization factor λ that balances the accuracy and fairness objectives, we can obtain a tradeoff curve by adjusting its value. Figure 1b shows two accuracy-fairness tradeoff curves obtained with the clean and poisoned synthetic datasets. Notice that adding data poisoning clearly shifts the curve to the left, which means both accuracy and fairness decrease. This coincides with our intuition. The poisoning confuses the model so that there are more biased examples to fix, which in turn makes it overreact and thus sacrifice more on accuracy. In the supplementary, we show similar trends when the data is poisoned via label flipping. In Section 5, we will show how data poisoning affects LABEL BIAS CORRECTION (Jiang & Nachum, 2019) and ADVERSARIAL DEBIASING (Zhang et al., 2018) .

4. FR-GAN:

We now describe FR-GAN, which is shown in Figure 2 . Unlike traditional GANs, the generator is a classifier that receives an example x ∈ X and returns a prediction ŷ. The other two discriminators optimize fairness and robustness as we describe in the following sections.

4.1. FAIRNESS:

We denote by D tr the training data set. Suppose D tr has m examples {(x (i) , z (i) , y (i) )} m i=1 where x (i) contains the non-sensitive attributes, z (i) contains the sensitive attributes, and y (i) is the label. Both the sensitive attribute and label can be multi-class, i.e., they can have one of multiple values. For notational simplicity, we assume there is one sensitive attribute, which can be viewed as a merged result of multiple sensitive attributes with a larger alphabet size. For illustrative purposes, we focus on disparate impact, leaving in the supplementary our formulation for equalized odds and equal opportunity.

The first discriminator in FR-GAN distinguishes predictions w.r.t. one sensitive group from those in the others. Disparate impact intends the sensitive attribute to be independent of the model's prediction, i.e., I(Z; Ŷ ) = 0.

We explain how FR-GAN can enforce the above constraint. Let P Z (z) be the distribution of Z where z ∈ Z and Z is the set of possible sensitive attribute values. Let

Ŷ |Z = z ∼ P Ŷ |z (•) and Ŷ ∼ P Ŷ (•). Then P Ŷ (•) = z∈Z P Z (z)P Ŷ |z (•).

We show that mutual information is equivalent to the following expression where the optimal discriminator D z (ŷ) =

P Ŷ ,Z (ŷ,z) P Ŷ (ŷ) and z∈Z D z (ŷ) = 1 ∀ŷ ∈ Y. Theorem 1. I(Z; Ŷ ) = max Dz(ŷ): z Dz(ŷ)=1∀ŷ z∈Z P Z (z)E P Ŷ |z log D z ( Ŷ ) + H(Z).

While deferring the detailed proof to the supplemental materials, we provide a brief overview of the proof. As the optimization problem in the RHS is convex, we find the optimal dicriminator by solving the KKT conditions. We then show that the maximum value attained by the optimal discriminator is equal to the mutual information by using the properties of mutual information and the generalized Jensen-Shannon divergence (Lin, 1991) .

What is more involved than showing the above equality is designing a right optimization problem. One needs to carefully handcraft a plausible optimization problem so that its unique solution matches the desired quantity. Here, we design the above optimization problem via a 'guess-&-check' approach aided by the structural insights across the KL divergences that appear in an alternative expression of mutual information.

We now discuss how to implement the above expression. Since we do not know P Ŷ |z (•) exactly, we compute the following empirical version:

max Dz(ŷ): z Dz(ŷ)=1∀ŷ z∈Z P Z (z) i:z (i) =z 1 m z log D z (ŷ (i) ) + H(Z).

Now for sufficiently large m, the number m z of examples with z (i) = z is approximately the same as P Z (z)m. Therefore, the above expression becomes:

max Dz(ŷ): z Dz(ŷ)=1∀ŷ z∈Z i:z (i) =z 1 m log D z (ŷ (i) ) + H(Z).

Interestingly, this formulation is exactly the same as that in the original GAN (Goodfellow et al., 2014) when |Z| = 2. We also remark that our formulation does not require a prior knowledge on P Z (z).

4.2. ROBUSTNESS:

We utilize a clean yet small validation set to ensure robust training. In particular, we add a second discriminator that distinguishes the training data with predictions {(

x (i) , z (i) , ŷ(i) )} m i=1 from the validation set {(x (i) val , z (i) val , y (i) val )} m val i=1 .

Intuitively, if the classifier is confused by data poisoning in the training data, then its predictions will not be consistent with the labels of the clean data, and the discriminator would be able to detect that difference. While using a validation set is inspired by the meta learning approach by Ren et al. (2018) , we take an adversarial learning approach. A key motivation behind this choice is that our GAN approach introduces a knob that controls the emphasis of robust training. We find that this knob enables FR-GAN to be more robust to the validation set size. See details in Section 5.1.

Our formulation is as follows. Let V ∼ Bern(β) denote whether an example comes from training data (V = 1) or from the validation set (V = 0). We then want to ensure that V is independent of the training data and predictions, i.e., I(V ; X, Z, Ŷ ) = 0, which means the predictions on the training data are indistinguishable from labels of the validation set. This way we can mimic the clean data set while expecting an indirect sanitization effect. We optimize this independence using a formulation similar to that of fairness. Another possible formulation is to use the hard decision value from D( Ŷ ), say Ẑ, instead of Z. However, this discriminator can only be trained after the fairness discriminator is trained. Instead, we take an approach that allows us to train the two discriminators at the same time.

4.3. ARCHITECTURE:

Figure 2 illustrates the architecture of FR-GAN. For the loss function in the generator, we employ the cross entropy loss:

L 1 = 1 m m i=1 -y (i) log ŷ(i) -(1 -y (i) ) log(1 -ŷ(i) ).

As per Theorem 1, we set the loss function w.r.t. the fairness discriminator as:

L 2 = max D(•) z∈Z i:z (i) =z 1 m log D z (ŷ (i) ) + H(Z)

where D(•) := (D 1 (•), . . . , D |Z| (•)). The condition z∈Z D z ( Ŷ ) = 1 can be enforced by adding a softmax layer to the discriminator. Finally, implementing I(V ; X, Z, Ŷ ) as advised by Theorem 1, we set the loss function w.r.t. the robustness discriminator as:

L 3 = max D r (•) i:v (i) =0 1 m log D r (x (i) , z (i) , ŷ(i) ) + i:v (i) =1 1 m log(1 -D r (x (i) , z (i) , ŷ(i) )) + H(V ).

The final objective function is to minimize the weighted sum of these value functions:

min G(•) L 1 + λ 1 L 2 + λ 2 L 3 .

Here λ 1 and λ 2 are tuning knobs that play roles to emphasize fair and robust training, respectively.

5. EXPERIMENTS:

We provide experimental results for disparate impact, while leaving in the supplementary those for equalized odds and equal opportunity. All of our results are on a separate test set. More implementation details are also in the supplementary.

5.1. SYNTHETIC DATA RESULTS:

For the synthetic data, we generate 2,000 examples with two non-sensitive attributes a and b, a sensitive attribute z, and a label y, using a method similar to the algorithm proposed by Zafar et al. (2017) . Both z and y are binary and generated uniformly at random. For the four possible (z, y) combinations, we generate different normal distributions for a and b. Finally for each example, the a and b values are sampled as per the normal distribution associated with the (z, y) pair. For data poisoning, we flip the z values of examples so as to lower the model's accuracy the most as described in Section 3.

To make a fair comparison in the context of robust training, we also improve the existing fairness methods with data sanitization and compare them with FR-GAN. Since FR-GAN is one of the first works to address both fairness and robustness, there are few baselines to compare with. Hence, we employed one reasonable baseline, which first sanitizes the poisoned data using a well-known sanitization technique and then performs each fairness algorithm. are not well fit by the model and result in a high loss are considered poisoned and are subsequently discarded. Loss defense is the only method that utilizes the trained model and thus has the best chance of defending poisoning attacks.

Accuracy and fairness:

We compare FR-GAN with FAIRNESS CONSTRAINTS (Zafar et al., 2017) , LABEL BIAS CORRECTION (Jiang & Nachum, 2019) , ADVERSARIAL DEBIASING (Zhang et al., 2018) , and a plain non-fairness method, logistic regression. We also compare with the meta learning approach by Ren et al. (2018) , which focuses on robust training. For this comparison, we tailor our framework to a setting where λ 1 = 0 (no fairness constraint), and name it R-GAN. For FR-GAN, we use a validation set that amounts to 10% of D tr . When setting λ 1 and λ 2 , we usually fix λ 2 to some value and then adjust λ 1 . For the clean data, we apply proper hyperparameters so that the disparate impacts are similar (around 0.8) across all distinct methods. We do the same for the poisoned data. There is no such hyperparameter tuning for logistic regression and the meta learning by Ren et al. (2018) , as the frameworks have no knob for adjusting fairness. We find that this leads to poor disparate impacts. For the three fairness methods, data poisoning aggravates performances in accuracy, fairness or both. For example, the disparate impact and accuracy of FAIRNESS CON-STRAINTS fall by 8.03% and 18.7%, respectively. On the other hand, the performance changes for FR-GAN are negligible: disparate impact increases by 2.09%, and accuracy decreases by 0.12%. Table 1 also shows that combining the fairness methods with loss defense (rows 6-8) does not always yield better accuracy and fairness. In fact, using the loss defense may lower the accuracy or fairness (e.g., LBC+LD has a disparate impact of 0.513 on poisoned data while LBC has 0.816).

Perhaps this is because the data sanitization sometimes mistakenly discards clean data as well. The results demonstrate that even a data sanitization technique may not help much. In the supplementary, we also perform experiments using equalized odds and show that FR-GAN has similar benefits.

We observe how accuracy trades off with fairness on clean and poisoned datasets. The results for FAIRNESS CONSTRAINTS are shown in Figure 1b . For LABEL BIAS CORRECTION, we employ the number of training iterations as a knob to trade accuracy off fairness. As shown in Figure 3a , the tradeoff curve shifts to the left, which demonstrates a clear tradeoff degradation. For ADVERSARIAL DEBIASING, we employ the α parameter (Zhang et al., 2018) analogous to λ 1 as a knob to trade accuracy off fairness. We see in Figure 3b that the tradeoff curve again shifts to the left.

Validation set size Figures 3c to 3e show how the validation set size affects the robustness of FR-GAN. In particular, we compare the accuracy-fairness tradeoff of FR-GAN on clean data and that on poisoned data while varying the size of the validation set. When running on poisoned data, we fixed λ 2 = 0.2 and varied λ 1 . We see that even 2% validation set (Figure 3d ) is sufficient to maintain the accuracy and fairness obtained on the clean data. When using 0.5% (Figure 3e ), the validation set is too small and has an adverse effect on the training. However, our framework includes a tuning knob λ 2 , so by decreasing λ 2 down to 0.1, we could de-emphasize robust training, thereby avoiding the adverse effect (Figure 3f ). This is in contrast to the meta learning approach, which suffers from a non-negligible performance degradation for a very small validation set. See details in the supplementary. The results show that only a holistic framework like FR-GAN can achieve both excellent model fairness and training robustness. In comparison, other methods tailored for only one of these objectives lose either accuracy, fairness or both.

Ablation study (Without 'R') For a small value of λ 2 , we observe in Figure 3f that the accuracyfairness tradeoff curve shifts to the left, just like other fairness-only methods. To make the comparison clearer, we also plot them together in the supplementary. (Without 'F') When λ 1 = 0, FR-GAN is expected to behave similarly to the other non-fair algorithms. The performance of FR-GAN with λ 1 = 0, which we dub as R-GAN, is reported in Table 1 . One can observe that R-GAN achieves a poor fairness performance just like the other non-fair algorithms.

5.2. REAL DATA RESULTS:

We use the following benchmark datasets: ProPublica COMPAS (Angwin et al., 2016) and Adult Census (Kohavi, 1996) . For sensitive attributes, we use SEX for both datasets. We do not employ the German Credit data set (Dua & Graff, 2017) as its highly biased towards positive labels. With such dataset, it is nearly impossible to observe any meaningful tradeoff.

For data poisoning, we use the same method employed on synthetic data: flipping the z values so as to maximize the accuracy performance degradation. While there may be stronger attacks, this poisoning attack is effective enough to fool the three fairness methods. In the supplementary, we also evaluate FR-GAN against label flipping attacks and show similar results. We use the same constructions for the validation set and poisoning as described in Section 3.

Accuracy and fairness See Table 2 . As in Table 1 , we apply proper hyperparameters so that disparate impacts are similar across all distinct methods, both for the clean and poisoned datasets.

The results are similar to Table 1 : logistic regression and meta learning exhibit poor disparate impacts; the three fairness methods have worse disparate impact and accuracy due to data poisoning; and FR-GAN again shows little degradation both in fairness and accuracy. Table 2 also shows that combining the fairness methods with loss defense (rows 6-8 and 15-17) does not always yield better accuracy and fairness and may even lower them, which is consistent to the results on synthetic data.

6. DISCUSSION:

In this section, we discuss possible ways to extend FR-GAN.

Subjective labels FR-GAN can be generalized to a setting where labels are subjective instead of being clean or poisoned. Here the dataset without the undesirable human biases becomes the clean validation set. In general, given two datasets with different distributions where one of them is desirable, FR-GAN can train robustly against the other distribution.

Constructing real datasets:

We propose a general methodology for constructing a real dataset that has both bias and poisoning, as well as a "clean" validation set. Suppose that we are performing evaluation tasks (say loan decisions) where there is a high chance of human bias (i.e., poisoning) due to the high workload. We can construct a clean validation set by selecting a few tasks, assigning more evaluators, and taking a majority vote of the evaluations to minimize the bias. While employing more evaluators can be expensive, constructing a small validation set is sufficient for robust training, which makes this method practical.

Generalizing to all attacks An interesting question is whether FR-GAN can defend against all poisoning attacks. If we know all the possible attacks, we can construct a training set containing these attacks. In the more challenging case where we do not know which attacks even exist, there seems to be a fundamental limitation in protecting against the attacks. Generalization is a universal problem in machine learning where a model trained on one dataset is not guaranteed to perform well in another dataset with a different distribution. Although the generalization is a critical issue to address, it is currently beyond the scope of this paper. 

7. CONCLUSION:

We proposed FR-GAN, which is a holistic framework that enables both unfairness mitigation and robust training. Our key contribution is proposing a novel GAN architecture that enjoys the synergistic effect of combining two approaches: (1) employing a fairness discriminator that distinguishes predictions w.r.t. one sensitive group from others and (2) employing a robust discriminator that distinguishes training data with predictions from a clean yet small validation set. We showed that existing fairness methods are vulnerable to data poisoning, even when combined with data sanitization techniques. In comparison, FR-GAN is robust to the poisoning and can be adjusted to maintain reasonable accuracy and fairness even if the validation set is too small or unavailable. We showed that FR-GAN is guaranteed to maximize fairness and supports all the prominent fairness measures: disparate impact, equalized odds, and equal opportunity.

A APPENDIX:

A.1 PROOF FOR THEOREM 1

Before we present the proof of the main theorem, we first recall our notation. Let P Z (z) be the distribution of Z where z ∈ Z and Z is the set of possible sensitive attribute values. Let Ŷ |Z = z ∼ P Ŷ |z (•) and Ŷ ∼ P Ŷ (•). Then P Ŷ (•) = z∈Z P Z (z)P Ŷ |z (•). Also, let Y ∼ P Y (•).

For convenience, let us repeat the statement of Theorem 1 here:

I(Z; Ŷ ) = max Dz(ŷ): z Dz(ŷ)=1,∀ŷ z∈Z P Z (z)E P Ŷ |z log D z ( Ŷ ) + H(Z).

We now prove the theorem.

Proof. Denote by D the collection of D z (ŷ) for all possible values of z and ŷ, and by ν the collection of ν ŷ for all values of ŷ. We can construct the Lagrangian function as follows:

L(D, ν) = z∈Z P Z (z)E P Ŷ |z log D z ( Ŷ ) + H(Z) + ŷ∈Y ν ŷ 1 - z∈Z D z (ŷ) .

We use the following KKT conditions:

∂L(D, ν) ∂D z (ŷ) = P Z (z) P Ŷ |z (ŷ) D z (ŷ) -ν ŷ = 0, ∀(ŷ, z) ∈ Y × Z, 1 - z∈Z D z (ŷ) = 0, ∀ŷ ∈ Y.

Solving the two equations, we obtain ν ŷ = P Ŷ (ŷ) for all ŷ. Thus, D z (ŷ) = P Z (z)P Ŷ |z (ŷ)

P Ŷ (ŷ) .

Putting this to the above optimization, Here, the second last equality is due to the definition of the generalized JS divergence, and the last equality is due to its equivalence to the mutual information (Lin, 1991) .

A.2 EXTENSIONS TO OTHER FAIRNESS MEASURES:

The following theorem relates the conditional mutual information I(Z; Ŷ |Y ) to the solution of an optimization problem. Recall that this conditional mutual information term can be used to capture equalized odds, which is another important fairness metric. We also note that the following theorem can be modified in a straightforward manner so that it can handle I(Z; Ŷ |Y = 1), which can be used to capture equal opportunity.

We now prove the theorem.

Proof. Denote by D the collection of D z|y (ŷ) for all possible values of (z, ŷ, and y) and by ν the collection of ν y,ŷ for all values of y and ŷ. We can construct the Lagrangian function as follows:

L(D, ν) = y∈Y z∈Z P Y,Z (y, z)E P Ŷ |y,z log D z|y ( Ŷ ) + H(Z|Y ) + ŷ∈Y y∈Y ν y,ŷ 1 - z∈Z D z|y (ŷ) .

We use the following KKT conditions:

∂L(D, ν) ∂D z|y (ŷ) = P Y,Z (y, z) P Ŷ |y,z (ŷ) D z|y (ŷ) -ν y,ŷ = 0, ∀(ŷ, y, z) ∈ Y × Y × Z 1 - z∈Z D z|y (ŷ) = 0, ∀(ŷ, y) ∈ Y × Y.

Solving the two equations, we obtain ν y,ŷ = P Y, Ŷ (y, ŷ) for all (y, ŷ) ∈ Y × Y. Thus, D z|y (ŷ) = P Z|y (z)P Ŷ |y,z (ŷ)

P Ŷ |y (ŷ) , ∀y, ŷ ∈ Y × Y.

Putting this to the above optimization, The third last equality is due to the definition of the generalized JS divergence; the second last equality is due to its equivalence to the mutual information (Lin, 1991) ; and the last equality is due to the definition of conditional mutual information.

We now discuss how to actually compute the mutual information. We compute the following empirical version using the example {(x (i) , z (i) , y Now for a sufficiently large value of m, m y,z ≈ P Y,Z (y, z)m. Therefore, the above expression is approximated as:

max D z|y (ŷ): z∈Z D z|y (ŷ)=1;∀ŷ y∈Y z∈Z i:(y (i) ,z (i) )=(y,z) 1 m log D z|y (ŷ (i) ) + H(Z|Y ).

Hence, we can set L 2 (i.e., the loss w.r.t. the fairness discriminator) to the above expression. The rest of the objective function is the same. Figure 4 shows the resulting FR-GAN architecture.

A.3 ADDITIONAL EXPERIMENTS:



A.3.1 SYNTHETIC DATA:

We continue our experiments from Sections 3 and 5.1.

Label flipping for data poisoning Figure 5a to 5d show the experimental results when the data is poisoned via label flipping. We used the same synthetic data, as illustrated in Figure 1a , and flipped 7% of labels from the clean data to make a poisoned data. Figures 5a, 5b , and 5c show the accuracy-fairness tradeoff curves of FAIRNESS CONSTRAINTS, LABEL BIAS CORRECTION, and ADVERSARIAL DEBIASING, respectively. While the tradeoff curves in Figure 5a to 5c considerably shift to the left, Figure 5d shows that FR-GAN is noticeably robust against label flipping. Meta learning with different validation set sizes Table 3 shows the accuracy and fairness results for the meta learning method by Ren et al. (2018) (ML) for different validation set sizes. We observe a drastic decrease of accuracy when the validation set size is 0.1% of the training data.

Ablation study for fairness Figure 6 shows the accuracy-fairness tradeoff curves for ADVER-SARIAL DEBIASING and FR-GAN on the synthetic dataset. ADVERSARIAL DEBIASING tradeoff curves are from Figure 3b , and the FR-GAN curve comes from Figure 3f . We observe in Figure 6 that the accuracy-fairness tradeoff of FR-GAN using a small value of λ 2 on the poisoned data is similar to the results of fairness-only methods.

A.3.2 REAL DATA:

We continue our experiments from Section 5.2.

Label flipping for data poisoning Table 4 uses the same experimental setting as in Table 2 , except that data poisoning is label flipping (not w.r.t. sensitive attributes). We observe similar trends as in Table 2 . For the non-fairness methods, the fairness results are significantly worse than other methods. For the fairness-only methods, both fairness and accuracy tend to be high on the clean data, but at least one of them decreases significantly on the poisoned data. On the other hand, FR-GAN yields high fairness and accuracy on the clean data and is also the most resilient against poisoned data. The results emphasize our holistic technique that takes both fairness and accuracy into consideration.

A.3.3 FR-GAN USING OTHER FAIRNESS MEASURES:

As we showed in Appendix A.2, FR-GAN respects equalized odds and equal opportunity. Table 5 shows the experimental results on the synthetic and real datasets for equalized odds. We see that FR-GAN significantly improves equalized odds with reasonable accuracy. The results w.r.t. equal opportunity are similar and thus not shown here.

Table 4 : Accuracy and fairness performances on real data for disparate impact where the data is poisoned using a label flipping attack (Paudice et al., 2018) . Two types of methods are compared:

(1) non-fairness methods: logistic regression (LR), the meta learning by Ren et al. (2018) We implement all algorithms in PyTorch (Paszke et al., 2017) , and all experiments are performed on a server with Intel i7-6850 CPUs.

The generator G is a neural network with zero or one hidden layer. The discriminator D f is a single layer neural network, and the discriminator D r is a neural network with one hidden layer. We used 8 or 16 nodes in the hidden layers. We set an Adam optimizer (Kingma & Ba, 2014) for the generator,

:



Figures and Tables:

Figure1: The left figure shows a synthetic dataset with a certain poisoning. Examples are divided into z = 0 (marked with circles) and z = 1 (crosses) as per a sensitive attribute z. The blue points indicate positive labels while the red points denote negative ones. For data poisoning, we flipped z values that reduce the model accuracy the most (similar to label flipping(Paudice et al., 2018), but specialized for fairness). The right figure shows that poisoning significantly worsens the accuracyfairness tradeoff (i.e., shifts to the left) of the FAIRNESS CONSTRAINTS method(Zafar et al., 2017).

Figure 2: The architecture of FR-GAN.

Figure 3: Accuracy-fairness tradeoff curves.

Z (z)E P Ŷ |z log P Z (z)P Ŷ |z ( Ŷ )P Ŷ ( Ŷ ) + H(Z) = z∈Z P Z (z)E P Ŷ |z log P Z (z)P Ŷ |z ( Ŷ ) z)D KL (P Ŷ |z P Ŷ ) JS P Z (P Ŷ |z1 , . . . , P Ŷ |z |Z| ) = I(Z; Ŷ ).

Theorem 2. I(Z; Ŷ |Y ) = max D z|y (ŷ): z∈Z D z|y (ŷ)=1;∀ŷ y∈Y z∈Z P Y,Z (y, z)E P Ŷ |y,z log D z|y ( Ŷ ) + H(Z|Y ).

Z (y, z)E P Ŷ |y,z log P Z|y (z)P Ŷ |y,z ( Ŷ ) P Ŷ |y ( Ŷ ) + H(Z|Y ) = y∈Y z∈Z P Y,Z (y, z)E P Ŷ |y,z log P Z|y (z)P Ŷ |y,z ( Ŷ ) Z (y, z)E P Ŷ |y,z log P Ŷ |y,z ( Ŷ ) P Ŷ |y ( Ŷ ) = y∈Y z∈Z P Y (y)P Z|y (z)E P Ŷ |y,z log P Ŷ |y,z ( Ŷ ) P Ŷ |y ( Ŷ ) = y∈Y P Y (y) z∈Z P Z|y (z)E P Ŷ |y,z log P Ŷ |y,z ( Ŷ ) Z|y (z)D KL (P Ŷ |y,z P Ŷ |y ) y∈Y P Y (y) • JS P Z|y (P Ŷ |z1,y , . . . , P Ŷ |z |Z| ,y ) = y∈Y P Y (y)I(Z; Ŷ |Y = y) = I(Z; Ŷ |Y ).

(i) )} m i=1 . max D z|y (ŷ): z∈Z D z|y (ŷ)=1;∀ŷ y∈Y z∈Z P Y,Z (y, z) i:(y (i) ,z (i) )=(y,z)1 m y,z log D z|y (ŷ (i) ) + H(Z|Y ).

Figure 4: The architecture of FR-GAN for equalized odds.

Figure 5: Accuracy-fairness tradeoff curves for poisoned synthetic data with label flipping (7%).

Figure 6: Comparison between ADVERSARIAL DEBIASING (AD) and FR-GAN with small validation set (0.5%) on synthetic data.

Accuracy and fairness performances on the clean and poisoned synthetic datasets w.r.t. disparate impact. Two types of methods are compared: (1) non-fairness methods: logistic regression (LR), the meta learning byRen et al. (2018) (ML), and R-GAN (FR-GAN with λ 1 = 0);(2) fairness-only methods: FAIRNESS CONSTRAINTS (Zafar et al., 2017) (FC), LABEL BIAS COR-RECTION(Jiang & Nachum, 2019) (LBC), and ADVERSARIAL DEBIASING (Zhang et al., 2018) (AD). Also "+ LD" denotes loss defense(Koh et al., 2018) (a data sanitization technique that is applied prior to model training). For FR-GAN, the validation set is 10% of D tr . For each fairness or accuracy result of the poisoned data, we make a comparison with the clean data result and show the percentage increase or decrease.

Table content: <table><tr><td>Type</td><td>Method</td><td colspan="2">Clean data</td><td>Poisoned data</td></tr><tr><td/><td/><td colspan="2">D. impact Acc.</td><td>D. impact</td><td>Acc.</td></tr><tr><td/><td>LR</td><td>0.409</td><td colspan="2">0.885 0.436 (6.60% ↑) 0.876 (1.02% ↓)</td></tr><tr><td>Non-fair</td><td>ML</td><td>0.429</td><td colspan="2">0.883 0.386 (10.0% ↓) 0.884 (0.11% ↑)</td></tr><tr><td/><td>R-GAN</td><td>0.414</td><td colspan="2">0.883 0.338 (18.4% ↓) 0.884 (0.11% ↑)</td></tr><tr><td/><td>FC</td><td>0.822</td><td colspan="2">0.806 0.756 (8.03% ↓) 0.655 (18.7% ↓)</td></tr><tr><td/><td>LBC</td><td>0.819</td><td colspan="2">0.760 0.816 (0.37% ↓) 0.738 (2.89% ↓)</td></tr><tr><td>Fair-only</td><td>AD FC+LD</td><td>0.807 0.822</td><td colspan="2">0.811 0.711 (11.9% ↓) 0.812 (0.12% ↑) 0.806 0.763 (7.18% ↓) 0.651 (19.2% ↓)</td></tr><tr><td/><td>LBC+LD</td><td>0.819</td><td colspan="2">0.760 0.513 (37.4% ↓) 0.859 (13.0% ↑)</td></tr><tr><td/><td>AD+LD</td><td>0.807</td><td colspan="2">0.811 0.730 (9.54% ↓) 0.818 (0.86% ↑)</td></tr><tr><td>Both</td><td>FR-GAN</td><td>0.813</td><td colspan="2">0.809 0.830 (2.09% ↑) 0.808 (0.12% ↓)</td></tr></table>

Accuracy and fairness performances on real data for disparate impact. Two types of methods are compared: (1) non-fairness methods: logistic regression (LR), the meta learning byRen et al. (2018) (ML), and R-GAN (FR-GAN with λ 1 = 0); (2) fairness-only methods: FAIRNESS CON-STRAINTS(Zafar et al., 2017) (FC), LABEL BIAS CORRECTION(Jiang & Nachum, 2019) (LBC), and ADVERSARIAL DEBIASING (Zhang et al., 2018) (AD). Also "+ LD" denotes loss defense(Koh et al., 2018) (a data sanitization technique that is applied prior to model training). For FR-GAN, the validation set is 10% of D tr . For each fairness or accuracy result of the poisoned data, we compare with the clean data result and show the percentage increase or decrease.

Table content: <table><tr><td>Dataset</td><td>Type</td><td>Method</td><td colspan="2">Clean data</td><td>Poisoned data</td></tr><tr><td/><td/><td/><td colspan="2">D. impact Acc.</td><td>D. impact</td><td>Acc.</td></tr><tr><td/><td/><td>LR</td><td>0.465</td><td colspan="2">0.674 0.431 (7.31% ↓) 0.667 (1.04% ↓)</td></tr><tr><td/><td>Non-fair</td><td>ML</td><td>0.493</td><td colspan="2">0.680 0.164 (66.7% ↓) 0.661 (2.79% ↓)</td></tr><tr><td/><td/><td>R-GAN</td><td>0.574</td><td colspan="2">0.683 0.393 (31.5% ↓) 0.650 (4.83% ↓)</td></tr><tr><td/><td/><td>FC</td><td>0.777</td><td colspan="2">0.682 0.790 (1.67% ↑) 0.671 (1.61% ↓)</td></tr><tr><td/><td/><td>LBC</td><td>0.866</td><td colspan="2">0.671 0.807 (6.81% ↓) 0.615 (8.35% ↓)</td></tr><tr><td colspan="2">COMPAS Fair-only</td><td>AD FC + LD</td><td>0.846 0.777</td><td colspan="2">0.680 0.822 (2.84% ↓) 0.655 (3.68% ↓) 0.682 0.790 (1.67% ↑) 0.668 (2.05% ↓)</td></tr><tr><td/><td/><td colspan="2">LBC + LD 0.866</td><td colspan="2">0.671 0.822 (5.08% ↓) 0.628 (6.41% ↓)</td></tr><tr><td/><td/><td colspan="2">AD + LD 0.846</td><td colspan="2">0.680 0.836 (1.18% ↓) 0.620 (8.82% ↓)</td></tr><tr><td/><td>Both</td><td>FR-GAN</td><td>0.936</td><td colspan="2">0.667 0.947 (1.18% ↑) 0.660 (1.05% ↓)</td></tr><tr><td/><td/><td>LR</td><td>0.328</td><td colspan="2">0.847 0.230 (29.9% ↓) 0.848 (0.12% ↑)</td></tr><tr><td/><td>Non-fair</td><td>ML</td><td>0.346</td><td colspan="2">0.844 0.215 (37.9% ↓) 0.843 (0.12% ↓)</td></tr><tr><td/><td/><td>R-GAN</td><td>0.335</td><td colspan="2">0.847 0.285 (14.9% ↓) 0.834 (1.53% ↓)</td></tr><tr><td/><td/><td>FC</td><td>0.825</td><td colspan="2">0.826 0.816 (1.09% ↓) 0.828 (0.24% ↑)</td></tr><tr><td/><td/><td>LBC</td><td>0.825</td><td colspan="2">0.825 0.825 (0.00% -) 0.795 (3.64% ↓)</td></tr><tr><td>Adult</td><td>Fair-only</td><td>AD FC + LD</td><td>0.850 0.825</td><td colspan="2">0.767 0.679 (20.1% ↓) 0.754 (1.69% ↓) 0.826 0.798 (3.27% ↓) 0.807 (2.30% ↓)</td></tr><tr><td/><td/><td colspan="2">LBC + LD 0.825</td><td colspan="2">0.825 0.815 (1.21% ↓) 0.811 (1.70% ↓)</td></tr><tr><td/><td/><td colspan="2">AD + LD 0.850</td><td colspan="2">0.767 0.821 (3.41% ↓) 0.728 (5.08% ↓)</td></tr><tr><td/><td>Both</td><td>FR-GAN</td><td>0.870</td><td colspan="2">0.819 0.866 (0.46% ↓) 0.827 (0.98% ↑)</td></tr></table>

Accuracy and fairness performances of the meta learning method byRen et al. (2018) on the clean and poisoned synthetic datasets for different validation set sizes. We used the same z-flipping poisoning attack described in Section 3.

Table content: <table><tr><td>Data</td><td/><td colspan="3">Validation set size Disparate impact Accuracy</td></tr><tr><td colspan="2">Clean Data</td><td>10%</td><td colspan="2">0.429</td><td>0.883</td></tr><tr><td/><td/><td>10%</td><td colspan="2">0.386</td><td>0.884</td></tr><tr><td/><td/><td>5%</td><td colspan="2">0.401</td><td>0.886</td></tr><tr><td colspan="2">Poisoned Data</td><td>1%</td><td colspan="2">0.297</td><td>0.845</td></tr><tr><td/><td/><td>0.5%</td><td colspan="2">0.269</td><td>0.829</td></tr><tr><td/><td/><td>0.1%</td><td colspan="2">0.348</td><td>0.585</td></tr><tr><td>Disparate Impact</td><td>0.4 0.5 0.6 0.7 0.8</td><td colspan="2">FR-GAN on clean data FR-GAN on poisoned data (val 0.5%) AD on clean data AD on poisoned data</td></tr><tr><td/><td>0.70</td><td>0.75</td><td>0.80 Accuracy</td><td>0.85</td></tr></table>

(ML), and R-GAN (FR-GAN with λ 1 = 0); (2) fairness-only methods: FAIRNESS CONSTRAINTS(Zafar et al., 2017) (FC), LABEL BIAS CORRECTION(Jiang & Nachum, 2019) (LBC), and ADVERSARIAL DEBIASING(Zhang et al., 2018) (AD). Also "+ LD" denotes loss defense(Koh et al., 2018) (a data sanitization technique that is applied prior to model training). For FR-GAN, the validation set is 10% of D tr . For each fairness or accuracy result of the poisoned data, we make a comparison with the clean data result and show the percentage increase or decrease. Accuracy and fairness performances on synthetic and real datasets w.r.t. equalized odds. Two algorithms are compared: (1) logistic regression (LR, non-fairness method) and (2) FR-GAN.

Table content: <table><tr><td>Dataset</td><td>Type</td><td>Method</td><td/><td colspan="2">Clean data</td><td>Poisoned data</td></tr><tr><td/><td/><td/><td/><td colspan="2">D. impact Acc.</td><td>D. impact</td><td>Acc.</td></tr><tr><td/><td/><td>LR</td><td/><td>0.465</td><td>0.674 0.856 (84.1% ↑) 0.487 (27.7% ↓)</td></tr><tr><td/><td>Non-fair</td><td>ML</td><td/><td>0.493</td><td>0.680 0.608 (23.3% ↑) 0.629 (7.50% ↓)</td></tr><tr><td/><td/><td>R-GAN</td><td/><td>0.574</td><td>0.683 0.919 (60.1% ↑) 0.621 (9.08% ↓)</td></tr><tr><td/><td/><td>FC</td><td/><td>0.777</td><td>0.682 0.931 (19.8% ↑) 0.490 (28.2% ↓)</td></tr><tr><td/><td/><td>LBC</td><td/><td>0.866</td><td>0.671 0.945 (9.12% ↑) 0.488 (27.3% ↓)</td></tr><tr><td colspan="2">COMPAS Fair-only</td><td>AD FC + LD</td><td/><td>0.846 0.777</td><td>0.680 0.941 (11.2% ↑) 0.615 (9.56% ↓) 0.682 0.820 (5.53% ↑) 0.544 (20.2% ↓)</td></tr><tr><td/><td/><td colspan="3">LBC + LD 0.866</td><td>0.671 0.928 (7.16% ↑) 0.550 (18.0% ↓)</td></tr><tr><td/><td/><td colspan="3">AD + LD 0.846</td><td>0.680 0.822 (2.84% ↓) 0.614 (9.71% ↓)</td></tr><tr><td/><td>Both</td><td colspan="2">FR-GAN</td><td>0.936</td><td>0.667 0.964 (2.99% ↑) 0.621 (6.90% ↓)</td></tr><tr><td/><td/><td>LR</td><td/><td>0.328</td><td>0.847 0.316 (3.66% ↓) 0.824 (2.72% ↓)</td></tr><tr><td/><td>Non-fair</td><td>ML</td><td/><td>0.346</td><td>0.844 0.413 (19.4% ↑) 0.826 (2.13% ↓)</td></tr><tr><td/><td/><td>R-GAN</td><td/><td>0.335</td><td>0.847 0.342 (2.09% ↑) 0.833 (2.13% ↓)</td></tr><tr><td/><td/><td>FC</td><td/><td>0.825</td><td>0.826 0.814 (1.33% ↓) 0.802 (2.91% ↓)</td></tr><tr><td/><td/><td>LBC</td><td/><td>0.825</td><td>0.825 0.734 (11.0% ↓) 0.819 (0.73% ↓)</td></tr><tr><td>Adult</td><td>Fair-only</td><td>AD FC + LD</td><td/><td>0.850 0.825</td><td>0.767 0.652 (23.3% ↓) 0.749 (2.35% ↓) 0.826 0.762 (7.64% ↓) 0.831 (0.61% ↑)</td></tr><tr><td/><td/><td colspan="3">LBC + LD 0.825</td><td>0.825 0.720 (12.7% ↓) 0.827 (0.24% ↑)</td></tr><tr><td/><td/><td colspan="3">AD + LD 0.850</td><td>0.767 0.761 (10.5% ↓) 0.803 (4.69% ↑)</td></tr><tr><td/><td>Both</td><td colspan="2">FR-GAN</td><td>0.870</td><td>0.819 0.859 (1.26% ↓) 0.789 (3.66% ↓)</td></tr><tr><td/><td>Dataset</td><td/><td colspan="2">Method</td><td>Equalized odds Accuracy</td></tr><tr><td/><td/><td/><td/><td/><td>Y = 0 Y = 1</td></tr><tr><td/><td colspan="2">Synthetic Data</td><td colspan="3">LR FR-GAN 0.888 0.936 0.351 0.804</td><td>0.885 0.865</td></tr><tr><td/><td colspan="2">Compas</td><td colspan="3">LR FR-GAN 0.718 0.959 0.427 0.557</td><td>0.674 0.628</td></tr><tr><td/><td colspan="2">AdultCensus</td><td colspan="3">LR FR-GAN 0.503 0.917 0.286 0.909</td><td>0.848 0.842</td></tr><tr><td colspan="3">A.4 TRAINING METHODOLOGY</td><td/><td/></tr></table>

Peer review (iclr2020):
This paper proposes to use a GAN style approach for training a classifier that is robust to data poisoning and can achieve a pre-specified notion of group fairness. The contribution of this paper is incremental in the context of prior Adversarial Debasing (AD) approach using essentially same GAN for group fairness and prior work presenting ideas of utilizing clean validation data to defend against data poisoning. This paper is proposing to add an additional discriminator to the AD approach that distinguishes training data and clean validation data. If the training data is poisoned, such distinguishment may be possible and maximizing loss of this discriminator can help to robustify against poisoned samples. Experimental results are insufficient to argue improvement over the AD. There are no AD results in the equalized odds Adult experiment in the supplement. I recommend more detailed comparison against the AD method (including results showing confusion matrices). Also note that AD, as presented in the original paper, is optimizing for demographic parity, but can also be adjusted to other group fairness metrics. Finally, in the context of Adult dataset, it is important to also report performance metrics such as balanced TPR since the labels are quite imbalanced. Are there any real data examples where poisoning does not need to be introduced artificially and the proposed method helps to improve the fairness properties? I think an interesting direction could be to consider data where labels are subjective. For example, a dataset on loan decisions can be naturally *poisoned* with human biases, i.e. information that someone did not receive the loan may be due to an error or bias of a human in charge of the decision making. Lastly I think that the discussion of the prior related work on fairness is incomplete. This paper exclusively covers group fairness, which indeed has been shown to have some disadvantages. For example, prior work [1] has shown that some group fairness notions can not be satisfied simultaneously in certain cases. In this regard it is also important to report multiple group fairness metrics simultaneously in the experiments. The other fairness definition, that has not been mentioned in this paper, is individual fairness [2]. It has legal and intuitive interpretations. Multiple recent papers have explored the direction of individual fairness [3,4,5,6]. [1] Kleinberg, J., Mullainathan, S., & Raghavan, M. (2016). Inherent trade-offs in the fair determination of risk scores. [2] Dwork, C., Hardt, M., Pitassi, T., Reingold, O., & Zemel, R. (2012, January). Fairness through awareness. [3] Garg, S., Perot, V., Limtiaco, N., Taly, A., Chi, E. H., & Beutel, A. (2019, January). Counterfactual fairness in text classification through robustness. [4] Yurochkin, M., Bower, A., & Sun, Y. (2019). Learning fair predictors with Sensitive Subspace Robustness. [5] Kearns, M., Roth, A., & Sharifi-Malvajerdi, S. (2019). Average Individual Fairness: Algorithms, Generalization and Experiments. [6] Jung, C., Kearns, M., Neel, S., Roth, A., Stapleton, L., & Wu, Z. S. (2019). Eliciting and Enforcing Subjective Individual Fairness.