Bias-Resilient Neural Network
Authors: Ehsan Adeli
Qingyu Zhao
Adolf Pfefferbaum
Edith V. Sullivan
L. Fei-Fei
Juan Carlos Niebles
Kilian M. Pohl
Venue:  (2020)
Keywords: Invariant Feature Learning
Vanished Correlation
Generative Adversarial Networks
Gender Shades
Fairness in Machine Learning

Abstract: Presence of bias and confounding effects is inarguably one of the most critical challenges in machine learning applications that has alluded to pivotal debates in the recent years. Such challenges range from spurious associations of confounding variables in medical studies to the bias of race in gender or face recognition systems. One solution is to enhance datasets and organize them such that they do not reflect biases, which is a cumbersome and intensive task. The alternative is to make use of available data and build models considering these biases. Traditional statistical methods apply straightforward techniques such as residualization or stratification to precomputed features to account for confounding variables. However, these techniques are not in general applicable to end-to-end deep learning methods. In this paper, we propose a method based on the adversarial training strategy to learn discriminative features unbiased and invariant to the confounder(s). This is enabled by incorporating a new adversarial loss function that encourages a vanished correlation between the bias and learned features. We apply our method to a synthetic, a medical diagnosis, and a gender classification (Gender Shades) dataset. Our results show that the learned features by our method not only result in superior prediction performance but also are uncorrelated with the bias or confounder variables. The code is available at http://blinded_for_review/.

Under review as a conference paper at ICLR 2020
BIAS-RESILIEN NEURAL NETWORK
Anonymous authors
Paper under double-blind review
ABSTRACT
Presence of bias and confounding effects is inarguably one of the most critical challenges in machine learning applications that has alluded to pivotal debates in the recent years. Such challenges range from spurious associations of confound- ing variables in medical studies to the bias of race in gender or face recognition systems. One solution is to enhance datasets and organize them such that they do not reflect biases, which is cumbersome and intensive task. The alterna- tive is to make use of available data and build models considering these biases. Traditional statistical methods apply straightforward techniques such residual- ization or stratification to precomputed features to account for confounding vari- ables. However, these techniques are generally not suitable for end-to-end deep learning methods. In this paper, we propose a method based on the adversar- ial training strategy learn discriminative features unbiased and invariant the confounder(s). This is enabled by incorporating a new adversarial loss function that encourages a vanished correlation between the bias and learned features. We apply our method to synthetic data, medical images, and gender classification (Gender Shades Pilot Parliaments Benchmark) dataset. Our results show that the learned features by our method not only result in superior prediction performance but also are uncorrelated with the bias or confounder variables. The code is avail- able at http:// /b bindedd for review/.
INTRODUCTION
A central challenge in practically all machine learning applications is the consideration of confound- ing biases. Confounders are extraneous variables that distort the relationship between the input (independent) and output (dependent) variables and hence lead to erroneous conclusions (Pourhose- ingholi et al., 2012). In a variety of applications ranging from disease prediction to face recognition, where machine learning models are built to predict labels from images, demographic variables (such as age, sex, race) of the study may confound the training process if the distribution of image labels skewed with respect to them. In this situation, the predictor may learn the influence of the con- founder and bias present in the data instead of actual discriminative cues.
It cumbersome task account for all biases when curating large-scale datasets (Yang et al., 2019). An alternative approach is to account for the bias in the model. Traditionally, confounding variables are often controlled by statistical methods in either design or analytical stages (Aschengrau & Seage, 2013). In the design stage, one can utilize randomization or matching of the confounding variables across different study groups. In the analytical stage, confounding can controlled by standardization or stratification (Pourhoseingholi et al., 2012; Aschengrau & Seage, 2013).
Another common solution is to learn the influence of the confounding variables on the input (in- dependent) variables by regression analysis. Then, the residuals derived from the optimal regres- sion model are regarded as the confounder-free input to train the predictor (Wodtke, 2018). The regression analysis works reasonably well under the assumption that the input variables represent deterministic features that are comparable across a population, egg., morphometrie measurements ex- tracted from medical images engineered features extracted from face images. The method fails, however, when this assumption does not hold such as for the pixel intensity values in images. Note, the intensities only meaningful within neighborhood but variant across images. Therefore, these regression approaches cannot be used in connection with deep learning methods that are di-
indicates equal contribution.

---

Under review as a conference paper at ICLR 2020
Shade
Shade
Shade
Shade
Shade
Shade
Average per Shade
tSNE Projection of the Learned Features (color encodes shade)
94.8%
96.2%
95.2%
95.6%
95.7%
86.4%
Baseline VGG16
96.2%
97.1%
97.2%
96.1%
96.7%
93.2%
BR-Net
VGG16
Backbone
Figure 1: Average face images across each shade category (first row), average saliency map of the trained baseline (second row), and BR-Net (third row) color-coded with the normalized saliency value for each pixel. BR-Net results in more stable patterns across all 6 shade categories. The last column shows the tSNE projection of the learned features by each method. Our method results in better feature space invariant to the bias variable (shade) while the baseline shows clear pattern affected by the bias. Average accuracy of per-shade gender classification over 5 rums of 5-fold cross- validation is shown on each average map. The models are pre-trained ImageNet and fine-ture fine-tuned on GS-PPB. BR-Net not only able to close the gap of accuracy for the darker shade but also regularizes the model to improve per-category accuracy.
rectly applied images, such convolutional neural networks (CNNs). Removing confounding factors for CNNs is an open question we aim to address here.
We propose feature learning scheme produce features that are predictive of class labels while being unbiased to confounding variables. The idea is inspired by the domain-adversarial training approaches (Ganin al., 2016) with controllable invariance (Xie et al., 2017) within the context of generative adversarial networks (GANs) (Goodfellow et al., 2014), but we argue that generic and widely used loss functions are not designed for controlling the invariance with respect to bias variables. Hence, we introduce adversarial loss function that aims to quantify the statistical dependence between the learned features and bias variables with the correlation coefficient. This strategy improves over the commonly used cross-entropy only aims to predict the exact value the bias variables and thereby achieves stabler results within the context of adversarial training. Since proposed model injects resilience towards the bias during training to produce confounder-invann invariant features, we refer to our approach as Bias-Resilienn Neural Network (BR-Net).
mean-squared error (MSE) loss that
We evaluate BR-Net on three datasets to examine different aspects of the method and compare with wide range of baselines. First, test synthetic dataset to outline how the learned fea- tures by our method are unbiased to controlled confounding variables. Then, we test it on a medical imaging application, i.e., predicting the human immunodeficiency virus (HIV) diagnosis directly from T1-weighted Magnetic Resonance Images (MRIs). As widely explored in the HIV literature, consider- HIV disease accentuates brain aging (Cole al., 2017) and if predictor is learned ing age as a confounder, the predictor may actually be learning the brain aging patterns rather than actual HIV markers. Lastly, we evaluate BR-Net for gender classification using the Gender Shades Pilot Parliaments Benchmark (GS-PPB) dataset (Buolamwini & Gebru, 2018). We use different backbones pre-trained on ImageNet (Deng et al., 2009) and fine-tune them for predicting gender from face images. We show that prediction of the vanilla models is dependent the race of the subject (alternatively we consider skin color quantified by the "shade" variable) and show poor re- sults for darker faces, while BR-Net successfully close the gap. Our comparison with methods based on multi-tase task (Lu et al., 2017) prediction (i.e,, predicting gender and shade as two tasks) and categorical GAN (Springenberg, 2015) (i.e., predicting shade categorical variable in the adver-
2

---

Under review as a conference paper at ICLR 2020
sarial component) shows that BR-Net is not only able to learn features impartial to the bias of race (verified by feature embedding and saliency visualization), it also results in better performance in gender prediction (see Fig. 1).
2 RELATED WORK
Fairness in Machine Learning: In recent years, developing fair machine learning models have been the center of many discussions (Liu et al., 2018; Hashimoto et al., 2018; Barocas et al., 2017) even in the news outlets and media (Khullar, 2019; Miller, 2015). It has been argued that the bias essentially comes from human or society biases induced by the training datasets (Barocas & Selbst, 2016). Recent efforts in solving this problem have been focused on building fairer and more diverse datasets (Yang et al., 2019; Celis et al., 2016). However, this approach is not always practical for large-scale datasets or especially in medical applications, where data is relatively scarce and expensive to generate. In this work, we propose to use existing sets of data but to build models mindful of biases by learning impartial features that are only predictive of the actual output variables.
Training: (Ganin et al., 2016) proposed for the first time to use adversarial training for domain adaptation tasks by creating a component in the network that uses the learned features to predict which domain the data is coming from (a binary variable; source or target). Ever since, several other works built on top of the same idea explored different loss functions (Bousmalis et al., 2017), domain discriminator settings (Tzeng et al., 2017), or cycle-consistency (Hoffman et al., 2017). The focus of all these works was to close the domain gap, which often encoded as a binary variable. To learn generic bias-resilient models, we argue that we need to go beyond this and learn features that are invariant to either discrete or continuous confounders.
Momain-Adversann
Invariant Feature Learning: There have been different attempts in the literature for learning rep- resentations that are invariant to specific factors in the data. For instance, (Zemel et al., 2013) took an information obfuscation approach to obfuscate membership in the protected group of data dur- ing training, and (Bechavod & Ligett, 2017; Ranzato et al., 2007) introduced a regularization based method. Recently, (Xie et al., 2017; Akuzawa et al., 2019; Zhang et al., 2018; Elazar & Goldberg, 2018) proposed to use dovainnadversarial training strategies for controllable invariant feature learn- ing with respect existing variables in the data. Some concurrent works (Sadeghi al., 2019; Wang et al., 2019) have also used adversarial techniques for mitigating the effect of bias. These methods used similar adversarial loss functions as in domain adaptation that aim predicting exact values of the bias variables. For instance, (Wang et al., 2019) used a binary cross-entropy for removing effect of "gender' and (Sadeghi al., 2019) used linear (and kernelized) least squares predictors as the ad- versarial component. Our study shows that these strategies fail at creating resilience against biases that take continuous or ordinal values. Instead, we introduce a loss function based on correlation coefficient to naturally alleviate the bias effects on the learned features.
Distribution Matching:
Some previous work attempted to learn distributionally robust techniques to avoid learning confounded effects from data (Oren et al., 2019). This can be done by matching the distributions of the data (Cao et al., 2018; Baktashmotlagh et al., 2016) across different domains. However, distribution matching techniques only model data of population as whole and fall short when it is crucial to remove the association between the learned features and a specific bias or confounding variable for each single input data point. Whereas, to close the gap with respect to the underlying bias in the data, our correlation-basel analysis minimizes the bias-predictive power of the learned features for every individual data point, which by construction harmonizes the data distribution on the population level.
3 BIAS-RESILIENT NEURAL NETWORK (BR-NET)
Suppose we have an M-class classification problem, for which we have N pairs of training images and their corresponding target label(s): ((X,, N Assume that the study is confounded or biased by a set of h variables, denoted by a vector b IR* To train a deep neural network for classifying each image to its label(s) while not being biased by the confounders in the study, we propose our end-to-end architecture (Ganin et al., 2016). Given the input image X, we first apply a Feature Extraction (FE) network, resulting in feature vector F. A Classifier (C) is built top of this feature vector predict the
in Fig. 2 similar to
training approaches
3

---

Under review as a conference paper at ICLR 2020
00
Classification
Feature Extraction
Lc
(C)
0g
F
X
(FFI)
@L.p
ffe
d0pp
Bias Prediction
a Abbp
Lop (b, b)
(B PP)
f0
bp
Figure 2: BR-Net architecture: FE learns features, F, that successfully classify (C) the input while being invariant (not correlated) to the bias variable(s), b, using BF and the adversarial loss compo- nent, LXp which is based correlation coefficient. Forward arrows show forward paths while the backward dashed ones indicate back-propagation with the respective gradient values.
class label y for the input X, and it forces FE to learn discriminative futures for the classification task. Now, to guarantee that these features are not biased to b, we build another network (denoted by BP)) with new loss function for predicting the bias variables from F. We propose to back- propagate this loss to the feature extraction module in an adversarial way. As a result, the feature extractor learns features that minimize the classification loss, while maximizing the bias predictor loss.
Each network has its underlying trainable parameters, defined f for FE, Og for C, and Obp for BP. If the predicted probability that subject belongs to class m is defined by dsm = C(FF X fe); 02), the classification loss can be characterized by cross-entropy:
M
L. y: 0e)
Vim log (fim).
(1)
define the adversarial component of the loss function. Standard methods for designing this loss function suggest to use a cross-entropy for nary/categorical variables (e.g., in (Ganin et al., 2016; Xie et al., 2017)) an & MSE loss for continuous variables. However, we argue that in the context of bias control, the ultimate goal of adversarial training is remove statistical association with respect the bias variables, opposed to maximizing the prediction error of them. In fact, the adversarial training based on MSE leads to the maximization of the & distance between b and b, which could be trivially achieved by uni- formly shifting the magnitude of b, thereby potentially resulting in ill-posed optimization and oscillation in the adversarial training. To address this issue, we define the following surrogate loss for predicting the bias confounders while quantifying the statistical dependence with respect to b:
Similarly, with b; (FFFEE(X; fe)); Obp).
Lap X, b; Sez 0bp) = Σ corr2 2bb, bx),
==1
where corr"(..) the squared Pearson correlation between its inputs and b, defines the vector of hh bias variable across all inputs. Through adversarial training, we aim to remove statistical depen- dence by encouraging zero correlation between b, and b,. Note, BP deems to maximize squared correlation and FE minimizes for it; Since corr is bounded the range [0, 1], both minimization and maximization schemes are deemed feasible. Having these loss functions defined, the overall objective of the network then defined as
0.)) ALtp (X, by 0 fes 0pp) min max (X, y; 0
e Ssp
where hyperparameter À controls the trade-off between the two objectives.
This scheme is similar to GAN (Goodfellow et al., 2014) and domain-adwersariil training (Ganin al., 2016; Xie et al., 2017), in which min-max game is portrayed between networks. In our
4

---

Under review as a conference paper at ICLR 2020
case, FE extracts features that minimize the classification criterion, while fooling BP (i.e., making BP incapable predicting the bias variables). Hence, the saddle point for this optimization objective obtained when the parameters fc minimize the classification loss while maximizing the loss of the bias prediction module. Simultaneously, Og and Op minimize their respective network losses.
3.1 THEORETICAL PROPERTY
In general, zero-correlation or zero-covariance only quantifies linear independence between vari- ables but cannot infer non-linear relationships. However, we now theoretically show that, under certain assumptions on the adversarial training of IPP, zero-covariance would guarantee the mean independence (Wooldridge, 2010) between bias variables and features, much stronger type of sta- tistical independence than the linear type.
A random variable B is said be mean independent of F if and only if E|B E] E[B] for all & with non-zero probability, where E[:] defines the expected value. In other words, the expected value of B is neither linearly following theorem then relates the mean independence between features F and bias variables B to the zero-covariance between B and the prediction of produced by the adversarial component, BIP.
non-linearly dependent F, but the variance of B might. The
Property 1: B is mean independent of B = Cov(B, B) O.
Property 2: B,F are mean independent + B is mean independent of B ((F) for any mapping function é.
Theorem 1, Given random variables F,B, B with finte second moment,
mean independent of +> for any arbitrary mapping &. s.t. B ((F). cou B)
Proof. The forward direction follows directly through Property and 2. We focus the proof on the reverse direction. Now, construct mapping function B o(F) E[B]F,, i.e, d(E) F = E). then Cou(B, [B, B) implies
[[BEBBF]]] = [|
(4)
Due to the self-adjointness of the mapping B + E[B]F], the left hand side of Eq. (4) reads
F F[BE[B]]]] = [ (F[B]]])* F F. By the law of total expectation F. [ F F.B). the
right hand side of Eq. (4) becomes E| By Jensen's (in eqqulitty E[ E[B holds if and only if 8 is constant, ie.,, by definition, B is mean independent F.
Remark. In practice normalize the covariance by standard deviations of variables optimiza- tion stability. In the unlikely singular case that IBP outputs a constant prediction, we add a small perturbation in computing the standard deviation.
This theorem echoes the validity of our adversarial training strategy: FE encourages zero- correlation between b, and bg. which enforces b, to be mean independent of F (one cannot infer the expected value of b, from F). In turn, assuming BP has the capacity to approximate any ar- bitrary mapping function, the mean independence between features and bias would correspond to nero-correlation between b, and b,, otherwise IPP would adversarially optimize for mapping function that increases the correlation.
3.2 IMPLEMENTATION DETAILS
Similar to the training of GANs, in each iteration, we first back paok-propagate the Le loss to update y and o... With f fixed, we then minimize the Lop loss to update Oxp- Finally, with Oop fixed, we maximize the Lbp loss to update f The last step can be considered as the bias effect removal component. Furthermore, in the present study, Ltp depends the correlation operation, which a population- based operation, as opposed to individual-level enror metrics such as cross-entropy or MSE losses. Therefore, calculate the correlations over each training batch batch-level
5

---

Under review as a conference paper at ICLR 2020
operation. Depending on the application, we can use different architectures for each of the three subnetworks. We use a 3D CNN (Esmaeilzadch et al., 2018; Nie et al., 2016) for FE to extract features from 3D medical images and use VGG16 (Simonyan & Zisserman, 2015) and ResNet50 (He et al., 2015) backbones for GS-PPB. For C and HIP, we use a two-layer fully connected network.
4 EXPERIMENTS
In this section, we evaluate our method on three different scenarios. First, we run synthetic exper- iment to verify the validity of our assumptions. Then, we apply BR-Net to predict diagnosis HIV from brain MRIs confounded by the subjects' age. Finally, test the model for predicting gender from face images and show how controlling for variables related to race (e.g.. face color shade) can robustly enhance prediction performance. We compare BR-Net with several baselines, and evaluate how the features learned by our method are invariant to the bias or confounding variables.
Baseline Methods. In line with the implementation of our approach, the baseline for all three experiments is a vanilla CNN, whose architecture is exactly the same as BR-net except that there is no bias prediction sub-network and hence the adversarial loss. We emphasize that BR-Net aims to remove the association between prediction and bias by encouraging vanished correlation, which is different from simply maximizing the prediction loss (w.r.t bias) as usually performed in many GAN settings. Therefore. the second comparison method is BR-Net with the adversarial loss being the MSE, denoted by 'BR-Net (w/ MSE). For the Gender Shades PPB experiment, we further add two other baseline methods, one predicting both 'gender' and 'shade' in multi-task setting (Lu et al., 2017), denoted by 'Multi-Task and one replacing correlation loss function Lop with a cross-entropy loss as the 'shade' variable has ordinal but categorical value. The adversarial training then relies on maximizing the entropy of BP predictions as motivated in Categoral GAN models ("CarGAN') (Springenberg, 2015). These baselines show how the correlation loss plays important role in delineating the bias and confounding effects.
4.1 SYNTIETIC DATA
We generate a synthetic dataset comprised of two groups of data, each containing 512 images of resolution 32 x 32 pixels. Each image is generated by 4 Gaussians (see Fig. 3a), the magnitude of which is controlled by σ and σn. For each image from Group 1, we sample σ and a from uniform distribution (1(1, 4) while we generate images of Group with stronger intensities by sampling from (433,6 6). Gaussian noise is added to the images with standard deviation 0.01. Now we assume the difference in a between the two groups is associated with the true discriminative cues that should be learned by a classifier, whereas dB is given confounder. In other words, an unbiased model should predict the group label purely based on the two diagonal Gaussians and not dependent the two off-diagonal ones. To show that the BR-Net can result in such models by controlling for o, we train it on the whole dataset of 1,024 images given their respective binary labels and confounder values dB.
For simplicity, we construct the FE Network with 3 stacks 2 x comvolution/ReLLUmaal poooon layers produce 32 features. Both the IBP and C networks have one hidden layer of dimension 16 with tanh as the non-linear activation function. After training, BR-Net achieves 89% training accuracy and BR-Net w/ MSE achieves 90%. Note that the theoretically maximum training accu- racy is 90% due to the overlapping sampling range of a between the two groups. The baseline model, however, achieves 95% accuracy, indicating that the model additionally relies on the con- founding effects og for predicting the group label, an undesired behavior. To further investigate the association between the learned features and σB, we measure their squared distance correlation (dcor?) (Székely et al., 2007) for the training samples in Group I (when there is no association be- widely-used measure
tween BB and prediction, door=0 for either group). Distance correlation of dependence between two paired vectors of arbitrary dimensions. Fig. 3b shows that our method successfully removes the statistical association w.r.t σB as the distance correlation drops dramat- ically with training iterations. On the other hand, the baseline model without the BP component learns features that constantly yield high correlation. Note that the adversarial loss based on MSE yields unstable ? measures potentially due to the ill-posed optimization of maximizing & dis- tance. Finally. the above results are further supported by the 2D iSNE (Maaten & Hinton, 2008) projection of the leamed features shown in Fig. 3c. The feature space learned by the baseline

---

Under review as a conference paper at ICLR 2020
0012
A
BB Confounder
-008
006
ØB
OA
01-
100
Confounder
402
600
1000
1200
14:00
(a) Image synthesis
(b) Distance correlation w.r.t dp
Baselie ICNN
BR-Niet (MSE)
BiR-Net (Comr)
20
25
15
(c) ISNE projection of the learned features for CNN (left), BR-Net w/ MSE (middle), and BR-Net (right)
Figure 3: Comparison of results the synthetic dataset. Color indicates the value of d-
0.25
Boceliin CMC 28 (61) BB fuet (Ourc
Balanced accuracy, F -coree and area under curve (AUC) of HIV diagnosis prediction. Best results each column are typeset bold.
Table 1:
Method
bAcc
F
AUC
Resid+SVM
69.5
0.65
71.2
Baseline (3D CNN)
71.8
0.64
80.8
BR-Net (w/ MSE)
64.8
0.58
75.2
20-
1.50
140
BR-Net (Ours)
74,2
0.67
80.9
Figure 4: Distance correlation between the learned features and age for the CTRL cohort.
model forms clear correlation with dB. whereas method results in space with apparent bias. This confirms that the proposed adversarial technique successfully removes the bias from the confounding variable.
4.2 PREDICTION OF HIV DIAGNOSIS BASED MEDICAL IMAGES
Our second task aims at predicting the diagnosis of HIV patients vy.control subjects (CTRL) based on brain MRIs. The study cohort includes 223 CTRLs and 122 HIV patients who are seropositive for the HIV-infection with CD4 count > 100 gells (average: 303.0). Since the HIV subjects sig- aL nificantly older in age than the CTRLs (CTRL: 45 + 17, HIV: 51 + 8.3, p .001) in this study, age becomes a potential confounder; prediction of diagnosis labels may be dependent on subjects' age instead of true HIV markers. The T1 T1-weighted MRIs are all skull stripped, affinely registered common template, and resized into a 64 x 64 x 64 volume. Classification accuracy is measured with 5-fold cross validation. For each run, the training folds are augmented by random shifting (within one-voxel distance), rotation (within one degrec) in all directions, and left-right flipping based on the assumption that HIV infection affects the brain bilaterally (Adeli et al., 2018). The data aug- mentation results in a balanced training set of 1024 CTRLS and 1024 HIVs. As the flipping removes left-right orientation, the ConvNet is built on half the 3D volume containing one hemisphere. The feature extractor FE has stacks of x 3D convolution/ReeLuban htt.://acc...oommmiiitooannaaaaa layers yielding 4096 intermediate features. Both BIP and C have one hidden layer of dimension 128 with tanh as the activation function. For this experiment, suggested in the previous work (Rao et al., 2017), confounding effects can only be reliably estimated among healthy subjects. So, in practice only perform the adversarial loss propppronnn step for the CTRL group.

---

Under review as a conference paper at ICLR 2020
classrication accuracy Baseline
classiiiation accuracy
0.9
0.9
6.8
AwM
0.7
May
0.5
0.-
Belanced Accuracy
0.4
Belanced Accuraty
0.3
True Negative Rate
Posttive Rate
(b)
True Nogattvve Rate
Fositive
100
120
140
20
40
60
80
100
120
140
Beralicen
Iberatoons
Figure 5: Accuracy, true negative, and positive rates of the HIV diagnosis experiment, as function of the number of iterations for (a) the 3D CNN baseline, (b) BR-Net. The results show that our adversarial training with correlation loss function robust against the imbalanced age distribution between HIV and CTRL subjects and achieves balanced prediction for both cohorts.
learned features using Baseline (3D CNN)
LSNE of the leaned features using BR (Ours)
80
70
60
50
(a)
20
Figure 6: (SNE visualization of the learned features by (a) the 3D CNN baseline and (b) our BR-Net. Each point shows subject in the CTRL cohort color-coded by their age.
Table shows the diagnosis prediction accuracy of BR-Net in comparison with 3D CNN, BR- Net (w/ MSE), and Resid+SVM (note, to compare with the traditional residualization methods, we extract 298 brain regional measurements, residualize the confounders using general linear model, and classify with a support vector machine). Our method (BR-Net) results in the most accurate prediction terms balanced accuracy (bAce), area under curve (AUC), and Fi-score from cross-vale validatio.. These results show that our method is able to learn discriminative features while controlling for confounders. In addition, we record the balanced accuracy, true positive, and true negative rate for each training iteration. As shown in Fig. 5, the baseline tends predict most subjects as CTRLS (high true negative rate). This is potentially caused by the CTRL group having wider age distribution, so an age-dependent predictor would bias the prediction towards CTRL. On the other hand, when controlling age as confounder, BR-Net reliably results in balanced true positive and true negative rates.
Similar to the previous experiment, we train different methods on the entire dataset and plot the squared distance correlation between the learned features and confounders for the CTRL cohort (Fig. 4). The figure shows that for BR-Net the distance correlation between the features and the confounding variable (age) decreases with the adversarial training. Whereas, the baseline model 3D CNN consistently produces features that are highly correlated with the confounder, and BR-Net w/ MSE produces inconsistent and unreliable associations with respect to the confounder. The t-SNE (Maaten & Hinton, 2008) projection of the leamed feature spaces are visualized in Fig. 6. The feature space learned by the baseline model forms a clear association with age, as older subjects are concentrated the top left region of the space. This again suggests predictions from the baseline may be dependent on age rather than true HIV markers. Whereas, our method results in space with apparent bias age.
8

---

Under review as a conference paper at ICLR 2020
Results Gender Shades Dataset- VGG16 Backbone
Results Gender Shades Dataset ResNet50 Backbone
BR-Net CatGan VGG16
Multi- Task
BR-Net
CalGan
ResNet Multi-Task
3.80
Balanced Accuracy
1.40
Balanced Accuracy
0.
6.0
0.53
0:5
6.61
0.33
shadel
shade2
shade3
shade4
shade5
shade6
shade1
shade2
shade3
shade4
shade5
shade6
Figure 7: Accuracy of gender prediction from face images across all shades (I to 6) of the GS-PPB dataset with two backbones, (left) VGGI6 and (right) ResNet50. BR-Net consistently results in more accurate predictions in all 6 shade categories by injecting bias-resilience into the model.
thE St aricamod tostures es by easznin 2b1
TaMl:: of tcarncc reatines By the Mutti mood
Ditt: iesimoo Teatures RoS (UUIS)
(a)
(b)
(c)
Figure 8: 1SNE visualization of the learned features by (a) the VGG16 baseline, (b) multi-task baseline, and (c) our BR-Net. Each point shows an image in the dataset color-coded with 'shade'.
4.3 GENDER PREDICTION USING THE GS-PPB DATASET
The last experiment on gender prediction from face images in the Gender Shades Pilot Parliaments Benchmark (GS-PPB) dataset (Buolamwini & Gebru, 2018). This dataset contains 1,253 facial images of 561 female and 692 male subjects. The face shade is quantified by the Fitzpatricksix-poe labeling system and is categorised from type 1 (lighter) to type 6 (darker). This quantization was used by dermatologistsuse for skin classification and determining risk for skin cancer (Buolamwini & Gebru, 2018).
To train our models on this dataset, we use backbones VGG16 (Simonyan & Zisserman, 2015) and ResNet50 (He et al., 2015) pre-trained on ImageNet (Deng et al., 2009). We fine-tune each model on GS-PPB dataset to predict the gender of subjects based on their face images using fair 5-fold cross-validation. The ImageNet dataset for pre-training the models has fewer cases of humans with darker faces (Yang et al., 2019), and hence the resulting models have an underlying bias to the shade.
BR-Net counts the variable 'shade' besides the vanilla VGG16 and ResNet50 models, we compare the results with multi-task baseline (Lu et al., 2017), which predicts both 'gender' and 'shade' simultaneously, and model that uses the entropy loss as the adversarial loss for the cross-entropy-basee categorical prediction (proposed by CatGAN (Springenberg, 2015)). Table 2 shows the results across five runs of 5-fold cross-validation. Fig. 7 plots the accuracy for each individual 'shade" category. As can be seen from the table and
ordinal and categorical bias variable. As discussed earlier,
Table 2: Average results over five runs of 5-fold cross-validation the GS-PPB dataset. Best results in each column are typeset in bold.
VGG16 Backbone
ResNetSi Backbone
Method
bAcc
F1
AUC
bAcc
F,
ACC
Baseline
94.1 + 0.2
93.5 - 0.3
98.9 : 0.1
75.7 - 2.0
68.0 =:000 3.0
96.2 0.3
CatGAN
96.0 + 0.5
95.7 0.5
99.4 + 0.2
90.1 + 1.0
90.0 + 1.0
96.3 + 0.7
Multi-Task
94.0 + 0.3
93.4
+.44+0..
98.9 0.1
94.0 + 0.3
93.4 0.3
98.6 0.3
BR-Net
96.3 + 0.6
96.0 + 0.7
99.4 + 0.2
94.1 + 0.2
93.6 - 0.2
98.6 +

---

Under review as a conference paper at ICLR 2020
the figure, BR-Net outperforms other methods on average while producing similar accuracy across all 'shade' categories. Prediction made by other methods, however, is more dependent on the bias variable by showing inconsistent recognition capabilities for different 'shade' categories and failing significantly on darker faces. This bias is confirmed by the iSNE projection of the feature spaces learned by different methods (see Fig. 8). The features learned by the vanilla baseline or even the multi-task model show a clear dependency on the 'shade" while BR-Net results in a roughly uniform distribution of subjects.
To gain more insight into the results, we visualize the saliency maps derived for the baseline and BR-Net. For this purpose, we use a similar technique as in (Simonyan al., 2014) extract the pixels in the original image space highlighting the areas that are discriminative for the gender labels. Generating such saliency maps for all inputs, we visualize the average map for each individual 'shade' category (Fig. 1). The value on each pixel corresponds to the attention from the network to that pixel within the classification process. Compared the baseline, BR-Net focuses more on specific face regions and results in more stable patterns across all 'shade' categories.
5 CONCLUSION
We proposed a method based on adversarial training strategies by encouraging vanished correlation to learn features for the prediction task while being unbiased to the confounding variables in the study. We evaluated our bias-resilient neural network (BR-Net) on synthetic, a medical diagnosis, and a gender prediction dataset. In all experiments, BR-Net resulted in a feature embedding space that was agnostic to the bias in the data while all other methods failed to do experiments we can conclude that, besides the attempt to improve datasets and curate unbiased ones (Yang al., 2019), it is crucial build models that properly account for the bias in data during training. Our bias-resilient model and some other recent works set on foot toward this direction. This is crucial as machine learning models are acceding to everyday lives, or are being developed for crucial medical applications. Failure to account for the underlying bias or confounding effects can lead to spurious associations and erroneous decisions. As a direction for the future work, other strategies such as deep canonical correlation analysis (Andrew et al., 2013) can be explored to form the adversarial component.
Based
REFERENCES
Ehsan Adeli et al. Chained regularization for identifying brain patterns specific to HIV infection. Neurolmage, 183:425 437,22-437... 2018.
Kei Akuzawa, Yusuke Iwasawa, and Yutaka Matsuo. Adversarial invariant feature learning with accuracy constraint for domain generalization. arXiv preprint (eXX4v:::04... 2019.
Galen Andrew, Raman Arora, Jeff Bilmes, and Karen Livescu. Deep canonical correlation analysis. In International conference on machine learning, pp. 1247- 2013.
Ann Aschengrau and George R Seage. Essentials of epidemiology in public health. Jones & Bartlett Publishers, 2013.
Mahsa Baktashmotlagh, Mehrtash Harandi, and Mathieu Salzmann. Distributionn-atching embed- ding for visual domain adaptation. The Journal of Machine Learning Research, 17(1))3760-3789, 2016.
Solon Barocas and Andrew D Selbst. Big data's disparate impact. Calif. L. Rev., 104:671, 2016.
Solon Barocas, Moritz Hardt, and Arvind Narayanan. Fairness in machine learning. NIPS Tutorial, 2017.
Yahav Bechavod and Katrina Ligett. Learning fair classifiers: A regularization-inss inspiree approach. arXiv preprint arXiv: 000044 pp. 1733-1782, 2017.
Konstantinos Bousmalis, Nathan Silberman, David Dohan, Dumitru Erhan, and Dilip Krishnan. Unsupervised pixel-level domain adaptation with generative adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3722-3731, 2017.
10

---

Under review as a conference paper at ICLR 2020
Joy Buolamwini and Timnit Gebru. Gender shades: Intersectional accuracy disparities in commer- cial gender classification. Conference on faimnss, accountability and transparency, pp. 77-91, 2018.
Yue Cao, Mingsheng Long. and Jianmin Wang. Unsupervised domain adaptation with distribution matching machines. In "hhrty--ecoon AAAI Conference on Artificial Intelligence, 2018.
Elisa Celis, Amit Deshpande, Tarun Kathuria, and Nisheeth Vishnoi. How to be fair and diverse?
arXiv preprint arXiv: /10.v:1610.07/883, 2016.
James Cole et al. Increased brain brain-predicted aging in treated HIV disease. Neurology, 88(14):1349- 2017.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale erarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 255, leee, 2009.
Yanai Elazar and Yoav Goldberg. Adversarial removal of demographic attributes from text data. In Proceedings of the 2018 Conference Empirical Methods in Natural Language Processing, pp- 11-21, Brussels, Belgium, October-November 2018. Association for Computational Linguistics.
Sohcil Esmacilzadch, Dimitrios Ioannis Belivanis, Kilian M Pohl, and Ehsan Adeli. End-to-end alzheimers disease diagnosis and biomarker identification. In MICCAI-MIMI, 2018.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, Franccois training of neural net-
Laviolette, Mario Marchand, and Victor Lempitsky. works. The Journal of Machine Learning Research, 1 17(1):2096220 2030, 2016.
lan Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde- Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. NeurIPS, 2014.
Tatsunori B Hashimoto, Megha Srivastava, Hongseok Namkoong. and Percy Liang. Fairness without demographics in repeated loss minimization. In Proceedings of the International Conference Machine Learning, 2018.
K He, X Zhang, Ren, and Sun. Deep residual learning for image recognition, computer vision and pattern recognition (evpr). In 2016 IEEE Conference on, volume pp. 2015.
Judy Hoffman, Eric Tzeng, Taesung Park, Man Zhu, Phillip Isola, Kate Saenko, Alexei A Efros, and Trevor Darrell. Cycada: Cycle-consistent adversarial domain adaptation. arXiv preprint urXiv:1711..00 2017.
Dhruv Khullar. A.I. could worsen health disparities, 2019. URL https::/weet www.:/yytmmms..... htm/2019901///
heallassssattthhhrreeea html.
Lydia T Liu, Sarah Dean, Esther Rolf, Max Simchowitz, and Moritz Hardt. Delayed impact of fair machine learning. In Proceedings of the International Conference on Machine Learning, 2018.
Yongxi Lu, Abhishek Kumar, Shuangfei Zhai, Yu Cheng, Tara Javidi, and Rogerio Feris. Fully- adaptive feature sharing in musk networks with applications in person attribute classification. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 5334
5343, 2017.
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-SNE. JMLR, 9, 2008
Clair Miller. Algorithms and bias, 2015. URL
wttp://imeeeeecc.m.eeeeeeettmmte 2015/08/1///// upshot/ / a http://woooota.aaaaaaaaooaaaaaa. httm...h... html.
Dong Nie, Han Zhang, Ehsan Adeli. Luyan Liu, and Dinggang Shen. 3d deep learning for multi- modal imaging-guidde survival time prediction of brain tumor patients. International Confer- ence on Medical Image Computing and Computer-Assisted Infervention, pp. 212-220. Springer, 2016.
Yonatan Oren, Shiori Sagawa, Tatsunori Hashimoto, and Percy Liang. Distributionally robust lan- guage modeling. In Empirical Methods in Natural Language Processing (EMNLP), 2019.
11

---

Under review as a conference paper at ICLR 2020
Mohamad Amin Pourhoseingholi, Ahmad Reza Baghestani, and Mohsen Vahedi. How to control confounding effects by statistical analysis. Gastroenterol Hepatol Bed Bench, 5(2):79, 2012.
Marc' aurelio Ranzato, Fu Jie Huang, Y-Lan Boureau, and Yann LeCun. Unsupervised learning of invariant feature hierarchies with applications object recognition. In 2007 IEEE conference on computer vision and pattern recognition, pp. 1-8. IEEE, 2007.
Anil Rao et al. Predictive modelling using neuroimaging data in the presence of confounds. Neu- rolmage, 150:23 49, 2017.
Bashir Sadeghi, Runyi Yu, and Vishmu Boddeti. On the global optima of kernelized adversarial rep- resentation learning. In Proceedings of the IEEE International Conference Computer Vision, pp. 7971-9999, 2019.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. ICLR, 2015.
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. In ICLR, 2014.
Jost Tobias Springenberg. Unsupervised and sumi-supervised learning with categorical generative adversarial networks. arXiv preprint arXiv11511.063900 2015.
Gábor J Székely, Maria L Rizzo, Nail K Bakirov, et al. Measuring and testing dependence by correlation of distances. The annals of statistics, 3566):2769-2794, 2007.
Eric Tzeng, Judy Hoffman, Kate Saenko, and Trevor Darrell. Adversarial discriminative domain adaptation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7167-7176, 2017.
Tianlu Wang, Jieyu Zhao, Mark Yatskar, Kai-Wei Chang, and Vicente Ordonez. Balanced datasets not enough: Estimating and mitigating gender bias in deep image representations. In Pro- ceedings of the IEEE International Conference on Computer Vision, pp. 5310-5319, 2019.
Geoffrey T Wodtke. Regression-based adjustment for time-varying confounders. Sociological Meth- ods & Research, pp. 00491241 2018
Jeffrey M. Wooldridge. Econometric Analysis of Cross Section and Punel Data (2nd ed.). The MIT
Press, London, 2010.
Qizhe Xie, Zihang Dai, Yulun Du, Eduard Hovy, and Graham Neubig. Controllable invariance through adversarial feature learning. In Advances in Neural Information Processing Systems, pp- 585-596, 2017.
Kaiyu Yang, Klint Qinami, Li Fei-Fei, Jia Deng, and Olga Russakovsky, Towards fairer datasets: Filtering and balancing the distribution of the people subtree in the imagenet hierarchy. Image- Net, 2019.
Rich Zemel, Yu Wu, Kevin Swersky, Toni Pitassi, and Cynthia Dwork. Learning fair representations. In International Conference on Machine Learning, pp. 325-333, 2013.
Brian Hu Zhang, Blake Lemoine, and Margaret Mitchell. Mitigating unwanted biases with adver- sarial learning. In Proceedings of the 20/8 AAAI/ACM Conference on AI, Ethics, and Society, pp- 335-340. ACM, 2018.
12


Reviews:
[{"review_id": "Syg8nq8pKr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents Bias-Resilient neural network (BR-Net) that is designed to learn representations that can accurately predict the desired target while being invariant to the confounding covariates in the data. The proposed method is based on domain adversarial training strategies, especially that of (Ganin et al., 2016), where the adversarial component is modified from “loss of distinguishing between the source and target domains” to “the squared Pearson correlation between the ground truth bias covariate and its estimation from the learned representation”. This design is based on the argument that the ultimate goal of bias control here is removing statistical association with respect to the bias variables, as opposed to maximizing the prediction error of them.\n\nThings to improve the paper that did not impact the score:\n\t- Introduction, line 4: Wrong citation format: use of \\citet instead of \\citep. Correct for all citations throughout the paper.\n\nReferences: \n\t- Ganin, Y., Ustinova, E., Ajakan, H., Germain, P., Larochelle, H., Laviolette, F., ... & Lempitsky, V. (2016). Domain-adversarial training of neural networks. The Journal of Machine Learning Research, 17(1), 2096-2030.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "SJlaTYsk9H", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes an adversarial approach toward debiasing neural network representations w.r.t protected attributes. The core idea is balance task loss with an adversarial loss from which protected attributes have low correlation with the feature representation used for the end task. The paper provides some synthetic experiments, and evaluates on HIV data (the bias variable being age) and gender classification in images, stratified by skin shade (the bias variable being skin shade). Results demonstrate improved balanced accuracy across all three experiments. \n\nOverall the direction is interesting and the methodology is intuitive and sound. Unfortunately, this paper seems unaware of extremely related works:\n\n1. (AAAI 2018) http://www.m-mitchell.com/papers/Adversarial_Bias_Mitigation.pdf\n2. (ACL 2018) https://arxiv.org/abs/1808.06640\n3. (ICCV 2019) https://arxiv.org/abs/1811.08489\n4. (ICCV 2019) http://hal.cse.msu.edu/assets/pdfs/papers/2019-iccv-kernel-adversarial-representation-learning.pdf\n\nBeyond experiments on different datasets, and slight modification of the adversarial loss for correlation, I am unsure what this paper contributes beyond these works. While some some of these papers can potentially be considered contemporary, authors must at least address these issues.  Furthermore, modification of the objective seems to have mixed results (Table 2 CatGAN vs. BR-NET, although I'm not sure whats going on with vgg vs resnet), where the baseline would correspond more closely to the setup in https://arxiv.org/abs/1811.08489 (3. above) . \n\nOverall I am positive about the direction, but I am unsure this paper represents a significant contribution over existing work. \n\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "B1xwaKqptr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose a method based on GAN  to classify data while automatically removing confounding effects during training, in order to obtain a classifier whose features are not biased by any confounding effect. The proposed idea is based on the extension of classical classification architectures to account for bias prediction. In practice, the parameters of the feature extraction component are updated to solve both bias prediction and the desired classification problem in adversarial fashion. Pearson’s correlation was chosen as a metric for bias.\n\nThe paper is interesting and addresses an important problem for the application of machine learning methods in several real context. The feeling is however that the paper should have better explored the implication of the proposed model of bias, and better investigated the relationship with simpler approaches relying on similar hypothesis.\n\nHere are my main comments for this work:\n\n- Why Pearson’s correlation should be a reliable metric to quantify bias? This metric is insensitive to affine scaling of the data, which is a quite common form of bias (for example in medical images).\n- The authors should have investigated the relationship between the proposed method and bias removal through canonical correlation analysis (CCA), and perhaps its non-linear variants. At the end this is what their network is doing, although in an end-to-end fashion. Using the CCA projections in the latent space for classification would be the closest approach to the state of the art for bias removal in statistical analysis (residual analysis). \n- The experimental setting illustrated in 4.1 is not clear. In which sense \\sigma_A is a common factor for the two groups? Why the theoretical maximum classification accuracy is 90%? Figure 2 is not clear either and doesn’t help understanding the structure of the generated data (e.g. axis labels missing, colorbars units not specified). \n- It is not clear why authors quantify the correlation in the latent space with tSNE projections. tSNE is highly sensitive to the choice of parameters and it would be important to ensure that it was a “fair competition” between all the methods, when showing the results of the dimensionality reduction. This is another modelling step relying on specific assumptions which decreases interpretability of the findings. The authors also proposed to assess the decorrelation of the estimated features throughout the different methods by measuring the squared distance correlation. Naturally, their method is the one which exhibits the best performances using this metric. However, this way of assessing the decorrelation of the features with the biases is unfair to the other methods, as in their case they specifically built their model to avoid statistical correlation between features and biases.  \n- Experiment 4.2 has some controversial aspects, as the bias correction is performed on the control population only, while the model is trained on the entire population. I understand the fact that confounding can be estimated only on healthy conditions, however in this case the network is going to be biased by the control group by construction. The effect of such a choice in the end-to-end optimisation scheme is really not clear.\n- We also observe that the results of the baseline CNN are very close to the BR-Net. The main difference lies in the fact that the CNN tends to have an unbalanced classification between true negatives and true positives. However, what would happen if we corrected for age before applying the CNN ?\n- In the case where the performances of the CNN would be improved, this last question would raise another one. Indeed, if I already know the confounding effects I want to correct for, why wouldn’t I correct them beforehand in order to avoid to train a complex GAN, which leads to more instability during training. This aspect points to the limit of having an online bias-correction (at least for the medical data case).\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper addressed the problem of machine bias when training machine learning models. The authors propose an approach based on representation learning with adversarial training. As opposed to the majority of previous works that trying to create a representation from which it is not possible to predict the sensitive feature (bias), the authors propose to minimize the dependency between the learned features and the sensitive feature with adversarial training. While acknowledging that the proposed model is addressing an important problem and is potentially useful, the reviewers and AC note the following potential weaknesses:  
(1) limited technical contribution -- the proposed approach is similar to a number of works published in machine learning and computer vision before the submission deadline that were overlooked by the authors. Specifically: i) adversarial training for learning fair representations [Edwards and Storkey, Censoring Representations with an Adversary, ICLR 2016], [Beutel, et al 2017, Data decisions and theoretical implications when adversarially learning fair representations], ii) learning fair representation by minimizing the dependency between the latent representation and the sensitive attributes [The variational fair autoencoder, ICLR 2016 by Louizos et al.; Fairness Constraints: Mechanisms for Fair Classification, by Zafar et al, 2015] or by minimizing the mutual information between feature embedding and bias [Learning Not to Learn: Training Deep Neural Networks with Biased Data, CVPR 2019]. 
(2) Limited empirical evidence -- the baseline methods used in the evaluation are not sufficient to assess the benefits of the proposed approach over the existing SOTA methods mentioned above. In fact, none of the baseline methods used in the evaluation tackle machine bias (via adversarial training or minimizing statistical dependence). 
(3) It would be beneficial to also report fairness metrics, e.g. equality of opportunity, statistical parity, to assess the effectiveness of bias removal. R1 has raised some concerns regarding empirical evidence -- see the point about mixed results. Also R2 has reported concerns regarding controversial results in experiment 4.2 and suggested ways to justify when and why the results of the CNNs baseline are close to the BR-Net. Addressing these concerns would strengthen the contributions of the proposed method.    

Among these, (3) did not have a decisive impact on the decision, but would be helpful to address in a subsequent revision. However, (1) and (2) make it very difficult to assess the benefits of the proposed approach, and were viewed by AC as critical issues. AC suggests, in its current state the manuscript is not ready for a publication. We hope the reviews are useful for improving and revising the paper.

Decision: Reject