Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper proposes an adversarial approach toward debiasing neural network representations w.r.t protected attributes. The core idea is balance task loss with an adversarial loss from which protected attributes have low correlation with the feature representation used for the end task. The paper provides some synthetic experiments, and evaluates on HIV data (the bias variable being age) and gender classification in images, stratified by skin shade (the bias variable being skin shade). Results demonstrate improved balanced accuracy across all three experiments. 

Overall the direction is interesting and the methodology is intuitive and sound. Unfortunately, this paper seems unaware of extremely related works:

1. (AAAI 2018) http://www.m-mitchell.com/papers/Adversarial_Bias_Mitigation.pdf
2. (ACL 2018) https://arxiv.org/abs/1808.06640
3. (ICCV 2019) https://arxiv.org/abs/1811.08489
4. (ICCV 2019) http://hal.cse.msu.edu/assets/pdfs/papers/2019-iccv-kernel-adversarial-representation-learning.pdf

Beyond experiments on different datasets, and slight modification of the adversarial loss for correlation, I am unsure what this paper contributes beyond these works. While some some of these papers can potentially be considered contemporary, authors must at least address these issues.  Furthermore, modification of the objective seems to have mixed results (Table 2 CatGAN vs. BR-NET, although I'm not sure whats going on with vgg vs resnet), where the baseline would correspond more closely to the setup in https://arxiv.org/abs/1811.08489 (3. above) . 

Overall I am positive about the direction, but I am unsure this paper represents a significant contribution over existing work. 


review_title: Official Blind Review #2

final_score: rating: 1: Reject
review_content: The authors propose a method based on GAN  to classify data while automatically removing confounding effects during training, in order to obtain a classifier whose features are not biased by any confounding effect. The proposed idea is based on the extension of classical classification architectures to account for bias prediction. In practice, the parameters of the feature extraction component are updated to solve both bias prediction and the desired classification problem in adversarial fashion. Pearson’s correlation was chosen as a metric for bias.

The paper is interesting and addresses an important problem for the application of machine learning methods in several real context. The feeling is however that the paper should have better explored the implication of the proposed model of bias, and better investigated the relationship with simpler approaches relying on similar hypothesis.

Here are my main comments for this work:

- Why Pearson’s correlation should be a reliable metric to quantify bias? This metric is insensitive to affine scaling of the data, which is a quite common form of bias (for example in medical images).
- The authors should have investigated the relationship between the proposed method and bias removal through canonical correlation analysis (CCA), and perhaps its non-linear variants. At the end this is what their network is doing, although in an end-to-end fashion. Using the CCA projections in the latent space for classification would be the closest approach to the state of the art for bias removal in statistical analysis (residual analysis). 
- The experimental setting illustrated in 4.1 is not clear. In which sense \sigma_A is a common factor for the two groups? Why the theoretical maximum classification accuracy is 90%? Figure 2 is not clear either and doesn’t help understanding the structure of the generated data (e.g. axis labels missing, colorbars units not specified). 
- It is not clear why authors quantify the correlation in the latent space with tSNE projections. tSNE is highly sensitive to the choice of parameters and it would be important to ensure that it was a “fair competition” between all the methods, when showing the results of the dimensionality reduction. This is another modelling step relying on specific assumptions which decreases interpretability of the findings. The authors also proposed to assess the decorrelation of the estimated features throughout the different methods by measuring the squared distance correlation. Naturally, their method is the one which exhibits the best performances using this metric. However, this way of assessing the decorrelation of the features with the biases is unfair to the other methods, as in their case they specifically built their model to avoid statistical correlation between features and biases.  
- Experiment 4.2 has some controversial aspects, as the bias correction is performed on the control population only, while the model is trained on the entire population. I understand the fact that confounding can be estimated only on healthy conditions, however in this case the network is going to be biased by the control group by construction. The effect of such a choice in the end-to-end optimisation scheme is really not clear.
- We also observe that the results of the baseline CNN are very close to the BR-Net. The main difference lies in the fact that the CNN tends to have an unbalanced classification between true negatives and true positives. However, what would happen if we corrected for age before applying the CNN ?
- In the case where the performances of the CNN would be improved, this last question would raise another one. Indeed, if I already know the confounding effects I want to correct for, why wouldn’t I correct them beforehand in order to avoid to train a complex GAN, which leads to more instability during training. This aspect points to the limit of having an online bias-correction (at least for the medical data case).

review_title: Official Blind Review #1

final_score: rating: 8: Accept
review_content: This paper presents Bias-Resilient neural network (BR-Net) that is designed to learn representations that can accurately predict the desired target while being invariant to the confounding covariates in the data. The proposed method is based on domain adversarial training strategies, especially that of (Ganin et al., 2016), where the adversarial component is modified from “loss of distinguishing between the source and target domains” to “the squared Pearson correlation between the ground truth bias covariate and its estimation from the learned representation”. This design is based on the argument that the ultimate goal of bias control here is removing statistical association with respect to the bias variables, as opposed to maximizing the prediction error of them.

Things to improve the paper that did not impact the score:
	- Introduction, line 4: Wrong citation format: use of \citet instead of \citep. Correct for all citations throughout the paper.

References: 
	- Ganin, Y., Ustinova, E., Ajakan, H., Germain, P., Larochelle, H., Laviolette, F., ... & Lempitsky, V. (2016). Domain-adversarial training of neural networks. The Journal of Machine Learning Research, 17(1), 2096-2030.

review_title: Official Blind Review #3

Meta-review: The paper addressed the problem of machine bias when training machine learning models. The authors propose an approach based on representation learning with adversarial training. As opposed to the majority of previous works that trying to create a representation from which it is not possible to predict the sensitive feature (bias), the authors propose to minimize the dependency between the learned features and the sensitive feature with adversarial training. While acknowledging that the proposed model is addressing an important problem and is potentially useful, the reviewers and AC note the following potential weaknesses:  
(1) limited technical contribution -- the proposed approach is similar to a number of works published in machine learning and computer vision before the submission deadline that were overlooked by the authors. Specifically: i) adversarial training for learning fair representations [Edwards and Storkey, Censoring Representations with an Adversary, ICLR 2016], [Beutel, et al 2017, Data decisions and theoretical implications when adversarially learning fair representations], ii) learning fair representation by minimizing the dependency between the latent representation and the sensitive attributes [The variational fair autoencoder, ICLR 2016 by Louizos et al.; Fairness Constraints: Mechanisms for Fair Classification, by Zafar et al, 2015] or by minimizing the mutual information between feature embedding and bias [Learning Not to Learn: Training Deep Neural Networks with Biased Data, CVPR 2019]. 
(2) Limited empirical evidence -- the baseline methods used in the evaluation are not sufficient to assess the benefits of the proposed approach over the existing SOTA methods mentioned above. In fact, none of the baseline methods used in the evaluation tackle machine bias (via adversarial training or minimizing statistical dependence). 
(3) It would be beneficial to also report fairness metrics, e.g. equality of opportunity, statistical parity, to assess the effectiveness of bias removal. R1 has raised some concerns regarding empirical evidence -- see the point about mixed results. Also R2 has reported concerns regarding controversial results in experiment 4.2 and suggested ways to justify when and why the results of the CNNs baseline are close to the BR-Net. Addressing these concerns would strengthen the contributions of the proposed method.    

Among these, (3) did not have a decisive impact on the decision, but would be helpful to address in a subsequent revision. However, (1) and (2) make it very difficult to assess the benefits of the proposed approach, and were viewed by AC as critical issues. AC suggests, in its current state the manuscript is not ready for a publication. We hope the reviews are useful for improving and revising the paper.

Decision: Reject