Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper proposes a synthetic dataset to use for benchmarking representation learning algorithms on a variety of fairness criteria. Using this benchmark, it compares several recently developed and prominent methods for fair representation learning and shows their relative strengths and weaknesses under different data generation assumptions and for different metrics.

The type of careful, systematic comparison provided by this paper is potentially an important contribution. With so many fairness metrics and methods to choose from, it is important for the community to understand limitations and under what sort of conditions a given method may be appropriate or perform best.

However, I recommend rejection for this paper for one main reason: I believe the synthetic dataset is inadequate for use as a realistic benchmark. It is too abstract and unlike examples where people are concerned about fairness, and the protected attribute--background color--is something which can be encoded far too simply. As a result, I believe that the relative performance of methods evaluated on this benchmark dataset may be not be indicative of their relative performance on a more realistic dataset with more complex sensitive attributes like race and gender.

It is not obvious how to improve this flaw, since more realistic datasets may also be less general and hence less useful for benchmarking purposes. One possibility is to abandon the goal of providing a benchmark, since fairness considerations may be too problem dependent for this anyway. In this way, the paper could propose a systematic comparison of methods for a particular synthetic task, provided that task is realistic enough to be related to some potential real world applications. I have no specific suggestion for how to modify the synthetic data generation process, but the features, especially the sensitive attribute(s), need to be rich enough to be more like a real fairness application.
review_title: Review of benchmarking bias mitigation in representation learning

final_score: rating: 5: Marginally below acceptance threshold
review_content: 1. Summarize what the paper claims to contribute. Be positive and generous.
The paper claims to contribute the following:
  (1) Show that models exploit any correlation between eligibility and the sensitive attribute found in training data even if the test data does not have such correlation; This leads to unfair predictions.
  (2) Show that models can exploit a correlation between non-sensitive attribute or small visual features and the eligibility in the dataset; This leads to unfair predictions.
  (3) Demonstrate that the choice of seed can affect the results significantly; the impact varies considerably across different models.
  (4) Provide a deep learning codebase composed of six debiasing models and a baseline.
  (5) Provide a dataset with different controllable sets of features and correlation among them.
Agreed that there isa  void of a common framework. Thanks for your proposed contributions with the paper!

2. List strong and weak points of the paper. Be as comprehensive as possible.
  (1) Strengths
    a. Exhaustive analyses and graphs. I especially liked Figure 8 (Correlation between dataset feature and model's prediction). I enjoyed reading the qualitative analyses too.
  (2) Weaknesses
    a. Sensitive attribute values are limited to binary values. In contrast, the real-world sensitive attributes are multi-label.
    b. What type of sensitive attribute does the variation of the number of stripes represent? I can see that the color may represent skin tone differences and that the small pixel changes may represent the size of the correlation/bias in the input image. I am not sure what stripes may represent.

3. Clearly state your recommendation (accept or reject) with one or two key reasons for this choice.
Accept because the paper helps the reader better understand and compare the prior arts in ML fairness and their practical performance against various fairness metrics. Would like to understand the answer to 2.(1).b (above) better though.

4. Provide supporting arguments for your recommendation.
See 2.(1) and 3 above.

5. Ask questions you would like answered by the authors to help you clarify your understanding of the paper and provide the additional evidence you need to be confident in your assessment. 
See 2.(2) above.

6. Provide additional feedback with the aim to improve the paper. Make it clear that these points are here to help, and not necessarily part of your decision assessment.
Just as FYI, it was difficult to read the labels of the third graph of Figure 2 when printed on A4 paper. I had to revisit the PDF and zoom in to see that the graph title is "clr-ratio(0.01, 0.01)" instead of "clr-ratio(0.03, 0.03)". Would be good if those titles are repeated in the Figure 2 description to help with reading.


**Final Update**
After reading other reviewers' comments and authors' responses, I decided to choose to reject the paper at this time for a few reasons.

1. I am afraid that the impact of the paper won't be good with only one out of the four reviewers understanding a paper meant to present an entry point benchmarking dataset.

2. I agree the binary, pseudo sensitive attributes, such as stripes, make the benchmarking and the analysis more manageable. However, as a future user of the benchmarking dataset, I see that there still remains the question of whether the metrics measured against the proposed benchmarking dataset with simple pseudo sensitive attributes can translate well to the model performance measured against some real world dataset with real sensitive attributes. It will help to add the best or chosen model's performance against some real world dataset with real and more complicated sensitive attributes.

3. Add something like what is proposed in #2 to help the readers understand how to choose the best model and make tradeoffs for a given scenario using the presented dataset with the proposed methods. The goal would be somewhat similar to that of the "INTENDED USES" section of the LFW dataset presented in http://vis-www.cs.umass.edu/papers/lfw.pdf.

Nevertheless, I believe the authors' core work is in the right direction. Just need some way to organize and present it better to make a bigger impact. Thank you for your work!
review_title: Like the various analyses and the idea of common benchmark framework for MLFairness

final_score: rating: 5: Marginally below acceptance threshold
review_content: ############# Summary of contributions ##############

This experimental paper proposes a method for simulating a dataset to evaluate algorithms for enforcing fairness criteria on deep learning models. The proposed simulated dataset pertubs MNIST in various ways to create data distributions that could impact the difficulty of satisfying various fairness criteria. The paper then uses this simulated dataset to empirically evaluate three existing approaches for enforcing fairness on deep learning models using adversarial techniques. For each approach, they evaluate group-based fairness criteria such as demographic parity, equal opportunity, and equal odds under different dataset simulations. 

The most potentially impactful contribution of this paper is the simulated dataset and the comparison of different approaches under different dataset simulations. 

############# Strengths ##############

- Their proposed simulated modifications of MNIST are well documented. 

- The methodology of stress-testing different algorithms by tuning different aspects of the simulated data distribution seems intuitive and useful.

- The experiments are well structured, and they provide a mostly clear explanation for tuning each of the dataset simulation parameters. 

- While they only evaluate a few group-based fairness criteria in their experiments, they provide an implementation to evaluate many more metrics listed in Table 2 in the appendix.

- The experiments provided seem thorough, with many different dataset simulation parameters. 

############# Weaknesses ##############

- The dataset features seem a bit contrived, and it’s not entirely clear how the different simulated features correspond to real data scenarios. For example, as one modification of MNIST, they add 6-20 drawn horizontal lines on some digits, and 2-6 drawn lines on the rest of the digits. They modify how many of the 6-20 line digits are odd, and how many are even, such that the number of lines on the digit can correlate with whether the digit is even or odd. It’s not clear to me if there are real image datasets where something like this would occur. Of course, I recognize that it’s impossible to capture every possible real data distribution in a simulation like this, but it would be helpful to have some explanation of why they use horizontal lines specifically. A similar explanation would be helpful for the use of the green column.

- They only use a binary sensitive attribute, encoded in the simulation as various shades of red/blue. It would be nice to see an example with more than two categories for the sensitive attribute, as the difficulty of satisfying many of these group-based fairness measures can increase with more categories.  

- The related work has limited discussion of other papers that evaluate multiple fairness criteria and algorithms. The closest citation is Friedler et al. 2019, but they do not discuss or compare to the findings of Friedler et al 2019, which also evaluates four different algorithms for enforcing group-based fairness. Specifically, this paper should discuss how their conclusions and evaluations relate to the conclusions and evaluations of Friedler et al. 2019. (As a concrete example, the Stability analysis in Section 7.2 of Friedler et al. 2019 is very similar to this paper’s “Impact of seed” analysis in the Experimental Results section.)

- The paper would benefit from deeper discussion of how or why the three evaluated approaches differ in performance. Otherwise, the experimental results are not particularly novel (for example, the fact that a correlation between the sensitive attribute and the label can lead to unfair predictions is well known). 

############# Recommendation ##############

Overall, my recommendation is 5: Marginally below acceptance threshold. While this paper has a nicely organized experimental section and a clear methodology for simulating a dataset, I don’t think there’s enough discussion of the dataset features and enough discussion or novelty in the experimental results themselves. I do think this type of survey work can be impactful, and would encourage the authors to add more discussion of the methodology, results, and related work, as suggested above.

############# Questions and clarifications ##############

- The dataset feature g_2 is introduced in Section 4.1, but not mentioned in experiments. Is g_2 used in any experiments? 

- What were the sizes of the train, validation, and test sets? 

- How many seeds were used for each experiment? For example, in the “Impact of seed” experiment, how many seeds were used to compute the standard deviation? 

- In Section 5, the authors say, “For each metric, the best value is reported, meaning the performance of the best-performing model on the fairness metric (chosen across hyper-parameters on the validation set) is reported on the test set...This would allow us to report the best performance on each metric, without choosing how to compromise between accuracy and fairness metric.” I don’t understand what this means. Does this mean that they chose a different model for each metric reported? Did they choose one model for the equal opportunity metric, and a different model for the accuracy metric? 

- On page 7, the authors say “LAFTR models drop performance in the biased setup of g_1 = 2.” Can the authors clarify what they mean by “drop performance”? Which metric is hurt? And why is g_1=2 considered to be a “biased” setup? It seems the g_1=2 is simply a setting when it is easy to identify odd from even numbers -- I don’t understand why the authors frame this as “biased.”

############# Additional feedback ##############

- The related work section is currently not organized well. I would suggest breaking the large block of text on page 2 into multiple paragraphs addressing different aspects of related work. For example, have one paragraph discussing different methods for enforcing fairness on deep learning models, and have a different paragraph discussing other survey papers for evaluating various fairness criteria and algorithms.

- The plots in the Experimental Results section are really tiny and hard to read (Figures 2-4). 

- [page 6, “The goal is to evaluate when such cases arise, do models perform fairly in test setups, where everything is completely balanced?”] It’s great to outline the goal so clearly, but this isn’t actually a grammatically correct sentence. I would suggest modifying this (and similar other sentences in the paper) to: “The goal is to evaluate the following: when such cases arise, do models perform fairly in test setups, where everything is completely balanced?”

review_title: Interesting and well documented dataset simulation, but lacking in discussion.

final_score: rating: 4: Ok but not good enough - rejection
review_content: #Summary:

This paper presents an interesting benchmark study on three bias-mitigation algorithms: 
- LAFTR (Madras et al., 2018), an adversarial method for learning fair and transferable representations;
- CFAIR (Zhao et al., 2020), conditional learning of fair representations;
- FFVAE (Creager et al., 2019), fair representation learning by disentanglement.

The experiments are conducted on a synthetic dataset (colored-MNIST), with various interventions to control the bias (background color, number of lines drawn on the digit, green pixels added with different widths and locations). Empirical comparison over the three bias-mitigation methods are reported.

#Pros:
- Due to the increased popularity of bias-mitigation methods recently, there is a need to benchmark them carefully and fairly. This paper presents a preliminary effort towards better comparing those algorithms under a unified framework.
- The proposed experimental framework is very interesting, by using a colored-MNIST dataset with various controllable interventions: 1) background color; 2) number of lines drawn on the digit; 3) green pixels added with different width and locations. Those interventions represent clues with various difficulties for the model to pick on, thus would be useful in understanding how sensitive bias-mitigation algorithms are under different situations.

#Cons:
- The most important thing lacking for this paper is a *deeper* understanding of the results based on the empirical observations. This paper seems to merely present the empirical results, i.e., under different settings which method works better, without further insights on why certain methods work under certain conditions. 
(1) For example, for LAFTR, since the fairness metric (DP, EqOpp0/1, EqOdd) is already built into the objective function, does this affect how this method performs when evaluated over different fairness metrics? 
(2) When changing the e-o-ratio, is there a reason for the LAFTR-DP metric to drop and LAFTR-EqOpp0 metric to improve? 
Currently the paper reads like different methods achieve different trade-offs under various fairness metrics, and it is hard to learn any practical lessons without further analysis over those empirical results.
- This might be a stretch but as a benchmark paper, I would expect to see comparisons of more methods and more datasets. Method-wise, I don't see a clear argument on how the three methods presented in this paper are chosen, is it one method each from the three representation-learning categories? Some clarification would be useful. 
Dataset-wise, it would also be interesting to see the comparison over datasets other than vision (e.g., NLP datasets, or some of the classic fairness datasets with numerical/categorical features like UCI adult), to gain a better understanding of how those methods perform under different types of feature input. 


#Overall Recommendation

Overall I tend to reject this paper, as this paper can benefit a lot from a deeper analysis on *why* different methods work under different conditions, and more studies on different datasets covering wider applications (e.g., NLP). It would be more useful if the paper can give more practical recommendations on which bias-mitigation method to choose under a certain scenario. 


#Minor comments

- The figures' readability can be improved. Some of the figures are really small to read, some of the error bars are truncated, and in some cases the bar plots for certain methods are missing (if it's because the metric achieves zero value please clarify).
- For experimental comparison, some of the error bars on the bar plot are really large. I'm not sure what's the particular reason causing this, could the authors do a deeper analysis on what are the factors? Is it purely because of the instability of a method, or because of the size of the training data, or the number of seeds, or it's due to the intervention added? Currently it's very hard to tell which method works reliably under which scenario due to the significance of the results.
- It might be worth to move some of the details from appendix to the main text, currently the basic details of the experimental setting is unclear to me based only on the main text.

#Post rebuttal

I appreciate the authors added a deeper discussion and analysis in their newest uploaded draft, which helps clarify some of my concerns. However, it would be better if the following can be better addressed:

- As a benchmark paper the experimental setting is relatively simple/artificial, more realistic datasets/settings would be more useful;
- Thanks for adding the stability study. It would be more helpful if the main results can be presented with a better control over the stability. Currently it is still hard to tell the statistical significance of the results.
- As the other reviewers have mentioned, it is still unclear how this simulated setting connects to real-world fairness applications.

review_title: A benchmark paper comparing representation-learning based bias-mitigation methods

Meta-review: The paper studies benchmarking of bias mitigation methods. The authors propose a synthetic dataset of images (alike colored-MNIST) that enables a controlled setup over different types of correlations between a binary sensitive attribute, dataset features, and a binary outcome label. The authors have evaluated 2K models that are the variants of three recently proposed debiasing methods using fair representation learning across various settings.

While the reviewers acknowledged the importance of benchmarking fair learning methods in a systematic controlled setting, they have raised several concerns: 
(1) the proposed benchmark is too abstract/unrealistic (R4, R2, R3); it is not clear whether the findings from this benchmark can be generalized to real-world data with real sensitive features, (2) the proposed benchmark is limited to pseudo sensitive attributes (R1) that are binary (R1, R2), (3) the paper lacks in-depth analysis on why certain methods work under certain conditions (R3). Among these, (2) did not have a major impact on the decision, but would be helpful to address in a subsequent revision, (3) was partially addressed in the rebuttal. However, (1) makes it very difficult to assess the benefits of the proposed benchmark, and was viewed by AC as a critical issue.

The authors provided a detailed rebuttal addressing multiple of the reviewers’ concerns. AC can confirm that all four reviewers have read the author responses and have contributed to the discussion. A general consensus among reviewers and AC suggests, in its current state the manuscript is not ready for a publication. See R1 post-rebuttal encouragement and suggestions how to strengthen the work. We hope the detailed reviews are useful for improving the paper.

Decision: Reject