Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round1

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This paper articulates one important problem in the current fairness community, that *the inconsistencies in the experimentation and dataset setups hinders a fair comparison of bias-mitigating methods*. To remedy this, this paper introduces a framework to benchmark fair representation learning algorithms in a wide variety of scenarios with controllable imbalanced labels and correlated sensitive attributes. The authors release the codebase to contribute to the fairness community, which is appreciated. They also discuss how the different factors may affect the fairness evaluation and the potential sources of bias.
strengths: - The authors release the code to contribute to the fairness community.
- The authors provide a comprehensive examination on 3 fair representation learning algorithms. Their observations could motivate researchers to seek for theoretical explanations.
weaknesses: Although the authors claim that their benchmark is designed for a wide variety of scenarios, it seems that their benchmark only supports binary labels and binary sensitive attributes (if I got wrong, please correct me). The fact that it doesn't support multiple protected groups or even continuous sensitive attributes narrows down the application and the contribution of this work. I am also concerning about whether their benchmark could be generalized to multi-class classification or regression.
correctness: The evaluation methods and experiment design are appropriate and performed correctly.
clarity: The paper is well written and easy to follow up. The rationale for evaluation setup is clearly explained. Although, there are some typos.
relation to prior work: This work discussed how this work differs from previous contributions. The authors emphasis that they evaluate *deep learning-based models*.  I recognize their novelty in fairness benchmark but don't think evaluating deep learning models is rather novel.
additional feedback: The authors have addressed all my concerns. Especially, they added a feature to enable multiple sensitive groups. It will definitely broaden the comparison of constrained optimization algorithms and enhance the significance to the fairness community, which is appreciated. Therefore, I would raise my rating to 7 (the original rating is 6).

------
Questions:

- Why does controlling the position of small boxes help to control the correlation? This needs more details.
- Do you consider controlling the imbalance of both labels $Y$ and sensitive attributes $S$?
review_title: A solid benchmark that may contribute to the fairness community

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: 1) A synthetic image dataset, CI-MNIST (Correlated and Imbalanced MNIST), where each data point has a group membership and a sensitive variable.  Users can set parameters to (i) control relative group size to represent **class imbalance**; and (ii) create **correlations** between a variable (that is sensitive or non-sensitive) and an outcome/eligibility variable. 
Similarly, they propose a modification to an existing real dataset (Adult) so that users can experiment with imbalance and correlation to sensitive variables, though this real data set is not imaging.

2) A protocol / procedure that (i) generates the above data sets with varying degrees of class imbalance and correlations (between sensitive & outcome variables), and (ii) evaluates the SOTA bias-correction models under standard fairness evaluation metrics, for the datasets generated as described. They do detailed discussion of results and comparisons between current best models. 

3) Well-documented code for (i) generating the datasets that are in contribution, (ii) implementing the SOTA models in fairness, and (iii) evaluating models under standard evaluation metrics.
strengths: Relevance: The synthetic dataset is a simple and effective proxy for simulating the class imbalance and correlation to sensitive variables in imaging. It is very understandable and easily communicated. 

The standardization of evaluation is a valuable contribution to allow researchers to compare models fairly, something that seems to be lacking in papers to date. It is especially valuable then that the code is freely available and appears to be well documented and detailed, and that there is some attention paid to the affect of random seeds (especially given the results reported in previous benchmarking studies, such as Locatello, Bauer et al 2019).

A thorough study of the SOTA models under progressively more challenging conditions, illustrating where there is a failure to ensure both fairness and higher accuracy. 

There are comparisons between methods that are disentangling vs methods that impose fairness on the representation. Authors make a novel contribution by combining those models to achieve an incremental improvement in the sota for fairness metrics. 

A discussion section contains interesting insights into the standard fairness models. For example, it shows that bias can be caused by data scarcity. As another example, it proposes a novel method for determining the extent of sensitive information representation in the latent space. 

weaknesses: Significance: having a real world data set is a strength of the paper. But a limitation is that it is not an imaging set. So there may be  doubts about whether a model tested under this framework is shown to be 'fair' in a more general, real-world imaging data setting (though of course it is useful for non-imaging settings). 

Significance: a current limitation of the synthetic dataset is that (at least by default), there are only 2 groups - 'privileged' and 'underprivileged' group. Using only 2 groups is reasonable for a first study, but future work may want to go into more groups, and it would be useful if CI-MNIST can easily handle this case. 

Relevance: the chosen models are mostly adversarial methods (and a vae), but other methods are referenced in the paper that have a reasonable number of citations (this may actually be fine if an argument is made that these methods are the most important methods).



correctness: The synthetic dataset is constructed in a simple and logical way. 
The evaluation methods are reasonable in that they simply use those metrics that are used in the current state-of-the-art models.
My checking of the codebase suggests the implementation is good, but this was a limited exploration. 
clarity: The paper is written logically, and is clear for the most part. 

There are some errors of expression, for example 86, 169-170, 391-393.

Paragraph starting 143 is a bit hard to follow. It's a little wordy, and uses the concepts of 'sensitive', 'priveleged', and 'eligibility' in quick succession in a way that was a bit confusing. It could be phrased more simply. 
relation to prior work: The categorisation of models into pre-processing, in-processing, post-processing is clear. 

It's made clear that this work is focusing on DL approaches (in particular adversarial methods), so as to be distinct from previous reviews on non-DL methods.

This reviewer does not have deep-enough knowledge of this field to judge whether the models that are chosen for the empirical study really are the SOTA. There is also not enough knowledge to say whether the related work section is missing important papers. 
additional feedback: Figure 1 has a column called "after shade", but that isn't explained in the figure caption, and the word "shade" does not appear in the main document.
review_title: Useful synthetic dataset & benchmarking procedure for evaluating fairness-enforcing dl methods.(modified: 19 Jul 2021)

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This paper evaluates the representations learned by 3000 different deep learning fairness methods on common datasets. It finds that model bias increases as datasets become more imbalanced or when dataset attributes become more correlated. It also finds that sensitive information remains in the learned representations regardless of the bias-mitigation algorithms.

The three main findings are: (1) imbalance or scarce underrepresented groups induce bias, (2) correlations between sensitive attribute and eligibility induce bias, (3) some model sare susceptible to bias from random seed initializations

strengths: Overall, I like the paper. It does a good job of designing different datasets and stress testing common fairness methods. The recognize and appreciate the amount of work that goes into working on a paper like this. I also really enjoyed reading the discussion section - it’s a vital component of this paper and lays out opportunities for future research. I think having such a benchmark would be beneficial for the community.

weaknesses: I like the paper but find it hard to appreciate how much the findings “actually matter”. The paper shows that bias-mitigation methods don’t work as well as dataset bias increases. But by what magnitude? How biased do datasets have to be to render bias-mitigation algorithms fail? What I think is missing is a stronger grounding between the dataset design choices and real world use cases. Let me explain this in a bit more detail:

What makes the results, and therefore the experiments section, hard to appreciate is the difficulty in deciphering to what degree existing bias-mitigation methods actually fail. What does it actually mean to test with an age-ratio of (0.06, 0.36)? Or with clr-ratio of (0.01, 0.99)? Without a clear explanation of how these ratios map onto ecologically valid situations that might arise in real world use cases, it’s hard to appreciate why it matters. 

On a similar vein, the raw numbers of all these fairness metrics are fine to report. But I wish the paper instead presented a relative decrease in these metrics from the balanced training set scenario. I wish the main paper had a plot where the x axis was the ratio change and the y axis was the relative decrease in the fairness metric from a balanced training dataset. A relative change, with an explanation of what that decrease implies, would have made the results much more impactful.

correctness: Yes, they appear correct to me.
clarity: Line 6: “into the working 6 of these methods”. Weird phrasing.

“Eligibility” and “sensitive variable” are defined later in the methods section and are general well known amongst people who work in fairness. But it’s worth defining these terms when you first use them in the introduction section.

Figures are really hard to read. The axes and labels need to be larger in font.

relation to prior work: The paper does a good job of laying out the related work.
additional feedback: NA
review_title: Good paper but I wish the results were presented better

Meta-review: This paper introduces a framework for benchmarking fair representation learning methods that systematically controls dataset imbalances and correlations. While the toy and synthetic nature of the dataset limits the conclusions that can be drawn, reviewers agree there is significant value in having a standardized and controllable dataset for benchmarking purposes. 

Reviewers are in agreement that this paper is a valuable contribution to this track and addresses an important open problem within algorithmic fairness. 

Decision: Accept