Benchmarking Bias Mitigation Algorithms in Representation Learning through Fairness Metrics

Abstract: With the recent expanding attention of machine learning researchers and practitioners to fairness, there is a void of a common framework to analyze and compare the capabilities of proposed models in deep representation learning. In this paper, we evaluate different fairness methods trained with deep neural networks on a common synthetic dataset to obtain a better insight into the working of these methods. In particular, we train about 2000 different models in various setups, including unbalanced and correlated data configurations, to verify the limits of the current models and better understand in which setups they are subject to failure. In doing so we present a dataset, a large subset of proposed fairness metrics in the literature, and rigorously evaluate recent promising debiasing algorithms in a common framework hoping the research community would take this benchmark as a common entry point for fair deep learning.

Reviews and author responses:

#Summary:

This paper presents an interesting benchmark study on three bias-mitigation algorithms: 
- LAFTR (Madras et al., 2018), an adversarial method for learning fair and transferable representations;
- CFAIR (Zhao et al., 2020), conditional learning of fair representations;
- FFVAE (Creager et al., 2019), fair representation learning by disentanglement.

The experiments are conducted on a synthetic dataset (colored-MNIST), with various interventions to control the bias (background color, number of lines drawn on the digit, green pixels added with different widths and locations). Empirical comparison over the three bias-mitigation methods are reported.

#Pros:
- Due to the increased popularity of bias-mitigation methods recently, there is a need to benchmark them carefully and fairly. This paper presents a preliminary effort towards better comparing those algorithms under a unified framework.
- The proposed experimental framework is very interesting, by using a colored-MNIST dataset with various controllable interventions: 1) background color; 2) number of lines drawn on the digit; 3) green pixels added with different width and locations. Those interventions represent clues with various difficulties for the model to pick on, thus would be useful in understanding how sensitive bias-mitigation algorithms are under different situations.

#Cons:
- The most important thing lacking for this paper is a *deeper* understanding of the results based on the empirical observations. This paper seems to merely present the empirical results, i.e., under different settings which method works better, without further insights on why certain methods work under certain conditions. 
(1) For example, for LAFTR, since the fairness metric (DP, EqOpp0/1, EqOdd) is already built into the objective function, does this affect how this method performs when evaluated over different fairness metrics? 
(2) When changing the e-o-ratio, is there a reason for the LAFTR-DP metric to drop and LAFTR-EqOpp0 metric to improve? 
Currently the paper reads like different methods achieve different trade-offs under various fairness metrics, and it is hard to learn any practical lessons without further analysis over those empirical results.
- This might be a stretch but as a benchmark paper, I would expect to see comparisons of more methods and more datasets. Method-wise, I don't see a clear argument on how the three methods presented in this paper are chosen, is it one method each from the three representation-learning categories? Some clarification would be useful. 
Dataset-wise, it would also be interesting to see the comparison over datasets other than vision (e.g., NLP datasets, or some of the classic fairness datasets with numerical/categorical features like UCI adult), to gain a better understanding of how those methods perform under different types of feature input. 


#Overall Recommendation

Overall I tend to reject this paper, as this paper can benefit a lot from a deeper analysis on *why* different methods work under different conditions, and more studies on different datasets covering wider applications (e.g., NLP). It would be more useful if the paper can give more practical recommendations on which bias-mitigation method to choose under a certain scenario. 


#Minor comments

- The figures' readability can be improved. Some of the figures are really small to read, some of the error bars are truncated, and in some cases the bar plots for certain methods are missing (if it's because the metric achieves zero value please clarify).
- For experimental comparison, some of the error bars on the bar plot are really large. I'm not sure what's the particular reason causing this, could the authors do a deeper analysis on what are the factors? Is it purely because of the instability of a method, or because of the size of the training data, or the number of seeds, or it's due to the intervention added? Currently it's very hard to tell which method works reliably under which scenario due to the significance of the results.
- It might be worth to move some of the details from appendix to the main text, currently the basic details of the experimental setting is unclear to me based only on the main text.

#Post rebuttal

I appreciate the authors added a deeper discussion and analysis in their newest uploaded draft, which helps clarify some of my concerns. However, it would be better if the following can be better addressed:

- As a benchmark paper the experimental setting is relatively simple/artificial, more realistic datasets/settings would be more useful;
- Thanks for adding the stability study. It would be more helpful if the main results can be presented with a better control over the stability. Currently it is still hard to tell the statistical significance of the results.
- As the other reviewers have mentioned, it is still unclear how this simulated setting connects to real-world fairness applications.


1. Summarize what the paper claims to contribute. Be positive and generous.
The paper claims to contribute the following:
  (1) Show that models exploit any correlation between eligibility and the sensitive attribute found in training data even if the test data does not have such correlation; This leads to unfair predictions.
  (2) Show that models can exploit a correlation between non-sensitive attribute or small visual features and the eligibility in the dataset; This leads to unfair predictions.
  (3) Demonstrate that the choice of seed can affect the results significantly; the impact varies considerably across different models.
  (4) Provide a deep learning codebase composed of six debiasing models and a baseline.
  (5) Provide a dataset with different controllable sets of features and correlation among them.
Agreed that there isa  void of a common framework. Thanks for your proposed contributions with the paper!

2. List strong and weak points of the paper. Be as comprehensive as possible.
  (1) Strengths
    a. Exhaustive analyses and graphs. I especially liked Figure 8 (Correlation between dataset feature and model's prediction). I enjoyed reading the qualitative analyses too.
  (2) Weaknesses
    a. Sensitive attribute values are limited to binary values. In contrast, the real-world sensitive attributes are multi-label.
    b. What type of sensitive attribute does the variation of the number of stripes represent? I can see that the color may represent skin tone differences and that the small pixel changes may represent the size of the correlation/bias in the input image. I am not sure what stripes may represent.

3. Clearly state your recommendation (accept or reject) with one or two key reasons for this choice.
Accept because the paper helps the reader better understand and compare the prior arts in ML fairness and their practical performance against various fairness metrics. Would like to understand the answer to 2.(1).b (above) better though.

4. Provide supporting arguments for your recommendation.
See 2.(1) and 3 above.

5. Ask questions you would like answered by the authors to help you clarify your understanding of the paper and provide the additional evidence you need to be confident in your assessment. 
See 2.(2) above.

6. Provide additional feedback with the aim to improve the paper. Make it clear that these points are here to help, and not necessarily part of your decision assessment.
Just as FYI, it was difficult to read the labels of the third graph of Figure 2 when printed on A4 paper. I had to revisit the PDF and zoom in to see that the graph title is "clr-ratio(0.01, 0.01)" instead of "clr-ratio(0.03, 0.03)". Would be good if those titles are repeated in the Figure 2 description to help with reading.


**Final Update**
After reading other reviewers' comments and authors' responses, I decided to choose to reject the paper at this time for a few reasons.

1. I am afraid that the impact of the paper won't be good with only one out of the four reviewers understanding a paper meant to present an entry point benchmarking dataset.

2. I agree the binary, pseudo sensitive attributes, such as stripes, make the benchmarking and the analysis more manageable. However, as a future user of the benchmarking dataset, I see that there still remains the question of whether the metrics measured against the proposed benchmarking dataset with simple pseudo sensitive attributes can translate well to the model performance measured against some real world dataset with real sensitive attributes. It will help to add the best or chosen model's performance against some real world dataset with real and more complicated sensitive attributes.

3. Add something like what is proposed in #2 to help the readers understand how to choose the best model and make tradeoffs for a given scenario using the presented dataset with the proposed methods. The goal would be somewhat similar to that of the "INTENDED USES" section of the LFW dataset presented in http://vis-www.cs.umass.edu/papers/lfw.pdf.

Nevertheless, I believe the authors' core work is in the right direction. Just need some way to organize and present it better to make a bigger impact. Thank you for your work!

We thank the reviewer for their thoughtful comments on the complexity of the problem of measuring bias / fairness, and the resulting difficulty of presenting a benchmark. They correctly point out that real world protected attributes can be far more complex. The usage of synthetic datasets has been extensively leveraged in the machine learning community and has served as a starting point for evaluating methods and the proposed concepts. Examples of synthetic data are MNIST, CLEVR [1], 3DChairs[2], shapes3D [3], dsprites [4], Synthetic hands[5], SMPL body pose and shape dataset [6] among many others. An example of model analysis [7], which is done only on synthetic data, and obtained the best paper award in ICML 2019. We are not the first ones to leverage a synthetic dataset. They have been used to verify the concepts in a setting that is controllable and does not have the complexity of real datasets. In debiasing algorithms, if a model cannot handle correlations that are the source of bias in a simple setting, there is no guarantee that it will handle it in other scenarios. Another point that should be considered is the feasibility of creating different unbalanced setups without changing any attribute rather than the attribute under study. In realistic datasets, this is almost impossible. In a synthetic dataset, different features can be easily tuned, which is not the case in real dataset, as we cannot control the data generation process. Hence, it is not trivial to change only one factor while keeping all other factors unchanged. We believe our proposed dataset can serve among the first efforts (if not the first) in verifying the performance of the debiasing models under different setups. Furthermore, the biases in our dataset are ‘transparent’, helping researchers pin-point potential deficiencies in their model’s performance.  

[1] Johnson, Hariharan, Van der Maaten, Fei-Fei, Zitnick, and Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning, CVPR 2017.

[2] Aubry, M., Maturana, D., Efros, A. A., Russell, B. C., and Sivic, J. Seeing 3D chairs: exemplar part-based 2D-3D alignment using a large dataset of cad models. In CVPR, 2014.

[3] Burgess and Kim. 3dshapes18, https://github.com/deepmind/3dshapes-dataset/, 2018.

[4] Matthe, Higgins, Hassabis, and Lerchner. dSprites: Disentanglement testing Sprites dataset, 2017.

[5] F. Mueller, D. Mehta, O. Sotnychenko, S. Sridhar, D. Casas, and C. Theobalt. Real-time Hand Tracking under Occlusion from an Egocentric RGB-D Sensor, ICCV 2017.

[6] Gül Varol, Javier Romero, Xavier Martin, Naureen Mahmood, Michael J. Black, Ivan Laptev, Cordelia Schmid. Learning from Synthetic Humans, CVPR 2017.

[7] Locatello, F., Bauer, S., Lucic, M., Raetsch, G., Gelly, S., Schölkopf, B., & Bachem, O. Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations, ICML 2019.

We thank the reviewers for their response, we address here the main issues raised by the reviewers:

**1- Deeper analysis on the results, why the three evaluated approaches differ in performance? what causes models performance to change? what are the source of instability in the results? (Reviewers #2, #3)**

Thank you for bringing up this issue. We have added a Discussion section to our paper. This Section discusses in more details the biases introduced in our framework and their impact in addition to the source of instability in different models.


**2- Why using a synthetic dataset? (Reviewer #4)**

We thank the reviewer for their thoughtful comments on the complexity of the problem of measuring bias / fairness, and the resulting difficulty of presenting a benchmark. They correctly point out that real world protected attributes can be far more complex. The usage of synthetic datasets has been extensively leveraged in the machine learning community and has served as a starting point for evaluating methods and the proposed concepts. Examples are MNIST, CLEVR [1], 3DChairs[2], shapes3D [3], dsprites [4], Synthetic hands[5], SMPL body pose and shape dataset [6] datasets among many others. An example is the study done in [7] on disentanglement, which is done only on synthetic data, and obtained the best paper award in ICML 2019. We are not the first ones to leverage a synthetic dataset. They have been used to verify the concepts in a setting that is controllable and does not have the complexity of real datasets. In debiasing algorithms, if a model cannot handle correlations that are the source of bias in a simple setting, there is no guarantee that it will handle it in other scenarios. Another point that should be considered is the feasibility of creating different unbalanced setups without changing any attribute rather than the attribute under study. In realistic datasets, this is almost impossible. In a synthetic dataset, different features can be easily tuned, which is not the case in real dataset, as we cannot control the data generation process. Hence, it is not trivial to change only one factor while keeping all other factors unchanged. We believe our proposed dataset can serve among the first efforts (if not the first) in verifying the performance of the debiasing models under different setups. Furthermore, the biases in our dataset are ‘transparent’, helping researchers pin-point potential deficiencies in their model’s performance.  

[1] Johnson, Hariharan, Van der Maaten, Fei-Fei, Zitnick, and Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning, CVPR 2017.

[2] Aubry, M., Maturana, D., Efros, A. A., Russell, B. C., and Sivic, J. Seeing 3D chairs: exemplar part-based 2D-3D alignment using a large dataset of cad models. In CVPR, 2014.

[3] Burgess and Kim. 3dshapes18, https://github.com/deepmind/3dshapes-dataset/, 2018.

[4] Matthe, Higgins, Hassabis, and Lerchner. dSprites: Disentanglement testing Sprites dataset, 2017.

[5] F. Mueller, D. Mehta, O. Sotnychenko, S. Sridhar, D. Casas, and C. Theobalt. Real-time Hand Tracking under Occlusion from an Egocentric RGB-D Sensor, ICCV 2017.

[6] Gül Varol, Javier Romero, Xavier Martin, Naureen Mahmood, Michael J. Black, Ivan Laptev, Cordelia Schmid. Learning from Synthetic Humans, CVPR 2017.

[7] Locatello, F., Bauer, S., Lucic, M., Raetsch, G., Gelly, S., Schölkopf, B., & Bachem, O. Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations, ICML 2019.


**3- Why not using multi-valued data or NLP data for evaluation?  (reviewers #1, #3)**

While multi-valued sensitive attributes can be also an interesting evaluation setup, since the majority of the proposed methods did not focus on this setting, we decided to focus on the mostly-studied case, and instead study it more thoroughly. This is the reason for evaluating only binary values in our work. We believe multi-valued scenarios can be the focus of a paper concentrated on this setting and out of focus of our work.
Regarding dataset variations such as NLP, while evaluation of models in different setups is interesting, the number of experiments that should be run is extensive. In our setup, to be fair to different models, we had to run over a big set of hyper-parameters. In particular, we had to train about 2,000 models, which took a long time to complete. Our goal was to show that the models are subject to bias even in simplistic setups and show it in our experiments.


We thank reviewer for their feedback. Please see below answers to the major raised issues:

**The most important thing lacking for this paper is a deeper understanding of the results based on the empirical observations. This paper seems to merely present the empirical results, i.e., under different settings which method works better, without further insights on why certain methods work under certain conditions.
(1) For example, for LAFTR, since the fairness metric (DP, EqOpp0/1, EqOdd) is already built into the objective function, does this affect how this method performs when evaluated over different fairness metrics?
(2) When changing the e-o-ratio, is there a reason for the LAFTR-DP metric to drop and LAFTR-EqOpp0 metric to improve? Currently the paper reads like different methods achieve different trade-offs under various fairness metrics, and it is hard to learn any practical lessons without further analysis over those empirical results.**


1) In general, as can also be observed in the tables and plots, LAFTR models did not perform reliably in different settings. Even among different LAFTR variants, the LAFTR model with a given fairness criteria introduced into its objective function, was not always the LAFTR variant performing the best on that particular metric. This can be also observed in the  Spearman correlation plot in Fig. 8, where LAFTR models show a higher correlation with input variables compared to other models.

2) This is due to instability of the models. As can be also observed in the plots, the LAFTR models show a high variance among runs with different seeds. So, we believe such changes indicate the instability of the model under small variations. 

We will clarify it in the paper. 

**This might be a stretch but as a benchmark paper, I would expect to see comparisons of more methods and more datasets. Method-wise, I don't see a clear argument on how the three methods presented in this paper are chosen, is it one method each from the three representation-learning categories? Some clarification would be useful. Dataset-wise, it would also be interesting to see the comparison over datasets other than vision (e.g., NLP datasets, or some of the classic fairness datasets with numerical/categorical features like UCI adult), to gain a better understanding of how those methods perform under different types of feature input.**

- We used methods only in the deep learning framework and due to the emergence of the adversarial methods in many debiasing algorithms, we focused on these methods. To choose models, while it was not possible to evaluate all models, we checked the proceedings of the recent ICML, ICLR, FAT*, and NeurIPS, and chose some of the highly cited papers. As mentioned in Section 3, we also considered another highly cited work, Mubal (Zhang et al., 2018), however, due to its high instability even under various hyper-parameter search, we dropped it from the results.

- Regarding dataset variations, while evaluation of models in different setups is interesting, the number of experiments that should be run is extensive. In our setup, to be fair to different models, we had to run over a big set of hyper-parameters. In particular, we had to train about 2,000 models, which took a long time to complete. Our goal was to show that the models are subject to bias even in simplistic setups and show it in our experiments.

**Overall I tend to reject this paper, as this paper can benefit a lot from a deeper analysis**

We will add a Discussion section to the paper, please check it.

**For experimental comparison, some of the error bars on the bar plot are really large. I'm not sure what's the particular reason causing this, could the authors do a deeper analysis on what are the factors? Is it purely because of the instability of a method, or because of the size of the training data, or the number of seeds, or it's due to the intervention added?**

For methods that have high variance, such as LAFTR, it is due to instability of the method that obtains different results under different seeds even in balanced scenarios. However, in extreme cases such as clr-ratio (.001, .001) or (0.01, 0.99) (the extreme data setup which causes small portions of data to be in some category and a large portion of data in another category), the unbalanced nature of data further causes instability of such models.

**and in some cases the bar plots for certain methods are missing (if it's because the metric achieves zero value please clarify).**

Regarding zero bar charts, in some cases bar plot values are zero, which imply biased predictions on the balanced test set.




We thank reviewer for their feedback. Please see below answers to the major raised issues:

**The dataset features seem a bit contrived...**

Our goal was to create different features on images with different sizes and various appearance formats. While changing the background image changes most of the pixels in the image, we also wanted to observe the impact of smaller visual features. This is the reason for using horizontal lines and also green vertical lines. To provide an example, one can think of skin color as a feature that affects most of the pixels observed on an image of a face, while facial wrinkles or someone’s emotion affect smaller positions of face. For example, if someone is angry, mostly the eyebrows are affected or if someone is smiling, the lips are mostly affected. We wanted to observe if some feature is big or small, how much it impacts the model’s level of fairness. The background is a feature that affects a big region of the image, while horizontal and vertical lines, in order, affect smaller regions of the image. Any other type of visualization could have been considered to observe the impact of the size of the visual features.

**They only use a binary sensitive attribute...**

Please check the response given to reviewer #1

**The related work has limited discussion...**

There are a couple of differences between our work and the referenced paper:
1) None of the models evaluated in [Friedler et al. 2019] are deep learning models. In particular, they evaluate logistic regression, svm, decision trees, and naive bayes. Hence, as far as we know, our work is the first evaluation of deep learning models for fairness.

2) Friedler et al. evaluate the models in only one setting of each dataset, so there is no measure of how the performance of the models change under different levels of bias or correlation among features. This is where our proposed dataset shows its impact in providing different fairness setups. 

3) The stability measurement reported in [Friedler et al.] is done by observing standard deviation of models trained/evaluated with different splits of train/test sets, while we report the standard deviation of models trained with different seeds. The former, known as cross-validation, measures how much a model generalises to a different test data, while the latter measures the impact of random sampling used in training models, e.g. in parameters initialization and sampling during training. These two measurements can be complementary evaluations of stability of a model. 

We will add the above-mentioned information to the paper.

**The paper would benefit from deeper discussion...**

Thanks for bringing up this point. In our experiments, we observed LAFTR to be the least stable model, observed both in terms of variation of results given different seeds and also in different experimental setups. We believe this is due to application of both accuracy and adversarial loss to the same latent space, which causes a competition between the two losses. This is the main reason for instability of this model. CFAIR has a more stable learning due to using two adversarial networks for two eligibility classes and using a balanced error rate as its loss function. Finally, FFVAE, obtains the most stable performance due to disentangling of attributes into sensitive and non-sensitive and also decoding features, which helps the model capture all of the information in the input image, which is not the case for models, such as LAFTR and CFAIR, that simply learn an encoding. 

**I don’t think there’s enough discussion of the dataset features...**

We will add a Discussion section to the paper. Please check the updated version.


**In Section 5, the authors say, “For each metric, the best value is reported..**

Given the chosen hyper-parameters, a model can be more accurate or more fair and there is a trade-off between these two metrics. Since a priori we did not know which hyper-parameter to choose and by setting any specific hyper-parameter, the model can be better-off on one metric and worse-off on the other, we wanted to report the best result a model can achieve on each metric and show that there exists a bias even when the best variation of the model is considered. Note that this is even in favor of the models as the best possible result is reported. Hence in each column we report the best obtained result, which can correspond to a different hyper-parametre or early-stopping point. We will clarify it in the text.

**The dataset feature g_2 is introduced in Section 4.1..**

g_2 is used in all experiments.

**What were the sizes of the train, validation, and test sets?**

50,000, 10,000, 10,000 are used for train, validation, and test sets. 

**How many seeds were used for each experiment?**

We used three seeds for each experiment and reported the average.

**why is g_1=2 biased?**
In this case for odd digits the pixel 1 is green and for even digits the pixel 0 is green, indicating the eligibility.




We thank reviewer for their feedback. Please see below answers to the raised issues:

**Sensitive attribute values are limited to binary values. In contrast, the real-world sensitive attributes are multi-label.**

While multi-valued sensitive attributes can be also an interesting evaluation setup, since the majority of the proposed methods did not focus on this setting, we decided to focus on the mostly-studied case, and instead study it more thoroughly. This is the reason for evaluating only binary values in our work. Please also note that in each setting many experiments need to be run as we trained about 2,000 models for our experiments. We believe multi-valued scenarios can be the focus of a paper concentrated on this setting. The multi-valued setting is not the focus of our work.

**What type of sensitive attribute does the variation of the number of stripes represent? I can see that the color may represent skin tone differences and that the small pixel changes may represent the size of the correlation/bias in the input image. I am not sure what stripes may represent.**

As these features are synthetic, they do not represent any real features. However, to give a related example, one can think of the wrinkles seen on a person’s face that is correlated with age. The stripes in our dataset can be correlated with eligibility in order to create scenarios where some attributes (which in our study were non-sensitive), are correlated with eligibility. Hence, one can evaluate the model’s performance under such correlations.

**Just as FYI, it was difficult to read the labels of the third graph of Figure 2 when printed on A4 paper. I had to revisit the PDF and zoom in to see that the graph title is "clr-ratio(0.01, 0.01)" instead of "clr-ratio(0.03, 0.03)". Would be good if those titles are repeated in the Figure 2 description to help with reading.**

Thank you for pointing it out, we have made the plots bigger and have moved the rest to the Supplementary section for easier reading.



############# Summary of contributions ##############

This experimental paper proposes a method for simulating a dataset to evaluate algorithms for enforcing fairness criteria on deep learning models. The proposed simulated dataset pertubs MNIST in various ways to create data distributions that could impact the difficulty of satisfying various fairness criteria. The paper then uses this simulated dataset to empirically evaluate three existing approaches for enforcing fairness on deep learning models using adversarial techniques. For each approach, they evaluate group-based fairness criteria such as demographic parity, equal opportunity, and equal odds under different dataset simulations. 

The most potentially impactful contribution of this paper is the simulated dataset and the comparison of different approaches under different dataset simulations. 

############# Strengths ##############

- Their proposed simulated modifications of MNIST are well documented. 

- The methodology of stress-testing different algorithms by tuning different aspects of the simulated data distribution seems intuitive and useful.

- The experiments are well structured, and they provide a mostly clear explanation for tuning each of the dataset simulation parameters. 

- While they only evaluate a few group-based fairness criteria in their experiments, they provide an implementation to evaluate many more metrics listed in Table 2 in the appendix.

- The experiments provided seem thorough, with many different dataset simulation parameters. 

############# Weaknesses ##############

- The dataset features seem a bit contrived, and it’s not entirely clear how the different simulated features correspond to real data scenarios. For example, as one modification of MNIST, they add 6-20 drawn horizontal lines on some digits, and 2-6 drawn lines on the rest of the digits. They modify how many of the 6-20 line digits are odd, and how many are even, such that the number of lines on the digit can correlate with whether the digit is even or odd. It’s not clear to me if there are real image datasets where something like this would occur. Of course, I recognize that it’s impossible to capture every possible real data distribution in a simulation like this, but it would be helpful to have some explanation of why they use horizontal lines specifically. A similar explanation would be helpful for the use of the green column.

- They only use a binary sensitive attribute, encoded in the simulation as various shades of red/blue. It would be nice to see an example with more than two categories for the sensitive attribute, as the difficulty of satisfying many of these group-based fairness measures can increase with more categories.  

- The related work has limited discussion of other papers that evaluate multiple fairness criteria and algorithms. The closest citation is Friedler et al. 2019, but they do not discuss or compare to the findings of Friedler et al 2019, which also evaluates four different algorithms for enforcing group-based fairness. Specifically, this paper should discuss how their conclusions and evaluations relate to the conclusions and evaluations of Friedler et al. 2019. (As a concrete example, the Stability analysis in Section 7.2 of Friedler et al. 2019 is very similar to this paper’s “Impact of seed” analysis in the Experimental Results section.)

- The paper would benefit from deeper discussion of how or why the three evaluated approaches differ in performance. Otherwise, the experimental results are not particularly novel (for example, the fact that a correlation between the sensitive attribute and the label can lead to unfair predictions is well known). 

############# Recommendation ##############

Overall, my recommendation is 5: Marginally below acceptance threshold. While this paper has a nicely organized experimental section and a clear methodology for simulating a dataset, I don’t think there’s enough discussion of the dataset features and enough discussion or novelty in the experimental results themselves. I do think this type of survey work can be impactful, and would encourage the authors to add more discussion of the methodology, results, and related work, as suggested above.

############# Questions and clarifications ##############

- The dataset feature g_2 is introduced in Section 4.1, but not mentioned in experiments. Is g_2 used in any experiments? 

- What were the sizes of the train, validation, and test sets? 

- How many seeds were used for each experiment? For example, in the “Impact of seed” experiment, how many seeds were used to compute the standard deviation? 

- In Section 5, the authors say, “For each metric, the best value is reported, meaning the performance of the best-performing model on the fairness metric (chosen across hyper-parameters on the validation set) is reported on the test set...This would allow us to report the best performance on each metric, without choosing how to compromise between accuracy and fairness metric.” I don’t understand what this means. Does this mean that they chose a different model for each metric reported? Did they choose one model for the equal opportunity metric, and a different model for the accuracy metric? 

- On page 7, the authors say “LAFTR models drop performance in the biased setup of g_1 = 2.” Can the authors clarify what they mean by “drop performance”? Which metric is hurt? And why is g_1=2 considered to be a “biased” setup? It seems the g_1=2 is simply a setting when it is easy to identify odd from even numbers -- I don’t understand why the authors frame this as “biased.”

############# Additional feedback ##############

- The related work section is currently not organized well. I would suggest breaking the large block of text on page 2 into multiple paragraphs addressing different aspects of related work. For example, have one paragraph discussing different methods for enforcing fairness on deep learning models, and have a different paragraph discussing other survey papers for evaluating various fairness criteria and algorithms.

- The plots in the Experimental Results section are really tiny and hard to read (Figures 2-4). 

- [page 6, “The goal is to evaluate when such cases arise, do models perform fairly in test setups, where everything is completely balanced?”] It’s great to outline the goal so clearly, but this isn’t actually a grammatically correct sentence. I would suggest modifying this (and similar other sentences in the paper) to: “The goal is to evaluate the following: when such cases arise, do models perform fairly in test setups, where everything is completely balanced?”


This paper proposes a synthetic dataset to use for benchmarking representation learning algorithms on a variety of fairness criteria. Using this benchmark, it compares several recently developed and prominent methods for fair representation learning and shows their relative strengths and weaknesses under different data generation assumptions and for different metrics.

The type of careful, systematic comparison provided by this paper is potentially an important contribution. With so many fairness metrics and methods to choose from, it is important for the community to understand limitations and under what sort of conditions a given method may be appropriate or perform best.

However, I recommend rejection for this paper for one main reason: I believe the synthetic dataset is inadequate for use as a realistic benchmark. It is too abstract and unlike examples where people are concerned about fairness, and the protected attribute--background color--is something which can be encoded far too simply. As a result, I believe that the relative performance of methods evaluated on this benchmark dataset may be not be indicative of their relative performance on a more realistic dataset with more complex sensitive attributes like race and gender.

It is not obvious how to improve this flaw, since more realistic datasets may also be less general and hence less useful for benchmarking purposes. One possibility is to abandon the goal of providing a benchmark, since fairness considerations may be too problem dependent for this anyway. In this way, the paper could propose a systematic comparison of methods for a particular synthetic task, provided that task is realistic enough to be related to some potential real world applications. I have no specific suggestion for how to modify the synthetic data generation process, but the features, especially the sensitive attribute(s), need to be rich enough to be more like a real fairness application.

Meta-review: The paper studies benchmarking of bias mitigation methods. The authors propose a synthetic dataset of images (alike colored-MNIST) that enables a controlled setup over different types of correlations between a binary sensitive attribute, dataset features, and a binary outcome label. The authors have evaluated 2K models that are the variants of three recently proposed debiasing methods using fair representation learning across various settings.

While the reviewers acknowledged the importance of benchmarking fair learning methods in a systematic controlled setting, they have raised several concerns: 
(1) the proposed benchmark is too abstract/unrealistic (R4, R2, R3); it is not clear whether the findings from this benchmark can be generalized to real-world data with real sensitive features, (2) the proposed benchmark is limited to pseudo sensitive attributes (R1) that are binary (R1, R2), (3) the paper lacks in-depth analysis on why certain methods work under certain conditions (R3). Among these, (2) did not have a major impact on the decision, but would be helpful to address in a subsequent revision, (3) was partially addressed in the rebuttal. However, (1) makes it very difficult to assess the benefits of the proposed benchmark, and was viewed by AC as a critical issue.

The authors provided a detailed rebuttal addressing multiple of the reviewers’ concerns. AC can confirm that all four reviewers have read the author responses and have contributed to the discussion. A general consensus among reviewers and AC suggests, in its current state the manuscript is not ready for a publication. See R1 post-rebuttal encouragement and suggestions how to strengthen the work. We hope the detailed reviews are useful for improving the paper.


Decision: withdrawn-rejected-submissions