Benchmarking Bias Mitigation Algorithms in Representation Learning through Fairness Metrics
Authors: Charan Reddy
Soroush Mehri
Deepak Sharma
Samira Shabanian
Sina Honari
Venue:  (2021)
Keywords: fairness model evaluation
fair deep learning
adversarial fairness

Abstract: With the recent expanding attention of machine learning researchers and practitioners to fairness, there is a void of a common framework to analyze and compare the capabilities of proposed models in deep representation learning. In this paper, we evaluate different fairness methods trained with deep neural networks on a common synthetic dataset to obtain a better insight into the working of these methods. In particular, we train about 2000 different models in various setups, including unbalanced and correlated data configurations, to verify the limits of the current models and better understand in which setups they are subject to failure. In doing so we present a dataset, a large subset of proposed fairness metrics in the literature, and rigorously evaluate recent promising debiasing algorithms in a common framework hoping the research community would take this benchmark as a common entry point for fair deep learning.

Under review as a conference paper at ICLR 2021
BENCHMARKING BIAS MITIGATION ALGORITHMS IN REPRESENTATION LEARNING THROUGH FAIRNESS METRICS
Anonymous authors Paper under double-blind review
ABSTRACT
With the recent expanding attention of machine learning researchers and practi- tioners to fairness, there a void of a common framework to analyze and compare the capabilities of proposed models in deep representation learning. In this paper, we evaluate different fairness methods trained with deep neural networks on a common synthetic dataset to obtain a better insight into the working of these methods. In particular, we train about 2000 different models in various setups, including unbalanced and correlated data configurations, to verify the limits of the current models and better understand in which setups they are subject to failure. In doing we present a dataset, large subset of proposed fairness metries in the literature, and rigorously evaluate recent promising debiasing algorithms in a common framework hoping the research community would take this benchmark a common entry point for fair deep learning.
INTRODUCTION
The emergence of deep learning models has brought up questions on the fairness of these models with respect to sensitive attributes. There have been studies the reliability and bias of deep learning approaches, such as the bias of the learning algorithms due to matching a data distribution in adversarial models (Cohen al., 2018), visual question answering (Agrawal al., 2018), image search setups (Kay et al., 2015). and gender classification (Buolamwini & Gebru, 2018). Recent approaches (Madras al., 2018; Zhao et al., 2020: Creager et al., 2019; Zhang et al., 2018) address the bias in deep learning models and propose fair models that better guarantee fairness criterion while maintaining accuracy. The general idea in these models is applying a fairness learning technique either on the learned latent representation (Madras et al., 2018; Zhao et al., 2020; Creager et al., 2019) or the output eligibility criterion (Zhang et al., 2018), given an input.
Adversarial learning techniques (Goodfellow et al., 2014; Ganin & Lempitsky, 2015) have been extensively used to learn the data distribution of interest, such as learning disentangled latent space (Kim & Mnih, 2018: Chen et al., 2018; 2016), subtraction of feature from the latent space (Lample et al., 2017; Denton et al., 2017). Adversarial learning has also been used in fairness models to either remove sensitive information from the latent space (Madras et al., 2018; Zhao et al., 2020; Zhang et al., 2018) or disentangle latent features into sensitive and non-sensitive attributes (Creager et al., 2019). The goal in these approaches is to remove the sensitive information, such as gender, from the latent space that is later used for other tasks, such as eligibility classification. In this paper, we aim to verify whether well-performing and ubiquitous models can effectively reduce bias when evaluated using well-established fairness metrics. Here we focus on deep learning models for classification and their adversarial bias mitigation counterparts.
The difficulty of task is determined by the distribution of the dataset, which in turn, is a function of the label imbalance, features imbalance, dependency of sensitive attributes to the rest of features, and distribution shift from training to development phase to name few. In addition, in the current state fairness community problems with reproducibility and inconsistency in the experimentation and dataset setups makes comparison difficult. Given the importance and possibly tangible impact of proposed algorithms when in production, we advocate for a rigorous and unified evaluation of the capabilities of debiasing models. We argue the need for systematic analysis that would assess different bias mitigation approaches under the perspective of different fairness metrics. This would help elucidate the most promising research contributions.

---

Under review as a conference paper at ICLR 2021
To analyse debiasing models we propose dataset that facilitates creation of different biased setups, such as data imbalance or correlation among eligibility and sensitive or non-sensitive attributes. In particular, contrary to real datasets where the intensity of each feature cannot be controlled, this dataset allows changing only one component while keeping all other components unchanged, something that allows study of different variations of biased setups.
To evaluate models under different biased setups, we provide an in-depth analysis of baselines and recently proposed bias mitigation models in the literature. In particular, we evaluate three promising debiasing models (six variants) together with a baseline model using a unified set of fairness metrics and report results by carrying extensive hyper-parameet
search in all cases, ensuring that the drawn conclusions can be attributed to modeling or loss choices. In doing so, we evaluate these models under different setups by training about 2000 models, in which we transition from balanced setups towards challenging unbalanced and correlated setups, where eligibility criterion is correlated with sensitive or non-sensitive attributes. Given the importance of fairness and the serious implications of their misuse, we intentionally try to push these models to their breaking point using extreme settings from our dataset. We consider different ratios for each sensitive group to create "unfair datasets". This is noteworthy that our analysis is not to undermine the effectiveness of any of these methods, rather setting the expectations and boundaries for different use cases and encouraging the community to follow. To summarize, our contributions are:
We show when there is correlation between eligibility and the sensitive attribute, models exploit for prediction of accuracy and use even in test cases when such correlation does not exist, which leads to unfair predictions.
We show that even when there are non-sensitive attributes in the dataset or small visual features in the input image that correlate with eligibility, models can still exploit it and hence be biased.
We demonstrate that the choice of seed can affect the results significantly and different models show considerable variation of performance given the seed.
We provide deep learning codebase composed of six debiasing models and a baseline to the fairness community for research and evaluation of fairness models.
We provide a dataset with different controllable sets of features and correlation among them, to facilitate research on fairness models in different unfair setups.
2 RELATED WORKS
There is an increasing literature studying how biased datasets can bias learning algorithms to discriminate (Bolukbasi et al., 2016; Cohen et al., 2018; Agrawal al., 2018). These studies have led researchers to propose new evaluation tools and datasets (Buolamwini & Gebru, 2018), with the goal of identifying potential machine learning error rate gaps among different groups. To remedy this issue, numerous contributions have emerged in order to probe machine learning systems at different levels and reduce discrimination from the perspective of modeling. Friedler et al. (2019) evaluate the fairness of pre-processing, in-processing, and proseprocessing machine learning approaches; however, no deep learning model is evaluated in their study. Moreover, the models are not evaluated while changing the level of dataset bias or correlation among features. Kamishima al. (2012) propose a regularization approach for models with probabilistic discriminative nature and evaluate it on logistic regression models; (Zafar al., 2017a) proposes flexible mechanism to optimize accuracy in the presence of non-conver faimess constraints (or vice-versa, optimize fairness with accuracy constraints) and evaluate it on logistic regressions and support vector machines.
In addition, several contributions have attempted to leverage adversarial learning to mitigate biases in learned representations or in classifiers. Adversarial learning was originally introduced within the framework of Generative Adversarial Networks (GANs) (Goodfellow et al., 2014)and it has been leveraged in the bias mitigation frameworks to make different groups indistinguishable from one another with respect to a sensitive attribute. In particular, (Edwards & Storkey, 2016) learns adversarially to debias encoded features to be used in classification tasks, while being fair with respect to demographic parity. Madras et al. (2018) adopt a group normalized & loss and adapt it for each fairness metric of interest. Beutel et al. (2017) debias the encoded features using an adversarial training procedure for classification tasks to achieve equality of opportunity. Zhang al. (2018)
2

---

Under review as a conference paper at ICLR 2021
directly pass the classifier's 's output, rather than the encoded features, to discriminator to debias the classifier with respect to different fairness metrics.
Despite the recent progress in fair deep learning models, the comparison of bias mitigation algorithms in a common setup is not straightforward. especially in biased and correlated dataset setups. Verma Rubin (2018) evaluate how fair off off-the-shelf logistic regression model is, given set of fairness metric definitions. In this work, we evaluate some of the most recent and promising deep learning models in common deep representation setup.
Adversarial techniques can be categorized into three main approaches: (i) applying debiasing through adversarial training directly the class label, where the class label is indicator of eligibility (Zhang et al.. 2018), (ii) mitigating bias through enforcing group-fairness on the learned latent space where is directly used for classification of cligibility (Beutel et al., 2017: Madras et al., 2018; Zhao et al., 2020), and (iii) discarding sensitive representation features for downstream tasks after disentangling learned latent space into sensitive and non-sensitive ones (Creager et al., 2019; Träuble et al., 2020). This line of research is motivated by recent development in disentangled representation learning (Kim & Mnih, 2018; Chen et al., 2018; Locatello et al., 2019). In this paper, we take some of the promising models from the above-mentioned categories and assess their merits in challenging scenarios, in particular, in under--epresented and correlated dataset setups. This exercise is not to undermine the applicability or achievements of these models but rather to understand the robustness and boundaries of these models under a variety of settings.
MODELS
In this study, we empirically analyze the performance of a baseline MLP (multi-layer perceptron) model without any bias mitigation learning criteria and compare it with LAFTR (Madras al., 2018), which applies adversarial learning to achieve group fairness, CFAIR (Zhao et al., 2020) which performs conditional alignment of representations to achieve a better accuracy-fairnes trade-off, and finally, FFVAE (Creager et al., 2019), which disentangles latent representation into sensitive and non-sensitive attributes.
For simplicity, we introduce the following shorthand notation. We denote by X' C Rd and y {0, 1} the set of inputs and outputs, respectively. Two sets of random variables and take associated values x E A' and y E Y. Variable determines eligibility which in our MNIST-based dataset refers to a digit even or odd. We denote the sensitive binary variable by S taking values 8 E {0, 1). Moreover, p indicates the output probability of the classifier as a real number and Y is the class predicted random variable based on p, which takes values á E {0, using a threshold. For each and y, the term DE is the conditional distribution of the joint distribution D over X x Y x S, given Y = y. =.
Baseline Model. We use an MLP as our baseline model, which given an input image x, predicts the probability p of the input being odd or even. This probability is then transformed into classification prediction ÿ. This model does not leverage any bias mitigation algorithm and is meant to show how fair base deep learning model would perform under different fairness criteria.
Learning Adversaiialyy Fair and Transferable Representations (LAFTR). LAFTR (Madras et al., 2018) an adversarial based bias mitigation algorithm within the scope of representation learning. In the supervised version of LAFTR, given an input image it first learns latent encoded representation e that is passed to the discriminator to be debiased. The learned representation then passed to a classifier to predict the task of interest y. The discriminator is trained by minimizing
SL.. (D((,, v)
where y is only passed in debiasing models aimed for equality of odds and equality of opportunity. The encoder and classifier are trained jointly by minimizing
Ea.y.eeDy(((())) - (otpee/ooooit))))oo(((sst.....
where E(x) is the encoded feature, which is passed to both the classifier C and the discriminator D. The first term measures the classification loss and the second term gets the adversarial gradients from the discriminator regarding the sensitive attribute s.
3

---

Under review as a conference paper at ICLR 2021
Following the original paper, considered four variants of this model. LAFTR-DP objective is defined as
E sE C (D(2). =1-
Σ SE (0,1)
Be,,ED,,
(3)
and LAFTR-EqOppl objective is considered as
Le F4Omr (D(z, ().
Σ v0.1),===
E BE
(4)
Fa00pl is obtained by replacing = in Eq. (4) and cg c- is the sum of C5 C5g0rr0 and C Lg
Conditional Learning of Fair Representations (CFAIR). Proposed by (Zhao et al., 2020) this model leverages two adversarial networks ho and h1, predicting sensitive attribute s respectively for class labels Y = 0 and Y 1. CFAIR depends on objective function called the balanced error rate (BER) (Feldman et al., 2015; Menon & Williamson, 2018), which guarantees small joint error across demographic groups. The BER represents the sum of false positive rate and false negative rate. Therefore, is equal to minimizing the below two conditional errors. BER (Y]J)) is defined
[|Y)
(Y x p(Ý = 1|Y = 0) + p(Ý = 0|Y =).
(5)
and BER is defined similarly, where $ is the predicted sensitive random variable. CFAIR is optimized based on the following min-max formulation.
min max C,E bo.h1 (ER(((((()))II)) >BERD (ho (B(X))ISS) + BER (h) (E())))(S)
(6) This approach proposes that using the balanced error rate along with the conditional alignment helps achieving equalized odds across the groups without impacting demographic parity.
Flexibly Fair Representation Learning by Disentanglement (FFVAE). Inspired by FactorVAE (Kim & Mnih, 2018), FFVAE (Creager et al., 2019) performs disentanglement by factorizing latent space. It learns a disentangled representation of the inputs, which is flexibly fair in the sense that it can be casily modified at test time to achieve demographic parity across various groups.
In setup, is the input image, which is considered to implicitly contain both sensitive and non-sensitive attributes. Moreover, a = (aI...., aN), = (b),..., 6x), and & are the sensitive attribute, the sensitive latent, and non-sensitive latent of 2. respectively, with N indicating the number of sensitive or non-sensitive features, depending on the dataset.
FFVAE trains encoder q(2, blx). decoder p(z|2, b), well adversarial network. The latent representation disentangled into sensitive and non sensitive latent attributes by encouraging both MT(b, =) and MI (be, a,)), Vi # to be low, where MT represents mutual information. FFVAE objective is defined as
CervAE(p. q) (l 22,b) + logpp(a b)]] DKL-FIVAE
(7)
where
DKL-FFVAE = =DDx((((2.)laf II (b,)))
-D x))lp(z,b))
(8)
The first term of Eq. (7) consists of reconstruction term (on left) and predictiveness term p(a b). which aligns sensitive attributes to its respective sensitive latents. (8) has two terms; the disentanglement term on the left decorrelates the sensitive latent representation b from using an adversarial network and the second term is the Kullback-Leibhe divergence between the prior and the latent distribution.
In addition to the above-mentioned models, we also did experiments with (Zhang et al., 2018) (based on the code released by authors), however, the model was very unstable on our dataset configurations after extensive adversarial training directly to the class labels, which makes the model unstable, as indicated by the authors. Due unstable results, dropped this model from evaluation.
search. We hypothesize that this is due to the application of
4

---

Under review as a conference paper at ICLR 2021
We show the conversion process used to generate our dataset and add our dataset attributes in the last columns to the resized image. e-o-ratio affects the number of lines observed on the image. g1/g2 and 9n respectively affect the location and size of the green pixels (small vertical lines on top-left of the image).
Figure 1:
3
3
3
6
6
6
orignnllmmae
readzing
dight pattern 32 x x
after 91. ge. gw 32 02
4
SETUP
Machine learning algorithms have attracted much attention and found application in wide range of highly risky domains such as medical tests and criminal justice. It is important to make sure that the unfairness present in datasets (Cohen al., 201 8; Agrawal al., 2018; Kay al., 2015; Buolamwini & Gebru, 2018), is not learned and later inferred by powerful deep learning models used for decision making (Barocas & Selbst, 2016; Berk et al., 2018; of the President, 2016). One way achieve this goal is to measure the ability of the current prominent debiasing models in ignoring the correlation existing in the datasets and evaluate these models setups where we have a mechanism to control the correlations among dataset features.
Colored-MNIST In order evaluate the debiasing models in challenging setups and be capable of controlling different dataset configurations, we design variant of the MNIST dataset (LeCun & Cortes, 2010), called Colored-MNIS! where we introduce different types of correlations between sensitive attribute, dataset features, and eligibility criterion. For an input image the label {0, 1} indicates eligibility or ineligibility, given
even or odd. We use the background colors as the protected or sensitive attribute E {0, 1}. where blue indicates the unprivileged group and red indicates the privileged group. The task of interest is to classify whether, given an input image r, the class label y is odd even. While model improves its classification accuracy, at observing its performance given different fairness metries. We would like to also observe how the model's classification accuracy and fairness measures change in different dataset scenarios. A common practice is to show reasonable performance on a few standard datasets. These, however, do cover the majority of possible distributions of dataset that model can encounter in real-world development and production. Here we are using this dataset as a benchmarking tool that will allow control over these factors give better sense of what model is capable of in different challenging setups.
aim
4.1 DATASET FEATURES
Label (y): The label indicates whether the digit even or odd. In our dataset, even is eligible (y and odd is ineligible (y 0).
Background color (bck): refers to the background color (sensitive attribute) which various shades of blue and red colors.
clr-ratio: shown in pair (be, b.). (bb)) refers the ratio of the even (odd) digits with blue backgrounds and the rest of the digits will have red backgrounds. Using this feature we control the correlation between the sensitive attribute and the eligibility.
This feature indicates the number of drawn lines on the digit. e-ratio (o-ratio) refers to the ratio of even (odd) digits that contain to 20 lines. The rest the digits have between lines. Using e-o-ratio, we can completely correlate (1, 0) completely decorrelate (0.5, 0.5) the number of lines on the digit (as a non- sensitive attribute) with the eligibility.
e-o-ratio:
Green width (gw): This value indicates the number of green pixels (cither or 5) in the image and is visualized as small vertical line (see Figures and 5). The location of green pixels is indicated by Ø1 and Ø2-
91: This variable indicates the location of the first green column in the image and is set to one of {2, (2). For each given (odd) digit, the location of the first green column is randomly drawn in the range [o, g1 /2) ((9) /2, 9: )). Hence, gi = 2, the location of the green column is an easy-to-spot indicator of eligibility since the same pixel location (0 1) indicates eligiblityiielliiibiitt while if
5

---

Under review as a conference paper at ICLR 2021
01 22, the model needs to observe over many cases, hence it requires stronger analysis across different cases.
92: This variable indicates the location of the second green column in the image and is set to
91 + digit where digit varies from to 9. We used 92 in all our experiments along with 91-
By using 91,
92, and Iv. evaluate the following: when we have features that visually small (1
or pixels big), can models exploit the correlation between 91 and eligibility, in an easy-to-spot case (g) 2) and more challenging case (g) 22)?
4.2 FAIRNESS METRICS
We use the same set of fairness metrics, as reported in LAFTR, CFAIR, and FFVAE, to evaluate the debiasing models. Table presents the fairness metries as well as their mathematical notations and abbreviations. In Table 2 we provide a comprehensive list of fairness metrics in the literature, which we have implemented in our framework and will release to the community. Due to the limit of space and more widespread usage of the metrics presented in Table 1, we only report on these metrics. The provided tool by (Friedler et al., 2019) is used to compute all of the metrics.
Fairmess Criferia
Formulation
Abbreviatin
Demegraphic Parny
Protected)
Unproteciol))
Equility of Opportaityy waty
1- 10 - 11 1,8
Ungrotected) MT 1/ 1,5 Protected]
EqUip1
Equality Oggoranity wayy
100
0.8
Ungrowectdd xF 1/ Protected)
Equallity
(500pp0 Eg0pe
EqDill
of v/Y = y. Ungeneeced)
up-scc
poteeted--ccuuccy
p(Y u/Y 9.5 Protectal)
polce
securacy
10p pacd
Table I: X, Y, S denote the input sample, ground truth label, and the sensitive attribute. and p are the model's prediction and the output probability of the model. For all metrics, 1 indicates the perfect and 0 the worst value.
5 EXPERIMENTAL RESULTS
In this section, we evaluate the baseline MLP model as well as the debiasing models (LAFTR-DP, LAFTR-EEQppO, LAFTR-EqOppl. Lddd CFAIR, and FFVAE) the CoNOTed-MNIST dataset and evaluate their performance on different fairness metrics. The goal is to provide an evaluation bascline for different debiasing models and meanwhile measure how fair they remain in unbalanced and biased situations. In particular, we change the dataset from a balanced setup to different unbalanced setups. In the balanced setup, the ratio of unprivileged (blue) and privileged (red) background is at 50% for both eligible and ineligible groups, giving clr-ratio of (0.5, 0.5), and the data features and eligibility are uncorrelated with e-o-ratio being (0.5, 0.5). In the unbalanced cases, while we deviate from the balanced setup during training, at test time, we evaluate the models balanced setup, meaning the dataset is at 50% for eligible and ineligible groups as well as for privileged and unprivileged groups This way, we report the model's 's performance of different sub-groups. Experimental and model details are provided in Section B.
balanced setup
Impact of reducing representation of unprivileged group. In this experiment, we change clr- ratio from the balanced setup of (0.5, 0.5) to unbalanced cases of (0.1, (0.01, 0.01), and (0.001, 0.001). In these setups, the unprivileged group with the blue background is less and less represented in both eligible and ineligible groups, hence evaluating the models where the unprivileged groups are under-represented, but represented equally in both eligible and ineligible groups. Figure 2 compares two models side-by-side, Figure 6 and Tables 4 to 10 in the Supplementary in Section C.1 show all models and give detailed results. We initially averaged metrics over three seeds, then for each metric, the best value is reported, meaning the performance of the best rest-performing model on the fairness metric (chosen across typer-parameters on the validation set) reported on the test set. This would allow us to report the best performance on each metric, without choosing how to compromise between accuracy and a fairness metric (which usually have a trade-off and improving one deteriorates the other). We want to show that there exists a bias even when the best variation of the model is considered. Note that this is even in favor of the models as the best possible result is reported. Hence in each column report the best obtained result, which different
correspond to or carly-stopping point. As observed in the results, through the transition from balanced to unbalanced setups, both accuracy and fairness metrics drop. While LAFTR models

---

Under review as a conference paper at ICLR 2021
maintain better fairness metries compared to other models, they perform poorly on accuracy metries. After LAFTR, FFVAE and then CFAIR, can less and less preserve fairness metrics while in the same order they improve on accuracy. The baseline model (MLP), without any debiasing, expected performs the best on the accuracy, while fails more than other models in maintaining fairness.
LAPTROD
LAFTH-SODes2
CPAIR
LAPTR.D
LAFTI-S000s2
OPAR
metrr
A0C
120
Eq00po0
Eqpipst
adc
p-00c
18-00C
UPP
E000p00
Eqpipst
ci-rati (5.5, 0.5)
CPretti 10 000,
0.000
Figure 2: Comparing different models while decreasing minority representation.
Impact of correlation of sensitive attribute with eligibility. In this setup, we change clr-ratio from the balanced setup of (0.5, 0.5) to unbalanced cases of (0.1, 0.9) and (0.01, 0.99), where the unprivileged group is under-represented in the eligible and over-represented in the ineligible groups. In unbalanced settings, the resistance of models the correlation between sensitive attribute (background color) and the eligibility metric is evaluated. The goal is to evaluate the following: when such cases arise, do models perform fairly in test setups. where everything completely balanced? and hence can they avoid carrying the correlation of training to test setups? Figure 3 compares two models side-by-side, Figure and Tables 11 to 17 in Section C.2 of Supplementary show all models and detailed results. In this setup, FFVAE preserves better the fairness metrics compared to other models. LAFTR and CFAIR models perform similarly or sometimes even worse than the baseline model on fairness metrics, while the baseline model preserves better the accuracy. Missing bars in the plots imply zero metric values and biased predictions on the balanced test set. The results show that even though debiasing models provided with the sensitive attribute during training, they still fail to decorrelate the sensitive attribute from eligibility in highly correlated setups.
LASTA Spfaso LAFTH Seboa)
Enpood
SPAAE
LAFTA Gotuso LAFTR Goboa)
Exfoodd
FPMA
LAFTR DA
CPAR
LAFTR D
CPAR
302
0.000
10:
1000
CP
Enp0p0
Enpoos
Enjiss
800
2.000
UP 100
CP
En00no
Empoool
Enpois
cisrabio
0.5)
diriatie 10.01,
Figure 3: Comparing different models while shifting correlation of sensitive attribute (bek) and the eligibility.
Impact of correlation of non-sensitive attribute with eligibility. In this setup, we keep cir-ratio at (0.5, 0.5). however, change e-o-ratio from balanced setting of (0.5, 0.5) unbalanced settings of (0.9. 0.1). This would make the number of lines on the digits correlated with eligibility. The goal is to evaluate the following: to what extent model can exploit the correlation between a non-sensitive attribute and the eligibility criterion and hence be biased at test time? Figure 8 and Tables 18 to 24 in Section C.3 show results for all models. While MIP, CFAIR, and FFVAE do not show much change in this setup, LAFTR models show more deviation, indicating they are more sensitive to such correlations. For example, LAFTR-DP dropped in fairness measures, while LAFTR AAFTP-'' performance improved. As show later, this is mainly due to the high variance of this model under different seeds.
Impact of small features in the input images. In this setup, we change 91 from 22, where the location of the green pixel has a more complex relation with eligibility, to 2, where the location of the green pixel (0 for eligibility and I for ineligibility) is easy-to-spot indicator of eligibility that can
7

---

Under review as a conference paper at ICLR 2021
be more easily learned by models and hence induce bias. The question is whether models find such small but non-trivial correlated features with eligibility? ? If the answer is yes, then information even in the small portion of the imput cause biased performance of the model. In this setup we run an experiment where for each one of gi = 2 and 91 = 22 we consider two cases with Iu = 1 and Ow respectively showing only and 5 green pixels in the image. This setup would verify whether the model observes the green part of the image and if yes how much impact the size of that region will have. Figure 4 shows two models, Figure 9 and Tables 25 31 show two extra models and give detailed results for all models. While models like MLP, CFAIR, and FFVAE show small difference two setups, LAFTR models become less fair in the biased setup of 91 2. indicating this model more prone to pick up such small details, when it can spot it. Results with fo 5 show very close performance to J = 1, indicating even one pixel can be enough for models to extract small details from the images. The results show that when there are small and sometimes not-noticeable features in the dataset, they can cause bias in the learning algorithm.
PLP LAITR-OP
LATTROGRESO LARTh-Spdes3
LAFTB-ENOO
ITWRC
LAFTR-CODeS LARTh-Sy0es3
LAFTB-ENOO
ITWC
metei
metnr
60C
5-40C
102-000
txpippt
80C
P-MCC
100-000c
S:00ppt0
txplppt
01:-2.
Figure 4: Impact small visual components on different models' performance.
6 DISCUSSION
6.1 SOURCES OF BIAS
In our experiments, we have considered four different sources of bias.
1. Small percentage of the unprivileged group. This case causes bias due to the small ratio of the unprivileged group compared the privileged group. In this case, the bias can be two-fold, one is due to the imbalance between the two groups, another can be due to rareness of the data from the unprivileged group. While the clr-ratios (0.1, 0.1), (0.01, 0.01), and (0.001, 0.001), all are subject to group imbalance, the latter two cases have 250 and 25 samples from the unprivileged group. compared to 24,750, and 24.975 samples respectively from the privileged group. So, the model has to also handle the secondary aspect of the bias, which different.
2. Correlation of feature with eligibility. In some data the unprivileged group might be less eligible, e.g. due to historic reasons of lack of access to facilities or limited resources compared to privileged group. Such correlations the data can be picked up by the model and cause bias at inference.
3. Impact of size of the correlated visual features. While large features that correlated with eligibility and cause bias can be easily observed by model, small visual features can also have correlations with eligibility. Given the size of the correlated features, some models can act more fair while others might remain biased.
4. Impact of unnoticed and small features. The debiasing algorithms work based on the assumption that the sensitive or biased attributes are known a-priori and hence can be addressed by knowing these features and removing them from the learned representation. However, some features, especially small ones, might not be noticed by the model or the annotator. For example, wearing earrings, can correlate with gender, or glasses with age. Such cases might not be observed and addressed properly by the debiasing model. if they are not know a-priori.
We have studied the impact of the four mentoooed sources of bias, and have observed a stronger level of bias in the first two cases. Specially, the correlation of big visual sensitive attributes with
8

---

Under review as a conference paper at ICLR 2021
eligibility, and imbalance among the privileged and unprivileged groups lead to higher bias. This trend was observed in all models. We observed smaller visual components that are correlated with eligibility cause less bias, the model can more easily get rid of them. Some models such as LAFTR suffered still in these cases. We believe study of such sources of bias can help better evaluate the dehiasing models and advocate their study in future works.
6.2 MODEL STABILITY
Variation due to seed. In deep learning models, seed is a great source of variation due to initialization of the model parameters and the sampling done during training, which can converge the model to different solutions in the course of training. To observe its impact, we evaluated how much standard deviation changes when the seed used for a model differs. The results for three seeds considered are shown in Figure 10 for all of the setups described in our experiments. We observed the choice of seed can impact some models more than others. In particular, LAFTR was the least stable model in terms of variation of results given different seeds. We advocate verification of any debiasing method under different seeds, as this indicates how much the model is susceptible to small training variations and hence not suitable for bias mitigation. This is complementary other studies of stability, such as cross-validation, studied in Friedler et al. (2019).
Correlation between dataset features and model's prediction. If model's prediction is correlated with dataset features, it indicates it can rely on biases in data in its predictions and hence be unfair when applied. To verify this, for each model, we measured the correlation between dataset features and fairness metrics using Spearman correlation matrices, measured over all of the experiments presented in this section. The results are presented in Figure 11 for the 64-dim architecture. On average the models that capture less correlation between dataset features and fairness metrics act fairer the experiments, as we also observe from results presented above.
Instability due to the debiasing method. Debiasing algorithm can cause instability due to the way the model mitigates bias. In particular, adversarial models can be unstable in practice and caution needs to be made in properly using them. We found Mubal (Zhang et al., 2018) to be the least stable model, as it applies the adversarial loss directly onto the class label. Applying the adversarial loss to the learned latent space, done in LAFTR, made this process more stable. However, even this approach performed poorly in many setups and was subject to high variations with a small change in the dataset. Even among different LAFTR variants, the LAFTR model with a given fairness criteria introduced into its objective function, was not always the LAFTR variant performing the best on that particular metric. We believe this is due application of both accuracy and adversarial loss the same latent space, which causes a competition between the two losses. CFAIR has more stable learning due using two adversarial networks for two eligibility classes and using balanced error rate as its loss function. Finally, FFVAE, obtained the most stable performance due to disentangling of attributes into sensitive and non-sensitive and also decoding features, which helps the model capture all of the information of the input image.
7 CONCLUSION
With representation learning algorithms getting more accepted automatic decision- makinn tools, we propose an evaluation scenario to better understand the limits of these methods. We systematically compare and analyze a few of the better-known models using our proposed dataset, where we control the correlation in-between different dataset components. Although these models proved their effectiveness to the community and can solve and improve upon some of our dataset variants, we show that we can purposefully push them to their breaking point. Our results indicate that in different setups the models can learn and hence induce a bias when correlations in the dataset exist. This suggests that not all models are suitable for all scenarios and using unsuited models for certain tasks could cause unwanted consequences. We also note that with the abundance of fairness metrics, there exists room for improvement when it comes to model selection we cannot pinpoint single metric that works best across all the metrics. This becomes imperative as specifically, practitioners need to be fair across more than one metric and find trade-off between faimess and accuracy. Finally, we will release our dataset and codebase to the community in order to provide a framework for the evaluation of deep fairness models.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Aishwarya Agrawal, Dhruv Batra, Devi Parikh, and Aniruddha Kembhavi. Don't just assume; look and answer: Overcoming priors for visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4971-4980, 2018.
Solon Barocas and Andrew D Selbst. Big data's disparate impact. Calif. L. Rev.. Califorria Law Review, 104(IR)661,, 2016. URL http: /ttp://lawwat... berkeley..e................. httpelllyy..drrrordd//eeooooeeee
Shai Ben-David, Nadav Eiron, and Philip M. Long. On the difficulty of approximately maximizing agreements. J. Comput. Syst. Sci., 66:496-514, 2003.
R. Berk, H. Heidari, S. Jabbari, M. Kearns, and A. Roth. Fairness in criminal justice risk assessments: The state of the art. Sociological Methods & Research, pp. 00491241 1878253, 2018.
Alex Beutel, Jilin Chen, Zhe Zhao, and Ed H Chi. Data decisions and theoretical implications when adversarially learning fair representations. arXiv preprint arXiv: arXiv1107.00077 2017.
Tolga Bolukbasi, Kai-Wei Chang, James Zou, Venkatesh Saligrama, and Adam Kalai. Man is to computer programmer woman is to homemaker? debiasing word embeddings. In Proceedings of the 30th International Conference on Neural Information Processing Systems, NIPS' 16, pp. 4356 4364, 2016.
Joy Buolamwini and Timmit Gebru. Gender shades: Intersectional accuracy disparities commercial gender classification. In Conference on fairness, accountability and transparency, pp. 77-91, 2018.
Toon Calders and Sicco Verwer. Three naive bayes approaches for discrimination-free classification.
Data Mining and Knowledge Discovery, 21(2):277-222 2010.
Ricky TQ Chen, Xuechen Li, Roger Grosse, and David K Duvenaud. Isolating sources of disentanglement in variational autoencoders. In Advances in Neural Information Processing Systems, pp. 2610-2620, 2018
Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. In Advances in neural information processing systems, pp. 2172-2180, 2016.
Alexandra Chouldechova. Fair prediction with disparate impact: A study of bias in recidivism
prediction instruments. Big data, 5(2): 2017.
Joseph Paul Cohen, Margaux Luck, and Sina Honari. Distribution matching losses can hallucinate features in medical image translation. In International conference on medical image computing and computer-assister intervention, pp. 529-536. Springer, 2018.
S. Corbett-Davies, F. Pierson, A. Feller, S. Goel, and Aziz Huq. Algorithmic decision making and the cost of fairness. Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2017.
Elliot Creager, David Madras, Jörn-Henrik Jacobsen, Marissa A Weis, Kevin Swersky, Toniann Pitassi, and Richard Zemel. Flexibly fair representation learning by disentanglement. volume 97 of Proceedings of Machine Learning Research, pp. 1436-1445. PMLR, 2019.
Emily L Denton al. Unsupervised learning of disentangled representations from video. In Advances in neural information processing systems, pp. 4414-4423, 2017.
Cynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Richard Zemel. Fairness through awareness. In Proceedings of the 3rd innovations in theoretical computer science conference, pp. 214-226. ACM, 2012.
Harrison Edwards and Amos Storkey. Censoring representations with an adversary. In International Conference Learning Representations, 2016.
Michael Feldman, Sorelle A Friedler, John Moeller, Carlos Scheidegger, and Suresh Venkatasubra- manian. Certifying and removing disparate impact. In Proceedings of the 21th ACM SIGKDD International Conference Knowledge Discovery and Data Mining, pp. 259-268 ACM, 2015.
10

---

Under review as a conference paper at ICLR 2021
V. Feldman, V. Guruswami, P. Raghavendra, and Yi Wu. Agnostic learning of monomials by halfspaces is hard. 2009 50th Annual IEEE Symposium on Foundations of Computer Science, pp. 385-394, 2009.
Sorelle A Friedler, Carlos Scheidegger, Suresh Venkatasubrammniian, Sonam Choudhary, Evan Hamilton, and Derek Roth. A comparative study of fairness- seaness-enhancing interventions in machine learning. In Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 329-338. ACM, 2019.
Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by backpropagation. In International conference on machine learning, pp. 1180-1189. PMLR, 2015.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In AISTATS, 2010.
lan Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural information processing systems, pp. 2672-2680, 2014.
Moritz Hardt, Eric Price, Nati Srebro, et al. Equality of opportunity in supervised learning. In Advances in neural information processing systems, pp. 3115-3325 2016.
Toshihiro Kamishima, Shotaro Akaho, Hideki Asoh, and Jun Sakuma. Fairness-aware classifier with prejudice remover regularizer, In Joint European Conference on Machine Learning and Knowledge Discovery Databases, pp. 35-50. Springer, 2012.
Matthew Kay, Cynthia Matuszek, and Sean A Munson. Unequal representation and gender stereotypes in image search results for occupations. In Proceedings of the 33rd Annual ACM Conference Human Factors in Computing Systems, pp. 3819-3828, 2015.
Hyunjik Kim and A. Mnih. Disentangling by factorising. In ICML, 2018.
Matt J Kusner, Joshua Loftus, Chris Russell, and Ricardo Silva. Counterfactual fairness. In Advances in Neural Information Processing Systems, pp. 4066-4076, 2017.
Guillaume Lample, Neil Zeghidour, Nicolas Usunier, Antoine Bordes, Ludovic DENOYER, et al. Fader networks: Manipulating images by sliding attributes. In Advances in Neural Information
Processing Systems, pp. 5963-5972, 2017.
Yann LeCun and Corinna Cortes. MNIST handwritten digit database. 2010. URL http::/yaann lecun. www.eedo/eeeeeeeeeeee mond//mnist.....
Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Raetsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learning of disentangled representations. volume 97, pp. 4424444. PMLR, 2019.
David Madras, Elliot Creager, Toniann Pitassi, and Richard S. Zemel. Learning adversarially fair and transferable representations. In Proceedings of the 35th International Conference on Machine Learning. ICML, pp. 338 1-3390, 2018.
A. Menon and R. Williamson. The cost of fairness in binary classification. In FAT, 2018.
Executive Office of the President. Big data: A report on algorithmic systems, opportunity, and civil rights. 2016.
Shiori Sagawa, Aditi Raghunathan, Pang Wei Koh, and Percy Liang. An investigation of why overparameterrizatio exacerbates spurious correlations, 2020.
Frederik Träuble, Elliot Creager, Niki Kilbertus, Anirudh Goyal, Francesco Locatello, Bernhard Schölkopf, and Stefan Bauer. Is independence all you need? on the generalization of representations learned from correlated data, 2020.
Sahil Verma and Julia Rubin. Fairness definitions explained. In Proceedings of the International Workshop Software Fairness, FairWare '18, pp. 1-7, 2018. ISBN 978- 978-1-4503-5744-3
11

---

Under review as conference paper at ICLR 2021
M. Zafar, I. Valera, M. Gomez- RodrzzRod and K. Gummadi. Fairness constraints: Mechanisms for fair classification. In AISTATS, 2017a.
Muhammad Bital Zafar, Isabel Valera, Manuel Gomez Rodriguez, and Krishna P. Gummadi, Fairness beyond disparate treatment & disparate impact: Learning classification without disparate mistreatment. In Proceedings of International Conference on World Wide Web, pp. 1171-1180, 2017b.
Brian Hu Zhang, Blake Lemoine, and Margaret Mitchell. Mitigating unwanted biases with adversarial learning. In Proceedings of the 2018 AAAT/ACM Conference on AI, Ethics, and Society, pp. 335-340. ACM, 2018.
Han Zhao, Amanda Coston, Tameem Adel, and Geoffrey J. Gordon. Conditional learning of fair representations. In International Conference Learning Representations, 2020.
Indre Zliobaite. On the relation between accuracy and fairness in binary classification. In The 2nd workshop on Fairness, Accountability, and Transparency in Machine Learning (FATML) a ICML' 15, 2015. doi:
12


Reviews:
[{"review_id": "uRtjhmOmFu", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "#Summary:\n\nThis paper presents an interesting benchmark study on three bias-mitigation algorithms: \n- LAFTR (Madras et al., 2018), an adversarial method for learning fair and transferable representations;\n- CFAIR (Zhao et al., 2020), conditional learning of fair representations;\n- FFVAE (Creager et al., 2019), fair representation learning by disentanglement.\n\nThe experiments are conducted on a synthetic dataset (colored-MNIST), with various interventions to control the bias (background color, number of lines drawn on the digit, green pixels added with different widths and locations). Empirical comparison over the three bias-mitigation methods are reported.\n\n#Pros:\n- Due to the increased popularity of bias-mitigation methods recently, there is a need to benchmark them carefully and fairly. This paper presents a preliminary effort towards better comparing those algorithms under a unified framework.\n- The proposed experimental framework is very interesting, by using a colored-MNIST dataset with various controllable interventions: 1) background color; 2) number of lines drawn on the digit; 3) green pixels added with different width and locations. Those interventions represent clues with various difficulties for the model to pick on, thus would be useful in understanding how sensitive bias-mitigation algorithms are under different situations.\n\n#Cons:\n- The most important thing lacking for this paper is a *deeper* understanding of the results based on the empirical observations. This paper seems to merely present the empirical results, i.e., under different settings which method works better, without further insights on why certain methods work under certain conditions. \n(1) For example, for LAFTR, since the fairness metric (DP, EqOpp0/1, EqOdd) is already built into the objective function, does this affect how this method performs when evaluated over different fairness metrics? \n(2) When changing the e-o-ratio, is there a reason for the LAFTR-DP metric to drop and LAFTR-EqOpp0 metric to improve? \nCurrently the paper reads like different methods achieve different trade-offs under various fairness metrics, and it is hard to learn any practical lessons without further analysis over those empirical results.\n- This might be a stretch but as a benchmark paper, I would expect to see comparisons of more methods and more datasets. Method-wise, I don't see a clear argument on how the three methods presented in this paper are chosen, is it one method each from the three representation-learning categories? Some clarification would be useful. \nDataset-wise, it would also be interesting to see the comparison over datasets other than vision (e.g., NLP datasets, or some of the classic fairness datasets with numerical/categorical features like UCI adult), to gain a better understanding of how those methods perform under different types of feature input. \n\n\n#Overall Recommendation\n\nOverall I tend to reject this paper, as this paper can benefit a lot from a deeper analysis on *why* different methods work under different conditions, and more studies on different datasets covering wider applications (e.g., NLP). It would be more useful if the paper can give more practical recommendations on which bias-mitigation method to choose under a certain scenario. \n\n\n#Minor comments\n\n- The figures' readability can be improved. Some of the figures are really small to read, some of the error bars are truncated, and in some cases the bar plots for certain methods are missing (if it's because the metric achieves zero value please clarify).\n- For experimental comparison, some of the error bars on the bar plot are really large. I'm not sure what's the particular reason causing this, could the authors do a deeper analysis on what are the factors? Is it purely because of the instability of a method, or because of the size of the training data, or the number of seeds, or it's due to the intervention added? Currently it's very hard to tell which method works reliably under which scenario due to the significance of the results.\n- It might be worth to move some of the details from appendix to the main text, currently the basic details of the experimental setting is unclear to me based only on the main text.\n\n#Post rebuttal\n\nI appreciate the authors added a deeper discussion and analysis in their newest uploaded draft, which helps clarify some of my concerns. However, it would be better if the following can be better addressed:\n\n- As a benchmark paper the experimental setting is relatively simple/artificial, more realistic datasets/settings would be more useful;\n- Thanks for adding the stability study. It would be more helpful if the main results can be presented with a better control over the stability. Currently it is still hard to tell the statistical significance of the results.\n- As the other reviewers have mentioned, it is still unclear how this simulated setting connects to real-world fairness applications.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "lc260k0mQ1", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "############# Summary of contributions ##############\n\nThis experimental paper proposes a method for simulating a dataset to evaluate algorithms for enforcing fairness criteria on deep learning models. The proposed simulated dataset pertubs MNIST in various ways to create data distributions that could impact the difficulty of satisfying various fairness criteria. The paper then uses this simulated dataset to empirically evaluate three existing approaches for enforcing fairness on deep learning models using adversarial techniques. For each approach, they evaluate group-based fairness criteria such as demographic parity, equal opportunity, and equal odds under different dataset simulations. \n\nThe most potentially impactful contribution of this paper is the simulated dataset and the comparison of different approaches under different dataset simulations. \n\n############# Strengths ##############\n\n- Their proposed simulated modifications of MNIST are well documented. \n\n- The methodology of stress-testing different algorithms by tuning different aspects of the simulated data distribution seems intuitive and useful.\n\n- The experiments are well structured, and they provide a mostly clear explanation for tuning each of the dataset simulation parameters. \n\n- While they only evaluate a few group-based fairness criteria in their experiments, they provide an implementation to evaluate many more metrics listed in Table 2 in the appendix.\n\n- The experiments provided seem thorough, with many different dataset simulation parameters. \n\n############# Weaknesses ##############\n\n- The dataset features seem a bit contrived, and it’s not entirely clear how the different simulated features correspond to real data scenarios. For example, as one modification of MNIST, they add 6-20 drawn horizontal lines on some digits, and 2-6 drawn lines on the rest of the digits. They modify how many of the 6-20 line digits are odd, and how many are even, such that the number of lines on the digit can correlate with whether the digit is even or odd. It’s not clear to me if there are real image datasets where something like this would occur. Of course, I recognize that it’s impossible to capture every possible real data distribution in a simulation like this, but it would be helpful to have some explanation of why they use horizontal lines specifically. A similar explanation would be helpful for the use of the green column.\n\n- They only use a binary sensitive attribute, encoded in the simulation as various shades of red/blue. It would be nice to see an example with more than two categories for the sensitive attribute, as the difficulty of satisfying many of these group-based fairness measures can increase with more categories.  \n\n- The related work has limited discussion of other papers that evaluate multiple fairness criteria and algorithms. The closest citation is Friedler et al. 2019, but they do not discuss or compare to the findings of Friedler et al 2019, which also evaluates four different algorithms for enforcing group-based fairness. Specifically, this paper should discuss how their conclusions and evaluations relate to the conclusions and evaluations of Friedler et al. 2019. (As a concrete example, the Stability analysis in Section 7.2 of Friedler et al. 2019 is very similar to this paper’s “Impact of seed” analysis in the Experimental Results section.)\n\n- The paper would benefit from deeper discussion of how or why the three evaluated approaches differ in performance. Otherwise, the experimental results are not particularly novel (for example, the fact that a correlation between the sensitive attribute and the label can lead to unfair predictions is well known). \n\n############# Recommendation ##############\n\nOverall, my recommendation is 5: Marginally below acceptance threshold. While this paper has a nicely organized experimental section and a clear methodology for simulating a dataset, I don’t think there’s enough discussion of the dataset features and enough discussion or novelty in the experimental results themselves. I do think this type of survey work can be impactful, and would encourage the authors to add more discussion of the methodology, results, and related work, as suggested above.\n\n############# Questions and clarifications ##############\n\n- The dataset feature g_2 is introduced in Section 4.1, but not mentioned in experiments. Is g_2 used in any experiments? \n\n- What were the sizes of the train, validation, and test sets? \n\n- How many seeds were used for each experiment? For example, in the “Impact of seed” experiment, how many seeds were used to compute the standard deviation? \n\n- In Section 5, the authors say, “For each metric, the best value is reported, meaning the performance of the best-performing model on the fairness metric (chosen across hyper-parameters on the validation set) is reported on the test set...This would allow us to report the best performance on each metric, without choosing how to compromise between accuracy and fairness metric.” I don’t understand what this means. Does this mean that they chose a different model for each metric reported? Did they choose one model for the equal opportunity metric, and a different model for the accuracy metric? \n\n- On page 7, the authors say “LAFTR models drop performance in the biased setup of g_1 = 2.” Can the authors clarify what they mean by “drop performance”? Which metric is hurt? And why is g_1=2 considered to be a “biased” setup? It seems the g_1=2 is simply a setting when it is easy to identify odd from even numbers -- I don’t understand why the authors frame this as “biased.”\n\n############# Additional feedback ##############\n\n- The related work section is currently not organized well. I would suggest breaking the large block of text on page 2 into multiple paragraphs addressing different aspects of related work. For example, have one paragraph discussing different methods for enforcing fairness on deep learning models, and have a different paragraph discussing other survey papers for evaluating various fairness criteria and algorithms.\n\n- The plots in the Experimental Results section are really tiny and hard to read (Figures 2-4). \n\n- [page 6, “The goal is to evaluate when such cases arise, do models perform fairly in test setups, where everything is completely balanced?”] It’s great to outline the goal so clearly, but this isn’t actually a grammatically correct sentence. I would suggest modifying this (and similar other sentences in the paper) to: “The goal is to evaluate the following: when such cases arise, do models perform fairly in test setups, where everything is completely balanced?”\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "h_e9AK7Zsq", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "1. Summarize what the paper claims to contribute. Be positive and generous.\nThe paper claims to contribute the following:\n  (1) Show that models exploit any correlation between eligibility and the sensitive attribute found in training data even if the test data does not have such correlation; This leads to unfair predictions.\n  (2) Show that models can exploit a correlation between non-sensitive attribute or small visual features and the eligibility in the dataset; This leads to unfair predictions.\n  (3) Demonstrate that the choice of seed can affect the results significantly; the impact varies considerably across different models.\n  (4) Provide a deep learning codebase composed of six debiasing models and a baseline.\n  (5) Provide a dataset with different controllable sets of features and correlation among them.\nAgreed that there isa  void of a common framework. Thanks for your proposed contributions with the paper!\n\n2. List strong and weak points of the paper. Be as comprehensive as possible.\n  (1) Strengths\n    a. Exhaustive analyses and graphs. I especially liked Figure 8 (Correlation between dataset feature and model's prediction). I enjoyed reading the qualitative analyses too.\n  (2) Weaknesses\n    a. Sensitive attribute values are limited to binary values. In contrast, the real-world sensitive attributes are multi-label.\n    b. What type of sensitive attribute does the variation of the number of stripes represent? I can see that the color may represent skin tone differences and that the small pixel changes may represent the size of the correlation/bias in the input image. I am not sure what stripes may represent.\n\n3. Clearly state your recommendation (accept or reject) with one or two key reasons for this choice.\nAccept because the paper helps the reader better understand and compare the prior arts in ML fairness and their practical performance against various fairness metrics. Would like to understand the answer to 2.(1).b (above) better though.\n\n4. Provide supporting arguments for your recommendation.\nSee 2.(1) and 3 above.\n\n5. Ask questions you would like answered by the authors to help you clarify your understanding of the paper and provide the additional evidence you need to be confident in your assessment. \nSee 2.(2) above.\n\n6. Provide additional feedback with the aim to improve the paper. Make it clear that these points are here to help, and not necessarily part of your decision assessment.\nJust as FYI, it was difficult to read the labels of the third graph of Figure 2 when printed on A4 paper. I had to revisit the PDF and zoom in to see that the graph title is \"clr-ratio(0.01, 0.01)\" instead of \"clr-ratio(0.03, 0.03)\". Would be good if those titles are repeated in the Figure 2 description to help with reading.\n\n\n**Final Update**\nAfter reading other reviewers' comments and authors' responses, I decided to choose to reject the paper at this time for a few reasons.\n\n1. I am afraid that the impact of the paper won't be good with only one out of the four reviewers understanding a paper meant to present an entry point benchmarking dataset.\n\n2. I agree the binary, pseudo sensitive attributes, such as stripes, make the benchmarking and the analysis more manageable. However, as a future user of the benchmarking dataset, I see that there still remains the question of whether the metrics measured against the proposed benchmarking dataset with simple pseudo sensitive attributes can translate well to the model performance measured against some real world dataset with real sensitive attributes. It will help to add the best or chosen model's performance against some real world dataset with real and more complicated sensitive attributes.\n\n3. Add something like what is proposed in #2 to help the readers understand how to choose the best model and make tradeoffs for a given scenario using the presented dataset with the proposed methods. The goal would be somewhat similar to that of the \"INTENDED USES\" section of the LFW dataset presented in http://vis-www.cs.umass.edu/papers/lfw.pdf.\n\nNevertheless, I believe the authors' core work is in the right direction. Just need some way to organize and present it better to make a bigger impact. Thank you for your work!", "extra_scores": {}, "extra_text": {}}, {"review_id": "E6Owj_hDhYQ", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a synthetic dataset to use for benchmarking representation learning algorithms on a variety of fairness criteria. Using this benchmark, it compares several recently developed and prominent methods for fair representation learning and shows their relative strengths and weaknesses under different data generation assumptions and for different metrics.\n\nThe type of careful, systematic comparison provided by this paper is potentially an important contribution. With so many fairness metrics and methods to choose from, it is important for the community to understand limitations and under what sort of conditions a given method may be appropriate or perform best.\n\nHowever, I recommend rejection for this paper for one main reason: I believe the synthetic dataset is inadequate for use as a realistic benchmark. It is too abstract and unlike examples where people are concerned about fairness, and the protected attribute--background color--is something which can be encoded far too simply. As a result, I believe that the relative performance of methods evaluated on this benchmark dataset may be not be indicative of their relative performance on a more realistic dataset with more complex sensitive attributes like race and gender.\n\nIt is not obvious how to improve this flaw, since more realistic datasets may also be less general and hence less useful for benchmarking purposes. One possibility is to abandon the goal of providing a benchmark, since fairness considerations may be too problem dependent for this anyway. In this way, the paper could propose a systematic comparison of methods for a particular synthetic task, provided that task is realistic enough to be related to some potential real world applications. I have no specific suggestion for how to modify the synthetic data generation process, but the features, especially the sensitive attribute(s), need to be rich enough to be more like a real fairness application.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper studies benchmarking of bias mitigation methods. The authors propose a synthetic dataset of images (alike colored-MNIST) that enables a controlled setup over different types of correlations between a binary sensitive attribute, dataset features, and a binary outcome label. The authors have evaluated 2K models that are the variants of three recently proposed debiasing methods using fair representation learning across various settings.

While the reviewers acknowledged the importance of benchmarking fair learning methods in a systematic controlled setting, they have raised several concerns: 
(1) the proposed benchmark is too abstract/unrealistic (R4, R2, R3); it is not clear whether the findings from this benchmark can be generalized to real-world data with real sensitive features, (2) the proposed benchmark is limited to pseudo sensitive attributes (R1) that are binary (R1, R2), (3) the paper lacks in-depth analysis on why certain methods work under certain conditions (R3). Among these, (2) did not have a major impact on the decision, but would be helpful to address in a subsequent revision, (3) was partially addressed in the rebuttal. However, (1) makes it very difficult to assess the benefits of the proposed benchmark, and was viewed by AC as a critical issue.

The authors provided a detailed rebuttal addressing multiple of the reviewers’ concerns. AC can confirm that all four reviewers have read the author responses and have contributed to the discussion. A general consensus among reviewers and AC suggests, in its current state the manuscript is not ready for a publication. See R1 post-rebuttal encouragement and suggestions how to strengthen the work. We hope the detailed reviews are useful for improving the paper.

Decision: Reject