Scaling Fair Learning to Hundreds of Intersectional Groups
Authors: Eric Zhao
De-An Huang
Hao Liu
Zhiding Yu
Anqi Liu
Olga Russakovsky
Anima Anandkumar
Venue: ICLR 2022 Submitted (2022)
Keywords: machine fairness
intersectional fairness
bias mitigation
fair learning
knowledge distillation

Abstract: Bias mitigation algorithms aim to reduce the performance disparity between different protected groups. Existing techniques focus on settings where there is a small number of protected groups arising from a single protected attribute, such as skin color, gender or age. In real-world applications, however, there are multiple protected attributes yielding a large number of intersectional protected groups. These intersectional groups are particularly prone to severe underrepresentation in datasets. We conduct the first thorough empirical analysis of how existing bias mitigation methods scale to this setting, using large-scale datasets including the ImageNet People Subtree and CelebA. We find that as more protected attributes are introduced to a task, it becomes more important to leverage the protected attribute labels during training to promote fairness. We also find that the use of knowledge distillation, in conjunction with group-specific models, can help scale existing fair learning methods to hundreds of protected intersectional groups and reduce bias. We show on ImageNet's People Subtree that combining these insights can further reduce the bias amplification of fair learning algorithms by 15% ---a surprising reduction given that the dataset has 196 protected groups but fewer than 10% of the training dataset has protected attribute labels.

Under review as a conference paper at ICLR 2022
SCALING FAIR LEARNING TO HUNDREDS OF INTER- SECTIONAL GROUPS
Anonymous authors Paper under double-blind review
ABSTRACT
Bias mitigation algorithms aim to reduce the performance disparity between dif- ferent protected groups. Existing techniques focus on settings where there is a small number of protected groups arising from a single protected attribute, such as skin color, gender or age. In real-world applications, however, there are multi- ple protected attributes yielding a large number of intersectional protected groups. These intersectional groups are particularly prone severe underrepresentation in datasets. We conduct the first thorough empirical analysis of how existing bias mitigation methods scale to this setting, using large-scale datasets including the ImageNet People Subtree and CelebA. We find that as protected attributes are introduced to a task, it becomes more important to leverage the protected at- tribute labels during training promote fairness. We also find that the of knowledge distillation, in conjunction with group-specific models, can help scale existing fair learning methods to hundreds of protected intersectional groups and reduce bias. We show on ImageNet's ' People Subtree that combining these insights can further reduce the bias amplification of fair learning algorithms by 15% surprising reduction given that the dataset has 196 protected groups but fewer than 10% of the training dataset has protected attribute labels.
INTRODUCTION
In many real-world applications, there is significant potential for harm when the predictive prop- erties and performance of machine learning models vary across different demographic populations. This is indeed the case for many applications including facial analysis (Buolamwini & Gebru, 2018), ad delivery (Sweeney, 2013) and search engines (Noble, 2018). Recent works aim to address this by designing learning algorithms that guarantee approximate formal notions of algorithmic fair- ness (Mehrabi et al., 2021). However, there still remains a gap between theory and practice.
This gap is especially pronounced in the case of intersectional fairness. In the real-world, it is critical to simultaneously protect multiple attributes such as gender, skin color and age. Due to increasing disadvantage along the intersecting dimensions of protected attributes (Crenshaw, 1989; Foulds et al., 2020), it is also important to protect the intersections of these attributes. However, there has been scarce work on intersectional and subgroup fairness deep learning contexts, with a large portion of the literature still focusing single (binary) protected attribute settings, such as protecting the Male/Not-Mor Male attribute the CelebA Dataset (Liu et al., 2015).
In this work, we focus intersectional fairness and seek to scale deep learning bias mitigation techniques to multiple protected attributes with hundreds of intersectional groups. Approaches such as training separate classifiers for each group (Wang et al., 2020), upweighting the losses of samples based on the group size, optimizing for the worst-case group outcome (Sagawa et al., 2020), or upweighting errors in carly epochs (Liu et al., 2021) have all been shown to be effective in the single attribute setting. However, is not clear whether they are effective given the intersection of multiple protected attributes, where there is combinatorial number of protected groups and fairness for individual protected attributes does not imply intersectional fairness (Figure 1b). We address the question of how scale existing bias mitigation techniques to this important but challenging setting.

---

Under review as a conference paper at ICLR 2022
Ago
Bulanced gender.
20yr
80yr
Average 80%
70%
50% Accur
Male
Accur
Accuracy
Ba aucoedd by skin color:
Gander
Average 80%
90% Ancuracy
70% Acturacy
Female
Aczunacy
Balanced by age:
Average 80% Accurscy
Avorage 80% Accuracy
Figure I: (a) A toy example of intersectional bias. Suppose all intersectional groups are equally represented in the data. A model has the same 80% average accuracy on each of the listed age groups and genders. However, some intersectional groups (e.g., 60 yr females) have only 70% accuracy. (b) Figure of samples from the ImageNet's "Programmer" synset but balanced along the attributes of gender, skin color and age (taken from Yang et al. (2020)). Our ImageNet experiments explore methods of mitigating bias against the intersection of these attributes.
Summary of Contributions:
1. We perform a thorough empirical analysis of existing deep learning bias mitigation techniques to explore their applicability in multi-attribute fairness settings. We find that as the number of protected attributes (k) increases, attribute labels become necessary to mitigate biases.
2. We conduct the first study of bias mitigation on the ImageNet People Subtree, with 196 protected groups but fewer than 10% of training data having protected attribute labels. In this challenging setting, we show existing bias mitigation methods can reduce bias amplification (Zhao et al., 2017) by 9% against empirical risk minimization (ERM). We identify two key issues with scaling up to so many intersectional groups: model complexity and overfitting to limited attribute labels.
3. We address these scaling challenges by proposing novel regularization method: Knowledge Distillation Independent Models as Regularization (DIR). DIR regularizes a single student classifier with teacher classifiers trained specifically for each intersectional group implicitly incorporate attribute label information whenever it's available. In contrast to existing techniques with complexity linear in the number of protected groups, DIR has constant inference complex- ity and model size when used as a stand-alone bias mitigation algorithm. We further demonstrate that DIR can regularize other bias mitigation algorithms, reducing bias amplification by addi- tional 15% on ImageNet.
We first conduct a thorough evaluation of existing bias mitigation algorithms through a series of fair learning tasks on the CelebA dataset (Liu et al., 2015). Each round introduces new protected attribute and evaluates whether existing methods can protect the resulting intersectional groups. Our findings demonstrate that, even with hundreds of intersectional groups, there are bias mitigation algorithms which can significantly reduce bias. However, they all require access to protected at- tribute labels during training. While bias mitigation methods which do not require attribute labels are occasionally effective in single-attribute settings, as suggested in prior literature (Liu et al., 2021; Shrestha al., 2021), we show they are not effective multi-attribute scenarios.
We then consider the more challenging task of learning a fair classifier on the ImageNet People Subtree, which Yang et al. (2020) labels for 196 intersectional groups. This setting especially challenging for two reasons. First, our previous results suggest that access to protected attribute labels during training is necessary to effectively protect intersectional groups. However, such labels are scarce our ImageNet dataset, with less than 10% of training datapoints having such informa- tion. This leaves bias mitigation algorithms prone to overfitting on the few available attribute labels. Second, bias mitigation algorithms often have model size and complexity that scale linearly with the number of protected groups, which, in turn, explode combinatorially due to intersectionality.
We address these challenges by proposing Knowledge Distillation of Independent Models Regu- larization (DIR). DIR uses group-specific teacher classifiers to train a single student classifier that, at each datapoint, mimics the output of the teacher classifier corresponding truth group. DIR, whose model complexity is independent of the number of protected attributes (k), efficient alternative existing bias mitigation algorithms which scale linearly with k.
the datapoint's ground-
2

---

Under review as a conference paper at ICLR 2022
DIR also provides regularization scheme for other bias mitigation techniques, reducing overfitting by implicitly incorporating attribute label information into learning objectives. Using DIR, we can significantly mitigate bias towards intersectional groups on ImageNet, empirically demonstrating 22% reduction in bias amplification over standard empirical risk minimization.
2 RELATED WORK
Algorithmic Fairness Prior works have explored algorithmic bias in a number of real-world ap- plications. Deep learning models, including commercial image classifiers (Buolamwini & Gebru, 2018) and natural langauge processing (Alvi et al., 2018; Bolukbasi et al., 2016; Garg et al., 2018), have been the subject of significant scrutiny. Recent literature have proposed methods for bias mit- igation (Edwards& & Storkey, 2015; Ramaswamy et al., 2021; Ryu et al., 2018; Wang et al., 2020; Zhao al., 2017), approximately optimizing metrics like worst-case accuracy (weighted the worst-off group), mean accuracy (balanced over protected groups), bias amplification scores (Zhao al., 2017), and intersectional bias scores (Foulds et al., 2020). Most build on existing techniques in classical fairness, such as reductions approaches, adversarial models, fairness through awareness, and importance weighting (Agarwal et al., 2018; Saerens et al., 2002; Dwork al., 2012: Zhang et al., 2018). Many also borrow on techniques from robust learning (Adragna et al., 2020; Liu et al., 2021; Sagawa et al., 2020), causal inference (Arjovsky et al., 2020; Creager al., 2021; Kusner et al., 2017; Madras et al., 2019), and representation learning (Pezeshki et al., 2020).
Intersectional Multi-Attribute Fairness One of our contributions replicating existing results (Wang et al., 2020; Shrestha et al., 2021; Liu et al., 2021; Sagawa et al., 2020) and extending their empirical analysis to analagous multi-attribute settings. Shrestha et al. (2021) is the most related work of multi-attribute fairness, but only compare existing algorithums against new sets of datasets. In contrast, we focus on the ImageNet and CelebA datasets and observe algorithms as we tune the number of protected attributes (on fixed dataset). We reach conflicting findings with Shrestha et al. (2021). While methods that do not access protected attribute labels may appear to be effective in certain single-attribute settings, their performance deteriorates in multi-attribute settings. We also find that a reweighting baseline is competitive with methods proposed by Wang et al. (2020). Beyond the deep learning settings considered by our paper and the aforementioned literature, Kang et al. (2021) considers variational method for muuli--atrribus faimess in classical learning settings.
Fairness with Partial Unknown Attributes In many settings, it may not be practical to label the protected attributes of all datapoints. The setting where only part of the dataset has attribute labels has been explored in previous works: Dai Wang (2021) targets a specific graph neural network setting and Ho et al. (2020) provides basic analysis of adversarial faimess. Some works have looked at the case where no labels are available (Chen et al., 2019; Hashimoto et al., 2018). Others have looked at learning proxy labels from different task (Kallus et al., 2020; Awasthi et al., 2021). Our work focuses on the partial unknown attributes case and provides an in-depth analysis with extensive experiments and larger coverage of methodologies.
3 PRELIMINARIES
We formalize our problem as learning a model h : X - Y, where X is the input space and Y a finite label space. We seek to learn a h that fair with respect to m-dimensional protected attribute vector A where is the number of protected attributes. Fairness datasets are thus in tuples (z, a, y) with image x ~ X, protected attribute vector a E A, and single/multi-label label y E Y. For example, protected attribute vector might look like (male, caucasian, teenager, not veteran).
For the entirety of this paper, "labeled/unlabeledd" refers to whether the protected attribute labels A available-we always assume the imput features X and class labels Y are available. Unlabeled data refers to datapoints where X and Y are available, but A is not.
To be fair with respect to the protected attributes A, we want be fair to the intersectional groups G arising from A. The number of such groups is generally combinatorial in m. An example is shown in Figure 1(a). We formalize faimess among groups by reviewing two common definitions.
First, demographic parity concerns parity in Pr(Ø g). the probability of predicting a label 9 e Y for datapoint belonging group y E G. This definition of fairness requires that, over some fixed
3

---

Under review as a conference paper at ICLR 2022
Table List of the bias mitigation algorithms evaluated in Sections 4 & 5.
Name
Abbreviation
Needs Attribte Labels
Reweighting
Adversarial
Empirical Risk Minimization
U-ERM Weighted ERM
No
No
No
Uniform Reweighting
Yes
Yes
No
Group Optim. (Sagawa al., 2020)
Yes
Adversarial Censoring (Edwards & Storkey, 2015)
Advensarial
Yes
Yes
Invariant Risk Minimization (Arjovsky al., 2020)
IRM
Yes
No
No
Domai Independence (Wang al., 2020)
Independent-SP
Yes
No
No
Discriminative (Dwork al., 2012)
Discriminative U-SD
Yes
Spectral Decoupling (Pezeshki et 2020)
No
No
Just Tran Twice (Liu et al., 2021)
U-IT
No
Yes
Yes
data distribution, the predictions of a model h are probabilistically independent of the protected group membership. Formally, Vg' E G. : Pr(ū g) = Prr( g'). This objective is generally more applicable to settings where predicted class a decision (e.g.. issuing a loan).
Second, the equalized odds definition refines the demographic parity constraint to enforce parity in Pr(ý g: y), the probability of predicting a label ý E Y, for a datapoint with true label y Y belonging group E G. This definition enforces, for instance, equivalent accuracies for each group. Formally, Vig' G.O.NEY: Pr(ğ 9.9) = Pr(O ())
3.1 FAIRNESS METRICS
Given the difficulty of analyzing neural networks, which notions of fairness are violated. For equalized odds, two common surrogate metrics include the sepportance-eiihhee accuracy and the worst-case accuracy. Importance weighted accuracy "reweighted accuracy") is the accuracy weighted such that each group has equal representation.
metrics that approximate the margin by
Accy (h) E Po..
|
1ly = n(w))]
Worst-case accuracy is the accuracy of the group with the lowest accuracy.
Acew (h) min (xx., (x))]]
g' EG
Similarly, bottom-quartile accuracy defined as the average of the bottom quartile of group aceu- racies. Papers based on robust learning often use worst-case accuracy metries (Sagawa et al., 2020; Liu et al., 2021). However, more general papers also use importance weighted loss, such as Wang al. (2020). We also define such reweighted and worst-case metrics for Mean Average Precision (mAP), which we use in place of accuracy for multi-label settings.
For demographic parity, two common metrics include bias amplification (Zhao et al., 2017) and in- tersectional bias (Foulds et al., 2020). Bias amplification measures the difference between a group's ground-truth representation for label class versus the group's predicted representation:
Sy = Pr(g h(z)) Pr(g y) where == augmax,yee Pr(g') h(z)).
(3)
A positive score indicates that the model amplifies biases in the data. Generally, we seek bias am- plification score close-to or below zero. However, differences between negative bias amplification scores non-informative. Bias amplification have been widely adopted by prior works like Wang et al. (2020), but has been found to be limited in what it captures (Wang & Russakovsky. 2021).
The intersectional bias (Foulds et al., 2020) more directly measures violations of the demo- graphic parity constraint. Larger intersectional bias indicates greater demographic parity violations.
Pr(h(z) argmin, s.t. exp(-c) Pr(h(z) g2)
91)
cy S1 22
exp(e)
(4)
In contrast to bias amplification, we seek as low an intersectional bias score as possible. However, due to noise, intersectional bias scores be -onioofrreeeve when different methods have very similar scores. Since these metrics are defined for each class label y. in multi-class and multi-label
settings we will occasionally average the metrics over their top quartile of class labels.
4

---

Under review as a conference paper at ICLR 2022
Varying Protected Attributes Number on CelebA Blond Hair/Not Blond Task
Roweighted Accuracy
3.0
0.90
0.00
2.5
Bias Amp
Inter: Blas
-0.01
2,0
1.5
-0.02
0.80
-0.03
-0.04
3
7
3 5
# prot. attributes
prot. attributes
prot. attributes
GDRO
Independent
U-ERM
U-JTT
U.SD
Weighted ERM
Figure 2: Bias mitigation algorithms CelebA-SL task for k [1. 7]. The left figure plots the reweighted accuracy. Independent-SP and U-JTT have low reweighted accuracy compared to the remaining methods. The middle figure plots bias amplification. Only Independent-SP has above zero bias amplification score. The right figure plots intersectional bias. U-ERM and U-JTT have poor intersectional bias. The unlabeled method U-SD has competitive intersectional bias for small k but, for large k, is outperformed by Weighted ERM with margin increasing in k.
3.2 METHODS FOR ANALYSIS
Table I lists the bias mitigation algorithms explored in this paper. We consider both "unlabeled" and "labeled" methods, which distinguished by whether they use protected group information during training. Our definition of "unlabeled" methods still assume access to class labels; they instead lack access to protected attribute labels. Abbreviations of unlabeled methods are prefixed with "U-". We include empirical risk minimization ("U-ERM") as a baseline. Other unlabeled methods we consider include spectral decoupling (Pezeshki et al., 2020) and Just Train Twice (Liu et al., 2021).
For labeled methods, we include weighted ERM where the risk in Equation is approximated with maximum maxiimmm-ikkelhoodd-eeoo likelihood-estimate of group importance weights. We also include robust optimization methods that use adversarial importance weights to either maximize the worst-case group accuracy (GDRO) (Sagawa et al., 2020) or eliminate protected group information (Adversarial) (Edwards & Storkey, 2015). We also include invariant risk minimization (IRM) (Arjovsky et al., 2020) for completeness. We include the domain independent and discriminative algorithms (Wang et al., 2020), which train collection of group-specific classifiers and ensemble them for inference.
4 ANALYSIS TOWARDS MULTIPLE PROTECTED ATTRIBUTES
In this section, we analyze how different bias mitigation techniques perform in settings with multiple protected attributes. We build the CelebA dataset (Liu al., 2015) experiment settings consid- ered in Wang et al. (2020). Liu et al. (2021), and Shrestha et al. (2021). Specifically, we expand their antliee experiments by incrementally introducing additional protected attributes. The Appendix includes additional replication results alongside other competitive baselines.
Experimental Setup. We consider two primary experiment settings: (I) binary classification task, CelebA-SL, on CelebA's blond hair class label, taken from Liu et al. (2021) and Shrestha et al. (2021). (2) multi-labe classification task, CelebA-ML, on 33 out of 40 of CelebA's class labels, taken from Wang et al. (2020). For CelebA-MI, the 7 omitted class labels are reserved as protected attributes and specified in the following paragraph.
For both CelebA-SL and CelebA-MI, we train residual networks (He et al., 2016) using the algo- rithms listed in Table 1. We evaluate how these methods perform as new protected attributes are introduced by repeating the learning task while each time introducing new attribute protect. Let denote the number of protected attributes. Since every CelebA attribute is binary, every protected attribute that is added doubles the number of intersectional groups. The protected attributes that are added are taken from a fixed set to reduce variance. The pool of potential protected attributes is, in order: Pale Skin, Male, Narrow Eyes, Big Nose, Young, Straight Hair, and Attractive. These
5

---

Under review as a conference paper at ICLR 2022
Varying Protected Attributes Number on CelebA Labei Task
0.8
0.8
0.100
Reweighted mAp
Top Bias Amp
0.075
Top Qr Inter. Blas
6
Bot Qt mAP
0.6
0.6
0.050
4
0.44
0.025
2
0.2
1
5
1
5
5
1
3
5
prot. attributes
prot. attributes
prot. attributes
prot. attributes
GDRO
Independent
U-ERM
U.JTT
Weighted ERM
Figure 3: Bias mitigation algorithms on CelebA-ML task for E [1, The left figure plots the reweighted mAP. The left middle plots the bottom quartile mAP scores. U-ERM and Independent- SP score marginally higher than Weighted ERM. The right middle plots bias amplification with Weighted ERM having the lowest bias amplification. The right figure plots intersectional bias with Independent-SP having the lowest intersectional bias. U-JTT starts with low intersectional bias, but grows beyond that of Weighted ERM and Independent-9 with the number of protected attributes.
tributes were selected pseudorandomly (i.e., without tuning) for their problematic associations with protected demographics. Further details on dataset and model hyperparameters are in the Appendix.
Primary Results. CelebA-SL and CelebA-ML are depicted in Figures 2. 3 with the key findings:
We can mitigate intersectional bias and bias amplification, even for hundreds of intersectional groups, using labeled methods like Domain Independent and Weighted ERM.
2. In contrast prior findings by Liu al. (2021); Shrestha al. (2021), find that unlabeled methods generalize poorly to other fair learning tasks and, even if they perform competitively for small k (number of protected attributes), they have bad intersectional bias for large k.
There is no consistently superior bias mitigation algorithm. Methods like GDRO, Independent- SP, and Weighted HRM may each perform best depending on the learning task, performance/bias metric, and the value of k.
Intersectional Bias Mitigation On Celeba-SL. (Figure 2), Weighted ERM and GDRO are the best overall methods, with similar accuracy, intersectional bias, and bias amplification for k S Both methods successfully reduce bias amplification and intersectional bias (versus U-ERM) by signif- icant margin without losing accuracy. On CelebA-ML (Figure 3), Weighted ERM and Independent- SP are the best overall methods, with Independent-SP outperforming Weighted ERM in terms of bottom quartile mAP (59% vs 53% for & = 3) and intersectional bias (5.5 vs 6.2 for k = 7). Again, these best methods for CelebA-ML successfully reduce bias amplification and intersectional bias (versus U-ERM) by a significant margin without significant loss of accuracy. These results suggest that bias reduction is possible, even under highly intersectional settings with large k, but the optimal choice of a bias mitigation algorithm is context-dependent. All the best methods we found in this section are labeled methods.
Unlabeled Methods Prior works found that unlabeled methods perform competitively with labeled methods when protecting a single attribute CelebA-SL (Shrestha et al., 2021; Liu et al., 2021). Appendix A.1 replicates their experiments with highlights in Tables 2-3. Table 3 confirms that U-JTT yields competitive worst-case accuracy and bias amplification (85% Weighted ERM's 89% accuracy; -0.041 vs -0.055 bias amp). Table 2 shows U-SD also yields competitive worst-case accuracy and bias amplification (82% vs GDRO's 83% accuracy; -0.057 vs -0.065 bias amp).
However, we find these prior results are fragile. Figures 2-3 shows U-JTT has consistently middle- of-the-pack bias scores and low reweighted accuracies for all k. Even on Figure 2's k=1 case, JTT's 88% reweighted accuracy is 3%+ below others. The only difference between Figure 2's k=1 case and Table 3 is that the former protects "Is/Not Pale Skin" attribute instead of "Is/Not Male". In Figure (CelebA-SL), U-SD fares better with competitive reweighted accuracy (92% weighted ERM's 92%), bias amplification (0.001 vs weighted ERM's 0.003), and intersectional bias scores (0.3 vs weighted ERM's 0.3) for k=1. However, for & > 3, the intersectional bias of U-SD is

---

Under review as a conference paper at ICLR 2022
Table 2: Highlights from Shrestha et al. (2021) replication, a variant of Figure 2's CelebA-SL task for k= A R is used and the protected attribute is "Male". U-SD is competitive with GDRO (the best labeled method), and reduces bias significantly versus U-ERM.
Model
Inter. Bias
Amp
Reweighted Accuracy
Accuracy
U-ERM
000 = 0. 109
0.024 + 0.012
0.848 + 0.012 0.895 = 0.009
0.604 = 0.065
GDRO
0.558 0.019
-0.063 + 0.008
0.833 034
U-SD
0.746 0.032
-0.057 + 0.001
0.887 0.005
0.817 + 0.025
Table 3: Highlights from Liu et al. (2021) replication, a variant of Figure 2's CelebA-SL task for k=1. The protected attribute is "Male". U-JTT is competitive with Weighted ERM (the best labeled method), and reduces bias significantly versus U-ERM.
Model U.E
Inter. Bias
Bias Amp
Reweighted Accuracy
Min Accuracy
1.107 1. 0.606
0.024 1 0.018
0.715 1. 0.148
0.281 0.197
U-ITT
0.911 0.005
-0.041 + 0.002
0.907 + 0.001
0.842 0.003
Weighted ERM
0.647 0.101
-0.055 0.012
0.921 1 0.003
0.891 1 0.026
consistently greater than the labeled baseline, Weighted ERM: 1.9 vs 1.5 for A = 5 and 3.0 vs 2.6 for k = 7. U-SD was not evaluated on CelebA-ML as its hyperparameters are intractable to tune for large numbers of label classes. These negative results highlight the limitation of unlabeled methods and thus the importance protected attribute labels, particularly in the multi-attribute domain.
Intersectional Bias and Bias Amplification Trends For both CelebA-SL and CelebA-ML, inter- sectional bias scores are generally increasing in & (see the right-most plots of Figures 2, 3). The noise of the metric itself, however, does not grow significantly with the number of protected at- tributes. Bias amplification scores are significantly more noisy (see all lines in Figure 2 and the U-JTT line in Figure 3). We also find that bias amplification is uninformative for CelebA-SI. with all methods but Independent-Si having a bias amplification score below zero However, we attribute this to the binary classification nature of CelebA-SL where, although there is class imbalance (85% to 15%), there is sufficient number of datapoints per class that there is little risk of underfitting with ResNets. Bias amplification remains non-trive non-trivial on the multi- label CelebA-ML. and, as we will see later, is also non-trivial on the multi-class prediction ImageNet task.
SCALING TO REAL-WORLD DATASETS
We now consider learning fair models on the ImageNet People Subtree, with protected attribute labels annotated by Yang et al. (2020). This dataset realistic benchmark for intersectional bias mitigation, featuring 196 intersectional groups arising from 3 protected attributes: gender, skin color, and age. In addition, only 15,981 images of the dataset's 124.693 images have protected attribute labels padling challenge with label scarcity. We found in Section 4 that bias mitigation is pos- sible with hundreds of intersectional groups but that labeled methods are imperative. Can we still effectively mitigate bias on ImageNet with hundreds of protected groups but scarce attribute labels?
We first analyze the performance of existing labeled and unlabeled bias mitigation algorithms on ImageNet. We then improve on those results by proposing a general knowledge distillation based approach (DIR) that helps scale bias mitigation algorithms to the ImageNet People Subtree.
5.1 BIAS MITIGATION TECHNIQUES ON THE IMAGENET PEOPLE SUBTREE
Experimental Setup. For this section, we use the ImageNet People Subtree annotated by Yang et al. (2020). We have 130,554 images for training (5.861 of which have attribute labels). We have 5,327 images for validation and 5,327 for test, all with attribute labels. 3 protected attributes (gender, skin color, age) are annotated with 196 corresponding intersectional groups. This raises the issue of zero-shot fairness, 237 out of 284 synsets the people subtree have zero representation of dark-skinned females (contrast this to the only out of 284 synsets that lack light skinned male rep- resentation). The methods considered in this section are described in Table 1 with the exception of weighted ERM, which we modify to square-root all importance weights- -his is to remedy the high-variance importance weights that arise on ImageNet.
Existing Methods on ImageNet. Table 4 shows the performance of various labeled and unlabeled methods ImageNet. The key result is that, using existing labeled bias mitigation methods, can
7

---

Under review as a conference paper at ICLR 2022
Table 4: Results of existing bias mitigation algorithms on the ImageNet People Subtree. Weighted ERM and Independent-SP significantly improve reweighted accuracy and reduce bias amplification versus standard ERM. All intersectional bias scores are within a standard deviation.
Roweighted
Amp.
Blas
U-ERM
45.38 + 0.404
0.08153 + 0.0095
2.684 + 0.025
U-JTT
45.42 0.460
0.07603 + 0.0105
2.694 = 0.00G
U-SD
36,39 0.352
0,08-189 + 0.0116
2.621 + 0.027
GDRO
45.20 1. 0.136
0.08910 1 0.0170
2.697 1: 0.014
Weighted ERM
47.67
+ 0.812
0.07611 + 0.0011
2.687 + 0.027
Independent-SP SP
47.24 1 0.817
0.07402 1 0.0095
2.688 1: 0.049
Table 5: Comparison of pretrained and not-pretrained bias mitigated models on ImageNet. Pretrain- ing significantly improves reweighted accuracy and reduces bias amplification. However, pretraining increases intersectional bias (clarified in Section
Reweighted Acc.
Bias Amp.
Inter. Bins
Weighted ERM
47.67
0.812
0.07611 +: 0.0011
2.687 = 0.027
w/o pretrain
14.88 0.670
0.09602 + 0.0154
2.107 0.037
Independeut-SP
47.24 0.817
0.07402 + 0.0095
2.688 = 0.049
w/o pretirain
17.36 0.255
0.1429 0.0110
2.268 0.012
significantly mitigate bias. Weighted ERM and Independent-SP both outperform standard ERM in terms of reweighted accuracy (47%, 47% 45%) and bias amplification (0.076, 0.074 vs 0.082) by a significant margin. The intersectional bias metric is uninformative, i.e., all methods are within standard deviation of each other.
We also find that unlabeled methods U-JTT and U-SD (45% 37% accuracy) are significantly out- performed by Weighted ERM and Independent-SP (47% accuracies). U-JTT and U-SD's bias am- plification and intersectional bias scores are within standard deviation of Weighted ERM and Independent-SP" This corroborates our Section findings of the limited applicability of unlabeled methods in multi-attribute settings where protected attribute information is critical.
Pretraining, An important note about the results in Table is that all methods are pretrained with ERM on the entire training set, including data points without attribute labels. This is justified in Table 5 which compares Table 4 results against their not-pretrained counterparts. Pretraining is im- portant for obtaining reasonable performance, improving reweighted accuracy (15% 47% for Weighted ERM and 17% + 47% for Independent-SV and reducing bias amplification (0.096 -> 0.076 and 0.143 + 0.074 respectively). Surprisingly, pretraining increases intersectional bias (2.16 + 2.69 for Weighted ERM and 2.27 + 2.69 for Independent-SP), This is a flaw of the metric. Low accuracy, high uncertainty neural networks act like random classifier, which have intersectional bias close to zero (Eq 4); this is not the case with bias amplification, for instance.
5.2 KNOWLEDGE DISTILLATION FOR SCALING LABELED BIAS MITIGATION TECHNIQUES
Our findings in Section 5.1 demonstrate that bias mitigation methods, including Weighted ERM and Independent-SP. can successfully mitigate bias in multi-attribute settings. However, we encountered two key issues from scaling to ImageNet.
Model Size. The number of protected groups grows combinatorially with the number of protected attributes. This poses an issue for methods which have a model size linear in the number of protected groups, including Independent-S!! Adversarial, and Discriminative. Their inference complexity increases combinatorially. This necessitated low batch sizes for the experiments Table
Overfitting. Given the large number of protected groups but scarce protected attribute informa- tion, labeled bias mitigation algorithms are prone to overfit to available attribute information. For instance, methods which estimate importance weights over protected groups, c.g. Weighted ERM, are prone to high-variance weight estimates. This also affects methods, like Independent-SP and Diseriminative, which use protected attribute labels to selectively update network weights: weights associated with underrepresented groups may overfit to the scarce datapoints labeled with the group.
Knowledge Distillation of Independent Models Distillation of Independent Models as Regularization (DIR) to address these challenges. Specif- ically, develop two-stage bias mitigation approach, where distill set of group-specific
Regularization. We propose Knowledge
8

---

Under review as a conference paper at ICLR 2022
Table 6: Comparison of ImageNet results of using DIR to modify the Weighted ERM and Independent-Shl methods. DIR reduces bias amplification while maintaining the reweighted ac- curacy gains of the original Weighted ERM and Independent-S SP methods.
Roweighted
Amp.
Blas
Independent-SP
47.24 0.817
0.07402 + 0.0095
2.688 + 0.049
DIR Weighted ERM w/ DIR
17.68 0.521
0.06318 0. 0.0100
2.693 + 0.012
47.67 0.812
0,07611 + 0.0011
2.687 + 0.027
47.73 1 0.920
0.07354 1 0.0031
2.880 1 0.015
classifiers into a single classifier. This provides: (1) a bias mitigation method with ((1) inference complexity but the same inductive biases the Ok) complexity Domain Independent algorithm: and (2) a form of regularization that can be used to augment any labeled bias mitigation algorithm.
Given G protected groups, the Domain Independent (Independent-SPP algorithm (Wang al., 2020)
trains a specialized model hg for each protected group g. training on the same X + Y task but exclusively on datapoints labeled with g. Given an input æ, the algorithm directly sums the outputs of the set of fgg.
Σ hg(2),
(5)
9EG
where a is final output of the Domain Independent model. Instead of summing over the specialized classifiers (hg g E G), we propose distillation-basee approach that trains single classifier ha with the following loss for each datapoint (x, y, g) ~ (X,Y, G)
((ha(a), v) + AK ((aa(x), (2)),
(6)
where hg are already trained, e(.,)) is some loss function and À is the weight for the distillation term. This formulation yields a single classifier ha that is regularized by the second term in Equation (6), which incentivizes the output ha(x) to similar to that of hg. the specialized classifier for the datapoint's 's protected group g. We refer to this scheme as DIR. When É is the original task loss function, c.g. cross-entropy. Equation 6 is an approximation of the Nomain-Indepenee with (1) rather than O(k) model complexity. However, we can also use DIR to augment other bias mitigation algorithms. For instance, we can augment Weighted ERM by selecting its importance weighted loss as f. This yields the regularized objective
algorithm
Σ w, vi) + AKI((((;), (z:)),
where wi denotes the importance weight estimate used by Weighted ERM, g' the protected group of x; and hgg the datapoint's group-specific classifier.
Experimental Results. Table depicts the performance gains on ImageNet from using DIR to augment Weighted ERM and Independent-SP The second and fourth rows correspond to models trained with Equation (6) and Equation (7), respectively. DIR reduces bias amplification for both Independent-SP (0.074 - 0.063) and Weighted ERM (0.076 + 0.073) while maintaining the accuracy gains realized by these two labeled methods. Intersectional bias again uninformative.
ETHICAL CONCERNS & CONCLUSION
In this paper, we conduct thorough review of bias mitigation deep learning algorithms in multi- attribute settings. We also propose a knowledge distillation framework for improving bias mitigation on the pootecte--label-seane nology that directly or indirectly interface with protected classes, the insights we find concerning multi-attribute fairness may help better inform interventions for bias mitigation.
ImageNet dataset. In many applications of machine learning tech-
Due to the nature of the datasets that our experiments use, there are number of potential ethical concerns arising from the use of protected attribute labels. Dataset schemas used commonly in deep learning literature, such the subjective "AtrractiveNNot-Attaacttt dataset, can be problematic. In addition, there are privacy and representation concerns inherent in the process of labeling protected attributes. For more detailed discussion, we refer readers to Yang et al. (2020); Paullada et al. (2020); Birhane & Prabhu (2021) which further explore issues in fair dataset design, especially concerning bias and privacy.
attribute adopted by the CelebA

---

Under review as a conference paper at ICLR 2022
REFERENCES
Robert Adragna, Elliot Creager, David Madras, and Richard Zemel. Fairness and Robustness in Invariant Learning: A Case Study in Toxicity Classification. arXiv:2011.006 206-4888106 [cs], December 2020. URL http:/ : /// /arxxvv..v.. org/abs 20110.1111. 06485. arXiv: 2011.06453
Alekh Agarwal, Alina Beygelzimer, Miroslav Dudík, John Langford, and Hanna Wallach. re- ductions approach fair classification. In International Conference on Machine Learning, pp- 60-69, PMLR, 2018.
Mohsan Alvi, Andrew Zisserman, and Christoffer Nelläker. Turning a blind eye: Explicit removal of biases and variation from deep neural network embeddings. In Proceedings of the European Conference on Computer Vision (ECCV) Workshops, pp. 2018.
Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant Risk Minimiza- tion. arXiv:1907.000 02899 [es, stat), March 2020. URL http: org/ haggg/bbb199. 02893.
Pranjal Awasthi, Alex Beutel, Matthäus Kleindessner, Jamie Morgenstern, and Xuezhi Wang. Eval- uating Fairness of Machine Learning Models Under Uncertain and Incomplete Information. In Proceedings of the 2021 ACM Conference Fairness, Accountability, and Transparency, pp. 206-214, 2021.
Abeba Birhane and Vinay Uday Prabhu. Large image datasets: A pyrrhic win for computer vision? In 2021 IEEE Winter Conference Applications of Computer Vision (WACV), pp. 1536-1546. IEEE, 2021.
Tolga Bolukbasi, Kai- Wei Chang, James Y. Zou, Venkatesh Saligrama, and Adam T. Kalai. Man is to computer programmer woman is to homemaker? debiasing word embeddings. Advances in neural information processing systems, 29:4349 4357, 2016.
Joy Buolamwini and Timnit Gebru. Gender shades: Intersectional accuracy disparities in commer- cial gender classification. In Conference on fairness, accountability and transparency, pp. 77-91. PMLR, 2018.
Jiahao Chen, Nathan Kallus, Xiaojie Mao, Geoffry Svacha, and Madeleine Udell. Fairness Under Unawareness: Assessing Disparity When Protected Class Is Unobserved. In Proceedings of the Conference on Fairness, Accountability, and Transparency, FAT* '19, pp. 339-348, New York, NY, USA. January 2019. Association for Computing Machinery. ISBN 978-1-4503-6125-5 doi: 10.1455/3287560.2285334 URL https: 111/1001145/33287566666111111110 3287594,
Elliot Creager, Hörn-Henrik Jacobsen, and Richard Zemel. Environment inference for invariant learning. International Conference on Machine Learning, pp. 2189-2200. PMLR, 2021.
Kimberlé Crenshaw. Demarginalizing the intersection of race and sex: black feminist critique of antidiscrimination doctrine, feminist theory and antiracist politics. Chi. Legal pp. 139, 1989. Publisher: HeinOnline.
Enyan Dai and Suhang Wang. Say No the Discrimination: Learning Fair Graph Neural Networks with Limited Sensitive Attibute Information. In Proceedings of the 14th ACM International Conference on Web Search and Data Mining, WSDM '21, pp. 680-688, New York, NY, USA, March 2021. Association for Computing Machinery. ISBN 978 978-1-4503-8297-7, doi: 10.1145/ 1437963.344175.. URL https:
10. 1tt.:.////.1.111333333353553 379433333333347452. 3441752.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 248-24888 255. leee, 2009.
Cynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Richard Zemel. Fairness through awareness. In Proceedings of the 3rd innovations theoretical computer science confer- ence, pp. 214-226, 2012.
Harrison Edwards and Amos Storkey. Censoring Representations with an Adversary. In ICLR, November 2015. URL https:://// http://axxxw...gaa./////
/arxivv.vvvv wwt./oog/////ooooooooooooeee /1511 05899733.
10

---

Under review as a conference paper at ICLR 2022
James R. Foulds, Rashidul Islam, Kamrun Naher Keya, and Shimei Pan. An intersectional definition of fairness. In 2020 JEEE 36th International Conference on Data Engineering (ICDE), pp. 1918- 1921. IEEE, 2020.
Nikhil Garg, Londa Schiebinger, Dan Jurafsky, and James Zou. Word embeddings quantify 100 years of gender and ethnic stereotypes. Proceedings of the National Academy Sciences, 115 (16):E3635-E364) April 2018. ISSN 0027-8424, 1091-6490. doi:
URL http:::t:://. /www. www.ppaaa... org/ cowtent//11//eene
Tatsunori Hashimoto, Megha Srivastava, Hongseok Namkoong, and Percy Liang. Fairness without demographics in repeated loss minimization. In International Conference on Machine Learning, pp. 1929-1938. PMLR, 2018.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Caroline Ho, Hugo Kitano, and Kevin Lee. Fair Image Classification with Semi-Supervised Learn- ing. Technical report, Stanford, 2020.
Nathan Kallus, Xiaojie Mao, and Angela Zhou. Assessing algorithmic fairness with unobserved protected class using data combination. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency, FAT* '20, pp. 110, New York, NY, USA, January 2020. As- sociation for Computing Machinery. ISBN 978- 1-77---1-53---36--7 doi: 10. 10.1145/3511095.373555 URL https: /ooiiii....// . org 100. 00044/53351595 00....9...3..111 3373154.
Jian Kang, Tiankai Xie, Xintao Wu, Ross Maciejewski, and Hanghang Tong. MultiFair: Multi-
Group Fairness in Machine Learning. arXiv:2105. /069 [cs, math, stat], May 2021. URLhttp:
www.iiiwrrgg/.....aa.aaa..a..e.e www.rr////aaaaaaeeeeeeeeeeeeeeee 2115..1199..
Matt Kusner, Joshua Loftus, Chris Russell, and Ricardo Silva. Counterfactual Fairness. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, 2017.
Evan Zheran Liu, Behzad Haghgoo, Annie S. Chen, Aditi Raghunathan, Pang Wei Koh, Shiori Sagawa, Percy Liang, and Chelsea Finn. Just Train Twice: Improving Group Robustness without Training Group Information. ICML, 2021. URL http://arxiv..rg/
org.abs/////oooooooooooo...... abs / 0011107.00444
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep Learning Face Attributes in the Wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
David Madras, Elliot Creager, Toniann Pitassi, and Richard Zemel. Fairesss through causal aware- ness: Learning causal latent-variable models for biased data. In Proceedings of the conference on fairness, accountability, and transparency, pp. 349-358. 2019.
Ninareh Mehrabi, Fred Morstatter, Nripsuta Saxena, Kristina Lerman, and Aram Galstyan. A survey bias and fairesss in machine learning. ACM Computing Surveys (CSUR), 54(6):1-35, 2021. Publisher: ACM New York, NY, USA.
Safyy Umoja Noble. Algorithms oppression: How search engines reinforce racism. Algorithms of oppression: How search engines reinforce racism. New York University Press, New York, NY, US, 2018. ISBN 978-1-4498-372433 978-1-4499--499-99
Amandalynne Paullada, Inioluwa Deborah Raji, Emily M. Bender, Emily Denton, and Alex Hanna. Data and its (dis)contents: A survey of dataset development and use in machine learning research. arXiv:2012.05344 1.505.5..345...... [cs], December 2020. URL http: / waw.i.xvvvvvroo.. abs / 200..05312..... 05345.
arXiv: 2012.05345.
Mohammad Pezeshki, Sékou-Oumar Kaba, Yoshua Bengio, Aaron Courville, Doina Precup, and Guillaume Lajoie. Gradient Starvation: A Learning Proclivity in Neural Networks. arXiv:2011.09-60 (cs, math, stat], November 2020. httpttp: arxiv. org/abs///11.. 09468.
11

---

Under review as a conference paper at ICLR 2022
Vikram V. Ramaswamy, Sunnie SY Kim, and Olga Russakovsky. Fair attribute classification through latent space de- biabiisi... In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 9301-9310, 2021.
Hee Jung Ryu, Hartwig Adam, and Margaret Mitchell. InclusiveFaceNet: Improving Face Attribute Detection with Race and Gender Diversity. arXiv:1712.00193 [cs], July 2018. URL http:
///aaxiiv.rg/aaaabaa.aa http://rggabb////////// 00193.
Marco Saerens, Patrice Latinne, and Christine Decaestecker. Adjusting the outputs of a classifier to new priori probabilities: simple procedure. Neural Compulation, 14(1):21-41, January 2002.
ISSN 0899-7667. doi: 10.1::1.1120000. 166168996660077924444
Shiori Sagawa, Pang Wei Koh, Tatsunori B. Hashimoto, and Percy Liang. Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Gener-
alization. ICLR, 2020, URL http: / wwtpaaixivvv.o/..aaaaa org/ /tass/991e 08731.
Robik Shrestha, Kushal Kafle, and Christopher Kanan. An Investigation of Critical Issues in Bias Mitigation Techniques. arXiv:2104.00170 [cs, stat], March 2021. URL abs/2104 00170.
Latanya Sweeney. Discrimination in online ad delivery: Google ads, black names and white names, racial discrimination, and click advertising. Queue, 11(3):0029, 2013. Publisher: ACM New York, NY, USA.
Angelina Wang and Olga Russakovsky. Directional bias amplification. ICMI., 2021.
Zeyu Wang, Klint Qinami, Ioannis Christos Karakozis, Kyle Genova, Prem Nair, Kenji Hata, and Olga Russakovsky. Towards fairness in visual recognition: Effective strategies for bias mitigation. In CVPR, 2020.
Kaiyu Yang, Klint Qinami, Li Fei-Fei, Jia Deng, and Olga Russakovsky. Towards fairer datasets: Filtering and balancing the distribution of the people subtree in the imagenet hierarchy. In Pro- ceedings of the 2020 Conference on Fairness, Accountability, and Transparency, 2020.
Brian Hu Zhang, Blake Lemoine, and Margaret Mitchell. Mitigating unwanted biases with adver- sarial learning. In Proceedings of the 2018 AAAI/ACM Conference AI, Ethics, and Society, pp. 335-340, 2018
Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. Men Also Like Shopping: Reducing Gender Bias Amplification using Corpus-level Constraints. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, 2017.
12


Reviews:
[{"review_id": "t5Ohke6MKBK", "reviewer": "Reviewer_CCHj", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength\n\n1. The empirical analysis of existing bias mitigation methods on two large datasets with multiple sensitive attributes/partially labeled protected attributes is very interesting. It shows the scalability of existing methods and the potential challenges when applying those methods to large-scale datasets.\n\n2. A new method for group-specific models is proposed to augment other mitigation algorithms.\n\n\nWeakness:\n\n1. The novelty of this work concerns me the most. The empirical analysis is interesting but far from the expectation of the standard of ICLR. In addition, the proposed DIR method is incremental for model simplification of bias mitigation. It would be considered as transferring the existing method **Domain-Independent model** into a smaller one whose inference complexity is less than the original version, through incorporating the knowledge distillation approach. The distillation process requires |G| pre-trained models for |G| groups.\n\n2. The claimed O(1) complexity is implausible because it assumes the availability of G group-specific models. It seems the proposed method is not able to reduce the complexity without G models. ~~In addition, it is unclear how to integrate multiple classifiers $h_g$ into Eq (6) and (7) because $\\lambda$ is not model-specific.~~ (Resolved)\n\n3. It is unclear how to integrate the DIR method with unlabeled mitigation algorithms as the empirical results show they are vulnerable to poor generalization.\n", "summary_of_the_paper": "This paper conducts an empirical analysis of existing bias mitigation methods on two large datasets CelebA and ImageNet where there are multiple sensitive attributes and some unavailable protected labels. The results show the existing can mitigate intersectional bias at scale but the unlabeled methods generalize poorly. This paper further proposes a knowledge distillation of independent models as regularization method (DIR) which is able to augment into other bias mitigation algorithms.", "summary_of_the_review": "The empirical analysis is interesting but the incremental contribution concerns me the most. In addition, the proposed DIR method is trivial and contains many uncertainties. I would not root for this paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Qr1mEZkEJmq", "reviewer": "Reviewer_F8SZ", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Strengths\n\n- Focus on an important and understudied problem. \n\n- Paper presents a new approach to handle intersectional attributes: Knowledge Distillation of Independent Models as Regularization (DIR).\n\n### Weaknesses\n\n- [Major] Paper evaluates methods on datasets where we have no ground truth. These findings should be confirmed on a synthetic dataset where we have ground truth labels (e.g., binary classification task with $k = 1, 2, 4$ binary group attributes).\n\n- [Major] Key findings of the empirical study are based on summary statistics that do not capture or reflect differences in accuracy across intersectional groups (see e.g., Figure 1, Figure 3, Tables 2 and 3). For example:\n\n     (*) The ``reweighted accuracy\" with k = 128 groups does account for the performance of all intersectional groups. However, it would be unable to flag instances where an intersectional group has an accuracy of 0\\% since any group affects at most 1/128 of the total performance. \n\n     (*) Measures such as Bias Amplication and \"Intersectional Bias Score,\" are surrogate measures that suffer from their own issues (as discussed by the authors on Page 4). The informative-ness of these issues is also hindered in that they represent population-level statistics.\n\n      I would recommend would be better to evaluate methods in ways that do not involve summary statistics at all (e.g., via a plot showing the distribution of group-specific accuracy). I would also recommend the authors use measures of group-level performance (e.g., worst-case accuracy over groups, # of groups with substantial bias). \n\n- [Minor]: The paper is missing references to some of the earlier work on this topic, namely: (1) work on training single classifiers with fairness guarantees over an infinite number of intersectional groups [Kearns et al - ICML 2018](https://proceedings.mlr.press/v80/kearns18a.html); (2) work on decoupled training across many intersetional attributes (see e.g., [Ustun et al at ICML 2019](http://proceedings.mlr.press/v97/ustun19a.html)).", "summary_of_the_paper": "This paper studies the fairness of deep learning models in classification tasks with a large number of intersectional groups. The paper has two primary contributions:\n\n1. The paper includes an empirical study that shows the inherent difficulty of this setting (i.e., due to the lack of labels), and the limitations of existing approaches.\n\n2. The paper presents a new approach to train deep models to settings with many intersectional groups called Knowledge Distillation of Independent Models as Regularization (DIR).", "summary_of_the_review": "This paper studies an important problem. In short, there is little work on intersectional subgroups, even less on deep learning. In this context, I see the empirical study as an opportunity to make a contribution as it can highlight previously unknown issues. \n\nMy recommendation is based on the following issues: (1) key findings are not well-supported by the empirical results (see comment above); (2) the proposed approach DIR is underdeveloped and under-evaluated in the text. I think that both issues could be addressed, but should be subject to another round of peer review.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "-pk82BCzj6a", "reviewer": "Reviewer_JEnE", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Intersectional fairness is a practical and important problem. The authors conduct a comprehensive comparison among existing bias mitigation techniques, which I find valuable. The paper begins reasonably well-written but the clarity deteriorates a bit in Section 3. \n\n1. My main concern is the fairness metrics considered in this paper. \n\nFirst, the authors introduced a notion called worst-case accuracy in Eq. (2) and argued that it was a surrogate metric for equalized odds. However, it is unclear to me why it is an appropriate surrogate. Note that equalized odds measure the *performance disparity* instead of the worst-case accuracy. \n\nSecond, it is unclear to me why the authors computed the surrogate metrics instead of the original measures (e.g., demographic parity) when analyzing the performance of bias mitigating algorithms (see e.g., Fig 2). I understand that training a model with the original fairness measures can be hard. However, once a binary classifier is fixed, verifying demographic parity is just to computing the mean of Bernoulli variables. \n\nThird, the definitions of the fairness metrics in Section 3.1 are puzzling from a mathematical perspective. \n\n(1)  the importance-weighted accuracy in Eq. (1) involves 1/|g| inside the expectation. It is unclear what |g| means. Is it the maximal number of subgroups? If so, then Acc_U(h) -> 0 when |g| -> \\infty, which does not make any sense. \n\n(2) the bias amplification in Eq (3) involves Pr(g|h(x)). This conditional probability distribution is unclear to me. First, is h(x) a random variable? If so, the authors should write it as h(X) to be consistent. Second, if h(x) is a random variable, why does the measure s_y only rely on y?\n\n(3) the intersectional bias in Eq (4) is also unclear. Isn’t it \\max \\min instead of \\max \\argmin?\n\nI highly suggest the authors carefully revise their definitions of the fairness metrics in Section 3.1! \n\n2. The primary results in Section 4 are interesting. I wonder if the authors can provide any interpretation of these results. ", "summary_of_the_paper": "This paper conducts a comprehensive comparison among existing bias reduction methods when there are multiple protected attributes. The authors also propose a knowledge distillation framework for improving fairness on the protected-label-scarce ImageNet dataset.", "summary_of_the_review": "Interesting experimental results but the fairness metrics adopted in this paper need to be clarified ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "-9R9Gp_Y6xY", "reviewer": "Reviewer_zw8E", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "+ The focus of this paper is an important problem: intersectionality in fairness. \n+ The empirical analysis provides insight into how existing methods perform on intersectional groups and highlights fairness challenges that are particularly vexing in the context of intersectionality: runtime complexity and stability. \n\n- The paper would benefit from a discussion on the hyperparameter tuning. Typically this involves a fairness-accuracy tradeoff which would seem to affect the results. e.g, we can always eliminate intersectional bias by returning a constant classifier. \n-The current definition of the sensitive attributes invokes the “pale male” default. For clarity and inclusivity, I would recommend revising to define the attribute in terms of the attribute, not the value: e.g, “Gender” instead of “Male.” Are each of these binary? It may also be more appropriate to term these sensitive attributes, since protected typically evokes a legal connotation. \n- It would be useful to clarify why we should be concerned about fairness with respect to attributes like “straight hair\" or “attractive.” Is this purely illustrative or is there a story the reader should have in mind?\n- Accuracy in table 4 seems to be reported on a 0-100 scale but was earlier reported on a 0-1 scale. Why are the reweighed accuracies in Table 4 so low?\n\n\nMinor:\nDefine Mean Average Precision. \nDefine the error bars in the figures. Are all figures for test sets?\nDefine synsets", "summary_of_the_paper": "\nThis paper provides an empirical investigation of deep learning bias mitigation methods, focusing on two problems: intersectionality and missing sensitive attribute labels. The paper evaluates several bias mitigation approaches on the Celeb dataset and on ImageNet, the latter having few sensitive attribute labels. The main takeaways from their empirical analysis is that the best performing mitigation method depends on the context and mitigation methods that operate without using the sensitive attribute do not perform well in terms of intersectionality. The paper also proposes a new distillation-based approach to resolve the runtime complexity problem that models like Domain Independent exhibit. On ImageNet their method significantly reduces bias amplification but there are no differences in group-reweighed accuracy or intersectional bias.", "summary_of_the_review": "This paper provides an extensive empirical analysis of an important question, how fairness methods perform with regards to intersectionality. The paper would benefit from additional discussion on a few points, like hyperparameter tuning.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The manuscript performs an empirical analysis of existing bias mitigation methods on two large datasets CelebA and ImageNet People Subtree where there are multiple sensitive attributes and some unavailable sensitive attribute labels. The results show that existing methods can mitigate intersectional bias at scale but unlabeled mitigation methods generalize poorly. The manuscript further proposes a knowledge distillation approach which can augment other labeled mitigation approaches.

On the positive aspect, the manuscript studies an important problem: intersectional subgroups on deep learning methods. Reviewers acknowledged that an empirical study on this problem is as an opportunity to make a contribution as it can highlight previously unknown issues.

There are however several major concerns including:
1. Methodological contribution (knowledge distillation) is under-developed, while empirical investigation is interesting but can be further developed;
2. The fairness metrics adopted in this manuscript need to be clarified;
3. A discussion on the hyperparameter tuning, maybe involving a fairness-accuracy tradeoff;
4. The claimed O(1) complexity for the knowledge distillation approach is implausible because it assumes the availability of G group-specific models. This has been clarified in the rebuttal that the claim is only for the inference complexity, and the approach does not improve the training complexity. 

Reviewers also concluded that while the empirical analysis is interesting, the results on CelebA to be of limited use because the sensitive attributes are "purely illustrative." It's not clear that the insights from these illustrative intersectional groups (e.g. big nose & attractive) will hold for groups that are meaningful in a fairness sense.
Decision: Reject