Simplicity bias leads to amplified performance disparities
Authors: Samuel Bell
Levent Sagun
Venue: Submitted to ICLR 2023 (2023)
Keywords: fairness
model bias
dataset bias
bias amplification
simplicity bias

Abstract: The simple idea that not all things are equally difficult has surprising implications when applied in a fairness context.
In this work we explore how "difficulty" is model-specific, such that different models find different parts of a dataset challenging.
When difficulty correlates with group information, we term this difficulty disparity.
Drawing a connection with recent work exploring the inductive bias towards simplicity of SGD-trained models, we show that when such a disparity exists, it is further amplified by commonly-used models.
We quantify this amplification factor across a range of settings aiming towards a fuller understanding of the role of model bias. We also present a challenge to the simplifying assumption that ``fixing'' a dataset is sufficient to ensure unbiased performance.

Under review as a conference paper at ICLR 2022
SIMPLICITY BIAS LEADS TO AMPLIFIED PERFORMANCE DISPARITIES
Anonymous authors Paper under double-blind review
ABSTRACT
The simple idea that not all things are equally difficult has surprising implications when applied in fairness context. In this work we explore how "difficulty" is model-specific, such that different models find different parts lenging. When difficulty correlates with group information, we term this difficulty disparity. Drawing a connection with recent work exploring the inductive bias towards simplicity of SGD-trained models, show that when such disparity exists, it is further amplified by commonly-used models. We quantify this ampli- fication factor across range of settings aiming towards fuller understanding of the role of model bias. We also present a challenge to the simplifying assumption that "fixing" dataset is sufficient ensure unbiased performance.
dataset chal-
INTRODUCTION
Without actually training, understanding what model will find challenging is far from trivial. A certain dataset may be hard for one model but not for another (Wolpert & Macready 1997). For given model, two classes may be easily separable, while for another they may be hard distinguish. Given this, it follows naturally that "difficulty" is function of both data and model, such that we can't properly account for difficulty by analyzing the dataset alone.
In the context of fairness in machine learning, for a given task, a data-model pair may be more difficult for one social group than another, Icading to disparate impact (Barocas & Selbst 2016)) For example, [uolamwini & Gebru's 2018 audit of commercial image recognition systems finds
1.0
Test accuracy
Est. disparity, Easter only Difficult group only
Obs disportty, d
Dificult group
-2
0.5
0.5
Step
Step
(a) Example distribution
(b) Train each group separately
(c) Train both groups together
Figure 1: What is difficulty amplification? (a) Consider binary classification of circles and triangles. Above
(light gray background) we have a simple group which is linearly separable. Below 0 (dark gray) we have a more complex group with non-linear decision boundary. (b) Illustration test accuracy when training on the simple group only (light gray) and the complex group only (dark gray). As expected we obtain better accuracy on the simple group. (c) However, when training on both groups at once, our model exacerbates the difference: the observed accuracy disparity d (height of pink area) exceeds the estimated accuracy disparity from individual group training d (height of green area). When d d, we call this difficulty amplification.

---

Under review as a conference paper at ICLR 2022
they exhibit worse accuracy for darker-skinned women than for any other group. Typically, accu- racy disparity of this kind is attributed to either uudresrepressntation of certain groups or spurious correlations between group information and the variable of interest. In this work, we further show that-even with perfectly balanced data and in the absence correlations between group labels and class labels-trained models can find certain groups harder than others. Crucially, group diffi culty is not always predictable from a dataset audit, providing key evidence for the necessity of complementary post-training model audit.
Having identified model-specific disparities in the post-dataset-audi setting, we turn to the role of the model itself. We show that implicit bias of certain model classes towards simple functions (Arit
et al. 2017; Kalimeris et al., (2019) Rahaman et al. 2019;) Valle-Perez et al.) [2019] [Shah et al.) 2020)
further amplifies disparity: When model finds one group easier than another, its bias towards the easy group leads to greater than expected performance disparity after training (see fig. T. We show that difficulty amplification is highly sensitive model architecture, training time, and parameter count. Seemingly innocuous design decisions, such as whether to use early stopping, can have significant impact on the amount of amplification and consequently the performance disparity.
Contributions
1. We identify difficulty disparity, a pervasive phenomenon that persists in the post-dataset- audit setting: using data with perfect representation and without spurious correlations.
2. We introduce difficulty amplification factor quantify how much model exacerbates difficulty disparity.
3. We empirically evaluate how choices including model architecture, training time, and parameter count impact difficulty amplification.
Paper structure. In $ provide background and related work. We begin in 3 by evaluating variability of dataset diffcllty across models. In $ 3ew formalize difficulty disparity and amplifica- tion and design synthetic task isolate it. In $ 5 we show how factors such model architecture, scale, training time and regularization impact amplification. After real-world example using Dollar Street in $ 6 we discuss the fairness implication of our work before concluding in
BACKGROUND AND RELATED WORK
2. BIASED DATASETS AND BIASED MODELS
Bias in MI. systems arises from many sources. At the most basic level, dataset itself is biased if certain groups are under-represented (Stock & Cisse 2018 Hendricks et al. 2018 Yang et al. 2020) Menon al. (02I). Proposals rectify under-repreesentatio include actively collecting more data for marginalized groups (Dutta et al. 2020), under/oversampling or reweighting (Byrd & Lipton
2019 Sagawa al. 2020 Idrissi et al. 2022 Arjovsky al. 2022) during training. and optimizing
for worst-group (as opposed to average) accuracy (Sagawa et al.
2019). Other recent work has 2022).
suggested fine-tuning on an explicitly balanced set (Kirichenko et al.
Alternatively, datasets can reinforce harmful associations (Goyal et al. 2022b). both due to sampling
error and by inadvertently capturing an undesirable association that is present in society. Bucketed as "spurious correlations" (Muthukumar et al.) 2018 Wang et al. 2019) Sagawa et al.) (020) or "shortcuts" (Geirhos et al., 2020). a large body of fairness work seeks to train models that learn some true function invariant to the spuriously correlated feature. Both under-repreeenntation and spurious correlations dominate the faimess literature landscape, though in both cases the onus is squarely the data.
2.2 BEYOND SPURIOUS CORRELATIONS
There is increasing focus the model itself, independent of the role of data (Hooker) 202 l). of which bias amplification is perfect example (Zhao et al. 2017 Wang & Russakovsky 2021 Here, a small correlation in the training set is amplified into larger correlation at test time. Models don't just replicate the bias in the data, but exacerbate it. In empirical experiments evaluating bias amplification, Hall al. (002) suggest that if group membership is easier to identify than class
2

---

Under review as a conference paper at ICLR 2022
membership, models prefer to the spurious correlation. Hall et al. report no bias amplification
in the case where there is no spurious correlation present in the data: this is expected given the definition of amplification involves a multiplication of existing data bias.
Other subtle biases have been identified in the balanced data setting. Leino et al. (2119) show that models trained with SGD overly rely upon moderately mpuriously-correlate
features if they sufficiently numerous relative to the size of training set. Khani & Liang (2020) find that adding feature noise equally across groups induces disparity, a fact that can also be attributed to the relative difficulty of group information versus the desired target. KKaan & Liang (202T) find removing spurious features can disproportionately lower performance on certain groups, and argue (as do we) that a balanced dataset is not sufficient guard against biased performance. MMannlli et al. (2022) use teacher-student networks to show subtle properties such as differences in group distance from the overall mean and differences in group variance are sufficient to induce biased outcomes in the absence of spurious correlations.
2.3 INDUCTIVE BIAS TOWARDS SIMPLICITY
Numerous recent works have identified the tendency for CGD-trained models to prioritize simple data points during training, resulting in simple functions being learned before more complex ones (Arpit et al.) 2017 Kalimeris et al., 2019, Valle-Perez et al. 2019). Jo & Bengio (2017) show that convolutional networks are overly dependent on surface-level statistical properties of images, such that applying Fourier filter to the training set is sufficient to radically degrade test perfor- mance. Rahaman et al. (2019) also show a bias towards low-frequency functions, learning these simpler functions before more complex, higher-frequency examples. Though often framed pos- itive, allowing neural networks to learn functions that generalize well by applying Occam's 's razor, Shah et al. (2020) cites this simplicity bias potentially harmful, root causing both vulnerabil- ity to adversarial attacks and over-reliance on spurious correlations. [agaev et al. (2021) reserve as much skepticism for overly simple solutions as for overly complicated, arguing that excessively simple solutions are likely to rely on potentially harmful "shortcuts" (Geirhos et al. (2020). Sagawa al. (2020) find that increasing model size yet further pushes the model to rely upon spuriously- correlated features where they carry more signal than the intended features. In our work, we show how this inductive bias toward simplicity manifests as disparate outcomes when there is perceived complexity difference between groups.
3 PRELUDE: DATA DIFFICULTY IS MODEL-SPECIFIC
This work is predicated on simple, perhaps even obvious idea: that certain facets of dataset may be more or less difficult to a given model. Whether individual data points, classes, or social groups, identifying what will be difficult to model isn't isn't always apparent ahead of time, nor need it align with human intuition. As an illustration, we begin by investigating difficulty difference between classes, though in following sections we will transition a focus on social groups
Measuring class difficulty. We measure difficulty using cross-validated test accuracy over the full dataset, i.e. after merging the source train and test sets (see appendix B). We train a classifier on coarse-grained CIF (Krizhevsky et al., 2009), after which we evaluate binary classification accuracy for each pair of labels by masking irrelevant outputs before the softmax layer. Figure 2 shows how ResNet-18 18 (He et al., (016) yields variable test accuracy between classes. For example, this model achieves near perfect test accuracy on foweerrslaatti manmals, and substantially lower performance on non-insect mwertebrates/inseee specific model. Were we to perform this evaluation on a different model, our results would differ.
Crucially, this view of difficulty is just that of this
Model specificity. To confirm this, we repeat the experiment using the following models: an SVM with an RBF kernel; a 3-layer and a 5-layer fully-connected network: LeNet, a simple CNN (Lecun) et al. 1998); AlexNet, a more complex CNN (Krizhevsky et al. 2012) and fully-connected layer
over pre- trained representations extracted from Res trained using SimCLR (Chen et al.
2020
on ImageNet- 1K (Russakovsky et al. (2015), and those of RegNet- 128Gf (Radosavovic et al. 2020
trained with SwAV (Caron et al. 2020) on I billion public images from Instagram (Goyal et al. 2022a) (see appendix C for a full description).
3

---

Under review as a conference paper at ICLR 2022
Pah
Honos
Hiwws
Par & veg.
0:93 1.0
dewuo
lousenald hami...
lousehsid
poects
insech
leke
Outsoer thlgg
Outtoor thinge
Ormatattionn
Omnihebtsono
man
stre
buste5))
People
Feople
Small nomnuh
Small mac
Thees
whickle
Veniclo les
Velk delices
(b)
Food
Agpas
Food
(a)
(c)
Figure 2: (a) ResNet-18 finds certain classes of CIFAR-100 more difficult than others, according to binary test accuracy. (b) Rank ordering of difficulty varies across models, measured here by Kendall's т. (c) ID PLS projection of binary accuracies onto cosine distance between class mean vectors. Red cells pairs where increasing inter-class distance decreases accuracy- there no clear way to identify what model will find difficult without training.
accuracy-demonstrace demonstrate
Figure 2b shows how the rank order correlation of pairwise difficulties varies between models, as measured by Kendall's т. Specifically, between the 3-layer FC model and LeNet, there is high (but not perfect) т, indicating that broadly what LeNet finds difficult too does the FC network. In contrast, between the RBF SVM and a linear layer on RegNet- much lower (though aligns with recent work (Hacohen et al. 2020) showing that the difficulty of individual data points shared across random initializations the same model architecture, but difficulty only partially consistent across architectures.
representations, there is zero) T, indicating that the pairwise ordering varies considerably. This
Data difficulty is not model difficulty. We test this using the partial least squares (PLS) analysis shown in fig. p. PLS attempts to find low-dimensional projections of both the input and output variables such that their covariance maximized. We apply PL. determine how much a data- only difficulty measure can explain a model+data measure, where the data-only measure is the rank cosine distance between the input data class means, and the model+data measure is the rank test ac- curacy (see appendix A). If model difficulty was purely function of data difficulty, we would expect PLS to find a well fitting linear regression model. Instead, PLS finds a near-zero (R² = 0.058). From the ID projection, increasing inter-class distance tends to increases binary accuracy, though for many class pairs (in red, outdoor scenesloutdoor things) the opposite is true.
that for some classes (in blue, e.g. carnivoreslfood containers),
Summary. There is no clear difficulty ordering that is consistent between all model classes. What a model finds difficult is solely function of the data.
4 NEURAL NETWORKS PRIORITIZE "EASY"
Having established that difficulty is a function of both model and data, we now measure how difficuly affects performance disparity and quantify the effect of the simplicity bias.
4.1 DEFINITIONS
Let acc(X, M) be the cross- vollddeddd test accuracy on the classification dataset (X, y) of a model M, and Mx,y y be model trained (with cross- ya) and Vz) denote corresponding slices of the dataset, such that a model trained only on group is Mx Mx..N..
cross-validaiee validation) on (X,y). Given two groups and B, let
4

---

Under review as a conference paper at ICLR 2022
2.0
0.4
C.3
Test accuracy
Observed diopa
0.2
0.6
G.a
0.1
10,000
10,000"
12
(a) Synthetic task
(b) Train accuracy
(c) Test accuracy
(d) Observed disparity
Figure 3: Accuracy and disparity for ResNet-18. (a) Binary classification (orange blue) comprising two groups. Easy group (light gray) is high-accuracy pair flowerslaquatic mammal. Difficult group (dark gray) is low-accuracy pair medium manmalsicarnivores (b) Train accuracy for complex group is learned slower but both groups reach perfect train accuracy. (c) However, test accuracy for complex group is persistently lower. Dashed lines are binary accuracy from single-group training. (d) Observed disparity d peaks early in training but large gap persists after training. Red dashed line estimated disparity d: observed disparity above this line indicates amplification. Shaded area is standard error over 10 runs each with different train/test split.
Estimated difficulty disparity. First, we define the estimated difficulty disparity d as the difference in accuracy between model trained and evaluated each group in isolation,
d (accoXa.eea Mx....
3. Mx
Observed difficulty disparity. Second, the observed difficulty disparity is the difference in aceu- racy between groups on a model trained on both groups,
d = Jaco(Xar Yas Mx,x))))) acc(Xx, Mxx)]
Difficulty amplification. Finally, if the model trained on both groups exhibits worse disparity than when trained in isolation, d d, we say that the model exhibits difficulty amplification. Over many model runs, groups, or samples from the dataset we can define an amplification factor d/d.
In practice, calculating amplification is two-stage process. First, we train N randomly-inittalized models on each group in isolation, and compute the average cross-validated test accuracy. Between each pair of groups, we calculate the estimated difficulty disparity d. Second, we train new set of N models on the full dataset including all groups, and compute average test accuracy broken out by group. For each group pair, we calculate the observed difficulty disparity d.
4.2 SIMULATING DIFFICULTY DISPARITY WITH CIFAR-100
To measure difficulty disparity in controlled setting, design task based CIFAR-100 100 that is group-balanced and absent correlations between group labels and target labels. We extract the binary test accuracies for each pair of coarse classes and treat their pairwise differences as estimated difficulty disparity d. We let group be the class pair V/ vf with the highest accuracy and 3 be the lowest v6 vf¹ To simulate binary classification task with two differently-difficult groups, we stitch these pairs together into single binary task, where
{ue . 8 J.y1 {vf 8 See fig. 3a. Finally, we train N models this task and calculate observed difficulty disparity d.
4.3 RESULTS
In fig. 3 we see the training accuracy of the simple group improves much more rapidly than the complex 3, though both groups reach perfect train accuracy eventually. In contrast, the test aceu- racies in fig. 3 remain notably different convergence, with the model displaying lower accuracy on the complex group. This gap is the observed accuracy disparity shown in Figure B1. Here, we observe that observed disparity peaks after just few steps, before slight decline to plateau.
5

---

Under review as a conference paper at ICLR 2022
1.8
1.6
1.0
0.0
Amplificatio
0.8
10.1
FC 3-bber
FC layer
Resflet34
Reshet50
Resflet1
0.4
Lefiet
Alextee
-0.2
ResNett1
SVM
Estimated accuracy disparity
.0.2 -0.1
0.0
0.1
0.2
Model
(a) Observed vs. estimated
(b) Amplification factor
Figure 4: Observed difficulty disparity as a function of estimated difficulty disparity. (a) ResNet- 18 amplifies difficulty disparity but LeNet atfenuates. Points are sampled tasks with varying estimated difficulty disparities. Solid line is linear regression. Dashed grey line is amplification factor k = 1. (b) Model architecture impacts difficulty amplification factor. See fig. S4 for R and fig. S3 for raw test accuracies.
Evidencing our difficulty amplification claim, observed disparity remains higher than estimated dis- parity we would have expected from separate training (the red dashed line). Replications of this experiment on both Fashion MNIST (Xiao et al.] (017)) and EMNIST Letters (Cohen et al., 2017) show similar results (see fig. and fig. (S)..
Summary. This simple experiment shows that models trained across groups with different difficulty do prioritize the simpler group, leading an outsized observed disparity, primarily driven by under- performance on the more difficult group.
AMPLIFICATION FACTOR VARIES ACROSS MODELS
To compute an amplification factor, we repeat the above experiment using different pairs of classes with different estimated disparity, and compute their observed disparity after combined training.
We retrain on 30 sampled pairs of label pairs, recomputing both estimated and observed difficulty disparity, and apply OLS linear regression
estimate the amplification factor (see appendix E for details). This method can easily be applied to any dataset annotated with group information, by replacing the sampling of pairs of classes with the sampling of different group combinations. We compute k for each model listed in $ 3
Furthermore, we evaluate the effect of model scale on amplification factor by varying the width of ResNet-18. 18; evaluate various settings of weight decay; and evaluate the role of carly stopping by computing amplification through training. Our choice investigate these three parameters motivated by their expected effect on simplicity bias. Following Kalimeris et al. (20T9) we expect models to exhibit a stronger preference for simplicity earlier training, which would often mate- rialize when using early stopping. Weight decay is a common regularization technique intended to limit overfitting by penalizing excessively complex functions, and the role of width over-reliance spurious correlations is reported by Sagawa al. 2220)).
For complementary test of the bias against complexity, we also try to push the model to choose a more complex solution. We enforce Lipschitz constraint by applying penalty the norm of the gradients, a technique commonly used to stabilize discriminator training in GANs (Gulrajani et al. 2017). We add the following penalty term loss function L.
= X(I <x)()) - C)²
where (x) is the gradient of the network's 's outputs with respect its inputs, the penalty coeffi- cient 10, and C determines the Lipschitz constraint: a low C pushes the model towards simpler functions, and high C' towards complex.

---

Under review as a conference paper at ICLR 2022
1.4
1.4-
1.4
1.4
Ampiniicaionn factor
Amplllicatton factor
1.2
Ampiriicatoon factor
1.2
Amplification fector
1:2
8.0
1.0
8.8.
0.0
0.8
0.0
0.8
0.4
0.4
0.
9.
8
100
10,000
0.0
10.6
1o-4
16-3 1e:2
0.001 0.016.03 0.1 0.3
Width
Weight decay
Gradient penalty
(a) Width
(b) Step
(c) Weight decay
(d) Gradient penalty
Figure 5: Design decisions influence amplification factor. All panels are a ResNet-18 on sam- pled synthetic tasks based on CIFAR-100 (see fig. 3p) (a) Network width has negative effect on amplification, reducing it to near 1. (b) & peaks early in training before plateauing just above 1. (c) Weight decay has no effect amplification. (d) Applying gradient penalty to bias the model toward a C-Lipschitz functions works. See fig. $5 for R
5.1 RESULTS
Model architecture. In fig. 4h, we illustrate the difference amplification factor between two models, LeNet and ResNet-18. We find that the ResNet-18 amplifies disparity by factor of k = 1.19 + 0.12. In contrast, LeNet diminishes disparity (A = 0.84 = 0.04), resulting in an observed disparity lower than expected. Thus, from this simple example we show that model choice influences difficulty amplification. Across the full suite of models (fig. { we again see significant variation in amplification factors across the different models, with the certain models (e.g. LeNet) attenuating and the others (e.g. ResNet) amplifying. However, the simpler models all exhibit poor test accuracy averaged over the entire dataset (see fig. (3), offering a candidate explanation for the lack of amplification. These models may be too simple to learn the dataset at all, resulting in equally poor performance across all groups
Width. However, within specific architecture, increasing width seems to reduce amplification. Figure 55p shows the amplification factor for ResNet- 18 rapidly decreasing to almost (no amplifi- cation) network width increases. These results align with those of (Sgawa et al. (2020), who port that while overparameterivation typically increases reliance on spurious correlations and harms worst-group error, this effect is reversed groups become more balanced, such that increasing parameter count becomes helpful (Sagawa et al. 2020 e.g. fig. 6).
Early stopping. As training proceeds (fig. 5p) k increases to a peak around 1.2 early in training, before decreasing to a plateau a little over I. This highlights the important role of early stopping in amplifying disparity, particularly in light of prior work arguing that models learn more simple functions earlier in training (Kalimeris et al. 2019).
Weight decay. Figure 5 shows next to no effect of scaling the weight decay parameter. This is surprising result, as our expectation is that applying stronger weight decay would further bias the model towards the simpler group, increasing amplification. One possible explanation is the sensi- tivity of the & penalty to choices of model and dataset, as reported by Sagawa et al. (2019). While the purpose of our work to introduce the notion of difficulty disparity and difficulty amplifica- tion, further research is needed to confirm the role of weight decay across various settings, and its interaction with other implicit regularization schemes.
Gradient penalty. In contrast, applying penalty to the norm of the gradients, rather than the parameters, sufficient to lower below for values of considered here. This suggests that applying gradient penalty to balance out the implicit bias towards simplicity may be helpful strategy in combating difficulty disparity.
Summary. High performing models- those optimized for average test accuracy consistently dis- play difficulty amplification. This phenomenon is exacerbated by early stopping, but may be reduced using a gradient penalty.
7

---

Under review as a conference paper at ICLR 2022
vs. 4th quartie
0.30
DiK. 4th quartic
0.10
0.05
0.05
0.05
2.33
3.4
0.00
0.00
0.00
003
0.2
-0.05
-0.05
0.00
12
100
1,000
10,000
12
10
100
1.000
30,000
12
10
100
1,000
10.00
32
100
1.000 10.000
Step
(a) Test accuracy
(b) 1st quartile
(c) 2nd quartile
(d) 3rd quartile
Figure 6: (a) Mcan test accuracy for each income quartile on balanced Dollar Street. Dashed lines are single-group group training; solid trained on entire dataset. (b-d) Performance disparity between each income quartile compared against the top income quartile (also the best performing). Dashed lines indicate estimated disparity d: solid indicate observed disparity d. This particular model amplifies the disparity between the highest and lowest income quartiles (b), but reduces disparity between middle and top quartiles (c-d). Error bars omitted for clarity.
DIFFICULTY AMPLIFICATION HAS WORLD IMPACT
We now evaluate difficulty disparity and amplification in a real-world case study. Dollar Stree is dataset of geographically-dieer images spanning a broad range of household incomes. We use the labels associated with each image in a 138-class object classification task, where group information is household income quartile. We explicitly rebalance the dataset via subsampling to ensure each group is has the same number of data points. We evaluate models trained on each income quartile independently, and models trained on all quartiles together. We train single-layer FC networks on representations extracted from ResNet-18 pretrained on ImageNet-IK. See appendix D for details.
Figure 6p shows the test accuracies of the models trained on the whole dataset (solid lines), and the models trained on each income quartile separately (dashed). In fg-Bb-d, we compare the perfor- mance of the quartile upon which the models perform best, the highest income quartile, against the other three quartiles. In fig. (b, we see that the gap between the highest and lowest income quar- tiles increases through training. Comparing estimated disparity (d:; dashed line) with observed (d; solid) reveals evidence of difficulty amplification between the 1st and 4th quartile. In fig, G d,, again observe that performance disparity between the middle and highest quartiles increases through training, though here we see slight disparity attenuation.
Thus, we present consistent evidence of performance disparity. such that the models continuously perform best on the highest income quartile. However, it is mixed picture regarding difficulty amplification, with amplified disparity between the lowest and highest income quartiles, but slightly reduced elsewhere. There are number of key takeaways. First, even using a dataset with a focus on diverse data collection, we still observe bias of several percentage points towards the highest- income group. Second, explicitly rebalancing the dataset to ensure equal group sizes is not sufficient to remove this bias, nor can it be attributed to spurious correlations due to our cross-validation methodology. Third, difficulty amplification present here, though not consistently throughout the dataset. In this case, by choosing to deploy this specific model we would magnify biases against low income houscholds, less so against middle income households, all the while giving preferential treatment to high income.
Summary. Example of difficully disparity social context: this model amplifies income disparity.
7 DISCUSSION
Auditing for bias. At its heart, our work presents yet another way which models exhibit bias and performance disparities across demographics. A frequent refrain in the ML community is that such disparities are the fault of the data, rather than algorithmic bias (Hooker) 202I). Indeed, a series of thorough audits have revealed that popular datasets under-represent minoritized groups (Shankar et al.) 2017 Stock & Cisse
2018 Buolamwini & Gebru 2018 de Vries et al, 2019.
Dulhanty& & Wong 2019 Wilson et al. 2019 reify harmful associations and perpetuate stereotypes
https:/ gapmi nder. ar street
8

---

Under review as a conference paper at ICLR 2022
(Bolukbasi et al.
2016: van Miltenburg 2016: Garg et al. 2018 Dixon al. 2018 Birhane &
Prabbu 2021 | Raji & Fried 2021): and operationalize concepts such as gender and race in a way and culturally specific concepts (Keyes) 2018 Paullada et al. 2021: Denton et al., 2021 Raji et al. 2021). Fixing these issues at the level of the data may not even be possible, for example it is often undesirable collect the demographic information needed to ensure balance in the first place (Veale & Binns 2017; Andrus et al. 2021
that applies veneer of "objectivity" to vocially-constructtt
Hooker 202T). That being said, acknowledging issues with our use of data does not absolve all that comes after, as exemplified by bias amplification (Zhao et al. 2017; Wang & Russakovsky 2021 [Hall et al. 2022). Here, in support of the role of post-training audit, we choose the setting where the data is "perfect", in that it is both explicitly balanced, and groups and labels are decorrelated. The variability of both difficulty disparity and amplification from model to model a strong reminder that both those who develop and deploy ML systems must take action to ensure their fairness.
Fairness definitions. By discussing issues of bias and disparity, we engage in a broader discussion about fairness in ML systems. Here, we follow others in focusing on the performance gap between
groups (Dwork et al. 2012; Hardt et al. 2016 Woodworth et al. 2017: Agarwal et al. 2018 Khani
jet al. 2019; Goyal al.] 2022b). though an alternative approach would be to focus explicitly on worst- groop performance instead (Mohri et al., 2019;; Sagawa et al., 2019 Zhang et al.) 2020). Oth-
ers rely upon counterfactual fairness (Kusner et al. 2017 Kilbertus al. 2017, Loftus al. 2018),
according to which a "fair" system reaches the same decision on two otherwise identical individuals belonging to different protected groups, though this draws increasing criticism due to its require- ment that concepts such as race or gender both be well-defined (Benthall & Haynes (019) and can be changed while only minimally impacting other attributes (Hu & et al. 2020). Our aim in this work is not to use metric by which to deem systems fair or unfair, but to highlight the possible role of model bias in this case, due to preference for simplicity that will have subsequent fairness impacts. Even assuming satisfactory yardstick by which to measure, and a model accordingly deemed fair, fairness of course not necessarily implied. When situated within broader societal context, any model can be put to harmful use, and it is common pitfall of the ML community to narrowly situate our work inside neatly- abstractions (Selbst et al. 2019).
2020 Hanna
Spurious correlations. Similarly, a key ambition of our work is to push research into sources of bias outside of the typical characterizations: spurious correlations and under-representation. Indeed we suggest that reducing the study of model bias to these two dimensions is an instance of excessive abstraction through formalization (Selbst al. 2019). By focusing on the settings where these issues are resolved, we hope that future research can take a more nuanced look at the biased behavior of models where not obviously the result of data issue. A plausible outcome of this kind of research could be that in certain situations ML might not be appropriate at all, if we can't 't guarantee that the system won't 't develop unpredictable and hidden biases.
CONCLUSION
We have argued that what model finds difficult is not simply a function of the data, but a function of both model and dataset. This is particularly a problem in a fairness context if difficulty is correlated with group information. We have found that certain models further amplify difficulty disparity, resulting in observed difficulty disparity over and above estimated difficulty disparity, as result of the bias of certain models towards easy examples. Difficulty amplification varies with model architecture, model scale, training time and regularization strategy, and seemingly innocuous design impact. Finally, we have shown how difficulty
decisions can have a substantial and counter-intuine disparity and amplification take place in the Dollar Street setting, where our simple model is biased against images in the low income quartile. Taken together, our results highlight the key role of the model-above and beyond the dataset-in creating group disparities.
Limitations. Our primary aim is to further highlight the key role of the model in accuracy disparity. We do however assume access to group information for audit purposes, which may not be available in many realistic scenarios, nor desirable to obtain. We intentionally choose to explore the balanced dataset setting, though separating difficulty disparity from other sources of bias may be difficult in practice. Future work may seek to explore a broader array of model families, and a more detailed investigation of the role of different regularization techniques.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Alckh Agarwal, Alina Beygelzimer, Miroslav Dudik, John Langford, and Hanna Wallach. A re- ductions approach fair classification. Proceedings of the 35th International Conference on Machine Learning, volume 80, pp. 60-69, 2018.
McKane Andrus, Elena Spitzer, Jeffrey Brown, and Alice Xiang. What can't measure, can't understand: Challenges to demographic data procurement in the pursuit of fairness. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, FAccT '21, pp. 249-260, 2021.
doi: 0411445/344218884445558
Eric Arazo, Diego Ortego, Paul Albert, Noel O'Connor, and Kevin Meguinness. Unsupervised label noise modeling and loss correction, In Proceedings of the 36th International Conference on Machine Learning, volume 97, pp. 312-321, 2019.
Martin Arjovsky, Kamalika Chaudhuri, and David Lopez-Par Paz. Throwing away data improves worst- class error in imbalanced classification, 2022. arXiv:2205.1167.
Devansh Arpit, Stanislaw Jastrzębski, Nicolas Ballas, David Krueger, Emmanuel Bengio. Maxin- der S. Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, and Simon Lacoste-Julien. A closer look at memorization deep networks. In Proceedings of the 34th International Conference on Machine Learning, volume 70, pp. 233-242, 2017.
Solon Barocas and Andrew D Selbst. Big data's disparate impact. Calif. L. Rev., 104:671, 2016.
Sebastian Benthall and Bruce D. Haynes. Racial categories in machine learning. In Proceedings of the Conference on Fairness, Accountability, and Transparency, FAT* '19, pp. 289-298, 2019. doi: 10.114532883560.2287555
Abeba Birhane and Vinay Uday Prabhu. Large image datasets: A pyrrhic win for computer vision? In 2021 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 1536-1546, 2021. doi: 10. S0G9W/AAC46630.222..000
Tolga Bolukbasi, Kai-Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam T Kalai. Man is to computer programmer as woman is to homemaker? debiasing word embeddings. In Advances in Neural Information Processing Systems, volume 29, 2016.
Joy Buolamwini and Timnit Gebru. Gender shades: Intersectional accuracy disparities commer- cial gender classification. In Proceedings of the 1st Conference Fairness, Accountability and Transparency, volume 81, pp. 77-91, 2018.
Jonathon Byrd and Zachary Lipton. What is the effect of importance weighting in deep learning? In Proceedings of the 36th International Conference on Machine Learning, volume 97, pp. 872-881, 2019.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Un- supervised learning of visual features by contrasting cluster assignments. In Advances in Neural Information Processing Systems, volume 33, pp. 9912-9924, 2020.
Haw-Shiuan Chang, Erik Learned- Miller, and Andrew McCallum. Active bias: Training more accu- rate neural networks by emphasizing high variance samples. In Advances in Neural Information Processing Systems, volume 30, 2017.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. Proceedings of the 37th International Conference on Machine Learning, volume 119, pp. 1597-1607, 2020.
Gregory Cohen, Saced Afshar, Jonathan Tapson, and André van Schaik. Emnist: Extending mnist to handwritten letters. In 2017 International Joint Conference on Neural Networks (LJCNN), pp. 2921-2926, 2017, doi: 10. 20.109//UCNN.201..796222.
Nikolay Dagaev, Brett D. Roads, Xiaoliang Luo, Daniel N. Barry, Kaustubh R. Patil, and Bradley C. Love. A too-good-to-be-te prior reduce shorteut reliance, 2021. arXiv:2102.06400
10

---

Under review as a conference paper at ICLR 2022
Terrance de Vries, Ishan Misra, Changhan Wang, and Laurens van der Maaten. Does object recog- nition work for everyone? In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2019.
Emily Denton, Alex Hanna, Razvan Amironesei, Andrew Smart, and Hilary Nicole. On the ge- nealogy of machine learning datasets: A critical history of imagenet. Big Data & Society, 8(2): 00539517211035999 2021. doi: 10. 10.117/2053951721103333
Lucas Dixon, John Li, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. Measuring and miti- gating unintended bias in text classification. In Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Sociery, AIES '18, pp- 67-73, 2018. doi: 10.114553278721.52720
Chris Dulhanty and Alexander Wong. Auditing imagenet: Towards a model-driven framework for annotating demographic attributes of large-scale image datasets, 2019. arXiv:1905.01347
Sanghamitra Dutta, Dennis Wei, Hazar Yueksel, Pin-Y Yu Chen, Sijia Liu, and Kush Varshney. Is there a trade-off between fairness and accuracy? perspective using mismatched hypothesis testing. In Proceedings of the 37th International Conference Machine Learning, volume 119, pp. 2803-2813, 2020.
Cynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Richard Zemel. Fairness through awareness. In Proceedings of the 3rd Innovations in Theoretical Computer Science Con- ference, ITCS 12, pp. 214-226, 2012. doi: 10.1 444200990236229025555
Nikhil Garg, Londa Schiebinger, Dan Jurafsky, and James Zou. Word embeddings quantify 100 years of gender and ethnic stereotypes. Proceedings of the National Academy of Sciences, 115 (16):F3333-E3344 2018. doi: 10. 10.1073/pnas. 1720347115.
Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. Shortcut learning in deep neural networks. Nature Machine Intelligence,
2020.
Priya Goyal, Quentin Duval, Jeremy Reizenstein, Matthew Leavitt, Min Xu, Benjamin Lefaudeux, Mannat Singh, Vinicius Reis, Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Ishan Misra.
2021.
Priya Goyal, Quentin Duval, Isaac Seessel, Mathilde Caron, Ishan Misra, Levent Sagun, Armand Joulin, and Piotr Bojanowski. Vision models are more robust and fair when pretrained on uncu- rated images without supervision. 2022a. arXiv:2202.08366
Priya Goyal, Adriana Romero Soriano, Caner Hazirbas, Levent Sagun, and Nicolas Usunier. Fair- ness indicators for systematic assessments of visual feature extractors, 2022b. arXiv:2202.06003.
Ishaan Guirajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville. Im- proved training of wasserstein gans. In Advances in Neural Information Processing Systems, volume 30, 2017.
Guy Hacohen, Leshem Choshen, and Daphna Weinshall. Let's agree to agree: Neural networks share classification order on real datasets. In Proceedings of the 37th International Conference on Machine Learning. volume 119, pp. 3950-3960, 2020.
Melissa Hall, Laurens van der Maaten, Laura Gustafson, and Aaron Adcock. A systematic study of bias amplification, 2022, arXiv:2201.
Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. In Advances in Neural Information Processing Systems, volume 31, 2018
Alex Hanna, Emily Denton, Andrew Smart, and Jamila Smith-Loud. Towards a critical race method- ology in algorithmic fairness. In Proceedings of the 2020 Conference Fairness, Accountability, and Transparency, FAT '20, pp. 501-512, 2020. doi: 10. /4114/335119953332232
Moritz Hardt, Eric Price, Erie Price, and Nati Srebro. Equality of opportunity in supervised learning. In Advances in Neural Information Processing Systems, volume 29, 2016.
11

---

Under review as a conference paper at ICLR 2022
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
Lisa Anne Hendricks, Kaylee Burns, Kate Saenko, Trevor Darrell, and Anna Rohrbach. Women also snowboard: Overcoming bias in captioning models. In Proceedings of the European Conference on Computer Vision (ECCV), 2018.
Sara Hooker. Moving beyond "algorithmic bias is a data problem".. Patterns, 2(4):100241, 2021.
doi: https://doi.org/ 10. 60j 20tter.2021.10024:1
Lily Hu and Issa Kohler Hausmann. What's sex got to do with machine learning? In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency, FAT* '20, pp. 513, 2020. ISBN 978 14110669367. doi: 10. 00.115/335199553356644
Badr Youbi Idrissi, Martin Arjovsky. Mohammad Pezeshki, and David Lopez-Paz. Simple data balancing achieves competitive worst-grout postt-rrop-accuuracy In Proceedings of the First Conference on Causal Learning and Reasoning, volume 177, pp. 336-351 2022.
Jason Jo and Yoshua Bengio. Measuring the tendency of cnns to learn surface statistical regularities, 2017. 1rXiv1171..1155
Dimitris Kalimeris, Gal Kaplun, Preetum Nakkiran, Benjamin Edelman, Tristan Yang, Boaz Barak, and Haofeng Zhang, Sgd on neural networks learns functions of increasing complexity. In Ad- vances Neural Information Processing Systems, volume 32, 2019.
Os Keyes. The misgendering machines: Trans/hci implications of automatic gender recognition. Proc. ACM HHam.-Comppe Interact., 2(CSCW), 2018. doi: 10.1 10.1145/3274357.
Fereshte Khani and Percy Liang. Feature noise induces loss discrepancy across groups. Proceed- ings of the 37th International Conference Machine Learning, volume 119, pp. 5209-5219, 2020.
Fereshte Khani and Percy Liang. Removing spurious features can hurt accuracy and affect groups disproportionately. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, FAccT '21, pp. 196 296,205,400 2021. doi: 10. 04.1445/3441188.445588
Fereshte Khani, Aditi Raghunathan, and Percy Liang. Maximum weighted loss discrepancy, 2019.
106.05190.
Niki Kilbertus, Mateo Rojas Carulla, Giambattista Parascandolo, Moritz Hardt, Dominik Janzing, and Bernhard Schölkopf. Avoiding discrimination through causal reasoning. In Advances in Neural Information Processing Systems, volume 30, 2017.
Polina Kirichenko, Pavel Izmailov, and Andrew Gordon Wilson. Last layer re-training sufficient for robustness to spurious correlations, 2022. arXiv:2204.02937.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep volutional neural networks. In Advances in Neural Information Processing Systems, volume 25, 2012.
Matt Kusner, Joshua Loftus, Chris Russell, and Ricardo Silva. Counterfactual fairness. In Advances in Neural Information Processing Systems, volume 30, 2017.
Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recog- nition. Proceedings of the IEEE, 86(1 (6(1)):277-2324 1998. doi: 10. 109/5.76777
Klas Leino, Matt Fredrikson, Emily Black, Shayak Sen, and Anupam Datta. Feature-wise bias amplification. In International Conference on Learning Representations, 2019. URL hetps: 7/openrevien
d=$1 ecm2C9K7
12

---

Under review as a conference paper at ICLR 2022
Joshua R. Loftus, Chris Russell, Matt J. Kusner, and Ricardo Silva. Causal reasoning for algorithmic fairness, 2018. arXiv:1805.0585.
Stefano Sarao Mannelli, Federica Gerace, Negar Rostamzadeh, and Luca Saglietti. Inducing bias is simpler than you think, 2022. arXiv:2205.1555 15935.
Aditya Krishna Menon, Ankit Singh Rawat, and Sanjiv Kumar. Overparameterisation and worst- case generalisation: friend or foe? In International Conference on Learning Representations, 2021.
net http://wwwweeeeeoooorooeee.a.e-. "
Mehryar Mohri, Gary Sivek, and Ananda Theertha Suresh. Agnostic federated learning. In Pro- ceedings of the 36th International Conference on Machine Learning, volume 97, pp. 4615-4625, 2019.
Vidya Muthukumar, Tejaswini Pedapati, Nalini Ratha, Prasanna Sattigeri, Chai Wah Wu, Brian Kingsbury, Abhishek Kumar, Samuel Thomas, Aleksandra Mojsilovic, and Kush R. Varshney. Understanding unequal gender classification accuracy from face images, 2018. arXiv:1812.00099.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems 32, pp. 8024-8035. 2019.
Amandalynne Paullada, Inioluwa Deborah Raji, Emily M. Bender, Emily Denton, and Alex Hanna. Data and its (dis)contents: A survey of dataset development and use in machine learning research. Patterns, 1111:000336, 2021. doi: https://doi.or orr////ooooo////// 10. 101 patp:rrtrrr..21111111 100336..
F. Pedregosa. G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Pretten- hofer, R. Weiss, V. Dubourg, Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825 2830, 2011.
Ilija Radosavovic, Raj Prateek Kosaraju, Ross Girshick, Kaiming He, and Piotr Dollar. Designing network design spaces. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred Hamprecht, Yoshua Bengio, and Aaron Courville. On the spectral bias of neural networks. In Proceedings of the 36th International Conference on Machine Learning, volume 97, pp. 5301-5310, 2019.
Inioluwa Deborah Raji and Genevieve Fried. About face: A survey of facial recognition evaluation, 2021. arXiv:2102.00813.
Inioluwa Deborah Raji, Emily Denton, Emily M. Bender, Alex Hanna, and Amandalynne Paullada. AI and the everything in the whole wide world benchmark. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021. https:
//operzeview.o wwt/ 1=xp-dddddoxxxxA-
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV, (15(3):211-253) 2015. doi: 10. 100//1 1263-015-0816-yy
Shiori Sagawa, Pang Wei Koh, Tatsunori B. Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generaliza- tion, 2019. arXiv: 1.007731...
Shiori Sagawa, Aditi Raghunathan, Pang Wei Koh, and Percy Liang. An investigation of why overparameterization exacerbates spurious correlations. In Proceedings of the 37th International Conference on Machine Learning, volume 119, pp. 8346-8356, 2020.
13

---

Under review as a conference paper at ICLR 2022
Skipper Seabold and Josef Perktold. statsmodels: Econometric and statistical modeling with python. In 9th Python in Science Conference, 2010.
Andrew D. Selbst, Danah Boyd, Sorelle A. Friedler, Suresh Venkatasubramanian, and Janet Vertesi. Fairness and abstraction in sociotechnical systems, In Proceedings of the Conference on Fairness, Accountabilltty and Transparency, FAT* '19, pp. 59-68, 2019. doi: 10. 115532875500332555 32285598.
Harshay Shah, Kaustav Tamuly, Aditi Raghunathan, Prateck Jain, and Pranecth Netrapalli. The pitfalls of simplicity bias neural networks. In Advances in Neural Information Processing Systems, volume 33, pp. 9573 9983- 2020.
Shreya Shankar, Yoni Halpern, Eric Breck, James Atwood, Jimbo Wilson, and D. Sculley. No classi- fication without representation: Assessing geodiversity issues open data sets for the developing world, 2017. arXiv:1711 1.1.0856...
Berfin Simsek, Melissa Hall, and Levent Sagun. Understanding distriibittiitio accuracies through quantifying difficulty of test samples, 2022. arXiv:2203.15100
Pierre Stock and Moustapha Cisse. Convnets and imagenet beyond accuracy: Understanding mis- takes and uncovering biases. Proceedings of the European Conference on Computer Vision (ECCV), 2018.
Swabha Swayamdipta, Roy Schwartz, Nicholas Lourie, Yizhong Wang. Hannaneh Hajishirzi, Noah A. Smith, and Yejin Choi. Dataset cartography: Mapping and diagnosing datasets with training dynamics, 2020. arXiv:2009.1079:
Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes, Adam Trischler, Yoshua Bengio, and Geoffrey J. Gordon. An empirical study of example forgetting during deep neural network learning. In International Conference on Learning Representations, 2019. URL https:// settrevview.ne//oourn wie-dddddddddeeeeeeeee
Guillermo Valle-Perez, Chico Q. Camargo, and Ard A. Louis. Deep learning generalizes be- cause the parameter-funntiin map is biased towards simple functions. In International Confer- ence on Learning Representations, 2019. URL
neteee.weeeeeteee www.:eee/ffooooo net/forumffddt
13y4433AFm
Emiel van Miltenburg, Stereotyping and bias in the flickr30k dataset, 2016. arXiv: 11w.:66050608..
Michael Veale and Reuben Binns. Fairer machine learning in the real world: Mitigating discrimina- tion without collecting sensitive data. Big Data & Society, 442::0033517117733555 2017. doi: 00.1177205395177774442
Angelina Wang and Olga Russakovsky. Directional bias amplification. In Proceedings of the 38th
International Conference Machine Learning, volume 139, pp. 10882-10893, 2021.
Tianlu Wang, Jieyu Zhao, Mark Yatskar, Kai-Wei Chang, and Vicente Ordonez. Balanced datasets not enough: Estimating and mitigating gender bias in deep image representations. In Pro- ceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2019.
Benjamin Wilson, Judy Hoffman, and Jamie Morgenstern. Predictive inequity in object detection, 2019.
D.H. Wolpert and W.G. Macready. No free lunch theorems for optimization. IEEE Transactions on Evolutionary Computation, 1(1):67-82, 1997. doi: 10.1 109/4235.5858993
Blake Woodworth, Suriya Gunasekar, Mesrob I. Ohannessian, and Nathan Srebro. Learning non- discriminatory predictors. Proceedings of the 2017 Conference on Learning Theory, vol- ume 65, pp. 1920-1953, 2017.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnine a novel image dataset for benchmark- ing machine learning algorithms, 2017.
14

---

Under review as a conference paper at ICLR 2022
Kaiyu Yang, Klint Qinami, Li Fei-Fei, Jia Deng, and Olga Russakovsky. Towards fairer datasets: Filtering and balancing the distribution of the people subtree in the imagenet hierarchy. In Pro- ceedings of the 2020 Conference on Fairness, Accountability, and Transparency, FAT '20, pp- 547-558, 2020. doi: 10.1 0..145//3510995.377799..
Jingzhao Zhang, Aditya Menon, Andreas Veit, Srinadh Bhojanapalli, Sanjiv Kumar, and Suvrit Sra. Coping with label shift via distributionally robust optimisation, 2020. arXiv:2010.12230.
Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. Men also like shopping: Reducing gender bias amplification using corpus-level constraints. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 2979-2989. Association for Computational Linguisties, 2017. doi: 10. 10.18653/v/DD17-1322.
15

---

Under review as a conference paper at ICLR 2022
A PLS METHODOLOGY
Given classes, we extract d = m m=1 binary accuracies corresponding to each possible pairing, and convert them to rank orders. We repeat this for each of the n models under consideration, resulting in d X matrix of difficulty ranks. We construct d x data difficulty matrix from the cosine distance between the mean of each class.
We fit partial least squares regression model both the model difficulties and the data difficulties using scikit-learn (Pedregosa et al.] (011). For visualization purposes we use a single component, though in subsequent tests find no difference in model fit when increasing the number of com- ponents. We evaluate how well the data difficulty is explained by the model difficulty using It²,
B MEASURING DIFFICULTY
In this work we choose to measure difficulty using cross-validated test accuracy, averaged over all samples in a group or class. Recent works have investigated alternative methods for quantifying model-specific example difficulty, including loss (Arazo al. 2019 Han et al. 2018) and predic- tion disagreement between models (Simsek et al. (2022), mini-bati batchestchs (Chang et al. 2017), and throughout training (Toneva al., 2019 (Saayamdaa ples that are often forgotten after compression. Applying these sample-level measures to evaluating group-level difficulty disparity remains an interesting future direction.
al., 2020 Hooker (2021) ) identifies sam-
C MODEL ARCHITECTURE AND HYPERPARAMETERS
C.I
SVM
For the SVM we use an RBF kernel with hyperparameters C 1.0 and y 3072 using the scikit- learn implementation.
C.2 NEURAL NETWORKS
All models are implemented in PyTorch (Paszke et al. 2019) with Torch Models are trained to minimize cross-entropy loss using SGD with learning rate 0.01, momentum 0.9, weight decay of 0.0001 for 500 epochs with batch size 128.
FC. The fully-connected networks are either 3 or 5 hidden layers with 256 units and ReLU activa- tion. Batch normalization applied to the inputs.
LeNet. LeNet (Lecun et al. 1998) is simple CNN, with two convolutional layers interleaved with max pooling, three fully-connected layers, and ReLU activation function.
AlexNet. AlexNet (Krizhevsky et al.) 2012) is deeper CNN, with five convolutional layers inter- leaved with max pooling, three fully-connected layers, and ReLU activation function.
R8sNet-18. We use the variable-width ResNet (He et al. 2016) implementation of (Sagawa et al. 2020).
SSL. For both SSL models, we extract final-layer representations for each data point from an SSL- pretrained model. We pass these representations through 1-layer FC network as described above. Representations are extracted from one two models. The first is from ResNet-50, pretrained with SimCLR (Chen et al., 2020) on ImageNet- 1K (Russakovsky et al. 2015). The second is from a RegNet- 128Gf model Radosavovic al.] 2020) trained with SwAV (Caron et al. (020)) billion public images from Instagram (Goyal et al. (022a)). Representations were extracted using VISSL (Goyal al. 2021 from models publicly available in the model zoo.
16

---

Under review as a conference paper at ICLR 2022
D DOLLAR STREET EXPERIMENT
D.1 DATASET
Dollar Stree dataset of gcographically-divess images spanning a broad range of household incomes. Dollar Street comprises 23724 RGB 480 480 images of objects and people in everyday environments around the world, each associated with one of 138 class labels. For our purposes, discard geographic information and use income quartiles group label.
Throughout this work, we have endeavored to remove bias resulting from group imbalance and la- bel/group correlation. However, in the Dollar Street example we introduce an additional possible source of bias via ImageNet-IK IK pretraining. ImageNet-1K, IK significantly under-represents many so- cial groups (Dulhanty & Wong 2019) and geographies (Shankar et al. 2017: Vries et al. 2019), and exhibits harmful associations between race and certain class labels (Stock Cisse (2018) Ge- ographic under- rnder-representation is plausible reason for income-quariia difficulty disparity. This, however, cannot explain difficulty amplification.
D.2 MODEL
We follow the representation extraction method outlined appendix C.2 though we use the repre- sentations from supervised learning models rather than SSL. Specifically, we extract from ResNet- 50 trained with supervised learning on ImageNet-IK (Russakovsky et al. 2015). We train a single layer fully-connected network of varying width, following the standard SGD training regime speci- fied above.
E CALCULATING AMPLIFICATION FACTOR WITH LINEAR REGRESSION
Given a vector estimated accuracy disparities and a vector of observed accuracy disparities d, estimate the amplification factor using OLS linear regression.
In our synthetic setup, we additionally control for the effect of confounds including within-class separability (e.g. aquatic mammal / medium mammal in fig. 3R), and diagonal separability (e.g. medium mammal fflowr), by including them as nuissance regressors. Our full model is of the form:
a.1
Sa.
d= XB+6
x=
$
b,1
(4)
$0.0
6.0
S1.1
where amplification factor is the first parameter, A Bu. and 5 is the separability (i.e. accuracy) between group a, label and group b, label 1. We use Python statsmodels (Seabold & Perktold 2010) to fit the model.
b.1
17

---

Under review as a conference paper at ICLR 2022
F SUPPLEMENTARY FIGURES
dizyortyy
0.3
0.8-
Test accuragy
0.8
0.7
0.2
Simple Complax
Simple Complex
100 1,500
10,000
12
10
100 1.000 10,000
0.8
12
10
100 1,000 10.000
Stop
Step
Step
(a) Train accuracy
(b) Test accuracy
Observed disparity
Figure S1: Accuracy and disparity for ResNet- 18 on Fashion MNIST (Xiao et al. 2017). Experi- ment design is identical to fig. [] Simple group Trouser/Sneaker; complex group 7-Shirt/Shirt. Results align with those observed on CIFAR-100. (a) Train accuracy for complex group is learned slower but both groups reach perfect train accuracy. (b) However, test accuracy for complex group is persistently lower. Dashed lines are binary accuracy from sigle-group training. (c) Observed disparity peaks early in training but large gap persists after training. Red dashed line is estimated disparity d: observed disparity above this line indicates amplification. Shaded area standard error over 10 runs each with different train/test split.
Olcarved disparity
0.3
0.8-
Test accumey
0.7
0.2
0.-
Simpla
100 1,000 10,000
12
10
100 1.000 10,000
100 1.000 10.000 Step
Step
(a) Train accuracy
(b) Test accuracy
(c) Observed disparity
Figure S2: Accuracy and disparity for ResNet-18 on EMNIST letters (Cohen et al. 2017). Exper- iment design is identical fig. 3 Simple group is Q/X; complex group I/L. Results align with those observed on CIFAR-100 100. (a) Train accuracy for complex group is learned slower but both groups reach perfect train accuracy. (b) However, test accuracy for complex group persistently lower. Dashed lines are binary accuracy from single-group training.
(c) Observed disparity d peaks early, drops below estimated disparity during training, and then stabilizes slight amplification.
18

---

Under review as a conference paper at ICLR 2022
1.5
Amplitication factor
Amplification factor
1.0
FC 3-layer 5-layer
LeNet
0.5
0.5
ResNet18
SSL RegNet128
SSL ReeNet50
SVM
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Full test accuracy
Sampled test accuracy
(a) Full CIFAR- 100 coarse
(b) Sampled tasks
Figure S3: Amplification factor k as function of average test accuracy on (a) the full CIFAR-100 coarse dataset, and (b) the 30 sampled tasks used for computing amplification factor. Choosing a higher accuracy model, e.g/ an SSL model, would increase amplification. Vertical bars are standard error of the coefficient k. Horizontal bars (barely visible in left panel) are standard deviation of test accuracy over (a) seeds and (b) 10 seeds and 30 tasks. Solid black line fit with linear regression. Dashed gray line is k = 1.
1.0
0.9
0.8
SSL RegNet128
ResNet50
ResNot34
ResNet18
3-layer
FC 5-layer
AlexNet
0.6
LeNet
SVM
Model
Figure S4: R² values for linear regression calculation of amplification factor, for various models, corresponding to fig.
0.9g
0.8
D8
0.8
0.8
0.74
0.7
0.7
0.5
0.4
04
0.4-
0.4
0.3
0.3
0.3
01
0.0
8
32
10.000
0.0
1e/6 26'4 2e:3
0.0
0.001 0.010.03 0.1 0.3
width
weight decay
gradient, peeaatty
(b) Step
(a) Width
(c) Weight decay
(d) Gradient penalty
Figure 85: R2 values for linear regression calculation of amplification factor, corresponding to fig.
19


Reviews:
[{"review_id": "fXAYdtTCea", "reviewer": "Reviewer_po3o", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper explores how “difficulty” is model-specific, such that different models find different parts of a dataset challenging. They measure difficulty as a function of both model and data,  and they measure how difficulty affects performance disparity and quantify the effect of the simplicity bias. Their main observations are: 1. They identify difficulty disparity persists in the post-dataset audit setting. 2. They introduce a difficulty amplification factor to quantify how much a model exacerbates difficulty disparity. 3. They evaluate how choices including model architecture, training time, and parameter count impact difficulty amplification.\n", "summary_of_the_review": "The paper presents an investigation on the association of task difficulty and model bias. However, the observations are not quite convincing.", "strength_and_weaknesses": "Strength\n\n- The paper performs investigation of the task difficulty and reveals it is a function of both model and dataset.\n\nWeakness\n- The main argument of the paper (the task difficulty is not a function of the dataset itself but relies on the model as well) seems to be a known fact as different models have different capacity or generalization ability and their difficulty of solving a task also varies. For example, a deep CNN (e.g, ResNet-152) can perform better than a shallower one on ImageNet and thus the difficulty of solving ImageNet is different for different models. \n\n- There is no explanation on how the model bias leads to different difficulty disparity. The model bias is also not clearly defined. The experiments are only based on a handful of different architectures/pre-trained models and it is not clear why these models are selected.\n\n- The paper does not provide a solution for precisely estimating the difficulty disparity of a model. Given section 6, it is not still clear about the real-world impact of the difficulty amplification and what practical lessons can be learned from the observations.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "UPDok9XcdFJ", "reviewer": "Reviewer_9g7M", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work proposes a notion of difficulty disparity between subsets of a dataset and claims that neural networks parameterize functions that better separate groups with lower difficulty. The claim is then validated empirically via a set of experiments on CIFAR-100 and the Dollar Street datasets. Results indicated that difficulty amplification is affected by factors such as model architecture and early stopping.", "summary_of_the_review": "This submission is clearly written and aims at shedding light on interesting and relevant aspects of SGD-trained models. My major concerns are related to the lack of support for central claims of the narrative in the paper, the generalization of the findings, and lack of depth in the experimental analysis, which was critical to my score given that the contributions of this work are mostly empirical. All in all, I found that the flaws in the current version of this submission outweigh its merits by a large margin and I believe this manuscript is not ready for publication yet. ", "strength_and_weaknesses": "Strengths:\n- The paper adds evidence to show that biased predictions are an effect of both datasets and models biases, a often discussed question within the machine learning community;\n- The paper is clearly written;\n- The authors made a clear effort to convey their message by providing pictures that clearly illustrate the issues tackled in the paper and summarizing the main findings of each experiment.   \n\nWeaknesses:\n- The paper contributions are mostly empirical and the scope of the experiments is quite narrow, which makes it difficult to conclude how robust the findings are to other common settings such as neural networks trained with optimizers different from SGD;\n- Critical claims to support the findings are not well explained and supported. Importantly, the authors claim that they perform experiments in a setting where models do not pick up on spurious correlations between input and labels, but in no way this claim is supported in the manuscript.\n- The proposed measure of difficult amplification relies on training predictors for each particular group in the dataset. I believe this is not practical in many real-world settings where examples from particular groups are very scarce, making it difficult to train and evaluate specialized models.\n- Computing the proposed measure of difficult amplification further requires knowledge about group labels so that the dataset can split. This information is not always available in practice and / or shouldn’t even be taken into account when collecting a dataset.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "JGYXv00upIh", "reviewer": "Reviewer_eHoV", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper focuses on the problem of difficulty amplification: trained models exhibit consistent differences in performance on groups that are \"easy\" and \"difficult\" even when there are no obvious spurious correlations or dataset imbalance. In particular, they show that the  simplicity bias is a major reason for difficulty amplification. First, they show that data difficulty is model-specific (different models find different aspects of the dataset difficult). Then, the paper shows that models trained on a variant of CIFAR-100 (with easy and hard groups) exhibit difficulty amplification. Section 5 focuses on understanding how the amplification varies as a function of design choices such as model architecture, model width, early stopping etc. Section 6 shows that difficulty amplification can be an issue in practice as well. ", "summary_of_the_review": "Please see strengths and weaknesses.", "strength_and_weaknesses": "Strengths:\n- CIFAR-100 setup. The synthetic variant of CIFAR-100 incorporates an intuitive notion of model-specific difficulty and clearly showcases that difficulty amplification can show up in standard image classification datasets. \n\n- Amplification factor analysis. The analysis on how the amplification factor changes depending on standard design choices such as architecure, early stopping etc is insightful. \n\n- The paper (especially the first few sections) is well written. \n\n\nWeaknesses:\n- Main finding not novel: The main finding of this paper (performance disparity is amplified when easy and hard groups are clumped in the dataset) is not new. This paper (https://proceedings.neurips.cc/paper/2020/hash/6cfe0e6127fa25df2a0ef2ae1067d915-Abstract.html) shows that simplicity bias hurts generalization when the dataset comprises a majority subpopulation of \"easy\" data points and a minority subpopulation of \"hard\" data points---the drop in accuracy stems from misclassifying the minority subpopulation, even though a model trained only on the minority subpopulations performs well. The authors should clearly contextualize the experiments in this paper vis-a-vis previous findings on this phenomenon.\n\n- Section 3: The findings in this section contradict previous work. There are several works (e.g. http://proceedings.mlr.press/v119/hacohen20a/hacohen20a.pdf and references therein) that show that models with different architectures etc learn similar classifiers + classify points correctly over time in a similar order. On the other hand, this paper says that dataset difficulty is different for different models. So, it would be nice to reconcile these findings with previous works. Is this because the models compared in this paper are too different (e.g, linear models and CNNs)? Also, PLS analysis hard to understand: The paper needs to introduce PLS briefly before it introduces the findings. The way it is written right now is hard to parse.\n\n- Analysis on disparity over time. It is not clear why the paper focuses on difficulty amplification over time. What is the practical implication of understanding how difficulty amplification varies over time? Having some clear motivation would be useful, right now it feels a bit ad-hoc.\n\n- Section 6. This section is too terse. I think the experiment setup and the analysis should be more in-depth given that this is the only experiment showcasing this phenomenon in practice. Most of the discussion can be deferred to an appendix to make room for this. \n\nOverall, this paper studies an important problem that is not well understood. However, my major concerns are (a) novelty---previous work on simplicity bias has analyzed this phenomenon, (b) section 3 claims contradict previous findings on this topic, (c) limited + unclear analysis on difficulty amplification in practice. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "6A952EoK2e", "reviewer": "Reviewer_Reqo", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper introduces the notions of difficulty disparity and difficulty amplification, which is the ratio of change in the accuracy gap between different groups (easy vs hard groups) when trained in a combined manner versus when trained on those groups individually. The experiments show different models have different difficulty amplification, and it is not just a data-only issue.", "summary_of_the_review": "While the measure might be useful, I think the paper needs to address at least these shortcomings: a) study on settings without group labels since that is more realistic b) confirm the weight decay finding better (perhaps using the underrepresented groups) to be accepted.", "strength_and_weaknesses": "**Strengths**\n\n[S1] The paper provides a general, intuitive measure for difficulty amplification and the experiments show that this is dependent on the models and not just data.\n\n[S2] If group labels are provided, then the paper can be used by newer works to measure difficulty disparities.\n\n**Weaknesses**\n\n[W1] This work requires group labels, which may not always be available. Recent works on debiasing techniques have realized that this is an unrealistic assumption, so use implicit measures of difficulty e.g., Learning From Failure, Just Train Twice etc. It is unclear how this work could leverage the implicit measures to compute amplification.\n\n[W2] Group underrepresentation is a real-world problem. The model unfortunately does not consider this setting. Furthermore, ‘underrepresentations’ may not be the only source of spurious correlations e.g., even with balanced groups certain factors in the group (not necessarily signal) may be easier to exploit. So, the claim of ‘absent correlations between group labels and target labels’ may require further verification.\n\n[W3] Weight decay is generally used to regularize the complexity of the model (to prevent overfitting), yet the paper surprisingly finds ‘next to no effect of weight decay’. Could you elaborate on this finding? Would it hold in general? One interesting experiment would be to analyze the models from [1], where the authors showed that with increased weight decay (decreasing overparameterization) even simple re-weighting could help debias models.\n\n[1] Sagawa, Shiori, et al. \"An investigation of why overparameterization exacerbates spurious correlations.\" International Conference on Machine Learning. PMLR, 2020.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject