Overparameterisation and worst-case generalisation: friend or foe?
Authors: Aditya Krishna Menon
Ankit Singh Rawat
Sanjiv Kumar
Venue: ICLR 2021 Poster (2021)
Keywords: overparameterisation
worst-case generalisation

Abstract: Overparameterised neural networks have demonstrated the remarkable ability to perfectly fit training samples, while still generalising to unseen test samples. However, several recent works have revealed that such models' good average performance does not always translate to good worst-case performance: in particular, they may perform poorly on subgroups that are under-represented in the training set. In this paper, we show that in certain settings, overparameterised models' performance on under-represented subgroups may be improved via post-hoc processing. Specifically, such models' bias can be restricted to their classification layers, and manifest as structured prediction shifts for rare subgroups. We detail two post-hoc correction techniques to mitigate this bias, which operate purely on the outputs of standard model training. We empirically verify that with such post-hoc correction, overparameterisation can improve average and worst-case performance.

Published as a conference paper at ICLR 2021
OVERPARAMETERISATION AND WORST-CASE GENER- ALISATION! FRIEND OR FOE?
Aditya Krishna Menon, Ankit Singh Rawat & Sanjiv Kumar Google Research New York, NY (adi tyakmenon, anki tswattt, sanjivkjegoogle.coom
ABSTRACT
Overparameterised neural networks have demonstrated the remarkable ability to perfectly fit training samples, while still generalising to unseen test samples. However, several recent works have revealed that such models' good average per- formance does not always translate to good worst-case performance: in particular, they may perform poorly on subgroups that are under-represented in the training set. In this paper, we show that in certain settings, overparameterised models' perfor- mance on under-represented subgroups may be improved via post-hoc processing. Specifically, such models' bias can be restricted to their classification layers, and manifest as structured prediction shifts for rare subgroups. We detail two post-hoc correction techniques to mitigate this bias, which operate purely on the outputs of standard model training. We empirically verify that with such post-hoe correction, overparameterisation can improve average and worst-case performance.
INTRODUCTION
Overparameterised neural networks have demonstrated the remarkable ability to perfectly fit training samples, while still generalising to unseen test samples (Zhang et al., 2017; Neyshabur et al., 2019; Nakkiran et al., 2020). However, several recent works have revealed that overparameterised models' good average performance does not translate to good worst-case performance (Buolamwini & Gebru, 2018; Hashimoto et al., 2018; Sagawa et al., 2020a:b). In particular, the test performance of such models may be poor on certain subgroups that are under-represented in the training data. Worse still, such degradation can be exacerbated as model complexity increases. This indicates the unsuitability of such models in ensuring uses of machine learning (Dwork et al., 2012; Hardt et al., 2016; Buolamwini & Gebru, 2018).
across subgroups, a topical concern given the growing societal
Why does overparameteriisaton induce such unfavourable bias, and how can one correct for it? Sagawa et al. (2020a) demonstrated how such models may fit to spurious correlations that explain under-represented samples, which can generalise poorly. Sagawa et al. (2020b) further posited that overparameterised models have an inductive bias towards memorising labels for as few samples as possible, which are invariably those from under-represennted subgroups. To mitigate such bias, existing approaches include subsampling majority subgroups (Sagawa et al., 2020b), and modifying the training objective (Sagawa et al., 2020a; Nam et al., 2020; Zhang et al., 2020; Goel et al., 2020). This suggests two important points regarding overparameterised models' performance:
(a) with standard training, increasing model complexity exacerbates degradation on rare subgroups; (b) controlling this degradation may require alternate training objectives or procedures.
In this paper, we establish that while overparameterised models are biased against under-represented examples, in certain settings, such bias may be easily corrected via post-hoc processing of the model outputs. Specifically, such models' bias can be largely restricted to their classification layers, and manifest structured shifts in predictions for rare subgroups. We thus show how two simple techniques applied to the model outputs - classifier retraining based on the learned representations, and correction of the classification threshold can help overparameterised models improve worst- subgroup performance over underparameterised counterparts. Consequently, even with standard training, overparameterised models can learn sufficient information to model rare subgroups.

---

Published as a conference paper at ICLR 2021
Positive class
0.10
Negative class
Dominant subgroup
Dominant subgroup
80
ERM
subgrrup
0.08
suSgroup
Corrected
70
Worst subgroup
0.04
Denole
0.006
60
0.04
55
40
0.02
30
0.00
0.00
-20
20 Decision score
-40
-20
20
Decision score
log m
(a) Distribution of overparameterised model scores.
(b) Effect of post-hoc correction.
Figure 1: Distribution of model decision scores test samples from a synthetic dataset of Sagawa et al. (2020b), comprising two labels with two subgroups each (left panel). The scores are expected to be 0 iff the label is positive. We train linear model with complexity controlled by its number of features m. For the overparameterised setting m = 10 (left), within each class, rare subgroups consistently appear on the wrong side of the decision boundary. Correcting this bias via post-hoc score translation improves the worst-subgroup enror as the model complexity is increased (right).
To make the above concrete, Figure 1 plots histogram of model predictions for synthetic dataset from Sagawa et al. (2020b) (cf. $2). The data comprises four subgroups generated from combinations (y, axx)) of labels y {+1) and a feature a(x) {+1]. Most samples (ar,) y) have y a(x), and so these comprise two dominant subgroups within the positive and negative samples. We train an overparameterised linear model, yielding logits f+1 (x). We then plot the decision scores f+) 1 which are expected to be > iff +1. Strikingly, there is distinct separation amongst the subgroup scores: e.g., samples with y = +1, a(x) = -1 have systematically lower scores than those with y +1, +1)a(z) +1. Consequently, the model incurs significant error rate on rare subgroups. The structured nature of the separation implies suggests to post-hoc shift the scores to align the distributions; this markedly improves performance on the rare subgroups (Figure 1b).
Scope and contributions. The primary aim of this work is furthering the understanding of the behaviour of overparameterised models, rather than proposing correction techniques we employ have been well-studied in the related problem setting of long- tail learning or learning under class imbalance (He & Garcia, 2009; Buda al., 2017; Van Horn & Perona, 2017). Several works have demonstrated that the representations learned by standard networks contain sufficient information to distinguish between dominant and rare labels (Liu et al., 2019; Zhang et al., 2019; Kang et al., 2020; Menon et al., 2020). Similar techniques are also common the fairness literature (Hardt et al., 2016; Chzhen et al., 2019). However, whether such techniques are effecttive for overparameterised models, whose ability to perfectly fit the training labels can thwart otherwise effective approaches (Sagawa et al., 2020a).
techniques. Indeed, the post-hoc
not a-priori clear
Existing techniques for improving the worst-subgroup error of overparameterised models involve altering the inputs to the model (Sagawa et al., 2020b), or the training objective (Sagawa et al., 2020a). By contrast, the techniques we study alter the outputs of standard network, trained to minimise the softmax cross- cross-entropy on the entire data. Our findings illustrate that such models do not necessarily require bespoke training modifications perform well rare subgroups: even with standard training, overparameterised models can (in certain settings) learn useful information about rare subgroups.
In summary, contributions
(i) we demonstrate that, in certain settings, overparameterised models' poor performance on under- represented subgroups is the result of structured bias in the classification layer (ef. $3); iii) we show that simple post-hoc correction procedures (cf. $4) can mitigate the above bias, and thus significantly reduce their worst- wubst-subgroup error (cf. $5).
2 BACKGROUND AND SETTING
Suppose we have labelled training sample S - {1x, Hi E (X x 4)", for instance space X c IRd and label space y. One typically assumes 8 is an i..d draw from some unknown distribution P(r, v). Further, suppose each (ar, y) has associated group membership g(x, y) S. with
2

---

Published as a conference paper at ICLR 2021
G This induces G data subgroups, with prior P(g) and conditional distributions P(x:, y g). Following Sagawa et al. (2020a;b), we consider groups g(z,, y) = (y. a(z)). where a(z) E R is some attribute within æ. We assume a(z) is fully specified during train and test time; while not always realistic, such an assumption has precedent in the fairness literature (Lipton et al., 2018).
The standard goal in classification is to learn classifier h: X y that minimises the average error Lavg(h (h) = E E [6o1 (0.()(())) h(x)),,
where for h(x)) - LV + h[z)] is the 0-1 loss. Typically, one constructs h(z) = argmax, (2), where f(x) RD comprises real-valued logits, learned by empirical risk minimisation (ERM): min fer n Ei= E(y,, (x:))) Here, is a surrogate loss such as the softmax cross-entropy, and F is function class, such as neural networks with fixed architecture. A network is overparametrised if it can perfectly fit the training labels, and thus drive the training error to zero. Remarkably and in apparent contrast to orthodox statistical wisdom this does not come at the expense of
generalisation on test samples (Zhang et al., 2017; Belkin et al., 2019; Nakkiran et al., 2020).
This apparent power comes at price, however. Let us define the worst- worst-subgroup error as Lunax (h) = max E gES x.ylg
[601 (y, h())).
(1)
i.e., the worst-case error over all data subgroups. Prior work (Sagawa et al., 2020a;b) established that for overparameterised models, the worst-subgroup training error can go to zero (since the model can fit all samples), but the worst-subgroup test error can devolve to that of random guessing (since the model can fit spurious correlations for rare subgroups). Further, the degree of degradation can increase with the model complexity. This indicates that the naïve use of overparametrised models may be odds with ensuring fairness across data subgroups, a core concern moden applications of machine learning (Calders & Verwer, 2010; Dwork et al., 2012; Hardt et al., 2016; Zafar et al., 2017).
There are several potential strategies to cope with this. One is to perform distributionally robust optimisation (Hashimoto et al., 2018; Mohri et al., 2019; Sagawa et al., 2020a), and minimise:
LORO = max 9E9
E [C (x)))+ F.yle
where S2g is some per-group regulariser. In settings where F(g) is non-uniform, Sagawa et al. (2020a) proposed to set (2)()) = / where ng is the number of training samples with group Alternatively, one can reweight samples upweight the contribution of rarer groups and minimise:
LRw (h) = Σ " gES
E [K(y, f(x)).
where, e.g., My PP(g) leads to the standard average error, while wy 1 implicitly upweights rare subgroups. While intuitive, Sagawa et al. (2020b) established that such an approach is also subject poor worst-subgroup performance, owing to a broader issue with using importance weighting in conjunction with neural networks (Wen et al., 2014; Byrd & Lipton, 2019). Sagawa et al. (2020b) established that one can achieve good performance by instead subsampling dominant groups, an operation equivalent in expectation to minimising Law(h, with w, = 1. Recent developments in the mitigation worst-subgroup errors include Nam al. (2020); Zhang et (2020); Goel et al. (2020).
In the sequel, we shall make extensive use of three datasets from Sagawa et al. (2020a;b), each of which involve binary labels y E y and a binary attribute a(x) E .A:
(i synth, synthetic dataset where X C IR 200 y {+1), and A E {+1}.
iii) waterbirds, a dataset of bird images with y = [land bird, water bird] corresponding to the bird type, and A {land background, water background) corresponding the background.
(iii) celebA, dataset of celebrity images with y (blond, dark) corresponding to individuals' hair colour, and A = {male, female).
For each dataset, we construct four subgroups g(z, y) (y. ax))), with two such subgroups being meder-represented. On synth and waterbirds, these correspond to subgroups with y + a(z). while on celebA, these corespond to the subgroups ((blond,mal) male)} and ((dark, female)). Owing to
3

---

Published as a conference paper at ICLR 2021
Negative (rane)
Negative
Poskiie (dominant)
Positiva tranc)
20
20
-40
-40
-80 20
66
-40
20
-40
-20
20
60
waterbirds.
(b) celebA.
Figure 2: Two-dimensional tSNE visualisation of test embeddings produced by overparameterised models. Within each class, samples from each the two subgroups tend to closely clustered. This suggests that the representations learned by the models contain sufficient information to help distinguish samples from rare versus dominant subgroups. The poor worst-subgroup performance of such models may thus be traced issues with the classification layer. (Best viewed in colour.)
the rarity of certain subgroups. it is intuitively easy for an overparameterised network to learn to predict axx) rather than y. and memorise spurious patterns to predict the rare subgroups.
To train overparameterised models, we follow the setup of Sagawa et al. (2020a;b), which we briefly summarise, For celebA and waterbirds, we use ResNet-50, which can attain perfect training accuracy. For synth, we train weakly regularised (A 10 16) logistic regression model on a fixed representation & constructed as follows: for fixed m. we construct ((x) = Re.. where V mR is random Gaussian matrix with normalised rows. Overparameterised models consistently demonstrate a significant gap between the average and worst-subgroup error: e.g. (see Figure 4), synth, the model achieves 91 % average accuracy, but 36% worst-subgroup accuracy.
3 THE DOMINANT SUBGROUP BIAS OF OVERPARAMETERISS MODELS
We study the nature of overparameterised models' poor performance on rare subgroups more closely. We make two observations: first, this under-performance is largely owing to a bias in the classification layer. Second, this bias manifests in the form of distribution shift in the model scores for rare subgroups. This shall subsequently motivate post-hoc correction procedures.
3.1 CAN THE LEARNED REPRESENTATIONS DISTINGUISH BETWEEN SUBGROUPS?
Neural models make predictions based on logits fy(x) = w. ((x), for classification weights wy IR and representations (xx) E K Suppose such a model performs poorly on subgroup aJ, This implies that for sample (z, g) in this subgroup, fr(r) < fy (z) for some competing y
Why do overparameterised models underperform logits suggests this arises from issues with the representations, the classification weights, or both. To determine which of these is likely, we begin by inspecting the representations. Recall that train a ResNet-50 embeddings. We may thus embed test instances from each of the four data subgroups, and study their geometric structure. Intuitively, if instances from rare and dominant subgroups with the same label have limited similarity, the representations for rare samples are insufficiently rich.
rare subgroups? The factorised nature of the
celebA and waterbirds, which produce K = 2048 dimensional instance
The high dimensionality of the embeddings prohibits exact inspection, but as rough surrogate, we employ a two-dimme wwo-dimensional tSNE (Maaten & Hinton, 2008) visualisation. As tSNE attempts to preserve neighbourhood information amongst samples, hope to evince the relative geometries of samples from the various subgroups. Figure reveals that for celebA and waterbirds, samples from the rare subgroups tend to be closely clustered with those belonging to the same class.
4

---

Published as a conference paper at ICLR 2021
Pusiiiee cas5
Neguliee Clss
Fookive Class
Negatiee dless
6.25
Geninartastgrrup Part subrg sou
6:c8
Centrantuddgaw Para subgrous
0.22
Comtrent adgroup Para subgrsus
6.10
Faru sungread
Demi
0.20
20.15
20.06
5.09
0.00
0.54
0.04
0.23
0.42
0.22
0.05
20
33
32 0.00
0:20
10
0:00
30
13
waterbirds.
(b) celebA.
Figure 3: Distributions of model decision scores (x) test samples on waterbirds and celebA, comprising two labels with two sub-groups each. The rare and dominant subgroup score distributions largely differ, with the former often on the incorrect side of the decision boundary (i.e., with decision scores < 0 for positive samples, and >> O for negative samples). Notably, the scores for rare subgroups tend to be shifted compared to those for dominant subgroups.
The above suggests that the representations learned by the models contain information to (at least partly) help distinguish samples from rare versus dominant subgroups." This suggests the poor worst-subgroup performance of such models may result from issues with the classification layer.
3.2 HOW DOES BIAS MANIFEST IN THE CLASSIFICATION LAYER?
To study the potential issues in the classification layer, we continue our strategy of visualising the model outputs. Since the datasets we consider involve binary labels, we may simply study the distribution of the decision scores (x) - fz) Since our predicted label is the highest scoring logit, we desire this score to be >> 0 iff the sample has a positive label. As in the previous section, we may break down these scores for each of the subgroups induced by label y and attribute a(x).
We earlier illustrated this distribution for synth in Figure 1; Figure 3 further provides distributions for celebA and waterbirds. On both datasets, within each label, there is a shift in the score distributions for one or both of the rare subgroups. This is in keeping with the poor performance of the model on these subgroups: e.g., on waterbirds, rare samples in the positive class systematically have negative decision scores, implying the model incurs high nelse-negative rate on these samples.
In light of the above findings, we may revisit ablations from prior work that tease apart the factors causing poor substusu performance. For example, Sagawa et al. (2020b) showed that increasing model complexity on synth can degrade worst-suhgroup performance beyond a certain critical point, and that this can be mitigated with combination of strong regularisation and subsampling. The same conclusions largely hold (see Appendix D) for the varying decision scores amongst subgroups: e.g.. the difference in rare and dominant subgroup scores is exacerbated we increase model complexity.
3.3 DISCUSSION AND IMPLICATIONS
The systematic under-predietto of scores for rare subgroups can be seen as a particular manifestation of neural networks producing uncalibrated probability estimates (Guo et al., 2017): from Figure 3, the model will systematically under- or over-estimate the probability of rare samples being positive.
We emphasise here that our illustrations above are for test samples not observed during training. Training samples exhibit qualitatively different trends, reflective of overparameterised models' ability to perfectly fit them: e.g., the decision scores for all samples are consistently on the correct side of the decision boundary (see Appendix D). The fact that the scores on unseen samples exhibit distinction amongst subgroups suggest the network encodes an implicit bias against such samples.
At the same time, this bias largely manifests as a translation of the scores. This suggests a simple post-hoc correction of the scores may suffice to improve performance; e.g., bumping up scores for samples with a(x)) = land background in waterbirds can make the error on the rare subgroup (water bird, land background) more equitable. It remains now to more carefully describe such host-hoc procedures, and study their performance.
We crucially rely on classification layer to help distinguish samples between subgroups; by themselves, however, the embeddings be systematically biased (Bolukbasi al., 2016; Gonen & Goldherg, 2019).
5

---

Published as a conference paper at ICLR 2021
4 CORRECTING THE SUBGROUP BIAS OF OVERPARAMETERISE MODELS
Drawing from the literature on long-tail learning (Zhang et al., 2017; Kang et al., 2020; Ye et al., 2020) and fairness (Hardt et al., 2016; Chzhen al., 2019), we now detail two post-hoc correction techniques to mitigate overparameterised models' bias against rare subgroups.
4.1 CLASSIFIER RETRAINING
Given that $3.1 demonstrates that learned representations ((x) appear meaningful across subgroups, a natural thought is to fit a linear classifier on top of them; i.e., we treat ( new training set for linear model. Oveeparamettrrisation introduces challenge, however: since the original network can find a classifier with the lowest possible (i.e., zero) training error, simple modifications to the loss (e.g., reweighting samples) will result in learning the same classifier.
())) as a
Fortunately, there are several options to find a distinct classifier than the original network: e.g., one can subsample elements from the majority subgroups, per Sagawa al. (2020b). We emphasise an important difference between employing such techniques in standard training, and in classifier retraining: the latter uses representations learned from standard network, as opposed changing the network objective itself. The success of the latter shall thus demonstrate that the standard network representations are rich enough to reasonably distinguish between subgroups.
4.2 THRESHOLD CORRECTION
The illustrations in Figures 3 suggest a simple approach to improving classification performance: rather than using identical classification threshold for all samples, employ per-subgroup thresholds. In detail, observe that in the case of binary labels, we predict h(x) = +1 f+i(2) f-i(x) > Instead, we can predict
h(z) = +1
f-1(x) ta(s))
where (La E R: a E A) are per-attribute thresholds. Equivalently, this translates the scores for all samples with given attribute a(x) so that the decision boundary distribution shifts observed in Figures and 3: intuitively, by enforcing a lower threshold for samples with a(x) = 1, we can account for the fact that most of these samples obtain a low model score.
This can compensate for the
It remains to specify how to choose the thresholds t.. One simple option is to perform a parameter sweep, and employ the thresholds that minimise the worst-subgroup error on a holdout set. This is feasible in settings where |A| and |3] are small, and can directly target the performance measure of interest. For more complex problems, one may suitably parameterise the thresholds, or attempt to learn so as to minimise a suitable objective (see discussion below).
4.3 DISCUSSION AND RELATED WORK
The long-tail learning literature has demonstrated the value of both classifier retraining (Zhang et al., 2017; Kang et al., 2020) and threshold correction (Zhou & Liu, 2006; Collell et al., 2016; Menon al., 2020). Similarly, in the fairness literature, post processing of classifier outputs based on per-subgroup thresholds is a similarly well-established technique (Hardt et al., 2016). Our aim here to investigate the effectiveness of such techniques in the overparameterised setting. Given such models can perfectly fit training labels, it is less clear whether the representations learned from standard training are sufficiently useful learn a good classifier, and whether their outputs can be easily corrected post-hoe; e.g., model that merely memorised certain training labels could not be meaningfully corrected to perform well on test samples.
To extend threshold correction to multi-class settings, one could tie the thresholds to the frequencies P((v, a(x))) of various subgroups, rather than tune them. This is akin to class prior or logit correction techniques from long-tail learning (Collell et al., 2016; Menon et al., 2020). In the faimess literature, relevant techniques include Hardt et al. (2016), who propose an objective to select optimal thresholds for ensuring a particular notion of fairness; and recent techniques that learn post-processing of model scores (Chzhen et al., 2019; Jiang et al., 2020; Wei et al., 2020) to de-bias predictions. Exploring such techniques in the overparameterised setting is an interesting direction for future work.

---

Published as a conference paper at ICLR 2021
Table 1: Summary of average and worst-subgroup errors on synth, waterbirds, and celebA. For synth and waterbirds with logistic regression model, we use the overparameterised setting where m 10 Standard ERM performs well on average, but poorly on the worst-subgroup. However, this is significantly improved by post-hoc correction of the ERM predictions, either through classifier retraining (CRT) or threshold correction (THR), The former is only applied to models with a learned representation. We also report the results of distributionally robust optimisation (DRO) and sub-sampling of the training data (SAM), per Sagawa et al. (2020a;b). The results marked with f are quoted from Sagawa et al. (2020a, Table 2).
Datasel
Model
Average error
Worst-subgroup error
ERM
SAM
DRO
CRT
ERM
DRO
CRT
THR
synth
Logistic
9.49
23.99
17.99
N/A
22.18
63.20
27.38
22.67
N/A
27.51
waterbirds
Lugistic
4.37
17.18
10.57
N/A
16.71
52.34
20.82
16.73
N/A
17.46
ResNet
2.15
15.66
6.301
2.58
5.23
17.23
18.41
9.501
8.30
8.70
celebA
ResNet
4.64
9.28
6.60t
9.38
7.90
56.11
10.63
12.20f
14.68
12.96
The contemporaneous work of Sohoni et al. (2020) also considers the viability of improving worst- subgroup performance using the learned representations. Specifically, they propose to cluster these embeddings, and train new distributionally robust classifier predict cluster assignments. While very much in the spirit of our classifier retraining proposal, their study does not consider the efficacy of such procedure under varying model complexity, nor correction techniques that operate purely on the classification outputs.
EXPERIMENTS: HOW EFFECTIVE IS POST-HOC CORRECTION?
We now show that the above post-hoc correction techniques can significantly improve overparame- terised models' performance on rare subgroups. This indicates that even when trained in the usual manner, such models can learn useful information about rare subgroups.
5.1 EXPERIMENTAL SETUP
We follow the same basic setup as Sagawa et al. (2020a;b), We instantiate overparameterised models on each dataset: ResNet-50 on celebA and waterbirds; linear logistic regression on synth, using fixed features as described in $2; and a linear logistic regression on waterbirds, using the embeddings from ResNet-18 pre-trained on ImageNet. See Appendix A for details of experimental hyper-parameters. We measure both the average and worst-subgroup errors both the train and test set, repeating each experiment 5 times.
We apply post-hoc correction to these learned models, via classifier retraining (CRT) on the learned representations, using linear logistic regression model with subsampling of the dominant subgroups per Sagawa al. (2020b); and threshold correction (THR) the decision scores, using holdout set to estimate thresholds {ta: a E {+1}} that minimise the worst-subgroup error. For waterbirds, use the holdout set from Sagawa al. (2020a); for celebA, we use the standard holdout set; and for synth, we construct a holdout set using 20% of the training samples.
As reference, we report the results of standard minimisation training set, where following Sagawa et al. (2020b) we down-sample each subgroup to have A'in examples, where Mmiin is the number of examples in the smallest subgroup. We also report the results of the regularised distributionally robust optimisation (DRO) procedure of Sagawa et al. (2020a). This involves strongly regularising the model, and modifying the training objective worst-subgroup error. Our aim is not to improve on the performance of this method, which directly trains minimise the subgroup error; rather, our goal understand and quantify how much useful information standard model training learns about rare subgroups.
balanced subsample (SAM) of the
target the
5.2 RESULTS AND DISCUSSION
Table 1 summarises the test results all datasets. We highlight several key points.
7

---

Published as a conference paper at ICLR 2021
Tah
Acriagg TAET
Aorrigg THE
Weors Trem
Wars, (ran
Werrl bril were tee
28
Wontetd
20
20
togm
1og
log
ERM (synth).
(syath).
(c) ERM (waterbirds).
(d) THR (vaterbirda)..
Figure 4: Performance of baseline (ERM) and threshold correction (THR) on synth and waterbirds datasets, with fixed input representations. We vary the feature dimensionality m between 10a and 10a and plot the average and subst-su error on both the train and test set over 5 independent trials. For the baseline, the worst-case error devolves m increases. For threshold correction, both average and worst-case error steadily improve on the test set. Of salience is that overparameterisation (large m) aids worst-subgroup performance under post-hoc correction.
ERM has poor worst-subgroup error. In keeping with Sagawa et al. (2020a;b), standard ERM performs poorly in terms of surst-subgroup error. This may be strongly mitigated by modifying the training objective per DRO, by modifying the training sample per SAM. (Each of these incurs only a mild penalty in terms of the average error.) This confirms the finding of prior work that modification of the training procedure can achieve a suitable trade-off between average- and worst-subgroup error.
Post-hoc correction improves worst-subgroup error. Encouragingly, post-hoc techniques consis- tently and significantly reduce the worst-subgroup error of ERM. For example, on CelebA, THR reduces the worst-subgroup error from 56. 11% % to 12.96%, with only a mild increase in the average error. Between the post-hoc techniques, we generally find THR to yield the best performance. We reiterate that this technique does not modify the training procedure directly, but simply post-processes the learned ERM model output. This confirms the analysis of the previous section, which indicates that (on the considered datasets) the result of standard ERM can by themselves contain sufficient information to overcome poor worst-subgroup error.
Post-hoc correction is comparable to training modification. Post-hoc techniques generally com- pare favourably with DRO and SAM in terms of the trade-off between average- and worst-subgroup error. While DRO is notably superior on synth, this gap appears to be in part due to the challenge of tuning hyperparameters given holdout data with limited subgroup representation. For example, using larger holdout set size of n 2400 samples improves the performance of THR synth to 24%.
The superior overall performance of DRO is in keeping with findings about the efficacy of training modification for faimess (Agarwal et al., 2018). Nonetheless, the generally competitive performance of CRT and THR suggests the latter can extract non-trivial gains from overparameterised models.
ineppaaaeterrsation with post-hoc correction improve worst-subgroup error. We now confirm that increasing model complexity can result in improved average and worst-subgroup performance, provided the ERM outputs suitably corrected. As the synth and waterbirds datasets with fixed features allow for modifying the feature dimensionality m - which controls the degree of overparameteriistion we perform an experiment where we vary m between 10 and 10 Figure 4 shows the average and worst-subgroup error on both the train and test set, over 5 independent trials. The overall and worst-case training error approaches 0% as m increases (i.e., the model becomes overparameterisee)). Further, the average test error appears reasonable, approaching 10%; however, the worst-case error devolves as m increases, exceeding 60%.
The plots also show the results of THR as m is varied. Here, both average and worst-case error steadily improve: indeed, overparameterisation aids worst-suhgroup performance. Further, the gains in the worst-subgroup test eитor are dramatic compared to ERM. Thus, while ERM is susceptible to a tradeoff' between overparametcrisation and worst group accuracy, this tension can be mitigated with simple mechanisms to encourage equitable performance. This general trend is largely robust to distributional properties such as the fraction of majority samples in synth, though significantly increasing this expectedly degrades the best achievable performance; see Appendix D.
Illustration of classification thresholds. Figure 5 studies the effect of tuning the classification thresholds La on each of the per-subgroup errors for synth (m 10²). Here, we plot the errors for
8

---

Published as a conference paper at ICLR 2021
Positive class
Negative class
Dominant subgroup
Dominont sulggrous
0.26
Hare suogroup
0.06
Hare subggroup
Enrr
80
y
Density
0.04
Density
60
La
+
0.02
0.02
-40 -20
20
0.50
20
20
0.50
-30
-20
-10
0
10
20
Threshold
Decision score
Decision
(a) Effect of threshold fa-
(b) Distribution of scores with optimal threshold.
Figure 5: Effect of tuning classification threshold la for synth, when m = 10 The case la 0 corresponds the baseline. Modifying ta systematically trades off performance on the rare subgroups compared to the dominant ones (left). By choosing the thresholds to minimise the worst-subgroup error, the resulting score distributions are aligned for rare and dominant subgroups (right).
each of the four subgroups as we sweep over the thresholds. The case ta corresponds to the baseline. Modifying to trades off performance on the rare subgroups compared to the dominant ones. One may then pick optimal thresholds for = =1 which correspond to cross-over points of the error curves for each value of a: for example, when 4+1 ~ 10, the errors on the subgroups y = +1, +1 and y = -1, a = +1 are equitable, and thus the maximum of the two errors is minimised. Choosing such thresholds aligns the scores distributions for rare and dominant subgroups (right).
DISCUSSION AND FUTURE WORK
Post-hoc correction relies on knowledge of the data subgroups at train and test time. An important practical challenge is extending this to settings with unknown subgroups. One natural strategy is to attempt to unearth these subgroups via clustering of the model outputs, but careful study is needed to inform design choices (c.g.. choosing the number of clusters). For contemporaneous study of a similar technique, see Sohoni et al. (2020). Exploring the viability of post-hoc correction in overparameterised settings with multi-class labels, and multiple subgroups e.g., through adaptations of techniques noted in $4.3 - is also of interest. More broadly, the study of learning bias-free representations has received significant interest (Bahng et al., 2020; Kim al., 2019; Li & Vasconcelos, 2019; Arjovsky et al., 2020; Nam et al., 2020). Exploring the efficacy of such approaches in overparameterised settings would be of interest.
REFERENCES
Alekh Agarwal, Alina Beygelzimer, Miroslav Dudik, John Langford, and Hanna Wallach. A reductions approach fair classification. In Jennifer Dy and Andreas Krause (eds.), International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 60-69, Stockholmsmässan, Stockholm Sweden, 10-15 Jul 2018. PMLR.
Martin Arjovsky, Léon Bottou, Ishaan Guirajani, and David Lopez-Paz, Invariant risk minimization, 2020.
Hyojin Bahng. Sanghyuk Chun, Sangdoo Yun, Jaegul Choo, and Seong Joon Oh. Learning de-biased representations with biased representations. In International Conference on Machine Learning (ICML), 2020.
Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal. Reconciling modern machine- machine-learning practice and the classical bias variance trade-off. Proceedings of the National Academy of Sciences, 15849))15844-15854 2019, ISSN 0027-8424.
Tolga Bolukbasi, Kai-Wei Chang, James Zou, Venkatesh Saligrama, and Adam Kalai. Man is to computer programmer woman is to homemaker? debiasing word embeddings. In Proceedings of the 30th International Conference on Neural Information Processing Systems, NIPS' 16, pp. 4356 4364, Red Hook, NY, USA, 2016. Curran Associates Inc. ISBN 97815 9781510827 5108388888

---

Published as a conference paper at ICLR 2021
Mateusz Buda, Atsuto Maki, and Maciej A. Mazurowski. A systematic study of the class imbalance problem in convolutional neural networks. arXiv:1710.05381 [cs, stat], October 2017.
Joy Buolamwini and Timnit Gebru. Gender shades: Intersectional accuracy disparities in commercial gender classification. In Sorelle A. Friedler and Christo Wilson (eds.). Conference on Fairness, Accountability, and Transparency. volume 81 of Proceedings of Machine Learning Research, pp. 77-91, New York, NY, USA, 23-24 Feb 2018. PMLR.
Jonathon Byrd and Zachary Chase Lipton. What is the effect of importance weighting in deep learning? In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th Inter- national Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pp. 872 172.881.72 PMLR, 2019.
Toon Calders and Sicco Verwer. Three Naive Bayes approaches for discrimination-free classification. Data Mining and Knowledge Discovery, 21(2):277-292 2010.
Evgenii Chzhen, Christophe Denis, Mohamed Hebiri, Luca Oneto, and Massimiliano Pontil. Leverag- ing labeled and unlabeled data for consistent fair binary classification. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' Alché Buc. E. Fox, and R. Garnet (eds.), Advances in Neural Information Processing Systems 32, pp. 2760-12770. Curran Associates, Inc., 2019.
Guillem Collell, Drazen Prelec, and Kaustubh R. Patil. Reviving greeshld-mmvving. simple plug-in bagging ensemble for binary and multiclass imbalanced data. CoRR, abs/1606.08698, 2016.
Cynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Richard Zemel. Fairness through awareness. In Innovations in Theoretical Computer Science Conference (ITCS), pp. 214-226, 2012.
Karan Goel, Albert Gu, Yixuan Li, and Christopher Ré. Model patching: Closing the subgroup performance gap with data augmentation, 2020.
Hila Gonen and Yoav Goldberg. Lipstick on pig: Debiasing methods cover up systematic gender biases in word embeddings but do not remove them, 2019.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. In Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, pp. 1321-1330, 2017.
Moritz Hardt, Eric Price, and Nathan Srebro. Equality opportunity in supervised learning. In Advances in Neurul Information Processing Systems (NIPS), December 2016.
Tatsunori Hashimoto, Megha Srivastava, Hongseok Namkoong, and Percy Liang. Fairness without demographics in repeated loss minimization. In Jennifer Dy and Andreas Krause (eds.), Interna- tional Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 1929-1938, Stockholmsmässan, Stockholm Sweden, 10-15 Jul 2018. PMLR.
Haibo He and Edwardo A. Garcia. Learning from imbalanced data. IEEE Transactions on Knowledge and Data Engineering. 21 2009.
Ray Jiang, Aldo Pacchiano, Tom Stepleton, Heinrich Jiang, and Silvia Chiappa. Wasserstein fair classification, In Ryan P. Adams and Vibhav Gogate (eds.), Uncertainty Artificial Intelligence, volume 1 115 of Proceedings of Machine Learning Research, pp. 362-872, Tel Aviv, Israel, 22-25 Jul 2020. PMLR.
Bingyi Kang, Saining Xie, Marcus Rohrbach, Zhicheng Yan, Albert Gordo, Jiashi Feng. and Yan- nis Kalantidis. Decoupling representation and classifier for long-tailed recognition. In Eighth International Conference on Learning Representations (ICLR), 2020.
Byungju Kim, Hyunwoo Kim, Kyungsu Kim, Sungjin Kim, and Junmo Kim. Learning not learn: Training deep neural networks with biased data. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
10

---

Published as a conference paper at ICLR 2021
Yi Li and Nuno Vasconcelos. REPAIR: removing representation bias by dataset resampling. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pp. 9572-9581. Computer Vision Foundation / IEEE, 2019.
Zachary Lipton, Julian McAuley, and Alexandra Chouldechova, Does mitigating ML's impact disparity require treatment disparity? In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems 31, Curran Associates, Inc., 2018.
pp. 8125-8135.
Ziwei Liu, Zhongqi Miao, Xiaohang Zhan, Jiayun Wang, Boqing Gong, and Stella X. Yu. Large-scale long-tailed recognition in open world. In IEEE Conference Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pp. 2537-2546. Computer Vision Foundation IEEE, 2019.
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-SNE. Journal of machine learning research, 9NNov):2579-2660 2008
Aditya Krishna Menon, Sadeep Jayasumana, Ankit Singh Rawat, Himanshu Jain, Andreas Veit, and Sanjiv Kumar. Long-tail learning via logit adjustment, 2020.
Mehryar Mohri, Gary Sivek, and Ananda Theertha Suresh. Agnostic federated learning. In Interna- fional Conference on Machine Learning, 2019.
Prectum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, and Ilya Sutskever. Deep double descent: Where bigger models and more data hurt. In International Conference Learning Representations, 2020.
Junhyun Nam, Hyuntak Cha, Sungsoo Ahn, Jacho Lee, and Jinwoo Shin. Learning from failure: Training debiased classifier from biased classifier, 2020.
Behnam Neyshabur, Zhiyuan Li, Srinadh Bhojanapalli, Yann LeCun, and Nathan Srebro. The role of over-parametrizationn in generalization of neural networks. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
S. Sagawa, P. W. Koh, T. B. Hashimoto, and P. Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. In International Conference Learning Representations (ICLR), 2020a.
S. Sagawa, A. Raghunathan, P. W. Koh, and Liang. An investigation of why overparameterization exacerbates spurious correlations. In International Conference Machine Learning (ICML), 2020b.
N. Sohoni, J. Dunnmon, G. Angus, A. Gu, and C. Ré. No subclass left behind: Fine-grained robustness in coarse-grained grained classification problems. In To appear in Conference on Neural Information Processing Systems (NeurIPS), 2020.
Grant Van Horn and Pietro Perona. The devil in the tails: Fine Fine-grainee grained classification in the wild. arXiv preprint arXiv: 1709.0145, 2017.
Dennis Wei, Karthikeyan Natesan Ramamurthy, and Flavio Calmon. Optimized score transformation for fair classification. In Silvia Chiappa and Roberto Calandra (eds.), International Conference Artificial Intelligence and Statistics, volume 108 of Proceedings of Machine Learning Research, pp. 1673-1683, Online, 26-28 Aug 2020. PMLR.
Junfeng Wen, Chun-Nam Yu, and Russell Greiner. Robust learning under uncertain test distributions: Relating covariate shift model misspecification. In Proceedings of the 31st International Conference on International Conference on Machine Learning, 1CML/14 14, pp. 11-631-11-639. JM....rrg 2014.
Han-Jia Ye, Hong- You Chen, De-Chuan Zhan, and Wei-Lun Chao. Identifying and compensating for feature deviation in imbalanced deep learning, 2020.

---

Published as a conference paper at ICLR 2021
Muhammad Bilal Zafar, Isabel Valera, Manuel Gomez Rodriguez, and Krishna P. Gummadi. Fair- ness beyond disparate treatment & disparate impact: Learning classification without disparate mistreatment. In Proceedings of the 26th International Conference World Wide Web, WWW '17, pp. 1171-1880, Republic and Canton Geneva, CHE, 2017. International World Wide Web Conferences Steering Committee. ISBN 9781450349130.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. In 5th International Conference on Learning Representations, ICI. 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017.
Junjie Zhang, Lingqiao Liu, Peng Wang, and Chunhua Shen. To balance not to balance: A simple-yet-effective approach for learning with long-tailed distributions, 2019.
Marvin Zhang, Henrik Marklund, Abhishek Gupta, Sergey Levine, and Chelsea Finn. Adaptive risk minimization: A meta-learning approach for tackling group shift, 2020.
Zhi-Hua Zhou and Xu- Ying Liu. Training sensisvee neural networks with methods addressing the class imbalance problem. IEEE Transactions on Knowledge and Data Engineering (TKDE), 18(1), 2006.
12

---

Published as a conference paper at ICLR 2021
A EXPERIMENTAL SETUP
For the logistic regression experiments, we train the models using LogisticRegresiin package in sklearn with C' = 1/(n A), for number of training examples n and minimal regularisation strength A= 10010--. 16 This employs the liblinear solver under-the-hood, and thus finds an approximate minimiser for the convex empirical risk objective on the entire training sample.
For the ResNet-50 experiments, we initalise the model using a ResNet-50 pre-trained on ImageNet. We train the models using SGD with momentum value of 0.9, We use batch size of 128, weight decay 10-4, and a learning rate of decayed according learning rate of 10-4 for 1000 epochs* on waterbirds, and base learning rate of 10-2 for 50 epochs on celebA. We also employ data augmentation the form of random cropping and flipping. which offers a consistent boost in rare subgroup performance.
cosine schedule. We train with a base
B ADDITIONAL EXPERIMENTS
We present additional experimental results that explore few key hypotheses:
how sensitive are the results to inexact subgroup specification?
how does the precise choice of target label and spurious attribute affect results?
how generalisable are the results to multi-class settings?
B. INEXACT SUBGROUP SPECIFICATION
The post-hoc modification techniques in the body crucially rely on knowledge of the precise sub- group specification of each example. This is unrealistic in practice, where the subgroups may be latent or inexactly specified. Following Sagawa et al. (2020a)Apppendx B], we simulate a setting of inexact specification of the subgroups on celebA. Here, we use as spurious attributes A' = EpearinLLipstide x Eyegl asses x Smi 1 ing x DoubleChin x OvalFace, comprising 32 distinct values. We then learn using the subgroups with respect to the original subgroups y x A'.
A' as input, and then measure performance
As noted in the body, the threshold adjustment technique (THR) is challenging to apply as-is in this setting, as it requires setting 32 distinct thresholds. As suggested in $4.3, we thus apply a simple heuristic of tying the thresholds to the subgroup frequencies, i.e., La(w) = log Pyy = +1 a(x)) log Pyy 1 oz))). This has the effect of implicitly requiring higher model confidences to classify examples into dominant subgroup. This technique is seen to have a worst woost-subgroup error of 16.67%. which is modest increase compared to the 12. 10% obtained when using the exact subgroups y x A'. This illustrates that one can still make useful predictions given imperfect subgroup information.
B.2 CHOICE OF TARGET LABEL. AND SPURIOUS ATTRIBUTE
The results in the body involve data with one or more rare subgroups. Is this rarity the primary factor that influences performance, or does the definition of the subgroups themselves matter? To test this, we consider a variant of the celebA dataset in the body where the target label and sensitive attribute are swapped. In this variant the dataset, we have (male, female] and A = (blond, dark), This exactly preserves the subgroup definitions and their rarity, but fundamentally changes the target label and feature used in training.
Interestingly, this simple modification dramatically improves performance of the baseline: on the rarest subgroup, the error is 13.67%, which is a significant reduction over the 56.9 .4.4% for the original dataset. This indicates that the precise choice of subgroup definition can play non-trivial role in final performance. Intuitively, performance can be hampered when the target variable is spuriously correlated with many features in the training set.
"This large number of epochs. coupled with the small learning rate, endows some stability in the worst subgroup performance. One may however obtain qualitatively similar results with much fewer epochs and larger learning rate.
13

---

Published as a conference paper at ICLR 2021
Nonetheless, even with this improved model, we find that threshold adjustment (THR) further improve performance to 9. 9.11%. The average subgroup errors of both techniques are similar, being 1.29% and 1.28% respectively.
B.3 MULSI-CLASS SETTINGS
The results in the body involve problems with binary labels. To assess the effect of working with multi- class labels, we employ modified version of MNIST based on Goel et al. (2020). Here, one mixes the standard MNIST dataset with samples from corrupted version of MNIST comprising zig-zag images. The zig-zaz images are made to be strongly correlated with the digit parity, so that most odd digits are zig-zz zggzzagge. We then consider subgroups defined by A, where A (normal, zig zag]. Note that we consider y = {0, 1........................ | to illustrate performance a multi-class setting, unlike Goel et al. (2020) who consider y {0, 1) to be the digit parity.
We train a LeNet-5 for 100 epochs using learning rate of 0.0001, momentum 0.9, weight decay 0.05, and batch size 100. Here, ERM achieves worst-subgroup error of 67.11%. Classifier retraining (cRT) based on subsampling all non-minority samples improves this to 74.52%. Similarly, threshold adjustment (THR) based the heuristic of tying the thresholds to the subgroup frequencies (as described in the previous section) achieves 78.95%. This illustrates the potential for post-hoc techniques to also be useful in scenarios other than binary classification.
C VISUALISATION EMBEDDINGS UNDER GROUP-BASED DRO
Figure shows a tSNE visualisation the embeddings learned by a model trained to minimise the group-based DRO objective of Sagawa et al. (2020a). Similar to the results of ERM, there is generally a notable separation of samples from the four subgroups.
60
Negat ve (rare]
Negat ve (dominant)
Positive (sominanc)
Prsitive (rane)
0
-20
20
-40
-30
-40
-20
20
40
-69
-00
-20
20
40
60
(a) waterbirds.
(b) celebA.
Figure 6: Two-dimensional tSNE visualisation of test embeddings produced by overparame- terised model trained to minimise group-based DRO.
D ADDITIONAL EXPERIMENTAL ABLATIONS
We present additional experimental results, highlighting several key points:
the separation of scores between rare and dominant subgroups is consistent across all datasets considered in the paper; however, the score distributions are markedly different on train and test sets, owing to models being overly confident on training samples.
increasing model complexity systematically exacerbates the distribution shift in decision scores.
early stopping has limited effect on the score distributions; even after a single epoch of training, there may be distinction between rare and dominant subgroups' scores.
increased & regularisation strength has a favourable effect on the score distributions, encouraging samples from both rare and dominant subgroups to be correctly classified.
subsampling (per Sagawa et al. (2020a)) has a positive effect on the score distributions, making them almost perfectly align across subgroups.
14

---

Published as a conference paper at ICLR 2021
Poolive dass
2.30
Nogative dass
Podiive dasss
Negazhe cless
Zom
Dontrart subgrrup
Daminaan
5.03
Com
0.96
Rars sucgroup grop
fure a0gguap
Fergosggroup
Rare hargospproup
0.03
0.04
5,03
0.04
0.02
0.21
6.00
-20
Deciiion
20
00
-23
20
0.00
-40
-20
20
40
6.02
-20
25
(a) Synthetic.
(b) Waterbirds (logistic).
Pesitive class
Negattee class
Peotive
Negustev dass
6.00
cuiiiaattuuggrep Rore subggra maus
0.06
Genirart soboraa Ram caggmau
0.22
Goninore aibon 4p Ram eatgmas
5.0
Form euogmau
Dom acheropp
0.30
6.00
4.15
0.06
0.10
2 0.04
0:34
6.00
0.03
0.02
0:22
6.08
-20
-33
32
0.00
-40
-20
0.30
-12
10
0:00
-30
-20
-13
10
Doc
Dedisio
(c) Waterbirds (ResNet).
(d) CelebA.
Figure 7: Histograms of model scores test samples various datasets, comprising two labels with two sub-groups each. In general, there are differences in the distributions for the rare and dominant subgroup scores, with the former often lying on the incorrect side of the decision boundary.
increasing the fraction of majority samples has a deleterious effect on overall performance; however, even extreme levels of imbalance, the score distribution for rare samples may be shifted to correct for bias.
D.1 HISTOGRAM OF TRAIN AND TEST SCORES
Figure 7 plots histograms of test scores for all datasets considered in this paper. We consistently find that there is a separation between the scores for rare and dominant subgroups.
We see similar behaviour on training scores in Figure 8. However, note the vastly different scale, owing the model being more confident in its predictions for these samples. In general, while there are differences in the distributions for the rare and dominant subgroup scores, nearly all such scores are on the correct side of the decision boundary. This expected, since overparameterised models perfectly fit the training data, and thus correctly classify all samples. The ability of these models to nonetheless produce meaningful results on test samples is owing to their inductive bias.
D.2 IMPACT OF MODEL COMPLEXITY ON SCORES
Figure 9 shows model scores on test samples synth number of projection features is varied. We see that as the model complexity increases, there is a steady increase in the separation of decision scores between the rare and dominant subgroups for label. This is in keeping with overparameterisatioon exacerbating worst-subgroup error: as the decision scores have more pronounced separation, using a default classification threshold will lead to significantly worse performance.
D.3 IMPACT OF EARLY STOPPING ON SCORES
Figures 10 and 11 shows the evolution of model scores on test samples on the CelebA and Waterbirds datasets. Here, the distinction between the scores amongst subgroups of the positive class is visible even after early stopping. With increased training epochs, there is a systematic shift of the negative scores, as the network becomes increasingly confident on them.
D.4 IMPACT OF REGULARISATION ON SCORES
Figures 12 and 13 shows how model scores on test samples vary as we modify the strength of regularisation. Increasing the strength is seen to favourably impact the scores the negative class, for both the dominant and rare subgroups. This provides another perspective on why regularisation can be somewhat effective at improving worst-subgroup performance.
15

---

Published as a conference paper at ICLR 2021
Pus tie dass
Negotive dass Rark suogroup
Postive dass
Nogative dass
Eontrartsudguu Rare adgisss
Corn
Rare suggnup
Dor inain sulgrra
2.30
Cerrinett irnn
Rare subgrous
2.23
9:15
6.5
0.2
6.3
10
33
40
0.0
-20
-42
-20
-23
-10
0.6
10
20
30
40
20
0.00
-50
-00
-33
-22
-10
(a) Synthttic.
(b) Waterbirds (logistic).
Feative class
Negative diass
tive cass
Wegative clsss
lemiosnt cungrus
bonfeeee eingiaa
6.06
6.4
kam shymup
0.30
kane a.homap
6.75
kare sungroup
6:00
8.15
6.
5.03
0.55
620
1.02
-20
-12
0.00 -32
0.00
-30
Dec
Dectoon
Dechilon
(c) Waterbirds (ResNet).
(d) CelebA
Figure 8: Histograms of model scores on train samples on various datasets, comprising two labels with two sub-group groups each. In general, while there are differences in the distributions for the rare and dominant subgroup scores, nearly all such scores are on the correct side of the decision boundary.
Peskive dass Comtaatsndgrapp Rara subgreus
Nogazies dats Darinentsuzgroo
Positive dass
Noyative class
3.2
0.35
0.25
Dom nant
3.0
Baro stbg 194
0.25
Rarssaba wag.agapppppp
Rara suuggropp
2,8
1.00
0.22
0.33
8.3
0.33
8.4
0.32
6.2
0.05
6:3
0.00
-1
0.00
0.00
-22
10
(b)
10
Postive dass
Nogative dass
Postive dass
Wegad we Class
Zom nunt sdbgrou
Contrunt subgoup
Daminaan
0.24
har supgroup
Rar subg tup
0.56
Mark audgrnp
fare Maroosoproup
Dendoy
Denstry
Denetr
Denicy
B-42
0.01
2.00
0.02
0.02
8.00
-50
2.00
-75
-30
-25
25
0.00
-22
Dediion
23
6.00
-40
-20
20
(c) 10
(8) 10
Figure 9: Histograms of model scores on test samples on synth as number of projection features m is varied. We see that as the model complexity increases, there is a steady increase in the separation of decision scores between the rare and dominant subgroups for label. This is in keeping with overparameterisation exacerbating worst-subgroup error.
D.5 IMPACT OF SUBSAMPLING ON SCORES
Figure 14 shows histograms of model scores test samples on synthetic dataset, with and without subsampling per Sagawa et al. (2020a). Subsampling is seen to make the scores equitable across the subgroups, which provides another perspective on why this technique can effectively mitigate a bias against rare subgroups.
D.6 IMPACT OF FRACTION OF RARE SUBGROUPS ON SCORES
The synth dataset involves a parameter Pdom its construction, which controls the relative number of samples belonging to the dominant class. By default, following Sagawa et al. (2020b), we use Paom= 0.90. Figure 15 shows how the tradeoff is affected by changing Pdom- As Paom incrases,,,,, as expected, it is more challenging to minimise the wurst-subgroup error at large m. Figure 16 further shows how the test scores for each subgroup are affected by the choice of Paom As Paom increasss the rare subgroup scores are seen to significantly diverge from the dominant one.
16

---

Published as a conference paper at ICLR 2021
Tesiiive C haa
Negative dass
Teative das
Vegazie clus
0.90
schopop
bonkour sigrau
0.25
Rare sungrup
Konn sahemop
Rum suggrsp
Raze sahereup
6.15
0.10
0:00
0.54
6:85
0.22
0:00
-20
-15
Decidor
13
20
0.00
-30
-27
-10
10
0.00
-27
-10
10
20
0.83
-30
-20
Sechian
=10
0
23
(a) I spoch.
(b) 5 apachs.
Negstive
0.500
Dorenam subgrnup
0.25
Rat autggaa
6.15
Arssipppup
6.15
0.10
00
006
6.28
0.64
0.00
-10
0.000
0.00
-10
-37
-10 Dods Score
Decolar-son Score
Dectzon Score
Dedsion
(c) 10 epochs.
(d) 50 epochs.
Figure 10: Evolution of histograms of model scores on test samples on CelebA. The distinction between the scores amongst subgroups of the positive class is visible even after early stopping. With increased training epochs, there is a systematic shift of the negative scores, as the network becomes increasingly confident on them.
Ton Chee
Vegative clece
Pos
Negativee
5.3
34
0.25
Saterup
Kare
6.20
8,15
6.26
6.1
311
0.15
6.05
6.2
3.0
0.00
6:30
-15
Decalon score
Decolon 300TE
Dediicn score
Decalton acore
I epoch.
(h) 10 epochs.
Tesifive c bux
Negative dass
Pootive class
Negutive cluss
Rain cubgrnus
6.550
0.323
Inwiivvr Mare-silinee
0.4
Rye arayea
6:38
Rape stherop
2626
2.15
6.10
A.50
6:95
0.25
4.1
0.22
6:00
-12
0,000
-20
-15
-13
-5
3.0 -13
6:38
-20
-15
-10
(4) 100 epochs.
(d) 1000 epochs.
Figure 11: Evolution of histograms of model scores test samples on waterbirds.
17

---

Published as a conference paper at ICLR 2021
Poxltec
Negabbre
Domtram subgruu
5.125
Domiramt suhgruu
Rune sutggrua
6.15
tarssspgroop
nant shhgoup
5.190
0,10
50055
0,055
1.03
0.000
0.55
1.23
0.25
Dectdon SCOTe
Dection score
Desicnn score
Dedi score
(a) A = 10
(b) A
10 2
Positive class
Negalive class
0.10
Dominant subgroup
Dominant subgroup Rare subgroup
Rare subgroup
0.20
0.08
Dendior
0.08
Density
0.15
0.04
0.10
0.02
0.05
0.00
0.00
-20
-10
Decision score
Decision score
(c) A
= 10
Figure 12: Histograms of model scores test samples celebλ with various strenghts of regulari- sation. Increasing the strength is seen to favourably impact the scores on the negative class, for both the dominant and rare subgroups.
Postive dass supgrrup
Nogative dass
Pas Diet dass
6:06
Mojative cass
sakgroup
0.10
Farn sukgroup
2.00
6.95
6.15
3.00
0.56
3.04
6:02
6.85
1.02
6.22
6.01
6.30
-20
10
3.00
-42
-20
0.00
-10
10
-40
-23 Decolor SCOTG
Deciloo Scora
Doctsin Scorc
Dedaion Score
A
10
10 2
Positive class
Ne gativive class
0.150
Dom hann subgroup
0.12
Dominant subgroup
Rare subgroup
Rare subgroup
0.125
0.10
Density
0.100
0.08
0.075
0.06
0.050
0.04
0.025
0.02
0.000
0.00
10 Decision score
40
-20
20
Decision
(c) A 10"
Figure 13: Histograms of model scores on test samples on waterbirds with various strengths of regularisation. Increasing the strength seen to favourably impact the scores on the negative class, for both the dominant and rare subgroups.
Poctiive dees
Negative
Pox
Negativee Cank
20m nort subgrou susgrrop
Doreramt sutgreup
0.08
Damiiann Suggroup
6.10
Dor nont subgroup sungreup
6.36
Kare 2by00.
Kare sutorop
6.00
8.00
0.04
3.00
0.24
6.02
2.00
6.02
0.00
Decidon
2.00
23
20
0.50
-0
-10
Dediion
15
20
0.00
-20
10
25
(8) Baseline.
(b) Baseline subsampling.
Figure 14: Histograms of model scores on test samples on synth, with and without subsampling per Sagawa et al. (2020a). Subsampling is seen to make the scores equitable across the subgroups.
18

---

Published as a conference paper at ICLR 2021
Average train
Average train
60
50
Average test
Average test
Worst train
40
Worst train
Error
40
Worst test
Error
Worst test
20
10
log m
log m
(a) Baseline Pmaj = 0.5.
(b) Threshold correction Pmaj = 0.5.
Average train
60
60
Average test
Worst train
Average train
Error
40
Average test
Error
40
Worst test
Worst train
Worst test
20
1
2
3
1
3
log m
log m
(c) Baseline Pmaj 0.9.
(d) Threshold correction Pmaj 0.9.
80
Average train
60
Average test
60
Average train
50
Worst train
Worst test
Average test
Error
40
Worst train
30
Worst test
20
20
10
3
log m
log m
(e) Baseline Pmaj 0.95.
(f) Threshold correction Pmaj = 0.95.
Figure 15: Performance of baseline and threshold correction synth. We vary the fraction of majority samples Pmaj- This seen to adversely affect the performance ERM. However, threshold correction effectively reduce this error, albeit at the expense of higher variance.
19

---

Published as a conference paper at ICLR 2021
6.25
Postive diss
Negsttan dees
Poctlan CIKE
6.05
Megative case
0.24
fure suogroup
0.04
fart
6.00
8
3.00
6.42
1.02
0.53
6.00
-20
23
3.00
-25
6.00
6:00
-25
SCore
score
Score
Pinaj
0.50.
(b) Pana
0.73
FenBliv rises
Reggrtive cloes
Foxttive
Megertie Cives
Foro subgroup
Domenant
Farn sugrrup
150
Conerentsu Rara subgreap
0.12
Dominut suboe
Rare sulgreup
0 1.5
100
0.00
6:02
0.04
3.00
3.005
0.63
6.30
Deci
20
0.00
-12
-23
Desteon score
Dec 3101
-02
Dechior
-20
(c) Pooaj
0.90.
(d) Proaj
0.95
Figure 16: Histograms of model scores on test samples on synth, for various values of fraction of majority samples Proo As this fraction becomes larger, the rare subgroups see progressive shift in their scores compared to the dominant ones.
20


Reviews:
[{"review_id": "wk3b5CrwFh", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThe paper builds upon prior work that shows that overparameterized networks learned by ERM can have poor worst-case performance over pre-defined groups. Specifically, the paper demonstrates that this result is not necessarily due to overparameterized learning poor representations for rare subgroups, but rather mis-calibration in the classification layer that can be addressed with two simple correct techniques: thresholding and re-training the classification layer. They show improvements over ERM in worst-case subgroup error.  \n\nStrengths:  \n1. The paper is very well-written and easy to follow.\n2. The paper provides a better understanding of worst-case generalization in overparamaterized models by isolating the issue to the classification layer, which can help machine learning practitioners better understanding how they can address the issue of poor worst-group performance. \n\n\nWeaknesses:  \n1. The scope of the work seems largely limited to the set-up from Sagawa a, b. Given that all three datasets are simplified/synthetic (only one attribute, max. 4 subgroups), it would have been great to see how this paper's analysis applied to more complex settings.   \n2. It is not immediately clear why we would not expect classifier retraining/threshold correction (which as the author notes, are standard techniques) to work for the overparameterized setting? The richness of learned representations is well known, so in some sense the findings are not too surprising, especially given the simple (e.g. binary label) settings that make post-hoc corrections less complex. Could the authors better explain why overparameterization reduces our expectations on the effectiveness of these post-hoc procedures? \n3. The requirement of knowing the groups a priori seems rather significant. While one of their main cited works (Sagawa a) seems to have touched on group attribute mis-specification, this was not explored here -- how does having imprecise knowledge of the groups effect performance when using threshold correction or classifier retraining? \n4. I have trouble fully understanding Figure 2 without a sense of what \"insufficient\" or poor representations would look like in a tSNE visualization.  \n\n\nRecommendation:  \nI recommend acceptance. While I remain concerned about the limited scope of the experiments, I believe the paper adds valuable insights to the overall important topic of robustness / worst-case generalization.  \n\nQuestions:\n1. Does the nature of the attribute (land vs water, or hair color) have any effect on the observed poor worst-group performance, or are the results are mainly due to the fact that some groups are rarer than others? For example, would the authors expect similar results if y=male,female and A=blond,dark for celebA?   \n2. Did the authors visualize the embeddings of models trained with DRO, to see whether there is any improvement in the learned representations ability to distinguish subgroups? ", "extra_scores": {}, "extra_text": {}}, {"review_id": "XFTo9ZrMlR0", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper is concerned with potential improvements to the worst-case\n(mainly minority class/group) generalizations in over-parametrized\nneural networks through post-hoc corrections. The authors demonstrate\nthe problem and the suggested corrections (that were used in previous\nliterature) on one artificial classification task as well as two\nimage classification tasks. The paper shows that post-hoc corrections\nmay improve the worst-subgroup scores similar to an earlier\nstate-of-the-art system that modifies the learning objective.\n\nThis topic is interesting. The paper is in general written well, and\ndemonstrates the problem convincingly. The post-hoc fix solution\nsuggested also seem to be performing reasonably well on the problems /\ndata sets used in the study.\n\nThat being said, it feels the study/paper builds on a few earlier\nstudies heavily, and I am not fully convinced that there is enough new\nfindings in the present paper to warrant publication in ICLR.\n\nI also have a few minor notes/suggestions:\n\n- The findings presented in Figure 4 is interesting (mainly the fact\n  that the overparametrization seem to be improving the worst-group\n  performance with threshold tuning). It would be interesting to see\n  more investigation/discussion of what could be the underlying\n  reason.\n\n- Page 2 (middle): \"also common the fairness literature\" ->   \"also common in the fairness literature\" \n- Figure 2 is not very readable. If increasing size/scale is not an\n  option due to limited space, taking legend out the figures may also\n  improve it. Colorblind friendly colors may also be a good idea.\n- There are case (normalization) issues in the references:\n  \"ml\", \"t-sne\" (not exhaustive, a through check is\n  recommended).\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "PUGFzGK_Yvu", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies how to improve the worst-case subgroup error in overparameterized models using two simple post-hoc processing techniques: (1) learning a new linear classification layer of a network, or (2) learning new per-group threshold on the logits. The efficacy of these techniques is evaluated on three synthetic datasets.\n\nPros:\n- The paper studies a timely topic--generalization in overparameterized models-- with some applications to fairness.\n- The algorithms presented are very simple and avoid the overhead of more complicated algorithms like DRO or subsampling strategies.\n- In the experiments presented, both the thresholding method and the \"learn a new classification layer\" method significantly improve worst-case group error over ERM and are competitive with DRO.\n- The paper is well-written and pleasant to read.\n\nCons:\n- The experimental evaluation is somewhat limited, focusing on three synthetic datasets and two models (ResNet/Logistic Regression). It remains an open question how well the presented technique work more generally both on different model families and on \"real\" distributions.\n\nMinor:\n- There appears to be a discrepancy between the text and the main results table, Table 1. The text says \"For example, on celebA, THR reduces the worst-subgroup error from 56.94% to 31.11%\", but the table shows THR gets a worst-subgroup error of 12.10. What's going on there?\n\n==============\n\nUpdate after rebuttal:\nThank you for clarifying the numbers in Table 1 should match the main text. I enjoyed this paper, and I'm keeping my score unchanged.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper studies how to improve the worst-case subgroup error in overparameterized models using two simple post-hoc processing techniques. All reviewers were positive about the paper, though R5 questioned the novelty of the paper which built heavily on a few previous papers (in particular, it builds heavily on Sagawa et al. 2020a,b). The AC is satisfied with the authors`' response clarifying the novelty. Given that this topic is quite timely and of interest to the ICLR community, and that this paper presented a clean investigation on it, the AC recommends acceptance.
Decision: Accept (Poster)