Selective Classification Can Magnify Disparities Across Groups
Authors: Erik Jones
Shiori Sagawa
Pang Wei Koh
Ananya Kumar
Percy Liang
Venue: ICLR 2021 Poster (2021)
Keywords: selective classification
group disparities
log-concavity
robustness

Abstract: Selective classification, in which models can abstain on uncertain predictions, is a natural approach to improving accuracy in settings where errors are costly but abstentions are manageable. In this paper, we find that while selective classification can improve average accuracies, it can simultaneously magnify existing accuracy disparities between various groups within a population, especially in the presence of spurious correlations. We observe this behavior consistently across five vision and NLP datasets. Surprisingly, increasing abstentions can even decrease accuracies on some groups. To better understand this phenomenon, we study the margin distribution, which captures the model’s confidences over all predictions. For symmetric margin distributions, we prove that whether selective classification monotonically improves or worsens accuracy is fully determined by the accuracy at full coverage (i.e., without any abstentions) and whether the distribution satisfies a property we call left-log-concavity. Our analysis also shows that selective classification tends to magnify full-coverage accuracy disparities. Motivated by our analysis, we train distributionally-robust models that achieve similar full-coverage accuracies across groups and show that selective classification uniformly improves each group on these models. Altogether, our results suggest that selective classification should be used with care and underscore the importance of training models to perform equally well across groups at full coverage.

Published as a conference paper at ICLR 2021
SELECTIVE CLASSIFICATION CAN MAGNIFY DISPARITIES ACROSS GROUPS
Erik Jones; Shiori Sagawa; Pang Wei Koh, Ananya Kumar & Percy Liang Department of Computer Science, Stanford University (erjones, ssagawa, pangwei, ananya, p lang) stanfoorrdd...ddddd.........
ABSTRACT
Selective classification, in which models can abstain on uncertain predictions, is natural approach improving accuracy in settings where errors are costly but abstentions are manageable. In this paper, we find that while selective classifi- cation can improve average accuracies, it can simultaneously magnify existing accuracy disparities between various groups within a population, especially in the presence of spurious correlations. We observe this behavior consistently across five vision and NLP datasets. Surprisingly, increasing abstentions can even de- crease accuracies on some groups. To better understand this phenomenon, we study the margin distribution, which captures the model's confidences over all predictions. For symmetric margin distributions, we prove that whether selective classification monotonically improves or worsens accuracy is fully determined by the accuracy at full coverage (i.e., without any abstentions) and whether the distri- bution satisfies property we call left-log-coneavityy Our analysis also shows that selective classification tends to magnify full-coverage accuracy disparities. Moti- vated by our analysis, we train distributionally-robus models that achieve similar full-coverage accuracies across groups and show that selective classification uni- formly improves each group on these models. Altogether, our results suggest that selective classification should be used with care and underscore the importance of training models to perform equally well across groups at full coverage.
INTRODUCTION
Selective classification, in which models make predictions only when their confidence is above threshold, is a natural approach when errors are costly but abstentions are manageable. For exam- ple, in medical and criminal justice applications. model mistakes have serious consequences, whereas abstentions can be handled by backing off to the appropriate human experts. Prior work has shown that, across a broad array of applications, more confident predictions tend to be more accu- rate (Hanczar& & Dougherty 2008 Yu et al. 1 Toplak et al. 20144 Mozannar & Sontag 2020 Kamath et 2020). By varying the confidence threshold, we can select an appropriate trade-off between the abstention rate and the (selective) accuracy of the predictions made.
In this paper, we report cautionary finding: while selective classification improves average ac- curacy, it can magnify existing accuracy disparities between various groups within a population, especially the presence of spurious correlations. We observe this behavior across five vision and NIP datasets and two popular selective classification methods: softmax response (Cordella et al., 1995 Geifman & EI- Yaniv] 2017) and Monte Carlo dropout (Gal & Ghahramani] (0166). Surpris- ingly, we find that increasing the abstention rate can even decrease accuracies on the groups that have lower accuracies at full coverage: on those groups, the models are not only wrong more frequently, but their confidence can actually be anticorrelated with whether they correct. Even datasets where selective classification improves accuracies across all groups, we find that it preferentially helps groups that already have high accuracies, further widening group disparities.
These group disparities are especially problematic in the same high-stakes areas where we might want to deploy selective classification, like medicine and criminal justice; there, poor performance on particular groups is already a significant issue (Chen et al. 2020 Hill 2020). For example, we study a variant of CheXpert (Irvin et al. 2019), where the task is predict if patient has pleural
"Equal contribution

---

Published as a conference paper at ICLR 2021
Incorrect
Abstain
Correct
Selective accuracy
1.0
0.5
Density
0.6
0.4
Average
0.2
Worst group
0.0
-2
0.0
0.4 0.6
1.0
Margin
Average coverage
Figure I: A selective classifier (9. é) makes prediction (z)) point if its confidence č(z) in that prediction is larger than or equal to some threshold т. We assume the data comprises different groups each with their own data distribution, and that these group identities are not available to the selective classifier. In this figure, we show a classifier with low accuracy on a particular group (red), but high overall accuracy (blue). Left: The margin distributions overall (blue) and on the red group. The margin is defined as ê(x) on correct predictions (i(z) = y) and -((x)) otherwise. For a threshold r. the selective classifier thus incorrect on points with margin S -7; abstains on points with margin between -T and т: and is correct on points with margin 2 Right: By varying т, we can plot the accuracy-cvveage curve, where the coverage is the proportion of predicted points. As coverage decreases, the average (selective) accuracy increases, but the worst-group accuracy decreases. The black dots correspond to the threshold T 1, which shaded on the left.
effusion (fluid around the lung) from chest x-ray. As these are commonly treated with chest tubes, models can latch onto this spurious correlation and fail on the group of patients with pleural effusion but not chest tubes other support devices. However, this group is the most clinically relevant it comprises potentially untreated and undiagnosed patients Rayner et al. 2020).
To better understand why selective classification analyze the margin distribution, which captures the model's confidences across all predictions and determines which examples it abstains on at each threshold (Figure []]. We prove that when the margin distribution is symmetric, whether selective classification monotonically improves or wors- ens accuracy is fully determined by the accuracy at full coverage (i.e., without any abstentions) and whether the distribution satisfies property we call left-log-concavity. To our knowledge, this is the first work to characterize whether selective classification (monotonically) helps or hurts accuracy in terms of the margin distribution, and to compare its relative effects on different groups.
worsen accuracy and magnify disparities,
Our analysis shows that selective classification tends magnify accuracy disparities that are present at full coverage. Motivated by our analysis, we find that selective classification on group DRO models (Sagawa et al., 2020) 2020)-which -which achieve similar accuracies across groups at full coverage by using group annotations during training uniformlly improves group accuracies at lower coverages, substantially mitigating the disparities observed on standard models that are instead optimized for average accuracy. This approach is not silver bullet: it relies knowing group identities during training, which are not always available (Hashimoto et al.) (018). However, these results illustrate that closing disparities full coverage can also mitigate disparities due selective classification.
RELATED WORK
Selective classification.
Abstaining when the model is uncertain is a classic idea (Chow) 1957 Hell- man] 1970), and uncertainty estimation is an active area of research, from the popular approach of us-
ing softmax probabilities (Geifman & BJ- Ellyynivy 2017) to more sophisticated methods using dropout
GGal & Ghahramani 2016), ensembles (Lakshminarayanal et al. (017). or training snapshots (Geif-
man et al. (018)) Others incorporate abstention into model tramiing (Bartlett & Wegkamp 2008
Geifman & EI- Yaniv 2019; Feng et al., (2019) and learn to abstain on examples human experts are
more likely to get correct (Raghu et al. 2019 Mozannar Sontag 2020 De al. 2020). Selec-
tive classification can also improve out- out-of-distribution accuracy (Pimentel et al. [014]] Hendrycks &
Gimpel, 2017 Liang et al. 20188 Ovadia et al., 2019 Kamath et al. 2020). On the theoretical side, early work characterized optimal abstention rules given well-specified models (Chow 1970 Hell- man & Raviv 1970). with more recent work on learning with perfect precision (EI- (El-Yaniy & Wieneer 2010|Kaham et al. 2016) and guaranteed risk (Geifman & El- Yaniy (017). We build on this liter- ature by establishing general conditions on the margin distribution for when selective classification helps, and importantly, by showing that it can magnify group disparities.
2

---

Published as a conference paper at ICLR 2021
Group disparities. The problem of models performing poorly some groups of data has been widely reported Hovy & Søgaard (2015): Blodgett et al. (2016); Corbett- Davies et al. (2017); arise when models latch onto spu- rious correlations, e.g., demographics (Buolamwini & Gebru 2018 Borkan et al. 2019). image backgrounds (Ribeiro et al., 2016 Xiao et al., (020), spurious clinical variables (Badgeley al.
Tatman ) Hashimoto et al. (2018) ). These disparities
2019
Oakden-Rayner et al. 2020). or linguistic artifacts (Gururangan et al. [2018] McCoy et al,,
2019
These disparities have implications for model robustness and equity, and mitigating them is
an important open challenge (Dwork et al.] 2012: Hardt et al., 2016)] Kleinberg et al., 2017 Duchi
al., 2019) Sagawa et al. 2020j. Our work shows that selective classification can exacerbate this
problem and must therefore be used with care.
3 SETUP
A selective classifier takes in input E X and either predicts label y Y abstains. We study standard confidence-baat based selective classifiers (Ú. é), where y : X + outputs a prediction and è X - R., outputs the model's confidence in that prediction. The selective classifier abstains on I whenever its confidence ë(x) is below some threshold r and predicts j(x) otherwise.
Data and training. We consider data distribution D over X' y XG where = (1, 2,,,,...,A k) cor- responds to a group variable that is unobserved by the model. We study the common setting where the model ý trained under full coverage (i.e., without taking into account any abstentions) and the confidence function is then derived from the trained model, as described in the next paragraph. We will primarily consider models ý trained by empirical risk minimization (i.e., to minimize the aver- age training loss); in that setting, the group the group DRO training algorithm (Sagawa et al., 2020). which observes g at training time only. In both cases, is not observed at test time and the model thus does not take in g. This is common assumption: e.g., we might want to ensure that a face recognition model has equal accuracies across genders, but the model only sees the photograph (x) and not the gender (g).
G is never observed. In Section we will consider
Confidence. We will primarily consider softmax response (SR) selective classifiers, which take ((x) to be the normalized logit of the predicted class. Formally, we consider models that estimate ply z) (e.g., through a softmax) and predict ((xx) = arg max,ey p(y zz), with the corresponding probability estimate p(ğ(x) r). For binary classifiers, we define the confidence é(x)
p(0(z) x))
c(x) = log 2 1 (()x(((x) xx)
(1)
This corresponds to a confidence of 8(x) = 0 when f(((x) x) = 0.5, i.e., the classifier is com- pletely unsure of its prediction. We generalize this notion to multi-class classifiers in Section A. Softmax response is a popular technique applicable to neural networks and has been shown to im- prove average accuracies offer alternative ways of computing ((x); in Appendix B. we also run experiments where ((x) is obtained via Monte Carlo (MC) dropout (Gal & Ghahramani 2016), with similar results.
a range of applications (Geifman & Yaniv 2017). Other methods
Metrics. The performance of a selective classifier at a threshold r is typically measured by its average (selective) accuracy on its predicted points, P()(x) = y e(x) 2 7]. and its average coverage, i.e., the fraction of predicted points IP(((x) 2 r]. We call the average accuracy at threshold O the full-coverage accuracy, which corresponds the standard notion of accuracy without any abstentions. We always use the term accuracy w.r.t. some threshold 2 0: where appropriate, we emphasize this by calling selective accuracy, but we use these terms interchangeably in this paper.
Following convention, we evaluate models by varying and tracing out the accuracy-coverage curve (E- & Wiener 2010). As Figure illustratts this eurve is fully determined by the distribution of the margin, which is c(r) on correct predictions (i(z) y) and -ê(x) otherwise. We are also interested in evaluating performance on each group. For a group g E G, we compute its group (selective) accuracy by conditioning on the group, PfO(x) y g, Ō(z) T]. and we define group coverage analogously as PP((x)) 2 T g]. We pay particular attention to the worst group under the model, i.e., the group argmin, y P()()) = y I g] with the lowest accuracy at full coverage. In our setting, we only use group information to evaluate the model, which does not observe at training or test time; we relax this later when studying distributionally-rouus models that use g during training.
3

---

Published as a conference paper at ICLR 2021
Dataset
Modality
examples
Prediction task
Spurious attributes A
CelchA
Photos
202,599
Hair color
Gender
CivilComments
Text
448,000
Toxicity
Mention of Christianity
Waterbirds
Photos
11,788
Waterbird or landbird
Water land background
CheXpert-device
X-rays
156,848
Pleural effusion
Has support device
MultiNLI
Sentence pairs
412,349
Entailment
Presence of negation words
Table 1: We study these datasets from Liu et al. (0015): Borkan et al. (2019); Sagawa et al. (2020);
Irvin et al. (2019;; Williams et al. (2018) respectively. For each dataset, we form group for each attribute a E AA, and evaluate the accuracy of selective classifiers on average and on each group. Dataset details in Appendix C.I
combination of label E y and
Datasets. We consider five datasets [ on which prior work has shown that models latch onto spurious correlations, thereby performing well on average but poorly on the groups of data where the spurious correlation does not hold up. Following (Sagwaa et al. (2020), we define set of labels Y as well as a set of attributes A that are spuriously correlated with the labels, and then form one group for each (y, a) E J A. For example, in the pleural effusion example from Section | one group would be patients with pleural effusion (y = 1) but no support devices (a 0). Each dataset has | A|| 2, except MultiNLI, which has | More dataset details Appendix C.I
EVALUATING SELECTIVE CLASSIFICATION ON GROUPS
We start by investigating how selective classification affects group (selective) accuracies across the five datasets in Table [ mize average training Joss, using ResNet50 for CelebA and Waterbirds; DenseNet121 for CheXpert- device; and BERT for CivilComments and MultiNLI II. Details are in Appendix C max response (SR) selective classífiers, but show similar results for MC-dropout in Appendix B.
We train standard models with empirical risk minimization, i.e., to mini-
We focus on soft-
Accuracy-coverage curves. Figure 2 shows group ceuurac--oveeage curves for each dataset, with the average in blue, worst group red, and other groups gray. On all datasets, average accuracies improve as coverage decreases. However, the worst-group curves fall into three categories:
1. Decreasing. Strikingly, on CelebA, worst-group accuracy decreases with coverage: the more confident the model on worst-group points, the more likely it is incorrect.
2. Mixed. On Waterbirds, CheXpert-device, and CivilComments, as coverage decreases, worst- group accuracy sometimes increases (though not by much, except noisy, low coverages) and sometimes decreases.
3. Slowly increasing. On MultiNLI, coverage decreases, worst-group accuracy consistently im- proves but more slowly than other groups: from full to 50% average coverage, worst-group accu- racy goes from 65% to 75% while the second-to-wors group accuracy goes from 77% to 95%.
CelebA
Waterbirds
Chexpert-device
CivilComments
MuitiNLI
Group coverage Group sel. accuracy
0.5
Worst group Average
0.25
Other groups
0-
1
0.5
0.25
0
0.5
0.5
0.5 Average coverage
0.5 Average coverage Average coverage
0.5
Average coverage
Average coverage
Figure 2: Accuracy (top) and coverage (bottom) for each group, as a function of the average cover- age. Each average coverage corresponds threshold 7. The red lines represent the worst group. At low coverages, accuracy estimates are noisy as only few predictions are made.
4

---

Published as a conference paper at ICLR 2021
CelebA
CivilComments
Waterbirds
CheXpert-device
MultiNLI
Selective accuracy
1.0
0.6
Worst group Group Group-agnostic
0.2
Average
0.0
0.25 0.5 0.75 0.25 D.5 0.75 0.25 0.5 0.75 Coverage
0.25 0.5 0.75 Coverage
0.25 0.5 0.75 Coverage
Coverage
Coverage
Figure 3: SR selective classifiers (solid line) substantially underperform their group-agnostic ref- erences (dotted line) on the worst group, which are in turn far behind the best-case Robin Hood references (dashed line). By construction, these share the same average accuracy-coverage curves (blue line). Similar results for MC-dropout are in Figure 7
Group-agnostic and Robin Hood references. The results above show that even when selective classification helping the worst group, seems help other groups more. We formalize this notion by comparing the selective classifier to a matching group-agnostic reference that is derived from it and that tries to abstain equally across groups. At each threshold, group-agnostic ref- erence makes the same numbers of correct and incorrect predictions its corresponding selective classifier, but distributes these predictions uniformly at random across points without regard to group identities (Algorithm I By construction, it has an identical average accuracy-coverage curve as its corresponding selective classifier, but can differ on the group accuracies. We show in Appendix A.22 that it satisfies equalized odds (Hardt et al. 2016) w.r.t. which points it predicts abstains on.
The group-are agnoosiic reference distributes abstentions equally across groups; from the perspective of closing disparities between groups, this is the least that might hope for. Idcally, selective clas- sification would preferentially increase worst-group accuracy until it matches the other groups. We capture this optimistic scenario by constructing, for given selective classifier, corresponding Robin Hood reference which, as above, also makes the same number of correct and incorrect pre- dictions (see Algorithm|2 P in Appendix A..)). However, unlike the group-agnostic reference, for the Robin Hood reference, the correct predictions are not chosen uniformly at random; instead, we pri- oritize picking them from the worst group, then the second worst group, etc. Likewise, we prioritize picking the incorrect predictions from the best group, then the second best group, etc. This results worst-group accuracy rapidly increasing the cost of the best group.
Both the group-agnostic and the Robin Hood references are not algorithms that we could implement in practice without already knowing all of the groups and labels. They act instead as references: selective classifiers that preferentially benefit the worst group would have worst-group accuracy- coverage curves that lie between the group-agnostic and Robin Hood curves. Unfortunately, Figure shows that SR selective classifiers substantially underperform even their group aaguosag counter- parts: they disproportionately help groups that already have higher accuracies, further exacerbating the disparities between groups. We show similar results for MC-dropout in Section B.
Algorithm 1: Group-agnoste reference for (g, é) at threshold
Input: Selective classifier (ū. é), threshold T, test data D
Output: The sets of correct predictions C D and incorrect predictions -edh C D that the group-agnostic reference for (ū. é) makes at threshold т.
Let C, be the set of all examples that (Ú. correctly predicts at threshold
C, {(ar, y. E D ((x) and ö(x) 2 T).
(2)
Sample subset C of size |C,| uniformly at random from Co, which is the of all examples that ý would have predicted correctly at full coverage.
Let I, be the analogous set of incorrect predictions at 7:
I. ((1,(11..0 9) D
((x) and ē(x) r}.
Sample subset _ of size |,| uniformly at random from Zo- Return C and Z Since | = |c- and | = |I-|. the group-agnostic reference makes
the same numbers of correct and incorrect predictions as (j. é), but in a group-agnostic way.
5

---

Published as a conference paper at ICLR 2021
CelebA
Waterbirds
CheXpert-device
CivilComments
MultiNLi
Worst-group Average density
density
10
-10
-2.5
-2.5
Margin
Margin
Margin
Margin
Margin
Figure 4: Margin distributions average (top) and on the worst group (bottom). Positive (negative) margins correspond to correct (incorrect) predictions, and we abstain on points with margins closest to zero first. The worst groups have disproportionately many confident but incorrect examples.
5 ANALYSIS: MARGIN DISTRIBUTIONS AND ACCURACY-COVERAL CURVES
We saw in the previous section that while selective classification typically increases average (selec- tive) accuracy, can either increase or decrease worst-group accuracy. We now turn theoretical analysis of this behavior. Specifically, its two extremes: when accuracy monotonically increases, or decreases, as a function of coverage. While these extremes do not fully explain the empirical phenomena, e.g., why worst-group accuracy sometimes increases and then decreases, our analysis broadly captures why accuracy monotonically increases average with decreasing coverage, but displays mixed behavior the worst group.
establish the conditions under which we expect to
Our central objects of study are the margin distributions for each group. Recall that the margin of selective classifier (ū, é) on point (Ir, y) is its confidence ē(x)) 2 if the prediction is correct (i(x) y) and -((x) S O otherwise. The selective accuracies on average and on the worst- group are thus completely determined by their respective margin distributions, which we show for our datasets in Figure 4 The worst-group and average distributions are very different: the worst- group distributions are consistently shifted to the left with many confident but incorrect examples. Our approach will be to characterize what properties of a general margin distribution F lead to monotonically increasing or decreasing accuracy. Then, by letting be the overall worst-group margin distribution, we can see how the differences in these distributions lead to differences in accuracy as a function of coverage.
Setup. We consider distributions over margins that have a differentiable cumulative distribution function (CDF) and a density, denoted by corresponding upper- and lowercase variables (e.g.. F and f. respectively). Each margin distribution F' corresponds to a selective classifier over some data distribution. We denote the corresponding (selective) accuracy of the classifier at threshold Ar(T) (1 (Fr))/(F(-7)+ + decreases coverage, we focus on studying accuracy as a function of r. All proofs are in Appendix
F(T)). Since inereasing the threshold 5 monotonically
5.1 SYMMETRIC MARGIN DISTRIBUTIONS
We begin with symmetric distributions. We introduce a generalization of log-concavity, which we call left-log-concaviny for symmetric distributions, of the accuracy-coverage curve, with the direction determined by the full-coverage accuracy.
corresponds to monotonicity
Definition 1 (Leet-log-coneave distributions). A distribution is left-log-concave if its CDF is log- concave on (-oc, /1], where y is the mean of the distribution.
Left-log-concave distributions are a superset of broad family of log-concave distributions (e.g.. Gaussian, beta, uniform), which require log-concave densities (instead of CDFs) on their entire support (Boyd & Vandenberghe/ 2004]. Notably, they can multimodal: symmetric mixture of but not generally log-concave (Lemma I in Appendix D).
two Gaussians is
Proposition (Left-log-coneaviiy and monotonicity). Let F be the CDF of a symmetric distri- bution. If F is left-log-concee log-conccaee, then AF(T) is monotonically increasing in T if Ap 2 1/2 and monotonically decreasing otherwise. Conversely, if Ap, (T) ) is monotonically increasing for all translations Fa such that Fa(T) = F(T - d) for all T and Ap, (0) 2 1/2, then F is left-log- concave.

---

Published as a conference paper at ICLR 2021
Proposition[ is consistent with the observation that selective classification tends to improve average accuracy but hurts worst-group accuracy. As an illustration, consider a margin distribution that is a symmetric mixture of two Gaussians, each corresponding is -50% at full coverage but the worst-group accuracy is <50%. As the overall and worst-group margin distributions both left-log-concave (Lemma D. Proposition []] implies that worst-group accuracy will decrease monotonically with 7 while the average accuracy improves monotonically.
group, and where the average accuracy
Applied to CelebA (Figure 4 Proposition 1 is also consistent with how average accuracy improves while worst-group accuracy, which is <50% at full coverage, worsens. Finally, Proposition T also helps explain why selective classification generally improves average accuracy in the literature, as average accuracies are typically high at full coverage and margin distributions often resemble Gaussians (Balasubramanian al. 2011
Lakshminarayanan et al. 2017).
5.2
SKEW-SYMMETRIC MARGIN DISTRIBUTIONS
The results above apply only to symmetric margin distributions. As not all of the margin distri- butions in Figure are symmetric, we extend our analysis to asymmetric margin distributions by building upon prior work on skew-symmetric distributions (Azzalini & Regoli (012)).
Definition 2. A distribution with density fa.p is skew-symmetric with skew a and center μs if
fa, = 2h(T ())G-a) - -1))
(4)
for all R. where h is the density of a distribution symmetric about O, and G is the CDF of potentially different distribution that is also synmettrcc about O.
In other words, Ja is a skewed form of the symmetric density where higher a means more right skew, and setting = 0 yields (translated) h. Skew-symmetrie distributions are broad family and include, e.g., sorw-normal distributions, as well as all symmetric distributions. In Appendix D.3 show some properties of skew-symmet daemesymmetric distributions pertains to selective classification, e.g., margin distributions that are more right-skewed have higher accuracies (Proposition 6).
Our result is that skewing symmetric distribution in the "same" direction preserves monotonicity: accuracy is monotone increasing, then right skew (which increases accuracy) preserves this.
Proposition 2 (Skew in the same direction preserves monotonicity). Let Fa.u be the CDF of skew-
symmetric distribution. If accuracy of its symmetric version, AFo., (T), is monotonically increasing in r. then AF is also monotonically increasing for any >> 0. Similarly, if AFo. (T) is monotonically decreasing in T, then FF,,, (T) is also monotonically decreasing in T ffor any a < O.
5.3 DISCUSSION
Proposition relates the left-log-concavity of symmetric margin distribution to the monotonicity of selective accuracy, with the direction of monotonicity (increasing or decreasing) determined by the full-coverage accuracy. Proposition 2 then states that if we have symmetric margin distribution with monotone accuracy, skewing it in the same direction preserves monotonicity. Combining both propositions, we have that if Fo,,. is symmetric left-log-concave concave with full-coverage accuracy >50% then the accuracy (T) of any skewed (T) with a > 0 is monotone increasing in т.
Since accuracy-coverage curves are preserved under all odd, monotone transformations of margins (Lemma 8 in Appendix D. these results also generalize to odd, monotone transformations of these (skew-Jsymmette distributions. As many margin distributions inn Figure 4 and in the broader lit- erature (Ballaubraamnnian et al. 2011 Takkshmnarayanan et al.) 2017) resemble the distributions studied above (e.g., Gaussians and skewed Gaussians), we thus expect selective classification to improve average accuracy but worsen worst-group accuracy when worst-group accuracy full cov- erage is low to begin with.
An open question is how to characterize the properties of margin distributions that lead to non- monotone behavior. For example, in Waterbirds and CheXpert-device, accuracies first increase and then decrease with decreasing coverage (Figure 2) These two datasets have worst- worst-group margin distributions that have full-coverage accuracies =50% but that are left-skewed with skewness -0.30 and -0.33 respectively (Figure 4) cannot apply Proposition 2 to describe them.
7

---

Published as a conference paper at ICLR 2021
6 ANALYSIS: COMPARISON TO GROUP-AGNOSTIC REFERENCE
Even if selective classification improves worst-group accuracy, it can still exacerbate group dispari- ties, underperforming the group-agnostic reference the worst group (Section 4 In this section, we continue our analysis and show that while it is possible to outperform the group-agnostic refer- ence, it challenging do so, especially when the accuracy disparity at full coverage is large.
Setup. Throughout this section, we decompose the margin distribution into two components pFwg (1 -)) Fothers: where Fug and Fathers correspond to the margin distributions of the worst group and of all other groups combined, respectively; p is the fraction of examples in the worst group: and the worst group has strictly worse accuracy at full coverage than the other groups (i.e., AF, (0) < Pone (O)). Recall from Section 4 that for any selective classifier (i.e., any margin distribution), its group-agnostic reference has the same average accuracy at each threshold r but potentially different group accuracies. We denote the worst-group accuracy of the group-agnostic reference as AF.,(T) which can be written in terms of Fug. Fothers, and (Appendix A.2). We continue with notation from Section jotherwise, and all proofs are in Appendix
A selective classifier with margin distribution F is said to outperform the group-agnostic reference on the worst group if A (T) 2 (T) for all r 2 0. To establish a necessary condition for out- performing the reference, we study the neighborhood of r = O, which corresponds to full coverage:
Proposition 3 (Necessary condition for outperforming the group-agnostic reference). Assume that
1/2 < A (0) < A (0) < and the worst group density fwg(0) > 0. If AP., (T) S AF, (T)
for all T 2 0, then
f fothers (0) 1 A Faters (0) fwg(0 S
(5)
A (0)
The RHS is the ratio of full-coverage errors: the larger the disparity between the worst group and the other groups at full coverage, the harder it is to satisfy this condition. In Appendix [F] we simulate mixtures of Gaussians and show that this condition is rarely fulfilled.
Motivated by the empirical margin distributions, we apply Proposition 3oto the setting where Fwg and Fothers are both log-concave and are translated and scaled versions of each other. We show that the worst group must have lower variance than the others to outperform the group-agnosti reference:
Corollary (Outperforming the group-agnostic reference requires smaller scaling for log-concave distributions). Assume that 1/2 AF (0) A (0) < 1, Fwg is log-concave, and (T) V fwg (T(T- - Mothers) + pwg)) for all T E IR, where v is a scaling factor. If AFue (T) S AFA, (T) for all
This is consistent with the empirical margin distributions on Waterbirds: the worst group has higher variance, implying u 1 is the ratio of the worst group's standard deviation to the other group's s, and it thus fails to satisfy the necessary condition for outperforming the group-agnostic reference.
A further special case is when Fwg and Fothers are log-concave and unscaled translations of each other. Here, selective classification underperforms the group-agnostic reference at all thresholds т. Proposition 4 (Translated log-coneave distributions underperform the group-agnostic reference). Assume Fwg and Fothers are log- log-concave and fothers (T) = fwg(T - d) for all T E R. Then for all T 720
AF < (T).
(6)
This helps to explain our results on CheXpert-device, where the worst-group and average margin distributions are approximately translations of each other, and selective classification significantly underperforms the group-agnostic reference at all confidence thresholds.
7 SELECTIVE CLASSIFICATION ON GROUP DRO MODELS
Our above analysis suggests that seleetive classification tends to exacerbate group disparities, espe- cially when the full-coverage disparities are large. This motivates a potential solution: by reducing
8

---

Published as a conference paper at ICLR 2021
CelebA
CivilComments
Waterbirds
CheXpert-device
MultiNLI
Selective accuracy
1.0
0.9
0.8
Worst group
Group-agnestic
Average
0.6
0.25 Coverage
Coverage
Coverage
Coverage
Coverage
Figure 5: When applied to group DRO models, which have more similar accuracies across groups than standard models, SR selective classifiers improve average and worst-group accuracies. At low coverages, accuracy estimates noisy only a few predictions are made.
CelebA
Waterbirds
CheXpert-device
CivilComments
MultiNLI
Worst-group Average density
density
-255 0.0
2.5
-2
-2.5
2.5
-2.5
0.0
2.5
Margin
Margin
Margin
Margin
Margin
Figure 6: Density of margins for the average (top) and the worst-group (bottom) distributions over margins for the group DRO model. Positive (negative) margins correspond correct (incorrect) predictions, and we abstain margins closest to zero first. Unlike the selective classifiers trained with ERM, we see similar average and worst-group distributions for group DRO.
group disparities at full coverage, models are more likely to satisfy the necessary condtion for out- performing the group-agnostic reference defined in Proposition 3 In this section, explore this approach by training models using group distributionally robust optimization (group DRO) (Sagawa et al. 2020), which minimizes the worst-group training loss LDRO maxgeg [0; (2,y) v)) g]. Unlike standard training, group DRO uses group annotations training time. As with prior work, we found that group DRO models have much smaller full-coverage disparities (Figure 5). Moreover, worst-group accuracies consistently improve as coverage decreases and rate that comparable to the group-agnostic reference, though small gaps remain on Waterbirds and Che Cheepprt-device
While our theoretical analysis motivates the above approach, the analysis ultimately depends on the margin distributions of each group. not just their full-coverage accuracies. Although group DRO only optimizes for similar full-coverage accuracies across groups, we found that it also leads to much more similar average and worst-group margin distributions compared to ERM (Figure 6 explaining why selective classification behaves more uniformly over groups across all datasets.
Group DRO is not silver bullet, as it relies on group annotations for training, which are not al- ways available. Nevertheless, these results show that closing full-coverage accuracy disparities can mitigate the downstream disparities caused by selective classification.
DISCUSSION
We have shown that selective classification can magnify group disparities and should therefore be applied with caution. This is an insidious failure mode, since selective classification generally im- proves average accuracy and can appear to be working well if we do not look at group accuracies. However, we also found that selective classification can still work well on models that have equal full-coverage accuracies across groups. Training such models, especially without relying too much additional information at training time, remains an important research direction. On the theo- retical side, we characterized the behavior of selective classification in terms of the margin distribu- tions; an open question is how different margin distributions arise from different data distributions, models, training procedures, and selective classification algorithms. Finally, in this paper focused on studying selective accuracy in isolation; accounting for the cost of abstention and the equity of different coverages different groups is important direction for future work.

---

Published as a conference paper at ICLR 2021
ACKNOWLEDGMENTS
We thank Emma Pierson, Jean Feng, Pranav Rajpurkar, and Tengyu Ma for helpful advice. This work was supported by NSF Award Grant no. 1804222. SS was supported by the Herbert Kunzel Stanford Graduate Fellowship and AK was supported by the Stanford Graduate Fellowship.
REPRODUCIRILITY
All code, data, and experiments are available on CodaLab https:/ https::/wrkkshetss The code is
coda org also available on GitHub at
REFERENCES
Adelchi Azzalini and Giuliana Regoli. Some properties of skew-symmetric distributions. Annals of the Institute of Statistical Mathematics, 64(4))857-879, 2012.
Marcus Badgeley, John R Zech, Luke Rayne,, Benjamin Glicksberg, Manway Liu, William Gale, Michael V McConnell, Bethany Percha, Thomas M Snyder, and Joel T Dudley. Deep learning predicts hip fracture using confounding patient and healthcare variables. npj Digital Medicine, 2, 2019.
Mark Bagnoli and Ted Bergstrom. Log-concave probability and its applications. Economic Theory, 26:445-469, 2005.
Krishnakumar Balasubramanian, Pinar Donmez, and Guy Lebanon. Unsupervised supervised learning II: Margin-based based classification without labels. Journal of Machine Learning Research (JMLR), 12:3119 11-5514,,,, 2011.
Peter L Bartlett and Marten H Wegkamp. Classification with a reject option using hinge loss. Journal of Machine Learning Research (JMLR), 90)):1823-1840, 2008
Su Lin Blodgett, Lisa Green, and Brendan O'Connor. Demographic dialectal variation social media: A case study of African-American English. In Empirical Methods in Natural Language Processing (EMNLP), pp. 1119-11300 2016.
Daniel Borkan, Lucas Dixon, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. Nuanced metrics for measuring unintended bias with real data for text classification. World Wide Web (WWW), pp. 491-500 2019.
Stephen Boyd and Lieven Vandenberghe. Convex Optimization. Cambridge University Press, 2004.
Joy Buolamwini and Timnit Gebru. Gender shades:
Intersectional accuracy disparities in com- mercial gender classification. Conference on Fairness, Accountability and Transparency, pp. 77-91, 2018.
Irene Y Chen, Emma Pierson, Sherri Rose, Shalmali Joshi, Kadija Ferryman, and Marzych Ghas- semi. Ethical machine learning in health. arXiv preprint arXiv:2009. /0576, 2020.
C. K. Chow. An optimum character recognition system using decision functions. In IRE Transac- tions on Electronic Computers, 1957.
Chao K Chow. On optimum recognition error and reject tradeoff. IEEE Transactions on Information Theory, 16(1):41-46 1970.
Sam Corbett-Davies. Emma Pierson, Avi Feller, Sharad Goel, and Aziz Huq. Algorithmic decision making and the cost of faimess. In International Conference Knowledge Discovery and Data Mining (KDD), pp. 797-806, 2017.
Luigi Pietro Cordella, Claudio De Stefano, Francesco Tortorella, and Mario Vento. A method for improving classification reliability of multilayer perceptrons. IEEE Transactions on Neural Net- works, 6(5): 140-11444 1995.
10

---

Published as a conference paper at ICLR 2021
Madeleine Cule, Richard Samworth, and Michael Stewart. Maximum likelihood estimation of multi-dimensional log-con log-concave density. Journal of the Royal Statistical Society, 73:545-603, 2010.
Abir De, Paramita Koley, Niloy Ganguly, and Manuel Gomez-Rodriguez. Regression under human assistance. In Association for the Advancement of Artificial Intelligence (AAAI), pp. 261 1-2620, 2020.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT; Pre-training of deep bidirectional transformers for language understanding. Association for Computational Linguis- lics (ACL), pp. 4171-41888 2019.
Lucas Dixon, John Li, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. Measuring and mit- igating unintended bias in text classification. In Association for the Advancement of Artificial Intelligence (AAAT), pp- 67-73, 2018.
John Duchi, Tatsunori Hashimoto, and Hongseok Namkoong. Distributionally robust losses against mixture covariate shifts. https://cs. stanford. edu/ www.hhhhhhmmmam.. assets/ publicat ions condrisk pdf 2019.
Cynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Rich Zemel. Faireess through awareness. In Innovations Theoretical Computer Science (ITCS), pp. 214-226, 2012.
Ran El-Yaniv and Yair Wiener. On the foundations of noise-free selective classification. Journal of Machine Learning Research (JMLR), 11, 2010.
Jean Feng, Arjun Sondhi, Jessica Perry, and Noah Simon. Selective prediction-set models with coverage guarantees. arXiv preprint arXiv: /906. 05473, 2019.
Yarin Gal and Zoubin Ghahramani. Dropout as a Bayesian approximation: Representing model uncertainty in deep learning. In International Conference on Machine Learning (ICML), 2016.
Yonatan Geifman and Ran EI- Yaniv. Selective classification for deep neural networks. In Advances in Neural Information Processing Systems (NeurIPS), 2017.
Yonatan Geifman and Ran Elnia..v. Selectivenet: A deep neural network with an integrated reject option. In International Conference Machine Learning (ICML), 2019.
Yonatan Geifman, Guy Uziel, and Ran El- Yaniv.. Bias-reduced uncertainty estimation for deep
neural classifiers. In International Conference on Learning Representations (ICLR), 2018.
Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A Smith. Annotation artifacts in natural language inference data. In Association for Com- putational Linguistics (ACL), pp. 107-112, 2018.
Blaise Hanczar and Edward R. Dougherty. Classification with reject option in gene expression data. Bioinformatics, 2008
Moritz Hardt, Eric Price, and Nathan Srebo. Equality of opportunity in supervised learning. In Advances in Neural Information Processing Systems (NeurIPS), pp. 3115-332533 2016.
Tatsunori B. Hashimoto, Megha Srivastava, Hongseok Namkoong, and Percy Liang. Fairness with- out demographics in repeated loss minimization. In International Conference on Machine Learn- ing (ICML), 2018
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. Computer Vision and Pattern Recognition (CVPR), 2016.
Martin Hellman and Josef Raviv. Probability of error, equivocation, and the chernoff bound. IEEE Transactions on Information Theory, 16(4):368-372 1970.
Martin E Hellman. The nearest neighbor classification rule with a reject option. JEEE Transactions Systems Science and Cybernetics, 6(3):179-185, 1970.

---

Published as a conference paper at ICLR 2021
Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and examples in neural networks. In International Conference on Learning Representations (ICLR), 2017.
Kashmir Hill. Wrongfully accused by an algorithm. The New York Times, 2020. URL https:// www. iyee http:///ww..e////////////coooooa http://oooooo......aeeeeooooooaa
recognition arrest.
Dirk Hovy and Anders Søgaard. Tagging performance correlates with age. In Association for Computational Linguisties (ACL), pp. 483 2015.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4700-4708, 2017.
Jeremy Irvin, Pranav Rajpurkar, Michael Ko, Yifan Yu, Silviana Ciurea-Ilcus, Chris Chute, Henrik Marklund, Behzad Haghgoo, Robyn Ball, Katie Shpanskaya, et al. Chexpert: A large chest radio- graph dataset with uncertainty labels and expert comparison. In Association for the Advancement of Artificial Intelligence (AAAI), volume 33, pp. 590-597, 2019.
Shalmali Joshi, Oluwasanmi Koyejo, Been Kim, and Joydeep Ghosh. xGEMs: Generating exam- plars to explain black-box models. arXiv preprint
2018.
Amita Kamath, Robin Jia, and Percy Liang. Selective question answering under domain shift. In Association for Computational Linguistics (ACL), 2020.
Fereshte Khani, Martin Rinard, and Percy Liang. Unanimous prediction for 100% precision with application to learning semantic mappings. In Association Computational Linguistics (ACL), 2016.
Jon Kleinberg. Sendhil Mullainathan, and Manish Raghavan. Inherent trade-off's in the fair determi- nation of risk scores. In Innovations in Theoretical Computer Science (ITCS), 2017.
Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsub- ramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, Tony Lee, Etienne David, Ian Stavness, Wei Guo, Berton A. Farnshaw, Imran S. Haque, Sara Beery, Jure Leskovec, Anshul Kundaje, Emma Pierson, Sergey Levine, Chelsea Finn, and Percy Liang. WILDS: A benchmark of in-the-wild distribution shifts. arXiv, 2020.
Balaji Lakshminarayanaan Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In Advances in Neural Information Processing Systems (NeurIPS), 2017.
Shiyu Liang, Yixuan Li, and R. Srikant. Enhaneing the reliability of out-of-distribution image detection in neural networks. In International Conference on Learning Representations (ICLR), 2018.
Ziwei Liu, Ping Luo. Xiaogang Wang. and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of the IEEE International Conference on Computer Vision, pp. 3730-3738, 2015.
Thomas McCoy, Ellie Pavlick, and Tal Linzen. Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. In Association for Computational Linguistics (ACL), 2019.
Hussein Mozannar and David Sontag. Consistent estimators for learning to defer to an expert. arXiv preprint 2rXiv:2006.0/880 2020.
Luke Oakden-Rayner, Jared Dunnmon, Gustavo Carneiro, and Christopher Ré. Hidden stratification causes clinically meaningful failures in machine learning for medical imaging. In Proceedings of the ACM Conference on Health, Inference, and Learning, pp. 151-159, 2020.
Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, D Sculley, Sebastian Nowozin, Joshua V. Dil- lon, Balaji Lakshminarayanan, and Jasper Snock. Can you trust your model's uncertainty? eval- uating predictive uncertainty under dataset shift. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
12

---

Published as a conference paper at ICLR 2021
Ji Ho Park, Jamin Shin, and Pascale Fung. Reducing gender bias in abusive language detection. In Empirical Methods Natural Language Processing (EMNLP), pp. 7799-2804, 2018.
Marco AF Pimentel, David A Clifton, Lei Clifton, and Lionel Tarassenko. A review of novelty detection. Signal Processing, 99:215-249, 2014.
José M Porcel. Chest tube drainage of the pleural space: a concise review for pulmonologists. Tuberculosis and Respiratory Diseases, 81(2):00--115, 2018.
Maithra Raghu, Katy Blumer, Rory Sayres, Ziad Obermeyer, Bobby Kleinberg, Sendhil Mul- lainathan, and Jon Kleinberg. Direct uncertainty prediction for medical second opinions. In International Conference Machine Learning (ICML), pp. 5281-5290, 2019.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. "Why Should Trust You?": Explaining the predictions of any classifier. In International Conference on Knowledge Discovery and Data Mining (KDD), 2016.
Shiori Sagawa, Pang Wei Koh, Tatsunori Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generaliza- tion. In International Conference on Learning Representations (ICI.R,, 2020.
Rachael Tatman. Gender and dialect bias in YouTube's ' automatic captions. In Workshop on Ethics in Natural Langauge Processing, volume 1, pp. 53 53-59, 2017.
Marko Toplak, Rok Močnik, Matija Polajnar, Zoran Bosnić, Lars Carlsson, Catrin Hasselgren, Janez Demšar, Scott Boyer, Blaz Zupan, and Jonna Stålring. Assessment of machine learning reliabil- ity methods for quantifying the applicability domain of QSAR regression models. Journal of Chemical Information and Modeling. 54, 2014.
C. Wah, S Branson, P Welinder, P Perona, and S Belongie. The Caltech-UCSD Birds-200-2011 dataset. Technical report, California Institute of Technology, 2011.
Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sen- tence understanding through inference. In Association for Computational Linguistics (ACL), pp- 1112-1122, 2018.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R'emi Louf, Morgan Funtowicz, and Jamie Brew. HuggingFace's transformers: State-of-the-art natural language processing. arXiv preprint arXiv:1910.03771 2019.
Kai Xiao, Logan Engstrom, Andrew Ilyas, and Aleksander Madry. Noisc or signal: The role of image backgrounds in object recognition. arXiv preprint WXXi:2006.09994 2020.
Dong Yu, Jinyu Li, and Li Deng- Calibration of confidence measures in speech recognition. Trans. Audio, Speech and Lang. Proc.. 2944:2244- 2011.
Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 mil- lion image database for scene recognition. IEEE Transuctions on Puttern Analysis and Machine Intelligence, 40(6): 1452-1464 2017.
13

---

Published as a conference paper at ICLR 2021
A SETUP
A.1 SOFTMAX RESPONSE SELECTIVE CLASSIFIERS
In this section, we describe our implementation of softmax response (SR) selective classifiers (Geif- man & El- El-Yaniv, 2017). Recall from Section that a selective classifier is a pair (8.C), where y: : y outputs prediction and é X R+ outputs the model's confidence, which always non-negative, in that prediction. SR classifiers are defined for neural networks (for classification), which generally have a last softmax layer over the possible classes. For an input point v, we denote its maximum softmax probability, which corresponds to its predicted class j(x), as p(ū(x)) x). We defined the confidence ((x) for binary classifiers as
(()(z) x) a(x) = log 1 p(((x) (1)
Since the maximum softmax probability p()((z) x) is least 0.5 for binary classification, ē(x) is nonnegative for each v, and is thus a valid confidence. For > classes, however, p(o(z) x) be less than 0.5, in which case ê(x) would be negative. To ensure that confidence is always nonnegative, we define ((x) for lt classes to be
1 ((x) log ((0(()) x²) p(O(z) x))
+ log(k 1).
With & classes, the maximum softmax probability ((ū(x) 2)
verify that Ō(x) 2 0 as desired. Moreover, when h: = 2. Equation reduces to our original binary
1/k, and therefore
confidence; we can therefore interpret the general form in Equation 8 as normalized logit.
Note that ((x) is a monotone transformation of the maximum softmax probability fū(x)) x)) Since the accuracy-coverage curve of a selective classifier only depends the relative ranking of ē(x) across points, we could have equivalently set č(x) to be p(((z) x). However, following prior work, choose the fogit-transformed version make the corresponding distribution of confidences easier to visualize (Balasubramanian et al. 2011 Lakshminarayanan et al. 2017).
Finally, we remark on one consequence of SR on the margin distribution for multi-class classifica- tion. Recall that we define the margin of an example to be ê(x) on correct predictions (f(x) = y) and -((x) otherwise, as described in Section 3 Figure we plot the margin distributions of SR selective classifiers on all five datasets. We observe that on MultiNLI, which is the only multi-class dataset (with k 3), there is a gap (region of lower density) in the margin distribution around O. We attribute this gap in part to the comparative rarity of seeing maximum softmax probability of when & versus seeing when & = 2; in the former, all three logits must be the same, while for the latter only two logits must be the same.
A.2 GROUP- REFERENCE
Here, we describe the group-agnostic reference described in Section i more detail. We elaborate on the construction from the main text and then define the reference formally. Finally, show that the group-agnostic reference satisfies equalized odds.
A.2.1 DEFINITION
We begin by recalling on the construction described in the main text for a finite test set D. In Algorithm we relied two important sets; the of correctly classified points at threshold 7, C,:
C {ax y. g) E D ((x) = and ((x) 2 r}
(9)
and similarly, the set of incorrectly classified points at threshold I.: I- = ((..(.)) E D ((x) # and 6(x) 2 r}.
(10)
To compute the accuracy of the croup-agnostic reference, we sample a subset Cf of size |C, uni- formly at random from Co, and similarly subset - of size [I| uniformly at random from Iy. The reference makes predictions when examples are in C U and abstains otherwise. We compute group accuracies over this set of predicted examples.
14

---

Published as a conference paper at ICLR 2021
Note that the group accuracies, as defined, are randomized due to the sampling. For the remainder of our analysis, we take the expectation over this randomness to compute the group accuracies. We now generalize the above construction by considering data distributions D. We first define the number of correctly and incorrectly classified points:
Definition 3 (Correctly and incorrectly classified points), Consider a selective classifier (ÿ, C). For each threshold t, we define the fractions of points that are predicied (not abstained on), and correctly or incorrectly classified, as
C(r) = ( = I(T) p(((x)) * y é(z) 2 T).
7),
(11)
(12)
We define analogous metrics for each group
Cy(T) p()(x) y) lg), [g(T) = p(((x) # yy 2 g).
(13)
(14)
For each threshold 7, we will make predictions C(r)/C(0) fraction of the C(O) total (proba- bility mass of) correctly classified points. Since each group g has (0) correctly classified points, threshold т, the group-agnostic reference will make predictions on C, (C)))))) correctly classified points in group We can reason similarly over the incorrectly classified points. Putting it all together, we can define the group-agnostic reference as satisfying the following:
Definition 4 denote the analogous quantities to Definition for its matching group-agnen proup-agnostic reference. For each threshold t, these satisfy
reference). Consider selective classifier (Ú, é) and let C, Í, Cg. Ig
Ö(r) -((r)
(15)
ï(T) I(T), -(T),
(16)
for each threshold T and group g.
Cg(T) = C,(0)C(()/(()) ((T)=11(000 [(())
(17)
(18)
The group-agnostic reference thus has the following accuracy on group g:
Ag(T) = (,(
(19)
+,(7)+
C,(0)C(()/(0))
(20)
A.2.2 CONNECTION TO EQUALIZED ODDS
We now show that the group- reference satisfies equalized odds with respect to which points it predicts or abstains on. The goal of a selective classifier is to make predictions on points it would get correct (i.e., f(x) y) while abstaining on points that it would have gotten incorrect (i.e., ((x) * y). We can view this as a meta classification problem, where a true positive is when the selective classifier decides to make prediction on a point and gets it correct (f(x) y), and false positive is when the selective classifier decides to make a prediction on point x and gets it incorrect (g(x) + y). As such, we can define the true positive rate TTP(1 (T) and false positive rate REP (T) of selective classifier:
Definition 5. The true positive rate of a selective classifier at threshold T is
C(T)
R TTP((T
(21)
C(0))
and the false positive rate at threshold is
RFP T(T)
(22)
I(0)
15

---

Published as a conference paper at ICLR 2021
Analogousty, the tue positive and false positive rates group
Hg (T) Cg(T)
(23)
R (T) [T(
(24)
(0)
The group-agnostic reference satisfies equalized odds (Hardt et al. 2016) with respect this defini- tion:
Proposition The group-agnostic reference defined in Definition has equal true positive and false positive rates for all groups 9 and satisfies equalized odds.
Proof. By construction of the group- aoousstgnottic reference (Definition we have that
= -,(0)((0())))/00000 C)(0))(()/((())
(25)
(26)
and therefore, for each group g. we can show that the true-positive of the refer- ence RT on g is equal to average true-positie true-positive rate RTP RTP(T).
Cg(T)
R (T)
(27)
Cg(0)
=Ō(r)/Ō(0)
(28)
=R.
(29)
Each group thus has the same true positive rate with the aroup-agnostic reference. Using similar reasoning, each group also has the same false positive rate. By the definition equalized odds, the group-agnostic reference thus satisfies equalized odds.
A.3 ROBIN HOOD REFERENCE
In this section, we define the Robin Hood reference, which preferentially increases the worst-group accuracy through abstentions until it matches the other groups Like the group-agnostic reference, we constrain the Robin Hood reference to make the same number of correct and incorrect predictions as given selective classifier.
We formalize the definition of the Robin Hood reference in Algorithm 2 ence makes predictions the subset of examples from D that has the smallest discrepency between best-group and worst-group accuracies, while still matching the number of correct and incorrect predictions of given selective classifier. Since enumerating over all possible subsets of the test data
The Robin Hood refer-
Algorithm 2: Robin Hood reference threshold
Input: Selective classifier (ū. 2), threshold r, test data D
Output: The set of points P D that the Robin Hood reference for (j. Z) makes predictions on at threshold
In Algorithm 1 we defined C, and I, be the sets of correct and incorrect points predicted on and abstamed on at threshold respectively. Define Q, the set of subsets of D that have the same number correct and incorrect predictions as (j. ?) at threshold r as follows:
{S D [P |c, P
[I,
(30)
Let (C) be the accuracy of points in S that belong group g. We return the set P that minimizes the difference between the best-group and worst- group accuracies.
P argmin SEQ
max acc, (S) min acc, (S) 9E9
(31)
gEg
16

---

Published as a conference paper at ICLR 2021
D is intractable, we compute the accuracies of the Robin Hood reference iteratively. Starting from full coverage, we abstain on examples from lowest to highest confidence. Whenever we abstain on incorrect example, larly whenever we abstain on a correctly classified example we assume it comes from the current highest accuracy group. This reduces disparities the maximum extent possible as the threshold increases.
assume it comes from the current lowest accuracy group, and simi-
B SUPPLEMENTAL EXPERIMENTS
B.I MONTE-CARLO DROPOUT
In the main text, observed that SR selective classifiers monotonically improve average accuracy as coverage decreases, but exacerbate accuracy disparities across groups on all five datasets. To demonstrate that these observations
not specific SR selective classifiers, we now present our empirical results on another standard selective classification method: Monte-Carlo (MC) dropout (Gal Ghahramani] 2016 Geifman & EI- Yaniv 2017). We find that MC-dropout selective classi- fiers exhibit similar empirical trends as SR selective classifiers.
MC-dropout selective classifiers. MC-dropout an alternate way of assigning confidences to points. Taking model with a dropout layer, the selective classifier first predicts j(x) simply by taking the model output without dropout. To estimate the confidence, it then samples n softmax probabilities corresponding to the label j(z) over the randomness of the dropout layer. The con- fidence is computed as ((x) = 1/8, where 52 is the variance the sampled probabilities. We implement MC-dropout by using the existing dropout layers for BERT and adding dropout to the final fully-coneected layer for ResNet and DenseNet, with a dropout probability 0.1. We present empirical results for n 10; Gal & Ghahramani (2016) observed that this was sufficient to produce good confidence estimates.
Results. The MC-dropout selective classifiers exhibit similar trends those that observed in Section demonstrating that the observed empirical trends are not specific to SR response; even though the average accuracy improves monotonically as coverage decreases across all five datasets, the worst- group accuracy tends to decrease for CelebA, fails to increase consistently for CheXpert, Waterbirds, and CivilComments, and increases consistently but slowly for MultiNLI. Comparing SR and MC- dropout selective classifiers, we observe that the MC-dropout selective classifiers performs slightly worse. For example, we see a more prominent drop in worst-group accuracy for Waterbirds and much smaller improvements in worst-group accuracy for CheXpert.
CelebA
Waterbirds
CheXpert-device
CiviiComments
MultiNLI
Group hoverage Group accuracy
0.75
Worst group Average Other groups
0.25
0
0.75
0.25
Average coverage Average coverage
Average coverage
Average coverage
Average coverage
Figure 7: Selective accuracy (top) and coverage (bottom) for each group, as function of the average
coverage for the MC-dropout selective classifier. Each average coverage corresponds to a threshold The red lines represent the worst group.
17

---

Published as a conference paper at ICLR 2021
CelebA
CivilComments
Waterbirds
CheXpert-device
MultiNLI
Group coverage Group sel, accuracy
1.0
0.9
0.8
Worst
0.7
Average
groups
0.6
1
0.75
0.5
0.25
0.5
0.5 Average coverage
0.5 Average coverage
0.5 Average coverage
0.5 Average coverage
Average coverage
Figure 8: Selective accuracy (top) and coverage (bottom) for each group as function of the average coverage for the softmax response selective classifier with model optimized with group DRO. Each average coverage corresponds to a specific threshold т. The red lines represent the worst group (i.c. the one with the lowest accuracy full coverage,) and the gray lines represent the other groups.
B.2 GROUP DRO
We showed in Section that SR selective classifiers trained with the group DRO objective success- fully improve worst-group selective accuracies as coverage decreases, and perform comparably to the group-agnostic reference. We now present additional empirical results for these selective classi- fiers.
Accuracy-coverare curves. We first present the accuracy-coverage curves for all groups, along with the group coverage trends, in Figure g Group selective accuracies tend to improve mono- tonically, in stark contrast with results standard (ERM) selective classifiers. These trends hold generally across groups and datasets, with few exceptions; selective accuracies drop on two groups in MultiNLI roughly 20% average coverage, and selective accuracies improve slowly on two groups in CivilComments. Below, we look into these anomalies and offer potential explanations.
For MultiNLI, first note that the drops observed very low group coverages-lower than 1%-at which point the accuracies are computed based on very few examples and thus are noisy. In addition, much of the anomaly can be explained by label noise; we manually inspect the 20 examples from the above two groups with the highest softmax confidences, and find that 17 of them are labeled incorrectly. The observations on CivilComments can also be potentially attributed to label noise. Removing examples with high inter-annotator disagreement (with the fraction of toxic annotations between 0.5 and 0.6) yields an accuracy-coverage curve that fits the broader empirical trends.
C EXPERIMENT DETAILS
C.I DATASETS
CelebA. Models have been shown to latch onto spurious correlations between labels and demo- graphic attributes such as race and gender (Buolamwini & Gebru 2018 Joshi et al.) 2018), and we study this on the CelebA dataset (Liu et al.) (0055). Following Sagawa et al. (2020). we consider the task of classifying hair color, which is spuriously correlated with the gender. Concretely, inputs are celebrity face images, labels are hair color y (blond, non-blond) and spurious attributes are gender, A (male, female), ) with blondness associated with being female. Of the four groups, blond males are the smallest group, with only 1,387 examples out of 162,770 training examples, and they tend to be the worst group empirically. We use the official train-val-split of the dataset.
18

---

Published as a conference paper at ICLR 2021
Waterbirds. Object recognition models are prone using image backgrounds as a proxy for the label RRieiiro et al.] 2016; Xiao et al. 2020). We study this on the Waterbirds dataset (Sagawal
et al. 2020), constructed using images of birds from the Caltech- UCCD Birds dataset (Wah et al. placed on backgrounds from the Places dataset (Zhou et al. 2017). The task is to classify a photograph of a bird one of y = (waterbird, landbird), and the label is spuriously correlated with the background A A { {water background, land background). Of the four groups, waterbirds on land backgrounds make up the smallest group with only 56 examples out of 4,795 training examples, and they tend to be the worst group empirically. We use the train-val-split provided by [Saaaaa et al. (2020), and also follow their protocol for computing average metrics; to compute average accuracies and coverages, we first compute the metrics for each group and obtain a weighted average according to group proportions in the training set, in order to account for the discrepancy in group proportions across the splits.
2011
CheXpert-device. Models can latch onto spurious correlations even in high-stakes applica- tions such as medical imaging. When models are trained to classify whether patient has certain pathologies from chest X-rays, models have been shown to spuriously detect the pres- ence of a support device, in particular a chest drain, instead (Oakden-Raynce et al., 2020 We study this phenomenon in a modified version of the CheXpert dataset (Irvin et al. 2019) which we call CheXpert-device. Concretely, the inputs are chest X-rays, labels are y {pleural effusion, no pleural effusion), and spurious attributes indicate the the presence of a support device, A = (support device, no support device]. We note that chest drain is one type of support device, and is used to treat suspected pleural effusion (Porcel 2018).
Che Cher a subsampled version of the full CheXpert dataset that manifests the spurious correlation more strongly. To create CheXpert-device, we first create a new 80/10/10 train/val/test split of examples from the publicly available CheXpert train and validation sets, randomly assigning patients splits so that all X-rays of the same patient fall in the same split. We then subsample the training set; in particular, we enforce that in 90% of examples, the label of support device matches the pleural effusion label. Of the four groups, cases of pleural effusion without support device make up the smallest group, with 5,467 examples out of 1 112,100 training examples, and they tend to be the worst group empirically.
To compute the average accuracies and coverages, we weight groups according to group proportions in the training set, similarly to Waterbirds. Another complication with CheXpert is that some patients can have multiple X-rays from one visit. Following Irvin et al. (2019). we treat these images as separate training examples at training time, but output one prediction for each patient-study pair at evaluation time. Coneretely, we predict pleural effusion if the model detects the condition in any of the X-ray images belonging to the patient-study pair, as pathologies may only appear clearly in some X- X-rays.
CivilComments. In toxicity comment detection, models have been shown to latch onto spurious correlations between the toxicity and mention of certain demographic groups (Park al. 2018 Dixon et al. 2018). We study this in the CivilComments dataset (Borkan et al. 2019). The task to classify the toxicity of comments on online articles with labels y = ttoxic, non-toxic). As spurious attributes, we consider whether each comment mentions a Christian identity, A = {mention of Christian identity, no mention of Christian identity]; non-toxicity is associated with the mention of Christian identity, often resulting in high false negative rate on comments with such mentions. Of the four groups, toxic comments with a mention of Christian identity make up the smallest group, with only 2,446 examples out of 269,038 training examples, and they tend to be the worst group empirically. and further split the development set into a training set and a validation set by randomly splitting on articles and associating all comments with each article to either set. We use the train, validation, and test set from the WILDS benchmark (Koh et al.) 2020). The training, validation, and test set all comprise comments from disjoint sets of articles. The original dataset also contains many additional examples that have toxicity annotations but not identity annotations; we do not use these in our experiments.
In the original CivilComments dataset, each comment is given a probabilistic labels for both the toxicity and the mention of Christian identity, where probabilistic label is the average of binary
19

---

Published as a conference paper at ICLR 2021
labels across annotators. Following the associated Kaggle competition binarized labels obtained by thresholding the probabilistic labels at 0.5.
MultiNLI Lastly, consider natural langage inference (NLD, where the task is to predict whether a hypothesis is entailed, contradicted by. or neutral to an associated premise, y fentailed, contradictory, neutral). NLI models have been shown to exploit annotation artifacts, for example predicting contradictory whenever negation words such as never or nobody are present (Gu- rurangan et al.) 2018). We study this on the MultiNLI dataset (Williams al. 2018). To annotate examples' spurious attributes A |negttion words, no negation words), we consider the following negation words following Gururangan et al. (2018): "nobody". "no", "never", and "nothing". We use the splits used in Sagawa et al. (2020), whose training set includes 206,175 examples with 1 1,521 examples from the smallest group (entailment with negations). The worst group for standard models tends to be neutral examples with negation words.
C.2 MODELS
We train ResNet (He et al. 2016) for CelebA and Waterbirds (images), DenseNet (Huang et al. 2017) for CheXpert (X-rays), and BERT (Devlin et al. 2019) for CivilComments and MultiL (text). For tasks studied in Sagawa et al. (2020) (CelebA, Waterbirds, MultiNLD, we use the hyper- parameters from SSaawaa et al. (2020). For others (CivilComments and CheXpert-device),, we test the same number of hyperparameter sets for each of ERM and DRO, and report the best set below. Across all image and X-ray tasks, inputs are downsampled to resolution 224 x 224.
CelebA. To train a model on CelebA, we initialize to pretrained ResNet-50. For ERM we optimize with learning rate le-4, weight decay Ie-4, batch size 128, and train for 50 epochs. For DRO we use learning rate le-5, weight decay le-1, and use generalization adjustment (described in Sagawa al. (020)). The batch size 128, and we train for 50 epochs.
Waterbirds. For Waterbirds, as with CelebA, we use pretrained ResNet-50 as an initialization. For ERM we use learning rate 1c-3, weight decay Ie-4, batch size 128, and train for 300 epochs. For DRO we use learning rate 1e-5, weight decay 1, geneneralization adjustment 1, batch size 128, and train for 300 epochs.
CheXpert-device. For -he we fine-tune pretrained DenseNet-121 for three epochs. For ERM we use learning rate 1e-3, no weight decay, batch size 16, and choose the model (out of the first three epochs) with highest average accuracy (epoch 2). For DRO, we use learning rate le- 4, weight decay le-1, and batch size 16, and choose the model with highest worst-group accuracy (epoch 1).
CivilComments. To train a model for CivilComments, we fine-tune bert-base-uncased using the implementation from Woll et al. (2019). For both ERM and DRO use learning rate 1e-5, weight decay le-2, and batch size 16. We train the ERM models and DRO models for three epochs (early stopping,) then choose the model with highest average accuracy for ERM (epoch 2), and highest wonst-group accuracy for DRO (epoch 1).
MultiNLI For MultiNLI again fine-tune hert-base-uncased using the implementation from Wolf jet al. (201)). For ERM, we fine-tune for three epochs with learning rate 2e-5, weight decay 0, and batch size 32. For DRO, we also use learning rate 2e-5, and weight decay 0, and batch size 32, but use generalization adjustment I. For both ERM and DRO the model after the third epoch is best in terms of average accuracy for ERM and worst-group accuracy for DRO.
D PROOFS: MARGIN DISTRIBUTIONS AND ACCURACY-COVERAGE CURVES
D. LEFT-LOC-CONCAVITY
Recall the definition of left-log-concave from Section
Definition 1 (Left-log-coneave distributions). distribution is left-log-concave if its CDF is log- concave on (-oo, pi], where μ is the mean of the distribution.
www. www..kaggee..om///////ooo http://w........oooea.oeeeeeeeea fiaatitioon
20

---

Published as a conference paper at ICLR 2021
We first prove that symmetric mixture of Gaussians is left-log-coneave, but not necessarily log- concave.
D.1.1 SYMMETRIC MIXTURES OF GAUSSIANS ARE LEFT-LOG-CONCAVE
Lemma (Symmetric mixtures of two gaussians are left-lo--concave) Consider a symmetric mix- ture of Gaussians with density f = 0.5 + 0.5f-p where fu is the density of a N NN,, random variable and likewise f-2 the density of a (N(-u.o)) random variable. Then the mixture is left-log-concant for all values of j RRo > 0, but only log-concave if [u] S
Proof. Without loss of generality, we can take o = 1, since (left-) left-)log-concavitty is invariant to scaling, and also assume that I is positive. First consider the case where μ I 1. Then the mixture is log-concave, and therefore left-log-concave (Cule et al. 2010).
Now, consider the case where μ 1. Cule al. (2010) show the mixture is no longer log- conccvecvve. However, we claim that it is still left-log- concave. We start by studying the gradient of log f.
f((x)
-epp (x-=)² (x - (4) exp( (x+x)²
(xx 1d)
(32)
f(x)
exp(
(ar (=-w)²
exp( (x+x)²
2
- exp(-22p -2zzp)
=-r+ 1
(33)
+ exp( -2044)
We claim that f(xx) has a local minimum at x = a O, and is an odd function, correspond- f(x)
ing local maximum at x a > 0. To show this, we first differentiate to obtain d f'(x) = -1+
4n² exp(-22pp
(34)
da f(x)
(1 expe 22xj)))
Setting the derivative to 0 gives us the quadratic equation
41² exx((x2210) (20p-2x1))
0 -1 +
(35)
(1 expl (2r(1))²
(1 + exp(-221) 2xp)) )² = exppe 227144
(36)
1 (exp(-2rTu)) (2 exp( (ap(-2 + +1
(37)
exp(-2x1) = 2/1² + 2 112
(38)
Since > 1, there are two distinct roots of this quadratic, and two corresponding critical points of f'/f. Let v((u) 2n² - + 2/a n22 - be the larger root. Then v(ta) is strictly increasing have that v(μ) > for all p > 1. Let æ a satisfy
function for μ 2 and since v(1) exp(2aμ) = v()). Then we have that -a -a, the smaller of the critical points of f'/f.is
log v(1)
(39)
24
1og(2 - -1 1 2 112
(40)
2/1
<.
(41)
To show that f((a)f((a) is a local minimum, we take the second derivative
d² f'(x) d -1 41r² exp( (xr())
(42)
dx f(x) dx (1+expl (xx)))))))
813 exp(-2x4)(expf 22rFu)
(43)
(exp( 22744) + 1)3
which at gives
dx² f(x) z=-a 8p²r(m)(v(1)) -1)
d f'(x).
(44)
(v(u) + 1)3
(45)
21

---

Published as a conference paper at ICLR 2021
since v(p) > 1. Since -a is the only critical point of f'/ that is less than and it is local minimum, f'/f must be decreasing on (-00,- -a], which in turn implies that f. and therefore F, is log-concave (-00,-00 -a].
remains to show that F' is also log-concave on [-a, 0j. We make use of two facts. First, since aa
local minimum and the only critical point less than O, we have that [[(-a) π(-a)
< ((()
< f(()
[( [(]
for all E I-a, 0]. Second, since f(x) and F(z) are non-negative for x, f(z) (x)) is also non-negative for all r. Thus, for all [-a,
d (xx (x) f())
(46)
dx Fxx)
F(z)²
f(a)
('(x)
f(x)
(47)
F(x)
f(x)
F(z)
20
20
20
S 0,
(48)
and therefore is also cog-coecave on [-a, 0].
Remark 1. Note that if f is (left- lleft-lag-conncuve, then F is also Bergstrom 2005). However, the reverse direction does not hold.
(Bagnoli &
D.2 SYMMETRIC MARGIN DISTRIBUTIONS
In this section we prove Proposition We first prove the following helpful lemma: Lemma 2 (Conditions for monotonicity of selective accuracy). Ar(T) is monotone increasing if and only if
f(-T)
f(T) F(T)
(49)
F(-T)
for all 2 Conversely, AF(T) is monotone decreasing in T if and only if the above inequality is flipped for all 2 0.
Proof. (F( is monotone increasing in if and only if dAp differentiating Ar and simplifying:
dr 2 0 for all r 2 0. We obtain dAr by
de
dAF d dr
d-
F()
(50)
dr (-F(7)+ F
1
(51)
((-F(T)+F())
(1-F())+((TT))
f(-T) ((-
(52)
(1 (1- F(-)) f(-r) -()))
(53)
(L=F(T)+F((7))
Since the denominator is always positive, we have that ddg 2 if and only if the numerator
de - 2 0, which in turn is equivalent
f(-r) f(T) F(-T) 1- F-F(7)'
(54)
as desired. The case for monotone decreasing Ap(r) is analogous.
In the next two lemmas, we prove the necessary and sufficient conditions for Proposition respec- tively.
22

---

Published as a conference paper at ICLR 2021
Lemma 3 LLeft-log-concviit and symmetry imply monotonicity.). Let be symmetric about and let F be left-log-concate If A0) 2 0.5, then AF(T) is monotone increasing. Conversely, if (0) < 0.5, then (T(T) is monotone decreasing.
Proof. Consider the case where (0( 2 0.5; the case where Ap(0) I 0.5 is analogous. From Lemma 2 and the symmetry of f. we have that Ap (T) is monotone increasing if (and only if)
f(-)) f(r) f(2u- r)
(55)
F(-T) F(T) F(2u
holds for all 7 2 O.
To show that this inequality holds for all 2 0, we first note that since Ar(0) > 0.5, we have that F(0) < 0.5, which together with the symmetry of implies that μ 0. Thus, - S 2/ T for all O.
Now, if 2 μr then 2μ - - p.. so the desired inequality (55p follows from the log-concavity of on (-o0, pl] (Remark of Bagnoli & Bergstrom (2005.)
instead 0 S S ue, we have ((-r) F F(-r) F(T)
S(T)
[by log-concavity of F on (-o0, [4]]
(56)
f(T)
[since F(r) S 0.5]
(57)
1 1--F(T)
and thus (55) also holds.
Lemma 4 (Monotonicity and symmetry imply left-log-concavity.. Let f be symmetric with mean and let f be a translated version of f that has mean μ.. If As, is monotone increasing for all O. then f is left- concave.
left-log-concave.
Proof. First consider 2 0. Since As, is monotone increasing for all 2 0, from Lemma 2 and the symmetry of fn.
fu(-T) Fx(T) 12(2p-T)
(58)
Fx(-T) 1- FT(T) (2p
must hold for all 2 O. Since (x) = f(x - 11) by construction, we can equivalently write
Ι(-μ - T) T( F(-μ-T) 2 F(u T)
(59)
which holds for all 2 0 as well, and by assumption, for all μ 2 0 as well. By letting a -14
and т, we can equivalently write the above inequality as
f(b) F(a) F(b)
f(a)
(60)
for all a, 6 where a S b,a+b500
By the definition of log-concavity, this means that F' is log-concave on (-30,
D.2.1 LEFT-LOG-CONCAVIIT AND MONOTONICITY.
Proposition peet-llo--coccavng and monotonicity). Let F be the CDF of a symmetric distri- bution. If F is gefl-log-concuve then AF(T) is monotonically increasing and monotonically decreasing otherwise. Conversely, if AF, ) is monotonically increasing for all translations Fa such that Fo(T) = F(r-d) for all T and AF. (0) 2 1/2, then F is left-log-concaae. concave.
T if Ar 1/2
Proof. If F is left--og--oncave, by Lemma Ap is monotonically increasing in F if AF, (0) and monotonically decreasing otherwise. Conversely, if AF is monotonically increasing for all translations F, such that Fa(T) = F(T - d) for all by Lemma 4 F is conccveee completing the proof.
23

---

Published as a conference paper at ICLR 2021
D.3 SKEW-SYMMETRIC MARGIN DISTRIBUTIONS
We turn to how skew affects selective classification. Recall in Section S we defined skew- symmetric distributions as follows:
Definition 2. A distribution with density fa.uu is skew-symmetric with skew and center μs if
(T. = - (a)(--))(((TT
(4)
for all T E R. where h is the density of a distribution symmetric about 0, and G the CDF of potentially different distribution that is also symmetric about O.
When a 0, f h and there is no skew. Increasing a increases rightward skew, and decreasing increases leftward skew. Note that in general, the mean of f' depends a well. We will also consider the translated distribution:
(a, Sc((x 1t)-
(61)
The CDF of fa.u is Fa.p and the CDF of fa is For We will use the following properties of these distributions:
Lemma (Skew-symmetry about μ.). These properties hold when we flip the skew from to -a:
fa,((z) = f f-/1 x)
(62)
(a) = F-a, (2 x)
(63)
Proof.
(a,(()) = 2h(x ()((((r 14)
(64)
2h(p ==)(((-a)(u=)))
(65)
f-a (2μ - x).
(66)
(x) =
Sa,u (t)dt
(67)
I l-a,u(2c l)dl
(68)
(t)ct
(69)
20mmx
F F- (2c- x)).
(70)
Lemma (Stochastic ordering with a [Proposition from Azzalini & Regoli (012 Let 01 S 0.2 Then, for any R, For. 2 Faz
Proof. Since the ordering is invariant to translation, without loss of generality we can take First consider æ < 0. Then
0.
Fa, (x)
2h(t)G(a,t)dt
(71)
2h(t)G(azt)dt
(72)
Faz(2)
(73)
24

---

(T) Published as a conference paper at ICLR 2021
Now consider 2 2 0. We have
Pai (x) 2h())G(antddt
(74)
21(1)(( G(-ai
[by symmetry of G]
(75)
(x)x)
2h(t)G(-art)dt
(76)
[
= =2H(x)-
2h(- 2h(-t)G(-arttdt
[by symmetry of h]
(77)
= =2( 2)(t)G(ant)dt
[change of variables]
(78)
= (x) - 1+ Fat (-x),
(79)
which reduces to the where S 0.
Lemma 7 (Log gradient ordering by skew). For all a 20, 2 O, and E R,
(T) 2 s0,u(T)
2 F (T)
(80)
F (T) (T)
Proof. Since the ordering is invariant to translation, without loss of generality we can take We have:
0.
fa(r) h(r)G(ar)
(81)
Fa(T) [_ h(t)G(attdt
h(T)
(82)
1_ h()) G(ot) dt
G(a7)
h(T)
fo(T)
(83)
S h(t)dl So(T)
h(T)
(84)
S h(t) G(-at) dt
((T)G(-ar)
(85)
S h(t)G(
(-a(T)
(86)
F-a(T)
To get the inequalities, note that when 0 and < T, we have 2 0, G(at)/G((r)) < since G is an increasing function. Similarly, G( G(-at)/G(-an) 2 1. Since h is non-negative, the inequalities then follow from the monotonicity of the integral.
We now prove the main results of this section.
D.3.1 ACCURACY MONOTONE WITH SKEW
Proposition 6 (Accuracy is monotone with skew). Let Fo,, be the CDF of a skew-symmetric dis- tribution. For all 2 0 and μ E R, AFa. (T) is monotonically increasing in a.
Proof. We use the skew-symmetry of fa, to write the selective accuracy as
A Sa.p (7)
(87)
1 Fa.ul-T)
(7)
(88)
Fa,u Fa. (2 34-7)
25

---

Published as a conference paper at ICLR 2021
We see that A (T) is monotone decreasing function of Pa.a(-1) merator decreases with increasing a while the denominator increases. Thus, this fraction decreases, which in turn implies that Af,, (7) is monotone increasing with a desired.
F- (2p-T) From Lemma 6 the nu
D.3.2 SKEW IN THE SAME DIRECTION PRESERVES MONOTONICITY
Proposition 2 (Skew in the same direction preserves monotonicity). Let Fa.u be the CDF of a skew-
symmetric distribution. If accuracy of its symmetric version, A (T), is monotonically increasing in т, then A (T) is also monotonically increasing in for any >> 0. Similarly, if A (T) is monotonically decreasing in T, then F is also monotonically decreasing in T for any a < O.
Proof. The idea is Lemma to reduce the statement the case where 0. that can apply monotonicity. First consider the case where A A (T) is monotonically increasing in т, and O. We have
fa,u(-T)
fo,u(1
[Lemma
(89)
Fa,ue (-))
Fame( (-)
(x((-))
[definition of /]
(90)
Hn(-T)
hT(T)
[from monotonicity of (Lemma
(91)
HT(T)
(2p T)
[symmetry of h]
(92)
H₂(2μ - T)
fup 7)
[definition of /]
(93)
Fo T)
f- (2ps
[Lemma
(94)
F F-a.n (2 T) (T. 1- (T.
[Lemma
(95)
Applying Lemma completes the proof. The case where a < 0 analogous.
D.4 MONOTONE ODD TRANSFORMATIONS PRESERVE ACCURACY- CURVES.
We now show that our results are unchanged by strictly monotone and odd transformations to the density.
Lemma (Odd and strictly monotonically increasing transformations preserve selective accuracy).
Let X be real-valued random variable with CDF Fx, T be strictly monotonically increasing function, and define random variable Y =XX) with CDF Fy. Then, Arx (T) Ary (T(T)) for each t.
Proof. For each т, since is a strictly monotonically increasing function, we apply the change of variables formula for transformations of univariate random variables get the following:
Fx (T(r))
(96)
Fx(-7)= Fy (T(-T)) FY(((()))
[T is odd]
(97)
We now solve for selective accuracy:
1 - Ex(T)
AFx = 1- Ex(T) + Fx(-T)
(98)
1- 1-FY(T(T))
(99)
T- + (+(-((+))
= (TT))(
(100)
n
26

---

Published as a conference paper at ICLR 2021
With Lemma results extend all random variables T(X) where X corresponds to left-log- concave distribution and T is odd and strictly monotonically increasing. particular, X and T(X) have the same accuracy-coverag curves.
E PROOFS: COMPARISON TO THE GROUP-AGNOSTIC REFERENCE
In this section, we present the proof' from Section 6 which outline conditions under which selective classifiers outperform the group-agnosta reference.
E. DEFINITIONS
We first define certain metrics on selective classifiers and their group-agnostic reference in terms of their margin distributions.
Definition 6. Consider margin distribution with CDF
F
Σ 9E9
Pg Fg.
(101)
where Py the mixture weight and Fg is the CDF for group g. We write the fraction of examples that are correct/incorrect and predicted on at threshold T from group g and on average as follows:
C₂(T) 1 (T(T)
(102)
[G(T) Fg(-7)
(103)
C(r) C
(104)
I(r) Σ (T
(105)
We write the true-positive rate R (T) and false-positive rate (FP(F) for given threshold
C(T)
[T)
(106)
C(O))
1(T)
(F((T)
(107)
T(O)
Finally, we write the accuracy the group-agnostic reference on group g
AF, (T)
AF,(T) Ar, (0, (1 AF, (0))
(108)
(OC)(())))( C,00))C))(( + [T(())((T/TT()))))
(109)
For convenience in our proofs below, we also define the fraction of each group g in correctly and incorrectly classified predictions at each threshold t. Definition 7. We define the fraction of group out of correctly and incorrectly classified predictions at threshold T, CF (T) and IF (T) respectively, as
CF = CTC
(110)
C(T)
IF (T) = PT
(111)
T(T)
E.2 GENERAL NECESSARY CONDITIONS TO OUTPERFORM THE GROUP-AGNOSTIC REFERENCE
We now present the proofs for Proposition 3 and Corollary starting with the supporting lemmas.
We first write the accuracy of the group-agnostic reference in terms of IF wg (0) and CFwg(0). (0).
27

---

Published as a conference paper at ICLR 2021
Lemma 9.
Aran (T)
(112)
1+ IF,, (0) 1(1)
(0) C(T)
Proof. We have:
Ar (0)RT (T)
AFon(1 (T) = AP., (0) (T) + +(1 ART. (O))
(113)
DC (0)RT((T
(114)
wC (0) (T) + pIwg (0) (O)RF((T)
WC (0) C(T)
C(0)
(115)
wCng (0) C(r) C(O)
+ + (0) I(T)
T(O)
CF wg (T)
(116)
CF wg (T)C(T) + IF wg (T)
(117)
1+ (0) I(r)
(F) C(F)
Lemma 10 (Bounding the derivative of 1/CF we (T) at T = 0). If Fatro (0) 2 A (0).
d 1
1 (1-P)
F (0)
)
fwg (0) Fothers (0) - fothers (0) F Fw (0)
dr
CFFw(T)
T=U
p
I
Fwg
F (0)
(118)
Proof.
(119)
dr
CF Cg) rm
d
(TC + (1 - p) Cothers (T)
(120)
dr
pCCugl (T)
1=0
d
1-p' p
Pothers(7) (T)
1
(121)
dr
p
Fwg(1
r=0
p
d
1
l'otherss (T)
(122)
P
dr
1
Fwo(T (T)
==
1-P
[we(T)((
Fothers (T))
Sothers (T) (T)
(123)
P
(1 Fwg( (2)
7=0
1-P
1
(T)
1 Fothhrs(TT)
fothers (T)
(124)
p
Fwg(T)
1
Fug (T)
r==0
1-p
p 1
F Fothers(O))
fwg(( (wg(0)) (0)
fothers (0)
(125)
p
Fwg(0)
1
Fwg((0)
1 p 1
F Fothers (0)
[wg (w)
Sathers (0))
10
Fothers (0) < (wg(0) 1]
p
Fwg(()
Fwg (0)
(126)
1 1-p 1
Swg. (0) Fothers (0) Sathers (0) Fwg(O)
(127)
p
Fwg(0)
(w)
1- p
Fwg (0) fwg(0) F Fothers
(0) fothers (0) F (0)
(128)
p
Fwg(()
F Fwg (0)2
28

---

Published as a conference paper at ICLR 2021
Lemma 11. If A Fameer (0) 2 A (0) > 0.5, then
IF (T)
2 C' (0) wg (0)
fwg (0) otherss (0))
(129)
dr CF wg
==0
for some positive constant C.
Proof.
d IF wg (T)
(130)
dr
CF wg! (T)
r=0
d IF (T)
(4 dr CF wE (T)
+ IF wg (0)
(131)
dr
CF wa (0)
7=0
r=
2
WF (T)
IF
(132)
dr
CF wg (0)
r=0/
p
1 Fwg(O) ) [wgg (0) fothers (0) Fwg (0)
Fwg (0)
+ IF wg (0)
(133)
F Fwg((1) (0)
p P
[oathera (0) (0) fwg (O) (0)
(134)
+
1-p Father (0)
(0)2
p
F Fothers (0))
Fwg (0) )
(135)
P
L-P P)
Fwg (0)
Pwg(() ) fwz (O) (0) fothers (0) (0) Fwg (0)
I-p Fecters (0) Fux (0)
Pwg(O)
P
(Jothers (wg())(0)) - fwg (0) (o))
(136)
+ 1 1-p Pather (0) (
Fwg (0)2
>0
>0
>0
1 1 1 p Foehar (0)
1- (0)
Fwig (0)
1-p Fathess (0)
1 Fun (0)
(U)
because A Fung 0.)
21 bocause A Fathon (0)2 (0)
(fothers Fwg (0) - fwg Fothers
(137)
where 133p follows from Lemma TO
E.2.1 NECESSARY CONDITION FOR OUTPERFORMING THE GROUP-AGNOSTIC REFERENCE
Proposition 3 (Necessary condition for outperforming the group-agnostic reference). Assume that 1/2 < A (0) < A (0) < and the worst-grow gorst-group density fwg(0) > 0. If A (T) S (F for all 7 2 0, then
Sothers (0)
1 A Fatter 00)
(5)
fwg(() (0) AF (0)
Proof. Recall that
A Arr (T)
(138)
1
IF (F)(T))
[[r]
CF [7]
((7)
À F (T)
(139)
IF (U) 167
+
CF CF (0)
C(r)
29

---

Published as a conference paper at ICLR 2021
If (F) 2 AF for all 0, then
wg dr CF wwgg(? (T((T))
00
(140)
From Lemma 11
(fotherss (aneers(00))) (0) Fwg (0) fwg Fothers())) d IF wg (T)
(141)
dr CF Cgwg(T) 1=
for some positive constant C. Combined,
Sothers Fwg(() - Swg Fothersss (0) S.
(142)
E.2.2 OUTPERFORMING THE GROUP-AGNOSTEC REFERANCE REQUIRES SMALLER SCALING FOR LOG-CONCAVE DISTRIBUTIONS
Corollary (Outperforming the group-agnostic reference requires smaller scaling for log-concave
distributions). Assume that 1/2 A A (0) F (0) 1, Fwg is log-coneave, and (Tthers(T)
Fwg (U(T Mothers) + Wwg) for all T E I, where u is a scaling factor. If A (F) I A (T) for all
Proof. By Proposition 3 if A (T) 2 A (T) for all 7 2 0, then:
fothers (0) fwg Fothers (0) s
(143)
and equivalently,
fothers
Swg (0)
(144)
F Fothers (0) Fwg (0)
From the definition of fathers:
fwg(0)) Fwg (0)
(145)
fwg Mothers + Mugg )/ Fing Mothers? (wg))
Because F (0) > AF (0), Uothers Uwz 0. Applying log-concavity yields
v
(146)
Thus, when v > 1. there exists some threshold T 2 0 where ÁF,(T)
(T) > Arm (T).
E.3 TRANSLATED, LOG-CONCAVE DISTRIBUTIONS ALWAYS UNDERPERFORM THE GROUP-AGNOSTIC REFERENCE
We now present a proof of Proposition Assume fwg and fothers are log concave and symmetric, the worst group has mean Awg. the combination of other group(s) has mean Mothers and fothers(z) = fugl (x (Hothers [wg)). (the densities translations of each other.) For convenience, define
Mothers Hlug. This implics:
Fothers (xr Fmg(# (xr
(Mothers - (wgg))
(147)
Fwg d).
(148)
We will first show that wg is indeed the worst group at full coverage; i.e. A (0) < A A (0):
Lemma 12. If fwg is PDF and fothers obtained by translating fwg to the right (i.e. fothers()) fag(# - d) for d > 0), and Fug and F Fothers are their associated CDF's A (0) < A A (0).
Proof. We have:
A (0)
Fwg (0)
(149)
<
Pwg
(150)
F Fothers (0)
(151)
A Feste
(152)
30

---

Published as a conference paper at ICLR 2021
We next show that CF (Fgg(T) (T) is monotonically decreasing in
Lemma 13, If Swg. Fug: Sothers
Fothers Ps and CFws wa are as described, CF wg is monotonically
decreasing
Proof. First, defining CF wg in terms of Fwg. we have:
p(1 Fwg(T))
CF Cggg(T)
(153)
p( - (Tg + (1 - p)(1 - P Pothers (T))
p(1 Fwg(T))
(154)
p( Fwwe( (T)) + (1 - -P)(1- Pww(T (T= - d))
p FFw (2/wng - 7)
(155)
Fwg (2 - T) + (1 Fug (2 T + d)
(156)
(1 D. Fire! 2e - +.1) p. (2Mmng
Taking the derivative of CF
wg: have:
d dr
d
CF wgT()
(157)
dr
1 +
(I-p) pl (2tung +d)
pFaz (2 7)
2
p d
Fwgg (2 T + d)
(158)
1 (1 (I-p) Fag (2tton (21tan ++d) P. (2 ()
p
dr Fwg (2 - T)
=-c(- (-ffg (0 (2/wog + d) Fug (2
T) Swg (2)twgg T) Fwg (2
d))
(159)
where e is positive constant (since p and 1 p are positive and all of the terms are squares.) Thus, we can say that CF (T) is monotonically decreasing for all
Sug( (2pug d) Fwg! (2 T) S Swg (2 T) Fug (2/wwn
d)
(160)
[wg (2 + d)
[wg (2/lin - T)
(161)
Fig (2pwg
d) Fwg (2 T)
Now, since d O, this follows from the log concavity of fwg. Therefore, the fraction of correct examples that come from group 1 is a decreasing function of т.
Next, we show that the IF (g(()) monotonically increasing in T: Lemma 14. If Swg. Fwg. Sathers: Fathers: p and IF wg are described, IF wg is monotonically in- creasing in T.
Proof. First, we note that:
WF (-r)
IF wg (T) = plw (-7) + (1 p) Pwwgl d)
(162)
(163)
+ (1-p) (- D. (T)
We will show that the derivative of IF (7) with respect to is positive. Doing so gives us:
d
d
IF
wgg(r)
(164)
dr
dr + (1-p) (p ( (-r-d
pFugl-
2
1 11-1 d (Fng(-T -d)
(165)
1+ 1 p) Fag d-
F
pPac(-1 (-)
-c c---wg (- d)FF Fwe (7)) + Smns( (c7) Fwg) - d))
(166)
31

---

Published as a conference paper at ICLR 2021
where is positive since it is the product of squared terms and (1 p)/p, which is also positive.
Thus, dr IF wg (T) 2 0 is equivalent to:
- -(-fwg(- Fwg + fwg( (o((-T) Fug( d))
2
(167)
fwgl (T ( 2 fwg (-T) Fwg! Fwg(-T d)
(168)
Swg. T d) fwgl (-))
(169)
Fw -T d)
Fwgl (-T)
Since fwg is log-coneave and is positive, the last inequality is true, so IF wg (T) is monotonically increasing in 7, desired.
Lastly, show that Lemma 13 and Lemma 14 imply that the ratio creasing in
IF (7) CF cx.
is monotonically in-
Lemma 15.
IF (t) CFx)(T)
monotonically increasing in
Proof. We simply follow the quotient rule:
d IF (T) CF wg (Tg(F))F((() IF wg (rg(())F((T)
(170)
dr CFwg(T)
20,
(171)
where we note that CF wg/ (T), IF wg (T) 2 0, CF((T)
< 0 from Lemma and IF((T) > from
Lemma
We will now prove the main result of this section:
E.3.1 TRANSLATED, LOG-CONCAVE DISTRIBUTIONS ALWAYS UNDERPERFORM THE TROUP-AGNOSTER REFERENCE
Proposition 4 (Translated log-concave distributions underperform the group-agnostic reference).
Assume Fug and Fothers are log-concave and fothers(T) = fwe(T - d) for all T E R. Then for all
T
AF,, (T) S (T).
(6)
Proof. We consider the case of underperforming the group-agnostic reference: the case overper- forming the group-agnostic reference is analogous. Assume the worst group is the lower accuracy group (so d is positive.) Using the definition of selective accuracy, we have:
Cwg(T)
A (T)
(172)
(Tg(T) + (T(
CF wg.
(T)
(173)
CF wg. (T)((()((T)))))))))))))))))
IF wg (T)
(174)
[Fg((T) 1(T)
1 1+ CFag!T) C(T)
[by Lemma 15
(175)
IFog(0 [(r)
1+ CF C(F)
CF wg (0)C(7)
(176)
CF ( + IF wg (0)((T)
= =A (T).
(177)
Thus, the worst group underperforms the group-agnostic reference, as desired.
32

---

Published as a conference paper at ICLR 2021
F SIMULATIONS
To demonstrate that is possible but challenging to outperform the group-agnostic reference, we present simulation results margin distributions that mixtures of two Gaussians. Following the setup from Section we consider best-group margin distribution (1,,)) 1) and a worst-group margin distribution N (1, varying the parameters p. We set the fraction of mass from the worst group. p to be 0.5. We evaluate whether selective classification outperforms the group-agnostic reference, plotting the results in Figure ? We observe that selective classification outperforms the group-agnostic reference under some parameter settings, notably when the necessary condition in Proposition met and when the variance smaller as implied by Corollary [ but it underperforms the group- -ggnotic reference under most parameter settings.
1.50
Outperforms the group-agnostic baseline Underperforms the agnastic baseline Satisfies the necessary condition Best-group parameter
1.25
1.00
Worst-group
0.75
0.50
0.25
0.5
1.0
1.5
2.0
Worst-group standard deviation
Figure 9; Simulation results margin distributions that are mixtures of two Gaussians, where we vary the mean and the variance of the worst-group margin distributions. For parameters corre- sponding to the blue and orange regions, the worst group outperforms and underperforms the group- agnostic reference, respectively. We do not consider parameters the white region to maintain that they yield full-coverage accuracies that worse than the other group. We shade parameters that satisfy the necessary condition in Proposition
In addition whether the worst group underperlorms or outperforms the group-agnostic reference, we study the magnitude the difference in the worst- porst-group accuracy with respect to the group- agnostic reference through the same simulations (Figure TO). Concretely, compute the difference in worst-group accuracy with respect to the group-agnostic reference, taking the worst-case differ-
ence across thresholds: max, A (r) Ar, (T)- We observe significant disparities between the observed worst-group accuracy and the group-agnostio reference.
Difference in worst-group accuracy from the group-agnostic baseline
1.50
0.1
Other-group parameter
Worst-grom Worst-group g--rrupp mean
1.25
max (+) Ar.
1.00
0.75
0.50
0.00
0.5
1.0
1.5
2.0
Worst-group standard deviation
Figure 10: Simulation results on margin distributions that are mixtures of two Gaussians, where we vary the mean and the variance of the worst-group margin distribution, and keep the other group's margin distribution fixed with mean and variance 1. We compute the difference in worst-group accu- racy with respect to the sooupaggnsttc reference, taking the worst-case difference across thresholds: max, AF (7) AF (T).
Lastly, we simulate the effects of varying the worst-group and other other-group group means while keeping the variance fixed at の² 1 for both groups, following the same simulation protocol otherwise.
33

---

Published as a conference paper at ICLR 2021
We similarly observe substantial gaps between the observed worst-group accuracy and the group- agnostic reference (Figure IT.
Difference worst-group accuracy from the group-agnostic baseline
1.50
0.1
Worst-group mean
1.25
max Ara,, (T) - (F. (T)
1.00
0.75
0.50
0.25
0.00
0.0
0.5
1.0
1.5
Other-group mean
Figure 11: Simulation results margin distributions that mixtures of two Gaussians, where vary the means of the two margin distributions, keeping their variances fixed at 1. We compute the difference in worst-group accuracy with respect to the group-agnostic reference, taking the worst- case difference across thresholds: max, ÅF,, (T) AF (T).
34


Reviews:
[{"review_id": "wVdiJZN2JGH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper draws attention to a problem in the selective classification. In particular, it implies that selective classifiers can perform unnegligible accuracy disparities across various groups. To capture such information, the authors theoretically analyze the margin distributions and compare to the group-agnostic baseline, which can help to explain the accuracy disparities shown in the training result of 5 datasets. \n\nOverall, I like the idea of theoretically analyzing the accuracy-coverage curve by margin distribution and I vote for accepting.\n\nPros:\n1. The paper takes an important issue of selective classification: accuracy disparities between groups. For me, it's possibly related to ''fairness'' in classification, which we shall treat carefully in real-world problems. \n\n2. The proposed margin distribution for analysis is reasonable. The theoretical analysis is helpful to explain the observation in experiments. \n\n3. This paper provides experiments to illustrate their findings in typical datasets and defend their analysis.   \n\nBelow are my major concerns. Hopefully the authors can address them in the rebuttal period. \n\nCons:\n1. Although exploiting the margin distribution can provide good reasoning on the monotonicity of accuracy-coverage curves, analysis of comparison to group-agnostic baseline seems not very clear to me: \n\n(1) In that section, the authors consider the case of a mixture of two groups. Is this case typical enough to demonstrate the issues the authors want to imply?\n\n(2) In proposition 5, the authors show that log-concave distributions always underperform the baseline. However, from my view, the importance of this issue lies on that the disparity is unnegligible, so a more refined estimate of the difference between accuracies, namely, $|A_F(\\tau)-\\tilde A_F(\\tau)|$ (may need some extra conditions) will be more helpful to explain results on CheXpert-device.\n\n\nSome typos:\n1. Page 4: there are extra \"]\" in both C(tau) and I(tau).", "extra_scores": {}, "extra_text": {}}, {"review_id": "b47HsU8FDH5", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: This paper demonstrates and analyzes the accuracy disparities induced by selective classification on subgroups. It evaluates these induced disparities across five datasets from different domains, and provides a theoretical analysis of the conditions under which selective classification may decrease accuracy or magnify disparities via evaluation of the margin distribution.\n\nOverall, this paper is a useful addition to a growing literature on the disparate impact of various machine learning techniques on subgroups. The analysis in terms of margin distribution is both appropriate to the selective classification task, and novel, as far as I am aware. The theoretical analysis clearly describes conditions under which the various observed experimental results may generally hold. The (short) section on DRO also suggests a promising way out of this problem, informed by the results and analysis in the paper. The paper could use some minor edits and clarifications, but I believe it would be a valuable addition to the conference.\n\nNote: I did not evaluate or check all of the proofs.\n\n#### Major Comments:\n\n* I found the description of the baselines confusing; it is not made clear that C and I are being defined (I had to scan to see if they were previously discussed) and a great deal of text is used to describe a baseline construction procedure which could be encapsulated in a simple algorithm (my recommendation) or set of numbered steps.\n\n* Throughout, the paper uses a definition of \"average accuracy\" (defined in Sec. 3) which is not actually average accuracy -- it is average accuracy on covered points. I find this to be a bit misleading, or at the very least a confusing overloading of the term, and would recommend introducing a new name for this metric and re-labeling the y-axis of the plots that use it.\n\n* The DRO section (7) feels like an afterthought, but it is actually quite important to the paper. There are several recent works which identify disparate model performance impacts on subgroups (e.g. due to model compression, differential privacy, etc.), with no clear solution. The connection to DRO suggests a potential solution, at least in practice, for the problem identified in this paper. It would be nice to, at the very least, see some of the discussion of DRO upgraded from the appendix to the main text, if not a more thorough analysis.\n\n#### Minor comments:\n\n* I did not feel that the actual process of training a selective classifier (vs. a standard classifier) was clearly explained; since this is fundamental to the analysis, it would be useful to have this clearly signposted for readers unfamiliar with selective classification (such as this reviewer).\n\n#### Typos etc.\n\nSec. 7: \"to their average accuracies Figure 5\"\n\n####\n\nUPDATE: I have reviewed the author response and the revisions. All of my main concerns were addressed. I believe that these improved the paper further, and while they do not change my rating, the paper is still a clear \"accept\" in my view.", "extra_scores": {}, "extra_text": {}}, {"review_id": "QUcIv2vjboV", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper discusses the impact of classifier abstention on the performance obtained for different groups of data. In particular, the authors show that, while abstaining in case of uncertainty in general leads to an improvement in predictive performance, it can also lead to worse results for specific groups. In a somewhat atypical paper structure, the authors first illustrate this empirically on five banchmark datasets. Later in the paper, they also present theoretical results that explain the empirical results. \n\nThis is an interesting paper about an original research topic. The take-home message is not so surprising, but the analysis is thorough. I particularly appreciated that the empirical results are also supported by theoretical results. I didn't do a detailed check of the proofs, but I don't see a reason why the propositions should not hold. \n\nOverall the paper is also well written and I enjoyed reading it. A few things are not very clear to me:\n\n1. In Section 3 it is briefly mentioned that two scores are considered as a representation of uncertainty: softmax and a score based on MC dropout. For the latter it is not clear to me how the score is computed. Is this the variance over different predictions for an instance? Moreover, in Figures 2 and 4 it is not mentioned which of the two scores is used to obtain the graphs. I would have expected to see the results for softmax in the main paper, since this is most often used as uncertainty score, but the horizontal axis in Figure 4 is not limited to the interval [-1,1], so this cannot be softmax. This needs to be explained better. \n\n2. Related to the previous remark, I would also argue that the margin distributions depend very strongly on the uncertainty measure that is used. As discussed e.g. in the work of Kendall and Gal, softmax looks at aleatoric uncertainty, whereas MC dropout  looks at epistemic uncertainty. Furthermore, also a lot of other uncertainty measures exist in the literature. For example, in multi-class classification, one often constructs models that abstain on instances that are very dissimilar to the training data (so-called out-of-distribution samples). Thus, different measures represent different types of uncertainty, so I also expect to see very different margin distributions on specific datasets. \n\n3.  In the description of the experimental setup, I found the notion \"spurious attribute\" unclear. For \"spurious correlation\" it is clear what is meant, but I have never heard of a spurious attribute, and I also don't know what that would be. Aren't you just referring to a confounding attribute, as commonly used in the statistical literature? Does it make a difference for the analysis whether the group variable is observed or unobserved? It might be good to elaborate on this. \n\n4. It is also not very clear what the practical usefulness of the obtained results is. It is nice to know that the performance can drop for specific groups when abstention is allowed, but this is not necessarily a bad thing. In practice one encounters a lot of situations where machine learning models are used as a prescreening tool, followed by a manual check for cases that are hard to classify by models. In such situations one could decide to use a particular algorithm only for specific groups, whereas badly-scoring groups would deserve a special treatment, such as a manual screening. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "0mBB_3HGZO", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper mainly focuses on reporting a cautionary finding on using selective classification, but didn’t provide any solution. That is, selective classification can magnify existing accuracy disparities between various groups within a population, especially when there are spurious correlations. The authors studied the margin distribution to get some understandings about the problem. \n\nIf the disparity across different groups is important, performing a uniform standard classification over the unified data is not a good idea in the first place. It seems the group disparity problem is inherent to the full coverage classification and should be addressed at the full coverage in the first place, e.g., by using group-DRO models.  This is consistent with their finding (which is intuitive) that selective classification can uniformly improve group accuracies on group-DRO models, because the problem is inherent to the classification model used. \n\nAlthough the authors tried to analyze their findings using margin distribution under the concept of the left-log-concave distribution, which does not lead to any solution for solving the disparity problem over the worst-group, but remains an observation. Moreover, it is unclear whether all the phenomena of selective classification magnifying disparity can be captured in their concepts and analysis, since their findings are based on empirical observations over a few datasets. \n\nOverall I feel the current observation report requires further effort. It would be natural to expect the analysis on observations can lead to a solution for the observed problem. \n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This is an interesting paper discussing the impact of classifier abstention on the performance obtained for different groups of data. The reviewers are either very (scores of 8, 7 and 7) or moderately (score of 5) positive about the paper. The main concern is that the paper does not directly propose a solution for the discovered problems. Nevertheless, it can initiate interesting discussions and research around them.
Decision: Accept (Poster)