Calibrated Selective Classification
Authors: Adam Fisch
Tommi S. Jaakkola
Regina Barzilay
Venue: Accepted by TMLR ()

Abstract: Selective classification  allows models to abstain from making predictions (e.g., say ``I don't know'') when in doubt in order to obtain better effective accuracy. While typical selective models can succeed at producing more accurate predictions on average, they may still allow for wrong predictions that have high confidence, or skip correct predictions that have low confidence. Providing calibrated uncertainty estimates alongside predictions---probabilities that correspond to true frequencies---can be as important as having predictions that are simply accurate on average. Uncertainty estimates, however,  can sometimes be unreliable. In this paper, we develop a new approach to selective classification in which we propose a method for rejecting examples with ``uncertain'' uncertainties. By doing so, we aim to make predictions with well-calibrated uncertainty estimates over the distribution of accepted examples, a property we call selective calibration. We present a framework for learning selectively calibrated models, where a separate selector network is trained to improve the selective calibration error of a given base model. In particular, our work focuses on achieving robust calibration, where the model is intentionally designed to be tested on out-of-domain data. We achieve this through a training strategy inspired by distributionally robust optimization, in which we apply simulated input perturbations to the known, in-domain training data. We  demonstrate the empirical effectiveness of our approach on multiple image classification and lung cancer risk assessment tasks.

Published in Transactions on Machine Learning Research (12/2022)
Calibrated Selective Classification
Adam Fisch
fisch Ascc&acsal.c mit. edu
Tommi Jaakkola
tommi@csail. mitt edu
Regina Barzilay
regina mit, edu
Computer Science and Artificial Intelligence Laboratory (CSATL)
Massachusetts Institute of Technology, Cambridge, MA, 02142, USA.
Reviewed on OpenReview: https: // openreiew.. net/ met/ forun? id- 2FANBs8GaV
Abstract
Selective classification allows models to abstain from making predictions (e.g,, say "I don't know") when in doubt in order to obtain better effective accuracy. While typical selective models can succeed at producing more accurate predictions on average, they may still allow for wrong predictions that have high confidence, or skip correct predictions that have low con- fidence. Providing calibrated uncertainty estimates alongside perdictions-probablltte that correspond to true frequencies can be as important as having predictions that are simply ac- curate average. Uncertainty estimates, however, can sometimes be unreliable. In this paper, we develop a approach to selective classification in which propose method for reject- ing examples with "uncertain" uncertainties. By doing so, we aim to make predictions with well-calibrated uncertainty estimates over the distribution of accepted examples, property we call selective calibration. We present a framework for learning selectively calibrated models, where a separate selector network trained improve the selective calibration error of given base model. In particular, our work focuses on achieving robust calibration, where the model is intentionally designed to be tested on out-of-domain data. We achieve this through training strategy inspired by distributionally robust optimization, in which apply simulated input perturbation ions to the known, in-domain training data. We demonstrate the empirical effective- ness of our approach on multiple image classification and lung cancer risk assessment tasks.' 1
1
Introduction
Even the best machine learning models can make errors during deployment. This especially true when there are shifts in training and testing distributions, which we can expect to happen in nearly any practical scenario Ceuionero-Candele approach to mitigating the negative consequences of palentially wrong predictions, by allowing models ab- stain uncertain inputs in order to achieve better accuracy (Chow, 1957; El- Llavv & Wiener, 2010; Geifman & El- El-Yaniv, 2017). While such systems may indeed be more accurate on average, they still fail to answer a critical question: how reliable are the uncertainties that they use to base their predictions? Often it can be important to precisely quantify the uncertainty in each prediction via calibrated confidence metrics (that reflect the true probability of an event interest, such as our model being correct). This particularly relevant in situations in which user cannot casily reason about themselves in order to verify predictions; in which there is inherent ran- domness, and accurate predictions may be impossible; or in which high-stakes decisions made, that require weighing risks of alternative possible outcomes (Amodci et al., 2016; Jiang et al., 2018; Ovadia et al., 2019).
al., 2009; Rabanser al., 2019; Koh al., 2021). Selective classification an
Consider the task of lung cancer risk assessment, where a model is asked if a patient will develop cancer within the next few years (Aberle et al., 2011). A standard selective classifier that aims to maximize accuracy may
'Qur code available at http://///aaaaaa..../a//////a//a

---

Published in Transactions on Machine Learning Research (12/2022)
0: Flag
vs.
g(X)
E[Y ((( 0.3, g(X)
0.3
1: Keep
0: Not at risk
vs.
)X)= 0.3
Y=1
Y=0
Reject
A: risk
Figure 1: A demonstration of calibrated selective classification for lung cancer risk assessment. The input X is a CT scan; the output Y indicates if the patient develops cancer within 6 years of the scan. f (vellow) provides a confidence estimate for the patient being at risk, while g (red) acts as a gatekeeper to decide when to trust f's outputs. Our goal is for the accepted predictions be calibrated. For example, of all patients with predicted (and non-rejected) risks of 0.3, 30% should be expected to develop cancer within years of the scan.
only make predictions on examples with clear outcomes; predictions that are less useful to the radiologists who these models are meant to help (Dromain al., 2013; Choy al., 2018). Furthermore, even if the selective classifier is accurate, it may fail to reliably reflect the risk variations of different patients. For example, suppose that patient has 30% chance of developing cancer in reality. The prediction that the patient will not develop cancer will most likely be correct. However, knowing the degree of uncertainty in this outcome- namely that the patient's 's true risk of developing cancer is plan. For these reasons, having a calibrated confidence estimate (in this case, that reflects the true likelihood of developing cancer) can be key to guiding clinical decision making (Jiang el al., 2012; Tsoukalas el al., 2015).
dramatically change the patient's care
We develop a new approach to selective classification in which we aim to only make predictions with well-calibrated uncertainties. In lung cancer setting, this means that if the model predicts that cancer will develop with confidence for a group of palients, then fraction of them should indeed develop cancer. Under our framework, model should abstain if it is uncertain about its confidence, rather than risk misleading user.² In other words, we shift the focus from abstaining on uncertain examples, to abstaining on examples with "uncertain" incertainties. Figure 1 illustrates our approach.
Concretely, consider a binary classification task (we consider multi-class tasks later) with input space X and label space y = {0, 1}, where inputs X E X and labels Y E y are random variables drawn from joint distribution P. Suppose we have model f X + [0, where fXX represents the model's confidence that =1 1, while 1 - -(X) represents the confidence that Y = O. Following the selective classification setting of Geifman & El-Van (2017), we assume that f is fized (c.g., it may be black-box or expensive to re-train) and we are now simply seeking a safe gate-keeping mechanism for its predictions. Typical calibration stipulates that Va in the range of f we have E[Y fXx) = a] = a, but this may not be satisfied by the provided f. Instead, here we let g: X + {0, 1] be our selection model, where we make a prodiction if g(X) = 1, and abstain otherwise. Our goal is to then make predictions that are conditionally calibrated, i.e.,
E [ [ (XX) = a, g(X) =1
Va E [0, 1] in the range of restricted to {a : g(x) = 1}.
(1)
We refer to this property as selective calibration. It is also important to preserve some minimum amount of prediction coverage for f to still be useful (e.g., we may want lo make predictions on al least. 80% of inputs). As a result, we seek to become as relatively calibrated as possible by minimizing the selective calibration error of (fig), ie., deviation from Eq. (1), subject user-specified coverage constraint on g. Inspired by the trainable Maximum Mean Calibration Error (MMCE) objective of Kumar et al. (2018), Maximum Mean Calibration Error (S-MMCF)- (easily estimated) upper bound of it that we compute in practice, in order to improve the selective calibration error. This loss can be effectively trained over a relatively small subset of held-out examples (e.g., s O(1²)))
propose the Selective with an efficient method for training g to minimize a
2 Alterratively, the model may make prediction, but should abstain from providing (or raise red flag about) its confidence.
2

---

Published in Transactions on Machine Learning Research (12/2022)
Still, the key empirical question that remains, is what subset of held-out examples can we train 9? We typ- ically do not know the test distribution, P Piest, that we may encounter, and our goal is for g to generalize beyond simple in-domain examples (for which standard, non-selective, calibration is more readily obtained). To address this, we formulate a robust calibration objective using syuthetically constructed dataset shifts. Specifi- cally, given any original training data Dusin ~ Purain that we do have, and some family T of perturbation that, we can define (e.g., via common data augmentation techniques), we minimize the S-MMCE over the uneali- brated datasets that we create by applying perturbations in T to the data in Dirain. Of course, we do not expect these augmentations to represent all possible test distributions Piest- Rather, by constructing broad, albeit still limited, set of example shifts, our intention is for 9 to learn to generalize well to new perturbation types, be- yond those seen during training. (naturally, g's generalization abilities will depend on how feasible it is to create a meaningful perturbation family T). Empirically, when training and testing on disjoinl sets of both simulated and naturally occurring perturbations on diverse datasets, we demonstrate consistent empirical reductions in se- lective calibration error metrics relative to typical confidence-based basclines across multiple tasks and datasots.
Contributions. To summarize, the main contributions of this work are as follows:
1. We introduce the concept of selective calibration as a selective classification objective;
2. We propose a coverage-constrained loss for training selectors to identify well-calibrated predictions:
3. We provide robust framework for training selectors that generalize to out-of-distribatioo data;
4. We demonstrate consistent empirical reductions in selective calibration error metrics (e.g., 18-25% reduction in &g calibration error AUC) relative to standard heuristic basclines across multiple tasks and datasets.
2 Related work
Selective classification. Selective classífication, or classification with reject option, altempts to abstain on examples that the model is likely to get wrong Yaniv & Wiener, 2010; Geifman & El- El-Yaniv, 2017). The lit- erature on selective classification is extensive (Hellman, 1970; De Stefano et al., 2000; Herbei & Wegkamp, 2006; Cortes et al., 2016a;b; Ni et al., 2019; Geifman & El- Yaniv, 2019). A straightforward and popular technique is to use some confidence measure to select the most certain examples (Cordella et al., 1995; Yaniv & Wiener, 2010; Geifinan& & EI- Yaniv, 2017). If the underlying confidence measure is unreliable, however, this approach can per- form poorly (Jones et al., 2021). As in our setting, Kamath et al. (2020) consider seleetive classification in the context of domain shift, though they focus on accuracy, rather than calibration. Approaches to selective classi- fication that build beyond simple accuracy include cost-scnsitive selective classification (Bartlett & Wogkamp, 2008; Charoenphakdee et al., 2021), selective classification optimized for experts in-the-loop (Mozannar & Son- tag, 2020), and selective classification with fairness constraints (Shah et al., 2021); all of which represent com- plementary directions to our work. Lin et al. (2022) also considers the possibility of a non-uniform significance over different classes, and provide a sct-based selective classifier with controlled class-spocific miscoverage rates. Here, we exlend selective classification to focus on improving model calibration over non-rejected instances.
Model calibration. Calibration has a rich history in machine learning (Brier, 1950; Murphy & Epstein, 1967; Dawid, 1982; Foster & Vohra, 1998; Gneiting et al., 2007). Recently, it has begun to experience resurgence in Kuleshov et al., 2018; Kumar el al., 2019; van Amersfoort et al., 2020; Gupta et al., 2020), partly motivated by observations that modern neural networks can be significantly miscalibrated out-of-the-box (Guo et al., 2017; Ashukha et al., 2020). To start, a number of efforts have focused on how to best define and measure calibration, especially in multi-class settings. A common approach (e.g., as taken by Guo et al. (2017)) measures the top-label calibration error, or the probability that the model's top prediction is correct (a definition we adopt). Other methods propose more precise notions of calibration, including calibration that is conditioned the predicted class (Gupta & Ramdas, 2022), across classes (Kull et al., 2019), for nub-populations or individuals (Hlebert-Johnson et al., 2018; Zhao et al., 2020), or with respect to decision making (Zhao et al., 2021). Additionally, conformal prediction provides tools for calibrating set-based predictions such that they provably satisly fixed risk limits (e.g., cover the correct label) with some specified probability (Vovk et al., 2005; Bates et al., 2020; Angelopoulos et al., 2022). In the case of providing calibrated probability estimates, conformal predictors can also be modified to output (scts of) probability predictions with the same coverage properties (Vovk& & Petej, 2014; Vovk et al., 2018). We note that aspects of these methods are complementary to our work, and could be incorporated in selective setting
the deep learning literature (Kuleshov & Liang, 2015;
3

---

Published in Transactions on Machine Learning Research (12/2022)
following, similar to, our framework. Finally, approaches to achieving better calibration over single point estimates include both parameteric and non-parametric post-processing techniques (Platt, 1999; Zadrozny & Elkan, 2001; Niculescu-Mizil & Caruana, 2005; Guo et al., 2017; Kumar et al., 2019), as well as modified losses introduced during model training (Kumar ct al., 2018; Mukhoti et al., 2020; Karandikar et al., 2021). Here, we atlempt to achieve better conditional calibration across domains through selective classification.
Robustness to domain shift. It is well-known that model performance can often degrade when the testing and training distributions different- and many methods have been proposed to help combat this poor behavior (Sugiyama & Müller, 2005; Wen et al., 2014; Reddi et al., 2015; Lipton et al., 2018; Tran et al., 2022). With respect to model calibration, even models that are well-calibrated in-domain can still suffer from considerable miscalibration under domain shift. (Ovadia et al., 2019; Minderer et al., 2021). Given some knowledge or assumptions about the possible new target distribution (e.g., such as an estimated likelihood ratios or maximum divergence) several works have considered corrections for calibration under distribution shift (Tibshirani et al., 2019; Cauchois et al., 2020; Gupta et al., 2020; Park et al., 2022). Similarly, given samples from two distributions (e.g., the training and testing distributions)) many lests exist. for detecting if they are different (Vovk et al., 2003; Gretton et al., 2012; Chwialkowski et al., 2015). A shift in input features, however, is neither necessary nor sufficient as a predictor of accuracy; methods such as Ginart et al. (2022) attempt to monitor when distribution drifts are severe enough to indeed cause performance drop, in order to trigger a decision to collect more supervised data. Closer to our work, a number of approaches have also attempted robust training (e.g, for temperature scaling) over multiple environments or domains (Wald et al., 2021; Yu et al., 2022), rather than relying on test-time adjustments. Our work adds to the broad array of tools in this area by providing robust selective classification mechanism with few assumptions or requirements.
Synthetic perturbations. Data augmentation is commonly used to improve model performance and generalization (Hendrycks et al., 2020; Rusak et al., 2020; Cubuk et al., 2020; Buslaev et al., 2020). Similarly, automated methods have been developed for identifying challenging sub-populations of a larger training set to target bias (Liu et al., 2021; Bao & Barzilay, 2022). Mcanwhile, we create perturbed datasets by sampling different augmentation types, in order to uncover relationships between types of perturbation and model calibration. Our line of work is a consumer of the other: as new approaches are developed to expand the space of possible perturbations, the more tools our framework has to use for robust training.
3 Background
We briefly review selective classification and calibration. We use upper-case letters (X) to denote random vari- ables; lower-case letters (x)) to denote scalars; and script letters (2) to denote sets, unless otherwise specified.
3.1 Selective classification
Returning to the setting in $1, let f: X' be our prediction model with input space X and label space Y, and 1ct g: X + {0, 1} be a binary selection function over the input space X. Let P be a joint distribution over X x V. For now, we make no distinction between Perain and Pustt and simply assume a general p PPrain Peett. A selective classification system (z) at an input x X can then be described by if f((x==1 1,
(f.g)(z)= = { f(x)
(2)
"abstain" otherwise.
Selective classifier performance is commonly evaluated in terms of risk versus coverage (El-Yaniv & Wiener, 2010). Coverage is defined as the probability mass of the region of X that is not rejected, Exg(X)]. []. In practice coverage is usually tunable via threshold E R, where given some soft scoring function g: X' + IR, g is defined as g(x) == (()() 2 r}. Given some loss function C, the selective risk with respect to P is then defined as
R(f.9; P) = E((((())) s(X) 1] (CCF(X)))))(X)) EXg(X))
L is typically the 0/1 loss, making R(f.g gi P) the selective error. As evident from Eq. (3), there is a strong dependency between risk and coverage. Rejecting more examples can result in lower seloctive risk, but also lower coverage. The risk-coverage curve, R(f, g; P) at different (X)())), and its AUC provide a standard way to evaluate models. We will also use the AUC to evaluate the culibratioon-ooerrag curve of our proposed method.
4

---

Published in Transactions on Machine Learning Research (12/2022)
3.2 Model calibration
We consider two simple marginal measures of binary and multi-class calibration error. Starting with the binary setting, let f: X + [0, 1] be our model, where f(X) is the confidence that Y = 1. The binary calibration error measures the expected difference between the estimate f(X), and the true probability of Y given that estimate.
Definition 3.1 (Binary calibration error). The bg binary calibration error of f w.r.t. P is given by
BCE(f:q, = (> 1 [(X))
(4)
Typical choices for the parameter q 2 are 1, 2, and oc (we will use both q == 2 and q = oc). We now turn to the multi-class case, where y [l...., K} is a set of K classes, and the model f: X [0, 1j outputs confidence score for each of the K classes. A common measure of calibration is the difference between the model's highest confidence in any class, maXye(K] J(X)y, and the probability that the top predicted class, arg yE[]] g()),, is correct given that estimate (Guo et al., 2017; Kumar et al., 2019):*
Definition 3.2 (Top-label calibration error). The lg top-label calibration error of f w.r.t. P is given by
TCE(f:g, = E B = arg max f(X)y) } | max f(X)yl - max f(X)y19)) vE(K]
(5)
yE[K]
DE(K]
In practice, we measure the empérical calibration error, where BCE and TCE approximated (see Appendix B). Having zero calibration error, however, does not imply that f is useful model. It is still important for to be a good predictor of Y. For example, the constant S(X) == E[(Y is calibrated, but is not necessarily accurate. As another measure of prediction quality, we also report the Brier score (Brier, 1950):
Definition 3.3 (Brier score). The mean-squared error (a.k.a., Brier score) of f w.r.t. P is given by
Brier f; P) = (((((X)-Y) Y)
(6)
In multi-class scenarios, we sot f(X) to [ f(X), and Y 1(Y arg max yE[K] f(X)y}.
4 Calibrated selective classification
We now propose an approach to defining and optimizing a robust selective calibration objective. We begin with definition selective calibration for selective classification ($4.1). We then present a trainable objective for selective calibration (54.2), together with a coverage constraint ($4.3). Finally, we construct framework for training robust selective classifiers that encounter domain shift at test time ($4.4). For notational convenience, we focus discussion binary classification, but evaluate both binary and multi-class tasks in $6. As previously stated, we assume that f is a pre-trained and fined black-box, and only train for now.
4.1 Selective calibration
Once more, let X' be our input space and y {0, 1} be our binary label space with joint distribution P. Again, we assume for now that P is accessible (i.e., we can sample from it) and fixed. When classification model f: X + [0, 1] is paired with selection model g: X + {0, 1}. we define selective calibration follows: Definition 4.1 (Selective calibration). A binary selective classifier (F.9) is selectively calbbrated w.r.t. P if
E y f(x) a, 9(X) 1] a, Va E [0 in the range of f restricted to (x g(x) 1}.
(7)
Similarly, we also define the selective calibration error, which measures deviation from selective calibration. Definition 4.2 (Selective calibration error). The Eg binary selective calibration error of (f.g) w.r.t P is
S-BCE(J. P) = ER. BIY (SX), g(X) = 1 (x(x) = =1)}=
(8)
3 As discussed $2 many different definitions multi-class calibration have been proposed in the literature. For simplicity, focus only on the straightforward TCE metric, and leave extensions more precise definitions future work.
5

---

Published in Transactions on Machine Learning Research (12/2022)
The top-label selective calibration error for multi-class classification is defined analogously, as is the selective
Brier score. What might we gain by selection? As basic observation, we show that the covernge-consttrinee selective calibration error of (f.g) can, at least in theory, always improve over its non-selective counterpart. Claim 4.3 (Existence of a good selector). For any fized f: X - [0. 1), distribution. P, and coverage & E (0, 1], there exists g: X' + { such that (i) [X] 2 &, and (i) S-BCE(J 9i 9;q, P) - BCE(fiq,PP)
See Appendix A for a proof. Note that in some (e.g., adversarially constructed) cases we may only be able to find trivial (i.e., the constant g(X) = 1 which recovers BCE(J;q, P) exactly), though this is typically not the case. For intuition on how an effective selection rule may be applied, consider the following toy example. Example 4.4 (Toy selting). Assume that. input X = (X1, X2) E [0, 1] x 10,1} follows the distribution Px = (0, 1) Bern(p) for some p 2 g (0, 1), and output Y E [0,1] follows the conditional distribution
Bern (ern(X1)
if X₂ 1,
prix: =
(9)
Bern (eemmmi((())))) X1 + A,1))
if X2 = 0,
for some A > Further assume that the given confidence predictor is simply S(X) = X1. That is, by design, [(X) X2 = 1 is calibrated, while f(X) X2 = 0 has an expected calibration error of A. Marginally, f(X) will have an expected calibration error of R A1--)) >> O. It is then easy see that defining the simple selector o(X) = X₂ will result in selectively calibrated model (f.9) with expected selective calibration error of O.
Extending this intuition to more realistic settings, our key hypothesis is that there often exist some predictive features of inputs that are likely to be uncalibrated (e.g., when X2 = 1 from Example 4.4) that can be identified and exploited via a selector g. In the next sections, we investigate objectives for learning empirically effective g-
4.2 A trainable objective for selective calibration
Our calibration objective is based on the Maximum Mean Calibration Error (MMCE) of Kumar et al. (2018). Let H denote a reproducing kernel Hilbert space (RKHS) induced by a universal kernel k(;,-) with feature map Φ: [0, 1] + H. Let R == f(X) E [0, 1]. The MMCE of f w.r.t. and kernel l is then
MMCF(;;k; P) = [ - -R)((R)))))))))))))) R)) H
(10)
where ||x11x denotes norm in the Hilbert space H. Kumar et al. (2018) showed that the MMCE is iff is almost surcly calibrated, a fundamental property we call faithfulness. Transitioning to selective classification, we now propose the Selective Maximum Mean Calibration Error (S-MMCE). For notational convenience, define the conditional random variable V == f(X) I 9(X) =16 [0, 1]. The S-MMCE of (f.g) w...t. Pand kernel k is then
S-MMCE(J.9g.. k. P) = B [ [V] VV
(11)
S-MMCE involves only few modifications to MMCE. The iterated expectation in Eq. (11), however, makes the objective difficult to estimate in a differentiable way. To overcome this, instead of directly trying to calculate the S-MMCE, we formulate and optimize a more practical upper bound to S-MMCE,
S.MMCE..ff = ||||I - V
H = -|
= EIY ||BIY
(x)))((XX((((X)) B[g(X))
(12)
As with MMCE and other kernel-based IPMs (such as MMD, see Gretton et al. (2012)), we can take ad advantage of the kernel trick when computing finite sample estimate of S-MMCE3 over mini-batch D = (0 He where (I). are drawn i.i.d. from P. Denote f(x) E [0, 1] as The empirical estimate is
G.MCEEUU. D) = (ENEED (( wu: r rul - (0
(13)
EneeD (xx))9(1,)
Intuitively, we can see that a penalty is incurred when n; and xy with similar confidence values rú, r, are both selected, but have ra, that are misaligned with the true labels ye, By. In the remainder of the paper we will informally collectively refer to S-MMCE, S-MMCF,, and the empirical estimate simply the S-MMCE.
6

---

Published in Transactions on Machine Learning Research (12/2022)
4.2.1 Theoretical analysis
We briefly highlight a few theoretical properties of S-MMCE, which are mainly analogous to those of MMCE.
See Appendix A for proofs. These results also hold in the multi-class case when following a multi-classtt--binarr reduction (as in the TCE). First, we show that S-MMCE a faithful measure of selective calibration.
Theorem 4.5 (Faithfulness). Let k a universal kernel, and let q 2 1. The S-MMCE is then 0 if and only if (..) almost surely selectively calibrated, i.e., P( V] - V) 1, where == =(x) I g(X) 1.
While faithfulness is appealing, we note that S-MMCE = is not always achievable for a fixed ʃ (i.e.; no selector 9 may exist). Next, we show a connection between S-MMCE and the expected selective calibration error.
Proposition 4.6 (Relationship to S-BCE). S-MMCE < SBBBE where K maxxae(01) k(a, a).
As can be expected, lower S-BCE leads to lower upper bound on S-MMCE (both are 0 when (f.g) is calibrated). Lastly, we show that S-MMCF,, which we always minimize in practice, upper bounds S-MMCF. Proposition 4.7 (S-MMCE upper bound). For any (f.9), P, and q 2 1 we have S-MMCE S S-MMCE,.
4.3 Incorporating coverage constraints
S-MMCE depends heavily on the rejection rate of g(X). As discussed in $1, we desire g to allow predictions on at least some g-fraction of inputs. This leads us to the following constrained optimization problem:
minimize S-MMCE(f.g;. k, s.t. B[g(X)] 2g
(14)
g
A typical approach to solving Eq. (14) might be to convert into an unconstrained problem with, e.g., a quadratic penalty (Bertsekas, 1996), and then to train a soft version of g, g: X - [o, 1], together with some form of simulated annealing to make it asymptotically discrete (as desired). However, we find that we can effectively change the problem slightly (and simplify it) by training the soft selector ğ the S-MMCE loss without the denominator that depends on g(X). Instead of explicitly training for a specific &. we simply use a logarithmic regularization term that prevents ğ from collapsing to 0 (we will later re-calibrate it for E), i.e.,
Greg(fgg D) = A1 Σ 1y - r My, - (1,1) 7
"- A2
Σ log (a:)) RED
(15)
I.EDD
Regularized S-MMCE. loss for training soft g.
where λι, A2 2 O are hyper-parameters. This also has the advantage of avoiding some of the numerical instability present in Eq. (13) for small effective batch sizes (i.e,, when either the batch size is small, or the rejection rate relatively large). The continuous ğ is then discretized as g(x) == 1(ā(x) 2 f), similar to the selective models in $3.1. can conveniently be tuned to satisfy multiple & (versus being fixed at training time), follows. Let S be a set of g(x) scores over an additional split of i.i.d. unlabeled data (ideally, test data). We set f Threshold((, S), where Threshold(g, S) is the largest value that at least E-fraction of S is greater than:
Threshold(g, = sup { E :: S
1
Σ 11 SES
(16)
While targeting the lowest viable coverage (i.c., = E vs > {) isn't necessarily optimal, we empirically find be good strategy. For large enough |8], we can further show that g will have coverage E with high probability. Proposition 4.8 (Coverage tuning). Let Deune == (X ........ X,,} be set of i..dd unlabeled random variables drawn from distribution P. Let random variable f == Threshold(E, (ē(X;): X, Drune)). Note that & is constant given Dune- Further assume that random variable x(X) is continuous for X ~ P. Then Ve > 0,
P( 2 + Deune) E
(17)
where the probability is over the draw of unlabeled threshold tuning data, D Drune-
7

---

Published in Transactions on Machine Learning Research (12/2022)
Algorithm 1 Robust training for calibrated selective classification.
Definitions f is the confidence model. Ptrain is sample of available training data. T is (task- family of perturbations that can be applied to Derain to simulate potential test-time domain shifts. Drune is an additional sample of unlabeled data available to us. If possible, Drane should be drawn from the lest-lime distribution. & is the larget coverage.
function TRAIN(J, Dirain, Donne: T.E)
2: b Initialize a soft selector model ğ: X' - [0, 1].
($1.3)
3: Define the binarized selector model, g(æ)= == (()()) 2 r), for some 7 IR that we will choose.
4: for iter = 1,2,... do
($4.4)
5: Randomly sample batch of m perturbation functions from T, Brunc == {ti T)
6: Apply each perturbation function 4 to Durain, obtaining Baiittdd == [4. o Dirain : 4, E B
7:
Calculate the selective calibration error of (f,g) for Q E Batitted using T = Threshold(S, 1ā(z): x E 2.)).
8:
(Optional) Identify the k-worst datasets, 2x (1) E Behiteed i 1, 1....: K; sorted by selective calibration error. b Take a gradient step for I using the r-worst Q (or all, if not L8), x 1 Cme(S. 9: k. 2x(1), see Eq. (15). Stop if 9 has converged, or if maximum number of iterations has been reached.
9:
10:
11: Set final f on Doune, where f == Threshold(S, (ō(z): Drune }), see Eq. (16).
($4.3)
12:
return ( = 1 [g(x) 2 f}
To summarize, the basic steps for obtaining g are to (1) train a soft 9 via Eq. (15), and (2) tune a threshold to ob- tain a discrete via Eq. (16). Proposition 4.8 shows that this gives us a model with close to our desired coverage with high probability. Still, performance may start to degrade when distributions shift, which we address next.
4.4 Learning robust selectors via simulated domain shifts
Until now, we have treated the distribution P as constant, and have not distinguished between a Perain and a Prest- A main motivation of our work is to create selective classifiers (f.g) that can effectively generalize to new domains not seen during training, i.e., Ptrain # Ptest, so that our models are more reliable when deployed in the wild. We adopt a simple recipe for learning a robust g for a fixed f through data augmentation. Let T be a fam- ily of perturbations that can be applied to a set of input examples. For example, in image classification, T can color adjustments. T need not solely include functions
include label- preservig geometrie transformations of individual examples. For example, in lung cancer risk assessment, randomly resample Dtrain according to the hospital each scan was taken in, to simulate covariate shift (without directly modifying the input CT scans).
Let. Dtrain == (I,, 1: ~ Purain be a split of available training data (ideally distinct from whatever data was used to train f). For a given perturbation t E T, lct Q == t a Dirain denote the result of applying t to Dtrain- Depending on the severity of the perturbation that is used (as measured relative to f), predictions on Q. may suffer from low, moderate, or high calibration error. Motivated by distributionally robust optimization (DRO) methods (Levy et al., 2020), optimize selective calibration error across perturbations by (1) sampling a batch of perturbations from T, (2) applying them to Derain and making predictions using f, (3) optionally identifying the k-worst selectively calibrated perturbed datasets in the batch when taking at coverage E,² and then (4) optimizing S-MMCE only over these k-worst examples (or all examples otherwise). 5 See Algorithm 1. Our mothod can be viowed as a form of group DRO (Sagawa et al., 2020), combined with ordered SGD (Kawaguchi & Lu, 2020). unique aspect of our up, however, is that our groups are defined over general functions of the data (i.e., requiring robustness over a broad set of derived distributions), rather than a fixed set of identified sub-populations of the data. Furthermore, the number of functions applied to the data can be combinatorially large- for example, when defining T to be comprised of compositions of set of base transformations.
4.5 Selector implementation
We implement 9 as a binary MLP that takes high-level meta features of the example x as input. Specifically, we define ğ == o (FF((eeetal (x))), where (()) is a sigmoid output, FF a 3-layer ReLU activated feed-forward neural
When not optimizing for one & (e.g., when sharing 9 for multiple f). we measure the selective calibration error AUC. 5 Not all pertarbations cause performance degradation, hence the motivation only optimize over the top-x worst per batch.
8

---

Published in Transactions on Machine Learning Research (12/2022)
network with 64 dimensional hidden states, and (meta()) extracts the following assortment of fixed features," 6
many of which are derived from typical out-of-distribation and confidence estimation method outputs:
Confidence score. max po(y 2), where po(y x)) is the model estimate of p(y | x) (i.e., equal to f())..
Predicted class index. One-hot encoding of arg max (v x) E {0,1)* where K 2.
Prediction entropy. The entropy, Σx= (v xx) log Ix), of the model prediction, where K 2 2.
Full confidence distribution. Raw model estimate pa(y x)) E [0, 1]K where K X22 2.
Kernel density estimate. 1D estimate of p(z) derived from a KDE with a Gaussian kernel.
Isolation forest score. 1D outlier score for x derived from an Isolation Forest (Liu et al., 2008).
One-class SVM score. 1D novelty score for æ derived from One-Class SVM (Schölkopf et al., 2001).
Outlier score, ID novelty score for x derived from a Local Outlier Factor model (Brennig et al., 2000).
kNN distance. ID outlier score for x derived from the mean distance to its k nearest training set neighbors.
We also use a subset of the derived features individually as baselines in $5.3. For all derived scores, we use the base network's last hidden layer representation of x as features. If the number of classes K is large (e.g., Ima- geNet where K = 1000), we omit the full confidence distribution and the one-hot predicted class index to avoid over-fitting All models are trained with q = for S-MMCF-basse losses. For additional details see Appendix B.
5
Experimental setup
5.1 Tasks
CIFAR-10-C. The CIFAR-10 dataset (Krizhevsky, 2012) contains 32 x 32 x 3 color images spanning 10 image categories. The dataset has 50k images for training and 10k for testing. We remove 5k images each from the training set for validation and perturbation datasets for training I. Our perturbation family T is based on the AugMix framework of Hendrycks et al. (2020), where various data augmentation operations (e.g., contrast, rotate, shear, etc) are randomly chained together with different mixture weights and intensities. Our base model f is WideResNNet40022 (Zagoruyko & Komodakis, 2016). The last layer features for deriving Qmeta (2) are in IR228 We lemperature scale J after training clean validation data (we use the same underlying set of validation images for training both f and 3).We evaluate on the CIFAR-10-C dataset (Hendrycks & Dietterich, 2019), where CIFAR-1O-C is a manually corrupted version of the original CIFAR-10 test set. CIFAR-10-C includes a total of 15 noise, blur, weather, and digital corruption types that are applied with 5 different intensity levels (to yield 50k images per corrupted test set split). Note that these types of corruptions are not part of the AugMix perturbation family, and therefore not seen during training, in any form.
ImageNet-C. The ImageNet dataset (Deng et al., 2009) contains 1.2 million color images (scaled to 224 x 224x x 3) spanning 1K image categories. We use standard off-the-shelf ResNet-50 (He et al., 2016) for f that was trained on the ImageNet training set. We then simply reuse images from the training set to create separate perturbed validation and training datasets for training G. and use the same perturbation family as used for CIFAR-10 above (i.e., based on AugMix). We also use the clean validation data for temperature scaleing. The last layer fcatures for deriving Pmeta (2) are projected down from 2048 128 using SVD. Since the number of classes is large, we omit the predicted class index and full confidence distribution from our selector input features. We evaluate on the ImageNet-C dataset (Hendrycks & Dietterich, 2019), which applies many of the same corruptions done in CIFAR-10-C. Again, none of these types of corruptions are seen during training.
Lung cancer (NLST-MGH). As described in $1, in lung cancer risk assessment, lung CT scans are used to predict whether or not the patient will develop a biopsy-confirmed lung cancer within the 6 years following the scan. In this work we utilize, with permission, data and models from Mikhael et al. (2022), all of the which was subject to IRB approval (including usage for this study). The base model for ff is a 3D CNN trained on scans from the National Lung Screening Trial (NLST) data (Aberle et al., 2011). We then use a separate split of 6,282 scans from the NLST data to train g. The last layer features for deriving Qmeeta (I) are in IR 256 As the NLST
WW opt use fixed, simple, meta features most of our training sets are small.
9

---

Published in Transactions on Machine Learning Research (12/2022)
data contains scans from multiple hospitals (33 hospitals in total), we are able to construct a perturbation family T using randomly weighted resamplings according to the hospital each scan is taken from. We evaluate our selective classifier on new set of 1,337 scans obtained from Massachusetts General Hospital (MGH), taken from patients with different demographics and varied types of CT scans. See Appendix B.3 for additional details.
5.2 Evaluation
For each task, we randomly reinitialize and retrain five times, resample the test data with replacement to crc- ate five splits, and report the mean and standard deviation across all 25 trials (# models x # splits). In order to compare performance across coverage levels, we plot each metric as function of Ç, and report the AUC (starting from a coverage of 5% to maintain a minimum sample size). Our primary metric is the Selective Calibration error AUC (lower is better). We report both the &g and &oo error, written as S-BCE, and S-BCE respec- tively (and, accordingly, S-TCE2 and 5) Our secondary metric is the Selective Brier Score AUC (lower is better). Thresholds for establishing the desirod coverage are obtained directly from the test data, with the labels removed. and perturbation type), including temperature scaling. Shaded areas in plots show + standard deviation across trials, while the solid line is the mcan. AUCs are computed using the mean across trials at cach coverage level.
are tuned on development data held out from the training sets (both data
5.3 Baselines
Full model. We use our full model with no selection applied (i.e., g(x)) = 1 always). All examples are retained, and the selective calibration error of the model is therefore unchanged from its original calibration error.
Confidence. We use the model's raw confidence score to select the most confident examples. For binary models, we define this as max(f(xx,, ((x),, and maxyε]]]] f(xx,, for multi-class models. Often, but not always, more confident predictions can also be more calibrated, making this a strong baseline.
Outlier and novelty detection models. To test if our Iearned method improves over common houristics, we directly use the features from $4.5 to select the examples that are deemed to be the least anomalous or outlying. Specifically, we compare to isolation forest scores, one-class SVM scores, and training set average kNN distance by simply replacing 9 with these outputs." 7 We then apply the same thresholding procodure in Eq. (16).
6 Experimental results
Selective calibration error. Figure 2 reports our main calibration results. In all cases, we observe that optimizing for our S-MMCF objective leads to significant reductions in measured calibration error over the selected examples. In particular, S-MMCE-derived scores outperform all bascline scoring mochanisms in both f2 and lo Selective Calibration Error AUC, as well as in Selective Brier Score AUC. This is most pronounced on the image classification tasks-which we conjecture is due to the effectiveness of AugMix perturbations at simulating the test-time shifts. Note that the lung cancer task has far less data (both in training and testing) than CIFAR-10-C or ImageNet-C,, hence its higher variance. Its perturbation class (reweighted resampling based on the source hospital) also relatively weak, and limited by the diversity of hospitals the NI.ST training set. Despite this, S-MMCE still improves over all baselines. Interestingly, while some baseline spproaches-such rejecting based on one-class SVM scores or isolation forest scores -an lead to worse calibration error than the full model without abstention, predictors produced using S-MMCE have reliably lower selective calibration error (per our goal) across all coverage rates &. The fact that our method significantly outperforms distance-based baselines, such as kNN distance or Isolation Forests, is also encouraging in that it suggests that has learned behaviors beyond identifying simple similarity to the training set.
Robustness to perturbation type. Figure 3 shows breakdown of Selective Calibration Error AUC across all 15 lest time perturbations for CIFAR-10-C (note that Figure 2 shows only averages across all perturbations). Naturally, the reduction in calibration error is more pronounced when there is significant calibration error to be- gin with (c.g., see Gaussian noise). That said, across all perturbation types, we observe substantial decreases in Seleetive Error AUC, both relative to the full model without abstentions, and relative to a standard confidence- based selection model. Similar results are also seen for ImageNet-C; which we provide in Appendix C.
We omit entropy, KDF, and LOF scores brevity as they perform similarly or worse compared to the other henristics.
10

---

Published in Transactions on Machine Learning Research (12/2022)
0.175
3.20
0.140
0.123
STCC
0.15
0.03
Rull (0.162)
Peseline (0.161)
Isolation Forest (0. (0.195)
One Class SVM (0.133)
0.000
Isolation Forest (0.278)
One Class SVM (0.135)
0.10
Cin--Class SVM (0.18/)
Confidence
Distance
0.25
Confideece (.105)
(0.123)
0.05
KNN Distance (0.183)
Confidence (3 15.35
0.00
S-MMCE (0.070)
0.000
S-MMICE (0.092)
0.00
Selective (0:124)
62
044
0.8
07
0.6
0.4
0.4
0.6
0.5
1.0
Elgt/]
=[g(X1)
Coverage
0.40
(0
Ispletton Forest 10.296)
0.30
0.1
0.35
Ora--Clavs SUM (D:716)
Confidence 10.1721
0.30
Distance
S/NCCE (0.118)
0.25
S-BCE.
0.20
0.2
0.15
Full
Bascline (0.277)
0.19
Isolation Forest (0.271)
Isolation Forest (0 2909
0.30
One-Class SVM (0.234)
One Class SVM (0.200) 209)
0.05
0.05
KNN Distarce (0.199)
Contidence (0.239)
Confidence 243)
(972
0.00
0.00
S-MAACE (0.153)
Selective (0.215)
0.2
0.4
0.6
0.8
1.0
0-2
4.
2.0
0.4
0.6
0.6
=[g(K1]
=tg(X))
0.20
0.20
0.30
Solactue Brier Store
0.55
selectve Brier Score
0.75
2.38
2.30
Selecive
0.10
0.15
Rull (0.172)
Full (0. 168)
Baseline (0.196)
Isotation Forest (0.210)
Isolation Forest (0.181)
0.10
Isolation Forest (0.207)
3.05
Class SVM (0.176)
0.05
One-Class (0.147)
Oni Closs SVM
Confidence 131)
Contidence
0.25
Confidence (0 (00
KNN Distance (0.80 (9.99)
KNN Distarce (0 152)
KNN Distance (0 (0.198)
S-MMCE (0. (0.082)
5: S-aVAce (0.050)
0.00
Seleciive (0.108)
07
Din
0.2
stgiXi)
slgtX)
EtaiX)
(a) CIFAR-10-C
(b) ImageNet-C
(c) Lung cancer (NLST-MGH)
Figure 2: Coverage vs. selective calibration error and Brier scores on CIFAR-10-C, ImageNet-C, and lung cancer data. For CIFAR-1O-C and ImageNet-C, we report average results across all perturbation types. For lung cancer, we report results on the diverse MGH test population. Empirically, across all coverage levels and metries, rejections based on g optimized for S-MMCE perform the best (or on par with the best in some cases).
Analysis of rejection behavior. We empirically investigate the types of sclections that g trained to optimize S-MMCE make, compared to typical confidence--bssed selections. For the lung cancer risk assessment task MGH data, Figure 4 shows the empirical distribution of confidence scores at various coverage levels & conditioncd on the example X being seleeted (o(X) = 1). As expected, the confidence-bass only the most confident examples, which in this case, is primarily examples where the patient is deemed least likely to be at risk (f(X)= x 0). Figure 4 (b) then demonstrates that an undesirable outcome of this approach is that a disproportionate number of patients that do develop cancer (Y = 1) are not serviced by the model (g(X)= = 0). In other words, the model avoids making wrong predictions on these patients, but as a result, doesn't make predictions on the very examples it is needed most. On the other hand, our S-MMCF-based model clcarly does not solcly reject examples based on raw confidence-thhe distribution of confidence scores of accepted examples matches the marginal distribution (i.e., without rejection) much more closely. Similarly, as demonstrated in Figure 5, this selector rejects relatively fewer examples where Y = =1 as compared to the confidence-based method. Together with the low calibration error results from Figure 2, this suggests that it is more capable of giving calibrated predictions both for non-cancerous and cancerous patients alike.
method seleets
11

---

Published in Transactions on Machine Learning Research (12/2022)
Tull
Confidence
S-MMCE
S-TCE; AUC
0.25
0.20
0.15
0.10
0.05
0.00
ganistinn, noist
shol, poise
Ingudee poioe
detocus,
glass,
mation, tour
zoemn
so00
trightness
contrad
ebettic transfenm
Biselate
gog compressio Dean
Average-
Figure 3: lg selective top-label calibration error AUC reported for each of the 15 test perturbations in CIFAR- 10-C (in addition to the average). Optimizing S-MMCE leads to significant error reductions across perturba tooooonation types, relative to a model without abstentions ("Full",, as well as the standard selective classifier ("Confidence")
Extimated Denilty
Fotmated Dendiy
Extimated
1000 |gXi
(a) Coverage 0.90.
(b) Coverage E 0.70
(c) Coverage 0.50.
Figure 4: Empirical distribution of f(X) 9(X) for different coverage rates MGH data (for f(X) S 0.4 for visualization). Empirically, by selecting the most confident predictions, confidence-based predictions mainly take examples that are thought to be not be cancerous (i.e., where fxx x P(Y X = x) is low). The behavior of the S-MMCE- based classifier, however, is less skewed towards only selecting examples of a particular confidence value, and f(X) g(X) more closely follows the marginal distribution of f(X) without selection.
Additional results. We briefly highlight a few addi- tional results that are included in Appendix C. Figure C.3 presents an ablation on CIFAR-10-C that explores the relative importance different components our training procedure. Interestingly, even g trained minimize the S-MMCE on in-domain data yield improvements in selective calibration error on out-of-domain data, though the results are significantly improved when including the perturbation family T. Figure C.2 shows the effect of first training f on perturbations from T (though our focus is on black-box f). We use a model that both trained and temperature scaled with AugMix perturbations on CIFAR-10. In line with Hendrycks et al. (2020), this leads to f(X) with lower calibration error CIFAR-10-C. Still, our framework lcads to substantially lower selective calibration error AUC, relative to this new baseline. We also report standard selective accuracy scores Figure C.5. Note that more calibrated classificrs are not necessarily more accurate. Indoed, on ImageNet-C, where S-MMCE yields the most selectively calibrated
6.
S-MNICE
6.-
c.-
Rejeted
No Cencer
Figure Rejection ratios by label type & = 0.90 on MGH data. Blue denotes the ratio of each class in the full data. Proportionally more of the confidence-baseee rejections are cancerous (presumably as they are "harder" to classify). S- MMCE rejections are relatively less imbalanced.
12

---

Published in Transactions on Machine Learning Research (12/2022)
predictions, it does not yield the best selectively accurate predictions. Still, accuracy is not always at odds with calibration (especially if the higher confidence scores are also calibrated): in CIFAR-10-C and the lung data, S-MMCE also achieves close to the best selective accuracy.
Limitations and future work. We note a few considerations that are unaddressed by this work. Marginal calibration does not account for differences between sub-populations or individuals. This is both relevant to fair- ness and personalized decision making (Zhao et al., 2020; Shah et al., 2021; Jones et al., 2021). While we do not study them here, algorithms for more complete notions of calibration (Hebert-Johnson et al., 2018; Vaicenavi- cius et al., 2019; Shah al., 2021), can be incorporated into our framework. We also note that the performance of our method is strongly coupled with the perturbation family T, which, here, is manually defined per task. This can fail when the family T is not straightforward to construct. For example, the image augmentation such as those in AugMix (Hendrycks et al., 2020) do not generalize across modalities (or even other image tasks, nec- essarily). Still, as tools for creating synthetic augmentations improve, they can directly benefit. our algorithm.
7 Conclusion
Reliable quantifications of model uncertainty can be critical to knowing when to trust deployed models, and how to base important decisions on the predictions that they make. Inevitably, deployed models will make mistakes. More fundamentally, not every query posod to even the best models can always be answered with complete certainty. In this paper, we introduced the concept of selective calibration, and proposed a framework for identifying when to abstain from making predictions that are potentially uncolibrated -ieeedddieee..... when the model's 's reported confidence levels may not accurately reflect the true probabilities of the predicted events. We showed that our method can be used to selectively make predictions that are better calibrated as a whole, while still being subject to basic coverage constraints. Finally, our experimental results retonstrated that, (1) our calibrated selective classifiers make more complex abstention decisions than simple confidence-basee selective classificrs, and (2) are able to generalize well under distribution shift.
Acknowledgements
We thank Dmitry Smirnov, Anastasios Angelopoulos, Tal Schuster, Hannes Stärk, Bracha Laufer- Goldshtein, members of the Regina Barzilay and Tommi Jaakkola research groups, and the anonymous reviewers for helpful discussions and feedback. We also thank Peter Mikhael for invaluable support on the lung cancer risk assessment experiments. AF is supported in part by the NSF GRFP and MIT MLPDS.
References
Denise R Aberle, Amanda M Adams, Christine D Berg, William C Black, Jonathan D Clapp, Richard M Fagerstrom, Ilana F Gareen, Constantine Gatsonis, Pamela M Marcus, JoRean D Sicks, National Lung Screening Trial Research Team, and Reginald F. Munden. Reduced lung-cancer mortality with low-dose computed tomographic screening. New England Journal of Medicine, 365(5):395-409) 2011.
Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Concrete problems in ai safety. ArXis preprint: (666.06565, 2016.
Anastasios N. Angelopoulos, Stephen Bates, Adain Fisch, Lihua Lei, and Tal Schuster. Conformal risk control. ArXie preprint: 2208. 02814, 2022.
Arsenii Ashukha, Alexander Lyzhov, Dmitry Molchanov, and Dmitry Vetrov. Pitsalls of in-domain uncertainty estimation and ensembling in deep learning. In International Conference on Learning Representations (ICLR), 2020.
Yujia Bao and Regina Barzilay. Learning to split for automatic bias detection. ArXiv preprint: 2204. 1874., 2022.
Peter L. Bartlett and Marten H. Wegkamp. Classification with a reject option using a hinge loss. Journal of Machine Learning Research, 1893) 1840, 2008
13

---

Published in Transactions on Machine Learning Research (12/2022)
Stephen Bates, Anastasios Nikolas Angelopoulos, Lihua Lei, Jitendra Malik, and Michael I. Jordan. Distribu- tion free, risk controlling prediction sets. ArXiv preprint: 2101.02703, 2020.
Dimitri P. Bertsekas. Constrained Optimization and Lagrange Multiplier Methods (Optimization and Neural Computation Series). Athena Scientific, edition, 1996. ISBN 1886529043.
Markus M Breunig, Hans-Peter Kriegel, Raymond T Ng, and Jörg Sander. Lof: identifying density-based local outliers. In ACM sigmod record, volune 29, pp. 93 104. ACM, 2000.
Glenn W. Brier. Verification of forecasts expressed in terms of probability. Monthly Weather Review, 78:1-3, 1950.
Alexander Buslaev, Vladimir I. Iglovikov, Eugene Khvedchenya, Alex Parinov, Mikhail Druzhinin, and Alexandr A. Kalinin. Albumentations: Fast and flexible image augmentations. Information, 11(2), 2020.
Maxime Canchois, Suyash Gupta, Alnur Ali, and John C. Duchi Robust validation: Confident predictions cven when distributions shift. ArXiv preprint: 2008. 04267, 2020.
Nontawat Charoenphakdee, Zhenghang Cui, Yivan Zhang, and Masashi Sugiyama. Classification with rejection based on cost-sensitive classification. In International Conference Machine Learning (ICML), 2021.
C. K. Chow. An optimum character recognition system using decision functions. IRE Transactions on Electronic Computers, 6:247 254, 1957.
Garry Choy, Omid Khalilzadch, Mark Michalski, Synho Do, Anthony E. Samir, Oleg S. Pianykh, J. Raymond Geis, Pari V. Pandharipande, James A. Brink, and Keith J. Dreyer. Current applications and future impact of machine learning in radiology. Radiology, 288(2):318-33) 328, 2018.
Kacper P Chwialkowski, Aaditya Ramdas, Dino Sejdinovic, and Arthur Gretton. Fast two-sample testing with analytic representations of probability measures. In Advances in Neural Information Processing Systems (NeurIPS), 2015.
L.P. Cordella, C. De Stefano, F. Tortorella, and M. Vento. A method for improving classification reliability of multilayer perceptrons. IEEE Transactions on Neural Networks, 6(5):1140-115 1147, 1995. doi: 10.1109/72. 410358.
Corinna Cortes, Giulia DeSalvo, and Mehryar Mohri. Boosting with abstention. In Advances in Neural Information Processing Systems (NeurIPS), 2016a.
Corinna Cortes, Giulia DeSalvo, and Mchryar Mohri. Learning with rejection. In International Conference on Algorithmic Learning Theory (ALT), 2016b.
Ekin Dogus Cubuk, Barret Zoph, Jon Shlens, and Quoe Le. Randaugment: Practical automated data augmentation with a reduced search space. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
A. P. Dawid. The well-calibrated bayesian. Journal of the American Statistical Association, 770339):005-610, 1982.
Claudio De Stefano, Carlo Sansone, and Mario Vento. To reject or not reject: That is the question an answer in case of neural classifiers. IEEE Transoctions on Systems, Man, and Cybernetics, Part C' (Applications and Revieus), 30:84 94, 03 2000.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Conference on Computer Vision and Pattern Recognition (CVPR), 2009.
C. Dromain, B. Boyer, R. Ferré, S. Canale, S. Delaloge, and C. Balleyguier. Computod-aided diagnosis (cad)
in the detection of breast cancer. European Journal of Radiology, 82(3):417-422 2013.
14

---

Published in Transactions on Machine Learning Research (12/2022)
Ran El-Yaniv and Yair Wiener. On the foundations of noise-free selective classification. Journal of Machine Lenrning Research (JMLR), 11, 2010.
Dean Foster and Rakesh V. Vohra. Asymptotic calibration. Biometrika, 85(2):379-33 390, 1998.
Yonatan Geifman and Ran Fl-Yaniv. Selective classification for deep neural networks. Advances Neural Information Processing Systems (NewrIPS), 2017.
Yonatan Geifinnn and Ran El-Yas Yaniv. SelectiveNet: A deep neural network with an integrated reject option. In International Conference on Machine Learning (ICML), 2019.
Tony Ginart, Martin Jinye Zhang, and James Zou. MLDemon: systems. In International Conference. on Artificial Intelligence and Statisties (ATSTATS), 2022.
Deployment monitoring for machine learning
Tilmann Gneiting, Fadoua Balabdaoui, and Adrian E. Raftery. Probabilistic forecasts, calibration and sharpness. Journal of the Royal Statistical Society Series B, 69(2):243 268, 2007.
Arthur Gretton, Karsten M. Borgwardt, Malte J. Rasch, Bernhard Schölkopf, and Alexander Smola. A kernel two-sample test. Journal of Machine Learning Research (JMLR), 13(25))723-773, 2012.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. International Conference on Machine Learning (ICMT., 2017.
Chirag Gupta. and Aaditya Ramdas. Top-label calibration and multiclass-to-binaryy reductions. In International Conference on Learning Representations (ICLR), 2022.
Chirag Gupta, Aleksandr Podkopaev, and Aaditya Ramdas. Distribution-free binary classification: prediction sets, confidence intervals and calibration. In Aduances in Neural Information Processing Systems (NeurIPS), 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
Ursula Hebert-Johnson, Michael Kim, Omer Reingold, and Guy Rothblum. Multicalibration: Calibration for the (Compuaaliinallyydeeddllaaae) masses. In International Conference on Machine Learning (ICML), 2018.
Martin E. Hellman. The nearest neighbor classification rule with reject option. IEEE Transactions Systems Science and Cybernetics, 6:179-185, 1970.
Dan Hendrycks and Thomas Dictterich. Benchmarking neural network robustness to common corruptions and perturbations. International Conference on Learning Represendations (ICLR), 2019.
Dan Hendrycks, Norman Mu, Ekin D. Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshminareyanan. AugMix: A simple data processing method to improve robustness and uncertainty. Proceedings of the International Conference on Learning Representations (ICLR), 2020.
Radu Herbei and Marten H. Wegkamp. Classification with reject option. Canadian Journal of Statistics, 34 (4):709 721, 2006.
Heinrich Jiang, Been Kim, Melody Guan, and Maya Gupta. To trust or not to trust classifier. In Advances in Neural Information Processing Systems (NewrIPS), 2018.
Xiaoqian Jiang, Melanie Osl, Jihoon Kim, and Lucila Ohno Machado Calibrating predictive model estimates to support personalized medicine. Journal of the American Medical Informatics Association, 19(2):263 274, 2012.
Erik Jones, Shiori Sagawa, Pang Wei Koh, Ananya Kumar, and Percy Liang. Selective classification can magnify disparities across groups. In International Conference on Learning Representalions (ICLR), 2021.
15

---

Published in Transactions on Machine Learning Research (12/2022)
Amita Kamath, Robin Jia, and Perey Liang. Selective question answering under domain shift. In Annual Meeting of the Associalion for Compulational Linguisties (ACT.), 2020.
Archit Karandikar, Nicholas Cain, Dustin Tran, Balaji Lakshminarayanan, Jonathon Shlens, Michael Curtis Mozer, and Rebecca Roclofs. Soft calibration objectives for neural networks. Tn Advances in Newral Information Processing Systems (NewrIPS), 2021.
Kenji Kawaguchi and Haíhao Lu. Ordered sgd: A new stochastic optimization framework for empirical risk minimization. In International Conference on Artificial Intelligence and Statistics (AISTATS), 2020.
Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, Tony Lee, Etienne David, Ian Stavness, Wei Guo, Berton Earnshaw, Imran Haque, Sara M Beery, Jure Leskovee, Anshul Kundaje, Emma Pierson, Sergey Levine, Chelsea Finn, and Percy Liang. Wilds: A benchmark of in-the-wild distribution shifts. In International Conference on Machine Learning (ICML), 2021.
Alex Krizhevsky. Learning multiple layers of fcatures from tiny images. University of Toronto, 2012.
Volodymyr Kuleshov and Percy Liang. Calibrated structured prediction. In Advances in Neural Information Processing Systems (NeurIPS), 2015.
Volodymyr Kuleshov, Nathan Fenner, and Stefano Ermon. Accurate uncertainties for deep learning using calibrated regression. In International Conference on Machine Learning (ICML), 2018.
Meelis Kull, Miquel Perelló-Nieto, Markus Kängsepp, Telmo de Menezes e Silva Filho, Hao Song, and Peter A. Flach. Beyond temperature scaling: Obtaining well-calibrated multi-class probabilities with dirichlet calibration. In Advances in Neural Information Processing (NeurIPS). 2019.
Ananya Kumar, Percy Liang, and Tengyu Ma. Verified uncertainty calibration. In Aduances in Neural Information Processing Systems (NeurIPS), 2019.
Aviral Kumar, Sunita Sarawagi, and Ujjwal Jain. Trainable calibration measures for neural networks from kernel mean embeddings. In International Conference on Machine Learning (ICML.) 2018.
Daniel Levy, Yair Carmon, John C Duchi, and Aaron Sidford. Large-scale methods for distributionally robust optimization. In Advances in Neural Information Processing Systems (NewrIPS), 2020.
Zhen Lin, Lucas Glass, M. Brandon Westover, Cao Xiao, and Jimeng Sun. SCRIB: Set-classifier with class-specific risk bounds for blackbox models. In AAAI Conference on Artificial Intelligence (AAAT), 2022.
Zachary Lipton, Yu-Xiang Wang, and Alexander Smola. Detecting and correcting for label shift with black box predictors. In International Conference on Machine Learning (ICML). 2018.
Evan Z Liu, Behzad Haghgoo, Annie S Chen, Aditi Raghunathan, Pang Wei Koh, Shiori Sagawa, Percy Liang, and Chelsea Finn. Just train twice: Improving group robustness without training group information. In Proceedings of the 38th International Conference on Machine Learning (ICML), 2021.
Fei Tony Liu, Kai Ming Ting, and Zhi-Hua Zhou. Isolation forest. In IEEE International Conference on Data Mining, 2008
Peter G. Mikhael, Jeremy Wohlwend, Ludvig Karstens Adam Yala, Justin Xiang, Angelo K. Takigami, Patrick Bourgouin, PuiYee Chan, Sofiane Mrah, Wael Amayri, Yu-Hsiang Juan, Cheng-Ta Ta Yang, Yung- Liang Wan, Gigin Lin, Lecia V. Sequist, Florian J. Fintelmann, and Regina Barzilay. Sybil: A validated deep learning model to predict future lung cancer risk from a single low-dose chest computed tomography, 2022.
Matthias Minderer, Josip Djolonga, Rob Romijnders, Frances Ann Hubis, Xiaohua Zhai, Neil Houlsby, Dustin Tran, and Mario Lucic. Revisiting the calibration of modern neural networks. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
16

---

Published in Transactions on Machine Learning Research (12/2022)
Hussein Mozannar and David Sontag. Consistent estimators for learning to defer to an expert. In International Conference on Machine Learnöng (TCMT.), 2020.
Jishnu Mukhoti, Viveka Kulharia, Amartya Sanyal, Stuart Golodetz, Philip Torr, and Puneet Dokania. Calibrating deep neural networks using focal loss. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
Allan H. Murphy and Edward S. Epstein. Verification of probabilistic predictions: A brief review. Journal of Applied Meleorology and Climalology, 6(5):748 - 755, 1967.
Chenri Ni, Nontawat Charoenphakdee, Junya Honda, and Masashi Sugiyama. On the calibration of multiclass classification with rejection. In Advances in Neural Information. Processing Systems (NeurIPS). 2019.
Alexandru Niculescu-Mizn and Rich Caruana. Predicting good probabilities with supervised learning. In International Conference on Machine Learning (ICML), 2005.
Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, D. Sculley, Sebastian Nowozin, Joshua Dillon, Balaji Lakshminarayanan, and Jasper Snock. Can you trust your model's uncertainty? evaluating prodictive uncertainly under dataset shift. In Aduances in Newral Information Processing Systems (NeurIPS), 2019.
Sangdon Park, Edgar Dobriban, Insup Lee, and Osbert Bastani. PAC prediction sets under covariate shift. In International Conference on Learning Representations (ICLR), 2022.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In Advonces in Neural Information Processing Systems (NewrIPS), 2019.
F. Podregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research (JMLR), 12:2825 2830, 2011.
John C. Platt. Probabilistic outputs for support vector machines and comparisons regularized likelihood methods. In Advances in Large Margin Classifiers, pp. 61-74. MIT Press, 1999.
Joaquin Quionero-Candela, Masashi Sugiyama, Anton Schwaighofer, and Neil D. Lawrence, Dataset Shift in Machine Learning. The MIT Press, 2009. ISBN 0262170051.
Stephan Rabanser, Stephan Günnemann, and Zachary Lipton. Failing loudly: An empirical study of methods for detecting dataset shift. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
Sashank Reddi, Barnabas Poczos, and Alex Smola. Doubly robust covariate shift correction. In AAAT Conference on Artificial Intelligence, 2015.
Rebecca Roclofs, Nicholas Cain, Jonathon Shlens, and Michael C. Mozer. Mitigating bias in calibration error estimation. In Proceedings of The 25th International Conference on Artificial Intelligence and Statistics (AISTATS), 2022.
Evgenia Rusak, Lukas Scholt, Roland S. Zimmermann, Julian Bitterwolf, Oliver Bringmann, Maithias Bethge, and Wieland Brendel. A simple way to make neural networks robust against diverse image corruptions. European Conference on Computer Vision (ECCV), 2020.
Shiori Sagawa, Pang Wei Koh, Tatsunori B. Hashimoto, and Percy Liang. Distributionally robust neural networks. In International Conference on Learning Representations (ICLR), 2020.
Bernhard Schölkopf, John C. Platt, John C. Shawe- Taylor, Alex J. Smola, and Robert C. Williamson. Estimating the support of high-dimensional distribution. Neural Compulation, 1343):1443-1477, 2001.
17

---

Published in Transactions on Machine Learning Research (12/2022)
Abhin Shah, Yuheng Bu, Joshua Ka- Wing Lee, Subhro Das, Rameswar Panda, Prasanna Sattigeri, and Gregory W Wornell. Selective regression under fairness criteria. arv preprint: 2110.15403 2021.
M. Shaked and J.G. Shanthikumar. Stochastic Orders. Springer Series in Statistics. Springer New York, 2007. ISBN 9780387346755.
Masashi Sugiyama and Klaus-Robert Müller. Input-dependent estimation of generalization error under covariate shift. Statistics and Decisions, 23(4):249 279, 2005.
Ryan J Tibshirani, Rina Foygel Barber, Emmanuel Candès, and Aaditya Ramdas. Conformal prediction under covariate shift. In Advances in Newral Information Processing Systems (NeurIPS), 2019.
Dustin Tran, Jeremiah Liu, Michael W. Dusenberry, Du Phan, Mark Collier, Jie Ren, Kehang Han, Zi Wang, Zelda Mariet, Huiyi Hu, Neil Band, Tim G. J. Rudner, Karan Singhal, Zachary Nado, Joost van Amersfoort, Andreas Kirsch, Rodolphe Jenatton, Nithum Thain, Honglin Yuan, Kelly Buchanan, Kevin Murphy, D. Sculley, Yarin Gal, Zoubin Ghahramani, Jasper Snock, and Balaji Lakshminarayanan. Plex: Towards reliability using pretrained large model extensions. ArXiv preprint: 2207.07411 2022.
Athanasios Tsoukalas, Timothy Albertson, and Ilias Tagkopoulos. From data to optimal decision making: A data-driven, probabilistic machine learning approach to decision support for patients with sepsis. JMIR Medical Informalies, 3(1),el1, 2015.
Juozas Vaicenavicius, David Widmann, Carl Andersson, Fredrik Lindsten, Jacob Roll, and Thomas Schön. Evaluating model calibration in classification. In International Conference on Artificial Intelligence and Statistics (AISTATS). 2019.
Joost van Amersfoort, Lewis Smith, Yee Whye Teh, and Yarin Gal. Uncertainty estimation using a single deep deterministic neural network. In International Conference on Machine Learning (ICML), 2020.
Vladimir Vovk and Ivan Petej. Venn-abers predictors. In Proceedings of the Thirtieth Conference on Uncertainly in Artificial Intelligence, 2014.
Vladimir Vovk, Ili Nouretdinov, and Alexander Gammerman. Testing exchangeability on-line. In ICML, 2003.
Vladimir Vovk, Alex Gammerman, and Glenn Shafer. Algorithmic Learning in a Random World. Springer- Verlag, Berlin, Heidelberg, 2005.
Vladimir Vovk, Llia Nouretdinov, Valery Manokhin, and Alexander Gammerman. Cross-conformal predictive distributions. Proceedings of the Seventh Workshop on Conformal and Probabilistic Prediction and Applications, 2018.
Yoav Wald, Amir Feder, Daniel Greenfeld, and Uri Shalit. On calibration and out-of-domain generalization. In Advances in Neural Information Processing Systems (NewrIPS), 2021.
Junfeng Wen, Chun-Nam Yu, and Russell Greiner. Robust learning under uncertain test distributions: Relating covariate shift to model misspecification. In International Conference on Machine Learning (ICML), 2014.
Yaodong Yu, Stephen Bates, Yi Ma, and Michael Jordan. Robust calibration with multi-domain temperature scaling, Xiv preprint: 2206. 02757, 2022.
Bianca Zadrozny and Charles Elkan. Obtaining calibrated probability estimates from docision trees and naive bayesian classifiers. In International Conference on Machine Learning (ICML), 2001.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In British Machine Vision Conference (BMVC), 2016.
Shengjia Zhao, Tengyu Ma, and Stefano Ermon. Individual calibration with randomized forecasting. In International Conference on Machine Learning (ICMT), 2020.
18

---

Published in Transactions on Machine Learning Research (12/2022)
Shengjia Zhao, Michael Kim, Roshni Sahoo, Tengyu Ma, and Stefano Ermon. Calibrating predictions to decisions: A novel approach to multi-class calibration. In Advances in Newral Informalion Processing Systems (NeurIPS), 2021.
A Proofs
A.1 Proof of Claim 4.3
Proof. We prove existence in Claim 4.3 by example. Let random variable : f(X). Define h(r) | R = r] - r| to be the true calibration error for confidence level E [0, 1]. Then take g(x) as
g(x) {
if h(f()) S
(18)
otherwise.
where we define Å as
Å = inf
(19)
5-th quantile of the calibration error of
We show that this choice of satisfies parts (i) and (ii) of Claim 4.3.
(i) As g(X) is binary, have [X) ((((R) 1). The CDF (P((RR) - X) per Eq. (19) is right-contimuous with sup, ((h(R) & A) 1, so that well-defined and satisfics P(h(R) À) 2
(ii) Let random variable : R g(X 1. Then let random variables E, E be defined
E = B R R].
(20)
VI. c = E Y v
(21)
For our choice of g, we have for all X E [0
P(E < A) P(E E x) S SP(E
(22)
giving E[)(E)] E[o(E) for all increasing functions by stochastic dominance (Shaked & Shanthikumar, 2007). For x 2 0, (xx) = æ is increasing Vq 2 1, hence EL < B[E"), which implies (E[ < (BI())) Comparing definitions, this is equivalent to S-BCE < BCE.
A.2 Proof of Theorem 4.5
Our proof follows that of Kumar al. (2018), differing its treatment of (1) random variables conditioned on g(X) and (2) non-negative by calibration terms. We begin with the following lemma.
Lemma A.1. Let X,Y [0, 1] be non-negative random variables with some joint distribution P. Define M the integral probability metric,
M(X,Y;C) = sup EJy h(X)),
(23)
hEe
where is the space of all continuous, bounded functions h(x)) defined over x E [0, 1]. Then, M(X,Y:C) = = almost surely.
(24)
Proof. Since is non-negative and bounded, and all h E C are bounded, we can write
5 M(X,Y;C)) < Bai E| <
(25)
for Bup = sup sup h(x), where 0 < Bup c0, C includes functions that not strictly negative. kec ae (0.1)
19

---

Published in Transactions on Machine Learning Research (12/2022)
Similarly,
00 M(X,Y: Bio 2
(26)
for Bio == sup inf h(z), where B) < oo as includes functions that are strictly positive. heC =E(0,1]
Combining bounds from Eqs. (25) and (25), we have
(i) P(Y = 0) = B|y = 0 - M(X,Y;C) = 0, and
(ii) P(Y = 0) < < - 0= 0
We now proceed to prove Theorem 4.5.
Proof- For of notation, define random variables : f()) (X) = 1 and = BY V] VI%, where V is the selective confidence and E is the selective calibration error V. First, we have that
S-MMCE = 0 > S-MMCE"
(SEE>>(]]]] 0,
and second that
E = Vj -
(27)
Therefore, it will suffice to show that
S-MMCE =0 + E = 0 almost surely.
We start by defining alternative (intractable) integral probability metric:
M(E, V;C) = ssup E E c(V)),
(28)
kEC
where C denotes the space of all continuous, bounded functions h(v) over [0,
Applying Lemma A.1 gives M(E, = 0 - E = almost surely.
We now replace C with the set functions = (h E H: || S 1} the RKHS with universal kernel k(.;). Since & is universal, F dense in the space of bounded continuous functions with respect the supremum Using the same
norm. That is, for every h C and 0, there 3M E F such that sup, (M(v)) [1(v)) arguments as Lemma A.1, can then immediately show that M(EVV; 0 iff E 0 almost surely.
Finally, using the reproducing property of the RKHS, we derive the equivalence to S-MMCE*
M(E.V:)) sup E E A(V))
heF
sup E. [ (h, (V,
hEF
sup heF
(h, E(E o(V)))
(29)
E[E o(V)]
EE ((V)) H
|ELE +(V) 1
| PE[EESOV1 *(v)]
S-MMCE
where à is the feature map associated with kernel k.
20

---

Published in Transactions on Machine Learning Research (12/2022)
A.3 Proof of Proposition 4.6
Lemma A.2. Let X be a random variable. For any family of functions F defined over the range of X,
sup REF
[[h(X)] S E/sup h(X)
(30)
hEF
Proof. For any fixed N' F, we have (X)) S supber h(X), which implies B[)((X) S Elsupner ((X)..
Since this holds for all h' E F, Elsuprez ((X) is an upper bound of the set [X]] N e F
Lemma A.3. Let H. be an RKHS with kernel Then the following holds for all functions h H:
h(v)= S / k(v, v) (hll
(31)
Proof. Using the reproducing property of the RKHS and the Cauchy-Schwann inoquality, we have that h(v) (h, x)) s ||Allzellk<<s w))l =vk(u,
(32)
We now proceed to prove Proposition 4.6.
Proof. We begin by writing S-MMCE in TPM form in A.2:
S-MMCE MY V] - VFF)
(33)
sup E [ V] V/ h[V)] heF
where F=
= (h E H: [1ll I 1}.
Applying Lemmas A.2 and A.3,
sup E | v] vp. ((V)]((]]] < E REF
E[Y [V] V sup h(V) heF
(34)
E
E[Y V V sup ok(u,))(/llH
SK [ [ - v19..
Comparing definitions, we can write S-MMCE < K BCCCEE
A.4 Proof of Proposition 4.7
Proof. We begin by writing S-MMCE in IPM form as in A.2:
S-MMCE" M V
sup E E V] (T).AA(V))
(35)
kEF
where F = {h E H: [ < 1).
Fix any E F and v E [0, 1). For notational convenience, let = Y V = c. and define the function
v(Z,v) 2 v/ h(v)
336)
Jensen's inequality yields E[p(Z, v)] 2 (E, v), as is convex in Z for every constant v. Applying the law of iterated expectations then gives (l V)) 2 ([E)) V)). Plugging back into Eq. (36), we have
Eijy ((V) > B|E V V M(V)).
(37)
21

---

Published in Transactions on Machine Learning Research (12/2022)
As all h F are part of the RKHS H. their evaluation functionals are bounded, giving
sup AEF
Y - V -[)) 2 sup
kEF EI
[] - vi9. [(V)].
(38)
Rewriting Eq. (38) norm in the RKHS H. completes the proof.
A.5 Proof of Proposition 4.8
Proof. Our proof is similar to that of Bates al. (2020). For case of notation, let S = o(X), where by assumplion S is continuous and samples St, = 1, ,,..... are i.dd. For some parameter E R, let
1 Σ
(T(T)
and
(39)
" -=
= : 2
(40)
Note that (T) is random variable depending on S ...... Sn while C(T) is constant. For every 7 and Ve > 0, Hoeffding's inequality gives pointwise bound on CT(T)'s deviation from C(r):
P(C,(T)-C(p 2 20 se 2nc? -2ne"
(41)
Recall the definition of f from Eq. 16:
f 6 =u sup
( R: C2((T) 2 c]
(42)
As On random variable, i is also random variable as C(f), the true expected coverage random threshold f. Let E be the event that C(f) <-c. We will now show that this event occurs with probability at is continuous, as S is continuous
most e22n Define the constant -" as 7* == sup (r R: C(r) 2 - c) by assumption, therefore r" well-defined, and satisfies C(7*) = E Suppose E. Then & P" C(T) is monotonically non-increasing. This implies C,(rT) 2 as (T) is also monotonically non-increasing.
As the event C(f) S 5 implics the event C,,r")
P(C(f) < é) s ((C)((T) 2 E).
(43)
However, by applying Hooffding's inequality at == = T" we also have that
C(7') + se 2nc?
(44)
Combining bounds Eqs. (44) and (43) yields P(C(f) s - 52-2
Remark A.4. Alternatively, to guarantee coverage 2 & with high probability for any (instead of providing probabilistic bound on its error), one can also use a corrected threshold via the conformal prediction, RCPS, or CRC algorithms (see Vovk al., 2005; Bates et al., 2020; Angelopoulos al., 2022, respectivcly).
B Technical details
B.1 Empirical calibration error
We use equal-mass binning (Kumar al., 2019; Roclofs et al., 2022) estimate the expected calibration error. Given dataset of confidence predictions, Drest {(((1). mn)..., (f(x,) yn)), sort and assign each
22

---

Published in Transactions on Machine Learning Research (12/2022)
prediction pair (f(4),, to one of m equally sized bins B (i.c., bin B3 will contain all examples with scores falling in the &-1 empirical quantiles). If y is binary, the expected binary calibration error is estimated by
:
Bl& Drest
EieB, yi [Bx]
Eies, Maxs 19 1)
BCE
(45)
[B&l
When y is multi-class, we replace yi and (3r) with 1 {v: = argmaxye(K] (1:)y and maxye[k] f(ar)) respec- tively to compute the TCE. We set the number of bins m to min(15, _ For selective calibration error, we first filter Deest to only include the examples that have g(X;) 1, and then apply the same binning procedure.
B.2 Training details
We use a Laplacian kernel with width 0.2 in S-MMCE, similar to previous work (Kumnr et al., 2018), i..,
-Irr - ryl
k(rer,)) = exp
(46)
0.2
All models are trained in PyTorch (Paszke et al., 2019), while high-level input features ($4.5) for g are computed with sklearn (Pedregosa et al., 2011). We compute the S-MMCE with samples of size 1024 (of examples with the same perturbation E T applied), with combined batch size of = 32. We train all models for 5 epochs, with 50k samples (where one "sample" is a batch D of 1024 perturbed examples) per epoch (= 7.8k updates total at the combined batch size of 32). The loss syper-parameters A and A2 were set to [D] 0.5 = 1/32 and 1e-2 x D = 1e-5, respectively. The k in our top-k calibration error loss is set to 4.
B.3 NLST-MGH lung cancer data
The CT scans in the NLST data are low-dose CT scans from patients enrolled in lung cancer screening programs across the 33 participating hospitals. The MGH data is aggregated across diverse assortment of all patients in their system who underwent any type of chest CT between 2008-2018, and matched NLST enrollment criteria (55-74 years old, current/ purrent/former former smokers with 30+ pack years). Note that since these patients are not emrolled in a regular screening program, not all patients have complete 6-year follow-up records. All patients without documented cancer diagnosis in the 2008-2018 period are assumed to be cancer- free.
C Additional results
Figure C.1 presents results per perturbation type on ImageNet-C. Like CIFAR-10-C (Figure 3), we see consistent and significant seleetive calibration error reduction across perturbation types. This in contrast to the confidence-based selective classifier, which does not always reliably lead to improved selective calibration error. Related to perturbations, a question that arises is what happens if perturbations are also included during the training of f(X)? Figure C.2 presents results using an f trained with AugMix perturbations over CIFAR-10, top of which we learn our g as before (also using the same AugMix perturbation family). While the absolute calibration error of f is lower, in line with Hendrycks et. al. (2020), using the selective further reduces the selective calibration eгror across all coverage levels. Next, Figure C.3 presents the results of an ablation study on the S-MMCE loss formulation, applied to CIFAR-10-C. Specifically, we look at the effect of not including any perturbations, including perturbations on an example level only (i.e., (xx) rather than o Derain), and using all perturbations in a batch instead of only the w-worst. Interestingly, even training the S-MMCE loss only on in-domain, unperturbed data can lead to moderate improvements in selective calibration error. Including perturbations, however, leads to significantly better results. Sharing perturbations across batches or training on the woworst then offer additional, albeit small, improvements. We also perform a similar ablation study on CIFAR-10-C over the features provided to our S-MMCE MLP ($4.5) in Figure C.4. Finally, Figure C.5 demonstrates trade-offs between selective calibration and selective accuracy.
23

---

Published in Transactions on Machine Learning Research (12/2022)
0.25
Full Confidence
0.20
S-MMCE
S-TCE, AUC
0.35
0.00
contrant
defocus, Blut
eaatir, transtor
gucaaian, noise
gless bluf
imputte paise
geg compression
mation, blur
pisilltt
shet nolse
zorm, blur
dean
Average
Figure C.1: lg selective top-label calibration error AUC, reported for each of the 11 test-time perturbations ImageNet-C C (and the average). Across perturbations, optimizing for S-MMCE consistently leads to significant error reductions relative model without abstentions ("Full"), as well the standard selective classifier (" ('Confidence".. In contrast, the confidence-basee selector sometimes lead to worse calibration error.
0.34
hull 1o [b:]]
10.0/4]
0.32
Isolation Forest 065)
SVM
Isolation Forest (0.134)
Dne-Class SUM (0.125)
6.20-
Isolation Forest (0. 0.08))
One-Casc SUM (0. 1021
0.30
Dorf dence (0.018]
KNN Distance (0 0100
0.20
KNN Distance 043)
(0.043)
Conticerce
KNN Distance 10 0.0231
SIMNCE (0.013)
S. MMCE (0.031)
Brier Score
0.15
MKCE (C.019)
9-TCE.
0.10
0.05
0.00
0.00
0.00
0.00
0.2
0.4
0.6
0.8
1.0
0.2
0.4
0.6
0.0
1.0
0.2
0.4
0.5
0.8
10
E1g00]
E[g00]
<[gt0)
Figure C.2: Results on CIFAR-10-C averaged over perturbation types when including AugMix pertubrations (T) during the original training of f(X). As also first reported in Hendrycks et al. (2020), including such pertur- bations during training indeed reduces calibration error oft-of--he-box. Using even the same perturbation class T during selective training, can yield a selective classifier with substantially lower relative calibration error.
0.175
0.55
0.24
0.150
0.12
0.30
Salective Bner Score
n 0.125
STCE.
0.108
0.08
0.675
0.06
0:10
Full (0.246)
Confidence (0.110)
Confidence (0.172)
0.288
Conficence
W/o 7(0.102)
wio (0(1182)
w/o 7(0.105)
0.00
per F10.076)
005
perx /10
0.075
per 088)
Shared F, w/o DRO (0.075)
shared wio DRO (0.123)
Shared F, w/o D8O (0.087)
shared v, w/ (0.070)
0.00
(0.1
0.000
shared v, w/
0.4
0.6
0.2
0.4
3a
1.0
0.2
6.8
=1g001
E[g(X)
[[gtt)
Figure C.3: Ablation on CIFAR-10-C for different configurations of our S-MMCE optimization. Without T: perturbations. Per-a T: perturbations are mixed within batch. Shared T, no DRO: update on all batches, not just the w-worst. Shared T, with DRO: update the x worst examples per batch (main algorithm).
24

---

Published in Transactions on Machine Learning Research (12/2022)
0.175
0.100
0.225
Salective Brar Score
0.20
0.100
STICE
0.075
0.000
Full (0.153)
0:10
(0.248)
Full (0.172)
wio confidence (0. 108)
vio confidence (0.197)
Weo confidence (0. (0.106)
w/o class info 10.092)
0.05
class info (9.158)
who class info (0.100)
w/o scores
scores (0.194)
w/o scores (0.112)
Full (0.070)
Rull (0.118)
RJll (0.082)
0.4
0.2
[(o(Xi]
[[g00)
[[G(X)]
Figure C.4: Ablation on CIFAR-1O-C holding out sets of input features to the MLP. Without confidence: all confidence features are removed (top-label confidence, entropy. the full distribution). Without class info: all features that can be used identify the predicted class are removed (one-hot class index, the full confidence distribution). Without outlier scores: all derived outlier/novelle novelty scores (One-Clast SVM, Isolation Forest, etc) are removed. We see that each of these feature sets has impact on the aggregate classifier in somewhat complementary ways: in particular, confidence helps performance most at high coverage levels, while outlier scores help performance most at low coverage levels.
10-
FUll
1.5
Isolation Forest (0.497)
One Che-Class SVM (0.288)
Confidence
Selechiv Accurary
Selective accuraty
Selective Accuracy
(0.426)
0.6
2.6-
0.44
0.4
(0.71.)
Isaation Forest (0 626)
Confidence (0 (0.66))
One One-Cless SVM [0.691]
0.-
One Clas-Clss SVM (0.769)
Confidence
Distance (0.878)
Selective (0. (Bd)
5.0
0.0
82
#igi011
06
0.5
10
02
04
#[g00]]
mm
6.8
3.0
0.6
EleXX]]
0.6
0.0
(a) CIFAR-10-C
(b) ImageNet-C
(c) Lung cancer (NLST-MGH)
Figure C.5: Selective accuracy results across CIFAR-10-C, TmageNet-CC, and Lung cancer (higher AUC is better). When more calibrated examples also tend to be more accurate (c.g., if higher confidence values are also well-calibrated), then the selective accuracy of S-MMCE can be quite high (see CIFAR-10-C and Lung cancer). Sometimes, however, there is tradeoff between calibration and accuracy: on ImageNet-C, selection based on S-MMCE scores results in significantly lower selective accuracy relative to simple confidence thresholding, but is significantly better calibrated (see Figure 2 in $6).
25


Reviews:
[{"review_id": "iHhAL2RYsZ", "reviewer": "Reviewer_hv31", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The proposed selective calibration objective seems like an interesting addition to the UQ literature. Compared to usual calibration which modifies the confidence scores, the key difference within selective calibration is that it does not directly change the confidence scores, but rather calibrates through a binary decision given by the selector. This may be useful in real-world applications where such selectors are permitted whereas a (full) calibrator is not. \n\n- Experiments seem solid (e.g. the performance improvement, range of baselines and datasets), and justify the advantage of the proposed method against baselines.\n\n- The presentation is very clear and contains lots of useful details.\n\n- A couple design choices are justified by theoretical results, e.g. good property of the population S-MMCE objective when a universal kernel is used. This feels secondary though compared with the above strengths.\n\nWeaknesses:\n- My main concern is about *accuracy*: The paper does not report selective accuracy when the proposed calibrated selector is deployed. In Figure 3, only selective calibration objectives like S-TCE or mixed objectives like Brier scores are evaluated.  \nSelective accuracy itself (e.g. its AUC) is at least as important in my opinion for this task, due to the nature of this problem: The mechanism for selective classification is that there can only be one such selector. It is important to understand whether the calibrated selector is itself a less accurate selector (e.g. compared with SOTA selectors for selective accuracy). Or, if there is a trade-off between selective accuracy vs. selective TCE, where does the proposed method lie on the trade-off curve, and where do the baselines lie.\n\n\nGiven the above strengths and weaknesses, overall, I feel like this paper is a solid UQ/deep learning paper, which proposes a new learning objective and designs a method that achieves it better than a range of baselines. However, there is a concern whether the selective calibration objective is at a cost of other important objectives like selective accuracy.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a new learning objective for uncertainty quantification (UQ) called selective calibration, which requires models to abstain predictions based on a selector, so that the remaining predictions are as calibrated as possible (in addition to being accurate). The paper proposes a method for training such a selector, by combining techniques from both the calibration and the distributionally robust optimization literature. The trained selector achieves better selective calibration error over baseline methods.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "Motivation for selective calibration:\n\nThe motivation for this task is discussed in the 2nd paragraph in the introduction. While there is an example about cancer development predictions, the example still reads to me like just saying selective classification makes sense, then calibration is also important… so we should do these two together. Could the authors provide more context / examples on who should care about selective calibration? I imagine (and hope!) this paper is going to be read by practitioners, so getting this convincing may be a worthy thing to do for this paper. \n\nDiscussions about calibration vs. selective accuracy:\n\nAnother point worth some discussions is about the relationship between calibration and selective classification itself. It was observed, e.g. in Malinin et al. (2019, Appendix B & Figure 5) that calibrating the top probability confidence score (which they do by ensemble + distillation) improves the selective classification accuracy defined by the (naive) top probability selector. That’s another connection between calibration and selective accuracy. The learning goal in this paper is different, but I am curious whether the authors think that point is related to this paper / worth to have some discussions.\n\n\nReference: Malinin, Andrey, Bruno Mlodozeniec, and Mark Gales. \"Ensemble distribution distillation.\" arXiv preprint arXiv:1905.00076 (2019).\n\n\nAdditional minor questions:\n\n- Brier score reported in Figure 3: Is it actually the *selective* Brier score?\n- Choice of the 1d kernel in MMCE (Appendix B.2) is fixed to be a Laplace kernel with $\\sigma=0.2$ and not tuned, any intuitions for this choice? (Is it following the literature e.g. the original MMCE paper?)\n", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "FDEYHGtnR6", "reviewer": "Reviewer_ffhS", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n* The paper is well-written, with the objective function clearly introduced and situated in the literature.\n* Author derived sufficient theoretical results to backup the design choices\n* The method is empirically validated against common baselines on challenging large-scale datasets (ImageNet-C), showing significant improvements.\n\nWeakness: (Please see \"Requested Changes\")\n* The use of S-MMCE_u upper bound is not sufficiently justified, some discussion on either the tightness of the bound, or if the estimation quality compromises the method's objective of minimizing the true S-MMCE (e.g., monitor the S-MMCE_u v.s.  S-MMCE curve during training) would be helpful.\n* The S-MMCE_u estimator (Eq. (3)) seems to be numerically unstable depending on the batch size. \n* The method's guarantee is currently extended only to the situation where the class of perturbation is known. Authors could consider the method's performance to unknown perturbations (e.g., conduct leave-one-perturbation-out evaluation) to provide a fuller view of the method performance.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work proposes a soft training objective for post-hoc training of selective prediction models, and introduced extensions to ensure the model's out-of-distribution robustness against classes of known and well-defined perturbation functions. On three image classification tasks (CIFAR-C, ImageNet-C and NLST-MGH), the proposed objective achieved superior calibration performance when compared to popular baselines (confidence-based and distance-based outlier detection methods).\n", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "\"universal kernel $k(·, ·)$\", should it be bound? If so, might be useful to mention it since otherwise \n\nEquation (12): please add a one-sentence explanation about why $S-MMCE_u$ is an upper-bound (maybe just add a pointer to Proposition 4.7).\n \nEquation (13): please comment on the robustness of $S-MMCE_u$ estimator with respect to batch size. I see that there is a possible tradeoff between computational complexity and quality of the S-MMCE estimator. Could the author comment on this possible tradeoff, and provide suggestions on how to decide batch size in practice? A small investigation on one of the dataset would also be very informative.\n\nProposition 4.7 please discuss how tight this bound is, and if it is not tight, would using $S-MMCE_u$ as the optimization objective impact the calibration quality? Some small experiments on CIFAR-10C / ImageNet-C in comparing the empirical distribution / gap between S-MMCE and S-MMCE_u might be useful. \n\nSmall typo: Section 5.1 \"ImageNet-C\" paragraph: \"scaleing -> scaling\"\n\nThe method's guarantee is currently extended only to the situation where the class of perturbation is known. Authors could consider the method's performance to unknown perturbations (e.g., conduct leave-one-perturbation-out evaluation) to provide a fuller view of the method performance across the datasets. (Just to be clear, even if the method underperforms in this situation, it does compromise the appeal of the method since it is still useful for the situation where all sources of perturbation are known. However, since TMLR insists on the quality and rigor of its publication, I believe presenting an investigation like this to provide a fuller view of the method performance is important).\n", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "yxQnO4ZwNs", "reviewer": "Reviewer_j41U", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n  * The paper combines research from selective prediction and calibration, which are two important fields of research.\n\n * A new tractable metric and corresponding loss function are proposed, whereby selectively calibrated classifiers can be trained/learned. \n\nWeaknesses:\n\n  * The paper seems to be divided between two topics. The introduction, method section and conclusion deal with calibration and selective prediction. Together named selectively calibrated classifiers. And the experimental section deals with OOD tasks. I understand that the two topics are related. But the paper misses either a) experimental results for the new metric (and its loss function), or b) methodical explanation and contributions to the OOD research field are missing.\n\n  * Same question holds for data augmentation, which is discussed in the introduction. Could you please explain in the rebuttal the theoretical connection between selective prediction and calibration on one side, and data augmentation etc on the other side?\n\n  * Training g() requires hold-out data (Section 5.1). Perhaps this weakness should be mentioned earlier on. Moreover, as a question: has research been done to identify the effect of the size of this hold-out data? And what is the effect on more or less held-out data? And finally to play into the experimental section: what is the effect of g() being trained OOD?\n\n  * Section 6: most results focus on OOD performance (ImageNet-C, AugMix). However, I wonder what are the fundamental properties of this new paradigm of ‘selectively calibrated classification’, and properties of the loss function and upper-bound proposed?\n\n* Equation (1): ‘G(x)=1’ seems to be outputting in {0,1} here. However, I assume also the function G() has uncertainty. How is that uncertainty dealt with?\n\n* Again, section 4.4, ‘ A main motivation of our work is to create selective classifiers (f, g) that can effectively generalise to new domains not seen during training‘. I don’t understand how OOD generalisation is a ‘main motivation of the work’.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "* The paper introduces a line of thinking where calibration could be dependent on ‘selection’ of samples, see equation (1).\n\n* Theorem 4.3 provides a proof that such a selectively calibrated classifier exists. \n\n* Measure and loss function are supplied. \n", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": " * This paper discusses selective prediction and calibration in this work. However, I am surprised to see no mention of conformal prediction [1]. Could the omission and possible relation be explained? (Same holds for ‘expected calibration error’ and its trainable loss functions [2].)\n\n * “Under our framework, a model should abstain if it is uncertain about its confidence” This should be motivated further. I would think that ‘Uncertainty’ about a prediction ‘far away from the threshold of g()’ should be considered less than ‘uncertainty’ about a prediction ‘close to the threshold of g()’. \n\n\n * Section 4.5: there are important features to use for a g() function. Has research been done to identify the most important features of these? I can imagine some are more important than others. Moreover, the (relative) importance of features could give insight into the working of selective prediction.\n\n\n * Most confusing of this paper is the switch between OOD-thinking, like ImageNet-C and Augmix and calibration. Both introduction and conclusion discuss mainly general claims to calibration and selective classification, while experiments relate mostly to OOD datasets and data augmentation\n\n\n[1] Shafer, G. and Vovk, V., 2008. A Tutorial on Conformal Prediction. Journal of Machine Learning Research.\n\n[2] Guo, C., Pleiss, G., Sun, Y. and Weinberger, K.Q., 2017. On calibration of modern neural networks. In: International conference on machine learning.\n", "broader_impact_concerns": "N.A. Calibration and selective prediction could benefit many other fields of science. Especially enterprises where decisions must be made under uncertainty, such as clinical settings, manufacturing settings, and policy settings. ", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers are in favour of accepting this work, a viewpoint that I am happy to concur with myself: I believe it is a useful contribution to the uncertainty quantification literature.  As the main concerns of the reviewers have already been addressed in the revision, I am happy to recommend the paper be accepted as is.
Decision: Accept (poster)