Towards Uncertainties in Deep Learning that Are Accurate and Calibrated
Authors: Volodymyr Kuleshov
Shachi Deshpande
Venue: ICLR 2022 Submitted (2022)

Abstract: Predictive uncertainties can be characterized by two properties---calibration and sharpness. This paper introduces algorithms that ensure the calibration of any model while maintaining sharpness. They apply in both classification and regression and guarantee the strong property of distribution calibration, while being simpler and more broadly applicable than previous methods (especially in the context of neural networks, which are often miscalibrated). Importantly, these algorithms achieve a long-standing statistical principle that forecasts should maximize sharpness subject to being fully calibrated. Using our algorithms, machine learning models can under some assumptions be calibrated without sacrificing accuracy: in a sense, calibration can be a free lunch. Empirically, we find that our methods improve predictive uncertainties on several tasks with minimal computational and implementation overhead.

Under review as a conference paper at ICLR 2022
TOWARDS UNCERTAINTIES IN DEEP LEARNING THAT ARE ACCURATE AND CALIBRATED
Anonymous authors Paper under double-blind review
ABSTRACT
Predictive uncertainties can be characterized by two noperties-calibree and sharpness. This paper introduces algorithms that ensure the calibration of any model while maintaining sharpness. They apply in both classification and regres- sion and guarantee the strong property of distribution calibration, while being sim- pler and more broadly applicable than previous methods (especially in the context of neural networks, which are often miscalibrated). Importantly, these algorithms achieve long-standing statistical principle that forecasts should maximize sharp- ness subject to being fully calibrated. Using our algorithms,, machine learning models can under some assumptions be calibrated without sacrificing accuracy: in sense, calibration can be a free lunch. Empirically, we find that our methods improve predictive uncertainties on several tasks with minimal computational and implementation overhead.
INTRODUCTION
Probabilistic forecasts can characterized by two propeties-ccalibratiee and sharpness (Gneiting et al., 2007). Intuitively, calibration means that a 90% confidence interval contains the true outcome 90% of the time. Sharpness means that these confidence intervals are narrow. These qualities are grounded in the statistics literature on proper scoring rules, and are widely used to evaluate forecasts (Gneiting and Raftery, 2005; 2007) in domains such as medicine (Saria, 2018), robotic control (Malik et al., 2019), and saman-in-the-loon learning (Werling et al., 2015).
This paper introduces algorithms that ensure the calibration of any predictive machine learning model while maintaining sharpness. They apply to both classification and regression tasks and guarantee the strong property of distribution calibration (which generalizes standard quantile and classification-based calibration; Song et al. (2019)) in any model, including deep learning models, which are often miscalibrated (Guo et al., 2017). Unlike existing methods for distribution calibration (Song et al., 2019), ours can be used with any model (not just ones that output Gaussians), are very simple to implement in differentiable programming frameworks, and have theoretical guarantees.
Importantly, our algorithms achieve for the first time long-standing statistical principle that fore- casts should "maximize sharpness subject to being calibrated" proposed by Gneiting et al. (2007). We prove that under some assumptions this principle is achievable in modern machine learning mod- els in black-box manner and without sacrificing overall performance. This lends strong support for this principle as a way of reasoning about uncertainty in machine learning.
In sense, calibration is a rare free lunch in machine learning, and we argue that it should be enforced predictive models and taken advantage of in downstream applications. Empirically, we find that our method consistently outputs well-calibrated predictions across wide range of experiments, while improving performance on downstream tasks with minimal implementation overhead.
Contributions. In summary, we make three contributions. We propose a new recalibration tech- nique that (a) among the only to guarantee distribution calibration besides Song et al. (2019). Unlike Song et al. (2019) can (b) recalibrate any parametric distribution (not just Gaussians) and (c) our method is simpler. While theirs is based on variational inference in Gaussian processes, ours uses neural network that be implemented in few lines of code, which encourages adoption. Our method (d) applies to both classification and regression and (c) outperforms methods by Song et al. (2019) and Kuleshov al. (2018) as well as Platt and temperature scaling.

---

Under review as a conference paper at ICLR 2022
We also formally prove that technique produces asymptotically distributionally calibrated fore- casts while minimizing regret. Most methods (e.g., Platt, Kuleshov scaling, Song et al., etc.) do not have a conrectness proof, except for conformal prediction, which is significantly more complex.
Finally, our analysis formalizes the well-known paradigm of Gneiting et al. (2007) and provides the first method that provably achieves it. This lends strong support for this principle and influences how one should reason about uncertainty in machine learning. An important takeaway is that calibration can be achieved in most applications of machine learning with very little cost. As such, calibration can be a rare free lunch that we believe should be leveraged throughout machine learning.
2 BACKGROUND
2.1 PREDICTIVE UNCERTAINTY IN MACHINE LEARNING
Supervised machine learning models commonly predict probability distribution over the output variables c.g. class membership probabilities or the parameters of exponential family distribu- tion. These predictive uncertainties are useful for interpretability, safety, and downstream decision- making. Aleatoric uncertainty captures the inherent noise in the data, while epistemic uncertainty arises from not having a large enough dataset to estimate model parameters (Kendall and Gal, 2017).
Notation. Formally, we say that a machine learning forecaster H : X - A()) outputs a proba- bility distribution F(y): y + [0. 1] in the space (()) of distributions over y. We use f to denote the probability density or probability mass function associated with The model H is trained on a labeled dataset 22, Y/ E X' xxx for t = 1,2, ..., T of i..dd realizations of random variables X, Y P. where PP is the data distribution.
2.2
WHAT DEFINES GOOD PREDICTIVE UNCERTAINTIES?
The standard tool in statistics for evaluating the quality of predictive uncertainties is proper scoring rule (Gneiting and Raftery, 2007). Formally, a scoring rule S A()) x y - R assigns a "score" to a probabilistic forecast F (()) and realized outcome y E y. Given a true distribution G E A()) for y. we use the notation S(F, G') for the expected score S(F, G) Ey-cc:(F,v))
We say that a score is proper if it is minimized by G when is the true distribution for y: S(F, G) 2 S(G, G) for all F. When S is proper, we also refer to it as proper loss. An example of a proper loss is the lig-likelhhoo S(F y) log fy). where f is the probability density or probability mass function of F. Another common loss is the check score Py (y. 7 - f) if y 2 f and (1 [f y) otherwise; it can be used to estimate the T-th quantile of a distribution. See Table for additional examples.
What are the qualities of a good probabilistic prediction, as measured by a proper scoring rule? can be shown that every proper score is sum of the following terms (Gneiting et al., 2007):
proper loss calibration -harpness irreducible term
refinement lerm
Thus, there are precisely two qualities that define an ideal forecast: calibration and sharpness. We examine each of them next.
2.3 CALIBRATION AND SHARPNESS - Two QUALITIES OF AN IDEAL PREDICTION
Formally, calibration can be defined by the equation
P(Y=)) y Fx = F) = f(y) for all y E, F A()),
where X,Y P are random variables corresponding to the input features and targets, and Fx (X) is the forecast at X, itself a random variable that takes values A(Y). We use f to denote the probability density probability mass function associated with F.
When y = {0, 13 and Fx is a Bernoulli distribution with parameter p. we can write (1) as P(Y =1 1| Fx p) p. This has simple intuition: the true probability y is conditioned
2

---

Under review as a conference paper at ICLR 2022
Proper Score
Loss L(F,G)
Calibration Lc(F.Q)
Refinement L,(
Logarithmic CRPS
Ey-c log f(y) Ey-a (F(y) - G(v))²
KL(gllf)
((g)
[ (P(v) Q())) (r) Lo So (0) (Q(v) r)dydr
1 Q(v) Q(v))dy BTEU y-Q p. (r))
Quantile
y-6
P. (y F (r))
Table 1: Proper loss functions. A proper loss a function L(F, G) over a forecast targeting a variable y E y whose true distribution is G and for which S(F, G) N S(G, G) for all F. Each L(P, G) decomposes into the sum of a calibration loss term (F, Q) (also known as reliability) and a refinement loss term (Q) (which itself decomposes into sharpness and an uncertainty term). Here, Q(y) denotes the cumulative distribution function of the conditional distribution P(Y y Fx = F) of Y given a forecast F. and g ft) are the probability density functions of Q and F, respectively. We give three examples of proper losses: the log-loss, the continuous ranked probability score (CRPS), and the quantile loss.
on predicting it p. Equation extends beyond binary classification to arbitrary distributions. For example, is Gaussian with variance o², this definition asks that the data distribution condi- tioned on predicting F also has variance o², This recently proposed definition is called distribution calibration (Song et al., 2019).
A closely related, but weaker concept is quantile calibration (Kuleshov et al., 2018), which asks that a 90% confidence interval contains the true value 90% of the time. Formally, it can be written as: F(Y < CDFF (p)) = p for all p E [0, 1], Quantile calibration implied by distributional calibration (Song et al., 2019).
Calibration by itself is not sufficient to produce a useful forecast. For example, is easy to see that binary classifier that always outputs P(Y 1) as the probability that Y is calibrated; however it does not even use the features X and thus cannot be accurate.
In order to be useful, forecasts must also be sharp. Intuitively, this means that predicted confidence intervals should be as tight as possible around a single value. This is captured by proper scoring rules as part of a refinement term (see Table 1), which equals an irreducible term minus sharpness term (Murphy, 1973; Brocker, 2009). The latter is maximized when we minimize the scoring rule.
Are Modern Machine Learning Models Calibrated And Sharp? Most machine learning mod- els are not calibrated out-of-the-box (Niculescu-Mizil and Caruana, 2005; Guo et al., 2017). Two reason for this are the limited expressivity of the model H-we we cannot perfectly fit the entirety of the level curves of the data distribution and computational approximations computing extract predictive uncertainties may be intractable, and approximations are not entirely accurate.
A final reason stems from how models tive functions induce a tradeoff between sharp and calibrated forecasts. Next, we will show that by achieve calibration without sacrificing performance.
cannot fit a perfect H, standard objec-
training models differently,
ENSURING DISTRIBUTION CALIBRATION IN ANY MODEL
This section introduces algorithms that ensure the distirbutional calibration of any predictive ma- chine learning model while maintaining sharpness. Unlike existing methods for distribution cali- bration, ours can be used with any model (not just ones that output Gaussians), are very simple to implement in differentiable programming frameworks, and have theoretical guarantees.
We first assume there exists a parameterization & of the probabilitics returned by forecaster H: for each p A()) returned by H. there exist parameters & E th that describe p. The é can be the natural parameters an exponential family distribution, such as (1,,²2) o²) describing a Gaussian.
We consider class of algorithms based classic approach called recalibration. First, train a base forecaster to minimize a proper loss L. Then, we train an auxiliary model : o + o (called the recalibrator) the outputs of H that outputs the parameters Φ2 E Φ2 of another
3

---

Under review as a conference paper at ICLR 2022
distribution such that L is minimized. Here 4y is second parameterization of A()) (possibly the same). As a result, the forecasts (R (X) will be calibrated. We provide details in Algorithm 1.
Algorithm 1 Calibrated Learning of Probabilistic Models.
Input: Model H X' + 11 recalibrator R: 11 22, training set D, recalibration set C Output: Recalibrated model R X 22-
1. Fit the base model on D: min ((z,)eD (
2. Fit the recalibration model I on the output of H on C: minR Elen)e L ((Ro (x),)
Implementing this approach requires choosing parameterizations of probabilities 1, 22 recali- bration model R, and an objective L. We discuss these choices below; then clarify how differs from existing recalibration algorithms.
Parameterizing Probability Distributions. When fitting a classification model, each distribution A()) is categorical and can be parameterized via its K 2 2 class membership probabilities. In regression, most widely used models such as neural networks already output parameterized proba- bilities, in which the are usually the natural parameters of exponential family model.
In the most general case, if we only have black-box access to a density function or cumulative distribution function, may form d-dimensional representation by evaluating the distribution a grid of d points. For example, if we have black-box access to quantile function F-1, we may featurize F via its sequence of quantiles o(F) (F-1(a;)) for some sequence of levels ag possibly chosen uniformly in [0, 1].
(ax))
In addition the above techniques, the representation of output probabilities in 22 coming from R can leverage flexible invertible models of the CDF, following methods developed in the normalizing flow literature, including monotonic neural networks, sum- -quuarrss polynomials (Wehenkel and Louppe. 2019; Jaini et al., 2019) spline functions (Muller et al., 2019; Durkan et al., 2019), piece- wise separable models (Wehenkel and Louppe, 2019), and others.
Choosing a Recalibrator. Ideal recalibrators are highly effective at optimizing the proper loss (see Section 4). In simple setting like binary classification, task reduces to one- mie-dimeniooal density estimation; in such cases we can provably achieve calibration asymptotically by using kernel density estimation for the recalibrator R. while controlling the kernel width function of the dataset size to trade off overfitting and underfitting (Wasserman, 2006). In regression settings, we may rely other non-parametric techniques such as Gaussian processes.
An alternative approach is to rely on expressive neural networks; although their optimization is a non-convex, they are very effective at fitting proper losses L, feature mature regularization tech- niques, and can be implemented easily within deep learning frameworks, possibly within the same computation graph as a neural forecaster H, which can simplify deployment.
In the classification setting, a natural architecture for R is sequence of dense layers mapping the simplex AK into Ág. In regression settings, R needs output a density function: natural architecture for this is a mixture density network (MDN; Bishop (1994)).
Choosing a Proper Loss A natural choice of proper loss is the log-loss. It applies in both calibra- tion and regression; optimizing it is standard supervised learning problem.
In regression settings, we found that using the quantile loss L = Ereu 1y--rjo.1EE--G-(00 (r)( (see Table 1) was numerically stable and produced the best performance. This objective fits a model (T; Φ) to estimate the 7-th conditional quantile (-) at Φ. When Ra(T; Φ) is neural network that takes in 7 and x, we minimize the quantile loss Ererro, 1yjerro.jJJyy..PP... Eyn.Gp (y - F (T)) using gradient descent, approximating both expectations using Monte Carlo (details are in the experiments section).
Comparing Against Song et al. Interestingly, the method of Song et al. (2019) is special case of ours when i consists of Gaussian natural parameters, 42 consists of parameters for the Beta link function, R is Gaussian process, and 7. is the log-likelihood. However, the resulting problem can
4

---

Under review as a conference paper at ICLR 2022
only be solved using variational inference, which is slow and complex to implement. Our framework instead admits simple solutions based on gradient descent.
Comparing Against Other Approaches. Algorithm I performs recalibration like many previous methods (e.g., Platt, Kuleshov et al., Song et al., etc.); it may thus appear to be the same as these methods. However, that not the case. First, existing recalibration approaches operate over the space of probabilities (class probabilities or CDF values); ours operates over functional parameters. This is what enables it to achieve distribution rather than quantile calibration (Kuleshov al., 2018).
Our approach also involves novel recalibration objectives (e.g., the quantile divergence in regression) which differ from the calibration error of Kuleshov et al. (2018). We also use different types of models (small neural networks instead of isotonic regression) and different optimization procedures used (stochastic gradient descent instead of variational inference). Thus, our recalibration strategy is distinct from previous work.
4 THEORETICAL ANALYSIS
Next, we now show that under some assumptions calibration is provably achievable in modern ma- chine learning models in black-box manner and without sacrificing overall performance. In that sense, calibration is a rare free lunch in machine learning.
We start with some notation. We have recalibration dataset of size T sampled from IP and train a recalibrator R : A()) - A(Y) over the outputs of a base model H to minimize a proper loss L. We denote the Bayes-optimal recalibrator by B IP(Y = y (X));; the distribution of Y conditioned on the forecast (R o (X) is Q == P(Y
(R o H)))). We are interested in expectations of various losses over X, Y; simplify notation, we omit the variable X, e.g. E[L(R o H)] ([L(
Next, we will assume the following condition under which Algorithm works. Assumption 1. The model R can minimize expected risk such that w.h.p. we have
ELL(B H, Y)] s EI H, Y) < B[L(B o H, Y)] + 8
where 8 > 00,8- = oT) is bound that decreases with T and E[L(B H)) is the irreducible loss.
This assumption implies that the recalibrator can perform density estimation in what is usually a small number of dimensions (one or two). For some recalibrators, e.g., neural nets, it may not provably hold (e.g.. because of non-convexity). However, neural networks are effective density estimators in practice, and we can quantify whether they estimate density well on hold-out set. This assumption provably holds for many non-parametric density estimation methods.
Fact (Wasserman (2006)). When R implements kernel density estimation and the log-loss. Assumption holds with 5 o31/T2/3).
We now prove two key lemmas. We show that Algorithm outputs calibrated forecasts without reducing the performance of the base model, as measured by regret relative to loss L. Lemma 1. The model R o H is asymptotically calibrated, in the sense that E LR Q) < for 8 oT)) w.h.p.
Proof. Recall that the loss E[L(R o H,Y) decomposes into a sum of calibration and refinement terms (R a H, Q)) + (Q)] where Qvy) == IP(Y = (R o H)))).
As shown by Kull and Flach (2015), refinement further decomposes into a group loss and an irre-
ducible term: (Q). = (Q, H)) + [... o H, Y)), where B(Y = y H(X)) the Bayes-optimal recalibrator. The form of the group loss Ly is the same as that of Le. We may then write:
E E o H,Y)H
< LL HQQ)
(Q, EL CIL(BOH,Y) H,Y)
Rayes-Optimal Loss
Calibration
Group Loss
Bayes-Opt Loss
E o H,Y)
E [(B H,
Proper Loss
Bayes Optimal Loss
5

---

Under review as a conference paper at ICLR 2022
where d > 0, o(T). In the first equality used the decomposition of Kull and Flach (2015) and the last inequality we used Assumption 1. follows that ( (R o H, Q)1 6, i.. the calibration loss is small.
n
Lemma 2. The recalibrated model is asymptotically as good as the base model: ElL(Ro H, EIL(H, Y) + Y., where y > 0, = o(T)) a bound that decreases with T.
Proof. The claim holds by empirical risk minimization, since R o H minimizes L, but is more expressive than H and can represent the identity map (by Assumption 1).
We now combine these two lemmas to show Algorithm ensures calibration and low regret.
Theorem 3. Algorithm I produces model that minimizes expected risk, while w.h.p. achieving asymptotically optimal calibration.
Proof. The base model H. is trained using empirical risk minimization (ERM). The model R H minimizes the same objective L, hence minimizes the same expected risk by ERM theory. Also, by Lemma 2, the expected risk of R H also asymptotically approaches lower value as that of H. By Lemma 1, the model R produces asymptotically calibrated forecasts w.h.p.
Thus, given enough data, guaranteed to produce calibrated forecasts and preserve base model performance (as measured by L). Thus, calibration a property that can be achieved most applications of machine learning with almost cost. As such, calibration is rare free lunch in machine learning.
Finite-Sample Bounds. Note that our analysis provides finite-sample and not only asymptotic bounds the regret and calibration error- the bounds are stated in terms of variables 6, and y that are each (T). The bound on the calibration error directly depends on the finite-sample bound on the generalization error of the algorithm used as the recalibrator.
Practical Considerations. Assumption 1 suggests that want minimize the expected risk (XL(H(X), Y)] well. Thus, in practice we want to select a highly flexible algorithms for which we can control overfitting and underfitting. This motivates our ear- lier advice of using density estimation algorithms- are expressive and feature effective regularization techniques
model family that can
have provable guarantees-and neural
WHAT UNCERTAINTIES ARE NEEDED IN MODERN DEEP LEARNING?
Good predictive uncertainties are calibrated and sharp and these two propertics yield optimal values of the ligeliikl and other proper loss functions. Thus, they characterize ideal forecast. In practice, however, modern machine learning models do not output such ideal predictions. What then the ideal type of forecast that we should aim obtain from our models?
Gneiting et al. (2007) argue that predictive uncertainties should be maximally sharp subject to being calibrated. They propose a diagnostic approach based on this principle; this approach is commonly used in statistics for evaluating the predictive performance of probabilistic models.
In this paper, we also argue for this general principle, but approach it in a prescriptive way we claim that this principle should be enforced in modern ML systems, and we show how to do so. Specifically,, we show that any model can be modified to output calibrated uncertainties, and this property can be provably achieved without sacrificing performance.
5.1 CALIBRATED RISK MINIMIZATION
We formalize the intuition behind "maximizing sharpness subject to being calibrated" follows. We argue that we should be training models to minimize expected risk (as measured by a proper loss) subject being perfectly calibrated. We call this principle calibrated risk minimization.

---

Under review as a conference paper at ICLR 2022
Definition 5.1 (Calibrated Risk Minimization). We select model H that minimizes the constrained
expected risk
min ( Y) subject to (HL, Q))] 0,
where a proper scoring rule, T.r is associated calibration loss derived from the calibration- reliability decomposition of T., and Quy) == P(Y =y Fx F).
Recall that proper loss [(F, G) decomposes into sum of calibration (X) Q) and reliability (Q); the latter equals an irreducible term minus sharpness. Thus, by minimizing L(F, G) subject to TLe(H (X) = 0, are maximizing sharpness subject calibration (Gneiting al., 2007).
Note that a special case the above principle is calibrated maximum likelihood, in which we seek a model that maximizes the expected log- likelihood Ex,y log FY( under the calibration constraint that EX || F) = O.
Machine learning models are normally trained to minimize expected risk; our principle asks that in addition they should be calibrated. Our main result (Theorem 3) shows that this criterion is achievable.
5.2 WHY Do WE NEED CALIBRATED AND SHARP UNCERTAINTIES?
Probabilistic models are important building blocks of machine learning systems in many domains- including medicine, robotics, industrial automation, and others. Calibration is not difficult to achieve in many of these domains; hence, we argue that it should be enforced in predictive models, which will unlock the following set of benefits downstream applications.
Safety and Interpretability. Good predictive uncertainties are important for model interpretabil- ity: in user-facing applications. humans make decisions based on model outputs and need assess the confidence of the model, for example when interpreting an automated medical diagnosis. Cal- ibration is also important for model safety: in areas such as robotics, we want minimize the probability of adverse outcomes (e.g., crash), and estimating these outcomes' probabilities is an important step for that (Berkenkamp al., 2017).
Model-Basse -aaed Planning. More generally, good predictive uncertainties also improve downstream decision-making applications such as model-based planning (Malik al., 2019), a setting in which agents learn model of the world to plan future decisions (Deisenroth and Rasmussen, 2011). 1). Plan- ning with a probabilistic model improves performance and sample complexity, especially when representing the model using a deep neural network. and improves the cumulative reward and the sample complexity of model-based agents (Rajeswaran al., 2016; Chua et al., 2018).
Efficient Exploration. Balancing exploration and exploitation is a common challenge in many applications on machine learning such as reinforcement learning, Bayesian optimization, and ac- tive learning. When probabilistic models are uncalibrated, inaccurate confidence intervals might incentivize model explore ineffective actions, degrading performance. Calibrated uncertain- ties have been shown to improve decision-makin in bandits (Malik et al., 2019) and likely to extend to Bayesian optimization and active learning as well.
Other Applications. The importance of accurate confidence estimates has been highlighted by practitioners in many fields, including medicine (Saria, 2018), meteorology (Raftery et al., 2005), and natural language processing (Nguyen and O' Connor, 2015). Accurate confidence estimates also play an important computer vision applications, such as depth estimation (Kendall and Gal, 2017).
EXPERIMENTS
6.1 SETUP
Datasets. We use a number of UCI regression datasets varying in size from 194 to 8192 training instances; each training input may have between 6 and 159 continuous features. We randomly use 25% of each dataset for testing, and use the rest for training. We also perform image classification on the following standard datasets: MNIST, SVHN, CIFAR10.

---

Under review as a conference paper at ICLR 2022
Rayesion Linear Regression
Uncalibrated
Kuleshov al.
Song et al.
Ours
MAE
MАPE
CHK
MAE
МАРE
CHK
MAE
MАPE
CHK
MAE
МАРЕ
CHK
dataset
mpg
2.456
2.398
boston
3.459
0.171
1.392
3.399
0.165
1.365
3.387
0.164
1.372
3.349
0.169
1.312
yacht
6.171
3,800
2.438
5.964
4,802
2.379
1.502
1.326
1.271
0.908
0,336
0.352
0.628
0.106
0.240
0.627
0.105
0.239
0.637
0.106
0.241
0.628
0.106
0.240
crime
0.516
0.087
0.202
0.514
0.086
0.202
0.526
0.092
0.205
0.516
0.086
0.198
0.635
0.058
0.251
0.629
0.057
0.250
0.635
0.062
0.258
0.636
0.058
0.250
upu
39.896
0.647
15.871
39. 09.0086
0.636
15.463
35. 16.166
0.462
13.915
28. 160
0,324
13,615
39.
0.387
Bayesian Neural Network Uncalibrated
Kuleshov et al. MAPE
Song et al.
Ours
dataset
mpg
2.736
0.122
1.198
2.973
0.127
1.176
2.678
0,118
1.101
2.601
0.119
1.083
boston
2.966
0.147
1.237
3.003
0.141
1.206
3.305
0.171
1.181
2.983
0.144
1.404
yacht
3.539
0.592
1.510
wine
0.625
0.105
0.252
0.630
0.104
0.241
0.623
0.109
0.239
0.621
0.103
0.238
crime
0.498
0.085
0.195
0.487
0.083
0.192
0.488
0.082
0.192
0.491
0.083
0.192
0.625
0,059
0.250
0.623
0,058
0.246
0.640
0,060
0.248
0.644
0.061
0.248
upu
74.001
0.518
35.528
71.033
0.633
28.683
68.428
0.641
31.810
66.428
0,630
31,318
bank
96.088
0.722
46.022
90 887
1,105
101
87.096
1,103
41,192
85.096
0.822
39.257
Deep Ensemble Uncalibrated
Kuleshov al.
Song al.
Ours
MAE
MAPE
CHK
MAE
MAPE
CHK
MAE
MAPE
CHK
MAE
MAPE
CHK
datasct
7.667
0.288
3,556
11.858
0.471
5.207
9.573
0.364
3.639
9.010
0.358
3.538
boston
8-427
0.328
3.820
12.800
0.508
5.495
8.132
0.321
3.712
8.004
0.318
3.623
yacht
9.406
0.774
4.604
10.836
1.684
4.932
9.459
2.211
4.312
9.359
2,012
4.278
wine
0.715
0.132
0.308
0.701
0.122
0.276
0.695
0.128
0.269
0.690
0.123
0.268
crime
0.685
0.255
0.779
0.344
0.684
0.251
0.686
0.252
auto
0.862
0.084
0.309
0.862
0.084
0.314
0.878
0.091
0.352
0.868
0.088
0.301
epu
bank 59.088
0.512
21.216
58.731
0.509
21.087
57.493
0.498
20.596
55 869
0.482
19.196
Table 2: Calibration and accuracy on UCT regression datasets. We evaluate Bayesian linear regression, Bayesian neural networks, and deep ensembles using average error (MAF), mean absolute percent ror (MAPE), and the check score (CHK); compare against Kuleshov al. (2018) and Song et al. (2019).
Models. Our first model is Bayesian Ridge Regression (MacKay, 1992). It uses spherical Gaus- sian prior over the weights and a Gamma prior over the precision parameter. Posterior inference is performed in closed form the prior is conjugate.
We also test a number of deep neural networks. We use variational dropout (Gal and Ghahramani, 2016) to produce probabilistic predictions. In our UCI experiments, ward neural networks with two layers of 128 hidden units with a dropout rate of 0.5 and parametric ReLU non-linearities. We use convolutional neural networks (CNNs) on the image classification tasks. These are formed by fine-tuning ResNet50 architecture on the training split for each dataset.
fully-connected feedfor-
We also compare against popular uncertainty estimation method recently developed specifically for deep learning models: deep ensembles (Lakshminarayanan et al., 2017). Deep ensembles average the predictive distributions of multiple models; we ensembled neural networks, each having the same architecture as our standard model.
Our recalibrator R was also densely connected neural network with two fully connected hidden layers of 20 units each and parametric ReLU non-linearities. We added dense skip connections be- tween the layers. In regression experiments, we featurized input distributions F using nine quantiles [0.1, ... 0.9]. We trained Rt using the quantile regression version of Algorithm 1; concatenated the quantile parameter 7 E [o, 1] to the featurization of F. classification experiments, the inputs and the ouputs of R are class probabilities, and R is trained using the ligeeiol maximization version of Algorithm 1. All other architectural details are unchanged.
We did not observe significant overfitting in experiments. We believe overfitting is mitigated by the fact that we perform quantile regression and thus learn a complex distribution function that is not easy overfit.
8

---

Under review as a conference paper at ICLR 2022
6.2 REGRESSION EXPERIMENTS ON UCI DATA
We report the results of our regression experiments on the UCI datasets in Table 2. We evaluate the quality of forecasts using check score pr f) = T f) y f and (1 11-7((f y) as in Song et al. (2019); we average it over nine quantile levels T E 0.1,..., 0.9. We measure regression performance using the mean absolute percent error and mean average error.
Our method improves over the accuracies and uncertainties of Kuleshov et al. (2018), and in many cases over those of Song et (2019) on Bayesian linear regression, Bayesian neural networks, and deep ensembles, without ever being worse. Note that also that out method is simpler and easier to implement than that of Song et al. (2019) (it does not require implementing variational inference). and applies to any input distribution, not just Gaussians.
6.3 CLASSIFICATION EXPERIMENTS ON MNIST, SVHN, CIFAR10
We report the results of the image classification experiments in Table 3. We measure perfor- mance using accuracy and calibration emror of Kuleshov et al. (2018) on the test set. We report these metrics for baseline and calibrated sions of convolutional neural network classifier. We perform recalibration with a simple soft- max regression (a multi-class generalization of Platt scaling) and with the neural network re- calibrator. The best uncertainties are produced by our method. Recalibrated and base models achieve similar levels of accuracy.
MNIST
SVHN
CIFARIO
Base Model
Accuracy Calibration
0,9952
0.9508 0.5975
0.9179
0.3166
0.5848
Platt Scaling
Accuracy
0.9952
0.9508
0.9181
Calibration
0.2212
0.3278
0.2233
Ours
Accuracy
0.9951 0.1030
0.9509 0.2674
0.9163
Calibration
0.1091
Table 3: Performance on Image Classification
PREVIOUS WORK
Probabilistic Forecasting. More modern discussions of probabilistic forecasting can be found in the literature on meteorology (Gneiting and Raftery, 2005). This influential work appears in meth- ods weather forecasting applications systems (Raftery et al., 2005). Most previous work focuses on classification, but recent work (Gneiting et al., 2007; Kuleshov et al., 2018) extends classical methods to regression.
Probabilistic forecasting has been studied extensively in the statistics literature (Murphy, 1973; Dawid, 1984), mainly in the context of evaluation using proper scoring rules (Gneiting and Raftery, 2007). Proper scores measure calibration and sharpness in classification (Murphy, 1973) and regres- sion (Hersbach, 2000).
Calibration. Recalibration is a widely used approach for improving probabilistic forecasts. takes it roots in the classification setting, where Platt scaling (Platt, 1999) and isotonic regression (Niculescu-Mizz and Caruana, 2005) are two widely used algorithms. The have been extended to multi-class (Zadrozny and Elkan, 2002), structured (Kuleshov and Liang, 2015), and online predic- tion Kuleshov and Ermon (2017). There is significant recent interest in calibration in deep learning (Guo et al., 2017; Lakshminarayamannn et al., 2017; Gal et al., 2017: Kuleshov et al., 2018).
CONCLUSION
We take inspiration from the statistics literature and argue that predictive uncertainties should be evaluated by proper scoring rules, which measure two specific qualities of probabilistic predictions: calibration and sharpness. Gneiting et al. (2007) argued that predictive uncertainties should maxi- mize calibration subject to sharpness and used this paradigm to evaluate forecasts. We formalize the paradigm of Gneiting et al. (2007) into a novel learning principle called calibrated risk minimization and propose general algorithm that meets the requirements of this paradigm. Overall, that calibration is a property that can be achieved in predictive models with almost no cost. As such, calibration is rare free lunch that should be enforced throughout applications of machine learning.
show

---

Under review as a conference paper at ICLR 2022
REFERENCES
Felix Berkenkamp, Mattco Turchetta, Angela Schoellig, and Andreas Krause. Safe model-based re- inforcement learning with stability guarantees. In I. Guyon, U. V. Luxbbug, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30, pages 908-918. Curran Associates, Inc., 2017.
Christopher M Bishop. Mixture density networks. 1994.
J. Brocker. Reliability, sufficiency, and the decomposition of proper scores. Quarterly Journal of the Royal Meteorological Society, 133(643):1512-1111) 2009.
Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforcement learning handful of trials using probabilistic dynamics models. S. Bengio, Wallach, H. Larochelle, K. Grauman, N. Cesa- Resa-Bianchi, and R. Garnett, editors, Advances in Neural Infor- mation Processing Systems 31, pages 4759 4770. Curran Associates, Inc., 2018.
A. P. Dawid. Present position and potential developments; Some personal views:
Statistical theory: The prequential approach. Journal of the Royal Statistical Society. Series A (General), 147:278- 292, 1984.
Mare Deisenroth and Carl E Rasmussen. Pilco: model-based and data-efficient approach policy search. In Proceedings of the 28th International Conference on machine learning (ICML-I I), pages 465 472, 2011.
Yarin Gal and Zoubin Ghahramani. Dropout as a Bayesian approximation: Representing model uncertainty deep learning. In Proceedings of the 33rd International Conference on Machine Learning (ICML-16), 2016.
Yarin Gal, Jiri Hron, and Alex Kendall. Concrete dropout. In Advances Neural Information Processing Systems, pages 3584-3593, 2017.
T. Gneiting, F. Balabdaoui, and A. H. Raftery. Probabilistic forecasts, calibration and sharpness. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 69(2):243-268, 2007.
Tilmann Gneiting and Adrian E Raftery. Weather forecasting with ensemble methods. Science, 310 (5746):248-22 2442 2005.
Tilmann Gneiting and Adrian E Raftery. Strietly proper scoring rules, prediction, and estimation. Journal of the American Statistical Association, 102(477):359-377 2007.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. arXiv preprint
2017.
Hans Hersbach. Decomposition of the continuous ranked probability score for ensemble prediction systems. Weather and Forecasting, 15(5):559 57000 2000.
Alex Kendall and Yarin Gal. What uncertainties do we need in bayesian deep learning for computer vision? Advances in Neural Information Processing Systems, pages 5580-5590, 2017.
V. Kuleshov and P. Liang. Calibrated structured prediction. In Advances in Neural Information Processing Systems (NIPS), 2015.
Volodymyr Kuleshov and Stefano Ermon. Estimating uncertainty online against an adversary. In AAAI, pages 21 10-21116. 2017.
Volodymyr Kuleshov, Nathan Fenner, and Stefano Ermon. Accurate uncertainties for deep learning using calibrated regression. In International Conference Machine Learning, pages 1796-2804 PMLR, 2018.
Meelis Kull and Peter Flach. Novel decompositions of proper scoring rules for classification: Score adjustment as precursor to calibration. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pages 68 85. Springer, 2015.
10

---

Under review as a conference paper at ICLR 2022
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. arXiv preprint drXiv:1612.01474. 2017.
David JC MacKay. Bayesian interpolation. Neural computation, 4(3):415-47,, 1992.
Ali Malik, Volodymyr Kuleshov, Jiaming Song, Danny Nemer, Harlan Seymour, and Stefano Er- mon. Calibrated model-based deep reinforcement learning. In International Conference on Ma- chine Learning, pages 43 14 4323. PMLR, 2019.
A. H. Murphy, A new vector partition of the probability score. Journal Applied Meteorology, 12 (4):595-600, 1973.
K. Nguyen and B. O'Connor. Posterior calibration and exploratory analysis for natural language processing models. In Empirical Methods in Natural Language Processing (EMNLP), pages 1587-1598, 2015.
A. Niculescu- Mizil and R. Caruana. Predicting good probabilities with supervised learning. In Proceedings of the 22nd international conference on Machine learning, pages 625-632, 2005.
Platt. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. Advances in Large Margin Classifiers, 10(3):61-74, 1999.
Adrian E Raftery, Tilmann Gneiting, Fadoua Balabdaoui, and Michael Polakowski. Using bayesian model averaging to calibrate forecast ensembles. Monthly weather review, 1336511155-1174, 2005.
Aravind Rajeswaran, Sarvjeet Ghotra, Balaraman Ravindran, and Sergey Levine. Epopt: Learning robust neural network policies using model ensembles. arXiv preprint arXiv: 2016.
Suchi Saria. Individualized sepsis treatment using reinforcement learning. Nature Medicine, 24(11): 1641-1642, 2018. ISSN 1078-8955 8956. doi: 10.10088/411911018-00555
Hao Song, Tom Diethe, Meelis Kull, and Peter Flach. Distribution calibration for regression. In International Conference Machine Learning, pages 5897-5906. PMLR, 2019.
Larry Wasserman. All of nonparametric statistics. Springer Science & Business Media, 2006.
K. Werling, A. Chaganty, P. Liang, and Manning. On-the-job learning with Bayesian decision theory. In Advances in Neural Information Processing Systems (NIPS), 2015.
B. Zadrozny and C. Elkan. Transforming classifier scores into accurate multiclass probability timates. In International Conference on Knowledge Discovery and Data Mining (KDD), pages 2002.


Reviews:
[{"review_id": "yOMqxuJl-Fz", "reviewer": "Reviewer_qXzz", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper addresses an important problem, namely that of ensuring that uncertainties output by supervised machine learning models are well calibrated. It is largely well written and well motivated. The method is simple and generally applicable. It mainly represents a generalization of prior recalibration approaches. Empirical results on the UCI regression datasets show that the model consistently improves calibration compared to the base model, and modestly improves over previous work by Song et al. It also shows strong improvements on image classification tasks.\n\nMy main concern with the paper is that its theoretical analysis appears somewhat tautological to me. It starts with the assumption that the calibrator R is close to Bayes-optimal, and shows that in this case, it ensures calibration without hurting sharpness. But given that R was trained on the same loss, and on the same data distribution as the base model H, why should it be any closer to Bayes optimal than H? Since the analysis makes no assumptions regarding the nature of R or H, it seems difficult to make any definitive statements in this regard. I suspect the answer is that, in practice, R will have much fewer parameters than H and be less prone to overfitting. It would be important for the paper to analyze the specific conditions under which recalibration is effective, instead of simply asserting that it always works.\n\nSpecific Questions/Comments:\n - I am confused by the definition of the check score in section 2.2. It appears that f denotes an outcome from Y, as opposed to a probability distribution function over y, which is how it was used previously.\n - The theoretical analysis states that base model and calibrator are trained on different training sets, but the experimental section does not mention anything in this regard. Were the sets actually diffirent, and if so, what was their size? Do you think this is an important setting?\n - It is unclear to me to what degree the experimental outcomes are effected by the increased capacity of the recalibrated model. It would be good to provide a baseline in which R and H are jointly trained on all the training data.\n - There are a few broken sentences, e.g. at the end of the model-based planning paragraph in section 5.2.", "summary_of_the_paper": "The paper proposes a new method for calibrating the uncertainty estimates of supervised machine learning models, i.e., ensuring that they match the probabilities of the data distribution. After regular model training, a calibrator model is trained to transform the model output, using the same loss but a separate dataset. Theoretical analysis and empirical experiments are conducted to support the claim that this is a way to calibrate \"any model\" at \"almost no cost\".", "summary_of_the_review": "In summary, the paper provides a simple method that empirically generally improves model calibration. However, the theoretical analysis is overly general and does little to identify the specific conditions under which is works. As a result, I am skeptical of the very broad claim that it will be effective on any model and learning problem.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "hpXu9RMQr79", "reviewer": "Reviewer_fwct", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n- The paper proposes a simple recalibration algorithm\n- The proposed method works if the number of outputs is reasonably small (one or two)\n- Extensive simulation\n\nWeaknesses:\n\n- Unclear / small novelty compared to previous work\n- Theoretical results are asymptotic\n- Empirical results are quite weak\n\nI have previously served as a reviewer for this paper for NeurIPS 2021. We ended up rejecting the paper there and recommended a few major points for improvement:\n- Performance in higher dimensions: The current method seems to only work well in very low-dimensional settings (one to two dimensions or so). It should be discussed how it would (or would not) generalize to higher-dimensional problems, which are relevant and common in many areas.\n- Empirical results: The current results are missing error bars, which makes it impossible to assess their statistical significance. In fact, the proposed method underperforms the baselines in many settings, which may or may not be significant.\n- Missing baselines: Previous work has proposed to optimize specialized calibration losses, such as [1,2,3]. It would be necessary to compare to at least some of these related approaches to judge whether the proposed approach actually improves over prior work.\n\nGiven that it seems like the authors have unfortunately implemented none of these suggestions yet, I will for now have to stick to my previous assessment of the work and recommend rejection again.\n\n\n\n[1] Khosravi et al. 2011, https://ieeexplore.ieee.org/document/5672788\n\n[2] Pearce et al. 2018, https://arxiv.org/abs/1802.07167\n\n[3] Tagasovska et al. 2018, https://arxiv.org/abs/1811.00908", "summary_of_the_paper": "The authors propose to decompose different common loss functions into calibration and refinement terms and show theoretically that under some (strong) assumptions, such as universal approximation, models can become perfectly calibrated without sacrificing predictive performance. They propose a constrained optimization framework to achieve such results and demonstrate it empirically on different tasks.", "summary_of_the_review": "This work has previously been rejected from NeurIPS 2021 due to the absence of a discussion on the performance in higher dimensions, the lack of errors bars in the experimental results, and missing baselines. Since none of these points have improved in the resubmission, I will again recommend rejection.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "fIkyQWR5Yex", "reviewer": "Reviewer_GeKX", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, I liked this paper. It is well-written in the sense that the text feels clear and readable, and the authors do a good job of situating their work with respect to calibration literature and recapitulating important results.\n\nI have a few reservations and concerns which currently prevent me from recommending acceptance.\n\nFirst, my worry is about what precisely we are calibrated with respect to. Imagine, for example, the algorithm involved taking D = {empty set}, H = identity, and R = {all training data}. At first glance, this ought to result in calibrated models, while it is also identical to standard practice. Can you explain to me what I'm missing here? Why does splitting the calibration into two non-trivial steps make the difference? How would you explain why standard training with a proper loss does not result in a calibrated model?\n\nSecond, there are other kinds of uncertainty that seem not to be represented in your model, which are what Bayesian deep learning tries to address.\nFor example, it seems straightforward to overfit the 'calibration' to C, and then be poorly calibrated on some other test data.\nThe algorithm might not be solving some sort of generalizable calibration problem, but quite a specific one.\nIn general, it would be useful to engage more on prior work about uncertainty in deep learning.\n\nThird, there is a bit of a jump between your lemmas ant Theorem 3 which make claims entirely asymptotically and probabilistically and then your assertion that perfect calibration is available at almost no cost as a `free lunch'.\nThis is also reflected in your definition of CRM (5.1), which builds off the idea of having zero expected calibration loss, but we don't know how to actually enforce that constraint, we can only bound the calibration loss with high probability.\n\nFourth, it did not feel like very much attention was paid to the probabilistic nature of the requirement.\nIn related fields like differential privacy, say, a number of epsilon/delta guarantees are available, but with specific values of epsilon and delta that are prohibitive in real data contexts.\nI would like to see an empirical examination of the bounds in your theorems under realistic contexts.\n\nFifth, the paper is currently unclear about the training/calibration/test data split.\nFor UCI you say 25% is used for testing and the rest for training.\nHow is the training data split between the training data and calibration data?\nOr is the test data being used to calibrate?\nIf so, I'm afraid that would be problematic.\nSimilarly, please be explicit about how this is done for the image data.\n\nSixth, for UCI I would like to see a more thorough assessment of the relative strengths of the different methods.\nYou write that your method improves on prior work in many places \"without ever being worse\", but that doesn't obviously seem to be the case (e.g., Deep ensemble mpg where your method seems worse than an uncalibrated baseline).\nIt's quite hard to make sense of giant tables of numbers like this, so it might be helpful to do some plots or post-processing to tease out the relationships.\n\nSeventh, and a minor point that doesn't affect my score, there are a number of typos (e.g., \"computing *extract* predictive uncertainties\", \"distirbutional\") so the paper could probably use another pass. The phrase \"w.h.p.\" is also introduced without being defined, but is actually important to discuss (I assume you mean 'with high probability').", "summary_of_the_paper": "The authors propose an algorithm scheme that they prove provides probabilisitic asymptotic guarantees of `distribution' calibration for models.\nThe algorithm provided by the authors fits a primary model, H, on a training dataset, D, as well as a recalibrator, R, which takes H(x) as an input, using a calibration dataset, C.\nH is fit using (x,y) pairs from D and a proper loss. R is then fit using (H(x), y) pairs from C and a proper loss.\nThe resulting model ought to be calibrated on C while also being (asymptotically) as sharp as the original model.\nThe authors also introduce the idea of calibrated risk minimization, which is a constrained optimization alternative to empirical risk minimization.", "summary_of_the_review": "I do not currently recommend acceptance.\nMy main concern is some conceptual clarity about what different data-sets are doing which is reflected both in the theoretical and experimental sections.\nI would like to see more thorough examination of the probabilistic bounds and the degree of slack they afford, which could help better ground the claims that calibration is a free lunch and justify CRM as an approach.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "9OGByQChNBS", "reviewer": "Reviewer_EFrC", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Significance and contribution**\n\n\nIn the abstract, introduction and theoretical analysis there are some ambitious claims such as “This paper introduces algorithms that ensure the calibration of any model while maintaining sharpness”,  “our algorithms achieve for the first time a long-standing statistical principle that forecasts should maximize sharpness subject to being calibrated”, and “in a sense, calibration can be a free lunch“. In my view, these are unsubstantiated.\n\nMethodologically speaking, it is not clear to me what is the contribution of this paper. The paper states that it proposes a framework for recalibration, within which any flexible model class can be used as a recalibrator.  In practise, as far as I can tell, one important difference from previous works is the use of NNs with a simultaneous quantile regression noise model as the recalibrator. \n\nThe theory section contains a series of propositions regarding asymptotically optimal calibration. My impression is that these are somewhat vacuous: we would not usually have an infinite amount of held-out data to train an infinitely flexible recalibration model. Again, I think that saying that “calibration can be a free lunch“ based on this is a bit wishful. It is like saying that machine learning is a free lunch because we have asymptotically optimal learners. \n\n\n\n**Clarity and Writing** \n\nThe introduction and background sections of the paper read very well. The problem is explained and motivated well. I would say that simple concepts are explained in an unnecessarily notation-heavy way at times. \n\nI was left somewhat confused by the methodological section (3). It reads a bit like a literature review. I did not understand what the methodological contribution of the paper was from reading this section. \n\nI was also somewhat confused by section 5. Here “Calibrated Risk Minimization” is introduced in definition 5.1 as a principle by which the predictive entropy is minimised while miscalibration is kept at a minimum. As far as I can tell, no procedure is actually given to obtain this result.\nSection 5.2 also feels disconnected. It gives a series of example settings in which calibrated uncertainty would be useful. I would suggest moving this information to the introduction or related work. \n\n**Experiments**\n\nI think key details are missing from the experimental section. It is unclear to me what the exact procedure used to recalibrate predictive distributions is. Did the authors employ the standard procedure of training a secondary model on some held out test data using a standard objective? Or did they employ some more sophisticated method designed to adhere to the “Calibrated Risk Minimization” principle introduced in 5.1?\n\nI could not find what percentage of training data was used for recalibration in the prose. I think the authors should mention this, as it is an important hyperparameter. \n\nI would recommend bolding important figures in the tables. In its current state, table 2 looks like a matrix of 288 numbers. It is very difficult to read. I would also encourage the authors to repeat their experiments with different random seeds and report error bars. ", "summary_of_the_paper": "This paper proposes to re-calibrate the probabilistic predictions of machine learning models by learning a secondary function that maps from their output parameters to a new set of calibrated distribution parameters. The secondary function is learnt on a held out set. The paper argues for the use of flexible recalibration methods, appealing to their asymptotic optimality. Experiments are performed on 8 UCI regression datasets, MNIST, SVHN and CIFAR10.  NNs are used as both primary and recalibration models. Superior results are obtained compared to some recent baselines.", "summary_of_the_review": "Although I do not have a large amount of research experience in the domain of calibration I do have quite a bit of applied experience in this domain. For that reason, after reading the introduction and abstract I was quite excited about this paper as I was expecting some sort of theoretical breakthrough. After reading the rest of the paper I was just left confused. Unless I am misunderstanding something important, I think there is a large mismatch between the claims this paper makes and the theoretical and empirical results it delivers. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers highlight that several of significant claims of the paper are not backed up by experiments, and the experiments themselves lack sufficient detail, therefore, at this stage, I recommend rejection. I suggest the authors address the questions and comments they have received before considering whether they might resubmit or not.
Decision: Reject