Year: 2021

Abstract:
Predictive uncertainties can be characterized by two properties—calibration and sharpness. This paper introduces algorithms that ensure the calibration of any model while maintaining sharpness. They apply in both classification and regression and guarantee the strong property of distribution calibration, while being simpler and more broadly applicable than previous methods (especially in the context of neural networks, which are often miscalibrated). Importantly, these algorithms achieve long-standing statistical principle that forecasts should maximize sharpness subject to being fully calibrated. Using our algorithms, machine learning models can under some assumptions be calibrated without sacrificing accuracy: in a sense, calibration can be a free lunch. Empirically, we find that our methods improve predictive uncertainties on several tasks with minimal computational and implementation overhead.

heading: 1 INTRODUCTION
text: Probabilistic forecasts can be characterized by two properties—calibration and sharpness (Gneiting et al., 2007). Intuitively, calibration means that a 90% confidence interval contains the true outcome 90% of the time. Sharpness means that these confidence intervals are narrow. These qualities are grounded in the statistics literature on proper scoring rules, and are widely used to evaluate forecasts (Gneiting and Raftery, 2005; 2007) in domains such as medicine (Saria, 2018), robotic control (Malik et al., 2019), and human-in-the-loop learning (Werling et al., 2015).
This paper introduces algorithms that ensure the calibration of any predictive machine learning model while maintaining sharpness. They apply to both classification and regression tasks and guarantee the strong property of distribution calibration (which generalizes standard quantile and classification-based calibration; Song et al. (2019)) in any model, including deep learning models, which are often miscalibrated (Guo et al., 2017). Unlike existing methods for distribution calibration (Song et al., 2019), ours can be used with any model (not just ones that output Gaussians), are very simple to implement in differentiable programming frameworks, and have theoretical guarantees.
Importantly, our algorithms achieve for the first time a long-standing statistical principle that forecasts should “maximize sharpness subject to being calibrated” proposed by Gneiting et al. (2007). We prove that under some assumptions this principle is achievable in modern machine learning models in a black-box manner and without sacrificing overall performance. This lends strong support for this principle as a way of reasoning about uncertainty in machine learning.
In a sense, calibration is a rare free lunch in machine learning, and we argue that it should be enforced in predictive models and taken advantage of in downstream applications. Empirically, we find that our method consistently outputs well-calibrated predictions across a wide range of experiments, while improving performance on downstream tasks with minimal implementation overhead.
Contributions. In summary, we make three contributions. We propose a new recalibration technique that (a) is among the only to guarantee distribution calibration besides Song et al. (2019). Unlike Song et al. (2019) we can (b) recalibrate any parametric distribution (not just Gaussians) and (c) our method is simpler. While theirs is based on variational inference in Gaussian processes, ours uses a neural network that can be implemented in a few lines of code, which encourages adoption. Our method (d) applies to both classification and regression and (e) outperforms methods by Song et al. (2019) and Kuleshov et al. (2018) as well as Platt and temperature scaling.
We also formally prove that our technique produces asymptotically distributionally calibrated forecasts while minimizing regret. Most methods (e.g., Platt, Kuleshov scaling, Song et al., etc.) do not have a correctness proof, except for conformal prediction, which is significantly more complex.
Finally, our analysis formalizes the well-known paradigm of Gneiting et al. (2007) and provides the first method that provably achieves it. This lends strong support for this principle and influences how one should reason about uncertainty in machine learning. An important takeaway is that calibration can be achieved in most applications of machine learning with very little cost. As such, calibration can be a rare free lunch that we believe should be leveraged throughout machine learning.

heading: 2 BACKGROUND

heading: 2.1 PREDICTIVE UNCERTAINTY IN MACHINE LEARNING
text: Supervised machine learning models commonly predict a probability distribution over the output variables — e.g. class membership probabilities or the parameters of an exponential family distribution. These predictive uncertainties are useful for interpretability, safety, and downstream decisionmaking. Aleatoric uncertainty captures the inherent noise in the data, while epistemic uncertainty arises from not having a large enough dataset to estimate model parameters (Kendall and Gal, 2017).
Notation. Formally, we say that a machine learning forecaster H : X → ∆(Y) outputs a probability distribution F (y) : Y → [0, 1] in the space ∆(Y) of distributions over y. We use f to denote the probability density or probability mass function associated with F . The model H is trained on a labeled dataset xt, yt ∈ X ×Y for t = 1, 2, ..., T of i.i.d. realizations of random variablesX,Y ∼ P, where P is the data distribution.

heading: 2.2 WHAT DEFINES GOOD PREDICTIVE UNCERTAINTIES?
text: The standard tool in statistics for evaluating the quality of predictive uncertainties is a proper scoring rule (Gneiting and Raftery, 2007). Formally, a scoring rule S : ∆(Y) × Y → R assigns a “score” to a probabilistic forecast F ∈ ∆(Y) and a realized outcome y ∈ Y . Given a true distribution G ∈ ∆(Y) for y, we use the notation S(F,G) for the expected score S(F,G) = Ey∼GS(F, y). We say that a score S is proper if it is minimized by G when G is the true distribution for y: S(F,G) ≥ S(G,G) for all F . When S is proper, we also refer to it as a proper loss. An example of a proper loss is the log-likelihood S(F, y) = log f(y), where f is the probability density or probability mass function of F . Another common loss is the check score ρτ (y, f) = τ(y − f) if y ≥ f and (1 − τ)(f − y) otherwise; it can be used to estimate the τ -th quantile of a distribution. See Table 1 for additional examples.
What are the qualities of a good probabilistic prediction, as measured by a proper scoring rule? It can be shown that every proper score is a sum of the following terms (Gneiting et al., 2007):
proper loss = calibration−sharpness + irreducible term︸ ︷︷ ︸ refinement term .
Thus, there are precisely two qualities that define an ideal forecast: calibration and sharpness. We examine each of them next.

heading: 2.3 CALIBRATION AND SHARPNESS — TWO QUALITIES OF AN IDEAL PREDICTION
text: Formally, calibration can be defined by the equation
P(Y = y | FX = F ) = f(y) for all y ∈ Y , F ∈ ∆(Y), (1)
where X,Y ∼ P are random variables corresponding to the input features and targets, and FX = H(X) is the forecast at X , itself a random variable that takes values in ∆(Y). We use f to denote the probability density or probability mass function associated with F .
When Y = {0, 1} and FX is a Bernoulli distribution with parameter p, we can write (1) as P(Y = 1 | FX = p) = p. This has a simple intuition: the true probability of Y = 1 is p conditioned
on predicting it as p. Equation 1 extends beyond binary classification to arbitrary distributions. For example, if F is a Gaussian with variance σ2, this definition asks that the data distribution conditioned on predicting F also has variance σ2. This recently proposed definition is called distribution calibration (Song et al., 2019).
A closely related, but weaker concept is quantile calibration (Kuleshov et al., 2018), which asks that a 90% confidence interval contains the true value 90% of the time. Formally, it can be written as: P(Y ≤ CDF−1FX (p)) = p for all p ∈ [0, 1], Quantile calibration is implied by distributional calibration (Song et al., 2019).
Calibration by itself is not sufficient to produce a useful forecast. For example, it is easy to see that a binary classifier that always outputs P(Y = 1) as the probability that Y = 1 is calibrated; however it does not even use the features X and thus cannot be accurate.
In order to be useful, forecasts must also be sharp. Intuitively, this means that predicted confidence intervals should be as tight as possible around a single value. This is captured by proper scoring rules as part of a refinement term (see Table 1), which equals an irreducible term minus a sharpness term (Murphy, 1973; Brocker, 2009). The latter is maximized when we minimize the scoring rule.
Are Modern Machine Learning Models Calibrated And Sharp? Most machine learning models are not calibrated out-of-the-box (Niculescu-Mizil and Caruana, 2005; Guo et al., 2017). Two reason for this are the limited expressivity of the model H—we cannot perfectly fit the entirety of the level curves of the data distribution—and computational approximations—computing extract predictive uncertainties may be intractable, and approximations are not entirely accurate.
A final reason stems from how models are trained—since we cannot fit a perfect H , standard objective functions induce a tradeoff between sharp and calibrated forecasts. Next, we will show that by training models differently, we can achieve calibration without sacrificing performance.

heading: 3 ENSURING DISTRIBUTION CALIBRATION IN ANY MODEL
text: This section introduces algorithms that ensure the distirbutional calibration of any predictive machine learning model while maintaining sharpness. Unlike existing methods for distribution calibration, ours can be used with any model (not just ones that output Gaussians), are very simple to implement in differentiable programming frameworks, and have theoretical guarantees.
We first assume there exists a parameterization Φ1 of the probabilities returned by forecaster H: for each p ∈ ∆(Y) returned by H , there exist parameters φ ∈ Φ1 that describe p. The φ can be the natural parameters of an exponential family distribution, such as (µ, σ2) describing a Gaussian.
We consider a class of algorithms based on a classic approach called recalibration. First, we train a base forecaster H to minimize a proper loss L. Then, we train an auxiliary model R : Φ1 → Φ2 (called the recalibrator) over the outputs of H that outputs the parameters φ2 ∈ Φ2 of another
distribution such that L is minimized. Here Φ2 is a second parameterization of ∆(Y) (possibly the same). As a result, the forecasts (R ◦H)(X) will be calibrated. We provide details in Algorithm 1.
Algorithm 1 Calibrated Learning of Probabilistic Models. Input: Model H : X → Φ1, recalibrator R : Φ1 → Φ2, training set D, recalibration set C Output: Recalibrated model R ◦H : X → Φ2.
1. Fit the base model on D: minH ∑ (x,y)∈D L(H(x), y)
2. Fit the recalibration model R on the output of H on C: minR ∑ (x,y)∈C L ((R ◦H)(x), y)
Implementing this approach requires choosing parameterizations of probabilities Φ1,Φ2, a recalibration model R, and an objective L. We discuss these choices below; then we clarify how 1 differs from existing recalibration algorithms.
Parameterizing Probability Distributions. When fitting a classification model, each distribution ∆(Y) is a categorical and can be parameterized via its K ≥ 2 class membership probabilities. In regression, most widely used models such as neural networks already output parameterized probabilities, in which the φ are usually the natural parameters of an exponential family model.
In the most general case, if we only have black-box access to a density function or a cumulative distribution function, we may form a d-dimensional representation by evaluating the distribution at a grid of d points. For example, if we have black-box access to a quantile function F−1, we may featurize F via its sequence of quantiles φ(F ) = (F−1(αi))di=1 for some sequence of d levels αi, possibly chosen uniformly in [0, 1].
In addition to the above techniques, the representation of output probabilities in Φ2 coming from R can leverage flexible invertible models of the CDF, following methods developed in the normalizing flow literature, including monotonic neural networks, sum-of-squares polynomials (Wehenkel and Louppe, 2019; Jaini et al., 2019) spline functions (Muller et al., 2019; Durkan et al., 2019), piecewise separable models (Wehenkel and Louppe, 2019), and others.
Choosing a Recalibrator. Ideal recalibrators are highly effective at optimizing the proper loss L (see Section 4). In a simple setting like binary classification, our task reduces to one-dimensional density estimation; in such cases we can provably achieve calibration asymptotically by using kernel density estimation for the recalibrator R, while controlling the kernel width as a function of the dataset size to trade off overfitting and underfitting (Wasserman, 2006). In regression settings, we may rely on other non-parametric techniques such as Gaussian processes.
An alternative approach is to rely on expressive neural networks; although their optimization is a non-convex, they are very effective at fitting proper losses L, feature mature regularization techniques, and can be implemented easily within deep learning frameworks, possibly within the same computation graph as a neural forecaster H , which can simplify deployment.
In the classification setting, a natural architecture for R is a sequence of dense layers mapping the simplex ∆K into ∆K . In regression settings, R needs to output a density function: a natural architecture for this is a mixture density network (MDN; Bishop (1994)).
Choosing a Proper Loss A natural choice of proper loss is the log-loss. It applies in both calibration and regression; optimizing it is a standard supervised learning problem.
In regression settings, we found that using the quantile loss L = Eτ∈U [0,1]Ey∼Gρτ (y − F−1(τ)) (see Table 1) was numerically stable and produced the best performance. This objective fits a model Rθ(τ ;φ) to estimate the τ -th conditional quantile F−1(τ) at φ. When Rθ(τ ;φ) is a neural network that takes in τ and x, we minimize the quantile loss Eτ∈U [0,1]Ey∼Gρτ (y − F−1(τ)) using gradient descent, approximating both expectations using Monte Carlo (details are in the experiments section).
Comparing Against Song et al. Interestingly, the method of Song et al. (2019) is a special case of ours when Φ1 consists of Gaussian natural parameters, Φ2 consists of parameters for the Beta link function, R is a Gaussian process, and L is the log-likelihood. However, the resulting problem can
only be solved using variational inference, which is slow and complex to implement. Our framework instead admits simple solutions based on gradient descent.
Comparing Against Other Approaches. Algorithm 1 performs recalibration like many previous methods (e.g., Platt, Kuleshov et al., Song et al., etc.); it may thus appear to be the same as these methods. However, that is not the case. First, existing recalibration approaches operate over the space of probabilities (class probabilities or CDF values); ours operates over functional parameters. This is what enables it to achieve distribution rather than quantile calibration (Kuleshov et al., 2018).
Our approach also involves novel recalibration objectives (e.g., the quantile divergence in regression) which differ from the calibration error of Kuleshov et al. (2018). We also use different types of models (small neural networks instead of isotonic regression) and different optimization procedures used (stochastic gradient descent instead of variational inference). Thus, our recalibration strategy is distinct from previous work.

heading: 4 THEORETICAL ANALYSIS
text: Next, we now show that under some assumptions calibration is provably achievable in modern machine learning models in a black-box manner and without sacrificing overall performance. In that sense, calibration is a rare free lunch in machine learning.
We start with some notation. We have a recalibration dataset of size T sampled from P and train a recalibrator R : ∆(Y) → ∆(Y) over the outputs of a base model H to minimize a proper loss L. We denote the Bayes-optimal recalibrator by B := P(Y = y | H(X)); the distribution of Y conditioned on the forecast (R ◦ H)(X) is Q := P(Y = y | (R ◦ H)(X)). We are interested in expectations of various losses over X,Y ; to simplify notation, we omit the variable X , e.g. E[L(R ◦H,Y )] = E[L(R(H(X)), Y )]. Next, we will assume the following condition under which Algorithm 1 works. Assumption 1. The model R can minimize expected risk such that w.h.p. we have
E[L(B ◦H,Y )] ≤ E[L(R ◦H,Y )] < E[L(B ◦H,Y )] + δ where δ > 0, δ = o(T ) is a bound that decreases with T and E[L(B ◦H,Y )] is the irreducible loss.
This assumption implies that the recalibrator can perform density estimation in what is usually a small number of dimensions (one or two). For some recalibrators, e.g., neural nets, it may not provably hold (e.g., because of non-convexity). However, neural networks are effective density estimators in practice, and we can quantify whether they estimate density well on a hold-out set. This assumption provably holds for many non-parametric density estimation methods. Fact 1 (Wasserman (2006)). When R implements kernel density estimation and L is the log-loss, Assumption 1 holds with δ = o(1/T 2/3).
We now prove two key lemmas. We show that Algorithm 1 outputs calibrated forecasts without reducing the performance of the base model, as measured by regret relative to loss L. Lemma 1. The model R ◦H is asymptotically calibrated, in the sense that E[Lc(R ◦H,Q)] < δ for δ = o(T ) w.h.p.
Proof. Recall that the loss E[L(R ◦ H,Y )] decomposes into a sum of calibration and refinement terms E[Lc(R ◦H,Q)] + E[Lr(Q)] where Q(y) := P(Y = y | (R ◦H)(X)). As shown by Kull and Flach (2015), refinement further decomposes into a group loss and an irreducible term: E[Lr(Q)] = E[Lg(Q,B ◦H)] + E[L(B ◦H,Y )], where B(Y = y | H(X)) is the Bayes-optimal recalibrator. The form of the group loss Lg is the same as that of Lc. We may then write:
E[L(B ◦H,Y )] Bayes-Optimal Loss ≤ E[Lc(R ◦H,Q)] Calibration Loss +E[Lg(Q,B ◦H)] Group Loss +E[L(B ◦H,Y )] Bayes-Opt Loss
= E[L(R ◦H,Y )] Proper Loss < E[L(B ◦H,Y )] Bayes-Optimal Loss +δ
where δ > 0, δ = o(T ). In the first equality we used the decomposition of Kull and Flach (2015) and in the last inequality we used Assumption 1. It follows that E[Lc(R ◦ H,Q)] < δ, i.e. the calibration loss is small.
Lemma 2. The recalibrated model is asymptotically as good as the base model: E[L(R ◦H,Y )] ≤ E[L(H,Y )] + γ, where γ > 0, γ = o(T ) is a bound that decreases with T .
Proof. The claim holds by empirical risk minimization, since R ◦ H minimizes L, but is more expressive than H and R can represent the identity map (by Assumption 1).
We now combine these two lemmas to show Algorithm 1 ensures calibration and low regret.
Theorem 3. Algorithm 1 produces a model that minimizes expected risk, while w.h.p. achieving asymptotically optimal calibration.
Proof. The base model H is trained using empirical risk minimization (ERM). The model R ◦ H minimizes the same objective L, hence minimizes the same expected risk by ERM theory. Also, by Lemma 2, the expected risk of R ◦H also asymptotically approaches a lower value as that of H . By Lemma 1, the model R ◦H produces asymptotically calibrated forecasts w.h.p.
Thus, given enough data, we are guaranteed to produce calibrated forecasts and preserve base model performance (as measured by L). Thus, calibration is a property that can be achieved in most applications of machine learning with almost no cost. As such, calibration is a rare free lunch in machine learning.
Finite-Sample Bounds. Note that our analysis provides finite-sample and not only asymptotic bounds on the regret and calibration error—the bounds are stated in terms of variables δ, and γ that are each o(T ). The bound δ on the calibration error directly depends on the finite-sample bound on the generalization error of the algorithm used as the recalibrator.
Practical Considerations. Assumption 1 suggests that we want to use a model family that can minimize the expected risk E[L(H(X), Y )] well. Thus, in practice we want to select a highly flexible algorithms for which we can control overfitting and underfitting. This motivates our earlier advice of using density estimation algorithms—which have provable guarantees—and neural networks—which are expressive and feature effective regularization techniques

heading: 5 WHAT UNCERTAINTIES ARE NEEDED IN MODERN DEEP LEARNING?
text: Good predictive uncertainties are calibrated and sharp and these two properties yield optimal values of the log-likelihood and other proper loss functions. Thus, they characterize an ideal forecast. In practice, however, modern machine learning models do not output such ideal predictions. What then is the ideal type of forecast that we should aim to obtain from our models?
Gneiting et al. (2007) argue that predictive uncertainties should be maximally sharp subject to being calibrated. They propose a diagnostic approach based on this principle; this approach is commonly used in statistics for evaluating the predictive performance of probabilistic models.
In this paper, we also argue for this general principle, but approach it in a prescriptive way — we claim that this principle should be enforced in modern ML systems, and we show how to do so. Specifically, we show that any model can be modified to output calibrated uncertainties, and this property can be provably achieved without sacrificing performance.

heading: 5.1 CALIBRATED RISK MINIMIZATION
text: We formalize the intuition behind ”maximizing sharpness subject to being calibrated” as follows. We argue that we should be training models to minimize expected risk (as measured by a proper loss) subject to being perfectly calibrated. We call this principle calibrated risk minimization.
Definition 5.1 (Calibrated Risk Minimization). We select a modelH that minimizes the constrained expected risk
min H E[L(H(X), Y )] subject to E[Lc(H(X), Q))] = 0, where L is a proper scoring rule, Lc is its associated calibration loss derived from the calibrationreliability decomposition of L, and Q(y) := P(Y = y | FX = F ).
Recall that a proper loss L(F,G) decomposes into a sum of calibration Lc(H(X), Q) and reliability Lr(Q); the latter equals an irreducible term minus sharpness. Thus, by minimizing L(F,G) subject to Lc(H(X) = 0, we are maximizing sharpness subject to calibration (Gneiting et al., 2007).
Note that a special case of the above principle is calibrated maximum likelihood, in which we seek a model that maximizes the expected log-likelihood EX,Y logFX(Y ) under the calibration constraint that EX,Y KL(Q || F ) = 0. Machine learning models are normally trained to minimize expected risk; our principle asks that in addition they should be calibrated. Our main result (Theorem 3) shows that this criterion is achievable.

heading: 5.2 WHY DO WE NEED CALIBRATED AND SHARP UNCERTAINTIES?
text: Probabilistic models are important building blocks of machine learning systems in many domains— including medicine, robotics, industrial automation, and others. Calibration is not difficult to achieve in many of these domains; hence, we argue that it should be enforced in predictive models, which will unlock the following set of benefits in downstream applications.
Safety and Interpretability. Good predictive uncertainties are important for model interpretability: in user-facing applications, humans make decisions based on model outputs and need to assess the confidence of the model, for example when interpreting an automated medical diagnosis. Calibration is also important for model safety: in areas such as robotics, we want to minimize the probability of adverse outcomes (e.g., a crash), and estimating these outcomes’ probabilities is an important step for that (Berkenkamp et al., 2017).
Model-Based Planning. More generally, good predictive uncertainties also improve downstream decision-making applications such as model-based planning (Malik et al., 2019), a setting in which agents learn a model of the world to plan future decisions (Deisenroth and Rasmussen, 2011). Planning with a probabilistic model improves performance and sample complexity, especially when representing the model using a deep neural network. and improves the cumulative reward and the sample complexity of model-based agents (Rajeswaran et al., 2016; Chua et al., 2018).
Efficient Exploration. Balancing exploration and exploitation is a common challenge in many applications on machine learning such as reinforcement learning, Bayesian optimization, and active learning. When probabilistic models are uncalibrated, inaccurate confidence intervals might incentivize the model to explore ineffective actions, degrading performance. Calibrated uncertainties have been shown to improve decision-making in bandits (Malik et al., 2019) and likely to extend to Bayesian optimization and active learning as well.
Other Applications. The importance of accurate confidence estimates has been highlighted by practitioners in many fields, including medicine (Saria, 2018), meteorology (Raftery et al., 2005), and natural language processing (Nguyen and O’Connor, 2015). Accurate confidence estimates also play an important in computer vision applications, such as depth estimation (Kendall and Gal, 2017).

heading: 6 EXPERIMENTS

heading: 6.1 SETUP
text: Datasets. We use a number of UCI regression datasets varying in size from 194 to 8192 training instances; each training input may have between 6 and 159 continuous features. We randomly use 25% of each dataset for testing, and use the rest for training. We also perform image classification on the following standard datasets: MNIST, SVHN, CIFAR10.
Models. Our first model is Bayesian Ridge Regression (MacKay, 1992). It uses a spherical Gaussian prior over the weights and a Gamma prior over the precision parameter. Posterior inference is performed in closed form as the prior is conjugate.
We also test a number of deep neural networks. We use variational dropout (Gal and Ghahramani, 2016) to produce probabilistic predictions. In our UCI experiments, we use fully-connected feedforward neural networks with two layers of 128 hidden units with a dropout rate of 0.5 and parametric ReLU non-linearities. We use convolutional neural networks (CNNs) on the image classification tasks. These are formed by fine-tuning a ResNet50 architecture on the training split for each dataset.
We also compare against a popular uncertainty estimation method recently developed specifically for deep learning models: deep ensembles (Lakshminarayanan et al., 2017). Deep ensembles average the predictive distributions of multiple models; we ensembled 5 neural networks, each having the same architecture as our standard model.
Our recalibrator R was also a densely connected neural network with two fully connected hidden layers of 20 units each and parametric ReLU non-linearities. We added dense skip connections between the layers. In regression experiments, we featurized input distributions F using nine quantiles [0.1, ..., 0.9]. We trained R using the quantile regression version of Algorithm 1; we concatenated the quantile parameter τ ∈ [0, 1] to the featurization of F . In classification experiments, the inputs and the ouputs of R are class probabilities, and R is trained using the log-likelihood maximization version of Algorithm 1. All other architectural details are unchanged.
We did not observe significant overfitting in our experiments. We believe overfitting is mitigated by the fact that we perform quantile regression and thus learn a complex distribution function that is not easy to overfit.

heading: 6.2 REGRESSION EXPERIMENTS ON UCI DATA
text: We report the results of our regression experiments on the UCI datasets in Table 2. We evaluate the quality of forecasts using a check score ρτ (y, f) = τ(y − f) if y ≥ f and (1 − τ)(f − y) as in Song et al. (2019); we average it over nine quantile levels τ ∈ 0.1, ..., 0.9. We measure regression performance using the mean absolute percent error and mean average error.
Our method improves over the accuracies and uncertainties of Kuleshov et al. (2018), and in many cases over those of Song et al. (2019) on Bayesian linear regression, Bayesian neural networks, and deep ensembles, without ever being worse. Note that also that out method is simpler and easier to implement than that of Song et al. (2019) (it does not require implementing variational inference), and applies to any input distribution, not just Gaussians.

heading: 6.3 CLASSIFICATION EXPERIMENTS ON MNIST, SVHN, CIFAR10
text: We report the results of the image classification experiments in Table 3. We measure performance using accuracy and calibration error of Kuleshov et al. (2018) on the test set. We report these metrics for baseline and calibrated versions of convolutional neural network classifier. We perform recalibration with a simple softmax regression (a multi-class generalization of Platt scaling) and with the neural network recalibrator. The best uncertainties are produced by our method. Recalibrated and base models achieve similar levels of accuracy.

heading: 7 PREVIOUS WORK
text: Probabilistic Forecasting. More modern discussions of probabilistic forecasting can be found in the literature on meteorology (Gneiting and Raftery, 2005). This influential work appears in methods weather forecasting applications systems (Raftery et al., 2005). Most previous work focuses on classification, but recent work (Gneiting et al., 2007; Kuleshov et al., 2018) extends classical methods to regression.
Probabilistic forecasting has been studied extensively in the statistics literature (Murphy, 1973; Dawid, 1984), mainly in the context of evaluation using proper scoring rules (Gneiting and Raftery, 2007). Proper scores measure calibration and sharpness in classification (Murphy, 1973) and regression (Hersbach, 2000).
Calibration. Recalibration is a widely used approach for improving probabilistic forecasts. It takes it roots in the classification setting, where Platt scaling (Platt, 1999) and isotonic regression (Niculescu-Mizil and Caruana, 2005) are two widely used algorithms. The have been extended to multi-class (Zadrozny and Elkan, 2002), structured (Kuleshov and Liang, 2015), and online prediction Kuleshov and Ermon (2017). There is significant recent interest in calibration in deep learning (Guo et al., 2017; Lakshminarayanan et al., 2017; Gal et al., 2017; Kuleshov et al., 2018).

heading: 8 CONCLUSION
text: We take inspiration from the statistics literature and argue that predictive uncertainties should be evaluated by proper scoring rules, which measure two specific qualities of probabilistic predictions: calibration and sharpness. Gneiting et al. (2007) argued that predictive uncertainties should maximize calibration subject to sharpness and used this paradigm to evaluate forecasts. We formalize the paradigm of Gneiting et al. (2007) into a novel learning principle called calibrated risk minimization and propose a general algorithm that meets the requirements of this paradigm. Overall, we show that calibration is a property that can be achieved in predictive models with almost no cost. As such, calibration is a rare free lunch that should be enforced throughout applications of machine learning.

Review summaries:
This paper proposes to re-calibrate the probabilistic predictions of machine learning models by learning a secondary function that maps from their output parameters to a new set of calibrated distribution parameters. The secondary function is learnt on a held out set. The paper argues for the use of flexible recalibration methods, appealing to their asymptotic optimality. Experiments are performed on 8 UCI regression datasets, MNIST, SVHN and CIFAR10.  NNs are used as both primary and recalibration models. Superior results are obtained compared to some recent baselines.

The paper proposes a new method for calibrating the uncertainty estimates of supervised machine learning models, i.e., ensuring that they match the probabilities of the data distribution. After regular model training, a calibrator model is trained to transform the model output, using the same loss but a separate dataset. Theoretical analysis and empirical experiments are conducted to support the claim that this is a way to calibrate "any model" at "almost no cost".

The authors propose an algorithm scheme that they prove provides probabilisitic asymptotic guarantees of `distribution' calibration for models. The algorithm provided by the authors fits a primary model, H, on a training dataset, D, as well as a recalibrator, R, which takes H(x) as an input, using a calibration dataset, C. H is fit using (x,y) pairs from D and a proper loss. R is then fit using (H(x), y) pairs from C and a proper loss. The resulting model ought to be calibrated on C while also being (asymptotically) as sharp as the original model. The authors also introduce the idea of calibrated risk minimization, which is a constrained optimization alternative to empirical risk minimization.