Year: 2021

Abstract:
Reliable probability estimation is of crucial importance in many real-world applications where there is inherent uncertainty, such as weather forecasting, medical prognosis, or collision avoidance in autonomous vehicles. Probability-estimation models are trained on observed outcomes (e.g. whether it has rained or not, or whether a patient has died or not), because the ground-truth probabilities of the events of interest are typically unknown. The problem is therefore analogous to binary classification, with the important difference that the objective is to estimate probabilities rather than predicting the specific outcome. The goal of this work is to investigate probability estimation from high-dimensional data using deep neural networks. There exist several methods to improve the probabilities generated by these models but they mostly focus on classification problems where the probabilities are related to model uncertainty. In the case of problems with inherent uncertainty, it is challenging to evaluate performance without access to ground-truth probabilities. To address this, we build a synthetic dataset to study and compare different computable metrics. We evaluate existing methods on the synthetic data as well as on three real-world probability estimation tasks, all of which involve inherent uncertainty: precipitation forecasting from radar images, predicting cancer patient survival from histopathology images, and predicting car crashes from dashcam videos. Finally, we also propose a new method for probability estimation using neural networks, which modifies the training process to promote output probabilities that are consistent with empirical probabilities computed from the data. The method outperforms existing approaches on most metrics on the simulated as well as real-world data.

heading: 1 INTRODUCTION
text: We consider the problem of building models that answer questions such as: Will it rain? Will a patient survive? Will a car collide with another vehicle? Due to the inherently-uncertain nature of these real-world phenomena, this requires performing probability estimation, i.e. estimating the probability of each possible outcome of the phenomenon of interest. Models for probability prediction must be trained on observed outcomes (e.g. whether it rained, a patient died, or a collision occurred), because the ground-truth probabilities are unknown. The problem is therefore analogous to binary classification, with the important difference that the objective is to estimate probabilities rather than predicting specific outcomes. In probability estimation, two identical inputs (e.g. histopathology images from cancer patients) can potentially result in two different outcomes (death vs. survival). In contrast, in classification the class label is usually completely determined by the data (a picture either shows a cat or it does not).
The goal of this work is to investigate probability estimation from high-dimensional data using deep neural networks. Deep networks trained for classification often generate probabilities, which quantify the uncertainty of the estimate (i.e. how likely the network is to classify correctly). This quantification has been observed to be inaccurate, and several methods have been developed to improve it (Platt, 1999; Guo et al., 2017; Szegedy et al., 2016; Zhang et al., 2020; Thulasidasan et al., 2020; Mukhoti et al., 2020; Thagaard et al., 2020), including Bayesian neural networks (Gal & Ghahramani, 2016; Wang et al., 2016; Shekhovtsov & Flach, 2019; Postels et al., 2019). However, these works restrict their attention almost exclusively to classification in datasets (e.g. CIFAR10/100 Krizhevsky (2009), or ImageNet (Deng et al., 2009)) where the label is not uncertain, and therefore the uncertainty is completely tied to the model: it quantifies the confidence of the model in its own prediction, not the probability of an event of interest.
Probability estimation from high-dimensional data is crucial in medical prognostics (Wulczyn et al., 2020), weather prediction (Agrawal et al., 2019), and autonomous driving (Kim et al., 2019). In order to advance deep-learning methodology for probability estimation it is crucial to build appropriate benchmark datasets. Here we build a synthetic dataset and gather three real-world datasets, which we use to systematically evaluate existing methodology. In addition, we propose a novel approach, which outperforms current state-of-the-art methods. Our contributions are the following:
• We introduce a new synthetic dataset for probability estimation where a population of people may have a certain disease connected to age. The task is to predict the probability that they contract the disease from a picture of their face. The data are generated based on the UTKFaces dataset (Zhang et al., 2017a), which contains age information. The dataset contains multiple versions of the synthetic labels, which are generated according to different distributions designed to mimic realworld probability-prediction datasets. The dataset serves two objectives. First, it allows us to evaluate existing methodology. Second, it enables us to evaluate different metrics in a controlled scenario where we have access to ground-truth probabilities. • We have used publicly available data to build probability-estimation benchmark datasets for three real-world applications: (1) precipitation forecasting from radar images, (2) prediction of cancerpatient survival from histopathology images, and (3) prediction of vehicle collisions from dashcam videos. We use these datasets to systematically evaluate existing approaches, which have been previously tested mainly on classification datasets. • We propose Calibrated Probability Estimation (CaPE), a novel technique which modifies the training process so that output probabilities are consistent with empirical probabilities computed from the data. CaPE outperforms existing approaches on most metrics on synthetic and real-world data.

heading: 2 PROBLEM FORMULATION: PROBABILITY ESTIMATION
text: The goal of probability estimation is to evaluate the likelihood of a certain event of interest, based on observed data. The available training data consists of n examples xi, 1 ≤ i ≤ n, each associated with a corresponding outcome yi. In our applications of interest, the input data are high dimensional: each xi corresponds to an image or a video. The corresponding label yi is either 0 or 1 depending on whether or not the event in question occurred. For example, in the cancer-survival application xi is a histopathology image of a patient, and yi equals 1 if the patient survived for 5 years after xi was collected. The data have inherent uncertainty: yi, the patient’s survival, does not depend deterministically on the histopathology image (due e.g. to comorbidities and other health factors). Instead, we assume that yi equals 1 with a certain probability pi associated with xi,, as illustrated in Figure 1, because the input data provides key information about the patient’s survival chances.
At inference, a probability-estimation model aims to generate an estimate p̂ of the underlying probability p, associated with a new input data point x (e.g. the probability of survival for over 5 years for new patients based on their histopathology data). To summarize, this is not a classification problem, because the labels are not completely predictable. Instead, the goal is to predict the probability of the outcome, which is critical in choosing a course of treatment for the patient.

heading: 3 EVALUATION METRICS
text: Probability estimation shares similar target labels and network outputs with binary classification. However, classification accuracy is not an appropriate metric for evaluating probability-estimation models due to the inherent uncertainty of the outcomes. This is illustrated by the example in Figure 2a where a perfect probability estimate would result in a classification accuracy of just 75%.1
Metrics when ground-truth probabilities are available. For synthetic datasets, we have access to the ground truth probability labels and can use them to evaluate performance. Two reasonable metrics are the mean squared error or `2 distance MSEp, and the Kullback–Leibler divergence KLp between the estimated and ground-truth probabilities:
MSEp = 1
N N∑ i=1 (p̂i − pi)2, and KLp = 1 N N∑ i=1 ( p̂i log ( p̂i pi ) + (1− p̂i) log ( 1− p̂i 1− pi )) . (1)
N is the number of data, and pi, p̂i are the ground-truth and predicted probabilities respectively.
Calibration metrics. In real-world data, ground-truth probabilities are not available. In order to evaluate the probabilities estimated by a model, we need to compare them to the observed probabilities. To this end, we aggregate the examples for which the model output equals a certain value (e.g. 0.5), and verify what fraction of them have outcomes equal to 1. If the fraction is close to the model output, then the model is said to be well calibrated. Definition 3.1. A model f is well calibrated if
P (y = 1 | f(x) ∈ I(q)) = q, ∀ 0 ≤ q ≤ 1, (2) where y is the observed outcome, f(x) is the probability predicted by model f for input x, and I(q) is a small interval around q.
Model calibration can be evaluated using the expected calibration error (ECE) (Guo et al., 2017) (note however that the definition Guo et al. (2017) is specific to classification). Given a probabilityestimation model f and a dataset of input data xi and associated outcomes yi, 1 ≤ i ≤ N , we partition the examples into B bins, I1I2, · · · , IB , according to the probabilities assigned to the examples by the model. Let Q1,. . . , QB−1 the B-quantiles of the set {f(x1), . . . , f(xN )}, we have Ib := [Qb−1, Qb]∩{f(xi)}Ni=1 (setting Q0 = 0). For each bin, we compute the mean predicted and empirical probabilities,
p(b)emp = E (y | f(x) ∈ Ib) = 1 |Ib| ∑
i∈Index(Ib)
yi, (3)
q(b) = 1 |Ib| ∑
i∈Index(Ib)
f(xi), (4)
1A perfect model (in terms of probability estimation), assigns 0.25 to the blue class and 0.75 to the red class. To maximize classification accuracy, we predict when the model outputs 0.75 (red examples) and 0 when it outputs 0.25 (blue examples). However, 25% of red examples have an outcome of 0, and 25% of blue examples have an outcome of 1. As a result, the model would only have 75% accuracy.
where Index(Ib) = {i | f(xi) ∈ Ib}.
The pairs (q(b), p(b)emp) can be plotted as a reliability diagram, shown in the second row of Figure 4 and in Figure 6. ECE is then defined as
ECE = 1
B B∑ b=1 ∣∣∣p(b)emp − q(b)∣∣∣ . (5) Other metrics for calibration include the maximum calibration error (MCE) defined as
MCE = max b=1,...,B ∣∣∣p(b)emp − q(b)∣∣∣ , and the Kolmogorov-Smirnov error (KS-error) (Gupta et al., 2021), a metric based on the cumulative distribution function, which is described in more detail in Appendix B.
Brier score. Crucially, a model without any discriminative power can be perfectly calibrated (see Figure 2b). The Brier score is a metric designed to evaluate both calibration and discriminative ability. It is the mean squared error between the predicted probability and the observed outcomes:
Brier = 1
N N∑ i=1 (p̂i − yi)2. (6)
This score can be decomposed into two terms associated to calibration and discrimination ability, as shown in Appendix C. Using the synthetic data in Section 6.1, where the ground-truth probabilities are known, we show that Brier score is indeed a reliable proxy for gold-standard MSE metric based on ground-truth probabilities MSEp, in contrast to calibration metrics such as ECE, MCE or KSerror, and to classification metrics such as AUC (see Figure 3 and Appendix D).

heading: 4 PROPOSED METHOD: CALIBRATED PROBABILITY ESTIMATION (CAPE)
text: Prediction models based on deep learning are typically trained by minimizing the cross entropy between the model output and the training labels (Goodfellow et al., 2016). This cost function is a proper scoring rule, which means that it evaluates probability estimates in a consistent manner and is therefore guaranteed to be well calibrated in an infinite-data regime (Buja et al., 2005), as illustrated by Figure 4 (first column).
Unfortunately, in practice prediction models are trained on finite data. This is crucial in the case of deep neural networks, because these models are highly overparametrized and therefore prone to overfitting (Goodfellow et al., 2016). In classification, networks have been shown to be capable of fitting arbitrary random labels (Zhang et al., 2017a). In probability estimation, we observe that neural networks indeed eventually overfit the observed outcomes completely. Moreover, the estimated probabilities collapse to 0 or 1 (Figure 4, second column), a phenomenon that has also been reported in classification (Mukhoti et al., 2020). However, calibration is preserved during the first stages of training (Figure 4, third column). This is reminiscent of the early-learning phenomenon observed
for classification from partially corrupted labels (Yao et al., 2020; Xia et al., 2020), where neural networks learn from the correct labels before eventually overfitting the false ones (Liu et al., 2020).
Here, we propose to exploit the training dynamics of cross-entropy minimization through a method that we name Calibrated Probability Estimation (CaPE). Our starting point is a model obtained via early stopping using validation data on the cross-entropy loss. CaPE is designed to further improve the discrimination ability of the model, while ensuring that it remains well calibrated. This is achieved by alternatively minimizing the following two loss functions:
Discrimination loss: Cross entropy between the model output and the observed binary outcomes,
LD = − N∑ i=1 [yi log(f(xi)) + (1− yi) log(1− f(xi))].
Calibration loss: Cross entropy between the output probability of the model and the empirical probability of the outcomes conditioned on the model output:
LC = − N∑ i=1 [ piemp log(f(xi)) + (1− piemp) log(1− f(xi)) ] ,
where piemp is an estimate of the conditional probability P[y = 1|f(x) ∈ I(f(xi))] where I(f(xi)) is a small interval centered at f(xi). As explained in Section 3 if f(xi) is close to this value, then the model is well calibrated. We consider two approaches for estimating piemp. (1) CaPE (bin) where we divide the training set into bins, select the bin bi containing f(xi) and set piemp = p (bi) emp in equation 3. (2) CaPE (kernel) where piemp is estimated through a moving average with a kernel function (see Appendix E for more details). Both methods are efficiently computed by sorting the predictions p̂i. The calibration loss requires a reasonable estimation of the empirical probabilities p (i) emp, which can be obtained from the model after early learning. Therefore using the calibration loss from the beginning is counterproductive, as demonstrated in Section J.
Algorithm 1 Pseudocode for CaPE Require: f . early stopped model Require: m . freq. of training with LC Require: {xi, yi}Ni=1 . training set Require: K(p, q) := exp [ − (p− q)2 /σ2 ] . Gaussian kernel
for t = 1 to num epochs do if t mod m = 0 then p̂i ← f(xi), ∀i Update piemp, ∀i, with BIN or KERNEL L ← LC . compute discrimination loss
else L ← LD . compute calibration loss end if f ← backprop with L . train with loss
end for
function BIN(B) . B-number of bins I1, · · · IB ← partitions by quantile of {p̂j}Nj=1 Find b such that p̂i ∈ Ib Index(Ib)← {j|p̂j ∈ Ib} . get indices in bin b piemp ← 1|Ib| ∑ i∈Index(Ib) yi . empirical mean of bin b end function
function KERNEL(r,K) . r-window size; kernel Nr(i)← r-nearest neighbor of p̂i (output probability space) Z ← ∑ p̂j∈Nr(i) K (p̂i, p̂j) . normalization factor
piemp ← ∑
p̂j∈Nr(i) K (p̂i, p̂j) yj/Z . kernel smooth
end function
CaPE is summarized in Algorithm 1. Figures 4 and 5 show that incorporating the calibration-loss minimization step indeed preserves calibration as training proceeds (this is not necessarily expected because CaPE minimizes a calibration loss on the training data), and prevents the model from overfitting the observed outputs. This is beneficial also for the discriminative ability of the model, because it enables it to further reduce the cross-entropy loss without overfitting, as shown in Figure 5. The experiments with synthetic and real-world data reported in Section 6 suggest that this approach results in accurate probability estimates across a variety of realistic scenarios.

heading: 5 RELATED WORK
text: Neural networks trained for classification often generate a probability associated with their prediction which quantifies its uncertainty. These estimates are often found to be inaccurate (Mukhoti et al., 2020; Guo et al., 2017). Techniques mitigating this issue are often described as calibration methods, and broadly fall into three categories depending on whether they: (1) postprocess the outputs of a trained model, (2) combine multiple model outputs, or (3) modify the training process.
Post-processing methods transform the output probabilities in order to improve calibration on heldout data (Zadrozny & Elkan, 2001; Gupta et al., 2021; Kull et al., 2017; 2019). For example, Platt scaling (Platt, 1999) fits a logistic function that minimizes the negative log-likelihood loss. Temperature scaling (Guo et al., 2017) does the same with a temperature parameter augmenting the softmax function. In contrast to these methods, CaPE enforces calibration during training, which has the advantage of enabling further improvements in the discriminative abilities of the model.
Ensembling methods combine multiple models to improve generalization. Mix-n-Match (Zhang et al., 2020) uses a single model, and ensembles predictions using multiple temperature scaling transformations. Other methods (Lakshminarayanan et al., 2017; Maddox et al., 2019) ensemble
multiple models obtained using different initializations. These approaches are compatible with the proposed method CaPE; how to combine them effectively is an interesting future research direction.
Modified training methods can be divided into two groups. The first group smooths the target 0/1 labels in order to prevent output estimates from collapsing to 0/1 (Mukhoti et al., 2020; Szegedy et al., 2016; Zhang et al., 2018; Thulasidasan et al., 2020). The second group, attaches additional calibration penalties to a cross entropy loss (Kumar et al., 2018; Pereyra et al., 2017; Liang et al., 2020). CaPE is most similar in spirit to the latter methods, although its data-driven calibration loss is different to the penalties used in these techniques.
Datasets for evaluation The methods discussed in this section were developed for calibration in classification, and tested on datasets such as CIFAR-10/100 (Krizhevsky, 2009), SVHN (Netzer et al., 2011), and ImageNet (Deng et al., 2009) where the relationship between labels and input data is completely deterministic. Here, we evaluate these methods on synthetic and real-world probability-estimation problems with inherent uncertainty.

heading: 6 EXPERIMENTS

heading: 6.1 SYNTHETIC DATASET: FACE-BASED RISK PREDICTION
text: To benchmark probability-estimation methods, we build a synthetic dataset based on UTKFace (Zhang et al., 2017b), containing face images and associated ages. We use the age of the ith person zi to assign them a risk of contracting a disease pi = ψ(zi) for a fixed function ψ : N→ [0, 1]. Then we simulate whether the person actually contracts the illness (label yi = 1) or not (yi = 0) with probability pi. The probability-estimation task is to estimate the ground-truth probability pi from the face image xi, which requires learning to discriminate age and map it to the corresponding risk. We design ψ to create five scenarios, inspired by real-world data (see Appendix G)):
• Linear: Equally-spaced, inspired by weather forecasting: ψ(z) = z/100 • Sigmoid: Concentrated near two extremes: ψ(z) = σ(25(z/100− 0.29)) • Skewed: Clustered close to zero, inspired by vehicle-collision detection: ψ(z) = z/250 • Centered: Clustered in the center, inspired by cancer-survival prediction: ψ(z) = z/300 + 0.35 • Discrete: Discretized: ψ(z) = 0.2 [ 1{z>20} + 1{z>40} + 1{z>60} + 1{z>80} ] + 0.1

heading: 6.2 REAL-WORLD DATASETS
text: We propose to use three open-source, real-world datasets to benchmark probability-estimation approaches (see Appendix H for further details on the datasets and experiments).
Survival of Cancer Patients. Histopathology aims to identify tumor cells, cancer subtypes, and the stage and level of differentiation of cancer. Hematoxylin and Eosin (H&E)-stained slides are the most common type of histopathology data used for clinical decision making. In particular, they can be used for survival prediction (Wulczyn et al., 2020), which is critical in evaluating the prognosis of patients. Treatments assigned to patients after diagnosis are not personalized and their impact on cancer trajectory is complex, so the survival status of a patient is not deterministic. In this work, we use the H&E slides of non-small cell lung cancers from The Cancer Genome Atlas Program (TCGA)2 to estimate the the 5-year survival probability of cancer patients. The outcome distribution is similar to the Centered scenario in our synthetic data.
Weather Forecasting. The atmosphere is governed by nonlinear dynamics, hence weather forecast models possess inherent uncertainties (Richardson, 2007). Nowcasting, weather prediction in the near future, is of great operational significance, especially with increasing number of extreme inclement weather conditions (Agrawal et al., 2019; Ravuri et al., 2021). We use the German Weather service dataset3, which contains quality-controlled rainfall-depth composites from 17 operational Doppler radars. We use 30 minutes of precipitation data to predict if the mean precipitation will increase or decrease after one hour. Three precipitation maps from the past 30 minutes serve as an input. The outcome distribution is similar to the Linear scenario in our synthetic data.
Collision Prediction. Vehicle collision is one of the leading causes of death in the world. Reliable collision prediction systems are therefore instrumental in saving human lives. These systems predict potential collisions from dashcam cameras. Collisions are influenced by many unknown factors, and hence are not deterministic. Following Kim et al. (2019), we use 0.3 seconds of real dashcam videos from YouTubeCrash dataset as input, and predict the probability of a collision in the next 2 seconds. The data are very imbalanced as the number of collisions is very low, so the outcome distribution is similar to the Skewed scenario in our synthetic data.

heading: 6.3 BASELINES
text: We apply existing calibration methods developed for classification to probability estimation (as well as cross-entropy minimization with early-stopping): (1) Three post-processing methods: Temperature Scaling (Guo et al., 2017), Platt Scaling (Platt, 1999), and Dirichlet Calibration (Kull et al., 2019) applied to the best CE model, (2) Two Ensemble Methods: Mix-n-Match (Zhang et al., 2020) applied to best CE model, and Deep Ensemble (Lakshminarayanan et al., 2017) with 5 networks, and (3) Three Modified Training methods: focal loss (Mukhoti et al., 2020), entropy-maximizing loss (Pereyra et al., 2017), and MMCE regularization (Kumar et al., 2018). Appendix F provides a detailed description. For our experiments on synthetic data, we also compare against a model trained on an infinite amount of data by repeatedly sampling new outcomes from the ground-truth probabilities at each epoch. This provides a best-case reference for each scenario.
2https://www.cancer.gov/tcga 3https://opendata.dwd.de/weather/radar/

heading: 7 RESULTS AND DISCUSSION
text: Table 1 shows that calibration methods developed for classification can be effective for probability estimation. However, the performance of some methods is not consistent across all scenarios. For instance, regularization with negative entropy, which penalizes very high/low confidence, performs worse than CE when the ground-truth probability is close to 0 or 1. In contrast, methods that do not make strong assumptions tend to generalize better to multiple scenarios (e.g. Platt scaling consistently beats CE). The proposed method CaPE outperforms other techniques in most scenarios, and even matches the performance of the infinite data baseline for the Sigmoid scenario. Finally, we observe that the Skewed scenario is very challenging: most methods barely improve the CE baseline.
Table 2 compares the baseline methods and CaPE on the three real-world datasets. We present AUC, ECE for 15 equally-sized bins, and Brier score, as complementary metrics since the underlying ground-truth probabilities are unobserved. As discussed in Section 3, Brier score is the metric that best captures the quality of probability estimates. CaPE has the lowest Brier score in all three datasets, while also achieving lower ECE values and higher AUC values than the other methods. This demonstrates that enforcing calibration during training also yields a more discriminative model. The reliability diagrams in Figure 6 depict the probability estimates produced by CE, CaPE and the best baseline method on the three datasets, demonstrating that CaPE produces a well-calibrated outputs.
Figure 6 also shows that each real-world dataset closely aligns with a particular synthetic scenario: cancer survival with Centered; weather forecasting with Linear; collision prediction with Skewed. This supports the significance of our synthetic benchmark dataset, and provides insights in the differences among baseline models. For example, model averaging with deep ensemble performs well on weather forecasting but has higher Brier scores than Platt scaling on the other two datasets (see Appendix I for further analysis based on pathological stages). Accordingly, deep ensemble also underperforms in the synthetic scenarios where ground-truth probabilities are clustered closely (Sigmoid, Linear), but is effective for Linear. Finally, as in the synthetic Skewed scenario, all methods had similar performance on the collision prediction task. This highlights the importance of considering different scenarios when evaluating methodology for probability estimation.

heading: 8 CONCLUSION
text: In this work we evaluate existing approaches to improve the output probabilities of neural networks on probability-estimation problems. To this end, we introduce a new synthetic benchmark dataset designed to reproduce several realistic scenarios, and also gather three real-world datasets relevant to medicine, climatology, and self-driving cars. In addition, we propose a novel approach for probability-estimation via deep learning that outperforms existing approaches for most datasets. An important application for probability estimation is in the context of survival analysis, which can be recast as estimation of conditional probabilities (Lee et al., 2018; Shamout et al., 2020; Goldstein et al., 2021). An interesting research direction is to consider problems with several possible uncertain outcomes (analogous to multiclass classification).

heading: A ADDITIONAL RESULTS
text: We present here supplementary results to the ones presented in Section 7.
A.1 FACE-BASED RISK PREDICTION
Full evaluation with confidence intervals derived using 1000 bootstraps for the five simulated scenarios are examined: Linear (Table.3); Sigmoid (Table.4); Centered (Table.5); Skewed (Table.6); Discrete (Table.7). Note that all numbers are downscaled by 10−2 in the tables.
A.2 SUPPLEMENTARY METRICS ON REAL-WORLD DATASET
We present here additional metrics on the real world data: Cancer Survival (Table 8); Climate Forecasting (Table 9); Collision Prediction (Table 10).
A.3 ADDITIONAL RELIABILITY DIAGRAM
We present here additional reliability curves to the ones illustrated in Figure 6
Cancer Survival Weather Forecasting Collision Prediction Temperature Entropy Reg.
Platt Scaling MMCE
Dirichlet Cal. Deep Ensemble
Mix-n-match CaPE(bin)
Temperature Entropy Reg.
Platt Scaling MMCE
Dirichlet Cal. Deep Ensemble
Mix-n-match CaPE(bin)
Temperature Entropy Reg.
Platt Scaling MMCE
Dirichlet Cal. Deep Ensemble
Mix-n-match CaPE(bin)
1.0
0.8
0.6
0.4
0.2
0.0
1.0
0.8
0.6
0.4
0.2
0.0
1.0
0.8
0.6
0.4
0.2
0.0
1.0
0.8
0.6
0.4
0.2
0.0
1.0
0.8
0.6
0.4
0.2
0.0
1.0
0.8
0.6
0.4
0.2
0.0
1.0
0.8
0.6
0.4
0.2
0.0
1.0
0.8
0.6
0.4
0.2
0.0
1.0
0.8
0.6
0.4
0.2
0.0
1.0
0.8
0.6
0.4
0.2
0.0
1.0
0.8
0.6
0.4
0.2
0.0
1.0
0.8
0.6
0.4
0.2
0.0
0.0 0.2 0.4 0.6 0.8 1.00.0 0.2 0.4 0.6 0.8 1.0
0.0 0.2 0.4 0.6 0.8 1.00.0 0.2 0.4 0.6 0.8 1.0
0.0 0.2 0.4 0.6 0.8 1.00.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.00.0 0.2 0.4 0.6 0.8 1.0
0.0 0.2 0.4 0.6 0.8 1.00.0 0.2 0.4 0.6 0.8 1.0
0.0 0.2 0.4 0.6 0.8 1.00.0 0.2 0.4 0.6 0.8 1.0
1.0
0.8
0.6
0.4
0.2
0.0
1.0
0.8
0.6
0.4
0.2
0.0 0.0 0.2 0.4 0.6 0.8 1.00.0 0.2 0.4 0.6 0.8 1.0
1.0
0.8
0.6
0.4
0.2
0.0
1.0
0.8
0.6
0.4
0.2
0.0 0.0 0.2 0.4 0.6 0.8 1.00.0 0.2 0.4 0.6 0.8 1.0
1.0
0.8
0.6
0.4
0.2
0.0
1.0
0.8
0.6
0.4
0.2
0.0 0.0 0.2 0.4 0.6 0.8 1.00.0 0.2 0.4 0.6 0.8 1.0
We present here additional reliability curves for the different synthetic data scenarios, mentioned in Figure 4.

heading: B KOLMOGOROV-SMIRNOV ERROR
text: We derive the KS-error, mentioned in Section 3.
For a calibrated estimator P[y = 1|f(x) ∈ I(q)] = q, ∀0 ≤ q ≤ 1,
for some small interval I(q) around q.
Hence P[y = 1, f(x) ∈ I(q)] = P[f(x) ∈ I(q)]q, ∀0 ≤ q ≤ 1
. Similarly to the Kolmogorov-Smirnov (KS) test for distribution functions, we can recast this property in integral form
φ1(σ) = σ∫ 0 P[y = 1, f(x) ∈ I(q)]dq, φ2(σ) = σ∫ 0 P[f(x) ∈ I(q)]qdq
We can evaluate φ1, φ2 from a finite sample (xi, yi), i = 1 . . . n,
φ1(σ) = 1
n n∑ i=1 1(yi = 1, f(xi) ≤ σ), φ2(σ) = 1 n n∑ i=1 1(f(xi) ≤ σ)f(xi)
The KS error is defined as KS = max
1≤σ≤1 |φ1(σ)− φ2(σ)|
φ1, φ2 can be efficiently computed by sorting the data points with respect to their confidence scores f(xi). The KS error has the advantage of being independent of binning configurations, unlike ECE and MCE.

heading: C BRIER SCORE DECOMPOSITION
text: We present here a decomposition of the Brier score into two components, discussed in Section 3.
The Brier score can be interpreted as a sum of two terms, calibration and refinement. Assume the network can output one of K distinct possible predictions, i.e., p̂ ∈ {q̂1, . . . , q̂K}. Denote Sk, the set of all inputs with output pk and q̄k the empirical probability over Sk, i.e.,
Sk = {x|f(x) = q̂k}, |Sk| = nk, q̄k = 1
nk ∑ xi∈Sk yi
Then we can write
Brier = 1
N N∑ i=1 (p̂i − yi)2 = 1 N K∑ k=1 nk(q̂k − q̄k)2 + 1 N K∑ k=1 nkq̄k(1− q̄k),
The first term on the RHS, calibration, is similar to MSEp, with the empirical probabilities q̄k substituting for the true labels. The second term, refinement, is an estimate of the confidence in determining q̄k. It is related to the area under curve (AUC), which measures to the achievable accuracy of the network as a classifier. The term is smaller as the prediction classes fi tend towards 0 or 1. Thus, this term penalizes empirically calibrated predictors, with low discriminative power, as in Figure 2b.
Linear
0.0 0.5 1.0 Predicted probability
0.0
0.5 1.0 Em pi ric al p ro ba
bi lit y pi
pi
Infinite Data
0.0 0.5 1.0 Predicted probability
0.0
0.5
1.0 pi
pi
Overfitting
0.0 0.5 1.0 Predicted probability
0.0
0.5
1.0 pi
pi
Early Stopping
0.0 0.5 1.0 Predicted probability
0.0
0.5
1.0 pi
pi
CaPE
Sigmoid
0.0 0.5 1.0 Predicted probability
0.0
0.5
1.0
Em pi
ric al
p ro
ba bi
lit y
pi
pi
Infinite Data
0.0 0.5 1.0 Predicted probability
0.0
0.5
1.0 pi
pi
Overfitting
0.0 0.5 1.0 Predicted probability
0.0
0.5
1.0 pi
pi
Early Stopping
0.0 0.5 1.0 Predicted probability
0.0
0.5
1.0 pi
pi
CaPE
Centered
0.0 0.5 1.0 Predicted probability
0.0
0.5
1.0
Em pi
ric al
p ro
ba bi
lit y
pi
pi
Infinite Data
0.0 0.5 1.0 Predicted probability
0.0
0.5
1.0 pi
pi
Overfitting
0.0 0.5 1.0 Predicted probability
0.0
0.5
1.0 pi
pi
Early Stopping
0.0 0.5 1.0 Predicted probability
0.0
0.5
1.0 pi
pi
CaPE
Skewed
0.0 0.5 1.0 Predicted probability
0.0
0.5
1.0
Em pi
ric al
p ro
ba bi
lit y
pi
pi
Infinite Data
0.0 0.5 1.0 Predicted probability
0.0
0.5
1.0 pi
pi
Overfitting
0.0 0.5 1.0 Predicted probability
0.0
0.5
1.0 pi
pi
Early Stopping
0.0 0.5 1.0 Predicted probability
0.0
0.5
1.0 pi
pi
CaPE
Discrete
Linear

heading: D METRIC COMPARISON
text: We present here the correlation between different calibration and accuracy metrics and metrics that have access to the ground truth probabilities, MSEp and KL-divergence, eveluated over all five scenarios in our Face-based Risk Prediction synthetic dataset, referred to in Section 3.

heading: E ESTIMATION OF EMPIRICAL PROBABILITY IN CAPE
text: We describe in further detail the two ways to estimate the conditional probability P[y = 1|f(x) ∈ I(q)], introduced in Section 4.
Linear
We wish to estimate the conditional probability of an output y given a network prediction f(x), P[y = 1|f(x) ∈ I(q)] We can approximate the probability by averaging over points p̂ ∈ I(q),
P[y = 1|f(x) ∈ I(q)] ≈ 1 |I(q)| ∑ p̂∈I(q) P[y = 1|f(x) = p] (7)
An empirical estimate of P[y = 1|f(x) ∈ I(q)] would be
P[y = 1|f(x) ∈ I(q)] ≈ 1 |Index(I(q))| ∑ f(xi)∈I(q) yi, (8)
where Index(Iq) = {i|f(xi) ∈ I(q)}.
Alternatively, we can use kernel estimation:
P[y = 1|f(x) ∈ I(q)] ≈ 1 Z ∑ p̂∈I(q) P[y = 1|f(x) = p̂] · exp
( − (p− q) 2
σ2
) , (9)
where Z = ∑ p∈I(q) exp ( − (p−q) 2 σ2 ) is the normalization factor. An empirical estimate of the conditional probability would then be
P[y = 1|f(x) ∈ I(q)] ≈ 1 Z ∑ f(xi)∈I(q) yi exp
( − (f(xi)− q) 2
σ2
) . (10)
Based on these two approximation methods, we can design an algorithm to estimate piemp.
Bin We divide our data into B bins of equal size. Q1, . . . , QB are the data B-quantiles. We wish to estimate P[y = 1|f(x) ∈ [Qb−1, Qb]], b = 1, . . . , B,Q0 = 0. Denote Ib := [Qb−1, Qb] ∩ {f(xi)}Ni=1, set of all predictions in [Qb−1, Qb], and Index(Ib) = {i|f(xi) ∈ Ib}. We have,
P[y = 1|f(x) ∈ [Qb−1, Qb]] ≈ p(b)emp = 1 |Ib| ∑
i∈Index(Ib)
yi
We assign p(b)emp to all data points i in the b-th quantile
piemp = p (b) emp ∀i ∈ Index(Ib)
Kernel In this case we use kernel estimation:
piemp = ∑ k∈NN(i,r)K (i, k) yk∑ k∈NN(i,r)K (i, k) . (11)
NN(i, r) defines r data points whose predictions are nearest to p̂i = f(xi). K(i, j) is the Gaussian kernel
K(i, j) = exp ( − (p̂i − p̂j) 2
σ2
) ,
with hyperparameter σ.

heading: F CALIBRATION BASELINES
text: This section includes a review of the baseline methods, discussed in Section 6
Post-processing Postprocessing for calibration requires finding a function f : [0, 1] → [0, 1], that augments the output of a the neural network p̂i → f(p̂i) in order to achieve better calibration properties
• Platt scaling (Platt, 1999) optimizes f on validation set within the following family, f1(p̂i) = σ ( WT p̂i + b ) (12)
where W ∈ R2, b ∈ R and σ is the Sigmoid function. The non-probabilistic predictions of a classifier are used as features for a logistic regression model, which is trained on the validation set to return probabilities. • Temperature scaling (Guo et al., 2017) is a single parameter variant of Platt Scaling where we only change the temperature of the softmax to obtain the calibrated probabilities.
f(p̂i) = Softmax (p̂i/T ) (13) where T ∈ R minimizes the negative log-likelihood of validation set. • Beta/Dirchlet calibration (Dir-ODIR) (Kull et al., 2017; 2019) assumes that the probabilities can be parametrized by a Beta/Dirchlet distribution i.e.
fj ∼ Beta(α(j), β(j)) (14) Assume the prior to be p(y = j) = πj , πj ∈ [0, 1], we have P (y|fj) ∝ πjfj , and then α(j), β(j) are estimated by maximizing the posterior.
Ensembling These calibration methods simultaneously train several neural networks from end to end, varying parameters in the training process. The final output is some function of all the different outputs.
• Mix-n-Match (Zhang et al., 2020) improves calibration by ensembling parametric and nonparametric calibrators. Denote the temperature scaling function with g(ŷi, T ). Then Mixn-Match ensembles different temperatures
fj(p̂i) = w1gj(p̂i, T ) + w2gj(p̂i, 0) + w3gj(p̂i,∞) (15) After ensembling the parametric temperature scaling, Mix-n-Match applies non-parametric isotonic regression. • Deep ensemble (Lakshminarayanan et al., 2017) trains M copies of the neural network with different initialization. The probability estimation is the average of all single model estimations
p(yi | xi) = 1
M M∑ j pθj (yi | xi) (16)
Modified training These calibration methods train the neural networks from end to end, modifying the training process to improve calibration.
• Confidence penalty (Pereyra et al., 2017) Penalizeslow entropy output distributions (confidence penalty). Label smoothing improve state-of-the-art models across benchmarks.
L(θ) = − ∑ i log pθ(yi|xi)− βH(pθ(yi|xi)) (17)
• Focal loss (Mukhoti et al., 2020) maximizes entropy while minimizing the KL divergence between the predicted and the target distributions. It also regularizes the weights of the model to avoid overfitting.
L(θ) = − ∑ i (1− pθ(yi|xi))γ log pθ(yi|xi), γ ∈ R. (18)
• Kernel MMCE (Kumar et al., 2018) is a reproducing kernel Hilbert space (RKHS) kernel based measure of calibration that is efficiently trainable, alongside the negative likelihood loss. Given data samples D = {(ci, ri)}mi=0, where ci = χ{ŷi=yi} and ri = P(ci = 1|ŷi), MMCE is computed on samples D as following,
MMCE2(D) = ∑ i,j (ci − ri)(cj − rj)k(ri, rj) m2
(19)
where k(ri, rj) is a kernel function. MMCE is optimized together with the cross entropy loss as a regularization term. The strength of calibration can be adjusted by a scale λ ∈ R.
L(θ) = − ∑ i log pθ(yi|xi) + λ ( MMCE2(D) ) 1 2 (20)

heading: G SYNTHETIC DATA EXPERIMENTS
text: We use ResNet-18 model for all our experiments with synthetic data.
The synthetic data is split into training, validation, and test sets with 16641, 4738, and 2329 samples, respectively. The training and validation sets contain only images xi and 0-1 labels yi for training and tuning the model. In order to evaluate the performance of the model for probability estimation, the held-out test set contains the ground truth probabilities pi, in addition to xi and yi. Note that we do not use the ground-truth probability labels pi values during training or inference - we only use them to compare the performance of different models.
Ground Truth Probability Generation The ground truth probability associated with example i is simulated by pi = ψ(zi) where zi is age of the person.
Label distribution After determining the probability pi using ψ(z), the label yi is sampled from a Bernoulli distribution parametrized by pi, so that it takes the value 1 with probability pi. The distributions of yi under five different scenarios are illustrated in Fig.11.

heading: H REAL-WORLD DATA AND EXPERIMENT DETAILS
text: We present here supplementary information for the real-world datasets used in our experiments.
Cancer Survival Histopathological features are useful in identification of tumor cells, cancer subtypes, and the stage and level of differentiation of the cancer. Hematoxylin and Eosin (H&E)-stained slides are the most common type of histopathology data and the basis for decision making in the clinics. With these properties, H&E are used for mortality prediction of cancer (Wulczyn et al., 2020). In this experiment, we use the H&E slides of non-small cell lung cancers from The Cancer Genome Atlas Program (TCGA)4 to predict the 5-year survival. The dataset has 1512 whole slide images from 1009 patients, and 352 of them died in 5-years. We split the samples by patients and source institutions into training, validation, and test set, which has 1203, 151, and 158 samples respectively.
The whole slide images contain numerous pixels, so we cropped the slides into tiles at 20x magnification with 1/4 overlapping, resized them to 299 × 299 with bicubic interpolation, and filtered out the tiles with more than 85% area covered by the background. The representations of each tile are trained with self-supervised momentum contrastive learning (MoCo) (Chen et al., 2020), and the slide-level prediction is obtained from a multiple-instance learning network (Ilse et al., 2018) trained with the binary label of survival in 5 years.
Weather Forecasting We use the German Weather service dataset5, which contains qualitycontrolled rainfall-depth composites from 17 operational Doppler radars. Three precipitation maps from the past 30 minutes serve as an input. The training labels are the 0/1 events indicating whether the mean precipitation increases (1) or not (0).
The German Weather service (DWD - Deutshce Wetter Dienst) dataset https://opendata. dwd.de/weather/radar/ contains quality-controlled rainfall-depth composites from 17 operational DWD Doppler radars. It has a spatial extent of 900x900 km, and covers the entirety of Germany. Data exists since 2006, with a spatial and temporal resolution of 1x1 km and 5 minutes, respectively. The dataset has been used to train RainNet, a pricipitation nowcasting model (Ayzel, 2020).
The network architecture is ResNet18, with 3 input channels and 2 output channels. The input to the network are 3 precipitation maps which cover a fixed area of 300km×300 km in the center of the grid (300× 300 pixels), set 10 minutes apart. The training, validation and test datasets consist of 20000, 6000 and 3000 samples, respectively, all separated temporally, over the span of 15 years.
Collision Prediction Vehicle collision is one of the leading causes of death in the world. Reliable collision prediction systems which can warn drivers about potential collisions can save a significant number of lives. A standard way to design such a system is to train a convolutional model for identifying if a particular vehicle in the dash-cam video feed might collide with the car in next few seconds. More formally, at time t = T the system tries to predict if any car in the video might collide with our given car in time t ∈ [T, T + Tlook-ahead]. Each labelled training sample consists of features X = (XT−δ, XT−2δ, . . . , XT−dδ) and a binary label Y ∈ {0, 1} denoting if an accident will occur in t ∈ [T, T + Tlook-ahead]. Each Xt is a tensor with 4 channels where the first 3 channels corresponds to an RGB image of the dashcam view at time t = t, and the fourth channel consists of a mask with a bounding box on a particular vehicle of interest. In this work, we use YouTubeCrash dataset (Kim et al., 2019) to train and test our model, which uses δ = 0.1s,Tlook-head = 18δ = 1.8s, and d = 3. Following Kim et al. (2019) we used a VGG-16 network architecture.
The dataset contains 122 accident scenes, and 2096 non-accident scenes, which after feature extraction gives us 2096 positive samples, and 11486 negative samples (the dataset is severely imbalanced, and similar to the Skewed situation in Section 6.1). We further split the dataset into train (6453 samples for label 0, and 1023 samples for label 1), validation (2348 samples for label 0, and 545 samples for label 1), and test (2685 samples for label 0, and 528 samples for label 1) sets. The samples in train, validation and test sets are generated from disjoint scenes/dashcam videos.
4https://www.cancer.gov/tcga 5https://opendata.dwd.de/weather/radar/

heading: I ANALYSIS OF CANCER SURVIVAL RESULTS
text: For cancer survival prediction, we visualize the estimated probabilities on the test set in different pathological stages in Figure 12. In general, patients in earlier stages should have higher probabilities of survival. Deep ensemble produces similar probability estimates for all stages (i.e the model is less discriminative). Cross-entropy minimization (CE) is more discriminative, but has very wide confidence intervals. CaPE is more discriminative than deep ensemble, while having narrower confidence intervals than CE.

heading: J CALIBRATING FROM THE BEGINNING
text: CaPE exploits a calibration-based cost function to improve its probability estimates without overfitting. The empirical probabilities in this loss are computed from the model itself. Consequently, applying this strategy from the beginning of training can be counterproductive, because the model predictions are essentially random. This is demonstrated in the following table, which compares CaPE with a model trained using the calibration loss from the beginning (in the same way as CaPE, alternating with cross-entropy minimization).

heading: K COMPARISON OF REAL-WORLD DATASETS WITH DIFFERENT SCENARIOS
text: OF THE SIMULATED DATASET
Figure 13 illustrates the similarity between the empirical probability curves of different real-world datasets and the different scenarios of our synthetic dataset. For the cancer survival dataset, the empirical probabilities are clustered in the center (0.4-0.6) similar to the Centered scenario. For the weather forecasting dataset, the probabilities are uniformly distributed across 0.1-0.8 similar to the Linear scenario. For the collision prediction dataset, the majority of the data points are clustered in the lower probability region which makes it similar to the Skewed scenario.

Review summaries:
This work tackles the problem of probability estimation. Current machine learning models do not fully reflect the uncertainty of the outcome but only of the model. The authors propose a loss that enforces both calibration and discrimination. Additionally, they present a semi-synthetic dataset for further study of this problem: from an image dataset of faces with associated age, they estimate the risk of developing a given disease.

The authors propose a method to ensure calibration of probability outputs during training. They do so by adding an explicit penalty to push the output probability values towards an empirical estimate of the actual probability for the current inferred probability. They compare the method to a number of baseline approaches, showing the benefits on both synthetic and real-world data sets.

The paper studies the problem of uncertainty quantification in deep neural nets. It introduces a concept called "probability estimation" and an uncertainty calibration method called "CaPE" based on this new concept. The studies uncertainty calibration in a few new data sets such a histopathology cancer diagnostics.