Deep Probability Estimation
Authors: Weicheng Zhu
Matan Leibovich
Sheng Liu
Sreyas Mohan
Aakash Kaku
Boyang Yu
Laure Zanna
Narges Razavian
Carlos Fernandez-Granda
Venue: ICLR 2022 Submitted (2022)
Keywords: Probability estimation
calibration
uncertainty
weather forecasting
medical prognosis
car crash
benchmark datasets
deep learning
high dimensional data

Abstract: Reliable probability estimation is of crucial importance in many real-world applications where there is inherent uncertainty, such as weather forecasting, medical prognosis, or collision avoidance in autonomous vehicles. Probability-estimation models are trained on observed outcomes (e.g. whether it has rained or not, or whether a patient has died or not), because the ground-truth probabilities of the events of interest are typically unknown. The problem is therefore analogous to binary classification, with the important difference that the objective is to estimate probabilities rather than predicting the specific outcome. The goal of this work is to investigate probability estimation from high-dimensional data using deep neural networks. There exist several methods to improve the probabilities generated by these models but they mostly focus on classification problems where the probabilities are related to model uncertainty. In the case of problems with inherent uncertainty, it is challenging to evaluate performance without access to ground-truth probabilities. To address this, we build a synthetic dataset to study and compare different computable metrics. We evaluate existing methods on the synthetic data as well as on three real-world probability estimation tasks, all of which involve inherent uncertainty: precipitation forecasting from radar images, predicting cancer patient survival from histopathology images, and predicting car crashes from dashcam videos. Finally, we also propose a new method for probability estimation using neural networks, which modifies the training process to promote output probabilities that are consistent with empirical probabilities computed from the data. The method outperforms existing approaches on most metrics on the simulated as well as real-world data.

Under review as a conference paper at ICLR 2022
DEEP PROBABILITY ESTIMATION
Anonymous authors
Paper under double-blind review
ABSTRACT
Reliable probability estimation is of crucial importance in many real- world appli- cations where there is inherent uncertainty, such as weather forecasting, medical prognosis, or collision avoidance in autonomous vehicles. Probability-estimatin models are trained on observed outcomes (e.g. whether it has rained not, whether a patient has died or not), because the ground-truth probabilities of the events of interest typically unknown. The problem is therefore analogous binary classification, with the important difference that the objective is to esti- mate probabilities rather than predicting the specific outcome. The goal of this work is to investigate probability estimation from high-dimensional data using deep neural networks. There exist several methods improve the probabilities generated by these models but they mostly focus on classification problems where the probabilities are related to model uncertainty. In the case of problems with inherent uncertainty, it is challenging to evaluate performance without access to ground probabilities. To address this, we build a synthetic dataset to study and compare different computable metrics. We evaluate existing methods the synthetic data as well as on three real-world probability estimation tasks, all of which involve inherent uncertainty: precipitation forecasting from radar images, predicting cancer patient survival from histopathology images, and predicting car crashes from dashcam videos. Finally, we also propose new method for prob- ability estimation using neural networks, which modifies the training process to promote output probabilities that are consistent with empirical probabilities com- puted from the data. The method outperforms existing approaches on most metrics on the simulated as well as real-world data.
INTRODUCTION
We consider the problem of building models that answer questions such as: Will it rain? Will patient survive? Will car collide with another vehicle? Due to the inherently of these real-world phenomena, this requires performing probability estimation, i.e. estimating the probability of each possible outcome of the phenomenon of interest. Models for probability pre- diction must be trained on observed outcomes (e.g. whether it rained, patient died, or collision occurred), because the ground-truth probabilities are unknown. The problem is therefore analogous to binary classification, with the important difference that the objective is to estimate probabili- ties rather than predicting specific outcomes. In probability estimation, two identical inputs (e.g. histopathology images from cancer patients) can potentially result in two different outcomes (death vs. survival). In contrast, in classification the class label is usually completely determined by the data (a picture either shows cat it does not).
nature
The goal of this work is to investigate probability estimation from high-dimensional data using deep neural networks. Deep networks trained for classification often generate probabilities, which quantify the uncertainty of the estimate (i.c. how likely the network is to classify correctly). This quantification has been observed to be inaccurate, and several methods have been developed to improve it (Plati, 1999 Guo et al. 2017;; (Szeedyy et al. 2016; Zhang et al.) 2020 Thulasidasan ct al. 2020 Mukhoti et al. 2020 Thagaard al. 2020), including Bayesian neural networks (Gal & Ghahramani 2016 Wang et al. 2016 Shekhovlsov & Flach 2019 Postels et al. 2019). However, these works restrict their attention almost exclusively to classification in datasets (e.g. CIFAR- 10/100 Krizhevsky p2009, or ImageNet (Dnng et al. 2009)) where the label is not uncertain, and therefore the uncertaityy is completely tied to the model: it quantifies the confidence of the model in its own prediction, not the probability of an event of interest.

---

Under review as a conference paper at ICLR 2022
Training
Inference
+ Trained f
P(survival)=?
pi
P(survival)=:0%%
7/4
Death
Figure I: The probability problem. In probability estimation, we assume that each ob- served outcome (e.g. death or survival in cancer patients) in the training set is randomly generated from a latent unobserved probability p; associated the corresponding data æ; (e.g. histopathology images). Training (left): Only æ; and yi can be used for training, because ps is not observed. In- ference (right): Given new data æ, the trained network produces probability estimate ý E (0, 1].
Probability estimation from high high-dimensional data is crucial in medical prognostics (Wulczyn al. 2020). weather prediction [Agrawal et al., 2019), and autonomous driving (Kim et al. 2019). j. In or- der advance deep-learning methodology for probability estimation it is crucial build appropriate benchmark datasets. Here we build synthetic dataset and gather three real-world datasets, which we use systematically evaluate existing methodology. In addition, we propose novel approach, which outperforms current state-of-the-an methods. Our contributions are the following:
We introduce a new synthetic dataset for probability estimation where a population of people may have certain disease connected to age. The task is to predict the probability that they contract the disease from a picture of their face. The data are generated based on the UTKFaces dataset (Zhang jet al., 2017a), which contains age information. The dataset contains multiple versions of the synthetic labels, which are generated according to different distributions designed to mimic real- datasets. The dataset serves two objectives. First, it allows to evaluate existing methodology. Second, enables us to evaluate different metrics in a controlled scenario where have access to ground-truth probabilities.
world probability-preddice
benchmark datasets for three real-world applications: (1) precipitation forecasting from radar images, (2) prediction of cancer- patient survival from histopathology images, and (3) prediction of vehicle collisions from dashcam videos. We use these datasets to systematically evaluate existing approaches, which have been previously tested mainly on classification datasets.
We have used publicly available data to build boobabilit--estimatie
We propose Calibrated Probability Estimation (CaPE), a novel technique which modifies the train- ing process so that output probabilities consistent with empirical probabilities computed from the data. CaPE outperforms existing approaches on most metrics on synthetic and real-world data.
2 PROBLEM FORMULATION PROBABILITY ESTIMATION
The goal of probability estimation is to evaluate the likelihood of a certain event of interest, based on observed data. The available training data consists of examples æ; i < n, each associated with corresponding outcome In our applications of interest, the input data are high dimensional: video. The corresponding label yı is either 0 depending on whether or not the event in question occurred. For example, in the sancer-survival application the patient survived for 5 years after
each æ; corresponds to an image
x; is histopathology image of a patient, and y equals æ; was collected. The data have inherent uncertainty: yt. the patient's survival, does not depend deterministically on the histopathology image (due e.g. to comorbidities and other health factors). Instead, we assume that y/ equals 1 with certain probability p: associated with æ,.. as illustrated in Figure because the input data provides key information about the patient's 's survival chances.
At inference, probability-estimatee bility p. associated with a new input data point x (e.g. the probability of survival for over years for new patients based on their histopathology data). To summarize, this is not classification problem, because the labels are not completely predictable. Instead, the goal is to predict the probability of the outcome, which is critical in choosing course of treatment for the patient.
model aims to generate estimate ý of the underlying proba-
2

---

Under review as a conference paper at ICLR 2022
Pr
f+)
f(5
(a) Accurate probability estimation / Calibration
(b) Inaccurate probability estimation Calibration
Figure 2: Calibration is not enough. Uncolored/colored markers denote y 0/1 outcomes, re- spectively. Blue/red stand for two classes with different associated ground-truth probabilities (1/4 and 3/4 respectively). (a) The model f retrieves the true probabilities, which requires discriminating between inputs with low and high probability. (6) The model has no discriminative power, it just assigns the same probability to all outputs. However, the model is perfectly calibrated because out of all outcomes assigned 0.5 by the model, the fraction that equal to I is 50%.
EVALUATION METRICS
Probability estimation shares similar target labels and network outputs with binary classification. However, classification accuracy is not an appropriate metric for evaluating babaability-sstimatiee models due to the inherent uncertainty of the outcomes. This is illustrated by the example in Fig- ure 2a where perfect probability estimate would result a classification accuracy of just 75% |
Metrics when ground-truth probabilities available. For synthetic datasets, have access to the ground truth probability labels and can use them to evaluate performance. Two reasonable metries are the mean squared error or lg distance MSEp. and the Kullback-Leibe divergence KLp between the estimated and ground-truth probabilities:
MSEp = N
Σ (6: - p.), and KI,p
N
2((n N
pr log
)
Pr +
-m:) log
1- T
(1)
1=1
N is the number of data, and Pe, P. are the ground-tren tround--ruth and predicted probabilities respectively.
Calibration metrics. In real-world data, ground-truth probabilities not available. In order to evaluate the probabilities estimated by a model, we need to compare them to the observed probabil- ities. To this end, aggregate the examples for which the model output equals a certain value (e.g. 0.5), and verify what fraction of them have outcomes equal to 1. If the fraction is close to the model output, then the model is said be well calibrated.
Definition 3.1. A model is well calibrated if
P (u = f(x) e I(g)) g. v0 where y is the observed outcome, f(x) is the probability predicted by model f for input æ, and (gg a small interval around q.
(2)
Model calibration can be evaluated using the expected calibration error (ECE) Guuo
2017
(note however that the definition Guo et al. (2017) is specific to classification), Given probability- estimation model f and dataset of imput data æ, and associated outcomes y;s 1 S S N, we partition the examples into B bins, I, /2, examples by the model. Let Q1..... QB 1 the B-quantiles of the set (f(x)), [(xxx())))))))())(( x ),,, we have Ib : 1Q Qo] (f(x))) ) (setting Qo 0). For each bin, we compute the mean predicted and empirical probabilities,
Jn. according to the probabilities assigned to the
(6) Pemp
= E (y f(x) E Ib) =
(3)
Ibl
(EIndex(Ts)
(b)
Σ
f(x)),
(4)
(EIndex(IA)
'A perfect model (in terms of probability estimation), assigns 0.25 to the blue class and 0.75 to the red class. To maximize classification accuracy. we predict when the model outputs 0.75 (red examples) and 0 when it outputs 0.25 (blue examples). However, 25% of red examples have an outcome of 0, and 25% of blue examples have outcome of 1. As result, the model would only have 75% accuracy.
3

---

Under review as a conference paper at ICLR 2022
Correlation: 0 0.7
Correlation: 0.712
Correlation 0.679
Correlation: 0.652
Correlation:: 0 0.996
KS Krrr
ECE
2
6.3
0.5
0:33 MSEa
3.64
6.32
MSEp
3:33
6.64
663
0.22
MSEp
2:33
9.04
641
9.62
MSEp
E.43
6.4
0.021
0.02
2.40 MSEp
0.34
CE Early ST0p Intinite Data
ceature Platt
Dirrchict Cal.
Focal
Entrosy Rog. CaPE (oi))
CaFE (kemal)
MNCE
Figure 3: Evaluating evaluation metrics. We use synthetic data to compare different metrics to the gold-standard MSEp that uses ground-truth probabilities. Brier score highly correlated with MSEp, in contrast to the classification metric AUC and the calibration metrics ECE, MCE and KS- Error. The graphs show the results of the proposed method CaPE, as well the baselines described Other scenarios and similar comparison
in Section 6.3 on the Linear scenario (see Section 6.1 I
with KP are included Appendix D
where Index(I,) = f(x;) E Tb).
The pairs (9) Pemp) can be plotted as reliability diagram, shown in the second row of Figure and in Figure 6 ECE is then defined
B
q0))
ECE
= B
Pomp
(5)
6=1
Other metrics for calibration include the maximum calibration error (MCE) defined as
MCE
max
(b) Pemp
g0)
d= d=1.....B
and the Kolmogorov-Sminn Smirnov error distribution function, which is described in more detail in Appendix B
(Gupta et al. 2021, metric based on the cumulative
Brier score. Crucially, model without any discriminative power can be perfectly calibrated (see Figure 2b). The Brier score is a metric designed to evaluate both calibration and discriminative ability. is the mean squared error between the predicted probability and the observed outcomes:
1 p. x
Brier
(6)
This score can be decomposed into two terms associated to calibration and discrimination ability, as shown in Appendix C Using the synthetic data Section where the ground-truth probabilities are known, we show that Brier score is indeed a reliable proxy for gold-standard MSE metric based on ground-trutt -ound-truth probabilities MSEp in contrast to calibration metrics such as ECE, MCE or KS- error, and to classification metrics such as AUC (see Figure 3 and Appendix D
4 PROPOSED METHOD: CALIBRATED PROBABILITY ESTIMATION (CAPE)
Prediction models based on deep learning are typically trained by minimizing the cross entropy between the model output and the training labels (Goodfellow al. 2016). This cost function is proper scoring rule, which means that it evaluates probability estimates in a consistent manner and is therefore guaranteed be well calibrated in an infinite-data regime (Buja et al. 2005). as illustrated by Figure 4 (first column).
Unfortunately, practice prediction models are trained on finite data. This crucial the case of deep neural networks, because these models are highly overparametrized and therefore prone to overfitting (Goodfellow et al.) 2016). In classification, networks have been shown to capable of fitting arbitrary random labels (Zhang et al., 2017a). In probability estimation, observe that neu- ral networks indeed eventually overfit the observed outcomes completely. Moreover, the estimated probabilities collapse to (Figure 4 second column), phenomenon that has also been reported classification (Mukhoti et al., 22220) However, calibration is preserved during the first stages of training (Figure third column). This is reminiscent of the early-learning phenomenon observed
4

---

Under review as a conference paper at ICLR 2022
Infinite data
Overfitting
Early-stopping
CaPE
p
p 0.3
0.5
0.7
0.9
800
100
s.samllsss 122
600
700
g
5-000
400
150
100
200
100
P
Empirical probability
1.0 P]
1.0 [)
P
1.0 P]
0.5
0.5
p;
0.01
0.0
0.0
0.0
0.0
0.5
1.0
0.0
0.5
1.0 0.0
0.5
1.0
0.0
0.5
1.0
Predicted probability
Predicted probability
Predicted probability
Predicted probability
Figure 4: Miscalibration due to overfitting and how to avoid it. When trained on infinite data (i.e. resampling outcome labels at each epoch according ground probabilities), models min- imizing cross-entropy are well calibrated (first column). The top row shows results for the synthetic Discrete scenario (see Section [..] (top). The bottom row shows results for the Lineur scenario (dashed line indicates perfect calibration). However, when trained on fixed observed outcomes, the model eventually overfits and the probabilities collapse to either 0 or (second column). This is mitigated via early stopping (i.c. selecting the model based on validation cross-entropy loss), which yields relatively good calibration (third column). The proposed Calibration Probability Estimation (CaPE) method exploits this to further improve the model while ensuring that the output remains well calibrated. Appendix A.3 shows plots for all synthetic data scenarios.
for classification from partially corrupted labels (Yao et al. 2020 Xia et al. 2020 where neural networks learn from the correct labels before eventually overfitiing the false ones (Liu et al.) 2020).
Here, we propose to exploit the training dynamics of cross-entropy minimization through a method that we name Calibrated Probability Estimation (CaPE). Our starting point is a model obtained via early stopping using validation data on the cross-entropy loss. CaPE is designed to further improve the discrimination ability of the model, while ensuring that it remains well calibrated. This is achieved by alternatively minimizing the following two loss functions:
Discrimination loss: Cross entropy between the model output and the observed binary outcomes,
CD
ly log(f(=;)) (1 y1) log lo:(1-((z)))
Calibration loss: Cross entropy between the output probability of the model and the empirical probability of the outcomes conditioned on the model output:
cc=
Pemp log(f(zz)) + Pemg)) log(1 f(x:))]
where Pemp an estimate of the conditional probability Ply 11)(x) E [(]():))) where T(f(a;;)
a small interval centered at fa:)). As explained in Section if (xx) close this value, then the
model is well calibrated. We consider two approaches for estimating Pemp- (1) CaPE (bin) where
divide the training set into bins, select the bin by containing f(x;) and set Pemp = Pemp in equation 3
(b.)
(2) CaPE (kernel) where pamp is estimated through moving average with kernel function (see Appendix F for more details). Both methods are efficiently computed by sorting the predictions
(4) P.. The calibration loss requires reasonable estimation of the empirical probabilities Pemp. which can be obtained from the model after early learning. Therefore using the calibration loss from the beginning is counterproductive, demonstrated in Section
5

---

Under review as a conference paper at ICLR 2022
Training loss
Validation loss
MSE,
KLP
0.80
0.80
0.14
0.20
0.75
0.75
0.12
0.15
0.70
0.70
0.10
0.65
0.08
0.10
0.60
0.55
0.60
0.04
0.05
0.50
0.55
0.02
0.00
10 20
40 epoch
50
60
0
10
20
40 50
10
30 epoch
40
50 60
10
20
40 50 60 epoch
epoch
Eross-Entropy minimization
CaPE
CaPE begins
Comparison between the learn- ing curves of cross-entropy (CE) minimization and the proposed calibrated probability estimation (CaPE), smoothed with a 5-epoch moving average. After an early-learning stage where both train- ing and validation losses decrease, CE minimization overfits (first and second graph), with disastrous consequences in terms of probability estimation (third and fourth graph). In contrast, CaPE prevents overfitting, continuing to improve the model while maintaining calibration (see Figure
Figure 5: Calibrated Probability Estimation prevents overfitting,
Algorithm Pseudocode for CaPE
Require:
early stopped model freq. of training with Le
function BIN(B)
B-number of
Require:
m Require: KC(, q) exp (p (r
11.- .- In +- partitions by quantile of {p, 1/=1 such that
Require: {Ir,, vil.
training
Jo
/o=]
Gaussian kernel
Index(Ta [a) +- [J]pg
E
1.)
> indices in bin h
for
to nun.epochs do
Pimp 15p Σ 9/
empirical mean of bin
mod
then
(glins)(1g))
f(
end function
Update pings Vi, with BIN KERNEL + Le
compute discrimination loss
fumetion KERNEL(F, K)
r-window sixe; kerel
else
N-(1) +- r nenarsst neighhor of P. (ouipul probability space) (fr,
C + LD
b compute calibration loss
Σ
normalization factor
if
By ()
f t- backprop with C
b train with loss
Piop Σ
K (b,.,,),,Z
kemell smooth
for
SgENr(6)
end function
CaPE is summarized in Algorithm [ Figures 4 and 5 show that incorporating the calibration- loss minimization step indeed preserves calibration as training proceeds (this is not necessarily expected because CaPE minimizes a calibration loss on the training data), and prevents the model from overfitting the observed outputs. This is beneficial also for the discriminative ability of the model, because it enables it further reduce the cross-entropy loss without overfitting, as shown in Figure The experiments with synthetic and real-world data reported in Section 6 suggest that this approach results in accurate probability estimates across a variety of realistic scenarios.
RELATED WORK
Neural networks trained for classification often generate a probability associated with their predic- tion which quantifies its uncertainty. These estimates are often found to be inaccurate (Mukhoti al. 2020 Guo et al. [017]. Techniques mitigating this issue are often described as calibration methods, and broadly fall into three categories depending whether they: (1) postprocess the
outputs of a trained model, (2) combine multiple model outputs, or (3) modify the training process.
Post Post-processing methods transform the output probabilities in order to improve calibration held- out data (Zadrozny Fikan 2001: Gupta et al., 2021 Kull et al. 2017 2019). For example, Platt scaling (Platt] 1999) fits logistic function that minimizes the negative log likelihood loss. Temper- ature scaling Guo et al. 2017) does the same with a temperature parameter augmenting the softmax function. In contrast to these methods, CaPE enforces calibration during training, which has the advantage of enabling further improvements in the discriminative abilities of the model.
Ensembling methods combine multiple models to improve generalization. Mix-n-Matel (Zhang et al. 2020) uses a single model, and ensembles predictions using multiple temperature scaling transformations. Other methods (Lakshminarryynaan al. 2017: Maddox al. 2019) ensemble

---

Under review as a conference paper at ICLR 2022
Methods
Linear
Sigmoid
Centered
Skewed
Discrete
(x10-?)
MSEp
KL. pp
MSEp
KLpp
MSFp
KLp
MSEp
KLp
MSEp
KI op
Infinite Data
1.13
2.81
5.35
14.86
0.20
0.41
0.22
0.92
1.52
3.64
CE early-stop
4.21
10.93
6.16
17.16
0.48
0.98
0.40
1.79
2.24
5.26
Temperature
2.71
6.70
6.13
17.05
0.48
0.98
0.40
1.75
2.20
5.12
Platt Scaling
2.48
6.06
5.78
16.12
0.41
0.83
0.39
1.71
2.06
4.82
Dirichlet Cal.
3.56
9.06
8.71
25.33
0.46
0.94
0.48
2.31
2.74
6.52
Mix-n-matee
2.69
6.70
6.13
17.10
0,48
0,98
0.39
1.74
2.20
5.12
Focal Loss
4.13
10.51
6.89
19.51
0.48
0.97
1.27
11.61
2.92
Entropy Reg.
2.84
7.37
7.03
21.19
0.42
0.87
1.17
10.57
2.84
6.62
MMCE Reg.
2.22
5.65
5.33
15.03
0.44
0.90
0.54
2.43
2.08
4.90
Deep Ensemble
1.90
4.55
5.85
16.43
0.44
0.89
0.55
2.58
1.97
4.61
CaPE (hin)
1.83
4,46
5.29
14.59
0.38
0.78
0.40
1.72
1.83
4.31
CaPE (kemel) 1.81
4.41
5.22
14.47
0.40
0.81
0.39
1.70
1.85
4.36
Table I: Results on synthetic data. Appendix A.
shows a table with confidence intervals using bootstrapping. is model trained with infinite data obtained by continuous label resampling.
multiple models obtained using different initializations. These approaches are compatible with the proposed method CaPE; how to combine them effectively is interesting future research direction.
Modified training methods can be divided into two groups. The first group smooths the target 0/1 labels in order to prevent output estimates from collapsing to 0/1 (Mukhoti et al.) 2020) Szegedy ct al. 2016; Zhang et al. 2018 Thulasidasan et al. 2020 The second group, attaches additional calibration penalties to cross entropy loss (Kumar et al. 2018 Pereyra et al.) [2017] [iang et al.,
2020). CaPE is most similar in spirit to the latter methods, although its data-driven calibration loss different the penalties used in these techniques.
Datasets for evaluation The methods discussed in this section were developed for calibration in classification, and tested datasets such CIFAR-1011 10/100 (Krizhevsky, (009), SVHN (Netzer et al., 2011), and ImageNet (Deng et al. 2009) where the relationship between labels and input data is completely deterministic. Here, evaluate these methods on synthetic and real-world probabiliy-eestmmatin problems with inherent uncertainty.
EXPERIMENTS
6.1 SYNTHETIC DATASET: FACE-BASED RISK PREDICTION
To benchmark probability-estimation methods, we build synthetic dataset based on UTK-
Face (Zhang et al. (017b). containing face images and associated ages. We use the age the ith per-
2; to assign them risk of contracting discase p; (22)) for fixed function & N |0, 1]. Then we simulate whether the person actually contracts the illness (label yi = 1) or not (y: = 0) with probability P:- The probability-estimatie from the face image I;, which requires learning to discriminate age and map it to the corresponding risk. We design &a create five scenarios, inspired by real-world data (see Appendix Gh):
task is to estimate the ground-truth probability p4
Linear: Equally-spaced, inspired by weather forecasting: (x2) z/100
Sigmoid: Concentrated near two extremes: (x2) = (25(2 /0000 0.29))
Skewed: Clustered close to zero, inspired by vehicle-collision detection: (²(2) = z/250
Centered: Clustered in the center, inspired by cancer-survival prediction: 1/(2) =/300 0.35
[ Discrete: Discretized: (zz)) 0.2 [12220) + [=>400 [[2260) + [{>280)| + 0.1
6.2 REAL-WORLDE DATASETS
We propose to use three open-source, real-world datasets to benchmark probability ap- proaches (see Appendix H for further details the datasets and experiments).

---

Under review as a conference paper at ICLR 2022
Method
Cancer Survival
Weather forecasting
Collision Prediction
(x10-3 3)
AUC
ECE
Brier
AUC
ECE
Brier
AUC
ECE
Brier
CE stopy-stop
58.88
12.25
23.96
77.64
10.91
20.57
85.68
4.36
8.59
Temperature
58.88
12.07
23.73
77.64
8.66
20.21
85.68
4,56
8.51
Platt Scaling
58.91
10.28
23.33
77.65
6,97
19.53
85.76
3.04
8.23
Dirichlet Cal.
49.89
13.83
24.08
77.51
14.29
21.89
83.36
5.78
8.78
Mix-n-match
58.88
12.16
23.67
77.64
8.65
20.21
85.68
4.40
8.52
Focal Loss
55.02
12.15
23.31
76.18
8.32
20.27
82.21
9.07
9.82
Entropy Reg.
56.29
11.73
23.62
79.01
10,53
19.77
83.15
14.54
11.10
MMCE Reg.
48.45
11.84
23.73
76.69
8,46
20.12
85.18
2.94
8.48
Deep Ensemble
52.46
9.99
23.47
79.86
7.41
18.82
85.27
3.15
8.55
CaPE (bin)
61.44
12.31
23.20
78.99
5,16
18.37
85.70
3,16
8.18
CaPE (kernel) 61.22
9.48
23.18
79.00
5.08
18.39
85.95
3.22
8.13
Table 2: Results on cancer-survival prediction, weather forecasting, and collision prediction. Tables with all the metrics described in Section are provided Appendix A.2
Survival of Cancer Patients. Histopathology aims to identify tumor cells, cancer subtypes, and the stage and level of differentiation of cancer. Hematoxylin and Eosin (H&E)-stained slides are the most common type of histopathology data used for clinical decision making. In particular, they can be used for survival prediction (Wulczyn et al.) (020), which is critical in evaluating the prognosis of patients. Treatments assigned patients after diagnosis are not personalized and their impact on cancer trajectory is complex, so the survival status of patient is not deterministic. In this work, we use the H&E slides of non-small cell lung cancers from The Cancer Genome Atlas Program (TCGA P to estimate the the 5-year survival probability of cancer patients. The outcome distribution is similar to the Centered scenario in our synthetic data.
Weather Forecasting. The atmosphere is governed by nonlinear dynamics, hence weather forecast models possess inherent uncertainties (Richardson] 2007). Nowcasting, weather prediction in the near future, is of great operational significance. especially with increasing number of extreme in- clement weather conditions (Agrawal et al.) 2019; Ravuri et al. (202I). We use the German Weather rainfall-depth composites from 17 operational
service d which contains quality- Doppler radars. We use 30 minutes of precipitation data to predict if the mean precipitation will increase or decrease after one hour. Three precipitation maps from the past 30 minutes serve as an input. The outcome distribution is similar to the Linear scenario in our synthetic data.
Collision Prediction.
Vehicle collision is one of the leading causes of death in the world. Reliable collision prediction systems are therefore instrumental in saving human lives. These systems predict potential collisions from dashcam cameras. Collisions are influenced by many unknown factors, and hence are not deterministic. Following Kim et al. (019)), we use 0.3 seconds of real dashcam videos from YouTubeCrass dataset as input, and predict the probability of collision in the next 2 seconds. 'Tee data are very imbalanced as the number of collisions is very low, so the outcome distribution is similar to the Skewed scenario in our synthetic data.
6.3 BASELINES
We apply existing calibration methods developed for classification to probability estimation (as well as cross-entropy minimization with carly-stopping): (1) Three post-processing methods: Temper- ature Scaling (Guo et al. 2017). Platt Scaling (Platt 1999). and Dirichlet Calibration (Kull et al., 2019) applied to the best CE model, (2) Two Ensemble Methods: Mix-n-Mas Match (Zhang al. 2020) applied to best CE model, and Deep Ensemble (Lakshminarayanan et al.) 2017) with 5 networks, and (3) Three Modified Training methods: focal loss (Mukhoti et al. 2020). cntropy-maximize loss (Pereyra et al., (017). and MMCE regularization (Kumar et al. 2018). Appendix [F] provides detailed description. For our experiments on synthetic data, we also compare against model trained on an infinite amount of data by repeatedly sampling new outcomes from the ground-truth probabilities each epoch. This provides best-case reference for each scenario.
https: dwd de/ wwt.e/eeeeeeaeeereeeeeeeee
8

---

Under review as a conference paper at ICLR 2022
Cancer Survival
Weather Forecast
Collision Prediction
1.0
Sary
Empirical Probabiity
0.81
Empirical Probabiltty
0.8
Empirical Probability
A.8
0.6
0.6
04
02
0.2
0.2
0.0
0.0
0.4
Predicted Probability
Predicted Probability
Predicted Probability
Figure 6: Reliability diagrams for real-world data. Reliability diagrams computed on test data for cross-entropy minimization with early stopping, the proposed method (CaPE) and the best baseline for each dataset. CaPE produces better calibrated outputs. Appendix A.3 shows additional diagrams.
RESULTS AND DISCUSSION
Table T shows that calibration methods developed for classification can be effective for probability estimation. However, the performance of some methods is not consistent across all scenarios. For instance, regularization with negative entropy, which penalizes very high/low confidence, performs worse than CE when the ground-truth probability is close 0 or 1. In contrast, methods that do not make strong assumptions tend to generalize better to multiple scenarios (e.g. Platt scaling con- sistently beats CE). The proposed method CaPE outperforms other techniques in most scenarios, and even matches the performance of the infinite data baseline for the Sigmoid scenario. Finally, we observe that the Skewed scenario is very challenging: most methods barely improve the CE baseline.
Table the baseline methods and CaPE on the three real-world datasets. We present AUC, ECE for 15 equally-sized bins, and Brier score, as complementary metrics since the underlying ground-truth probabilities are unobserved. As discussed in Section 3 Brier score is the metric that best captures the quality of probability estimates. CaPE has the lowest Brier score in all three datasets, while also achieving lower ECE values and higher AUC values than the other methods. This demonstrates that enforcing calibration during training also yields a more discriminative model. The reliability diagrams in Figure depiect the probability estimates produced by CE, CaPE and the best baseline method on the three datasets, demonstrating that CaPE produces a well-calibrated outputs.
Falsre shows that each real-world dataset closely aligns with particular synthetic scenario: cancer survival with Centered; weather forecasting with Linear; collision prediction with Skewed. This supports the significance of our synthetic benchmark dataset, and provides insights in the dif- ferences among baseline models. For example, model averaging with deep ensemble performs well on weather forecasting but has higher Brier scores than Platt scaling on the other two datasets (see Appendix
D for further analysis based on pathological stages). Accordingly, deep ensemble also underperforms in the synthetic scenarios where ground-truth probabilities are clustered closely (Sig- moid, Linear), but is effective for Linear. Finally, as in the synthetic Skewed scenario, all methods had similar performance on the collision prediction task. This highlights the importance of consid- ering different scenarios when evaluating methodology for probability estimation.
8 CONCLUSION
In this work we evaluate existing approaches improve the output probabilities of neural net- works on prohability-estimen
problems. To this end, we introduce a new synthetic benchmark dataset designed to reproduce several realistic scenarios, and also gather three real-world datasets relevant to medicine, climatology, and self-driving cars. In addition, we propose a novel approach via deep learning that outperforms existing approaches for most datasets.
for probability-estine
An important application for probability estimation is in the context of survival analysis, which can
be recast as estimation of conditional probabilities (Lee al. 2018 Shamout al.) 2020 Gold-
stein et al. 202T). An interesting research direction is to consider problems with several possible uncertain outcomes (analogous to multiclass classification).

---

Under review as a conference paper at ICLR 2022
REFERENCES
Shreya Agrawal, Luke Barrington, Carla Bromberg, John Burge, Cenk Gazen, and Jason Hickey. Machine learning for precipitation nowcasting from radar images. arXiv preprint
arXiv:1912.1213. 2019.
Georgy Ayzel. Rainnet: a convolutional neural network for radar-based precipitation nowcasting. 2020.
btttp://gittub..co http://u/cccoooyyarrr/rraraaaaaa
Andreas Buja, Werner Stuetzle, and Yi Shen. Loss functions for binary class probability estimation and classification: Structure and applications," manuscript, available at www- stat. nottthaaroon.uemm..dee buja, 2005.
Xinlei Chen, Haoqi Fan, Ross B. Girshick, and Kaiming He. Improved baselines with momen- tum contrastive learning. CoRR, abs/2003.04297, 2020. URL https:://axiv.. /bs/
2003.04233
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hier- archical image database. In 2009 IEEE Conference Computer Vision and Pattern Recognition, pp. 245, 2009. doi: 10. N0.11099VPR.22000000000 5506848.
Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. 2016.
Mark Goldstein, Xintian Han, Aahlad Puli, Adler J. Perotte, and Rajesh Ranganath. X-cal: Explicit calibration for survival analysis, 2021.
Ian Goodfellow, Yoshua Bengio, Aaron Courville, and Yoshua Bengio. Deep learning, volume 1. MIT Press, 2016.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International Conference on Machine Learning, pp. 1321-1330. PMLR, 2017.
Kartik Gupta, Amir Rahimi, Thalaiyasingam Ajanthan, Thomas Mensink, Cristian Sminchisescu, and Richard Hartley. Calibration of neural networks using splines. In International Confer- ence on Learning Representations, 2021. URL @Q@8DEWNN22)
wet.:/iwwwwwee/eee..... htwpswwwwet/ffoummmm forum? ?id
Maximilian Ilse. Jakub Tomezak, and Max Welling. Attention-based deep multiple instance learn- ing. In Jennifer Dy and Andreas Krause (eds.). Proceedings of the 35th International Con- ference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 2127 2136. PMLR, 10-15 Jul 2018. URL https::/proceeciinn wttpgss.m...prrs.. w.w.pppees/vve.
ilsel8a. 1aa1l8..mmmm html
Hoon Kim, Kangwook Lee, Gyeongjo Hwang, and Changho Suh. Crash to not crash: Learn to identify dangerous vehicles using simulator. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 978-985, 2019.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009.
Meelis Kull, Telmo M. Silva Filho, and Peter Flach. Beyond sigmoids: How to obtain well- calibrated probabilities from binary classifiers with beta calibration. Electronic Journal of Statis- tics, 5052)50022~ 5080, 2017. doi: 10.1214/7-EIS13388LL URL httpts:///o.iii orq/1C. 1214/17-B3S1338S1
Meelis Kull, Miquel Pereoló-Nieto, Markus Kängsepp. Telmo de Menezes Silva Filho, Hao Song, and Peter A. Flach. Beyond temperature scaling: Obtaining well-calibrated multiclass probabili- ties with dirichlet calibration. In NewrIPS, 2019.
Aviral Kumar, Sunita Sarawagi, and Ujjwal Jain. Trainable calibration measures for neural networks from kernel mean embeddings. In ICML, 2018.
Balaji Lakshminarayanan. Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles, 2017.
10

---

Under review as a conference paper at ICLR 2022
Changhee Lee, William Zame, Jinsung Yoon, and Mihaela van der Schaar. Deephit: A deep learning approach to survival analysis with competing risks. Proceedings of the AAAI Conference on Arti- ficial Intelligence, 32(1), Apr. 2018. URL http:: www.aaaaaaaaaaaaaaaaaaaaaaaaaeea index. wwwpppppppa.... AAAI/
article/viewe
Gongbo Liang, Yu Zhang, Xiaoqin Wang, and Nathan Jacobs. Improved trainable calibration method for neural networks medical imaging classification. In British Machine Vision Conference (BMVC), 2020.
Sheng Liu, Jonathan Niles- Weed, Narges Razavian, and Carlos Fernandez-Grande Early-learning regularization prevents memorization of noisy labels, Advances in Neural Information Processing Systems, 33, 2020.
Wesley Maddox, Timur Garipov, Pavel Izmailov, Dmitry P. Vetrov, and Andrew Gordon Wilson. A simple baseline for bayesian uncertainty deep learning. CoRR, abs/1902.02476, 2019. URL
org/ abs/ 1100/99020000 02477
Jishnu Mukhoti, Viveka Kulharia, Amartya Sanyal, S. Golodetz, Philip H. S. Torr, and P. Dokania. Calibrating deep neural networks using focal loss. ArXiv, abs/2002.09477, 2020.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Ng. Reading digits in natural images with unsupervised feature learning. 2011. URL http: /ufldl.
hww.oluuoouuseeberrr...ooomm pdf
Gabriel Pereyra, George Tucker, Jan Chorowski, Lukasz Kaiser, and Geoffrey E. Hinton. Regular-
izing neural networks by penalizing confident output distributions. CoRR, ////1701.066 2017.
URL http::ttttiiiccooooc
abs//10..... 06548
J. Platt. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. Advances in Large Margin Classifiers, 10(3), 1999.
Janis Postels, Francesco Ferroni, Huseyin Coskun, Nassir Navab, and Federico Tombari. Sampling- free epistemic uncertainty estimation using approximated variance propagation. CoRR, abs/1908.00598, 2019. htLp:ttt://aa wwwpxiivoorrr/oooooooooo abs 1908 190..00598...0.
Suman Ravuri, Karel Lenc, Matthew Willson, Dmitry Kangin, Remi Lam, Piotr Mirowski, Megan Fitzsimons, Maria Athanassiadou, Sheleem Kashem, Sam Madge, al. Skillful precipitation nowcasting using deep generative models of radar. arXiv preprint arXiv:2104. 009540 2021.
Lewis Fry Richardson. Weather prediction by numerical process. Cambridge university press, 2007.
Farah E. Shamout, Yiqiu Shen, Nan Wu, Aakash Kaku, Jungkyu Park, Taro Makino, Stanislaw Jas- trzebski, Duo Wang, Ben Zhang, Siddhant Dogra, Meng Cao, Narges Razavian, David Kudlowitz, Lea Azour, William Moore, Yvonne W. Lui, Yindalon Aphinyanaphongs, Carlos Fernandez- Granda, and Krzysztof Geras. An artificial intelligence system for predicting the deteriora- tion of COVID-19 patients in the emergency department. CoRR, abs/2008.017 2020. URL https://arxiv. http:///aaxiv.oor///////000 wttp/////////ooo/o 2008 01774
Alexander Shekhovtsov and Boris Flach. Feed- forward propagation in probabilistic neural networks with categorical and max layers. In International Conference on Learning Representations, 2019.
URL /ettps://openreviie
net forum? id= SkMuP iRcKQ
Christian Szegedy, V. Vanhoucke, Ioffe, Jonathon Shlens, and Z.. Wojna. Rethinking the incep- tion architecture for computer vision. 2016 IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp. 2818-2826, 2016.
Jeppe Thagaard, Søren Hauberg, Bert van der Vegt, Thomas Ebstrup. Johan D Hansen, and Anders B Dahl. Can you trust predictive uncertainty under real dataset shifts in digital pathology? In International Conference on Medical Image Computing and Computer-Assister Intervention, pp- 824 833. Springer, 2020.
Sunil Thulasidasan, Gopinath Chennupati, Jeff Bilmes. Tanmoy Bhattacharya, and Sarah Michalak. On mixup training: Improved calibration and predictive uncertainty for deep neural networks, 2020.
11

---

Under review as a conference paper at ICLR 2022
Hao Wang, Xingjian Shi, and Din Yeung. Natural-parannn parammeee networks: A class of probabillistic neural networks. CoRR, abs/1611.00448, 2016. URL htttp://axxiv.orgee
00448
Ellery Wulczyn, David Steiner, Zhaoyang Xu, Apaar Sadhwani, Hongwu Wang, I. Flament, C. Mermel, Po-Hsuan Cameron Chen, Yun Liu, and Martin C. Stumpe. Deep learning-basee survival prediction for multiple cancer types using histopathology images. PLoS ONE, 15, 2020.
Xiaobo Xia, Tongliang Liu, Bo Han, Chen Gong, Nannan Wang, Zongyuan Ge, and Chang- Robust early-learning: Hindering the memorization of noisy labels. In International Conference on Learning Reprexentations, 2020.
Quanming Yao, Hansi Yang, Bo Han, Gang Niu, and James Tin- Yau Kwok. Searching to exploit memorization effect in learning with noisy labels. In International Conference Machine Learn- ing, pp. 10789-10798. PMLR, 2020.
B. Zadrozny and C. Elkan. Obtaining calibrated probability estimates from decision trees and naive bayesian classifiers. JCML, 2001.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. ICLR, 2017a.
Hongyi Zhang, Moustapha Cissé, Yann Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. ArXiv, abs/17 110.09912, 2018.
Jize Zhang, B. Kailkhura, and T. Y. Han. Mix-n-match: Ensemble and compositional methods for uncertainty calibration in deep learning. ICML, 2020.
Zhifei Zhang, Yang Song, and Hairong Qi. Age progression/egression by conditional adversarial autoencoder. In IEEE Conference Computer Vision and Pattern Recognition (CVPR). IEEE, 2017b.
12

---

Under review as a conference paper at ICLR 2022
A
ADDITIONAL RESULTS
We present here supplementary results to the ones presented in Section 7
A.1
FACE-BASED RISK PREDICTION
Full evaluation with confidence intervals derived using 1000 bootstraps for the five simulated sce- narios examined: Linear 3T Sigmoid : Centered (Table Skewed (Table
Discrete (Table Note that all numbers are downscaled by 10 the tables.
Linear
ECE
MCE
KS
Brier
MSEp
KLp
Infinite Data
4.14
12.07+3.29
2.24
18.97-00.3
1.14=0.04
2.82+:0.11
CE early-stop
12.32:1..83
21.79::.997
12.16:10.3
21.82:0.51
4.21
10.94=0.36
Temperature
5.7:0.744
13.82+2.71
2.36=0.74
20.47+0.37
2.73=0.11
6.75+t0.25
Platt Scaling
4.29-+0.77
10.94
1.340.45
20.18=0.36
2.48=0.09
6.07
Dirichlet Cal.
7.38:11.12
22.58447.24
3.78-0.46
21.32+10.3
3.56-10133
9.08
Focal Loss
5.34:10.68
13.3142.67
3.86
21.99+10.2
4.13=0.11
10.52+
Mix-n-match
5.46:10.8
12.942.51
1,92#044
20.43+0.35
2.710.11
6.72
Entropy Reg.
5.7:0.774
13.52:1.67
4.9410.9
20.42::..3
2.58=0.09
6.65
MMCE Reg.
4.89:10.74
12.5742.27
1.92=0.66
20.04
2.24=0.08
5.68
Deep Ensemble
4.26+0072
11.33+2.38
1.95+0061
19.88
1.9+0.07
4.55
CaPE (bin)
11.85:2.49 4.58:10.75
1.71=0.51
19.68
1.78-0.07
1.35
CaPE (kernel)
4.62+t0.62
12.25+2.31
1.65=0.38
19.71
1.74=0.07
4.3700117
Table 3: Performance on Face-based Risk Prediction. Linear scenario.
Sigmoid
ECE
MCE
KS
Brier
MSE
KL
Infinite Data
6.410..71
20.6313.44
2.74::0.45
16.28:10.44
5.34
14.52-0.51
CE early-stop
6.19:10.5
17.043.68
5.86:10.8
16.68:0.42
6.16:10.17
17.16:10.48
Temperature
5.57:10.71
15.32-13.09
5.02-10.83
16.58
6.13:0.17
17.09
Platt Scaling
3.45+00.8
10.32+2.79
1.340.43
16.33+0.34
5.78+0.19
16.15=0.47
Dirichlet Cal.
14.5+1.15
25.68
4.67::032
19.21#0.43
8.6410.26
28.18
Focal Loss
4.65+00.7
11.8442.78
2.6610.77
16.9640.34
6.86:0.21
19.46:10.5
Mix-n-match
5.65++0.76
15.32+3.41
5.09+0.94
16.6+0.36
6.12+0.17
1..06
Entropy Reg.
9.51+0.79
18.7742.38
7.26+0.78
17.17+0.31
7.02+0.17
21.16=0.42
MMCE Reg.
4.67:10.76
13.6342.59
2.5.0.533
15.9+0.51
5.35
15.06-0.49
Deep Ensemble
5.17:0.74
16.1243.11
2.04
16.39.0.45
5.86+0022
16.4640.66
CaPE (bin)
3.78:10.6
11.96:12.59
2.22+10.7
5.17 15,8411.43
14.27
CaPE (kernel)
3.9:00.55
11.7342.79
2.05-LO.54
15.85
5.1640.2
14.34.10.49
Table 4: Performance Face-based Risk Prediction. Sigmoid scenario.
13

---

Under review as a conference paper at ICLR 2022
Centered
ECE
MCE
KS
Brier
MSE,
KLp
Infinite Data
4.29:10774
12.38+2.92
0.8
24.22++0.13
0.2:00011
0.41
CE early-stop
5.76:10.84
15.32:1..07
4.19
24.68:10.08
0.48
0.98
Temperature
6.09:10.82
15.83+2.91
4.74
24.74+0.06
0.48+0001
0.98+0.03
Platt Scaling
4.57=0.76
11.85=2.5
2.79=0.85
24.622:..8
0.41
0.83+0.03
Dirichlet Cal.
4.84+11115
13.1347.61
2.16+0.86
24.74.0.1
0.46+00.11
0.94+0.03
Mix-n-match
6.05++0883
15.71+2..2
4.68
24.74+0.06
0.48+0.01
0.98+0.02
Focal Loss
5.09-+0.83
13.4+2.87
3.44+1.02
24.8+
0.48+0.01
0.97+00.03
Entropy Reg.
5.02:10.86
12.6913.42
3.27+0.96
24.7440.06
0.45
0.92:50.03
MMCE Reg.
5.56:10.86
13.59-22.65
2.71
24.7
0.44
0,910.03
Deep Ensemble
4.84:10.7
12.39+2.52
2.64+0.71
24.69
0.44
0.89
CaPE (bin)
11.81 4.73:10.82
0.67
24.56:10.11
0.38
0.78
CaPE (kernel)
5.41+0087
12.71=2.5
2.39=0.78
24.59+0.11
0.4
0.81+0.03
Table 5: Performance on Face-based Risk Prediction. Centered scenario.
Skewed
ECE
MCE
KS
Brier
MSEp
KLp
Infinite Data
2.7:10.46
7.64412.14
1.05-L0.39
11.0++0.51
0.22:10.01
0.92+10.03
CE early-stop
3.07-4-057
1.88
1.28
11.18+00.5
0.4-10.01
1.79-10.06
Temperature
3.14++0.49
7.92+1.84
1.12++0.3
11.22+0.47
0.4++0.02
1.76++0.06
Platt Scaling
2.99+-0.53
7.73+1.55
1.07+0.37
11.1+0.54
0.39++0.01
1.72+0.06
Dirichlet Cal.
3.04:10.73
7.81
0.9710.3
11.22::.42
0.4740.02
2.31
Focal Loss
8.29+1067
14.93+1.44
6.16100.7
12.0140.41
1.28
1.6310.66
Mix m-tch
2.99::0.3
7.78
1.08+0.32
11.18=0.49
0.4:0.01
1.75+0.05
Entropy Reg.
7.67=00557
14.43+t1.5
5,2+0.71
11.94:1045
1.18:0..3
10.74=0.65
MMCE Reg.
3.68:10.59
10.94:12.76
1.47.1031
11.14:0.44
0.54
2.44
Deep Ensemble
2.87+10.5
1.21
1.36400.4
11.28
0.55
2.50
CaPE (bin)
8.18=1.51 3.29
1.1740.34
11.07=0.47
0.4
1.73+0.06
CaPE (kernel)
3.16++0.5
1.14
1.09
11.1740.53
0.39+0..01
1.69++0.06
Table 6: Performance on Face-based Risk Prediction. Skewed scenario.
Discrete
ECE
MCE
KS
Brier
MSF.
KI,p
Infinite Data
4.23++0.74
11.16=2.5
1.45++0.49
20.38+
1.52++0.05
3.6
CE early-stop
6.7-1086
18.6243.52
2.6110.53
21.911:.36
2.2440.08
5.2740017
Temperature
6.12.0.87
16.8243.56
3.3710.86
21.76:10.35
2.2140.08
5.1540.18
Platt Scaling
4.7.0.72
11.69+2.44
1.67+0.51
21.44+0.32
2.06+0.08
4.83+:0.17
Dirichlet Cal.
7.13+10.86
22.67+5.08
3.1810.68
22.1+0.34
2.74+0.1
6.53+0.22
Focal Loss
5.7-10.75
13.68+2.32
4.6240.91
21.77:10.28
2.92-10.09
6.77:00.1
Mix-n-match
6.27-10.76
16.8312.95
3.47+0.93
21.77.1.33
2.21100.88
5.1410.18
Entropy Reg.
6.69-10.7
15.38+2.43
6.03+1.13
21.7940.31
2.8440.08
6.62+0.19
MMCE Reg.
3.96+00.7
10.422.4
1.51+0.47
21.12+0.35
2.09
4.9240.18
Deep Ensemble
4.76++0.74
11.49+2.23
2.04+00.1
21.17+0.31
1.97+0.00
4.61+0.117
CaPE (bin)
5.4110077
14.4543.15
2.2410.59
21.3340.36
18110.08
4.28
CaPE (kernel)
4.96+00.8
12.9742..3
2.18
21.2140.42
1.8400.08
4.35
Table 7: Performance on Face-based Risk Prediction. Discrete scenario.
14

---

Under review as a conference paper at ICLR 2022
A.2 SUPPLEMENTARY METRICS ON REAL-WORLD DATASET
We present here additional metrics on the real world data: Cancer Survival (Table 8); Climate Fore- casting (Table 9 Collision Prediction (Table 10
Methods
(x10-3)
AUC
ECE
MCE
NLL
Brier
KS
CE Early-stop
58.88
12.25
25.35
67.92
23.96
6.44
Temperature
58.88
12.07
24.65
67.11
23.73
6.92
Platt Scaling
58.91
10.28
27.69
66.11
23.33
4.91
Dirichlet Cal.
49.89
13.83
35.52
67.57
24.08
6.00
Mix-n- Mix-n-match
58.88
12.16
24.52
66.89
23.67
7.18
Focal loss
55.02
12.15
26.34
65.92
23.31
6.38
Entropy Reg.
56.29
11.73
30.81
66.49
23.62
6.83
MMCE Reg.
48.45
11.84
37.36
66.83
23.73
3.64
Deep Ensemble
52,26
9.99
28.30
66.22
23.47
5.02
CaPE (bin)
61.44
12.31
25.27
65.75
23.20
2,59
CaPE (kernel)
61.22
9.48
32.40
65.70
23.18
3.70
Table 8: Baselines with full metrics for cancer survival
Methods
(x10-2)
AUC
ECE
MCE
NLL
Brier
KS
CE Early-stop
77.64
10.91
25.50
59.97
20.57
11.03
Temperature
77.64
8.66
23.56
58.77
20.21
7,41
Platt Scaling
77.65
6.97
16.47
57.38
19.53
3.26
Dirichlet Cal.
77.51
14.29
30.09
62.83
21.89
5.21
Mix-n-match
77.64
8.65
23.58
58.77
20.21
7.39
Focal Loss
76.18
8.32
21.25
59.01
20.27
4.45
Entropy Reg
79.01
10.53
20.72
57.83
19.77
5.00
MMCE Reg
76.69
8.46
19.73
59.25
20.12
7.31
Deep Ensemble
79.86
7.41
18.24
55.28
18.82
7.57
CaPE (bin)
78.99
5.16
15.09
79.00
18.37
2.34
CaPE (kernel)
79.00
5.08
13.28
54.32
18.39
2.34
Table 9: Baselines with full metrics for weather prediction
Methods
(x10-2)
AUC
ECE
MCE
NLL
Brier
KS
CE Early-stop
85.68
4.36
19.87
31.67
8.59
1.54
Temperature
85.68
4.56
16.79
30.36
8.52
2.9
Platt Scaling
85.76
3.04
12.39
29.42
8.23
1.52
Dirichlet Cal.
83.36
5.78
18.13
30.90
8.77
1,60
Mix-n-match
85.68
4.40
17.41
30.25
8.52
2.60
Focal Loss
82.21
9.07
19.85
34.41
9.82
8.72
Entropy Reg
83.15
14.54
21.27
38.74
11.10
13.44
MMCE Reg-
85.18
2.94
8.95
30.65
8.48
2.44
Deep Ensemble
85.27
3.15
16.53
30.20
8.54
2.01
CaPE (bin)
8.57
3.16
12.21
30.61
8.18
2.13
CaPE (kernel)
85.95
3.22
13.32
30.44
8.13
2.10
Table 10: Basclines with full metries for collision prediction
15

---

Under review as a conference paper at ICLR 2022
A.3
ADDITIONAL RELIABILITY DIAGRAM
We present here additional reliability curves to the ones illustrated in Figure 6
Cancer Survival
Weather Forecasting
Collision Prediction
Tempesatuur
Entropy leg.
Temperature
Entropy Reg
Temperature
Entropy Ree
14
1.
1.0
0.
06
0.6
0.4
04
04
04
04
02
02
05
022 04
56 08 1.0
00.
02 04 06 04 1.0 MWCE
0.0
02 04 06 08 10
03
0.2 04 56 08 1.0
0.0
0.
04
04 06
1/6
0.0
0.7
0.4 06 08 1/
Plat: Scaling
Platt Scaling
MMCE
Platt Scaling
MMCCF
1.0
1.0
1.0
0.6
0.5
04
04
04
M4
0.2
0.
68
da
1/1
00
0.
68 DJ 06
00
07 04 05 ba
1.0
CD B2 34 65 CR 13
Circhlet
Deep Ensemble
Darchlet Cal.
Deep Ensemble
Dirichiet
Deep Ensemble
1.3
1.4
1.0
06
06
0.
04
0.4
0.4
04
0.4
CD 02 04 04 CB 1.0
03 22 04 64 GE 10 CaPElbin)
3D
02 C4 04 26
1.5
GD
02 0.4 05 0B 14
0. Mben match
22 04
00 0. 04 05 CaPElbin)
Mit-n-matth
Mos-n-match
CaPEibi))
1.0
0.6
06
06
54
04
6A
9.
04
90
V
0.0 0.2 0.4
D6 08 1.0
05
32 04 06 0. 1.0
0.5
02
04 06
08 1.0
05
02 04 06 0. 100
0.5
0.2 04 06
05
00 02 0.4 36 CaPElkemaa)
Focal Loss
CaPEilermal
Foral Loss
CaPE)uernal!
Focal Loos
1.0
1.0
0.6
06
24
04
0.4
B4
84
022
02
00 0.7 04 DA
GB 1.0
00 32 04 06 0. 1.0
30
0.
06 08 1.0 Earlystopp
0.4
04 06 05 10
03 0.2
02 02 04 06
CE Early-stop
1.2
1.0
04
04
02
0.2
02
06 08 08 1/2
Figure 7: The reliability diagrams of all the baselines on real-world datasets. We train all base- line methods on each of the datasets and plot the empirical probability(y-xxis) against predicted probabiliy(x-axxis. The axis labels are removed due to space constraints.
16

---

Under review as a conference paper at ICLR 2022
We present here additional reliability curves for the different synthetic data scenarios, mentioned in Figure
B KOLMOGOROV- SOMOOGOROV-SMINOO ERROR
We derive the KS-error, mentioned Section
For calibrated estimator
Ply = (a)( E [(4)]
vO 5
for some small interval [()) around q.
Hence
Ply 1, f()) E I(g)) = P[f(x) [(g)]g, vo 5
Similarly to the Kolmogorov-Smingg (KS) test for distribution functions, we can recast this property in integral form
Φι Ply 1 f()) E (q)]dgg
62(0) P(f(x) a())lddq
We evaluate Φ 02 from finite sample (2 1/), 1...
2
Σ Iy = f(x,) I {)), Φ2( = ((f();) S {))((a))
1=1
The KS error is defined
KS = max -200)~22(0 150
Φ1, 22 be efficiently computed by sorting the data points with respect to their confidence scores fx;)). The KS error has the advantage of being independent of binning configurations, unlike ECE and MCE.
C BRIER SCORE DECOMPOSITION
We present here decomposition of the Brier score into components, discussed in Section
The Brier score can be interpreted as sum of two terms, calibration and refinement. Assume the network can output one of K distinct possible predictions, i.c., & ..t.....qk)
Denote Sk. the set of all inputs with output pt and Ōk the empirical probability over Sy. i...
S (x) 9k ) S
T&R.
Σ
nuk
Then we can write
1 N
Brier
(p: v1)
nu
Tk )x
ng [x))
N
The first term on the RHS, calibration, is similar to MSE,, with the empirical probabilities qı sub- stituting for the true labels. The second term, refinement, is an estimate of the confidence deter- mining Te It is related to the area under curve (AUC), which measures to the achievable accuracy of the network as classifier. The term is smaller as the prediction classes fi tend towards or 1. Thus, this term penalizes empirically calibrated predictors, with low discriminative power, as in Figure 2b
17

---

Under review as a conference paper at ICLR 2022
Linear
Empirical probability
Infinite Data
Overfitting
Early Stopping
CaPE
1.0 P
1.0
1.0
1.0
0.0
0.0
0.0
0.5
1.0
0.0
1.0
0.0
1.0
0.0
1.0
Predicted probability
Predicted probability
Predicted probability
Predicted probability
Sigmoid
Empirical probability
Infinite Data
Overfitting
Early Stopping
CaPE
1.0
1.0
1.0
1.0
0.5
0.5
0.5
0.5
6
0.0
Pi
0.0
p
Predicted probability
Predicted probability
Predicted probability
Predicted probability
Centered
Infinite Data
Overfitting
Early Stopping
CaPE
Empirical probability
P;
1.0 Pi
1. Pi
1. P
0.5
0.5
0.5
0.5
0.0
0.0
0.0
0.0
0.0
1.0
0.0
1.0
0.0
0.5
1.0
0.0
1.0
Predicted probability
Predicted probability
Predicted probability
Predicted probability
Skewed
Empirical probability
Infinite Data
Overfitting
Early Stopping
CaPE
P
P
10
Pi
1. Pi
0.0
0.0
0.0
0.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
Predicted probability
Predicted probability
Predicted probability
Predicted probability
Discrete
Empirical probability
Infinite Data
Overfitting
Early Stopping
CaPE
1.0
1.0
10
10
6
0.0
1.0
0.0
1.0
0.0
1.0
0.0
1.0
Predicted probability
Predicted probability
Predicted probability
Predicted probability
Figure 8:
Reliability diagrams for different synthetic data scenarios. We can see that CaPE out- performs early stopping, prevents overfitting, and achieves a performance on par with training on infinite resampled data.
18

---

Under review as a conference paper at ICLR 2022
Linear
Correlation: -0.50
Corelation:
Corelation
Conelation:
Correlation: 9.996
50
ECE
MCE
KS Emor
MSE,
MSED
MSE
MSEE
MSE,
Sigmoid
Conelaaiio:
Corelation
Correlation
Correlation:
ECE
KS
MSEpp
MSF
MSSPp
MSF
MSEp
Centered
Corrlaatonn
Conrelation 0.692
Correlation
Corralation:
Correlations 0.979
KS Smor
*6
MCE
5,0
MSE,
MSE;
MSE,
MSE,
MSE,
Skewed
ECE
KS
MSE,
MSE,
MSE,
MSE,
MSE,
Discrete Coreeaation
Comelation: 0.673
Correlation
Conrelation:
MCB
Brler
KS
MSS..
MSF,
MMEF,
MSFF
MSE,
Earty-stop Infintte
Dirchett
Focal MMCE
Cafi (oemw))
Post Scaliin
Figure 9: The comparison between MSE, and other metrics on synthetic data. Brier score presents the most consistent correlation with MSE.
D METRIC COMPARISON
We present here the correlation between different calibration and accuracy metrics and metrics that have access the ground truth probabilities, MSE, and KL-divergence, eveluated over all five scenarios our Face-based Risk Prediction synthetic dataset, referred to in Section 3
E ESTIMATION OF EMPIRICAL PROBABILITY IN CAPE
We describe in further detail the two ways to estimate the conditional probability Ply = 11f(x) e T(g)), introduced in Section 4
19

---

Under review as a conference paper at ICLR 2022
Linear Corelation
Correlation -0.77
Conelation: 0.731
Conelation:
Correlation: 0.985
10
MCE
KS Emor
Klpp
Klop
Klpp
Klp
Sigmoid
Conelation:
Corelaaion
Correlation:
ECE
MCE
KS
Kl,p
Kl.p
Kl
Kl.p
Kl
Centered
Corrulation -0.771
Conelation: 0.603
Comelation 0.569
Correlation: 0.491
Correlation: 0.379
KS enor
MCE
8
KL.
Kl.
KLp
KLpp
Klg
Skewed
Ciy Stp Si Stp Sxg
Ciy 5% 5g
ECE
MCCE
5,5
Klg
KL
Kl.0
Kl.y
KLg
Discrete
Correlation:
Correlations 0.681
Corelaaiio
Correlation:
0.794
Sana
Sit
MCE
Ulg
Klg
Klpp
Klgp
Kl
Kl
Early-stop
Temperatire
Divinde
Foral Lows
CaPt (twenet)
Scaling
Figure 9: Comparison between KL, and other metrics on synthetic data. Brier score presents the most consistent correlation with KL
We wish to estimate the conditional probability of an output given network prediction f(x),
Ply fx)()) e [q)] We can approximate the probability by averaging over points E I(g).
Ply = 1l)(x) E T(4) ~ T(a)
Σ (E
Ply = 1x)(x)
(7)
An empirical estimate of PPyy (x) [(g)) would be
1
Ply = 1x) e I(g)] x Index([(g))) f(x))el()))
Σ
Vi
(8)
where Index(T,) (m)) T()))
20

---

Under review as a conference paper at ICLR 2022
Alternatively, use kernel estimation:
(-6---²²)
Ply = 11f(2) E T(g))
~
Σ
Ply = 1[f(=)= = - exp
Z
pEf(g)
where Z = (g) exp (
the normalization factor. An empirical estimate of the
conditional probability would then be
(f(x))) 9) σ²
Ply = 11f(x) E [(1)
Σ
Yi exp
(10)
[(m))E
Based on these two approximation methods, we can design an algorithm to estimate Pemp
Bin We divide data into B bins of equal size. Q1..... ................................ QB the data B-quantiles. We wish . B, Qn = 0. Denote Ia := [Qu-1, Qul n
to estimate Ply = (x) E [Qa-1- Qoll, = 1.....
{f(x) set of all predictions (Qb-1, Q6). and Index (T)) (f)f();))))))))))))))))))))))( To}- We have,
Fly = fx) E [Qo-1, Qoll s pemp
1
Σ y1
(EI)de((Ta)
We assign pemp to all data points in the b-th quantile
Pemp
Pemp
Vi E Index(Is)
Kernel In this case we use kernel estimation:
ΣkENN(E.T) K (i, kb) 4/
Pemp
(11)
EkENN(i.r) K (i,k) k)
NN(i, r) defines r data points whose predictions are nearest to fo: = (x::). K(1, is the Gaussian kernel
k(i,)) exp (- (pi - 2,)² -,,)²
σ²
with hyperparameter a.
F CALIBRATION BASELINES
This section includes review of the baseline methods, discussed in Section 6
Post-processing Postprocessing for calibration requires finding function f [0, 1] + |0, 1], that augments the output of a the neural network p. > f(.)) in order to achieve better calibration properties
Platt scaling Platt 1999
optimizes f on validation set within the following family,
fi = (w D. +
(12)
where W E IR, 6 e R and a is the Sigmoid function. The non-probabilistic predictions of classifier used features for logistic regression model, which is trained on the validation set to return probabilities.
Temperature scaling (Guo al.) (2017) is single parameter variant of Platt Scaling where we only change the temperature of the softmax to obtain the calibrated probabilities.
f(;: Softmax (p:/T")
(13)
where T E R minimizes the negative log-likelihood of validation set.
Beta/Dirchlet calibration (Dir-ODIR (Kull et al. 2017 2019) assumes that the probabili- ties can be parametrized by a Beta/Dirchlet distribution i.c.
ff ~ (e) (,),
(14)
Assume the prior to be p(y j) kj. E [0, 1], we have P(y)ff) Nyfs. and then a (g6 are estimated by maximizing the posterior.
21

---

Under review as a conference paper at ICLR 2022
These calibration methods simultaneously train several neural networks from end to end, varying parameters in the training process. The final output is some function of all the different outputs.
Ensembling
Mix-n-Match (Zhang et al. 2020) improves calibration by ensembling parametric and non- parametric calibrators. Denote the temperature scaling function with g T)). Then Mix- n-Match ensembles different temperatures
f) W19, (f:, T') wag, (f., 0) + wagj (p:, oo)
(15)
After ensembling the parametric temperature scaling, Mix-n- --kthhhhhhh applies non parametric isotonic regression.
Deep ensemble ([aassminarayanaa et al. 2017) trains M copies of the neural network with different initialization. The probability estimation is the average of all single model estimations
p(ya (x))
1 M
Σ po; (yi xx)
(16)
Modified training These calibration methods train the neural networks from end to end, modify- ing the training process to improve calibration.
Confidence penalty (Pereyra al. 2017) Penalizeslow entropy output distributions (confi- dence penalty). Label smoothing improve state-of-the-art models across benchmarks.
( Σ
log po (x) 3H (pe()(az:))
(17)
Focal loss (Mukhoti et al. 2020) maximizes entropy while minimizing the KL divergence between the predicted and the target distributions. It also regularizes the weights of the model to avoid overfitting
((0) -0 (1 pe (y (z))) log po (x)(2))) 7 R.
(18)
Kernel MMCE (Kumar et al. 2018) is reproducing kernel Hilbert space (RKHS) kernel based measure of calibration that is efficiently trainable, alongside the negative likelihood loss. Given data samples D = {(,, rr)) where = X(K-=w) and r, = P(c: = 1(0;), MMCE is computed on samples as following,
(a ri)ci -ry)) (r))k(r)rr)))
MMCE'(D) (D)
Σ
(19)
m
where k(r;, r3) is kernel function. MMCE is optimized together with the cross entropy loss as a regularization term. The strength of calibration can be adjusted by scale À E R.
[( = Σ
log + A (MMCE?(D))* 1
(20)
G SYNTHETIC DATA EXPERIMENTS
We use ResNet- 18 model for all our experiments with synthetic data.
The synthetic data is split into training, validation, and test sets with 16641, 4738, and 2329 samples, respectively. The training and validation sets contain only images x; and 0-1 labels yı for training and tuning the model. In order to evaluate the performance of the model for probability estimation, the held-out test set contains the ground truth probabilities Pt, in addition to 2; and ys- Note that we do not use the ground-truth probability labels p; values during training or inference we only use them to compare the performance of different models.
Ground Truth Probability Generation The ground truth probability associated with example i is simulated by p: (=2) where z; is age of the person.
22

---

Under review as a conference paper at ICLR 2022
Linear
Sigmoid
Centered
1.0
1.0
0.8
0.8
0.6
0.6
0.4
0.4
0.4
0.0
0.0
0.0
50
75
100
2
2
Skewed
Discrete
1.0
1.0
0.8
0.8
0.4
0.4
0.0
0.0
100
75
100
Figure 10: Illustration of the function (=)) used generate the different synthetic-data scenarios.
Label distribution After determining the probability p: using (=),, the label y: is sampled from a Bernoulli distribution parametrized by so that it takes the value 1 with probability The distributions of y, under five different scenarios illustrated in Fig I1
Linear
Sigmoid
Centered
800
1000
1000
0
600
800
800
Count
Count
Count
600
400
200
200
0
100
100
50
100
Age
Age
Age
Skewed
Discrete
1250
1000
1
0
1000
800
Count
Count
750
600
500
400
100
50
100
Age
Age
Figure 11: Histograms of the outcomes (y;) for the different synthetic-data scenarios.
23

---

Under review as a conference paper at ICLR 2022
H REAL-WORLD DATA AND EXPERIMENT DETAILS
We present here supplementary information for the real-world datasets used in our experiments.
Cancer Survival Histopathological features are useful in identification of tumor cells, cancer sub- types, and the stage and level of diffcrentiation of the cancer. Hematoxylin and Eosin -HtEe-stained slides are the most common type of histopathology data and the basis for decision making in the clin- ics. With these properties, H&E are used for mortality prediction of cancer (Wulezyn et al. 2020 In this experiment, we use the H&E slides of non-small cell lung cancers from The Cancer Genome Atlas Program (TCGA) to predict the 5-year survival. The dataset has 1512 whole slide images from 1009 patients, and 352 of them died in 5-years. We split the samples by patients and source institutions into training, validation, and test set, which has 1203, 151, and 158 samples respectively.
The whole slide images contain numerous pixels, so we cropped the slides into tiles at 20x mag- nification with 1/4 overlapping. resized them to 299 299 with bicubic interpolation, and filtered out the tiles with more than 85% area covered by the background. The representations of each tile are trained with self-super momentum contrastive learning (MoCo) (Chen et al. 2020 and the slide-level prediction is obtained from a multiple-instance learning network llse et al. 2018) trained with the binary label of survival in 5 years.
Weather Forecasting We use the German Weather service datase which contains quality- controlled rainfall-depth composites from 17 operational Doppler radars. Three precipitation maps from the past 30 minutes serve as input. The training labels are the 0/1 events indicating whether the mean precipitation increases (1) or not (0).
The German Weather service (DWD Deutshee Wetter Dienst) dataset https:
dwd. de / breatherrradaze contains quality-controlled rainfall-depth composites from 17 op- erational DWD) Doppler radars. It has spatial extent of 900x900 km, and covers the entirety of Germany. Data exists since 2006, with spatial and temporal resolution of Ixl km and 5 minutes, respectively. The dataset has been used to train RainNet, a pricipitation nowcasting model (Ayzel 2020).
The network architecture is ResNet18, with 3 input channels and output channels. The input to the network are 3 precipitation maps which cover a fixed area of 300km x 3300 km in the center of the grid (300x 300 pixels), set 10 minutes apart. The training, validation and test datasets consist of 20000, 6000 and 3000 samples, respectively, all separated temporally, over the span of 15 years.
Collision Prediction Vehicle collision is one of the leading causes of death in the world. Reliable collision prediction systems which can warn drivers about potential collisions can save significant number of lives. A standard way to design such a system is to train a convolutional model for identifying if a particular vehicle in the dash-cam video feed might collide with the car in next few seconds. More formally, at time t = T the system tries to predict if any car in the video might collide with our given car in time & [T, T Trook Each labelled training sample consists of XT and a binary label Y E {0, 1} denoting if an accident will occur in f E [T,T + ahoa- Each X is tensor with 4 channels where the first channels corresponds to an RGB image of the dashcam view at time & = f, and the fourth channel consists of mask with a bounding box on a particular vehicle of interest. In this work, we use YouTubeCrash
features X = (XT-5 XT-26
dataset Kim et al.) 2019) to train and test our model, which uses 8 = 0.1s,Tiook-ben. = 185 = 1.8s, and d 3. Following
Kim et al. (0199 we used a VGG- 16 network architecture.
The dataset contains 122 accident scenes, and 2096 non-accident scenes, which after feature extrac- tion gives us 2096 positive samples, and 11486 negative samples (the dataset is severely imbalanced, and similar to the Skewed situation in Section 6. We further split the dataset into train (6453 sam- ples for label O, and 1023 samples for label 1), validation (2348 samples for label O, and 545 samples for label 1), and test (2685 samples for label 0, and 528 samples for label 1) sets. The samples in train, validation and test sets are generated from disjoint scenes/dashcam videos.
https:/ de/ wwt.:/deeeeeeeeee/eeaaaaaeeeeeae
24

---

Under review as a conference paper at ICLR 2022
I
ANALYSIS OF CANCER SURVIVAL RESULTS
DeepErs CaPE (kernel)
1.0
Estimated Probability
0.8
0.6
0.4
0.2
I1/I
Pathological stage
Figure 12: Estimated probability of survival grouped by pathological stages. The plot shows median, samples between 25th 75th percentile in the box, samples between Oth and 100th percentile the line, and the outliers as dots. Deep ensemble produces similar probability estimates for patients across all the stages; CE is more discriminative but has very large variance; CaPE achieves trade-off between the two baselines.
For cancer survival prediction, we visualize the estimated probabilities on the test set in different pathological stages in Figure [2] general, patients in earlier stages should have higher prob- abilities of survival. Deep ensemble produces similar probability estimates for all stages (i.e the model is less discriminative). Cross-entropy minimization (CE) is more discriminative, but has very wide confidence intervals. CaPE is more discriminative than deep ensemble, while having narrower confidence intervals than CE.
CALIBRATING FROM THE BEGINNING
CaPE exploits a calibration-base cost function to improve its probability estimates without over- fitting. The empirical probabilities in this loss computed from the model itself. Consequently, applying this strategy from the beginning of training can be counterproductive, because the model predictions are essentially random. This is demonstrated in the following table, which compares CaPE with model trained using the calibration loss from the beginning (in the same way as CaPE, alternating with cross-entropy minimization).
Methods
Linear
Sigmoid MSE
Centered MSE,
Skewed MSE,
Discrete MSE,
(x10
MSE.
KL.p
KL.p
KLp
KL.p
KL.p
Bin (start)
2.59
6.81
8.07
22.10
0.48
0.98
0.51
2.37
2.74
6.36
Kernel (start)
2.23
5.68
7.60
21.15
0.54
1.10
0.68
2.84
2.40
5.63
Bin (CaPB)
4.46
5.29
14.59
0.38
0.78
0.40
1.72
4.31
Kernel (CaPE)
1.81
4.41
5.22
14.47
0.40
0.81
0.39
1.70
1.85
4.36
Table 1: Comparison between CaPE and a model that uses the calibration loss from the beginning (in the same way as CaPE, alternating with cross-cntropy minimization) on synthetic data.
25

---

Under review as a conference paper at ICLR 2022
K COMPARISON OF REAL-WORLD DATASETS WITH DIFFERENT SCENARIOS OF THE SIMULATED DATASET
Figure 13 illustrates the similarity between the empirical probability curves of different real-world datasets and the different scenarios of our synthetic dataset. For the cancer survival dataset, the empirical probabilities are clustered in the center (0.4-0.6) similar to the Centered scenario. For the weather forecasting dataset, the probabilities are uniformly distributed across 0. similar to the Linear scenario. For the collision prediction dataset, the majority of the data points are clustered in the lower probability region which makes it similar to the Skewed scenario.
26

---

Under review as a conference paper at ICLR 2022
Cancer Survival
Centered
1.0
CE Earty-soap
Empirical probability
Ensemble
Empirical probability
CaPE (kernel)
0.6
0.
0.0
0.0
0.2
0.4
3.6
0.8
1.0
15
Predicted probability
Predicted probability
Weather Forecast
Linear
1.
CE Early-scap
Empirical probability
Deep Ensemble
Empirical probability
CaPE (kernel)
0.6
0.4
Predicted probability
Predicted probability
Collision Prediction
Skewed
1.0
Early-stop
Empirical probability
MMCE
Empirical probability
(kemel)
0.9
0.4
0.2
0.0
Predicted probability
Predicted probability
Figure 13: Comparison reliability diagrams for real-world data with different scenarios of simulated data. For the cancer survival dataset, the empirical probabilities are clustered in around (0.4-0.6), similar to the Centered scenario. For the weather forecasting dataset, the probabilities uniformly distributed across 0.1-0.8, similar to the Linear scenario. For the collision prediction dataset, the majority of the output probabilities are clustered in the lower probability region, similar to the Skewed scenario.
27


Reviews:
[{"review_id": "ZDTLMa4sYXb", "reviewer": "Reviewer_QWNy", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The idea is quite straightforward (which is not bad) and seems to be working well, as the authors compare their method to a number of baselines and show it is competitive. The idea is to a degree non-intuitive (at least to me a bit), but it does show good results. The writing is confusing though, and the authors could have done a much better job on that side. Please see details comments below.", "summary_of_the_paper": "The authors propose a method to ensure calibration of probability outputs during training. They do so by adding an explicit penalty to push the output probability values towards an empirical estimate of the actual probability for the current inferred probability. They compare the method to a number of baseline approaches, showing the benefits on both synthetic and real-world data sets.", "summary_of_the_review": "Detailed comments:\n- \"classification problems without inherent uncertainty\", I am not sure that such problem exist. Please rephrase or elaborate further.\n- Minor: make sure that all notation is correct (e.g., scalars are not bolded), both in the figures and in the text.\n- \"of just 75%\", can you please clarify how?\n- It is not fully clear why MSE_p is declared gold-standard, please clarify why.\n- For Fig 4 results, more details are needed. E.g., early stopping is not well explained.\n- I suggest moving Fig 5 much later, it seems introduced too early.\n- For the kernel function, it is not clear whether the r-window is in the input or output space.\n- In Section 6, uniform is not really uniform? Since it still increases with z.\n- It is unclear which models were actually used for different data sets. This should be explained to a greater detail.\n- In the last paragraph in Section 6, please explain better how each scenario aligns with the data sets, it is unclear as is. In fact, currently the entire section feels hand-wavy.\n- Also, the last sentence doesn’t parse well, please rephrase.\n- I recommend removing the last sentence in the Conclusion section.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "BSd2yVUxPqQ", "reviewer": "Reviewer_r2Pt", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The main message of the paper is not clear. What is essentially the mathematical definition of probability estimation? All definitions in Eqs 1 - 6 are well-established terms in the uncertainty calibration literature and the phenomenon in Fig 2 is also well known to the community. It is accounted for simply as reporting prediction accuracy and calibration scores together. I am then missing what is new thing the \"probability estimation\" notion is telling us and where we see it in the experiments, which are \ndesigned simply as standard calibration experiments.\n\nI am having hard time to make sense of the calibration loss presented as an alternative to the discrimination loss. Where does p_emp^i come from? How can one define it without giving reference to a sample-based estimation of the confidence score of f(x), which eventually boils down to the calibration we know? Having given a reference to a confidence score estimate, how is it different then from ordinary calibration?\n\nThe \"function kernel\" inside Algorithm 1 appears to be the only part where one can argue about a methodological novelty. This function only replaces the Dirac measure used by standard binning with a kernel density estimator. This is a standard trick to improve numerical stability, but it is hard to say that this much makes a scientific novelty worthwhile to be published as a main-track conference paper.\n\nThe paper lacks a clear focus in its core theme and a consistent logical flow. Is the main message how good Algorithm 1 is? Then the justification of the algorithm and explanation of why it works in the way it works is missing. Is it the allegedly new applications where uncertainty calibration is used? Then references to the closest work are missing. I also do not think these are such novel applications of uncertainty calibration. For instance see:\n\nThagaard et al., Can you trust predictive uncertainty under real dataset shifts in digital pathology?, MICCAI 2020", "summary_of_the_paper": "The paper studies the problem of uncertainty quantification in deep neural nets. It introduces a concept called \"probability estimation\" and an uncertainty calibration method called \"CaPE\" based on this new concept. The studies uncertainty calibration in a few new data sets such a histopathology cancer diagnostics.", "summary_of_the_review": "The paper lacks a clear problem statement, sufficient technical novelty, and a consistent logical flow. Under these conditions, I am not able to recommend an accept. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "Ammv_LWxUDS", "reviewer": "Reviewer_vz2e", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The definition of of ECE and MCE is a special case (for binary classification) of the more general one that handles multi-class classification. Mathematically the definition is accurate, but overly obscure and complicated. I suggest the authors refer to the definition from Guo et al., 2017. Essentially, ECE calculate the difference between confidence and accuracy in each bin. It is therefore a bit confusing to call the accuracy term p_emp. \n\nIn term of methodology, it makes sense to alternate between the two losses. However, it seems CaPE uses the training set to compute p_emp. Would p_emp itself run into the problem of over-confidence? For example, a well trained neural network (NN) is always over-confident and therefore most predictions (probability) will concentrate around 1, meaning that most bins will have nearly zero data point. Could you provide more intuition or theoretical analysis on why over-confidence would not happen? \n\nWhy is the discrimination loss only one-sided? In other words, what happens if the normal cross-entropy loss is used? \n\nThis brings me to another concern which is the extension to multi-class classification. The current discrimination loss will not work probably when there are more classes since it only focuses on the single correct class. Therefore, I would expect that the performance will get worse and worse as the number of classes increases.\n\nThe plots in Figure 5 show the benefit of CaPE. However, the drop of training loss and increase of validation loss are quite steep for CE. In practice, this rarely happens for high dimensional data, especially if the network architecture is well chosen. \n\nThe baselines seem comprehensive. Are there results for ECE/MCE/Brier Score for Table 1?\n\nIn Table 2, it seems CaPE is worse than Deep Ensemble and MMCE Reg. in Collision Prediction and worse than Deep Ensemble  in Weather Forecasting (in terms of AUC). I was wondering how many models are used for Deep Ensemble, since the performance of Deep Ensemble is also related to the number of individual models. \n\n\n\nMinor:\nThere are two entries for the same reference Gupta et al., 2021.\n\nSome related work on probabilistic neural network for uncertainty estimation is missing [a,b,c]. \n\n[a] Natural-Parameter Networks: A Class of Probabilistic Neural Networks, NIPS 2016\n[b] Feed-forward Propagation in Probabilistic Neural Networks with Categorical and Max Layers, ICLR 2018\n[c] Sampling-free Epistemic Uncertainty Estimation Using Approximated Variance Propagation, CVPR 2019\n", "summary_of_the_paper": "The paper tackles the problem of uncertainty calibration in the special case of binary classification. The authors propose the so-called CaPE to iteratively minimize the discrimination loss and the calibration loss. ", "summary_of_the_review": "Overall, the authors proposed a simple and effective method for improving the uncertainty estimation for binary classification. Several drawbacks include: the reason why training directly on the training will not lead to over-confidence is unclear; performance is worse than some baselines in some dataset; the method seems limited to binary classification and it would be interesting to see at least its performance on multi-class classification. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "619sH2a2zfU", "reviewer": "Reviewer_Lp12", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The authors address an interesting problem. However, it would be interesting to further justify why current models do not offer an accurate estimate of uncertainty, particularly bayesian modelling. It is also important to note that calibration is a key issue in the survival literature and penalties have been developed to tackle this issue [1]. \n\nAn interesting point is the decomposition of the Brier loss, which would benefit from further descriptions.\n\nFinally, it is interesting that the authors use their loss only as a fine-tuning of the neural network. From figure 5, it seems that the introduced loss does not allow to reach a lower minimum but stops the training of the neural network. It would be interesting to see what is the effect of training the neural network from the start with this given loss.\n\n[1] Lee, C., Zame, W.R., Yoon, J. and van der Schaar, M., 2018, April. Deephit: A deep learning approach to survival analysis with competing risks. In Thirty-second AAAI conference on artificial intelligence.", "summary_of_the_paper": "This work tackles the problem of probability estimation. Current machine learning models do not fully reflect the uncertainty of the outcome but only of the model. The authors propose a loss that enforces both calibration and discrimination. Additionally, they present a semi-synthetic dataset for further study of this problem: from an image dataset of faces with associated age, they estimate the risk of developing a given disease.", "summary_of_the_review": "The work is clear and concise. The authors propose to tackle an interesting problem. However, it would benefit from further justifications of the utility and contextualisation with other domains of machine learning.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper addresses the problem of uncertainty quantification in deep neural nets. The authors introduces the CaPE calibration loss to deal with the inherent uncertainty in probabilistic prediction, e.g. medical prognosis, weather prediction or collision prediction.

The paper initially received contrasted reviews: two weak acceptance, one weak rejection, and one strong rejection recommendation. The main limitation pointed out by reviewers related to the unclear definition of the problem setting, the limited contributions, and clarifications on experiments (comparison with deep ensembles). After authors' feedback, the reviewers were not convinced by the clarification on the problem setting, and there was a consensus among reviewers to reject the paper. 

The AC's own readings confirmed the concerns raised by the reviewers, and also identifies additional shortcomings of the current submission. The paper addresses the problem of proper quantification of data uncertainty (generally referred as aleatoric uncertainty), and the CaPE calibration loss should be positioned with respect to the literature on the topic. The AC thus recommends rejection, but encourages the authors to re-submit their work after specifying the focus and motivation of their work.
Decision: Reject