Empirical confidence estimates for classification by deep neural networks
Authors: Chris Finlay
Adam M. Oberman
Venue:  (2020)
Keywords: confidence
classification
uncertainty
anomaly
robustness

Abstract: How well can we estimate the probability that the classification predicted by a deep neural network is correct (or in the Top 5)?  It is well-known that the softmax values of the network are not estimates of the probabilities of class labels.  However, there is a misconception that these values are not informative.  We define the notion of implied loss and prove that if an uncertainty measure is an implied loss, then low uncertainty means high probability of correct (or Top-k) classification on the test set.   We demonstrate empirically that these values can be used to measure the confidence that the classification is correct.  Our method is simple to use on existing networks: we proposed confidence measures for Top-k which can be evaluated by binning values on the test set. 

Under review as a conference paper at ICLR 2020
EMPIRICAL CONFIDENCE ESTIMATES FOR CLASSIFICA- TION BY DEEP NEURAL NETWORKS
Anonymous authors Paper under double-blind review
ABSTRACT
How well can we estimate the probability that the classification predicted by a deep neural network is correct (or in the Top 5)? well-known that the softmax values of the network not estimates of the probabilities of class labels. However, there is a misconception that these values are not informative. We define the notion of implied loss and prove that if uncertainty measure is an implied loss, then low uncertainty means high probability correct (or top k) classification on the test set. We demonstrate empirically that these values can be used to measure the confidence that the classification is correct. Our method is simple to use on existing networks: proposed confidence measures for Top l which can be evaluated by binning values on the test set.
INTRODUCTION
Despite lots of effort build confidence measures for classification by deep neural networks, there is still lot of confusion about the value and applicability of these measures. In this article we present simple method for estimating confidence based on implied loss values which leads to results which are empirically more accurate than benchmarks on test sets. We prove that high confidence values imply high probability of correct classification on test sets.
Many have observed that used blindly, the maximum softmax probability of network does a poor job of predicting uncertainty (Nguyen & O' Coonnor, 2015; Provost et al., 1998; Nguyen et al., 2015; Yu et al., 2011; Lakshminarayanan et al., 2017). However, Zaragoza & d' Alché Buc (1998) showed in the 1990s that on on shallow networks the maximum softmax probability and the (negative) entropy of the probabilities strongly correlate with model confidence in-distribution images. More recently, in the deep setting, Hendrycks & Gimpel (2017) showed empirically that the maximum softmax probability can be used to predict network confidence. We demonstrate that both the maximum softmax and the model entropy are uncertainly measures. We extend the uncertainty metric to Top k predictions. We show that, in conjunction with binning, simple uncertainty statistics outperform common approaches like MC-dropout as a measure of confidence, at a fraction of the computational cost.
Using this simple idea, we make the following contributions.
1. We estimate the probability that the classification of the model on a test set is correct which works for existing models (no need retrain), using simple tabular form (see Table 2 for Imagenet).
2. We give a simple definition of uncertainty, which applies to previously proposed methods, and leads to a proof that low uncertainty (high confidence) implies high probability of correct classification. It applies to both Top and Top & uncertainty.
3. We can discover mislabelled data, Figure ((c) and we can detect off manifold data and adversarial examples.
We advocate evaluating model uncertainty via expected Bayes factors (Kass & Raftery, 1995), which provide rigorous probabilistic approach to evaluating uncertainty, and are widely used for hypothesis testing in other scientific fields, see for example (Good, 1979) and (Jeffreys, 2003). Compared to other methods (such as AUROC Brier scores) Bayes factors better distinguish improvements

---

Under review as a conference paper at ICLR 2020
confidence for methods which are already quite accurate, is the case for top 1 or top uncertainty for image classification.
2
PRIOR WORK
As neural networks are adopted into safety critical systems, the need for neural network uncertainty estimates has become abundantly clear. Indeed, any accident adverse system must by design incorpo- rate notions of uncertainty (Amodei et al., 2016). Real-world examples abound: uncertainty measures needed in autonomous vehicles (Feng et al., 2018), robotics (Richter & Roy, 2017), medical imaging (Ching et al., 2018; DeVries & Taylor, 2018b) and medical decision making (Begoli et al., 2019), and semantic understanding (Kendall al., 2017).
Much effort has been dedicated to addressing this deficiency. Many works have placed neural networks within a Bayesian probabilistic framework. Initial work placed Bayesian priors model weights (MacKay, 1992a; Neal, 1996), leading to Bayesian neural networks, however this has proven difficult implement in practice. Many techniques have been developed to overcome this difficulty (MacKay, 1992b; Neal, 1996; Graves, 2011 Hasenclever et al., 2017: Li et al., 2015; Balan et al., 2015; Welling & Teh, 2011; Springenberg et al., 2016). One promising approach in the deep learning setting is to perform approximate posterior inference (Louizos & Welling, 2016; Hernández-Lobal & Adams, 2015; Blundell al., 2015; Sun al., 2017).
Due to its simplicity, dropout is widely used as surrogate for uncertainty. Dropout (Srivastava et al., 2014) was interpreted Bayesian setting by Gal & Ghahramani (2016) and Kingma al. (2015), however, there are problems with this interpretation, see (Hron et al., 201 8) for recent discussion. Dropout involves evaluating an ensemble of models at test time, which can be both memory and computationally intensive for very large networks.
Non-Bayesian model ensembles have also been developed (Dietterich, 2000), for a recent survey see (Li et al., 2018). Lakshminarayanan et al. (2017) train an ensemble of adversarially robust models and empirically showed an improvement in uncertainty estimates over dropout based methods. Geifman et al. (2018) () proposed using an early stopping criteria to collate ensemble of models. Kristiadi & Fischer (2019) use mixture modeling to chose ensemble weights.
Several deep learning specific approaches have been proposed in recent years, especially in the context of detecting out-of- dittr--distribttion samples. Oliveira et al. (2016) suggest detecting outliers via an anomaly detector. Lee et al. (2018) generator -iiitii-iitiiittinnnn images through GAN; the classifier is trained to assign the equal weight probability vector to these images. Hendrycks al. (2018) train networks on two distributions: the in-distribution samples, and -ut-o--distribution samples. Liu et al. (2018) develop PAC-style guarantees on detection of out-of-diisributtin samples. Several recent works (Jiang et al., 2018: Papernot& & McDaniel, 2018; Mandelbaum & Weinshall, 2017) have suggested using nearest neighbour distances, in feature space, for outlier detection and confidence measures. DeVries Taylor (2018a) suggest training an additional network to predict uncertainty; Malinin & Gales (2018) specifically model prediction probabilities with a Dirichlet distribution, which implicitly describes model uncertainty.
Platt (2000) proposed scaling SVM predictions to better match the validation set; this has been generalized to neural networks and multiclass classification (Niculescu-Mizil & Caruana, 2005; Guo et al., 2017). Other scaling approaches, such as changing the softmax temperature, have shown promise (Guo et al., 2017; Liang et al., 2018). Another popular approach to calibration is based binning model probabilities, developed by Zadrozny & Elkan (2001). Each bin is assigned probability of being correct, which is obtained by minimizing the Brier score of the bins (Brier, 1950). Bins edges may be optimized well (Zadrozny & Elkan, 2002); and can be extended to the Bayesian setting by assigning prior on binning schemes (Naeini et al., 2015).
3 CONFIDENCE MEASURES AND UNCERTAINTY ESTIMATES
Suppose a model f(x), generalizes well, so that it has a high probability, p. of a correct prediction on image sampled from the same underlying distribution. Write I&()) (indices of the & largest components of f} for the top & indices. The classification of the vector f is given by the largest component, C(J) = (f). Define the random variables X₃ (000)))
2

---

Under review as a conference paper at ICLR 2020
to be the Bernuulii random variables with expected value pk: = E [X the probability that the correct label in the Top k. Our goal is to estimate Pk- We do this by defining random variables, Ug. which we call uncertainties, whose statistics allow to better estimate Ps. The histogram of the uncertainty variables will result in an estimate of the conditional probability that the classification is correct, given the uncertainty value,
Prob (xx(x) = 1 (x(r) = =t).
Definition 3.1. Given > O, and the uncertainty measure (zxx), define the set
S = {Ux(=)= S c and y & [x))
(1)
The uncertainty measure Ux(x)) is an implied loss if the event St has high expected loss. Example 3.2. For the Kullback-Leibler loss, the (negative) entropy of the probabilities is an uncer- lainty measure. Write fsort for the indices of f sorted in decreasing order. Define
(1) = - -log(/fnr)),
Uz(2) = - log Σ J
Example 3.3. For general losses, f, the top 1 uncertainty can be given by (1) (C(f(z),)) v) = C(f())), which is the loss, given that the classification was correct. Also Ux(z) L(f, yw) is implied loss, for top k, where yo is the (Is + 1)-th ranked label.
3.1 ILLUSTRATION OF UNCERTAINTY RANDOM VARIABLES
The histogram of U1 is plotted on the test set in Figure ((a). Note that for small values of U1. the images have a very high probability of being correct. In fact, we can use U1 to detect incorrectly classified images: we visualized the images which smallest value of U1 (i.e. highest confidence), which correspond to the few isolated points in the upper left of the figure. turned out that all of these were either incorrectly labelled, were ambiguous images, For example, in the second image, the animal is a wallaby, not a wombat. In the fourth image, a paintbrush kind plant, but there also a pot the image. In the second part Figure 1(b) we illustrate more quantitatively the Top 1 (green) and Top 5 (green or blue) probabilities conditioned on the 100 histograms bins of (Pmax) on test set for ResNet152 on ImageNet. The Top probability conditioned on the lower bins is very close to 100%. The Top 5 probability is no better than 50% on the last few bins. The intermediate bins are less informative.
illustrations in Figure (c).
3.2 TOP UNCERTAINTY ESTIMATES
The next theorem shows that if the uncertainty is small, then the probability of correct classification must be high.
Theorem 3.4 (Confidence estimate). Define (1(3) by (2) and define S by (1), and let LKL be the Selbback-Leibler loss. Then
Prob (5*) < E [[(KI((()),)) log (2)
(3)
Proof. Claim: Let > 0 be small. By assumption, - Sport S Thus srort 2 exp(-r) Let
be the correct label. Then fk poort so
1 1 x1-exp(-e) exp(-e)
and
- log(ff)) 2 - log((1 exp(xe)()) 2 log(1/e).
Thus for x. E St, (f)) y(x)) 2 log(1/c). Apply Markov's inequality (13) to the random variable L(x) ((f(x) to obtain the result.
Remark 3.5 (Neural Networks are always overconfident). Note that the uncertainly is always less than the loss,
Ui(f) S CkI(f.e))
with equality when C((()) v(x).
3

---

Under review as a conference paper at ICLR 2020
1.00
100
P(Correcti
0.75
top5 mis-classified topl
Loss
10-2
0.50
10-4
0.25
10
0.00 4
10-6
10-3
100
10-2
100
(aa)(fa))
-log(fa))
(a) Loss vs U1
(b) Histogram
Prediction:
Bearskin Assault Rifle
Wallaby Wombat
School
Pot Paintbrush
Ping pong ball Beaker
Baseball Bucket
Label:
Minibus
(c) mislabelled or ambiguous images found using U1-
Figure 1: Figure ((a) Scatter plot to indicate how predictive U1 is compared to the loss. For small values of U1, the loss is small with high probability. Figure 1(b): the probability correct (green) or Top5 (blue) given the value of UT. Figure Icc): visualization of the images in the upper left of Figure (a). The confident images which were labelled incorrectly turned be mislabelled or ambiguous.
3.3 TOP UNCERTAINTY ESTIMATE
In the next result we show that if the top i uncertainty is small, then the probability that the correct labels is in the top & must be high. The result can also be proven in the case of general losses, and uncertainly measures satisfying (1).
Consider the event S (1) for a given 2 If the correct label in the top k; then the probability of the correct label, fe. must satisfy
fe S f8tt
with
saort < 1 offort + + foort)
Thus
CKI(f,ee) 2 - log(1 (ffort +.... fFort)) Then, by an argument similar the for Top error,
that
E [X+]
Prob (S%) < log ())
(5)
4 EMPIRICAL RESULTS
The previous section proved that, under fairly general conditions, we can define uncertainty measure which ensure that the top & classification is correct with high probability. The theory applies to uncertainties used the literature, such as the negative entropy of the probabilities, and negative log softmax.
In practice, once we have an uncertainty measure, the method is simple
1. Compute the statistics on the test set of the uncertainty estimates.
2. Divide the test set into bins, based on uncertainty values.
3. Estimate the conditional probabilities based the bin populations.
4

---

Under review as a conference paper at ICLR 2020
4.1 THE BAYES FACTOR
The Bayes factor is way to measure the value of new information, in terms of how much the expected winnings of fair bet increase, when the information is available. Unlike other measures of confidence, which are additive, the Bayes factor is multiplicative. On a model which is correct 95% of the time, there still a lot of value in knowing when the probability correct increases to 99.5%. In this case, the Bayes factor is close to 10. On the other hand, going from 50% to 54.5% gives Bayes factor close to 1.2. On the other hand, additive scoring methods give equal weight both improvements. As example, we show in Table 5 that the Brier scores of eight different measures of confidence lie close together, between .033 and .076. On the other hand, the expected Bayes factors range more widely, from 1.3 to 16.6.
PX Consider a Bernoulli random variable = B(px). The odds for X are given by O(px) I-Px Now consider a test, Y = BBpy), ) for which
px,y Prob (X - 1x=1|Y=]] 1)
Then the odds, given the test succeeds, are O(px,r) = Px.v I-Rx,y" If the odds have increased, we define the Bayes Factor to be BF(X Y) O(px,r)
Olpx). On the other hand. if the odds have decreased, then the value of the information provided by Y is bet against, so define the Bayes factor to be O(px). Combining these possibilities, define the Bayes factor by
(F Y)= O(px,y]
BF(X IY) max (Olxxr) 0(px) (Olxx) O
4.2 EXPECTED BAYES FACTOR
We can also define a metric for measuring the quality of an uncertainty random variable, such as the value the loss (or another random variable) for predicting the probability correct classification. In practice, we will define the tests based on bin values for some uncertainty variable. If we have more data, we can use more bins.
Definition 4.1 (Histogram random variables). Given random variable (x)) E [a, b] and a partition of [a, bj into bins
to < ... < to = b,
Define the histogram (or bin) random variables B, corresponding to each interval
B,(x) = {
4-1 U((x) < t
(8)
otherwise
so that
Prob (4:-1 < = [B,
Each Bayes factor measures the value of information that x lies in each quantile. The value of the test itself is defined be the expected value of the Bayes factors.
Definition 4.2 (Histogram Bayes Factors). Given X, U and the histogram random variables B, define the conditional probabilities
Px,: Prob (X 1 B, = 1), i=
(10)
The predictive value for X of the random variable U with respect to the histogram, is given by
Q E [BF X) = Σ BF(X B)
(11)
1=1
When the number of bins is large, we defined the bins to be quantiles, that they have equal number of examples in each bin. In addition, in order to make the histogram Bayes factors finite, we require that each bin have least one correct and one incorrect example.
See Appendix C for a detailed example of the expected Bayes factor. Next we present an example based on actual confidence measures for networks.
5

---

Under review as a conference paper at ICLR 2020
Table I: Bayes factor [R against various measures of confidence. For CIFAR-10 we used X1, the probability of the correct label; for CIFAR- 100 and ImageNet-1K used X5 the probability that the correct label is in the Top5. Data binned into 100 bins, chosen to have equal weight.
Confidence measure
CIFAR-10
CIFAR-100
ImageNet-1K
Model Entropy
4.29
3.64
8.18
log Pmax
4.22
3.77
8.87
log > P1:5
4.25
8.45
V x [p||
8.32
3.47
7.17
Dropout variance (p = 0.002)
10.39
3.11
6.84
Dropout variance (p 0.01)
4.67
2.38
7.81
Dropout variance (p = 0.05)
1.69
1.35
1.60
Ensemble variance
16.66
4.03
6.13
Loss
00
228.94
1242.55
Example 4.3. This example follows closely the confidence bins for top 5 on 1n, using the Model Entropy, as in the first row of Table 2. Consider model with px .94. In this the odds 94 to 6, so O(px)
15.6. Define three bins for Model Entropy with bin edges 0.31, .40.. Then with probability ,55, data is the first bin, in which case the probability correct is .99. So the Bayes factor in this bin is (.99/.90 .//.))94(006)) (.06/.0))) = 6.3. Thus knowing the Model Entropy is less than .31 tell you that you are 6.3 times more likely to be correct than on average. The second bin consists of data with with Model Entropy between .31 and .14, which occurs with probability .31, in this case, the Bayes Factor (95 (6/5)) = 1.2 is nearly one, so there is little additional value to knowing data is in this bin. Finally, when the Model Entropy is greater than .14, which occurs with probability .14, the probability correct is only 88. In this case, the relative probability to correct is worse, to the Bayes Factor given by (5/80)/66/200 Bayes factor of each bin, weighted by the probability of the bins
3.9. The expected Bayes factor is the weighted average of the
E [[F(X B,) 6.3 x +5 1.2 x 31 3.9 x 14 4.4
So the expected value of the Model Entropy, for the chosen bins, is 4.4. By fine graining the bins we capture relatively small and relatively large values of the Model Entropy which have Bayes factors on the order of 20, see Figure 5. Thus the expected Bayes factor with 100 bins is 8.18, as shown in Table 1.
4.3 CONFIDENCE BINS AND BAYES FACTOR
In this section we present confidence bins for ImageNet-IK. IK. These bins are concise summaries of the information presented in the larger bins. Table presents short bins for ImageNet. Using these bins, we can simply read of from the Uncertainty values, the probability that the model is correct. For example, on the model, P(top 5) = 0.9406, however, using entropy, 55% of the images had entropy low enough to be confidently classified with probability .99. Using Us. 66% of images could be binned to have probability .99.
Bins for CIFAR-10 and CIFAR-100 are given in Tables 7 and 6, respectively.
In Table we show the expected Bayes factor for various confidence measures, on CIFAR-10, CIFAR- 100, and ImageNet- IK. In addition to the confidence measures already discussed, considered Bayesian dropout, and the norm of the gradient the model. Larger expected Bayes factors means the information is more valuable.
In Figure 5 we plot the regularized Bayes factor for our two main measure confidence, U1 and Us along with the loss and the model Entropy. The entropy and U1. Us have very large Bayes factor in the first 10 and last 3 bins, meaning that for these bins, the prediction is 10X (or more) likely to be correct (for the first 10) wrong (for the last bins) than average.

---

Under review as a conference paper at ICLR 2020
Table Confidence bins for ImageNet-1k K. The values of a and b are chosen such that P(top5
Y a) 0.99 and P(a < top5 Y < b) 0.95. For the model used here, P(top5) 0.9406.
Confidence measure Y
(a,
P(Y <a)
P(a S Y < 6) P(Y 2 6)
Model Entropy
(0.31, 1.40)
0.55
0.31
0.14
llog Pmax
(0.047, 0.41)
0.52
0.26
0.22
log 2 P1:5
(6.2e-3, 0.03)
0.66
0.13
0,21
IV |Il||
(0.19, 0.30)
0.52
0.08
0.40
Dropout variance (p 0.002)
(8.5e-4) 4.7e-3)
0.50
0.15
0.35
Ensemble variance
(0.014, 0.023)
0.54
0.05
0,41
0.20
0.20
frequency
0.15
frequency
0.15
mis-classified
top5
0.10
0.10
topl
0.05
0.05
0.00
0.00
10
10-3
100
10
6
10-3
100
Model Entropy
Model Entropy
(a) COCO images
(b) ImageNet images
Figure Figure 2(a): Confidence of model trained on ImageNet-Ie 1k, evaluated on the COCO dataset. Figure 2(b): ImageNet images.
EXTENSIONS
In this section we discuss some extensions of the confidence results. We show that we can detect mislabeled images in the test set. We also show that we can obtain some confidence results for off manifold images, as well as adversarial images.
5.1 DETECTION OF MISLABELED IMAGES
We are able detect test images which are mis-labeled: images which the network correctly classified, but who's label is incorrect, or for which multiple labels could apply. These are images with high loss but low model entropy. For example Figure ((c) we show six images from the ImageNet- lk test set who's 's predictions where not in the top5, but had low model entropy. All six of these images either have an incorrect dataset label, or could described by multiple labels. For example, in the second image, the animal is a wallaby, not wombat. In the fourth image, paintbrush is kind of plant, but there also a pot in the image.
5.2 CONFIDENCE ON OUT-OF-DISTRIBUE DISTRIBUTION AND ADVERSARIAL IMAGES
Next we studied whether could detect cnt-of-distribution images generated by COCO. In Figure we show how the histogram of the model entropy shifted to the right compared to the on-distribution images. Table give the results of our test: choosing confidence measure which rejects 10% of the on-distribution images, our confidence measures rejected as much as 38% of COCO images (for Entropy) with similar values for U1 Us. On the other hand Dropout was completely ineffective.
7

---

Under review as a conference paper at ICLR 2020
Table Discarding out-of-distribution images from ImageNer-1K, For each confidence measure Y, the value of a is chosen such that P(Y S image is from ImageNet- 1k) = 0.9.
Image source
Confidence measure
a
P(image discarded)
Model Entropy
1.75
0.38
log Pmnx
0.77
0.34
COCO
log > P1:5
0.13
0.37
v |pll|
1.06
0.23
Dropout variance (p 0.002)
0.024
0.
Model Entropy
1.75
0.28
adversarially perturbed (I2)
log Pmax log Σ P1:5
0.77
0,25
0.13
0.28
V |pll|
1.06
0.58
Dropout variance (p 0.002)
0.024
0.39
Table 4: Adversarial detection with ResNeXt-34 34 (2x32) on CIFAR-10. Clean images which the model correctly labels are perturbed until they are misclassified with four attack methods (PGD, Boundary attack, Carlini-Ve Carlini-Wagner, and an evasive Carlini- Wagner designed to avoid detection). Images are rejected if V (x) 2,00 > 2.45.
clean
PGD
Boundary
CW
evasive CW
percent detected median &
6%
96%
100%
100%
22%
0.31
0.36
0.34
0.81
CONCLUSIONS
With the goal of using measures such model entropy a surrogate for the (unknown) model loss, we defined confidence measures as random variables which are large when the loss is large. Using this definition we proved that confidence variables can be used estimate the probability that model low expected loss makes correct prediction. In practical terms, this amounts to defining confidence measure (such as model entropy or log pmax) and binning the values.
We presented the expected Bayes factor as an effective measure of confidence. Since models are already very accurate, it is important measure the relative confidence. The Bayes factor multiplicative factor to the probability (or odds) that model is correct. For example, showing the that model entropy on ImageNet with 100 bins is means that knowledge of the model entropy (and the bayes factors for the bins) allows us to predict the probability that the model is correct 8 times more effectively.
The Bayes factors was used compare existing confidence measures on different tasks. The main task was estimating the probability a correct prediction on the images from the data set. Additional tasks included: detection of off manifold data, detection of adversarial examples, and detection of mislabelled images. The latter were found by searching for highly confident predictions which were labelled incorrect.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Dario Amodei, Chris Olah, Jacob Steinhardt, Paul F. Christiano, John Schulman, and Dan Mané. Concrete Problems in AI Safety. CoRR, 0bs/1606.06665 2016. URL http://arxiv.org/ 00ee/666..0665.
Anoop Korattikara Balan, Vivek Rathod, Kevin P. Murphy, and Max Welling, Bayesian dark knowledge. Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada, -aaeeee-bbaeaaaaaaaaa
pp. -438-344,, 2015. URL UtL.htpp://ppgrrsnii..cc/ knowledge.
Edmon Begoli, Tanmoy Bhattacharya, and Dimitri Kusnezov. The need for uncertainty quantification in machine-asse assisted medical decision making. Nature Machine Intelligence, 1(1):20, January 2019. ISSN 2522-5839. doi: 10. 1..1018/42256-018-0000
URL https: /www.natuue..omm
articles/s4225 018 -20-02220000000---11
Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. Weight uncer- tainty in neural networks. CoRR, abs/1505.00524, 2015. URL http:::/axxi...ogg////////
1505.05424.
Glenn W Brier. Verification of forecasts expressed in terms of probability, Monthly Weather Review, 78(1):1-3, January 1950. doi: 10.
Travers Ching, Daniel S Himmelstein, Brett K Beaulieu-Jones, Alexandr A Kalinin, Brian T Do. Gregory P Way, Enrico Ferrero, Paul-Michael Agapow, Michael Zietz, Michael M Hoffman, and others. Opportunities and obstacles for deep learning in biology and medicine. Journal of The Royal Society Interface, (:(141):2011) 70387, 2018.
Terrance DeVries and Graham W. Taylor. Learning Confidence for Out-of-Distribution Detec- tion in Neural Networks. CoRR, lbs//802.04885, 2018a. URL http::/aaxiivozz///////////// 1802.04865.
Terrance Veeries and Graham W. Taylor. Leveraging Uncertainty Estimates for Predicting Segmenta- tion Quality. CoRR, bbs//807.00000. 2018b. URL
Thomas G. Dietterich. Ensemble methods in machine learning. In Multiple Classifier Systems, First International Workshop, MCS 2000, Cagliari, Italy, June 21-23, 2000, Proceedings. pp. 1-15, 2000. doi: 10.1007/3-540-45500 - 1 URL https:
Di Feng. Lars Rosenbaum, and Klaus Dietmayer. Towards Safe Autonomous Driving: Capture Uncertainty in the Deep Neural Network For Lidar 3d Vehicle Detection. 21xt International Conference
Intelligent Transportation Systems, ITSC 2018. Maui, HI, USA. November 4-7, 2018, pp. 0266-3273. 2018. doi: 10. 10.1109/TSC..2018855565 10.1 04.1094 10910099000 ITSC.201 TTT.T.........9111111 8.8569814.
URL https:/ /doi.org/
Yarin Gal and Zoubin Ghahramani. Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning. In Proceedings of the 33nd International Conference on Machine Learning, ICMI. 2016, New York City, NY, USA, June 19-24, 2016, pp. 1050-1059, 2016. URL http:// http://proceeee http///oooddi/eo// ngs.ml www.smlr.press. r.press/ / v /ga 6.html.
Yonatan Geifman, Guy Uziel, and Ran El-Yaniv. Bias-Reduced Uncertainty Estimation for Deep Neural Classifiers. September 2018. URL httpps:///peenneeeee..ttttt
SJfb5jjCa Km
Irving J Good. Studies in the history of probability and statistics. XXXVII AM Turing's statistical work in world war ii Biometrika, pp. 393-396, 1979.
Alex Graves. Practical variational inference for neural networks. In Advances in Neural Information Processing Systems 24: 25th Annual Conference on Neural Information Pro- cessing Systems 2011. Proceedings of meeting held 12-14 December 2011, Granada, Spain., pp. 2348 2356, 2011. URL /tapatttp//pppppss.ii..... variational in for- - -ntworrks...

---

Under review as a conference paper at ICLR 2020
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On Calibration of Modern Neural Networks. In Proceedings of the 34th International Conference on Machine Learn- ing, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, pp. 1321-1330, 2017. URL ngs.m www.gngw..............e
r.press/ sviees/vv00. https//s///////////////o/ /t//guolloo ga..7a..........
Leonard Hasenclever, Stefan Webb, Thibaut Lienart, Sebastian Vollmer, Balaji Lakshminarayanan, Charles Blundell, and Yee Whye Teh. Distributed bayesian learning with stochastic natural gradient expectation propagation and the posterior server. Journal of Machine Learning Research, 18:
106:1-106:37, 2017. URL http: /atee//mmmmrrro jmlr.org wwwpor////ppp.//aa/eeeeeaeeeeere /aarrs/v8B. v1///v/////// 6 478.html.
Dan Hendryeks and Kevin Gimpel. A Baseline for Detecting Misclassified and Distr--Distributiinn Examples in Neural Networks. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings, 2017. URL https: //ppenreview.net/////e forum? iedd-idddii indidddkkee HHHH44141x1.
Dan Hendrycks, Mantas Mazeika, and Thomas G. Dietterich, Deep Anomaly Detection with Outlier Exposure. CoRR, abs/ 2018. URL http:://axx...or///baaaa wbt.aba.......000000
José Miguel Hernández- Lobato and Ryan P. Adams. Probabilistic backpropagation for scalable learning of bayesian neural networks. In Proceedings of the 32nd International Conference on Machine Learning, ICML 2015, Lille, France, 6-11 July 2015, pp. 1861-1869, 2015. URL http::/proceeeee www.:innnsmm........ www.mmmr..ppssse vv7 hht///henandee--e wobatoolllloooooooooooooooooo.oe 5.html.
Jini Hron, Alexander G. de G. Matthews, and Zoubin Ghahramani. Variational Bayesian dropout: pitfalls and fixes. In Proceedings of the 35th International Conference on Machine Learning, ICMI. Stockholmsmüsssun, Stockholm, Sweden, July 10-15, 2018, pp. 2024-2033, 2018. URL http:/ http//pooceedc///////cooocoocccc wwt.oeeeela.m.....rr.r...... p..peess/ / vv. 2.00//// htt/roonnle0 8a.tmml.
Harold Jeffreys. Theory of probability, 3rd Edition. Oxford Classic Texts in the Physical Sci- ences. Clarendon Press, 3 edition, 2003. ISBN 019985066877778811888008800
URL http://
get.lib.rus.ec/bbe
2md5=4 http::/ennnooooe0000000000000ee0
Heinrich Jiang, Been Kim, Melody Y. Guan, and Maya R. Gupta. To Trust Or Not To Trust A Classi- fier. In Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, 3-8 December 2018, Montréal, Canada., pp. 5546-5557, 2018 URL http: httptt::pppppp.ssiiiietm httpp/ppppppppcccc.ppp/ to
to trust-or-noot
fier.
Robert E Kass and Adrian Raftery. Bayes factors. Journal of the American Statistical Association, 90(430):773-755, 1995.
Alex Kendall, Vijay Badrinarayanan, and Roberto Cipolla. Bayesian SegNet: Model Uncertainty in Deep Convolutional Encoder-Decodd Architectures for Scene Understanding. In British Machine Vision Conference 2017, BMVC 2017, London, UK. September 4-7, 2017, 2017. URL https: //t:://wwwww.... /ttp:/wwpbbxxccoo/////////////// / wwt.sggoozobbooooo atw.b////aagggg............a.a/a 2ttp://aaaay002...........
Durk P Kingma, Tim Salimans, and Max Welling. Variational Dropout and the Local Reparame- terization Trick. In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett (eds.), Advances in Neural Information Processing Systems 28, pp. 2575-2583, Curran Associates, Inc., 2015. URL http: theerbooaa--ppppzzmmtteeraaarot-
wttpp/cccpppppe/66666 vari atinallional dropout
iottiiionaandoootttee and-
Agustinus Kristiadi and Asja Fischer. Predictive uncertainty quantification with compound density abs// 1902.01080.
networks. CoRR, abs/1902.000000 2019. URL
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and Scal- able Predictive Uncertainty Estimation using Deep Ensembles. In Advances in Neu- ral Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, 4.9 December 2017, Long Beach. CA. USA, pp. 6405- wawmppll-ad--salaallleeee
6416, 2017. URL httttp/://porssaappppcc/pppppppp preeipiiieeeeeeeereraaaaaaaittae
10

---

Under review as a conference paper at ICLR 2020
Kimin Lee, Honglak Lee, Kibok Lee, and Jinwoo Shin. Training Confidence-caliirated Classifiers for Detecting cut-of-Distributie Samples. In 6th International Conference on Learning Representa- tions, ICLR 2018, Vancouver, BC. Canada, April 30 May 3, 2018, Conference Track Proceedings, 2018. URL https::////.. openooveevv ew.ntt. forum? d=ry Av2xA7.
Hui Li, Xuesong Wang, and Shifei Ding. Research and development of neural network ensembles: a survey. Artif. Intell. Rev. 49(4):455-479. 2018. doi: 10. 10:1007/400462-06--933--1 URL
Yingzhen Li, José Miguel Hernández-Lobato and Richard E. Turner. Stochastic expectation prop- agation. In Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada, pp. 2323-2331 2015. URL http:/ expectation prt.naatiioopppaatttiiiio propagation.
Shiyu Liang, Yixuan Li, and R. Srikant. Enhancing The Reliability of -ut-of-distribution Image Detection in Neural Networks. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings, 2018. URL https:://// htttps://ppenrewee/ htt////pnnneeeeeeeeeeeeeeees www.eeeiiwiweeeeeommememm forum? Hii.iidhHHiiIiPP.
Si Liu, Risheek Garrepalli, Thomas G. Dietterich, Alan Fern, and Dan Hendrycks. Open Category Detection with PAC Guarantees. Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässann Stockholm, Sweden, July 10-15, 2018, pp. 3175-3184, 2018. URL http: http://pprreeddng......eeeeeeeee htt/////////1//// 1iul8e.html.
Christos Louizos and Max Welling. Structured and efficient variational deep learning with matrix gaussian posteriors. In Proceedings of the 33nd International Conference on Machine Learning, ICML 2016, New York City, NY, USA, June 19-24, 2016, pp. 1708-1716, 2016. URL http: //ppoceedings.....ppp......e.eee
///////////// http///////////////////e h.tmm.................mm.mmmm... h..ml.
David J. C. MacKay. A practical bayesian framework for backpropagation networks. Neural Computation, 4(3)448-472, 1992a. doi: (0.1162/neco.1992. 1992.4.3.448. URL WRRLhtts:////i.orr 199......999.....44..
David JC MacKay. Bayesian methods for adaptive models. PhD thesis, California Institute of Technology, 1992b.
Andrey Malinin and Mark J. F. Gales. Predictive Uncertainty Estimation via Prior Networks. In Advances Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurlPS 2018, 3-8 December 2018, Montréal, Canada., pp. 7047-7058. 2018. URL est ma ieetiion-ae a -e http://pp....ppppeppee/aaeeaeeee
Amit Mandelbaum and Daphna Weinshall. Distance-based confidence score for neural network classifiers. CoRR,
2017. URL http: http::://axxvv.orr///o /ttp://9aaa/109998444444
Mahdi Pakdaman Naeini, Gregory Cooper, and Milos Hauskrecht. Obtaining well calibrated probabilities using bayesian binning. In Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence, January 25-30, 2015, Austin, Texas, USA., pp. 2901-2907, 2015. URL occ/////////////// i inndx..phhi paddex.php//e TAAATAAATT hww.T///ppper/view//..
Radford M Neal. Bayesian learning for neural networks. 1996.
Anh Mai Nguyen, Jason Yosinski, and Jeff Clune. Deep neural networks are easily fooled: High confidence predictions for unrecognizable images. In JEEE Conference Computer Vision and Puttern Recognition, CVPR 2015, Boston, MA, USA, June 7-12, 2015, pp. 427-436, 2015. doi: URL httpt:ttp/... http://oooooooooo/o/ooooooooooo/ 10.1 1. /Tt.////RR..21.222.....
N0..1009/CV
Khanh Nguyen and Brendan O'Connor. Posterior calibration and exploratory analysis for natural language processing models. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, EMNLP 2015, Lisbon, Portugal, September 17-21, 2015, pp. 1587-1598, 2015. URL. http: /acl htttt:://awwebbbrrg/aaaaaa www..rgg/anthooogg///////aa
1182.pdf.
11

---

Under review as a conference paper at ICLR 2020
Alexandru Niculescu-Mizze and Rich Caruana. Predicting good probabilities with supervised learning. In Machine Learning. Proceedings of the Twenty-Secae Second International Conference (ICML 2005), Bonn. Germany, August 7-11, 2005, pp. 225-632, 2005, doi: 10.1145/112235111104300 URL
https:: htt.//oooogg/oo///ooo/ooooooog// 0. 4 02331.111111 10240..
Ramon Oliveira, Pedro Tabacof, and Eduardo Valle. Known Unknowns: Uncertainty Quality in Bayesian Neural Networks. CoRR, abs/1612.01251, 2016. URL htt.:t:://aaxxi.oog////aa/
1612.01251.
Nicolas Papernot and Patrick McDaniel. Deep k-Nearest Neighbors: Towards Confident, Interpretable and Robust Deep Learning. March 2018. URL https:
http:/ogg/bb//1000000 65v1.
John Platt. Probabilistic Outputs for Support Vector Machines and Comparisons to Regularized Likelihood Methods. Adv. Large Margin Classif. 10, 2000.
Foster J. Provost, Tom Fawcett, and Ron Kohavi. The case against accuracy estimation for comparing induction algorithms. In Proceedings of the Fifteenth International Conference on Machine Learning (ICML 1998), Madison, Wisconsin, USA, July 24-27, 1998, pp. 445-453, 1998.
Charles Richter and Nicholas Roy. Safe Visual Navigation via Deep Learning and Novelty De- tection. In Robotics: Science and Systems XIII, Massachusetts Institute of Technology, Cam- bridge, Massachusetts, USA, July 12-16, 2017, 2017. doi: 10. 10.15507/RSS. 03600//SSS.001.XI1I. 2017.X111.064. URL rss13 / ht1///64... 4.hmml.
hetp::/wwwwrrbbltt wwwp:ttceeccccccccccceccoee
Jost Tobias Springenberg, Aaron Klein, Stefan Falkner, and Frank Hutter. Bayesian optimization with robust bayesian neural networks. In Advances in Neural Information Processing Systems 29: Annual Conference Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, pp. 4134-4142, 2016. URL http:: http::/pppprrr.nnss....///
6117-
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research, 15(1):1929-1988 2014.
Shengyang Sun, Changyou Chen, and Lawrence Carin. Learning structured weight uncertainty in bayesian neural networks. In Proceedings of the 20th International Conference Artificial Intelligence and Statistics, AISTATS 2017, 20-22 April 2017, Fort Lauderdale, FL, USA, pp. 1283-1292, 2017. URL http:
wtw.perres.gs................... /v54/ sun//unlline manm7b.mm...
Max Welling and Yee Whye Teh. Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of the 28th International Conference on Machine Learning, ICML 2011, Bellevue, Washington, USA, June 28 July 2, 2011, pp. 681-688, 2011. URL https::pp//. http://cccccccc/1////// papers/39E icmlpaper.pdt
Dong Yu, Jinyu Li, and Li Deng. Calibration of confidence measures in speech recognition. IEEE Transactions on Audio, Speech, and Language Processing, 19(8):44612247 2011.
Bianca Zadrozny and Charles Elkan. Obtaining calibrated probability estimates from decision trees and naive Bayesian classifiers. In Proceedings of the Eighteenth International Conference Machine Learning (ICMI 2001), Williams College, Williamstown, MA, USA, June 28 July I, 2001, pp. 609-616, 2001.
Bianca Zadrozny and Charles Elkan. Transforming classifier scores into accurate multiclass probabil- ity estimates. Proceedings of the Eighth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, July 23-26, 2002, Edmonton, Alberta, Canada, pp. 694-699, 2002. doi: 10. 1115/775047.775111 URL https:/ http://pp//oooooo///.11/////////
Hugo Zaragoza and Florence d' Alché Buc. Confidence measures for neural network classifiers. In Proceedings of the Seventh Int. Conf. Information Processing and Management of Uncertainty in Knowlegde Based Systems, 1998.
12

---

Under review as a conference paper at ICLR 2020
0.6
0.6
0.6
frequency
0.4
frequency
0.4
frequency
0.4
0.2
0.2
0.2
0.0
0.0
0.0
10-3
10"
10-3
109
10-3
10
||
(a) Clean
(b) PGD attacked
(c) Boundary attack
0.6
frequency
0.4
0.2
0.0
10-3
10
100
||
(d) evasive CW attack
Figure 3: Frequency distribution of the norm the model Jacobian [ on ResNeXt-34 (2x32) on CIFAR-10, using 3(a): Clean, 3(b): PGD attacked 3(c): Boundary attacked, 3(d): evasive- CW attacked test images.
A ADVERSARIAL ATTACK DETECTION
In this section empirically demonstrate that image vulnerability may also be used defect adversarial examples. We hypothesize that unless otherwise penalized, gradient based attacks will tend move images to regions where the gradient of the loss is large. Based on this heuristic, we propose the norm of the loss gradient norm as criterion for detecting adversarial perturbations. Because the loss available during inference, we propose using the norm of the model gradient as rejection criteria: an image has been adversarially perturbed if
(12) for some threshold value, The threshold is determined by setting the significance level (the rate of false positives) to 5%. For example on CIFAR- 10 we obtained e 2.45 for our model. The results are reported in Table 4 and in Figure 3. Only 6% of clean test images were rejected. However, 100% of Boundary attacks and Carlini -aginir attacks were detected, as well as 96% PGD attacked images.
| (z))222. 2 c,
This leads to the question, is it possible to successfully perturb all images in the test set, and avoid detection? We built a targeted attack, designed to avoid detection. We use Carlini-Wagner Wagner style attack, modified with penalty avoid detection. We augmented the attack loss function with penalty for |[V I|2. which penalizes attacks for being detectable. We call this attack an evasive Carlini- Carlini-Wagner attack. The evasive CW attack was successful at avoiding detection 78% of the time, but order to do so, increased the median adversarial distance significantly, from 0.31 to 0.81, see Table 4.
B MARKOV'S INEQUALITY
Lemma B.1 (Markov's Inequality). For random variable Z with with expectation, let S C c 2
a} then
E [
Prob (S)
(13)
13

---

Under review as a conference paper at ICLR 2020
100
100
Loss
Loss
10
10
10
10 4
10
6
10
10 8
10 33
10
100 Model Entropy
6
10-3 -
Dropout Variance
(a) Dropout variance
(b) Model entropy
Figure 4: Illustration of uncertainty measures on ImageNet. Dropout 0.002.
103
Bayes ratio
102
Loss log(fai)
-log (og(Eff:)))
10
Model Entropy
100
0
100
Bin
Figure 5: Bayes factor equal 100 quantile bins test set for ImageNet: loss, entropy, U1, Uz. The entropy and U1, U3 have very large Bayes factors in the first 10 and last 3 bins.
Table 5: Brier score of various measures of confidence. For CIFAR-10 used X1, the probability of the correct label; for CIFAR-101 and ImageNet-Ie 1K we used X₃ the probability that the correct label is in the Top5. Data is binned into 100 bins, chosen to have equal weight.
Confidence measure
CIFAR-10
CIFAR- 100
ImageNet-IK
Model Entropy
0.033
0.067
0.041
log Pmax
0.033
0.067
0.042
log Σ P1:5
0.067
0.040
V
[p|||
0.034
0.073
0.046
Dropout variance (p 0.002)
0.036
0.074
0.047
Dropout variance (p = 0.01)
0.04
0.075
0.048
Dropout variance (p 0.05)
0.043
0.076
0.049
Ensemble variance
0.040
0.050
0.047
Loss
0
0.029
0.019
14

---

Under review as a conference paper at ICLR 2020
Table Confidence bins for CIFAR-100. The values of and are chosen such that P(top5
0.99 and P(a S top5 Y < b) = 0.95. For the model used here, P(oop5) 0.916.
Confidence measure Y
(a,
P(Y <a)
P(a P(asY= Y < 6)
P(Y 2 6)
Model Entropy
(0.082, 2.1)
0.24
0.50
0.26
llog Pmax
(7.9c-3, 0.42)
0.24
0.49
0.27
log 2 P1:5
(4.8e-3, 0.34)
0.19
0.57
0,24
IV |Il||
(0.46, 1.70)
0.27
0.17
0.56
Dropout variance (p 0.002)
(6.4c- 4, 2.2c-3)
0.27
0.06
0.67
Ensemble variance
(4.2e 4, 0.052)
0.42
0.18
0.40
Table 7: Confidence bins for CIFAR-10. The value of is chosen such that P(topl Y < a) 0.975.
Confidence measure Y
a
P(Y < <a)
P(Y 2 a)
Model Entropy
1.6
0.95
0,05
log Pmax
0.57
0.95
0.05
IV |plll
8.16
0.93
0.07
Dropout variance (p - 0.002)
0.045
0.92
0.08
Ensemble variance
0.019
0.88
0.12
C WORKED EXAMPLE OF BAYES FACTORS
Consider the situation where you have exchanged phone numbers with someone, and you wish to contact them. The question is whether to send text message phone their number. Approximately 95% of people prefer message. Let the probability that person prefers to message. The expected value and odds for X is given by
px 0.95, ( = 19
Now suppose we have additional information, which gives these statistics based on age. Suppose we wish predict X. Knowing the age U has value. Let (xx) be the age, and consider three bins for U given by the values 20, 65 and let Yi, Y2, Y3 be the corresponding histogram random variables.
Y L(U=20),
E [Y]
Y2 = 122SUS665)
E
2 = E
(14)
Y3 ((0>65),
[Y2] = .1
Since older people are more likely to prefer to use phone, the conditional probabilities and corresponding odds are given by
p(X Y1) = -999,
O(px,x) 999
p(X Y2) = .94,
O(xx,v2) 15.7
(15)
p(X Y3) .9. O(px.y,)
In particular, knowing if they are younger or older more valuable than the middle range. The Bayes factor (relative odds) expresses the value of knowing the age if someone is willing to bet with the odds O(px). So this information allows an expected profit on the bet given by the factor.
BF(X YY) 999/19 53
BF(X Y1) 19/15.7 1.2
(16)
BF(X Y2 = 19/9 = 221
So the value of the information depends on the cases. Finally, if we wish to find the expected value of the information, we take an expectation with respect to the probabilities of the events.
E (BR(X]V)) 53 x 4 1.2 2.1
22
(17)
15

---

Under review as a conference paper at ICLR 2020
Some other information about the person may be much less useful in prediction their preference. For example, suppose you know the region where they live and let Y1, 12, Y3 be the histogram random variables. Suppose
p(X Yi .03
E Yi .3
p(X
Y2) .05
E 2 .
(18)
p(X
Y3) = -07
E
= 3 33g
Since E [ .95,
BF(X (1)) 1.9
BF(X Y1) = 1.1
[[BF(X])) = 1.5
(19)
BF(X (Y.) = 1.3
So with an expected value of 1.5, compared to age, with an expected value of 22, the location information much less valuable.
16


Reviews:
[{"review_id": "rkewbIiu9B", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper shows that the maximum softmax probability is useful for uncertainty estimation on in-class data and not just for detecting out-of-distribution data. They argue this with the Bayes Ratio, which here is an uncertainty estimation quality measure that seems worth exploring more for assessing the quality of uncertainty estimation techniques. The community is still in need of a good uncertainty estimation measure (Brier score is too tangled with accuracy, has low numerical resolution, and hardly penalizes consistent overconfidence; AUROC for error detection doesn't budge; the Soft-F1 score is without theoretical motivation; area under the response rate recall curve is too close to 1 when accuracy is high), and this could be it. They evaluate on ImageNet-1K, which most uncertainty estimation papers fail to consider. By considering CIFAR-10, CIFAR-100, and ImageNet, we get fuller picture of the ranking of the utility of many uncertainty estimators, which is somewhat important for the community. This paper is currently borderline, in that what it proposes is simple, but perhaps too simple. Its empirical contributions are fairly minimal, though its proposal of the Bayes Ratio for uncertainty estimation quality assessment could quite impactful.\n\nSmaller Notes:\nFigure 1 should be on page 3 not 2. \n\nSome content in Appendix B should be incorporated in Section 5.1\n\n> In the odds have increased\nIf\n\nB.3 should have a worked example in our setting as well\nEqn (19) should say > 65 not <65\n\nNotation of Y is confusing when it means a binning of U. Perhaps use \"B\"?\n\nIs the histogram adaptively computed? Is that what “chosen to have equal weight” means?\n\nPerhaps compare the Bayes ratio with RMS Calibration Error, AURRA, or AUROC for error detection.\n\nUpdate: The other reviewers are concerned about lack of clarity which is separate from why I like the paper.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "Hkgur3Lg9B", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nThe paper provides a comparison of several uncertainty measures that have been proposed for neural networks. The goal of the work is to alleviate the lack of clear interpretability of softmax outputs as measures of uncertainty in neural networks for multi-class classification (in particular, here image classification). The authors propose the use of expected Bayes factors as aggregate measures of how much information is conveyed by a measure of uncertainty, and compares several proposed approaches based on this measure on CIFAR-10/100 and ImageNet-1k. The authors also discuss the application of uncertainty measures to detect mislabeled or ambiguous images, detecting out-of-distribution samples and adversarial examples. Passing by, the authors propose a measure based on the norm of the gradient to detect adversarial examples that seems to work well. \n\nThere are interesting ideas in the paper. The use of Bayes factors for measuring the quality of the uncertainty estimates, and the comparison of various existing methods on that measure seems useful. The experiments on various tasks also have their own merits.The extension to top-k uncertainty is also interesting.\n\nOn the other hand, I found the paper difficult to follow because the contributions are scattered over the paper and the appendices without clear link. A lot of space is devoted to the definition of the implied risk, which does not bring much to the overall interpretation of the results. The criterion for detecting adversarial examples based on the norm of the gradient appears at the end of the paper somewhat independently from what was before, and the definition and computation of the Expected Bayes factor is entirely deferred to Appendices, which makes the main paper not really self-contained.\n\nThe paper also lacks a conclusion. The various uncertainty measures seem to perform differently depending on the datasets, and it is unclear what the authors recommend in the end. \n\nOverall, it seems to me that with a bit of restructuration, the paper would be an interesting contribution -- for instance in terms of the assessment of dropout variance vs direct model entropy. It seems to me though that for now the paper lacks coherence and clarity in the message.\n\n\n====== after author rebuttal\n\nThe rebuttal answer most of my concerns, and I raised my score to weak accept. Overall, even if there is no clear evidence that one method to evaluate uncertainty is consistently better than the others, I feel that the idea of using Bayes factors to compare uncertainty measures is possibly impactful.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "B1egi4UoYH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: This paper proposes an uncertainty measure called an implied loss. The authors suggest that it is a simple way to quantify the uncertainty of the model. It is suggested that \"Low implied loss (uncertainty) means a high probability of correct classification on the test set.\". They suggest that the analysis of an implied loss justifies the maximum confidence value of softmax-cross entropy. They also extend to evaluate Top-k uncertainty (the uncertainty whether our prediction is in the Top-5 maximum values of our confidence score or not). \n\n========================================================\nClarity:\nI found that this paper content does not seem to be difficult mathematically, however, it is difficult to follow the paper and here I list several parts that can potentially be improved:\n\n1. INTRO: the sentence \"Our implied loss interpretation justifies both methods, since we demonstrate that \"both these quantities\" are uncertainty measures.\". What is both quantities here, the maximum softmax probability and something? \n\n2. INTRO: first contribution, accurate estimates of the probability that the classification of the model on a test set is correct. What do you mean by this sentence? I couldn't see the accuracy of the estimation in Table 2.\n\n3. second contribution in INTRO: what is the meaning of \"a consistent manner\" here. If it means \"successfully\", is there a case that your method fail? It would be nice to be precise when describing the contribution. Figure 2 is more like a small example but can be considered difficult to convince to the readers about the capability of the proposed method.\n\n4. Figure 1 can be much improved. I found the caption is hard to understand. Loss (y-axis) may be written as Kullback-Leibler loss to be more precise in this context (if I understood correctly).\nminor comment: Figure 1(a) [colon missing], since Figure 1(b) has \":\" right after. The authors made an effort to explain the color in the caption of Figure 1(b) but the explanation of (yellow) is missing. The explanation of how to train a network to get Figure 1(a) seems to be missing. It seems -log(f_(1)) should be -log(f^{sort}_(1)). Finally, I'm not sure what is P(Correct|x). Is this histogram suggested that when the U_1 is large and the P(Correct|x) is small, we get a correct prediction, or maybe this means the ratio of correct prediction under the value of loss (histogram)? I think it would help the reader to make it more concise. In the description in page 4 -\\log(p_{max}) seems never define before, was it a typo?\n\n5.1 First page, last sentence: what do you mean by the current setting. Before this point there is no explanation about the problem setting, only we are interested in quantifying an uncertainty measure of the networks.\n5.2 First page, last sentence:  I'm not familiar with Bayes factors, is the last sentence your contribution or it's the finding of the existing work, if it's the latter one, it would be nice to cite them. However, I found this sentence a bit vague: Bayes factor more informative (not sure what is the definition of informative here) better than Brier score under the situation where prob of correct classification is high (is this means high accuracy on the test set?).\n \n6. PRIOR-WORK: Although the authors suggested many existing works, it would be highly useful if the authors discuss the relationship between existing works and their proposed work, e.g., where to put your work in the literature. And since they proposed an uncertainty function, it would be nice to see a few definitions of uncertainty existing works described (doesn't need to be mathematical formulation, I think just an intuitive explanation is sufficient).\n\n7. I am confused with the definition of an implied loss. It is first defined in page 3 with a fixed k (as 1) as a loss where the prediction is correct but in Eq. (4), it looks like a set with one element where y is the maximum prediction score, not a correct label. Then there is U_k(x), if k!=1, is this an implied loss? Although in (5) it is a real number, not a set anymore, I read this paper and took it as a \"yes U_k(x) is also an implied loss\". Also it would be better to define Kullback-Leibler here to be concise and kind to the readers. Then in def 4.1, it's mentioned that the uncertainty measure U_k(x) is an implied loss if the event has high expected loss, does that mean if the event has low expected loss, it is not an implied loss? My opinion is that the authors may use a definition environment and define precisely what is an implied loss. For example, given \"\\ell: X x Y \\to R, a correct label y \\in Y, and an integer k <= K (number of classes), an implied loss is defined as\" to avoid confusion.\n\n8. Def 4.1: U(x) without subscript is undefined (perhaps U_k(x)?). What is an element of the set S^\\eps_k? If it is a set then what is the meaning of the event S has a high expected loss? Does the definition of implied loss after Eq. (6) and the definition of implied loss in Eq. (5) identical when it is Kullback-Leibler loss?\n\n9. Theorem 4.2: S^\\eps seems to be undefined without k. Moreover, how to interpret the bound in (7), it would be nicer to explain the bound after stating the theorem. It is only an inequality that says the left-hand side is smaller or equal to the right-hand side. And does (7) hold for any y? And how tight is the bound?\n\n10. Remark 4.3: what is e_k? what is k here if k in U is set to one? And the name of the remark, how to interpret (8) as \"Neural networks are always overconfident?\" Is this about neural networks or this apply to any function?\n\n11. Sec 5: Figure 6 is not in the main body but the appendix... If this a mistake (and the paper is supposed to be 9 pages without ref.) or it is supposed to be in the appendix? If It's in the appendix, it would be better to mention that this figure is in the appendix.\n\n12. Sec5: since Bayes factor is highly used in this paper to motivate the use of the measure, I don't think it is a good idea to put the explanation of Bayes factor in the appendix, i.e., it is impossible to understand this paper without knowing Bayes factor. \n\n13: Fig 6: I think it is better and kinder to use U_1, U_5 in the legend of the plot instead of f. What is the model entropy?\n\n14. Table 1: why there is \"-\" in CIFAR-10, it is better to clarify it in the paper (or maybe I missed it). I am not sure how to interpret the result, if the higher the better, does that mean the Loss is great?, I'm confused with the experimental results.\n\n15. Before the beginning of 6.2: Tables 7 and 6 are in the appendix and we should state clearly it is in the appendix.\n\n========================================================\nComments:\nThis paper lacks of clarity and difficult to understand. Although it is claimed to be better than existing measure, I am not convinced about that despite many experiments were conducted unfortunately. \nFor the criticism of using the maximum confidence of the softmax score from the softmax-cross entropy loss may not quantify the uncertainty, it is known theoretically that the score of softmax-cross entropy corresponds to p(y|x) if our prediction function achieve the global minimizer this loss function and our function class to be considered is all measurable functions (Zhang, JMLR2004: Statistical Analysis of Some Multi-Category Large Margin Classification Method). For other losses, see Williamson+ JMLR2016: Composite Multiclass Losses. However, it may not be accurate empirically when we use a deep network as it is reported in Guo+, 2017. Thus, one direction is to do post-processing or finding a way to modify a network. For U_1(k), I feel that it should suffer from the same problem as using maximum confidence score of softmax. Extending to top-k may have a good point when discussing about uncertainty and I believe it is good to explore that direction. For experiments, I would like to know how many trials did the authors run the experiment? and it would be helpful to see the standard deviation of the reported value. I believe this paper can still be improved a lot. For these reasons, I vote to reject this paper this time.\n\n========================================================\nMinor comments:\nthere exists the writing convention of \"Top 5\", \"top 5\", \"top5\". It's better to pick one way to describe it if there is no reason to make it different.\n========================================================\nAfter the rebuttal:\nI have read the rebuttal.\n\nI appreciate the authors' effort to modify the paper. \nAlso, please let me state that I totally agree that the problem the authors try to solve is indeed important and relevant for using machine learning in the real-world. \n\nI feel that the structure of the modified version is better than the first version. It is a nice to include the explanation of the Bayes factor in the main body. Appendix C is also very useful for everyone to understand the Bayes factor. \n\nAs the author requested, I have read through the whole revised paper (including the appendix) carefully . I am aware of the positive sides of this paper. For example, it is interesting that we can find wrongly labeled data. Utilizing the uncertainty information for several applications. However, I found that the paper still requires a lot of modifications. The authors have modified many of my concerns, but still several of them were not addressed. I also emphasized the comments for parts that are unrelated to clarity (please see below). For these reasons, I decide not to change my score. \n\nBelow are my comments after reading the rebuttal (which some of them may be overlapped with the issues that were not addressed in the revised version).\n============================\nGeneral comments:\n\n1. Although the work is about tackling the empirical confidence estimation problem, Theorem 3.4 and Eq. (5) are providing insights about the population, not finite data points for empirical estimation. If we focus on the population case, it is known that the minimizer of the softmax cross-entropy loss must be a conditional probability $p(y|x)$. Thus, it is natural that as we minimize such a loss, the probability of correct classification must be high, since we can pick the best choice for classification, i.e., argmax of $p(y|x)$. But the most challenging part of the research in this direction is that, although the theory suggests we can get nice confidence information (in population), when we use the deep neural networks, the quality of confidence estimation can be very bad (overconfidence in empirical estimation) compared with the high accuracy we can achieve. As a result, a finer theory that can quantify the quality of confidence estimate for the finite sample case is highly needed, but I think Theorem 3.4 fails to capture this. I am aware that this theorem only concerns the KL loss. I believe that even only for KL-loss, the result can be significant if we discuss a finer theory for empirical estimation.\n\n2. Regarding the Remark 3.5 (Neural networks are always overconfident), in my understanding, the result has no relationship with neural networks at all since it is true regardless of the hypothesis class of interest (e.g., linear models, kernel models, deep networks). This is because it is simply the definition of a loss function (and the result in the paper focus on KL loss, but I believe we can derive for many other losses). We know that the quality of confidence estimation of simple models can be better although the accuracy is worse. Thus, if the objective is to visualize the problem of neural networks, Remark 3.5 does not seem to help and adding neural networks in the remark title can be misleading. Thus, the implication of Remark 3.5 is insufficient to state that Neural networks are always overconfident.\n\n3. What is the advantage of an implied loss? It seems the paper has two separate stories, the first one is implied loss (Sec. 3) and then move on to Bayes factor (Sec.4). Then, there is an adversarial detection problem using the gradient norm in the last experiment. From Sec. 4, the discussion about implied loss is very limited and if I understand correctly, the $-\\log p_\\max$ and $-log\\sum p_{1:5}$ (the latter seems to require a superscript $sort$) are the implied loss, which does not seem to have the clear advantages over other methods. My impression is that the contributions of this paper are unclear and I do not know what is the main point of this paper. While the abstract dedicates most space to highlight the implied loss (nothing about Bayes factor), the conclusion dedicates most space to highlight the Bayes factor. Improving the connection between two parts may help to signify the contributions. Despite all that, I do like the idea of introducing the Bayes factor in this paper. \n============================\nClarity:\n\n1. Most importantly, I think the clear and solid definition of implied loss is missing.\nAccording to Definition 3.1, \"The uncertainty measure $U_k(x)$ is an implied loss if the event $S^\\epsilon_k$ has high expected loss\". I believe the implied loss is one of the most important contributions of this paper. I am not sure what does it means by \"if the event $S^\\epsilon_k$ has high expected loss\" How can we define \"high expected loss\"?. I tried to read the paper many times to understand what exactly is the implied loss, and what is the scope of implied loss (what is and what is not an implied loss).\n\n2. Following my first issue on clarity, what is the definition of uncertainty measure in this paper? According to the paper, it is defined roughly as $U_k$, whose statistics allow us to better estimate $p_k$. And in the abstract, it is emphasized that if uncertainty measure is an implied loss, then low uncertainty means a high probability of correct classification. Is there any uncertainty measure that is not implied loss? Also in the introduction, it seems that both softmax and the model entropy are uncertainty measures, are they implied losses? Is MC-dropout an uncertainty measure or even an implied loss in this context?\n\n3. Eq. (11) left side: I think it is useful to the expectation with respect to which variable, I believe it is $B$. As a result, is it a typo to have $Y_i$ instead of $B_i$ on the left-hand side of (11)?\n\n4. Example 3.3: If we set $k=1$ according to the definition of the implied loss at the last sentence of this example. Will it contradict $U_1(x)$ defined just right before that sentence? Because $y_w$ will become the $2$-nd ranked label, which I feel it is different from $U_1(x)$ defined in Example 3.3\n\n5. I think it is better to clearly state that the figures/tables are in the appendix when referring to them from the main body. I saw the authors refer to Table 5 in Sec. 4.1, Figure 5 in Sec. 4.2 and Tables 6 and 7 in Sec. 4.3. All of them are in the appendix. And it seems some of them are highly needed. For example, the authors said that \"by fine graining the bins we can capture relatively small ... on the order of 20\" (before Sec. 4.3), then suggested the reader to see Figure 5. I feel Figure 5 must be included in the main body because it is hard to understand that without seeing the figure. \n\n6. I think all figures that have $f_{(1)}$ (Figures 1a, 1b, and 5) must have a superscript $sort$ for all of them. Otherwise, it is wrong.\n\n7. Kullback-Leibler loss is used extensively here without definition. It is important to clarify the clear definition of it. $U_1(x)$ also used extensively for the KL loss case and non-KL loss cases. This can make the paper hard to read. I suggest using $U^{\\mathrm{KL}}_1(x)$ when referring to the uncertainty measure with respect to the KL-loss.\n\n8. I saw $-\\log(p_\\max), -\\log(f_{(1)}), -\\log(f_1^{sort}), U_1$. Are these all refer to the same thing? And I found that sometimes the argument $(x)$ is ignored in the paper sometimes it doesn't in a quite random way. If they are the same, it would be nice to unify them.\n\n9. The caption of figure 2(b) is uninformative. The authors may consider improving it.\n\n10. It would be very helpful to add the implication or interpretation of the theoretical results to help the readers understand the intuition of the proven results. For example, how does Eq. (3) implies that when small uncertainty implies high chance of correct classification.\n============================\nMinor typos:\nINTRO: \"uncertainly measures\" -> \"uncertainty measures\"\n5.2: \"on-distribution\" -> \"in-distribution\"\nConclusion: logpmax -> write clearly with $$ should be better\n\nMinor comments in the appendix:\n1. How does Appendix A related to implied loss or Bayes factor in this paper? Did I miss something?\n2. Figure 5:\n\t2.1 y-axis: is it Bayes ratio or Bayes factor? It seems the Bayes ratio and Bayes factor to be a different thing. Even if it is the same in some literature, I think it's better to use the Bayes factor here for the consistency of this paper. \n\t2.2 Caption: \"entropy\" -> \"model entropy\". \n\t2.3 Caption: Is it mistakes or do you want to insist on using $U_1$ and $U_5$ in the caption but using different notions in the figure? (in figure, they are $-\\log(f_{(1)})$ and $-\\log(\\sum f_{(1:5)})$).\n3. Appendix C: Eq. 16 and Eq. 19: I believe there is a typo. I think it should be $BF(X|Y_1), BF(X|Y_2), BF(X|Y_3)$, respectively.\n4. Appendix C: Eq. 18-right: does it need to sum to 1 not 0.3+0.5+0.3 = 1.1?\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes to model uncertainty using expected Bayes factors, and empirically show that the proposed measure correlates well with the probability that the classification is correct.

All the reviewers agreed that the idea of using Bayes factors for uncertainty estimation is an interesting approach. However, the reviewers also found the presentation a bit hard to follow. While the rebuttal addressed some of these concerns, there were still some remaining concerns (see R3's comments). 

I think this is a really promising direction of research and I appreciate the authors' efforts to revise the draft during the rebuttal (which led to some reviewers increasing the score). This is a borderline paper right now but I feel that the paper has the potential to turn into a great paper with another round of revision. I encourage the authors to revise the draft and resubmit to a different venue.
Decision: Reject