Title: EMPIRICAL CONFIDENCE ESTIMATES FOR CLASSIFICA-TION BY DEEP NEURAL NETWORKS

Abstract: How well can we estimate the probability that the classification predicted by a deep neural network is correct (or in the Top 5)? It is well-known that the softmax values of the network are not estimates of the probabilities of class labels. However, there is a misconception that these values are not informative. We define the notion of implied loss and prove that if an uncertainty measure is an implied loss, then low uncertainty means high probability of correct (or top k) classification on the test set. We demonstrate empirically that these values can be used to measure the confidence that the classification is correct. Our method is simple to use on existing networks: we proposed confidence measures for Top k which can be evaluated by binning values on the test set.

1. INTRODUCTION:

Despite lots of effort to build confidence measures for classification by deep neural networks, there is still a lot of confusion about the value and applicability of these measures. In this article we present a simple method for estimating confidence based on implied loss values which leads to results which are empirically more accurate than benchmarks on test sets. We prove that high confidence values imply a high probability of correct classification on test sets.

Many have observed that used blindly, the maximum softmax probability of a network does a poor job of predicting uncertainty (Nguyen & O'Connor, 2015; Provost et al., 1998; Nguyen et al., 2015; Yu et al., 2011; Lakshminarayanan et al., 2017) . However, Zaragoza & d'Alché Buc (1998) showed in the 1990s that on on shallow networks the maximum softmax probability and the (negative) entropy of the probabilities strongly correlate with model confidence on in-distribution images. More recently, in the deep setting, Hendrycks & Gimpel (2017) showed empirically that the maximum softmax probability can be used to predict network confidence. We demonstrate that both the maximum softmax and the model entropy are uncertainly measures. We extend the uncertainty metric to Top k predictions. We show that, in conjunction with binning, simple uncertainty statistics outperform common approaches like MC-dropout as a measure of confidence, at a fraction of the computational cost.

Using this simple idea, we make the following contributions.

1. We estimate the probability that the classification of the model on a test set is correct which works for existing models (no need to retrain), using a simple tabular form (see Table 2 for Imagenet).

2. We give a simple definition of uncertainty, which applies to previously proposed methods, and leads to a proof that low uncertainty (high confidence) implies high probability of correct classification. It applies to both Top 1 and Top k uncertainty.

3. We can discover mislabelled data, see Figure 1 (c) and we can detect off manifold data and adversarial examples.

We advocate evaluating model uncertainty via expected Bayes factors (Kass & Raftery, 1995) , which provide a rigorous probabilistic approach to evaluating uncertainty, and are widely used for hypothesis testing in other scientific fields, see for example (Good, 1979) and (Jeffreys, 2003) . Compared to other methods (such as AUROC or Brier scores) Bayes factors better distinguish improvements to confidence for methods which are already quite accurate, as is the case for top 1 or top 5 uncertainty for image classification.

2. PRIOR WORK:

As neural networks are adopted into safety critical systems, the need for neural network uncertainty estimates has become abundantly clear. Indeed, any accident adverse system must by design incorporate notions of uncertainty (Amodei et al., 2016) . Real-world examples abound: uncertainty measures are needed in autonomous vehicles (Feng et al., 2018) , robotics (Richter & Roy, 2017) , medical imaging (Ching et al., 2018; DeVries & Taylor, 2018b) and medical decision making (Begoli et al., 2019) , and semantic understanding (Kendall et al., 2017) .

Much effort has been dedicated to addressing this deficiency. Many works have placed neural networks within a Bayesian probabilistic framework. Initial work placed Bayesian priors on model weights (MacKay, 1992a; Neal, 1996) , leading to Bayesian neural networks, however this has proven difficult to implement in practice. Many techniques have been developed to overcome this difficulty (MacKay, 1992b; Neal, 1996; Graves, 2011; Hasenclever et al., 2017; Li et al., 2015; Balan et al., 2015; Welling & Teh, 2011; Springenberg et al., 2016) . One promising approach in the deep learning setting is to perform approximate posterior inference (Louizos & Welling, 2016; Hernández-Lobato & Adams, 2015; Blundell et al., 2015; Sun et al., 2017) .

Due to its simplicity, dropout is widely used as a surrogate for uncertainty. Dropout (Srivastava et al., 2014) was interpreted in a Bayesian setting by Gal & Ghahramani (2016) and Kingma et al. (2015) , however, there are problems with this interpretation, see (Hron et al., 2018) for a recent discussion. Dropout involves evaluating an ensemble of models at test time, which can be both memory and computationally intensive for very large networks.

Non-Bayesian model ensembles have also been developed (Dietterich, 2000) , for a recent survey see (Li et al., 2018) . Lakshminarayanan et al. ( 2017) train an ensemble of adversarially robust models and empirically showed an improvement in uncertainty estimates over dropout based methods. Geifman et al. (2018) proposed using an early stopping criteria to collate an ensemble of models. Kristiadi & Fischer (2019) use mixture modeling to chose ensemble weights.

Several deep learning specific approaches have been proposed in recent years, especially in the context of detecting out-of-distribution samples. Oliveira et al. (2016) suggest detecting outliers via an anomaly detector. Lee et al. (2018) generator out-of-distribution images through a GAN; the classifier is trained to assign the equal weight probability vector to these images. Hendrycks et al. (2018) train networks on two distributions: the in-distribution samples, and out-of-distribution samples. Liu et al. (2018) develop PAC-style guarantees on detection of out-of-distribution samples. Several recent works (Jiang et al., 2018; Papernot & McDaniel, 2018; Mandelbaum & Weinshall, 2017) have suggested using nearest neighbour distances, in feature space, for outlier detection and confidence measures. DeVries & Taylor (2018a) suggest training an additional network to predict uncertainty; Malinin & Gales (2018) specifically model prediction probabilities with a Dirichlet distribution, which implicitly describes model uncertainty. Platt (2000) proposed scaling SVM predictions to better match the validation set; this has been generalized to neural networks and multiclass classification (Niculescu-Mizil & Caruana, 2005; Guo et al., 2017) . Other scaling approaches, such as changing the softmax temperature, have shown promise (Guo et al., 2017; Liang et al., 2018) . Another popular approach to calibration is based on binning model probabilities, developed by Zadrozny & Elkan (2001) . Each bin is assigned a probability of being correct, which is obtained by minimizing the Brier score of the bins (Brier, 1950) . Bins edges may be optimized as well (Zadrozny & Elkan, 2002) ; and can be extended to the Bayesian setting by assigning a prior on binning schemes (Naeini et al., 2015) .

3. CONFIDENCE MEASURES AND UNCERTAINTY ESTIMATES:

Suppose a model f (x), generalizes well, so that it has a high probability, p, of a correct prediction on an image x sampled from the same underlying distribution. Write I k (f ) = {indices of the k largest components of f } for the top k indices. The classification of the vector f is given by the largest component, C(f ) = I 1 (f ). Define the random variables

X k = 1 {y(x)∈I k (f (x))}

to be the Bernoulli random variables with expected value p k = E [X k ], the probability that the correct label is in the Top k. Our goal is to estimate p k . We do this by defining random variables, U k , which we call uncertainties, whose statistics allow us to better estimate p k . The histogram of the uncertainty variables will result in an estimate of the conditional probability that the classification is correct, given the uncertainty value,

Prob (X k (x) = 1 | U k (x) = t) .

Definition 3.1. Given > 0, and the uncertainty measure U (x), define the set

EQUATION

The uncertainty measure U k (x) is an implied loss if the event S k has high expected loss.

Example 3.2. For the Kullback-Leibler loss, the (negative) entropy of the probabilities is an uncertainty measure. Write f sort for the indices of f sorted in decreasing order. Define

EQUATION

Example 3.3. For general losses, L, the top 1 uncertainty can be given by U

1 (x) = {L(f (x), y) | y = C(f (x)

)}, which is the loss, given that the classification was correct. Also U k (x) = L(f, y w ) is an implied loss, for top k, where y w is the (k + 1)-th ranked label.

3.1. ILLUSTRATION OF UNCERTAINTY RANDOM VARIABLES:

The histogram of U 1 is plotted on the test set in Figure 1 (a). Note that for small values of U 1 , the images have a very high probability of being correct. In fact, we can use U 1 to detect incorrectly classified images: we visualized the images which smallest value of U 1 (i.e. highest confidence), which correspond to the few isolated points in the upper left of the figure. It turned out that all of these were either incorrectly labelled, or were ambiguous images, see illustrations in Figure 1 (c). For example, in the second image, the animal is a wallaby, not a wombat. In the fourth image, a paintbrush is a kind of plant, but there is also a pot in the image. In the second part of Figure 1 (b) we illustrate more quantitatively the Top 1 (green) and Top 5 (green or blue) probabilities conditioned on the 100 histograms bins of -log(p max ) on test set for ResNet152 on ImageNet. The Top 1 probability conditioned on the lower bins is very close to 100%. The Top 5 probability is no better than 50% on the last few bins. The intermediate bins are less informative.

3.2. TOP 1 UNCERTAINTY ESTIMATES:

The next theorem shows that if the uncertainty is small, then the probability of correct classification must be high. Theorem 3.4 (Confidence estimate). Define U 1 (x) by ( 2) and define S by (1), and let L KL be the Kullback-Leibler loss. Then

Prob (S ) ≤ E [L KL (f (x), y)] log 1 (3) Proof. Claim: Let > 0 be small. By assumption, -log f sort 1 ≤ . Thus f sort 1 ≥ exp(-). Let e k be the correct label. Then f k ≤ f sort 1 , so f k ≤ 1 -exp(-) and -log(f k ) ≥ -log(1 -exp(-)) ≥ log(1/ ).

Thus for x ∈ S , L KL (f (x), y(x)) ≥ log(1/ ). Apply Markov's inequality (13) to the random variable L(x) = L(f (x), y(x)) to obtain the result.

Remark 3.5 (Neural Networks are always overconfident). Note that the uncertainly is always less than the loss,

EQUATION

with equality when C(f (x)) = y(x). 

3.3. TOP k UNCERTAINTY ESTIMATE:

In the next result we show that if the top k uncertainty is small, then the probability that the correct labels is in the top k must be high. The result can also be proven in the case of general losses, and uncertainly measures satisfying (1).

Consider the event S k (1) for a given k ≥ 1. If the correct label is not in the top k, then the probability of the correct label, f c , must satisfy

f c ≤ f sort k+1 with f sort k+1 ≤ 1 -(f sort 1 + • • • + f sort k ) Thus L KL (f, e c ) ≥ -log(1 -(f sort 1 + • • • + f sort k

)) Then, by an argument similar to the one for Top 1 error, we see that

Prob (S k ) ≤ E [X k ] log 1 (5)

4. EMPIRICAL RESULTS:

The previous section proved that, under fairly general conditions, we can define uncertainty measure which ensure that the top k classification is correct with high probability. The theory applies to uncertainties used in the literature, such as the negative entropy of the probabilities, and negative log softmax.

In practice, once we have an uncertainty measure, the method is simple 1. Compute the statistics on the test set of the uncertainty estimates.

2. Divide the test set into bins, based on uncertainty values. 3. Estimate the conditional probabilities based on the bin populations.

4.1. THE BAYES FACTOR:

The Bayes factor is a way to measure the value of new information, in terms of how much the expected winnings of a fair bet increase, when the information is available. Unlike other measures of confidence, which are additive, the Bayes factor is multiplicative. On a model which is correct 95% of the time, there still a lot of value in knowing when the probability correct increases to 99.5%. In this case, the Bayes factor is close to 10. On the other hand, going from 50% to 54.5% gives a Bayes factor close to 1.2. On the other hand, additive scoring methods give equal weight to both improvements. As an example, we show in Table 5 that the Brier scores of eight different measures of confidence all lie close together, between .033 and .076. On the other hand, the expected Bayes factors range more widely, from 1.3 to 16.6.

Consider a Bernoulli random variable X = B(p X ). The odds for X are given by O(p

X ) = p X 1-p X . Now consider a test, Y = B(p Y ), for which p X,Y = Prob (X = 1 | Y = 1)

Then the odds, given the test succeeds, are

O(p X,Y ) = p X,Y 1-p X,Y . If the odds have increased, we define the Bayes Factor to be BF (X | Y ) = O(p X,Y )

O(p X ) . On the other hand, if the odds have decreased, then the value of the information provided by Y is to bet against, so we define the Bayes factor to be

BF (X | Y ) = O(p X )

O(p X,Y ) . Combining these possibilities, define the Bayes factor by

EQUATION

4.2. EXPECTED BAYES FACTOR:

We can also define a metric for measuring the quality of an uncertainty random variable, such as the value of the loss (or another random variable) for predicting the probability of correct classification.

In practice, we will define the tests based on bin values for some uncertainty variable. If we have more data, we can use more bins. 

EQUATION

Define the histogram (or bin) random variables B i , corresponding to each interval

EQUATION

Each Bayes factor measures the value of information that x lies in each quantile. The value of the test itself is defined to be the expected value of the Bayes factors. Definition 4.2 (Histogram Bayes Factors). Given X, U and the histogram random variables B i , define the conditional probabilities

EQUATION

The predictive value for X of the random variable U with respect to the histogram, is given by

EQUATION

When the number of bins is large, we defined the bins to be quantiles, so that they have an equal number of examples in each bin. In addition, in order to make the histogram Bayes factors finite, we require that each bin have at least one correct and one incorrect example.

See Appendix C for a detailed example of the expected Bayes factor. Next we present an example based on actual confidence measures for networks. Bayes Factor (95/94)(6/5) = 1.2 is nearly one, so there is little additional value to knowing data is in this bin. Finally, when the Model Entropy is greater than .14, which occurs with probability .14, the probability correct is only .8. In this case, the relative probability to correct is worse, to the Bayes Factor is given by (94/80)/(6/20) = 3.9. The expected Bayes factor is the weighted average of the Bayes factor of each bin, weighted by the probability of the bins E [BF (X | B i )] = 6.3 × .55 + 1.2 × .31 + 3.9 × .14 = 4.4

So the expected value of the Model Entropy, for the chosen bins, is 4.4. By fine graining the bins we can capture relatively small and relatively large values of the Model Entropy which can have Bayes factors on the order of 20, see Figure 5 . Thus the expected Bayes factor with 100 bins is 8.18, as shown in Table 1 .

4.3. CONFIDENCE BINS AND BAYES FACTOR:

In this section we present confidence bins for ImageNet-1K. These bins are concise summaries of the information presented in the larger bins. Table 2 presents short bins for ImageNet. Using these bins, we can simply read of from the Uncertainty values, the probability that the model is correct. For example, on the model, P (top 5) = 0.9406, however, using entropy, 55% of the images had entropy low enough to be confidently classified with probability .99. Using U 5 , 66% of images could be binned to have probability .99.

Bins for CIFAR-10 and CIFAR-100 are given in Tables 7 and 6 , respectively.

In Table 1 we show the expected Bayes factor for various confidence measures, on CIFAR-10, CIFAR-100, and ImageNet-1K. In addition to the confidence measures already discussed, we considered Bayesian dropout, and the norm of the gradient of the model. Larger expected Bayes factors means the information is more valuable.

In Figure 5 we plot the regularized Bayes factor for our two main measure of confidence, U 1 and U 5 along with the loss and the model Entropy. The entropy and U 1 , U 5 have very large Bayes factor in the first 10 and last 3 bins, meaning that for these bins, the prediction is 10X (or more) likely to be correct (for the first 10) or wrong (for the last 3 bins) than average. 

5. EXTENSIONS:

In this section we discuss some extensions of the confidence results. We show that we can detect mislabeled images in the test set. We also show that we can obtain some confidence results for off manifold images, as well as adversarial images.

5.1. DETECTION OF MISLABELED IMAGES:

We are able to detect test images which are mis-labeled: images which the network correctly classified, but who's label is incorrect, or for which multiple labels could apply. These are images with high loss but low model entropy. For example in Figure 1 (c) we show six images from the ImageNet-1k test set who's predictions where not in the top5, but had low model entropy. All six of these images either have an incorrect dataset label, or could be described by multiple labels. For example, in the second image, the animal is a wallaby, not a wombat. In the fourth image, a paintbrush is a kind of plant, but there is also a pot in the image.

5.2. CONFIDENCE ON OUT-OF-DISTRIBUTION AND ADVERSARIAL IMAGES:

Next we studied whether we could detect out-of-distribution images generated by COCO. In Figure 2 we show how the histogram of the model entropy is shifted to the right compared to the on-distribution images. Table 3 give the results of our test: choosing a confidence measure which rejects 10% of the on-distribution images, our confidence measures rejected as much as 38% of COCO images (for Entropy) with similar values for U 1 , U 5 . On the other hand Dropout was completely ineffective. 

6. CONCLUSIONS:

With the goal of using measures such as model entropy as a surrogate for the (unknown) model loss, we defined confidence measures as random variables which are large when the loss is large. Using this definition we proved that confidence variables can be used to estimate the probability that a model low expected loss makes a correct prediction. In practical terms, this amounts to defining a confidence measure (such as model entropy or log pmax) and binning the values.

We presented the expected Bayes factor as an effective measure of confidence. Since models are already very accurate, it is important to measure the relative confidence. The Bayes factor is a multiplicative factor to the probability (or odds) that a model is correct. For example, showing the that model entropy on ImageNet with 100 bins is 8 means that knowledge of the model entropy (and the bayes factors for the bins) allows us to predict the probability that the model is correct 8 times more effectively.

The Bayes factors was used to compare existing confidence measures on different tasks. The main task was estimating the probability of a correct prediction on the images from the data set. Additional tasks included: detection of off manifold data, detection of adversarial examples, and detection of mislabelled images. The latter were found by searching for highly confident predictions which were labelled incorrect. 

A ADVERSARIAL ATTACK DETECTION:

In this section we empirically demonstrate that image vulnerability may also be used to detect adversarial examples. We hypothesize that unless otherwise penalized, gradient based attacks will tend to move images to regions where the gradient of the loss is large. Based on this heuristic, we propose the norm of the loss gradient norm as criterion for detecting adversarial perturbations. Because the loss is not available during inference, we propose using the norm of the model gradient as a rejection criteria: an image has been adversarially perturbed if

EQUATION

for some threshold value, c. The threshold is determined by setting the significance level (the rate of false positives) to 5%. For example on CIFAR-10 we obtained c = 2.45 for our model. The results are reported in Table 4 and in Figure 3 . Only 6% of clean test images were rejected. However, 100% of Boundary attacks and Carlini-Wagner attacks were detected, as well as 96% of PGD attacked images.

This leads to the question, is it possible to successfully perturb all images in the test set, and avoid detection? We built a targeted attack, designed to avoid detection. We use a Carlini-Wagner style attack, modified with a penalty to avoid detection. We augmented the attack loss function with a penalty for ∇ (x) 2 * , which penalizes attacks for being detectable. We call this attack an evasive Carlini-Wagner attack. The evasive CW attack was successful at avoiding detection 78% of the time, but in order to do so, it increased the median adversarial distance significantly, from 0.31 to 0.81, see Table 4 . Table 5 : Brier score of various measures of confidence. For CIFAR-10 we used X 1 , the probability of the correct label; for CIFAR-100 and ImageNet-1K we used X 5 the probability that the correct label is in the Top5. Data is binned into 100 bins, chosen to have equal weight. 

B MARKOV'S INEQUALITY:



C WORKED EXAMPLE OF BAYES FACTORS:

Consider the situation where you have exchanged phone numbers with someone, and you wish to contact them. The question is whether to send a text message or phone their number. Approximately 95% of people prefer to message. Let X be the probability that a person prefers to message. The expected value and odds for X is given by p X = 0.95, O(p X ) = 19

Now suppose we have additional information, which gives these statistics based on age. Suppose we wish to predict X. Knowing the age U has a value. Let U (x) be the age, and consider three bins for U given by the values 20, 65 and let Y 1 , Y 2 , Y 3 be the corresponding histogram random variables.

EQUATION

Since older people are more likely to prefer to use a phone, the conditional probabilities and corresponding odds are given by In particular, knowing if they are younger or older is more valuable than the middle range. The Bayes factor (relative odds) expresses the value of knowing the age if someone is willing to bet with the odds O(p X ). So this information allows an expected profit on the bet given by the factor. So the value of the information depends on the cases. Finally, if we wish to find the expected value of the information, we take an expectation with respect to the probabilities of the events. 

So with an expected value of 1.5, compared to age, with an expected value of 22, the location information is much less valuable.

:



Figures and Tables:

mislabelled or ambiguous images found using U1.

Figure 1: Figure 1(a) Scatter plot to indicate how predictive U 1 is compared to the loss. For small values of U 1 , the loss is small with high probability. Figure 1(b): the probability correct (green) or Top5 (blue) given the value of U 1 . Figure 1(c): visualization of the images in the upper left of Figure 1(a). The confident images which were labelled incorrectly turned out to be mislabelled or ambiguous.

Definition 4.1 (Histogram random variables). Given a random variable U (x) ∈ [a, b] and a partition of [a, b] into bins

Figure 2: Figure 2(a): Confidence of a model trained on ImageNet-1k, evaluated on the COCO dataset. Figure 2(b): ImageNet images.

Figure 3: Frequency distribution of the norm of the model Jacobian |∇f (x)| 2,∞ on ResNeXt-34 (2x32) on CIFAR-10, using 3(a): Clean, 3(b): PGD attacked 3(c): Boundary attacked, 3(d): evasive-CW attacked test images.

Figure 4: Illustration of uncertainty measures on ImageNet. Dropout p = 0.002.

Figure 5: Bayes factor over equal 100 quantile bins on test set for ImageNet: loss, entropy, U 1 , U 5 . The entropy and U 1 , U 5 have very large Bayes factors in the first 10 and last 3 bins.

| Y 1 ) = .999, O(p X,Y1 ) = 999 p(X | Y 2 ) = .94, O(p X,Y2 ) = 15.7 p(X | Y 3 ) = .9, O(p X,Y2 ) = 9(15)

| Y 1 ) = 999/19 = 53 BF (X | Y 1 ) = 19/15.7 = 1.2 BF (X | Y 1 ) = 19/9 = 2.1(16)

[BR(X|Y i )] = 53 × .4 + 1.2 × .5 + 2.1 × .1 = 22 (17)Some other information about the person may be much less useful in prediction their preference. For example, suppose you know the region where they live and let Y 1 , Y 2 , Y 3 be the histogram random variables| Y 1 ) = 1.9BF (X | Y 1 ) = 1.1 BF (X | Y 1 ) = 1.3 E [BF (X|Y i )] = 1.5

Bayes factor E[BR] against various measures of confidence. For CIFAR-10 we used X 1 , the probability of the correct label; for CIFAR-100 and ImageNet-1K we used X 5 the probability that the correct label is in the Top5. Data is binned into 100 bins, chosen to have equal weight.Example 4.3. This example follows closely the confidence bins for top 5 on ImageNet-1K, using the Model Entropy, as in the first row of Table2. Consider a model with p X = .94. In this the odds are 94 to 6, so O(p X ) = 15.6. Define three bins for Model Entropy with bin edges 0.31, .140. Then with probability .55, data is in the first bin, in which case the probability correct is .99. So the Bayes factor in this bin is (.99/.94)(.06/.01) = 6.3. Thus knowing the Model Entropy is less than .31 tell you that you are 6.3 times more likely to be correct than on average. The second bin consists of data with with Model Entropy between .31 and .14, which occurs with probability .31, in this case, the

Table content: <table><tr><td>Confidence measure</td><td colspan="3">CIFAR-10 CIFAR-100 ImageNet-1K</td></tr><tr><td>Model Entropy</td><td>4.29</td><td>3.64</td><td>8.18</td></tr><tr><td>-log p max</td><td>4.22</td><td>3.77</td><td>8.87</td></tr><tr><td>-log p 1:5</td><td>-</td><td>4.25</td><td>8.45</td></tr><tr><td>∇ x p</td><td>8.32</td><td>3.47</td><td>7.17</td></tr><tr><td>Dropout variance (p = 0.002)</td><td>10.39</td><td>3.11</td><td>6.84</td></tr><tr><td>Dropout variance (p = 0.01)</td><td>4.67</td><td>2.38</td><td>7.81</td></tr><tr><td>Dropout variance (p = 0.05)</td><td>1.69</td><td>1.35</td><td>1.60</td></tr><tr><td>Ensemble variance</td><td>16.66</td><td>4.03</td><td>6.13</td></tr><tr><td>Loss</td><td>∞</td><td>228.94</td><td>1242.55</td></tr></table>

Confidence bins for ImageNet-1K. The values of a and b are chosen such that P (top5 | Y < a) = 0.99 and P (a ≤ top5 | Y < b) = 0.95. For the model used here, P (top5) = 0.9406.

Table content: <table><tr><td colspan="2">Confidence measure Y</td><td colspan="2">(a, b)</td><td colspan="2">P (Y &lt; a) P (a ≤ Y &lt; b) P (Y ≥ b)</td></tr><tr><td colspan="2">Model Entropy</td><td colspan="2">(0.31, 1.40)</td><td>0.55</td><td>0.31</td><td>0.14</td></tr><tr><td colspan="2">-log p max</td><td colspan="2">(0.047, 0.41)</td><td>0.52</td><td>0.26</td><td>0.22</td></tr><tr><td colspan="2">-log p 1:5</td><td colspan="2">(6.2e-3, 0.03)</td><td>0.66</td><td>0.13</td><td>0.21</td></tr><tr><td colspan="2">∇ x p</td><td colspan="2">(0.19, 0.30)</td><td>0.52</td><td>0.08</td><td>0.40</td></tr><tr><td colspan="4">Dropout variance (p = 0.002) (8.5e-4, 4.7e-3)</td><td>0.50</td><td>0.15</td><td>0.35</td></tr><tr><td colspan="2">Ensemble variance</td><td colspan="2">(0.014, 0.023)</td><td>0.54</td><td>0.05</td><td>0.41</td></tr><tr><td/><td>0.20</td><td/><td>0.20</td><td/></tr><tr><td>frequency</td><td>0.05 0.10 0.15</td><td>frequency</td><td>0.05 0.10 0.15</td><td/><td>mis-classified top5 top1</td></tr><tr><td/><td colspan="2">10 6 10 3 10 0 Model Entropy 0.00</td><td colspan="3">Model Entropy 10 6 10 3 10 0 0.00</td></tr><tr><td/><td>(a) COCO images</td><td/><td/><td colspan="2">(b) ImageNet images</td></tr></table>

Discarding out-of-distribution images from ImageNet-1K. For each confidence measure Y , the value of a is chosen such that P (Y ≤ a | image is from ImageNet-1k) = 0.9.

Table content: <table><tr><td>Image source</td><td>Confidence measure</td><td>a</td><td>P (image discarded)</td></tr><tr><td/><td>Model Entropy</td><td>1.75</td><td>0.38</td></tr><tr><td/><td>-log p max</td><td>0.77</td><td>0.34</td></tr><tr><td>COCO</td><td>-log p 1:5</td><td>0.13</td><td>0.37</td></tr><tr><td/><td>∇ x p</td><td>1.06</td><td>0.23</td></tr><tr><td/><td colspan="2">Dropout variance (p = 0.002) 0.024</td><td>0.</td></tr><tr><td/><td>Model Entropy</td><td>1.75</td><td>0.28</td></tr><tr><td>adversarially</td><td>-log p max</td><td>0.77</td><td>0.25</td></tr><tr><td>perturbed</td><td>-log p 1:5</td><td>0.13</td><td>0.28</td></tr><tr><td>(L 2 )</td><td>∇ x p</td><td>1.06</td><td>0.58</td></tr><tr><td/><td colspan="2">Dropout variance (p = 0.002) 0.024</td><td>0.39</td></tr></table>

Adversarial detection with ResNeXt-34 (2x32) on CIFAR-10. Clean images which the model correctly labels are perturbed until they are misclassified with four attack methods (PGD, Boundary attack, Carlini-Wagner, and an evasive Carlini-Wagner designed to avoid detection). Images are rejected if |∇f (x)| 2,∞ > 2.45.

Table content: <table><tr><td/><td colspan="2">clean PGD Boundary</td><td colspan="2">CW evasive CW</td></tr><tr><td>percent detected</td><td>6% 96%</td><td colspan="2">100% 100%</td><td>22%</td></tr><tr><td>median 2</td><td>-0.31</td><td>0.36</td><td>0.34</td><td>0.81</td></tr></table>

Confidence bins for CIFAR-100. The values of a and b are chosen such that P (top5 | Y < a) = 0.99 and P (a ≤ top5 | Y < b) = 0.95. For the model used here, P (top5) = 0.916.

Table content: <table><tr><td>Confidence measure</td><td colspan="3">CIFAR-10 CIFAR-100 ImageNet-1K</td></tr><tr><td>Model Entropy</td><td>0.033</td><td>0.067</td><td>0.041</td></tr><tr><td>-log p max</td><td>0.033</td><td>0.067</td><td>0.042</td></tr><tr><td>-log p 1:5</td><td>-</td><td>0.067</td><td>0.040</td></tr><tr><td>∇ x p</td><td>0.034</td><td>0.073</td><td>0.046</td></tr><tr><td>Dropout variance (p = 0.002)</td><td>0.036</td><td>0.074</td><td>0.047</td></tr><tr><td>Dropout variance (p = 0.01)</td><td>0.04</td><td>0.075</td><td>0.048</td></tr><tr><td>Dropout variance (p = 0.05)</td><td>0.043</td><td>0.076</td><td>0.049</td></tr><tr><td>Ensemble variance</td><td>0.040</td><td>0.050</td><td>0.047</td></tr><tr><td>Loss</td><td>0</td><td>0.029</td><td>0.019</td></tr></table>

Confidence bins for CIFAR-10. The value of a is chosen such that P (top1 | Y < a) = 0.975.

Table content: <table><tr><td>Confidence measure Y</td><td>a</td><td colspan="2">P (Y &lt; a) P (Y ≥ a)</td></tr><tr><td>Model Entropy</td><td>1.6</td><td>0.95</td><td>0.05</td></tr><tr><td>-log p max</td><td>0.57</td><td>0.95</td><td>0.05</td></tr><tr><td>∇ x p</td><td>8.16</td><td>0.93</td><td>0.07</td></tr><tr><td colspan="2">Dropout variance (p = 0.002) 0.045</td><td>0.92</td><td>0.08</td></tr><tr><td>Ensemble variance</td><td>0.019</td><td>0.88</td><td>0.12</td></tr></table>

Peer review (iclr2020):
Summary: This paper proposes an uncertainty measure called an implied loss. The authors suggest that it is a simple way to quantify the uncertainty of the model. It is suggested that *Low implied loss (uncertainty) means a high probability of correct classification on the test set.*. They suggest that the analysis of an implied loss justifies the maximum confidence value of softmax-cross entropy. They also extend to evaluate Top-k uncertainty (the uncertainty whether our prediction is in the Top-5 maximum values of our confidence score or not). ======================================================== Clarity: I found that this paper content does not seem to be difficult mathematically, however, it is difficult to follow the paper and here I list several parts that can potentially be improved: 1. INTRO: the sentence *Our implied loss interpretation justifies both methods, since we demonstrate that *both these quantities* are uncertainty measures.*. What is both quantities here, the maximum softmax probability and something? 2. INTRO: first contribution, accurate estimates of the probability that the classification of the model on a test set is correct. What do you mean by this sentence? I couldn*t see the accuracy of the estimation in Table 2. 3. second contribution in INTRO: what is the meaning of *a consistent manner* here. If it means *successfully*, is there a case that your method fail? It would be nice to be precise when describing the contribution. Figure 2 is more like a small example but can be considered difficult to convince to the readers about the capability of the proposed method. 4. Figure 1 can be much improved. I found the caption is hard to understand. Loss (y-axis) may be written as Kullback-Leibler loss to be more precise in this context (if I understood correctly). minor comment: Figure 1(a) [colon missing], since Figure 1(b) has *:* right after. The authors made an effort to explain the color in the caption of Figure 1(b) but the explanation of (yellow) is missing. The explanation of how to train a network to get Figure 1(a) seems to be missing. It seems -log(f_(1)) should be -log(f^{sort}_(1)). Finally, I*m not sure what is P(Correct|x). Is this histogram suggested that when the U_1 is large and the P(Correct|x) is small, we get a correct prediction, or maybe this means the ratio of correct prediction under the value of loss (histogram)? I think it would help the reader to make it more concise. In the description in page 4 -log(p_{max}) seems never define before, was it a typo? 5.1 First page, last sentence: what do you mean by the current setting. Before this point there is no explanation about the problem setting, only we are interested in quantifying an uncertainty measure of the networks. 5.2 First page, last sentence: I*m not familiar with Bayes factors, is the last sentence your contribution or it*s the finding of the existing work, if it*s the latter one, it would be nice to cite them. However, I found this sentence a bit vague: Bayes factor more informative (not sure what is the definition of informative here) better than Brier score under the situation where prob of correct classification is high (is this means high accuracy on the test set?). 6. PRIOR-WORK: Although the authors suggested many existing works, it would be highly useful if the authors discuss the relationship between existing works and their proposed work, e.g., where to put your work in the literature. And since they proposed an uncertainty function, it would be nice to see a few definitions of uncertainty existing works described (doesn*t need to be mathematical formulation, I think just an intuitive explanation is sufficient). 7. I am confused with the definition of an implied loss. It is first defined in page 3 with a fixed k (as 1) as a loss where the prediction is correct but in Eq. (4), it looks like a set with one element where y is the maximum prediction score, not a correct label. Then there is U_k(x), if k!=1, is this an implied loss? Although in (5) it is a real number, not a set anymore, I read this paper and took it as a *yes U_k(x) is also an implied loss*. Also it would be better to define Kullback-Leibler here to be concise and kind to the readers. Then in def 4.1, it*s mentioned that the uncertainty measure U_k(x) is an implied loss if the event has high expected loss, does that mean if the event has low expected loss, it is not an implied loss? My opinion is that the authors may use a definition environment and define precisely what is an implied loss. For example, given *ell: X x Y 	o R, a correct label y in Y, and an integer k <= K (number of classes), an implied loss is defined as* to avoid confusion. 8. Def 4.1: U(x) without subscript is undefined (perhaps U_k(x)?). What is an element of the set S^eps_k? If it is a set then what is the meaning of the event S has a high expected loss? Does the definition of implied loss after Eq. (6) and the definition of implied loss in Eq. (5) identical when it is Kullback-Leibler loss? 9. Theorem 4.2: S^eps seems to be undefined without k. Moreover, how to interpret the bound in (7), it would be nicer to explain the bound after stating the theorem. It is only an inequality that says the left-hand side is smaller or equal to the right-hand side. And does (7) hold for any y? And how tight is the bound? 10. Remark 4.3: what is e_k? what is k here if k in U is set to one? And the name of the remark, how to interpret (8) as *Neural networks are always overconfident?* Is this about neural networks or this apply to any function? 11. Sec 5: Figure 6 is not in the main body but the appendix... If this a mistake (and the paper is supposed to be 9 pages without ref.) or it is supposed to be in the appendix? If It*s in the appendix, it would be better to mention that this figure is in the appendix. 12. Sec5: since Bayes factor is highly used in this paper to motivate the use of the measure, I don*t think it is a good idea to put the explanation of Bayes factor in the appendix, i.e., it is impossible to understand this paper without knowing Bayes factor. 13: Fig 6: I think it is better and kinder to use U_1, U_5 in the legend of the plot instead of f. What is the model entropy? 14. Table 1: why there is *-* in CIFAR-10, it is better to clarify it in the paper (or maybe I missed it). I am not sure how to interpret the result, if the higher the better, does that mean the Loss is great?, I*m confused with the experimental results. 15. Before the beginning of 6.2: Tables 7 and 6 are in the appendix and we should state clearly it is in the appendix. ======================================================== Comments: This paper lacks of clarity and difficult to understand. Although it is claimed to be better than existing measure, I am not convinced about that despite many experiments were conducted unfortunately. For the criticism of using the maximum confidence of the softmax score from the softmax-cross entropy loss may not quantify the uncertainty, it is known theoretically that the score of softmax-cross entropy corresponds to p(y|x) if our prediction function achieve the global minimizer this loss function and our function class to be considered is all measurable functions (Zhang, JMLR2004: Statistical Analysis of Some Multi-Category Large Margin Classification Method). For other losses, see Williamson+ JMLR2016: Composite Multiclass Losses. However, it may not be accurate empirically when we use a deep network as it is reported in Guo+, 2017. Thus, one direction is to do post-processing or finding a way to modify a network. For U_1(k), I feel that it should suffer from the same problem as using maximum confidence score of softmax. Extending to top-k may have a good point when discussing about uncertainty and I believe it is good to explore that direction. For experiments, I would like to know how many trials did the authors run the experiment? and it would be helpful to see the standard deviation of the reported value. I believe this paper can still be improved a lot. For these reasons, I vote to reject this paper this time. ======================================================== Minor comments: there exists the writing convention of *Top 5*, *top 5*, *top5*. It*s better to pick one way to describe it if there is no reason to make it different. ======================================================== After the rebuttal: I have read the rebuttal. I appreciate the authors* effort to modify the paper. Also, please let me state that I totally agree that the problem the authors try to solve is indeed important and relevant for using machine learning in the real-world. I feel that the structure of the modified version is better than the first version. It is a nice to include the explanation of the Bayes factor in the main body. Appendix C is also very useful for everyone to understand the Bayes factor. As the author requested, I have read through the whole revised paper (including the appendix) carefully . I am aware of the positive sides of this paper. For example, it is interesting that we can find wrongly labeled data. Utilizing the uncertainty information for several applications. However, I found that the paper still requires a lot of modifications. The authors have modified many of my concerns, but still several of them were not addressed. I also emphasized the comments for parts that are unrelated to clarity (please see below). For these reasons, I decide not to change my score. Below are my comments after reading the rebuttal (which some of them may be overlapped with the issues that were not addressed in the revised version). ============================ General comments: 1. Although the work is about tackling the empirical confidence estimation problem, Theorem 3.4 and Eq. (5) are providing insights about the population, not finite data points for empirical estimation. If we focus on the population case, it is known that the minimizer of the softmax cross-entropy loss must be a conditional probability . Thus, it is natural that as we minimize such a loss, the probability of correct classification must be high, since we can pick the best choice for classification, i.e., argmax of . But the most challenging part of the research in this direction is that, although the theory suggests we can get nice confidence information (in population), when we use the deep neural networks, the quality of confidence estimation can be very bad (overconfidence in empirical estimation) compared with the high accuracy we can achieve. As a result, a finer theory that can quantify the quality of confidence estimate for the finite sample case is highly needed, but I think Theorem 3.4 fails to capture this. I am aware that this theorem only concerns the KL loss. I believe that even only for KL-loss, the result can be significant if we discuss a finer theory for empirical estimation. 2. Regarding the Remark 3.5 (Neural networks are always overconfident), in my understanding, the result has no relationship with neural networks at all since it is true regardless of the hypothesis class of interest (e.g., linear models, kernel models, deep networks). This is because it is simply the definition of a loss function (and the result in the paper focus on KL loss, but I believe we can derive for many other losses). We know that the quality of confidence estimation of simple models can be better although the accuracy is worse. Thus, if the objective is to visualize the problem of neural networks, Remark 3.5 does not seem to help and adding neural networks in the remark title can be misleading. Thus, the implication of Remark 3.5 is insufficient to state that Neural networks are always overconfident. 3. What is the advantage of an implied loss? It seems the paper has two separate stories, the first one is implied loss (Sec. 3) and then move on to Bayes factor (Sec.4). Then, there is an adversarial detection problem using the gradient norm in the last experiment. From Sec. 4, the discussion about implied loss is very limited and if I understand correctly, the and (the latter seems to require a superscript ) are the implied loss, which does not seem to have the clear advantages over other methods. My impression is that the contributions of this paper are unclear and I do not know what is the main point of this paper. While the abstract dedicates most space to highlight the implied loss (nothing about Bayes factor), the conclusion dedicates most space to highlight the Bayes factor. Improving the connection between two parts may help to signify the contributions. Despite all that, I do like the idea of introducing the Bayes factor in this paper. ============================ Clarity: 1. Most importantly, I think the clear and solid definition of implied loss is missing. According to Definition 3.1, *The uncertainty measure is an implied loss if the event has high expected loss*. I believe the implied loss is one of the most important contributions of this paper. I am not sure what does it means by *if the event has high expected loss* How can we define *high expected loss*?. I tried to read the paper many times to understand what exactly is the implied loss, and what is the scope of implied loss (what is and what is not an implied loss). 2. Following my first issue on clarity, what is the definition of uncertainty measure in this paper? According to the paper, it is defined roughly as , whose statistics allow us to better estimate . And in the abstract, it is emphasized that if uncertainty measure is an implied loss, then low uncertainty means a high probability of correct classification. Is there any uncertainty measure that is not implied loss? Also in the introduction, it seems that both softmax and the model entropy are uncertainty measures, are they implied losses? Is MC-dropout an uncertainty measure or even an implied loss in this context? 3. Eq. (11) left side: I think it is useful to the expectation with respect to which variable, I believe it is . As a result, is it a typo to have instead of on the left-hand side of (11)? 4. Example 3.3: If we set according to the definition of the implied loss at the last sentence of this example. Will it contradict defined just right before that sentence? Because will become the -nd ranked label, which I feel it is different from defined in Example 3.3 5. I think it is better to clearly state that the figures/tables are in the appendix when referring to them from the main body. I saw the authors refer to Table 5 in Sec. 4.1, Figure 5 in Sec. 4.2 and Tables 6 and 7 in Sec. 4.3. All of them are in the appendix. And it seems some of them are highly needed. For example, the authors said that *by fine graining the bins we can capture relatively small ... on the order of 20* (before Sec. 4.3), then suggested the reader to see Figure 5. I feel Figure 5 must be included in the main body because it is hard to understand that without seeing the figure. 6. I think all figures that have (Figures 1a, 1b, and 5) must have a superscript for all of them. Otherwise, it is wrong. 7. Kullback-Leibler loss is used extensively here without definition. It is important to clarify the clear definition of it. also used extensively for the KL loss case and non-KL loss cases. This can make the paper hard to read. I suggest using when referring to the uncertainty measure with respect to the KL-loss. 8. I saw . Are these all refer to the same thing? And I found that sometimes the argument is ignored in the paper sometimes it doesn*t in a quite random way. If they are the same, it would be nice to unify them. 9. The caption of figure 2(b) is uninformative. The authors may consider improving it. 10. It would be very helpful to add the implication or interpretation of the theoretical results to help the readers understand the intuition of the proven results. For example, how does Eq. (3) implies that when small uncertainty implies high chance of correct classification. ============================ Minor typos: INTRO: *uncertainly measures* -> *uncertainty measures* 5.2: *on-distribution* -> *in-distribution* Conclusion: logpmax -> write clearly with $$ should be better Minor comments in the appendix: 1. How does Appendix A related to implied loss or Bayes factor in this paper? Did I miss something? 2. Figure 5: 2.1 y-axis: is it Bayes ratio or Bayes factor? It seems the Bayes ratio and Bayes factor to be a different thing. Even if it is the same in some literature, I think it*s better to use the Bayes factor here for the consistency of this paper. 2.2 Caption: *entropy* -> *model entropy*. 2.3 Caption: Is it mistakes or do you want to insist on using and in the caption but using different notions in the figure? (in figure, they are and ). 3. Appendix C: Eq. 16 and Eq. 19: I believe there is a typo. I think it should be , respectively. 4. Appendix C: Eq. 18-right: does it need to sum to 1 not 0.3+0.5+0.3 = 1.1?