Empirical Frequentist Coverage of Deep Learning Uncertainty Quantification Procedures
Authors: Benjamin Kompa
Jasper Snoek
Andrew Beam
Venue:  (2021)
Keywords: uncertainty quantification
coverage
dataset shift

Abstract: Uncertainty quantification for complex deep learning models is increasingly important as these techniques see growing use in high-stakes, real-world settings. Currently, the quality of a model's uncertainty is evaluated using point-prediction metrics such as negative log-likelihood or the Brier score on heldout data. In this study, we provide the first large scale evaluation of the empirical frequentist coverage properties of well known uncertainty quantification techniques on a suite of regression and classification tasks. We find that, in general, some methods do achieve desirable coverage properties on \emph{in distribution} samples, but that coverage is not maintained on out-of-distribution data. Our results demonstrate the failings of current uncertainty quantification techniques as dataset shift increases and establish coverage as an important metric in developing models for real-world applications.

Under review as a conference paper at ICLR 2020
EMPIRICAL FREQUENTIST COVERAGE OF DEEP LEARNING UNCERTAINTY QUANTIFICATION PROCEDURES
Anonymous authors Paper under double-blind review
ABSTRACT
Uncertainty quantification for complex deep learning models is increasingly im- portant as these techniques see growing use in high-stakes, real-world settings. Currently, the quality of model's uncertainty is evaluated using point-prediction metrics such as negative fog-likelihoon or the Brier score on heldout data. In this study, provide the first large scale evaluation of the empirical frequentist cov- erage properties of well known uncertainty quantification techniques on a suite of regression and classification tasks. We find that, in general, some methods do achieve desirable coverage properties on in distribution samples, but that cov- erage is not maintained on -isttriiitiibttionn data. Our results demonstrate the failings of current uncertainty quantification techniques as dataset shift increases and establish coverage as an important metric in developing models for real-world applications.
INTRODUCTION
Predictive models based on deep learning have seen dramatic improvement in recent years (LeCun ctt al., 2013). which has led widespread adoption in many areas. For critical, high-stakes do- mains such as medicine or self-driving cars, it is imperative that mechanisms are in place to ensure safe and reliable operation. Crucial to the notion of safe and reliable deep learning the effective quantification and communication of predictive uncertainty to potential end-users of a system. Many approaches have recently been proposed that fall into two broad categories: ensembles and Bayesian methods. Ensembles (Lakshminarayann
et al. 2017) aggregate information from many individual models to provide a measure of uncertainty that reflects the ensembles agreement about a given data point. Bayesian methods offer direct access to predictive uncertainty through the posterior predictive distribution, which combines prior knowledge with the observed data. Although conceptually ele- gant, calculating exact posteriors of even simple neural models is computationally intractable (Yao) ct al., 2019; Neal] 1996), and many approximations have been developed AiernándezzLLbaal & Adams 2015 Blundell al. 2015 Graves 201T: Pawlowski al. 2017 [Hernández-Lobato al.) 2016 Louizos & Welling 2016; 2017). Though approximate Bayesian methods scale to modern sized data and models, recent work has questioned the quality of the uncertainty provided by these
approximations (aao et al. 2019 Wenzel et al. 2020 Ovadia et al. 2019).
Previous work assessing the quality of uncertainty estimates have focused on calibration metrics and scoring rules such as the lgglkive-oglikellioood (NLL), expected calibration error (ECE), and Brier score. Here we provide complementary perspective based on the notion of empirical coverage, well-established concept in the statistical literature (Wasserman) 2013) that evaluates the quality of a predictive set or interval instead a point prediction. Informally, coverage asks the question: If model produces predictive uncertainty interval, how often does that interval actually contain the observed value? Ideally, predictions on examples for which a model uncertain would produce larger intervals and thus be more likely cover the observed value. More formally, given features Xn E IRd and a response yv. E I, coverage is defined in terms of set (x) and level a E [0, 1]. The set C,, (x) said to have coverage at the - a level for all distributions P IRd where (x, v) ~ P. the following inequality holds:
P(yn C,, 2

---

Under review as a conference paper at ICLR 2020
Model
Model
x
X
x
X
Coverage full Widhhi 5Ds
Prodictedd Y,
Coveragee 2u Widthc
Figure 1: An example of the coverage properties for two methods of uncertainty quantification. In this scenario, each model produces an uncertainty interval for each æ; which attempts to cover the true yi, represented by the red points. Coverage is calculated the fraction of true values contained in these regions, while the width of these regions is reported in terms of multiples of the standard deviation of the training set y/s values.
The set čn (x) can be constructed using a variety of procedures. For example, in the case of simple linear regression a prediction interval for new point Fn++1 can be constructed using simple, closed-form solution. Figure provides graphical depiction of coverage for two hypothetical regression models.
A complementary metric to coverage is width, which is the size of the prediction interval or set. Width can provide relative ranking of different methods, i.e. given two methods with the same level of coverage we should prefer the method that provides intervals with smaller widths.
Contributions: this study we investigate the empirical coverage properties of prediction inter- vals constructed from catalog of popular uncertainty quantification techniques such as ensembling, Monte-Carke dropout, Gaussian processes, and stochastic variational inference. We assess the cov- erage properties of these methods on nine regression tasks and two classification tasks with and without dataset shift. These tasks help us make the following contributions:
We introduce coverage and width as a natural and interpretable metrics for evaluating pre- dictive uncertainty.
A comprehensive set of coverage evaluations on suite of popular uncertainty quantifica- tion techniques.
An examination of how dataset shift affects these coverage properties.
2 BACKGROUND AND RELATED WORK
Obtaining Predictive Uncertainty Estimates
Several lines of work focus on improving approximations of the posterior of Bayesian neural net-
work (Graves] 201T, Zlemándeez-Lobtt Adams] 2013 Blundell et al. 2013 Hernandez
al.
2016 Louizos & Welling] 2016 Pawlowski al. 2017 Louizos & Welling 2017). Yao al.
well-known result from the statisties literature (c.f. chapter 13 of Wasserman
(2013)
the interval
given by j.+1 4 fu-22,, /1/n- (xx+41 #)2/(nn 1)s3), where 0h is the predicted value, f.-2 is the a/2 critical value from t-distribution with 2 degrees freedom, is the mean in the training data, and sy, S, are the standard deviations for and x respectively. such that ([) holds asymptotically. How- ever, for more complicated models such as deep learning. closed form solutions with coverage guarantees are unavailable, and constructing these intervals via the bootstrap (Efron) 1982)) can be computationally infeasible fail provide the correct coverage (Chatterjee Lahiri 2011
2

---

Under review as a conference paper at ICLR 2020
2019) provide comparison of many of these methods and highlight issues with common metrics of comparison, such as test-set log likelihood and RMSE. Good scores on these metrics often in- dicates that the model posterior happens to match the test data rather than the true posterior (Yao) et al. 2019). Maddox et al. (019)) developed a technique to sample the approximate posterior from the first moment of SGD iterates. Wenzel al. (2020) demonstrated that despite advances in these approximations, there are still outstanding challenges with Bayesian modeling for deep networks.
Alternative methods that do not rely on estimating a posterior over the weights of model can also
be used to provide uncertainty estimates, [Ga & Ghahramani (2016), for instance, demonstrated that the Bayesian posterior implied by
Monte Carlo dropout is related variational approximation
the dropout procedure. Lakshminaaayaaauu et al. (2017) used ensembling of several neural networks to obtain uncertainty estimates. Guo et al. (2017) established that temperature scaling provides well calibrated predictions on an i.i.d test set. More recently, van Amersfoort et al. (2020) showed that the distance from the centroids in a RBF neural network yields high quality uncertainty estimates. Liu et al. (2220) also leveraged the notion of distance (in this case, the distance from test to train examples) to obtain uncertainty estimates with their Spectral- normalized Neural Gaussian Processes.
Assessments of Uncertainty Properties under Dataset Shift
Ovadia et al. (2019) analyzed the effect of dataset shift on the accuracy and calibration of Bayesian deep learning methods. Their large scale empirical study assessed these methods standard datasets such as MNIST, CIFAR-10, ImageNet, and other non-image based datasets. Additionally, they used translations, rotations, and corruptions (Hendrycks & Gimpell (017)) of these datasets to quantify performance under dataset shift. They found stochastic variational inference (SVI) to be promising on simpler datasets such MNIST and CIFAR-10, but more difficult train on larger datasets. Deep ensembles had the most robust response to dataset shift.
Theoretical Coverage Guarantees
The Bernstein-von Mises theorem connects Bayesian credible sets and frequentist confidence in- tervals. Under certain conditions, Bayesian credible sets of level a are asymptotically frequentist confidence sets of level a and thus have the same coverage properties. However, when there is model misspecification, coverage properties no longer hold (Kleijn & van der Vaart (0022).
Barber et al. (019)) explored under what conditions conditional coverage guarantees can hold for arbitrary models (i.e. guarantees for P(y, E (x|x xn)). which are per sample guarantees). They show that even when these coverage properties are not desired to hold for any possible dis- tribution, there provably no methods that can give such guarantees. By extension, no Bayesian deep learning methods can provide conditional coverage guarantees.
METHODS
In both the regression and classification settings, we analyzed the coverage properties of prediction intervals and sets of five different approximate Bayesian and non- non-Bayesian approaches for uncer- tainty quantification. These include Dropout (Gal & Ghahramani 2016 Srivastava et al. 2015)
ensembles (Lakshminarayanan et al. (017)) Stochastic Variational Inference (Blundell et al. 2015 Graves 2011 Louizos & Welling 2016 2017: Wen al. 2018). and last layer approximations of SVI and Dropout (Riquelme et al. 2011). Additionally, we considered prediction intervals from linear regression and the 95% credible interval of a Gaussian process with the squared exponential kernel as baselines in regression tasks. For classification, we also considered temperature scaling (Guo et al. 2017) and the softmax output of vanilla deep networks (Hendrycks & Gimpel 2017).
3.1 REGRESSION METHOOS AND METRICS
We evaluated the coverage properties of these methods on nine large real world regression datasets used as a benchmark in Hornández-Lobato & Adams (2015) and later Gal and Ghahramani (Gal & Ghahramani 2016). We used the training, validation, and testing splits publicly available from Gal and Ghahramani and performed nested cross validation to find hyperparameters and evaluated coverage properties, defined as the fraction of prediction intervals which contained the true value in the test set. On the training sets, we did 100 trials of random search over hyperparameter space of a multi-layer-per pertepptyyrreroettroon architecture with an Adam optimizer (Kingma & Ba 2013) and selected hyperparameters based RMSE on the validation set.
3

---

Under review as a conference paper at ICLR 2020
An example of the corruptions in CIFAR-1O-C. The 16 different corruptions have 5 dis- crete levels of shift, of which are shown here. The same corruptions were applied to ImageNet to form the ImageNet-C dataset.
Figure 2:
Each approach required slightly different ways to obtain a 95% prediction interval. For an ensemble of neural networks, we trained N = 40 vanilla networks and used the 2.5% and 97.5% quantiles as the boundaries the prediction interval. For dropout and last layer dropout, we made 200 predictions per sample and similarly discarded the top and bottom 2.5% quantiles. For SVI, last layer SVI (LL SVI), and Gaussian processes we had approximate variances available for the posterior which used calculate the prediction interval. We calculated 95% prediction intervals from linear regression using the closed-form solution.
Then calculated two metrics:
Coverage: A sample is considered covered if the true label is contained in this 95% predic- tion interval. We average over all samples test set to estimate the coverage method on this dataset.
Width: The width is the average over the test set of the ranges of the 95% prediction intervals.
Coverage measures how often the true label is in the prediction region while width measures how specific that prediction region is. Ideally, we would have high levels of coverage with low levels of width on in-distribution data. As data becomes increasingly out of distribution, we would like coverage to remain high while width increases to indicate model uncertainty.
3.2 CLASSIFICATION METHODS AND METRICS
Ovadia et al. (2019) evaluated model uncertainty variety of datasets publicly available. These predictions were made with the five apprxoimate Bayesian methods describe above, plus vanilla neural networks, with and without temperature scaling. We focus on the predictions from MNIST, CIFAR-10, CIFAR-10-C, ImageNet, and ImageNet-C datasets. For MNIST, Ovadia et al. (2019) measured model predictions rotated and translated versions of the test set. For CIFAR- 10. Ova- dia et al. (2019) measured model predictions on translated and corrupted versions of the test set. For Net, Ovadia al. (20T9) only analyzed model predictions on the corrupted images of ImageNet-C. Each of these transformations (rotation, translation, or any of the 16 corruptions) has multiple levels of shift. Rotations range from 15 to 180 degrees in 15 degrees increments. Transla- tions shift images every 2 and 4 pixels for MNIST and CIFAR-10, respectively. Corruptions have increasing levels of intensity. Figure shows the effects of the 16 corruptions in CIFAR-10-C the first, third, and fifth levels of intensity.
We calculate the prediction set of a model's output. Given a E (0, 1), the 1 prediction set S for a sample x; is the minimum sized set of classes such that
E(uddx)) 2 SES
This consists of the top probabilities such that probability has been accumulated. Then we can define:
Coverage: For each dataset point, we calculate the 1- a prediction set of the label proba-
bilities, then coverage is what fraction of these prediction sets contain the true label.
Width: The width of a prediction set is simply the number of labels in the set, [S]. We report the average width of prediction sets over dataset in our figures.
4

---

Under review as a conference paper at ICLR 2020
Although both calibration (Guo et al. 2017) and coverage can involve a probability over model's output, calibration only considers the most likely label and it's corresponding probability, while coverage considers the the top-k; probabilities. In the classification setting, coverage is more robust to label errors as it does not penalize models for putting probability on similar classes.
4 RESULTS
4.1 REGRESSION
Table shows the mean and standard error of coverage levels for the methods we evaluated. In the regression setting, we find high levels of coverage for linear regression, Gaussian processes, SVI, and LL SVI. Ensembles and Dropout had lower levels of coverage, while LL Dropout had the lowest average coverage. Table reports the average width of the 95% prediction interval in terms of standard deviations of the response variable. We see that higher coverage correlates with higher average width.
Datraet Method
Linee Regression
Ensamble
Despout
DI.Deapou
SVI
Roston Housing
0.946 (5.61e-03)
0.9765 (5.05e-(13)
015912 (1:47-
0,402 (1 (1644-00)
0:1002 (2.0le01)
0.9434 (6 (6-044-11)
(8.48e085)
Cinerste
0.9437 (3.66--5)
(3.13-03)
0,5854
044--33
(1 (7
0.0932 (1.75c-0)
0.9581
(3.61c-41))
0,9443
72-85)
Energy
0.8957 (4.86e
0.8877 (6.96e 03)
0.865
033
0.5013 202000e000 02)
0.2597
(2)
0.9773
(33
0,9938
(2.99-03)
Kinânm
0.9514 (1.20e00))
0.9705 (1.56-03)
0.706
(4.43e-033
0,8037 (8 15e-03)
0.1984 (1 (633
0.9618
(266e-33
0,9633
(1.36-88)
Naval Propulsion Plant
0.9373 11.59 05)
0.994 (4.12e
0.8030 (5 99. 03)
0,5212 16.762 03)
0,2683
(2.51e-02)
0.5797
(1.88e (33)
0,9941 (1.25e 03)
Power Plant
0.9646 (1314e-33)
0.9614
(1.26e 03)
0.4008 (1.12c 02)
0.432 (1.47c 02)
(1.43e 02)
0.9626
(1.13e (33
0.9623 (1.50e 03)
Proncin Tertacy Simucture
0.9619 (4.71c-04)
0,999
(472--2)))
0-4125 (2 433
0.3846 (1 36:-02)
0.1182 (1 35c-00)
0.9609
(2.27c-413
0,9559
(1.72a-85)
Wine Quality Red
0.9425 (2. 22 03)
0.9472
(3:25-003)
0.3919 (1 (2)
0.3566
(1 (3--02)
0.1616 (7.45e-0))
0,9059 (8. (9e-03)
0.8647
(8.77e- 03)
Yacht Hlddoognamiss
0 19449 (7.86e-05)
0.9726 (6 736-05
0.161 (7. (7.3
0.3871 (2882-02)
0.2081 (2.54e-00)
0,9607 (6.97e-033
0,9899 (6.03e-83)
Table 1: The average coverage of six methods across nine datasets with the standard error over 20 cross validation folds in parentheses.
Dataset Method
Linear Regresiio
GP
Ensemble
Dneposst
LL Drepost
SVI
LL
Rodea Hossing
2.0424 (6878-48)
188716 1.776611176--2
0.4422 (7 (288--03
0.6882 (2. 196-0)))
0.1855 (2 (55-402)
1.301 (2.566-02)
1.148 (2, 36E-00)
Canciete
2.4562 (2:228-6))
(3.326--)3
0.4776 (9.538-055 433)
1.0342 (1 (.779)-22
0,1028 (2 (22044402)
1.5116 (1.726-0))
1.2293 (1 (1-415-02)
Energy
1.148 (2 (2229-40)
10773 (2 645-01)
0.2394 ( 5-8556000 033
0.5938 (1 226-200
0. 0.14177..61800 (B-02)
0.426 (1.335-00)
0.7974 (1 956-022
KinBrm
3,0090 (9 (9-268-04)
23795 (7 (7.00--0)
0.5403 ( 376-03)
1 (1.37 37F-00)
0:2024
(1 225-02)
1 16697 (2.78P-03)
1,2634 (2. (96-99--)
Naval Prepulsion Plane
1.5551 (2.128-04)
0.300 (1 (0F000))
0.0088 (4 868-09)
1.1998 (6-45E-01)
0:2281
(1.856-42)
1.3064 (1 (.P88-01)
0.488 448-00)
Power Plant
1.9475 (7.098-444)
09768 (9 (3683--44)
0.2494 (6. 728-03)
0.3385 (1 40E-22)
10035 (1.888,08)
0.9815 (3 (3.645-05)
Proteit Terriary Sirccture
3.3182 (3.28113.211 (3.21D-04) 04)
5.2123 (3.436-03
0,6804 (). (3.77E--0 035
09164(1-41E000
0:3454 (1. 9201-02)
2.9535 (3.820-00 (0)
2.6605 (2. 201 02)
Wine Quality Red
(.1575(1.828000
3.1829 (4.096-01)
0,7763 (1 31E-02)
0.7841 (2.91E-02)
0:5481
(1.611-02)
2. 2.3469 (2.726-00)
2,3697 (2. 200-0-000 02))
Yachi Hydredyramiis
2. 2,3635 (2.895-03)
(757447)57--00)
0.4455 (9 768-03)
0.5443 (2.227-00
0,1081 (0.855-03)
0.677 (3.546-03)
0.69 (3.11P-00)
Table 2: The average width of the posterior prediction interval of six methods across nine datasets
with the standard error over 20 cross validation folds in parentheses. Width is reported in terms of standard deviations of the response variable in the training set.
Mettod
Regresioo
Dropent
[T. Dropont
SVT
LLS
Biston Housing
4.0592 (1.226-01)
3,5397 (2. (2.30--1)
3.1484 31E-01
4,9654 (1 201277---0
3,6251 (1 (1.61E-01)
(197E-B)))
3.4225 (1333-111
7.6025 (1.128-0))) 011
7,8245
5.8107 18E
10.3653 19.00E-00 02)
6.5621 (.200-01)
5.6199 (1.47E-011
6.4618 26..26E.0 011
Energy
2,3029 (6.638 02)
2.7454 (5 (5.68E
1,9922
42))
3,172
1.5211 (6.17E-02)
1,2781
2.7032
(5.47E 02)
KinBom
(K0055-04)
0.1366
0.0855 (2.735-04)
0,2007 696--44
1-511-)))
0.6116 (5.400
0 1091 (77-055)
Propulsion Plant
0.8054 (45545555)
(3.43E-55)
(3:065-05)
0.0059
0.006 (3.038)44)
0,0012 (4:55-005)
448-04)
Plant
4.5639 (5.67E:02)
4.2551 (3.4265-02
4,2952 (2.625-02)
4.5793
4.7504
31E-0)
4,19K3
(4.68--08 -021
4.2503 (3.56--22)
Protcia Tortary
4.514 (.06E-02)
4.9695
4.2396 (7.038:-43)
5,2122
4.5219 (2.7HE-02)
4.3824
(1.933-0))
4.645% 122444--00 02)
Quality Red
(6.56E 03)
0.6432
0,643 (6.411-33
(5.0--5)
0.6555 (6.6HE-03)
0.647 (8.95
0.6762
02)
Yachi Hydroddmamicss
4,4647 (1.40E 01)
4.7585
3,1594 (3.95E (2)
9,4761 (2.648
2.5562 (2.47E-01)
2.765 (1
2.6775 (9.06E 02)
Table 3:
The average RMSE of six methods across nine datasets with the standard error over 20
cross validation folds in parentheses. These values are comparable to other reported in the literature for these benchmarks (Gal & Ghahramani] 2016)), though the intention was not to produce state of the art results, but merely demonstrate the models were trained in a reasonable manner.
4.2 MNIST
We begin by calculating coverage and width for predictions from Ovadia al. (0019) on MNIST and shifted MNIST data. Ovadia et al. (2019) used LeNet architecture and we reter to their manuscript for more details on their implementation.
how coverage and width co-vary as dataset shift increases. We observe high coverage and low width for all models on training, validation, and non-shifted test set data. The elevated width for SVI on these dataset splits indicate that the posterior predictions of label probabilities were the most diffuse to begin with among all models. In Figure 3 all seven models have at least 0.95 coverage with 15 degree rotation shift. Most models don't 't see appreciable increase in the average width of the 0.95 prediction set, except for SVI. The average width for SVI jumps to over 2 at 15 degrees rotation. As the amount of shift increases, coverage decreases across all methods in a comparable way. SVI maintains higher levels of coverage, but with a compensatory increase in width.
5

---

Under review as a conference paper at ICLR 2020
1.00
3.0
Methed
0.75
2.5
Dropout
Coverage
Ensemble
LL Dropout
0.50
2.0
LL Svi
0.25
1.5
Temp Scaling
0.00
1.0
4px
16pk
20px
2dpz
28px
4px
Bpx
12px
18px
20px
2
28px
Shif
Shift
Figure 4: The effect of translation shifts on coverage and width in CIFAR-10 images. Coverage remains robust across all pixel shifts, while width increases.
In Figure 3 we observe the same coverage-widte pattern at the lowest level of shift, 2 pixels. All methods have least 0.95 coverage, but only SVI has a distinct jump in the average width of its prediction set. The average width of the prediction set increases slightly then plateaus for all methods but SVI as the amount of translation increases.
For this simple dataset, SVI outperforms other models with regards to coverage and width properties. is the only model that has an average width that corresponds to the amount of shift observed.
Method
Coverags (SE)
Test
Mean Set
Width (SE)
Coverage (SE)
Rotation Shift
Mean Rotation Width (SE)
Mean Translation Shift Coverage (SE)
Width (SE)
Translation
Dropour
0.9987 (6. 328-05)
1,06 (1. 380-14)
0,5519 (2.918402)
2.5279 (6 64E-02)
0,5333 (3 54F-02)
2,3527 (6 34F:02)
Ensemble
0.9984 (7.07E-05)
1.0424 (2.07E-04)
0,5157 (3.119-02)
2,0892 (5.44E.02)
0.5424 (3.33)002
2.3276 66F:42)
L1. Dropout
0.9985 (1 (55-044)
1,0561 89E-03)
0.552 (2.935-02)
2.5162 (6.733-22)
0.5388 (3. 52E-02)
2. 2,365% 66E-02)
11.8VI
0.9964 (1 14H-044
1.0637
656-01)
0.5746 (2.778-02)
2.6324 (8. (8.41)-022
0.535 (3.5 (1.51F-422)
2. 2.3294 (6, 46F-02)
SVI
0.9997
(7.35E-05)
1,5192
(2.19E-02)
0.7148
(2.665-02)
4,8549 (L.ME-0L)
0.754 (1.96E-02)
5,6803 (9EE-0))
Temp scaling
9.9986
(.366-44)
98E-033
0.5243 (3.106-02)
2.2683 (6.177)-22
0,5375
(6.21))02)
Vanilla
0,9972 (1.16E-04)
1.032 (9.06E-01)
0.4715 (3,28B-02)
1.7492 (3.78E-02)
0.4798 (3.50E-02)
1,801 (3.84E-02)
Table 4: MNIST average coverage and width for the test set, rotation shift, and translation shift.
Machod
Coverage
Exeemble
0.50
030
LL Depoutt
0.25
8. 5.
3.
Shitt
Shif
Dispoct
Wicth
Emerta
Terp Scalig
Var
Figure 3: The effect of rotation and translation on coverage and width, respectively, for MNIST.
4.3 CIFAR-10
Next, we consider more complex image dataset, CIFAR-10. Ovadia al. (2019) trained 20 layer and 50 layer ResNets. lation shifts. Temperature scaling and ensemble, in particular, have at least 0.95 coverage for every translation. We find that this high coverage comes with increases in width as shift increases. Figure 4 shows that temperature scaling has the highest average width across all models and shifts. All models have the same pattern of width increases, with peak average widths at 16 pixels translation.
shows how all seven models have high coverage levels over all trans-
Between the models which satisfy 0.95 coverage levels on all shifts, ensemble models have lower width than temperature scaling models. Under translation shifts CIFAR-10, ensemble methods perform the best given their high coverage and lower width.
Additionally, we consider the coverage properties of models 16 different corruptions of CIFAR- 10 from Hendrycks and Gimpel (Hendrycks & Gimpel (017).. Figure shows coverage vs. width over varying levels of shift intensity. Models that have more dispersed points to the right have higher

---

Under review as a conference paper at ICLR 2020
1.00
Mothod
Dropout
Ensomble
Coveraga
LL Dropout
0.50
LL Svi
BM
Temp Scoling
0.00
Texll
6
Width
Figure 5: The effect of corruption intensity on coverage levels vs. width in CIFAR- 10-C. Each facet panel represents a different corruption level, while points are the coverage of model on one of 16 corruptions. Each facet has 80 points per method, since 5 iterations were trained per method. For methods with points at the same coverage level, the superior method to the left it has lower width.
widths for the same level of coverage. An ideal model would have a cluster of points above the 0.95 coverage line and be far to the left portion of each facet. For models that have similar levels of coverage, the superior method will have points further to the left.
Figure 5 demonstrates that the lowest shift intensity, ensemble models, dropout, temperature scaling, and SVI were able to generally provide high levels of coverage most corruption types. However, as the intensity of the shift increases, coverage decreases. Ensembles and dropout models have for at least half of their 80 model-corruption evaluations at Ieast 0.95 coverage up the third intensity level. At higher levels of shift intensity, ensembles, dropout, and temperature scaling consistently have the highest levels of coverage. Although these higher performing methods have similar levels of coverage, they have different widths. See Figure Al] for a further examination of the coverage and widths of these methods.
Method
Mean Test Set Coverage (SE)
Mean Text Set Width (SE)
Mean Corruption Coverage (SE)
Mean Curruption Widkh (SE)
Dropout
0.987 (3.726-04)
1.578 (2.68E-03)
0.886 (6 (4E-03)
2,313 (3 (3.033-02))
Ensemble
0.992 (9.70F.05)
1.492 (1.52F.03)
0.911 (6F.6)))))
2.425 (3.69F-02)
LL Drapout
0.960 (8.77E-04)
1,301 (3.99E-013 03)
0.815 (7 (7.48E-03)
1.699 (1 53E-02)
LL SVI
0.964 (6.64E-04)
1.258 (2. 2.660--03)
0.817 (7.52E-03)
1.781 (2.15E-02)
SVI
0.976 (5, (0E-04))
1.558 (6.31E--33
0.881 (5 (5E45)))3)
2.161 (2.32E-02)
Temp Scaling
0.985 (4.54E-04)
1.599 (1.19E)02)
0.899 (4.83E-03)
2.636 (3 86E-022)
Vanilla
0.964 (6.36E-04)
(3.61(3.90E-03)
0.823 (7. 17E00))3)
1,790 (2. (2.16E)02)
Table The mean coverage and widths on the test of CIFAR- 10 as well on the mean coverage and width averaged over 16 corruptions and intensities.
4.4 IMAGENET
Finally, we analyze coverage and width on ImageNet and ImageNet-C from Hendrycks Gimpel (017). Figure O shows similar coverage vs. width plots to Figure 5 We find that over the 16 dif- ferent corruptions at 5 levels, ensembles, temperature scaling, and dropout models had consistently higher levels of coverage. Unsurprisingly, T shows that these methods have correspondingly higher widths. At the first three levels of corruption, ensembling has the lowest level of width of the top performing methods (see Figure A2). However, at the highest two levels of corruption, dropout has lower width than ensembling. None of the methods have a commensurate increase in width to maintain the 0.95 coverage levels seen in-distribution test data dataset shift increases.

---

Under review as a conference paper at ICLR 2020
1.00
Mothod
Deopout
Coveraga
0.50
Dropout
Temp Scaling
Vanillu
0.00
Texl
0 100 250 300 400
100 200 300
400
100 200
300
403
100 200 300
400
100 200 300 400
100 200 800 400
Width
Figure 6: The effect of corruption intensity coverage levels vs. width in ImageNet-C. Each facet panel represents a different corruption level, while points are the coverage of model on one of 16 corruptions. Each facet has 16 points per method, only only 1 iteration For methods equal coverage, the superior method is to the left as it has a lower width.
trained per method.
Method
Mean Test Set Coverage
Mean Test Set Width
Mean Corruption Coverage (SE)
Mean Corupttin Width (SE)
Dropout 0.9613
13.2699
0.8579 (1.61E-02)
87.5784 (7. 7.80=+000)
Ensemble
0,9701
13.0613
0.9231 (7.13E-08)
105.3608 (8.57E+00)
I.I. Dropout
0.9552
10.7707
0.8688 (1. (1)88F002
88 0326 (8. 04F++00)
LL SVI 0.9327
10.5624
0.777 (1.76E-02)
65 99.9982 (5.01E+00)
Temp Scaling
0,9613
15,4811
0.8829 (1. (0E-0)))
1459 (8. 43))+00)
Vanilla
0.9525
11,0255
0.8529 (1.27E-02)
80.687 (7.16E+00))
Table 6: The mean coverage and widths the test set of ImageNet well on the coverage and width averaged over 16 corruptions and intensities.
DISCUSSION
We have provided the first comprehensive empirical study of the frequentist-style coverage proper- ties of popular uncertainty quantification techniques for deep learning models. In regression tasks, Gaussian Processes were the clear winner in terms of coverage across nearly all benchmarks, with smaller widths than linear regression, whose prediction intervals come with formal guarantees. SVI and LL SVI also had excellent coverage properties across most tasks with tighter intervals than GPs and linear regression. In contrast, the methods based on ensembles and Monte Carlo dropout had significantly worse coverage due to their overly confident and tight prediction intervals. Another interesting finding is that despite higher levels of uncertainty (e.g. larger widths), SVI was also the most accurate model based on RMSE as reported in Table
In the classification setting, all methods showed very high coverage in the i.i.d setting (i.e. no dataset shift), as coverage reflective of top-1 accuracy this scenario. On MNIST data, SVI had the best performance, maintaining high levels of coverage under slight dataset shift and scaling the width of its prediction intervals more appropriately as shift increased relative other methods. On CIFAR- 10 data, ensemble models were superior. They had the highest levels of coverage at the third of five intensity levels on CIFAR-10-C data, while have lower width than the next best method, temperature scaling. Dropout and SVI also had slightly worse coverage levels, but lower widths as well. Last layer dropout and last layer SVI performed poorly, oftentimes having lower coverage than vanilla neural networks.
In summary, we find that popular uncertainty quantification methods for deep learning models do not provide good coverage properties under moderate levels of datset shift. Although the width of prediction regions do increase under increasing amounts of shift, these changes are not enough to maintain the levels of coverage seen i.i.d data. We conclude that the methods evaluated for uncertainty quantification are likely insufficient for use in high-stakes, real-worke applications where dataset shift is likely to occur.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Rina Foygel Barber, Emmanuel Candès, Aaditya Ramdas, and Ryan J Tibshirani. The limits of distribution-free conditional predictive inference. March 2019. URL http::://axxvvvv/oe
abs /15//193... 64684
Charles Blundell, Julien Corneiise, Koray Kavukcuoglu, and Daan Wierstra. Weight uncertainty in neural networks. In International Conference Machine Learning, 2015.
Arindam Chatterjee and Soumendra Nath Lahiri. Bootstrapping lasso estimators. Journal of the American Statistical Association, 106(494)6608-00 625, 2011.
Bradley Efron. The jackknife, the bootstrap and other resampling plans. SIAM, 1982.
Yarin Gal and Zoubin Ghahramani. Dropout as bayesian approximation: Representing model uncertainty in deep learning. In International Conference on Machine Learning, 2016.
Alex Graves. Practical variational inference for neural networks. In Advances in Neural Information Processing Systems. 2011.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Weinberger. On calibration of modern neural networks. In International Conference on Machine, 2017.
Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and Out-of-Distributtio examples in neural networks. In International Conference on Learning Representations, 2017.
José Miguel Hernández-Lobate and Ryan P Adams. Probabilistic backpropagation for scalable learning of bayesian neural networks. International Conference on Machine Learning, 2015.
José Miguel
Yingzhen Li, Mark Rowland, Daniel Bernández-Lbbat., Thang Bui, and Richard E Turnr. Black-box a-divergence minimization. In International Conference on Machine Learning, 2016.
Diederík Kingma and Jimmy Ba. Adam: method for stochastic optimization. In International Conference on Learning Representations, 2015.
B J.K. Kleijn and A W van der Vaart. The BernsteinVoon-Mise Von-Mises theorem under misspecification. Electronic Journal of Statistics, 6:354-381, 2012.
Balaji Lakshninarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In Advances in Neural Information Processing Systems. 2017.
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. Nature, 521(7553):436 444, 2015.
Jeremiah Zhe Liu, Zi Lin, Shreyas Padhy, Dustin Tran, Tania Bedrax- Weiss, and Balaji Lakshmi- narayaman. Simple and principled uncertainty estimation with deterministic deep learning via distance awareness. In Advances in Neural Information Processing Systems, 2020.
C Louizos and M Welling. Multiplicative normalizing flows for variational Bayesian neural net- works. In International Conference of Machine Learning. 2017.
Christos Louizos and Max Welling. Structured and efficient variational deep learning with matrix Gaussian posteriors. In International Conference on Machine Learning, 2016.
Wesley Maddox, Timur Garipov, Pavel Izmailov, Dmitry Vetrov, and Andrew Gordon Wilson. A simple baseline for Bayesian uncertainty in deep learning. In Advances in Neural Information Processing Systems. 2019.
Radford M Neal. Bayesian Learning for Neural Networks. Springer, New York, NY, 1996.
Yaniv Ovadia, Emily Fertig. Jie Ren, Zachary Nado, Sculley, Sebastian Nowozin, Joshua V Dil- lon, Balaji Lakshminarayanan, and Jasper Snoek. Can you trust your model's 's uncertainty? Eval- uating predictive uncertainty under dataset shift. In Advances in Neural Information Processing Systems, 2019.

---

Under review as a conference paper at ICLR 2020
Nick Pawlowski, Andrew Brock, Matthew C H Lee, Martin Rajchl, and Ben Glocker. Implicit weight uncertainty in neural networks. 2017. URL http://axxiv..oo wtt.
01297
Carlos Riquelme, George Tucker, and Jasper Snoek. Deep bayesian bandits showdown: An empiri- cal comparison of bayesian deep networks for Thompson sampling. In International Conference on Learning Representations, 2019.
Rupesh Srivastava, Klaus Greff, and Jürgen Schmidhuber. Training very deep networks. In Advances in Neural Information Processing Systems. 2015.
Joost Amersfoort, Lewis Smith, Yee Whye Teh, and Yarin Gal. Uncertainty estimation using a single deep deterministic neural network. In International Conference on Machine Learning, 2020.
Larry Wasserman. All of statisties: a concise course in statistical inference. Springer Science & Business Media, 2013.
Yeming Wen, Paul Vicol, Jimmy Ba, Dustin Tran, and Roger Grosse. Flipout: Efficient Pseudo- Independent weight perturbations on Mini-Batches. International Conference on Learning Representations, 2018
Florian Wenzel, Kevin Roth, Bastiaan S Veeling, Jakub Swiętkowski, Linh Tran, Stephan Mandt, Jasper Snoek, Tim Salimans, Rodolphe Jenatton, and Sebastian Nowozin. How good is the bayes posterior deep neural networks really? International Conference on Machine Learning, 2020.
Jiayu Yao, Weiwei Pan, Soumya Ghosh, and Finale Doshi-Velez. Quality of uncertainty quantifi- cation for bayesian neural network inference. June 2019. URL 1906. 09686
org /bs/
10

---

Under review as a conference paper at ICLR 2020
A
APPENDIX
Code Availability
The code and data to reproduce our results will be made available after the anonymous review period.
Mean Test Set
Mean Test Set
Mean Translation Shift
Mean Translation Shift
Method
Coverage (SE)
Width (SE)
Coverage (SE)
Width (SE)
Dropout
0,9883 (3.79E-04 -04)
1.5778 (2. (2)
0,9696 (2.48E-03)
2.0709 (5, (5.11E02)
Ensemble
0,9922 (3.08E-04
1.4925 (1352E-03)
0,9806 (1 (655E-03)
1.9246 (4,49E-02)
LL Dropout
0,9628 (1.40E-03)
1.3007 (3.99E-03)
0,9184 (5.59E-03)
1.6678 (4.16E-02)
LL SVI
0,9677 (1.10E-03)
1,2585 (2.60E-03)
0,929 (4.55E-03)
1.5044 (2.61E-02)
SVI
0.9789 (6.41E-04)
1,5579 (6.31E-03)
0,9543 (2.89E-03)
1.9286 (3 (3.69E-02)
Temp scaling
0,9871 (3.51E-04)
1.5987 (1.19E-02)
0.9707 (1.97E-03)
2.1266 (5. (5.30E-02)
Vanilla
0.9686 (6.06E-04)
1.2611 (3.90E-03)
0.9296 (4.36E-03)
1.5064 (2.58E-02)
Table A1: CIFAR- 10 average coverage and width for the test set and translation shift.
1
0.5
(a) CIFAR-10 coverage
(b) CIFAR-10 10 width
Figure Al: The effect corruption intensity on coverage levels in CIFAR-10. This is averaged over 16 different corruption types. As shift intensity increases, coverage decreases and width increases. In general, Dropout, ensembling, and temperature scaling have the highest levels of coverage across corruptions levels.
Mattaa
2406 Hordl
(a) ImageNet coverage
(b) ImageNet width
Figure A2: The effect of corruption intensity coverage levels in ImageNet. This is averaged over 16 different corruption types. As shift intensity increases, coverage decreases and width increases. In general, Dropout, ensembling, and temperature scaling have the highest levels of coverage across corruptions levels.
11


Reviews:
[{"review_id": "g1QtYFTJ6s7", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Summary and key claims**\n\nThis paper provides a comprehensive evaluation of the empirical frequentist coverage properties of existing uncertainty quantification baselines on both regression and classification tasks. The paper focuses on frequentist coverage as a faithful metric for the quality of uncertainty estimates. The experimental evaluations in the paper imply that accurate out-of-distribution coverage is a hard target for most existing baselines; a problem exacerbated by settings were dataset shifts are prevalent. \n\n*The key contributions claimed by the paper are:*\n- Introduces coverage and width as a natural and interpretable metrics for evaluating predictive uncertainty.\n- Provides a comprehensive set of coverage evaluations for popular uncertainty quantification baselines.\n- Examines how dataset shift affects these coverage properties.\n  \n**Originality and Significance**\n\nFrequentist coverage is perhaps the most classical (and straightforward) measure of the quality of uncertainty estimates in statistics, so it's a bit odd that the authors claim the introduction of coverage and interval width as one of their key contributions. Despite not being as popular in the machine learning community, frequentist coverage has been considered in [R1] and [R2], and even coverage under dataset shifts was considered in [R3]. These existing papers not only consider frequentist coverage as a metric for uncertainty estimates, but they go as far as developing methods that provide theoretical guarantees on coverage. In fact, [R2] gives a more complete picture of uncertainty estimates by assessing both coverage and discriminative accuracy as both metrics do not necessarily correlate. \n\nI think that the key contribution of the paper is the experimental evaluations on many baselines and many datasets to analyze the performance of different methods with respect to coverage. While this analysis is interesting, it lacked insights into baselines' performances and the role of the evaluation metric used in assessing the comparative performances of baselines. For the most part, the experimental section was limited to reporting performance of all baselines on all datasets without providing insights into **why some methods perform better than others w.r.t this specific coverage metric** and **how the introduction of the coverage metric changes our perception on which methods are best**. I was expecting to see more evaluations that rank baselines w.r.t say calibration or Brier score, and then show that a ranking based on coverage would be significantly different, thereby motivating the usage of coverage in the uncertainty analysis toolbox. I would have also appreciated a breakdown of aleatoric and epistemic uncertainty, and how coverage may be a good metric for assessing either types of uncertainties, etc. Having read the experimental section---which is the key section in this paper---I was not exactly sure what to make of it. \n\nThe key take away of the experiments highlighted in the abstract and discussion is that uncertainty estimates do not generalize well to out-of-distribution samples. However, such finding is not new and has been discussed before in (Ovadia et al. (2019)). Also, it is not clear how the introduction of the coverage metric helps us arrive at this conclusion; it seems to me that the same conclusion could have been arrived at with calibration or AUC-ROC on out-of-distribution samples.\n\n**Technical comments**\n\nI have two main comments on the technical aspects of the paper:\n\n1) The authors found that GPs are clear winners when it comes to coverage. However, I am afraid that the frequentist coverage of Bayesian uncertainty (credible) intervals are extremely sensitive to the selection of the Bayesian prior (See the works of Szabo and van der Vaart in [R4] and references therein). Frequentist coverage is a specifically sensitive quantity in Bayesian analysis as a very large or very small prior length-scale of a GP kernel may give us very good or very bad coverage. The same issues are relevant (in a more subtle way) in Dropout NNs and any Bayesian NN approximation. Since most baselines considered in your frequentist analysis are actually Bayesian models, it is very important to report how robust are your findings to different selections of the priors (in this case, priors will correspond to hyperparameters). I did not find any discussion on the impact of hyperparameters in the resulting quality of uncertainty intervals and their impact on dataset shifts, despite this being a central concern in Bayesian models. A different approach for tuning hyperparameters may render models other than GPs come on top in your comparison.  \n\n2) Frequentist coverage is a concept associated with **regression** problems: we want a **contiguous** coverage set $C$ to contain the **real-valued** prediction target $y$ with a probability $1-\\alpha$. \n\nIn $K$-classification problems, the true real-valued target is the class probabilities $p_K$, and a confidence set in this case would comprise a $K$-simplex that covers the true class probability $1-\\alpha$ of the time. But the true class probabilities $p_k$ are never observed; we only observe discrete values for one out of K classes. So calculating empirical coverage of class probabilities is impossible in classification problems.\n\nThe authors extend the notion of coverage to classification in a different way: a coverage set $C$ is a discrete set of possible labels whose sum predicted probabilities add up to $1-\\alpha$, and coverage is achieved if the true label belongs to this set (Equation (2)). I find this definition incomplete because your coverage set $C$ is not contiguous anymore; it may contain labels 1 and $K$ and excludes 2,...,$K-1$. As you can see, in this scenario a coverage set wouldn't make sense unless the targets 1 to $K$ are unordered. So I think you have to say that these applies only to unordered categorical targets for this to make sense.\n \nAlso, I do not see how this definition would work for binary classification, which is always ordered? In the case of binary classification, it seems to me that calibration is actually a more expressive metric than coverage as it accounts for class probability even when $K=2$.\n\n**References**\n\n[R1] Rina Foygel Barber, Emmanuel J Candes, Aaditya Ramdas, Ryan J Tibshirani, \"Predictive inference with the jackknife+\", arXiv, 2019.\n\n[R2] Alaa, Ahmed M., and Mihaela van der Schaar. \"Discriminative jackknife: Quantifying uncertainty in deep learning via higher-order influence functions.\" ICML (2020).\n\n[R3]  Tibshirani, R. J., Barber, R. F., Candes, E., & Ramdas, A. (2019). Conformal prediction under covariate shift. In Advances in Neural Information Processing Systems (pp. 2530-2540).\n\n[R4] Botond Szabó, A. W. van der Vaart, and J. H. van Zanten, Frequentist coverage of adaptive nonparametric Bayesian credible sets, Annals of statistics, 2015.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "Mdt_15ElLcc", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Paper provides an  evaluation of the reliability of confidence levels of well known uncertainty quantification techniques in deep learning on classification and regression tasks. The question that the authors are trying to answer empirically is: when a model claims accuracy at a confidence level within a certain interval , how often does the actual accuracy   fall within  that interval?  This is conceptually similar to the recent slew of papers seeking to empirically evaluate the softmax calibration of deep models where the question there is how often do predicted probabilities of the winning class reflect the true probability of the correct answer, but in this paper the focus is on confidence level and confidence intervals.  \n\nStudies are conducted for both regression and classification.  Confidence levels and intervals are evaluated using the notion of coverage probability and width. While these have a straightforward interpretation in the regression setting, for classification the authors use the top K probabilities that captures 95% of the prediction probability mass to evaluate coverage and width. Thus for classification, the width is the number of classes over which 95% of the probability is smeared. Ideally one would want a model that has a low width, and high coverage probability (i.e a model that is both reliable and accurate). The aim of the paper is not to produce this ideal model, but rather to  empirically evaluate whether the predictive uncertainty of various methods proposed in the DL literature can be relied upon.  Various UQ methods are tested for both regression and classification datasets, and for the latter case, also under dataset shift.\n\nPros:\n+ Paper is well written and ideas are, for the most part,  presented well.\n+ Experiments test a variety of  state-of-the-art UQ methods.\n+ There has not been work looking at this specific metric -- i.e., the reliability of prediction intervals. And with increasing usage of DL in high-risk applications, an evaluation of this kind might be useful.\n\nCons\n- The authors appear to be conspicuously avoiding much usage of the terms \"confidence levels\" and \"confidence intervals\", but it appears that this is really what the paper is about.  Justify why you are taking this stance. The section on \"theoretical coverage guarantees\" is not sufficiently explanatory or convincing in this regard. \n\n- A quantitative discussion on the mismatch between the coverage probability and the quality of softmax calibration is missing. \n\n- My biggest concern is the conclusion of the paper: the authors state \"we conclude that the methods we evaluated for\nuncertainty quantification are likely insufficient for use in high-stakes, real-world applications where dataset shift is likely to occur.\" Yes, the models' coverage probabilities  are indeed significantly below the reported confidence level when data is corrupted (both for CIFAR-10 and ImageNet), but the fact that the width increases should give us an attack vector into the problem. You say this is not sufficient, but I'm not convinced this is the case.  95% of the probability mass is now smeared over a much larger number of classes. In other words, an increasing width necessarily means the predictions have increased in entropy, and also  that the probability mass in the winning class is now significantly lower under data corruption than what it was for the clean set. Both of these quantities (entropy and winning softmax) can be used to filter out predictions when the model is not confident (subject to a suitable confidence threshold), at least in the non-adversarial case. And in the real-world, this could be a practical approach to ascertain when a model's predictions should be trusted or discarded.\n\nIn summary, while the authors have done a commendable job with experimental evaluations, the conclusion is too strong and -- in my opinion -- incorrect to justify acceptance.", "extra_scores": {}, "extra_text": {}}, {"review_id": "EexjCtRnul", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In the submitted manuscript, \"Empirical Frequentist coverage of deep learning uncertainty quantification procedures\", the authors propose to investigate the Frequentist coverage properties of predictive intervals by numerical experiment for a number of machine learning models applied to benchmark datasets.  I can't say that I find this a strong submission because:\n1. the authors give a confused (mis-)definition of coverage; essentially they seem to have taken Barber et al.'s definition of \"marginal distribution free prediction intervals\", mangled it and then called it Frequentist coverage citing Wasserman\n2. the authors claim one of the contributions of this manuscript to be \"introduce coverage and width as a natural and interpretable metrics for evaluation predictive uncertainty\" but in fact these aspects of predictive intervals from ML models has been studied for many years, as a simple google search will confirm\n3. the results shown will not generalise in any meaningful sense: for example, GPs are found to have excellent coverage over the set of regression tasks shown, but in fact GPs are themselves a case study in the difficulties of achieving Frequentist style coverage in the domain of Bayesian non-parametrics (e.g. Hadji & Szabo 2019; Neiswanger & Ramdas 2020; Rousseau 2016 ; prior over-smoothing being the root of many problems ).", "extra_scores": {}, "extra_text": {}}, {"review_id": "3pUPyzAUWp", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "## Summary\nThe authors compare empirical frequentist coverage of predictive intervals for several uncertainty quantification methods.  The paper covers both classification and regression.  The authors define an analogue of a confidence interval for classification.  Coverage properties are also studied under covariate shift between training and test sets.\n\n## Pros\nCoverage and width are a standard benchmarks for uncertainty quantification in statistics, and to my knowledge, this is the first work that undertakes a large-scale comparison for deep learning models.  Some inspiration seems to have been drawn from Ovadia et al. 2019 in that the set of methods compared are similar and the same architectures are used.  However, this work makes an important contribution in focusing on coverage / width, which I would agree are more interpretable metrics for practitioners.  The set of methods spans several important strains of the literature: ensembling, Bayesian approximation, Dropout, GPs.\n\n## Cons\nThe work is timely and of broad interest; however, I think the presentation of results still needs some refining.  Given that this paper focuses on empirical results, I would suggest the authors spend more time developing effective visualizations to communicate their conclusions.\n\nTables 1 2 and 3 are perhaps necessary as a reference, but cry out for a visual aid.  The authors state \"We see that higher coverage correlates with a higher average width.\"  This is something that seems like it could be communicated more immediately with the right plot.\n\nFigure 4 conveys some visual trends, but also could be improved.  The coverage plot contains mostly blank space.  The dots are clustered together and impossible to differentiate.  In the width plot, what is communicated is that all methods have wider intervals with more shift.  However, it is hard to differentiate the methods: again, the dots are on top of each other, the colors blend such that they do not seem to actually correspond to the colors on the legend (perhaps alpha should not be used here?).\n\nFigure 5 has similar problems.  The use of alpha means that colors blend together and cannot be looked up in the legend. As before, the dots in the legend are tiny, so it is hard to differentiate the shades even in the legend. What is visually communicated is the spread of performance, but conclusions about any particular method are nearly impossible.\n\nClearly this is a difficult problem to solve: there are 7 methods, and several axes of variation (coverage, width, shift).  However, the plots at the moment do not convey much information aside from overall trends. Visual understanding method-specific results is not possible at the moment.\n\nI would also suggest that the authors devote more attention to the definition of coverage for predictive intervals, and how it relates to distribution shift.  For example, the authors define coverage as equation (1) holding for any distribution P.  It is not explicitly stated, but the implication here seems to be that the set $\\hat{C}_n(x_n)$ is determined from training data distributed as P, and coverage is measured from data drawn from the same distribution (i.e. this definition does not allow covariate shift).  It would be useful for the authors to state in mathematical terms, what it means for coverage to hold under covariate shift.  Is this equivalent to the notion of conditional coverage as defined in Barber et al. 2020?  These are subtle enough concepts that I think they should be more precisely spelled out in the paper, even if some intuitive definition of covariate shift is widely understood. Clearly we cannot expect coverage to hold under a distribution shift that changes the conditional P(Y | X) between training and eval. What are the limits of what the authors allow?\n\nI would also suggest that the authors might include an explicit analysis of conditional coverage.  For example, all methods seem to enjoy 95% coverage for in-distribution eval sets.  However, it would be interesting to know if this coverage is uniform across classes or any other useful clustering of the data.  \n\n### A few specifics:\n*    The authors' analogue of confidence interval for a classifier is novel to me, and is a convenient way to unify the presentation of results between classification and regression.  If this is a novel definition, I would suggest the authors more explicitly point this out, as future literature may use it and should cite it.\n*   Figure 4 is out of order with figure 3 - this needs to be fixed.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Overall, the reviewers agree that there is definite value in the empirical evaluation you have provided. However, as you have acknowledged in your responses to the reviewers, the presentation could be significantly improved. A final point that was not touched upon by the reviewers--where possible (e.g. certainly not ImageNet, but for some of the smaller datasets in Table 1) it would be helpful to have a comparison to fully Bayesian methods (you have linear regression and GPs, but I don't see the implementation details; my suggestion is to implement these within an MCMC framework, specifying reasonable priors over the (hyper)parameters).
Decision: Reject