Pitfalls of In-Domain Uncertainty Estimation and Ensembling in Deep Learning
Authors: Arsenii Ashukha
Alexander Lyzhov
Dmitry Molchanov
Dmitry Vetrov
Venue:  (2020)
Keywords: uncertainty
in-domain uncertainty
deep ensembles
ensemble learning
deep learning

Abstract: Uncertainty estimation and ensembling methods go hand-in-hand. Uncertainty estimation is one of the main benchmarks for assessment of ensembling performance. At the same time, deep learning ensembles have provided state-of-the-art results in uncertainty estimation. In this work, we focus on in-domain uncertainty for image classification. We explore the standards for its quantification and point out pitfalls of existing metrics. Avoiding these pitfalls, we perform a broad study of different ensembling techniques. To provide more insight in this study, we introduce the deep ensemble equivalent score (DEE) and show that many sophisticated ensembling techniques are equivalent to an ensemble of only few independently trained networks in terms of test performance.

Published as a conference paper at ICLR 2020
PITFALLS OF IN-DOMAIN UNCERTAINTY ESTIMATION AND ENSEMBLING IN DEEP LEARNING
Arsenii Ashukha
Alexander Lyzhov
Samsung Al Center Moscow, HSE eashukha@bayesgroup. ru
Samsung AI Center Moscow, Skoltech! HSE$
alyzhov@bayysgrropp. ru
Dmitry Molchanov
Dmitry Vetrov
Samsung Al Center Moscow, HSE dmolch emolchhbayessrropp ru
HSE Samsung Al Center Moscow
cvetrovebayesgroup.o ru
ABSTRACT
Uncertainty estimation and ensembling methods go hand-in-hand. Uncertainty estimation one of the main benchmarks for assessment of ensembling perfor- mance. At the same time, deep learning ensembles have provided state-of-the-a results in uncertainty estimation. In this work, we focus on in-domain uncertainty for image classification. We explore the standards for its quantification and point out pitfalls of existing metrics. Avoiding these pitfalls, we perform a broad study of different ensembling techniques. To provide more insight in this study, we introduce the deep ensemble equivalent score (DEE) and show that many sophisticated ensembling techniques are equivalent independently trained networks in terms of test performance.
ensemble of only few
video code / blog post.
INTRODUCTION
Deep neural networks (DNNs) have become one of the most popular families of machine learning models. The predictive performance of DNNs for classification is often measured in of accu- racy, However, DNNs have been shown to yield inaccurate and unreliable probability estimates, or predictive uncertainty (Guo et al., 2017). This has brought considerable attention the problem of uncertainty estimation with deep neural networks.
There are many faces uncertainty estimation. Different desirable uncertainty estimation properties of model require different settings and metrics to capture them. Out-of-domain uncertainty of the model is measured on the data that does not follow the same distribution as the training dataset (out-of-domain data). Out-of-domain data can include images corrupted with rotations or blurring, adversarial attacks (Szegedy et al., 2013) or data points from a completely different dataset. The model is expected to be resistant to data corruptions and to be more uncertain out-of-domain data than on in-domain data. On the contrary, in-domain uncertainty of the model measured on data taken from the training data distribution, i.c. data from the same domain. In this setting the model is expected to produce reliable probability estimates, e.g. the model shouldn't 't be too overconfident in its wrong predictions.
Pitfalls of metrics We show that many common metrics of in-domain uncertainty estimation (e.g. log-likelihood, Brier score, calibration metries, etc.) are either comparable across dif- ferent models or fail to provide a reliable ranking. We address some of the stated pitfalls and point out more reasonable evaluation schemes. For instance, although temperature scaling is not a standard for ensembling techniques, it is must for fair evaluation. With this in mind, the
"Equal contribution $HISE refers National Research University Higher School of Economics
Skoltech refers to Skolkovo Institute of Science and Technology
1 HHE refers Samsung-HSi HSE Lahoratory, National Research University Higher School of Economics

---

Published as a conference paper at ICLR 2020
calibrated log- aigllikelihood avoids most of the stated pitfalls and generally is reasonable metric for in-domain uncertainty estimation task.
Pitfalls of ensembles Equipped with the proposed evaluation framework, are revisiting the eval- uation of ensembles of DNNs- one of the major tools for uncertainty estimation. We introduce the deep ensemble equivalent (DEE) score that measures the number of independently trained models that, when ensembled, achieve the same performance as the ensembling technique interest. The DEE score allows us compare ensembling techniques across different datasets and architectures using a unified scale. Our study shows that most of the popular ensembling techniques require aver- aging predictions across dozens of samples (members of ensemble), yet are essentially equivalent to an ensemble of only few independently trained models.
Missing part of ensembling In our study, test-time data augmentation (TTA) turned out to be surprisingly strong baseline for uncertainty estimation and a simple way to improve ensembles. Despite being a popular technique in large-scale classification, TTA seems be overlooked in the community of uncertainty estimation and ensembling.
2 SCOPE OF THE PAPER
We use standard benchmark problems of image classification which comprise a common setting in research learning ensembles of neural networks. There are other relevant settings where the correctness of probability estimates can be a priority, and ensembling techniques are used to improve it. These settings include, but not limited to, regression, language modeling (Gal, 2016), image segmentation (Gustafsson et al., 2019), active learning (Settles, 2012) and reinforcement learning (Buckman et al., 2018; Chua al., 2018).
We focus on in-domain uncertainty, as opposed to out-of-domain uncertainty. Out-of-domain un- certainty includes detection of inputs that come from a completely different domain or have been corrupted by noise or adversarial attacks. This setting has been thoroughly explored by (Ovadia al., 2019).
We only consider methods that are trained on clean data with simple data augmentation. Some other methods use oft-of-domain data (Malinin & Gales, 2018) or more elaborate data augmentation, e.g. mixup (Zhang et al., 2017) adversarial training (Lakshminarayanan et al., 2017) to improve accuracy, robustness and uncertainty.
We use conventional training procedures. We use the stochastic gradient descent (SGD) and use batch normalization (loffe Szegedy, 2015), both being the de-facto standards in modern deep learning. We refrain from using more elaborate optimization techniques including works super- convergence (Smith & Topin, 2019) and stochastic weight averaging (SWA) (Izmailov et al., 2018). These techniques can be used drastically accelerate training and to improve the predictive perfor- mance. Thus, we do not comment on the training time of different ensembling methods since the use of these and other more efficient training techniques would render such comparison obsolete.
number of related works study ways of approximating and accelerating prediction in ensembles. The distillation mechanism allows to approximate the prediction of an ensemble by single neural network (Hinton et al., 2015; Balan et al., 2015; Tran et al., 2020), whereas fast dropout (Wang & Manning, 2013) and deterministic variational inference (Wu et al., 2018) allow to approximate the predictive distribution of specific stochastic computation graphs. We measure the raw power of ensembling techniques without these approximations.
All of the aforementioned alternative settings are orthogonal to the scope of this paper and are promising points of interest for further research.
PITFALLS OF IN-DOMAIN UNCERTAINTY ESTIMATION
No single metric measures all the desirable properties of uncertainty estimates obtained by model of interest. Because of this, the community is using many different metrics in an attempt to cap- ture the quality of uncertainty estimation, such the Brier score (Brier, 1950), likeliked (Quinonero-aanded et al., 2005), metrics of calibration (Guo et al., 2017: Nixon et al., 2019), per- formance of misclassification detection (Malinin & Gales, 2018), and threshold-accural curves
2

---

Published as a conference paper at ICLR 2020
-07
Calibrated log log-llkellhool (dashed)
-0.8
Log -ellilllhhod (solid)
3.0
VI
-09
K-FAC-L Single net
2.5
Deep
2.0
100 bins
50 bins
ensemble
Deep ensemble (lect-ime aug.)
1.5
13
25
10-1
10
10
samples (ensemble size)
Threshold
Figure 1: The average log-likeldhood of two different ensembling techniques for ResNet50 on ImageNet dataset before (solid) and after (dashed) temperature scaling. Without the tem- perature scaling, test-time data augmentation de- creases the log-likelihood of plain deep ensem- bles. However, when the temperature scaling is enabled, deep ensembles with test-time data aug- mentation outperform plain deep ensembles.
Thresholded adaptive calibration error (TACE) is highly sensitive to the threshold and the number of bins. It does not provide consis- tent ranking of different ensembling techniques.
Figure 2:
Here TACE is reported for VGGI6BN model on
CIFAR- dataset and is evaluated at the opti- mal temperature.
(Lakshminarayanan et al., 2017). In the section we highlight the pitfalls of the aforementioned met- rics, and demonstrate that these pitfalls significantly affect evaluation, changing the ranking of the methods.
Notation We consider a classification problem with a dataset that consists of N training and n testing pairs (I,, vt p(x, y), where 2; is object and y E fI,..., C) is discrete class label. A probabilistic classifier maps an object 2; into a predictive distribution p(y) æ;). The predictive distribution p(y Lx,) of deep neural network is typically defined by the softmax function p(y) x) (alimax(((x/TTT))) where =(x)) is vector of logits and T is a scalar parameter standing for the temperature of the predictive distribution. This scalar parameter is usually set to T 1 or is tuned on validation set (Guo et al., 2017). The maximum probability max, p(y cl xx) is called the confidence of classifier P on an object æ4. 1[-] denotes the indicator function throughout the text.
3.1 LOG-LIKELIIOOD AND BRIER SCORE
The average test log-likelibann likelihood LL = - log ply v,5 xx)) is a popular metric for measuring the quality of in-domain uncertainty of deep learning models. It directly penalizes high probability scores assigned to incorrect labels and low probability scores assigned to the correct labels y
LL is sensitive the softmax temperature T. The temperature that has been implicitly learned during training can be far from optimal for the test data. However, a nearly optimal temperature validation data. This approach is called
can be found post-hoc by maximizing the log-likelihood temperature scaling or calibration (Guo et al., 2017). Despite its simplicity, the temperature scaling results in notable improvement the LL.
While ensembling techniques tend to have better temperature than single models, the default choice of T still suboptimal. Comparing the LL with suboptimal temperatures- that often the case in practice-apa can potentially produce an arbitrary ranking of different methods.
Comparison of the log-likelihood should only be performed the optimal temperature.
Empirically, we demonstrate that the overall ordering of methods and also the best ensembling method according the L.I. vary depending temperature T. While this applies most
3

---

Published as a conference paper at ICLR 2020
ensembling techniques (see Figure 10), this effect is most noticeable on experiments with data aug- mentation on ImageNet (Figure I).
We introduce new metric called the calibrated log-likelihood that is the log-likelihood at the optimal temperature.
The calibrated log- legllikeiihodd considers model and post-training calibration as unified system, targeting to measure all models the equal conditions of optimal temperature. That allows avoid measuring calibration error that can be eliminated by simple temperature scaling. The metric significantly affects the results of the comparison. For example, in Figure 10 the differences between Bayesian (VI, K-FAC, SWAG, dropout) and conventional non- non-Bayesian networks become much less pronounced, and in most cases making conventional non-Bayesian networks match the performance of Bayesian ones (VI, K-FAC, Dropout) ResNetl 10, ResNet1 64, and NeseResNet.
We show how to obtain an unbiased estimate of the calibrated log- log-likeldhood without a held-out validation set in Section 3.5.
LL also demonstrates a high correlation with accuracy (p > 0.86), that in case of calibrated LL be- comes stronger (p 0.95). That suggests that while (calibrated) LL. measures the uncertainty of the model, it still has a significant dependence on the accuracy and vice versa. A model with higher accuracy would likely have higher log- log-likelihood. See Figure 9 in Appendix C for more details.
Brier score BS = - as a metric for verification of predicted probabilities (Brier, 1950). Similarly to the log-likelihood, the Brier score penalizes low probabilities assigned to correct predictions and high probabilities as- signed to wrong ones. It is also sensitive to the temperature of the softmax distribution and behaves similarly to the log-likelihood pirical correlation for wide range of models on CIFAR-10, 10, CIFAR- 100 and ImageNet datasets (see Figure in Appendix C ).
-1 c c| xx))² has also been known for long time
a.
While these metrics are not strictly equivalent, they show high em-
3.2 MISCLASSIFICATION DETECTION
Detection of wrong predictions of the model, misclassifications, is popular downstream prob- lem relevant to the problem in-domain uncertainty estimation. Since misclassification detection is essentially binary classification problem, some papers measure its quality using conventional met- rics for binary classification such as AUC-ROC and AUC-PR (Malinin & Gales, 2018; Cui et al., 2019; Možejko et al., 2018). These papers use uncertainty criterion like confidence predictive entropy H pvy [x] as a prediction score.
While these metrics can be used assess the misclassification detection performance of a single model, they cannot be used to directly compare misclassification performance across different mod- els. Correct and incorrect predictions are specific for every model, therefore, every model induces its own binary classification problem. The induced problems can differ significantly, since different models produce different confidences and misclassify different objects. In other words, comparing such metrics implies comparison of performance of classifiers that solve different classification problems, Such metries are therefore incomparable.
AUCs for misclassification detection cannot be directly compared between different models.
While comparing AUCs is incorrect in the setting of misclassification detection, it is correct to compare these metrics in many out-of-domain data detection problems. In that case, both objects and targets of the induced binary classification problems remain the same for all models. All out- of-domain objects have positive label and all in-domain objects have a negative label. Note that this condition does not necessarily hold in the problem of detection of adversarial attacks. Different models generally have different inputs after an adversarial attack, so such AUC-based metrics might still be flawed.
4

---

Published as a conference paper at ICLR 2020
3.3 CLASSIFICATION WITH REJECTION
Accuracy-confidence curves are another way to measure the performance of misclassification detec- tion. These curves measure the accuracy on the of objects with confidence max, poy cl ar)) above a certain threshold т (Lakshminarayanan et al., 2017) and ignoring or rejecting the others.
The main problem with accuracy-confidence curves that they rely too much on calibration and the actual values of confidence. Models with different temperatures have different numbers of objects at each confidence level which does not allow for a meaningful comparison. To overcome this problem, one can swhich from thresholding by the confidence level to thresholding by the number of rejected objects. The corresponding curves are then less sensitive to temperature scaling and thus allow to compare the rejection ability in a more meaningful way. Such curves have been known accuracy- rejection curvey (Nadeem al., 2009). In order to obtain a scalar metric for easy comparisons, one can compute the area under this curve, resulting in AU-ARC (Nadeem et al., 2009).
3.4 CALIBRATION METRICS
Informally speaking, a probabilistic classifier is calibrated if any predicted class probability equal the true class probability according the underlying data distribution (see (Vaicenavicius al., 2019) for formal definitions). Any deviation from the perfect calibration is called miscalibration. For brevity, we will use Pr.c denote p(y (x2) in the current section.
Expected calibration error (ECH) (Naeini et al., 2015) a metric that estimates model miscalibration by bimning the assigned probability scores and comparing them to average accuracies inside these bins. Assuming Bm denotes the m-th bin and M is overall number of bins, the ECE is defined as follows:
Bml (aco(Bm) - conf (Bm))
ECE
(1)
where acc (Cc(B) 11 EiEB Marg max Pi.e = v/7 and con(( | 1 Eien Pist-
recent line of works on measuring calibration in deep learning (Vaicenavicius et al., 2019; Kumar al., 2019; Nixon al., 2019) outlines several problems of the ECE score. Firstly, ECE is a bi- ased estimate of the true calibration. Secondly, ECE-like scores cannot be optimized directly since they are minimized by model with constant uniform predictions, making the infinite temperature T +oo its global optimum. Thirdly, ECE only estimates miscalibration in terms of the maxi- mum assigned probability whereas practical applications may require the full predicted probability vector to be calibrated. Finally, biases of ECE on different models may not be equal, rendering the miscalibration estimates incompatible. Similar concerns are also discussed by Ding et al. (2019).
Thresholded adaptive calibration error (TACE) was proposed as a step towards solving some of these problems (Nixon et al., 2019). TACE disregards all predicted probabilities that are less than a certain threshold (hence thresholded), chooses the bin locations adaptively so that each bin has the same number of objects (hence adaptive), and estimates miscalibration of probabilties across all classes the prediction (not just the top-1 predicted class as in ECE). Assuming that BTA denotes the m-th thresholded adaptive bin and M is the overall number of bins, TACE is defined as follows:
TACE =
1
TTA
Jobjs( BT coonf (B TA ,c]l,
(2)
CM
71
where objs(BTA, = |BTAI- - Eie [[y = c) and (BTA BTA | EIEBTA Pi.e-
Although TACE does solve several problems of ECE and is useful for measuring calibration of specific model, it still cannot be used as reliable criterion for comparing different models. Theory suggests that is still biased estimate of true calibration with different bias for each model (Vaice- navicius et al., 2019). In practice, we find that TACE is sensitive to its two parameters, the number of bins and the threshold, and does not provide consistent ranking of different models, as shown in Figure 2.
3.5 CALIBRATED LOG-LIKELIHOOK AND TEST-TIME MOSS-VVLLIAATON
There are two common ways to perform temperature scaling using a validation set when training on datasets that only feature public training and test sets (e.g. CIFARs). The public training set might
5

---

Published as a conference paper at ICLR 2020
be divided into smaller training set and validation set. the public test set can be split into test and validation parts (Guo et al., 2017: Nixon et al., 2019). The problem with the first method is that the resulting models cannot be directly compared with all the other models that have been trained on the full training set. The second approach, however, provides an unbiased estimate of metrics such as log-likelihood and Brier score but introduces more variance.
In order to reduce the variance of the second approach, we perform a "test-time cross- cross-validation". We randomly divide the test into two equal parts, then compute metries for each half of the test set using the temperature optimized on another half. We repeat this procedure five times and average the results across different random partitions to reduce the variance of the computed metries.
A STUDY OF ENSEMBLING & DEEP ENSEMBLE EQUIVALENT
Ensembles of deep neural networks have become de-facto standard for uncertainty estimation and improving the quality of deep learning models (Hansen & Salamon, 1990; Krizhevsky et al., 2009; Lakshminarayanan et al., 2017). There are two main directions of training ensembles of DNNs: training stochastic computation graphs and obtaining separate snapshots of neural network parameters.
Methods based on the paradigm of stochastic computation graphs introduce some kind of random noise over the weights or activations of deep learning models. When the model is trained, each sample of the noise corresponds to member of the ensemble. During test time, the predictions are averaged across the noise samples. These methods include (test-time) data augmentation, dropout (Srivastava et al., 2014; Gal & Ghahramani, 2016), variational inference (Blundell et al., 2015; Kingma et al., 2015; Louizos & Welling, 2017), batch normalization (loffe & Szegedy, 2015; Teye et al., 2018; Atanov et al., 2019), Laplace approximation (Ritter et al., 2018) and many more.
Snapshot- based methods aim to obtain sets of weights for deep learning models and then to average the predictions across these weights. The weights can be trained independently (e.g. deep ensembles [Lakshminarayanan et al., 2017)), collected on different stages a training trajectory (e.g. snapshot ensembles (Huang et al., 2017) and fast geometric ensembles (Garipov et al., 2018)), or obtained from a sampling process (e.g. MCMC-hased methods (Welling & Teh, 2011; Zhang et al., 2019))- These two paradigms can be combined. Some works suggest construction of ensembles of stochastic computation graphs (Tomczak et al., 2018), while others make use of the collected snapshots to construct a stochastic computation graph (Wang et al., 2018: Maddox et al., 2019).
In this paper we consider the following ensembling techniques: deep ensembles (Lakshminarayanan al., 2017), snapshot ensembles (SSE by Huang al. (2017)), fast geometric ensembling (FGE by Garipov et al. (2018)). SWA-Gaussiae (SWAG by Maddox et al. (2019)), cyclical SGLD (cSGLD by Zhang et al. (2019)), variational inference (VI by Blundell al. (2015)), K-FAC Laplace ap- proximation (Ritter et al., 201 8), dropout (Srivastava et al., 2014) and test-time data augmentation (Krizhevsky et al., 2009). These techniques were chosen to cover diverse set of approaches keeping their predictive performance in mind.
All these techniques can be summarized as distributions zm (w) over parameters w of computation graphs, where m stands for the technique. During testing, can average the predictions across parameters w ~ qm (w) to approximate the predictive distribution
p(y: xx))) Re f p(ue x,, (w) dw E p(y: la, wk), We ~ am (w)
K
(3)
K
For example, deep ensemble of S networks can be represented in this form as mixture of S Dirac's deltas (w) = 3 Σs 8(w w.), centered at independently trained snapshots w.. Simi- larly, Bayesian neural network with fully-factorized Gaussian approximate posterior distribution over the weight matrices and convolutional kernels w is represented as qu((w) = NV(w p, diag(o")), j and o² being the optimal variational means and variances respectively.
If one considers data augmentation as a part of the computational graph, can be parameterized by the coordinates of the random crop and the flag for whether to flip the image horizontally or not. Sampling from the corresponding (aug(w) would generate different ways to augment the data during inference. However, data augmentation is present by default during the training of all othe

---

Published as a conference paper at ICLR 2020
cSGLD
Deep ensemble
Dropout
FGE
K-FAC Laplace
Single model
SSE
SWAG
FFG
Deep ensemble equivalent (DEE)
15
CIFAR10
15
CIFAR100
ImageNet
10
10
5
2
1
1
20 40
60
80
100
20
40 60
80
100
10
20
30
40 50
samples (ensemble size)
# samples (ensemble size)
samples (ensemble size)
Figure 3: The deep ensemble equivalent score (DEE) for different numbers samples on CIFAR- 10, CIFAR-100, and ImageNet datasets averaged across different deep convolutional architectures. deep ensemble equivalent score (DEE) model is equal to the minimum size of a deep ensem- ble (an ensemble of independently train networks) that achieves the same performance the model under consideration. The score measured in the number of models (higher is better). The area between average lower and upper bounds of DEE is shaded. The plot demonstrates that all of the ensembling techniques are far less efficient than deep ensembles during inference and fail to produce the same level of performance as deep ensembles. The comparison that is normalized on training time is presented in Appendix A.
mentioned ensembling techniques, it is suitable to study it in combination with these methods and separate ensembling technique. We perform such an evaluation in Section 4.3.
Typically, the approximation (equation 3) requires K independent forward passes through a neural network, making the test-time budget directly comparable across all methods.
4.1 DEEP ENSEMBLE EQUIVALENT
Most ensembling techniques under consideration are either bounded to single mode, or provide positively correlated samples. Deep ensemble, on the other hand, a simple technique that provides independent samples from different modes of the loss landscape, which, intuitively, should result in better ensemble. Therefore deep ensembles can considered strong baseline for performance of other ensembling techniques given a fixed test-time computation budget.
Comparing the performance ensembling techniques is, however, a challenging problem. Different models on different datasets achieve different values of metrics; their dependence the number of samples is non-trivial, and varies depending on a specific model and dataset. Values of the metrics thus lacking in interpretability the gain in performance has to be compared against model- and dataset-specific baseline.
Aiming to introduce perspective and interpretability in study, introduce the deep ensem- ble equivalent score that employs deep ensembles to measure the performance of other ensembling techniques. Specifically, the deep ensemble equivalent answers the following question:
What size deep ensemble yields the same performance as a particular ensembling method?
Following the insights from the previous sections, we base the deep ensemble equivalent the calibrated log-likelinn likelihood (CLL). Formally speaking, we define deep ensemble equivalent (DEE) for ensembling method and its upper and lower bounds as follows:
DEEm (k) min {I R, Z
CLL DE
(1) >
CLL CLLInaa mean (k)
(4)
(k) min { R. > CLL DE (1) CLL / (1) CLL (A)).
DEEI
(5)

---

Published as a conference paper at ICLR 2020
Augmentations
Ensemble
Predictions
Frog
Original Image
Ensemble Prediction
Bind
Free
Frog
Seake
Snake
Sind
Bird
Bird
Figure 4: An illustration of test-time augmentation (TTA) for an ensemble. We apply every member of an ensemble to a separate random augmentation of an image. The predictions of all members are averaged to produce final prediction of an ensemble. In our experiments, TTA leads to significant boost the performance for most of the ensembling techniques on ImageNet with a sufficient computational budget (see Figure 5).
where CLL mean//std (!) are the mean and the standard deviation of the calibrated log-likelihood achieved by an ensembling method m with samples. We compute CLLDE meam (l) and CLL (I) for
natural numbers 1 E N and use linear interpolation to define them for real values 2 1. In the following plots we report DEE, (k) for different methods m with different numbers of samples k and shade the area between the respective lower and upper bounds DEElower (k) and DEEupper (k))
4.2 EXPERIMENTS
We compute the deep ensemble equivalent (DEE) of various ensembling techniques for four popular deep architectures: VGG16 (Simonyan & Zisserman, 2014), PreResNetl 10/164 (He et al., 2016), and WideResNet28x10 (Zagoruyko & Komodakis, 2016) on CIFAR-10/100 datasets (Krizhevsky et al., 2009), and ResNet50 (He et al., 2016) ImageNet dataset (Russakovsky et al., 2015). We use PyTorch (Paszke et al., 2017) for implementation of these models, building upon available public implementations. Our implementation closely matches the quality of methods that has been reported in original works. Technical details on training, hyperparameters and implementations can be found in Appendix B. The source code and all computed metrics are available on GitHub'.
As one can see on Figure 3, ensembling methods clearly fall into three categories. SSE and cSGLD outperform all other techniques except deep ensembles and enjoy a near-linear scaling of DEE with the number of samples on CIFAR datasets. The investigation of weight-space trajectories cSGLD and SSE (Huang et al., 2017: Zhang al., 2019) suggests that these methods can efficiently explore different modes of the loss landscape. In terms of the deep ensemble equivalent, these methods do not saturate unlike other methods that are bound to a single mode. We found SSE still saturate on ImageNet. This is likely due to suboptimal hyperparameters of the cyclic learning rate schedule. More verbose results are presented Figures 1 11-13 and Table 5 and Table in Appendix C.
In our experiments SSE typically outperforms eSGLD. This is mostly due to the fact that SSE has much larger training budget. The cycle lengths and learning rates of SSE and cSGLD are comparable, however, SSE collects one snapshot per cycle while eSGLD collects three snapshots. This makes samples from SSE less correlated with each other while increasing the training budget threefold. Both SSE and cSGLD can be adjusted to obtain different trade-off between the training budget and the DEE-to-samples ratio. We reused the schedules provided in the original papers (Huang et al., 2017; Zhang et al., 2019).
Source code: http:/hhhhh..ooobeeee///pttttee/
8

---

Published as a conference paper at ICLR 2020
X
central crop (ensemble size 10)
X
central crop (ensemble size 50)
test-lmme sug. (ensemble size 10)
test-lime aug. (ensemble size 50)
Neg. calibrated log-likelhhodd
0.95
Top-1 error (%)
23
0.90
22
0.80
0.75
19
Single model
KFFACL
FFG
FGE
SSE
DE
Sirgle model
KFACL
FFG W
FGE
99E
DE
Figure 5: How read results: X
lest aug
dest-time aug
The negative calibrated log-
samples
samples
likelihood (lower is better) for different ensembling techniques on ImageNet. We report perfor- mance for two regimes. Central-crop evaluation (xx) means every member of an ensemble is applied to a central crop of an image, and test-lime data augmentation (+ means each member of the ensemble is applied to separate random augmentation of the image. Test-time data augmen- tation significantly improves ensembles with no additional computational cost. Interestingly, single model with TTA performs competitively with methods that require significantly larger param- eters budget, and training complexity, e.g., single model with TTA performs closely to pure deep ensembles (DE) of the same size.
Being more "local" methods, FGE and SWAG perform worse than SSE and cSGLD, but still sig- nificantly outperform "single-snapshot" methods like dropout, K-FAC Laplace approximation and variational inference. We hypothesize that by covering a single mode with set of snapshots, FGE and SWAG provide a better fit for the local geometry than models trained as stochastic computation graphs. This implies that the performance of FGE and SWAG should be achievable by single- snapshot methods. However, one might need more elaborate posterior approximations and better inference techniques in order to match the performance of FGE and SWAG by training a stochastic computation graph end-to-end (as opposed to SWAG that constructs a stochastic computation graph post-hoc).
The deep ensemble equivalent curves allow us to notice the common behaviour of different methods, e.g. the relation between deep ensembles, snapshot methods, advanced local methods and single- snapshot local methods. They also allow us to notice inconsistencies that may indicate suboptimal choice of hyperparameters. For example, we find that SSE on ImageNet quickly saturates, unlike SSE on CIFAR datasets (Figure 3). This may indicate that the hyperparameters used on ImageNet are not good enough for efficient coverage different modes of the loss landscape. We also find that SSE WideResNet on 10 achieves DEE score of 100 on approx. 70 samples (Fig- ure 12). This may indicate that the members of the deep ensemble for this dataset-architectue pair underfitted and may benefit from longer training different learning rate schedule. Such inconsistencies might be more difficult to spot using plain calibrated log-likelihood plots.
4.3 TEST-TIME DATA AUGMENTATION IMPROVES ENSEMBLES FOR FREE
honored technique that is widely used in deep learning. and a crucial component for training modern DNNs. Test-time data augmentation has been used for long time to improve the performance of convolutional networks. For example, multi-crop evaluation has been standard procedure for the ImageNet challenge (Simonyan & Zisserman, 2014; Szegedy et al., 2015; He et al., 2016). is, however, often overlooked in the literature on ensembling techniques in deep learning. In this section, we study the effect of test-time data augmentation on the aforementioned ensembling techniques. To keep the test-time computation budget the same, we sample one random augmentation for each member of ensemble. Figure 5 reports the calibrated log- likelihooood on combination of ensembles and test-time data augmentation for ImageNet. Other metrics and results on CIFAR- 1 datasets reported in Appendix C. We have used the standard data augmen-
Data augmentation

---

Published as a conference paper at ICLR 2020
tation: random horizontal flips and random padded crops for CIFAR- 10/100 datasets, and random horizontal flips and random resized crops for ImageNet (see more details in Appendix B).
Test-time data augmentation (Figure 4) consistently improves most ensembling methods, especially on ImageNet, where we see a clear improvement across all methods (Figure and Table 7). The performance gain for powerful ensembles (deep ensembles, SSE and eSGLD) CIFAR datasets not as dramatic (Figures 14-15 and Table 4). This is likely due to the fact that CIFAR images small, making data augmentation limited, whereas images from ImageNet allow for a large number of diverse samples of augmented images. On the other hand, while the performance of "single-smi snapssott methods (e.g. variational inference, K-FAC Laplace and dropout) is improved significantly, they perform approximately as good as an augmented version of single model across all d
Interestingly, test-time data augmentation on ImageNet improves accuracy but decreases the un- calibrated ligliielihoodd of deep ensembles (Table 7 in Appendix C). Test-time data augmentation breaks the nearly optimal temperature of deep ensembles and requires temperature scaling reveal the actual performance of the method, as discussed in Section 3.1. The experiment demonstrates that ensembles may be highly miscalibrated by default while still providing superior predictive per- formance after calibration.
We would like to note that test-time data augmentation does not always break the calibration of an ensemble, and, on the contrary, test-time data augmentation often improves the calibration of an ensemble. In experiments, decalibration was caused by the extreme magnitude of a random crop, that is conventionally used for ImageNet augmentation. Using less extreme magnitude of the random crop fixes decalibration, that makes test-time data augmentation a more practical method that provides out-of-the-box calibration. Although, as we demonstrated earlier, there is no guarantee that any ensemble is calibrated out-of-the-box If we are willing apply post-hoe calibration, the final performance can be much better with more severe augmentations.
5 DISCUSSION & CONCLUSION
We have explored the field of in-domain uncertainty estimation and performed an extensive evalua- tion of modern ensembling techniques. Our main findings can be summarized as follows:
Temperature scaling must even for ensembles. While ensembles generally have bet- ter calibration out-of-the-box. they are not calibrated perfectly and can benefit from the procedure. A comparison of fog-likelihoodd of different ensembling methods without tem- perature scaling might not provide fair ranking, especially if some models happen to be miscalibrated.
Many common metrics for measuring in-domain uncertainty are either unreliable (ECE and analogues) or cannot be used to compare different methods (AUC-ROC, AUC-PR for mis- classification detection; accuracy-cone -coffcc--oneieence curves). In order to perform fair comparison of different methods, one needs to be cautious of these pitfalls.
Many popular ensembling techniques require dozens of samples for test-time averaging, yet are essentially equivalent to a handful of independently trained models. Deep ensembles dominate other methods given a fixed test-time budget. The results indicate, in particu- lar, that exploration of different modes in the loss landscape is crucial for good predictive performance.
Methods that are stuck in a single mode are unable to compete with methods that are de- signed to explore different modes of the loss landscape. Would more elaborate posterior approximations and better inference techniques shorten this gap?
Test-time data augmentation is surprisingly strong baseline for in-domain uncertainty estimation. It can significantly improve other methods without increasing training time or model size since data augmentation is usually already present during training.
Our takeaways are aligned with the nake-home messages of Ovadia et al. (2019) that relate to in- domain uncertainty estimation. We also observe stable ordering of different methods in exper- iments, and observe that deep ensembles with few members outperform methods based on stochastic computation graphs.
10

---

Published as a conference paper at ICLR 2020
A large number of unreliable metrics inhibits fair comparison of different methods. Because of this, we urge the community to aim for more reliable benchmarks in the numerous setups of uncertainty estimation.
ACKNOWLEDGMENTS
Dmitry Vetrov and Dmitry Molchanov were supported by the Russian Science Foundation grant no. 19-71-30020. This research was supported part through computational resources of HPC facilities at NRU HSE.
REFERENCES
Andrei Atanov, Arsenii Ashukha, Dmitry Molchanov, Kirill Neklyudov, and Dmitry Vetrov. Un- certainty estimation via stochastic batch normalization. In International Symposium on Neural Networks, pp. 261 169.. Springer, 2019.
Anoop Korattikara Balan, Vivek Rathod, Kevin P Murphy, and Max Welling. Bayesian dark knowl- edge. In Advances in Neural Information Processing Systems, pp. 1438-3446, 2015.
Charles Blundell, Julien Corebiie,, Koray Kavukcuoglu, and Daan Wierstra. Weight uncertainty in neural networks. arXiv preprint arXiv: 2015.
Glenn W Brier. Verification forecasts expressed in terms of probability. Monthly weather review, 78(1):1-3, 1950.
Jacob Buckman, Danijar Hafner, George Tucker, Eugene Brevdo, and Honglak Lee. Sample- efficient reinforcement learning with stochastic ensemble value expansion. In Advances Neural Information Processing Systems, pp.
2018.
Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforcement learn- ing handful of trials using probabilistic dynamics models. In Advances Neural Information Processing Systems, pp. 4754 4765, 2018.
Yufei Cui, Wuguannan Yao, Qiao Li, Antoni B Chan, and Chun Jason Xue. Accelerating monte carlo bayesian inference via approximating predictive uncertainty over simplex. arXiv preprint ArXiv: 2019.
Yukun Ding, Jinglan Liu, Jinjun Xiong, and Yiyu Shi. Evaluation of neural network uncertainty estimation with application to resource-constrained platforms. arXiv preprint arXiv:1903.02000 2019.
Yarin Gal. Uncertainty in deep learning. PhD thesis, PhD thesis, University of Cambridge, 2016.
Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In international conference machine learning, pp. 1050-1059, 2016.
Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry Vetrov, and Andrew G Wilson. Loss surfaces, mode connectivity, and fast ensembling of dnns. In Advances in Neural Information Processing Systems, pp. 8789-8798, 2018
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Weinberger. On calibration of modern neural networks. In Proceedings of the 34th International Conference on Machine Eeurning-Volumee 70, pp. 1321-1330. JMLR. org, 2017.
Fredrik K Gustafsson, Martin Danelljan, and Thomas B Schön. Evaluating scalable bayesian deep learning methods for robust computer vision. arXiv preprint arXiv:1906.01620 2019.
Lars Kai Hansen and Peter Salamon. Neural network ensembles. IEEE Transactions on Pattern
Analysis & Machine Intelligence, (10):993-1001, 1990.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
11

---

Published as a conference paper at ICLR 2020
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint arXiv:1503.02531, 2015.
Gao Huang, Yixuan Li, Geoff Pleiss, Zhuang Liu, John E Hopcroft, and Kilian Q Weinberger. Snapshot ensembles: Train get for free. arXiv preprint AxKiv:/704.00109 2017.
Sergey loffe and Christian Szegedy. Batch normalization; Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv: 2015.
Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wil- son. Averaging weights leads to wider optima and better generalization. arXiv preprint arXiv: 100.00803..557 2018.
Durk P Kingma, Tim Salimans, and Max Welling. Variational dropout and the local reparameteri- zation trick. In Advances in Neural Information Processing Systems, pp. 2575-2585 2015.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
Ananya Kumar, Percy S Liang, and Tengyu Ma. Verified uncertainty calibration. In Advances in
Neural Information Processing Systems, pp. 3787-3798, 2019.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In Advances in Neural Information Processing Systems, pp. 6402-6413, 2017.
Christos Louizos and Max Welling. Multiplicative normalizing flows for variational bayesian neural networks. In Proceedings of the 34th International Conference Machine Learning- Vollum 70, pp. 2218-2227 JMLR. org, 2017.
Wesley Maddox, Timur Garipov, Pavel Izmailov, Dmitry Vetrov, and Andrew Gordon Wilson. A simple baseline for bayesian uncertainty deep learning. arXiv preprint arXiv: (90..0299902277700 2019.
Andrey Malinin and Mark Gales. Predictive uncertainty estimation via prior networks. In Advances in Neural Information Processing Systems, pp. 7047-7058, 2018.
James Martens and Roger Grosse. Optimizing neural networks with kronecker- approximate curvature. In International conference on machine learning, pp. 2408-2417, 2015.
Dmitry Molchanov, Arsenii Ashukha, and Dmitry Vetrov. Variational dropout sparsifies deep neural networks. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 2498-2507. JMLR. org, 2017.
Marcin Možejko, Mateusz Susik, and Rafal Karczewski. Inhibited softmax for uncertainty estima- tion in neural networks. arXiv preprint 1xXiv:1810.01161,, 2018.
Malik Sajjad Ahmed Nadeem, fean-Daniel Zucker, and Blaise Hanezar. Accuracy-rejection curves (ares) for comparing classification methods with reject option. In Machine Learning in Systems Biology, pp. 65-81, 2009.
Mahdi Pakdaman Naeini, Gregory Cooper, and Milos Hauskrecht. Obtaining well calibrated prob- abilities using bayesian binning. In Twenty-Ninth AAAI Conference on Artificial Intelligence, 2015.
Jeremy Nixon, Mike Dusenberry, Linchuan Zhang, Ghassen Jerfel, and Dustin Tran. Measuring calibration in deep learning. 2019.
Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, D Sculley, Sebastian Nowozin, Joshua V Dil- lon, Balaji aaksmminaraaaann, and Jasper Snoek. Can you trust your model's uncertainty? eval- uating predictive uncertainty under dataset shift. arXiv preprint arXiv: /906.02566000 2019.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in PyTorch. In NIPS Autodiff Workshop, 2017.
12

---

Published as a conference paper at ICLR 2020
Joaquin Cantonerr-Canndlle Carl Edward Rasmussen, Fabian Sinz, Olivier Bousquet, and Bemn- hard Schölkopf. Evaluating predictive uncertainty challenge. In Machine Learning Challenges Workshop, pp. 1-27. Springer, 2005.
Hippolyt Ritter, Aleksandar Botev, and David Barber. A scalable laplace approximation for neural networks. 2018.
Olga Russakovsky, Jia Deng. Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang. Andrej Karpathy, Aditya Khosla, Michael Bernstein, al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3)2211-255 2015.
Burr Settles. Active learning. Synthesis Lectures Artificial Intelligence and Machine Learning, 6 (1):1-114, 2012.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1409, 1556, 2014.
Leslie N Smith and Nicholay Topin. Super-convergence: Very fast training of neural networks using large learning rates. In Artificial Intelligence and Machine Learning for Multi-D Domain Operations Applications, volume 1 11006. pp. 1100612. International Society for Optics and Photonics, 2019.
Casper Kaae Sønderby, Tapani Raiko, Lars Maaløe, Søren Kaae Sønderby, and Ole Winther. How to train deep variational autoencoders and probabilistic ladder networks. In 33rd International Conference on Machine Learning (ICML 2016), 2016.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: simple way prevent neural networks from overfitting. The journal of machine learning research, 15()):1229-198, 2014.
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv: 2013.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Du- mitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1-9, 2015.
Mattias Teye, Hossein Azizpour, and Kevin Smith. Bayesian uncertainty estimation for batch nor- malized deep networks. In International Conference Machine Learning (ICML), 2018.
Marein B Tomezak, Siddharth Swaroop, and Richard E Turner. Neural network ensembles and variational inference revisited. In /st Symposium on Advances in Approximate Bayesian Inference, pp. 1-11, 2018
Linh Tran, Bastiaan S Veeling, Kevin Roth, Jakub Swiatkowski, Joshua V Dillon, Jasper Snoek, Stephan Mandt, Tim Salimans, Sebastian Nowozin, and Rodolphe Jenatton. Hydra: Preserving ensemble diversity for model distillation. arXiv preprint arXiv::0011 T00::2000.006944, 2020.
Karen Ullrich, Edward Meeds, and Max Welling. Soft weight-sharing for neural network compres- sion. arXiv preprint arXiv:1702.04000 2017.
Juozas Vaicenavicius, David Widmann, Carl Andersson, Fredrik Lindsten, Jacob Roll, and Thomas B Schon. Evaluating model calibration in classification. arXiv preprint
arXiv: 195:::990200677, 2019.
Kuan-Chich Wang. Paul Vicol, James Lucas, Li Gu, Roger Grosse, and Richard Zemel. Adversar- ial distillation of bayesian neural network posteriors. In International Conference on Machine Learning, pp. 5177-5186, 2018.
Sida Wang and Christopher Manning. Fast dropout training. In international conference on machine learning, pp. 118-126. 2013.
Max Welling and Yee W Teh. Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of the 28th international conference on machine learning (ICML-11), pp. 681-688, 2011.
13

---

Published as a conference paper at ICLR 2020
Anqi Wu, Sebastian Nowozin, Edward Meeds, Richard E Turner, José Miguel Hernández-Lobate and Alexander Gaunt. Deterministic variational inference for robust bayesian neural networks. 2018.
Sergey Zagonuyko. 92.45 cifar-10 in torch, 2015. URL http:://toorhh..c//log///1111 07/30/ ci mar.hhml.
Sergey Zagoruyko and Nikos Komodakis. arXiv:1605.07146. 2016.
Wide
residual networks.
arXiv preprint
Hongyi Zhang. Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint 17X::11700.09422, 2017.
Ruqi Zhang, Chunyuan Li, Jianyi Zhang, Changyou Chen, and Andrew Gordon Wilson. Cyclical stochastic gradient meme for bayesian deep learning, arXiv preprint arXiv: 2019.
14

---

Published as a conference paper at ICLR 2020
CIFAR10
CIFAR100
10
deep 4tsemb souhaient
Mean deep wrsemmbme aquiovaant
Snapshot Ensembles (SSE)
Snapshot Ensembles (SSE)
Cyclc SGLD (cSGLD)
Cyclc SGLD (cSGLD)
Deep ensemble
Deep ensembll
Fast Geometric Ens. (FGE)
Fast Geometric Ens. (FGE)
000
1000
1500
2000 2500 3000 3600 4000
500
1000
1500
2000
2500 3000 3500 4000
Mean oosi of solutin epochs)
Mean czsi solution (ft epodhs)
Figure 6: The mean cost of training of ensemble the mean deep ensemble equivalent score. Each marker on the plot denotes one snapshot of weights.
Derg
C5GLD
650L0
(SOLD
YOUTS
502
555
Figure 7: Cost of training of an ensemble vs. its quality (DEE), Each marker on plot denotes one snapshot of weights.
A Is "EFFICIENT" TRAINING OF ENSEMBLES EFFICIENT AT ALL?
Yes! If you most about training time efficiency. All snapshot based methods (SSE, cSGLD and FGE) on average (Figure 6) tend to achieve the same performance as deep ensembles using 2-5x less training epochs on CIFAR datasets.
The gain comes at a cost of inference efficiency and memory consumption. "Efficient" snapshot- based methods require to store much more samples of weights compared to deep ensembles making inference significantly more expensive (up to 25) given the same predictive performance.
You need to get lucky with hyperparameter choice. While on average "efficient" snapshot-based methods require less training resources, they might be completely inefficient if your hyperparameter choice is sub-optimal (see Figure 7). Such hyperparameters as the maximum learning rate, the length of learning rate decay cycle, the snapshot saving schedule all impact the performance significantly.
This analysis assumes that we only conventional training procedure. Many models can most likely be trained, stored and executed much more efficiently with methods like stochastic weight averaging, compression, distillation and others. These methods are out of the scope of the paper but are interesting topics for future research. The current choice of hyperparameters may also not be optimal. We reuse the hyperparameters used in the original papers.
15

---

Published as a conference paper at ICLR 2020
B EXPERIMENTAL DETAILS
Implementations of deep ensembles, SWAG, FGE and K-FAC Laplace are heavily based the original PyTorch implementations of stochastic weight averaging (SWA) and SWAG 3 Imple- mentations of cyclical MCMC and snapshot ensembles are based the original implementation of eyclical MCMC 4, We hypothesize that the optimal hyperparameters of ensembling methods may vary widely depending on the computational budget and the number of samples in the ensemble. Searching for the optimal values for each configuration is outside the scope of this paper so we stick to the originally proposed hyperparameters whenever possible.
Implied probabilistic model Conventional neural networks for classification are usually trained using the average cross-entropy loss function with weight decay regularization hidden inside an optimizer in deep learning framework like PyTorch. The underlying optimization problem can be written as follows:
L(w) = N log plui Tz. w) + 2 || - min,
(6)
where [(T, vz ) is the training dataset of N objects D; with corresponding labels yf" À is the weight decay scale and p(j 24,, w) denotes the probability that a neural network with parameters w assigns to class j when evaluated on object Tū-
The cross-entropy loss defines likelihood function ply" Ir, w) and weight decay regularization, or Le regularization corresponding to a certain Gaussian prior distribution p(w). The whole optimiza- tion objective then corresponds the maximum a posteriori inference in the following probabilistic model:
ply", w x) p(y" |2, w)p(w)
N
N
logp(y" Iz, (x = log Il
p(vi 2,, w) = Σ
log ply [,,
(8)
1=1
-N || const > p(w) = N (w 0. (NA)-11)
log p(w)
In order to make the results comparable across all ensembling techniques, used the same prob- abilistic model for all methods, choosing fixed weight decay parameters for each architecture. We used the softmax-based likelihood for all models. We also use the fully-factorized zero- mero Gaus- sian prior distribution with variances o² = (NA) where the number of objects N and the weight decay scale À are dictated by the particular datasets and neural architectures defined in the fol- lowing paragraph.
Conventional networks To train single network CIFAR-10/100, we used SGD with batch size of 128, momentum 0.9 and model-specific parameters, i.e. the initial learning rate (Irinst), the weight decay coefficient (wd), and the number of optimization epochs (epoch). Specific hyperpa- rameters are shown in Table 1. The models were trained with a unified learning rate scheduler that shown in equation 10. All models have been trained using data augmentation that consists of hor- izontal flips and random crop of 32 pixels with a padding of 4 pixels". The standard data normal- ization has also been applied. Weight decays, initial learning rates, and the learning rate scheduler were taken from (Garipov et al., 2018) paper. Compared with hyperparameters of (Garipov et al., 2018), we increased the number of optimization epochs since we found that all models were un- derfitted. While the original 1etcResNet28x10 network includes number of dropout layers with p = 0.3 and is trained for 200 epochs, we find that the WideResNNe28xx10 10 underfits in this setting and requires longer training. Therefore, used p which reduces training time while bearing
'https: hattesss//gihhbbb.mmmmm/mprrrppp
"Compose (#aaddaniorizonnta11111pp kandomCrop (32, padding=4) ])
16

---

Published as a conference paper at ICLR 2020
Model
Irant
epochs
wd
VGG
0.05
400
5c-4
PreResNetl 10
0.1
300
3e-4
PreResNer164
0.1
300
3e-4
WideRes
0.1
300
5c-4
Table 1: Hyperparameters of models trained on CIFARs for single-model evaluation
no significant effect on final model performance in our experiments.
Irinits
E 0.5 epochs]
1r(1)=
Irinat (1.0 0.99
(i/epochi 0.5)/0.4)
e [0.5 . epochs, 0.9 - epochs]
(10)
Irintt
0.01,
otherwise
On NeageNet dataset we used ResNet50 with default hyperparameters taken from PyTTorch examples Specifically, we used SGD with momentum 0.9, batch size of 256, initial learning rate 0.1, weight decay 1e - 4. Training included data augmentation' (scaling. random crops of size 224 X 224, horizontal flips), normalization and learning rate scheduler Ir = Irimit 0.1 /30 where / denotes integer division. We only deviated from standard parameters by increasing the number of training epochs from 90 to 130. Or models achieve top-1 error of 23.81 + 0.15 that closely matches the accuracy of ResNet50 provided by Py PoTec which is 23.85 Training of one model on a single NVIDIA Tesla V100 GPU takes approximately 5.5 days.
Deep ensembles Deep ensembles (Lakshminarayanan et al., 2017) average the predictions across networks trained independently starting from different initializations. To obtain a deep ensemble we repeat the described procedure of training standard networks 128 times for all architectures on CIFAR-10 10 and CIFAR- datasets (1024 networks over all) and times for ImageNet dataset. Ev- ery member of deep ensembles was trained with exactly the same hyperparameters conventional models of the same architecture.
Dropout Binary dropout (or MC dropout) (Srivastava et al., 2014: Gal & Ghahramani, 2016) is one of the most widely known ensembling techniques. It involves putting multiplicative Bernoulli noise with a parameter over the activations of either a fully connected layer or a convolutional layer, averaging predictions of the network w.r.t. noise test-time. Dropout layers were applied to VGG and WideResNNt networks on CIFAR-10 and CIFAR-100 datasets. Dropout for VGG was applied to fully connected layers with p 0.5. Two dropout layers were applied: one before the first fully connected layer and one before the second one. While the original version of VGG for CIFARs (Zagoruyko, 2015) exploits more dropout layers, we observed that any additional dropout layer deteriorates the performance of the model in ether deterministic or stochastic mode. Dropout for WideResNet was applied in accordance with the original paper (Zagoruyko & Komodakis, 2016) with p = 0.3. Dropout usually increases the time needed to achieve convergence. Because of this, WideResNet networks with dropout were trained for 400 epochs instead of 300 epochs for determin- istic case, and VGG networks have always been trained with dropout. All the other hyperparameters were the same as in the case of conventional models.
Variational Inference Variational Inference (VI) approximates the true posterior distribution over weights p(w] Data) with a tractable variational approximation go(w) by maximizing a so-called variational lower bound & (eq. 11) w.r.t the parameters of variational approximation 0. We used a fully-factorized Gaussian approximation (ww) and Gaussian prior distribution p(w).
((0) E.g log p(y"
xx w) (w) p(w)) - max
(11)
((w) N(w H, diag(o²)) p(w)= N(w 0, diag(o where of = -
(12)
1ttto//oorzzotalll1111 ()) "https: hhtres://pytorreh..rg/////sss/s http:///t.aaaaeettttrciriiiioooa 1
"compose ([Randonkesi wew.e
17

---

Published as a conference paper at ICLR 2020
Optimal noise scale CIFAR-IO-aug CIFAR- 100 CIFAR-100-aus 100-aug
Architecture
CIFAR-10
VGG16BN
0.042
0.042
0.100
0.100
PreResNetl 10
0.213
0.141
0.478
0.401
PreResNet164
0.120
0.105
0.285
0.225
10
0.022
0.018
0.022
0.004
Table 2: Optimal noise scale for K-FAC Laplace for different datasets and architectures. For ResNet50 on ImageNet, the optimal scale found was 2.0 with test-time augmentation and 6.8 with- out test-time augmentation.
In the case of such a prior, the probabilistic model remains consistent with the conventional training which corresponds MAP inference in the same probabilistic model. We used variational inference for both convolutional and fully-connected layers where variances of the weights were parameterized by log a. For fully-connected layers applied the local reparameterization trick (LRT, (Kingma et al., 2015)).
While variational inference provides theoretically grounded way to approximate the true posterior, tends to underfit deep learning models in practice (Kingma et al., 2015). The following tricks applied deal with it: pre-training (Molchanov et al., 2017) equivalently annealing of (Sønderby et al., 2016). and scaling 3 down (Kingma et al., 2015; Ullrich et al., 2017).
During pre-training we initialize μ with snapshot of weights of pre-trained conventional model, and initialize log a with a model-specific constant log Gnit. The KL-divergence except for the term corresponding the weight decay is scaled with model-specific parameter B. The weight decay term is implemented as part of the optimizer. We used fact that the KL-divergence between two Gaussian distributions can be rewritten as two terms, one of which is equivalent the weight decay regularization.
On CIFAR-10 and CIFAR-100 we used / equal to le-4 for VGG, ResNet1 and ResNet164 net- works, and / equal to le-5 for WideResNet. Log-variance log dinit was initialized with -5 for all models. Parameters μ were optimized with SGD in the same manner as in the case of conven- tional networks except that the initial learning rate (rinit optimizer with a constant learning rate of le-3 to optimize log- lorivariances the weights log a. Pre- training was done for 300 epochs, and after that the remaining part of training was done for 100 epochs. On ImageNet we used B = 1e-3, lrinit = 0.01, log Fnit 6, and trained the model for 45 epochs after pre-training.
set to le-3. We used separate Adam
K-FAC Laplace The Laplace approximation uses the curvature information of the appropriately scaled loss function to construct a Gaussian approximation to the posterior distribution. Ideally, one would use the Hessian of the loss function as covariance matrix and use the maximum posteriori estimate w M AP as a mean of the Gaussian approximation:
p(w v, w,,") log |x, w) + logp(w) + const
(13)
w MAP arg max logr(w ar, ",,"); Σ - VV log x,y")
(14)
p(w a. N(w | MAP Σ)
(15)
In order keep the method scalable. we use the Fisher Information Matrix as an approximation to the true Hessian (Martens & Grosse, 2015). For K-FAC Laplace, we use the whole dataset to construct an approximation to the empirical Fisher Information Matrix, and use the w correction to reduce the bias (Ritter et al., 2018; Martens & Grosse, 2015). Following (Ritter et al., 2018), we find the optimal noise scale for K-FAC Laplace on held-out validation set by averaging across five random initializations. We then reuse this scale for networks trained without a hold-out validation set. We report the optimal values scales in Table 2. Note that the optimal scale is different depending whether introduces some amount of additional noise, the optimal noise scale for K-FAC Laplace with data augmentation is lower.
test-time data augmentation not. Since the data augmentation also
18

---

Published as a conference paper at ICLR 2020
Snapshot ensembles Snapshot ensembles (SSE) (Huang et al., 2017) is simple example of an array of methods which collect samples from a training trajectory of network in weight space to construct an ensemble. Samples are collected in a cyclical manner: during each cycle the learning rate goes from a large value to near-zero and snapshot of weights of the network is taken at the end of the cycle. SSE uses SGD with cosine learning schedule defined as follows:
(t))
au
0Os
π mod (t - 1, 1TTT// )+1)
[T/M]
(16)
where is the initial learning rate, T is the total number of training iterations and M is the number of cycles.
For all datasets and models hyperparameters from the original SSE paper are reused. For CIFAR- 10/100 length of the cycle is epochs, maximum learning rate is 0.2, batch size 64. On ResNer50 and ImageNet length of the eycle is 45 epochs, maximum learning rate is 0.1, batch size is 256.
Cyclical SGLD Cyclical Stochastic Gradient Langevin Dynamics (cSGLD) (Zhang et al., 2019) state-of-the-ard ensembling method for deep neural networks pertaining to stochastic Markov Chain Monte Carlo family of methods. It bears similarity to SSE, e.g. it employs SGD with a learn- ing rate schedule described with the equation 16 and training is cyclical in the same manner. Its main differences from SSE are the introduction of gradient noise and the capturing of several snapshots per cycle, both of which can aid in sampling from posterior distribution over neural network weights efficiently.
Some parameters from the original paper are reused: length of cycle is 50 epochs, maximum learning rate is 0.5, batch size is 64. Number of epochs with gradient noise per cycle is 3 epochs. This was found to yield much higher predictive performance and better uncertainty estimation compared to the original paper choice of 10 epochs for CIFAR-10 and 3 epochs for CIFAR-100.
Finally, the results of cyclical Stochastic Gradient Hamiltonian Monte Carlo (SGHMC), (Zhang et al., 2019) which reportedly has marginally better performance compared with cyclical SGLD, could not be reproduced with any value of SGD momentum term. Because of this, we only include eyclical SGLD in our benchmark.
FGE Fast Geometric Ensembling (FGE) is an ensembling method that is similar to SSE in that it collects weight samples from a training trajectory to construct an ensemble. Its main differences from SSE are pretraining, short cycle length and a piecewise- linear learning rate schedule:
(1- (1-2(())) + 21)))
0 < E(1) S 1
a(i) -
(17)
(2 (2)))()))222) + (2t(1) 1)an
1(1) <
Hyperparameters of the original implementation of FGE are reused. Model pretraining is done with SGD for 160 epochs according the standard learning rate schedule described in equation 10 with maximum learning rates from Table 1. After that, a desired number of FGE cycles is done with one snapshot per cycle collected. For VGG the learning rate is changed with parameters a₁ le - 2, a2 5e - 4 in cycle with cycle length of 2 epochs. For other networks the learning rate changed with parameters a1 5e 2, ag 5e with cycle length of epochs. Batch size is 128.
SWAG SWA-Gaussian (SWAG) (Maddox et al., 2019) is an ensembling method based on fitting Gaussian distribution to model weights the SGD training trajectory and sampling from this distribution to construct an ensemble.
Like FGE, SWAG has a pretraining stage which is done according to the standard learning rate schedule described in equation 10 with maximum learning rates from Table 1. After that, training continues with a constant learning rate of le-2 for all models except for PreResNetl 10 and PreRes- Net164 64 on CIFAR-100 where it continues with a constant learning rate of 5e-2 in accordance with the original paper. Rank of the empirical covariance matrix which is used for estimation of Gaussian distribution parameters is set to be 20.
19

---

Published as a conference paper at ICLR 2020
C
ADDITIONAL EXPERIMENTAL RESULTS
ResNett
VeG16 WideResNee
ResNet164
-0.100
Calibrated Log-Likelihood
-0.125
-0.150
-0.175
Deep ensemble
cSGLD
Dropout
SSE
-0.200
K-FAC-L
SWAG
Single model
FGE
-0.225
0.05
0.06
0.07
0.08
0.09
Calibrated Brier
(a) CIFAR-10 dataset |p|
0.985
ResNet110
VGG16
-0.6
Calibrated Log-Like Likelihood
-0.7
-0.8
-0.9
Deep ensemble
cSGLD
Dropout
SSE
K-FAC-L
SWAG
-1.0
Single model
FFG
-1.1
0.225
0.250
0.275
0.300
0.325
0.350
Calibrated Brier
(b) CIFAR 100 dataset |p|
0.972
ResNet50
50
-0.80
d0
Calibrated Log-Likelihood
$0
40
-0.85
-0.90
80
Deep ensemble
FFG VI SSE
K-FAC-L
-0.95
Single model
FGE
0.29
0.30
0.31
0.32 Calibrated Brier
0.33
0.34
(c) ImageNet dataset |p| 0.999
Figure 8: The average log- log-likelihood the Brier score test dataset for different ensemble methods on CIFAR-10 (a) and CIFAR-10 (b) and ImageNet (c) datasets. While not being equivalent, these metrics demonstrate strong linear correlation. The correlation coefficient is denoted as p.
20

---

Published as a conference paper at ICLR 2020
ReaHeet
ReoMurt64
voGte
Re
Rea Wertoe
voste Widebeerten
Calibrated Log
-0.100
Lag Likelkkilhodd
-0.15
-0.125
-0.20
-0.150
Deep Ensemble
cSGLD
-0.175
Deep ensomble
6BBLD
-0.25
Dropout
Dropout
SSE
K-FAC-L
SWAG
-0.200
K-FAC-L
SWAG
Single model
-0.30
FFG VI
FIG VI
-0.225
94.5
96.0
95.5
96.0
96.5
97.0
94.5
95.0
95.8 5
98.0
96.5
97.0
Accuracy (%)
Accuracy (%6
(a) CIFAR-10, 10, p 0.861
(b) CIFAR- 10, p 0.957
Heahwe110 Rechwet6i
VGGns
RedNett Redtetid4
VS616 WiteRooder
-0.6
-0.6
Calibrated Log Likellhood
Log-Likke Litellhood
-0.8
-0.7
-1.0
-0.8
-1.2
Deep Ensemble
eSGLD
-0.9
Deep ensemble
(SGLD
Dropout
SSE
Dropout
SSE
KFFACL
-1.4
Single model
FGE
-1.0
Single model
FCE
-1.1
76
78
80
82
74
76
80 Accuracy (%)
82
84
Accuracy (%)
(c) CIFAR- 100, p 0.877
CIFAR-100, p == 0.956
ReaNedS
Reshwr50
80
-0.80
Calibrated Logil Logollkklliii ikalhood
-0.80
80
50
Log Likolihood
-0.85
-0.85
-0.90
-0.90
iddalo d0
K-FAC-L
SSE
K-FACLL
SSE
-0.95
Single
FGE
-0.95
FGE
Accuracy (%)
Accuracy (%)
(c) ImageNet, 0.995
(f) ImageNet, 0.997
Figure 9: log-likelihood vs accuracy for different ensembles before (a, c, e) and after (b, d, f) calibration. Both plain log-likelihood and especially calibrated log- lieelikeeiood are highly correlated with accuracy.
21

---

Published as a conference paper at ICLR 2020
VGG16 CIFAR10
VSG16 CIFAR10
VGG16 CIFAR100
VGG18 CIFAR100
4.10
Calintted Log Pood
4.20
0 5
3.0
samples
# samples
46mppis
ssmples
RosNait 10 CIFAR10
R10 CIFAR10
ResNett CIFAR100
Ne110 CFFR1000
-1.0
Callunted Log Lio Boo1
4..90
Logg 10000000000
-:20
4.20
samples
samples
semples
samples
Restittt 64 CIFAR10
ResNer116 CIFAR10
ResNerte4 CIFAR100
ResNett66 CIFAR100
-110
Coltettdd Uig
41120
Log Food
0.200
12
4 samples
# samples
#
samples
samples
WideResNNt FAR10
WedeReeNNt CIFAR10
WizeFesNNet CIFAR100
WiideResNet CIFAR100
Coltrned Log Uiidhiid
camples
samples
samples
samples
(a) CIFAR-10
(b) CIFAR-100
Figure 10: A bide-by-side comparison of log- ligeiiil and calibrated log- ogglikelihood CIFAR-10 10 (a) and CIFAR- 100 (b) datasets. On CIFAR-10 the performance of one network becomes close to dropout, variational inference (vi), and K-FAC Laplace approximation (kfacl) after calibration all models except VGG. On CIFAR-100 deep ensembles move to the first position in the ranking after calibration on WideResNet and VGG. See Section 3.1 for details on the calibrated logllikeli
FGE
K-FAC Laplace
Single model
SSE
FFG VI
ImageNet ResNet50
ImageNet ResNet50
ImageNet ResNet50
ImageNet ResNet50
Brier DEE
AU-AR ARAACCOO DIE
Accuracy
10
30
20
10
20 &samples
40
10
30
samples
&samples
&samples
Figure 11: The deep ensemble equivalent of various ensembling techniques on ImageNet. Solid lines: mean DEE for different methods and architectures. Area between DEglower and Dppr is shaded. Columns 2-4 correspond DEE based on other metrics, defined similarly to the log- likelihood-based DEE. The results are consistent for all metrics
22

---

Published as a conference paper at ICLR 2020
<SGLD
Dropout
FGE
K.FAC Laplace
Single madel
SSE
SWAG
FFG VI
OFARR D VGG1 6AN
CIFARID PreResNet110
CIFARIO ProResNot164
CIFARID WideResNer2Bx10
40
20
30
30
15
80
60
20
40
20
0
0
50
100
0
50
100
0
50
100
0
50
100
samples
+ samples
# samples
samples
CIFARIO o VGGI 6BN
CIFARIO PreResNet110
PreResNet164
CIFAR10 WideResNe-28x10
100
20
20
80
Brier DEE
15
15
60
10
10
40
5
5
20
2
50
100
samples
samples
samples
samples
CIFARI o VGG1 6BN
CIFAR10 PreResNet110
CIFARI0 PreResNet164
CIFAR10 WideResNe12Bx10
100
20
30
60
Accuraty DEE
80
15
20
40
10
40
20
0
0
50
100
0
50
100
50
100
50
100
samples
# samples
samples
samples
CIFARI o VGGI 6BN
CIFAR10 PreResNet110
CIFAR10 PreResNet164
CIFAR10 0 128x
50
30
15
Al-ARC DEE
20
30
10
10
10
5
0
50 samples
100
sampless
0 samppess
samples
The deep ensemble equivalent of various ensembling techniques on CIFAR-10. Solid lines: mean DEE for different methods and architectures. Area between DEFloowrr and DEEupper is shaded. Lines 2-4 correspond DEE based on other metrics, defined similarly to the log-likelihood- based DEE. Note that while the actual scale of DEE varies from metric to metric, the ordering of different methods and the overall behaviour of the lines remain the same.
Figure 12:
SSE outperforms deep ensembles on CIFAR-10 on the WideResNet architecture. It possibly indi- cates that the cosine learning rate schedule and longer training of SSE are more suitable for this architecture than the piecewise-linear learning rate schedule and the number of epochs used in deep ensembles.
23

---

Published as a conference paper at ICLR 2020
<SGLD
Drepout
FGE
K-FAC Laplace
Single model
SSE
SWAG
FFG V
CIFAR100 VGG16BN
CIFAR100 Pr11e
CIFAR100 PreResNe
CIFAR100 Wide 28:
25
30
12.5
12.5
10.0
20
10.0
20
7.5
7.5
5.0
2.5
5
50
100
100
samples
CIFAR100 VGG168N
CIFAR100 e111
CIFA1100 64
CIFAR100 218eRes
12.5
12.5
10.0
Biter DEE
6
10.0
7.5
5.0
4
2.5
50
100
100
samples
samples
samples
samples
CIFAR100 VGG16BN
CIFAR100 PreResN et110
CIFA1100 6reResNet164
CIFAR100 WideResNNet2ex10
Accuracy
4
2
1000
0
50
100
100
0
50
100
# samples
samples
samples
# samples
CIFARI0O VGG16BN
CIFAR100 PrelesNNell10
CIFAR100 11
CIFAR100 2811
10
12.5
8
AU-ARE DEE
10.0
7.5
6
10
4
2.5
50
100
0
50
100
50
100
50
100
4 samples
samples
samples
# samples
Figure 13: The deep ensemble equivalent of various ensembling techniques CIFAR-100. Solid lines: mean DEE for different methods and architectures. Area between DEFllowr and upper is shaded. Lines 2-4 correspond to DEE based on other metrics, defined similarly to the log-likelihood- based DEE. Note that while the actual scale of DEE varies from metrie metric, the ordering of different methods and the overall behaviour of the lines remain the same.
24

---

Published as a conference paper at ICLR 2020
Error (%)
Negative calibruted log- likellhond
Model
Method
10
100
5
10
100
Dropout
5.81 0.
0..2
5.79
0.232
0. 0.225
0.224
0. n..03
SWA Gaussian
7.0310:0
5.6610cs
5.
5.25.1.11
0.2901a.044
0.18220.00
0.171
0.160_o.00z
Cyclic SGLD
1.
6.562009
5.71
4.84
0.234
0.196.2
0.176
0.147
Fast Geometric Ens.
6.521.11
5.95
5.69_.06
5.10
0.2131
0.187 20.01
0.1781a.00
0. 0.105
VGG16 CIFAR-10 10
Deep Ensemhles
14
4
0.
4.30
0,158 0.226
0.148
0.134
Single model
5.83
5.83emm
6.83
6.83
0. 2.2233u.ox 0.223
0.223
0. 222
Variational Inf. (FFG)
6,57
5.63en.1s
5,50
5,46.0.22
0.239
0.192=0.202
0.181+
0. 0.175
KFAC-Laplace
6.
5.82
1.80
0.210
2.333..mm
+:20rr
2.200=u.00s
Snapshot Ensembles
7.76
5.52=a.15
5,00=0.30
0.247
0.176=0.202
0,100eeo.ox
0. 0.137
SWA-Gansian
4.77
1.56e0.17
1.46=0.12
4.3
0.178
0.143=uo.04
0,139
0. =0.0s
Cyclic SGLD
2.
5.32
1.35=0as
3.83
0.156==.205
0.138
0.115=0.001
Fast Geometric Ens.
5.521008
4.83
0.
4.28
1.163
0.141_0.003
0.1 0.126 1.
.o.00
ResNet110
Deep Ensembles
1.66eo.11
3.77
3.63-oar
3.53
0.148
0.117=s0.00
0.112ea.esz
0.106-ixx
CIFAR-10 10
Single model
4.69
4.69
4.69.o.1
4.69
1.150
0.150go.202
0.150
0. 0.150_..00z
Variational Inf. (FFG)
5.57
g.91
4.72_.0
4.60200.0
D.
0.149. 080
oon
0.144ga.00r
0.1402.o.00
KFAC-Laplace
6.14
4.90
4. 4.78
187
0. 0.160=0..0
2.153
0. 14.700000
Snapshot Ensembles
8.4
4.85
0.16
3.5220.10
0.252
0.153_00.0
0.1322a.ora
0.107_2..00
SWA- UNA-Gaussian
5.411om
1.21
4.21_o03
4.02...14
0.171
0.130
1.1281
0.121_0.002
Cyclic SGL.D
5.8020
4.97
4.30
3.66 00
0. 0,149go 0.178
0.131 oca
0.1 10
Fast Geometric Ens.
5.22
4.49200w
4.00
0.134 4.157
0.130
19
ReaNet164
Deep Ensembles
4.53eo.11
3.51
3.50=0.e
3.31
0.147
0.113=0.201
+.107
0.100
CIFAR-10
Single model
4.52
4.52
4.52
4.52
D.
0.144g2..0
0.
0. 1.4420000
Variational Inf. (FFG)
5.62+0.44
1.78epcs
1.66=oas
1.35=0.0
0.183ea.004
0.131=0.00
0.1
16
0.141-0.00
KFAC-Laplace
2.23
4.77
0.65
4.60
0,168
co.mce
0.151:0.007
0.142 0.146
Snapshot Ensembles
1.06
4.50go0e
3.80_ on
3.50g0.00 20
503 0.144 1.201
0.1241
0.104_...0m
Dropout
3.88
3.70
3.63_.o
3.64
1.130
0.120,0.002
0.1191a000
0.117
SWA-Gaussan
1.98+1a7
3.53-a.2
3.31-0.14
3,28
+,157
0.111=0,204
0.105+
0. 0.101=0.002
Cyclic SGLD
4.7810.16
4.09
3.
3.19
0.128 1..55
502
0.11413.001
0.099...002
Fast Geometric Fis.
4.86
3.95go07
3.
3.34_o00
0. 0.120_g 0.148
002
0.1131a.wr
0. 0.1022.0.001
WideResNet
Deep Ensembles
3.65g000
3.11
3.01=va0
2.83
0. 0.097:20.0 w.123
0.095
0.090
CIFAR- 10
Single model
3.7010
3.7010i5
3.
3.7010.1
1.124
0.124_00.00
0.1251
0.124_0.00s
Variational Inf, (PFG)
5.61
4.
1.15
3.
3.64
D.
0. 134:2..0 002
0.
0.127a
0. 0.11720.00
KFAC-Laplace
1.03e019
3.90=0.1
3.88-022
3.83=0.1
0.121=0.oo 0.131ea..s
0.122
1.0000000
Snapshot Ensembles
5.56
3.68
3.33
2.89
0.179
0. 0.119
0.105
0.090
Dropout
26.10=u.0
25.68=0.u
25
25.6622.14
25. 60
1.176
1. 1.11=20.00
1.098
100.0
SWA-Gaussian
27.74=1a7
24.53=0.0
23.64++0.2
22
1.109
0.
0.931=0.207
0.879
0. 0.826
Cyclic SGLD
29.75...17
26.79.0.18
24.14
21.15
1.11449.00a
0.976...001
1.881
0.749..0.00
Fast Geometric Ens.
27.07=0.34
25.35e0.20
21.6840.40
22.78+s.2:
1.057+a.a0
0. 0.965ee..05
0.930
0. 0.827=0.004
VGG16 CIFAR- 100
Deep Ensembles
25.72.0.17
21.60:0.1
20.79.0.16
19.
1.00220.004
0.840
0.794
0.723
Single model
25.44_0.m
25.44_..m
25.4410.39
25.4416.m
1.08719.000
1.087.0.00s
1.0871...r
1.087_.0.oos
Variational Inf, (PFG)
27.24:20.0
25.24
24.8520.05
24.56a..or
1.1542a.0s
1.00120.002
0.973aa.oo
0. 0.930=...0)
KFAC Laplace
27.11.0.00
25.98.0.a
25.8410.s
25.7010.as
1.17415.07
1.0891o.a
507
1.06919.00s
1.050_...os
Snapshot Ensembles
31.1920.s0
23.8720.1
22.31
21.03a6.1 10
1.170g
0.899 001
0.8342 ocs
0. 0.751
-NN--Gussian
27.75_0.m
22.31120.0
21.5226.0 50
20.60 is
0.9602a00
0.7812e.0
0.74520.00
0.701_nn.0s
Cyclic SGLD
25.73=0.14
23.30.2..
21.20
18.07
0.914
0.81822..04
0.753
0. 0.630=z.0zz
Fast Geometric Ens.
22.81=0.34
21.22=0.20
20. 20.79
19.61+e.1
0,798
0. 0.729=0..0
0.713ea.wzz
0.679=0.002
10
Deep Ensembles
22.55
18.3020.2
17.50 at
16.97a
0.675 0.84723.0
oo
0.63822.oo
0.504
CIFAR-100
Single model
22.66=m.31
22. 22.66=0.51
22 22.66+0.51
22
06.666..
0.818
0.818==0.05
0.818
0. 0.8-88=n.0s
Variational Inf. (FFG)
24.27
22, 22.41=0.11
22. 22.1410.11
21.86...or
0.924ea.0r7
0.829
0.813
0.795-o.001
Laplace
24.88
22.87.0.44
22.4116.a6
22.1416.:
0.94819amm
0.858_.0.01
1.83610.0
0.812_.n.0s
Snapshot Ensembles
30,30
22. 22.83
21,13
18.
48.2
1.069
0.820-=o.03
0.761
0. 0.6622-n.002
SWA-Gansian
24.38-0.55
20. 20.62=oo.s
20.08+0..9
19.
19.18+o.u
0.81/es..u
0.719:x0.0s
0.700
0. 0.667=0.004
Cyclic SGLD
20.3
22.37
20.23:0.22
17.13
p.888
0.7900:..00
0.722
0.606=...07
Fast Geometric Ens.
21.92_0.15
20.10.2..0
19.8710.5
18.731e.s
0.7652 0.699_o.001
0.6861a..c4
0.650_n.co3
ResNer164
Deep Ensembles
21.4
17.5320.1
16.90
16.50aNs
0.8192aww
0.647:20.on
0.6152a
0.574
CIFAR- 100
Single model
21.39 21.39_o.0
21.391...0
21.391..40
0.8171amm
0.817_o..ts
0.81719.044
0.817_...01
Variational Inf, (FFG)
23.47 26 21.35.20.1
21.10ao.1
16
20.824a.o
0.9102
0.801 002
0.78223
0.762_0..00
KFAC-Laplace
23.
23.44 21.77=0.30
21.29
21.03+e.ss
0.902sx.m
0.813320.0s
0.792
0. 0.772=0.007
Snapshot Ensembles
29.48-0.29
21.92=0.1s
20.27:+0.21
17.68
1.045es..co
0.789
0.729
0. 0.634=o..os
Dropout
20.19=0.11 19.41=0.17
19.36=0.12
19. t0.22
0.823ea.00
0.768=2..0s
0.760+a.oce
0.751-0..00
SWA-Gmusiin
20.45 17.57
17:21
17.08
0.7942
0.653
.01 0.61 0.6342a.
Cyclic SGLD
21.42==0.2
19.
17.88+0.1s
16.29+o.uc
0.813+a.mm
0.713=0.20
0.6511+a.ca
0.
0.583=0.004
Fast Geometric Ens.
21.48-00.11
18.54=0.14
18.00s0.19
17.12
0.770sa.ez7
0.6552-o.20s
0.630
0.5966n.003
WideResNet
Deep Ensembles
19.38_0.50
16.55_o..0
16.172o.1
15.77
0.797
0.623_
0.5951
1.57
CIFAR-100
Single model
19.31-0.24
19.31-0.24
19.31+0.24
19.31+a.24
0.797+a.mm
0. 0.799
0.797
Variational Inf. (FFG)
24.38.0.37
20.17
19.28.6.m
18.74
1.004
0.767 20.00
0.7271a.0a
0. 0.685_...02
KFAAC-
20.02.0.18 18 19.76...1
19.531o.1
10
19.431a.:
0.83423000
0,803 sos
0.7951
0.789_o..0m
Snapshot Ensembles
23.01=0.w 18.20 17.12so.:
16.074mar
0.67 0.6782o..m 0.869
a.633
0. 0.58220000
Table 3: Classification error and negative calibrated log-likelihood for difforent models and numbers of samples on CIFAR-1001000
25

---

Published as a conference paper at ICLR 2020
Faror (93
Negarive callibrated Iog-likelhhood
Model
Method
5
5
Dropout
5.81 vs 5.52.
5.82 vs 5.34;
5.70 vs 5.21
0.225 vs 0.187
0.224 vs 0.177,
0.223 vs 0.167.
SWA-Ganssia
v. 5.65.
5.49 5.36
5.25 vs
0.182 0.180.
0.166
0.160 0.115
Cyclic SGLDD
6.56 6.07.
5.71 vs 5.47
4.84 vo 4.88
0.196 vs 0.186
0.176 vs 0.160
0.147 vs 0.1477
Fast Geometric Lis.
v..64.
5.69 5.36
5.10 vs 4.98
0.187 vs 0.177
0.178
0.167
0.155 0.
VGG16
Deep Ensembles
4.79 vs 1.90
4.57 vs 4.73
4.39 vs 4.55
0.158 vs 0.162
0.118 vs 0.152
0.134 vs 0.136
CIFAR-10
Single model
5.83 5.
5.83 5.35
5.83 vs 5.19
0.223 vs 0.183
0.223 0.174
0.223 vs 0.16.
Variational (FFG)
5.63 5.43.
5.50 5.25
5.46 5.07
0.192 vs 0.182
0.169;
0.175 0..154
KFAC-Laplace
v..2.s55.199
5.82 vs 5.32
5.80 vs 5.117
0,203 vs 0.177,
0.201 vs 0.171,
0,200 v.
Snapshot Ensembles
6.66l..
5.00 5.03..
4.54 4.64
0.176 0.178
0.160
0. 0.100.
0.137 0.1
SWA-Gaussian
4.56
4.46 vs 4.34
4.34 vs 4.16,
0.143 vs 0.142
0.139 vs 0.135
0.131
0.15.
Cyclic SGLD
v.32.5.4.88.
4.55 4.35
3.81 vs 3.50
0.156 vs 0.148,
0.138 0.131
0.115 0.11
Fast Geometric Ens.
4.83 4.59.
4.73 vs 4.44)
4.28 vs 4.14.
0.141 vs 0.138
0.197 vs 0.132)
0.126 vs 0.121
ResNer 110
Deep Ensembles
3.77 vs 3.70
3.63 3.58
3.53 vs 3.45
0.1 vs 1.
0.112 0.112
0,106 vs 0.105.
CIFAR-10
Single model
vx 4.312
4.69 4,231
4.10
0.150 vs 0.137
0,150 vx 0.134
0,150 0.111
Variational Inf, (FFG)
1.91 1.66
1.72 vs 4.49
4.60 vs 1.25
0.149 vs 0.145
0114D vs 0.138
0,140 vs 0.130
KFAC-Laplace
5.14
4.90 v5-0..
4.78 4.17
0.100 0.139,
0,153 134,
0.147 0.
Snapshot Ensembles
4.85 4.788
4.16 18
3.52 vs 3.48
0.153 vs 0.150
0.132 vs 03130
0. vs 0.
SWA- Gaussinn
4.21 =
4.21 vs 4.04
4.02 vs 3.84
0.130
130.:
0.128 0. 126
0.121 0.16.
Cyclic SGLD
1.97 vs 4.66.
1.30 1,01,
3.06 3.48
0.149 vs 0.1-11,
0.131 vs 0, 125,
0.110 0s 0.107.
Fast Geometric Fins.
4.49 4.29
4.36 vs 4.
4.00 vs 3.85
0.134 vs 0.131
0.130 vs 0.126.
0. vs 0.15.
R6sNet164
Deep Ensembles
3.51 3.558
3.50 3.46
3.34 3.30
0.113 vs 0.114
0,107 vs 0.107
0,100 vs 0.100
CIFAR-10
Single model
4.52 4.15,
4.52 vs 4.08;
4.52 vs 3.98
0.144 vs 0.131
4,140 vs 0.128
0,141 vs d.126.
Variational Inf. (PFG)
1.78
1.66 1.22,
4.35 1.05
0.151 vs 0.142.
0,116 vs 135.
0.1/11 0.
Lapplce
4.77 vs 4.26.
4.65 vs. 4.21;
4.60 vs 4.08
0.151 vs 0.135
0.146 vx 0.132
0,142 vs 0.127.
Snapshot Ensembles
1.50 1.112
3.89 3.87
3.50 vs 3.35
0,1441 vs 0.142
0,124 0.123
0,101 0.10..
Drepout
3.70 62.
3.63 3.52;
3.61 vs 3.44,
0.120 vs 0.117,
0.119 vs 0.114,
0.117 0.11111
WWA-Gaussian
v..39.
3.34 vs 3.34
3.28 vs 3.24
0.111 vs 0.114
0.105 vs 0.107,
0.101 vs 10102..
Cyclic SGLDD
3..95...
3.63 3.58
3.19 3.22
0.128 vs 125
0.114 vs 0.112,
0.009 0.
Fast Geometric Ens.
3.95 v..5.s
3.77 vs 3,64,
3.34 vs 3.27
0.120 va 0.120
0,113 vs 0.113..
0,102 vs 0.12
WideResNet
Deep Ensembles
v.3.21
3.01 3.02
2,80 2.91-
0.097 0.103
0.085 vs 0.008;
0.090 0.084-
CIFAR-10
Single model
3.70
3.70 3.45
3.70 vs 3.40
0.124 vs 0.117
0.125 vs 01114,
0.124 vs 0.113.
Variational Inf. (FFG)
1.15 1.05
3.91 3.80
3.61 vs 3.63.
0.134 vs 0.136..
0.127 0,126,
0,117 vs 0.116
KPAC-Laplace
3.90 3 633
3.88 3.58,
3.83 vs 3.50
0.121 vs 0.115,
0,122 vs 0,113,
0.120 vs 0.11.111
Snapshot Ensembles
3.68 vs 3.74
3.33 vs. 3.36
2.80 vs 2.90
19 vs 0.122
0.105 0.107,
0.090 vs 0.093
Dropout
25.68 vs 21.37
25. 66 23 89
25. GO vs 23. 11
1. 1.111 vs 0. 0.999,
1. 1.098 0. 0.960_
1.081 vs 0.911.
SWA- -WA-Ganssia
24.53 vs
23.64 vs 23.27
22
22
22
0.931 vs 0.926..
0.879 vx 0,859
0.826 0.795.
Cyclic SGLD
26.79 va 25. 666
21.14 vs 23, 15
21 15
21
(4)
0.976 vy 0.929,
(18881 vs 0.848
0.7.1 vs 0.740
Fast Geometric Eos.
25.35 24.53,
23.62
22.
22 20
0.963 vA 0.921,
0,930 vs 0.878,
0.827 0.800
VGG16
Deep Ensembles
21.60 vx 21.90;
20.79 vs 21.03
19.88 vs 20.23
0.840 vs 0.805
0.794 vs 0.811
0.723 vs 0.731-
CIFAR-100 100
Single model
25.44 vs 24.35
25.44 23.92
25.44 23.48
1,087 vx 0.974
1.087 vx 0.945,
1.087 0.912,
Variational (FFG)
25.24 vx 24. 19
24.85 vs 23.56
24.56
22.80
1,001 vs 0.064.
0,973 vs 0911,
0.939 0.864
KFAC- Laplace
25.98 vs. 24.53
25.84 vs 24.03
25.70 vs 23.57
1.089 va 0.989.
1.069 vs 0.949,
1.050 vs 0.909.
Snapshot Ensembles
23.87 vs 28..
22.30)..
21.03
21 1M.
0.899 va 0.901
0.834 vs 0.836
0.751 0.750
SWA-Gaussian
22.31 va 20..
21. vs 21.21
20. vs 20.
0.781 va 0,770,
(1.45 D. 7.73)
0.701 vs 0.683
Cyelic SGLD
23.30 22.32
20.43,
18.07
17 67.67
0,818 vs 0,787
0.753
0, 725,
0.630 vs 0.616
Fast Geometric Ers.
21.22 vs 20 69
20.79 vs 20. 18
19.64 vs 19:25
0.729 vs 0.7
0.713 0.6944
0.679 vs 0.681.
ResNer110
Deep Ensembles
18.30 vs 18.36..
17.59 17.61
16.97 16.71
0.675 vs 0.672
018338
06635
0.591 vs 0.591
CIFAR-100
Single model
22.06 vs 21.37
22.
21.17,
22 06 vs 20.98
0.848 vs 0.2977
0,848 vA 01186
0.848 vs 0.775
Variational Inf. (PFG)
22.41 vv. 21 21,67
22.14
21.21
21 86 vs 20.77
0.829 vs 0.798
0.813 yx 0.775
0.795 vs 0.748.
KFAC Laplace
22.87 vs
22.41 vx
22. 20.99
0.858 vs 0.810
0.836 vs 0.788
0.812 vs 0.766
Snapshot Ensembles
22.83 va 22. 33
21.13 vs 20.71
18. 18 vs 18 13.
0,820 va 0.806
0.761 vs 0.744.
0.662 vs 0.661.
SWA-Ganssia
20.62 vs 20.61..
20.08.,
19:18 19, 33-
0.719 vs 7.75
0.700
0.690.
0.687 0.65L
Cyclic SGLD
22.37 vs 21.57
20.23 vs 19.58
17.13 vs 16.99
0.790 vs 0.787
0.722 vs 0.702
0.606 vs 0.585.
Fist Geometric Eus.
20.10 vs 19.82
19.87 19.39
18.73 18.33
0.699 vs 0.689
(1686 0.671,
0.650 vs 0.636
6esNer164
Deep Ensembles
17.53 vs 17.37
16.90 vs 16.81
16.50 vs 16.22
0.647 vs 0 650
0,615 vs 0.613.,
0.574 vs 0.375
CIFAR-100
Single model
21.30 vs. 20.60
21.30 vs 20.30
21 30 vs 20
0.817 vs 0.772,
0.817 vs 0.761.
0.817 vs 0.751.
Variational (FFG)
21.35 xs
.6.
21.10 vs 20.54
20.82 19.97
0.801 vx 0.785
0.762 vx 0,759,
0.762 vs 0.731
LPPCC-
21.77 vs 20. 66
21.29 vs 20.36
21 03 vs 20.18
0.813 vs 0.778,
0.792 ys 0.708
0.772 vs 0.740
Snupshot Ensembles
21.92 21 10
20.27 19.92
17.66
17.66
0.789 vs 0.781
0.729 vs 0.720
0.634 0.629
Drepout
19.41 vs 19.27
19.36 vs 19.11
19:22 vs 18.58
0.768 vs 0.751
0.760 vs 0.738
0.751 vs 0.723.
SWA-Gaussian Cyclic SGLD
17.57 vs 17.79.
17.21 17.27
17:08 16.89.
0.653 vs 0.638,
(16334 0.635..
0.614 vs 0.610.
19.12 vs 18.83,
17.88 vs 17.50,
16. vs 16.21
0.713 vs 0. 0.696
0.654 vs 0,611,
0.583 vs 0.380
Past Geometric Fin.
18.54 vs 18.39
18.00 vs 17.84;
17. 12 vs 16.93
0.652 vs 0.649:
0.630 vs 0.624,
0.596 vs 0.592.
WideResNet Deep Ensembles
16.35
16.81.
16.30
15.77 15.77
0.623 vs 0.632,
0.602,
0.371 0.373
CIFAR-100
Single model
19.31 vS 88,
19.31 vs 18,80
19.31 vs 18.72
0.797 vs 0.756
0.797 vs 0.746
0.797 vs 0.738.
Variational Inf. (PFG)
20.17 vx 20. 12.
19.28 19.20.,
18.74 18.54
0.767 0.766.,
0.727 0.724
0.685 0.679
Laplace
19.76 vs 19.21
19.53 vs 19.03;
19.43 vs 18.93
0.803 vs 0.784,
0.795 vs 0.757
0.789 vs 0.747.
Snapshot Ensembles
18.20 vs 18.22
17.12 17.20).
16.07 16.27 0.678 vs 0.680., 01633 0.633.
0.582 (.386
Table Classification error negative calibrated log- likelibood before vs. after test-time augmentation on CIFAR-101100.
26

---

Published as a conference paper at ICLR 2020
Deep ensemble equivalent
Model
Method
sample
samples 10 samples 50 samples
100 samples
Deep Ensembles
1.0
5.0
10.0
50.0
100.0
Snapshot Ensembles
1.0
2.6
4.7
21.9
33.8
Cyche SGLD
1.0
1.7
2.6
MWA-Gaussian
1.0
2.2
2.9
4.4
4.6
VGGI6
Fast Geometric Ens.
1.3
2.0
2.4
4.7
5.8
CIFAR-10
Dropout
1.0
1.0
1.0
1.1
Variational Inf. (FFG)
1.0
1.8
2.6
KFAC-Laplace
1.4
1.6
1.6
1.6
1.6
Single model
1.1
1.1
1.1
1.1
1.1
Deep Ensembles
1.0
5.0
10.0
48.0
94.6
Snupshot Ensembles
1.0
1.0
14.6
32.6
Cyclic SGLD
1.0
1.0
1.6
4.6
6.0
ResNet110 CIFAR-10
SWA-Gaussian
1.0
1.3
1.5
1.9
2.0
Fast Geometric
1.0
1.4
2.9
Variational Inf. (FFG)
1.0
1.0
1.5
KFAC-Laplace
1.0
1.0
1.0
1.0
1.1
Single model
1.0
1.0
1.0
1.0
1.0
Deep Ensembles
1.0
5.0
10.0
43.7
100.0
Snapshot Ensembles
1.0
1.1
11.3
17.2
Cyclic SGLD
1.0
1.0
1.8
4.8
6.7
ResNer164 CIFAR-10
SWA WWA-Gaussian
1.0
1.8
1.9
2.5
2.7
Fast Geometric Ens.
1.0
Variational Inf. (FFG)
1.0
1.0
1.3
1.3
LFAC-Laplace
1.0
1.0
1.0
1.2
1.2
Single model
1.1
1.1
Deep Ensembles
10.0
Snapshor Ensembles
1.0
1.3
2.5
30.6
100.0
Cyclic SGLD
1.0
1.0
1.6
4.0
4.1
SWA-Gassian
1.0
1.8
3.3
WideResNet
Fast Geometric Ens.
1.0
1.2
3.2
CIFAR-10
Dropout
1.0
1.2
1.3
1.4
1.4
Variational Inf. (FFG)
1.0
1.0
1.0
1.3
1.4
KFAC-Laplace
1.0
Single model
1.0
1.0
1.0
1.0
Deep Ensembles
1.0
5.0
10.0
50.0
100.0
Snapshot Ensembles
1.0
18.8
Cyclic SGILD>
1.0
1.8
3.3
15.9
28.2
SWA-Gaussian
1.0
2.3
3.4
5.2
5.9
VGGI6
Fust Geometric
1.2
1.9
4.5
CIFAR-100
Dropout
1.0
1.0
1.1
Variational Inf. (FTG)
1.0
1.6
1.8
2.1
2.2
1.0
1.0
1.3
1.3
Single model
Deep Ensembles
1.0
5.0
10.0
50.0
Snapshor Easembles
1.0
1.3
1.9
4.7
6.2
Cyclic SGLD
1.0
1.3
2.0
7.0
12.2
ResNer110
SWA-Gaussian
1.7
CIFAR-100
Fast Geometric Lnn.
1.5
2.6
4.2
4.7
Variational Inf. (FFG)
1.0
1.2
1.3
1.5
1.5
KFAC-Laplace
1.0
1.0
1.3
1.4
Single model
1.0
Deep Ensembles
1.0
5.0
10.0
50.0
100.0
Snapshot Ensembles
1.0
1.3
1.9
4.8
6.3
Cychi SGLLD)
1.0
1.3
2.0
6.8
13.3
ResNet164
SWA-Gaussian
1.0
2.1
3.8
CIFAR-100
Fast Geometric Ens.
1.6
2.6
3.0
4.2
4.8
Variational Inf. (FFG)
1.0
1.2
1A
1.6
1.6
KFAC-Laplace
1.0
1.1
1.5
Single model
1.0
1.0
1.0
Deep Ensembles
1.0
5.0
10.0
49.0
95.9
Snapshot Ensembles
1.0
2.5
4.2
16.2
20.0
Cyclic SGLD
1.0
1.9
13.6
18.8
SWA-Gaussian
1.0
3.2
6.0
WideResNet
Fist Genmetric Fins.
1.3
3.2
4.4
7.5
9.5
CIFAR-100 Dropout
1.0
1.3
1.4
1.5
1.5
Variational Inf.
KFAC-L Laplaee
1.0
1.0
1.1
Single model
1.0
1.0
1.0
1.0
1.0
Table Deep ensemble equivalent CIFAR-10100
27

---

Published as a conference paper at ICLR 2020
Error (%)
Negative calibrated log-likclibood
Model
Method
Fast Geometric Ens.
23.71-n.c0
23.61=0.0
23.56=0.m0
28.28+0.00
0.929
0.921=a.coc
0.910
0.901eo.cm
Deep Ensembles
23.79=a.14
21.19
20.90
20.63aMA
0.935e0.00r
0.823aooo
0.805
0.788=NA
ResNet50
Single model
23.86_e.00
23.86g0.0
23.8620.3
23.8610.9
0.938 0.938
0.93822.mx noe
0.938 oos
Variational Inf. (FFG)
24.50=n.00
23.8
23.77.0.04
23.07
0.927 0.907
0.920 0.923
KFAC- SIFC--Laphan Laplace
25.01
24.19
23.93.0.a
23.861o.1
0.988
10.022
0.948
1..091...01
0.9344. oos
Snapshot Ensembles
24.92+xA
22.21
21.75
21.48=xa
0.983
0.805
0. 0.943
0.830
Table Classification error and negative calibrated log-likelihood for different numbers of samples ImageNet.
Erorr (9)
Negative calibrated log- aigeliil
Model
Methed
5
10
10
50
Geometric
23.61
56
21.37,
vs 0.894
0.842
0.904 0.793
Deep Ensembles
21.19 vs. 21.20.
20.00 vs 20 16
20.63 vs 19.39
0.823 vs 0.855
0.505 vs 0.793
0.788 vs 0. 0370
ResNet50
Single model
23.86
23.86
0.900
Variational Inf. (FFG)
23.82 vs 22.58
20.77 vs 21.74
23.67 vs 21.10
0.927 vs 0.905
0.923 vs 0.851
0.920 vs 0.805;
KFAC-Laplace
24.19 22.50
23.93 vs 21.67
23.86 vs 04,
0.948 vs 0.906,
0.939 0.855;
0.934 0.809
Snapshot Ensembles
22.21 vs 21.90.
21.75 vs 20.81
21.48 vs 19.86, 0.865 vs 0.879;
0.843 vs 0.815; 0.830 vs 0.766,
Table Classitication negative log- log-likelibood
augmentation ImageNet.
Deep ensemble equivalent score
Model
Method
sample 5 samples
10 samples
25 samples 50 samples
Deep Ensembles
1.0
5.0
10.0
25.0
50.0
Snapshot Ensembles
1.0
2.2
4.0
ResNet50 Fast Geometric Ens.
1.1
1.2
1.4
Variational Inf. (FFG)
1.0
1.1
1.2
1.2
EFAC-Lappace
1.0
1.0
1.0
Single model
1.0
1.0
1.0
1.0
Table 8: Deep ensemble equivalent score for ImageNet.
28

---

Published as a conference paper at ICLR 2020
CIFAR10
x central crop (onsemble size 10)
X
central drop
(ensemble size 50)
aug: (onsemble 10)
tost-tma sug. (ensemble
VGG16
VGG16
5.8
Neg. callbrated log-likellhood
0.22
x
Top-1 error (%)
5.4
0.20
5.2
0.18
X
5.0
X
4.8
0.16
4.6
4.4
Single K-FAC-L
FFG
SWAG
FGE
cSGLD
SSE
DE
Single K-FAC-L
FTG
SWAG
FGE
cSGLD
SSE
DC
model
VI
model
VI
ResNet110
ResNet1 10
4.8
x
X
Neg. calbrratd log-llkeee se--iiellhoodd
0.15
4.6
Top-1 error (%)
0.14
4.2
0.13
4.0
0.12
3.8
3.6
0.11
3.4
Single K-FAC-L
FTG
SWAG
FGE
cSGLD
SSE
DE
Single K-FAC-L
FFG
SWAG
FGE
cSGLD
SSE
DE
model
VI
model
VI
ResNet164
ResNet164
4.6
Neg. calibrated log -iikellhood
x
0.14
4.4
Top-1 error (%)
4.2
0.13
4.0
X
3.8
3.6
0.11
3.4
0.10
Single K-FAC-L
FFG
SWAG
FGE
eSGLD
SSE
DE
Single K-FAC-L
FFG
SWAN
FGE
cSGLD
SBE
DE
model
M
model
VI
WideResNet
WideResNet
x
0.125
3.8
log-lkkelhhod
0.120
Top-1 error (%)
0.115
X
0.110
0.105
3.2
0.100
0.095
2.8
0.090
Single K-FAC-L
FFG
SWAG
FGE
<SGLD
SSE
DE
Single KFFA-L
FFG
SWING
FGE
cSSLD
SSE
DE
modal
MI
mosel
VI
Figure 14: Classification error and negative calibrated log-likelihood before after test-time aug- mentation on CIFAR-10
29

---

Published as a conference paper at ICLR 2020
CIFAR100
x
central crop (ensemble size 10)
X
central drop
(ensemble size 50)
aug: (onsemble 10)
tost-tma sug. (ensemble
VGG16
VGG16
28
1.10
Neg. callbrated log-likellhodd
x
25
1.05
Top-1 error (%)
1.00
X
0.95
23
0.90
22
0.85
0.80
0.75
20
Single K-FAC-L modal
ITG
SWAG
FGE
cSGLD
SSE
DC
Single K-FAC-L model
FFG
SWAG
FGE
cSGLD
SSE
DC
VI
ResNet1 10
ResNet110
x
0.85
x
22
X
X
Top-1 error (%)
Neg. log-atodlgg-lkklhiie
0.80
0.75
20
X
0.70
19
18
0.65
17
Single K-FAC-L
IFG
SWAG
FGE
cSGLD
SSE
DE
Single K-FAC-L model
FFG
SWAG
FGE
dSGLD
SSE
DE
model
VI
ResNet164
ResNet164
Neg. calibrated log-likellhood
0.80
X
X
Top-1 error (%)
20
0.75
19
0.70
18
0.65
17
0.60
Single K-FAC-L
IFG
SWAG
FCE
cSGLD
SSE
DE
Single K-FAC-L
FFG
SWAG
FGE
<SGLD
SSE
DE
model
VI
model
VI
WideResNet
WideResNet
0.80
Neg. calibrated log-likelitod ood ood
19
Top-1 error (%)
0.75
0.70
17
0.65
0.60
16
Single K-FACAL FFG
SWAG
FGE
cSGLD
SSE
DE
Single K-FAC-L model
FFG
SWAG
FGE
dSGLD
SSE
DE
modal
VI
Figure 15: Classification error and negative calibrated log-likelihood before after test-time aug- mentation on CIFAR-100
30


Reviews:
[{"review_id": "rkgh0fP3FS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper provide an extensive review of current advances in uncertainty estimation in neural networks with the analysis of drawbacks of currently used uncertainty metrics and comparison on scale the recent method to estimate uncertainty. The paper covers a lot of uncertainty metrics and a wide range of methods. The paper focuses on in-domain uncertainty estimation complementing the recent similar review on out-of-domain uncertainty estimation.\n\nIt seems that the paper provides the analysis missing in the current literature. Whereas as mentioned Yukun Ding in a public comment there is a related work on identifying issues with popular uncertainty metrics, the mentioned paper is missing the through comparison of the methods for estimating uncertainty. \n\nSuch kind of thorough analysis (especially performed on scale on large datasets) and comparison is of obvious interest to the community as well as objective comparison of the current state-of-the-art. \n\nThe paper is clearly written and easy to follow.\n\nBased on this, I believe this is a strong technical paper and it should be accepted. However, the analysis in the paper is not overwhelmingly exhaustive. Some of the arguments on that are listed below.\n\nBelow is the list of comments/thoughts for potential improvement of the paper:\n1.\t“In this case, a model is expected to provide correct probability estimates:” – may be not the best choice of words, because for out of domain uncertainty estimation we still expect a model to provide correct probability estimates\n2.\tThe first paragraph on page 3 seems to better fit in Section 2, for example, on the very beginning of Section 2.\n3.\t“Comparison of the log-likelihood should only be performed at the optimal temperature.” and others alike – personally, I do not support this kind of formatting for a scientific paper\n4.\t“can produce an arbitrary ranking of different methods. <…> Empirically,” – in the current form it seems that the first statement is somehow theoretically justified and then additionally it is confirmed empirically in this paper. I believe that the authors use empirical observation itself as the justification of the first statement, if that the case it should be reworded here. For example, “can produce an arbitrary ranking of different methods as we show below/ as we show empirically. We demonstrate that the overall …” If my belief is incorrect and there are other grounds that justify the first statement that it is required a reference after this statement.\n5.\tItalic and non-italic LL usage is unclear\n6.\tHaving “Brier score” emphasised as a paragraph, it seems that there should be a paragraph log-likelihood as well\n7.\t“In that case, both objects and targets of induced binary classification problems remain fixed for all models” – do the authors consider in this case all out-of-domain objects as having a positive class and all in-domain objects as having a negative class? Because the models are still going to make individual misclassification mistakes\n8.\tFigure 2 – legend occupies too much space of the plot occluding almost a third part of the plot. Maybe taking the legend out of the plot to the right and squeezing the plot to make a room for the legend would be a better solution\n9.\tIn eq. (4) and (5) subscript DE is not defined\n10.\t“SSE and cSGLD outperform all other techniques except deep ensembles” –cSGLD was not applied on ImageNet, therefore this statement is a bit misleading\n11.\tColour of SWAG in Figure 3 is not very clear. Only excluding other colours I can determine which line is SWAG. Similar to cSGLD, is seems that SWAG was not applied on ImageNet. Why is that if that is the case? And it should be clearly stated at least in experimental setup in Supplementary  \nFor colours in general, lines in legends are very thin and it is difficult to assess their colour. I appreciate the authors compare a lot of methods and therefore have to use a lot of colours, but it is quite difficult to assess them even on screen not to mention if the paper is printed out. Could the authors please use thicker lines in legends at least?\n12.\t“Being more “local” methods” – without any context in the main paper this referral to “local” methods is unclear. Also it is good to add a reference to Appendix review of the considered methods in the main text.\n13.\tMissing details of what kind of augmentation is used in Section 4.3. Is it the same as training augmentation specified in Supplementary? It would require a reference to Supplementary\n14.\t“(Figure 1, Table REF)” – missing number for Table\n15.\t“Our experiments demonstrate that ensembles may be severely miscalibrated by default while still providing superior predictive performance after calibration.” – unclear which experiments demonstrate this and superior in comparison to what\n16.\tThe issues of uncalibrated log-likelihood and TACE are clearly shown in the paper, whereas the issues with misclassification detection are only verbally discussed. An illustrative example, at least a toy thought example, could really improve the paper here\n17.\tThe chosen main performance metric is not very convincingly motivated. It is clear why it is based on calibrated log-likelihood, but it is not very convincing why one cannot just used calibrated log-likelihood as a performance metric, why one should base the metric on deep ensembles instead. Also from the long-term perspective, if the community comes up with methods clearly outperforming deep ensembles, the metric would need to be based on one of these new methods \n18.\tThere is an indirect uncertainty metric that is not mentioned in the paper – uncertainty used in active learning (see, e.g., Hernández-Lobato and Adams, 2015. Probabilistic backpropagation for scalable learning of Bayesian neural networks)\n19.\tFigures 4 and 5 are too small\n20.\t“the original PyTorch implementations of SWA” – SWA is not considered in the paper\n21.\t“hidden inside an optimizer … The actual underlying optimization problem” – it seems that the ICLR audience should be familiar with “actual optimization problems” rather than using blindly the optimizer. It is always good to explicitly write down an equation that is used in a paper, but this wording seems a bit off for ICLR \n22.\t“\\hat{p}(y^∗_i = j | x_i, w) denotes the probability that a neural network with parameters w assigns to class j when evaluated on object x_i” – it should be \\hat{p}(y_i = j | x_i, w), y^*_i is observed\n23.\t\na.\tWhy was dropout applied only for limited number of architectures and not applied on ImageNet at all?\nb.\tWhy wasn’t cSGLD applied on ImageNet\n24.\t“On CIFAR-10/100 parameters from the original paper are reused” – it is better to repeat the reference here\n\n\nMinor:\n1.\tFont size in eq. (10) should be the same as the rest of the paper\n2.\t“Or models achived top-1 error of”: “Or” - ?, “achived” -> achieved \n3.\t“for a 45 epoch form a per-trained model”: “form” -> “from”?", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "rJg0iIgAFH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors response on 13th nov regarding the main concerns I have are valid. They make sense. I thank the authors for the detailed explanations. I went back and did another thorough read of the work. As it stands, I am OK to change my review to weak accept. \n\n---------------------------------\n\nThe authors evaluate a variety of ensemble models in terms of their ability to capture in-domain uncertainity. A set of metrics are used to perform these evaluations and in turn, using deep ensenble as a reference, the authors study the behaviour/capacity of the rest of the methods. \n\nAlthough the motivation for the work is sensible, there are several critical issues with the paper and the summary is not necessarily conclusive in terms of gaining any new insights. \n1. Much of the evaluations rely on the choice/nature of the optimal temperature, which would be different for different models. The authors suggest to use the model-specific optimal values when comparing instead of fixing the temperature? Why is this the case? Further, if we take this into account (i.e., allow for comparing different temperatures) then much of the differences between DEE and others cannot be directly interpreted. This is the case when using log-likelihood and Brier scores. \n2. AUC can be transformed into a normalized probability distribution (CDF), and hence in principle it is model/hyperparameters agnostic. This is one of the reasons i is used as information criterion in Bayesian model selection. Area of AUC is a valid metric as well. To that end, why do the authors suggest that it cannot be used as criteria for comparison across models? \n3. From section 3.5 it is not clear how test time cross validation is tackling temperature scaling? \n4. In section 4.1, the hypothesis on #independent trained networks is great and it makes sense? How is this translating ito the evaluations? None of the results actually talk about this aspect directly? Or am I missing something here?\n5. Setting the evaluations with DEE as reference is problematic because we already know from random sampling theory that deep ensemble is better than the normal ensembles (tech results on random sampling for model fitting and RL etc. optimization results on mode finding with single mode vs. multi model methods also say similar things) and in fact that was the main motivation. Also normal regularization (like dropout or K-facL are more towards overfitting than ensembling) are not really an ensemble. Putting these together, most of the conclusions and the lots (figure 3 in particular) is by definition true. Nothing surprising. \n\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "S1lvoIaRtH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThis paper mainly concerns the quality of in-domain uncertainty for image classification. After exploring common standards for uncertainty quantification, the authors point out pitfalls of existing metrics by investigating different ensembling techniques and introduce a novel metric called deep ensemble equivalent (DEE) that essentially measures the number of independent models in an ensemble of DNNs. Based on the DEE score, a detailed evaluation of modern DNN ensembles is performed on CIFAR-10/100 and ImageNet datasets.\n\nStrengths:\nThe paper is well written and easy to follow. The relationship to previous works is also well described. Overall, I think this is a good paper, which gives a detailed overview of existing metrics for accessing the quality in in-domain uncertainty estimation. The idea behind the proposed DEE score is nice and simple, clearly showing the quality of different ensembling methods (in Fig. 3). Given the importance of uncertainty analysis to deep learning, I believe this work will have a positive impact on the community.\n\nWeaknesses:\n- On page 6, the authors mention that the prior in Eq. 3 is taken to be a Gaussian N(\\mu, diag(\\sigma^2)) for Bayesian neural networks, however, many other choices of a prior distribution are available in the literature. What is the impact of changing prior distributions on the quality of uncertainty estimates in the case of variational inference? \n- Data augmentation is commonly used for improving model performance. However, I find the results presented in Sect 4.3 are not clear enough, note that for a given ensembling method in Table 1, the negative calibrated log-likelihood may increase or decrease when using different networks (VGG, ResNet, etc.). I think it would be interesting to elaborate a bit more on the influence of model complexity.\n- On page 15, in Eq. 12, the choice of the variance parameter \\sigma_p^2=1/(N*wd) seems unclear and should be better explained.\n\nMinor comments:\nThe size of some figures appears too small, for example Fig. 4 and Fig. 5, which may hinder readability.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper points out pitfalls of existing metrics for in-domain uncertainty quantification, and also studies different strategies for ensembling techniques.

The authors also satisfactorily addressed the reviewers' questions during the rebuttal phase. In the end, all the reviewers agreed that this is a valuable contribution and paper deserves to be accepted. 

Nice work!
Decision: Accept (Poster)