Year: 2021

Abstract:
Uncertainty estimation and ensembling methods go hand-in-hand. Uncertainty estimation is one of the main benchmarks for assessment of ensembling performance. At the same time, deep learning ensembles have provided state-of-the-art results in uncertainty estimation. In this work, we focus on in-domain uncertainty for image classification. We explore the standards for its quantification and point out pitfalls of existing metrics. Avoiding these pitfalls, we perform a broad study of different ensembling techniques. To provide more insight in this study, we introduce the deep ensemble equivalent score (DEE) and show that many sophisticated ensembling techniques are equivalent to an ensemble of only few independently trained networks in terms of test performance. video / code / blog post

text: video / code / blog post

heading: 1 INTRODUCTION
text: Deep neural networks (DNNs) have become one of the most popular families of machine learning models. The predictive performance of DNNs for classification is often measured in terms of accuracy. However, DNNs have been shown to yield inaccurate and unreliable probability estimates, or predictive uncertainty (Guo et al., 2017). This has brought considerable attention to the problem of uncertainty estimation with deep neural networks.
There are many faces to uncertainty estimation. Different desirable uncertainty estimation properties of a model require different settings and metrics to capture them. Out-of-domain uncertainty of the model is measured on the data that does not follow the same distribution as the training dataset (out-of-domain data). Out-of-domain data can include images corrupted with rotations or blurring, adversarial attacks (Szegedy et al., 2013) or data points from a completely different dataset. The model is expected to be resistant to data corruptions and to be more uncertain on out-of-domain data than on in-domain data. On the contrary, in-domain uncertainty of the model is measured on data taken from the training data distribution, i.e. data from the same domain. In this setting the model is expected to produce reliable probability estimates, e.g. the model shouldn’t be too overconfident in its wrong predictions.
Pitfalls of metrics We show that many common metrics of in-domain uncertainty estimation (e.g. log-likelihood, Brier score, calibration metrics, etc.) are either not comparable across different models or fail to provide a reliable ranking. We address some of the stated pitfalls and point out more reasonable evaluation schemes. For instance, although temperature scaling is not a standard for ensembling techniques, it is a must for a fair evaluation. With this in mind, the
∗Equal contribution §HSE refers to National Research University Higher School of Economics †Skoltech refers to Skolkovo Institute of Science and Technology ‡HSE refers to Samsung-HSE Laboratory, National Research University Higher School of Economics
calibrated log-likelihood avoids most of the stated pitfalls and generally is a reasonable metric for in-domain uncertainty estimation task.
Pitfalls of ensembles Equipped with the proposed evaluation framework, we are revisiting the evaluation of ensembles of DNNs—one of the major tools for uncertainty estimation. We introduce the deep ensemble equivalent (DEE) score that measures the number of independently trained models that, when ensembled, achieve the same performance as the ensembling technique of interest. The DEE score allows us to compare ensembling techniques across different datasets and architectures using a unified scale. Our study shows that most of the popular ensembling techniques require averaging predictions across dozens of samples (members of an ensemble), yet are essentially equivalent to an ensemble of only few independently trained models.
Missing part of ensembling In our study, test-time data augmentation (TTA) turned out to be a surprisingly strong baseline for uncertainty estimation and a simple way to improve ensembles. Despite being a popular technique in large-scale classification, TTA seems to be overlooked in the community of uncertainty estimation and ensembling.

heading: 2 SCOPE OF THE PAPER
text: We use standard benchmark problems of image classification which comprise a common setting in research on learning ensembles of neural networks. There are other relevant settings where the correctness of probability estimates can be a priority, and ensembling techniques are used to improve it. These settings include, but are not limited to, regression, language modeling (Gal, 2016), image segmentation (Gustafsson et al., 2019), active learning (Settles, 2012) and reinforcement learning (Buckman et al., 2018; Chua et al., 2018).
We focus on in-domain uncertainty, as opposed to out-of-domain uncertainty. Out-of-domain uncertainty includes detection of inputs that come from a completely different domain or have been corrupted by noise or adversarial attacks. This setting has been thoroughly explored by (Ovadia et al., 2019).
We only consider methods that are trained on clean data with simple data augmentation. Some other methods use out-of-domain data (Malinin & Gales, 2018) or more elaborate data augmentation, e.g. mixup (Zhang et al., 2017) or adversarial training (Lakshminarayanan et al., 2017) to improve accuracy, robustness and uncertainty.
We use conventional training procedures. We use the stochastic gradient descent (SGD) and use batch normalization (Ioffe & Szegedy, 2015), both being the de-facto standards in modern deep learning. We refrain from using more elaborate optimization techniques including works on superconvergence (Smith & Topin, 2019) and stochastic weight averaging (SWA) (Izmailov et al., 2018). These techniques can be used to drastically accelerate training and to improve the predictive performance. Thus, we do not comment on the training time of different ensembling methods since the use of these and other more efficient training techniques would render such a comparison obsolete.
A number of related works study ways of approximating and accelerating prediction in ensembles. The distillation mechanism allows to approximate the prediction of an ensemble by a single neural network (Hinton et al., 2015; Balan et al., 2015; Tran et al., 2020), whereas fast dropout (Wang & Manning, 2013) and deterministic variational inference (Wu et al., 2018) allow to approximate the predictive distribution of specific stochastic computation graphs. We measure the raw power of ensembling techniques without these approximations.
All of the aforementioned alternative settings are orthogonal to the scope of this paper and are promising points of interest for further research.

heading: 3 PITFALLS OF IN-DOMAIN UNCERTAINTY ESTIMATION
text: No single metric measures all the desirable properties of uncertainty estimates obtained by a model of interest. Because of this, the community is using many different metrics in an attempt to capture the quality of uncertainty estimation, such as the Brier score (Brier, 1950), log-likelihood (Quinonero-Candela et al., 2005), metrics of calibration (Guo et al., 2017; Nixon et al., 2019), performance of misclassification detection (Malinin & Gales, 2018), and threshold–accuracy curves
(Lakshminarayanan et al., 2017). In the section we highlight the pitfalls of the aforementioned metrics, and demonstrate that these pitfalls can significantly affect evaluation, changing the ranking of the methods.
Notation We consider a classification problem with a dataset that consists of N training and n testing pairs (xi, y∗i ) ∼ p(x, y), where xi is an object and y∗i ∈ {1, . . . , C} is a discrete class label. A probabilistic classifier maps an object xi into a predictive distribution p̂(y |xi). The predictive distribution p̂(y |xi) of a deep neural network is typically defined by the softmax function p̂(y |x) = Softmax(z(x)/T ), where z(x) is a vector of logits and T is a scalar parameter standing for the temperature of the predictive distribution. This scalar parameter is usually set to T = 1 or is tuned on a validation set (Guo et al., 2017). The maximum probability maxc p̂(y = c |xi) is called the confidence of a classifier p̂ on an object xi. I[·] denotes the indicator function throughout the text.

heading: 3.1 LOG-LIKELIHOOD AND BRIER SCORE
text: The average test log-likelihood LL = 1n ∑n i=1 log p̂(y = y ∗ i |xi) is a popular metric for measuring the quality of in-domain uncertainty of deep learning models. It directly penalizes high probability scores assigned to incorrect labels and low probability scores assigned to the correct labels y∗i .
LL is sensitive to the softmax temperature T . The temperature that has been implicitly learned during training can be far from optimal for the test data. However, a nearly optimal temperature can be found post-hoc by maximizing the log-likelihood on validation data. This approach is called temperature scaling or calibration (Guo et al., 2017). Despite its simplicity, the temperature scaling results in a notable improvement in the LL.
While ensembling techniques tend to have better temperature than single models, the default choice of T = 1 is still suboptimal. Comparing the LL with suboptimal temperatures—that is often the case in practice—can potentially produce an arbitrary ranking of different methods.
Comparison of the log-likelihood should only be performed at the optimal temperature.
Empirically, we demonstrate that the overall ordering of methods and also the best ensembling method according to the LL can vary depending on temperature T . While this applies to most
ensembling techniques (see Figure 10), this effect is most noticeable on experiments with data augmentation on ImageNet (Figure 1).
We introduce a new metric called the calibrated log-likelihood that is the log-likelihood at the optimal temperature.
The calibrated log-likelihood considers a model and a post-training calibration as a unified system, targeting to measure all models in the equal conditions of optimal temperature. That allows to avoid measuring calibration error that can be eliminated by a simple temperature scaling. The metric significantly affects the results of the comparison. For example, in Figure 10 the differences between Bayesian (VI, K-FAC, SWAG, dropout) and conventional non-Bayesian networks become much less pronounced, and in most cases making conventional non-Bayesian networks match the performance of Bayesian ones (VI, K-FAC, Dropout) on ResNet110, ResNet164, and WideResNet.
We show how to obtain an unbiased estimate of the calibrated log-likelihood without a held-out validation set in Section 3.5.
LL also demonstrates a high correlation with accuracy (ρ > 0.86), that in case of calibrated LL becomes even stronger (ρ > 0.95). That suggests that while (calibrated) LL measures the uncertainty of the model, it still has a significant dependence on the accuracy and vice versa. A model with higher accuracy would likely have a higher log-likelihood. See Figure 9 in Appendix C for more details.
Brier score BS = 1n 1 C ∑n i=1 ∑C c=1(I[y∗i = c]− p̂(y = c |xi))2 has also been known for a long time as a metric for verification of predicted probabilities (Brier, 1950). Similarly to the log-likelihood, the Brier score penalizes low probabilities assigned to correct predictions and high probabilities assigned to wrong ones. It is also sensitive to the temperature of the softmax distribution and behaves similarly to the log-likelihood. While these metrics are not strictly equivalent, they show a high empirical correlation for a wide range of models on CIFAR-10, CIFAR-100 and ImageNet datasets (see Figure 8 in Appendix C ).

heading: 3.2 MISCLASSIFICATION DETECTION
text: Detection of wrong predictions of the model, or misclassifications, is a popular downstream problem relevant to the problem of in-domain uncertainty estimation. Since misclassification detection is essentially a binary classification problem, some papers measure its quality using conventional metrics for binary classification such as AUC-ROC and AUC-PR (Malinin & Gales, 2018; Cui et al., 2019; Możejko et al., 2018). These papers use an uncertainty criterion like confidence or predictive entropyH[p̂(y |xi)] as a prediction score. While these metrics can be used to assess the misclassification detection performance of a single model, they cannot be used to directly compare misclassification performance across different models. Correct and incorrect predictions are specific for every model, therefore, every model induces its own binary classification problem. The induced problems can differ significantly, since different models produce different confidences and misclassify different objects. In other words, comparing such metrics implies a comparison of performance of classifiers that solve different classification problems. Such metrics are therefore incomparable.
AUCs for misclassification detection cannot be directly compared between different models.
While comparing AUCs is incorrect in the setting of misclassification detection, it is correct to compare these metrics in many out-of-domain data detection problems. In that case, both objects and targets of the induced binary classification problems remain the same for all models. All outof-domain objects have a positive label and all in-domain objects have a negative label. Note that this condition does not necessarily hold in the problem of detection of adversarial attacks. Different models generally have different inputs after an adversarial attack, so such AUC-based metrics might still be flawed.

heading: 3.3 CLASSIFICATION WITH REJECTION
text: Accuracy-confidence curves are another way to measure the performance of misclassification detection. These curves measure the accuracy on the set of objects with confidence maxc p̂(y = c |xi) above a certain threshold τ (Lakshminarayanan et al., 2017) and ignoring or rejecting the others.
The main problem with accuracy-confidence curves is that they rely too much on calibration and the actual values of confidence. Models with different temperatures have different numbers of objects at each confidence level which does not allow for a meaningful comparison. To overcome this problem, one can swhich from thresholding by the confidence level to thresholding by the number of rejected objects. The corresponding curves are then less sensitive to temperature scaling and thus allow to compare the rejection ability in a more meaningful way. Such curves have been known as accuracyrejection curves (Nadeem et al., 2009). In order to obtain a scalar metric for easy comparisons, one can compute the area under this curve, resulting in AU-ARC (Nadeem et al., 2009).

heading: 3.4 CALIBRATION METRICS
text: Informally speaking, a probabilistic classifier is calibrated if any predicted class probability is equal to the true class probability according to the underlying data distribution (see (Vaicenavicius et al., 2019) for formal definitions). Any deviation from the perfect calibration is called miscalibration. For brevity, we will use p̂i,c to denote p̂(y = c |xi) in the current section. Expected calibration error (ECE) (Naeini et al., 2015) is a metric that estimates model miscalibration by binning the assigned probability scores and comparing them to average accuracies inside these bins. Assuming Bm denotes the m-th bin and M is overall number of bins, the ECE is defined as follows:
ECE = M∑
m=1
|Bm| n |acc(Bm)− conf(Bm)| , (1)
where acc(B) = |B|−1 ∑ i∈B I[arg maxc p̂i,c = y∗i ] and conf(B) = |B|−1 ∑ i∈B p̂i,y∗i .
A recent line of works on measuring calibration in deep learning (Vaicenavicius et al., 2019; Kumar et al., 2019; Nixon et al., 2019) outlines several problems of the ECE score. Firstly, ECE is a biased estimate of the true calibration. Secondly, ECE-like scores cannot be optimized directly since they are minimized by a model with constant uniform predictions, making the infinite temperature T = +∞ its global optimum. Thirdly, ECE only estimates miscalibration in terms of the maximum assigned probability whereas practical applications may require the full predicted probability vector to be calibrated. Finally, biases of ECE on different models may not be equal, rendering the miscalibration estimates incompatible. Similar concerns are also discussed by Ding et al. (2019).
Thresholded adaptive calibration error (TACE) was proposed as a step towards solving some of these problems (Nixon et al., 2019). TACE disregards all predicted probabilities that are less than a certain threshold (hence thresholded), chooses the bin locations adaptively so that each bin has the same number of objects (hence adaptive), and estimates miscalibration of probabilties across all classes in the prediction (not just the top-1 predicted class as in ECE). Assuming that BTAm denotes the m-th thresholded adaptive bin and M is the overall number of bins, TACE is defined as follows:
TACE = 1
CM C∑ c=1 M∑ m=1 |BTAm | n ∣∣objs(BTAm , c)− conf(BTAm , c)∣∣ , (2) where objs(BTA, c) = |BTA|−1 ∑ i∈BTA I[y∗i = c] and conf(BTA, c) = |BTA|−1 ∑ i∈BTA p̂i,c.
Although TACE does solve several problems of ECE and is useful for measuring calibration of a specific model, it still cannot be used as a reliable criterion for comparing different models. Theory suggests that it is still a biased estimate of true calibration with different bias for each model (Vaicenavicius et al., 2019). In practice, we find that TACE is sensitive to its two parameters, the number of bins and the threshold, and does not provide a consistent ranking of different models, as shown in Figure 2.

heading: 3.5 CALIBRATED LOG-LIKELIHOOD AND TEST-TIME CROSS-VALIDATION
text: There are two common ways to perform temperature scaling using a validation set when training on datasets that only feature public training and test sets (e.g. CIFARs). The public training set might
be divided into a smaller training set and validation set, or the public test set can be split into test and validation parts (Guo et al., 2017; Nixon et al., 2019). The problem with the first method is that the resulting models cannot be directly compared with all the other models that have been trained on the full training set. The second approach, however, provides an unbiased estimate of metrics such as log-likelihood and Brier score but introduces more variance.
In order to reduce the variance of the second approach, we perform a “test-time cross-validation”. We randomly divide the test set into two equal parts, then compute metrics for each half of the test set using the temperature optimized on another half. We repeat this procedure five times and average the results across different random partitions to reduce the variance of the computed metrics.

heading: 4 A STUDY OF ENSEMBLING & DEEP ENSEMBLE EQUIVALENT
text: Ensembles of deep neural networks have become a de-facto standard for uncertainty estimation and improving the quality of deep learning models (Hansen & Salamon, 1990; Krizhevsky et al., 2009; Lakshminarayanan et al., 2017). There are two main directions of training ensembles of DNNs: training stochastic computation graphs and obtaining separate snapshots of neural network parameters.
Methods based on the paradigm of stochastic computation graphs introduce some kind of random noise over the weights or activations of deep learning models. When the model is trained, each sample of the noise corresponds to a member of the ensemble. During test time, the predictions are averaged across the noise samples. These methods include (test-time) data augmentation, dropout (Srivastava et al., 2014; Gal & Ghahramani, 2016), variational inference (Blundell et al., 2015; Kingma et al., 2015; Louizos & Welling, 2017), batch normalization (Ioffe & Szegedy, 2015; Teye et al., 2018; Atanov et al., 2019), Laplace approximation (Ritter et al., 2018) and many more.
Snapshot-based methods aim to obtain sets of weights for deep learning models and then to average the predictions across these weights. The weights can be trained independently (e.g. deep ensembles (Lakshminarayanan et al., 2017)), collected on different stages of a training trajectory (e.g. snapshot ensembles (Huang et al., 2017) and fast geometric ensembles (Garipov et al., 2018)), or obtained from a sampling process (e.g. MCMC-based methods (Welling & Teh, 2011; Zhang et al., 2019)). These two paradigms can be combined. Some works suggest construction of ensembles of stochastic computation graphs (Tomczak et al., 2018), while others make use of the collected snapshots to construct a stochastic computation graph (Wang et al., 2018; Maddox et al., 2019).
In this paper we consider the following ensembling techniques: deep ensembles (Lakshminarayanan et al., 2017), snapshot ensembles (SSE by Huang et al. (2017)), fast geometric ensembling (FGE by Garipov et al. (2018)), SWA-Gaussian (SWAG by Maddox et al. (2019)), cyclical SGLD (cSGLD by Zhang et al. (2019)), variational inference (VI by Blundell et al. (2015)), K-FAC Laplace approximation (Ritter et al., 2018), dropout (Srivastava et al., 2014) and test-time data augmentation (Krizhevsky et al., 2009). These techniques were chosen to cover a diverse set of approaches keeping their predictive performance in mind.
All these techniques can be summarized as distributions qm(ω) over parameters ω of computation graphs, where m stands for the technique. During testing, one can average the predictions across parameters ω ∼ qm(ω) to approximate the predictive distribution
p̂(yi |xi) ≈ ∫ p(yi |xi, ω)qm(ω) dω ' 1
K K∑ k=1 p(yi |xi, ωk), ωk ∼ qm(ω) (3)
For example, a deep ensemble of S networks can be represented in this form as a mixture of S Dirac’s deltas qDE(ω) = 1S ∑S s=1 δ(ω − ωs), centered at independently trained snapshots ωs. Similarly, a Bayesian neural network with a fully-factorized Gaussian approximate posterior distribution over the weight matrices and convolutional kernels ω is represented as qVI(ω) = N (ω |µ,diag(σ2)), µ and σ2 being the optimal variational means and variances respectively.
If one considers data augmentation as a part of the computational graph, it can be parameterized by the coordinates of the random crop and the flag for whether to flip the image horizontally or not. Sampling from the corresponding qaug(ω) would generate different ways to augment the data during inference. However, as data augmentation is present by default during the training of all othe
mentioned ensembling techniques, it is suitable to study it in combination with these methods and not as a separate ensembling technique. We perform such an evaluation in Section 4.3.
Typically, the approximation (equation 3) requires K independent forward passes through a neural network, making the test-time budget directly comparable across all methods.

heading: 4.1 DEEP ENSEMBLE EQUIVALENT
text: Most ensembling techniques under consideration are either bounded to a single mode, or provide positively correlated samples. Deep ensemble, on the other hand, is a simple technique that provides independent samples from different modes of the loss landscape, which, intuitively, should result in a better ensemble. Therefore deep ensembles can be considered a strong baseline for the performance of other ensembling techniques given a fixed test-time computation budget.
Comparing the performance of ensembling techniques is, however, a challenging problem. Different models on different datasets achieve different values of metrics; their dependence on the number of samples is non-trivial, and varies depending on a specific model and dataset. Values of the metrics are thus lacking in interpretability as the gain in performance has to be compared against a modeland dataset-specific baseline.
Aiming to introduce perspective and interpretability in our study, we introduce the deep ensemble equivalent score that employs deep ensembles to measure the performance of other ensembling techniques. Specifically, the deep ensemble equivalent score answers the following question:
What size of deep ensemble yields the same performance as a particular ensembling method?
Following the insights from the previous sections, we base the deep ensemble equivalent on the calibrated log-likelihood (CLL). Formally speaking, we define the deep ensemble equivalent (DEE) for an ensembling method m and its upper and lower bounds as follows:
DEEm(k) = min { l ∈ R, l ≥ 1 ∣∣∣CLLmeanDE (l) ≥ CLLmeanm (k)}, (4) DEE upper lower m (k) = min { l ∈ R, l ≥ 1 ∣∣∣CLLmeanDE (l)∓ CLLstdDE(l) ≥ CLLmeanm (k)}, (5)
where CLLmean/stdm (l) are the mean and the standard deviation of the calibrated log-likelihood achieved by an ensembling method m with l samples. We compute CLLmeanDE (l) and CLL std DE(l) for natural numbers l ∈ N>0 and use linear interpolation to define them for real values l ≥ 1. In the following plots we report DEEm(k) for different methods m with different numbers of samples k, and shade the area between the respective lower and upper bounds DEElowerm (k) and DEE upper m (k).

heading: 4.2 EXPERIMENTS
text: We compute the deep ensemble equivalent (DEE) of various ensembling techniques for four popular deep architectures: VGG16 (Simonyan & Zisserman, 2014), PreResNet110/164 (He et al., 2016), and WideResNet28x10 (Zagoruyko & Komodakis, 2016) on CIFAR-10/100 datasets (Krizhevsky et al., 2009), and ResNet50 (He et al., 2016) on ImageNet dataset (Russakovsky et al., 2015). We use PyTorch (Paszke et al., 2017) for implementation of these models, building upon available public implementations. Our implementation closely matches the quality of methods that has been reported in original works. Technical details on training, hyperparameters and implementations can be found in Appendix B. The source code and all computed metrics are available on GitHub1.
As one can see on Figure 3, ensembling methods clearly fall into three categories. SSE and cSGLD outperform all other techniques except deep ensembles and enjoy a near-linear scaling of DEE with the number of samples on CIFAR datasets. The investigation of weight-space trajectories of cSGLD and SSE (Huang et al., 2017; Zhang et al., 2019) suggests that these methods can efficiently explore different modes of the loss landscape. In terms of the deep ensemble equivalent, these methods do not saturate unlike other methods that are bound to a single mode. We found SSE to still saturate on ImageNet. This is likely due to suboptimal hyperparameters of the cyclic learning rate schedule. More verbose results are presented in Figures 11–13 and in Table 5 and Table 8 in Appendix C.
In our experiments SSE typically outperforms cSGLD. This is mostly due to the fact that SSE has a much larger training budget. The cycle lengths and learning rates of SSE and cSGLD are comparable, however, SSE collects one snapshot per cycle while cSGLD collects three snapshots. This makes samples from SSE less correlated with each other while increasing the training budget threefold. Both SSE and cSGLD can be adjusted to obtain a different trade-off between the training budget and the DEE-to-samples ratio. We reused the schedules provided in the original papers (Huang et al., 2017; Zhang et al., 2019).
1Source code: https://github.com/bayesgroup/pytorch-ensembles
Being more “local” methods, FGE and SWAG perform worse than SSE and cSGLD, but still significantly outperform “single-snapshot” methods like dropout, K-FAC Laplace approximation and variational inference. We hypothesize that by covering a single mode with a set of snapshots, FGE and SWAG provide a better fit for the local geometry than models trained as stochastic computation graphs. This implies that the performance of FGE and SWAG should be achievable by singlesnapshot methods. However, one might need more elaborate posterior approximations and better inference techniques in order to match the performance of FGE and SWAG by training a stochastic computation graph end-to-end (as opposed to SWAG that constructs a stochastic computation graph post-hoc).
The deep ensemble equivalent curves allow us to notice the common behaviour of different methods, e.g. the relation between deep ensembles, snapshot methods, advanced local methods and singlesnapshot local methods. They also allow us to notice inconsistencies that may indicate a suboptimal choice of hyperparameters. For example, we find that SSE on ImageNet quickly saturates, unlike SSE on CIFAR datasets (Figure 3). This may indicate that the hyperparameters used on ImageNet are not good enough for efficient coverage of different modes of the loss landscape. We also find that SSE on WideResNet on CIFAR-10 achieves a DEE score of 100 on approx. 70 samples (Figure 12). This may indicate that the members of the deep ensemble for this dataset-architecture pair are underfitted and may benefit from longer training or a different learning rate schedule. Such inconsistencies might be more difficult to spot using plain calibrated log-likelihood plots.

heading: 4.3 TEST-TIME DATA AUGMENTATION IMPROVES ENSEMBLES FOR FREE
text: Data augmentation is a time-honored technique that is widely used in deep learning, and is a crucial component for training modern DNNs. Test-time data augmentation has been used for a long time to improve the performance of convolutional networks. For example, multi-crop evaluation has been a standard procedure for the ImageNet challenge (Simonyan & Zisserman, 2014; Szegedy et al., 2015; He et al., 2016). It is, however, often overlooked in the literature on ensembling techniques in deep learning. In this section, we study the effect of test-time data augmentation on the aforementioned ensembling techniques. To keep the test-time computation budget the same, we sample one random augmentation for each member of an ensemble. Figure 5 reports the calibrated log-likelihood on combination of ensembles and test-time data augmentation for ImageNet. Other metrics and results on CIFAR-10/100 datasets are reported in Appendix C. We have used the standard data augmen-
tation: random horizontal flips and random padded crops for CIFAR-10/100 datasets, and random horizontal flips and random resized crops for ImageNet (see more details in Appendix B).
Test-time data augmentation (Figure 4) consistently improves most ensembling methods, especially on ImageNet, where we see a clear improvement across all methods (Figure 5 and Table 7). The performance gain for powerful ensembles (deep ensembles, SSE and cSGLD) on CIFAR datasets is not as dramatic (Figures 14–15 and Table 4). This is likely due to the fact that CIFAR images are small, making data augmentation limited, whereas images from ImageNet allow for a large number of diverse samples of augmented images. On the other hand, while the performance of “single-snapshot” methods (e.g. variational inference, K-FAC Laplace and dropout) is improved significantly, they perform approximately as good as an augmented version of a single model across all datasets.w
Interestingly, test-time data augmentation on ImageNet improves accuracy but decreases the uncalibrated log-likelihood of deep ensembles (Table 7 in Appendix C). Test-time data augmentation breaks the nearly optimal temperature of deep ensembles and requires temperature scaling to reveal the actual performance of the method, as discussed in Section 3.1. The experiment demonstrates that ensembles may be highly miscalibrated by default while still providing superior predictive performance after calibration.
We would like to note that test-time data augmentation does not always break the calibration of an ensemble, and, on the contrary, test-time data augmentation often improves the calibration of an ensemble. In our experiments, decalibration was caused by the extreme magnitude of a random crop, that is conventionally used for ImageNet augmentation. Using less extreme magnitude of the random crop fixes decalibration, that makes test-time data augmentation a more practical method that provides out-of-the-box calibration. Although, as we demonstrated earlier, there is no guarantee that any ensemble is calibrated out-of-the-box. If we are willing to apply post-hoc calibration, the final performance can be much better with more severe augmentations.

heading: 5 DISCUSSION & CONCLUSION
text: We have explored the field of in-domain uncertainty estimation and performed an extensive evaluation of modern ensembling techniques. Our main findings can be summarized as follows:
• Temperature scaling is a must even for ensembles. While ensembles generally have better calibration out-of-the-box, they are not calibrated perfectly and can benefit from the procedure. A comparison of log-likelihoods of different ensembling methods without temperature scaling might not provide a fair ranking, especially if some models happen to be miscalibrated.
• Many common metrics for measuring in-domain uncertainty are either unreliable (ECE and analogues) or cannot be used to compare different methods (AUC-ROC, AUC-PR for misclassification detection; accuracy-confidence curves). In order to perform a fair comparison of different methods, one needs to be cautious of these pitfalls.
• Many popular ensembling techniques require dozens of samples for test-time averaging, yet are essentially equivalent to a handful of independently trained models. Deep ensembles dominate other methods given a fixed test-time budget. The results indicate, in particular, that exploration of different modes in the loss landscape is crucial for good predictive performance.
• Methods that are stuck in a single mode are unable to compete with methods that are designed to explore different modes of the loss landscape. Would more elaborate posterior approximations and better inference techniques shorten this gap?
• Test-time data augmentation is a surprisingly strong baseline for in-domain uncertainty estimation. It can significantly improve other methods without increasing training time or model size since data augmentation is usually already present during training.
Our takeaways are aligned with the take-home messages of Ovadia et al. (2019) that relate to indomain uncertainty estimation. We also observe a stable ordering of different methods in our experiments, and observe that deep ensembles with few members outperform methods based on stochastic computation graphs.
A large number of unreliable metrics inhibits a fair comparison of different methods. Because of this, we urge the community to aim for more reliable benchmarks in the numerous setups of uncertainty estimation.

heading: ACKNOWLEDGMENTS
text: Dmitry Vetrov and Dmitry Molchanov were supported by the Russian Science Foundation grant no. 19-71-30020. This research was supported in part through computational resources of HPC facilities at NRU HSE.

heading: B EXPERIMENTAL DETAILS
text: Implementations of deep ensembles, SWAG, FGE and K-FAC Laplace are heavily based on the original PyTorch implementations of stochastic weight averaging (SWA) 2 and SWAG 3. Implementations of cyclical MCMC and snapshot ensembles are based on the original implementation of cyclical MCMC 4. We hypothesize that the optimal hyperparameters of ensembling methods may vary widely depending on the computational budget and the number of samples in the ensemble. Searching for the optimal values for each configuration is outside the scope of this paper so we stick to the originally proposed hyperparameters whenever possible.
Implied probabilistic model Conventional neural networks for classification are usually trained using the average cross-entropy loss function with weight decay regularization hidden inside an optimizer in a deep learning framework like PyTorch. The underlying optimization problem can be written as follows:
L(w) = − 1 N N∑ i=1 log p̂(y∗i |xi, w) + λ 2 ‖w‖2 → min w , (6)
where {(xi, y∗i )}Ni=1 is the training dataset of N objects xi with corresponding labels y∗i , λ is the weight decay scale and p̂(j |xi, w) denotes the probability that a neural network with parameters w assigns to class j when evaluated on object xi.
The cross-entropy loss defines a likelihood function p(y∗ |x,w) and weight decay regularization, or L2 regularization corresponding to a certain Gaussian prior distribution p(w). The whole optimization objective then corresponds to the maximum a posteriori inference in the following probabilistic model:
p(y∗, w |x) = p(y∗ |x,w)p(w), (7)
log p(y∗ |x,w) = log N∏ i=1 p(y∗i |xi, w) = N∑ i=1 log p̂(y∗i |xi, w), (8)
log p(w) = −Nλ
2 ‖w‖2 + const ⇐⇒ p(w) = N
( w ∣∣ 0, (Nλ)−1I) (9)
In order to make the results comparable across all ensembling techniques, we used the same probabilistic model for all methods, choosing fixed weight decay parameters for each architecture. We used the softmax-based likelihood for all models. We also use the fully-factorized zero-mean Gaussian prior distribution with variances σ2 = (Nλ)−1 where the number of objects N and the weight decay scale λ are dictated by the particular datasets and neural architectures as defined in the following paragraph.
Conventional networks To train a single network on CIFAR-10/100, we used SGD with batch size of 128, momentum 0.9 and model-specific parameters, i.e. the initial learning rate (lrinit), the weight decay coefficient (wd), and the number of optimization epochs (epoch). Specific hyperparameters are shown in Table 1. The models were trained with a unified learning rate scheduler that is shown in equation 10. All models have been trained using data augmentation that consists of horizontal flips and a random crop of 32 pixels with a padding of 4 pixels5. The standard data normalization has also been applied. Weight decays, initial learning rates, and the learning rate scheduler were taken from (Garipov et al., 2018) paper. Compared with hyperparameters of (Garipov et al., 2018), we increased the number of optimization epochs since we found that all models were underfitted. While the original WideResNet28x10 network includes a number of dropout layers with p = 0.3 and is trained for 200 epochs, we find that the WideResNet28x10 underfits in this setting and requires a longer training. Therefore, we used p = 0 which reduces training time while bearing
2https://github.com/timgaripov/swa 3https://github.com/wjmaddox/swa_gaussian 4https://github.com/ruqizhang/csgmcmc/tree/master/experiments 5Compose([RandomHorizontalFlip(), RandomCrop(32, padding=4)])
no significant effect on final model performance in our experiments.
lr(i) =  lrinit, i ∈ [0, 0.5 · epochs] lrinit · (1.0− 0.99 ∗ (i/epochs− 0.5)/0.4), i ∈ [0.5 · epochs, 0.9 · epochs] lrinit · 0.01, otherwise
(10)
On ImageNet dataset we used ResNet50 with default hyperparameters taken from PyTorch examples 6. Specifically, we used SGD with momentum 0.9, batch size of 256, initial learning rate 0.1, weight decay 1e − 4. Training included data augmentation7 (scaling, random crops of size 224 × 224, horizontal flips), normalization and learning rate scheduler lr = lrinit · 0.1epoch//30 where // denotes integer division. We only deviated from standard parameters by increasing the number of training epochs from 90 to 130. Or models achieve top-1 error of 23.81± 0.15 that closely matches the accuracy of ResNet50 provided by PyTorch which is 23.85 8. Training of one model on a single NVIDIA Tesla V100 GPU takes approximately 5.5 days.
Deep ensembles Deep ensembles (Lakshminarayanan et al., 2017) average the predictions across networks trained independently starting from different initializations. To obtain a deep ensemble we repeat the described procedure of training standard networks 128 times for all architectures on CIFAR-10 and CIFAR-100 datasets (1024 networks over all) and 50 times for ImageNet dataset. Every member of deep ensembles was trained with exactly the same hyperparameters as conventional models of the same architecture.
Dropout Binary dropout (or MC dropout) (Srivastava et al., 2014; Gal & Ghahramani, 2016) is one of the most widely known ensembling techniques. It involves putting a multiplicative Bernoulli noise with a parameter p over the activations of either a fully connected layer or a convolutional layer, averaging predictions of the network w.r.t. noise at test-time. Dropout layers were applied to VGG and WideResNet networks on CIFAR-10 and CIFAR-100 datasets. Dropout for VGG was applied to fully connected layers with p = 0.5. Two dropout layers were applied: one before the first fully connected layer and one before the second one. While the original version of VGG for CIFARs (Zagoruyko, 2015) exploits more dropout layers, we observed that any additional dropout layer deteriorates the performance of the model in ether deterministic or stochastic mode. Dropout for WideResNet was applied in accordance with the original paper (Zagoruyko & Komodakis, 2016) with p = 0.3. Dropout usually increases the time needed to achieve convergence. Because of this, WideResNet networks with dropout were trained for 400 epochs instead of 300 epochs for deterministic case, and VGG networks have always been trained with dropout. All the other hyperparameters were the same as in the case of conventional models.
Variational Inference Variational Inference (VI) approximates the true posterior distribution over weights p(w |Data) with a tractable variational approximation qθ(w) by maximizing a so-called variational lower bound L (eq. 11) w.r.t. the parameters of variational approximation θ. We used a fully-factorized Gaussian approximation q(w) and Gaussian prior distribution p(w).
L(θ) = Eq log p(y∗ |x,w)−KL(qθ(w) || p(w))→ max θ
(11)
q(w) = N (w |µ, diag(σ2)) p(w) = N(w | 0, diag(σ2p)), where σ2p = (N · wd)−1 (12)
6https://github.com/pytorch/examples/tree/ee964a2/imagenet 7Compose([RandomResizedCrop(224), RandomHorizontalFlip()]) 8https://pytorch.org/docs/stable/torchvision/models.html
In the case of such a prior, the probabilistic model remains consistent with the conventional training which corresponds to MAP inference in the same probabilistic model. We used variational inference for both convolutional and fully-connected layers where variances of the weights were parameterized by log σ. For fully-connected layers we applied the local reparameterization trick (LRT, (Kingma et al., 2015)).
While variational inference provides a theoretically grounded way to approximate the true posterior, it tends to underfit deep learning models in practice (Kingma et al., 2015). The following tricks are applied to deal with it: pre-training (Molchanov et al., 2017) or equivalently annealing of β (Sønderby et al., 2016), and scaling β down (Kingma et al., 2015; Ullrich et al., 2017).
During pre-training we initialize µ with a snapshot of weights of a pre-trained conventional model, and initialize log σ with a model-specific constant log σinit. The KL-divergence – except for the term corresponding to the weight decay – is scaled with a model-specific parameter β. The weight decay term is implemented as a part of the optimizer. We used a fact that the KL-divergence between two Gaussian distributions can be rewritten as two terms, one of which is equivalent to the weight decay regularization.
On CIFAR-10 and CIFAR-100 we used β equal to 1e-4 for VGG, ResNet100 and ResNet164 networks, and β equal to 1e-5 for WideResNet. Log-variance log σinit was initialized with −5 for all models. Parameters µ were optimized with SGD in the same manner as in the case of conventional networks except that the initial learning rate lrinit was set to 1e-3. We used a separate Adam optimizer with a constant learning rate of 1e-3 to optimize log-variances of the weights log σ. Pretraining was done for 300 epochs, and after that the remaining part of training was done for 100 epochs. On ImageNet we used β = 1e-3, lrinit = 0.01, log σinit = −6, and trained the model for 45 epochs after pre-training.
K-FAC Laplace The Laplace approximation uses the curvature information of the appropriately scaled loss function to construct a Gaussian approximation to the posterior distribution. Ideally, one would use the Hessian of the loss function as a covariance matrix and use the maximum a posteriori estimate wMAP as a mean of the Gaussian approximation:
log p(w |x, y∗) = log p(y∗ |x,w) + log p(w) + const (13) wMAP = arg max
w log p(w |x, y∗); Σ = −∇∇ log p(w |x, y∗) (14)
p(w |x, y∗) ≈ N (w |wMAP ,Σ) (15)
In order to keep the method scalable, we use the Fisher Information Matrix as an approximation to the true Hessian (Martens & Grosse, 2015). For K-FAC Laplace, we use the whole dataset to construct an approximation to the empirical Fisher Information Matrix, and use the π correction to reduce the bias (Ritter et al., 2018; Martens & Grosse, 2015). Following (Ritter et al., 2018), we find the optimal noise scale for K-FAC Laplace on a held-out validation set by averaging across five random initializations. We then reuse this scale for networks trained without a hold-out validation set. We report the optimal values of scales in Table 2. Note that the optimal scale is different depending on whether we use test-time data augmentation or not. Since the data augmentation also introduces some amount of additional noise, the optimal noise scale for K-FAC Laplace with data augmentation is lower.
Snapshot ensembles Snapshot ensembles (SSE) (Huang et al., 2017) is a simple example of an array of methods which collect samples from a training trajectory of a network in weight space to construct an ensemble. Samples are collected in a cyclical manner: during each cycle the learning rate goes from a large value to near-zero and snapshot of weights of the network is taken at the end of the cycle. SSE uses SGD with a cosine learning schedule defined as follows:
α(t) = α0 2
( cos ( π mod (t− 1, dT/Me)
dT/Me
) + 1 ) , (16)
where α0 is the initial learning rate, T is the total number of training iterations and M is the number of cycles.
For all datasets and models hyperparameters from the original SSE paper are reused. For CIFAR10/100 length of the cycle is 40 epochs, maximum learning rate is 0.2, batch size is 64. On ResNet50 and ImageNet length of the cycle is 45 epochs, maximum learning rate is 0.1, batch size is 256.
Cyclical SGLD Cyclical Stochastic Gradient Langevin Dynamics (cSGLD) (Zhang et al., 2019) is a state-of-the-art ensembling method for deep neural networks pertaining to stochastic Markov Chain Monte Carlo family of methods. It bears similarity to SSE, e.g. it employs SGD with a learning rate schedule described with the equation 16 and training is cyclical in the same manner. Its main differences from SSE are the introduction of gradient noise and the capturing of several snapshots per cycle, both of which can aid in sampling from posterior distribution over neural network weights efficiently.
Some parameters from the original paper are reused: length of cycle is 50 epochs, maximum learning rate is 0.5, batch size is 64. Number of epochs with gradient noise per cycle is 3 epochs. This was found to yield much higher predictive performance and better uncertainty estimation compared to the original paper choice of 10 epochs for CIFAR-10 and 3 epochs for CIFAR-100.
Finally, the results of cyclical Stochastic Gradient Hamiltonian Monte Carlo (SGHMC), (Zhang et al., 2019) which reportedly has marginally better performance compared with cyclical SGLD, could not be reproduced with any value of SGD momentum term. Because of this, we only include cyclical SGLD in our benchmark.
FGE Fast Geometric Ensembling (FGE) is an ensembling method that is similar to SSE in that it collects weight samples from a training trajectory to construct an ensemble. Its main differences from SSE are pretraining, short cycle length and a piecewise-linear learning rate schedule:
α(i) = { (1− 2t(i))α1 + 2t(i)α2 0 < t(i) ≤ 12 (2− 2t(i))α2 + (2t(i)− 1)α1 12 < t(i) ≤ 1 . (17)
Hyperparameters of the original implementation of FGE are reused. Model pretraining is done with SGD for 160 epochs according to the standard learning rate schedule described in equation 10 with maximum learning rates from Table 1. After that, a desired number of FGE cycles is done with one snapshot per cycle collected. For VGG the learning rate is changed with parameters α1 = 1e − 2, α2 = 5e − 4 in a cycle with cycle length of 2 epochs. For other networks the learning rate is changed with parameters α1 = 5e− 2, α2 = 5e− 4 with cycle length of 4 epochs. Batch size is 128.
SWAG SWA-Gaussian (SWAG) (Maddox et al., 2019) is an ensembling method based on fitting a Gaussian distribution to model weights on the SGD training trajectory and sampling from this distribution to construct an ensemble.
Like FGE, SWAG has a pretraining stage which is done according to the standard learning rate schedule described in equation 10 with maximum learning rates from Table 1. After that, training continues with a constant learning rate of 1e-2 for all models except for PreResNet110 and PreResNet164 on CIFAR-100 where it continues with a constant learning rate of 5e-2 in accordance with the original paper. Rank of the empirical covariance matrix which is used for estimation of Gaussian distribution parameters is set to be 20.

heading: C ADDITIONAL EXPERIMENTAL RESULTS
text: Error (%) Negative calibrated log-likelihood
Model Method 1 5 10 100 1 5 10 100
Dropout 5.86±0.09 5.81±0.08 5.82±0.06 5.79±0.07 0.232±0.005 0.225±0.004 0.224±0.004 0.223±0.003 SWA-Gaussian 7.03±0.50 5.66±0.08 5.49±0.12 5.25±0.13 0.230±0.014 0.182±0.003 0.171±0.002 0.160±0.002 Cyclic SGLD 7.37±0.16 6.56±0.09 5.71±0.06 4.84±0.04 0.234±0.004 0.196±0.004 0.176±0.003 0.147±0.003 Fast Geometric Ens. 6.52±0.16 5.95±0.16 5.69±0.16 5.10±0.13 0.213±0.005 0.187±0.003 0.178±0.003 0.155±0.004
VGG16 Deep Ensembles 5.95±0.14 4.79±0.11 4.57±0.07 4.39±NA 0.226±0.001 0.158±0.002 0.148±0.001 0.134±NA CIFAR-10 Single model 5.83±0.11 5.83±0.11 5.83±0.11 5.83±0.11 0.223±0.002 0.223±0.002 0.223±0.002 0.223±0.002
Variational Inf. (FFG) 6.57±0.09 5.63±0.13 5.50±0.10 5.46±0.03 0.239±0.002 0.192±0.002 0.184±0.002 0.175±0.001 KFAC-Laplace 6.00±0.13 5.82±0.12 5.82±0.19 5.80±0.19 0.210±0.005 0.203±0.007 0.201±0.007 0.200±0.008 Snapshot Ensembles 7.76±0.22 5.52±0.13 5.00±0.10 4.54±0.05 0.247±0.005 0.176±0.001 0.160±0.001 0.137±0.001
SWA-Gaussian 5.77±0.45 4.56±0.17 4.46±0.12 4.34±0.13 0.178±0.009 0.143±0.004 0.139±0.003 0.131±0.003 Cyclic SGLD 6.18±0.20 5.32±0.15 4.55±0.13 3.83±0.02 0.185±0.006 0.156±0.005 0.138±0.002 0.115±0.001 Fast Geometric Ens. 5.52±0.09 4.83±0.08 4.73±0.10 4.28±0.05 0.163±0.002 0.141±0.003 0.137±0.003 0.126±0.002
ResNet110 Deep Ensembles 4.66±0.11 3.77±0.11 3.63±0.07 3.53±NA 0.148±0.004 0.117±0.002 0.112±0.002 0.106±NA CIFAR-10 Single model 4.69±0.11 4.69±0.11 4.69±0.11 4.69±0.11 0.150±0.002 0.150±0.002 0.150±0.003 0.150±0.002
Variational Inf. (FFG) 5.57±0.26 4.91±0.15 4.72±0.13 4.60±0.03 0.178±0.003 0.149±0.001 0.144±0.001 0.140±0.000 KFAC-Laplace 5.81±0.39 5.14±0.15 4.90±0.14 4.78±0.08 0.187±0.014 0.160±0.007 0.153±0.005 0.147±0.003 Snapshot Ensembles 8.41±0.27 4.85±0.11 4.16±0.16 3.52±0.10 0.252±0.006 0.153±0.002 0.132±0.002 0.107±0.001
SWA-Gaussian 5.41±0.71 4.21±0.19 4.21±0.23 4.02±0.14 0.171±0.028 0.130±0.004 0.128±0.004 0.121±0.002 Cyclic SGLD 5.80±0.21 4.97±0.12 4.30±0.08 3.66±0.06 0.178±0.004 0.149±0.004 0.131±0.003 0.110±0.001 Fast Geometric Ens. 5.22±0.07 4.49±0.06 4.36±0.07 4.09±0.12 0.157±0.003 0.134±0.002 0.130±0.001 0.119±0.002
ResNet164 Deep Ensembles 4.53±0.11 3.51±0.09 3.50±0.06 3.34±NA 0.147±0.002 0.113±0.001 0.107±0.001 0.100±NA CIFAR-10 Single model 4.52±0.11 4.52±0.11 4.52±0.11 4.52±0.11 0.144±0.002 0.144±0.003 0.144±0.002 0.144±0.003
Variational Inf. (FFG) 5.62±0.14 4.78±0.05 4.66±0.05 4.55±0.08 0.183±0.004 0.151±0.001 0.146±0.001 0.141±0.001 KFAC-Laplace 5.23±0.29 4.77±0.23 4.65±0.17 4.60±0.09 0.168±0.008 0.151±0.007 0.146±0.005 0.142±0.004 Snapshot Ensembles 8.06±0.10 4.50±0.04 3.89±0.09 3.50±0.05 0.241±0.004 0.144±0.003 0.124±0.002 0.104±0.001
Dropout 3.88±0.12 3.70±0.18 3.63±0.19 3.64±0.17 0.130±0.002 0.120±0.002 0.119±0.001 0.117±0.002 SWA-Gaussian 4.98±1.17 3.53±0.09 3.34±0.14 3.28±0.10 0.157±0.036 0.111±0.004 0.105±0.003 0.101±0.002 Cyclic SGLD 4.78±0.16 4.09±0.11 3.63±0.13 3.19±0.04 0.155±0.003 0.128±0.002 0.114±0.001 0.099±0.002 Fast Geometric Ens. 4.86±0.17 3.95±0.07 3.77±0.10 3.34±0.06 0.148±0.003 0.120±0.002 0.113±0.002 0.102±0.001
WideResNet Deep Ensembles 3.65±0.02 3.11±0.10 3.01±0.06 2.83±NA 0.123±0.002 0.097±0.001 0.095±0.001 0.090±NA CIFAR-10 Single model 3.70±0.15 3.70±0.15 3.70±0.15 3.70±0.15 0.124±0.005 0.124±0.005 0.125±0.005 0.124±0.005
Variational Inf. (FFG) 5.61±0.04 4.15±0.15 3.94±0.10 3.64±0.07 0.189±0.002 0.134±0.002 0.127±0.002 0.117±0.001 KFAC-Laplace 4.03±0.19 3.90±0.15 3.88±0.22 3.83±0.16 0.134±0.004 0.124±0.004 0.122±0.005 0.120±0.003 Snapshot Ensembles 5.56±0.15 3.68±0.09 3.33±0.10 2.89±0.07 0.179±0.005 0.119±0.001 0.105±0.001 0.090±0.001
Dropout 26.10±0.20 25.68±0.18 25.66±0.14 25.60±0.17 1.176±0.008 1.111±0.008 1.098±0.009 1.084±0.009 SWA-Gaussian 27.74±1.87 24.53±0.09 23.64±0.28 22.97±0.20 1.109±0.073 0.931±0.007 0.879±0.007 0.826±0.005 Cyclic SGLD 29.75±0.17 26.79±0.19 24.14±0.11 21.15±0.11 1.114±0.003 0.976±0.004 0.881±0.006 0.749±0.004 Fast Geometric Ens. 27.07±0.24 25.35±0.29 24.68±0.40 22.78±0.22 1.057±0.010 0.965±0.003 0.930±0.003 0.827±0.004
VGG16 Deep Ensembles 25.72±0.17 21.60±0.13 20.79±0.16 19.88±NA 1.092±0.004 0.840±0.005 0.794±0.002 0.723±NA CIFAR-100 Single model 25.44±0.29 25.44±0.29 25.44±0.29 25.44±0.29 1.087±0.006 1.087±0.006 1.087±0.006 1.087±0.006
Variational Inf. (FFG) 27.24±0.09 25.24±0.11 24.85±0.05 24.56±0.07 1.154±0.004 1.001±0.002 0.973±0.002 0.939±0.001 KFAC-Laplace 27.11±0.59 25.98±0.21 25.84±0.38 25.70±0.38 1.174±0.037 1.089±0.007 1.069±0.005 1.050±0.008 Snapshot Ensembles 31.19±0.33 23.87±0.18 22.31±0.31 21.03±0.10 1.170±0.012 0.899±0.004 0.834±0.005 0.751±0.003
SWA-Gaussian 27.75±0.76 22.31±0.22 21.52±0.30 20.69±0.19 0.960±0.033 0.781±0.011 0.745±0.010 0.701±0.008 Cyclic SGLD 25.73±0.14 23.30±0.19 21.20±0.21 18.07±0.16 0.914±0.006 0.818±0.004 0.753±0.002 0.630±0.002 Fast Geometric Ens. 22.84±0.16 21.22±0.20 20.79±0.23 19.64±0.15 0.798±0.006 0.729±0.003 0.713±0.002 0.679±0.002
ResNet110 Deep Ensembles 22.55±0.28 18.30±0.22 17.59±0.21 16.97±NA 0.847±0.007 0.675±0.001 0.638±0.001 0.594±NA CIFAR-100 Single model 22.66±0.31 22.66±0.31 22.66±0.31 22.66±0.31 0.848±0.014 0.848±0.015 0.848±0.014 0.848±0.015
Variational Inf. (FFG) 24.27±0.26 22.41±0.13 22.14±0.12 21.86±0.07 0.924±0.007 0.829±0.003 0.813±0.001 0.795±0.001 KFAC-Laplace 24.88±0.97 22.87±0.44 22.41±0.26 22.14±0.29 0.948±0.036 0.858±0.014 0.836±0.010 0.812±0.010 Snapshot Ensembles 30.30±0.40 22.83±0.23 21.13±0.14 18.48±0.25 1.069±0.006 0.820±0.003 0.761±0.002 0.662±0.002
SWA-Gaussian 24.38±0.93 20.62±0.18 20.08±0.19 19.48±0.19 0.844±0.042 0.719±0.006 0.700±0.006 0.667±0.004 Cyclic SGLD 24.87±0.39 22.37±0.27 20.23±0.22 17.13±0.18 0.888±0.008 0.790±0.009 0.722±0.009 0.606±0.005 Fast Geometric Ens. 21.92±0.15 20.10±0.22 19.87±0.25 18.73±0.25 0.765±0.003 0.699±0.004 0.686±0.004 0.650±0.003
ResNet164 Deep Ensembles 21.41±0.25 17.53±0.17 16.90±0.15 16.50±NA 0.819±0.008 0.647±0.003 0.615±0.002 0.574±NA CIFAR-100 Single model 21.39±0.40 21.39±0.40 21.39±0.40 21.39±0.40 0.817±0.014 0.817±0.014 0.817±0.014 0.817±0.014
Variational Inf. (FFG) 23.47±0.26 21.35±0.11 21.10±0.16 20.82±0.04 0.910±0.001 0.801±0.002 0.782±0.002 0.762±0.000 KFAC-Laplace 23.44±0.45 21.77±0.20 21.29±0.23 21.03±0.38 0.902±0.019 0.813±0.006 0.792±0.005 0.772±0.007 Snapshot Ensembles 29.48±0.19 21.92±0.18 20.27±0.23 17.68±0.07 1.045±0.005 0.789±0.005 0.729±0.004 0.634±0.003
Dropout 20.19±0.11 19.41±0.17 19.36±0.12 19.22±0.15 0.823±0.008 0.768±0.005 0.760±0.006 0.751±0.005 SWA-Gaussian 20.45±0.73 17.57±0.17 17.21±0.22 17.08±0.19 0.794±0.025 0.653±0.004 0.634±0.005 0.614±0.005 Cyclic SGLD 21.42±0.32 19.42±0.28 17.88±0.16 16.29±0.10 0.813±0.010 0.713±0.009 0.654±0.005 0.583±0.004 Fast Geometric Ens. 21.48±0.31 18.54±0.16 18.00±0.19 17.12±0.16 0.770±0.007 0.652±0.006 0.630±0.006 0.596±0.003
WideResNet Deep Ensembles 19.38±0.20 16.55±0.08 16.17±0.15 15.77±NA 0.797±0.007 0.623±0.003 0.595±0.003 0.571±NA CIFAR-100 Single model 19.31±0.24 19.31±0.24 19.31±0.24 19.31±0.24 0.797±0.010 0.797±0.010 0.797±0.010 0.797±0.010
Error (%) Negative calibrated log-likelihood
Model Method 1 5 10 50 1 5 10 50
Fast Geometric Ens. 23.71±0.00 23.61±0.00 23.56±0.00 23.28±0.00 0.929±0.000 0.921±0.000 0.916±0.000 0.904±0.000 Deep Ensembles 23.79±0.14 21.19±0.14 20.90±0.08 20.63±NA 0.935±0.007 0.823±0.002 0.805±0.000 0.788±NA
ResNet50 Single model 23.86±0.20 23.86±0.20 23.86±0.20 23.86±0.20 0.938±0.006 0.938±0.006 0.938±0.006 0.938±0.006 Variational Inf. (FFG) 24.50±0.06 23.82±0.03 23.77±0.04 23.67±0.00 0.957±0.001 0.927±0.000 0.923±0.001 0.920±0.000 KFAC-Laplace 25.01±0.49 24.19±0.29 23.93±0.20 23.86±0.16 0.988±0.022 0.948±0.013 0.939±0.011 0.934±0.008 Snapshot Ensembles 24.92±NA 22.21±NA 21.75±NA 21.48±NA 0.983±NA 0.865±NA 0.843±NA 0.830±NA

Review summaries:
The paper provide an extensive review of current advances in uncertainty estimation in neural networks with the analysis of drawbacks of currently used uncertainty metrics and comparison on scale the recent method to estimate uncertainty. The paper covers a lot of uncertainty metrics and a wide range of methods. The paper focuses on in-domain uncertainty estimation complementing the recent similar review on out-of-domain uncertainty estimation.

This paper mainly concerns the quality of in-domain uncertainty for image classification. After exploring common standards for uncertainty quantification, the authors point out pitfalls of existing metrics by investigating different ensembling techniques and introduce a novel metric called deep ensemble equivalent (DEE) that essentially measures the number of independent models in an ensemble of DNNs. Based on the DEE score, a detailed evaluation of modern DNN ensembles is performed on CIFAR-10/100 and ImageNet datasets.