Pitfalls of In-Domain Uncertainty Estimation and Ensembling in Deep Learning

video / code / blog post 1 INTRODUCTION . Deep neural networks ( DNNs ) have become one of the most popular families of machine learning models . The predictive performance of DNNs for classification is often measured in terms of accuracy . However , DNNs have been shown to yield inaccurate and unreliable probability estimates , or predictive uncertainty ( Guo et al. , 2017 ) . This has brought considerable attention to the problem of uncertainty estimation with deep neural networks . There are many faces to uncertainty estimation . Different desirable uncertainty estimation properties of a model require different settings and metrics to capture them . Out-of-domain uncertainty of the model is measured on the data that does not follow the same distribution as the training dataset ( out-of-domain data ) . Out-of-domain data can include images corrupted with rotations or blurring , adversarial attacks ( Szegedy et al. , 2013 ) or data points from a completely different dataset . The model is expected to be resistant to data corruptions and to be more uncertain on out-of-domain data than on in-domain data . On the contrary , in-domain uncertainty of the model is measured on data taken from the training data distribution , i.e . data from the same domain . In this setting the model is expected to produce reliable probability estimates , e.g . the model shouldn ’ t be too overconfident in its wrong predictions . Pitfalls of metrics We show that many common metrics of in-domain uncertainty estimation ( e.g . log-likelihood , Brier score , calibration metrics , etc . ) are either not comparable across different models or fail to provide a reliable ranking . We address some of the stated pitfalls and point out more reasonable evaluation schemes . For instance , although temperature scaling is not a standard for ensembling techniques , it is a must for a fair evaluation . With this in mind , the ∗Equal contribution §HSE refers to National Research University Higher School of Economics †Skoltech refers to Skolkovo Institute of Science and Technology ‡HSE refers to Samsung-HSE Laboratory , National Research University Higher School of Economics calibrated log-likelihood avoids most of the stated pitfalls and generally is a reasonable metric for in-domain uncertainty estimation task . Pitfalls of ensembles Equipped with the proposed evaluation framework , we are revisiting the evaluation of ensembles of DNNs—one of the major tools for uncertainty estimation . We introduce the deep ensemble equivalent ( DEE ) score that measures the number of independently trained models that , when ensembled , achieve the same performance as the ensembling technique of interest . The DEE score allows us to compare ensembling techniques across different datasets and architectures using a unified scale . Our study shows that most of the popular ensembling techniques require averaging predictions across dozens of samples ( members of an ensemble ) , yet are essentially equivalent to an ensemble of only few independently trained models . Missing part of ensembling In our study , test-time data augmentation ( TTA ) turned out to be a surprisingly strong baseline for uncertainty estimation and a simple way to improve ensembles . Despite being a popular technique in large-scale classification , TTA seems to be overlooked in the community of uncertainty estimation and ensembling . 2 SCOPE OF THE PAPER . We use standard benchmark problems of image classification which comprise a common setting in research on learning ensembles of neural networks . There are other relevant settings where the correctness of probability estimates can be a priority , and ensembling techniques are used to improve it . These settings include , but are not limited to , regression , language modeling ( Gal , 2016 ) , image segmentation ( Gustafsson et al. , 2019 ) , active learning ( Settles , 2012 ) and reinforcement learning ( Buckman et al. , 2018 ; Chua et al. , 2018 ) . We focus on in-domain uncertainty , as opposed to out-of-domain uncertainty . Out-of-domain uncertainty includes detection of inputs that come from a completely different domain or have been corrupted by noise or adversarial attacks . This setting has been thoroughly explored by ( Ovadia et al. , 2019 ) . We only consider methods that are trained on clean data with simple data augmentation . Some other methods use out-of-domain data ( Malinin & Gales , 2018 ) or more elaborate data augmentation , e.g . mixup ( Zhang et al. , 2017 ) or adversarial training ( Lakshminarayanan et al. , 2017 ) to improve accuracy , robustness and uncertainty . We use conventional training procedures . We use the stochastic gradient descent ( SGD ) and use batch normalization ( Ioffe & Szegedy , 2015 ) , both being the de-facto standards in modern deep learning . We refrain from using more elaborate optimization techniques including works on superconvergence ( Smith & Topin , 2019 ) and stochastic weight averaging ( SWA ) ( Izmailov et al. , 2018 ) . These techniques can be used to drastically accelerate training and to improve the predictive performance . Thus , we do not comment on the training time of different ensembling methods since the use of these and other more efficient training techniques would render such a comparison obsolete . A number of related works study ways of approximating and accelerating prediction in ensembles . The distillation mechanism allows to approximate the prediction of an ensemble by a single neural network ( Hinton et al. , 2015 ; Balan et al. , 2015 ; Tran et al. , 2020 ) , whereas fast dropout ( Wang & Manning , 2013 ) and deterministic variational inference ( Wu et al. , 2018 ) allow to approximate the predictive distribution of specific stochastic computation graphs . We measure the raw power of ensembling techniques without these approximations . All of the aforementioned alternative settings are orthogonal to the scope of this paper and are promising points of interest for further research . 3 PITFALLS OF IN-DOMAIN UNCERTAINTY ESTIMATION . No single metric measures all the desirable properties of uncertainty estimates obtained by a model of interest . Because of this , the community is using many different metrics in an attempt to capture the quality of uncertainty estimation , such as the Brier score ( Brier , 1950 ) , log-likelihood ( Quinonero-Candela et al. , 2005 ) , metrics of calibration ( Guo et al. , 2017 ; Nixon et al. , 2019 ) , performance of misclassification detection ( Malinin & Gales , 2018 ) , and threshold–accuracy curves ( Lakshminarayanan et al. , 2017 ) . In the section we highlight the pitfalls of the aforementioned metrics , and demonstrate that these pitfalls can significantly affect evaluation , changing the ranking of the methods . Notation We consider a classification problem with a dataset that consists of N training and n testing pairs ( xi , y∗i ) ∼ p ( x , y ) , where xi is an object and y∗i ∈ { 1 , . . . , C } is a discrete class label . A probabilistic classifier maps an object xi into a predictive distribution p̂ ( y |xi ) . The predictive distribution p̂ ( y |xi ) of a deep neural network is typically defined by the softmax function p̂ ( y |x ) = Softmax ( z ( x ) /T ) , where z ( x ) is a vector of logits and T is a scalar parameter standing for the temperature of the predictive distribution . This scalar parameter is usually set to T = 1 or is tuned on a validation set ( Guo et al. , 2017 ) . The maximum probability maxc p̂ ( y = c |xi ) is called the confidence of a classifier p̂ on an object xi . I [ · ] denotes the indicator function throughout the text . 3.1 LOG-LIKELIHOOD AND BRIER SCORE . The average test log-likelihood LL = 1n ∑n i=1 log p̂ ( y = y ∗ i |xi ) is a popular metric for measuring the quality of in-domain uncertainty of deep learning models . It directly penalizes high probability scores assigned to incorrect labels and low probability scores assigned to the correct labels y∗i . LL is sensitive to the softmax temperature T . The temperature that has been implicitly learned during training can be far from optimal for the test data . However , a nearly optimal temperature can be found post-hoc by maximizing the log-likelihood on validation data . This approach is called temperature scaling or calibration ( Guo et al. , 2017 ) . Despite its simplicity , the temperature scaling results in a notable improvement in the LL . While ensembling techniques tend to have better temperature than single models , the default choice of T = 1 is still suboptimal . Comparing the LL with suboptimal temperatures—that is often the case in practice—can potentially produce an arbitrary ranking of different methods . Comparison of the log-likelihood should only be performed at the optimal temperature . Empirically , we demonstrate that the overall ordering of methods and also the best ensembling method according to the LL can vary depending on temperature T . While this applies to most ensembling techniques ( see Figure 10 ) , this effect is most noticeable on experiments with data augmentation on ImageNet ( Figure 1 ) . We introduce a new metric called the calibrated log-likelihood that is the log-likelihood at the optimal temperature . The calibrated log-likelihood considers a model and a post-training calibration as a unified system , targeting to measure all models in the equal conditions of optimal temperature . That allows to avoid measuring calibration error that can be eliminated by a simple temperature scaling . The metric significantly affects the results of the comparison . For example , in Figure 10 the differences between Bayesian ( VI , K-FAC , SWAG , dropout ) and conventional non-Bayesian networks become much less pronounced , and in most cases making conventional non-Bayesian networks match the performance of Bayesian ones ( VI , K-FAC , Dropout ) on ResNet110 , ResNet164 , and WideResNet . We show how to obtain an unbiased estimate of the calibrated log-likelihood without a held-out validation set in Section 3.5 . LL also demonstrates a high correlation with accuracy ( ρ > 0.86 ) , that in case of calibrated LL becomes even stronger ( ρ > 0.95 ) . That suggests that while ( calibrated ) LL measures the uncertainty of the model , it still has a significant dependence on the accuracy and vice versa . A model with higher accuracy would likely have a higher log-likelihood . See Figure 9 in Appendix C for more details . Brier score BS = 1n 1 C ∑n i=1 ∑C c=1 ( I [ y∗i = c ] − p̂ ( y = c |xi ) ) 2 has also been known for a long time as a metric for verification of predicted probabilities ( Brier , 1950 ) . Similarly to the log-likelihood , the Brier score penalizes low probabilities assigned to correct predictions and high probabilities assigned to wrong ones . It is also sensitive to the temperature of the softmax distribution and behaves similarly to the log-likelihood . While these metrics are not strictly equivalent , they show a high empirical correlation for a wide range of models on CIFAR-10 , CIFAR-100 and ImageNet datasets ( see Figure 8 in Appendix C ) . 3.2 MISCLASSIFICATION DETECTION . Detection of wrong predictions of the model , or misclassifications , is a popular downstream problem relevant to the problem of in-domain uncertainty estimation . Since misclassification detection is essentially a binary classification problem , some papers measure its quality using conventional metrics for binary classification such as AUC-ROC and AUC-PR ( Malinin & Gales , 2018 ; Cui et al. , 2019 ; Możejko et al. , 2018 ) . These papers use an uncertainty criterion like confidence or predictive entropyH [ p̂ ( y |xi ) ] as a prediction score . While these metrics can be used to assess the misclassification detection performance of a single model , they can not be used to directly compare misclassification performance across different models . Correct and incorrect predictions are specific for every model , therefore , every model induces its own binary classification problem . The induced problems can differ significantly , since different models produce different confidences and misclassify different objects . In other words , comparing such metrics implies a comparison of performance of classifiers that solve different classification problems . Such metrics are therefore incomparable . AUCs for misclassification detection can not be directly compared between different models . While comparing AUCs is incorrect in the setting of misclassification detection , it is correct to compare these metrics in many out-of-domain data detection problems . In that case , both objects and targets of the induced binary classification problems remain the same for all models . All outof-domain objects have a positive label and all in-domain objects have a negative label . Note that this condition does not necessarily hold in the problem of detection of adversarial attacks . Different models generally have different inputs after an adversarial attack , so such AUC-based metrics might still be flawed .

Summary: The paper provide an extensive review of current advances in uncertainty estimation in neural networks with the analysis of drawbacks of currently used uncertainty metrics and comparison on scale the recent method to estimate uncertainty. The paper covers a lot of uncertainty metrics and a wide range of methods. The paper focuses on in-domain uncertainty estimation complementing the recent similar review on out-of-domain uncertainty estimation.