What classifiers know what they don't know?

1 INTRODUCTION I don ’ t think I ’ ve ever seen anything quite like this before —HAL 9000 in 2001 : A Space Odyssey Deep image classifiers exceed at discriminating the set of in-domain classes observed during training . However , when confronting test examples from unseen out-domain classes , these classifiers can only predict in terms of known categories , leading to wrong and overly-confident decisions ( Hein et al. , 2019 ; Ulmer & Cinà , 2020 ) . In short , machine learning systems are unaware of their own limits , since “ they do not know what they do not know ” . Since out-domain data can not be safely identified and treated accordingly , it is reasonable to fear that , when deployed in-the-wild , the safety and performance of these classifiers crumbles by leaps and bounds ( Ovadia et al. , 2019 ) . The inability of machine learning systems to estimate their uncertainty and abstaining to classify outdomain classes roadblocks their use in critical applications . These include self-driving ( Michelmore et al. , 2018 ) , medicine ( Begoli et al. , 2019 ) , and the analysis of satellite imagery ( Wadoux , 2019 ) . Good uncertainty estimates are also a key ingredient in anomaly detection ( Chalapathy & Chawla , 2019 ) , active learning ( Settles , 2009 ) , safe reinforcement learning ( Henaff et al. , 2019 ) , defending against adversarial examples ( Goodfellow et al. , 2014 ) , and model interpretability ( Alvarez-Melis & Jaakkola , 2017 ) . For an extensive literature review on uncertainty estimation and its applications , we refer the curious reader to the surveys of Abdar et al . ( 2020 ) and Ruff et al . ( 2021 ) . Despite a research effort spanning multiple decades , machine learning systems still lack trustworthy estimates of their predictive uncertainty . In our view , one hindrance to this research program is the absence of realistic benchmarking and evaluation protocols . More specifically , prior attempts are limited in two fundamental ways . First , these experiment on small datasets such as SVHN and CIFAR-10 ( van Amersfoort et al. , 2021 ) . Second , these do not provide a challenging set of out-domain data . Instead , they construct out-domain classes by using a second dataset ( e.g. , using MNIST in-domain versus FashionMNIST out-domain , cf . Van Amersfoort et al . ( 2020 ) ) or by perturbing the in-domain classes using handcrafted transformations ( such as Gaussian noise or blur , see ImageNet-C Hendrycks & Dietterich ( 2019 ) ) . Both approaches result in simplistic benchmarking , and little is learned about uncertainty estimation for the real world . One exception is the ImageNet-O dataset ( Hendrycks et al. , 2021 ) , where out-domain data is selected from ImageNet-22k classes not contained in ImageNet1k . However , the images in ImageNet-O are collected adversarially as to maximize the prediction confidence of ResNet50 ImageNet-1k classifiers , creating what ’ s possibly an unnecessarily difficult benchmark . The purpose of this work is to introduce an end-to-end benchmark and evaluation protocol as realistic as possible . At the time of writing , UIMNET is the most exhaustive benchmark for uncertainty estimation in the literature . Formal setup We learn classifiers f using in-domain data from some distribution Pin ( X , Y ) . After training , we endow the classifier with a real-valued uncertainty measure u ( f , x† ) . Given a test example ( x† , y† ) ⇠ P with unobserved label y† , we declare x† in-domain ( hypothesizing P = Pin ) if u ( f , x† ) is small , whereas we declare x† out-domain ( hypothesizing P 6= Pin ) if u ( f , x† ) is large . Using these tools , our goal is to abstain from classifying out-domain test examples , and to classify with calibrated probabilities in-domain test examples . The sequel assumes that the difference between in- and out-domain resides in that the two groups of data concern disjoint classes . Contributions We introduce UIMNET , a test-bed for large-scale , realistic evaluation of uncertainty estimates in deep image classifiers . We build UIMNET as follows ( see also Table 1 ) . ( Sec . 2 ) We construct ImageNot , a perceptual partition of ImageNet into in-domain and outdomain classes . Unlike most prior work focusing on small datasets like SVHN and CIFAR-10 , ImageNot provides a benchmark for uncertainty estimators at a much larger scale . Moreover , both in-domain and out-domain categories in ImageNot originate from the original ImageNet dataset . This provides realistic out-domain data , as opposed to prior work relying on a second dataset ( e.g. , MNIST as in-domain versus SVHN as out-domain ) , or handcrafted perturbations of in-domain classes ( Gaussian noise or blur as out-domain ) . ( Sec . 3 ) We re-implement eight state-of-the-art algorithms from scratch , listed in Table 1 . This allows a fair comparison under the exact same experimental conditions ( training/validation splits , hyper-parameter search , neural network architectures and random initializations ) . Furthermore , we also study ensembles of multiple training instances for each algorithm . ( Sec . 4 ) Each algorithm can be endowed with one out of six possible uncertainty measures , allowing an exhaustive study of what algorithms play well with what measures . Listed in Table 1 , these are the largest softmax score , the gap between the two largest softmax scores , the softmax entropy , the norm of the Jacobian , a per-class Gaussian density model , and ( for those available ) an algorithm-specific measure . ( Sec . 5 ) For each classifier-measure pair , we study four in-domain metrics ( top-1 and top-5 classification accuracy , log-likelihood , expected calibration error ) and three out-domain metrics ( the AUC at classifying in-domain versus out-domain samples using the selected uncertainty measure , as well as the confusion matrix at a fixed uncertainty threshold computed over an in-domain validation set ) . ( Sec . 6 ) We explore three popular ablations to understand the impact of model calibration by temperature scaling , model size , and the use of spectral normalization . ( Sec . 7 ) UIMNET is entirely hands-off , since the pipeline from zero to LATEX tables is fully automated : this includes hyper-parameter search , model calibration , model ensembling , and the production of all the tables included in our experimental results . ( Sec . 8 ) Our experimental results reveal that , in order to obtain the best possible uncertainty estimates in large-scale image classification , the practitioner should favor large , calibrated models . We recommend the use of single ERM models , single MIMO models , or ensembles of ERM models , in order of increasing performance and required computational budget . UIMNET is open sourced at https : //github.com/ANONYMOUS . All of the tables presented in this paper are reproducible by running the main script in the repository at commit 0xANON . 2 CONSTRUCTING THE IMAGENOT BENCHMARK . The ImageNet dataset ( Russakovsky et al. , 2015 ) is a gold standard to conduct research in computer vision pertaining image data of 1000 different classes . Here we use the ImageNet dataset to derive ImageNot , a large-scale and realistic benchmark for uncertainty estimation . ImageNot partitions the 1000 classes of the original ImageNet dataset into in-domain classes ( used to train and evaluate algorithms in-distribution ) and out-domain classes ( used to evaluate algorithms out-of-distribution ) . To partition ImageNet into in-domain and out-domain , we featurize the entire dataset to understand the perceptual similarity between classes . To this end , we use a pre-trained ResNet-18 ( He et al. , 2016 ) to compute the average last-layer representation for each of the classes . Next , we use agglomerative hierarchical clustering Ward Jr ( 1963 ) to construct a tree describing the perceptual similarities between the 1000 average feature vectors . Such perceptual tree has 1000 leafs , each of them being a cluster containing one of the classes . During each step of the iterative agglomerative clustering algorithm the two closest clusters are merged , where the distance between two clusters is computed using the criterion of Ward Jr ( 1963 ) . The algorithm halts when there are only two clusters left to merge , forming the root node of the tree . At this point , we declare the 266 classes to the left of the root as in-domain , and the first 266 classes to the right of the root as out-domain . In the sequel , we call “ training set ” and “ validation set ” to a 90/10 random split from the original ImageNet “ train ” set . We call “ testing set ” to the original ImageNet “ val ” split . The exact in-/out-domain class partition as well as the considered train/validation splits are specified in Appendix D. Our agglomerative clustering procedure ended-up congregating different types of objects as in-domain classes , while grouping animals as out-domain classes . While inspired by the BREEDS ImageNet splits ( Santurkar et al. , 2020 ) , our benchmark ImageNot is conceived to tackle a different problem . The aim of the BREEDS dataset is to classify ImageNet into a small number of super-classes , each of them containing a number of perceptually-similar sub-classes . The BREEDS training and testing distributions differ on the sub-class proportions contributing to their super-classes . Since the BREEDS task is to classify super-classes , the set of labels remains constant from training to testing conditions . This is in contrast to ImageNot , where the algorithm observes only in-domain classes during training , but both in-domain and out-domain classes during evaluation . While BREEDS studies the important problem of domain generalization ( Gulrajani & Lopez-Paz , 2020 ) , where there is always a right prediction to make within the in-domain classes during evaluation , here we focus on measuring uncertainty and abstaining from predicting about those out-domain classes unavailable during training . ImageNot is also similar to the ImageNet-O dataset of ( Hendrycks et al. , 2021 ) . However , their outdomain images are collected adversarially , that is , to maximize the prediction confidence of ResNet50 classifiers . We believe that this drastic change in selection bias from ImageNet to ImageNet-O may result in an unnecessarily difficult uncertainty estimation benchmark . Thus , here we favor using only the original ImageNet data , as described above . In contrast , the starting point for both in-domain and out-domain classes of our ImageNot is the ImageNet dataset , and thus should maximally overlap in terms of image statistics , leading to a challenging and realistic benchmark . 3 ALGORITHMS . We benchmark ten supervised learning algorithms commonly applied to tasks involving uncertainty estimation . Each algorithm consumes one in-domain training set of image-label pairs { ( xi , yi ) } ni=1 and returns a predictor f ( x ) = w ( ( x ) ) , composed by a featurizer : R3⇥224⇥224 ! Rk and a classifier w : Rk ! RC . We consider predictors implemented using deep convolutional neural networks ( LeCun et al. , 2015 ) . Given an input image x† , all predictors return a softmax vector f ( x† ) = ( f ( x† ) c ) Cc=1 over C classes . The considered algorithms are : • Empirical Risk Minimization , or ERM ( Vapnik , 1992 ) , or vanilla training . • Mixup ( Zhang et al. , 2017 ) chooses a predictor minimizing the empirical risk on mixed examples ( · xi + ( 1 ) · xj , · yi + ( 1 ) · yj ) , where ⇠ Beta ( ↵ , ↵ ) , ↵ is a mixing parameter , and ( ( xi , yi ) , ( xj , yj ) ) is a random pair of training examples . Mixup improves generalization performance ( Zhang et al. , 2017 ) and calibration ( Thulasidasan et al. , 2019 ) . • Random Network Distillation , or RND ( Burda et al. , 2018 ) , finds an ERM predictor f ( x ) = w ( ( x ) ) , but simultaneously trains an auxiliary classifier wstudent to minimize kwstudent ( ( x ) ) wteacher ( ( x ) ) k22 , where wteacher is a fixed classifier with random weights . RND has shown good performance as a tool for exploration in reinforcement learning . • Orthogonal Certificates , or OC ( Tagasovska & Lopez-Paz , 2018 ) , is analogous to RND for wteacher ( ( x ) ) = ~0k for all x . That is , the goal of wstudent is to map all the in-domain training examples to zero in k different ways ( or certificates ) . To ensure diverse and non-trivial certificates , we regularize each weight matrix W of wstudent to be orthogonal by adding a regularization term kW > W Ik22 . OCs have shown good performance at the task of estimating uncertainty across a variety of classification tasks . • Autoencoder , or DeepAE ( Vincent et al. , 2010 ) , is analogous to RND for wteacher ( ( x ) ) = ( x ) and a wstudent with a bottleneck . • MC-Dropout ( Gal & Ghahramani , 2016 ) trains ERMs with one or more dropout layers ( Srivastava et al. , 2014 ) . These stochastic dropout layers remain active at test time , allowing the predictor to produce multiple softmax vectors { f ( x† , dropoutt ) } Tt=1 for each test example x† . Here , dropoutt is a random dropout mask sampled anew . MCDropout is one of the most popular baselines to estimate uncertainty . • MIMO ( Havasi et al. , 2021 ) is a variant of ERM over predictors accepting T images and producing T softmax vectors . For example , MIMO with T = 3 is trained to predict jointly the label vector ( yi , yj , yk ) using a predictor h ( xi , xj , xk ) , where ( ( xt , yt ) ) 3i=1 is a random triplet of training examples . Given a test point x† , we form predictions by replicating and averaging , that is f ( x† ) = 13 P3 t=1 h ( x † , x † , x † ) t. • Radial Basis Function , or RBF ( Broomhead & Lowe , 1988 ) , is a variant of ERM where we transform the logit vector z 7 ! e z2 before passing them to the final softmax layer . In such a way , as the logit norm kzk ! 1 , the predicted softmax vector tends to the maximum entropy solution ( 1C ) C c=1 , signaling high uncertainty far away from the training data . RBFs have been proposed as defense to adversarial examples ( Goodfellow et al. , 2014 ) , but they remain under-explored given the difficulties involved in their training . • Soft labeler ( Hinton et al. , 2015 ; Szegedy et al. , 2016 ) is a variant of ERM where the one-hot vector labels yi are smoothed such that every zero becomes ` min > 0 and the solitary one becomes ` max < 1 . Using soft labels , we can identify softmax vectors with entries exceeding ` max as “ over-shoots ” , and regard them as uncertain predictions . • DUE ( van Amersfoort et al. , 2021 ) enforces the smoothness of the featurizer using spectral normalization ( Miyato et al. , 2018 ) , implements the classifier w as a sparse Gaussian Process ( Quinonero-Candela & Rasmussen , 2005 ) , and trains the resulting predictor using variational inference ( Titsias , 2009 ) . Gaussian processes are considered one of the main tools to estimate predictive uncertainty in machine learning systems . Ensembles of predictors We also consider ensembles of predictors trained by each of the algorithms above . Ensembles are commonly regarded as the state-of-the-art in uncertainty estimation ( Lakshminarayanan et al. , 2016 ) . In particular , and for each algorithm , we construct bagging ensembles by ( i ) selecting the best K 2 { 1 , 5 } predictors { fk } Kk=1 from all considered random initializations and hyper-parameters , and ( ii ) returning the average function f ( x† ) : = 1K PM k=1 f k ( x† ) .

Summary: This paper proposes the UIMNET benchmark for uncertainty estimation. This benchmark includes (1) "ImageNot", a remix of ImageNet using hierarchical clustering of pairwise distances between features (2) framework for evaluating uncertainty estimation, including a suite of algorithms, metrics, and ablations studies. The authors report empirical results on their dataset using their suite, and make a number of recommendations based on these findings. Reproducible software is also provided.