Year: 2021

Abstract:
Being uncertain when facing the unknown is key to intelligent decision making. However, machine learning algorithms lack reliable estimates about their predictive uncertainty. This leads to wrong and overly-confident decisions when encountering classes unseen during training. Despite the importance of equipping classifiers with uncertainty estimates ready for the real world, most prior work focuses on small datasets with little or no class discrepancy between training and testing data. To close this gap, we introduce UIMNET: a realistic, ImageNet-scale test-bed to evaluate predictive uncertainty estimates for deep image classifiers. Our benchmark provides implementations of ten state-of-the-art algorithms, six uncertainty measures, four in-domain metrics, three out-domain metrics, and a fully automated pipeline to train, calibrate, ensemble, select, and evaluate models. Our test-bed is open-source and all of our results are reproducible from a fixed commit in our repository. Adding new datasets, algorithms, measures, or metrics is a matter of a few lines of code—in so hoping that UIMNET becomes a stepping stone towards realistic, rigorous, and reproducible research in uncertainty estimation. Our experimental results reveal that, in order to obtain the best possible uncertainty estimates in large-scale image classification, the practitioner should favor large, calibrated models. We recommend the use of single ERM models, single MIMO models, or ensembles of ERM models, in order of increasing performance and required computational budget.

text: 1 INTRODUCTION
I don’t think I’ve ever seen anything quite like this before —HAL 9000 in 2001: A Space Odyssey
Deep image classifiers exceed at discriminating the set of in-domain classes observed during training. However, when confronting test examples from unseen out-domain classes, these classifiers can only predict in terms of known categories, leading to wrong and overly-confident decisions (Hein et al., 2019; Ulmer & Cinà, 2020). In short, machine learning systems are unaware of their own limits, since “they do not know what they do not know”. Since out-domain data cannot be safely identified and treated accordingly, it is reasonable to fear that, when deployed in-the-wild, the safety and performance of these classifiers crumbles by leaps and bounds (Ovadia et al., 2019). The inability of machine learning systems to estimate their uncertainty and abstaining to classify outdomain classes roadblocks their use in critical applications. These include self-driving (Michelmore et al., 2018), medicine (Begoli et al., 2019), and the analysis of satellite imagery (Wadoux, 2019). Good uncertainty estimates are also a key ingredient in anomaly detection (Chalapathy & Chawla, 2019), active learning (Settles, 2009), safe reinforcement learning (Henaff et al., 2019), defending against adversarial examples (Goodfellow et al., 2014), and model interpretability (Alvarez-Melis & Jaakkola, 2017). For an extensive literature review on uncertainty estimation and its applications, we refer the curious reader to the surveys of Abdar et al. (2020) and Ruff et al. (2021). Despite a research effort spanning multiple decades, machine learning systems still lack trustworthy estimates of their predictive uncertainty. In our view, one hindrance to this research program is the absence of realistic benchmarking and evaluation protocols. More specifically, prior attempts are limited in two fundamental ways. First, these experiment on small datasets such as SVHN and CIFAR-10 (van Amersfoort et al., 2021). Second, these do not provide a challenging set of out-domain data. Instead, they construct out-domain classes by using a second dataset (e.g., using MNIST in-domain versus FashionMNIST out-domain, cf. Van Amersfoort et al. (2020)) or by perturbing the in-domain classes
using handcrafted transformations (such as Gaussian noise or blur, see ImageNet-C Hendrycks & Dietterich (2019)). Both approaches result in simplistic benchmarking, and little is learned about uncertainty estimation for the real world. One exception is the ImageNet-O dataset (Hendrycks et al., 2021), where out-domain data is selected from ImageNet-22k classes not contained in ImageNet1k. However, the images in ImageNet-O are collected adversarially as to maximize the prediction confidence of ResNet50 ImageNet-1k classifiers, creating what’s possibly an unnecessarily difficult benchmark. The purpose of this work is to introduce an end-to-end benchmark and evaluation protocol as realistic as possible. At the time of writing, UIMNET is the most exhaustive benchmark for uncertainty estimation in the literature.
Formal setup We learn classifiers f using in-domain data from some distribution Pin(X,Y ). After training, we endow the classifier with a real-valued uncertainty measure u(f, x†). Given a test example (x†, y†) ⇠ P with unobserved label y†, we declare x† in-domain (hypothesizing P = Pin) if u(f, x†) is small, whereas we declare x† out-domain (hypothesizing P 6= Pin) if u(f, x†) is large. Using these tools, our goal is to abstain from classifying out-domain test examples, and to classify with calibrated probabilities in-domain test examples. The sequel assumes that the difference between in- and out-domain resides in that the two groups of data concern disjoint classes.
Contributions We introduce UIMNET, a test-bed for large-scale, realistic evaluation of uncertainty estimates in deep image classifiers. We build UIMNET as follows (see also Table 1).
(Sec. 2) We construct ImageNot, a perceptual partition of ImageNet into in-domain and outdomain classes. Unlike most prior work focusing on small datasets like SVHN and CIFAR-10, ImageNot provides a benchmark for uncertainty estimators at a much larger scale. Moreover, both in-domain and out-domain categories in ImageNot originate from the original ImageNet dataset. This provides realistic out-domain data, as opposed to prior work relying on a second dataset (e.g., MNIST as in-domain versus SVHN as out-domain), or handcrafted perturbations of in-domain classes (Gaussian noise or blur as out-domain).
(Sec. 3) We re-implement eight state-of-the-art algorithms from scratch, listed in Table 1. This allows a fair comparison under the exact same experimental conditions (training/validation splits, hyper-parameter search, neural network architectures and random initializations). Furthermore, we also study ensembles of multiple training instances for each algorithm.
(Sec. 4) Each algorithm can be endowed with one out of six possible uncertainty measures, allowing an exhaustive study of what algorithms play well with what measures. Listed in Table 1, these are the largest softmax score, the gap between the two largest softmax scores, the softmax entropy, the norm of the Jacobian, a per-class Gaussian density model, and (for those available) an algorithm-specific measure.
(Sec. 5) For each classifier-measure pair, we study four in-domain metrics (top-1 and top-5 classification accuracy, log-likelihood, expected calibration error) and three out-domain metrics (the AUC at classifying in-domain versus out-domain samples using the selected
uncertainty measure, as well as the confusion matrix at a fixed uncertainty threshold computed over an in-domain validation set).
(Sec. 6) We explore three popular ablations to understand the impact of model calibration by temperature scaling, model size, and the use of spectral normalization.
(Sec. 7) UIMNET is entirely hands-off, since the pipeline from zero to LATEX tables is fully automated: this includes hyper-parameter search, model calibration, model ensembling, and the production of all the tables included in our experimental results.
(Sec. 8) Our experimental results reveal that, in order to obtain the best possible uncertainty estimates in large-scale image classification, the practitioner should favor large, calibrated models. We recommend the use of single ERM models, single MIMO models, or ensembles of ERM models, in order of increasing performance and required computational budget.
UIMNET is open sourced at https://github.com/ANONYMOUS. All of the tables presented in this paper are reproducible by running the main script in the repository at commit 0xANON.

heading: 2 CONSTRUCTING THE IMAGENOT BENCHMARK
text: The ImageNet dataset (Russakovsky et al., 2015) is a gold standard to conduct research in computer vision pertaining image data of 1000 different classes. Here we use the ImageNet dataset to derive ImageNot, a large-scale and realistic benchmark for uncertainty estimation. ImageNot partitions the 1000 classes of the original ImageNet dataset into in-domain classes (used to train and evaluate algorithms in-distribution) and out-domain classes (used to evaluate algorithms out-of-distribution).
To partition ImageNet into in-domain and out-domain, we featurize the entire dataset to understand the perceptual similarity between classes. To this end, we use a pre-trained ResNet-18 (He et al., 2016) to compute the average last-layer representation for each of the classes. Next, we use agglomerative hierarchical clustering Ward Jr (1963) to construct a tree describing the perceptual similarities between the 1000 average feature vectors. Such perceptual tree has 1000 leafs, each of them being a cluster containing one of the classes. During each step of the iterative agglomerative clustering algorithm the two closest clusters are merged, where the distance between two clusters is computed using the criterion of Ward Jr (1963). The algorithm halts when there are only two clusters left to merge, forming the root node of the tree.
At this point, we declare the 266 classes to the left of the root as in-domain, and the first 266 classes to the right of the root as out-domain. In the sequel, we call “training set” and “validation set” to a 90/10 random split from the original ImageNet “train” set. We call “testing set” to the original ImageNet “val” split. The exact in-/out-domain class partition as well as the considered train/validation splits are specified in Appendix D. Our agglomerative clustering procedure ended-up congregating different types of objects as in-domain classes, while grouping animals as out-domain classes.
While inspired by the BREEDS ImageNet splits (Santurkar et al., 2020), our benchmark ImageNot is conceived to tackle a different problem. The aim of the BREEDS dataset is to classify ImageNet into a small number of super-classes, each of them containing a number of perceptually-similar sub-classes. The BREEDS training and testing distributions differ on the sub-class proportions contributing to their super-classes. Since the BREEDS task is to classify super-classes, the set of labels remains constant from training to testing conditions. This is in contrast to ImageNot, where the algorithm observes only in-domain classes during training, but both in-domain and out-domain classes during evaluation. While BREEDS studies the important problem of domain generalization (Gulrajani & Lopez-Paz, 2020), where there is always a right prediction to make within the in-domain classes during evaluation, here we focus on measuring uncertainty and abstaining from predicting about those out-domain classes unavailable during training.
ImageNot is also similar to the ImageNet-O dataset of (Hendrycks et al., 2021). However, their outdomain images are collected adversarially, that is, to maximize the prediction confidence of ResNet50 classifiers. We believe that this drastic change in selection bias from ImageNet to ImageNet-O may result in an unnecessarily difficult uncertainty estimation benchmark. Thus, here we favor using only the original ImageNet data, as described above. In contrast, the starting point for both in-domain and
out-domain classes of our ImageNot is the ImageNet dataset, and thus should maximally overlap in terms of image statistics, leading to a challenging and realistic benchmark.

heading: 3 ALGORITHMS
text: We benchmark ten supervised learning algorithms commonly applied to tasks involving uncertainty estimation. Each algorithm consumes one in-domain training set of image-label pairs {(xi, yi)}ni=1 and returns a predictor f(x) = w( (x)), composed by a featurizer : R3⇥224⇥224 ! Rk and a classifier w : Rk ! RC . We consider predictors implemented using deep convolutional neural networks (LeCun et al., 2015). Given an input image x†, all predictors return a softmax vector f(x†) = (f(x†)c)Cc=1 over C classes. The considered algorithms are:
• Empirical Risk Minimization, or ERM (Vapnik, 1992), or vanilla training. • Mixup (Zhang et al., 2017) chooses a predictor minimizing the empirical risk on mixed examples ( · xi + (1 ) · xj , · yi + (1 ) · yj), where ⇠ Beta(↵,↵), ↵ is a mixing parameter, and ((xi, yi), (xj , yj)) is a random pair of training examples. Mixup improves generalization performance (Zhang et al., 2017) and calibration (Thulasidasan et al., 2019). • Random Network Distillation, or RND (Burda et al., 2018), finds an ERM predictor f(x) = w( (x)), but simultaneously trains an auxiliary classifier wstudent to minimize kwstudent( (x)) wteacher( (x))k22, where wteacher is a fixed classifier with random weights. RND has shown good performance as a tool for exploration in reinforcement learning. • Orthogonal Certificates, or OC (Tagasovska & Lopez-Paz, 2018), is analogous to RND for wteacher( (x)) = ~0k for all x. That is, the goal of wstudent is to map all the in-domain training examples to zero in k different ways (or certificates). To ensure diverse and non-trivial certificates, we regularize each weight matrix W of wstudent to be orthogonal by adding a regularization term kW>W Ik22. OCs have shown good performance at the task of estimating uncertainty across a variety of classification tasks. • Autoencoder, or DeepAE (Vincent et al., 2010), is analogous to RND for wteacher( (x)) = (x) and a wstudent with a bottleneck. • MC-Dropout (Gal & Ghahramani, 2016) trains ERMs with one or more dropout layers (Srivastava et al., 2014). These stochastic dropout layers remain active at test time, allowing the predictor to produce multiple softmax vectors {f(x†, dropoutt)}Tt=1 for each test example x†. Here, dropoutt is a random dropout mask sampled anew. MCDropout is one of the most popular baselines to estimate uncertainty. • MIMO (Havasi et al., 2021) is a variant of ERM over predictors accepting T images and producing T softmax vectors. For example, MIMO with T = 3 is trained to predict jointly the label vector (yi, yj , yk) using a predictor h(xi, xj , xk), where ((xt, yt))3i=1 is a random triplet of training examples. Given a test point x†, we form predictions by replicating and averaging, that is f(x†) = 13 P3 t=1 h(x † , x † , x
†)t. • Radial Basis Function, or RBF (Broomhead & Lowe, 1988), is a variant of ERM where we
transform the logit vector z 7! e z2 before passing them to the final softmax layer. In such a way, as the logit norm kzk ! 1, the predicted softmax vector tends to the maximum entropy solution ( 1C ) C c=1, signaling high uncertainty far away from the training data. RBFs have been proposed as defense to adversarial examples (Goodfellow et al., 2014), but they remain under-explored given the difficulties involved in their training. • Soft labeler (Hinton et al., 2015; Szegedy et al., 2016) is a variant of ERM where the one-hot vector labels yi are smoothed such that every zero becomes `min > 0 and the solitary one becomes `max < 1. Using soft labels, we can identify softmax vectors with entries exceeding `max as “over-shoots”, and regard them as uncertain predictions. • DUE (van Amersfoort et al., 2021) enforces the smoothness of the featurizer using spectral normalization (Miyato et al., 2018), implements the classifier w as a sparse Gaussian Process (Quinonero-Candela & Rasmussen, 2005), and trains the resulting predictor using variational inference (Titsias, 2009). Gaussian processes are considered one of the main tools to estimate predictive uncertainty in machine learning systems.
Ensembles of predictors We also consider ensembles of predictors trained by each of the algorithms above. Ensembles are commonly regarded as the state-of-the-art in uncertainty estimation (Lakshminarayanan et al., 2016). In particular, and for each algorithm, we construct bagging ensembles
by (i) selecting the best K 2 {1, 5} predictors {fk}Kk=1 from all considered random initializations and hyper-parameters, and (ii) returning the average function f(x†) := 1K PM k=1 f k(x†).

heading: 4 UNCERTAINTY MEASURES
text: We equip a trained predictor f with six different uncertainty measures. An uncertainty measure is a real-valued function u(f, x†) designed to return small values for in-domain instances x†, and large values for out-domain instances x†. To describe the different measures, let {s(1), . . . , s(C)} be the softmax scores returned by f(x†) sorted in decreasing order.
• Largest (Hendrycks & Gimpel, 2016) returns minus the largest softmax score, s(1) • Softmax gap (Tagasovska & Lopez-Paz, 2018) returns s(2) s(1). • Entropy (Shannon, 1948) returns PC c=1 s(c)I{s(c) > 0} log s(c). • Norm of the Jacobian (Novak et al., 2018) returns krxf(x†)k22. • GMM (Mukhoti et al., 2021) estimates one Gaussian density N ( (x);µc,⌃c) per-class, on
top of the feature vectors (x) collected from a in-domain validation set. Given a test example x †, return PC c=1 c · N ( (x†);µc,⌃c), where c is the proportion of in-domain validation
examples from class c. • Test-time augmentation (Ashukha et al., 2020) returns maxc( 1A PA a=1 f(x † a))c. This is the
measure “Largest” about the average prediction over A random data augmentations {x†a}Aa=1 of the test instance x†.
These uncertainty measures are applicable to all the algorithms considered in Section 3. Additionally, some algorithms provide their Native uncertainty measures, outlined below.
• For Mixup, we return 1K PK
k=1 k · f(x†) + (1 ) · ȳk f( · x† + (1 ) · x̄k)k22, where ⇠ Beta(↵,↵), and (x̄k, ȳk) is an example saved from the training set. This measures if the test example x† violates the Mixup criterion wrt the training dataset average. • For RND, OC, and DeepAE we return kwstudent( (x†)) wteacher( (x†))k22, that is, we consider a prediction uncertain if the outputs of the student and teacher disagree. We expect this disagreement to be related predictive uncertainty, as the student did not observe the behaviour of the teacher at out-domain instances x†. • For Soft labeler we return (s(1) `max)2. This measures the discrepancy between the largest softmax and the positive soft label target, able to signal overly-confident predictions. • For MC-Dropout and Ensembles, and following (Lakshminarayanan et al., 2016), we return the Jensen-Shannon divergence between the K members (or stochastic forward passes) f1, . . . , fK
of the ensemble, H ⇣
1 K PK k=1 f k(x†) ⌘ 1K PK k=1 H(f
k(x†)). • For DUE, we return the predictive variance of the Gaussian process classifier.

heading: 5 EVALUATION METRICS
text: For each algorithm-measure pair, we evaluate several metrics both in-domain and out-domain.
Following (Havasi et al., 2021), we implement four in-domain metrics to assess the performance and calibration of predictors when facing in-domain test examples.
• Top-1 and Top-5 classification accuracy (Russakovsky et al., 2015). • Expected Calibration Error or ECE (Guo et al., 2017):
1 B PB b=1 |Bb| n |acc(f,Bb) conf(f,Bb)|, where Bb contains the examples where the algorithm predicts a softmax score of b. The functions acc and conf compute the average classification accuracy and largest softmax score of f over Bb. ECE is minimized when f is calibrated, that is, f is wrong p% of the times it predicts a largest softmax score p. Following (Guo et al., 2017), we discretize b 2 [0, 1] into 15 equally-spaced bins. • Negative Log Likelihood (NLL) Also known as the cross-entropy loss, this is the objective minimized during the training process of the algorithms.
We assess the uncertainty estimates of each predictor-measure pair using three out-domain metrics.
• Area Under the Curve, or AUC (Tagasovska & Lopez-Paz, 2018), describes how well does the predictor-measure pair distinguish between in-domain and out-domain examples over all thresholds of the uncertainty measure. • Confusion matrix at fixed threshold. To reject out-domain examples in real scenarios, one must fix a threshold ✓ for the selected uncertainty measure. We do so by computing the 95% quantile of the uncertainty measure, computed over an in-domain validation set. Then, at testing time, we declare one example out-domain if the uncertainty measure exceeds ✓.1 To understand where does the uncertainty measure hit or miss, we monitor the metrics InAsIn (percentage of in-domain examples classified as in-domain) and OutAsOut (percentage of in-domain examples classified as out-domain).

heading: 6 ABLATIONS
text: We execute our entire test-bed under three popular ablations.
• We study the effect of calibration by temperature scaling (Platt et al., 1999). To this end, we introduce a temperature scaling ⌧ > 0 before the softmax layer, resulting in predictions Softmax( z⌧ ) about the logit vector z. We estimate the optimal temperature ⌧̂ by minimizing the NLL of the predictor across an in-domain validation set. We evaluate all metrics for both the un-calibrated (⌧ = 1) and calibrated (⌧ = ⌧̂ ) predictors. According to previous literature (Guo et al., 2017), calibrated models provide better in-domain uncertainty estimates. • We analyze the impact of spectral normalization to control the behavior of the featurizer . More specifically, recent works (Liu et al., 2020; Van Amersfoort et al., 2020; van Amersfoort et al., 2021; Mukhoti et al., 2021) have highlighted the importance of controlling both the smoothness and sensitivity of the feature extraction process to avoid feature collapse and achieve high-quality uncertainty estimates. On the one hand, enforcing smoothness upper bounds the Lipschitz constant of , limiting its reaction to changes in the input. Smoothness is often enforced by normalizing each weight matrix in by its spectral norm (Miyato et al., 2018). On the other hand, enforcing sensitivity lower bounds the Lipschitz constant of , ensuring that the feature space reacts in some amount when the input changes. Sensitivity is often enforced by the residual connections of the hereby used ResNet models (He et al., 2016). We apply one-sided spectral normalization, with a target spectral norm of 5, to the weights and the batch normalization layers following Miyato et al. (2018) and Gouk et al. (2021) respectively. • We analyze the impact of the model size (ResNet-18 versus ResNet-50).

heading: 7 EXPERIMENTAL PROTOCOL
text: We now conduct experiments on the ImageNot benchmark (Section 2) for all algorithms (Section 3) and measures (Section 4), wrt all metrics (Section 5) and ablations (Section 6). Empirical oracle upper bounds To have a measure of the maximally achievable separation of the in-domain and out-of-domain partitions, we train a ResNet-18 classifier to discriminate between the In-domain classes and the out-of-domain partition. The oracle performance is 0.985± 0.002, 0.944± 0.001 and 0.947± 0.002 for the AUC, InAsIn and OutAsOut metrics respectively. Hyper-parameter search. We train each algorithm using (i) ResNet-18 or ResNet-50 architectures (ii) spectral normalization or not, (iii) ten hyper-parameter trials, and (iv) three random train/validation splits of the in-domain data (data seeds). We opt for a random hyper-parameter search (Bergstra & Bengio, 2012), where the search grid for each algorithm is detailed in Appendix C. More specifically, while the first trial uses the default hyper-parameter configuration suggested by the authors of each algorithm, the additional four trials explore random hyper-parameters.
Model selection. After training all instances of a given algorithm, we report the test average and standard deviation (over data seeds) of all metrics. We report these metrics for (a) the best model
1This strategy is equivalent to the statistical hypothesis test with null “H0: the observed example is indomain”.
(k = 1), and (b) the ensemble containing the best five models (k = 5) in terms of the validation average (over data seeds) negative log-likelihood2.
Optimization. We use PyTorch (Paszke et al., 2019) and SGD (Bottou, 2012) for 100 epochs using mini-batches of 256 examples distributed over 8 NVIDIA Tesla V100 GPUs, and a learning rate decaying by a factor of 10 every 30 epochs.

heading: 8 RESULTS
text: The full experimental results are available in Appendix A (in-domain) and Appendix B (out-domain). Below, we summarize our findings.
From the in-domain results summarized in Table 2, we identify the following key takeaways:
• No single method out-performs ERM significantly on any in-domain metric, except MIMO on ECE (-30% ECE). • No ensemble method outperforms ensembles of ERMs in any in-domain metric. • Increasing model size is the most effective strategy to improve in-domain metrics (+5%
ACC@1, +2% ACC@5, -17% NLL, -13% ECE). • Ensembling is the second most effective strategy to improve in-domain metrics (+2.5%
ACC@1, +1% ACC@5, -8.5% NLL, -8% ECE). • Calibration is an effective strategy to improve test negative log-likelihood and expected
calibration error (-2% NLL, -45% ECE). • Spectral normalization has little effect to improve in-domain metrics (<1%). • DUE has a brittle behavior, performing well only for default hyper-parameters and
ResNet18 architectures.
From the out-domain results summarized in Table 3, corresponding to the best performing uncertainty measure “Entropy”, we identify the following key takeaways:
• No single method outperforms ERM significantly on any out-domain metric, except MIMO on OutAsOut (+5% OutAsOut). • No ensemble method outperforms ensembles of ERMs in any out-domain metric. • Increasing model size is the most effective strategy to improve out-domain metrics (+4%
AUC, +35% OutAsOut). • Ensembling is the second most effective strategy to improve out-domain metrics (+1%
AUC, +4% OutAsOut). • Calibration has a small negative effect on out-domain metrics (-1%). • Spectral normalization has a small positive effect on out-domain metrics (<1%). • All methods are able to upper-bound their type-I errors to the requested threshold of 5%. • The best performing uncertainty measure is Entropy; then Largest, and Gap follow. The
worst performing measures are Augmentations, Jacobian, and Native (Appendix B).
Other results We were unable to obtain competitive performances for the algorithm RBF (Broomhead & Lowe, 1988) and the measure GMM (Mukhoti et al., 2021). We believe that training RBFs at this large scale are challenging optimization problems that deserve further study in our community. Furthermore, the large number of classes in our study (266 ImageNet classes instead of the 10 CIFAR-10 classes often considered) poses a difficult problem for the density-based measures GMM.

heading: 9 CONCLUSION
text: To obtain the best possible uncertainty estimates in large-scale image classification, we recommend the use of large and calibrated models. We favor single ERM models (low computational budget), single MIMO models (medium budget), or ensembles of ERM models (high budget).
2Graphs showing the relationship between validation set negative likelihood and out-of-domain measures on the test set are show in the appendix.
ETHICS STATEMENT
This work does not involve the release of any new data, or the development of any new algorithm. However, we believe that the development of better uncertainty estimates for our predictive models is at the core of responsible, fair machine learning. As such, we hope that the UIMNET baseline is a stepping stone towards advancing the ability of our systems to say “I don’t know”. Also, we hope that our manuscript and software aids future research to study uncertainty estimates as components also vulnerable to attacks and with their own set of limitations.
REPRODUCIBILITY
A main focus of this manuscript is reproducibility. To replicate all of our experimental results, obtain our code from the Supplementary Material and follow the instructions at README.md. Upon publication, our code will be open-sourced in a GitHub repository, and all of our experimental results will be replicable from a fixed commit hash.

Review summaries:
This paper proposes the UIMNET benchmark for uncertainty estimation. This benchmark includes (1) "ImageNot", a remix of ImageNet using hierarchical clustering of pairwise distances between features (2) framework for evaluating uncertainty estimation, including a suite of algorithms, metrics, and ablations studies. The authors report empirical results on their dataset using their suite, and make a number of recommendations based on these findings. Reproducible software is also provided.