What classifiers know what they don't know?

Abstract: Being uncertain when facing the unknown is key to intelligent decision making. However, machine learning algorithms lack reliable estimates about their predictive uncertainty. This leads to wrong and overly-confident decisions when encountering classes unseen during training. Despite the importance of equipping classifiers with uncertainty estimates ready for the real world, prior work has focused on small datasets and little or no class discrepancy between training and testing data. To close this gap, we introduce UIMNET: a realistic, ImageNet-scale test-bed to evaluate predictive uncertainty estimates for deep image classifiers. Our benchmark provides implementations of eight state-of-the-art algorithms, six uncertainty measures, four in-domain metrics, three out-domain metrics, and a fully automated pipeline to train, calibrate, ensemble, select, and evaluate models. Our test-bed is open-source and all of our results are reproducible from a fixed commit in our repository. Adding new datasets, algorithms, measures, or metrics is a matter of a few lines of code-in so hoping that UIMNET becomes a stepping stone towards realistic, rigorous, and reproducible research in uncertainty estimation. Our results show that ensembles of ERM classifiers as well as single MIMO classifiers are the two best alternatives currently available to measure uncertainty about both in-domain and out-domain classe.

Reviews and author responses:

The paper introduce a benchmark UIMNET to evaluate predictive uncertainty estimates for deep image clasifiers. The authors provides implements of ten state-of-the-art algorithms and six uncertainty measures with four in-domain metrics and three out-domain metrics. For Strengths, the paper provides a solid test-bed for evaluating the uncertainty estimation for deep image classifiers. Also, the paper is well-written and both the data construction and results part are clearly described.
The main weakness of the paper is that the uncertainty measurements in Section 4 lack the SOTA uncertainty estimation methods. For example, no method in Section 4 measures the epistemic uncertainty such as deep ensemble/dropout and aleatoric uncertainty [1] explicitly. Also, recent dirichlet-distribution-based approaches [2][3] are not included to measure the evidence-based uncertainty. 

[1] Kendall, Alex, and Yarin Gal. "What uncertainties do we need in bayesian deep learning for computer vision?." arXiv preprint arXiv:1703.04977 (2017).
[2] Sensoy, Murat, Lance Kaplan, and Melih Kandemir. "Evidential deep learning to quantify classification uncertainty." arXiv preprint arXiv:1806.01768 (2018).
[3] Charpentier, Bertrand, Daniel Zügner, and Stephan Günnemann. "Posterior network: Uncertainty estimation without ood samples via density-based pseudo-counts." arXiv preprint arXiv:2006.09239 (2020).
 Overall, I think the paper is well-written and the benchmark is valuble as testbed for uncertainty estimation. But the uncertainty measurement part lacks some important uncertainty estimation methods, which are highly encouraged to be included in the paper.

This paper proposes the UIMNET benchmark for uncertainty estimation. This benchmark includes (1) "ImageNot", a remix of ImageNet using hierarchical clustering of pairwise distances between features (2) framework for evaluating uncertainty estimation, including a suite of algorithms, metrics, and ablations studies. The authors report empirical results on their dataset using their suite, and make a number of recommendations based on these findings. Reproducible software is also provided. ##########################################################################

Pros: 
 
* Uncertainty estimation is an important topic.

* I commend the authors for their efforts on reproducibility.

##########################################################################

Cons: 

* The novelty of the research is low. The data source, algorithms, and metrics are all prior work. 

* What value will the community get from the release of ImageNot? The authors have already reported their results. Can the authors outline some additional use cases? Otherwise there is not much point in branding it as a "new dataset".

##########################################################################

Questions during rebuttal period: 


* The authors claim in/out domain data should be defined from the same dataset, and that "this provides realistic out-domain data", as opposed to the alternatives (different datasets, augmentations). What is the justification for this claim of greater "realism"? 
 
* How extensible is the software framework to new datasets? What would I have to implement to run the framework on a new dataset? New empirical benchmark with reproducible software but with unclear research novelty and value to community. 

This paper proposes a new testbed to test the uncertainty of image classifiers based on a split of ImageNet. Strengths:
- The paper is well written.
- The benchmarking is solid and very thorough. Many baselines and metrics have been implemented, and results are presented with confidence intervals.

Weaknesses:
- Overall, the major weakness is in the design of this benchmark. For a study claiming to be "the most exhaustive uncertainty estimation benchmark," the only test-time distribution shift considered is the presentation of novel classes. The authors do not appear to justify this design decision anywhere in the paper. The authors make multiple appeals to "providing realistic out-domain data," but there are many, many ways out-domain data can be presented in real world applications. This disjoint class assumption is very strong and is not clear why or when this is representative.
- From reading about the construction of the benchmark, it's not entirely clear why BREEDS wasn't a suitable candidate and why something new had to be created. The authors state that the BREEDS task is to classify super-classes and so the label set remains fixed from train to test - that is true, but could the authors not have just used the same split and kept the leaf node original class labels instead of using the superclass labels?
- The agglomerative clustering method used to construct the splits seems round-about. The ImageNet hierarchy contains a lot of metadata that could have been used to create the splits - do the authors have any comment on why this wasn't used? Also, the result of this split is that the train-test gap is over object classes to animal classes, which is perhaps (a priori) not as realistic of a setting as in the BREEDS dataset, where the train-test gap is between very similar class categories. This design decision was again never justified. In summary, this study has a large number of design flaws. However, given the benchmark, the authors have done a great job in thoroughly benchmarking existing algorithms.

Meta-review: This paper introduces an ImageNet-scale benchmark UIMNET for uncertainty estimation of deep image classifiers and evaluates prior works under the proposed benchmark. Two reviewers suggest reject, and one reviewer does acceptance. In the discussion period, the authors did not provide any response for many concerns of reviewers, e.g., weak baselines, weak novelty, and lack of justification for the current design. Hence, given the current status, AC recommends reject.

Decision: Reject