Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper presents a benchmark for out-of-distribution detection for ImageNet(-1K) models, using data from ImageNet-21k as out-of-distribution samples. The work presents a strategy for evaluating OOD detection, avoiding pitfalls encountered in prior work. Additionally, the authors evaluate a wide array of ImageNet models, and highlight findings on which models and confidence functions outperform others.
strength_and_weaknesses: Strength:

- The authors point out a key issue in prior work on evaluating OOD detection: when OOD datasets are constructed by filtering to be difficult for one model, the bias in filtering results in other models naturally outperforming the original model. 

- The authors evaluate a large array of models on their data, in contrast to prior work, and find interesting conclusions about OOD detection (e.g., that CLIP models outperform models trained on ImageNet)

Weaknesses:

- The notion of severity levels needs to be motivated further. The paper states that the proposed method of computing severity levels is better than prior work – but why do we need severity levels? A plausible alternative metric is to simply compute AUROC on the entire OOD dataset, without the severity levels. Why is this metric insufficient? What conclusions would change if you were to use this metric?
    - If the AUROC overall metric is sufficient, the severity levels can still be used as a diagnostic. It’s unclear that one needs to look at AUROC at each level individually when comparing models.
- While interesting, the result about CLIP’s OOD detection needs further explanation and comparison. The paper claims it outperforms “96%” of other models. But this is a strange metric: the 96% presumably includes models like AlexNet. Using this metric, the ViT-L/32-382 outperforms “99.99%” of other models. It would be more interesting to compare a CLIP model head-to-head with a model with the same architecture, trained with different training strategies, such as training on ImageNet-1k (or pretraining on IN21k then finetuning on IN1K).
- Notation could be more clear – it’s still unclear to me exactly how AUROC is computed. 
    - In Eq (1), what is \hat{y}? Is it the prediction of f on x1, or on x2? Is it the same when computing k(x1, \hat{y} | f) as k(x2, \hat{y} | f)? I assume not.
    - In Section 3, k is changed from k(x1, \hat{y} | f) to k(x | f). How is this defined?
    - Is AUROC computed per OOD class, then averaged across classes within a severity level? Or are samples for all classes in a severity level merged together, and then AUROC is computed once for the severity?
- How is ID AUROC defined?
    - Do you compute AUROC per class, with samples from all other classes treated as negative, and then averaged?
- Nits / questions:
    - It would be useful to explain why ODIN, entropy and MC dropout specifically were chosen for evaluation in the work.
    - Fig 6 – what severity level is this evaluated at?
    - Sec 5, (1) – is this on all 525 models?
    - Sec 5, (1) – any ideas why imagenet 21k training hurts OOD detection?
    - Would it be possible to evaluate the effect of removing softmax at test time, and re-evaluating? Depending on how AUROC is defined (see my questions above), this may or may not change the results. This may also allow a more fair comparison between ImageNet models (which usually have a softmax) to CLIP models (which are just using cosine similarity without a softmax). Relatedly, evaluating the impact of the softmax temperature on OOD detection would be interesting. These experiments are not necessary for acceptance, but I would appreciate the authors’ thoughts on the impact of the softmax – it will also help me understand the evaluation better.
    - If possible, it would help to share the AUROC for all models at all severity levels in a table in the supplementary, or in a webpage


clarity,_quality,_novelty_and_reproducibility: - Clarity: Please see weaknesses section. There are some key clarity questions that need to be resolved.
- Quality / novelty: No concerns.
- Reproducibility: Will the authors commit to publicly releasing code for evaluating all models, along with the AUROC per severity level for each model plotted in the paper? This will significantly help reproducibility.

summary_of_the_review: Overall, this is a good paper, with some issues in presentation and clarity. I have raised some questions about the presentation and clarity above, and particularly about the use of severity levels in the evaluation. Assuming these are addressed, I would vote to accept the work.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper presents a method for benchmarking models (combined with a dataset and a confidence function $\kappa$) for their ability to recognize out-of-distribution examples at test time. In particular, the paper considers the OOD case of when the model is presented with an example with a label that that model had not seen during training. The paper uses the methodology to evaluate 525 different models as well as some different confidence functions to produce some observations about different types of models and about the OOD task generally.
strength_and_weaknesses: The paper well grounded in previous works is attacking a significant problem of interest to the community and has a very extensive empirical validation backing up its claims/observations (which are numerous). The most interesting aspect of the paper for me was the observations and claims about OOD detection, such as the performance of vision transformer models or how knowledge distillation helps with detecting OOD samples. I believe these types of observations are both important for practitioners and also as starting points for additional research.

The paper’s only real weakness is in its clarity, and, possibly its correctness. There are some points in the paper that need further explanation. Specifically

-	In section 2, how is AUC used to measure OOD performance? It seems like there is some cut-off score for $\kappa$ below which the paper assigns the value of OOD versus ID, and then uses these assigned values to compute AUC. If so, what is the cut-off value? Is it different for different $\kappa$’s or models? 

-	How does one make apples-to-apples comparisons between models for the OOD performance? As is mentioned in the paper (section 5, point 6) OOD hardness is subjective and the proposed method will have different classes in different hardness ratings for different model + $\kappa$ configurations. This would lead me to believe that one cannot really conclude that one model is better than another in OOD detection as data breakdowns between the two models are not the same. Is there a way to overcome this limitation or otherwise have a comparison between two models on their OOD performance where the only difference is the model itself? Or, is this not a problem for comparing models?

-	A minor point, but could you add the web addresses, probably as footnotes, for timm and torchvision for total clarity on the packages?

clarity,_quality,_novelty_and_reproducibility: The paper is high in quality and reproducibility. Then issues with its clarity are fixable and not serious (stated previously in limitations). In terms of the novelty of the paper, while the paper is not the first to address this idea of evaluating models for OOD performance, it does propose a significant advancement in this line of research.
summary_of_the_review: Sometimes the best papers are those papers that raise more questions than they answer. This is one of those papers. While the technical contributions are not especially monumental or flashy, the observations coming from the solid empirical work in the paper are very intriguing, especially from a practicing data scientist lens. As such, I believe this paper does merit acceptance and publication.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a benchmarking framework for class-out-of-distribution detection with various levels of detection difficulty. This work benchmarks this technique's application to ImageNet with 525 publicly available pre-trained ImageNet-1K classifiers. Based on this benchmark, the authors identify several trends in out-of-distribution detection robustness and serve as a foothold for future research.
strength_and_weaknesses: Strengths:

S1. This work provides a new benchmarking framework to detect class-out-of-distribution instances with different levels of difficulty. This opens new avenues for future OOD detection research.

S2. The authors analyze the results of this benchmarking which lead to numerous interesting observations.

S3. This paper presents a good viewpoint on designing large-scale objective benchmarks with different levels of detection difficulty for class-out-of-distribution, which is useful for real-world applications.

Weaknesses:

W1. It is not clear how and the intuition why a subset of ViTs achieves the best C-OOD detection performance.

W2. The computational complexity is not discussed when analyzing baseline models, e.g., knowledge distillation, and language-vision CLIP.


clarity,_quality,_novelty_and_reproducibility: The quality of this paper is good. The authors present a new benchmarking framework for class-out-of-distribution detection.
summary_of_the_review: This is a benchmarking paper focusing on an important problem. The contributions of the paper will have some impact to the OOD detection community.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proses a methodology for evaluating class out-of-distribution detection with various levels of detection difficulty that is model-specific. The main contributions are observations, a.k.a best practices, for OOD benchmarking that are thoroughly validated in an extensive experimental setup. AC agrees with the reviewers that the contributions of the work are important for practitioners and have potential to spark further research in this area. We urge the authors to include the clarifications of the rebuttal into the manuscript, as well as the empirical results on the effects of not applying softmax. Congratulations to the authors!
justification_for_why_not_higher_score: A spotlight recommendation will help to bring attention to the findings in this work, which is aimed at practitioners in this area. 
justification_for_why_not_lower_score: Extensive discussions with the reviewers demonstrate a potential of fruitful discussions at and after the conference, which is great to have. AC agrees with this summary provided by one of the reviewers: 'Sometimes the best papers are those papers that raise more questions than they answer.  And this is one of those papers.'
Decision: Accept: notable-top-25%