Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors extend the out-of-distribution (OOD) detection from not-seen in small-scale settings to large-scale multiclass and multi-label ones. They provide large-scale benchmarks for evaluating ODD detectors on classification as well as segmentation. Additionally, they propose a simple yet strong baseline for this practical problem.
main_review: Strength:
1. The paper is well-organized and easy to understand.
2. Out-of-distribution is a critical problem, the authors establish several large-scale benchmarks on classification and segmentation to facilitate future research.
3. The proposed MaxLogit is practical and the experiment results on large-scale benchmarks demonstrate its consistent efficacy.

Weakness:
1. I like the large-scale benchmarks constructed in the paper. But the proposed solution (MaxLogit) is just a simple extension of MSP. I cannot even tell it is a new method even though it can get significantly better results.
2. One interesting conclusion of the paper is the vision transformer-based method does not make OOD detection (on large-scale benchmarks) trivial. I would like to see more analysis and reasons behind this.
3. The benchmarks are interesting, but the comparison baselines are not sufficient. Please include more SOTA OOD results like [2, 3] (see more in [1]).


[1] Yang, Jingkang, et al. "Generalized Out-of-Distribution Detection: A Survey." arXiv preprint arXiv:2110.11334 (2021).
[2] Di Biase, Giancarlo, et al. "Pixel-wise Anomaly Detection in Complex Driving Scenes." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.
[3] Lis, Krzysztof, et al. "Detecting the unexpected via image resynthesis." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2019.
summary_of_the_review: Overall, the authors study an practical problem at scale. They establish a large scale benchmark and propose a simple yet efficient approach for OOD detection. But they need to study more SOTA baselines. I would like to give an initial positive score and keep tuning during discussion period.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper presents a collection of somewhat disjoint contributions to outlier detection. First, the authors propose Species - a novel OOD test dataset. The main advantage of this dataset is being disjoint from ImageNet-22k. Second, the authors propose to detect outliers according to the max-logit criterion. The authors claim that max-logit is especially suitable for OOD detection in multi-label environments. Third, the authors propose two novel datasets for dense outlier detection. StreetHazards is especially interesting since it allows proper rendering of introduced outliers. 
main_review: Strengths

S1. Species may be a valuable tool for evaluating OOD detection approaches

S2. StreetHazards is recognized as a valuable tool for evaluating dense outlier detection in several pieces of previous work.

Weaknesses

W1. max-logit has not been compared with its differentiable counterpart (log-sum-exp logit). Previous work shows that logsumexp logit can be interpreted as likelihood of the input [gratwohl00iclr] and that it can be used as a principled criterion for outlier detection [liu20nips]. A comparison with standardized max-logit [jung21iccv] would increase the value of the manuscript.

W2. experimental evaluation leaves to desire:
- Tables 1 and 2 do not contain recent baselines such as ODIN, generalized ODIN, logsumexp logit, standardized max-logit
- Table 3: it is unclear why do max-logit and max-sigmoid-logit deliver such different rankings
- Table 4 does not contain recent baselines, cf. [grcic21visapp] and the references within
- Figure 5 should report AP and FPR95 (the official benchmark metrics) in order to allow comparison across the leaderboard; it would be a good idea to include these metrics into all other tables since they are also used in Segment Me If You Can. 

W3. BDD anomaly has a poor choice of the OOD class since parts of trains can occur on buses and parts of motorcycles can occur on cars. Thus many OOD pixels get very low OOD scores. Such occurences devastate the AP score and cause all approaches to come out equally bad. Thus, Fishyscapes, Segment Me If You Can and Wilddash 2 present better opportunities for dense OOD detection

Suggestions
- some details are missing or unclear: which Pascal VOC (2007 or 2012?), why does the training use a small batch size so that batchnorm has to be freezed, which batch size has been used, why does Table 2 show per-category results. 
- regarding "the object’s exact class is difficult to determine": this problem also arises at semantic borders when performing dense OOD detection [grcic21visapp]
- it would be a good idea to describe WildDash 2

[jung21iccv] https://arxiv.org/abs/2107.11264

[gratwohl00iclr] https://arxiv.org/pdf/1912.03263.pdf

[liu20nips] https://arxiv.org/pdf/2010.03759.pdf

[grcic21visapp] https://arxiv.org/pdf/2011.11094.pdf
summary_of_the_review: Creating datasets requires a lot of hard work. Yet, the weaknesses outweigh advantages. Hence, I must conclude that this manuscript is below ICLR acceptance threshold.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The work designs three large-scale settings for evaluating out-of-distribution detection and performs empirical evaluation on the settings to establish some baselines for future work.
main_review: Strengths.
+ Three three large-scale OOD settings are presented, together with datasets for empirical evaluation
+ A set of experiments is given on all three settings to provide baseline for future work
+ A anomaly scoring method called MaxLogit, which is a minor variant of the popular method MSP, is introduced, and shows effective performance on the three settings

Weaknesses.
+ Several key claims of the paper are incorrect. For example, rather than what it is claimed in the paper about small-scale multi-class OOD detection, there have been increasing efforts put on large-scale multi-class settings, such as [1-2], but the paper fully ignores those closely related work; in terms of large-scale anomaly segmentation settings, the datasets and tasks introduced in [3] clearly have more advantageous features (dataset scale, diversity of anomalies, realistic of the settings, etc.) than the proposed one in this work.
+ All the proposed datasets are created by some simple combination of existing benchmarks, wherein I cannot find any major contributions
+ The presented method MaxLogit is a trivial variant of the popular method MSP, so no major technical novelties are presented
+ The competing methods across all three settings are outdated models or simple baselines. Without comparison to recent, state-of-the-art models, it is difficult to evaluate how the presented method advances the area.

References\
[1]  "Are out-of-distribution detection methods effective on large-scale datasets?." arXiv preprint arXiv:1910.14034 (2019).\
[2] "MOS: Towards Scaling Out-of-distribution Detection for Large Semantic Space." In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8710-8719. 2021.\
[3] "SegmentMeIfYouCan: A Benchmark for Anomaly Segmentation." arXiv preprint arXiv:2104.14812 (2021).
summary_of_the_review: Overall, the paper presents some good empirical results of large-scale OOD detection in three settings, but the work ignores a number of closely related studies and it is weak in terms of both technical novelty and empirical evaluation, thus not good enough for publication at ICLR.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper presents the negative of the maximum unnormalized logit (MaxLogit) as an anomaly score for out-of-distribution (OOD) detection. Also, it introduces a large-scale setup for ODD. The proposed metric shows promising results compared to the maximum softmax probability (MSP) in the proposed setup (in-distribution ImageNet-1K and out-distribution Places365). For the multi-label experiment, the PASCAL VOC and MS-COCO are in-distribution and ImageNet-22K out-distribution. The proposed MaxLogit works better than in this MSP too. Finally, the proposed metric shows promising results in the  CAOS benchmark.
main_review: Paper strengths:

+ The paper spots the reliability issue of the maximum softmax probability (MSP) on large-scale datasets.

+ The paper addresses a limited explored problem, large-scale OOD.

+ The paper is well-written and easy to follow. 

+ The proposed anomaly MaxLogit score (negative of the maximum unnormalized logit for an anomaly score) leads to good performance for large-scale OOD.


Paper weaknesses: 

- The MaxLogit score is nowhere formally defined. It's straightforward how is computed but a formal distribution is necessary since it composes a major contribution to the paper. 

- It's not clear whether the proposed setup is near- or far-ODD setup. This claim could be well-supported by a corresponding analysis.

- The paper focuses only on the maximum softmax probability metric for the evaluations. However, there are other methodologies such as ODIN (Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networks, 2018), Mahalanobis distance (A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks, 2018) or OECC (Outlier Exposure with Confidence Control for Out-of-Distribution Detection, 2021). It is important to include one more baseline for comparisons. 

- Multi-label OOD: There is already the setup with ImageNet-1k as out-of-distribution (A benchmark for anomaly segmentation, 2019). It would be more helpful to follow an existing protocol next to the proposed one.

- Small-scale evaluation: it would be useful to see how the proposed metric performs on small-scale evaluations. There is not a single standard evaluation on this setup although approaches from the small-scale evaluation are used, e.g. MSP.


Improvements:

- "In contrast to medical anomaly segmentation and fault detection, we consider complex images from street scenes." -> This argument could be reformulated since medical imaging is quite challenging.
summary_of_the_review: Overall the paper proposes a metric and a large scale setup of out-of-distribution detection. Both contributions are valuable and can be considered novel too. There are few a points to be addressed in the paper structure, as discussed above. The major issue is to add more related approaches in the comparisons. Moreover, the small-scale evaluation would be useful to show the generalization of the proposed metric to different types of setups. Even if MaxLogit does not deliver state-of-the-art results on a small scale, it would still be important to show the results and interpret the outcome.


Post-rebuttal: 

The rebuttal showed that there are still changes to be performed in the paper. It would need further work before acceptance.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This work explores out-of-distribution (ODD) detection in three large-scale settings: multi-class OOD detection, multi-label OOD detection and anomaly segmentation. To facilitate large-scale experiments, it introduces a novel species dataset and a road anomaly dataset for multi-class OOD detection and anomaly segmentation respectively. It also demonstrates a new setup for multi-label OOD detection. In addition, this work establishes a new baseline via a simple detector based on the maximum logit in all the three large-scale settings.
main_review: Strengths:
1. This paper is well motivated, which aims to push out-of-distribution (ODD) detection from small-scale settings to large-scale and real-world settings.
2. This works seems solid, which explores three large-scale settings and introduces two new datasets.
3. The proposed method (MaxLogit) shows good performance in all settings.  

Weaknesses:
1. This manuscript is not very well written, some contents of this manuscript are not clearly organized. It would be better to list contributions for three different settings, and introduce datasets and methods/findings more separately.
2. The proposed method seems simple, but it's still not very clear how it works in three settings. More specific descriptions of the method would be better.
summary_of_the_review: This paper is well motivated and solid, but the writing needs to be improved.
review_title: null

Meta-review: The authors focus on large scale out-of-distribution (OOD) detection for which they propose three benchmarks with multiclass and multi-label 
high-resolution images. In these settings, they find that a simple extension, using maximum logits (MaxLogit), of a common baseline  maximum softmax probability (MSP), is surprisingly competitive to prior methods.

Five knowledgeable reviewers found the idea of having these novel benchmarks potentially interesting, but highlighted some issues that needs to be taken into account before the paper can be publishable. 

First, reviewers highlighted how the presentation can be better organized (more structure on and stronger overall motivation for the three different contributions) as to present the three ideas in a more cohesive way and more formal in introducing methods (e.g. the MaxLogic and MSP) as to clearly highlight the technical contributions and the differences with other models. 

Second, more baselines need to be introduced and therefore experiments extended. For example, the comparison with another detector, LogSumExpLogit, a relaxation of MaxLogit already used for (small-scale) OOD in the context of generative models. Authors provided in the rebuttal some preliminary experimental results but promised more (for a camera-ready) that could not be evaluated by the reviewers.

Third, the scope of the proposed benchmarks raised some concerns by some reviewers. If not in the motivation behind the task of treating whole objects as anomalies, additional care shall be put into the provided annotations. As one reviewer highlighted a certain percentage of images are mis-annotated. While this percentage is somehow low (3.9%) and should not change the empirical conclusions drawn in the paper, it highlights that the core contributions of the paper might have been rushed.
Decision: Reject