Venue: NeurIPS 2022 Track Datasets and Benchmarks

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This paper introduces a benchmark for three related research topics, including AD, OSR, and OOD. Experimental settings are specified for each topic. In total, 31 methods are evaluated. This work facilitates a unified comparison of AD, OSR and OOD algorithms.
strengths: (1) Overall a well-written paper and easy to follow.

(2) The paper introduces a benchmark for three related research topics, including AD, OSR, and OOD, facilitating a unified comparison of AD, OSR and OOD algorithms.
weaknesses: (1) I suggest to include a specific definition of AD, OSR and OOD and a detailed discussion on their similarities/differences before introducing the benchmark. 

(2) Compared to the OOD methods, algorithms in AD and OSR are not updated. Specifically, knowledge-distillation based methods such as MKD (CVPR'21) [2]  and reverse distillation (CVPR'22) [2] have reached SOTA performance in AD. Similarly, a task-adaptive negative class envision method (CVPR'22) [3] is proposed to tackle OSR. I highly suggest to include these methods in the benchmark.

(3) I doubt that Cifar-10 and Cifar-100 could be used to build a near-OOD benchmark. Cifar-10 and Cifar-100 contain identical image content but with different label information. In my opinion, it doesn't follow the definition of OOD.

(4) It is good to have a public open-source benchmark. But it neither provides instructions for benchmark usage nor presents demos for experimental result reproduction. It is highly suggested to include specific documentation on the benchmark usage.

Ref:

[1] Mohammadreza et al., Multiresolution Knowledge Distillation for Anomaly Detection, CVPR 2021.

[2] Hanqiu et al., Anomaly Detection via Reverse Distillation from One-Class Embedding, CVPR 2022.

[3] Shiyuan er al., Task-Adaptive Negative Class Envision for Few-Shot Open-Set Recognition, CVPR 2022.

correctness: Yes.
clarity: Well-written.
relation to prior work: Yes.
additional feedback: Please refer to the weakness section. Thanks.
review_title: Good Benchmark, but some weakness

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: - This paper proposes an open-source unified OOD detection codebase with 9 benchmarks.
- This paper makes comprehensive comparison across different OOD detection tasks, including anomaly detection , open set recognition and model uncertainty.  
- This paper obtains some valuable findings using the proposed codebase.

strengths: - 9 OOD detection benchmarks are provided in the codebase OpenOOD.
- The proposed codebase supports 33 methods originated from AD, OSR, OOD detection and model uncertainty.
- The OpenOOD abstracts a unified pipeline to meet the requirements of different types of methods.
- Extensive experiments are conducted to obtain some interesting results about model uncertainty, extra data, Post-Hoc etc.
- The source codes and dataset have be released.

weaknesses: - The metrics of FPR@5 and AUPR are introduced in the paper, but the experimental results in terms of these metrics are not reported.  Are their results same as ones with  AUROC?
-  Although the codebase contains 33 methods, but it seems to be not enough to discuss on the characteristics of these methods according to experimental results.
- In the experiments, every method only run one time.


correctness: The nine benchmarks are well evaluated using over 30 methods.
clarity: The paper is well-written and easy to follow. 
relation to prior work: The related work has been systematically discussed, and the contributions of this paper are obviously different compared with existing researches. 
additional feedback: The benchmarks include four types of methods, including classification-based, density-based, distance-based and reconstruction-based. If possible, it would be better to discuss the advantages of the four categories according to the experimental results, respectively.
review_title: Review for OpenOOD: Benchmarking Generalized Out-of-Distribution Detection

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This paper compares more than 30 algorithms on 9 datasets, including one anomaly detection (AD) dataset, four open set recognition  (OSR) datasets and four OOD datasets. Based on comprehensive experiments, the authors draw several insightful conclusions, which are regard to be helpful for the OOD community.
strengths: Algorithms
- The authors propose a comprehensive benchmark called OpenOOD, which intergrates more than 30 cutting-edge OOD algorithms. Collecting so many DL-based methods could be served as a contribution for the OOD community, where both researchers and practitioner can quickly implement algorithms on the newcomming dataset and compare their performance.

Datasets
- The authors propose 9 datasets, i.e., one anomaly detection (AD) dataset, four open set recognition  (OSR) datasets and four OOD datasets), these datasets could be regarded as a good testbed for evaluating newly proposed OOD methods.

Framework
- OpenOOD is fully integrated as a pipeline for providing more convinient task processing. The OpenOOD pipeline include Dataloader——Preprocessor——Network——Trainer——Evaluator——Postprocessor——Tools, where each steps include the currently popular OOD methods, such as the GradNorm method in the Postprocessor step.

Experimental Results
- Based on the OpenOOD, the authors draw several interesting conclusions in Section 5.2. For example, "Extra Data Seems Not Necessary" is superies but meaningful observation, since collecting additional datasets for OOD problem could be laborious.
weaknesses: The weaknesses of this paper are mainly two points:
- The computational cost makes the model runtime only once (mentioned in the Weakness subsection), this could lead to a certain extent of randomness in the current experimental results. Besides, statistical test should be considered for model comparison, not just the absolute difference of model performance.
- Currently the experimental results tell the readers: which OOD algorithms are better for different OOD tasks? The authors could provide more perspectives about: which OOD algorithms are better for a specific type of data? Besides, the authors could further provide some explainable results for understanding the inherent architecture of different OOD algorithms.
correctness: The claims made in this paper are correct.
clarity: Yes
relation to prior work: Yes
additional feedback: No
review_title: Generally Good Work for OOD Task

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary and contributions: The paper proposes a new benchmark for out-of-distribution detection that generalizes over datasets and methods from the anomaly detection, open set recognition, and out-of-distribution detection communities. They construct a number of  datasets from existing image classification databases and evaluate around 30 methods.
strengths: The ability of a model to recognise data samples that deviate from the training distribution is an important real-world problem. The research community is in need of a unifying benchmark dataset that can reliably assess the performance of OOD methods and that can serve as a standardised evaluation tool. The authors evaluate a large number of well-selected methods from various domains.

weaknesses: 1. The majority of included datasets are of very low-resolution (i.e. MNIST, CIFAR10/100, TinyImageNet). In real-world applications, most classification problems do not operate on such low-resolution inputs. Hence, it is not clear how well the results obtained on this benchmark transfer to many other real-world problems.

2. The benchmark only covers a few very specific settings for which no motivation is given. For example, MNIST and CIFAR 10 are split into ratios of 6/4, but there is no clear motivation for doing so in the paper.

3. The description of the codebase structure is insufficiently detailed. Key components in the pipeline in Figure 3, such as the trainer component, are only roughly outlined in the paper. Some of the displayed components, e.g., the MOSTrainer, OETrainer, etc. are not mentioned in the paper. This may be due to space limitations, but then I would suggest to remove these descriptions altogether.
correctness:  
clarity: The paper is written in an understandable way. However, there are still numerous grammar issues. When fixed, this would improve the readability of the paper.
relation to prior work:  
additional feedback:
review_title: Review for OpenOOD: Benchmarking Generalized Out-of-Distribution Detection

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This paper proposes a benchmark designed to provide a quantitative analysis complement to a previous
survey paper by the same authors, which studied OOD detection and related fields to build a unified and comprehensive research framework called
Generalized Out-of-Distribution Detection.

The authors consider 9 different benchmarks from 3 related research fields (AD, OSR, OOD detection)
and evaluate the performance of over 30 methods on them, considering approaches for the
different tasks in an attempt to build a comprehensive picture of the state of the art in the
Generalized OOD detection framework. 

Contributions include: 

 - a large benchmark based on 9 different benchmark tracks 
 - a large-scale comparison among methods designed for different but related fields. Results
   analysis allows to better understand which is the current state-of-the-art and how methods based
   on extremely different strategies compare;
 - an open source codebase which implements the tested methods and allows to easily implement
   additional ones for future research. 

strengths: Paper strengths: 

 - the main strength of the paper, which corresponds in my opinion with the most important
   contribution is the open source codebase that the authors have built and released as part of this
   work. According to the authors, the code should not only allow replicating all the paper's
   results, but it should also be easily extensible in order to represent an extremely suitable
   solution to start developing new algorithms to study Generalized OOD detection tasks;
 - the presentation of a benchmark tackling OOD detection (and related fields) is certainly timely.
   Existing literature in these research fields has progressed in parallel streams without many points of
   contact;
 - the paper's motivations are well presented;
 - it is clear from the entire paper that the research work has been huge, considering various
   fields, a very large number of benchmark tracks and methodologies;
 - the proposed insights, inferred from experimental results, are quite interesting, showing that
   often more complex or more computationally expensive approaches do not provide better results in
   practice when evaluating on a large number of settings. 

weaknesses: Main high level weaknesses:

 - the most important problem of this work is that the authors' target was too high, considering too
   many methods and settings. As a result, the time and space to effectively present everything were
   probably not enough. The main consequence is that the paper has obvious writing problems, and in 
   many places the choices of what to describe in detail and what not to describe were wrong. Just
   to make an example many of the considered methodologies are not described well enough so that the
   reader cannot understand how they are executed (e.g. is the considered ARPL [5, 6] version the
   ECCV20 one or the TPAMI 21 one? Is the CSI [56] the completely self-supervised version or the
   supervised one? These pieces of information are quite important, given that results of these methods in various versions
   may certainly change a lot). Authors could have probably saved some space discarding some less
   useful discussions/descriptions like the detailed code presentation of Sec 4;
 - the experimental evaluation could benefit from a better and clearer organization. All the paper's
   experimental results are provided in a single very large table whose purpose is to allow easy
   comparison among all methods, but that at the same time pushes the reader to compare numbers
   which are not really comparable. For example some methods use additional auxiliary information
   and therefore should not be included among the others. This is the case for example of MOS which
   uses hierarchical categorical information. Also methods based on OE, even if they are grouped in
   a separated part of the table should not be compared with the others as the choice of the
   training time OOD data may highly influence their performance and thus the numbers reported here
   may not be really relevant. Moreover the "Avg" column is certainly misleading. Indeed even if 
   the authors do not refer to it in the text the fact that it appears in the table pushes
   the reader to compare numbers for papers of different groups, which cannot be done considering
   that not all methods are tested on all benchmarks. 

Other weaknesses:

 - there are some inconsistencies between the figures/tables and text. For example, section 3
   describes two methods (KL mathing, AugMix) which did not appear in the schematic organization of
   Figure 2, which on the other hand considered an unknown "KDAD" method. Moreover, while KL matching
   appears in Table 1, this is not the case for AugMix, which is substituted by CutMix;
 - the citations in Table 1 are not consistent in format with the others, it is therefore not easy
   to link a method name to the corresponding paper in the references;
 - Table 1 does not contain results for many combinations of methods and benchmarks. While there are
   undoubtedly some reasons behind these choices some of them may be not obvious to the reader, the authors
   should justify them. Why, for example, have the AD methods been run for all benchmarks, including the OSR
   and OOD detection ones, but not for ImageNet? Why are PatchCore results limited to MVTec? 
 - there are some questionable choices. For example, the inclusion of the MVTec benchmark is not
   useful considering that only AD methods have been tested on it. Moreover, the methods using OOD
   data at training time have been provided with a strange choice for them: by the authors' admission
   TinyImageNet is not purely OOD and only UDG has been designed to deal with this situation, so why
   did they choose this dataset? 
 - in general, the choices behind some benchmarks are not completely sound. In particular, I do not
   find it useful to consider far OOD cases in which there is a covariate shift between ID and OOD data
   in the test set, it makes the problem trivial and the results useless. 
 - besides AUROC sec 2.4 describes 2 additional metrics (i.e.: FPR@95 and AUPR), where are the
   numbers for these metrics? There is no supplementary material and the github repo does not
   include them. Moreover even results for all individual experiments should be provided somewhere, for
   example with MNIST Near OOD there should be two experiments (NOTMNIST and FashionMNIST), is the
   reported number an average of them? Or have they been merged for the experiment as a single OOD
   set?
correctness: The main correctness problem rely in the fact that, as I said above, some methods should not be
included in the same table with the others as they use additional information. 
clarity: The paper is clear, however the writing could be improved a lot. There are errors, typos and
repetitions which should be fixed. 

Examples: 

 - "Diagram of benchmarks that supported by OpenOOD" in the caption of Fig. 1; 
 - "In the main paper, we use AUROC as the main metric in the paper." last line of sec 2.4; 
 - "Methods that we include in this category are all require training, except temperature scaling"
   last line of Sec 3.4 
 - "We run all the 33 methods that supported by OpenOOD" first line of sec 5
 - "Although AD methods was originally" and "OSR Benchmarks Aligns with OOD" in sec 5.2.

By the way, submissions under review should include line numbers so that the reviewers can more
easily highlight specific problems in the text. 

relation to prior work: The relation with prior work is clear. Previous works did not try to unify different but related research fields
under the same framework which has long be due. This paper provides this unification and effectively
allows to build a coherent picture and find out where the state of the art really is. 

additional feedback: I do not find the whole Sec 4 to be really useful (including figure 3). Given the limited amount of space in the paper
and even if the code is an important contribution of this research work I would have preferred a
shorter description of the implementation (maybe as part of sec 5.1) in order to save some space to
be dedicated to additional details about the tested methodologies or additional experimental
results/analyses. 

Overall I'm recommending acceptance of this work, as I really think that its contribution in terms of the benchmark, research fields unification, and especially codebase are important, however I would have appreciated a better presentation.
review_title: A poor presentation of a huge research work.

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: This paper organizes several OOD detection-related tasks, including OOD detection, AD, OSR, etc. By releasing a codebase containing methods, evaluation, and benchmark tasks, the authors undoubtedly contribute significantly to the machine learning community in an open environment. Finally, the authors comprehensively compare methods and summarize this field's progress over the past few years. 
strengths: * This codebase contains 30+ OOD detection-related methods and divides these methods into categories on several dimensions. This codebase helps the researchers in this community, especially newcomers, fully get in touch with OOD detection.
* This codebase release an OpenOOD pipeline that contains many OOD detection-related benchmarks. This helps the following researchers fairly compare different methods on different tasks. 
* With the help of this unified pipeline, the authors give insightful comparison results between tasks inside one single task and tasks among different tasks.
weaknesses: * This codebase lacks documents right now (at least there is no intuitive link for documentation). Nevertheless, the given code can still provide enough reproducibility.
* This codebase mainly focuses on the performance of the model and ignores the training costs, which are also crucial metrics of different methods.
correctness: This submission is a benchmark, and the experiment design and evaluation are appropriate.
clarity: The paper is thoughtfully written and easy to follow.
relation to prior work: This work relates to the prior work "Generalized Out-of-Distribution Detection: A Survey," which was also developed by this team of authors. This work focuses on the codebase and experiments, yet the prior work focuses on the comprehensive survey of this field.
additional feedback: N/A
review_title: Review for OpenOOD

Meta-review: metareview: All the reviews are mostly aligned, in recognition of the merits and contribution of this paper, and its potential impacts to the community.
Therefore, I would like to recommend acceptance. Even though, I still recommend authors to make further enhancements and include the response changes in this paper.

confidence: 5: The area chair is absolutely certain
Decision: Accept