Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper presents an approach to detect out-of-distribution (ODD) samples by an ensemble of pre-trained models. For a given test sample, OOD scores are computed based on a set of pre-trained models and a suitable p-value is determined whether the sample is OOD while maintaining a given true positive rate. experiments are conducted on common OOD benchmarks such as CIFAR-10, ImageNet, and Places365.
strength_and_weaknesses: Strengths:

1. Considering an ensemble of models to detect OODs is a simple yet effective idea.

2. The draft is clearly written and easy to follow.

3. Authors conducted extensive experiments of the standard benchmarks and the results are state of the art.

Weaknesses:

1. Overall novelty is limited as p-value based OOD detection is explored in conformal prediction literature.

2. What is the score function used in this work? This is important and needs to be discussed.

3. In OOD detection, FPR is more important than TPR. Does theorem 1 provide any guarantee on FPR?

4. Authors are encouraged to discuss the computational complexity of Algorithm 1. For each test sample, it requires computing the score 
of all the training samples for all the models. This can be expensive.

5. Does the OOD detection performance depend on the quality of pre-trained models?

6. Some related references are missing 

a. Cai, F.; and Koutsoukos, X. Real-time out-of-distribution detection in learning-enabled cyber-physical systems. ICCPS, 2020: 
b. Kaur, R. et al., iDECODe: In-distribution equivariance for conformal out-of-distribution detection. AAAI, 2022

(a) considers p-value based conformal scores for OOD detection.
(b) proposed an approach to combine multiple p-values with a bound on TPR. 
clarity,_quality,_novelty_and_reproducibility: Please refer to the above section for clarity and novelty. 
Some additional details are required to ensure reproducibility
summary_of_the_review: The idea of considering an ensemble of models for detecting OOD is interesting. However, the overall novelty is lacking. Please address the concerns in the weaknesses section.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes the method ZODE, an ensemble scheme using pretrained models for OOD detection. The proposed scheme employs the Benjamini–Hochberg procedure to decrease the false discovery rate of OOD samples by combining the base model outputs. The authors conduct experiments on CIFAR10 and Imagenet datasets to evaluate the ZODE performance compared with SOTA ODD methods, and they show that the method improves the ODD detection performance.
strength_and_weaknesses: - The proposed approach is interesting and presents promising results. 
- The idea of using an ensemble of classifiers for OOD is not new, but using the BH procedure to combine the base classifiers is clever. BH generally produces fewer Type I errors and performs best in sparse cases. What about the non-sparse cases?


- I miss some qualitative discussions. What is the impact of the pretrained models on the results? Is there a minimum quantity of base classifiers? Do they need to be trained in some specific dataset? Is it necessary that the training dataset of the base classifiers be related to the test samples? Does the increase in the number of base classifiers improve the results? Is there a limit to that? What is the impact of adding diversity in the base classifiers in this context?
- The authors do not consider the computational cost of using all base classifiers in the inference process when comparing the performance with the standalone SOTA methods. Could distilling the ensemble's knowledge using a distillation approach improve the process for computational cost purposes? 
- Is the approach generic to another type of data or just work with images?
- How the SOTA methods were chosen? Are those the SOTA?
- The authors did not discuss the limitation of the proposed method. It will be meaningful to discuss the gap between the experiments in the current version of the paper and the real-world applications.

clarity,_quality,_novelty_and_reproducibility: The paper is clear and logically structured. Unfortunately, the code is unavailable, making it difficult to reproduce the experiments.
The idea of using an ensemble for OOD detection is simple but seems effective based on the paper's results. However, the idea is not entirely new, and the authors should compare the proposed model with other ensemble approaches. Therefore, the comparison with standalone methods seems not fair. 
The analysis does not consider the computational cost added by the base classifiers. 

summary_of_the_review: The trustworthiness of ML approaches is significantly related to their ability to recognize out-of-distribution samples. This is an important topic with a lot of applications. The proposed paper presents one interesting approach but fails to analyze different perspectives of the problem. The authors focus on error and do not consider the computational cost of the proposed method or the comparison with other ensemble methods. The author should discuss the proposed approach's limitations and explore the qualitative aspects of the results.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes to ensemble multiple pre-trained models with different architecture by estimating and thresholding p-value for out-of-distribution detection. Experimental results show the effectiveness of the proposed method.

strength_and_weaknesses: Strengths

+ Theoretical analysis on the p-value looks interesting.

+ The proposed method clearly outperforms other methods in most cases.

Weaknesses

- No ablation study on the ensembling strategy. For example, authors could simply average the detection score or do majority voting and take them as baselines. 

- The comparison is not fair. Model ensembling is a kind of guaranteed strategy to improve the performance in machine learning. Even authors showed consistently good performance, empirical contribution is somewhat limited in that sense. Rather, comparison with other (baseline) ensemble strategies should be fair, but missed.

- Is there a reason why the proposed method is combined with MSP, Energy, and KNN only? I wonder if the rank of methods holds, e.g., if ZODE-Mahalanobis is on average worse than ZODE-KNN. Authors could add more results in the appendix.

- Typo: DensNet @ Table 2
clarity,_quality,_novelty_and_reproducibility: Writing is clear and well-written in general. The ensemble strategy is not novel, but the method to ensemble them, estimating and thresholding p-value might be novel. They also provided some theoretical analysis on it. I believe this work is reproducible with the provided code.
summary_of_the_review: I am mostly happy with this paper, except for missing ablation study on the choice of ensembling strategy. Again, I believe ensemble method almost always improves the performance of machine learning models, so comparison with other ensemble methods should be more reasonable than standalone models. Please answer my concerns above.


**post-rebuttal**

The additional experiments for comparison with simple ensemble methods addressed my concern, but it turned out that the performance of the proposed method is not so better than baseline ensemble methods, in terms of AUC. Ensembling based on the p-value and corresponding analysis are interesting, but by looking at the performance, I can't see the reason why we need such a sophisticated ensemble strategy.
At this point, I decided not to change my initial rating.

By the way, authors claim that TPR is "out of control" and show FPR under somewhat random TPR for other ensemble methods, which I do not fully understand what they actually mean by. Maybe authors observed discrete changes on TPR?

It seems authors also count the fact that their work is first to take advantage of "(publicly available) model zoo" for OOD detection as the main contribution, but by looking at other reviews, it seems other reviewers do not care about it much?
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: **Summary**: 
This paper proposes to ensemble multiple pre-trained models with different architecture by estimating and thresholding p-value for out-of-distribution detection. Experimental results show the effectiveness of the proposed method.
Overall, the paper has borderline mixed reviews (556).

**Strengths**:
- Considering an ensemble of models to detect OODs is a simple yet effective idea.
- The draft is clearly written and easy to follow.
- Theoretical analysis on the p-value looks interesting.

**Weaknesses**: The reviewers identified the following major weaknesses: 

1. Limited novelty as p-values have been used in OOD detection before. [Ty7t]
2. missing related works (Cai & Koutsoukos, Kaur et al). [Ty7t]
3. Score function not described clearly. [Ty7t]
4. Since FPR is more important in OOD detection, does The 1 provide any guarantee on FPR? [Ty7t]
5. what is the computational complexity? [Ty7t, Wjqr]
6. Does the OOD detection performance depend on the quality of the pre-trained models? [Ty7t]
7. idea of using ensembles for OOD detection is not new, but using BH to combine base classifiers is clever. [Wjqr]
8. What about non-sparse cases? [Wjqr]
9. What is the impact of the pertained models on the results? Is there a minimum number of base classifiers? Does increasing the number improve results? What is the impact of diversity in the base classifiers? [Wjqr]
10. Could knowledge distillation on the ensemble be used to improve computational cost? [Wjqr]
11. Does it work with other types of data? or just images? [Wjqr]
12. How were the SOTA methods selected? [Wjqr]
13. What are the limitations (e.g., gap with real-world applications)? [Wjqr]
14. How does it compare to other ensemble methods? [Wjqr, FQTf]
15. No ablation study on the ensembling strategy (e.g., simple averaging or majority voting)
16. Why combine with MSP, Energy and KNN only? Does ranking of methods hold (ZODE-Mahal vs. ZODE-kNN)?


justification_for_why_not_higher_score: All reviewers agreed after the discussion that the paper is borderline reject. The AC agrees with their concerns. 
justification_for_why_not_lower_score: n/a
Decision: Reject