MetaOOD: Automatic Selection of OOD Detection Models

Authors: Yuehan Qin, Yichi Zhang, Yi Nian, Xueying Ding, Yue Zhao
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: Out-of-distribution Detection, Meta-learning, Language Modeling, AutoML

Abstract:
How can we automatically select an out-of-distribution (OOD) detection model for various underlying tasks? This is crucial for maintaining the reliability of open-world applications by identifying data distribution shifts, particularly in critical domains such as online transactions, autonomous driving, and real-time patient diagnosis. Despite the availability of numerous OOD detection methods, the challenge of selecting an optimal model for diverse tasks remains largely underexplored, especially in scenarios lacking ground truth labels. In this work, we introduce MetaOOD, the first zero-shot, unsupervised framework that utilizes meta-learning to select an OOD detection model automatically. As a meta-learning approach, MetaOOD leverages historical performance data of existing methods across various benchmark OOD detection datasets, enabling the effective selection of a suitable model for new datasets without the need for labeled data at the test time. To quantify task similarities more accurately, we introduce language model-based embeddings that capture the distinctive OOD characteristics of both datasets and detection models. Through extensive experimentation with 24 unique test dataset pairs to choose from among 11 OOD detection models, we demonstrate that MetaOOD significantly outperforms existing methods and only brings marginal time overhead. Our results, validated by Wilcoxon statistical tests, show that MetaOOD surpasses a diverse group of 11 baselines, including established OOD detectors and advanced unsupervised selection methods.

Decision: accept (poster)
Reviewer scores: 6, 3, 6, 6

Reviews and discussion:
<thread>
creation date:
1730298901898

thread_type:
official_review

<thread_object>
creator: reviewer_katw

<content>
summary:
The paper presents MetaOOD, a framework for automatic selection of out-of-distribution (OOD) detection models without requiring labeled data. It leverages historical performance data and language model embeddings. The approach aims to improve the reliability of OOD detection in critical applications, such as autonomous driving and online transactions. Overall, MetaOOD addresses the challenge of adapting to data shifts effectively and efficiently.

strengths:
The paper's strengths include the introduction of a zero-shot, unsupervised framework for OOD detection model selection, which enhances adaptability to new datasets. It effectively utilizes language model-generated embeddings to capture nuanced dataset characteristics, improving model selection accuracy. The extensive experimentation demonstrates superior performance compared to eleven established methods, showcasing its robustness. Additionally, the framework incurs minimal runtime overhead, making it efficient for practical applications. The use of the Wilcoxon signed-rank test is a plus of the paper. The p-values suggests the proposed approach works well.

weaknesses:
The paper's weaknesses include a reliance on the quality of language model embeddings, which may vary based on the model used and the nature of the input data. Additionally, the framework's performance may be limited by the diversity of the historical data pool, potentially affecting generalization to unseen datasets. The lack of extensive real-world testing could raise concerns about its applicability in practical scenarios. Lastly, the complexity of the approach may pose challenges for reproducibility and implementation in different contexts. Obtain datasets and model feature/embeddings from their textual descriptions appear a bit strange and somewhat unreliable.

questions:
No questions.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730535277237

thread_type:
official_review

<thread_object>
creator: reviewer_n69e

<content>
summary:
The paper postulates that by identifying which OOD detection models have historically performed well on datasets similar to the one currently being considered, one can select the model most likely to be effective without needing labels for supervised training.  A meta-learning approach is used to take past performance data from various models (across data sets); when new dataset arrives, the approach checks for similarity between the new dataset and historical ones using embeddings. The assumption is that the selected model will perform well as it is closest to the data set under use.  Meta-learning (training) is done offline with curated data sets; while OOD model selection is online as a specific data point arrives.  Results show that their approach works better than compared other techniques.  Experiment approach itself is reasonable and approach is sound.

strengths:
Integration of language model based embeddings; Empirical eval. is done to good detail. this technique is actually useful. though it is a logical next step ~ the approach itself can be used in other contexts or at-least the idea can be adapted. Sufficient detail is provided makes work transparent.

weaknesses:
Eval is too narrow and limited ~ so results may not generalise or this approach may be limited to the data set / domain attempted; esp. since there is no formal conceptual development as such we do not know when and where this method will work or have an intuition for where the limit may be.  Although there is the claim of unsupervised world-first etc. ~ there is still a need for other forms of supervised and curated training. Assumes text descriptions are good in the evals and curated data sets - but does it hold for the real world?  All the usual limits of language models apply here too. Scalability not known -- again since we do not have specific underlying theory.

questions:
Can the authors add a diagram to better illustrate how this technique would work in practice? This will help translatability of this work into other contexts faster.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730556612049

thread_type:
official_review

<thread_object>
creator: reviewer_bkxx

<content>
summary:
This paper presents MetaOOD, a “model” selection approach for out-of-distribution (OOD) detection. MetaOOD utilizes language models to generate feature embeddings of both the meta dataset and “models”, allowing for the optimal “model” selection based on anticipated performance on the test set. The results on the Wilcoxon statistical tests show the promising performance of MetaOOD.

strengths:
1. The motivation is sound. It is interesting to see a meta-selection approach to the OOD detection problem since there are so many methods in this OOD domain.
2. The proposed method is simple and straightforward. The results on the traditional methods are promising.

weaknesses:
1. The definition of "OOD model" is confusing. There are many post-hoc detection methods in the detection problem, which should not be classified as “models”. For instance, the paper includes the MSP method for the selection experiments. However, MSP is just a simple post-hoc technique that can be applied to most classification models (e.g., ResNet) using the SoftMax function. This method should not be considered as a model, which is misleading considering another factor, “model architecture,” in the experiments.

2. The methodology lacks depth. MetaOOD merely utilizes language models to extract embeddings for dataset and model descriptions, and then select the top-1 method based on these embeddings. The approach lacks insight and overlooks potential issues. For instance, the embeddings derived from descriptions may not accurately capture the true characteristics of the models and datasets. Also, simply selecting the top 1 can overlook the nuances of methods and the potential problems of the utilized datasets.

3. The experimental results are unconvincing. The baseline methods included are outdated, with the most recent method (NCF) dating back to 2017.

4. The terms OOD and OOD detection should not be used interchangeably. It is unclear what is meant by "OOD dataset" given such a name strategy. Is it referring to a commonly recognized OOD dataset distinct from the in-distribution (InD) dataset, or simply an OOD detection dataset (includes train, val, and test splits for detection methods)? 

5. I am curious whether this paper was generated by a language model, such as GPT-4. The writing style, particularly in Section 3.3.1, resembles AI-generated text. Given the simplicity of the method, the Method Section could be more concise, potentially requiring only 0.5 pages to convey the core elements of the approach. However, the current version spans 2.5 pages.

**Post-rebuttal Comments**

I'd like to thank the authors' response. I agree that weakness 5 could be too strong as I assumed the method may be generated from GPTs. However, the length and the redundancy of the method section are still a problem. I still cannot accept that the choice of embedding strategy could occupy half of the Method Section, given that the embedding method is that simple.

I also agree that the simplicity could be the advantage of the proposed method. However, I still cannot view this method as sufficiently advanced or novel for a research paper at ICLR. As I've mentioned, this method lacks depth, not to mention the absent theoretical insights. The approach merely utilizes the rough descriptions of the dataset and models to predict the score on the OOD dataset. The performance relies heavily on LLMs, which can introduce a series of problems and are heavily limited to the application scenarios. 

Also, I acknowledge that the paper may present the "first" selection strategy for the OOD detection method. However, this should not be an excuse for the obvious shortcomings. Overall, given current experiments, I consider this paper more of an exploration of leveraging LLMs in OOD detection rather than a substantial research contribution. Thus, I will maintain my original score.

**Additional Post-rebuttal Comments**

After reviewing the authors' responses, I still consider this paper as a preliminary exploration of leveraging LLMs in OOD detection. So I will maintain my original score.

questions:
NA
</content>
</thread_object>

</thread>
<thread>
creation date:
1730601565251

thread_type:
official_review

<thread_object>
creator: reviewer_na6w

<content>
summary:
In this paper, the authors propose MetaOOD, which utilizes meta-learning to select an OOD detection model automatically. The motivation is that each OOD detection algorithm might excel in specific scenarios but may not perform well universally, therefore it is important to select one particular OOD detection for each task. MetaOOD utilizes historical performance data of existing methods across a variety of benchmark out-of-distribution (OOD) datasets to enable efficient model selection for new datasets, eliminating the need for labeled data at test time. To more accurately measure task similarities, the authors incorporate language model-based embeddings that capture the unique OOD characteristics of both datasets and detection models. Through extensive testing across 24 unique test dataset pairs and 11 OOD detection models, the authors show that MetaOOD consistently outperforms current methods with minimal additional computation time.

strengths:
1. The idea of using meta-learning to select the best OOD detection method for each specific task is interesting.
2. The paper is generally easy to understand and clearly written.
3. The experiments show the effectiveness of the proposed method.

weaknesses:
1. Figure 1 needs to be improved. The notations in the figure are confusing and unclear.
2. The design of the textual description seems ad-hoc and cannot be applied in the case of without detailed dataset information.
3. Detailed results on the selected OOD method for each dataset are missing.

questions:
1. Does the proposed method rely on the architecture of the trained model?
2. What is the training time of the proposed method?
3. If there is one additional OOD method, how can incorporate this method into the proposed MetaOOD?
4. What are the main factors that influence the choice of an OOD method based on the characteristics of the training and test sets?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732083363182

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
### Summary of Our Responses and Contributions 
We sincerely thank all reviewers for their thoughtful and constructive feedback, as well as for recognizing the novelty and importance of our work. We have carefully addressed each individual comment and made revisions to improve the manuscript (changes highlighted in blue). Below, we summarize the major contributions of our work and highlight points endorsed by the reviewers:

**Novelty**: We introduced **MetaOOD**, the first zero-shot, unsupervised framework for automatic selection of OOD detection models. We are grateful that multiple reviewers (e.g., Reviewer N69E and kAtw) acknowledged the significance of this problem, describing it as a "logical next step" and an "effective and efficient" solution to the critical challenge of adapting OOD detection to real-world data shifts in domains such as autonomous driving and healthcare.

**Specialized Framework**: MetaOOD leverages **meta-learning** with language model-based embeddings to capture the distinctive characteristics of datasets and OOD detection models. This enables robust model selection without requiring labeled data. Reviewers kAtw and nA6W highlighted using embeddings and meta-learning as a **"sound and interesting"** approach that offers a principled solution to an underexplored challenge.

**Extensive Experimental Validation**: Our experiments demonstrate the effectiveness and efficiency of MetaOOD across 24 test dataset pairs and 11 OOD detection models, significantly outperforming existing methods with minimal computational overhead. Reviewers kAtw and N69E appreciated the robustness of our results, with Reviewer kAtw commending our use of the **Wilcoxon statistical test** to validate performance claims. Reviewer nA6W noted that the experimental setup is "extensive" and the results convincingly support our claims.

**Practical Contributions**: MetaOOD enhances reliability in critical open-world applications by automating model selection, and eliminating the need for manual tuning or labeled data. Reviewer N69E highlighted its potential for broader applicability, stating that "the approach itself can be adapted to other contexts" and offers valuable contributions to practical OOD detection.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732510205976

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
We sincerely thank the reviewer again for your valuable time and thoughtful comments. We have updated the paper in response to your feedback, with changes highlighted in blue in the updated file and additional tables/figures included in the appendix. As we are approaching the end of the discussion stage, we would greatly appreciate it if you could kindly read our responses and update the scores if your concerns have been addressed. We are more than happy to further discuss any concerns that you find not fully addressed. Thank you very much.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper presents a meta-learning framework for model selection of out-of-distribution detection models. The core idea is to use embeddings from language models to represent datasets and models as described by textual descriptions. The framework is evaluated on image-based OOD detection tasks and shown to outperform other baselines.

The paper addresses an important practical problem of model selection for OOD detection. The proposed method is simple and shown to be effective, using rigorous statistical tests. The work could be strengthened by additional analysis on the embeddings and how they are capturing task and dataset similarity, as performance appears to be somewhat sensitive to choice of language model used (Fig 4). Experiments on a broader range of datasets will also help to strengthen evidence for the generalizability of the approach. 

Overall, the AC leans towards accepting this paper as an interesting first approach to an important practical problem. The authors should incorporate all changes and additional results as promised in the discussion.