MetaOOD: Automatic Selection of OOD Detection Models
Authors: Yuehan Qin
Yichi Zhang
Yi Nian
Xueying Ding
Yue Zhao
Venue: ICLR 2025 Poster (2025)
Keywords: Out-of-distribution Detection
Meta-learning
Language Modeling
AutoML
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Abstract: How can we automatically select an out-of-distribution (OOD) detection model for various underlying tasks? This is crucial for maintaining the reliability of open-world applications by identifying data distribution shifts, particularly in critical domains such as online transactions, autonomous driving, and real-time patient diagnosis. Despite the availability of numerous OOD detection methods, the challenge of selecting an optimal model for diverse tasks remains largely underexplored, especially in scenarios lacking ground truth labels. In this work, we introduce MetaOOD, the first zero-shot, unsupervised framework that utilizes meta-learning to select an OOD detection model automatically. As a meta-learning approach, MetaOOD leverages historical performance data of existing methods across various benchmark OOD detection datasets, enabling the effective selection of a suitable model for new datasets without the need for labeled data at the test time. To quantify task similarities more accurately, we introduce language model-based embeddings that capture the distinctive OOD characteristics of both datasets and detection models. Through extensive experimentation with 24 unique test dataset pairs to choose from among 11 OOD detection models, we demonstrate that MetaOOD significantly outperforms existing methods and only brings marginal time overhead. Our results, validated by Wilcoxon statistical tests, show that MetaOOD surpasses a diverse group of 11 baselines, including established OOD detectors and advanced unsupervised selection methods.

Published as a conference paper at ICLR 2025
METAOOD: AUTOMATIC SELECTION OF OOD DE- TECTION MODELS
Yuehan Qin
Yichi Zhang'
Xueying Ding Yue Zhao'
University of Southern California fyuehanqi, y2hang42, yow eus....eeeuuueeeeo...o edu, nian@uchica cago. edu, xding2 cww.ccu...uuuoommm "Co-frrtt authors
University of Chicago 'Carnegie Mellon University
ABSTRACT
How automatically select nut-of-distribution (OOD) detection model for various underlying tasks? This is crucial for maintaining the reliability of open-world applications by identifying data distribution shifts, particularly in crit- ical domains such as online transactions, autonomous driving, and real-time pa- tient diagnosis. Despite the availability of numerous OOD detection methods, the challenge of selecting an optimal model for diverse tasks remains largely under- explored, especially in scenarios lacking ground truth labels. this work, we in- troduce MetaOOD, the first zero-shol, unsupervised framework that utilizes meta- learning to select an OOD detection model automatically. As a meta-learning approach, MetaOOD leverages historical performance data of existing methods across various benchmark OOD detection datasets, enabling the effective selec- tion of suitable model for new datasets without the need for labeled data the test time. To quantify task similarities more accurately, we introduce language model-based embeddings that capture the distinctive OOD characteristics of both datasets and detection models. Through extensive experimentation with 24 unique test dataset pairs to choose from among 11 OOD detection models, we demon- strate that MetaOOD significantly outperforms existing methods and only brings marginal time overhead. Our results, validated by Wilcoxon statistical tests, show that MetaOOD surpasses diverse group of 11 baselines, including established OOD detectors and advanced unsupervised selection methods.
INTRODUCTION
out-of-distribution (OOD) detection is the process of identifying data points that deviate signifi- cantly from the distribution of the training data. This capability is essential for ensuring the reliabil- ity of machine learning models when they encounter new, unseen data (Yang et al. (021). Common applications of OOD detection include safety-critical systems like autonomous vehicles (Filos al.) 2020 Li et al. 2024a) and medical diagnosis (Ulmer et al., 2020) to prevent erroneous predic- tions. Notably, many OOD detection algorithms are unsupervised due to the high annotation cost, leveraging statistical methods or reconstruction errors from models such as autoencoders to identify deviated samples without labeled OOD data (Yang al. 2022, Dong al., 2024 [uun al. 2022). However, despite the variety of OOD detection algorithms, each adepts at identifying different as- pects of in-distribution (ID) and OOD data, there lacks a systematic method for choosing the best OOD detection model(s) under the unsupervised setting without labels. It is acknowledged that each OOD detection algorithm might excel in specific scenarios but may not perform well universally due to the no-free no-free-lunchl theorem (Wolpert & Macready, 1997). Moreover, without labels, it is difficult to objectively evaluate and compare the performance different OOD detection models.
Present Work. In the most related field, unsupervised outlier detection (OD) has benefitted sig- nificantly from meta-learning with models like MetaOD (Zhao et al. 2021). ELECT (Zhao et al.) 2022ah, ADGym (Jiang et al. (2024), and HyPer (Ding 2024) demonstrating notable advance- ments. By leveraging historical performance data, these methods estimate a model's efficacy on new datasets. However, they are not directly adaptable to OOD detection due to several key differences:

---

Published as a conference paper at ICLR 2025
(Offline) Meta-Learner Training
Dalasel pair
M2
running mn OOD detection methods on dalasel pairs
Training meta-learner f
in in Performanie Matrix P
OOD detecior
Datasel pair
E gdataact
Meta-Prediche Predictor
D
Pre-trained Language Model
X,
OOD detector
moodel
Performance Matrix
Embeddings
3.4 (OnlIne) Model Selection
Predicted Performance P Preas
Dataset pair
Embeddings
Pre-trained Language Model
m
max Pacus
X test
Donie
Predictor Selected Model M
P new
glest
My..
Figure I: MetaOOD overview (.22;; offline meta-training phase is shown on the top ($3.3) the key is train meta performance predictor f (denoted in @) to map language embeddings of the datasets and models to their performance P; the online model selection ($3.4) is shown at the bottom by transferring the meta-predictor predict the test data paired with OOD detectors for selection.
First, OD and OOD detection differ in their problem settings, OD models are trained on both nor- mal and outlier samples (Zhao et al., 2019 Ma al. [202)) focusing on anomalies within a single distribution (Zhao et al.) 2022b; Zhao [224]] Chen et al. 2024). In contrast, OOD detection involves training solely on ID data and aims to identify samples from entirely different distributions, often across multiple datasets (Li et al.) 2024b Liu et al.) 2024). Second, OOD detection tasks primarily deal with images, presenting greater complexity compared to time series and tabular data OD. Third, the embeddings used to measure dataset similarity in OD do not translate well to OOD de- tection contexts. Current OD embedding generation are largely heuristic and reature-speccfic (Zhao) et al. 2021). inadequate for the diverse and complex nature of OOD datasets, especially when sidering the different data modalities. Thus, addressing these challenges with a tailored method for OOD detection model selection is crucial.
Our Work. In this study, we introduce MetaOOD, the first unsupervised OOD detection model Figlinggillllllsstttrtre selection method that employs meta-learning Fig. Iillustrates the overall workflow of the proposed approach. The idea is that an OOD detector that performed well on similar historical datasets is train suite of OOD detection mod-
likely to excel on new ones. As meta-learning approach, els offline using a variety of carefully curated datasets to gauge their performance across different scenarios during the meta-train phase (Figl [1] top). When new dataset arrives, we apply knowledge derived from historical data to select an appropriate OOD detection model (Fig. bottom). This selection is based on the similarity between the new dataset and those used in the meta-train phase. To enhance the accuracy of this similarity assessment, we designed two versions of data embeddings via specialized OOD meta-features and language embeddings from language models. As our exper- iment shows, these language model-based embeddings effectively capture complex, nuanced dataset characteristics that traditional meta-feature might miss. We summarize our technical contributions:
First OOD Detection Model Selection Framework. We introduce the first meta-learning-bee framework for zero-shot OOD detection model selection without training and evaluation.
Specialized Embeddings for OOD Datasets and Models. We use language model-generan features to quantify the similarity among OOD detection tasks, facilitating better understanding of OOD characteristics and enhancing OOD detection model selection.
Effectiveness. The proposed MetaOOD outperforms eleven well-known model selection methods and unsupervised meta-learners on a testbed with 24 unique test data pairs. is superior and statistically better than all the baselines w.r.t. average rank, and efficient with small runtime. Accessibility and Reproducibility. We release the testbed, corresponding code, and the proposed meta-learner https:: /ttp:::///tttttt /qgtttb........ com/ /gin433mettaodd
2

---

Published as a conference paper at ICLR 2025
2 RELATED WORK
2.1 UNSUPERVISED OOD DETECTION MODEL SELECTION
OOD detection faces the challenge of both OOD samples and OOD distributions being unknown during the training phase (Hendrycks et al. 2019 Liang et al. 2018). Consequently, selecting models for OOD detection relies solely on in-distribution or training data (Lee et al. 2018b). Once deployed an open-world setting, the OOD detector often handles a diverse range of test inputs originating from various OOD distributions (Yang al. 2021), and consequently. unsupervised methods are expected to be more suitable for selecting OOD detection models (Liu et al. 2020b).
Within these unsupervised methods, some recent works focus actively choosing sample- wise detectors (e.g., choosing different groups of detectors for different input samples) (Xue et al. 2024), while it is not doing actual model selection at the single model level as this work tries to address.
Most existing methods rely on trial-and-error or empirical heuristics. For instance, the simplest approach may be not to actively "select" model, but just use popular OOD detectors like Maximum Softmax (Hendrycks & Gimpel| 2017) and ODIN (Liang et al., 2017). There are other simple approaches: the confidence level of the ID data may be used as an indicator for OOD detection model selection. However, such a simple method may lead to a low true positive rate (TPR) (Hendrycks & methods, where the model selection is based on the similarity or cluster of the dataset, which has also been utilized for algorithm recommendation
Gimpel, 2017). Another direction similarity-
(Kadioglu al., 2010 Nikolić et al., 2013 Xu et al.) 2012 Misir & Sebag (017). We thus include these algorithms as our baselines in $4
SUPERVISED OOD DETECTION MODEL SELECTION 2.2
There has been considerable research into the use of supervised model selection techniques for effec- tively training and evaluating predictive models on labeled datasets (James et al. 2013 Hastie et al.) Tibshirani
2015
1996). These methods are partieularly valuable in scenarios where generalization to unseen data is critical. Model selection in supervised learning encompasses a variety of strategies, including cross-validation, grid search, and performance metrics such accuracy and Fl score to assess model efficacy. Randomized (Bergstra & Bengio 2012), bandit-based (Li et al. 2017). and Bayesian optimization (BO) techniques (Shahriari et al. 2015) are various SOTA approaches to hy- perparameter optimization and/or algorithm selection. However, it is notable that such methods do not apply to unsupervised OOD detection model selection, as ground truth values are absent.
2.3 REPRESENT DATASETS AND MODELS AS EMBEDDINGS FOR META-LFARRING
In terms of data representation, especially when it comes to meta-learning, embeddings play an im- portant role measure dataset/task similarity. Traditionally, computational- based meta-features are used as data representation in the meta-learing process (Vanschoren) (0018). More recently, more sophisticated, learning-basse based meta-features have been designed, as in dataset2vec (Jomaa et al. 2021) and HyPer (Ding et al. 2024). Additionally, there has been a notable integration of language embeddings to encapsulate data features, aiming to enhance model understanding and data repre- sentation (Drori et al. 2019 9 Fang et al. 2024). While the first approach primarily relies on heuristic methods and the latter can be hindered by the slow pace of model training, we adopt language em- beddings to represent data for this OOD detection model selection task. In this study, the traditional meta-feature approach is implemented as well for comparison and evaluation.
3 METAOOD FOR OOD DETECTION MODEL SELECTION
3.1 PRELIMINARIES ON OOD DETECTION
The OOD detection task involves training datasets Xirain sampled from the in-distribution (ID) Pi.. denoted X ~ Pun, and testing datasets X which may contain both ID and OOD samples. The goal is to train model M to classify if new sample X belongs Pin.
M(x) { i & Pmn
[1 if E Pin:
Generally, a model is trained exclusively on the ID data to learn to perform tasks such as classifica- tion on this dataset. For example, one common approach is using the model's confidence scores as the method M to distinguish between ID and OOD data.
When evaluating an OOD detector on X both ID and OOD test data present to assess the detector's 's capability to accurately distinguish between known and unknown data samples. There- fore, in this study, we structure the data into dataset pairs D = (X X each consisting of the training dataset with ID samples only and the test dataset with both ID and OOD samples.
3

---

Published as a conference paper at ICLR 2025
3.2 PROBLEM STATEMENT AND FRAMEWORK OVERVIEW
Given a new, unseen dataset pair for OOD detection, the goal is to select the best model from a heterogeneous set of OOD detection models without requiring any model evaluations at test. In this work, we leverage meta-learning to transfer model performance information from prior experiences to the new OOD detection task. Meta-learning |Vanschoren| 2018), often called "learning to learn", is technique where an algorithm learns from a collection of historical/meta tasks and uses this experience perform well on new, unseen tasks. The rationale is that an OOD detector is likely to outperform on new dataset if it excels in a similar historical dataset. This approach is particularly useful when evaluation is infeasible or costly due to a lack of labels or the need for rapid adaptation.
The proposed meta-learner, MetaOOD, relies on:
collection of n historical (i.e., meta- meta-train) OOD detection dataset pairs, Durin {DI.... Dr with ground truth labels, i.e., D = { {Xuain: (Xust Yles)).
Historical performance P of the pre-set model set set M = {M....., Mm (with models), on the meta-train datasets. We refer to P E [R=xm as the performance matrix, where P.,, corresponds to the j-th model Mj's 's performance on the i-th meta-train dataset pair D:.
Our objective is to choose the best candidate OOD detection model M E M, given a new pair of datasets D Dnow train X.pew } as input, where we have no ground truth labels yles for evaluation.
Problem (OOD detection model selection) Given a new input dataset Dnew = (Xirii X (detect OOD samples on X with only in-distribution data X main and no labels), select model M E M to employ on the new (test) task.
The problem is similar the OD model selection task such MetaOD, ELECT, and HPOD Zhaao 2024). However, in contrast to OD model selection, which focuses solely on a single dataset, OOD detection model selection requires the consideration of both training and test datasets. This necessity stems from the need to take into account the similarities and differences among both datasets that impact pre-training on the ID data and the actual OOD detection on the test data, all crucial to measuring the inherent characteristics of OOD detection task.
In nutshell, our MetaOOD consists of phases: (i) offline (meta-) training of the meta-leamnr on Deain where the goal is to learn the mapping from OOD detection models' performance on various (historical/meta) datasets, and (ii) online model selection that uses the meta-train information to choose the model at test time for the new dataset Dnew. Fig.] outlines the workflow and key elements of MetaOOD, with the offline training phase shown in white and the online model selection stage shown in grey. The details of the two phases are discussed in $3.3 and 3... respectively.
3.3 OFFLINE TRAINING
During offline training, we generate embeddings for dataset pair Duain and method M, and train the latent mapping from these embeddings the performances P. The meta-learner can generalize and select the best-performing model for new, unseen datasets by learning the relationship between (Dirain: M) + P. Note this training process supervised. Prior works have shown that perfor- mance mapping is empirically learnable, although imperfect, in related fields like OD (Zhao 2024).
To predict the performance of the candidate model on a new datasel pair, we propose training a meta-predictor as a regression problem. The input to the meta-predictor consists of Emmeta F. Emmddl corresponding to the embedding of the i-th dataset pair and the embedding of the j-th OOD detector. Dataset embedding of dataset pair D is denoted as Edata (D), and method embedding extracted from M is denoted as E Emmoeel
& (M,, M); we provide more details below on the embedding generation in 13.3.1 Our goal is to train the meta-predictor f to map the characteristics of the datasets and the OOD detectors to their corresponding performance ranking across all historical dataset pairs. The steps of the (offline) meta-train are shown in Fig. [1 top and Appx. Algo.
f: Eelaaa
pumadel + Piih i (1,..,,,,,
n}, JE (1,.....,
'The use of the term 'odels' more for following the tradition of model selection research. 2The format of f can be any regression models; in this work, use an XGBoost (Chen & Guestrin 2016 model due its balance of simplicity and expressiveness, well strong feature selection characteristic.
4

---

Published as a conference paper at ICLR 2025
3.3.1 DATA AND MODEL EMBEDDINGS
Data and model embeddings, as inputs of f. offer compact, standardized context about the data and models utilized in the meta-learning process. Instead of directly using data with arbitrary sizes, we hope to generate embeddings that can effectively represent the data and help develop models that can adapt rapidly to new tasks. In this work, try two types of data and model embeddings: (1) classical (data) meta-features and (method) one-hot encoding embedding, and (2) language model- based approach to get data and model embeddings via language models.
Classical Way: Data Embedding via Meta-Features and Model Embeddings via One-hot En- coding. Meta-features, or "features of features", are attributes used in leta-learnnng, model selec- tion. and feature engineering to provide higher-level information about a dataset or its characteristics (Vanschoren 2018). These include basic statistics, such as the mean and maximum values of the data, as well as outputs or performance metries from preliminary models tested on the dataset. Meta-features facilitate understanding the nuances of various learning tasks and guide the selection or configuration of models for new tasks by drawing parallels with previously encountered tasks. In this work, we first create meta-features that capture OOD characteristics across different dataset pairs. These features aim to identify similarities between data in the meta- veta-training database and test data, enhancing the model selection process based on prior successful applications. To achieve this, we categorize the extracted meta-features into statistical and landmarker categories.
Statistical meta-features capture the underlying data distributions' characteristics, including vari- ance, skewness, and other relevant statistics of individual features and their combinations. For image datasets used in OOD detection, we incorporate meta-fcatures that reflect pertinent image characteristics (Aguiar al. 2019), such as: (i) Color-based: Simple statistical measures from color channels; (ii) basser- Statistical measures obtained after applying border-detector filters; (iii) Histogram: Statistics from histograms of color and intensity; and (iv) Texture features: Values de- rived from an image's 's texture, analyzed using the co-occurrence matrix and Fast Fourier Transform.
Landmarker features summarize the performance of specific learning algorithms on a dataset, pro- viding quick and effective estimates of algorithm performance. These features, derived from evaluat- ing simple/fast models, offer a snapshot of dataset characteristics and approximate the performance of more sophisticated models in specific tasks. For OOD detection, our landmarkers focus on fea- tures related to Softmax probability outputs that can be rapidly generated without model fitting. Detailed descriptions and complete list of our OOD meta-features are in Appx. Tab. B
New Approach: Data Embedding via Language Models. Meta-features, often handcrafted and heuristic, have limitations in scalability and adaptability. This manual approach to feature engi- neering can be labor-intensive and may not capture all the nuanced relationships within OOI) data, potentially affecting the efficacy of model selection. In contrast, leveraging language models offers a transformative approach for generating data embeddings (Peng et al., 2024). These models have been recently utilized to generate embeddings from textual descriptions of datasets and methods, capturing essential information that reflects the datasets' and methods' intrinsic properties due to their comprehensive training (Drori al., 2019). Given that language models are designed to pro- cess text inputs, we hypothesize that utilizing text descriptions of data as input to language model for embedding generation could effectively encapsulate the inherent features of the data. In this study, we experiment with various language models to produce the language embeddings.
To capture the dataset features, we input dataset metadata such as size, object type, and description. For example, the input for the CIFAR-10 dataset (Krizhevsky 2009) is formatted as:
Contains images of 10 types of objects, including airplanes, cars, birds, cats, deer, dogs, frogs, horses, ships, and trucks. Each image is a small 32x32 RGB image.
For OOD detection model embeddings, take the textual descriptions of methods from the Pytorch-OOD library (Kirchheim et al. (2022) as input. This includes detailed information about the OOD detector, such as its components, whether it needs fitting, and what is used as OOD indi- cator. An example of this is the description for the Openmax method (Bendale & Boult 2016):
Determines center for each class in the logits space of model, and then creates a statistical model of the distances of correctly classified inputs. It uses extreme value theory to
5

---

Published as a conference paper at ICLR 2025
detect outliers by fitting Weibull function to the tail of the distance distribution. The activation of the unknown class is used as the outlier score.
The complete lists of the datasets and methods used are listed in $4.1 In brief, this approach lever- ages language models' natural language processing capabilities transform qualitative descriptions into quantitative embeddings. Compared to the traditional meta-feature approach, it is more com- putationally efficient and has better generalization ability. Also, is interesting to whether could use language models to choose an OOD detector directly. We provide all these analyses in $
3.4 ONLINE MODEL SELECTION
In the online model selection process, generate the embeddings for the test dataset pair Dew and reuse the model embeddings of M. apply the trained meta performance predictor f from the offline stage to predict different OOD detection models' performance, and select the model with the highest predicted performance, as described in Eq. (2)
where new... = f(Enew Liguudel)
== arg max new M,EM
Specifically, for new dataset pair, we acquire the predicted relative performance ranking of differ- ent OOD detection methods using the trained f. and select the top-1 method] as shown in Eq. (2). It is important to note that this procedure is zero-shof, and does not require any training the test sample. The (online) model selection steps are given in Fig. [ bottom and Appx. Algo. 2
EXPERIMENTS
Our experiments answer the following research questions (RQ): RQ1 ($4.)): How effective is the
proposed MetaOOD in unsupervised OOD detection model selection in comparison to other leading
baselines? RQ2 ($4.4.I): How do different design choices in MetaOOD impact its effectiveness? How much time overhead/saving MetaOOD brings to OOD detection in general?
RQ3 ($ 4.4.2
4.1 EXPERIMENT SETTING
The model Set M. We compose a model set M choose from with 11 popular OOD detection models as shown in Tab. covering different types of detection methods.
The OOD Datasets. We utilize the train-test split of datasets preprocessed described in (Yang et al. 2022). To summarize, we create our ID-OOD dataset pairs using the following datasets:
ID Datasets: CIFARIO (Krizhevsky) 2009). CIFAR100 (Krizhevsky
2009), ImageNet (Deng
et al. 2009), FashionMMISTT (Xiao et al.,
2. Classic OOD Group: CIFAR10, CIFARIO0, MNIST (Deng 2012).
Places365 Zhou et al.
2018), SVHN (Netzer et al. 2011), Textures (Cimpoi et al. 2014), TIN (Le Yang 2015)
Large-Scale OOD Group: SSB haad (Vaze al.
2022
NINCO (Bitterwolf al. 2023), ). iNatu-
ralist (Horn et al. 2017), Textures (Cimpoi et al.
2014)
OpenImage-O (Wang et al. 2022)
where semantic overlap images between the ID data and the OOD data are removed (1,203 images from TIN, and 1,305 images removed from MNIST, SVHN, Textures, and Places365).
We construct the ID-OOD dataset pair, and set the training and testing set follows: (i)
Category
OOD Detection Model
Probubility Based
MSP
Hendryeks & Gimpel 2017
Gaa Ghahramann 201b
Training: CIFAR10 from ID and OOD from the classic OOD group shown above; and (ii) Testing: CIFAR 100, ImageNet, and Fashion- MNIST from ID, and OOD from large-scale OOD dataset group. The other ID datasets un- dergo a similar train test split, which results in 24 unique testing pairs. Our split guaran- tees that no dataset the testing pair has been seen/leaked during the meta-train process.
KLM Hendryeks al 2016
Entropy Chaa el 4040
Logit-based
MaxLogit [Hendrycks
2022
Openmax (Bendale & Boull
2016
EnergyBased [Ln et 2020
ODIN [iang 21 2017)
Feature-hused
Mahalanohis
1018a
ViM Wang c( 20/4
ANN Cover Hart 1967
Table 1: OOD detection models in this study.
It may choose the top-k candidates an ensemble, while this work focuses on top-1 selection.

---

Published as a conference paper at ICLR 2025
Hardware. For consistency, all models are built using the pytorch-ood library Kirchheim et al. 2022) on NVIDIA RTX 6000 Ada, 48 GB RAM workstations.
XGBoost (Chen & Guestrin 2016) model as f due to its simplicity and expressiveness. Meanwhile, we use the data and model embeddings generated by the pre-trained BERT-basee all-mpnet-base-22 model by Hugging Face (Reimers & Gurevych 2019). We provide more ablations on f in $4.4.1
Training the peea--reedcctr (see details in $3.3). In this work,
Evaluation. To compare the performance of MetaOOD and baselines, examine the performance rank" of the OOD detector chosen by each method among all the candidate detectors through a boxplot and the rank diagram (which is the average across all dataset pairs). Clearly, the best rank is I, and the worst is 12 (i.e.. 11 baselines and MetaOOD). To compare our algorithm with baseline, we employ the pairwise Wilcoxon rank test on performances across dataset pairs (significance level
p < 0.05). The full selection results are in Appx. Tab.
4.2
MODEL SELECTION BASELINES
We select the baselines following the literature in meta-learning for unsupervised model selection
(Zhao et al.
2021 2022a Jiang al., 2024 Park et al. 2023) with four categories:
(a) No model selection or random selection: always employs either the ensemble of all the models or the same single model, or randomly selects a model: (1) Maximum Softmax Probability (MSP) (Hendrycks & Gimpel| 2017), as popular OOD detection model, uses the maximum softmax score of a neural network's 's output as threshold to identify whether an input belongs to the distribution the network trained on. (2) ODIN (Liang al.) 2017) applies temperature scaling and small perturbations to the input data, which helps to amplify the difference in softmax scores between ID and OOD samples. (3) Mega Ensemble (ME) averages OOD scores from all the models for a given dataset. As such, ME does not perform model selection but rather uses all the models. (4) Random Selection (Random) randomly selects model from the pool of candidate models.
(b) Simple meta-learners that do not involve optimization: (5) Global Best (GB) is the simplest meta-learner that selects the model with the largest average performance across all meta-train datasets. GB does not use any meta-fcatures. (6) ISAC (Kadioglu et al. 2010) clusters the meta- train datasets based on meta-features. Given new dataset pair, identilies its closest cluster and selects the best model of the cluster. (7) ARGOSMART (AS) (Nikolić et al., (013)) finds the closest meta-train dataset (1 nearest neighbor) given test dataset, based on meta-feature similarity, and selects the model with the best performance on the INN dataset.
(c) Optimization-based meta-learners which involves learning process: (9) ALORS (Misir & Sebag. 2017) factorizes the perf. matrix to extract latent factors and estimate perf. as the dot product of the latent factors. A regressor maps meta-features onto latent factors. (9) NCF (He et al.] 2017) replaces the dot product used in ALORS with more general neural architecture that predicts performance by combining the linearity of matrix factorization and non-linearity deep neural networks. (10) MetaOOD O uses manually crafted heuristic meta-features, which consist of both statistical and landmarker features for dataset meta- meta-feeturee, along with one-hot encoding for model embeddings, as discussed in $3..11
(d) Large language models (LLMs) as a model selector: (11) GPT-4o mini (OpenAl et al. 2024) is used zero-shot meta-selector. The dataset and method descriptions directly provided the LLM, allowing it to select the methods based on these descriptions. Note there is no meta-lcarning
here. The details are presented in Appx. $B.1
AUROC
12
10
8
6
ODIN LS51
MetaOOD MetaOOD 0 GPT-4a mini
ME Luma
Random ISAC NCF Loss
GB ALORS AS
Figure 2: Average rank (lower is better) of methods w.r.t. performance across datasets; MetaOOD outperforms all baselines with the lowest rank.
In this work, we choose Area Under the Receiver Operating Characteristic Curve (AUC-ROC), or ROC, the performance metrie; can replace by any other metrics interest.
7

---

Published as a conference paper at ICLR 2025
Ours
Baseline MetaOOD 0
p-value
MSP
MetaOOD
0.0357
MetaOOD
ME
<0.0001
Random
MetaOOD
AS
0.0004
MetaOOD
ISAC
0,0001
NCF
MetaOOD
ALORS
0.018
MetaOOD
Random
0.0005
ALORS
MetaOOD
MSP
<0.0001
MetaOOD
ODIN
<0.0001
Metab00_0
MetaOOD
NCF
0.0027
GPT-4o
MetaOOD
GB
0,018
MetaOOD
MetaOOD
GPT-4o mini
0,018
11
10
Rank on AUROC
Table 2: Wilcoxon signed-rank test re-
Boxplot the rank distribution of MetaOOD and baselines (the lower, the better). MetaOOD is the lowest/best.
Figure 3:
sult. MetaOOD is statistically better than all the baselines.
4.3 OVERALL RESULTS
First, we present the rank distribution of the actual rank of the top-1 OOD detector chosen by each model selection method across the 24 test data pairs in Fig. To compare two model selection algorithms (c.g.,
with bascline), we perform Wilcoxon rank test on the AUROC value of the top-1 models selected by our method and the baseline method, as shown in Tab. 2 Also, we show the aggregated average rank plot in Fig. 2 where the average performance rank of the OOD detection model selected by each algorithm is shown. Here are the key findings of the results:
1. MetaOOD outperforms all baselines. As shown in Fig. MetaOOD consistently performs well with small variance. On the aggregated level, the average rank MetaOOD surpasses all of the 11 baselines in Fig. 2 Moreover, MetaOOD has demonstrated statistically superior performance compared to all these baselines, as shown in Tab. 2 Such enhancement in performance is indicative of MetaOOD's robust approach tackling complex datasets while generating stable performance. We credit its superiority to the combination of meta-learning framework and the use of language models in embedding datasets and models.
2. Optimization-arn based meta-learning methods (i.e.. MetaOOD, MetaOOD_0 0 and ALORS) per- form better in unsupervised model selection over other baselines. One reason is that meta-learning leverages prior experiences to adapt to new tasks. It optimizes the learning process by extracting common patterns and representations across different tasks, enhancing the model's generalization ability. Meanwhile, the optimization process helps models converge to optimal solution effi- ciently. Compared to simple meta-learners, such optimization-base methods take advantage of the meta-features for better mapping of the model performance. For instance, AS and ISAC do not involve optimization and resemble finding the closest sample or closest sample cluster for choosing the top-1 detector. They inferior MetaOOD there is optimization the OOD detector selection, but fully depends on the data embeddings for performance knowledge transfer.
3. The underperformance of model selection and random selection baselines justifies the need for OOD model selection. We analyze their performance below.
ME: Averaging the OOD detection scores of all models does not yield strong results, demon- strated in Fig. 2 and 3 This may stem from certain models consistently underperform across various datasets, and combining all models indiscriminately reduces overall effectiveness. While using selective ensembles might offer improvements (Zhao & Hryniewicki 2019), constructing ensembles from numerous models can be impractical due to high costs. In contrast, MetaOOD learns to make optimal selections without constructing any models, allowing it to operate effi- ciently during testing-
Random: According to Fig. and Tab. random selection performs worse than all the meta- learners. This indicates that all the meta-learner baselines we chose do have some improvements compared to random choice, and it is not advised to select an OOD detection model randomly. ODIN and MSP: As expected, a single method does not perform well across all datasets. This result is not surprising since different OOD detectors emphasize various aspects of the datasets, and real-world datasets have diverse characteristics. Relying on a single approach tends to limit the scope of solutions, making it difficult capture the distribution shift among different datasets.
8

---

Published as a conference paper at ICLR 2025
4. LLM acts a reasonable zero-shot model selector in OOD detection model selection. This highlights the potential LLM in model selection, while it can be improved by meta-learning. While the performance of LLM selector may be slightly worse (ranked top 3) than MetaOOD, LLM consistently selects the globally best model (Openmax (Bendale & Boult 2016)) across diverse tasks, with prior knowledge about the performances of different OOD detection methods on different models. This is valuable when prior knowledge is limited or when facing novel tasks, highlighting the potential of LLMs in unsupervised model selection. It can be improved via meta-learning by receiving more performance-relate language embeddings along with the meta- leatnle. thus achieving better performance. Future work may consider LLMs' robustness and trustworthiness in model selection Huang et al. (2025).
information. Arguably, MetaOOD leverages
4.4
ABLATION STUDIES AND ADDITIONAL ANALYSES
4.4.1
ABLATION STUDY
In $3.3.1 we discussed how train the meta meta-pertie performance predictor with the choices on data and model embeddings and f itself. Here, we conduct two ablations on these choices:
Ablation 1 on Data and Model Embeddings. We compare the MetaOOD's performance on differ- ent LLM-generain embeddings and different combinations of embeddings. The variants include:
OpenAI text-embedding-3-eeee 3--mmlll (OpenAL embb (Reimers & Gurevych] 2019): It is an em- bedding model designed by OpenAl to generate compact and meaningful representations of text for various natural language processing tasks.
LLama2-7b (Llama) TTovrro et al., 2023): LLama2 is an advanced language model that pro- vides enhanced capabilities a variety applications, including text summarization, translation, and conversational AI. Since it is transformer style decoder-based model, the sequence level embeddings are produced by pooling token level embeddings together"] MetaOOD 0 uses meta-features for dataset embeddings, along with the simple one-hot encoding for model embeddings (see $33 for more details).
Meta0OD_o 0
OpenAl cmb
Uama
emb
omb
MetaOOD
MetaOOD
Rank on AUROC
Rank on AUROC
Figure 4: Ablation study different data and model embeddings. MetaOOD has better per- formance over its variants.
Figure 5: Ablation study on different choices of meta-predictor f. Tree-based models have better performance.
Fig. 4 presents ablation studies results on MetaOOD that uses different embedding variants with f fixed to XGBoost. The results indicate that language models are highly effective at generating em- beddings. MetaOOD, along with the other two language model-generated embeddings, demonstrate solid performance. We observed that the BERT-based embeddings perform slightly better than those derived from LLMs. Fig. shows the dataset embeddings produced by different language models, with the embeddings reduced to 2D using t-SNE. Note that MetaOOD (BERT-based embedding) is slightly better to capture similar datasets (e.g. Texture and Textures) than decoder-based model em- beddings. This could be attributed to the decoder model's causal attention, where the representation of token is influenced only by preceding tokens, making it less effective for text embedding tasks as it restricts capturing information from the entire input sequence (BehnamGGhader et al. 2024).
OpenAl
Figure 6: Visualization of dataset embeddings generated by different language models. Bert- based model performs slightly better (e.g.. Texture and Textures datasets are close).
get
ge
d
gioten store
Usually this is done by averaging the token level embeddings or using the last token. In this study, opt for the latter approach.

---

Published as a conference paper at ICLR 2025
Ablation 2 the Choice of Meta-predictor f. We evaluate the performances of MetaOOD and OpenALeebbNN, which use the language embeddings to train a two-layer MLP meta- predictor. The result is shown in Fig. [ Consistent with the finding in (Jiang et al. 2024 using a tree-based model such as XGBoos! for meta-predictor leads to more stable results and enhanced performance compared initializing with neural network models. Additionally, tree-based models offer superior feature selection capabilities and greater interpretability. In contrast, neural networks, while powerful, can be more sensitive to initialization and hyperparameter choices, leading to less predictable performance. This finding underscores the significance of selecting features of impor- tance during meta-learning especially in complex tasks where stability and performance are critical.
4.4.2
RUNTIME ANALYSIS OF METAOOD
One of the big advantages of MetaOOD over MetaOOD 0 is the fast dataset embedding gen- eration via language models, making the model selection overhead negligible to the actual OOD detection fitting on the image datasets. We demonstrate the time of MetaOOD and the fit- ting of the selected OOD detector on selected dataset pairs in Fig. 7 Notably, the language embedding for both datasets and models and online model selection via f just takes sec- onds to finish. Thus, the MetaOOD only brings the marginal cost to the entire OOD detection pipeline while finding a lop-performing model.
(MetaO0D) model selection Wi data embedsiing (MuraDo)) data embadiing ganeration
Runtime (Seconas)
80
time for trainin the selected model
60-
40-
20
OFARLO-CIFARL00
CIFARID-SVHN CIFARIDD-CIFAROO CIBAR100-SVHN
Figure 7: Runtime of MetaOOD vs. selected O0D detector. MetaOOD incurs small overhead (difference shown with black arrows).
CONCLUSION, LIMITATIONS, AND FUTURE DIRECTIONS
In this work, we introduced MetaOOD, the first unsupervised out-of-distribuiin (OOD) detection model selection framework. This meta-learner utilizes an extensive pool of historical data on OOD detection models and dataset pairs, employing language model-based embeddings to enhance the model selection process based on past performances. Despite its innovative approach, MetaOOD depends on the availability and quality of historical dataset pairs. This reliance may limit its effec- tiveness in scenarios where such data are sparse less similar. Moreover, the current framework is designed for single-modality data, which restricts its application in highly diverse or multimodal environments. Looking ahead, we plan to broaden our testbed to include a more diverse group of datasets and models, thereby enhancing the meta-learning capabilities of MetaOOD. We also aim to extend MetaOOD to support top-k selection, offering a range of viable models rather than a single recommendation. Additionally, equipping MetaOOD with an uncertainty quantification mechanism will enable it to output an "I do not know" response when applicable, further refining its utility in complex scenarios where no suitable meta-train knowledge can be transferred.
BROADER IMPACT STATEMENT, ETHICS STATEMENT, AND REPRODUCIBILITY
MetaOOD revolutionizes OOD detection model selection by enabling practitioners to choose appropriate models for unlabeled tasks automatically. This is particularly crucial in sectors like healthcare, finance, and security, where rapid adaptation new data types can significantly enhance system reliability and prevent critical errors. By providing a systematic approach to select the most effective models, MetaOOD promotes robust applications dynamically changing environments, ensuring ongoing reliability and accuracy in critical systems.
Broader Impact Statement:
Our research adheres to the ICLR Code of Ethics, ensuring that MetaOOD is developed and applied with ethical considerations at the forefront, particularly in privacy, bias, and fairness across diverse applications. By facilitating more accurate and unbiased model selec- tion, MetaOOD helps mitigate potential ethical risks in its deployments, such as in surveillance and healthcare, promoting fairness and protecting privacy. Continuous ethical evaluations accompany MetaOOD's development ensure it meets societal and legal standards.
Ethics Statement:
We advocate reproducibility in MetaOOD. Comprehensive documen- tation of our methodologies and experimental designs is detailed in the main text and appendices. We have made our code, testbed, and leaanee fully accessible at httttt:t///iii yqi n 4 aooo providing the resources for replication and exploration.
Reproducibility Statement:
com/
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work was partially supported by the National Science Foundation under Award No. 2346158. Any opinions, findings, and conclusions or recommendations expressed this material are those of the authors and do not necessarily reflect the views of the National Science Foundation.
REFERENCES
Gabriel Jonas Aguiar, Rafael Gomes Mantovani, Saulo M. Mastelini, André C.P.F.L. de Car- valho, Gabriel F.C. Campos, and Sylvio Barbon Junior. A meta-learning approach for se- lecting image segmentation algorithm. Puttern Recognition Letters, 128:480-487, 2019. ISSN 0167-8655. URL
pii pii 051/001678655193022
Parishad BehnamGhader, Vaibhav Adlakha, Marius Mosbach, Dzmitry Bahdanau, Nicolas Chapa- dos, and Siva Reddy. Llm2vec: Large language models are secretly powerful text encoders, 2024.
URL https: /awiiiiiivv.////a///aaaa///aaavv org/ abs/2404 05961
Abhijit Bendale and Terrance Boult. Towards open set deep networks. In Computer Vision and
Pattern Recognition (CVPR), 2016 IEEE Conference IEEE, 2016.
James Bergstra and Yoshua Bengio. Random search for hyper-parameter optimization. The Journal of Machine Learning Research, 13(1):281-305, 2012.
Julian Bitterwolf, Maximilian Mueller, and Matthias Hein. In or out? fixing imagenet out-of- distribution detection evaluation. In JCML, 2023. URL https::/proceedd mml.
html
Robin Chan, Matthias Rottmann, and Hanno Gottschalk. Entropy maximization and meta classifi- cation for out-of-distribution detection in semantic segmentation. 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 5108-5117, 2020.
Sihan Chen, Zhuangzhuang Qian, Wingchun Siu, Xingcan Hu, Jiaqi Li, Shawn Li, Yuehan Qin, Tiankai Yang, Zhuo Xiao, Wanghao Ye, Yichi Zhang, Yushun Dong, and Yue Zhao. Pyod 2: A python library for outlier detection with llm-powered model selection, 2024. URL https:
12154
Tianqi Chen and Carlos Guestrin. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd acm sigkdd international conference knowledge discovery and data mining, pp. 785-794, 2016.
Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. De- scribing textures the wild. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2014.
T. Cover and P. Hart. Nearest neighbor pattern classification. IEEE Transactions Information Theory, 13(1)211-27, 1967.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Imagenet: A large-scale hier- archical image database. 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 2009.
Li Deng. The mnist database of handwritten digit images for machine learning research [best of the web]. IEEE Signal Processing Magazine, 29: 141-142, 2012. URL httpp::/
Xueying Ding, Yue Zhao, and Leman Akoglu. Fast unsupervised deep outlier model selection with hypernewwrrss. In ACM SIGKDD Conference on Knowledge Discovery and Data Mining. ACM, 2024.
Hao Dong, Yue Zhao, Eleni Chatzi, and Olga Fink. Multiood: Scaling for multiple modalities. Advances Neural Information Processing Systems, 37, 2024.
detection

---

Published as a conference paper at ICLR 2025
Iddo Drori, Lu Liu, Yi Nian, Sharath C. Koorathota, Jie S. Li, Antonio Khalil Moretti, Juliana Freire, and Madeleine Udell. Automl using metadata language embeddings, 2019. URL https:
//arxiv.org/aaaaa.vv. wwt..rgggabaa11.1.1 03698
Xi Fang, Weijie Xu, Fiona Anting Tan, Jiani Zhang. Ziqing Hu, Yanjun Qi, Scott Nickleach, Diego Socolinsky, Srinivasan Sengamedu, and Christos Faloutsos. Large language models(lmms) on tabular data: Prediction, generation, and understanding - a survey, 2024. URL https:
//arxiv.crg/aaab http:/bba2422..11 17944
Angelos Filos, Panagiotis Tigkas, Rowan Mcallister, Nicholas Rhinehart, Sergey Levine, and Yarin Gal. Can autonomous vehicles identify, recover from, and adapt distribution shifts? In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th ICML, volume 119 of Proceed- ings of Machine Learning Research, pp. 3145 PMLR, 13-18 Jul 2020. URL https: //proceedings. mlr.. press / wttp///1111/l/eee html
Yarin Gal and Zoubin Ghahramani. Dropout bayesian approximation: Representing model un- certainty in deep learning. In Maria Florina Balcan and Kilian Q. Weinberger (eds.), Proceedings of The 33rd ICML, volume 48 of Proceedings of Machine Learning Research, pp. 1050-1055 New York, New York, USA, 20-22 Jun 2016. PMLR.
Trevor Hastie, Robert Tibshirani, and Martin Wainwright. Statistical Learning with Sparsity: The Lasso and Generalizations. CRC Press, 2015. ISBN 9781498712163
Xiangnan He, Lizi Liao, Hanwang Zhang, Liqiang Nie, Xia Hu, and Tat-Seng Chua. Neural collab- orative filtering. Proceedings of the 26th International Conference on World Wide Web, WWW '17, pp. 173-182, Republic and Canton of Geneva, CHE, 2017. International World Wide Web Conferences Steering Committee. ISBN 9781450349130.
Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of-distribution examples in neural networks. Proceedings of International Conference on Learning Representa- tions, 2017.
Dan Hendrycks, Mantas Mazeika, and Thomas G Dietterich. Benchmark for out-of- distribution detection and robustness. In International Conference Learning Representations, 2019.
Dan Hendrycks, Steven Basart, Mantas Mazeika, Mohammadreza Mostajabi, Jacob Steinhardt, and Dawn Xiaodong Song. Scaling duttof-distribution detection for real-worke settings. In ICML, 2022.
Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alexander Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. The inaturalist species classification and detection dataset. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8769-
8778, 2017. httpss/ps///pi www.ssemattiioom wtw..sstccacattcilll.......... wowpechoooar.rrrg//orpssssssi
Yue Huang, Chujie Gao, Siyuan Wu, Haoran Wang, Xiangqi Wang, Yujun Zhou, Yanbo Wang, Jiayi Ye, Jiawen Shi, Qihui Zhang, et al. On the trustworthiness of generative foundation models: Guideline, assessment, and perspective. arXiv preprint arXiv:2502.14290 2025.
Gareth James, Daniela Witten, Trevor Hastie, and Robert Tibshirani. An Introduction Statistical Learning. Springer New York, 1 edition, 2013. ISBN 9781461471370.
Minqi Jiang, Chaochuan Hou, Ao Zheng, Songqiao Han, Hailiang Huang, Qingsong Wen, Xiyang Hu, and Yue Zhao. Adgym: Design choices for deep anomaly detection. Advances in Neural Information Processing Systems, 36, 2024.
Hadi S Jomaa, Lars Schmidt- Schmidt-Thieme, and Josif Grabocka. Dataset2vec: Learning dataset meta- features. Data Mining and Knowledge Discovery, 35(3)96449955, 2021.
Serdar Kadioglu, Yuri Malitsky, Meinolf Sellmann, and Kevin Tierney Isac instance algo-
rithm configuration. In ECAI, volume 215 of Frontiers in Artificial Intelligence and Applications, pp. 751-756. IOS Press, 2010. URL http::tpp////// /tct:://dbl..... uni trier.rr... de de/db/con///ccl/ ecaa i 2010 0. html iKadi ooadiogluMMMTI 1MMTT10
12

---

Published as a conference paper at ICLR 2025
Konstantin Kirchheim, Marco Filax, and Frank Ortmeier. Pytorch-ood: A library for out-of- distribution detection based on pytorch. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition (CVPR) Workshops, pp. 1551-4 June 2022.
Alex Krizhevsky. Learning multiple layers of features from tiny images. 2009. URL https:
//api www.ssmmttnccccoo.e...... wwt. http:/CrppsssD... 8268744
Ya Le and Xuan S. Yang. Tiny imagenet visual recognition challenge. 2015. URL https:
/tt.a////0oopS11111677799999000 16664799
Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for detecting out-of-distribution samples and adversarial attacks. In Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018a.
Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for detecting out-of-distribution samples and adversarial attacks. In Advances in Neural Information Process- ing Systems, pp. 7167-7177, 2018b.
Li Li, Wei Ji, Yiming Wu, Mengze Li, You Qin, Lina Wei, and Roger Zimmermann. Panoptic scene graph generation with semantics-prototyye learning. Proceedings of the AAAI Conference on Artificial Intelligence, 3844)3145-31153, Mar. 2024a. doi: 10.1609/aaai, 33884.2809, URL
Lisha Li, Kevin Jamieson, Giulia DeSalvo, Afshin Rostamizadeh, and Ameet Talwalkar. Hyperband: A novel bandit- approach to hyperparameter optimization. The Journal of Machine Learning Research, 18(1):6765-66) 68 6816,88666666666666611 2017.
Shawn Li, Huixian Gong, Hao Dong, Tiankai Yang, Zhengzhong Tu, and Yue Zhao. Dpu: Dy-
namic prototype updating for multimodal out-of-distribution detection, 2024b. URL https:
waaxii.erg/abs///// /
Shiyu Liang, Yixuan Li, and Rayadurgam Srikant. Enhancing the reliability of -uitrf-distribution image detection in neural networks. arXiv: Learning. 2017.
Shiyu Liang, Yixuan Li, and Srikant. Enhancing the reliability -ustr--disttibuttoon image detec- tion in neural networks. In International Conference on Learning Representations, 2018.
Weitang Liu, Xiaoyun Wang. John Owens, and Yixuan Li. Energy-based out-of-distribution de- tection. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 21464-21475. Curran Associates, Inc., 2020a.
Weitang Liu, Xiaoyun Wang, John Owens, and Yixuan Li. Energy-based out-of-distribution de- tection. In Advances in Neural Information Processing Systems, volume 33, pp. 2020b.
Zhendong Liu, Yi Nian, Henry Peng Zou, Li Li, Xiyang Hu, and Yue Zhao. Cood: Concept-based zero-shot ood detection, 2024. URL https:://arxtt: org abs 221121111111111
Martin Ma, Yue Zhao, Xiaorong Zhang, and Leman Akoglu. The need for unsupervised out- lier model selection: A review and evaluation of internal evaluation strategies. ACM SIGKDD Explorations Newsletter, 25(1):19-35, 2023.
Mustafa Misir and Michèle Sebag. Alors: An algorithm recommender system. Artif. Intell., 244: 291-314, 2017.
Yuval Netzer, Tao Wang, Adam Coates, A. Bissacco, Bo Wu, and A. Ng. Reading digits in natural images with unsupervised feature learning. 2011. httpttss: api
U.t.Ltpss//p.....aat........ semantccscool......
org/CorpusTD: 1181:ss1:11111115
Mladen Nikolić, Filip Marić, and Predrag Janičié. Simple algorithm portfolio for sat. Artificial Intelligence Review, 40(4):4574465 2013.
13

---

Published as a conference paper at ICLR 2025
OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, et al. Gpt-4 technical report, 2024. URL PtRLbttps:// /attpps:/aaxxv...o arxaa.iiivv..... awtpvv..rr//b///333...00 08774
Namyong Park, Ryan A. Rossi, Nesreen Ahmed, and Christos Faloutsos. MetaGL: Evaluation-fru selection of graph learning models via meta-learning. In The Eleventh International Confer- ence Learning Representations, 2023. URL https:/ /wwpnreveeeeieeeeeooeeeeee /
net bet/forurm forum?id
CIns08q9:2
Letian Peng, Zilong Wang, Feng Yao, Zihan Wang, and Jingho Shang. Metaie: Distilling model from IIm for all kinds of information extraction tasks. arXiv preprint arXiv:2404.00457, 2024.
Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert- networks, 2019. URL peatttss:/aariiin http///axvvvv. http://rgggbbbs//seego /tt.
10084
Bobak Shahriari, Kevin Swersky, Ziyu Wang, Ryan P Adams, and Nando De Freitas. Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE, 104(1):
148-175, 2015.
Zhensu Sun, Li Li, Yan Liu, Xiaoning Du, and Li Li. On the importance of building high-quality training datasets for neural code search. In Proceedings of the 44th International Conference on Software Engineering. ICSE '22, pp. 1609-1620, New York, NY, USA, 2022. Association for Computing Machinery. ISBN 9781 47034223. doi: 10.1 (0.11445/510000335511060 URL https:
//doi. 10. 1et....11513510000 3510160
Robert Tibshirani. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society: Series B (Methodological), 58(1):2677-) 1996.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, et al. Llama 2: Open foundation and fine-tuned chat models, 2023. URL https://aax.lvvor///bbbb//
09288
Dennis Ulmer, Lotta Meijerink, and Giovanni Cinà. Trust issues: Uncertainty estimation does not enable reliable ood detection on medical tabular data. In Machine Learning for Health, pp. 341- 354. PMLR, 2020.
Joaquin Vanschoren.
Meta-learning: A survey, arXiv preprint arXiv:1810.03558. 2018.
Sagar Vaze, Kai Han, Andrea Vedaldi, and Andrew Zisserman. Open-set recognition a good closed- set classifier is all you need? In International Conference on Learning Representationss, 2022.
Haoqi Wang, Zhizhong Li, Litong Feng, and Wayne Zhang. Vim: Out-of-distribution with virtual- logit matching, In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, 2022.
David H Wolpert and William G Macready. No free lunch theorems for optimization. IEEE trans- actions on evolutionary computation, 1(1):67-82, 1997.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: novel image dataset for bench- marking machine learning algorithms. ArXiv, abs/1708.07747, 2017. URL https://app.
semanticschon org/ Corpus ID: 702279
Lin Xu, Frank Hutter, Jonathan Shen, Holger Hoos, and Kevin Leyton- Broton Satzilla2012: Improved algorithm selection based cost-sensitive classification models. Proceedings of SAT Challenge, pp. 57-58, 2012.
Feng Xue, Zi He, Yuan Zhang, Chuanlong Xie. Zhenguo Li. and Falong Tan. Enhancing the power of ood detection via sample-aware model selection. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 17148-17157, 2024.
Jingkang Yang, Kaiyang Zhou, Yixuan Li, and Ziwei Liu. Generalized out-of-distribution detection: A survey. arXiv preprint arXiv:21 10. 1 2021.
14

---

Published as a conference paper at ICLR 2025
Jingkang Yang, Pengyun Wang, Dejian Zou, Zitang Zhou, Kunyuan Ding, Wenxuan Peng, Haoqi Wang, Guangyao Chen, Bo Li, Yiyou Sun, et al. Openood: Benchmarking generalized out-of- distribution detection. Advances in Neural Information Processing Systems, 35:32598-32611 2022.
Y. Zhao, S. Zhang, and L. Akoglu. Toward unsupervised outlier model selection. In 2022 IEEE International Conference on Data Mining (ICDM), pp. 773-782, Los Alamitos, CA, USA, dec 2022a. IEEE Computer Society.
Yue Zhao. Towards reproducible, automated, and scalable anomaly detection. In AAAI, volume 38, pp. 22687-22687, 2024.
Yue Zhao and Maciej K Hryniewicki. Deso: dynamic combination of detector scores for outlier ensembles. arXiv preprint arXiv:1911.104/6. 2019.
Yue Zhao, Zain Nasrullah, and Zheng Li. Pyod: A python toolbox for scalable outlier detection. Journal of machine learning research, 20196))1-7,2011
Yue Zhao, Ryan Rossi, and Leman Akoglu. Automatic unsupervised outlier model selec- tion. M. Ranzato, Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 4489-4502. Cur- ran Associates, Inc., 2021. URL
palllppppeeaps///a/palpapppeeese files/ pdf
Yue Zhao, George H Chen, and Zhihao Jia. Tod: Gpu-accelerated outlier detection via tensor oper- ations. Proceedings of the VLDB Endowment, 16(3):546-560, 2022b.
Bolei Zhou, Àgata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. JEEE Transactions on Pattern Analysis and Ma- chine Intelligence, 40:1452-1464, 2018. URL https: / /api....... senanticccholar..orr.
CorpusiDD:2609922
15

---

Published as a conference paper at ICLR 2025
SUPPLEMENTARY MATERIAL FOR METAOOD
A DETAILS ON METAOOD
A.1 PSEUDO-CODE FOR META-TRAIN AND ONLINE MODEL SELECTION
We discussed meta-training and online model selection 3.3 and $3.4 respectively. Here the pseudo-code for the two phases.
Algorithm Offline OOD detection learnee training
Input: Meta-train database Duain- model set M Output: Meta-learner f for OOD detection model selection
1: Train and evaluate M on Dirin to get performance matrix
2: for i E (
n do 3: Extract data embedding E, (D))
4: for j E (I,.... do
5: Encode methods set as L.Tuudel 6(M,M)
6: Train f by Eq. (I) with the j-th model the i-th dataset
7: end for
8: end for
9: return the meta-learner ʃ
Algorithm 2 Online OOD detection model selection
Input: the meta-learner New ID-OOD dataset pair Dnew Output: Selected model for Dnew
Extract data embedding. Liess := (Dnow)
2: for j E ( m (for clarity, written as a for loop) do
3: Encode methods set as Eypoddl My,
4: Predict the j-th model performance by the meta-learner f, i.e., P new. = f [EEae Empodel)
5: end for
6: return the model with the highest predicted perf. by Eq.
A.2 DETAILS ON NOTATIONS
The following notations are used in Fig. which provides comprehensive MetaOOD overview
3.2
Notations
Description
Training Loss
M
# OOD Detection Methods
N
# Dataset Pairs
Embedding Notation for OOD Detection Methods Embedding Notation for Dataset Pairs
Pij
Performance of OOD Detection Method j on Dataset Pair i
Pr.
Predicted Performance of OOD Detection Method j on Dataset Pair Meta-pread Peta--reeiccto Input Sample
X
Table A: Notations with details used in Fig.
A.3 DETAILS ON META-FEATURES
The classical way of data embedding via meta- meta-features is discussed in $33.1 Tab. B complete list of meta-features constructed in this study.
presents the
16

---

Published as a conference paper at ICLR 2025
Category
Description
Variants
Statistical Features
Sample features
n. X, o², minx, maxx, o percentile P; 975 q25
mean, median, var, min, max, std q1. q25, 975, q99
IQR
normalized mean, normalized median sample range, sample gini
maxx maxy maxx minx, Gini(X)
median (X X)
median absolute deviation average absolute deviation
xxg(X X)
475-925 925
Quantile Coefficient Dispersion
Coefficient of variance normality
If a sample differs from normal dist.
5th to 10th moments
skewness,
skewness, kurtosis Co-occurence Matrix mean, of the HSV channel
M color-based
Image features
color-based
std the intensity channel entropy of the RGB channel std of the (RGB, HSV, intensity) channel Average white pixels Average Hu Moments of sobel image
color-based
histogram
border
border
texture
(Co-occurence Matrix)
contrast mean, std dissimilarity mean, std homogeneity mean, std energy mean, std correlation mean, std entropy mean, std entropy mean, std inertia mean, std energy mean, std homogeneity mean, std num samples, num of features dim, num of class Earth Mover's 's Distance
(Co-occurence Matrix)
(Co-occurence Matrix)
(Co-oecurence Matrix)
(Co-occurence Matrix)
(Co-occurence Matrix)
(FFT)
(FFT)
(FFT)
(FFT)
Dataset features
n.
d,
EMI)
Landmarker Features
Probability-
softmax probability
mean, std, min, max entropy, range
topl softmax probability top2 softmax probability considence margin skewness, kurtosis
Table Selected MetaOOD features. Part of the statistical features are based on Vanschoren (0088;; Zhao et al. (2021), and specialized landmarker features are newly designed for OOD detection. See $3. for more details.
17

---

Published as a conference paper at ICLR 2025
B ADDITIONAL EXPERIMENT SETTINGS AND RESULTS
B. PROMPTS LLM FOR SERO SELECTION OF THE OPTIMAL OOD DETECTOR
Section $4. discusses the baseline GPT-4o mini, where LLM is used as zero-shot meta-predictor. In this baseline, the prompt provided the LLM is structured follows, with text descriptions of both the datasets and models provided. To ensure consistency, we set temperature parameter to 0, and top p parameter 0.999.
[Dataset descriptions provided]
Your task is to select the best OOD detection method for set of 24 test ID-OOD dataset pairs. You will be provided with descriptions of both the ID-OOD dataset pairs and the available OOD detection methods. You should pick the best model that has the highest AUROC metric. For each dataset pair, output the recommended OOD detection method in the format: 'Recomnended Method: [Recommended Method]'.
[Model descriptions provided]
B.2 FULL PERFORMANCE RESULTS
Below is the full performance matrix P, which shows the performance of eleven OOD detection models (refer to Tab. M on our constructed ID-OOD dataset pairs (see dataset pair construction in We report AUROC the metric for OOD detection.
OOD Datasel
ID Datasel
CIFAR-I0 CIFAR- 100
MNIST Places365
SVHN
Textune TIN
SSB.hard NINCO INaturalisi Textures Opentmage-O
OOD Detectoo
CIFAR-TO
Upennax
N/A
90.68
V3.95
91.17
93.31
92.71
19.67
82.33
91.35
V0.07
92.35
91.36
MCT)
N/A
88.47
92.88
88.31
90.31
67.22
86.91
80.15
88.59
85.39
87.37
87.88
ODIN
65.91
88.58
67.71
72.44
63.28
58.85
65.52
50.47
72.59
62.08
Mahulanobis
BasegyBased
NIA
87.25
97.45
90.12
91.56
85.42
86.60 78.43
87.57
81.52
85.30
36.94
Entropy
N/A
$8.29
94.05
$3.91
92.55
88.91
86.97
79.08
$8.42
81.97
88.95
$7.58
MaxLogit
N/A
57.29
97.27
90.04
91.58
85.55
86.54
78.44
87.55
81
85.44
86
KLM
83.78
90.95
$4.96
85.64
83.36
74.72
$4.47
76.89
65.79
$3.43
MSP
87.95
93.27
83.45
92,09
58
86. 86.59
88:05
84.78
62
87
KNN
N/A
83.
92.47
89.94
92.62
91.20
87.53
80 7M
6%.61
84.01
91.29
89 .5
CIFAR-10
Openmax
77,40
N/A
81.30
83.77
93.71
87.35
81.11 75.60
75.10
71.31
B7.37
79.23
MCD
ODIN
Mahalanobis
66.51
N/A
73.35
70.13
85.21
89.84
71.04 71.46
72.87
75.63
90.20
74.73
Energy Based
77.15
NAA
93.92
78.20
72.49
76.20
80.63
75.64
76.65
74.16
76.05
74.10
Entropy
76.45
NAA
86.37
77.30
71.70
74.93
80.17 90
77. 77.98
77.90
74.79
75.37
MaxLagit
77.32
75.79
VIM
69.50
N/A
86.06
71.14
81.81
88.60
77.14 73.32
77.39
75.32
89.12
72.50
MSP
75.10
NAA
83.01
75.85
70.02
73:41
78.41 74.48
76.37
77.13
73.26
74.21
KNN
71.53
NAX
67.42
70.96
83.29
79.73
79.38
74.96
75.20
73.38
79.99
71 44
TmugoNei
Openmax
MCT>
82.94
79.40
97.70
79.16
ODIN
98.74
97.85
96.56
64.67
99.80
70.86
83.64 67.59
69 10
74.27
72.09
71
Mahalanokis
82,35
$0.45
88.25
58.25
72.91
89.95
90.72 47.91
62.07
61.18
89.78
70.54
Energy Rased
85.13
86.78
96.79
82.45
98.35
86.74
80.97 72.35
79.70
90.59
88.71
80.
Entropy
85.72
88.14
MaxLagit
85.68
KLM
87.73
$8.95
91.49
78.84
97.37
82.39
82.33 68.60
79.82
89.54
84.07
85.0 $5.95
VIM
91.34
941.26
95.89
77.52
95.10
91.82
87.21
63.83
77.53
87.23
92.51
$7.05
MSP
82.94
83.63
89.74
79.40
97.70
80.46
79.16
72.
79.9 97
88
82.47
84
KNN
71.37
FashionNNIST
Openmax
92.18
92.27
MCD
86.05
86.88
85.65
89.17
93.85
$6.77
98.43 87.74
$7.88
89.06
86.59
88.67
ODIN
57.52
59.33
$5.66
61.53
89.01
60.94
59.83 60.88
61.89
47.23
61.05
$7.55
Mahalanckis
99.46
99.44
97.22
99.35
99.90
99.75
99.33
99.17
99.
99.63
99,77
99.44
Baseg
Entropy
85.69
81.76
89.00
94.27
86.97
88.21
K7.65
88
MaxLogit
91.22
92.26
79.32
93.94
98.24
91.35
93.16 92.99
92.81
93.51
91.22
93.31
KLM
71.28
73.85
64.34
79.02
91.84
77.16
77.19
75.26
75.87
76.58
77.12
77.78
VIM
96.06
96.
80.93
96 84
99.22
97.69
96. 16.65
96.86
97.44
47.71
97.03
MSP
84.72
97.71
97.72
97.36
98.14
Table C: Various OOD detection models' performance on ID-OOD dataset pairs. See experiment
setting in 4.1
We highlight the selected OOD method for each dataset the test set in bold.
18

---

Published as a conference paper at ICLR 2025
B.3 DATASET DESCRIPTIONS
Below is the full list of the dataset descriptions use in the study, which includes basic dataset metadata such as the types of objects in each dataset, the image type, and the size of the dataset. This information can be compiled swiftly and easily.
Dataset CI
Description
Contains images of 10 types of objects, including airplanes, cars, birds, cats, deer, dogs, frogs, horses, ships, and trucks. Each image is a small 32x32 RGB image.
CIFAR-I00
Similar to CIFAR-I0 but includes 100 classes grouped into 20 superclasses.
Each class contains images of specific objects, animals, or people.
MNIST
Comprises black-and- white images of handwritten digits from () to 9, with each image normalized to in 28x28 pixel bounding box.
Places365
Features images of various places like homes, parks, offices, and cities, spread
across 365 different scene categories.
SVHN
Includes images of digit sequences found in natural scene images, primarily focusing on house numbers from street view images.
Texture
Generally consists of images depicting various surface textures such as bricks, tiles, fabrics, and other patterned surfaces.
TIN
Contains tiny 32x32 images sourced from the internet, depicting wide variety of everyday objects, animals, and scenes.
SSB-Hard
Contains images of specific breeds of birds, types of aircraft, and models of cars.
NINCO
Contains natural scenes, textures, and objects, providing diverse of im- ages.
iNaturalist
Consists of images submitted by users of various species of plants, animals, and fungi, used for species identification and classification.
Textures
Focuses on images that capture the surface quality of various materials.
Openlmages
Features wide range of images with various objects, scenes, and activities, annotated with labels and bounding boxes.
ImageNet
Contains diverse images across a wide range of categories like different types of animals, plants, vehicles, and everyday objects.
Fashion-MNIST
Includes images of fashion products from 10 categories, such as shirts, dresses, shoes, and bags. Each image 28x28 grayscale image.
Table D: Dataset descriptions used in the study, which contain basic information of the dataset such as dataset content (e.g., what kind of objects are in the dataset), image type, and dataset size.
B.4 FEATURE IMPORTANCE OF LANGUAGE EMREDDINGS
Fig. shsowss the feature importance of the language embeddings, where the F score of feature is the total number of times the feature is used to split the data in all trees in the model. We show the top- 10 most important features. According the analysis, the initial dimensions of these embeddings carry greater significance.
Embeddings
ID dataset
OOD dataset
Method
Metric
Weight
0.53
0.52
0.37
Cover
7.27e-07
2.88e-07
7.08e-07
Gain
0.026
0.025
0.018
Table E: Average feature importance for different language embeddings on the Weight, Cover, and Gain metries. Dataset embeddings play a more significant role in the selection process compared to method embeddings.
19

---

Published as a conference paper at ICLR 2025
Feature Importance
dimi
408.0
00D 300m
Method_dime dimb
281.0
Method diml
112.0
Features
ID dim1
ODD dim1
101.0
81.0
00D dim3
71.0
Method_dine dim3
66.0
100
150
250
300
Fcoore
Figure A: Top-10 feature importance of the language embeddings used in the study; dimensions from the ID dataset are labeled as ID ID.dim, those from the OOD dataset as DOD_dim, and from the Method Method .im. The initial dimensions of language embeddings the most critical.
Tab. B presents the average feature importance based on three metries: 1) Weight: the number of
times a feature is used to split the data across all trees, 2) Cover: the average coverage across all splits the feature is used in, and 3) Gain: the average gain across all splits the feature is used in the ID dataset embeddings. These metrics are computed for the ID dataset embeddings, OOD dataset embeddings, and method embeddings. The results indicate that dataset embeddings have greater impact on the selection process compared to method embeddings.
20

---

Published as a conference paper at ICLR 2025
C FURTHER STUDY ON EMBEDDINGS
C.I COMBINED META-FEATURE AND LANGUAGE EMBEDDING
Some might argue that relying solely on language embeddings could be overly basic. To address this, we also experimented with combining statistical and landmarker meta-features alongside lan- guage embeddings for dataset and method embeddings (denoted as Combined). As shown Fig. [B] this approach achieves performance comparable to MetaOOD, However, the computational and time costs of extracting statistical features can be substantial, particularly for large-scale datasets like Im- ageNet and LSUN. Language embeddings were chosen primarily for their reproducibility. efficiency, and significantly lower computational overhead.
C.2 VARIATION OF DATASET DESCRIPTIONS
To test the robustness of language embeddings against different input descriptions of datasets, manually adjust dataset descriptions by rephrasing them to alter the wording while preserving the original meaning, as detailed in Tab. F This variation is referred as MetaOOD' According to Fig. C changes in dataset descriptions containing basic information do not lead to significant differ- ences, and performance remains stable.
00.
Model Selectioon Methodd
M5F
Methods
M2
NS
A5
A5
ALORS
G8
Retco0
VetxCaa
GF144
GFT-le mh
12
31
15
Rank AUROC
Figure C: Boxplot of the rank distribution of variation of dataset descriptions and baselines. Variations in dataset descriptions, which in- clude basic information, do not lead to signifi- cant differences, and performance remains sta- ble.
Figure B: Boxplot of the rank distribution of combined meta-feature and language embed- ding, and baselines. The performance is com- parable to that of MetaOOD.
21

---

Published as a conference paper at ICLR 2025
Dataset CIFAR-10
Description
A standard dataset in computer vision, containing 60,000 color images (32x32 pixels) of 10 distinct object categories: airplanes, cars, birds, cats, deer, dogs, frogs, horses, ships, and trucks.
CIFAR-100
An extension of CIFAR-10. featuring 100 classes grouped into 20 broader perclasses. includes 60,000 color images (32x32 pixels), each class repre- senting finer granularity of objects, animals, or people.
MNIST
A seminal dataset comprising 70,000 grayscale images (28x28 pixels) of hand- written digits (0-9), each centered a bounding box.
Places365
Contains 1.8 million images spanning 365 distinct scene categories, including various indoor and outdoor environments like homes, parks, offices, and cities.
SVHN
A real-world image dataset featuring over 600,000 images of digit sequences (32x32 pixels) extracted from house numbers in street view imagery.
Texture
Composed of diverse images highlighting surface textures such as bricks, tiles, fabrics, and other patterned materials, often used to study material properties.
TiN
A massive dataset of 80 million tiny images (32x32 pixels) collected from the web, representing broad spectrum of everyday objects, animals, and scenes.
SSB Hard
Features a diverse collection of images, including specific bird species, distinct types of aircraft, and detailed models of cars.
NINCO
Comprises wide range of images, including natural scenes, intricate textures, and various objects.
iNaturalist
Comprises over 859,000 images of various species of plants, animals, and fungi, contributed by users for species identification and classification.
Textures
Similar to the Texture dataset, but may have a different focus or a wider range of surface textures, including natural and artificial materials.
OpenImages
A vast dataset containing 9 million images annotated with labels, bounding boxes, and object segmentation across diverse objects, scenes, and activities.
ImageNet
One of the most comprehensive datasets, with over 14 million images catego- rized into 1,000 classes, including animals, plants, vehicles, and various every- day objects.
Pashion-MNIST
A modern alternative to MNIST, containing 70,000 grayscale images (28x28 pixels) of fashion items categorized into 10 classes, such as shirts, dresses, shoes, and bags.
Table F: Manually modified dataset descriptions used in additional study C.2 which contain basic information of the dataset such as dataset content (e.g., what kind of objects are in the dataset), image type, and dataset size.
22


Reviews:
[{"review_id": "3eCYPvM311", "reviewer": "Reviewer_nA6W", "summary": "In this paper, the authors propose MetaOOD, which utilizes meta-learning to select an OOD detection model automatically. The motivation is that each OOD detection algorithm might excel in specific scenarios but may not perform well universally, therefore it is important to select one particular OOD detection for each task. MetaOOD utilizes historical performance data of existing methods across a variety of benchmark out-of-distribution (OOD) datasets to enable efficient model selection for new datasets, eliminating the need for labeled data at test time. To more accurately measure task similarities, the authors incorporate language model-based embeddings that capture the unique OOD characteristics of both datasets and detection models. Through extensive testing across 24 unique test dataset pairs and 11 OOD detection models, the authors show that MetaOOD consistently outperforms current methods with minimal additional computation time.", "questions": "1. Does the proposed method rely on the architecture of the trained model?\n2. What is the training time of the proposed method?\n3. If there is one additional OOD method, how can incorporate this method into the proposed MetaOOD?\n4. What are the main factors that influence the choice of an OOD method based on the characteristics of the training and test sets?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The idea of using meta-learning to select the best OOD detection method for each specific task is interesting.\n2. The paper is generally easy to understand and clearly written.\n3. The experiments show the effectiveness of the proposed method.", "weaknesses": "1. Figure 1 needs to be improved. The notations in the figure are confusing and unclear.\n2. The design of the textual description seems ad-hoc and cannot be applied in the case of without detailed dataset information.\n3. Detailed results on the selected OOD method for each dataset are missing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6FhfDi1F1q", "reviewer": "Reviewer_bKXx", "summary": "This paper presents MetaOOD, a “model” selection approach for out-of-distribution (OOD) detection. MetaOOD utilizes language models to generate feature embeddings of both the meta dataset and “models”, allowing for the optimal “model” selection based on anticipated performance on the test set. The results on the Wilcoxon statistical tests show the promising performance of MetaOOD.", "questions": "NA", "limitations": "", "rating": 3, "confidence": 4, "soundness": 1, "presentation": 1, "contribution": 1, "strengths": "1. The motivation is sound. It is interesting to see a meta-selection approach to the OOD detection problem since there are so many methods in this OOD domain.\n2. The proposed method is simple and straightforward. The results on the traditional methods are promising.", "weaknesses": "1. The definition of \"OOD model\" is confusing. There are many post-hoc detection methods in the detection problem, which should not be classified as “models”. For instance, the paper includes the MSP method for the selection experiments. However, MSP is just a simple post-hoc technique that can be applied to most classification models (e.g., ResNet) using the SoftMax function. This method should not be considered as a model, which is misleading considering another factor, “model architecture,” in the experiments.\n\n2. The methodology lacks depth. MetaOOD merely utilizes language models to extract embeddings for dataset and model descriptions, and then select the top-1 method based on these embeddings. The approach lacks insight and overlooks potential issues. For instance, the embeddings derived from descriptions may not accurately capture the true characteristics of the models and datasets. Also, simply selecting the top 1 can overlook the nuances of methods and the potential problems of the utilized datasets.\n\n3. The experimental results are unconvincing. The baseline methods included are outdated, with the most recent method (NCF) dating back to 2017.\n\n4. The terms OOD and OOD detection should not be used interchangeably. It is unclear what is meant by \"OOD dataset\" given such a name strategy. Is it referring to a commonly recognized OOD dataset distinct from the in-distribution (InD) dataset, or simply an OOD detection dataset (includes train, val, and test splits for detection methods)? \n\n5. I am curious whether this paper was generated by a language model, such as GPT-4. The writing style, particularly in Section 3.3.1, resembles AI-generated text. Given the simplicity of the method, the Method Section could be more concise, potentially requiring only 0.5 pages to convey the core elements of the approach. However, the current version spans 2.5 pages.\n\n**Post-rebuttal Comments**\n\nI'd like to thank the authors' response. I agree that weakness 5 could be too strong as I assumed the method may be generated from GPTs. However, the length and the redundancy of the method section are still a problem. I still cannot accept that the choice of embedding strategy could occupy half of the Method Section, given that the embedding method is that simple.\n\nI also agree that the simplicity could be the advantage of the proposed method. However, I still cannot view this method as sufficiently advanced or novel for a research paper at ICLR. As I've mentioned, this method lacks depth, not to mention the absent theoretical insights. The approach merely utilizes the rough descriptions of the dataset and models to predict the score on the OOD dataset. The performance relies heavily on LLMs, which can introduce a series of problems and are heavily limited to the application scenarios. \n\nAlso, I acknowledge that the paper may present the \"first\" selection strategy for the OOD detection method. However, this should not be an excuse for the obvious shortcomings. Overall, given current experiments, I consider this paper more of an exploration of leveraging LLMs in OOD detection rather than a substantial research contribution. Thus, I will maintain my original score.\n\n**Additional Post-rebuttal Comments**\n\nAfter reviewing the authors' responses, I still consider this paper as a preliminary exploration of leveraging LLMs in OOD detection. So I will maintain my original score.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TmgAfHI9U9", "reviewer": "Reviewer_N69E", "summary": "The paper postulates that by identifying which OOD detection models have historically performed well on datasets similar to the one currently being considered, one can select the model most likely to be effective without needing labels for supervised training.  A meta-learning approach is used to take past performance data from various models (across data sets); when new dataset arrives, the approach checks for similarity between the new dataset and historical ones using embeddings. The assumption is that the selected model will perform well as it is closest to the data set under use.  Meta-learning (training) is done offline with curated data sets; while OOD model selection is online as a specific data point arrives.  Results show that their approach works better than compared other techniques.  Experiment approach itself is reasonable and approach is sound.", "questions": "Can the authors add a diagram to better illustrate how this technique would work in practice? This will help translatability of this work into other contexts faster.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "Integration of language model based embeddings; Empirical eval. is done to good detail. this technique is actually useful. though it is a logical next step ~ the approach itself can be used in other contexts or at-least the idea can be adapted. Sufficient detail is provided makes work transparent.", "weaknesses": "Eval is too narrow and limited ~ so results may not generalise or this approach may be limited to the data set / domain attempted; esp. since there is no formal conceptual development as such we do not know when and where this method will work or have an intuition for where the limit may be.  Although there is the claim of unsupervised world-first etc. ~ there is still a need for other forms of supervised and curated training. Assumes text descriptions are good in the evals and curated data sets - but does it hold for the real world?  All the usual limits of language models apply here too. Scalability not known -- again since we do not have specific underlying theory.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "I2twsLv0LZ", "reviewer": "Reviewer_kAtw", "summary": "The paper presents MetaOOD, a framework for automatic selection of out-of-distribution (OOD) detection models without requiring labeled data. It leverages historical performance data and language model embeddings. The approach aims to improve the reliability of OOD detection in critical applications, such as autonomous driving and online transactions. Overall, MetaOOD addresses the challenge of adapting to data shifts effectively and efficiently.", "questions": "No questions.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The paper's strengths include the introduction of a zero-shot, unsupervised framework for OOD detection model selection, which enhances adaptability to new datasets. It effectively utilizes language model-generated embeddings to capture nuanced dataset characteristics, improving model selection accuracy. The extensive experimentation demonstrates superior performance compared to eleven established methods, showcasing its robustness. Additionally, the framework incurs minimal runtime overhead, making it efficient for practical applications. The use of the Wilcoxon signed-rank test is a plus of the paper. The p-values suggests the proposed approach works well.", "weaknesses": "The paper's weaknesses include a reliance on the quality of language model embeddings, which may vary based on the model used and the nature of the input data. Additionally, the framework's performance may be limited by the diversity of the historical data pool, potentially affecting generalization to unseen datasets. The lack of extensive real-world testing could raise concerns about its applicability in practical scenarios. Lastly, the complexity of the approach may pose challenges for reproducibility and implementation in different contexts. Obtain datasets and model feature/embeddings from their textual descriptions appear a bit strange and somewhat unreliable.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)