MetaOOD: Automatic Selection of OOD Detection Models

## Abstract

Abstract How can we automatically select an out-of-distribution (OOD) detection model for various underlying tasks? This is crucial for maintaining the reliability of open-world applications by identifying data distribution shifts, particularly in critical domains such as online transactions, autonomous driving, and real-time patient diagnosis.
Despite the availability of numerous OOD detection methods, the challenge of selecting an optimal model for diverse tasks remains largely underexplored, especially in scenarios lacking ground truth labels.
In this work, we introduce MetaOOD, the first zero-shot , unsupervised framework that utilizes meta-learning to select an OOD detection model automatically.
As a meta-learning approach, MetaOOD leverages historical performance data of existing methods across various benchmark OOD datasets, enabling the effective selection of a suitable model for new datasets without the need for labeled data at the test time.
To quantify task similarities more accurately, we introduce language model-based embeddings that capture the distinctive OOD characteristics of both datasets and detection models.
Through extensive experimentation with 24 unique test dataset pairs to choose from among 11 OOD detection models, we demonstrate that the MetaOOD significantly outperforms existing methods and only brings marginal time overhead.
Our results, validated by Wilcoxon statistical tests, show that MetaOOD surpasses a diverse group of 11 baselines, including established OOD detectors and advanced unsupervised selection methods.

## 1 Introduction

Out-of-distribution (OOD) detection
is the process of identifying data points that deviate significantly from the distribution of the training data. This capability is essential for ensuring the reliability of machine learning (ML) models when they encounter new, unseen data (Yang et al., ).
Common applications of OOD detection include safety-critical systems like autonomous vehicles (Filos et al., ) and medical diagnosis (Ulmer et al., ) to prevent erroneous predictions.
Notably, many OOD detection algorithms are unsupervised due to the high annotation cost, leveraging statistical methods or reconstruction errors from models such as autoencoders to identify deviated samples without labeled OOD data (Yang et al., ; Dong et al., ).
However, despite the variety of OOD detection algorithms, each adepts at identifying different aspects of in-distribution (ID) and OOD data, there lacks a systematic method for choosing the best OOD detection model(s) under the unsupervised setting without labels.
It is acknowledged that each OOD detection algorithm might excel in specific scenarios but may not perform well universally due to the no-free-lunch theorem (Wolpert & Macready, ).
Moreover, without labels, it is difficult to objectively evaluate and compare the performance of different OOD detection models.

Present Work.
In the most related field, unsupervised outlier detection (OD) has benefitted significantly from meta-learning, with models like MetaOD (Zhao et al., ), ELECT (Zhao et al., ), ADGym (Jiang et al., ), and HyPer (Ding et al., ) demonstrating notable advancements.
By leveraging historical performance data, these methods estimate a model’s efficacy on new datasets. However, they are not directly adaptable to OOD detection due to several key differences:
First, OD and OOD detection differ in their problem settings. OD models are trained on both normal and outlier samples (Zhao et al., ; Ma et al., ), focusing on anomalies within a single distribution (Zhao et al., ; Zhao, ). In contrast, OOD detection involves training solely on in-distribution (ID) data and aims to identify samples from entirely different distributions, often across multiple datasets.
Second, OOD detection tasks primarily deal images, presenting greater complexity compared to time series and tabular data in OD.
Third, the embeddings used to measure dataset similarity in OD do not translate well to OOD contexts. Current OD embedding generation are largely heuristic and feature-specific (Zhao et al., ), inadequate for the diverse and complex nature of OOD datasets, especially when considering the different data modalities.
Thus, addressing these challenges with a tailored method
for OOD model selection is crucial.

Figure: Figure 1: MetaOOD overview (§3.2); offline meta-training phase is shown on the top (§3.3)–the key is to train a meta performance predictor $f$ (denoted in ) to map language embeddings of the datasets and models to their performance $\mathbf{P}$; the online model selection (§3.4) is shown at the bottom by transferring the meta-predictor $f$ to predict the test data paired with OOD models for selection.
Refer to caption: x1.png

Our Work.
In this study, we introduce MetaOOD, the first unsupervised OOD
(^1^11For brevity, we may refer OOD detection as OOD throughout the paper.)
model selection method that employs meta-learning. Figure 1 illustrates the overall workflow of the proposed approach.
The key rationale is that an OOD model is likely to perform well on a new dataset if it excels in a similar historical dataset.
As a meta-learning approach, we train a suite of OOD detection models offline using a variety of carefully curated datasets to gauge their performance across different scenarios during the meta-train phase (Figure 1, top).
When a new dataset arrives, we apply knowledge derived from historical data to select an appropriate OOD model (Figure 1, bottom).
This selection is based on the similarity between the new dataset and those used in the meta-train phase.
To enhance the accuracy of this similarity assessment, we designed two versions of data embeddings via specialized OOD meta-features and language embeddings from language models.
As our experiment shows, these language model-based embeddings effectively capture complex, nuanced dataset characteristics that traditional meta-features might miss.
We summarize our technical contributions:

- •
First OOD Detection Model Selection Framework. We introduce the first meta-learning-based framework for zero-shot OOD detection model selection without training and evaluation.
- •
Specialized Embeddings for OOD Datasets and Models. We use language model-generated features to quantify the similarity among OOD tasks, facilitating a better understanding of OOD characteristics and enhancing OOD detection model selection.
- •
Effectiveness. The proposed MetaOOD outperforms eleven well-known model selection methods and unsupervised meta-learners on a testbed with 24 unique test data pairs.
It is superior and statistically better than all the baselines w.r.t. average rank, and efficient with small runtime.
- •
Accessibility and Reproducibility. We release the testbed, corresponding code, and the proposed meta-learner at [https://anonymous.4open.science/r/metaood-DA67](https://anonymous.4open.science/r/metaood-DA67).

## 2 Related Work

### 2.1 Unsupervised OOD Model Selection

OOD detection faces the challenge of both OOD samples and OOD distributions being unknown during the training phase (Hendrycks et al., ; Liang et al., ).
Consequently, selecting models for OOD relies solely on in-distribution or training data (Lee et al., ).
Once deployed in an open-world setting, the OOD detector often handles a diverse range of test inputs originating from various OOD distributions (Yang et al., ),
and consequently, unsupervised methods are expected to be more suitable for selecting OOD detection models (Liu et al., ).
Within these unsupervised OOD detection models, some recent works focus on actively choosing sample-wise detectors (e.g., choosing different groups of detectors for different input samples) (Xue et al., ), while it is not doing actual model selection at the single model level as this work tries to address.

Most existing methods rely on trial-and-error or empirical heuristics.
For instance, the simplest approach may be not to actively “select” a model, but just use popular OOD detection models like Maximum Softmax (Hendrycks & Gimpel, ) and ODIN (Liang et al., ).
There are other simple approaches; the confidence level of the ID data may be used as an indicator for OOD detection model selection.
However, such a simple method may lead to a low true positive rate (TPR) (Hendrycks & Gimpel, ).
Another direction is similarity-based methods,
where the model selection is based on the similarity or cluster of the dataset, which has also been utilized for algorithm recommendation
(Kadioglu et al., ; Nikolić et al., ; Xu et al., ; Misir & Sebag, ).
We thus include these algorithms as our baselines in §4.

### 2.2 Supervised OOD Model Selection

There has been considerable research into the use of supervised model selection techniques for effectively training and evaluating predictive models on labeled datasets (James et al., ; Hastie et al., ; Tibshirani, ). These methods are particularly valuable in scenarios where generalization to unseen data is critical. Model selection in supervised learning encompasses a variety of strategies, including cross-validation, grid search, and performance metrics such as accuracy and F1 score to assess model efficacy. Randomized (Bergstra & Bengio, ), bandit-based (Li et al., ), and Bayesian optimization (BO) techniques (Shahriari et al., ) are various SOTA approaches to hyperparameter optimization and/or algorithm selection.
However, it is notable that such methods do not apply to unsupervised OOD model selection, as ground truth values are absent.

### 2.3 Represent Datasets and Models as Embeddings for Meta-learning

In terms of data representation, especially when it comes to meta-learning, embeddings play an important role to measure dataset/task similarity. Traditionally, computational-based meta-features are used as data representation in the meta-learning process (Vanschoren, ). More recently, more sophisticated, learning-based meta-features have been designed, as in dataset2vec (Jomaa et al., ) and HyPer (Ding et al., ). Additionally, there has been a notable integration of language embeddings to encapsulate data features, aiming to enhance model understanding and data representation (Drori et al., ; Fang et al., ). While the first approach primarily relies on heuristic methods and the latter can be hindered by the slow pace of model training, we adopt language embeddings to represent data for this OOD detection model selection task. In this study, the traditional meta-feature approach is implemented as well for comparison and evaluation.

## 3 MetaOOD for OOD Detection Model Selection

### 3.1 Preliminaries on OOD Detection

The OOD detection task involves training datasets $\mathbf{X}_{\text{train}}$ sampled from the in-distribution (ID) $\mathcal{P}_{\text{in}}$, denoted $\mathbf{X}_{\text{train}}\sim\mathcal{P}_{\text{in}}$, and testing datasets $\mathbf{X}_{\text{test}}$, which may contain both ID and OOD samples. The goal is to train a model $M$ to classify if a new sample $x\in\mathbf{X}_{\text{test}}$ belongs to $\mathcal{P}_{\text{in}}$.

Generally, a model is trained exclusively on the ID data to learn to perform tasks such as classification on this dataset. For example, one common approach is using the model’s confidence scores as the method $M$ to distinguish between ID and OOD data.

When evaluating an OOD detector on $\mathbf{X}_{\text{test}}$, both ID and OOD test data are present to assess the detector’s capability to accurately distinguish between known and unknown data samples. Therefore, in this study, we structure the data into dataset pairs ${D}=\{\mathbf{X}_{\text{train}},\mathbf{X}_{\text{test}}\}$, each consisting of the training dataset with ID samples only and the test dataset with both ID and OOD samples.

### 3.2 Problem Statement and Framework Overview

Given a new, unseen dataset pair for OOD detection,
the goal is to select the best model from a heterogeneous set of OOD detection models without requiring any model evaluations at the test.

In this work, we leverage meta-learning to transfer model performance information from prior experiences to the new OOD task.
Meta-learning (Vanschoren, ), often called “learning to learn,” is a technique where an algorithm learns from a collection of historical/meta tasks and uses this experience to perform well on new, unseen tasks.
The rationale is that an OOD model is likely to outperform on a new dataset if it excels in a similar historical dataset.
This approach is particularly useful when evaluation is infeasible or costly due to a lack of labels or the need for rapid adaptation.

The proposed meta-learner, MetaOOD, relies on:

- •
A collection of $n$ historical (i.e., meta-train) OOD detection dataset pairs, $\mathcal{D}_{\text{train}}=\{D_{1},\ldots,D_{n}\}$ with ground truth labels,
i.e., ${D}=\{\mathbf{X}_{\text{train}},(\mathbf{X}_{\text{test}},\underline{\mathbf{y
}_{\text{test}}})\}$
- •
Historical performance $\mathbf{P}$ of the pre-set model set $\mathcal{M}=\{M_{1},\ldots,M_{m}\}$ (with $m$ models), on the meta-train datasets.
We refer to $\mathbf{P}\in\mathbb{R}^{n\times m}$ as the performance matrix, where
$\mathbf{P}_{i,j}$ corresponds to the $j$-th model $M_{j}$’s performance on the
$i$-th meta-train dataset $D_{i}$.

Our objective is to choose the best candidate OOD detection model $M\in\mathcal{M}$, given a new pair of datasets $D_{\text{new}}=\{\mathbf{X}_{\text{train}}^{\text{new}},\mathbf{X}_{\text{test
}}^{\text{new}}\}$ as input, where we have no ground truth labels $\mathbf{y}_{\text{test}}^{\text{new}}$ for evaluation.

###### Problem 1 (OOD detection model selection)

Given a new input dataset $D_{\text{new}}=\{\mathbf{X}_{\text{train}}^{\text{new}},\mathbf{X}_{\text{test
}}^{\text{new}}\}$ (detect OOD samples on $\mathbf{X}_{\text{test}}^{\text{new}}$ with only in-distribution data $\mathbf{X}_{\text{train}}^{\text{new}}$ and no labels), select a model $M\in\mathcal{M}$ to employ on the new (test) task.

The problem is similar to the OD model selection task such as MetaOD, ELECT, and HPOD (Zhao, ). However, in contrast to OD model selection, which focuses solely on a single dataset, OOD detection model selection requires the consideration of both training and test datasets.
This necessity stems from the need to take into account the similarities and differences among both datasets that impact pre-training on the ID data and the actual OOD detection on the test data, all crucial to measuring the inherent characteristics of OOD detection task.

In a nutshell, our MetaOOD consists of two phases: (i) offline (meta-) training of the meta-learner on $\mathcal{D}_{\text{train}}$ where the goal is to learn the mapping from OOD detection models’ performance on various (historical/meta) datasets, and (ii)
online model selection that uses the meta-train information to choose the model at test time for the new dataset $D_{\text{new}}$.
Figure 1 outlines the workflow and key elements of MetaOOD, with the offline training phase shown in white and the online model selection stage shown in grey.
The details of the two phases are discussed in §3.3 and §3.4, respectively.

###### Problem 1 (OOD detection model selection)

### 3.3 Offline Meta-Training

During the offline training, we generate embeddings for both dataset pair $\mathcal{D}_{\text{train}}$ and method $\mathcal{M}$,
and train the latent mapping from these embeddings
to the performances $\mathbf{P}$.
The meta-learner can generalize and select the best-performing model for new, unseen datasets by learning the relationship between $\{\mathcal{D}_{\text{train}},\mathcal{M}\}\rightarrow\mathbf{P}$.
Prior works have shown that performance mapping is empirically learnable, although imperfect, in related fields like outlier detection (Zhao, ).

To predict the performance of the candidate model on a new dataset pair, we propose training a meta-predictor as a regression problem.
The input to the meta-predictor consists of ${E_{i}^{\text{meta}},E_{j}^{\text{model}}}$, corresponding to the embedding
of the $i$-th dataset pair and the embedding of the $j$-th OOD detector.
Dataset embedding of dataset pair $\mathcal{D}$ is denoted as $E^{\text{data}}=\psi(\mathcal{D})$, and method embedding extracted from $\mathcal{M}$ is denoted as $E^{\text{model}}=\phi(\mathcal{M},M)$;
we provide more details below on the embedding generation in §3.3.1.
Our goal is to train the meta-predictor $f$(^2^22The format of $f$ can be any regression models; in this work, we use an XGBoost Chen & Guestrin () model due to its balance of simplicity and expressiveness, as well as strong feature selection characteristic.) to map the characteristics of the datasets and the OOD detectors to their corresponding performance ranking across all historical dataset pairs. The steps of the (offline) meta-train are shown in Fig. 1, top and Appx. Algo. 1.

$$ ${f:E_{i}^{\text{data}},E_{j}^{\text{model}}}\rightarrow\mathbf{P}_{i,j},\quad i \in\{1,\dots,n\},\quad j\in\{1,\dots,m\}$ (1) $$

#### 3.3.1 Data and Model Embeddings

Data and model embeddings, as inputs of $f$, offer compact, standardized context about the data and models utilized in the meta-learning process.
Instead of directly using data with arbitrary sizes, we hope to generate embeddings that can effectively represent the data and help develop models that can adapt rapidly to new tasks.
In this work, we try two types of data and model embeddings: (1) classical (data) meta-features and (method) one-hot encoding embedding, and (2) language model based approach to get data and model embedding via language models.

Classical Way: Data Embedding via Meta-Features and Model Embeddings via One-hot Encoding.
Meta-features, or “features of features”, are attributes used in meta-learning, model selection, and feature engineering to provide higher-level information about a dataset or its characteristics (Vanschoren, ). These include basic statistics, such as the mean and maximum values of the data, as well as outputs or performance metrics from preliminary models tested on the dataset. Meta-features facilitate understanding the nuances of various learning tasks and guide the selection or configuration of models for new tasks by drawing parallels with previously encountered tasks.
In this work, we first create meta-features that capture OOD characteristics across different dataset pairs. These features aim to identify similarities between data in the meta-training database and test data, enhancing the model selection process based on prior successful applications.
To achieve this, we categorize the extracted meta-features into statistical and landmarker categories.

Statistical meta-features capture the underlying data distributions’ characteristics, including variance, skewness, covariance, and other relevant statistics of individual features and their combinations. For image datasets utilized in OOD, we incorporate meta-features that reflect pertinent image characteristics (Aguiar et al., ), such as:
(i) Color-based: Simple statistical measures from color channels;
(ii) Border-based: Statistical measures obtained after applying border-detector filters;
(iii) Histogram: Statistics from histograms of color and intensity;
and
(iv) Texture features: Values derived from an image’s texture, analyzed using the co-occurrence matrix and Fast Fourier Transform.

Landmarker features summarize the performance of specific learning algorithms on a dataset, providing quick and effective estimates of algorithm performance. These features, derived from evaluating simple/fast models, offer a snapshot of dataset characteristics and approximate the performance of more sophisticated models in specific tasks.
For OOD detection, our landmarkers focus on features related to Softmax probability outputs that can be rapidly generated without model fitting. Detailed descriptions and a complete list of our OOD meta-features are in Appx. Table  A.

New Approach: Data Embedding via Language Models.
Meta-features, often handcrafted and heuristic, have limitations in scalability and adaptability.
This manual approach to feature engineering can be labor-intensive and may not capture all the nuanced relationships within OOD data, potentially affecting the efficacy of model selection.
In contrast, leveraging language models
offers a transformative approach for generating data embeddings (Peng et al., ).
These models have been recently utilized to generate embeddings from textual descriptions of datasets and methods, capturing essential information that reflects the datasets’ and methods’ intrinsic properties due to their comprehensive training (Drori et al., ).
Given that language models are designed to process text inputs, we hypothesize that utilizing text descriptions of data as input to a language model for embedding generation could effectively encapsulate the inherent features of the data. In this study, we experiment with various language models to produce the language embeddings.

To capture the dataset features, we input dataset metadata such as size, object type, and description. For example, the input for the CIFAR-10 dataset (Krizhevsky, ) is formatted as:

Contains images of 10 types of objects, including airplanes, cars, birds, cats, deer, dogs, frogs, horses, ships, and trucks. Each image is a small 32x32 RGB image.

For OOD model embeddings, we take the textual descriptions of methods from the Pytorch-OOD library (Kirchheim et al., ) as input. This includes detailed information about the OOD detector, such as its components, whether it needs fitting, and what is used as OOD indicator. An example of this is the description for the Openmax method (Bendale & Boult, ):

Determines a center for each class in the logits space of a model, and then creates a statistical model of the distances of correctly classified inputs. It uses extreme value theory to detect outliers by fitting a Weibull function to the tail of the distance distribution. The activation of the unknown class is used as the outlier score.

The complete lists of the datasets and methods used are listed in §4.1. In brief, this approach leverages language models’ natural language processing capabilities to transform qualitative descriptions into quantitative embeddings. Compared to the traditional meta-feature approach, it is more computationally efficient and has better generalization ability.
Also, it is interesting to see whether we could use language models to choose an OOD model directly.
We provide all these analyses in §4.

### 3.4 Online Model Selection

In the online model selection process, we generate the embeddings for the test dataset pair $D_{\text{new}}$ and reuse the model embeddings of $\mathcal{M}$, apply the trained meta performance predictor $f$ from the offline stage to predict different OOD detection models’ performance, and select the model with the highest predicted performance, as described in Eq. (2).

$$ $M^{*}:=\underset{M_{j}\in\mathcal{M}}{\arg\max}\,\widehat{\mathbf{P}}_{\text{ new},j},\quad\text{where}\quad\widehat{\mathbf{P}}_{\text{new},j}=f(E^{\text{ meta}}_{\text{new}},E^{\text{model}}_{j})$ (2) $$

Specifically, for a new dataset pair, we acquire the predicted relative performance ranking of different OOD detection methods using the trained $f$, and select the top-1 method(^3^33It may choose the top-$k$ candidates as an ensemble, while this work focuses on top-1 selection.), as shown in Eq. (2).
It is important to note that this procedure is zero-shot, and does not require any network training on the test sample.
The (online) model selection steps are given in Fig. 1, bottom and Appx. Algo. 2.

## 4 Experiments

Our experiments answer the following research questions (RQ):
RQ1 (§4.3): How effective is the proposed MetaOOD in unsupervised OOD model selection in comparison to other leading baselines?
RQ2 (§4.4.1): How do different design choices in MetaOOD impact its effectiveness?
RQ3 (§4.4.2): How much time overhead/saving MetaOOD brings to OOD detection in general?

### 4.1 Experiment Setting

The model Set $\mathcal{M}$.
We compose a model set $\mathcal{M}$ to choose from with 11 popular OOD detection models as shown in Table  1, covering different types of detection methods.

The OOD Datasets.
We utilize the train-test split of datasets preprocessed as described in (Yang et al., ). To summarize, we create our ID-OOD dataset pairs using the following datasets:

- 1.
ID Datasets:
CIFAR10 (Krizhevsky, ),
CIFAR100 (Krizhevsky, ),
ImageNet (Deng et al., ),
FashionMNIST (Xiao et al., )
- 2.
Classic OOD Group: CIFAR10, CIFAR100, MNIST (Deng, ),
Places365 (Zhou et al., ),
SVHN (Netzer et al., ),
Textures (Cimpoi et al., ),
TIN (Le & Yang, )
- 3.
Large-Scale OOD Group:
SSB_hard (Vaze et al., ),
NINCO (Bitterwolf et al., ),
iNaturalist (Horn et al., ),
Textures (Cimpoi et al., ),
OpenImage-O (Wang et al., )

where semantic overlap images between the ID data and the OOD data are removed (1,203 images from TIN, and 1,305 images removed from MNIST, SVHN, Textures, and Places365).

**Table 1: OOD detection models in this study.**
| Category | OOD Detection Model |
| --- | --- |
| Probability Based | MSP (Hendrycks & Gimpel, ) |
|  | MCD (Gal & Ghahramani, ) |
|  | KLM (Hendrycks et al., ) |
|  | Entropy (Chan et al., ) |
| Logit-based | MaxLogit (Hendrycks et al., ) |
|  | Openmax (Bendale & Boult, ) |
|  | EnergyBased (Liu et al., ) |
|  | ODIN (Liang et al., ) |
| Feature-based | Mahalanobis (Lee et al., ) |
|  | ViM (Wang et al., ) |
|  | $k$NN (Cover & Hart, ) |

We construct the ID-OOD dataset pair, and set the training and testing set as follows:
(i) Training: CIFAR10 from ID and OOD from the classic OOD group shown above; and (ii) Testing: CIFAR100, ImageNet, and FashionMNIST from ID, and OOD from large-scale OOD dataset group.
The other ID datasets undergo a similar train test split, which results in
24 unique testing pairs.
Our split guarantees that no dataset in the testing pair has been seen/leaked during the meta-train process.

Hardware.
For consistency, all models are built using the pytorch-ood library (Kirchheim et al., ) on NVIDIA RTX 6000 Ada, 48 GB RAM workstations.

Training the meta-predictor $f$ (see details in §3.3).
In this work, we use an XGBoost (Chen & Guestrin, ) model as $f$ due to its simplicity and expressiveness.
Meanwhile, we use the data and model embeddings generated by the pre-trained BERT-based all-mpnet-base-v2 model by Hugging Face (Reimers & Gurevych, ).
We provide more ablations on $f$ in §4.4.1.

Evaluation.
To compare the performance of MetaOOD and baselines, we examine the performance rank (^4^44In this work, we choose Area Under the Receiver Operating Characteristic Curve (AUC-ROC), or ROC, as the performance metric; can replace by any other metrics at interest.) of the OOD detector chosen by each method among all the candidate detectors through a boxplot and the rank diagram (which is the average across all dataset pairs).
Clearly, the best rank is 1, and the worst is 12 (i.e., 11 baselines and MetaOOD)
To compare our algorithm with a baseline,
we employ the pairwise Wilcoxon rank test on performances across dataset pairs (significance level $p<0.05$).
The full selection results are in Appendix Table B.

### 4.2 Model Selection Baselines

We select the baselines following the literature in meta-learning for unsupervised model selection (Zhao et al., ; ; Jiang et al., ; Park et al., ) with four categories:

(a) No model selection or random selection: always employs either the ensemble of all the models or the same single model, or randomly selects a model:
(1) Maximum Softmax Probability (MSP) (Hendrycks & Gimpel, ), as a popular OOD model,
uses the maximum softmax score of a neural network’s output as threshold to identify whether an input belongs to the distribution the network was trained on.
(2) ODIN (Liang et al., ) applies temperature scaling and small perturbations to the input data, which helps to amplify the difference in softmax scores between ID and OOD samples.
(3) Mega Ensemble (ME) averages OOD scores from all the models for a given dataset. As such, ME does not perform model selection but rather uses all the models.
(4) Random Selection (Random) randomly selects a model from the pool of candidate models.

(b) Simple meta-learners that do not involve optimization:
(5) Global Best (GB) is the simplest meta-learner that selects the model with the largest average performance across all meta-train datasets. GB does not use any meta-features.
(6) ISAC  (Kadioglu et al., ) clusters the meta-train datasets based on meta-features. Given a new dataset pair, it identifies its closest cluster and selects the best model of the cluster.
(7) ARGOSMART (AS)  (Nikolić et al., ) finds the closest meta-train dataset (1 nearest neighbor) to a given test dataset, based on meta-feature similarity, and selects the model with the best performance on the 1NN dataset.

(c) Optimization-based meta-learners which involves a learning process:
(9) ALORS (Misir & Sebag, ) factorizes the perf. matrix to extract latent factors and estimate perf. as the dot product of the latent factors. A regressor maps meta-features onto latent factors.
(9) NCF  (He et al., ) replaces the dot product used in ALORS with a more general neural architecture that predicts performance by combining the linearity of matrix factorization and non-linearity of deep neural networks.
(10) MetaOOD$_$0 uses manually crafted heuristic meta-features, which consist of both statistical and landmarker features for dataset meta-features, along with one-hot encoding for model embeddings, as discussed in §3.3.1.

(d) Large language models (LLMs) as a model selector:
(11) GPT-4o mini (Reimers & Gurevych, ) is used as a zero-shot meta-selector. The dataset and method descriptions are directly provided to the LLM, allowing it to select the methods based on these descriptions. Note there is no meta-learning here.
The details are presented in Appendix §B.1.

Figure: Figure 2: Average rank (lower is better) of methods w.r.t. performance across datasets; MetaOOD outperforms all baselines with the lowest rank.
Refer to caption: x2.png

Figure: Figure 3: Boxplot of the rank distribution of MetaOOD and baselines (the lower, the better). MetaOOD is the lowest/best.

| Ours | Baseline | p-value |
| --- | --- | --- |
| MetaOOD | MetaOOD_0 | 0.0357 |
| MetaOOD | ME | $<$0.0001 |
| MetaOOD | AS | 0.0004 |
| MetaOOD | ISAC | 0.0001 |
| MetaOOD | ALORS | 0.018 |
| MetaOOD | Random | 0.0005 |
| MetaOOD | MSP | $<$0.0001 |
| MetaOOD | ODIN | $<$0.0001 |
| MetaOOD | NCF | 0.0027 |
| MetaOOD | GB | 0.018 |
| MetaOOD | GPT-4o mini | 0.018 |

Table: Table 2: Wilcoxon signed-rank test result. MetaOOD is statistically better than all the baselines.

### 4.3 Overall Results

First, we present the rank distribution of the actual rank of the top-1 OOD detector chosen by each model selection method across the 24 test data pairs in Fig. 3.
To compare two model selection algorithms (e.g., ours with a baseline), we perform Wilcoxon rank test on the AUROC value of the top-1 models selected by our method and the baseline method, as shown in Table 2.
Also, we show the aggregated average rank plot in Fig. 2, where the average performance rank of the OOD detection model selected by each algorithm is shown. Here are the key findings of the results:

1. MetaOOD outperforms all baselines,
as shown in Fig. 3, Table 2, and Fig. 2.
As shown in Fig. 3, MetaOOD consistently performs well with small variance.
On the aggregated level, the average rank of MetaOOD surpasses all of the 11 baselines in Fig. 2.
Moreover, MetaOOD has demonstrated statistically superior performance compared to all these baselines.
Such enhancement in performance is indicative of MetaOOD’s robust approach to tackling complex datasets while generating stable performance. We credit its superiority to the combination of a meta-learning framework and the use of language models in embedding datasets and models.

2. Optimization-based meta-learning methods (i.e., MetaOOD, MetaOOD_0 and ALORS) perform better in unsupervised model selection over other baselines.
One reason is that meta-learning leverages prior experiences to adapt to new tasks. It optimizes the learning process by extracting common patterns and representations across different tasks, enhancing the model’s generalization ability.
Meanwhile, the optimization process helps models converge to an optimal solution efficiently.
Compared to simple meta-learners, such optimization-based methods take advantage of the meta-features for a better mapping of the model performance.
For instance, AS and ISAC do not involve optimization and resemble finding the closest sample or closest sample cluster for choosing the top-1 detector.
They are inferior to MetaOOD as there is no optimization on the OOD model performance, but fully depends on the data embeddings for performance knowledge transfer.

3. The underperformance of no model selection and random selection baselines justifies the need for OOD model selection.
We analyze their performance below.

- •
ME: Averaging the OOD scores of all models does not yield strong results, as demonstrated in Fig. 3 and 2.
This may stem from certain models consistently underperform across various datasets, and combining all models indiscriminately reduces overall effectiveness. While using selective ensembles might offer improvements (Zhao & Hryniewicki, ), constructing ensembles from numerous models can be impractical due to high costs.
In contrast,
MetaOOD learns to make optimal selections without constructing any models, allowing it to operate efficiently during testing.
- •
Random: According to Fig. 3 and Table 2, random selection performs worse than all the meta-learners.
This indicates that all the meta-learner baselines we chose do have some improvements compared to random choice, and it is not advised to select an OOD detection model randomly.
- •
ODIN and MSP:
As expected, a single method does not perform well across all datasets. This result is not surprising since different OOD detectors emphasize various aspects of the datasets, and real-world datasets have diverse characteristics. Relying on a single approach tends to limit the scope of solutions, making it difficult to capture the distribution shift among different datasets.

4. LLM acts as a reasonable zero-shot model selector in OOD model selection.
This highlights the potential of LLM in model selection, while it can be improved by meta-learning.
While the performance of LLM as a selector may be slightly worse (ranked as top 3) than our MetaOOD, LLM consistently selects the globally best model (Openmax (Bendale & Boult, )) across diverse tasks, with no prior knowledge about the performances of different OOD detection methods on different models.
This is particularly valuable when prior knowledge is limited or when facing novel tasks, highlighting the potential of LLMs in unsupervised model selection.
It can be further improved via meta-learning by receiving more performance-related information.
Arguably, MetaOOD leverages language embeddings along with the meta-learning, thus achieving better performance.

### 4.4 Ablation Studies and Additional Analyses

#### 4.4.1 Ablation Study

In §3.3.1, we discussed how to train the meta-performance predictor $f$, with the choices on
data and model embeddings and $f$ itself. Here, we conduct two ablations on these choices:

Ablation 1 on Data and Model Embeddings.
We compare the MetaOOD’s performance on different LLM-generated embeddings
and different combinations of embeddings.
The variants include:

- •
OpenAI text-embedding-3-small (OpenAI_emb) (Reimers & Gurevych, ): It is an embedding model designed by OpenAI to generate compact and meaningful representations of text for various natural language processing tasks.
- •
LLama2-7b (Llama) (Touvron et al., ):
LLama2 is an advanced language model that provides enhanced capabilities in a variety of applications, including text summarization, translation, and conversational AI. Since it is a transformer style decoder-based model, the sequence level embeddings are produced by pooling token level embeddings together(^5^55Usually this is done by averaging the token level embeddings or using the first token. In this study, we opt for the latter approach.).
- •
MetaOOD$_$0 uses meta-features for dataset embeddings, along with the simple one-hot encoding for model embeddings (see §3.3 for more details).

Figure: Figure 4: Ablation study on different data and model embeddings. MetaOOD has better performance over its variants.

Figure: Figure 5: Ablation study on different choices of meta-predictor $f$. Tree-based models have better performance.

Fig. 4 presents ablation studies results on MetaOOD that uses different embedding variants with $f$ fixed to XGBoost.
The results indicate that language models are highly effective at generating embeddings. MetaOOD, along with the other two language model-generated embeddings, demonstrate solid performance.
We observed that the BERT-based embeddings perform slightly better than those derived from LLMs.
Fig. 6 shows the dataset embeddings produced by different language models, with the embeddings reduced to 2D using t-SNE. Note that MetaOOD (BERT-based embedding) is slightly better to capture similar datasets (e.g. Texture and Textures) than decoder-based model embeddings. This could be attributed to the decoder model’s causal attention, where the representation of a token is influenced only by preceding tokens, making it less effective for text embedding tasks as it restricts capturing information from the entire input sequence
(BehnamGhader et al., ).

Figure: Figure 6: Visualization of dataset embeddings generated by different language models. Bert-based model performs slightly better (e.g., Texture and Textures datasets are close).
Refer to caption: x6.png

Ablation 2 on the Choice of Meta-predictor $f$.
We evaluate the performances of MetaOOD_NN and OpenAI_emb_NN, which use the language embeddings to train a two-layer MLP meta-predictor.
The result is shown in Fig. 5.
Consistent with the finding in (Jiang et al., ), using a tree-based model such as XGBoost for meta-predictor leads to more stable results and enhanced performance compared to initializing with neural network models.
Additionally, tree-based models offer superior feature selection capabilities and greater interpretability. In contrast, neural networks, while powerful, can be more sensitive to initialization and hyperparameter choices, leading to less predictable performance. This finding underscores the significance of selecting features of importance during meta-learning, especially in complex tasks where stability and performance are critical.

#### 4.4.2 Runtime Analysis of MetaOOD

Figure: Figure 7: Runtime of MetaOOD vs. selected OOD detector. MetaOOD incurs a small overhead (difference shown with black arrows).
Refer to caption: x7.png

One of the big advantages of MetaOOD over MetaOOD_0 is the fast dataset embedding generation via language models, making the model selection overhead negligible to the actual OOD detection fitting on the image datasets.
We demonstrate the time of MetaOOD and the fitting of the selected OOD detector on selected dataset pairs in Fig. 7.
Notably, the language embedding for both datasets and models and online model selection via $f$ just takes seconds to finish.
Thus, the MetaOOD only brings the marginal cost to the entire OOD detection pipeline while finding a top-performing model.

## 5 Conclusion, Limitations, and Future Directions

In this work, we introduced MetaOOD, the first unsupervised out-of-distribution (OOD) detection model selection framework. This meta-learner utilizes an extensive pool of historical data on OOD detection models and dataset pairs, employing language model-based embeddings to enhance the model selection process based on past performances.
Despite its innovative approach, MetaOOD depends on the availability and quality of historical dataset pairs. This reliance may limit its effectiveness in scenarios where such data are sparse or less similar. Moreover, the current framework is designed for single-modality data, which restricts its application in highly diverse or multimodal environments.
Looking ahead, we plan to broaden our testbed to include a more diverse group of datasets and models, thereby enhancing the meta-learning capabilities of MetaOOD. We also aim to extend MetaOOD to support top-$k$ selection, offering a range of viable models rather than a single recommendation.
Additionally, equipping MetaOOD with an uncertainty quantification mechanism will enable it to output an “I do not know” response when applicable, further refining its utility in complex scenarios where no suitable meta-train knowledge can be transferred.

## Broader Impact Statement, Ethics Statement, and Reproducibility

Broader Impact Statement: MetaOOD revolutionizes OOD detection model selection by enabling practitioners to choose appropriate models for unlabeled tasks automatically.
This is particularly crucial in sectors like healthcare, finance, and security, where rapid adaptation to new data types can significantly enhance system reliability and prevent critical errors. By providing a systematic approach to select the most effective models, MetaOOD promotes robust applications in dynamically changing environments, ensuring ongoing reliability and accuracy in critical systems.

Ethics Statement:
Our research adheres to the general Code of Ethics, ensuring that MetaOOD is developed and applied with ethical considerations at the forefront, particularly in privacy, bias, and fairness across diverse applications. By facilitating more accurate and unbiased model selection, MetaOOD helps mitigate potential ethical risks in its deployments, such as in surveillance and healthcare, promoting fairness and protecting privacy. Continuous ethical evaluations accompany MetaOOD’s development to ensure it meets societal and legal standards.

Reproducibility Statement: We advocate reproducibility in MetaOOD. Comprehensive documentation of our methodologies and experimental designs is detailed in the main text and appendices. We have made our code, testbed, and meta-learner fully accessible at [https://anonymous.4open.science/r/metaood-DA67](https://anonymous.4open.science/r/metaood-DA67), providing the resources for replication and exploration.

## Supplementary Material for MetaOOD

## Appendix A Details on MetaOOD

### A.1 Pseudo-code for Meta-train and Online Model Selection

We discussed meta-training and online model selection in §3.3 and §3.4, respectively.
Here is the pseudo-code for the two phases.

Figure: Algorithm 1 Offline OOD detection meta-learner training

Figure: Algorithm 2 Online OOD detection model selection

### A.2 Details on Meta-features

The classical way of data embedding via meta-features is discussed in §3.3.1.
Table A presents the complete list of meta-features constructed in this study.

**Table A: Selected MetaOOD features. Part of the statistical features are based on Vanschoren (); Zhao et al. (), and specialized landmarker features are newly designed for OOD detection. See §3.3 for more details.**
| Category | Description | Variants |
| --- | --- | --- |
| Statistical Features |  |  |
| Sample features | $\mu,\tilde{X},\sigma^{2},\min_{X},\max_{X},\sigma$ | mean, median, var, min, max, std |
|  | Percentile $P_{i}$ | q1, q25, q75, q99 |
|  | $q75-q25$ | IQR |
|  | $\frac{\mu}{\max_{X}},\frac{\tilde{X}}{\max_{X}}$ | normalized mean, normalized median |
|  | $\max_{X}-\min_{X}$, Gini$(X)$ | sample range, sample gini |
|  | edian$m(X-\tilde{X})$ | median absolute deviation |
|  | avg$(X-\tilde{X})$ | average absolute deviation |
|  | $\frac{q75-q25}{q75+q25}$ | Quantile Coefficient Dispersion |
|  |  | Coefficient of variance |
|  | If a sample differs from a normal dist. | normality |
|  |  | 5th to 10th moments |
|  | skewness, $\frac{\mu^{4}}{\sigma^{4}}$ | skewness, kurtosis |
|  | $\mathcal{M}$ | Co-occurence Matrix |
| Image features | color-based | mean, std of the HSV channel |
|  | color-based | std of the intensity channel |
|  | color-based | entropy of the RGB channel |
|  | histogram | std of the (RGB, HSV, intensity) channel |
|  | border | Average white pixels |
|  | border | Average Hu Moments of sobel image |
|  | texture |  |
|  | (Co-occurence Matrix) | contrast mean, std |
|  | (Co-occurence Matrix) | dissimilarity mean, std |
|  | (Co-occurence Matrix) | homogeneity mean, std |
|  | (Co-occurence Matrix) | energy mean, std |
|  | (Co-occurence Matrix) | correlation mean, std |
|  | (Co-occurence Matrix) | entropy mean, std |
|  | (FFT) | entropy mean, std |
|  | (FFT) | inertia mean, std |
|  | (FFT) | energy mean, std |
|  | (FFT) | homogeneity mean, std |
| Dataset features | $n,p$ | num of samples, num of features |
|  | $d,c$ | dim, num of class |
|  | EMD | Earth Mover’s Distance |
| Landmarker Features |  |  |
| Probability-based | softmax probability | mean, std, min, max |
|  |  | entropy, range |
|  |  | top1 softmax probability |
|  |  | top2 softmax probability |
|  |  | considence margin |
|  |  | skewness, kurtosis |

## Appendix B Additional Experiment Settings and Results

### B.1 Prompts to LLM for zero-shot selection of the optimal OOD detector

Section §4.2 discusses the baseline GPT-4o mini, where LLM is used as a zero-shot meta-predictor.
In this baseline, the prompt provided to the LLM is structured as follows, with text descriptions of both the datasets and models provided.
To ensure consistency, we set temperature parameter to 0, and top_p parameter to 0.999.

[Dataset descriptions provided]

Your task is to select the best OOD detection method for a set of 24 test ID-OOD dataset pairs.
You will be provided with descriptions of both the ID-OOD dataset pairs and the available OOD detection methods.
You should pick the best model that has the highest AUROC metric.
For each dataset pair, output the recommended OOD detection method in the format: ’Recommended Method: [Recommended Method]’.

[Model descriptions provided]

### B.2 Full Performance Results

Below is the full performance matrix $P$, which shows the performance of eleven OOD detection models (refer to Table 1) on our constructed ID-OOD dataset pairs (see dataset pair construction in §4.1). We report AUROC as the metric for OOD detection.

**Table B: Various OOD detection models’ performance on ID-OOD dataset pairs. See experiment setting in §4.1.**
| ID Dataset | OOD Detector<br>OOD Dataset | CIFAR-10 | CIFAR-100 | MNIST | Places365 | SVHN | Texture | TIN | SSB_hard | NINCO | iNaturalist | Textures | OpenImage-O |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| CIFAR-10 | Openmax | N/A | 90.68 | 93.95 | 91.17 | 93.31 | 92.71 | 89.67 | 82.53 | 91.53 | 90.07 | 92.85 | 91.36 |
|  | MCD | N/A | 88.47 | 92.88 | 88.31 | 90.31 | 87.22 | 86.94 | 80.15 | 88.59 | 85.39 | 87.37 | 87.88 |
|  | ODIN | N/A | 65.91 | 88.58 | 67.71 | 94.25 | 72.44 | 63.28 | 58.85 | 65.52 | 50.47 | 72.59 | 62.08 |
|  | Mahalanobis | N/A | 82.77 | 94.38 | 83.92 | 95.65 | 96.78 | 82.62 | 76.38 | 85.47 | 86.46 | 96.83 | 90.03 |
|  | EnergyBased | N/A | 87.25 | 97.45 | 90.12 | 91.56 | 85.42 | 86.60 | 78.43 | 87.57 | 81.52 | 85.30 | 86.94 |
|  | Entropy | N/A | 88.29 | 94.05 | 88.91 | 92.55 | 88.91 | 86.97 | 79.08 | 88.42 | 84.97 | 88.95 | 87.88 |
|  | MaxLogit | N/A | 87.29 | 97.27 | 90.04 | 91.58 | 85.55 | 86.59 | 78.44 | 87.58 | 81.66 | 85.44 | 86.97 |
|  | KLM | N/A | 83.78 | 90.95 | 84.96 | 88.88 | 85.64 | 83.36 | 74.72 | 84.47 | 76.89 | 85.79 | 83.43 |
|  | ViM | N/A | 83.03 | 89.84 | 83.54 | 91.62 | 92.14 | 82.16 | 78.18 | 86.19 | 86.09 | 92.16 | 88.00 |
|  | MSP | N/A | 87.95 | 93.27 | 88.45 | 92.09 | 88.58 | 86.59 | 78.85 | 88.05 | 84.78 | 88.62 | 87.52 |
|  | KNN | N/A | 88.34 | 92.47 | 89.94 | 92.62 | 91.20 | 87.53 | 80.28 | 88.61 | 84.01 | 91.29 | 89.45 |
| CIFAR-100 | Openmax | 77.40 | N/A | 81.30 | 83.77 | 93.71 | 87.38 | 81.11 | 75.60 | 78.40 | 77.22 | 87.57 | 79.23 |
|  | MCD | 76.51 | N/A | 81.78 | 76.40 | 68.19 | 71.16 | 79.50 | 75.68 | 76.89 | 76.59 | 71.03 | 73.83 |
|  | ODIN | 59.55 | N/A | 82.72 | 63.01 | 81.51 | 64.27 | 62.73 | 60.57 | 64.48 | 54.65 | 64.31 | 62.14 |
|  | Mahalanobis | 66.51 | N/A | 73.35 | 70.13 | 85.21 | 89.84 | 74.04 | 71.46 | 72.87 | 75.63 | 90.20 | 74.73 |
|  | EnergyBased | 77.15 | N/A | 93.92 | 78.20 | 72.49 | 76.20 | 80.63 | 75.64 | 78.65 | 74.16 | 76.05 | 74.10 |
|  | Entropy | 76.45 | N/A | 86.37 | 77.30 | 71.70 | 74.93 | 80.17 | 75.90 | 77.98 | 77.90 | 74.79 | 75.37 |
|  | MaxLogit | 77.32 | N/A | 93.15 | 78.33 | 72.55 | 76.27 | 80.78 | 75.83 | 78.78 | 74.72 | 76.13 | 74.37 |
|  | KLM | 74.45 | N/A | 86.02 | 75.79 | 73.86 | 75.41 | 78.63 | 74.85 | 76.73 | 76.89 | 75.26 | 74.38 |
|  | ViM | 69.50 | N/A | 86.06 | 71.14 | 81.81 | 88.89 | 77.14 | 73.32 | 77.39 | 75.32 | 89.12 | 72.80 |
|  | MSP | 75.10 | N/A | 83.04 | 75.85 | 70.02 | 73.41 | 78.41 | 74.48 | 76.37 | 77.13 | 73.26 | 74.21 |
|  | KNN | 71.53 | N/A | 67.42 | 70.96 | 83.29 | 79.73 | 78.38 | 74.96 | 75.20 | 73.38 | 79.99 | 71.44 |
| Imagenet | Openmax | 98.12 | 96.77 | 95.89 | 82.52 | 97.63 | 87.95 | 90.73 | 73.66 | 81.04 | 93.65 | 89.65 | 88.34 |
|  | MCD | 82.94 | 85.63 | 89.79 | 79.40 | 97.70 | 80.46 | 79.16 | 72.16 | 79.97 | 88.42 | 82.47 | 84.98 |
|  | ODIN | 98.74 | 97.85 | 96.56 | 64.67 | 99.80 | 70.86 | 83.64 | 67.57 | 69.10 | 74.27 | 72.09 | 71.39 |
|  | Mahalanobis | 82.33 | 80.45 | 88.25 | 58.25 | 72.91 | 89.95 | 80.72 | 47.91 | 62.07 | 61.18 | 89.78 | 70.54 |
|  | EnergyBased | 85.13 | 86.78 | 96.79 | 82.45 | 98.35 | 86.74 | 80.97 | 72.35 | 79.70 | 90.59 | 88.73 | 89.14 |
|  | Entropy | 85.72 | 88.14 | 94.38 | 81.43 | 98.64 | 83.23 | 81.46 | 73.07 | 81.80 | 91.03 | 85.34 | 87.78 |
|  | MaxLogit | 85.68 | 87.32 | 96.07 | 82.64 | 98.55 | 86.40 | 81.38 | 72.75 | 80.40 | 91.13 | 88.41 | 89.25 |
|  | KLM | 87.73 | 88.95 | 91.49 | 78.84 | 97.37 | 82.39 | 82.33 | 68.60 | 79.82 | 89.54 | 84.07 | 85.98 |
|  | ViM | 94.34 | 94.26 | 95.89 | 77.52 | 95.10 | 91.82 | 87.21 | 63.83 | 77.53 | 87.23 | 92.51 | 87.05 |
|  | MSP | 82.94 | 85.63 | 89.79 | 79.40 | 97.70 | 80.46 | 79.16 | 72.16 | 79.97 | 88.42 | 82.47 | 84.98 |
|  | KNN | 71.37 | 71.02 | 69.83 | 61.75 | 64.20 | 60.94 | 70.13 | 55.35 | 63.81 | 56.47 | 61.49 | 60.35 |
| FashionMNIST | Openmax | 92.18 | 90.61 | 93.65 | 91.70 | 91.75 | 92.22 | 90.34 | 91.71 | 91.57 | 94.08 | 92.27 | 90.81 |
|  | MCD | 86.03 | 86.88 | 85.65 | 89.17 | 93.85 | 86.77 | 88.43 | 87.74 | 87.83 | 89.06 | 86.59 | 88.67 |
|  | ODIN | 57.52 | 59.33 | 55.66 | 61.53 | 89.01 | 60.94 | 59.83 | 60.88 | 61.89 | 47.23 | 61.05 | 57.55 |
|  | Mahalanobis | 99.46 | 99.44 | 97.22 | 99.35 | 99.90 | 99.75 | 99.33 | 99.47 | 99.40 | 99.63 | 99.77 | 99.44 |
|  | EnergyBased | 91.32 | 92.35 | 79.32 | 94.03 | 98.30 | 91.44 | 93.26 | 93.09 | 92.90 | 93.62 | 91.31 | 93.40 |
|  | Entropy | 85.69 | 86.69 | 81.76 | 89.00 | 94.27 | 86.97 | 88.21 | 87.58 | 87.65 | 88.12 | 86.88 | 88.32 |
|  | MaxLogit | 91.22 | 92.26 | 79.32 | 93.94 | 98.24 | 91.35 | 93.16 | 92.99 | 92.81 | 93.51 | 91.22 | 93.31 |
|  | KLM | 71.28 | 73.85 | 64.34 | 79.02 | 91.84 | 77.16 | 77.49 | 75.26 | 75.87 | 76.58 | 77.12 | 77.78 |
|  | ViM | 96.06 | 96.23 | 80.93 | 96.84 | 99.22 | 97.69 | 96.65 | 96.58 | 96.86 | 97.44 | 97.71 | 97.03 |
|  | MSP | 84.72 | 85.78 | 80.52 | 88.36 | 93.90 | 86.25 | 87.46 | 86.87 | 86.86 | 87.54 | 86.15 | 87.64 |
|  | KNN | 97.71 | 97.70 | 98.38 | 97.85 | 99.45 | 98.12 | 97.72 | 97.72 | 97.74 | 97.76 | 98.14 | 97.95 |

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. Does the proposed method rely on the architecture of the trained model?
2. What is the training time of the proposed method?
3. If there is one additional OOD method, how can incorporate this method into the proposed MetaOOD?
4. What are the main factors that influence the choice of an OOD method based on the characteristics of the training and test sets?
soundness: 3
strengths: 1. The idea of using meta-learning to select the best OOD detection method for each specific task is interesting.
2. The paper is generally easy to understand and clearly written.
3. The experiments show the effectiveness of the proposed method.
summary: In this paper, the authors propose MetaOOD, which utilizes meta-learning to select an OOD detection model automatically. The motivation is that each OOD detection algorithm might excel in specific scenarios but may not perform well universally, therefore it is important to select one particular OOD detection for each task. MetaOOD utilizes historical performance data of existing methods across a variety of benchmark out-of-distribution (OOD) datasets to enable efficient model selection for new datasets, eliminating the need for labeled data at test time. To more accurately measure task similarities, the authors incorporate language model-based embeddings that capture the unique OOD characteristics of both datasets and detection models. Through extensive testing across 24 unique test dataset pairs and 11 OOD detection models, the authors show that MetaOOD consistently outperforms current methods with minimal additional computation time.
weaknesses: 1. Figure 1 needs to be improved. The notations in the figure are confusing and unclear.
2. The design of the textual description seems ad-hoc and cannot be applied in the case of without detailed dataset information.
3. Detailed results on the selected OOD method for each dataset are missing.

---

confidence: 4
contribution: 1
initial_rating: 3
presentation: 1
questions: NA
soundness: 1
strengths: 1. The motivation is sound. It is interesting to see a meta-selection approach to the OOD detection problem since there are so many methods in this OOD domain.
2. The proposed method is simple and straightforward. The results on the traditional methods are promising.
summary: This paper presents MetaOOD, a “model” selection approach for out-of-distribution (OOD) detection. MetaOOD utilizes language models to generate feature embeddings of both the meta dataset and “models”, allowing for the optimal “model” selection based on anticipated performance on the test set. The results on the Wilcoxon statistical tests show the promising performance of MetaOOD.
weaknesses: 1. The definition of "OOD model" is confusing. There are many post-hoc detection methods in the detection problem, which should not be classified as “models”. For instance, the paper includes the MSP method for the selection experiments. However, MSP is just a simple post-hoc technique that can be applied to most classification models (e.g., ResNet) using the SoftMax function. This method should not be considered as a model, which is misleading considering another factor, “model architecture,” in the experiments.

2. The methodology lacks depth. MetaOOD merely utilizes language models to extract embeddings for dataset and model descriptions, and then select the top-1 method based on these embeddings. The approach lacks insight and overlooks potential issues. For instance, the embeddings derived from descriptions may not accurately capture the true characteristics of the models and datasets. Also, simply selecting the top 1 can overlook the nuances of methods and the potential problems of the utilized datasets.

3. The experimental results are unconvincing. The baseline methods included are outdated, with the most recent method (NCF) dating back to 2017.

4. The terms OOD and OOD detection should not be used interchangeably. It is unclear what is meant by "OOD dataset" given such a name strategy. Is it referring to a commonly recognized OOD dataset distinct from the in-distribution (InD) dataset, or simply an OOD detection dataset (includes train, val, and test splits for detection methods)? 

5. I am curious whether this paper was generated by a language model, such as GPT-4. The writing style, particularly in Section 3.3.1, resembles AI-generated text. Given the simplicity of the method, the Method Section could be more concise, potentially requiring only 0.5 pages to convey the core elements of the approach. However, the current version spans 2.5 pages.

**Post-rebuttal Comments**

I'd like to thank the authors' response. I agree that weakness 5 could be too strong as I assumed the method may be generated from GPTs. However, the length and the redundancy of the method section are still a problem. I still cannot accept that the choice of embedding strategy could occupy half of the Method Section, given that the embedding method is that simple.

I also agree that the simplicity could be the advantage of the proposed method. However, I still cannot view this method as sufficiently advanced or novel for a research paper at ICLR. As I've mentioned, this method lacks depth, not to mention the absent theoretical insights. The approach merely utilizes the rough descriptions of the dataset and models to predict the score on the OOD dataset. The performance relies heavily on LLMs, which can introduce a series of problems and are heavily limited to the application scenarios. 

Also, I acknowledge that the paper may present the "first" selection strategy for the OOD detection method. However, this should not be an excuse for the obvious shortcomings. Overall, given current experiments, I consider this paper more of an exploration of leveraging LLMs in OOD detection rather than a substantial research contribution. Thus, I will maintain my original score.

**Additional Post-rebuttal Comments**

After reviewing the authors' responses, I still consider this paper as a preliminary exploration of leveraging LLMs in OOD detection. So I will maintain my original score.

---

confidence: 5
contribution: 3
initial_rating: 6
presentation: 2
questions: Can the authors add a diagram to better illustrate how this technique would work in practice? This will help translatability of this work into other contexts faster.
soundness: 2
strengths: Integration of language model based embeddings; Empirical eval. is done to good detail. this technique is actually useful. though it is a logical next step ~ the approach itself can be used in other contexts or at-least the idea can be adapted. Sufficient detail is provided makes work transparent.
summary: The paper postulates that by identifying which OOD detection models have historically performed well on datasets similar to the one currently being considered, one can select the model most likely to be effective without needing labels for supervised training.  A meta-learning approach is used to take past performance data from various models (across data sets); when new dataset arrives, the approach checks for similarity between the new dataset and historical ones using embeddings. The assumption is that the selected model will perform well as it is closest to the data set under use.  Meta-learning (training) is done offline with curated data sets; while OOD model selection is online as a specific data point arrives.  Results show that their approach works better than compared other techniques.  Experiment approach itself is reasonable and approach is sound.
weaknesses: Eval is too narrow and limited ~ so results may not generalise or this approach may be limited to the data set / domain attempted; esp. since there is no formal conceptual development as such we do not know when and where this method will work or have an intuition for where the limit may be.  Although there is the claim of unsupervised world-first etc. ~ there is still a need for other forms of supervised and curated training. Assumes text descriptions are good in the evals and curated data sets - but does it hold for the real world?  All the usual limits of language models apply here too. Scalability not known -- again since we do not have specific underlying theory.

---

confidence: 5
contribution: 3
initial_rating: 6
presentation: 2
questions: No questions.
soundness: 3
strengths: The paper's strengths include the introduction of a zero-shot, unsupervised framework for OOD detection model selection, which enhances adaptability to new datasets. It effectively utilizes language model-generated embeddings to capture nuanced dataset characteristics, improving model selection accuracy. The extensive experimentation demonstrates superior performance compared to eleven established methods, showcasing its robustness. Additionally, the framework incurs minimal runtime overhead, making it efficient for practical applications. The use of the Wilcoxon signed-rank test is a plus of the paper. The p-values suggests the proposed approach works well.
summary: The paper presents MetaOOD, a framework for automatic selection of out-of-distribution (OOD) detection models without requiring labeled data. It leverages historical performance data and language model embeddings. The approach aims to improve the reliability of OOD detection in critical applications, such as autonomous driving and online transactions. Overall, MetaOOD addresses the challenge of adapting to data shifts effectively and efficiently.
weaknesses: The paper's weaknesses include a reliance on the quality of language model embeddings, which may vary based on the model used and the nature of the input data. Additionally, the framework's performance may be limited by the diversity of the historical data pool, potentially affecting generalization to unseen datasets. The lack of extensive real-world testing could raise concerns about its applicability in practical scenarios. Lastly, the complexity of the approach may pose challenges for reproducibility and implementation in different contexts. Obtain datasets and model feature/embeddings from their textual descriptions appear a bit strange and somewhat unreliable.

Meta-review: This paper presents a meta-learning framework for model selection of out-of-distribution detection models. The core idea is to use embeddings from language models to represent datasets and models as described by textual descriptions. The framework is evaluated on image-based OOD detection tasks and shown to outperform other baselines.

The paper addresses an important practical problem of model selection for OOD detection. The proposed method is simple and shown to be effective, using rigorous statistical tests. The work could be strengthened by additional analysis on the embeddings and how they are capturing task and dataset similarity, as performance appears to be somewhat sensitive to choice of language model used (Fig 4). Experiments on a broader range of datasets will also help to strengthen evidence for the generalizability of the approach. 

Overall, the AC leans towards accepting this paper as an interesting first approach to an important practical problem. The authors should incorporate all changes and additional results as promised in the discussion.

Decision: Accept (Poster)