Enhancing Near OOD Detection in Prompt Learning: Maximum Gains, Minimal Costs

## Abstract

Abstract Prompt learning has shown to be an efficient and effective fine-tuning method for vision-language models like CLIP. While numerous studies have focused on the generalisation of these models in few-shot classification, their capability in near out-of-distribution (OOD) detection has been overlooked. A few recent works have highlighted the promising performance of prompt learning in far OOD detection. However, the more challenging task of few-shot near OOD detection has not yet been addressed. In this study, we investigate the near OOD detection capabilities of prompt learning models and observe that commonly used OOD scores have limited performance in near OOD detection. To enhance the performance, we propose a fast and simple post-hoc method that complements existing logit-based scores, improving near OOD detection AUROC by up to 11.67% with minimal computational cost. Our method can be easily applied to any prompt learning model without change in architecture or re-training the models. Comprehensive empirical evaluations across 13 datasets and 8 models demonstrate the effectiveness and adaptability of our method.

## 1 Introduction

Pre-trained vision-language models such as ALIGN [] and CLIP [] have shown outstanding visual-text understanding by learning to align image features and textual features of a large-scale image-text dataset via contrastive learning. Consequently, CLIP naturally excels at zero-shot image classification, utilising a class name as natural language text instead of arbitrarily numbered category. For instance, cosine similarity between encoded image feature of an image and encoded text feature of “a photo of a [CLASS].” with a class name as “[CLASS]” can perform zero-shot classification without requiring any additional classifier. Logits with different class names can then be transformed to probabilities by using the softmax function.

Despite the powerful classification capability of CLIP, a study [] has shown that the CLIP’s classification accuracy is sensitive to wording of the prompt. For instance, using “a photo of [CLASS].” instead of “a photo of a [CLASS].” results in more than 5% decrease in accuracy in Caltech101 [] where the only difference in the prompt is a single character “a”.
Inspired by these observations,
CoOp [] was proposed to optimise the prompt in word embedding space instead of handcrafting the prompt, which has been shown to be an effective method with superior performance with little training cost.

**Table 1: Improvement of near OOD AUROC using MaxLogit score with our method on 1-shot Caltech101 [] with wall-clock time measured for the method. Refer to Section 5 for details.**
| Model | $\triangle$ AUROC | Computation Time (s) |
| --- | --- | --- |
| CoOp [] | +2.66 | 0.988 |
| CoCoOp [] | +5.05 | 1.703 |
| IVLP [] | +3.79 | 1.055 |
| KgCoOp [] | +7.09 | 0.989 |
| ProGrad [] | +5.90 | 0.994 |
| MaPLe [] | +5.74 | 0.974 |
| PromptSRC [] | +6.09 | 1.005 |
| LoCoOp [] | +10.57 | 1.153 |

As CoOp learns a single global prompt with few-shot training samples,
one notable drawback of CoOp is that the learned prompt is not generalisable to unseen classes.
Subsequently, significant research efforts have been made to improve accuracy on samples from unseen classes by generalising the learned prompt [, , , , ]. In safety-critical applications, beyond achieving high accuracy, it is crucial for a model to distinguish between samples from unseen classes and those from seen classes. This capability is an equally important research question that has not been adequately explored.

Recently, a few studies have shown promising improvement in out-of-distribution (OOD) detection of CoOp [, ]. Nevertheless, a few gaps still exist. Firstly, these methods require to train a separate model, which is computationally costly and not applicable to various existing prompt learning models. Secondly, they were only compared with the early works of prompt learning models such as CoOp and CoCoOp [] without being compared with more recent models such as MaPLe [] and PromptSRC []. More importantly, no study has yet addressed near OOD detection of prompt learning models, which is a more challenging task in which in-distribution (ID) and OOD samples share the same image domain without any overlapping label classes [, , , , ].

To address this gap, we propose a post-hoc framework applicable to any prompt learning models improving near OOD detection performance. The framework involves using new OOD score functions that complement existing logit-based scores. Our method is simple and effective that does not require any OOD samples, architecture change, or re-training models (see Talbe 1). Our contributions are:

- 1.
We analyse limitation of the existing OOD scores when applied to prompt learning models and introduce a new concept of Empty-Class score that complements existing logit-based scores.
- 2.
We propose a simple post-hoc framework that can be applied to the existing score functions to improve near OOD detection performance without affecting its classification accuracy.
- 3.
We extensively validate our method with 13 datasets and 8 prompt learning models, improving near OOD AUROC up to 11.67%.

## 2 Background

### 2.1 Contrastive Language–Image Pre-training (CLIP)

Contrastive Language–Image Pre-training (CLIP) [], a pre-trained vision-language model learned to align 400 million image-text pairs, is renowned for its powerful zero-shot image classification performance. It measures cosine similarity between image feature of an unseen image and text feature of a text prompt formatted as ‘‘a photo of a [CLASS]’’ where [CLASS] is the name of a class in a label space of interest. Formally, given an image $I\in\mathbb{R}^{H\times W\times 3}$ with $H$ being the height and $W$ being the width and a text prompt $T=\texttt{``a photo of a [CLASS]''}$, a classification logit is computed by $\langle\text{Enc}_{\text{I}}(I),\text{Enc}_{\text{T}}(T)\rangle$ where $\langle\cdot,\cdot\rangle$ is cosine similarity, $\text{Enc}_{\text{I}}(\cdot)$ is an image encoder, and $\text{Enc}_{\text{T}}(\cdot)$ is a text encoder. The image encoder can be either ResNet [] or Vision Transformer (ViT) [], and the text encoder is Transformer []. For the brevity of notation, we omit the notations of the encoders for the remainder of the paper.

### 2.2 Context Optimization (CoOp)

Context Optimization (CoOp) [] is a pioneering work that applied prevalent prompt learning technique from the natural language processing (NLP) literature to CLIP. It addresses the issue of CLIP’s classification being sensitive to the prompt’s prefix (e.g., a large performance gap between when using ‘‘a photo of a [CLASS]’’ and ‘‘a [CLASS]’’) by optimising the prefix with few-shot samples. CoOp learns $M$ continuous context vectors within word-embedding space $V=\{V_{1},V_{2},\cdots,V_{M}\}$ where $V_{i}\in\mathbb{R}^{D}$ is the $i^{th}$ vector with $D$ being the word-embedding dimension which is 512 for CLIP. The learnable prompt is formalised as $P=\{V_{1},V_{2},\cdots,V_{M},C\}$ where $C\in\mathbb{R}^{D}$ is the word-embedding of a class name appended to the context vectors. The classification logit of $i^{th}$ class is then computed by $\langle I,P_{i}\rangle$ where $P_{i}$ is the learnable prompt with the $i^{th}$ class name. The probability is estimated by the Softmax function as $p(y=i|I,P_{i})=\frac{\exp{(\langle I,P_{i}\rangle/\tau)}}{\sum_{k=1}^{K}{\exp{
(\langle I,P_{k}\rangle/\tau)}}}$
where $K$ is the total number of classes and $\tau$ is the temperature scale. Cross-entropy loss is then minimised to learn the context vectors. Note that the only learnable parameters that are common among different prompt learning models are the $M$ context vectors.

Figure: Figure 1: Density plots of Energy scores (left) and MaxLogit (right) computed with CoOp [] on Flowers102 []. Large regions of ID and near OOD samples overlap, which are highlighted by shaded boxes.
Refer to caption: x1.png

### 2.3 Near Out-Of-Distribution Detection

OOD detections are largely categorised as far OOD detection and near OOD detection based on the distribution shift between an ID test dataset and an OOD dataset along with difficulty of detection [, , , , ]. Far OOD datasets have covariate shift in images (i.e., OOD samples are from different domains), and near OOD datasets which are more challenging to detect involve semantic shift (i.e., OOD samples are drawn from unseen classes). Near OOD detection with CLIP requires computing a score for each sample to determine whether a given image is from in-distribution (ID) or from OOD, which can be written as:

$$ $g\left(I;\{P_{i}\}_{i=1}^{K},\alpha\right)=\begin{cases}1&S\left(I;\{P_{i}\}_{ i=1}^{K}\right)\geq\alpha\\ 0&S\left(I;\{P_{i}\}_{i=1}^{K}\right)<\alpha\end{cases}$ (1) $$

where $g(\cdot)$ is a OOD detector, $\alpha$ is the threshold, and $S(\cdot)$ is a score function. By convention, the ground truth label is 1 for ID samples and 0 for OOD samples. We leverage two logit-based score functions of Energy score [] and MaxLogit score [] that are widely used and directly applicable to fine-tuned prompt learning models [, , , , , , , ]:

$$ $\displaystyle\textbf{Energy Score}\quad S_{Energy}=\tau\log{\sum_{i=1}^{K}{ \exp{(\langle I,P_{i}\rangle/\tau)}}}$ (2) $\displaystyle\textbf{MaxLogit Score}\quad S_{MaxLogit}=\max\limits_{i}\langle I ,P_{i}\rangle$ (3) $$

Energy score can be viewed as an approximation of MaxLogit score when $\tau=1$ as $\max\limits_{i}\langle I,P_{i}\rangle\leq\log{\sum_{i=1}^{K}{\exp{(\langle I,P
_{i}\rangle)}}}\leq\max\limits_{i}\langle I,P_{i}\rangle+\log{K}$.

## 3 Method

In this work, we focus on a near OOD detection problem for prompt learning models of CLIP, which is to detect whether a given image $I_{test}$ is from the ID test dataset $\mathcal{D}^{ID}_{test}$ of $(I^{ID}_{test},y^{ID})$ pairs or a near OOD dataset $\mathcal{D}^{nearOOD}_{test}$ of $(I^{nearOOD}_{test},y^{nearOOD})$ pairs where $y^{ID}\in\{1,\cdots,K\}$ is the ID label with $K$ classes and $y^{nearOOD}\in\{1,\cdots,L\}$ is the OOD with $L$ classes. The ID dataset and the near OOD dataset contain the same types of images (i.e., no covariate shift in images) but have no overlapping classes (i.e., $y^{ID}\cap y^{nearOOD}=\varnothing$). Without loss of generality, we assume that the context vectors $V$ have already been fine-tuned using a prompt learning model with a few-shot ID training dataset.

Energy and MaxLogit scores are two widely-used scores in the OOD detection literature [, , , , , , , ]. These scores measure a model’s confidence in its predictions, based on the assumption that ID samples typically have higher scores than OOD samples.
Despite their simplicity and renowned effectiveness in far OOD detection, we find that they are insufficient for near OOD detection, leading to suboptimal performance. This is illustrated in Figure 1, where a significant overlap between ID and near OOD scores can be observed.
Our objective is to develop a post-hoc method that improves MaxLogit and Energy scores to enhance near OOD detection for any CLIP-based prompt learning model with minimal learning cost.

We first introduce the concept of Empty-Class Score $S_{EC}=\langle I,V\rangle$, which represents the cosine similarity between the image and the context vectors *without* any class name. This score reflects how well the image aligns with generic, non-class-specific features, indicating the likelihood that the image does not belong to any particular ID class. In practice, Empty-Class Score can be easily obtained by adding an additional label class with no class name (i.e., extending $K$ classes to $K+1$ classes).

We argue that Empty-Class score is a valuable complement to MaxLogit or Energy score.
Ideally, if a model is confident that the image belongs to a specific ID class, its Energy or MaxLogit score should be large while its Empty-Class score should be small.
Conversely, if the model is uncertain about an image belonging to any ID class, its Energy or MaxLogit score should be small while its Empty-Class score should be large. This contrast between Energy or MaxLogit score and Empty-Class score can be viewed as an alternative score, which we name *Relative MaxLogit Score* and *Relative Energy Score* defined as:

$$ $\displaystyle\textbf{Relative Energy Score}\quad S_{R\text{-}Energy}=S_{Energy }-S_{EC}$ (4) $\displaystyle\textbf{Relative MaxLogit Score}\quad S_{R\text{-}MaxLogit}=S_{ MaxLogit}-S_{EC}$ (5) $$

Figure: (a) $S_{MaxLogit}$
Refer to caption: x2.png

We illustrate the effectiveness of the proposed scores in Figure 2(a) and Figure 2(b) where MaxLogit and Relative MaxLogit scores are plotted at y-axis and Empty-Class scores are plotted at x-axis. Geometrically, subtracting Empty-Class score from MaxLogit score is the same as applying vertical shearing transformation [] to the samples, which essentially reduces the overlapping area highlighted by shaded boxes. With the original MaxLogit score, the near OOD AUROC is 0.785 which is increased to 0.879 with Relative MaxLogit score.

In spite of the illustrated large improvement in AUROC, we observe that the performance is still suboptimal. In Figure 2(c), we introduce a coefficient $\beta$ termed *margin scale* that controls the amount of reduction of Empty-Class score from MaxLogit or Energy score. By enlarging the reduction by 2.2 times, the near OOD AUROC is improved from 0.879 to 0.942. With the margin scale, the relative scores can be rewritten as:

$$ $\displaystyle\textbf{Relative Energy Score}\quad S_{R\text{-}Energy}=S_{Energy }-\beta\cdot S_{EC}$ (6) $\displaystyle\textbf{Relative MaxLogit Score}\quad S_{R\text{-}MaxLogit}=S_{ MaxLogit}-\beta\cdot S_{EC}.$ (7) $$

A naive way to find the optimal margin scale is to find a scalar that maximises the near OOD AUROC. This, however, requires to have ground-truth near OOD samples, which is not available in most practical scenarios. Thus, we propose to estimate the margin scale by only using *ID training samples*. Initially, MaxLogit score and Empty-Class score in Figure 2(a) exhibit positive correlation. When the near OOD detection AUROC is maximised as shown in Figure 2(c), the correlation is minimised. Thus, we formulate this problem as finding the margin scale that minimises the correlation between MaxLogit or Energy score and Empty-Class score.

###### Lemma 3.1 .

Given $N$ scalar observations $\{\hat{x}_{i}\}_{i=1}^{N}$ and $\{\hat{y}_{i}\}_{i=1}^{N}$, we define two variables $x=\hat{x}$ and $y=\hat{y}-\beta\cdot\hat{x}$. The scale parameter $\beta$ that zeros out the covariance of two variables (i.e., the off-diagonals of a covariance matrix) which is approximated by maximum likelihood estimation is:

$$ $\beta=\frac{\sum_{i=1}^{N}{(\hat{x}_{i}-\mu_{\hat{x}})(\hat{y}_{i}-\mu_{\hat{y }})}}{\sum_{i=1}^{N}{(\hat{x}_{i}-\mu_{\hat{x}})^{2}}}$ (8) $$

where $\mu_{\hat{x}}=\frac{1}{N}\sum_{i=1}^{N}\hat{x}_{i}$ and $\mu_{\hat{y}}=\frac{1}{N}\sum_{i=1}^{N}\hat{y}_{i}$.

We propose to estimate the margin scale by fitting a bivariate normal distribution with maximum likelihood estimation. By using Lemma 3.1 with $\hat{y}$ being the MaxLogit score or Energy score and $\hat{x}$ being the Empty-Class score, the margin scale can be easily estimated with ID training samples (see Appendix A.1 for proof of the lemma). This scale estimation needs to be conducted only once after training finishes.
Figure 3 shows the approximation of the margin scale in red dotted lines. Even with a small number of ID training samples, we find that the estimation well approximate the margin scale close to the optimal point. Overall, the benefits of our approach are:

- 1.
Adding an additional class of Empty-Class is cost-effective that requires no additional forward pass.
- 2.
The approach can be applied to any prompt learning model as long as it learns the context vectors.
- 3.
The post-hoc process does not require any training, being applicable to fine-tuned prompt learning models and fast in terms of computation.

Figure: Figure 3: Near OOD detection AUROC using Relative MaxLogit score vs. margin scale for CoOp [], CoCoOp [], IVLP [], KgCoOp [], ProGrad [], MaPLe [], PromptSRC [], and LoCoOp [] on 16-shots UCF101 []. The margin scale is approximated by Eq. (8), shown as red dotted lines.
Refer to caption: x6.png

###### Lemma 3.1 .

## 4 Related Work

#### Vision-Language Models

Vision-language models have significantly advanced in recent years, bridging the gap between visual and textual data. Early approaches, such as image captioning models [, , ], typically used convolutional neural networks (CNNs) to extract visual features and recurrent neural networks (RNNs) to generate descriptive text. The advent of transformers [] handling long-range dependencies more effectively and contrastive learning [] revolutionised this field. Notably, ALIGN [], CLIP [], and LiT [] leveraged a contrastive learning framework that aligns image and text embeddings in a multimodal space, allowing for zero-shot learning capabilities and impressive generalisation to unseen tasks and datasets. In this work, we leverage the powerful vision-language model CLIP and extend its near OOD capability.

#### CLIP-based Prompt Learning

Despite the remarkable zero-shot performance of CLIP, CLIP shows inherently unstable classification accuracy that varies by wording of prompt. To mitigate this issue, CoOp [] was proposed to optimise a prompt in word embedding space, leveraging prompt learning from the NLP literature. CoCoOp [] identified that CoOp has limited generalisation and proposed to condition image features to the learnable prompt. Subsequently, many studies have proposed different techniques to improve the generalisation [, , , ]. While its generalisation has been largely improved, its OOD detection has been overlooked. LoCoOp [] proposed a OOD regularisation to improve OOD detection performance. Nevertheless, no study has addressed near OOD detection of prompt learning models.

#### OOD Detection

An early work of Hendrycks and Gimpel [] utilised the maximum softmax probability (MSP) as a score to identify OOD samples. Another notable work is Out-of-DIstribution detector for Neural networks (ODIN) [] which extends MSP by introducing temperature scaling and input pre-processing to enhance separation of the scores from ID samples and OOD samples. Similar to ODIN, Mahalanobis [] score also uses input pre-processing in addition to measuring distance in feature space. Delving into a more challenging task of near OOD detection, several studies analysed benchmarks of pre-trained networks in near OOD detection [, , , ], and different training methods and score functions were proposed for near OOD detection [, ]. Although significant advancements have been made in OOD detection for classifier-based neural networks, there is still room for improvement in detecting near OOD detection in vision-language models like CLIP, particularly in prompt learning models.

## 5 Experiments

### 5.1 Experimental Settings

#### Datasets

Following previous works of CLIP-based prompt learning models [, , , , , , ], we use 11 publicly available datasets of ImageNet [], Caltech101 [], OxfordPets [], StanfordCars [], Flowers102 [], Food101 [], , FGVCAircraft [], SUN397 [], DTD [], EuroSAT [], and UCF101 []. A common task of these works involves training models on half of the label classes (e.g., base classes) and evaluating them on the other half classes (e.g., new classes) to measure base-to-new generalisation. We reframe this task as a near OOD detection problem. Specifically, the models trained on base classes are tested with a dataset where half of the samples belong to the base classes (ID) and the other half to new classes (near OOD). In addition, we include CIFAR10 [] and CIFAR100 [], which are near OOD detection benchmarks [, , , , ]. For CIFAR10 and CIFAR100, we use all classes and evaluate with a test dataset consisting of both CIFAR10 test samples and CIAFR100 test samples, following the literature.

**Table 2: Near OOD AUROC ($\uparrow$) of prompt learning models over 13 datasets using the original and relative MaxLogit score.**
|  | Original | Relative | $\triangle$ |
| --- | --- | --- | --- |
| CoOp | 80.74 | 81.84 | +1.09 |
| CoCoOp | 81.09 | 82.74 | +1.65 |
| IVLP | 81.12 | 84.34 | +3.23 |
| KgCoOp | 80.84 | 83.12 | +2.28 |
| ProGrad | 79.77 | 82.35 | +2.58 |
| MaPLe | 81.06 | 83.94 | +2.88 |
| PromptSRC | 83.85 | 85.77 | +1.92 |
| LoCoOp | 77.55 | 81.74 | +4.18 |

#### Models

We use 8 prompt learning models: CoOp [], CoCoOp [], IVLP [], KgCoOp [], ProGrad [], MaPLe [], PromptSRC [], and LoCoOp [], all of which uses ViT-B/16 [] for the visual encoder and Transformer [] for the text encoder. We follow their training details to train them with 16, 8, 4, 2, and 1 shots using 3 random seeds (see Appendix A.2 for implementation details). For each model, we use MaxLogit score, Energy score, and MCM [] score as baselines. MCM is the SOTA score for CLIP. As our focus is on the post-hoc method directly applicable to fine-tuned prompt learning models without requiring near OOD samples, we exclude other scoring methods that require OOD samples, modifications of training procedures, or architecture changes in models.

### 5.2 Experimental Results

We report average AUROC of 16, 8, 4, 2, and 1 shots settings with 3 randoms seeds. Refer to Appendix A.3 for results of individual few-shot settings with standard deviations across the random seeds. In Table 2(n), we compare near OOD detection AUROC using the original and relative MaxLogit score. Positive improvements in AUROC are observed in 100 out of 104 evaluations (i.e., 13 datasets$\times$8 models) when the relative score is used. On average, the largest improvement was observed with LoCoOp, the recent prompt learning model for OOD detection. The same results for Energy score are shown in Table 12(n). Similar to MaxLogit score, AUROC is improved in 101 out of 104 evaluations when using Energy score.

Figure: Figure 4: Density plot of dataset distance between ID test dataset and near OOD dataset. The distance is measured when a relative score or MCM outperforms others in each dataset.
Refer to caption: x7.png

Refer to Table 6(n) and Table 13(n) for false positive rate (FPR) of near OOD samples when true positive rate (TPR) of ID samples is at 95%.

Table 3 shows a comparison between MaxLogit score (original and relative score), Energy score (original and relative score), and MCM, in terms of both AUROC and FPR95 averaged across 13 datasets. The relative MaxLogit and Energy scores outperform MCM in both AUROC and FPR95. When relative MaxLogit score and relative Energy score are compared, the relative MaxLogit score outperforms the relative Energy score for the half of the models in AUROC and all models in FPR95. This highlights that the proposed relative scores are better scores for near OOD detection tasks than MCM, and the relative MaxLogit score is better than the relative Energy score. Refer to Table 21(n) and Table 22(n) for the results of all datasets.

**Table 3: Near OOD AUROC ($\uparrow$) and FPR95 ($\downarrow$) of prompt learning models averaged over 13 datasets using the original and relative MaxLogit score and Energy score compared with MCM.**
|  | AUROC $\uparrow$ | FPR95 $\downarrow$ |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | MaxLogit | Energy | MCM | MaxLogit | Energy | MCM |  |  |  |  |
|  | Original | Relative | Original | Relative | Original | Relative | Original | Relative |  |  |
| CoOp | 80.74 | 81.84 | 80.44 | 81.71 | 79.41 | 58.23 | 54.85 | 59.45 | 55.37 | 63.70 |
| CoCoOp | 81.09 | 82.74 | 80.53 | 82.74 | 79.28 | 55.78 | 51.67 | 57.49 | 51.90 | 63.76 |
| IVLP | 81.12 | 84.34 | 80.49 | 84.40 | 80.95 | 55.65 | 48.58 | 57.66 | 49.19 | 60.05 |
| KgCoOp | 80.84 | 83.12 | 80.14 | 83.23 | 79.82 | 57.16 | 51.52 | 59.69 | 51.90 | 62.20 |
| ProGrad | 79.77 | 82.35 | 78.79 | 81.93 | 79.91 | 60.07 | 54.02 | 62.74 | 55.34 | 62.89 |
| MaPLe | 81.06 | 83.94 | 80.39 | 83.99 | 80.54 | 55.58 | 48.36 | 57.82 | 48.76 | 60.68 |
| PromptSRC | 83.85 | 85.77 | 83.48 | 85.88 | 82.34 | 49.65 | 44.60 | 51.86 | 45.15 | 55.86 |
| LoCoOp | 77.55 | 81.74 | 75.94 | 81.25 | 79.22 | 64.89 | 55.76 | 69.07 | 57.80 | 63.37 |

## 6 Discussion

#### Comparison with MCM

As discussed in Section 5.2, the proposed relative scores outperform MCM in AUROC and FPR95 averaged across 13 datasets. We notice that MCM outperforms these scores in some datasets. We analyse this difference from the perspective of dataset distance. As MCM is the SOTA OOD detection score for far OOD detection, MCM is expected to outperform as the dataset distance between an ID test dataset and a near OOD dataset increases. We validate this assumption by measuring the dataset distance by Optimal Transport Dataset Distance (OTDD) [] which is shown in Figure 4.

For each of MCM and relative scores, we find a dataset in which each score outperforms and measure the dataset distance. While the relative scores outperform across a wide range of distance, it excels when the distance decreases. MCM, on the other hand, tends to outperform as the distance increases. Nevertheless, the relative scores outperform in the majority of cases, resulting in superior average performance.

#### Application to MCM

A straightforward extension of our method is the application of post hoc process to the MCM score (i.e., $S_{MCM}-\beta\cdot S_{EC}$). The differences in AUROC and FPR95 with the original and relative MCM are presented in Table 19(n) and Table 20(n).
While it improves performance with MCM on most datasets, some datasets exhibit a decline in performance. This decline is due to the lack of a positive correlation between MCM and the Empty-Class score, as illustrated in Figure 2. Figure 5 further demonstrates the differing relationships between the score and the Empty-Class score. This discrepancy primarily arises from the softmax normalisation, where MCM only considers the relative magnitude of logits between classes, allowing non-maximum logits to unexpectedly change the relationship.

#### Application to Far OOD Detection

Another extension of our method is far OOD detection. Following the OOD literature of CLIP [, , , ], we use ImageNet as an ID dataset and iNaturalist [], SUN [], Places [], and Texture [] as far OOD datasets. We leverage the fine-tuned models in our experiments and show AUROC results in Table 23(d) and FPR95 results in Table 24(d). The post hoc framework is effective for both MaxLogit score and energy score, improving AUROC and FPR95 in 126 out of 128 evaluations where the relative scores outperform MCM in a half of evaluations. This aligns with the near OOD results, being effective across various models and OOD datasets. The limited performance also aligns with our observation in Figure 4 that MCM is more likely to outperform when the dataset distance increases.

## 7 Conclusion

In this work, we address few-shot near OOD detection of CLIP-based prompt learning models. To enhance existing logit-based scores, we propose a simple and fast post-hoc method applicable to any proompt learning model. Without changing training procedures of the existing models nor compromising classification accuracy, our method effectively enhances near OOD AUROC and lowers FPR95 for 8 recent prompt learning models across 13 real-world datasets. One limitation is that its effectiveness decreases for far OOD detection. Nonetheless, our method is more effective in a more challenging task of near OOD detection.

## Appendix A Appendix

### A.1 Proof of Lemma

We provide the proof of Lemma 3.1. For completeness of proof, we duplicate the lemma here.

###### Lemma A.1 .

Given $N$ scalar observations $\{\hat{x}_{i}\}_{i=1}^{N}$ and $\{\hat{y}_{i}\}_{i=1}^{N}$, we define two variables $x=\hat{x}$ and $y=\hat{y}-\beta\cdot\hat{x}$. The scale parameter $\beta$ that zeros out the covariance of two variables (i.e., the off-diagonals of a covariance matrix) which is approximated by maximum likelihood estimation is:

$$ $\beta=\frac{\sum_{i=1}^{N}{(\hat{x}_{i}-\mu_{\hat{x}})(\hat{y}_{i}-\mu_{\hat{y }})}}{\sum_{i=1}^{N}{(\hat{x}_{i}-\mu_{\hat{x}})^{2}}}$ (9) $$

where $\mu_{\hat{x}}=\frac{1}{N}\sum_{i=1}^{N}\hat{x}_{i}$ and $\mu_{\hat{y}}=\frac{1}{N}\sum_{i=1}^{N}\hat{y}_{i}$.

###### Proof.

It is well known that maximum likelihood estimation (MLE) of bivariate normal distribution for $N$ observations of variables $x$ and $y$ results in []:

$$ $\displaystyle\mu_{x}=\frac{1}{N}\sum_{i}{x_{i}},\quad\mu_{y}=\frac{1}{N}\sum_{ i}{y_{i}}$ (10) $\displaystyle\Sigma=\begin{bmatrix}\sigma_{xx}&\sigma_{xy}\\ \sigma_{xy}&\sigma_{yy}\end{bmatrix}=\frac{1}{N}\sum_{i}{\left(\begin{bmatrix} x_{i}\\ y_{i}\end{bmatrix}-\begin{bmatrix}\mu_{x}\\ \mu_{y}\end{bmatrix}\right)\left(\begin{bmatrix}x_{i}\\ y_{i}\end{bmatrix}-\begin{bmatrix}\mu_{x}\\ \mu_{y}\end{bmatrix}\right)^{T}}$ (11) $\displaystyle\sigma_{xy}=\frac{1}{N}\sum_{i}{(x_{i}-\mu_{x})(y_{i}-\mu_{y})}$ (12) $$

where $\mu_{x}$ and $\mu_{y}$ are the means of $x$ and $y$, and $\Sigma$ is the covariance matrix. We let $x=\hat{x}$ and $y=\hat{y}-\beta\cdot\hat{x}$ and find $\beta$ that makes $\sigma_{xy}=0$. By rewriting $\sigma_{xy}$ in terms of $\hat{x}$ and $\hat{y}$, we obtain $\beta$ as:

$$ $\displaystyle\sigma_{xy}$ $\displaystyle=\frac{1}{N}\sum_{i}{(\hat{x}_{i}-\mu_{\hat{x}})(\hat{y}_{i}- \beta\cdot\hat{x}_{i}-\mu_{\hat{y}}+\beta\cdot\mu_{\hat{x}})}=0$ (13) $\displaystyle\beta$ $\displaystyle=\frac{\sum_{i}{(\hat{x}_{i}-\mu_{\hat{x}})(\hat{y}_{i}-\mu_{\hat {y}})}}{\sum_{i}{(\hat{x}_{i}-\mu_{\hat{x}})^{2}}}$ (14) $$

The resulting $\beta$ is the ratio of covariance of $\hat{x}$ and $\hat{y}$ to variance of $\hat{x}$.
∎

###### Lemma A.1 .

###### Proof.

### A.2 Implementation Details

We follow officially released training guidelines for each prompt learning model using the same configuration files. The only additional line of code required is beta=(((y-y.mean())*(x-x.mean())).sum())/(((x-x.mean())**2).sum()) to estimate the margin scale in Eq. (8). Table 4 shows common hyperparameters which are the number of epochs, batch size, and context vectors initialisation. Refer to their officially released codes for other model-specific hyperparameters. All models were trained on a single NVIDIA GeForce RTX 3090 GPU with PyTorch framework. The temperature scaling is 0.01 for the Energy score and 1 for the MCM score.

**Table 4: Training details of the prompt learning models..**
|  | # Epochs | Batch Size | Context Vectors Initialisation |
| --- | --- | --- | --- |
| CoOp | 50 (ImageNet)<br>200 (Others) | 32 | “a photo of a” |
| CoCoOp | 10 | 1 |  |
| IVLP | 5 | 4 |  |
| KgCoOp | 100 | 128 |  |
| ProGrad | 200 | 32 |  |
| MaPLe | 5 | 4 |  |
| PromptSRC | 20 | 4 |  |
| LoCoOp | 50 | 32 | 16 vectors drawn from $\mathcal{N}(0,0.02)$ |

### A.3 Additional Experimental Results

We provide additional experimental results other than the results in the main section.

#### MaxLogit Score

We provide average AUROC and FPR95 across 13 datasets using MaxLogit in Table 5(n) and Table 6(n). Note that Table 5(n) is duplicated for completeness of this Section. From Table 7(n) to Table 11(n), average AUROC across 3 random seeds with standard deviations are reported for each few-shot setting. While overall improvements are observed across different few-shot settings, slight performance degradations are observed in EuroSAT and CIAFR10 at 1-shot. This is because of insufficient training samples used to find the optimal margin scale where EuroSAT and CIFAR10 have 5 classes and 10 classes in their training datasets respectively.

**Table 5: Near OOD AUROC ($\uparrow$) of prompt learning models over 13 datasets using the original and relative MaxLogit score.**
|  | Original | Relative | $\triangle$ |
| --- | --- | --- | --- |
| CoOp | 80.74 | 81.84 | +1.09 |
| CoCoOp | 81.09 | 82.74 | +1.65 |
| IVLP | 81.12 | 84.34 | +3.23 |
| KgCoOp | 80.84 | 83.12 | +2.28 |
| ProGrad | 79.77 | 82.35 | +2.58 |
| MaPLe | 81.06 | 83.94 | +2.88 |
| PromptSRC | 83.85 | 85.77 | +1.92 |
| LoCoOp | 77.55 | 81.74 | +4.18 |

**Table 6: Near OOD FPR95 ($\downarrow$) of prompt learning models over 13 datasets using the original and relative MaxLogit score.**
|  | Original | Relative | $\triangle$ |
| --- | --- | --- | --- |
| CoOp | 58.23 | 54.85 | -3.38 |
| CoCoOp | 55.78 | 51.67 | -4.11 |
| IVLP | 55.65 | 48.58 | -7.07 |
| KgCoOp | 57.16 | 51.52 | -5.64 |
| ProGrad | 60.07 | 54.02 | -6.05 |
| MaPLe | 55.58 | 48.36 | -7.22 |
| PromptSRC | 49.65 | 44.60 | -5.05 |
| LoCoOp | 64.89 | 55.76 | -9.12 |

**Table 7: Near OOD AUROC ($\uparrow$) of prompt learning models over 13 datasets using the original and relative MaxLogit score with 16-shots.**
|  | Original | Relative | $\triangle$ |
| --- | --- | --- | --- |
| CoOp | 82.82±10.81 | 83.86±10.79 | +1.03±1.08 |
| CoCoOp | 81.51±12.87 | 83.65±11.76 | +2.14±2.32 |
| IVLP | 83.46±10.49 | 87.48±8.02 | +4.02±3.96 |
| KgCoOp | 81.41±12.12 | 84.08±11.38 | +2.67±2.38 |
| ProGrad | 81.76±11.56 | 83.83±11.66 | +2.07±2.43 |
| MaPLe | 83.83±10.89 | 86.35±10.06 | +2.52±1.97 |
| PromptSRC | 85.76±9.23 | 87.52±9.07 | +1.76±1.55 |
| LoCoOp | 79.26±11.23 | 82.30±12.45 | +3.05±3.61 |

**Table 8: Near OOD AUROC ($\uparrow$) of prompt learning models over 13 datasets using the original and relative MaxLogit score with 8-shots.**
|  | Original | Relative | $\triangle$ |
| --- | --- | --- | --- |
| CoOp | 81.60±11.10 | 82.84±11.08 | +1.24±1.01 |
| CoCoOp | 81.48±12.41 | 83.33±11.93 | +1.85±1.55 |
| IVLP | 82.20±11.80 | 85.11±10.87 | +2.91±3.80 |
| KgCoOp | 80.97±12.22 | 83.38±11.86 | +2.40±2.18 |
| ProGrad | 80.39±12.37 | 82.96±12.03 | +2.58±2.37 |
| MaPLe | 82.67±12.39 | 85.32±12.13 | +2.64±3.29 |
| PromptSRC | 85.44±9.45 | 87.36±8.88 | +1.92±1.70 |
| LoCoOp | 79.25±10.77 | 82.96±10.27 | +3.71±4.19 |

**Table 9: Near OOD AUROC ($\uparrow$) of prompt learning models over 13 datasets using the original and relative MaxLogit score with 4-shots.**
|  | Original | Relative | $\triangle$ |
| --- | --- | --- | --- |
| CoOp | 81.06±11.11 | 82.10±11.35 | +1.04±1.56 |
| CoCoOp | 81.45±12.10 | 83.09±12.43 | +1.63±2.06 |
| IVLP | 81.14±12.63 | 84.08±12.27 | +2.94±2.39 |
| KgCoOp | 80.69±12.01 | 83.03±12.20 | +2.34±2.02 |
| ProGrad | 80.58±11.41 | 82.50±11.50 | +1.92±1.76 |
| MaPLe | 80.76±11.98 | 83.53±11.47 | +2.78±3.03 |
| PromptSRC | 84.27±9.89 | 86.10±9.92 | +1.83±1.54 |
| LoCoOp | 78.52±11.12 | 81.57±11.62 | +3.06±4.44 |

**Table 10: Near OOD AUROC ($\uparrow$) of prompt learning models over 13 datasets using the original and relative MaxLogit score with 2-shots.**
|  | Original | Relative | $\triangle$ |
| --- | --- | --- | --- |
| CoOp | 80.05±11.11 | 81.19±11.05 | +1.14±1.59 |
| CoCoOp | 80.76±12.16 | 82.12±12.60 | +1.35±1.70 |
| IVLP | 80.12±11.76 | 82.87±10.97 | +2.75±2.35 |
| KgCoOp | 80.83±11.54 | 83.13±11.49 | +2.30±2.01 |
| ProGrad | 79.43±11.87 | 82.07±11.48 | +2.64±2.91 |
| MaPLe | 80.49±11.63 | 83.42±11.27 | +2.92±2.35 |
| PromptSRC | 83.18±10.43 | 84.99±10.53 | +1.82±1.58 |
| LoCoOp | 76.28±12.95 | 80.62±11.92 | +4.34±4.81 |

**Table 11: Near OOD AUROC ($\uparrow$) of prompt learning models over 13 datasets using the original and relative MaxLogit score with 1-shot.**
|  | Original | Relative | $\triangle$ |
| --- | --- | --- | --- |
| CoOp | 77.78±11.44 | 78.69±11.97 | +0.92±2.15 |
| CoCoOp | 79.70±13.00 | 81.26±12.66 | +1.56±2.54 |
| IVLP | 80.00±10.91 | 82.27±11.08 | +2.26±3.19 |
| KgCoOp | 80.19±12.35 | 81.98±12.90 | +1.79±2.74 |
| ProGrad | 78.26±12.72 | 80.32±12.20 | +2.06±2.87 |
| MaPLe | 78.86±13.76 | 81.13±13.97 | +2.27±4.69 |
| PromptSRC | 81.56±12.17 | 82.81±13.06 | +1.25±2.45 |
| LoCoOp | 76.37±12.38 | 79.56±11.73 | +3.19±5.99 |

#### Energy Score

We provide the same results of Table 5(n) to Table 11(n) using Energy score in Table 12(n) to Table 18(n).

**Table 12: Near OOD AUROC ($\uparrow$) of prompt learning models over 13 datasets using the original and relative Energy score.**
|  | Original | Relative | $\triangle$ |
| --- | --- | --- | --- |
| CoOp | 80.44 | 81.71 | +1.27 |
| CoCoOp | 80.53 | 82.74 | +2.21 |
| IVLP | 80.49 | 84.40 | +3.91 |
| KgCoOp | 80.14 | 83.23 | +3.09 |
| ProGrad | 78.79 | 81.93 | +3.14 |
| MaPLe | 80.39 | 83.99 | +3.60 |
| PromptSRC | 83.48 | 85.88 | +2.40 |
| LoCoOp | 75.94 | 81.25 | +5.31 |

**Table 13: Near OOD FPR95 ($\downarrow$) of prompt learning models over 13 datasets using the original and relative Energy score.**
|  | Original | Relative | $\triangle$ |
| --- | --- | --- | --- |
| CoOp | 59.45 | 55.37 | -4.08 |
| CoCoOp | 57.49 | 51.90 | -5.59 |
| IVLP | 57.66 | 49.19 | -8.47 |
| KgCoOp | 59.69 | 51.90 | -7.80 |
| ProGrad | 62.74 | 55.34 | -7.41 |
| MaPLe | 57.82 | 48.76 | -9.07 |
| PromptSRC | 51.86 | 45.15 | -6.71 |
| LoCoOp | 69.07 | 57.80 | -11.27 |

**Table 14: Near OOD AUROC ($\uparrow$) of prompt learning models over 13 datasets using the original and relative Energy score with 16-shots.**
|  | Original | Relative | $\triangle$ |
| --- | --- | --- | --- |
| CoOp | 82.67±10.20 | 83.84±10.18 | +1.17±1.20 |
| CoCoOp | 81.21±12.20 | 83.70±10.99 | +2.49±2.54 |
| IVLP | 83.25±9.90 | 87.55±7.60 | +4.30±4.05 |
| KgCoOp | 81.02±11.18 | 84.36±9.84 | +3.34±2.72 |
| ProGrad | 80.90±11.25 | 83.32±11.27 | +2.42±2.81 |
| MaPLe | 83.60±10.11 | 86.54±9.03 | +2.94±2.28 |
| PromptSRC | 85.53±8.47 | 87.67±8.19 | +2.13±1.90 |
| LoCoOp | 78.03±10.52 | 81.70±11.89 | +3.67±4.56 |

**Table 15: Near OOD AUROC ($\uparrow$) of prompt learning models over 13 datasets using the original and relative Energy score with 8-shots.**
|  | Original | Relative | $\triangle$ |
| --- | --- | --- | --- |
| CoOp | 81.38±10.64 | 82.77±10.65 | +1.39±1.17 |
| CoCoOp | 81.16±11.79 | 83.44±10.96 | +2.28±1.87 |
| IVLP | 81.92±11.35 | 85.27±10.31 | +3.34±3.78 |
| KgCoOp | 80.61±11.35 | 83.71±10.47 | +3.10±2.60 |
| ProGrad | 79.71±12.08 | 82.65±11.65 | +2.93±2.57 |
| MaPLe | 82.29±11.58 | 85.45±11.08 | +3.16±3.24 |
| PromptSRC | 85.28±8.51 | 87.53±8.05 | +2.26±1.90 |
| LoCoOp | 78.06±10.24 | 82.63±9.46 | +4.57±4.78 |

**Table 16: Near OOD AUROC ($\uparrow$) of prompt learning models over 13 datasets using the original and relative Energy score with 4-shots.**
|  | Original | Relative | $\triangle$ |
| --- | --- | --- | --- |
| CoOp | 80.96±10.65 | 82.10±10.96 | +1.15±1.67 |
| CoCoOp | 81.06±11.56 | 83.05±11.69 | +1.99±2.22 |
| IVLP | 80.84±12.25 | 84.23±11.72 | +3.39±2.85 |
| KgCoOp | 80.36±11.15 | 83.31±10.95 | +2.95±2.45 |
| ProGrad | 79.95±10.93 | 82.21±10.85 | +2.26±1.91 |
| MaPLe | 80.39±11.25 | 83.76±10.48 | +3.37±3.27 |
| PromptSRC | 84.22±8.81 | 86.37±8.78 | +2.15±1.78 |
| LoCoOp | 77.45±10.27 | 81.35±10.79 | +3.90±5.10 |

**Table 17: Near OOD AUROC ($\uparrow$) of prompt learning models over 13 datasets using the original and relative Energy score with 2-shots.**
|  | Original | Relative | $\triangle$ |
| --- | --- | --- | --- |
| CoOp | 79.88±10.72 | 81.13±10.73 | +1.25±1.73 |
| CoCoOp | 80.48±11.52 | 82.25±11.63 | +1.77±1.74 |
| IVLP | 79.75±11.35 | 83.11±10.12 | +3.35±2.75 |
| KgCoOp | 80.47±10.84 | 83.35±10.44 | +2.88±2.41 |
| ProGrad | 78.86±11.41 | 81.87±11.07 | +3.02±3.25 |
| MaPLe | 80.20±10.89 | 83.68±10.41 | +3.48±2.64 |
| PromptSRC | 82.99±9.69 | 85.12±9.76 | +2.14±1.75 |
| LoCoOp | 75.13±12.35 | 80.57±10.96 | +5.44±5.58 |

**Table 18: Near OOD AUROC ($\uparrow$) of prompt learning models over 13 datasets using the original and relative Energy score with 1-shot.**
|  | Original | Relative | $\triangle$ |
| --- | --- | --- | --- |
| CoOp | 77.68±10.89 | 78.75±11.51 | +1.08±2.34 |
| CoCoOp | 79.42±12.18 | 81.51±11.59 | +2.09±2.95 |
| IVLP | 79.62±10.43 | 82.58±10.29 | +2.96±3.38 |
| KgCoOp | 79.88±11.50 | 82.26±11.74 | +2.38±3.14 |
| ProGrad | 77.86±12.21 | 80.27±11.67 | +2.41±3.09 |
| MaPLe | 78.46±13.27 | 81.19±13.11 | +2.73±5.06 |
| PromptSRC | 81.76±10.60 | 83.23±11.69 | +1.47±3.32 |
| LoCoOp | 75.17±11.73 | 79.38±10.84 | +4.22±6.61 |

#### MCM Score

We show the effectiveness of MCM measured by average AUROC and FPR95 across 13 datasets in Table 19(n) and Table 20(n). Also, we compare the correlation between scores and the Empty-Class score using MaxLogit and MCM in Figure 5.

**Table 19: Near OOD AUROC ($\uparrow$) of prompt learning models over 13 datasets using the original and relative MCM score.**
|  | Original | Relative | $\triangle$ |
| --- | --- | --- | --- |
| CoOp | 79.41 | 81.31 | +1.90 |
| CoCoOp | 79.28 | 82.59 | +3.31 |
| IVLP | 80.95 | 84.48 | +3.53 |
| KgCoOp | 79.82 | 83.31 | +3.49 |
| ProGrad | 79.91 | 81.81 | +1.90 |
| MaPLe | 80.54 | 83.81 | +3.27 |
| PromptSRC | 82.34 | 85.49 | +3.15 |
| LoCoOp | 79.22 | 80.36 | +1.14 |

**Table 20: Near OOD FPR95 ($\downarrow$) of prompt learning models over 13 datasets using the original and relative MCM score.**
|  | Original | Relative | $\triangle$ |
| --- | --- | --- | --- |
| CoOp | 63.70 | 57.48 | -6.21 |
| CoCoOp | 63.76 | 53.62 | -10.14 |
| IVLP | 60.05 | 49.23 | -10.83 |
| KgCoOp | 62.20 | 51.85 | -10.35 |
| ProGrad | 62.89 | 56.22 | -6.67 |
| MaPLe | 60.68 | 49.00 | -11.68 |
| PromptSRC | 55.86 | 46.13 | -9.73 |
| LoCoOp | 63.37 | 60.63 | -2.74 |

Figure: Figure 5: Comparison of MaxLogit (left) or MCM score (right) vs. Empty-Class score for near OOD samples with IVLP [] on EuroSAT [].
Refer to caption: x8.png

#### Score Comparison

In the main section, average AUROC and FPR95 of different scores are compared in Table 3. Here, we provide the individual’s dataset results averaged across 16, 8, 4, 2, and 1 shots with 3 random seeds in Table 21(n) and Table 22(n).

**Table 21: Near OOD AUROC ($\uparrow$) of prompt learning models across 13 datasets using the original and relative MaxLogit score and Energy score compared with MCM.**
|  | MaxLogit | Energy | MCM |  |  |
| --- | --- | --- | --- | --- | --- |
|  | Original | Relative | Original | Relative |  |
| CoOp | 80.74 | 81.84 | 80.44 | 81.71 | 79.41 |
| CoCoOp | 81.09 | 82.74 | 80.53 | 82.74 | 79.28 |
| IVLP | 81.12 | 84.34 | 80.49 | 84.40 | 80.95 |
| KgCoOp | 80.84 | 83.12 | 80.14 | 83.23 | 79.82 |
| ProGrad | 79.77 | 82.35 | 78.79 | 81.93 | 79.91 |
| MaPLe | 81.06 | 83.94 | 80.39 | 83.99 | 80.54 |
| PromptSRC | 83.85 | 85.77 | 83.48 | 85.88 | 82.34 |
| LoCoOp | 77.55 | 81.74 | 75.94 | 81.25 | 79.22 |

**Table 22: Near OOD FPR95 ($\downarrow$) of prompt learning models across 13 datasets using the original and relative MaxLogit score and Energy score compared with MCM.**
|  | MaxLogit | Energy | MCM |  |  |
| --- | --- | --- | --- | --- | --- |
|  | Original | Relative | Original | Relative |  |
| CoOp | 58.23 | 54.85 | 59.45 | 55.37 | 63.70 |
| CoCoOp | 55.78 | 51.67 | 57.49 | 51.90 | 63.76 |
| IVLP | 55.65 | 48.58 | 57.66 | 49.19 | 60.05 |
| KgCoOp | 57.16 | 51.52 | 59.69 | 51.90 | 62.20 |
| ProGrad | 60.07 | 54.02 | 62.74 | 55.34 | 62.89 |
| MaPLe | 55.58 | 48.36 | 57.82 | 48.76 | 60.68 |
| PromptSRC | 49.65 | 44.60 | 51.86 | 45.15 | 55.86 |
| LoCoOp | 64.89 | 55.76 | 69.07 | 57.80 | 63.37 |

#### Far OOD Detection

We provide additional far OOD detection results of AUROC and FPR in Table 23(d) and Table 24(d) respectively.

**Table 23: OOD detection AUROC ($\uparrow$) averaged over 16, 8, ,4, 2, and 1 few-shot settings with 3 random seeds.**
|  | MaxLogit | Energy | MCM |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
|  | Original | Relative | $\triangle$ | Original | Relative | $\triangle$ |  |
| CoOp | 96.22 | 96.78 | +0.56 | 95.80 | 96.65 | +0.85 | 95.92 |
| CoCoOp | 96.90 | 96.79 | -0.11 | 96.50 | 96.51 | +0.01 | 96.21 |
| IVLP | 95.79 | 96.01 | +0.22 | 95.18 | 95.77 | +0.60 | 95.97 |
| KgCoOp | 96.67 | 96.91 | +0.24 | 96.12 | 96.74 | +0.62 | 96.30 |
| ProGrad | 95.21 | 96.05 | +0.84 | 94.52 | 95.76 | +1.25 | 95.30 |
| MaPLe | 96.61 | 96.52 | -0.09 | 96.00 | 96.11 | +0.11 | 96.26 |
| PromptSRC | 96.54 | 96.59 | +0.05 | 96.00 | 96.20 | +0.21 | 96.40 |
| LoCoOp | 94.67 | 97.10 | +2.43 | 91.59 | 96.45 | +4.86 | 96.57 |

**Table 24: OOD detection FPR95 ($\downarrow$) averaged over 16, 8, ,4, 2, and 1 few-shot settings with 3 random seeds.**
|  | MaxLogit | Energy | MCM |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
|  | Original | Relative | $\triangle$ | Original | Relative | $\triangle$ |  |
| CoOp | 17.58 | 13.56 | -4.01 | 20.90 | 14.42 | -6.48 | 16.40 |
| CoCoOp | 13.01 | 12.92 | -0.09 | 15.41 | 14.09 | -1.33 | 14.78 |
| IVLP | 17.82 | 15.32 | -2.50 | 22.14 | 16.67 | -5.47 | 15.41 |
| KgCoOp | 14.43 | 12.46 | -1.97 | 18.32 | 13.33 | -4.99 | 14.34 |
| ProGrad | 22.47 | 16.28 | -6.19 | 28.38 | 17.85 | -10.52 | 18.51 |
| MaPLe | 15.37 | 14.01 | -1.37 | 19.35 | 16.74 | -2.62 | 14.32 |
| PromptSRC | 14.88 | 13.77 | -1.11 | 18.70 | 16.04 | -2.66 | 13.77 |
| LoCoOp | 27.58 | 11.05 | -16.52 | 53.58 | 14.25 | -39.33 | 12.43 |

#### Broader Impacts

This work have a number of potential positive societal impacts. Firstly, as our work aims to improve reliability of CLIP-based prompt learning models, the model will be more trustworthy, which is an important factor in many safety-critical applications. Secondly, as our method does not cost any re-training models, it is computationally efficient, being eco-friendly with minimal carbon footprint. Nonetheless, a potential negative impact may possibly exist when the model is deployed without fully understanding its limitations.

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: - I believe the paper would benefit from an ablation study on how the number of shots impacts the learning of the margin scale hyperparameter and, subsequently, the near-OOD detection performance.

- How does MLE compare to MCM on datasets that contain both near- and far-OOD samples?
soundness: 3
strengths: I believe the paper addresses an important and often overlooked problem. The proposed Marginal Logit Score is simple and appears effective across a wide range of models and datasets, outperforming conventional OOD metrics in near-OOD detection. The metric is intuitively appealing and supported by experimental results.
summary: This paper explores the few-shot near-OOD identification capabilities of various prompt-based vision-language models (VLMs), such as CLIP. To enhance the identification of near-OOD samples in these models, it introduces a set of post-hoc metrics based on existing logit-based OOD techniques, which can be applied across a wide range of prompt-based VLMs.
weaknesses: -The authors mention that the margin scale is learned from the few-shot ID examples provided. However, there is insufficient analysis of how the number of shots impacts parameter learning and, ultimately, near-OOD detection performance.

- Although far-OOD detection is not the focus of this work, far-OOD samples are important for a model’s overall effectiveness in identifying OOD samples, especially in real-world scenarios. From this perspective, it is crucial to compare the performance of MLE with MCM on datasets or in scenarios where both near- and far-OOD samples are present.

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 3
questions: Please refer to the weakness.
soundness: 3
strengths: 1.	They introduced an MLS score based on the observation that the logit scores from CLIP, such as max logit, are positively correlated with their proposed context score. This MLS score reduces the overlap between ID data and near ID data compared to the max logit score.

2.	The MLS scoring method proposed in this paper requires no modifications to the model architecture and does not involve retraining. This characteristic makes it efficient and highly adaptable.
summary: In response to the near OOD dataset and CLIP, this paper designed an OOD score based on logit, referred to as MLS. MLS computes the difference between the logit scores inferred by CLIP and a newly introduced complementary score known as the Context score. This approach effectively enhances the separation between ID samples and near OOD samples, resulting in a significant performance improvement.
weaknesses: 1.	The authors mention that logit score and context score are proportional, but this claim is not sufficiently accurate. 
a)	The paper presents the case that Maxlogit is proportional to context score, but it remains unclear whether energy score and other logit-based scores follow this relationship as well. It is recommended that corresponding images be provided in the appendix. 
b)	The proof that Maxlogit is proportional to the context score is based on experiments with MaPLe on the Caltech101 dataset, where we observed that ID and OOD data showed approximately linearly separable patterns in the images. It is recommended that the authors validate whether this proportionality and linear separability occur in other datasets and include those images in the appendix.
2.	The paper considers the Base to new dataset as a near OOD dataset. However, it is suggested that the authors also explore more classic near OOD datasets, such as those used in the MCM(Delving into Out-of-Distribution Detection with Vision-Language Representations) paper, where Imagenet10 is used as the ID dataset and Imagenet20 as the OOD dataset. Adding experiments involving these classic near OOD datasets would be beneficial.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: Please refer to the Weaknesses.
soundness: 3
strengths: 1. The finding is interesting.

2. The experimental results have validated the effectiveness of the proposed method.
summary: This paper introduces a novel post-hoc method to improve the near Out-Of-Distribution (OOD) detection capabilities of prompt learning models for vision-language tasks, without requiring architectural changes or retraining. The proposed method, named Marginal Logit Score (MLS), leverages the Context Score computed based on a variant of existing prompts to enhance near OOD detection performance . Experiments on 13 datasets demonstrate that MLS significantly boosts the near OOD detection performance, while maintaining the same classification accuracy.
weaknesses: 1. The cause of the phenomenon is not clearly explained. 

The author find: Compared to ID images, OOD images have a smaller gap between the original score and the Context Score. Based on this phenomenon, the author proposes combining Energy (or MaxLogit) and Context Score to identify OOD. However, the author does not deeply analyze this phenomenon and explain its cause. I speculate that the cause might be because when CLASS is present, CLASS plays the main role in the Prompt, and at this time, regardless of whether it is an ID image or an OOD image, the attention is on CLASS, so the scores are similar. When CLASS is removed, the attention of ID images and OOD images to the remaining vectors in the Prompt will differ, thus producing different scores.

2. The discussion is not thorough enough. 

Why combine Energy (or MaxLogit) with Context Score? I see that there is already a significant difference in Context Score between ID images and OOD images. Can't we directly use the Context Score to distinguish between the two types of images?

This work discovers a phenomenon in prompt-based learning and designs an OOD detection algorithm based on this phenomenon, but it does not delve into the cause for the occurrence of this phenomenon, so I am somewhat concerned that this may not be a universal phenomenon. If the authors could provide more analysis on it, the article could be further improved.

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 3
questions: 1. The OOD score is a measure of OOD detection performance. There have been numerous methods for calculating OOD scores in CLIP-based out-of-distribution detection, including, but not limited to, the papers mentioned [1, 2, 3, 4]. The comparison is too narrow, and I suggest that the authors compare with more methods to analyze the effectiveness of the proposed approach.

2. The comparison is not fair. In Table 2, the paper merely compares with prompt-based methods and achieves seemingly significant improvements. However, as shown in Table 3, the improvement becomes marginal when assessed using different metrics and OOD-based methods.

3. The writing may be somewhat confusing. For example, in the near OOD setting, which dataset is the ID dataset? I did my best but could not find any description.

4. Based on the results shown for the given datasets, the effectiveness of the method is not convincing. I suggest that the authors provide more visualizations and additional results to demonstrate its effectiveness and generalization.


[1] Learning Transferable Negative Prompts for Out-of-Distribution Detection.

[2] Negative Label Guided OOD Detection with Pretrained Vision-Language Models.

[3] Enhancing Outlier Knowledge for Few-Shot Out-of-Distribution Detection with Extensible Local Prompts.

[4] CLIPN for Zero-Shot OOD Detection: Teaching CLIP to Say No.
soundness: 2
strengths: 1. The proposed method boosts existing prompt learning methods' near OOD detection performance in AUROC by up to 11.67% with minimal computational

2. The method can be easily applied to any prompt learning model without change in architecture or model re-training.
summary: The paper introduces a new logit-based score named Marginal Logit Score (MLS) based on existing logit scores and a new complementary score named Context score.
weaknesses: 1. Lack of novelty: The contribution of the paper is limited. All I could discover is a post-hoc OOD score. More description and deeper analyses are needed to certify its usefulness.

2. Incomplete comparison with current methods: The paper mainly compares with few-shot based methods but not OOD-based methods for VLM [1, 2, 3], which should be addressed.

3. Marginal improvements: The performance gains in the main tables are marginal, even when compared with the basic MaxLogit.

4. The writing of the paper should be improved and standardized, including but not limited to the usage of quotation marks in Sec. 2.1 and the usage of superscripts and subscripts in Sec. 3.1, which may confuse readers.

Meta-review: In this work, the authors propose a logit-based MLS score and context score to effectively leverage domain-specific and class-specific features, significantly enhancing near-OOD detection performance by up to 11.67% in AUROC through few-shot prompt learning. Extensive experiments conducted on 13 datasets and 8 prompt learning methods demonstrate the effectiveness of the proposed approach. Most reviewers acknowledge the method’s ability to substantially boost near-OOD detection with minimal computational cost, without modifying the model architecture or requiring model retraining. However, the paper received mixed reviews — with two borderline accept, one borderline reject, and one reject — resulting in an average score of 5. The reviewers who assigned negative ratings expressed concerns about the paper's clarity, specifically the need for a more detailed explanation of the underlying reasons behind the performance improvements observed with MLS and context scores. They also suggested the inclusion of additional baseline comparisons. Despite the authors' rebuttal, these concerns persisted. After reviewing the feedback and rebuttal, I agree that the authors should incorporate comparisons with more recent OOD methods, such as NegLabel [1], and further refine the manuscript for resubmission to a future venue.

[1] Jiang, Xue, Feng Liu, Zhen Fang, Hong Chen, Tongliang Liu, Feng Zheng, and Bo Han. "Negative label guided ood detection with pretrained vision-language models." ICLR 2024.

Decision: Reject