LoCoOp: Few-Shot Out-of-Distribution Detection via Prompt Learning
Authors: Atsuyuki Miyai
Qing Yu
Go Irie
Kiyoharu Aizawa
Venue: NeurIPS 2023 poster (2023)
Keywords: out-of-distribution detection
vision-language foundation model
prompt learning
TL;DR: An effective prompt learning approach for few-shot out-of-distribution detection

Abstract: We present a novel vision-language prompt learning approach for few-shot out-of-distribution (OOD) detection. Few-shot OOD detection aims to detect OOD images from classes that are unseen during training using only a few labeled in-distribution (ID) images. While prompt learning methods such as CoOp have shown effectiveness and efficiency in few-shot ID classification, they still face limitations in OOD detection due to the potential presence of ID-irrelevant information in text embeddings. To address this issue, we introduce a new approach called $\textbf{Lo}$cal regularized $\textbf{Co}$ntext $\textbf{Op}$timization (LoCoOp), which performs OOD regularization that utilizes the portions of CLIP local features as OOD features during training. CLIP's local features have a lot of ID-irrelevant nuisances ($\textit{e.g.}$, backgrounds), and by learning to push them away from the ID class text embeddings, we can remove the nuisances in the ID class text embeddings and enhance the separation between ID and OOD. Experiments on the large-scale ImageNet OOD detection benchmarks demonstrate the superiority of our LoCoOp over zero-shot, fully supervised detection methods and prompt learning methods. Notably, even in a one-shot setting -- just one label per class, LoCoOp outperforms existing zero-shot and fully supervised detection methods. The code is available via https://github.com/AtsuMiyai/LoCoOp.

LoCoOp: Few-Shot Out-of-Distribution Detection via Prompt Learning
Atsuyuki Miyai' Qing Yu Go Irie
Kiyoharu Aizawa'
'The University of Tokyo fmiyai, yu, aizaaa u-t.uoyo..ac... jc.jp goirießiese. org
LY Corporation
Tooyoo University of Science
Abstract
We present novel vision-language prompt learning approach for few-shot out-of- distribution (OOD) detection. Few-shot OOD detection aims detect OOD images from classes that are unseen during training using only a few labeled in-distribution (ID) images. While prompt learning methods such as CoOp [60] have shown effectiveness and efficiency in few-shot ID classification, they still face limitations OOD detection due to the potential presence ID-irrelevant information text embeddings. To address this issue, we introduce a new approach called Local regularized Context Optimization (LoCoOp), which performs OOD) regularization that utilizes the portions of CLIP local features as OOD features during training.
CLIP's local features have a lot of ID-irrelevant nuisances (e.g., backgrounds), and by learning to push them away from the ID class text embeddings, remove the nuisances in the ID class text embeddings and enhance the separation between ID and OOD. Experiments on the large-scale ImageNet OOD detection benchmarks demonstrate the superiority supervised detection methods and prompt learning methods. Notably, even in a one-shot setting - just one label per class, LoCoOp outperforms existing zero- shot and fully supervised detection methods. The code is available via https: //gi thtb....... com/ AtsuMiyai/LLooopp
our LoCoOp over zero-shot, fully
Introduction
Detecting -fittiiibiiiitoo (OOD) samples is crucial for deploying machine learning models in
real-world scenarios, where new classes of samples can naturally arise and require caution. Common approaches for OOD detection use single-modal supervised learning [15, 17, 19, 28, 45, 29, 26, 50, 1]. These supervised methods achieve good results but have limitations e.g., these methods require lot of computational and annotation costs for training. These days, CLIP [37] achieves surprising performances for various downstream tasks, and the application of CLIP to OOD detection is beginning to attract increasing attention [11, 30, 33,
Previous studies CLIP-based OOD detection have focused on extreme settings. Some studies [12, 11, 30, 33] have explored zero-shot OOD detection methods that do not require any in in-distribution (ID) training data, whereas other studies [46] have developed fully supervised methods that require the entire training data. These methods have comparable performances, but both approaches have their own limitations. On the one hand, the zero-shot methods [30, 33] do not require any training data, but they may encounter domain gap with downstream data, which limits the performance of zero-shot methods. On the other hand, fully supervised methods [46] utilize the entire ID training data but may destroy the rich representations of CLIP by fine-tuning, which also limits the performance despite requiring enormous training costs. To overcome the limitations of fully supervised and zero-shot methods, is essential to develop a few-shot OOD detection method that utilizes few ID training images for OOD detection. By finding balance between the zero-shot
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

horse
cat
dog
1
2
Pixel-based Entropy Maximize
ID
Image
2
1
Pick nulsance
Logits
dog
1
1
Text
W1
W2
W
3
3
horse
Exclude reogg-te regions
M classes
Figure 1: Overview of the OOD regularization for LoCoOp. Our approach first extracts ID-irreverent nuisances (OOD) from CLIP's local features. Then, it performs entropy maximization on a per-pixel basis for the extracted OOD features. This OOD regularization allows OOD nuisances and all ID text embeddings to be dissimilar, thus preventing the inclusion of nuisances in ID text embeddings and improving test-time detection performance.
and fine-tuned methods, feo-shot OOD detection has the potential to provide a more efficient and effective solution for OOD detection.
In this study, we tackle CLIP-based few-shot OOD detection, where we utilize only a few ID images for training CLIP. The most common approach to train CLIP with few ID training data is prompt learning [60, 61], which trains the prompt's context words while keeping the entire pre-trained parameters fixed. CoOp [60] is the representative work of prompt learning, but we observe that CoOp does not still work well for OOD detection. While CoOp is designed to bring the given ID image and its corresponding class text embedding closer together, it also has the unintended effect of bringing the text embeddings closer to the backgrounds or ID-irrelevant objects in the ID image. Consequently, the text embeddings may contain information that is not relevant to the ID image, resulting in incorrectly high confidence scores for OOD images.
To apply prompt learning to OOD detection, we propose simple yet effective approach called Local regularized Context Optimization (LoCoOp). An overview of our LoCoOp approach shown in Fig. 1. We observe that the local features of CLIP contain many ID-irrelevant nuisances, such as backgrounds. LoCoOp treats such ID-irrelevant nuisances as OOD and learns to push them away from the ID class text embeddings. which removes unnecessary information from the text embeddings of ID classes and prevents the model from producing high ID confidence scores for the OOD features. To achieve this, LoCoOp first identifies the ID-irrelevant regions using the segmentation classification prediction results [59] for each local region. Next, it performs entropy maximization of the classification probabilities for the extracted regions, which ensures the features in ID ID-irrelevant regions are dissimilar to any ID text embedding.
Our proposed method, LoCoOp, offers two significant advantages. Firstly, it can perform OOD regularization at minimal cost. The standard approach to OOD regularization is OOD exposure [16, 54, 52], which utilizes external OOD samples for training. However, it can be quite cumbersome to collect lot OOD images. In addition, it can computationally expensive as the number of OOD images and training time are proportional. On the other hand, LoCoOp uses only local features of CLIP as OOD. Therefore, even when 100 OOD features are used per ID image, the training time is only about 1.4x longer and GPU memory consumption is only about 1.1 x higher than those of the baseline method without OOD regularization (i.e., CoOp). Secondly, LoCoOp is highly compatible with GL-MCM [33], a test-time OOD detection method that separates ID and OOD using both global and local features. Since LoCoOp learns local OOD features, it can accurately identify whether the given local region is ID OOD. This enables LoCoOp to improve the OOD detection performance with GL-MCM, significantly outperforming other methods.
Through experiments on the large-scaled ImageNet OOD benchmarks [18], we observe that our LoCoOp significantly outperforms existing zero-shot, few-shot, and fully supervised OOD detection methods. Notably, even in one-shot setting - just one label per class, LoCoOp outperforms existing

---

zero-shot and fully supervised detection methods, which is remarkable finding, The contributions of our study are summarized as follows:
We first tackle CLIP-based fho-shot OOD detection, where we utilize only a few labeled ID images to train CLIP for OOD detection.
We propose a novel prompt learning approach called LoCoOp, which leverages the portions of CLIP local features as OOD features for OOD regularization (see Fig. 1).
LoCoOp brings substantial improvements over prior zero-shot, fully supervised, and prompt learning methods on the large-scale ImageNet OOD benchmarks. Notably, LoCoOp outper- forms them with only one label per class (see Table 1).
2 Method
2.1 Problem statement
For OOD detection with pre-trained models [12, 30, 24], the ID classes refer to the classes used in the downstream classification task, which are different from those of the upstream pre-training. Accordingly, OOD classes are those that do not belong to any of the ID classes of the downstream task. An OOD detector can be viewed as binary classifier that identifies whether the image an ID image or an OOD image. Formally, we assume that we have an ID dataset D of (x"", y²") pairs where as denotes the input ID image, and 1 E Y == I,... M denotes the class label. Let Dout denote an OOD dataset of (æ:out, youu) pairs where out denotes the input OOD image, and your yout = M + 1, ,,.,, M + O denotes the OOD class label. Then, there are no overlaps of classes between ID and OOD. Formally, yout n yin
We study the scenario where the model is fine-tuned only on the training set Dis Dirain without any access to OOD data. The test contains Dig and Dout for evaluating OOD performance. Unlike existing studies using large number of ID training samples [46] or no ID training samples [30], we use only few samples (1, 8, and 16 samples per class) for Din Drrin
2.2 Reviews of CoOp
CoOp [60] is a pioneering method that utilizes vision-language pre-trained knowledge, such as CLIP [37], for downstream open- open-world visual recognition. While CLIP uses manually designed prompt templates, CoOp sets a portion context words the template as continuous learnable parameters that can be learned from few-shot data. Thus, the classification weights can be represented by the distance between the learned prompt and the visual feature.
Given an ID image _t global visual feature fod = f(xi") is obtained by the visual encoder f of CLIP. Then, the textual prompt can be formulated as f.m. {w 1. W2,,W,--. ...,,.... w. Cm ). where cm denotes the word embedding of the ID class name, has the same dimension as the original word embedding and N denotes the length of context words.
iN { are learnable vectors where each vector
With prompt t., as the input, the text encoder outputs the textual feature as gm g(tm). The final prediction probability is computed by the matching score as follows:
exp (sim (9.) /T)
ply m m "n
(I)
Σm= exp (sim (f"" 9m)) /T)
where sim (+-)) denotes cosine similarity, and 7 stands for the temperature of Softmax. The final loss Cooop is the cross-entree cross-entropy loss with Eq. (1) and its ground truth label y"".
2.3 Proposed approach
In this section, we present our LoCoOp approach for few-shot OOD detection. An overview of our proposed method shown in Fig. 1. Our methods consist of two components. The first to extract iD-irrelevant regions from CLIP local features, and the second is perform OOD regularization training with the extracted regions. It performs these two operations for each iteration.
In the following, we will first provide brief overview of how to obtain local features from CLIP. Then, we will describe our two key components: the method to extract object-ireelevan regions from ID images and the OOD regularization loss used during training.
3

---

Preliminary. To obtain local features of CLIP, we project the visual feature fA of each region i from the feature map the textual space [59, 42, 33] as follows:
f." in Projo (f' in
(2)
where v denotes the value projection and Proj.-> denotes the projection from the visual space into the textual space. These projections are inherent in CLIP and do not need to be trained. This local feature f. has rich local visual and textual alignment [59].
Extraction ID-irrelevant regions. We need select the region indices of -i-reeleeaan regions from set of all region indices I = {0, 1, 2, H x W - 1}. where H and W denote the height and width of the feature map. We handle this in a very simple and intuitive way. Similar to the segmentation task [59], we can obtain the classification prediction probabilities for each region i during training by computing the similarity between the image features f: in of each region i and the text features the ID classes. The formulation is as follows:
exp (sim (f.",gm)
9m) /T)
pcyy 10
M Σ
Dm'=1 exp
(sim (f, I.m)) /T))
When a region i of TM corresponds to part of an ID object, the ground truth class y" should be among the top-K predictions. In contrast, if region is ID-irrelevant, such as the background, yt should not appear among the top- top-K predictions because of the lack of semantic relationship between region and the ground truth label y'n. Based on this observation, we identify regions that do not include their ground truth class in the top- K predicted classes as ID ID-rrellvvan regions J. The formulation as follows:
J= I rank(p, = ""))""")) > K},
where rank(p, y'"| )) denotes the rank of the GT class y among all ID) classes. Note that set of regions of an image i is updated during training since pily = m anin) is updated.
This method may appear to be a parameter-depender method since it depends on the hyperparameter K. However, we argue that the optimal K is not difficult search for. This because, when applying OOD detection in the real world, it is assumed that the users understand what ID data is. For example, when ID is ImageNet- 1K [5], users know the number of fine-gra fineggraieed classes or semantic relationships between classes in advance. Using prior knowledge, such as the number of fine-grained classes, helps determine the value of K (up to top-K, the prediction may be wrong).
OOD regularization. With J/ (a set of region indices of ID-irrelevant regions), we perform OOD regularization, which removes unnecessary information from the text feature. The OOD features f, in of each OOD region J should be dissimilar to any ID text embedding. Therefore, use entropy maximization for regularization, which is used for detecting unknown samples during training [40, 55]. Entropy maximization makes the entropy of py py (y/(:"") larger and enables the
OOD image features f," be dissimilar to any ID text embedding. The loss function for this regularization is as follows:
Lood = (p;),,
(5)
where H(-) is the entropy function and denotes the classification prediction probabilities for region j E J.
Final Objective. The final objective is as follows:
C Leoop + XLood:
(6)
where Cooop is the loss for CoOp [60] and À is hyperparameter. Here, Ccoop is applied for the entire image.
2.4 Test-time OOD detection
In testing, we use MCM score [30] and GL-MCM score [33]. With these functions, we classify test-time images into x and x out The details are as follows:

---

Locoop (QUES)
LeCtOp GE (ours) Dtlpp, MCM
93
FPR
40
CoOp OL
AUROC
35
LeCoOp MCM (ours)
91
1:00p
CoOg_MCM
Chog Gl.
90
1
2 4
8
2
8 # of labels
16
# of labels
(a) FPR95
(b) AUROC
Figure 2: Few-shot OOD detection results with different numbers of ID labeled samples. We report average FPR95 and AUROC scores on four OOD datasets in Table 1. The lower value is better for FPR95, and the larger value is better for AUROC. We find that in all settings, our proposed LoCoOp with GL-MCM (red one) outperforms CoOp by large margin.
MCM score. The MCM score [30] utilizes the softmax score of global image features and text features. Formally, SMOM = maxm Eni Emn'= (sp(sim((fsemm// where r = 1. The rationale that, for ID data, will matched to one of the prototype vectors with a high score and vice versa.
exp(simmffgg.. (m9)//
GL-MCM score. The GL-MCM score [33] utilizes the sofumax score of both global and local image features and text features. The score function SCL-MCM = SMCM pxppsim((f.gmmmll
max M exp(alm[]; (.n)) When OOD objects appear the given ID image, the match- ing score with global features might be incorrectly low. Utilizing the matching score with local features can compensate for the low global matching score and produce correct ID confidence.
Experiment
3.1 Experimental Detail
Datasets. We use the ImageNet-IK dataset [5] as the ID data. Therefore, A is set to 1,000. For OOD datasets, we adopt the same ones as in [18], including subsets of iNaturalist [47], SUN [51], Places [58], and TEXTURE [4]. For the few-shot training, we follow the few-shot evaluation protocol adopted in CLIP [37] and CoOp [60]. using 1, 2, 4, 8, and 16 shots for training, respectively, and deploying models in the full test sets. The average results over three runs are reported for comparison.
Setup. Following existing studies [46, 30, 33], we use ViT-B/16 [6] as a backbone. Specifically, we use the publicly available CLIP-ViT-B/16 models (https: //t::://gttii thub. com/ Copenai/CLin The resolution of CLIP's feature map 14 14 for CLIP-VIT-B/16. For K, we use 200 in all experiments. For X, we use 0,25 in all experiments. The sensitivities of K and À are described in the Analysis section for K and in the supplementary material for X. Other hyperparameters (e.g., training epoch=50, learning rate=0.002, batch size=32, and token lengths N=16) are the same as those of CoOp [60]. We use a single Nvidia A100 GPU for all experiments, Since LoCoOp takes approximately 1.4 times longer than CoOp, additional experiment was conducted using 70 epochs for CoOp, but the results did not differ from those of 50 epochs. Therefore, we report all the results with 50 epochs.
Comparison Methods. To evaluate the effectiveness of our LoCoOp, we compare it with zero-shot detection methods, fully-supervised detection methods, and the baseline prompt learning method. For zero-shot OOD detection methods, we use MCM [30] and GL-MCM [33], which are state-of-the zero-shot detection methods. For fully-supervised detection methods, we follow the comparison methods [28, 50, 45, 46] in NPOS [46]. All the other baseline methods are fine-tuned using the same pre-trained model weights (i.e., CLIP-B/16). For NPOS, the MCM score function is used in the original paper. For fair comparison, we reproduce NPOS and also report the results with the GL-MCM score. For prompt learning methods, we use CoOp [60], which is the representative work and the baseline for our LoCoOp. As for the comparison with CoCoOp [61], which is another representative work, we show the results in Analysis Section 4.
5

---

Table 1: Comparison results ImtgeNet OOD benchmarks. We use ImageNet- ID. We CLIP-B/16 as a backbone. Bold values represent the highest performance. i cited from [46]. is reproduction. We find that LoCoOp with GL-MCM (LoCoOpul) is the most effective method.
NNaturalist
SUN
Places
Texture
Average
Method
FPR95
AUROCT
FPR95
AUROCT
FPR95, AUROC
FPR95
AUROCT
[PR95]. AUROCT
Zero-shot
MCM [30]*
30.94
94.61
37.67
92.56
44.76
89.76
57.91
86.10
42.82
90.76
GL-MCM [33]*
15.18
96.71
30.42
93.09
38.85
89.90
57.93
83.63
35.47
90.83
Fine-tuned
ODIN [28]*
30.22
94.65
54.04
87.17
55.06
85.54
51.67
87.85
47.75
88.80
ViM [50]
32.19
93.16
54.01
87.19
60.67
83.75
53.94
87.18
50.20
87.82
KNN [45]*
29.17
94.52
35.62
92.67
39.61
91.02
64.35
85.67
42.19
90.97
NPOSMcM [46]*
16.58
96.19
43.77
90.44
45.27
89.44
46.12
88.80
37.93
91.22
NPOSMcM [46]]
19.59
95.68
48.26
89.70
49.82
88.77
51.12
87.58
42.20
90.43
NPOS
18.70
95.36
38.99
90.33
41.86
89.36
47.89
86.44
36.86
90.37
Prompt learning
one-shot (one label per class)
CоOPиCM
43.38
91.26
38.53
91.95
46.68
89.09
50.64
87.83
44.81
90.03
CoOpan
21.30
95.27
31.66
92.16
40.44
89.31
52.93
84.25
36.58
90.25
LoCoOрMсM (ours)
38.49
92.49
33.27
93.67
39.23
91.07
49.25
89.13
40.17
91.53
LoCoOper. (ours)
24.61
94.89
25.62
94.59
34.00
92.12
49.86
87.49
33.52
92.14
16-shot (16 labels per class)
CоOPисм
28.00
94.43
36.95
92.29
43.03
89.74
39.33
91.24
36.83
91.93
CoOPaL
14.60
96.62
28.48
92.65
36.49
89.98
43.13
88.03
30.67
91.82
LoCoOрMeM (ours)
23.06
95.45
32.70
93.35
39.92
90.64
40.23
91.32
33.98
92.69
LoCoOpar (ours)
16.05
96.86
23.44
95.07
32.87
91.98
42.28
90.19
28.66
93.52
40
Lotiotyy MCN
93.5
LoCuOp
AUROC
93.0
FPR
35-
92.5
92.0
30
LoCoOp, MCM
91.5
LatnoOptii
200
400
600
800
1000
200
400
600
800
1000
The value of
The value of K
(a) FPR95
(b) AUROC
Figure 3: Ablation studies on K. We report average FPR95 and AUROC scores on four OOD datasets in Table 1. The performance is not very sensitive to K values except for extreme values such and 1,000.
Evaluation Metrics. For evaluation, we use the following metrics: (1) the false positive rate of OOD images when the true positive rate of in-distribution images 95% (FPR95), (2) the area under the receiver operating characteristic curve (AUROC).
3.2
Main results
The OOD detection performances are summarized in Table 1. We show that LoCoOp with GL- MCM (LoCoOpau) can achieve superior OOD detection performance, outperforming the competitive methods by a large margin. In particular, with a 16-shot setting, LoCoOp with GL-MCM reached an AUROC score of 93.52%, while the other methods failed to reach an AUROC score of 92%.
In addition, we can see, even in a one-shot setting, LoCoOp with GL-MCM achieved average FPR95 and AUROC scores of 33.52 and 14, respectively, which outperforms the zero-shot and fine-tuned methods. Considering that LoCoOp utilizes only one training sample per class while fine-tuned methods use 1,000 training images per class, these results remarkable.

---

Table 2: Comparison results with CoCoOp [61]. We report average FPR and AUROC scores on four OOD datasets in Table 1.
Table 3: Comparison in ID accuracy on ImageNet-Il IK validation data.
Method
Top-1 1 Accuracy
Average
CoOp
72.1
Method
Infer time.
FPR95
AUROCT
LoCoOp
71.7
CоCoOPpcM [61]
149 ms
35.53
91.99
LoCoOPMcCM
2.59 ms
33.98
92.69
LoCoOPar
5.97 ms
28.66
93.52
In Fig. 2, we show the results with different numbers of ID labeled samples for CoOp and LoCoOp, where the green line denotes our LoCoOp with MCM, the red one denotes LoCoOp with GL- MCM, the blue one is CoOp with MCM, and the orange line denotes CoOp with GL-MCM. The detailed accuracy can be found in the supplementary materials. We observed that GL-MCM boosts the performance of LoCoOp much more than CoOp in all settings.
3.3 Ablation studies on hyperparameter
We conduct ablation studies on K in Eq. (4). K is an important parameter pick up OOD regions from the local features. We experimented with a 16-shot setup using K-= (all the local regions are used for OOD), 100, 200, 300, 400, 500, 900, 980, and 1,000 (same as CoOp). In Fig. 3, we summarize the results of ablation studies. First, let look at the leftmost point of the graph, K=0,, i.e. the result when all the local regions are treated as OOD, We can see that K=0 degrades the detection performance in all settings. This is because it treats all object-relevant regions as OOD, Then, going around K=100, we can see that the performance improves dramatically because the number of such false negative OOD features is decreasing. LoCoOp maintains high performance after K=200.. In particular, even at K=980, large performance gap exists with K=1,000 (CoOp). At K=980, the number of OOD regions per image is about one, indicating that LoCoOp is effective even when only a few local OOD features are used for training. In general, the value of K is not sensitive the performance and is not difficult to select long as it is not set to extreme values such as 0 or 1,000.
4 Analysis
Comparison with CoCoOp. We compare our LoCoOp with CoCoOp [61]. CoCoOp is also representative work for prompt learning, which generates an input-conditional text token for each image. For comparison, we add per-image inference time (in milliseconds, averaged across test images) for evaluation metrics [45]. This because OOD detection methods are applied before test-time input comes to the close-set classifier, so the short inference time is important. We use single Nvidia A100 GPU for the measurements. We show the 16-shot results of CoCoOp and LoCoOp in Table 2. This result demonstrates that our LoCoOp outperforms CoCoOp in performance and inference speed. Especially, CoCoOp takes much longer inference time than our LoCoOp. That is because it requires an independent forward pass of instance-specific prompts for each image through the text encoder. When inferring ImageNet OOD benchmarks (total number of images is 85,640), CoCoOp takes 3.55 hours, while LoCoOp with MCM takes only 3.7 minutes.
ID accuracy of LoCoOp. We examine the ID accuracy of LoCoOp. OOD detection is preprocessing before input data comes into the ID classifier, and its purpose is to perform binary classification of ID or OOD [15]. However, is also important to investigate the relationship between close-set ID accuracy and OOD detection performance [49]. In Table 3, we show the 16-shot ID accuracy of CoOp and LoCoOp on the ImageNet- IK validation set. This result shows that LoCoOp slightly inferior to CoOp in ID accuracy. We hypothesize that excluding OOD nuisances that are correlated with ID objects will degrade the ID accuracy. For example, in some images of dogs, the presence of green grass in the background may help identify the image as dog. Therefore, learning to remove the background information could make difficult to rely on such background information to determine that the image is dog. However, this study demonstrates that excluding such backgrounds improves
7

---

Table 4:
Comparison results with CLIP-ResNet-50. (LoCoOpaL) is effective with ResNet-50.
We find that LoCoOp with GL-MCM
iNaturalist
SUN
Places
Texture
Average
Method
FPR95,
AUROCT
FPR95]
AUROCT
FPR95)
AUROCT
FPR95]
AUROCT
FPR954
AUROC1
MCM [30]
31.98
93.86
46.09
90.75
60.56
85.67
60.00
85.72
49.66
89.00
GL-MCM [33]
37.20
90.98
38.74
91.21
49.49
86.40
43.46
88.14
42.22
89.18
CoOPaL
33.68
92.59
40.0
90.77
48.14
86.8
31.27
91.69
38.27
90.46
LoCoOpen. (ours)
22.81
95.38
36.44
92.22
44.10
88.3
34.47
91.89
34.45
91.95
Table 5: Comparison with other methods for ID-irrelevant regions extraction.
iNaturalist
SUN
Places
Texture
Average
Method
FPR95I
AUROCT
FPR95
AUROC1
FPR95
AUROCT
FPR95
AUROC
FPR95L
AUROCT
Ent.
25.05
94.28
31.92
92.97
39.85
89.94
48.71
87.52
36.38
91.18
Prob.
16.19
96.55
23.66
94.76
32.62
91.83
42.12
89,96
28.65
93.27
Rank (ours)
16.05
96.86
23.44
95.07
32.87
91.98
42.28
90.19
28.66
93.52
the OOD detection performance. This study exhibits that a strong ID classifier is not always a robust OOD detector for prompt learning.
Effectiveness with CNN architectures. We examine the effectiveness of LoCoOp with CNN architectures. In the main experiments, we use Vision Transformer architecture as a backbone following existing work [46, 30]. However, for the fields of OOD detection, CNN-based models also have been widely explored [17, 43). Following this practice, we use ResNet-50 as a CNN backhone and examine the effectiveness of LoCoOp. Table 4 shows the 16-shot results with ResNet-50. We find that similar to the results with ViT, our LoCoOp is effective for ResNet-50.
Comparison with other methods for ID-irrelevant region extraction. We explore the effectiveness of several methods for ID-irrelevant region extraction. Although we use a Top- rank-based other approaches, entropy-based thresholding and probability-bate thresholding, approaches are also candidates for the methods. Entropy-based thresholding methods use the threshold of the entropy of Pt. whereas probability-based thresholding methods use the threshold of the value of (. = (x). For entropy-based thresholding methods, we extract regions where the entropy of p; is less than log because l is the half value of the entropy of M- dimmensional probabilities, which is following existing studies [40, 55]. For probability-based thresholding methods, we extract regions where (ı = yid (x'") is less than 1/M simply because the ID dataset has M classes. We summarize the 16-shot OOD detection results in Table 5. For the entropy- based method, the performance is poor since it is challenging to determine the appropriate threshold. On the other hand, for the probability- rank-based method since the user's 's perspective, the rank easier to interpret and handle than the probability value. Therefore, use rank-based thresholding in this paper.
thresholding approach,
2
method, the performance is comparable to our determine the threshold intuitively. However, we consider that, from
Visualization of extracted OOD regions. In Fig.4, we show visualization results of extracted OOD regions. The performance of OOD extraction is key to our method. From this visualization result, we find that our Rank-based approach can accurately identify OOD regions.
Theoretical background
The effectiveness of separating foregrounds (ID objects) and backgrounds is theoretically guaranteed by existing research 39]. Ming et al.[31] states that when the classifier only uses foreground features, the optimal decision boundary can obtained. With conventional classifiers, it was difficult for the classifiers to use only foreground features. Contrary to this, CLIP has rich local visual-text alignments and can identify the foregrounds and backgrounds. Based on the theoretical analysis in [31], LoCoOp achieves high OOD detection performance.
8

---

tench (fish)
ID: scarf
French horn
ID: Appenzeller Sennenhund (dog)
ID: military aircraft
ID: guinea pig
Figure 4: Visualization of extracted OOD regions. We find that our approach can correctly extract ID-irrelevant regions.
Limitations
Application to other light-weight tuning methods. In this study, we focus on text prompt learning methods [60, 61] because they the most representative methods for tuning CLIP. These days, other weight tuning methods such as Tip-Adapter, which learns a lightweight residual feature adapter [57] and visual prompt methods [21, 2], which learn a visual prompt added the input image [2] or inside the layer [21]. have been proposed for image classifications. It is our future work apply our LoCoOp these methods [57, 21, 2].
Application to the model without the rich local visual-text alignment. This approach requires models with strong local visual-text aligmment capabilities, such the image encoder in CLIP. Consequently, applying this approach to models lacking such capabilities can prove challenging. Nonetheless, given the numerous methods currently emerging based CLIP, our research is influen- tial in this research field.
Extending to other visual recognition tasks. In this study, we focus only classification tasks. However, prompt learning widely used other tasks, such as object detection [10] and segmenta- tion [10]. We will solve other tasks important future work. However, existing studies about OOD detection do not deal with both classification and other tasks simultaneously [8, 26, 7], and most studies in OOD detection address classification tasks [53]. Therefore, we believe that our findings in the classification task are influential.
7 Related Work
Out-of- dinfriiittiittion detection. Various approaches have been developed address OOD detection in computer vision [14, 23, 34-36, 15, 17, 19, 28, 45, 29, 26, 50, 53, 56, 32]. One of the common OOD detection approaches is devise score functions such probability-basee
method [15, 17, 19, 28, 44, 45], logit-based method [29, 17, 44), and feature-based method [26, 50]. Another approach is to leverage OOD data for training-time regularization [16, 54, 9, 52]. These OOD detection methods assume the use of backbones trained with single-modal data. In recent years, much attention has been paid how multimodal pre-trained models, such as CLIP, can be used for out-of- ont-o--distribution detection. Existing CLIP-based OOD detection methods have explored the zero- shot methods [12, 11, 30, 33] and fine-tuned method [46]. However, these methods have their own limitations regarding performance and training efficiency. To bridge the gap between fully supervised and zero-shot methods, developing few-shot OOD detection method that utilizes a few ID training images for OOD detection is essential. Before CLIP emerged, Jeong et al. [20] addressed the few-shot OOD detection, using only a few training data OOD detection, but their experiments have only been done small data sets (e.g., Omniglot [25], CIFAR-FS [3]), showing the limitations of the performances of few-shot learning methods without CLIP. By leveraging the powerful representations of CLIP, shot OOD detection could provide more efficient and effective solution.

---

Prompting for foundation models. Prompting is heuristic way to directly apply foundation models to downstream tasks in a zero-shot manner. Prompt works as a part of the text input that instructs the model to perform accordingly on a specific task. However, such zero-shot generalization relies heavily on a well-designed prompt. Prompt tuning [27] proposes to learn the prompt from downstream data in the continual input embedding space, which is a parameter-effiiient way of fine-tuning foundation models. Although prompt tuning was initially developed for language models, it has been applied to other domains, including vision-language models [60, 61, 42, 38, 13]. CoOp [60], which applies prompt tuning to CLIP, is a representative work that effectively improves CLIP's performance on corresponding downstream tasks. However, existing prompt learning methods focus improving only close-set ID classification accuracy, and their performance for OOD detection remains unclear. In this work, propose novel method for prompt learning for OOD detection.
Local features of CLIP. Recent work [59] revealed that CLIP has local visual features aligned with textual concepts. For classification tasks, global features are used, which are created by pooling the feature map with a multi-headed self-attention layer [48]. However, these features are known to lose spatial information [41]. so they are not suitable for tasks that require spatial information, such as semantic segmentation. To adopt CLIP to segmentation, Zhou et al. [59] proposed the CLIP's local visual features aligned with textual concepts, which can be obtained by projecting the value features of the last attention layer into the textual space. Existing studies used CLIP's local features for semantic segmentation [59, 62, 22]. multi-label classification [42] and ID detection [33], which need to detect local objects in an image. Unlike previous studies, we use local features to pick up ID-irrelevant parts (e.g., backgrounds), which can be regarded OOD features for OOD detection.
8 Conclusion
We present a novel vision-language prompt learning approach for few-shot OOD detection. To solve this problem, we present LoCoOp, which performs OOD regularization during training using the portions of CLIP local features as OOD images. CLIP's local features have a lot of ID-irrelevant information, and by learning to push them away from the ID) class text embeddings, we can remove the nuisances in the ID class text embeddings, leading to better separation of ID and OOD. Experiments on the large-scale ImageNet OOD detection benchmark demonstrate the advantages of our LoCoOp over zero-shot, fully supervised detection methods and existing prompt learning methods.
Broader Impact. This work proposed a prompt learning approach for few-shot OOD detection, improving the performance and efficiency of existing detection methods. Our work reduces the data-gathering effort for data-ex expensive applications, so it makes the technology more accessible to individuals and institutions that lack abundant resources. However, it is important to exercise caution in sensitive applications as our approach may make these systems vulnerable to misuse by individuals or organizations seeking to engage fraudulent or criminal activities. Therefore, human oversight necessary to ensure that our approach is used ethically and not relied upon solely for making critical decisions.
Acknowledgement
This work was partially supported by JST JPMJCR22U4, CSTI-SIP, and ISPS KAKENHI 21H03460.
References
[1] Udit Arora, William Huang, and He He. Types of out-of-distribution texts and how to detect them. In EMLP, 2021.
[2] Hyojin Bahng, Ali Jahanian, Swami Sankaranarayanan, and Phillip Isola. Exploring visual prompts for adapting large-scale models. arXiv preprint arXiv:2203. /7274, 2022.
[3] Luca Bertinetto, Joao Henriques, Philip Torr, and Andrea Vedaldi. Meta-learning with differentiable closed-form solvers. In ICLR, 2019.
[4] Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In CVPR, 2014.
10

---

[5] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. CVPR, 2009.
[6] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenbom,, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In ICIR, 2021.
[7] Xuefeng Du, Gabriel Gozum, Yifei Ming, and Yixuan Li. Siren: Shaping representations for detecting out-of-distribution objects. In NeurIPS, 2022.
[8] Xuefeng Du, Xin Wang, Gabriel Gozum, and Yixuan Li. Unknown-aware object detection: Learning what you don't It know from videos in the wild. In CVPR, 2022.
[9] Xuefeng Du, Zhaoning Wang, Mu Cai, and Yixuan Li. Vos: Learning what you don't know by virtual outlier synthesis. In ICLR, 2022.
[10] Yu Du, Fangyun Wei, Zihe Zhang, Miaojing Shi, Yue Gao, and Guoqi Li. Learning to prompt for open-vocabulary object detection with vision-language model. In CVPR, 2022.
[11] Sepidch Esmacilpour, Bing Liu, Eric Robertson, and Lei Shu. Zero-shot out-of-distribution detection based on the pretrained model clip. In AAAI, 2022.
[12] Stanislav Fort, Jie Ren, and Balaji Lakshminarayanan. Exploring the limits of -iitriiiittiittioonn detection. In NewrIPS, 2021.
[13] Chunjiang Ge, Rui Huang, Mixue Xie, Zihang Lai, Shiji Song, Shuang Li, and Gao Huang. Domain adaptation via prompt learning. arXiv preprint Siti::20020068777 2022.
[14] Yuan Ge, Sergey Demyanov, Zetao Chen, and Rahil Garnavi. Generative openmax for multi-class open set classification. arXiv preprint arXiv: 1707.0777. 2017.
[15] Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and ouito--distr examples neural networks. In ICLR, 2017.
[16] Dan Hendrycks, Mantas Mazeika, and Thomas Dietterich. Deep anomaly detection with outlier exposure. In ICLR, 2019.
[17] Dan Hendrycks, Steven Basart, Mantas Mazeika, Mohammadreza Mostajabi, Jacob Steinhardt, and Dawn Song. Scaling out-of-distribution detection for real-world settings. In ICML, 2022.
[18] Rui Huang and Yixuan Li. Mos: Towards scaling dustt--dist detection for large semantic space. In CVPR, 2021.
[19] Rui Huang, Andrew Geng, and Yixuan Li. On the importance of gradients for detecting distributional shifts in the wild. In NeurlPS, 2021.
[20] Taewon Jeong and Heeyoung Kim. maml:: Meta-learning for few-shot out-of-distribution detection and classification. In NeurlPS, 2020.
[21] Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In ECCV, 2022.
[22] Kwanyoung Kim, Yujin Oh, and Jong Chul Ye. Zegot: Zero-shot segmentation through optimal transport of text prompts. arXiv preprint 231.:2301112177, 2023.
[23] Polina Kirichenko, Pavel Izmailov, and Andrew G Wilson. Why normalizing flows fail to detect out-of-distributio data. NeurIPS, 2020.
[24] Rajat Koner, Poulami Sinhamahapatra, Karsten Roscher, Stephan Günnemann, and Volker Tresp. Oodformer: Out-of-distributioo detection transformer. In BMVC, 2021.
[25] Brenden M Lake, Ruslan Salakhutdinov, and Joshua B Tenenbaum. Human-level concept learning through probabilistic program induction. Science, 1332-166 2015.
11

---

[26] Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for detecting
samples and adversarial attacks. In NeurlPS, 2018.
[27] Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In ACL, 2021.
[28] Shiyu Liang. Yixuan Li, and Rayadurgam Srikant. Enhancing the reliability of out-of- distribution image detection in neural networks. In ICLR, 2018.
[29] Weitang Liu, Xiaoyun Wang, John Owens, and Yixuan Li. Energy- baaegy-based out-of-distribution detection. In NeurlPS, 2020.
[30] Yifei Ming, Ziyang Cai, Jiuxiang Gu, Yiyou Sun, Wei Li, and Yixuan Li. Delving into out-of-distribution detection with vision-language representations. In NeurIPS, 2022.
[31] Yifei Ming, Hang Yin, and Yixuan Li. On the impact of spurious correlation for out-of- distribution detection. In AAAI, 2022.
[32] Atsuyuki Miyai, Qing Yu, Go Irie, and Kiyoharu Aizawa. Can pre-trained networks detect familiar out-of- distrribbt data? arXiv preprint arXiv:2310.008/ 2023.
[33] Atsuyuki Miyai, Qing Yu, Go Irie, and Kiyoharu Aizawa. Zero-shot in-distribution de- tection in multi-object settings using vision-language foundation models. arXiv preprint arXiv:2304.05521, 2023.
[34] Eric Nalisnick, Akihiro Matsukawa, Yee Whye Tch, Dilan Gorur, and Balaji Lakshminarayanan. Do deep generative models know what they don't 't know? In ICLR, 2019.
[35] Lawrence Neal, Matthew Olson, Xiaoli Fern, Weng- Keen Wong, and Fuxin Li. Open set learning with counterfactual images. In ECCV, 2018.
[36] Poojan Oza and Vishal M Patel. C2ac: Class conditioned auto-encoder for open-set recognition. In CVPR, 2019.
[37] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, 2021.
[38] Yongming Rao, Wenliang Zhao, Guangyi Chen, Yansong Tang, Zheng Zhu, Guan Huang, Jie Zhou, and Jiwen Lu. Denseclip: Language-guided dense prediction with context-aware prompting. In CVPR, 2022.
[39] Jie Ren, Peter J Liu, Emily Fertig, Jasper Snoek, Ryan Poplin, Mark Depristo, Joshua Dillon, and Balaji Lakshminarayanan. Likelihood ratios for -usto--distt detection. In NeurlPS, 2019.
[40] Kuniaki Saito, Donghyun Kim, Stan Sclaroff, and Kate Saenko. Universal domain adaptation through self supervision. In NeurIPS, 2020.
[41] Sanjay Subramanian, Will Merrill, Trevor Darrell, Matt Gardner, Sameer Singh, and Anna Rohrbach. Reclip: A strong zero-shot baseline for referring expression comprehension. In ACL, 2022.
[42] Ximeng Sun, Ping Hu, and Kate Saenko. Dualcoop: Fast adaptation to multi-label recognition with limited annotations. In NeurlPS, 2022.
[43] Yiyou Sun and Yixuan Li. Dice: Leveraging sparsification for out-of-distribution detection. In ECCV, 2022.
[44] Yiyou Sun, Chuan Guo, and Yixuan Li. React: Out-of-distribution detection with rectified activations. In NeurIPS, 2021.
[45] Yiyou Sun, Yifei Ming, Xiaojin Zhu, and Yixuan Li. Out-of-distribution detection with deep nearest neighbors. In ICML, 2022.
12

---

[46] Leitian Tao, Xuefeng Du, Xiaojin Zhu, and Yixuan Li. Non-parametric outlier synthesis. In ICLR, 2023.
[47] Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. The inaturalist species classification and detection dataset. In CVPR, 2018.
[48] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurlPS, 2017.
[49] Sagar Vaze, Kai Han, Andrea Vedaldi, and Andrew Zisserman. Open-set recognition: A good closed-set classifier is all you need. In ICLR, 2022.
[50] Haoqi Wang. Zhizhong Li, Litong Feng, and Wayne Zhang. Vim: Out-of-distribttinn with virtual-logit matching. In CVPR, 2022.
[51] Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey zoo. In CVPR, 2010.
[52] Jingkang Yang. Haoqi Wang, Litong Feng, Xiaopeng Yan, Huabin Zheng, Wayne Zhang, and Ziwei Liu. Semantically coherent out-of-distribune detection. In ICCV, 2021.
[53] Jingkang Yang, Pengyun Wang, Dejian Zou, Zitang Zhou, Kunyuan Ding, Wenxuan Peng, Haoqi Wang, Guangyao Chen, Bo Li, Yiyou Sun, Xuefeng Du, Kaiyang Zhou, Wayne Zhang, Dan Hendrycks, Yixuan Li, and Ziwei Liu. Openood: Benchmarking generalized out-of-distribution detection. NeurlPS Datasets and Benchmarks Track, 2022.
[54] Qing Yu and Kiyoharu Aizawa. Unsupervised out-of-distribution detection by maximum classifier discrepancy. In ICCV, 2019.
[55] Qing Yu, Atsushi Hashimoto, and Yoshitaka Ushiku. Divergence optimization for noisy universal domain adaptation. In CVPR, 2021.
[56] Jingyang Zhang, Jingkang Yang, Pengyun Wang, Haoqi Wang, Yueqian Lin, Haoran Zhang, Yiyou Sun, Xuefeng Du, Kaiyang Zhou, Wayne Zhang, et al. Openood v1. 5: Enhanced benchmark for out-of-distribution detection. arXiv preprint arXiv:2306.0930/ 2023.
[57] Renrui Zhang, Rongyao Fang, Wei Zhang, Peng Gao, Kunchang Li, Jifeng Dai, Yu Qiao, and Hongsheng Li. Tip-adapter: Training-free free clip-adapter for better vision language modeling. In ECCV, 2022.
[58] Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. TPAMI, 40(6): 1452-1464, 2017.
[59] Chong Zhou, Chen Change Loy, and Bo Dai. Extract free dense labels from clip. In ECCV, 2022.
[60] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning prompt for vision-language models. IJCV, 2022.
[61] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. In CVPR, 2022.
[62] Ziqin Zhou, Bowen Zhang, Yinjie Lei, Lingqiao Liu, and Yifan Liu. Zegelip: Towards adapting clip for zero-shot semantic segmentation. In CVPR, 2023.
13


Reviews:
[{"review_id": "tUqoppzN5i", "reviewer": "Reviewer_Lcii", "summary": "This paper proposes a vision-language prompt learning method named local regularized context optimization (LoCoOp) for few-shot out-of-distribution detection. The proposed LoCoOp method performs OOD regularization  that uses the portions of CLIP local features as OOD features during training. The experimental results show the effectiveness of the proposed LoCoOp method over zero-shot and fully supervised methods.", "questions": "- Will the proposed method also improve the results in fully supervised setting? And what are the results?\n- Will the region segmentation results be a key obstacle to the proposed method?\n- What is the reason that the performance of one-shot LoCoOp method is even worse than the zero-shot CL-MCM method (and only comparable for 16-shot LoCoOp) on iNaturalist dataset?", "limitations": "The authors have addressed the limitations.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The addressed problem of few-shot out-of-distribution detection is interesting and practical.\n- The proposed CLIP based few-shot OOD detection method with local regularized context optimization is well motivated and achieves good results.", "weaknesses": "- The proposed local regularized context optimization method is not specific to the few-shot setting, and can actually also be applied to fully supervised OOD detection setting, where the ID-irrelevant nuisances could also be learned. Then will the proposed method also improve the results in fully supervised setting? And what are the results?\n- Similar to the CLIP based segmentation task in [55], the ID-irrelevant regions are predicted based on CLIP prediction results. However, the segmentation results may not be good, will that be a key obstacle to the proposed method?\n- The training process of the proposed method is a little bit confusing. Does the L_coop loss apply to the whole image or only the ID-relevant regions?\n- In Table 1, it shows that the performance of one-shot LoCoOp method is even worse than the zero-shot CL-MCM method (and only comparable for 16-shot LoCoOp) on iNaturalist dataset. What is the reason?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the valuable comments and respond to them as follows. We will add suggested experiments and explanations in the updated manuscript.\n\n> Q1.  The effectiveness of LoCoOp in a fully supervised setting.\n    \n    \n\nA1. As the reviewer pointed out, our LoCoOp can be applied to a fully supervised setting (i.e., using all training data). However, Figure 2 shows that the improvement in performance will become slower after 16-shot training. This is due to the fact that even if many images are used for training, CoOp and LoCoOp only train text prompts that have only a few parameters, and a small number of samples is sufficient to reach the upper score.\n\n> Q2. Is the segmentation result a key obstacle to the proposed method?\n    \n    \n\nA2.  That is correct, and the accuracy of the segmentation results is a major factor.\n\nIn our setting, unlike typical segmentation tasks, correctly guessing the segmentation label is not necessary. Hence, to account for potential errors in the segmentation results, we introduce Rank in Eq.(4) to allow segmentation errors up to top-K.\n\nWe also attach some visualization results in the global response to show how well the OOD regions are extracted. This shows that our method can correctly extract OOD regions.\n\n> Q3. Does the L_coop loss apply to the whole image or only the ID-relevant regions?\n    \n    \n\nA3. Thanks for pointing this out. L_coop is the loss applied for the entire image.  We will add an explanation in the final version.\n\n> Q4. The reason for low performance on the iNaturalist dataset\n\nA4. The reason for this may be difficult to analyze. In the case of ResNet in Table 4, the detection performance of LoCoOp with GL-MCM is superior to that of zero-shot GL-MCM on iNaturalist, so it cannot be said that LoCoOp's training strategy is the main cause of the problem.\n\nBesides, comparisons of different backbone networks for OOD detection often have been explored [1, 2] but no conclusions have yet been drawn because of the different performance on each dataset. \n\n[1] Fort et al., Exploring the limits of out-of-distribution detection, NeurIPS 2021.  \n[2] Hendrycks et al., Scaling out-of-distribution detection for real-world settings, ICML 2022."}, {"review_id": "B6iITwlBQg", "reviewer": "Reviewer_uDwi", "summary": "The task of this article is to use the CLIP model for image classification, and the target scene is few-shot data and out-of-distribution (OoD). The author confirms the final OoD region by querying the ID-independent region in the image, and further optimizes the model through this region. Remarkably, the authors validated their method on the large-scale dataset Image-Net 1K. As far as I know the task is a new one, and its application makes sense. ", "questions": "My main questions are in the weaknesses section, and I would like to improve my score if the author can address my concerns convincingly.\n\nIn addition, I still want to know how long (how many hours or how many days) does this method take to train the ImageNet dataset on an A100GPU?", "limitations": "Yes, the authors have listed the limitation in the article.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This paper proposed a new and meaningful task.\n- The performance of the method is fine.\n- The author completed method verification on a very challenging dataset (ImageNet-1K full dataset).\n- It's an interesting idea to remove regions not related to IDs.\n- The author provides the source code to ensure the reproducibility of the article.", "weaknesses": "The idea of this article is very interesting and theoretically feasible. However, I have the following concerns:\n\n- My biggest concern is the accuracy of locating regions not related to IDs. Because the author did not provide an experimental indicator of the accuracy of the ID-independent region positioning, it is only shown from the results of training and detecting OoD. My concerns come from another article I've read that points to the limitations of CLIP's current interpretation of location regions [1]. It is a pity not to see related visualization results in the article and supplementary material. If the author can provide some indicators to prove that his method can well locate regions that have nothing to do with ID, I think it will be more convincing.\n- In addition to OoD detection, the model should have Zero-Shot capability, should the author also compare the zero-shot performance with CoOp or CoCoOp methods?\n\n[1] Li, Yi, et al. \"CLIP surgery for better explainability with enhancement in open-vocabulary tasks.\" *arXiv preprint arXiv:2304.05653* (2023).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the valuable comments and respond to them as follows. We will add suggested experiments and explanations in the updated manuscript.\n\n> Q1. The accuracy of locating regions not related to IDs.\n    \nA1. Thanks for the question.  That is correct, and the accuracy of the segmentation results is a major factor. However, unlike the usual segmentation task, in this setting, it is not necessary to correctly guess the segmentation label. Therefore, to compensate for errors in the segmentation results, we introduce ranks into equation (4) to allow for segmentation errors up to top-K predictions.\n\nAs for the evaluation metrics, the ImageNet dataset does not have segmentation masks, so quantitative evaluation is difficult. Therefore, we have attached visualization results for some samples in the global response to demonstrate the effectiveness of our method. This shows that our method can correctly extract OOD regions.\n\n> Q2. Zero-shot generalization ID accuracy\n\nA2. We report the zero-shot generalization ID accuracy (not OOD detection performance) with CoOp and LoCoOp on Flower102, Food101, and Oxford-Pets datasets. Note that LoCoOp aims to improve the performance of OOD detection on specific ID datasets, so the zero-shot generalization ID accuracy is not objective.\n| Method      | Flower | Food  | Pets  |\n|--|--------|-------|-------|\n| CoOp  | 65.63  | 84.00  | 87.53 |\n| LoCoOp| 61.27  | 83.13 | 88.17 |\n\n\n\nFor the Flower dataset, the ID accuracy of LoCoOp is considerably lower than that of CoOp. This is because the information similar to flowers (e.g., grass) might be removed as backgrounds during training on ImageNet. On the other hand, the ID accuracy of LoCoOp is higher than that of CoOp for the Pets dataset. The background in ImageNet is similar to that of Pets. In addition, many of the images in Pets have similar backgrounds. Therefore, LoCoOp can remove unnecessary information (e.g., common backgrounds) to identify fine-grained kinds of dogs, so the ID accuracy is improved.\n\n> Q3. Training time\n\nA3. The training efficiency is also one of the strengths of our LoCoOp.\n\nFor a 1-shot setting on ImageNet-1K, LoCoOp takes about 13 minutes with a single A100 GPU.  For a 16-shot setting on ImageNet-1K, LoCoOp takes about 3.5 hours with a single A100 GPU.  \n\nTherefore, LoCoOp is easy to implement in environments with limited GPU resources."}, {"review_id": "562eC58hdZ", "reviewer": "Reviewer_wEUW", "summary": "In this paper, the authors propose a CLIP-based few-shot OOD detector named Local regularized Context Optimization (LoCoOp). The method LoCoOp uses learnable prompts and local tokens' class scores to optimize the OOD detection performance. During the few-shot training phase, the authors extract the ID-irrelevant regions by top-k ranking w.r.t. the ground truth class and minimize their class entropy. In this way, the affinities between local tokens and ID classes are suppressed, thus enhancing the distinction between ID and OOD samples. The experimental results show the proposed method outperforms baseline methods.", "questions": "Some minor issues.\n   - The concept *OOD image features* in Line 146 is prone to ambiguity. Readers may think that it is a feature taken from the OOD image.\n   - There may be a typo in Figure 2 x-axis label *# of  labels*. Perhaps *# of samples* is the correct one?", "limitations": "N/A", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well-organized and clearly presented. The proposed method is easy to follow.\n2. LoCoOp is a novel method that uses OOD region regularization to suppress the ID similarity of the background components. The idea of ID-irrelevant local context suppression is interesting. The proposed method raises a valuable point that suppressing extraneous background helps distinguish ID from OOD data.", "weaknesses": "1. The method needs more analysis. \n   - The authors may provide some visualizations for ID and OOD samples to show which regions are suppressed. This kind of case study helps demonstrate that the proposed method works as the authors envision. \n   - The foreground object F and its background context B may tend to co-occurrence, so there may be a spurious correlation issue [1]. Hence, the GT label may also rank higher in the background context. For example, for the background, like the beach, categories associated with the beach (e.g., beach chairs, surfboards) may rank higher than categories that have no relations. Based on this, for some categories with spurious correlation, will this training strategy fail? Because the ground-truth label may occur in top-k candidates for almost all background regions.\n   - The OOD regularization loss (Eqn. 5) needs to be further analyzed. The authors may investigate the training dynamics of the regularization loss. How does this loss affect prompt learning convergence? Will it cause instability and affect ID accuracy in a longer non-few-shot training schedule? How does it affect hard samples?\n   - The proposed method may need some theoretical analysis and support.\n2. More experiments and validation are necessary.\n   - I suggest that the authors extensively verify the effectiveness of the proposed method on various OOD benchmarks. In particular, it is necessary to pay attention to whether the proposed OOD regularization loss will behave as expected when the network is trained with fewer categories (such as CIFAR10)?\n   - The authors only train the network with up to 16 samples per class. I'm curious if one can achieve better results by using more samples to finetune the network? Where is the bottleneck of OOD detection based on prompt learning? Can we unfreeze more layers to obtain stronger OOD regularization capabilities?\n   - In Table 1, why does one-shot prompt learning of CoOp+MCM result in lower performances compared to zero-shot MCM?\n\n[1] Ming, Yifei, Hang Yin, and Yixuan Li. \"On the impact of spurious correlation for out-of-distribution detection.\" *Proceedings of the AAAI Conference on Artificial Intelligence*. Vol. 36. No. 9. 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the valuable comments and respond to them as follows. We will add suggested experiments and explanations in the updated manuscript.\n\n> Q1. Visualization results\n\nA1. Thanks for pointing this out. We attached the visualization results. From this result, we can see the OOD regions are extracted correctly.\n\n> Q2. Spurious correlation issue\n\nA2. Thanks for the interesting point of view.  We consider that the spurious correlation issue does not occur in the CLIP’s local features. Unlike global features, CLIP's local features are features before pooling, so the feature in each region has no mixed concepts. In other words, areas with beaches have the concept of beaches only, and areas with beach chairs have the concept of beach chairs only. Therefore, ground-truth labels are unlikely to be included in the top-k candidates in background regions. This may be more readily understood by examining the visualization results that I have attached. The unmasked areas in these visualization examples are areas where the ground-truth labels are not included in the top-200 prediction. Therefore, we consider that spurious correlation does not occur in the local features.\n\nBased on this observation, in our view, LoCoOp can directly solve the problem of the spurious correlation issue.  LoCoOp separates the background from the foreground, which results in the elimination of spurious correlations between things that are normally correlated, such as beach and beach chairs. We will document this discussion in detail in the final version.\n\n> Q3. Analysis of the OOD regularization loss\n\nA3. The training convergence and instability for LoCoOp are similar to those of CoOp, and the training config is the same as that of CoOp.\n\n> Q4. Theoretical analysis and support.\n \nA4. Thanks for the important suggestion. In terms of the effectiveness of disentangling background information, our work is supported by existing theoretical studies [1, 2, 3], which state, when the classifier only uses foreground features, the optimal decision boundary can be obtained. Therefore, we will cite these theoretical studies [1, 2, 3] to reinforce the theoretical background in the final version.\n\n[1] Ren et al., Likelihood Ratios for Out-of-Distribution Detection, NeurIPS 2019.  \n[2] Nagarajan et al., Understanding the Failure Modes of Out-of-Distribution Generalization, ICLR 2021.  \n[3] Ming et al., On the impact of spurious correlation for out-of-distribution detection, AAAI2022.\n\n> Q5. The effectiveness of LoCoOp on small-scale datasets\n\nA5. As for the small-scale dataset, there are no existing studies dealing with CIFAR-10 in CLIP, because CLIP is not compatible with toy datasets (e.g., too low resolution) like CIFAR-10.   Hence, we experiment with an ImageNet subset dataset.\n\nPrevious work [30] reported that the result on ImageNet-10 [30] (a 10-class subset of ImageNet) and ImageNet-20 [30] (a 20-class subset of ImageNet) with zero-shot MCM reached the upper-bound score (e.g., average AUROC is 99.78 on ImageNet-10). Therefore, we use ImageNet-100  (a 100-class subset of ImageNet) as the ID dataset. As for the OOD datasets, we adopt the same ones as the ImageNet-1K OOD datasets.\n\nThe result on ImageNet100 OOD datasets is as follows.  \n\n|  | FPR (%) | AUROC (%) |\n| --- | --- | --- |\n| CoOp with MCM | 14.57 | 97.12 |\n| CoOp with GL-MCM | 13.82 | 96.93 |\n| LoCoOp with MCM (ours) | 12.68 | 97.49 |\n| LoCoOp with GL-MCM (ours) | **10.77** | **97.67** |\n\nFrom this result, LoCoOp outperforms CoOp on small-scale datasets. \n\n> Q6. The performance of LoCoOp with more than 16-shot samples\n\nA6. In this paper, we conducted experiments up to 16 shots following the setting of CoOp.  Figure 2 shows that the improvement in performance will become slower after 16-shot training. This is due to the fact that CoOp and LoCoOp only train text prompts that have only a few parameters, and a small number of samples is sufficient to reach the upper score. \n\nA key element to further improve performance is CLIP's vision encoder. LoCoOp freezes the vision encoder, so the image features may contain some OOD information. Removing OOD information in image features can be a possible solution to further OOD detection performance.\n\n> Q7. The lower performance of 1-shot CoOp with MCM than that of zero-shot MCM\n    \nA7. According to the paper in CoOp, the classification accuracy of prompt learning methods with 1-shot training is inferior to the zero-shot performance because there are not enough samples to learn enough discriminative representations in prompts. Similarly, in the case of OOD detection, CoOp cannot learn enough features with only one sample. However, LoCoOp is capable of learning multiple OOD features from a single image. Therefore, even in a setting with only one image, it can learn a sufficient amount of features to achieve robust performance in OOD detection.\n\n> Minor issues: ambiguous description and  typo\n    \nThanks for the point out. We will fix them in the final version."}, {"review_id": "lKKnQQBCVl", "reviewer": "Reviewer_RaSR", "summary": "This paper focused on the problem of vision-language prompt learning for few-shot OOD detection, i.e., using CLIP model to detect OOD images from unseen classes using only a few labeled in-distribution (ID) images. Previous zero-shot methods may encounter a domain gap with ID downstream data, while fully fine-tuned methods require enormous training costs. This work focused on the few-shot learning setting and extends CoOp, a few-shot vision-language prompt learning, to OOD detection. The main difference is to extract ID-irrelevant regions based on region-class similarity for OOD regularization. Experiments are conducted on iNaturalist, SUN, Places, and TEXTURE as OOD datasets and take ImageNet-1K as ID data. Experimental results verified the effectiveness of the proposed LoCoOp, even performing better than zero-shot and fully fine-tuned methods with 1-shot training data.", "questions": "Please see weaknesses for my quesions.", "limitations": "The authors have adequately addressed the limitations", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "+ The problem of vision-language prompt learning for few-shot OOD detection is novel and interesting. Previous works focused on zero-shot or fully fine-tuning settings, while few-shot OOD detection is not well explored. Also, it makes sense to adapt few-shot prompt tuning method CoOp for the few-shot OOD detection.\n\n+ The proposed method is simple but effective and intuitive. It directly extract pseudo OOD samples from few-shot ID samples. The motivation and solution are clear and inspiring.\n\n+ The experimental results verified the effectiveness of the proposed method. Especially, it can perform better than zero-shot or fine-tuned methods even with one-shot training data. Compared to complex method CoCoOp, it can also achieve faster inference speed.\n\n+ The writing quality is satisfying. It is easy to follow the idea and implementation.", "weaknesses": "- Although Table 3 compares the ID accuracies of CoOp and LoCoOp, it is interesting to know how other zero-shot and fine-tuned methods perform on ID accuracy, as CoOp may not achieve good ID performance due to the limitation of sample numbers.\n\n- As the motivation is to extract OOD regions from ID images, it would be interesting to visualize the OOD regions to illustrate how well CLIP can output OOD regions.\n\n\n------------------\n\nAfter rebuttal:\n\nThank the authors for providing further comparisons and discussions. The comparisons and discussions are comprehensive. I will keep my positive rating.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the valuable comments and respond to them as follows. We will add suggested experiments and explanations in the updated manuscript.\n\n> Q1. The ID accuracy with zero-shot and fully-supervised methods\n\nA1. The results of the ID accuracy and OOD performance on the ImageNet validation set for the zero-shot, fully supervised, and prompt-based methods are shown below.\n\n|  | ID acc. (%) | AUROC (%) |\n| --- | --- | --- |\n| Zero-shot (CLIP [35])  | 67.01 | 90.83 |\n| Fine-tune (NPOS [43]) | 79.42 | 90.37 |\n| CoOp [56] | 72.10 | 91.82 |\n| LoCoOp (ours) | 71.70 | 93.52 |\n\nEven though MCM, LoCoOp, and CoOp are inferior to fully supervised methods in ID accuracy, they performed better in OOD detection.  \n\n> Q2. Visualization of OOD regions\n    \nA2. Thanks for the suggestion.  We attached visualization results for some samples to demonstrate the effectiveness of our method in the global response. This shows that our method can correctly extract OOD regions."}, {"review_id": "desnhzvnxA", "reviewer": "Reviewer_epH7", "summary": "The paper proposed locally regularized Context Optimization for OOD detection inspired by CoOP. Their primary claim is that the CLIP feature contains many ID-irrelevant nuances, such as backgrounds. Hence pushing the ID and these embedding from each other will lead to a better separation between ID and OOD samples. It is a few shot OOD detection methods that utilize learnable prompts. The author conducted experiments on zero/one/few shot methods, which shows their proposed method outperforms existing approaches. ", "questions": "Are there any results on near OOD or LargeScale datasets?", "limitations": "No negative social impact and limitations are addressed adequately ", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The strengths of the paper are : \n1. The paper is well-written and easy to follow. \n2. It is a simple approach combined with CoOP, MCM, and GL-MCM scores for test time prediction, giving 3. SOTA performance in zero-shot settings. \n4. Pushing apart the ID-irrelevant features from CLIP, similar to MaskCLIP, would lead to a better OOD classification is a novel idea and has much potential. \n5. The results are also good; it outperforms most existing models using both ResNet and ViT.\n", "weaknesses": "There are a small few weaknesses of the paper which the author needs to address  :\n1. The prompt learning used here is the same as CoOP, then why is it novel?\n2. Table 1. Zero-shot uses CLIP or CoOP, it is not clear. \n3. The performance using CoOP with MCM/GL-MCM is already quite good, and LoCoOP improves an additional 1-2% gain in AUROC only in the one-shot or few-shot setting. Hence I doubt its efficacy in large-scale OOD detection or in near OOD detection. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the valuable comments and respond to them appropriately as follows. We will add suggested experiments and explanations in the updated manuscript.\n\n> Q1. The novelty of LoCoOp\n    \nA1. While CoOp learns to bring the global image features and GT text feature closer together, LoCoOp treats the portions of CLIP's local features as OOD and performs OOD regularization in addition to the CoOp's loss. Although it seems like just adding an OOD regularization term to CoOp, it is very effective for OOD detection. Therefore, we consider that LoCoOp has a solid novelty for few-shot OOD detection.\n\n> Q2. Zero-shot method\n    \nA2.  We use CLIP for zero-shot results.\n\n> Q3. Performance improvements with LoCoOp\n    \nA3. That is true that LoCoOp brings 1-2% gain in AUROC over CoOp, but we argue that this improvement is not small on ImageNet-1K OOD benchmarks. Existing work (NPOS [43]) reported that NPOS brings only a 0.03 % gain over VOS [9] on ImageNet-1K benchmarks. Therefore, we consider a 1~2% gain to be very significant in this tight setting where we have only a few labels data with the large ImageNet benchmarks.\n"}]

Author rebuttal:
We would like to express our gratitude to the reviewers for giving excellent and positive comments and recognizing our contributions: “Few-shot out-of-distribution detection is interesting and practical” (**Lcii, uDwi, RaSR**), “LoCoOp is well motivated and interesting” (**Lcii, uDwi, wEUW, RaSR, epH7**), and “The effectiveness of LoCoOp is clear” (**Lcii, uDwi, RaSR, epH7**), and “This paper is well written” (**wEUW, RaSR, epH7**).

We write responses to each reviewer in each thread. Visualization is a common question, and we share the visualization results here. 
## Visualization of OOD regions

Thanks for the really important point. We have attached the visualization results of OOD regions LoCoOp extracts in the pdf.  This shows that our method can correctly extract OOD regions. 

As reviewers (**Lcii, uDwi, wEUW, RaSR**) pointed out, the performance of segmentation results is a key to our method. However, unlike normal segmentation tasks, it is not necessary to correctly guess the segmentation label in our setting. Therefore, to compensate for some errors in segmentation results, we introduce Rank in Eq. (4) in the main, which allows segmentation mistakes up to top-K. 

We will add this figure and this explanation in the final version.

Meta-review: The paper has received unanimous acceptance recommendations, with one accept, one weak accept, and three borderline accept ratings. The topic of few-shot out-of-distribution detection is considered highly relevant and interesting to the research community. Reviewers have lauded the simplicity and effectiveness of the method, which is substantiated by compelling experimental results.

The Area Chair concurs with the reviewers' assessments and joins them in recommending the acceptance of the paper.
Decision: Accept (poster)