Local-Prompt: Extensible Local Prompts for Few-Shot Out-of-Distribution Detection
Authors: Fanhu Zeng
Zhen Cheng
Fei Zhu
Hongxin Wei
Xu-Yao Zhang
Venue: ICLR 2025 Poster (2025)
Keywords: Out-of-distribution detection
Prompt learning
Vision-language model
Few-shot learning
Primary area: alignment, fairness, safety, privacy, and societal considerations
TL;DR: learning local prompts to effectively enhance few-shot out-of-distribution detection with local outlier knowledge

Abstract: Out-of-Distribution (OOD) detection, aiming to distinguish outliers from known categories, has gained prominence in practical scenarios. Recently, the advent of vision-language models (VLM) has heightened interest in enhancing OOD detection for VLM through few-shot tuning. However, existing methods mainly focus on optimizing global prompts, ignoring refined utilization of local information with regard to outliers. Motivated by this, we freeze global prompts and introduce Local-Prompt, a novel coarse-to-fine tuning paradigm to emphasize regional enhancement with local prompts. Our method comprises two integral components: global prompt guided negative augmentation and local prompt enhanced regional regularization. The former utilizes frozen, coarse global prompts as guiding cues to incorporate negative augmentation, thereby leveraging local outlier knowledge. The latter employs trainable local prompts and a regional regularization to capture local information effectively, aiding in outlier identification. We also propose regional-related metric to empower the enrichment of OOD detection. Moreover, since our approach explores enhancing local prompts only, it can be seamlessly integrated with trained global prompts during inference to boost the performance. Comprehensive experiments demonstrate the effectiveness and potential of our method. Notably, our method reduces average FPR95 by 5.17% against state-of-the-art method in 4-shot tuning on challenging ImageNet-1k dataset, even outperforming 16-shot results of previous methods.

Published as a conference paper at ICLR 2025
LOCAL-PROMPHT EXTENSIBLE LOCAL PROMPTS FOR FEW-SHOT OUT-OF-DISTRIBUTIA DETECTION
Fanhu Zeng'.2, Zhen Cheng',2, Fei Zhu', Hongxin Wei', Yao Zhang 1.2+
'State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation,
Chinese Academy of Sciences
School of Artificial Intelligence, University of Chinese Academy of Sciences
*Centre for Artificial Intelligence and Robotics, HKISI-CAS
*Department of Statistics and Data Science, Southern University of Science and Technology {zengf anhuu202,, chengzhen2019, zhufei201 8 @ cn xyz@nlpr pr. ia. ac. weihx@sustecch en
ABSTRACT
Out-of-Distribuud (OOD) detection, aiming to distinguish outliers from known categories, has gained prominence in practical scenarios. Recently, the advent of vision-language models (VLM) has heightened interest in enhancing OOD de- tection for VLM through few-shot tuning. However, existing methods mainly focus on optimizing global prompts, ignoring refined utilization of local infor- mation with regard to outliers. Motivated by this, we freeze global prompts and introduce Local-Prompt, novel coarse-to-fine tuning paradigm to emphasize regional enhancement with local prompts. Our method comprises two integral components: global prompt guided negative augmentation and local prompt en- hanced regional regularization. The former utilizes frozen, coarse global prompts as guiding cues to incorporate negative augmentation, thereby leveraging local outlier knowledge. The latter employs trainable local prompts and regional reg- ularization to capture local information effectively, aiding in outlier identification. We also propose regional-related metric to empower the enrichment of OOD de- tection. Moreover, since our approach explores enhancing local prompts only, it can be seamlessly integrated with trained global prompts during inference boost the performance. Comprehensive experiments demonstrate the effective- ness and potential of our method. Notably, our method reduces average FPR95 by 5.17% against state-of-the-an method in 4-shot tuning on challenging ImageNet- Ik dataset, even outperforming 16-shot results of previous methods. Code is re- leased at https: / / thub. com htmm//rroraZeggh//icceo larooccaaapprppppse -Prompt.
INTRODUCTION
Out-of-distribution (OOD) detection (Liu et al., 2020; Cheng al., 2023b; Jaeger et al., 2022; Tao et al., 2022) aims to distinguish outliers, i.e., samples that do not belong to known in-distribution (ID) classes. is crucial for industries that require a high level of safety, such as face recognition (Lopez- Lopez al., 2022) and autonomous driving (Filos et al., 2020). Most previous methods in the field of OOD detection handle the problem using single-modal modal methods and concentrate on post-hoc processing (Hendrycks & Gimpel, 2016; Wang al., 2022) or leveraging outliers (Hendrycks al., 2018; Jiang et al., 2024). However, they suffer from computational inefficiency or consuming data collection. Recently, as the emergence of vision-language models (Radford et al., 2021; Li al., 2021) shows promising results in multi-modal tasks including image caption (Li et al., 2022a), video understanding (Xu et al., 2021), and on, it is promising to exploit textual representations to improve performance of OOD detection given a vision-language model (Radford et al., 2021) as a prior model. Considering that zero-shot undergoes domain gap between upstream and target distribution while full-tuning may pose threats to the leamned representation, it is meaningful to explore few-shot learning for lasion-language model OOD detection, in which the detector prohibited from real outliers and only has access to several ID images.
"Corresponding author.

---

Published as a conference paper at ICLR 2025
Feature Space
O0D
Tocus overall similaaity
Optimization Space
Failure
x
Global Optinitation
Existing methods
0OD
Globei poumpe
Vision
11
IDripe
Langrege
Oetioggiaal optlantotionn divection
Modei
Our method
Enharcement
oat obout regionel difference
Local Prompes
sample
OOD sample
challeage GOD sumple local ID region
local OOD region
Figure I: Comparison of prompt learning for OOD detection task. Prompts with global optimization may fail in challenging OOD samples as they are overall similar to ID samples and only have subtle regional differences. For example, cat and tiger are generally similar (blue boxes) and only differ in forehead (red box). Our approach with local outlier knowledge cares about region difference and tackles the issue to some extent.
The most challenging scene for OOD detection is that one hard OOD sample is similar to a known class on the whole and only has subtle differences locally, which naturally requires the detector to identify outliers through local outlier regions. However, existing research falls short of refining OOD task via rich local information when subtle OOD samples are exhibited in certain regions, as is shown in Fig. 1. Some methods merely focus on utilizing global features only (Ming et al., 2022) (blue boxes in Fig. 1), which ignores local features (red boxes in Fig. 1) and inevitably brings about coarse description. Others use the same prompts to match both global and local image features (Miyai et al., 2023b;a), so the gap between them may lead to inaccurate local outlier identification. Consequently, it is straightforward that enhaneing regional information to empower the model with local outlier knowledge could be significant to OOD detection.
Motivated by the above observations, focus how to explicitly exploit regional-related knowl- edge, ie., the learned knowledge is determined by the position of regions, about not only known classes but also unknown samples, which has not been explored before. To tackle the issue, propose Local-Prompt, which concentrates on enhancing local features solely and introduces ex- tensible local prompts for fine local information utilization. Concretely, we decompose global and local prompts to exploit global and local features, and term prompts that interact with global/local features as global/local prompts. Then we propose two integral components: global prompt guided negative augmentation and local prompt enhanced regional regularization. Global prompt is frozen to provide guidance for generating negative augmentation, which could facilitate the exploitation of local outlier knowledge. Fine local prompt is designed for regional regularization to refine regional- related local prompts, which captures the fine local information to better distinguish OOD inputs. Moreover, we also propose corresponding regional-relater evaluation metrics to leverage the power of local prompts for OOD detection.
Another advantage of our approach is that it orthogonally explores the benefits of local prompts shown in Fig. and uses hand-crafiee global prompts, i.e., a photo a {class}. Hence, it is ex- tensible to all existing global prompt designing methods (Miyai al., 2023a; Wang et al., 2023). Therefore, we can extend it by seamlessly replacing coarse global prompts with trained ones during inference improve the performance.
Comprehensive experiments demonstrate the effectiveness of the proposed method. Specifically, our results outperform the state-of-the-art methods in both OOD detection (5.17% reduction on FPR95) and ID accuracy evaluation (1.02%) under 4-shot tuning by a large margin, exhibiting the superiority and the potential of enhancing local prompts for OOD detection and ID classification ability. We additionally carry out various analysis on and all quantitative outcomes and qualitative visualization validate effectiveness. We summarize our contributions as follows:
We focus on enhancing local information with local prompts and propose a coarse-to-fin few-shot tuning paradigm. Specifically, we propose effective negative augmentation and regional regularization to learn local prompts with local outlier knowledge.
We propose regional-related OOD score and ID classification metrics, which benefit from enhanced local prompts in OOD detection. Moreover, our method is orthogonal to global
2

---

Published as a conference paper at ICLR 2025
prompt optimization methods and is extensible to get notable performance when integrated with trained local global prompts during inference.
We conduct comprehensive experiments and achieve competitive results on various OOD detection datasets, demonstrating the superiority of our method. Notably, our 4-shot results on challenging ImageNet- Ik even outperform 16-shot results of previous methods.
2 RELATED WORK
2.1 DITTRIUDDITTIIIIIONN DETECTION WITH VISION-LANGUAGE MODEL
Out-of-distribune detection, as a crucial part of open environment (Zeng et al., 2024; Zhu et al., 2023b; Cheng et al., 2023b), has been widely studied the past decades (Hendrycks et al., 2018; Cheng et al., 2023a; Shu et al., 2023; Zhu et al., 2023a; Liu et al., 2024). With the emergence of CLIP (Radford et al., 2021), which learns outstanding visual and textual representations pre- trained on large-scale image-text pairs and displays excellent ability in various downstream tasks, much attention has been paid to transferring vision- language model to OOD detection task (Fort al., 2021; Ming al., 2022; Esmaeilpour et al., 2022; Wang al., 2023; Miyai et al., 2023a; Park et al., 2023; Bai et al., 2024). For instance, MCM (Ming et al., 2022) adopts the visual and textual representations extracted from CLIP encoders and applies softmax with temperature to better separate ID and OOD data. ZOC (Esmacilpour et al., 2022) generates pseudo unseen class labels with additional modules and defines a novel confidence score accordingly. Clipn (Wang al., 2023) enhances the discrimination of OOD samples with extra encoder and data (Sharma et al., 2018).
2.2 MULTI-MODAL FEW-SHOT PROMPT LEARNING
Since lasion-language model like CLIP (Radford et al., 2021) achieves superior performance in numerous image-text tasks, many attempts have been made to explore few-shot tuning to leverage the power of large pre-trained models. There are mainly two lines of approaches, namely Adapter Learning (Zhang et al., 2021; Gao et al., 2023; Guo et al., 2023; Udandarao et al., 2023; Zhu al., 2023c) and Prompt Learning (Zhou et al., 2022b; Bahng et al., 2022). Both of them froze the pre- trained encoders when fine-tuning a few additional parameters.
Prompt learning promotes the performance of vision- langun-a model by enhancing the prompts in both visual (Jia et al., 2022; Bahng et al., 2022) and textual (Zhou et al., 2022b;a) aspects. A common practice is setting textual prompts learnable and forwarding them together with visual inputs. CoOp (Zhou et al., 2022b) replaces the hand-crafted prompts with contextual learnable embeddings and optimizes them during few-shot training. In our approach, we freeze global prompts and fine-tune local prompts individually.
2.3 GLOBAL AND LOCAL INFORMATION
In Vision Transformer (Dosovitskiy et al., 2020; Zeng & Yu), global tokens are specifically designed for image classification task (Touvron al., 2021; He al., 2022). They represent overall character- istics of images. Local tokens are utilized for dense prediction tasks including segmentation (Cheng al., 2022; Strudel et al., 2021), object detection (Chen et al., 2022; Meng et al., 2021), and so on. The utilization of local features has been explored as well, such pyramid features (Wang et al., 2021) and window attention (Liu et al., 2021; Li et al., 2022b; Liu al., 2022).
In the field of OOD detection, global features are primarily employed and numerous studies have been conducted concerning local information (Miyai et al., 2023b;a; Zhang et al., 2023). GL- MCM (Miyai et al., 2023b) takes all tokens into consideration and uses the sum of global and local OOD scores to measure the confidence. LoCoOp (Miyai et al., 2023a) keeps global ID textual embeddings away from the interference of ID-irrelevant regions. However, all existing approaches take same prompts for all features. By contrast, our method directly enhances OOD detection with ID-related areas and refines local prompts leverage local outlier knowledge.
3

---

Published as a conference paper at ICLR 2025
3 PRELIMINARY
3.1 PROBLEM DEFINITION
Few-shot out-of-distribution detection. Formally, dustii-dittiibutiin detection be viewed as a binary classification that the detector has to identify whether the input image is from ID or OOD space. Labeled training data is composed of Din trgin {3 y4) (=))==>- where æ; is sampled i.idd from joint data distribution space Pxym and y/4 E ) ) {1..... CC is the label space of ID data and C' is the number of classes. We transfer the definition of few-shot learning in classification to OOD detection fine-tune given vision-language model instead of training from scratch. Under few-shot setting, a small proportion of images in each class (e.g-, 1, 4, or 16 images) are extracted for training. youu is the label space of OOD) data and there is overlap between IID and OOD label, i.., ) yout = 0. Typically, the model is unable to obtain any OOD data during training process.
The test dataset is mixture of Die and Diett When testing, OOD performance is evaluated by discriminating which distribution each sample comes from. A discriminant function is calculated to identify ID and OOD samples:
S(z) 27 D (OOD,
ID,
(1)
S(x) <>
where ((z) is the confidence score function measuring the uncertainty and 7 the threshold.
Prompt learning with CLIP. CLIP (Radford et al., 2021) is compromised of image encoder Ep and text encoder ET. Given an image and text T describing the corresponding image, they are extracted into feature embeddings, respectively. In prompt learning, textual prompts can be hand- crafted templates (Radford et al., 2021), e.g.. "a photo of {class}" or learnable context words (Zhou et al., 2022a), e.g., t [V1,... , UL; Uel. where Ve (i = 1.... L) is the learnable embedding, and is the length of context words. v. (c E {I,... CC) is the embedding of class name. Features of prompts are obtained from text encoder (t.) : [R(+1)xd - Rd. For simple notation, we use t t/ described below to represent features of prompts as text encoder is not the focus of the paper.
Image first split into several patches with an additional class token embedding x E (N+))xxd N and d denote the number of local tokens and hidden dimension, respectively. Both global and local visual features are then extracted from image encoder [28, =!| Ex(x) R (NN+))x& + ((N+1) xd where global features 2 and local features represents overall features and patch-wise regional features, respectively.
In this paper, we denote global prompts and local prompts hand-craffeed/eammbie prompts that interact with overall local features and fine local features, respectively.
OOD detection with CLIP. The MCM score (Ming al., 2022) is defined as the maximum of similarity after softmax with temperature:
SMOM (z) = max exp(sim(23,, f./T))
(2)
j=1 exp(sim(2x t2)/T)
and GL-MCM score (Miyai et al., 2023b) considers local information. It applies softmax for local features as well and the maximum is selected local MCM score. Final OOD score is the sum of MCM score and local MCM score:
exp(sim(2), bn)TT) Σj=1 exp(sim( 2),, t,)T))
(c)) (x) + max i,
where z represents global feature and 2/, (h (1,..., .N)) represents all extracted local features.
METHODOLOGY
We propose Local-Prompt, coarse-to-fine paradigm to strengthen OOD detection with local outlier knowledge. We first present negative augmentation that leverages local and unknown outlier infor- mation. Then, we describe local prompt enhancement along with their training regularization and a regional OOD score for better regional information utilization. Finally, we extend our method to trained global prompts boost the performance. Detailed structures are shown in Fig. 2.
4

---

Published as a conference paper at ICLR 2025
Global Prompt Guided Negative ugmentation
Global Prompts
Negative
Evaluation
photo cat
Text Encoder
ugmented Inputs
High
Low
Similarity
Random Crop
image Encoder
0.15
5051
Global Feutures
(b) Local Prompt Enhauced Regional Regularization
Local Features
Local Dutlier Enhancoment
Trainable
Local features of Augmented Inputs
Frozen
Local Prompts
Loal Prompts
Gat
5/
Text Encoder
Regional
0.3
Negative Local Prompts
Selection
Diversity
Contrastive Regularization
Regularization
Loss
Classiicatiin
DOD
Detailed structure of the proposed Local- Our method consists of global prompt guided negative augmentation and local prompt enhanced regional regularization. We froze global prompts select regional augmented samples enhance local prompts learn regional-related representation that helps im- prove both ID accuracy and OOD detection.
Figure 2:
4.1 GLOBAL PROMPT GUIDED NEGATIVE AUGMENTATION
As it is costly and not always effective to rely solely on real outliers, one crucial problem for OOD detection is how to leverage imaginary local outlier knowledge to the detector. We propose to syn- thesize hard negative images to force the model to learn fcatures with strong relation only to simulate outlier situations while avoiding the requirement of real OOD samples. To this end, we apply sim- ple and straightforward random crop augmentation to empower the detector with unknown (i.e., OOD) information. Specifically, we randomly crop image input m times and use the hand-crafted template with corresponding class name, i.e., te "a photo of [class)"' c {I, ,C} as text in- puts to calculate the image-text similarity with global features from image encoder. We then select m1/my images with the largest/least similarity, respectively. Images with the largest similarity can be seen as positive samples and those with the least similarity serve as hard negative samples. No- tably, random crop is helpful to leverage local information in that local prompts learn regional- related outlier information through space translation by means random crop. Effectiveness of random crop is verified in Sec. 5.2.
Global prompts can be viewed as coarse guidance for negative augmentation standing for overall representation and frozen in framework. Motivated by previous observations(Ming al., 2022; Miyai et al., 2023b), we simply use the basic global prompts during training and select aug- mented images as they are fairly good for extracting global features. Concretely, the global prompts are used to (1) guide the negative augmentation selection, which has been described above; (2) guide the global OOD score calculation for evaluation, as shown Sec. 4.3.
It is worth emphasizing that our approach is orthogonal to all existing global prompt optimization strategies, i.e., global prompts are built without tuning in our structure (Miyai et al., 2023a), so dur- ing inference, it is expected that extending the hand-crafted global prompts with carefully designed ones can promote the performance. Concretely, we replace hand-crafted global prompts with trained ones that have the same shape. Note that our main purpose is to decompose global and local prompts and showcase the effectiveness of local outlier enhancement for OOD detection. Therefore, we do not specifically select the template as it is not the focus of the paper.
globa
4.2 LOCAL PROMPT ENHANCED REGIONAL REGULARIZATION
Once augmented inputs decided, corresponding local features then used to optimize local prompts to identify ID and OOD samples from the perspective of regional enhancement. We aim at utilizing the local features extracted from image encoder in a fine way to (1) leam regional-related prompts that better characterize local similarity for both ID and OOD regions; (2) enhance detection ability with refined local features. Specifically, design local prompts with learnable context
5

---

Published as a conference paper at ICLR 2025
words to represent local textual information for each known class. Moreover, account of the existence of OOD regions, we build a few local negative prompts to handle possible local outliers.
Contrary to previous global prompt optimization methods, local prompts that work on local features are expected to learn regional-relate textual representations and be aware of outliers during training. To this end, we enhance local prompts through regional regularization, which consists contrastive regularization and diversity regularization, to better distinguish between ID and OOD samples.
Contrastive regularization. We propose regional contrastive regularization to enhance local textual prompts for better local information utilization. It is composed of local loss and local negative loss. Given a positive sample of training image, local loss is employed to learn regional information of known classes:
T Ts|exp(sim tyy/TT}
( y) -log
(4)
Ei=1 To (xx)(((sm(()) 4)/ + E Ta (exp(sim(=),
where y) stands for training image-label pairs, To (xx) is the sum of k largest elements in æ, is regional number and T is the temperature. We use cosine similarity as the similarity metric:
t
sim (4)
(5)
[4
1|,
Similarly, given hard negative sample of the image denoted as ã. local negative loss is defined to make the model aware outliers:
Ei=t Te(exp(sim( 2 4)TT)
Naog
nag ( - log Ei= Tk TTxxxpssim(2 2) +)/T)} + Em Tk {exxpssim( 2) E)/T)}
(6)
Intuitively, contrastive regularization forces the model to focus on ID-related regions through ran- dom crop, and keep away from outlier-related regions through negative augmentation. We demon- strate the necessity of local negative loss to make margins for unknown categories in OOD detection in Sec.5.2.
Diversity regularization. As local negative prompts randomly initialized, regularization has to be imposed on them to ensure their diversity. Therefore, in addition to the proposed contrastive regularization, we apply diversity regularization on local negative prompts, follows:
1
Greg
sim(f, Ey),,
Nneg( (Nueg 1)/2 SiiejEN
where Nueg is the number of local negative prompts. Final Loss is a weighted sum of the above loss:
Liotal = Lpos + >neg + Areg
(8)
where Aneg and Areg are the coefficients of the corresponding losses.
4.3 REGIONAL OOD SCORE
On consideration that regional information is especially enhanced during training, we propose Regional-MCM score enhance OOD score beyond simple MCM and GL-MCM:
exp(sim 2/,0 (:)/T)
SR-MCM = (M) + Timean
1==1 exp(sim(z), f2)/T) + E/nrr Nuee exp(sim (2), E
where T.mean (x: is the mean of is largest elements in æ and k represents regional number. Intuitively, Regional-MCM serves as a general form of GL-MCM that takes the k most similar regions into consideration and additionally contains local negative prompts, which is helpful consider more than one certain region with high similarity and improve OOD detection ability. Effectiveness of the score is verified in Sec. 5.2.
As the proposed OOD score is only applicable for OOD detection, we further propose a focal-aware score that assigns a score to each class and thus determines ID category with the largest score. Specifically, ID classification takes both global and local features into consideration:
h (x:)
sim (25 t.):
(10)

---

Published as a conference paper at ICLR 2025
Table 1: Results when ImageNet- 1k used as ID data. We compare the methods in different tuning manners. it represents the result by our re-implementation. Bold values represent the best results.
Method
NNaturalist
SUN
Places
Texture
Average
EPROEL
AUROCT
FPROEL
AUROCT
FPRI51
AUROCT
FPRSS)
AUROCT
FFROS)
AUROCT
MCM
30.91
94,61
37.59
92.57
44.69
89.77
57.77
86.11
42.74
90.77
GL-MCM
96.71
30.42
93.09
38.85
89,90
Fudl-naming
MSP
54.05
87.43
73.37
78.05
72.98
78.03
79.06
67.31
80,
Energy
29.75
94.68
53.18
87.33
56.40
85.60
51.35
88.00
47.67
88,90
KNN
29.17
94.52
35.62
92.67
02
85.67
42.19
90.97
NPOS
16.58
96
43.77
90.44
45.27
89.44
46.
88.80
37.93
91.22
Textual
87.34
66.82
83.71
Few--hor
4-xhot
CoOp
18.95
95.52
29.58
92,90
38.72
89.64
48.03
85.87
33.82
90.98
Locul-Prompt
9.65
97.87
20.40
95.57
29.39
92.67
51.20
88.00
27.66
93.53
LoCoOp
21.67
95.69
22.98
95.07
31.41
92.10
49.79
87.85
31.46
92.68
Local Prooootoompopppppppppppppp
97.29
95.85
27.53
94.03
16-shor
CoOp
14.60
96.62
28.48
92.65
36.49
89.98
43.13
88.03
30.67
91.82
Pocal-Prompt
8.71
98.10
23.97
94,88
32,50
92.32
47.93
89.04
28.27
93.58
LoCoOp
16.05
96.86
23.44
95.07
32.87
91.98
42.28
90.19
28.66
93.52
Mucal-Prompt-L.aCo00
8.63
98.07
23.23
95.12
31.74
92.42
34.50
92.29
24.52
94.48
(x))
Thean (exp(sim(2). (4) (T)
(11)
where (a)), (z)) are global and local portions and represents the category label. The score attached to each category is measured by:
fly = iz) hx(z) (x)).
(12)
each sample is then classified corresponding to the maximum score.
5 EXPERIMENTS
Datasets. Following existing works (Ming et al., 2022; Miyai et al., 2023b;a), large scale ImageNet- 1K (Deng et al., 2009) along with a 100-category subset denoted as ImageNet- 100 are used as ID dataset and OOD dataset is a combination of Naturalist (Van Horn et al., 2018), SUN (Xiao et al., 2010), Places (Zhou et al., 2017) and Texture (Cimpoi et al., 2014). In addition, We also use two semantically similar subsets of ImageNet- 1k, ie.., ImageNet- 10 and ImageNet-20, to evaluate near OOD detection performance (Ming et al., 2022). Detailed information is provided in Appendix A.
Implementation details. We use C the backbone. Image encoder, text encoder, and global prompts are frozen and only local prompts are learnable. We set one local prompt for each known category (1000 in total for ImageNet- Aueg and Axeg are and 0.5. respectively. As the stability of temperature T' has been verified by previous research (Ming et al., 2022; Miyai et al., 2023b), we set T be by default. More details are shown Appendix B.
Evaluation metrics. We report the following metrics for evaluation: (1) the area under the receiver operating characteristic curve (AUROC); (2) false positive rate of OOD samples when true positive rate of ID samples is 95% (FPR95); (3) in in-distribution data classification accuracy (ID accuracy).
5.1 MAIN RESULTS
ImageNet- 1k as ID dataset. We report 4 and 16-shot results on four OOD datasets using ImageNet- 1k as ID dataset. It can be seen in Tab. that tuming local prompts only using hand-crafted global prompts can achieve competitive results, especially in datasets with challenging fine local textures like iNauualist. Concretely, we get impressive progress in 4-shot tuning (outperforming by 3.80% on FPR95). In 16-shot setting, approach gets competitive results as well. All results above strongly showease the potential regional enhancement for OOD detection as an orthogonal direction to global prompt optimization methods. Results of more shots shown in Appendix C.
ID accuracy. In addition to OOD performance, it is also meaningful to evaluate ID accuracy as is expected to ensure classification accuracy and separate outliers at the same time. However, weighing too much about separating outliers may narrow the inter-class distance, thus harming accuracy. Con- sequently, evaluate IDD accuracy for ImageNet-1k and the results shown in Tab. 2 conclude that

---

Published as a conference paper at ICLR 2025
Table 2: IDD accuracy evaluation using Table Comparison results of near OOD detection tasks. We report 16-shot results for few-shot methods. Our method can achieve performance against other methods.
ImageNet- Ik as ID data.
Method
Tuning Manner
ID accurncy
MCM
zeto-shot
67.01
ID
IetgeNet-10
ImageNet-20
GL-MCM
67.01
Method
Average
OOD
ImngeNNtt220
ImageNet-10 10
KNN NPOS
full-nuning
79.42
79.64
Fite0s]
AUROCT
F1005]
AUROC1
AL3OCCT
LoCoOp
69.50
MCM
5.00
98.71
12.51
97.70
8.75
98.21
4-shot
Local-Prompt
70.52
GL-MCM
10.10
98.04
9.00
98.62
9.55
98.33
LoCoOp
LoCoOp
11.20
97.49
12,00
97.79
11.60
97.61
Local-Prompe
16-shor
74.24
Loral-Prompt
3,90
99.06
98.84
98.95
Table 4: Experiments on ImageNet- 100. 4-shot results are reported for few-shot tuning.
Method
iNaturalist
SUN
Places
Texture
Average
FPRO5L
AUROC1
AUROCT
F9R951
ALBOCT
FPR95L
AUROCT
AUROCT
MCM
18.13
96.77
36.45
94.54
34.52
94.36
41.22
92.25
32.58
94.48
LoCoOp!
44.94
93.31
30.70
94.70
34.31
93.93
55.92
90.01
41.47
92.99
Local-Prompt
6.99
98.05
24.34
96.03
26.04
95.49
36.32
93.67
23.42
95.81
our method significantly surpasses previous methods (1.02% and 2.38% respectively). demon- strates that the proposed regional enhancement is also beneficial to improve OOD separation while increasing inter-class distance. which is of great significance for real upplication.
Extensive to global prompts. Benefiting from the convenient integration with well-trained global prompts, we combine the advantage of trained global prompts from LoCoOp (Miyai et al., 2023a) without further de- and get further substan- tial improvements on all datasets. For example, we sur- pass the state-of-the-at method by large margin (5.17% on FPR95 and 1.35% on AUROC), even against previ- methods with 16 shots. Specifically, extending local prompts to global prompts is helpful to datasets with major global representations like Texture, ver- ifying the compatibility and potential of local prompt optimization. It is notable that the extension process is seamless and can be integrated with any global optimization prompts, strongly showing the extensibility of our approach. Density map in Fig. 3 also illustrates the superiority of our method. More visualization can be found in Appendix D.
178.0/244
sign Procal-Prompt+LoCCC
LoGoOp
Our
Figure 3: Comparison of density map on iNaturalist. Ours are more separable.
Near OOD detection tasks. Near OOD task typically evaluates semantically similar categories and is relatively more challenging. Following existing work (Ming et al., 2022), we conduct experiments on ImageNet-10 vs. ImageNet-20 to verify the effectiveness of the proposed approach. Results in Tab. 3 indicate that our method exceeds previous methods by a large margin. It can be observed that previous few-shot method fails in near OOD tasks. We attribute to the refinement of local prompts by local outlier knowledge, which greatly assists in distinguishing semantically similar categories. By contrast, relying solely on global prompts may confuse the distribution. Therefore, it underlines that our local outlier enhancement approach is especially effective in semantically near OOD tasks.
ImageNet-1000 as ID dataset. We also conduct experiments using ImageNet-106 as ID dataset. We compare with other zero-shot and few-shot methods and results reported in Tab. 4. The substantial performance improvements (average 9.16% reduction on FPR95 and 1.33% promotion on AUROC) strongly demonstrate the effectiveness and transferability of our approach.
Comparison with outlier methods. Negl (Jiang et al., 2024) designs a novel scheme for OOD score with negative labels. It leverages real outlier information with negative labels from extensive corpus databases. This kind of knowledge helps to great extent pointed out by OE (Hendrycks al., 2018) and is inconsistent with real-world application, where negative categories are infinite. As can be seen from Tab. 5, we observe that (1) although without outlier, our model achieves better
Table 5: Comparison with NegLabel, which leverages powerful outlier samples.
iNaturalist
SUN
Places
Texture
Average
Method
FPROS1
AUROCT
FPRO5
AUROCT
EPROS
AUROCT
FPRUS1
AUROCT
FPRO5L
AUROCT
NegLabel (w outlier)
1.91
99.49
20.53
95.49
35.59
91.64
43.56
90.22
25.40
94.21
Pooal- (w/ outlier)
8.63
98.07
23.23
95.12
31.74
92.42
34.50
92.29
24.52
94.48
8

---

Published as a conference paper at ICLR 2025
Table 6: Effectiveness of each component in loss function.
iNaturalist
SUN
Places
Texture
Average
Loss
FPR95L
AUROCT
FPR95,
AUROC
FPR951
AUROCT
FFRSEJ
ALROCT
FPR95,
AUROCT
Baseline (Cooa)
11.64
97.59
26.81
94.59
34,50
91.59
50.46
87.84
30.85
92.90
+ Laag
9.84
97.88
24.37
94.97
32.84
91.94
50.02
88.32
29.27
93.27
+ Long Line
9.39
97.97
24.20
95.10
32.13
92.21
49.37
88.66
28.77
93.48
Table 7: Influence of different negative augmentation strategies.
iNaturalist
Texture
Average
BPR95L
AUROCT
FPR:51
AUROCT
EPR95L
AUROC1
1PR95
AUROCY
EPR9S..
AUROCT
Baseline
11.18
97.68
23.30
95.25
30.97
92.44
50.44
87.89
28.97
93.31
17.47
94.73
+gaussian
94.39
51.06
92.96
+random crop
9.39
97.97
24.20
95.10
32.13
92.21
49.37
88.66
28.77
93.48
average performance against NegLabel: (2) method achieves better balance between different kinds of OOD datasets, which strongly validates the effectiveness of incorporating local information and strengthens its application diverse and infinite read-world scenarios.
5.2 ABLATION STUDY
We conduct various and comprehensive ablation studies to investigate the effectiveness of each component. We train a 4-shot model to showease the function of components and hyper-parameters unless otherwise stated. More studies about training time and coefficient shown in Appendix C.
Effectiveness of loss components. We conduct experiments to analyze the effectiveness of each loss component and results in Tab. 6 imply that local prompts optimization (baseline) achieves com- parable results with global prompts optimization methods, strongly demonstrating the effectiveness of our proposed regional enhancement strategy. Furthermore, taking hard negative samples together with local negative prompts to optimize OOD-considered representation is beneficial to the dis- tinction of outliers. Outcome without regularization loss is better in FPR95 but slightly inferior in AUROC. Considering overall performance and the fact that the optimization of i relies heavily on the initialization without regularization, we reserve regularization loss for stability and convergence.
Influence of negative augmentation strategy. We conduct experiments analyse the influence of negative augmentation strategy. In Tab. 7, baseline does not employ augmentation. Cutout, gaussian noise and random crop are used as augmentation, respectively. It is concluded that improper augmentation is unfavorable to detection ability. By contrast, random crop is helpful enhance regional-related representation and distinguish outliers, which confirms our analysis above.
Number of local negative prompts. The number of local negative prompts N Nneg stands for local outlier knowledge the model learns from negative augmentation. Results in Tab. 8 indicate that the performance is positively related Nneg- However, larger Nneg requires resources. Considering both computational cost and performance, the number 300 is selected.
computational
OOD score strategy. We compare our proposed OOD score with existing scores (Ming et al., 2022; Miyai et al., 2023b) on 16-shots. We also remove local negative prompts in our OOD score analyse their effectiveness. Results in Fig. 4 imply that our regional OOD score attains better performance than existing strategy, displaying the strength of regional enhancement. Moreover, the performance further promoted when negative prompts are introduced. We attribute to possible outlier leverage through negative prompts.
FPR95-
AUROC
CL-MCM
Ours Info LMP
GLMMM
Durs LNP)
LNP)
Figure 4: Ablation study of different OOD score strategies. LNP denotes local negative prompts.

---

Published as a conference paper at ICLR 2025
Table Influence different number local negative prompts.
Noeg
iNaturalist
Average
199891
19R951
199891
199895
AUROCT
100
9.02
98.00
24.71
94.82
32.77
91.87
48.81
88.50
28.82
93.30
200
9.65
97.90
22.58
95.26
32.03
92.18
49.85
88.00
28.52
93.33
300
9.39
97.97
24.20
95.10
32.13
92.21
49.37
88.66
28.77
93.48
5.3
VISUALIZATIION
Visualization of local prompts. We give a detailed explanation of local prompts and visualize the regions that attract attention from local prompts and local negative prompts. Visualization is shown in Fig. in the order of original picture, ID-related, and several typical OOD-related areas. implies that local prompts concentrate ID regions and local negative prompts are helpful to extract ID- irrelevant, i.e,, outlier areas corresponding to certain semantics. For instance, local negative prompts successfully focus sky, sea, and beach, respectively in pirate ship, which is beneficial to enhance OOD detection ability through local and unknown knowledge leverage.
ID: wizte Sork
ID: ship
Lt
original
local prompt
local negative prompts
Figure 5: Visualization of ID and OOD related regions. Local prompts and local negative prompts successfully focus on ID-related and OOD-related regions, respectively. which helps OOD detection.
Visualization of hard OOD samples. We showcase several hard OOD samples that previous global-based method fails to detect, while our method with local enhancement successfully dis- criminates the outlier. For example, in the first image, previous global-based method fails to detect the outlier as it is similar to ID category ocean liner on the whole, with only subtle difference me- chanical devices the deck indicating that it is actually icebreaker (also demonstrated by the iceberg next to it). The same phenomenon are also observed other examples.
Icobrcake
Ground Juffowcri00
Ground Truth: Asple 10 (D0O)
Gibba methed 10)
kkbbal bosed mothed Rose 001
Oues: Outler
Figure 6: Examples of outlier from SUN dataset that fails to be detected by previous global-based method, and our method with local enhancement successfully discriminates them with subtle differences in certain regions.
CONCLUSION
In this paper, we investigate the relationship between global and local prompts for few-shot OOD detection. Concentrating on enhancing outlier knowledge for better local textual representation uti- lization, we decompose global and local features using respective prompts. We use straightforward negative augmentation to leverage local outlier knowledge and propose local prompts to enhance re- gional information by means of regional regularization. We also propose regional-related evaluation metric to enrich the detector with local outlier information. Moreover, our approach is extensible to well-trained global prompts to get better performance. We conduct various experiments and perior results underline the effectiveness of our method. We hope our method can provide a new perspective studying local prompts for OOD detection tasks based on vasion-language models.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENT
This work supported by National Science and Technology Major Project (2022ZD01 16500), Na- tional Natural Science Foundation of China (62222609, 62076236), and CAS Project for Young Scientists in Basic Research (YSBR-083).
REFERENCES
Hyojin Bahng, Ali Jahanian, Swami Sankaranarayanan, and Phillip Isola. Exploring visual prompts for adapting large-scale models. arXiv preprint arXiv:2203.17274, 2022.
Yichen Bai, Zongbo Han. Bing Cao, Xiaoheng Jiang, Qinghua Hu, and Changqing Zhang. Id- like prompt learning for few-shot out-of-distribune detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 17480-17489, 2024.
Qiang Chen, Xiaokang Chen, Gang Zeng, and Jingdong Wang. Group detr: Fast training conver- gence with decoupled one-to-many label assignment. arXiv preprint arXiv:2007. 13085, 2022.
Bowen Cheng, Ishan Misra, Alexander G Schwing, Alexander Kirillov, and Rohit Girdhar. Masked- attention mask transformer for universal image segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 1290-1299 2022.
Zhen Cheng, Yao Zhang, and Cheng- Bheng-Lin Liu. Unified classification and rejection: A one-versus- all framework. arXiv preprint arXiv:2311.13355, 2023a.
Zhen Cheng, Fei Zhu, Xu- Yao Zhang, and Cheng-Lin Liu. Average of pruning: Improving perfor- mance and stability of distiiiiibttion detection. arXiv preprint arXiv:2303.0/201, 2023b.
Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. De- scribing textures in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3606-3613, 2014.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei. Imagenet: A large-scall scale hi- erarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 248-24882 255. leee, 2009.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An im- age is worth 16x16 words: Transformers for image recognition at scale. International Confer- ence on Learning Representations, 2020.
Sepideh Esmaeilpour, Bing Liu, Eric Robertson, and Lei Shu. Zero-shot disttr-disttibutioon detec- tion based on the pre-trainee traitre model clip. In Proceedings of the AAAI conference on artificial intelligence, volume 36, pp. 6568-6576, 2022.
Angelos Filos, Panagiotis Tigkas, Rowan McAllister, Nicholas Rhinehart, Sergey Levine, and Yarin Gal. Can autonomous vehicles identify, recover from, and adapt to distribution shifts? In Inter- national Conference on Machine Learning, pp. 3145-3153. PMLR, 2020.
Stanislav Fort, Jie Ren, and Balaji Lakshminarayanan. Exploring the limits of out-of-distribttion detection. Advances in Neural Information Processing Systems, 34:7068-7088 2021.
Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma, Rongyao Fang, Yongfeng Zhang, Hongsheng Li, and Yu Qiao. Clip-adapter: Better vision- language models with feature adapters. International Journal of Computer Vision, pp. 1-15, 2023.
Ziyu Guo, Renrui Zhang, Longtian Qiu, Xianzheng Ma, Xupeng Miao, Xuming He, and Bin Cui. Calip: Zero-shot enhancement of clip with parameter-fie free attention. In Proceedings of the AAAI Conference Artificial Intelligence, volume 37, pp. 746-754, 2023.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked au- toencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 16000-16009, 2022.

---

Published as a conference paper at ICLR 2025
Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of- examples in neural networks. In International Conference on Learning Representations, 2016.
Dan Hendrycks, Mantas Mazeika, and Thomas Dietterich. Deep anomaly detection with outlier exposure. In International Conference on Learning Representations, 2018.
Paul F Jaeger, Carsten Tim Lüth, Lukas Klein, and Till J Bungert. A call reflect evaluation practices for failure detection in image classification. The Eleventh International Conference
Learning Reprexentations, 2022.
Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In European Conference on Computer Vision, pp. 709-727. Springer, 2022.
Xue Jiang, Feng Liu, Zhen Fang, Hong Chen, Tongliang Liu, Feng Zheng, and Bo Han. Negative label guided ood detection with pretrained vision-language models. In The Twelfth International Conference Learning Representations, 2024.
Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven
Chu Hong Hoi. Align before fuse: Vision and language representation learning with momentum distillation. Advances in neural information processing systems, 34:9694-9705, 2021.
Wei Li, Linchao Zhu, Longyin Wen, and Yi Yang. Decap: Decoding clip latents for zero-shot captioning via text-only training. In The Eleventh International Conference on Learning Repre- sentations, 2022a.
Yanghao Li, Hanzi Mao, Ross Girshick, and Kaiming He. Exploring plain vision transformer back- bones for object detection. In European Conference on Computer Vision, pp. 280-296. Springer, 2022b.
Weitang Liu, Xiaoyun Wang. John Owens, and Yixuan Li. Energy-based tion. Advances in neural information processing systems, 33:21464-2144 2020.
detec-
Yangyang Liu, Chen, Fei Yin, and Cheng- Liu. Caate confidence estimation for rejec- tion in handwritten chinese text recognition. In International Conference on Document Analysis and Recognition, pp. 134 151. Springer, 2024.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference computer vision, pp. 10012-10022, 2021.
Ze Liu, Han Hu, Yutong Lin, Zhuliang Yao, Zhenda Xie, Yixuan Wei, Jia Ning, Yue Cao, Zheng Zhang, Li Dong, et al. Swin transformer v2: Scaling up capacity and resolution. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 12009-12019, 2022.
Eric Lopez-Lopez, Xose M Pardo, and Carlos V Regueiro. Incremental learning from low-lebelled stream data open-set video face recognition. Pattern Recognition, 131:108885, 2022.
Depu Meng, Xiaokang Chen, Zejia Fan, Gang Zeng, Houqiang Li, Yuhui Yuan, Lei Sun, and Jing- dong Wang. Conditional detr for fast training convergence. In Proceedings of the IEEF/CVF International Conference Computer Vision, pp. 3651-3660, 2021.
Yifei Ming, Ziyang Cai, Jiuxiang Gu, Yiyou Sun, Wei Li, and Yixuan Li. Delving into out-of- distribution detection with rision-languaee representations. Advances in Neural Information Pro- cessing Systems, 35:35087-35102 2022.
Atsuyuki Miyai, Qing Yu, Go Irie, and Kiyoharu Aizawa. Locoop: Few-shot detection via prompt learning. arXiv preprint arXiv:2306.01293, 2023a.
Atsuyuki Miyai, Qing Yu, Go Irie, and Kiyoharu Aizawa. Zero-shot in-distribution detection in multi-object settings using langunel foundation models. arXiv preprint arXiv:2304.0452.1 2023b.
12

---

Published as a conference paper at ICLR 2025
Sangha Park, Jisoo Mok, Dahuin Jung, Saehyung Lee, and Sungroh Yoon. On the powerfulness of textual outlier exposure for visual ood detection. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp- 8748-8763. PMLR, 2021.
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp- 2556-2565. 2018.
Yang Shu, Xingzhuo Guo, Jialong Wu, Ximei Wang, Jianmin Wang, and Mingsheng Long. Clipood: Generalizing clip to
2023.
Robin Strudel, Ricardo Garcia, Ivan Laptev, and Cordelia Schmid. Segmenter: Transformer for semantic segmentation. In Proceedings of the IEEE/CVF international conference computer vision, pp. 7262-7272, 2021.
Leitian Tao, Xuefeng Du, Jenry Zhu, and Yixuan Li. Non-parametric outlier synthesis. In The Eleventh International Conference on Learning Representations, 2022.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International conference on machine learning, pp. 10347-10357. PMLR, 2021.
Vishaal Udandarao, Ankush Gupta, and Samuel Albanie. Sus-x: Training-free name-only transfer of vision-language models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2725-2736, 2023.
Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. The inaturalist species classification and detection dataset. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 8779-8778, 2018.
Haoqi Wang, Zhizhong Li, Litong Feng, and Wayne Zhang. Vim: Out-of-distribution with virtual- logit matching. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 4921-4933 2022.
Hualiang Wang, Yi Li, Huifeng Yao, and Xiaomeng Li. Clipn for zero-shot ood detection: Teaching clip say In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 1802-1812, 2023.
Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. Proceedings of the IEEE/CVF international conference on computer vision, pp. 568-578, 2021.
Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In 2010 IEEE computer society conference on computer vision and pattern recognition, pp. 3485-3492. IEEE, 2010.
Hu Xu, Gargi Ghosh, Yao Huang, Dmytro Okhonko, Armen Aghajanyan, Florian Metze, Luke Zettlemoyer, and Christoph Feichtenhofer. Videoclip: Contrastive pre-training for zero-shot video-text understanding. In Proceedings of the 2021 Conference on Empirical Methods Nat- ural Lunguage Processing, pp. 6787-6800, 2021.
Fanhu Zeng and Deli Yu. M2m-tag: Training-fre free many-to-man -o-mmayy token aggregation for vision transformer acceleration. In Workshop Machine Learning and Compression, NeurlPS 2024.
13

---

Published as a conference paper at ICLR 2025
Fanhu Zeng, Fei Zhu, Haiyang Guo, Ya-Yao Zhang, and Cheng-Lin Liu. Modalprompt: Dual- modality guided prompt for continual learning of large multimodal models. arXiv preprint arXiv:2410.05889, 2024.
Ji Zhang, Lianli Gao, Bingguang Hao, Hao Huang, Jingkuan Song, and Hengtao Shen. From global to local: Multi-scale cut-of-distribution detection. IEEE Transactions on Image Processing, 2023.
Renrui Zhang, Rongyao Fang. Wei Zhang, Peng Gao, Kunchang Li, Jifeng Dai, Yu Qiao, and Hong- sheng Li. Tip-adapter: Training-fre free clip-adapter for better vision-language modeling. arXiv preprint urXiv:2111.03930. 2021.
Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. JEEE transactions on pattern analysis and machine intelligence, 40(6):1452 1464, 2017.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16816-16825, 2022a.
Kaiyang Zhou, Jingkang Yang. Chen Change Loy, and Ziwei Liu. Learning to prompt for vision- language models. International Journal of Computer Vision, 130(9):2337-2348 2022b.
Fei Zhu, Zhen Cheng, Xu- Yao Zhang, and Cheng-Lin Liu. Openmix: Exploring outlier samples for miselassification detection. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 12074-12083, 2023a.
Fei Zhu, Xu-Yao Zhang, Zhen Cheng, and Cheng- Liu. Revisiting confidence estimation: To- wards reliable failure prediction. IEEE Transactions on Pattern Analysis and Machine Intelli- gence, 2023b.
Xiangyang Zhu, Renrui Zhang, Bowei He, Aojun Zhou, Dong Wang, Bin Zhao, and Peng Gao. Not all features matter: Enhancing few-shot clip with adaptive prior refinement. arXiv preprint 25X 2023c.
14

---

Published as a conference paper at ICLR 2025
APPENDIX
A DATASET DETAILS
ImageNet-10 contains 10 classes of ImageNet-IK, Ik, including military aircraft (n04552348), sports car (n04285008), brambling (n01530575). Siamese cat (n02123597), impala (n02422699), Greater Swiss Mountain Dog (n02107574), American bullfrog (M01641577), garhage truck (n03417042), common sorrel horse (n02389026) and container ship (n03095699).
ImageNet-200 is 20-class subset in ImageNet-1k. Concretely, they are schooner (n04147183), canoe (002951158)) balloon (n02782093), tank (n04389033), missile (n03773504), high-speed train (n02917067)) starfish (n02317335), spotted salamander (n01632458), smooth newt (n0163067 0), newt 6663),,,,, zebra (n02391044) 1049), European green lizard (n01693334). Nile crocodile (n01 697457), Arctic fox (n02120079)) grey wolf (n021 00021,4367, brown bear (n02132136), moped (n0378 5016), steam locomotive (n04310018), space shuttle (n04266014) and snowmobile (n04252077).
ImageNet- is the same 100-class subset of ImageNet- k MCM (Ming al., 2022) to have fair comparison with previous vision-language based OOD detection methods. Detail categories are provided in https:/ggihub.com//deellaamii -wic//MMM.
B TRAINING DETAILS
Number of learnable context words is 16. We use SGD optimizer to train the model with 30 epochs. Learning rate is 10-3 with cosine schedule and batch size 256. m set to 24 by default. We empirically set positive/negative augmentation to be 8/1, which are discussed thoroughly below. Training and testing are 50 and 10, respectively in main results. We use one single NVIDIA A6000 to run all experiments. The reported results are averaged for three runs. Extra experimental results are shown below.
C MORE EXPERIMENTAL RESULTS
Experiments of different shots. We use ImageNet-Ik ID dataset in the main experiment. For training, we use different shots, i.e., numbers of images from each class, following few-shot setting. For evaluation, we use validation set of ImageNet-1k, 1k, which is composed of 50000 images from 1000 categories. Results are shown in Tab. 9.
Influence coefficient. Both Aneg and Areg determine the importance of the corresponding loss. We report average FPR95 and AUROC to show the trend when the coefficients vary. The results in Tab. 10 indicate that the performance tends to be stable when the coefficients are wide range. During the entire ablation process, the results are relatively robust when the ratio of the two corre- sponding coefficients is approximately within [2, 10), with performance drop under extreme cases (Aneg 0.1 and 50). For Aneg 50, the loss for local prompts is neglected, resulting in severe
Table More few-shot experimental results on ImageNet- 1k.
INaturalist
SUN
Places
Texture
Average
Method
AUROC1
FPR95
AUROC:
FPR95L
AUROCT
AUROC1
FPR951
AUROCT
4-shot
CoOp
18.95
95.52
29.58
92.90
38.72
89.64
48.03
85.87
33.82
90.98
LoCoOp
21.67
95.69
22.98
95.07
31.41
92.10
49.79
87.85
31.46
92.68
Local-Prompt
97.87
20.40
95.57
29.39
92.67
51.20
88.00
27.66
93.53
8-shot
CoOp
15.23
96.69
27.78
93.08
35.93
90.22
48.26
85.91
31.80
91.47
LoCoOp
16.34
96.47
22.40
94.96
31.86
91.83
42.20
89.81
28.20
93.27
Local-Prompt
10.17
97.83
19.82
95.42
30.83
92.54
48.26
88.47
27.27
93.56
16-shot
CoOp
14.60
96.62
28.48
92.65
36.49
89.98
43.13
88.03
30.67
91.82
LoCoOp
16.05
96.86
23.44
95.07
32.87
91.98
42.28
90.19
28.66
93.52
Local-Prompt
8.71
10
23.97
94.85
32.50
92.32
47.93
89.04
28.27
93.58
15

---

Published as a conference paper at ICLR 2025
Table 10: Influence of different coefficient in loss function.
iNaturalist
SUN
Places
Texture
Average
FPRO5]
AUROCT
FPR9S]
AUROCT
FPR99
AUROCT
FPR95]
AUROCT
FPR9S]
AUROCT
17.64
95.97
27.18
94.76
34.45
91.41
48.82
32.02
92.25
2
9.47
97.97
24.11
94.99
32.28
92.07
47.94
88.86
28.45
93.47
Anea
9.39
97.97
24.20
95.10
32.13
92.21
49.37
88.66
28.77
93.48
10.19
97.83
23.46
95,26
31,59
92.31
49.57
88.81
28.70
93.55
50
34.09
92.06
42.34
88.63
50.88
83.52
62.47
85.70
47.44
87.48
0.2
9.83
97.88
24.60
94.92
32.65
92.10
49.82
88.63
29.22
93.38
Avek
97.97
95.10
32.13
49.37
28.77
11.24
97.67
25.87
94.77
33.54
91.89
48.38
88.97
29.75
93.32
Training k=
29.5
Training
93.5
Training k-100
29.0
93.4
19999
AUROC
28.5
93.3
Training K-20
Training k-50
Training
27.5
93.1
Testing
Test
Figure 7: Results on different values of k for training and inference.
performance drop. For Aneg 0.1, the local negative prompts are over-regularized by diversity constraints. The effectiveness of the proposed loss can be further verified accordingly.
Regional number. Regional number & expresses the extent to which local prompts pay attention to areas. As regional information not always ID or OOD-related, the selection of regional number k largely determines performance. During training, Regions are selected emphasize suitable ID- related regions (outlier regions when hard negative images) and learn regional- -elated local prompts. During inference, A is relatively smaller guarantee the confidence of the selected region for met- ric calculation. We conduct various experiments on different values & during both training and testing. Curves shown in Fig. 7 illustrate that for both during training and inference, larger will confuse confidence of the boundary between ID and OOD regions, while smaller k fails to extract precise regional-related textual representations, which corresponds to our analysis. We underline that varying & brings in subtle differences and even simplified outcome without careful selection of still achieves better results than previous approaches.
We extend the training time to analyse the benefits of longer training. It can be seen Tab. 11 that epochs is sufficient to get superior representations few-shot prompt tuning and longer training time does not necessarily bring about the improvements of performance. By contrast, LoCoOp (Miyai et al., 2023a) requires 50 epochs and indicates that our approach is well and fast converged, which is in line with the purpose of few-shot learning.
Extra training time.
Number of negative augmentation. The number of negative augmentation influences the perfor- mance. It can be in Tab. 12 that setting negative augmentation to be achieves obviously better results. A potential reason for the phenomenon is that random crops of original images are hard enough for the detector to learn local outlier information and much negative augmentation may confuse the learning process. Consequently, we choose the least similar image as the negative augmentation.
Table 11: Influence of longer training time.
iNaturalist
SUN
Places
Texture
Average
epoch
FPR95
AUROCT
FPR95L
AUROCT
EPR95L
AUROCT
FPR95L
AUROCT
FPR95
AUROCT
30
9.39
97.97
24.20
95.10
32.13
92.21
49.37
88.66
28.77
93.48
50
97.83
23.71
95.05
32.31
92.02
48.47
88.68
28.52
93.40
16

---

Published as a conference paper at ICLR 2025
Table 12: Influence of negative augmentation number.
INaturalist
SUN
Places
Texture
Average
N
FPR95]
AUROCT
FPR95
AUROCT
FPR95]
AUROCT
FPR95]
AUROCT
FPR9S]
AUROCT
9.39
97.97
24.20
95.10
32.13
92.21
49.37
88.66
28.77
93,48
10.31
97.80
25.18
94.96
33.36
92.06
51.32
87.64
30.04
93.12
Table 13: Comparison with ID-like.
iNaturalist
SUN
Places
Texture
Average
Method
FP8951
AUROC1
BPR95L
AUROCT
HPR9SL
AURUCT
HPR55,
AUROCT
BPRMS
AUROCT
ID-like
8.98
98.19
42.03
91.64
44,00
90.57
25.27
94.32
30.07
93.68
Ours
12.81
97.29
19.34
95.85
27.53
92.97
45.51
89.99
26.29
94.03
Comparison with ID-like. We additionally compare with ID-like (Bai et al., 2024). and the re- sults shown in Tab. 13 reveal that our model achieves better average performance against ID-like, demonstrating the utility of our local information enhancement strategy.
F9R9
FPR: 19,445
FPR:B660%
INaturalis
100
4000
200
40011
5090
FF9-40.57%
1ER-3150%
OOD
Tasturr
4100
CIOT
HPR:37.5%%
FIPEZZI 653%
SUN
69R-44.2%%
FPR:31.18%
19R:31.4%%
Placss
MCM
LoCoOp
Ours
Figure 8: Density map of different OOD datasets measured by various methods.
D VISUALIZATION AND ANALYSIS
Visualization of heatmaps. we additionally include heatmaps maps of typical images to illustrate the attention of our methods. It can be seen Fig. 9 that our method accurately focuses on ID regions despite the interference of complex background surroundings, e.g., tabby cat, lion and leonberg with complex background of grassland, which showcases the effectiveness of pseudo background aug- mentation serving local outlier. Moreover, method also captures multi-objects, e.g., the ships on the sea (third picture on the first row). strongly validating the usefulness of local enhancement in the framework.
17

---

Published as a conference paper at ICLR 2025
ID: tabby cat
ID: fountain pen
ID: liner (ship)
ID: lion
ID: leonberg (dog)
ID: minivan (car)
Figure 9: Visualization of heatmaps.
Table 14: The top-k average similarity of both ID-dataset and OOD datasets with local prompts.
10
20
50
w/o local enhancereett
ID-dataset
0.297
0.286
0.272
OOD-datascts
0.280
0.276
0.261
W local enhancement
ID DD-dataset
0.312
0.306
0.282
OOD-datasets 0.273
0.266
More visualization. We visualize the density map of 4 OOD datasets, i.e., iNaturalist, SUN, Places and Texture and compare them with typical existing methods. Fig. 8 exhibits that our approach is capable of distinguishing IDD and OOD samples with more confidence and has obviously less overlap between them than previous zero-shot and few-shot methods. It strongly demonstrates the effectiveness and potential of our local prompt based method.
Similarity of enhanced regions. We calculate the top-k average similarity of both ID-dataset and OOD datasets with local prompts qualitatively showcase the effectiveness of local enhancement. It can be clearly seen in Tab. 14 that the method without local enhancement has no obvious boundary between and OOD datasets (0.272 for top-50 and 0.276 for top-20). which may confuse the model when detecting hard outlier samples where subtle difference occurs in certain regions of the image. By contrast, our method appears to be more discriminant and has an obvious boundary between them (0.282 for ID and 0.273 for OOD). It additionally demonstrates the core problem that current method encounters and our method refining local regions successfully addresses the issue to some extent.
E LIMITATIONS AND FUTURE WORK
Training time. The primary limitation of our work is that it may require more training time due to negative augmentation. However, as few shot tuning paradigm for vision-language model is efficient and time-saving, it will not be the bottleneck of the model. Practically, fine-tune CLIP one single NVIDIA A6000 and 4-shot tuning only needs hours.
Joint training of both global and local prompts. In work, mainly concentrate the effec- tiveness of local prompts as an enhancement and set global prompts frozen/from existing methods. We believe that exploiting the connection between global and local features is promising. Therefore, we are in an effort to explore the advantage of training global and local prompts jointly.
Expansion to other recognition tasks. In this work, we focus on OOD detection for classification task. It will be an interesting topic to handle other vision tasks in open-world such as segmentation and object detection. However, existing methods also merely deal with one task at a time. Never- theless, we believe the proposed local prompts can be generally useful as long as the applied tasks have rich local information.
18


Reviews:
[{"review_id": "3gHIqBeQRh", "reviewer": "Reviewer_dbE1", "summary": "The proposed work focuses on enhancing the model's ability to detect local features by training local feature prompts using localized image information. Furthermore, based on these local feature prompts, this paper introduce a new OOD score that integrates with MCM Score.", "questions": "Please refer to the weakness.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1.\tLocal image information is utilized to train local feature prompts, enhancing the model's ability to detect local features.\n\n2.\tBased on local feature prompts, an OOD metric combining MCM has been proposed, along with a new ID classification metric.", "weaknesses": "1.\tIn the latest works related to OOD detection, there are several studies similar to the method in this paper, such as \"ID-like (ID-like Prompt Learning for Few-Shot Out-of-Distribution Detection)\" and \"NegLabel (NEGATIVE LABEL GUIDED OOD DETECTION WITH PRETRAINED VISION-LANGUAGE MODELS).\" The structure of this paper is similar to that of ID-like, as it also employs random crop combined with CLIP to construct ID and OOD data, and utilizes positive prompts and negative prompts. However, the paper lacks a comparison and analysis of strengths and weaknesses with the ID-like paper. It is recommended that the authors add relevant discussions to enhance the depth and breadth of the paper.\n\n2.\tThe OOD Score strategy proposed in this paper presents results for SMCM and SR-MCM but lacks results that only use the Regional OOD score and corresponding ablation experiments. This makes it difficult to effectively demonstrate that the SR-MCM outperforms the strategy that solely relies on the Regional OOD score. It is suggested that the authors include this part of the experiment to strengthen the paper's persuasiveness and the comparability of the results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uXDgXiBKBJ", "reviewer": "Reviewer_NiJR", "summary": "The paper presents a novel approach to few-shot out-of-distribution (OOD) detection. The aim of this paper is at addressing a limitation of existing few-shot OOD detection methods, which predominantly rely on global prompts and struggle to identify challenging OOD images that differ only locally from in-distribution (ID) images. \n\nTo enhance detection accuracy for these difficult cases, the proposed method introduces learnable local prompts. Specifically, the proposed method first picks out multiple randomly cropped regions from the original image and classifies them as positive or negative based on their similarity to the global text embedding. These regions are then used as tarining exampes to learn the positive and negative learnable local  prompts.\n\nExperimental results on several popular benchmark datasets for few-shot OOD detection show that the proposed method can achieve better accuracy than several existing few-shot OOD detection methods.", "questions": "As noted in the Weaknesses section, I believe the experiments have several significant shortcomings. While all of these issues are important, I have listed them in descending order of priority. The first four points (**W1-W4**) directly concern the effectiveness of the proposed method and the motivation of this paper. I would like to encourage the authors to address these points in their response.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This paper proposes a novel idea of learning local prompts, which has not been well explored in the context of few-shot OOD detection.\n\n- Experimental results reported in this paper show that the proposed method outperforms existing methods.\n\n- This paper is well-organized and easy to follow.", "weaknesses": "My main concerns are about the experimental results.\n\n**W1**. Some of the results, such as AUROC in Table 1, show very small performance differences (less than 1%) between the proposed method and the existing methods. Since the proposed method uses random cropping to learn local prompts, the standard deviation of accuracy should be reported to show the significance of the performance differences.\n\n**W2**. Looking at Fig. 4, the accuracy differences between Ours w/ LNP and Ours w/o LNP appear to be quite small. LNP is one of the core ideas of the proposed method. If the accuracy differences are small, the superiority of the proposed method would be questionable.\n\n**W3**. As stated in the second paragraph of the introduction section, the aim of this paper is to detect hard OOD samples that are similar to the ID classes as a whole, but can be distinguished only by looking at subtle local differences. However, there is no support in this paper for whether the proposed method can actually detect such a hard OOD sample. It would be good to show some examples of OOD images that could not be detected with the global prompts alone, but could be detected with the introduction of the local prompts.\n\n**W4**. Based on the results in Table 9, the authors claim that the accuracy of the proposed method is not sensitive to the values of the two hyperparameters $\\lambda_{\\text{neg}}$ and $\\lambda_{\\text{reg}}$. However, this claim seems to contradict the effectiveness of the proposed method since the learning of the local prompt, which is the core of the proposed method, is controlled by these two hyperparameters (see Eq. 8).\n\n**W5**. Table 8 shows that the FPR95 values change nonlinearly with the number of shots; the FPR95 values improve as the number of shots increases from 4-shot to 8-shot, but deteriorate as it increases to 16-shot. What is the reason for this?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sBM6e4NVv9", "reviewer": "Reviewer_KX2t", "summary": "This work proposes to enhance Out-of-Distribution (OOD) detection through local prompt learning. Specifically, the method involves randomly cropping training images and identifying hard negatives by comparing their similarity to global prompts, which are provided in textual format. These hard negatives, along with the associated object samples, are then used to train local negative prompts.", "questions": "1.  How are the local prompts and negative local prompts initialized, and what are their dimensions?\n2.  Moreover, from lines 270 and 237, it appears that global and local prompts t_k share the same format, \"a photo of {class}.\" However, Figure 2 indicates that local prompts are trainable, which seems contradictory. Could the authors provide further clarification? Are the local prompts in Equation 4 simply using the same symbol t_k but representing different meanings? If so, using distinct symbols might be more appropriate.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well-written overall, but it lacks some crucial symbol definitions and implementation details.\n2. The method addresses a significant task aimed at detecting OOD images using only a few images from the ID data. Extensive experiments demonstrate the effectiveness of the method.", "weaknesses": "1. Some crucial symbol definitions, such as T_k and ​\\hat{t}_i , are not clearly explained.\n2. Apart from replacing the maximum process with an average in Equation 3, what distinguishes the proposed method from [Miyai et al. 2023b]?\n3. Although hard negative samples can serve as a form of OOD data, they primarily consist of cluttered backgrounds or parts of in-distribution (ID) objects lacking distinctive features. Essentially, they are quite different from true OOD classes. The work lacks an explanation of the underlying theories.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TTFMhFlkq5", "reviewer": "Reviewer_GNZR", "summary": "The authors propose an approach to Out-of-Distribution (OOD) detection in Vision Language Models through few-shot fine-tuning. The paper uses random crops from the images to generate negative global prompts, which are used to guide the learning of learnable local prompts aimed at detecting local or regional outliers. They also propose a metric for quantifying the OOD scores that takes into account local or regional information.", "questions": "I kindly request that the authors clarify the motivation behind few-shot tuning and address the discrepancy between few-shot and many-shot performance in Table 1, as highlighted in the weaknesses section. I believe an ablation study on the impact of additional negative local prompts is necessary to justify their use. Additionally, I would appreciate it if the authors could provide a clearer explanation of the image in Figure 2.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "I think the paper sheds light on an interesting perspective regarding the role of local or regional features in OOD detection. The idea of randomly cropping the image and creating negative global samples based on image-text similarity scores, while simple, is both effective and clever. The authors have empirically demonstrated the effectiveness of learning local prompts, achieving strong performance.", "weaknesses": "I am unsure about the justification for using few-shot fine-tuning. I do not understand what the authors meant in lines 46-48. Having access to the full dataset for a class can provide a better estimate of the outliers. Identifying outliers with only a few examples, while challenging, may be biased toward the examples of the class being used, especially if the number of shots is low and one of the examples is an outlier. The reason I am focusing on this is that the results in Table 1 seem counterintuitive to me. It appears that OOD detection worsens with 16 shots on the SUN and Places datasets, where we would expect the model to identify OODs better with more examples.\n\nThere also seems to be a lack of ablation studies regarding the model's performance without the learnable negative local prompts. I believe this ablation is important.\n\nI have a comment about Figure 2. It mentions local features of augmented inputs, which I believe refers to the image patches from the randomly cropped images. However, from the figure, it appears that they are only taking patches from the entire image, which is confusing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)