AdaNeg: Adaptive Negative Proxy Guided OOD Detection with Vision-Language Models
Authors: Yabin Zhang
Lei Zhang
Venue: NeurIPS 2024 poster (2024)
Keywords: Adaptive negative proxy
OOD detection
vision-language models
Primary area: safety_in_machine_learning
TL;DR: We propose adaptive negative proxies by exploring potential OOD images during testing for OOD detection

Abstract: Recent research has shown that pre-trained vision-language models are effective at identifying out-of-distribution (OOD) samples by using negative labels as guidance. However, employing consistent negative labels across different OOD datasets often results in semantic misalignments, as these text labels may not accurately reflect the actual space of OOD images. To overcome this issue, we introduce \textit{adaptive negative proxies}, which are dynamically generated during testing by exploring actual OOD images, to align more closely with the underlying OOD label space and enhance the efficacy of negative proxy guidance. Specifically, our approach utilizes a feature memory bank to selectively cache discriminative features from test images, representing the targeted OOD distribution. This facilitates the creation of proxies that can better align with specific OOD datasets. While task-adaptive proxies average features to reflect the unique characteristics of each dataset, the sample-adaptive proxies weight features based on their similarity to individual test samples, exploring detailed sample-level nuances. The final score for identifying OOD samples integrates static negative labels with our proposed adaptive proxies, effectively combining textual and visual knowledge for enhanced performance. Our method is training-free and annotation-free, and it maintains fast testing speed. Extensive experiments across various benchmarks demonstrate the effectiveness of our approach, abbreviated as AdaNeg. Notably, on the large-scale ImageNet benchmark, our AdaNeg significantly outperforms existing methods, with a 2.45\% increase in AUROC and a 6.48\% reduction in FPR95. Codes are available at \url{https://github.com/YBZh/OpenOOD-VLM}.

AdaNeg: Adaptive Negative Proxy Guided OOD Detection with Vision-Language Models
Yabin Zhang
Lei Zhang"
The Hong Kong Polytechnic University csybzhang@comp. polyu.edi edu.h? .hk
The Hong Kong Polytechnic University cslzhang@comp. polyu. polyu. edu. .k
Abstract
Recent research has shown that pre-trained rision-language models are effective at identifying out-of-distribution (OOD) samples by using negative labels as guidance. However, employing consistent negative labels across different OOD datasets often results in semantic misalignments, these text labels may not accurately reflect the actual space of OOD images. To overcome this issue, we introduce adaptive negative proxies, which dynamically generated during testing by exploring actual OOD images, to align more closely with the underlying OOD label space and enhance the efficacy negative proxy guidance. Specifically, our approach utilizes a feature memory bank to selectively cache discriminative features from images, representing the targeted OOD distribution. This facilitates the creation of proxies that can better align with specific OOD datasets. While aask-adapivve proxies average features to reflect the unique characteristics of each dataset, the sample-adaptive proxies weight features based on their similarity to individual test samples, exploring detailed sample-level nuances. The final score for identifying OOD samples integrates static negative labels with our proposed adaptive proxies, effectively combining textual and visual knowledge for enhanced performance. Our method is training-free and annotation-free, and it maintains fast testing speed. Extensive experiments across various benchmarks demonstrate the effectiveness of our approach, abbreviated AdaNeg. Notably, the large-scale ImageNet benchmark, our AdaNeg significantly outperforms existing methods, with a 2.45% increase in AUROC and 6.48% reduetion in FPR95. Codes available
https: https:://gtthbbe com YBZh/ 100Zhh0pen00D-VV
1 Introduction
In real applications, artificial intelligence (AI) systems often encounter test samples of unknown classes, termed out-of-distribution (OOD) data. These OOD data often result in overly confident errors (52) [4], posing security threats. Therefore, accurately identifying OOD data is essential for ensuring the reliability and security of AI systems in open-world environments.
Traditional OOD detection methods in image domain primarily rely on vision-only models 20 33
36
Recent advancements in vision- langual models (VLMs) have demonstrated remarkable OOD detection performance by leveraging multi-modal knowledge [17 [5] 39]. Recently, NegLabel [2] explores negative labels by identifying text labels that semantically distant from the in-distribution (ID) labels. This method achieves state-of-the-ar performance by detecting test images closer to negative labels OOD. In other words, Negl aabll regards these negative labels proxies of OOD data. However, employing consistent negative labels across different OOD datasets often leads to semantic misalignment, where these text labels may not accurately reflect the actual label space
*Corresponding Author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Neguabel Adalling
0GD
Adwliigg
Noglabel
120-
130-
Mean Value: 0.039
Mean Value: 0.073
60-
8259
(a) I-SNE Visualization
(b) Histogram of ISOR
Figure I: Qualitative and quantitative analyses of semantic misalignment between OOD labels and negative proxies using ImageNet (ID) and SUN (OOD) datasets. (a) Visualization of ID labels, OOD labels, negative labels from NegLabel, and adaptive negative proxies (AdaNeg). (b) Quantitative analysis based on ID-Similarity to OOD Ratio (ISOR in short. see Appendix A..I). Lower ISOR indicates higher similarity to OOD labels and reduced similarity to IDD labels. AdaNeg consistently achieves lower ISOR, demonstrating enhanced alignment with OOD characteristics. Visualizations include the top 1,000 discriminative proxies from both NegLabel and AdaNeg.
of OOD images, as shown in Fig. [ This misalignment between the proxies and targeted OOD distribution leads to sub-optimal performance.
To promote the alignment between the negative proxies and target OOD distribution, we introduce the Adaptive Negative proxies (AdaNeg), which are dynamically generated during testing by exploring actual OOD images. Specifically, we start by initializing an empty category-split memory bank for each OOD dataset and selectively cache features of discriminative OOD images during testing. The OOD discrimination is assessed using mined negative labels, as detailed in [27]. With this feature memory, develop task-adaptive proxies by simply averaging cached features within each category. These proxies, derived from actual OOD images, reflect the distinct characteristics of the target OOD dataset and align more closely with the underlying OOD label space.
The task-adaptive proxies mentioned previously provide unique proxies for different OOD datasets while maintaining consistency across various test samples within the same dataset. To delve into the fine-grained nuances at the sample level, we introduce the sample-adaptive proxies by weighting cached features based on their similarity to a particular test sample. This is achieved with an attention mechanism, where the feature memory serves as both keys and values, and the test feature acts as the query. The final score for detecting OOD samples integrates static negative labels with our adaptive proxies, effectively combining textual and visual knowledge for enhanced performance.
We conduct extensive experiments on standard benchmarks to validate the effectiveness of AdaNeg, where our proposed adaptive proxies outperform the negative-label-buseee one, enhancing performance with complementary multi-modal knowledge. Particularly, on the large-scale ImageNet dataset, our AdaNeg method outperforms existing methods by 2.45% AUROC and 6.48% FPR95. Notably, our method is training-free free and annotation- free, and it maintains fast testing speed, analyzed in Tab.
The ability to dynamically adjust to new OOD datasets without affecting testing speed or labor- intensive annotation/training makes our approach particularly valuable for real-world applications where adaptability and efficiency are crucial. We summarize our contribution as follows:
We first identify the label space misalignment between existing negative-label-bat proxies and the target OOD distributions. In response, we introduce adaptive negative proxies that are dynamically generated during testing by exploring actual OOD images, resulting in more effective alignment with the OOD label space.
Our adaptive negative proxies are constructed with feature memory bank that selectively caches discriminative image features during testing. We instantiate this concept by devel- oping task-adaptive proxies to reflect the unique characteristics of each OOD dataset and

---

sample-adaptive proxies to capture detailed sample-level nuances. The final OOD detection score combines these insights with complementary textual and visual knowledge.
We conduct thorough analyses of the proposed components and perform extensive exper- iments on standard benchmarks. Our method is training- free and annotation free, and maintains fast testing speed and achieves state-of-the-ie art performance. Notably, our method significantly outperforms existing methods, with a 2.54% increase in AUROC and a 6.48% reduction in FPR95 the large-scale ImageNet dataset.
2
Related Work
OOD Detection focuses on identifying OOD test samples with semantic shifts, thus distinguishing
it from generalization studies which typically focus covariate shifts [35 [5]. A variety of
OOD) detection techniques have been developed, which can be roughly categorized into score-based
20
33
36
37
66,
26 64 [56), distance-hased [58 59 57 12 41 [53, and generative-based [50) [2]
methods. Among them, score-based methods are particularly notable by employing a variety of
scoring functions differentiate between ID and OOD samples. These functions include confidence-
based 220 36 [56] 64), discriminator-bast
(29]. energy-based [37 66). and gradient-based (25)
scores. In contrast, distance-based methods determine OOD samples by evaluating the distance in the
feature space between test data and the closest ID samples (58) or ID prototypes (59), using metrics
such as KNN [57 12 or Mahalanobis distance
B3 53
Despite their achievements, traditional OOD detection methods generally rely on manually annotated ID images and often overlook the integration of textual information. To leverage the textual knowl-
edge, recent advancements have focused on employing VLMs [39 40 27
76 15
65 45)
Specifically, ZOC (15) applies VLMs to discern OOD instances by training a captioner that generates potential OOD labels. Nevertheless, this captioner often fails to produce effective OOD labels, particularly for ID datasets containing many classes. LoCoOp [42] adopts a novel approach by learning ID prompts from few-shot ID samples, and further enhances the robustness of these prompts by incorporating OOD features mined from the backgrounds of images. CLIPN 655. LSN 45 and LAPT (76 explore learning text prompts for expressing negative concepts. In specific, CLIPN initializes text prompts with the word 'no' combined with ID labels and refines them with large-scale multi-modal data; LSN starts with manually collected ID samples to learn negative prompts, offering different approach to leveraging textual information in OOD detection; LAPT conducts automated prompt tuning with automatically collected training samples, boosting OOD detection without any manual effort. MCM 39 utilizes ID class names to facilitate effective zero-shot OOD detection. is further refined by NegLabel (27). which incorporates additional negative class names mined from available data sources as negative proxies. However, as illustrated Fig. [ there is a mismatch be- tween the Segative-label-baseee proxies and the target OOD distribution, underscoring the limitations of this strategy. This observation has inspired us to construct adaptive proxies by exploring potential OOD test images during testing. This leads to efficient method that aligns better with the target OOD distribution, resulting in enhanced OOD detection performance.
Furthermore, we clarify the relationship between our method and existing approaches on OOD Most OOD exposure methods introduce manually collected negative images during training, where manual labor is necessary to ensure that the labels of negative images are different from ID ones. Moreover, involving negative images in training typically introduces additional computational overhead, impeding its practical deployment. Unlike these methods, NegLabel [27 is exposed to negative labels during the test phase in training-free manner. However, given fixed ID dataset, the exposed negative texts remain consistent for different OOD datasets, inevitably resulting in label misalignment, as shown in Fig. D To address this, we expose the VLMs to adaptive negative proxies, which explore actual OOD samples during testing and align more effectively with OOD distribution. Our method does not require manual annotations and works in a training- -free manner, making it an appealing solution for real applications.
exposure 17 21 7
Test-time Adaptation. We adopt an online update of the negative proxies during testing, resembling test-time adaptation (TTA) methods [35] 63 [54]. Existing TTA methods primarily address covariate shifts between training and testing domains. In contrast, our approach mitigates the label shift between negative proxies and the target OOD distribution by exploring online test samples. Recently, TTA strategies have been considered the field of OOD detection. However, these methods typically require test-time optimization [18] 72 16). slowing down the testing process. In contrast, our method
3

---

ID feature/proxy
Image Encoder
Cat
Negative feature/proxy
ID labels
Test Image
Selectively Caching
Empty slot
NegMining
G1
Text Encoder
C2
Scoring
C7 C3
C1
Weighting
Text-based Proxles
Adaptive Proxies
Sut
S
Memory
Frozen
Plant
Negative labels
Multi-modal Score
Update W/O Optimization
The overall framework of AdaNeg, where we selectively cache test images and generate adaptive proxies with an external feature memory bank. The final score combines textual and visual knowledge from static negative labels and our adaptive proxies, integrating multi-modal information.
Figure 2:
optimization-free and introduces only a lightweight memory interaction operation, enabling rapid and accurate testing, as analyzed in Tab. 4
Memory Networks. The use of memory networks for storing and retrieving past knowledge 67 55 has been extensively applied across various fields, including classification [28] 77 segmentation 46 69), detection (8]6), and NLP [47). To our best knowledge, our work is the first apply memory networks to the field of OOD detection. By caching and retrieving test images with feature memory, training-free
we propose adaptive proxies to more effectively align with the OOD distribution manner. This innovative approach significantly enhances the OOD detection performance.
3 Methodology
3.1 Preliminaries
OOD Detection Setup. Consider X' as the image domain and y = {y1, fy/1...., yc } as the space of ID class labels, where y' comprises text elements such as y = fcat, dog dog..... ...., bird), and C' represents the total number of classes. Let ah and pod be random variables representing ID) and OOD samples from X, respectively. We define Px and Pxo as the marginal distributions for and OOD, respectively. In conventional classification scenarios, it is assumed that the test image originates from the ID and is associated with a specific ID label, specifically, æ E Px and y E V, with y being the label of æ. However, in real applications, AI systems often face data from unknown classes, denoted by a E ood and é ) Such occurrences can make AI models incorrectly categorize these instances into familiar ID categories with substantial certainty [52 44 resulting in security concerns. To address these challenges, OOD detection is proposed to accurately categorize ID samples into their respective classes and reject OOD samples non-ID. Recognition within the ID categories is performed using a C-way classifier, following standard classification approaches
31 19 Concurrently, OOD detection typically employs scoring mechanism [33 36 37 to
differentiate between ID and OOD inputs:
G. ID, if S(x) 2 otherwise, G, (2) = OOD,
(1)
where G., represents the OOD detector set at a threshold E R. The test sample æ is identified as an ID sample if and only if 5(x) 2 7.
CLIP and Labla For an ID test image æ within the label space y, we derive the image feature vector v = fimg(1) (1) E RD and the text feature matrix Cia = Jtseep(( (p())) E RCXD using pre-trained CLIP encoders, where D represents the feature dimension. The functions Smmg() (.) and ftz((-) are encoders for images and text, respectively. The function p(-) is the text prompt mechanism, typically defined as 'a photo a <label>.', where label is the actual class name, for example, 'cat' or 'dog' Both and Cia undergo L2 normalization across the dimension D. Then, zero-shot classification probabilities are computed utilizing as the classifier:
p Softmax(vCl., (//T) R.
(2)

---

where 7 > O is the scaling temperature.
The vanilla CLIP is proposed for zero-shot ID recognition and has recently been extended to OOD detection. Specifically, the NegL.bbe approach (27) introduces negative class names - = (VC+1,--- 3C+MM sourced from broad text corpora, where M is the length of negative classes and (=nu 0. Then, we can obtain the full text feature matrix C Jt J (p() uy-) R(C+M)) D with the pre-trained CLIP text encoder, leading to the classification probability across C' + M classes:
p = Softmax Assuming that ID samples exhibit greater similarity to ID labels and lesser similarity to negative labels compared to OOD samples, NegLabel introduces the following score for OOD detection:
E RC+MM.
(
Σ Ps.
Ps.
where P; is the i-th entry of p. indicating the classification probability of the i-th class. Intuitively, the NegLabel method uses negative labels proxies of the OOD distribution, detecting OOD images based on the similarity to these negative labels.
3.2 AdaNeg
Although NegLabel has successfully employed negative labels as the OOD proxies, there is a semantic misalignment between such OOD proxies and actual OOD labels, as illustrated in Fig. [ We aim to obtain OOD proxies that align better to the targeted OOD distribution. However, acquiring such OOD proxies is challenging, the OOD information is unknown prior to actual testing.
From another perspective, we can access real OOD information during testing, motivating us to refine OOD proxies in the testing stage. We can identify discriminative negative images during testing and then adjust the OOD proxies toward detected images. This is achieved by selectively eaching features of test images into a task-aware memory bank, followed by memory reading operations to produce adaptive proxies. We detail our implementation as follows.
Task-aware Memory. We construct task-aware memory as a category-split tensor M E R(C+M)XLXD, where L is the memory length for each category. M is initialized with zero values and gradually filled with features of selected images during testing. Specifically, for test image with feature v, we first calculate its score Snt(v) with Eq. (4) If (T)(v) < 7, we detect this test point as a negative image, and otherwise, it is identified as a positive sample. For negative and positive images, we respectively identify their closest labels as:
Negative: y arg max podd + +C,
(5)
sd Positive y = arg max p.
(6)
where pood = p|C C + M] and pdd =p/0: CJ are the classification probabilities corresponding to negative and ID class names, respectively. Then, we cache this feature v into the task-aware memory:
M =
(7)
where indicates an empty slot of M, E RL Once M,, is filled, we drop the image feature with the highest prediction entropy, as detailed in Appendix A.2 In one word, we keep confident image features with low prediction entropy in the memory.
The aforementioned strategy attempts to cache all test images, including those with high confusion between ID and OOD. However, we found that selectively retaining only those image features that exhibit strong ID/OOD distinguishing capabilities can effectively reduce this confusion. Specifically, we modify the selection criterion for memorization as follows:
Negative: : (w) < - (T) < 7 - 97,
Positive (v) 2 7 + (w) 2 + g(1 - 7),,
(8)
where y E [0, 1] is hyperparameter that introduces gap in the score space. Consequently, image features falling within the gap 7 g7 S (1) < 7 + g(1 - 7) are considered to have low distinguishing confidence and are not cached.
5

---

Algorithm 1 Adaptive Negative Proxy Guided OOD Detection
Require: ID label space y and test set X'
1: y <- NegMine())) following (27)
2: Constructing an empty memory M
3: for x E X do
4: Generating detection score with negative labels using Eq.
5: Determine whether a: should be cached using Eq.
6: Caching z with Eq. [ needed
7: Generating adaptive proxies with memory bank using Eq. 9 or Eq.
8: Generating adaptive scores with Eq. [0)or Eq. [2
9: Generating final score San by merging multi-modal knowledge with Eq. 13
10: end for
11: Return Collected final scores (Sarr
Task Vask-adeptive Proxies. Given the updated M, we can easily get the task-adaptive proxy by averag- ing along the length dimension L:
L+1
C =
1
Ma
E RR(+MM)DD
L+1
where L2(+) indicates the L2 normalization along feature dimension D, and M [M,C] R(C+M)x( is the extended memory with vanilla text proxies C. Such memory extension is necessary since M is initially empty and uninformative. Initially, this extension initializes the adaptive proxies t with the basic text proxies C. However, there are two key distinctions between the adaptive t and the vanilla C. First, unlike the NegLabel approach, which employs fixed proxy C' across various OOD datasets, our C dynamically adjusts the targeted OOD domain as the memory accumulates data, thereby providing dataset-specins adaptive proxies. Second, the C primarily incorporates image features, offering modal knowledge that complements the text-based proxies C. The benefits of this approach are further analyzed in Tab.
The score function for OOD detection with the adakttdaptive proxy C is derived as:
E cooe(v,c("//T
Sta(v) = EE @cos(,,d"")/= Ej=ct1 cos(e,ef")//"
(10)
where cos(:.) measures the cosine similarity, and cea is the i-th entry of C
Sample-adaptive Proxies. As evidenced in Table 3 our task-adaptive proxies significantly out- perform the fixed text proxies by effectively adapting to the characteristics of target OOD dataset. Building on this success, we further refine our approach by leveraging finer-grained, sample-level nuances to introduce even more effective hample-adaptiy proxies. Specifically, given the extended memory M and the test image feature via the following cross cross-attention operation:
introduce the sample-adaptive proxy C E (C+A5)
cieh = Le (MMM)) M,) E ERD
(11)
where v TM,,) II(((-1) measures the cosine similarities between normalized features of v and M ;, and p(x) exp(- 1ap(-5(1 .)) modulates the sharpness of with hyper-parament B3. cta and M are the i-th entry of S and M, respectively.
Both the task-adaptive and the sample-adaptive proxies are derived from the memorized image features stored in M. The primary distinction between them lies in their respective weighting strategies. For t each feature M in the memory is assigned a uniform weighting coefficient of L-I Conversely, in constructing C the weighting coefficient for each cached feature is dynamically determined based on its cosine similarity to the test image feature, denoted as v (M,)² Consequently, while cd remains constant across different test samples, 8 adapts to each individual test sample. This adaptability allows a to tailor its response based on the specific characteristics

---

of each test image, thereby enhancing discrimination between ID and OOD samples, particularly in diverse and variable testing scenarios.
The score function for OOD detection with the sample-adaptive proxies C is derived as:
Σ coo((,,(")/=
(w)) = E ct))(
(12)
CCM 2j=C+1 e coo((,,
Multi-modal Score. As previously discussed, the score function Snu(1) relies primarily on text features, whereas the sumple-adaptive score function (v) utilizes cached image features. Given the complementary nature of text and image modalities, we derive the final score function by integrating knowledge from both modalities:
Sau(v) = S.)(v) + (v),
(13)
where À >>0 0 is the hyperparameter balancing the two modalities. The overall pipeline of our method is illustrated in Fig. 2 and summarized in Algorithm
Experiments
4.1 Setup
Datasets. We conduct extensive experiments with the large-scale ImageNet-Ik 1k
9 as ID data.
Following prior practice 26, 39 27], four OOD datasets of iNaturalist [60], SUN 688, Places (88
and Textures 7 are evaluated. We also validate our method on the OpenOOD benchmark 774 71).
where OOD datasets grouped into near-OOD (e.g.. SSB-hard [62]. NINCO (2)) and for
(e.g., iNaturalist [60], Textures (7], OpenImage-O (64) according to their similarity to ImageNet dataset. Besides ImageNet, also evaluate our method on smaller-sized CIFAR 10/100 datasets (30
with the OpenOOD setup. Specifically, with the ID) dataset of CIFAR 1 00, we adopt near-OOD datasets of CIFAR 10010 and TIN (32), and far-ood datasets of MNIST [O, SVHN 43 Texture 77.
and Plances365 77 These experiments with various ID and OOD datasets enable a comprehensive evaluation on various OOD settings.
Implementation Details. We adopt the visual encoder VITB/16 pretrained by CLIP 48 and analyze more backbone architectures in Tab. AIl For parper-parameters. length 1=10, threshold 0=055 with the gap g=0.5 in Eq. 8 3=5.5 in Eq. II and A= in Eq. 13 in all experiments. These syper-parameters are carefully analyzed in Sec. 4.3 Following NegLabel, adopt the text prompt of 'The nice <label>.', set temperature ==0.01, and define the number M of negative labels as 10, 000 for the ImageNet dataset. For the CIFAR datasets, set the number M as 70, 000 since we find that a larger M leads to better results for CIFAR. Following common practice (26 39 27,, we employ the evaluation metrics of FPR95, AUROC, and ID ACC, which are detailed All experiments are conducted with a single Tesla V100 GPU.
adopt the memory
in Appendix A.3
4.2 Main Results
ImageNet Results with Four OOD Datasets. As illustrated in Tab. AdaNeg significantly outperforms existing griiiing-free methods and even surpasses approaches requiring additional training. Specifically, report traditional methods 220 36 37
25 64 57 B
59 by
CLIP-encoders with labeled training samples following 27 and report results of (45 42 27 34
from their original paper. Compared the closest competitor 27, our method achieves consistent and notable improvements, validating the advantage of the proposed adaptive proxies over the negative-label-base ones.
ImageNet Results with OpenOOD Setup. Our method is extensively evaluated against a range of OOD datasets in Tab. 2 on the OpenOOD benchmark. Competing methods that require training referred from OpenOOD. These methods utilize the full ImageNet training set and hold an unfair advantage zero-shot, training-fre free methods like ours. Our AdaNeg consistently outperforms its closest competitor (27] in both near-OOD and far-OOD scenarios. Additionally, AdaNeg not only enhances OOD detection capabilities but also improves ID classification, presenting higher robustness under diverse conditions.
Results CIFAR10/100 datasets are provided in Appendix A.5 where advantages still hold.
7

---

Table OOD detection results with ImageNet-Ik, where VITB/16 CLIP encoder adopted.
ODD datasets
Methods
INaturalist
Places
Textures
Average
AUROCT
FPRSE
AUROCT
FPR95
AUROCT
[PR95]
AUROC"
FPR95_
AUROCT
FPR95]
Methods requiring training for fine-tuning)
MSP
20
87.44
58.36
79.73
73.72
79.67
74.41
79.69
71.93
81.63
69.61
ODIN
36
94.65
30.22
87.17
54,04
85.54
55.06
87.85
51.67
88. 80
47.75
Energy
17
95.33
26.12
92.66
35.97
91.41
39.87
86.76
57.61
91.54 31.54
39.89
GoadNorm
72.56
81.50
72.86
82.00
73.70
80.41
70.26
79.36
72.35
80.82
VIM
64
93.16
32.19
87.19
54,01
83.75
60.67
87.18
53.94
87.82
50.20
94.52
29.17
92.67
VOS 18
94.62
28.99
92.57
36,88
91.23
38.39
86.33
61.02
91.19
41.32
NPOS
96.19
16.58
90.44
43.77
89.44
43.27
88.80
46.12
91
37.93
ZOC 15
86,09
$7.30
81.20
81.51
83.39
73.06
76.46
98.90
81.79
85.19
1,5N (45
95.83
21.56
94.35
26.32
91.25
34.48
90.42
38.54
92.96
30.22
CLIPN [83
95.27
23.94
93.93
26.17
92.28
33.45
31.10
LoCoOp
96.86
16.05
95.07
23.44
91.98
32.87
90.19
42.28
93.52
28.66
LAPT
99.63
96.01
19.12
92.01
33.01
91.06
NegPrompt
84
98.73
6.32
95.55
22.89
93.34
27.60
91.60
35.21
94.81
23.01
Zero Rero-Shot Training-free Methods
Mahalanobis
83
55.89
99.33
59.94
99.41
65 65.96
98.54
64.23
98.46
61.50
98.94
Energy 37
MCM 39
94.59
32,20
92.25
38.80
90.31
46.20
86.12
58.50
90.82
43.93
NegT
99.49
95.49
20.53
AdaNeg (Ours)
99.71
0.59
97.44
9.50
94.55
34.34
94.93
31,27
96.66
18.92
Table 2: OOD detection results on the OpenOOD benchmark, where ImageNet-IK is adopted as ID dataset. Full results available in Tab. A7
FPR95
AUROC
ACC 1 ID
Methods
Ncar-OOD Far-OOD
Near-OOD
Far-OOD
Methods requiring training (or fine-tuning)
GEN
38
78.97
90.98
81.59
Aug Mix
ReAct 56
79.94
93.70
77.63
RMDS
80.09
92.60
81.14
SCALE
70
81.36
96.53
76.18
AugMix
23
ASH
82.16
96.05
77.63
LAPT 766
58.94
24.86
82.63
94.26
67.86
Zero-shot Training-free Methods
MCM 39
79.02
68.54
60.11
84.77
66.28
NegLabel 27
69.45
23.73
75.18
94.85
66.82
AdaNeg (Ours)
67.51
17.31
76.70
96.43
67.13
4.3
Analyses and Discussions
Score Functions. As illustrated in Tab. 3 the adaptive score functions Sto and S.a consistently outperforms the fixed S.,, validating the advantage of adaptive proxies over fixed label proxies. The sample-adaptive score S.. slightly surpasses the task-adaptive one S.a. verifying usefulness of fine-grained sample characteristics. Combining adaptive image-based proxies and fixed text-based proxies leads to the best performance, proving their complementarity.
Table 3: OOD detection results with different score functions, where results are reported with ImageNet ID dataset under the OpenOOD setup.
Saa
Sea
Saa
Near-OOD AUROCT
Far-OOD AUROC
75.18
94.85
75.76
96.20
76.03
96.35
76.49
96.45
76.70
96.63

---

76.75
05255
76.0-
90.25
Near 000 AU000
76.0
76.8
98.4
Tex 1e0-000 ALROC
Nar 000 AUROC
76.55
98.00
25.5
99.75
266
26.3
95.73
25.0
74.5
Nee-COp FarOCD
90.50
762
Near 000
75.75
+ Far
- Neer-OID
95.50
74.0
95.00
76.0
FanOOD
7525
95.00
10
20
(a) Threshold 7
(b) Gap value
(c) Memory length L
Figure 3: Analyses on the
(a) threshold in Eq. (b) gap value g in Eq. and
(c) memory length L. on the ImageNet dataset under OpenOOD setting.
Table 4:
Analyses on the time complexity of our AdaNeg and its competitors. 'Training' measures the training time, and 'Param.' presents the number of learnable parameters. 'FPS' reflects the inference speed, measured with batch size of 256. Results are achieved with NVIDIA V100 GPU.
Methods
Training
FPS
Param.
FPR95 4
ZOC
15]
224
287
336M
85.19
CLIPN 66
>24h
605
37.8M
31.10
LoCoOp 42
9h
625
8K
28.66
MCM 39
625
43.93
NegLabel
27
592
25.40
AdaNeg (Ours)
476
18.92
Threshold y and gap Eq. 8 As illustrated Fig. employing a moderate threshold y (e.g., 0.4 < < 0.6) proves effective in distinguishing OOD samples across various scenarios. However, the efficacy of the gap parameter depends upon the specific characteristics of different OOD datasets. A larger g facilitates the identification of more confidently classified ID/OOD samples, thereby improving the detection of far-OOD samples. Conversely, smaller g is advantagcous for near-OOD detection it accommodates Low-confidence OOD samples, which typically more prevalent in near-OOD scenarios.
In scenarios where the OOD distribution is entirely unknown, we adopt a conservative approach by setting g to 0.5 all experiments. This balanced setting provides a robust baseline performance across variety of conditions. However, if prior knowledge regarding the difficulty levels of OOD datasets accessible, tailoring the hyperparameters- such as opting for a smaller g more challenging OOD contexts can yield enhanced detection performance.
ID-OOD imbalanced Test Data. To investigate the stability of our method with imbalanced ID-OOD test data, we construct test sets with various mixture ratios of ID and OOD samples. Specifically, we adopted the 1. 1.28M ImageNet training data as ID and randomly sampled 12.8K and 1.28K instances from the SUN OOD dataset to construct the ID:OOD ratios of 100:1 and 1000:1 settings, respectively. To construct the ID:OOD ratios of 1:100, 1:10, 1:1, and 10:1 settings, we randomly sampled 40K samples from the SUN dataset OOD and randomly sampled 400, 4K. 40K, and 400K instances from the ImageNet training data. shown Tab. S. our method outperforms Neg across wide range of mixture ratios (from 1:100 to 100:1), validating the robustness and reliability of our approach. However, unbalanced mixture ratios do pose a challenge to our method. Our approach performs the best in scenarios with balanced mixture of ID and OOD samples, reducing the FPR95 by 11.18%. As the mixture ratio becomes increasingly unbalanced, the improvement brought by our method gradually decreases. When the unbalanced ratio reaches 1000:1, our method shows some negative impact.
After more detailed analysis, we discovered that the fundamental issue stems from the increased proportion of misclassified samples in the memory due to the growing ID-OOD imbalance. To effectively address this problem, we refine the selection criterion for memorizing OOD samples by adaptively adjusting the gap g. refer to this adaptive gap strategy as AdaGap, which significantly improves the robustness of our method against ID-OOD imbalanced test data, as demonstrated in

---

Table 5: FPR95 (1) with different mixture ratios of ID and OOD samples.
ID:OOD Ratio
1:100
1:10
1:1
10:1
100:1
1K:1
NegLabel
22.42
21.11
20.99
20.92
21.48
23.69
AdaNeg
21.00
12.49
9.81
26.28 15.61
20.71
AdaNeg (With AdaGap)
20.50
12.22
9.73
12.98
15.61
18.43
Table 6: OOD detection results with BIMCV-COVIDI9+ adopted.
61 where a VITB/16 CLIP encoder is
OOD datasets
CT-SCAN
X-Ray-Bone
Average
Methods
AUROC1
FPR95
AUROC1
FPR95
AUROCT
FPR95
NegLabel
63.53
100
99.68
0.56
81.61
50,28
AdaNeg (Ours)
93.48
100
99.99
0.11
96.74
50.06
T To summarize, we first estimate the ratio of ID to OOD in the test data online. If there is a higher proportion of ID/OOD compared to OOD/ID, we adjust the standards for caching ID/OOD data into memory by dynamically modifying the gap in Eq. For more detailed implementation, please refer the Appendix A.6
Generalization to Other Domains. Besides experiments with natural images, we further validate method on the BIMCV-COVID19+ dataset [61), which includes medical images, following the OpenOOD setup. Specifically, we select BIMCV as the ID dataset, which includes chest X-ray images CXR (CR, DX) of COVID-19 patients and healthy individuals. For the OOD datasets, follow the OpenOOD setup and use CT-SCAN and R-ay-Bone datasets. The CT-SCAN dataset includes computed tomography (CT) images of COVID-19 patients and healthy individuals, while the X- X-Ray-Bonn dataset contains X-ray images of hands. As illustrated in Tab. our AdaNeg method consistently outperforms NegLabel on this medical image dataset.
Memory Length. As demonstrated in Fig. 3e there positive correlation between the memory length L. and the performance outcomes, affirming the efficacy of feature memorization from another perspective. In all our experiments, we have set 10.
Complexity Analyses. As analyzed in Table 4 our AdaNeg approach does not introduce any learnable parameters require model training. Furthermore, it significantly enhances performance while maintaining fast testing speed.
More analyses and discussions the À in Eq. 13 B in Eq. various backbone architectures, the ordering test samples, complementarity to training-bre based method, and the number test data can be found in Appendix A.6
Conclusion and Limitations
We proposed adaptive negative proxies that aligned more effectively with OOD distributions, thereby providing more effective guidance for OOD detection. These proxies were constructed using task-aware feature memory that selectively cached discriminative image features during testing. Our approach facilitated the generation of both task-adaptive and sample-adaptive proxies through carefully designed memory reading operations. Notably, our method was training-free free and annotation- free, and it maintained fast testing speed and achieved state-of-the art results various benchmarks. These results validated the effectiveness of the proposed adaptive proxies.
One minor limitation of our method is the introduction of an external memory requirement. For example, this memory occupies storage space 214.75MB when using the ImageNet dataset as the ID, which may pose challenges for storage-constrained applications.
10

---

References
[1] Yichen Bai, Zongbo Han, Changqing Zhang, Bing Cao, Xiaoheng Jiang. and Qinghua Hu. Id-like prompt learning for few-shot out-of-distribution detection. arXiv preprint arXiv:2311. 2023.
[2] Julian Bitterwolf, Maximilian Mueleer, and Matthias Hein. In or out? fixing imagenet out-of-distribution detection evaluation. In ICML, 2023.
[3] Chaoqi Chen, Jiongeheng Li, Hong-Yo Yu Zhou, Xiaoguang Han, Yue Huang, Xinghao Ding, and Yizhou Yu. Relation matters: Foreground-awand -ware graph-based relational reasoning for domain adaptive object detection. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(3):3677-366 2022.
[4] Chaoqi Chen, Luyao Tang, Yue Huang, Xiaoguang Han, and Yizhou Yu. Coda: generalizing to open and unseen domains with compaction and disambiguation. Advances in Neural Information Processing Systems, 36, 2023.
[5] Chaoqi Chen, Luyao Tang, Leitian Tao, Hong- Zhou, Yue Huang, Xiaoguang Han, and Yizhou Yu. Activate and reject: towards safe domain generalization under category shift. In Proceedings of the JEEE/CVF International Conference on Computer Vision, pages 11552-11563, 2023.
[6] Yihong Chen, Yue Cao, Han Hu, and Liwei Wang. Memory enhanced global-local aggregation for video object detection. In Proceedings of the IEEFICVF conference computer vision and pattern recognition, pages 10337-10346, 2020.
[7] Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3606-360 3613, 2014.
[8] Hanming Deng. Yang Hua, Tao Song. Zongpu Zhang, Zhengui Xue, Ruhui Ma, Neil Robertson, and Haibing Guan. Object guided external memory network for video object detection. In Proceedings of the IEEF/CVF International Conference Computer Vision, pages
2019.
[9] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hicrarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255. Iece, 2009.
[10] Li Deng. The mnist database handwritten digit images for machine learning research [best the web]. IEEE signal processing magazine, 141-1424 2012.
[11] Andrija Djurisic, Nebojsa Bozanic, Arjun Ashok, and Rosanne Liu. Extremely simple activation shaping for out-of distriibbbt detection. arXiv preprint arXiv:2209.0985 2022.
[12] Xuefeng Du, Gabriel Gozum, Yifei Ming, and Yixuan Li. Siren: Shaping representations for detecting out-of-distributio objects. Advances in Neural Information Processing Systems, 45:20434-20449 2022.
[13] Xuefeng Du, Xin Wang, Gabriel Gozum, and Yixuan Li. Unknown-aware object detection: Learning what you don't 't know from videos in the wild. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages (6678-1668,, 2022.
[14] Xuefeng Du, Zhaoning Wang, Mu Cai, and Yixuan Li. Vos: Learning what you don't know by virtual outlier synthesis. arXiv preprint arXiv:2202.01 2022.
[15] Sepideh Esmaeilpour, Bing Liu, Eric Robertson, and Lei Shu. Zero-shot -iittribbtiiitooon detection based on the pre-trained model clip. In Proceedings of the AAAI conference on artificial intelligence, volume 36, pages 6568 6576, 2022.
[16] Ke Fan, Yikai Wang, Qian Yu, Di, and Yanwei Fu, A simple test-time method for out-of-distributioo detection. arXiv preprint arXiv:2207.082. 2022.
[17] Stanislav Fort, Jie Ren, and Balaji Lakshminarayanan. Exploring the limits of out-of-distribution detection. Advances in Neural Information Processing Systems, 34:7068-7081, 2021.
[18] Zhitong Gao, Shipeng Yan, and Xuming He. Atta: Anomaly-aware test-time adaptation for out-of- distribution detection in segmentation. Advances in Neural Information Processing Systems, 36:45150- 45171, 2023.
[19] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 778, 2016.
11

---

[20] Dan Hendrycks and Kevin Cimpel. A baseline for detecting misclassified and out-of-distribution examples in neural networks. arXiv preprini 161.v1661..021366 2016.
[21] Dan Hendryeks, Mantas Mazeika, and Thomas Dietterich. Deep anomaly detection with outlier exposure. arXiv preprint arXiv:1812.04666. 2018.
[22] Dan Hendrycks, Mantas Mazeika, Saurav Kadavath, and Dawn Song. Using self-supervised learning can improve model robustness and uncertainty. Advances in neural information processing systems, 32, 2019.
[23] Dan Hendrycks, Norman Mu, Ekin D Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshminarayanan. Augmix: A simple data processing method improve robustness and uncertainty. arXiv preprint urXiv:1912.0278..22 2019.
[24] Dan Hendrycks, Andy Zou, Mantas Mazeika, Leonard Tang, Bo Li, Dawn Song, and Jacob Steinhardt. Pixmix: Dreamlike pictures comprehensively improve safety measures. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16783-16792, 2022.
[25] Rui Huang, Andrew Geng, and Yixuan Li. On the importance of gradients for detecting distributional shifts in the wild. Advances in Neural Information Processing Systems, 34:677-689, 2021.
[26] Rui Huang and Yixuan Li. Mos: Towards scaling out-of-distribution detection for large semantic space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8710-8719,
2021.
[27] Xue Jiang, Feng Liu, Zhen Fang, Hong Chen, Tongliang Liu, Feng Zheng, and Bo Han. Negative label guided OOD detection with pretrained vision-language models. In The Twelfth International Conference Learning Representations, 2024.
[28] Geethan Karunaratne, Manuel Schmuck, Manuel Le Gallo, Giovanni Cherubini, Luca Benini, Abu Sebastian, and Abbas Rahimi. Robust high-dimensional memory-angmented neural networks. Nature communications, 12(1))2468, 2021.
[29] Shu Kong and Deva Ramanan. Opengan: Open-set recognition via open data generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 813-822, 2021.
[30] Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. 2009.
[31] Alex Krizhevsky, llya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25, 2012.
[32] Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 7(7):3, 2015.
[33] Kimin Lee, Kibok Lec, Honglak Lec, and Jinwoo Shin. A simple unified framework for detecting out- of-distribution samples and adversarial attacks. Advances in neural information processing systems, 31, 2018.
[34] Tianqi Li, Guansong Pang. Xiao Bai, Wenjun Miao, and Jin Zheng. Learning transferable negative prompts for -ut-of-distribuutio detection. Proceedings of the IEFF//VV Conference on Computer Vision and Pattern Recognition, pages 17594, 2024.
[35] Jian Liang, Ran He, and Tieniu Tan. comprehensive survey on test-time adaptation under distribution shifts. arXiv preprint Xiv: /5361, 2023.
[36] Shiyu Liang, Yixuan Li, and Rayadurgam Srikant. Enhancing the reliability of out-of-distribution image detection in neural networks. arXiv preprint arXiv:/706.02690 2017.
[37] Weitang Liu. Xiaoyun Wang, John Owens. and Yixuan Li. Energy- out- -isttrribbtto detection. Advances in neural information processing systems, 33:21464 21475, 2020.
[38] Xixi Liu, Yaroslava Lochman, and Christopher Zach. Gen: Pushing the limits of softmax- based out-of- distribution detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patten Recognition, pages
2023.
[39] Yifei Ming. Ziyang Cai, Jiuxiang Gu, Yiyou Sun, Wei Li. and Yixuan Li. Delving into out-of-distribution detection with vision-language representations. Advances in Neural Information Processing Systems, 35:35087-35102. 2022.
[40] Yifei Ming and Yixuan Li. How does fine-tuning impact out-of -fdistribbution detection for vision- languaeee models? Internatoonal Journal of Computer Vision, 132(2):596 6099 2024.
12

---

[41] Yifei Ming. Yiyou Sun, Ousmane Dia, and Yixuan Li. How to exploit hyperspherical embeddings for out-of-distribution detection? arXiv preprint urX1v:2203.04450. 2022.
[42] Atsuyuki Miyai, Qing Yu, Go Irie, and Kiyoharu Aizawa. Locoop: Few-shot uutto--distriuution detection via prompt learning. Advances Neural Information Processing Systems, 36, 2024.
[43] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Baolin Wu, Andrew Y Ng. et al. Reading digits in natural images with unsupervised fcature learning. In NIPS workshop on deep learning and unsupervised feature feature learning, volume 2011, page 7. Granada, Spain, 2011.
[44] Anh Nguyen, Jason Yosinski, and Jeff Clune. Deep neural networks casily fooled: High confidence predictions unrecognizable images. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 427-436, 2015.
[45] Jun Nie, Yonggang Zhang. Zhen Fang. Tongliang Liu, Bo Han, and Xinmei Tian. Out-of-distribtion detection with negative prompts. In The Twelfth International Conference on Learning Representations, 2024.
[46] Scoung Wug Oh, Joon- Young Lee, Ning Xu, and Scon Joo Kim. Video object segmentation using space- time memory networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9226 9235, 2019.
[47] Charles Packer, Vivian Fang. Shishir G Patil, Kevin Lin. Sarah Wooders, and Joseph E Gonzalez. Memgpt: Towards ilms as operating systems. arXiv preprint urXiv:2310.08560, 2023.
[48] Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. International conference on machine learning, pages 8748 8763, PMLR, 2021.
[49] Jie Ren, Stanislav Fort, Jeremiah Liu, Abhijit Guha Roy, Shreyas Padhy, and Balaji Lakshminarayanan. A simple fix to mahalanobis distance for improving near-ood detection. arXiv preprini 1rXiv:2100.09022. 2021.
[50] Seonghan Ryu, Sangjun Koo, Hwanjo Yu, and Gary Geunbae Lee. Out-of-domain detection based on generative adversarial network. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 714-718, 2018
[51] Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, and Timothy Lillicrap. Mcta- learning with memery-augmented neural networks. In International conference on machine learning. pages 1842-1850. PMLR, 2016.
[52] Walter J Scheirer, Anderson de Rezende Rocha, Archana Sapkota, and Terrance E Boult. Toward open set recognition. IEEE transactions on pattern analysis and machine intelligence, 35(7):157-1177,, 2012.
[53] Vikash Sehwag, Mung Chiang, and Prateck Mittal, Ssd: A unified framework for self-supervised outlier detection. arXiv preprint arXiv:2103. /2051, 2021.
[54] Manli Shu, Weili Nie, De-An Huang, Zhiding Yu, Tom Goldstein, Anima Anandkumar, and Chaowei Xiao. Test-time prompt tuning for zero-shot generalization in lision-language models, Advances in Neural Information Processing Systems, 35:14274- 14289, 2022.
[55] Sainbayar Sukhbaatar, Jason Weston, Rob Fergus, al. End-to-end memory networks. Advances in neural information processing systems, 28, 2015.
[56] Yiyou Sun, Chuan Guo, and Yixuan Li. React: Out-of-distribution detection with rectified activations. Advances in Neural Information Processing Systems, 34: 144-17,7 2021.
[57] Yiyou Sun, Yifei Ming, Xiaojin 7hu, and Yixuan Li. Out-of-distribution detection with deep nearest. neighbors. In International Conference on Machine Leamning, pages 20827-20840. PMI.R, 2022.
[58] Jihoon Tack, Sangwoo Mo, Jongheon Jeong, and Jinwoo Shin. Csi: Novelty detection via contrastive learning on distributionally shifted instances. Advances in neural information processing systems, 33:1 1839- 11852, 2020.
[59] Leitian Tao, Xuefeng Du, Xiaojin Zhu, and Yixuan Li. Non- Non-parametric outlier synthesis. arXiv preprint arXiv:2303.0294 022.6666..000...... 2023,
13

---

[60] Grant Van Horn, Oisin Mac Aodha, Yang Song. Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. The inaturalist species classification and detection dataset. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8769-8778, 2018.
[61] Maria De La Iglesia Vayá, Jose Manuel Saborit, Joaquim Angel Montell, Antonio Pertusa, Aurelia Bustos, Miguel Cazorla, Joaquin Galant, Xavier Barber, Domingo Orozeo-Beltrán, Francisco Garefa-Garcia al. Bimev covid- 19+: a large annotated dataset rx and ct images from covid-19 patients. arXiv preprint arXiv:2006.001 174, 2020.
[62] Sagar Vaze, Kai Han, Andrea Vedaldi, and Andrew Zisserman. Open-set recognition: good closed-set classifier is all you need? arXiv preprint arXiv:21 10.06207, 2021.
[63] Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. Tent: Fully test-time adaptation by entropy minimization. arXiv preprint arXiv:2006. 10726, 2020.
[64] Haoqi Wang. Zhizhong Li, Litong Feng, and Wayne Zhang. Vim: Out-of-distributin with virtual-logit matching. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4921 0921-4930, 2022.
[65] Hualiang Wang, Li, Huifeng Yao, and Xiaomeng Li. Clipn for zero-shol ood detection: Teaching clip to say no. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages I 1802-1812, 2023.
[66] Yezhen Wang, Bo Li, Tong Che, Kaiyang Zhou, Ziwei Liu, and Dongsheng Li. Energy-based open-world uncertainty modeling for confidence calibration. In Proceedings of the IEEL/CVF International Conference on Computer Vision, pages 9302-9311, 1, 2021.
[67] Jason Weston, Sumit Chopra, and Antoine Bordes. Memory networks, arXiv preprint 11X.v.4400.396.. 2014.
[68] Jianxiong Xiao, James Hays, Krista A Ehinger. Aude Oliva, and Antonio Torralba. Sun database: Large- scale scene recognition from abbey to zoo. In 2010 IEFE computer society conference on computer vision and pattern recognition, pages 3485-3492. IEFE, 2010.
[69] Guo-Sen Xie, Huan Xiong, Jie Liu, Yazhou Yao, and Ling Shao. Few-shot semantic segmentation with cyclic memory network. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7293-7302, 2021.
[70] Kai Xu, Rongyu Chen, Gianni Franchi, and Angela Yao. Scaling for training time and post-hoc out-of- distribution detection enhancement. arXiv preprint arXiv:2310.00227, 2023.
[71] Jingkang Yang. Pengyun Wang. Dejian Zou, Zitang Zhou, Kunyuan Ding, Wenxuan Peng, Haoqi Wang. Guangyao Chen, Bo Li. Yiyou Sun. Xuefeng Du Kaiyang Zhou, Wayne Zhang, Dan Hendrycks, Yixuan Li, and Ziwei Liu. Openood: Benchmarking generalized out-of-distribution detection. 2022.
[72] Puning Yang, Jian Liang, Jie Cao, Ran He. Auto: Adaptive outlier optimization for online test-time ood detection. arXiv preprint arXiv:2303. 2...2.2232...... /2267, 2023.
[73] Jingyang Zhang, Nathan Inkawhich, Yiran Chen, and Hai Li. Fine-grained out-of-distribution detection with mixup outlier exposure. CoRR, (abs/2106.03917)) 2021.
[74] Jingyang Zhang, Jingkang Yang. Pengyun Wang, Haoqi Wang, Yueqian Lin, Haoran Zhang, Yiyou Sun, Xuefeng Du Kaiyang Zhou, Wayne Zhang. Yixuan Li. Ziwei Liu. Yiran Chen, and Hai Li. Openood v1.5: Enhanced benchmark for oft-t--distiibution detection. arXiv preprint arXiv:2306.0940 09301, 2023.
[75] Yabin Zhang, Bin Deng, Hui Tang, Lei Zhang, and Kui Jia. Unsupervised multi-class domain adaptation: Theory, algorithms, and practice. JEEE Transactions on Pattern Analysis and Machine Intelligence, 44(5):2755-792, 2020.
[76] Yabin Zhang, Wenjic Zhu, Chenhang Hc, and Lei Zhang. Lapt: Label-driven automated prompt tuning for ood detection with vision-language models. In European Conference on Computer Vision, 2024.
[77] Yabin Zhang, Wenjie Zhu, Hui Tang, Zhiyuan Ma, Kaiyang Zhou, and Lei Zhang. Dual memory networks: A versatile adaptation approach for lasion-language models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 28718 18-28888,, 2024.
[78] Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEEE transactions on pattern analysis and machine intelligence, 40(6):1452 1464,4442222 2017.
14

---

Table A7: Detailed OOD detection results on the OpenOOD benchmark, where ImageNet- is
adopted as ID dataset.
Near-FFa--OOD
Datasets
FPR95
AUROC t
SSB-hard 662
74.91
75.11
Near-OOD
NINCO 2
60.10
78.30
Mean
67.51
76.70
iNaturalist 660
0.72
99.72
Textures
21.40
95.71
Far-OOI)
OpenImage-O
64
29.81
93.87
Mean
17.31
96.43
A Appendix
A.1 ID-Similarity to OOD Ratio with ground truth ID and OOD labels
We introduce the ID-Similarity OOD Ratio (ISOR) to quantitatively measure the relative alignment of negative proxies with ground truth OOD labels, compared to their alignment with ID labels. In implementation, we adapt the score function of NegLabel (i.e., Eq. 4. which originally measures the similarity of a test image to ID labels over negative labels. We modify this function by replacing the negative labels with ground truth OOD labels and changing the input from test images negative proxies (e.g., negative texts), while keeping other aspects consistent with Eq. 4 This modified score function allows
assess the degree of similarity between the inputs and ground truth ID/OOD labels, thereby quantifying the relative alignment between negative proxies and OOD labels. Specifically, lower ISOR indicates higher similarity OOD labels and a reduced similarity to labels.
A.2 Entropy-based caching strategy for full memory
The memory construct is of finite length; hence, as the number of cached images increases, it may become fully occupied. To address this, we have devised a simple strategy to retain only those images with low entropy, e.g., high confidence. Specifically, when storing image in memory, also record its entropy pertinent to OOD detection:
Entropy (.) log(S,:(1)) (1 (n)( log(1 - Sn((1)),
(A.1)
where Sni(v) represents the probability of belonging to the ID, as shown Eq. (4) Given the entropy of the current test image and full memory M y. we replace the item with the maximum entropy in M, y with the current image feature if the current test image exhibits lower entropy. Otherwise, do not cache the current test sample.
A.3 Evaluation criteria
Following common practice 126 39 27
we employ the following three evaluation metrics: (1)
FPR95, which measures the false positive rate for OOD samples when the detection accuracy for ID samples is 95%; (2) AUROC, the area under the receiver operating characteristic curve; and (3) ID ACC, which quantifies the accuracy of correctly identifying and classifying ID samples.
A.4 Detailed results on ImageNet dataset under OpenOOD setting
The detailed OOD detection results on the OpenOOD benchmark are presented in Tab. A7
A.5
Results CIFARI0/100
As illustrated in Tab. A8 the advantage of our AdaNeg also holds on the CIFAR 10/100 dataset. Notably, our method achieves new state-of-the-ar results in the far-OOD setting under a zero-shot training-free free manner, outperforming its competitors training on the full labeled training set.
15

---

Table A8: OOD detection results with CIFARI0/100 on the OpenOOD benchmark. Full results are provided in Tables A10 and A9
FPR95 4
AUROC 1
Methods
Ncar-OOD Far-OOD Near-OOD
Far-OOD
Methods requiring training (or fine-tuning)
PixMix
KNN S
93.10
95.94
OE 21
MSP
20
94.82
96.00
PixMix 24
RotPred 22
94.86
98.18
Zero-shot Training-fres Methods
MCM 39
30.86
17.99
91.92
95.54
NegLabel
27
28.75
6.60
94.58
98.39
AdaNeg (Ours)
20.40
2.79
94.78
99.26
CIFARIO ID dataset
FPR95
AUROC1
Methods
Near-OOD
Far-OOD
Near-OOD
Far-OOD
Methods requiring training (or fine-tuning)
GEN 38
81.31
79.68
VOS
EBO
37
80.93
81.32
SCALE
70
80.99
81.42
OE 21
MSP
20
88.30
81.41
Zero-shot Training-free Methods
MCM 39
75.20
59.32
71.00
76,00
NegL .abel
27
71.44
40.92
70.58
89.68
AdaNeg (Ours)
59.07
29.35
84.62
95.25
(b) CIFARIO0 ID dataset
Table A9: Detailed OOD detection results of on the OpenOOD benchmark, where CIFAR100
adopted as ID dataset.
Near-/Far-OOD
Datasets
FPR95 4
AUROC 1
CIFARIO [30
58.24
79.91
Near-OOD
TIN 32
59.90
89.34
Mean
59.07
84.62
MNIST 10
4.18
97.90
SVHN 43
6.03
97.60
Far-OOD
Texture
30.00
95.14
Places365
78
77.20
90.35
Mean
29.35
95.25
A.6 More analyses
Detailed Implementation of AdaGap Module. We have implemented an adaptive gap (AdaGap) module to adjust the memorization selection criteria dynamically. This strategy builds on observation that the score Sni increases/decreases, the probability that sample is ID/OOD also increases accordingly. By enforcing a stringent selection criterion, we can effectively minimize the inclusion of misclassified samples in memory. Specifically, first online estimate the ratio of ID to OOD samples in the test data using a First-In-First- queue, which caches the ID/OOD estimation (cf. Eq. 8) of the most recent N samples:
Estimated ID Number
MR Estimated ID Number + Estimated OOD number'
(A.2)
where the ID and OOD numbers are acquired within the queue.
Leveraging the estimated mix ratio (M R), we can dynamically adjust the gap g in memory caching to avoid majority of misclassified samples within the memory. For instance, if we find that ID
16

---

Table A10: Detailed OOD detection results of our method on the OpenOOD benchmark, where CIFARIO is adopted as ID dataset.
Near-FFar-OOD
Datasets
FPR95 +
AUROC 1
CIFAR 100
30
35.80
90.93
Near-OOD
TIN
32
5.01
98.63
Mean
20.40
94.78
MNIST 10
0.13
99.96
SVHN 43
0.04
99.97
Far-OOD)
Texture
7
0.04
99.82
Places365
78
10.93
97.29
Mean
2.79
99.26
77.00
97.00
76.6
76.75
96.75
76.4
96
Far-OCD AUROC
Near-DOD AUROC
76.50
96.50
76.25
96.25
76.0
Near- -0-0000
76.00
96.00
75.8
+ Near-DOD
Near-O0D
75.6
75.50
95.50
95
75.25
95.25
75.2
2 À values
@ values
(a) À value
(b) B value
Figure A4: Analyses on the hyper-parameters of (a) À in Eq. and (b) 3 in Eq. 11 on the ImageNet dataset under OpenOOD setting.
samples constitute the majority of the test samples (i.e., MR 0.5), this could lead to an increased adjust the selection criterion for
proportion of ID samples in the OOD memory. In response, OOD) memorization to only cache higher confidence OOD samples. This adjustment is achieved by modifying the selection criterion for memorization in Eq. 8 as follows:
Negative :S,((v) < < 97 Sni(v) <9 max(g, MR)7,
Positive (n)(() 2 7 + 9(1 7) Sa(() 2 max(g, MR) - 7),
(A.3)
where = 0.5 is the default gap analyzed in Figure 3b conditions (i.e,, MR 0.5). our method aligns with our original version. However, the proportion of ID samples is higher in the test samples' estimation (i.e., M R 0.5), we raise the standard for storing negative samples the memory. an extreme case when M ft = 1, we estimate that there might be no OOD samples among the test samples; thus, stop storing test samples in the negative memory and only selectively cache test samples into the positive memory. We adjust our approach conversely when the M R value is lower than 0.5.
In this way. under ID/OOD balanced
Please note that the selection criterion dynamically adjusted online because the M is estimated with the most recent N test samples. Here, we set N = 10, 000 by default. This dynamic adjustment ensures that our memory caching strategy remains responsive to the evolving nature of the test sample distribution, thereby optimizing memory utilization and enhancing the accuracy of our domain distinction process.
in Eq. 13 As illustrated in Fig. A4a the OOD detection performance remains robust across wide range of À values, e.g., from 0.01 For experiments, we have À to 0.1.
in Eq. I Results with different 3 values are shown in Fig. A4b where OOD detection performance robust the 3 values. We adopt 3=5.5 in all experiments.
Ordering of Testing Data. Our AdaNeg selectively caches test data into memory, potentially causing variations in the results depending on the ordering of the test data. To rigorously test this
17

---

aspect, we randomly shuffled the order of the test data using three distinct seeds. We observed that our method exhibits robustness to changes in the ordering of data. Specifically, across three experiments conducted on the ImageNet dataset, the AUROC scores were 96.65% 96.69% and 96.64% respectively, demonstrating fluctuations of less than 0.1%. We report the average results from three random runs in our paper.
Various Backbone Architectures. Results with various VLMs architectures are illustrated in Tab. A where better results are typically achieved with stronger backbones.
Table A11: OOD detection results of our AdaNeg with different VLMs architectures, where ImageNet- IK is used as the dataset.
OOD datasets
Backhone
INaturalist
Textures
Average
AUROC1
FPR95]
AUROCT
FPR95]
AUROCT
FPR95]
AUROCT
FPR95)
AUROCT
FPR95]
ResNer50
99.58
1.18
97.37
10.56
93.84
43.19
94,18
35.00
96.24
22.48
VITL/32
99.59
1,02
97.53
9.63
93.99
38.45
94,21
37.92
96.33
21.76
VITB/16
99.71
97.44
96.66
18.92
Complementarity to Training-based Method. We validated the complementarity between our AdaNeg method and the latest works, NegPrompt [4 and LAPT [76), which use learned prompts. As shown in Table A12 our method significantly improves performance over these approaches, demonstrating its complementarity with training-basse based methods.
Table A12: FPR95 (4) with the ID dataset of ImageNet.
Methods
INaturalist
SUN
Places
Textures
Average
NegPrompt
34
6.76
23.41
28.32
34.57
23.27
AdaNeg
3.87
11.35
25.45
29.79
17.62
LAPT (76
1.10
20.59
35.38
40.11
24.29
AdaNeg
0.58
9.98
30.47
24.25
16.32
Number of Testing Data. We examine the dependency of our approach on the number of test images by evaluating its performance across different scales of test samples. As the number of test samples increases (from 900 90K), the cached feature data also increases, leading improvement in method's 's results, as shown in Tab. A13 Even with a small number test samples (e.g., 90 and 900), method significantly reduces FPR95 compared to NegLabel, demonstrating its robustness across different numbers of test images.
Note that with only 90 test images, the task of distinguishing between ID and OOD samples degener- ates into a simpler task since the number of test images is even smaller than the number of classes (e.g., 1000 for ImageNet). Consequently, both NegLabel and our method achieve lower FPR95 in such an easier scenario.
Table A 13: FPR95 (4) with different numbers of test images, where test samples are randomly sampled from ImageNet (ID) and SUN (OOD) datasets, and we maintain a consistent ratio of ID to OOD samples 5:4 throughout the experiments.
Num. of Test Images
90
900
9K
45K
90K
NegLabel
14.00
20.44
20.71
20.51
20.53
AdaNeg
6.00
10.12
9.78
9.66
9.50
18

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Guidelines:
This paper contributes the adaptive negative proxies for OOD detection, which matches the main claims in the abstract and introduction.
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Please refer Sec. i the main paper.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
19

---

The main contributions are the algorithm design and experimental validation. No theory proofs are provided.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide detailed algorithm designs in Sec. 3 and hyper-parametert Sec to facilitate reproducibility.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
20

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
The adopted datasets are publicly available. Codes are available at https V/github. com YBZh/ OpenOOD-VLM
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
cttips.cc/uublic//
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The proposed method does not require training, and the testing details are illustrated in 1.c
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: As analyzed Sec. A.6 in the appendix, we examined the experimental results the ImageNet dataset with different orderings of the testing data. We found that the order of the test data has minimal impact on the final results, e.g., the variation in three random experiments was less than 0.1 %, which verified the statistical stability of method. Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
21

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for caleulating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
Question:
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification: Our experiments can be reproduced with only one V100 GPU, as shown in Sec. 4.1
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics httpp//////.aappp.cccpppp///////
Question:
Answer: [Yes]
Justification:
I have read the NeurIPS Code of Ethics and confirm that this paper conforms.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Positive Societal Impacts: The proposed method for OOD detection can enhance the reliability and safety of AI systems, particularly in critical applications such healthcare, autonomous driving, and cybersecurity. By improving the detection out- of-distribution data, the system can better avoid potentially harmful decisions based on
22

---

anomalous inputs, thereby increasing trust in Al technologies and their deployment in various fields.
Negative Societal Impacts: One potential negative impact is the risk of privacy issues due to online eaching and processing of test data, which may inadvertently store sensitive information. Additionally, the improved OOD detection might be misused in surveillance and monitoring applications, leading to ethical concerns regarding privacy and autonomy.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
The used codes and datasets well cited.
Guidelines:
23

---

The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name the license (e.g.. CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does release assets.
Researchers should communicate the details the dataset/code/mooel as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer:[NA]
Justification:
The paper does not involve crowdsourcing research with human subjects.

---

Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
25


Reviews:
[{"review_id": "a8RDzZ2FOt", "reviewer": "Reviewer_ACdF", "summary": "The authors introduce a new approach to leverage the pre-trained vision-language model for identifying out-of-distribution (OOD) samples. Compared to prior works that employ consistent negative labels across different OOD datasets, they introduce adaptive negative proxies to dynamically generate text labels during testing by exploring actual OOD images, thereby aligning more closely with the underlying OOD label space. Empirically, the proposed method demonstrates state-of-the-art performance across various OOD detection benchmarks especially on the large-scale ImageNet benchmark.", "questions": "Please refer to the weakness.", "limitations": "yes", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "- Dynamically generating negative proxies is a simple and effective strategy. \n\n-  The setting studied is very natural and this paper can easily stimulate further research in the area.\n\n- The proposed approach performs well, particularly on large-scale datasets such as ImageNet, effectively demonstrating its scalability.\n\n- The paper is nicely written.", "weaknesses": "- While the proposed AdaNeg shows clear improvements over training-free baselines, its overall performance on ImageNet still lags behind training-based methods. This raises the question of whether there are opportunities for complementarity between the two approaches.\n\n- Can the dynamic update of the memory bank and refinement of OOD proxies during the testing stage be considered a form of test-time training? The authors are requested to clarify the inherent connections and distinctions, especially from the perspectives of training versus training-free approaches.\n\n- If negative proxies can directly identify true out-of-distribution (OOD) test images during the testing phase, is it possible to use the identified OOD samples to update the model parameters online?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer *ACdF*，\n\nWe sincerely thank you for the constructive comments and recognition on our work! Please find our responses below.\n\n\n### **Q1: While the proposed AdaNeg shows clear improvements over training-free baselines, its overall performance on ImageNet still lags behind training-based methods. This raises the question of whether there are opportunities for complementarity between the two approaches.**\n\n**A1:** Thanks for the nice suggestion! \nWe validated the complementarity between our AdaNeg and the existing state-of-the-art method NegPrompt [31], which is characterized by learnable negative prompts. \nWe reproduced the results of NegPrompt with the author released codes. As shown in the table below, our method brings significant performance improvements over NegPrompt, validating the complementarity between our approach and training-based methods. We will include these analyses in the revision.\n\n| Methods        | INaturalist | SUN   | Places | Textures | Average |\n|----------------|:-----------:|:-----:|:------:|:--------:|:-------:|\n| **NegPrompt**       | 6.76        | 23.41 | 28.32  | 34.57    | 23.27   |\n| + **AdaNeg**   | **3.87**    | **11.35**  | **25.45** | **29.79** | **17.62** |\n\n**Table**: FPR95 ($\\downarrow$) of AdaNeg based on the training-based NegPrompt method.\n\n### **Q2:  Can the dynamic update of the memory bank and refinement of OOD proxies during the testing stage be considered a form of test-time training? The authors are requested to clarify the inherent connections and distinctions, especially from the perspectives of training versus training-free approaches.**\n\n**A2:** Thanks for the suggestions.\nOur online update of the memory bank and refinement of OOD proxies is a kind of training-free test-time adaptation. Unlike existing training-required test-time adaptation methods [13, 15, 69], which typically require test-time optimization and subsequently slow down the testing process, our approach is optimization-free. It introduces only a lightweight memory interaction operation, enabling rapid and accurate testing, as analyzed in Table 4.\nWe will further clarify this point in the revision.\n\n### **Q3:  If negative proxies can directly identify true out-of-distribution (OOD) test images during the testing phase, is it possible to use the identified OOD samples to update the model parameters online?**\n\n**A3:** Thank you for the comments. \nWe performed experiments to use the identified OOD samples to update the text prompt with cross-entropy loss, similar to the training objective in [1]. However, it is unstable and often leads to model collapse, especially in challenging settings. For example, in the ImageNet (ID) and SSB-hard (OOD) setup, around 44\\% of OOD samples were misclassified as ID, disrupting model learning.\n\nOur training-free method is more robust to such misclassifications due to the weighted combination of cached features. Additionally, updating model parameters online would significantly slow down the testing process. In contrast, our method ensures rapid and accurate testing with only lightweight memory interaction.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Nx44vVzMAL", "reviewer": "Reviewer_yxd9", "summary": "This paper introduces a new algorithm for Out-Of-Distribution (OOD) sample detection. First, it analyzes the shortcomings of previous Vision-Language OOD detection methods and proposes improvements based on these findings. Specifically, the paper presents a scheme for online updating of the memory bank during testing to design better negative proxies. The authors conducted experiments on datasets such as ImageNet and CIFAR. According to the experimental results, the newly proposed method can enhance OOD detection performance.", "questions": "As shown in weakness. What is the meaning of Figure 1, and is the proposed method effective when the base model predicts most samples incorrectly?", "limitations": "Potential negative societal impact is not applicable.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. Currently, vision-language models are developing rapidly, and using them for OOD sample detection is a promising direction. Approaching from this perspective may yield better results.\n2. The experiments in this paper are relatively thorough, encompassing both large datasets based on ImageNet and smaller datasets based on CIFAR. According to the authors' experimental results, the newly proposed method can improve the accuracy of OOD detection.", "weaknesses": "1. The motivation in this paper is not very clear. Specifically, in Figure 1(a), it is not evident why the newly proposed AdaNeg is better than NegLabel. On the contrary, the distribution of OOD samples seems to be closer to NegLabel.\n\n2. The method proposed in this paper is based on the features and results of test samples during testing, which limits the upper bound of the method. In my opinion, the effectiveness of the proposed method relies on the vision-language model's strong inherent OOD detection capability, meaning that most test samples can be correctly processed. Based on these correctly processed samples, the method can further improve the detection accuracy of other samples. However, if in a certain scenario, the model itself cannot correctly estimate most of the samples, this method might actually make the results worse.\n\n3. This paper merely performs optimizations based on the NegLabel framework, without many innovative points. The novelty of this improvement is insufficient to support a NeurIPS paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer *yxd9*，\n\nWe sincerely thank you for the constructive comments! We hope our following responses can address this reviewer's concerns.\n\n###  **Q1: The motivation in this paper is not very clear. Specifically, in Figure 1(a), it is not evident why the newly proposed AdaNeg is better than NegLabel. On the contrary, the distribution of OOD samples seems to be closer to NegLabel.**\n\n**A1:** Thank you for pointing out this issue. In the visualization of our submission, the adaptive negative proxies are not $L_2$ normalized, whereas the features of ID, OOD, and NegLabel are normalized vectors. This makes the visualization not very clear. We have corrected this issue by revising the visualization with normalized adaptive negative proxies. As expected, the distribution of adaptive negative proxies is much closer to the ground truth OOD labels than to the negative labels in NegLabel.\nWe have attached an **updated Figure in the PDF of the Global Response** and will revise Figure 1(a) in the manuscript.\n\n### **Q2: The method proposed in this paper is based on the features and results of test samples during testing, which limits the upper bound of the method. In my opinion, the effectiveness of the proposed method relies on the vision-language model's strong inherent OOD detection capability, meaning that most test samples can be correctly processed. Based on these correctly processed samples, the method can further improve the detection accuracy of other samples. However, if in a certain scenario, the model itself cannot correctly estimate most of the samples, this method might actually make the results worse.**\n\n**A2:** Thanks for the comments.\nWe agree that our method leverages the inherent strong capability of vision-language models (VLMs).  However, we would like to emphasize that our method is highly robust to misclassified samples. \n\nFor instance, we investigate a challenging task setting with ImageNet as the ID dataset and SSB-hard [58] as the OOD dataset, where NegLabel achieves an FPR95 of 77.26%.\nWith a threshold $\\gamma = 0.5$ and a gap $g = 0$, about 44\\% of OOD samples were incorrectly stored in memory slots for ID classes. In this scenario, our method achieves an FPR95 of 72.90\\%, showing a 4.36\\% improvement over NegLabel. This robustness comes from the weighted combination of cached features used to form the proxies, which resists interference from a few misclassified features.\n\nIn a worst-case scenario, where test samples are randomly cached into ID or OOD memories (50\\% OOD misclassified), our method achieves an FPR95 of 77.69\\%, comparable to NegLabel's 77.26\\%. Such extreme confusion is unlikely in real-world scenarios, where there are usually some clues to distinguish between ID and OOD samples.\n\n### **Q3: This paper merely performs optimizations based on the NegLabel framework, without many innovative points. The novelty of this improvement is insufficient to support a NeurIPS paper.**\n\n**A3:** We appreciate this reviewer's comments; however, we'd like to argue that our method introduces significant improvements to the NegLabel framework and makes notable contributions to negative proxies guided OOD detection. Our contributions and distinguished features are as follows:\n- We identify the label space misalignment between existing negative-label-based proxies and the target OOD distributions. To address this issue, we dynamically generate adaptive negative proxies to align with the OOD label space more effectively. This is a \"novel approach\" and a \"significant advancement,\" as recognized by Reviewers ACdF and eV4A.\n- We construct adaptive negative proxies using a feature memory bank that incorporates carefully designed write and read strategies. Additionally, we propose a novel multi-modal score that combines complementary textual and visual knowledge, which is an \"interesting design\" as highlighted by Reviewer Pmx9.\n- Our method is simple yet effective (cf. Reviewer ACdF), training-free, and annotation-free (cf. Reviewer eV4A). It exhibits good scalability (cf. Reviewers ACdF and eV4A), has been comprehensively evaluated (cf. Reviewers yxd9 and eV4A), and achieves substantial performance improvements (cf. Reviewers yxd9 and eV4A).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kXHqgABTra", "reviewer": "Reviewer_Pmx9", "summary": "In this paper, the authors propose AdaNeg, a test-time adaption method for CLIP-based post-hoc OOD detection. AdaNeg is an extension of NegLabel and introduces a class-wise memory bank for each ID and negative labels. The memory bank is gradually filled with ID and OOD features during the model deployment. The author design a margin-based approach to select positive and negative samples with high confidence. And they propose a cache elimination mechanism to update the memory bank. Besides, AdaNeg uses cross attention between the input sample and the memory bank to reweight the cached features. The experimental results show the proposed method outperforms the baseline methods under various benchmarks.", "questions": "see Cons", "limitations": "Yes", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "<1> AdaNeg uses dynamic OOD proxies instead of the static design of NegLabel, achieving SOTA performance in CLIP-based zero-shot OOD detection. \n\n<2> The multi-modal score is an interesting design and explanation that demonstrates the improvement brought by using both text and image encoding capabilities in a multi-modal model.\n\n<3> The paper is well organized and easy to follow.", "weaknesses": "**Major concerns**\n\n<1> AdaNeg is a test-time adaption approach that caches features w.r.t. ID labels and negative labels by maintaining a class-wise memory bank. For OOD detection, the biggest problem of the test-time adaptation method is that the arrival time of the OOD sample is uncertain. Compared with non-TTA methods, AdaNeg has greater uncertainty in its performance during the deployment phase and may even risk causing model collapse. \n\nFor example, when the model is deployed to a close-world environment, almost all input samples are ID samples (I believe this is a very common scenario). In this case, the memory banks of negative labels will gradually be filled with ID samples (in long-term deployment, there will always be misclassified ID samples that enter the negative memory banks). Since the number of negative labels is much greater than that of ID labels, more and more ID samples will be misclassified as OOD over time. I suggest the author conduct an experiment using the 1.28M training set images of ImageNet-1k as input (this still meets the zero-shot setting of CLIP) and observe how the proportion of samples misclassified as OOD changes with the number of input samples. If 1.28M images are repeatedly input into multiple rounds, will the misclassification rate increase further? In contrast, the other case is that the OOD samples are far more than the ID samples. Will this cause a greater false positive risk? I hope the authors can test their method with different ID and OOD sample mixture ratios, such as 1:100, 1:10, 1:1, 10:1, 100:1.\n\nIn summary, I suggest the authors to further study the setting of TTA in OOD detection to improve the motivation of the work, since the input samples may come from two different distributions, ID and OOD. How to ensure the stability of TTA OOD detection algorithm when the input stream is a non-stationary process is a problem worth studying.\n\n<2> The negative labels provide the initial memory bank slots for AdaNeg, but it seems to me that the negative labels are not necessary. This suggests that we need to rethink AdaNeg's motivation for negative labels. Why do samples that are judged as negative need to be placed in the memory bank w.r.t. the negative label? What if the authors directly use the MCM score to judge negative samples and then let them organize themselves into OOD proxies? The authors need to provide a more detailed analysis (preferably theoretical analysis) to prove that the **negative label-based** memory bank design is necessary.\n\nFurther, negative labels simply select words that are semantically far from ID labels. For some OOD samples, they may be far away from both ID labels and negative labels. According to the mechanism of AdaNeg, they cannot enter the memory bank of negative labels. Is this a negative impact of designing a memory bank based on negative labels?\n\n**Minor concerns**\n\n<1> The authors need to provide more detailed experimental settings. The paper mentions that memory banks are task specific. When evaluating the model, taking the ImageNet-1k benchmark as an example, do the authors maintain an independent memory bank for each OOD dataset (precisely, each ID-OOD pair), or did the four OOD datasets share one memory bank?\n\n<2> There seem to be some typos and symbol issues in the paper.\n\na) L247: temperature $\\tau = 100$ seems to be $\\tau = 0.01$ because $\\tau$ is in the denominator.\n\nb) The subscript NL is not case-inconsistent, e.g., Eq. (4) and Eq. (8).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer *Pmx9*，\n\nWe sincerely thank you for the constructive comments and recognition on our work! Please find our responses below.\n\n### **Q1: Analyses on the stability of our method with different ID and OOD sample mixture ratios**\n\n**A1:**  Many thanks for the detailed and valuable comments. Following this reviewer's suggestion, we investigated the stability of our method by constructing test sets with various mixture ratios of ID and OOD samples. Specifically, we adopted the 1.28M ImageNet training data as ID and randomly sampled 12.8K and 1.28K instances from the SUN OOD dataset to construct the ID:OOD ratios of 100:1 and 1000:1 settings, respectively. To construct the ID:OOD ratios of 1:100, 1:10, 1:1, and 10:1 settings, we used the full 40K SUN dataset as OOD and randomly sampled 400, 4K, 40K, and 400K instances from the ImageNet training data. We did not validate the setting where the test set contains only ID data, as the absence of OOD data makes it impossible to calculate evaluation metrics (e.g., FPR95 and AUROC).\n\nAs shown in the table below, our method outperforms NegLabel across a wide range of mixture ratios (from 1:100 to 100:1), validating the robustness and reliability of our approach. As pointed out by this reviewer, unbalanced mixture ratios do pose a challenge to our method. Our approach performs the best in scenarios with a balanced mixture of ID and OOD samples, reducing the FPR95 by 11.18\\%. As the mixture ratio becomes increasingly unbalanced, the improvement brought by our method gradually decreases. When the unbalanced ratio reaches 1000:1, our method shows some negative impact. We will include these analyses in the limitation part of our revised manuscript, and attempt to address this challenging setting in future work.\n\n| ID:OOD Ratio | 1:100 | 1:10 | 1:1  | 10:1 | 100:1 | 1000:1 |\n|--------------|-------|------|------|------|-------|------|\n| **NegLabel** | 22.42 | 21.11 | 20.99 | 20.92 | 21.48 | **23.69** |\n| **AdaNeg**   | **21.00** | **12.49** | **9.81**  | **15.61** | **20.71** | 26.28 |\n\n**Table Caption**: FPR95 ($\\downarrow$) with different mixture ratios of ID and OOD samples.\n\n### **Q2: The necessity of negative labels in the initialization of the memory bank.**\n\n**A2:** Thanks for the insightful questions. Our use of negative labels to extend the memory bank is carefully designed for effective implementation. Initially, our memory bank is empty, containing only zero values. Consequently, the derived negative proxies are also zero vectors. In other words, during the early stages of testing, it is impossible to generate effective negative proxies solely based on the memory bank. To enable our method to be operational from the beginning of the testing phase, we extend the memory bank with negative labels. It is important to note that while negative labels play a crucial role during the early stages, their influence diminishes as the memory bank gradually accumulates data, and the negative proxies will be progressively dominated by cached negative images.\n\nRegarding the reviewer's suggestion to use the MCM score to judge negative samples and organize them into OOD proxies, this is indeed feasible. One could cluster the negative samples selected by the MCM score and use the cluster centers as OOD proxies. However, this method has a significant drawback: it requires detecting a portion of negative samples before applying our method. In other words, this approach cannot be applied online to the initial test data. From this perspective, we utilize the negative labels as predefined initial cluster centers in our AdaNeg. \nDuring testing, these cluster centers are gradually refined as more image features are cached into the memory.\n\nConsidering that negative labels are words semantically distant from ID labels, if OOD samples are similarly distant from both ID and negative labels, they typically fall into the intermediate hard examples between ID and negative labels. Handling such intermediate hard examples is a long-standing challenge for all methods, including NegLabel and our AdaNeg. Our AdaNeg partially addresses these hard examples via an easy-to-hard strategy. Specifically, our method first caches easy OOD samples, which are closer to negative labels and farther from ID labels, into the memory. These easy OOD samples, being closer to the intermediate hard OOD examples, can serve as bridges to facilitate the detection of hard OOD samples. The effectiveness of our method is validated by the improved OOD detection performance shown in Table 2 of the main paper.\n\n### **Q3: More detailed experimental settings.**\n\n**A3:** Thanks for the kind suggestion. In our experiments, we maintain an independent memory bank for each OOD dataset (i.e., each ID-OOD pair). In implementation, we clear the memory bank when switching to a new OOD dataset.\n\nWe validated performance using a shared memory bank across four OOD datasets: iNaturalist, SUN, Places, and Textures. In this setup, the memory bank retains features from previous OOD datasets when testing a new one. As shown in the table, shared memory banks outperform independent ones, suggesting that cached features aid in recognizing new OOD datasets. However, shared memory banks may leak information between datasets, which can be problematic in practice. Therefore, we use independent memory banks by default.\n\n| Types of Memory Bank | INaturalist | SUN  | Places | Textures | Average |\n|----------------------|:-----------:|:----:|:------:|:--------:|:-------:|\n| Independent      | **0.59**    | 9.50 | 34.34  | 31.27    | 18.92   |\n| Shared           | **0.59**    | **9.13** | **32.68** | **29.08** | **17.87** |\n\n**Table Caption**: FPR95 ($\\downarrow$) with different types of memory banks.\n\n### **Q4: Typos and symbol issues**\n\n**A4:** Thank you for the kind correction. We will correct $\\tau$ = 0.01 in L247 and unify the subscript NL as $S_{nl}$ in the revision. We will further proofread the manuscript carefully.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RIIFNogE9p", "reviewer": "Reviewer_eV4A", "summary": "The paper \"AdaNeg: Adaptive Negative Proxy Guided OOD Detection with Vision-Language Models\" presents a novel approach to out-of-distribution (OOD) detection using pre-trained vision-language models (VLMs). The primary innovation is the introduction of adaptive negative proxies, which are dynamically generated during testing by exploring actual OOD images. This method addresses the semantic misalignment issues of previous approaches that use static negative labels. AdaNeg utilizes a feature memory bank to cache discriminative features from test images, creating task-adaptive and sample-adaptive proxies that better align with the specific OOD datasets. The approach combines static negative labels with adaptive proxies to enhance the performance of OOD detection, achieving significant improvements in benchmarks like ImageNet. The method is training-free, annotation-free, and maintains fast testing speeds.", "questions": "See weakness.", "limitations": "1. Generalization to Other Domains\n2. Dependency on test data.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "1.\tInnovative Approach: The introduction of adaptive negative proxies to address semantic misalignment is a significant advancement. This dynamic generation of proxies during testing offers a novel solution to improve OOD detection.\n2.\tEffective Use of Vision-Language Models: Leveraging VLMs to integrate textual and visual knowledge enhances the robustness and accuracy of OOD detection.\n3.\tPerformance Improvement: The method shows substantial improvements in standard benchmarks, particularly a 2.45% increase in AUROC and a 6.48% reduction in FPR95 on the ImageNet dataset.\n4.\tTraining-Free and Annotation-Free: AdaNeg does not require additional training or manual annotations, making it highly efficient and practical for real-world applications.\n5.\tScalability and Efficiency: The method maintains fast testing speeds and can dynamically adapt to new OOD datasets without significant computational overhead.\n6.\tComprehensive Evaluation: Extensive experiments and analyses demonstrate the effectiveness and robustness of the proposed approach across various benchmarks.", "weaknesses": "1. Potential Overhead in Memory Management: The implementation of a memory bank for caching features may introduce significant overhead in memory management, especially when dealing with large-scale datasets or high-dimensional feature spaces.\n2. Generalization to Other Domains: Although the approach demonstrates promising results on existing public datasets, its effectiveness in other domains or with different types of data remains uncertain and requires further investigation.\n3. Testing Phase Dependency: It is unclear whether the approach can maintain the same level of reliable performance when only a small number of images are tested in practical applications. This dependency on the number of test images warrants additional examination.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer *eV4A*，\n\nWe sincerely thank you for the constructive comments and recognition on our work! Please find our responses below.\n\n### **Q1: Potential Overhead in Memory Management: The implementation of a memory bank for caching features may introduce significant overhead in memory management, especially when dealing with large-scale datasets or high-dimensional feature spaces.**\n\n **A1:** Thanks for the question.  Indeed, the introduction of a memory bank introduces additional memory overhead, which we have briefly discussed in Section 5 of the main paper. However, we clarify that our memory overhead does not continuously increase with the scale of the dataset. This is because we drop image features with high prediction entropy when the memory bank is full, as detailed in Line 184 and Appendix A.2 of our submission\n\nFor a typical high-dimensional feature of 512 dimensions and a maximum memory length $L$ of 10 for each class, our memory bank occupies a storage space of 214.75MB when using the ImageNet dataset as ID. This storage requirement is negligible compared to the memory consumption of the CLIP model during forward passes.\n\n### **Q2: Generalization to Other Domains: Although the approach demonstrates promising results on existing public datasets, its effectiveness in other domains or with different types of data remains uncertain and requires further investigation**\n\n**A2:**  Thanks for the nice suggestion. We further validate our method on the BIMCV-COVID19+ dataset [a], which includes medical images, following the OpenOOD setup [68, 71]. Specifically, we selected BIMCV as the ID dataset, which includes chest X-ray images CXR (CR, DX) of COVID-19 patients and healthy individuals. For the OOD datasets, we follow the OpenOOD setup and use CT-SCAN and X-Ray-Bone datasets. The CT-SCAN dataset includes computed tomography (CT) images of COVID-19 patients and healthy individuals, while the X-Ray-Bone dataset contains X-ray images of hands.\nAs illustrated in the table below, our AdaNeg method consistently outperforms NegLabel on this medical image dataset. We will add these analyses in the revision.\n\n| Methods          | CT-SCAN AUROC$\\uparrow$ | CT-SCAN FPR95$\\downarrow$ | X-Ray-Bone AUROC$\\uparrow$ | X-Ray-Bone FPR95$\\downarrow$ | Average AUROC$\\uparrow$ | Average FPR95$\\downarrow$ |\n|------------------|:------------------------:|:--------------------------:|:--------------------------:|:----------------------------:|:------------------------:|:--------------------------:|\n| NegLabel         | 63.53                    | 100                        | 99.68                      | 0.56                          | 81.61                    | 50.28                      |\n| **AdaNeg (Ours)**| **93.48**                | 100                        | **99.99**                  | **0.11**                      | **96.74**                | **50.06**                  |\n\n[a] BIMCV COVID-19+: a large annotated dataset of RX and CT images from COVID-19 patients\n\n### **Q3: Testing Phase Dependency: It is unclear whether the approach can maintain the same level of reliable performance when only a small number of images are tested in practical applications. This dependency on the number of test images warrants additional examination.**\n###\n**A3:** Thank you for your suggestion.  We examined the dependency of our approach on the number of test images by evaluating its performance across different scales of test samples.  As the number of test samples increases (from 900 to 90K), the cached feature data also increases, leading to an improvement in our method's results, as shown in the table below. Even with a small number of test samples (e.g., 90 and 900), our method significantly reduces FPR95 compared to NegLabel, demonstrating its robustness across different numbers of test images.\n\nNote that with only 90 test images, the task of distinguishing between ID and OOD samples degenerates into a simpler task since the number of test images is even smaller than the number of classes (e.g., 1000 for ImageNet). Consequently, both NegLabel and our method achieve lower FPR95 in such an easier scenario.\n\nWe will add these findings to the revised paper.\n\n| Num. of Test Images | 90   | 900  | 9K   | 45K  | 90K  |\n|---------------------|------|------|------|------|------|\n| NegLabel            | 14.00| 20.44| 20.71| 20.51| 20.53|\n| **AdaNeg**          | **6.00** | **10.12** | **9.78** | **9.66** | **9.50** |\n\n**Table Caption:** FPR95 ($\\downarrow$) with different numbers of test images, where test samples are randomly sampled from ImageNet (ID) and SUN (OOD) datasets while maintaining their relative proportions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
## **Common Responses to All Reviewers**

**Dear Reviewers, Area Chairs, and Program Chairs:**

We are grateful for the reconstructive comments and valuable feedback from the reviewers. We are glad that the reviewers found our idea novel (Reviewers ACdF and eV4A) and design interesting (Reviewer Pmx9), and we appreciate their recognition on the wide scalability (Reviewers ACdF and eV4A), comprehensive evaluation (Reviewers yxd9 and eV4A), and substantially improved performance (Reviewers yxd9 and eV4A) of our method.

To address the reviewers' concerns, we have attached an updated Figure 1(a) in the enclosed PDF file and provided additional experiments and analyses on training-based methods, misclassified samples, different setups of test data, and medical image datasets. Please find our itemized responses to all reviewer’s comments below, and we sincerely hope our responses can well address the reviewers' concerns.

Best regards,

Authors of Paper9248

Meta-review: The paper was reviewed by four experts in the field. The paper mostly reviewed positive comments from reviewers on innovation, scalability, interesting design, and easy-to-follow paper. Among the concerns described by reviewers were analyses on stability, necessity of negative labels, unclear motivation, and innovativeness. The rebuttal attempted to address all the concerns and reviewers acknowledged that most of their concerns (e.g., not very clear motivation, possibly limited innovation) were adequately addressed. Therefore, AC recommends the paper for acceptance. Authors are encouraged to include important points from rebuttal and post-rebuttal discussion into the final version of the paper. We congratulate authors on the acceptance of their paper at Neurips 2024.
Decision: Accept (poster)