Negative Label Guided OOD Detection with Pretrained Vision-Language Models
Authors: Xue Jiang
Feng Liu
Zhen Fang
Hong Chen
Tongliang Liu
Feng Zheng
Bo Han
Venue: ICLR 2024 spotlight (2024)
Keywords: OOD detection
Primary area: societal considerations including fairness, safety, privacy

Abstract: Out-of-distribution (OOD) detection aims at identifying samples from unknown classes, playing a crucial role in trustworthy models against errors on unexpected inputs.  
Extensive research has been dedicated to exploring OOD detection in the vision modality. 
{Vision-language models (VLMs) can leverage both textual and visual information for various multi-modal applications, whereas few OOD detection methods take into account information from the text modality. 
In this paper, we propose a novel post hoc OOD detection method, called NegLabel, which takes a vast number of negative labels from extensive corpus databases. We design a novel scheme for the OOD score collaborated with negative labels.
Theoretical analysis helps to understand the mechanism of negative labels. Extensive experiments demonstrate that our method NegLabel achieves state-of-the-art performance on various OOD detection benchmarks and generalizes well on multiple VLM architectures. Furthermore, our method NegLabel exhibits remarkable robustness against diverse domain shifts. The codes are available at https://github.com/tmlr-group/NegLabel.

Published as a conference paper at ICLR 2024
NEGATIVE LABEL GUIDED OOD DETECTION WITH PRETRAINED VISION-LANGUAGE MODELS
Xue Jiang! 1.
Feng Liu Feng Zhengl* Bo Han
Zhen Fang*
Hong Chen
Tongliang Li
2TMLR Group, Hong Kong Baptist University 8TMLR Group, University of Melbourne *University of Techhnloog Sydney
Sootthern University of Science and Technology
$ Huazhong Agricultural University "Mohamed bin Zayed University of Artificial Intelligence "Sydney Al Centre, University of Sydney
(csxjiang, bhanml| } 8 @comp. hkbu.................... edd.. hk, feng. lilllllull @un mom lb, eedu.uu zhen. fang@uts edu au, chenh@mail www...hauu... edu.cn, cn, tongliang liuesydnneee edu. au, f. z.zzngieeeeeeooo org
ABSTRACT
Oft-of-distribution (OOD) detection aims at identifying samples from unknown classes, playing a crucial role in trustworthy models against errors on unexpected inputs. Extensive research has been dedicated to exploring OOD detection in the vision modality. Vision-languag models (VLMs) can leverage both textual and visual information for various multi-modal applications, whereas few OOD detection methods take into account information from the text modality. In this paper, we propose novel post hoc OOD detection method, called NegLabel, which takes a vast number of negative labels from extensive corpus databases. We design novel scheme for the OOD score collaborated with negative labels. Theoretical analysis helps to understand the mechanism of negative labels. Ex- tensive experiments demonstrate that our method NegLabel achieves state-of- the-art performance on various OOD detection benchmarks and generalizes well on multiple VI.M architectures. Furthermore, our method LegLa exhibits re- markable robustness against diverse domain shifts. The codes available at attps: /ttpp:://gttaan whtb.tb.t.ttgggg/ggg com/tmle http:/mmlllrrrrr///e Caoup//eeLabll.
INTRODUCTION
In open-world scenarios, deploying machine learning models faces a critical challenge: how to handle data from unknown classes, commonly referred to as out-of-distributioo (OOD) data (Hendrycks Gimpel, 2017). The presence of OOD data can lead to models exhibiting overconfidence, potentially resulting in severe errors or security risks. This issue is particularly pronounced critical applications, such autonomous vehicles and medical diagnosis. Therefore, detecting and rejecting OOD data plays a crucial role in ensuring the reliability and safety of the model.
Traditional visual OOD detection methods (Hsu et al., 2020a: Wang al., 2021b; Huang et al., 2021; Sun et al., 2021; Wang et al., 202 la) typically rely solely on image information, ignoring the rich textual information carried by labels. Vision-languag models (VLMs) can leverage multimodal information, which also beneficial for OOD detection. Some recently proposed methods attempt to design dedicated OOD detectors for VLMs. Specifically, ZOC (Esmaeilpour et al., 2022) defines the new task - zero-shot OOD detection, and uses a trainable captioner to generate candidate OOD labels match OOD images. However, when dealing with large-scale datasets encompassing multitude of in-distribution (ID) classes, like ImageNet- 1k. the captioner may not generate effective candidate OOD labels, resulting in poor performance. MCM (Ming et al., 2022a) uses the maximum logit of scaled softmax to identify OOD images. However, MCM only employs information from the ID label space and does not effectively exploit the text interpretation capabilities of VLMs. Therefore, there exists untapped potential for enhancing OOD detection utilizing VL.Ms.
In this paper, we design method to leverage better the knowledge carried by VLMs for OOD detection. To achieve this, we introduce a large number of negative labels that enable the model to
"Correspondence to Feng Zheng (f.zheng@ @iece.org)

---

Published as a conference paper at ICLR 2024
In-diotrittton Tabale y
bird
distnhurcion Image Similanttes
Smilery
Negative Labels yy
plass +labdo.
Encoder
bost
plano
Insezt
Negalve
6v
Ous-of-dieriistion Emage Similaniizss
In-distribution Image
Image
Encader
Indioonhation Tabels
Tabels
Oui-of-ddittrbutton Image
A2
Porrs
Pots
PoP:
Texp(n)
(B) Ex)p(B)
simck 5)
Figure 1: Overview of NegLabel. The image encoder extracts input images into image embeddings h. The text encoder extracts ID and negative labels into text embeddings e. All encoders are frozen in the inference time. The negative labels are selected by NegMining (see Section 3.1) from large-scale corpus. The image-text similarities are quantified through h e, represented by purple blocks (darker shades indicating higher similarity). The right part illustrates that ID images tend to produce lower similarities with neagative labels than OOD images. Our NegL.abel score (see Section 3.2) fuses the similarities of image-II) labels (green) and image-negation labels (blue).
distinguish OOD samples in more nuanced and detailed manner. Our proposed method NegLabel detects OOD samples by examining their affinities between ID labels and negative labels. As shown in Fig. I, ID samples have higher OOD scores than OOD samples due the affinity difference.
Specifically, we design the NegMining algorithm to select high-quality negative labels from extensive corpus databases. This algorithm utilizes the distance between negative label and ID labels metric to assess their adequacy in terms of semantic divergence. By selecting negative labels with sufficient semantic differences from ID labels, the algorithm enhances the separability between ID and OOD samples. Additionally, we introduce novel scheme for the OOD score, which is positively correlated with the affinities between images and the ID labels and negatively correlated with those of the negative labels. The score combines the knowledge from the II) and negative label space, thus better leveraging the VL.Ms' capabilities of comprehending text. Furthermore, we provide a theoretical analysis to aid in understanding the mechanism of negative labels.
Extensive experiments demonstrate that our method NegLabel achieves state-of-the-ar performance on various zero-shot OOD detection benchmarks and multiple VLM architectures, like CLIP (Radford et al., 2021), ALIGN (Jia et al., 2021), Group ViT (Xu et al., 2022), and etc. Combining with CLIP, NegLabel achieves 94.21% AUROC and 25.400 15.40% FPR95 in the large-scale ImageNet-1k OOD detection benchmark (Huang et al., 2021), even surpasses numerous existing methods in fully- supervised settings. Furthermore, NegLabel exhibits remarkable robustness against diverse domain shifts. In summary, our key contributions are as follows:
Our proposed framework, NegLabel, introduces massive negative labels to boost OOD detection. The extended label space provides novel perspective to distinguish OOD samples, leveraging extra clues by examining the similarities between images and the extended labels.
We propose the NegMining algorithm (see Section 3.1) to select high-quality negative labels to enhance the distinction between ID and OOD samples. Based the picked negative labels, design new scheme for the post hoc OOD score (see Section 3.2) that effectively leverages the knowledge encoded in VLMs.
Extensive experimental analyses (see Section 4.2) demonstrate that our method, NegLabel, achieves state-of-the--rt performance multiple OOD detection benchmarks. Moreover, NegLabel demonstrates strong generalization capabilities across various VLM architectures. NegLabel also shows remarkable robustness in the face of diverse domain shifts.
PRELIMINARIES
Let X' and y (vi,..., y/K be the image space and ID) label space, respectively. Note that y is a set containing words, e.g., ) {cat, dog, ... bird), and K is the number of IDD classes. Given the ID feature random variable X E X and OOD feature random variable Xout E X, we use the Pxi and P xwr present the ID marginal distribution and OOD marginal distribution, respectively.
CLIP-like models (Radford et al., 2021). Given any input image x ~ Pxin and any label yj E y, we extract features of x and yj using the CLIP-like model f, which consists of image encoder fing

---

Published as a conference paper at ICLR 2024
Algorithm 1: NegMining
Input :Candidate labels y ID labels V/
Text encoder feext
Output :
:Negative labels y
8
// Calculate text embeddings for y: E y do
L 6 = flext (promp((y:)):
kodke as Labsk
for y) E 3 do
Negulie Labis
6; frext (prompt(())))
Paum Decmm
// Measure candidate-ID label distance.
Figure 2: Illustration of NegMining, The al- gorithm selects negative labels with larger dis- tances (lower similarities) from the IID labels. Darker blue squares represent the higher priori- ties to be picked. Dashed squares represent neg- ative labels that are impossible to be selected.
d percentile, cos(ē, ex)
Choose M negative labels from top-k distances.
y = topk([d], d2.... dc), M).
and text encoder flext. Then the extracted features h E TR and ej E ERD are
h fimg(()) Ej
frext (prompt Vj = 1,,,.....
where prompt(-) represents the prompt template for input labels, and D the embedding dimension. In zero-shot classification, the goal is to predict the correct label or class for image without prior training on that specific class. CLIP-like models' predictions are based on the cosine similarity of the image embedding features h and text embedding features e1, -2. ,eK:
= arg max (cos( h, e₁)}, where e, = -f ptext (promp((y)))
(2)
My Ey
Score functions. Following many representative OOD detection methods (Huang et al., 2021; Liang et al., 2018; 8; Liu al., 2020), given threshold 7 and score function S, then is detected as II) data if and only if (x) 2 7:
(x ID,, if if 2 y; otherwise, Gx,() OOD.
The performance of OOD detection depends on how to design a score function S to make OOD) samples obtain lower scores while IDD samples have higher scores.
Problem Setting. Zero-shot image classification for a pre-trained CLIP-like model is to classify images into one given label names (i.e., ID classes), without any fine-tuning on ID classes (Radford et al., 2021). In this paper, we work with pre-trained CLIP-like model and specific set of class labels or names for a zero-shot classification task. Therefore, the primary objective of VLM-based OOD detection in our paper is to identify samples that do not belong to any known ID classes as OOD without sacrificing ID classification accuracy.
METHODOLOGY
As mentioned in Section the semantic labels of ID samples are confined within the bounds of V whereas OOD labels lie outside this label space. In contrast to traditional image classification tasks, zero-shot classifiers are trained on more extensive datasets (Radford et al., 2021; Jia et al., 2021) to gain a broader understanding of the training data. However, when conducting zero-shot image classification, the ID label space y can weaken the model's ability to detect OOD samples, since the limited label space may not fully capture the differences between ID) and OOD samples, ultimately reducing the model's sensitivity.
As zero-shot classifier, CLIP-like model can scale its label space using any text prompts with minimal overhead cosine similarity calculations. Thus, the central concept behind NegLabel is to enlarge the model's zero-shot generalization ability by expanding the label space. By capitalizing the scalability of the model's 's label space, we can effectively utilize the VLMs' text comprehension capabilities for zero-shot OOD detection.
3.1 SELECTION OF NEGATIVE LABELS
If there are labels y & y that have lower affinities with ID images compared to OOD images, they provide hints for detecting OOD samples. These labels are referred to as negative labels, denoted by y E = and the objective of selecting negative labels is to increase the gap between y and y-

---

Published as a conference paper at ICLR 2024
The large-scale corpus database contains almost all common concepts and entities worldwide. Its semantic space effectively exploit the capacity of CLIP-like zero-shot models, allowing all input samples to be mapped to richer labels. As shown in Fig. 2, when selecting negative labels from the corpus, words with high similarity the ID label space should be excluded so that the selected negative labels have lower affinities with ID samples than OOD samples. More analysis can be found in Appendix B.4. Then, we propose the NegMining algorithm select suitable negative labels.
Given a large group of words' (contains tens of thousands of nouns, adjectives, etc.) fetched from lexical database like WordNet (Fellbaum, 1998), we can construct a candidate label space y (TI, V2.... Wc J. where C' is the total number of the candidate words. For each Tt = 1, 2, ..... C), we calculate the negative cosine similarities as the distance measure between the candidate label and the whole ID label space:
d, percentile, (fdux)
cos(ē;, co ex)
where ē; ftext (7.) ) is the text embedding of the candidate word Di. the K is the number of ID classes, and percentile,, 77 E [o, represents the 100p-th percentile of the data. When 1j = 0 means the minimum distance, we usually choose 17 = 0.05. We use the percentile of the distance between the candidate label y; and all ID labels Y instead of the minimum distance as the metric to obtain robustness against outlier ID labels, shown in Fig. 2.
Then we design the negative label selection criteria as y = topk((d1, d2.... dc),yo, MM. operator topk( A, B, M) first identifies the indices of the top-M elements in set A, and then selects the corresponding elements from set B based on these indices. The details are shown in Algorithm 1. In this work, unless noted otherwise, we use M = 10000 the typical number of negative labels, which can achieve robust performance on various benchmarks.
;e),),,M. The
seems that it is better to directly select the negative labels far away from the ID image embeddings. However, in the zero-shot setting, the ID marginal distribution Pxis is unreachable, i... there is no training data to guide the selection of negative labels. Fortunately, the CLIP-like model minimizes the embedding feature distance between paired image-text in a contrastive learning framework (Radford et al., 2021). Hence, we use ID labels to search for OOD concepts as negative labels that are far away from ID labels.
3.2 OOD DETECTION WITH NEGATIVE LABELS
NegLabel Score. Given the selected negative label set y we merge them with the ID labels y to obtain extended label space ) y U.- Then, the extended labels are fed into the text encoder of CLIP to obtain text embeddings. CLIP calculates the cosine similarities between the text and image embeddings. The proposed OOD score can be formulated as
S(x) S* (sim(x,)) ))) sim(x,)")
where (----) represents a fusion function that combines the similarity of a sample with ID labels sim(x, V) and the similarity of the sample with negative labels sim(x. y²).
As analyzed in Section 3.1, the selected negative labels are far from D, so ID samples have lower similarity with negative labels than those of OOD samples. Therefore, the design criterion of S* can be summarized as follows: S" positively correlated with the similarity of the sample the ID label space and negatively correlated with the similarity of the sample in the negative label space.
In the extended label space yextt the OOD detection task can considered as the model's confidence that the image belongs to y. A natural design is to use normalization function to check the proportion of the similarity sim(x, (x, J) in the extended label space yext, Here, we propose a NegLabel score in a sum-softmax form follows, and more designs are discussed in Appendix A.5:
K
M
S(x) =
co/(h
+
cos(h,,,)//))
(6)
where K is the number of ID labels, 7 is the temperature coefficient of the softmax function, and M is the number of negative labels. Based on the analysis in Section 3.1, the affinities of the ID
For the lexical datahase, we only retained nouns and adjectives as candidate labels hecause most of the other words either have their corresponding nouns or adjectives, or do not represent specific concepts or entities.

---

Published as a conference paper at ICLR 2024
images to the negative labels are lower than those of the OOD images. Therefore, the ID samples will generate higher similarities the ID labels, resulting in higher OOD scores.
Eq. (6) provides two special considerations for CLIP-like models:
Firstly, the cosine function clamps the similarity in [-1, 1] and results in even smaller divergences between positive and negative pairs (e.g., the cosine similarities of positive pairs in CLIP are typically around 0.3 and that of negative pairs are around 0.1 (Ming al., 2022b)). Such small differences may not adequately reflect the confidence variance of ID and OOD samples. Therefore, a temperature coefficient is introduced to scale the similarity, reflecting the model's confidence better.
Secondly, zero-shot models are not finetuned on task-specific ID data. Their classification perfor- mance is typically weaker than the fully supervised models, making it more susceptible super-class confusion (c.g., mistaking Husky for an Alaskan Malamute). The numerator of Eq. (6) uses the sum of the sample's ' similarity all ID labels, making the OOD score more robust to super-class confusion. This design also improves the robustness of the zero zero-shot OOD detector to domain shifts, which will be analyzed in Section 4.2.
Grouping Strategy. In Section 3.1, we introduce the NegMining algorithm, which involves picking words that are as far away from the ID) label as possible to expand the label space and avoid high similarities between ID images and negative labels (called false positives). Nevertheless, false positives may occur in the ID sample due to limited model performances and semantic overlaps between ID images and negative labels. This risk increases when more negative labels are used, increasing the variance of the OOD score. To balance the trade-off between the additional knowledge gained from negative labels and the risk of false positives, we propose grouping strategy divide negative labels into multiple groups and calculate the OOD score as Eq. (6) in each group separately, and then obtain the average OOD score across groups.
Specifically, we start by evenly dividing the selected M negative labels into Ng groups. Each group contains [M/n,, /ng] negative labels, and any remaining negative labels discarded. We then calculate a separate NegLabel score for each group. Finally, we sum up the scores of all groups and take the average as the final NegLabel score. The grouping strategy limits the risk of false positives within each group, and further smoothes this noise through averaging across groups. More detailed analysis can be found in Appendix A.5.
3.3 UNDERSTANDING NEGATIVE LABELS FROM THE PERSPECTIVE OF MULTI-LABEI. CLASSIFICATION
To better understand how negative labels can facilitate zero-shot OOD detection, we provide theoreti- cal analysis from the perspective of multi-label classification to demonstrate that negative labels can improve the separability of ID and OOD samples.
For VLMs like CLIP, whether an image and a label match is determined by calculating the similarities between their embeddings. To simplify the analysis, we assume that the selected negative labels, y E y" are independently and identically distributed (we will discuss the validity of this assumption later). For each y/ E y i 1, M, the input image x has an implicit distribution noted as (s;; f. x, Vt) for its similarity score 8; sim(x, x, V.). By applying a threshold y,, the similarity score can be converted into binary label s 1 [s, 2w) Thus, we can define p = P (s; 2 eif x, 7%) as the probability of classifying the image as positive for the given label T.. The introduced negative labels can be regarded as an M M-way multi-label classification task on )
Under the multi-labo classification setting, for each negative label T. i = 1, 2, ..., M, the single-label classification result of an input image x follows the Bernoulli distribution with parameter p. where p is the probability that the label of x is Vi- Hence, based on the relation between Bernoulli and binomial distributions, for the input image x, the positive count = Σ3 st among M negative labels follows the binomial distribution B(M, ?)? Specifically, we define the in-distribution positive count
HHoweeer, due to the large number of negative labels, covering wide semantic space, their affinity with the sample is variable. Thus, we provide more general case based on Poisson binomial distribution in Appendix B.6. Specifically, we assume that the probability an image x matching with a negative label is pi. where different negative labels have different probabilities. This better reflects the real-world scenario, and thus, we consider the number of matches between an image x and them to follow a Poisson binomial distribution.

---

Published as a conference paper at ICLR 2024
as cm ~ (1M, ~ Pxin, and the citto--distribttion positive count as c ((M, P2), ~ P Xouu. Recall that the affinities between ID images and negative labels are lower than that of OOD images, so we have p1 < P2 holds.
Next, consider toy OOD score function that only takes the positive count as criteria, i... S(x) -, which is special case of Eq. (5). Then we have S B(M, and Sout B(M, Because the number of negative labels is sufficiently large, according to the binomial approximation rules 3, the OOD scores Sin and Suut can be regarded following the normal distribution pM Mp1 (1 - Pı)) and N (Mp2, Mp2(1 - P2)). respectively.
We use FPRA as the performance metric of OOD detector to show the separability between the ID and OOD samples. The FPR, metrie in relation to the toy function S can be defined as follows:
FPR; 4 4 [x: /Mp p1] ;P2,, v MM2 P2)
1 erf (1 (1 (1-p)) P2 (1
(1/((1170) erl e2x -
VM (pi P2)
2 +
2
2pp (1 P2)
where T [[ a) represent the CDF of normal distribution with μ as mean and a as standard deviation, er(( = - 10¹ e- dl represents the error function.
We target investigating the relationship between the FPRx metric and the number of nega- tive labels M. Therefore, we calculate the partial derivative with respect to M and use z
(1-p) 1 1) v 2222(1-22)
VM(Pi-Pa simplify the expression:
V p2(1-p2)
OFPR 1 der((z) Oz 0M
P2
< 0.
(8)
2
Oz
0M
2
VMp2 (1-p2)
It is evident that as M increases, FPRA consistently decreases. This observation suggests that the performance of the OOD detector is enhanced by incorporating more negative labels (note that Pi P2 0). Consequently, the inclusion of negative labels provides additional information that aids in distinguishing between ID and OOD samples. Furthermore, Eq. (8) is negatively correlated with
P2 Pr, i.e., the difference between the probability that OOD and ID samples are positively classified w.r.t. the negative labels. This indicates that selecting negative labels with significant differences in similarity with ID and OOD samples can help improve OOD detection performance, and it explains the role of the NegMining algorithm.
Finally, we discuss the plausibility of this theoretical modeling. Readers may notice that as M increases, z approaches +oo and FPRA approaches O. However, this does not mean that a large number of negative labels can infinitely optimize the performance of the OOD detector, as the semantic space formed by images is limited. With M increases, the newly added negative labels either become less differentiated on ID and OOD images or exhibit semantic overlap with the previous label. Consequently, this leads or in some cases, these additional labels may no longer satisfy the assumption of independent and identically distributed data. In particular, if the similarities between negative labels and ID images larger (i.e., P1 > P2). it will cause an increase in FPR,, The detailed derivations of Eqs. (7) and (8) can be found in Appendix B.5 and more theoretical analysis can be found in Appendix B.6.
decline in the difference between the probabilities P2 and Pl
EXPERIMENTS
4.1 EXPERIMENT SETUP
Datasets and benchmarks. We evaluate our method on the ImageNet- OOD benchmark (Huang et al., 2021) and compare it with various previous methods. The ImageNet- OOD benchmark is a widely used performance validation method that uses the large-scale visual dataset ImageNet- Ik as ID data and iNaturalist (Horn et al., 2018), SUN (Xiao et al., 2010), Places (Zhou et al., 201 8), and Textures (Cimpoi et al., 2014) as OOD data, covering diverse range of scenes and semantics. We
'If Mp 2 and M(1 - p) 2 5, the binomial distribution can he approximated by Gaussian distribution (Parameswaran, 1979).

---

Published as a conference paper at ICLR 2024
Table I: OOD detection performance comparison with baselines. All methods are based on CLIP- B/16. The ID data are ImageNet- 1k. All values are percentages. 1 indicates larger values are better and J. indicates smaller values are better. The shadow part represents our method.
0OD Dataset
Average
Methods
iNaturalist AUROC FPR95]
SUN
Places
Textures
AUROCT
FPR951 AUROCT FPR95,
AUROCT FPR95]
AUROCT
FPR95,
Requires
training (or w. line-tuning)
MSP (Hlendrycks & Gimpel, 2017)
67.44
58.36
79.73
73.72
79.67
74.41
79.69
71.93
81.63
69.61
ODIN (Linng et al, 2018)
94.65
30.22
87.17
54.04
85.54
55.06
87.85
51.67
88. 80
47.75
Energy (Liu al., 2020)
95.33
26.12
92.66
35.97
91.41
39.87
86.76
57.61
91.54
39,89
GradNorm (Hunng et al., 2021)
72.56
81.50
72.86
82.00
73.70
80.41
70.26
79.36
72.35
80.82
VIM (Wang
2022)
93.16
32.19
87.19
54.01
83.75
60.67
87.18
53.94
87.82
50.20
KNN (Sun et al., 2022)
94.52
29. 17
92.67
35.62
91.02
39.61
85.67
64.35
90.97
42.19
VOS (Du al., 2022)
94.62
28.99
92.57
36.88
91.23
38.39
86.33
61.02
99.19
41.32
NPOS (Tao et al., 2023)
96.19
16.58
90.44
43.77
89.44
45.27
88.80
46.12
91.22
37.93
Zero-shot (no training required)
Mahalanobis (Lee al., 2018)
55,89
99.33
59.94
99.41
65.96
98.54
64,23
98.46
61.50
98.94
Energy (Liu et al., 2020)
85.09
81.08
84.24
79.02
83.38
75.08
65.56
93.65
79.57
82.21
ZOC (Esmacilpoor et al. 2022)
86.09
87.30
81.20
81.51
83.39
73.06
76.46
98.90
81.79
85,19
MCM (Ming et al., 2022a)
94.59
32.20
92.25
38.80
90.31
46,20
86.12
58,50
90.82
43,93
CLIPN (Wang et al., 2023)
95.27
23.94
93.93
26.17
92.28
33.45
90.93
40.83
93.10
31.10
NegLabel
99.49
1.91
95.49
20.53
91.64
35.59
90.22
43.56
94.21
25.40
also follow the settings of MCM (Ming et al., 2022a) and use Stanford-Cars (Krause al., 2013),
CUB-200 (Wah et al., 2011), 1). Oxford-Pet (Parkhi et al., 2012), Food-101 (Bossard et al., 2014), and some subsets of ImageNet- Ik (Deng et al., 2009) as ID data, and iNaturalist, SUN, Places, and Textures as OOD data to conduct validation experiments on our method. MCM uses subsets of ImageNet- 1k for ane-grained analysis. For example, MCM constructs ImageNet- 10, which mimics the class distribution of CIFAR-10 10 (Krizhevsky, 2009) but has high-resolution images. Each OOD dataset has no classes that overlap with the ID dataset.
Implement details. In this paper, we implement Negl laeel on various VLM architectures, including CLIP (Radford et al., 2021) ), ALIGN (Jia et al., 2021). Group VIT (Xu et al., 2022), AICCIP (Chen al., 2022). Unless otherwise specified, we employ CLIP-B/16 for zero- shot OOD detection, and the hyperparameters use the following default configurations. The NegMining algorithm takes WordNet as the corpus and selects M 10000 negative labels under 1) = 0.05. We use the NegLabel score in the sum- sum-softmax form and set = 0.01 as the temperature, and use ng 100 for grouping strategy by default. More experiment settings can be found in Appendix A.1.
Computational cost. NegLabel is post hoc OOD detector with about O(2MD) FLOPs extra computational burden per image, which usually introduce < 1% network forward latency. When taking M = 10000 negative labels, it takes ~ mm per sample for OOD detection. NegLabel detects OOD samples in parallel with the zero-shot classification task without any impact on ID classification performance. The NegMining algorithm selects and dumps the negative label before the inference phase, and it usually takes a few minutes to process a large-scale corpus on a single 3090Ti GPU.
4.2 EXPERIMENTAL RESULTS AND ANALYSIS
OOD detection performance comparison with baselines. We compare our proposed method, NegLabel, with other existing OOD detection methods in Table 1. The methods compared with include MCM, a method specifically designed for zero-shot OOD detection, as well as traditional methods that were re-implemented using finetuned CLIP on the ImageNet- 1k, see Appendix A.1 for more details. Our method NegLabel achieves the state-of-the-art on the ImageNet- Ik benchmark, which highlights its superior performance in the zero-shot setting. Furthermore, our method can surpass traditional methods with a finetuned CLIP, demonstrating CLIP's strong OOD detection capabilities in zero-shot scenarios. This is because CLIP can parse images in fine-graine manner, which is achieved through its pre- pre-training on a large-scale image-text dataset. Additionally, our choice of negative labels provides more semantic information, which further enhances the separability of ID and OOD samples. It is worth noting that our proposed method achieved surprisingly high performance on the iNaturalist dataset, and we analyze the reason in Appendix B.1. More discussions about ZOC and CLIPN are in Appendix B.2 and Appendix B.3. We also follow MCM to conduct experiments on the small-scale benchmark in Appendix A.2. The further investigations with some case studies can be found in Appendix B.4.
Zero-shot OOD detection performance comparison on hard OOD detection. Following MCM's setup, we also explore the performance of NegLabel on hard OOD detection tasks, as shown in Table 2. For semantically hard OOD detection, we alternate using ImageNet- 10 and ImageNet-20 as

---

Published as a conference paper at ICLR 2024
Table 2: Zero-shot OOD detection performance
MCM Segt aostatel
comparison on hard OOD detection tasks.
70-
1 
ID Daravet
OOD Dataset
Method
AUROCT
FPR95)
50-
ImageNet-10
ImageNet-20
MCM
98. 98.60
6.00
Negl.ahel
98.86
5.10
20
ImageNet-20
ImageNet-10
NegLabel
MCM
98.81
13.04
4.60
ImageNet-10
ImageNNtt1100
MCM
99.39
2.50
20-
Neglaahll
99.51
1.68
ImageNot.4
ImageNitR
Imapotett Sketch
imageNettz
ImageNet- 100
ImageNet-10 10
MCM
60.00
NegLabel
90.19
40.20
Figure 3: Zero-shot OOD detection performance
WaterBinds
Spurious OOD
NegLaaell
MCM
94.67
93.30
14.45
9.50
robustness domain shift.
ID and OOD data, as well as using ImageNet-10 and ImageNet- 100 to mimic the setting proposed by (Fort et al., 2021) with high-resolution images. The results in Table 2 show that NegLabel can significantly outperform MCM in all four experimental settings, demonstrating that NegLabel has strong discriminative power for semantically hard OOD data. For spurious OOD detection, since MCM does not provide specific parameters for generating spurious OOD samples, we reproduce one setting from Ming et al. (2022b) for experimental comparison. The last row of the experimental data also shows that NegLabel can surpass MCM in zero-shot spurious OOD detection tasks.
Robustness to domain shift. To test the domain generalization capability, use several versions of ImageNet with diverse domain shifts as ID samples. OOD samples are still taken from iNaturalist, SUN, Places, and Textures. The experimental details can be found in Appendix A.3 and the average results on four OOD datasets are shown in Fig. 3. It is apparent that the zero-shot OOD detection performance of MCM significantly deteriorates across diverse domain shifts, indicating the difficulty of zero-shot OOD detection under such conditions. Nevertheless, our method NegLabel continues to achieve noteworthy performances cross diverse ID datasets, thus demonstrating its remarkable robustness against diverse domain shifts. MCM heavily relies on the model's classification ability on the ID dataset. Therefore, when a domain shift leads a decrease in ID classification accuracy, the OOD detection performance of MCM also decreases accordingly. In contrast, our approach NegLabel treats the ID labels y as a whole, which less affected by inter-class confusion, making it more robust to domain shifts.
Ablation on each module in NegLabel. We conduct an ablation analysis on each module in our proposed zero-shot OOD detection framework, as shown in Table 3. Note that when NegMining is not used, negative labels are randomly selected from the corpus.
Comparing the second and fourth rows, negative labels selected by NegMining show +1.81 81% AUROC and -6.63% FPR95, indicating that negative labels require greater semantic differences from ID labels. Comparing the second and third rows, the grouping strategy only provides a slight improvement when MegMining is not used. However, comparing the first and fourth rows, yields +0.45% AUROC and -2.79% FPR95 with selected negative labels, further demonstrating that the grouping strategy aims to reduce coincidental false positives between ID images and negative labels. The first row and the last three rows demonstrate that an increasing number of negative labels gradually improves the performance of OOD detection and then declines. This indicates that NegMining prioritizes selecting negative labels that are farther away from the ID labels, as they are more discriminative for detecting OOD images. More detailed analyses can be found in Appendices A.4 and A.5.
Ablation on VLM architectures. We evaluate NegLabel with different VLM architectures and the results are shown in Table 4. The first part shows results with different backbones of CLIP, illustrating that the larger backbone boosts the performance of OOD detection. The second part shows NegLabel can generalize well to various VLM architectures. It is important to note that take the same hyperparameters across all architectures, demonstrating the robustness NegLabel's hyperparameters different VLM architectures.
RELATED WORKS
Traditional OOD detection. There exist two major categories of visual OOD detection: training-time
regularization (Hsu et al., 2020a: Bitterwolf et al., 2020; Wang et al., 2021b; Bitterwolf et al., 2020; Liu et al., 2019; Vaze et al., 2022; Yang et al., 2021) and post hoc methods (Liang et al., 2018: Liu et al., 2020; Hendrycks & Gimpel, 2017; Lee et al., 2018; Wang et al., 2021a; Jiang et al., 2023).

---

Published as a conference paper at ICLR 2024
Table 3: Module ablation in NegLabel. All methods are based on CLIP-B/IE The ID data are ImageNet- 1k. More can be found in Appendices A.4 and A.5.
Table 4: Comparison with different VLM ar- chitectures on ImageNet- Ik (ID). Detailed re- sults can be found in Appendix A.6.
Number of Negative Labels
Grouping Strategy
Architecture
Backhone
AUROC1
FPR95
NegMining
AUROC
FPR95
ResNer50
92.97
30.70
10,000
/
V
94.21
25.40
Re
93.43
29.50
10,000
X
x
91.95
34.82
CLIP
ViT-B/32
93.67
27.92
10,000
x
/
92.01
34.69
ViT-B/16
94.21
25.40
10,000
/
X
93.76
28.19
VIT-L/14
94.47
24.81
20,000
1,000
>
/
/ 91.60
/
93.85
36.14
26.56
AIICIIP
ALIGN
EfficientNet-B7
VIT-L/14
94.60
90.76
23.63
39.66
50,000
92.70
31.67
Group VIT
GroupViT 91.72
33.10
Training-time time regularization methods assume that a portion of OOD data can be accessed during model training. ConfBranch VDeVries & Taylor, 2018) constructs an additional branch from the penultimate layer of the classifier to learn the ID/OOD discrimination task. The confidence output from this branch serves as the OOD score during the inference time. G-ODIN (Hsu et al., 2020b) decomposes prior knowledge to model the probability of ID-ness. CSI (Tack et al., 2020) optimizes the OOD detector using contrastive learning. MOS (Huang et al., 2021) groups all categories in advance and adds an additional class to each group to redesign the loss for training. VOS (Du et al., 2022) generates better energy scores by synthesizing virtual anomalies. LogitNorm (Wei et al., 2022) provides an alternative loss for cross-entropy loss that separates the influence of logit norm from the training process. CIDER (Ming et al., 2023) improves OOD detection performance by optimizing contrastive loss based on the distance between class prototypes and the distance between samples and class prototypes the basis of KNN (Sun et al., 2022).
Post hoc methods do not change the model's parameters and are typically accomplished by designing the OOD score. MSP (Hendrycks & Gimpel, 2017) uses the maximum predicted softmax probability as the OOD score. ODIN (Liang et al., 2018) enhances MSP by perturbing the inputs and rescaling the logits. Energy (Liu et al., 2020) proposes use the energy function (LeCun et al., 2006) for measuring OOD-ness. ReAct (Sun et al., 2021) further improves the performance of the Energy score by feature clipping, while DICE (Sun & Li, 2022) improves the performance of the Energy score by discarding most salient weights in the fully connected layer. Mahalanobis (Lee et al., 2018) computes the minimum Mahalanobis distance between the feature and the class-wise centroids as the OOD score. GradNorm (Huang et al., 2021) designs the OOD score in the gradient space. Lastly, ViM (Wang et al., 2022) combines the norm of feature residual with the principal space formed by training features and the original logits to compute the degree of OOD-ness. KNN (Sun et al., 2022) investigates the effectiveness of parammeeettiic nearest-neighhor distance for detecting OOD samples. ASH (Djurisic et al., 2023) works by pruning large portion of input sample's activation and lightly adjusting the remaining.
CLiP-based OOD detection. The concept of zero-shot outlier exposure, as defined by (Fort et al., 2021), assumes that the labels of OOD images are known and uses them as candidate labels to complete the outlier exposure task. Recently, there has been some research into the problem of zero-shot OOD detection. For instance, ZOC (Esmaeilpour et al., 2022) proposes using a trainable generator and additional data to generate candidate OOD labels. MCM (Ming al.. 2022a) is the earliest post hoc zero-shot OOD detection method that uses temperature scaling strategy and maximum predicted softmax value as the OOD score. Based on MCM, NPOS (Tao et al., 2023) finetunes CLIP's s image encoder using generated OOD data to optimize the decision boundary between ID and OOD data for the visual OOD detection task. Our method, NegLabel, is post hoc zero-shot OOD detection method that leverages CL.IP's zero-shot capabilities for text to design OOD scores.
CONCLUSION
This paper proposes a simple yet effective post hoc zero-shot detection framework called NegLabel. The approach involves incorporating a large set of negative labels that exhibit significant semantic differences from the IDD labels. The proposed method determines whether an image belongs to OOD by comparing its affinity towards ID and negative labels. To enhance the distinction between ID and OOD samples, the NegMining algorithm is proposed to select high-quality negative labels. Moreover, we design
scheme for the zero-shot OOD score that effectively leverages the knowledge of VLMs. Experimental results show that the proposed method achieves state-or thethe-rrt performance on multiple zero-shot shot OOD detection benchmarks and generalizes well on multiple VLM architectures. Negl abel also presents remarkable robustness against diverse domain shifts.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
This work supported by the National Natural Science Foundation of China (Grant NO. 62122035, and 62376104). XJ and BH were supported by the NSFC General Program No. 62376235, Guangdong Basic and Applied Basic Research Foundation No. 2022A151501 1652, HKBU Faculty Niche Research Areas No. CC-FFNRA-I//2222//SCIII FL was supported by the Australian Research Council with grant numbers DP230101540 and DE240101089, and the NSF&CSIRO Responsible Al program with grant number 2303037. TLL was partially supported by the following Australian Research Council projects: FT220100318, DP220102121, L.P20100522, L.P20220949, and IC190100031.
and HKBU CSD Departmental Incentive Scheme.
REFERENCES
Julian Bitterwolf, Alexander Meinke, and Matthias Hein. Certifiably adversarially robust detection of out-of-distribution data. In NeurIPS, 2020.
Lukas Bossard, Matthieu Guillaumin, and Lue Van Gool. Food-101 10ll-001-miiinn discriminative compo- nents with random forests. In ECCV, 2014.
Zhongzhi Chen, Guang Liu, Bo-Wen Zhang, Fulong Ye, Qinghong Yang, and Ledell Wu. Altelip: Altering the language encoder in clip for extended language capabilities. arXiv preprint, 2022.
Mircea Cimpoi, Subhransu Maji, lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. De- scribing textures in the wild. In CVPR, 2014.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
Terrance Ve and Graham W Taylor. Learning confidence for out-of-distribution detection in neural networks. arXiv preprint arXiv: 100::/80..00000 04865, 2018.
Andrija Djurisic, Nebojsa Bozanic, Arjun Ashok, and Rosanne Liu. Extremely simple activation shaping for -uittf-distribution detection. 2023.
Xuefeng Du, Xin Wang. Gabriel Gozum, and Yixuan Li. Unknown-aware object detection: Learning what you don't 't know from videos in the wild. In CVPR, 2022.
Sepideh Esmacilpour, Bing Liu, Eric Robertson, and Lei Shu. Zero-shot dusto--distribution detection based on the pre-trained model clip. AAAI, 2022.
Christiane Fellbaum. WordNet: An Electronic Lexical Database. Bradford Books, 1998. URL
pettps:/mitprrett mit www./7800225611166666
Stanislav Fort, Jie Ren, and Balaji Lakshminarayanan. Exploring the limits of out-of-distribution
detection. In NeurlPS, 2021.
Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of-distribution examples in neural networks. In ICLR, 2017.
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, and Justin Gilmer. The many faces of robustness: critical analysis of out-of-distribution generalization. ICCV, 2021a.
Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. CVPR, 2021b.
Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alexander Shepard, Hartwig Adam, Pictro Perona, and Serge J. Belongie. The inaturalist species classification and detection dataset. In CVPR, 2018.
Yen-Chang Hsu, Yilin Shen, Hongxia Jin, and Zsolt Kira. Generalized ODIN: detecting out-of- distribution image without learning from out-of-distribution data. In CVPR, 2020a.
10

---

Published as a conference paper at ICLR 2024
Yen-Chang Hsu, Yilin Shen, Hongxia Jin, and Zsolt Kira. Generalized odin: Detecting out-of-
distribution image without learning from out-of-distribute data. In CVPR, 2020b.
Rui Huang, Andrew Geng, and Yixuan Li. On the importance gradients for detecting distributional shifts in the wild. In NeurIPS, 2021.
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In JCML,,, 2021.
Xue Jiang, Feng Liu, Zhen Fang, Hong Chen, Tongliang Liu, Feng Zheng, and Bo Han. Detecting out-of-distribution data through in-distribution class prior. In ICML, 2023.
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. ICCV, 2013.
Alex Krizhevsky. Learning multiple layers of features from tiny images. 2009.
Yann LeCun, Sumit Chopra, Raia Hadsell, M Ranzato, and Huang. A tutorial on energy-based learning. Predicting structured data, 1(0), 2006.
Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for detecting out-of-distribution samples and adversarial attacks. In NeurlPS, 2018.
Shiyu Liang, Yixuan Li, and R. Srikant. Enhancing the reliability of out-of-distribution image detection in neural networks. In ICLR, 2018.
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In NeurIPS. 2023.
Weitang Liu, Xiaoyun Wang, John D. Owens, and Yixuan Li. Energy- out-of-distribution detection. In NeurlPS, 2020.
Ziwei Liu, Zhongqi Miao, Xiaohang Zhan, Jiayun Wang, Boging Gong, and Stella X. Yu. Large-scale long-tailed recognition in an open world. In CVPR, 2019.
Yifei Ming, Ziyang Cai, Jiuxiang Gu, Yiyou Sun, Wei Li, and Yixuan Li. Delving into out-of- distribution detection with vision-language representations. NeurlPS, 2022a.
Yifei Ming, Hang Yin, and Yixuan Li. On the impact of spurious correlation for out-of -iitr--distribution detection. In AAAI, 2022b.
Yifei Ming, Yiyou Sun, Ousmane Dia, and Yixuan Li. Cider: Exploiting hyperspherical embeddings for out-of-diitribution detection. ICLR, 2023.
Ravi Parameswaran. Statistics for experimenters: introduction to design, data analysis, and model building. JMR, Journal of Marketing Research, 16(000002):291, 1979.
Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In CVPR, 2012.
Foster J. Provost, Tom Fawcett, and Ron Kohavi. The case against accuracy estimation for comparing induction algorithms. In ICML, 1998.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML,, 2021.
Benjamin Recht, Rebecca Roclofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In ICML, 2019.
11

---

Published as a conference paper at ICLR 2024
Yiyou Sun and Yixuan Li. Dice: Leveraging sparsification for out-of-distribution detection. In ECCV, 2022.
Yiyou Sun, Chuan Guo, and Yixuan Li. React: Out-of-distribttion detection with rectified activations. In NeurIPS, 2021.
Yiyou Sun, Yifei Ming. Xiaojin Zhu, and Yixuan Li. Out-of-distribution detection with deep nearest neighbors. ICML,, 2022.
Jihoon Tack, Sangwoo Mo, Jongheon Jeong, and Jinwoo Shin. Csi: Novelty detection via contrastive learning distributionally shifted instances. NeurIPS, 2020.
Leitian Tao, Xuefeng Du, Xiaojin Zhu, and Yixuan Li. Non-parametric outlier synthesis. ICLR, 2023.
Sagar Vaze, Kai Han, Andrea Vedaldi, and Andrew Zisserman. Open-set recognition: good closed-set classifier is all you need. In ICLR, 2022.
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-ucsd birds-200-2011 dataset. 2011.
Haohan Wang. Songwei Ge, Zachary C. Lipton, and Eric P. Xing. Learning robust global representa- tions by penalizing local predictive power. In NeurIPS, 2019.
Haoqi Wang, Zhizhong Li, Litong Feng, and Wayne Zhang. Vim: Out-of-distribution with virtual- logit matching. In CVPR, 2022.
Haoran Wang, Weitang Liu, Alex Bocchieri, and Yixuan Li. Can multi-label classification networks know what they don't 't know? NeurIPS, 2021a.
Hualiang Wang, Li, Huifeng Yao, and Xiaomeng Li. Clipn for zero-shot ood detection: Teaching clip say ICCV, 2023.
Yezhen Wang, Bo Li, Tong Che, Kaiyang Zhou, Ziwei Liu, and Dongsheng Li. Energy-based open-world uncertainty modeling for confidence calibration. In ICCV, 2021b.
Hongxin Wei, Renchunzi Xie, Hao Cheng, Lei Feng, Bo An, and Yixuan Li. Mitigating neural network overconfidence with logit normalization. In ICML, 2022.
Jianxiong Xiao, James Hays, Krista A. Ehinger, Aude Oliva, and Antonio Torralba. SUN database: Large-scale scene recognition from abbey zoo. In CVPR, 2010.
Jiarui Xu, Shalini De Mello, Sifei Liu, Wonmin Byeon, Thomas Breuel, Jan Kautz, and Xiaolong Wang. Groupvit: Semantic segmentation emerges from text supervision. In CVPR, 2022.
Jingkang Yang, Haoqi Wang, Litong Feng. Xiaopeng Yan, Huabin Zheng, Wayne Zhang, and Ziwei Liu. Semantically coherent out-of-distributiio detection. In ICCV, 2021.
Bolei Zhou, Àgata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEEE transactions on pattern analysis and machine 2018.
intelligence,
12

---

Published as a conference paper at ICLR 2024
A FURTHER EXPERIMENTS
A.1 DETAILED EXPERIMENT SETTING.
A.1.1 IMPLEMENT DETAILS.
The proposed method LegLa is implemented with Python 3.9 and PyTorch 1.9. All the experiments on a single NVIDIA GeForce RTX 3090Ti GPU. Negl is a post hoc OOD detector with about (22MD) FLOPS extra computational burden per image (note that M is the number of negative labels and D is the dimension of the embedding feature), which usually introduce < 1% network forward latency. NegLabel detects OOD samples in parallel with the zero-shot classification task, without any impact on ID classification performance. The NegMining algorithm selects and dumps the negative label in advance, and it usually takes few minutes for processing the large-scale corpus like WordNet on single 3090Ti GPU.
In this paper, we implement NegLabel on various VLM architectures, including CLIP (Radford et al., 2021), ALIGN (Jia et al., 2021). Group ViT (Xu et al., 2022), AIICIIP (Chen et al., 2022). Unless otherwise specified, we employ CLIP-B/16 for zero-shot OOD detection and the hyperparameters use the following default configurations. The Ne..iniin algorithm takes WordNet as the corpus and select M = 10000 negative labels under 7 = 0.05. We use the NegLabel score the sum-softmax form and set 0.01 as the temperature, and use ny 100 for grouping strategy by default.
Table reports the results of various mathods that perform OOD detection on fine-tuned CLIP. We follow the fine-tuning strategy of NPOS (Tao et al., 2023), which fine-tunes the last two blocks of CLIP's image encoder and equips a fully connected layer after the image encoder, which maps the image features into class predictions.
A.1.2 EVALUATION METRICS.
We use common metrics evaluate OOD detection methods Huang et al. (2021): the false positive rate that OOD data are classified as ID data when 95% of ID data are correctly classified (FPR95) Provost et al. (1998) and the area under the receiver operating characteristic curve (AUROC) Huang et al. (2021).
A.2 ZERO-SHOT OOD DETECTION PERFORMANCE COMPARISON ACROSS DIVERSE TASKS.
We compare the zero-shot OOD detection performance of NegLabel and MCM on seven different in-distribution (ID) datasets. As shown in Table 5, our method outperforms MCM on all OOD datasets regardless of the ID dataset used. Specifically, NegL.abe achieved an AUROC of around 99.9% and FPR95 below 1 1% Standford-Cars CUB-200, Oxford-Pet, Food-101. and ImageNet-10, 10, while MCM could only achieve this performance on Standford-Can and ImageNet-10 10. This demonstrates the superior OOD detection performance of our method on these datasets. When ImageNet-1000 is used as the ID dataset, our method was able to achieve an average improvement of approximately 4% AUROC and decrease of around 20% FPR95 on four OOD datasets compared MCM. Overall, our method showes strong zero-shot OOD detection capabilities on different ID datasets.
A.3 ROBUSTNESS TO DOMAIN SHIFT.
To test the domain generalization capability, we use several versions of ImageNet with diverse domain shifts as ID samples. The ImageNet-Sketar dataset (Wang et al., 2019) consists of 50,000 images, with 50 images for each of the 1,000 ImageNet classes, rendered in "black and white" color scheme. ImageNet-A (Hendrycks et al., 2021b) is a dataset of images labeled with ImageNet labels, collected by selecting only the images that ResNet-50 models fail to classify correctly. ImageNet-R (Hendrycks et al., 2021a) is dataset that includes various artistic representations such as art, cartoons, deviantart, graffiti, embroidery, graphics, origami, paintings, patterns, plastic objects, plush objects, sculptures, sketches, tattoos, toys, and video game renditions of ImageNet classes. It consists of 30,000 images, representing 200 ImageNet classes. The ImageNetV2 dataset (Recht et al., 2019) includes new test data specifically created for the ImageNet benchmark. These test sets were sampled after a decade of progress the original ImageNet dataset. OOD samples are still taken from iNaturalist, SUN, Places, and Textures. There is no overlapping classes between ID and OOD samples.
13

---

Published as a conference paper at ICLR 2024
Table 5: Zero-shot OOD detection results based on CLIP-B/16 with various ID datasets.
OOD Dataset
ID Dataset
Method
NNaturalist
SUN
Places
Textures
Averuge
AUROCT
FPR95]
AUROC1
FPR95]
AUROCT
FPR95]
AUROCT
FPR95]
AUROC1
FPR95,
Stanford-Cars
MCM
99.77
0.05
99.95
0.02
99.
0.24
99.96
0.02
99.89
0.08
NegLahel
99.99
0.01
0.01
99.99
0.03
0.01
99.99
0.01
CUB-200
MCM 98.24
9.83
99.10
4.93
98.57
6.65
98.75
6.97
98.66
NegLahel
99.96
0.18
99.99
0.02
99.90
0.33
99.99
0.01
99,96
0.13
Oxford-Pet
MCM
99.38
2.85
99.73
1.06
99.56
2.11
99.81
0.80
99.62
1.70
NegLabel
99.99
0.01
99,99
0.02
99.96
0.17
99.97
0.11
99,98
0.07
Food-101
MCM
99.78
0.64
99.75
0.90
99.58 99.58
1.86
98.62
4.04
99.43
1.86
NegLahel
99.99
0.01
99.99
0.01
99.90
ImageNet-10
MCM
99.80 99.80
0.12
99.79
99.65
0.88
0.04
99.78
Negl Lahel
99.83
0.02
99.88
0.20
99.75
0.71
99.94
0.02
99.85
0.24
ImageNet-20
MCM
99.66
1.02
99.50
2.55
99.11
4.40
99.03
2.43
99.32
2.60
NegLabel
99.95
0.15
99.51
1.93
98.97
4.40
99.11
2,41
99.39
2.22
ImageNet- 100
MCM 96.77
18.13
94.54
36.45
94.36
34.52
92.25
41.22
94.48
32.58
NegLa
99.87
0.57
97.89
96.25
20.37
97.50
12.84
The detailed results shown in Table 6 illustrate that the zero-shot OOD detection performance of MCM significantly deteriorates across diverse domain shifts, indicating the difficulty of zero-shot OOD detection under such conditions. Nevertheless, our method NegLabel continues to achieve noteworthy performances cross diverse ID datasets, thus demonstrating its remarkable robustness against diverse domain shifts. MCM heavily relies on the model's s classification ability on the ID dataset. Therefore, when domain shift leads to decrease in ID classification accuracy, the OOD detection performance of MCM also decreases accordingly. In contrast, our approach NegLabel treats the ID labels as whole, which is less affected by inter-class confusion, making it more robust to domain shifts.
Table Zero-shot OOD detection performance robustness domain shift. All methods are based on CLIP-B/16. The ID data are ImageNet-Sketer All values are percentages. 1 indicates larger values are better and indicates smaller values are better. The shadow part represents our method.
0OD Daraset
ID Dutaset
Method
INaturalist
SUN
Placss
Textures
Average
AUROCT
FPR95]
AUROC1
FPR95]
AUROCT
FPR95]
AUROCT
TPR95]
AUROC1
FPR95]
ImageNot-Skotch
Negl abel
MCM
99.34
87.74
63.06
2.24
94,93
85.35
67.24
22.73
90.78
81.19
70.64
38.62
89.29
74.77
79.59
46.10
82.26
93.59
27.42
70.13
ImageNet-A
NegLabel
MCM
98.80
79.50
4.09
89.83
44.38
82.88
80.51
60.10
80.25
64.34
87.94
43.23
ImageNet-R
MCM
71.51
75.53
76.67
67
76.72
NegLabel
99.58
1.60
96.03
15.77
91.97
29.48
90.60
35.67
94.54
20.63
ImageNerV2
MCM
91.79
45.90
89.88
50.73
86.52
56.25
81.51
69.57
87.43
55.61
NegLabel
99.40
A.4 ABLATION STUDY ON THE NEGMINING ALGORITIM..
We conduct an ablation analysis on each module in our proposed zero-shot OOD detection framework and the detailed results are shown in Table 7.
The first row displays the performance of NegLabel, while the subsequent rows represent ablation studies on modules and the number of negative labels. Note that when NegMining is not used, negative labels are randomly selected from the corpus.
Comparing the second and fourth rows, negative labels selected by NegMining show +1.81 1% AUROC and -6.63% FPR95, indicating that negative labels require greater semantic differences from ID labels. Comparing the second and third rows, the grouping strategy only provides slight improvement when NegMining is not used. However, comparing the first and fourth rows, yields +0.45% AUROC and -2.79% FPR95 with selected negative labels, further demonstrating that the grouping strategy aims to reduce coincidental false positives between ID images and negative labels.
14

---

Published as a conference paper at ICLR 2024
The first row and the last three rows of Table demonstrate the impact of an increasing number of negative labels the performance of OOD detection. The fifthh row shows that even with small number of negative labels, our method can still achieve state-of-the offthhe-rr the-art performance. Furthermore, there is noticeable performance gain in OOD detection when the number of negative labels increases to 10,000. However, as the NegMinnng algorithm gives priority to candidate labels that are more dissimilar from the ID labels, when the number further increases 20,000, the newly added labels exhibit more dissimilarities with the ID labels. Consequently, this leads to a decline in performance. This finding is consistent with the analysis presented in Section 3.3.
Interestingly, even when we indiscriminately use all the words in WordNet, the OOD detector still demonstrates remarkable performance and surpasses previous methods. This phenomenon suggests that the broad semantic space constructed by negative labels contributes to harnessing the latent knowledge of VL.Ms and provides new clues for OOD detection, For more detailed experiments regarding the number of negative labels. please refer to Appendix 1.4.1.
Table 7: Module ablation in NegLabel. All methods are based CLIP-B/16. The ID data ImageNet- 1k.
Numbar Negutive Labels
NegMining
Grouping Strategy
iNaturalist
SUN
OOD
Places
Textures
Average
AUROCT
FPR95
AUROCT
HP895]
AUROCT
HPR95]
AUROCT
HPK95]
AUROCT
PPK95]
10,000
99.49
1.91
95.49
20.53
91.64
35.59
90.22
43.56
94.21
25.40
10,000
8
97.96
9.24
93.89
28.91
89.53
45.08
86.40
56.03
91.95
34.82
10,000
86.6
34.69
10,000
99.30
23:11
89.76
93.76
28.19
1,000
1.35
92.42
36,96
86.
56.79
91.00
20,000
99.18
3.16
95.24
21.47
91.16
37.23
89.87
44.40
93.85
26.56
50,000
98.39
6.89
94,48
25.15
90.20
41.62
87.71
53.01
92.70
31.67
A.4.1 IMPACT OF THE NUMBER OF SELECTED NEGATIVE LABELS.
We explore the impact of the number of selected negative labels on OOD detection. The results shown in Table demonstrate that increasing the number of negative labels gradually improves the performance of OOD detection and then declines. This indicates that NegMining prioritizes selecting negative labels that are farther away from the ID labels, as they are more discriminative for detecting OOD images. This is consistent with the analysis in Section 3.3.
Table 9 shows the results of randomly selecting words in WordNet as negative labels. The comparison of Table 8 and Table 9 illustrates the role of NegMining algorithm. Concretely, when the number of negative label is less than 1,000, we notice that the OOD detection performances of random selection are better than MegMining. This is because that the negative labels selected by NegMining with higher rank (i.e., the top-M ranked candidate labels when M is small) are actually both far from ID and OOD samples, resulting in less divergences in P1 and P2 analysed in Section 3.3. Although the divergences are small, they can still provide a positive return in detecting OOD samples (i.e., AUROC 50% for a binary classifier).
Furthermore, it is unfair to compare the performance of two methods only in the small M case. With M increases, the performances of random selection gradually converge to 92. 98. AUROC as is performance bottleneck, while NegMining reaches a higher performance of 94.21 % AUROC. The reason is that random selection indiscriminately introduces labels in WordNet, which may toxic for OOD detection. In contrast, NegMining carefully selects words that are farther away from the ID semantics as negative labels, with a longer-term perspective. Therefore, the comparison of Table and Table reveals the effectiveness of the NegMining algorithm.
A.4.2 DISCUSSION OF PERCENTILE DISTANCES.
To further investigate the impact of percentile distances, conduct an ablation experiment. and the results are presented in Table 10. The study findings demonstrate that OOD detection performance improves significantly as the percentile decreases initially, but the rate of improvement gradually slows down, reaching its peak at the 5th percentile.
This phenomenon be attributed to two main reasons. Firstly, ID labels that are closer to the candidate negative label are more likely to represent similar semantic meaning. As a result, by
15

---

Published as a conference paper at ICLR 2024
Table 8: Impact of the number of negative labels M selected by NegMining on ImageNet- Ik benchmark. The negative labels are selected from WordNet. All values are percentages. 1 indicates larger values are better and &. indicates smaller values are better.
Negative Label Number
OOD Dataset
NNturalistt
SUN
Places
Textures
Average
AUROCT
PPR95_
AUROC1
[PR95]
AUROCT
FPR95]
AUROCT
[PR95]
AUROCT
FPR95,.
100
53.18
99.30
53.34
99.44
53.09
99.05
53.04
99.18
53.16
99.24
200
99.23
2.59
89.83
50.87
85.15
62.33
84.23
69.01
89.61
46.20
1,000
99.62
1.35
92.42
36.96
88.24
86.11
56.79
91.60
36.14
2,000
99.67
1.17
93.60
29.58
89.90
42.83
88.84
48.10
93.01
30.42
5,000
99.64
1.39
94.83
23.28
91.28
37.58
89.97
44.26
93.93
26.63
10,000
99.49
1.91
95.49
20.53
91.64
35.59
90.22
43.56
94.21
25.40
20,000
99.18
3.16
95.24
21.47
91.16
26.56
30,000
98.90
4.42
94.98
22.40
90.75
39.11
89.12
47.00
93.44
28.23
50,000
98.39
6.89
94.48
25.15
90.20
41.62
87.71
53.01
92,70
31.67
Table 9: Results of randomly selecting negative labels on ImageNet- 1k benchmark. The negative labels are selected from WordNet. Each row of different experimental setting reports the average performance of 10 repeated runs, and the superscript represents its standard deviation.
Selection
INaturalist
SUN
OOD Dataset
Places
Textures
Average
AUROCT
FPR951
AUROCT
FPR95]
AUROCT
FPR9S]
AUROCT
FPR951
AUROC1
FPR95]
+3.00
12..00
72.77
56 0g
45:40
81.84=1.0
61.7223..2
80.27790.11
25.00022.00
200
96. 76.70
16. 16.1222.57
92.31
36.09+220
88 7259
50.71=177
84.
60.9742.01
500 90.80t00.1
40.37799.5
1,000
97.6320.11
10.68
40.75
29.5
20.45
45.00=1.0s
85.86=0.30
57.55
35.69 35.88930.0
2,000
97. 63
10:27700.4
95.9y+0.12
28. 28.233
59
43.82=98:
57.64
91. #4..0
34.9yt0.44
5,000
97.71
10.36
94.01 10.00
28.21 19.51
86. 86.4830011 43.30=9955
36.88738.30
02.01
34.67:39.35
10,000
97.96
#0.15
53.9g40..0
28.67
se 00.55
m8.22
55.87
92. 022.0
34.09
20,000
077 07.84400.02
9.78-9.s
94 94.0849.00
27.66-
80 04
42.83=022
86 86.53e000
066.6340.11
592, 0y2.0tsto.os
34 14.2239..5
30,000
97.87=0..0
9.05
84.10+9..02
27.64
80 00.0220000
42. 42.88-9.13
86.62 86.62
56.54
92, 12:.1.020
34.18:t..2
50,000
01
9.39
04.1220.01
27.43=0..1
42.80
36.42220.00
U22 12.116
34. 34.0g30.00
selecting the ID label as the anchor for the candidate negative label that is in close proximity it, more accurate measurement the candidate negative label's distance from the entire ID label space can be obtained. Secondly, illustrated in Fig. 2, there are several outliers in the ID label space. If these outliers are used as anchors, the distance between the candidate negative label and the entire ID label space would be inaccurate. To improve the quality of negative labels and enhance OOD detection performance, we use the ID label at the 5th percentile as the anchor. This approach strikes balance between accurately measuring the distance between the candidate negative label and the ID label space while also avoiding outliers that could lead to inaccurate distance measurements.
A.4.3 ABLATION ON CORPUS SOURCES OF NEGATIVE LABEL.S.
The role of the corpus provide a larger and more comprehensive semantic space, so that ID and OOD data can be characterized and distinguished in a more fine-grained manner in this space. We conduct ablative analysis using corpus sources for selecting negative labels and the results are shown in Table 11. We take Wikipedia dataset containing cleaned articles of all languages. Part-of-Speech Tags dataset is a 370k English words corpus.
In detail, use the title field in Wikipedia dataset, which contains concepts entities in the world. We first randomly sample 70,000 items as candidates, and then pick top-10,000 items as negative labels by applying NegMining algorithm. As for Part-of-Speech Tags dataset, it is processed like WordNet, i.c., only using nouns and adjectives in the dataset. We first randomly subsample the words to 70,000 candidates, and then select top- 10,000 words negative labels through NegMining algorithm.
The results of Table 11 show that the proposed negative label is robust scheme for zero-shot OOD detection, which can get reasonable performances on multiple corpora, and it is even robust across multiple languages.
"https://dumm" malkimedia.. org/ Phtpps: / /www. kaggle e wtt//otaasseeeeeoe ruchi 798 wttppcchhtt/aaaaf-aaaapae
16

---

Published as a conference paper at ICLR 2024
Table 10: The choice of distances for the selection of negative labels.
OOD Dataset
Average
Percentile
iNaturaiist
SUN
Places
Textures
AUROC
FPR95_
AUROCT
FPR95]
AUROC
FPR95.
AUROC1
FPR95]
AUROCT
FPR95,
0.95
98.81
4.88
94.69
24.39
91.18
36.25
89.32
46.81
93.50
28.08
99.06
3.65
94.62
24.89
90.97
37.31
93.60
27.56
0.85
99.26
2.88
94.82
23.71
90.99
37.09
89.96
43.62
93.76
26.82
0.80
99.37
2.43
94.95
22.91
91.14
36.80
89,97
43.76
93.86
26.47
0.75
99.43
2.26
95.20
21.70
91.28
36.55
90.14
43.65
94.01
26.04
0.70
99.45
2.14
95.31
21.01
91.41
36.10
90.21
43.49
25.69
0.65
99.46
2.08
95.34
21.07
91.47
36.13
90.16
43.85
94.11
25.78
0.60
99.47
2.07
95.39
21.00
91.77
36.31
90.16
43,90
94.12
25.82
99.47
1.94
20.81
91.53
35.81
90.13
25.60
0.50
99.47
195
95.44
20.73
91.54
35.94
90.14
43,90
94.15
25.63
0.45
99.47
1.5
95.48
20.50
91.57
35.83
98
43.67
94.18
25.49
0.40
99.88
197
95.48
20.56
91.56
35,90
90.21
43.76
94.18
25.55
0.35
99.48
1.95
35.81
43.58
0.30
99.48
1.97
95.49
20.71
91.59
35,94
90.21
43.48
94.19
25.52
0.25
99.48
194
95.49
20.52
91.59
35,90
90.19
43.72
94.19
25.52
99.48
1.96
35.73
0.15
99.48
1.90
95.49
20.42
91.60
35.72
90.18
43.78
94.19
25.45
0.10
99.49
1.90
95.50
20.44
91.62
35,92
90.14
44.06
94.19
25.58
99.49
191
95.49
20.53
91.64
35,59
90.22
43.56
94.21
25.40
0.04
99.49
91.65
35.84
44.06
25.59
0.03
99.49
1.92
95.42
20.62
91.63
35.98
90.17
44.11
94.18
25.66
0.02
99.49
1.93
95.39
20.79
91.61
35.98
90.28
43.79
94.20
25.62
99.50
1.96
95.36
91.63
36.45
90.03
94.13
26.03
0.00
99.47
2.08
95.27
21.14
91.55
36.91
89.58
47.09
93.97
26.81
Table 11: Comparison with diffrent corpus sources.
OOD Dataset
Source
Language
INaturalist
SUN
Places
Textures
Average
AUROC1
FPR95
AUROC1
FPR95
AUROC1
FPR95
AUROCT
FPR95
AUROCT
FPR95]
WordNer
English
99.49
1.91
95.49
20.53
91.64
35.59
9X.22
43.56
94.21
25.40
Wikipedia
English
French
91.99
47.18
90.62
48.79
90.37
46.30
80.86
76.33
88.46
$4.65
Part-of-Spochh Tags
English
99.23
3.25
94.20
25.93
90.17
43.09
87.77
56.11
92.84
30.59
A.5 ABLATION STUDY ON THE NEGLABEL SCORE
A.5.1
THE FORM OF THE NEGLABEL SCORE.
In Section 3.2, we discuss about the scheme of designing zero-shot OOD score with negative labels:
S(x) 5" (sim V), sim(x,)")))
The design criterion of S* can be summarized as follows: S* is positively correlated with the similarity the sample in the ID label space and negatively correlated with the similarity of the sample in the negative label space.
Here, we provide two implementations, linear and proportional, as follows:
Sinear (x) =
Ti -sim(x(x, (xiimxx)))) y)) - a . (sim(x,
(10)
T (zimm((x,)) )1)
Sproportional (x) Ty Ty (x, })) (sim (-)
(11)
where T, (-) are usually nonlinear transforms, which can map similarities to more discriminative values. And is a factor that balances the number of ID and negative labels. Specially, when T is an exponential-sum transform with temperature coefficient 7, Eq. (11) becomes the sum-softmax form in Eq. (6).
Table 12 shows the results of different implementations for OOD score function. All experiments in this table evaluated ImageNet- benchmark without the grouping strategy. The results indicate that the proportional scores perform better than the linear scores. This because the linear
17

---

Published as a conference paper at ICLR 2024
form just simply fuses the similarities by linear combination, while the proportional form introduces an extra nonlinearity that normalize and stabilize the similarities.
Comparing row and row 6, the result of sum-softmax score is better than max -oftmmx with negative labels. Because the sum-softmax measures the confidence that samples belong to the whole ID label space, which is more robust against ID inter-class confusion in zero-shot classification. Row 5 and row 7 show the comparison exponential function can significantly improve the performance. Because the image-text similarities in the CLIP-like model are smooth, as analyzed in Section 3.2.
whether use exponential transform. It can be observed that using
In particular, the optimal temperature coefficient of sum-softmax score is T = 0.01, which makes the score act as a soft binarization function (each similarity is magnified 100 times and then scaled by exponential function, so that only the similarities near to the highest value are treated as positives). This is similar to our assumption in theoretical analysis (Section 3.3) that each negative label is a binary classifier. So, we provide some "binarized version" of the score functions, as shown in the Table 12 with I symbols. 1 [cos(- is a function to check the cosine similarity is larger than B, and the value equals to 1 if the condition holds. 3 is a binarization threshold and we set it to 0.25 for CLIP.
By replacing the exponential transform with the binarization transform (see row 5 and row 9), the results drop to 90.07% AUROC and 40.53% FPR95, but still show reasonable performances in zero-shot OOD detection. The binarized scores in linear form (refer to row 3 and row 4) also show the effectiveness of negative labels. Specially, the score function in row 4 is align with the "toy score function" mentioned in theoretical analysis (Section 3.3), without directly using ID label to detect OOD samples. It still gets 78.12% AUROC, and illustrates the rationality of our theoretical analysis.
Table 12:
The form of the NegLabel score. All experiments are evaluated on ImageNet-1 1 benchmark without the grouping strategy.
Score Function
INasuniist
Places
Textures
Average
AUROC:
FPR351
AUBOCT
FFROET
AUROCT
FPRSS,
AUBOCT
FPRA5]
AUROCY
FPRO5,
S coo/A..) oi) 0.1 E 2:))
98.14
9.53
80.26
30.63
08 68
59.29
78,02
82,80
81.27
05.51
Linear
mix
$7.75
66.27
58:35
51,90
N2,30
72.02
86:87
61.25
ppneerooo 0.1. 4pochaylo
95.64
9.34
85.12
44.74
$3.93
49.59
75,02
62.06
85.33
41.43
21 4ymiki) K.PA
98.97
3,47
62.04
72.62
78.12
60.97
99.30
2,65
95.08
23.11
90.90
40.35
59.76
46,03
93.76
28.19
99.12
3.75
94.54
24.01
50.92
37.45
89.36
44.72
93.46
27.48
Propertioall
97.39
13.48
75.90
90.24
76
85.46
80.21
68.26
97.92
10.38
94.03
28.24
59.05
44.34
90.94
40.51
92.98
30,87
96.89
5.70
91,80
46.68
56.75
$1.00
$4.82
60.74
90.07
40.53
A.5.2 COMPARISON WITH DIFFERENT TEMPERATURE VALUES OF SOFTMAX FUNCTION.
As analyzed in Section 3.2, the cosine function clamps the similarity score in [-1, 1] and results in even smaller divergences between positive and negative pairs (e.g., the cosine similarities of positive pairs in CLIP are typically around 0.3 and that of negative pairs are around 0. (Ming et al., 2022b)). Such small differences may not adequately reflect the confidence variance of ID and OOD samples. Therefore, a suitable temperature coefficient r is important to scale the similarity, allowing better reflecting the model's confidence.
We conduct ablation analysis towards the temperature coefficient T in Eq. (6). The results in Table 13 illustrate that as the temperature coefficient 7 increases, the performance of OOD detection shows an initial increase followed by a decrease trend, achieving the best performance at T = 0.01.
18

---

Published as conference paper at ICLR 2024
Table 13: Comparison with different temperature values of softmax function. The temperature values are sampled uniformly on the logarithmic axis.
OOD Dataset
Average
Temperature
iNaturalist
SUN
Places
Textures
AUROCT
FPR95
AUROC
HPR95]
AUROCT
HPR95
AUROCT
FPR95
AUROCT
HPR95_
0.0010
99.
2.96
94.37
23.46
90.23
36.95
88.65
44.08
93.10
26.86
0.0013
99.19
2.91
94.53
23.44
90.55
36.96
89.01
44.24
93.32
26.89
0.0016
99.22
2.80
94.68
23.29
90.74
36.60
89.21
44.11
93.46
26.70
0.0020
99.27
2.70
94.82
22.80
90.94
36.23
89.39
43.67
93.61
26.35
0.0025
99.32
2.60
94,97
22.38
91.13
35.82
89.56
43.51
93.75
26.08
0.0032
99.36
2.39
95.12
21.73
91.33
35.32
89.74
42.89
93,89
25.58
0.0040
99.40
2.33
95.26
21.40
35.21
43.03
25,49
0.0050
99.44
2.22
95.37
20.81
91.61
35,36
90.04
42.82
94.11
25,30
0.0063
99.47
2.09
95.45
20.38
35,39
90.15
43.10
94.18
25,24
0.0079
99.49
1,90
95.49
20.58
91.64
35.82
90.23
43.72
94.21
25.51
0.0100
99.49
1.91
95.49
20.53
91.64
35.59
90.22
43.56
94.21
25.40
0.0126
99.52
1.86
95.44
21.18
91.35
38.19
90.23
45.28
94.14
26.63
0.0158
99.53
1.82
95.34
22.13
39.73
90.16
46.67
94.02
27.59
0.0200
99.46
2.06
93.13
37.88
87.12
58.11
87.88
$9.34
91.90
39.35
5.05
70.57
75.32
83.09
80.02
77
0.0316
99.14
2.98
94.36
23.79
90.22
36.82
88.65
44.61
93.09
27.05
0.0398
99.15
2.97
94.42
23.68
90.34
36.85
88.75
44.54
93.17
27.01
99.17
2.94
23.44
36.86
44.54
93.23
26.94
0.0631
99.19
2.87
94.53
23.41
90.54
36.65
89.01
44.17
93.32
26.77
0.0794
99.21
2.85
94.61
23.24
90.65
36.61
89.11
44.24
93.39
26.73
99.23
23.16
36.49
0.1259
99.27
2.75
94.82
22.74
90.94
36.26
89.38
44.06
93.60
26.45
0.1585
99.32
2.58
94.98
22.43
91.14
35.76
89.57
43.56
93.75
26.08
2.39
21.55
35.34
0.2512
99.44
2.22
95.37
20.74
91.61
35,20
90.04
42.66
94.11
25,20
0.3162
99 49
1.22
95.49
20.59
91.65
35.82
90.23
43.67
94.22
25,50
99.52
1.85
95.39
21.81
38.97
27.18
0.5012
99.52
1.75
94.72
26.92
89.82
46.69
89.58
$1.19
93.41
31.64
0.6310
99.46
2.07
93.15
37.73
87.16
57.93
87.90
59.34
91 91.92
39.27
0.7943
50.85
69.90
85.48
66.60
89.84
47.50
1,0000
99.16
3,44
88.61
60.33
80.22
76.39
83.11
71.24
87.78
52.85
1,2589
99.00
4.36
86.65
66.66
77.40
80.79
81.28
74.33
86.08
56.53
1.5849
85.13
83,10
84.82
$8.95
1,9953
98.70
5.84
83,96
73.06
73.76
84.56
79.16
78.72
83.90
60.55
2,5119
98.58
6.73
83.06
74.74
72.62
85.67
78.56
79.82
83,21
61.74
3.1623
98.47
7.45
82.36
76.15
86.43
78.13
80.57
82,68
62.65
3,9811
98.37
8.02
81.11
77.47
71.07
87.09
77.81
81.31
82.27
63.47
5.0119
98.28
8.48
81.37
78.27
70.55
87.60
77.57
81.95
81.94
64.08
6.3096
98.21
81.11
78.90
86.00
77.39
82.34
81.69
64.50
7,9433
98.15
9.06
80.73
79.37
69.81
88.28
77.24
82,70
81.48
64,85
10.0000
98.10
9.34
80.51
79.80
88.52
77.12
82.84
81.32
65.12
A.5.3 EFFECT OF PROMPT ENGINEERING.
We evaluate NegLabel with different prompt template on the ImageNet- 1k benchmark, shown in Table 14. The first six rows are selected from the 80 prompt templates proposed by CLIP (Radford al., 2021). The last two rows are our designed prompt templates.
Comparing the first three rows with the sixth row, is evident that the performance of OOD detection significantly decreases when the prompt templates contain negative terms such "dark", "blurry", and "low resolution". This may occur because OOD images might exhibit high similarity to prompts with negative words and ID labels, leading to misclassification. Conversely, when the prompt templates include positive terms like "good", the OOD detection performance improves noticeably. This is because OOD images are more likely to have low similarity with prompts containing positive words and ID labels, thereby enhancing the discrimination between ID and OOD images.
Furthermore, it can be observed from fourth and fifth rows that excessive neutral words neutral prompt templates tend to result in higher similarity between the negative labels and ID labels, thereby affecting the quality of selected negative labels. Therefore, we design two prompt templates with fewer neutral words and more positive adjectives shown in the last two rows. Experimental results demonstrate that using "The nice <labels." as the prompt template achieves the best performance.
19

---

Published as a conference paper at ICLR 2024
Table 14: Prompt engineering.
GOD Datoser
Prompt
INararalist
SUN
Places
Textares
Average
FTR95_
AUROCT
AUROCT
FPR95]
dark photo <libelt.
97.03
14.16
91.50
90.16
64.75
71.42
86.91
86,
51.00
A bluny photo of a <labels.
98.64
5.74
91.26
47.36
$5.66
61.76
79.14
71.70
88.68
46.64
low resolution photo of <labels,
99.39
2.74
93.06
37.79
88.30
52.88
78.33
77.50
89.89
42.73
cropped photo <labels.
A photo of <labels.
99.59
1.74
94.83
26.35
90.17
46.92
80.79
72.11
91.34
36.78
A good photo of <labels.
99.53
2.01
95.32
22.71
90.39
43.99
$1.09
61.90
92.33
32.65
<labels
99.52
1.91
95.77
19,32
92.43
32.79
86.89
59,34
93.65
28.34
A.5.4 EXPLORING THE GROUP NUMBERS OF THE GROUPING STRATEGY.
We investigate the impact of the hyperparameter (group number) in the grouping strategy and the corresponding results are presented in Table 15. We use M 10, 000 negative labels for all the experiments in this table, and the group number indicates how many subgroups should the negative labels be divided equally. The first row represents the baseline without using the grouping strategy when the group number is equal to one. The experimental results demonstrate that using the grouping strategy substantially enhances the OOD detection performance across four OOD datasets. Furthermore, as the group number gradually increases, the OOD detection performance remains stable, oscillating around 94.21 AUROC and 25.40% FPR95. These findings suggest that our grouping strategy is resilient to the hyperparameter (group number). Hence, select group number as 100 for our main experiments.
Table 15:
Impact of the group numbers in the grouping strategy OOD detection. The experiments are conducted on ImageNet- benchmark.
OOD Dutaset
Group Number
iNaturalist
SUN
Places
Textures
Average
AUROC
FPR95]
AUROCT
FPR95
AUROCT
FPR95]
AUROCT
FPR95
AUROC1
FPR95,
99.30
2.65
95.06
23.11
90.90
40.35
89.76
46.63
93.76
28.19
99.48
95.47
20.63
91.62
35.91
90.19
43.56
94.19
25.51
100
95.49
20.53
91.64
35.59
90.22
43.56
94.21
25.40
150
99.49
1.89
95.52
20.47
91.66
35.70
90.20
43.95
94.22
25.50
200
99.49
1.90
95.50
20.58
91.66
35.66
90.24
43.60
94.22
25.43
99.50
95.55
20.22
91.65
35.69
90.18
43.97
94.22
25,44
300
1.85
95.53
20.32
35,95
90.14
44.18
94.19
25.58
350
99.51
1,84
95.54
20.32
91.58
35,94
90.13
44.26
94.19
25.59
400
99.50
L90
95.50
20.61
91.66
35.76
90.25
43.62
94.23
25.47
99.50
95.53
20.48
91.67
35.73
90.21
43.92
94.23
25.50
500
99.52
1.80
95.57
20.20
91.55
36.33
90.02
44.95
94.16
25.82
1000
99.52
1.81
95.57
20.19
91.55
36.32
90.01
44.95
94.16
25.82
A.5.5 EXPLORING THE RELATIONSHIP BETWEEN THE NUMBER OF FUSED GROUPS AND OOD DETECTION PERFORMANCE IN THE GROUPING STRATEGY.
Another question about grouping strategy is that why the performances increases by calculating the average OOD score over multiple independent and identically distributed subgroups. To explore this further, we conduct series of ablation experiments with fusing different number of groups. Based on the 100-subgroup setting in Table 15, we randomly pick 1, 2, 5, 10, 20, and 50 subgroups, with 100 negative labels each subgroup. And then we compute the average NegLabel score over the picked subgroups, repeating each experiment ten times for more accurate performances. The results in Table 16 indicate that OOD detection performance improves gradually with an increase in the number of selected groups. Additionally, the standard deviation generated from multiple experiments decreases progressively. This trend shows that the performance for each subgroup may be affected by the randomness of the negative labels. The average strategy of grouping helps NegLabel to stabilize the OOD score and reduce the variances of zero-shot OOD detection performances.
20

---

Published as a conference paper at ICLR 2024
Table 16: Impact of the number of fused group in the grouping strategy on OOD detection. We choose the group number as 100 so the last row represents the original grouping strategy (baseline).
Number Fosed
iNaturalist
Averue
Groups
AUROCT
FTR951
AUROCT
FPR95L
AUROCT
FPR95
AUROCT
FPR951
AUROCT
FFR95,
98, 90..9sss
6.15#9::37
02.93=9.00
37, 37.27346515
89 10.1499.115
50.6540.111
87 87.814402m
56,20
02.20=88206
36. 36.58-60.23
B8 8ges00z2
4.20=2:::3
02.85-9.56
38. F1-.500
89 1313340111
50.55491201
$7.66
55.07
92.13=nore
36, 10l.9--44
2,00
94.33=R
27.62:500022
-0.41=00022
42.57t51
88.872t00322
08.71
90.24=R.o4
200.39-0.200
80, 43 30.0:3
2.20
04.91 =.0.00
23.30
20.594mmms
89. 40.11
45.7g*2.54
03 83.00=R..om
27. 48:04
20
2.07#36640
05.20 250
21 68 68-66-222
91 32.2--144
37 17.44tu.1
89 80
d4.41=2.152
03. 03.96=9..0x
26. 30..000-0335
99.48 99.0849..04
1.98933.43
95. 95.42 8..000
20. 70.722-0.00
36.22 36.2340
90, 10.12440.0
03.75-4.-222
94.15 #0.002
26.67440333
95.49
20.53
35,59
90.22
43.56
94.21
25.40
A.5.6 EXPLORING SIMILARITY CHOICES EQ. (5).
We tried using 11 distance and KI. divergence to measure the similarity between images and negative labels, and the results are shown below. As the CLIP-like VLM models are trained under cosine similarity supervision, the features are naturally measured in the cosine space.
Table 17: Choices for similarity measurement between images and negative labels.
OOD Dataset
Choices
NNaturalist
SUN
Places
Textures
Average
AUROC1
FPR95]
AUROC1
FPR95_
AUROCT
FPR951
AUROC1
FPR95]
AUROC1
FPR95_
Cosine (baseline)
99.49
1,91
95.49
20.53
91.64
35.59
90.22
43,56
94.21
25.40
KL. Divergence
98.19
9.19
78.75
85.47
68.61
90.47
73.84
90.07
79.85
68.80
1.1 Distance 97.34
11.95
80.28
79.39
69.29
89.14
66.57
86.51
78.37
66.75
Observing the experimental results, can be seen that when using L1 distance and KL. divergence metries, there is significant drop in performance SUN, Places, and Textures datasets, while the impact on iNaturalist dataset is relatively small. This is because our method (based on cosine similarity) achieves 99.49 % AUROC on iNaturalist, almost completely distinguishing between ID and OOD data. Therefore, even when using metrics such as Ll and KL divergence that are not suitable for cosine space, there is still significant difference between ID samples and OOD samples from iNaturalist. For more discussions on iNaturalist, please refer to Appendix B.1.
A.6 ABLATION ON VLM ARCIITECTURES.
The detailed results are shown Table 18.
Table 18: Comparison with different VLM architectures on ImageNet-1K (ID). All values are percentages. indicates larger values are better and indicates smaller values better.
Architocture
Backbone
INaturalist
SUN
OOD Dataser
Places
Textures
Averuge
AURROC
FPR95
ALRROC1
EPR95
AUROC1
FPR95]
AUROC1
FPR95]
AUROCI
FPR95]
16
99.48
2,00
94. 18
29.11
88.85
48.14
91.23
38.74
93.43
29.50
CLIP
VIT-B/32
99.11
3.73
95.27
22.48
91.72
34,91
88.57
50.51
93.67
27.92
VIT:B/16
20.53
VIF- /14
99.53
1.77
95.63
22.33
93.01
32,22
89.71
42.92
94.47
24.81
ALIGN AICCIIP
EfliiienNee-BB B7
98.86
4.55
90.93
43.18
85.03
61.01
88.24
49.89
90.76
39.66
GroupViT
GroupViT
98.077
8.60
91.52
35.12
88.83
41.63
88.45
47.06
91.72
$3.10
A.7 DISCUSSION ABOUT FORT AL. (2021).
Fort et al. (2021) the ground-truth OOD labels for OOD detection, which is inconsistent with the OOD detection setting in real scenarios. may not be appropriate and fair to compare Fort et al. (2021) 1) in Table 1. The performance comparison between our method and Fort et al. (2021) is show in Table 19.
21

---

Published as a conference paper at ICLR 2024
Celarst Narre: Irage lot
Baraye Nore: Peagettt Than SNTEOT
Oraatt tha here LINCOEE Scsene
Catautrame
Poitin
Dizaut lame: Frage Me
Catsuttiams maggle
Swrsen Narter Insortee
Cetastiinn magela
KGT (10 1/M
Figure 4: Case visualization of ID images. The left part of each subfigure contains the original image, its filename and its dataset name. The right part shows the affinities among ID (blue) and negative (green) labels.
22

---

Published as a conference paper at ICLR 2024
A.tiicinneee with www.haaaeeehe
Ostasit Nuturale
14
Catanot Name: Packs
Rarmett Hanes
Swogen Name SIM
Namnc SUN
8.0
wu Newe catel cir
Figure 5: Case visualization of OOD images. The left part of each subfigure contains the original image, its filename and its dataset name. The right part shows the coftmax-normalin affinities among ID (blue) and negative (green) labels.
23

---

Published as a conference paper at ICLR 2024
In the original implementation of Fort et al. (2021), they directly apply softmax function on the cosine similarities between image and text embeddings. ie.. take temperature 1 in Eq. (6) in paper. However, we observe that its performance on the ImageNet- Ik benchmark is not as expected because the difference between positive matching and negative matching is too small, mentioned in the discussion about the temperature coefficient of NegLabel Score in Section 3.2. We replace the temperature by r 0.01 and achieved satisfactory results, with AUROC of 96. 86.88 and FPR95 of 14.69%, exceeding all existing results (including zero-shot and non-zero-shot OOD detectors) reported on the ImageNet-I 1k benchmark. But we remind again that now the OOD detector works in an oracle situation because all OOD labels are added to the negative labels as known information.
Table 19: The performance comparison about Fort et al. (2021) ImageNet- 1k benchmark.
iNaturalist
SUN
Places
Textures
Average
Method
7
AUROC
FPR95
AUROC
FPR95
AUROC
FPR95
AUROC
FPR95
AUROC
FPR95
NegLabel
0.01
99.49
1.91
95.49
20.53
91 64
35.59
90.22
43.56
94.21
25.40
Fort et al. (2021)
1
98.48
7.73
85.41
75.54
76.42
86.14
79.33
87.39
84.91
64.20
Fort et (2021)
0.01
99.57
1.47
98.37
7.11
94.28
25.32
95.30
24.88
96.88
14.69
B FURTHER ANALYSIS
B.I ANALYSIS OF THE PERFORMANCE ON INATURALIST
We think that the performance differences between iNatuallist and other OOD datasets mainly stem from the characteristics of the OOD datasets themselves. In Fig. 4 and Fig. 5, we provide a few sample images from the ID and OOD datasets for reference. The ID dataset ImageNet-Ik contains large number of categories related to animals and food, while iNaturalist consists of images of natural plants. SUN and Places datasets contain images of natural landscapes. Therefore, compared to other OOD datasets, iNaturalist has the largest differences in category terms compared to ImageNet, such as animals vs. plants. Therefore, introducing negative labels can significantly improve the OOD detection performance. On the other hand, SUN and Places datasets often contain multiple elements from the natural world beyond their annotated ground truth, making them more prone to confusion with the ID data.
B.2 DISCUSSION ABOUT ZOC.
ZOC (Esmacilpour et al., 2022) employs captioner that has been pretrained the COCO dataset (Lin et al., 2014) to generate multiple candidate labels for each image. These labels are then merged with the existing ID labels to calculate OOD scores. Consequently, the generated labels by ZOC are all derived from the COCO dataset, which poses a disadvantage when applying ZOC to large-scale datasets.
To address this limitation, we try to reproduce an upgraded version of ZOC on ImageNet-I Ik bench- mark. We leveraged the capabilities of the multimodal model LLaVA (Liu al., 2023) and ChatGPT to migrate ZOC to large-scale data sets. The idca comes from MCM's discussion and reproduction of ZOC.
We first used LLaVA to describe each sample and extract the entities contained in the description. Then we encountered a difficulty. LLaVA's descriptions of a large number of ID samples were slightly different from their corresponding ImageNet- 1k labels, such as dog v.s. Alaskan Malamute. As result, the ID label filtering policy cannot take effect. In order to overcome this problem, we collected all entities generated by VA and used ChatGPT to filter words similar to the ImageNet-16 1k label. The remaining words are treated as candidate OOD labels of ZOC. The results shown in Table 20 demonstrate that our method NegLabel significantly outperforms ZOC.
There are several reasons for this phenomenon: Firstly, ZOC generates candidate out-of-distribution (OOD) labels for both in in-distribution (ID) and OOD data. However, for ID data, the captioner may produce synonyms or closely related words to the ID labels, which can have high similarity with the image. This can lead to ID images being incorrectly classified as OOD. Secondly, the performance of ZOC heavily relies on the dataset used for pretraining the captioner. If the OOD data is not part of
24

---

Published as a conference paper at ICLR 2024
Table 20: Performance comparison with ZOC.
OOD Dataset
Method
Naturalist
SUN
Places
Textures
Average
AUROCT
FPR95]
AUROC
FPR95]
AUROCT
FPR95]
AUROC1
FPR95]
AUROCT
FPR95.
ZOC
86.09
87.30
81.20
81.51
83.39
73.06
76.46
98.90
81.79 94.21
85.19
NegLabel
99.49
1,91
95.49
20.53
91.64
35.59
90.22
43.56
25,40
the captioner's pretrained categories, the generated candidate labels will be inaccurate, resulting in inaccurate scores for OOD data.
Compared to ZOC, our method, NegLabel, does not require additional complex pretraining processes and additional images. NegLabel is simpler to use and can easily generalize to different data and models. NegLabel also offers more stable and powerful performance of zero-shot OOD detection.
B.3 DISCUSSION ABOUT CLIPN.
CLIPN utilizes additional datasets train text encoder that understand negative prompts. It discriminates OOD data by comparing the similarity differences between the outputs of two text encoders and the image encoder. On the other hand, our method does not require additional data or training. It only introduces negative labels from the corpus to accomplish zero-shot OOD detection tasks. Therefore, category to reject inputs unrelated to the ID category. In terms of computational complexity, our method slightly higher than CLIPN, complexity of CLIP itself. In terms of specific performance, our method has a higher average performance on the ImageNet- Ik benchmark. Morcover, compared to CLIPN and NegLabel on CLIP-B/16 and CLIP-B/32, our method NegLabel is more robust model architectures.
terms of the idea, both methods aim to introduce negative semantics of the ID
almost negligible compared to the computational
Table 21: Performance comparison with CLIPN.
0OD Dataset
Method
INaturalist
SUN
Places
Textures
Average
AUROCT
EPR95
AUROCT
FPR95L
AUROCT
FPR95L
AUROC1
FPR95]
AUROCT
FPR95
CLIPN (CLIP-B/32)
94.67
28.75
31.87
86.93
50.17
87.68
49.49
90.53
40.07
CLI (CI R/16)
95.
23.94
93.93
26. 17
90.93
40.83
92.28
33.45
93.10
31.10
NegLa (CLIP B/32)
99.11
3.73
95.27
22.48
91.72
34.94
88.57
50.51
93.67
27.92
NegLabel (CLIP-B/16)
99.49
191
95.49
20.53
91.64
35.59
90.22
43.56
94.21
25.40
B.4 CASE ANALYSIS
To better illustrate the effectiveness of NegLabel, we provide some visualization results of ID and OOD images, shown in Fig. 4 and Fig. 5. All the images are picked from the datasets in ImageNet- 1K OOD detection benchmark. Fach subfigure shows the original image and the top-5 affinities between the image and ID (blue) and negative (green) labels given by CLIP. The affinities defined as softmax- normalized cosine similarities with temperature coefficient J = 0.01.
In Fig. it is obvious that ID images have higher affinities on ID labels, and usually have low affinities on negative labels because of our proposed NegMining algorithm. For example, the first subfigure (row column 1, ILSVRC2012_ val 00000965) gets high affinity its ground-truth ID label and gets low affinities on other labels, which is a good case of CLIP classifier, demonstrating typical working scenario of NegLabel. However, CLIP does not always accurately determine which label an image belongs to. The image in row 4, column (ILSVRC2012 val val_00000059) shows bad case of CLIP that the image is misclassified as "great white shark" and even confused between Alaskan and Husky. In fact, the image contains husky wearing shark coat, but suffers severe inter-class confusion, resulting in poor performances in MCM (Ming et al., 2022a), Energy (Liu et al., 2020), and etc. Fortunately, NegLabel with the sum- aum-softmax score checks the affinities between the image and the whole ID label space, thus maintaining stable OOD detection performances even when the zero-shot classifier performs poorly. The same results can also be in row 2, column 1 UISVRC2012 2_ val_00001
(00000 764) and row 3, column (ILSVRC2012 00004842).
25

---

Published as a conference paper at ICLR 2024
In Fig. 5, OOD images usually have higher affinities on negative labels, because the negative labels contain various concepts and entities different from ID labels. As zero-shot classifier, CLIP- like VLMs may not be good at fine- grained classification, making the affinities between OOD images and negative labels away from the one-hot distribution. As a result, the classification results for OOD images may be confused (like row 1, column 2, I (425555c3)) or even incorrect (like row column 1, ef4509 180199801100 us how "unlike" the image is from the ID data. Benefiting from the extensiveness of the large-scale corpus, almost all OOD images can have reasonable negative labels corresponding to them. This shows that NegLabel utilizes more knowledge from the text domain of VLMs and provides new clues for detecting OOD samples.
But these negative labels tell
Some readers may be concerned about whether it is fair that the corpus contains wide range of concepts, and may even directly cover the semantic labels of OOD samples. We think this is fair when the corpus is large enough, just VLMSs are reasonable for evaluation zero-shot tasks, even though they may have seen this task-relevant data when they were trained. Besides, developers have prior knowledge of potential OOD labels, they manually include them in the negative label space to achieve better OOD detection performance.
B.5 DETAILED DERIVATIONS
About FPRx, We use FPRA as the performance metric of an OOD detector to show the separability between the ID and OOD samples, which is defined as
FPRA Fout (F (A))
(12)
where À [0, 1] represents the true positive rate (TPR), which indicates the proportion of samples that are correctly classified as ID. The cumulative distribution functions (CDFs) Fin and Fou. correspond the scores obtained by the ID and OOD samples, respectively. The metric FPR, quantifies the degree of overlap between the scores assigned by the OOD detector to the ID and OOD samples, with lower values indicating better performance. Therefore, the FPRA metric, specifically in relation to the toy function S, can be defined as follows:
FPRA
X; Mr v Mp1(1 ppi] Mp2, v Mp2(11 - p2)
Φ-1 1
>; v Mp (1-p)) Mp2
1
+ orf
2
22Mp2( - -22)
(13)
1 1
Pi (1 PI)
M (pl P2) v 2p2 (1 P2)
+
erf
erl (2X 1)
2
P2 (1 p2)
where
MrL, Mp1(1 p1)
2Mp prf e
(2) - 11) + Mp1.
(14)
About the deviation of FPR We target investigating the relationship between the FPR, metric and the number of negative labels M. Therefore, we calculate the partial derivative with respect to M and use = P1
P1
erf 1 (2X 1) + MMp Pa) to simplify the expression:
Pa
/2pal (1-2))
OFPR X
der((z)
0z 0M
0M
2
Oz
2c =
P1 P2
(15)
2
2 2M p2 (1 p2)
-
P1 P2 < 0.
2
MM (1 - P2)
26

---

Published as a conference paper at ICLR 2024
B.6 MORE THEORETICAL ANALYSIS
In this section, we will consider a more general case for our analysis, where the probability that the label of æ: is Dt is Pi instead of used in Section 3.3. Thus, the positive count c = >; of will follow Poisson binomial distribution instead of binomial distribution. In the following, that Poisson binomial distribution can also be approximated by a Normal distribution based on the Lyapunov central limit theorem where the -iittillli assumption made in the traditional central limit theorem is related.
find
Formally. let S; be a Bernoulli random variable with parameter Pt. which means that the probability of S; = 1 is Pt and the probability of S; 0 is Pi. Here, we assume that, for any E Z+, ps E 'Pminn Pmax] where Pmin > 0 and Pmax < 1. Then, we C = S + S2 + ....... SM- Thus, we know [[ = pi Var[S:] = (1 - Pt), E[[] Σ Pi, and Var[C] = Σi= p: p.). Based on the above expectation and variance values, we want verify the Lyapunov condition: for some
5 > 0,
lim M-+00
(Var (ar// EE [S. [ B[S.]]=+]] 0.
(16)
We first analyze the term B||S, B[S 2- based on S. and E[S.].
E S. - E[s.] 2+8
= |1 -[[[S]]
2+ Pr(S; = 1) + 10- -
0)
= |1 pil p212ttppp ps + 10 - pıl 2 - p:)
(1 p:)² )2 p: + p,:
2+ (1 pt)
= (1 - pt )p)pp(( [(1 - p.) 1+8
12) + p, 6
(17)
Thus, we know El|s; E[SS]]1++)] < Then, we analyze (Var (Var(C))8/2.
M
3/2
M
3/2
(Var(C))3/2== =
Σ Px( pp)) pi)
Σ
3/2
M3/2
(18)
where 6 = min Ppmin - Pinian Pmux - Pmax } > 0. Based on Eq. (17) and Eq. (18), we have
M
1 (Var(C))3/2
2M
2
(19)
3/2 3/2 M3/2
3/2
/M
G
Thus, as M + 00,
0
1
Σ
B [is: [[s]]]2t]]
+ +0
(20)
(Var(C))3/2
meaning that
lim M (Var(C])3/2
Σ
E L
E[S.]]++]
0.
(21)
Hence, we verify the Lyapunov condition for C and S,. Based on the Lyapunov central limit theorem, we know that, as M goes to infinity,
M
Σ
S - E(s.) 4 NO 1),
(22)
VVrlCC
where 4 means "converges in distribution". Thus, C= IM S,, Poisson binomial random variable, approximately follows NC B[S.], Var(CT) = NEECC Var(C]), for a sufficiently large M. Then, following Section B.5, we can obtain the same result.
'It means that we only consider random variables with real randomness. i.e., these random variables have positive probabilities to at least take two values as their outputs. If p. = (or p. = 1) for some i, then we know S, can only take the value of 0 (or 1) and has no probability to make S, = 1 (or S, = 0).
27

---

Published as a conference paper at ICLR 2024
C
ALGORITHMS
Here give detailed implementations of the proposed algorithms. The central concept behind the NegMining algorithm (see Algorithm 2) is as follows: firstly. calculates the 100y-th percentile nearest distance between negative candidate and the entire ID label space. Next, it selects the top-k candidates that have the greatest distances from the ID labels as the negative labels. This strategy guarantees that for every y y, there exists substantial margin between the negative label and the ID label.
Algorithm 2: NegMining (Detailed Version)
Input :Candidate labels y ID labels ) Text encoder flext Output Negative labels y
// Calculate text embeddings
for y/s E y do
L e; ftest (prompt(y.))
for Y/ E y do
é, = prext (prompt(j.)):
// Measure the distance from ID labels for each candidate label.
negsim; cos(e, ex))
negsim, <- reverse_son srversessort((iii sort (eegsimm):::;;
d, 4tthe 100r-th element in negsim,
Pick M negative labels with respect to the top-k distances.
d, d2 de) reverse reverse_sort([dd__ de......el
According the rank of d, dz,... .. dc] reorganize y to y':;
10 y + first M elements in y'c
The NeglL score in sum-softmax form is shown in Algorithm 3 and the NegLabee score with the grouping strategy is shown in Algorithm
Algorithm 3: NegLabel Score
Input Model input x, ID labels V, Negative labels y", Text encoder flext, Image encoder fimg
Output: NegLabel score S(x)
// Preprocess: calculate text embeddings.
for y/ E y do
L e; ftext (prompt(y.));
for y E y do e ptext (prompt(7))::
// Inference time. (x);
con(h.g))+
5(x)
() y coothey))
28

---

Published as a conference paper at ICLR 2024
Algorithm 4: Lael Score with the Grouping Strategy
Input Model input x, ID labels V Negative lahels y", Text encoder flext, Image encoder fime, Number of grouping ?g
Output: NegLabel score S(x)
// Preprocess: calculate text embeddings.
for y/ E y do
L e; neet (I))m
for y; E y do L é, = plext (p))mpt(77))):
Divide the text embeddings of negative labels into ng groups.
// Inference time.
h = f
for each group do
co)(k.a))/
81
Ming
conkhig/
S(x)
Ei 81
29


Reviews:
[{"review_id": "xRKHi28pC7", "reviewer": "Reviewer_RPtB", "summary": "This paper introduces a novel method for zero-shot out-of-distribution (OOD) detection, leveraging the capabilities of vision-language models (VLMs). The primary objective is to enhance OOD detection by incorporating textual information. NegLabel introduces a large number of negative labels, extending the label space to distinguish OOD samples more effectively. The method uses a novel scheme for the OOD score, taking into account affinities between images and both ID labels and negative labels. The paper provides a theoretical analysis, justifying the use of negative labels.", "questions": "- I wonder why the authors did not add the simple and strong baseline Mahaloanobis (Lee et al. 2018) /SSD (Sehwag et al. 2022).\n- No ablations are provided on $n_g$. How do the authors set this parameter and what is its impact?\n- Would the hard OOD detection task be better qualified as Open-Set Recognition?\n- Why the comparison with ZOC is performed using ImageNet-200 and not ImageNet-1k as in Table 2? A Homogeneous comparison setup would greatly enhance the paper quality. \n\n**References**\n\nLee, K., Lee, K., Lee, H., & Shin, J. (2018). A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks. Advances in Neural Information Processing Systems, 31. \n\nSehwag, V., Chiang, M., & Mittal, P. (2022, February 10). SSD: A Unified Framework for Self-Supervised Outlier Detection. International Conference on Learning Representations.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- **Well-written and technically sound paper** The paper is well-structured and its concepts and contributions are clearly presented.\n\n- **Quality illustrations**\n\n- **Extensive experiments** The paper conducts extensive experiments, including OOD detection, hard OOD detection, and robustness to domain shifts. \n\n- **Large and consistent gains compared to several SOTA baselines** The paper's results show substantial improvements over several state-of-the-art (SOTA) baselines. \n\n- **Intuitive justification and theoretical analysis provided** The paper offers an intuitive justification for the proposed method and includes theoretical analysis to explain the mechanism of negative labels.\n\n- **Extensive ablations**", "weaknesses": "- **Assumption about CLIP's latent space** The assumption that ID image and corresponding ID label embeddings are close in CLIP's latent space is actually false. The contrastive learning strategy of CLIP is based on cosine similarity and thus textual and visual embeddings are actually on different manifolds see (Weixin Liang et al. 2022) for more details. Using ID labels as proxies for the ID images may thus produce unexpected results. In my understanding, the negative labels are compared to the images using cosine similarity, and thus having them close to ID textual embeddings still makes sense but the authors should be cautious here.\n\n- **Comparison with related work** The method is close in spirit to CLIPN (Wang et al. 2023) as well as ZOC (Esmaeilpour et al 2022). The former fine-tunes CLIP to incorporate negative prompts to assess the probability of a concept not being present in the image. An extensive comparison with CLIPN would be overkill but some discussion about it would be welcome.  The second also performs zero-shot OOD using maximum softmax probability on the labels extended with the object detected in the images. Comparisons with ZOC are presented in the supplementary but would better belong in the main paper IMO as it is one of the few recent zero-shot OOD detection baselines. To be fair this comparison should be provided with the same set-up as in Table 2.\n\n- **Misleading baseline presentation** Even if the MSP, ODIN, or Energy Logit baseline are generally used on models fine-tuned on a downstream task, one could also use these detectors in a zero-shot setting. Classifying them in purely zero-shot can be misleading eg. MCM can be seen as zero-shot MSP on CLIP's output probabilities.\n\n- **Short Related Work** The related work section is relatively short, especially for such a prolific field as OOD detection. I don't think it is really detrimental to the paper but a discussion on the sense of OOD detection for foundation models trained on vast and various data would be welcome. \n\nI am willing to improve my rating depending on the author's rebuttal.\n\n**References** \n\nWang, H., Li, Y., Yao, H., & Li, X. (2023). CLIPN for Zero-Shot OOD Detection: Teaching CLIP to Say No, ICCV 2023.\n\nEsmaeilpour, S., Liu, B., Robertson, E., & Shu, L. (2022). Zero-Shot Out-of-Distribution Detection Based on the Pre-trained Model CLIP. Proceedings of the AAAI Conference on Artificial Intelligence, 36(6), Article 6.\n\nLiang, W., Zhang, Y., Kwon, Y., Yeung, S., & Zou, J. (2022, May 16). Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning. Advances in Neural Information Processing Systems.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "M36BQvuzTV", "reviewer": "Reviewer_JPmP", "summary": "This paper is a pioneer work to study the OOD detection problem under VLMs. Compared to state-of-the-art papers in this frontier, this paper addresses several issues that SOTA methods do not cover. These issues are justified well in this paper, and the corresponding explanation is reasonable and solid. Then, a OOD-word selection method and a corresponding score function is proposed to address the observed issues. The proposed method makes sense and works well in the extensive experiments.", "questions": "1 Figure 1 is clear and important. However, it makes me think about one question: in a statistical view, what should the right subfigure be?\n\n2 Why does grouping strategy appear in Section 3.2 instead of 3.1? It seems that Grouping Strategy is a more advanced way to pre-process negative labels?\n\n3 Can the authors explain why is the case considered in B.3 more general? More explanation can make readers know your analysis better.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1 As a pioneer study in this field, I find that this paper is easy to follow and demonstrates issues of SOTA methods very clearly. Thus, in my point of view, this paper has a great potential to motivate more relevant work in this important field: OOD detection with VLMs/Zero-shot OOD detection.\n\n2 I am convinced that the proposed work has a solid contribution by the extensive experiments conducted by the authors. The experiments cover many scenarios, which is a solid evidence that the proposed method has its own contribution to this field.\n\n3 This paper also benefits from the theoretical understanding of the proposed score function. The analysis is easy to follow and looks totally new to me.", "weaknesses": "1 Figure 1 is clear and important. However, it makes me think about one question: in a statistical view, what should the right subfigure be?\n\n2 In Algorithm 1, the comments are too long. It would be better to say the key functionality of this part.\n\n3 Why does grouping strategy appear in Section 3.2 instead of 3.1? It seems that Grouping Strategy is a more advanced way to pre-process negative labels?\n\n4 Can the authors explain why is the case considered in B.3 more general? More explanation can make readers know your analysis better.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KaPTt5Iouf", "reviewer": "Reviewer_9gus", "summary": "In the traditional OOD detection, the text information of labels will be abandoned, which actually loses some information regarding labels. With the assist of VLMs, this kind of information might be useful for OOD detection. The authors present a study aligning with this interesting and new research direction: OOD detection with VLMs. The existing methods are quite on the early stage, and this paper challenges these methods and proposes a new OOD detection method consisting of a negative label selection method and a new score function. Experiments are solid and form a solid contribution to OOD detection with VLMs.", "questions": "In general, this paper addresses a significant problem and makes a solid contribution to this field. However, please answer questions listed in the Weakness:\n\n- OOD detection methods have different performance on different datasets. On some datasets, the performance is extremely good for all methods. Can we see the difference among these datasets? Why can OOD detection be easily addressed on some datasets?\n\n- What is the relationship between Bernoulli distribution and binomial distribution? This is a well-known result in statistics?\n\n- Why is B.3 more general than things demonstrated in main context?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "+ Experiments are solid and cover many aspects, which can address many in-the-mind issues\nautomatically. It is enjoyable to the whole experimental section, which also contains some interesting insights directly from the experiments conducted in this paper.\n\n+ The idea to design the new score function is straightforward after demonstrating an intuitive motivation. The motivation of this paper is supported by some evidence instead of \"wording\", which appreciated. \n\n+ The presentation is generally good. The flow is great to follow, and necessary analysis is easy to follow as well.", "weaknesses": "- OOD detection methods have different performance on different datasets. On some datasets, the performance is extremely good. Can we see the difference among these datasets? Why can OOD detection be easily addressed on some datasets?\n\n- Especially for results based on CLIP-B/16 with various ID datasets, the performance is too high. More explanation is need. \n\n- Figures might be better to illustrate the final performance in Section 4. \n\n- What is the relationship between Bernoulli distribution and binomial distribution? This is a well-known result in statistics?\n\n- Why is B.3 more general than things demonstrated in main context? \n\n- Please keep the naming strategy consistent in Algorithm 1 and other algorithms.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eSDr0YK1j1", "reviewer": "Reviewer_dUt1", "summary": "The field of out-of-distribution (OOD) detection seeks to recognize samples originating from unknown classes, thereby ensuring the trustworthiness of models when confronted with unanticipated inputs. While there exists a vast body of research delving into OOD detection within the visual modality, vision-language models (VLMs) stand out by synergizing both textual and visual data for a range of multi-modal tasks. However, there's a noticeable gap, with only a few OOD detection techniques capitalizing on the textual modality. In this study, the authors introduce a groundbreaking post hoc OOD detection technique, termed NegLabel, which harnesses a plethora of negative labels sourced from expansive corpus databases. They meticulously craft an innovative scheme, wherein the OOD score seamlessly collaborates with these negative labels. A detailed theoretical scrutiny aids in unraveling the intricate workings of these negative labels. Rigorous experimentation underscores that their NegLabel approach not only sets new standards in OOD detection benchmarks but also exhibits a laudable adaptability across diverse VLM architectures. Moreover, the NegLabel technique showcases exemplary resilience when faced with various domain shifts.", "questions": "1 In Table 2, all methods can obtain good results (AUROC>90% or FPR95<10% in the most cases). I actually do not see the necessarity to put that table in the main content. Can we move it to appendix?\n\n2 What will happen if we select all different words from the ID classes?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1 The research topic is very novel and important. There are a few papers focusing on this field. After reviewing this paper and relevant literature, it can be found that this paper moves a solid step to detect OOD samples when VLMs are available. \n\n2 This paper identifies an important issue of relevant literature: how to select the negative words to improve the OOD detection power. Correspondingly, this paper proposes a novel method to address this important issue, which is a significant contribution in my view. The method's effectiveness is also verified by the extensive experiment.\n\n3 The claims of this paper obtains strong support via extensive experiments that are based on various zero-shot OOD detection benchmarks and multiple VLM architectures.\n\n4 The theoretical understanding starts from the data modelling, which is an elegant way to build up an understanding from the scratch. I am glad to see how the score is built up via a well-motivated data modelling method. The use of the probability tools is very interesting in the field, as we can directly see the distribution.", "weaknesses": "1 In Eq. (5), the cosine similarity is selected. Although the reason is given in the paper, it is still interesting to see if we can have other choices.\n\n2 I enjoy reading how you move from basic data modelling to a more realistic data modelling in B.3. However, it would be better to point out why the contents in B.3 is more general, which can increase the interests of readers for understanding the proposed score.\n\n3 In Table 2, all methods can obtain good results (AUROC>90% or FPR95<10% in the most cases). I actually do not see the necessarity to put that table in the main content. It would be better to move some interesting parts in Appendix to the main content.\n\n4 What will happen if we select all different words from the ID classes?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)