Conjugated Semantic Pool Improves OOD Detection with Pre-trained Vision-Language Models
Authors: Mengyuan Chen
Junyu Gao
Changsheng Xu
Venue: NeurIPS 2024 poster (2024)
Keywords: out-of-distribution detection
zero-shot OOD detection
pre-trained vision-language models
Primary area: machine_vision
TL;DR: We show theoretical conditions for improving zero-shot OOD detection and propose a feasible method.

Abstract: A straightforward pipeline for zero-shot out-of-distribution (OOD) detection involves selecting potential OOD labels from an extensive semantic pool and then leveraging a pre-trained vision-language model to perform classification on both in-distribution (ID) and OOD labels. In this paper, we theorize that enhancing performance requires expanding the semantic pool, while increasing the expected probability of selected OOD labels being activated by OOD samples, and ensuring low mutual dependence among the activations of these OOD labels. A natural expansion manner is to adopt a larger lexicon; however, the inevitable introduction of numerous synonyms and uncommon words fails to meet the above requirements, indicating that viable expansion manners move beyond merely selecting words from a lexicon. Since OOD detection aims to correctly classify input images into ID/OOD class groups, we can "make up" OOD label candidates which are not standard class names but beneficial for the process. Observing that the original semantic pool is comprised of unmodified specific class names, we correspondingly construct a conjugated semantic pool (CSP) consisting of modified superclass names, each serving as a cluster center for samples sharing similar properties across different categories. Consistent with our established theory, expanding OOD label candidates with the CSP satisfies the requirements and outperforms existing works by 7.89% in FPR95. Codes are available in https://github.com/MengyuanChen21/NeurIPS2024-CSP.

Conjugated Semantic Pool Improves OOD Detection with Pre-trained Vision-Language Models
Mengyuan Chen
Junyu Gao"
MAIS, Institute of Automation, CAS
MAIS, Institute of Automation, CAS School of Artificial Intelligence, UCAS junyu. grognl.. iia.aaa. ac. cn
School of Artificial Intelligence, UCAS chemeengyuan222110i.. www.:o221ia.........
Changsheng Xu
MAIS, Institute of Automation, CAS School of Artificial Intelligence, UCAS Pengcheng Laboratory csxu@nlpr. wwwppr.aaac......m
Abstract
A straightforward pipeline for zero-shot out-of-distributioo (OOD) detection in- volves selecting potential OOD labels from extensive semantic pool and then leveraging a pre-trained vision-language model to perform classification on both in-distribution (ID) and OOD labels. In this paper, we theorize that enhancing performance requires expanding the semantic pool, while increasing the expected probability of selected OOD labels being activated by OOD samples, and en- suring low mutual dependence among the activations of these OOD labels. A natural expansion manner is to adopt larger lexicon; however, the inevitable introduction of numerous synonyms and uncommon words fails to meet the above requirements, indicating that viable expansion manners move beyond merely lecting words from a lexicon. Since OOD detection aims to correctly classify input images into ID/OOD class groups, we can "make up" OOD label candidates which are not standard class names but beneficial for the process. Observing that the original semantic pool is comprised of unmodified specific class names, we correspondingly construct a conjugated semantic pool (CSP) consisting of modified superclass names, each serving as a cluster center for samples sharing similar properties across different categories. Consistent with our established theory, expanding OOD label candidates with the CSP satisfies the requirements and outperforms existing works by 7.89% in FPR95. Codes are available in https: // rithub..... comm///angyyaaaChee2/////aaa/SS
Introduction
The efficacy of machine learning models typically diminishes out-of-distribution (OOD) data, thereby underscoring the significance of flagging OOD samples for caution. Traditional visual OOD detection methods typically driven by single image modality, leaving the rich information in textual labels untapped [21, 34, 63, 56, 11]. As pre-trained vision-language models (VLMs) develop. employing textual information in visual OOD detection has become a burgeoning paradigm [15, 12, 40, 58, 64, 43, 29]. A straightforward pipeline [29] is to select potential OOD labels from semantic pool and leverages the text-image alignment ability of pre-trained VLM. Specifically, potential OOD labels are selected from WordNet [39] based on their similarities to the In-distribution (ID) label space, and then CLIP [47] is employed classify input images into ID/OOD class groups
"Coresponding authors
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

In this paper, we establish a mathematic model to describe the performance of the above pipeline. Specifically, the activation status of selected OOD labels, aka whether the similarities between OOD labels and an input image exceed an implicit threshold, can be modeled as a series of independent Bernoulli random variables [29]. Depending on the class group of the input image, we refer these as II) and OOD Bernoulli variables for short. As the proportion of selected OOD labels in the semantic pool increases, our theory indicates an inverted-V V performance variation trend, which aligns with the actual observation. We further derive that the peak performance is positively correlated with two factors: the size of the semantic pool and the average expectation of the OOD Bernoulli variables. Considering the mutual influence between factors, clear strategy for enhancing performance involves concurrently enlarging these two factors while maintaining low mutual dependence among the Bernoulli variables. As result, with an existing semantic pool, what need to do is to expand it with additional OOD labels which have higher and independent probabilities of being activated by OOD images.
A straightforward manner of semantic pool expansion is to adopt larger lexicons. However, simple lexicon expansion fails to yield consistent satisfactory outcomes, and we conclude that the inefficacy is attributed to the following reasons: On the one hand, larger lexicons bring numerous uncommon words, whose expected probabilities of being activated by OOD images are minimal, thus resulting in reduction of the average expectation of the OOD Bernoulli variables. On the other hand, larger lexicons introduce plenty of (near-)synonel Jsynonyms for existing OOD label candidates, leading to a high degree of functional overlap and little benefit. The corresponding Bernoulli random variables for )syoonyymsss are highly mutual dependent, which severely violates the independence assumption required by Lyapunov central limit theorem [1], thus failing to achieve the expected enhancement.
The above analysis suggests that viable strategies for semantic pool expansion require moving beyond the paradigm of simply selecting labels from larger lexicons. Since the goal of OOD detection is to correctly classify input images into ID/OOD class groups, we can freely "make up" OOD label candidates which are not standard class names but beneficial for the process. Inspired by the fact that the original semantic pool is comprised of unmodified specific class names (e.g.. "cat", "waller", , "barbershop"), each of which serves as a cluster center for samples from the same category but with varying properties, we correspondingly construct conjugated semantic pool (CSP) consisting of specifically modified superclass names (e.g.. "white creature". "valuable item", "communal place"), "), each of which serves as a cluster center for samples sharing similar properties across different categories. Expanding OOD label candidates with the CSP satisfies the requirements of our theoretical scheme. Specifically, since superclasses used in constructing the CSP include broad semantic objects, the property clusters encompass samples from numerous potential OOD categories. Therefore, these cluster centers, serving as OOD labels, have much higher expected probabilities of being activated by OOD samples, thus increasing the average expectation of the OOD Bernoulli variables. Furthermore, the distribution of these property cluster centers in the feature space is distinctly different from that of the original category cluster centers, resulting relatively low mutual dependence between the new and original labels. Consistent with the established theory, our method outperforms the SOTA method NegLabel [29] with an improvement of 7.89% in FPR95, which underscores the efficacy of our method, Our contributions include:
A theoretical scheme for improving OOD> detection with pre-trained VLMS (Section 3. We derive that an unequivocal strategy for performance enhancement requires concurrently increasing the semantic pool size and the expected activation probability of OOD labels and ensuring low mutual dependence among the activations of selected OOD labels.
An analysis of the inefficacy of simple lexicon expansion (Section 3.2). We attribute the inefficacy to the introduction of numerous uncommon words and meear-)synnnyms, which respectively reduces the expected activation probabilities of OOD labels and brings severe mutual dependence, thereby failing to achieve theoretical enhancement.
An expansion manner beyond selecting labels from existing lexicons (Section 3.3). We construct an additional conjugated semantic pool (CSP), consisting of modified superclass names, each serving as a cluster center for samples with similar properties across different categories. Consistent with our established theory, expanding OOD label candidates with the CSP satisfies the requirements and achieves satisfactory performance improvements.
Extensive experiments and related analysis on multiple OOD detection benchmarks with -tate-of-the-art performances (Section 5), which demonstrate the effectiveness of our method.
Proof and derivations, visualizations, additional experiment results and details are given in Appendix.

---

2
Preliminaries
Task setup. OOD detection leveraging pre-trained vision-language models (VLMs), also termed as zero-shot OOD detection [15, 12, 40, 58, 64, 43, 29I, aims to identify OOD images from ID ones with only natural-language labels of ID classes available. Formally, given the testing image set X' = X U where R n r 0,, and ID label (class name) set ) = {y1..... yK J. where K is the number of ID classes, our target is to obtain an OOD detector G( ("): : :
{in, out), where z E X denotes a test image. It is noteworthy that the zero-shot setting does not require that there be no overlap between the pre-training data of VLMs and the testing data Xt but only stipulates that no ID images are available for model fine-tuning. In other words, the split of ID and OOD data completely depends on how users manually preset the ID label set Vin-
OOD detection with pre-trained VLM and a semantic pool. A straightforward pipeline of this task to select potential OOD labels from a semantic pool and leverages the text-image alignment ability of pre-trained VLM to perform zero-shot OOD detection [29]. Specifically, there are three steps: (1) Fetching numerous words from semantic pool like WordNet [39] as OOD label
candidates; (2) Selecting portion of OOD label candidates most dissimilar to the entire ID label space; (3) Employing a pre-trained VLM like CLIP [47] to obtain similarities between testing images and ID/OOD labels and then performing OOD detection with a designed OOD score.
The OOD detection performance of this pipeline can be modeled as follows [29]. Given the selected OOD label set y = {zI...., }. 0 < m < M, where m is the number of selected OOD labels, and M is the size of the semantic pool. By applying threshold v. we can naturally define p = Ps; 2 f 24, )i as the probability of classifying ID input images a E 2 as positive for the given label 2,, where s; = (fm(((2) fz+)) is the similarity score given by the pre-trained model f. To derive an analytic form for the model's OOD detection performance, we employ a straightforward OOD score function S(x), aka the total positive count across categories for a sample Σ; oi" where s!" is a Bernoulli random variable with parameter y²",, ie., the probability of s!" 1 is p."h and the probability of s = 0 is 1 p.. Consequently, S(x'") follows a Poisson binomial distribution with parameters { pmm }. p.at and ((z are defined similarly. Based on the Lyapunov central limit theorem (CLT) [I], we can obtain the following lemma:
Specifically, S(x²))
Lemma 1. Given independent Bernoulli random variables {51,..., )m) with parameters {pr.... ................................ Pm }.
where O < Ps < 1, as m goes to infinity, the Poisson binomial random variable C - 8 con-
verges in distribution to a normal random variable with distribution N (E Pi, Ei= Ps - p))).
According to Lemma 1, proved in Appendix 1, the distribution of C can be approximated as
C N (E x= p.", Di=1 p." (1- p(") and the distribution of Coul can be approximated similarly.
By denoting Q1 = [p[r U1 Var, [p]), 92 = [p,"], 0/2 Var, [p?"t, we have
Cr N (mq1, mq1 (1 )1) mm C N (mq2, mga(1 92) mv2).
(1)
Thereafter, with the derivation provided in Appendix A.2, we can obtain the closed-fom expression of one of the most commonly adopted OOD performance metric, aka the false positive rate (FPR) when the true positive rate (TPR) is À E [0, 1). denoted by FPRA,
1 1 FPRA-
(1
9 (1 91) U1 erf (2X-11 + 2242 (2) 202 92((1 Φ)) U2
/m(q: - 92)
+ erf
(2)
2
2
where erf(x) 3x So c- e dt. However, contrary the monotonie trend suggested by Eqn. 2, the v actual performances in experiments exhibit an inverted-V V trend as the ratio of selected OOD labels in the semantic pool increases. Therefore, we further optimize the mathematic model by incorporating finer-grained variable relationships, seeking theoretical guidance for performance enhancement.
Methodology
3.1 A Theoretical Scheme for Performance Enhancement
Since selecting OOD labels is typically based on the reverse-order of similarities to the ID label space to minimize semantic overlap, i.e., the most dissimilar OOD label candidates are most likely
3

---

to be selected, the expected probability, 91. of OOD labels being activated by ID images is not static. Specifically. as the ratio of selected OOD labels r = m m/M increases, the expected activation probability ς = [p[p of existing OOD labels for IDD images will monotonically increase, since more OOD labels with higher affinities to ID labels are selected. When all labels in the semantic pool are finally selected, !i will achieve 92. which means the expected probabilities of OOD labels being activated by ID and OOD images are close. Meanwhile, Q2 is considered as a constant when the semantic pool is fixed and the ratio varies, since whether element in the pool (excluding ID labels) corresponds to a potential OOD sample is independent its similarity to the ID label space. Formally, defining Qu as the lower bound of 91. we model the accumulated increase in q1 the ratio r increases from zero with the function u(r). which exhibits following properties:
u(r)= = (r) qo: u(r = Olgo: 92) = 0, u(r = 1|g0- 92) Q2 go > 0, u (r) 2 0.
(3)
Besides, we assume that the absolute value of the curvature of u is constrained within specific range, thereby preventing abrupt changes in the trend of u, which facilitates subsequent analysis. With u(r). we set À 0.5 in Eqn. 2 for convenience and then explore the properties of
1 + erf
m
40 92
u( 10- 92)
0.5
(V2
2
2
Vga 92)
1/2
Denote
2 from Eqn. 4, we can derive that the first-order derivative of FPR..5.
2
V (1=))-))-
denoted as Gr)), can be expressed as
G(rlgo: 92, u, M) = OFPRo.5 Me-#²
42 + u + 2ru vm
90
(5)
Ör
2x2x
92 (1 - 92) - U/
which can be further proved to monotonically increase over the interval (0, 1] with respect to r with the above assumptions. Besides, according to Eqn. 5, we can obtain that
<((u-- - 42)
lim G(r) = lim r-0+
-0,
lim G(r) nau (rddr11)) (1r==)22 2) r-1
(6)
2/r
where x =
- 92) -22 > O.
DDa aNiglate
+Oun -Nelaos
Since FP) is continuous function within our frame- work and satisfies Eqn. 6, it can be deduced that there ex- ists a value ro E (0, 1] where FPR0.5 5 reaches its minimum. Furthermore, FPRg.5 monotonically decreases over the in- terval (0, ro] and increases over the interval [ro, 1]. Since FPR, is smooth continuous function with respect to X, we deduce that FPR, and FPRu.s share similar trends as the parameter r varies, which aligns with the actual results presented in Fig. 1. A more detailed calculation process from Eqn. 4 to Eqn. 6 is provided in Appendix A.3.
Figure 1: Model performances evaluated by FPR50 and FPR95 (lower is better) of our method and NegLabel against the ratio r, which exhibit a trend of initial de- cline followed by increase. Detailed results can be found in Table 8.
Subsequently, we delve deeper into the factors influenc- ing the optimal value of the OOD detection performance evaluated by FPRo.5 (r). When reaches the critical point Tu. the expected performance improvement resulting from "OOD samples being correctly identified due to the addi- tion of new OOD labels" will be equal to the performance degradation caused by "ID samples being misclassified due to the addition of new OOD labels". As a result, the model performance achieves its peak. Specifically, from Eqn. 5, it can be inferred that satisfies
40 22 + u(ro 40: (2) + 2r (rolqo, 92) 0.
Given the complex interdependencies among the variables in the above equation, it is challenging to derive any definitive conclusions with the undefined form of the function u. Consequently, we simplify by assuming that the function u(r) is linear. Under this assumption, by substituting Eqn. into Eqn. 4, we obtain that the optimal value of FPRo.s can be expressed as
(2M) 1 (92 90) V 42 (1- 92) 02
1 + erf 2 2
FPRp.s
(8)
4

---

where the variables M and g2, aka the semantic pool size and the expected probability of OOD labels being activated by OOD samples, are the predominant factors influencing the right side of the equation. The other variables qo. ro. and 12 remain nearly constant with sufficiently large semantic pool (refer to Appendix A.4 for analysis), thus exerting marginal impact.
If we disregard the interdependencies among the variables, the impact of M and q2 on the optimal value of FPR is straightforward: (1) With q2 fixed, can be easily observed that FPRo.5 (ro) monotonically decreases with M. (2) With M fixed, and denoting the input to the erf(-) function as Ç, it can be derived from Eqn. 8 that,
OFPRu.a(ro) 1 derf(() ōç_
Mr0 2m
00(( + qu 2q002 2022)
<0
(9)
0q2
2 aç
āq2
(12(1 - -22) - 02)
holds in almost all practical cases (see Appendix A.5 for analysis), thus FPRo.5 also monotonically decreases with respect to 92. However, in real-world scenarios, the complex interactions between M and d2 prevent either variable from being adjusted in isolation. For instance, utilizing larger lexicon to expand the size M of the semantic pool may cause decline in q2 (see Section 3.2). Conversely, discarding candidates with lower activation probabilities to elevate @2 leads to reduction of M. The variable changes in both strategies exert opposing effects, ultimately leading to minimal improvements or even degradation in model performance. Besides, the Lyapunov central limit theorem (CLT) used in proof of Lemma requires that the activations of selected OOD labels are independent. Although complete independence is impossible to achieve in real-world scenarios, it is essential to maintain a relatively low level of mutual dependence to reduce the errors in theoretical derivations. Therefore, an unequivocal strategy for performance enhancement is concurrently increasing the variables M and d2 and ensuring that there is no strong dependence among the activations of selected OOD labels. With an existing semantic pool, what we need to do is to expand it with additional OOD labels which have higher and independent probabilities of being activated by OOD images.
3.2 A Closer Look at the Inefficacy of Simple Lexicon Expansion
Therefore, it is time to consider how to expand the original semantic pool, which already includes most common words, while ensuring the increase of q2 and low mutual dependence. The most straightfor- ward strategy for expansion, adopting larger existing lexicons, fails to consistently yield satisfactory outcomes, as shown in Fig. 2. Sub- sequently, we analyze that the inefficacy of simple lexicon expansion is attributed to the following reasons.
Negl
2::4
On the one hand, larger lexicons bring numerous uncommon words. whose expected probability of being activated by OOD images are minimal, thus resulting in reduction of q2- As derived in Sec- tion 3.1, the decrease of Q2 attenuates the performance improvements yielded by enlarging M. There two potential reasons for the acti- vation probability p. of an uncommon OOD label 2; being minimal: (I) Pre-trained VLMs lack semantic matching capability for label 2, This issue is particularly pronounced when 2, pertains to concepts such as highly abstract notions (e.g., "idealism", "metaphysics"). complex mathematical concepts (e.g., "Lyapunov condition". "central limit theorem"), "), or specific knowledge of individuals (e.g., personal names excluding celebrities). Pre-trained VLMs are unable to recognize the corresponding content of these text inputs, resulting in p; remaining close to zero (2) The set out lacks testing samples similar to label z, For instance, when the test dataset primarily consists of images of everyday items, new labels constructed from astronomical terms are likely to maintain p: close to zero. 2 'he above scenarios are much more prevalent lexicons of uncommon terms than those of common words. Thereafter, larger proportion of labels with minimal activation probability p.t will diminish 9 = E, [p?" thus attenuating the performance improvement.
10
Lesionss Constructing Semanti Pool
Figure 2: Model performances evaluated by FPR95 (lower is better) with lexicons of differ- ent sizes. Detailed results can be found in Table 9.
On the other hand, larger lexicons introduce plenty of synonyms and near-synonyms for existing OOD label candidates, leading to a high degree of functional overlap with little additional benefit.
In practical applications, input images typically encompass limited categories. Therefore, it is reasonable that infinite expansion of vocabulary richness does not necessarily yield sustained performance improvement.
5

---

For example, the common word "smartphone" can be expanded by adding synonyms such as "mobile phone" and "cellphone". However, the selection results for these words are consistent due to their similar meanings, and if they are selected, the activation of these labels still depends solely on the presence of a smartphone in the input image. This demonstrates a high level of mutual dependency and provides little additional benefit compared to only including "smartphone" in the semantic pool. Despite no reduction in Q2, the corresponding Bernoulli random variables for synonyms, representing whether they are activated by input OOD image, severely violate the independent assumption required by Lemma 1. Although the Lyaрunov CLT used in proof Lemma relaxes the requirement for random variables to have strictly identical distributions mandated by the traditional CLT, it still requires that the variables maintain mutual independence. Despite the random variables corresponding to semantically dissimilar labels are not strictly independent, the intensity of their mutual dependency is generally much lower than that observed in (near-)synonyms. Contrarily, synonyms and near- synonyms lead to significant bias in the approximation of Eqn. 1 and the conclusions derived, thereby failing to achieve the theoretical enhancement.
3.3 Expanding Label Candidates with Conjugated Semantic Pool
The above analysis suggests that viable strategies for semantic pool expansion require moving beyond the paradigm of simply selecting labels from lexicon. Since the goal of OOD detection is to correctly classify input images into ID/OOD class groups, we can freely "make up" OOD label candidates which are not standard class names but beneficial for the process. Inspired by the fact that the original semantic pool is comprised of unmodified specific class names, each of which serves as a cluster center for samples from the same category but with varying properties, we correspondingly construct a conjugated semantic pool (CSP) consisting of specifically modified superclass names, each of which serves as a cluster center for samples sharing similar properties across different categories.
We notice that a class name inherently encompasses a broad semantic range. As shown in the bottom right of Fig. 3, when an image is attached with the class label "cat", it actually depicts one of various more specific situations, such as a "white cat", "tabby kitten", "gray cat", "yawning cat", or "cat on mat". etc. Considering all feature points that correspond more specific descriptions of cats as a cluster within the feature space, the feature point of "cat" can be regarded as its cluster center.
In ideal scenario, each input image in the feature space would be closest (most similar) to the clus- ter center that corresponds to its category, thereby achieving perfect OOD identification. However, the following issues impair the ideal case: (1) Due to the limited capabilities of pre-trained VLMs, some OOD images, such as "white polar bear" in Fig. 3, are closer to incorrect cluster centers than to the correct ones. (2) Owing to the limited scope of lexicons and the inaccuracy of label selection, the category name corresponding to an OOD image, such as "white pea- cock" in Fig. 3, may not exist in selected labels, re- sulting in the absence of an appropriate cluster center. To summarize briefly: not every input OOD image locates close to a correct OOD cluster center.
Toy
Ind
Incesecc resenee Libe
adgid pio Anddcaez werp
Perros Ebgiieen Putlies
ia up iice
Creature
Cat
Butterfly
Figure 3: An illustrative diagram of ele-
ment in the conjugated semantic pool (CSP). Category names can be regarded as the cen- ters of category clusters. Similarly, elements in CSP can be considered as cluster centers of superclass objects with similar properties.
Thereafter, it naturally occurs to us that we should construct more suitable cluster centers to attract such "homeless" OOD images. This is why we expand the original semantic pool by constructing the CSP as follows: Instead of specifying concrete category names (e.g.. "cat", "wallet", "barbershop") "), we utilize superclass names to encompass a wider range of categories (e.g.. "creature". "item". "place"), Instead of leaving category names undecorated, we using adjectives from lexicon as modifiers to attract objects sharing similar properties. As a result, we obtain numerous label candidates of random combinations of adjectives and superclasses (e.g., "white creature", "valuable item", "communal place"). As Fig. 3 shows, in the feature space, "white creature" can be considered as the cluster center of all feature points corresponding to creatures modified by "white", " such as "white cat". "white butterfly", "white polar bear". etc. Note that the semantic scopes of label candidates in the CSP may

---

overlap with ID categories, For example, when "Cat" or "Butterfly" in Fig, 3 are included in ID classes, the label candidate "White Creature" " in the CSP may not be selected as an OOD label.
Consistent with our established theory, our proposed method achieves satisfactory performance improvements by concurrently enlarging the semantic pool size M and the expected activation probability Q2 of OOD labels and ensuring that there is no severe mutual dependence among the activations of selected OOD labels. Firstly, when we expand the original semantic pool with the CSP, the enlargement of M is obvious. Then, since the superclasses used in constructing the CSP typically include broad semantic objects, the property clusters encompass samples from numerous potential OOD categories. Therefore, their centers have much higher expected probabilities of being activated by OOD samples, which brings an increase in g2. Furthermore, the distribution of these property cluster centers in the feature space is distinctly different from that of the original category cluster centers. As a result, the mutual dependence between the new and original labels is relatively low, and the functions of labels from the CSP will not be overshadowed, enhancing the likelihood that an OOD image locates close to a correct OOD cluster center. Experiment results and analysis which support the above claims are provided in Appendix C.I.
Related works
Traditional visual OOD detection. Traditional visual OOD detection methods, driven by the single image modality, can be broadly categorized into four distinct types: (1) Output-based methods, which aims to obtain improved OOD scores from network output, can be further classified into post-hoc methods [21, 34. 25, 54, 55, 44, 38] and training-based ones [10, 23, 57, 26, 71, 65, 30]. (2) Density-basee methods [37, 49, 53, 67, 11] explicitly model the ID data with probabilistic models and identify test data located in regions of low density as OOD. (3) Distance- based methods [32, 51, 63, 56, 41, 7, 70, 59, 24, 18] originate from the core idea that OOD samples should be relatively far away from ID prototypes or centroids. (4) Reconstruutiin-baase
methods [76, 69, 28, 33], which employ an encoder-decoder framework trained on ID data, leverage the performance discrepancies between ID and OOD samples as indicators for anomaly detection. Furthermore, numerous studies [52, 72, 14, 42, 6] offer theoretical contributions.
OOD detection leveraging pre-trained VLMs. By adopting pre-trained VLMs, employing textual information in visual OOD detection has become a burgconing paradigm with remarkable perfor- mance [15, 12, 40, 58, 64, 43, 29]. Fort et al.[ propose feed the names of potential outlier classes to image-text pre-trained transformers like CLIP [47] for OOD detection. ZOC [12] extends CLIP with text-based image description generator to output OOD label candidates for testing. MCM [40] simply adopts maximum predicted softmax value as the OOD score, which is an effective and representative post-hoc OOD detection method based on vision-language pre-training. Based on MCM, NPOS [58] generates artificial OOD training data and facilitates learning a reliable decision boundary between ID and OOD data. CLIPN [64] trains text encoder to teach CLIP to comprehend negative prompts, effectively discriminating OOD samples through the similarity discrepancies between two text encoders and the frozen image encoder. Also based on CLIP, LSN [43] constructs negative classifiers by learning negative prompts to identify images not belonging to a given category. NegLabel [29] proposes straightforward pipeline, that is, selecting potential OOD labels from an extensive semantic pool like WordNet [39], and then leveraging pre- trained VLM like CLIP to classify input images into ID/OOD class groups this study, we explore the theoretical requirements for performance enhancement in this pipeline, and thus construct conjugated semantic pool to expand OOD label candidates, which achieves performances improvements as expected.
Further discussion about Negl Label NegLaabel [29] undertakes a rudimentary theoretical analysis of the correlation between OOD detection performance and the quantity of adopted potential labels, concluding that an increase in selected labels correlates with enhanced performance. However, this conclusion contradicts the observed actual trend. The contradiction arises from that [29] simply assume constant higher similarity between OOD labels and OOD images compared to ID images, neglecting that this similarity discrepancy originates from the strategy of reverse-ordes order selection of OOD labels based on their similarity to the ID label space. As the set of selected OOD labels transitions from "a small subset of labels with the lowest similarity to the entire ID label space" to "the whole semuntic pool, which is unrelated to the setting of ID and 0OD labels". the discrepancy in similarity of ID images to OOD labels versus OOD images to OOD labels will progressively diminish until it disappears. Incorporating the above dynamic to optimize the mathematic model, focus
7

---

on the correlation between OOD detection performance and the ratio of selected OOD labels in the semantic pool, seeking theoretical guidance for performance enhancement.
Experiments
5.1 Experiment Setup
Benchmarks. We mainly evaluate method the widely-used ImageNet- 1k OOD detection benchmark [26]. This benchmark utilizes the large-scale ImageNet- Ik dataset as the ID data, and select samples from iNNaturalsst [60], SUN [66], Places [73]. and Textures [8] as the OOD data. The categories the OOD data have been manually selected to prevent overlap with ImageNet- kk Furthermore, conduct experiments
hard OOD detection tasks, with various ID datasets. Besides, we access whether our method generalizes well to different VLM architectures, including ALIGN [27], GioupViT [68], EVA [13], etc. More details of datasets be found in Appendix B.
Implementation details. Unless otherwise specified, we employ the CLIP ViT-B/16 model as pre-trained VLM and WordNet as the lexicon. The superclass set for constructing the conjugated semantic pool (area, creature, environment, item, landscape, object, pattern, place, scene, space, structure, thing, view, vista), which nearly encompasses all real-world objects. The ablation in Appendix C.5 shows that numerous alternative selections can also yield significant performance improvements. All hyper-parameters including the ratio r which is set to 15%. Additionally, we adopt the same NegMining algorithm, OOD score calculation method, and grouping strategy described in [29]. All experiments conducted using GeForce RTX 3090 GPUs.
directly inherited from [29] without any modification,
Prompt ensemble. We use the following prefixes to construct prompts for labels in the original semantic pool: the, the good (nice), photo of (with) the nice, a good (close-up) photo of the nice. For labels in the conjugated semantic pool, we apply the prefixes: nice (good, close-up) photo of. The bascline results reported the ablation study (see Table 3) also utilize this technique.
Computational cost. The prompt ensemble is constructed over the embedding space to avoid any additional inference cost. Similar to NegLabel, our method is post hoc OOD detector with negligible extra computational burden, which introduces 1% network forward latency.
Evaluation metries. Following previous works [40, 29, 64], adopt the following metrics: the area under the receiver operating characteristic curve (AUROC), and the false positive rate of OOD data when the true positive rate of ID data is 95% (FPR95) [46].
5.2 Evaluation on OOD detection benchmarks
Evaluation on ImageNet- Ik OOD detection benchmark. We compare our method with existing OOD detection methods on the ImageNet- benchmark organized by [26] Table The methods
Table 1: Comparative performance of OOD detection across baseline methods utilizing CLIP ViT- B/16 architecture with ImageNet- k as ID data. Performance metrics are presented as percentages.
OOD Datasets
Methods
INaturalist
SUN
Places
Textures
Average
AUROC?
FPR95
AUROC1
FPR95
AUROCI
FPR95
AUROC1
FPR95
AUROC1
FPR95
MSP [21]
87.44
58.36
79.73
73.72
79.67
74.41
79.69
71.93
81.63
69.61
[34]
Energy [37]
95.33
26.12
92.66
35.97
91.41
39.87
86.76
57.61
91 54
39.89
GradNom [25]
VIM [63]
93.16
32.19
87.19
54.01
83.75
60.67
87.18
51.94
87.82
50.20
39.61
42.19
VOS []]]
94.62
28.99
92.57
36,88
91.23
38.39
86.33
61.02
91.19
41,32
20C [12]
86.09
87.30
81.20
81.51
83.39
73.06
76.46
98
81.79
85.19
MCM [40]
94.59
32.20
92.25
38.80
90.31
46.20
86.12
58.50
90.82
43.93
NPOS [58]
96.19
16.58
90.44
43.77
89.44
43.27
88.80
46.12
91 22
37.93
CoOp [75]
94.89
29.47
93.36
31.34
90.07
40.28
87.58
54.25
38.83
CoCoOp [74]
94.73
30.74
93.15
31.18
90.63
38.75
87.92
53.84
91.61
38.63
CLIPN
LSN [33]
95.83
21.56
94.35
26.32
91.25
34.48
90.42
38.54
92.96
30.22
NegLabel
Ours
99.60
1.54
96.66
13.66
92.90
29.32
93.86
25.52
95.76
17.51

---

Table 2: OOD detection performance comparison on hard OOD detection tasks.
datasets
ImtgeNet-10
ImageNet-20
ImageNet-10
ImageNet-100 ImageNet-10
ImageNet-Ik ImageNet-OI
WaterBirds Placesbg
OOD datnsets
IetgeNet-20
ImageNet- 10
ImageNet-100 100
Methods
AUROC
AUROC
AUROC
IPR95
AUROC
TPR95
AUROC
FPR95
AUROC
FPR995
MCM
98.60
6,00
98.09
13.04
99.39
2.50
87,20
60.00
78.59
64.27
87.45
33.62
Negl
98.80
98.04
11.60
99.37
87.93
49.40
56.65
87.99
Ours
99.02
3.30
98.79
3,40
99.33
2.22
89,59
42.40
88.08
51.50
92.88
12.07
Table 3; Ablation study with CLIP (ViT-B/16) as the backbone on ImageNet- Ik as ID,
Components of the Whole Semantic Pool
INaramlist
SUN
0OD Datasots
Places
Textures
Average
Original Semantic Pool
Simple Labels
Conjugated Semsntic Pool
AUROC:
FPR95
AUROC
FPRO5
AUROC
FIRMS
AUROOC
FPRU5
AUROC
FPRU5
99.63
1.40
96.25
16.63
92.02
33.90
$6.91
57.62
93.70
27.39
/
97.06
14,00
93.52
31.82
90.80
40.23
90.93
40.89
93.08
31.74
V
97.05
13.94
32.43
95.74
18.74
/
99:60
1.5]
96.15
16.52
92.38
32.00
89:26
47.39
91.35
24.36
v
99.61
1.54
96.69
13882
92.85
29.69
93.78
25.78
95.73
17.71
listed in the upper section of Table 1, ranging from MSP [21] to VOS [11], represent traditional visual OOD detection methods. Conversely, the methods in the lower section, extending from ZOC [12] to Legla [29], employ pre-trained VLMMs like CL.IP. is evident that the integration of
textual information through VLMS has increasingly become the predominant paradigm. Our method outperforms the baseline method Neglabel with a considerable improvement of 1.55% in AUROC and 7.89% in FPR95, which underscores the efficacy of our method. The reported results are averaged from runs of 10 different random seeds, whose results are provided in Appendix C.2.
Evaluation on hard OOD detection tasks. Following [40], we also evaluate our method on the hard OOD detection tasks in Table The results of NegLabel [29] are reproduced with its released setting. Our method shows consistently high performances on various hard ID-OOD dataset pairs.
Evaluation with various ID datasets. We also experiment on various ID datasets, including Stanford- Cars [31], CUB-200 [61]. Oxford-Pet [45], Food-101 [2], mageeNet-Sketel [62], ImageNet-A [22], ImageNet-R [20], ImageNetV2 [48], etc. Refer Appendix C.3 for details.
5.3 Empirical evidence supporting our assertions
Performance trends related to the ratio r. In Fig. and Table 8 (Appendix C.4), present the FPR performances of our method and NegLabel against a progressively increasing ratio r, which represents the proportion of selected OOD labels in the whole semantic pool. The color gradations displayed in the table clearly illustrate an initial improvement in model performance followed by a subsequent decline as the ratio increases. This trend aligns with derivation in Section 3.1.
Inefficacy of simple lexicon expansion. In Fig. 2 and Table 9 (Appendix C.4), we assess whether adopting larger lexicons enhances performances. Our findings indicate that it does not always hold. When the semantic pool covers the vast majority of common words, further expansion will introduce an excessive number of uncommon words and (near- Jsynonyms, e thus failing to meet the derived requirements for theoretical performance enhancement. The inefficacy of simple lexicon expansion indicates that viable expansion manners move beyond merely selecting words from existing lexicons.
Validation of consistency between methodology and theory. Refer Appendix C.I.
5.4 A Abltion study
Ablation of different semantic pool components. As shown in Table 3, we explore the effect of three semantic pool components: (1) the original semantic pool, which consists exclusively of specific class names; (2) simple adjective labels employed by NegLabel; and (3) our conjugated semantic pool (CSP). The results demonstrate that the CSP consistently outperforms the simple adjective labels. Furthermore, the highest average performance across the four OOD datasets is achieved when the CSP is employed to expand the original semantic pool.

---

Table 4: Performances of OOD detection with different CLIP architectures on ImageNet- 1k as ID.
OOD Dutnsets
Backbones
Methods
iNatoralist
SUN
Places
Average
AUROCT
FPR9S]
AUROCT
FPR9S]
AUROCT
FPRYS]
AUROC
FPR95L
AUROCT
FPR995
ResNec50
Negl.abel
99.34
2,88
94.54
26.51
42.60
88.40
50.80
92.97
30.70
ResNert01
NegLabel
Ours
99.47
99.27
3.11
2.04
94.96
95.71
24.55
19.50
90.27
89.42
39.57
44.82
90.59
57.22
52.78
38.67
94.01
92.72
24.95
31.32
ResNet5004
NegLabel
99.45
2.27
95.53
21.95
91.62
35.29
89.48
47.77
94.02
26.82
Ours
99.65
1.48
96.26
17.02
92.01
33.42
92.97
29.40
95.22
20.35
ResNec50x16
NegLabel
94.18
29.11
99.68
1.25
95.89
17.89
91.52
35.77
93.80
26.61
95.22
20.38
ResNet50l.61
NegLabel
99.63
1.46
94.29
29.34
91.23
39.18
88.27
49.43
93.36
29.85
Ours
99.69
96.21
92.81
VIT-R/32
NegLabel
Ours
99.46
99.11
3.73
2.37
96.49
95.27
22.48
15.01
92.42
91.72
34,91
31,47
93.64
88.57
25.09
50.51
93.67
95.50
27.92
18.49
NegLabee
1.91
20.53
VIT-R/16
99.61
96.69
13.82
92.85
29.69
93.78
25.78
95.73
17.71
WT-L1114
Negl aahel
Ours
99.72
99.53
1.21
1:77
96.73
95.63
22.33
93.58
93.011
28.41
52.22
92.71
89:71
28.16
42.92
95.69
94.47
24.81
18.17
VIF-L/44-366x
NegLabell Ours
99.67
99.79
1.31
0.86
96.82
95.71
21.60
14.01
93.02
93.64
27.65
32.15
90.38
93.12
27.39
40.44
94.70
95.84
23.88
17.48
Analysis of different CLIP architectures. Table shows that our proposed method consistently outperforms the baseline method NegLabel by large margin with differernt CLIP architectures, which demonstrates our effectiveness.
Ablation of different superclass sets. Refer to Appendix C.5 for details.
6 Conclusion
Summary. In this paper, we propose that enhancing the performance of zero-shot OOD detection theoretically requires: (1) concurrently increasing the semantic pool size and the expected activation probability selected OOD labels; (2) ensuring low mutual dependence among the label activations. Furthermore, analyze that the inefficacy of simply adopting larger lexicons is attributed to the introduction of numerous uncommon words and (near- Saar---snoonms,. thus failing to meet the above requirements. Observing that the original semantic pool is comprised of unmodified specific class names, we correspondingly construct a conjugated semantic pool consisting of specifically modified superclass names, each serving different categories. Consistent with the established theory, expanding OOD label candidates with the conjugated semantic pool satisfies the requirements and achieves consideaale improvements.
cluster center for samples sharing similar properties across
Limitations and Future directions. Our method has following limitations worth further exploration: (1) The effectiveness of CSP depends the implicit assumption that the OOD samples exhibit variety of distinct visual properties. When this assumption does not hold, i.e., OOD samples most share similar visual properties, such the plant images in iNaturalist, the addition of CSP results in a slight performance decline, since most newly added labels are not likely to be activated. Reducing dependency on this assumption is valuable direction for future research. (2) In this work,
primarily focus on analyzing and optimizing the activation status of OOD labels while making no modification to the ID label set. However, there is possibility that selecting additional labels from the semantic pool, including the CSP, to expand the ID label set could enhance the identification of difficult ID samples. We consider this promising direction for future exploration.
Acknowledgments and Disclosure of Funding
This work was supported in part by the National Key Research and Development Plan of China under Grant 2021 12200, in part by the National Natural Science Foundation of China under Grants 62036012, U2 1214444, 62236008, 62102415, U2333215, 62072286, and 62106262, in part by Beijing Natural Science Foundation under Grant 4242051.
10

---

References
[1] Patrick Billingsley. Probability and measure. John Wiley & Sons, 2017.
[2] Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101-mining discriminative components with random forests. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014 4, Proceedings, Part VI 13, pages 446-461. Springer, 2014.
[3] Mengyuan Chen, Junyu Gao, and Changsheng Xu. Uneertaint- aware dual- evidentiat learning for weakly- supervised temporal action localization. IEEE transactions pattern analysis and machine intelligence, 2023.
[4] Mengyuan Chen, Junyu Gao, and Changsheng Xu. R-edl: Relaxing nonessential settings of evidential deep learning. In The Twelfth International Conference on Learning Representations, 2024.
[5] Mengyuan Chen, Junyu Gao, and Changsheng Xu. Revisiting essential and nonessential settings of evidential deep learning. arXiv preprint arXiv:2410.00330 2024.
[6] Ruoyu Chen, Hua Zhang. Siyuan Liang, Jingzhi Li, and Xiaochun Cao. Less is more: Fewer interpretable region via submodular subset selection. arXiv preprint arXiv:2402.09164 2024.
[7] Xingyu Chen, Xuguang Lan, Fuchun Sun, and Nanning Zheng. A boundary based -itt----iitiitttion classifier for generalized zero-shot learning. In European conference on computer vision, pages 372-588. Springer, 2020.
[8] Mircea Cimpoi, Subhransu Maji. Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 3606-3613, 2014.
[9] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hicrarchical image datahase. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255 Iece, 2009.
[10] Terrance Veie and Graham W Taylor. Learning confidence for out-of- of-distribution detection in neural networks. arXiv preprint arXiv:1802.04880 2018.
[11] Xuefeng Du, Zhaoning Wang, Mu Cai, and Yixuan Li. Vos: Learning what you don't know by virtual outlier synthesis. In International Conference on Learning Representations, 2021.
[12] Sepideh Esmaeilpour, Bing Liu, Eric Robertson, and Lei Shu. Zero-shot autoof-distrbuutin detection based on the pre-trained model clip. In Proceedings of the AAAI conference artificial intelligence, volume 36. pages
2022.
[13] Yuxin Fang, Wen Wang, Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. Eva: Exploring the limits of masked visual representation learning at scale. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 19358-19369, 2023.
[14] Zhen Fang, Yixuan Li, Jie Lu, Jiahua Dong, Bo Han, and Feng Liu. Is out-of -fsiiirbbbtt detection learnable? Advances in Neural Information Processing Systems, 35:37199-3723 2022.
[15] Stanislav Fort, Jie Ren, and Balaji Lakshminarayanan. Exploring the limits of out-of-distribution detection. Advancex in Neural Information Processing Systems, 34:7068-7081, 2021.
[16] Junyu Gao, Mengyuan Chen, Liangyu Xiang, and Changsheng Xu. A comprehensive survey on evidential deep learning and its applications. arXiv preprint arXiv:2409.04720) 04720, 2024.
[17] Junyu Gao, Mengyuan Chen, and Changsheng Xu. Vectorized evidential learning for yeakly-suueerisee temporal action localization. IEEE transactions on pattern analysis and machine intelligence, 2023.
[18] Eduardo Dadalto Câmara Gomes, Florence Alberge, Pierre Duhamel, and Pablo Piantanida. Igeood: An information geometry approach to out-of-distribution detection. In International Conference on Learning Representations, 2022.
[19] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pottern recognition, pages 770-778, 2016.
[20] Dan Hendryeks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: A critical analysis of oittof--dstrbuttton generalization. In Proceedings of the IEEE/CVF international conference on computer vision, pages 8340 8349,8339,4000 2021.
11

---

[21] Dan Hendrycks and Kevin Cimpel. A baseline for detecting misclassified and out-of-distribution examples in neural networks. arXiv preprini urXiv:1610.02136, 2016.
[22] Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 15262-15271, 2021.
[23] Yen-Chang Hsu, Yilin Shen, Hongxia Jin, and Zsolt Kira. Generalized odin: Detecting out-of-distribution image without learning from out-of-distribution data. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 10951-10960, 2020.
[24] Haiwen Huang, Zhihan Li, Lulu Wang. Sishuo Chen, Bin Dong, and Xinyu Zhou. Feature space singularity for out-of-distribution detection. arXiv preprint arXiv:2011 146554, 2020.
[25] Rui Huang, Andrew Geng, and Yixuan Li, On the importance of gradients for detecting distributional shifts in the wild. Advances in Neural Information Processing Systems, 34:677-688 2021.
[26] Rui Huang and Yixuan Li. Mos: Towards scaling out-of-diitribution detection for large semantic space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8710-8719,
2021.
[27] Chao Jia, Yinfei Yang. Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoe Le, Kun-Hsuan Sung. Zhen Li. and Tom Duerig. Scaling up visual and vision- language representation learning with noisy text supervision. In International conference on machine learning, pages 4904-996.. PMIR, 2021.
[28] Wenyu Jiang, Yuxin Ge. Hao Cheng. Mingcai Chen, Shuai Feng, and Chongjun Wang. Read: Aggregating reconstruction error into out-of-distribution detection. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 14910-1491 2023.
[29] Xue Jiang, Feng Liu, Zhen Fang, Hong Chen, Tongliang Liu, Feng Zheng, and Bo Han. Negative label guided ond detection with pretrained vision-language models. In The Twelfth International Conferencee on Learning Representations, 2023.
[30] Jang-Hyun Kim, Sangdoo Yun, and Hyun Oh Song. Neural relation graph: A unified framework for identitying label noise and outlier data. Advances in Neural Information Processing Systems, 36, 2024.
[31] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In Proceedings of the IEEE international conference on computer vision workshops, pages 554-561. 2013.
[32] Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for detecting out- of-distribution samples and adversarial attacks. Advances in neural information processing systems, 31, 2018.
[33] Jingyao Li, Pengguang Chen, Zexin He, Shaozuo Yu, Shu Liu, and Jiaya Jia. Rethinking out-o--dittr (ood) detection: Masked image modeling is all you need. In Proceedings of the IEEE/CVF conference on computer vision and pattemn recognition, pages 11578-11589, 2023.
[34] Shiyu Liang, Yixuan Li, and Rayadurgam Srikant. Enhancing the reliability of out-of-distribution image detection in neural networks. arXiv preprint arXiv:/706.06666 2017.
[35] Siyuan Liang, Wei Wang, Ruoyu Chen, Aishan Liu, Boxi Wu, Ee-Chien Chang, Xiaochun Cao, and Dacheng Tao. Object detectors in the open environment: Challenges, solutions, and outlook. arXiv preprint arXiv:2403.1 16271, 2024.
[36] Jeremiah Liu, 7i Lin, Shreyas Padhy, Dustin Tran, Tania Bedrax Weiss, and Balaji Lakshminarayanan. Simple and principled uncertainty estimation with deterministic deep learning via distance awareness. Advances in neural information processing systems, 33:7498-7512. 2020.
[37] Weitang Liu, Xiaoyun Wang, John Owens, and Yixuan Li. Energy- based out-of oit-offrrdibutiio detection. Advances in neural information processing systems, 33:21464 21475, 2020.
[38] Xixi Liu, Yaroslava Lochman, and Christopher Zach. Gen: Pushing the limits of softmax-basse out-of- distribution detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 23946-23955, 2023.
[39] George A Miller. Wordnet: a lexical database for english. Communications of the ACM, 38(1 38(11):39 41. 1995.
12

---

[40] Yifei Ming. Ziyang Cai, Jiuxiang Gu, Yiyou Sun, Wei Li. and Yixuan Li. Delving into out-of-distribution detection with vision-language representations. Advances Neural Information Processing Systems. 35:35087-35102, 2022.
[41] Yifei Ming, Yiyou Sun, Ousmane Dia, and Yixuan Li. How to exploit hyperspherical embeddings for out-of-distribttion detection? In The Eleventh International Conference on Learning Represeniations, 2022.
[42] Peyman Morteza and Yixuan Li. Provable guarantees for understanding out-of-distribution detection. In Proceedings of the AAAI Conference Artificial Intelligence, volume 36, pages 7831-7840, 2022.
[43] Jun Nie, Yonggang Zhang, Zhen Fang, Tongliang Liu, Bo Han, and Xinmei Tian. Out-of-distribttion detection with negative prompts. In The Twelfth International Conference Learning Representations, 2023.
[44] Jacwoo Park, Yoon Gyo Jung, and Andrew Beng Jin Teoh. Ncarest neighbor guidance for out-of-distribution detection. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 1686- 1695, 2023.
[45] Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In 2012 IEEE conference computer vision and pattern recognition, pages 1498-3505. IEBE, 2012.
[46] Foster Provost, Tom Fawcett, Ron Kohavi, et al. The case against accuracy estimation for comparing induction algorithms. In ICMI,, volume 98, pages 445-453, 1998.
[47] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748 8763, PMLR, 2021.
[48] Benjamin Recht, Rebecca Roclofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In International conference on machine learning, pages 5389-538-009 5400. PMLR, 2019.
[49] Ren, Peter Liu, Emily Fertig, Jasper Snoek, Ryan Poplin, Mark Depristo, Joshua Dillon, Balaji Lakshminarayanan, Likelihood ratios for out-of-distribution detection. Advances in neural information processing systems, 32, 2019.
[50] Shiori Sagawa, Pang Wei Koh, Tatsunori Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. arXiv preprint 1rXiv:I911.00 1,8731. 2019.
[51] Chandramouli Shama Sastry and Sageev Oore. Detecting out-of-distribution examples with gram matrices. In International Conference on Machine Learning, pages 8491-8501 PMLR, 2020.
[52] Walter J Scheirer, Anderson de Rezende Rocha, Archana Sapkota, and Terrance E Boult. Toward open set recognition. IEEE transactions on pattern analysis and machine intelligence, 35(7)1175711772, 2012.
[53] Joan Serrå, David Álvarez, Vicenç Gómez, Olga Slizovskaia, José Núñez, and Jordi Luque. Input complexity and out-of-distribution detection with likelihood-based generative models. In International Conference on Learning Representarions, 2019.
[54] Yiyou Sun, Chuan Guo, and Yixuan Li. React: -uitoff-isttibuttin detection with rectified activations. Advances in Neural Information Processing Systems, 34: 144-157, 2021.
[55] Yiyou Sun and Yixuan Li. Dice: Leveraging sparsification for out-of-distributiio detection. In European Conference on Computer Vision, pages 691-70s. Springer, 2022.
[56] Yiyou Sun, Yifei Ming, Xiaojin Zhu, and Yixuan Li. Out-of-distribution detection with deep nearest neighbors. In International Conference Machine Learning, pages 20827-20840. PMLR, 2022.
[57] Jihoon Tack, Sangwoo Mo, Jonghcon Jeong, and Jinwoo Shin. Csi: Novelty detection via contrastive lcarning on distributionally shifted instances. Advances in neural information processing systems, 33:1 1839- 11852, 2020.
[58] Leitian Tao, Xuefeng Du Jerry Zhu, and Yixuan Li. Non-parametric outlier synthesis. In The Eleventh International Conference on Learning Representations, 2022.
[59] Joost Van Amersfoort, Lewis Smith, Yee whye Teh, and Yarin Gal. Uncertainty estimation using a single deep deterministic neural network. In International conference on machine learning, pages 9690 9700. PMLR, 2020.
13

---

[60] Grant Van Horn, Oisin Mac Aodha, Yang Song. Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam. Pietro Perona, and Serge Belongie. The inaturalist species classification and detection dataset. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8769-8778, 2018.
[61] Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-ucse birds-200-2011 dataset. 2011.
[62] Haohan Wang, Songwei Ge, Zachary Lipton, and Erie P Xing. Learning robust global representations by penalizing local predictive power. Advances in Neural Information Processing Systems, 32, 2019.
[63] Haoqi Wang. Zhizhong Li, Litong Feng, and Wayne Zhang. Vim: dustri--ist with virtual-logit matching. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4921 1921-4930, 2022.
[64] Hualiang Wang. Li, Huifeng Yao, and Xiaomeng Li. Clipn for zero-shot ood detection: Teaching clip to say no. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1802-1812, 2023.
[65] Qizhou Wang. Zhen Fang. Yonggang Zhang, Feng Liu, Yixuan Li, and Bo Han. Learning to augment distributions for out-of- of-distribution detection. Advances in Neural Information Processing Systems, 36, 2024.
[66] Jianxiong Xiao, James Hays, Krista A Ehinger. Aude Oliva, and Antonio Torralba. Sun database: Large- scale scene recognition from abbey to In 2010 IEEE computer society conference on computer vision and pattern recognition, pages 1485-3492. IEEE, 2010.
[67] Zhisheng Xiao, Qing Yan, and Yali Amit. Likelihood regret: An out-of-distribution detection score for variational auto-encoder. Advances neural information processing systems, 33:20685-20666, 2020.
[68] Jiarui Xu, Shalini De Mello, Sifci Liu, Wonmin Bycon, Thomas Breuel, Jan Kautz, and Xiaolong Wang. Groupvit: Semantic segmentation emerges from text supervision. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18134-1 144-18144, 2022.
[69] Yijun Yang. Ruiyuan Gao, and Qiang Xu. Qustrfudistoonutioon detection with semantic mismatch under masking. In European Conference on Computer Vision, pages 373-390. Springer, 2022.
[70] Alireza Zaeemzadeh, Niccolo Bisagno, Zeno Sambugaro, Nicola Conci. Nazanin Rahnavard, and Mubarak Shah. Out-of distr--disttibttion detection using union of I-dimensional subspaces. In Proceedings of the IEEF/CVF conference on Computer Vision and Pattern Recognition, pages 9452-9461, 2021.
[71] Jinsong Zhang, Qiang Fu, Xu Chen, Lun Du, Zelin Li, Gang Wang, Shi Han, Dongmei Zhang, et al. -itto--distribution detection based on in- distrriibuuoon data patterns memorization with modern hopfield energy. In The Eleventh International Conference Learning Representations, 2022.
[72] Lily Zhang, Mark Goldstein, and Rajesh Rangamath. Understanding failures in aut-of-distribation detection with deep generative models. In International Conference Machine Learning, pages 12427-12436. PMLR, 2021.
[73] Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image datahase for scene recognition. IEEE transactions on pattern analysis and machine intelligence, 40(6): 1452-1464, 2017.
[74] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision- language models. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition. pages 16816- 16825,,, 2022.
[75] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. International Journal of Computer Vision, 13009):2337-228 2022.
[76] Yibo Zhou. Rethinking reconstruction autoencoder-based ouitt--distii detection. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 7379-7387, 2022.
14

---

A Proof and Derivation
The proof is partially adapted from the appendix of [29].
A.1 Proof of Lemma
Lyapunov Central Limit Theorem. Suppose {81,..., Sm.... " is sequence of independent random
variables, each with finite expected value μ4 and variance o7. Define p 2i=1 07. If for some > 0, Lyapunov's condition
m
m EE
lim
p4 -=+
(10)
2+6 Pm
satisfied, then sum of Si goes infinity:
converges in distribution standard normal random variable,
Pm
m
Σ 5 11-44) 14) 1 NO.
(11)
Lemma 1. Given sequence of independent Bernoulli random variables (81,...,8m. 8m] with parameters . Pm where < ps < 1, as m goes to infinity, the Poisson binomial random variable
C == 8t converges in distribution to the normal random variable:
m
17
C CSN
N
Pi pt)
(12)
Proof. The proof process involves application of the Lyapumov central limit theorem (CLT) [1]. particular form of CLT which relaxes the ddentical-distriuttiio assumption. Since 8; is Bernoulli random variable with parameter Ps. know that μ plsı] Pi, [ P. pt). To keep the notation uncluttered, we use 14 and of for substitution. Based on the above expectation and variance values, we try to verify the Lyapumov condition: Denote pr, Ei= of. for some 8 >> 0,
11a
lim
E
|s
Mil
2+
(13)
70-000
2+5
Analyzing the term E[[s; 14 [2+]] based 8; and μi, we know
B[|s; - Mil [2+5]
= (1 - (1)) 2 Pr(s; = =) 1) (0 24) Pr(st = 0)
= (1 pz)) 2+8 + (0 - 22) - pt)
(14)
(1 pt p:)p; (11- + + -
Thus, we know 0 < El[se - B[st]|2+0] < 2. Then, we analyze
1+5/2
1+5/2
220
(1 - -p))
c 1+6/ 2 1-5/2
(15)
1=1
where = = mpi - pt)
0. Based on Eqn. 14 and Eqn. 15, we have
m
2m
0
[| /4 22
(16)
24 Pm
e1+6/2m1+6/2
61/6
Thus, as m - oc, the squeeze theorem tells us that
m
lim T7 100
Σ 14, [2+5
(17)
2+5
Pi
Hence, we verify the Lyapunov condition for and s;. Based on the Lyapunov CLT, we know that, goes infinity,
m
88 - pi) $ N(),
(18)
15

---

where 4 means "converges in distribution". Thus, for sufficiently large m, the Poisson binomial
random variable C' = Ei= approximately follows
C~M N
Σ 1=1
Mis Pin
=N
Σ 1=1
Pi.
Pc(1 pi)
(19)
i=1
A.2
Derivation of Eq. (3)
First, we denote the cumulative distribution function (CDF) of normal distribution with mean μ and standard deviation as (2; ut o²)), which can be expressed by
1 + erf x --
( p. x²)
(20)
2
V2a
and its inverse function can be expressed by
i Φ (x; u. o²) = V2 eri e - 1) +
(21)
where erf( denotes the integral of the standard normal distribution from æ, termed as error function, and can be given by
erf(x) = VT / 12 dt.
2
(22)
Besides, with the distributions of C and C given by
C N (mqq, mg: - Q1) mm1). C (mqq, mgz(1 (2)) -mm2),
(23)
the false positive rate (FPR) when the true positive rate (TPR) is [0, 1J. denoted by FPR,, be calculated as
FPRA Four (
(A))
= + ( (A; mqn, min (1 q1) mmv1) mq2, mg (22 1002
1
1
Φ (x; mq1 (1 91) - -Me1) mq2
+
erf
2
2
v 1mqq (1 (2) 2mm22
1
1
2mmq (1 91) 22mweer ert 1(2A 1) + mQn - mmg2
+
erf
2
2
/ 2mqq2 (1 - -92) -2m22
1 + erf
1
9i (1 91) U1
/m 92)
erf (22 1) V 242 (1 (22) 2v2
2
92(1
42) 122
(24)
where Fin and Fol denote the cumulative distribution functions which correspond to the scores obtained by ID and OOD samples.
A.3
Calculation process from E3.(3) to Eq.(6)
First, from
FPRA = + erf (Va(1 - 91) (1) 1i erf (2A - 11)
V /migi 42))
(25)
2
ge(1 - 92) -
24g (1 92))
2212
we know that
1 + erf
m
9o 42
u q0- 92)
FPRo.s
(26)
2 2
2
v
92
92)
02
16

---

Denote
v
20 qortlggrrl))
then we can derive that
2
02(1
OFPRo.5
OF.Ro.5
Oz dm
1 [erl((z) Oz dm
G(rlgo: 92, u,
dr
0z dm dr
2 dz
dm Ōr
vmm Du
M
2e
2 400
422 + u)
+
M or
(27)
2
VT
v 2222(1 Q2)
202
Me
90
92
u + M 8r
2v2T
Vm Vm 92 92) 1/
Denote w = vm
1
(90
42 + u +
2m du M लr
then we have
dw
M
du
2 du
2m du
2m Ou
Ör
M Dr
+
M dr
M 0r
VT 2/m
40
+ u +
M dr
M
4m 8² u
4m du
+ 92
2m²
M2 2r
M dr
(28)
M
4m² Du
4m du
90 + Q2
2m
M Dr
M dr
M
du
4r(1
+ (42
40 w)
2 0,
2m3
Dr
where the derivation from the second to the third line utilizes an assumption on the limited range of the curvature of the function u(r), aka |u"| S w'. While recognizing the challenge of consistently maintaining this assumption in complex real-world scenarios, we consider it reasonable for use in intuitive quantitative analyses to simplify derivations. Therefore, we come to the conclusion that
(r) 8PFRRe.
20 0. Besides, according to Eqn. 27, we have
dr
< lim G(r) lim r-t+
q0 92
Ki(00 92)
lim m0
-00;
(29)
r-0+
2V2
Vr (2 (2) 1/2
2/r
lim G(r) -ー
VMu((r= = 1l90. 92)
ku' (r- 1) 2 0,
(30)
V 92) -/2
where x = (M/2T) (g2 (2) - -²==
==²=²²² 0.
A.4 Analysis of some variables in Eq.(8)
Defined as the lower bound of q1- go represents the expected probability of the OOD) label, which is most dissimilar to the ID label space, being activated by ID images. When the semantic pool size is large enough, this OOD label has very different meaning from ID) labels, thus its expected probability qo of being activated by ID images is close zero. With the assumption that the function u(r) is linear, from Eqn. 7 we can derive that = 1/3, which is a constant unrelated to other factors. Empirical evidence also indicates that the change of ro is relatively slight. Besides, 12 defined as the variance of the probabilities of OOD labels being activated by OOD samples. When the semantic pool large enough, this variance changes very slightly with further expansion. Therefore, we come to the conclusion that variables qo. To, and U2 remain nearly constant with a sufficiently large semantic pool, thus exerting marginal impact the right side of Eqn.
A.5 Analysis of a condition in Eq.(9)
By adjusting its shape parameters, a and 3, the Beta distribution can flexibly simulate a range of distinct probability distribution profiles. This flexibility is particularly useful in statistical modeling and analysis where the behavior of probabilities needs to be accurately described. Assuming that y follows Beta distribution, we know that
aB
42 = E, 11
1/2 = Var, (p. out
(31)
a + B¹
(a + () + 3 1+1)
17

---

If 92 202 2 0 does not hold, it means that
2a.3 ( < <0,
(32)
+ 3
3)
which is equivalent
2 20.3 + 3
<0
(33)
and a < min (0.5,,, < 1 is a necessary condition for the above equation. In this situation, the Beta distribution exhibits a pronounced peak at 0 and a long. thin tail stretching towards 1. In our experiments, we observe that the similarity distribution between OOD imput images and OOD text labels exhibits distinct unimodal concentration, with probability densities near 0 and 1 approaching zero, which is entirely different from the distribution shape derived theoretically. Consequently, we can conclude that in the vast majority of practical cases, the following inequality holds,
q2 Q0
20jo22 2v2 Q2 2vz + 20(1 -242) 2 0.
(34)
B Datasets and Lexicons
B.1 Main benchmark
We mainly evaluate our method on the widely-used ImageNet-Ik OOD detection benchmark [26]. This benchmark utilizes the large-scale ImageNet- dataset as the ID data, and select samples from iNaturalist [60], SUN [66], Places [73], and Textures [8] the OOD data. The categories of the OOD data have been manually selected to prevent overlap with the classes of ImageNet-Ik.
ImageNet- 1k, also referred to as ILSVRC 2012, is subset of the larger ImageNet dataset [9]. This dataset encompasses 1,000 object classes and includes 1,281,167 images for training. 50,000 images for validation, and 100,000 images for testing. In the widely used benchmark for OOD detection organized by [26], the validation set of ImageNet-IK is designated as the ID data.
iNaturalist [60] is a fine-grained dataset containing 859,000 images across more than 5,000 species of plants and animals. [26] randomly sample 10,000 images from 110 manually selected plant classes which are not present in ImageNet- 1k as the OOD data.
SUN [66] is scene database which includes 130,519 images from 397 categories. [26] sample 10,000 images from 50 nature-related classes that do not overlap with ImageNet- 1k as OOD data.
Places [73] is another scene dataset containing more than 2.5 million images covering more than 205 scene categories with more than 5,000 images per category. [26] manually select 50 categories from this dataset and then randomly sample 10,000 images OOD data.
Describable Textures Dataset (DTD), consists of 5,640 images from 47 categories of textural patterns inspired from human perception. There are 120 images for each category. [26] use the entire dataset as OOD data.
Textures [8], also referred
B.2 Datesets of hard OOD detection
We also evaluate our method on the hard OOD detection tasks as shown in Table 2. Specifically, the ID-vs-OOD dataset pairs includes ImageNet- 10 vs ImageNet-20, ImageNet- 10 vs ImageNet-100, ImageNet- 1k vs ImageNet-O [22], WaterBirds [50)-vs-Placebg. etc.
IetgeNet-O [22] is dataset of adversarially filtered examples for ImageNet OOD detectors. To create this dataset, the authors delete examples of ImageNet-lk Ik from ImageNet-22k,, and then select examples that ResNet-50 [19] model misclassifics as belonging to ImageNet- Ik class with high confidence. This dataset contains 2,000 images across 200 classes. In a hard OOD detection task, we use ImageNet-Ik Ik the ID data and use ImageNet-O as OOD data.
WaterBirds [50] is constructed by combining bird photographs from the CUB-200 dataset [61] with image backgrounds from the Places dataset [73]. Therefore, WaterBirds keeps the same size with CUB-2000, i..,, contains 11,788 images from 200 bird classes. To construct this dataset, the authors label each bird as waterbird or landbird and place it on one image of water background or land background. hard OOD detection task, we use WaterBirds as the ID data and use its background images OOD data.
18

---

B.3 Datasets of various ID data
As depicted Table 7, our method evaluated against baseline methods, including MCM [40] and NegLabel [29], across various ID datasets. These datasets encompass (1) specialized domain-focused datasets such as Stanford- -arrs [31], CUB-200 [61], Oxford Pet [45], and Food-101 [2]: (2) subsets of ImageNet, including ImageNet10, ImageNe:20, and ImageNet100; (3) ImageNet domain shift datasets, namely ImageNet-Sketch [62], ImageNet-A [22], ImageNet-R [20], and ImageNetV2 [48].
Stanford-Cars, [31] contains 16,185 images of 196 classes of cars. Classes are typically at the level of Make, Model, Year, e.g., 2012 Tesla Model or 2012 BMW M3 coupe. The data is split into 8,144 training images and 8,041 testing images.
CUB-200 [61], formally recognized as Caltech-UCSD Birds-200-2011, comprises 11,788 images across 200 bird subcategories, with 5,994 images for training and 5,794 for testing.
Oxford-Pet [45] is 37 category pet dataset with roughly 200 images for each class created by the Visual Geometry Group at Oxford. The total image number is 7,390.
Food-101 [2] dataset consists of 101 food categories with 750 training and 250 test images per category, making a total of 101k images.
ImageNett-Sktch [62] dataset consists of 50,889 images, approximately 50 images for each of the 1,000 ImageNet classes. The dataset was created using Google Image searches for "sketch of (Class Name]", specifically limiting results to the "black and white" color scheme.
ImageNett- [22] is dataset of 7,500 world adversarially filtered images, which misclassified by a ResNet-50 ImageNet classifier. from 200 classes. The user-tagged images are downloaded from websites including iNaturalist, Flicker, and DuckDuckGo.
ImageNet-R [20], formally recognized as ImageNet- ImageNet-Renditions contains 30,000 images of ImageNet objects from 200 classes with different textures and styles.
ImageNetV2 [48] contains 10,000 new images across the 1,000 categories of ImageNet-Ik: The new images are gathered from the same source of ImageNet to avoid bias.
B.4 Lexicons
As shown in Table 9, we conduct experiments with lexicons of different sizes, and observe that simply adopting larger lexicons does not yield consistent performance improvement. From each lexicon, we select all the nouns to construct the original semantic pool, and use all the adjectives to construct the additional conjugated semantic pool for expansion.
WordNet [39] a large lexical database of English. Nouns, verbs, adjectives and adverbs are grouped into sets of cognitive synonyms, each expressing a distinct concept. In experiments, 70K nouns and adjectives to construct a semantic pool.
the
Common-2001 is a list of the 20,000 most common English words in order of frequency, as determined by n-gram frequency analysis of the Google's Trillion Word Corpus. In our experiments, use the 17K nouns and adjectives to construct semantic pool.
Part-of-Speede Tagging' is 370K English words corpus, In our experiments, we use the 319K nouns and adjectives to construct semantic pool.
C More Results and Analysis
C.1 Validation of consistency between methodology and theoretical framework
Consistent with our established theory, expanding label candidates with the CSP satisfies the require- ments derived in Section 3.1: (1) Concurrently enlarging the semantic pool size and the expected activation probability 42 of OOD labels; (2) Ensuring that there is no severe mutual dependence among the activations of selected OOD labels.
"https://wwi
spechhsesshhsteeeeeeeeeeeeeeeeee tagging
19

---

(I) The enlargement of the semantic pool size M is evident. Besides, since the superclasses used in constructing the CSP typically include broad semantic objects, the property clusters encompass samples from numerous potential OOD categories. Therefore, their centers have much higher expected probabilities of being activated by OOD samples, which brings an increase in 92- In Table 5, we present the expected softmax scores for a single OOD label from both the original semantic pool and the CSP. These scores, averaged across OOD samples, serve as an approximation of q2. which is defined as the expected probability of OOD labels being activated by OOD samples. Table 5 reveals that the average score of our CSP across four OOD datasets is distinctly higher than that of the original pool, indicating that this expansion leads to an increase in 92-
Table 5: The expected Softmax scores of a single OOD label, an approximation of q2, from the original semantic pool and the conjugated semantic pool, scaled up by factor of 1000.
The expected Softmax score of single OOD label
Semantic Pools
iNaturalist
SUN
Places
Textures
Average
Original / Conjugated
0.1356 / /0.0308
0.0923 0.2176 0.0864 / 0.2213 0.0404 / 0.4435 0.0887 / 0.2283
The effectiveness of the CSP is based on the implicit assumption that OOD samples exhibit various visual properties. However, the degree of visual diversity varies across different OOD datasets, resulting in different expected probabilities of OOD labels in the CSP being activated, as reflected in the varying scores of conjugated labels shown in Table 5. For instance, plant images in iNaturalist have limited visual diversity, leading to low scores for conjugated labels, whereas the Texture dataset, with its higher visual diversity, exhibits the opposite phenomenon. We can observe that across different OOD datasets, there is correlation between these scores and the performance improvements achieved by our method: the score is lower on iNaturalist compared to the original pool, relatively higher SUN and Places, and significantly higher Textures. Consequently, our method achieves only modest gains on iNaturalist, normal improvements on SUN and Places, and substantial enhancements on Textures. This correlation further corroborates the validity of our theory.
(2) Since the labels in CSP are centers of property clusters, while the labels in the original semantic pool are centers of category clusters, it is highly improbable that numerous synonym pairs would exist between these two semantic pools. Our statistical analysis supports this claim: we calculate the average maximum similarity between each label and other labels within the semantic pool, metric which reflects the proportion synonymous pairs within the pool and tends to increase monotonically as the semantic pool expands. Our findings indicate that only 3.94% of the original labels find more similar counterparts in the expanded CSP, resulting in a negligible increase in the aforementioned metric from 0.8721 to 0.8726. As a result, the mutual dependence between the new and original labels is relatively low, and the functions of labels from the CSP will not be overshadowed, enhancing the likelihood that an OOD image locates close an OOD cluster center.
C.2 Random analysis
Table 6 shows the results of our method under random seeds from 0 9, whose average is reported in the main text. It is evident that the performance of our method is minimally impacted by randomness, consistently exhibiting superior efficacy.
C.3 Various ID datasets
As depicted in Table 7, our method is evaluated against baseline methods, including MCM [40] and NegLabel [29], across various ID datasets. These datasets encompass (1) specialized domain-
focused datasets such as Stanford-Cars [31], CUB-200 [61], Oxford-Pet [45], and Food-101 [2]; (2) subsets of ImageNet, including ImageNet10, ImageNet20, and ImageNet100; (3) ImageNet domain shift datasets, namely ImageNet-Sketae [62], ImageNet-A [22], ImageNet-R [20], and ImageNerV2 [48]. Our proposed method consistently achieves satisfactory results on all the above ID datasets. For example, our method outperforms NegLabel by 9.23% and 11. 1.71 % evaluated by FPR95 with ImageNet-A and ImageNet-R as the ID data, respectively.
20

---

Table Mean and standard deviation of OOD detection performance across various random seeds with CLIP-B/ B/16 on ImageNet- 1k as ID data. Performance metrics are presented as percentages.
0OD Datasets
Sceds
iNaturalist
SUN
Places
Textures
Average
AUROC1
FPR95)
AUROC1
FPR951
AUROC
FPR95]
AUROC1
FPR95J
AUROCT
FPR95
99.61
1.54
96.69
13.82
92.85
29.69
93.78
25.78
95.73
17.71
99.61
1.51
96.67
13.57
92.99
29.03
93.69
25.57
95.74
17.42
99.60
13.61
92.95
29.18
93.74
26.10
95.74
17.61
99.60
1.56
96,69
13.57
92.93
29.21
93.64
26.01
95.72
17.59
99.50
1.54
96.65
13.65
92.87
29:26
94.10
24.95
95.81
17.35
99.60
1.55
96.66
13.61
92.88
29.37
94.01
24.79
95.79
17.33
99 60
1.33
96.70
13.53
92.92
29.27
93.87
25.96
95.77
17.57
13.78
29.50
25.74
17.64
99.60
1.53
96.64
13.90
92.89
29.52
94.09
24.75
95.81
17.43
99.60
1,56
96.64
13.58
92.88
29.20
93.80
25.59
95.73
17.48
29.32
Std
0.00
0.02
0.03
0.12
0.04
0.19
0.15
0.48
0.03
0.12
Table 7: OOD detection performance comparison various IDD datasets.
ODD Datasess
datasets
Methods
INaturalist
SUN
Places
Textures
Average
AUROCT
FPR95
AUROCT
FPR95]
AUROC:
FPR95]
AUROC1
FPR95]
AUROC1
FPR95L
MCM
99.77
0.05
99.95
0.02
99.59 89
0.24
99.96
0.02
99.89
0.08
Stanford-Cars
LagLabel
99.99
Ours
100.00
100.00
100.00
100.00
MCM
98.24
988
99.10
4.9%
98.57
6.45
98.75
6.97
98.67
7.10
CUB-200
NegLabel
99.96
0.18
99.99
0.02
99.90
0.3
0.01
99.96
0.14
Ours
99.96
99.99
88
100.00
0,00
99.96
MCM
99.38
2.85
99.73
1.06
99.56
2.11
99.81
0.80
99.62
1.71
Oxford-Per
Negl Label
0.02
99.96
0.17
99.97
99.98
0.08
100.00
MCM
99.75
0.64
99.75
0.90
5%
1.86
98.62
4.04
99:43
1.86
Food-101
NegLabel
99.99
0.01
99.99
0.01
99.99
0.01
99.60
1.61
99.89
0.41
Ours
MCM
99.80
0.12
99.79
0.29
99.62
0.88
90.90
0.04
99.78
0.33
ImageNet10
Negl Labla
99.83
0.02
99.88
0.20
99.75
0.71
99.94
0,01
99.85
0..4
Ours
99,85
0.20
MCM
99.66
1.02
99.50
2.55
99.11
4.40
99.03
2.43
99.33
2,60
ImageNer20
NegLabel
99.95
0.15
99.51
1.93
98.97
4,40
99.11
2.41
99.39
2.22
Ours
MCM
96.77
18.13
94.54
36.45
94.36
34.52
92.25
41,22
94.45
32.58
ImageNet100
NegLabel
99.87
0.57
97.89
11.26
96.25
19.15
96.00
20.37
97.50
12.8/1
Ours
99.90
0.46
98.78
4.84
97.19
13.31
98.16
8.83
98.51
6.86
MCM
87.74
63.06
85.35
67.24
81.19
70.64
74.77
79.59
82.26
70.13
ImageNet Skotch
NegLabel
99.34
2.24
94.93
22.75
90.78
38.62
$9.29
46.10
93.59
27.42
Ours
99.49
1.60
96.41
15.30
92.51
31.41
92.95
29,86
95.34
19.54
MCM
79.50
76.85
76.19
79.78
70.95
80.51
61.9%
86.37
72.16
80.88
ImugeNet-A
NegLabel
98. 80
4.09
89.83
44. 38
82. 8%
60.10
80.25
64.34
87.94
43.23
Ours
99.15
2.91
91.06
42.70
85.16
59.87
93.08
34.50
92.11
34,00
MCM
75.53
66
76.68
76.72
ImageNes-R
Negl.abel
99.58
1,60
96.03
15.77
91.97
29.48
90.50
35.67
94.55
20.63
Ours
99.79
0.89
98..99
6.16
95.41
18.46
96.44
10.16
97.53
8.92
ImngeNetV2
NegLabel
99.10
2.17
94.46
25.69
90.00
42.03
88:46
48.90
93.08
29.77
Ours
99.54
1.76
96.10
17.16
91.66
34.12
92.76
29.65
95.02
20.67
C.4 Empirical evidence supporting our assertions
Performance trends related to the ratio In Fig, I and Table 8, we present the FPR95 performances of our method and NegLabel against a progressively increasing ratio r, which represents the proportion of selected OOD labels in the whole semantic pool. The color gradations displayed in the table clearly illustrate an initial improvement model performance followed by a subsequent decline as the ratio increases. This trend aligns with derivation in Section 3.1.
Effect simple lexicon expansion. Fig. 2 and Table we assess whether adopting larger lexicons enhances performances. Our findings indicate that it does not always hold. When the semantic pool covers the vast majority of common words, further expansion will introduce an excessive number of uncommon words and (near- seeao-ysnnommms, thus failing to meet the derived requirements for theoretical
21

---

performance enhancement. The inefficacy of simple lexicon expansion indicates that viable expansion manners move beyond merely selecting words from existing lexicons.
Table 8: OOD detection performance evaluated by the FPR95 metric with different candidate selection ratios r. The results of our method and the baseline method NegLabel share similar trends.
Performance Ours
Performance NegLabel
Ratio
INaturalist
SUN
Places
Textures
Average
iNaturalist
SUN
Places
Textures
Avernge
0.02
1.28
23.50
37.34
26.10
22:05
1,31
33.66
46.26
50,32
32.89
27.98
0.10
1.28
15,44
30.78
25.41
18:25
1.54
22.20
36.80
43.40
25.99
0.15
1.54
13.82
29.69
25.78
17:71
1.95
20.84
36,00
43.40
25.55
0.30
1.82
25.43
2.46
20.70
36.29
43.49
25.74
2.17
29.70
0.30
2.39
13.70
29.67
25.94
17.93
3.30
22.04
57.71
45.00
27.01
0.40
2.96
14.27
30.34
27.32
18.72
4,22
22.48
38,91
46.95
28.14
0.50
3.58
30.96
28.69
23.16
39.36
43.44
29.09
0.60
4.18
31.33
5.79
0.80
5.19
16.28
31.77
32.70
21 29
7.67
25.75
41.72
$3.85
32:25
1.00
6.27
17.41
37,60
35.00
22.82
9.26
27.25
42.85
$6.29
33.91
Table Evaluation with different corpus sources. "Size" refers to the size of semantic pools.
Source
Size
Method
iNaturalist
SUN
OOD Datneets
Places
Textures
Average
AURO::
FPR35
AUROC
AUROC
FPR35
FPR
Commom--20K
17K
NegLahei Ours
90.50
47.94
19.13
92.35
87.70
51.82
92.06
36.56
WoedNer-v20
T0K
NegLabal
Ours
99.49
99.61
1.91
1.54
95.49
96.69
20.53
13.82
91.64
92.85
29.69
35.59
93.78
90.22
43.56
25.78
94.21
95.73
25,40
17.71
Wo.dNer-y3.0
131K
Noglabbl
Ours
99.65
99.44
2.18
1,37
96.43
94.73
25 12
15.73
92.25
90.50
41.85
31.62
89.46
93.69
26.26
47.59
93.53
05.51
29.19
18.75
Speech
319
24.42
C.5 Analysis of superclasses in conjugated semantic pool
The indices through 14 in Table 10 represent the following superclasses, listed in alphabetical order: area, creature, environment, item, landscape, object, pattern, place, scene, space, structure, thing, view, vista. Table 10 displays the outcomes of multiple runs with 4, 7, and 10 randomly selected superclasses. Although the selection of different superclasses results in some performance fluctuations, any selection significantly enhances performance compared to not employing CSP (as shown in the first row of the table), and achieves state-of-the-an results.
Acute readers may be concerned about performance fluctuations caused by different superclass sets. However, despite the specific OOD categories being unknown in real-world applications, is likely that an approximate range of OOD superclasses can be estimated in advance based on the deployment scenario and empirical evidence. Generally, users can preset a suitable superclass set to achieve satisfactory performance enhancements provided by the conjugated semantic pool.
D
Visualization
In this section, we present visualization results of images picked from the ImageNet- 1 OOD detection benchmark. Each subfigure includes the original image, the ground-trutt label (for ID images only), the image name the dataset, and the top-5 softmax scores for ID labels (orange), OOD labels from the original semantic pool (green), and OOD labels from the conjugated semantic pool (blue).
D.1 In-distribution examples
In Figs. 4 and 5, we present ID examples that have been correctly classified into the ground-truth ID class with high and low confidence, respectively. Fig. 6 presents examples that have been correctly classified into the ID class group but assigned the wrong specific classes. Figs. 7 and 8 display failure cases where the ID image is misclassified into labels of the original semantic pool or our conjugated semantic pool, respectively.
22

---

Table 10: Analysis of the number of the superclasses constructing the conjugated Semantic Pool.
Superciasies of Conjegrned Semanik Pool
INatraaist
SUN
OGD Dhnaccis
Places
Textures
Aveniee
10
AUBRCC
FPR95
AUDOC
FE9255
AUROC
F1905
AUTOCC
FI105
ALBRCC
1.48
13.25
27.92
$9.87
94.81
1:56
06.64
13.88
02.95
13.91
511.77
05.04)
99.62
1.51
96.84
12.57
03.05
28.43
18.91
45.00
94.56
21,99
99.97
1.66
06-20
15.44
97.16
11.00
05.17
30.59
05.64
17:41
09.61
1,48
97:13
12:22
03.52
27.07
50.80
43.53
95.27
21.08
Ancruge wing suprrclascss
91.29
96.65
03.25
50.57
02.71
30.09
51.12
93.00
99.59
1,59
96,
15:45
02.44
31.84
94.67
22.66
95.75
17.89
99.61
1:92
06
13:47
91.00
75.51
01.70
480.74
95.17
21.06
Astrage ahowe uoin? superilssoss
99.01
1.55
96.53
14.31
97.76
30.00
92.65
33.91
95.34
19.96
54.63
95.57
1:35
13.79
42.45
754.89
59.22
95.02
17.34
99.51
1.51
96.81
13.15
93.34
28.43
90.55
43.17
95.03
21.56
Average of shore 5 runs ssng 10 sepeiclasses
99.61
1.58
90.61
13.82
92.83
29.05
92.77
31,77
95.46
19.20
99.61
1.54
96.09
13.82
92.55
20.69
93.77
25.78
95.73
17.71
We observe that, for ID samples. incorrect OOD detection results mainly stem from the following reasons: (1) Low image clarity or VLM limitations: Due to the low clarity of the images or the limited capabilities of the VLM, the VLM provides incorrect classification results. For instance, the category of image ImageNer MaageNNr_JI.SVRR220 _tI.ISSSVRR222222 2 00002334 in Fig. 7 is "coho (silver salmon)", but CLIP mistakenly classifies it as "chum salmon" ", thereby identifying it an OOD) sample. (2) Inaccurate ground truth labels: Some images have ground truth labels that are not precise enough. For example, the image ImageNet ILSVRC2012 val 00004471 in Fig. 8 is labeled not commonly used to refer to spiral staircase. This leads the model to classify the image as "helter-skelter structure", a more accurate OOD category. (3) Multiple elements in images: Certain images contain multiple elements that correspond to several appropriate labels. Although we selected OOD labels with low similarity the ID label space, it does not ensure that OOD labels are entirely unrelated to the specific ID images. For instance, the image ImageNet ILSVRC2012 val 00003031 Fig. is labeled "doormat". but "cursive view" and "frosty view" are also suitable descriptions. resulting in incorrect OOD detection.
"coil spiral", which is
Figure 4: ID Examples of correct OOD detection, correct classification, and high confidence.
D.2 Out-of-distributtio examples
Figs. 9,11,13, and 15 show OOD images from the iNaturalist, Places, SUN, and Textures datasets, respectively, that have been correctly classified as OOD samples. Contrarily, Figs. 10, 14, and 16 display the failure cases, where OOD images are misclassified as ID ones.
23

---

lume MACIRCOUARNH
211
Figure 5: ID Examples of correct OOD detection, correct classification, and low confidence.
Cord
co
Figure 6: ID Examples of correct OOD detection and incorrect classification.
We observe that there are primary reasons for OOD misdetection results in OOD datasets: (1) Absence of suitable OOD labels: In the process of selecting potential OOD labels, elements in the semantic pool that have high similarity to IDD labels are discarded. This may lead to the absence of corresponding labels for OOD images. For example, in Fig. 16, the image named waffled _103 depicts waffle. However, the OOD label candidates do not include the label "waffle" resulting in the image being incorrectly classified as the ID category "waffle iron". (2) Presence of ID category objects within some OOD images: For instance, in Fig. 12, the image Places dataset, whose label is "ski slope". depicts man skiing on ski slope. Actually, classifying it as the ID category "ski" is entirely correct, and this image should not be considered an OOD sample. Similarly, in Fig. 16, the label of the image striped 0032 from the Texture dataset, which shows part of a zebra, "striped", but it is also reasonable that CLIP directly classifies as the category "zebra". ". Thus, although the ImageNet- k OOD detection benchmark established by [26] has been widely used, constructing more accurate and comprehensive OOD datasets remains crucial for further advancements in this field. We will pursue this as future research direction.
00004460 from the
24

---

Figure 7: ID Examples of incorrect OOD detection classified into the original semantic pool.
LUPCCLL 2080111
Wetcoms
LANOH22 200211
Figure 8: ID Examples of incorrect OOD detection classified into the conjugated semantic pool.
Figure 9: OOD Examples of correct OOD detection from iNaturalist.
25

---

Srudd
Figure 10: OOD Examples of incorrect OOD detection from NNtturalist.
Figure 11: OOD Examples correct OOD detection from Places.
Figure 12: OOD Examples of incorrect OOD detection from Places.
26

---

Figure 13: OOD Examples of correct OOD detection from SUN.
Figure 14: OOD Examples of incorrect OOD detection from SUN.
10
Figure 15: 0OD Examples of correct OOD detection from Textures.
27

---

Figure 16: OOD Examples of incorrect OOD detection from Textures.
28

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction clearly state the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The limitations are discussed in Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]

---

Justification:
The full set of assumptions and proofs is provided in Appendix A.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We have provided necessary information for reproduction in Section 5.1.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
30

---

Answer: [Yes]
Justification: Our source codes are available in https:///githi: http::///ggthuub...o/eeeeggggeee www./enngyunnhhen22 Neur IPs2024--G
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run reproduce the results. See the NeurlPS code and data submission guidelines (https: /nips. ccips.cc/public//uu guwdee/dss///// CCodeSubeesiaceecc ssisnsolssiiiiiosso
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Our method does not require training. All the test details necessary to under- stand the results have been provided in Section 5.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We provide the standard deviations, which are commonly used as error bars, for the main experiment results under different random seeds Appendix C.2.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
31

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The computational cost is analyzed in Section 5.1.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: ptttps://neurs http///nurripp.peeee cc/ppbbicc/thiccccueeeeee
Answer: [Yes]
Justification:
The research conforms with the NeurlPS Code of Ethics in every respect.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA].
Justification: Given the focus on zero-shot OOD detection, paper prioritizes discussing technical methodologies and results rather than societal impacts.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
32

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper exploring zero-shot OOD detection poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We have properly credited the used assets. All the datasets and lexicons, introduced in Appendix B, are available for free to researchers for non-commercial use.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
33

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: New assets introduced in the paper are well documented and the documentation is provided alongside the assets.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/moodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA].
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA].
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
34


Reviews:
[{"review_id": "0OBx70SG0l", "reviewer": "Reviewer_pEkd", "summary": "The paper proposes a method for zero-shot out-of-distribution (OOD) detection using an expanded semantic pool of modified superclass names. By leveraging a pre-trained vision-language model, the approach aims to improve OOD classification performance by ensuring low mutual dependence among selected OOD labels. This method outperforms existing techniques by 7.89% in FPR95, highlighting its effectiveness in handling OOD detection tasks.", "questions": "see weakness", "limitations": "YES", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The theoretical analysis of expanding NegLabel using the u-function is very clear and intuitive.\n2. The experiments are conducted very comprehensively.", "weaknesses": "1. The method proposed in this paper lacks innovation; its main framework and basic performance are entirely derived from NegLabel.\n2.The preliminary section lacks clear references to the original theory of NegLabel and the proofs in the appendix.\n3.The description of the proposed method is very concise, which makes it unclear for readers to understand the details of the method design. It is recommended to provide pseudocode for the algorithm to clarify the method's design.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We express our sincere gratitude for your constructive comments. Below, we first reiterate your comments, subsequently providing our detailed responses to each point.\n\n> 1. The method proposed in this paper lacks innovation; its main framework and basic performance are entirely derived from NegLabel.\n\nThank you for your comments! While we utilized the existing SOTA method, NegLabel, as our primary framework, we believe that our work provides meaningful theoretical and performance advancements, showcasing unique innovation. We sincerely hope the following explanation addresses your concerns, and we welcome any further discussion on this matter.\n\nRegarding the methodological framework, we have discussed the theoretical shortcomings in NegLabel's derivation (L1030-1042) : NegLabel undertakes a rudimentary theoretical analysis of the correlation between OOD detection performance and the quantity of adopted potential labels, concluding that an increase in selected labels correlates with enhanced performance. However, this conclusion contradicts the observed actual trend. The contradiction arises from that NegLabel simply assumes a constant higher similarity between OOD labels and OOD images compared to ID images, **neglecting that this similarity discrepancy originates from the strategy of reverse-order selection of OOD labels based on their similarity to the ID label space**. As the set of selected OOD labels transitions from \"*a small subset of labels with the lowest similarity to the entire ID label space*\" to \"*the whole semantic pool, which is unrelated to the setting of ID and OOD labels*\", the discrepancy in similarity of ID images to OOD labels versus OOD images to OOD labels will progressively diminish until it disappears. **Reviewer p3ao also comments that** \"the supplement to the shortcomings of the NegLabel theory is very enlightening\".\n\nBuilding on this insight, we incorporated this dynamic process into our analysis and optimize the performance modeling of NegLabel, leading to the derivation of the unique mathematical model presented in Section 3.1. With a series of derivation, this model establishes the conditions for performance improvement through semantic pool expansion: an unequivocal strategy for performance enhancement requires **concurrently increasing the semantic pool size and the expected activation probability of OOD labels** and **ensuring low mutual dependence among the activations of selected OOD labels**.\n\nIn Section 3.2, based on the optimized mathematical model, we analyzed why simple lexicon expansion fails to yield further performance improvements. In Section 3.3, we proposed a novel method for constructing a conjugate semantic pool, expanding the semantic pool in a manner that satisfies the theoretical conditions mentioned above, and achieved satisfactory performance gains. Therefore, **all the derivations and design choices detailed in Methodology are contributions we have made to this framework**.\n\nIn terms of basic performance, while NegLabel, as the current SOTA method, indeed demonstrates strong performance, our approach has achieved considerable improvements over NegLabel on the standard ImageNet-1k OOD detection benchmark, with a **1.55% increase in AUROC and a 7.89% reduction in FPR95**. Furthermore, our method significantly outperforms NegLabel in various scenarios, including hard OOD detection tasks (*Table 2*), different CLIP models (*Table 3*), different ID datasets (*Table 7*), different corpus sources (*Table 9*), and different VLM architectures (*Table 10*). The effectiveness of our approach has been validated through extensive experiments. **The strong performances we achieved stem from our theoretical analysis and the proposed conjugated semantic pool**, rather than simply from the use of NegLabel.\n\n> 2. The preliminary section lacks clear references to the original theory of NegLabel and the proofs in the appendix.\n\nThank you for the valuable comment! We originally structured the paper to clearly differentiate contributions, with NegLabel's contributions discussed in the **Preliminary** section and our own in **Methodology**. We apologize for any ambiguity and will clarify this in the revised version.\n\nIn Preliminary, we have cited NegLabel at **L99 and L104**, to introduce **NegLabel's methodology** (L97-103) and its **theoretical performance modeling** (L104-124), respectively, summarizing the contributions made by NegLabel. We will make the references more explicit in the revised version.\n\nIn Appendix A.1 and A.2, to ensure clarity and proper acknowledgment, we will add the following statement: “**The proof in this part is adapted from the appendix of [24]**”.\n\nAdditionally, we have cited NegLabel [24] a total of **15** times, with its name, \"NegLabel,\" appearing **20** times, throughout the paper. This should make it clear that our method has strong relevance with the NegLabel framework, upon which we have introduced further innovations. If there are any aspects where the contributions of NegLabel have not been clearly articulated, we would greatly appreciate the opportunity for further discussion and are fully supportive of providing any necessary clarifications.\n\n> 3. The description of the proposed method is very concise, which makes it unclear for readers to understand the details of the method design. It is recommended to provide pseudocode for the algorithm to clarify the method's design.\n\nThank you for the constructive comment! Following the suggestion, we have **provided the pseudocode of our algorithm in the rebuttal PDF file** as Algorithm 1, and we will incorporate it, along with a more detailed version of the method design, into the appendix in the revised version. If there are any aspects of the algorithmic process that remain unclear, we would be more than happy to engage in further discussion.\n\n**Finally, we extend our sincerest thanks for all your time and consideration!**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FqGqIAnnmt", "reviewer": "Reviewer_QWh5", "summary": "This paper presents improvements to zero-shot OOD detection methods based on pre-trained vision-language models. The study first models factors that influence the performance of existing pipelines and theoretically derive two necessary conditions for enhancing performance: expanding the OOD label candidate pool and maintaining low interdependence. Subsequently, the author analyzes why simple expansion methods do not meet these conditions and proposes constructing a conjugated semantic pool for expansion. This method meets the theoretical conditions and achieves SOTA performance on several public benchmarks.", "questions": "NA", "limitations": "Please see the weaknesses.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The article presents reliable mathematical modeling and theoretical derivations, and the proposed method is simple and efficient.\n\n2. The proposed method achieves considerable performance improvements over the latest methods.\n\n3. The article features a clear structure and logical coherence, demonstrating a high standard of presentation and writing quality.", "weaknesses": "1. The description in the caption of Table 1 is unclear. In the experimental section, the author mentioned that the upper part of Table 1 represents traditional OOD detection methods, while the lower part pertains to detection methods based on pre-trained models like CLIP. However, this distinction is not explicitly marked in the caption, which might lead readers to misunderstand that all methods are based on the CLIP framework.\n\n2. There is a possibility that in some cases, the chosen OOD labels from CSP may also have high similarity to ID images. The authors should provide more explanation on this point.\n\n3. I do not quite understand the significance of the experiments in Table 5. I suggest the authors provide a more detailed discussion.\n\n4. In Table 9, the meaning of \"Size\" of the corpus is not clear. Does it refer to the total number of words in the dictionary or the number of selected OOD labels?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We express our sincere gratitude for your valuable suggestions and the positive rating. Below, we first reiterate your comments, subsequently providing our detailed responses to each point.\n\n> 1.  The description in the caption of Table 1 is unclear. In the experimental section, the author mentioned that the upper part of Table 1 represents traditional OOD detection methods, while the lower part pertains to detection methods based on pre-trained models like CLIP. However, this distinction is not explicitly marked in the caption, which might lead readers to misunderstand that all methods are based on the CLIP framework.\n\nThank you for the comment! We will clarify the distinction between the methods in the upper and lower parts of Table 1 in the caption to avoid any potential misunderstandings.\n    \n> 2.  There is a possibility that in some cases, the chosen OOD labels from CSP may also have high similarity to ID images. The authors should provide more explanation on this point.\n\nThank you for the insightful comment! As with the NegLabel method we followed, we used a negative label mining algorithm to minimize the similarity between selected negative labels and ID images. Specifically, we calculated the similarity between each OOD label in the CSP and the ID label space, selecting the least similar ones as the negative labels for actual use. We have provided the pseudocode for this process in the rebuttal PDF file. Although we cannot completely avoid instances where certain OOD labels may exhibit high similarity with ID images, the strategy we employed effectively decreases the likelihood of such occurrences.\n    \n> 3.  I do not quite understand the significance of the experiments in Table 5. I suggest the authors provide a more detailed discussion.\n\nThank you for your suggestion! The experiments in Table 5 primarily aim to demonstrate that, consistent with our established theory, expanding label candidates with the CSP satisfies the requirement derived in Section 3.1: concurrently enlarging the semantic pool size M and the expected activation probability q_2 of OOD labels. Specifically:\n\nSince the superclasses used in constructing the CSP typically include broad semantic objects, the property clusters encompass samples from numerous potential OOD categories. Therefore, their centers have much higher expected probabilities of being activated by OOD samples, which brings an increase in q_2. In Table 5, we present the expected softmax scores for a single OOD label from both the original semantic pool and the CSP. These scores, averaged across OOD samples, serve as an approximation of q_2, which is defined as the expected probability of OOD labels being activated by OOD samples. Table 5 reveals that the average score of our CSP across four OOD datasets is distinctly higher than that of the original pool, indicating that this expansion leads to an increase in q_2.\n    \n> 4.  In Table 9, the meaning of \"Size\" of the corpus is not clear. Does it refer to the total number of words in the dictionary or the number of selected OOD labels?\n\nThank you for your question! Since we used nouns from the lexicon to construct the original semantic pool and adjectives to build the conjugated semantic pool, the \"Size\" in Table 9 refers to the total number of nouns and adjectives in the lexicon, not the number of selected OOD labels.\n\n**In the end, we express our sincerest gratitude for your time and consideration!**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "I0Q5ZPKekz", "reviewer": "Reviewer_gvS3", "summary": "This paper explores how to set potential OOD labels to facilitate OOD detection with vision-language models. The paper first conducts a theoretical analysis, revealing that in addition to increasing the negative label space, it is also important to increase the probability of negative labels being activated and to reduce mutual dependence between negative labels. Therefore, the paper proposes a new strategy for constructing negative labels by introducing modified superclass names to construct a conjugated semantic pool (CSP). Experiments are conducted on standard benchmarks.", "questions": "See weaknesses", "limitations": "NA", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1. The paper theoretically analyzes the criteria for selecting negative labels and proposes a new method for designing negative labels.\n2. The motivation is clear, and the method design is straightforward and effective.\n3. The proposed method achieves state-of-the-art results on standard benchmarks.", "weaknesses": "While the overall method design is agreeable, there are some details that need clarification:\n\n1. In Line 276, the authors claim that CSPs overlapping with ID classes will not be selected as negative labels. How is this implemented, and does it significantly impact the results?\n2. In Line 321, the authors introduce the superclasses used and mention combining these superclasses with adjectives to form the CSP. Could you provide more details on the adjectives used and the selection process? Is the pairing with superclasses done randomly?\n3. How many negative labels are ultimately used? Beyond the 10,000 classes in the negative label pool, how many new negative labels are introduced? Does the number of newly introduced negative labels significantly affect OOD detection performance?\n4. In my experiments, on ImageNet, performance initially improves and then declines as the number of negative labels increases; however, in experiments with CIFAR as the ID dataset, performance continuously improves with more negative labels. Have the authors observed similar phenomena, and can they provide an explanation?\n5. It is suggested to follow Neglabel and refer to the introduced classes as \"negative labels\" instead of \"OOD labels\" to avoid confusion with the labels of practical OOD datasets.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We express our sincere gratitude for your valuable comments. Below, we first reiterate your comments, subsequently providing our detailed responses to each point.\n\n> 1.  In Line 276, the authors claim that CSPs overlapping with ID classes will not be selected as negative labels. How is this implemented, and does it significantly impact the results?\n\nThank you for the insightful comment! In the revised version, we will clarify the statement in L276 to avoid any ambiguity.\n\nApart from the negative label selection strategy based on inverse similarity provided by NegLabel, **we do NOT have additional implementation to ensure that CSP labels with semantic overlap with ID classes are not selected**. What we intended to convey in L276 is that the labels in the CSP are also selected as negative labels **based on inverse similarity**, thereby reducing the likelihood of CSPs with semantic overlap with ID classes being chosen as negative labels. There are no specific strategies in our method that can completely prevent semantic overlap between CSP and ID classes.\n\n> 2.  In Line 321, the authors introduce the superclasses used and mention combining these superclasses with adjectives to form the CSP. Could you provide more details on the adjectives used and the selection process? Is the pairing with superclasses done randomly?\n\nThank you for your suggestion!\n\nWhen constructing the conjugated semantic pool, the adjectives we used were unfiltered adjectives from the lexicon adopted (\"adj.all.txt\" in WordNet). For selecting negative labels, we employed the NegMining algorithm provided by NegLabel. The pseudocode is provided in the rebuttal PDF file.\n\nThe pairing of superclasses and adjectives was entirely random; that is, for each adjective, we randomly selected a superclass from the set introduced in L321 to form a phrase.\n\n> 3.  How many negative labels are ultimately used? Beyond the 10,000 classes in the negative label pool, how many new negative labels are introduced? Does the number of newly introduced negative labels significantly affect OOD detection performance?\n\nThank you for your question!\n\n**We ultimately used 8,492 negative labels, of which 7,005 were simple noun labels from NegLabel, and 1,487 were from our constructed CSP.** The proportion of negative labels selected from the semantic pool was 15%, consistent with NegLabel. However, the total number of noun labels we obtained was smaller than in NegLabel because we removed word categories such as personal names, organization names, and quantifiers, which generally have little utility in activating OOD images. All the ablation experiments were conducted after removing these categories.\n\nThe number of newly introduced negative labels has a noticeable impact on OOD detection performance. When keeping the selection proportion of noun labels at 15%, and gradually adjusting the proportion of negative labels selected from CSP from 2% (198 labels) to 100% (9,916 labels), the model’s performance (FPR95) increased from **23.22%**, peaked at **16.66%** when the proportion was 40% (3,966 labels), and then began to decline, reaching **18.64%** when the proportion was 100%. **The specific experimental results are presented in Table 1 of the Rebuttal PDF file.** In other words, if a separate selection ratio parameter for CSP was set, a higher performance than reported could be achieved at 40%. However, to avoid parameter tuning, we directly adopted the 15% ratio parameter adopted by NegLabel.\n\n> 4.  In my experiments with CIFAR as the ID dataset, performance continuously improves with more negative labels. Have the authors observed similar phenomena, and can they provide an explanation?\n\nThank you for your comment!\n\nWe had not previously conducted experiments using CIFAR-100 as the ID dataset. To explore this issue, we conducted experiments with CIFAR-100 as the ID dataset and iNaturalist, Places, SUN, and Textures as the OOD datasets. Due to the presence of overlapping categories between CIFAR-100 and the OOD datasets, we manually removed the following categories from CIFAR-100: flowers (orchids, poppies, roses, sunflowers, tulips), large man-made outdoor things (bridge, castle, house, road, skyscraper), large natural outdoor scenes (cloud, forest, mountain, plain, sea), and trees (maple, oak, palm, pine, willow). This means that the ID dataset we used contains only 80 distinct categories. **The experimental results are presented in Table 2 of the Rebuttal PDF file.**\n\nIn brief, we did **NOT** observe a continuous improvement in performance as the number of **total** negative labels increased. The performance trend remains an inverse-V curve, similar to what we observed with ImageNet-1k. However, the peak point of the average performance across the four OOD datasets is indeed **reached at a larger selection ratio** compared to experiments on ImageNet-1k, shifting from about 15% to approximately 50%.\n\nTheoretically, continuous performance improvement seems somewhat abnormal. When the selection ratio increases to 100%, the negative labels actually **cover all words in the lexicon without utilizing any information from the ID categories**, making it improbable to achieve optimal results. If this analysis does not fully address your concerns, we welcome you to provide more details and results of your experiments for further discussion.\n\n\n> 5.  It is suggested to follow Neglabel and refer to the introduced classes as \"negative labels\" instead of \"OOD labels\" to avoid confusion with the labels of practical OOD datasets.\n\nThank you for your valuable suggestion! We agree that referring to the newly introduced classes as \"negative labels\" is indeed a more appropriate approach. We will incorporate this correction in the revised version.\n\n**In the end, we express our sincerest gratitude for your valuable suggestions and positive rating!**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dEFtmC7RSw", "reviewer": "Reviewer_p3ao", "summary": "This paper presents a novel zero-shot out-of-distribution (OOD) detection pipeline that enhances performance by expanding the semantic pool with a conjugated semantic pool (CSP), which consists of modified superclass names that cluster similar properties from different categories. The approach aims to increase the activation probability of selected OOD labels by OOD samples while ensuring low mutual dependence among these labels. By moving beyond traditional lexicon-based expansion and using the CSP, the paper's method outperforms existing works by 7.89% in FPR95, demonstrating a significant improvement in OOD detection without the need for additional training data.", "questions": "see Cons", "limitations": "Yes", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well-written and easy to understand.\n2. The supplement to the shortcomings of the NegLabel theory is very enlightening.\n3. Extensive experiments demonstrate the superiority of the method.", "weaknesses": "1. The primary contribution of this work is its enhancement of the semantic pool expansion issue within NegLabel, representing an incremental step within the NegLabel framework.\n\n2. There is ambiguity regarding the distinction between citations and contributions. For instance, the content related to Lemma 1 in both the preliminary section and the appendix A.1, A.2 seems to originate directly from the original NegLabel paper, yet this source is not explicitly referenced in the main text and appendix.\n\n3. In Section 3.3, the authors think that OOD images may not include their true label \"white peacock\" among the negative labels is a issue. However, this argument appears debatable. As evidenced in NegLabel's visualizations and Figures 6 and 7 of this paper, OOD images may not consistently exhibit the highest similarity to their ground truth (GT) label and could show significant similarity with multiple negative labels. Thus, even if the true GT label of an OOD image is absent from the negative labels, the NegLabel method can still effectively detect OOD data.\n\n4. From an implementation standpoint, how does including terms like \"smartphone\" and \"cellphone\" as negative labels impact the overall OOD score? The claim in Section 3.2 about the impact of synonym disruption on Lemma 1's independence seems somewhat overstated.\n\n5. I am curious whether, aside from the semantic pool, all other aspects of the technical implementation directly derive from NegLabel. Specifically, does the NegMining Algorithm, the OOD score calculation, and the Grouping Strategy align exactly with those in NegLabel? This aspect requires further clarification.\n\n6. Additionally, is Figure 3 a representation of actual distribution distance data or a schematic created by the authors? Utilizing real experimental data to support the findings presented in Figure 3 would strengthen its validity. If it is a schematic, it may introduce subjective biases from the authors.\n\n7. The paper extensively discusses the limitations of previous methods and the intended outcomes in Section 3, yet provides relatively minimal detail on the actual implementation of the proposed method. It would be beneficial for the authors to expand on the specifics of their proposed approach in Section 3.3. For instance, further elaboration on how superclass selection is conducted and integrated with adjectives would be valuable.\n\n8. According to the description in the paper, the method used to select cluster centers appears crucial. Have the authors considered employing clustering techniques for superclass selection rather than relying solely on manual selection?\n\n9. Another point of confusion arises regarding how variations in corpus size impact the acquisition of semantic pools. Does this necessitate selecting additional superclasses?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your constructive comments and positive rating! Below, we briefly restate your comments (due to length limitations) and then provide our point-by-point responses.\n\n> 1. Enhancement of semantic pool expansion within NegLabel\n\nThanks for the valuable comment. Our method builds on the NegLabel framework, introducing further innovations. **NegLabel is the most effective paradigm for this task.** In addressing its limitations, we proposed a performance modeling approach that better aligns with experimental results and identified the conditions for effective semantic pool expansion. This allowed us to optimize this step, resulting in satisfactory performance improvements. Therefore, we believe our work **provides meaningful theoretical and performance advancements, showcasing unique innovation**.\n    \n> 2.  Ambiguity between citations and contributions. \n\nWe originally structured the paper to clearly differentiate contributions, with NegLabel's contributions discussed in the Preliminary section and our own in the Methodology. We apologize for any ambiguity and will clarify this in the revised version.\n\nIn Preliminary, we have cited NegLabel at **L99 and L104**, to introduce NegLabel's methodology (L97-103) and its theoretical performance modeling (L104-124). We **will make these references more explicit** and add the statement in Appendix A.1 and A.2: “**The proof in this part is adapted from the appendix of [24]**”.\n\nAdditionally, we have cited NegLabel [24] a total of **16** times, with \"NegLabel\" appearing **20** times, throughout the paper. This should make it clear that our method is based on NegLabel, upon which we have introduced further innovations.\n    \n> 3.  Debatable augument about not including true labels.\n\nThank you for the insightful comment! We agree that NegLabel can effectively detect OOD data when the true label (GT) of the OOD image is excluded from the negative labels. However, this isn’t always guaranteed. If an OOD image’s most similar classes are not OOD classes, the model may fail to detect it as OOD.\n\nFor instance, in a scenario where the ID classes are \"*hognose snake*\", \"**basset hound**\", and \"**Afghan hound**\", and the OOD classes are \"**toy terrier**\", \"*green snake*\", and \"*king snake*\" (all are ImageNet-1k categories), if \"**toy terrier**\" is not included in negative labels, OOD detection for \"toy terrier\" images might fail, even with a strong zero-shot classifier, because **their most similar classes are within the ID set**.\n\nMoreover, even if NegLabel works when the GT is excluded from negative labels, **it doesn't mean including the GT in negative labels is pointless**. Ideally, with a sufficiently powerful VLM, OOD images should exhibit the highest similarity to their GT labels. Including the GT in negative labels **can naturally result in higher OOD scores for OOD images**, improving detection in more challenging scenarios.\n    \n> 4.  Impact of synonym disruption on Lemma 1's independence.\n\nThanks for the valuable comment! Regarding this issue, we suggest considering **an extreme case**: if we duplicate each negative label 10 times, will the model's performance improve simply because the semantic pool is larger? Clearly not, and this contradicts the derivation in NegLabel and our work, which advocates for larger semantic pools. The reason is that this duplication violates the independence assumption in Lemma 1. When there is significant interdependence among Bernoulli variables, actual performance can deviate substantially from the mathematical model. **Adding synonyms has a similar effect to duplicating labels.**\n\nHowever, we do **NOT** consider that using Lemma 1 in NegLabel and our work for performance modeling is flawed, as any mathematical modeling is an approximation of real-world conditions under ideal assumptions.\n    \n> 5.  Question about technical implementations from NegLabel.\n\nYes, we adopted the exact same NegMining algorithm, OOD score calculation, and Grouping Strategy as in NegLabel. In the revised version, we will clarify these technical details more explicitly.\n    \n> 6.  Question about Figure 3.\n\nFig 3 is indeed a schematic diagram created by us to illustrate our motivations. Following your suggestion, we plan to generate visualizations using real experimental data to support the findings presented in Fig 3 in the revised version.\n\n> 7.  More implementation details.\n\nWe will incorporate more specific implementation details in Sec 3.3 and 5.1 of the paper. Specifically, we manually chose terms with broad semantic meanings, such as \"item\", \"place\", \"creature\", as superclasses. Experiments showed that various superclass sets can all achieve satisfactory performance improvements. The combination of adjectives with superclasses is entirely random: for each different adjective, we randomly select a superclass from the set and pair it with the adjective to form a phrase.\n    \n> 8.  Employing clustering techniques for superclass selection.\n\nThanks for the constructive comment. The clustering technique is not employed for superclass selection for its **higher complexity and weaker interpretability**. Clustering results vary with factors like the algorithm, sample size, and number of clusters, adding complexity. In contrast, our method introduces no additional hyperparameters, requires no extra data or training, and enhances interpretability by using explicit superclass terms instead of abstract cluster centers.\n    \n> 9.  How variations in corpus size impact semantic pools.\n\nRegardless of the corpus size, our method for constructing CSP is consistent, as mentioned in response to comment 7, and the pseudocode is provided in the rebuttal PDF file. Therefore, the size of the CSP is corresponds to the number of adjectives in the corpus, but a larger corpus does not necessarily require more superclasses.\n\n**In the end, we express our sincerest gratitude for your time and thoughtful consideration!**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gu8tQGSJQh", "reviewer": "Reviewer_KrTH", "summary": "This paper proposes a novel method utilizing pre-trained vision-language models (VLMs) to enhance out-of-distribution (OOD) detection. Their core idea lies in constructing a Conjugated Semantic Pool (CSP) to enrich the pool from which OOD labels are drawn. Unlike simply expanding lexicons with synonyms and uncommon words, the CSP leverages modified superclass names that serve as cluster centers for samples with similar properties across categories. This, the authors theorize, not only increases pool size but also amplifies the activation probability of OOD labels while maintaining low label dependence. Extensive experiments demonstrate significant performance improvements over existing approaches.", "questions": "See Weakness", "limitations": "See Weakness", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "Originality: The paper introduces a unique concept, conjugated semantic labels, for enhancing the semantic pool, offering a valuable contribution to OOD detection.\nTheoretical Underpinnings: The authors provide a robust theoretical framework, including mathematical models and justifications demonstrating how the CSP improves OOD detection.\nEmpirical Validation: Extensive experiments showcase the effectiveness of the proposed method, achieving significant advancements over current state-of-the-art techniques.\nGeneralizability: The method's applicability across diverse VLM architectures suggests broad utility and adaptability.\nClear Problem Definition: The paper clearly identifies limitations in existing methods, particularly regarding simple lexicon expansion, and proposes a well-reasoned solution.", "weaknesses": "1. Ablation Study & OOD Score Function:\nThe ablation study in Table 11 showcases that competitive performance on iNaturalist and SUN datasets is achievable without the CSP. However, it remains unclear how the proposed OOD score function (designed by the authors) contributes to performance when CSP is not employed. Further clarification is needed on: (1)Performance Source: If strong results are achievable without CSP on these datasets, what factors besides the CSP are driving the overall performance improvement observed in the main experiments? (2) OOD Score Function Usage: How can the authors' OOD score function be effectively utilized in scenarios where CSP is not implemented?\n2. Synonym Handling and CSP Overlap:\nThe paper acknowledges the limitations of including synonyms in larger lexicons. However, it lacks a deeper exploration of how the CSP specifically addresses this issue. Additionally, the potential for semantic overlap within the CSP itself needs to be addressed. It would be beneficial to see a discussion on: (1) CSP vs. Synonyms: How does the design of the CSP inherently mitigate the problems introduced by synonyms in traditional lexicon expansion? (2) Mitigating Overlap: How do the authors handle potential semantic overlap between elements within the CSP? Are there strategies to ensure distinctiveness between labels?\n3. Systematic Analysis on Place and Texture Datasets:\nA more systematic analysis of the CSP's impact on the Place and Texture datasets, particularly concerning the design principles of semantic pooling, would be valuable. The authors' own experiments suggest that designing effective semantic pooling requires considering multiple factors. Therefore, a more in-depth exploration of: (1) Place and Texture Performance: How does the CSP specifically improve performance on Place and Texture datasets? Can these improvements be linked to specific design choices within the CSP? (2) Semantic Pooling Design Factors: Building on the authors' findings, what factors are crucial for designing effective semantic pooling mechanisms, especially in the context of Place and Texture datasets?\n\nOverall, this paper proposes a promising approach to improve OOD detection by innovatively using conjugate semantic labels to expand the semantic pool, which is supported by theoretical analysis and empirical results. However, I still have some questions about the method that the authors need to address.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive comments and the positive rating! In the following, your comments are concisely mentioned (due to the length limitation) and then followed by our point-by-point responses.\n\n> 1.  Ablation Study & OOD Score Function: ...\n\nThank you for the insightful comment! The OOD score function we adopted is proposed by NegLabel (ICLR2024) without modifications. The performance improvement on certain datasets when not using CSP stems from minor adjustments we made to the lexicon used to construct the original semantic pool. Specifically:\n\n1.  **Performance Source:** In the baseline presented in Table 11, we removed all adjectives and proper nouns, including personal names, organization names, and quantifiers, from the semantic pool constructed by NegLabel. This led to a higher expected activation probability of negative labels for OOD images in the iNaturalist and SUN datasets. However, this did not result in an overall performance improvement. Compared to the NegLabel method, the performance gains on the iNaturalist and SUN datasets were offset by performance drops on the Places and Texture datasets, resulting in a decrease in overall model performance (25.40% -> 27.39% on FPR95).\n    \n2.  **OOD Score Function Usage:** Our Conjugated Semantic Pool (CSP) is a method for expanding the semantic pool. Without using CSP, we still have the original semantic pool composed of simple noun labels, which can be used with the OOD score function. We will include additional details in the implementation part of the revised version to ensure clearer explanations.\n\n> 2.  Synonym Handling and CSP Overlap: ...\n\nThank you for the valuable comment! We provided related discussion in Appendix D.1 (L1143-1152). Specifically:\n\n1.  **CSP vs. Synonyms:** When the size of a traditional lexicon reaches a certain point, further expansion inevitably introduces a large number of synonyms, which are unlikely to contribute to performance improvement. However, using CSP to expand the semantic pool effectively mitigates this issue without introducing numerous synonyms, since the labels in CSP are centers of property clusters, while the labels in the original semantic pool are centers of category clusters. As discussed in Appendix D.1, our statistical analysis supports this claim: we calculate the average maximum similarity between each label and other labels within the semantic pool—a metric that reflects the proportion of synonymous pairs within the pool and tends to increase monotonically as the semantic pool expands. Our findings indicate that only 3.94% of the original labels find more similar counterparts in the expanded CSP, resulting in a negligible increase in the aforementioned metric from 0.8721 to 0.8726. Consequently, the mutual dependence between the new and original labels is relatively low.\n    \n3.  **Mitigating Overlap in CSP:** When constructing the conjugated semantic pool, each label has a different adjective and a randomly selected superclass from a set of 14 superclasses. For two labels in the CSP, significant semantic overlap would only occur if both their adjectives and superclass words are synonymous, thus the likelihood of such overlap within the CSP is very low.\n    \n\nOf course, we acknowledge that CSP, as a method for expanding the semantic pool, cannot eliminate synonyms that already exist within the original pool. However, the main limitation of synonyms is that their activations are highly dependent on each other, making it difficult for synonym-based expansion to improve performance in line with the theoretical guidance of mathematical models. The presence of a small number of synonyms should not significantly harm model performance.\n\n> 3.  Systematic Analysis on Place and Texture Datasets: ...\n\nThanks for the valuable comment! We provide a more in-depth exploration of these issues and will add them to the revised version:\n\n1. **Place and Texture Performance:** We provide related discussion in Appendix D.1 (L1132-1142) and provide supporting data in Table 5, which may be helpful to clarify this issue. Specifically, by linking the performance improvements on the Places and Textures datasets (as well as SUN) to the design choices of CSP, we conclude that because the CSP was constructed using a diverse range of adjectives and superclass words with broad semantic meanings, labels in the CSP can be considered as centers of property clusters. Therefore, the effectiveness of the CSP is based on the implicit assumption that OOD samples exhibit various visual properties. The images in SUN, Places, and Texture primarily depict natural environments and textures, which have strong visual property diversity, leading to relatively larger performance improvements. In contrast, iNaturalist, where images are mostly focused on various plants with limited visual property diversity, does not benefit from the inclusion of the CSP. \n\n2.  **Semantic Pooling Design Factors:** Focusing on the following factors of CSP can contribute to performance improvement: (1) When constructing the CSP, avoid reusing adjectives and instead pair each adjective with a randomly selected superclass to minimize semantic overlap between labels. (2) When setting up the superclass set, aim to include a broad semantic range in the set. Generally, more superclasses tend to bring better performance due to increased diversity. (3) As mentioned in our response to Reviewer gvS3’s comment 3, the ratio of negative label selection impacts performance, and selecting negative labels from the CSP achieves the best performance at 40%.\n\n**In the end, thanks again for all your time and consideration in reviewing our paper!**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all the reviewers for their thorough reading of our work and the high-quality feedback they provided. Their comments have been immensely beneficial in enhancing the quality of our manuscript and deepening our own understanding of this field. We have uploaded detailed, point-by-point responses for each reviewer, which we believe address all the concerns raised. Additionally, we have provided the requested algorithm pseudocode and extra experimental results in the uploaded rebuttal PDF file, as referenced in our responses to the corresponding reviewers

We look forward to engaging in further discussion and exchange with the reviewers during the next stage of the review process. Finally, we extend our sincerest thanks to all the reviewers for their time and thoughtful consideration!

Meta-review: This paper proposes a novel method to build a Conjugated Semantic Pool to improve OOD detection by pretrained vision and language model. Concretely, the paper identifies two key factors to build the semantic pool: 1) increasing the activation probability of selected OOD labels, and  2) maintaining low interdependence among the chosen OOD labels. The proposed approach has sound theoretical support and comprehensive experiments to show the significant improvement over prior works e.g. NegLabel.

All reviewers converge on high scores 7, 6, 7, 6 to accept the paper (except one with a score of 4). The reviewers like the clear and intuitive theoretical justification (pEkd, gvS3, KrTH), and analysis of the shortcomings of existing methods (p3ao, KrTH). In addition, the experiments are thorough (pEkd, p3ao, KrTH), and demonstrate significant gains over current SoTA (gvS3, KrTH, QWh5). After considering the author reviews, rebuttal and the paper, I'd recommend acceptance of the paper.
Decision: Accept (poster)