Category-Extensible Out-of-Distribution Detection via Hierarchical Context Descriptions
Authors: Kai Liu
Zhihang Fu
Chao Chen
Sheng Jin
Ze Chen
Mingyuan Tao
Rongxin Jiang
Jieping Ye
Venue: NeurIPS 2023 poster (2023)
Keywords: out-of-distribution detection
vision-language models
category-extendable classification

Abstract: The key to OOD detection has two aspects: generalized feature representation and precise category description. Recently, vision-language models such as CLIP provide significant advances in both two issues, but constructing precise category descriptions is still in its infancy due to the absence of unseen categories. This work introduces two hierarchical contexts, namely perceptual context and spurious context, to carefully describe the precise category boundary through automatic prompt tuning. Specifically, perceptual contexts perceive the inter-category difference (e.g., cats vs apples) for current classification tasks, while spurious contexts further identify spurious (similar but exactly not) OOD samples for every single category (e.g., cats vs panthers, apples vs peaches). The two contexts hierarchically construct the precise description for a certain category, which is, first roughly classifying a sample to the predicted category and then delicately identifying whether it is truly an ID sample or actually OOD. Moreover, the precise descriptions for those categories within the vision-language framework present a novel application: CATegory-EXtensible OOD detection (CATEX). One can efficiently extend the set of recognizable categories by simply merging the hierarchical contexts learned under different sub-task settings. And extensive experiments are conducted to demonstrate CATEX’s effectiveness, robustness, and category-extensibility. For instance, CATEX consistently surpasses the rivals by a large margin with several protocols on the challenging ImageNet-1K dataset. In addition, we offer new insights on how to efficiently scale up the prompt engineering in vision-language models to recognize thousands of object categories, as well as how to incorporate large language models (like GPT-3) to boost zero-shot applications.

Category-Extensibbe Out-of-Distribution Detection via Hierarchical Context Descriptions
Kai Liu Zhihang Fu²t, Chao Chen', Sheng Jin?, Ze Chen', Mingyuan Tao?, Rongxin Jiang', Jieping Ye²
ZZejjiang University, Alibaba Cloud
Abstract
The key to OOD detection has two aspects: generalized feature representation and precise category description. Recently, vision-language models such as CLIP provide significant advances in both two issues, but constructing precise category descriptions is still in its infancy due to the absence of unseen categories. This work introduces two hierarchical contexts, namely perceptual context and spurious context, to carefully describe the precise category boundary through automatic prompt tuning. Specifically, perceptual contexts perceive the inter-category differ- ence (e.g., cats vs apples) for current classification tasks, while spurious contexts further identify spurious (similar but exactly not) OOD samples for every single category (e.g., cats vs panthers, apples vs peaches). The two contexts hierarchically construct the precise description for certain category, which first roughly clas- sifying sample to the predicted category and then delicately identifying whether it is truly ID sample actually OOD. Moreover, the precise descriptions for those categories within the vision-language framework present a novel application: CATegory- eetenile OOD detection (CATEX). One can efficiently extend the set of recognizable categories by simply merging the hierarchical contexts learned under different sub-task settings. And extensive experiments are conducted For
demonstrate CATEX's effectiveness, robustness, and cattegory-extesibiili instance, CATEX consistently surpasses the rivals by large margin with several protocols on the challenging ImageNet- 1K dataset. In addition, we offer new insights on how efficiently scale up the prompt engineering in vision-language models to recognize thousands of object categories, well as how to incorporate large language models (like GPT-3) to boost zero-shot applications.
Introduction
disf-iiittiittion (OOD) detection focuses on determining whether an input image in-distribution (ID) or OOD, while classifying the ID data into respective categories [17]. The key to OOD detection has two aspects: (1) constructing sufficiently generalized feature representation capability, so that images of arbitrarily different categories
be roughly separated from each other regardless of semantic shifts [67] and (2) acquiring precise descriptions (namely decision boundary) for each ID category in the image feature space (in CNNs the last fully-connected layer plays this role),
to
determine whether the input image belongs to the corresponding category or just OOD [22].
Specifically, generalized feature representation comes from large-scale and diverse training data for learning-bant based data-driven models. Therefore, OOD detection works 33 13 50] have made great progress the basis of large-scale pre-trained models such as ResNet [9 and ViT [1]. To achieve
Work done during Kai Liu's research internship at Alibaba Cloud. Email: kail@ www.:jj..uu.c.n... Corresponding authors. Email: rongxinj zju.edu.en, zhihang.fzht fzh www.f/iiaaaaaaaaaaaa alibabb-inn.com..
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

83
8
8
83
83
corted deariptions
crntect dosrigtione
4
4
4
4
D
conedyyy wlwis wlw.sffeeddDaannna....
83
13
8
4
4
83
8
4
classiiei sanplei
cxcreilly Ode.r samples
Previous approaches
Our method
Figure I: Method comparison. Compared to previous approaches, our method utilizes the perceptual
context to classify different categories under the current ID task (solid lines), and leverages the spurious context to strictly define the category boundaries independent of the current setting (dashed lines). The hierarchical perceptual and spurious contexts jointly describe the precise and universal boundaries for each category (combination of solid and dashed lines).
the precise description for certain ID category, on the other hand, we need let the model acquire as much "prior knowledge" as possible. For instance, if the description comes from a binary classifier that has only seen cal and apple during training, the description of category cat is obviously not precise enough to distinguish panther image as an OOD: it lacks prior knowledge to distinguish different quadruped mammals.
Recently, vision-language models such as CLIP provided significant advances in the two key issues above for OOD detection. Huge corpora of paired image-text data training brings both powerful feature representation capabilities and more comprehensive prior knowledge than single-vision-me ingle-vision-modal training (7] Therefore, CLIP-based OOD detection methods have been proposed successively, such as the zero-shot method MCM (38) and the encoder fine-tuning method NPOS (50). However, there still exist the issues of generalization of feature representations and precision of category descriptions under the multi-modal framework.
Our observations can be summarized into two aspects: 1. based OOD methods [38] is limited. As CLIP is trained by contrastive Icarning with informative image-text caption pairs, simply using the category names all text information constrains the potential image-text discriminability. 2. Even though fine-tuning CLIP's encoder may boost the the generalization of multi-moda feature representation is sacrificed. In other words, such methods impair the model's 's ability to resist data shifts. Although the existing works are promising and inspiring academia, constructing precise category descriptions via CLIP-like multi-modal features for OOD detection is still in its infancy [8].
The performance of the zero-shot CLIP-
performance 118 50,
Based on the observations above, we take step towards generalized OOD detection by seeking the precise description prompt for each category and meanwhile maintaining the representation capacity. To this end, we develop pair of perceptual context and spurious context for each in-distribution category to hierarchically construct the precise category description. CLIP encoders are kept frozen, and the contexts are learned through automatic prompt tuning.
As illustrated in Fig. D perceptual contexts first roughly classify a sample to the predicted category (colored regions), and spurious contexts then delicately identify whether it is truly an ID sample or actually spurious OOD (gray regions). Specifically, perceptual contexts perceive the descriptions of inter-category differences across all ID categories, while spurious contexts are relatively independent of the specific ID data task. The latter models a more strictly defined description of the current category itself by training both real ID data and well-designed adversarial samples. We also introduce a robust sample synthesis strategy that leverages the prior knowledge of the VLM to select the qualified syntheses inspired by [50 60).
We conduct extensive experiments to show the proposed hierarchical context descriptions are crucial to precisely and universally define each category. As result, our method consistently outperforms the competitors on the large-scale OOD datasets, while showing comparable or even better generalization than the remarkable zero-shot methods. With the vision-language prompting framework, the precise and universal category descriptions via hierarchical contexts present a novel application: CATegory- EXtensible OOD detection (CATEX). We then merge the context descriptions leamned from different

---

task data, and directly test on the union ID setting, The competitive results demonstrate that the learned descriptions can be used across tasks. Consequently, to illustrate the salegory-extensibinn we incrementally extend the context descriptions to the whole ImageNet-21 K categories [1] at an acceptable GPÚ memory cost, and achieve superior performance to rivals. In addition, CATEX offers new insights on how to incorporate large language models (like ChatGPT) to boost zero-shot classifications (e.g., implicitly constructing spurious contexts to perform one-class OOD detection).
In summary, we make the following contributions:
1. We construct the perceptual and spurious contexts to hierarchically describe each category to perform ID/OOD detection. Our method consistently surpasses the SOTA method on the challenging ImageNet- IK benchmark, leading to an 8.27% decrease in FPR95.
2. We empirically demonstrate the proposed hierarchical contexts are prone to learning precise category descriptions while keeping the generalized feature representation. When data shift occurs, our method shows stable robustness on both ID classification and OOD tasks.
3. We present novel category-extensse merging the successively learned hierarchical contexts, we show that precise descriptions enable cross-task ID classification and OOD detection. We hope to offer new insights on how to scale up the prompt engineering in VL.Ms to recognize thousands of categories, as well how to incorporate LLMSs to boost zero-shot applications.
(CATEX) OOD detection framework. By simply
2
Preliminaries
This paper considers the multi-elass classification as the in-distribution (ID) situation, and the OOD detection task is formalized as follows.
Notations. Let X' E #x x3 denote the input image space and yin = (1, ... ,C} denote the ID label space of C' categories in total. The labeled training set Di = (4, } is drawn i..d from the joint data distribution P. Payin. Let Px denote the marginal distribution on X. which is referred to as the in-distribution (ID). For ID classification task, let f X + R denote a function predicting the label of an input sample by ú: = orgmar, [TT))
In-distribution classification. In lasion-language models, let x E IR denote the (2- no-normalized d-dimension visual feature of input images via the image encoder: I(I), the mapping function can be expressed as [() WTI((I) = W x. where W [ ;; W2; wc) E Rd C is the collection of /z-normalized text features for C categories. Specifically, each text feature is extracted from the category description via text encoder: Wi= = [[ 1: 1 ;
CLS²)) A T((x: CLS&)). where VA presents the m word embedding vectors for k-th ID category and CLS; is the given category name. The ID-label prediction becomes 9 = argmaz, w argmaa. x).
Out-of-distribution detection. When an OOD sample with unknown category y & y emerges test time, it should not be predicted to any known category. Hence, the OOD detection task can be viewed binary classification problem. Let g X' + Rd denote function distinguishing between the in- E.S. out-of-distributtin samples. OOD detection estimates the lower level set G == {z g(z) S A}. where samples with lower scores gzz) are detected as OOD and vice versa. The threshold is typically chosen by ensuring a high fraction of ID data (e.g., 95%) is correctly preserved.
3 Method
From the data-distribution perspective, the key to OOD detection is to acquire precise descriptions for each ID category to determine the distribution boundary. Inputs beyond the specific boundary should belong to either OOD samples or other ID categories. However, has always been challenge 222 17 38). In conventional methods using vision or multi-modal models, one may fine-tune the image encoder to reshape the feature representation space. Then, the classification layer (for vision models) or text features (for VLMs) play the role in category descriptions, where calculated logits measure the relative distances between inputs and corresponding ID categories. However, due to the absence of unseen categories, both the over-fitted feature representations [28] and the ID-label-biased category descriptions are prone to overconfident predictions on unseen OOD data [40].
3

---

ID image samples
spurious synthesis spurious category
peroeptaal description
spurious description
perturbed description
ID category
classification boundary
category boundary
Veba only used in training
learmable contextr
(ers
(CLL
Image Encoder
Text Encoder
(CLS
[CLS
Figure Illustration of our method. Perceptual context perceives a certain ID category, and spurious context explicitly describes a spurious category around this ID category. Random perturbation is applied to the perceptual context for synthesizing outliers to train the non-trivial spurious context. The hierarchical perceptual and spurious contexts jointly describe the precise category boundary.
To alleviate these problems, we take the vision-language prompting framework 71 to learn precise and universal descriptions for each ID category. First, both the image and text encoders are frozen to preserve the generalized representation capacity. Second, propose hierarchically construct the perceptual and spurious context for each category to mitigate label bias.
In the following sections, we present the proposed CATEX by elaborating on the following questions: How do perceptual and spurious contexts model the precise and universal category descriptions (Sec. 3.1 ? 2. How to learn such hierarchical contexts (Sec. 3.2 3. How to leverage the precise
category descriptions for ID classification and OOD detection (Sec. 3.3 The overview framework of our method is illustrated in Fig.
3.1 Hierarchical Contexts
Recall that empirical risk minimization (ERM) (54 operates under the closed-world assumption. When an OOD sample Px emerges, the ID classifier might produce overconfident predictions as 2 argmar, (w. 2). where w T T[V"; CLS)) is the encoded text feature for k-th category. Here vx is denoted as perceptual context that perceives the original multi-category classifications. However, with vğ only, whether z actually belongs to the k-thh ID category or is just an unseen OOD semantically close this category is still unclear. Hence, propose to add hierarchical context to model such OOD samples as another spurious category. The image-text similarity is measured by (w,, , z). where w T([vī: CLS)). Here vř is denoted spurious context that identifies such semantically similar but spurious OOD samples. We propose to utilize such two types of hierarchical contexts to jointly model the precise and universal boundaries for each category.
First, perceptual context serves to carefully adapt to the current ID classification task. As we propose to model potentially spurious category for C' ID categories respectively, the total category number equivalent to grows to C C = 2C in the scope. When an ID sample (x,, yi) emerges, to learn more strict classification boundary, we put it apart from both other ID categories and their corresponding spurious categories by:
(W xx))
or argmin W w
Σ
log (w).).)) + SC
(wf.x)) Ek
Then, spurious context is hierarchically combined describe the specific category boundary to handle the OOD detection. To formulate the open- open-world situation, when an ID sample x or OOD sample z with the predicted category emerges, the OOD detection risk can be expressed as:
R= ExxPx
[[((W(x) < (w x)}}
ExuPx [[(WW) 2) (wz.2)}

---

For each ID category. if we can proactively draw the spurious OOD samples and their category- predictions as Dir {(z,, 9: Mi=1- the risk in Eq. (2) can be empirically minimized by:
( xi)
(W2, ...)
ar argmin W
Σ log e xx.) + (Wa, xx)
+ ñi
Σ
log (w))
(w 2.)
In short, based on Eq. (T). perceptual context first learns to model a more striet inter-class decision boundary. Hierarchically, combining with Eq. (3), perceptual context and spurious context jointly model the precise category boundary, which is the key to OOD detection. More details on the whole training procedure are provided in Appendix A
Intuitively, how draw the spurious training set Di, is crucial. This paper presents a well-designed sampling strategy to learn the hierarchical contexts.
3.2 Perturbation Guidance
Generating adversarial data samples has been widely studied in recent years (39
43
among
which feature-space sampling is proven more tractable 50). In general, previous approaches randomly synthesize spurious samples away from ID features' clustering centers (with low likelihood or large distance), and treat them as visual outliers. To improve the quality of spurious syntheses, we provide perturbation-uuide sampling strategy that leverages the prior knowledge of the pre-trained large-scale aision-langual models.
In practice, an unseen spurious sample generally shares part of visual characteristics with the images, and the rest remains different [18] Under the vision-language prompting framework, in the learned per- ceptual context of a certain ID category v = [v 2 ... m)).))
Perturbation
peresphial contert
unooder
each word embedding uh actually describes a visual characteristic for this category 77.. According to the contrastive learning ob- jective during the pre-training process image-text pairs, if the text perturbed, it should not be paired with the original image. Correspondingly. if the learned word embeddings the encoded text feature should also not describe images from the original ID category. Thus, as shown in Fig. 3 can proactively leverage the perturbation to guide the spurious syntheses to boost context learning. Specifically, after randomly applying pertur- bation (e.g., masking (29)) onto the word k of perceptual context, the perturbed text feature becomes w -( u. ;.; km CLS,.).. Rather than the original ID category, w now describes a kind of unknown spurious category, of which the images should share higher similarities with perturbed text feature w than the original wh. Hence, we propose to re-sample the randomly synthesized spurious candidates by:
perturbal content
Guidance
Albered sperlestiy
pereygian
perturbed,
somples
porturbed
sedected symtheria
Figure 3: Guiding process.
Di, ((2,, 0:): (WP 23) (w 2,))
(4)
Combining with Eq. (3) and Eq. (T. the spurious context vî is able to learn to describe the synthesized spurious samples with such perturbation. After numerous iterations, random perturbations on perceptual contexts bring diverse spurious syntheses, which lead to learning non-trivial spurious contexts. More details on perturbation guidance (e.g., perturbed ratio) are discussed in Appendix A.2
After learning the hierarchical perceptual and spurious contexts for each category, we are able to probe the precise and universal ID category boundaries.
3.3 Integrated Inference
After learning the hierarchical perceptual and spurious contexts, to further alleviate the overconfidence predictions on unseen OOD samples, propose regularize the vanilla image-text similarities by:
Th = (wf, x) X xw(xx) + (wgxx)
x
(5)
5

---

Eq. (5) is unified measurement. For ID classification, the category is predicted by 9 = argmazz Tk- For OOD detection, the commonly-used scoring function 38 50 is adopted: g(x) max
>, e
The motivation intuitive. When perceptual context w2 produces a relatively higher image-text similarity & unseen OOD samples misclassified ID inputs, the hierarchical spurious context w could reduce the overconfident similarity by the regularization item y. More empirical experiments presented in the following sections.
4 Experiment
In this section, we empirically validate the effectiveness of our CATEX on real-word large-scale classification and OOD detection tasks. The setup is described below, based on which extensive experiments and analysis are displayed in Sec. Sec. 4.2
Datasets. Following the common benchmarks in the literature 59
50
60 38 we mainly consider
the large-scale ImageNet [[T] as the in-distribution data. Subsets of itaturaaiist [53]. SUN [65], Places [69], and Texture S adopted as the OOD datasets. The categories in each OOD dataset are disjoint with the ID dataset 25 38, 50]. This paper investigates four practical scenarios in real-world application of ID classification and OOD detection tasks: (1) standard OOD detection, (2) ID-shifted OOD detection, (3) category- category-extended extended ID classification and OOD detection, and (4)
zero-shot ID classification. Details are presented in Sec. 4.1
Evaluation metrics. For OOD detection, two metrics are used: (1) FPR95, the false positive rate of OOD samples when the true positive rate of ID samples is 95%, and (2) AUROC, the area under the receiver operating characteristic curve. For ID classification, we report the mean accuracy (ACC).
Implementation details. Without loss of generality. our method is implemented based on the vision- Lasounlanguage prompting framework [7]] with the CLIP model 4T, one of the most popular and publicly available pre-trained models. CLIP aligns an image and its corresponding textual description the feature space through a contrastive objective. We mainly take CLIP-B/16 in our experiments, which comprises of ViT-B/16 Transformer [4] as the image encoder and a masked self-attention Transformer (55) the text encoder. During training, all parameters of the image and text encoders are frozen. We only learn pair of perceptual and spurious contexts for each ID category. Following default setting (71), each context consists of learnable 512-D prompt embeddings, which are trained for 50 epochs using the SGD optimizer with momentum of 0.9. The initial learning rate is 0.002, which is decayed by the cosine annealing rule. The contexts are optimized with synthesized samples, and surrogate OOD datasets are involved during training.
4.1 Main Results
Table 1: OOD detection performance for ImageNet-I Ik || as dataset.
OOD Datasets
Method
iNatualist
SUN
Places
Texture
Average
FPR95]
AUROC1
FPR95
AUROC1
FPR95
AUROCT
FPR95]
AUROCT
FPR95.
AUROCT
MCM
32.08
94.41
39.21
92.28
44.88
89.83
58.05
85.96
43.55
MSP 18
54.05
87.43
73.37
78.03
72.98
78.03
68.85
79.06
67.31
80.64
Energy
$1.35
VIM 57
32.19
93.16
54.01
87.19
60.67
83.75
53.94
87.18
50.20
87.82
KNN
29.17
92.67
39.61
91.02
64.35
85.67
SSD+ 45
59.60
85.54
75.62
73.80
83.60
68.11
39.40
82,40
64.55
77.46
DOE 66
55.87
85.98
80.94
76.26
67.84
83.05
34.67
88.90
59.83
83.54
VOS
31.65
94.53
43.03
91.92
41.62
90.23
56.67
86.74
43.24
90.86
NPOS
50
16.58
96. 19
43.77
90.44
45.27
89. 90.44
46.12
88.80
37.93
91.22
41.43
33.17
92.73
CATEX significantly improves standard 0OD detection. We first compare the proposed CATEX with competitive OOD detection methods that also delve into describing the ID category boundary. In particular, MCM 38 is the latest one adopt the pre-trained CLIP to perform the ID boundary description zero-shot way, and others turn to fine-tune the encoders to optimize the ID) boundary. The results are presented in Tab. I where the best performance is marked bold, and the

---

Table Generalization across ID domains. Models are trained on ImageNet- IK and directly tested on shifted ID datasets.
Target Datasets
Method
ImageNet-A
ImageNei-RR
ImugeNet-Sketch
ACC1
FPR95,
AUROCT
ACC
FPR951
AUROCT
ACC1
FPR95
AUROCT
MCM
38
50.61
71.52
80.65
75.95
$2.67
89.49
47.42
73.73
80.74
MSP 18
43.96
86.25
63.25
66.50
82.21
75.07
45.18
91.67
58.45
Energy
43,96
89.78
63.96
66.50
89:55
68.43
45.18
95.56
54.12
VOS
43.79
80.03
73.45
66.83
78.42
80.09
46.06
86.68
68.58
NPOS [50
50.16
74.57
75.37
73.58
75.64
82.71
50.29
82.87
71.56
50.87
81.04
76.67
Table 3: Generalization across ID tasks. Models are independently trained on disjoint ImageNet- 100 (I) and ImageNet-100 100 (II), and then tested on the union ImageNet-200 (I U II) without fine-tuning
Datasets
Method
ImageNet 100 (I)
ImageNet 100 (II)
ImageNet 200 (1 II)
ACC1
FPR95]
AUROCT
ACC1
FPR95
AUROCT
ACC1
HPRYS
AUROCT
MCM 38
89.00
24.38
95.59
90.48
19.84
96.44
83.35
27.18
94.88
MSP
41.34
93.39
94.82
93.75
60.93
87.26
Energy
33
94,70
32.11
94.36
94.82
32.95
93.86
87.69
37.78
92,62
VOS 15
94.68
25.19
95.60
94.72
20.97
96.01
86.14
34.16
93.42
NPOS 50
94,24 (40.00)
16.54 (-0.00)
96.62
94.32 (10.00)
16.84 (-0.00)
96.35
86.23 (40.00)
25.54 (-0.00)
94.35
Ours
94.12 (-0.12)
10.31 (-6.25)
97.82
94.42 (+0.
7.91 (-8.93)
98.31
89.61
(63.38)
13.34 (-12.20)
97.19
second best is underlined. Accordingly, our CATEX achieves superior OOD detection performance, outperforming competitive rivals by a large margin. Specifically, CATEX consistently surpass the SOTA method NPOS (50) on all of the four OOD datasets, leading to an 8.27% decrease in FPR95 and 2.24% increase in AUROC on average. It implies that compared to zero-shot probing and macoder-fine-tune learning the hierarchical perceptual and spurious contexts for each category is more effective acquiring precise ID boundaries. In addition, our method dose not conflict with other post-hoe approaches (such as ReAct (46) and ASH (12),, and appropriate combinations may bring further improvements (e.g., 27.56% of FPR95), as discussed in Appendix B.2
CATEX properly generalizes to ID-shifted OOD detection. If the description of each category precise enough, it will not only perform well in OOD tasks, but also be discriminative even when the distribution of ID data is shifted. Hence, we evaluate both the ID classification and OOD detection ability of models trained on ImageNet- [I]] while tested on shifted ID datasets including ImageNet-A A [20). ImageNet-R (23), and ImageNet- Sketch (56). As shown in Tab. 2 the best ID and OOD performance on both ImageNet-A A and ImageNet-R R and competitive results on ImageNet-Sketch. It indicates that the generalization ability from the large pre-trained models is well-maintained and even straightened. The fine-tuned models like NPOS are unstable across different datasets, which may be a sign of generalization degradation caused by parameter min-ttuing. Because the data shift is inevitable in real-world applications, we suggest that the proposed hierarchical contexts are instructive for model generalization.
CATEX reaches
CATEX carefully adapts to eategdry-extended ID classification and OOD detection. If hierarchical contexts learn precise category description, it should prevent overconfident predictions on any unseen samples in open-world but beyond the current ID task scope. We thus conduct a cross-task experiment to further evaluate the open-set discriminability. Specifically, two models are independently trained on ImageNet- 100 (I) (38) and another disjoint ImageNet- 100 (II) randomly selected from ImageNet- Then, models are directly tested the union ImageNet-200 (I II). OOD detection is simultaneously evaluated, and the results are displayed in Tab. 3
Accordingly, our CATEX achieves the highest performance on the union ImageNet-200 and signifi- cantly surpasses the competitors (e.g.. 3.38% increase on accuracy and 12.2% decrease on FPR95), In the category-extended scenario, our classification accuracy only drops by 4%, which is much lower than the fine-tuning methods (e.g., 8% decrease from VOS (5).. implies that fine-tuning the encoder is prone to overfitting seen training samples, and sacrificing the ability (obtained from
7

---

x seen ID images
unscen OOD images
leamed pereeptual context
learned spurious context
distances
features after finetuming
feutures w/o finetuming
messure with single context
mensure with hisrurchical contexts
Figure 4: Feature visualization by t-SNE. (a) Previous approaches that fine-tune the encoders may distort the generalized feature space and make unseen OOD samples inseparable; (b) instead, our method freezes the encoders to maintain the discriminability. prompting methods using single perceptual context only, (d) our spurious context provides better metric for unseen OOD detection.
(c) Compared with traditional
large-scale pre-training) to distinguish unseen categories. We randomly select five categories from each ImageNet-100) subset respectively, whose feature distribution shown in Fig. 4 is consistent with Tab. 3 Moreover, compared to the SOTA method NPOS, our CATEX outperforms more on the union ImageNet-200 than separated ImageNet- subsets (e.g.. 12.2% v.s. 6.23% decrease on FPR95). It further demonstrates that in each ImageNet- 100 subset, besides our Iearned perceptual contexts that classify the current 100 categories, the spurious contexts are able to identify the unseen categories (both of the new-coming ID samples and OOD samples), illustrated in Fig. In this way, the precise boundaries for each category are acquired. On the other hand, to deal with such category-exteniblle tasks in real-world practice [10 62 6I], this paper provides separately learning the precise descriptions for the new-coming categories besides the existing ones, and jointly testing on the full ID category scope.
perspective:
To further evaluate the efficacy of the precise descriptions, we conduct more challenging larger-scale category-extensine experiments: scaling up ImageNet-IK to ImageNet-21K [[T]. In particular, we randomly split the huge ImageNet-2 dataset into several subsets, individually train the hierarchical percep- tual and spurious contexts for each category on each subset, and concurrently test on the full ImageNet-21K. Such paradigm is similar to category-incrementaa -nnremeeneel learning, and the results shown in Fig. 5 where we mainly compare with zero-shot CLIP 41 and baseline CoOp (71.. According Fig. [ our CATEX consistently outperforms CLIP and CoOp. Specifically, when the total number of categories extends, CoOp has a higher accuracy drop, since its learned contexts on each separated subset share limited discriminability to novel categories across subsets. And training on -thous- categories together is challenging due to the generally prohibitive computational costs, which require over 300 GB GPU memory the text-encoder on 21K categories. To alleviate this problem, this paper proposes to learn the hierarchical contexts to describe the precise and universal category boundaries, and achieves 38% accuracy on the full ImageNet-21 K with single V V100-32G GPU card. We hope to offer new insights to the community on how to adopt large-scale VLMs to classify numerous categories with limited GPU resources.
CLIP
CoOp
Accuracy
Ours
IK
21K
ImageNet
Figure 5: Scaling up ImageNet- IK 11I to ImageNet-21K [I] with
category-incrrmental learning.
CATEX encouragingly boosts zero-shot ID classification. Under the category-extended scenario, the hierarchical contexts help establish the category boundary for each given category, and prevent overconfidence on other unseen categories by adjusting the image-text similarity with the spurious- regulareeegl item 7, as introduced in Eq. (5). Intuitively, in zero-shot classification scenarios, the regularized score r s X " may also rectify the category predictions and lead to a higher classification accuracy, with proper perceptual and spurious contexts. To verify it, as CLIP's default prompt template "A photo of [CLS]" contains no visual information, we adopt rich visual descriptions from large language models 37 (such as GPT-3 3 as our perceptual context wp to perform classification (denoted as VisDesc). Then, we randomly perturb the visual description to simulate spurious contexts W,, and regularize text-image similarities via Eq. (5). The results shown in Tab, 4 indicate the regularized score r brings higher classification accuracy, without any training
8

---

Table 4: Zero-shot classification via text-image similarity regularized by simulated spurious contexts.
Method
Prompt Example
ACC-Topl1 ACC-Top5t
CLIP
A photo of goldfish.
63.50
88,99
VisDesc
A yellow fish with long flowing tail is goldfish.
65.47
90.14
Ours
[MASK] fish with long [MASK] [MASK] is goldfish.
65.84
90.36
cost. We hope to provide new insights that in the category-extensible setting or a true zero-shot classification scenario, explicitly constructing spurious contexts can perform the one-class OOD detection task, as other categories can also be viewed as OOD for certain ID category.
4.2 Ablation Studies
In this section, we take ImageNet-100 33 as the ID dataset, and other settings follow Sec. 4.1 We mainly verify the key contributions of this work in Tab. and the influence on model capacity in Tab. 6 In particular, we evaluate the number of spurious contexts used to help probe the category boundary in Tab. 7 More experiments and discussions are displayed in Appendix A and Appendix B
Table 5: Ablation study on proposed framework.
Table 6: Ablation study on model capacity.
Method
ID-ACC f
FPR95]
AUROC1
Model
Method
TD-ACC
FPR95..
AUROC1
baseline 71
94.12
13.07
97.42
RNS0
MCM
84.26
32,31 19.58
94.61
Hieemrchical.c Contexts
93.84
11.39
97.66
Ours
90.58
96.25
+ -oortrbation-Guuddnne
94.10
10.59
97.81
+Integrated-Infeer 94.12
10.31
97.82
VIT-L/14
MCM
20.79
96.35
Our
96.18
6.97
98.43
The proposed components are effective. In Tab. 5 baseline model is first constructed following the vision-language prompt-learning framework CoOp (7I) which only learns the perceptual contexts with the vanilla softmax cross-entrops loss. Then, we learn the hierarchical perceptual and spurious contexts using initially synthesized OOD samples with Eq. [ and Eq. (3) According to Tab. 5 the OOD detection performance is improved immediately, whereas the ID classification accuracy slightly drops by 0.3%. indicates that the quality of initial OOD syntheses is limited, which puts the learned descriptions at risk of identifying ID inputs as OOD or even other ID categories. On the contrary, the proposed perturbation guidance mechanism in Eq. 4 provides extra constraints on OOD syntheses to improve the quality, which brings consistent gains in both ID classification and OOD detection protocol. Finally, the integrated inference strategy in Eq. ( brings free improvement on FPR95 without loss of ID accuracy. The efficacy of our proposed method is further demonstrated.
Our method is scalable model capacity. In Tab. we investigate the scalability of our method with different model capacities. i.e., lighter model with modified ResNet50 (RN50) and heavier model with ViT-L/14 as the image encoder. The results imply larger models indeed lead to superior performance, suggesting larger models are endowed with better representation qualities [38] [50]. Interestingly, though the zero-shot method MCM 38 with ViT-L/14 surpasses our method with RN50 by 1.1% of ID classification accuracy, the OOD performance is inferior to ours (e.g., over 1.2% increase on FPR95). It further demonstrates that learning precise category descriptions via our hierarchical contexts is still key to deriving better OOD detector in the open-world.
Multiple spurious contexts bring more precise descriptions. For each ID category (e.g., cat), the spurious OOD samples can be diverse (e.g., panthers, lions, etc). Thus, we aim to only describe the spurious sample surrounding the ID category, rather than the whole OOD space (which is too complicated). According to common sense, it is intuitive to use more spurious contexts to describe better category boundaries. To verify it, we have tested the number of spurious contexts for each ID category (denoted as W/o Constraints), and the results shown in Tab. immplie using more spurious contexts only leads to 0.1% gain on performance. The reason may that the learned 2 or more spurious contexts are too redundant without any constraints. To alleviate this problem, we simply add an orthogonal constraint (making the similarities between each two spurious contexts close to zero) (denoted as W/ Constraints), and the OOD detection performance is significantly boosted, as displayed in Tab. 7 Therefore, how to effectively and efficiently leverage more spurious contexts to better describe the category boundary deserves further exploration, and we view it as future work.

---

Table Ablation study on multiple spurious contexts.
W/o Constraints
W/ Constraints
N₂
FPR95
AUROCT
FPR95L
AUROC1
1
10.31
97.82
10.31
97.82
2
10.21
97.86
10.17
97.86
4
10.27
97.88
9.89
97.89
10.25
97.86
9.76
97.84
Related Works
OOD detection in computer vision. To keep the safe deployment of multi-category classification models in open-world scenarios, the goal of OOD detection is to derive a binary ID-v.s-OOD detector for the visual inputs. To reduce the overconfidence on unseen OOD samples M1 surge of post-hoc scoring functions has been devised based on various information, including output confidence [2 32]
25 free energy [33 58 15 Bayesian inference
30
34
51, gradient information [24], model/data
sparsity [46 73 [2], and visual distance [31 49
50 Another promising approach is adding open-set making models produce lower confidence
regularization in the training time [66] 22
26
52
or higher energy on OOD data. Manually Hamually-colleee collected [22 60 or synthesized outliers [15] 50 are
required for auxiliary constraints. Works insofar have mostly focused on regularizing the task-specific classification models trained on downstream scenarios using only visual information. In contrast, this paper explores the OOD detection on information to probe the task-agnostic category boundaries.
scenarios, which incorporates rich textual
OOD detection with vision-language models (LVMss. Exploring textual information for visual OOD detection with large-scale pre-trained models has recently attracted a surge of interest. One may simply use the vision-language model CLIP 41 explicitly collect potential OOD labels [ [6] or conduct zero-shot OOD detection (38) with manually pre-defined text prompts. However, as such manual text prompts lack the description of unseen categories, the prior knowledge learned by VLMs during pre-training is not fully exploited, leading to inferior OOD detection performance. On the contrary, we propose to learn hierarchical contexts to describe the category boundaries against unseen spurious categories via vision-language prompt tuning (71. RPL (7] and ARPL 66 explored similar idea of "Reciprocal Points" to constrain more compact in-distribution feature space, which may distort the generalized representation like NPOS (50). Our method largely boosts the OOD detection performance, and simultaneously maintains the generalization capacity of pre-trained VLMs.
Conclusion
This paper proposes novel framework to learn the precise category boundaries via the hierarchical perceptual and spurious contexts. Specifically, the perceptual context first perceives relatively more strict classification boundary on the current IDD task, and then integrates the spurious context to determine more precise category boundary that is independent of the current task setting. The extensive empirical experiments demonstrate the proposed hierarchical contexts are prone to learning precise category descriptions, which shows robustness to various data shifts. By simply merging the task-independent precise category descriptions, we provide a new perspective efficiently extend the ID classification scope to twenty-thousand categories at an acceptable resource cost. We also offers new insights to boost zero-she shot classification via text-image similarities regularized from an one-class OOD detection perspective.
I Limitations and societal impact. To keep the representation capacity of visual features and the generalization ability to identify unseen samples, we freeze the image encoder of the pre-trained CLIP. It limits the upper-bound performance on ID classification and OOD detection of our method. Larger models (e.g., ViT-L/1 @ 336px) might bring higher performance but require more costs. There is a trade-off between efficacy and efficiency. Besides, accumulated during pre-training on uncurated web-scale datasets are inherited. Directly applying our method may cause biased predictions containing stereotypes or racist content. Leveraging debiasing techniques could be the key to alleviating this problem.
do not fine-tune the encoder, the social biases
10

---

Acknowledgments and Disclosure of Funding
This work was supported part by the Fundamental Research Funds for the Central Universities, and in part by the National Key R&D Program of China under Grant 2020/AA0000302..
References
Abhijit Bendale and Terrance Boult. Towards open world recogmition. In Proceedings of the IEFE conference on computer vision and pattern recognition, pages 1893-1902, 2015.
[2] Daniel Bolya, Cheng- Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. Token merging: Your ViT but faster. In International Conference on Learning Representations, 2023.
[3] Tom Brown, Benjamin Mann, Nick Ryder, Melanic Subhiah, Jared D) Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
[4] Boxi Cao, Hongyu Lin, Xianpei Han, Le Sun, Lingyong Yan, Meng Liao, Tong Xue, and Jin Xu. Knowledgeable or educated guess? revisiting language models as knowledge bases. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 11th International Joint Conference on Natural Language Processing (Volume I: Long Pupers), pages 1860-1874, 2021.
[5] Senqi Cao and Zhongfei Zhang. Deep hybrid models for out-of-distributtinn detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4733-4743, 2022.
[6] Guangyao Chen, Peixi Peng, Xiangqian Wang, and Yonghong Tian. Adversarial reciprocal points learning for open set recognition. IEEE Transactions on Pattern Anaysis and Machine Intelligence, 44(1 1):8065- 8081, 2021.
[7] Guangyao Chen, Limeng Qiao, Yemin Shi, Peixi Peng, Jia Li, Tiejun Huang, Shiliang Pu, and Yonghong Tian. Learning open set network with discriminative reciprocal points. In Cumputer Vision-ECC Aition-ECVVP2 2020: 16th European Conference, Glaxgow, UK, August 23-28, 2020, Proceedings. Part III 16, pages 507-522. Springer, 2020.
[8] Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages (606-3613) 2014.
[9] Jeremy Cohen, Elan Rosenfeld, and Zico Kolter. Certified adversarial robustness via randomized smoothing. In international conference machine learning, pages 1310-1320. PMLR, 2019.
[10] Matthias De Lange, Rahaf Aljundi, Mare Masana, Sarah Parisot, Xu Jia, Aleš Leonardis, Gregory Slabaugh, and Tinne Tuytelaars. A continual learning survey: Defying forgetting in classification tasks. IEEE transactions on pattern analysis and machine intelligence, 44(7):3366-333 2021.
[11] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and puttern recognition, pages 248-255. IEEE, 2009.
[12] Andrija Djurisic, Nebojsa Bozanic, Arjun Ashok, and Rosanne Liu. Extremely simple activation shaping for out-of-distribution detection. International Conference on Learning Representations. 2023.
[13] Andrija Djurisic, Nebojsa Bozanic, Arjun Ashok. Rosanne Liu. search lost domain generalization. In International Conference on Learning Representations. 2023.
[14] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer. Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image worth 16x16 words: Transformers for image recognition at scale. International Conference on Machine Learning. 2021.
[15] Xuefeng Du, Zhaoning Wang, Cai, and Yixuan Li. Vos: Learning what you don't know by virtual outlier synthesis. In International Conference on Learning Representations, 2022.
[16] Sepideh Esmaeilpour, Bing Liu, Eric Robertson, and Lei Shu. Zero-shot out-of distribbubt detection based on the pre-trained model clip. In Proceedings of the AAAI conference on artificial intelligence, volume 36, pages 6568 6576, 2022.

---

[17] Zhen Fang, Yixuan Li, Jie Lu, Jiahua Dong, Bo Han, and Feng Liu. Is out-of-distribution detection learnable? In Advances in Neural Information Processing Systems, 2022.
[18] Stanislav Fort, Jie Ren, and Balaji Lakshminarayanan. Exploring the limits of out-of-distribution detection. Advances in Neural Information Processing Systems, 34:7068-7081, 2021.
[19] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.
[20] Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, al. The many faces of robustness: A critical analysis of distiiiiittiitttion generalization. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8340-8349, 2021.
[21] Dan Hendrycks and Kevin Cimpel. A baseline for detecting misclassified and in neural networks. In International Conference on Learning Representations, 2017.
examples
[22] Dan Hendrycks, Mantas Mazeika, and Thomas Dietterich. Deep anomaly detection with outlier exposure. Proceedings of the International Conference Learning Representations, 2019.
[23] Dan Hendrycks, Kevin Zhao, Steven Basart, Jacoh Steinhardt, and Dawn Song. Natural adversarial examples. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pages 15262-15271, 2021.
[24] Rui Huang, Andrew Geng, and Yixuan Li, On the importance of gradients for detecting distributional shifts in the wild. Advances in Neural Information Processing Systems, 34:677-689, 2021.
[25] Rui Huang and Yixuan Li. Mos: Towards scaling out-of-distribution detection for large semantic space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8710-8719,
2021.
[26] Tacwon Jeong and Hecyoung Kim. Ood-maml: Meta-learning for few-shot out- -iittiibbttinnooonnnnnn detection and classification. Advances in Neural Information Processing Systems, 33:3907-390 3916, 2020.
[27] Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4401-4410, 2019.
[28] Ananya Kumar, Aditi Raghunathan, Robbie Jones, Tengyu Ma, and Percy Liang. Fine-tuning can distort pretrained features and underperform out-of-distribution In International Conference on Learning Representations, 2022.
[29] Gukyeong Kwon, Zhaowei Cai, Avinash Ravichandran, Erhan Bas, Rahul Bhotika, and Stefano Soatto. Masked vision and language modeling for multi-modal representation learning. In International Conference on Learning Representations, 2023.
[30] Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in neural information processing systems, 30, 2017.
[31] Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for detecting out- of-distribution samples and adversarial attacks. Advances in neural information processing systems, 31, 2018
[32] Shiyu Liang, Yixuan Li, and Srikant. Enhancing the reliability out-of-distribution image detection in neural networks. In International Conference on Learning Representations, 2018.
[33] Weitang Liu, Xiaoyun Wang, John Owens, and Yixuan Li. Energy-based out-of-distributiin detection. Advances in neural information processing systems, 33:21464-22
2020.
[34] Wesley J Maddox, Pavel Izmailov, Timur Garipov, Dmitry P. Vetrov, and Andrew Gordon Wilson. A simple baseline for bayesian uncertainty in deep learning. Advances in neural information processing systems, 32, 2019.
[35] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. To- wards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations, 2018.
12

---

[36] Andrey Malinin and Mark Gales. Predictive uncertainty estimation via prior networks. Advances in neural information processing systems, 31, 2018.
[37] Sachit Menon and Carl Vondrick. Visual classification via description from large language models. Proceedings of the International Conference on Learning Representations, 2023.
[38] Yifei Ming. Ziyang Cai, Jiuxiang Gu, Yiyou Sun, Wei Li, and Yixuan Li. Delving into out-of -f--tiiiittttiionn detection with vision-language representations. In Advances in Neural Information Processing Systems, 2022.
[39] Mchdi Mirza and Simon Osindero. Conditional generative adversarial nets. arXiv preprint arXiv:1411.1788 2014.
[40] Anh Nguyen. Jason Yosinski, and Jeff Clune. Deep neural networks are easily fooled: High confidence predictions for unrecognizable images. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 427-436, 2015.
[41] Alee Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Muchine Learning, pages 8748-8763 PMLR, 2021.
[42] Tal Ridnik, Emanuel Ben-Baruch, Asaf Noy, and Lihi Zelnik-Manor Imagenet-21k pretraining for the masses. arXiv preprint arXiv:2104.19972 2021.
[43] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10684-10695, 2022.
[44] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorcalistic text-to- image diffusion models with deep language understanding. Advances in Neural Information Processing Systems, 35:36479-36694, 2022.
[45] Vikash Schwag. Mung Chiang, and Prateck Mittal. Ssd: A unified framework for self-supervised outlier detection. In International Conference on Learning Representations, 2021.
[46] Yiyou Sun, Chuan Guo, and Yixuan Li. React: distr---iittiuution detection with rectified activations. In Advances in Neural Information Processing Systems, 2021.
[47] Viyou Sun and Yixuan Li. Dice: Leveraging sparsification for out-of-distribatioo detection. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXIV, pages 691-708 Springer, 2022.
[48] Yiyou Sun, Yifei Ming, Xiaojin Zhu, and Yixuan Li. Out-of-distribution detection with deep nearest neighbors. In International Conference on Machine Learning pages 20827-20840. PMLR, 2022.
[49] Yiyou Sun, Yifci Ming. Xiaojin Zhu, and Yixuan Li. Out-of-distribution detection with deep nearest neighbors. In International Conference on Machine Learning, pages 20827-20840. PMLR, 2022.
[50] Leitian Tao, Xuefeng Du Jemry Zhu, and Yixuan Li. Non-parametris outlier synthesis. In The Eleventh International Conference on Learning Representations, 2023.
[51] Antonio Torralba, Rob Fergus, and William T Freeman. 80 million tiny images: A large data set for nonparametric object and scene recognition. IEEE transactions pattern analysis and machine intelligence, 10(11):195 1970, 2008.
[52] Joost Van Amersfoort, Lewis Smith, Yee Whye Teh, and Yarin Gal. Uncertainty estimation using single deep deterministic neural network. In International Conference on Machine Learning, pages 9690-9700. PMLR, 2020.
[53] Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pictro Perona, and Serge Belongie. The inaturalist species classification and detection dataset. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8769-8778, 2018
[54] Vladimir Vapnik. The nature of statistical learning theory. Springer science & business media, 1999.
[55] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems. 30, 2017.
13

---

[56] Haohan Wang, Songwei Ge. Zachary Lipton, and Eric P Xing. Learning robust global representations by penalizing local predictive power. In Advances in Neural Information Processing Systems, pages 10506-10518, 2019.
[57] Haoqi Wang. Zhizhong Li. Litong Feng, and Wayne Zhang. Vim: Out-of-distribution with virtual-logit matching. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pages 4921-4930,2022 2022.
[58] Haoran Wang, Weitang Liu, Alex Bocchieri, and Yixuan Li. Can multi classification networks know what they don't know? Advances in Neural Information Processing Systems, 34:29074 29087, 2021.
[59] Haotao Wang. Aston Zhang, Yi Zhu, Shuai Zheng, Mu Li, Alex J Smola, and Zhangyang Wang. Partial and asymmetric contrastive learning for out-of-distribution detection in long-tailed recognition. In International Conference on Machine Learning, pages 23446-2344 23458. PMLR, 2022.
[60] Qizhou Wang. Junjie Ye, Feng Liu, Quanyu Dai, Marcus Kalander. Tongliang Liu, HAO Jianye, and Bo Han. Out-of-distribution detection with implicit outlier transformation. In The Eleventh International Conference on Learning Representations, 2023.
[61] Zifeng Wang, Zizhao Zhang, Sayna Ebrahimi, Ruoxi Sun, Han Zhang, Chen- Yu Lee, Xiaoqi Ren, Guolong Su. Vincent Perot, Jennifer Dy, et al. Dualprompt: Complementary prompting for rehearsal-free continual learning. Computer Vision ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23 27, 2022, Proceedings, Part XXVI, pages 631 -648. Springer, 2022.
[62] Zifeng Wang. Zizhao Zhang, Chen- Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, and Tomas Pfister. Learning to prompt for continual lcarning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 139-149, 2022.
[63] Hongxin Wei, Renchunzi Xie, Hao Cheng, Lei Feng, Bo An, and Yixuan Li. Mitigating neural network overconfidence with logit normalization. In International Conference on Machine Learning, pages 23631- 23644. PMLR, 2022.
[64] Ross Wightman. Pytorch image models. 2019.
[65] Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large- scale scene recognition from abbey to In 2010 IEEE computer society conference on computer vision and pattern recognition, pages 3485-3492. IEEE, 2010.
[66] Jingkang Yang, Haoqi Wang. Litong Feng. Xiaopeng Yan, Huabin Zheng, Wayne Zhang, and Ziwei Liu. Semantically coherent out-of-distriiuttio detection. In Proceedings of the IEFF/CVF International Conference on Computer Vision, pages 8301-8309, 2021.
[67] Jingkang Yang. Kaiyang Zhou, Yixuan Li, and Ziwei Liu. Generalized out-of-distribution detection: A survey. arXiv preprint arXiv: 10. 1334, 2021.
[68] Tao Yu, Runseng Feng, Ruoyu Feng, Jinming Liu, Xin Jin, Wenjun Zeng, and Zhibo Chen. Inpaint anything: Segment anything meets image inpainting. arXiv preprint arXiv:2304.06790. 2023.
[69] Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralha. Places: A 10 million image database for scene recognition. IEEE transactions on pattern analysis and machine intelligence, 1452):44521466 2017.
[70] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[71] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. International Journal of Computer Vision, 130(9):2337-2344 2022.
[72] Yao Zhu, Yuefeng Chen, Xiaodan Li, Kejiang Chen, Yuan He, Xiang Tian, Bolun Zheng, Yaowu Chen, and Qingming Huang. Toward understanding and boosting adversarial transferability from distribution perspective. IEEE Transactions on Image Processing, 31:6487-6501 2022.
[73] Yao Zhu, YueFeng Chen, Chuanlong Xie, Xiaodan Li, Rong Zhang, Xiang Tian, Yaowu Chen, et al. Boosting out-of-disttibuttin detection with typical features. In Advances in Neural Information Processing Systems, 2022.
14

---

Appendices
A Implementation Details
A.1 Training Procedure
This section provides a walkthrough example illustrate the training procedure presented Sec. 3.1 of the manuscript. As displayed in Fig. [] we first generate spurious OOD syntheses by perturbation guidance in Sec. 3.2, and then learn in-distribution classification via Eq. ( and duttrf-dist detection via Eq. Specifically, Fig. A captures the training procedure for one category. For the multi-category scenario, please refer to Fig. in the manuscript.
sporsous synihhese
learnable contexts
perturbation
perturbed
grad-stop
boundary
[CLS]
perceptual
text encoder
2
[CLS]
grad- flow
[CLS]
sparions
ID Images
Figure Al: Training illustration for one category. The randomly perturbed context is adopted to select spurious OOD syntheses, where gradients are stopped. Then perceptual context describes the ID images, and spurious context describes those spurious OOD syntheses surrounding the ID category. The two contexts jointly model the category boundary.
A.2 Perturbation Guidance
Detailed perturbation guidance process. For arbitrary k-th ID category, use the spurious context to explicitly describe a corresponding spurious category, and critical consideration is how to synthesize training samples spurious to that k-thh ID category. Recently, generating adversarial data samples have been widely studied, including GAN networks
33 27). diffusion models 43 For simplicity, we take the tractable feature-space sampling as NPOS (50) to generate spurious candidates. We calculate the k-NN distance for each ID sample in the specific category, and sample OOD candidates from multivariate Gaussian distribution around the ID data points with largest distances (basically away from the clustering center). Then, leverage the perturbed descriptions of perceptual context to guide the further filtering for high-quality spurious syntheses. Visualization examples are provided in Fig. A2
image attacks 35 [72. and feature-space sampling [15 50
ID images
random OOD syntheses
selected OOD syntheses
perceptuzl context
perturbed context
spurious context
peroob
pull
select
(a) randomly synthesized OOD samples
(b) filtering with perturbed context
(c) iterativv syatheses to learn spurious context
Figure A2: Visualization of the Perturbation Guidance process. (a) Given a set of in distribution image samples, several OOD syntheses are randomly generated in the feature space. (b) We perturb the perceptual context to simulate a kind of OOD category, and select OOD syntheses subject to (more similar to) the perturbed context. (c) After iteratively sample generation and perturbation-guil guided selection, the synthesized OOD samples are adopted to learn the spurious context.
15

---

" - of k-th ID category, randomly apply a perturbation onto arbitrary o to produce a perturbed descrip- tion w. through text-encoder. Intuitively, there are two ways to perturb context v erasing or replacing the specific visual character o
Perturbation approach. Given the perceptual context v [ "..2i...
'E, Consequently, we design three types of perturbation: (I) masking with a placeholder u [MASK]. (2) noise from Gaussian distribution
o, and (3) swapping with another category u = 1 J" And the perturbed text-feature is produced by:
w = (( u ... ; m CLS As shown in Fig. A3 all of the three types of perturbed text-features W slightly deviate from the original w while keep the affinity (e.g., shares 97% similarity against the original one.) Specifically, the noised w leads to a greater deviation, since the noised visual character o is more unpredictable than the masked swapped ones.
masked
clean
masked
noised swapped images
noised
swapped
2400
6.00
166
(a) similarity distribution.
(b) t-SNE visualization
Figure A3: Statistics of perturbations, including (a) similarities between original and perturbed text-features, and (b) distribution of original text-feature, perturbed text-features, and image-features.
Perturbing ratio. As an important factor, perturbing how many visual characters {up) Jm to simulate a spurious category seems to be very sensitive. A too-small ratio may lead noneffective perturbation and cause invalid candidates from ID data, while too-large ratios can distort the spurious category description and bring invalid candidates from far OOD. In fact, as previous analysis in Fig. [3 implies perturbing one word/token v can produce effective deviation on the encoded text feature w Tab. A suggests that one-word perturbation is effective enough to perform perturbation guidance neeeewoo--returrttion
described in Sec. 3.2 Perturbing 4 or more words even leads performance degradation, which means severely perturbed contexts may choose the noisy OOD candidates (e.g. random noise), making the learned spurious context unable capture the true spurious OOD samples and further describe the category boundary.
Table A1: Ablation for perturbing ratio on ImageNet-100 benchmark.
Perturbing ratio
FPR95
AUROCT
1 16
10.31
97.82
2/ 16
10.27
97.81
4 16
10.47
97.78
8/ 16
11.02
97.73
16/16 16
11.70
97.62
Perturbing position. Another hyper-parameter to perform perturbation on perceptual contexts the perturbing position, i.e., which word/token to perturb? Sec. 3.2 from the manuscript, we randomly choose the word in perceptual contexts to perturb at each training iteration. Indeed, deciding the optimal word perturb is challenging, especially when the learned word embeddings do not correspond to actual words in natural language. Though, we have made simple step to find the "optimal" word/token the embedding space. After computing the prototype vector by averaging the 16 learned words, we take the most distant (denoted as MaxDist) or closed (denoted as MinDist) word perturbation guidance for spurious OOD syntheses. However, as shown Tab. A2 the final OOD detection performance even gets worse. Thus, we argue that randomly choosing the words to perturb each iteration is still an effective way. After several iterations, we can statistically perturb every word to guide the spurious sample generation, covering the optimal situation.
16

---

Table A2: Ablation for perturbing position on ImageNet- 100 benchmark.
Method
FPR954
AUROC
MaxDist
10.42
97.75
MinDist
10.86
97.73
Random
10.31
97.82
A.3
Cross-ID-Domain Generalization
As indicated in the manuscript, the precise category boundary learned by our method shows robust OOD performance when the ID data is shifted. In fact, the shifted ID classification be further boosted by our proposed integrated inference strategy (Eq. (5) in the manuscript), as shown in Tab. A3 implies the regularization item y successfully modulates the relative similarities between input images and learned perceptual descriptions for each category, leading to more precise boundaries.
Table A3: Additionally improved accuracy on shifted datasets.
Target Datasets
Method
ImageNet-A
ImageNet-R
ImageNet-Sketch
CATEX
50.87
76.67
48.59
+nntegInfer.
50.98
76.72
48.65
However, our method only takes secondary place on ImageNet-Sketch (66 on both ID classifi- cation (inferior to NPOS (50)) and OOD detection (inferior to MCM (88). It is mainly because of huge domain gap between vanilla ImageNet- I [T]] and shifted ImageNet-Sketch. As shown in Fig. A4 compared to the shifted ImageNet-A (20] and ImageNet-R R [23], images from ImageNet-
Sketch only preserve objects' shape and main texture, while the color information is totally vanished. We leave the generalization to -hifie ID datasets future work.
Figure A4: Left right: examples from ImageNet, ImageNet-A, ImageNet-R, and ImageNet-
A.4 Cross-ID- -D-ssk Generalization
To verify the efficacy of our proposed framework, we conduct a category-extended experiment in Sec.4.1 and Tab.3. Here more implementation details are provided for reproducibility.
Given two models independently trained on the separated ImageNet- (I) and ImageNet-100 (II, how to test them the union ImageNet-2200 (I U II) with CATEX is simple. In the vision- language prompt-tuning framework, the image-encoder I and text-encoder T are frozen, and we only learn the perceptual and spurious contexts (i.e., v and v*). And the Iz-normalized text-feature can be pre-extracted with the 100 category names each subset, taking the perceptual descriptions for example, which are denoted as /w,, T(v T, CLS ] k= and WI. T (vV CLS11, k=
k=1- Given an input image 1, the lz-normalized image-feature is extracted by I(I), and the perceptual image-text similarities computed as s" [(wf, (w?, x).
During inference, one may concatenate the 200 text-features together as [w 200
(w 2000 x)] & [s Sh - Similarly, the spurious similarities become s* [s s², ... .2n Then we can leverage the measurement defined in Eq. (5) for both ID classification and OOD detection.
As for the competitors, (e.g,, VOS [15 and NPOS (50)). two image-encoders are trained separately (denoted as I₁ and Imz). And for each input image I, there are two corresponding image-fcatures:
17

---

XI = (I) x11 XII = (I) Consequently, there also two sets of image-text similarity vector: [(wI.x)) xt), (W2. xr)..... (2200: xx) x.) ) k=1 and Str {(W,: X11) ]= (the superscript
is hidden for simplicity). For compatibility, we choose the one for ID classification and OOD
Sr maz(s:) > maz($$I)
detection according to its highest image-text similarity.
Now, the
S11 otherwise
performance of our method and other rivals are evaluated under the same measurements.
Note that since only take one image encoder throughout, the inference time is fixed (because the lext-features can be pre-extracted). In contrast, applying other methods brings multiple time cost (e.g., twice slower than ours in this case). When the training subsets extend intensely (e.g. from ImageNet-1K K to ImageNet-21K in our manuscript), our method still keeps fast speed (e.g., 100FPS on V100) during inference, which can even enable real-time applications in practice.
A.5 Software and Hardware
We use Python 3.7.13 and PyTorch 1.8.1, and NVIDIA V V100-32G GPUs.
B Additional Experiments
B.1 Comparison to CoCoOp
Compared with the vanilla vision-languase prompt tuning method CoOp (71). the newer version CoCoOP (70 was explicitly designed to deal with out-of-distribution issues through image conditional contexts. However, Tab. A4 indicates that CoCoOp is surprisingly much worse than our method. The reason may be that during training, CoCoOp only takes ID images context conditions, while neither OOD samples nor OOD contexts are involved. Thus, when employed in the open-world and asked reject OOD samples not belonging to any ID category, CoCoOp failed. The image conditions even aggravate the overconfidence in OOD samples.
Table A4: Comparison to CoCoOp on ImageNet-100 benchmark.
Method ID-ACC1
FPR95
AUROCT
CoCoOp
92.90
39.22
92.69
Ours 94.12
10.31
97.82
B.2 Combination with Post-hoc Enhancements
Recently, post-hoc OOD detection methods that enhance the single-vision-mode networks (e.g., ResNet [19] and ViT (4) have been widely studied 9 46
73 12] In this section, make step towards combining vision-language models with previous post-hoc enhancements for better OOD performance. The results are shown in Tab. A5 where ReAct 46 achieves remarkable improvement. It indicates that pruning the extreme feature values according to the unified distributional statistics may be more suitable for VLMs reduce the overconfidence on OOD samples. We hope this can bring new insights to the community.
47
Table A5: Combination with post-hoc methods.
ImageNet-
ImageNet- 1K
Cmobine
FPR95
AUROC1
FPR95]
AUROC1
None
10.31
97.82
29.66
93.48
ReAct 46
10.06
97.82
27.56
93.77
BATS 73
10.16
97.84
29.37
93.59
ASH 12
10.19
97.81
29.14
93.27
18

---

B.3 Inference-time perturbation
Now that every perturbation can directly produce the description (i.e., text-feature) of an unknown spurious category, one may try to take the perturbed description as a substitute for the learned spurious contexts execute OOD detection. That is, use the perturbed w replace the learned wE in Eq. (5) in the manuscript. And the results are shown in Tab. A6 where ImageNet-100 100 (38 is the ID dataset. Given baseline model [71] learned with perceptual contexts only, simply using the perturbed descriptions (denoted as + +Perturb-Desc) brings slight improvements (e.g., 0.2% decrease on FPR95). The insignificant advantage is not due to the limited capacity of only one perturbed description for each ID category. Because ensembling (38] several perturbed descriptions for ID category at once (denoted as (Perturb-Ensem)) dose not bring remarkable improvements. In contrast, our proposed CATEX can significantly enhance the OOD detection performance, which demonstrates it is still necessary to explicitly learn the spurious contexts for each ID category.
Table A6: Comparison with directly using perturbed descriptions for OOD detection.
Method
FPR95L
AUROCT
baseline 71
13.07
97.42
PPerturb-Dese
12.84
97.43
+Perturb-Ensem
12.87
97.45
CATEX (Ours)
10.31
97.82
B.4 Experiments on CIFAR Benchmarks
To further verify the robustness of our method, we conduct additional experiments on CIFAR-10 and CIFAR- datasets, and evaluate the OOD detection performance on SCOOD (66) benchmark. We train our CATEX for 20 epochs, and the other settings are the same as Sec.4 in the manuscript. The results are shown in Tab. A7 and Tab. A8 where "Surr." means the extra TinyImageS0MM (5I] is adopted for surrogate OOD training set. Accordingly, our CATEX consistently ourperforms the competitors well, and even surpasses those who adopts the extra OOD training data. implies the pre- tre-trained knowledge for large-scale CLIP [4]] model leveraged by our method is capable enough detect the OOD samples in the open-worle The efficacy of our CATEX is further demonstrated.
Table A7: Performance on CIFAR-10.
Table A8: Performance on CIFAR- 100.
Method
Surr:
ID-ACC 1
FPR95L
AUROC
Method
Surr.
ID-ACC
FPR95J
AUROCT
MCM
38
90.79
23.14
94.68
MCM
38
X
66.91
71.93
79.39
ODIN
X
95.36
52.00
82.00
ODIN 32
X
81. $1.84
81.89
77.98
Energy
33
X
95.36
50.03
83.83
Energy
X
81 84
83.66
79.31
OF 23
94,90
50.53
88.93
OR 22
81.31
80.06
78.46
UDG
M+
94.71
36.22
93.78
UDG 66
80.89
75.45
79.63
CATEX
X
95.57
21.17
95.33
CATEX
81.99
67.95
84.04
B.5 Error Bars
To verify the robustness, we repeat the training our method and the rivals on ImageNer- 100 38
with CLIP-B/16 for 3 times, and the results are shown in Tab. A9 Our CATEX consistently outperforms the rivals on OOD detection by significant margin.
Table A9: Error Bars on ImageNet-100 after 3 runs
Method
ACC1
FPR951
AUROC1
MSP 18
94.77 (20.05)
41.90 (=0.61)
93.38 (10005)
Energy 33
94.77 (20.05)
31.89 (=0.50)
94.53 (10.11)
VOS 15
94.75 (20.07)
24.48 (=0.71)
96.04 (40.36)
NPOS
[0
94.34 (20.12)
17.32 (=0.87)
96.46 (2:0.13
CATEX
94.11 (20.03)
10.97 (=00799
97.75 (1:0.07)
19

---

C Additional Analysis
Performance improvement. To further evaluate the improvement brought by method (e.g.. 8% decrease of FPR95 against NPOS), we conduct a comparative experiment on ImageNet-1K. To provide unified analysis across models, take a third-pan ResNet--50 model [64] (pre-trained on ImageNet- K classification only) to produce the Maximum SoftMax Probability for each OOD sample that is correctly detected by CATEX while wrongly viewed as ID samples by NPOS. According to Fig. A5 our method consistently improves the OOD detection on each interval, where the sigh-prbbbbility OOD (generally hard samples) detection is significantly enhanced. It indicates that properly leveraging the prior knowledge from pre-trained VLMs can alleviate the OOD problem when the fine-tuned visual features are indistinguishable, which is consistent with our motivation.
380
conrected
292
133
180
0.8
62
8/ SoftMux Probability
04
1a
Figure A5: Corrected OOD detections compaired with NPOS. The softmax probability predictions
on those OOD samples are produced by another pre-trained ResNet-50 [64] classifier.
Failure cases. As our method still gets 29% FPR95 on ImageNet-IIK, we provide some failure cases Fig. A6 which can be summarized into three kinds:
Noisy label, where the ID objects (e.g., dam) also exists in some OOD images from the test set. And the dataset composition may need a further examination.
Similar texture, shared by some OOD samples (e.g..flowwe) against ID images (e.g., starfishh, and the pre-trained encoders of CLIP are unable distinguish their features. Applying image-level spurious OOD syntheses (e.g., image attacks [35] 72 may reduce the texture-
bias.
Same background (e.g., sky) that seizes a large proportion of the image may lead to similar feature representations. Adopting image-level automatic masking techniques 2 68 synthesize spurious OOD samples may alleviate such problem.
Similar failure cases are also observed in recent SOTA methods, which reveal the unsolved challenges of OOD detection and suggest the potential directions for future works.
D Datasets and Baselines
For reproducibility, we present the details of datasets and baselines as follows.
ImageNet-100 100 (I). Following MCM 388 we take the randomly-sampled 100 classes from ImageNet-
IK 111 as the ImageNet-100 100 (I) subset, which contains the following categories: n03877845, n03000684,
0031 10669, H03710721, n02825657, n021 13186, n01817953, n04239074, n02002556, m04356056, 183187595, n03355925, n03125729,
m02058221, u01580077, 003016953, 002843684, n04371430,, n01944390, n03857697, n04037443, n02493793, n01518878, nl38440611,
n04179913, u01871265, 003866082, 103180011, n01910747, 003388549, n03908714, n01855032, n02134084, a03400231, n04483307,
n03721384, u02033041, n01775062, 002808304, n13052670, n01601694, n04136333, n03272562, n03895866, n03995372, n06785654,
102111889, a03444721, n03666591, 004376876, n03929855, n02128757. n02326432, n07614500, n01695060, n02484975, n02105412.
n04090263, n03127925, m|4550184, ndM606251, n02488702, n03444251,, n03363091, n02091635, n03457902, m02233338, n02483362.
nD4461696, a02871525,
601498041, #02107312, n01632458, n03394916, n041477833, n04418357, a03218198, n01917289.
n02102318, a020883644 n09K35506, n02095570, a03982430, m0M411544, n04562935, n03933933,, n01843065, m02128925, n02480495.
n03425411,, n03935335, n02971356, n02124075, n07714571, n031.3378,, m02097130. 132113799, 22213399,00339999 m09399592, m03594945.
ImageNet-1000 (II). Disjoint from ImageNet- (I), ImageNet-106 (IT) contains another 100 classes
randomly sampled from ImageNet-IK: m02096177, 103769881, #01629819, n04033995, n04357314, 002101388, n02328150.
20

---

Noisy Label
Similar Texture
Same Background
OOD Sample
ID Category
dam
starfish
flagpole
Figure A6: Failed OOD detections of our CATEX
m03729826, 002665020, 001985128, 002109525, N07715103, n02099429, n04517823, n02088632, n03207743, n03657121, n02948072,
n02 102106662, 001631663, n09229709, 003793489, n03776460, 007860988, n02129604 n03483316, n02107574, n07716358, n04208210,
102107908, af4372370, 102119912, n12144580, 601643334, n04548280, n03785016, n03535780, n03599486, nU2859443, nD4335433,
102110341, m03902225, n0M146614, n01774750, 8033147800 003045698, n01697457, n02869837, nU2276258, a04081281, a03956157,
a02487347, af4311174, 002084114, 004409515, #03022079, n03384352, n04532106, n02087394, nU4612504, nU2100583, 111939491,
n02107142, n01669191, 112998815, n04522168, n02894605, n03529860, n10148035, n01677366, n03775071, n03208938, n04238763,
n02363005, n02804414, n02106382, n03950228, 202128385, n02028035, n04099969, n02481823, n01729322, mf2939988, n02483708.
m04162706, n03857828, n02093647, n02927161, n03160309, n02840245, n03920288, n07871810, n04404412, n03947888, nB45094177
m02006910, n02256656, 20241 m02410509, n03584829.
ImageNet-2IK. The ImageNet-21K dataset on which we conduct the category-extended experiment the official winter 2021 released version For pre-processing, we follow Ridnik el al 42 clean invalid classes, allocating 50 images per class for validation, and crop-resizing all the images to 224 resolution. Training settings are the same as Sec.4 in our manuscript.
OOD datasets. Following the literature
50
60
38
we mainly consider subsets of
iNaturalist (53) SUN (65), Places (69) and Texture 35640 images in total.
the OOD datasets, which contains
Baselines. To evaluate the baselines on experiment settings, re-implement the most represen- tative and relevant methods, including MSP [21 18), Energy [33], VOS (15), and NPOS [50]. For fair comparison, we train all the baselines with NPOS's codebase and only fine-tune the last two transformer blocks of image encoder [50].
For MSP and Energy, we train single model with standard cross-entropy loss function for ID classification only, and infer with respective OOD metrics.
For VOS, we take the likelihood-based sampling strategy to generate spurious OOD synthe- ses, and train the model with uncertainty regularization as suggested [15]
For NPOS, take the non-pare conpaaametric distance- based sampling strategy generate spurious OOD syntheses, and train the model with open-set ERM as suggested [50].
htnpss//image-net.rr// hhapss/gittibbco//deepeeaii wisc/upos
21


Reviews:
[{"review_id": "fXRprN9SSx", "reviewer": "Reviewer_ufZG", "summary": "This paper presents a solution for the task of out-of-distribution detection with a hierarchical context. Specifically, it introduces the concepts of spurious context as negative descriptions to learn the distribution of unseen categories implicitly. When equipped with perceptual context, it can effectively detect OOD samples. Experimental results verify its robustness and effectiveness on several benchmarks.", "questions": "What is the masking ratio in Pertubantion Guidance? And is the performance sensitive to it?\n\nWhere will the spurious context locate? How about plotting them in Fig 4(a-b)?\nAlthough there has been an illustrative plotting in Fig.1, what would it be like in TSNE visualization?", "limitations": "No potential negative societal impact", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "- The proposed method is simple yet effective. Instead of finding a negative context for all categories, it proposes to adopt one for each category. Besides, the approach proposed to find negative samples is novel and intuitive.\n\n- The writing structure is clear and easy to follow. However, I find it hard to have an initial guess for the meanings of \"perceptual context\" and \"spurious\" context in the abstract section.\n\n- The experimental results are strong and solid. Besides, the ablation study in Tab.4 verifies the effectiveness of both proposed modules.", "weaknesses": "- The masking ratio in section 3.2, as a very important factor, hasn't been studied. A too-small ratio may lead to some false negative candidates while too-large ratios can neglect false positive candidates. From this perspective, this hyper-parameter can be very sensitive, thus influencing performance vastly.\n\n- I just doubt whether a single vector w^p_k for a spurious can model the complex category boundary. Modeling the interior region of a category is intuitive as in the ideal case it can model a quasi-hypersphere. But for OOD samples surrounding a category, the manifold can be more complex.\n\n- Fig.4 (a-b) as the main visualization, is a bit confusing. It's hard to tell what it aims to show.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are really encouraged that the reviewer recognizes our method to be simple, novel, intuitive, and effective.\n\nWe thank the valuable comments and insightful suggestions, and we hope our detailed responses below can address your concerns.\n\n> Q0: However, I find it hard to have an initial guess for the meanings of \"perceptual context\" and \"spurious context\" in the abstract section.\n\nThank for pointing out this. We did miss a concise and precise description in the abstract section. Specifically, we are planning to revise corresponding sentences into: \n\n\"Perceptual contexts perceive the inter-category difference (e.g., cats vs apples) for current classification tasks, while spurious contexts further identify spurious (similar but exactly not) OOD samples for each single category (e.g., cats vs panthers, apples vs peaches). \"\n\nWe hope such a revision can help readers get into our paper faster.\n\n> Q1: The masking ratio in section 3.2 hasn't been studied.\n\nThanks for the essential suggestion! In fact, in the supplementary materials we have demonstrated that perturbing (masking is one of the perturbing approaches) one word/token in the perceptual context is **effective** to guide the OOD syntheses in section 3.2. Moreover, we have conducted a series of ablations to study how the masking ratio influences the final OOD detection performance. Taking ImageNet-100 as the ID dataset, we learn 16 words/tokens for each category, and randomly perturb 1/16 to 16/16 words (the *classname* is always preserved) to guide the OOD synthesis for training. The results are shown below:\n\n| Mask Ratio | FPR95↓ | AUROC↑ |\n|:----------:|:------:|:------:|\n|    1/16    |  10.31 |  **97.82** |\n|    2/16    |  **10.27** |  97.81 |\n|    4/16    |  10.47 |  97.78 |\n|    8/16    |  11.02 |  97.73 |\n|    16/16   |  11.70 |  97.62 |\n\nIt indicates that masking 1~2 words/tokens is **effective enough** for perturbation guidance. Masking 4 or more words even leads to performance degradation, which means severely perturbed contexts may choose the noisy OOD candidates (e.g., random noise), making the learned spurious context unable to capture the true spurious OOD samples and further describe the category boundary.\n\nWe will add the ablation studies and discussions to the manuscript.\n\n> Q2: Whether a single vector $w_k^p$ for a spurious can model the complex category boundary. \n\nWe really appreciate the comment! We recognize that for each ID category (e.g., cat), the spurious OOD samples are diverse (e.g., panthers, lions, etc). Thus, we aim to only describe the spurious sample surrounding the ID category, rather than the whole OOD space (which is too complicated). According to common sense, it is intuitive to use more spurious contexts ($w_k^p$) to describe better category boundaries. To verify it, we have tested the number of spurious contexts for each ID category (taking ImageNet-100 as the ID dataset), and the results are shown below:\n\n| Spurious Context Number | FPR95↓ | AUROC↑ |\n|:-----------------------:|:------:|:------:|\n|            1            |  10.31 |  97.82 |\n|            2            |  **10.21** |  97.86 |\n|            4            |  10.27 |  **97.88** |\n|            8            |  10.25 |  97.86 |\n\nIt implies using more spurious contexts only leads to 0.1% gain on performance. The reason may be that the learned 2 or more spurious contexts are too redundant without any constraints. To alleviate this problem, we simply add an orthogonal constraint (making the similarities between each two spurious contexts close to zero), and the OOD detection performance is significantly boosted:\n\n\n| Spurious Context Number | FPR95↓ | AUROC↑ |\n|:-----------------------:|:------:|:------:|\n|            1            |  10.31 |  97.82 |\n|         2 + orth        |  10.17 |  97.86 |\n|         4 + orth        |   9.89 |  **97.89** |\n|         8 + orth        |   **9.76** |  97.84 |\n\nTherefore, how to effectively and efficiently leverage more spurious contexts to better describe the category boundary deserves further exploration, and we view it as our future work.\n\nWe will update the experiments and discussions.\n\n> Q3: Fig.4 (a-b) as the main visualization, is a bit confusing. It's hard to tell what it aims to show. Where will the spurious context locate? How about plotting them in Fig 4(a-b)? Although there has been an illustrative plotting in Fig.1, what would it be like in TSNE visualization?\n\nFig.4 highlights the advantages of our method against competitors, including:\n1. Fig.4 (a) indicates finetuning the model's encoder will damage the generalized feature space, making the unseen OOD images indistinguishable from seen ID images. On the contrary, Fig.4 (b) shows our method maintains the feature-level separability by freezing the whole encoders. \n2. Fig.4 (c-d) implies our learned spurious context (blue star in (d)) assists perceptual context (yellow star) in better distinguishing the ID and OOD samples. \n\n\nWe have revised the figure and caption in the uploaded PDF file (**Figure A3**), which may be more clear and comprehensible. Besides, visualizing the spurious contexts in Fig.4 (a-b) will make the pattern too complicated to highlight the necessity to freeze the generalized feature space. Thus, we turn to visualize the spurious context in Fig.4 (d) for clarity."}, {"review_id": "5jMlzgJJsz", "reviewer": "Reviewer_rNwN", "summary": "This paper proposes a framework for detecting out-of-distribution (OOD) samples using hierarchical description - perceptual and spurious contexts. Authors consider a category-extensible setup where categories can merge hierarchically. The proposed approach is evaluated by considering ImageNet as the in-distribution data and iNaturalist, SUN, Places, and Texture as OOD datasets.", "questions": "Please address the questions in the 'Weaknesses' section.", "limitations": "Yes", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. Considering image-text models to learn precise class boundaries is interesting and effective.\n\n2. Authors performed extensive experiments on relevant benchmarks.\n\n3. Results are state of the art.", "weaknesses": "1. It is not clear from the introduction what the hierarchical contexts are and how they affect OOD detection in a category-extensible way. Precisely, how do these contexts influence Fig. 1(a) to get updated to Fig. 1(b)?\n\n2. Authors talk about learning precise class boundaries. This is perhaps more suitable for novel class detection i.e., open-set learning than OOD detection. For example, the same class 'Car' can be in-distribution in a sunny environment and OOD in a rainy environment. How does the proposed approach address this scenario?\n\n3. Line 150-152: how does the combination of the contexts implemented? It is not clear from the description.\n\n4. section 3.2 and Fig. 3: how are the spurious samples generated and perturbed? Are the perturbations always random or informed by text cues?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are glad that the reviewer considers our method to be interesting and effective. \n\nHowever, we are sorry that you get confused with the mechanism and procedure of our method, due to our imperfect presentation. With the illustration figures in our uploaded PDF file, we hope the following responses can successfully address your concerns.\n\n> Q1: It is not clear from the introduction what the hierarchical contexts are and how they affect OOD detection in a category-extensible way. Precisely, how do these contexts influence Fig. 1(a) to get updated to Fig. 1(b)?\n\nIn our paper, the term \"hierarchical\" does NOT mean a taxonomic hierarchy, e.g., a fish node/category with two leave nodes/subcategories (goldfish and lionfish) in the ImageNet hierarchy. Instead, our \"hierarchical\" term refers to a mathematical/logical hierarchy, that is, first using the perceptual context to classify the samples into different categories (different colors in Fig.1 (a)), then using the spurious context for each category to further determine whether the sample truly belongs to this category or just comes from out-of-distribution (dark regions in Fig.1 (b)).\n\nAs illustrated in Fig.4 (c-d), the learned hierarchical contexts (i.e., perceptual and spurious contexts) are essential for distinguishing base/novel classes and ID/OOD samples in the category-extensible way. Besides, we have revised the figure and caption in the uploaded PDF file (**Figure A3**), which may be more clear and comprehensible.\n\n> Q2: Authors talk about learning precise class boundaries. This is perhaps more suitable for novel class detection i.e., open-set learning than OOD detection. For example, the same class 'Car' can be in-distribution in a sunny environment and OOD in a rainy environment. How does the proposed approach address this scenario?\n\nThanks for pointing out this problem. Indeed, in the early stage, the task definitions were confusing, but nowadays out-of-distribution detection (OOD) and open-set recognition (OSR) tend to be a unified scenario $^{[1][2][3]}$. Here is a brief summary of the two tasks.\n\nTraditionally, the OSR task aims at classifying samples from predefined classes and identifying the rest samples as unknown, while the initial OOD task only focuses on detecting and rejecting the unknown samples. However, modern OOD methods simultaneously take in-distribution classification and out-of-distribution detection into account. Hence, in today's community, OOD and OSR are exactly performing the same task, and the only difference lies in the evaluation settings $^{[3]}$. So you may simply view OOD and OSR the same scenario, and the precise class boundaries play the key role.\n\nBesides, in a general OOD detection setting, when defining the 'car' as an in-distribution category, a car in sunny or rainy environments will possess a large intra-class variance. This is indeed a challenging problem, and our solution is freezing the encoders of CLIP to maintain the generalized representation capability to overcome the variance. \n\nIf we misunderstood or have not addressed your concern, please feel free to inform us anytime.\n\n\n[1] Yang Jingkang. Generalized Out-of-Distribution Detection: A Survey. ArXiv, 2021.\n\n[2] Yang Jingkang. OpenOOD: Benchmarking Generalized Out-of-Distribution Detection. NIPS 2022.\n\n[3] Jun Cen. The Devil is in the Wrongly-classified Samples, ICLR 2023.\n\n> Q3: Line 150-152: how does the combination of the contexts implemented?\n\nAs for the combination of the perceptual and spurious contexts, the methodological mechanism is as we discussed in Q1, and the mathematical formulation please refers to Eq.(1-3) (for training) and Eq.(5) (for inference). For example, in Eq.(5), we use the perceptual context $w_k^p$ to compute the initial image-text similarity $s_k$ with the $k$-th ID category. Then the spurious contexts $w_k^s$ is adopted to calculate the regularization item $\\gamma_k$, which presents the probability that this input image truly belongs to the $k$-th ID category or comes from out-of-distribution. Finally, we combine the two contexts by multiplying the initial similarity $s_k$ by the regularization item $\\gamma_k$, to derive the final score $r_k=s_k \\times \\gamma_k$ for ID classification and OOD detection.\n\nBesides, we have illustrated the whole procedure in **Figure A2** in the uploaded PDF file. If there is still something unclear, please let us know and we are happy to answer your further questions at our best effort.\n\n> Q4: section 3.2 and Fig. 3: how are the spurious samples generated and perturbed? Are the perturbations always random or informed by text cues?\n\nWe first randomly generate a set of candidate samples in the image feature space, and then use the perturbed text features to select valid samples. The selected samples are so-called spurious samples. And indeed, the perturbations are always random, because during the training iterations, randomly perturbing is effective enough to guide the OOD sample generation, as illustrated in our supplementary material. \n\nFor clear illustration, we provide the **visualizations** of the initial OOD **sample generation**, our **perturbation guidance**, and further **training goals** in **Figure A1** in the uploaded PDF file. We hope it can help understand our method."}, {"review_id": "3VzzzX0P0S", "reviewer": "Reviewer_oU43", "summary": "This paper contributes a new method for OOD detection by learning the precise category boundaries, specifically, the category boundaries are defined by one perceptual context and one spurious context, these two contexts are learned text embeddings for a frozen CLIP model, the spurious context is learned by perturbing the perceptual context.\nExperiments on large scale OOD detection benchmarks show the effectiveness of the proposed method.\n", "questions": "1. I don't quite understand why the perceptual context is called hierarchical.\n", "limitations": "The limitations have been discussed in the paper.\n\n", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The experiments are comprehensive.\n2. Because the proposed method does not change the parameter the frozen CLIP model, it is shown that the proposed method is more generalizable under multiple category extended scenarios.\n3. The writing of the paper is clear.\n", "weaknesses": "1. The spurious context is learned by perturbing the perceptual context, and the perturbation is done by changing one word embedding, this could also be changing multiple word embeddings, but there are no ablations for this.\n2. There is one paper[R1] explore similar idea of spuriour context (termed reciprocal points in [R1]), which I think should be cited and discussed.\n\n[R1] Learning Open Set Network with Discriminative Reciprocal Points, ECCV 2020.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are really encouraged that the reviewer thinks our method makes sense and the experiments are comprehensive.\n\nWe appreciate your spot-on summary and constructive comments and suggestions, which we address below:\n\n> Q1: When learning spurious contexts, the perturbation could also be changing multiple word embeddings.\n\nThanks for the suggestion. In fact, in the supplementary materials we have demonstrated that perturbing (masking is one of the perturbing approaches) one word/token in the perceptual context is **effective** to guide the OOD syntheses in section 3.2. Moreover, we have conducted a series of ablations to study how the masking ratio influences the final OOD detection performance. Taking ImageNet-100 as the ID dataset, we learn 16 words/tokens for each category, and randomly perturb 1/16 to 16/16 words (the *classname* is always preserved) to guide the OOD synthesis for training. The results are shown below:\n\n| Mask Ratio | FPR95↓ | AUROC↑ |\n|:----------:|:------:|:------:|\n|    1/16    |  10.31 |  **97.82** |\n|    2/16    |  **10.27** |  97.81 |\n|    4/16    |  10.47 |  97.78 |\n|    8/16    |  11.02 |  97.73 |\n|    16/16   |  11.70 |  97.62 |\n\nIt indicates that masking 1~2 words/tokens is **effective enough** for perturbation guidance. Masking 4 or more words even leads to performance degradation, which means severely perturbed contexts may choose the noisy OOD candidates (e.g., random noise), making the learned spurious context unable to capture the true spurious OOD samples and further describe the category boundary.\n\nWe will add the ablation studies and discussions to the manuscript.\n\n> Q2: Reciprocal Points Learning (RPL)$^{[1]}$ explores a similar idea of spurious context, which should be cited and discussed.\n\nThanks for your constructive suggestion. We will add the discussion and cite it in the manuscript.\n\nThe similarity between our method and Reciprocal Points Learning (RPL)$^{[1]}$ and its sequel ARPL $^{[2]}$ is that we all explicitly learn the spurious contexts / reciprocal points beyond the ID category to deal with unknown samples in the open-world. As follows, we briefly discuss the difference from two main aspects:\n1. The **motivation** between our method and RPL/ARPL is different. RPL/ARPL aim at modeling the whole complementary feature space against a certain ID category for a visual classifier. On the contrary, our paper focuses on leveraging the pretrained vision-language models to capture the spurious OOD samples surrounding a certain ID category, and hierarchically describe the category boundary.\n2. The **approach** between our method and RPL/ARPL is different. RPL/ARPL are essentially learning a more compact in-distribution feature space by the constraints from reciprocal points, which are more similar to NPOS $^{[3]}$ and VOS $^{[4]}$. In contrast, our method freezes the model's parameters to maintain the generalized feature space, and learns the spurious contexts in a specially-designed prompt-tuning way.\n\n\n[1] Guangyao Chen. Learning Open Set Network with Discriminative Reciprocal Points, ECCV 2020.\n\n[2] Guangyao Chen. Adversarial Reciprocal Points Learning for Open Set Recognition, TPAMI 2021.\n\n[3] Leitian Tao. Non-Parametric Outlier Synthesis, ICLR 2023.\n\n[4] Xuefeng Du. VOS: Learning What You Don't Know by Virtual Outlier Synthesis, ICLR 2022.\n\n> Q3: I don't quite understand why the perceptual context is called hierarchical.\n\nWe are sorry for that. Actually, the perceptual context together with the spurious contexts are called hierarchical contexts.\n\nIn our paper, the term \"hierarchical\" does NOT mean a taxonomic hierarchy, e.g., a fish node/category with two leaves/subcategories (goldfish and lionfish) in the ImageNet hierarchy. Instead, our \"hierarchical\" term refers to a mathematical/logical hierarchy, that is, first using the perceptual context to classify the samples into different categories (different colors in Fig.1 (a)), then using the spurious context for each category to further determine whether the sample truly belongs to this category or just comes from out-of-distribution (dark regions in Fig.1 (b)).\n\nIn the manuscript, we will make a more rigorous definition of the term \"hierarchical\" with more appropriate descriptions."}, {"review_id": "wsH4ijjkby", "reviewer": "Reviewer_R97w", "summary": "The paper proposes a method to incorporate perceptual context and spurious context to handle the OOD detection problem. The experimental results seem quite promising.", "questions": "Kindly refer to  the weakness part", "limitations": "The authors can test that finetuning the encoders from the CLIP can produce what extent of the performance improvement for OOD detection.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The results of the proposed method is obviously better than previous  OOD detection methods;\n2.  The proposed method is quite interesting, which i think can be extended to other areas involving large-scale vision-and-language models.", "weaknesses": "1. The presentation of this paper is not very clear. For example, the authors mentioned \"label bias\" in the contributions part, but there is no explicit explanation about this so-called label bias, which I believe needs more clarifications;\n2. The motivations for the proposed hierarchical context mechanism is not very clear. Why a single context cannot produce a precise classification boundary? Moreover, the spurious samples w.r.t. a specific ID category can have large variance, so using a single spurious context can encode such large intra-class variance?\n3. It would be better if the authors can provide some visualizations of  the generated samples using the perturbation guidance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are very cheerful that the reviewer finds our work interesting and transferable to other areas involving large-scale VLMs. We hope the detailed responses below can address your concerns.\n\n> Q1: The presentation of this paper is not very clear. For example, the mentioned \"label bias\" in the contributions part needs more clarification.\n\nWe are sorry for that. The term \"label bias\" basically means the overconfidence in predicting unknown samples into known categories $^{[1][2]}$. To avoid ambiguity and reduce the burden of understanding, we will change the term \"mitigate label bias\" to the straightaway \"detect OOD samples\" in our contribution claim.\n\nIn addition, we will make a throughout proofreading to improve readability and rigorousness.\n\n[1] Anh Nguyen. Deep neural networks are easily fooled: High confidence predictions for unrecognizable images. CVPR 2015.\n\n[2] Boxi Cao. Knowledgeable or educated guess? revisiting language models as knowledge bases. ACL 2021.\n\n> Q2: The motivation for the proposed hierarchical context mechanism is not very clear. Why a single context cannot produce a precise classification boundary? Moreover, the spurious samples w.r.t. a specific ID category can have large variance, so using a single spurious context can encode such large intra-class variance?\n\nActually, a single perceptual context CAN produce a precise classification boundary in the closed-set, but CANNOT produce the precise category boundary in the open-set. As a result, the spurious OOD samples (similar to a certain in-distribution category) will lead to overconfidence predictions by the single perceptual context. Therefore, we propose to learn the spurious context to help perceptual context establish the precise category boundary.\n\nBesides, we recognize that for each ID category (e.g., cat), the spurious OOD samples are diverse (e.g., panthers, lions, etc). Thus, we aim to only describe the spurious sample surrounding the ID category, rather than the whole OOD space (which is too complicated). According to common sense, it is intuitive to use more spurious contexts ($w_k^p$) to describe better category boundaries. To verify it, we have tested the number of spurious contexts for each ID category (taking ImageNet-100 as the ID dataset), and the results are shown below:\n\n| Spurious Context Number | FPR95↓ | AUROC↑ |\n|:-----------------------:|:------:|:------:|\n|            1            |  10.31 |  97.82 |\n|            2            |  **10.21** |  97.86 |\n|            4            |  10.27 |  **97.88** |\n|            8            |  10.25 |  97.86 |\n\nIt implies using more spurious contexts only leads to a 0.1% gain in performance. The reason may be that the learned 2 or more spurious contexts are too redundant without any constraints. To alleviate this problem, we simply add an orthogonal constraint (making the similarities between every two spurious contexts close to zero, denoted as *+orth*), and the OOD detection performance is significantly boosted:\n\n\n| Spurious Context Number | FPR95↓ | AUROC↑ |\n|:-----------------------:|:------:|:------:|\n|            1            |  10.31 |  97.82 |\n|         2 + orth        |  10.17 |  97.86 |\n|         4 + orth        |   9.89 |  **97.89** |\n|         8 + orth        |   **9.76** |  97.84 |\n\nTherefore, how to effectively and efficiently leverage more spurious contexts to better describe the category boundary deserves further exploration, and we view it as our future work.\n\nWe will update the experiments and discussions.\n\n> Q3: It would be better if the authors can provide some visualizations of the generated samples using the perturbation guidance.\n\nThanks for such a constructive suggestion. We have provided the visualizations in **Figure A1** in the uploaded PDF file. We hope it can better illustrate our approach.\n\n> Q4: The authors can test that finetuning the encoders from the CLIP can produce what extent of performance improvement for OOD detection.\n\nWe have tested finetuning CLIP's encoders in the same category-extensible setting as Table 3 in our manuscript (separately training on two ImageNet-100 subsets (IN100-I, IN100-II), while directly testing on the merged ImageNet-200 (IN200)). Following NPOS $^{[3]}$, we only train the encoder's last two blocks, and the results are shown below. \n\n|          | IN100-I |        |        | IN100-II |        |        | IN200 |        |        |\n|:--------:|:-------:|:------:|:------:|:--------:|:------:|:------:|:-----:|:------:|:------:|\n| FineTune |   ACC↑  | FPR95↓ | AUROC↑ |   ACC↑   | FPR95↓ | AUROC↑ |  ACC↑ | FPR95↓ | AUROC↑ |\n|    No    |  94.12  |  10.31 |  97.82 |   94.42  |  7.91  |  98.31 | 89.61 |  13.13 |  97.19 |\n|    Yes   |  95.24  |  10.55 |  97.77 |   95.16  |  9.56  |  97.98 | 89.29 |  14.71 |  96.89 |\n\n\nSpecifically, encoder-finetuning can indeed improve the in-distribution classification accuracy by 1% in each subset, but the OOD detection performance (i.e., FPR95 and AUROC) decreases. Moreover, when testing on the merged union set, the model obtains a lower ID classification accuracy with worse OOD detection performance. It is consistent with our motivation that freezing the encoders of large-scale vision-language models is necessary to describe the precision category boundaries.\n\n[3] Leitian Tao. Non-Parametric Outlier Synthesis, ICLR 2023."}, {"review_id": "ElnPLMbSnP", "reviewer": "Reviewer_194m", "summary": "The paper presents a new technique to improve OOD, as well as IID, prediction for a pre-trained Language Vision Model (LVM). The heart of the proposed method is to learn both an ID context (i.e. perceptual context) and an OOD context (i.e. spurious context) to improve the classification of both ID classes and OOD samples. The paper proposes a new loss function to combine the ID and OOD losses together and a sampling strategy to produce spurious samples.", "questions": "1. How does the proposed method find optimal perturbations of the textual input? How does the method decide on which tokens or words to mask?\n\n2. How could the proposed method be used when you have no labeled data (i.e. a true zero-shot setting) to improve performance? Or can it? For example, can perturbations be included in at inference time – combined with the OOD scoring function – to do zero-shot labeling?\n\n3. What is the performance of the proposed method versus CoCoOP? While the paper does investigate the performance of its proposed method versus CoOp, and rightly concludes its does better with OOD, it does not evaluate the performance of its proposed method against the newer version of the CoOP method (i.e. CoCoOP) which was explicitly designed to deal with OOD issues of CoOP. \n\n\n-------- Following author's responses --------------\nI believe the authors have answered all three of my questions. In particular, I found their answer around 2, of how to use the proposed method to improve the zero-shot performance of LVMS, to be very interesting and notable. I also find it quite interesting that there was such a performance gap between the proposed method and CoCoOP. ", "limitations": "The authors have addressed nearly all of their limitations and addressed those dealing with societal impact. The only limitation they have not addressed is that the proposed method still requires labeled (or captioned) data in order to work and cannot work in a zero-shot setting as was the promise of CLIP. I welcome the author’s reply on this, as I am not sure if the proposed method couldn’t be used without labeled data. ", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The strengths of the paper are in its empirical validation and the originality around combining insights from different works. The proposed technique in the paper shows strong empirical performance, including ablation testing, on the main OOD tasks and measures. The quality of this validation well supports the claims made in the article about the need to consider both ID and some kind of spurious context when doing open-world detection. \n\nThe paper also combines elements from previous work, like the idea of having perturbed examples from VOS and NPOS, and the idea of learning prompts in the text vector space from Learning to Prompt into one framework.\n", "weaknesses": "Despite its strong empirical validation, the paper does have some weaknesses in its clarity and novelty. Beginning with novelty, the proposed technique of CATEX seems to be only an incremental improvement on NPOS (e.g., creating perturbed examples as part of training for OOD) and directly uses the technique from Learning to Prompt with only a change in the loss function. In essence, the paper doesn’t present any insight that the NPOS/VOS papers already presented, namely the inclusion of perturbed samples into learning for an LVM can help with OOD performance. If the paper were more explicit, especially in the methods section and discussion section, about how the proposed method differs from previous ones, it would help for establishing the novelty. For example, I believe both VOS and NPOS train the underlying CLIP model, while the proposed technique of CATEX uses the Learning to Prompt technique of training a lightweight layer on top of the CLIP model. Such a change seems to balance between being good at ID tasks, while not distorting the feature space.\n\nIn terms of clarity, there is not enough detail in the methods section to both deal with the aforementioned novelty issues as well as to fully understand the training process and the perturbation guidance. For the perturbation guidance, lines 174-176 make it sound like its changing the actual words or tokens (as is done in the Kwon et al. article with masked language modeling), versus the embeddings of the text, as is done in Learning to Prompt article. If the proposed technique is actually masking the tokens, versus changing the context vectors in the embed space, then how is the training done to optimize the text, as the method in Learning to Prompt only deals with a vector space? Also, does perturbation by masking tokens fully make sense? For example, is the perturbation of “a photo of a dog” to “a [MASK] of a dog” or “a photo of a [MASK]” really a meaningful perturbation for the spurious context? I wish there was something like a walkthrough example of the perturbation as well as some more explanation of the training method and intuition behind the perturbation to better understand the contribution of the work.\n\nFinally, there are a couple of areas where the writing could be improved. There are some sentences throughout that need to be proofread for grammar. For example, the last sentence of the abstract is a run-on sentence and the sentence on lines 48-50 is unclear in what its trying to say. \n\n-------- Following author's responses --------------\n\nI believe the authors have significantly addressed my concerns about the perturbation guidance. They have both added additional explanations as well as done some additional experimentation around ideas like how many tokens to mask.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are pleased that the reviewer recognizes our work‘s originality and empirical validity.\n\nWe appreciate your experienced comments and valuable suggestions, which are addressed below in detail:\n\n> Q1: No insights beyond NPOS/VOS. Clarifying the method difference helps establish novelty.\n\nThanks for your kind comment. As follows, we provide at least three new insights. \n1. **Large-scale pretrained VLM itself provides significant advances in OOD detection**. Huge corpora bring powerful feature representation capability, which is the key to detecting diverse unknown OOD samples beyond the limited ID datasets. Furthermore, massive paired image-text data builds comprehensive multi-modal prior knowledge, offering valuable extra information for OOD detection.\n2. **Previous methods have not fully utilized VLM's advances**. NPOS/VOS optimize a more compact ID feature space via **random** OOD syntheses, but the generalized feature space is distorted and the image-text prior knowledge is not utilized. Besides, vanilla CoOp lacks explicit descriptions for OOD samples, which hinders further OOD detection.\n2. **Our method leverages VLM's advances via hierarchical contexts**. Freezing CLIP's encoders to maintain the generalized feature space, we learn two contexts (i.e., perceptual and spurious) to hierarchically describe each ID category's boundary. Moreover, to exploit VLM's prior knowledge, a novel textual-perturbation-guided approach is developed to generate OOD samples to learn the contexts.\n\nHence, this paper is specifically designed for OOD detection with VLMs, but not an incremental improvement on NPOS.\n\n> Q2: About training process and perturbation guidance. How to optimize the text? Does the perturbation of “a [MASK] of a dog” or “a photo of a [MASK]” makes sense?\n\nThanks for your comment. First, in the training procedure, the perturbation itself only serves to generate OOD samples and no gradient is involved. Detailed diagrams (**Figure A1-A2**) are provided in the uploaded PDF file to illustrate the whole training procedure.\n\nSecond, rather than the initial prompt template as \"a photo of a\", the perturbation is applied to the tuned contexts [V1][V2]...[Vm]+[CLS], where [CLS] is the fixed class name (e.g., dog or goldfish). As each word [Vi] has a certain semantic meaning $^{[1]}$, let's view the learned contexts as (even though not) \"yellow fish long flowing tail\" for [CLS] as [*goldfish*]. Thus, the perturbations of \"[MASK] fish long flowing tail\" and \"yellow fish long flowing [MASK]\" both make sense, which eliminate some kinds of visual features of [*goldfish*], and generally describe another spurious OOD category against the certain ID category. \n\n[1] Kaiyang Zhou. Learning to Prompt for Vision-Language Models, IJCV 2022.\n\n> Q3: The writing can be improved. Lines 48-50 are unclear.\n\nThanks to this, we have made a throughout proofreading to improve readability.\n\nIn particular, the core idea conveyed by lines 48-50 is that even though fine-tuning CLIP's encoder may boost the performance (e.g., classification accuracy on ImageNet), CLIP's generalized feature space is destroyed. So test data shifts (e.g., ImageNet-R) will cause severe performance degradation.\n\n> Q4: How to find optimal perturbations of the textual input?\n\nA thoughtful comment! Indeed, deciding the optimal word to perturb is challenging, especially when the learned word embeddings do not correspond to actual words in natural language. \nThough, we have made a simple step to find the \"optimal\" word/token in the embedding space. After computing the prototype vector by averaging the 16 learned words, we take the most distant (denoted as *MaxDist*) or closed (denoted as *MinDist*) word for perturbation guidance. However, as shown below, the results even get worse.\n\n|Method|FPR95↓|AUROC↑|\n|:----:|:----:|:----:|\n|MaxDist|10.42|97.75|\n|MinDist|10.86|97.73|\n|Random|**10.31**|**97.82**|\n\nSo, we argue that randomly choosing the words to perturb at each iteration is still an effective way. After several iterations, we can statistically perturb every word to guide the spurious sample generation, covering the optimal situation.\n\n> Q5: How to deploy without labeled data? Can perturbations be used to do zero-shot labeling? \n\nAn insightful suggestion! Our method actually CAN be used for zero-shot labeling. For ImageNet-1K, since CLIP's default prompt \"a photo of a [CLS]\" contains no visual info, we adopt the visual description from LLM $^{[2]}$ as the initial prompt input (denoted as VisDesc). Then, we randomly mask the visual description to simulate spurious contexts, and adopt Eq.(5) in our manuscript to regularize image-text similarities for classification. The results shown below indicate our method boosts zero-shot classification without any training cost.\n\n|Method|Prompt|ACC-Top1|ACC-Top5|\n|:----:|:------:|:-----:|:----:|\n|CLIP|A photo of a goldfish.|63.50|88.99|\n|VisDesc|A yellow fish with a long flowing tail is goldfish.|65.47|90.14|\n|Ours|(plus) A [MASK] fish with a long [MASK] [MASK] is goldfish.|**65.83**|**90.36**|\n\nIt provides another new insight that in the category-extensible setting or a true zero-shot classification scenario, explicitly constructing spurious contexts can perform the one-class OOD detection task, as other categories can also be viewed as OOD for a certain ID category. \n\n[2] Sachit Menon, Visual Classification via Description from Large Language Models. ICLR 2023.\n\n> Q6: Comparison to CoCoOP?\n\nOn ImageNet-100, CoCoOp is surprisingly much worse than our method. The reason may be that during training, CoCoOp only takes ID images as context conditions, while neither OOD samples nor OOD contexts are involved. Thus, when employed in the open-world and asked to reject OOD samples not belonging to any ID category, CoCoOp failed. The image conditions even aggravate the overconfidence in OOD samples.\n\n|Method|ACC↑|FPR95↓|AUROC↑|\n|:----:|:--:|:---:|:----:|\n|CoCoOp|92.90|39.22|92.69|\n|Ours|**94.12**|**10.31**|**97.82**|"}]

Author rebuttal:
We thank all the reviewers for their time, insightful suggestions, and valuable comments. We are encouraged to see **ALL** reviewers find our method **interesting** and **effective** (ufZG, rNwN, oU43, R97w, 194m), **comprehensive** experiments and **promising** results (ufZG, rNwN, oU43, R97w, 194m), and **transferable** to other areas involving large-scale vision-language models (R97w).

However, reviewers still have some extra concerns, which mainly focus on the following aspects:
* insufficient presentation on the term definition, method operation, and novelty clarification;
* lack of additional experiments and ablation studies;
* extension and combination with zero-shot applications, future explorations, etc.

We respond to each reviewer's comments in detail below, and here is a brief summary of the main discussions and experiments:
* more precise explanations of our proposed perceptual context, spurious context, hierarchical mechanism, etc;
* walkthrough examples to illustrate the perturbation process, visualize the generated samples, and demonstrate the full training procedure (in the **uploaded PDF file**);
* extensive ablation studies and discussions on the mask ratio for Perturbation Guidance, the number of learnable spurious contexts, as well as comparison and discussion with relevant works like RPL and CoCoOp;
* new insights and validations for zero-shot applications, including the combination with large language models (like GPT).

We thank the reviewers' valuable suggestions again, and we believe those make our paper much stronger.

Meta-review: This paper was reviewed by five experts in the field. The authors' rebuttal resolved most of the concerns. Reviewers liked the overall proposed method and comprehensive experiments.

The AC agrees with the reviewers' assessments and believes the paper provides a simple yet effective method for OOD detection using vision-language models. The decision is to recommend the paper for acceptance. The reviewers did raise some valuable suggestions in the discussion that should be incorporated in the final camera-ready version of the paper (e.g., improving paper presentation and clarity). The authors are encouraged to make the necessary changes to the best of their ability.

Why not spotlight / oral: The current paper still requires further refinement in its final version to incorporate reviewers' suggestions.
Decision: Accept (poster)