On the Powerfulness of Textual Outlier Exposure for Visual OoD Detection
Authors: Sangha Park
Jisoo Mok
Dahuin Jung
Saehyung Lee
Sungroh Yoon
Venue: NeurIPS 2023 poster (2023)
Keywords: Out-of-distribution detection
TL;DR: We present textual outliers in diverse forms and demonstrate their effectiveness in visual out-of-distribution (OoD) detection tasks

Abstract: Successful detection of Out-of-Distribution (OoD) data is becoming increasingly important to ensure safe deployment of neural networks. One of the main challenges in OoD detection is that neural networks output overconfident predictions on OoD data, make it difficult to determine OoD-ness of data solely based on their predictions. Outlier exposure addresses this issue by introducing an additional loss that encourages low-confidence predictions on OoD data during training. While outlier exposure has shown promising potential in improving OoD detection performance, all previous studies on outlier exposure have been limited to utilizing visual outliers. Drawing inspiration from the recent advancements in vision-language pre-training, this paper venture out to the uncharted territory of textual outlier exposure. First, we uncover the benefits of using textual outliers by replacing real or virtual outliers in the image-domain with textual equivalents. Then, we propose various ways of generating preferable textual outliers. Our extensive experiments demonstrate that generated textual outliers achieve competitive performance on large-scale OoD and hard OoD benchmarks. Furthermore, we conduct empirical analyses of textual outliers to provide primary criteria for designing advantageous textual outliers: near-distribution, descriptiveness, and inclusion of visual semantics.

On the Powerfulness of Textual Outlier Exposure for Visual OoD Detection
Sangha Park', Jisoo Mok', Dahuin Jung', Saehyung Lee', Sungroh Yoon 122, 'Department of Electrical and Computer Engineering, Seoul National University IInterdisciplinary Program in Artificial Intelligence, Seoul National University (wiarae, magicshopl118, annajung0625, halo8218, sryoon]0smu. ac kr
Abstract
Successful detection of Out-of-Distribution (OoD) data becoming increasingly important to ensure safe deployment of neural networks. One of the main chal- lenges in OoD detection is that neural networks output overconfident predictions OoD data, make it difficult to determine OoD-ness of data solely based on their pre- dictions. Outlier exposure addresses this issue by introducing an additional loss that encourages low-confidence predictions on OoD data during training. While outlier exposure has shown promising potential in improving Ool detection performance, all previous studies on outlier exposure have been limited to utilizing visual outliers. Drawing inspiration from the recent advancements vision- lsion-language pre-training, this paper venture out to the uncharted territory of textual outlier exposure. First, we uncover the benefits of using textual outliers by replacing real or virtual outliers in the image-domain with textual equivalents. Then, we propose various ways generating preferable textual outliers. Our extensive experiments demonstrate that generated textual outliers achieve competitive performance large-scale OoD and hard OoD benchmarks. Furthermore, we conduct empirical analyses of textual outliers to provide primary criteria for designing advantageous textual outliers: near-distribution, deseriptiveness, and inclusion visual semantics. Code is available htttpss:/ / thub. com com/wiarae/TO:
Introduction
The standard assumption when deploying neural networks is that both the train and test data will fall under the same distribution, in which case, test data are considered to be in-distribution (ID). However, in the real world, they often encounter out-of-distribution (OoD) data, which refers to data that are distant from the training data distribution. Because the generalization capacity of neural networks does not extend outside the distribution of observed train data, OoD data cause them to perform poorly or even fail [I]] Therefore, it is important for neural networks to not only achieve high accuracy ID data but also successfully detect Ool) data preclude them from the inference process. Unfortunately, neural networks yield highly over-confident predictions on OoD data, making OoD detection non-trivial research problem (39 43
One promising approach to OoD detection is "outlier exposure," which modifies the training procedure of neural networks [23 18), such that they can accurately classify ID data while also reliably detecting OoD data. To simultancously accomplish the two training objectives, outlier exposure utilizes proxy data that can emulate actual OoD data that the neural network may confront test time. Outlier exposure then defines a regularization term that encourages low-confidence predictions on those proxy data. Finally, the neural network is optimized to minimize the regularization term in addition
*Corresponding author
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

to the supervised loss. The major challenge in outlier exposure is providing proxy data for training without explicit knowledge of OoD data.
Previously when neural networks could only process single-modal data, outlier exposure could only rely on visual outliers from the image domain. The recent emergence of multi-modal neural networks [37[21] has opened new research opportunities. In this work, we study the powerfulness of textual outlier exposure by utilizing multi-modal neural networks. We first conduct preliminary studies of textual outlier exposure by replacing visual outliers with their textual counterparts in two widely-studied outlier exposure methods: real and virtual outliers. Our empirical results reveal that visual outliers suffer from high performance variation according to the choice of an auxiliary dataset and non-negligible time consumption. Textual outliers, on the contrary. do not experience the above problems, demonstrating that they can serve as attractive proxy data for outlier exposure.
Textual outliers can be designed in various forms, from single words detailed descriptions. Initially, it may be unclear which type of textual outlier is most effective for facilitating textual outlier exposure. To address this ambiguity, we explore textual outliers three verbosity levels: word, description, and caption. These textual outliers are generated using large-language models (LLMs) such as GPT-3, BERT, or vision-language models (VLMs) like BLIP-2. Among generated outputs, only those that can be considered OoD are utilized for outlier exposure. Our textual outlier generation methods rely solely on images and class labels of ID data. As result, utilizing generated textual outliers eliminates the need to collect and curate proxy data to perform outlier exposure.
Our comprehensive experimental results present compelling evidence that the proposed textual outlier approach outperforms existing methods for OoD detection. Specifically, compared to the method in Hendryeks et al. 18] that employs real auxiliary datasets of outliers, our caption-level textual outlier reduces the FPR95 from 73.80% to 58.21 %. yielding a direct improvement of 15. 5,59%p on the challenging ImageNet-IK IK benchmark (6). Additionally, our textual outlier approach surpasses advanced baselines in visual outlier exposure, demonstrating the advantages of the outlier synthesis approach in the textual space. Furthermore, through comparative analyses of visual versus textual outliers and different forms of textual outliers, we establish three key criteria for advantageous textual outliers: dietriistributoon, descriptiveness, and inclusion of visual semantics.
Our contributions are summarized as follows:
This is the first work to investigate the potential of textual outlier exposure with multi- modal neural networks. We reveal that textual outlier exposure can outperform its visual counterparts and overcome inherent limitations associated with them.
We propose to utilize LLMs or VLMs to generate diverse drafts of textual outliers at different verbosity levels: word, description, and caption. Generative models facilitate the convenient and cost-effective collection of potential texts suitable for formatting into textual outliers. The generated outputs are then refined into more effective forms for textual outlier exposure.
Proposed textual outliers are validated in various OoD detection settings through compar- ison with visual outlier exposure baselines. In particular, description-level textual outlier outperforms a competitive baseline from visual outlier exposure by 38.60 38.60%p (AUROC) on a challenging detection scenario with semantically similar classes.
2 Problem Setup
Background and Notations. Let X' represent the input space, and y = (1...., C) denote the label space. In our framework, we consider two main datasets: the ID dataset, DID, and the OoD dataset, DOOD. The DID = {x4, y:) } 1 is set of every sample from ID and is defined over X' x V. On the other hand, Do.D is defined solely over the input space X' and does not have any associated labels. During training, the neural network never observes the OoD dataset. In the inference process, the neural network should be able to reliably filter out OoD samples.
Out-of-distribution detection can be formulated as a binary classification problem. Given a classifier f: X - R trained on samples in subset of DID, the objective is to design binary function estimator,
in, if if f) 2 Y if f) <>
9(x)
(1)
out,

---

(a) Training Phase
(b) Testing Phase
Images
ship
ID Images
Image Encoder
Linear Classifler
LE
s(x) 2
Linear Classifier
Modality
Image Encoder
"a photo ses" ta cargo" "a photo blue" photo water"
Gap
Text Encoder
LoF
S(x) <
Proxy of
Images
Figure I: Framework overview, (a) During the training phase, we utilize textual outliers embeddings as proxies for test OoD data. For the ID image data, we train the model using cross-entropy loss, while for the textual outlier data, we employ outlier exposure loss, which enforce the model to output the low-confidence predictions on them. To reduce the modality gap between the two domains, we inject noise to textual outlier. (b) During the testing phase, the trained classifier filters out OoO data by thresholding output score.
that classifies whether sample E X' belongs DID or not. This is determined by the score function S(x, f), which is defined based on the classifier's output. The threshold parameter is typically chosen to ensure high percentage (e.g., 95%) of correctly classified nn-distribution samples.
Evaluation Metrics. The OoD detection performance can be assessed by using the following two evaluation metries: (1) the false positive rate of Ool) samples at fixed positive rate of 95% (FPR95) for ID samples, with lower values indicating better performance, and (2) the area under the receiver operating characteristic curve (AUROC), with higher values indicating better performance. Additionally, we report the accuracy of the ID classification.
Framework Overview. CLIP 37 is large-scale vision-language model that has been trained vast dataset consisting of 400 million image-text pairs, collected from the internet. Its remarkable performance has been demonstrated across various tasks, leading to numerous studies exploring its
transferable feature representation (59 13 Several previous works have shown improvements in
diverse tasks by operating in the CLIP space with the frozen CL.IP encoder [42
14
15
9
35 1.
Likewise, we leverage the strength the CLIIP encoder to build a reliable flssifier by training a linear classifier on top of it. The CLIP embedding space that jointly models images and texts enables usage of textual cues for visual OoD detection. Therefore, we employ textual, instead of visual, outliers to regularize the classifier; train data for the supervised loss and data, both ID and OoD, remain defined in the image domain. The general framework for textual outlier exposure is illustrated in Figure
3 Understanding the Potential of Textual Outlier
Prior to exploring various forms of textual outliers, we first show that even naïve approaches to textual outlier exposure can overcome the inherent limitations of its visual counterpart. We reveal the advantages of using textual outliers over images by replacing images with texts in two representative types of outlier exposure: (1) the use of real auxiliary datasets and (2) virtual outlier synthesis in the feature space. The empirical results demonstrate that textual outliers can achieve superior OoD detection performance, as well as significantly reducing the variance of performance and time consumption in (1) and (2). In this section, all of the experiments are conducted following the procedure Figure
3.1 Auxiliary datasets
Experimental setting. Hendrycks et al. [18 proposed to utilize real auxiliary datasets as proxy data for outlier exposure improve the detection performance. We leverage four auxiliary datasets to evaluate visual and textual outliers [ To use texts instead of images, we adopt class labels from auxiliary dataset as outliers, e.g., "a photo of bamboo forest" when SUN [52) is used as an

---

Table 2: Results of using images vs. texts (class labels) of auxiliary datasets as outliers. We compare the results of utilizing 4 different auxiliary datasets. In the testing phase, we use ImageNet20 and ImageNet10 as ID and OoD datasets, respectively. The best result in each column is in bold.
Auxiliary Dataset
Texture
Places
SUN
iNaturalist
Average
Metrics
FPR AUROC FPR AUROC FPR AUROC FPR AUROC ID Acc
FPR
AUROC
None
95.2
21.49
96.27
Image
29.80
95.96
13.60
97.53
18.60
96.89
36.20
93.34
95.6
24.55
10.05
95,93 1.84
Text
19.60
97.23
15.40
97.37
15.00
97.37
13.40
97.28
95.9
16.10
3.12 97.25
auxiliary dataset. To mitigate the imbalance in size between image and text datasets, we, following the experimental setting in Zhang et al. [57]. train the model for 300 epochs with textual outliers and 25 epochs with visual outliers. We use ImageNet20 as ID data and ImageNet10 as OoD data for
evaluation as proposed in Ming et al. [32].
Results. Table 2 presents comparison of the detec- tion performance and classification accuracy between visual and textual outliers. Notably, textual outliers demonstrate superior performance across all evaluation metrics. Furthermore, the detection performance of models trained with visual outliers is highly contingent on the choice of an auxiliary dataset, leading to signifi- cant performance variation. This poses considerable challenge in determining the optimal dataset
Table 1: Number of images / texts (class
labels) in auxiliary datasets.
Textures Places
SUN iNaturalist
Image
5640
10000
10000
10000
Text
47
50
50
110
for effective outlier exposure. contrast, the use of textual outliers noticeably reduces performance variation. Additional results on other benchmark datasets can be found in the Appendix C.I.
3.2 Synthesis in feature space
Experimental setting. As an alternative to using auxiliary datasets, virtual outlier synthesis in the feature space, which does not rely on real datasets, has been proposed 88 46). The feature space of classifier can be represented as a Gaussian reason-
Table 3: AUROC achieved after
varying amounts of time consump- tion for virtual outlier synthesis.
mixture model with the class mean and variance. able to consider instances sampled from the low-likelihood regions of this distribution as outliers. To obtain means and variances per class using text instead of images, we adopt class labels, synonyms class labels, and descriptions of classes obtained from GPT-3 (2). Examples of labels and associated synonyms and descriptions are provided in the Appendix C.2. We choose VOS 8 from a family of methods in outlier synthesis the testbed to compare virtual outliers. For both textual and visual outiers, we utilize the CLIP embedding space to obtain class means and variances. We train the linear classifier with 1etgeNet and use ImageNet20 OoD dataset for evaluation 32 The number of training epochs is set to 300 for both images and texts.
Time
18m
30m
3h 54m
Image Text
91.65
96.59
97.77
97.85
Results. Our findings, as shown in Table 3, indicate that utilizing class means and variances derived from texts for outlier synthesis outperforms those derived from images, while also demonstrating better sample efficiency. The majority of computational overhead in virtual outlier synthesis arises from computing the updated class mean and variance after each epoch to generate a new set of virtual outliers. Notably, when using text data, the training time is significantly reduced by factor of 13 (18 minutes vs. approximately 4 hours), while still achieving comparable performance. When we reduce the amount of image data match the synthesis and training time for textual outliers, the OoD detection performance experiences a considerable decline from 97.77 to 91.65 in terms of AUROC.
3.3 Analysis
The superior performance of textual outliers can be attributed to their proximity to ID data. To analyze the relationship between ID data and different outliers, we utilize UMAP (30, popu- lar technique for dimension reduction. In Figure 2 (a), we visualize the latent space that con-

---

tains ID data and two types of real auxiliary datasets, Texture 4 and iNaturalist 47 They are the two auxiliary datasets whose textual outliers led to greatest performance improvements. Similarly, Figure 2 (b) shows the latent space representing three types of data: ID data, class- wise means of ID data both image and text domains, and outliers sampled from each do- main's distribution. Textual outliers, whether real or virtual, lie closer to ID data compared to their visual counterparts. This observation implies that textual outliers, which reside in the vicinity of ID data distribution, can offer informative signals about boundary regions g 33
As a result, the usage of textual outliers contributes to improving OoD detection performance by assisting the neural net- work in modeling the ID-OoD boundary.
Auxiliary dataset
(b) Synthesis
Image
Teot
Outterr
Outters from Test
Based on these empirical findings, we conclude that exposing the neural net- work to textual outliers offers significant advantages, including robustness to per- formance variation and enhanced sample efficiency. In the subsequent sections, we delve into potential designs for tex- tual outliers (Section 4 and analyze the factors that contribute to their effective- ness in outlier exposure (Section 5)
Tealures (imaga)
Textures (txx)
Platuraisi (mage)
Posturwisi (weal)
Figure 2: Feature space analysis of image and text outliers in (a) auxiliary datasets and (b) synthesis settings.
4 Method
Unlike visual outliers, textual outliers can take various forms, ranging from a single word to more verbose descriptions. We investigate three different designs for textual outliers: 1) words retrieved from ID images, 2) descriptions of ID class-labels, and 3) captions of ID images generated by an image captioning model. Figure visualizes the generation process for each type of textual outlier. We utilize LLMs and a VI.M to generate textual outliers and selectively employ the generated outputs that are OoD but still lie adjacent the ID data. We highlight that additional data except for II) data is used to generate textual outliers.
4.1 Generating textual outliers in three types
4.1.1 Word-level textual outliers
(i) Motivation. The most straightforward way of designing textual outliers is in the form of photo of {word)," " which we coin word-level textual outliers. As discussed Section 3.1 word-level textual outliers can generated by simply leveraging class labels or comparable terms from auxiliary datasets, but they may not always lie in the proximity of ID data. In Section 3.3 we have observed that textual outliers that are closer to the ID data tend to provide more informative signals. To obtain word-level textual outliers that are near the ID data, we utilize image-based text retrieval. Utilizing texts that close images enhances the classifier's ability to capture the ID-OoD boundary, thereby improving its performance in distinguishing between ID and OoD data.
(ii) Generation. We exploit the framework proposed by Es- maeilpour el al. 10 to perform text retrieval from ID images. The retrieval process consists of the following steps. First, we use the CLIP image encoder to extract image embeddings of ID data. These embeddings are then treated as sequences of tokens and fed into the multi-head cross-attention head of text decoder. Finally, the result- ing text embeddings are linearly projected onto the text space. Figure 3 (a) provides high-level overview of the retrieval process. Images from the validation set of the ID dataset, which does not overlap with the test set, are used inputs.
Table 4: Templates for prompt en- sembling with textual outliers.
"A photo of O
"A blurry photo of O
"A photo of many O)"
BERT, which functions
"A photo of the large O}"
"A photo of the small O
(iii) Filtering. Initial versions of generated text outputs are not always guaranteed to be OoD. Therefore, we select a subset of retrieved words based on their cosine similarity to the ID image
5

---

(a) Word-level Textual Outlier
Image Enooder
[CLIP)
[BERT] Text Decoder
Linear Projection
+ Top-k For Animall Tiger Noze Feline
Kammal)n (Cless Labels)
Images from
Validation
(b) Description-evel Textual Outlier
(c) Caption-level Textual Outlier
Step 1. Caption generation
Step 2. Outter caption selection
What useful features for diatineuishing
Blanese with
Inlier
in a photo?
Category Names from ID Validation
[sianese cat]
Slue eyea sitting
on the floo="
8 Blanese cat with blue eyes sittiing fioor"
[BLIP-2]
[GPT-3]
blue eyes
long, slender body
Images Validation Set
case lying on couch.
Slamese Cat
on a couch."
lying
shozt
Outller
Figure 3: Illustration of (a) word-level, (b) description-level and (c) caption-level textual outlier generation processes. A different generative model is used for each type of textual outlier. In (a), CLIP and BERT are used to perform image-based text retrieval to create word-level textual outliers. In (b) and (c), description-level and caption-level textual outliers are generated from GPT-3 and BLIP-2, respectively. For word- and caption-level textual outliers, the generated outliers are refined for the purpose of outlier exposure. See Section for details.
embeddings. Only those with top-k to k + 8 similarity values selected, and afterwards, class labels are explicitly removed. Through this filtering process, we can effectively keep texts that are OoD but near ID data. To prompt CLIP with selected word-level textual outliers, employ prompt ensemble using the templates outlined Table
4.1.2 Description- textual outliers
(i) Motivation. We now consider generating textual outliers from class labels of ID data, instead of images. With LLMs, class labels can easily be expanded into illustrative descriptions that include linguistic semantics; we name these description-leve textual outliers.
(ii) Generation. We utilize GPT-3, powerful and widely-adopted LLM, obtain class descriptions. GPT-3 is prompted with the phrase "describe what the (class name} looks like" (31]
(iii) Filtering. These class descriptions can be utilized textual outliers without further refinement because they already are OoD from CLIP's viewpoint. For instance, when we prompt CLIP with description of Siamese cat (e.g.. "a photo of light-colored fur with dark points the face, ears, legs, and tail"), it retrieves images that noticeably deviate from the actual visual characteristics of Siamese cat. An illustrative example demonstrating this phenomenon Appendix E. Therefore, we employ the class descriptions obtained from GPT-3, after excluding the corresponding class labels, as textual outliers and prepend templates in Table to each description.
be found in the
4.1.3 Caption-level textual outliers
(i) Motivation. To generate descriptive textual outliers that also contain visual semantics, we propose to reformulate captions into textual outliers. The so-called caption-level textual outliers, generated with visual cues, can capture the complex visual semantics in images and also contain rich linguistic semantics, encompassing the benefits of word- and description-level textual outliers.
(ii) Generation.
To acquire set of captions, denoted S {s1, 82; ...; S.. ). utilize BLIP-2 25 known for its exceptional descriptive capabilities [40]. Images of the validation set of the ID data are used inputs to BLIP-2. However, some of the captions that resemble the ID samples too closely are ineffective for outlier exposure.
(iii) Filtering. To filter out such ineffective captions, utilize the Mahalanobis distance M(s)
24
as a measure of closeness to ID data. The Mahalanobis distance computed as follows:
M(s) max (-(beexc(s) Pc)) T -pe)).
(2)

---

where j and Σ denote class-wise mean and variance of the training set. Pa and Σ are derived using the following equations:
Σ
Mimago xi);
Σ
(Mimage(x)) ) (Himagge (xi) iáo) Aggi=
(3)
No
N
where (-) refers to the CLIP image text encoder. We only consider the top-p% of the data with the largest M(s).
4.2 Optimizing ID embeddings with textual outlier
We train our classifier using the loss as defined below:
L(f) (xx))>DDI [CEE (f (himage (x)): y)] + AE gDDD
(s))
(4)
out
where the left term is cross-entropy loss and the right term is Kullback- divergence to the uniform distribution, which can be expressed as - E. softmax.f())) Douf refers to the generated textual outliers. Our loss aims to increase the classification performance train data and reduce the confidence on DOE We add Gaussian noise to the textual outliers to reduce the modality gap [36] out
4.3
Test time OoD detection
Ex(; (x; where When testing, we use Energy score 29 for OoD detection S(x, f) -
E( - log
Σ
(5)
Energy score is known to be less susceptible to the overconfident issue. It is worth mentioning that the sign of the energy function, denoted as E(x). is reversed align with the convention where samples with higher scores are classified as ID, while samples with lower scores are classified as OoD, Our method can be used with any OoD score that utilizes output probabilities.
5 Experiments
In this section, we conduct extensive experiments to showcase the effectiveness of our novel technique for textual outlier exposure. Through comprehensive experiments, we evaluate and investigate our approach from three main perspectives: (1) comparison of our method with existing methods in outlier exposure, (2) comparison of performance among proposed textual outliers, and (3) assessment of the proposed textual outlier exposure in challenging OoD detection scenarios.
We use the large-scale ImageNet- IK 66 OoD detection benchmark proposed by Huang et al. 20 We conduct experiments on four OoD datasets: subsets of iNaturalist 47,, SUN (52).
Datasets.
Places [58) Texture
4 The categories of OoD datasets are disjoint from those of the ID dataset.
Experimental setting. The CLIP model used in this paper is adapted from OpenAl's public repository, with ViT-B/16 serving as the default vision and language backbone. For word-level textual outliers, the BERT 7 large model with 24 layers and hidden size of 1024 from huggingface [51 is used.
The model is trained on the MS-COCO (88) dataset using the Adam optimizer [22 with a fixed learning rate 10- for 100 epochs. A simple teacher-foreing algorithm is employed retrieve text during the training process. To generate description-levé textual outliers, the GPT-3-text-davinci-OU0 model is used. For caption-level textual outliers, BLIP-2 opt 2.7b from huggingface is used. The Adam optimizer is used to train our linear classifier for OoD detection, and batch size, learning rate, and other hyperparameters are tuned on the validation set. We use 0.5 for in our training loss, and batch size for both ID and train time outlier is 32. The temparature value T for Energy score is set to as per the original paper. Model checkpoints with the highest validation accuracy are evaluated on the test set. We employ value of 30 for k, which is used in word-level outliers filtering, and a value of for 5. Furthermore, filtering ratio 15 is utilized as the p for caption-level outlier analysis. The ablation study examining these parameters can be found in the Appendix B.
7

---

Table 5: Comparison of proposed textual outliers and competitive baselines from visual outlier
exposure on the ImageNet- IK dataset. The best result in each column is in bold.
OoD Data
INaturalist
SUN
Places
Texture
Average
Metries
FPR95 AUROC FPR95
AUROC FPR95
AUROC FPR95 AUROC ID Ace FPR95 AUROC
Visual OE
OE
78.31
75.23
80.10
76.55
70.41
81.78
66.38
82.04
75.51
73.80
78.90
CSI
75.85
82.63
90.62
47.83
94.90
44.62
85.85
87.11
74.27
86.80
65.54
SSD+
59.60
85,54
75.62
73.80
83.60
68.11
39.40
82.40
78.80
64.55
77.46
MixOE
80.51
74.30
74.62
79.81
84.33
69.20
58.00
85.83
74.62
74.36
77.28
VOS
94.83
$7.69
98.72
38.50
87.75
65.65
70.20
83.62
74.43
87.87
61.36
DOE
55.87
85.98
80.94
76.26
67.84
83.05
34.67
88.90
75.50
59.83
83.54
Textual OE
None
84.74
74.02
92.52
63.85
99.21
41.72
99.01
27.30
72.43
93.87
51.72
Word
31.72
94.56
58.76
86.73
67.68
83.51
80.12
77.10
76.60
59.57
85.47
Desc.
33.12
94.28
54.72
87.70
72.47
81.82
86.03
70.72
77.48
83.63
Caption
29.61
94.74
57.12
87.34
66.82
83.71
79.29
77.76
76.00
58.21
85.88
5.1
Comparison with competitive baseline approaches
Textual outliers outperform previous advanced methods. To compare the effectiveness of textual outlier exposure method with other outlier exposure methods, we consider the following baselines: OE [18. CSI 45). SSD+ [41, MixOE (55), VOS [8, and DOE [50]. The performances of compared methods in Table 5 are borrowed from those reported in Wang et al. [50]. All of these baseline
methods utilize CNN architectures, specifically ResNet 16 or WideResNet
54
Our textual outlier methods at every level consistently achieve superior Ool detection performance, surpassing competing approaches terms of AUROC. Caption- and word-level textual outliers suc- ceed at improving the FPR95 metric as well. We point out that the caption-level textual outliers reduce FPR95 from 87.87% (VOS) 58.21% (ours), resulting in substantial improvement of 29.6666 29.66% This significant performance gap highlights the effectiveness of our textual outlier generation method for model regularization. The results on the ImageNet- benchmark and comparison with post-hoc methods can be found in the Appendix A. We include experimental results for CLIP-RM LPP-RN50 and RN50x4 in the Appendix D.4 and D.5 demonstrate that the observed performance gains do not stem from different architectures, namely ResNet and VIT. The performance comparison among the proposed textual outliers included in the lower half of T We also report the results of using CLIP without outlier exposure (None) to demonstrate that the performance improvement is not simply due to the inductive bias of the CLIP embedding space. Among the three levels of textual outliers, the caption-level textual outliers demonstrate superior performance average.
Based these results, we can deduce that textual outliers must the characteristies summarized below to function effectively for outlier exposure. While the near-din near-distribution criterion is in line with previous insights from the outlier exposure literature, the other two are unique to textual outliers.
Near-Distribution: Textual outliers should be situated near the boundary samples of the
training distribution, as several approaches in visual outlier exposure have emphasized 23 33 3] This allows the neural network to compactly estimate the decision boundary between ID and OoD data [8]
Textual outliers should possess rich linguistic semantics to include comprehensive and diverse explanations of OoD data. Word-level textual outliers, which relatively terse, often fall short of the other two forms of textual outliers this aspect.
Descriptiveness:
Inclusion of Visual Semanties: Textual outlies should incorporate visual elements, such that generated textual outliers do not deviate far from visual boundary samples of ID data.
The characteristics satisfied by textual outliers at each level detailed in the Appendix F.3.

---

Table 6: Effectiveness of textual outliers exposure in hard OoD situation, a more challenging OoD detection scenario. Proposed textual outliers are compared against strong baseline approaches on three different ID/OOD dataset combinations. The best result each column is in bold.
ID OoD
ImageNet10
ImageNet20
CUB 100
Dataset
ImageNet20
ImageNet10
CUB 100
FPR
AUROC
FPR
AUROC
FPR
AUROC
OF VOS
93.20
54.51
89.80
57.63
87.20
60.70
Visual OF
91.00
60.42
93.50
44.38
94.79
49.25
Word
4.60
98.92
9.00
98.39
80.34
71.62
Textual OE
Desc.
3.60
99.02
7.20
98.69
82.56
72.33
Caption
5.30
98.85
8.60
98.43
85.92
64.79
5.2 Textual outliers in hard OoD situation
Description-level textual outliers outperforms in hard OoD. Furthermore, we explore the effec- tiveness of textual outliers in handling challenging OoD inputs. In these scenarios, the OoD samples closely resemble the ID samples in terms of their semantic content. To evaluate the performance of OoD detection in such semantically hard scenario, first utilize the ImageNet10 vs. ImageNet20 benchmark proposed by Ming et al. (4 This task involves distinguishing between high-resolution images with semantically similar categories, such dog vs. wolf. Our experimental results, shown in Table (first and second columns), demonstrate the superiority of our textual outliers over OE, achieving 89.6%p improvement in FPR95 for ImageNet10 (ID) vs. ImageNet20 (OoD), and 82.6%p improvement when ID and OoD data are switched.
Additionally, conduct experiments the CUB dataset [49] benchmark, which Vaze et al. [48] proposed (third column in (). This benchmark is designed to discriminate between ID and OoD samples that possess numerous shared attributes, thereby introducing difficulties in accurately identifying OoD instances. As shown in Table 6 our description-basse textual outliers outperform the auxiliary dataset and synthesis methods. While descriptions performance, they can also be effective outliers in situations where class names are not considered. The class labels for the hard OoD benchmarks are provided in the Appendix F.2.
known to improve classification
Related work
OoD detection. A considerable body of research has been dedicated to designing scoring functions for detecting samples that lie outside the training categories. These methods include Maximum Softmax Probability 17], ODIN score (27). Mahalanobis-based score [24). Energy score (29), gradient-
score 199, and non- non-parametric KNN score 44). While there have been studies proposing OoD scores
using CLIP's 's textual information [32
12
10] none of them have incorporated textual information
into outlier exposure during training. An alternative approach address the out-of-distributton (OoD) detection problem is through training- timen regularization, as discussed in previous research [23] [8] These methods encourage models provide predictions with lower confidence or higher energies, aiming to improve OoD detection. However, most of these approaches rely on the availability of an auxiliary outlier dataset. To overcome this limitation, recent studies have explored the synthesis of virtual outliers in the feature space, seen in works such as g 46 This approach aims generate synthetic outliers without the need for an additional dataset. Our proposed textual outliers differ from previous methods in that they are defined in the input space and do not rely on auxiliary dataset.
Multi-modal models. The adoption of pre-trained VLMs multi-modal tasks has attracted consid- erable interest and yielded impressive results. Prominent among these approaches are dual-stream models such CLIP (37) ALIGN (2T), and FILIP [53]. These models utilize distinct encoders for text and image data and optimize them through contrastive objectives to align semantically similar features across heterogeneous modalities. The success of CLIP-like models inspired numerous subsequent investigations [26 56 59, aiming to enhance data efficiency and adaptability for various downstream tasks. Recent progress this field, exemplified by innovations such as DALL-E 2 (38), ClipCap 35. and related studies (5] [3, has pushed the boundaries of exploration in multi-modal contrastive representation spaces, uncovering their untapped potential. To the best of our knowledge, this is the first time that joint embedding has been utilized for outlier exposure.

---

7 Conclusion
In this paper, we propose novel outlier framework, transitioning from the traditional single-modal paradigm to a multi-modal regime. Unlike existing methods that rely on image data, we leverage textual outliers during the training process. We analyze the advantages of textual outliers compared to visual outliers and propose three levels of textual outliers. Our proposed textual outliers significantly enhance the neural network's 's capability to distinguish between ID and OoD data, leading to superior OoD detection performance while maintaining the classification performance on ID data. Furthermore, our analysis of each level of outliers provides valuable insights into the characteristics of effective textual outliers. We anticipate that our work will serve source of inspiration for future research on OoD detection methods that leverage textual outlier exposure. Limitations. Our proposed textual outlier method involves heuristic selection process to refine outputs of generative models. In future work, it is necessary to reduce the dependence on heuristic methods and explore alternative approaches.
Acknowledgments
This work was supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (2022R A3B 1077720 and 2022RIA5A7083900 1)), Institute of Information & Communications Technology Planning & Evaluation (ITTP) grants funded by the Korea government (MSIT) [2021-0-0134) Artificial Intelligence Graduate School Program (Scoul National University) and 2022-0-00959] and the BK21 FOUR program of the Education and Research Program for Future ICT Pioneers, Scoul National University in 2023.
A5
References
[1] Manuele Barraco, Marcella Comia,, Silvia Cascianelli, Lorenzo Baraldi, and Rita Cucchiara. The unreason- able effectiveness of clip fcatures for image captioning: an experimental analysis. In proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4662-4670, 2022.
[2] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh. Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Fric Sigler, Mateusz I Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Bemer, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodci. Language models are few-shot learners. In H. I Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc., 2020.
[3] Jiefeng Chen, Yixuan Li, Xi Wo, Yingyu Liang, and Somesh Jha. Atom: Robustifying diist--iittiittton detection using outlier mining. In Proceedings of European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD), 2021.
[4] Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In 2014 IEEE Conference on Computer Vision and Pattern Recognition, pages 3606-3613, 2014.
[5] Niv Cohen, Rinon Gal, Eli A Mcirom, Gal Chechik, and Yuval Atzmon. "this is my unicom, fluffy": Personalizing frozen vision-language representations. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviy, Israel, October 23 27, 2022, Proceedings, Part XX, pages 558 577. Springer, 2022.
[6] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: large-scale hierarchical image database. 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248-255. 2009.
[7] Jacob Devlin, Ming-Was Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Volume (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesola, June 2019. Association for Computational Linguistics.
[8] Xuefeng Du, Zhaoning Wang, Mu Cai, and Yixuan Li. Vos: Learning what you don't know by virtual outlier synthesis. Proceedings of the International Conference Learning Representations, 2022.
10

---

[9] Yu Du, Fangyun Wei, Zihe Zhang, Miaojing Shi, Yue Gao, and Guoqi Li. Learning to prompt for open- vocabulary object detection with vision-language model. In Proceedings the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 14084-14093, 2022.
[10] Sepideh Psur Bing Liu, Fric Robertson, and Iei Shu. Zero-shot out-of-distribution detection based on the pre-trained model clip. Proceedings of the AAAI Conference on Artificial Intelligence. 36(6):6568 6576, Jun. 2022.
[1]] Zhen Fang. Yixuan Li, Jie Lu, Jiahua Dong, Bo Han, and Feng Liu. Is out-of-distribution detection learnalle? In Advances in Neural Information Processing Systems, 2022.
[12] Stanislav Fort, Jie Ren, and Balaji Lakshminarayanan. Exploring the limits of
detection. In A. Beygelzimer. Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, 2021.
[13] Peng Gao, Shijic Geng, Renrui Zhang, Teli Ma, Rongyao Fang, Yongfeng Zhang, Hongsheng Li, and Yu Qiao. Clip-adapter: Better vision-language models with feature adapters. arXiv preprint arXiv:21 0404444, 2021.
[14] Golnaz Ghiasi. Xiuye Gu. Yin Cui, and Tsung- Yi Lin. Open-vocabulary image segmentation. arXiv preprint arXiv:21 2021.
[15] Xiuye Gu, Tsung-Y Yi Lin, Weicheng Kuo, and Yin Cui. Open-vocabulary object detection via vision and language knowledge distillation. In International Conference on Learning Representations, 2022.
[16] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pottern recognition, pages 770-778, 2016.
[17] Dan Hendryeks and Kevin Gimpel. A baseline for detecting misclassified and diiit-iiiititttiinn examples in neural networks. In International Conference on Learning Representations, 2017.
[18] Dan Hendrycks, Mantas Mazeika, and Thomas Dietterich. Deep anomaly detection with outlier exposure. Proceedings of the International Conference on Learning Representations, 2019.
[19] Rui Huang, Andrew Geng, and Yixuan Li. On the importance of gradients for detecting distributional shifts in the wild. In A. Beygelzimer, Y. Dauphin, Liang, and J. Wortman Vaughan, editors, Advances Neural Information Processing Systems, 2021.
[20] Rui Huang and Yixuan Li. Mos: Towards scaling out-of-distribution detection for large semantic space. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, 2021.
[21] Chao Jia, Yinfci Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hicu Pham, Quoe Lc, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vasion-language representation learning with noisy text supervision. In International Conference on Machine Learning, pages 4904-494 4916. PMLR, 2021.
[22] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 1412.6980 2014.
[23] Kimin Lee, Honglak Lee, Kibok Lee, and Jinwoo Shin. Training confidence-calibaate classifiers for detecting out-of-distributoo samples. In International Conference on Learning Representations, 2018.
[24] Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for detecting out- of-distribution samples and adversarial attacks. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.
[25] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models, 2023.
[26] Yangguang Li, Feng Liang, Lichen Zhao, Yufeng Cui, Wanli Ouyang, Jing Shao, Fengwei Yu, and Junjic Yan. Supervision exists everywhere: A data efficient contrastive language-image pro-training paradigm. In International Conference on Learning Representations, 2022.
[27] Shiyu Liang, Yixuan Li, and R. Srikant. Enhancing the reliability of out- of distributiin image detection in neural networks. In International Conference on Learning Representations, 2018.
[28] Tsung- Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision- ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6- 2014, Proceedings, Part V 13, pages 740-755. Springer, 2014.

---

[29] Weitang Liu. Xiaoyun Wang, John Owens, and Yixuan Li. Energy-based out-of-distribute detection. Advances in Neural Information Processing Systems, 2020.
[30] Leland McInnes, John Healy. Nathaniel Saul, and Lukas Grossberger. Umap: Uniform manifold approxi- mation and projection. The Journal of Open Source Software, 3(29):661, 2018.
[31] Sachit Menon and Carl Vondrick. Visual classification via description from large language models. In The Eleventh International Conference on Learning Representations, 2023.
[32] Yifei Ming. Ziyang Cai, Jiuxiang Gu, Yiyou Sun, Wei Li. and Yixuan Li. Delving into out-of-distribution detection with vision-language representations. In Advances in Neural Information Processing Systems, 2022.
[33] Yifei Ming. Ying Fan, and Yixuan Li. Poem: Out-of-distributi- detection with posterior sampling. In International Conference on Machine Learning. PMLR, 2022.
[34] Yifei Ming, Hang Yin, and Yixuan Li. On the impact of spurious correlation for out-of-distribution detection. The AAAI Conference on Artificial Intelligence (AAAI), 2022.
[35] Ron Mokady, Amir Hertz, and Amit H Bermano. Clipcap: Clip prefix for image captioning. arXiv preprint 10Xiv:211.099344 2021.
[36] David Nukrai, Ron Mokady, and Amir Globerson. Text-only training for image captioning using noise- injected CLIP. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 4055-4063, Abu Dhabi, United Arah Emirates, December 2022. Association for Computational Linguistics.
[37] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 8748 8763. PMLR, 18 24 Jul 2021.
[38] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint urXiv:2204. 0602000,....... 2022.
[39] Jie Ren, Peter J Liu, Emily Fertig, Jasper Snoek, Ryan Poplin. Mark Depristo, Joshua Dillon, and Balaji Lakshminarayanan. Likelihood ratios for out-of dfstiiitiitttiinn detection. Advances in neural information processing systems, 32, 2019.
[40] Shibani Santurkar, Yann Dubois, Rohan Taori, Percy Liang, and Tatsunori Hashimoto. Is a caption worth thousand images? study on representation leaming. In The Eleventh International Conference on Learning Representations, 2023.
[41] Vikash Sehwag, Mung Chiang, and Pratcek Mittal. Ssd: A unified framework for self-supervised outlier detection. In International Conference on Learning Representations, 2021.
[42] Gyungin Shin, Weidi Xie, and Samuel Alhanie. Reco: Retrieve and co-segment for zero-shot transfer. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.
[43] Yiyou Sun, Chuan Guo, and Yixuan Li. React: Out-of-distribution detection with rectified activations. In Advances in Neural Information Processing Systems, 2021.
[44] Yiyou Sun, Yifei Ming. Xiaojin Zhu, and Yixuan Li. Out-o--distribution detection with deep nearest neighbors. ICML, 2022.
[45] Jihoon Tack, Sangwoo Mo, Jongheon Jeong, and Jinwoo Shin. Csi: Novelty detection via contrastive learning on distributionally shifted instances. In Advances in Neural Information Processing Systems, 2020.
[46] Leitian Tao, Xuefeng Du, Jerry Zhu, and Yixuan Li. Non-parametric outlier synthesis. In The Eleventh International Conference on Learning Representations, 2023.
[47] Grant Van Horn, Oisin Mac Aodha, Yang Song. Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona. and Serge Belongie. The inaturalist species classification and detection dataset. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), June 2018.
[48] Sagar Vaze, Kai Han, Andrea Vedaldi, and Andrew Zisserman. Open-set recognition: A good closed-set classifier is all you need. In International Conference on Learning Representations, 2022.
12

---

[49] Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-ucsd birds-200-2011 dataset. 2011.
[50] Qizhou Wang, Junjie Ye, Feng Liu, Quanyu Dai, Marcus Kalander, Tongliang Liu, Jianye HAO, and Bo Han. -istof-disti detection with implicit outlier transformation. In The Eleventh International Conference on Learning Representations. 2023.
[51] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistae, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. Transformers: State-of-the art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online, October 2020. Association for Computational Linguistics.
[52] Jianxiong Xiao, James Hays, Krista A. Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large- scale scene recognition from abbey to zoo. 2010 JEEE Computer Society Conference on Computer Vision and Pattern Recognition, pages 3485-3492, 2010.
[53] Lewei Yao, Runhui Huang, Lu Hou, Guansong Lu, Minzhe Niu, Hang Xu, Xiaodan Liang, Zhenguo Li, Xin Jiang, and Chunjing Xu. FILIP: Fine-grained interactive language-image pre-training. International Conference on Learning Representations, 2022.
[54] Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In Edwin R. Hancock Richard Wilson and William A. P. Smith, editors, Proceedings of the British Machine Vision Conference (BMVC), pages 87.1-87.12. BMVA Press, September 2016.
[55] Jingyang Zhang, Nathan Inkawhich, Randolph Linderman, Yiran Chen, and Hai Li. Mixture outlier exposure: Towards out-of-distribttion detection in fine-grained environments. In Proceedings of the IEEE/CVF Winter Conference Applications of Computer Vision (WACV), pages 5531-5540, January 2023.
[56] Renrui Zhang, Rongyao Fang, Wei Zhang, Peng Gao, Kunchang Li, Jifeng Dai, Yu Qiao, and Hong- sheng Li. Tip-adapter: Training-free clip-adapter for better vision-language modeling. arXiv preprint arXiv:2111.03930, 2021.
[57] Yuhui Zhang, Jeff Z. HaoChen, Shih- Cheng Huang. Kuan-Chieh Wang. James Zou, and Serena Yeung. Diagnosing and rectifying vision models using language. In The Eleventh International Conference on Learning Reprexentations, 2023.
[58] Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence. 40(6):1452 1464, 2018.
[59] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. In /EEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2022.
13


Reviews:
[{"review_id": "HCjuzZW5ku", "reviewer": "Reviewer_T6K7", "summary": "\nThis paper addresses the challenge of detecting Out-of-Distribution (OOD) data by introducing \"textual outlier exposure\" as an alternative to visual outliers. Instead of relying on visual examples, the authors explore the benefits of using textual equivalents in OOD detection. They propose various methods for generating textual outliers, which are validated through extensive experiments on large-scale OOD benchmarks. The findings demonstrate that the generated textual outliers outperform visual outliers and establish criteria for effective textual outliers, including their proximity to the data distribution, descriptiveness, and incorporation of visual semantics.\n\nThe contributions of this work include investigating the potential of textual outlier exposure with multi-modal neural networks, utilizing large-language models for generating textual outliers at different levels of detail, and validating their effectiveness in various OOD detection scenarios. The paper presents a novel and promising approach to OOD detection, showcasing the advantages of textual outliers over visual counterparts and providing valuable insights for designing impactful textual outliers.", "questions": "\nThe experimental evaluation in your paper is commendable. However, I have two inquiries regarding the experiments:\n\n- It appears that the recent state-of-the-art methods, such as ASH [1] and GradNorm [2], have not been included in your experimental comparisons. Could you provide insights into the rationale behind this omission? It would be valuable to report and compare the performance of your proposed method with these state-of-the-art approaches to provide a comprehensive assessment.\n\n- Additionally, considering the impact of different network architectures on your method is crucial. Could you elaborate on how varying network architectures affect the performance and effectiveness of your proposed approach? Investigating this aspect would provide deeper insights into the robustness and generalizability of your method.\n\n[1] Djurisic, Andrija, et al. \"Extremely simple activation shaping for out-of-distribution detection.\" arXiv preprint arXiv:2209.09858 (2022).\n\n[2] Huang, Rui, Andrew Geng, and Yixuan Li. \"On the importance of gradients for detecting distributional shifts in the wild.\" Advances in Neural Information Processing Systems 34 (2021): 677-689.\n\n\n\n\n\n\n\n", "limitations": "No negative social impact.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper presents an innovative and compelling idea, offering valuable insights for open-world learning, Out-of-Distribution (OOD) detection, and online learning. The findings of this study have significant implications for various research studies in these domains.\n\n- The clarity and coherence of the paper are commendable, making it easy to comprehend and follow the authors' methodological approach. \n\n- The inclusion of illustrative examples effectively enhances the understanding of key concepts.\n\n- The figures and plots presented in the paper are visually clear, aiding in the visualization and interpretation of the experimental results.\n\n- The paper offers comprehensive experimental studies, demonstrating the efficacy of the proposed method. The promising performance observed in these experiments further strengthens the validity and potential impact of the proposed approach.\n\n\n", "weaknesses": "While the paper presents detailed experimental studies, there are two notable weaknesses that should be addressed:\n\n- The focus of the experimental studies is primarily limited to outlier exposure methods, neglecting the exploration of other types of Out-of-Distribution (OOD) detection methods. It would be valuable to consider and compare the proposed approach against alternative techniques in order to provide a more comprehensive evaluation.\n\n- The omission of an ablation study to assess the impact of network architecture on the proposed methodology is a notable oversight. Investigating the influence of different network architectures on the performance of the proposed approach would enhance the understanding of its strengths and limitations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for acknowledging the novelty of our method and the extensive experiments. Here, based on your comments, we show various newly-conducted experimental results to further verify our method’s effectiveness. \n\n> Q1 (W1). It appears that the recent state-of-the-art methods, such as ASH [1] and GradNorm [2], have not been included in your experimental comparisons. \n\nFollowing the suggestion of the reviewer, we have compared our method with ASH and GradNorm, demonstrating superior performance in comparison to both. The table below presents the results of OoD detection performance, evaluated using AUROC, for a model trained on ImageNet1K across four OoD datasets. We would like to note that all the results are derived from the ResNet-101 architecture (BiT-S-R101 for GradNorm and ASH, CLIP-RN101 for our approach). In this experiment, we solely present results based on caption-level outliers for our method. In the revised manuscript, we will incorporate these results to offer a more comprehensive set of experiments. \n\n| | **iNaturalist** | **SUN** | **Places** | **Textures** | **AVG** | \n|----------|:---------------:|:-------:|:----------:|:------------:|:-------:| \n| GradNorm | 90.32 | 89.02 | 84.82 | 81.07 | 86.30 | \n| ASH | 94.84 | 88.72 | 86.61 | 88.40 | 89.64 | \n| ours | **95.54** | **91.84** | **88.42** | **91.70** | **91.93** | \n\nAdditionally, for your reference, we have conducted and reported a comparative analysis of our method with post-hoc OoD detection methods in Appendix A.3, presented in Table 4. \n\n> Q2 (W2). Could you elaborate on how varying network architectures affect the performance and effectiveness of your proposed approach? \n\nFollowing the suggestion of the reviewer, we conducted experiments to compare the performance of our method across two different image encoder architectures, ResNet and ViT, the two architectures offered by CLIP. Importantly, our method consistently produces encouraging outcomes, even when applied to CLIP models built upon the ResNet architecture. We selected RN50x4, designed with a parameter size akin to that of ViT-B/32 (174M vs 176M). The performance analysis reveals a comparable trend between RN50x4 and ViT-B/32, resulting in AUROC scores of 86.43 and 87.55, respectively. For this experiment, we utilized ImageNet1K as the in-distribution dataset. The results are obtained from caption-level textual outliers. \n\n| | **parameter size** | **iNaturalist** | **SUN** | **Places** | **Textures** | **AVG** | \n|----------|:-------:|:---------------:|:-------:|:----------:|:------------:|:-------:| \n| RN50x4 |174M| 91.69 | 85.19 | 82.93 | 85.94 | 86.43 | \n| ViT-B/32 |176M| 94.55 | 86.59 | 81.51 | 79.48 | 87.55 | \n\nIn the revised manuscript, we will incorporate these results to offer a more comprehensive set of experiments."}, {"review_id": "tMyGAzl5WK", "reviewer": "Reviewer_Lz76", "summary": "This paper studies visual OOD detection by introducing the textual outlier under the outlier exposure paradigm. Different from previous research focused on utilizing visual outliers, this work explores the benefits of textual outliers in the image domain. Specifically, they propose different ways to generate textual outliers based on the powerful GPT model. Comprehensive experiments have been conducted to demonstrate the effectiveness of textual outliers in OOD detection.", "questions": "Regarding the current version of the draft, I have the following question:\n1. Could the authors discuss more the unique contribution of OOD detection beyond the general paradigm of utilizing textual information? \n2. Could the authors provide more examples or system comparisons of the three different types of textual outlier generation?\n3. Could the authors discuss more why the performance of textual OE is not consistently performing better than visual OE?", "limitations": "This paper has discussed the limitation, e.g., the proposed textual outlier method involves a heuristic selection process to refine the outputs of the generative model and also provided the potential solution in future work.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. From the perspective of originality, to the best knowledge, this paper is the first work to explore the potential of textual exposure with multimodal neural networks for outlier exposure. \n2. The overall presentation of this work is good, having intuitive illustrations and clear organization. The framework and the proposed textual outliers generation are easy to understand.\n3. Instead of straightforwardly utilizing the LLM under the multimodal paradigm for visual OOD detection, this paper investigates the practical adaptation of different textual outlier types in the outlier exposure framework. \n4. Experiments from different perspectives have been conducted to demonstrate the effectiveness of description-level textual outliers.", "weaknesses": "1. Regarding the technical level, exploring the textual information (like description level or caption level) under the multimodal setting for the image domain has been studied in [1]. The general paradigm, e.g., using LLM to generate the corresponding description or caption and then utilizing them in the image task, is similar.  \n2. Compared with the conventional outlier exposure, the optimizing objective and test time OOD detection have limited advanced design beyond the previous method.\n3. Compared with the generation part, the filtering part seems to be more important as it has a more close relationship with the data quality for outlier exposure. However, the current filtering process is a little bit heuristic.\n4. Considering the performance, the improvement compared with other advanced methods of using pure image outliers is not very significant (like Table 5 compared with DOE). Could the authors provide more explanation or discussion? \n\nI generally appreciate the idea of introducing and excavating the potential of textual outliers. I hope the previous comments or the later questions can help the current contribution to be clearer and enhanced.\n\n[1] Sachit Menon and Carl Vondrick. Visual classification via description from large language models. In The Eleventh International Conference on Learning Representations, 2023.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for providing constructive feedback and suggestions. In particular, we appreciate your recognition of the novelty and efficiency of our work and the insightfulness of our analyses. Here, we hope that a more detailed explanation can address your concerns.\n\n> Q1 (W1, W2). Could the authors discuss more the unique contribution of OOD detection beyond the general paradigm of utilizing textual information?\n\nOur contribution is primarily centered around the proposition of textual outliers. Through a comprehensive analysis, we illuminate the potential of textual outliers as valuable indicators for detecting visual OoD detection (as elaborated in Section 3). We propose a range of textual outlier categories (outlined in Section 4) and identify key attributes of effective textual outliers (discussed in Section 5). As the reviewer pointed out, the distinctiveness of our method does not lie in the process of generating text through the LLM. Our focus is on utilizing these texts as textual outliers for the OoD detection task. Identifying informative textual outliers within the generated text demands a nuanced and intricate approach (W1). Our textual outlier approach is not constrained to using only uniform loss or energy scores, as demonstrated in our paper; it can also be applied with other objective functions or OoD scores. This flexibility enhances the scalability of our method (W2). \n\nGoing beyond the utilization of textual outliers, our method exhibits computational efficiency by achieving competitive performance solely through training of a linear classifier. This advantage in computational efficiency can come in particularly handy when new OoD instances emerge frequently, and outlier exposure must be performed repeatedly.\n\n> W3. The current filtering process is a little bit heuristic.\n\nWhile our filtering approach includes heuristic elements, we address this by conducting ablation studies for each method to determine the optimal parameters (Appendix B.4, B.5).\n\n> Q2. Could the authors provide more examples or system comparisons of the three different types of textual outlier generation?\n\nHere are a few examples of textual outliers for ImageNet. We offer three distinct types of textual outliers for each class.\n\nClass: Warplane \n* word: “This is a photo of army”, “This is a photo of airport”, “This is a photo of airliner” \n* desc: “a photo of large and powerful\", \"a photo of designed for carrying weapons and other military equipment\", \"a photo of typically has a camouflage paint job\",\n* caption: “a small plane with a window on the side”, “a silver airplane”, “a plane with a large window on the front”, “a plane taking off” \n\nClass: Greater Swiss Mountain Dog\n* word: “This is a photo of green”, “This is a photo of puppy”, “This is a photo of pets”\n* desc: \"a photo of large, fluffy white dog\", \"a photo of black or brown markings on the face, ears, and tail\", \"a photo of long, thick coat\"\n* caption: “two dogs are playing with each other”, “a dog with a white face”, “a dog with its tongue out”\n\nBased on the provided examples, our word-level outliers convey more abstract concepts. Similarly, caption-level outliers mostly contain descriptions of background elements or lack class-specific attributes. Description-level outliers include class-relevant information, but when the class label is omitted, they become very vague and difficult to interpret. In the revised manuscript, we will incorporate additional examples to enhance understanding.\n\n> Q3 (W4).  Could the authors discuss more why the performance of textual OE is not consistently performing better than visual OE?\n\nThe results of using pure image outliers are presented in Table 5 under the label 'OE,' where our textual outlier approach notably outperforms the OE baseline. It should be noted that while DOE is not strictly categorized as a pure image outlier method. Therefore, the fact that textual outliers do not consistently outperform DOE does not imply that textual outliers do not consistently outperform pure image outliers. While DOE utilizes model perturbations, which are computationally demanding, our approach only requires training a lightweight linear classifier, making it computationally efficient. A key advantage of textual outlier exposure over DOE, albeit insufficiently highlighted in the current manuscript, lies in its computational efficiency. We will elaborate on this discussion in the revised manuscript to further underscore the benefits of adopting textual outlier exposure.\n"}, {"review_id": "MdRuy4Y03H", "reviewer": "Reviewer_JSZe", "summary": "The paper proposes a new method that takes the textual outliers to help the model better detect out-of-distribution samples. Specifically, they build the pipeline on top of CLIP models with a classifier and use different representations of the text sample to synthesize outliers in the CLIP space, then use them to train the model with ood awareness. The authors provide extensive experiments and analysis of the method.", "questions": "How can you prove the ood robustness is come from your method but not from CLIP itself? \n\nIn the appendix, only the larger CLIP model is studied. However,  in the open-sourced CLIP API, they provided smaller models. I wonder how much impact it would have on the metrics as the CLIP shrinks.\n", "limitations": "The method seems only experimented with classifiers. The appliance of the CLIP Image encoder makes it hard to extend to other problems like Object Detection, semantic segmentations, .etc.\n", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "I think the paper is well-written, and the paper is in a great flow. Figures and texts are clear and easy to follow. The authors also give a good rationale and background of the problem. \n\nThe authors approach the OOD problem with the textual outliers and the help of CLIP fundamental models, which is a novel and plausible method.\n\nThe authors provide extensive experiments and detailed analysis of the problem and extensively worked on different types of textual outliers.  \n", "weaknesses": "The authors build the classifiers on top of CLIP, which is a large model with 151 Million parameters (~88 Million in the Image encoder) and implies a large hidden feature space. Comparing this with previous methods may lead to unfair comparisons.\n\nThe use of CLIP may limit this to other fields like object detection/segmentations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for acknowledging the novelty of our method and the extensive experimental results. Here, we show various newly-conducted experimental results to further verify the effectiveness of our method.\n\n> Q1 (W1). How can you prove the ood robustness is come from your method but not from CLIP itself?\n\nIn order to demonstrate that the success of our method cannot be attributed solely to the use of CLIP, we have provided results of using CLIP without outlier exposure (labeled as 'None' in Table 5) to substantiate that the performance enhancement is not merely a result of the inherent bias within the CLIP embedding space. Notably, without the integration of textual outliers, the AUROC score is only 51.72, whereas our caption-level textual outliers achieve an AUROC score of 85.53. This disparity underscores that CLIP, without supplementary techniques tailored for OoD detection, yields significantly lower AUROC scores, further attesting that the success of textual outliers is not merely a byproduct of the CLIP embedding space.\n\nNPOS [1] is another work that leverages CLIP's embedding space. NPOS synthesizes outliers from in-distribution data and fine-tunes the CLIP image encoder using these synthesized examples. In the table below, we compare the performance of NPOS vs. Ours. When compared against NPOS, we employed the ImageNet100 dataset as the in-distribution data and utilized CLIP-L/14, which incorporates a ViT-L/14 Transformer as the image encoder for both approaches. The outcome for our method is derived from caption-level outliers. Our approach achieves slightly superior average AUROC values across all four OoD datasets, all without necessitating resource-intensive fine-tuning steps. This implies that textual outliers are potent tools for advancing visual OoD detection.\n\n|  | **iNaturalist** | **SUN** | **Places** | **Textures** | **AVG** |\n|------|:-----------------:|:---------:|:------------:|:--------------:|:---------:|\n| NPOS | 99.14           | 98.06   | 97.69      | 98.19        | 98.27   |\n| ours | 98.86           | 97.93   | 98.29      | 98.48        | **98.39**   |\n\nIn the revised manuscript, we will incorporate these results to offer a more comprehensive set of experiments.\n\n[1] Leitian Tao, Xuefeng Du, Jerry Zhu, and Yixuan Li. Non-parametric outlier synthesis. In The Eleventh International Conference on Learning Representations, 2023.\n\n> W2. The use of CLIP may limit this to other fields like object detection/segmentations.\n\nThe primary objective of our paper is to improve OOD detection in the context of classification. However, the CLIP encoder can also be extended to object detection or segmentation tasks. Notably, there are approaches designed for object detection [2] and segmentation [3] that are built upon the CLIP architecture.\n\n[2] Yiwu Zhong, et al. RegionCLIP: Region-based Language-Image Pretraining. CVPR 2022 \n\n[3] Chong Zhou, Chen Change Loy, and Bo Dai. Extract Free Dense Labels from CLIP. ECCV 2022 \n\n> Q2. I wonder how much impact it would have on the metrics as the CLIP shrinks.\n\nFollowing the suggestion of the reviewer, we conducted an ablation study using a smaller model (ViT-B/32 vs ResNet50). For this experiment, we utilized ImageNet1K as the in-distribution dataset. Remarkably, our textual outlier exhibited comparable performance levels (AUROC) even with a significantly smaller model size (176M vs 77M). The results are obtained from description-level textual outliers.\n\n|  | **parameter size** | **iNaturalist** | **SUN** | **Places** | **Textures** | **AVG** |\n|----------|:------------------:|:---------------:|:-------:|:----------:|:------------:|:-------:|\n| ViT-B/32 | 176M               | 92.07           | 85.24   | 77.40       | 79.77        | 83.62   |\n| RN50     | 77M                | 91.01           | 82.44   | 74.95      | 81.70        | 82.52   |\n\nIn the revised manuscript, we will incorporate these results to offer a more comprehensive set of experiments."}, {"review_id": "ZRN0rfczvh", "reviewer": "Reviewer_yebH", "summary": "The paper notices that while outlier exposure has shown promising potential in improving OoD detection performance, all previous studies on outlier exposure have been limited to utilizing visual outliers. The paper uncovers the benefits of using textual outliers by replacing real or virtual outliers in the image domain with textual equivalents. Then, it proposes various ways of generating preferable textual outliers. The extensive experiments demonstrate that generated textual outliers achieve competitive performance on large-scale OoD and hard OoD benchmarks.", "questions": "see above", "limitations": "yes", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The paper is written well and is easy to understand.\n2. The studied problem is very important. section 3 is quite interesting.\n3. The results seem to outperform state-of-the-art.\n", "weaknesses": "1. I am curious about section 4.1.1 and section 4.1.3, why do the authors choose to use the images from the validation set of the ID dataset for generation?\n2. It is not clear that if the class labels of the outlier dataset is missing, how can we generation description-level supervision in section 4.1.2?\n3. I am curious about the performance of the method without the large-scale pretrained models (CLIP) as the classification backbone. The current approach seems to be highly model-specific, which hinders its general usage.\n4. The comparison with a reasonable baseline, i.e., NPOS [1] is missing. \n\n[1] Leitian Tao, Xuefeng Du, Jerry Zhu, and Yixuan Li. Non-parametric outlier synthesis. In The Eleventh International Conference on Learning Representations, 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for all of the constructive feedback and suggestions. In particular, we appreciate your recognition of the novelty and efficiency of our work and the insightfulness of our analyses. Here, we show additional experimental results to address your concerns.\n\n\n> W1. Why do the authors choose to use the images from the validation set of the ID dataset for generation?\n\nDue to computational constraints, utilizing the entire training set for generating textual outliers proves time-consuming. Our findings demonstrate that employing only a validation set yields sufficiently favorable performance.\n\n> W2. How can we generation description-level supervision in section 4.1.2?\n\nOur method operates without the need for class labels from the outlier dataset. Instead, we leverage the class labels from the in-distribution data to generate textual outliers, excluding the class label from the descriptions. Descriptions without class labels can be identified as outliers. Illustrative examples can be found in Appendix C, Figure 2.\n\n> W3. I am curious about the performance of the method without the large-scale pretrained models (CLIP) as the classification backbone.\n\nOur method is designed to leverage the power of a joint embedding space that contains both visual and textual embeddings. By definition, such an embedding space is only available in vision-language models, and thus, our method is contingent on the use of VLM models. Regarding your concern with the scalability of our method to smaller models, it's worth noting that CLIP offers a smaller model option, such as ResNet50. Our method still demonstrates comparable performance even on models of reduced size (176M vs 77M). For this experiment, we utilized ImageNet1K as the in-distribution dataset. The results are obtained from description-level textual outliers. \n\n|  | **parameter size** | **iNaturalist** | **SUN** | **Places** | **Textures** | **AVG** |\n|----------|:------------------:|:---------------:|:-------:|:----------:|:------------:|:-------:|\n| ViT-B/32 | 176M               | 92.07           | 85.24   | 77.40       | 79.77        | 83.62   |\n| RN50     | 77M                | 91.01           | 82.44   | 74.95      | 81.70        | 82.52   |\n\nIn the revised manuscript, we will incorporate these results to offer a more comprehensive set of experiments.\n\n> W4. The comparison with a reasonable baseline, i.e., NPOS [1] is missing.\n\nFollowing the suggestion of the reviewer, we experimented with NPOS and the results are as follows. \n\n|  | **iNaturalist** | **SUN** | **Places** | **Textures** | **AVG** |\n|------|:-----------------:|:---------:|:------------:|:--------------:|:---------:|\n| NPOS | 99.14           | 98.06   | 97.69      | 98.19        | 98.27   |\n| ours | 98.86           | 97.93   | 98.29      | 98.48        | **98.39**   |\n\nWe employed the ImageNet100 dataset as the in-distribution data and utilized CLIP-L/14, which incorporates a ViT-L/14 Transformer as the image encoder for both approaches. The outcome for our method is derived from caption-level outliers. The average AUROC values across all four OoD datasets highlight that our textual outlier approach outperforms NPOS. In the revised manuscript, we will incorporate these results to offer a more comprehensive set of experiments."}]

Author rebuttal:

We sincerely appreciate the reviewers' time and invaluable feedback. The unanimous consensus among reviewers highlights our paper's insightful contribution on textual outliers (T6K7, Lz76, JSZe, yebH). The reviewers also commend the novelty and significance of the addressed problem (T6K7, JSZe, yebH), along with acknowledging the method's effectiveness through extensive experiments (T6K7, Lz76, JSZe). Additionally, we are pleased that the reviewers found the paper clear and easily understandable (Lz76, JSZe, yebH).

We respond to each reviewer's comments in detail below. We will incorporate the reviewers' suggestions into the manuscript revisions, which we believe will significantly enhance the paper's strength.

Meta-review: Congratulations to the authors on their work. The reviewers unanimously vote for acceptance. Please add the new results into the final version of the paper.
Decision: Accept (poster)