Textual Training for the Hassle-Free Removal of Unwanted Visual Data: Case Studies on OOD and Hateful Image Detection
Authors: Saehyung Lee
Jisoo Mok
Sangha Park
Yongho Shin
Dahuin Jung
Sungroh Yoon
Venue: NeurIPS 2024 poster (2024)
Keywords: Vision-Language models
Multimodal models
CLIP
unwanted visual data detection
text-only training
hateful image detection
Primary area: other
TL;DR: We introduce Hassle-Free Textual Training (HFTT), a method designed to improve the performance of VLMs in identifying unwanted visual data, utilizing solely synthetic textual data.

Abstract: In our study, we explore methods for detecting unwanted content lurking in visual datasets. We provide a theoretical analysis demonstrating that a model capable of successfully partitioning visual data can be obtained using only textual data. Based on the analysis, we propose Hassle-Free Textual Training (HFTT), a streamlined method capable of acquiring detectors for unwanted visual content, using only textual data in conjunction with pre-trained vision-language models. HFTT features an innovative objective function that significantly reduces the necessity for human involvement in data annotation. Furthermore, HFTT employs a clever textual data synthesis method, effectively emulating the integration of unknown visual data distribution into the training process at no extra cost. The unique characteristics of HFTT extend its utility beyond traditional out-of-distribution detection, making it applicable to tasks that address more abstract concepts. We complement our analyses with experiments in hateful image detection and out-of-distribution detection. Our codes are available at https://github.com/HFTT-anonymous/HFTT.

Textual Training for the Hassle-Free Removal of Unwanted Visual Data Case Studies on OOD and Hateful Image Detection
Sachyung Lee Jisoo Mokk Sangha Park! Yongho Shin Dahuin Jung Sungroh Yoon 1.41
*Department of Electrical and Computer Engineering, Scoul National University
Qualcomm Korea YH, Scoul, South Korea
School of Computer Science and Engineering, Soongsil University IInterdisciplinary Program in Artificial Intelligence, Scoul National University
(halo8218, magicshopl118, wiarae @snu. ac.uz kr, manaineetiqualcom... qualcomm. com, dahuin. jungessu. ac.kr, sryocnesnu.ac ac. kr
yshin0gti
Abstract
In study, explore methods for detecting unwanted content lurking in visual datasets. We provide a theoretical analysis demonstrating that a model capable successfully partitioning visual data can be obtained using only textual data. Based on the analysis, we propose Hassle-Free Textual Training (HFTT), a streamlined method capable of acquiring detectors for unwanted visual content, using only syn- thetic textual data in conjunction with pre-trate pre-trained lision-language models. HFTT features innovative objective function that significantly reduces the necessity for human involvement in data annotation. Furthermore, HFTT employs a clever tex- tual data synthesis method, effectively emulating the integration of unknown visual data distribution into the training process at no extra cost. The unique characteristics of HFTT extend its utility beyond traditional out-of-din -il-o--diittibttionn detection, making it applicable to tasks that address more abstract concepts. We complement our anal- yses with experiments in out-of-distribttion detection and hateful image detection. Our codes are available at
com Sow/Saehyung Lee/HFTH
Introduction
We are currently in the midst of what is known as the large-scale Al era. The growth in both the size of deep neural networks and training datasets led unparalleled achievements wide array of tasks However, this transition to large-scale Al presents new, unforeseen challenges. Particularly. the recent reports the biased behavior of large Al models raise significant concerns surrounding the continuously expanding size of training datasets without proper quality control and regulation 41). The massive scale of visual training datasets necessary to train large-scale models presents a challenge in curating data to ensure unbiased and safe datasets. This is primarily due to the impracticality of manually selecting and removing unwanted content from such an extensive collection of images. This issue of data curation has traditionally been addressed by: (i) creating supervised dataset for specific objective; (ii) training a model on this dataset; and then (iii) utilizing the model to develop larger dataset (28). However, this approach exploits considerable human labor and needs to be re-initiated from beginning whenever there are changes
43
training objective.
"This publication was created by Saehyung Lee while an intern at QTI and currently attends Seoul National University.
I Corresponding Authors
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

(a) Training Phase
(b) Test Phase
ACE
Thresholding
1
I (exp(CosSim(-))
Eexp(CosSim(-))
P(Y 1/x)
P(Y -
Σ Exxp(CosSim (-)) Eexp(CosSim(-))
Σ Eexp(CosSim(-)) Ixxp(CosSim(-))
Task Embeddings
Trainable Embeddings
Task Embeddings
Trainable Embeddings
Dog
...
Cat
Bus
Dog
...
Cat
Bus
Measure Cosine-similarity
Measure Cosine-similarit
Text Encoder
Image Encoder
Xy ~ Text Corpus
X, (Images)
Figure 1: Overview our proposed method. Task embeddings define the task to be performed. For example, in the case of hateful image detection, hate speeches would serve as task embeddings. while in OOD detection, the names of classes from the training distribution would be the task embeddings. Trainable embeddings are the only parameters that trained in method, defined in the joint embedding space. During the training phase, only textual data used, and in the testing phase, these trained parameters are employed to classify images. Detailed explanations are provided Sections
The field of out-of-distributie (OOD) detection, which aims identify OOD data that lie outside the training data distribution, can be considered sub-branch of data curation research. Recent works in OOD detection utilize vision-language models (VLMs) 40 29
30 to take advantage of the rich and human-all alignee representations learned by these models. For instance, Esmaeilpour al. [II augmented the pre-trained CLIP model [40] with an additional decoder module, trained similar vein. Wang et al. (49) incorporated an extra "no" text encoder, trained on a vision-language dataset, into CLIP. These previous approaches, however, suffer from a significant limitation: They require vast amount of additional vision- language data. Using data samples targeted for detection improve sample efficiency, but this leads to the dilemma of needing to collect unwanted data for the purpose of removing them.
vision-language dataset, for visual OOD detection.
In this work, we propose a novel method that no longer relies on additional visual data or compu- tationally expensive training process. We first outline our theoretical rationale, particularly demon- strating that with a successfully trained model on bimodal dataset, like CLIP, one can obtain classifier to partition data from one mode using solely the data from the other mode. Building upon our motivation, we propose a method called Hassle-Free Textual Training (HFTT). HFTT consists of newly proposed loss and a clever textual data synthesis method, updating trainable parameters defined in the joint embedding space to improve the detection of undesirable visual content. Specifically, decompose the weighted cross-entropy loss into a formula that includes a regularization term, which is tailored to our use. Additionally, to achieve higher detection accuracy, we introduce the concept of focal loss (33. Moreover, our textual data synthesis method, which combines prompt templates and words, can effectively imitate the involvement of the entire visual data distribution in the training process. We illustrate an overview of our proposed method in Figure D
The proposed loss function brings considerable convenience in achieving our objective. To train an unwanted data detection model, it necessary to define out-distribution for given data distribution (in-distribution), which is not always straightforward due to vague boundaries between the two. For instance, in hateful content detection tasks [13,, the divide between what is hateful and what is not is influenced by various contexts, e.g., historical and social backgrounds. Our proposed loss climinates the need for human labor to annotate aut-distribution data because it does not involve a clearly defined set of out-distribution data. Furthermore, our textual data synthesis method requires no cost, employing a rule-based approach using only prompt templates and a set of words.
Based on the principle that HFTT can detect out-distribution samples by merely defining the in- distribution in natural language, we propose that this method can extended to tasks beyond traditional OOD detection, including hateful image detection. Current OOD detection methods often fail such extended tasks for two main reasons: firstly, they are based on the assumption of distinct boundary between in- and out-distributions, which is unsuitable for tasks with abstract concepts.

---

Secondly, methods requiring training images may lead to ethical concerns. Our proposed method, however, is not subject to these limitations.
Through empirical validation, we verify that HFTT can enhance the performance of VLMs in identifying unwanted visual data, whether it be OOD instances or hateful images. Additionally, we demonstrate through feature visualization results that HFTT, despite not observing any visual data in the training phase, appears to have been trained as if it had. Furthermore, we provide various analyses of HFTT, including comparative results of using different textual data synthesis methods.
In summary, our contributions are as follows: (i) We theoretically demonstrate how textual data can serve as a substitute for visual data in our scenario; (ii) We introduce a new loss function. Its use eliminates the need for labor in annotating out-distribution data; (iii) We propose a textual data synthesis method that can efficiently imitate the visual data distribution in our training; (iv) We empirically analyze HFTT, a method composed of the above proposals. Our experiments show that HFTT is effective in a range of scenarios, from traditional OOD detection to situations involving abstract concepts, like the identification of hateful images.
2
Related Work
Vision-language models. With the advancements in deep learning, tackling sophisticated tasks that demand an understanding of both vision and language modalities has become viable. The methodologies employed to encode image and text data exhibit notable distinctions owing to their inherent differences. Prominent within this domain are dual-stream models exemplified by CLIP (40), ALIGN 23 and FILIP [51] These models employ separate encoders for text and image data, optimizing them through contrastive objectives to align semantically similar features across heteroge- neous modalities. Primarily, VLMs integrate transformer-base of architectures, encompassing convolutional neural networks [25] [ and vision transformers [2], are deployed for image encoding. The success of CLIP-like models has spurred numerous subsequent inquiries, with a focus on enhancing data efficiency and adaptability for diverse downstream tasks.
encoders for text data, while variety
cuttof-distributtion detection. Traditionally, OOD detection has evolved by defining post- hoc OOD scores [18 32 27
V-1
14-1
A Photo
(34) or formulating learning algorithms based on outlier exposure methods [26] 20 [0]. With the advancement of VLMs, methods for OOD detection that lever- age both image and text embeddings have also progressed. Post-hoc OOD score methods based on VLM typically involve utilizing the OOD class name [12 [[]] defining OOD scores us- ing the top similarity values between images and class names (37]. In the case of outlier exposure, which requires a training algorithm, some approaches employ prompt learning or fine-tune the image encoder (455 of models like CLIP. Transitioning from conventional methods to VL. -bsed approaches, none of these meth- ods have attempted text-only training and sub- sequently applied their techniques to tasks such as hateful image detection.
1
of (Dog)
Decision Boundary (from Texts)
24+
Photo of (Cat)
0
V+1 |P
Figure 2: Overview of Section 3.1 The red and blue col- ors symbolize the two classes -1 and +1, respectively. In our theoretical model, and can be interpreted as text and image, respectively.
Text-only training for vision tasks. Given the progress in VLMs, there have been numerous studies aimed at replacing images with textual representations in vision and multimodal tasks. Textual data presents the advantage of being easily collectible compared to visual data. Previous studies have demonstrated the effectiveness of using only textual information for various vision tasks, including image classification [8) image captioning [3I], and multi-label image recognition [4] Our work represents a pioneering effort in applying text-only supervision to unwanted visual data detection.
3

---

3 Method
In this section, we propose new textual training method for the convenient and successful removal of unwanted visual data. In Section 3.1. we theoretically demonstrate, through motivating example, that when there is well-trained model on a bimodal dataset, such CLIP, is possible to train a binary classifier successfully partitioning data from one mode using only data from the other. This theoretical revelation leads us to our novel loss function that allows hassle-free training of an unwanted visual data detector in Section 3.2 Lastly, in Section 3.3 we present our proposed method that includes a simple yet effective synthesis of textual data. The proposed method executable even without access to the parameters of the backbone model, making it lightweight and applicable to black-box foundational models.
3.1 A Motivating Example
We theoretically demonstrate that when there exists well-trained bimodal model F G x 7 - Z for given bimodal data distribution, it possible train a classifier successfully partitioning data from one mode (H) using only the dataset from the other mode (9). To align with the operations of VLMs like CLIP our scenario, we assume that the output vectors of F are normalized. We define bimodal dataset D as follows:
D = [[gi, his v:)) where u.4.7 [-1,+1} and (g, h) i.i.d. Gy x Hy.
(ge, h2) represents the input vectors from the two modes for given data sample, where yi denotes the binary class of the i-th sample. We can partition the dataset D as follows:
D = D-1 U D+1, where D, = ((,,, y = 3/
We assume that samples belonging to the same class in the dataset D exhibit similar semantic patterns. Given F' that successfully builds the joint embedding space for the bimodal data distribution, we can posit the following:
1410+11 u U- and wI wI1041 < -uIv-- where By
E
[u]
Uy
E Vv
[v],
MEU,
(F (9.) 1as E DyJ. and Vg, {F (h:) (y Dy). = (F
U, and Vy are class- embedding sets for each of the two modes, respectively. For simplicity. we assume that the variances of the angular distributions relative their mean vectors are equal for sets U-1 and U+1, well as for sets V-1 and V+1.
We investigate whether the cosine-similarity classifier 0* trained solely on the unimodal dataset (gi, v using can successfully be applied to (h;; y:) We establish the following theorem:
Theorem For the quadratic loss function L (u, y/5 (1 y1y u)² the optimal cosine-similarity classifier 0** that classifies sets U-1 and U+1 is
arg min 0
[T. (u, 1100]+ + uEU
E wEU+1
(u, +1:0)] U4 11
|1+1
Proofs are Appendix A Theorem demonstrates the optimal classifier 0* orthogonal to u + u+1. We present an illustration in Figure 2 enhance understanding of both the problem under investigation and the results our analysis.
Applying the classifier 6* trained to classify U-1 and U+1 to distinguish between V-1 and V41 leads the following:
Corollary 1. The classifier 0*, with respect to V-1 and V41, satisfies the double inequalities of
E
< <
6
vEV
This implies that we can successfully classify V-, and V+ by observing the cosine similarities with 0*. Motivated by these theoretical examples, we hypothesize that classifiers obtained solely using textual data can operate on visual data as well. Section eemprrically demonstrates that the arguments developed based on thcoretical model can be applied modern machine-learmin settings.

---

3.2 Our Proposed Loss Function
Our objective is to distinguish in m--iisribbttion data samples (Din), conforming to given data distribu- tion, from out-distribution data samples ( (Dou)). ). The development of our new loss function begins with defining the binary cross-entropy loss as follows:
1 + logp (u) 1
L(u v)
log p(u))
2
We employ the notations introduced in Section 3.1 plu) denotes the probability that the label of an embedding is +1, where +1 signifies an out-distribution. With respect to datasets U-1 and U+1, we minimize
Σ wEU-11
AL -1,
Σ (1- (u, wEU-11
(1)
We introduce hyper-parameter, À E [0, 1], adjust the balance between in-distribution learning (the first term) and out-distribution learning (the second term). Equation (T) can be reformulated as
-u.-1)-
Σ (1- WEU-1
Σ L MEU
WEU-1
The second term can be understood as regularization for in-distribution learning. As approaches in-distribution learning is more heavily impeded. Rather than employing the original regularization term, Enev_, (1 L (u, 1), we propose changing it to
Σ (1-)) NEUL1
Before analyzing the significance of this modification to the objective function, we first examine the effects resulting from this change. Along with the modification, objective function can be formulated as follows:
Σ WEU-1
Lu, +1)+
Σ (1- L
Σ uEU+1
uEU-1
Σ L 11,+
Σ
(1 (1- L
100
To minimize Eq. 2. it is imperative to distinguish between the in-distribution dataset and the out- distribution dataset. In-distribution data aligns with the objective of the given task, and any data not included in it becomes out-distribution data. However, in real-world scenarios, distinguishing between these distributions not straightforward.
instance, we consider as the text embedding space, collecting out-distribution texts for given set of in-distribution texts involves considerations such as homonyms, synonyms, and various forms of linguistic variations. Particularly, tasks where the boundaries between in-distribution and out-distribution are ambiguous, as seen in challenges such as hate content detection [33, constructing a dataset for Eq. becomes difficult and requires considerable human labor. However, the utilization of Eq. 3 alleviates us from such challenges. In other words, the union of the two sets U-1 U42 in Eq, 3 allows us to treat all data samples as out-distribution without the need to ponder their relationship with the in-distribution, providing solution to the intricacies involved in dataset construction. The distinction between Eq. 2 and becomes evident when comparing the gradient signals produced by the two different regularization terms. Gradients of the regularization terms in Eq. and can he computed as follows:
ØL 0L (u, -1)
(original)
Σ
Σ
0p(u)
1
p(u)
NEW
WEU/
(proposed)
0L (u,+1)
Σ
-1
Op(u)
p(u)
uEU-1
The original regularization term weakly regularizes in-distribution samples that were sufficiently learned by the model (i.e., samples with low p(u)). However, the proposed regularization term does the exact opposite; it imposes stronger regularization on in-distribution samples with low
5

---

p(u). In essence, our proposed regularization prevents the model from exhibiting high confidence in in-distribution samples and enforces the decision boundary be created near the in-distribution.
Recent studies show that the closer the decision boundary of the out-distribution data detector is to the in-distribution, the more effective the detector is at identifying various out-distribution data 126 19 10 33]. Subsequent research efforts have been directed at obtaining out-distribution samples that reside close to the in-distribution while training a detector to bring its decision boundary closer to the in-distribution. For instance, Lee et al. [26] utilizes a generative adversarial network to acquire samples placed on the in in-distribution boundary. Du et al. [0] models the in-distribution using Gaussian distribution and samples embeddings from the low-likelihood regions of the defined Gaussian distribution. Likewise, we focus on training samples situated in the region close to the in-distribution by incorporating an additional focal loss.
The focal loss was initially proposed to forcefully suppress the gradients for background pixels, which dominate the image, and intensify the learning signals from foreground pixels. Under our scenario, the in-distribution, like foreground pixels, tends to inhabit small portion of the entire embedding space. In light of the similarity between the in-distribution and foreground pixels, we utilize the focal loss restrain the loss from far out-distribution samples and amplify learning signals from samples near the in-distribution. The proposed loss can thus be defined as:
Let B-1 = { and B { denote mini-batches that are respectively drawn from the specified in-distribution and the overall data distribution. Let L be the cross-entropy loss. Then, our proposed loss function
Definition 1.
Naj and aj = -1 p1-)(1,))) (4)
Σ Z,EB_1
-1
X)
Σ ZyEB
Bj - Σx,eB 0k
p (z) is the predictive probability of z belonging in out-distribution. 7 2 0 is treated as hyperpa- rameter of the focal loss. In Section we compare the results of using loss terms in Eqs. 2 [lddd] 3 and demonstrate the particular effectiveness of the focal loss.
3.3 Hassle-Free Textual Training (HFTT)
So far, we have assumed access data sampled from the out-distribution. However, we may not always be able to anticipate the out-distribution in advance, and even if we can, sampling a subset of data that is representative of the entire distribution is not straightforward problem in nature. In our scenario, we solely utilize textual data to learn an unwanted visual data detector. Therefore, the proposed scenario requires texts to define the in-distribution and a comprehensive corpus of textual data that can replace the entirety of visual data. VLMs, such as CLIP, obtained impressive zero-shot classification accuracy on diverse visual data benchmarks through the usage of prompts, e.g., "a photo of )." Inspired by the success of prompting in VLMs, we conjecture that all visual data can be expressed textually through prompts. This assumption allows the textual dataset to replace the unknown visual data distribution by integrating words associated with the visual data into prompts, drastically simplifying the process of textual data sampling in our method. One example of a prompt design utilized in our approach is: "This is photo of a ()." To emulate the effect of using the entire visual data distribution, we adopt a word se that includes approximately 370k English words. We report the results of using other prompt designs textual data acquisition processes in Appendix C While the optimization procedure in our method additionally requires in-distribution textual data according to Eq. 4 these can be obtained with minimal effort by creating arbitrary sentences or prompts related to the given task. Section I details how in-distribution textual data are attained for each experimental setting. Even though the task of unwanted visual data detection is type of binary classification problem. learning a plain linear classifier in the embedding space of pre-trained VLMs through approaches like linear probing is not necessarily compatible with our task because the in- and out-distributions are not expected to be linearly separable. To accurately estimate the probability that input belongs in the out-distribution p (z), we must take advantage of the informative signals in the text encoder of pre-trained VLMs. In our method, p (x) is computed as follows:
1. Obtaining embeddings {wi") ) for K texts that effectively represent in-distribution visual data is equivalent to defining the task. This process is akin to obtaining zero-shot classifiers using VLMs.
We will refer these text embeddings as task embeddings.
https:: / htt.a//ww.w...ooooeeeeeeoooooooa

---

Algorithm 1 Hassle-Free Textual Training (HFTT)
Require: word set W, prompt templates P. in-distribution textual data G-1. task embeddings
[wi") trainable embeddings {w(n)) N
f=1. pre-trained model F, hyper-parameter
1: Initialize the trainable embeddings f 1j=1
2: for mini- bini-batches (B-1, words) ) ~ (G-1, do
3: B 4- wwwdddatt
P) # textual data synthesis (the entire data distribution)
4: Compute the proposed loss by Eq. f
5: Update {uRuE L. # the incurred cost is negligible
j==1
6: end for
7: Output: out-distribution data detector (F, {u: Je {ug 1==1)
2. With a pre-trained vision-language model and the set of N trainable embeddings {wi²) 1 1j=1 defined in the joint embedding space, p (x)) is obtained as
N Σド= exp
(F (Ix) (
ΣR exp (x))
+ j=1 exp F (x) with w."
Our method minimizes the custom loss defined in Eq. 4 by learning {wi Jj=1 only with textual data with the task embeddings {u:") t=1 and the model F kept frozen. Because the trainable embeddings are tuned in the output space of the backbone network, the proposed method results in little memory need to access the parameters of the backbone network
and computational cost. Furthermore, makes the proposed method extensible to black-box foundation models. The overall procedure of the proposed method is summarized in Algorithm
4 Experimental Results and Discussion
4.1 Experimental Setup
We complement analysis with studies conducted OOD and hateful image detection. For the OOD detection task, ImageNet-Ik 1k [7] is treated as in-distribution, and the following datasets are used as out-distribution data: iNaturalist [47), SUN (50). Places (52), and Textures [5]. We specifically
utilize OOD datasets that are carefully curated to be disjoint from ImageNet, as described in [22]. For hateful image detection, we utilize a dataset containing 892 Antisemitic/slamophhhi images and 420 phrases (Hate) 13]. The Hate dataset is a human- annotated dataset whose usage is limited to individuals with academic purposes prevent its unethical and unregulated use.
We adopt CLIP, the most extensively studied VLM, specifically using ViT-B/16 as the vision back- bone. Unless specified otherwise, we the batch size=256, learning rate=1.0. epoch=1, 1=1.0 (the focal loss hyper-parameter) A=0, and N=10 (the number of trainable embeddings) for all experi- ments. Note that the majority scenarios, there a substantial predominance out-distribution textual data compared to in-distribution textual data, and our approach involves mini-batch sampling. Consequently, given the rarity of in-distribution data sampling, the training on in-distribution data remains largely unaffected even though À = 0. All values presented in the tables of this paper are the average results over five runs. We conduct a comparative analysis of our approach against existing methods requiring in distribution images, namely Mahalanobis (27). MSP [18. KNN [44, and NPOS
46 Additionally, we include methods that do not necessitate in-distribution data, Energy (34 ZOC
11 Maxl [21], and MCM [36], in our comparison. The evaluation is performed using OOD scores proposed by the aforementioned works, as well as the scores introduced in this paper (refer to
p (x:) in Sec 3.3). The calculation of Area Under the Receiver Operating Characteristic (AUROC)
and False Positive Rate at 95% True Positive Rate (FPR95) is based on these scores.
HFTT training and inference costs. The backbone model for HFTT remains untrained, while only trainable parameters (trainable embeddings) defined in the model output space are updated. Thus, the cost of this update process is almost equivalent to the forward propagation cost of the synthetic textual data. For the corpus (~<30k samples) and CLIP utilized in the experiments, the update process takes less than 2 minutes with single V100 GPU, During inference time, the computation of HFTT
7

---

Table 1: Comparison of HFTT and competitive baselines with and without in-distribution image requirements on the ImageNet-II IK dataset. The best and second-best results indicated in bold and underlined, respectively. Our method surpasses even strong baselines that utilize in-distribution images. This complements our analysis Section 3.] demonstrating that textual data can substitute for visual data in such tasks.
OOD Dataset
Average
Method
iNaturalist
SUN
Places
Texture
AUROC
FPR
AUROC
FPR
AUROC
FPR
AUROC
AUROC
In-distribution images required
Mahalanobis
99.33
$5.89
99.41
59.94
98.54
65.96
98.46
64.23
98.94
61.51
MSP
40.17
89.76
63.99
79.40
63.50
80.19
67.01
79.33
58.67
82.17
KNN
29.17
94.52
35.62
92.67
39.61
91.02
64.35
85.67
42.19
90.97
NPOS
16.58
96.19
43.77
90.44
45.27
89.44
46.12
88.80
37.94
91.22
In-distribution images not required
Energy
34.70
90.55
32.33
90.58
40.29
89.32
51.24
72.36
39.64
85.70
MaxLo
35.03
89.46
32.86
90.33
41.15
89.60
68.17
75.63
44.30
86.26
ZOC
87.30
86.09
81.51
81.20
73.06
83.39
98.90
76.46
85.19
81.79
MCM
34.33
91.36
32.27
91.86
47.48
88.68
50,90
87.52
41,25
89.86
HFTT (ours)
27.44
93.27
19.24
95.28
43.54
90.26
43.08
88.23
33.33
91.76
involves obtaining cosine similarities between trainable embeddings and input embeddings. This cost amounts to (batch size) (embedding dimension) x (the number of trainable embeddings) FLOOS, which is negligible compared to the inference cost of the entire model.
4.2 Out-of Distr--Disttiittionn Detection
In OOD detection experiments, we utilize weights of zero-shot classifiers of pre-trained VLMS as task embeddings for HFTT. In- distribution textual data are obtained via com- binations of various prompt templates and class names of in-distribution data, which, in our ex- perimental setting, are equivalent 1,000 Ima- geNet classes. We adopt the prompt set released by OpenAl for prompt ensembling templates. The comparison results reported in Table I Despite the simple and lightweight nature of the proposed method, it outperforms even strong baselines that utilize images, on av- erage.
Trained embeddings
Task embeddings ImageNet
prompt
To understand how the trained embeddings pro- vide the task embeddings with informative sig- nals for identifying OOD data points, we visu- ally analyze the joint embedding space of CLIP Figure Even though visual OOD data were not involved in the training process, the trained embeddings are positioned on a sub-region of the embedding space inhabited by OOD data and thus can function as additional pointers for where OOD data lie in the joint embedding space. Therefore, this deliberate positioning of trained embeddings precludes the task embeddings from accidentally confusing out-distribution data as in-distribution by refining the decision boundary to intricately separate in-distribution and out-distribution regions.
Figure 3: UMAP visualization the joint embed- ding space of CLIP. The dispersed, transparent markers represent the OOD data samples used in our experiment (iNaturalist: brown; SUN: grey; Places: pink; Texture: purple; NINCO 3 red). The trained embeddings (bluc stars) are located in sub-region of the embedding space occupied by OOD data. We trained 2000 embeddings for this plot. It is important note that these trainable embeddings did not incorporate any information about the OOD data during their training time.
htttps://gitie http://wwww..cooo///oo/cccc/oe/e

---

Table 2: Comparison of HFTT with state-of-the-art methods for OOD detection that do not require in-distribution images, conducted the Hate dataset. The hest result in each column is in bold. HFTT outperforms baseline approaches. showing that can effectively be used for the general purpose of unwanted data detection.
Innocuous Dataset
Average
Method
iNaturalist
SUN
Places
Texture
NINCO
FPR
AUROC
FPR
AUROC
FPR
AUROC
FPR
AUROC
FPR
AUROC
FPR
AUROC
Energy
12.30
97.53
3.88
98.51
13.87
97.40
39.70
94.98
26.89
96.12
17.43
97.
MaxLogit
23.65
96.89
18.49
97.49
27.84
96.48
33.33
96.00
33.03
95.99
25.82
96.71
ZOC
87.76
71.05
66.51
85.23
69.96
82.57
65.48
83,22
81.06
78.36
74.15
84.09
MCM
80.53
76.70
87.54
69.38
81.37
74.12
60.39
84.97
81.95
78.00
77.45
76.29
CLIPN
47.71
92.78
36.36
95.16
40.62
94.52
53.36
92.36
68.40
89.58
49.29
92.88
NegLabel
0.03
99.84
1.09
99.10
5.16
98.50
3.56
98.82
12.62
97.86
4.49
98.82
HFTT (ours)
0.17
99.44
1.05
99.13
4.38
98.60
1.73
99.08
4.18
98.52
1.83
99.06
This visualization result can be attributed to two methodological characteristics that are unique to our method. First, our method directly optimizes the trainable embeddings and is not bounded by the modality gap between texts and images. Second, our method successfully places the trained embeddings on top of OOD data only through textual data; this consolidates that textual data can replace visual data, providing strong empirical support for the theory presented in Section 3.1 Together, these two aspects of HFTT yield the joint embedding space as illustrated in Figure
4.3 Hateful Image Detection
In this task, the hateful data that contains offensive content against Muslims and Jews is treated as in-distribution data, whereas innocuous data void of such content is treated as out-distribution data. Consequently, embeddings of distinct phrases from a collection of offensive and hateful phrases, provided as part of the Hate dataset, are utilized as task embeddings. The entire set of offensive and hateful phrases is employed in-distribution textual data.
The Mahalanobis, MSP, KNN, and NPOS methods necessitate the construction of an mi-iistrbbution image dataset. Therefore, they should not be used as methods for unethical image detection tasks such as hate image detection, as doing so would require the construction of unethical image datasets, leading to numerous ethical problems such as direct or indirect leakage of sensitive information. In contrast, HFTT requires no usage of any image, thus can be applied to any unethical image detection task without ethical concerns. To highlight the differences between traditional OOD detection methods and HFTT, we include two additional baselines (49 24
and one extra dataset 3.
In Table 2. we can observe that most OOD detection methods show significantly lower performance compared to HFTT. These results arise because OOD detection methods assume classification prob- lem with clear distinctions between classes. In tasks dealing with abstract concepts, the boundaries between data clusters within the in-distribution are ambiguous, which results in the underperformance of existing OOD detection methods. NegLabel shows different results compared to traditional OOD detection methods but still falls short of our proposed approach. We provide a further comparison of our method to CLIPN 44 and Label (24] in Appendix B
To further study the generalizability of HFTT, observe the effectiveness of HFTT in low-quality [48. The findings demonstrate the
image detection [17] and within the medical image domain |6
potential extension of HFTT's applicability beyond conventional OOD detection tasks. The results of
these experiments and an ablation study on hyper-parameters are provided in Appendices B and
5 Conclusion and Limitation
In this paper, we proposed novel methodology for identifying undesirable content hidden within visual datasets. Close theoretical scrutiny of the joint embedding space of VLMs led to the devel- opment HFTT, an efficient framework for training detectors to automatically identify unwanted visual content by leveraging solely textual data together with pre-trained VLMs. HFTT is comprised of a creative objective function that markedly diminishes human involvement in data annotation and the textual data synthesis technique in HFTT that can simulate the usage of unknown visual

---

data distributions in the training process without additional cost. The distinctive attributes of HFTT broaden its upplicability from a clearly-scoped OOD detection task to a far more general set of tasks that are more abstract. Because HFTT requires some type of VLM as the base model, its capabilities are bounded by the representative capacity of pre-trained VL.s.. This dependency on pre-trained VLMs makes the use of HFTT in tasks that VLMs struggle with challenging.
Impact Statements. This paper contributes to the growing field of data curation and selection research. As datasets for training large Al models expand without adequate safeguards, identifying unwanted data points, such as biased or offensive content, from training datasets is becoming crucial. We believe our work will make positive contribution this area, opening up new possibilities for the effortless removal of unwanted visual data. While our method could potentially be misused for content censorship, we believe the positive impact it provides significantly outweighs these concerns.
Acknowledgments and Disclosure of F Funding
This work was supported by Institute of Information & communications Technology Planning & Eval- uation (IITP) grant funded by the Korea government (MSIT) Noo.RS-2021-11211 1343, 2022-0-00959, RS- 1-2220122209555 Artificial Intelligence Graduate School Program (Seoul National University)]. ) the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (No. SABRRAA3B1777222 0022RAA5A7839000 II), and the BK21 FOUR program of the Education and the Research Program for Future ICT Pioneers, Scoul National University in 2024.
References
[1] Alessio Ansuini, Alessandro Laio, Jakob H Macke, and Davide Zoccolan. Intrinsic dimension of data representations in deep neural networks. Advances in Neural Information Processing Systems, 32, 2019.
[2] Abeba Birhane, Sanghyun Han, Vishnu Boddeti, Sasha Luccioni, al. Into the laion's den: Investigating hate in multimodal datasets. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023.
[3] Julian Bitterwolf, Maximilian Mueller, and Matthias Hein. In or out? fixing imagenet out- of-distribution detection evaluation. In ICML, 2023. URL https: htttpp::/pproceedingg mllr
[4] Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005. 14166, 2020.
[5] Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In 2014 IEEE Conference on Computer Vision and Pattern Recognition, pages 3606-3613, 2014. doi: 10.1 L09/C//R.2004.445
[6] Noel Codella, Veronica Rotemberg, Philipp Tschandl, M Emre Celebi, Stephen Dusza, David Gutman, Brian Helba, Aadi Kalloo, Konstantinos Liopyris, Michael Marchetti, et al. Skin lesion analysis toward melanoma detection 2018: A challenge hosted by the international skin imaging collaboration (isic). arXiv preprint
2019.
[7] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. ImageNet: A Large-Scale Hierarchical Image Database. In CVPR09, 2009.
[8] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint 1xXXv:1810004805 2018.
[9] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x 16 words: Transformers for image recognition at scale, arXiv preprint 2020.
10

---

[10] Xuefeng Du, Zhaoning Wang. Mu Cai, and Yixuan Li. Vos: Learning what you don't 't know by virtual outlier synthesis. arXiv preprint arXiv:2202.01197, 2022.
[11] Sepideh Esu, Bing Liu, Eric Robertson, and Lei Shu. Zero-shot out-of-distribution detection based on the pre-trained model clip. In Proceedings of the AAAI conference artificial intelligence, volume 36, pages 6568-6576 2022.
[12] Stanislav Fort, Jie Ren, and Balaji Lakshminarayanan. Exploring the limits of out-of-distribution detection. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, 2021. URL hettps://openreviees net/
Forum?1G==1MrrNBEEE
[13] Felipe González-Pizane and Savvas Zannettou. Understanding and detecting hateful content using contrastive learning. In Proceedings of the International AAAI Conference on Web and Social Media, volume 17, pages 257-268, 2023.
[14] Zixian Guo. Bowen Dong, Zhilong Ji, Jinfeng Bai. Yiwen Guo, and Wangmeng Zuo. Texts as images in prompt tuning for multi-label image recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2808-2817, June 2023.
[15] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In European conference computer vision, pages 630 645. Springer, 2016.
[16] Xuehai He, Yichen Zhang, Luntian Mou, Eric Xing, and Pengtao Xie. Pathvqa: 30000+ questions for medical visual question answering. arXiv preprini arXiv:2003. /0200/0000200 10286, 2020.
[17] Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. arXiv preprint 19Xiv:1903./2261 2019.
[18) Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and -uttr--disttiiuution examples neural networks. International Conference on Learning Representations, 2017.
URL / /wwpe/eieeeeeeeeeeeeeeeeee h htt://oormmm1dddkk4111
[19] Dan Hendrycks, Mantas Mazeika, and Thomas Dietterich. Deep anomaly detection with outlier exposure. arXiv preprint 1rXiv:1812.046000 2018.
[20] Dan Hendrycks, Mantas Mazeika, and Thomas Dietterich. Deep anomaly detection with outlier exposure. In International Conference on Learning Representations, 2019. URL
net worum /twp://rrrd.llyxxxcciiYcYc
[21] Dan Hendrycks, Steven Basart, Mantas Mazeika, Andy Zou, Joseph Kwon, Mohammadreza Mostajabi, Jacob Steinhardt, and Dawn Song. Scaling out-of-distribution detection for real- world settings. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song. Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 3759-8773, PMLR, 17-23 Jul 2022. URL htttht:p///// htttps::/proceeddnng
mlr www.presssseee https///111///eeecccce22222/
html
[22] Rui Huang and Yixuan Li. Mos: Towards scaling out-of distribution detection for large se- mantic space. In Proceedings the IEEE/CVF Conference Computer Vision and Pattern Recognition, 2021.
[23] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In ICML, pages 4904-4916 PMLR, 2021.
[24] Xue Jiang, Feng Liu, Zhen Fang, Hong Chen, Tongliang Liu, Feng Zheng, and Bo Han. Negative label guided ood detection with pretrained vision-language models. arXiv preprint arXiv:2403.20078. 2024.
[25] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Advances neural information processing systems, 25, 2012.
[26] Kimin Lee, Honglak Lee, Kibok Lee, and Jinwoo Shin. Training confidence-caliivaaeed classifiers for detecting diitr---isttibutionn samples. arXiv preprint arXiv:1711.09335, 2017.
11

---

[27] Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for detecting out-of-distribution samples and adversarial attacks. In S. Bengio, H. Wal- lach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018. wttrrffils/pppp////ee 1010///flee// URL hgtp:///pppooceeee........e papp:
[28] Kyubum Lee, Maria Livia Famiglietri, Aoife McMahon, Chih-Hsuan Wei, Jacqueline Ann Lang- don MacArthur, Sylvain Poux, Lionel Breuza, Alan Bridge, Fiona Cunningham, Ioannis Xe- narios, et al. Scaling up data curation using deep learning: an application to literature triage in genomic variation resources. PLoS computational biology, 14(8)=e0006300, 2018.
[29] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre- training for unified vision-language understanding and generation. In International Conference on Machine Learning, pages 12888-12900 PML.R, 2022.
[30] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301. 12597, 2023.
[31] Wei Li, Linchao Zhu, Longyin Wen, and Yi Yang. Decap: Decoding CLIP latents for zero- shot captioning via text-only training. In The Eleventh International Conference on Learning Representations, 2023. URL
net.:iiiwweeeeeeee.ee.e..ee..eee fonttfforumtt um? LttMiiiiit
[32] Shiyu Liang, Yixuan Li, and R. Srikant. Enhancing the reliability of out-of-distribution image detection in neural networks. In International Conference on Learning Representations, 2018.
URL
net
[33] Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, pages 2980-2988, 2017.
[34] Weitang Liu, Xiaoyun Wang, John Owens, and Yixuan Li. Energy-based out-of-distribution detection. Advances in Neural Information Processing Systems, 2020.
[35] Leland McInnes, John Healy, and James Melville. Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint arXiv: 2018
[36] Yifei Ming, Ziyang Cai, Jiuxiang Gu, Yiyou Sun, Wei Li, and Yixuan Li. Delving into out-of- distribution detection with vision-language representations. In Advances Neural Information Processing Systems, 2022.
[37] Yifei Ming, Ziyang Cai, Jiuxiang Gu, Yiyou Sun, Wei Li, and Yixuan Li. Delving into out-of- distribution detection with vision-language representations. In Advances in Neural Information Processing Systems, 2022.
[38] Muhammad Jehanzeb Mirza, Leonid Karlinsky, Wei Lin, Horst Possegger, Mateusz Kozinski, Rogerio Feris, and Horst Bischof. LaFTer: Label-free tuning of zero-shot classifier using lan- guage and unlabeled image collections. In Shirty- Conference on Neural Information Pro- cessing Systems, 2023. URL
net http://www.ne//rmemedddddeeeeaee
jpH
[39] Sangha Park, Jisoo Mok, Dahuin Jung, Saehyung Lee, and Sungroh Yoon. On the powerfulness of textual outlier exposure for visual ood detection. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
[40] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning. pages 8748-8763 8763. PMLR, 2021.
[41] Patrick Schramowski, Manuel Brack, Björn Deiseroth, and Kristian Kersting. Safe latent diffusion: Mitigating inappropriate degeneration in diffusion models. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 22522-22531,
12

---

[42] Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filterdd 400 million image-text pairs. arXiv preprint 1..i:22/11002//11 14, 2021.
[43] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion- 5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 35:25278-25299 2022.
[44] Yiyou Sun, Yifei Ming, Xiaojin Zhu, and Yixuan Li. Out-of-distriuution detection with deep nearest neighbors. In International Conference on Machine Learning, pages 20827-20840. PMLR, 2022.
[45] Leitian Tao, Xuefeng Du, Jerry Zhu, and Yixuan Li. Non-parametric outlier synthesis. In The Eleventh International Conference on Learning Representations, 2023. URL https?
//ppenreview net wwtp/eeeeeiieaa..eeeeee.... http:/ooruunii--a-p///E/d//
[46] Leitian Tao, Xuefeng Du, Xiaojin Zhu, and Yixuan Li. Non-parametric outlier synthesis. arXiv preprint arXiv: 2023.
[47] Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. The inaturalist species classification and detection dataset. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8769-8778, 2018.
[48] Bastiaan Veeling, Jasper Linmans, Jim Winkens, Taco Cohen, and Max Welling. Rotation cquivariant cnns for digital pathology. In Medical Image Computing and Computer Assisted InterventionMMICAAI 2018: 21st International Conference, Granada, Spain, September 16-20, 2018, Proceedings, Part II 11, pages 210-218. Springer, 2018.
[49] Hualiang Wang, Yi Li, Huifeng Yao, and Xiaomeng Li. Clipn for zero-shot ood detection: Teaching clip to say no. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1802-1812, 2023.
[50] Jianxiong Xiao, James Hays, Krista A. Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pages 3485-3499 2010. doi: 10. 1109/CVPR.2010.553399
[51] Lewei Yao, Runhui Huang, Lu Hou, Guansong Lu, Minzhe Niu. Hang Xu, Xiaodan Liang, Zhenguo Li, Xin Jiang, and Chunjing Xu. Filip: Fine- time-grained interactive language-image pre-training. In International Conference on Learning Representations, 2021.
[52] Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A
10 million image database for scene recognition. IEEE Transactions on Puttern Analysis and Machine Intelligence, 2017.
13

---

A Proofs
Theorem I. For the quadratic loss function I. (u, 0: (1- 11-y6 y00 u) the optimal cosine-similarity classifier 8 that classifies sets U-1 and U+1 is
arg
LL (u, -1:0] + [L (u, +1:0]] uEU-1
wEU+1
min (1 + 0 w1n²² 2 + (1 0 u444 1441 u
#+1
0
Proof. Our assumption validates the following equations:
0 Tu u uTv 1, Σ+1 E-1 el,, ||+I|| = ||u-
where By and denote the covariance matrix of U, and the identity matrix, respectively, and >> 0 is a constant. Then, arg min
E wED-1
[L (u, 1;0)]+ +
E uEU+1
[L (u, +1;0]]
0
arg min E
(1 + OT ()
+ E
(1 11
0 OTu) =)²
WEU-1
uEU+1
arg min
E
[1 0 u]
E uEU+1
[1- [1-0 -Tu) u)]
0+0 + 1
NEU
arg min
E
[1 + 1 u]
E
[] 11-0 w]
+ 2e
0
uEU-1
uEU+1
arg min (1 + 0 u-
+
0
u4+1
The gradient of the objective function with respect to 0 is
-2 (1 0 u-1) u-1 + 2 (1 + H u+1) u+1
Therefore, the optimal cosine-similarity classifier 0 satisfies the following equation:
1 0- __ u-1 (1 0¹ (+1) u41
Then,
1 1 0+ 1 + 0² u+1 or 0r #- = -1 0" u41
The second equation is not true for any u and u+ Hence,
0t u+ 1 u Ill
|141
Corollary 1. The classifier 8", with respect to V-1 and V+L, satisfies the double inequalities of
0 <
vEV-1
vEV41
Proof. Based on the inequalities
u 141 > _II 1-1 and I II u+1 < u -uIV-I- U-1
E VEV-1
0
T
0
T
1_
0 < E
TT v)]
u 1 U41 - -I, 141
u41
u-1
vE441
U411 u-
B Additional Experiments
Low-quality image detection. We additionally demonstrate the applicability of our method, HFTT. in detecting low-quality images, which are commonly unwanted visual data beyond OOD and hate images. Specifically, we assume the task of detecting corrupted images lurking within a raw visual dataset consisting of 1000 ImageNet classes. For this experiment, we employ ImageNet and ImageNet-C as in-distribution and out-distribution data, respectively. As shown in Table HFTT consistently surpasses existing methods in the detection of corrupted images.
14

---

Table Comparison of HFTT with baselines the low-quality image detection.
Method
FPR
AUROC
MSP
64.17
83.94
Energy
99.99
09.
MaxLogit
78.01
68.47
MCM
51.54
89.06
HFTT (ours)
42.13
92.81
Table 4: Comparison of HFTT with MCM on the medical image datascts.
OOD Dataset
Method
PVQA
PCAM
FPR
AUROC
AUROC
MCM MCM + description
95.44
47.10
71.49
68.74
86.84
60.39
84.50
43.44
HFTT
22.58
93.60
8.07
96.94
HFTT description HFTT + description corpus engincering
13.72
97.05
4.95
98.35
6.24
98.69
4.33
98.73
Medical image domain. We compare the performance of HFTT and MCM the medical image domain. Specifically, we treat the ISIC-18 skin lesion diagnosis dataset [6] in-distribution and the PathVQA [16) and PatchCamelyon [48] datasets as out-distribution. The ISIC-18 skin lcsion diagnosis dataset is an image classification benchmark for seven skin discase categories. We apply MCM and HFTT to CLIP on the seven disease categories. YableeJrevent a significantly low detection performance of MCM, attributed to the limited medical domain knowledge of CLIP. Even appending descriptions (generated by GPT-4) to the disease names does not yield favorable results for MCM (+ description). In contrast, our proposed method achieves significantly better results by leveraging the model knowledge and the medical-related information in the corpus. The utilization medical- related information within the corpus by HFTT is evidenced by the fact that further improvements can be achieved by modifying the corpus to align with the medical domain (+ corpus engineering).
The experimental results on other pre-trained models. As discussed in Section 3.1 of our paper, our method assumes that text and image are well-aligned through contrastive learning, similar to CLIP. Therefore, if CLIP is used the vision encoder, the text encoder must also be CLIP. If text and image are well-aligned, our method can be applied models other than CLIP. Here, we provide the results for CLIP-L/14, BLIP-B/16, and BLIP-L/16 in addition to the CLIP-B/16 used in our study, Table ( that our method is effective across various vision-language models.
HFTT vs. CLIPN and NegLabel. NegLabel 24 constructs an OOD corpus by selecting texts distant from the in-distribution texts from predefined corpus, then compares the distances between the input image and those texts in the CLIP embedding space to detect OOD. While NegLabel shows high OOD detection performance on ImageNet (see Table 7. it has the following limitations compared to our method: 1) Although NegLabel does not require training additional parameters, must compute the embeddings of all texts in the corpus and measure their similarity to the in- distribution texts to find the optimal OOD corpus for given in-distribution. Our training method also requires nearly the same cost as obtaining the embeddings of all texts within predefined corpus and calculating the similarities between those embeddings and the task+trainable embeddings, as discussed Section 4.I Thus, NegLabel and our method require the same level of optimization cost; 2) Since NegLabel uses the embeddings of the determined OOD corpus as they are, it falls behind our method, which has trainable parameters, terms of generalization. To demonstrate this, we further compare our method and NegLabel in the medical image domain. Specifically, we treat the ISIC-18 18 skin lesion diagnosis dataset [1] as in-distribution and PathVQA (16) and PatchCamelyon 48 datasets as out-of-distribution The ISIC-18 skin lesion diagnosis dataset is an image classification benchmark for seven skin disease categories. Table illlustrates the limitations of Negl. in terms of generalization. While NegLabel fails to construct effective OOD corpus for the medical image
15

---

Table 5: Comparison of HFTT and competitive baselines on the ImageNet-I IK dataset. The best result in each column hold. Our method outperforms all baselines both variants of CLIP and BL.PP, demonstrating that can be used to improve the OOD detection performance various VLMs.
OOD Dataset Places
Model
Method
iNaturalist
SUN
Texture
NINCO
FPR
AUROC
FPR
AUROC
FPR
AUROC
FPR
AUROC
FPR
AUROC
MSP
34.63
91.35
32.06
91.86
47.62
88.86
49.78
87.65
72.83
71.98
Energy
34.70
90.55
32.33
90.58
40.29
89.32
51.24
72.36
70.06
73.85
CLIP-B
MaxLogit
35.03
89.46
32.86
90.33
41.15
89.60
68.17
75.63
68.96
74.24
MCM 34.33
91.36
32.27
91.86
47.48
88.68
50.90
87.52
73.26
71.98
HFTT (ours)
27.32
93.28
19.68
95,20
43.24
90.32
43.26
88.20
70.08
74.61
MSP
26.66
94.20
22.37
94.37
36,82
92.45
52.83
86.57
67.27
78.70
Energy
30.84
91.25
25.94
94.10
32.94
92.30
64.33
79.26
63.49
79.72
CLIP-L
MaxLogit
32.76
90.96
26.48
92.96
31.88
92.39
72.08
73.85
60.67
81.07
MCM 26.96
94.19
22.77
94.37
36.74
92.44
52.66
86.56
68.16
78.65
HETT (ours)
24.10
94.58
17.80
95.39
33,83
93.09
52.06
86.58
69.19
78.98
MSP 64.70
82.22
30.38
91.06
71.40
78.82
76.99
81.30
71.47
72.07
Energy
67.15
79.30
45.21
89.07
70.28
77.49
91.24
75.38
80.29
77.20
BLIP-B
MaxLogit
69.57
75.44
69.57
71.19
69.86
76.26
93.55
60.31
88.58
56.37
MCM
64.41
82.29
30.21
91.05
70.53
79.32
75.84
81.55
71.56
72.02
HFTT (ours)
63.28
82.22
19.16
95.12
68.48
79.50
63.74
84.53
72.12
73.86
MSP
51.20
87.91
22.37
93.86
61.63
84.68
64.85
85.28
65.96
78.29
Energy
45.63
87.23
33.94
90.29
55.73
85.91
72.38
82.16
71.23
77.49
BLIP-L
MaxLogit
44.59
86.94
35.56
86.45
50.96
86.46
86.38
71.22
79.78
67.59
MCM 50.75
88.03
22.34
93.88
60.88
85.38
64.71
85.39
66.04
78.32
HFTT (ours)
44.24
89.88
6.81
98.40
62.20
84.16
63.35
83.39
64.82
80.46
Table 6: OOD detection the medical image domain.
PVQA
PCAM
Method
FPR
AUROC
FPR
AUROC
CLIPN
35.47
84.64
3.10
98.76
NegLabel
37.44
94.11
48.07
94.86
HFTT (ours)
13.72
97.05
4.95
98.35
dataset, method achieves significantly higher performance by learning optimal embeddings for detection.
CLIPN utilizes additional "no" text encoder alongside CLIP. This additional text encoder predicts the probability that a given object is not present in an image. Thus, CLIPN predicts whether a out-distribution by using the original CLIP text encoder and the
given image in-distribution "no" text encoder to estimate the probabilities, respectively. Images with low probability of being in-distribution and high probability of being out-distribution are identified as OOD.
Although CLIPN achieves high OOD detection performance on ImageNet (see Table 7) has following limitations compared to our method:
1) CLIPN requires significantly higher inference costs due the of additional text encoder;
2) While our method requires lightweight training that does not involve images, CLIPN demands extensive and expensive training of the "no" text encoder on large lisionnlanguage datasets; 3) CLIPN can only be applied to tasks where the distinction between in-distribution and out-distribution is be
clear and straightforward, such as classification datasets. This is because all training images classified as either "yes" or "no" images. Therefore, it is unsuitable for tasks dealing with abstract concepts, such as hateful image detection, as discussed in Section 4.3 our paper; 4) Our method can be easily applied to any detection task defined in natural language, whereas CLIPN shows significantly degraded performance for in-distribution tasks that fall outside the training distribution the "no" text encoder. In terms of applicability, proposed method surpasses CLINN, To demonstrate this,
16

---

Table 7: OOD detection performance ImageNet in-distribution (average for Texture, Places, SUN, and iNaturalist).
Method
FPR
AUROC
CLIPN
31.10
93.10
NegLabel
25.40
94.21
HFTT (ours)
33.33
91.76
Table 8: Results of using different textual data synthesis methods, HFTT outperforms other methods that complex.
OOD Dataset
Average
Method
iNaturalist
SUN
Places
Texture
NINCO
FPR
AUROC
FPR
AUROC
FPR
AUROC
FPR
AUROC
FPR
AUROC
FPR
AUROC
WTO
29.26
92.23
23.31
93.93
40.18
90.77
42.96
88.01
69.07
76.54
40.96
88.30
CTO
27.53
91.66
38.41
86.78
43.31
89.31
60.13
79.64
76.25
69.08
49.13
83.29
DTO
29.32
92.32
24.57
94.04
43.37
89.63
42.26
89.34
70.36
74.76
41.98
88.02
Caption 54.07
79.49
57.15
74.23
63.44
77.04
41.21
91.12
89.30
56.42
47.65
86.28
Dedupl, 28.17
93.03
21.08
94.75
43.67
90.10
42.69
88.54
68.86
75.22
40.89
88.33
Ours 27.44
93.27
19.24
95.28
43.54
90.26
43.08
88.23
70.15
74.48
40.69
88.30
we further compare our method with CLIPN in the medical image domain. Table A illustrates the limitations of CLIPN in terms of generalization. While CLIPN effectively detects PCAM, it exhibits very low detection performance on PVQA. In contrast, our method achieves high performance on both OOD tasks.
C Ablation Study
In this section, we analyze how different components affect the performance of HFTT.
Textual data synthesis method. While we propose textual data synthesis method that incurs no additional cost, alternative approaches beyond this can also be explored. We apply the following methods in conjunction with HFTT and list their results in Table 8
WTO, CTO, DTO: Recently, Park et al. proposed a method that utilizes textual outliers instead of visual outliers in outlier exposure [19]. For experiments, use word-level textual outliers (WTO) generated using in-distribution images along with CLIP and BERT [8), caption-level textual outliers (CTO) generated by an image captioning model (3), and description-level textual outliers (DTO) created using a large language model.
2. Caption: We can consider the extensive use of image captions from LAION-400M 42 for the entire visual data distribution.
substitute
3. Deduplication: To experimentally compare Eq. and 3 applied HFTT after removing words from our word set that have meanings identical to ImageNet classes as much as possible.
Table 8 reveals that textual outliers generated using additional models and in-distribution images show comparable or inferior performance to our textual data synthesis method. Furthermore, the captions results suggest that heavily relying image captions does not effectively enhance the average detection performance for various OOD data. Lastly, there appears to be no discernible performance difference between the application of Eq. [ and This indicates that our proposed loss minimizes the need for human labor by eliminating the process of selecting out-of of-diittibution data without sacrificing performance.
The focal loss hyper-pane syper-parameter. HFTT incorporates the concept focal loss to shape the decision boundary of detectors near the in-distribution. We observe its effect by incrementally increasing the focal loss byper-parameter 7 from zero. Tamossddemnsstraees that using the focal loss (y 0) generally leads to better performance compared to not using (y 0).
17

---

Table 9: Results of using different values of for the focal loss. The performance of HFTT appears be relatively robust changes in the choice of 7, with the adoption of focal loss with y generally leading improved results.
OOD Dataset Places
Average
iNaturalist
SUN
Texture
NINCO
FPR
AUROC
FPR
AUROC
FPR
AUROC
FPR
AUROC
FPR
AUROC
FPR
AUROC
0
27.67
93.08
20.62
94.98
43.97
90.12
44.42
87.57
69.23
75.22
41.18
88.19
27.44
93.27
19.24
95.28
43.54
90.26
43.08
88.23
70.15
74.48
40.69
88.30
27.10
93.32
19.48
95.20
43.17
90.32
42.95
88.33
70.19
74.40
40.58
88.31
27.03
93.32
19.56
95.17
43.21
90.32
42.82
88.38
70.42
74.26
40.61
88.29
Table 10: Results of changing the temperature of the final Softmax layer.
OOD Dataset
Average
Temp.
iNaturalist
SUN
Places
Texture
NINCO
FPR
AUROC
FPR
AUROC
FPR
AUROC
FPR
AUROC
FPR
AUROC
FPR
AUROC
1.0
94.97
46.92
76.17
48.52
93.00
55.90
99.80
23.96
94.96
56.42
91.78
46.34
0.1
93.52
50.17
95.22
51.30
91.67
58.43
99.75
25.37
94.62
56.93
94.96
48.44
0.01
27.44
93.27
19.24
95.28
43.54
90.26
43.08
88.23
70.15
74.48
40.69
88.30
Temperature. In HFTT, a temperature parameter is utilized for computing p (x). CLIP learns a temperature parameter during their pre-training phase, and employ these same learned temperature values in all our experiments. Table illlustrates that modifying the temperature value may reduce the efficacy of HFTT.
The number of trainable embeddings. the dimensionality of the data manifold in the joint embedding space of VLMs is low, HFTT can be effective even with small number of trainable embeddings. To validate this, we present the OOD detection performance of HFTT in Table [T] illustrating how it varies with the number of trainable embeddings (N). Remarkably, HFTT can improve OOD detection performance on ImageNet, which has 1,000 classes, even with very limited number of trainable embeddings. This is possible due the inherently low dimension of data in the actual model output space [
Table 11: Results of shrinking or expanding the number of trainable embeddings (N).
OOD Dataset Places
Average
iNaturalist
SUN
Texture
NINCO
AUROC
FPR
AUROC
FPR
AUROC
FPR
AUROC
FPR
AUROC
FPR
AUROC
27.50
93.26
19.96
95.07
43.55
90.25
43.07
88.23
70.15
74.38
40.85
88.24
27.44
93.27
19.24
95.28
43.54
90.26
43.08
88.23
70.15
74.48
40.69
88.30
100
27.40
93.30
19.63
95.14
43.67
90.28
42.93
88.27
70.14
74.53
40.75
88.30
2000
27.32
93.28
19.68
95.20
43.24
90.32
43.26
88.20
70.08
74.61
40.72
88.32
18

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Abstract and Section Guidelines:
The answer NA means that the abstract and introduction do not include the claims
made in the paper.
The abstract and/or introduction should clearly state the claims made, including the
contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
Justification: Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't aecnowledgeed in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: Section and Appendix A
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
19

---

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: URL in Abstract and Section 4 Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results. 5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: URL in Abstract
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
public/ ceciicc/guiesseeee CCtp://odeieiiiicccc ss www.ssooosssoooooooooo
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
20

---

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https}
/ //nips wtt.cc/ppeeoo ptbllc/ggppioeeeep des CooeeSum ss onPol 1 icy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
reproducible, they
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes] Justification: URL in Abstract and Section 4
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: We report the results over five runs. The variances of the results are close to zero reporting them would be redundant.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi-
dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula,
call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [Yes]
Justification: Section 4.1
Guidelines:
The answer NA means that the paper does not include experiments.
21

---

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't Iodn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect. with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: Our submission conform, in every respect, with the NeurlPS Code of Ethics. Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there special consid-
eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Section
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. des.lomeetm of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following unintentional) misuse of the technology.
from (intentional
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (c.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA]
Justification: This paper does not release any data or model.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
22

---

Datasets that have been scraped from the Internet could pose safety risks. The authors
should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Section 4 and Appendix Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, papprr sepperswihhooe com/ cat asets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: URL in abstract and Section 4
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects. Guidelines:
The answer NA means that the paper does not involve crowdsoureing research with
human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
23

---

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects. Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
24


Reviews:
[{"review_id": "2gKg04c6FH", "reviewer": "Reviewer_CzuR", "summary": "This paper proposes an objective function for CLIP-based architecture to enhance out-of-distribution (OOD) detection. Instead of relying on OOD image data, the approach extracts OOD words from various sources and updates some trainable embeddings using predefined text embedding. Results show that the proposed approach outperforms previous methods that do not require in-distribution images and has comparable performance with methods that require in-distribution images.", "questions": "Authors should explain why the comparison with NegLabel and CLIPN is partial and the method are not even cited by the paper.\n\nIn addition authors should highlight advantages and disadvantages of their method which is in-domain image free versus approaches that are training free, such as NegLabel.", "limitations": "Authors addressed the limitations.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "\\+ This paper shows that is possible to learn OOD without the need of in-distribution images, which I think it is quite valuable and useful in practice\n\n\\+ The justification of the proposed idea seems well motivated and clearly presented", "weaknesses": "\\- The related work section lacks comprehensiveness. While the authors mention some CLIP-based methods, they overlook approaches closely related to their own. Notably, NegLabel (ICLR2024) and CLIPN (ICCV2023) also use textual features for OOD detection and are very similar to the proposed method. The authors should include these works in their related work section, provide benchmarks, and critically analyze the advantages of their approach compared to these existing methods.\n\n\\- In the experimental results, authors compare results with CLIPN and NegLabel only in Tab. 2, while they are not mentioned in Tab.1. Also, those two methods are not even cited nor explained in related work. I believe it might have been a last minute effort.\n\n\\- Some methods in the literature do not require any training (e.g. NegLabel). Given that the proposed approach involves additional training (even though without images), it is crucial to highlight the differences with these training-free methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Firstly, we express our gratitude for your thorough review of our manuscript. Particularly, we appreciate your recognition of our method as valuable and useful in practice, as well as your acknowledgment that the justification of our method is well-motivated and clearly presented. Furthermore, we appreciate your invaluable suggestions, which significantly enrich the substance of our work. In the subsequent sections, we meticulously address each of your concerns as follows:\n\n>**W1**. The related work section lacks comprehensiveness. While the authors mention some CLIP-based methods, they overlook approaches closely related to their own. Notably, NegLabel (ICLR2024) and CLIPN (ICCV2023) also use textual features for OOD detection and are very similar to the proposed method. The authors should include these works in their related work section, provide benchmarks, and critically analyze the advantages of their approach compared to these existing methods.\n\n**A1**. Thank you for your feedback on the two related studies. We can compare our method with each of these studies as follows. This discussion will be added to the final version of our paper:\n\n**Comparison with NegLabel**  \nNegLabel constructs an OOD corpus by selecting texts distant from the in-distribution texts from a predefined corpus, then compares the distances between the input image and those texts in the CLIP embedding space to detect OOD. While NegLabel shows high OOD detection performance on ImageNet (see Table B), it has the following limitations compared to our method:\n- Although NegLabel does not require training additional parameters, it must compute the embeddings of all texts in the corpus and measure their similarities to the in-distribution texts to find the optimal OOD corpus for a given in-distribution.\nOur training method also requires nearly the same cost as obtaining the embeddings of all texts within a predefined corpus and calculating the similarities between those embeddings and the task+trainable embeddings, as discussed in Section 4.1. Thus, NegLabel and our method require the same level of optimization cost.\n- Since NegLabel uses the embeddings of the determined OOD corpus as they are, it falls behind our method, which has trainable parameters, in terms of generalization. To demonstrate this, we further compare our method and NegLabel in the medical image domain. Specifically, we treat the ISIC-18 skin lesion diagnosis dataset [1] as in-distribution and the PathVQA [2] and PatchCamelyon [3] datasets as out-of-distribution. The ISIC-18 skin lesion diagnosis dataset is an image classification benchmark for seven skin disease categories.\n\nTable A: OOD detection in the medical image domain\n|    OOD:   | PVQA | | PCAM | |\n| ----------------- | ----- | ----- | ----- | ----- |\n| **Method**     | **FPR** &#8595; | **AUROC** &#8593; | **FPR** &#8595; | **AUROC** &#8593; |\n| NegLabel    | 37.44   | 94.11   |48.07   | 94.86   |\n| CLIPN  | 35.47   |  84.64  |3.10   | 98.76   |\n| HFTT (ours)   | 13.72   | 97.05   |4.95   | 98.35   |\n\nTable A illustrates the limitations of NegLabel in terms of generalization. While NegLabel fails to construct an effective OOD corpus for the medical image dataset, our method achieves significantly higher performance by learning optimal embeddings for detection.\n\n**Comparison with CLIPN**  \nCLIPN utilizes an additional \"no\" text encoder alongside CLIP. This additional text encoder predicts the probability that a given object is not present in an image. Thus, CLIPN predicts whether a given image is in-distribution or out-distribution by using the original CLIP text encoder and the \"no\" text encoder to estimate the probabilities, respectively. Images with a low probability of being in-distribution and a high probability of being out-distribution are identified as OOD.\n\nAlthough CLIPN achieves high OOD detection performance on ImageNet (see Table B), it has the following limitations compared to our method:\n\n- CLIPN requires significantly higher inference costs due to the use of an additional text encoder.\n- While our method requires lightweight training that does not involve images, CLIPN demands extensive and expensive training of the \"no\" text encoder on large vision-language datasets.\n- CLIPN can only be applied to tasks where the distinction between in-distribution and out-distribution is clear and straightforward, such as classification datasets. This is because all training images must be classified as either \"yes\" or \"no\" images. Therefore, it is unsuitable for tasks dealing with abstract concepts, such as hateful image detection, as discussed in Section 4.3 of our paper.\n- Our method can be easily applied to any detection task defined in natural language, whereas CLIPN shows significantly degraded performance for in-distribution tasks that fall outside the training distribution of the \"no\" text encoder. In terms of applicability, our proposed method surpasses CLIPN. To demonstrate this, we further compare our method with CLIPN in the medical image domain. Table A illustrates the limitations of CLIPN in terms of generalization. While CLIPN effectively detects PCAM, it exhibits very low detection performance on PVQA. In contrast, our method achieves high performance on both OOD tasks.\n\nTable B: OOD detection performance on ImageNet in-distribution (average for Texture, Places, SUN, and iNaturalist)\n| OOD:  | Average ||\n| ----------------- | ----- | ----- |\n| **Method**            | **FPR** &#8595; | **AUROC** &#8593; |\n| CLIPN    | 31.10   | 93.10 |\n| NegLabel  | 25.40  |  94.21 |\n| HFTT (ours)   | 33.33   | 91.76 |\n\n[1] Codella, Noel, et al. \"Skin lesion analysis toward melanoma detection 2018: A challenge hosted by the international skin imaging collaboration (isic).\" arXiv 2019.  \n[2] He, Xuehai, et al. \"Pathvqa: 30000+ questions for medical visual question answering.\" arXiv 2020.  \n[3] Veeling, Bastiaan S., et al. \"Rotation equivariant CNNs for digital pathology.\" MICCAI 2018.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ri9cFpBcqz", "reviewer": "Reviewer_LurW", "summary": "This paper focuses on textual training methods to remove undesirable (such as biased or offensive) visual content and proposes a method for detecting unwanted visual content using only synthetic textual data to partition visual data. The classifier trained on textual content can be successfully transferred to visual content. The method consists of an objective function and a textual data synthesis method. The design of the loss does not require data annotation and the textual data synthesis method can emulate unknown visual data distribution into the training process with no extra cost. The proposed method was proven to be effective for out-of-distribution detection and hateful image detection.", "questions": "1. This idea of training on textual content for transferring to visual content is interesting, could the authors provide more insights on more examples of applications of this method except for the OOD detection and hateful image detection? Is it possible to apply it to more binary classification tasks?\n2. What encoders are used to obtain the task embedding? The authors mentioned using CLIP as the vision backbone, and is the text encoder during training also from CLIP?\n3. How do you decide the number of task embeddings?\n4. The paper mentions using the proposed method to detect unwanted content and uses two tasks (OOD detection and hateful image detection. If I understand correctly, my key concern is that the focus of this paper is closer to a novel method for binary classification with an application that classifies unwanted content. Could the authors share more thoughts regarding this? If so, I would suggest slightly paraphrasing the introduction to better reflect the contribution and indicate more potential applications.\n5. Can this method be extended to multi-class classifications?", "limitations": "The authors adequately addressed the limitations.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. This paper is clear and well-written.\n\n2. The idea is innovative and solid with theoretical results. The proposed method is simple but effective and can be applied to even black-box foundation models.\n\n3. The experiments are comprehensive and the results showed the effectiveness of this method.\n\n4. The training and testing costs are discussed in this paper which could benefit the application of the proposed method.", "weaknesses": "I do not see a significant weakness in the paper since it’s well-organized. Several clarification questions are listed below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Firstly, we thank you for your thorough review of our paper. We particularly appreciate your recognition that our paper has no significant weaknesses and is well-organized. We have made our best efforts to address your remaining concerns as follows. If our responses meet your expectations, we would be grateful if you could consider reflecting this in your rating:\n\n>**Q1**. This idea of training on textual content for transferring to visual content is interesting, could the authors provide more insights on more examples of applications of this method except for the OOD detection and hateful image detection? Is it possible to apply it to more binary classification tasks?\n\n**A1**. Our method can be applied to more binary classification tasks. We additionally demonstrate the applicability of our method, HFTT, in detecting low-quality images, which are commonly unwanted visual data beyond OOD and hate images. Specifically, we assume the task of detecting corrupted images lurking within a raw visual dataset consisting of 1000 ImageNet classes. For this experiment, we use ImageNet and ImageNet-C. As shown in Table A below, HFTT consistently surpasses existing methods in the detection of corrupted images.\n\nTable A: Corrupted image detection\n|Method|FPR &#8595;|AUROC &#8593;|\n|-|-|-|\n|MSP|64.17|83.94|\n|Energy|99.99|09.16|\n|MaxLogit|78.01|68.47|\n|MCM|51.54|89.06|\n|HFTT (ours)|**42.13**|**92.81**|\n\n>**Q2**. What encoders are used to obtain the task embedding? The authors mentioned using CLIP as the vision backbone, and is the text encoder during training also from CLIP?\n\n**A2**. As discussed in Section 3.1 of our paper, our method assumes that text and image are well-aligned through contrastive learning, similar to CLIP. Therefore, if CLIP is used as the vision encoder, the text encoder must also be CLIP. If text and image are well-aligned, our method can be applied to models other than CLIP. To demonstrate this, we additionally apply our method to BLIP and observe the results. Table B shows that our method is also effective for BLIP.\n\nTable B: The experimental results on BLIP (in-distribution: ImageNet)\n|OOD:|iNaturalist||SUN||Places||Texture||NINCO|| \n|-|-|-|-|-|-|-|-|-|-|-|\n|**Method**|**FPR** &#8595;|**AUROC** &#8593;|**FPR** &#8595;|**AUROC** &#8593;|**FPR** &#8595;|**AUROC** &#8593; |**FPR** &#8595;|**AUROC** &#8593;|**FPR** &#8595;|**AUROC** &#8593;|\n|MSP|64.70|82.22|30.38|91.06|71.40|78.82|76.99|81.30|**71.47**|72.07|\n|Energy | 67.15 |  79.30 |  45.21 |  89.07 |  70.28  | 77.49 |  91.24 |  75.38 |  80.29  | **77.20**  |\n|MaxLogit |69.57 | 75.44 | 69.57 | 71.19 | 69.86 | 76.26 | 93.55 | 60.31 | 88.58 | 56.37 |\n|MCM| 64.41| **82.29** | 30.21 | 91.05 | 70.53 | 79.32 | 75.84 | 81.55 | 71.56 | 72.02  |\n|HFTT (ours)| **63.28** | 82.22 | **19.16** | **95.12** | **68.48** | **79.50** | **63.74** | **84.53** | 72.12 | 73.86   |\n\n>**Q3**. How do you decide the number of task embeddings?\n\n**A3**. The number of task embeddings is not a hyper-parameter but is determined by the in-distribution task. For example, if the in-distribution is ImageNet, the task embeddings are the text embeddings for the 1,000 ImageNet classes, such as [\"a photo of a tench\", ..., \"a photo of a toilet paper\"]. In the case of hateful image detection, we used representative hate phrases defined by the dataset as task embeddings. The number of trainable embeddings in our method is a hyper-parameter. We present an ablation study on this in Table 8 of Appendix C, and the results show that our method is not sensitive to this parameter.\n\n>**Q4**. The paper mentions using the proposed method to detect unwanted content and uses two tasks (OOD detection and hateful image detection. If I understand correctly, my key concern is that the focus of this paper is closer to a novel method for binary classification with an application that classifies unwanted content. Could the authors share more thoughts regarding this? If so, I would suggest slightly paraphrasing the introduction to better reflect the contribution and indicate more potential applications.\n\n**A4**. We are hesitant to consider our method as a novel approach for binary classification. Generally, binary classification is defined as the task of finding patterns that best distinguish between two different classes. However, in our scenario, we assume that only one (in-distribution) of the two classes has patterns, while we make no assumptions about the other class to cover all possible cases. Therefore, for binary classification tasks where both classes have specific patterns, our method may not lead to a good solution compared to methods that consider patterns of both classes. Thus, we would like to distinguish our method from general binary classification methods. We appreciate the reviewer for prompting us to view our method in this light.\n\n>**Q5**. Can this method be extended to multi-class classifications?\n\n**A5**. In fact, our proposed method can always perform multi-class classification and unwanted data detection simultaneously. Figure 1 in our paper or the figure in our repository (https://github.com/HFTT-anonymous/HFTT) can aid in understanding this. HFTT utilizes the input embeddings and logit values as they are for zero-shot classification while simultaneously estimating the probability that an input sample is unwanted. For example, our method can be applied to solve a 1001-class classification task consisting of 1000 ImageNet classes plus one OOD class.\n\nHowever, applying our method does not enhance the zero-shot image classification performance of VLMs. This is because our method focuses solely on optimizing trainable embeddings for out-distribution detection while keeping CLIP frozen. Consequently, the image classification performance of VLMs remains the same as their zero-shot classification performance even after applying our method. For instance, the ImageNet classification accuracy of CLIP-ViT-B/16 remains at 68.6%, regardless of whether our method is applied.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qMQIOGdQ3w", "reviewer": "Reviewer_QxHY", "summary": "This paper introduces an efficient and effective text-only training method for detecting undesired visual content. Its key contributions include a theoretical demonstration of how text data can substitute visual data, a new loss function, and a method for synthesizing textual data. These efforts aim to segregate data from one mode using only the dataset from the other mode. The method is evaluated through experiments on OOD detection tasks and hateful image detection, demonstrating comparable performance.", "questions": "My main questions are (1) making the trainable embedding learning clearer and (2) the test on other pre-trained models to view how the  pre-learned cross-modal alignment knowledge impacts the text-only learning.", "limitations": "The main limitations of this paper also lie in the two concerns: unclear description of the trainable embedding learning and no examination of other pre-trained vision/video-text models.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The utilization of text-only mode for detecting data in the other mode is a compelling approach, supported by existing research in various domains.\n2. The training and inference processes are efficient, with a small number of trainable parameters and minimal additional computational costs during inference.\n3. Despite the experimental results not consistently outperforming other methods, the high efficiency (no use of image datasets during training) can offset this limitation.", "weaknesses": "1. The trainable embedding learning process is unclear. According to the description, trainable embeddings are learned for N out-distribution data instances, which are then frozen during the test phase. It appears that the image embeddings obtained by the image encoder align with the frozen embeddings. If this is the case, the alignment process should be clearly elucidated. If not, the method for using the learned embeddings for images needs to be explained. The parameter N, representing the number of out-distribution data used during training, intuitively suggests that a larger N would enhance accurate distribution learning. However, the results in Table 8 present a contradictory viewpoint. More explanation should be given.\n2. The proposed HFTT method can be applied to different pretrained models and OOD tasks. In addition to the image CLIP model, it is recommended that the authors conduct experiments using pretrained video-text models to assess performance changes in related video detection tasks. This additional experiment can shed light on how pretrained models influence performance when only one mode is utilized for learning, considering the crucial role of pre-learned cross-modal alignment knowledge in enabling text-only learning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Firstly, we thank you for your thorough review of our paper. We particularly appreciate your recognition that our method is the compelling approach of utilizing text-only mode for detecting data in other modes and that the training and inference processes are highly efficient. We are also immensely grateful for your excellent suggestions that can enhance the contents of our paper. We will address each of your concerns in detail in the following:\n\n>**Q1**. The trainable embedding learning process is unclear. According to the description, trainable embeddings are learned for N out-distribution data instances, which are then frozen during the test phase. It appears that the image embeddings obtained by the image encoder align with the frozen embeddings. If this is the case, the alignment process should be clearly elucidated. If not, the method for using the learned embeddings for images needs to be explained. \n\n**A1**. In Section 3.1 of our paper, we presented a motivating example demonstrating that a classifier obtained using only textual data in the output space of a model like CLIP, where images and texts are well-aligned, can also be applied to image data. Based on this, we proceed as follows:\n1. We define trainable embeddings in the output space of CLIP that serve as the parameters of a classifier distinguishing between in-distribution and out-distribution data.\n2. We train these embeddings using only textual data.\n3. We use these trained embeddings to detect unwanted images.\nIn summary, the model parameters of CLIP remain fixed, and our trainable parameters are defined and trained within CLIP’s (image-text) joint embedding space. Thus, our method does not require any additional alignment process.\n\n>**Q2**. The parameter N, representing the number of out-distribution data used during training, intuitively suggests that a larger N would enhance accurate distribution learning. However, the results in Table 8 present a contradictory viewpoint. More explanation should be given.\n\n**A2**. We apologize for any confusion caused by our notation. The \"N\" in Table 8 refers to the number of trainable embeddings, not the number of out-distribution data. The number of trainable embeddings in our method represents the complexity of the classifier that distinguishes between in-distribution and out-distribution data. If the data in CLIP's output space form a very low-dimensional manifold, it would be possible to distinguish between in- and out-distribution data with a small number of trainable embeddings. As we can see in Table 8, a large \"N\" is not necessary for high unwanted image detection results. Indeed, various studies support the notion that deep learning models possess low-dimensional data manifolds [1,2].  \n[1] Ansuini et al. \"Intrinsic dimension of data representations in deep neural networks.\" NeurIPS 2019.  \n[2] Moayeri et al. \"Text-to-concept (and back) via cross-model alignment.\" ICML 2023.\n\n>**Q3**. The test on other pre-trained models to view how the pre-learned cross-modal alignment knowledge impacts the text-only learning.\n\n**A3**. Thank you very much for suggesting an expansion of our research domain. Video embedding models and detection tasks are less established than those in the image domain. We will investigate the experimental environment further and include results in the next version of our paper. Here, we provide the results for CLIP-L/14, BLIP-B/16, and BLIP-L/16 in addition to the CLIP-B/16 used in our study. Tables A, B, and C demonstrate that our method is effective across various vision-language models.\n\nTable A: The experimental results on CLIP-L/14 (in-distribution: ImageNet)\n|OOD:|iNaturalist||SUN||Places||Texture||NINCO|| \n|-|-|-|-|-|-|-|-|-|-|-|\n|**Method**|**FPR** &#8595; |**AUROC** &#8593;|**FPR** &#8595;|**AUROC** &#8593;|**FPR** &#8595;| **AUROC** &#8593; |**FPR** &#8595;|**AUROC** &#8593;|**FPR** &#8595;|**AUROC** &#8593;|\n|MSP|26.66|94.20|22.37|94.37|36.82|92.45|52.83|86.57|67.27|78.70|\n|Energy|30.84|91.25|25.94|94.10|32.94|92.30|64.33|79.26|63.49|79.72|\n|MaxLogit|32.76|90.96|26.48|92.96|**31.88**|92.39|72.08|73.85|**60.67**|**81.07**|\n|MCM|64.41|26.96|94.19|22.77|94.37|36.74|92.44|52.66|86.56|68.16|78.65|\n|HFTT (ours)|**24.10**|**94.58**|**17.80**|**95.39**|33.83|**93.09**|**52.06**|**86.58**|69.19|78.98|\n\nTable B: The experimental results on BLIP-B/16 (in-distribution: ImageNet)\n|OOD:|iNaturalist||SUN||Places||Texture||NINCO||\n|-|-|-|-|-|-|-|-|-|-|-|\n|**Method**|**FPR** &#8595; | **AUROC** &#8593; |**FPR** &#8595; | **AUROC** &#8593; |**FPR** &#8595; | **AUROC** &#8593; |**FPR** &#8595; | **AUROC** &#8593; |**FPR** &#8595; | **AUROC** &#8593; |\n| MSP |  64.70 | 82.22 | 30.38 | 91.06 | 71.40 | 78.82 | 76.99 | 81.30 | **71.47** | 72.07   |\n| Energy | 67.15 |  79.30 |  45.21 |  89.07 |  70.28  | 77.49 |  91.24 |  75.38 |  80.29  | **77.20**  |\n| MaxLogit |  69.57 | 75.44 | 69.57 | 71.19 | 69.86 | 76.26 | 93.55 | 60.31 | 88.58 | 56.37 |\n| MCM| 64.41  | **82.29** | 30.21 | 91.05 | 70.53 | 79.32 | 75.84 | 81.55 | 71.56 | 72.02  |\n| HFTT (ours)    | **63.28** | 82.22 | **19.16** | **95.12** | **68.48** | **79.50** | **63.74** | **84.53** | 72.12 | 73.86   |\n\nTable C: The experimental results on BLIP-L/16 (in-distribution: ImageNet)\n| OOD:  | iNaturalist || SUN ||  Places || Texture || NINCO || \n|-|-|-|-|-|-|-|-|-|-|-|\n|**Method**|**FPR** &#8595;| **AUROC** &#8593; |**FPR** &#8595; | **AUROC** &#8593; |**FPR** &#8595; | **AUROC** &#8593; |**FPR** &#8595; | **AUROC** &#8593; |**FPR** &#8595; | **AUROC** &#8593; |\n| MSP |   51.20| 87.91| 22.37| 93.86| 61.63| 84.68| 64.85| 85.28| 65.96| 78.29 |\n| Energy | 45.63 |87.23 |33.94 |90.29 |55.73 |85.91 |72.38 |82.16 |71.23| 77.49  |\n| MaxLogit |  44.59| 86.94| 35.56| 86.45| **50.96**| **86.46**| 86.38| 71.22| 79.78| 67.59  |\n| MCM| 50.75 |88.03 |22.34 |93.88 |60.88 |85.38 |64.71 |**85.39** |66.04| 78.32  |\n| HFTT (ours)  | **44.24**| **89.88**| **6.81**| **98.40**| 62.20| 84.16| **63.35**| 83.39| **64.82**| **80.46** |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZMbxobw50C", "reviewer": "Reviewer_kqnV", "summary": "This paper proposes a purely textual-based training method for detecting out-of-distribution or hateful images. The authors train an additional embedding layer over frozen CLIP encoders using text data. The authors propose the use of a novel loss function for this training. The method improves performance over baselines for most of the datasets that they evaluate on.", "questions": "Clarifications to my questions in weaknesses section is sufficient", "limitations": "The authors have described limitations.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Using only textual data for hate detection training is an interesting approach as it eliminates the need to source hateful images or ethical issues of creating paired image-caption datasets of hateful scenarios. The authors provide sufficient experimentation and ablation analysis to validate their claims that only training on text data is enough to identify OOD or hate content. Implementation details are listed and code is provided for reproducibility.", "weaknesses": "The writing structure could be improved. The textual synthesis aspect is not clear to me and should be discussed in more detail in the main paper. The method could also be used for generic image classification tasks which brings me to question the choice of why such classification results have not been shown. Some more discussion over the baselines would be good too.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Firstly, we thank you for your thorough review of our paper. We particularly appreciate your recognition of our innovative approach using only textual data for hate detection, which eliminates the need to source hateful images and addresses ethical concerns. Additionally, we value your acknowledgment of our comprehensive experimentation, ablation analysis, and the provision of implementation details and code for reproducibility.\n\nWe are also immensely grateful for your excellent suggestions that can enhance the contents of our paper. We will address each of your concerns in detail in the following:\n\n>**Q1**. The writing structure could be improved. The textual synthesis aspect is not clear to me and should be discussed in more detail in the main paper. \n\n**A1**. We apologize for any inconvenience caused in understanding our paper. In our method, the purpose of textual data synthesis is to emulate the entire visual dataset for training. We assumed that using prompt templates, such as \"a photo of a {}\", to generate text data could effectively emulate visual data, based on observations of high CLIP zero-shot classification performance with similar prompts. Therefore, our textual data synthesis involves simply inserting numerous words into the prompt \"This is a photo of a {}\". In our experiments, we used approximately 370k predefined English words. Our results demonstrate that even this lightweight textual data synthesis method can produce a detector capable of identifying various types of unwanted images.\n\n>**Q2**. The method could also be used for generic image classification tasks which brings me to question the choice of why such classification results have not been shown.\n\n**A2**. In fact, our proposed method can always perform generic image classification and unwanted data detection simultaneously. Figure 1 in our paper or the figure in our repository (https://github.com/HFTT-anonymous/HFTT) can aid in understanding this. HFTT utilizes the input embeddings and logit values as they are for zero-shot classification while simultaneously estimating the probability that an input sample is unwanted. For example, our method can be applied to solve a 1001-class classification task consisting of 1000 ImageNet classes plus one OOD class.\n\nHowever, applying our method does not enhance the zero-shot image classification performance of VLMs. This is because our method focuses solely on optimizing trainable embeddings for out-distribution detection while keeping CLIP frozen. Consequently, the image classification performance of VLMs remains the same as their zero-shot classification performance even after applying our method. Hence, we did not present classification results. For instance, the ImageNet classification accuracy of CLIP-ViT-B/16 remains at 68.6%, regardless of whether our method is applied.\n\n>**Q3**. Some more discussion over the baselines would be good too.\n\n**A3**. Thank you for your suggestion. We provide further discussion on the strongest baselines and present results for various foundation models here:\n\n**Comparison with SOTA OOD detection methods**  \nNegLabel [1] and CLIPN [2] are cutting-edge OOD detection methods.\nNegLabel constructs an OOD corpus by selecting texts distant from the in-distribution texts from a predefined corpus, then compares the distances between the input image and those texts in the CLIP embedding space to detect OOD. While NegLabel shows high OOD detection performance on ImageNet (see Table B), it has the following limitations compared to our method:\n- Although NegLabel does not require training additional parameters, it must compute the embeddings of all texts in the corpus and measure their similarity to the in-distribution texts to find the optimal OOD corpus for a given in-distribution. Our training method also requires nearly the same cost as obtaining the embeddings of all texts within a predefined corpus and calculating the similarities between those embeddings and the task+trainable embeddings, as discussed in Section 4.1. Thus, NegLabel and our method require the same level of optimization cost.\n- Since NegLabel uses the embeddings of the determined OOD corpus as they are, it falls behind our method, which has trainable parameters, in terms of generalization. To demonstrate this, we further compare our method and NegLabel in the medical image domain. Specifically, we treat the ISIC-18 skin lesion diagnosis dataset [3] as in-distribution and the PathVQA [4] and PatchCamelyon [5] datasets as out-of-distribution. The ISIC-18 skin lesion diagnosis dataset is an image classification benchmark for seven skin disease categories.\n\nTable A: OOD detection in the medical image domain\n|OOD:|PVQA||PCAM||\n|-|-|-|-|-|\n|**Method**|**FPR** &#8595;|**AUROC** &#8593;|**FPR** &#8595;|**AUROC** &#8593;|\n|NegLabel|37.44|94.11|48.07|94.86|\n|CLIPN|35.47|84.64|3.10|98.76|\n|HFTT (ours)|13.72|97.05|4.95|98.35|\n\nTable A illustrates the limitations of NegLabel in terms of generalization. While NegLabel fails to construct an effective OOD corpus for the medical image dataset, our method achieves significantly higher performance by learning optimal embeddings for detection.\n\nCLIPN utilizes an additional \"no\" text encoder alongside CLIP. This additional text encoder predicts the probability that a given object is not present in an image. Thus, CLIPN predicts whether a given image is in-distribution or out-distribution by using the original CLIP text encoder and the \"no\" text encoder to estimate the probabilities, respectively. Images with a low probability of being in-distribution and a high probability of being out-distribution are identified as OOD.\n\nAlthough CLIPN achieves high OOD detection performance on ImageNet (see Table B), it has the following limitations compared to our method:", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We genuinely appreciate the reviewers' dedicated time and their valuable feedback. Before addressing each reviewer's specific concerns in detail below, we would like to summarize here the contributions of our research that have been recognized and the aspects that have been enhanced in our study.
## Reviewer Acknowledgements of Our Paper's Strengths
- The theoretical justification of our proposed method is solid (CzuR, LurW).
- We demonstrate that it is possible to obtain an effective unwanted image detector without using images (CzuR, QxHY, kqnV).
- Our method is innovative, efficient, effective, and useful in practice (CzuR, LurW, QxHY, kqnV). 
- The experiments are comprehensive, and the results show the effectiveness of our proposed method (LurW, kqnV).
- For the reproducibility of our method, we provide a cost analysis, implementation details, and code (LurW, kqnV).

## Enhancements in Our Paper
- We provide additional discussion on the comparison with state-of-the-art OOD detection methods.
- We present comparative results against NegLabel and CLIPN in the medical image domain.
- We offer experimental results and discussion regarding other foundation models, such as BLIP.
- We provide a discussion on the application of our method to multi-class classification tasks.

We provide detailed responses to each reviewer's comments below. We have diligently addressed each of the four reviewers' concerns to the best of our abilities, and believe that the results further underscore the contributions of our study. We plan to integrate the reviewers' suggestions into the revisions of the manuscript, as we believe these adjustments will significantly bolster the paper's overall strength.

Meta-review: The paper originally received borderline scores leaning towards acceptance. The authors provided a detailed rebuttal and the paper has been discussed by the reviewers. The reviewers raised some concerns about missing experiments/comparisons (e.g. with NegLabel and CLIPN) that have been mostly addressed in the rebuttal. The authors included in their responses several new result and, overall, the reviewers where satisfied by the rebuttal and they all manteined or increased their scores to 6:WeakAccept. Overall this is a solid submission and the authors are encouraged to carefully follow and implement all the reviewers' suggestion in the final revision of the paper.
Decision: Accept (poster)