A General-Purpose Multi-Modal OOD Detection Framework
Authors: Viet Quoc Duong
Qiong Wu
Zhengyi Zhou
Eric Zavesky
WenLing Hsu
Han Zhao
Huajie Shao
Venue: Accepted by TMLR ()

Abstract: Out-of-distribution (OOD) detection seeks to identify test samples that deviate from the training data, which is critical to ensuring the safety and reliability of machine learning (ML) systems. While a plethora of methods have been developed to detect uni-modal OOD samples, only a few have focused on multi-modal OOD detection. Current contrastive learning-based methods primarily address multi-modal OOD detection in a scenario where an image is not related to the class labels in training data. However, ML systems in the real-world applications may encounter a broader spectrum of anomalies caused by different factors like  systematic errors in labeling, environmental changes, and sensor malfunctions. Hence, we propose a new method to be able to simultaneously detect anomalies from multiple different OOD scenarios, arising from fine-grained image features and textual descriptions, instead of large categorical information. To achieve this goal, we propose a general-purpose weakly-supervised OOD detection framework, called WOOD, that combines a binary classifier and a contrastive learning module to reap the benefits of both. In order to better distinguish in-distribution (ID) samples from OOD ones, we employ the Hinge loss to constrain the similarity of their latent representations. Moreover, we devise a new scoring metric that fuses predictions from both the binary classifier and contrastive learning to enhance OOD detection. Extensive experimental results on multiple benchmarks demonstrate that the proposed WOOD significantly outperforms the state-of-the-art methods for multi-modal OOD detection. Importantly, our approach can achieve superior detection performance in a variety of OOD scenarios.

Published in Transactions on Machine Learning Research (07/2024)
A General-Purpose Multi-Modal OOD Detection Framework
Viet Duong
vyduongg@wmm.com eduu
Department of Computer Science, William & Mory
Qiong Wu
gw65470att.cm com
AT&T Labs
Zhengyi Zhou
22547k@utt com
AT8T Labs
Eric Zavesky
ez2685 com
AT8T Labs
Wen-Ling Hsu
wenlhsu@@gmaii. com
AT&T Labs
Han Zhao
honzhao@illiniis edu
Department Computer Science, University of Illinois at Urbana- Urbana-Champiite Champoign
Huajic Shao*
hshuo Burm. edu
Department of Computer Science, William 8 Mary
Reviewed on OpenReview: https: openreview. net/ forum? id= n Yzus 7s$2o
Abstract
Out-of-distribution (OOD) detection seeks to identify test samples that deviate from the training data, which is critical to ensuring the safety and reliability of machine learning (ML. systems. While plethora of methods have been developed to detect uni-modal OOD sam- ples, only few have focused on multi-modal OOD detection. Current contrastive learning- based methods primarily address multi-modal OOD detection in scenario where an image not related to the class labels training data. However, ML systems in the real-world applications may encounter broader spectrum of anomalies caused by different factors like systematic errors in labeling, environmental changes, and sensor malfunctions. Hence, we propose a method to be able to simultancously detect anomalies from multiple different OOD scenarios, arising from fine-grained image features and textual descriptions, instead of large categorical information. To achieve this goal, propose a general-purposs weakly- supervised OOD detection framework, called WOOD, that combines binary classifier and a contrastive lcarning module to reap the benefits of both. In order to better distinguish in-distribution (ID) samples from OOD ones, we employ the Hinge loss to constrain the similarity of their latent representations. Moreover, we devise a new scoring metric that fuses predictions from both the binary classifier and contrastive learning to enhance OOD detection. Extensive experimental results multiple benchmarks demonstrate that the proposed WOOD significantly outperforms the state-of-the-art methods for multi-modal OOD detection. Importantly, our approach can achieve superior detection performance in a varicty of OOD secnarios.
*Corresponding author.

---

Published in Transactions on Machine Learning Research (07/2024)
a small yellow colored bird with a black head and body strips and small beak
two black birds are floating around the river one looks curious about the other mouth
medium size bird with brown and white speckled wings white
(a) Misaligned - an ID image not aligned with its textual description (egg,, the textual description be- longs to another class).
(b) New-domain an aligned image-text pair from dataset of another domain (e.g., containing two birds instead of one).
(c) Noisy - an ID and aligned image-caption pair but with blurry patches (highlighted in red squares) in the image (sensory faults).
Examples of three OOD scenarios in CUB-200 datasct, with the textual descriptions for cach scenario. Images are indicated as OOD with red border, and ID with green. Regarding the text modality, corresponding textual descriptions in red for OODs and green for IDs.
Figure 1:
1
Introduction
Out-of-distribution (OOD) detection (Yang et al. 2021 Bogdoll et al. 2022: Ming et al.] 2022c
Ruff et al.
2021 Ma al. 2021 aims at identifying whether a test sample differs from the training data. Such detection
is crucial for ensuring the safety and reliability of machine learning (ML) systems (Wang al. 2020 Hussain
& Zeadally 2018), such autonomous driving and Al diagnosis (Davenport & Kalakota 2019 Han et al.
2020
MacDonald et al. (022). Due to its importance, OOD detection has received a lot of attention from
both industry and academia.
While extensive studies have focused on single-modal OOD detection Zadorozhny et al., 2022 Roy et al. 2022 Narayanaswamy et al. 2023). multi-modal OOD detection remains underexplored. In reality, multi- modal OOD detection is particularly important lo many high-stakes applications like clinical diagnosis and autonomous driving. These applications often fuse information from different data sources and modalities to improve the reliability of decision making, For instance, majority of Al diagnosis systems integrate multi-modal data, such as radiology imaging (X-ray, ultrasound, CT, MRI, etc.), physician's annotations, and electronic health records, to enhance their prediction performance. In the real-world applications, those multi-modal ML systems inevitably encounter "outlier" conditions caused by various factors, such as machine defects, sensor faults, and operator errors. Hence, a few recent works (Ming et al.] 2022b Esmacilpour et al. 2022) have explored multi-modal vinual-language OOD detection, but they narrow their scope to a special scenario where images come from new domain. Moreover, these work often employ Cassd techniques (Ming et al., 2022a Miyai et al. 2023 Tao et al. 2023 Miyai et al.] 2024) to identify whether a query image matches one of the training set's class-level labels (e.g., pneumonia). As result, these methods fail to identify OOD scenarios such as misaligned pairings of an image and its fine-grained tertual description.
The goal of this work is to develop a general-purpose multi-modal OOD detection model that can identify OOD in various scenarios a fine-grained manner. We focus on three different OOD scenarios for multi- sensory data, illustrated in Fig [ (1) Misaligned unaligned pairs of data samples, e.g., an in-distribution (ID) image is not aligned with its textual information; (2) New-domail aligned pairs of data samples col- lected from a new domain, e-g., aligned images and text from a new environment with different distribution from training; (3) Noisy partially noisy data samples, e.g., images coming from the same environment but with blurry patches. The research question is: How can we detect OOD samples from all these OOD sce- narios simultaneously? Existing OOD detection approaches, such as the CLiP-based methods (Ming et al. 2022a: Esmacilpour et al. 2022 Fort et al. 2021 and weakly-supervised classífications (Ji et al. 2020 Wang 2021), only focus on one of these scenarios (new-domain or noisy), thus failing to generalize to all of them. Fig. illustrates motivating example of multi-modal OOD detection on CUB-200 dataset (Wah)
et al.
let al.)
2011 using our weakly-supervisee CLiP-based contrastive learning method and the binary classi- fier. We can observe from Fig. 2 (a) that weakly-supervised CLIP can only detect misaligned OODs, but is not effective in new-domain or noisy scenarios. Specifically, weakly-supervised CLIP is not effective for new- domain OODs because we deliberately create a more difficult new-domain scenario by choosing OODs as samples labeled with "bird" from COCO data (Lin et al.) (2014). which has the same label but different textual information as those ID samples in the training set of CUB-200, instead of samples with other classes
(e.g.; person, dog, etc.) in previous works (Ming et al.] 2022a Zhou al. 2022a Miyai et al.] 2023 Tn
2

---

Published in Transactions on Machine Learning Research (07 (204))
Misalgnee
Misaiineed
50-
Mizaigned
New-domain
New-domain
Newe-tomain
Nune
Nokey
Sample Count
50-
Nilley
Thresholo: 0.56
Thrushold:
0.88
60-
Thrnashola:0005
Sample
00-
30
30
20-
20-
20-
10-
10-
10-
0.2
0.3
0.4
0.5
0.6
0.7
9
0.1
0.2
0.3
0.4
0.5
0.7
0.9
1.0
0.0
0.1
0.3
0.4
0.5
0.6
0.7
0.8
1.0
Probaolitees
ID
Probabilties
ID
Probabiittes
(a) Weakly-sueervised CLIP.
(b) Binary Classification.
(c) Our Model.
Figure 2: A motivating example of multi-modal OOD detection on CUB-200 dataset. Following prior work (Ming et al., (022b); we choose a threshold that includes 95% of in-distribution test samples (gray), and identify test sample as OOD if its prediction score below that threshold, i.., to the left of the dashed line. If an OOD detector works effectively, the prediction scores for all samples in all three OOD scenarios should be concentrated to the left of the dashed line. We can observe from (a) that weakly-supervise contrastive learning can only detect misaligned OOD samples (bluc), but fails to detect new-domain and noisy OODs. Conversely, a binary classifier in (b) can identify OOD samples in new-domain (orange) and noisy (green) scenarios, but it does not work for misaligned test samples. Motivated by these observations, we develop a general-purpose OOD detection model (c) that can effectively detect OOD) samples from all three scenarios.
contrast, a binary classifier can only identify new-dommin noisy OOD samples, but fails in the misaligned (b). This observation motivates us to develop new OOD detection model that combines both approaches to complement each other.
scenario, as illustrated in Fig. 2
Our Contributions. We propose the WOOD, a weakly-superviee multi-modal OOD detection model that can identify three OOD scenarios at the same time. As illustrated in Fig. 3 the proposed WOOD composed of two components: a binary classifier for classifying OOD samples and contrastive learning module for measuring the similarity scores between multiple data modalities. On the contrastive learning side, we adopt the Hinge loss to maximize the similarity scores of ID) samples and minimize those of OOD samples to better distinguish them. On the binary classifier side, we develop a novel Feature Sparsity Regularizer to better integrate important features from data of multiple modalities. Then a new scoring metric is designed to fuse the prediction results from these two components. Finally, we evaluate the proposed WOOD model on three real-world benchmark datasets. Experimental results demonstrate that our method can successfully identify OOD samples with high accuracy under three different scenarios, which significantly outperforms the CLiP-based baselines.
2 Related Work
In this section, review and contrast relatod work on OOD detection in machine learning. We group them into two categories: single-modal OOD detection and multi-modal OOD detection. Additionally, further discuss related work on multi-modal OOD generalization in Appendix A
Single-Mode Modal OOD Detection. There exists a plethora of works on single-modal OOD detection (Salehi
et al. detection, (ii) text OOD detection, and (iii) time-sories OOD detection (Du et al.
2022 for machine learning. They can be generally categorized into three types: (i) vision OOD
2022p. For vision OOD
detection, various methods have been developed, including softmax confidence score
DeVries & Taylor 2018
Hein et al. 2019; Hendrycks & Gimpell 2017 Huang & Li| 2021 energy-based score
Yang et al. 2021 Sun
et al. 2021 Sun & Li 2022), distance- based method (Sun et al. 2022 Ren et al. 2021 Podolskiy et al. 2021
Tochapanurak et al. 2020 Zacemzadeh et al. 2021 Van Amersfoort et al. 2020). and generative models (Li
et al. 2022b Xiao et al.) 2020). For instance, Liu el al. (2020) proposed an energy-based OOD detection
(2022) developed nearest neighbors to improve the flexibility and generality of OOD detection. Yoo et al.
method with theoretical analysis. SSnn et al.
(2023) considered data augmentation as hyperparameter
for generating pseudo-anomalies, and investigated its effoctiveness for image- based self-supervised anomaly
detection. For text. OOD detection, pre-trained language models (Podolskiy et al. 2021 Zhou et al.) 2021
3

---

Published in Transactions on Machine Learning Research (07/2024)
are commonly used due their robustness in identifying OOD samples in natural languages. Other methods, such as data augmentation (Zhan et al. 2021 and contrastive ve learning (Zhou et al.
2022b; 2021). have also
been developed for OOD detection. Furthermore, some researchers have focused on OOD detection in time-
scries data (Romero& & Estévez 2022: Kaur et al. 2022 Georgescu et al. 2021). where several ML models have
been developed for video anomaly detection [Georgescu et al.) 2021 Wang et al.] 2018p. Wang el al. (2018) combined LSTM with CNN to improve anomaly detection using a spatio-temporal auto-encoder. Li ct al. (2022a) leveraged generative models to predict middle frames based on past and future frames. However, these methods mainly detect OOD samples using unimodal data, such images or text. In contrast, we develop a general-purpose model that combines multi-modal data, such as images and textual information, to enhance the performance of OOD detection.
Multi-Modal OOD Detection. Some studies (Sun et al., 2020: Wang et al. 2021 have adopted multi- modal data to improve the OOD detection accuracy based on deep neural networks (DNNs). (Wang et al. 2021) proposed a multi-modal transformer network that combines Radar and LiDAR data to detect radar ghost largets. Ji et al. (2020) developed supervised VAE (SVAE) model that integrates sensor data of multiple modalities to detect anomalous operation mode of the car. To improve the accuracy of detecting abnormal driving segments, Qiu et al. (2022) developed an unsupervised contrastive approach that uses generative adversarial networks to extract latent features from five modalities. More recently, CLiP-based methods (Ming et al. 2022a: Esmacilpour et al.] 2022 Fort et al. 2021 Jeong et al., 2028) have been developed to detect OOD samples. (Esmaeilpour et al. 2022) designed a zero-shot OOD detection model based on pre-trained CLIP (Radford et al., [021]] to detect and generate candidate labels for test images of set of candidate labels. To overcome this issue, Ming et al. (2022a) developed zero-shot method called Maximum Concept Matching (MCM) based pre-train CLIP model, that can detect visual OOD samples, i.e., OODs arising from the image modality. which utilizes local CLIP visual features to enable zero-shot OOD detection for background-relatte OOD features. Additionally, Jeong et al. (2023)
unknown classes. However, this method heavily relies
Miyai et al. 2023) further introduced
developed WinCLIP to detect multiple scenarios of visual anomaly by matching multi-scale CLIP image embeddings obtained from image windows of varying sizes with text prompts describing the anomaly (e.g.. 'a blurry photo"), but known anomaly prompts are required to achieve high zero-shot detection accuracy, Furthermore, latest rescarch explored various fine-tuning strategics for CLIP, such as prompt learning (Zhoul et al. 2022al Miyai et al. 2024) and outlier synthesis (Tao et al. 2023). for enhanced OOD detection based on MCM and GL-MCM. While these methods have shown good performance on multi-modal OOD detection,
they can only detect visual OOD samples rather than both an image and its associated textual description. Hence, they are not applicable to other scenarios we are exploring, such as misalignment and noisy samples.
Different from prior works, we develop a general-purpose multi-modal OOD detector that can identify OOD samples arising from three different scenarios in a fine-grained manner. Our proposed method leverages both weakly supervised learning and contrastive learning for improving OOD detection.
3
Preliminaries
This section begins by introducing the problem of detecting OOD samples in multi-modal data, Subsequently, we delve into the applications of contrastive learning for multi-modal OOD detection
3.1 Problem Statement
We consider the problem of detecting multi-modal OOD samples under three different scenarios, as mentioned in Sec. [ In this paper, we use vision- languala modeling as running problem for multi-modal OOD detection. Given a batch of N pairs of images and texts, along with their positive ID labels, denoted by {II,, (n), yn = 1] we generate a small number of OOD samples K prior to training- Specifically, we select K < N pairs of images and texts using a random seed, and convert them to OODs according to the three scenarios above, such that {(T, f2), yk = 0]k (k)...., kx) C (1,...,,Ny N)), and the remaining N K ID pairs are unchanged. The goal of this work is to improve the OOD detection performance for multiple scenarios using weak supervision a small number of generated OOD samples.
4

---

Published in Transactions on Machine Learning Research (07 (07/2024)
N-K
Feature Sparsity Regularizer
Unintomaa va feature
Image Encoder
Contrastive
Informativv feature
Learning
00O
Similarity Matrix
Image
weights
The bird has grey wings and long bill
Text Encoder
Feature Sparsity
Binary Classifier
Lbe
Regularizer
Text Features
weights
(a)
(b)
Figure 3: (a) The overall framework of the WOOD detector. It consists of two main parts: 1) constrastive learning module and 2) a binary classifier. Contrastive learning aims to maximize the difference similarity scores between ID and OOD samples while the binary classifier used to predicl the probability of being OOD. (b) An expanded detailed view of the Feature Sparsity Regularizer in the WOOD framework. This regularizer integrates important features from data of multiple modalities to improve classification accuracy,
3.2 Contrastive Learning
Contrastive learning aims to encode pairs of data samples into latent representations by making similar samples close to each other and dissimilar ones far apart. One well-known method the vision-language pre-trained model CLIP, which jointly trains an image encoder and a lext encoder to learn the latent representations from text paired with images using zero-shot learning. Specifically, it encodes a pair of image 2n and text t, into the latent reprosentations I(#,) and T(t,), respoctively, and then adopts cosine similarity to minimize the distance of their represenlations I(x,,) and T(,.). Due to its excellent performance methods have been developed
in learning the latent representations of images and texts, few CLIP- to detect multi-modal OOD samples. However, existing methods can only detect one specific scenario where a given image and its textual information come from a new domain with a distribution shift. In the following section, we develop a general-purpose OOD detection framework identily OODs arising from several different scenarios.
4 Proposed Model
In this section, we describe the proposed multi-modal OOD detection framework and then devise a new scoring metric for detecting OODs under three different scenarios.
We propose weakly-supervial OOD detector, called WOOD, that combines a classifier and contrastive learning to simultancously detect three different OOD scenarios mentioned in Scction 1 Fig. 3 illustrates the overall multi-modal OOD detection framework that consists of two components: contrastive learning module and a binary classifier. The core idea is to use contrastive learning to learn the representations of different data modalities by enforcing the similarity scores of ID pairs to be higher than those of OOD samples. Then WOOD combines the similarity scores from contrastive learning and the prediction results from a binary classifier to identify OOD samples. Below, we detail the two main components of the proposed WOOD method.
Contrastive Learning with Hinge Loss. Inspired by CLIP-based OOD detection methods, adopt image encoder and a text encoder to learn the representations of input pairs (image and text) using contrastive learning. However, unlike existing zero-shot CLIP-based detectors, we add a small number of OOD samples to better separate the representations of ID and OOD samples. Thus, our goal is to maximize the cosine similarity of representations lcarned from ID samples but and minimize those of OOD samples. Let (z,, tx) be a pair of input image and text, and their corresponding representations denote (I(z,). (tt,)). Then, the CLIP contrastive loss (Radford et al. 2021 h can be modified to maximize the cross entropy of the
5

---

Published in Transactions on Machine Learning Research (07/2024)
cosine similarity between labeled OOD pairs and minimize that of ID pairs, which is given by:
K
E stl) eSultsi)))
Sood(1x .t6)
L1
log
log K i=1 (ood(t
(1)
N
K
NN- Liml
K
where (1a t,t)
[i(Zz >I III represent the cosine similarity between image and text features for ID and OOD pairs, respectively. Here "+" and 5_" mean the sample is from ID and OOD, respectively. addition, N K and K respectively denote the number of ID and OOD pairs.
[ 3IIlI III and Sood
[(=))) )(ts))
I(
To further maximize the difference in latent representations between ID and OOD samples, we adopt Hinge loss instead of Lj in Eq. | to constrain their cosine similarity. In this work, we consider Hinge loss for both ID pairs and labeled OOD pairs in the objective function.
First, we introduce Hinge loss for N - K ID samples, as shown in the upper part of the similarity matrix in Fig 3 It is given by:
N K
N
max (0.m - StalEt (n)) + Sta(=n, (//)t+//))))
Lid =
Σ n=1
Σ Ligin
(2)
N
where m is a margin, Sia( t represents the cosine similarity of N - K aligned ID pairs and Sia( t-/") represents the cosine similarity bet N K ID images and all N texts (including OOD samples), where each text either does not align with its corresponding ID image or belongs to an OOD sample. In short, the objective, Cid, aims to maximize the difference between the aligned ID pairs and incorrect pairings by ensuring that such difference is larger than a margin
Second, we introduce Hinge loss for K OOD samples to constrain their cosine similarity to small value. Hence, we have:
K
N
max (0, -m + Sand(T ( ))
Lood
(3)
N
where Sood(0 :t: ( represents the cosine similarity between each K OOD images and all N texts (including OOD samples). By definition, the K OOD images should not align with any of the N K ID texts, since each OOD image not aligned with its corresponding text described in the misaligned scenario. Additionally, should discourage the model from aligning new-domain and noisy OOD pairs.
By combining the above loss functions, Lid and Lood, for ID and OOD samples, the overall contrastive loss is given by:
1 La N (Lid + Lood).
(4)
During inference, inspired by Wang et al. 2023a we apply the sigmoid function to the cosine similarity to obtain the ID probabilities:
(S(or)=fn)
S(In: fn)
Pa Pa= = o
(5)
where S(xn,tt) denotes the cosine similarity, and S the temperature for scaling the cosine similarity. Note that as the cosine similarity in the range [-1,1], when we set a small value of 0.1 in our experiments, the range of Pa is close to 10,
Binary Classifier. We further adopt weakly-supervised lcarning to classify OOD samples since recent studies have illustrated that it can significantly outperform unsupervised learning methods by adding some OOD samples (Tian et al.) 2020 Sultani et al. 2018 Majhi et al.) 2021). However, the challenge lies in how integrate image and text features for improved classification accuracy. One naive method concatenate their embeddings directly and then feed them into a classifier. But this simple fusion approach does not perform well since the informativeness of different features may vary for different samples (Jiang

---

Published in Transactions on Machine Learning Research (07 2024))
Algorithm 1 The Proposed WOOD Model
1: Input: A batch of N pairs of images and texts, with N - K ID pairs and K labeled OOD pairs.
2: Output: Samples labeled as OOD or ID.
3: Encode ID pairs into I(=;)) and )
4: Encode OOD pairs into ) and T(,)).
Compute Hinge loss on cosine similarity for ID samples according to Eq. 2 and OOD samples according
to Eq. 3 for contrastive learning.
Compute the total contrastive loss La in Eq.
7: Compute loss of the binary classifier Live in Eq. 6
8: Jointly train the binary classifier and contrastive learning based on the overall objective in Eq. 7
9: Identify OOD samples based on the designed scoring metric in Eq.
et al., 2023 Han al., 2022). To address this issue, we develop a feature sparsity regularizer to select and integrate important features from data of the two modalities, as illustrated in Fig, B(). Specifically, train single-layer encoder MLP networks E I(œn) w and E" T(t,) - wo? to use features obtained from image encoder and text encoders. Here, the dimension of WN matches that of In, and the dimension of wi matches that of fn These weight features are updated by a sigmoid activation a to assign higher weights to informative features and lower weights to the uninformative features. Namcly, the weight vectors (ww)) and or (w, are multiplied element-wise (denoted by &) with I(x,,) and T(,,,) respectively. Afler that, we fuse the features from two modalities by concatenation (#), yielding h, + (f[(x,,) @ (w T(t,) oo Next, we adopt a binary classifier to identify ID/OOD samples based on the fused features hn. To introduce sparsity in the weight vectors,
L1 normalization (lor(w,)]l and [[o l| and add them to our binary cross entropy (BCE) loss function, which is given by:
1 N
Lbc
Σ BCE( 0.) + 1 (w + [arlw")) Il1
(6)
where (CE ()) - (ys log(ū.) (1 vi) log(1 - 0.))- Note that here y, 0 mcans the i-th test sample is OOD while it is an ID sample when y; 1.
Overall objective. Finally, we jointly train the contrastive learning module and binary classifier for OOD detection. The overall objective for multi-OOD detection given by:
= (1 X)Ca + ALter
where Lbe is the contrastive loss, Lie the binary cross-entropy loss for the classifier, and [0,1] is the weight for balancing the two respoctive terms.
4.1 New Scoring Metric
To improve the performance of OOD detection during inference, we introduce novel scoring metric that combines predictions from the contrastive learning module and the binary classifier. The key insight is that we identify an image and text pair as ID only when both the contrastive learning and binary classifier predict that sample as ID. In all other cases, we identify as OOD, We codify this condition with the following scoring metric for identifying OOD samples:
Pood = 1 PoePas
where Phe and Pa denote the ID probabilitics in (0, 1) from the binary classifier and contrastive learning, respectively. Subsequent sections demonstrate that the new scoring metric can help detect OOD samples all three OOD scenarios effectively.

---

Published in Transactions on Machine Learning Research (07/ (224))
4.2 Summary of Proposed Model
We summarize the proposed WOOD model in Algorithm 1 The basic idea is to map ID and OOD pairs (image and text) into latent representations, and then calculate their cosine similarity. Then, we use Hinge loss to maximize the difference in similarity scores between ID and OOD samples. Moreover, we feod the fused latent representations into a binary classifier to classify ID or OOD samples. Finally, we jointly train the binary classifier and the rontrastive learning component for OOD detection.
5 Experiments
In this section, we carry out extensive experimentation to evaluate the performance of the proposed WOOD model on multiple benchmark datasets. Then, we conduct ablation studies to explore how the main compo- nents in model design and hyperparameters impact OOD detection performance.
5.1 Datasets
We implement experiments on the three real-world datasets: CUB-200 (Wah et al. 2011 MIMIC- CXR (Johnson al. 2019), and COCO (Lin et al. 2014p. For CUB-200, the textual information comes from literature (Reed et al. 2016). COCO and MIMIC-CXR contain images and their textual descriptions.
Three OOD scenarios. We generate three different OOD scenarios using the above datasets follows.
Misaligned. Randomly select a subset of ID images and their textual description from given dataset and shuflle them so that each image is not aligned with its corresponding textual information. Specifically, we select images from one category and the unaligned textual descriptions from another category, ensuring that each pair of OOD images and text are not aligned.
New-domain. Choose OOD samples from another new dataset different from the training data. For instance, when conducting experiments images from Indiana. University (IU) X-ray Fuuhhann et al. 2016) OOD samples.
MIMIC-CXR data, we select some pairs of texts and
Noisy. Add Gaussian blur (whilening for X-ray images) lo several palches in the ID) images, such that each image is partially blurry but its corresponding textual information is correct. Additionally, we evaluate CLIP's OOD genoralization to noisy images on the CUB-200 dataset in Appendix
Table summaaries the detailed information about generating three OOD scenarios using the above dalasets in the experiments.
Table Detailed Summary of Our Three OOD Scenarios.
Scenarios
CUB-200
MIMIC-CXR
COCO
Randomly select pairs of ID images and texts, swap their textual descriptions with textual descriptions images from different categories, and label these misaligned pairs as OOD.
Misaligned
New-domain
Samples from COCO with birds IU X-ray (Demner-Fusee Fushman et al. 2016 GCO (Sharma et. al. 2018
Randomly select pairs ID images and texts, add small number of blurry/ whitened patches each, and label these blurry pairs as OOD.
Noisy
5.2 Model Configurations
Following prior CLIP-based detectors, we also use the CLIP model (ViT-B/16 (Radford et al, 2021 as the backbone of the contrastive learning module. The two encoders are CLIP image and CLJP text, which are pre-trained Transformer models for image and text (Radford et al. 2021) respectively. We do not change the base encoders but fine-tune them with Hinge loss in Eq. 4 for both feature alignment and OOD detection. Recall that WOOD also has Feature Sparsity Regularizer module (Figure 3) which is a single projection layer (MLP) with a sigmoid activation. We set its hidden size to 512, the same as the dimensions of the output embeddings from CLIPimage image and CLIP The Binary Classifier is a 3-layer fully connected network with ReLU activation, which outputs a single probability score for binary OOD classification and the layer hidden size 1024, 512, and 128 respectively. We train the proposed WOOD model using Adam optimizer (Kingma & Ba 2014) with initial learning rate 5e -6-6 and stepped learning rate schedule. Additionally, the batch size set to 128 in all experiments, and X = 0.8 for the overall training adjective in Eq. 7 Regarding
8

---

Published in Transactions on Machine Learning Research (07/2024)
the margin of the Hinge loss, choose m = 0.2 for CUB-200 and MIMIC-CXR and m 0.3 for COCO after grid search. Note that subsequent ablation studies in Section 5.5 explore the impact of these hyper- parameters on detection performance. To improve detection performance on three scenarios simultaneously, we randomly select 1% of the training data and replace them with labeled OOD samples for cach scenario, as described in Tab. | resulting a lotal of 3% OOD samples and 97% of ID samples for weakly-superviser training. During evaluation, we use the same ratio (25%) of test samples for ID and three OOD scenarios. Finally, regarding inference, following previous research (Ming et al. 2022b), we choose a threshold 6 (e.g., 0.6) at inference so that high fraction of ID data is above the threshold (ie., OOD samples are identified when Pood > 1 6). Following prior work (Ming et al., 2022b we report the false positive rate for 95% of recall of ID samples, FPR95, to measure the prediction error when OODs are identified by threshold d. Additionally, we report the AUROC scores, which is a threshold-free OOD detection metric commonly used in the literature (Hendrycks & Gimpel 2016
Liu et al. 2020 Ming et al., 2022a
5.3 Baselines
We compare the WOOD model against a broad range of CLIP-based OOD detection basclines, including zero-shot (1-2) and finetuned approaches (3-6), as well as the weakly-supervised implementation (7-8) of previous detection methods based on auxiliary OOD data.
MCM-OOD (Ming et al. 2022a). This method uses zero-shot CLIP for multi-modal OOD detection based on Maximum Concept Matching (MCM). It can only detect visual OODs, ie. OOD images, in one type of scenario in which a given image is not aligned with its label in the training data. 2. GL-MCM-OOD (Miyai et al., 2023). As an extension of MCM-OOD, this work leverages both local and global visual-text alignments of CLIP features, known as Global-Local MCM (GL-MCM), to detect OOD samples containing ID classes with OOD features in the background in the zero-shot manner. However, it still only dotects visual OODs.
3. CoOp-MCM (Miyai et al. 2023). This method adopts prompt learning to finetune the pre-trained CLIP using set of learnable context vectors (Context Optimization). In addition, it uses MCM scores to detect visual OODs.
4. NPOS-MCM (Tao et al. 2023 This method finetunes the pretrained CLIP for enhanced OOD delection by generating artificial outliers from the boundary ID data selected based on the non- parametric n-nnearest-niighbor (k-NN) distance. It also utilizes MCM scores to detect visual OODs. CLIP-N (Wang et al., 2023b). This model adopts a Icarnable "no" text encoder addition to the pre-trained CLIP to capture negation semantics. It also uses a threshold-free inference algorithm perform OOD detection by utilizing the association between negation semantics from "no" prompts and the image encoder.
6. LoCoOp (Miyai et al., 2024b. This work extends CoOp by performing OOD regularization that utilizes the CL.IP local features as OOD features during training, since local features tend to have OOD related nuances such as background objects. It adopts GL-MCM scores to detect visual OODs.
7. CLIP-Energy (Liu et al.] 2020). This approach adopts auxiliary OOD data to finetune the CLIP model using energy scores. For fair comparison, we utilize the same number of OOD samples as our method for energy finetuning.
8. CLIP-BCE (Liznerski et al.] 2022 This model finetunes the pre-trained CLIP with a binary classifier (BCE) in the supervised manner using our OOD samples as outliers for Outlier Expo- sure (Hendrycks et al.) 2018), which aims to maximize the similarity of an image and the ID label set and minimize that for OOD samples.
5.4 Main Results
We conduct extensive experiments to thoroughly evaluate the delection performance of WOOD using three benchmark datasets. Here, recall that the proposed WOOD is weakly-superrised on training datasets con- taining 97% ID samples and 1% generated OOD samples for cach of the three scenarios. To create a balanced evaluation setting, the test datasets are equally distributed for ID and our three OOD scenarios (25% each). We report the results on three subsets of the test dataset for each scenario (i.e. Misaligned
9

---

Published in Transactions on Machine Learning Research (07/2024)
New-domain and Noisy+ID), as well as the overall performance the entire test set. Furthermore, we
conduct experiments to compare the computational cost of WOOD with that of the baselines in Appendix C
Table 2: Performance comparison of different methods for OOD detection on CUB-200 dataset averaged over three random sceds. The standard deviations across random sceds are reported following +. Higher AUROC and lower FPR95 indicate better performance.
Methods
II
New-domain+
Noisy
All Sommarice- + ID
AUROCT
FPR05
AUROCI
PPR951
AUROCI
PPR951
AUROCC
FPRO5
Zavo-shot
MCM-OOD
47.16 + 2.74
94.76 + 1.26
81.42
= 0.17
62.72 + 1.01
51.62
2.68
93.82 + 5.79
60.07
= 0.41
90.52 + 1.29
GI-MCM-OOD
46.90 1: 1:37
96.33 4 0.38
79.88 1.07
72.50 1.09
49.60 4: 3.91
96.25 4: 0.90
58.80
2.11
94.42 4: 1.53
Finetuned
CoOp-MCM
48.61 1,49
96.76
+ 0.67
90.30
48.83 2.02
52.63
3.30
96.36 + 0.65
63.91
+ 0.77
NPOS-MCM
19.46 + 1.76
91.38
+ 1.51
32.31
2.69
92.67 + 1.81
46.71
0.64
93.83 # 1.01
19.50
0.96
91.00
+ 1.36
CLIP-
49.96 +: 2.36
16,000
+: 1,80
80.21
= 4.86
83,33 1: 9.75
49.64
2: 0.64
95.17
4: 1.51
59.94
= 1.74
95.17
1 2.75
LoCoOp
49:25 1. 2.68
95.92
1 1.13
91.72
1.66
42.58
3.50
54.20
3,66
92.92 1. 0.14
65.05
2.62
91.90
1. 0.98
Wewkly-supervised
CLiP-Eaergy
49.45
1.45
05.25
1: 2.05
78.51
1.54
63,50
4.11
50.08
1.36
95.67
+: 0.63
39.35
= 2.81
93.58 1: 0.72
CLIP-BCE
50.37 1 1.25
84.83
1 0.29
88.61
3.16
29.50
1 5.25
95.10
0.39
15.67
1 3.13
78.08
1.87
46.87
1. 2.49
Qurs
91.71 0.53
28.47
1.25
99.08
0.92
3.63
+ 1.18
96.73 + 0.86
6.83
1.26
95.18
+ 0.25
13.98 +: 0.60
Table 3: Performance comparison of different methods for OOD detection on MIMIC-CXR dataset averaged over three random seeds. The standard deviations across random seeds are reported following +. Higher AUROC and lower FPR95 indicate better performance.
Misalinee+11
New-domain--D
Noiey +ID
All Scenatice 1D
Meshods
AUROCT
FFR851
AUROOC
FPR
AUROCT
PPRS6]
AUROOC
FPRO0L
Zero-shot
MCM-OOD
53.13 + 1.20
96.89 + 0.50
50,92 =
93.51 0.32
47.19
2.17
96.01 + 1.70
50.19
1.46
95.55 + 1.18
GL- MCM-OOOO
51.15 + 0.45
86.51 + 0.33
57.05
1.00
90.05 2.14
47.15
1.58
96.29 + 0.95
52.78
0.95
91.41 + 0.67
Finetuned
CoOp-MCM
33,11 + 1.51
97.90
+ 0.78
66.82 = 0.92
69.46 0.54
50.70 = 1.02
94,80 = 0.88
33.00
= 1.15
95.27 = 1.02
NCOS-MCM
47.83 1 0.54
94.90
1.35
37.81
88.61 1.62
52:22
0.72
94.94 4: 1.57
32.62
CLIP-N
64.21 1. 0.93
58.58
1 2.34
30.69
3.98
86.56
3.39
50.93
1.07
95,
1. 2.05
55.28
1.72
91.24
1. 0.77
LoCoOp
33.73 3: 0.78
97.34
4: 0.45
62.27
0.88
68.44
2.98
51.11
0.60
94.12 4: 1.04
49.04
0.30
94.80 J: 0.81
Waakl-sspervised
EnerrE
64.71 + 2.56
80.80 + 2.64
93.32 = 1.99
54.51
9.14
50.05
0.44
94.12 + 1.93
69.33
1.00
89.94 + 2.00
CLIP-BCE
56.52 4: 3.35
83.24 + 1.52
99.21 1.15
1.18
1.08
96.37
2.31
11.19 4: 7.09
85.45
1.66
32. 70
0.97
Ours
89.02 0.67
37.66 + 0.85
99.81 + 0.25
0.47 + 0.63
966.95 + 0.55
8.03 + 1.15
94.48 + 0.50
15.39 + 0.31
Table 4: Performance comparison of different methods for OOD detection on COCO datasct averaged over three random seeds. The standard deviations across random seeds are reported following t. Higher AUROC and lower FPR95 indicate botter performance.
Methods
Misaligned
New-dommin
Noiey
Sermarice ID
AUROCT
FPR951
AUROCT
FPR951
AUROCT
FPR95L
AUROCT
FPR951
Zera-shot
MCM-OOD
40.86
2.31
97.99 + 0.53
71.15 1.44 86.20
68.05 + 1.24
47.39 + 1.75
96.15 + 0.62
53.13 4 1.71
95.42 + 1.04
GL-MCM-OOD
35.44 4.01
90.73 + 0.46
3.87
46.40 + 16.86
47.00
+ 6.50
96.33 # 1.67
56.50 = 4.81
98.40 + 1.31
Finetuned
MCM
58.31
1.39
95.50
2:00
68.21
0.72
72.16 +: 3.64
44.51 4 1.94
96.13 +: 0.78
56.34 +: 1.11
93.45
4 2.05
MCM
34.06 + 0.93
96.04 + 0.74
74.81 + 1.33
92.57 # 1.17
49.32
4 0.95
95.98 + 0.71
52.73 + 0.46
96.57
+ 0.73
CLIP-N
61.97 = 1.66
85.21 + 2.31
72.73 +: 1.45
78.61 = 2.01
55.33
+ 0.13
92.11 + 1.67
63.35 +: 0.96
87.91
+ 1.47
LoCoOp
55.11 2.17
91.68 = 1.53
89.56 + 0.57
43.77 = 3.92
43.74 + 1.96
96.10 + 0.79
62.80 = 1.34
93.75
+ 0.83
Wewkily-npprvieed
57.83
1.34
84.73 1 4.18
96.07
0.50
24.46 1 9.46
52.65 1.84
93.84 1. 2.18
68.85 1 1.16
86.88
1 3.31
CLIP-BCEE
56.95
1.85
93.56
+ 2.40
98.39 0.17
6.69 + 0.38
98.73
+: 0.42
6.00
+: 1.30
84.69
+ 0.77
35.47
+ 1.38
Ours
99.78 + 0.07
1.14
1 0.32
99.88 :: 0.06
0.88 + 0.19
99.08 1 0.21
2.89
-: 0.35
99.58 + 0.07
1.54 + 0.31
We first assess the performance of the proposed WOOD CUB-200 dataset. Table illustrates the compar- ison results of different OOD detection methods using three random seeds. We can observe from this table that our method is effective in all three OOD sccnarios, and its overall performance significantly excceds the baselines. The main reason is that our method leverages both contrastive learning and a binary classifier for OOD detection on all three scenarios, as motivated by Fig. 2 Evidently, zero-shot methods such as MCM- OOD and GL-MCM-OOD do not perform well since they are only designed for detecting visual OODs by querying an image to check whether the returned features from CLIP align with training labels. In addition, although finetuned CLIP methods are designed to improve the detection performance on new-domain OODs,
10

---

Published in Transactions on Machine Learning Research (07/2024)
we discover that finetuning approaches such NPOS-MCM and CLIP-N can have adverse effect on per- formance when when there are not many OOD training images from classes distinguishable from "bird'. In particular, when NPOS-M MCMSMMMMPP generates training OOD examples on the k-NN boundary of ID images, these examples can overlap with ID images from another classes instead of OODs from COCO-bird, resulting in worse performance than zero-shot methods. Also, while CLIP-N can embed negation semantics from images to 'no" text prompts, these semantics may not be sufficiently distinct from ID "bird" features. On the other hand, CoOp and LoCoOp effectively build contexts from CLIP image features around the associated class labels, resulting in better new- domaón detection. Despite taking advantage of OOD training data, the small number of training OODs is not sufficient for CLIP-Energy and CLIP-BCE to beat context prompt fine- tuning for new-domain examples. Similarly to our method, CLIP-BCE can utilize weakly-supervised binary classification to elfectively detect. noisy visual OOD examples, but it still fails to detect misaligned textual OODs. In contrast, our method can detect both visual and textual OODs. Still, it is very challenging to identify a pair of OOD samples when an image is slightly unaligned with its textual information. Specifically in our experiment, misaligned lest samples include many similar images as those in the ID samples, while the textual descriptions also contain "bird' with hardly different fine- grained features. As result, the FPR95 score for detecting misaligned OODs is higher compared to other scenarios.
Next, we also show that WOOD can identify OOD samples in multi-modal medical data, MIMIC-CXR. As illustrated in Table 3 it can be observed that the proposed method effectively detects OODs in all three OOD scenarios and its overall performance is better than the baselines. Here, note that similar to CUB-200, the FPR95 of the misaligned scenario is also high since the test images have the same class label "chest scans" those in the training data and only have slightly different tertual descriptions. Furthermore, zero-shot and finetuned CLIP methods fail to detect OODs in all three scenarios because the pre-trained CLIP lacks the aligument capability for the medical image features and data labels, which is absent from its pretraining data, With additional seakly-supervisee training, better energy-based OOD scores or binary predictions respectively to significantly improve the detection performance for new-domain samples. However, CLIP-BCE falls short in detecting noisy OOD samples in this experiment with higher false positives. One possible explanation is that while a classifier is effective at detecting visual noise, the binary classifier in CLIP-BCE aligus the image features with the collection of ID class labels, instead of the corresponding textual descriptions using contrastive learning. On the other hand, this issue is clearly miligated by the interplay of our two components where our proposed method first aligns medical images with their detailed textual description with constrastive learning, then combines the learned features in conjunction with feature sparsity to make accurate predictions for all three OOD scenarios simultancously
and CLIP-BCE can learn
Furthermore, we apply the proposod WOOD to detect OOD samples on COCO dataset. We compare the detection performance of different methods as shown in Table 4 It can be seen that our approach is able to effectively detect OODs in all three OOD scenarios while the baseline methods can only detect new-domoin moisy OODs. In addition, since COCO and its corresponding new-domain dataset GCC both mostly contain images with the class label "person", zero-shot and finctuned CLIP basclines with except GL-MCM-OOD and LoCoOp struggle to detect these OODs. The reason why GL-MCM-OOD and LoCoOp oulperform the other baselines is that they can capture local CLIP features representative of OOD in the image background in the learnable context prompts around the class labels, as previously demonstrated by Miyai et al. (2023) and Miyai et al. (2024). However, by utilizing a small number of OOD training data, weakly-supervisel classification methods such as CLIP-BCF. and ours can capture these OOD nuances even more effectively, as evidenced by their superior performance for detecting new-domain OODs, as well as noisy visual OOD examples. Morcover, the detection performance for misaligned OODs is much higher than the previous two datasets, since COCO has a much larger training set that contains many easily distinguishable classes (e.g. person, cat, bird, etc.).
5.5 Ablation Studies
In this section, we conduct ablation studies to investigate the effect of some important components, the weight in the objective function, as well as the Hinge loss and its hyperparameter on the detection performance.
11

---

Published in Transactions on Machine Learning Research (07/2024)
Table 5: Impact ablating the binary classifier (BC) and the contrastive learning (CL) components from the proposed WOOD method. We report the results averaged over three random sceds. It can be seen that our proposed method outperforms both its counterparts without the binary classifier or the contrastive learning module for detecting three OOD scenarios all three datasets.
Misaligned + ID
New-domain + ID
Noisy +ID
Scenarios +
Modela
ACROCT
FPR95
ACROCT FPR95
AUROCT
FPR951
AUROCT
FPR05]
CUB-200
WOOD w/o BC
90.84
31.67
86.94
40.08
51.19
94.67
76.32
55.47
WOOD w/o CL
49.26
93.25
97.62
7.25
88. 74
33.83
78.52
44.72
WOOD
91.71
28.47
99.08
3.63
96.73
6.83
95.18
13.98
MIMIC-CXR
WOOD w/o BC
86.75
51.52
86.75
48.50
51.52
94.48
72.58
68.45
WOOD w/o CL
68.34
85.91
99.32
3.39
51.21
95.00
73.15
61.42
WOOD
89.02
37.66
99.81
0.47
96.95
8.03
94.48
15.39
COCO
WOOD w/o BC
99.73
1.60
90.43
27.12
53.59
94.02
81.25
41.24
WOOD w/o CL
77.90
80.91
99.48
2.02
96.80
10.22
91.39
31.05
WOOD
99.78
1.14
99.88
0.88
99.08
2.89
99.58
1.54
Effect of Main Model Components. First, to assess the efficacy of distinct components within the proposed WOOD method, we conduct an ablation study by systematically excluding individual components and measuring the OOD detection performance. The results are presented in Table 5 From this table, we can draw two conclusions: (i) notably, our method incorporating both the binary classifier (BC) and the contrastive learning (CL) components demonstrates superior performance; and (ii) each component exhibits notable influence on performance, affirming the necessity of their inclusion in our method. More specifically, the WOOD model without the binary classifier can detect misaligned OODs well, but its detection accuracy suffers when the OOD features appear outside the scope of the textual descriptions (i.e., in the background) for the new-domain and noisy scenarios. On the other hand, the WOOD model without contrastive learning lacks knowledge of the alignment between images and their corresponding textual descriptions, thus failing to detect. misaligned OODs. Therefore, our design of the proposed WOOD allows these two components to complement each other and significantly enhance the OOD detection performance for three scenarios simultaneously.
Effect of Weight in the Objective. We study the effect of the weight. in the objective function (A in Eq. 7 on the performance of OOD detection. We can see from Table 6 that the proposed WOOD model has the best performance when X = 0.4 as À decreases from 0.5 to 0.2 on all three datasets. Note that in the previous ablation study (Table 5) we demonstrate the detection performance when our method utilizes only the contrastive learning module when A in Eq. 7 or only the binary classifier when X == 1. Additionally, when = 0.5, the rastiveeive learning module and the binary classifier have equal weights in the objective. As illustrated in Table for more challenging datasets with highly similar images such as CUB-200 and MIMIC-CXR, low À values have much more pronounced negative impact on the OOD detection performance, the model becomes less effective in detecting new- domain and noisy scenarios. This further supports our motivation for combining two different. modules in the proposed WOOD.
Table 6: Impact of the weight of the binary classifier in the objective on detection performance. We report the results averaged over three random seeds. It can be seen that the proposed WOOD performs very well when x = 0.4 on all three datasets.
Dataset MIMIC-CXR
A
CUB-200
COCO
AUROCT
FPR95
AUROCT
FPR95
AUROCT
FPR95
0.5
95.15
15.72
94.41
17.20
99.49
1.66
0.4
95.18
13.98
94.48
15.39
99.58
1.54
0.3
90.55
30.20
94.08
19.33
99.46
1.20
0.2
82.57
40.97
85.28
38.11
99.39
1.56
Effect of Hinge Loss and its Hyperparameter, We also explore the impact of the margin hyperpa- rameter used in the Hinge loss (in Eq. 2 and Eq.
3) on the detection performance. Table illustrates
12

---

Published in Transactions on Machine Learning Research (07 (07/2024)
the detection performance under different margin parameters. In addition, ablate the Hinge loss (Eq. to investigate its impact on the model performance. Specifically, the model without the Hinge loss utilizes C1 as defined in Eq. ! for contrastive learning. We can observe from Table 7 that our method performs the best when m = 0.2 for CUB-200 and MIMIC-XRR. Since COCO has many casily distinguishable classes, our model benefits from using slightly large margin m = 0.3. Moreover, our detection method without the Hinge loss (L1) does not perform well for CUB-200 and MIMIC-CXR. Therefore, we can conclude that this Hinge loss plays an important role in multi-modal OOD detection.
Table 7: Impact of Hinge loss in the objective detection performance. We report the averaged results from three random seeds. It can be observed that when 0.2, the proposed model has the best performance on CUB-200 and MIMIC-XXR datascts. For COCO data, it performs best as m = 0.3.
Datasel
Let
CUB-200
MIMIC-CXR
COCO
AUROCT
FPR954
AUROC1
FPR95
AUROCT
FPR95
No Hinge
81.42
41.11
86,90
29.78
99.01
2.09
0.1
94.27
18.95
92.74
21.84
99.35
1.87
m 0.2
95.18
13.98
94.48
15.39
99.48
0.3
04.68
17.41
93.21
20.75
99.58
1.54
0.4
92.57
25.61
93.77
19.80
99.37
1.87
Effect of the Feature Sparsity Regularizer. Last but not least, we conduct ablations of the Feature Spansity Regularizer (FSR) used in the binary classifier measure its usefulness for OOD detection. From Table it can be observed that using sparse features for the binary classifier is beneficial for the detection performance, especially for the medical data. Since the textual description for each image contains about 40 words on average in MIMIC-XRR, the FSR module encourages the binary classifier to gather highly informative features from the lext modality for OOD classification.
Table 8: Impact of the Feature Sparsity Regularizer (FSR) used in the binary classifier on detection perfor- mance. Averaged results from three random seeds show that employing FSR leads to superior performance across all three datasets.
Dataset
CUB-200
MIMIC-CXR
COCO
FSR
AUROCI
FPR95,
AUROC1
FPR95,
AUROC
FPR95
No
94.75
15.64
84.41
37.21
99.23
1.64
Yes
95.18
13.98
94.48
15.39
99.58
1.54
6 Conclusions and Future Work
In this paper, we developed WOOD, general-purpose multi-modal OOD detection framework that inte- grates contrastive learning and binary classifier in a weakly-supervised manner. Concretely, we employed Hinge loss in the contrastive learning component to amplify the dissimilarity between ID and OOD pairs. In addition, we developed Feature Sparsity Regularizer to selectively incorporate crucial features from data of two modalities for the binary classifier. We also devised a novel scoring metric to effectively fuse pre- dictions from both components. The evaluation results validated that the proposed hybrid approach excels in identifying anomalies across three different OOD scenarios. Our future work will focus on applying the proposed WOOD to more OOD scenarios to better represent the wide spectrum of OOD environments the real world.
Acknowledgements
Research reported in this paper was sponsored in part by NSF under award CPS SSSF-231102 and Faculty
Research Grant at William & Mary 141446.
13

---

Published in Transactions on Machine Learning Research (07 2024)
References
Daniel Bogdoll, Maximilian Nitsche, and J Marius Zöllner. Anomaly detection in autonomous driving: A survey. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1488-4499, 2022.
Thomas Davenport and Ravi Kalakota. The potential for artificial intelligence in healthcare. Future healtheare journal, 6(2)994, 2019.
Dina Demner-Fushman, Mare D Kohli, Marc B Rosemman, Sonya E Shooshan, Laritza Rodriguez, Sameer Antani, George R Thoma, and Clement J McDonald. Preparing a collection of radiology examinations for distribution and retrieval. Journal of the American Medical Informalics Associalion, 23(2):304-3111 2016.
Terrance DeVries and Graham W Taylor. Learning confidence for out-of-distribution detection in neural networks. arXiv preprint Xiv: 1002. 04865, 2018.
Xuefeng Du, Xin Wang, Gabriel Gozum, and Yixuan Li. Unknown-aware object detection: Learning what you don't know from videos in the wild. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13678 13688, 2022.
Sepideh Esmaeilpour, Bing Liu, Eric Robertson, and Lei Shu. Zero-shot out-of-distribution detection based on the pretrained model clip. In Proceedings of the AAAI conference artificial intelligence, 2022.
Stanislav Fort, Jie Ren, and Balaji Takshminarayanan. Exploring the limits of out-of-distribbttio detection. Advances in Newral Information Processing Systems, 34:7068 708188778,1101 2021.
Mariana-Tuliana Georgescu, Antonio Barbalau, Radu Tudor Tonescu, Fahad Shahbaz Khan, Marius Popescu, and Mubarak Shah. Anomaly detection in video via self-supervised and multi-task learning. In Proceedings of the TEEE/CVF conference on compuler vision and pullern recognilion, pp. 12742-15 12742-1275, 2021.
Changhee Han, Leonardo Rundo, Kohei Murao, Takafumi Nemoto, and Hideki Nakayama. Bridging the gap between ai and healthcare sides: towards developing clinically relevant ai-powered diagnosis systems. In IFIP International Conference on Artificial Intelligence Applications and Innovations, pp. 320 333. Springer, 2020.
Zongbo Han, Fan Yang, Junzhou Huang, Changqing Zhang, and Jianhua Yao. Multimodal dynamics: Dynamical fusion for trustworthy multimodal classification. In Proceedings of the IEFE/CVF Conference on Computer Vision and Pattern Recognition, pp. 20707 20717, 2022.
Matthias Hein, Maksym Andriushchenko, and Julian Bitterwolf. Why relu networks yield high-confidence predictions far away from the training data and how to mitigate the problem. In Proceedings of the ICFF Conference on Compuler Vision and Pallern Recognition, pp. 41 11-50, 2019.
Dan Hendryeks and Kevin Gimpel. A baseline for detecting misclassified and out-of distribuutn examples neural networks. arXin preprint arXiv: NNi::6610002190 2016.
Dan Hendryeks and Kevin Gimpel. A baseline for detecting misclassified and duttof-disttibuttin examples in neural networks. In International Conference on Learning Representations, 2017.
Dan Hendrycks, Mantas Mazeika, and Thomas Dietterich. Deep anomaly detection with outlier exposure. arXiv preprint ar X 1at::8122000 04606, 2018.
Rui Huang and Yixuan Li. Mos: Towards scaling outtof-ddistrbbation detection for large semantic space. In Proceedings of the IEFE/CV Conference on Computer Vision and Pattern Recognition, pp. 8710-8719, 2021.
Zhuo Huang, Chang Liu, Yinpeng Dong, Hang Su, Shibao Zheng, and Tongliang Liu. Machine vision therapy: Multimodal large language models can enhance visual robustness via denoising in-context learning. arXiv preprint ar 2fo:2312.025.. 2023.
14

---

Published in Transactions on Machine Learning Research (07/2024)
Rasheed Hussain and Sherali Zeadally. Autonomous cars: Research results, issues, and future challenges. IEFE Commaniicalions Surveys 8 Tulorials, (12)):1255-1313, 2018.
Jongheon Jeong, Yang Zou, Taewan Kim, Dongqing Zhang, Avinash Ravichandran, and Onkar Dabeer. Win- clip: Zero-/few-shot anomaly classification and segmentation. In Proceedings of the TEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 19606 19616, 2023.
Tianchen Ji, Sri Theja Vuppala, Girish Chowdhary, and Katherine Driggs-Camplen Campbell. Multi-modal anomaly detection for unstruct ured and uncertain enviromments. arXiv preprinl wrXiv22012.06673 2020.
Qian Jiang, Changyou Chen, Han Zhao, Liqum Chen, Qing Ping, Son Dinh Tran, Yi Xu, Belinda Zeng, and Trishul Chilimbi. Understanding and constructing latent modality structures in multi-modal representation learning. In Proceedings of the IEEE/CVE Conference Computer Vision and Pattern Recognition, pp. 7661-7671, 2023.
Alistair EW Johnson, Tom Pollard, Seth Berkowitz, Nathaniel Greenbaum, Matthew P Lungren, Chih-ying Deng, Roger G Mark, and Steven Horng. Mimic-cxi, de-identified publicly available database of chest radiographs with free-text reports. Scientific data, 6(1):1 8, 2019.
Myong Chol Jung, He Zhao, Joanna Dipnall, and Lan Du. Beyond unimodal: Generalising neural processes for multimodal uncertainty estimation. Advances in Newral Information Processing Systems, 36, 2023.
Ramneet Kaur, Kaustubh Sridhar, Sangdon Park, Susmit Jha, Anirban Roy, Oleg Sokolsky, and Insup Lee. Codit: Conformal out-of-distribution detection in time-series data. ar preprinl or Xxxx:227.11769,, 2022.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv:1412.600 2014.
arXiv preprint
Daoheng Li, Xiushan Nie, Xiaofeng Li, Yu Zhang, and Yilong Yin. Context-related video anomaly detection via generative adversarial network. Pattern Recognition Letters, 156:183-189, 2022a.
Yewen Li, Chaojie Wang, Xiaobo Xia, Tongliang Liu, Xin Miao, and Bo An. Out-of-distribttoo detection with an adaptive likelihood ratio on informative hierarchical vac. In Advances in Neural Information Processing Systems, 2022b.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740 755. Springer, 2014.
Jiashuo Liu, Zheyan Shon, Yue Hc, Xingxuan Zhang, Renzhe Xu, Han Yu, and Peng Cui. Towards out-of- distribution generalization: A survey, arXiv preprint ar Xiv:21108 13624, 2021.
Weitang Liu, Xiaoyun Wang, John Owens, and Yixuan Li. Energy-based out-of-distribution detection. Advances in Neural Information Processing Systems, 33:21464-2144 2020.
Philipp Liznerski, Lukas Ruff, Robert A Vandermeulen, Billy Joe Franks, Klaus-Robert Müller, and Marius Kloft. Exposing outlier exposure: What can be learned from few, one, and zero outlier images. arXiv preprint ar 2iv: 114 74, 2022.
Xiaoxiao Ma, Jia Wu, Shan Xuc, Jian Yang, Chuan Zhou, Quan Shong, Hui Xiong, and Leman Akoglu. A comprehensive survey on graph anomaly delection with deep learning. IEEE Trunsactions on Knowledge and Data Engineering, 2021.
Samual MacDonald, Kaiah Steven, and Maciej Trzaskowski. Interpretable ai in healthcare: Enhancing fairness, safety, and trust. In Artificial Intelligence in Medicine, pp. 241-258. Springer, 2022.
Snchashis Majhi, Srijan Das, François Brémond, Ratnakar Dash, and Pankaj Kumar Sa. Weakly-supervise joint anomaly detection and classification. In 2021 16th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2021), pp- 1 1-7. IEEE, 2021.
15

---

Published in Transactions on Machine Learning Research (07/2024)
Yifei Ming, Ziyang Cai, Jiuxiang Gu, Yiyon Sun, Wei Li, and Yixuan Li. Delving into out-of-distribution detection with vision-language representations. In Advances in Newral Information Processing Systems. 2022a.
Yifei Ming, Ziyang Cai, Jiuxiang Gu, Yiyou Sun, Wei Li, and Yixuan Li. Delving into out-of-distribution detection with vision-language representations. In Advances in Neural Information Processing Systems, 2022b.
Yifei Ming, Ying Fan, and Yixuan Li. Poem: Out-of-distribution detection with posterior sampling. In International Conference on Machine Learning, pp. 15650-15665. PMTR, 2022c.
Atsuyuki Miyai, Qing Yu, Go Trie, and Kiyoharu Aizawa. Zero-shot in-distribution detection in multi-object settings using Aision-language foundation models. arXiv preprint orXiv:2304.0452,, 2023.
Atsuyuki Miyai, Qing Yu, Go Irie, and Kiyoharu Aizawa. Locoop: Few-shot out-of-distribution detection via prompt learning. Advances in Neural Information Processing Systems, 36, 2024.
Vivek Narayanaswamy, Yamen Mubarka, Rushil Anirudh, Deepta Rajan, Andreas Spanias, and Jayaraman J Thiagarajan. Know your space: Inlier and outlier construction for calibrating medical ood detectors. In Medical Imaging with Deep Learning, 2023.
Alexander Podolskiy, Dmilry Lipin, Andrey Bout, Ekaterina Artemova, and Irina Piontkovskaya. Revis- iting mahalanobis distance for transformer-based out-of-domain detection. In Proceedings of the AAAT Conference on Artificial Intelligence, volume 35, pp. 13675 13682, 2021.
Yuning Qiu, Teruhisa Misu, and Carlos Busso. Unsupervised scalable multimodal driving anomaly detection.
IEEE Transoctions on Intelligent Vehicles, 2022.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkinn Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pp. 8748-8763. PMLR, 2021.
Scott Reed, Zeynep Akata, Honglak Lee, and Bernt Schiele. Learning deep representations of fine-grained visual descriptions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 49 58, 2016.
Jie Ren, Stanislav Fort, Jeremiah Liu, Abhijit Guha Roy, Shreyas Padhy, and Balaji Laishminarayanan. A simple fix to mahalanobis distance for improving near-ood detection. arxiv preprint arXiv:2106 090222 2021.
Mauricio Romero and Pablo A Estévez. Outlier detection based on transformations for astronomical time series. In 2022 International Joinl. Conference on Neural Networks (IJCNN), pp. 1-8. IEFE, 2022.
Abhijit Guha Roy, Jie Ren, Shekoofeh Azizi, Aaron Loh, Vivek Natarajan, Basil Mustafa, Nick Pawlowski, Jan Freyberg, Yuan Liu, Zach Beaver, et al. Does your dermatology classifier know what it doesn't know? detecting the long-tail of unseen conditions. Medical Image Analysis, 75:102274, 2022.
Lukas Ruff, Jacob R Kauffmann, Robert A Vandermeulen, Grégoire Montavon, Wojeiech Samek, Marius Kloft, Thomas G Dietterich, and Klaus-Robert Müller. A unifying review of deep and shallow anomaly detection. Proceedings of the JEFF, (09(5):756-799) 2021.
Mohammadreza Salehi, Hossein Mirzaei, Dan Hendrycks, Yixuan Li, Mohammad Hossein Rohban, and Mohammad Sabokrou. A unified survey anomaly, novelty, open-set, and out of-distribution detection: Solutions and future challenges. Transactions of Machine Learning Research, 2022.
Piyush Sharma, Nan Ding, Scbastian Goodman, and Radu Soricut. Conceptual captions: cleaned, hyper- nymed, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Lúnguisties (Volume 1: Long Papers), pp. 2556-2565, 2018.
16

---

Published in Transactions on Machine Learning Research (07/2024)
Yang Shu, Xingzhuo Guo, Jialong Wu, Ximei Wang, Jianmin Wang, and Mingsheng Long. Clipood: Gen- eralizing clip to out-of-ddistributions. In International Conference on Machine Learning, pp- 31716-31731. PMLR, 2023.
Waqas Sultani, Chen Chen, and Mubarak Shah. Real-world anomaly detection in surveillance videos. Tn Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 6479 6488, 2018.
Lei Sun, Kailun Yang, Xinxin Hu, Weijian Hu, and Kaiwei Wang. Real-time fusion network for rgb-d semantic segmentation incorporating unexpected obstacle detection for road-driving images. IEEE Robotics and Automation Letters, 5(4):5558-555) 2020.
Yiyou Sun and Yixuan Li. Dice: Leveraging sparsification for out-of-distribution detection. In Compuler Viston-ECOO 2022: 17th European Conference, Tel Aviu, Israel, October 23- 23-27, 2022, Proceedings, Part XXIV, pp. 691-708. Springer, 2022.
Yiyou Sun, Chuan Guo, and Yixuan Li. React: Out-of-distribution detection with rectified activations. Advances in Neural Information Processing Systems, 34:144-157, 2021.
Yiyou Sun, Yifei Ming, Xiaojin Zhu, and Yixuan Li. Out-of-distribution detection with deep nearest neigh- bors. In International Conference on Machine Learning, 2022.
Leitian Tao, Xuefeng Du, Xiaojin Zhu, and Yixuan Li. Non-parametric outlier synthesis. arXiv preprint arXiv:2303.00 02966, 2023.
Engkarat Techspanurak, Masanori Suganuma, and Takayuki Okatani. Hyperparameter-free out-of- distribution detection using cosine similarity. In Proceedings of the Asian conference on computer vision, 2020.
Yu Tian, Gabriel Maicas, Leonardo Zorron Cheng Tao Pu, Rajvinder Singh, Johan W Verjans, and Gustavo Carneiro. Few-shot anomaly detection for polyp frames from colonoscopy. In International Conference Medical Image Computing and Computer-Assister Intervention, pp. 274 284. Springer, 2020.
Weijie Tu, Weijian Deng, and Tom Gedeon. A closer look at the robustness of contrastive language-image pre-training (clip). Advances in Neural Information Processing Systems, 36, 2023.
Joost Van Amersfoort, Lewis Smith, Yee Whye Tch, and Yarin Gal. Uncortainty estimation using a sin- gle deep deterministic neural network. In International conference on machine learning, pp. 9690 9700. PMLR, 2020.
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-ucsd birds- 200-2011 dataset. 2011.
Fengjun Wang, Sarai Mizrachi, Moran Beladev, Guy Nadav, Gil Amsalem, Karen Lastmann Assaraf, and Hadas Harush Boker. Mumic multimodal embedding for multi-label image classification with tempered sigmoid. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 15603 15611, 2023a.
Hualiang Wang, Yi Li, Huifeng Yao, and Xiaomeng Li. Clipn for zero-shot ood detection: Teaching clip to say no. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. 1802-18122 2023b.
Jun Wang, Li Zhang, Yanjun Huang, and Jian Zhao. Safety of autonomous vehicles. Journal of advanced transportation, 2020, 2020.
LeiChen Wang, Simon Giebenhain, Carsten Anklam, and Bastian Goldluecke. Radar ghost target detection via multimodal transformers. IEEE Robotics and Automation Letters, 6(4):7758-7765 2021.
Lin Wang, Fuqiang Zhou, Zuoxin Li, Wangxia Zuo, and Haishu Tan. Abnormal event detection in vidcos using hybrid spatio-temporal autoencoder. In 2018 25th IEEE International Conference on Image Processing (ICTP), pp. 2276 2280. IEFE, 2018.
17

---

Published in Transactions on Machine Learning Research (07/2024)
Zhisheng Xiao, Qing Yan, and Yali Amit. Likelihood regret: An out-of-distribution detection score for variational auto-encoder. Advances in neural information processing systems, 33:20685- 20696, 2020.
Jingkang Yang, Kaiyang Zhou, Yixuan Li, and Ziwei Liu. Generalized distr--distribution detection: A
survey. Xixin preprint orXiu:210 10. 1334, 2021.
Jaemin Yoo, Tiancheng Zhao, and Leman A koglu. Data augmentation is a hyperparameter: Cherry-picked self-supervision for unsupervised anomaly detection is creating the illusion of success. Transactions on Machine Learning Research, 2023.
Karina Zadorozhny, Patrick Thoral, Paul Elbers, and Giovanni Cinà. Out-of-distribution detection for medical applications: Guidelines for practical evaluation. In Mullimodal Al in healthcare: A paradigm shift in health intelligence, pp. 133-153. Springer, 2022.
Alireza Zaeemzadeh, Niccolo Bisagno, Zeno Sambugaro, Nicola Conci, Nazanin Rahnavard, and Mubarak Shah. Duttof-distributtinn detection using union of 1-dimensional subspaces. In Proceedings of the IEEE/CVF Conference on Compuler Vision and Pattern Recognition, pp. 9452 9461, 2021.
Li Ming Zhan, Haowen Liang, Bo Liu, Lu Fan, Xiao Ming Wu, and Albert YS Lam. Out-of-scope intent de- tection with self-supervision and discriminative training. In Joinl Conference of the 59th Annual Meeling of the Association for Computational Linguistics and the 11th International Joint Conference on Natu- ral Language Processing, ACL-IJCNLP 2021, pp. 3521 3532. Association for Computational Linguistics (ACL), 2021.
Xingxuan Zhang, Jiansheng Li, Wenjing Chu, Junjia Hai, Renzhe Xu, Yuqing Yang, Shikai Guan, Jiazheng Xu, and Peng Cui. On the preprint arXic:2402.06599 2024.
generalization of multimodal large language models. arXiv
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Záwei Liu. Conditional prompt learning for vision- language models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recog- nition, pp. 16816-16825, 2022a.
Wenxuan Zhou, Fangyu Liu, and Muhao Chen. Contrastive out-of-distribution detection for prctrained transformers. arXiv preprint Xiv::22104 08812, 2021.
Yunhua Zhou, Peiju Liu, and Xipeng Qiu. Kun-contrastive learning for out-of-domain intent classification. Proceedings of the 60th Avnual Meeling of the Association for Compulational Linguistics (Volume 1: Long Papers), pp. 5129-52 5129-5141, 2022b.
18

---

Published in Transactions on Machine Learning Research (07/2024)
A Additional Related Work on Multi-Modal OOD Generalization
Besides OOD detection, OOD generalization is also an emerging topic of machinc learning rescarch that focuses on complex scenarios where the testing distribution is different from those of the training data Liu et al.
2021 In particular, multi-modal OOD generalization aims to improve the robustness of multi-modal
models on -iitr--iditrinttiion testing data (Shu et al., 2023 Tu et al. 2023 Jung et al. 2023 Huang et al, 2023 Zhang et al. 2024). Shuu et al. (023)) developed a fno-tuning approach using class adaptive margins
adapt CLIP models to OOD scenarios where both domain shifts and open classes may occur on the unseen testing data. [Tu et al. (2023) studied the robustness of CLIP to variations in fine-grained visual factors such as shapes and textures, and adopted post-hoc prediction uncertainty calibration for better classification accuracy across diverse distributions. Juung et al. (2023) considered various types of noise as OOD multi- modal inputs, and proposed Multi-modal Neural Processes to generalize neural processes for multi-modal classification and uncertainty estimation. Moreover, Huang et al. (2023) focused on generalizing Multi-modal Large Language Models (MLLMs) for out-of-domain vision lasks, utilizing in-contexl learning to detect and rectify incorrect model predictions. Similarly to Huang et al. (2023), Zhang et al. (2024) investigated the generalization ability of MLLMs in out-of-domain OOD sccnarios and studied the effectiveness of in-context learning in bolstering MLLMss adaplability.
In sunmarry these studies primarily addressed the generalization of multi-modal models for specific types of OOD, such as out-of-domain or noisy classcs, where OOD detection took the back seat as a sccondary objective. In contrast, our study focused on enhancing the detection ability of multi-modal models for OOD samples belonging to several scenarios including misalignment, out-of-domain, and noise simultancously the fine-grained manner.
B Evaluation of CLIP's Generalization to Noisy OOD Samples
In this section, we evaluate the CLIP model (ViT-B/16) using linear probing for bird species classification on the CUB-200 dataset. Then, experimentally add Gaussian blur to different percentages of image patches, ranging from 10% up to 50%, and report the Macro-F1 scores to measure the model's classification accuracy for noisy images, as shown in Table 9 below. In particular, even when little noise is added (10% of patches), the prediction performance of the CLIP model drops by 6% if half of the testing images contain noise, and by 14% if all images are noisy. Additionally, the CLIP model only achieves a 60.12% Macro-F1 score on the noisy testing dataset with 50% noisy patches, which a 22% reduction in performance. Therefore, we conclude that CLIP is not robust to noisy images.
Table 9: The OOD generalization evaluation for classification noisy samples on the CUB-200 dataset. The performance is measured using Macro-F1 scores (%) averaged over throe random sceds.
% Samples \% Patches
10
50
0
77.62
25
75.62
74.43
73.58
50
72.78
70.04
69.22
100
66.74
62.40
60.12
C Evaluation of Computational Cost
In this section, conduct experiments analyze the computational costs of our proposed method, and other baselines which utilize fine-tuning for CLIP. Additionally, develop lightweight variant of our approach, enabling training solely on the final two transformers layers of the CLIP image and text encoders. The computational cost. results are outlined in Table 10 while Table showcasse a comparative evaluation of OOD detection performance between the lightweight and fully-trained versions across three datasets. Notably, our lightweight method demonstrates competitive training speed compared to the basclines while improving OOD detection performance. Morcover, partially training the CLIP model achieves competitive OOD detection performance compared to both the best baseline, CL.P--BF,, and its fully-trained counterpart
19

---

Published in Transactions on Machine Learning Research (07/2024)
the CUB-200 and COCO datasets, shown in Table 11 However, the lightweight approach does not perform well on the MIMIC-CXR dataset due to CLIP's lack of pretraining on medical data.
Table 10: Computational cost comparison between proposed method and the baselines that finetune CLIP. The training speed is measured by training throughput in samples/seconee averaged over three random seeds. It can be observed that our lightweight method demonstrates competitive training speed compared to the baselines while improving OOD detection performance. Additionally, we specify whether each model component such as Image Encoder, Text Encoder, and Classifier is fully-trained partially-trained
frozen (O), or absent. (-)
Throughput (samples/s)
Methods
Image Encoder
Text Encoder
Classifier
CUB2200
MIMIC-CXR
COCO
CoOp-MCM
107.0 0.19
123.3 0.27
139.9 0.43
NPOS-MCM
123.3 = 0.36
108.7 0.44
125.1 0.57
CLIP-N
99.7 0.28
90.5 0.37
102.7
0.49
LoCoOp
96.3 = 0.17
110.0
0.24
125.9 + 0.39
CLIP-EAA CLIP-Energy
67.3 0.24
62.8 + 0.30
69.0 0.46
CLIP-BCE
80.4 = 0.27
73.5 0.37
83.0 + 0.45
Ours (light.)
110.3 0.17
105.1 + 0.28
115.4 0.38
Ours (full)
71.7 = 0.12
66.0
0.25
73.1 0.34
Table 11: Performance comparison between the lightweight and fully-trained version of our proposed method averaged over three random seeds. It can be seen that partially training the CLIP encoders results in marginal performance reduction on the CUB-200 and COCO datasets, but full training is necessary for MIMIC-CXR due to the lack of CLIP pretraining on medical data.
Misaligned ID
New-domain-
Noisy
Soenarios
Components
AUROCT
FPR95,
AUROCT
FPR951
AUROCT
FPR95]
AUROCT
FPR95,
CUB-2200
CLIP-BCE
50.37 + 3.25
94.83 +: 0.29
88.61 = 3.16
29.50 +: 5.25
95.10 + 0.39
15.67 + 3.13
78.08 1.87
16.87 = 2.49
Ours (Light)
89.84 + 1.60
36.67
3.82
98.90 - 0.81
4.33 1: 3.17
90.74 = 1.69
21.25
1: 3.54
93.39
0.33
19.86
0.40
(full)
MIMIC-CXR
CLIP-BCE
36.32 3.35
93.24
1.52
99.21 = 1.15
1.16 = 1.08
96.37 + 2.31
11.19
7.09
85,45
1.66
32.70
0.97
Ours (light)
78.37
1.82
63.23 1: 1.60
99.69
1: 0.12
1.28 1. 0.48
51.72 1. 0.87
93.97 1: 1.88
76.59
0.79
52.82
1.10
Ours (full)
89.02 + 0.67
37.66 = 0.85
99.81 - 0.25
0.17 + 0.63
96.95
0.55
8.03 + 1.15
91.48 = 0.50
15.39 = 0.31
COCO
CL.PP-BCE
56:95 1.85
93.50
30
0.17
0.69 +
98.73
0.42
1.30
$4.69
0.77
35.47 =
Ours (light)
99.55 0.23
1.60
93.58
15.43
0.75
97.68
5.75
Ours [Tul]]
99.78 0.07
1.14 = 0.32
99.88
0.06
0.88 = 0.19
99.08
0.21
2.89
0.35
99.58
0.07
1.54 = 0.31
20


Reviews:
[{"review_id": "leMcvq5gnx", "reviewer": "Reviewer_QqpD", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- The paper addresses multi-modal OOD beyond image-text alignment. Two additional challenging scenarios are considered where the images are either from a different domain or are noisy.\n\n- The proposed hinge loss for contrastive learning further helps increase the similarity between text and image of ID samples. Additionally, it constrains the image-text similarity for OOD samples, improving their detection.\n\n- Empirical results demonstrate improvements over various baselines on three benchmark datasets.\n\nWeaknesses:\n\n- Experimental Setup:\n\nIn Section 5.1, it is mentioned that misaligned samples are generated by replacing the text of an image-text pair with text from another category. However, while discussing the main results in Section 5.4, it is said that the misaligned samples have only fine-grained differences such as colour or appearance. How do you explain this discrepancy?\n\n- Baselines:\n\nThe baseline CLIP-BCE has a very similar objective to the proposed mehtod, where they minimize the similarity between image-text pairs for ID samples and maximize it for OOD samples. However, in the results, it seems that this baseline performs the worst at detecting the misaligned OOD samples for which it is trained rather than detecting OOD samples that are noisy or belong to a new domain. How do you explain this? This would allow us to understand the shortcomings of CLIP-BCE and the strengths of WOOD.\n\n\n- Readability: \n\nThe paper is hard to follow in places. For example, the encoders introduced while discussing the binary classifier are not well defined. It is only in the discussion of the model setup, that the readers are informed that the encoders are single layer MLPs.\n\nFurthermore, without knowing that the input and output of the encoders are the same, it is difficult to understand how the weights can be multiplied by the features. Mentioning the mapping or the dimensions could further improve the readability.\n\n- Additional Information:\n\nThe scoring metric in Equation 8 is not clear. To the best of my understanding, the value of $P_{cl}$ does not have a range of 0 to 1 as it is obtained by applying sigmoid on the cosine similarity, whose value lies in the range -1 to 1. Thus, the scoring metric would also never be 1. Please give more details and provide the range in which the values of the scoring metric lie.\n\n\n- Extended Ablations:\n\nWhile discussing the results from the ablation studies, the authors refer to the three OOD scenarios and discuss how each is addressed by a specific choice. The authors should provide performance for each OOD scenario separately rather than just the mean performance.\n\n\n- Experimental Results:\n\nIn the paper it is mentioned that the results are averaged over three random seeds but the authors do not provide standard errors.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The goal of this paper is to address multi-modal OOD detection. Specifically, it examines three distinct OOD scenarios: misaligned, new-domain, and noisy. The paper introduces a weakly supervised multi-modal OOD detection model named WOOD, which includes a binary classifier and a contrastive learning module to assess the similarity between various modalities. For the contrastive learning module, the authors use a hinge loss, whereas the binary classifier employs a feature sparsity regularizer to effectively integrate relevant features from multiple modalities. Empirical evaluations demonstrate that the WOOD model provides improvements over existing baselines.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Major comments\n\n- See weaknesses above.\n\n- The authors use three datasets in the experiments. It is not clear if these are the only datasets used by SOTA methods. Can the authors comment on this?\n\n\n- I think the following paper is relevant to this work. Could the authors discuss it? How does the proposed approach addresses issues raised in this paper? \n\t- https://openreview.net/forum?id=HyzCuCV1jH\n\n\nMinor comments\n\n- \"We generate a small number of OOD samples K during training\"\n\t- It is not during the training process that these samples are generated.\n-  \"we adopt Hinge loss instead of L1 in Eq. 1\"\n\t- Please provide the L1 expression, at least in the appendix. Was it used before in other papers?\n\n\n- \"to obtain the OOD predictions\", \"the prediction results\", etc.\n\t- what do you mean by predictions? Please be more specific, i.e. probability, label, etc.\n\n- Page 6, circled plus and product are not defined.\n\n- In (7), please give the range for lambda (lambda in [0, 1]?).\n\n- Page 11, \"performs the best performance\"\n\n- Table 6, why did not not consider more extreme values of lambda (lambda = 0 or 1?)\n\n- The plots in Figure 2 are missing the axis labels which make it difficult to follow.\nIn Equation 2, there is a missing bracket.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "9bTuroSnTp", "reviewer": "Reviewer_8hZH", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. Multi-model anomaly detection is a relatively unexplored field, making it worthwhile to focus on. \n2. The paper is well-written and easy to follow.\n3. The proposed methods are straightforward yet efficient.\n\nWeaknesses:\n1. The assumption of noisy images as anomalies may impact the model's out-of-distribution (OOD) generalization. An OOD generalization experiment is necessary to assess the method's robustness.\n2. While the rationale behind the method is sound, its novelty is limited. Exploring the use of zero-shot anomaly detection on each modality and merging the results, as demonstrated in approaches like WinCLIP, could enhance novelty.\n3. The method mentions using outliers for training, but it's unclear how these outliers are selected and why. Additionally, assumptions like outlier exposure need further clarification and validation.\n4. Experimentation should include comparisons with single-modality approaches. The expectation is that the proposed method should perform significantly better due to its utilization of more information during inference.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents a novel approach for multi-model anomaly detection, leveraging a methodology akin to CLIP. It employs separate encoders for text and images, and learns a unified representation through joint encoding. Three anomaly scenarios are considered: anomalous images or text, text unrelated to images, and noisy images. Experimental results validate the efficacy of the proposed method, demonstrating its efficiency in anomaly detection.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The assumption that noisy images represent anomalies might affect the model's out-of-distribution (OOD) generalization. Therefore, conducting experiments specifically designed to evaluate OOD generalization is crucial to gauge the method's robustness.\n\n Although the method's rationale is well-founded, its novelty appears limited. To enhance novelty, consider investigating the application of zero-shot anomaly detection independently on each modality and subsequently merging the results, akin to methodologies demonstrated in approaches like WinCLIP.\n\nClarification is needed regarding the selection criteria and rationale for using outliers in training. Additionally, assumptions such as outlier exposure require further elaboration and validation to ensure their reliability and applicability.\n\n It is imperative to compare the proposed method with single-modality approaches in the experimentation phase. Given the utilization of additional information during inference, it is expected that the proposed method will outperform single-modality approaches by a significant margin.", "broader_impact_concerns": "n/a", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "UaBI5HYV7C", "reviewer": "Reviewer_wSrD", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:  \n- This paper studies OOD detection using multi-modal models, which is quite a novel and interesting study direction.  \n- The writing is good, is not hard to understand the intention of the authors.  \n- The Experimental improvement is very good and significantly surpasses many baseline methods.  \n\nWeaknesses:  \n- The proposed method needs clearer motivation to justify its design. Why using Hinge loss instead of typical contrastive loss is not sufficiently and clearly explained. Why can Hinge loss constrain the similarity score? Why is a feature sparsity regularizer employed? Why can sparsity help OOD detection in this scenario?  \nThe authors are referred to the following literature for OOD data under sparsity:  \n[1] Zhang et al., Can Subnetwork Structure be the Key to Out-of-Distribution Generalization?  \n\n- Training CLIP is computationally expensive. However, the training efficiency is not discussed. Compared to other baseline methods, the effectiveness has been largely improved, but the computational cost compared to other methods should be more elaboratively demonstrated.  \n- There are many loss terms and designs in the proposed approach. However, only contrastive learning and binary classification have been considered in the ablation study. There should be more components in this method, including using Hinge loss, sparsity terms, etc.\n- Is this framework only applicable to CLIP? Are there any other models that can be used for this method?  \n- Missing several references about multi-modal models under OOD scenario:  \n[2] Jung et al., Beyond Unimodal: Generalising Neural Processes for Multimodal Uncertainty Estimation.  \n[2] Huang et al., Machine Vision Therapy: Multimodal Large Language Models Can Enhance Visual Robustness via Denoising In-Context Learning.  \n[3] Zhang et al., On the Out-Of-Distribution Generalization of Multimodal Large Language Models.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies OOD detection using multi-modal models such as CLIP. There are three types of data that need to be detected in multi-modal OOD detection, namely ID misaligned data, OOD aligned data, and OOD misaligned data. In order to address this problem, the CLIP models need to be trained to identify both the misalignment between image and text and the image from new domains. In this paper, the authors propose to conduct contrastive learning to minimize the confidence in OOD data meanwhile increasing the confidence in ID data. Instead of using typical contrastive loss, the authors propose to use Hinge loss to constrain the similarity score. Moreover, for the detection of the OOD data, the authors propose a binary classifier to classify OOD data from ID data.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please see weaknesses for details.", "broader_impact_concerns": "No ethic issue.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a method for multi-model out-of-distribution detection. Unlike previous work, various forms of "out-of-distributionness" are considered, and the empirical results are good. Reviewers support acceptance, and I do as well.
Decision: Accept (poster)