Adapting Contrastive Language-Image Pretrained (CLIP) Models for Out-of-Distribution Detection
Authors: Nikolas Adaloglou
Felix Michels
Tim Kaiser
Markus Kollmann
Venue: Accepted by TMLR ()

Abstract: We present a comprehensive experimental study on pre-trained feature extractors for visual out-of-distribution (OOD) detection, focusing on leveraging contrastive language-image pre-trained (CLIP) models. Without fine-tuning on the training data, we are able to establish a positive correlation ($R^2\geq0.92$) between in-distribution classification and unsupervised OOD detection for CLIP models in $4$ benchmarks. We further propose a new simple and scalable method called \textit{pseudo-label probing} (PLP) that adapts vision-language models for OOD detection. Given a set of label names of the training set, PLP trains a linear layer using the pseudo-labels derived from the text encoder of CLIP. Intriguingly, we show that without modifying the weights of CLIP or training additional image/text encoders (i) PLP outperforms the previous state-of-the-art on all $5$ large-scale benchmarks based on ImageNet, specifically by an average AUROC gain of 3.4\% using the largest CLIP model (ViT-G), (ii) linear probing outperforms fine-tuning by large margins for CLIP architectures (i.e. CLIP ViT-H achieves a mean gain of 7.3\% AUROC on average on all ImageNet-based benchmarks), and (iii) billion-parameter CLIP models still fail at detecting feature-based adversarially manipulated OOD images. The code is available at https://github.com/HHU-MMBS/plp-official-tmlr2024.

Tim Kaiser Published in Transactions on Machine Learning Research (03/2024)
Adapting Contrastive Language-lmage Pretrained (CLIP) Models for Out-of-Distribution Detection
Nikolas Adaloglou
adaloglo@thha..om
Heinrich Heine University, Duesseldorf
Felix Michels
felkz. Rekem.iccellsiiee @hho..
Heinrich Heine University, Duesseldorf
tikai 1ikai103aml 03t.h003hhhhcco de
Heinrich Heine University, Duesseldorf
Markus Kollman Heinrich Heine Universily, Duesseldorf
korkus.kollmazzse.eee de
Reviewed on OpenReview: https: // optp//vpenee.e.... net/ forum? id= YCgX7sJRF1
Abstract
We present comprehensive experimental study pre-trained feature extractors for visual (OOD) detection, focusing on leveraging contrastive language-image pre- trained (CLIP) models. Without fine-tuning on the training data, we are able to establish a positive correlation (R² 2 0.92) between in-distribution classification and unsupervised OOD detection for CLIP models in 4 benchmarks. We further propose a new simple and scalable method called pseudo-label probing (PLP) that adapts vision-language models for OOD detection. Given set of label names of the training set, PLP trains linear layer using the pseudo-labels derived from the text encoder of CLIP. Intriguingly, we show that without modifying the weights of CLIP or training additional image/ stagge/txxtoo encoders (i) PLP outperforms the previous state-of-the-ar on all 5 large-scale benchmarks based on ImageNet, specifically by an average AUROC gain 3.4% using the largest CLIP model (ViT-G), (ii) linear probing outperforms fine-tuning by large margins for CLIP architectures (i.c. CLIP ViT-H achieves a mean gain of 7.3% AUROC on average on all ImageNeet-bsed benchmarks), and (iii) billion- parameter CLIP models still fail at detecting feature-based adversarially manipulated OOD images. The code is available at https: : http//ggttttttt//e/ http://UU--iiis/pp-ffiiiiia-a---
1 Introduction
Transferring the representations of pretrained vision models has improved the performance on a plethora of
image recognition tasks (Yosinski et al.] 2014 Tan et al. 2018 Park et al.) 2023 Adaloglou et al. 2023 To date, these models are trained with various
types supervision, which accelerates training convergence
compared to random initialization (He al.
2019
Examples include self-supervision (Chen et al. 2020b
natural language supervision (Radford et al.
2021
weakly-supervised learning (Mahajan al. 2018
standard supervised learning. Concurrently, Dosovitskiy et al. (2021) have established vision transformers
(ViTs), along with enormous number of variants (Liu et al.] 2021 Touvron et al. 2021 Beyer et al. 2023
as suitable architecture for training large-scale models in the visual domain (Degghani et al. 2023
Nevertheless, the applicability the learned features of pretrained models crucial and non-trivial, especially for unsupervised downstream tasks (Bommasani et al.] 2021 Adaloglou et al.) 2023). This work centers on adapting pretrained feature extractors for various visual OOD detection setups, focusing on contrastive language-image pretraining (CIIP) models.

---

Published in Transactions on Machine Learning Research (03/2024)
The task of OOD, novelty, or anomaly detection aims at identifying whether a given test sample is drawn from the in-distribulión (the training set) or an alternative distribution, known as the out-distribution. Accurate detection of anomalies is indispensable for real-world applications to ensure safety during deployment (Amodei ct al. 2016; Ren et al.) 2019). The detected unfamiliar samples can be processed separately, possibly with a human expert in the loop, rather than making a polentially uncalibrated prediction (Guo el. al. 2017 Despite significant advances in deep learning, neural networks tend to generate systematic errors for test examples far from the training set. (Nguyen et al., 2015) or assign higher likelihoods to OOD samples compared to in-distribution samples (Nalisnick et al. 2019).
Recent studies have established firm connection between the training data distribution accuracy and OOD generalization (Hendrycks et al.) 2021a Wenzel et. al.) 2022 Dehghani et al. 2023). A similar connection has been identified for supervised OOD detection when in- in-distribution training labels are available for training or fine-tuning. Supervised training leads to intermediate representations that likely form tight label-related clusters (Fort et al., 2021). An ideal representation for OOD detection should capture semantic propertics, such the pose and shape of an object, while remaining sensitive to the properties of the imaging process (e.g., lighting, resolution) (Winkens et al. 2020).
A suitable choice of visual feature representations is critical for detecting anomalics. However, learning informative representations for unsupervised OOD detection Taac et al., 2020 Sehwag et al. 2021 Rafiee
et al. 2022, where no in-distribution labels are available, is a challenging and active research area Coohnn et al,
2023 Unsupervised methods often adopt self-supervision to learn the in-distribution features by defining pretext tasks such as rotation prediction (Gidaris et al.) 2018). A major milestone in visual representation learning was reached by Chen et al. (2020b) with the development of contrastive and non-contrastive visual self-supervised methods (Caron et al. 2021). Recently, CLIP has enabled learning from vast amounts of raw text (Radford et al., 2021]. lasiun-languaee models offer the umprecedented advantage of zero-shot image classification by leveraging the label names of the downstream dataset. Nonetheless, adapting CLIP models is not straightforward, even in the supervised scenario. For instance, Phha et al. (2023) showed that finc-tuning CLIP models degrades their robustness against classilying distribution-shiilee samples.
Labeled OOD samples are typically unavailable in real-world applications, and the number of in-distribution samples is usually limited. Therefore, external data have been widely employed (Rafice et al. 2022) in two ways: a) outlier exposure where the external data is treated anomalous (Hendrycks et al. 2019a and b) using models pretrained on auxiliary data (Sun et al., 2022). Outlier exposure leads to performance gains only if the auxiliary data are sufficiently diverse and disjoint from the in-distribution (Hendrycks et al. 2019a
Liznerski et al., 2022). ). On the other hand, [endryyks et al. 2020 2021a) showed that pretrained backbones can enhance OOD detection performance and robustness without relying on dataset-specific shortcuts (Geirhos) et al. 2020). Consequently, pretrained models are suitable candidates for OOD detection, while Galil et al. (2023 has recently established CLIP models for OOD detection, especially when in-distribution label names are present [Ming et al.
2022
In parallel, several OOD detection methods still rely on similar small-scale benchmarks based on low-resolution images (Mohseni et al.] 2021 Rafiee et al., 2022 Esmaeilpour el al.) (022), such as CIFAR (Krizhevsky et al. 2009). Huang & Li (2021) argued that methods explicitly tuned for these benchmarks may not always translate effectively into larger-scalc and real-life applications. Towards this direction, new large-scale and
more challenging benchmarks have been introduced (Hendrycks et al.) 2021b Yang al.] 2022 Bitterwolf et al. 2023), which consider TmageNet (Deng et al.) 2009) as in-distribution. Finally, even though the robustness against adversarial attacks has been sufficiently explored in image classification (Szegedy et al.
2014 Goodfellow et al.] 2015 Mao et al.] 2023). less attention has been given to studying the construction of robust. visual OOD detectors Azizmalayeri et al. 2022 Yin et al., 2021]. ). Even though several ad vancemeetss
in visual feature extractors have been made and new large-scale OOD detection benchmarks (Bitterwolf et al,
2023) have been proposed, limited research has been conducted in OOD detection regarding the choice of
pretrained model and evaluation scheme, especially regarding CLIP models (Ming et al. 2022 Galil et al.
2023 Esmaeilpour et al. 2022).
In this paper, we present experimental study across 25 feature extractors and several visual OOD detection benchmarks. Using the existing publicly available models, we demonstrate that large-scale CLIP models are
2

---

Published in Transactions on Machine Learning Research (03/2024)
robust unsupervised OOD detectors and focus adapting the representations of CLIP under different OOD detection settings. Under this scope, examine several OOD detection setups based on the availability of labels or image captions (i.e. in-distribution class names). The core contributions of this work are summarized as follows:
To investigate whether the dependence between in-distribution accuracy and unsupervised OOD detection performance can be confirmed without in-distribution fine-tuning, we quantify the perfor- mance of 25 pretrained models across 4 benchmarks (Fig. 2 CLIP models exhibit the strongest positive correlation without fine-tuning across all benchmarks (R² coefficient > 0.92). Interest- ingly, the features of CLIP ViT-G outperform the ones from supervised ImageNet pretraining on ImageNet-based OOD detection.
To adapt the representations of CLIP for OOD detection, we propose simple and scalable method called pscudo-label probing (PLP). The text-based pscudo-labels are computed using its text encoder based on the maximum image-text feature similarity. We leverage the obtained Lext-based pseudo- labels to train a linear layer on top of CLIP, Without modifying the weights of CLIP or training separate /imaae encoders, PLP surpasses the previous state-of-the-art (Ming et al. 2022) on 5 ImageNet benchmarks by an average AUROC gain of 1.8% and 3.4% for CLIP VIT-H and CLIP ViT-G, respectively. Moreover, linear probing achieves superior performance on ImageNet-basol OOD benchmarks compared to fine-tuning, notably by a mean AUROC gain of 7.3% using CLIP ViT-H.
Finally, we show that a simple feature-based adversarial attack tailored for OOD detection can easily fool CLIP ViT-G trained on billions of samples (86.2% + 50.3% AUROC deterioration) by changes that are invisible to humans.
2 Related work
2.1 Supervised OOD detection methods
Supervised OOD detection methods rely on the fact that in-distribution classification accuracy positively correlated with OOD detection performance (Fort al.
2021 Galil et al., 2023). For that reason, many OOD detection methods derive anomaly scores from supervised in-distribution classifiers. Hendrycks & Gimpel (2017) developed the maximum softmax probability (MSP) as OOD detection score, which is frequently used, its temperature scaled version (Liang et al. 2018). More recently developed scores based the logits of the in-distribution classifier are the maximum logit (Hendrycks et al. 2022) or the negative energy scores by Liu et al. (2020). An alternative OOD detection score, which requires the in-distribution labels, is the parametric Mahalanobis-basse score (Lee et al., 2018). The Mahalanobis score assumes that the representations from each class are normally distributed around the per-class mean and conform to a mixture of Gaussians (Ren et al., 2021]] Fort et al.] (0021). Later on, [Sun et al.] (2022) established an important yet simple OOD detection score, namely the k-nearest neighbors (NN) distance, without requiring the feature
norms TTak et al. 2020) temperature tuning (Rafiee et al. 2022 The k-NN distance has the advantage
of being non-parametri and model- and disttibuttoon-agnotti..
Supervised learning may not always produce sufficiently informative features for identifying OOD samples Winkens et al. 2020). To this end, additional tasks have been proposed to enrich the supervised-learned fea- tures. Examples include determining the key in-distribution transformations and predicting them (Hendrycks)) et al. 2019b or contrastive learning. Mohseni et al. (2021) attempt first learn the domain-specifi transformations for each in-distribution using Bayesian optimization. [Zhang et al. (2020) present a two-branch framework, where generative flow-based model and supervised classifier are jointly trained.
2.2 Unsupervised OOD detection methods
Unsupervised OOD detection methods rely on learning in- in-distribution features, typically accomplished with contrastive or supervised rastive learning (Sehwag et al. 2021: Khosla et. al. 2020). Contrastive-based
3

---

Published in Transactions on Machine Learning Research (03/2024)
methods can be further enhanced by designing hand-crafted transformations that provide estimate of near OOD data (Rafiee et al. 2022) or by developing better OOD detection scores Tack et al. (2020). For example, TTack et al. (2020) add a transformation prediction objective (i.e. rotation prediction such as [09, 90", 1800, 1)0")). Schwag et al. (2021) define a simpler contrastive-bastt OOD detection method, where the Mahalanobis distance is computed in the feature space using the cluster centers of k-means (Lloyd 1982
2.3 OOD detection methods using external data or pretrained models
Early works attempted to external data to generate examples near the OOD decision boundary incorporate them for outlier exposure and negative sampling (Hendrycks et al. 2019a Rafice et al. 2022] Nonetheless, disjointness belween the in and out-distribution cannot be guaranteed, and applying shifting transformations depends on the in-distribution (Mohseni et al. 2021), which limits the applicability such approaches.
Hendrycks al. (2020) show that large-scale models pretrained on diverse external datasets can boost OOD detection performance. Recent OOD detection scores, which leverage pretrained models, deal with the large semantic space by, for instance, grouping images with similar concepts into small groups as in Huang &
Li (201). 1. The majority of existing methods focus on supervised OOD detection (Sun et al., 2022 Galil et al. 2023 Such approaches include fine-tuning the whole network or parts of it (Reiss et al. 2021
Contrarily, Ren et al. (2021) showed that a Mahalanobis-basse based score could achieve comparable OOD detection performance on srnall-scale benchmarks without fine-tuning.
Out of the limited label-free methods based on pre-trained models, a simple approach is the k-NN feature similarity, which has not been studied systematically across pro-trained models (Sun al. 2022). Another unsupervised approach aims at initially detecting an priori set of visual clusters. The obtained clusters are subsequently used as pseudo-labels for fine-tuning (Cohen et al. 2023). Apart from unsupervised OOD detection, CLIP models can additionally leverage in-distribution class names (Esmacilpour et al.) 2022 ,
and in- nn-disrribution prototypes can be obtained using the textual encoder, which has not been thoroughly investigated. Esmaeilpour et al. (202)) extend the CL.IP framework by training a text-based generator on top, while Ming et al. [2022] compute the maximum softmax probability of image-text feature similarities. Finally, [Wang et al. 2023) hold the current large-scale OOD state-of-the-art using CLIP by training separate text encoder on 3 million auxiliary text-image pairs. Still, it is unknown whether this approach scales on larger-scale CLIP models. Approaches that work with off-the-shelffeetuure
without requiring external
image-lext data (Wang el al. 2023), ; fine-tuning, or training of separate encoders (Wang et al. 2023 remain
relatively unexplored, especially in conjunction with CLIP models Wortsman et al. 20222)
2.4 OOD detection robustness
Hendrycks & Dietterich (219) analyzed the robustness under corruptions and geometric perturbations, such Gaussian noise and brightness shift. Since it is not always clear which manually perturbed images are present in the in-distribution (and which are not), attention has been given to adversarial robustness (Chen et al. 2020ab. Existing works have primarily focused on fooling supervised OOD detection methods on small-scale benchmarks {Yin et al.) 2021. Even though the zero-shot classification performance of CLIP deteriorates significantly when the inpul images are constructed adversarially (Mao el al.) (202)). their adversarial OOD detection robustness has not been thoroughly investigated. Finally, even though adversarial Luo et al. 2022 they have not been applied
feature- based methods exist (Guo et al. 2018 Heng al.] 2018 to OOD detection.
3
The proposed OOD detection setup
3.1 Considered pretrained models
Several supervised CNNs, such as ResNet50 (He et al. 2016). ConvNext (Liu et al. 2022) and VIT (Dosovitskiy
ctt al. 2021) models trained on ImageNet and ImageNet-21K (Dnng et al.] 2009 Russakovsky al. 2015)
were utilized. Regarding Imagenet--redraineed self-supervised models, the DINO (Caron al. 2021 MoCov3
4

---

Published in Transactions on Machine Learning Research (03/2024)
Dataset
Classes
Train images
Validation images
Architecture
Images/ / second
Number of Params (M)
Pretraining datasets
ImageNet
1K
1.28M
ImageNet-21K
21K
14M
ResNet50
2719
24
OpenAI-400M
400M
ConvNext-S
1576
49
LATON-2B
2B
ConvNext-BB
1157
88
In- distiittbbtto datasets
ConvNext-L
695
196
CIFAR10
10
50K
10K
ConvNext-B wide
888
88
CIFAR100
100
50K
10K
ConvNext-LL deep
531
200
ImageNet
1K
1.28M
50K
Next-XXX
180
847
Out-distribuleen datasets
ViT-S/16
3210
22
iNaturalist.
110
10K
ViT-B/16
1382
86
SUN
50
10K
VIT-L/16
475
303
Places
50
10K
V1T-H/14
183
631
IN-O
200
2K
VIT-G/14
78
1843
NINCO
64
5.88K
Texture
47
5.54K
Table 1: Number of parameters (in mil- lions) and inference time (images per second on a single GPU) for the uti- lized network architectures.
CIFARIO-A
10
1000
CIFARIO-AS
10
1000
Table 2: An overview the number of classes and the number of samples on the considered datasets.
(Chen et al. 2021). and MSN (Assran et al. 2022) were selected. Finally, CLIP-based models were either
trained on AT-0AM (Radford et al. 2021 or LAION-2B (Schuhmann al. 2022). which consists of 400M and 2 billion image-text description pairs, respectively. Further information regarding the net work
complexities is reported in Tab. [ To quantify the complexity, performed inference single GPU with batch size of 256 to compute the images per second that were processed at 224 x 224 resolution.
3.2 Datasets and metrics
We denote the in-distribution Din, and the out-distribution Dout The corresponding train and test splits are indicated with superscript. To design fair comparison between vision and vision-language models, we define unsupereised 0OD delection without having access to the set of Di, label names. Following [Huang & (021), we use ImageNet as Dm for large-scale OOD detection benchmarks and CIFAR for small-scale benchmarks.
For the large-scale benchmarks, we use the following OOD datasets: Imagenet-O (TN-O) (Hendrycks et al.
2021bb, a flower-basod subset of iNauralist (Van Horn et al., 2018). Texture (Cimpoi et al. [014)) subsct of the SUN scene database (Xiao et al. (0100), subset of Places (Zhou et. al. 2017]. IN-O contains 2K samples from ImageNet-2111 21K, excluding ImageNet. It is worth noting that Places, Textures, and IN-O have class overlap of 59.5%, 25.6%, and 20.5% with ImageNet, according to Bitterwolf et al. (2023) Therefore, we
additionally use the newly proposed NINCO Bitterwolf et al. (2023), which contains 5879 samples belonging to 64 classes with zero class overlap with ImageNet's classes. Datasel information is summarized in Tab. 2
To quantify the OOD detection performance, the area under the receiver operating characteristic curve (AUROC) and the false positive rate 95% recall (FRP95) are computed between Drest test and Diest Below, present. the used OOD detection scores, given a pretrained backbone model g.
5

---

Published in Transactions on Machine Learning Research (03/2024)
1-NN. For the unsupervised evaluations, the maximum of the cosine similarity (Sun al 2022)
between a test. image x' and 2; Dirain = (21, [N as an OOD score:
ANN(2') max
(1)
where sim(.) the cosine similarity and N the number of Dirain samples
Mahalanobis distance (MD). The MD be either applied directly the feature space of the pre-trained model, g(:)) or on the trained linear head, labels Yı E ........ JN are available. We denote the class index C (I...., C}, with C being the number of
h(g(x))). However, MD assumes that the in-distribution
Din classes and N₂ the number of samples in class c. For each class c, we fit a Gaussian distribution to the representations 2 Lee et al. 2018). Specifically, we first compute the per-class mean Hec = N. Eing and a shared covariance matrix
C Σ Σ
E
- (c- - Mc)
(2)
C 1::=c
The Mahalanobis score is then computed for each test sample as
MD+(2') (z'- 1c) -He) (MD(2') - min (z).
He)
(3)
(4)
MD can also be applied with cluster-wise means, for instance, using the k-means cluster centers computed on the feature space of g (Sehwag et al. 2021). We denote this score as k-means MD and use the number of ground truth Di classes to compute the cluster centers.
Relative Mahalanobis distance (RMD). Given the in-distribution mean po
N Ex 24, we additionally compute Σg V Σ (z+ (2) Po) to compute MDs analogously to Eq. 3 Subsequently, the RMD
score Ren el al. 2021 can be defined as
(R')(#') -
min (minnMM.(2')) -MDo(z)).
55)
Energy. As in Liu et al. (2020). negative free energy is computed over the logits 2 of an Dn, classifier as
Senergy Senergy(x') (x))= T. log
A/T
(6)
where T is a temperature hyperparameter.
3.3 Pseudo-Label Probing (PLP) using CLIP's textual encoder
The pseudo-MSP baseline. Ming et al. 2022 proposed simple visual OOD detection method to leverage the text encoder of CLIP by feeding the Dnn class names in parallel with computing the image representations using the image encoder of CLIP. set of captions such as "a photo of {label}" is used to produce the text representations. The text-based representations are averaged after L.2 normalization. Subsequently, L2 normalized text-image cosine similaritics for cach test image are computed. The maximum softmax probability (MSP) used MSP the logits of supervised Dm classifier. We use MSe as baseline.
OOD score for lext-image similarities. This conceptually similar to using
PLP. We propose simple method called pseudo-label probing (PLP) that extends pscudo-MSP. After computing the image-text similarity, we derive a pseudo-label for each Dérain image using argmaz, the class index that corresponds to the maximum image-text similarity. To enhance the predictions of CLIP to assign high probabilities Dirain a linear layer then trained with the obtained pscudo-labels on the features of g. During the OOD detection evaluation stage, the RMD or energy score used, namely PLP + RMD and PLP Energy-

---

Published in Transactions on Machine Learning Research (03/2024)
(a)
(b)
(c)
(d)
(e)
(f)
Figure I: Generating an adversarial OOD example of a deer that is close enough in the feature space to an in-distribution image of an apple. From left to right: a) the original OOD image from the CIFAR10 test set, b) the adversarial example without smoothing, c) the adversarial example with smoothing, d) the randomly sampled in-distribution larget image from CIPAR100, e) the per-pixel Euclidean distance between the original image and perturbed image, and f) the distance between the original and the smoothly perturbed image.
Similar to pseudo-MSP, PLP remains OOD data-agnostic and assumes no prior information on D Dout. PLP is conceptually similar to self-training Xie et al. (202))_ Differently from standard self-training, we only train a linear layer using the text- based prototype that closest to each Din image in feature space derived from the textual encoder of CLIP. Finally, while both pscudo-MSP and PLP do not modify the paramcters of CLIP or train external image/ /ext encoders psua el. al. (2022 Wang et al. (2023), , we show that PLP scales better with larger-scale models (Section while training linear layer adds minimal overhead compared to pseudo-MSP.
We additionally consider supervised linear probing as an alternative to the existing paradigm of fine-tuning visual feature extractors. Linear probing refers to training a linear head on the features of the backbone g, using the Disai labels and acts as an upper bound for PLP. Subsequently, RMD or Energy is computed. This is in contrast to existing approaches (Fort et al.) 2021 Huann & Lil (021)) that typically fine-tune the visual backbone, which is significantly slower, computationally more expensive, and may lead to catastrophic forgetting [Kemker et al. 2018
3.4 Feature-based adversarial OOD data manipulation
The unsupervised OOD detection performance of CLIP VIT-G, combined with its known classification
robustness against natural distribution shifts, raises the question ion of whether these models are also adversarially robust OOD detectors. To expose the vulnerability of CLIP as an OOD detector, we develop a feature-based
adversarial data manipulation tailored for OOD detection similar to Guo et al. (2018): Heng et al. (2018) Luo
et al. 2022). We solely aim to test adversarial OOD detection; thus, we do not compare with methods that can be modified for OOD detection.
For test image x' Diest we randomly choose an in-distribution image x E Ditran as the target. Instead of maximizing the OOD scores directly (Azizmalayeri et al.) 2022; Chen et al. 2020a), create an adversarial perturbation P with the same dimensions as x' that maximizes the cosine similarity between the in-distribution feature g(x) and g(z' + p). We use the Adam optimizer to compute by minimizing sim(((r)) + p)), starting with Gaussian noise p ~ (0, 10-)) and clipping x' + to the pixel range [0, 1] after every update step, similar to Yin et al. (2021). J. We emphasize that we do not directly restrict the perturbatinn size and only limit the number of steps, as opposed to Yin et al. (2021).
We experimentally observe that in the case of ViTs, the perturbations are quite visible along the edges of the transformer patches (Fig, D. To create more natural appearing adversarial examples, we enforce the smoothness of the perturbation by regularizing the allowed perturbation difference between neighboring pixels. We compute the image gradient 0p/Oh and 0p/ in the horizontal and vertical direction, respectively. The image gradients have the same shape as the image, x H x W, and we define the regularization term as
0p 2
Op 2
L (p)
3HW
ah
Dw
Ejk
ijk
ijk
7

---

Published in Transactions on Machine Learning Research (03/2024)
OOD Dataset
Method
INaturalist Plants
SUN
IN-O
Texture
NINCO
FPR95
AUROCT
FPR951
AUROC
FPR95
AUROCT
FPR95
AUROCT
FPR951 AUROCT
CLIP VIT-B
Ming 2022
30.91
94.61
37.59
92.57
57.77
86,11
Pseudo-MSI
52.10
89.55
40.34
91.92
78.20
79.47
55.28
86.82
83.09
72. 44
PLP + Energy
47.70
93.36
G8.15
88.69
79.00
80.72
54.63
86.44
81.93
75.65
CLIP VIT-L
Ming et al. 2022 28,33 28
94.95
29.00
94.14
59.88
84.88
Pseudo-MS MS PLP + Energy
48.37
90.98
29.85
93.91
72.15
81.49
57.92
85,42
74.72
78. 47
27.24
95.40
44.99
91.05
64.70
85.81
46.75
88.22
71.95
83.83
CLIP VIT-H (LAION-2B)
Pseade-MSPt PLP + Energy
60.35
90.45
42.92
91.50
59.65
87.00
46.95
89.67
74.24
82,99
48.61
93.02
40.90
91.78
52.50
89.24
35.00
91.96
71.9
84.88
CLIP VIT-G (LATON-2B)
Pseudo-MSPt 56.49
90.55
38, 69
92.75
55.50
88.11
46.95
89.70
71:30
83.38
PLP | Energy
23.51
96.01
34.41
92.90
42.80
91.39
30.38
92.74
59.32
88.65
Table 3: OOD detection performance metrics using CLIP models and label names for ImageNet-11K in-distribution. We report the best metric in bold across different ViTs. The symbol indicates our reproduction of Ming
2022
where i, ls run over image dimensions. We then minimize the loss
badv sim(g(x) g(x + p)) + Nemootth
(8)
with respect to the pertubation P, where À is hyperparameter. During the evaluation, remove the chosen target image, from Dirain to show that the adversarial example, x' + p, cannot be detected OOD from the remaining in-distribution examples. As a proof of concept, we create two adversarial OOD datasels for the CIFAR100 - CIFAR10 benchmark, namely CIFARIO-A (A = 0) and its smoothened version CIFARIO-AS (A > 0). The generation of an adversarial example is shown Fig. More adversarial examples can be found in the supplementary material.
3.5 Experimental evaluations
First, we benchmark 25 publicly available pretrained models on common ImageNNet and CTFAR OOD detection benchmarks, illustrated in Fig. Second, we compare PLP against the aforementioned baselines on ImageNet benchmarks in Tab. For fair comparison with Ming et al. (2022),
the CLIP ViT-B and ViT-L trained on OpenAI-400M Radford et al. (2021), but also report results with the recently released CLIP VIT-G trained on LATON-2B (Cherti et al. 2022). In contrast to Huang & Li (2021), we remove the Places OOD subset due to its high class overlap of 59.5% with ImageNet (Bitterwolf et al. 2023); instead, we add IN-O and NINCO. Third, in Tab. ! we compare linear probing on CLIP's visual representations to standard supervised fine-tuning for OOD detection. Fourth, Tab. [] we conduct further OOD detection evaluations with CLIP ViT-G, basod on the availability of Din class names or (few-shot) labeled images and different detection scores. In Tab. 6 compare linear probing to fine-tuning for ImageNet-211K pretrained models. Finally, we study the robustness against the adversarially created OOD samples (CIFAR10-A, CIFAR10-AS) using CIFAR100 as Di..
3.6 Implementation details
Since probing and PLP only train a linear layer precomputed representations, it is more scalable and significantly faster than fine-tuning while having minimal overhead compared to pscudo-MSP. We used the Adam optimizer (Kingma & Ba, 2014) with a mini-batch size of 256 for CIFAR10 and CIFAR100 and 8192 for ImageNet and trained for 100 epochs with a weight decay of 10-3 The learning rate is set to 10-3, (mini-batch /256 with a linear warm-up over the first ten epochs and cosine decay after that. All
https://ddriie www.iieeeeoooo........ http::/wwwt..ooooooeeeessssssstr

---

Published in Transactions on Machine Learning Research (03/2024)
OOD Dataset
Method
INsturalist Plants
SUN
Places
Texture
NINCO
FPR95)
AUROCT
FPR951
AUROCT
FPR051
AUROCT
PPR95
AUROCT
FPR95,
AUROCT
CLIP
VIT-B
Finetune Energy
83.95
60.87
64.39
61.16
73.18
67.57
70.36
Finetune MSP
36.80
90.49
60.53
63.29
80.54
82.60
80.94
Probing Energy
95.72
50.30
50.26
88.60
50.97
88.10
79.23
CLIP
VIT-L
Finetune Energy
30.35
88.75
46.75
79.42
54.60
74.11
53.03
76.37
55.70
77:70
Finetune MSP
31.33
91.90
31.33
84.97
55.68
82.90
48.68
84.52
56.79
84.27
Probing Energy
8.65
97.89
41.42
91.91
43.05
90.81
44.14
90.27
67.05
86.32
CLIP ViT-II (LAION-2D)
Finetune Euergy
27.99
90.07
39.77
83.27
49.66
77.84
42.37
81.52
51.62
78.18
Finetune MSP
26.70
92.90
19.19
85.69
54.51
83.28
46.66
84.70
55.48
83.90
Probing Energy
98.28
32.42
38.89
26.45
93.93
55.39
CLIP VIT-G (LAION-2B)
Probing Energy
6.29
98.27
32.21
98.27
37.16
92.02
24.63
94.31
49.35
90.4
Table 4: Supervised OOD detection performance on ImageNet-11K 1K using CLIP: linear probing outper- forms fine-tuning on average, especially for larger-scale models, while being computationally cheaper and significantly faster to train. The best method for each model architecture is bolded, excluding CLIP ViT-G. The reported results from fine-tuning are based on our evaluations using publicly available CLIP weights fine-tueed ImageNet-1K from timm Wightman 2019).
the experiments were carried out in a single NVIDIA A100 with 40GB VRAM. Moreover, emphasize that the standard deviation of probing and PLP is less than 0.01%, measured over 10 independent runs. To create the adversarial datascts CIFAR10-A and CIFAR10-AS, we perform 250 stops with the Adam optimizer with a learning rale of 10-3 on 1K OOD images. We set to 10 when applying smoothing (Eq. S].
4 Experimental results
Unsupervised OOD detection. In Fig. initially investigate whether there is connection between the Diest classification accuracy and unsupervised OOD detection AUROC by benchmarking 25 fcature extractors. Out of them, CLIP models exhibit the strongest. correlation (R²² coelficient 2 0.92) independent of their network architecture (i.e. ConvNext, ViTs, etc). CLIP's best instances (ViT-G, ConvNext-XXI XXLL are currently the best-performing unsupervised OOD detectors, aligning with recent results from Galil et al. (2023). The features CLIP ViT-G even outperform the ones from supervised training on ImageNet as Din- Moreover, we observe that when the Ppretrain different from Dn (CIFAR benchmarks), a positive correlation can still be identified for self-supervised and supervised models pretrained on ImageNet (R² 0.82), yet not strong as CLIP (R² = 0.95).
Large-scale OOD detection with access to class names. We report absolute gains and AUROC scores. In Tab. 3 we compare pseudo-MSI based on our reproduction of Ming et al. (2022), to PLP on all five large-scale benchmarks and find an average improvement of 2.81%, 1.85%, 3.44% AUROC for CLIP VIT-I,, ViT-H, and VIT-G, respectively. The reported gains are computed using the energy soore, which is slightly superior to RMD on average and significantly faster compute. Interestingly, PLP has consistent improvements on the newly proposed NINCO dataset, wherein all samples have been visually verified to be semantically different from Dmn with average AUROC gain of 3.93% across CLIP models. We highlight that the discrepancy between our reproduction and the reported results of Ming et al. (2022) using ViT-B and VIT-L is not due to different CLIP weights. We provide a detailed analysis of our reproduction in the supplementary and show that. we were not able to get the same results with Ming et al. (2022).
Large-scale supervised OOD detection. As shown in Tab. fine-tuning CLIP models is often unncoossary for supervised OOD detection. More precisely, probing on larger models such as CLIP ViT-H consistently outperforms fine-tuning by large margin of 7.34% average. The performance discrepancy of CLIP VIT-I.
9

---

Published in Transactions on Machine Learning Research (03/2024)
Di, Dogt ImageNet Image Vet O
2n Dad CIEARTO CIEARTO
90
A
80-
AUROC (96)
AUROC (96)
75
70
70
*
20
70
80
I:-NN accuracy (%)
I-NN socuracy (96)
Des Dan: ImageNet / NINCO
Dis Dasi CIFAR10 / CIFAR100
90
100
85
AOROC (36)
80
AUROC
75
70
75 I-NN aceuracy (%)
100
k-NN accuracy (%)
CLIP LAION-2B
Supervised IN
MSN IN
4 DINO IN
VIT
CLIP OpenA1-400M
Supervised IN21K
MoCoV3 IN
ReNNet
CouxNeXi
Figure 2:
In-distribution test accuracy using k=20 nearest neighbours (k-NN) (x-axis)
versus unsupervised out-of-distribbutio (OOD) detection score (AUROC %) (y-axis) for DaaeNeett-mmaeeet-O..
SAAARIOO~CIFARIO imageNet-NNINCE and CIFARIO-CCIARRII CCIFAR10
Deng et al. (2009): Hondrycks et al. (2021b;; Krizhevsky et al. (2009). CLIP models Radford et al. (021) |
Cherti et al. (2022) exlibit. a strong dependence between OOD performance and in-distribution test accuracy
even without fine-tuning. The black dotted line fitted using the CLIP models (starred datapoints), where a
cocfficient R² > 0.92 for all benchmarks was found. The OOD detection score is computed using the top-1 NN cosine similarity. Different colors are utilized for different architectures (ViT Dosovitskiy et al. (2021) ConvNeXt Liu et al. (022]), ResNet He et al. (2016) while symbol sizes roughly indicate architecture size (i.e. Small, Base, Large, Huge, Giga). IN indicates ImageNet and IN21K indicates ImageNet-211 21K Russakovsky
ct al. 2015). Best viewed color.
10

---

Published in Transactions on Machine Learning Research (03/2024)
Din labels/ names
Din:CIFARION CIFARIO
CIFAR10 CIFAR100
ImageNet IN-O
ImageNNtt
CLIP ViT-G/14
Dout: :IF:RIIAA
NINCO
k-means MD
X
X
72.8
89.5
87.6
80.7
1-NN
X
X
87.6
98.2
88.0
84.0
Pscudo-MSP
X
V
94.2
97.3
88.1
83.4
PLP + MSP
X
V
92.7
97.9
86.6
86.7
PLP + RMD
X
97.1
98.3
91.9
88.4
PLP + Energy
95.0
98.5
91.4
88.7
MD
73.1
91.1
88.1
81.5
RMD
96.3
98.8
92.4
89.3
Few-shot p 10 MSP
89.4
96.5
88.1
84.8
Few-shot p 10 + Energy
90.9
96.6
90.8
83.0
Probing + MSP
94.1
98.7
88.1
89.1
Probing + RMD
97.3
98.8
92.5
89.5
Probing + Energy
96.3
99.1
92.9
90.4
Table OOD detection AUROCs (%) for multiple evaluations and scores using CLIP VIT-G/14
trained on LAION-2B.
on Places and SUN compared to the other benchmarks is partially attributed to the class overlap. By contrast, INaturalist, Texture, and NINCO are sufficiently disjoint from ImageNet. terms of Din class overlap.
Adversarial OOD detection robustness. By evaluating CLIP VIT-G on the introduced CIFARI00+CIFARDDO benchmark, found that it is possible to drop the AUROC score from 87.6% -50.3% using 1-NN and degrades performance to 55.8% and 51.9% AUROC on CIPARIO0-CIFARIO- respectively. Note that an AUROC score of 50% is a random guess's 's score, meaning that CLIP ViT-G performs slightly better than a random guess.
using pscudo-MSP. Introducing the smoothness restriction (Eq. 8
using 1-NN and pseudo-MSP
Ablation study for CLIP VIT-G for all OOD detection setups. Tab. we conduct additional experimental evaluations using CLIP ViT-G for all three OOD detection scenarios (unsupervised, class names are available, supervised). In the unsupervised case, k-means MD yields an inferior AUROC compared to 1-NN, preciscly lower by 6.8% on average. By incorporating the Din class names using CLIP's text encoder, we find that the PLP consistently outperforms pseudo-MSP with mean improvement of 3.16% and 2.63% over PLP+RMD and PLP+Energy respectively. In the supervised scenario, we highlight that RMD is a strong baseline, surpassing MD by 10.75% AUROC on average, while our PLP+Energy marginally improves RMD by 0.47%. MSP is constantly the worst choice as an OOD detection score after linear probing.
5 Discussion
Pixel-related features in CLIP's learned representations. Similar to Mao et al. (2023) for image classification, the OOD detection performance close to a random guess demonstrates that even the top- performing CT.IP models trained on billion-scale image-text pairs can be fooled by visual signal manipulation that invisible to humans. This finding gives further evidence that besides label-rclated features, (local) pixel information a affects the learned representations and needs to be explored in greater depth (Park al. 2023 Dravid et al., 2023). Another possible research avenue is how adversarial OOD samples transfer between different feature extractors, which is left for future work.
Does PLP improve Diest accuracy? By comparing the Diest accuracy on ImageNet of the trained head with the pseudo-labels versus the zero-shot classification of CLIP VIT-G, we found a minor accuracy improvement of 0.5% on ImageNet. However, accuracies deteriorate for CLIP ViT-B and CLIP ViT-L on average, which suggests that their Di, text-based pseudo-labels have less true positives. The fact that PT.P improves the OOD detection performance (compared to pseudo-MSP) without significantly increasing
11

---

Published in Transactions on Machine Learning Research (03/2024)
Finetuned on Dn
Di=:NNK Dout :NICO
IN1K
CIFAR100 CIFARIO
CIFARIO CIFAR100
IN-O
ConyNext-B
X
95.4
94.9
93.0
98.0
ConvNext-B
V
88.8
85.8
90.8*
96.9*
ViT-I
X
95.2
95.3
89.8
94.3
VIT-L
V
91.3
92.1
97.9*
98.5*
VesNet50+ViT-B
X
95.9
95.8
89.2
96.6
RosNet50+ViT-B
V
92.9
92.1
96.2*
98.5
Table 6: Comparing the OOD detection AUROC (%) of ImageNet-21K pretrained models: linear probing versus fino-tuning. We use publicly available models with and without fine-tuning on ImageNet from limm and evaluate them using the energy score. Following |oort al. (2021), we use the MD CIEAR fine-tuned models (indicated with asterisk), which is inefficient at the scale of ImageNet.
Distt accuracy is a new direction left for future work. Another interesting avenue whether the text-based pseudo-labels from CLIP can be used with other, possibly smaller-scale, visual feature extractors.
Does linear probing lead to similar OOD detection performance compared to fine-tuning for compare linear probing versus fine-tuning for publicly
ImageNeet211K pretrained models? In Tab. available models, where there is a corresponding fine-tuned model on ImageNet. Even though fine-tuning is the standard practice in supervised OOD detection Fort et al. (2021), the obtained results indicate that it performs inferior to linear probing on the ImageNet OOD-related benchmarks even for ImageNet-211 pretrained models. Fine-tuning is, however, still the best approach on the small-scale OOD benchmarks (e.g. CIFAR1O - CIFAR100), given ImageNat21K pretraining. This explains why a simple and scalable approach such as probing may have been overlooked. As suggested by Huang & Li (2021), we confirm that OOD detection approaches tested on small-scale benchmarks do not always translate effectively into larger-scale setups, where simpler methods need to be revisited. Combined with the results from Tab. || we validate that the OOD-related information is readily available on large-scale foundational models, as recently stated in Tichwhich et al. (2023): Oquab et al. (2023)
Is the choice of the best feature extractor consistent for smaller-sized models? Not in the scale of ViT-B and CIFAR benchmarks. In Fig. 3 we keep the model architecture fixed (ViT-B) and visualize the unsupervised OOD detection performance (1-NN) for different datasets (ImageNet, TmageNet21KK, OpenAT- 400M, LAION-2B) and pretraining types (CLIP, supervised, self-supervised). The rationale behind using the CIFAR-based benchmarks is to focus on feature transferability. We highlight that large pretraining datasets such as LAION-2B are bottlenecked by the model size of ViT-B as explained in Cherti et al. (0022). Unlike Fig. 2 we show that the ranking of backbones is not always consistent between different Di, and Dont at this scale. Finally, DINO ViT-B surpasses supervised ImageNet pretraining on both benchmarks, outlining the transferability of features of self-supervised methods, which is consistent with the results of Ericsson al.
2021
Naseer et al. 2021); Adaloglou al. (2023).
6
Conclusion
This work presented a thorough experimental study by leveraging pretrained models for visual OOD detection, focusing on CLIP. It was demonstrated that CLIP models are powerful unsupervised OOD detectors, outperforming even Di supervised models large-scale OOD detection benchmarks. A fast, simple, OOD data- data-agnostic, and scalable method called PLP that trains a linear layer based on Dmn text-based pseudo-labels was introduced. PLP outperformed the previous statc-of-the-art (pscudo-MSP) for most CLIP models and ImageNee-baased benchmarks while substantially improving the larger architectures (i.e. ViT-G achieves a mean AUROC gain of 3.4%). Furthermore, was demonstrated that probing can replace the costly finc-tuning on CLIP and even ImageNett221K models on largo-scale benchmarks, where CLIP ViT-H exhibitod an average AUROC improvement of 7. 7.3%. Finally, a simple feature-based adversarial OOD data manipulation method showed that billion-scale feature extractors (CLIP ViT-G) vulnerable to adversarial OOD attacks
12

---

Published in Transactions on Machine Learning Research (03/2024)
Dio / Dmnt CIFAR100 /
CIFARIO
Din / Den: CIFARIO / CIFAR100
human
95
90
AUROC (%)
AUROC (%)
80
70
75
60
50
MAE-IN
Sup-IN21K
McCoV3-IN
MSN-IN
OpenAL-100M
Sup-IN
DINO-IN
LAION-2B
65
MAE-IN
Sup-IN2IK
OpenAL-400M
Sup-IN
IN
MSN-IN
LAION-2B
McCoV3-IN
DINO-N
Figure AUROC values for unsupervised OOD detection using ViT-B/16 pretrained on different datasets (IN, IN-21K, ApenAA-,00M, LAION-2B) and pretext tasks. IN indicates ImageNet. The performance of CLIP on LAION-2B is bottlenecked by the model size reported in Cherti et al. [2022 and larger networks are needed when scaling to billion-scale datasets. The horizontal line indicates human-level performance, as reported in Fort et al. (2021).
7 Acknowledgements
We would like to thank Georgios Zoumpourlis and Kaspar Senft for their insightful remarks and discussions the manuscript.
13

---

Published in Transactions on Machine Learning Research (03/2024)
References
Nikolas Adaloglon, Felix Michels, Hamza Kalisch, and Markus Kollmann. Exploring the limits of deep image clustering using pretrained models. arXiv preprint Xix::2303. 17896, 2023.
Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Concrete problems in ai safety. arXiv preprint 160i 06565, 2016.
Mahmoud Assran, Mathilde Caron, Ishan Misra, Piotr Bojanowski, Florian Bordes, Pascal Vincent, Armand Joulin, Mike Rabbat, and Nicolas Ballas. Masked siamese networks for label-efficient learning. In Shai Avidan, Gabriel J. Brostow, Moustapha Cissé, Giovanni Maria Farinclla, and Tal Hassner (eds.), Computer Vision ECCV 2022 17th European Conference, Tel Aviu, Isruel, Oclober 23-27, 2022, Proceedings, Part XXXI, volume 13691 of Lecture Notes in Computer Science, pp. 456 473. Springer, 2022. doi: 10.1007 99000/99778--01-12212-222 26. URL hattps://oi..or 00t//0...///...... 1-0077798-33001-11101 -__26
Mohammad Azizmalayeri, Arshia Soltani Moakar, Arman Zarei, Reihaneh Zohrabi, Mohammad Taghi Manzuri, and Mohammad Hossein Rohban. Your aut-of-distributin detection method is not robust! In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information
Processing Systems, 2022. URL https: http::::///eeeeiieeeeee http:///ppperreeeeeee////o////nn wttp://wwwe..ooooo////////////1.
Lucas Beyer, Pavel Izmailov, Alexander Kolesnikov, Mathilde Caron, Simon Kornblith, Xiaohua Zhai, Matthias Minderer, Michael Tschannen, Tbrahim Alabdulmohsin, and Filip Pavetic. Flexivit: One model for all patch sizes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14496-14506, 2023.
Julian Bitterwolf, Maximilian Mueller, and Matthias Hein. In or out? fixing imagenet out-ff-diittiiibtiion detection evaluation. In ICML, 2023. URL https://proceeddiggs mlr preess//200/bittteewoorr2.. htt.://ccccaao......rrr////////a
html
Rishi Bommasani, Drew Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models. arxiv preprint Sti::2008.0255 2021.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. 9650-9660, 2021.
Jiefeng Chen, Yixuan Li, Xi Wu, Yingyu Liang, and Somesh Jha. Robust out-of-disttibbtiio detection for neural networks. arXie preprint orXiv:2003.0977 2020a.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. simple framework for contrastive learning of visual representations. In International conference machine learning, pp. 1597-1607. PMLR, 2020b.
Xinlci Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. In Proceedings of the TEEE/CVE International Conference on Computer Vision, pp. 9640-96499, 2021.
Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ihharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-image learning. arXiv preprint arXio:2212. 2200:2212007143, 2022.
Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3606-3613, 2014.
Niv Cohen, Ron Abutbul, and Yodid Hoshen. Ont-of-distribution detection without class labels. In Computer Vision-ECCCV 2022 Workshops: Tel Aviv Israel, October 23-27, 2022, Proceedings, Part II, pp. 101-117. Springer, 2023.
14

---

Published in Transactions on Machine Learning Research (03/2024)
Mostafa Dehghani, Josip Djolonga, Basil Mustafa, Piotr Padlewski, Jonathan Heek, Justin Gilmer, Andreas Steiner, Mathilde Caron, Robert Geirhos, Ibrahim Alabdulmohsin, et al. Scaling vision transformers to 22 billion parameters. arXiv preprint arXiv:2302.05. 05442, 2023.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image databasc. In 2009 IEEE conference computer vision and pottern recognition, pp. 248-255 Iece, 2009.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image worth 16x16 words: Transformers for image recognition at scale. In International Conference Learning Representations, 2021. URL batt:s//wgpaarrrreoeeeeee/rrrrra
Amil Dravid, Yossi Gandelsman, Alexei A Efros, and Assaf Shocher. Rosetta neurons: Mining the common units in a model zoo. In Proceedings of the IEEE/CVE /CF International Conference on Computer Vision, pp. 1934-1943, 2023.
Linus Fricsson, Henry Gouk, and Timothy M Hospedales. How well do self-supervised models transfer? Tn Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5414 5423, 2021.
Sepideh Esmaeilpour, Bing Liu, Eric Robertson, and Lei Shu. Zero-shot out-of-distributiio detection based on the pre-trained model clip. In Proceedings of the AAAI conference artificial intelligence, volume 36, 2022.
pp.
Stanislav Fort, Jie Ren, and Balaji Takshminarayanan. Exploring the limits of out-of-distribution detection. Advances in Neural Information Processing Systems, 34:7068 7081, 2021.
Ido Galil, Mohammed Dabbah, and Ran El-ianiv. A framework for benchmarking clas-out-of--dssribbtte detection and its application to imagenet. In The Eleventh International Conference on Learning Repre- sentations, 2023. URL https:://penreviiee met//forrumidd11uubbbs.....
Robert Geirhos, Jörn- Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2(11): 665-673, 2020.
Spyros Gidaris, Pravoor Singh, and Nikos Komodakis. Unsupervised representation learning by prodicting image rotations. In International Conference on Learning Representations, 2018. URL httpss://peenrevier met/forrmmid--Siv4400000
lan Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Representations, ICLR 2015, Son Diego CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL http: Tlarxiv.org/bbse 0tt///112.667.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International conference on machine learning, pp. 1321-1330. PMLR, 2017.
Chuan Guo, Jared S Frank, and Kilian Q Weinberger. Low frequency adversarial perturbation. arxie preprint arXiv:1809. 100.00889.877, 2018.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Kaiming He, Ross Girshick, and Piotr Dollár. Rethinking imagenct pro-training, 2019.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness common corruptions and perturbations, In International Conference on Learning Representations, 2019. URL https: https://openrevien
15

---

Published in Transactions on Machine Learning Research (03/2024)
Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of-distribution examples in neural networks. In International Conference on Learning Represenlations, 2017. URL https:// openreview.net/Forunne http://oorrriddii///11.....
Dan Hendrycks, Mantas Mazeika, and Thomas Dietterich. Deep anomaly detection with outlier exposure. In International Conference on Learning Representations, 2019a.
Dan Hendrycks, Mantas Mazeika, Saurav Kadavath, and Dawn Song. Using self-supervised learning can improve model robustness and uncertainty. Advances in neural information processing systems, 32, 2019b.
Dan Hendrycks, Xiaoyuan Liu, Eric Wallace, Adam Dziedzic, Rishabh Krishnan, and Dawn Song. Pretrained transformers improve out-of-distribttion robustness. In ACI, pp. 2744-22 2251,, 2020. URL httph:://// //doi hrg/10.18653/vv
acl-main........mam..m maman.244
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: A critical analysis of out-o--distribution generalization. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 8340-834 -300,,84,440 2021a.
Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15262 15271, 2021b.
Dan Hendrycks, Steven Basart, Mantas Mazeika, Andy Zou, Joseph Kwon, Mohammadreza Mostajabi, Jacob Steinhardt, and Dawn Song. Scaling dut-of-distributiin detection for real-world settings. In Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 8759 8773. PMLR, 2022. URL https://proceedinggs weteesmmr..pprrreeee wttpppra//11/2///rrreeeeeee html
mlr
Wen Heng, Shuchang Zhou, and Tingting Jiang. Harmonic adversarial atlack method. arXiv preprinl arXiv:1807.10590 2018.
Rui Huang and Yixuan Li. Mos: Towards scaling Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognation, pp. 8710-8719, 2021.
detection for large semantic space. In
Nathan Inkawhich, Gwendolyn McDonald, and Ryan Luley, Adversarial attacks on foundational vision models. arXiv preprinl arXiv:230.115599 145977, 2023.
Ronald Kemker, Mare McClure, Angelina Abitino, Tyler Hayes, and Christopher Kanan. Measuring catastrophic forgetting in neural networks. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. Advances in neural information processing syslems, 11661,, 2020.
Diederík Kingma and Jimmy Ba. Adam: A method for stochastic optimization. International Conference on Learning Representations, 12 2014.
Alex Krizhevsky, GeofTrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. simple unified framework for detecting out-of- distribution samples and adversarial attacks. Advances in neurol information processing systems, 31, 2018.
Shiyu Liang, Yixuan Li, and R. Srikant. Enhancing the reliability of out-o--distribution image detection in neural networks. In International Conference on Learning Representations, 2018. URL https:// hwtprrevwee.ett//ooo////allllloe
16

---

Published in Transactions on Machine Learning Research (03/2024)
Weitang Liu, Xiaoyun Wang, John Owens, and Yixuan Li. Energy-based out-of-distribution detection. Aduances in neural informalion processing systems, 33:21464-21475, 2020.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedinggs of the IEEE/CVF international conference on computer vision, pp. 10012-10022, 2021.
Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet. for the 2020s. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11976 11986, 2022.
Philipp Liznerski, Lukas Ruff, Robert A. Vandermeulen, Billy Joe Franks, Klaus Robert Muller, and Marius Kloft. Exposing outlier exposure: What can be learned from few, one, and zero outlier images. Transactions on Machine Learning Research, 2022. ISSN 2835-8856.
Stuart Lloyd. Least squares quantization in pem. IEEE transactions on information theory, 28(2):129-133 137, 1982.
Cheng Luo, Qinliang Lin, Weicheng Xie, Bizhu Wu, Jinheng Xie, and Linlin Shen. dreiuency-driven imperceptible adversarial attack on semantic similarity. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 15315 15324, 2022.
Dhruv Mahajan, Ross Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li, Ashwin Bharambe, and Laurens Van Der Maaten. Exploring the limits of weakly supervised pretraining. In Proceedings of the Europeon conference on computer vision (ECCV), pp. 181-196, 2018.
Chengzhi Mao, Scott Geng, Junfeng Yang, Xin Wang, and Carl Vondrick. Understanding scro-shot adversarial robustness for large-scale models. In The Flleventh International Conference on Learning Represenlations, 2023. URL http::/pppaaareeeee..eee/rrrrrrp
Yifei Ming, Ziyang Cai, Jiuxiang Gu, Yiyou Sun, Wei Li, and Yixuan Li. Delving into distiiddist detection with vision-language representations. Advances in Neural Information Processing Systems, 35: 35087-35102, 2022.
Sina Mohseni, Arash Vahdat, and Jay Yadawa. Shifting transformation learning for out-of-distribution detection. Xiv preprinl +xXi::2106.0389, 2021.
Eric Nalisnick, Akihiro Matsukawa, Yee Whye Teh, Dilan Gorur, and Balaji Lakshminarayanan. Do deep generative models know what they don't know? In International Conference on Learning Representations, 2019. URT.
Muhammad Muzammal Naseer, Kanchana Ranasinghe, Salman H Khan, Munawar Hayal, Fahad Shah- baz Khan, and Ming-Hsuan Yang. Intriguing properties of vision transformers. Advonces in Newral 2021.
Information Processing Systems,
Anh Nguyen, Jason Yosīnski, and Jeff Clune. Deep neural networks are casily fooled: High confidence predictions for unrecognizable images. In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 427-44457 436, 2015. doi: N0.1109/CVPR.2015.777
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.071 071193, 2023.
Namuk Park, Wonjae Kim, Byeongho Heo, Taekyung Kim, and Sangdoo Yun. What do self-supervised vision transformers learn? In The Eleventh International Conference on Learning Representations, 2023. URL
https://openreviee
Hieu Pham, Zihang Dai, Golnaz Ghiasi, Kenji Kawaguchi, Hanxiao Liu, Adams Wei Yu, Jiahui Yu, Yi- Ting Chen, Minh-Thang Luong, Yonghui Wu, al. Combined scaling for zero-shot transfer learning.
Neurocompuling, 555:1 12658, 2023.
17

---

Published in Transactions on Machine Learning Research (03/2024)
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pp. 8748 8763. PMLR, 2021.
Nima Rafice, Rahil Gholamipoor, Nikolas Adaloglou, Simon Jaxy, Julius Ramakers, and Markus Kollmann. Self-supervised anomaly detection by self-distillation and negative sampling. In Artificial Neural Networks and Machine
2022: 31st International Conference on Artificial Newral Nelworks, Bristol, Springer, 2022.
UK, September 6 9, 2022, Proceedings; Part IV, pp.
Tal Reiss, Niv Cohen, Liron Bergman, and Yedid Hoshen. Panda: Adapting pretrained features for anomaly detection and segmentation. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 1506-28144 2021.
Jie Ren, Peter J Liu, Emily Fertig, Jasper Snoek, Ryan Poplin, Mark Depristo, Joshua Dillon, and Balaji Takshminarayanan. Tikelihood ratios for out-of-distribution detection. Advances in neural information processing systems, 32, 2019.
Jie Ren, Stanislav Fort, Jeremiah Liu, Abhijit Guha Roy, Shreyas Padhy, and Balaji Lakshminarayanan. A simple fix to mahalanobis distance for improving near-ood detection. arXiv preprinl arXiv:2106.0900 2021.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenct large scale visual recognition challenge. International journal of computer vision, 115:211 252, 2015.
Christoph Schuhmann, Romain Beaumont, Richard Veneu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Newral Information Processing Systems, 35:25278 25294, 2022.
Vikash Schwag, Mung Chiang, and Pratock Mittal. (SSD): A unified framework for self-supervised outlier detection. In International Conference on Learning Representations, 2021. URL htttps://openrevier
het/foruuriid-v5gjjppne
Yiyou Sun, Yifei Ming, Xiaojin Zhu, and Yixuan Li. Out-of-distribution detection with deep nearest neighbors. In International Conference on Machine Learning, pp. 30827-20840 PMLR, 2022.
Christian Szegedy, Wojciech Zaremba, 1lya Sutskever, Joan Bruna, Dumitru Erhan, lan J. Goodfellow, and Rob Fergus. Intriguing properties of neural networks. In Yoshua Bengio and Yann LeCun (eds.), 2nd International Conference on Learning Representalions, ICLR 2014, Banfl, AB, Canada, April 14- 16, 2014, Conference Track Proceedings, 2014. CRL http://arxiv.orrt wtgg/bs////bb/a/aeeee///// 6199
Jihoon Tack, Sangwoo Mo, Jongheon Jeong, and Jinwoo Shin. Csi: Novelty detection via contrastive learning on distributionally shifted instances. Advances in neural information processing systems, 08:11839-11852 2020.
Chuanqi Tan, Fuchun Sun, Tao Kong, Wenchang Zhang, Chao Yang, and Chunfang Liu. A survey on deep transfer learning. In Artificial Neural Networks and Machine Learning-CCANN ICANN 2018: 27th International Conference on Artificial Neural Networks, Rhodes, Greece, October 4-7, 2018, Proceedings, Part III 27, pp. 270 279. Springer, 2018.
Hugo Touvron, Matthieu Cord, Alexandre Sablayrolles, Gabriel Synnaeve, and Hervé Jégou. Going deeper with image transformers. In Proceedings of the IEFF/CVF International Conference on Computer Vision, pp. 32 42, 2021.
Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pictro Perona, and Serge Belongie. The inaturalist species classification and detection dataset. In Proceedings of the IEFE conference on computer vision and pallern recognition, pp. 8769 8769-8778, 8778, 2018.
18

---

Published in Transactions on Machine Learning Research (03/2024)
Hualiang Wang, Yi Li, Huifeng Yao, and Xiaomeng Li. Clipn for zero-shot ood detection: Teaching clip to say no. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1012-1812, 2023.
Florian Wenzel, Andrea Dittadi, Peter Vincent Gehler, Carl-Johann Simon-Gabriel Max Horn, Dominik Zietlow, David Kernert, Chris Russell, Thomas Brox, Bernt Schiele, et al. Assaying out-of-distribution generalization in transfer learning. arXiv preprint arX1v:2207.09239, 2022.
Ross Wightman. Pytorch image models. http:://gghh..........aaaaaaaaaa
2019.
Jim Winkens, Rudy Bunel, Abhijit Guha Roy, Robert Stanforth, Vivek Natarajan, Joseph R Ledsam, Patricia MacWilliams,, Pushmeet Kohli, Alan Karthikesalingam, Simon Kohl, et al. Contrastive training for improved out-of-distribution detection. arxiv preprint orXiv:2007.05000 2020.
Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roclofs, Raphael Gontijo Lopes, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong, et al. Robust fine- tuming of zero-shot models. In Proceedings of the LEEE/CVE Conference on Computer Vision and Pattern Recomnition, pp. 7959 7971, 2022.
Jianxiong Xiao, Jamos Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun databasc: Largo-scale scene recognition from abbey to 20o. In 2010 IEEE compuler sociely conference computer vision and pattern recognition, pp. 3485-3492. IEEE, 2010.
Qizhe Xie, Minh- Minh-Thang Luong, Eduard Hovy, and Quoe Le. Self-training with noisy student improves imagenet classification. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10687 10698, 2020.
Jingkang Yang, Pengyun Wang, Dejian Zou, Zitang Zhou, Kunyuan Ding, Wenxuan Peng, Haoqi Wang, Guangyao Chen, Bo Li, Yiyou Sun, al. Openood: Benchmarking generalizod out-of-distribution detection. Advances in Neural Information Processing Systems,
2022.
Heng Yin, Hengwei Zhang, Jindong Wang, and Ruiyu Dou. Boosting adversarial attacks on neural networks with better optimizer. Secur. Commun. Networks, 2021 -993809911,,,,3099, 2021. doi: 10.1155/2021] 9983309. URL hnttps://oo1. http:::://oo..r//.....oo 115.111111///22/988333000
Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks? Advances in neural information processing systems, 27, 2014.
Hongjic Zhang, Ang Li, Jie Guo, and Yanwen Guo. Hybrid models for open set recognition. In Computer Vision ECCV 2020: 16th European Conference, Glasgow, UK, August 23 28, 2020, Proceedings, Part III 16, pp. 102-117. Springer, 2020.
Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEEE transactions pattern analysis and machine intelligence, 40(6): 1452 1464, 2017.
19


Reviews:
[{"review_id": "0jNrVnaJMm", "reviewer": "Reviewer_a6kb", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n=======\n* The insights provided by the authors in this work are interesting\n* The methodology is sound\n* The authors provide an ablation and answer a series of interesting questions in the discussion (e.g. whether linear probing is also best for ImageNet 21K\n\nWeaknesses\n=========\n* While the authors claim ``PLP outperforms the previous state-of-the-art (Ming et al., 2022)'', they miss recent work such as [A]\n* While it is possible that any state of the art adversarial attack for computer vision would have sufficed to show the vulnerability of the OOD detector, the authors decided to introduce a new method without properly quantifying its performance nor comparing it to previous SoTA. \n\n[A] Wang, Hualiang, et al. \"Clipn for zero-shot ood detection: Teaching clip to say no.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors note that CLIP's performance on ImageNet is correlated with its OOD detection performance. Based on this finding, they propose using CLIP features for OOD detection. In that sense they find linear probing to be superior than fine-tuning. They focus on unsupervised OOD detection, since labels are not available, they follow the procedure introduced by Ming et al. (2022), where ImageNet label names are compared to in-distribution images in order to derive pseudo-labels. The authors explore whether the capabilities of CLIP extend to adversarial detection, concluding this is not the case. The authors include ablations and discussion including different pre-training tasks and datasets.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "* The authors claim to \"introduce a novel method that adversarially manipulates OOD images by matching their representations\" without a proper quantitative evaluation, ablation and comparison with other adversarial methods. For example, why not using any of these already existing smoothing techniques? [B,C,D]. Another option would be that instead of \"introducing a novel method\", you  \"show OOD detection with CLIP is vulnerable to adversarial attacks\". (until this point is resolved I am going to check \"no\" at whether the \"the claims made in the submission supported by accurate, convincing and clear evidence?\")\n* Could you update the text to reflect how your work compares to [A]? Note I am not asking for \"beating\" [A] but at least putting your submission in context of more recent work and adding some discussion about how your works compare and what are the advantages of your method (e.g. not needing to learn a \"no\" text encoder).\n* The linear relation between ImageNet and OOD detection performance is very similar to the effective robustness curve shown in the original CLIP paper (e.g. Figure 13)\n\n[B] Frequency-driven Imperceptible Adversarial Attack on Semantic Similarity\n\n[C] Low Frequency Adversarial Perturbation\n\n[D] Harmonic Adversarial Attack Method", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Uq7GfXUDV5", "reviewer": "Reviewer_V5a6", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n------------\n\n1. As large-scale pre-trained models/foundation models are getting widely adopted it is important to study if they can also be effective in unsupervised OOD detection. The study establishes that large-scale CLIP models serve as robust unsupervised OOD detectors without the need for in-distribution fine-tuning. The strong positive correlation (R2 coefficient ≥ 0.92) between in-distribution accuracy and unsupervised OOD detection across multiple benchmarks underscores the effectiveness of CLIP models.\n\n\n2. The paper also introduces a novel method, pseudo-label probing (PLP),  a simple and scalable method to adapt CLIP representations for OOD detection. Leveraging text-based pseudo-labels computed using the text encoder, PLP outperforms the previous state-of-the-art method on ImageNet benchmarks, showcasing its effectiveness in enhancing OOD detection performance.\t\n\t\t\t\t\n3. The paper presents a novel method for adversarially manipulating out-of-distribution images, revealing that CLIP ViT-G, trained on billions of samples, shows substantial vulnerability to these perturbations with an AUROC deterioration from 86.2% to 50.3% due to imperceptible changes.\n\n\nWeaknesses\n----------------\n1. The paper lacks a deeper investigation of the observed results. In particular, it would be helpful to have further experiments to explain why should one expect CLIP embeddings to work well for OOD detection. Evaluation is all on natural images ( that are plentiful on the internet) and CLIP has likely been trained on these or similar images. Is it the wide range of training data that makes them suitable for OOD detection or is it the architecture or training procedure? Would similar claims be valid for CLIP models trained for specific domains e.g. https://github.com/Mauville/MedCLIP? \n\n2. I found the discussion of PLP (section 3.3.) to be inadequate and it is written with the assumption that the reader is familiar with the prior work (Ming et al. 2022). It would be helpful to provide the necessary background and the details of the method and also discuss the differences with Ming et al. 2022.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper presents a comprehensive experimental study on pre-trained feature extractors such as CLIP models for visual out-of-distribution (OOD) detection. Without fine-tuning on the training data, the authors establish a strong positive correlation (R2 ≥ 0.92) between in-distribution classification and unsupervised OOD detection for CLIP models across four benchmarks. They introduce a new and scalable method called pseudo-label probing (PLP), which utilizes label names of the training set to train a linear layer using pseudo-labels derived from the text encoder of CLIP for OOD detection. The authors develop a novel feature-based adversarial OOD data manipulation approach to evaluate the OOD detection robustness of pre-trained models. The study shows that PLP surpasses the previous best method on all five large-scale ImageNet benchmarks, achieving a 3.4% AUROC gain with ViT-G. Linear probing outperforms fine-tuning for CLIP architectures, with ViT-H gaining 7.3% AUROC on average across all ImageNet benchmarks. However, billion-parameter CLIP models struggle to detect adversarially manipulated out-of-distribution images.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please see the weaknesses above.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "bq9VvvUQME", "reviewer": "Reviewer_FVMe", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\nThe paper is well-written and easy to follow. Related works and the experimental settings are clearly presented. The proposed way to curate adversarial examples is also interesting.\n\n**Weakness**\n\nThe comparison with the baseline seems unfair and need justification. The author should discuss their contribution compared with the existing works more clearly. See requested changes.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper propose to fine-tune a linear probing layer with the pseudo-label obtained by CLIP to train a visual OOD detector using two existing scores, the RMD and Energy scores. The paper also studies the adversarial image curation by optimizing towards higher cosine similarity discrepancy while keeping the perturbation invisible by adding a regularization term.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. The author says in Section 3.3 that the baselines (Fort et al., 2021; Huang & Li, 2021) differs from the proposed PLP method by fine-tuning the whole visual backbone instead of linear probing. The other components, e.g., the objectives and the pseudo-label, all seem to have been adopted in the baselines. Could the author clarify what is the key contribution of PLP part, as parameter-efficient fine-tuning is a well-known and widely adopted method when scaling method up. \n\n2. Could the author conduct experiments using other parameter-efficient fine-tuning methods like adapter tuning and compare the trade-off between the number of trainable parameters and the performance?\n\n3. Table 3 reports a very low baseline score than its original reported ones in Ming et al. (2022), which the author claims to be caused by the different source weights of the base model. The gap is too large to make me believe it is caused by that the authors downloaded the pretrained weights from a different place. Could the author supplement one experiment by using the same HuggingFace weights as in Ming et al. and compare again?\n\n4. Figure 1 takes up the whole page 2, but it is first mentioned in page 9, which seems to be an unreasonable design, making me very confused when reading through the paper.\n\n5. Why Table 4 compares the fine-tuning with MSP with the proposed Probing with the MSP objective? From Table 5, the Energy objectively is obviously better than MSP. Isn't the fair comparison being Finetune+MSP vs Probing+MSP and Finetune+Energy vs Probing Energy? Please supplement corresponding experiments.", "broader_impact_concerns": "NA.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The following concerns were also raised by reviewers and addressed to their satisfaction by the authors' response and the updated submission:

* The presentation of PLP in Section 3.3 is not accessible to readers with no prior knowledge of related work.
* The differences between PLP and the Fort et al. (2021) and Huang & Li (2021) baselines need clarification.
* The submission should present results on the trainable parameters / performance trade-off.
* The placement of Figure 1 hurts readability.
Decision: Accept (poster)