Contrastive Vision Transformer for Self-supervised Out-of-distribution Detection
Authors: Hengding Wang
Yuchen Lu
Xi Chen
Venue: Submitted to ICLR 2023 (2023)
Keywords: Out-of-distribution
self-supervised learning
contrastive learning
vision transformer

Abstract: Out-of-distribution (OOD) detection is a type of technique that aims to detect abnormal samples that don't belong to the distribution of training data (or in-distribution (ID) data). The technique has been applied to various image classification tasks to identify abnormal image samples for which the abnormality is caused by semantic shift (from different classes) or covariate shift (from different domains). However, disentangling OOD samples caused by different shifts remains a challenge in image OOD detection. This paper proposes Contrastive Vision Transformer (CVT), an attention-based contrastive learning model, for self-supervised OOD detection in image classification tasks. Specifically, vision transformer architecture is integrated as a feature extracting module under a contrastive learning framework. An empirical ensemble module is developed to extract representative ensemble features, from which a balance can be achieved between semantic and covariate OOD samples. The proposed CVT model is tested in various self-supervised OOD detection tasks, and our approach outperforms state-of-the-art methods by 5.5% AUROC on CIFAR-10 (ID) vs. CIFAR-100 (OOD), and by 10.7% AUROC on CIFAR-100 (ID) vs. CIFAR-10 (OOD).

Under review as a conference paper at ICLR 2023
CONTRASTIVE VISION TRANSFORMER FOR SELF- SUPERVISED OUT-OF-DISTRIBUTION DETECTION
Anonymous authors
Paper under double-blind review
ABSTRACT
Out-of-distribution (OOD) detection is type of technique that aims detect abnormal samples that don't belong to the distribution of training data (or in- distribution (ID) data). The technique has been applied to various image clas- sification tasks to identify abnormal image samples for which the abnormality is caused by semantic shift (from different classes) or covariate shift (from dif- ferent domains). However, disentangling OOD samples caused by different shifts remains a challenge in image OOD detection. This paper proposes Contrastive Vi- sion Transformer (CVT), an attention-basse contrastive learning model, for self- supervised OOD detection in image classification tasks. Specifically, vision trans- former architecture is integrated as feature extracting module under contrastive learning framework. An empirical ensemble module is developed to extract rep- resentative ensemble features, from which a balance can be achieved between se- mantic and covariate OOD samples. The proposed CVT model is tested in various self- sufesuupevieed OOD detection tasks, and our approach outperforms state-of- t-the- art methods by 5.12% AUROC on CIFAR-10 (ID) vs. CIFAR-100 (OOD), and by 9.77% AUROC on CIFAR-100 (ID) vs. CIFAR-10 (OOD).
INTRODUCTION
As many deep neural networks (DNNs) are deployed in real-world applications, the safety and bustness of the models get more and more attention. Most existing DNNs are trained under the closed-world assumption, i.c., the test data assumed be drawn i.i.d. from the same distribution as the training data (Yang et al./ (2021). Although the deployed DNNs can perfectly deal with such ID samples, they would blindly classify the data coming from other classes or domains (i.e., OOD samples) into existing classes in an open-world scenario. Nguyen et al. discovered that neural net- works can be easily fooled by unrecognizable images, which means that most DNNs are unreliable when encountering unknown or unseen samples. Such few mistakes may be tolerable in some sce- narios (e.g., chatbot, interactive entertainment), whereas they will bring catastrophic damage when the application area requires great safety benefits, such as automated vehicles, medical imaging and biometric security system. Therefore, it is essential to equip the model with the ability of detecting out-of-distribution data and make it more robust and reliable.
unrecoggunn
Generally, the outlier arises because of the mechanical failure, fraudulent behaviour, human error, instrument error and natural deviations in populations (Hodge & Austin 2004). In the field of machine learning, compared with ID samples, OOD samples are regarded as the outliers due to distributional shifts. The distributional shifts can be caused by semantic shift (i.e., OOD samples from different classes) or covariate shift (i.e., OOD samples from different domains) (Yang et al., 2021I. Meanwhile, the OOD samples that are semantically and stylistically very different from ID samples are referred to as far-OOD samples, and those that are semantically similar to ID samples but different from ID samples in domains are referred to as nOD samples (Ren et al., 2021). The autoo-ddistribution detection, also known as outlier detection or novelty detection, is developed to identify whether a new input belongs to the same distribution as the training data. A natural idea is to build a classifier to identify the ID and OOD data, using such as Deep Neural Network (DNN) and Support Vector Machine (SVM). However, the sample space of OOD data is almost infinite as OOD dataset is the complementary set of ID dataset, which leads to that creating representative OOD dataset is impracticable. Moreover, OOD samples are scarce and costly in some industries (e.g., medical imaging, fraud prevention). These main issues in the research on OOD detection.

---

Under review as a conference paper at ICLR 2023
To address these problems, researchers focus on the latent features of ID data, assuming distin- guishable distributional shifts exist between ID and OOD samples in the latent feature space. Some researchers (Nalisnick et al.] 2019 Serrà et al. 2019 Xiao et al.) 2020) use generative models, like Variational Auto-encoders (VAH), to extract the latent features for both ID and OOD samples, and specific OOD socres are designed and used the metric. As alternative, contrastive learn- ing models can be employed to learn the latent features, such as Self-Supervised Outlier Detection (SSD) (Sehwag et al. 2020) and Contrasting Shifted Instances (CSI) (Tack et al. 2020). However, in contrastive learning, researchers usually adopt standard convolutional neural network (CNN) and its variants like ResNet (He 2016) as the encoder. By contrast, the transformer- -baee architec- tures (such as the earliest Vision Transformer (ViT) (Dosovitskiy et al. 2020). DeiT (Touvron et al.) 2021 and Swin Transformer (Liu et al. 202T) gradually outperform CNNs in terms of extract- ing robust latent features as they can learn global long-range relationships for visual representation
learning, which would facilitate the identification of ID and OOD samples.
In this paper, Contrastive Vision Transformer (CVT) model is proposed for OOD detection under self-supervised regime for image classification tasks. The framework of contrastive learning, in- cluding data augmentation and contrastive loss, is adopted to learn the representation for all inputs, which has been shown to be reasonably effective for detecting OOD samples (Tack et al.] 2020)) On this basis, four extra modules are introduced into this framework: (i) To improve the distin- guishability between ID and OOD samples in the latent space, vision transformer architecture rather than CNN embedded as a feature extracting module; (ii) Since the collapse of representation is a noteworthy problem in self-supervised and unsupervised scenarios, an additional predictor structure (inspired by BYOL (Grill et al.) 2020 is employed to avoid collapsed solutions. (iii) Considering that the size of negative samples plays an important role in contrastive learning, memory queue scheme from MoCo (He et al.) (020) is integrated to maintain the model's performance especially when the batch size is extremely small. (iv) An ensemble module is developed to build representa- tive ensemble features for achieving the balance between semantic and covariate OOD detection, as observe that in our experiments the latent features from the encoder perform better semantic OOD samples but on the contrary the latent fcatures from the predictor perform better on covariate OOD samples. To further improve performance, Mahalanobis distance-based OOD score func- tion is utilised for the OOD detection, the effectiveness of which has been shown in recent papers (Sehwag et al. 2020, Ren et al. 202T).
To conclude, the key contributions of the paper are as follows:
We integrate vision transformer architecture into contrastive learning framework and de- velop a new paradigm specifically for self-supervised OOD detection in image classifica- tion tasks, results outperform state-of-the-arr algorithms
We develop ensemble module to compute representative features that balance OOD samples from different types of data shifts
We conduct extensive ablation studies to report the influences of various on OOD detection tasks and benchmark the performance of CVT using different vision transformer modules including ViT, ResNet50, and Swin transformer
In the rest of the paper, related work is described in Section 2 and the main CVT model is introduced in Section Followed by numerical results Section 4 and the paper is concluded in Section
2 RELATED WORK
Contrastive learning is a self-supervised technique that has seen fast development in recent years. Chen et al. (2020) proposed contrastive learning framework consists of four components: data augmentation module, neural network base encoder, MLP (multilayer perceptron) projection head, and contrastive loss. It incorporated a strong inductive bias by gathering samples from the same class and repelling others and achieved promising results in visual representation learning. Under a similar paradigm, many influential variants were develooped in recent years, such as SimCLR (Chen) et al., 2020). MoCo, SWAV (Caron et al., 2020). BYOL and MoCo-v3 (Chen et al. 2021). MoCo introduced queue module store the key representations of negative samples since the number of negative samples can effectively improve performance. To maintain the consistency of keys in the queue, momentum strategy was developed for MoCo to update the parameters of the key encoder.
2

---

Under review as a conference paper at ICLR 2023
BYOL adopted an asymmetric structure to prevent the model from collapsing by adding a extra predictor module after the projector for the online networks.
Current practice in contrastive learning adopts standard convolutional neural networks (CNN) like ResNet(He et al. 2016) as the encoder for image feature extraction. However, the attention-base architectures has recently outperformed CNN architectures in major vision tasks such as image classification, object detection, and semantic segmentation. It has been shown that attention mod- ule is capable of extracting global long-range relationships for visual representation learning. For example, ViooDosovitskiy et al., 2020) applied standard Transformer encoder to images classifica- tion tasks and reported good performance. It chopped image into sequences of patches fit the sequence- Transformer model. It also demonstrated that the multi-head self-attention (MSA) was beneficial for representation learning in images classification. To achieve excellent results, ViT required large training data and extensive computing resources. Dei'T (Touvron et al. 2021) intro- duced a distillation procedure (a teacher-student strategy) to enable Transformer learning from a convnet and shrinking the training time. To bring the benefit of self-attention to object detection and semantic segmentation tasks, Swin Transformer(Liu et al. (02T)) employed a hierarchical ar- chitecture to produce features at various scales. It is worth to mention that, the shifted windowing approach in Swin Transformer brought a linear computational complexity compared to standard self-attention.
The generalised OOD detection operations were proposed by Yang et al. 2021) where they di- vided the methodology into four categories: classification-basete density-based, distance-based and reconstruction-bet based methods. The classification-base method usually applies a softmax func- tion to the output layer, which enables the model producing a probabilistic result for all classes. Hendrycks & Gimpel (2016) observed that a well-trained model can give higher probability to ID samples than OOD ones. Based on the findings, ODIN (Liang et al. al., 2017) adopted temperature scal- ing in softmax function to separate ID/OOD softmax scores under standard classification models. However, the model may sometimes assign a high probability on known classes for OOD samples. To address this issue, Dirichlet-base uncertainty (DBU) (Kopetzki et al. 2021) incorporated uncer- tainty estimates by predicting the parameters of a Dirichlet distribution. An outstanding advantage of DBU models is that it can effectively compute epistemic distribution, alcatoric distribution, and class labels. With respect to density-based method, it established probability distribution of ID ex- amples and placed the OOD samples in low-density areas. [Pichorskyi et al. (2018) utilised autoen- coder network to capture underlying structure of data distribution and computed novelty probability. Compared to classification-basedd approaches, generative models usually had worse performance Yang et al., 202T). The idea of distance-based methods was that the distance from OOD input to ID samples may be relatively farther than distance from ID input to ID samples. People can lever- age distances with Euclidean distance, geodesic distance, cosine similarity, or Mahalanobis distance between the feature embeddings of input and the centroids of all classes. SSD (Sehwag et al., 2020) showed that the Mahalanobis distance is effective in OOD detection based on contrastive learning framework. Another OOD detector, CSI Tack et al.] (2020) was also based on contrastive learning and it used distribution-shiftin augmentations (e.g., rotations) to promote OOD detection.
3 METHODOLOGY
This section details the architecture of the proposed CVT model, simple yet efficient self- supervised OOD detector, followed by the design of loss functions and evaluation metrics.
3.1 MODEL ARCHITECTURE
In some previous research (Tack et al. 2020 Ren et al. 2021 Yang et al. 2021), OOD detection divided into two stages: first, representation learner is built and trained to extract latent features; second, an OOD score (c.g., softmax probability and distance-based score) is computed as the metric to indicate whether sample is an OOD one. As shown in Fig. I the general architecture of CVT model is composed by two parts: (i) a contrastive representation learner based on contrastive learning framework, and (ii) Mahalanobis distance-base based OOD score function.
The contrastive representation learner consists of two parallel networks: an online network, whose parameters are updated by contrastive loss and back-prop,,,tion, and targel network, where
3

---

Under review as a conference paper at ICLR 2023
moving average of the corresponding online parameters is used its parameters (Grill al, 2020). Unlike the traditional contrastive learning model, all the encoder modules in our CVT are flexible and can be replaced by any DNN designed for extracting latent features. Here, the ViT (Fig. 2 is given as an example for the encoder module, and a pre-trained version preferrd since the training of it requires lot of samples. Furthermore, additional predictor structure in BYOL. is employed in the online network to avoid collapsed solutions, as well as a memory queue scheme MoCo is integrated into the target network maintain the model's performance under different batch size settings. Although MoCo-v3 abandoned the memory queue design and used a large batch size train the model, we still keep available especially on low-memory computing infrastructure.
Encoder f(.)
Projector g(t)
Predictor g(+)
madule
Ensemble
Data ane Augmentatie
Computing maximum
(2)
Mahalanobis distance confidence
Contrastive
Loss
111
Enccdder 1o1
Projector
f12
Queue
threshold
<= threshold
72
sample
sample
Online Network
Contrastive representation learner
Target Network
OOD socre function
Figure 1: Architecture of the CVT model, in which the left part is contrastive representation learne composed by two parallel deep networks (online and target networks highlighted in blue and grey respectively) with ireettoo-bbsed encoder ((), projector 9()). and/or predictor q()). The ensemble module computes output features from Encoder (f,) and Predictor (21), then send ensemble features for OOD score computation on the right of the figure.
Transformer Encoder Block
12
SL
MLP
Encoder Block
Encoder Block
Encoder Block
224 x
fc
F
24
Norm
892-
332
224
Mulit-Heac Attention
768
Norm
196 768
Embedding Patches
Encoder
Projector
Predictor
Figure 2: Design of the contrastive representation learner in Fig. I CLS is a placeholder to be used for prediction downstream tasks and fc represents fully connected layer. Numbers the figure indicate the dimension of the corresponding input/output. The encoder block highlighted in green detailed on the right panel the figure. Here we take a standard Transformer encoder as an example, however, the encoder block is flexible for other (non) attention-based architectures.
Given a dataset { without labels, two views {a (1) i-1 and {= ( each sample are generated with data augmentation by using random ensemble transformations (Grill et al.) 2020). and 2N samples are obtained in total. Then, the augmented data will be fed into the model in batches. For each view pair of the sample, (7(1), (2)),, the representation pairs of (2 222)) and (zf 2/ (2)) are produced by the predictor q(.) in the online network and the projector g(.) in the target network respectively. Meanwhile, two memory queues, Q1 and Q2, created in the target

---

Under review as a conference paper at ICLR 2023
(1) and 2 respectively. The contrastive loss is calculated network store the presentations 22 among the representations given by the two networks (Section (.2). While predicting, only the online network used generate the representations for each sample (no data augmentation).
(2)
For the OOD score function, we use the ensemble features to compute the maximum Mahalanobis distance between a new input and the clusters of the training in latent space. The high-level features (e.g., from the predictor) usually contain more semantic information and low-level features (e.g., from the encoder) have more general properties. The semantic information helps distinguish near-OOD samples but fails on far-OOD samples. General properties can contribute to the identi- fication of samples from different domains. Therefore, an ensemble module in the online network, where the element-wiss wise mean of featuree/reprrsenntaioons from the encoder and predictor is com- puted, is developed to build representative ensemble features for achieving the balance between se- mantic and covariate OOD detection. To keep the ensemble module feasible, we set the dimension of the output from the predictor to be the same as that from the encoder.
3.2 LOSs FUNCTION DESIGN
For unlabeled training data, the representations be extracted by contrastive learning models. Our model absorbs the strengths of MoCo, MoCo-v3, and BYOL, and employs InfoNCE (Oord et al. 2018) as our training objective. The total loss is described as follows:
Lrotal Lg(0 2(00)
exp(2)
(1) .2/ /T)
200) = - log
(2)
E exp(x) (1) (T)
(,0) log exp(2 (2)
11
22 /T)
En exp(2 (2 2 (2) 2 [T)
where z is the embedding stored in queue, and queueg, K is the sum of the size of both queues and r is hyper-parameter temperature.
After the training, we just use the online network to extract embeddings from new inputs. Given another two dataset {x, Ji=s and {a,0od J,M, our OOD score function should assign a value to each input. we choose distance-based method compute the scores, which the Mahalanobis distance from input to centroids in training dataset. Therefore we have to divide training data into C clusters. Here we use k-means clustering approach to classify every training samples to particular class. Finally, we consider the maximum Mahalanobis distance as the OOD score, show in Eq. (
score(x) = max - (x -te)"
(4)
cEC
where is the class of training data, the embedding of input samples,pe the mean of training data from class c.
3.3 EVALUATION METRICS
As shown in Fig. we define an OOD score based threshold classify whether a new sample belongs to OOD or not. Specifically, sample is regarded as an OOD sample if its OOD score is larger than the predefined threshold, otherwise it is treated an ID sample.
We adopt three commonly used metrics to evaluate the performanc eof OOD detector. AUROC, which is a threshold- inddeeddn evaluation method and can reflect the ability of model to discrimi- nate between positive examples and negative examples. A high AUROC value means the model has good discriminatory ability and an AUROC of 0.5 corresponds to a random guess (useless model). Area Under the Precision-Recall (AUPR) is another evaluation metric that represents the ability to correctly distinguish positive samples without treating positives as negatives. The third one, FPR95 represents the false positive rate (FPR) when the true positive rate (TPR) equals to 95%. It measures the size of OOD samples that are not correctly recognised when 95% of ID samples are correctly classified.
5

---

Under review as a conference paper at ICLR 2023
4 EXPERIMENTS
We follow commonly used benchmarks from previous work and consider CIFAR-I0 and CIFAR- 100 (Krizhevsky et al. 2009) as ID datasets. For the OOD dataset, we choose Describable Textures Dataset (DTD) (Cimpoi et al. 2014) and SVHN (Netzer et al. 2011) in order to compare the CVT model with other competing OOD detectors. ID samples are divided into ID training set and ID test set, with data leakage issue considered. For all experiments, models are trained with the ID training set in the training phase, and tested with both ID test set and OOD set in the evaluation phase. All the experiments are repeated 5 times by using different random seeds. We evaluate our model by three metrics: AUROC, AUPR, and FPR95.
In our benchmarking and ablation studies, we benchmark the performance of different representative base encoder networks under the CVT model, including V-T-B-16 (Dosovitskiy et al. 2020), pre- trained ViT variant on Imagenet2012 (aeeee20122RRssaaaaa (Russakovsky et al. 2015); Swin Transformer, the popular and powerful transformer architecture; and ResNet50, the basic but widely used CNN architecture. We train the proposed CVT model using optimizer Adam ([oshcchilo & Hutter 2018) with initial learning rate 1e-6, half-cycle cosine decay, weight decay 0.1, and batch size equals to 64. Following the setup in literature SSD (Schwag al., 2020), we set the default temperature to 0.5 and ablate different temperature and number of clusters pairs, see Section (.5 more details. All experiments on Ubuntu 12.04 with two NVIDIA RTX A4000 GPU cards. We use Python 3.8. 10 and the single queue size in CVT is set to 4096.
4.1 CVT MODEL PERFORMANCE
As shown in the first row of Table T we test 6 pairs of combinations of ID and OOD datasets. Both CIFAR- 10 and CIFAR-100 datasets shares common image characteristics (e.g.. format, quality, style etc.), while SVHN dataset and Describable Texture dataset are more dissimilar to them.
We first compare CVT performance in self-supervised OOD detection tasks, where models are trained with unlabeled training data. We compare AUROC performance (higher score is better) across 6 different models including FixeCCNN+((Salmane
et al.) (017)). Deep- (RDu-f et al. & Gidaris) 2018), CSI(Tack et al. (020), and SSD(Sehwag et al. 2020 (shown in the top part of Table The results demonstrates that the proposed CVT model
2018 Craation-los(Kooodaaa
clearly outperforms all other competing models. The averaged AUROC of CVT model has improved by 6.6% when comparing with the the 2nd best one (i.e., SSD). In the challenging scenario where CIFAR-100 100 is the ID set and CIFAR-10 10 serves as the OOD set, most existing OOD detectors per- form poorly due the similarity and size of CIFAR- over CIFAR-10. Our CVT model achieves a noteable score of 79.37%, which is almost 10% improvement from that of the 2nd best model.
Table I: Model comparison in AUROC with 6 self-supervised OOD detectors using unlabelled training data and 5 supervised OOD detectors using labelled training data. The best performed number is highlighted in bold for each ID/OOD pair. The last column reports the averaged AUROC over all available tests (when the number of tests is no less than 4).
ACC
CIFAR- OOD:
CIPAR-100 vs. OOD:
Method
fon CIFAR-10)
SVHN
CIFAR-100
Texture
SVHN
CIFAR-10
Texture
Averago
Pixe
15.8%
52.45%
DecD-SVDD
14.5%
52.1%
16.3%
51.4%
33.6%%
Rotation-loss
97.0%%
81.25
84.45
30.1%
80.9%
CSI
94.38%
99.8%
80.2%
SSD
95.07%
99.65
90.8%
07.6%
94.9%
60.8%
82.9%%
80.25
CVT (ours)
97.92_m
900 %
72.0720
100
98. %
o02%
100
CE+SiNCLR!
99.5%
92.0%
95.6%
78.3%
91.6%
CST
16.1 9
97.9 L..1
92.2 1..1%
70.71%
SSDD"
94.35%
99.9%
93, 15%
98.5
88 256
78.35 28.355
81.2%
CIDERT 94.463%
90.735%
033.04%
97. .444
97.753
77.02%
91.96%
92. 92%
Fort al.
98.70%
55%552%
The repecatdd resuilss using differeni random
The suporvised detection mathood uses
training.
We also compares CVT model against supervised OOD detectors, where models are trained with labelled ID training set and tested by both unseen ID test set and OOD set. This is an unfair com- parison to the CVT model as it is still trained by unlabelled (but the same) ID training set, i.c., no label information has been provided to CVT model. Surprisingly CVT still outperforms majority of the supervised OOD detectors except model in (Fort al. 2021). As shown in the bottom part

---

Under review as a conference paper at ICLR 2023
of Table 1 we benchmark CE+SimCLR, SSSSimC(RWiWeenn Winkens al. 2020 supervised CSI, supervised SSD, Cing et al.) 2022), and (Fort et al. (022]) in the two pairs of CIFAR datasets. To briefly conclude, the proposed CVT model performs well in different OOD detection tasks for both near/far OOD datasets.
4.2 COMPARISON BETWEEN ENSEMBLE FEATURES AND OTHER FEATURES
As depicted in Fig. 1 develop ensemble module to balance the features extracted from both encoder and predictor. In this subsection, we test and show the effectiveness of the ensemble module in the CVT model.
Table 2: Performance of ensemble module in CVT. 'Encoder features' and 'Predictor features' represent output features produced by the Encoder component and the Predictor component in the CVT model, respectively. 'Ensemble features' describes the merged features computed on both encoder/predictor features.
ID dataset
OOD dataset
Encoder features Predictor features
Ensemble features
CIFAR-100
94.89%
96.56%
96.06%
CIFAR-10
SVHN
99.95%
99.62%
99.93%
CIFAR-10
78.00%
77.26%
80.34%
CIFAR-100
SVHN 99.06%
95.41%
99. 99.21%
The first two columns in Table four settings that grouped into semantic shift scenarios (e.g.,
vs. SVHN) and covariate shift scenarios (e.g.. CIFAR-I0 vs. CIFAR- 100). By using the features from either the Encoder or the Predictor to compute the OOD score for detecting OOD samples, the results display different trends under the above two scenarios. The race of the OOD detection under the covariate shift scenario between low-level features (i.e., the features from the encoder) and high-level features (i.e., the features from the predictor) ends in draw. By contrast, low-level features dominate the OOD detection under the semantic shift scenario. Therefore, it is hard to decide which kind of features should be finally used to compute the OOD score. However, by comparing the performance of ensemble features with others under CIFAR-10 ID dataset, one sees that the ensemble module balances the performance between low-level and high- level features. Interestingly, one also observes that ensemble features achieves better performance under CIFAR-100 ID dataset. We reckon the ensemble features not only balance features but also enhance features for OOD detection.
4.3 ENCODER USING VIT VS. RESNET VS. SWIN TRANSFOMER
To evaluate our hypothesis that the attention-based models are capable of extracting more effective features for OOD detection, we compare the performance between three architecture options for the encoder module in CVT, namely ResNet50, ViT-B-16 (Base model of ViT), and Swin-B (Base model of Swin Transfomer). In this experiment, we calculate OOD scores using features output from predictor and the number of dimensions is as 768.
Fig. shows experimental results of using CIFAR-10 as the ID set while CIFAR-100 (left panel) and SVHN (right panel) as the OOD set. One sees that both ViT and Swin Transformer architec- tures clearly outperform standard ResNet50 by about 26% in both AUROC and AUPR (higher the better), and with 67% decreases FPR95 (lower the better). On the other hand, ResNet50 achieves slightly better performance in the far-OOD scenario (right panel). Back to the comparison between ViT-B-16 and Swin-B, B. one sees comparable performance performs slightly better in FPR95. This might be due to shifted window introduced in Swin Trans- former, which reduces its computational complexity but makes it slightly less capable in information extraction.
AUROC and AUPR, with ViT model
7

---

Under review as a conference paper at ICLR 2023
CIFAR1O vs. CIFAR100
CIFAR1O vs. SVHN
ResNet50
VIT-B-16
Swin-B
ResNet50
VIT-B-16
Swin-B
96.6 96.1
96.3
95.8
99.9 99.6 99.5
99.8 99.2 98.8
83.7
69.9
16.6
19.3
0.2 1.2 2.7
AUROC
FPR95
AUPR
AUROC
FPR95
AUPR
Figure 3: Performance comparison using ResNet50, ViT-B-16, and Swin-B.
4.4 DISTRIBUTIONS OF SEMANTIC SHIFTED AND COVARIATE SHIFTED DATASETS
We visualise and compare the ID and OOD distributions over the four datasets (CIFAR-10/100, SVHN and Texture) in feature space, in which CIFAR-I0 is set as the ID) set and the rest three are the OOD sets.
1.0
FNR, FNR_CIFAR CIFAR-10
CIFAR-10
CIFAR-100
0.8
SVHN
FPR. Texture
Textures
FPR FNR
0.6
Dudp
0.2
0.0
score
Figure 4: Visualisation of in-distribution and out-of-distribution. The left figure is the FNR of ID datasets and the FPR of OOD datasets under different threshold of OOD score, and the right one visualization of embedding on feature space.
From the right panel of Fig. one sees that both CIFAR-10 and CIFAR-100 shares some points and the contrary SVHN points more focal and locates further away from the CIFAR datasets. The left panel of Fig. 4 depicts the FNR and FPR of ID and OOD datasets separately in terms of the threshold of OOD score, which can illustrate the overlap level among different datasets. One observes that only CIFAR datasets are overlapped with each other obviously (about 16% of CIFAR- 100 samples are mixed with CIFAR-10) while SVHN and Texture are slightly overlapped, even non-overlapped, with CIFAR-10. The main reason of these observations is that SVHN and Texture belong to domains further away from CIFAR-10. We also compare distributions of CIFAR-10/100 under different hyper-paramete settings, see Fig. in Appendix A. for details.
4.5 THE ABLATION STUDY OF TEMPERATURE AND NUMBER OF CLUSTERS
In the absence of training data labels, we can divide the training dataset into & clusters and evaluate whether the number of clusters & affect the performance. The left panel of Fig. shows that the model achieves good performance when the number of clusters is 1. The dataset (CIFAR- 100) has 10 classes, thus select and 10 number of clusters for temperature ablation. For
8

---

Under review as a conference paper at ICLR 2023
1 cluster case, the AUROC is maximised at 0.5 temperature and the lowest AUROC is 0.07 temperature.
AUROC under temperature=0.5
AUROC under different temperature values
1.00
CIFAR-10 vs. CIFAR-100
CITAR-100 (olusterm:1
0.95
CIFAR-10 vs. SVHN
0.95
SVIN (clusters::)
CIFAR-100 (clusters:10)
0.90
AUROC
0.85
0.85
0.80
0.75
0.70
0.75
0.65
2
8
9
10
0.07
0.2
0.5
06
Number of clusters
Hyper-parämeter temperature
Figure 5: The left panel shows AUROC trends with to 10 clusters and the right panel depicts the performance under different temperatures in single cluster scenario. CIFAR-10 is the ID set.
Table 3:
AUROC with different temperature values in self-supervised training. dataset is CIFAR-10, the OOD datasets are CIFAR-100 and SVHN respectively
The in in-distribution
Dataset
Number of clusters
Temperature (0.07)
(0.2)
(0.5)
(0.8)
1
65.05%
93.14%
96.48%
95.81
CIFAR-100
10
72.23%
73.69%
75.29%
76.98%
1
74.94%
98.51%
99,41
99.29%
SVHN
10
83.47%
81.38%
80.72%
84.66%
On the contrary, the AUROC on CIFAR-10 vs. SVHN the minimum at 0.5 temperature when cluster number equals 10, as shown Table The observed trend of accuracy is consistent to those in Fig. 55) In general, when we have one cluster and temperature is 0.5, the model gains the best performance.
5 CONCLUSIONS
In this paper, we proposed a new self-supervised OOD detector, the CVT model, which outper- formed competing OOD detectors in benchmark tests over three different datasets (CIFAR-10/100 and SVHN). In the CVT model, we developed an ensemble module, which not only balanced the performance on far and near OOD datasets, but also enhanced overall OOD detection performance. The CVT model achieved a notable 80.34% AUROC accuracy in the challenging unlabeled far- OOD detection task. Although the results were remarkable, there exists open issues and promising directions for future work. Firstly, the adopted datasets are well established and recognised, but rela- tively constrained. Images in CIFAR-10. CIFAR-10M and SVHN normally contain only object. Further evaluation of the proposed CVT model may requires more complicated and real datasets. Besides, with label information, supervised OOD detector may learn more features than unsuper- vised learning. Hence, the supervised OOD detector is worth to be further explored and we believe the ensemble module in CVT model will also benefit supervised OOD detectors. Finally, the CVT model has good potential to be integrated to other (non-OOD) self-supervised tasks, OOD detec- tion enabled CVT model may save time and improve reliability of the other OOD self-supervised model.
REFERENCES
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. Advances in Neural Information Processing Systems, 33:9912-9924, 2020.

---

Under review as a conference paper at ICLR 2023
Ting Chen, Simon Korbllihh, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. International conference on machine learning, pp. 1597-1607. PMLR, 2020.
Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9640-9649, 2021.
M. Cimpoi, S. Maji, I. Kokkinos, S. Mohamed, and A. Vedaldi. Describing textures in the wild. In Proceedings of the IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2014.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An im- age is worth 16x16 words: Transformers for image recognition at scale. In International Confer- ence on Learning Representations, 2020.
Stanislav Fort, Jie Ren, and Balaji Lakshminarayanan. Exploring the limits of out-of-distribution detection. Advances in Neural Information Processing Systems, 34:7068-7081, 2021.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach self-supervised learning. Advances in neural information processing systems, 33:21271-21288 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp. 9729-9738. 2020.
Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of out-of-distribution examples in neural networks. In International Conference on Learning Representations, 2016.
Victoria Hodge and Jim Austin. A survey of outlier detection methodologies. Artificial intelligence review, 22(2):85-126, 2004.
Nikos Komodakis and Spyros Gidaris. Unsupervised representation learning by predicting image rotations. In International Conference Learning Representations (ICLR), 2018.
Anna-Kathrin Kopetzki, Bertrand Charpentier, Daniel Zügnerr, Sandhya Giri, and Stephan Günnemann. Evaluating robustness of predictive uncertainty estimation: Are dirichlet-based models reliable? In International Conference on Machine Learning, pp. 5707-5718. PMLR, 2021.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Shiyu Liang Yixuan Li, and Srikant. Principled detection of out- tiatof-dditributtio examples in neural networks. arXiv preprint 1mXiv:1766.02690 pp. 655 655-662,44 2017.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 10012-10022, 2021.
Ilya Loshchilov and Frank Hutter. Fixing weight decay regularization in adam, 2018. URL https:
//openzevies http://wwwwweeetttooeeererrrorea
Yifei Ming, Yiyou Sun, Ousmane Dia, and Yixuan Li. Cider: Exploiting hyperspherical embeddings for out-of-distribution detection. arXiv preprint arXiv:2203.044550 2022.
Eric T Nalisnick, Akihiro Matsukawa, Yee Whye Teh, and Balaji Lakshminarayanan. Detecting
out-of-distribution inputs to deep generative models using test for typicality. 2019.
10

---

Under review as a conference paper at ICLR 2023
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. 2011.
Anh Nguyen, Jason Yosinski, and Jeff Clune. Deep neural networks are easily fooled: High confi-
dence predictions for unrecognizable images. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 427-436, 2015.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv preprint 1xXiv:/807.03746 2018.
Stanislav Pidhorskyi, Ranya Almohsen, and Gianfranco Doretto. Generative probabilistic novelty detection with adversarial autoencoders. Advances in neural information processing systems, 31, 2018.
Jie Ren, Stanislav Fort, Jeremiah Liu, Abhijit Guha Roy, Shreyas Padhy, and Balaji Lakshmi- narayanan. simple fix to mahalanobis distance for improving near-ood detection. arXiv preprint wrXiv:2106.09022, 2021.
Lukas Ruff, Robert Vandermeulen, Nico Goernitz, Lucas Deecke, Shoaib Ahmed Siddiqui, Alexan- der Binder, Emmanuel Müller, and Marius Kloft. Deep one-class classification. In International PMLR, 2018.
conference on machine learning, pp.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 1153))2112252,22 2015.
Tim Salimans, Andrej Karpathy, Xi Chen, and Diederik P Kingma. Pixelcnn++: Improving the pixelenn with discretized logistic mixture likelihood and other modifications. arXiv preprint VrXivv/701.055/// 2017.
Vikash Sehwag, Mung Chiang, and Prateek Mittal. Ssd: A unified framework for self-supervised outlier detection. In International Conference on Learning Representations, 2020.
Joan Serrà, David Álvarez, Vicenç Gómez, Olga Slizovskaia, José F Núñez, and Jordi Luque. In- put complexity and -itto--distribtionn detection with likelihood- generative models. arXiv preprint arXiv: 1480, 2019.
Jihoon Tack, Sangwoo Mo, Jongheon Jeong, and Jinwoo Shin. Csi: Novelty detection via contrastive learning distributionally shifted instances. Advances in neural information processing systems, 33:1 18:11839-11 1852. 2020.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International Conference Machine Learning, pp. 10347-10357. PMLR, 2021.
Jim Winkens, Rudy Bunel, Abhijit Guha Roy, Robert Stanforth, Vivek Natarajan, Joseph R Ledsam, Patricia MacWilliams, Pushmeet Kohli, Alan Karthikesalingam, Simon Kohl, et al. Contrastive training for improved out-of-distribution detection. arXiv preprint arXiv:2007.05566. 2020.
Zhisheng Xiao, Qing Yan, and Yali Amit. Likelihood regret: An out-of-distribution detection score for variational auto-encoder. Advances in neural information processing systems, 33:20685- 20696, 2020.
Jingkang Yang, Kaiyang Zhou, Yixuan Li, and Ziwei Liu. Generalized out-of-distribution detection: A survey, arXiv preprint arXiv:2110.7133,, 2021.
11

---

Under review as a conference paper at ICLR 2023
A APPENDIX
A.1 ABLATION
A.1.1 THE INFLUENCE TEMPERATURE AND NUMBER CLUSTERS
We visualize the features of ID and OOD datasets under different temperature parameters in Fig. 6 Here the ID dataset is CIFAR- 10, and the OOD datascts CIFAR 100 and SVHN respectively. When observing the divergence become more cohesive when the temperature is 0.5. From the pictures CIFAR-10 temperature 0.07, the features from SVHN are farther from features from CIFAR-10 than features from CIFAR-100. Hence, temperature controls the distance among each sample in the feature space, while the contrastive loss makes the samples from the same class close and repel others in the feature space. low temperature will spread features, and the features will be evenly distributed when is small enough. Although large temperature can make features from the same class more cohesive, also shortens the distance between different classes.
CIFAR- 10 under 0.07 and 0.5 temperatures, we find the clusters SVHN
temperature-0..7
temperature=0.5
OFARID
SVIN
OFARRO
CFAR10
OFAR100
CIARROO
Figure 6: Visualization of features under different temperature parameters. The temperature of the
two pictures the left is 0.07, and the pictures on the right are 0.5.
12

---

Under review as a conference paper at ICLR 2023
A.1.2 DIMENSION OF FEATURE EMBEDDINGS
To discover how many dimensions of feature embeddings are suitable for OOD score calculation, we experiment with different dimensions to compute the OOD score. Here, we use the features from predictor and encoder respectively calculate metries. As shown in Table 4 the AUROC begins with the lowest value at 128 dimensions on CIFAR-100 and SVHN OOD datasets when using features from predictor. becomes stable when the dimension larger than 256.
Table AUROC with different dimension values self-supervised training and the in in-distribution dataset is CIFAR-10.
OOD Dataset
Feature type
Dimensions (128)
(256)
(512)
(768)
Predictor features
95.05%
96.48%
96.61%
96.56%
CIFAR-100
Encoder features
93.59%
95.47%
94.07%
94.89%
Predictor features
99%
99.41%
99.42%
99.62%
SVHN
Encoder features
99.83%
99.9%
99.94%
99.95%
A.2 THE IMPACT OF DATA AUGMENTATION ON OOD DETECTION
Considering that data augmentation with a few specific techniques is applied to ID dataset for gen- erating positive pairs in contrastive learning, it is valuable explore how the data augmentation impact OOD score, especially for the ID test set. In training process of our method, the Random- ResizedCrop in torchvision and the GaussianBlur Pillow are utilised for data augmentation. The scale for cropping image is from 0.08 to 1, which is denoted by Crop(0.08, 1), whilst the radius of Gaussian kernel GaussianBlur from 0.1 2. which denoted by GaussianBlur(O.1, 2). To explore whether the augmented samples from ID test set will still be recognised as ID ones, new pa- rameters different from those in the training are for these two techniques generate augmented ID test set, and then the trained model is used to detect these new augmented samples. The results shown in Table demonstrates that data augmentation has almost no impacts on OOD score for ID samples.
Table 5: The impact of data augmentation on OOD detection. This toy experiment is conducted on two scenarios, where CIFAR- 10 and CIFAR-100 100 are used as dataset respectively, and FPR95 is selected the metric, where higher FPR95 is, the less impact data augmentation has OOD detection.
Augmented ID test set
ID dataset
Crop(O.03, 0.08)
GaussianBlur(2.3 3) Crop(0.03, 0.08) GaussianBlur(2, 3)
CIFAR-10
99.98%
99.76%
99.98%
CIFAR-100
99.99%
99.22%
99.98%
13


Reviews:
[{"review_id": "kqW8bGUgn77", "reviewer": "Reviewer_YUrT", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes to use a contrastive method that uses ViT models as encoders for out-of-distribution detection. Different from previous works, this paper uses contrastive methods with Vision Transformers encoders instead of convolutional ones. Different representations obtained by the model seem to be more appropriate to different settings (near vs far OOD) and they are combined. The resulting method obtains good results on OOD detection datasets.", "summary_of_the_review": "The paper proposes to use a contrastive learning based on ViT models for OOD detection. It results in a good method, but it needs to be evaluated more rigorously. ", "strength_and_weaknesses": "Strong Points:\n\nHaving a strong encoder is very relevant for OOD detection, thus having a contrastive method that works with ViT model is beneficial.\n\nWeak Points:\n\nSince the contribution of this paper is empirical, coming from replacing a convolutional model with a stronger, Transformer model, the evaluation should be more rigorous. At the moment only CIFAR10-100 and SVHN are used to form IN / OOD pairs. The setups from CSI and [A] should also be evaluated.\n\nThe claim that Encoder features and Predictor features are more appropriate in different settings is based on the experiments in Table 3. But these are not conclusive, as Encoder features are better in 3 of the 4 cases so it cannot be said that they are better for semantic or covariate shifts. \n\nMultiple seeds should be used when computing the results and confidence intervals should be given.\n\n[A] Sun et al. “Out-of-Distribution Detection with Deep Nearest Neighbors” ICML 2022\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "3ct4veX2mWi", "reviewer": "Reviewer_nmrp", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose an attention-based contrastive learning model (CVT)  for OOD detection in image classification tasks based on self-supervised methods by using a ViT as a feature extractor.  The proposed model outperforms several SOTA methods based on CIFAR-10/100.", "summary_of_the_review": "The authors propose a clear, relatively straightforward approach to identifying OOD data using a supervised framework.  Overall the writing is clear, but a few improvements could be made.  The impact is limited by the focus on small toy datasets (CIFAR 10/100, SVHN) as opposed to more real world datasets where such approaches would be needed. \n\n The results when compared against a binary ID/OOD task are much better, but this doesn't feel like a fair comparison since such information would never be known in advance; when comparing to the more standard task (Table 2) the results are still quite good, but not better than Fort et al.  Additional discussion needs to be provided to this key work.  The paper can still be accepted even if it does not beat Fort et al., but it can't be overlooked as it is currently since this is SOTA (not the approaches in Table 1, as the authors claim).", "strength_and_weaknesses": "Strengths:\nIdentifying OOD data is an important topic.  The ability to identify OOD data using the strengths of SSL is a valuable endeavor.\n\nWeaknesses:\nI think it would be valuable to see how the choice/amount of augmentation impacts OOD score.  For example, if a sample is augmented with new/significant noise, will it be declared out of domain?\n\nIt's not clear to me from 3.3 and 4.1/2 how the other models in Table 1 were trained.  If they were just trained to predict ID vs. OOD, this doesn't seem like a realistic task since that will never be known in the real world.  Or were the models trained to carry out the normal prediction task and only if the overall class likelihood was near chance, was it declared OOD?   And in contrast is the proposed model scored using the new metric defined in (4)?\n\nThe authors reference VAEs as another common approach which has been successful in the past, but that is not explored here.  Why?\n\nCIFAR is a small dataset (i.e. the images are small) so the queue size can be quite large on a \"standard\" machine.  If the batch size has to be reduced because \"normal\" sized images are used (512x512 or much larger), how will this impact performance and/or training/inference feasibility?\n\nWhy is the approach of Fort et al. not shown in Table 1? \n\nThe authors claim better than SOTA, but this isn't quite true.  The binary task isn't really \"fair\" at some level because ID vs. OOD is not known in advance.  The best comparison seems to be with Fort et al., but all related work and discussion around this is missing.  This omission feels intentional, but the authors need to discuss the strengths/weaknesses to each.\n\nComments:\nThe authors state a number of factors generating outliers.  In the context of self supervision, in particular, often \"outliers\" arise because no massive dataset exists for the task of interest.  The pretraining dataset may be a large public dataset like imagenet.  The learned representation is then transferred or fine-tuned to the domain/task of interest.  Can such an approach still be used in those scenarios.\n\nWhat impact does augmentation procedure have on the ability to identify OOD samples?\n\nHow dependent is this approach on the specific constrastive learning approach selected?\n\nThe authors state three metrics are used in 3.3, but tables 1 and 2 only focus on AUROC.  What do the other metrics reveal?\n\nFigure 4 seems confusing.  For as clean as the embedding clusters are on the right, there is still fairly significant overlap of the distributions on the left.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "-xYm8_ygwGg", "reviewer": "Reviewer_RtUc", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed a contrastive vision transformer for self-supervised OOD detection. An ensemble module is developed to build representative ensemble features for achieving the balance between semantic and covariate OOD detection. The experimental results show the effectiveness of the proposed method.", "summary_of_the_review": "The novelty and the contributions of this paper are limited as indicated in the weaknesses. Thus, I tend to give a negative rating.", "strength_and_weaknesses": "## Strength\n### An ensemble module is developed to build representative ensemble features for achieving the balance between semantic and covariate OOD detection.\n\n## Weaknesses\n(1) The architectures and the modules in the proposed method are not novel. The four extra modules (except for the ensemble module) are mostly existing modules from previous methods, such as MOCO, MOCO v3, BYOL, et. al.\n(2) The contributions of this paper are not significant. The major contribution is apply the contrastive learning architecture for OOD detection.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject