Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes to use a contrastive method that uses ViT models as encoders for out-of-distribution detection. Different from previous works, this paper uses contrastive methods with Vision Transformers encoders instead of convolutional ones. Different representations obtained by the model seem to be more appropriate to different settings (near vs far OOD) and they are combined. The resulting method obtains good results on OOD detection datasets.
strength_and_weaknesses: Strong Points:

Having a strong encoder is very relevant for OOD detection, thus having a contrastive method that works with ViT model is beneficial.

Weak Points:

Since the contribution of this paper is empirical, coming from replacing a convolutional model with a stronger, Transformer model, the evaluation should be more rigorous. At the moment only CIFAR10-100 and SVHN are used to form IN / OOD pairs. The setups from CSI and [A] should also be evaluated.

The claim that Encoder features and Predictor features are more appropriate in different settings is based on the experiments in Table 3. But these are not conclusive, as Encoder features are better in 3 of the 4 cases so it cannot be said that they are better for semantic or covariate shifts. 

Multiple seeds should be used when computing the results and confidence intervals should be given.

[A] Sun et al. “Out-of-Distribution Detection with Deep Nearest Neighbors” ICML 2022

clarity,_quality,_novelty_and_reproducibility: The paper can be reproduced. The originality of the work is low. Given that the paper repurpose existing methods with different encoders, the quality should come from rigorous evaluations, but the paper still needs to improve in this regard.
summary_of_the_review: The paper proposes to use a contrastive learning based on ViT models for OOD detection. It results in a good method, but it needs to be evaluated more rigorously.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors propose an attention-based contrastive learning model (CVT)  for OOD detection in image classification tasks based on self-supervised methods by using a ViT as a feature extractor.  The proposed model outperforms several SOTA methods based on CIFAR-10/100.
strength_and_weaknesses: Strengths:
Identifying OOD data is an important topic.  The ability to identify OOD data using the strengths of SSL is a valuable endeavor.

Weaknesses:
I think it would be valuable to see how the choice/amount of augmentation impacts OOD score.  For example, if a sample is augmented with new/significant noise, will it be declared out of domain?

It's not clear to me from 3.3 and 4.1/2 how the other models in Table 1 were trained.  If they were just trained to predict ID vs. OOD, this doesn't seem like a realistic task since that will never be known in the real world.  Or were the models trained to carry out the normal prediction task and only if the overall class likelihood was near chance, was it declared OOD?   And in contrast is the proposed model scored using the new metric defined in (4)?

The authors reference VAEs as another common approach which has been successful in the past, but that is not explored here.  Why?

CIFAR is a small dataset (i.e. the images are small) so the queue size can be quite large on a "standard" machine.  If the batch size has to be reduced because "normal" sized images are used (512x512 or much larger), how will this impact performance and/or training/inference feasibility?

Why is the approach of Fort et al. not shown in Table 1? 

The authors claim better than SOTA, but this isn't quite true.  The binary task isn't really "fair" at some level because ID vs. OOD is not known in advance.  The best comparison seems to be with Fort et al., but all related work and discussion around this is missing.  This omission feels intentional, but the authors need to discuss the strengths/weaknesses to each.

Comments:
The authors state a number of factors generating outliers.  In the context of self supervision, in particular, often "outliers" arise because no massive dataset exists for the task of interest.  The pretraining dataset may be a large public dataset like imagenet.  The learned representation is then transferred or fine-tuned to the domain/task of interest.  Can such an approach still be used in those scenarios.

What impact does augmentation procedure have on the ability to identify OOD samples?

How dependent is this approach on the specific constrastive learning approach selected?

The authors state three metrics are used in 3.3, but tables 1 and 2 only focus on AUROC.  What do the other metrics reveal?

Figure 4 seems confusing.  For as clean as the embedding clusters are on the right, there is still fairly significant overlap of the distributions on the left.
clarity,_quality,_novelty_and_reproducibility: It's not clear to me how the other models in Table 1 were trained.  Were these models just trained as binary classifiers to predict ID vs. OOD?  How the metrics were defined for both the alternate and proposed approach in this table is unclear.

Fort et al. is shown to be significantly better than the proposed approach, but this approach is never discussed anywhere- attention needs to be given to this in the related works.
summary_of_the_review: The authors propose a clear, relatively straightforward approach to identifying OOD data using a supervised framework.  Overall the writing is clear, but a few improvements could be made.  The impact is limited by the focus on small toy datasets (CIFAR 10/100, SVHN) as opposed to more real world datasets where such approaches would be needed. 

 The results when compared against a binary ID/OOD task are much better, but this doesn't feel like a fair comparison since such information would never be known in advance; when comparing to the more standard task (Table 2) the results are still quite good, but not better than Fort et al.  Additional discussion needs to be provided to this key work.  The paper can still be accepted even if it does not beat Fort et al., but it can't be overlooked as it is currently since this is SOTA (not the approaches in Table 1, as the authors claim).
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposed a contrastive vision transformer for self-supervised OOD detection. An ensemble module is developed to build representative ensemble features for achieving the balance between semantic and covariate OOD detection. The experimental results show the effectiveness of the proposed method.
strength_and_weaknesses: ## Strength
### An ensemble module is developed to build representative ensemble features for achieving the balance between semantic and covariate OOD detection.

## Weaknesses
(1) The architectures and the modules in the proposed method are not novel. The four extra modules (except for the ensemble module) are mostly existing modules from previous methods, such as MOCO, MOCO v3, BYOL, et. al.
(2) The contributions of this paper are not significant. The major contribution is apply the contrastive learning architecture for OOD detection.
clarity,_quality,_novelty_and_reproducibility: Clarity: good

Quality: fair

Novelty: poor
summary_of_the_review: The novelty and the contributions of this paper are limited as indicated in the weaknesses. Thus, I tend to give a negative rating.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposes contrastive vision transformer for self-supervised OOD detection. While the paper has some interesting contributions, the main concerns from the reviewers were the limited novelty of the proposed approach and the limited empirical evaluations. Overall, the current version falls below the acceptance threshold. I encourage the authors to revise and resubmit to a different venue.
justification_for_why_not_higher_score: Limited Novelty
Limited Empirical evaluation
Decision: Reject