Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper studies the few-shot outlier detection setting where a few examples from outlier classes are provided to the ODD detector.  The main motivation behind this work are situations that require high level of ODD detection. It is shown that classifiers trained by fine-tuning large-scale pre-trained transformers are significantly better at near-ODD detection. Simulations are performed on well-known data sets for visual classification and genomic sequences. Performance evaluation are reported in term of AUROC scores. However, it would have been more convenient to report results in terms of FPR at 95% scores which allow to better understand if the performance gains with respect to similar methods are relevant. The reported gains are not very significant but consistent. 

main_review: The main limitation of this work relies on the assumption that out-of-distribution samples can be labelled and be available to train the detector. I strongly believe this is not the case in many of practical scenarios where these samples evolve over time and this cannot be given in advance to the learner. On the other hand, the paper lacks of novelty in terms of exceptions for a NeurIPS work. The proposed method relies on the addition of the soft-probabilities of the concatenated  predictions for in and out distributions. Finally, it is not clear how much this method improve on state-of-the-art methods since most of the existent literature on ODD detection does not assume of the presence of ODD samples from the same distribution than for testing. This also brings the question of how much the present method degrades  if the available ODD samples do not correspond to the same testing distributions. Additional simulations would have been beneficial to clarify this issue. Finally, I believe that FPR at 95% scores are much more relevant than  AUROC scores which have not been reported. 
limitations_and_societal_impact: I do not think the authors addressed the main limitation related to the impossibility of disposing of the ODD samples corresponding to the testing distribution. As for the societal impact, the investigated topic (e.g., ODD detection) is very relevant and can have significant impact on real-word systems.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: * The paper presents detailed experiments which show that pre-trained large scale transformers significantly improves OOD detection performance.
* The paper especially focuses on the case where in-distribution (InD) and OOD samples are close to each other (near OOD) (e.g. when CIFAR100 is InD and CIFAR10 is OOD, SoTA AUROC is improved from 85% to 96% ).
* It is demonstrated that transformers can yield better well-separated representations for each class on InD dataset compared to ResNet without even fine tuning on InD samples. Separation of the representations become even better after fine-tuning in InD dataset (see Fig. 1). This motivates that transformers can be better for OOD detection compared to ResNet-like architectures and improves the SoTA.
* The paper presents experiments in different settings on multiple datasets:
* First, OOD detection performance between CIFAR10 <-> CIFAR100 is evaluated. The results demonstrate that fine-tuning transformers significantly improves SoTA OOD detection performance. Additional experiments are presented where it is assumed that a few examples from each class of target OOD dataset is available (called outlier exposure). This leads to further boost in OOD detection performance.
* The second experiments are performed on a genomics dataset where InD dataset consist of 10 known bacteria classes and OOD dataset consist of another 60 bacteria classes which are not seen during training. The results are consistent with the CIFAR10-CIFAR100 experiments. 
* Finally, experiments on multimodal text-image models are presented where candidate labels are used for OOD detection. The results show that it leads to significant performance improvement.
main_review: **Strengths**
- The paper presents extensive performance evaluation existing OOD detection methods on transformers architectures. The results presented in the paper demonstrate significant improvement over SoTA in challenging near OOD detection tasks. 
- Despite of the fact that the paper doesn't propose a novel algorithm, the presented experiments are quite extensive.
- Transformers are popular architectures which achieve SoTA performance in variety of tasks. The paper demonstrate that they not only achieve high prediction performance, but also achieves significant OOD detection performance which makes it an ideal choice for safety-critical applications.

**Weaknesses**
I don't see any major weakness of the paper. There are just a few points that needs a bit clarification for me and a few suggestions for improvements.
- According to Fig. 1, ViT achieves perfect separation of classes without even fine-tuned on InD dataset. This is quite surprising to me because this mean that transformers can identify images from same classes without seeing any training examples. Is it because ViT is pre-trained on very large datasets? I think some intuition about how transformers can achieve this would be quite useful.
- In the outlier exposure experiments, a few examples from each class of target OOD are used to improve the OOD detection performance. However, target OOD dataset is usually unknown in real applications. So, although teams are willing to annotate some, they may not overlap with the target OOD dataset. For example, let's consider the scenario where CIFAR10 is InD, sample OOD data available for outlier exposure is CIFAR100, and OOD samples given to the model in test time is SVHN. Do the model assign SVHN samples to one of the InD or OOD classes?
- One generic way of performing outlier exposure could be using adversarial samples as OOD and testing how they generalise to various unseen OOD datasets.
- In lines 278-279, the paper mentions that the best model is checkpointed based on best validation AUROC. However, the best model is usually checkpointed based on the best prediction accuracy rather than the best OOD detection performance. How would the results change if the models were saved based on the best prediction accuracy? Do this affect the OOD detection performance significantly?
limitations_and_societal_impact: - Please see the weaknesses section for some limitations of the paper.
- I don't see any potential negative social impact.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper experimentally examines the effect of using pre-trained (transformer) models for OOD detection and reports the following three results: i) pre-trained transformers lead to significant improvements on near-OOD benchmarks; ii) pre-trained transformers are well-suited for few-shot outlier exposure; iii) exploring pre-trained multi-modal image-text transformers in the zero-shot classification setting, just using the names of outlier classes as candidate text labels for CLIP can achieve good OOD detection performance.
main_review: I think this is a good paper that contains a lot of useful experimental results. But, on the other hand, I feel that its academic impact is slightly insufficient for a NeurIPS paper. 

The three results can be summarized by the effectiveness of pretrained Transformer models for OOD detection. However, the effectiveness of pretraining for OOD detection is already known, as shown in, e.g., the following paper. (It is a problem that it is not cited.) 

Hendrycks et al., Using Pre-Training Can Improve Model Robustness and Uncertainty, ICML2019

The paper’s claim (i) is that the two, i.e., pretraining + transformer, are important for improved performance. However, is it true? I suspect that not the transformer model but pretraining (on ImageNet21k) is the essence. Thus, even ResNet/EfficeinetNet models would show similarly good performance if they are pretrained on ImageNet21k and achieve similar ID classification accuracy. Table 1 shows comparisons with BiT-M R50x1, but they may not be fair since the compared models differ in the ID classification accuracy. Thus, the paper lacks sufficient evidence for the above claim. If my guess is true (i.e., the transformer is not necessary), then the results reported in this paper may not be so novel.

Regarding claim (ii), the method of few-shot outlier exposure is already known. It is unclear how novel it is to use transformer models for the backbone network. 

Regarding claim (iii), the proposed method may be regarded as a new CLIP application but remains one of many of its applications. 

In addition, one may question if we can still call it OOD detection even when we use pretrained models, especially those trained on a dataset sharing its classes with the OOD dataset. For instance, I suspect that most (or even all?) of CIFAR100 classes are the ImageNet21k classes. Can we call it OOD if an input belongs to a class learned in the pretraining?
limitations_and_societal_impact: If my concerns above are not resolved, they will be limitations of this study.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper finds and demonstrates that large-scale pre-trained transformers can significantly improve the state-of-the-art  on a range of near OOD tasks across different data modalities. The main contribution is that they find an efficient way to apply pre-trained transformer to OOD tasks and boost the performance significantly.
main_review: Though not surprised about this finding and conclusion, it is impressive that the authors provide detailed study and experiments on different comparisons. The paper is easy to read and follow, this would provide an important method to deal with OOD based on pretrained transformers. Also, beyond OOD, this paper could also bring some insights about how powerful the pretrained transformer will be for various vision tasks.

Strength:
I like the analytical experiments in the paper, such as Figure 1. The figure is very clear and easy to understand.
The paper is well-written and easy to follow.
The proposed method is simple but effective and could bring some insights for future research.

Concerns:
In line 123, the authors mention that `Self-supervised learning techniques have also been shown to improve OOD robustness`, what’s the underlying difference between the self-supervised methods and pretrained model. Does large-scale data-driven pretrained model should be the main issue for OOD tasks? Is there some cases that pretrained strategy works worse than self-supervised learning techniques? If yes or no, could the authors explain it?

In line 134, the authors claim that `We found that we get better generalization and higher quality embeddings when we do not use data augmentation for fine-tuning.` I am curious about which data augmentation the authors used for the experiments, is there any explanation for this part?

In general the whole paper is very clear and easy to follow. While I still have concerns regarding only conduct OOD experiments on CIFAR10 and a few relevant very small datasets with very low resolution ($32 \times 32$). Since the pretrained transformer is trained with large-scale datasets, so it is possible that the transformers already learn some related patterns. The improvement is significant, but wondering have the authors tried on other dataset with higher resolutions. If yes, how it goes with the pre-trained model? I feel like it might be harder to achieve a much higher performance on those dataset based on the pretrained model using current straight-forward strategy. I will change my scores based on the explanation of this part.

------------------- post rebuttal-----------------------
Thanks for the rebuttal, I am satisfied with the explanation, but still conservative about the potential usage of this method for the community, therefore, I keep my score unchanged.
limitations_and_societal_impact: Since the pretrained transformer is trained with large-scale dataset, while in this paper, the comparisons are primarily based on CIFAR10 and CIFAR100, which is too small to convince people thoroughly. It would be suggested that the authors could apply the same strategy to various OOD tasks and justify the efficiency and robustness of the proposed method.
review_title: null

Meta-review: The reviewers are in agreement that the paper results are clearly presented and thoroughly validated, and the takeaways on ViTs helping with (few-shot) OOD identification should be of broad interest to the community.
Decision: Accept (Poster)