Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper presents an out-of-distribution(OOD) detection method for the image classification models based on the CLIP model. The authors propose Maximum Concept Matching which detects OOD by aligning visual features with textual concepts. Experiments show that this simple yet effective method can outperform many methods. Since the method does not need any training, it can be easily applied to different datasets with minimum effort.
strengths_and_weaknesses: Strengths:
- The method is simple and can work well without bells and whistles.
    - There're four compelling advantages: generalizable to many tasks; OOD-agnostic; training-free; scalable.
- The authors justify their design with theoretical proof.

Weaknesses:
- Although the definition of MCM in eq. 1 is intuitive, there could be many ways to detect uniformity of the cosine similarities. For example, entropy after softmax scaling, the variance of the cosine similarities, exp(max - second_max). More options should be compared in the experiments.
- It will be great if the authors can try more different CLIP models like resnet CLIP, and some community open sourced CLIP-like models. Would it be possible that it is actually the vision transformer or the training data of CLIP to be the root cause of strong OOD detection performance?
- Wrong fact: the authors claim that CLIP uses a temperature of 0.07. However, the temperature of CLIP is initialized as 0.07 and learned during training.

Final rating:
The authors' rebuttal addresses my questions very well. I will keep my original rating.
questions:  
limitations: No limitations are discussed by the authors.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposed a new out-of-distribution (OOD) detection method based on pretrained vision-language model. Unlike previous works, this work relies on pretrained models like CLIP to infer whether a given image is in or out of the target distribution by matching the visual features to a set of concept embeddings. To mitigate the score overlap between in-domain and out-of-domain data, the authors proposed a simple but effective softmax-scaling to recalibrate the matching scores between images and texts. According to the experimental results, the proposed method called Maximum Concept Matching (MCM) achieves superior performance to previous works on a variety of benchmarks and settings. It does not need any finetuning but can be directly applied to different datasets and label spaces in a zero-shot manner thanks to the zero-shot capacity learned by vision-langauge models. 
strengths_and_weaknesses: [Strenghts]

1. This paper introduced a simple yet effective OOD detection method called MCM based on multi-modal pre-trained models. This is one of the first works that leverage vision-language multi-modal models for OOD, in contrast to previous single-modal methods.

2. Based on the vision-language pretrained model like CLIP, the authors further proposed a simple softmax scaling to re-calibrate the matching scores to magnify the difference between in-distribution and out-of-distribution samples. 

3. Comprehensive experiments demonstrate that the proposed MCM outperforms previous work proposed by Fort et al. and MSP. The superiority is universal on both small-scale datasets and large-scale ones line ImageNet-1K. The authors further demonstrated that MCM can benefit hard OOD detection very well. 

[Weaknesses]

1. The technical contribution in this paper is limited. First of all, using softmax on the classification logits is natural and straightforward as shown in MSP and other works. Despite the authors gave proof that softmax scaling can improve the OOD detection when using clip-based models, I think this change is incremental and should be considered as a baseline method.

2. The proposed MCM is one of the first works that leveraged vision-language pretrained models. However, I do think the claims made in lines 155-168 are exaggerated. Most of the claimed merits for MCM are due to vision-language pretrained CLIP, such as generalization ability, training-free and scalable.

3. In Fig.4, the authors showed the effect of softmax scaling and the corresponding FPR at different settings. For completeness, the authors should report the results on all used datasets for the proposed method without softmax scaling. 

4. The authors merely reported the performance on zero-shot settings, it would be better to also report the performance after finetuning. It is not clear whether after finetuning the OOD detection performance will be further improved or not. 

5. It is not clear in the paper why the vision-language model achieves much better zero-shot OOD detection than the single-modal method. The authors comapred with Maha in lines 289-307. However, why text features as the prototypes are supposed to be better than visual features as the prototypes? 
questions: Questions:

1. What is the performance after finetuning CLIP models for OOD detection? It would be better to show the performance after finetuning on downstream datasets.

2. It is still not clear to me why vision-language pretrained model with simple softmax scaling can outperform previous work significantly. There is not much explanation in the paper about this.

3. The authors should report the performance without softmax scaling for all datasets and evaluations. The role of softmax scaling is not clear to me.

Suggestions:

1. I do not think the claimed contributions in lines 155-168 are from the proposed MCM method. It mainly inherits from the off-the-shelf CLIP models. All methods that use vision and language encoders in CLIP are supposed to possess such properties.

2. The authors should have more discussions about the contribution of proposing softmax scaling, which I think is an incremental and normal technique.


limitations: The authors did not discuss the limitations and societal impact in their submission.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors proposed Maximum Concept Matching (MCM), a distance-based zero-shot OOD detection method based on aligning visual features with textual concepts, and provided theoretical justifications that softmax scaling used by MCM improves the separability between ID and OOD data for CLIP-based OOD detection. The experiments demonstrate that MCM achieves superior performance on a wide variety of real-world tasks. MCM with vision-language (multi-modal) features outperforms Mahalanobis with visual features only (single-modal) on a hard OOD task. MCM offers several advantages: OOD-agnostic (no information required from OOD data), training-free (no downstream fine-tuning required), and generalizable (one model supports many tasks).
strengths_and_weaknesses: Strengths

Originality: the authors adopted softmax scaling, which improves the separability between ID and OOD data.

Quality: MCM achieves superior performance on a wide variety of real-world tasks, compared with the state-of-the-art OOD detection methods. MCM with vision-language (multi-modal) features outperforms Mahalanobis with visual features only (single-modal) on a hard OOD task, with an improvement of 56.60% on FPR95.

Clarity: the manuscript is well-organized and the text is clear and concise.

Significance: The use of the powerful CLIP representation and softmax scaling, enables zero-shot OOD detection. More important, MCM is OOD-agnostic and generalizes to many tasks, making it suitable for real-world applications, where we don’t know OOD labels in prior.


Weaknesses

Originality: the authors proposed MCM inspired by concept match which has been adopted in previous OOD detection work (Mahalanobis, which defines class prototypes based on pure visual embeddings). And multi-modal OOD detection has been practiced in previous OOD detection works yet.
questions: Can MCM be applicable for non-contrastive vision-language pre-training models?
limitations: Ablation study on softmax scaling should be given.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper presents an OOD detection approach termed MCM based on vision-language pretrained models, by formatting the OOD detection as a distance-based matching problem. Specifically, they take the advantage of CLIP for distinguishing OOD data from known classes by cross-modal matching, which establishes a powerful cross-modal subspace for metric learning. The author gives some theoretical and empirical evidence to explain the superiority of MCM over cross-entropy-based models.
strengths_and_weaknesses: Strengths:
* The paper is well written and mostly clear.
* The idea of the maximum matching score is simple but effective and shows promising results compared with some baselines.
* The theoretical explanation of softmax scaling seems reasonable and insightful.

Weaknesses:
* Missing comparisons with recent works. The compared works in Table 2 all adopt a Transformer trained on large datasets, a large portion of CNN-based methods is missing, e.g. GradNorm[R1], Energy[R2]. Note that when replacing ResNet with CLIP,  the simple baseline MSP could boost from 63.69/87.59 (reported in [R1]) to 26.65/95.25 on FPR95/AUROC on iNaturalist. I am worried that previous CNN-based methods may also benefit from a strong backbone significantly. The paper lacks a fair comparison with recent CNN-based SOTA methods (MSP and Mahalanobis are somewhat old) using a similar backbone.

* Prompt ensemble. For the same datasets, I guess different prompts may produce unstable performance and thus ensemble works well. I wonder whether the authors have to design new prompts for new datasets. 

[R1] On the Importance of Gradients for Detecting Distributional Shifts in the Wild, NIPS 2021.

[R2] Energy-based out-of-distribution detection, NIPS 2020.

questions: 1. In Table 2, why do Fort et al. (ViT-B) and MSP (CLIP-B) have the same performance on all metrics? Also, some values should not be bold.
2. What about the variance of the experiments?
limitations: The paper did not describe limitations.
review_title: null

Meta-review: metareview: This paper presents an interesting and novel try at using vision-language multi-modal models for OOD tasks. The experiments are sufficiently validated on diverse OOD datasets. Besides, the theoretical explanations of softmax scaling are quite insightful. All reviewers give positive scores. During the discussion phase, the authors have accordingly added more ablation studies and comparisons. Despite some reviewers raising concerns about the possible limited novelty of using softmax scaling for the CLIP-like model. The authors give sufficient discussions and provide theoretical justifications, which will inspire the community. The meta-reviewers thus recommend accepting it.
confidence: Certain
Decision: Accept