GRIC: General Representation and Informative Content for Enhanced Out-of-Distribution Detection

Authors: Sima Behpour, Thang Doan, Xin Li, Wenbin He, Liang Gou, Liu Ren
Primary area: other topics in machine learning (i.e., none of the above)
Keywords: Out-of-Distribution Detection

Abstract:
Out-of-distribution (OOD) detection is crucial for ensuring the robustness of machine learning models in open-world scenarios by identifying inputs from unknown classes. Vision-language models like CLIP have enabled zero-shot OOD detection without requiring labels or training on in-distribution (ID) data. However, current approaches are limited by their dependence on \textit{closed-set text-based labels} and \textit{full image feature representations}, constraining CLIP’s capacity to generalize across diverse labels. In this work, we propose GRIC, a novel method that improves zero-shot multi-modal OOD detection by leveraging two key insights: (1) OOD detection is driven by general ID representations rather than class-specific features, and (2) large language models (LLMs) can enrich the model’s understanding of ID data and simulate potential OOD scenarios without actual OOD samples. GRIC is simple yet highly effective, reducing the false positive rate at $95\%$ recall (FPR95) by up to $19\%$, significantly surpassing state-of-the-art methods.

Decision: reject
Reviewer scores: 6, 6, 5, 6

Reviews and discussion:
<thread>
creation date:
1730108882502

thread_type:
official_review

<thread_object>
creator: reviewer_jgok

<content>
summary:
This paper introduces GRIC, a novel approach for zero-shot multi-modal OOD detection aimed at enhancing the robustness of machine learning models in open-world environments. Unlike existing methods that rely on closed-set text-based labels and complete image features, GRIC leverages general ID representations and LLMs to improve OOD detection. GRIC's approach rests on two main insights: (1) using general ID representations instead of class-specific features, and (2) enriching the model’s understanding with LLMs to simulate potential OOD scenarios. This method is straightforward yet effective.

strengths:
1. The paper is well-crafted and clearly presented, with an engaging motivation and good performance results.
2. Extensive experiments demonstrate the effectiveness of the proposed method.
3. The supplementary material provides useful experiments and details.

weaknesses:
1. The authors claim that "GRIC reduces the FPR95 by up to 19%, significantly surpassing SOTA methods." However, this statement is inaccurate. For instance, NegLabel [1], published a year ago, achieved an FPR95 of 25.40% on the ImageNet-1k benchmark, while the proposed method achieves 20.32%. Thus, the actual improvement is, at most, 5%.

2. I understand that it may be overkill to ask the authors to compare their methods with [2]. However, since [2] also utilizes superclasses for constructing prompts and achieves even higher performance (17.51% evaluated by FPR95), I consider it valuable for authors to add a discussion about the similarities and differences between their proposed method and [2]. If possible, [1] and [2] should be mentioned in the related work part and added to Table 2 to provide a more comprehensive comparison, which will not harm the unique contribution of this work.

[1] Negative Label Guided OOD Detection with Pretrained Vision-Language Models. ICLR, 2024.
[2] Conjugated Semantic Pool Improves OOD Detection with Pre-trained Vision-Language Models. NeurIPS, 2024.

3. If possible, the authors are recommended to provide more visualization results for deeper analysis.

4. There are multiple typos. It is recommended that the authors conduct a thorough review of the writing. For example, Line 110: G(x;Yin). L278: FOr this sake.

5. The paper has severe formatting weaknesses.

questions:
See Weakness
</content>
</thread_object>

</thread>
<thread>
creation date:
1730274380319

thread_type:
official_review

<thread_object>
creator: reviewer_v9pr

<content>
summary:
Out-of-distribution (OOD) detection is essential for enhancing the robustness of machine learning models in open-world environments by identifying inputs from unknown classes. While vision-language models like CLIP facilitate zero-shot OOD detection without the need for labels or training on in-distribution (ID) data, existing methods are constrained by their reliance on closed-set text-based labels and complete image feature representations, limiting CLIP's generalization capabilities. This work introduces GRIC, a novel approach that enhances zero-shot multi-modal OOD detection by focusing on general ID representations instead of class-specific features and utilizing large language models (LLMs) to enrich the understanding of ID data and simulate potential OOD scenarios without requiring actual OOD samples. GRIC demonstrates significant effectiveness, achieving a notable reduction in the false positive rate at recall (FPR95) and outperforming state-of-the-art methods.

strengths:
1. The  concept of general representation of ID data is novel to CLIP-based OOD detection.
2. The method is well designed with  various modules.
3. The extensive experiments show the effectiveness of the proposed method.

weaknesses:
1. One major issue with this paper is that it claims to be in a zero-shot OOD detection setting, but it should actually be classified as a few-shot setting. This is because the calculation of PCA requires the use of ID data, whereas in a zero-shot setting, ID images should be mixed with OOD images to form the test set, making them unavailable. The entire setting of the paper is flawed and needs to be revised

2. There are more state-of-the-art (SOTA) methods for zero-shot OOD detection that have not been compared, such as NegLabel [1], which demonstrates superior performance, and EOE [2], which also utilizes large language models (LLMs) for CLIP-based OOD detection.

3. The results in Table 1 are not representative, as the baseline MCM has already achieved a score of 99%, indicating that the OOD issue in this benchmark has been effectively addressed.

4. There are many more adjustable benchmarks that have not been explored, such as: hard OOD detection, robustness to domain shift and transfer the method to other CLIP-like models (ALIGN, AltCLIP, GroupViT)

[1] Jiang, X., Liu, F., Fang, Z., Chen, H., Liu, T., Zheng, F., & Han, B. Negative label guided ood detection with pretrained vision-language models. ICLR, 2024.
[2] Cao, C., Zhong, Z., Zhou, Z., Liu, Y., Liu, T., & Han, B. Envisioning Outlier Exposure by Large Language Models for Out-of-Distribution Detection. In Forty-first International Conference on Machine Learning.

questions:
1. The figures and algorithm seems screenshot and too ambiguous.
2. Many typos are in the paper and need to be revised. For  example,  'fPR95' in 412 is wrong spelling. When using "citation" as the subject, parentheses should not be added. Additionally, lines 493 and 494 overlap due to insufficient line spacing.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730447506077

thread_type:
official_review

<thread_object>
creator: reviewer_z3z1

<content>
summary:
This paper proposes a new enhancement method called GRIC for CLIP-based OOD detection. GRIC extracts general ID representations rather than class-specific features and introduces LLM-based informative prompts for OOD detection. Experimental results show the proposed GRIC outperforms existing methods.

strengths:
- GRIC surpasses the two baseline methods, MCM and GL-MCM.

weaknesses:
- There is a limited novelty. DICE [1] has a similar concept to drop unnecessary dimensions and shows the effectiveness for OOD detection.

- The motivation in this paper’s method—that class-specific information is unnecessary—raises some questions. In DICE, the motivation is to exclude signals that introduce noise.  Rather than removing information specific to the ID class, I consider this method actually exclude noise signals. Including the ID accuracy of GRIC without informative prompts in Table 6 would help clarify whether the information being removed is indeed ID class-specific.

- A recent challenge in OOD detection is accurately identifying "OOD images that are semantically similar to ID." In this problem setting, known as Hard OOD detection, certain classes within a dataset (e.g., ImageNet) are treated as ID, while other classes in the same dataset are treated as OOD. Therefore, I believe class-specific information is necessary rather than relying on the general representation of the dataset. I would like to see results on the effectiveness of this method when experimenting on Hard OOD detection benchmarks [2, 3].

- The approach is defined as a zero-shot method in L518. However, since it utilizes ID images for PCA processing, I consider this method to be a few-shot learning method, not a zero-shot. The definition of Zero-shot is not using ID images in preprocessing, regardless of whether training is involved [4].

- The code has not been shared, raising concerns about the reproducibility of the method.

[1] Sun+, DICE: Leveraging Sparsification for Out-of-Distribution Detection, ECCV2022. 

[2] Li+, Learning Transferable Negative Prompts for Out-of-Distribution Detection, CVPR2024. 

[3] Jung+, Enhancing Near OOD Detection in Prompt Learning: Maximum Gains, Minimal Costs, arXiv2024. 

[4] Miyai+, Generalized Out-of-Distribution Detection and Beyond in Vision Language Model Era: A Survey, arXiv2024.

questions:
I wonder about the motivation of this method that class-specific information is unnecessary.  To validate this statement, I would like to know the result of GRIC without informative prompts in Table 6 to clarify whether the information being removed is indeed ID class-specific, not a noisy signal.

Also, I would like to know the result of hard OOD detection.

For more details, please refer to the Weakness section.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730809538371

thread_type:
official_review

<thread_object>
creator: reviewer_dnku

<content>
summary:
The paper presents a method called GRIC (General Representation for Inference and Classification), designed to improve zero-shot and few-shot learning by leveraging representations from large-scale pre-trained models. GRIC integrates domain-specific knowledge into a unified embedding space that allows the model to transfer knowledge effectively across tasks and domains. The major contributions are the introduction of general ID features for OOD detection with hierarchical prompting.

strengths:
**1. Presentation**

The paper is generally well-presented and easy to follow. It begins with a clear hypothesis that using a generalized feature segment from the full feature space can improve ID/OOD sample distinction, followed by a systematic explanation of the proposed method for extracting this general feature space.

**2. Algorithm**

The algorithm is straightforward and effective, yielding significant performance gains on both small- and large-scale datasets. Ablation studies demonstrate that both the proposed general subspace extraction and hierarchical prompting contribute substantially to the performance improvements.

weaknesses:
**1. Formatting Issues**  
- **Text Accessibility**: The text is not selectable or OCR-scanned, which complicates review and readability.
- **Font and Equation Sizing**: Equations appear in a very small font, raising concerns about compliance with the `.sty` file specifications; table and figure fonts are also difficult to read.
- **Inconsistent Spacing**: Vertical spacing is uneven throughout, affecting readability. Additionally, Section 3.2 would be more appropriately placed in the related work section to improve structure.

**2. Missing Experiments and Analysis**  
While the paper presents a solid set of experiments across multiple datasets, further analysis would strengthen the justification of the approach:
   - **Single-Modality Vision Models**: The paper should demonstrate the effectiveness of general feature extraction in **vision-only models**, without hierarchical prompting, to show that the method generalizes beyond multi-modal settings.
   - **Integration with Other OOD Scoring Methods**: It would be valuable to evaluate GRIC with alternative OOD scoring metrics, such as **energy-based scores** and **feature-based scores**, to understand its compatibility with established scoring methods beyond MSP.
   - **ID Accuracy**: Given that real-world deployment typically involves handling both ID and OOD data, the paper should report ID accuracy to confirm that GRIC performs reliably on ID data without regression.

**3. Additional Ablation Studies**  
- **PCA Transformed Feature Space**: Examine the effectiveness of using PCA-transformed features (from \( R^{s \times r} \) to \( R^{s \times k} \), where \( k \) is the number of principal components) for OOD detection. 
- **Principal Component Masking**: Evaluate whether masking high-variance principal components in the PCA-transformed space, while using the remaining components, can improve OOD detection by focusing on features less affected by dominant ID patterns.
- **Full Feature Matrix for PCA**: Justify why the paper does not use the full feature matrix across all samples per class to compute PCA, as this could potentially improve the robustness of general feature extraction.
- **Hyperparameter Sensitivity**: Include a sensitivity analysis on the threshold in Equation 3, as this parameter may significantly influence detection performance.

Would re-consider the scoring based on authors' response.

questions:
Please refer to the weakness
</content>
</thread_object>

</thread>


Metareview:
metareview:
This work proposes a simple yet effective zero-shot multi-modal approach for OOD detection by leveraging general in-distribution (ID) representations with PCA and informative prompts generated by LLMs based on the super-class names of the ID data. The proposed method significantly enhances detection performance. Most reviewers agree that the paper is well-organized and easy to follow, recognizing the effectiveness of the approach in improving OOD detection by a large margin. However, reviewers raised concerns regarding the need for additional experiments and analyses, such as more baseline comparisons, few-shot evaluations, and ablation studies (DnKU, Z3Z1, V9Pr). Some reviewers also noted limited novelty, citing the existence of similar works (Z3Z1, V9Pr, jgok). Reviewers Z3Z1 and V9Pr specifically pointed out that the method should potentially be classified as few-shot detection due to the use of PCA. Following discussions, the authors successfully addressed some of the concerns, resulting in 3 borderline accept ratings (DnKU, Z3Z1, jgok) and 1 borderline reject (V9Pr), with an average score of 5.75. While Z3Z1 and jgok ultimately provided positive ratings, they recommended further refinements to the paper. Given the current feedback, the authors are encouraged to polish the manuscript to clearly articulate the benefits of the method’s training-free nature and highlight its differences from existing few-shot approaches. Additionally, it is recommended to include comparisons with other few-shot methods and expand baseline evaluations in Table 1, as suggested by reviewer V9Pr, for resubmission to a future venue.