Title: An Information-theoretical Framework for Understanding Out-of-distribution Detection with Pretrained Vision-Language Models
Venue: NeurIPS 2025
Accepted: yes
Keywords: Out-of-distribution Detection

Pretrained Vision-Language Models

Point-wise Mutual Information
Abstract: Out-of-distribution (OOD) detection, recognized for its ability to identify samples of unknown classes, provides solid advantages in ensuring the reliability of machine learning models. 
Among existing OOD detection methods, pre-trained vision-language models have emerged as powerful post-hoc OOD detectors by leveraging textual and visual information. 
Despite the empirical success, there still remains a lack of research on a formal understanding of their effectiveness. 
This paper bridges the gap by theoretically demonstrating that existing CLIP-based post-hoc methods effectively perform a stochastic estimation of the point-wise mutual information (PMI) between the input image and each in-distribution label. This estimation is then utilized to construct energy functions for modeling in-distribution distributions.
Different from prior methods that inherently consider PMI estimation as a whole task, we, motivated by the divide-and-conquer philosophy, decompose PMI estimation into multiple easier sub-tasks by applying the chain rule of PMI, which not only reduces the estimation complexity but also provably increases the estimation upper bound to reduce the underestimation bias. 
Extensive evaluations across mainstream benchmarks empirically manifest that our method establishes a new state-of-the-art in a variety of OOD detection setups.


Peer reviews:
comments: SUMMARY:
This paper introduces an information theory-based approach to enhance OOD detection using pretrained vision-language models such as CLIP. The method involves stochastic estimation of pointwise mutual information (PMI) and its decomposition via the chain rule. The authors validate the effectiveness of the proposed method through experiments on several classification benchmarks.

STRENGTHS AND WEAKNESSES:
Strengths:

Shows promising results on standard OOD detection tasks

Weakness:

1. The intuition behind the proposed method is not clearly presented, and the absence of illustrations or pseudocode makes the results difficult to reproduce
2. The evaluations do not clearly address why and how the proposed method enhances OOD detection
3. The evaluations are primarily conducted on standard classification benchmarks to demonstrate the effectiveness of the proposed method; including benchmarks from real-world applications would further strengthen the results

QUESTIONS:
What is the underlying intuition of the proposed method? Are there any illustrations or pseudocode to help convey the core idea?
It is important to link the evaluation results to the underlying effects of the proposed method. Rather than simply showing that it outperforms existing approaches on standard benchmarks, I’m more interested in understanding why and how it improves OOD detection

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-07-01

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposed an information theory-based method to solve the OOD detection via the VLM. The main motivation for this paper is that point-wise mutual information (PMI) could clearly differentiate the (in-distribution) ID and OOD samples. The problem here is how to calculate the PMI. This paper proposes an approximation method to estimate the PMI. The experimental results show that PMI could achieve the better performance on OOD detection.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The writing for this paper is good.\n\n2. The proposed theory is solid.\n\n\nWeaknesses:\n\n1. Limited novelty. The motivation of the paper is to compute the OOD score using an energy function. However, this idea has already been proposed in \"Energy-based Out-of-Distribution Detection.\" The authors suggest that VLMs (Visual-Language Models) can estimate PMI to derive the energy function, but this innovation seems insufficient, as the method does not fundamentally differ from existing energy-based methods. The authors have not clearly distinguished their approach from the one presented in \"Energy-based Out-of-Distribution Detection,\" which weakens the novelty of the proposed work.\n\n2. Lacking an ablation study about the computation cost. The proposed method requires sampling multiple sub-views for Monte Carlo estimation. However, there is no discussion on the computational overhead incurred by this process. It would be useful to include experiments or comparisons to assess whether this sampling step leads to a significant increase in computational cost.\n\n3. Lacking the ablation study about the different backbones. Since the paper aims to provide a theoretical framework for VLMs, it is surprising that the experiments focus solely on CLIP as the backbone. Including comparisons with other VLMs would strengthen the validity of the claims and provide a more comprehensive evaluation of the proposed method\n\nTo sum up, this paper leverages the information theory by connecting the PMI and energy function to detect OOD samples, which is interesting. However, due to the weaknesses, the contribution of this paper is limited. Meanwhile, this paper lacks the necessary ablation study. Thus, I rate it as borderline reject. If the author could address these concerns, I am willing to increase my score.\n\nQUESTIONS:\nNo other question, please refer Weaknesses.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper provides a theoretical view of post-hoc CLIP-based OOD detection methods from a point-wise mutual information perspective. It demonstrates that two representative methods in this area, MCM and NegLabel, can both be interpreted as Monte Carlo estimators of the energy function, and suggests theoretical motivation of incorporating negative labels. Then, following the framework, the paper proposes to introduce the use of subviews by randomly masking or cropping parts of an image, to enhance OOD detection performance. Empirical results show that the proposed approach brings consistent performance gain over existing baselines.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n* The paper provides a theoretical framework for post-hoc CLIP-based OOD detection, providing a systematic way to analyze methods and complementing the often empirical nature of research in this direction. \n* The experiments are extensive, covering challenging settings and compatibility with both training free and prompt learning methods. Results show that the proposed method shows strong performance and brings consistent improvement over the baselines.\n\nWeaknesses:\n* It is not entirely clear how the proposed theoretical framework is specifically connected to CLIP-based OOD detection besides the fact that CLIP is used as the estimator. It would be helpful to provide clarifications on the motivation behind focusing specifically on CLIP-based post-hoc OOD detection, instead of examining a more general OOD detection setting.  (See Questions section below.)\n\n* Aspects of the theoretical derivation are built upon various existing works cited in the paper, so there is some contribution offset in this sense.\n\nQUESTIONS:\n* It is not entirely clear how this paper’s proposed theoretical framework is specific to CLIP-based post-hoc OOD detection. While MCM and NegLabel consider the CLIP-based setting due to its generalizability and emergent properties, their core ideas often mirror traditional OOD detection methods. For instance, MCM can be considered as EnergyScore [1] but with CLIP replacing standard image classifiers, and NegLabel can also in some sense be viewed as a variant of contrastive/outlier exposure based approach. It appears that in this paper’s theoretical framework, CLIP is not specifically relevant except for setting it as the critic. Thus, it would be helpful to provide clarifications on the motivation behind focusing specifically on CLIP-based post-hoc OOD detection, instead of considering a more general OOD detection setting. \n\n* The current theoretical formulation, especially for NegLabel, is somewhat coarse-grained, in the sense that it primarily models the incorporation of non-ID prompts, but does not account for their specific properties or \"negativeness\". In practice, the selection of negative prompts has a significant impact on performance. However, from the theoretical perspective, it seems that incorporating any kind of non-ID prompt would help over MCM. It would be helpful to see more clarification or discussion on this point.\n\n[1] Liu et al. \"Energy-based Out-of-distribution Detection.\" Neurips 2020.\n\nLIMITATIONS:\nYes the authors have discussed potential limitations.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper provides a theoretical perspective on CLIP-based OOD detection by interpreting it as stochastic PMI estimation, and enhances performance and robustness through a divide-and-conquer approach using the chain rule of PMI.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n- The proposed method leverages the chain rule of PMI to decompose the original estimation into simpler conditional and unconditional components. This divide-and-conquer approach not only reduces estimation complexity but also provably increases the upper bound.\n- The method achieves state-of-the-art performance across multiple OOD benchmarks, including ImageNet-S, ImageNet-A, and hard OOD tasks like SSB-hard and NINCO. Notably, it shows improvements in both AUROC and FPR95.\n\nWeaknesses\n- The decomposition of PMI using the chain rule depends on heuristic-based transformations such as Cutout or Random Crop to generate sub-views. These augmentations, while effective, lack theoretical justification and may not generalize well across different domains or modalities.\n- The decomposition strategy introduces additional computation for PMI estimation, yet the paper does not discuss the impact on inference latency or memory overhead, which is particularly important in real-world post-hoc detection settings.\n- While the paper includes many CLIP-based and post-hoc baselines (e.g., MCM, NegLabel, AdaNeg), the overall scope of comparison feels limited.\n\nQUESTIONS:\n- Evaluate the effect of different sub-view generation methods on OOD detection performance to assess how sensitive the chain-rule decomposition is to the type of augmentation used.\n- Test the proposed method on OOD benchmarks from domains other than natural images, such as medical, satellite (e.g., EuroSAT), or sketch datasets, to evaluate whether heuristic sub-view generation still holds in these settings.\n- Provide quantitative analysis on inference time and GPU memory usage (e.g., FLOPs, runtime per image) compared to baselines like MCM or NegLabel to demonstrate practical viability of the proposed decomposition strategy.\n- Include comparisons with recent OOD detectors (HFTT (NeurIPS 2024), NPOS (ICLR 2023), ZOC (AAAI 2022), etc.) to better position the proposed method in the broader OOD detection landscape.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}