Title: $\Delta \mathrm{Energy}$: Optimizing Energy Change During Vision-Language Alignment Improves both OOD Detection and OOD Generalization
Venue: NeurIPS 2025
Accepted: yes
Keywords: Robust Vision-Language Models

OOD generalization

OOD detection
Abstract: Recent approaches for vision-language models (VLMs) have shown remarkable success in achieving fast downstream adaptation. 
When applied to real-world downstream tasks, VLMs inevitably encounter both the in-distribution (ID) data and out-of-distribution (OOD) data. The OOD datasets often include both covariate shifts (e.g., known classes with changes in image styles) and semantic shifts (e.g., test-time unseen classes). This highlights the importance of improving VLMs' generalization ability to covariate-shifted OOD data, while effectively detecting open-set semantic-shifted OOD classes. In this paper, inspired by the substantial energy change observed in closed-set data when re-aligning vision-language modalities—specifically by directly reducing the maximum cosine similarity to a low value—we introduce a novel OOD score, named $\Delta\mathrm{Energy}$. $\Delta\mathrm{Energy}$ significantly outperforms the vanilla energy-based OOD score and provides a more reliable approach for OOD detection. Furthermore, $\Delta\mathrm{Energy}$ can simultaneously improve OOD generalization under covariate shifts, which is achieved by lower-bound maximization for $\Delta\mathrm{Energy}$ (termed EBM). EBM is theoretically proven to not only enhance OOD detection but also yields a domain-consistent Hessian, which serves as a strong indicator for OOD generalization. Based on this finding, we developed a unified fine-tuning framework that allows for improving VLMs' robustness in both OOD generalization and OOD detection. Extensive experiments on challenging OOD detection and generalization benchmarks demonstrate the superiority of our method, outperforming recent approaches by 10\%–25\% in AUROC.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces **∆Energy**, an OOD detection score for vision-language models based on the energy change after masking top vision-language similarities. The approach combines ∆Energy scoring with an energy bound maximization (EBM) framework during prompt tuning. This jointly improves OOD detection and generalization under covariate shifts. Theoretical analysis and extensive experiments show improved performance over previous methods. The method is efficient and easy to integrate into existing vision-language models.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n1. The paper is written clearly, and its method is described in a straightforward manner.\n2. It provides detailed theoretical derivations supporting the proposed approach.\n3. Experiments are thorough and conducted on multiple benchmarks.\n\n**Weaknesses**\n\n1. The motivation is not sufficiently clear: the method is not proposed to address any specific issue with existing methods, nor to fill a well-defined gap.\n2. Originality is limited: pruning or zeroing the maximum score has been used in traditional OOD detection (e.g., DICE, ASH) to amplify ID–OOD differences, yet these are neither cited nor compared.\n3. I am concerned about the method’s performance: most comparisons are against very basic methods, and the reported gains in Tables 1 and 2 are only marginal(e.g., MSP, MCM). More recent and stronger baselines should be included.\n4. I doubt the claim that ΔEnergy outperforms MCM: energy-based scoring already often outperforms MSP (with existing theoretical support), and applying an energy-style normalization to MCM could yield similar improvements. Therefore, it is unclear that the gains stem specifically from the authors’ ΔEnergy formulation.\n5. The authors’ method is merely an incremental combination of an ASH-style pruning technique with the Energy approach on top of MCM.\n\nQUESTIONS:\nIn Tables 1 and 2, why do the Energy and MaxLogit performances differ so drastically? In my understanding, their results should be quite similar.\n\nLIMITATIONS:\nSee the above.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a novel OOD score, ΔEnergy, which enhances vision-language models' (VLMs) robustness in detecting out-of-distribution (OOD) data. ΔEnergy outperforms existing methods by reducing maximum cosine similarity and improving generalization. It also provides a domain-consistent Hessian for better OOD detection. Experiments show ΔEnergy improves performance by 10%-25% in AUROC. This work looks interesting, with a unique research perspective, good performance, and theoretical support.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: \n\n1)  An interesting research starting point. \n\n2) The effectiveness of the method has been supported both theoretically and experimentally.\n\nWeaknesses: \n\n1) This work heavily relies on the MCM method, and the performance improvement is not significant. For example, on PACS and\nVLCS, AUC, 93.7% VS 93.5%. \n\n2) The writing is too cumbersome and complex. It is recommended to briefly describe the idea of the theorem in the main text section and provide a more formal description in the appendix. In addition, the energy design in the paper lacks in-depth analysis, including EQ 4. It is not very clear why these designs can help OOD tasks.\n\n3) There is a problem with the application scenario of EBM, which requires obtaining several samples from the source domain. Is there any practical application background? How to satisfy the conditions of its theorem and how to detect its satisfaction.\n\n4) Lack of parameter analysis experiments, especially on how to determine the threshold of delta E in the experiment.\n\nQUESTIONS:\n1. Why design deltaE and EBM, and what is the underlying logic behind them.\n\n2. How to determine the parameters for OOD detection in deltaE and whether the parameters are robust. \n\n3. On some datasets, the improvement is acceptable, but on some datasets, the improvement is very small. Please explain the reason.\n\n4. Why EBM？ Is the EBM application scenario established? How to satisfy the conditions of the theorem.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel method to enhance the robustness of Vision-Language Models (VLMs) on two critical tasks simultaneously: out-of-distribution (OOD) generalization to covariate shifts and OOD detection of semantic shifts. The core contributions of the paper are: 1) A zero-shot OOD detection score, ∆Energy, which is based on the insight that this energy change is substantially larger for in-distribution (ID) data than for OOD data. 2) A unified fine-tuning framework, named EBM (Energy-based Bound Maximization). Minimizing the EBM loss not only increases the lower bound of the ∆Energy score but also encourages the model to learn a domain-consistent Hessian matrix.\n\nExtensive experiments on OOD detection and generalization benchmarks validate the effectiveness of both ∆Energy and EBM.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1.\tNovel Insight: The core idea of the paper—using the \"energy change\" (∆Energy) after perturbing the vision-language alignment as a basis for OOD detection—is novel. \n\n2.\tUnified Framework: Building upon the ∆Energy, the paper integrates optimization of OOD generalization and OOD detection in one framework. Theorem 3.5 links the EBM loss to a domain-consistent Hessian matrix, skillfully provides a strong justification for why this unified approach improves OOD generalization.\n\nWeaknesses:\n1.\tClarity of the Methodology: Although the overall idea of the paper is clear, some parts of the methodology (Section 3) are presented in a dense and abstract manner, which can make them difficult to follow.\n\nQUESTIONS:\n1.\tIntuitive explanation of the EBM loss: The goal of EBM is to minimize L_∆E = E_2 – E_0. Minimizing their difference intuitively encourages the model to make equally high-confidence predictions for both the original and partially masked features. Could you please explain intuitively how this objective guides the model to learn a \"domain-consistent Hessian matrix\"?\n2.\tHyperparameter Selection: The method introduces new hyperparameters, such as c in ∆Energy and the masking proportion p and loss weight λ_0 in EBM. As shown in Table 10 of the appendix, the choice of p has a noticeable impact. Could you elaborate on the selection strategy for these hyperparameters? Is there a principled way to set them, or do they require extensive validation?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nMotivated by heuristic observations regarding the energy change of cosine similarity in closed-set and open-set data, this paper introduces a novel out-of-distribution (OOD) score, $\\Delta Energy$, which simultaneously improves both OOD detection and OOD generalization performance of vision-language models during prompt fine-tuning. Theoretical analysis is provided to demonstrate that the proposed method guarantees better theoretical performance in OOD detection and generalization compared to the baselines. Evaluation on several OOD detection and generalization benchmarks further demonstrates the effectiveness of the proposed method for classification tasks.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n1. This paper is well-written and easy to follow. The heuristic observations behind the introduced out-of-distribution score are interesting and provide a straightforward approach to improving both out-of-distribution detection and generalization performance for vision-language models during prompt tuning.\n\n2. A detailed analysis is provided to demonstrate that the proposed method can improve both out-of-distribution detection and generalization performance from a theoretical perspective.\n\n3. The empirical results on commonly used OOD datasets demonstrate the effectiveness of the proposed method in improving both OOD detection and generalization performance compared to existing methods.\n\n**Weaknesses:**\n\n1. In this paper, the proposed method is specifically designed, discussed, and evaluated for distribution shift caused by covariate shift (i.e., $P_{train}(X) \\ne P_{test}(X)$ and $P_{train}(Y|X) = P_{test}(Y|X)$). However, its performance under another common out-of-distribution scenario, namely concept shift (where $P_{train}(X) = P_{test}(X)$ and $P_{train}(Y|X) \\ne P_{test}(Y|X)$), remains unclear, which significantly limits its applicability.\n\n2. In the proposed method, the objective is designed to raise the lower bound of $\\Delta Energy$ to improve OOD detection. However, the effect of increasing this lower bound on classification performance in closed-set data is unclear. I am concerned that this may negatively impact classification performance on closed-set data. It is recommended to verify this by evaluating the fine-tuning accuracy of the proposed method on standard datasets used in CLIP [1].\n\n[1] Radford, Alec, et al. \"Learning transferable visual models from natural language supervision.\" International conference on machine learning. PmLR, 2021.\n\n3. At the evaluation stage, the performance under another common out-of-distribution scenario, i.e., concept shift, is not assessed. It is suggested to include this evaluation to determine whether the proposed method can be applied to a broader range of out-of-distribution scenarios.\n\nQUESTIONS:\n1. In the proposed method, the objective is designed to raise the lower bound of $\\Delta Energy$ to improve OOD detection. However, the effect of increasing this lower bound on classification performance in closed-set data is unclear. I am concerned that this may negatively impact classification performance on closed-set data.\n\n2. At the evaluation stage, the performance under another common out-of-distribution scenario, i.e., concept shift, is not assessed. It is suggested to include this evaluation to determine whether the proposed method can be applied to a broader range of out-of-distribution scenarios.\n\nLIMITATIONS:\n**Limitation:**\n\n1. In this paper, the proposed method is specifically designed, discussed, and evaluated for distribution shift caused by covariate shift (i.e., $P_{train}(X) \\ne P_{test}(X)$ and $P_{train}(Y|X) = P_{test}(Y|X)$). However, its performance under another common out-of-distribution scenario, namely concept shift (where $P_{train}(X) = P_{test}(X)$ and $P_{train}(Y|X) \\ne P_{test}(Y|X)$), remains unclear, which significantly limits its applicability.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}