$\Delta \mathrm{Energy}$: Optimizing Energy Change During Vision-Language Alignment Improves both OOD Detection and OOD Generalization
Authors: Lin Zhu
Yifeng Yang
Xinbing Wang
Qinying Gu
Nanyang Ye
Venue: NeurIPS 2025 poster ()
Keywords: Robust Vision-Language Models
OOD generalization
OOD detection
Primary area: applications

Abstract: Recent approaches for vision-language models (VLMs) have shown remarkable success in achieving fast downstream adaptation. 
When applied to real-world downstream tasks, VLMs inevitably encounter both the in-distribution (ID) data and out-of-distribution (OOD) data. The OOD datasets often include both covariate shifts (e.g., known classes with changes in image styles) and semantic shifts (e.g., test-time unseen classes). This highlights the importance of improving VLMs' generalization ability to covariate-shifted OOD data, while effectively detecting open-set semantic-shifted OOD classes. In this paper, inspired by the substantial energy change observed in closed-set data when re-aligning vision-language modalities—specifically by directly reducing the maximum cosine similarity to a low value—we introduce a novel OOD score, named $\Delta\mathrm{Energy}$. $\Delta\mathrm{Energy}$ significantly outperforms the vanilla energy-based OOD score and provides a more reliable approach for OOD detection. Furthermore, $\Delta\mathrm{Energy}$ can simultaneously improve OOD generalization under covariate shifts, which is achieved by lower-bound maximization for $\Delta\mathrm{Energy}$ (termed EBM). EBM is theoretically proven to not only enhance OOD detection but also yields a domain-consistent Hessian, which serves as a strong indicator for OOD generalization. Based on this finding, we developed a unified fine-tuning framework that allows for improving VLMs' robustness in both OOD generalization and OOD detection. Extensive experiments on challenging OOD detection and generalization benchmarks demonstrate the superiority of our method, outperforming recent approaches by 10\%–25\% in AUROC.

Ennrgy: Optimizing Energy Change During Vision-Language Alignment Improves both OOD Detection and OOD Generalization
Lin Zhu', Yifeng Yang', Xinbing Wang', Qinying Gu², Nanyang Ye
Shanghai Jiao Tong University, Shanghai Artificial Intelligence Laboratory (zhulin_sjtu, xwang8, ynylincoln]@sjjt edu. cn,
maxvellquadyang@@mmi.. com guginyingepjlab.co org.cn cn
Abstract
Recent approaches for vision-language models (VLMs) have shown remarkable success in achieving fast downstream adaptation. When applied to real-world downstream tasks, VLMs inevitably encounter both the in-distribution (ID) data and out-of-distribution (OOD) data. The OOD datasets often include both covariate shifts (e.g., known classes with changes in image styles) and semantic shifts (e.g., test-time unseen classes). This highlights the importance of improving VLMs* generalization ability to covariate-shiffen OOD data, while effectively detecting open-set semantic-shie shifted OOD classes. In this paper, inspired by the substantial energy change observed in closed-set data when re-aligning vision-language modal- ities speciiclc by directly reducing the maximum cosine similarity to low value-we we introduce novel OOD score, named Energy. Energy significantly outperforms the vanilla energy-based OOD score and provides more reliable approach for OOD detection. Furthermore, A Energy can simultancously improve OOD generalization under covariate shifts, which is achieved by lower-bound max- imization for Ennrry (termed EBM). EBM theoretically proven to not only enhance OOD detection but also yields domain-consistent Hessian, which serves as a strong indicator for OOD generalization. Based on this finding, we developed a unified fine-tuning framework that allows for improving VLMs' robustness in both OOD generalization and OOD detection. Extensive experiments on challenging OOD detection and generalization benchmarks demonstrate the superiority of our method, outperforming recent approaches by 10%-25% % in AUROC.
1
Introduction
Recent advances in pre-trained vision-language models (VLMs), such as CLIP (Radford et al. 2021 VLMo (Bao et al., 2022). MiniGPT-4 (Zhu al., 2023a). etc., have shown promising results in visual-semantic learning. However, downstream use cases often involve further fine-tuning of VLMs. When applied to real-world downstream tasks, VLMs inevitably face challenges related out-of- distribution (OOD) data, stemming from differences in data distributions between the training and test sets (Meinshausen and Bühlmann 2015 Koh al. 2020). As illustrated in Figure [ these OOD datasets often involve closed-ser OOD data that exhibit covariate shifts (i.c., changes in environments, while class labels remain the same as the in-distribution data), as well as open-ser 0OD data with semantic shifts (i.e., test-time new categories that were unseen during fine-tuning). It is crucial to distinguish these unknown categories from known ones, rather than blindly predicting them as known
Nanyang Ye is the corresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

(A) Illustration AEnergy for OOD Detection
Different Types of Duta in Real World
Moiivstion of &'nergy
(&, Eo AEmergy)
Dara
Closad-Set
ID
Closal-Set
Open-Set
00D
Close Data
Sinstarrty
Similuitt
Energy Change
High
Cropped
Alnnenyy
Image Input
Tops
Topl Toy? Topis
Eneryy Eo
Classification
Soary
Soarf
Du Not Perforns Clossiication
Open 0OD
Similarit
Cropped V-I. Sanilartt
Low
AlEnery
Energ Changes
Huge
Huge
Small
Teg): Top2 Tog?
Topl Top2 Top3
Ensig 6
Franyy F,
Illustration of &Energyy Generalization
EBM
Top
Frenzy E2
Driginall valuis demengg Original lower bound) Rolsed lower bourd
Image Feature
Masked Image
Feature
Text Feature
Masked Image
Domsin Consisteccy
Optimizes AEnerga
Feature
Product Festtrre
Energy
Soand ssaed Ly EBN
Original Image
Masked Element
xT,
Peature
(C) Compartion with Recent Methods
Eneegy Distribution
Ænnergy Distribution
Enorgy
Mothods
CoOp
BuAci
CoCoOp
Close Sei 000
Closed Set OOD
Zero (c0b
CLIN
Bayes-CAL
One Sei 0DD
Open-Se OCD
Haxdoggt
DPLCUP
CRaFT
Neglabel
NopPormpt
LoboOp
Annergy (Ourg)
Conrs SoY
19905
Figure 1: (A) Illustration of AEnergy for OOD detection. Significant differences in AEnergy observed between closed-set data and open-set OOD data when the maximum cosine similarity is cropped to zero. (B) Illustration of the Annergy for OOD generalization. We introduce the EBM method to achieve domain-consistent Hessians, which simultaneously triggers bound optimization for A Energy. More details in Section 3.2 (C) Comparison between our A Ennrgy and EBM with state-of-the-ar methods. In the radar plots, all values are normalized to the range [0, 1]. observed that recent methods aimed improving VLMs' OOD detection may not scale well to handling different types of distribution shifts in challenging ImageNet- Ik OOD datasets.
classes (Wang et al. 2023bla). Therefore, it is essential to develop robust models that enhance VLMs' generalization ability to cloxed-set OOD data, while also effectively detecting open-set OOD classes during mieiiunn
However, most previous studies (Wortsman al. 2022 Chen et 2024 Jiang et al. 2023 Goyal ct al.] [2023] Wang et al.) 2202aa Ming et al. [202aa] Bai et al. 2024 Li et al. 2024a) have primarily focused on improving VLMs' robustness to training classes or developing OOD detection method for unseen classes independently. Consequently, existing approaches are often highly specialized for single task and are not capable of simultaneously addressing both aspects. Recent works Yangg et al., 2023; Zhang et al. 2023a) have taken into account both shift types and introduced full-spectrum OOD (FS-OOD) detection, which considers both detecting semantic shifts and being tolerant to covariate shifts. While the FS-OOD benchmark evaluates OOD detection performance across various distribution types, it may not focus on improving VL.M's classification accuracy on covariate-shifted data. Several studies (Laffon et al. (0224) Zhu et al. 2024 2025a) also have attempted to tackle this issue using multiple diverse prompts or through energy optimization techniques. However, these approaches (Lafon et al. 2024 Zhu al. (024) often require significantly more computational resources to train additional local prompts or have been evaluated on a narrow set of post-hoc functions for OOD detection. Thus, when fine-tuning VLMs for downstream tasks, the challenge of improving the VLMs' generalization ability to closed-set OOD data while simultaneously detecting open-set OOD classes that unseen during fine fine-tuning remains largely underexplored.
In this paper, we develop novel zero-shot and few-shot fine-tuning paradigms to go beyond the limitations of previous studies. We begin by proposing new post-hoc OOD detection method, inspired by the following heuristic observation: Given the text prompts corresponding to ID data and input images, compute the cosine similarities between the image features and text features.

---

As shown in Figure (A), when we crop the maximum cosine similarity to a low value (such as by resetting to zero), the resulting change energy score (Liu et al., 2020) substantially different between closed-set data and open-set semantic-shifted classes.
Takeaways for A Energy when aligning vision language modalities
When re-aligning vision- languala modalities by setting the maximum cosine similarity to zero, define the resulting change in energy score as AEnergy. As demonstrated in Theo- rem 3.2] the AEnergy for ID data is consistently larger than that for OOD data, indicating that it provides discriminative and effective method for OOD detection. Meanwhile, compared to the MCM method and raw energy scores, A Enorgy amplifies the difference between ID and OOD data- property supported by both theoretical analysis and empirical evidence. Extensive experiments further demonstrate that our method outperforms state-of-the-art zero-shot OOD detection approaches hard OOD detection benchmarks.
Building on this insight, we propose leveraging the energy change to distinguish closed-set set classes from open-set OOD classes. We introduce a zero-shot OOD detection method, termed AEnergy, which quantifies the energy change resulting from modifying vision-language alignment (i.e., the cosine similarities). As demonstrated in Section Aneergy significantly outperforms recent methods in detecting hard OOD classes, providing a more reliable approach for OOD detection.
Moreover, AEnergy can be further optimized to enhance OOD detection while simultaneously improving OOD generalization. This achieved through AEnergy-based bound maximization (termed EBM) during few-shot adaptation of VLMs. As depicted in Figure T (B), we modify the vision-langue langugge alignment by retaining the p% of the image feature elements (with p as a hyperparameter) and masking the remaining elements. The resulting masked features then used to compute a new energy change between the original and masked models, which we refer to as EBM. As demonstrated in Theorem 3.4, minimizing EBM is theoretically shown to maximize the lower bound of A Energy. Moreover, the EBM method not only theoretically enhances the discrimination between closed-set known classes and open-set OOD classes based the newly introduced OOD score (See Theorem (.4), but also leads to stronger OOD generalization under covariate shifts (See Theorem 3.5 This allows us fine-tune VLMs in unified framework, enhancing both OOD generalization for closed-set OOD data and OOD detection for open-set OOD data.
2
Preliminary
In this section, we first provide the data setting in Notation 2.1 and formally define the target tasks. Based on the widely-used vision-language model CLIP (Radford et al.) 2021), we then present the motivation for modifying the vision- language alignment through a masking operation.
Notation 2.1. Given the in-distribution (ID) samples from the downstream task, {x,,y)} y we define the classes of these samples as closed-set classes, while the other classes are considered as open- set OOD classes. The text prompts for the closed-set classes are defined as Tin = (2,, tx where K represents the number of closed-set classes. Each text prompt i; can be formulated as "a photo of (CLASS NAMEJ". Based on pre-trained VLM, we can obtain the zero-shot image features and text features, denoted {2; and {zr(tt)) respectively. Both 2 (x;) and (T(4.) are a D-dimensional features.
Task definition Given a set of closed-set ID samples (x,.s>) drawn from a source domain S, the model is tasked with learning a robust predictor f X - V/, which maps inputs x E X R Do to outputs y E y RK where Do is the dimension of x and K is the class number. Here, N is the total number of the few-shot ID samples. To effectively address both closed-set OOD data (covariate shifts) and open-set OOD data (semantic shifts), we aim to enhance the robustness of predictor from two perspectives: 1) OOD generalization, which requires the model to generalize on closed-set classes from new domains T that exhibit covariate shifts; and 2) OOD detection, which enables the model to detect open-set OOD classes during test time.
Effecc of vision-language re-alignment through masking We illustrate how vision- vision-language re- alignment is achieved through a specific masking strategy during fine-tuning. Given an image input xi and the text prompt that yields the maximum cosine similarity, we denote the corresponding
3

---

Similanity
Bound Maximization AEnergy
Cal
Bird
Dog
Before Masking
Product Features with Similarity
The Mask
Masked Image
Featture
Text Fealures
B2
Bg.
Top 1
Teat Eisoddr
Names].
Similarit After Masking
Lce
Top
Similarity Before Masking
ID
Image Encoder
Image Fenture
OOD Score: AEnergy
Similarity After Re-alignment
Reset by (1)
Reset Top-I Sumilaiillrr
AEnergy
Learnable
Prodact Fonture
Compute
Resei Top-2 Similarity
AEnergy
Sum Over
AEnergy
Changes
Energy
Text Feature
Modified
AEnergy
Top-c Similarity
Image Fearure
Similarity
Figure 2: Overview of the proposed method. Based on the prompt-tuning approach, we freeze both the image encoder and the text encoder, making only the context vectors (8 1 .,,)) learnable under the proposed objective function, as shown in Equation g During fine-tuning, we apply masking operation each ID image feature based on the top-1 similarity, as defined in Equation We then compute the resulting energy change after modifying the vision-language alignment via masking, which allows following Equation D we reset the top-c cosine similarities and then compute AEnergy for OOD detection. Simultaneously. we use the fine-tuned text feature and unmasked image feature for classification at test time. The complete algorithm can be seen in Appendix G
perform bound optimization on AEnergy.
the inference phase,
zero-shot image feature and text feature as (xi) and Zr (t). respectively. We then compute their element-wee wwie product, represented as Zp (x,) := z (xx) ) @ zr (t). Let J,. T,, and Pj denote the j-th element of (x4). Zr (t), and Zp (x1), respectively, such that 1, Ty = P,. Based on the product vector Zp (X;), ) we mask (zero-out) elements in (x;)) where P; 0. This masking operation thus effectively reduces the maximum cosine similarity to a low value, achieving modified vision-language alignment. Moreover, from the attention visualization Figure (B), the pre-trained VLM initially focuses on the foreground object. However, after masking the elements of the image feature where Pj > O, the model's attention shifts and becomes more reliant on background information. In contrast, masking the elements where Pj < 0 preserves the model's original attention, which motivates to leverage this consistency between original and masked domains to improve OOD generalization. Additional visualizations provided in Figure 3 in Appendix G
Methodology
Building upon the heuristic observations as shown in Figure I we propose novel OOD score, named Å Energy, that measures the energy change when re-aligning vision-language modalities. We theoretically demonstrate that AEnergy outperforms the widely-used MCM (Ming et al.) 2022a (see Theorems 3.2 3.3 Moreover, we introduce Eneegyy-bar -bssed bound maximization, which is proven to not only enhance OOD detection (see Theorem 3.4p but also lead to stronger OOD generalization (see Theorem 3.5). Before delving into the details, we provide the notations in this section as follows:
Notation 3.1. We define the cosine similarity between the image feature (x(x) and text feature (tz)) as 8y (x;) = (x;)) Zt(t)). Let Ú1 := argmax, sx(x;) denote the index of the maximum cosine similarity and ú == argmax (d1.. ... dy 1) sx(xi) denote the index of the j-th largest cosine similarity, And we denote the corresponding text features that have the j-th largest similarity with (x1), as (x,) == (x))). Here, (x,) refers to the text prompt corresponding to the j-th largest cosine similarity.
In this paper, (xi) and (t;))) normalized features

---

3.1 A Energy for OOD detection
The proposed A Ennrgyy, which measures the energy change after modifying the top-c maximum cosine similarities unfolds as follows:
Based on pre-trained VLM, for each image feature (xi),, we first select the text feature sets (x}( that have the top similarity with 21 (xi)-
We then compute the product between each image feature zr (xi) and the selected text feature (x().). The product feature is represented as Zp [x, (2) - zr(xi) @ (xi))..
For each text feature (x;) (j
[1..... c), we denote the j-th largest cosine similarity
between the image feature and the text feature as Sq, (x1) = (x,)) hy1).. Let $6, (xi)
represents the new cosine similarity after re- re-alignment, which is achieved by:
q, (x:)
Finally, we can compute the new OOD score (Energy(x;) E1 (x,) Ex (x:). Based on the scaling temperature т, (x4) is the energy score before the re-alignment:
K
Ex(x,) - log (xii//
(2)
Ej (x;) is the energy score after the re-alignment:
(x1)
Σ log
(xxi//
(3)
P#l
We provide formal guarantees that the proposed AEnergy can provably surpass the widely-used VLM-based OOD detection method MCM (Ming et al. 2022a).
Theorem 3.2. (OOD Detection Ability of AEnergy] Suppose that the maximum cosine similarity for an ID sample X is greater than that of an open-set OOD sample XOOD, i.e., Sÿ, (Xm)) > Sg, (XOOD). Let SMethod(x) denote the score assigned to sample x under given method. We have the following properties: 1) S Enerry (XID)
Buergy (ooerry(XXo)) (XOOD) for ID (Xı)) and open-set OOD (XooD) samples. 2) Compared to the MCM method, ΔEnergy amplifies the difference between ID and OOD data, i.e., d'Feneryy > dMCM, where dMMttod = SMethod (xm) - SMethod (Xoon)
Theorem 3.3. [The proposed 0OD Score Ennergy gets lower FPR than MCM] Given task with elosed-set ID label set Vin {V1, V/..... U.. and pre-trained VLM, for any test input x', based on the scaling temperature t, the maximum concept matching (MCM) score is computed as follows:
(x('
(M)M(x(;)m) max
Σx e0((0)/T
For any c E {1,2,-.. .. KK) if 8q (x) S 2 we have
FPR FEneRyy (T, S FPR AT)
where FPR AEneryy (r, X) and respectively, based on the temperature т and detection threshold
is the false positive rate of AEnergy and MCM,
3.2 The AEnergy-based bound maximization enhances OOD detection and generalization
Furthermore, illustrated in Equation
introduce AEnergy- -ased bound maximization function (EBM) during the fine-tuning process, which aims at increasing the lower bound of A Energy score for closed-set set classes as demonstrated in Theorem 3.4. As illustrated in Theorem 3.5 the proposed objective function is theoretically proven to not only improve OOD detection but also lead consiseconsistent Hessian, which serves as a strong indicator of OOD generalization.
TTo enlarge the energy change for closed-set data, perform re-alignment based the top-c similarities.
5

---

Specifically, motivated by further enlarging AEnergy, we propose to minimize the following term:
N LAE
(2(()) Ex)) N
(4)
where N is the number of few-shot ID samples during fine-tuning and Ex(xi) is the energy score for X₁ after masking on the image feature, which is formally calculated
E2 Ex(x:)
log
e (x/)/
(5)
sx(x = (Z1(x;) O m (x1)) Zr(tz)
(6)
Here, m' is the text feature corresponding to the top-1 cosine similarity.
(x;) m'(x;)
the mask that retains the top p- p-proportion elements in zí (x;) @ h1 (x;) and h1 (x:)
Theorem 3.4. [EBM increase the lower bound of A Annergy] Let h1 (xx) denote the the text feature that have the top-1 similarity with the image feature zı (x;). The corresponding similarity is computed
Sg. (x:) (x:) h1 (xx). Suppose that LAE EE. with = L, under the condition that:
Σ ch (x)/T (x)/T
Σ
(e 1)e F2(x;)
we have N x- AEnorgy LAR-
Ee
Theorem 3.4 implies that if the change in the VLM's predictions after re-alignment is not too small
and satisfies the condition as shown Equation 7 minimizing LA can increase lower bound of A Energy for closed-set classes. Moreover, we theoretically demonstrated that the proposed EBM loss Equation can also lead to strong indicator for OOD generalization (Rame et al. 2022; Hemati et al., (223)..
Hessians classification loss, which serves as
Theorem 3.5. [EBM leads to domain--onsistent Hessians] Given the ID training data sampled from domain S and the learnable parameter 0 in VLM, we denote the masked domain as S'. We represent the empirical classification loss on the domain D as Ep(O). Let Gp(0) and HD(0) be the gradient vector and Hessian matrix of empirical risk (8) over parameter 0 respectively. In this paper, propose to minimize LAE. The distance between the unmasked and masked image feature is assumed satisfy: [|zT(xi) (Zr(xi) m (x1))]l2 Then the local optimum 0 of min LAE satisfies:
(0))) (s))
V3
Proposition 3.6. [EBM bound 0OD generalization] Let 21(xs) and 21 (xx)) denote the image feature from source domain (S) and target domain (T), respectively. We assume that ((xx))-((())))) S Et- By applying the second-order Taylor expansion and utilizing the domain-consistent Hessians as outlined in Theorem 3.5 the OOD generalization gap between source domain (S) and target domain (T) is upper bounded by the following inequality:
max E -0)))Esser =1) -()(()))00)) max 2
1 Hs(0")0 + 0(s))
where is a local minimum across all domains, i.e., V0E =
Therefore, by connecting the EBM loss with the Hessians of empirical classification loss, we theoretically discover that the EBM loss can lead to bound of the performance gap between closed- set data and closed-set OOD data. This implies that optimizing for AFnergy with EBM loss also involves optimizing for OOD generalization.

---

Table 1: OOD detection between closed-set data and open-set OOD data based on ImageNet-1kg
OOD detection measured by AUROC and FPR95 over the mixture of closed-set test sets and open-set
OOD test sets.
DATA
Mechod
Energy
ODIN
ReAct
CLIPN
MSP
MaxLogit
MCM
NegLabel
CSP
A Energy (Ours)
ID Semantic-chifted
FPRS5]
16.72
51.31
80.38
64.37
51.12
69.12
53.34
77.31
68.49
46.40 (2.58)
AUROC
85.61
74.06
$1.44
87.10 (0.75)
Coeriaat-ssnileed OOD vs.
EPROS]
83.94
61.25
85.05
84.44
69.20
79.80
70.30
82.81
79.76
67.16 16)
Semunti-shhited OOD
AUROC
67.21
77.10
64,66
64.64
78.64
70.52
75.66
67.38
67.89
78.68 (0.57)
Table 2: OOD detection between closed-set data and open-set OOD data based on PACS and
VLCS: OOD detection measured by AUROC and FPR95 over the mixture of closed-set OOD and open-set OOD test sets.
DATA
PACS vs. Open Set (AUC) [PR95[)
VLCS vs. Open Set (AUC)/ / FPR95()
AVG
Method
DTD
Food101
Caltech101
DTD
Food101
Caltech101
AUC1 FPR95,
Energy
87.4/67.6
95.97 26.0
86.7
55.3/88.8
85.8/48.3
53.3/86.3
76.6 61.5
ReAct
89.5 44,9
98.1 79.9
89.8 43.2
52.8/89.7
86.7 44.7/47.8
61.4/83.2
79.7/53.1
CLIPN
93.6 40.3
25.2
16.11
80.4/62.7
88.9 88.9/46.66
72.5/74.1
86.6
Maxl Loght
45.5
97.9/1311
88.9 46.8
68.8/87.6
88.9/45.4
61.4/8166
53.3
MSP
97.91
9..914.6
95.8 20.9
84.4/55.3
93.7/35.1
88.9 88.9/48.8 44..8
93.3 29.1
ODIN
9..1/1.8
99.37
9..4/6.7
83.1/46.5
91.3/28.2
86.8 46.8/
92.8 17.9
MCM
98.9/4.3
99.213.4
97.0 13.7
84.2/55.1
93.3/36.4
88.5 50.4
93.5 27.2
Lahel
9..3/4.2
97.7/15.8
95.7 /31.6
84.8/54.
79.4/74.6
/2.7
86.6 49.1
CSP
99.6/1.8
9..2
97.8 12,4
38.7
667.
67.3/73.0
88.5 32.7
AEnergy (Ours)
9.5176.5
9..21224
96.1 14.3
85.3/53.2
/4.1/319
84.5
93.7 25.9
3.3
Overview of the proposed method
Our theoretical analysis thus leads to the design of fine-tuning framework with concurrent optimization for both tasks. As illustrated in Figure 2 we prioritize computational efficiency by adopting prompt-tuning techniques. In the fine-tuning process, both the image encoder and text encoder are frozen and only the context vectors O are learnable. Let Lce denote the Cross-Entropy loss and A denote the hyperparameter that can be chosen based on the validation procedure. Then the final optimization objective of the EBM method is expressed as:
CEBM LCE + Coe
(8)
4 Experiments
In this section, motivated by the remarkable success of the vision-language model CLIP (Radford al. (02T) in learning general visual knowledge, we conduct experiments based CLIP. First, conduct extensive experiments to validate the effectiveness of the proposed A Energy in zero-shot OOD detection across various datasets. Furthermore, we evaluate the effect of the proposed EBM method in enhancing both OOD generalization and OOD detection. Due to space limitations, we provide ablation studies in Appendix G to validate our theoretical findings.
4.1 Effectiveness of A Energy for OOD detection
Dataset We evaluate OOD detection performance over different benchmarks, including 1) the discrimination between closed-set OOD data and open-set OOD data based on ImageNet-1k: 2) the discrimination between closed-set OOD data and open-set OOD data based on cross-datased images, 3) hard OOD detection on different splits of ImageNet-Ik, 4) the conventional OOD detection benchmark. For the first two benchmark datasets, we evaluate the models' OOD detection capabilities under more challenging scenarios, where the datasets exhibit both covariate and semantic shifts. Models are required to distinguish between various types of closed-set OOD data (covariate shifts) and open-set OOD data (semantic shifts). Details of the two data settings are illustrated as follows:
1) Setup-I: open-set discrimination on the large-scale scale ImageNet-Ie dataset. Following the prior work (Zhu et al. 2024), we split ImageNet- 1k (Krizhevsky et al. 2017) into open and closed sets w.r.t class labels. We randomly define 40% classes of ImageNet as the closed-set, and the remaining 60% as the open-set. The samples from ImageNet-A (Hendrycks al.) (022b), ImageNet-R (Hendrycks et al.) (Wang et al. 2019). and ImageNet- V2 (Recht et al. (019). which share the same class labels the closed-set ID data, considered as closed-set OOD data.
2021a), ImageNet-Sken
2) Setup-Il: open-set set discrimination on cross-datasef images. Using cross-dataset examples as the
open-set set is another established protocol (Shafaci et al., 2018 Kong and Ramanan 2021) Following
7

---

Table 3: Tuning-based results on Setup-I: Comparison with competitive fine-tuning methods based
on CLIP ViT-B/16 using 16 samples per class. In the testing phase of LoCoOp, NegPrompt and
GalLoP, we use the GL-MCM score (Miyai et al. (022)) to compute OOD detection results.
Algorithm
CoOp
CoCoOp
CLIP-Adapter
Baya--AL MCM
DPLCLIP
CRoFT
LoCoOp
NogPrompt
GalLoP
EBM (Ours)
ODD Score
MCM
MCM
MCM
MCM
MCM
GI.
Gl.
G1.
AFarryy
1
82.11
81.59
79.91
82.31
82.46
82.03
84,51
(0.4)
ODD ACC f
61,36
62.58
60.58
61.95
61.53
62.83
61.18
60.39
61.75
63.28 (0.2)
AUROC
72.94
76.38
74.86
74.44
72.81
76.30
70.03
60.86
56.97
81,90 (1.9)
FPR95 1
73.15
70.30
70.92
72.34
73.07
69.78
74.33
86.66
91.17
65,90 (1.7)
the prior work (Gulrajani and Lopez- Paz
1021
Cha et al.
[2022]
Ye et al.
202I), we leverage popular
datasets like PACS (Li et al., (017)) or VLCS (Li et al. 2017) for domain generalization studies as
the closed-set data. We evaluate the models' ability to distinguish between closed-set OOD and
cross-dataset images by utilizing different styles of datasets like Caltech101 (Bansal et al. [021]) )
DTD (Sharan et al.] 2)14), and Food101 (Bossard al.] 2014) as open-set OOD examples. All overlapping classes are removed from the three open-set OOD datasets.
In addressing the hard OOD detection scenarios, we follow the prior works Ming et al. 2022a Li
ct al. 2024a Chen ct al. 2024) and partition the ImageNetIk dataset into two parts: one part of the data serves as the ID, while the other serves as OOD, For conventional OOD detection, we use popular benchmark in which ImageNet- Ik (Krizhevsky et al., 2017) with 1,000 classes is used as
the ID dataset, and the OOD datasets including subsets of Texture (Cimpoi et al. 2014), iNaturalist
(Van Horn et al. 2018). Places (Zhou et al,, 2017) and SUN (Xiao et al., 2010)..
Comparison methods To substantiate the effectiveness of the proposed OOD score, we conduct an empirical analysis of distinct categories of methodologies for OOD detection utilizing VLMS. These categories encompass zero-shot approaches and the methods that combine the CLIP image encoder with classical approaches. For zero-shot methods, we opted for 4 recent methods, MCM (Ming et al., 2022a), CLIPN (Wang et al.] 2023a), Negl.abel (Jiang et al. (024) and CSP (Chen et al. 2024). MCM employs the original CLIP, utilizing the maximum softmax probability operation on the similarities for detection, and CLIPN involves additional training phase during pre-training, specifically training a negative text encoder using large external data. Both the NegLabel and CSP methods introduce additional negative labels, enabling more accurate OOD detection. For the second group of methods, we adapt previous logits- based methodologies to the use of the CLIP image encoder, including MSP (Hendrycks and Gimpel| 2016), Energy (Liu al. 2020). MaxLogit (Hendryeks) eet al. 2019), ReAct (Sun et al. 2021) and ODIN (Liang et al. 2017a). Following the previous studies (Wang et al. 2023a we use CLIP based on ViT-B/16, which is pre-trained from OpenCIII.
Metrics Two OOD detection metrics are used. The first the False Positive Rate at a 95% True Negative Rate (FPR95), which denotes the rate of falsely identified OOD instances when the true neg- ative rate is maintained at 95%. The second is the Area Under the Receiver Operating Characteristic curve (AUROC), representing the measure of OOD ranking across various classification thresholds.
Experiments results For zero-shot OOD detection, we set c = 2 and = 0.01 in our AEnergy. We present the results of A Energy and competitors in discriminating between deseed-set data and open-set OOD data in Table I Due to space limitations, we provide the results on the traditional OOD detection and hard OOD detection datasets in Table 7 in Appendix G It is observed that the proposed A &nnergy obtains the top-1 AUROC performance on all benchmarks. Compared with the vanilla energy-based OOD detection method (Liu et al., 2020), our A.nergy method consistently achieves better OOD detection performance across all 4 benchmarks. Notably, as shown in Table D approach surpasses the competitive NegLabel and CSP methods by large margin, demonstrating the superiority of the proposed method in distinguishing different semantics in open-world scenarios. The inferior performances of NegLabel and CSP may stem from the underlying assumption of these may not hold
methods-thin OOD samples possess a variety of distinct visual properties-wite in hard OOD detection scenarios where OOD samples are distributed closely with closed-set data. Furthermore, the presence of covariate shifts reduces the similarity between closed-set OOD and ID data, thereby making it more difficult distinguish closed-set OOD data from open-set OOD data.
4.2 Effectiveness of the EBM loss for both OOD generalization and OOD detection
Datasets To substantiate the effectiveness of the proposed EBM method for both tasks, we evaluate our method under two data settings (Setup-I and Semu--II,, each incorporating both covariate shifts and semantic shifts, as introduced in Section 4.1 For evaluating OOD generalization performance on
8

---

Table 4: Tuning-based results on Setup-ll: Comparison with competitive fine-tuning methods based
on CLIP ViT- using 16 samples per class.
DATA
PACS
PAC'S Open-Set (AURUCT /FPR55))
VLCS
VLCS
Open-Set (AUROCT (PPR95))
AVG
Algorihh
OOD ACC
D913
Food11
Caheeh101
OOD ACC1
DTD
Food101
Caltech101
FPR.
CLLP MCM
96.1 (0.)
991734
97077
75.1
95:73364
27.2
C1.1 ARRungy
96.1 (0.)
98. 16.5
9..2/2.4
96.17 14.3
(00)
85.3/53.2
94.1
89.5/47.3
25.5
CoOp
(6.3
98.974.6
99.2/3.1
974411.2 11.2
78. (1
39.3/37.6
91:4740.1
85.9/47,4
24.0
CoCoOp
96.8 (0.5)
98.8/4.0
97.5/ 10.9
78.9 (0.))
88.3/44.99
89.2/44..5
85.8 48.6
26.5
CLIP-Adapter
96.1 (0.0)
9../4.1
99:21 3.6
97.4 12.1
77.3 (0.7)
85.0
/3.8/35.5
89.3/48.5
26.0
Bayes CAL
96.6 (1.5)
9..5/7.2
98.1
95,91 16.6
79.6 (0.9)
88.0/44.7
84.9/61.1
84.87 16.77
33.0
DPLCLIP
95.6 (0.2
96.6
97.1/18.
92.2135.11
76.5 (1.3)
88.9 /37.1
80.8 /43.2
/4.2/50.2
34.2
LoCoOp
96.9 (1.3)
96.1 79.7
98.4/86
95.97 19.4
(0.7
86. 750.9
B4.3:62.8
63.37 59.7
35.2
GolLop
9.93/8.9
81.3
NT.4 39,0
91.07434
29.0
CRaFT
33.0
/57.7
840/71.2
40.6
770.5
NegPrumpi
97.100.4
42.2
94.122.6
93.4/
89.8/398
72.4
36.9
EBM MCM
97.2.0011
9866566
90.074.
96.87
(0A))
91.6/32.6
93.7
86.57 44.114411
22.2
EBM AEnergy
97.2
96.2/6.3
98.7/3.8
96.67 14.8
81.7 (0.0)
91.7/30.2
93.6/28.6
86.4742..5
21.0
Setup-II, we utilize the leave-one-domain-e validation protocol (Gulrajani and Lopez-Par Paz 2020 that uses three domains as closed-set ID data and the remaining one as closed-set OOD data.
Comparison methods We conduct an empirical analysis of distinct categories of CLIP-based lightweight fine-tuning methods. In addition to comparing our approach with popular fine-tuning techniques, such the widely-used CoOp (Zhou et al. 2021), CoCoOp (Zhou et al., [022) and CLIP- Adapter (Gao et al. 2023), we also evaluate it against CLiP-based methods designed specifically for OOD generalization or OOD detection. For OOD generalization, we compare our EBM with methods like DPLCLIP (Zhang et al. (002b) and Bayes-CAL. (Zhu et al. 2023b 2025b). For OOD detection, we consider approaches such as LoCoOp (Miyai et al. [0024b), NegPrompt (Li et al. (0244). CRoFT (Zhu et al. 2024), and GalLoP (Lafon et al.] 2024). with CRoFT and GalLoP explicitly designed to optimize both OOD generalization and detection.
Experiment details We conduct experiments based on the CLIP ViT-B/16 model using 16 samples per ID classes. For the prompt learning methods, set the number of context tokens to 16. Without otherwise specified, methods are trained using the SGD optimizer with learning rate of 0.002 and batch size of 32 for fair comparisons. We set the maximum training epoch to 30 for all models. For all baseline methods, we follow the hyperparameter searching protocols recommended in their original papers. For EBM method, we search for Ao in 0.5, 1.0] for Setup-I and we set Au = 2 for Setup-II. We set T = 0.01 and c = 2 in A Energy, and we vary the masking proportion p% within the range of [0.4, 0.5, 0.6] in CAE. For experiments on each method, we repeat 3 times with different random splits to eliminate the effects of randomness. Finally, we report the average classification accuracy on closed-set test sets, as well the average FPR95 and AUROC results for distinguishing between open-set OOD data and closed-set OOD data based on MCM (GL-MCM) (Ming al., 2022a Miyai et al. 2023) or our AEnergy.
random initialization for context vectors and
Experiment results We present the results of setup-I in Table 3 where the proposed EBM method establishes the overall best performance in both OOD generalization and OOD detection. Notably, our method outperforms the competitive CRoFT (Zhu et al. 2024) method, which is introduced to achieve concurrent optimization both tasks. Our method obtains 0.45% improvement on OOD accuracy in closed-set OOD data and more than 5% improvements on AUROC when discriminating closed-set OOD datasets and open-set OOD classes. In contrast, although the competitors can achieve higher ID test accuracy, they often struggle with the OOD generalization or OOD detection task, even resulting in worse performance when compared to the zero-shot CLIP model. Moreover, it is observed that recent methods, such as Gall LoP (Lafon et al. (024) and NegPrompt (Li et al., 2024a), aimed at improving VLMs' OOD detection may not scale well to handling different types of distribution shifts in large-scale ImageNet-I1 Ik datasets. The results of setup-Il II are shown in Table 4 where the EBM method also demonstrates the best overall performance on both tasks. Consistent with the results of Setup-I, its competitors, especially those (Zhu et al.) 2023b Zhang et al.) 2021b Zhu et al., 2024) designed for OOD generalization, achieve even higher FPR95 scores compared to the CLIP model. Notably, for the more challenging VLCS dataset, our EBM method shows significant performance improvements over CRoFT and GaLLoP on both tasks. This is likely due to the fact that CRoFT performs adapter-tuning on both the image and text inputs, which can lead to forgetting the general knowledge encoded in the pre-trained CLIP model. contrast, based on fine-tuning language model, method optimizes the energy change when re-aligning vision-languee language modalities, resulting in more robust performance across various datasets. These results highlight the EBM method's 's ability to more effectively enhance both OOD generalization and OOD detection.
9

---

Conclusions
Different from the vanilla energy-based OOD score, we propose a novel zero-shot OOD detection method, AEnergy, which measures the energy change when re-aligning vision-language modalities. Both theoretical and experimental results demonstrate that A Ennerg provides more reliable OOD detection than previous methods. Furthermore, we introduce a A AEnergy-base bound maximization during fine-tuning VLMs. The proposed bound maximization theoretically proven to not only improve OOD detection but also lead to optimization for OOD generalization. Building this insight, we have developed unified Lie-tunnig framework that enables the concurrent optimization of both tasks. Extensive experiments on challenging OOD detection and generalization benchmarks demonstrate the superiority of our method.
Acknowledgements
This work is supported by National Natural Science Foundation of China (30.62572313, No.62106139).
References
Haoyue Bai, Gregory Canal, Xuefeng Du, Jeongyeol Kwon, Robert D Nowak, and Yixuan Li. Feed two birds with one scone: Exploiting wild data for both out-of-distribution generalization and detection. In International Conference on Machine Learning, pages 1454-144 1471. PMLR, 2023.
Yichen Bai, Zongbo Han, Bing Cao, Xiaoheng Jiang, Qinghua Hu, and Changqing Zhang. Id-like prompt learning for few-shot out-of-distribution detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition, pages 17480-17489, 2024.
Monika Bansal, Munish Kumar, Monika Sachdeva, and Ajay Mittal. Transfer learning for image classification using vggl 19: Caltech-101 image data set. Journal of ambient intelligence and humanized computing, pages 1-12, 2021.
Hangbo Bao, Wenhui Wang, Li Dong, Qiang Liu, Owais Khan Mohammed, Kriti Aggarwal, Subhojit Som, Songhao Piao, and Furu Wei. Vlmo: Unified vision-language pre-training with mixture- of-nodality-experss. Advances in Neural Information Processing Systems, 35:32897-32912, 2022.
Petra Bevandić, Ivan Krešo, Marin Oršié, and Siniša Šegvić. Dense outlier detection and open-set recognition based training with noisy negative images. arXiv preprint arXiv:2101.09193 2021.
Lukas Bossard, Matthicu Guillaumin, and Lue Van Gool. Food-101-a 101-mmning discriminative compo- nents with random forests. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part VI 13, pages 446-461. Springer, 2014.
Junbum Cha, Kyungjae Lee, Sungrae Park, and Sanghyuk Chun. Domain generalization by mutual- information regularization with pre-trained models. In European Conference on Computer Vision, pages 440-457. Springer, 2022.
Mengyuan Chen, Junyu Gao, and Changsheng Xu. Conjugated semantic pool improves ood detection with pre-trained lasion-language models. arXiv preprint -XXv:24400086111 2024.
Mircea Cimpoi, Subhransu Maji, lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describ- ing textures in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3606-3613, 2014.
Xuefeng Du, Gabriel Gozum, Yifei Ming, and Yixuan Li. Siren: Shaping representations for detecting out-of-distribution objects. Advances in Neural Information Processing Systems, 35:20434-20449, 2022a.
Xuefeng Du, Zhaoning Wang, Mu Cai, and Yixuan Li. Vos: Learning what you don't know by virtual outlier synthesis. arXiv preprint arXiv:2202.01119 2022b.
10

---

Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In international conference on machine learning. pages 1050-1059. PMLR, 2016.
Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma, Rongyao Fang, Yongfeng Zhang, Hongsheng Li, and Yu Qiao. Clip-adapter: Better languuge models with feature adapters. International Journal of Computer Vision, pages 1-15, 2023.
Chuanxing Geng, Sheng-jun Huang, and Songcan Chen. Recent advances in open set recognition: A survey. IEEE transactions on pattern analysis and machine intelligence, 43(10):311--361, 2020.
Muhammad Waleed Gondal, Jochen Gast, Inigo Alonso Ruiz, Richard Droste, Tommaso Macri, Suren Kumar, and I Luitpold Staudigl. Domain aligned clip for few-shot classificatton.. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 5721-5730, 2024.
Koustava Goswami, Srikrishna Karanam, Prateksha Udhayanan, KJ Joseph, and Balaji Vasan Srini- vasan, Copl: Contextual prompt learning for vision-language understanding. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 18090-18098, 2024.
Sachin Goyal, Ananya Kumar, Sankalp Garg, Zico Kolter, and Aditi Raghunathan. Finetune like you pretrain: Improved finetuning of zero-shot vision models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 19338-19347, 2023.
Ishaan Gulrajani and David Lopez-Paz. In search of lost domain generalization. arXiv preprint arXiv:2007.01434, 2020.
Ishaan Gulrajani and David Lopez-Paz. search of lost domain generalization. In ICLR, 2021.
Sobhan Hemati, Guojun Zhang, Amir Estiri, and Xi Chen. Understanding hessian alignment for domain generalization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 19004-19014, 2023.
Dan Hendrycks and Kevin Gimpel. baseline for detecting misclassified and out-of-distribution examples in neural networks. arXiv preprint 1610.02 2016.
Dan Hendrycks, Steven Basart, Mantas Mazeika, Andy Zou, Joe Kwon, Mohammadreza Mostajabi, Jacob Steinhardt. and Dawn Song. Scaling out-of-distribution detection for real-world settings. arXiv preprint arXiv:1911.1113: 2019.
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, al. The many faces of robustness: A critical analysis of out-of-distribution generalization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8340-8348 2021a.
Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adver- sarial examples. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15262-15271, 2021b.
Chen Huang, Skyler Seto, Samira Abnar, David Grangier, Navdeep Jaitly, and Josh Susskind. Aggregate-and-adaan natural language prompts for downstream generalization of clip. arXiv preprint arXiv:2410.220:
2024.
Tony Huang, Jack Chu, and Fangyun Wei. Unsupervised prompt learning for rision-language models. arXiv preprini MXX::22004006449 2022.
Jingjing Jiang, Ziyi Liu, and Nanning Zheng. Correlation information bottleneck: Towards adapting pretrained multimodal models for robust visual question answering, 2023.
Xue Jiang, Feng Liu, Zhen Fang, Hong Chen, Tongliang Liu, Feng Zheng, and Bo Han. Negative label guided ood detection with pretrained langunel models. arXiv preprint arXiv:2403.20077, 2024.
Muhammad Uzair Khattak, Hanoona Rasheed, Muhammad Maaz, Salman Khan, and Fahad Shahbaz Khan. Maple: Multi-modal prompt learning. In Proceedings of the IEEF/CVF Conference on Computer Vision and Patern Recognition, pages 19113-1912, 2023a.
11

---

Muhammad Uzair Khattak, Syed Talal Wasim, Muzammal Naseer, Salman Khan, Ming- Musann Yang, and Fahad Shahbaz Khan. Self-regulating prompts: Foundational model adaptation without forgetting. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 15190-15200, 2023b.
Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Bal- subramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips. Sara Beery, al. Wilds: A benchmark of in-the-wild distribution shifts. arXiv:2012.074211 2020.
Shu Kong and Deva Ramanan. Opengan: Open-set recognition via open data generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 813-822, 2021.
Alex Krizhevsky, llya Sutskever, and Geoffrey Hinton. Imagenet classification with deep convolu- tional neural networks. Communications of the ACM, 60(6):84-90, 2017.
Mare Lafon, Elias Ramzi, Clément Rambour, Nicolas Audebert, and Nicolas Thome. Gallop: Learning global and local prompts for vision-language models. In European Conference on Computer Vision, pages 264-282. Springer, 2024.
Chaohua Li, Enhao Zhang, Chuanxing Geng, and Songean Chen. Recent advances in out-of- distribution detection with clip-like models: A survey. arXiv preprint wrXiv:2505502448, 2025.
Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M Hospedales. Deeper, broader and artier domain generalization. In Proceedings of the IEEE international conference computer vision, pages 5542-5550, 2017.
Tianqi Li, Guansong Pang. Xiao Bai, Wenjun Miao, and Jin Zheng. Learning transferable nega- tive prompts for out-of-distribution detection. Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 17584-17594, 2024a.
Zheng Li, Xiang Li, Xinyi Fu, Xin Zhang, Weiqiang Wang, Shuo Chen, and Jian Yang. Promptkd: Unsupervised prompt distillation for vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26617-26622 2024b.
Shiyu Liang, Yixuan Li, and Rayadurgam Srikant. Enhancing the reliability of of image detection in neural networks. arXiv preprint 170.v:1706.026600 20174 7a.
Shiyu Liang, Yixuan Li, and Rayadurgam Srikant. Enhancing the reliability of out-of-distrrbution
image detection in neural networks. arXiv preprint arXiv:1706.026900 2017b.
Weitang Liu, Xiaoyun Wang, John Owens, and Yixuan Li. Energy- based out tott-f-distributiin detection. Advances in neural information processing systems,
2020.
Andrey Malinin and Mark Gales. Predictive uncertainty estimation via prior networks. Advances in neural information processing systems, 31, 2018
Nicolai Meinshausen and Peter Bühlmann. Maximin effects in inhomogeneous large-scale data. The Annals of Statistics, 43(4), 2015.
Yifei Ming and Yixuan Li. How does fine-tuning impact out-of-distribution detection for vision- language models? International Journal of Computer Vision, 132(2):596-60) 0099966669,1600 2024.
Yifei Ming, Ziyang Cai, Jiuxiang Gu, Yiyou Sun, Wei Li, and Yixuan Li. Delving into out-of- distribution detection with vision-language representations. Advances in neural information processing systems, 35:35087-35102, 2022a.
Yifei Ming, Ying Fan, and Yixuan Li. Poem: Out-of-distribution detection with posterior sampling.
In International Conference on Machine Learning, pages 15650-15665. PMLR, 2022b.
Atsuyuki Miyai, Qing Yu, Go Irie, and Kiyoharu Aizawa. Zero-shot in-distribution detection in multi-object settings using vision- vision-language foundation models. arXiv preprint 2304::2300.0452,,, 2023.
12

---

Atsuyuki Miyai. Jingkang Yang, Jingyang Zhang, Yifei Ming. Yueqian Lin, Qing Yu, Go Irie, Shafig Joty, Yixuan Li, Hai Li, et al. Generalized -iitrii-iitiitttinn detection and beyond in vision language model era: A survey. arXiv preprini arXiv:240.221794 2024a.
Atsuyuki Miyai, Qing Yu, Go Irie, and Kiyoharu Aizawa. Locoop: Few-shot out-of-distribution detection via prompt learning. Advances in Neural Information Processing Systems, 36, 2024b.
Vivek Narayanaswamy, Yamen Mubarka, Rushil Anirudh, Deepta Rajan, and Jayaraman J. Thi- agarajan. Exploring inlier and outlier specification for improved medical ood detection. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV) Workshops. pages 4589-4598, 2023.
Jun Nie, Yonggang Zhang, Zhen Fang, Tongliang Liu, Bo Han, and Xinmei Tian. detection with negative prompts. In The twelfth international conference on learning representa- tions, 2024.
Stanislav Pidhorskyi. Ranya Almohsen, and Gianfranco Doretto. Generative probabilistic novelty detection with adversarial autoencoders. Advances in neural information processing systems, 31, 2018.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. CoRR, abs/2103.00020, 2021.
Alexandre Rame, Corentin Dancette, and Matthieu Cord. Fishr: Invariant gradient variances for out-of-distribution generalization. In International Conference on Machine Learning, pages 18347-18377. PMLR, 2022.
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize imagenet? In International conference on machine learning, pages 5389-5400. PMLR, 2019.
Kuniaki Saito, Donghyun Kim, and Kate Saenko. Openmatch: Open-set semi-supervised learning with open-set consistency regularization. Advances in Neural Information Processing Systems, 34: 25956 25966,, 2021.
Shibani Santurkar, Dimitris Tsipras, and Aleksander Madry. Breeds: Benchmarks for subpopulation shift. 2020.
Walter Scheirer, Lalit P Jain, and Terrance E Boult. Probability models for open recognition. IEEE transactions on pattern analysis and machine intelligence, 36(11):2317-2322 2014.
Alireza Shafaei, Mark Schmidt, and James J Little. A less biased evaluation of out-of--istribuut sample detectors. arXiv preprint veXiv:1809 1809.04729, 2018.
Lavanya Sharan, Ruth Rosenholtz, and Edward H Adelson. Accuracy and speed of material catego- rization real-world images. Journal of vision, 12-2):12-12, 2014.
Mainak Singha, Harsh Pal, Ankit Jha, and Biplab Banerjee. Ad-clip: Adapting domains in prompt space using clip. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4355-4364, 2023.
Lin Song, Ruoyi Xue, Hang Wang, Hongbin Sun, Yixiao Ge. Ying Shan, et al. Meta-adapter: An online few-shot learner for vision-language model. Advances in Neural Information Processing Systems, 36:55361-5537444 2023.
Xin Sun, Zhenning Yang. Chi Zhang, Keck- Voon Ling, and Guohao Peng. Conditional gaussian distribution learning for open set recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages I 13480-13489, 2020.
Yiyou Sun, Chuan Guo, and Yixuan Li. React: Qut-of-diitriibtiio detection with rectified activations. Advances in Neural Information Processing Systems, 34: 14:144-157, 2021.
13

---

Yiyou Sun, Yifei Ming, Xiaojin Zhu, and Yixuan Li. neighbors. International Conference on Machine Learning, pages 20827-20840. PMLR, 2022.
detection with deep nearest
Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. The inaturalist species classification and detection dataset. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8769-8778, 2018.
Haohan Wang. Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global representations by penalizing local predictive power. Advances Neural Information Processing Systems, 32, 2019.
Haoqi Wang, Zhizhong Li, Litong Feng, and Wayne Zhang. Vim: Out-of distribiitti with virtual- logit matching. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4921-4930, 2022.
Hualiang Wang, Yi Li, Huifeng Yao. and Xiaomeng Li. Clipn for zero-shot ood detection: Teaching clip to say no. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1802-1812, 2023a.
Haoliang Wang, Chen Zhao, Yunhui Guo, Kai Jiang, and Feng Chen. Towards effective seman- tic ood detection in unseen domains: A domain generalization perspective. arXiv preprint arXiv:2309. 10209, 2023b.
Zhengbo Wang. Jian Liang, Ran He, Nan Xu, Zilei Wang, and Tieniu Tan. Improving zero-shot generalization for clip with synthesized prompts. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 3032-3044 2023c.
Syed Talal Wasim, Muzammal Naseer, Salman Khan, Fahad Shahbaz Khan, and Mubarak Shah. Vita-clip: Video and text adaptive clip via multimodal prompting. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pages 23034-23044, 2023.
Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Komblith, et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In International Conference on Machine Learning, pages 23965-23998, PMLR, 2022.
Jianxiong Xiao, James Hays, Krista Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In 2010 IEEE computer society conference on computer vision and pattern recognition, pages 3485-3492. IEEE, 2010.
Zehao Xiao, Jiayi Shen, Mohammad Mahdi Derakhshani, Shengeai Liao, and Cees G. M. Snoek. Any-shift prompting for generalization over distributions. 2024.
Jingkang Yang, Pengyun Wang, Dejian Zou, Zitang Zhou, Kunyuan Ding, Wenxuan Peng, Haoqi Wang, Guangyao Chen, Bo Li, Yiyou Sun, Xuefeng Du, Kaiyang Zhou, Wayne Zhang, Dan Hendrycks, Yixuan Li, and Ziwei Liu. Openood: Benchmarking generalized out-offdiisrributio detection. 2022.
Jingkang Yang. Kaiyang Zhou, and Ziwei Liu. Spel-spectrum of distirtrinttion detection. Interna- tional Journal of Computer Vision, 131 131( 2622, 2023.
Nanyang Ye, Kaican Li, Lanqing Hong, Haoyue Bai, Yiting Chen, Fengwei Zhou, and Zhenguo Li. Ood-bench: Benchmarking and understanding duitt--distriuution generalization datasets and algorithms. CoRR, Abs/2106.03721 2021.
Hongjie Zhang, Ang Li, Jie Guo, and Yanwen Guo. Hybrid models for open set recognition. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part III 16, pages 102-117. Springer, 2020.
Jingyang Zhang, Jingkang Yang, Pengyun Wang, Haoqi Wang. Yueqian Lin, Haoran Zhang, Yiyou Sun, Xuefeng Du, Yixuan Li, Ziwei Liu, Yiran Chen, and Hai Li. Openood v1.5: Enhanced benchmark for out-of-distribution detection. arXiv preprint arXiv:2306. 09301, 2023a.
14

---

Renrui Zhang, Rongyao Fang, Peng Gao, Wei Zhang, Kunchang Li, Jifeng Dai, Yu Qiao, and Hongsheng Li. Tip-adapter: Training- free clip-adapter for better -anguule modeling. arXiv preprint arXiv:21 11211111.000 .33900, 2021a
Xin Zhang, Yusuke Iwasawa, Yutaka Matsuo, and Shixiang Shane Gu. Amortized prompt: Guide clip to domain transfer learning. arXiv preprint arXiv:2111.12852 202
Yabin Zhang and Lei Zhang. Adaneg: Adaptive negative proxy guided ood detection with vision- language models. arXiv preprint arXiv:2410.20149 2024.
Yi Zhang, Ce Zhang, Xueting Hu, and Zhihai He. Unsupervised prototype adapter for vision- language models, In Chinese Conference on Pattern Recognition and Computer Vision (PRCV), pages 197-209. Springer, 2023b.
Yabin Zhang, Wenjie Zhu, Chenhang He, and Lei Zhang. Lapt: Label-driven automated prompt tuning for ood detection with vision-language models. In European Conference on Computer Vision, pages 271 271-288. Springer, 2024.
Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEEE transactions pattern analysis and machine intelligence, 40(6):1452-1464 2017.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision- language models. arXiv preprint arXiv:2109.01 134, 2021.
Kaiyang Zhou, Jingkang Yang. Chen Change Loy, and Ziwei Liu. Conditional prompt learning for lisiune models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
Zhi Zhou, Ming Yang, Jiang-Xin Shi, Lan-Zhe Guo, and Yu-Feng Li. Decoop: Robust prompt tuning with outtof-distribution detection. arXiv preprint vrXiv:2406.00355 2024.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models, 2023a.
Lin Zhu, Xinbing Wang, Chenghu Zhou, and Nanyang Ye. Bayesian cross-modal alignment learning for few-shot duttof-distribution generalization. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 11461-1146) 2023b.
Lin Zhu, Yifeng Yang. Qinying Gu, Xinbing Wang. Chenghu Zhou, and Nanyang Ye. Croft: Robust fin-tuning with concurrent optimization for ood generalization and open-set set ood detection. arXiv 2024.
preprint arXiv:2405.
Lin Zhu, Yifeng Yang. Zichao Nie, Yuan Gao, Jiarui Li, Qinying Gu, Xinbing Wang, Chenghu Zhou, and Nanyang Ye. Infobound: A provable information-boune inspired framework for both ood generalization and ood detection. IEEE Transactions Pattern Analysis and Machine Intelligence, 2025a.
Lin Zhu, Weihan Yin, Fan Wu, Qinying Gu, Xinbing Wang, Chenghu Zhou, and Nanyang Ye. Bayes-cal: Robust mross-modal alignment by bayesian approach for few-shot ood generalization. International Journal of Computer Vision, pages 1-34, 2025b.
Yao Zhu, YueFeng Chen, Chuanlong Xie, Xiaodan Li, Rong Zhang, Hui Xue, Xiang Tian, Yaowu Chen, et al. Boosting out-of- distri-itttii detection with typical features. Advances in Neural Information Processing Systems, 00220758-27766 2022.
Bo Zong, Qi Song, Martin Renqiang Min, Wei Cheng. Cristian Lumezanu, Dacki Cho, and Haifeng Chen. Deep autoencoding gaussian mixture model for unsupervised anomaly detection. In International conference on learning representations, 2018.
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction clearly state the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The limitations are discussed in Appendix H
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
16

---

Justification:
The full set of assumptions is provided in our theorems and the proofs is provided Appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification:
We have provided necessary information for reproduction in Section
Guidelines:
The answer NA means that the paper does include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
17

---

Answer: [Yes]
Justification: All utilized data are sourced from open-aceess platforms. The source code will be made publicly available upon the acceptance of this paper.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run reproduce the results. See the NeurlPS code and data submission guidelines (https /nips..cc/pulli//uuui guwdes/ddee/eee Coc//ooooeeesiseeeeeee hwtpbbmmsssiooooooooiooo for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All the training and test details necessary to understand the results have been provided in Section 4 and Appendix G
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We provide the standard deviations in our main experiment results, which are commonly used as error bars, for the main experiment results under different random seeds.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
18

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative eтror rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The computational cost is analyzed in Appednix !
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
The research conforms with the NeurIPS Code of Ethics in every respect.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Given the focus on improving VLMs in both OOD generalization and OOD detection, the paper prioritizes discussing technical methodologies and results rather than societal impacts.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
19

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper aims improve VLMs in both OOD generalization and OOD detection poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We have properly credited the used assets. All our codes will be available for free to researchers for non-commercial use.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URI
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswithcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
20

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subject.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
21

---

16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does not involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy buttps://neure wwwparripp..ccoooeeeeeeeeeea /t/CCooffeeencs///2222 for what should or should be described.
22

---

A Related Works
Robust fine-tuning methods for VLM For training efficiency, there have been many lightweight
CLIP-baser fine-tuning methods to enhance generalization performance via prompt tuning (Singha
al.
2023 Huang et al.] [022] Khattak et al. 2023a] Wang et al., 2023c Wasim et al. 2023]
Goswami et al. [202]] Huang et al. (024) or adapter tuning (Gondal et al. 2024 Zhang et al.) 2023b
Song et al. 2023). Prompt tuning methods aim to get better lision-languag alignment via only
fine-tuning the input prompts. For example, with only few-shot samples for learning, CoOp (Zhou
ct al.
202T) improved significantly in generalization ability over intensively-tuna manual prompts prompt learning. Motivated by learning generalization prompts, CoCoOp (Zhou et al. 2022) is proposed to achieve generalization on unseen classes via conditional prompt learning. Adapter-tuning another popular lightweight fine-tuning method, like
(Gao al.) 2023), Tip-Adapter /hang et af. 2021a). Both of them inject a lightweight bottleneck architecture after the image encoder or text encoder and perform residual-style feature blending with the original pre-trained embeddings. However, most previous studies have focused on improving models' robustness to covariate shifts, without being able effectively address OOD detection.
Note that several studies, such as CoCoOp (Zhou et al. 2022), PromptSRC (Khattak et al. 2023b), SHIP (Wang et al. (023c), and Promptkd (Li et al. 2024b), have explored the base-to-new ability of VLMs, aiming to improve VLMs' classification performance to unseen classes. However, we emphasize that our approach differs from these methods in its handling of classes that were unseen during training. While these methods focus on classifying unseen classes, our method is designed specifically to detect these unseen classes without performing classification, ensuring safety in real- world applications. Although there have been several studies (Bai et al. 2023 Zhu et al. 2024) to handle OOD generalization and OOD detection simultaneously, these approaches are typically limited to traditional vision models or have been evaluated with a narrow set of post-hoc functions for OOD Detection. Thus, when fine-tuning VLMs for downstream tasks, the challenge of improving the models' generalization ability to closed-set OOD data while simultaneously detecting open-set OOD classes that were unseen during fine-tuning remains largely underexplored.
OOD detection methods There are multiple lines of work addressing OOD detection, such as anomaly detection (Zong et al.] 2018: Liang et al. (0077)). outlier detection (Bevandić al. 1021
Saito et al.] (02T), and open-set OOD recognition (Kong and Ramanan [021] Geng ct al. 2020
Scheirer 2014)))))). These methods can be categorized into two main groups: post hoc methods
(Zhu et al.
2022
Liu et al. 2020 Sun et al. 2021 Hendrycks and Gimpel 2016; Liang et al. 2017b
Wang al.
2022 Sun al. (022) and training time regularization (Naray aarayanaswamyy et al. 2023 Bai
cet al. 2023
Malinin and Gales] 2018 Du et al. 2022b a Ming et 2022b). The former typically
resort to post-hoe functions recognize open-set without altering the DNN training process, like density estimation (Zhang et al., 2020), uncertainty modeling (Gal and Ghahramani] (2016), and input image reconstruction (Pidhorskyi et al. 2018 Sun et al., 2020). On the other hand, regularization- based methods aim to rectify the training process, compelling models to provide predictions with lower confidence. Recent studies (Zhou et al. 2024 Bai et al.) 2024 Nie et al.) 2024: Li al. 2024a)
Wang et al. 2023a; Miyai et al.) [024bb Zhang et al. 2024]] Jiang et al. 2024 Ming et al. 2022a
Zhang and Zhang 2024] Ming and Li 2024) have explored the capability of zero-shot or few-shot
OOD detection based on VLMs. More details about the CLIP-based OOD detection methods can be seen in these surveys MMyai et al.) [024a]] Li et al., 2025). However, while these studies primarily focus on handling semantic-shified datasets, our research aims to simultaneously improve both OOD detection for semantic shifts and OOD generalization for covariate shifts, enabling more effective handling of diverse OOD datasets in real-world scenarios.
Full-spectrum OOD detection Recent works, such as OpenOOD v1.5 (Yang et al. 2022 Zhang et al. 2023a) and SEM (Yang et al. (022), have taken into account both covariate shifts and semantic shifts and introduced full-spectrum OOD (FS-OOD) detection, which considers both detecting semantic shifts and being tolerant to covariate shifts. OpenOOD v1.5 extends benchmark evaluations to large- scale datasets (e.g., ImageNet) and foundation models (e.g., CLIP and DINOv2), broadening its scope to study FS-OOD detection under both types of distribution shifts. While the FS-OOD benchmark assesses OOD detection performance across diverse distribution types, it does not primarily focus on improving VLMM' classification accuracy (i.e., OOD generalization) under covariate shifts. Moreover, in practical applications, there strong motivation to create models that can not only detect semantically shifted OOD inputs but also generalize to covariate-shifted data. Within ELIP-bases
23

---

methods, OOD detection and generalization are often discussed in separate contexts, resulting in trade-off between detection and generalization performance (Miyai al. 2024a). In this paper, we focus on improving VI.Ms' OOD) generalization ability on covariate- OOD data while simultaneously detecting semantic-shifted samples. To substantiate the effectiveness of the proposed method for both tasks, we conduct an empirical analysis of distinct categories of competitive methods, including the CLIP-based zero-shot OOD detection method, lightweight tuning-based OOD detection methods, and lightweight tuning-based OOD generalization methods.
B Proof of Theorem 3.2
We first provide the proof for property (1) Theorem 3.2 In this paper. we achieve re-aligning vision-language modalities by disrupting the top-c maximum cosine similarity to a low value. Let h1 == argmax;g(K] (x')) denote the index of the maximum cosine similarity for an OOD input x'. We then reduce the maximum cosine similarity to zero according to Equation and denote the difference between the original and re-aligned cosine similarity as Sg, (x:) Sú (x4) (xx;).
Without loss of generality, we set c 1, the energy score before re-alignment is denoted as Eg (x'), which is calculated
Eo(x') log Σ
(9)
The energy score after re-alignment is denoted Er (x'). which is calculated as:
(x'x) = -loog Σ (x()/ + ao (x))/
(10)
#
Given the ID class labels Vin the newly proposed OOD score (AEnergy) is defined as:
S SAFnergy (x': )in)
E, Ex(xx Eo(x')
K
logl Σ (x((x// c50 (x)/T log Σ (x((T
Σieg, ex((x)// + Ev (x')/T + (x')/7 (x'//
log
Eieor e-i(x"1/T + c (x/)/7
e (x'1/7 c39, (x))/T
log
1
1 Eieoo c*i(x')/T c+ (x')/T
(II)
e (x'1/7 cl (x)--(()))+
log
1 1 Eiso, ex1(xx)/= clos, (xx) -(x)))/
eH(xi)/T
log
1
Eiegy en10x/1/7
T (-:)epi +
If we set 8g, (x)) = 0, i.e., u(xx)) 882 (x'), we have:
e (x)/7
S SaEnrgg ( E, Eo(x') = log 1 Eijo ex((x
(12)
Given the ID sample Xp and OOD sample XOOD, we assume that the sum of non non-maximal cosine similarities is similar for ID and OOD samples, i.e, E,y,, (x//x/ x Eingn, e (xoco)/7 which is reasonable under uniform similarity distributions as discussed prior work (Ming et al.) 2022a). Since the SaFnergy )in) is monotonically increasing function with respect the maximum similarity 86 (x), the energy change for ID data is greater than that for OOD data.
We then provide the more detailed results and proof for property (2) in Theorem 3.2 as follows:
24

---

Theorem B.1. [Difference Amplification between ID and OOD by AEnergyl Let SaEnergy (x. Vin) and SaMCM (x,Yan Vnn) be the OOD score for any sample by the AEnergy and MCM method, respectively, where:
e (x)/7 -1
SaEnergy (x; Pin) = E1 (x) Eo(x) = log 1 +
(13)
Eiter e0i(x)/++11
e (xx//7
SAMCM (x; Din)
(14)
e (x)/T + Eijin (xi(x)
Suppose that the maximum cosine similarity for an ID sample XID is greater than that of an OOD sample XOOD, i.e., Sÿ, (X1) > Sÿ, (XOOD). We also assume that the sum of non-maximal cosine similarities is similar for and OOD samples, ie, Eitii (d((x)) Lijii (x((o0))/ which
reasonable under uniform similarity distributions as discussed prior work (Ming et al. 2022a). Then the difference between ID and OOD under AEnergy exceeds that of the MCM method:
d d&Euneryy >> dMCM
where d'Eenergy = SEnnryy (XIDi Vin) - Saenrrgy (KooD: and dMCM = SAMCM(XD: Pin) (XMCM(X00D;;
SAMCM (XOOD; Min). Thus, our AEnergy exhibits strictly stronger separability between ID and OOD distributions.
Proof: Let s = (x(x)/T, we simplify the AEnergy score as S Vin) = log(1 + (+t) and =² simplify the MCM score SAMCM Vi.) 4'bb where b Σijg, ex((x)]T
First, we analyze the gradient with respect to 881
aS Anergy
5 exin/T T + = SMMMM
(15)
Osg
0SMcM
1
e/bi
SMCCM (1- (SMMMM
(16)
0ggi
(e* +)²
Since 0 < SMOM I 1, we have
8SAEauRY Deg1
0Sarm and daRnergy > dMCM- 8eg1
Next, consider the large-scale setting where the number of IDD classes K is large. Let b R Eifgh (*((x) In this case, the denominator term b becomes significantly larger than the maximum similarity term, i.e., >> e We then perform Taylor Expansion for S AAEnegy (x, Vn) and SAMCM (x, ).n)
Taylor Expansion of SA Energy (x, Min): For b >>>>>> e", we approximate SAEnergy (x, Vin) using log(1 20
<) R e - S (x, vnm 2s
26
Taylor Expansion of SAMCM Pi:): Similarly, we expand Sx,CM(X, Vin): SAMCM (x, Vin) = T.
Then we can compute the difference between ID and OOD under the AEnergy and MCM score, respectively:
c (Xw)/T
ca (Xooo)/T ir
1
(ee2s 280 (Xw)/T
@200) (Xoc0)/1
dAEuergy ~
(17)
b
2
b2
62
c00 (xw)//
(Xw)/T
e (Xxcs)/T
2000 (Xw)/r
@ (Mucws) /T>
dMcM
(18)
62
12
which leads to the following property:
25g, (xw)/r
2*0, (xo00)/T
daFnergy dMCM
(since Sg, (xm) > 8g, (XooD))
(19)
262
25

---

In large-scale hard OOD detection scenarios where Sg, (Xi) Sgg (XOOD) (small maximum similarity gap) and K >> Sq, (Xm) > Sj, (Noon). As demonstrated Equation 17 and Equation 18 AFnergy's s logarithmic form introduces less aggressive decay in the higher-order term compared to MCM. preserving discriminability. For large K, MCM's separability diminishes quadratically (0(1 (K))),), while AEnergy decays more slowly due to the smaller coefficient in the second-order term. Thus, Á Energy is better for large-scale OOD detection where K large and the maximum similarity gap is small ($g, (XI) St (XOOD) << 1).
C Proof of Theorem 3.3
we provide formal guarantees that the proposed AEnergy can provably reduce the false positive rate (FPR) compared to the widely-used VLM-based OOD detection method MCM (Ming et al. 2022a). Before the proof of Theorem 3.3 we first introduce the MCM method as tollows:
In Theorem 3
For any test input x', we calculate the label-wise matching score based on the cosine similarity between the image feature z (xx) and the concept vector (text feature) zr(t)): sx('') = (xx) (T(4). Note that both the image feature zr (z'') and the concept vector 2144) are normalized features this paper. The maximum concept matching (MCM) score is computed as follows:
(x/)
(x) max
>==1 e3s(x')/T
Ming et.al (Ming al. 2022a have provided the formal guarantees for MCM that using softmax can provably reduce the false positive rate (FPR) compared to that without softmax, illustrated in Theorem C.2
Under the Assumption C.I
Assumption C.1. [Mingg et al., 2022a Let == 1 E V.r. Qx denotes the duttof-distribution
P =x|==0 (marginal distribution of conditioned 0). Assume 18 > 0 such that
Qx
Σ intiin
18g2 (x) - sx(x) < 1,
K
where ú := argmax,elj [[[K] (x(x) and d22 == argmax, + SE[K] (x)) denote the indices of the largest and second-lare largest cosine similarities for an OOD input x.
Theorem C.2. Miing et al. ] Given task with ID label set Din 1311 1/2 ... 3K and
pre-trained VLM. 1x satisfies Assumption C.I then there exists constant M(K 1 -5 such that for any temperature > T. we have
KX-1
FPR(T,A) FPR"
where FPR(T, X) is the false positive rate based on sofimax scaling with temperature and detection threshold X; FPR wo Aw) is the false positive rate without softmax scaling based on threshold x
Now present the proof of Theorem 3.3
follows:
26

---

Proof: The newly proposed OOD score (AEnergy) is defined as:
S Anerrg (x': :ma) E1 Eu
K Σ c<((XO// + (2n1(11/7 + log Σ
- lool
ex(x1//T
1/11
i=1
E Eistie ex1(x2//+ + e (x 1/7 + e0 (x (x)(1/
log
Eieor ex1(x)// + e0n (x)/TT
(0 - 2x)1111//
log
1+
Eiph ea((x(//T c (x1)/T
e'5)(x2)/T 5 (x/)/T
e -
E Eig exx(xx//T e (x))/T
Σ ea((x')/T e (x2)/7 e"ú (x0)/T
1x1
exxx\/1 + 26 (x')/= e²01 (x))/7
(20)
When c E (2,... K). without loss of generality, we take c = as example and we have:
SaEnergy vin)
E1
Ep
1
logl <x(x"// + +=11(x0)/ log
(x()//
2
K
+
2
1
log
<x((x'1/7 + [000(x)//+] log
(x((x)) 1/7
(21)
1
e (x'/T
2
Di=1
K
(x,('))/7 1/7 + (x)1/111/7 e01
(x))
+
2
2i=1 edi(x"1/7 + 20 (x')TT (x11/7
For if (x )/r c6 (x1)/T 2 0, i.e, (x) - 150 (x') S 2n we have
(x)/1
c (x')////>>
(A
Σ ex((x1//7 + 2 (x)) (x9 -/)()))-))) 5
Σ cx((x///T = (Xi
For e = 2, we have sg, (x)()))))))))))))))"))) S.y (1 we have
0, i.c. (x') -x (x) S 11n2
(x))T
e (x)/T
DK ex((x'//T
Σx K
Exyestte es((x')/+
20 (x))/7 (x')//
(22)
(x')/
S ΣK ea(x')) ea((x'//
(MCM(('))I.)
which leads to
SaFnergy (in) S SMCM(X(;P..
Let the OOD detection functions be represented by:
G(x';Din) = {
SaEnegy vin) 2 A
(23)
SAEnery (x: xmm)m <
27

---

then we have
FPR FPR AEnergy
(r, A)
PP = 1| 0)
Qx (G((z' =
(24)
Qx (Saenergy (x') > A)
Qx (SMCM(x' :ii)) A) = FPR MPMM (T,
(T
Thus, complete the proof.
D Proof of Theorem 3.4
Proof: To further enlarge the energy change between the masked VLM and unmasked VLM for closed-set classes, we propose to minimize the following loss:
N/ Σ
1 (x((x) Fx((1) N
LAE
(25)
where (x:) is the energy score for x; after masking on the image feature, which is formally calculated as:
Ex(x1) = - log (xi)
j=1
(x:) = (21 O m' (xi)) ZT (tz)))
where mx;x is the mask that retains the top p-proportion elements in zr(x:) @ h1 (xi)
AEnergy(xi). Here, we represent the opti- mization term for xı as: LAE(xi) E2(x1) Ex(x)). Then the relationship between AEnergy (xi) and Ca (x;) can be formulated as:
Now we prove that -LAE is the lower bound of >M
Ej Ef/xill +elh (x1//T
(xnergy(xs) e- (xi((X))
Ejjn, (x)(x/// + c (xi)/T
Σ e3,(x0)/7
Ejeor Aslaill + t (xi)/T
Ejto, e*)(x))/ + c² (xi)/T
(26)
ethi (xil/7 - (2)//111/7 (mi)/7
K E - (x)(1)
Ejtú, (s)(x)/5 + etn (xi)/T
E ess(x//7
(xil/ - e (xi)/7
Ejeth ea (xi)/T + e (x1)/T
where the inequality in the last line follows šg. (xi) S $z, (xx).
Under the condition that e² (x1)/ - e (xi)/T 2 (e - 1) Ex= (x))/T (c - 1)e Fa(x;)
and that LAE S Sg it is straightforward to derive the following inequality:
K
(v)00)// - 5o (xI)/T
Σ
S0xi/ - (x)(x
j=1
Thus we have AEnergy (x;) 2 (xAE(xi). Thus complete the proof.
E Proof of Theorem 3,5
Proof:
In the prompt-tuning framework of our proposed EBM, only n context vectors are learnable,
and we denote the learnalle context vectors as 0 (0,,... e,)). The LAE be represented by:
N
K log exp (x)) log > j=1
K
LAE =
Σ
exp (m (x:) ZT (ty:00)
(27)
N
28

---

where m1(x;) = z1(x4) ② m (x;) is the masked image feature. Now we expand VeLse as follows:
N
Ve j=1 exp (ZT ZT;;
Ve Ej exp (mr (xi) x 2T
VoLaE
j=1 exp (T(())
Σj exp (mr(( 2t (1z 0))
(28)
We denote
a0
log Σ
exp (2n (T(zi:
N
j=l
a1
1 N
log
Σ exp (xi), (T(;;))
Then have VOLLE-
V Voa1. The local optimization of LAE lead to Vaao Voa
Let 8 represent the cosine similarity between the image feature zxi) and the text feature cor- responding its ground-truth label. The empirical classification loss, ED(O), be calculated
as:
1
exp (()
(D)
Σ
log
N
j=l exp (x)) zT(;;
1=1
(29)
N
K
1
log
exp (x1(),) Zr (4yi 0)) -5(3)
N
Accordingly, the gradient vector of empirical risk (e) with respect to parameter 0 represented
as:
Ve Σ K jj= exp (z))
N
GD = V (0)
1 N i=
Vas(i
Vas
Σ exp (2I (t;;;;
N
(30)
And the Hessian matrix of empirical risk with respect parameter 0 is calculated as:
N
(e) = vgEn(0) = Vgaa
Past
(31)
N
The local optimum solution of Equation
i.c., VeLsE = 0, gives the following equation:
N
Hs(0) H N i=1
Σ
((S(())
1
vo (x) -m(") zr(x;)) (32)
N
Finally, we can conclude that the local optimum solution of Equation Caatioonnlade leads to the following property:
10 (fs(() [s)o0) T
vol(z((xi) mn()) 2T (x:)0]] <0())
(33)
F
Proof of Proposition 3.6
Let 0* be the local minimum across all domains, i.e., (0*) and D (S, (S, By Taylor expansion, the OOD generalization gap between source domain (S) and target domain (7) is
Proof:
29

---

Table 5: Conventional OOD detection Results: OOD detection performance for ImageNet- 1k ID. In the table, we extend our A Energy method to the zero-shot method CSP. leveraging the informative information from extra OOD labels as detailed in Equation 36
Texture
iNaturalist AUC1
Places
SUN
Avg
Method
AUC1
FPR95L
FPRO5
AUC1
FPR95..
AUC
FPR95]
AUC1
PPR95_
CLII--based post-hoc methods
MSP
74.84
73.66
77.74
74.57
72.18
79.12
73.97
76.95
74,98
76.22
MaxLogit
88.63
48.72
88.03
60.88
87.45
55.54
91.16
44,83
88.82
52.49
Energy
50.39
87.18
64.98
87.33
57.40
91.17
46.42
88,48
54.80
ReAct
88.13
49.88
86.87
65.57
87.42
56.85
91.04
46.17
88.37
54,62
ODIN
$7.85
94.65
30.22
85.54
55.06
47.75
muning-based methods
NegPrompe
91.60
35.21
98.73
6.32
93.34
27.60
95.55
22.89
94.81
23.01
ID-Like
94.32
25.27
98,
8.98
91.15
41.74
91.64
42.03
93.83
29.51
LoCoOp
90.19
42.28
96.86
16.05
91.98
32.87
95.07
23.44
93.52
28.66
LSN-CoOp
89.52
95.47
36.43
29.84
LSN-CoCoOp
90.42
38.54
95.83
21.56
91.25
34.48
94.35
26.32
92.96
30,22
Ga
90.40
97.10
13.70
91.30
32.50
24.90
7em shot methods
MCM
86.11
57.77
94.61
30.91
89.77
44.69
92.57
34,59
90.76
42.74
CLIPN
90.93
95.27
23.94
92.28
33.45
26.17
31.10
NegLabel
90.22
43.56
99.49
1.91
91.64
35.59
95.49
20.53
94.21
25.40
99.60
29.32
CSP+-Energ (Ours)
94.33
21.44
99.72
0.82
92.66
28.87
96.60
13.75
95.83
16.22
Table 6: Hard OOD detection Results #1: OOD detection measured by AUROC and FPR95 over different splits of ImageNet- Details of the 4 splits are in Table g
Split-1
Split-2
Split-3
Split-4
Avg
Method
AUC
FPR95,
AUC
FPR95
AUC1
FPR95
AUC1
FPR95
AUC
FPR95,
MCM
97.93
9.17
88.10
56.40
90.34
33.05
98.72
4.73
93.77
25.83
CLIPN
99.38
2.07
97.77
10.55
90.03
36.85
98.83
4.68
96.50
13.53
MSP
77.85
63.60
68.73
83.63
79.10
70.55
82.40
65.52
77.02
70.83
MaxLogit
99.87
0.49
98.06
34.34
99.35
97.06
Energy
99.88
0.46
98.18
8.40
90.65
35.02
99.36
2.83
97.02
11.68
ReAct
99.34
0.72
97.91
9.33
90.72
35.65
99.12
2,94
96.77
12.16
ODIN
98.78
1.12
98.23
8.18
89.92
37
98.76
13.20
96.42
14,92
AEnergy (Ours)
99.93
0.37
99.00
5.16
91.14
30.09
99.40
2.83
97.37
9.61
upper bounded as shown in the following equation:
( Ee()")
max
(0-)))
(7') + Le ( Ess(8))
(0: e (00))0)
$ (8") εs(0')
max
0 (
(34)
[o: (0
2
$ (T7(00) εs(0*))
0 ['[īi(00)) -i((0) f
max
(0: [0:310T (0-)0
x (70) Es(o') max
e [17) Hss(0')0)+ (00))= 2
For each image feature 21 from the source domain, the image features ž from the target domain, which share the same label with 21, is assumed satisfy: || 2 21|2 S 61- Since optimize the EBM loss based on the unmasked image features and masked image features, we have the following approximation:
(0*) (e)) < ((s))
(355
G More Experiment Results
More experiment details We present experiment details for the baseline models follows:
30

---

Table 7: Hard OOD detection Results #2 Comparison with state-of-the-an zero-shot methods on
hard OOD detection datasets. In the table, OOD detection measured by AUROC and FPR95 over hard OOD detection datasets. Details of those datasets can be seen in prior researches (Chen et al.
2024 Ming al. 2022a).
ImageNet-10 InaggNNe-20
ImageNet-200 InagocNet110
ImageNett11
ImageNet- 100 ImageNet-10
ImageNet-IR ImageNet-O
WaterBirds Placesig
OOD datasets
ImageNaa-100
Method
AUROC
FPR95
AUROC
FPR95
AUROC
FPR95
AUROC
FPR95
AUROC
FPR95
AUROC
FPR95
MCM
6.00
98.09
13.04
2.50
87.20
78.59
64.27
87.45
33.62
NegLabel
98,90
5.00
98.04
11.60
99.37
2.50
87.93
49.40
85.78
56.65
87.99
29.16
CSP
99.02
3.30
98.79
3.40
99.33
2.22
89.59
42.40
88.08
51.50
92.88
12.07
AFwergy (Ours)
99.01
90.49
41.25
Table The 4 ImageNet- splits for hard OOD detection following the prior work (Li et al. 2024a).
Given are the numbers of classes training / test samples.
ID
OOD
Split-1
All dog classes
Non-animal classes
116: 1856 5800
166: - 8300
Split-2
Half of hunting dog classes
Other 4-legged animal classes
30: 480 1500
55: / 2750
Split-3
Mix of common classes
Mix of common classes
151: 2416 7550
164: - 8200
Split-4
First 100 classes
Remaining 900 classes
100: 1600 5000
900: /45000
Algorithm Algorithm the proposed EBM method
1: Input: ID data (xi, J (i E 1,... N). ID class names of the K-way classification, masking proportion p. text prompts (4,422 (k, hyperparameter λo and maximum epoch
2: for ( 1 to T do
3: Calculate the ID image features 2I(x)) and fine-tuned text features zr(t;; 0) when prompt-
tuning the VLM;
4: Compute the cosine similarity between image features 21 (x1) and text features zr(t;; 0) and denote the text feature with the top-1 similarity as h, (x;: 0)
5: Compute the element-wise product Zp := zı(x: @ h1 (x;; 9) and generate the mask, denoted m' (x:), which retains the top p-proportion elements in Zp (x;);;;
6: Perform masking on the image feature and represent the masked image feature as 21(x;) O
m' (x:):
7: Gradient update under the proposed loss as illustrated in Equation
8: end for
9: Output: Learnable content vectors 0.
For zero-shot OOD detection methods such CSP and NegLabel, all hyperparameters and OOD score calculation procedures are directly adopted from their respective papers (Chen et al. 2024
Jiang et al. 2024) without modification.
, we train models with SGD optimizer with a learning rate of 2e 2. The batch size is set to 32 for all tuning-based experiments. For the specific hyperparameter for each method, we follow the setting the original paper.
For the tuning-basee methods, based on the code of CoOp (Zhou et al. 2021
For the tuning-based methods for improving performances on closed-set data, such as CoOp Zhou et al. 2021), CoCoOp (Zhou et al. 2022), DPLCLIP (Zhang et al., (021b), and Bayes-CAL CAL iZhu al. 2023b), we use random initialization for context vectors and the number of context tokens as 16, set the class token position (CTP) as "end", and set the class-specific context (CSC) as "False". This configuration has shown the best average performance according to CoOp's paper. For DPLCLIP (Zhang et al., 2021b) method, we set the additional hyperparameters of DPLCLIP (Zhang al., 202Tb) as: "mip_ deilp__deph="", "mip_widee width=512", and "mip_drop crlp_ddroouut0.11 For the CI.Ie-Adapter (Gao et al. 2023) method, adopt image adapter only with the residual ratio of 0.5 for Sctup-I and
31

---

Data
Closed-Set
Closed-Set
Open-Set
Closed-Set
Closed-Set
Open-Set
Closed-Set
Clused-Set
Open-Set
OOD
OOD
OOD
Model
Text Prompt: photo of wheelbarrow
Text Prompt: a photo of haseball player
Text Prompi: photo a giant panda
Original CLIP
zs
CLIP(+)
zs CLIP (+)
Model
Text Prompt: photo a searf
Text Prucupt: photo of
Text Prumpt: photo of suxophune
Original 7S CLIP
CLIP(+)
28 (-)
Figure 3: The significant prediction difference between closed-set data and open-set OOD data when vision-language re-alignment is applied to the zero-shot CLIP model (Radford et al. 2021) This difference offers a novel approach to distinguishing between closed-set and open-set classes. Based on the element-wise product between CLIP's image and text features, the masked 7S CLIP(+) model zeroes out the elements negative. In contrast, the opposite operation is applied in ZS CLIP(-). It is observed that masking the elements where P2 < 0 preserves the model's original attention, which motivates us to leverage this consistency between the original and masked domains to improve OOD generalization.
the image feature where the corresponding values in the product are
0.2 for Setup-II, and we use the bottleneck adapter with a hidden dimension that is 1/4 of the original feature dimension. This hyperparameter configuration has been demonstrated as the most effective for generic image datasets, such as ImageNet, the original research (Gao et al. 2023).
For tuning-based OOD detection methods, we adopt the recommended hyperparameter settings for LoCoOp, NegPrompt, and GalL.o.. For LoCoOp (Miyai al.) (024b)), following the original paper, set À 0.25. The hyperparameter K is searched over the range [I00, 200] for Sctup-I, and [2, 3, 4, 5] for Setup-II, based on validation data. For GalLoP, we follow its publicly available source code and adopt the same hyperparameter settings reported in Table 3 of (Lafon al. 2024), including configurations for local prompts, global prompts, tokens per prompt, and other relevant settings. In the NegPrompt method (Li et al., 2024a), we follow its source code and train the model using all the classes from the ID dataset and train shared positive prompt and two shared negative prompts w.r.t. each training ID class. The hyperparameters 3 and y set to 0.1 and 0.05, respectively. In the first stage, CoOp is trained for 100 epochs to obtain the positive prompts. In the second stage, the positive prompts frozen, and model is trained for 10 epochs to learn the negative prompts. During the testing phase of LoCoOp, NegPrompt, and GalloP, we use the GL-MCM score (Miyai et al. 2023) to compute OOD detection results.
For experiments on each method, we repeat 3 times with different random splits to eliminate the effects of randomness. The hyperparameters in each method are selected based on the test accuracy on validation sets.
AEnergy based on negative OOD labels The NegLabel (Jiagg et al. 2024) and CSP (Chen et al. 2024) methods introduce massive negative labels to boost OOD detection. The extended label space provides a novel perspective to distinguish OOD samples, leveraging extra clues by examining the similarities between images and the extended labels. The CSP method extends the Negl Label by "make up" the OOD label candidates, which are not standard class names but beneficial for the process.
32

---

features, thereby enhancing the Table 9: Ablations on the hyperparameter c. Energy achieves the overall best performance on both AUROC and FPR95 when c 2.
Data
c=l
022
CR3
c=1
CE5
FPR
AUROC
FPR
AUROC
FPR
AUTROC
FPR
AUROC:
FPR
AURRCC
FPR
ATROC
IT) VS Open-Set ODD
49 60
86.46
46.40
87. 10
47.23
86.8
47.
86.44
48.13
86.06
49.97
K5.69
Closed OOD VS Open-Set OOD
66.54
78.49
67.16
78.68
68.30
78.43
68.10
78.07
68.01
77.79
69.27
77.50
Table 10: Performance of ImageNet- K-trained model on the sets with covariate shifts (such as ImageNet_ V2, ImageNet_R, ImageNet_A and ImageNet_S) and concept shifts (such as ImageNet- Superclass).
Dataset
ImageNet (ID)
ImageNet_
ImageNet_R
ImageNet_A
ImaggeNet_S
ImageNat-Superelas
Avg OOD
CLIP
68.80
73.97
46.09
47.77
60.90
33.18
52,38
CoOp
71.86
76.00
48.34
50.13
64,23
36.90
55.12
CoCoOp
64.07
37.17
CoDp+EBB (Ours)
71.70
77.10
49.02
$1.35
64.78
38.24
$6.10
Thus, we also extend our AEnergy method to CSP, leveraging the informative information from extra OOD labels. Let Vin denote the ID labels and VOOD denote the OOD labels. Thus, Annergy is calculated as follows:
AEnergy(x; = = AEnergy(x; )..) A Energyyxx; Voooo
(36)
Ablations on the hyperparameter e We conduct ablation studies on the hyperparameter e, focusing on its effect the discrimination between closed-set data and open-ser OOD data using ImageNet- 1k.
The AUROC and FPR95 performances are reported in Table The results demonstrate that: 1) There trade-off between the AUROC and FPR95. 2) AEnergy achieves the overall best performance on both AUROC and FPR95 when e = Therefore, unless otherwise specified, we set e = 2 in A AEnergy for all experiments.
OOD generalization performance under concept shift We conduct experiments on the concept- shifted SupperllttS dataset (Xiao et al. 2024 Santurkar et al., 2020), where each im- age is annotated with its corresponding superclass label. For the superclass labels, we use the open-source annotations provided in https: giggubu...... com/s S www.//oue//o////oaaaaaeeaaaaaaae wttp:///eemeeeeeesseeee....e.eee http//aainnsuprrceeeee
names txt Table T0 we report the performance of our ImageNet- 1K-trained model on test sets annotated with superclasses. From Table our EBMM-aased method outperforms baseline models under both covariate and concept shifts. These results demonstrate the effectiveness of our proposed approach in learning model's s ability to generalize under various distribution shifts.
Fine-tuning accuracy of the proposed EBM on standard datasets used in CLIP We also evaluate the effect of the proposed EBM loss on fine-tuning accuracy across 11 standard datasets used in CLIP. We implement the EBM loss on PromptSRC and train the models using 16-shot settings with ViT-B/16 backbone. We train 50 epochs for Imagenet and 200 epochs for other datasets with SGD, following the same training setup as in PromptSRC. The EBM hyperparameter was set to p% 0.6. and we searched over the values of (0.1, 0.5. 1,.,1.100 1.0, 2.0).
The performance results are reported in Table [1 demonstrating that the proposed EBM method further improves test accuracy on PromptSRC. Since the objective of EBM is minimize C g. encourages the model to make equally high-confidence masked features, thus facilitating the learning of predictions for both the original and partially features between the original domain and the masked domain. The improvements observed in Table I1 empirically suggest that regularization on masked features effectively enhances the model's generalization performance.
H Limitations and Future Work
As demonstrated in Theorems 3.2 3.3 and B.I our method outperforms the MCM approach by enlarging the difference between the ID and open-set OOD samples. The empirical results in Tables 2 and Tables 7 further support our theoretical findings. While our method also improves performance on conventional OOD detection benchmarks, the gains are less pronounced compared to the hard OOD scenarios. This may be due to the reduced impact of amplifying the distinction between
33

---

Table 11: Performances of fine-tuning accuracy on 11 standard datasets used in CLIP.
Data
ImogeNet
Callech101
OsfordPets
Cars
Flowers
Food101
Airerall
SUN397
DTD
EuroSAT
UCF101
Avg
CLIP
66.7
92.2
88.4
65.5
70.7
84.8
24.8
62.3
44.1
48.3
64.7
64.8
CaOp
CoCoOp
71.0
95.2
93.3
71.6
87.8
87.2
31.2
72.2
73.3
78.1
74,9
MaPLe
72.3
96.0
92.8
83.6
97.0
85.3
48.4
75.5
71.3
92.3
85.0
$1.8
PLOT
72.6
96.0
93.6
84.6
97.6
87.1
46.7
76.0
71.4
92.0
85.3
52.1
PrompeSRC
Ours
73.6
96.5
94.4
85.3
98.2
87.6
51.3
77.3
73.3
93.5
86.7
83.4
Table 12: Comparison of computational efficiency between method and prior approaches.
Category
Method
Time
GPU (MB)
Batch Size
17s
Zero-shol
AEnergy (Ours)
18s
1706
100
CoOp
18min
7658
32
LoCoOp
25min
8320
32
Fine-tuning
GalLoP
140min
47492
32
EBM (Ours)
18min
7756
32
ID and OOD samples when the inherent difference between ID and open-set OOD samples is already substantial. Future work may explore strategies further enhance AEnergy by incorporating CSP's informative negative labels.
I Computation Efficiency
The proposed zero-shot OOD detection method does not require fine-tuning of VI.M parameters. Instead, it introduces novel post-hoe scoring function to identify open-set OOD samples. All computations are performed the latent space during the alignment between vision and language representations, enabling improved OOD detection performances with comparable inference time and computational cost to existing methods such as MCM and raw energy scores. Compared to methods like NegLabel and CSP, AEnergy does not rely additional negative labels, making it more computationally efficient. For tuning-base based approaches, our proposed EBM method enables joint optimization for both OOD generalization and OOD detection by the novel optimization objective representations in the latent space without introducing extra prompts, required by vanilla CoOp (Zhou et al. 2021). thereby it is more efficient over methods such as LoCoOp (Miyai et al.) 2024b) and GalLoP (Lafon et al.) (022).. A detailed comparison of computation cost is provided in Table 12 Here, GalLoP oP is trained using four NVIDIA RTX 4090 GPUs, while all other experiments are conducted on single NVIDIA RTX 4090 GPU.
as defined in Equation g It re-aligns vision-languee
34


Reviews:
[{"review_id": "Q0EdBeoHpF", "reviewer": "Reviewer_7zhN", "summary": "Motivated by heuristic observations regarding the energy change of cosine similarity in closed-set and open-set data, this paper introduces a novel out-of-distribution (OOD) score, $\\Delta Energy$, which simultaneously improves both OOD detection and OOD generalization performance of vision-language models during prompt fine-tuning. Theoretical analysis is provided to demonstrate that the proposed method guarantees better theoretical performance in OOD detection and generalization compared to the baselines. Evaluation on several OOD detection and generalization benchmarks further demonstrates the effectiveness of the proposed method for classification tasks.", "questions": "1. In the proposed method, the objective is designed to raise the lower bound of $\\Delta Energy$ to improve OOD detection. However, the effect of increasing this lower bound on classification performance in closed-set data is unclear. I am concerned that this may negatively impact classification performance on closed-set data.\n\n2. At the evaluation stage, the performance under another common out-of-distribution scenario, i.e., concept shift, is not assessed. It is suggested to include this evaluation to determine whether the proposed method can be applied to a broader range of out-of-distribution scenarios.", "limitations": "**Limitation:**\n\n1. In this paper, the proposed method is specifically designed, discussed, and evaluated for distribution shift caused by covariate shift (i.e., $P_{train}(X) \\ne P_{test}(X)$ and $P_{train}(Y|X) = P_{test}(Y|X)$). However, its performance under another common out-of-distribution scenario, namely concept shift (where $P_{train}(X) = P_{test}(X)$ and $P_{train}(Y|X) \\ne P_{test}(Y|X)$), remains unclear, which significantly limits its applicability.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths:**\n\n1. This paper is well-written and easy to follow. The heuristic observations behind the introduced out-of-distribution score are interesting and provide a straightforward approach to improving both out-of-distribution detection and generalization performance for vision-language models during prompt tuning.\n\n2. A detailed analysis is provided to demonstrate that the proposed method can improve both out-of-distribution detection and generalization performance from a theoretical perspective.\n\n3. The empirical results on commonly used OOD datasets demonstrate the effectiveness of the proposed method in improving both OOD detection and generalization performance compared to existing methods.\n\n**Weaknesses:**\n\n1. In this paper, the proposed method is specifically designed, discussed, and evaluated for distribution shift caused by covariate shift (i.e., $P_{train}(X) \\ne P_{test}(X)$ and $P_{train}(Y|X) = P_{test}(Y|X)$). However, its performance under another common out-of-distribution scenario, namely concept shift (where $P_{train}(X) = P_{test}(X)$ and $P_{train}(Y|X) \\ne P_{test}(Y|X)$), remains unclear, which significantly limits its applicability.\n\n2. In the proposed method, the objective is designed to raise the lower bound of $\\Delta Energy$ to improve OOD detection. However, the effect of increasing this lower bound on classification performance in closed-set data is unclear. I am concerned that this may negatively impact classification performance on closed-set data. It is recommended to verify this by evaluating the fine-tuning accuracy of the proposed method on standard datasets used in CLIP [1].\n\n[1] Radford, Alec, et al. \"Learning transferable visual models from natural language supervision.\" International conference on machine learning. PmLR, 2021.\n\n3. At the evaluation stage, the performance under another common out-of-distribution scenario, i.e., concept shift, is not assessed. It is suggested to include this evaluation to determine whether the proposed method can be applied to a broader range of out-of-distribution scenarios.", "was_revised": true, "final_justification": "The authors' clarifications and additional experimental results has effectively addressed most of my concerns. After carefully considering the authors' responses and other reviews' comments, I have decided to raise my original rating to 4.", "rebuttal": "We thank the reviewer for the constructive comments. Below, we provide detailed responses to all the questions raised:\n\n> **R1. Reply to ''the performance under concept shift remains unclear'':**\n\nIn our method, we primarily focus on enhancing the robustness of vision-language models under covariate shifts, while simultaneously enabling the detection of open-set semantic shifts. Specifically, **we keep the class names consistent with those used during training in the test phase—a common and practical setting in real-world applications**—and meanwhile aim to prevent the model from incorrectly assigning open-set semantic-shifted OOD samples to any ID class.\n\nFollowing your valuable suggestion, we conduct experiments on the concept-shifted ImageNet-Superclass dataset [1], where each image is annotated with its corresponding superclass label. For the superclass labels, we use the open-source annotations provided in [1]. In Table 1-1, we demonstrate examples of ImageNet-1K training data alongside concept-shifted test samples. In Table 1-2, we report the performance of our ImageNet-1K-trained model on test sets annotated with superclasses. From Table 1-2, our EBM-based method outperforms baseline models under both covariate and concept shifts. These results demonstrate the effectiveness of our proposed approach in learning domain-invariant features, thereby enhancing the model's ability to generalize under various distribution shifts. All experimental results and related work [2, 3] on concept shifts will be incorporated into the later version of our manuscript to improve its overall quality.\n\n[1] Soichiro Kumano, et al. Dataset Source: github.com/s-kumano/imagenet-superclass/blob/main/superclass_names.txt\n\n[2] Xiao, Zehao, et al. \"Any-shift prompting for generalization over distributions.\" CVPR, 2024.\n\n[3] Santurkar, Shibani, et al. \"Breeds: Benchmarks for subpopulation shift.\" *arXiv preprint arXiv:2008.04859* (2020).\n\nTable 1-1.  Examples of the ImageNet-1K training data and the corresponding concept-shifted test data.\n\n| DATA               | Imagenet-1k    | Concept Shift           |\n| :----------------- | -------------- | ----------------------- |\n| Total Class Number | 1000           | 558                     |\n| **Folder**         | **Class Name** | **Class Name**          |\n| n01440764          | tench          | cyprinid, cyprinid fish |\n| n01443537          | goldfish       | cyprinid, cyprinid fish |\n| n01530575          | brambling      | finch                   |\n| n01531178          | goldfinch      | finch                   |\n| n01532829          | house finch    | finch                   |\n| n01534433          | junco          | finch                   |\n\n\nTable 1-2. Performance of ImageNet-1K-trained model on the test sets with covariate shifts and concept shifts.\n\n| Dataset             | ImageNet (ID) | ImageNet_V2  (covariate shift) | ImageNet_R (covariate shift) | ImageNet_A (covariate shift) | ImageNet_S (covariate shift) | ImageNet-Superclass (concept shift) | Avg OOD Acc |\n| ------------------- | ------------- | ------------------------------ | ---------------------------- | ---------------------------- | ---------------------------- | ----------------------------------- | ----------- |\n| CLIP                | 68.80         | 73.97                          | 46.09                        | 47.77                        | 60.90                        | 33.18                               | 52.38       |\n| CoOp                | **71.86**     | 76.00                          | 48.34                        | 50.13                        | 64.23                        | 36.90                               | 55.12       |\n| CoCoOp              | 71.10         | 76.18                          | 48.75                        | 50.63                        | 64.07                        | 37.17                               | 55.36       |\n| **CoOp+EBM (Ours)** | 71.70         | **77.10**                      | **49.02**                    | **51.35**                    | **64.78**                    | **38.24**                           | **56.10**   |\n\n\n\n> **R2. Reply to '' I am concerned that EBM may negatively impact classification performance on closed-set data.  It is recommended to verify the fine-tuning accuracy of the proposed method on standard datasets used in CLIP'':**\n\nAs defined in Equation 4, the proposed objective function—namely, the $\\Delta\\mathrm{Energy}$-based bound maximization (EBM)—aims to increase the lower bound of the $\\Delta\\mathrm{Energy}$ for closed-set classes, as demonstrated in Theorem 3.4. We have theoretically shown its impact on closed-set OOD data, particularly on data exhibiting covariate shifts, in Theorem 3.5. The results in Theorem 3.5 indicate that optimizing Equation 4 leads to domain-consistent Hessians of the classification loss, which implies improved model generalization on closed-set data [4, 5]. \nEmpirically, this theoretical advantage is supported by the OOD accuracy gains observed in Tables 3 and 4 of our manuscript, as well as by the ablation study on EBM presented in Table 2-2 of our response to Reviewer 3pyx, further highlighting the effectiveness of EBM in enhancing OOD generalization.\n\nFollowing your insightful suggestion, we evaluate the effect of the proposed EBM loss on fine-tuning accuracy across 11 standard datasets used in CLIP. We implement the EBM loss on PromptSRC [6] and train the models using 16-shot settings with a ViT-B/16 backbone. We train 50 epochs for Imagenet and 200 epochs for other datasets with SGD, following the same training setup as in [6]. The EBM hyperparameter was set to $p\\\\% = 0.6$, and we searched $\\lambda_0$ over the values of $\\\\{0.1,0.5,1.0,2.0\\\\}$.\n\nThe performance results are reported in Table 1-3, demonstrating that the proposed EBM method further improves test accuracy on PromptSRC. Since the objective of EBM is to minimize $\\mathcal{L}_{\\Delta E}=E_2 - E_0$, it encourages the model to make equally high-confidence predictions for both the original and partially masked features, thus facilitating the learning of domain-invariant features between the original domain and the masked domain. The improvements observed in Table 1-3 empirically suggest that regularization on masked features effectively enhances the model’s generalization performance.\n\n[4] Rame, Alexandre, et al. \"Fishr: Invariant gradient variances for out-of-distribution generalization.\" ICML, 2022.\n\n[5] Hemati, Sobhan, et al. \"Understanding hessian alignment for domain generalization.\" ICCV, 2023.\n\n[6] Khattak, Muhammad Uzair, et al. \"Self-regulating prompts: Foundational model adaptation without forgetting.\" ICCV, 2023.\n\n\nTable 1-3. Performances of fine-tuning accuracy on 11 standard datasets used in CLIP.\n\n| Data      | ImageNet | Caltech101 | OxfordPets | Cars     | Flowers102 | Food101  | Aircraft | SUN397   | DTD      | EuroSAT  | UCF101   | Avg      |\n| --------- | -------- | ---------- | ---------- | -------- | ---------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- |\n| CLIP      | 66.7     | 92.2       | 88.4       | 65.5     | 70.7       | 84.8     | 24.8     | 62.3     | 44.1     | 48.3     | 64.7     | 64.8     |\n| CoOp      | 71.7     | 95.6       | 91.9       | 83.1     | 97.1       | 84.2     | 43.4     | 74.7     | 69.9     | 84.9     | 82.2     | 79.9     |\n| CoCoOp    | 71.0     | 95.2       | 93.3       | 71.6     | 87.8       | 87.2     | 31.2     | 72.2     | 63.0     | 73.3     | 78.1     | 74.9     |\n| MaPLe     | 72.3     | 96.0       | 92.8       | 83.6     | 97.0       | 85.3     | 48.4     | 75.5     | 71.3     | 92.3     | 85       | 81.8     |\n| PLOT      | 72.6     | 96.0       | 93.6       | 84.6     | 97.6       | 87.1     | 46.7     | 76.0     | 71.4     | 92.0     | 85.3     | 82.1     |\n| PromptSRC | 73.2     | 96.1       | 93.7       | 83.8     | 97.6       | 86.5     | 50.8     | 77.2     | 72.7     | 92.4     | 86.5     | 82.8     |\n| **Ours**     | **73.6** | **96.5**   | **94.4**   | **85.3** | **98.2**   | **87.6** | **51.3** | **77.3** | **73.3** | **93.5** | **86.7** | **83.4** |\n\n\n>  **R3. Reply to ''Whether the proposed method can be applied to a broader range of OOD scenarios'':**\n\nWe thank the reviewer for the insightful comment. Following your insightful suggestion, we have conducted additional experiments on concept shifts and evaluated model's fine-tuning accuracy on 11 standard datasets used in CLIP. The improvements observed in Tables 1-2 and Table 1-3 demonstrate that the proposed EBM loss, which is designed to facilitate the learning of domain-invariant features between the original and masked domains, can effectively enhance the model’s generalization performance. All experimental results reported in Tables 1-2 and 1-3, along with the related work on concept shifts [2, 3], will be incorporated into a later version of our manuscript to improve its overall quality."}, {"review_id": "ft4ZKUaVZ0", "reviewer": "Reviewer_3pyx", "summary": "This paper proposes a novel method to enhance the robustness of Vision-Language Models (VLMs) on two critical tasks simultaneously: out-of-distribution (OOD) generalization to covariate shifts and OOD detection of semantic shifts. The core contributions of the paper are: 1) A zero-shot OOD detection score, ∆Energy, which is based on the insight that this energy change is substantially larger for in-distribution (ID) data than for OOD data. 2) A unified fine-tuning framework, named EBM (Energy-based Bound Maximization). Minimizing the EBM loss not only increases the lower bound of the ∆Energy score but also encourages the model to learn a domain-consistent Hessian matrix.\n\nExtensive experiments on OOD detection and generalization benchmarks validate the effectiveness of both ∆Energy and EBM.", "questions": "1.\tIntuitive explanation of the EBM loss: The goal of EBM is to minimize L_∆E = E_2 – E_0. Minimizing their difference intuitively encourages the model to make equally high-confidence predictions for both the original and partially masked features. Could you please explain intuitively how this objective guides the model to learn a \"domain-consistent Hessian matrix\"?\n2.\tHyperparameter Selection: The method introduces new hyperparameters, such as c in ∆Energy and the masking proportion p and loss weight λ_0 in EBM. As shown in Table 10 of the appendix, the choice of p has a noticeable impact. Could you elaborate on the selection strategy for these hyperparameters? Is there a principled way to set them, or do they require extensive validation?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1.\tNovel Insight: The core idea of the paper—using the \"energy change\" (∆Energy) after perturbing the vision-language alignment as a basis for OOD detection—is novel. \n\n2.\tUnified Framework: Building upon the ∆Energy, the paper integrates optimization of OOD generalization and OOD detection in one framework. Theorem 3.5 links the EBM loss to a domain-consistent Hessian matrix, skillfully provides a strong justification for why this unified approach improves OOD generalization.\n\nWeaknesses:\n1.\tClarity of the Methodology: Although the overall idea of the paper is clear, some parts of the methodology (Section 3) are presented in a dense and abstract manner, which can make them difficult to follow.", "was_revised": true, "final_justification": "The author has addressed my concerns and I will keep the original score for this paper.", "rebuttal": "We thank the reviewer for the positive comments and constructive suggestions. Below, we provide detailed responses to all the questions raised:\n\n> **R1. Reply to ''Clarity of the Methodology'':**\n\nWe apologize for the confusion and will revise the methodology section in a future version of our manuscript to improve readability. Following reviewers' constructive suggestions, we have provided detailed and intuitive explanations of the EBM loss and outlined the core ideas behind each theorem. A summary of the overall logic flow—illustrating the relationships among $\\Delta\\mathrm{Energy}$, the EBM loss, and domain-consistent Hessians—is presented in R2, with supporting empirical evidence shown in Table 2-1.\nBelow, we briefly summarize the intuition behind each theorem:\n\nTheorem 3.2 provides a formal guarantee that $\\Delta\\mathrm{Energy}$ can provably surpass MCM by amplifying the separation between ID and open-set OOD data.\n\nTheorem 3.3 provides a formal guarantee that $\\Delta\\mathrm{Energy}$ can provably reduce the false positive rate (FPR) in open-set OOD detection compared to MCM.\n\nTheorem 3.4 implies that minimizing the proposed EBM loss can increase the lower bound of $\\Delta\\mathrm{Energy}$ for closed-set classes, which effectively enhances the separation between ID and open-set OOD data.\n\nTheorem 3.5 demonstrates that the EBM loss in Equation 4 can lead to domain-consistent Hessians of classification loss between original and masked samples—an indicator of improved OOD generalization.\n\nProposition 3.6 shows that minimizing EBM provides a bound on the performance gap between ID data and closed-set OOD data, indicating that optimizing $\\Delta\\mathrm{Energy}$ for ID data implicitly promotes OOD generalization.\n\nAll these intuitive explanations will be incorporated into the revised manuscript to enhance clarity and accessibility.\n\n\n> **R2. Reply to ''Intuitive explanation of the EBM loss to guide the model to learn domain-consistent Hessians'':**\n\nAs illustrated in the attention maps in Figure 1(B) and Figure 3 of our manuscript, retaining the top-$p$ proportion of elements in $P_j$ and masking the rest corresponds to masking environment-related information, after which we compute the resulting energy score, denoted as $E_2$. The EBM loss then measures the change between $E_2$ and the original energy $E_0$. The key insight is that masking environment-related information typically induces a small energy change, whereas masking domain-invariant information results in a large energy shift. By optimizing the EBM loss to minimize the energy change before and after masking, the model is in turn encouraged to focus on domain-invariant features, thereby promoting stable classification across domains. Thus, it encourages the Hessians of the classification loss to become domain-invariant as well.\n\nTo provide an intuitive understanding of EBM’s impact on learning domain-consistent Hessians and improving OOD generalization, we present the relationship between the EBM loss, Hessian distances between the original and masked domains, and OOD accuracy during the training process. We denote the Hessian of the classification loss with respect to the learnable model parameters for the original data as $H_\\mathcal{s}$, and that for the masked data as $H_{\\mathcal{s}^\\prime}$. Following the prior work [1],  we compute the Hessian distances between the original and masked domains as $\\Vert H_\\mathcal{s} − H_{\\mathcal{s}^\\prime}\\Vert_F$, where $\\Vert \\cdot \\Vert_F$ denotes the Frobenius norm. Table 2-1 reports these quantities throughout training on the ImageNet-based dataset. We observe that EBM consistently yields smaller Hessian distances and higher OOD accuracies, supporting our claim that EBM promotes domain-invariant optimization dynamics.\n\n[1] Hemati, Sobhan, et al. \"Understanding Hessian alignment for domain generalization.\" ICCV, 2023.\n\n\nTable 2-1. Correlation between Hessian distances ($\\Vert H_\\mathcal{s} − H_{\\mathcal{s}^\\prime}\\Vert_F$) and OOD accuracies / EBM losses under the EBM regularization during the training on the ImageNet-based dataset used in our main experiment. The OOD accuracies are evaluated on the closed-set subset of the ImageNet-R test data.\n\n| Metric            | Epoch       | 1         | 3        | 5        | 7        | 9        | 11       | 13       | 15       | 17       | 19       |\n| ----------------- | ----------- | --------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- |\n| loss_EBM          | w / o EBM   | 0.91      | 0.92     | 0.93     | 0.93     | 0.93     | 0.93     | 0.93     | 0.93     | 0.93     | 0.93     |\n| loss_EBM          | **w / EBM** | **0.91**  | **0.91** | **0.89** | **0.88** | **0.87** | **0.87** | **0.85** | **0.84** | **0.83** | **0.83** |\n| Hessian Distances | w / o EBM   | 25.33     | 4.05     | 5.16     | 5.30     | 5.35     | 5.58     | 5.42     | 5.79     | 5.86     | 6.60     |\n| Hessian Distances | **w / EBM** | **19.98** | **3.24** | **2.97** | **3.17** | **3.56** | **3.47** | **3.81** | **4.23** | **4.26** | **4.35** |\n| OOD Acc           | w / o EBM   | 67.5      | 68.8     | **72.5** | 69.4     | 68.8     | 67.2     | 67.8     | 68.4     | 67.2     | 68.8     |\n| OOD Acc           | **w / EBM** | **70.0**  | **71.6** | 69.1     | **70.9** | **70.6** | **68.8** | **70.9** | **69.4** | **71.9** | **72.8** |\n\nIn addition, as shown in Theorem 3.5, we theoretically demonstrate that optimizing the EBM loss leads to domain-consistent Hessians. We briefly introduce the logic behind the theoretical relationships as follows:\nAs shown in Equation 31, we establish a connection between the Hessians of the classification loss and the gradient of the energy score with respect to the model parameters. \nBy comparing the gradients of the energy scores across the original and masked domains, we derive in Equation 33 that smaller differences in these gradients imply smaller differences in the Hessians of the classification loss across domains.\nTherefore, minimizing the EBM loss not only aligns energy score gradients but also encourages Hessian consistency, thereby improving OOD generalization.\nTo make the logic flow clearer, we summarize the logic as follows:\n\n---\n\n**A. Eq. (31): Classification loss Hessians ←→ Energy score gradients**  \n↓  \n**B. Domain-consistent Hessians ←→ Domain-consistent energy score gradients**  \n↓  \n**C. Optimizing EBM → Small distance between energy score gradients across original and masked domains**  \n↓  \n**D. Optimizing EBM → Small distance between Hessians across original and masked domains (i.e. domain consistent Hessians)**\n\n---\n\n\n> **R3. Reply to ''Elaborate on the selection strategy for hyperparameters'':**\n\nIn our main experiments on open-set discrimination using the large-scale ImageNet-1k dataset, we set $c = 2$ and search for the hyperparameters $p$ and $\\lambda_0$ within the ranges $p\\\\% \\in \\\\{0.4, 0.5, 0.6\\\\}$ and $\\lambda_0 \\in \\\\{0.09, 0.10, 0.11\\\\}$. **Model selection follows the training-domain validation protocol, where models are trained using few-shot samples under different combinations of $p$ and $\\lambda_0$, and the model that achieves the highest accuracy on 4-shot subsets—randomly sampled from the training data—is selected.** In our manuscript, we have presented ablation study results for the hyperparameters $c$ and $p$ in Tables 9 and 10 of our manuscript, respectively. Here, we include the ablation results for $\\lambda_0$, provided in Table 2-2. The results show that the EBM method achieves the best overall performance in terms of both AUROC and OOD accuracy when $\\lambda_0 = 0.1$. **Notably, models trained with EBM loss regularization (i.e., $\\lambda_0 \\neq 0$) achieve improved performance in both AUROC and OOD accuracy compared to the baseline model without EBM loss.** This demonstrates the effectiveness of our EBM method in jointly optimizing for both OOD generalization and OOD detection.\n\nThrough empirical observations from the ablation study, we found that the masking proportion $p\\\\%$ exhibits a robust performance sweet spot within the range of 0.5 to 0.6, while the coefficient $\\lambda_0$ performs best around 0.1. A masking proportion near 0.6 consistently balances the trade-off between retaining sufficient input information for accurate recognition and introducing meaningful corruption to enhance robustness. Similarly, setting $\\lambda_0$ around 0.1 achieves a good balance between maintaining classification performance and enforcing energy consistency between the original and masked data. While these empirical observations provide a principled starting point for hyperparameter selection for other downstream tasks, we acknowledge that the optimal choices may vary slightly across different domains. Our ablation studies offer practical guidance for setting the search space of $p$ and $\\lambda_0$ on other datasets.\n\nTable 2-2. Ablations on the hyperparameter $\\lambda_0$. In the table, we set $c=2$ and $p\\\\%=0.6$ according to the optimal hyperparameter observed in Table 9 and Table 10. The EBM method achieves the overall best performance on both AUROC and OOD accuracy when $\\lambda_0=0.1$.\n\n| $\\lambda_0$ | ID ACC    | OOD ACC   | AUROC     | FPR95     |\n| ----------- | --------- | --------- | --------- | --------- |\n| 0.00        | **82.11** | 61.36     | 75.64     | 73.20     |\n| 0.05        | 81.78     | 61.56     | 77.94     | 71.58     |\n| 0.10        | 81.62     | **63.28** | **80.19** | **65.75** |\n| 0.20        | 81.54     | 62.85     | 77.96     | 69.95     |\n| 0.50        | 81.74     | 61.80     | 78.41     | 69.75     |\n| 1.00        | 81.10     | 61.67     | 79.39     | 68.20     |\n| 2.00        | 81.32     | 62.01     | 79.26     | 71.45     |"}, {"review_id": "FKLd4qWzDC", "reviewer": "Reviewer_C3Gj", "summary": "The paper introduces a novel OOD score, ΔEnergy, which enhances vision-language models' (VLMs) robustness in detecting out-of-distribution (OOD) data. ΔEnergy outperforms existing methods by reducing maximum cosine similarity and improving generalization. It also provides a domain-consistent Hessian for better OOD detection. Experiments show ΔEnergy improves performance by 10%-25% in AUROC. This work looks interesting, with a unique research perspective, good performance, and theoretical support.", "questions": "1. Why design deltaE and EBM, and what is the underlying logic behind them.\n\n2. How to determine the parameters for OOD detection in deltaE and whether the parameters are robust. \n\n3. On some datasets, the improvement is acceptable, but on some datasets, the improvement is very small. Please explain the reason.\n\n4. Why EBM？ Is the EBM application scenario established? How to satisfy the conditions of the theorem.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths: \n\n1)  An interesting research starting point. \n\n2) The effectiveness of the method has been supported both theoretically and experimentally.\n\nWeaknesses: \n\n1) This work heavily relies on the MCM method, and the performance improvement is not significant. For example, on PACS and\nVLCS, AUC, 93.7% VS 93.5%. \n\n2) The writing is too cumbersome and complex. It is recommended to briefly describe the idea of the theorem in the main text section and provide a more formal description in the appendix. In addition, the energy design in the paper lacks in-depth analysis, including EQ 4. It is not very clear why these designs can help OOD tasks.\n\n3) There is a problem with the application scenario of EBM, which requires obtaining several samples from the source domain. Is there any practical application background? How to satisfy the conditions of its theorem and how to detect its satisfaction.\n\n4) Lack of parameter analysis experiments, especially on how to determine the threshold of delta E in the experiment.", "was_revised": true, "final_justification": "My questions have been partially resolved. However, based on the current status of the submitted paper, I have decided to maintain my current score for the following reasons: 1) The correlation between EBM and the improvement of final classification performance is unclear; 2) Insufficient discussion of parameters, especially in the OOD detection case and the few shot case where N values are taken, it is recommended to use a figure instead of a table; 3) How to verify that the conditions are met without an accurate answer; 4) Some typos, such as Eq (6) at line 147; 5) This article has some contributions but is not well distinguished from MCM in the main text, as the entire energy function design was inspired by MCM; 6) Additionally, it is suggested to theoretically demonstrate that inconsistent changes in \\ delta E are indeed valid for OOD.", "rebuttal": ">**R1. Reply to ''This work relies on the MCM method, and the improvement is not significant on some data'':**\n\nThe proposed $\\Delta\\mathrm{Energy}$ computes the change in energy score resulting from re-aligning the vision-language modalities by setting the top-2 maximum cosine similarities to zero. We would like to clarify the difference between our $\\Delta\\mathrm{Energy}$ and the MCM method, and we also provide an explanation for why the performance improvements are less significant on PACS and VLCS. This discussion will be included in a later version of the manuscript to improve clarity and completeness.\n\n**We highlight the key differences between our method and MCM as follows:**\n\n1. Theoretical support for the OOD detection capability of $\\Delta\\mathrm{Energy}$ compared to MCM: We have theoretically demonstrated in Theorem 3.2 that the $\\Delta\\mathrm{Energy}$ score for ID data is consistently larger than that for OOD data, indicating its effectiveness and discriminative power for OOD detection. Compared to MCM and raw energy scores, $\\Delta\\mathrm{Energy}$ amplifies the gap between ID and OOD data, a property validated by both theoretical analysis (Theorem 3.2) and empirical results (Tables 1, 2, 5 and 7).\n2. A $\\Delta\\mathrm{Energy}$-based optimization algorithm (EBM) for both OOD detection and generalization: Further performance gains can be achieved through our $\\Delta\\mathrm{Energy}$ lower-bound maximization algorithm (EBM). We have both theoretically and empirically shown that EBM not only enhances OOD detection but also secretly optimizes for improved OOD generalization. Theoretical support is provided in Theorem 3.5 and Proposition 3.6. This ''*feed two birds with one scone*” approach is novel compared to MCM, which focuses solely on detection.\n\n**Regarding the observation that improvements are less significant on PACS and VLCS**, we attribute the more modest gains to two key factors: (1) the inherent discrepancy between ID and easily separable open-set OOD samples is already substantial in these benchmarks; and (2) for hard-to-detect open-set OOD samples, the maximum cosine similarities do not satisfy the condition: $s_\\hat{y_1}(x_{ID}) > s_\\hat{y_1}(x_{OOD})$.\nAs a result, the benefit of amplifying this separation using $\\Delta\\mathrm{Energy}$ is reduced. This outcome is consistent with our theoretical analysis in Theorem B.1 (Lines 662–668), which suggests that $\\Delta\\mathrm{Energy}$ is more effective in hard OOD detection scenarios—specifically, when the similarity gap between closed-set and open-set samples is small. In contrast, when the distributional shift between closed-set and open-set data is already pronounced, the benefit of enhancing ID–OOD separability through $\\Delta\\mathrm{Energy}$ diminishes. However, we emphasize that hard OOD scenarios are more prevalent in open-world settings, which we evaluate using ImageNet-based hard OOD detection benchmarks. The results, presented in Tables 1, 3, 6, and 7, show substantial performance gains, demonstrating the effectiveness of our method.\n\n> **R2. Reply to ''Briefly describe the idea of the theorem, and it is not clear why EBM can help OOD tasks'':**\n\nFollowing your suggestion, we will improve the writing of methodology in the revised paper to improve its readability. Below, we briefly describe the main ideas behind each theorem and explain why Equation 4 is beneficial for OOD tasks.\n\n**Brief description of the idea behind each theorem:**\n\nTheorem 3.2 provides a formal guarantee that $\\Delta\\mathrm{Energy}$ can provably surpass the widely-used OOD detection method MCM by amplifying the separation between ID and open-set OOD data.\n\nTheorem 3.3 provides a formal guarantee that $\\Delta\\mathrm{Energy}$  can provably reduce the false positive rate (FPR) in open-set OOD detection compared to MCM.\n\nTheorem 3.4 implies that minimizing the EBM loss can increase the lower bound of $\\Delta\\mathrm{Energy}$ for closed-set classes, which effectively enhances the separation between ID and open-set OOD data.\n\nTheorem 3.5 demonstrates that minimizing the EBM loss can lead to domain-consistent Hessians of classification loss between original and masked samples, which serves as a strong indicator for OOD generalization.\n\nProposition 3.6 shows that minimizing EBM provides a bound on the performance gap between ID data and closed-set OOD data, indicating that optimizing $\\Delta\\mathrm{Energy}$ for ID data implicitly promotes OOD generalization.\n\n**Why Equation 4 can help OOD tasks:**\n\nEquation 4 introduces a $\\Delta\\mathrm{Energy}$-based bound maximization function (EBM) during the fine-tuning process, which aims at increasing the lower bound of $\\Delta\\mathrm{Energy}$ score for closed-set classes, as formally established in Theorem 3.4. We explain why Equation 4 is beneficial for OOD tasks from both theoretical and empirical evidence.\n\nTheoretical Perspective: (a) For OOD detection: minimizing Equation 4 effectively increases the $\\Delta\\mathrm{Energy}$ score for ID data, thereby enhancing the separation between ID and open-set OOD samples. This improved separability facilitates more accurate OOD detection. (b) For OOD generalization: as shown in Theorem 3.5, minimizing Equation 4 leads to domain-consistent Hessians of the classification loss between original and masked samples, which is a strong indicator of OOD generalization. \n**We have also provided detailed and intuitive explanations of the relationship between the EBM and domain-consistent Hessians, as well as the mathematical logic flow, in R2 to Reviewer 3pyx.**\n\nEmpirical Evidence: We also empirically demonstrate the impact of the EBM loss on both OOD tasks through an ablation study on the coefficient $\\lambda_0$. The results, shown in Table 2-2, indicate that models trained with EBM loss regularization (i.e., $\\lambda_0 \\neq 0$) achieve consistently better performance in both AUROC and OOD accuracy compared to the baseline without EBM. This confirms the effectiveness of EBM in jointly optimizing for OOD generalization and OOD detection.\n\n\n> **R3. Reply to ''the application scenario of EBM'':**\n\nOur EBM framework follows a common practice in few-shot OOD generalization settings, where only limited source domain samples are available. Few-shot OOD generalization has been extensively studied in prior works. However, these existing methods primarily focus on generalization to closed-set OOD data. In contrast, real-world deployment environments typically involve a mixture of distribution shifts, including both covariate shifts (e.g., changes in style or domain) and semantic shifts (e.g., unseen classes during fine-tuning). To better reflect this reality, our work considers a more practical and challenging setting: the test distribution is composed of both closed-set OOD data (covariate shifts) and open-set OOD data (semantic shifts). This setting has attracted increasing interest in recent years [1-5]. Importantly, this problem is especially critical for high-risk applications (e.g., medical imaging or autonomous driving), where models must simultaneously generalize across domains and detect unseen semantic categories.\n\n[1] Bai, Haoyue, et al. \"Feed two birds with one scone: Exploiting wild data for both out-of-distribution generalization and detection.\" ICML, 2023.\n\n[2] Bai, Haoyue, et al. \"Aha: Human-assisted out-of-distribution generalization and detection.\" NeurIPS, 2024.\n\n[3] Gwon, Kyungpil, et al. \"Out-of-distribution (OOD) detection and generalization improved by augmenting adversarial mixup samples.\" Electronics, 2023.\n\n[4] Zhu, Lin, et al. \"CRoFT: robust fine-tuning with concurrent optimization for OOD generalization and open-set OOD detection.\" ICML, 2024.\n\n[5] Lu, Wang, et al. \"Diversify: A general framework for time series out-of-distribution detection and generalization.\" TPAMI, 2024\n\n> **R4. Reply to ''How to detect the satisfaction of theorem conditions'':**\n\nThe key assumption underlying the EBM method is formalized in Equation 7. Since the cosine similarity after re-alignment is defined as $\\tilde{s}_{\\hat{y}_j}(\\mathbf{x_i})= 0$, Equation 7 simplifies to:\n\n$\\sum_{i=1}^N e^{s_{\\hat{y}_1}(\\mathbf{x}_i)/\\tau} \\geq $\n\n$\\sum_{i=1}^N [(e^{\\varepsilon_E} - 1) e^{-E_2(\\mathbf{x}_i)}+1]$\n\nwhere $s_{\\hat{y}_1}(\\mathbf{x_i})$ is the maximum cosine similarity for input $\\mathbf{x_i}$, $ \\varepsilon_E$ is the upper bound of EBM loss, and $E_2(\\mathbf{x_i})$ is the energy score for masked feature.\n\nThis inequality implies that, after fine-tuning, the maximum cosine similarity scores for ID samples (i.e., $s_{\\hat{y_1}}$) must be sufficiently large. At the same time, it requires the bound $\\varepsilon_E$ (in the constraint $\\mathcal{L}_{\\Delta E} \\leq \\varepsilon_E$) not to be too loose. These conditions are both reasonable: the first encourages confident classification, while the second ensures effective optimization of the EBM loss. Since all terms in Equation 7 are computable during fine-tuning, the theoretical condition is verifiable in practice.\nOverall, both the practical application scenarios and the verifiable theoretical conditions highlight the broad applicability of our method.\n\n> **R5. Reply to ''Lack of parameter analysis experiments'':**\n\nIn our manuscript, we have presented ablation study results for the hyperparameters $c$ and $p$ in Tables 9 and 10, respectively. Additionally, the ablation results for $\\lambda_0$ are provided in Table 2-2 of R3 to Reviewer 3pyx. Due to space constraints, we kindly refer the reviewer to our response (R3) to Reviewer 3pyx for details regarding the hyperparameter selection strategy.\n\n> **R6. Reply to Questions:**\n\nA1. The underlying rationale behind $\\Delta\\mathrm{Energy}$ and EBM is elaborated in R2.\n\nA2. The hyperparameter selection strategy is detailed in R5.\n\nA3. We discussed the reasons why the improvement is small on some datasets in R1.\n\nA4. The practical application scenarios and how to verify theoretical conditions are explained in R3 and R4."}, {"review_id": "TN4nyodDYn", "reviewer": "Reviewer_zrkV", "summary": "This paper introduces **∆Energy**, an OOD detection score for vision-language models based on the energy change after masking top vision-language similarities. The approach combines ∆Energy scoring with an energy bound maximization (EBM) framework during prompt tuning. This jointly improves OOD detection and generalization under covariate shifts. Theoretical analysis and extensive experiments show improved performance over previous methods. The method is efficient and easy to integrate into existing vision-language models.", "questions": "In Tables 1 and 2, why do the Energy and MaxLogit performances differ so drastically? In my understanding, their results should be quite similar.", "limitations": "See the above.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n\n1. The paper is written clearly, and its method is described in a straightforward manner.\n2. It provides detailed theoretical derivations supporting the proposed approach.\n3. Experiments are thorough and conducted on multiple benchmarks.\n\n**Weaknesses**\n\n1. The motivation is not sufficiently clear: the method is not proposed to address any specific issue with existing methods, nor to fill a well-defined gap.\n2. Originality is limited: pruning or zeroing the maximum score has been used in traditional OOD detection (e.g., DICE, ASH) to amplify ID–OOD differences, yet these are neither cited nor compared.\n3. I am concerned about the method’s performance: most comparisons are against very basic methods, and the reported gains in Tables 1 and 2 are only marginal(e.g., MSP, MCM). More recent and stronger baselines should be included.\n4. I doubt the claim that ΔEnergy outperforms MCM: energy-based scoring already often outperforms MSP (with existing theoretical support), and applying an energy-style normalization to MCM could yield similar improvements. Therefore, it is unclear that the gains stem specifically from the authors’ ΔEnergy formulation.\n5. The authors’ method is merely an incremental combination of an ASH-style pruning technique with the Energy approach on top of MCM.", "was_revised": true, "final_justification": "Most of my concerns have been addressed.", "rebuttal": ">**R1. Reply to ''The motivation is not sufficiently clear'':**\n\nWe apologize for any confusion regarding the motivation and the issue we aim to address. We provide clarification as follows:\n\n**Problem Setting of Our Target Issue:**\nIn real-world deployment scenarios, models often encounter a mixture of distribution shifts, including both covariate shifts (e.g., changes in style or environment) and semantic shifts (e.g., unseen classes during fine-tuning). To reflect this reality, our work considers a more practical and challenging setting where the test distribution consists of both closed-set OOD data (covariate shifts) and open-set OOD data (semantic shifts).\n\nUnder this setting, the model is required to:\n(1) Maintain accurate classification of covariate-shifted samples—a capability referred to as OOD generalization;\n(2) Recognize semantic-shifted OOD samples and abstain from blindly classifying them into any ID class—highlighting the importance of OOD detection.\n\nThis setting has attracted increasing interest in recent years [1-4]. Importantly, this problem is especially critical for high-stakes applications (e.g., medical imaging or autonomous driving), where models must simultaneously generalize across domains and detect unseen semantic categories.\n\n**Clarification of the Motivation and Effectiveness of Our Method:**\nWe have explained the benefits of $\\Delta\\mathrm{Energy}$ and EBM for OOD detection and OOD generalization from both theoretical and empirical perspectives. The theoretical motivations are thoroughly discussed in R1 and R2 to Reviewer 3pyx, particularly through the logical flow behind the theorems. Due to space limitations, we kindly refer the reviewer to R2 to Reviewer 3pyx for the in-depth explanation of our theoretical motivations. Empirical evidence supporting the effectiveness of EBM is presented in Tables 3 and 4 of the manuscript, and in Tables 2-1 and 2-2 in response to Reviewer 3pyx, where consistent improvements are observed across both OOD tasks.\n\n[1] Feed two birds with one scone: Exploiting wild data for both out-of-distribution generalization and detection ICML, 2023.\n\n[2] Aha: Human-assisted out-of-distribution generalization and detection. NeurIPS, 2024.\n\n[3] Diversify: A general framework for time series out-of-distribution detection and generalization. TPAMI, 2024\n\n[4] Bridging ood detection and generalization: A graph-theoretic view. NeurIPS, 2024\n\n> **R2. Reply to ''Pruning-based methods are not discussed':**\n\nWe apologize for any confusion caused by the lack of discussion on pruning-based OOD detection methods. We will include a corresponding discussion and cite all mentioned papers. Below, we highlight the key differences between our method and related works such as DICE and ASH: \n\n- Different problem setting: Our method connects $\\Delta\\mathrm{Energy}$ with domain-consistent Hessians and is specifically designed to jointly optimize OOD generalization and OOD detection, backed by theoretical guarantees. In contrast, DICE and ASH primarily target OOD detection by removing noisy weights or activations, without addressing the generalization aspect. Moreover, our method performs V-L re-alignment and explicitly measures energy changes, while prior methods reshape the model outputs through pruning without considering such energy-change criteria.\n\n- Different model assumptions: Our method is tailored for CLIP-based scenarios, where the output logits tend to be uniform-like, posing greater challenges for OOD detection. We particularly focus on hard OOD detection settings, where the semantic gap between ID and OOD samples is small. In contrast, prior works such as DICE and ASH primarily focused on traditional visual models, where the logit corresponding to the ground-truth label is often much higher than the others.\n\n> **R3. Reply to ''Concerns about the method’s performance':**\n\n**Reply to the concern on comparisons:**\n\nWe have compared our method with recent CLIP-based state-of-the-art OOD detection methods, including NegLabel (ICLR 2024) and CSP (NeurIPS 2024).  As shown in Tables 1,2, and 7 of our manuscript, our method significantly outperforms both NegLabel and CSP, demonstrating its superiority in distinguishing different semantics in hard OOD detection scenarios. The underlying reasons for the inferior performance of NegLabel and CSP have been discussed in Lines 239-246. In contrast, aligning with the results in Theorem B.1, our method is effective in hard OOD detection scenarios, where the number of classes is high and the similarity gap between closed-set and open-set samples is small.\n\n**Reply to the concern that gains are less significant on some data:**\n\nWe attribute the modest gains on PACS and VLCS to two key factors: (1) the inherent discrepancy between ID and easily separable open-set OOD samples is already substantial in these benchmarks; and (2) for hard-to-detect open-set OOD samples, the maximum cosine similarities may not satisfy the condition: \n$s_\\hat{y_1}(x_{ID}) > s_\\hat{y_1}(x_{OOD})$.\nThus, the benefit of amplifying ID-OOD separation using $\\Delta\\mathrm{Energy}$ is diminished. This outcome aligns with our theoretical findings in Theorem B.1, which suggest that $\\Delta\\mathrm{Energy}$ is more effective in challenging OOD detection scenarios—specifically, when the similarity gap between closed-set and open-set samples is small. Such cases are more prevalent in open-world settings, which we evaluate using ImageNet-based benchmarks. The results, presented in Tables 1, 3, 6, and 7, show substantial performance gains, demonstrating the effectiveness of $\\Delta\\mathrm{Energy}$ in handling hard OOD scenarios.\n\n> **R4. Reply to ''Doubt on the claim that $\\Delta\\mathrm{Energy}$ outperforms MCM':**\n\nAs shown in the experimental results in Tables 1, 2, and 6 in the manuscript, our method consistently outperforms the energy-based scoring baseline. **The superior performance of $\\Delta\\mathrm{Energy}$ over the raw energy-based method indicates that the $\\Delta\\mathrm{Energy}$ formulation itself contributes meaningfully to enhanced OOD detection performance.** \n\nMoreover, in Theorem 3.2, we provide a theoretical justification showing that $\\Delta\\mathrm{Energy}$ leads to stronger separation between ID and open-set OOD samples than MCM. We emphasize that this theoretical analysis is conducted under CLIP-like model conditions—where the output logits are typically uniform-like (as noted in MCM). This differs significantly from visual models trained with cross-entropy loss, where the logit corresponding to the ground-truth label is often much higher than the others. **As noted in the original MCM paper, for CLIP-like models, applying softmax helps sharpen the uniform-like logits output and increases the separation between ID and OOD samples. By tuning the temperature in the softmax, MCM significantly outperforms the standard energy-based scoring.**\n\n**Taken together, both our theoretical and empirical results support the conclusion that, in the context of CLIP-like models, the $\\Delta\\mathrm{Energy}$ formulation itself plays a critical role in enhancing OOD detection.**\n\n> **R5. Reply to ''The method is merely an incremental combination of an ASH-style pruning technique with the Energy on top of MCM:**\n\nOur work offers both theoretical advancements and a broader application scope across two OOD tasks within the vision-language model (VLM) context—namely, OOD generalization and OOD detection—rather than simply combining ASH-style pruning with energy-based methods. The key distinctions are as follows:\n\n- Different model assumptions and data settings: We focus on CLIP-based models, where the output logits tend to be uniform-like, making OOD detection more challenging. Moreover, we target hard OOD detection scenarios, where the semantic gap between closed-set and open-set OOD samples is small and the model must robustly handle both covariate and semantic distribution shifts.\n\n- Different methodology: Our method performs vision-language re-alignment and introduces a novel post-hoc OOD score that explicitly measures energy changes. In contrast, ASH modifies model outputs by pruning activations and relies on existing post-hoc OOD scores, such as the energy score or MSP, without incorporating energy-change criteria.\n\n- Unified framework for both OOD generalization and OOD detection: We theoretically prove that the $\\Delta\\mathrm{Energy}$-based lower-bound maximization (EBM) can theoretically improve both OOD tasks, explicitly increasing $\\Delta\\mathrm{Energy}$ for closed-set data and learning domain-invariant information between original data and masked data. In contrast, MCM and Energy Score focus solely on OOD detection without addressing OOD generalization, and ASH is an empirical, pruning-based method tailored to visual models trained with cross-entropy loss, lacking theoretical support for application to VLMs.\n\n> **R6. Reply to ''Why do the Energy and MaxLogit performance differ:**\n\nIn CLIP-like models, the output logits tend to be uniform-like, meaning that the similarity scores across different classes are relatively close in magnitude. In such cases:\n\n- The Energy score, which is based on the log-sum-exp of all logits, becomes more sensitive to the overall distribution of logits output and less discriminative when the logits are not sharply separated. This can reduce its ability to distinguish between ID and OOD samples.\n- In contrast, MaxLogit focuses solely on the maximum class score, which provides a sharper and more focused signal for confidence—even if all logits are close. This makes it more robust in detecting anomalies when the distribution of scores across classes is flat, as it picks out the highest response directly without aggregating all logits.\n\nAdditional empirical evidence of MaxLogit surpassing Energy in CLIP-based cases can be found in Table 1–2 of the CLIPN paper [5]. \n\n[5] Clipn for zero-shot ood detection: Teaching clip to say no. ICCV, 2023."}]

Author rebuttal:

Meta-review: This paper introduces ∆Energy out-of-distribution (OOD) score for VLMs. ∆Energy is motivated by heuristic observations that the energy change is larger for in-distribution (ID) data than for OOD data. This paper also proposes a fine-tuning framework, named Energy-based Bound Maximization (EBM). Experiments are shown in OOD detection and generalization benchmarks.

This paper has four borderline recommendations, but slightly learning towards positive. The reviewers found that the proposed method is well-motivated and sound (3pyx, 7zhN, C3Gj), and theoretically supported (7zhN, C3Gj).

Meanwhile, there were concerns related to the clarity and the presentation of the paper (3pyx, C3Gj), scope of evaluation (7zhN, C3Gj), and hyperparameter sensitivity (3pyx, C3Gj).

During the discussion period, the rebuttal document provides new experiments on concept shift to handle the evaluation scope issue, and a hyperparameter ablation study.

In summary, this paper has meaningful novelty, theoretical grounding, and empirical results. Also, the most critical concerns were addressed during rebuttal. Overall, I lean towards accept. While clarity and parameter analysis could be further improved, the contributions are meaningful for the vision-language and OOD communities.
Decision: Accept (poster)