Self-Calibrated Tuning of Vision-Language Models for Out-of-Distribution Detection
Authors: Geng Yu
Jianing Zhu
Jiangchao Yao
Bo Han
Venue: NeurIPS 2024 poster (2024)
Keywords: out-of-distribution detection
vision-language model
prompt-tuning
Primary area: safety_in_machine_learning
TL;DR: We propose a framework, named SCT, to mitigate the problem of spurious OOD features mined from ID data in prompt-tuning based OOD detection methods.

Abstract: Out-of-distribution (OOD) detection is crucial for deploying reliable machine learning models in open-world applications. Recent advances in CLIP-based OOD detection have shown promising results via regularizing prompt tuning with OOD features extracted from ID data. However, the irrelevant context mined from ID data can be spurious due to the inaccurate foreground-background decomposition, thus limiting the OOD detection performance. In this work, we propose a novel framework, namely, \textit{Self-Calibrated Tuning (SCT)}, to mitigate this problem for effective OOD detection with only the given few-shot ID data. Specifically, SCT introduces modulating factors respectively on the two components of the original learning objective. It adaptively directs the optimization process between the two tasks during training on data with different prediction uncertainty to calibrate the influence of OOD regularization, which is compatible with many prompt tuning based OOD detection methods. Extensive experiments and analyses have been conducted to characterize and demonstrate the effectiveness of the proposed SCT. The code is publicly available at: https://github.com/tmlr-group/SCT.

Self-Calibrate Tuning of Vision-Languaae Models for Out-of-Distributio Detection
Geng Yu²
Jianing Zhu
Jiangehao 1.0 Bo Han
CMIC, Shanghai Jiao Tong University
TMLR Group, Hong Kong Baptist University
Shanghai A Laboratory "RIKEN Center for Advanced Intelligence Project
{warriors30, Sunarker @sj.u.edd.. edu. cn {csjuzhu, bhanml)6comp hkbu. Mkbbu.ddu.de hk
Abstract
ditoof-distribution (OOD) detection is crucial for deploying reliable machine learning models in open- open-world applications. Recent advances in CLIP-based OOD detection have shown promising results via regularizing prompt tuning with OOD features extracted from ID data. However, the irrelevant context mined from ID data be spurious due the inaccurate greerround-backrrodd thus limiting the OOD detection performance. In this work, we propose a novel framework, namely, Sallbbaaaddteedd Tuning (SCT), mitigate this problem for effective OOD detection with only the given few-shot ID data. Specifically, SCT introduces modulating factors respectively learning objective. It adaptively directs the optimization process between the two tasks during training on data with different prediction uncertainty to calibrate the influence of OOD regularization, which is compatible with many prompt tuning based OOD detection methods. Extensive experiments and analyses have been conducted to characterize and demonstrate the effectiveness of the proposed SCT. The code is publicly available at: https: h htt///gitttt/ com/ womm/mlr-eeeeeeeee SaeeggrruupSSC
decomposition,
the two components of the original
Introduction
The deep neural networks (DNNs) are demonstrated to be overconfident on the OOD data out of the pre-defined label space [Hendrycks and Gimpel, 2017], which can induce severe problems in those safety-critical applications like autonomous driving or medical intelligence. Various explo- rations [Liang et al., 2018, Djurisic et al., 2022, Du et al., 2022, Zhu et al., 2023b] thus have been conducted in designing scoring functions or fine-tuning methods with auxiliary outliers to improve the OOD distinguishability. Specially, with the emergence of the powerful pretrained langunllagguage models (VLMs) [Radford et al., 2021], a series of prompt tuning based methods [Miyai et al., 2024b, Tao et al., 2023, Bai et al.. 2023, Ming et al., 2022c] show impressive performance in current OOD detection benchmarks, with the regularization given only few-shot in-distribution (ID) data.
Generally, these regularizations [Wang et al., 2023, Miyai et al., 2024b] are built upon the ID-irrelevant local context as the surrogate OOD source, which is extracted by VLMs (refer to Figure I) based on its alignment with ID-class text features. Although this saves the costly collection of auxiliary outliers from the open world, the quality of the ID-irrelevant local context also becomes the bottleneck, which can be greatly affected by the foreground-backgronme revealed in previous studies [Oh et al., 2023, Tu et al., 2024, Wang et al., 2024], the prevalent VLMs struggle with poor calibration, which means that the decomposition performance downstream data might not be well guarantced. Thus, it naturally motivates the following question:
decomposition with VLMs. Specifically, as
Correspondencce Jiangehao Yao (Sumarker@sjtu.cdu.co
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Keeshond
Green Mamba
Combination Lock
Fiddler Crab
Vase
Amecican Coet
Figure 1: Imperfect foreground background decomposition. The top row shows the original images from ImageNet-Il k and the bottom row shows the ID-irrelevant context extracted from the original images (shown as the colored patches of images on the second row), using CLIP fine-tuned with fine-tuned vision-language models,
CoOp on 16-shot data. Due to the imperfect decomposition large portions of the extracted local features from ID data belong to ID-related regions, thus harming the performance of OOD detection. More illustrations are presented in the Appendix A.3.3.
Can we flexibly leverage the imperfect OOD features extracted by the VLM itself. to facilitate the few-shot prompt tuning for effective OOD detection?
As illustrated in Figure a significant portion of extracted local context from ID data not valid OOD features due to the inevitable imperfect decomposition. Consequently, OOD regularization based on such unreliable OOD features may potentially constrain the improvement of OOD detection. To investigate this problem, we conduct
ofooe-of-coneept experiment on CLIP with ImageNet as the ID dataset and prompt-tune the model with different groups of data divided by their overall prediction uncertainty. Specifically, we find that the performance of prompt tuning based methods significantly deteriorates as the uncertainty the given ID) data rises, as presented Figure 2, which motivates us to leverage such clues to overcome the current issue. Intuitively, as the model prediction ID) samples is less certain, the OOD features extracted from these data are less reliable. Performing OOD regularization performance CLIP. Therefore, a potential idea is to adaptively adjust the importance of OOD features extracted from ID data according to their prediction uncertainty during model training.
such unreliable surrogate OOD features can degrade the OOD detection
Based on the previous observation, we propose a new learning framework, i.e., Self- Calibrated Tuming (SCT), to alleviate the problem induced by spurious OOD features. At the high level, we aim to dynamically adjust the weight of OOD regularization from different training samples based on their prediction uncertainty to calibrate their influence model training. In detail, we introduce modulating factors based on the sample uncertainty estimation respectively on the two parts of the original learning objective of prompt tuning for OOD detection (refer to Eq (4)). Under this new learning framework, the model's s attention is directed towards the classification task to better generalize to the downstream ID dataset when training with low-confidence data. OOD features extracted from high-confidence
data are attached more importance to achieve more effective OOD regularization. The redirection effect of these two modulating factors facilitates VLMs learning from imperfect OOD features to ultimately improve the OOD detection of prompt tuning. Our main contributions can be summarized as follows,
* Conceptually, we investigate the problem of imperfect OOD features extracted in prompt tuning based OOD detection methods and observe that ID data with different prediction uncertainty exhibits a distinctive influence on the OOD regularization. (in Section 3.2)
Technically, we propose novel learning framework, namely Self- Delf-Calibrated Tuning (SCT), for facilitating prompt tuning for effective OOD detection given only few-shot ID samples, which conducts adaptive redirection of model optimization process between the two tasks calibrate the influence of OOD features mined from different ID data. (in Sections 3.3)
Empirically, extensive experiments from different perspectives are conducted to verify the effectiveness of SCT improving OOD detection performance. To be specific, SCT improves the false positive rate (FPR95) by 3% compared the previous best method on the

---

large-scale ImageNet-1k [Deng et al., 2009] benchmark. Furthermore, we perform various ablations and further discussions to provide thorough understanding. (in Section 4)
2
Related works
Prompt tuning for VLMs. The concept of prompt tuning was originally applied in the field of natural language processing [Radford et al., 2018]. To eliminate the need for manual prompt crafting, prompt tuning exploits supervision signals from downstream tasks to automate the process of prompt generation. Autoprompt [Shin et al., 2020] searches for tokens that cause the greatest changes in gradients based on the label likelihood. Prefix-Tuning [Li and Liang, 2021] introduces a sequence of continuous vectors that can be end-to-end optimized in the token embedding space. Incorporating prompt tuning into computer vision. CoOp [Zhou et al., 2022a] adapts pretrained vision-langue ssiion-angguae models by optimizing a set of learnable continuous prompt vectors. Various prompt tuning methods [Zhou et al., 2022b, Khattak et al., 2023, Sun et al., 2022] have been subsequently proposed to address different vision tasks. However, since these methods are not developed for OOD detection, they face challenges in identifying unknown OOD samples encountered at inference stages.
detection with VLMs. Large pretrained vision-language models have enriched the landscape of OOD detection through their remarkable generalization capability in both visual and textual domains. MCM [Ming al., 2022a] employs the concept of maximum softmax proba- bility [Hendrycks and Gimpel, 2017] into the inference process of CLIP for OOD detection, while CLIPN [Wang al., 2023] trains an additional text encoder using and set of prompts with large external dataset to improve its negative semantic understanding. Compared with zero-shot methods, prompt tuning based approaches achieve better OOD detection with access to few-shot II) training data. LoCoOp [Miyai et al., 2024b] adopts prompt tuning and extracts ID-irrelevant background from CLIP's local features as surrogate OOD data to regularize the learned prompts. [Bai et al., 2023] discovers ID-like outliers from ID samples via random cropping to learn set of negative prompts. However, these two representative prompt learning-basee methods suffer from spurious OOD features extracted from ID data due to the imperfect seegrruund-bccgroound VLMs. In addition, LSN [Nie et al., 2023] introduce negative prompts to empower VLMs to learn negative semantics from ID samples while NegPrompt [Li et al., 2024] leverages negative prompts to investigate the novel setting of vocabuloyyy OOD detection.
decomposition of
3 Method
In this section, we introduce our new framework, i.e., Self-Calibrated Tuning (SCT), which conducts adaptive redirection of model learning far away from OOD data region during prompt tuning with only the few-shot ID data. Firstly, we provide preliminaries and notations about prompt tuning based OOD detection (Section 3.1). Secondly, we present and discuss the critical motivation that inspires our method (Section 3.2). Thirdly, introduce its newly derived learning objective with the explanation and analysis of the underlying intuition and present its algorithmic realization (Section 3.3).
3.1 Preliminaries
VLM-based OOD detection aims to identify test samples that do not belong to any ID class designated by the downstream tasks [Miyai et al., 2024a]. Therefore, the ID distribution is defined by the ID classes from the downstream tasks, which are different from those of the upstream pretraining. Formally, we consider multi-class classification as the original training task [Nguyen al., 2015], where &' C IRd denotes the input space and y = (1....,M] denotes the label space. A reliable classifier should be able to detect the OOD input, which problem. We consider Din as the distribution of ID data over pairs of examples x E Rd and corresponding labels y E y. At test time, the environment can present distribution Dour over X of OOD data. In general, the OOD distribution Dow. is defined as an irrelevant distribution of which the label set has no intersection with y [Zhu et al., 2023a] and thus should not be predicted by the model parameterized by 0. A decision model I(-) can be made with the threshold μ.
be considered binary classification
ID
S(z f) 4 S(x; 0) < 4
T (x;; 0)
(1)
OOD
3

---

66
high uncertainty
60-
low-uncurtainty
66-
50
FPR95
45
40
36
30
25
20
Shot 16
0.02
0.04
0,67
0.89
Figure 2: Empirical demonstration about invalid OOD features extracted from ID data in LoCoOp and the influence of sample uncertainty on OOD detection performance. In the left and middle panels, we illustrate the extracted OOD features at different levels of uncertainty and find that they become unreliable as the uncertainty increases. The numbers at the bottom denote the prediction probability for the ground-truth labels from fine-tuned CLIP. In the right panel, we collect ID samples of different uncertainty levels based on prompt-tuneds CLIP and divide them into 2 groups. The result demonstrates that the OOD detection performance of LoCoOp significantly degrades as the uncertainty level of ID data rises. We leave the experimental details in Appendix A.3.1 for reference.
Vanilla prompt tuning. Given ID image æ and its corresponding label y, global visual feature f [(xr is obtained by the visual encoder f CLIP. Then, the textual prompt vectors can be formulated as fm = [Wi,
WN. Cm), where Cm denotes the word embedding of the ID class name and w = fw [ } are N learnable context vectors, each of which has the same dimension as the word embedding. The text encoder y takes prompt fm as the input and outputs the textual feature as Jm g(.... The final prediction probability of the CLIP model is computed as follows:
exp (sim gm) /T)
ply m
Σm exp (sim (f 9m') (mn)/rr) /T)
where sim (-.-) denotes cosine similarity, and 7 represents the temperature of Softmax. We use p(æ; w) to represent the probability vector [p(y = 1|x: w), ply = 2/x; w). ply = Mzx w)], denoting the prediction probability of ID image for every ID class.
Prompt tuning for OOD detection. Compared with vanilla prompt tuning methods like CoOp [Zhou et al., 2022a], advanced prompt tuning based OOD detection methods extract surrogate OOD features from ID data via various methods to perform OOD regularization. LoCoOp [Miyai et al., 2024b] can further improve the detection performance by regularizing the learnable prompts on OOD features X extracted from ID local features using a ranking- based method, and its corresponding learning objective is defined as follows,
LoCCOP ((n.))D,,, (cu(ppu) + ALOOD (p (poop(((())))) (xx:)
where is the balancing parameter, (cE(-) is the Cross-Entropy (CE) loss, x(X) is the predic- tion probability vector for X and (-) is the negative entropy of the given probability vector. IDLike [Bai et al., 2023] conducts multiple random cropping on ID data and chooses cropped regions as OOD features based on their feature similarity with textual features of ID classes
3.2 Motivation
Given the only ID data, previous studies propose to extract the ID-irrelevant local context as the surrogate OOD source, which depends on the However, since the model itself has uncertainty on the prediction results, the correctness of the decomposition cannot always be guaranteed. Thus, as illustrated in Figure 1, the extracted local regions based on the prediction of CLIP may result in spurious OOD features, which then limits the OOD detection performance. Thus, it naturally motivates the following critical research question:
decomposition using CLIP.
How could we better utilize the surrogate 0OD features extracted by imperfect brreground-backgroone decomposition of CLIP for effective OOD regularization?

---

In this work, we conduct proof-of proof-of-concep experiment to investigate the relationship between sample uncertainty and OOD detection performance of prompt tuning based methods. We use ViT-B/16 CLIP as the model and large-scale ImageNet-1k OOD benchmarks. First, we observe that the quality of OOD features extracted by CLIP is highly correlated with the uncertainty estimation of ID data. The extracted OOD features become more spurious as the uncertainty escalates, as illustrated in the left and middle panel of Figure 2. Furthermore, as empirically shown in the right panel of Figure 2, the performance of prompt tuning based methods is heavily affected by the overall uncertainty level of the given ID data. Intuitively, as the predictions of the model on ID samples are less confident, the OOD features extracted from these data are less reliable. Regularizing on such invalid OOD features can undermine the calibration ability and OOD detection performance of CLIP, which can be reflected by the higher FPR95 score (indicating higher error on OOD detection). Therefore, new mechanism is required to take the sample uncertainty into consideration to assist the model in learning from these imperfect OOD features for more effective OOD detection.
3.3 Self-calibrated tuning
As aforementioned, the LoCoOp-based OOD detection paradigm relies on the extracted ID- background local features for OOD regularization. Given the imperfect wreground-backkroune decomposition, the model is expected to effectively learn from the inaccurate OOD features for better OOD detection. Inspired by the previous observation shown in Section 3.2, one conceptual idea to mitigate this problem is to adaptively adjust the importance of OOD regularization generated from different ID samples based on uncertainty estimation to alleviate the wrong guidance of invalid OOD features. Under this learning paradigm, the model can be regularized by more valid OOD features and simultaneously prevent itself from overconfidence improve OOD detection. To this intuition, we consider reformulating the learning objective under the framework of prompt tuning as follows,
LcTt = -E(a.))>Da (CCE(P()) x;; w). v) d(plyla;
Mood(p(XX:aa <(p(ylz; w (4)
where o= RMM- R and RM + R indicate the -ewly-introduced modulating functions that calculate adaptive factors for the two components of the original loss function (i.e.. Eq (3)) of LoCoOp based on the uncertainty estimation. In this loss function, the left part is for the ID classification task, and the right part is for the OOD regularization. Specifically, e should be monotonically decreasing and y should be monotonically increasing with respect to p(y) ax; w) so that the modulating factors shift the focus of prompt learning between the two tasks during the training process.
In detail, when the model outputs low-confidence prediction for the ground-trui -ruth label, the importance of the ID classification task is highlighted in order to better generalize to the downstream task and simultaneously reduce the effect of regularization from invalid OOD features extracted from ID data. When the model can accurately and confidently classify the ID samples, its attention is redirected towards OOD regularization to strengthen the positive effect of useful ID ID-irrelevant features for better OOD detection. In the meantime, the loss contribution of the classification task is reduced to avoid the model overfitting to the downstream dataset, which benefits the calibration of the model [Mukhoti et al., 2020] and further enhances the validity of extracted OOD features.
Under this learning framework, the goal of confidence calibration and OOD detection can benefit from each other. This method calibrates the influence of OOD features mined from different ID data based on model prediction confidence during the training process to facilitate capturing more reliable OOD features from ID data. Among a wide range of functions that satisfy the simple requirements as discussed above, we choose the linear function due to its simple design. Coneretely, we formulate the loss function of SCT as follows,
LSCT = (( [CEE (i) * p(y æ w)) + MooD(p(X;:)) p(yla; (5)
This implementation of Lsct introduces no extra hyperparameters and we empirically demonstrate its effectiveness in the following experiment section 4. In the Appendix A.3.2, we consider other instantiations of the modulating function and demonstrate that these can be comparably effective.
Extraction of OOD local features. We adopt the ranking-based method as suggested by Lo- CoOp [Miyai et al., 2024b] to extract OOD local features from ID samples for OOD regularization.
5

---

Algorithm/ Self-Calibrated Tuning(SCT)
Input:
learnable prompt: w, fine-tuning epochs: T, learning rate: 17, training set of ID data,
extraction rank parameter: K, regularization term weight: X;
Output: fine-tuned prompt w:
for epoch L, ...... do
2: for mini- = I..... .M do
from the training set 4: Sample surrogate OOD features X from ID local feature map by Eq. (6)
3: Sample mini-batch { 36) } ji=
5:
V w- w-w-
+ (CE(p()) ( ((p()la;
(poo (X); p(yla:
6: end for
7:
end for
To be specific, we select the region indices of ID-irrelevant regions from set of all region indices 1= (0, 1,2, H x W 1}, where H and W denote the height and width of the feature map. We calculate the classification prediction probabilities for each region i during training by computing the similarity between the image features f()) of each region and the text features the ID classes. Then choose regions that do not include their ground truth class in the top- K predicted classes as ID-irrelevant regions J. The corresponding formulation is presented as follows:
= {i E I rank(p(() (ylz: K}.
(6)
where pO0 (y) w) denotes the prediction probability of region for the ground-truth label and rank(p() (0) (yl az; w)) denotes the rank of the ground-truth label among all ID classes. We summarize the whole procedure of the proposed SCT in Algorithm 1.
Test-time OOD detection. At the testing stage, we use the GL-MCM score proposed by [Miyai al., 2023] since it has been empirically proved outperform the conventional MCM score. It combines the maximum softmax probability scores for both global and local image features. The detailed formulation is presented as follows:
exp (sim (f (f.9mm/7
exp (sim (f(),, .mm /T) Σm exp (sim (f(, 9m.) /T)
SCL-MCM = max
max
(7)
EM=n exp (sim (f, gm)) /T)
where (() denotes the local image feature of ID samples æ for region and we set 1.
Comparison and compatibility. Compared with the previous prompt tuning based OOD detection algorithm [Miyai et al., 2024b]. the critical idea behind SCT is trying to adaptively adjust the contribution of OOD features extracted from ID data with different uncertainty during training. It provides general framework (under the guidance of the learning objective in Eq. (4)) assist VLMs in learning from spurious OOD features for effective OOD detection. The discriminative feature regularized by our SCT can be utilized by those advanced post-hoc scoring functions [Sun al., 2021, Huang et al., 2021, Sun and Li, 2022]. For prompt tuning based methods, the adaptive modulation introduced in SCT is orthogonal current tuning objectives [Liu et al., 2020] and also compatible with different augmentation [Lu et al., 2023] or mining strategies [Bai et al., 2023, Zhu et al., 2023b].
Experiment
In this section, present the comprehensive verification of the proposed SCT in the CLIP--bseed OOD detection scenario. First, we provide the experimental setups in detail (in Section 4. 4.1). Secondly, provide the performance comparison of our approach with a series of CLIP-based post-hoc methods and prompt tuning based methods (in Section 4.2). Thirdly, we conduct various ablation studies and further discussions to understand our method (in Section 4.3).
4.1 Experimental setup
Datasets. Following the common benchmarks used in previous works, we adopt the ImageNet- dataset [Deng et al., 2009] as the ID data. For OOD datasets, we adopt the same ones as in [Huang

---

and Li, 2021], including subsets of iNaturalist [Van Horn et al., 2018]. SUN [Xiao et al., 2010], Places [Zhou et al., 2017], and TEXTURE [Cimpoi et al., 2014]. For the few-shot training, we use 1, 2, 4, and 16 shots IID data for training, respectively, and evaluate models in the full test set. We also present the comparison results on conventional CIFAR benchmarks [Hendrycks et al., 2019, Liu et al., 2020], which adopt CIFAR-10 10 and CIFAR-100 as ID datasets [Krizhevsky, 2009], in Appendix A.3.2.
Evaluation metries. We employ the following two common metrics to evaluate the performance of OOD detection: (i) Area Under the Receiver Operating Characteristic curve (AUROC) [Davis and Goadrich, 2006] can be interpreted as the probability for positive sample to have a higher discriminating score than a negative sample [Fawcett, 2006]; (ii) False Positive Rate (FPR) at 95% True Positive Rate (TPR) [Liang et al., 2018] indicates the probability for negative sample to be misclassified as positive when the true positive rate is at 95%. We also adopt in-distribution testing accuracy (ID-ACC) to measure the preservation level of the performance for the original classification task on ID) data and use Expected Calibration Error (ECE) [Naeini et al., 2015] to measure the performance of SCT on confidence calibration, which are both presented in Appendix A.3.2.
Implementation details. Following previous works [Tao et al., 2023, Ming et al., 2022a, Miyai et al., 2023], we use ViT-B/16 [Dosovitskiy et al., 2021] as the backbone model for the main experiment For the hyperparameter K in the surrogate OOD features extraction, we use 200 in all experiments as recommended by [Miyai et al., 2024b]. For SCT, we adopt À 0.4 under the 1-shot setting and A = 0.2 under the 16-shot setting. We train the CLIP for 25 epochs with a learning rate of 0.002 and other hyperparameterss (e.g. batch size=32, SGD optimizer and token lengths N=16) are the same as those of CoOp [Zhou et al., 2022a]. We use two Nvidia 3090 GPUs for all experiments.
OOD detection baselines. We compare SCT with several competitive CLiP-based OOD detection methods in the two directions, including post-hoc methods and prompt tuning based methods. For post-hoc methods, we compare with MCM [Ming et al., 2022a] and GL-MCM [Miyai al., 2024b] as zero-shot baselines. We also adopt Maximum Softmax Probability (MSP) [Hendrycks and Gimpel, 2017], ODIN [Liang et al., 2018], ReAct [Sun al., 2021], MaxLogit [Hendrycks et al., 2022], and Energy score [Liu et al., 2020] as conventional scoring function baselines. In addition. we provide more discussions about post-hoc methods and our methods in Appendix A.3.2. For prompt tuning based methods, we adopt CoOp [Zhou et al., 2022a], LoCoOp [Miyai et al., 2024b]. IDLike [Bai et al., 2023], NegPrompt [L.i et al., 2024] and LSS [Nie et al., 2023] as baselines. For all prompt tuning based methods, we constrain all major experiments to few-shot learning scenarios, which is more practical in real cases. Note that LSN is a general learning framework that can be combined with various prompt tuning methods and we report the result of LSN incorporated with LoCoOp in Table We leave more definitions and implementation details in the Appendix A.
4.2
Main results
In this part, present the major performance comparison with some representative baseline methods for OOD detection to demonstrate the effectiveness of the proposed SCT. Specifically, we consider several zero-shot methods as the performance reference based on the pretrained CLIP and some prompt tuning based methods for specific comparison on fine-tuning with few-shot ID data. Note that we leave the experiment results on 2-shot and 4-ohot settings in Appendix A.3.2.
Comparisons on conventional OOD detection In Table 1, we present the overall results of the comparison between different baseline methods and SCT for OOD detection. Since the prompt tuning based methods engage the ID data during training. the model will generally gain better empirical performance on OOD detection, reflected by evaluation metries like FPR95 and AUROC. IDLike, NegPrompt, and LSN all introduce a set of negative prompts for each ID class to Iearn negative semantics of ID objects using different strategies, which obtain different levels of detection performance gains. Without sacrificing much classification performance (i.e., ID classification accuracy) on ID data, shown in Table 6, our SCT can consistently achieve better OOD detection performance on the large-scale ImageNet- 1k benchmark, which verifies the effectiveness of our methods with the newly proposed modulation factors.
Compatibility with other baselines. In Table 2, we report the results of compatibility experiments, in which we compare those prompt tuning based methods with their variants, incorporating our SCT to dynamically adjust the importance of OOD regularization from samples with different uncertainty levels. We can find that our SCT can consistently help them gain better or comparable
7

---

Table 1: Comparison results on ImageNet-1k OOD benchmarks. All methods are trained on the same backbone CLIP-ViTE ViT--/16. Bold numbers are superior results. f indicates larger values are better, and indicates smaller values are better. Results marked with are taken from [Wang et al., 2023] and [Miyai et al., 2024b]. The prompt tuning based methods are run under multiple trials with reporting the mean and standard deviation of the performance.
OOD dataset
Methud
iNaturalist
SUN
Places365
Textures
Average
FPR95,
AUROC1
FPR95L
AUROC
FPR55L
AUROCT
FPR951
AUROCT
FPR951
AUROCT
Zero-tho methods
MCM
31,86
94,17
37.28
92.55
42 42.94
90.09
98.37
85.83
42.61
90.66
GL-MCM
15.16
96.71
29.16
93.41
37.07
90.37
58.85
83.11
35.06
90,90
CLIP-baved
part- hoo- merhodss
MSPI
74.57
77.74
76.95
73.97
79.72
18
73.66
74.84
74.98
76.22
ODIN
98.93
57.73
$8.72
78.42
87,
76.88
85.47
71.49
90.23
71.13
Energy!
61.98
87.18
46.42
91.17
57.40
87.33
50.39
88.22
51.30
88.48
ReActt
65.57
$6.87
46.17
91.04
87.42
54.62
MaxLogi'
60.88
$8.03
44.83
91.16
55.54
87.45
48.72
88.63
$2.49
88.82
Prempir tuning bosed murtheds
1-chot
CoOp
43. 83.80
91. 10..00
35, $5..22411
92.66-90.3
40.70
90: 100
49.61 49.61
87 77.999077
42 42.38511.0
90 90.6644000
LoCoOp
28.83 13.76
44.05
20.76
94.51 100.5
33.88 30002
61.13g
-
91, 91.75, 34.3
IDLike
12.0726..4
40.56 37.650
91.07 91,07=1
47.0g
38 388.31-1..0
89 07-77-
34.7240
91.67 67.67700
NegPrompt
65.08
84.5642.22
44.30
89.63.40.0
51.31 (16.1
80,55 60.55-2.11
87 87.00-33.1
63.70 63.76-3.02
62:08 3.771
81. 81.134.37
LSN
50.28 18.28477.0
87.2049.55
40.1549. as
91.47:5
46.11-3 les
88. (8..7420.00
60.34
83.92=0
51.47=1.50
87 87.84+
SCT
18.1641.35
95. 06.70
23.5241199
94.58
32.81 11.14
91. 22
48.87 1::04
86 66
31.09 50.48
92.04
16-shot
CoOp
24.25
93.9241.37
31.15
93.1320.a
30.12:3101
90. 20
41 41.88633.0
90.40
35.09=1.m
91 01.09
LoCo
17.584--99
96.39499.57
22.82-00.31
95,20 05.20-2005
32.21
92:033
46.27
88 86-00m
29.47 29:474.0m
983 10:00000
IDLike
8.71
98.0540.07
38.93
90.51100s
47.00-1-44
68.00
92.82---112
91.89-1.44
32. 12.12:3..0
92.14 92.14
NegPrompt
37.79
01
32.1133-77
92.25 on
35.8220011
43.0g==-m
37.3481-41
80.5749000
LSN
36.1754421
02.6641..5
34.2779.44
03.5320
41 41.47300s
90 52
46.43#000
89 39
39 10
91 53:5000
SCT
13.9
25.8649.25
20.55 or
95.33=00 12
29, 29.80=9.0
22
41 =0
89.00 50.00=t0.0
6344750.00
93,3740.00
Table 2: OOD detection performance on compatibility experiments. All methods are trained on the same backbone. 1 indicates larger values are better, and 4. indicates smaller values are better. Methods are run under multiple trials reporting the mean and standard deviation of the performance.
0OD Dataset
Method
INaturalist
SUN
Placex365
Textures
Average
HPR95L
AUROCT
L
AUROCT
FPR05,
AUROCT
EPR95]
AUROC!
FPROS
AUROC
LaCsOp
28.81
54.054002
94.5189.15 25.76
33.
33.68m0-02
91 2p..n-a
51. 51.53499.0
86.85
34.944066
91.75 01.75499.8
SCT
19.16=1:15
95, 70
23.5241.
94.5499:
32.8111116
91 23
48 18,8721.1
86.68
310992-ca
92,00=90
IDLike
12,07=8.6
40.55 15.64
4..99-5.24
12.05
28.34
44.72
91.67 19007
IDI RReSSTT
97.58 12.2422.0
31 31.98=1.0r
92.227 22.227:9:25
44.79=2200
87 40.00
43.5741.00
86.71
33.14-
90.97
LSN
10.28 47.02
40.15
91.4710.1
46.11
88 74 74.377
00.34
83.92 83.92 19.42
$7.80
1.5N-SCT
51.3
35. to
92.48
10.84=0..0
90 to
55, 25.2
86.00=
48.76
29.5
16-shot
LoCoOp
17.5848.22
96. 96.9919.57
06.2010.2 22.82
32.21 m0.21
92 (0..0.000
45 45.2749.45
88 88.8
28.47110.0
93.10 80.co
SCT
13.0
86 10.25
20.55 1..07
10.11
0.47
92 10.0s
41.51 40.41
10.00
26.47
93.37 19.07
8.71
38. 38.8330.0
00.5
47.06=1144
88 tn
32 22
01.80=1.40
32. 1.
92.14
IDL
3.
98 gpannas
33.9849
89.21 19
37,81
87 $7.4
29, 24
90.07
26 26.175h
91.46
LSN
36.17-481
92.00 1.1
41.47-0055
50 50.52
46.4340.30
28.2
10.73
91.00
1SN-SCT
35.14
92.64
29.80*1.6
94.0gt0ac
37.00=0.00
91. 23.23
45.4551.34
80.40 80.40
36.85 t..0
91.85
OOD detection performance across two evaluation metrics while keeping the classification accuracy comparable with the vanilla prompt-tuned model, as shown in Appendix A.3.2.
Comparisons on hard OOD detection. Following the setup in MCM [Ming al., 2022a], also explore the performance of SCT on hard OOD detection tasks, as shown in Table 3. SCT significantly outperforms LoCoOp in all four experimental settings, demonstrating that SCT has strong discriminative power for semantically hard OOD data.
4.3
Ablation study
In this part, we conduct various ablation experiments and further explorations to provide thorough understanding of the characteristies of our proposed SCT. For the extra results and discussions (e.g.. computational cost and social impact), we leave more details in Appendix A.3.2.
8

---

Table 3: OOD detection performance compari- son with LoCoOp on hard OOD detection tasks.
Table Ablation study on the modulation factors for the classification task and regularization term.
Bold numbers represents superior results.
Detailed results can be found in Appendix A.3.2.
ID
00D
Method
FPRU5L
AUROCT
Shot
FPRO51
AUROCT
ID-ACC7
ImageNet-10
ImageNet-20
LoCoOp SCT
28.20
92.75
X
34.94
91.75
69.03
92.38
ImageNet-20
ImageNet-10 10
LoCoOp SCT
34,40
25.00
92.34
94.95
1-shot
x
/
V
31.09
31.90
91.74
92.04
68.80
68.70
ImageNei-10
ImageNat-100
LoCoOp SCT
26.64
30,06
93.00
93.90
x
X
29.30 30
29.47
92.66
93.10
71.43
71.50
ImageNer-100
ImageNes-10
LoCoOp SCT
16-shot
X
/
28.
26.47
93.37
92.62
71.77
71.90
Importance of the modulation factors in SCT. As an important aspect of the learning objective of SCT in Eq. (4), the newly introduced modulating factors in the learning objective of SCT conduct adaptive redirection of prompt tuning towards suitable tasks. Although each of modulating factors seemingly achieve the redirection effect alone, empirically find that both modulating factors play significant role in enhancing OOD detection performance of VLMs, as shown in Table 4.
Influence of regularization weight in OOD regularization. The regularization weight À controls the contribution of OOD regularization to the prompt learning process like the role of the two modulation factors. In Figure 3(a), we show the performance by varying the ratio À 1. It is worth noting that setting high regularization weights such as X = 1 may even degrade the performance, indicating that the ID classification task should be attached more importance for better OOD detection.
Generality of using different OOD regularization functions. Since SCT introduces a general learning framework of prompt tuning for OOD detection, the specific realization for the OOD regularization function can have multiple choices (e.g., MSP [Hendrycks and Gimpel, 2017] or Energy score function [Liu al., 2020)). Here we report the performance using different OOD regularization functions in Figure 3(b), where they have different performance improvements compared with the original LoCoOp baseline. Specifically, the energy regularization needs tuning the two energy threshold hyperparameters min and Mout, limiting its advantages over other regularization functions.
Implementation with different CLIP architectures. We evaluate SCT with different VLM architectures and the results are shown in Figure 3(c). The result illustrates that the larger backbone boosts the performance of OOD detection and also shows SCT outperform LoCoOp across various VI.M architectures. It is important to note that we take the same hyperparameters across all architectures, demonstrating the robustness of SCT hyperparameters on different VLM architectures.
Comparison between different methods for extracting OOD features. In Figure 3(d), we perform the comparison of our method adopting different methods for extracting OOD features, including the probability-base on OOD detection across all the OOD feature extraction methods. Specifically, the probability-bane -based method and entropy-based method both have a threshold hyperparameter to discriminate between ID and OOD features. The sensitivity to hyperparameters of different methods might be the reason behind the different OOD detection performance. For the entropy-based method, the performance is significantly poor since it challenging determine the appropriate threshold [Miyai al., 2024b].
and entropy-based methods. The results verify the superiority of SCT
5 Diseussions and limitations
Comparisons with advanced post-hoc methods. Recently, advanced post-hoc ap- proaches [Djurisic al., 2022, Xu et al., 2024] exhibit comparable OOD detection performance to tuning based methods, despite the lack of training data. However, prompt tuning based methods can leverage the generalization ability VLMs to better fit the domains of the downstream tasks given only few-shot ID training data. What's more, post-hoc methods and prompt tuning based methods are compatible with each other, further boosting the OOD detection performance. We provide more detailed discussions, including empirical experiments, on the compatibility of SCT with advanced post-hoc methods in Appendix A.3.2. Furthermore, future research efforts into post-hoc calibration methods for prompt tuning based OOD detection could also contribute to the community.

---

480
SCT
LuG4l
27.2
26.0
FT-005
222
22.
180
Rogulatzafon Weight
Regilarzaton Regular nation Funation
Mode Archilecture
Extracioon Method
(a) Regularization Weight
(b) Regularization Func.
(c) Architecture
(d) Extraction Method
Figure 3: Ablation study. (a) performance of using different regularization weights (b) exploration of different regularization functions for OOD regularization; (c) using different CLIP architectures; (d) comparison of different methods for extracting OOD features.
Performance improvement on the AUROC metric. The experiment results in Table and indicate that the improvements of SCT over LoCoOp AUROC are less notable than the FPR95 metric. However, as shown in the comparison of different baselines in Table I, the improvement space for FPR95 is significantly larger than AUROC. Therefore, the progress these two metrics should not treated equally. Nevertheless, further investigation is necessary to enhance the OOD detection performance specifically targeting improvement on the AUROC metric.
The sensitivity to training data under the few-shot setting. VLMs are exposed limited ID data samples under the few-shot scenarios, which means that the OOD detection performance of prompt tuning based methods, including SCT, can susceptible to the quality limited ID training data in practice. Exploration of selection strategies of suitable training data or overcoming the data sensitivity inherent in the few-shot setting could further facilitate the practical application of prompt tuning based OOD detection methods in real-world scenarios.
Theoretical analysis of the proposed method. Although we propose novel framework to mitigate the problem of invalid extracted OOD features, we have not yet provided sufficient theoretical analysis prove the effectiveness of our method. We choose the linear function as the modulation function for the sake of simplicity, instead of based on theoretical justification. Conducting theoretical analyses on the relationship between sample uncertainty and OOD detection performance of prompt tuning based methods under few-shot settings is also potential direction for future work.
6 Conclusion
In this paper, we propose a novel learning framework, i.c., Self-Calibrated Tuning (SCT), that improves the OOD detection capability of VLMs with only the given ID training data. To mitigate the problem caused by invalid OOD features mined from ID data, SCT introduces two modulating factors to the original learning objective to conduct adaptive redirection of prompt tuning process between the tasks of classification and OOD regularization. Through the redirection effect, our method calibrates the impact of OOD features extracted from different ID samples based on the sample uncertainty estimation during the training process, which facilitates the model learning from imperfect surrogate OOD features for OOD regularization. We have conducted extensive experiments to demonstrate the effectiveness SCT and its compatibility with range prompt tuning based methods, along with various ablation studies and further explorations to characterize the framework.
Acknowledgments and Disclosure of Funding
GY and JCY were supported by the National Key R&D Program of China (No. 20222D0160703), 111 plan (No. BP0719010) and National Natural Science Foundation of China (No. 62306178). JNZ and BH were supported by NSFC General Program No. 62376235, Guangdong Basic and Applied Basic Research Foundation Nos. 2022A151500 1652 and 2024A151501 2399, RIKEN Collaborative Research Fund, HKBU Faculty Niche Research Areas No. DC-FNRRA-IG/22-23/5 CSD Departmental Incentive Scheme.
and HKBU
10

---

References
Yichen Bai, Zongbo Han, Changqing Zhang, Bing Cao, Xiaoheng Jiang, and Qinghua Hu. Id-like prompt learning for few-shot -uttr--distribution detection. arXiv preprint arXiv:2311.15443 2023.
Mircea Cimpoi, Subhransu Maji, lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. De- scribing textures in the wild. In CVPR, 2014.
Jesse Davis and Mark Goadrich. The relationship between precision-recall and roe curves. In JCML, 2006.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
Andrija Djurisic, Nebojsa Bozanic, Arjun Ashok, and Rosanne Liu. Extremely simple activation shaping for out-of-distribution detection. arXiv preprint arXiv:2209.08858, 2022.
Alexey Dosovitskiy. Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021.
Xuefeng Du, Zhaoning Wang, Mu Cai, and Yixuan Li. Vos: Learning what you don't ton't know by virtual outlier synthesis. In ICLR, 2022.
Tom Fawcett. An introduction to roc analysis. Pattern recognition letters, 2006.
Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of-distribution examples in neural networks. In ICLR, 2017.
Dan Hendrycks, Mantas Mazeika, and Thomas Dietterich. Deep anomaly detection with outlier exposure. ICLR, 2019.
Dan Hendrycks, Steven Basart, Mantas Mazeika, Mohammadreza Mostajabi, Jacob Steinhardt, and Dawn Song. Scaling out-of-disti detection for real-world settings. In ICML, 2022.
Rui Huang and Yixuan Li. Mos: Towards scaling space. In CVPR, 2021.
detection for large semantic
Rui Huang, Andrew Geng, and Yixuan Li. On the importance of gradients for detecting distributional shifts in the wild. In NeurIPS, 2021.
Xue Jiang, Feng Liu, Zhen Fang, Hong Chen, Tongliang Liu, Feng Zheng, and Bo Han. Negative label guided ood detection with pretrained vision- languaee models. arXiv preprint arXiv:2403.200778 2024.
Muhammad Uzair Khattak, Hanoona Rasheed, Muhammad Maaz, Salman Khan, and Fahad Shahbaz Khan. Maple: Multi-modal prompt learning. In CVPR, 2023.
Alex Krizhevsky. Learning multiple layers of features from tiny images. In arXiv, 2009.
Tianqi Li, Guansong Pang. Xiao Bai, Wenjun Miao, and Jin Zheng. Learning transferable negative prompts for -iitt--diitiibuttinn detection. arXiv preprint XXXv:2444.03224: 2024.
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In ACL, 2021.
Shiyu Liang, Yixuan Li, and Rayadurgam Srikant. Enhancing the reliability of out-of-distribution image detection in neural networks. In ICLR, 2018.
T Lin. Focal loss for dense object detection. arXiv preprint arXiv: (708.000000000000 2017.
Weitang Liu, Xiaoyun Wang, John Owens, and Yixuan Li. Energy-based out-of-dissribution detection. In NewrIPS, 2020.

---

Fan Lu, Kai Zhu, Kecheng Zheng, Wei Zhai, and Yang Cao. Likelihood semantic alignment for full spectrum -iittii-ittiittiionn detection. arXiv preprint arXiv:2312.01752, 2023.
Yifei Ming. Ziyang Cai, Jiuxiang Gu, Yiyou Sun, Wei Li, and Yixuan Li. Delving into out-of-
distribution detection with vision-language representations. In NewrIPS, 2022a.
Yifei Ming, Ying Fan, and Yixuan Li. Poem: Out-of-distribution detection with posterior sampling. In ICML, 2022b.
Yifei Ming, Hang Yin, and Yixuan Li. On the impact of spurious correlation for out-of-distribution detection. In AAAI, 2022c.
Atsuyuki Miyai, Qing Yu, Go Irie, and Kiyoharu Aizawa. Zero-shot in-distribution detection in multi-object settings using vision-languaa foundation models. arXiv preprint arXiv:2304.04521, 2023.
Atsuyuki Miyai. Jingkang Yang, Jingyang Zhang, Yifei Ming. Yueqian Lin, Qing Yu, Go Irie, Shafig Joty, Yixuan Li, Hai Li, et al. Generalized outtof-ddittibution detection and beyond in vision language model era: A survey. arXiv preprint arXi::240.221794 2024a.
Atsuyuki Miyai, Qing Yu, Go Irie, and Kiyoharu Aizawa. Locoop: Few-shot -itt---diittibttion detection via prompt learning. NeurlPS, 36, 2024b.
Jishnu Mukhoti, Viveka Kulharia, Amartya Sanyal, Stuart Golodetz, Philip Torr, and Puneet Dokania. Calibrating deep neural networks using focal loss. In NeurlPS, 2020.
Mahdi Pakdaman Naeini, Gregory Cooper, and Milos Hauskrecht. Obtaining well calibrated proba- bilities using bayesian binning. In AAAI, 2015.
Anh Nguyen, Jason Yosinski, and Jeff Clune. Deep neural networks are easily fooled: High confidence predictions for unrecognizable images. In CVPR, 2015.
Jun Nie, Yonggang Zhang, Zhen Fang, Tongliang Liu, Bo Han, and Xinmei Tian. Oft-t--distributiin detection with negative prompts. In ICLR, 2023.
Changdae Oh, Mijoo Kim, Hyesu Lim, Junhyeok Park, Euiscog Jeong, Zhi-Qi Cheng, and Kyung- woo Song. Towards calibrated robust fine-tuning of vision-language models. arXiv preprint arXiv:2311.01723, 2023.
Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training. OpenAl, 2018.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, 2021.
Kuniaki Saito, Donghyun Kim, Stan Sclaroff, and Kate Saenko. Universal domain adaptation through self supervision. In NeurIPS, 2020.
Taylor Shin, Yasaman Razeghi, Robert L Logan IV, Eric Wallace, and Sameer Singh. Autoprompt: Eliciting knowledge from language models with automatically generated prompts. arXiv preprint arXiv:2010.15990 2020.
Ximeng Sun, Ping Hu, and Kate Saenko. Dualcoop: Fast adaptation multi-label recognition with limited annotations. In NeurIPS, 2022.
Yiyou Sun and Yixuan Li. Dice: Leveraging sparsification for out-of-distributiio detection. In ECCV, 2022.
Yiyou Sun, Chuan Guo, and Yixuan Li. React: Out-of-distributioo detection with rectified activations. In NeurIPS, 2021.
Leitian Tao, Xuefeng Du, Xiaojin Zhu, and Yixuan Li. Non-parametric outlier synthesis. In ICLR, 2023.
12

---

Weijie Tu, Weijian Deng, Dylan Campbell, Stephen Gould, and Tom Gedeon. An empirical study into what matters for calibrating vision-language models. In JCML, 2024.
Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. The inaturalist species classification and detection dataset. In CVPR, 2018.
Hualiang Wang, Yi Li, Huifeng Yao, and Xiaomeng Li. Clipn for zero-shot ood detection: Teaching clip say In ICCV, 2023.
Shuoyuan Wang, Jindong Wang, Guoqing Wang. Bob Zhang, Kaiyang Zhou, and Hongxin Wei. Open-vocabulary calibration for vision-language models. In ICML, 2024.
Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey zoo. CVPR, 2010.
Kai Xu, Rongyu Chen, Gianni Franchi, and Angela Yao. Scaling for training time and post-hoc diloo--dittrbbution detection enhancement. In ICLR, 2024.
Pingmei Xu, Krista A Ehinger, Yinda Zhang, Adam Finkelstein, Sanjeev R Kulkarni, and Jianxiong Xiao. Turkergaze: Crowdsourcing saliency with webcam based eye tracking. arXiv preprint arXiv:1504.0675 2015.
Fisher Yu, Ari Seff, Yinda Zhang. Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. Lsun: Construction of a large-scale image dataset using deep learning with humans in the loop. In arXiv, 2015.
Bolei Zhou, Aditya Khosla, Agata Lapedriza, Antonio Torralba, and Aude Oliva. Places: An image database for deep scene understanding. arXiv preprint 1610000555 2016.
Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralb.. Places: A 10 million image database for scene recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017.
Kaiyang Zhou, Jingkang Yang. Chen Change Loy, and Ziwei Liu. Learning prompt for vision- language models. International Journal of Computer Vision, 2022a.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for lision-language models. CVPR, 2022b.
Jianing Zhu, Hengzhuang Li, Jiangehao Yao, Tongliang Liu, Jianliang Xu, and Bo Han. Unleashing mask: explore the intrinsic out-of-distribttion detection capability. In ICML, 2023a.
Jianing Zhu, Geng Yu, Jiangchao Yao, Tongliang Liu, Gang Niu, Masashi Sugiyama, and Bo Han. Diversified outlier exposure for out-of-distribution detection via informative extrapolation. In NeurlPS, 2023b.
13

---

A Appendix Supplemental Material
The whole Appendix is organized follows. Appendix A.I, we present the detailed definitions and implementation of zero-shot, post-hoc methods and several prompt tuning based methods that are considered in our experiments. In Appendix A.3, we provide our extra experimental details and more comprehensive results with further discussion on the underlying implications. Finally, in Appendix A.4, we discuss the potential broader impact and limitations of our work.
Reproducibility statement
To ensure reproducibility, we outline several key aspects about the experiments below:
Datasets. The datasets we used are publicly accessible, which are introduced in Sec- tion 4.1.
Open source. The source code is publicly available at https://githe: com/ tmir-group//ST.. We provide backhone for experiments well several aux- iliary components, such as OOD detection performance evaluation.
Environment. All experiments are conducted with multiple runs on NVIDIA GeForce RTX 3090 GPUs with Python 3.8 and PyTorch 1.12.
A.1 Details about considered baselines
In this section, we present the details about the baselines, including zero-shot, post-hoc methods, and several prompt tuning based methods, well as related hyper-parameters that considered in work.
MSP. Hendrycks and Gimpel [2017] proposes to utilize maximum softmax probabilityMMSP) the scoring function to differentiate between ID and OOD samples, of which the definition is as follows,
SMsP( a:; f) max P(y m æ;; f) = max sof (of(max
where / represents given well-trained model and m is one of the classes ) = (1...., M). A higher MSP score signifies a greater probability that a sample belongs to the mn-distribttin distribution, indicating the model's confidence the sample.
Liang et al. [2018] designs the ODIN score function, utilizing the temperature scaling and minor perturbations to test samples to widen the gap between the distributions of ID and OOD data. The ODIN score is defined as follows,
ODIN.
f(
SODIN (a:; f) max P(y m/ m max sof tmax
m
where 7 denotes the perturbed samples (controled by e) and T denotes the temperature.
Energy. Liu et al. [2020] proposes to the Energy of the prediction logits to discriminate between the ID and OOD samples. The Energy score is formulated as follows.
M
Senergy (x: f) = -T -Tlog log
(w))/T
(10)
m=1
where T' denotes the temperature parameter. As theoretically proved in Liu et al. [2020], lower Energy score represents higher probability for sample to belong to ID.
ReAct. Sun et al. [2021] designs simple and effective approaches, named Rectified Activations
(ReAct), to alleviate model overconfidence on out-of-distribution data. This work observes that OOD samples can induce unusually high unit activation in the deep layer of neural networks. ReAct improves OOD detection by simply rectifying the activations an upper limit e > 0, which can be performed on pretrained model without any modification to training.
14

---

MaxLogit. Hendrycks et al. [2022] demonstrates that the conventional MSP OOD detector does not scale well to challenging large-scale multiclass and multi-label OOD detection setting and proposes a surprisingly simple detector based on the maximum logit, called MaxLogit that greatly outperforms previous post-hoc methods. The definition of MaxL.ogit is as follows,
SMaxLogit (x; f) max f(x)
(11)
Ming et al. [2022a] explores the potential of large-scale vision-language models like CLIP for OOD detection and proposes simple yet effective zero-shot OOD detection method called Maximum Concept Matching (MCM), which is based on aligning visual features with textual concepts. This method characterizes OOD uncertainty by appropriately scaling of the distance from the visual feature to the nearest ID prototype, of which the formulation is as follows,
MCM.
exp (sim (f. (m.)mmm) /T)
SMCM max m
(12)
M Em'=1 exp (sim (f (m))))))))) /T)
where 7 denotes the temperature parameter, f and 9m represents image and text features respectively, and sim() represents the cosine similarity between the image and text features.
GL-MCM. Miyai et al. [2023] designs a zero-shot OOD detection method called Global-Local Maximum Concept Matching(GL-MCM) that leverages the maximum softmax score of both global and local image features and designs. Utilizing the matching score with local features can compensate for the low global matching score and produce more accurate ID confidence. The detailed formulation of GL-MCM is shown in Eq (7).
CoOp. Drawing inspiration from the success in prompt learning within natural language processing, Zhou et al. [2022a] proposes Context Optimization (CoOp), a simple framework specifically to adapt vision-language models like CLIP for downstream image recognition. Concretely, CoOp represents the context words of a prompt as learnable vectors while keeping the entire pretrained encoders fixed. The specific framework of CoOp is presented in Section 3.1.
LoCoOp.
Miyai al. [2024b] introduces an approach named Local regularized Context Opti- mization (LoCoOp) to enhance the OOD detection performance of prompt tuning. It first extracts ID-irreverent contexts from CLIP's local features and utilizes it as OOD features to perform OOD regularization with entropy maximization. The details of this method are provided in Section 3.1
IDLike. Bai al. [2023] designs a new framework of prompt tuning for OOD detection to focus on challenging OOD detection scenarios, It first constructs surrogate outliers from ID samples by conducting multiple random cropping on ID data and filtering them based on their cosine similarity with textual features of ID classes. Then it introduces a set of OOD prompts along with conventional ID prompts and optimizes them with a new loss function that consists of in-distribution loss, out-of- distribution loss and diversification regularization.
NegPrompt. Li et al. [2024] proposes prompt tuning based OOD detection method, named NegPrompt, which learns a set of negative prompts for each ID class, with only II) data, to capture the negative semantics associated with these classes. The training process is divided into two stages. In the first stage, the model learns the positive prompts. In the second stage, the positive prompts are kept frozen and the negative prompts are optimized via three loss functions that enforce the separation between negative prompts and ID images, proper degree of similarity between negative and positive prompts, and the diversity of the negative prompts.
Nie et al. [2023] reveals that CLIP cannot fully understand the negative semantics of textual information and proposes to learn a set of negative prompts for each class to alleviate this problem. The learned positive prompt (for all classes) and negative prompts (for each class) are utilized simultaneously to calculate similarity and dissimilarity in the feature space, thereby enhancing the accuracy of OOD sample detection.
LSN.
15

---

A.2 In-depth comparison between SCT and hard example mining.
The part of ID classification in the learning framework of SCT bears strong resemblance hard example mining methods, such as focal loss [Lin, 2017]. In this section, we clarify the novelty and insights of our SCT by analyzing the difference between SCT and hard example mining as follows.
Conceptually, the motivation of SCT is to mitigate the problem of unreliable QOD features in prompt tuning based OOD detection methods. Generally, these methods rely on the ID>-rrelevant local context extracted by VLMs as the surrogate OOD features to perform regularization, the quality of which is greatly affected by the inaccurate gareground-backrroune shown in Figure and Figure 5, although VLMs can mask out some ID-related regions (shown as the grey patches of images), large portions of the extracted OOD features (shown the colored patches of images) obviously belongs to ID features.
decomposition of VLMs. As
Empirically, find that the quality of extracted OOD features significantly correlated with the uncertainty level of ID data. As illustrated in the left panel of Figure 2, the extracted OOD features become more inaccurate the uncertainty increases. In the right panel of Figure 2, we train LoCoOp on multiple data groups with different uncertainty levels. The results demonstrate that the OOD detection performance of LoCoOp can be significantly impacted by the uncertainty level of ID data. Therefore, to mitigate the issue of unreliable OOD features, we propose SCT to calibrate the influence of OOD regularization from different ID samples based on their uncertainty level.
Technically, despite the simple design, SCT is significantly different from hard sample mining. The latter conducts reweighting directly the samples based the classification difficulty during training. The former adaptively adjusts the importance between the two components of the original learning objectives for every single sample. Data with high uncertainty are directly down- weighwe in hard sample mining while they are utilized more for OOD regularization in SCT. As shown in Table 4, under 16-shot ID data, the OOD detection performance of simply assigning 1 p(y)=)) to Lee (denoted as & v and d x) are significantly inferior to SCT (denoted as Φ v and y v), demonstrating the difference of SCT and hard sample mining.
A.3 Additional experimental results and further discussion
In this section, we provide more experiment results from various perspectives to characterize our proposed SCT. First, we introduce the additional experimental setups for the empirical verification in previous figures and our learning framework. Second, we offer more detailed results and analyses of our method in comparison to other advanced baselines. Finally, more demonstrations of the motivation of our method are provided.
A.3.1 Additional experimental setups
In the right panel of Figure 2, conduct experiments to investigate the relationship between sample uncertainty and OOD detection performance of prompt tuning based methods. We first calculate the prediction probability for ground-truth labels of all the training samples in 64-shot training set using a prompt-tuned CLIP model. This model is prompt-tuned with LoCoOp on a 4-shot training which contains no overlapping samples with the 64-shot set. We use the prediction probability for ground -truh labels to represent uncertainty. hiighhuncerainty samples are assigned low prediction probability for their ground-truth labels and vice versa. We choose the data with the lowest and highest uncertainty for every ID class to generate two data groups of specific shots with different uncertainty levels respectively, and train the model with LoCoOp on these two data groups.
Figure 2.
Figure 3. In Figure 3(b), we explore different regularization functions to perform OOD regular- ization, including entropy maximization [Miyai et al., 2024b], cross-entropy loss the uniform distribution [Hendrycks et al., 2019, Ming et al., 2022b] and the energy-based function [Liu et al., 2020], under the 16-shot setting. In Figure 3(a), Figure 3(c) and Figure 3(d), we train the models of all the architectures with 16-shot training datasets. In Figure 3(d), we evaluate the performance of SCT with various methods for ID-irrelevant region extraction. To be specific, we consider three different methods, including the making-bbssed method, probability-base method, and entropy-based method. Following the setups in [Miyai et al., 2024b], for the antropy-based method, we extract local regions where the entropy of p. (a))) is lower than log since it the half value of the maximum entropy of M -immensiooall probabilities, done in previous studies [Saito et al., 2020]. For the
16

---

Table 5: OOD detection performance of SCT under various few-shot settings. All methods are trained on the same backbone CLIP- ViT-B/16. Bold numbers are superior results. 1 indicates larger values better, and 4 indicates smaller values are better.
OOD dataset
Method
INaturalist
SUN
Places365
Textures
Average
FPR95,
AUROC1
FPR95]
AUROC1
FPR95
AUROCT
FPR95
AUROCT
FPR95L
AUROC1
2-shot
LoCoOp SCT
96.01 96.15
23.02
32.32
46.35
88.60
30,28
92.91
16,99
21.68
94.79
31.01
92.21
42.62
88.64
28.08
92.95
4-shot
LoCoOp
17.32
96.26
23.78
94.82
32.19
91.85
44,41 45.53
89.22 87.96
29.42
93.04 92.98
SCT
13.88
97.03
22.13
94.85
30.20
92.09
27.93
Table 6: ID classification performance of SCT and all the considered baselines. All methods are trained on the same backbone
ID Accuracy
Method
1-shot 16-shot
Zero-shot methods
MCM
66.7
GL-MCM
66.7
CLIP-based post-hoc methods
MSP
66.7
ODIN
66.7
Energy
66.7
ReAct
66.7
MaxLogit
66.7
Prompt tuning based methods
CoOp
68.83
71.93
LoCoOp
69.03
71.43
IDLike
68.90
71.04
NegPrompt
68,83
71.93
LSN
68,83
71.93
SCT
68.80
71.77
IDLike+SCT
69.70
71.14
LSN+SCT
68.98
71.50
probability- method, we extract regions where (x((z) is lower than 1/M simply because the ID dataset has M classes.
A.3.2 Additional experimental results and illustrations
Experiments multiple few-shot settings. In order to further understand the effectiveness of our SCT on different few-shot settings, we report the evaluation results of our SCT with 2-shot and 4-shot training datasets, which summarized in Table 5. The results demonstrate that SCT can gain significant improvement on OOD detection under all the few-shot settings.
ID classification performance of baselines and SCT Table and Table 2. To evaluate the performance of all the considered baselines and SCT on the original classification task, we report the classification accuracy on the test set in Table Since NegPrompt and LSN learn positive prompts and negative prompts separately, they have the same ID classification performance vanilla CoOp. The results show that SCT maintains comparable ID classification performance compared with vanilla prompt tuning CoOp and other advanced prompt tuning based OOD detection methods while achieving more effective OOD detection.
Fine-grained results of OOD test dataset for experiments in Table further evaluate the influence of both modulating factors introduced in Eq (4) the performance every OOD test
17

---

Table 7: Fine- teine-gained results of OOD detection performance for experiments in Table 4. All methods are trained on the same backbone ViT-B/16. Bold numbers are superior results. 1 indicates larger values better, and indicates smaller values better.
OOD dataset
NNaturalist
SUN
Places665
Textures
Average
FPR95
AUROCY
FPR95
AUROCT
FPR9S
AUROCT
FPR9SL
AUROC
FPR95
AUROC1
1-shot
x
28.81
94.05
25.76
94.51
33.08
91.59
51.53
86.85
34.94
91.75
x
21.94
95.14
22.46
95.08
30.54
92.04
49.61
87.14
31.14
92.35
X
/
19.26
95.72
23.70
94.44
33.56
91.04
51.10
85.77
31.90
91.74
19.16 /
95.70
23.52
94.58
32.81
91.23
48.87
86.66
31.09
92.04
16-shot
17.58
96.30
32.21
92.03
45.27
88.86
29.47
93.10
/
X
18.14
94.50
22.42
94.04
31.90
91.18
44.72
90.09
29.30
92.66
X
15.08
96.92
21.42
95.16
30.60
92.07
48.64
86.35
28.94
92.65
13.94
95.86
92.24
26.47
93.37
Table Different implementations of modulation functions.
Modulation function
vb
power
(1 ply x w)
ply an; w)
logarithmic
log (p(pla=w)11 log
log (ply ++]
log
trigonometric
cos (u w))
sin (fp(yla:w)
test, we present the fine-grained OOD detection results in Table 7. We observe from the results that although each of the modulation factors can individually contribute redirect model training, their combination maximizes the improvement on OOD detection, which achieves the best results on most OOD test sets and shot numbers.
Experiments other instantiations of modulation functions. To explore the generality of our proposed learning framework, we consider other instantiations of modulation functions. Specifically, consider the power, logarithmic, and trigonometric functions, which are formulated in Table 8. For the experiment on the power function, we set = 0.25, a = 0.5 for 1-shot and À = 0.25, a = 4 for 16-shot. The experiment results, shown in Table 9, indicate that all the considered choices of modulation function can achieve significantly better OOD detection performance than LoCoOp, which further empirically proves the effectiveness of the learning framework of SCT.
Experiments on conventional CIFAR benchmarks. We conduct the experiments on conventional CIFAR benchmarks, following previous works [Liu al., 2020]. We adopt CIFAR- CIFAR- 100[Krizhevsky. 2009] the II) datasets and use Textures [Cimpoi et al., 2014], Places365 [Zhou et al., 2016], iSUN [Xu et al., 2015], LSUN _SUN_Crop [Yu et al., 2015], and LSUN Resize [Yu et al., LSUN_Resize 2015] as the OOD test datasets. We summarize the comparison results averaged across all OOD test datasets Table 10, which confirm that SCT achieves superior performance to LoCoOp.
Experiments on Computational Cost SCT doesn't incur any extra computational cost to the LoCoOp due to its simple design. Technically, SCT introduces modulating factors respectively on the two components of the original learning objective. The modulating factors (instantiated
p(y]<> for è and z) for db in Equation (4) in the submission) only involves the computation of the prediction probability of ground truth classes p(y]æ), which can be repeatedly used after the original forward pass of CLIP models. What's 's more, the operation of local features involved in LoCoOp and SCT are also relatively low-cost in terms computation. The local are generated from forward pass of the vision encoders of CLIP, which doesn't i bring additional computational cost compared to regular training. Regarding the extraction of OOD features, we compute the similarity between every local feature and text feature of all the ID classes and we identify regions that do not include their
18

---

LoCoop
SCT
0.10
0.08
0.08
0.00
Shot
Figure 4: The comparison of calibration measured by ECE between SCT and LoCoOp trained with 1, 4, 16 shots data. The evaluation performed on the original validation ImageNet 1k.
Table 9:
OOD detection performance different instantiations of modulation functions. All methods trained on the same backbone CLIP-ViT-B/IE ViT-B/16. Bold numbers superior results. 1 indicates larger values are better, and J indicates smaller values are better. SCT-L denotes SCT with the linear function, SCT-Pow denotes SCT with the power function, SCT-Log denotes SCT with the logarithmic function and SCT-Tri denotes SCT with the trigonometric function as the modulation function.
OOD dataset
Method
iNaturalist
SUN
Places365
Textures
Average
FPR95
AUROC1
FPR95
AUROCT
FPR95,
AUROCT
FPR95,
AUROC1
FPR95,
AUROCT
1-shot
LoCoOp
28.81
91.05
25.76
91.51
33.68
91.39
51.53
86.85
34.94
91.75
SCT-I
95.70
94.58
32.81
91.23
86.66
SCT-Pow
18.91
95.84
25.06
94,61
33.36
91.68
49.13
86.68
31.62
92.20
SCT-Log
18.32
95.98
26.00
94.62
33.42
91.39
51.17
85.97
32.23
.9
SCT-Tri
18.46
95.84
24.32
94.98
33.23
91.69
54.10
86.15
32.53
92.17
16-shot
LoCoOp 17.58
96.30
22.82
95.20
32.21
92.03
45.27
88.56
29.47
93.10
SCT-L
13.94
95.86
20.35
95.33
29.86
92.24
41.51
89.06
26.47
93.37
SCT-
14.07
96.70
20.74
94.75
30.11
91.96
43.49
87.90
27.10
92.83
SCT-Log
13.11
20.56
95.50
29.03
92.
87.61
27.06
93
SCT-Tri
14.88
96.81
20.30
95.44
29.33
92.50
44.84
87.90
27.34
93.16
Table 10: Experiments CIFAR benchmark with 16-shot data.
D
Method
CIFAR-10 AUROCT
CIFAR-100 AUROC1
FPR95.
ID-ACC1
FPR95JJ
ID-ACC1
MCM
16.36
95.68
90.10
74.92
77.62
68.40
LoCoOp
15.30
95.37
93.00
64.66
82.45
72.20
SCT
12.89
96.10
93.10
60.74
84.91
72.30
ground truth class in the top-K K predicted classes as ID-irrelevant regions. Empirically, we evaluate the time and memory consumption of SCT compared with other baselines in Table 11 and the results show that SCT is relatively compute-efficient. The evaluation is conducted on single GTX-3090 GPU with a batch size 32.
Table 11: Time and memory cost of different methods on the ImageNet benchmark.
Method
Time for one iteration (s)
GPU Memory (MiB)
FPR95
AUROC
ID-ACC
CoOp
0.70
21140
35.09
91.99
71.93
LoCoOp
0.96
23036
29.47
93.10
71.43
SCT
0.96
23036
26.47
93.37
71.77
19

---

+
W
A merican coot
vase
combination lock
school bus
green mamba
cuirass
parking meter
hair slide
coral fungus
siamang
Figure 5: Examples of the invalid OOD features extracted by CLIP. The odd-numbered rows show original images from ImageNet-Ik and even-numbered rows show the extracted ID-irrelevant context from the corresponding images. The ground-truth labels are annotated below every even- numbered row. Although CLIP can mask out some ID-related regions (shown as the gray patches of images), large portions of the extracted OOD features (shown the colored patches of images) obviously belong to features.
Discussions about current advanced post-hoc methods and our method. prompt tuning based method can leverage the generalization ability of VLMSs better fit the domains of the downstream tasks with relatively low computational cost. Post-hoc methods need to be built on a well-trained model, the capacity of which greatly affects the OOD detection performance. What's more, post-hoc methods and prompt tuning based methods are compatible with each other, further boosting the OOD detection performance. We conduct experiments on the compatibility of an advanced post-doc method, NegLabel [Jiang et al., 2024], with SCT in the Table 12, and the results show that SCT can be combined with post-hoc methods for better OOD detection.
In addition, recent advanced post-hoc methods. such ASH [Djurisic et al., 2022] and ISH [Xu et al., 2024], improve OOD detection from the perspective of activation scale. However, VLMs like CLIP compute prediction probability based on the cosine similarity between image and text features. The computation of cosine similarity involves the normalization of features, which naturally eliminates the effect of the activation scale. Therefore, ASH and ISH can't can't apply to VLM models.
Improvement confidence calibration. We Expected Calibration Error (ECE) [Naeini et al., 2015] to measure the improvement of SCT on confidence calibration over LoCoOp. with lower values indicating better calibration. ECE is calculated by dividing predictions equally-spaced bins by confidence scores, then computing the mean average of the difference between each bin's is accuracy and confidence. It can be formulated ECE
samples into M
Σ m=1
.
Bin..l
|ace Binm.
20

---

Table 12: Experiments on compatibility of Neg-Label and SCT on 16-shot data.
OOD dataset
throshold
INaturalist
SUN
Places365
Textures
Average
FPR95]
AUROCT
FPR95)
AUROCT
FPR95,
AUROCT
FPR95,
FPR95]
AUTROCT
NegLabel
20.53
35.59
43.56
90.22
25.40
94.21
NogLabel+SCT
2.03
99.51
17.42
95.92
31.43
92.46
38.46
91.23
22.34
94,78
(Bitm where denotes the number of samples. As shown in Figure 4, SCT can consistently enhance the calibration VLMs across all few-shot settings.
A.3.3 More empirical demonstrations and illustrations of our research problem.
In this section, present more empirical evidence and illustrations of the research problem that inspires our SCT in Fig 5. As illustrated, portions of the extracted local features from data are invalid OOD features, thus degrading the performance of OOD detection.
A.4 Broader impact
OOD detection is crucial for deploying reliable deep learning systems real-world scenarios INguyen et al., 2015, Hendrycks et al., 2022]. This significance is particularly evident in safety- critical domains such as finance or medical intelligence, where trustworthy model must accurately distinguish between samples belonging providing predictions based on known classes (c.g., financial products or medical conditions). Our rescarch focuses general and practical challenge concerning prompt tuning methods for improving OOD detection efficacy. We specifically focus on the issue spurious OOD features extracted from ID samples. It is important for effective OOD detection to gain better empirical performance through regularizing prompt tuning.
distinct label spaces (e.g., animals) rather than simply
21

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The main claims made in the abstract and the introduction accurately reflect the paper's 's contributions and the scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We have discussed the limitations of the work in Section 5.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
22

---

Justification:
This paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibiliyy
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
The experimental setups are provided in Section 4.1 to ensure the result reproducibility.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
23

---

Answer:
[Yes]
Justification:
The code is publicly available at: httpp:://github.com.
com/ wttp://cooootmmmeeeeeeeeeeeeeeet http://hwww...ooommooooooou/////
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/ http://dddde/dceddieeeeee
ooiicylyyy))oo for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: / //nips. wtt.cc/ppuuc wtte//ublccggggiileeeeeee
Sttpa///ooeSSbiiiieeee wwwsssinooliooooooo for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide the detailed experimental setups in Section 4.1 and more discussion in Appendix A.1.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We demonstrate the experimental statistical significance by reporting the mean and std value based on multiple trials of the experiments Tables 1, 2, and other results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide the details of the used experiment compute resources in the reproducibility statement in Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics attps://neuripg: http:://wwrrpps.ccpppcl///e/eeee hccpppublcc/.hhhccsuuuieeeeee ines?
Answer: [Yes]
Justification: The research conducted in the paper conforms, every respect, with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We discuss the potential positive and negative societal impacts of the work in Appendix A.4.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
25

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: This paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The creators or original owners of the datasets and referred codes are properly mentioned in the references and their credits are properly respected.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
26

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
27


Reviews:
[{"review_id": "RWUCm7uvv2", "reviewer": "Reviewer_wD2g", "summary": "This paper focuses open-set detection method based on CLIP. The authors propose an additional weighting mechanism based on the LoCoOp method to alleviate the problem that the outlier related regions extracted by the LoCoOp method are not trustworthy in some cases.", "questions": "Please clarify the novelty and new insights.", "limitations": "N/A", "rating": 4, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "Outlier detection with VLM is an interesting research direction.", "weaknesses": "The contribution over LoCoOp is incremental. The only difference is an extra reweighting term based on the current prediction score. And the reweighting mechanism is purely based on heuristics - for example, $1-p(y|x)$ for $L_{ce}$ implicitly enforce hard sample mining.\n\nMinor: \nThe intuition in Figure 1/4 is not clear to me. The shown examples validate that LoCoOp can detect and mask-out the inlier-related regions well. Also, the GT label should be annotated.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time devoted to reviewing this paper and your constructive suggestions. Here are our detailed replies to your questions.\n> **W1,Q1:** The contribution over LoCoOp is incremental. The only difference is an extra reweighting term based on the current prediction score. And the reweighting mechanism is purely based on heuristics - for example, $1-p(y|x)$ for $L_{ce}$ implicitly enforce hard sample mining. Please clarify the novelty and new insights.\n\nThanks for the valuable comments! We would like to **re-clarify the novelty and insights of our SCT** as follows.\n\nConceptually, the motivation of SCT is to **mitigate the problem of unreliable OOD features** in prompt-tuning based OOD detection methods. Generally, these methods rely on the ID-irrelevant local context extracted by VLMs as the surrogate OOD features to perform regularization, **the quality of which is greatly affected by the inaccurate foreground-background decomposition of VLMs**. As shown in Figure 1/4 in the submission, although VLMs can mask out some ID-related regions (shown as the grey patches of images), **large portions of the extracted OOD features (shown as the colored patches of images) obviously belong to ID features**.\n\nEmpirically, we find that the quality of extracted OOD features is significantly correlated with the uncertainty level of ID data. **As illustrated in the left panel of Figure 2** in the submission, the extracted OOD features become more inaccurate as the uncertainty increases. In the right panel of Figure 2, we train LoCoOp on multiple data groups with different uncertainty levels, and **the results demonstrate that the OOD detection performance of LoCoOp can be significantly impacted by the uncertainty level of ID data.** Therefore, to mitigate the issue of unreliable OOD features, we propose SCT to calibrate the influence of OOD regularization from different ID samples based on their uncertainty level.\n\nTechnically, despite the simple design, SCT is significantly different from hard sample mining. The latter **conducts reweighting directly on the samples** based on the classification difficulty during training. The former **adaptively adjusts the importance between the two components of the original learning objectives** for every single sample. Data with high uncertainty are directly down-weighted in hard sample mining while they are utilized more for OOD regularization in SCT. **As shown in Table 4 in the submission**, under 16-shot ID data, the OOD detection performance of simply assigning $1-p(y|x)$ to $L_{ce}$ (denoted as $\\phi$ $\\checkmark$ and $\\psi$ $\\times$) are significantly inferior to SCT (denoted as $\\phi$ $\\checkmark$ and $\\psi$ $\\checkmark$), demonstrating the difference of SCT and hard sample mining.\n\n| $\\phi$ | $\\psi$ | FPR95 | AUROC | ID-ACC |\n| --- | --- | --- | --- | --- |\n| $\\times$ | $\\times$ | 29.47 | 93.10 | 71.43 |\n| $\\checkmark$  | $\\times$ | 29.30 | 92.66 | 71.50 |\n| $\\times$ | $\\checkmark$  | 28.94 | 92.62 | 71.90 |\n| $\\checkmark$ | $\\checkmark$  | 26.47 | 93.37 | 71.77 |\n\n\n> **W2:** The intuition in Figure 1/4 is not clear to me. The shown examples validate that LoCoOp can detect and mask-out the inlier-related regions well. Also, the GT label should be annotated.\n> \n\nThanks for the suggestions! As shown in Figure 1/4, **although VLMs can mask out some ID-related regions (shown as the gray patches of images), large portions of the extracted OOD features (shown as the colored patches of images) obviously belong to ID features.** We will make the captions and figures clearer as suggested in our revised version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mEPca8mQxy", "reviewer": "Reviewer_34fm", "summary": "In response to challenges in OOD detection using CLIP-based methods, this paper introduces Self-Calibrated Tuning (SCT), a novel framework that addresses issues with unreliable OOD features extracted from ID data. SCT dynamically adjusts the influence of OOD regularization during model training based on the prediction uncertainty of ID samples. By introducing modulating factors into the learning objective, SCT directs the model's attention more effectively towards classification tasks, especially when training with low-confidence data. This adaptive approach improves the calibration of OOD features extracted from high-confidence ID data, enhancing the overall OOD detection performance of prompt tuning methods. Empirical evaluations on ImageNet-1k demonstrate SCT's effectiveness.", "questions": "1. My concern is mainly about the computational cost and training cost of SCT, since it involves operations on dense/local features. \n2. My second concern is about the rationality of using pre-trained models (CLIPs .etc) to perform OOD detection tasks, because the concepts in both ID and OOD datasets are probably seen during pre-training stage. I want to know the authors' opinions towards the benchmarking and research paradigm.", "limitations": "Please refer to weaknesses.", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. This paper is well-motivated and well-written. In particular, authors propose to adaptively adjust the importance of OOD features and introduce SCT, which are motivated by the following finding: performance of prompt tuning based methods is significantly affected by the uncertainty of the given ID data. \n2. Authors have a comprehensive review of the whole research literature.\n3. Authors conduct a large amount of experiments and the experimental results demonstrate the effectiveness of SCT on both official benchmarks and hard OOD detection tasks. \n4. In summary, I think SCT could become a great contribution towards OOD detection community.", "weaknesses": "None in particular", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time devoted to reviewing this paper and your constructive suggestions. Here are our detailed replies to your questions.\n> **Q1:** My concern is mainly about the computational cost and training cost of SCT, since it involves operations on dense/local features.\n> \n\nThanks for your valuable question!\n\nFirst, SCT **doesn't incur any extra computational cost to the LoCoOp due to its simple design.** Technically, SCT introduces modulating factors respectively on the two components of the original learning objective. The modulating factors (instantiated as $1-p(y|x)$ for $\\phi$ and $p(y|x)$ for $\\psi$ in Equation (4) in the submission) only involves the computation of the prediction probability of ground truth classes $p(y|x)$, **which can be repeatedly used after the original forward pass of CLIP models.**\n\nSecondly, **the operation of local features involved in LoCoOp and SCT are also relatively low-cost in terms of computation.** The local are generated from the forward pass of the vision encoders of CLIP, which doesn't bring additional computational cost compared to regular training. Regarding the extraction of OOD features, we compute the similarity between local features and text features of all the ID classes and we identify regions that do not include their ground truth class in the top-K predicted classes as ID-irrelevant regions. Empirically, we evaluate the time and memory consumption of SCT compared with other baselines **in Table 1 and the results show that SCT is relatively compute-efficient.** The evaluation is conducted on a single A100 GPU with the batch size of 32.\n\n**Table 1.** Evaluation of computational cost of SCT and baselines.\n\n| Method | Time per iteration (s) | GPU Memory (MiB) | FPR95 | AUROC | ID-ACC |\n| --- | --- | --- | --- | --- | --- |\n| CoOp | 0.70 | 21140 | 35.09 | 91.99 | 71.93 |\n| LoCoOp | 0.96 | 23036 | 29.47 | 93.10 | 71.43 |\n| SCT | 0.96 | 23036 | 26.47 | 93.37 | 71.77 |\n\n\n> **Q2:** My second concern is about the rationality of using pre-trained models (CLIPs .etc) to perform OOD detection tasks, because the concepts in both ID and OOD datasets are probably seen during pre-training stage. I want to know the authors' opinions towards the benchmarking and research paradigm.\n> \n\n**A2:**\n\nThanks for your valuable comments! The definition of VLM-based OOD detection differs significantly from that of conventional OOD detection. VLM-based OOD detection aims to **detect samples that do not belong to any ID class text designated by the downstream task [1].** Therefore, the current benchmarks, such as the large-scale ImageNet-1k benchmark, still apply to VLM-based OOD detection as long as they satisfy the rule that ID and OOD concepts don't overlap. **For the better development of this field, future works should be focused on building benchmarks based on realistic datasets and scenarios.**\n\nReferences:\n[1] Miyai, Yang, et al. \"Generalized Out-of-Distribution Detection and Beyond in Vision Language Model Era: A Survey.\" 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bYMyB3lkDg", "reviewer": "Reviewer_fpY7", "summary": "Based on the observation that CLIP undercalibration will affect the existing prompt-tuning-based method's OOD regularization, i.e. samples with uncertain True-Class Probability (referred to as ID uncertainty in this paper) may provide false OOD features and harm to negative training used in the existing methods; therefore, the author proposes a simple training strategy Self-Calibrated Tuning (SCT) weighted with the ID uncertainty, which can help improve FPR95 through experimental verification.", "questions": "Please see the weaknesses.", "limitations": "Limitations of lack of theoretical analysis (L571-574)'d better be put in the manuscript.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The author also observed and attempted to study the important CLIP calibration problem.\n\n- The paper is relatively easy to understand overall.", "weaknesses": "My main concerns about this work are that the work is relatively incremental and empirical; there is insufficient discussion on the pros and cons of field-related methods (including other paradigms); the experiments are not sufficient, rigorous, and analyzed; and the method's effect on improving common benchmarks is rather one-sided, etc. The details are as follows:\n\n[Method]\n\n- (Inaccurate motivation verifications) If I understand correctly, the misclassified ratio on the horizontal axis in the right panel of Fig. 2 refers to $p(pred\\neq gt), pred=\\arg\\max_yp(y|x)$ while your ID uncertainty refers to sth like True-Class Probability, marginal softmax $p(y=gt|x)$. These are not two identical things, though there is a certain correlation: only within some ranges, TCP could indicate accuracy [1]. Therefore, I think the author may have a biased understanding here, and the experimental results cannot fully reflect the motivation of the work: when \"uncertain\" ID samples are used as OOD regularization, some ID data are misdetected as OOD (FPR). If I have misunderstood, could the author clarify? Or maybe additional correct experiments are needed?\n\n- (No calibration verifications) The claim and results in the work show that SCT helps with CLIP calibration, but there is no visualization of the calibration after training to illustrate the point (e.g. could add the before and after calibration comparison like in Fig. 2).\n\n- (Lack of discussion of weighted training; modulating function rationale) The idea of ​​weighted loss is very direct and easy to think of. Previous work should also be mentioned and discussed. For example, [2] is based on the feature representation paradigm and uses activation scale as the ID-ness (similar to ID uncertainty in the context) indicator for weighted training to improve OOD detection. In comparison, I do not quite understand why $\\phi$ must be monotonically decreasing w.r.t. $p$ (e.g. $1-p$) and not monotonically increasing (e.g. $p$), because the weighting method in [2] is monotonically increasing, and the result is also improved. Could the authors elaborate on this?\n\n- (How about post-hoc CLIP calibration?) Usually, calibration is divided into two types: training and post-hoc [3] (calibration related works are lacked in the paper). The former is used in this paper. The latter may be explored in OOD feature extraction methods, e.g. changing the rank operation (Eq. (6) & Fig. 3(d)). The author may lack discussion in this aspect.\n\n[Experiments]\n\n- (No much AUROC improvement) I understand that the method in this work is mainly to improve FPR95, but unilaterally only improving FPR95 does not seem to be comprehensive enough, because AUROC is also an equally important indicator and methods need to be proposed to improve it.\n\n- (Lack of CIFAR results) Although the comparison method LoCoOp has not been experimented on CIFAR, CIFAR is indeed another important benchmarks in the field of OOD detection, and I think it is necessary to supplement it.\n\n- (Discussions with simpler yet more effective pre-trained features + post-hoc?) I just would like to know what the author thinks about the (potential) advantages of the prompt-tuning-based method studied in the paper compared to the post-hoc method. After all, post-hoc does not require additional training and uses the basic ResNet backbone; the FPR95 and AUROC on the main task of Tab. 1 on ImageNet-1k have reached **20.05** and **95.71** respectively, which are much better than the results reported in the paper (26.47, 93.37).\n\n- Could the authors clarify on what validation set are the hyperparameters tuned?\n\n- (Interpretations of the ablations.) Figure 3(b) shows that the results of selecting other regularization functions are very different, and the paper (L294-298) does not provide any analysis. I am curious about how the author could try to interprete these ablation study experiment results. Similarly, the quality of OOD features extracted by different extraction methods also varies greatly, which seems very empirical (Fig. 3(d)).\n\n- Table 1 is suggested to include results of combining more newer post-hoc methods (e.g. ASH (Djurisic et al., 2022), Scale [2]) and fine-tuned methods, which will give readers a more comprehensive sense.\n\n[Presentation]\n\n- The paragraph introducing the OOD features (L189) should be moved forward, or at least before refer to Fig. 1, which will give readers a clearer background.\n\n- Why is the left panel in Fig. 2 not arranged in ascending order of softmax output? The arrangement of 0.02, 0.89, 0.04, and 0.67 affects reading. What is it trying to say? It would be better to display the classes and images together for clarity.\n\nReferences:\n\n[1] Corbière, Charles, et al. \"Addressing failure prediction by learning model confidence.\" NeurIPS, 2019.\n\n[2] Xu, Kai, et al. \"Scaling for Training-Time and Post-hoc Out-of-distribution Detection Enhancement.\" ICLR, 2024.\n\n[3] Guo, Chuan, et al. \"On calibration of modern neural networks.\" ICML, 2017.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time devoted to reviewing this paper and your constructive suggestions. Here are our detailed replies to your questions.\n> **W1:** Inaccurate motivation verifications\n\nThanks for the constructive comments! Although true-class probability (TCP) and accuracy are not two identical things, **classification accuracy could indicate TCP to a certain degree under the large-scale ImageNet-1k dataset.** Since ImageNet-1k has a large number of classes and CLIP models are prompt-tuned with one-hot labels, the prediction probability of CLIP can be overconfident, **which means that CLIP models normally output high maximum softmax probability $max_y p(y|x)$.** Therefore, when models correctly classify samples, TCP are likely to be very high. Nevertheless, we would acknowledge it has some non-accurate parts.\n\n**To make our motivation verification clearer and more accurate**, we conduct a new experiment on the correlation between data uncertainty and OOD detection performance. Specifically, we calculate the TCP of all the training samples in a 64-shot set using a CLIP model prompt-tuned with a 4-shot training set which contains no overlapping samples with the 64-shot set. **We choose the data with the lowest and highest TCP for every ID class to generate two data groups with different uncertainty levels respectively,** and train LoCoOp on these two data groups. **As shown in Table 1 in the attached PDF**, the OOD detection performance of LoCoOp is significantly impacted by the data uncertainty level, which is consistent with Figure 2 in the submission.\n\n> **W2:** No calibration verifications\n\nThanks for the valuable suggestions! We add the figures of calibration comparison **as Figure 1 in the attached PDF.** The illustrations show that our SCT can significantly help with CLIP calibration for more accurate OOD feature extraction. The extracted OOD features (shown as grey patches of images) of SCT models are significantly more accurate and contain less ID-relevant regions than LoCoOp models.\n\n> **W3,W4** \n\nWe leave the answers in General Response.\n\n> **W5:** No much AUROC improvement\n\nThanks for the comments! As shown in Table 1 in the submission, **the space for improvement of AUROC** of the prompt-tuning based method **is close to saturation**, both exceeding 90. However, the space for improvement of FPR95 is still very large, so these two metrics should not be treated equally. We will continue to promote the improvement of AUROC in future work!\n\n>**W6:** (Lack of CIFAR results) \n\nThanks for the valuable suggestions! We conduct experiments on the CIFAR benchmark **in Table 4 of the attached PDF** and the results show that our SCT can still outperform the baselines under CIFAR benchmark.\n\n> **W7:** Discussions with pre-trained features + post-hoc\n\nThanks for the comment! First, **prompt-tuning based method can leverage the generalization ability** of VLMs to better fit the domains of the downstream tasks with relatively low computational cost. Secondly, post-hoc methods need to be built on a well-trained model, the capacity of which greatly affects the OOD detection performance. Thirdly, **post-hoc methods and prompt-tuning based methods are compatible with each other**, further boosting the OOD detection performance. We conduct experiments on the compatibility of An advanced post-doc method, NegLabel [1], with SCT **in Table 5 in the attached PDF**. The results show that SCT can be combined with post-doc methods for better OOD detection.\n\n> **W8:** Validation set\n\nThanks for the question! We tune the hyperparameter $\\lambda$ on the dedicated OOD validation set in the OpenOOD v1.5 benchmark[2]. The other hyperparameters are chosen following previous works.\n\n> **W9:** About ablations. \n\nThanks for the constructive question! \nIn the ablation study of Figure 3(b), we follow OE [3] and Energy-OE [4] to implement prompt-tuning based OOD detection with MSP and Energy regularization functions, respectively. **The motivation of conducting this experiment is to verify that SCT can outperform LoCoOp in different regularization functions.** Specifically, the energy regularization **needs tuning the two energy threshold hyperparameters $m_{in}$ and $m_{out}$, limiting its advantages over other regularization.** As shown in Fig. 3(b), we conjecture that directly forcing the probability distribution of OOD features to the uniform distribution(MSP) performs worse than entropy maximization under the setting of prompt tuning.\n\nRegarding OOD feature extraction methods, the probability-based and entropy-based methods both have a threshold hyperparameter to discriminate between ID and OOD features. The sensitivity to hyperparameters of different methods may be the reason behind the different OOD detection performance. **For the entropy-based method, the performance is poor since it is challenging to determine the appropriate threshold.** [5]\n\n>**W10:** About Table 1\n\nThanks for the suggestion! We will conduct new experiments of newer post-hoc and fine-tuned methods on CLIP models in our revision. **Since ASH and Scale can't apply to CLIP as analyzed in W3**, we will include the results of them on conventional CNN models in the appendix for fair comparison.\n\n>**W11, W12, W13:** About presentation and limitations.\n\nThanks for the suggestion! The original images arrangement is designed to compare the neighboring images with different uncertainty levels. we will make the modifications in our manuscript.\n\nReference: \n[1] Jiang, Liu, et al. \"Negative Label Guided OOD Detection with Pretrained Vision-Language Models.\" ICLR, 2024.\n[2] Jingyang Zhang,Jingkang Yang, et al \"Openood v1.5: Enhanced benchmark for out-of-distribution detection.\"\n[3] Dan Hendrycks, et al, \"Deep Anomaly Detection with Outlier Exposure\", ICLR 2019\n[4] Weitang Liu, et al, \"Energy-based Out-of-distribution Detection\", NeurIPS 2020\n[5] Miyai1, Yu, et al. \"LoCoOp: Few-Shot Out-of-Distribution Detection via Prompt Learning.\" NeurIPS, 2023", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CM6tJmD7CM", "reviewer": "Reviewer_pUkR", "summary": "This paper presents a novel few-shot approach to regularizing prompt tuning-based OOD detection methods called Self-Calibrated Tuning (SCT). SCT is specifically built to address the problems of incorrect OOD features being used in prompt tuning-based OOD detection methods. More specifically, by weighting regions of the image based on model confidence, SCT can better alleviate these issues in prompt tuning-based OOD detection methods. The resulting SCT method shows strong empirical improvements across a wide range of OOD detection methods.", "questions": "The reviewer would like to see some additional evaluations of SCT in the traditional CIFAR setting of OOD detection. The reviewer would also like to point out some small inconsistencies in the bolding for Table 2 (IDLike+SCT).", "limitations": "The author provides adequate discussions on any limitations and broader impacts. Additionally, the reviewer does not forsee any potential negative social impacts from the work.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "Strengths:\n- The paper is well written and the authors provide a clear and concise motivation justifying the use of SCT.\n- The author provides a timely analysis of the problem of incorrect OOD features extracted from ID data.\n- SCT shows strong empirical performance across a wide range of traditional OOD detection methods and prompt tuning-based OOD detection methods.\n- Additionally, given the nature of prompt tuning-based OOD detection methods, SCT can act in the more relevant few-shot setting.", "weaknesses": "Weakness:\n- A primary concern of the reviewer is the lack of evaluations against the more traditional CIFAR set of benchmarks for OOD detection.\n- Additionally, the empirical performance gain of SCT (table 2) in combination with other prompt-tuning-based methods, seems minimal.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time devoted to reviewing this paper and your constructive suggestions. Here are our detailed replies to your questions.\n\n>**W1:** A primary concern of the reviewer is the lack of evaluations against the more traditional CIFAR set of benchmarks for OOD detection.\n> \n\nThanks for the valuable suggestions! We conduct experiments on the **CIFAR benchmark in Table 4 in the attached PDF** and the results show that our SCT can still outperform the baselines under the CIFAR benchmark.\n\n\n>**W2:** Additionally, the empirical performance gain of SCT (table 2) in combination with other prompt-tuning-based methods, seems minimal.\n>\n\nThanks for the comments! As shown in Table 1 in the submission, the space for improvement of AUROC of the prompt-tuning based method is close to saturation, both exceeding 90%. However, the space for improvement of FPR95 is still very large, so these two metrics should not be treated equally. **The improvement of SCT on FPR95 (e.g. +5.95% for IDLike and +2.73% for LSN under 16-shot setting) is significant**. Nevertheless, we will continue to promote the improvement of AUROC in future work!\n\n>**Q1:** The reviewer would also like to point out some small inconsistencies in the bolding for Table 2 (IDLike+SCT).\n> \nThank you for pointing out this issue! We are sorry for the carelessness and the original bolded data should be 92.44 instead of 91.44. We will make the modification in the revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vsures9stI", "reviewer": "Reviewer_pBeU", "summary": "This paper first reveals the relationship between the quality of out-of-distribution (OOD) features and the prediction uncertainty of in-distribution (ID) data. Then, the paper introduces modulating factors to weight the ID loss and OOD loss, with the weights being related to the ID data prediction confidence.  The experiments are carried out on standard datasets.", "questions": "See weaknesses", "limitations": "NA", "rating": 3, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The analysis of the relationship between OOD feature quality and ID prediction confidence is well-reasoned; lower ID confidence indeed affects the accuracy of foreground-background separation.\n2. Weighting the loss components is a straightforward approach, making it easier to understand.\n3. The writing is clear and easy to comprehend.", "weaknesses": "1. Overall, the technical contribution of this paper is relatively incremental, primarily focusing on how to weight the two loss components.\n2. The effectiveness of the proposed method is quite limited. For example, as shown in Table 2, the improvement in averaged AUROC under the 16-shot scenario is minimal, only around 0.3%, and there is even a slight decrease in results on ID-like data.\n3. There is a lack of comparison with existing state-of-the-art (SOTA) methods. For instance, the results reported in this paper are not as good as those of NegLabel[1] (AUROC 94.21 > 93.37), which is a zero-shot method that does not require training and training samples. The results for ID-like method reported in this paper are also lower; the official paper reports 94.36 AUROC under 4-shot, while this paper reports 92.14 AUROC under 16-shot.\n4. More exploration is needed regarding the settings of function $\\phi$ and $\\psi$ in Equation 4.\n5. The statement in lines 158-159 is somewhat unclear. Should it be that inaccurate OOD features hinder the effective learning of better OOD detection?\n\n[1] Jiang, Xue, et al. \"Negative label guided ood detection with pretrained vision-language models.\" ICLR (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time devoted to reviewing this paper and your constructive suggestions. Here are our detailed replies to your questions.\n\n>**Q1:** Overall, the technical contribution of this paper is relatively incremental, primarily focusing on how to weight the two loss components.\n> \n\nThanks for the valuable comments! We would like to re-clarify the **novelty and insights of our SCT** as follows.\n\nConceptually, **the motivation of SCT is to mitigate the problem of spurious OOD features in prompt-tuning based OOD detection methods.** Generally, these methods rely on the ID-irrelevant local context extracted by VLMs as the surrogate OOD features to perform regularization, **the quality of which is greatly affected by the foreground-background decomposition of VLMs.** As shown in Figure 1/4 in the submission, although VLMs can mask out some ID-related regions, **large portions of the extracted OOD features(shown as the colored patches of images) obviously belong to ID features.**\n\nEmpirically, **we find that the quality of extracted OOD features is significantly correlated with the uncertainty level of ID data.** As illustrated in the left panel of Figure 2 in the submission, the extracted OOD features become more inaccurate as the uncertainty increases. In the right panel of Figure2, we train LoCoOp on multiple data groups with different uncertainty levels and the results demonstrate that **the OOD detection performance of LoCoOp can be significantly impacted by the uncertainty level of ID data.** Therefore, to mitigate the issue of unreliable OOD features, we propose SCT to calibrate the influence of OOD regularization from different ID samples based on their uncertainty level.\n\n>**Q2:** The effectiveness of the proposed method is quite limited. For example, as shown in Table 2, the improvement in averaged AUROC under the 16-shot scenario is minimal, only around 0.3%, and there is even a slight decrease in results on ID-like data.\n> \n\nThanks for the comments! As shown in Table 1 in the submission, **the space for improvement of AUROC of the prompt-tuning based method is close to saturation,** both exceeding 90%. However, the space for improvement of FPR95 is still very large, so these two metrics should not be treated equally. **The improvement of SCT on FPR95 (e.g. +5.95% for IDLike and +2.73% for LSN under 16-shot setting) is significant**. Nevertheless, we will continue to promote the improvement of AUROC in future work!\n\n\n>**Q3:** There is a lack of comparison with existing state-of-the-art (SOTA) methods. For instance, the results reported in this paper are not as good as those of NegLabel[1] (AUROC 94.21 > 93.37), which is a zero-shot method that does not require training and training samples. The results for ID-like method reported in this paper are also lower; the official paper reports 94.36 AUROC under 4-shot, while this paper reports 92.14 AUROC under 16-shot.\n> \n\nThanks for the comment! **Zero-shot methods and prompt-tuning based methods are compatible with each other, further boosting the OOD detection performance.** We conduct experiments on the compatibility of NegLabel with SCT **in Table 5 in the attached PDF** and the results show that SCT can be combined with advanced Neg-Label for better OOD detection.\n\nRegarding the results for ID-like method, we strictly follow the official source code [1] and the hyperparameter settings of the official paper on a single A100 GPU. We will add this to our revised version.\n\n>**Q4:** More exploration is needed regarding the settings of function $\\phi$ and $\\psi$ in Equation 4.\n> \n\nThanks for the suggestions! We conduct experiments on more instantiations of modulation functions **in the following table**. The results demonstrate that All the instantiations show significant improvement over LoCoOp, which verifies the effectiveness of the learning framework of SCT.\n\n| method   | $\\phi$                           | $\\psi$                         | FPR95 | AUROC | ID-ACC |\n|----------|----------------------------------|--------------------------------|-------|-------|--------|\n| LoCoOp   | 1                                | 1                              | 29.47 | 93.10 | 71.43  |\n| power-2  | $(1-p(y\\|x))^2$                   | $p(y\\|x)^2$                     | 27.41 | 93.14 | 71.42  |\n| power-4  | $(1-p(y\\|x))^4$                   | $p(y\\|x)^4$                     | 27.10 | 93.21 | 71.49  |\n| log      | $ 1-\\frac{log(p(y\\|x)+1)}{log2} $ | $ \\frac{log(p(y\\|x)+1)}{log2} $ | 27.06 | 93.20 | 71.39  |\n| triangle | $cos(\\frac{\\pi}{2}p(y\\|x))$       | $sin(\\frac{\\pi}{2}p(y\\|x))$     | 27.34 | 93.16 | 71.53  |\n\n\n> **Q5:** The statement in lines 158-159 is somewhat unclear. Should it be that inaccurate OOD features hinder the effective learning of better OOD detection?\n> \n\nThanks for the comments! What we mean by the original statement is that we aim to design a mechanism to mitigate the issue of unreliable extracted OOD features. We will make this statement more clear in our revision.\n\nReference: [1] Bai, Han, et al. \"ID-like Prompt Learning for Few-Shot Out-of-Distribution Detection.\" CVPR, 2024", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
# General Response
We appreciate all the reviewers for their thoughtful comments and suggestions on our paper.

We are very glad to see that the reviewers find our focused problem is **important** (R1,R2,R3,R4) within the OOD detection research, and **simple but adaptable** (R1,R2,R4) to various other techniques, and the experiments are **good**, **comprehensive** and demonstrate the **general effectiveness** of our SCT (R2,R4). We are also pleased that the reviewers find our writing is **very clear** and **easy to understand** (R2,R3,R4,R5).

We have tried our best to address the reviewers' comments and concerns in **individual responses to each reviewer** with comprehensive experimental justification. The reviews allowed us to improve our draft and **the contents added** in the revised version and **the attached PDF** are summarized below:

**From Reviewer wD2g**

- Clarify the novelty and insights of SCT(see Figure 1 and 2 in original draft)
- Explain and compare the difference between SCT and hard sample mining (see Table 4 in the original draft)

**From Reviewer 34fm**

- Conduct evaluation on the computational cost of SCT
- Discuss the rationality of utilization of pre-trained models for OOD detection.

**From Reviewer fpY7**

- Conduct experiments for more accurate motivation verifications. (see Table 1 in PDF)
- Supplement illustrations of calibration gains of SCT. (see Figure 1 in PDF)
- Discuss and compare the difference between SCT and other weighted training methods. (see Table 2 in PDF)
- Conduct experiments on post-doc CLIP calibration (see Table 3 in PDF)
- Explain the performance gain of SCT on AUROC.
- Conduct experiments on the CIFAR benchmarks. (see Table 4 in PDF)
- Conduct experiments on the compatibility of SCT and advanced zero-shot method. (see Table 5 in PDF)
- Provide more analysis on the ablation study results. (see Figure 4 in the original submission)

**From Reviewer pUkR**

- Conduct experiments on the CIFAR benchmarks. (see Table 4 in PDF)
- Explain the performance gain of SCT in combination with other baselines.
- Correct the inconsistencies of experiment data in the original submission.

**From Reviewer pBeU**

- Clarify the novelty and insights of SCT (see Figure 1 and 2 in original draft)
- Explain the performance gain of SCT in combination with other baselines.
- Conduct experiments on the compatibility of SCT and advanced zero-shot method. (see Table 5 in PDF)
- Conduct more explorations on the function $\phi$ and $\psi$.
- Clarify the unclarity of some statements in the original submission.

**We appreciate your comments and time!** We have tried our best to address your concerns and revised the paper following the suggestions. **Would you mind checking it and confirming if you have further questions?**

----

# Some rest answers:
## For reviewer fpY7:
> **W3:** Lack of discussion of weighted training; modulating function rationale 

Thank you for recommending this work! We will clarify the difference between SCT and ISH as follows.

Conceptually, **the activation scale factor**, denoted as $Q/Q_p$ in the official paper, is derived as the quotient of the sum of all activations and the sum of un-pruned activations, which **has no direct mathematical correlation with true-class probability of VLMs.** Specifically, VLMs like CLIP compute prediction probability based on the cosine similarity between image and text features. **The computation of cosine similarity involves the normalization of features,** which naturally eliminates the effect of activation scale. **Furthermore, we compute the Pearson correlation coefficient** of the activation scale factors and $p(y|x)$ of ImageNet-1k validation set utilizing CLIP, and the result is -0.05 with the p-value equal to 0.0002. showing that these two variables **have no significant linear correlation**. 

Technically, **ISH performs reweighting directly on the samples** based on their activation scale factor. Whereas, **SCT adaptively adjusts the importance between the two components** of the original learning objectives for every single sample. Data with high uncertainty are not directly down-weighted but are utilized more for OOD regularization in SCT.

Empirically, we conduct an experiment to make $\phi$ monotonically increasing with respect to $p(y|x)$ **in Table 2 in the attached PDF** and results show that the performance of monotonically increasing $\phi$ is much worse than monotonically decreasing $\phi$.

> **W4:** Post-hoc CLIP calibration

Thanks for the valuable suggestions! We conduct experiments on the effect of different $K$ in the rank operation **in Table 3 in the attached PDF**. The results demonstrate that post-hoc CLIP calibration shows less effective performance than training-based methods. We will include more calibration related works in our revision.

Meta-review: **Summary:** This paper focused on OOD detection using CLIP. Specifically, it aims to improve the prompting-based method with OOD regularization. It found that the potentially poor quality of surrogate OOD features may hinder the performance, and reveals the relationship between the quality of OOD features and the prediction uncertainty of ID data. Then, the paper introduces Self-Calibrated Tuning (SCT), using modulating factors to weigh the ID loss and OOD loss, with the weights being related to the ID data prediction confidence. The resulting SCT method shows strong empirical improvements.

**Strength:** Interesting and important research direction; well-motivated and well-written; the finding/analysis was interesting and well-reasoned; comprehensive review; extensive experiments; effective and easy-to-understand algorithm. One reviewer thinks SCT could become a great contribution to the OOD detection community.

**Weakness:** Relatively incremental and empirical; inaccurate motivation verifications; lack of details/explanations on algorithm design; lack discussion/comparison to post-hoc methods; the experiments are not sufficient (lack comparisons to SOTA); missing results on CIFAR; the improvement is not clear on AUROC. 

**After rebuttal:** The authors provided the rebuttal, including further explanations and experiments attempting to address all the reviewers’ concerns. Three reviewers (with lower ratings) provided the feedback. They acknowledged that a portion of their concerns are addressed. However, they still have concerns about the performance gain on AUROC and the incremental technical contributions. One reviewer is still concerned about the weight design. **After rebuttal, the paper received 3-4-4-5-7 (average 4.6), with one reviewer (rating 4) acknowledging the nonfamiliarity with the topic.**

**Recommendation:** Given the mixed rating, the AC read the paper, the review, the rebuttal, and the further discussion. The AC agrees with the multiple strengths mentioned by the reviewers. Regarding the major two unresolved concerns 1) performance gain on AUROC, and 2) incremental technical contributions, the AC has the following opinions. First, the AC agrees with the authors’ arguments that FPR95 is the more challenging metric. Although the overall gains across metrics may not be outstanding, it is nice to see that such a simple STC method leads to *consistent* and *compatible gains. While the final STC method is simple, seemingly a simple add-on to LoCoOp, its generality and compatibility could be impactful to the community. Such a simple change to the objective function made the AC think of the focal loss, which, while simple, has become a go-to approach to resolve the imbalance. The AC also thinks the design of the weight is reasonable. The AC also appreciates the reviewers’ feedback, especially regarding other branches of OOD detections (e.g., feature-based, post-hoc). In the AC’s humble opinion, a paper does not need to beat all the existing methods and claim the state of the art if it offers sufficient novel insights. When an area is under development, the AC thinks allowing different branches to thrive should be encouraged. The AC sees that the paper makes some valuable fundamental observations/insights and proposes an easily extensible solution, which the following-up work can build upon. Regarding all these aspects, the AC recommends **acceptance.**

That said, the AC finds that the reviewers' comments and discussions are highly valuable and asks the authors to carefully incorporate them into the final version.
Decision: Accept (poster)