Understanding and Mitigating Miscalibration in Prompt Tuning for Vision-Language Models
Authors: Shuoyuan Wang
Yixuan Li
Hongxin Wei
Venue: ICML 2025 poster (2025)
Keywords: Vision-Language Models
CLIP
Confidence Calibration
Fine-tuning
Primary area: social_aspects->robustness
TL;DR: We propose Dynamic Outlier Regularization to improve fine-tuned CLIP’s calibration under various evaluations without compromising vanilla fine-tuning.

Abstract: Confidence calibration is critical for the safe deployment of machine learning models in the real world. However, such issue in vision-language models like CLIP, particularly after fine-tuning, has not been fully addressed. In this work, we demonstrate that existing prompt tuning methods usually lead to a trade-off of calibration between base and new classes: the cross-entropy loss used in standard fine-tuning (e.g., CoOp) causes overconfidence in new classes by increasing textual label divergence, whereas regularization-based tuning (e.g., KgCoOp) maintains the confidence level but results in underconfidence in base classes due to the improved accuracy. Inspired by the observations, we introduce Dynamic Outlier Regularization (DOR) to ensure the confidence calibration on both base and new classes after fine-tuning. In particular, we propose to minimize the feature deviation of novel textual labels (instead of base classes) sampled from a large vocabulary. In effect, DOR prevents the increase in textual divergence for new labels while easing restrictions on base classes. Extensive experiments demonstrate that DOR can enhance the calibration performance of current fine-tuning methods on base and new classes.

Understanding and Mitigating Miscalibration in Prompt Tuning for Vision-Language Models
Shuoyuan Wang'
Yixuan Li Hongxin Wei
Abstract
tation (Liang al., 2023) and knowledge-angmented re- trieval (Ming & Li, 2024). To efficiently adapt CLIP to domain-specific downstream tasks, various parameter- efficient fine-tuning (PEFT) techniques, such as prompt tuning (Zhou et al., 2022b) and adapter (Gao et al., 2024), have been proposed. While these methods improve ac- curacy, they largely overlook reliability concerns such as confidence calibration in fine-tuned CLIP. Without fully un- derstanding the miscalibration in fine-tuned CLIP, it can exacerbate safety concerns in high-stakes applications like medical diagnosis and autonomous driving.
Confidence calibration is critical for the safe de- ployment of machine learning models in the real world. However, such issue in vision-language models like CLIP, particularly after fine-tuning, has not been fully addressed. In this work, we demonstrate that existing prompt tuning methods usually lead trade-off of calibration between base and new classes: the cross-entropy loss used in standard prompt tuning (e.g.. CoOp) causes overconfidence in new classes by increasing tex- tual label divergence, whereas regularization- based tuning (e.g.. KgCoOp) maintains the confi- dence level but results in underconfidence in base classes due to the improved accuracy. Inspired by the observations, we introduce Dynamic Outlier Regularization (DOR) ensure the confidence calibration on both base and new classes after fine- tuning. In particular, DOR minimizes the feature deviation of novel textual labels (instead of base classes) sampled from large vocabulary set. In effect, DOR prevents the increase in textual di- vergence for new labels while easing restrictions base classes. Extensive experiments demon- strate that DOR can notably enhance the calibra- tion performance of current fine-tuning methods. Our code is available at https//github..co/mlltaaa CaatscchOutiier-Caaibattoo Calibration.
In confidence calibration, we generally expect the model's confidence level to be consistent with its empirical accu- racy (Tu et al., 2023). In the literature, zero-shot CLIP is often recognized for its excellent performance in confidence calibration (Minderer et al., 2021). Prior work (Wang et al., 2024) has shown that fine-tuning CLIP on base classes often leads to miscalibration on novel classes within the same task, where the model is expected to generalize (Zhou et al., 2022b; Yao et al., 2023). While these studies focus on miscalibration in novel classes unseen during fine-tuning, they do not adequately address calibration issues on base classes. Overall, the community still lacks a comprehensive understanding of the fundamental cause of miscalibration in fine-tuned CLIP and effective strategies for mitigation.
In this work, we first investigate how existing prompt tun- ing methods affect the confidence calibration of CLIP. In general, these methods can be fell into two categories: stan- dard fine-tuning (c.g.. CoOp (Zhou et al., 2022b)) and iggularization-basee fine-tuning (e.g., KgCoOp (Yao et al., 2023)). Our empirical analysis first reveals that both ap- proaches struggle to maintain calibration across base and novel classes simultaneously, inevitably compromising one: CoOp exhibits overconfidence on new classes while Kg- CoOp provides underconfident predictions on base classes. To thoroughly understand such phenomenon, we explain it from the perspective of textual feature divergence. In particular, CoOp increases the divergence of textual label distribution through cross-entropy loss, resulting in exces- sively high confidence misaligned with actual accuracy on new classes. Conversely, KgCoOp constrains the divergence increase, preserving confidence levels across both base and
1. Introduction
Large pre-trained vision-language models (VLMs) like CLIP (Radford et al., 2021) have become the de facto standard in today's s zero-shot tasks including image recog- nition (Wortsman et al., 2022), open-vocabulary segmen-
'Department of Statistics and Data Science, Southern Univer- sity of Science and Technology. Shenzhen, China. 'Department of Computer Sciences, University Wisconsin-Madison WI, USA. Correspondence to: Hongxin Wei <weihx www.sccc.cdu..c.........
Proceedings of the 4nnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Understanding and Mitigating Miscalibration in Prompt Tuning for Vision-Language Models
novel classes. However, this leads the underconfidence issue due to the improved accuracy base classes. This raises key question: Can we achieve reliable calibration on both base and novel classes after fine-tuning?
2. Preliminaries
Contrastive Language-Image Pretraining (CLIP) CLIP is a visual-language model that measures the aligmment between images and texts (Radford al., 2021). Recendy, CLIP has shown great potential in zero-shot inference for arbitrary classes. Let Φ æ - IR and 1b - IR denote CLIP's image and text encoders, respectively. Given an image instance a and a text label the logit function of CLIP can be formulated as:
To tackle the above challenges, we introduce Dynamic Outlier Regularization (DOR), a regularization technique incorporated into the fine-tuning phase. The core idea of DOR to leverage outliers to regulate the divergence of unseen textual distribution in the fine-tuned CLIP, without interfering with the vanilla fine-tuning objectives. Specifi- cally, we first collect textual outliers from the large lexical database (e.g., WordNet (Miller, 1995)), while ensuring that the selected words do not overlap with the base classes in the fine-tuning task. During fine-tuning, we minimize the feature discrepancy of novel textual labels between the fine-tuned model and the zero-shot CLIP. To further enhance flexibility, the textual outliers are dynamically sampled from the constructed set in each epoch. By incorporating dynamic textual outliers, DOR mitigates the increase in textual diver- gence for novel labels while reducing constraints on base classes, improving overall calibration stability.
Ie (a:)) = sim (ō(x), ((te))
(1)
Here t. is derived from a hand-crafted prompt like "a photo of a (class]", where the "[class)" is filled with the text label c. T is generally set as a pre-trained constant of 100.
For multi-class classification, we predict by selecting the la- bel with the highest probabilities among the label candidate set C = ) iimi 1. as shown below:
Celp (ar)
c' arg max p(clar) - arg max cEC
(2)
We verify the effectiveness of DOR across 11 image clas- sification datasets and 4 types of ImageNets with covariant shifts. Extensive experiments show that DOR can enhance the calibration of existing prompt-tuning methods (see Ta- ble 1). For instance, DOR reduces the Expected Calibration Error (ECE) by an average of 8.09% for CoOp across the 11 downstream datasets. Moreover, our method can maintain and even improve the generalization performance of those tuning methods (See Table 6) DOR also achieves competi- tive improvements in the presence of covariate shifts, further validating its robustness. Beyond prompt-based tuning, we show that this regularization criterion can be extended to vi- sual fine-tuning methods using image outliers, highlighting its broader applicability.
CEC
Li=1
where p(clær) is the predicted probability of class e for the instance x.
Prompt tuning To boost performance of CLIP in down- stream tasks, prompt tuning methods have been proposed to efficiently fine-tune CLIP on datasets of interest (Zhou et al., 2022a;b). particular, prompt tuning optimizes the context prompt only, without retraining the model and up- dating its weights. CoOp (Zhou et al., 2022b) replaces the hand-crafted textual tokens with a set of learnable textual token T = {0i, u2...., Uar
}. where M is the length of tokens. Thus, the output of fine-tuned CLIP is: Leoop (x) T-sim (ø(z), <b(t)),, where t' = [w1, U2 V3. c] and e denotes the textual embedding of class e. Using a few labeled samples Dn = { c the learnable textual tokens T are optimized entropy (CE) loss lee. We refer to the classes used in fine- tuning as base classes, and the remaining labels within the same task as new or novel classes.
We summarize our main contributions as follows:
1. We show that current prompt-tuning methods typically lead to a trade-off' between base and new classes, com- promising one of them. To understand such miscalibra- tion, we provide an in-depth analysis from the perspec- tive of textual distribution divergence.
minimize the standard cross-
To enhance the generalization ability of the learnable prompt for unseen classes within the task, regularization-bun based methods like KgCoOp (Yao et al., 2023) introduces a reg- ularization term align the learned prompt to the hand- crafted prompt. The optimization of KgCoOp is:
2. We propose DOR, simple yet effective regularization that ensures calibration performance on both base and new classes. Our method is compatible with existing prompt-tuning methods and can be extended to visual fine-tuning methods with image outliers.
N
T = arg min
La (c x;))
3. Extensive experiments on a wide range of real-world benchmarks demonstrate that DOR is effective and casy-to-use. DOR can be easily incorporated with various prompt tuning methods.
N
(3)
sim (1b (t') bb (lo)) )
+ A
1
Σ
C
2

---

Understanding and Mitigating Miscalibration in Prompt Tuning for Vision-Language Models
ZeroshotCLIP
CoOp
KgCoOp
ZeroshotCLIP
CoOp
KgCoOp
1.0
1.3
Outputx
3.8 Capp
0.3
0.8
0.0
0.6
0.3
03
0.5
08
0.0
0.8
0.4
0.8
02
ECE: 2.35%
BCE: 11.01%
BCE: 3.31%
ECE: 11.90%
ECE: 3.38%
"
Confidence
Conddonnc
Confisence
Confidinnos
Confidence
Confieence
(a) Base
(b) New
Figure 1. Reliability diagram of fine-tuned CLIP (VIT-B/16) on StanfordCars dataset. ECE: Expected Calibration Error (lower is better).
Miscalibration is depicted in pink for overconfidence and purple for underconfidence.
Here, the first term is the standard cross-entropy loss used in CoOp and the hyperparameter À is used control the weight of regularization. With À = 0, KgCoOp is degraded to the original CoOp.
base and new classes. The model is trained only on a few examples from the base classes and evaluated on examples from both base and new classes.
Prompt tuning leads to a trade-off between base and new classes. Figure illustrates the calibration performance of zero-shot CLIP, CoOp and KgCoOp on base and new classes. The results show that zero-shot CLIP achieves almost perfect calibration on all classes, while the fine- tuned models cannot maintain the calibration base and classes simultaneously, compromising one of them. In particular, CoOp maintains the excellent calibration on base classes but exhibits overconfidence on new classes. Instead, KgCoOp provides underconfident predictions on base classes while preserving the calibration on new classes. This motivates us to further investigate the fundamental cause of miscalibration occurring after fine-tuning.
Confidence calibration In addition to predictive perfor- mance, is generally expected for deep models to be well calibrated, i.e., the predicted class probabilities can faith- fully estimate the true probabilities of correctness (Guo et al., 2017). To quantify miscalibration, the Expected Calibration Error (ECE) (Guo et al., 2017) is defined as the difference between accuracy and confidence. With N samples grouped the ECE is formulated as:
into G bins {61,
lbg N
ECE
Jace (by) conf (by)l,
(4)
where acc (-) and conf (-) denotes the average accuracy and confidence in bin bm- the literature, has been shown that pre-trained CLIP archives excellent performance of con- fidence calibration in zero-shot inference (Minderer et al., 2021). However, prior work (Wang et al., 2024) finds that fined-tuned CLIP generally suffers from miscalibration on novel classes within the same task, where the model ex- pected to generalize (Zhou et al., 2022b; Yao et al., 2023). Yet to date, the community still has a limited understand- ing of the fundamental cause and mitigation strategies of miscalibration during fine-tume funiinu We proceed by analyzing how the fine-tuning of CLIP affects the calibration.
3.2. Understanding the miscalibration in CLIP
Given the above observation, we investigate how prompt tuning leads to the miscalibration issue. Since the visual features remain unchanged in the prompt tuning, the tex- tual features thus play a key role in confidence calibration. We first introduce feature divergence score to quantify the textual feature variation, inspired by the KNN- metrics commonly used for distribution estimation in cali- bration (Xiong et al., 2023; Yuksekgonul al., 2023).
Definition 3.1 (Feature Divergence). Consider a feature set Z {z, > each feature z, E Rd is embedded by the modality encoder in CLIP. Feature divergence (FD) score of z, measures the average distances from each feature to its M nearest neighbors in the set.
3. Motivation
3.1. Empirical study on CLIP calibration
Setup To show the miscalibration in fine-tuned CLIP, we first empirically study the calibration performance of fine- tuned VLMs. We use ViT-B-16 pre-trained by OpenAl (Rad- ford et al., 2021) as the zero-shot classification model. In particular, we compare the zero-shot CLIP with standard prompt tuning CoOp (Zhou et al., 2022b) and regularization- based method KgCoOp (Yao et al., 2023) on StanfordCars dataset (Krause et al., 2013). We evaluate the fine-tuned CLIP under base-to-new protocol: the dataset split into
8;
Σ (x)
dist(z;, Zj),
M
where Nar (z,) denotes the set of M nearest neighbors of 2, and dist (. is distance metric like cosine similar- ity. By averaging these similarity scores across all fea- tures, we obtain the overall FD score of a given feature set FD(Z)= = 1 ΣN $4, which can represent the divergence of the textual distribution. To investigate the miscalibration
3

---

Understanding and Mitigating Miscalibration in Prompt Tuning for Vision-Language Models
0.6
0.35
Max
Otbers
2SCT.IP KgCoOp CoOp
0.33
0.31
0.5
0.30
0,30
Soore
0.4
Confideace
0.8-
0.28
0.25
0.3
0.2
0.7
0.20
0.1
0.19
0.17
0.0
New
0.6
0.15
2SC1PP
Co
0.15
2SC1UP
CUOP
ReCope
(a) FD score
(b) Confidence
(a) Base
(h) New
Figure 2. Results of zero-shot and sundd CLIPs with different Figure 3. Comparison between the maximum logit and the average other logits, using different prompt tuning methods on DTD.
prompt tuning methods on UCF101 dataset.
issue, we vary the hyperparameter À in Eq. (3). In KgCoOp, is set to 8.0, and it degenerates to CoOp when À = 0. We conduct the experiments on UCF101 (Soomro et al., 2012). We present the results Figure 2.
stantially improves the accuracy of CLIP on base classes, resulting in the underconfidence issue due to the anchored confidence level. In this way, we explain why KgCoOp leads to underconfidence on base classes.
CoOp leads overconfidence on new classes by increas- ing the textual divergence. In the analysis of Subsection 3.1 and Figure 2b, we show that CLIP tuned by CoOp exhibits overconfidence on new classes, but keeps calibration on base classes. This is caused by the CE loss, which maximizes the posterior p(y z) for the ground-truth label and mini- mizes the probability for other labels. other words, CE loss tends to enlarge the distance between the image feature and all textual features except the ground truth. As the im- age feature remains unchanged during fine-tuning: it can be translated to large distances among all textual labels, includ- ing base and new classes. This is supported by the results presented Figure 2a, which shows that CoOp significantly increases the FD score of textual features compared to zero- shot CLIP, Consequently, As shown in Figure 3, the gap between the maximum logit and the others widens for both base and new classes. Hence, the tuned CLIP with CE loss will make softmax predictions with high confidence on both base and new classes. It aligns with the improved accuracy on base classes, but is not consistent with the nearly un- changed accuracy on new classes. This explains why CLIP tuned by CoOp tends to overconfident on new classes.
Through the perspective of textual divergence, we provide a thorough explanation for the calibration trade-off caused by different prompt-tuning methods. We provide the compre- hensive empirical results on more prompt tuning methods to thoroughly support our observationin Appendix B.1 (See Figure 5 and 6). In addition, we present theoretical jus- tification for the relationship between textual divergence and model confidence in Appendix C. Ideally, we expect to maintain the excellent zero-shot calibration for both base and classes after fine-tun ane-tuning. In the following, proceed by introducing our method, targeting this problem.
4. Method: Dynamic Outlier Regularization
In the previous analysis, we show that textual divergence is the key for CLIP calibration. To preserve the calibration capacity of zero-shot CLIP, our key idea is to regularize the textual divergence of new classes without restricting that of base classes. To this end, we use textual outliers to regularize fine-tuned CLIP. We provide clear evidence to show that the regularization using textual outliers can easily mitigate the overconfidence in the new classes without altering the fine-tuning objective in Appendix B.2.
KgCoOp anchors the confidence level by hindering the increase of textual divergence. In the previous analysis, KgCoOp leads to underconfident predictions on base classes while preserving the calibration on new classes. In Figure 2a, we illustrate the FD scores textual labels from the zero- shot CLIP and the tuned CLIP by KgCoOp with various A. The results show that large value of reduces the FD score of both base and new classes, approaching that of zero-shot CLIP. This phenomenon indicates that the regularization in KgCoOp can prevent the model from increasing the tex- tual divergence caused by CE loss. Correspondingly, the fine-tuned CLIP by KgCoOp preserves the same confidence level as the zero-shot CLIP. However, the fine-tumin tne-tuning sub-
Selecting textual outliers With this in mind, we construct a set of textual outliers using nouns from the large lan- guage lexical database. Specifically, we mainly use Word- Net (Miller, 1995) as database, which a large English lexical database containing over 150,000 words. We sclect nouns from WordNet that do not overlap but share higher- level concept relations with the base classes used in the fine- tuning, and then incorporate them into our regularization term for prompt tuning. We demonstrate the effectiveness of using relevant textual outliers Table
{c1, C2 .., Cn) be the base classes used in Let Cn

---

Understanding and Mitigating Miscalibration in Prompt Tuning for Vision-Language Models
the fine-tuning. First, we obtain candidate set Cword for, (01,02,.... aaa....,....... by filtering out the base classes from WordNet. Then, we rank the nouns according to the av- erage semantic similarity among the candidate Cwerd and each base class Cj E Cn. For candidate word oi we use zero-shot CLIP to quantify the semantic similarity s;²
et al., 2022b) when À = 0. As À increases, the optimization will encourage the model to maintain the confidence level new classes, alleviating the overconfidence issue. We illustrate the effect of X in Figure 4.
Extension to other robust fine-tuning algorithms It is worth noting that our regularization is a general method and can be easily incorporated into existing regularization- based tuning for CLIP, including knowledge-guided fine- tuning (Yao et al., 2023; 2024), multimodal consis- tency (Khattak et al., 2023b; Roy & Etemad, 2023), Self- Regularization (Khattak et al., 2023b) etc. Given the ex- isting robust fine-tuning objective frobust. we formalize the lie-ttuing objective as:
n
sim (tax) db (fc;));
8
where to; represents the textual tokens of noun of using a fixed prompt like "a photo of a cclass-name]", and i E (1, (1.2,.... 2......).. up is the text encoder of zero-shot CLIP. Then we get the set of textual outliers using the score ranking.
Liaill = Lrobust + x Ldor-
(10)
out fo; | i TopK (s1, $2,.... 8m))
Noticeably, DOR offers several compelling ad advantaee::
where (-p represents selecting the indices with the top K' largest scores for nouns in the candidate set Cword-
Easy-to-use: DOR leverages text outliers for regular- ization, which is readily available and easy to collect.
Dynamic Outlier Regularization Given a fine-tuning dataset Dn with base classes Ca, construct large set of textual outliers Oour described above. To prevent the increase of textual divergence on new classes, we propose Dynamic Outlier Regularization (DOR), which minimizes the feature discrepancy of textual outliers between the zero- shot CLIP and the fine-tuned CLIP.
Algorithm-agnostic: DOR can be easily incorporated into existing fine-tuning methods (See Table 1 and 2) calibration algorithms for CLIP (See Appendix I). Furthermore, our method can be extended to visual fine-tuning methods with image outliers (See Table 7),
Compared with existing regularization-be based methods, DOR does not conflict with the fine-tuning objective and breaks the calibration trade-off. (See Table and Appendix B.2).
Fine-tuning-nonte
In each iteration, we randomly sample a batch of textual outliers from the constructed set Owa We denote the batch of textual outliers as B= = {o:) h= where B is the number of textual outliers in the batch. By default, we set B as the same as the batch size of fine-tuning data. Then, we build the regularization by aligning the textual features to those of zero-shot CLIP. Formally, the regularization is defined as:
5. Experiments
5.1. Experimental Setup
Benchmark setting. Following recent works (Zhou et al., 2022b; Wang et al., 2024), we perform two evaluations in two standard benchmark settings: 1) Generalization from Base-to-New Classes: A downstream dataset will be equally split into base and new classes. The model is trained only on the base classes in a few-shot setting and evaluated on base and new classes. In practice, the harmonic mean value from both classes 2) Domain Generalization: The model is trained on ImageNet- Ik in few-shot manner and evaluated on four other ImageNet datasets that contain various types of domain shifts.
B Σ sim (1b (t%,)) 2b (tox)),
Loor
B
where sim(-) denotes the cosine similarity function and to. denotes the token of the textual outlier og- Using the reg- ularization, the textual divergence of the fine-tuned CLIP will be regularized to be consistent with the zero-shot CLIP. Different from KgCoOp (Yao al., 2023), the outlier regu- larization does not restrict the textual feature deviation of base classes, which is explicitly shown in Figure 9.
mainly focus on
Datasets. For the base-to-new evaluation, we cover diverse classification tasks including ImageNet (Deng et al., 2009). Caltech101 (Fei-Fei al., 2004), OxfordPets (Parkhi et al., 2012), StanfordCars (Krause et al., 2013), Flowers 102 (Nils- back & Zisserman, 2008), Food101 (Bossard et al., 2014), FGVCAireraft (Maji et al., 2013). SUN397 (Xiao et al., 2010), UCF101 (Soomro et al., 2012), DTD (Cimpoi et al., 2014) and EuroSAT (Helber et al., 2019). For do- main generalization, we use ImageNet-1k as the source
Equipped with dynamic outlier regularization, the final train- ing objective for fine-tuning CLIP is:
Liotal = Lee + X Litor;
where Loe and Ldor are the cross-entropy loss of fine-tuning data and the proposed regularization, respectively. À denotes the hyperparameter that controls the weight of the proposed regularization. Our method will degrade to CoOp (Zhou
5

---

Understanding and Mitigating Miscalibration in Prompt Tuning for Vision-Language Models
Table Average calibration across datasets. "+DOR(Ours)" to our method applied to standard tuning methods. & indicates smaller values are better, Calibration error is given by x10- "HM"" denotes the harmonic mean. Bold numbers are significantly superior results.
ECE(J)
ACE(L)
MCE()
PIECE(1)
Method
Basc
New
HM
Base
New
HM
Base
New
HM
Base
New
HM
ZSCLIP
3.58
4.61
4.10
3.62
4.58
4.10
0.97
1.21
1.09
6.35
6.55
6.45
CoOp
3.07
14.58
8.82
2.97
14.50
8.73
1.07
3.72
2.40
4.68
15.27
9.98
+DOR(Ours)
2.67
6.49
4.58
2.64
6.47
4.55
0.83
1.65
1.24
4,45
8.33
6.39
CoCoOp
3.60
6.14
4.87
3.53
6,08
4.81
0.96
1.72
1.34
5,53
7.86
6.70
+DOR(Ours)
4.22
4.02
4.12
4.30
3.94
4.12
1.07
1.11
1.09
6.00
6.41
6.20
MaPI.c
2.75
5.46
4.11
2.65
5.42
4.04
0.82
1.52
1.17
4.71
7.37
6.04
+DOR(Ours)
2.83
4.44
3.63
2.86
4.33
3.60
0.81
1.29
1.05
4.86
6.39
5.62
DEPT
6.04
14.58
10.31
6.00
14.52
10.26
1,44
4.58
3.01
7.31
15.42
11 11.37
+DOR(Ours)
7.67
7.50
7.58
7.66
7.44
7.55
1.73
1.87
1.80
8.68
8.86
8.77
Table Average ECE (%) regularization- based methods across 11 datasets. Bold numbers are significuntly superior results. DOR constantly improves the calibration when incorporated with existing eegulaarzzationnaseed methods.
KgCoOp
TCP
PrompiSRC
CoPrompt
PromptKD
Metric
Vanilla
+DOR
Vanilla
+DOR
Vanilla
+DOR
Vanilla
+DOR
Vanilla
+DOR
Base
5.82
6.07
4.71
4.79
3.7527
3.88
2.56
2.96
4.73
4.81
New
4.48
3.99
4.07
3.80
4.15
3.80
5,96
4.69
4.38
3.66
HM
5.15
5.03
4.39
4.29
3.95
3.84
4.26
3.83
4.56
4.24
dataset and its four variants as target datasets including Im- ageNetV2 (Recht et al., 2019), ImageNet-Sketch (Wang al., 2019), ImageNet-A (Hendrycks et al., 2021b), and ImageNet-R (Hendrycks et al., 2021a).
5.2. Results
DOR enhances the calibration existing prompt-tuning methods. Table shows the calibration performance of standard tuning baselines w/ w/o our DOR. We find that DOR can consistently reduce the calibration error new classes. For instance, DOR significantly reduces the ECE from 14.58% to 6.49% on new classes and maintains the ECE from 3.07% to 2.67% on base classes, which makes CoOp more reliable in terms of predicted confidence. More- over, we incorporate DOR with other robust tuning methods in Table 2. We observe a trade-off between the base and new class for these methods, e.g., CoOp outperforms TCP base classes but significantly underperforms TCP on new classes Notably, DOR can continually reduce mis- calibration on new classes across various metrics without calibration trade-offs on base and new classes. In sum- mary, our proposed DOR can consistently boost calibration performance on new classes upon existing state-of ffte-of-the-ae --tth-rrt prompt tuning methods without compromising the vanilla fine-tuning objectives. Due to space constraints, vide detailed calibration results of all datasets in Appendix F. Moreover, to illustrate how DOR affect the probability distribution, we provide a visualization in Appendix G.
Baselines. We compare our method with standard prompt tuning algorithms including CoOp (Zhou et al., 2022b), Co- CoOp (Zhou et al., 2022a), MaPLe (Khattak et al., 2023a) and DEPT (Zhang et al., 2024b). We also incorporate DOR tuning including KgCoOp (Yao
with regularization-basse al., 2023), TCP (Yao et al., 2024), PromptSRC (Khat- tak et al., 2023b), CoPrompt (Roy & Etemad, 2023) and PromptKD (Li et al., 2024).
Implementation details. We use CLIP ( ViT-B/16) (Rad- ford et al., 2021) as the pre-trained VLM throughout our experiments and report results averaged over 3 runs. We fine-tune the model with 16 samples per class in a few- shot setting (Zhou al., 2022a). We list the details of the compared methods in Appendix D.
Evaluation metrics. We use 4 standard metrics of confi- dence calibration in our evaluation including Expected Cali- bration Error (ECE) (Guo et al., 2017), Maximum Calibra- tion Error (MCE) (Guo et al., 2017), Adaptive Calibration Error (ACE) (Nixon et al., 2019) and moximitt-IInfrmmn Expected Calibration Error (PIECE) (Xiong et al., 2023).
pro-
DOR effectively breaks the calibration trade-off. To di- rectly demonstrate the role of DOR in the regularization during CLIP fine-tuning, compare DOR and KgCoOp
6

---

Understanding and Mitigating Miscalibration in Prompt Tuning for Vision-Language Models
Table Average calibration results of ECE (%) using various regularization on 11 datasets. DOR breaks the calibration trade-off between base and novel classes.
Table 4. Average calibration results of ECE (%) 11 datasets. DOR' denotes that the outlier pool excludes all new classes.
Method
Variant
Base
New
HM
Method
Variant
Base
New
HM
CoOp
Vanilla
3.07
14.49
8.78
CoOp
Vanilla
3.07
14.49
8.78
+KG
5.82
4.48
5.15
+DOR
2.67
6.49
4.58
+DOR
2.47
6.48
4.47
+DORT
2.82
6.77
4.80
MaPLe
Vanilla
2.75
5.46
4.11
MaPLe
Vanilla
2.75
5,46
4.11
+KG
4.01
4.29
4.15
+DOR
2.83
4.44
3.64
+DOR
3.06
4.26
3.66
+DORT
2.89
4.51
3.70
CoPrompt
Vanilla
2.60
5.96
4.28
CoPrompt
Vanilla
2.60
5.96
4.28
+KG
4.01
4.99
4.50
+DOR
2.96
4.69
3.83
+DOR
2.98
5.14
4.06
+DORT
2.71
4.93
3.82
on 3 baselines (CoOp, MaPLe, and CoPrompt), and present the results in the table 3. The results demonstrate that inte- grating with DOR can consistently outperform those with KgCoOp on overall performance. In particular, our method achieves much better performance than KgCoOp on Base classes, while KgCoOp performs well on New classes. This consistent with the analysis presented Subsection 3.2: KgCoOp anchors the confidence level, leading undercon- fidence Base classes. In short, the results demonstrate the superiority of DOR in breaking the calibration trade-off between base and novel classes.
325
KG (Jlane)
10
Accuracy (1)
77.5
DOK ((Bsne)
BCE())
22.5
65.0
0.10
0.50
1.00
2.00
5,00
16.00
0.10
1.00
4.00
16.00
Figure 4. Parameter sensitivity between KgCoOp (KG) and DOR (w/ CoOp). Compared with KG, the accuracy and ECE of DOR are not sensitive base classes. Left: Accuracy. Right: ECE.
DOR do not rely on the overlap with new classes. As we mentioned Section 4, DOR samples leverages text outliers for regularization. To further analyze the potential overlap with new classes used in the test time, we exclude all new classes from the outlier pool (denoted as DOR$). As is shown in Table 4, the average results on 11 datasets show that DOR without overlap achieves comparable per- formance to DOR, significantly improving the performance on all three baselines. Therefore, the effectiveness our method does not rely on the overlap with new classes. More- over, we present all selected outlier texts for 6 datasets, Stanford-
tion, with its accuracy consistently outperforming all exist- ing baselines in the harmonic mean of base and new classes. Moreover, DOR almost entirely preserves model capability in the classification performance on base classes without degeneration. For instance, applying DOR in DEPT can increase accuracy on new classes from 65.04% to 71.39%, while keeping the accuracy of base classes similar to the baseline. Given that most prompt tuning methods lag be- hind zero-shot CLIP on the accuracy of new classes, an intuitive explanation is that DOR aligns the features of un- seen classes with the zero-shot features, which can preserve the zero-shot generalization on new classes. In addition, we observe that some methods (e.g., CoCoOp and MaPLe) with DOR, resulting in improvements of 2.12% and 2.00% re- spectively, outperforming zero-shot accuracy on new classes. This demonstrates that DOR can significantly enhance the base-to-new generalization capacity of fine-tuned CLIP. To further demonstrate DOR can be applied to domain-specifit open-vocabulary tasks, we provide more results on medical datasets in Appendix
showing almost no class overlap, especially Cars and FGVCAireraft in Table 10. In summary, leverages general semantic information from language space rather than memorizing target classes, which ensures its faimeess and generalizability.
DOR benefits base-to-new generalization. To further ver- ify that DOR is effective on new classes and non-toxic for performance base classes, parison of average test accuracy in Table 6. Similar to the evaluation of calibration, a salient observation is that our proposed DOR drastically improves base-to-new generaliza-
summarize the com-
DOR is robust to covariate shifts. To comprehensively

---

Understanding and Mitigating Miscalibration in Prompt Tuning for Vision-Language Models
Table Accuracy comparison on domain generalization datasets. DOR boosts the calibration and generalization of existing methods.
BCE (L)
Accuracy (1)
Source
Target
Source
Target
ImageNet
-V2
-S
-A
-R
AVG
ImageNet
-V2
-S
-A
-R
AVG
CLIP
1.86
2.44
4.88
8,34
3.51
4.79
66.73
60.87
46.09
47.81
73.98
57.19
CoOp
1.10
4.19
8.40
15.34
0.80
7.18
71.44
63.55
45.76
47.81
73.74
57.72
+DOR(ours)
1.64
1.95
4.97
11.07
1.58
4.89
71.47
64.47
48.28
50.12
76.05
59.73
MaPLe
1.13
2.56
4.88
12.42
1.06
5.23
72.05
64.57
48.78
47.66
76,61
59,41
+DOR(ours)
1.46
1.89
3.96
11.08
1.37
4.58
71.93
64.94
48.77
48.29
76.20
59.55
Table Average accuracy (%) across 11 base-to-new datasets. "Vanilla" denotes the baseline w/o DOR. DOR can improve the generalization capacity on unseen classes while maintaining the performance on base classes.
ZSCLIP
CoOp
CoCoOp
MaPLe
KgCoOp
DEPT
CoPrompt
PromptKD
Class
Vanilla
Vanilla +DOR
Vanilla +DOR
Vanilla
+DOR
Vanilla +DOR
Vanilla +DOR
Vanilla +DOR
Vanilla
+DOR
Base
69.49
82,97
83,20
80.57
79.89
82.11
82.08
82.29
82.13
83.70
83,81
82.32
82.39
85.74
85.52
New
74.32
61.74
72.01
72.47
74.59
73.89
75.89
72.21
73.14
65.0M
71.39
73.29
74.50
79.80
80.81
HM
71.90
72.36
77.61
76.52
77.24
78.00
78.98
77.25
77.64
74.37
77.60
77.81
78.44
82.77
83.17
verify the robustness of DOR, we further evaluate its per- formance in the domain generalization setting, i.e., there covariate shift between training and testing datasets. Specifically, we first fine-tune the model with all classes of ImageNet on the 16-shot setting and then evaluate it on 4 types of datasets with covariate shifts. As shown in Table 5, the calibration performance indicates DOR main- tains stability in the presence of covariate shifts. Although DOR is not specially designed for the covariate shift, it outperforms the calibration baseline of CoOp and MaPLe, reducing ECE by 2.29% and 0.65% under distribution shifts respectively. Meanwhile, DOR demonstrates superior accu- racy in domain generalization and successfully maintains in-distribution performance. Such regularization will not hinder the generalization capacity of fine-tuned CLIP.
6. Discussion
Can the criterion of DOR be extended to visual tuning? In this paper, we primarily focus on prompt tuning and analyze how textual divergence impacts confidence calibra- tion. Such analysis may limit the potential scope of CLIP fine-tuning methods. To address this, we further consider a similar regularization approach based on image outliers for fine-tuning on visual representation. Specifically, we use ImageNet- lk (Deng et al., 2009) as an outlier repository and conduct experiments on four downstream datasets. To avoid potential semantic overlap between the outlier and fine- tuning data, we construct the outlier set by retaining images from 50% of the classes of ImageNet-Ik, which ensures these classes differ much as possible from those used during fine-tuning. For visual representation fine-tuning methods, we utilize CLIP-adapter (Gao et al., 2024) and visual prompt tuning (VPT) (Jia et al., 2022).
DOR is insensitive to hyperparameters. To further il- lustrate the influence of hyperparameter we present sensitivity analysis. We report the average performance 11 datasets under the base-to-new evaluation protocol. As shown in Figure 4, we can observe that DOR demonstrates robustness in model calibration as À in (10) varies. Al- though KgCoOp has better calibration on new classes as increases, it sacrifices accuracy and calibration on base classes while our method does not. The results verify that DOR is an effective approach to boosting calibration perfor- mance new classes while maintaining performance base classes. We provide more ablations on DOR including selection strategy, similarity metric, outlier numbers, update frequency, and outliers databases in Appendix H.
As shown in Table 7, we observe that visual-based DOR can successfully reduce the calibration error on new classes For example, DOR outperforms VPT and CLIP-adapter baseline by reducing ECE by 4.64% and 1.82% on the DTD dataset, respectively. We provide additional analysis of visual distribution in Appendix K. In general, visual-based DOR achieves better average calibration across various down- stream datasets and leaves space for further improvement. Given that expected image outliers are not always accessi- ble easily as text, potential method is to generate them by diffusion (Du et al., 2024) for high-quality image outliers.
8

---

Understanding and Mitigating Miscalibration in Prompt Tuning for Vision-Language Models
Table Calibration results of ECE (%) on fine-tuning of visual representation. DOR-V(ision) is effective for better calibration via visual
representation regularization. HM refers to harmonic mean.
Flowers
Cars
DTD
UCF101
AVG
Method
Base
New
Base
New
Base
New
Base
New
Base
New
HM
VPT
7.98
8.20
5.32
1.93
2.54
13.04
4.04
4.79
4.97
6.99
5.98
+DOR-V(ours)
8.19
7.54
4.90
1.78
2.68
8.40
3.73
4.58
4.88
5.58
5.23
CLIP-adapter
3.70
6.55
6.09
5.73
3.00
7.45
4.04
7.09
4.21
6.71
5.46
+DOR-V(ours)
4.02
4.86
7.13
4.85
3.25
5.63
1.90
5.23
4.08
5.14
4.61
Table 8. Calibration results of ECE (%) with different outliers. Oracle* denotes new classes that meet during test time.
confidence calibration on both base and new classes. We show that current prompt tuning methods typically lead to a tradeoff between the base and new classes. Through the textual divergence, we provide a thorough explanation for the limitations of those tuning methods. By utilizing relevant but non-overlapped outliers, DOR regularizes the textual distribution to preserve calibration capacity in zero- shot CLIP. Our method compatible with existing prompt- tuning methods and can be extended improve visual fine- tuning methods, like adapters. We hope future research can extend the insight in this work to other VLMs.
w/o
Near
Far
Random
Oracle*
Base
3.07
2.68
2.95
2.80
3.13
New
14.58
7.09
7.72
7.33
4.34
HM
8.83
4.89
5.34
5.07
3.74
What makes good regularization for CLIP fine-tuning? In the experiments, we use the relevant but non -verrlapedpe outlier with the fine-tuning task (referred as near- -OD) Such selection raises the question: why do we prefer to choose hear-OOD> To address this, conducted an abla- tion on the policies of outlier selection. We considered four -OOD,
Limitations. Similar to previous regularization methods of CLIP, DOR uses a hyperparameter À to control the weight of regularization, which will require extra computational costs. Morcover, our analysis is limited in the scope of CLIP, leaving other kinds of VLMs to be explored in the future.
types of outliers: near-OOD (ours), far-OOD, and new classes used at test time. The For is selected by applying the opposite operation of Eq. 7. "w/o" denotes the baseline without outlier regularization. Since target new classes are unknown during finettunin, we view them as an oracle, which serves as a performance upper bound for base-to-new tasks. We report the average performance the base-to-new datasets.
Impact Statement
Foundational model plays an important role in today's ma- chine learning research. These models typically show re- markable zero-shot generalization capabilities and achieve better performance on downstream tasks via fine-tuning, Unfortunately, it often comes at the cost of generalization, which posing a significant challenge in real-world applica- tions. In this paper, we investigate the confidence calibration of vision-language models (e.g., CLIP) after fine-tuning. Our goal is to mitigate overconfidence while preserving the models' adaptability to diverse tasks. We hope our find- ings can extend to larger language models or more vision- language architectures, aiming enhance their robustness after post-training in real-world scenarios.
We present the results in Table 8. Since we primarily fine- tune the model for a specific downstream task, selecting random data or far-OOD data may not be optimal for base- to-new evaluation. Additionally, the target class unknown during the fine-tuning phase. Therefore, we dynamically use near- Maar-OODaaaa as regularization data, which reduces calibration errors on new classes while preserving performance on base classes. We present several word extracted by near-OOD in Appendix E. Interestingly, the random selection can serve as a strong baseline, demonstrating the robustness of our pro- posed regularization item. Moreover, while the oracle can achieve impressive performance on the calibration of new classes, the fixed number of outliers may cause the model to overfit them, resulting in a decline in generalization.
A Acknowledgment
Shuoyuan Wang and Hongxin Wei are supported by the Shenzhen Fundamental Research Program (Grant No. JCYJ2023080709 (09000000009800000) We gratefully acknowledge the support of the Center for Computational Science and Engineering at the Southemn University of Science and Tech- nology for our research.
7. Conclusion & Limitations
In this paper, we introduce Dynamic Outlier Regularization (DOR), a simple yet effective technique that enhances the

---

Understanding and Mitigating Miscalibration in Prompt Tuning for Vision-Language Models
References
Jia, C., Yang, Y., Xia, Y., Chen, Y.-T., Parekh, Z., Pham, H., Le, Q., Sung, Y.-H., Li, Z.,, and Duerig, T. Scaling up visual and vision-language representation learning with noisy text supervision. In International Conference on Machine Learning, pp. 4904-4916, PMLR, 2021.
Bossard, L.,, Guillaumin, M., and Van Gool, L. Fold-101- mining discriminative components with random forests. In European Conference on Computer Vision, pp. 446- 461. Springer, 2014.
Jia, M., Tang, L., Chen, B.-C., Cardie, C., Belongie, S., Hariharan, B., and Lim, S.-N. Visual prompt tuning. In European Conference on Computer Vision, pp. 709-727. Springer, 2022.
Cimpoi, M., Maji, S., Kokkinos, I., Mohamed, S., and Vedaldi, A. Describing textures in the wild. In Pro- ceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3606-3613, 2014.
Jiang, W., Cheng, H., Chen, M., Wang. C., and Wei, H. DOS: Diverse outlier sampling for -uttrf-distributionn detection. In The Twelfth International Conference on Learning Representations, 2024.
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pat- tern Recognition, pp. 248-255 IEEE, 2009.
Joy, T., Pinto, F,, Lim, S.-N., Torr, P. H., and Dokania, P. K. Sample-dependen adaptive temperature scaling for improved calibration. In Proceedings of the AAAI Con- ference on Artificial Intelligence, volume 37, pp. 14919- 14926. 2023.
Du, X., Sun, Y.. Zhu, J., and Li, Y. Dream the impossible: Outlier imagination with diffusion models. Advances in Neural Information Processing Systems, 36, 2024.
Fei-Fei, L.,, Fergus, R., and Perona, P. Learning generative visual models from few training examples: An incremen- tal bayesian approach tested on 101 object categories. In 2004 Conference Computer Vision and Pattern Recog- nition workshop, pp. 178-178. IEEE, 2004.
Khattak, M. U., Rasheed, H., Maaz, M., Khan, S., and Khan, F. S. Maple: Multi-modal prompt learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19113-19122, 2023a.
Gao, P., Geng, S., Zhang, R., Ma, T., Fang, R., Zhang, Y., Li, H., and Qiao, Y. Clip-adapter: Better vision- language models with feature adapters. International Journal of Computer Vision, 132(():581-55)) 2024.
Khattak, M. U., Wasim, S. T., Naseer, M., Khan, S., Yang, M.-H., and Khan, S. Self-regulating prompts: Foun- dational model adaptation without forgetting. In Pro- ceedings of the IEEF/CVF International Conference on Computer Vision, pp. 15190-152000 2023b.
Guo, C., Pleiss, G., Sun, Y., and Weinberger, Q. On cali- bration of modern neural networks. In International Con- ference on Machine Learning, pp. 1321-1330. PMLR, 2017.
Krause, J., Stark, M., Deng, J., and Fei-Fei, L. 3d ob- ject representations for fine-grained categorization. In Proceedings of the IEEE International Conference on Computer Vision Workshops, pp. 554 1844-561,22 2013. 2013.
Helber, P., Bischke, B., Dengel, A., and Borth, D. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Se- lected Topics in Applied Earth Observations and Remote Sensing, 12(7):2117-2266, 2019.
Lee, S., Park, C., Lee, H., Yi, J., Lec, J., and Yoon, S. Removing undesirable feature contributions using out-of- distribution data. In International Conference on Learn- ing Representationss 2021.
Le W., Pikus, B., Raj, P., and Gil, F, A. Enabling cali- bration in the zero-shot inference of large vision-language models. ICLR workshop on Pitfalls of limited data and computation for Trustworthy ML, 2023.
Hendryeks, D., Mazeika, M., and Dietterich, T. Deep anomaly detection with outlier exposure. In International Conference on Learning Representations, 2019.
Hendrycks, D., Basart, S., Mu, N., Kadavath, S., Wang, F., Dorundo, E., Desai, R., Zhu, T.. Parajuli, S., Guo, M., et al. The many faces of robustness: A critical analysis of ditlo---iittibttionn generalization. In Proceedings of the IEFF/CVF international conference on computer vision, pp. 8340-8349, 2021a.
Li, Z., Li, X., Fu, X., Zhang, X., Wang, W., Chen, S., and Yang. J. Promptkd: Unsupervised prompt distillation for vasion-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 26617-26626, 2024.
Liang, F. Wu, B., Dai, X., Li, K., Zhao, Y., Zhang, H., Zhang, P., Vajda, P., and Marculescu, D. Open-vocabulary semantic segmentation with mask-adapted clip. In Pro- ceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition, pp. 7061-7070, 2023.
Hendrycks, D., Zhao, K., Basart, S., Steinhardt, J., and Song, D. Natural adversarial examples. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 15262-15271, 2021b.
10

---

Understanding and Mitigating Miscalibration in Prompt Tuning for Vision-Language Models
Liu, W., Wang, X., Owens, J., and Li, Y. Energy-based out- of-distribution detection. Advances in neural information processing systems, 33:21464-21475, 2020.
Nixon, J., Dusenberry, M. W., Zhang, L., Jerfel, G., and Tran, D. Measuring calibration in deep learning. In Computer Vision and Pattern Recognition workshops, volume 2, 2019.
Lu, Y., Liu, J., Zhang, Y., Liu, Y., and Tian, X. Prompt distribution learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pp. 5206-5215, 2022.
Oh, C., Lim, H., Kim, M.. Han, D., Yun, S., Choo, J., Hauptmann, A. G., Cheng, Z.-Q., and Song, K. Towards calibrated robust fine-tuning of vision-languee In The Thirty-eighth Annual Conference on Neural Infor- mation Processing Systems, 2024.
models.
Maji, S.,, Rahtu, E., Kannala, J., Blaschko, M., and Vedaldi, A. Fine-gras Fine-grained visual classification of aireraft. arXiv preprint arXiv:1306.55:
2013.
Parelli, M.. Delitzas, A., Hars, N., Vlassis, G., Anagnos- tidis, S., Bachmann, G., and Hofmann, T. Clip-guided vision- vision-language pre-training for question answering in 3d scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5606- 5611, 2023.
Mayilvahanan, P., Wiedemer, T., Rusak, E., Bethge, M., and Brendel, W. Does CLIP's generalization performance mainly stem from high train-test similarity? In The Twelfth International Conference on Learning Represen- tations, 2024.
Miller, G. A. Wordnet: lexical database for english. Com- munications of the ACM, 38(11):39944 1995.
Parkhi, O. M., Vedaldi, A., Zisserman, A., and Jawahar, C. Cats and dogs. In 2012 IEEE Conference on Computer Vision and Pattern Recognition, pp. 3498-3505. IEEE, 2012.
Minderer, M., Djolonga. J., Romijnders, R., Hubis, F., Zhai, X., Houlsby, N., Tran, D., and Lucic, M. Revisiting the calibration of modern neural networks. Advances in Neural Information Processing Systems, 34:15682- 15694, 2021.
Pereyra, G., Tucker, G., Chorowski, J., Kaiser, L.,, and Hinton, G. Regularizing neural networks by penalizing confident output distributions. 2017.
Ming, Y. and Li, Y. Understanding vetrieval-augment task adaptation for vision-language models. In Forty-first International Conference on Machine Learning, 2024.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pp. 8748-8763, PMLR, 2021.
Ming, Y., Fan, Y., and Li, Y. Poem: Out-of-distribution detection with posterior sampling. In International Con- ference on Machine Learning, pp. 15650-15665. PMLR, 2022.
Recht, B., Roelofs, R., Schmidt, L., and Shankar, V. Do imagenet classifiers generalize to imagenet? In Interna- tional conference machine learning, pp. 5389-5400. PMLR, 2019.
Mukhoti, J., Kulharia, V., Sanyal, A., Golodetz, S., Torr, P., and Dokania, P. Calibrating deep neural networks using focal loss. Advances in Neural Information Processing Systems, 33:15288-15299 2020.
Roy, S. and Etemad, A. Consistency-guide prompt learning for vision-language models. arXiv preprint arXiv:2306.01195, 2023.
Murugesan, B., Silva-Rodrigus J., Ayed, I. B., and Dolz, J. Robust calibration of large vision languuel adapters. In
European Conference on Computer Vision, pp. 147-165. Springer, 2025.
Shi, G., Li, Q., Zhang, W., Chen, J., and Wu, X.-M. Recon: Reducing conflicting gradients from the root for multi- task learning. In The Eleventh International Conference Learning Representations, 2023.
Nacem, M. F., Khan, M. G. Z. A., Xian, Y., Afzal, M. Z., Stricker, D., Van Gool, I.., and Tombari, F. 12mvformer: Large language model generated multi-view document supervision for zero-shot image classification. In Proceed- ings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pp. 15169-15179, 2023.
Soomro, K., Zamir, A. R., and Shah, M. Ucf101: A dataset 101 human actions classes from videos in the wild. arXiv preprint 1rXiv:1212.0402 2012.
Nilsback, M.-E. and Zisserman, A. Automated flower clas- sification over a large number of classes. In 2008 Sixth Indian Conference on Computer Vision, Graphics & Im- age Processing, pp. 722-729. IEFF, 2008
Tu, W., Deng, W., and Gedeon, T. A closer look at the robust- ness of contrastive language-image pre-training (CLIP). In Seveth- Conference on Neural Information Pro- cessing Systems, 2023.

---

Understanding and Mitigating Miscalibration in Prompt Tuning for Vision-Language Models
Wang, H., Ge, S., Lipton, Z.,, and Xing, E. P. Learning robust global representations by penalizing local predic- tive power. Advances in Neural Information Processing Systems, 32, 2019.
Yuksekgonul, M., Zhang, L.,, Zou, J., and Guestrin, C. Be- yond confidence: Reliable models should also consider atypicality. In Thirty-seventh Conference Neural In- formation Processing Systems, 2023.
Wang. S., Wang. J.. Wang. G., Zhang, B., Zhou, K., and Wei, H. Open-vocabulary calibration for fine-tuned CLIP. In Forty-first International Conference on Machine Learn- ing, 2024.
Zadrozny, B. and Elkan, C. Obtaining calibrated probability estimates from decision trees and naive bayesian classi- fiers. In International Conference on Machine Learning, volume 1, pp. 609-616 2001.
Wei, H.. Tao, L., Xie, R., and An, B. Open-set label noise can improve robustness against inherent lahel noise. Ad- vances in Neural Information Processing Systems, 34: 2021.
Zadrozny, B. and Elkan, C. Transforming classifier scores into accurate multiclass probability estimates. In Pro- ceedings of the eighth ACM SIGKDD International Con- ference on Knowledge Discovery and Data mining, Pp- 694-699, 2002.
Wei, H., Tao, L., Xie, R., Feng, L., and An, B. Open- sampling: Exploring out-of-distribution data for re- balancing long-tailed datasets. In International Confer- ence on Machine Learning, pp. 23615-23634 PMLR, 2022.
Zhang, J., Kailkhura, B., and Han, T. Y.J. Mix-n-match: Ensemble and compositional methods for uncertainty cal- ibration in deep learning. In International Conference on Machine Learning, pp. 11117-11128, PMLR, 2020.
Wortsman, M., Ilharco, G., Kim, J. W., Li, M., Kornblith, S., Roelofs, R., Lopes, R. G., Hajishirzi, H., Farhadi, A., Namkoong, H., et al. Robust fine-tuning of zero-shot models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 7959-7971, 2022.
Zhang, J., Huang, J., Jin, S., and Lu, S. Vision-language models for vision tasks: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024a
Zhang, J., Wu, S., Gao, L., Shen, H. T., and Song, J.
Dept: Decoupled prompt tuning, In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12924-12933, 2024b.
Xiao, J., Hays, J., Fhinger, K. A., Oliva, A., and Torralba, A. Sun database: Large-scale scene recognition from abbey to 200. In 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 3485-3492. IEEE, 2010.
Zhang, R., Zhang, W., Fang. R., Gao, P., Li, K., Dai, J., Qiao, Y., and Li, Tip-adapter: Training-free free adaption of clip for few-shot classification. In European Conference on Computer Vision, pp. 493-510. Springer, 2022.
Xiong, M., Deng, A., Koh, P. W., Wu, J.. Li, S., Xu, J., and Hooi, B. Proximity-informed calibration for deep neural networks. In Shirty-eeventh Conference on Neural Information Processing Systems, 2023.
Zhou, K., Yang, J., Loy, C. C., and Liu, Conditional prompt learning for vision-language models. In Proceed- ings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 16816-16888 2022a.
Yang, J., Shi, R., Wei, D., Liu, Z., Zhao, L., Ke, B., Pfister, H., and Ni, B. Medmnist v2-a large-scale lightweight benchmark for 2d and 3d biomedical image classification. Scientific Data, 10(1):41, 2023.
Zhou, K., Yang, J., Loy, C. C., and Liu, Learning to prompt for vision-language models. International Jour- nal of Computer Vision, 130(0))2337-2348, 2022b.
Yao, H., Zhang, R., and Xu, C. Visual-language prompt tuning with knowledge-guides context optimization. In Proceedings of the IEFE/CVF Conference on Computer Vision and Puttern Recognition, pp. 6757-6767, 2023.
Zhu, B., Niu, Y., Han, Y., Wu, Y., and Zhang, H. Prompt- aligned gradient for prompt tuning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 15659-15669, 2023.
Yao, H., Zhang, R., and Xu, C. Tep: Textual- based class- aware prompt uning for visual-language model. In Pro- ceedings of the IEEF/CVF Conference on Computer Vi- sion and Pattern Recognition, pp.
2024.
Yi, C., Ren, L.,, Zhan, D.-C., and Ye, H.-J Leveraging cross-modal neighbor representation for improved clip classification. In Proceedings of the IEFE/CVF Confer- ence on Computer Vision and Pattern Recognition, pp. 27402-27411, 2024.
12

---

Understanding and Mitigating Miscalibration in Prompt Tuning for Vision-Language Models
A. Related work
Vision-language models. Large pre-trained large vision-language models (Jia et al., 2021; Radford et al., 2021) have been verified effectively comprehend visual concepts using language supervision and apply them in downstream tasks (e.g, image classification (Radford al., 2021; Zhou et al., 2022b; Lu et al., 2022; Naeem et al., 2023), aoowlede-auugeened Metn & Li, 2024) and visual question answering (Parelli et al., 2023)) in zero-shot manner. Despite VLM's effectiveness in generalizing new visual concepts, the performance of zero-shot CLIP still lags behind the fine-tuned performance on specific downstream tasks (Zhang et al., 2024a). To further boost the downstream adaptation of pre-trained VLMs, many parameter-efficient tuning methods like vanilla prompt tuning (Zhou et al., 2022b;a; Khattak et al., 2023a) and adapter tuning (Gao et al., 2024; Zhang et al., 2022) have been proposed for high efficiency. Moreover, many regularization- based tuning methods have been proposed to preserve the generalization performance on unseen classes (Yao et al., 2023; 2024; Zhu et al., 2023; Roy & Etemad, 2023; Khattak al., 2023b). Despite the greater success of CLIP fine-tuning, the effectiveness of safety-related evaluation like confidence calibration has largely been overlooked, which is essential for real-world deployment.
Confidence calibration. Confidence calibration has been widely studied to ensure that the confidence levels output by models accurately reflect their empirical accuracy. To achieve this, the state-of-the the-art calibration methods can be categorized into regularization methods and post-hoc methods. For regularization methods, they either explicitly or implicitly regularize modern neural networks to have better calibration. Although regularization methods may not designed for calibration, they generally have better calibration performance including L2 regularization (Guo et al., 2017), Entropy regularization (Pereyra et al., 2017), focal loss (Mukhoti et al., 2020). etc. On the other hand, post-hoc methods fix the output probability after the training phase. For post-hoc methods, the most representative and simple method is temperature scaling (Guo et al.. 2017), which learns single scalar for rescaling the softmax logit. ATS (Joy et al., 2023) modifies the predicted confidence by per-data-porn poinnt adaptive temperature. Another type of post-hoc calibration is binning-based calibration (Zadrozny & Elkan, 2001; 2002). For instance, Mix-n-Match (Zhang et al., 2020) leverages ensemble and composition techniques to achieve data efficiency and maintain accuracy in confidence estimates. Recently, several works have explored the calibration in CLIP. (Murugesan et al., 2025) investigate the calibration of CLIP under covariate shift. Given that existing post-hoc calibration on base classes can not transfer to new classes, DAC (Wang et al., 2024) fixes the logit scale of prediction via textual deviation-informed score in a post-hoe manner. Different from them, we address the calibration issue during the fine-tuning phase. In this work, we introduce regularization based on dynamic outliers. We demonstrate that DOR boosts the calibration performance of many existing wate-of-the-arr prompt tuning methods of CLIP without affecting the vanilla fine-tuning objective.
Outlier regularization in trustworthy machine learning. The outlier plays an important role in trustworthy machine learning research including -isto--distribution (OOD) detection, noisy label learning, adversarial attack, long-tailed datasets re- ee-balancing, etc. In OOD detection, outliers are typically used to simulate the distribution of OOD data, thereby increasing the distinction between ID data and OOD data (Hendrycks et al., 2019; Liu et al., 2020; Ming et al., 2022; Jiang et al., 2024). Recently, Dream-OOD generate the expected OOD data by diffusion (Du et al., 2024). In noisy label learning, ODNL (Wei et al., 2021) leverages outliers dynamical noisy labels to improve model robustness against noisy labels. To address the extreme class imbalance, Open-samplin (Wei et al., 2022) re- re-balance class priors via open-set noisy labels. OAT (Lee et al., 2021) leverages outlier data to improve model generalization in adversarial robustness, which regularizes the softmax probabilities to be uniform distribution for outliers. In this paper, we utilize textual outliers to control the divergence of unseen textual distribution, and further improve the calibration of fine-tuned CLIP.
B. Additional analysis of the motivation
B.1. Detailed Results of textual divergence
In section 3.2, mainly derive the reason for the miscalibration issue from CoOp and KgCoOp. specifically, We use FD score to measure the diversity of textual representation and evaluate the prompt tuning methods on the value of output confidence and logit gap. To comprehensively verify our motivation, we further compare more prompt tuning methods including CoCoOp, MaPLe, PromptSRC and CoPrompt.
As is in Figure 5a, we can observe a similar phenomenon as we discussed in section 3.2. We find that fine-tuning can significantly increase the FD score of textual features compared to the zero-shot CLIP. Notably, this observation can extend
13

---

Understanding and Mitigating Miscalibration in Prompt Tuning for Vision-Language Models
0.5
ZSCLIP
Promp1SRC CoPrumpp
ZSCLIP
MaPLe
PrompeSRC
CoOp
KyCoOp
CoOp CoCoOp
KgCoOp
Cottomge
0.4
CoCoOp
0.85
FD Score
0.3
Confidence
0.80
0.70
0.1
0.65
0.0
0.60
Base
New
Base
New
(a) FD score
Confidence
Figure 5. Comparison between zero-shot CL.IP and different prompt tuning methods on UCF101 dataset. Fine-tuned CL.IP tends to have higher confidence and FD score on both base and new classes.
0.35
0.35
Max
Others
Max
Others
0.33
0.33
0.30
0.30
0.28
Logit
0.25
0.25
0.23
0.20
0.17
ZSCLIP
CoOp
CoCoOp
MaPle
KgCoOp
PromptSRC
CoPrompl
ZSCLIP
CoOp
CoCoOp
MaPLe
KgCoOp
PrompiSRC
CoPrompi
(a) Base
(b) New
Figure 6. Comparison between the maximum logit and the average of other logits DTD dataset. The logit gap become wider in fine-tuned CLIP.
to new classes, even if they are not explicitly optimized during the fine-tuning phase. Consequently, the gap between the maximum logit and the others widens after fine-tuning in Figure 6. Therefore, the tuned CLIP with CE loss will make softmax predictions with high confidence, which can generate higher average confidence (See Figure 5b), Such an increase in confidence aligns with the improved accuracy on base classes. However, consistent with the nearly unchanged accuracy classes, which leads overconfidence.
B.2. Why using outliers for the regularization?
To further demonstrate the superiority of outliers in the regularization for CLIP fine-tuning, we provide additional analysis in this section. Specifically, first analyze the calibration issue from the perspective of gradient conflicts (Shi al., 2023) and present empirical evidence to understand previous regularization terms hinder the calibration of base classes. We can decouple CLIP's optimization objective
Lelip Loe + X Lreg:
where Los is the cross-entropy loss for classification, and Grog, denotes the regularization term. Previously proposed regularization terms include Lkg (KgCoOp), Laistu (CoPrompt). and Lael (PromptSRC). For reasonable comparison, we set hyperparameters be the same including optimizer, learning rate, etc. We calculate the cosine similarity of the prompt
14

---

Understanding and Mitigating Miscalibration in Prompt Tuning for Vision- Yision-Language Models
0.6
+ dor (Ours)
0.5
+ Lig (KgCoOp)
5.0
Probability
0.4
+sd (PromptSRC)
disttll (CoPrompt)
ECE(%)
4.5
0.3
4.0
0.2
3.5
0.1
Base
3.0
New
0.0
1-1.0,
-0.5|
-0.33
(-1.5,
(..13,-0.11
(40.3,
-0.11
(-0.1.
0.11
0.31
(0.1.
0.55
10.5.
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
Outlier Ratio
Figure The distributions of gradient conflicts terms of cos Φ). DOR shows less gradient conflict compared with recent prompt tun- ing methods.
Figure ECE comparison the proportion of outlicrs changed. Outliers can break the calibration trade-off on base and new classes.
gradients between Creg and Lee to reflect the degree of gradient conflicts.
Regularization from base classes hinders original fine-tuning objective. As shown in Figure 7, the gradient conflict distributions for KgCoOp. CoPrompt, and PromptSRC are predominantly within the range of [-1, 0j. which indicates a conflict with the original learning objective. Considering that CoOp with vanilla Lee efficient calibrator for base classes, we can infer that these regularization terms may hinder the calibration performance for base classes. As an alternative. proposed DOR leverages outliers construct the regularization term. We observe that the gradient conflicts for DOR primarily concentrated within the range (-0.1, 0.1). Compared to previous regularization terms, it shows significantly fewer conflicts in the [-1, 0] range. The phenomenon supports our claim that outliers can be used in regularization without interfering with the original fine-tuning objective.
Outlier-based regularization can break the calibration trade-off. To further illustrate the actual performance of outliers in calibration, we conducted an analysis based on KgCoOp. Since KgCoOp uses a fixed number of base classes as the regularization term, we progressively replaced these texts with textual outliers at varying proportions [0.1, 1, 0.2... As shown Figure 8, as the proportion outliers increases, the calibration base classes improves while the performance on new classes remains largely unaffected. Such observation strongly supports our claim that outlier can effectively mitigate the miscalibration issue
classes while maintaining the calibration performance the base classes
C. Theoretical justification
To help readers understand the insights, we formalize our observations of CLIP fine-tuning that the textual divergence is a significant factor for confidence estimation in this part. As the image feature remains unchanged in fine-tuning, the textual divergence can be translated to variance of logits, which is computed by the similarity between the image feature and different text features. In the following, we formally show the relationship between the logit variance and the confidence.
For simplicity, consider binary classification problem. Let |=)) be set of logit vectors (i.e., model outputs), where each vector = [=1, 22] T consists of two logits. We assume that the logit is independent random variable drawn from a normal distribution A (1,, o²)). The confidence (i.c., maximum softmax probability) p; is given by the softmax (or sigmoid) function defined as in (2). We have the following proposition.
Proposition C.I. Let E[po] denote the expected value of the maximum probability p; when the logits are distributed as (i,, d²). Then, for any 01, 02 > 0 and μ. we have E(pa,] Elpe,], if 02 > d1-
This suggests that the high divergence in the logit distribution tends to generate larger predicted confidence, which is induced by the textual divergence using CE loss. In Section 5, we empirically verify that our proposed method can preserve the textual divergence on the new classes, thereby improving the calibration performance.
15

---

Understanding and Mitigating Miscalibration in Prompt Tuning for Vision-Language Models
Proof. We first remove the influence of μ.. For any constant 6, have:
Pj
Eド=1 c3
21=1 e
Thus, the mean value ji of the logits does not affect the softmax output. Therefore, without loss of generality, we can assume that j = 0 in our proof. For binary classification, we have:
e e 1 ( sigmoid (21 22) P2 sigmoid (22 21).
1
P1
Hence, the maximum softmax probability is:
Pmax max(p1, p2) sigmoid =2)).
Since ~ (O 2,²), the absolute difference Z = |2| z2| follows folded normal distribution with the probability density function:
fz(2)
4m2 0.
V T
Thus, the expected value the maximum softmax probability is:
El Pmp = F| 2)] = Jo sigmoid(z) fz(2)dz /0- 1
do dz.
VTO²
For simplicity, we perform the substitution u
The integral can be simplified to:
ov2
1e 0o0
V2 + du.
E Pmax]
1 1 e-v2u
Using Leibniz's rule, can differentiate with respect under the integral sign:
d. Pmax rcc 0 (
V2 du = [ e e 2u
Jo do (1+e-ovz
e du 2 0.
do
(1+e
2
Hence, the expected maximum probability Pmax increase alone with a. Then, We have E(poz| > Eipoy 02 > 01. The proposition is proven.
D. Implementation details
Table Hyperparameters for VLM tuning methods. "BS" denotes the batch size. "LR" denotes the learning rate. "CTX" is the context length of the learnable prompt.
CoOp
CoCoOp
DEPT
KgCoOp
MaPLe
TCP
CLIP-Adapter
VPT
Epochs
200
10
200
200
50
200
5
BS
32
32
32
32
4
LR
0.002
0.002
0.002
0.002
0.0026
0.002
0.002
0.0025
CTX 16
16
16
8
Our implementations are based on the open-source repository of DAC (Wang et al., 2024). Generally, We use CLIP ( ViT-B/16) (Radford et al., 2021) as the pre-trained VLM throughout our experiments and report results averaged over 3 runs. We fine-tune the model with 16 samples per class few-shot setting (Zhou al., 2022a). Following the corresponding
16

---

Understanding and Mitigating Miscalibration in Prompt Tuning for Vision-Language Models
Table 10.
Outlier selection from WordNet hased on zero-shot CLIP.
Dataset
Base class
Selected outier
('pink primrose". : 'hard-leaved pocket orchid'. 'sweet pea'. 'english marigold'. 'iiger lily". "moon orchid'. 'bird of paradise', 'monkshood', ; "glohe thistle', 'snapdragon', ; "colt's font", 'king protea'. 'spear thistle'. 'yellow iris', globe-flower,. ppuplee coneflower'. 'penvvian lily", "ballnon flower']
('May_ling Jily', "Roweriig planll,_pppa_.. 'dayllower'. 'ilyst_llly'.. 'llwwer, 'lily'. ''rchid', "African 'non- "non-flowere fooweinwening plant, 'plant', 'sego_lily', 'plant_ material",
Flowers102
'Bowesof-aathoour, 'Olora'. "liliaccous_ plant. 'Liliaccae'.
"apetaloos_ flowrr, 'tongueflower" "herbaceous_ plant,, "daisyhush']
['spaniel', 'bulldog'. 'dog'.
l'ubyssinian', 'american ammericanbailldog'. 'ameerican_pt Lactican_ptt__ll______ hiL_boll_terrer [errierr" baaset_boundd "beagle", "bengal", 'birman', 'bombay'. 'boxer',
'dog_o',, 'doggie", 'doggy, 'domestic_dee _ogestticddg.....
OxfordPets
"canine', , 'pug-dog'. "pooch", Sapaaese_sspaiill
, 'chihuahua', "egyptian mau', "german shorthaired"]
"Labrador_ "Labradorrettrieverr, 'smusage_dog', "Lahrador', "Litle_Dog', 'housedog'. , 'CAT', 'retriever', 'French_ "French_bullddg! dog')
"english_encker___eee cocker_spaniel! "english_seen setiiss___ttrr,, /
['estate_car', "automotive_ vehhcle"", "sedan", 'used-car",
2012 Acura RL Sedan', '2012 Acura TL Sedan',
'tesla", "Tesla,, "pickup_truck', 'SUV", hhatchback'' 'subccompcc_ccrr" 'putrol_car', "stattin "station_wagane wagon", 'sports_cat sidiuuittyyyyhhiieek
StanfordCars
22088 Acura TL Type-S", '2012 Acurn TSX Sodan', 2001 Acura Integra Type R', '2012 Acura ZDX Hatchback']
vehiille".. 'secondhand_ce 'vehicle", 'sport 'touring.car"
['707-320', '727-200', '737-200', "737-300", '737-400', '737-500', '747-200', '747-300', '747-400", '757-200', A340-600", , 'A380',
('airliner,, "widebody_ 'aircraft", 'aitbus", 'jetliner', hide-body_airrriit 'jumbojet', "air__ranssee iis_transpor... 'narrow-bode aiicaaircrafft',, 'muutiingine_siiplann" "airline",
FGVCAincraft
72'. 'BAE 146 46-2000,, 'BAE 146-300 300', 'BAE-125", 'Beecheraft Boeiing 717', 'C-130', 'C-47', 'CRI-200', 'CRJ-700", 'CRJ-900' "Cessna 172', "Cossna 208', 'Cessnu
"attack_ aircraft" "reconnissance__ sattaeee__e_e____eeeee "air__ramm transportationn 'military_phana plancy,_ppeeeee "acroplane", 'plane', "multiengine_piiee planciie__paae______ 1
['apple_pie' "baby_bbbuu back_ribb'. 'bakluva'. "beef_ parppcccc,, "heef_ tartare', ; salad,, 'beignets'. 'bibimbap', 'brend_puxdding. breakfasttbbrrito 'bruschetta,, ccaesar_salud', 'cannoli', caprese_salade 'carrot_cake', 'ceviche', 'cheese cheesse_patee, 'cheesecake,, "chicken_eury" 'chicken quesadilaa,
[' I'entres', 'pastry'. 'bread', 'breakfast_food 'food',
'salad', , 'burger', 'Burger', 'dessen'', 'soup'. trrench_pastry' 'sandwich', 'steak', 'meat, 'pizza', 'cuisine", 'pie', 'PIE', 'French_ bread", 'dish")
Food101
I'weightlifting, ; 'athletics', 'sports_ imppltsiippeeeen"" "hitting', : "physical_exeetee exercise'. "neur_thing' conletic_competttionn 'phot', 'depicting'. . 'fimess', 'athletic sartiii__mmmm,,, "physical_finess fitness". 'batting". 'goal', "musical_style 'photogmphy', hhap 'guing']
['Apply_Fun Beeeeeeeeeeaaee Makeup'. Lipstick,p 'Archery', StablnCn "Balance_ Beamm,, 'Band "Band_ Marching'. (Baseball_Pitchh 'Baskehalll', 'Baskethall Dunk", "Bench_ 'Biking']
UCF101
official implementation, We list the general hyperparameters in Table 9, For hyperparameter À in DOR, we set À 8.0 for CoOp, x 4.0 for MaPLe and 2.0 for other fine-tuning methods. We set the number of selected dynamic outlier repository to 5000. The number of outliers in each batch is the same as the base classes. Here, we adopt these VLM tuning methods from the corresponding official implementation and briefly introduce the corresponding hyperparameters of them. All the methods are adopted from their official implementation. For CoOp and CoCoOp, they do not contain other hyperparameters. For KgCoOp, we set A = 8.0. For MaPLe, we set prompt depth J to 0 and the language and vision prompt lengths to 2. For DePT, the learning rate for updating the parameters in the devised CAT head is set to 6.5 x 8. where ó is the adopted learning rate of CoOp. Moreover, the weight in the linear probe is set to 0.7. For TCP, the weight for prompt fusion is 1.0, and the loss weight is the same as KgCoOp. For alIPead we set 0.6, which is trade-off hyperparameter between fine-tuned and zero-shot visual representation. Finally, following MaPLe, we set the context length to 8.0 and prompt depth to 12 for VPT.
E. A close look at selected outliers
In this section, we present the detailed results for the dynamic outlier selection. Specifically, we select nouns from WordNet that do not overlap but share higher-level concept relations with the base classes seen in the fine-tuning task. We use the textual encoder of zero-shot CLIP as the modality encoder.
As shown in Table 10, we can conclude that the selected outlier meets our requirement. For example, if our base class contains certain aircraft models such as 'A340-600', 'A380', and ATTR722' the outliers we selected include words like "air_ transport', "air transportation'. and imiiitary_plane overlap with the base class. For further influence, we show that using outliers that are relevant but do not overlap with our fine-tuning task are helpful in reducing calibration error while preserving performance in the base classes. To verify it, we
These nouns are highly relevant to the downstream task but do not
17

---

Understanding and Mitigating Miscalibration in Prompt Tuning for Vision-Language Models
empirically demonstrate that using base classes the regularization may sacrifice its accuracy and calibration in Figure 4. Moreover, As shown in Table dynamic text is better than fixed text since the fixed number of text may cause the model to overfit them.
F. The detailed experimental results
In this section, present the detailed results proposed DOR in Table 11.
of Expected Calibration Error (ECE) to verify the effectiveness of our
Table 11. ECE (%) comparison of existing prompt tuning in the base-to-new generalization.
Methods
Caltech101
Pets
Cars
Flowers
Food101
FGVC
SUN397
DTD
EuroSAT
UCF101
ImageNet
AVG
ZeroshoiCLIP
6.49
2.25
1.59
4.53
3.24
1.51
CoCoOp
1.45
2.32
6.61
7.67
1.10
3.41
1.66
2.61
8.06
2.08
2.66
3.60
CoCoOp+DOR
2.20
2.97
6.85
8.49
0.97
3.33
3.19
2.86
10.12
2.96
2.52
4.22
CoOp
0.95
0.96
6.73
4.04
1.38
CoOp+DOR
1.59
1.78
4.20
4.90
0.87
5.81
1.61
1.96
DEPT
2.22
6.83
12.08
7.75
5.41
5.23
2.90
3.29
8.26
3.32
9.15
6.04
DEPT+DOR
2.95
8.32
13.37
10.26
7.04
6.51
5.55
4.78
9.50
5.08
10.96
7.67
KgCoOp
2.30
2.95
10.05
4.69
8.02
10,97
4.18
KgCoOp+DOR
2.48
2.96
11.02
10.19
1.39
7.64
4.84
8.34
11.03
4.32
2.57
6.07
MaPLe
1.21
2.09
5.81
4.23
0.82
3.46
1.04
4.34
3.53
1.77
1.95
2.75
MaPLc=DOR
1.84
2.05
6.49
4.47
0.86
2.40
2.28
2.32
4.44
3.08
2.02
2.93
TCP
1.99
2.44
2.64
6.83
3.58
2.12
4.71
TCP+DOR
2.03
2.46
9.34
7.10
5.28
2.90
6.52
9.84
3.48
2.09
4.79
PromptSRC
2.41
2.37
8.14
4,62
0.89
4.29
2.08
2.87
9.15
2.43
2.01
3.75
PrompiSR++DOR
2.32
2.56
8.19
4.92
0.95
4.12
2.16
3.53
9.20
2.59
2.13
3.88
CoPrompt
1.56
2.81
3.91
4.92
0.99
2.47
0.90
2.78
2.10
1.68
2.56
CoPRompt-DOR
1.96
2.86
4.25
6.24
1.02
2.50
1,53
2.97
5.63
1.74
1.91
2.96
(a) Base
Methods
1oltech101
Pets
Cars
Flowers
Food101
FGVC
SUN397
DTD
EuroSAT
UCF101
ImageNet
AVG
ZeroshorCLIP
1.60
3.42
3.31
4.91
1.83
6.55
3.48
9.12
5.52
2.09
4.61
CoCoOp
3.94
2.35
2.26
11.33
1.63
12.51
2.03
16.40
9.17
4.39
1.57
6.14
CoCoOp+DOR
1,30
2.98
3.03
6,77
1.82
7.67
1.12
6.00
8.26
3.65
1.67
4.02
CoOp
4.11
1.57
11.81
19.84
4.42
32.12
15.98
26.49
15.50
18.09
10.40
14.58
CaOp+DOR
1.42
2.94
8.01
7.34
1.22
21.21
2.31
11.34
8.67
5.07
1.89
6.49
DEPT
4.23
2.71
11.15
18.32
2.71
34.21
15.15
24.30
18.90
18.94
9.73
14.58
DEPT+DOR
2.51
2.64
7.53
6,58
0.74
22.62
5.01
17.10
7.34
7.65
2.77
7.50
KgCoOp
2.02
3.15
3.35
5.92
1.87
12.76
1.51
7.41
6.56
2.95
1.74
4.48
KgCuOp+DOR
1.43
3.04
3.46
6.68
1.83
9.63
2.33
5.78
$.29
2.52
1.86
3.99
MaPLe
2.66
2.35
2.95
10.32
1.16
10.72
2.42
15.54
6.06
3.65
2.27
5,46
MaPLc+DOR
1.71
2.50
2.42
10.27
1.51
10.56
0.90
10.64
7.15
2.52
1.68
4.71
TCP
1.15
2.94
2.46
5.14
2.34
8.07
1.98
4.91
8.36
5.78
1.59
4.07
TCP+DOR
1.21
3.03
2.43
4.26
2.23
7.51
2.56
4.72
6.21
5.91
1.70
3.80
PromptSRC
1.55
3.07
2.02
5,53
1.66
11.31
0.66
6.42
8.53
3.24
1.71
4.15
PrompISRCDDOR
1.64
2.82
1.84
5.58
1.49
9.58
0.74
5.72
7.56
3.04
1.82
3.80
CoPrompt
1.69
2.41
5.59
10.19
1.67
11.54
2.28
8.64
16.18
2.60
2.79
5.96
CoPromp++DOR
1.43
3.13
5.50
8.48
1.70
13.16
1.17
5.68
6.28
2.66
2.40
4.69
(b) New
18

---

Understanding and Mitigating Miscalibration in Prompt Tuning for Vision-Language Models
Base
New
Base
New
CoCpp Cudp
Dunity
Dundit
Denshy
6.11
022
255
0/4
0.
0.2
0.68
0.00
0.
Logit
Legil
Masimmmm Probakility
Figure 9. Distribution visualization of logit and maximum softmax probability on FGVCAircraft datasct. Our method generates logit and probability distributions that closely resemble those CoOp base class are similar zero-shot CLIP the new class.
Table 12. Average calibration performance (ECE%) across 11 datasets using different similarity metrics. Calibration error is given by "HM" denotes the harmonic mean.
CoOp
MaPLe
CoPrompt
Metric
Base
New
HM
Base
New
HM
Base
New
HM
w/o DOR
3.07
14.58
8.83
2.75
5.46
4.11
2.56
5.96
4.26
Cosine
2.67
6.49
4.58
2.93
4.71
3.82
2.96
4.96
3,96
Euclidean
2.92
4.74
3.83
2.83
4.63
3.73
2.92
4.78
3.85
Mahalanobis
2.83
6.73
4.78
3.05
4.86
3.96
2.96
4.76
3.86
G. How does DOR affect the distribution of logit and probability?
To further illustrate the influence of DOR on confidence calibration, we visualize and compare the distribution of output logit and softmax confidence score for base and new classes in Figure 9. We compare zero-shot CLIP and CoOp w/ or w/o DOR on FGVCAircraft dataset. The results verify that DOR modifies the logit distribution and confidence level. We can observe that the model tuning With CoOp+DOR, the logit distribution base class approximate CoOp, and the new class similar zero-shot CLIP, respectively. A similar phenomenon observed in the softmax probability distribution. The results meet our vision mentioned in Section 3.1, DOR can leverage the advantages of both models, which ensure the confidence calibration both base and new classes after fine-tuning.
H. The ablations of DOR
H.1. The similarity metric in outlier selection
In the outlier selection, we use cosine similarity as the distance metric for selecting textual outliers. To assess the metric sensitivity of our proposed DOR, we conduct an ablation and use three metrics including Cosine similarity, Euclidean distance (I.2), and Mahalanobis distance. We report the average calibration performance on the base-to-new datasets across various prompt tuning methods, including CoOp, MaPLe, and CoPrompt.
We present the results in Table 12. We find that our DOR framework is not very sensitive to the choice of similarity metric and consistently reduces the calibration error. Surprisingly, Fuclidean distance can outperform Cosine similarity across different prompt tuning methods and achieve better harmonic mean (HM). For example, using Euclidean distance with CoOp yielded an HM of 3.83%, compared to 4.58% with Cosine similarity. Despite this, we use Cosine similarity as the default distance metric throughout this work since it is widely used in the feature selection vision-language models (Yi al., 2024; Mayilvahanan et al., 2024; Wang et al., 2024).
H.2. The size selected outlier set
We evaluate how the number of selected outliers in DOR affects the calibration performance. Specifically, we present this ablation with average ECE on the base-to-new datasets with three prompt tuning methods including CoOp. MaPLe, and
19

---

Understanding and Mitigating Miscalibration in Prompt Tuning for Vision-Language Models
14
CoOp
CoOp
MaPLe
MaPLe
3.1
12
CoPrompt
CoPrompt
ECH0%)
ECE(%
10
2.8
2.6
10
50
100
500
1000
5000
10000
20000
0
10
50
100
500
1000
5000
10000
20000
Number
Number
(a) Base
(b) New
Figure 10. The ablation the number of selected outliers. ECE (%) is calculated from the average performance base-to-new datasets.
Table Average ECE (%) across datasets with different frequencies of outlier update the batch. We use "I" denotes that the outliers are updated in every iteration.
CoOp
CoPrompt
10
100
1000
10
100
1000
Base
2.67
2.76
2.71
2.65
2.96
3.00
2.80
2.86
New
6.49
6.72
7.00
7.43
4.69
4.68
4.85
5.03
HM
4.58
4.74
4.86
5.04
3.83
3.84
3.83
3.95
CoPrompt. We vary the number of outliers k = {10,50, 50, 100,..., 20000}.
As shown in Figure 10, increasing the number of selected outliers leads to an evident reduction BCE on the new classes. The performance starts to reach point of saturation with more outliers. Notably, even setting k: 10 yields significant calibration improvements on the new classes. For the base classes, the outliers may be fixed in the batch during the fine-tuning due to the small number, leading to poor performance due to overfitting. We observe similar phenomenon in Table (KgCoOp) or Table 8. Hence, we suggest to set moderate number (e.g., 5000). Furthermore, when the number of outliers is sufficiently large, DOR becomes less sensitive to the exact value of numbers, demonstrating its robustness.
H.3. The frequency of outliers
In DOR, we randomly sample batch of textual outliers from the selected textual outlier set, in each iteration. Therefore, the textual outliers used in each iteration can be different, which establishes a dynamic regularization. To evaluate the impact of outlier update frequency on performance, we conduct an ablation study by varying the frequency at which outliers are sampled for the batch with the update intervals in the range [1, 10, 100, 1000). We present this ablation with average ECE on the base-to-new datasets with two prompt tuning methods including CoOp and CoPrompt.
As shown in Table 13, low update frequencies (or large update intervals) are observed negatively impact the calibration performance of DOR. The phenomenon suggests that infrequent updates may allow the model to overfit to noise and reduce its calibration performance. In short, The experimental results highlight the benefits of employing dynamic update strategy in DOR, since it helps mitigate overfitting to noise (Wei et al., 2021) and achieves superior calibration performance.
H.4. The choice of lexical database
In this work, we construct set of textual outliers using nouns from WordNet. To evaluate whether different lexical databases significantly affect the results, we performed an ablation study on the choice of lexical databases. Specifically, we consider two additional textual databases: CLIP's vocabulary and ConceptNet 5.7. For CLIP's vocabulary, it includes 49,407 characters and words. For ConceptNet, we use raw sentences and filter out those exceeding CLIP's s input limit (77 tokens). Finally. it consists of (05,662 short sentences. We report the average calibration results on the base-to-new datasets.
20

---

Understanding and Mitigating Miscalibration in Prompt Tuning for Vision-Language Models
Table 14. The ablation the repository of the outlier The average ECE the base-to-new datasets is compared. DOR insensitive to lexical databases.
Method
Class
Vanilla
WerdNet
CLIP
ConceptNeeS
Base
3.07
2.67
2.91
3.02
CoOp
New
14.58
6.49
8,43
8.37
HM
8.83
4.58
5.67
5.70
Base
2.75
2.93
2.86
3.19
MaPLe
New
5.46
4.71
5.11
5.24
HM
4.11
3.82
3.99
4.22
As shown in Table 14, we find that DOR can achieve the best calibration performance with WordNet and all databases can achieve better results than the baseline. Additionally, we observed that short sentences may not perform as well as prompts like photo of [class]."
Table Calibration results ECE (%) of prompt tuning methods DOR on pathMNIST. "Vanilla" denotes the baseline fine-tuning methods. Bold numbers are significantly superior results. DOR boosts the performance of existing methods confidence calibration.
ZSCLIP
CoOp
KgCoOp
MaPLe
DEPT
PromptSRC
CoPrompt
Class
Vanilla
Vanilla
DOR
Vanilla
DOR
Vanilla
DOR
Vanilla + DOR
Vanilla + DOR
Vanilla
DOR
Base
29.80
1,56
1.25
13.45
12.15
14.12
15.47
6.57
7.63
12.43
11.52
12.26
6.27
New
15.27
61.28
14.99
12.45
7.48
13.47
6,54
62.18
13.91
11.08
10.34
8.39
7.57
HM
22.54
31.42
8.12
12.95
9.82
13.80
11.01
34.38
10.77
11.76
10.93
10.33
6.92
Table 16. Accuracy (%) of prompt tuning methods with DOR pathMNIST. "Vanilla" denotes the baseline of fine-tuning methods. Bold numbers are significantly superior results. DOR boosts the performance of existing methods on generalization.
ZSCLIP
CoOp
KgCoOp
MaPLe
DEPT
PromptSRC
CoPrompt
Class
Vanilla
Vanilla + DOR
Vanilla DOR
Vanilla DOR Vanilla + DOR
Vanilla + DOR
Vanilla
DOR
Base
24.64
92.52
93.52
87.77
86.89
85.87
85.73
92.33
91.37
92.16
91.58
88.70
88.55
New
44.71
31.58
35.68
40.80
47.31
35.46
40.13
30.06
44.57
42.46
43.47
41.72
50.27
HM
34.68
62.05
64.60
64.29
67.10
60.67
62.93
61.20
67.97
67.31
67.53
65.21
69.41
I. Comparison with existing calibration methods
To further validate the effectiveness of our proposed DOR, we compare it with two recent calibration approaches for CLIP. We consider main calibration strategies: post-hoc scaling and regularization-basse training. For post-hoc scaling, we compare DOR with Pero-Shot-Enabled Temperature Scaling (ZS (SS-TS) (Le al., 2023). Specifically, perform post-hoc calibration the fine-tuned model using ImageNet- 1k and evaluate the learnable temperature 7 on both the base and new classes. For regularization baasedd calibration, we incorporate DOR into calibrated robust fine-tuning method (CaRot) (Oh et al., 2024) like Equation 10. We report the average ECE performance on the base-to-new datasets.
DOR outperforms post-hoe scaling under base-to-new evaluation. We present the results in Table 17. We observe that the temperature 7 optimized on ImageNet- 1k significantly mitigates the overconfidence and improves the calibration the
21

---

Understanding and Mitigating Miscalibration in Prompt Tuning for Vision-Language Models
Table 17. Average ECE (%) comparison with post-hoe scaling methods on base-to-new datasets. 7S-TS fails calibrate the hase classes.
DOR effectively mitigates the miscalibration issue on new classes while maintaining the calibration performance on the base classes.
CoOp
CoCoOp
KgCoOp
MaPLe
PromptSRC
Vanilla
+ZS-TS
+DOR
Vanilla
+78-18
+DOR
Vanilla
+ZS-TS
+DOR
Vanilla
+ZS-TS
+DOR
Vanilla
+ZS-T8
+DOR
base
3.07
8.25
2.67
3.60
9.12
4.22
5.82
8.49
6.07
2.75
6.68
2.83
3.75
6.74
3.88
new
14,58
7.96
6.49
6.14
4.81
4,02
4.48
3,36
3.99
5.46
4.05
4.44
4.15
3.48
3.80
HM
8.83
8.11
4.58
4.87
6.97
4.12
5.15
5.93
5.03
4.11
5.37
3.64
3.95
5.11
3.84
Table 18. BCE (%) comparsion with regularization-basee methods on base-to-new datasets. DOR incorporate with CaRot to achieve better calibration.
Caltech101
Pets
Cars
Flowers
Food101
FGVC
SUN397
DTD
EuroSAT
UCF101
ImageNet
AVG
CaRot
7.68
6.14
12.93
8.34
6.92
6.15
5.31
6.68
10.12
6.76
3.05
7.28
Base
CaRot+DOR
5.71
5,10
11.35
8.55
6.13
5.72
5.23
6.86
9.67
6.81
2.91
6.73
CaRot
2.51
6,46
4.32
5.66
7.16
5,20
3.54
6.03
6.37
5,09
1.78
4.92
New
DoR
2.87
4.93
3.71
4.66
6.39
5.74
3.62
5.10
8.74
4.92
1.80
4.77
fine-tuned CLIP on new classes. For instance, it reduces the Expected Calibration Error (ECE) of the state-of thettart method PromptSRC from 4.15% to 3.48%. However, such calibration can not used on the base classes. ZS-TS exacerbates the model's 's underconfidence and increases the ECE from 3.75% to 6.74% on PromptSRC. In contrast, our approach effectively mitigates the miscalibration issue on new classes while maintaining the calibration performance on the hase classes.
DOR boosts existing regularization-basee methods for better calibration. As shown in Table 18, we observe that CaRot achieves decent calibration performance on both base and new classes. Furthermore, our DOR method can be effectively integrated into CaRot and improve calibration on both base and new classes. For instance, DOR reduces the ECE by 1.14% and 1.21% on base and new classes of the OxfordPets dataset, respectively. This demonstrates that DOR is a flexible regularization strategy compatible with various fine-tuning methods.
J. Application on medical imaging
To verify our proposed DOR can be applied in real-world tasks, we conduct the experiments on PathMNIST from MedMNIST+ (Yang et al., 2023) as the medical benchmark. PathMNIST is comprised of types of tissues, resulting in multi-class classification task. For the text of each label,
the caption from the official implementation. Specifically, the dataset includes the following labels: adipose tissue (0), background (1), debris (2), lymphocytes (3), mucus (4), smooth muscle (5), normal colon mucosa (6), cancer-associated stroma (7), and colorectal adenocarcinoma epithelium (8). we use We fine-tune the CLIP with 16 shots from the first 5 classes and evaluate the model on all 9 classes under the base-to-new evaluation protocol.
As shown in Table 15 and 16, we find that DOR can effectively help with the calibration of fine-tuned CLIP on medical datasets. Noted that existing prompt tuning methods can be effectively applied medical image datasets. For instance, after fine-tuning, the accuracy of the base class improved significantly from 29.80% to over 85% for all methods. However, compared with standard benchmarks used in the main experiment, the calibration performance could be worse and output worse ECE for new classes To this end, DOR effectively reduces ECE across all methods. For example, DOR lowers the ECE for new classes from 61. 51.28% to 14.99% in CoOp. Moreover, DOR can fit existing advanced methods like CoPrompt and significantly reduces the overall ECE from 10.33% to 6.92%. In summary, DOR can notably improve the calibration performance of prompt-tuning methods and is capable of real-world domain -seciffc tasks.
K. Feature visualization of DOR-V
In the discussion (Section 6), we show that DOR-V can successfully reduce the calibration error for visual feature adaptation. To investigate how DOR influences the feature space of base classes when incorporating visual outliers, we visualized the performance of CLIP-Adapter on the DTD dataset. For a better view, we randomly selected 10 base classes for visualization. We denote DOR-V to our method combined with CL.P-Adapter.
22

---

Understanding and Mitigating Miscalibration in Prompt Tuning for Vision-Language Models
ACC-13.24%
ACC-R2 00%
ACC-R7. 18%
10
10
10
-10
-10
-20
-30
-20
-10
10 20
-30
-30
-10
20
30
-30
-30
-10
30
(a) ZS-CLIP
(b) CLIP-Adaptie
(c) DOR-V
Figure The 1-SNE plots visualizing visual features on the base classes dataset. DOR-V denotes our method applied on CLIP-Adapter. CLIP-Adapter generates morc discriminative features compared with ZS- ZS-CLI.. and DOR does significantly affect the visual feature space.
Table 19. Distribution similarity between visual features in Zero-Shot CLIP (Z), CLIP-Adapter (C), and CLIP-Adapter with DOR-V (D) on the DTD datasct.
Metric 7
7 D
C<>>
D
MMD 0.39
0.84
0.22
Wasserstein
1,48
1.01
0.47
We present the visualization in Figure 11. Compared to ZS- CLP, CLIP-Adapter generates more discriminative features. Importantly, we observe that DOR does not significantly affect the visual feature space and maintains accuracy on the base class. To further quantify the difference between visual distributions, we measure the distance between distributions via Maximum Mean Discrepancy (MMD) and Wasserstein distance. As shown in Table 19, compared with ZS-CLIP, the gap between CLIP- -Lipter and DOR-V relatively smaller. These results confirm that DOR does significantly affect the feature space and can achieve better calibration results evidenced in Table 7.
23


Reviews:
[{"review_id": "eNcOQBMeUP", "reviewer": "Reviewer_YQTf", "summary": "The authors propose a method called Dynamic Outlier Regularization (DOR) to improve confidence calibration for both base and new classes after fine-tuning. Extensive experiments are conducted on multiple benchmark datasets to evaluate its effectiveness.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the constructive feedback. Please find our response below:\n\n\n### 1. Visualization of DOR\nThanks for the great suggestion. We'd like to clarify that our method outperforms KgCoOp as our method does not fix the confidence level on the base classes while preventing the increase of textual divergence on new classes. In contrast, KgCoOp anchors the confidence level, leading to underconfidence on Base classes. To illustrate this, we provide new visualization in Figures 1-2 [[link](https://anonymous.4open.science/r/icml_rebuttal-5881/rebuttal.pdf)]. The results show that KgCoOp significantly decreases the FD scores on both Base and New classes, whereas our method primarily impacts New classes. Consequently, our approach provides confidence scores that align with the improved accuracy on Base classes. We believe the visualization suggested by the reviewer can enhance the clarity of this paper.\n\n\n### 2. Clarification on result inconsistency\n\nThank you for your thorough review. We have carefully revisited the CoOp performance reported in our manuscript and DAC. We found that the slight differences occur specifically on two datasets—FGVCAircraft and EuroST—while the results for the remaining nine datasets align closely with those in DAC. We attribute these minor variations to differences in experimental settings, such as the random seed, hardware specifications, or software environment, which can subtly affect model performance in certain cases. Notably, these minor variations do not impact the contribution of DOR, which largely improves the calibration performance. For reproduction, we will release the full code of baselines on GitHub after publication. \n\n\n\n### 3. Comparison between DOR and KG\n\nThank you for the great suggestion. We conduct new experiments by comparing DOR and KgCoOp on 3 baselines (CoOp, MaPLe, and CoPrompt), and present the results in the table below. The results show that integrating with our method can **consistently outperform** those with KgCoOp on overall performance. In particular, our method achieves much better performance than KgCoOp on Base classes, while KgCoOp performs well on New classes. This is consistent with the analysis presented in Subsection 3.2: KgCoOp anchors the confidence level, leading to underconfidence on Base classes. In short, the results demonstrate the superiority of DOR in breaking the calibration trade-off between base and novel classes.\n\n|          | Method  | Base | New  | HM    |\n|----------|---------|------|------|-------|\n| **CoOp**     | Vanilla | 3.07 | 14.49 | 8.78  |\n|          | +KG     | 5.82 | 4.48  | 5.15  |\n|          | +DOR    | 2.47 | 6.48  | **4.47** |\n| **MaPLe**    | Vanilla | 2.75 | 5.46  | 4.11  |\n|          | +KG     | 4.01 | 4.29  | 4.15  |\n|          | +DOR    | 3.06 | 4.26  | **3.66** |\n| **CoPrompt** | Vanilla | 2.60 | 5.96  | 4.28  |\n|          | +KG     | 4.01 | 4.99  | 4.50  |\n|          | +DOR    | 2.98 | 5.14  | **4.06** |\n\n\n\n### 4. Ablation on the size of the outlier set\n\nYes, we present the ablation study in `Appendix H2`. The results show that our method is not sensitive to the outlier set size, especially when the size is larger than 500. In particular, the impact of set size is negligible on Base classes, while a larger set size promotes better performance on New classes. In an extreme case with $k = 10$, our method can still significantly improve the performance on new classes. Due to the low cost of outlier texts, we recommend $K=5000$ as the default to achieve optimal calibration performance.\n\n\n\n### 5. Overlap between outlier texts and new classes\n\nThank you for raising the concern. In Table 8, we present all selected outlier texts for 6 datasets, showing almost no class overlap especially on StanfordCars and FGVCAircraft. Moreover, the ablation study in Table 6 demonstrates that using even far-OOD and random-OOD can significantly improve the overall calibration performance. To further analyze the impact of overlap, we conduct a new ablation study by excluding all new classes from the outlier pool (denoted as DOR$^\\dagger$). The average results on 11 datasets below show that DOR$^\\dagger$ without overlap achieves comparable performance to DOR, significantly improving the performance on all three baselines. **Therefore, the effectiveness of our method does not rely on the overlap with new classes**.\n\n\n\n| Method       | Variant       | Base  | New   | HM    |\n|--------------|---------------|-------|-------|-------|\n| **CoOp**     | Vanilla       | 3.07  | 14.58 | 8.83  |\n|              | +DOR          | 2.67  | 6.49  | 4.58 |\n|              | +DOR$^\\dagger$| 2.82  | 6.77  | 4.80  |\n| **MaPLe**    | Vanilla       | 2.75  | 5.46  | 4.11  |\n|              | +DOR          | 2.83  | 4.44  | 3.64 |\n|              | +DOR$^\\dagger$| 2.89  | 4.51  | 3.70  |\n| **CoPrompt** | Vanilla       | 2.56  | 5.96  | 4.26  |\n|              | +DOR          | 2.96  | 4.69  | 3.83  |\n|              | +DOR$^\\dagger$| 2.71  | 4.93  | 3.82 |", "claims_and_evidence": "One of the major claims is that DOR outperforms kGCoOp due to its ability to prevent the increase of textual divergence on new classes. To strengthen this claim, I suggest visualizing DOR in the same manner as Figure 2 and Figure 6 in the supplementary material.\nThis would better illustrate how DOR improves on textual divergence and solidify the contribution.", "methods_and_evaluation_criteria": "I am concerned about the fair comparison in the experimental demonstration:\n\n1. I noticed inconsistencies between the baseline results reported in the manuscript and those in [1]. For example, on new classes, in [1] CoOp reports ECE - 13.84, ACE - 13.76, MCE - 3.80, PIECE - 14.71; while in this paper they are ECE -14.58, ACE - 14.50, MCE - 3.73, PIECE - 15.27.\nCan the authors clarify why these numbers differ? If not justified, this could affect the validity of the comparison.\n\n2. The authors claim that DOR improves over kGCoOp by preventing textual divergence from increasing on new classes. To this end, to ensure a fair comparison, I believe the manuscript should compare various baselines+DOR vs. various baselines+kGCoOp, as done in Table 2 & 3 of [1].\n\n\n[1] Wang et al., Open-Vocabulary Calibration for Fine-tuned CLIP, ICML 2024", "theoretical_claims": "I think the theoretical claims are sound to me.", "experimental_designs_or_analyses": "Please see Methods And Evaluation.", "relation_to_broader_scientific_literature": "The authors provide a good coverage of the related work.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "I find the overall presentation flow to be clear and easy to follow. The method is also well-described.\n\nThe core idea is interesting, but it lacks solid support in certain areas, as I mentioned in Claims and Evidence & Methods and Evaluation Criteria. Strengthening these aspects would definitely enhance the technical contribution.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "One of the core components of the proposed method is the choice of textual outliers in Equation (7). I believe more discussion and experimentation on this part is necessary. \n\nAblation study on outlier set size:\n\n(1) Do the authors have an ablation study on how the size of the outlier set affects performance? Is the method sensitive to the outlier set size?\n\n(2) How similar is the sampled outlier set to the actual new classes? Because I feel like one possible reason this method is effective might be that the sampled outliers overlap with or are similar to new classes.\n\nI am open to raising my score if the authors can address these concerns.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AZz97sCUeP", "reviewer": "Reviewer_QPox", "summary": "1. This paper analyzes the trade-off between base and novel classes from the perspective of textual distribution divergence.\n\n2. This paper proposes a simple DOR regularization method, which can compile with existing prompt learning methods.\n\n3. Experimental results show promising performance compared with related methods.\n\n\n## update after rebuttal\nThanks for the author's responses, and I will maintain my original score.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### 1. Clarification on \"information leakage\"\n\nThanks for your insightful review. We believe this concern is closely related to the response #5 for Reviewer YQTf, which is about the overlap between outliers and new classes. We'd like to clarify that our method does not leak the information of new classes, as the outliers are selected with base classes. Moreover, the ablation study in Table 6 demonstrates that using even far-OOD and random-OOD can significantly improve the overall calibration performance. In the response #5 for Reviewer YQTf, we also provide an additional ablation study by removing the new classes from the outlier pool (e.g., WordNet). The results demonstrate that the effectiveness of our method does not rely on the overlap with new classes. Instead of causing \"information leakage\", our work opens up the possibility of utilizing cheap public texts for improving calibration performance. In summary, leverages general semantic information from language space rather than memorizing target classes, which ensures its fairness and generalizability. We will add this discussion to the final version.\n\n\n\n### 2. Results on PromptKD\n\nThank you for the great suggestion. we provide new results by incorporating DOR into a recent method - PromptKD. In particular, we apply DOR in the first stage to train a large teacher model(ViT-L-14), and keep the second stage unchanged. The results below show that our method can achieve meaningful improvements over PromptKD despite its strong performance.\n\n| Method   | Base       | New        | HM          |\n|----------|------------|------------|-------------|\n| PromptKD | 4.73±0.36  | 4.38±0.53  | 4.56±0.45   |\n| +DOR     | 4.81±0.23  | **3.66±0.43**  | **4.24±0.33**   |\n\n\n### 3. Performance decrease on base classes\n\nThank you for the careful review. We'd like to clarify that our method achieves comparable accuracy to baselines on base classes. To illustrate this, we provide the accuracy with standard deviation in the table below. In particular, the performance gaps are negligible with most of them are less than 0.2%. We conjecture that adding regularization may slightly affect the training dynamic during the prompt tuning, leading to trivial changes on the accuracy (also increasing a little on new classes). Overall, our method can maintain the generalization performance of baselines on Base classes.   \n\n\n|         | CoOp      | CoCoOp    | KgCoOp    | MaPLe     | DEPT      | TCP       | CoPrompt  | PromptSRC | PromptKD  |\n|---------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|\n| Vanilla | 82.97±0.56| 80.57±0.47| 82.29±0.25| 82.11±0.54| 83.70±0.32| 83.95±0.40| 82.32±0.51| 84.77±0.29| 85.74±0.35|\n| +DOR    | 83.20±0.47| 79.89±0.57| 82.13±0.35| 82.08±0.50| 83.81±0.47| 83.89±0.29| 82.39±0.57| 84.79±0.34| 85.52±0.36|\n\n### 4. Visualization on DOR\n\n\nThank you for the great suggestion. we provide new visualization in Figures 1-2 [[link](https://anonymous.4open.science/r/icml_rebuttal-5881/rebuttal.pdf)]. The results show that KgCoOp significantly decreases the FD scores on both Base and New classes, whereas our method primarily impacts New classes. Consequently, our approach provides confidence scores that align with the improved accuracy on Base classes. We believe the visualization suggested by the reviewer can enhance the clarity of this paper.", "claims_and_evidence": "The claims about the calibrating are basically supported by experimental results and theoretical analysis.", "methods_and_evaluation_criteria": "The used benchmark datasets are reasonable in prompt learning fields.", "theoretical_claims": "This paper provides a theoretical analysis of the feature divergence and miscalibration, and gives the theoretical justification in the supplementary material.", "experimental_designs_or_analyses": "The experimental design is basically reasonable.", "relation_to_broader_scientific_literature": "This paper gives insight into the miscalibration in existing prompt learning methods, which has not been discussed and digged before.\nMeanwhile, the calibration idea can be further explored in this field, so I think it contributes to the broader scientific literature to some degree.", "essential_references_not_discussed": "More recent prompt learning methods should be analyzed and discussed, such as \"Gallop\" \"PromptKD\" and more.", "other_strengths_and_weaknesses": "Strengths:\n\n1. The idea of calibration is reasonable, which has not been explored enough in the field of prompt learning before.\n\n2. This paper gives the theoretical analysis from feature divergence to miscalibration along with the trade-off between base and novel classes.\n\n3. The designed method is simple, and the results are promising.\n\nWeaknesses:\n\nPlease refer to \"Questions For Authors\".", "other_comments_or_suggestions": "None.", "questions_for_authors": "1.  In my opinion, this paper uses the extra-textual information to mimic the new classes, through the similarity connection and existing word datasets. So, maybe it is not fair to compare with other methods directly to some degree, considering the \"information leakage\". This paper should consider this part carefully.\n\n2. More recent prompt learning methods, such as Gallop and PromptKD, should be equipped with DOR, to show the performance. In fact, PromptKD also involves extra information, so the performance of PromptKD with DOR will give some insights and discussions.\n\n3. The performance change should be discussed. As shown in Table 3, DOR decreases the base classes' performance in some methods, and what are the reasons?\n\n4. Some visualization results about outliers or methods with/without DOR are helpful for understanding the effect of the proposed method.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PJMlA3kl5X", "reviewer": "Reviewer_MfZm", "summary": "This paper identifies a calibration trade-off in existing prompt tuning methods: standard tuning (e.g., CoOp) leads to overconfidence on new classes due to increased textual label divergence, while regularization-based tuning (e.g., KgCoOp) results in underconfidence on base classes despite improved accuracy. To address this, the authors propose Dynamic Outlier Regularization (DOR), a method that regulates textual divergence of novel classes using dynamically sampled textual outliers from a large vocabulary (e.g., WordNet), without restricting base class features.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your helpful and positive feedback. Our response is below:\n\n### 1. Theoretical analysis on multi-class settings\n\nThank you for the insightful suggestion. We presented the theoretical results in binary classification to help readers quickly understand the relationship between feature divergence and output confidence. Here, we draft a theoretical analysis in the multi-class setting. In particular, a set of logits $\\\\{z_j\\\\}_ {j=1}^K$ sampled from a Gaussian distribution $\\mathcal{N}(\\mu, \\sigma^2)$ in $K$-class classification and maximum softmax probability is denoted by $p_{\\max} = \\max_j \\frac{e^{z_j}}{\\sum_k e^{z_k}}$. Now, we prove that the expected maximum softmax probability $\\mathbb{E}[p_{\\max}]$ increases strictly with $\\sigma$.\n\nSimilar to our former analysis, we assume $\\mu = 0$ without loss of generality. We then standardized the logits as $z_j = \\sigma u_j$, with $u_j \\sim \\mathcal{N}(0, 1)$ i.i.d., so $p_{\\max} = \\max_{1 \\leq j \\leq K} \\frac{e^{\\sigma u_j}}{\\sum_{k=1}^K e^{\\sigma u_k}}$. To analyze the monotonicity of $\\mathbb{E}[p_{\\max}]$ with respect to $\\sigma$, we compute the derivative $\\frac{d}{d\\sigma} \\mathbb{E}[p_{\\max}]$. By using the Dominated Convergence Theorem, we interchange differentiation and expectation. The derivative simplifies to $\\frac{d}{d\\sigma} \\mathbb{E}[p_{\\max}] = \\mathbb{E} \\left[ p_m \\left( u_m - \\sum_{k=1}^K p_k u_k \\right) \\right]$, where $m = \\arg\\max_j p_j$. We ensure this step is valid by bounding the derivative $\\left| \\frac{\\partial p_m}{\\partial \\sigma} \\right| \\leq 2 \\max_j |u_j|$, which has a finite expectation for fixed $K$. Now, since $u_m = \\max_j u_j$ and $e^{\\sigma u_j}$ is increasing in $u_j$, we have $u_m - \\sum_{k=1}^K p_k u_k = \\sum_{k \\neq m} p_k (u_m - u_k)$. Since $u_m > u_k$ for $k \\neq m$ almost surely, and $\\sum_{k \\neq m} p_k = 1 - p_m > 0$, this expression is strictly positive. Given $p_m > 0$, the expectation $\\mathbb{E} \\left[ p_m \\left( u_m - \\sum_{k=1}^K p_k u_k \\right) \\right]$ is positive, implying $\\frac{d}{d\\sigma} \\mathbb{E}[p_{\\max}] > 0$. Hence, for $\\sigma_2 > \\sigma_1$, we conclude $\\mathbb{E}[p_{\\sigma_2}] > \\mathbb{E}[p_{\\sigma_1}]$ in the multi-class setting. We will incorporate this detailed analysis into the final manuscript.\n\n\n\n\n### 2. Minor improvements on some datasets\n\nThank you for raising the concern. We also identify that the severity of the calibration problem differs on various datasets: current prompt-tuning methods are miscalibrated on some datasets (e.g., DTD, EuroSAT, and FGVCAircraft) and well-calibrated on some other datasets (e.g., Caltech101 and ImageNet). We conjecture that the calibration performance might be relevant to the distribution distances between the pretraining dataset of CLIP and downstream datasets (Larger distances may cause worse calibration). Notably, our method can consistently improve the calibration performance on various datasets, especially significant on those challenging cases.\n\n\n\n\n### 3. Add standard deviation to results\n\nThanks for your valuable suggestion. We present the main results with standard deviation in Table 1 of [[link](https://anonymous.4open.science/r/icml_rebuttal-5881/rebuttal.pdf)]. The results confirm the significance of the improvement from our method.\n\n\n### 4. The choice of cosine similarity\n\nWe use the cosine similarity since it is widely adopted in the literature of CLIP. In addition, the features of CLIP are typically normalized so that using either Euclidean distance or cosine distance achieves similar impacts. In particular, the squared Euclidean distance is proportional to the cosine distance: $||\\mathbf{x} - \\mathbf{y}||_2^2 = 2 - 2 \\cos \\langle \\mathbf{x}, \\mathbf{y} \\rangle$. Therefore, we default to using the cosine similarity due to its popularity in vision-language models.", "claims_and_evidence": "The primary claim—that existing prompt tuning methods compromise calibration on either base or new classes—is backed by empirical analysis on datasets like StanfordCars and UCF101, showing CoOp’s overconfidence (ECE 14.58% on new classes) and KgCoOp’s underconfidence (ECE 5.82% on base classes). The explanation via textual feature divergence is substantiated with Feature Divergence (FD) scores and logit gap visualizations (e.g., Figures 2 and 3). The claim that DOR mitigates this trade-off is supported by comprehensive results.", "methods_and_evaluation_criteria": "The proposed DOR method—minimizing textual feature discrepancy of dynamically sampled outliers—is sensible for addressing calibration in VLMs, as it targets the identified cause (textual divergence) without altering core fine-tuning objectives.", "theoretical_claims": "The paper includes a theoretical justification in Appendix C, linking textual divergence to confidence via logit variance in a binary classification setting (Proposition C.1). I checked the proof, which assumes logits follow a normal distribution and shows that higher variance (σ) increases expected maximum softmax probability.", "experimental_designs_or_analyses": "I reviewed the experimental designs and analyses, focusing on Sections 5.1-5.2 and Appendices F-J. The results are convincing.", "relation_to_broader_scientific_literature": "The paper builds on prior work in VLM fine-tuning and calibration. It extends CoOp, KgCoOp and more by addressing their calibration limitations, aligning with findings in existing works on novel class miscalibration.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strength\n- The paper is original in combining dynamic outlier regularization with prompt tuning to break the calibration trade-off, a novel synthesis of ideas. \n- Its significance lies in improving VLM reliability for safety-critical applications (e.g., medical diagnosis), and the clarity of writing, figures (e.g., Figure 1), and tables enhance accessibility. T\n- he extensive evaluation across 11 datasets and 4 ImageNet variants is a major strength.\n\nWeakness\n- The theoretical analysis is limited to binary classification, reducing its generalizability to multi-class settings. \n- Method shows minor improvements on some of the dataset \n- Results do not include standard deviation over multiple runs, making it difficult to understand the significance of the improvement", "other_comments_or_suggestions": "N/A", "questions_for_authors": "- The in Appendix C assumes binary classification. How do the authors expect textual divergence to influence confidence in multi-class settings with more complex logit distributions? \n- Table 10 shows Euclidean distance slightly outperforms cosine similarity. Why do the authors prefer cosine as the default, beyond convention? An explanation (e.g., stability, alignment with CLIP’s training) could strengthen the method’s justification.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ei2JWVJ0va", "reviewer": "Reviewer_Miv5", "summary": "This paper investigates miscalibration issues in fine-tuned vision-language models (VLMs) like CLIP, revealing a trade-off between base and new classes where standard prompt tuning (e.g., CoOp) leads to overconfidence in new classes, while regularization-based methods (e.g., KgCoOp) cause underconfidence in base classes​\n. To address this, the authors propose Dynamic Outlier Regularization (DOR), which samples textual outliers from WordNet and minimizes their feature deviation before and after fine-tuning, preventing excessive divergence in new class representations while maintaining base class accuracy​\n. Extensive experiments across multiple datasets demonstrate that DOR significantly improves calibration without compromising model performance, and the technique can also be extended to visual fine-tuning methods​.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful and positive suggestions. Here’s our response below:\n\n### 1. The choice of Near-Outliers or Random-Outliers\n\nWe agree that random outliers offer a practical and efficient approach for real-world out-of-distribution (OOD) applications. To investigate their feasibility, we conduct a detailed analysis of when random outliers are sufficiently effective. We hypothesize that the performance gap between near OOD and random OOD may depend on the task’s scope. Accordingly, we compare them across datasets of varying breadth: ImageNet, spanning diverse object classes, versus the fine-grained DTD (textures) and Flowers (flowers). In the table below, our results reveal a larger gap between random and near OOD on ImageNet compared to the two fine-grained datasets, suggesting that random outliers may be particularly viable for tasks with broad and diverse categories in our method.\n\n \n | Method  | DTD   | Flowers | ImageNet |\n|---------|-------|---------|----------|\n| Near    | 8.58  | 6.12    | 1.64     |\n| Random  | 9.54  | 6.61    | 1.66     |\n| Δ(Gap)  | 0.96  | 0.49    | 0.02     |", "claims_and_evidence": "The claims made in the paper are **generally well-supported** by evidence:\n\n**1. Identification of Calibration Trade-Off in Fine-Tuned CLIP**\nClaim: Fine-tuning CLIP with prompt tuning (e.g., CoOp) leads to overconfidence in new classes, whereas regularization-based tuning (e.g., KgCoOp) results in underconfidence in base classes.\nEvidence:\nEmpirical Analysis (Section 3.1 & 3.2): The authors analyze textual feature divergence (FD score) and show that CoOp increases FD, causing overconfidence in new classes, while KgCoOp reduces FD, leading to underconfidence in base classes.\n\n**2. Effectiveness of Dynamic Outlier Regularization (DOR)**\nClaim: DOR mitigates the calibration trade-off by sampling textual outliers from WordNet and minimizing their feature deviation, improving both base and new class calibration.\nEvidence:\nTable 1 & Table 2: Show that DOR consistently reduces ECE across 11 datasets when applied to various tuning methods (CoOp, CoCoOp, MaPLe, etc.).\n\n**3. Generalization of DOR to Visual Fine-Tuning (DOR-V)**\nClaim: DOR can be extended to visual fine-tuning by applying a similar regularization to image representations (DOR-V).\nEvidence:\nTable 5: Shows that DOR-V improves calibration in visual fine-tuning methods (VPT and CLIP-adapter) across multiple datasets.", "methods_and_evaluation_criteria": "Basically, the authors sample new outliers to increase the diversity of text prompts (without making them exactly the same as the original prompts). This introduces an appropriate constraint during model optimization, enabling the fine-tuned model to **reduce overconfidence on new classes** when encountering them, thereby enhancing its generalization ability. Overall, the approach **makes sense**.", "theoretical_claims": "I have checked **Appendix C. Theoretical Justification**, and it looks correct.", "experimental_designs_or_analyses": "The authors conducted evaluations using the **two standard evaluation protocols**, **Base-to-New Generalization** and **Domain Generalization**, while comparing against **comprehensive and recent baselines**, such as **MaPLe and DEPT**. The **calibration metrics** used are also comprehensive, overall, the **experimental design and analysis are effective**.", "relation_to_broader_scientific_literature": "Miscalibration is an overlooked problem in the field of **prompt tuning**, and this work proposes an effective method to address it.", "essential_references_not_discussed": "Not Discovered.", "other_strengths_and_weaknesses": "Clear motivation, reasonable methodology, effective experiments, a well-structured paper.", "other_comments_or_suggestions": "None", "questions_for_authors": "In your experiments, you mentioned that **randomly selecting outliers** also provides a **strong baseline**. Does this suggest a **valuable insight** for real-world **OOD applications**? How do you evaluate the improvement in **generalization performance** between **Near-Outliers and Random-Outliers** in practical applications?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper received four weak accepts. In the pre-rebuttal phase, the reviewers raised following points:

- fair comparison in some cases
- integration with more recent prompt learning methods
- possible information leakage
- theoretical analyses only for single class settings
- small improvement in some instances.

After the rebuttal stage, reviewers acknowledged the rebuttal and agree that their concerns have been resolved adequately. The post-rebuttal discussion left no major concerns. Some reviewers also appreciated the possible broader significance of the work. Therefore, the decision is to recommend acceptance of the paper. Authors are encouraged to include important comments from reviewers in the final verision.
Decision: Accept (poster)