C-TPT: Calibrated Test-Time Prompt Tuning for Vision-Language Models via Text Feature Dispersion
Authors: Hee Suk Yoon
Eunseop Yoon
Joshua Tian Jin Tee
Mark A. Hasegawa-Johnson
Yingzhen Li
Chang D. Yoo
Venue: ICLR 2024 poster (2024)
Keywords: Calibration
Test-time adaptation
CLIP
Prompt tuning
Primary area: societal considerations including fairness, safety, privacy
TL;DR: We address the critical yet under-explored challenge of achieving calibrated zero-shot inference during test-time prompt tuning in large-scale vision-language models.

Abstract: In deep learning, test-time adaptation has gained attention as a method for model fine-tuning without the need for labeled data. A prime exemplification is the recently proposed test-time prompt tuning for large-scale vision-language models such as CLIP. Unfortunately, these prompts have been mainly developed to improve accuracy, overlooking the importance of calibration, which is a crucial aspect for quantifying prediction uncertainty. However, traditional calibration methods rely on substantial amounts of labeled data, making them impractical for test-time scenarios. To this end, this paper explores calibration during test-time prompt tuning by leveraging the inherent properties of CLIP. Through a series of observations, we find that the prompt choice significantly affects the calibration in CLIP, where the prompts leading to higher text feature dispersion result in better-calibrated predictions. Introducing the Average Text Feature Dispersion (ATFD), we establish its relationship with calibration error and present a novel method, Calibrated Test-time Prompt Tuning (C-TPT), for optimizing prompts during test-time with enhanced calibration. Through extensive experiments on different CLIP architectures and datasets, we show that C-TPT can effectively improve the calibration of test-time prompt tuning without needing labeled data. The code is publicly accessible at https://github.com/hee-suk-yoon/C-TPT.

Published as a conference paper at ICLR 2024
C-TPT: CALIBRATED TEST-TIME PROMPT TUNING FOR VISION-LANGUAGE MODELS VIA TEXT FEATURE DIS- PERSION
Hee Suk Yoon!* Eunseop Yoon
Joshua Tian Jin Tee*
Chang D. Yoo't 1 Korea Advanced Institute of Science and Technology (KAIST)
Mark Hasegawa-Johnsonn
Yingzhen Li
University of Illinois at Urbana- TarbanaChhamaighe (UTUC) Imperial College London
(hskyoon, esyoon97, joshuateet ed loa.sooooo..aaa.....aa..
Shasegawe il wllliniis..eeeeeeee edu yingzhen. weiieerral.ac...
ABSTRACT
In deep learning, test-time adaptation has gained attention as a method for model fine-tuning without the need for labeled data. A prime exemplification is the re- cently proposed test-time prompt tuning for large-scale vision-language models such as CLIP. Unfortunately, these prompts have been mainly developed improve accuracy, overlooking the importance of calibration. which is a crucial aspect for quantifying prediction uncertainty. However, traditional calibration methods rely substantial amounts of labeled data, making them impractical for test-time sce- narios. To this end, this paper explores calibration during test-time prompt tuning by leveraging the inherent properties of CLIP. Through series of observations, we find that the prompt choice significantly affects the calibration in CLIP, where the prompts leading to higher text feature dispersion result in better-calibrated predic- tions. Introducing the Average Text Feature Dispersion (ATFD), we establish its relationship with calibration error and present a novel method, Calibrated Test-time Prompt Tuning (C-TPT), for optimizing prompts during test-time with enhanced calibration. Through extensive experiments on different CL.IP architectures and datasets, we show that C-TPT can effectively improve the calibration of test-time prompt tuning without needing labeled data. The code publicly accessible at
INTRODUCTION
Pre-trained large-scale vision-language models, such as CLIP (Radford et al. 2021). have demon- strated the potential as foundation models by leveraging their zero-shot inference capabilities. These models are pre-trained on a massive dataset of image-text caption pairs and learn to associate the image and its corresponding text caption in shared latent space, which allows the model to accu- rately classify newfound visual categories in zero-shot manner based on carefully designed prompt templates. hand-crafted prompts consisting of predefined vocabulary tokens (i.e., hard prompts) may not be optimal, significant attention is being directed towards prompt tuning, which treats the prompts as learnable vectors that can be optimized through gradient descent. Specifically, Test-time Prompt Tuning (TPT) (Manli et al. 2022) adaptively refines the prompt for individual unlabeled image sample. In line with well-known rest-time adaptation methods for classifications (Wang et al.) 2021 Zhang al.) 2022). TPT aims enhance the accuracy of CLIP models by minimizing the entropy in the prediction distribution as a self-supervision signal during test time.
However, reduction in entropy leads the model to generate overconfident predictions, which is characteristic often observed in models trained with cross-entropy loss (Guo er al.) [017] Mukhoti al. 2020 Eom al. (0224). This overconfidence is intrinsically linked to worsening the model's calibration a property that evaluates the degree to which the predicted probabilities align with the
"Equal contribution 1 "Corrsponning Author

---

Published as a conference paper at ICLR 2024
true underlying probability distribution (Guo et al. 2017). For instance, if a perfectly calibrated classifier assigns a confidence of 0.8 to its predictions, it should be correct 80% of the time. This
property is particularly crucial in real-world applications where knowing the prediction uncertainty can ensure the trustworthiness and safety of the model. Although CLIP is increasingly employed in
decision-making applications, such as healthcare diagnostics (Wang al. [022] Zhang al. [022
Chen et al. [023aa Liu et al. (023) and autonomous vehicles (Dorbala et al. 2022 Gadre et al. 2022
Khandelwal al.] 2022 Bucker al.), calibration has been largely overlooked in existing
CLIP prompt tuning literature, which has primarily focused on enhancing the classification accuracy.
This paper addresses this critical yet under-explored challenge of calibrated prompt tuning in large- scale ussion-langue models. Specifically, in light of the recent success of test-time prompt tuning on enhancing accuracy without labeled data (Manli et al. 2022), we aim accomplish calibration during test-fime prompt tuning to mitigate the adverse scenario where the prompt optimization, although enhancing accuracy, results in poor calibration. This may seem infeasible since various calibration techniques employed in standard supervised training of neural networks require substantial amounts of labeled training data, which restricts their applicability in test-time prompt tuning scenarios for models like CLIP. Instead, to enable calibration without labeled data, we capitalize the intrinsic structures and properties of CLIP. In detail, our contributions can be summarized as follows:
Through a series of observations, this paper reveals that the calibration of CLIP models is signifi- cantly influenced by the prompt choice, with certain prompts offering superior calibration with the same prediction accuracy level. We identify that the critical difference between these prompts can be characterized by the distribution of the class-embedded text features, with a noticeable negative correlation between the dispersion of the text features and the calibration error.
This paper defines the Average Text Feature Dispersion (ATFD) and quantitatively establishes its strong negative correlation with Expected Calibration Error (ECE), reinforeing our finding of the relationship between the text feature dispersion and the calibration error.
This paper introduces the Calibrated Test-time Prompt Tuning (C-TPT), which is used to jointly optimize the prompt during test time to achieve better calibration by maximizing ATFD. Extensive experiments demonstrate that across various datasets and CLIP models, incorporating C-TPT allows improved test-time optimization of the prompts, resulting in better-calibrated predictions.
2 RELATED WORK
Prompt Tuning for Vision-Language Models Pre-trained vision-language models like CLIP (Radford et al.) (021) excel in zero-shot inference by learning to associate images and captions created using prompt templates. Prompt tuning treats the prompt templates as learnable vectors that can be optimized by gradient descent. For instance, CoOp (Zhou et al. 2022a) tunes the prompt in CLIP using a dataset of labeled training samples to improve its classification accuracy. However, CoCoOp (Zhou al. 2022b) identifies that CoOp struggles with generalizing to auttof-distribution data and recommends conditioning the prompt on input images. While effective, these methods require access annotated training data, which limits the zero-shot utilization of pre-trained models. To tackle this challenge, recent research has introduced Test-time Prompt Tuning (TPT) (Manli et al. 2022), method that enables adaptive prompt learning on the fly using just one test sample.
Calibration of Neural Network Calibration of neural networks measures the extent which its prediction aligns with the true probability distribution (Guo et al., 2017). Among various methods for achieving better calibration. post-progssiing calibration strategies Guo al. 2017 Platt 1999
Zadrozny and Elkan 2001] [002] Vook et al. 2005 Lei et al. 2018 Pakdaman Nacini et al. 2015a) have been employed to calibrate an already trained model using a held-out validation dataset. Notable examples include temperature scaling (Guo et al., 2017). Platt scaling (Plati) 1999). and conformal prediction (Vokk et al. 2005 Lei et al. 2018). Alternatively, trainable calibration methods leverage hybrid objective during the training of a neural network by combining a primary training loss with an auxiliary calibration objective loss. In this context, MMCE (Kumar et al., 2018). SB-ECE, S-AvUC (Karandikar et al.) 22021), and ESD (Yoon et al.] 2023a) offer differentiable calibration objective loss that can be used in the training process. Also, involving mixup augmentation (Zhang eet al. 2018 Yoon al. 2022 during training has been shown improve the calibration of the model
2

---

Published as a conference paper at ICLR 2024
(Tulasidaaaan et al., 2019 Wang et al. 2023). These methods often target calibration error reduction at the expense of a minor compromise in accuracy - typically less than 1 to 2% degradation.
However, the direct application of these widely established calibration methods shares common limitation: a reliance on substantial amounts of labeled data, which restricts their applicability in few-shot test-time prompt tuning scenarios in CLIP. To address these challenges, our work proposes a calibration technique tailored for prompt tuning in CLIP without labeled data.
3 BACKGROUND AND PROBLEM SETUP
Zero-Shot Classification using Lisgua-L Model (CLIP) The structure of CLIP consists of two separate encoders: visual encoder F Viiuual for transforming the image input into a visual feature vector and text encoder F Fux for transforming textual inputs into text feature vectors. Suppose we have single test image, Xees, belonging to class Yees for classification problem involving N distinet classes. In the fundamental zero-shot scenario with CLIP, we attach a manually designed prompt prefix (e.g./P P ="a photo of a") to each possible class y. of Y E 3 {y1, 1/2,- .., UN), generating class-related textual descriptions [p: y.]. Next, these class descriptions are fed into the text encoder to produce text features (Mp::)) |p:val ................................ U/paw)). where Up:as)
Fixu( ([p; y/l), and the test image is fed into the visual encoder to produce image feature v = F Vvisual (Xiest). The image feature is paired with each text feature [p:y.] to determine a similarity score 8; = s v), where sim(-) refers to the cosine similarity. The probability of predicting class y/ for the test image Xust can be exp(sinn(fe.,,.))/11
described as p(yil X(es) = E. (pp where T is the temperature of Softmax (set to 0.01
during inference). Then, the predicted class becomes y arg maxy, ply Xies) with its associated confidence as P = maxg, p(y:) Xwe)
Prompt Tuning for CLIP Instead of hand-crafted prompts (i.e., hard prompts), prompt tuning has been adopted in various domains (Lester et al. 2022 Jia et al. 2022 Yoon et al. 2023b Koo et al., 2024), which treat prompts as trainable embeddings, allowing optimization by gradient descent to maximize performance. Specifically, prior research on prompt tuning for CLIP (Zhou et al. 2022a b
Chen al. 2023b Hantao Yao, 2023) tunes the prompt p RLXD in the text embedding space, where L. represents the number of tokens and D is the embedding size, using 16 samples per class from ImageNet dataset. These studies have demonstrated that the learned prompt can be generalized to different datasets. Recently, Test-time Prompt Tuning (TPT) (Manli et al. (022) has emerged as an alternative approach that enables prompt optimization without requiring labeled data, achieving performance on par with traditional prompt tuning methods that rely on labeled training data.
Calibration Error and Metric Consider an input X with its corresponding label Y. For the input X to a classifier, Y is the class prediction and f is its associated confidence. Formally, we define a classifier as perfectly calibrated when
IP = Y|P = p. Vp [0,
(I)
In essence, this means that if classifier assigns confidence of p to its predictions, then ideally, they should be correct p proportion of the time. This property of calibration is orthogonal to accuracy, where the property of a classifier's 's accuracy cannot guarantee its calibration and vice versa. Mathematically, calibration can be quantified using metrics such as the Expected Calibration Error (ECE) (Pakdaman Naeini al. 2015b). ECE is binning-basee based metric that quantifies the calibration of a model's predicted probabilities. It works by dividing the confidence of predictions into equally sized bins and then measuring the discrepancy between the predicted and true probabilities within each bin. Given a set of images, their corresponding ground-trute class labels, and model predictions, the ECE can be computed as follows:
[Bxl m
ECE Σ
Jaco( - conf(Bx))s
where K is the total number of bins, B& represents the set of images, Bg denotes the number of images, acc(B,) is the accuracy of the predictions, and conf(B;) is the average prediction confidence, all respectively associated with bin k. A lower ECE value indicates better calibration, as the predicted probabilities more closely aligned with the true probabilities of the prediction being correct.
3

---

Published as a conference paper at ICLR 2024
CLIP
CLIP + TPT
Apc DC12
Nrc. (TH 6448
(7
Apc: A2
(
2.21
DCED 8:35
Confidece
Confileree
Accenacz (1)
(4) Observation 1
(b) Observation
(c) Observation 3
Figure 1: Observations. (The plots are based on the CLIP- ILT--VVT-B/16 on the StanfordCars dataset.) (a) Observation shows the Reliability Diagrams (Guo al. 2017) of the prediction made with the hard prompt template ('an example of [class]') (left) and after applying TPT (right). The diagrams highlight the negative impact of TPT on calibration due to overconfident predictions. (b) Observation 2 demonstrates the varying calibration error (i.e., ECE), although similar accuracy, plotted using 80 different hard prompt templates. (c) Observation features a t-SNE visualization of text feature clustering patterns of different prompts with similar accuracy but different ECE, suggesting that text feature dispersion has a strong relationship with the calibration error of CLIP.
REVISITING THE CALIBRATION OF CLIP MODELS
In this section, we introduce a series of observations revealing the intrinsic properties of CLIP which allows us to propose a calibration technique tailored for prompt tuning in CLIP without labeled data.
Observation 1: Test-Time Prompt Tuning (TPT) Increases the Calibration Error. Previous
literature on "test-time adaptation" tries to improve accuracy by adapting the model to increase its prediction confidence on unlabeled target data (Mummadi et al. 2021). One of the mechanisms to achieve this is by employing entropy minimization loss (Wang al. 2021 Zhang et al.,, 2022) that uses maximization of the prediction confidence as a self-supervision signal during test-time adaptation. However, models trained with such loss are prone to overconfidence, which is one of the direct causes of calibration error (Guo et al. 2017; Mukhoti et al. 2020). We observe a parallel trend with Test-time prompt tuning (TPT) (Manli al. (022). which adopts a similar objective of entropy minimization during test-time to improve the classification accuracy of zero-shot inference using CLIP. As depicted Figure (a), while applying TPT successfully enhances the accuracy over hard prompts, there is corresponding trend of increase in calibration error.
Observation 2: Prompt Sensitivity in Calibration of CLIP Models. The following observation
begins by comparing the accuracy and calibration error (i.e., ECE) of various hard prompts to better comprehend the impact of the 'prompt' on the performance of CLIP. Figure[[.(b shows the accuracy and ECE of zero-shot inference of CLIP using different hard prompt templates A Although previous literature suggests that CL.IP models are generally cell-calibrrted (Minderer et al., 2021 Galil et al.), we discover that different prompts can yield notable variation in calibration performance while exhibiting similar accuracy (additional details in Appendix A.44). This observation signifies that certain prompts could have better-calibrated predictions with similar accuracy levels, and thus, is possible to guide the prompt optimization process toward better calibration.
Observation 3: Well-calibrated Prompts have High Text-Feature Dispersion. Following the results from Observation 2, we further investigated why particular prompts give better-calibrated predictions than others, even when the accuracy level is similar. Recognizing that the image feature does not change for different prompts during classification using CLIP, we directed our attention toward the lass-edbedded text features. We visualized the t-SNE (van der Maaten 2014) of text features from prompts with comparable accuracy, as depicted in Figure (c). and identified a distinct pattern; poorly calibrated prompts typically led text features cluster cohesively. In contrast, well- calibrated prompts manifested a wider dispersion in the text feature space. This insight suggests that promoting text feature dispersion could be a guiding regularizer for test-time optimization, potentially mitigating our observed calibration issues in Observation 1.
'The full list of prompt templates used can be found in Appendix A.3
4

---

Published as a conference paper at ICLR 2024
0.0
Can
0.00
Calochhii
Foodid1
SUNDO
0.90
048
0.4
ECE
ICE
(A) CLIP-RNSO
(b) VIE--VI
Figure 2: Plot illustrating the correlation between ECE and ATFD for hard prompts that achieve accuracies within 3% of the highest accuracy observed for each dataset. A notable negative association observed for CLIP-RN5I and CLIP- ViT-B/16 across different datasets, with Pearson correlation coefficients (Freedman et al. 2007) averaging -0.70 and -0.76, respectively.
C-TPT: CALIBRATED TEST-TIME PROMPT TUNING
Motivated by our observations in Section 4
we introduce the concept of Average Text Feature Dispersion (ATFD) and examine its relationship with the calibration of CLIP in Section 5.1
Finally,
we introduce our proposed Calibrated Test-Time Prompt Tuning (C-TPT) in Section 5.2
5.1 CORRELATION BETWEEN CALIBRATION AND TEXT FEATURE DISPERSION
Observation 3 highlighted that well-calibrated prompts exhibit wider dispersion of class-embedded text features. To quantitatively capture this dispersion for a given prompt p. we first compute the centroid of the text features associated with each class description (i.e., [pws): (paw)...... [pwxx]> "ppx))
1
Centroii
Following this, we evaluate the spread of the text features by determining the mean 1.2 distance between this centroid and each individual text feature. This measure, termed Average Text Feature Dispersion (ATFD), is defined as:
AtFD(tipan) (p:n)) ((:va)) ..... (PPN)))
(ocatroid
(4)
N
1=1
smaller ATFD indicates that the text features are closely clustered in the feature space, whereas larger ATFD suggests more dispersed distribution of the text features. Across multiple datasets, we conducted an evaluation using various hard prompts (Appendix A..3.. calculating their respective accuracy and ECE. Subsequently, rigorously understand the interplay between ATFD and ECE, we collated the prompts that yield similar accuracy, allowing for a more focused comparison. Our comprehensive analysis showed notable negative correlation between ATFD and ECE across different CLIP models and datasets, as illustrated Figure 2 This quantitative analysis reinforces our finding where the prompts leading enhanced calibration are associated with more dispersed distribution of text features across the possible classes.
5.2 C-TPT: CALIBRATED TEST-TIME PROMPT TUNING
Using our findings of the strong correlation between calibration and the Average Text Feature Dispersion (ATFD), we introduce Calibrated Test-time Prompt Tuning (C-TPT), which can be used to jointly train the prompt with tuning methods devised for accuracy (Figure Bp. C-TPT focuses on tuning the prompt during test time to enhance calibration by spreading the text description features each possible class by maximizing ATFD. Thus, our objective formulated as:
p" = arg miinnmmr + X LCTPT [[pm))) [paa): ... (pax)))
(5)
where LCTPT ATFD and [ fpax2) ... (paw) the class-emhedded text representations.
5

---

Published as a conference paper at ICLR 2024
Calibrated Prompt Tuning
ATFD
Maximization
[class
Text Encoder
Tunable Prompt
Inference
Back Propagation
Learnable Parameters
Image Encoder
Figure 3: Illustration of the Calibrated Test-time Prompt Tuning (C-TPT) for zero-shot image classification using CLIP. C-TPT improves calibration by optimizing the prompt so that it maximizes the Average Text Feature Dispersion (ATFD) during test-time prompt tuning.
EXPERIMENTS
This section presents the benchmarks employed for evaluating approach, discusses the implemen- tation specifics, and the results of the experiments. In line with prior works on the prompt tuning of vision-language models (Zhou al., 2022a Chen al. (203b Manli al. (022), evaluation focuses on I) multiple fine-grained classification and 2) natural distribution shift.
6.1 FINE-GRAINED CLASSIFICATION
Dataset The evaluation is carried out across 11 diverse datasets for the fine- ane-grained classification. These datasets include fine-grained classification of plants and animals (Flower 102, OxfordPets), scene recognition (SUN397), texture identification (DTD), food categorization (Food101), transporta- tion classification (StanfordCars, Aircraft), human action recognition (UCF101), satellite imagery analysis (EuroSAT), and general object classification (Caltech1O1, ImageNet). The details of each dataset are provided in Appendix A.5
Setup We showcase the effectiveness of C-TPT in reducing calibration error during test-time prompt tuning by incorporating it into the joint training with the previously proposed Test-time Prompt Tuning (TPT) (Manli et al. 2022). We initialize the prompt embeddings as the hard prompt photo of (CLIP and optimize the corresponding embeddings using TPT (TPTHarPPomp)) or jointly using TPT and C-TPT 1taT
Moreover, include an ensemble setting where average the logits from different hard- prompt initializations using ('a photo of a', 'a photo of the', 'a picture of a', 'a picture of the") (CLIP Ensemble). Similarly, we optimize using TPT (TPTllee) or jointly using TPT and C-TPT ETPTEmemtle +C on each of the hard-promp initializations and average the resulting logits. Additional experiments with different prompt initializations can be found in Appendix A.9
Implementation Details we employ TPT (Manli et al. 2002)) as the primary loss function to maximize accuracy while using C-TPT auxiliary objective enhance calibration in Eq. 5 We fix the À value 50.0 for all cases and in accordance with the previous test-time prompt tuning setup (Manli et al. 2022). we optimize the prompt for single step using the AdamW (Loshchilov and Hutter) optimizer with learning rate 0.005. All other settings for training TPT are conducted following Manli et al. (2022). All experiments were performed NVIDIA A100 80GB PCIe.
We use CLIP-RN50 and CLIP-ViT-B/I architectures. For all settings,
TThe details the choice of lambda provided in Appendix A.6

---

Published as a conference paper at ICLR 2024
Table I: Fine-Grained Classification. We present the results of CLIP-RN5O and CLIP-ViT-B/16 For each setting, we report the Acc. (f) and ECE (L) of the initialization, after applying TPT, and after jointly employing TPT and our proposed C-TPT- the values highlighted in bold signify the best ECE achieved after test-time prompt tuning. Std. is reported in Appendix A.7
ImageNet
Caltech
Flower
Food101
Aircraft
SUN397
EuroSAT
UCF101
Average
Pets
Cars
DTD
Method
Acc.
58.1
85.8
83.8
55.7
61.0
74.0
15.6
58.6
40.0
23.7
58.4
55.9
Cl.I-RN5OHnndPonn
ECE
2.09
4.33
5.91
4.70
3.19
3.11
6.45
3.54
9.91
15.4
3.05
5.61
Acc.
60.7
87.0
84.5
58.0
62.5
74.9
17.0
61.1
41.5
28.3
59.5
57.7
+TP HardPropp
ECE
11.4
5.04
3.65
3.76
13.4
5.25
16.1
9.24
25.7
22.5
12.4
Acc.
60.2
86.9
84.1
56.5
65.2
74.7
17.0
61.0
42.2
27.8
59.7
57.8
+TPT PaTTBaProometCCC +C-TPT
ECE
3.01
2.07
2.77
1.94
4.14
1.86
10.7
2.93
19.8
15.1
3.83
6.20
Acc.
59.7
87.1 82.9
55.6
60.5
75.6
16.4
60.2
41.0
29.3
59.8
57.1
CLIP-RN50paaane
ECE
5.15
6.43
6.46
7.34
5.02
5.04
3.92
6.19
4.54
7.70
3.55
5.58
Acc.
61.1
87.4
83.2
59.2
61.4
76.2
17.9
62.0
42.8
28.4
60.2
58.2
Eneemll
ECE
11.2
4.29
4.79
3.08
14.1
5.27
14.6
7.68
22.2
18.9
11.1
10.7
Acc.
61.2
87.4
84.0
57.3
65.3
76.0
17.5
62.1
43.1
29.4
60.7
58.5
ETPTememble+C-11
ECE
4.13
2.15
2.71
1.68
3.60
1.47
10.9
2.96
15.7
8.70
3.27
5.20
Acc.
66.7
92.9
88.0
65.3
67.3
83.6
23.9
62.5
44.3
41.3
65.0
63.7
CLIP-ViT-B/6Guaadde
ECE
2.12
5.50
4.37
4.25
3.00
2.39
5.11
2.53
8.50
7.40
3.59
4.43
Acc.
69.0
93.8
87.1
66.3
69.0
84.7
23.4
65.3
46.7
42.4
67.3
63.0
+TPT HardProupp
ECE
10.6
4.51
5.77
5.16
13.5
3.98
16.8
11.3
21.2
21.5
13.0
11.6
Acc.
68.5
93.6
1882
65.8
69.8
83.7
24.0
64.8
46.0
43.2"
65.7
64.88
+TP [PTTandrenpeeCC.. +T-TPT
ECE
3.15
4.24
1.90
1.59
5.04
3.43
4.36
5.04
11.9
13.2
2.54
5.13
Acc.
68.2
93.4
86.3
65.4
65.7
85.2
23.5
64.0
45.6
43.0
66.1
64.2
CLIP-ViT-B/IGeweenkt
ECE
3.70
6.16
4.88
7.09
6.01
3.78
4.56
4.01
13.8
6.01
4.05
5.82
Acc.
69.6
94.1
86.1
67.1
67.6
85.1
24.4
66.5
47.2
44.0
68.5
65.5
<TPTEmmmble
ECE
9.82
4.48
5.72
4.00
13.9
4.27
14.6
9.01
18.6
14.1
10.5
9.91
Acc.
69.3
94.1
87.4
66.7
69.9
84.5
23.9
66.0
46.8
48.7
66.7
65.8
+TPT -C-embil+C-TPP
ECE
4.48 3.14
1.54
1.84
5.77
2.38
6.40
3.09
13.7
5.49
3.04
4.62
Results Prior to discussing our results, it is worth recapping the ideal outcome: through the joint training with C-TPT, we desire reduce the ECF relative TPT while the accuracy drop is retained within margin of 1% (details in Appendix A.2 and A.8). Table I presents results for the fine-grained classification task. As noted in Observation from Section 4 TPT improves the prediction accuracy but also results in higher calibration error. However, when using both TPT and C-TPT, observe better calibration across all settings without compromising the accuracy benefits of TPT.
Specifically, in the Hard Prompt initialization settings, the average ECE drops from 11.7 to 6.20 for CLIP-RN50 and from 11.6 to 5.13 3 for CLJP-ViT-B/LL representing 47% and 56% reduction in calibration error, respectively. In the Ensemble settings, the average ECE decreases from 10.7 to 5.20 for CLIP-RN50 and from 9.91 to 4.62 for CLIP-ViT-BE ViT-B/66 leading to a 52% and 53% reduction in calibration error, respectively. Furthermore, Appendix (488 provides a visual representation illustrating the effectiveness of C-TPT.
6.2 NATURAL DISTRIBUTION SHIFTS
Dataset & Setup & Implementation Details For the natural distribution shift, we use ImageNet variants, including ImageNet-V2 V2, ImageNet-A, ImageNet-R. R, and ImageNet-Sketeh (details are in Appendix A.5). All the hyperparameters and configurations are set identically to Section 6.1 except for X, which we set to 20.0.
Results Table 2 shows the outcomes under natural distribution shifts. Similar to the results from Table I joint application of C-TPT improves the calibration error compared to utilizing TPT alone while still taking advantage of the accuracy increase of TPT. Specifically, for the Hard Prompt setting, there is a 28% and 52% drop of ECE for CLIP- -N500 and CLIP- ViT-B/16. respectively. For the Ensemble setting, there is 14% and 24% drop in ECE, respectively.
7

---

Published as a conference paper at ICLR 2024
Table Natural Distribution Shifts. We report the Acc. (1) and ECE (4) of the initialization, after applying TPT, and after jointly employing TPT and our proposed C-TPT- the values highlighted in bold signify the best ECE achieved after test-time prompt tuning. Std. reported in Appendix A.7
Method
ImageNet-A
ImageNet-V2
ImageNet-R
ImageNet-Sketch
Average
Acc.
21.7
$1.4
56.0
33.3
40.6
ECE
21.3
3.33
2.07
3.15
7.46
Acc.
25.2
54.6
58.9
35.1
43.5
HardPepp
ECE
31.0
13.1
9.18
13.7
16.7
Acc.
23.4
54.7
58.00
35.1
42.8
+TPT PanTRePPon +C-TPT
ECE
25.4
8,58
4.57
9.70
12.1
Acc.
22.7
$2.5
57.9
34.7
42.0
ELIP-RN50Eacae
ECE
17.0
2.68
5.64
10.9
9.06
Acc.
26.9
55.0
60.4
35.6
44.5
+TP Ensemble
BCE
29.1
12.7
7.50
14.0
15.8
Acc.
25.6
55.88"
59.7
35.7
44.0
+TPT +Cnem
ECE
27.0
5.17
12.2
13.6
Acc.
47.8
60.8
74.0
46.1
57.2
CLIP-ViT-B/1GHaaaraaa
BCE
8.61
3.01
3.58
4.95
5.04
Acc.
52.6
63.0
76.7
47.5
59.9
+TPTHudPrmn HardPropp
ECE
16.4
11.1
4.36
16.1
12.0
Acc.
51.6
62.7
76.0
47.9
59.6
+ IIPTitadPromptC-II
ECE
8.16
6.23
1.54
7.35
5.82
Acc.
50.9
74.5
46.0
ECE
8.85
3.01
2.85
9.70
6.10
Acc.
54.2
63.9
78.2
48.5
61.2
+TPT Ensemble
ECE
13.5
11.2
3.64
15.3
10.9
Acc.
52.9
63.4
78.00
48.5
660.7
+TPT Ensemble CCTTT +C-TPT
ECE
10.9
8.38
1.40
12.6
8.32
ABLATION STUDY
7.1 COMPARISON WITH PREVIOUS CALIBRATION METHOD
TPT
TPTamp
TPT a C-TPT (Ours)
Average
KCE
10
Caltech
Pets
Cara
Flawer111
Food101
Airozaft
SUN107
DTD
EmISAT
UCFI8I
ImageNet-
ImageNit-122
ImugeNNt--
ImageNet-k
Attrage
Figure 4:
Comparison of calibration error between TPT, temperature scaled TPT (TPTummp) and the joint use our proposed C-TPT (TPT+C-TPT), Results are based on CLIP-ViT-BIA
Temperature scaling (Guo et al. 2017) a widely used post-processing technique that calibrates predictions by adjusting the logits before softmax using learned temperature value, which trained on separate held-out validation set. A recent study by Vine et al. (2023) suggests that such temperature can be generalized across various datasets and prompts in CLIP. In this section, we apply temperature scaling training with the TPT output logits of the entire labeled ImageNet test set. Subsequently, this learned temperature is employed for TPT on different datasets, including fine-grained classification and natural data shift datasets, to assess its generalization capabilities. As illustrated in Figure 4 the temperature-scan scaled approach (denoted as TPT does reduce calibration compared to standard TPT. However, our proposed C-TPT exhibits better performance in most instances. Notably, C-TPT surpasses TPT in diminishing the average calibration error, even without leveraging any labeled training data.
8

---

Published as a conference paper at ICLR 2024
(class) (Ace: BCT: ATT--
bay (clandd (fce: 7K., 488, ATED (siI)
D my CaFFOE Ace LoN, 8260
this 15 phota of cclass3 Mecc 4 ALAD 829
TFT EaFTTIA Ace ITA 20T: ARTE ed30
there fclass] objects (Aec. ATTD
TET CaTIT Prruuin ARTO ecbl
Uhe rescest shape 21 this imaze
Men thA AIPEO
Prompt Visalication
Viszalization
Prompt Nassaliatioo
Clas Viuulication
(a) Hard Prompts
Prompts
Figure 5: I-SNE visualization of class-embedded textual representations for (a) Hard Prompts and (b) Tuned Prompts, utilizing the CLIP-RNSI model on the 101tech101 dataset. In both cases, each unique color signifies a distinct prompt the Prompt Visualization (left) and a distinct class in the Class Visualization (right). The legends belong to the Prompt Visualization (leff) for both cases.
7.2
WHY DOES TEXT FEATURE DISPERSION INFLUENCE CALIBRATION?
To better understand the relationship between Average Text Feature Dispersion (ATFD) and Expected Calibration Error (ECE), we visualize the class-embedded textual representations using t-SNE (van der Maaten 2014) plot of various hard prompts with similar accuracies but varying calibration (Figure 5 (a)). In Figure (left), in accordance with our findings, we can observe that the poorly- calibrated prompts (( l'there are /class) objects', 'the nearest shape in this image is "J)) are clustered together, showcasing low ATFD. Conversely, the well-calibrated prompts (I 'aa', 'a toy', 'this is photo )f")) are dispersed throughout the feature space, indicative high ATFD, Interestingly, for well-calibrated prompts, the features tend to cluster cohesively with respect the class (Figure (a) (right)), suggesting that the high dispersion in well-calibrated prompts a result of the text features aligning closely its respective class-label locations. On the contrary, in the case of poorly-calibran prompts, the text features tend to cluster together regardless the class labels. This pattern implies that certain intrinsic properties of these prompts such their higher descriptiveness, could hinder the text features from clustering near their corresponding class labels, leading to poor calibration.
In Figure (b)
we replicate the previous analysis, but with shift in focus prompt tuning. Observations indicate that when we employ standard Test-time Prompt Tuning (TPT), the text descriptions appear to cluster with each other, much like the poorly calibrated hard prompts discussed carlier. However, as we apply our proposed C-TPT and progressively increase its influence during the optimization process by increasing the À value (i.e., 45, 50, 55), the text features become more dispersed (Figure (b) (left). Notably, analogous to the well-calibrated hard prompt scenario presented earlier, these features cluster in relation to their respective class labels ( (righ()).
Within the hard prompt engineering, users may have control over calibration by selecting appropriate prompts that are not overly descriptive. However, this intuitiveness fades in prompt tuning, where prompts are sequences of embeddings rather than readable words. To address this, our proposed C- TPT can guide the optimization process for better calibration by increasing the text feature dispersion.
CONCLUSION
This paper presents the first comprehensive study of calibration in CLIP prompt tuning. A primary challenge in this domain is that the traditional calibration methods rely extensive labeled data, making them unsuitable for prompt tuning scenarios. To navigate this constraint, we turn to the inherent properties of CLIP, uncovering the role of prompts in calibration. Despite comparable accuracy across prompts, calibration errors vary significantly due to the dispersion of class-embedded textual features. Well-calibrated prompts exhibit high feature dispersion poorly-calibrated prompts show low dispersion with clustering features. Building on this, we introduce Calibrated Test-time Prompt Tuning (C-TPT), which prioritizes the Average Text Feature Dispersion (ATFD) during tuning. Our experiments across various datasets demonstrate the effectiveness of C-TPT in refining the calibration of predictions without the dependence on labeled data.
class labels, while

---

Published as a conference paper at ICLR 2024
9 ACKNOWLEDGEMENT
This work was supported by Institute of Information & communications Technology Planning & Evaluation (ITTP) grant funded by the Korea government(MSIT) (No.2022-0-00184) Development and Study of AI Technologies to Inexpensively Conform to Evolving Policy on Ethics), and Institute of Information & communications Technology Planning Evaluation (IITP) grant funded by the Korea (MV))) (No. 2022-0-00951, Development of Uncertainty-Aware Agents Learning by Asking Questions).
10 ETHICS STATEMENT
We confirm that our research adheres to the highest standards of ethical considerations. All work presented in this paper original, and any external contributions or sources have been appropri- ately cited. Our study does not introduce new datasets, nor does it involve experiments utilizing demographic identity characteristics.
11 REPRODUCIBILITY STATEMENT
To ensure the reproducibility of our work, we have detailed the comprehensive training procedures, hyperparameters, and experimental settings in Sections 6.1 6.2 of the paper. Moreover, show the complete list of hard prompts used for analysis and experiments in Appendix A.3
REFERENCES
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763. PMLR, 2021.
Shu Manli, Nie Weili, Huang De-An, Yu Zhiding, Goldstein Tom, Anandkumar Anima, and Xiao Chaowei. Test-time prompt tuning for zero-shot generalization in vision-language models. In NeurIPS, 2022.
Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. Tent: Fully test- time adaptation by entropy minimization. In International Conference Learning Representations, 2021. utLpttss: httpp::::pppeeeeeeeeee wwtp:otumttttt.u..te....a 2 332kkr33
Marvin Zhang, Sergey Levine, and Chelsea Finn. Memo: Test time robustness via adaptation and augmentation. Advances in Neural Information Processing Systems, 01138629-38644 2022.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In Doina Precup and Yee Whye Teh, editors, Proceedings of the 34th International Conference on Machine Learning. volume 70 of Proceedings of Machine Learning Research. pages 1321-1330. PMLR, 06-11 Aug 2017. URL htaps:://proceed. ngs. ml ptt.gsssl.pr.s 370//uoo/ html
Jishnu Mukhoti, Viveka Kulharia, Amartya Sanyal, Stuart Golodetz, Philip Torr, and Puneet Dokania. Calibrating deep neural networks using focal loss. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural In- formation Processing Systems, volume 33, pages 15288-15299. Curran Associates, Inc., files//aperr/2200
2020. URL https::/
nwwrips. pdf File/ 227733001022441222224443302-2-20
SooHwan Eom, Eunseop Yoon, Hee Suk Yoon, Chanwoo Kim, Mark Hasegawa-Johnson, and Chang Yoo. Adamer-cte: Connectionist temporal classification with adaptive maximum en- tropy regularization for automatic speech recognition. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 12707-12711. IEEE, 2024.
10

---

Published as a conference paper at ICLR 2024
Zifeng Wang, Zhenbang Wu, Dinesh Agarwal, and Jimeng Sun. MedCLIP: Contrastive learning from unpaired medical images and text. Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3876-388 Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. URL /tttpss//aci antho logy org/2022. emn 1p-malm..... 2.
Sheng Zhang. Yanbo Xu, Naoto Usuyama, Jaspreet Bagga, Robert Tinn, Sam Preston, Rajesh Rao, Mu Wei, Naveen Valluri, Cliff Wong, Matthew P. Lungren, Tristan Naumann, and Hoifung Poon. Large-scale domain-specifie pretraining for biomedical vision-language processing, 2023.
Zhihong Chen, Shizhe Diao, Benyou Wang, Guanbin Li, and Xiang Wan. Towards unifying medical vision-and- lannguug pre-training via soft prompts, 2023a.
Jie Liu, Yixiao Zhang, Jie-Neng Chen, Junfei Xiao, Yongyi Lu, Bennett A. Landman, Yixuan Yuan, Alan Yuille, Yucheng Tang, and Zongwei Zhou. Clip-driven universal model for organ segmentation and tumor detection, 2023.
Vishnu Sashank Dorbala, Gunnar Sigurdsson, Robinson Piramuthu, Jesse Thomason, and Gaurav Sukhatme. Clip-nav: Using clip for zero-shot rision-and-language navigation, 2022.
Samir Yitzhak Gadre, Mitchell Wortsman, Gabriel Ilharco, Ludwig Schmidt, and Shuran Song. Clip on wheels: Zero-shot object navigation as object localization and exploration. ArXiv, abs/2203. 10421, 2022.
Apoorv Khandelwal, Luca Weihs, Roozbeh Mottaghi, and Aniruddha Kembhavi. Simple but effective: Clip embeddings for embodied ai. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 14829- 14838, 2022.
A Fender Coelho Bucker, Luis Figueredo, Sami Haddadin, Ashish Kapoor, Shuang Ma, Sai Vemprala, and Rogerio Bonatti. Latte: Language trajectory transformer. In NeurlPS 2022 Foundation Models for Decision Making Workshop.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision- language models. International Journal of Computer Vision (ICV),, 2022a.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. In JEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022b.
John C. Platt. Probabilistic outputs for support vector machines and comparisons regularized likelihood methods. In ADVANCES IN LARGE MARGIN CLASSIFIERS, pages 61-74. MIT Press, 1999.
Bianca Zadrozny and Charles Elkan. Obtaining calibrated probability estimates from decision trees and naive bayesian classifiers. In Proceedings of the Eighteenth International Conference on Machine Learning, ICML '01, page 609-616, San Francisco, CA, USA, 2001. Morgan Kaufmann Publishers Inc. ISBN 1558607781.
Bianca Zadrozny and Charles Elkan. Transforming classifier scores into accurate multiclass prob- ability estimates. In Proceedings of the Eighth ACM SIGKDI) International Conference on Knowledge Discovery and Data Mining, KDD '02, page 694, New York, NY, USA, 2002. Association for Computing Machinery. ISBN 158113567X. doi: 10.1145/775047.77555111 URL 775111
Vladimir Vovk, Alex Gammerman, and Glenn Shafer. Algorithmic Learning in a Random World. Springer-Verlag, Berlin, Heidelberg, 2005. ISBN 0387001522.
Jing Lei, Max G Sell, Alessandro Rinaldo, Ryan Tibshirani, and Larry Wasserman. Distribution-fies predictive inference for regression. Journal of the American Statistical Association, 113(523): 1094-1111, 2018.
11

---

Published as a conference paper at ICLR 2024
Mahdi Pakdaman Nacini, Gregory Cooper, and Milos Hauskrecht. Obtaining well calibrated prob- abilities using bayesian binning. Proceedings of the AAAI Conference on Artificial Intelligence. 29(1), Feb. 2015a. doi: 10.1609/aai..v22 v99.1.962.....0..
httph:hts///oo.. aaai. org/ /indxx
/6002
Aviral Kumar, Sunita Sarawagi, and Ujjwal Jain. Trainable calibration measures for neural networks from kernel mean embeddings. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning. volume 80 of Proceedings of Machine Learning Research, pages 2805-2814. PMLR, 10-15 Jul 2018. URL htttt:t:////:ceeeeee
www.ppress/vee kumar Ba html
Archit Karandikar, Nicholas Cain, Dustin Tran, Balaji Lakshminarayanan, Jonathon Shlens, Michael Curtis Mozer, and Rebecca Roelofs. Soft calibration objectives for neural networks. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neu- ral Information Processing Systems, 2021. URL tVD13hosqs
net/ forum?idd
Hee Suk Yoon, Joshua Tian Jin Tee, Eunseop Yoon, Sunjae Yoon, Gwangsu Kim, Yingzhen Li, and Chang D. Yoo. ESD: Expected squared difference a tuning- free trainable calibration measure. In The Eleventh International Conference on Learning Representations, 2023a. URL net
JOSON
Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In International Conference on Learning Representations, 2018. URL https
//openreview.o net. for www.mmmidrrr.... Dddp1 -Rb
Hee Suk Yoon, Funseop Yoon, John Harvill, Sunjae Yoon, Mark Hasegawa-Johnson, and Chang Yoo. SMSMix: Sense-maintained sentence mixup for word sense disambiguation. In Yoav Goldberg, Zomitsa Kozareva, and Yue Zhang, editors, Findings of the Association for Computational Lin- guistics: EMNLP 2022, pages 1493-1502, Abu Dhabi, United Arab Emirates, December 2022. condding--enll11000 URL nitts://saclanthee ceaethohoogy....o www.oog....oo.oooo..oo...o...ooo 2022. findings----ii emr 1m.
Association for Computational Linguistics. doi: 10. N0.18653/11/202.. 10186665w1//2222nnnnnggsss.
Sunil Thulasidasan, Gopinath Chennupati, Jeff A Bilmes, Tanmoy Bhattacharya, and Sarah Michalak. On mixup training: Improved calibration and predictive uncertainty for deep neural networks. Advances in Neural Information Processing Systems, 32, 2019.
Deng-Bao Wang, Lanqing Li, Peilin Zhao, Pheng-Ann Heng, and Min-Ling Zhang, On the pitfall of mixup for uncertainty calibration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 7609-7618, June 2023.
Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for rerrameter-efficieee prompt tuning. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen- tau Yih, editors, Proceedings of the 2021 Conference on Empirical Methods in Natural Lan- guage Processing, pages 3045-3059, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10. 18.13//3/v 1/2021. 2e2202eeemlmmmmmmmmm mapp--mii..433 URL https://acLantho..oo wwtp://ooooooo..///2/////// emnnp ma inmain.22 243
Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. European Conference on Computer Vision, pages 709-727. Springer, 2022.
Eunseop Yoon, Hee Suk Yoon, John Harvill, Mark Hasegawa-Johnson and Chang Yoo. INTapt: Information-theoretic adversarial prompt tuning for enhanced non-native speech recognition. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Findings of the Association for Computational Linguistics: ACL 2023, pages 9893-9902, Toronto, Canada, July 2023b. UtLhttps!
Association for Computational Linguistics. doi: 10. 122.118833/111223..f.l org/ findings-acl. 627
7/ //aclanthology.oooo orgg:///hooooooo.o/22223
Sunjae Yoon, Gwanhycong Koo, Ji Woo Hong, and Chang Yoo. Neutral editing framework for diffusion-based video editing. arXiv preprint arXiv:2312.06778, 2023c.
12

---

Published as a conference paper at ICLR 2024
Gwanhyeong Koo, Sunjae Yoon, and Chang D Yoo. Wavelet-guided acceleration of text inversion in diffusion-based image editing. arXiv preprint 1eXiv:2401.0979, 2024.
Guangyi Chen, Weiran Yao, Xiangchen Song, Xinyue Li, Yongming Rao, and Kun Zhang. PLOT: Prompt learning with optimal transport for laiguag models. In The Eleventh International Conference on Learning Representations, 2023b. URL Stllttts: http://ppppevviieeee www.neee/eeeeee forrfo?um?
id=zgwryBoXYni
Changsheng Xu Hantao Yao, Rui Zhang. Visual-language prompt tuning with knowledge-guide context optimization. In The IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.
Mahdi Pakdaman Naeini, Gregory Cooper, and Milos Hauskrecht. Obtaining well calibrated prob- abilities using bayesian binning. Proceedings of the AAAI Conference on Artificial Intelligence. 29(1), Feb. 2015b. doi: 10.1609/amai v291196622 URL htttp::///// aaai. orq/ index
php Thpp/AAAT/N /alAAT/ariiim hetetiieeeeeeeeee/ee..oee 9962
Chaithanya Kumar Mummadi, Robin Hutmacher, Kilian Rambach, Evgeny Levinkov. Thomas Brox, and Jan Hendrik Metzen. Test-time adaptation to distribution shift by confidence maximization and input transformation. arXiv preprini arXiv:2106. 14999, 2021.
Matthias Minderer, Josip Djolonga, Rob Romijnders, Frances Hubis, Xiaohua Zhai Neil Houlsby, Dustin Tran, and Mario Lucie. Revisiting the calibration of modern neural networks. Advances in Neural Information Processing Systems, 34:15682-15694 2021.
Ido Galil, Mohammed Dabbah, and Ran El-vaniv. What can we learn from selective prediction and uncertainty estimation performance of 523 imagenet classifier?? In The Eleventh International Conference on Learning Representations.
Laurens van der Maaten. Accelerating t-sne using tree-based algorithms. Journal of Machine Learning Research, 15(93):3221-322 2014. URL
wandermaaten 4a htmll
David Freedman, Robert Pisani, and Roger Purves. Statisties (international student edition). Pisani, R. Purves, 4th edn. WW Norton & Company, New York, 2007.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Representations.
Will Le'ine, Benjamin Pikus, Pranav Vishnu Raja, and Fernando Amat. Enabling calibration in the zero-shot inference of large vision-language models. In ICLR 2023 Workshop on Pitfalls of limited data and computation for Trustworthy ML, 2023. URLhttps: / wpw.revie...eeeeeeemmee/e. n
Forum?id=nal1
James Urquhart Allingham. Jie Ren, Michael W Dusenberry, Jeremiah Zhe Liu, Xiuye Gu, Yin Cui, Dustin Tran, and Balaji Lakshminarayanan. A simple zero-shot prompt weighting technique to improve prompt ensembling in text-image models, 2023.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255. leee, 2009.
Fei-Fei Li, Marco Andrecto, Marc' Aurelio Ranzato, and Pietro Perona. Caltech 101, Apr 2022.
Omkar M. Parkhi, Andrea Vedaldi, Andrew Zisserman, and C. V. Jawahar. Cats and dogs. In IEEE Conference Computer Vision and Pattern Recognition, 2012.
Jonathan Krause, Michael Stark, Jia Deng, and Fei-Fei. 3d object representations for fine- grained categorization. In 4th International IEEE Workshop on 3D Representation and Recognition (3dRR-13), Sydney, Australia, 2013.
Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In Indian Conference Computer Vision, Graphics and Image Processing, Dec 2008.
13

---

Published as a conference paper at ICLR 2024
Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101 mining discriminative com- ponents with random forests. In David Fleet, Tomas Pajdla, Bernt Schiele, and Tinne Tuytelaars, editors, Computer Vision ECCV 2014, pages 446-661, Cham, 2014. Springer International Publishing. ISBN 978-3-319-11599944
S. Maji, J. Kannala, E. Rahtu, M. Blaschko, and A. Vedaldi. Fine-grained visual classification of aircraft. Technical report, 2013.
Jianxiong Xiao, James Hays, Krista Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to 2oo. In 2010 IEEE Computer Society Conference on Computer Vision and Puttern Recognition, pages 3485-3492, 2010. doi: 10.1109/CVPR.2010
5539970.
Mircea Cimpoi, Subhransu Maji, lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describ- ing textures in the wild. In 2014 IEEE Conference on Computer Vision and Puttern Recognition, pages 3606-3613, 2014. doi: 10. 11111//P..20444...
Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Introducing eurosat: novel dataset and deep learning benchmark for land use and land cover classification. In IGARSS 2018-2018 IEEE International Geoscience and Remote Sensing Symposium, pages 204-207. IEEE, 2018.
Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. UCF101: A dataset of 101 human actions classes from videos in the wild. CoRR, Abs/1222, 2012. URL http://axxiv
org/abs/1212, 0402
Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. CVPR, 2021a.
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In International conference on machine learning, pages 5389-5400.
PMLR, 2019.
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, and Justin Gilmer. The many faces of robustness: A critical analysis of out-of-distribution generalization. ICCV, 2021b.
Haohan Wang. Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global representations by penalizing local predictive power. In Advances in Neural Information Processing Systems, pages 10506-10518. 2019.
Yaochu Jin and Bernhard Sendhoff. Pareto-based multiobjective machine learning: An overview and case studies. Systems, Man, and Cybernetics, Part C: Applications and Reviews, IEEE Transactions on, 38:397 415, 06 2008. doi: 10. 10199//SMCC.2088.19999111
Xi Lin, Hui-Ling Zhen, Zhenhua Li, Qingfu Zhang, and Sam Kwong. Pareto multi- task learning. In Third Conference on Neural Information Processing Systems (NeurIPS), pages 12037-12047, 2019.
Aviv Navon, Aviv Shamsian, Gal Chechik, and Ethan Fetaya. Learning the pareto front with hypernetworks. In International Conference on Learning Representations, 2021. URL https
//openreview..o wwwp:nrwwiwwweeeeeeeeeme 2od
Jameel Hassan, Hanan Gani, Noor Hussein, Uzair Khattak, Muzammal Naseer, Fahad Khan, and Salman Khan. Align your prompts: Test-time prompting with distribution alignment for zero-shot generalization. In NewrIPS, 2023.
Jerrold H Zar. Significance testing of the spearman rank correlation coefficient. Journal of the American Statistical Association, (7(3)))5778
533o3iation,67((33)55788 5800 1972.
14

---

Published as a conference paper at ICLR 2024
A APPENDIX
A.
BROADER IMPACT
This paper emphasizes the importance of calibration in vision-language foundation models, a critical yet often overlooked aspect in the drive for increasing accuracy in AI development. By achieving calibration without labeled data, a first in the field, it demonstrates how we can utilize the intrinsic properties of AI models for better calibration, setting precedent for future research. The insights derived from this study could be particularly beneficial enhancing the trustworthiness and safety of Al applications in scenarios where labeled data is scarce.
A.2 LIMITATIONS
One limitation of regularization-batn based calibration methods is that there could be instances of accuracy degradation. Although the accuracy is not significantly affected based on our experimental results, there are some minor degradations in few cases. This slight reduction is in line with many trainable calibration methods (Kumar al., 2018 Karandikar et al., 2021] Yoon et al. (023a). which often target ECE reduction at the expense of minor compromise in accuracy typically less than to 2% degradation.
Another limitation worth noting is that since this paper leverages a test-time adaptation method, access to the training or validation sets for hyperparameter tuning limited. Our approach involves the joint optimization of the accuracy-centric and calibration-centric loss, for which we introduce the balancing hyperparameter lambda (A), as depicted in Eq. 5 In this study, we fix À to 50.0 for test scenarios, as validated in Section (A.6) While acknowledge that this choice may not be optimal for every data sample, it represents a pragmatic initial step. Future research could explore strategies that dynamically adapt the balancing hyperparameter for each data sample, potentially enhancing model performance even further.
A.3 LIST OF HARD PROMPTS USED FOR ANALYSIS
The plots and analyses depicted in Figure and Figure 2 are derived from a curated set of 80 prompts originally developed by Radford et al. (2021). Theses prompts are listed as follows:
List of Hard Prompts
drawing of the [class], art of (class), J, itap of the [class], drawing of a [class), a origami (class), photo of a nice (class), blurry photo of (class), close-up photo of the (class), a photo of a clean (class), , a photo of a weird [class), a photo of a small (class), a photo of the large (class), pixelated photo of the [class), embroidered [class), photo of the clean (class), the origami (class), ), the plushie (class), photo cool (class), ), a sculpture of the (class), low resolution photo of the (class), bad photo of the (class), ), jpeg corrupted photo of a [class), a rendition of the (class), a photo the cool (class), a low resolution photo of (class), cropped photo of the (class), the plastic (class), sculpture of (class), a pixelated photo of a [class). plastic (class), itap of my (class), close-up photo of (class), ) bright photo of a (class). art of the [class], graffiti of the (class), tattoo of a [class], sketch of the [class], J. a dark photo of (class), tattoo of the [class], photo of the dirty (class), a black and white photo of the (class), ), photo of a (class), a painting of the [class), cropped photo of (class), a photo of large (class). a photo of the weird (class), graffiti of (class) ), a painting of a (class), cartoon (class). the cartoon (class), good photo of the (class), jpeg corrupted photo of the (class), a bad photo of a (class), photo of the small (class), ), rendering of the (class), a photo of a dirty (class), rendition of a [class), ). a blurry photo of the (class), the toy {class), J, the embroidered (class), ). a rendering of a (class), ). a photo of a hard to see {class], a dark photo of the (class), doodle of the (class), good photo of a (class), photo of the (class), a photo of many (class), a plushie (class), ), a photo the nice (class), a bright photo of the (class), ), toy (class), ), photo of the hard see (class), photo of one (class), a photo of my (class), a black and white photo of a (class), sketch of a (class)
,. itap of a (class), a doodle of a (class). a sketch of a [class), a
15

---

Published as a conference paper at ICLR 2024
A.4 CALIBRATION OF CLIP MODELS
In Section 4 we illustrated the insight that emerges from our research, demonstrating that the choice of prompts can lead varying calibration error, even when maintaining similar levels of accuracy. In Figure we offer an additional visualization to further substantiate our finding of the pivotal role of prompt selection, not only in determining the accuracy but also the calibration error
tbe Caltebho
1d Pees
Suwarttt
to
B301
Fowatia
Airorat
10
Accemoo
d6 SUNO39
@e DTD
T1
Figure 6: Visualization of the variation in calibration error for different hard prompts in CLIP-ViT-B/IE across 11 datasets. For each dataset, the left plot presents scatter plot comparing the calibration еπто (ECE) and accuracy (Acc.) for diverse set of prompts. In the right plot, we categorize the seatter points into bins, each encompassing range of % accuracy. For each bin with more then 3 points, we plot the average ECE and Acc. well the bin't 's max and min ECE.
IIn this section, expand upon the initial set of 80 prompts specified in Section A.3 incorporating additional prompts provided by Allingham al. (2023). This results in diverse set of 320 hard prompts.
16

---

Published as a conference paper at ICLR 2024
A.5 DETAILS ON THE DATASET
In our experiments, we utilized the datasets adhering to CoOp (Zhou et al. (002aa). This encompasses 11 datasets designated for fine-grained classification and 4 ImageNet-derived datasets for evaluating natural distribution shifts. The details for each dataset, such as the number of classes, test-set size, and the respective tasks, can be found in Table
Table 3: The detailed statisties of datasets used in the experiments.
Dataset
Classes 1,000
Test set size
ImageNet DDng et al
2009
50,000
Caltech 101 Li et al. 2022
100
2,465
OxfordPets Parkkh et al. 2012
37
3,669
StanfordCars (Krause et al. 20113)
196
8,041
Flowers 102 (Nilsback and Zisserman
2008
102
2,463
Food101 (Bossard ct al.
2014)
101
30,300
FGVCAircratt (Maji al.
2013
100
3,333
SUN397 (Xiao et al. 2010)
397
19,850
DTD (Cimpor et 2014)
47
1,692
ETr (Helber et al.
2018
10
8,100
UCF101 (Soomro et al. 2012
101
3,783
ImageNet-A (Hendrycks et al.
2021a)
200
7,500
ImageNetV2 (Recht et 2211)
1,000
10,000
ImageNet-R (Hendrycks al.
202
200
30,000
ImageNet-Skee Skccchhkhhhhhhhhhhh (Wang el al.
2019)
1000
50,889
A.6 INFORMATION ON THE CHOICE OF LAMBDA
Caltech1ol
Pets
Cars
Plower102
Food101
Aircraft
SUNW7
DTD
EuraSAT
UCF10L
080
075
612
650
630
Lambda
Lambda
(2)
Figure 7: The trend of Accuracy, ECE, and ATFD with respect to varying lambda across the validation multiple datasets. Both the ECE and ATFD appear saturate at equal to 50.0 across all datasets. While the accuracy is not significantly affected, there are instances of minor degradation of approximately The values are based on CLIP-RN50.
Our method employs a test-time prompt tuning strategy, which does not allow access to data for hyperparameter tuning real-use cases. Nevertheless, our approach incorporates a lambda (A) term as defined in Eq. [5] This term balances the accuracy-based objective and our proposed calibration objective, C-TPT. By analyzing the trend observed in the validation set of various datasets (depicted in Figure 7. both the accuracy and ATFD appear saturate when À is at 50.0 across all datasets, suggesting that this value can served as generalizable hyperparameter regardless of the input. Consequently, in the experiments presented in main paper, we set the À term to 50.0 during test-time optimization.
Additionally, interesting observation worth highlighting from the plots presented in this section is their illustration of our key finding: the strong negative relationship between ECE and ATFD. As the ATFD increases according to increased X, can clear downward trend in ECE, thereby reinforcing our findings about their inverse relationship.
17

---

Published as a conference paper at ICLR 2024
A.7 RESULT: STANDARD DEVIATION
Table and Table 5 presents the standard deviation of three random run with different seeds for the
results of Table 1 and Table 2
Table 4: Standard Deviation of Fine-Grained Classification.
Flower102
ImageNet
Caltech
Food101
Aireraft
SUN397
EuroSAT
UCF101
Pets
Cars
DTD
Method
CLIP-RN50p Hrnddoop
Acc.
1.03
1.24
1.14
1.
16
1.
12
1
16
1.32
1.07
1.35
1
13
1.24
-TPTHanPPomn
ECE
1.18
1.22
1.19
1.21
1
10
1.24
1.13
1.09
1.26
1.19
1.22
Acc.
004
1.10
1.35
1.23
1.
18
1.26
1
22
1
10
1
20
1.13
1.17
+TPT HardPrompt C-TTP
ECE
1.15
1.09
1.20
1.10
1.14
CLIP-RNS SPP--RN50msseas
Acc.
1.11
1.22
1.20
1.26
1.32
1.24
1.25
1.26
1.13
1.21
1.19
+TPT Ensemble
ECE
2.23
2.24
1.15
12
12
1.
1.
2.
15
-
13
1.12
Acc.
1.22
1.19
1.34
1.21
1.15
1.34
1.13
1.12
1
1.25
1.12
1.14
+TPT Rasembii=C-TPP
ECE
1.17
1.14
4.20
4.22
1.19
4.11
4.17
1.16
1.22
1.15
1.13
Acc.
5.05
4.20
4.17
4.25
4.
4.27
4.
24
4.11
34
E.18
4TPT HandPrompt
ECE
1.19
4.23
4.17
1.21
4.
4.10
1.21
4.
13
18
4.11
Acc.
2.07
4:22
2.12
2
#31
231
+TP Htandpopm T +C-TPT
ECE
2.12 4.12
4.15
4.
2.20
CLIP-ViT-B/16 CLI--ViT-B/GGeeeuhee GEnsemble
Acc.
2.23
4.29
4.33
E.26
TTP Ensemble
ECE
1.11
1.11
1.21
1.21
1.12
4.23
1.15
Acc.
32
2.26
4
20
29
ECE
22
1.
19
+. 15
19
2.
1.17
+
09
+
23
+
08
1.13
Table 5: Standard Deviation of Natural Distribution Shifts.
Method
ImageNet-A
ImageNet-Ve V2
ImageNet-R ImageNet-Sketee
CLIP- CLIP-RN50 RN50padPiompt
Acc.
1.17
+.10
+.00
+.05
*TPTHudRogge
ECE
+.19
+.20
4.16
+.13
Acc.
E.16
".122"
E.07
"E.07"
+TPT HudPPooget CTTP
ECE
+.18
+.11
+.13
+.09
ALIP-RN0puscoll
Acc.
+.21
+.06
4.11
+.03
+TPT Ensemble
ECE
1.13
1.18
1.13
1.16
Acc.
4.17
+.07
4.10
+.05
+TPT +CTTT
ECB
+.16
+.09
+.11
+.08
CLIP-VIT-B/1 VTT-B/161aaaaommm
Acc.
E.19
I.II
E.07
+.09
+TPT Tardrrorooggg
BCE
1.16
1.18
#.13
#.20
Acc.
1.09
1.09
E.06
E.08
+TPT BardPooro WirrrrrogeeC--TT +C-TPT
ECE
1.13
1.15
1.10
1.12
NLIP--IT-B/16eweac Ggnsembble
Acc.
1.21
1.12
E.08
1.04
+TPTEmante Ensemble
ECE
1.17
+.22
+.14
+.12
Acc.
1.14
1.12
E.07
1.00
TPPEneabllc En +C-TPT
BCE
1.18
1.10
1.13
1.09
18

---

Published as a conference paper at ICLR 2024
A.8
PARETO FRONT: VISUALIZING THE EFFECT OF C-TPT
OCLP ATPT CTTP
85
AA
C-IFT (Increase A)
50 (Ours)
20
TPT
ECED
16
ECE.()
35
prompt
(TPT)
65
63 Accuracy (f)
Accuracy (1)
(a) Flower102
OCLPP ATPL O C.PPT
10
(Ours)
ECE-0)
Hard prompt
25
(Increase
00
56
56.5
56.0
50.5
$7.0
57.5
$8.0
68.5
59.9
Accuracy
Accuracy (t)
(b) Cars
30
lOCLIP ÂTPT e TPT C-TPT
C-IPT (lncrease 2)
26
5o (Ours)
TPT
BCE
30
promge
10
15
10
(TPT)
Accuracy
Accuracy (1)
(c) UCFI01
Figure 8: Pareto Front of BCE vs. Accuracy.
In multi-objective optimization scenarios, there exists trade-off between the two objectives. Simi- larly, calibration and accuracy are a trade-off in most regularization-bar approaches (Kumar et al.) 2018 Karandikar al. 2021 Yoon et al. (023a). This trade-off is often depicted through the Pareto front where each point on the front represents a different equilibrium between the conflicting objec- tives (Jin and Sendhoff, 2008 Lin al. 2019: Navon al.) (021). Figure @jillustrates this concept by presenting a scatter plot comparison among various approaches: the hard prompt (CLIP), TPT, and our method (TPT+C-TPT). Figure left shows the scatter plot where the points represent their corresponding accuracy and ECE using 80 different hard prompt initialization during the test-time prompt tuning. The plots show that our solution tends to form a Pareto front, rather than simply obtaining prompt with lower ECE (i.e., better calibration) and lower accuracy. Figure 8 right focuses on the seatter plot for specific hard prompt initialization ("a photo of a') under varying term in Eq. [ This visualization reveals how altering the À value causes the corresponding solutions to traverse along a Pareto front. Notably, this contrasts with the hard prompt, which clearly does not lie within the Pareto front
19

---

Published as a conference paper at ICLR 2024
A.9 ADDITIONAL EXPERIMENTAL RESULTS: EXPLORING DIFFERENT INITIALIZATION
This section shows additional experimental results where the hard prompt is initialized differentl. Table Thows the results when initializing the prompt with 'an example of", and Table dispays the results for the initialization 'a photo of the cool'. Moreover, we include the setting where the prompt initialized with the supervised trained embeddings CoOp (Zhou al. 2022a) and CoCoOp (Zhou et al. 2022b) in Table 8 For CoOp we utilize the officially published checkpoint and for CoCoOp we train on 16 images per class from half of the ImageNet's total classes with learnable prompt embeddings.
displays
Table 6: Fine-Grained Classification. TPT is initialized with 'an example of". We report the Acc. (f) and ECE (L) of the initialization, after applying TPT, and after jointly employing TPT and our proposed C-TPT- the values highlighted in bold signify the best ECE achieved after esst-tmme prompt tuning. Std. is reported in Appendix A.II
A.IT
ImageNet
Caltech
Flower102
Food 101
Aireraft
SUN397
EurOSAT
UCF101
Average
Pets
Cars
DTD
Method
Acc.
55.1
80.3
75.7
55.8
58.1
75.2
16.1
56.2
41.1
25.5
56.3
54.1
IPP-RNOHardPiompp
ECE
7.91
2.52
4.80
3.04
3.31
4.80
3.68
5.20
9.43
3.76
4.64
Acc.
60.9
87.1
77.2
57.7
62.7
76.1
17.9
60.7
41.2
29.4
57.7
57.1
[PPTaddPoog
ECE
8.51
5.12
6.98
5.52
12.2
4.83
15.2
8.19
20.2
11.1
15.3
10.3
Acc.
61.2
88.4
78.0
57.1
65.4
75.8
17.6
61.4
41.2
30.4
58.4
57.7
TTPTTuddrrome C-TPT
ECE
3.85
2.89
2.72
2.05
2.97
1.90
7.16
4.84
15.6
7.69
6.99
5.33
Acc.
65.2
90.9
82.5
64.6
64.7
83.9
22.3
61.4
42.4
38.8
64.8
62.0
T.IP-VVTTBB16Gaaddhoe
ECE
3.65
7.51
2.91
2.49
4.70
2.78
7.09
3.33
4.94
13.4
2.79
5.05
Acc.
69.3
93.0
83.0
67.3
69.4
84.8
22.9
65.3
45.8
40.7
67.1
64.4
TTPTaadPrrogg
ECE
8.72
2.91
7.30
6.26
12.2
5.05
16.2
7.94
20.5
20.8
11.6
10.9
Acc.
69.4
93.8
86.9
66.6
71.5
84.3
23.6
66.0
45.4
5T.5
66.4
65.9
+TPT IPTTadProngeCCC +C-TPT
ECE
4.04 1.62
2.89
1.75
4.49
1.36
9.05
3.54
15.5
5.18
3.87
4.85
Table 7: Fine-Grained Classification. TPT is initialized with photo of the cool'. We report the Acc. (1) and ECE (L) of the initialization, after applying TPT, and after jointly employing TPT and our proposed C-TPT-the the values highlighted in bold signify the best ECE achieved after test-time prompt tuiigg Std. is reported in Appendix A.11
ImageNet
Caltech
Flower102
Food101
Aireraft
SUN397
EuroSAT
UCF101
Average
Pets
Cars
DTD
Method
Acc.
56.9
80.9
79.8
56.9
57.7
73.0
16.1
56.5
39.6
21.9
56.3
54.1
CLIP-RN50HaadPmnpe
ECE
2.36
4.79
3.30
4.83
5.14
1.49
6.42
3.33
6.94
13.9
3.76
5.11
Acc.
61.5
86.5
82.6
58.8
61.6
75.8
17.4
60.2
39.2
26.3
59.7
57.2
TTPTuudPeeemg
ECE
12.6
4.49
17.0
7.93
17.5
11.4
24.8
15.7
Acc.
61.1
87.1
83.5
57.2
67.0
76.0
17.4
60.3
39.1
26.1
59.6
57.7
ITPTitadPooppeCC TTTTP
ECE
8.74
2.85
1.75
1.65
6.34
3.70
13.5
8.28
18.0
11.2
8.82
7.71
Acc.
64.7
88.1
86.2
66.5
64.5
81.4
22.7
62.4
38.4
34.6
67.6
61.6
CLIP-VIT-BI VTT-B/166
ECE
2.32
4.37
4.59
4.59
1.10
6.11
2.83
7.43
14.1
2.65
4.94
Acc.
69.2
91.5
87.4
67.2
67.9
84,9
24.5
65.9
45.5
43.3
66.4
65.0
TTPT
ECE
12.2
3.11
6.34
6.36
14.6
5.74
19.2
13.3
20.0
18.2
14.1
12.1
Acc.
69.1
91.7
88.8
66.9
69.6
84.1
24.7
65.5
46.
43.0
67.0
65.2
TTPTRddenog CCTPPT
ECE
1.59
1.64
10.6
2.43
10.5
10.7
18.0
8.73
7.42
7.34
20

---

Published as a conference paper at ICLR 2024
Table 8: Fine-Grained Classification. TPT is initialized with the embeddings from CoOp and CoCoOp. We report the Acc. (f) and ECE (1) of the initialization, after applying TPT, and after jointly employing TPT and our proposed C-TPT-the values highlighted in bold signify the best ECE achieved after test-time prompt tuning. Std. reported in Appendix A.11
ImageNet
Caltech
Flower102
Food101
Aireraft
SUN397
EuroSAT
UCF101
Average
Pets
Cars
DTD
Method
Acc.
63.5
86.6
87.1
55.4
61.4
75.6
15.1
58.2
37.3
26.2
59.0
56.9
CLIP-RN5Ocuop
ECE
1.97 3.12
6.65
7.17
4.59
2.56
6.20
1.99
10.4
12.2
3,92
5.52
Acc.
65.5
87.3
87.6
57.7
61.5
76.4
15.8
60.0
38.2
27.1
60.8
58.0
+TPTc.op
ECE
17.1
7.71
4.86
5.63
20.1
11.0
23.0
21.1
35.7
32.1
21.4
18.2
Acc.
64.2
87.6
76.0
15.4
59.4
37.7
26.3
60.1
57.6
+TPTcap4C-TP
ECE
7.53 3.26
3.14
2.57
7.75
4.39
14.4
11.2
27.1
19.7
11.0
10.2
CLIP-RN50cacapp
Acc.
61.5
88.6
87.4
54.7
64.0
75.3
14.0
60.8
39.2
29.1
61.5
57.8
BCE
2.63
4.24
7.39
6.14
3.86
2.52
4.15
3.79
7.25
5.24
2.63
4.53
Acc.
62.8
89.1
85.0
56.5
64.1
75.9
15.0
62.0
39.4
29.2
60.0
58.4
+TPTcoso0p
ECE
3.71
1.59
4.39
2.30
3.85
2.67
7.31
4.50
15.0
6.31
7.87
5.41
Acc.
62.0
88.7
55.66
64.1
14.1175..7
14.3
61.3
39.6
2818
55.1
57.9
4TPTcc00p
ECE
1.55
2.90
6.29
4.42
2.14
6.14
4.08
4.33
Acc.
71.6
93.6
89.2
63.1
67.4
83.2
18.0
63.7
43.1
40.1
66.0
63.5
CLIP-ViT-B/lccop
ECE
1.41
3.65
2.92
6,86
3.92
1.55
9.21
1.72
7.71
15.3
3.47
5.25
Acc.
73.7
94.0
89.1
65.6
68.7
83.8
20.0
65.6
44.5
40.6
67.2
64.8
+TPTcopp
ECE
15.2
3.65
7.40
6.63
19.9
9.66
29.6
20.8
34.8
31.3
19.9
18.1
Acc.
72.3
93.9
63.1
69.0
83.7
192
65.1
45.0
40.4
66.6
64.3
+TPTc.opC--TPT
ECE
6.80
1.66
2.12
2.45
10.2
4.49
21.5
11.8
21.0
13.2
12.0
9.75
CLIP-ViT-B/IG6 VIT-B//66Gccco
Acc.
67.8
91.0
88.3
64.9
84.1
24.2
63.0
44.6
44.1
67.0
64.3
ECE
2,89
3.52
4.60
6.51
3.82
3.25
4.06
4.61
3.82
5.81
3.28
4.20
Acc.
68.8
91.2
88.5
65.9
68.6
84.6
24.9
64.0
45.0
44.5
67.8
64.9
+TPTccos0p
ECE
2.33 2.74
2.22
5.22
4.70
1.94
6.13
3.16
6.91
9.03
3.47
4.35
Acč.
68.
91.4
188.5
64.9
69.3
84.2
24.6
63.6
44.7
44.3
67.1
64.7
4C-TT
ECE
1.72
3.45
3.76
5.09
3.13
2.66
4.80
2.96
4.18
5.79
2.91
3.68
21

---

Published as a conference paper at ICLR 2024
A.10 ADDITIONAL EXPERIMENTAL RESULTS: APPLICATION TO DIFFERENT TEST-TIME PROMPT TUNING METHODS
In this Section, further show that C-TPT can also be applied other test-time prompt tuning methods. Specifically, we apply C-TPT to PromptAlign (Hassan et al. 2023). which introduces distribution alignment strategy for CLIP to improve test-time adaptation.
We initialize the prompt embeddings as the hard prompt 'a photo of a' and optimize the corresponding embeddings using PromptAlign (PromptAlign
or jointly using PromptAlign and C-TPT PromppAlig@audPPPonn + C-TPT). The experimental settings for applying PromptAlign follow its original settings from Hassan et al. (202)). As with our previous experiments, we fixed the À value 50.0 for Fine-Grainee Classification and 20.0 for Natural Distribution Shifts. Table presents the results for the fine- fine-grained classification task and Table I0 shows the outcomes under natural distribution shift task.
The results show that C-TPT can effectively reduce the calibration error while still taking advantage of the accuracy increase of PromptAlign. This signifies that other test-time prompt tuning methods can benefit from C-TPT to decrease the calibration error.
Table Fine-Grained Classification. PromptAlign is initialized with 'a photo of a'. We report the Acc. (?) and ECE (U) of the initialization, after applying PromptAlign, and after jointly employing PromptAlign and our proposed C-TPT- the values highlighted in bold signify the best ECE achieved after test-time prompt tuning. Std. is reported in Appendix A.11
A.11
ImageNet
Caltech
Flower102 102
Food101
Aireraft
SUN397
EurOATT
UCF101
Average
Pets
Cars
DTD
Method
Acc.
65.2
90.9
82.5
64.6
64.7
83.9
22.3
61.4
42.4
38.8
64.8
62.0
CLIP- Baat///oooooopg
ECE
3.65
7.51
2.91
2.49
4.70
2.78
7.09
3.33
4.94
13.4
2.79
5.05
Acc.
72.1
94.1
90.5
68.0
72.1
87.6
25.5
68.1
47.9
44.8
69.8
67.3
-PromppAAlignan.ceee pudPonu
ECE
7.69
1.98
11.2
3.04
8.30
8.39
25.6
24.7
9.83
Acc.
72.2
94.0
90.6
67.8
72.1
87.5
25.3
67.8
47.7
45.9
69.8
67.3
-PromptAligDeuddren +C-TPT
BCE
5.87
2.20
2.09
1.79
9.26
2.25
6.57
6.29
22.1
21.8
9.95
8.19
Table 10: Natural Distribution Shifts. We report the Aec. (1) and ECE (L) of the initialization, after applying PromptAlign, and after jointly employing PromptAlign and our proposed C-TPT-the values highlighted in bold signify the best ECE achieved after test-time prompt tuning. Std. is reported in Appendix A.Il
Method
ImageNet-A
ImageNet-V2 ImageNet-R ImageNet-Sketch Average
Acc.
47.8
60.8
74.0
46.1
57.2
LLP--VTT-B16rteeeee
ECE
8.61
3.01
3.78
4.95
5.09
Acc.
55.8
65.3
78.5
50.3
62.5
PrromptAlignnaddrrra
ECE
16.1
10.8
4.70
16.8
12.1
-PromptAlignmaaareee CCTPT
Acc.
55.6
65.1
78.6
50.2
62.4
BCE
13.1
7.10
2.98
12.8
9.00
22

---

Published as a conference paper at ICLR 2024
A.11
ADDITIONAL EXPERIMENTAL RESULTS: STANDARD DEVIATION
The followings are the standard deviation of the experiments done in Appendix A.9
Table 11: Standard Deviation of the initialization using 'an example of'.
ImgNet
Caltech
Flower
Food101
Aircraft
SUN397
EuroSAT
UCF101
Pets
Cars
DTD
Method
CIJP-RN50HtrdRao
Acc.
..09
1.16
1.25
1.20
1.18
1.23
1.32
1.
2.
1.
1.17
+TP IPTHopPpmpp
RCE
+.1
+.
14
13
2.22
+313
18
4
12
4
4.25
+
14
Acc.
2.10
1.20
1.
E.24
1.16
1.16
1.21
1.22
1
22
1.
13
1
+TPT PIPPitandDoopptC-- CCTTPT
ECE
1.
1.10
1.12
1.11
1.17
2.24
1.25
1.16
1.20
2.20
22
CIJP-ViT-B/ HIP-Vii-B/1Gganddeer 61lrdPpoopp
Acc.
+.22
2.17
2.21
2.22
22
1.15
1.21
1.25
1.
1.14
1.17
+TPT HaaddPoompt
ECE
+.2
1.12
2.25
2.20
1.
1.
13
1.14
1.21
1.20
1
1.18
Aec.
-.20
1.16
1.17
1
11
1.15
1.23
1.20
1
22
1
23
1.17
1.13
+TP itaattaePooppp CCTTPP +C-TPT
ECE
1.21
1.15
1.11
2.
1.17
1.11
1. 16
1
2.24
Table 12: Standard Deviation of the initialization using photo of the cool'.
Acc.
1.10
1.19
1
1.
13
1.29
1.21
1.28
1.24
1.15
1.19
+TP HardProppt
ECE
1.16
1.13
4.
18
1
19
1.21
1
16
1.13
4.31
1.22
1.23
1.17
Ačc.
=.11
E.14
E.26
E
E.28
1.
3.14
3.25
4.15
2.18
2.
E.
PPTTHaPPoopp CCTTP CCTTT
ECE
1.19
1.14
4.21
2.29
4.28
4.20
1.12
1.25
1.14
1.
1.27
TLIP-ViT-B16Gaannaae
Acc.
+.08
+.28
4.27
2.21
2.15
#.26
2.29
4.32
2.24
2.21
2.21
+TPT HandPPompt
BCE
1.13
1.18
1.24
1.18
1.13
1.12
1.30
4. 16
1
25
1
16
1.14
Acc.
-.1
1.19
1.18
E
13
2.26
30
4 19
4.30
4
28
23
TTP +C-TPT
ECE
1.
2.22
12
+ 22
30
+ 28
22
4 24
18
1.
22
Table 13: Standard Deviation of the initialization using CoOp and CoCoOp.
CLIP-RN5OCooz
Acc.
I.
E.22
2.20
1.26
1.32
1.24
2.25
1.26
1.13
1.21
1.
+TPTcoop
ECE
+.23
1.24
1.15
1.12
1.12
1.15
1.17
2.24
1.15
1.13
1.12
Acc.
+.22
1.19
4.15
4.31
1.13
+IPTcc0pt--IPT
ECE
1.17
1.14
1.20
1.22
1.19
4.11
1.17
4.16
4.22
4.15
1.13
CLIP-RN50CccCO
Acc.
1.24
E.32
1.27
E.35
1.2
4.31
1.10
E.20
+TPTcccO0
ECE
1.11
1.12
24
4.
4.
1.19
1.10
Acc.
1.34
1.17
134
1.
13
1.20
1.30
1.16
+CPTCG
ECE
1.15
1.12
1.23
1.16
1.
1.20
1.15
1.15
1.19
1.19
CLIP-VIT-B/IGccco
Acc.
7.23 4.29
4.25
4.33
4.1
15
4.3
30
23
17
+.30
1.11
4.26
+TPTccop
ECE
+.11
+.11
4.24
4.21
4
+.22
+
08
+.
19
1.12
2.23
1.
Acc.
1.32
1.18
1.0
1.20
1.29
-C-ToOppC-TPT
ECE
+.22
+.14
1.19
4.15
4.19
1.25
2.77
4.00
4.23
4.08
1.13
CLIP-VIT-B/IGccoon
Acc.
+.22
F.29
4.
4.5
4.17
E.3
*TPCccoOp
ECE
+.17
+.19
-.14
4.
4.
4.22
+.15
Acc.
#:21
22
22
2.23
4CTTT
BCE
+.24
+.18
1.
1.12
1.14
1.12
22
1.17
09
15
13
23

---

Published as a conference paper at ICLR 2024
Table 14: Standard Deviation of applying PromptAlign on Fine-grained Classification Task.
EuroSAT
ImgNer
Caltech
Flower
Food10)
Aireraft
SUN397
UCFI01
Pets
Cars
DTD
Method
CLIP-ViT-B/ FLIP-ViT-BIloaddeee Buaddoon
-PromptAli@eeadPoaare
Acc.
1.18
1.16
1.19
4.21
4.23
1.14
14
2.20
4.21
1.17
1.15
1.16
ECE
1.
1.14
2.22
1.18
1.
12
15
2.20
1.
13
1.17
Acc.
1.19
1.15
1.18
1.
1.14
22
19
2.21
2.22
1.16
1.14
trrnmmt +C-TPT
ECE
+.20
1313
1.
4.19
1.16
18
+.
1.18
4.15
1.17
2.
Table 15: Standard Deviation of applying PromptAlign on Natural Distribution Shifts.
Method
ImageNet-A ImageNet-V2 ImageNet-R ImageNet-Sketch
CLIP-ViT-BI WTT-B/16taddPrree
PPromptAlignttddPonnn
Acc.
1.13
+.15
+.20
=.21
ECE
+.18
+.22
#.11
=.24
1.10
1.15
1.19
1.19
PrromptAligmunneammpp CCTTPT
ECE
+.07
:.16
#.11
#.17
24

---

Published as a conference paper at ICLR 2024
A.12 APPLICABILITY OF C-TPT ON TRAINING-TIME PROMPT TUNING METHOD
Poodiut
Floner
Oflirr
SUN3Y?
ImageNet ATFD, rank
Plot illustrating the correlation between the ATFD rankings from ImageNet and those from various other datasets for 300 different hard prompts. A notable positive correlation evident, as indicated by an average Spearman rank correlation coefficient Zar 1972) of 0.92.
Figure 9:
Training-time prompt tuning methods like CoOp (Zhou et al. 2022b) typically involves training the prompt using a subset of ImageNet samples, emphasizing its generalizability across various datasets. This process often leads transition to datasets with differing class types. this context, our focus turns to whether our proposed C-TPT regularization, which depends the class types (as outlined in Equation can apply to training time prompt tuning methods. The core our evaluation in this section lies determining whether the dispersion text features among classes in new test dataset can be generalized. For instance, if a prompt shows high text feature dispersion ImageNet, does it replicate this behavior in other datasets like Caltech, Pets, or Food101?
2022a and CoCoOp (Zhou et al.
We began with a detailed analysis of text feature dispersion using the prompts outlined in Appendix A.4 specifically focusing on the ImageNet classes, Our methodology involved ranking each prompt according to its text feature dispersion within the ImageNet classes. Subsequently, we extended to other datasets, ranking each prompt in the same manner based on its text feature dispersion in these other dataset classes. This process allowed to directly compare each prompt's text feature dispersion ranking across different datasets with initial ranking in ImageNet, providing a comprehensive view of their relative performance in diverse contexts. Our findings, as depicted in Figure reveal a consistent trend: prompts that demonstrate high text feature dispersion within one dataset consistently exhibit similar level of dispersion across other datasets.
This underlines the possibility of applying C-TPT regularization in training-time prompt tuning broadly apply methods. By identifying prompts with high textual dispersion in dataset, proposed C-TPT this across diverse datasets. In this section, we perform the CoOp training with our on the cross-domain generalization setting proposed by Zhoo al. (2022a), which uses 16 samples per class of ImageNet to train a generalized prompt. We train on CLIP-RNSO with batch size 128, learning rate 0.002, and epoch 50 using SGD optimizer. The regularizer strength of C-TPT, which is denoted by X, is set to 1. The result in Table I6 shows that C-TPT can work as a regularizer for better calibration error for the training- time prompt tuning method.
Table 16: Fine-Grained Classification. We report the Aec. (1) and ECE ()) of CoOp and CoOp + C-TPT on the CLIP-RN50. The values highlighted in bold signify the best ECE.
ImageNer
Caltech
Flower102
Food101
Aireraft
SUN397
EUrSAA
UCF101
Average
Pets
Cars
DTD
Method
Acc.
63.3
86.5
87.0
55.3
61.4
75.6
15.1
58.1
37.3
26.2
59.0
56.8
CoOp
ATFD
0.64
0.60
0.65
0.51
0.69
0.58
0.43
0.51
0.41
0.33
0.55
0.54
BCE
3.00
3.32
6.75
7.27
5.01
3.06
7.20
3.59
12.4
20.2
5.02
6.98
ACc.
62.4
87.9
87.0
55.3
61.8
74.6
14.0
59.2
36.3
25.3
58.2
56.5
Co0p+C-TPT
ATFD
0,92
0.91
0.66
0.74
0.75
0.67
0.60
0.64
0.65
0.56
0.71
0.71
ECE
1.48
3.09
6.03
1.79
3.55
2.98
7.01
3.05
12.0
19.2
4.00
5.83
25


Reviews:
[{"review_id": "PFSsP56hjX", "reviewer": "Reviewer_fxyk", "summary": "This paper studies an interesting problem related to test-time prompt tuning of a vision-language model. Through a series of observations, this paper finds out that the calibration is linked to the choice of prompts. Therefore, this paper attempts to improve the calibration error (ECE) by using the average dispersion of text features. The main idea is to distribute the text features in the feature representation space. The experiments show that the proposed method successfully improves the ECE of test-time prompt methods. The topic of this paper is relatively interesting and proposes a novel perspective for prompt tuning.", "questions": "Please refer to Weakness 1 & 2. I will raise my score if you address my concerns.\n\n[UPDATE]\nI have raised my score.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "1. The solution (i.e., ATFD) of this paper is reasonable. The choice of prompts affects the ECE metric, indicating that the feature representation is well-calibrated. Therefore, prompt tuning for placing textual class vectors separately is important.\n2. The experiments of this paper are comprehensive, and the improvement of ECE is significant. Both the ablation study and additional analysis can provide valuable information to readers.\n3. The presentation of this paper is good, and the technical content is easy to follow.", "weaknesses": "1. The experimental results show that the accuracy (ACC) of C-TPT drops in many cases. However, the ECE and ACC are a trade-off in most cases. The author should explain whether the drop in accuracy is worth the improvement of ECE. My advice is that authors can provide a Pareto frontier of ECE vs. ACC for C-TPT methods and hand-crafted prompts to demonstrate that C-TPT achieves a superior solution, rather than simply obtaining a prompt with higher ECE and lower ACC.\n2. The related work should discuss a related field, i.e., Test-time Adaptation [1-5].\n\n**Reference:**\n\n[1] Shuaicheng Niu, Jiaxiang Wu, Yifan Zhang, Yaofo Chen, Shijian Zheng, Peilin Zhao, Mingkui Tan: Efficient Test-Time Model Adaptation without Forgetting. ICML 2022: 16888-16905\n\n[2] Hao Zhao, Yuejiang Liu, Alexandre Alahi, Tao Lin: On Pitfalls of Test-Time Adaptation. ICML 2023\n\n[3] Zhi Zhou, Lan-Zhe Guo, Lin-Han Jia, Dingchu Zhang, Yu-Feng Li: ODS: Test-Time Adaptation in the Presence of Open-World Data Shift. ICML 2023\n\n[4] Tong Wu, Feiran Jia, Xiangyu Qi, Jiachen T. Wang, Vikash Sehwag, Saeed Mahloujifar, Prateek Mittal: Uncovering Adversarial Risks of Test-Time Adaptation. ICML 2023\n\n[5] Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno A. Olshausen, Trevor Darrell: Tent: Fully Test-Time Adaptation by Entropy Minimization. ICLR 2021", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vrWXnszrn3", "reviewer": "Reviewer_H39i", "summary": "This paper first makes observations on the effect of text prompt to the CLIP model calibration (consistency of predicted confidence with the real accuracy), where there is a negative correlation between the dispersion of the text features and the calibration error. Then it proposes the Average Text Feature Dispersion (ATFD) to measure this dispersion of the text features. It then proposes a loss function based on the ATFD to increase the ATFD thus improving calibration during test-time adaptation. The approach is validated on different datasets with CLIP models and shows the improved results with smaller calibration error and similar accuracy compared to an existing test-time prompt tuning methods.", "questions": "Is there specific reason to prevent this method to be used for other prompt tuning methods such as CoOp and CoCoOp (Conditional Prompt Learning for Vision-Language Models) ?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "-\tThe authors make several observations before introduction their technique to improve the calibration property of the model. The observations (figure 1&2) are intuitive and clearly shows the correlation between the calibration error and the ATFD metric.\n-\tThe proposed ATFD loss combined with an existing test-time adaptation method shows consistently improved results on various downstream tasks (ImageNet, Caltech, Pets etc)\n-\tPaper writing is clear and easy to understand. The key claims are well supported by the evidence from the experiments.", "weaknesses": "-\tThe application of improving calibration on just one algorithm (test-time prompt tuning) is kind of limited. I would expect this method to help improve different prompt tuning methods such as CoOp. Additional experiments on other prompt tuning tasks are strongly recommended to improve the impact of this work.\n-\tFigure 5 is problematic. First, the color scheme of Figure 5 is incorrect. The legends of (a) and (b) both have the black color, but the dots in the prompt visualization figure does not have the black dots. And the class visualization is weird. Do the authors want to show the class of some samples on the visualized plan? Then the location of the class samples should be different from the class prompt. However, I find that in Figure 5(a) the class samples are perfectly aligned with the class prompts.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Rq3DmXUfYH", "reviewer": "Reviewer_eM6j", "summary": "This paper focused on the calibration issue in prompt tuning of vision-language models, especially for test-time prompt tuning, which is not fully explored in previous studies. The main findings are (1) test-time prompt tuning increases the calibration error, (2) the variance of calibration is high with different hard prompts, (3) calibration and text-feature dispersion are highly correlated. Based on the third finding, this work proposed a regularization term based on the text feature dispersion during test-time prompt tuning. Experiments on fine-grained classification and distribution shift datasets show that the proposed method achieves comparable performance with test-time prompt tuning method while lowering the calibration error.", "questions": "Please see Weaknesses for detailed comments.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "+ The calibration error issue in test-time prompt tuning is not fully discussed in previous works. The research problem is novel and interesting.\n\n+ The proposed regularization term is simple but effectively decreases the calibration error.\n\n+ Experiments on fine-grained classification and distribution shift datasets show that the proposed method achieves comparable performance with test-time prompt tuning method while lowering the calibration error.\n\n+ The paper is well written and organized. It is easy to follow the idea and figure out the main massages.", "weaknesses": "- Although the contributions of this work are obvious, it is somehow limited. The main messages I took from this paper are (1) this work pointed out the calibration error issue for test-time prompt tuning of VLMs, (2) provided an analysis on the relations between prompts and calibration error, (3) this work found the correlations between text-feature dispersion and calibration. Although the analysis is interesting and novel, it is not comprehensive enough. It would be interesting to know whether training-time prompt tuning also suffers from calibration error. If so, does the proposed method work for training-time prompt methods. If not, what is the reason. Also, the conclusions are empirical rather than theoretical. It is okay if the method is motivated merely on empirical observations, but the lack of theories decreased the contributions of this work.\n\n- The proposed method is only adopted on TPT method. Do other test-time prompt tuning methods suffer from calibration error? Can the proposed method be applied to other methods? Only one method is discussed is not comprehensive to draw the conclusions.\n\n- This work used 80 hard prompts to evaluate the impact of prompts on calibration error. However, according to the prompts shown in Appendix A.3, some of the prompts are not optimal. For example, \"a black and white photo of a {class}\" is not suitable to colored photos, and \"a sketch of a {class}\" may not work for photos. I am wondering whether the discrepancy between prompts and calibration error are due to the bad choices of prompts. If so, the conclusions about the relation between prompts and calibration error may not be convincing enough.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "I5lKFFKJRu", "reviewer": "Reviewer_CqA1", "summary": "This paper aims to calibrate the output of CLIP during test time. The authors discover the negative relationship between ATFD and ECE and use ATFD as regularization during test time prompt tuning.", "questions": "One interesting thing is that CLIP (before TTA or fine-tuning) achieves the lowest ECE most time. It would be nice if the authors could shed light on this point.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-written.\n2. I appreciate the story of the paper.\n3. I recognize the contribution of the paper --- calibration during test time. The proposed method is simple but effective. They maintain competitive performance with TPT and achieve better calibration.", "weaknesses": "Although I recognize the contribution of the paper, one practical problem is that the proposed method does not improve the metrics --- no positive influence is observed besides the ECE. A lower ECE does not show its significance in the tasks of the paper unlike the mentioned medical application. Hope authors can shed more light on this weaknesses", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)