A-TPT: Angular Diversity Calibration Properties for Test-Time Prompt Tuning of Vision-Language Models
Authors: Shihab Aaqil Ahamed
Udaya Sampath K. Perera Miriya Thanthrige
Ranga Rodrigo
Muhammad Haris Khan
Venue: ICLR 2026 Poster (2026)
Keywords: Model Calibration
Angular Diversity
Uniformity
Vision-Language Models
CLIP
Prompt Tuning
Test-Time Adaptation
Primary area: alignment, fairness, safety, privacy, and societal considerations
TL;DR: We propose A-TPT, a novel angular diversity method that improves calibration for test-time prompt tuning of vision-language models by maximizing minimum pairwise angular distance among textual features.

Abstract: Test-time prompt tuning (TPT) has emerged as a promising technique for adapting large vision-language models (VLMs) to unseen tasks without relying on labeled data. However, the lack of dispersion between textual features can hurt calibration performance, which raises concerns about VLMs' reliability, trustworthiness, and safety. Current TPT approaches primarily focus on improving prompt calibration by either maximizing average textual feature dispersion or enforcing orthogonality constraints to encourage angular separation. However, these methods may not always have optimal angular separation between class-wise textual features, which implies overlooking the critical role of angular diversity. To address this, we propose A-TPT, a novel TPT framework that introduces angular diversity to encourage uniformity in the distribution of normalized textual features induced by corresponding learnable prompts. This uniformity is achieved by maximizing the minimum pairwise angular distance between features on the unit hypersphere. We show that our approach consistently surpasses state-of-the-art TPT methods in reducing the aggregate average calibration error while maintaining comparable accuracy through extensive experiments with various backbones on different datasets. Notably, our approach exhibits superior zero-shot calibration performance on natural distribution shifts and generalizes well to medical datasets. We provide extensive analyses, including theoretical aspects, to establish the grounding of A-TPT. These results highlight the potency of promoting angular diversity to achieve well-dispersed textual features, significantly improving VLM calibration during test-time adaptation. Our code is available at https://github.com/MB-Shihab-Aaqil-Ahamed/A-TPT/.

Published as a conference paper at ICLR 2026
A- T-TPT: ANGULAR DIVERSITY CALIBRATION PROP- ERTIES FOR TEST-TIME PROMPT TUNING OF VISION- LANGUAGE MODELS
Shihab Aaqil Ahamed 1.2
Udaya S.K.P. Miriya Thanthrige'
Ranga Rodrigo'
Muhammad Haris Khan
Dept. of Electronic and Telecommunication Engineering, University of Moratuwa, Sri Lanka 2 Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, UAE
sh wtw.aaaagllaaaaeeeeee
com
mb -
Project aage
11t.0-ooAAPPII A-TPT/
ABSTRACT
Test-time prompt tuning (TPT) has emerged as a promising technique for adapt- ing large vision-language models (VLMss) to unseen tasks without relying on labeled data. However, the lack of dispersion between textual features can hurt calibration performance, which raises concerns about VLMs' reliability, trust- worthiness, and safety. Current TPT approaches primarily focus on improving prompt calibration by either maximizing average textual feature dispersion enforcing orthogonality constraints to encourage angular separation. However, these methods may not always have optimal angular separation between class-wise textual features, which implies overlooking the critical role of angular diversity. To address this, we propose A-TPT, novel TPT framework that introduces an- gular diversity to encourage uniformity in the distribution of normalized textual features induced by corresponding learnable prompts. This uniformity is achieved by maximizing the minimum pairwise angular distance between features on the unit hypersphere. We show that our approach consistently surpasses state-of- the-art TPT methods in reducing the aggregate average calibration error while maintaining comparable accuracy through extensive experiments with various backbones different datasets. Notably, our approach exhibits superior zero- shot calibration performance on natural distribution shifts and generalizes well to medical datasets. We provide extensive analyses, including theoretical aspects, to establish the grounding of A-TPT. These results highlight the potency of pro- moting angular diversity achieve well-dispersed textual features, significantly improving VLM calibration during test-time adaptation. Our code is available at nttps://gi thubb com Shihab Aaqil AAhamed/--aaaa A-TPT/
INTRODUCTION
Foundational large-scale vision-language models (VLMs), such as CLIP RRdford et al. 202
ALIGN (Jia et al. 2021). and FILIP (Yao et al., 2021). have demonstrated remarkable zero-shot wide range of downstream tasks (Jia al., 202T Radford al. 202T).
inference capabilities
These models are pre-trained with contrastive learning on massive web-scale data e.g. 400 million image-text caption pairs-
align visual and textual modalities within a shared multimodal latent space. This alignment allows VLMs to classify instances from novel visual categories in a zero-shot setting realized by carefully constructed textual prompts - hand-crafted class-conditioned templates (c.g., "a photo of [CLS] "") - crucial for effective zero-shot transfer. However, manually designing such prompts often requires domain-specific heuristics and may not be optimal across diverse tasks (Shu et al. (022)..
To address these limitations, recent works have explored prompt tuning that learns prompts from training data specific to downstream tasks (Zhou et al. 2022a b). However, such approaches often rely on annotated data, which can be expensive and searce for zero-shot scenarios (Socher et al. 2013). To address this challenge, test-time prompt tuning (TPT) (Shu et al. 2022 has garnered

---

Published as a conference paper at ICLR 2026
Auglla wotaeroaaaaaaa ATTrveree MEM 13
Euedar
Fevenbio
Tees Seallca cotto
Foodses
3:6m Adn-saazndaa
Sacon can
CARWAAOO
SUN0R7
Figure 1: Comparison calibration perfor- mance (ECE) with C-TPT (Yoon et al. 2024). and O-TPT (Sharitdeen et al. 2025) on fine- grained classification datasets with CLIP ViT- B/16 backbone. Ours (lower ECE) shows im- proved prompt calibration.
Seove
Figure 2: Comparison of numerical optimization (P-TPT (Ours)) with angular optimization (O-TPT Sharifdeen al. (2025) and ATFD optimization (C-TPT Yoon al.
)224
significant attention focused prompt tuning. TPT optimizes learnable prompt vectors through gradient descent that adaptively refines them during inference with unlabelled test image samples adapt VLMs to novel tasks. Although TPT boost the accuracy of VLMs, it often suffers from poor calibration, where the model's predicted confidence does not reliably reflect the true accuracy (Guo et al., 2017 Murugesan al. 2024 Yoon al. 2024). Such miscalibration can result in overconfident predictions, raising concerns about the reliability and trustworthiness of VLMs, particularly for real-world safety-critical applications that require reliable uncertainty estimates, including medical diagnostics (Ji et al. 2021 Wang et al. 2022. Zhang et al. 2023 Chen et al. 2023
Liu et al., (203)) and autonomous driving (Dorbala et al. [022] Gadre et al. 2022] Khandelwal et al.,
2022 Bucker et al.] 2023: Cui et al.
2024 You et al. [2224] Zhou et al. 2024). To date, calibration in
test-time prompt tuning of VLMs less explored, with limited efforts to address it.
To do better calibratton, prior works, such as C-TPT (Yoon et al. 2024) and O-TPT (Sharifdeen ctt al. 2025) have explored methods to encourage dispersion between pairwise textual features, which can be categorized into the following two types: The first type, known as Average Textual Feature Dispersion (ATFD), spreads textual features away from their centroid. However, this can still result in textual features lying closely together (Fig. and cause poor calibration performance (Fig. TT. The second type enforces orthogonality constraits to encourage angular separation, which exploits an auxiliary orthogonal regularization term in the loss function encourage pairwise textual features as orthogonal as possible. However, we observe that it tends to group textual features closer, particularly when the number of classes N is greater than the embedding dimension |D] (c.g.. (N > [D), where CLIP's ViT-B/16 512-d (Radford et al. 2221 Liang et al. 2022) vs. 1000 classes in ImageNet- Ik, V2, K) does not guarantee uniformity of angular separation (Fig.) 2 4b). When the number of classes is less than the embedding dimension ((N < DD), e.g.. classes: 10 in ETroSAT, 37 in OxfordPets, and 47 in DTD), fails to fully utilize the hyperspherical space of feature points effectively across the hypersphere (Fig. 2 Ab). This eventually leads to poor calibration (Fig. D Although these methods increase feature dispersion to some extent, they often overlook the importance of angular diversity. Without sufficient angular separation, prompts may become highly correlated, which limits the model's 's ability to generate well well-calibrated predictions. Prior work (Wang & Isola] (0200) has shown that uniformity (uniformly distributed feature points the unit hypersphere) preserves maximal information, closely associated with strong zero-shot CLIP performance.
The uniformity problem is well-studied in Tammes problem (Tammes 1930) (best- blest-packing), that is to find the optimal arrangement of a given number of feature points on the surface of a unit hypersphere such that the minimum distance between any two points is maximized. Inspired by this insight, we propose A-TPT, a numerical optimization approach that introduces a simple yet effective angular diversity into the test-time prompt tuning framework. Our method maximizes the minimum pairwise angular distance between normalized textual features on the unit hypersphere to promote uniform and diverse prompt distribution, fully utilizing the hyperspherical space (Fig. 2 By penalizing closely aligned prompt directions, the proposed A-TPT promotes the greatest possible angular distance between them (Fig. (b), thus achieving better prompt calibration performance (Fig. []. Notably, by maximizing the minimum angular distance between prompt vectors, A-TPT naturally solves the number of classes exceeding the embedding dimension problem, e.g., 1000 classes in a 512-d space, making all prompt vectors pairwise orthogonal impossible. In such cases, hypersphere can still achieve a good class separation in A-TPT maximizing angular distance leads to better performance (Fig. ())
2

---

Published as a conference paper at ICLR 2026
Our major contributions are summarized as follows:
We introduce a numerical optimization method, called A-TPT, for better calibration of test-time prompt tuning for VLMs. This resolves the suboptimal performance of existing leading calibration techniques for test-time prompt tuning.
We introduce novel angular diversity that effectively promotes the diversity among textual features, thereby improving the calibration capabilities of VLMs when N > |D] and N |D]. This is accomplished by maximizing the minimum pairwise angular distance between normalized textual features.
We conduct extensive experiments to validate the generalizability of our approach on different datasets, including medical datasets, across various baselines. The results show that A- A-TPT surpasses state-of-the-an methods in calibration performance. We also provide thorough analyses, including theoretical aspects. Moreover, our approach provides superior calibration compared to the zero-shot CLIP model, which reveals improved calibration.
RELATED WORKS
2021) and ALIGN (Jia itt al. 2021) are pre-trained on extensive image-text pair datasets to learn a shared multimodal latent space (Menon & Vondrick 2022), thereby enabling strong zero-shot performance through prompt- based inference. In large vision-language models (VLMs), predictions are guided by hand-crafted textual prompts that require Nomain-speeifer heuristics. While effective, manually designed prompts may be suboptimal across various newer domains. To address this, prompt tuning techniques treat prompts as trainable vectors and optimize them via gradient descent. Notably, CoOp (Zhou et al.) 2022b) introduced supervised prompt tuning framework for CLIP (Radford et al., 2021). which improves the classification accuracy by leveraging labeled training samples. However, follow-up work CoCoOp (Zhou et al. 2022a) showed that CoOp (Zhou et al. 2022b) struggles to generalize to out-of-distribution (OOD) data and proposed input
Prompt tuning for large VLMSs. Large VI.Ms such CLIP (Radford et al.
prompts to enhance the model's ability to adapt to new, novel domains. Despite these advances, such methods rely on annotated training data, which limits their utility when working with pre-trained models in zero-shot settings. To address this gap. Test-time Prompt Tuning (TPT) (Shu et al.. (2022) has been introduced to enable on -n-the-fly adaptive prompt adaptation using just one unlabelled test image sample during inference. TPT optimizes prompts by minimizing prediction entropy and boosts model accuracy in zero-shot scenarios. However, recent works Yoon et al. [2024, Sharifdeen et al. 2025) have revealed that it leads to poorly calibrated, overconfident predictions.
Calibration of deep neural networks. Model calibration evaluates how well the model's predicted confidence aligns with the actual likelihood of correctness (Guo et al.] (017). Well-calibrated predictions are crucial in high-stakes applications such as healthcare, autonomous systems, and safety-critical systems (Ghahramani] 2015), where reliable uncertainty estimation is crucial for trustworthy decision decision-making, Calibration techniques for deep neural networks can be categorized into two categories: post-hoc hoc and train-time methods. Post-hoc calibration strategies, such as temperature scaling (Guo al.) 2017). platt scaling (Platt et al., 1999), and conformal prediction (Vovk et al.) 2005; Lei et al. 2018) calibrate a model's prediction confidence after training using a held-out validation set. However, these methods rely on access to labeled datasets collected from distribution similar to the target data (Liu et al. (022)), often impractical in zero-shot and -uitt--dist (OOD) contexts. In contrast, train-time calibration methods integrate a hybrid calibration objective into the training of deep neural networks, with an auxiliary calibration loss as regularizer in conjunction with the primary training loss. These include techniques (Kumar et al., 2018 Munir et al. 2022: 2023 Yoon et al. 2023) (i.e., for object classification and detection) that incorporate differentiable auxiliary regularization loss functions during training to reduce calibration error for reliable predictions. However, these train-time calibration methods are supervised and require labeled training data, limiting their applicability in the test-time prompt tuning of VLMs without supervision.
Calibration of large VLMs. Large VLMs such as CLIP demonstrate strong zero-shot performance by leveraging seree-scale pretraining on image- text pairs. Despite the efficacy of VLMs in generalizing to new tasks, they often suffer from poor calibration. Recent works have shown that test-time prompt tuning (TPT) (Shu et al., 2022) can boost task task-specific accuracy in zero-shot settings; however, could increase the model's overconfidence by expanding the logit range during inference. To address
3

---

Published as a conference paper at ICLR 2026
this, (Murugesan et al. 2024) have proposed logit normalization strategies. For example, zero-shot logit normalization adjusts the model's 's prediction confidence by refining the logits with (original) zero-shot baselines, while -ampleeddaptive logit scaling dynamically calibrates the normalized logits per instance to reduce overconfidence during inference. In parallel, C-TPT (Yoon et al. 2024) explored the relationship between textual feature dispersion and model calibration and proposed Average Text Feature Dispersion (ATFD) loss to maximize inter-class dispersion. It has been observed that well-calibrated prompts tend to generate class-specific text embeddings that are more widely separated in embedding space. To this end, the Average Text Feature Dispersion (ATFD) loss was introduced to maximize inter-class dispersion. Although ATFD effectively reduces calibration error without compromising accuracy. it may struggle in challenging cases, where it is limited in establishing enough dispersion and fails to sufficiently utilize the embedding space. To address these shortcomings, sethogonality-basee regularization has been proposed (Sharifdeen et al. (025), which enforces orthogonality constraints to encourage angular separation between textual features to promote greater dispersion. While this improves feature dispersion to some extent, it tends to group prompts closer when the number of classes is greater than the embedding dimension and does not guarantee uniform angular separation across all prompt vectors, resulting in poor calibration. Encouraged by these insights, we introduce a novel angular diversity technique that explicitly maximizes the minimum pairwise angular distance between normalized prompt vectors. This method promotes a more uniform and diverse distribution of prompts on the hypersphere. and hence better utilizes the embedding space, thereby consistently achieving better calibration performance during test-time adaptation without relying on labeled data.
PROPOSED METHOD
Zero-shot classification with large VLMs (CLIP). CLIP (Radford et al. 2021) consists of two encoders: an image encoder (Si) and a text encoder (ft), which map visual and textual inputs into the corresponding feature space vectors. The model is pre-trained with contrastive learning that maximizes the cosine similarity between corresponding image-text feature vectors, thereby aligning the visual and textual modalities within a shared multimodal latent space. In the zero- shot setting with CLIP, class-related textual prompts are constructed with hand-crafted templates - e.g.," "a photo of [CLS] " where " [CLSS] corresponds to the name C& of each possible N
class C = {c+ k=1 from predefined set of classes to classify images. Next, each textual prompt P& prompt (c&) corresponding to a specific class cs is fed into the text encoder (Jx) to generate the textual feature vector: 4 = Je (px). Simultaneously, a given test image x is fed into the image encoder (ft) to generate the image feature vector (x) To classify the image, cosine similarities 9y = sim (v, (x) are computed between the image feature vector v and each class-specific text feature vector tg. These similarity scores are then converted into probabilities of predicting class Ok for the test image x using a Softmax function controlled by a temperature parameter т, which is fixed at 0.01 during inference. Then, the predicted class becomes the class with the highest probability, é = arg max,, p(c+ x) with its associated predicted confidence is p maxey P(c, xr) This zero-shot framework enables efficient classification across wide range of categories without additional fine-tuning. relying solely on the learned multimodal alignment between images and textual prompts. In contrast to hand-crafted prompts (i.e., hard prompts). prompt tuning has been explored in CLIP (Chen et al. 2022 Radford et al. [202T; Yao et al. 2023 Zhou et al.) 20226 a) to optimize trainable prompt embeddings using 16 samples per class from the ImageNet dataset, which allows the learned prompts to generalize across cross cross-datasets. Recently, TPT (Shu et al.) 2022) has enabled prompt tuning without labeled data during inference. Although it boosts accuracy, TPT often raises calibration errors due to overconfident predictions (Guo et al. 2017
Expected calibration error and evaluation metric. Lower expected calibration error (ECE) (Naeini et al.) 2015) indicates perfect calibration, where the model ensures that predicted probabilities correspond accurately to the likelihood of true accuracy. We define this as: (2 = C' C f= = p. p. [0, 1], where input image x with its corresponding ground truth label c, predicted class label é with its predicted confidence = ý. Expected Calibration Error (ECE) partitions predictions into bins based on the confidence metric and evaluates the absolute difference between the accuracy and the mean confidence within each bin. Mathematically, ECE is formulated as N
follows: ECF = Σ
|all Jaco (Bn)= - conf (Bn)l, where N denotes the total number of bins, B, M
n=l
4

---

Published as a conference paper at ICLR 2026
a tiieaal (Me:: 2322 DCE: 56, AG: 0.6421
tay Hes: 12.8. 1.65, 0.8821
ATTE 75.000 PESS FOF: 2-71 6APG
thane suuca fAce: 61.1 SCE: 9:23 5.2)
the teertstttaas 1his Image (ckoal (Acs: 90.1 DCE: 10,4, AD:014612
Prompt Viauatization
Clasa Vausttztion
Pronept Mosuttaation
Class Visuatization
(a) Hard prompts
(b) Tuned prompts
Figure 3: t-SNE visualization of class-wise embedded textual features with CLIP RN50 model the fine- een-ggained classification dataset (Fei-Fei al. (004) for (a) hard prompts and (h) tuned prompts. In both subfigures, each unique color represents a distinct prompt the prompt visualization (left) and distinct class the class visualization (right). The legends belong to the prompt visualization (left) for hoth subfigures.
defines the image set with predicted confidence falling into the n-th bin, [B,] is the number of images within this bin, and M is the total number of predictions, Furthermore, acc (B,), and conf (B,) represents the accuracy of the predictions and average prediction confidence of all associated with bin n, respectively.
Why is angular diversity better for prompt calibration? While different prompts may yield text dispersion that results in comparable classification accuracies, their calibration performance can vary significantly. To further understand the relationship between angular diversity - maximizing the angular distance (AD) and expected calibration error (ECE) we conduct experiments with 80 different hard prompt styles (Radford et al. (021). We followed the C-TPT (Yoon et al.) (024) to evaluate the impact of angular distance (AD) and calibration error within the same accuracy group. Specifically, we focus on the 3 well-calibrated ('a', 'a doy',"thi" is a photo of") and 2 poor-calibrated ('there are [CLS] ob jecct'', 'the nearest. shape in this image is') prompt styles that provide lower and higher ECEs. For illustration, consider the following examples from the Caltech101 dataset, using CLIP RN50 are categorized into well-
calibrated and poor-calibrater prompts:
Hard prompts (See Fig. 3a legend)
Tuned prompts (See Fig. 3b lelendd)
[CLS] Acc: 83.2, ECE: 5.66, AD: 0.643
toy [CLS]: TPT Acc: 83.9, ECE: 6.18, AD: 0.6216
toy [CLS] Acc: 82.8, ECE: 6.65, AD: 0.600
this is photo of [CLS]: TPT A-TPT Acc: 86.0, ECE: 3.74, AD: 0.6333
this is photo of [CLS]
Ace: 82.4.
ECE: 6.15, AD: 0.622
a [CLS]: [OLS]:TPT:: TPT + A-TPT Acc: 86.6, ECE: 2.23,
there are [CLS] objects - Acc: 81.1,
AD: 0.6486
ECE: 9.25, AD: 0.543
there are [CLS] objects: TPT A-TPT Ace: 84.1, ECE: 2.12, AD: 0.6585
the nearest shape in this image is [CLS]] Acc: 80.1, ECE: 10.84, AD: 0.461
While well-calibrated prompts typically yield higher accuracy, the calibration error varies significantly within the same accuracy group. Similarly, poll-calibrated prompts tend to yield lower accuracy, and the calibration error varies significantly within the group. That is why in our analysis in Fig. 3 we collected the prompts that yielded similar accuracy and tried to determine what caused the difference in the calibration eгror within the same accuracy group.
We visualize t-SNE of class-wise embedded textual features in Fig. 3 In the Prompt Visualization (left), each point represents a text feature from different prompts for all possible classes. In the Class Visualization (right). the class embeddings represent text features corresponding to the actual class labels. These visualizations illustrate the distribution of different hard prompt vectors in the feature space, providing insights into how the angular diversity affects model calibration. In the case of Fig. (a we observed distinct pattern; poorly calibrated prompts - low angular diversity are clustered closely together regardless of their corresponding class labels. This low angular diversity results in poor calibration, as the text features become highly correlated. In contrast, well-calibrated prompts- - high angular diversity - are well dispersed across the feature space, as shown Fig, Ba (left). Notably, the features for well-calibrated prompts tend to group cohesively associated with their class labels (Fig. (a (right).. These well- dis--deeereed features allow the model to achieve better calibration, the features align closely with their respective class label locations in the feature space.
5

---

Published as a conference paper at ICLR 2026
This pattern suggests that angular diversity disperses class-specific prompts and clusters text features near their corresponding class labels, reducing calibration errors. In Fig. 3b when applying Test-time Prompt Tuning (TPT), the text features tend to cluster together, similar to the poorly calibrated hard prompts. However, as we introduce angular diversity (A-TPT) and progressively increase the regularization strength during the optimization process (via A), the angular distance between the features increases, As shown in Fig. (lefff)) A-TPT results in greater angular diversity than TPT alone. Notably, the features align more closely with their respective class labels (Fig. 3b (right)), similar to the well-calibrated hard prompt scenario. As shown in Figure 4, the examples provided show a negative correlation between ECE and angular diversity - maximizing the angular distance (AD) within each group. which aligns with the empirical findings of our paper. These findings suggest the importance of angular diversity in improving VLM calibration for test-time prompt tuning. By increasing the angular distance between learned prompt vectors, A-TPT promotes well-calibrated predictions, crucial for real-world applications that require reliable uncertainty estimation, such as medical diagnostics and autonomous systems.
Motivation for angular diversity. Prior works have shown that well-calibrated textual features tend to be more dispersed with L2 distance (Yoon et al., 2024), which is spread farther apart or benefits from greater angular separation with enforced orthogonality constraints (Sharifdeen et al., 2025) in the embedding space. Prior work O-TPT (Sharifdeen et al. (205) found that lower cosine similarity between textual features corresponds to better model calibration. This suggests that promoting greater angular separation among class-wise text features can improve the calibration of VLMs, However, these analyses overlook crucial aspect: angular diversity among textual features. We empirically observe that orthogonalization tends to group textual features closer, particularly when the number of classes is greater than the embedding dimension (Fig. 4.), and does not guarantee uniformity of angular separation across all prompt vectors. We hypothesize that promoting angular diversity is more important for better calibration of VLMs than enforcing orthogonality among textual features. Unlike existing techniques that disperse prompts via L2 dispersion (Yoon) et al. 2024) orthogonality constraints (Sharifdeen et al. 2025). our approach explicitly focuses on maximizing the minimum pairwise angular distance among normalized textual features. This numerical method constructs each prompt-induced feature point uniformly distributed on unit hypersphere. Maximizing the minimum pairwise angular distance between the features ensures each feature vector points in diverse direction, thereby sharpening class boundaries and improving zero-shot calibration performance in VLMs (Wang & Isola 2020) during inference.
Comparison of dispersion, orthogonality, and angular diversity. The ATFD (Yoon al. 2024 objective disperses textual features by maximizing the 1.2 distance from their centroid, without enforcing pairwise angular separation. In contrast, orthogonality constraints (Sharifdeen et al., 2025) that enforce angular separation by pushing features to be orthogonal to minimize their cosine similarity, tend to group textual features closer together when the number of classes is greater than the embedding dimension N > [D]. When the number of classes is lesser N < |D], it fails to fully utilize the hyperspherical space effectively. As a result, neither of these methods guarantees the uniformity of angular separation of features across the hypersphere (Liang et al. 2022). In zero-shot CLIP settings, normalized textual features are constrained to the surface unit hypersphere (Wang & Isola 2020). While ATFD may shift the centroid of the text features toward the center of the hypersphere by adjusting the features, similarly, orthogonality constraints may encourage angular separation by pairwise orthogonalizing the features the surface. However, these methods may not effectively distribute features uniformly across the hypersphere's s surface. To further validate our findings, we experimented for both where N < [D] (Helber al. 2018). and N |D] (Recht al. 2019) cases. For each data sample, we extract test-time prompt-tunee tuned text features generated by O-TPI and A-TPT (ours), compute pairwise cosine similarities, and plot their mean, as shown in Fig. The results show that О-TPT, which lacks clibratiinnsspccinn when N > ID] (O-TPT fails), exhibits lower, but high fluctuations in cosine similarities. reflecting its inconsistent calibration performance. In contrast, our method's angular diversity consistently produces text features with slightly higher but more consistent cosine similarities, indicating stable, uniform angular separation. Similarly, O-TPT underutilizes hyperspherical space, showing higher, slightly fluctuating cosine similarities when N < [D], while A-TPT shows lower, more consistent cosine similarities, achieving the greatest possible angular separation. That's why hypersphere offers optimal textual feature separation in A-TPT. (suppl. carries more details) In Tab. [ we present the accuracy and ECE results for CLIP ViT-B/16 backbone, dividing data into two groups based on the number of classes relative to the embedding dimension of TPT text features. Group 1 includes
constraints, tends to group textual features closer

---

Published as a conference paper at ICLR 2026
Method
Metrie
Gromp (N [P]
Group 2
Overall
(N <. (P])
Bandine
ECR
Acc.
$7.87
3.36
62:99
544
60.43
4.40
TPT
PCR
Acc.
59.83
12,60
64.54
9.50
62.19
11.25
CTIPT
ECB
Acc.
99.70
5.58
64.44
3:25
02.07
5.22
O-TPT
Acc.
ECB
53.70
4.27
63.33
4.44
61.17
4,36
A-TPT (Ours)
ECE
Acc.
2.92
64.30
3.60
61.27
3.26
N [D]
(b) [D]
Figure 4: Comparison of mean cosine similarity changes for both
Table 1: Comparison of Accuracy and ECE categories with CLIP ViT-B/16 backbone. Where, O-TPT fails, but across methods with CLIP B/16 back- our A-TPT offers consistent cosine similarity values and achieves hone and categories based the number the greatest minimum pairwise angular distance among text features of classes and TPT text features embedding for all the data points. (suppl. carries more details.) dimension (ID)).
cases with .N > [D], while Group 2 includes N < ID]. We then calculate the ECE and accuracy separately for each group, allowing more fine-grained analysis of each method's performance. As hypothesized, cases with N > DI tend to show elevated ECE, indicating poor calibration and suggesting these are more challenging points. In these challenging cases (Group 1), our method significantly outperforms C-TPT as well as O-TPT in terms of calibration performance, resulting in an overall lower ECE. These results highlight the efficacy of our approach in handling both groups. To achieve well-calibrated predictions, we argue that simple feature dispersion or orthogonal separation is insufficient, instead promoting angular diversity - by maximizing the minimum pairwise angular distance an effective approach to uniformly distributing features across the hypersphere 's surface and improving calibration in VLMs.
Angular diversity. Motivated by these insights, we introduce angular diversity to better the test-time prompt calibration of VLMs by promoting angular diversity within the textual feature matrix. Let each class Cj be associated with a textual feature vector t E IR PI where [D] is the embedding dimension. Define the text feature matrix E that contains textual feature vectors for all classes, such that E Nx DD,, where N denotes the total number classes. Each element E.y corresponds to the embedding of the i-th class in the j-th dimension. This matrix E captures the spatial distribution of class-specife features across the shared latent space. We normalize E to É. To promote uniformity on unit hypersphere (Fig. 2 we compute the matrix product ËĚT, which contains the pairwise cosine similarities between the text features. Inspired by insights from the ArcFace (Deng et al.. 2019), we propose an angular variant of the cosine loss the objective function maximize the minimum pairwise angular distance between the normalized prompt vectors.
1 N AD = N jE(1 NYM1
Σ min
Oij, 0 = arccos TET), s.t. # v. E,
e = e,
Here, O E NNN is the matrix of pairwise angular distances. The angular diversity term, denoted as AD, maximizes the minimum pairwise angular distance between normalized prompt vectors while ensuring uniformity of text features across the feature space. Thus, we integrate this regularization term into the overall objective function for the test-time prompt tuning process better the calibration performance is formulated as:
p" arg min (CTPT + À CA-TPT), where CA-PPT AAD,
P
Lirr is the TPT negative maximum class log probability (entropy minimization) loss function, denotes the learnable prompt parameters, and À hyperparameter that promotes uniformity of distributed features across the hypersphere and ensures effective utilization of the full feature space while controlling the strength the regularization term. By explicitly promoting the angular diversity term, we systematically achieve maximum angular distance between pairwise textual features to better prompt calibration test-time prompt tuning.
Numerical stability analyses. When computing arccos which could suffer from gradient explosion its input leaks outside the boundaries of [-1, 1] and gets too close to the ends (as its gradient is Bh (arccos(z)) = -1/ - )²), In Eq. where 0 is the arccos of pairwise cosine similarities (dot
products of normalized text features ÉÉ' we clamp within (-1, 1), specifically from -0.99999 to 0.99999, to prevent NaNs in the arccos calculation, and to prevent the infinite gradient at +1 during backpropagation.
7

---

Published as a conference paper at ICLR 2026
"A
Figure 6: Comparison of expected calibration error (ECE) between C-TPT, O-TPT, and A-TPT (Ours). Results are based on CLIP ViT-B/16 backbone. Lower ECE provides better prompt calibration.
Figure 7: Comparison of the gradient norm changed with pairwise angular distance. Unlike O-TPT's gradient, which vanishes and consistent with 6.
A-TPT's gradient stable
Gradient analyses comparison to O-TPT. O-TPT's orthogonality loss yields gradients that shrink to zero as the pairwise angular distance - O, making optimization hard when features are already close. In contrast, A-TPT optimizes the angular distance directly; its gradient norm is angle-independent so it stays stable even at small 0. That's why directly optimizing angular distance rather than using cosine similarity at test time improves VLM calibration and avoids the stuck near-colinear regime that hurts calibration. (See Fig. 7 for gradient-none norm, and Appendix A.. for derivations (eqs.
Computational complexity. A-IPT's asymptotic complexity same O-TPT, with negligible run- time/memory overhead over C-TPT, while substantially reducing ECE. (See Tab, 8 and Appendix (A.6)
4 EXPERIMENTS
We evaluate on different datasets, across various baselines, with CLIP VIT-B/16 (512-d) and RN50 (1024-d); suppl. carries datasets and implementation details in Appendix A.7 and A.9
Calibration performance on fine-grained classification tasks. We evaluate the proposed A-TPT method, and observe better calibration performance across multiple fine-grained classification tasks with both CLIP ViT-B/16 and CLIP RN50 backbones (Tab. _ A-TPT consistently reduces ECE compared O-TPT (Sharifdeen al., 2025) and C-TPT (Shu et al., 2022): For CLIP VYT-B/16. average ECE drops from 5.13 (C-TPT) and 4.23 (O-TPT) to 2.92. For CLIP RN50, ECE reduces from 6.19 (C-TPT) and 5.45 (O-TPT) to 2.79 These results highlight the efficacy of A-TPT both N < |D] and N D cases.
Calibration performance under natural distribution shifts. Tab. B shows the calibration results under natural distribution shifts. All hyperparameters and experimental configurations match with the implementation section, except for the regularization weight X, which we set to 10.0. Similar to Tab. 2 A-TPT shows better calibration performance across ImageNet variants by reducing the ECE for both CLIP ViT-B/16 and CLIP RN50. For CLIP ViT-B/16, A-TPT lowers the average ECE to 3.92, drops from 4.88 (O-TPT) and 5.82 (C-TPT). For CLIP RN50, A-TPT achieves average ECE of 7.82 drops from 9.69 (O-TPT) and 12.1 (C-TPT). Importantly, A-TPT also surpasses the zero-shot baseline in calibration performance, showing lower ECE on both backbones, without compromising the high accuracy benefits of TPT for N DI and N < D cases, which is a feat unmatched by any other approach.
Medical prompt tuning with A- TPT. We evaluate the generalizability of A- on medical datasets with medical baselines under the N [D] regime. Tab. 4 presents the performance of FPT (Huang iet al. 2024) and FPT combined with O-TPT and A-TPT on ISIC 2018 dataset, where A-TPT leads to a notable reduction in Expected Calibration Error (ECE) while preserving high classification accuracy. Tab. Tab. eJvaluates PLIP with Prompt Smooth (PS) (Hussein et al.) 2024), on KatherColon, where the combination of A-TPT further better calibration performance. Tab. (]rrports results using MedCLIP with BAPLe (Hanif et al.] 2024), showing that the integration of A-TPT substantially improves calibration metrics over the baseline.
Comparison with previous calibration methods. As illustrated in Fig. 6 average ECE across different datasets, including fine-grained classification and natural distribution shifts, where cali- bration techniques applied to TPT (Shu al.) 2022). We compare C-TPT (Yoon al.) 2024). O-TPT (Sharifdeen et al. 2025), and A-TPT (Ours). T-TPT consistently shows better calibration, demonstrating its efficacy improving model reliability across datasets.
Reliability plots. To address that A-TPT achieves superior calibration performance across multiple fine-grained datasets both
and over-confidence, while Tables 2 and 3 illlstrate
8

---

Published as a conference paper at ICLR 2026
Method
Metric
ImageNket
1770
Flwersttz
Foed
SUNN97
Alrzzatte
OsterdPets
Cattechi1l
UCFIOI
EaroSAT T
Sandford
Care
Avrrar
Pre-rained Backbome:
CLIP
ViT-R/16
Frebeoking dimension: 512-d
Buseline
Acc.
66.00
44,30
67.30
83.60
62.50
23.90
$8.00
92.90
65.00
41.30
65.30
63.70
BCE
2,12
5.50
3,00
2:39
2.53
5.11
4.37
5.50
3.39
13.89
4.25
4.43
TPT
FCF
Ace.
6900
10.60
46:70
21.20
69.00
13.50
84.70
3.08
64.50
11:30
23.40
16.80
87.10
5.77
95.80
4.51
67.30
2.54
42,40
13:20
66:30
5.16
65.00
11.60
CTTPT
BCB
Acc.
68,50
3.15
46.00
11.90
69.50
5.04
63.70
3:43
64.80
5.04
24.85
4.36
55.20
1,90
93.63
4.24
65.70
2,54
43.20
13.20
65.80
1.59
64.57
5.13
O-TPT
FCF
Acc.
67.33
1.96
45.08
7.88
70.07
3.87
84.15
1.46
64.23
4,95
23.04
3.68
87.95
1.90
93.95
3.80
64.16
2.34
42.54
12.98
64.53
1.78
64.41
4.23
A-TPT (Oun)
Arc. RCE
67.70
1.4
45.51
4.76
69.23
3,61
83.64
1.37
66.04
3.28
23.76
3.14
88.33
1.17
93.87
2.76
66.16
2.12
44.06
3.92
65.78
1.09
64.92
2.61
Prestrinned Backhome: CLIP
Foshvilding dimontioo: 1034d
Buseline
BCB
Acc:
2.09
40.00
9.91
61.00
3.19
74.00
3,11
58.4
3.54
15.60
6.45
$3.80
3.91
85
4.33
3.05
23.00
15.40
55.70
4.70
5.61
TPT
PCF
Acc.
60.70
11.40
41.50
25.70
62.50
13.40
74.90
5.25
61.10
9.24
17.00
16.10
$4.50
3.65
87.00
5.04
59.50
12.40
22.50
28.30
58.00
3.76
57.70
11.70
CTTPT
BCD
Acc:
60.20
3.01
43.20
19:80
65.30
4.14
74.70
1.86
61.00
2,93
10.70
17.00
$4.10
2.77
86.90
2.07
59.70
3.83
15.10
27.50
56.50
1.94
57.75
6.19
O-TPT
BCE
Acc.
58,97
3.10
41.90
16.53
65.61
2,50
74,22
1.20
60.85
3.20
16.77
8.18
$3.40
3,50
86.86
2.75
58.84
2.60
28.35
14.71
56.48
1.69
57.47
5.45
A-TPT (Oun)
RCE
Acc.
53.44
2.49
40.90
6.41
64.80
2,39
74.10
1,11
60.46
2,50
14.58
6.14
83.45
2.47
86.57
1.98
60.34
2.34
32.14
2,51
57.08
1.38
57.53
2.92
Table 2: Comparison of methods across ane-grained datasets for Accuracy (Acc.) and Expected Calibration
Error (ECE) with CLIP VYT-B/16 and CLIP RN50 pre-trained backbone for both N ID and N < ID| cases. The overall top best-performing result is in bold.
Naa
Arcente
Method
Metrie
Pre Pre-triine Bockbaan:
VI--/16
Eistesiiig diveraiio: 512-l
Baelise
DCB
Acc.
47,80
k6l
60:80
3.01
74.00
3.58
4.95
46.10
$7.20
504
TPT
BCB
52,60
18400
63.00
11:0
16.10
47.50
12:00
59:50
C-TFE
BCE
Art.
$1.00
8.16
61:00
2000
47:00
7.35
19:00
582
(a) C-TP:: Food101
(b) C-TPT: DTD
(c) C-TPE: Flower102
O TPT
Acc.
ECB
46.87
7.22
61.65
3.97
22.55
1.46
47.12
6.87
$1.30
438
Act.
55.06
Becine
BCE
Acc.
21.30
31.70
$1.40
3.35
2.07
50.99
35.30
3.15
40.60
7.45
IPT
BCE
31.00
13.10
16.70
CTPE
Ast.
BCE
23.40
25.40
$4.70
5.58
58.00
35.10
9.70
42.80
12.10
O-TPT: 1ood101
O-TPT: DTD
( O-TPT: Flower102
O-TYE
ICB
24.56
23.07
3.87
4.47
5.85
41.16
9.08
A-TPT (Ours)
Art.
21.88
40.57
Table Comparison of methods across natural distri- bution shift datasets for Accuracy (Acc.) and Expected Calibration Error (ECE) with TPT (baseline) with
CLIP VTT-B/16 (top) and CLIP RN50 (bottom) pre-
(g) T-TPT: Food101
(b) A-TPT: DTTD
() A-TPT: Flower102
trained backbones for both N > DJ and N < (D) cases. The overall top best-performing result in bold.
Figure 5: Reliability diagrams for CLIP ViT-B/16 back-
bone (suppl. carries additional reliability diagrams).
Method
Metric
ISIC' (N
?)
Method
Metric
KatherColon (.
Method
Metric
BA (VV 2)
Covid
CA(N 11)
FPT (512-1)
FPT + A-TPT
ECE
FOB
Acc.
Acc.
Acc.
0.1381
0.0794
0.2328
98.43
98.3]
98.25
PS (768-8)
PS e A-TPT
PS O-IPT
ECE
ECE
Acc.
Acc.
12:73
8.86
76.2
76.4
BAPLe (768-1)
BAPLe O TPT
A-TPT
Asc. ECE
Acc.
ECE
Acc.
99.50
99.78
99.62
0.91
0.42
3.21
$1.36
15.64
597
2.34
82.5
Table 5: PLIP: Promptsmooth
Table 4: FPT: FPT + A-TPT ISIC 2018
(PS) + A-TPT on KatherColon (KC).
Table 6: MedCLIP: BAPLe A-TPT
on Covid dataset.
CLIP ViT-B/16 and CLIP RN-50 backbones, they do not reveal insights into whether the model tends to be over-confident or under-confident. To further analyze this, we plot reliability diagrams in Fig. 5
for the Food101, DTD, and Flowers 102 datasets with the CLIP ViT-B/16 backbone. C-TPT Yoon ct al. 2024) displays coder-confidence on Food101 (Fig. 5a) and over-confidence on DTD (Fig. 5b)
and Plowers102 (Fig. 5c). O-TPT (Sharifdeen al. 2025) partially mitigates these issues some

---

Published as a conference paper at ICLR 2026
Method
Metric
pTB
Flwwerst22
Fond10
SUN297
Airzzlie
UstordPats
Cattebh 101
UCFI01
Eura847 1
Sandorr
Care
Avrrare
Pre-trined Backbome CLIP ViT-R/16
Baseline- + CuOp
Emnbeddnng domentioon 512-d
Duseline + CoOp
Azc. ECE
43,10
7.71
67.40
3.92
63,20
1.55
63.70
1.72
18.00
9.21
$9.20
2.92
93.60
3.05
66,00
3.47
40.10
15.30
63,10
0.86
63.50
5.25
TPT + C30p
ECF
Ace.
44,50
34.80
68.70
19.90
53.80
9.66
65,40
20.30
20.00
20.60
89.30
7.40
94.00
3,65
67.20
19.90
31.30
40.00
65.60)
6.63
63.91
18.
TPT Co0p+ TTT
Ace.
45,00
69.00
53.70
65.10
19.20
89.30
03.90
66,60
40.70
63.10
63.56
ECE
21,00
10.20
4.49
11.30
21.50
2.12
1.06
12.00
13:20
2.45
10.04
TPT à C00p O-TPT
Ace.
EDE
45.45
16.02
68.57
681
83.55
3.59
64.01
7:23
18.69
16.82
89.07
1.92
93.71
0.92
05.64
9.16
40.17
13.76
64.12
2.85
63.14
7.91
TPT CoOp A-IPT
ECE
Arc.
43.21
6.33
68.94
2.91
63.25
3,12
65.34
2
20.58
5.31
90.02
1,05
93.23
69.99
3.78
40.25
65.89
2.04
64.07
3.63
Pre-triined Backbone: CLIP VIT-8/16
Baseline CoCoOp [imbedding disenusion: 1024
Boseline + CoCoUp
ECE
Ace.
44,50
3.82
68.40
3.82
34.10
3.25
63.00
4.61
24.20
4.06
88.30
4.00
91.00
3.52
67.00
3.28
48.10
5.81
64.90
6.51
64.30
4.20
TPT + CoCoOp
ECE
Ace.
45,00
6.91
68.60
4.70
84.60
1,94
64.00
3.16
24.90
6.13
88.50
2.22
91.20
2.74
67,80
3.47
44.50
9.03
65.90
5.22
54 64.90
4.35
TPT + CoCoOp C-TPT
ECC
Ace:
44.70
4.18
60.30
3.13
84.20
2,66
63.60
2,96
34.60
4.90
88. 80
3.76
91:40
3:45
67.10
2.91
44.30
5.79
64.90
5.09
64.70
3.68
IPT + CoCoOp- A-IPT
Acc.
ECE
44.28
3,52
68.75
3.08
$4.12
1.91
63,68
2.74
24.14
4.79
88.37
2,09
91.21
2.67
67.89
2.85
44.36
3.49
64.91
4.95
64.15
3.22
Table 7: Comparison of methods when using CoOp (Zhou et al.
2022b) and CoCoOp (Zhou et al.
2022a) as
baseline across fine-grained datasets for Accuracy (Acc.) and Expected Calibration Error (ECE) with CLIP ViT-B/16 pre-trained backbone. The overall best-perforning result bold.
extent (Figs. 5d, 5e, 5f), but noticeable calibration gaps persist, particularly on DTD (Fig. 5e). In contrast, A-TPT produces the most reliable predictions correcting under-confidence in Food101 (Fig. 5g) and significantly reducing over-confidence in the other datasets (Figs. 5h, 5i). These results highlight that A-TPT effectively balances confidence and accuracy, making it an ideal solution for both under-confidence and over-confidence in VLM calibration.
Supervised-trainet prompt embeddings. In addition to tuning prompt parameters during inference time, we further evaluate the calibration capabilities of A-TPT when combined with supervised- trained prompt embedding parameters for test-time prompt tuning. Specifically, utilized the officially published checkpoints of CoOp (Zhou et al. (022b) and CoCoOp (Zhou et al.) (022a) as presented in Tab. 7 For CoCoOp, we trained on 16 images per class from half of ImageNet's total classes with 4 learnable prompt embeddings (Shu et al., 2022 Yoon et al., 2024). As reported in Tab. [ for CoOp across 10 fine-grained datasets, A-TPT reduces the overall average ECE to 3.63. This reflects a marked improvement relative to O-TPT CoOp, C-TPT + CoOp, and TPT CoOp. which report average ECEs of 7.91, 10.4, and 18.36, respectively. Similarly, when integrating A-TPT with CoCoOp results in considerable calibration gains across most datasets compared to C-TPT CoCoOp and TPT- + CoCoOp. As in Tab. 7 our approach reduces the overall ECE to 3.22, compared to 3.68, 4.35 for C-TPT CoCoOp, TPT CoCoOp, respectively. These results show that integrating A-TPT with supervised-trained prompt embeddings during test-time tuning preserves classification accuracy while substantially better calibration performance.
5 CONCLUSION
We propose a novel technique, called angular diversity, to promote the uniformity of textual features and disperse them calibrate test-time prompt tuning of vision-language models. We reveal that maximizing the minimum pairwise angular distance between textual features while prompt learning associated with lower calibration error. We show that achieving uniformity between textual features is more effective than orthogonalization and dispersion through 1.2 distance objectives. Moreover, angular diversity is also effective for prompt learning with significant margins, due to enlarging the inter-class separability. Therefore, we propose angular diversity on textual features during test-time prompt tuning, abbreviated as A-TPT. Our approach consistently outperforms state-of-the-an methods with different backbones and baselines.
Acknowledgments. The computational resources for this research were supported by the Accelerating Higher Education Expansion and Development (AHEAD) Operation Grant No. from the Ministry of Higher Education, Sri Lanka, funded by the World Bank and the National Research Council of Sri Lanka Grant No. 19-080.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101-ming discriminative compo- nents with random forests. In Computer vision-ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, proceedings, part VI 13, pp. 446-461. Springer, 2014.
Arthur Bucker, Luis Figueredo, Sami Haddadin, Ashish Kapoor, Shuang Ma, Sai Vemprala, and Rogerio Bonatti. Latte: Language trajectory transformer. In 2023 IEEE international conference on robotics and automation (ICRA), pp. 7287-7294, IEEE, 2023.
Guangyi Chen, Weiran Yao, Xiangchen Song, Xinyue Li, Yongming Rao, and Kun Zhang. Plot: Prompt learning with optimal transport for vision-language models. arXiv preprint arXiv:2210.01253, 2022.
Zhihong Chen, Shizhe Diao, Benyou Wang, Guanbin Li, and Xiang Wan. Towards unifying medical vision-and vision-and-language pre-training via soft prompts. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 23403-23413, 2023.
Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describ- ing textures in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3606-3613, 2014.
Can Cui, Yunsheng Ma, Xu Cao, Wenqian Ye, Yang Zhou, Kaizhao Liang, Jintai Chen, Juanwu Lu, Zichong Yang, Kuei-Da Liao, et al. A survey on multimodal large language models for autonomous driving. In Proceedings of the IEEE/CVE Winter Conference on Applications of Computer Vision, pp. 958-979, 2024.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. Icce, 2009.
Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou. Areface: Additive angular margin loss for deep face recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 4690-4699, 2019.
Vishnu Sashank Dorbala, Gunnar Sigurdsson, Robinson Piramuthu, Jesse Thomason, and Gaurav S Sukhatme. Clip--av: Using clip for zero-shot lanlauagd navigation. arXiv preprint
2022.
Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In 2004 conference on computer vision and pattern recognition workshop, pp. 178-178 178. IEEE, 2004.
Samir Yitzhak Gadre, Mitchell Wortsman, Gabriel Ilharco, Ludwig Schmidt, and Shuran Song. Clip on wheels: Zero-shot object navigation as object localization and exploration. arXiv preprint WrXiv:2203.1042) 3(4):7, 2022.
Zoubin Ghahramani. Probabilistic machine learning and artificial intelligence. Nature, 521(7553): 452-459, 2015.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International conference on machine learning, pp. 1321-1330. PMLR, 2017.
Asif Hanif, Fahad Shamshad, Muhammad Awais, Muzammal Naseer, Fahad Shahbaz Khan, Karthik Nandakumar, Salman Khan, and Rao Muhammad Anwer. Baple: Backdoor attacks medical foundational models using prompt learning. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 443-453 Springer, 2024.
Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Introducing eurosat: A novel dataset and deep leamiigg benchmark for land use and land cover classification. In IGARSS 2018-2018 IEEE international geoscience and remote sensing symposium, pp. 204-207. IEEE, 2018.
11

---

Published as a conference paper at ICLR 2026
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: A critical analysis of out-of-distribution generalization. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 8340-8349, 2021a.
Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 15262-15271, 2021b.
Yijin Huang, Pujin Cheng, Roger Tam, and Xiaoying Tang. Fine-grained prompt tuning: A parameter and memory efficient transfer learning method for high-resolution medical image classification. In International Conference on Medical Image Computing and Computer-Assister Intervention, pp. 120-130. Springer, 2024.
Noor Hussein, Fahad Shamshad, Muzammal Naseer, and Karthik Nandakumar. Promptsmooth: Certifying robustness of medical vision-language models via prompt learning. In International Conference Medical Image Computing and Computer-Assitte Intervention, pp. 698-708. Springer, 2024.
Zhanghexuan Ji, Mohammad Abuzar Shaikh, Dana Moukheiher, Sargur N Srihari, Yifan Peng, and Mingehen Gao. Improving joint learning of chest x-ray and radiology report by word region alignment. In Machine Learning in Medical Imaging: 12th International Workshop, MLMI 2021, Held in Conjunction with MICCAI 2021, Strasbourg. France, September 27, 2021, Proceedings 12, pp. 110-119. Springer, 2021.
Chao Jia, Yinfci Yang. Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le. Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In International conference on machine learning, pp. 4904 PMLR, 2021.
Apoorv Khandelwal, Luca Weihs, Roozbeh Mottaghi, and Aniruddha Kembhavi. Simple but effective: Clip embeddings for embodied ai. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2022.
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization, Pruceedings of the IEEE international conference on computer vision workshops, pp. 554-564, 2013.
Aviral Kumar, Sunita Sarawagi, and Ujjwal Jain. Trainable calibration measures for neural networks from kernel mean embeddings. International Conference on Machine Learning, pp. 2805-2814 14. PMLR, 2018.
Jing Lei, Max G'Sell, Alessandro Rinaldo, Ryan J Tibshirani, and Larry Wasserman. Distribution-Fre freee predictive inference for regression. Journal of the American Statistical Association, 113(523): 1094-1111, 2018.
Victor Weixin Liang, Yuhui Zhang, Yongchan Kwon, Serena Yeung, and James Y Zou. Mind the gap: Understanding the modality gap multi-modal contrastive representation learning. Advances in Neural Information Processing Systems, 35:1 112117661176777 17625, 2022.
Bingyuan Liu, Ismail Ben Ayed, Adrian Galdran, and Jose Dolz. The devil is in the margin: Margin- based label smoothing for network calibration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 80-88, 2022.
Jie Liu, Yixiao Zhang, Jie-Neng Chen, Junfei Xiao, Yongyi Lu, Bennett A Landman, Yixuan Yuan, Alan Yuille, Yucheng Tang, and Zongwei Zhou. Clip-driven universal model for organ segmentation and tumor detection, In Proceedings of the IEEE/CVF international conference on computer vision, pp. 21152-21164 2023.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv:1711.05101 2017.
arXiv preprint
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained visual classification of aircraft. arXiv preprint 13065:1306.515: 2013.
12

---

Published as a conference paper at ICLR 2026
Sachit Menon and Carl Vondrick. Visual classification via description from large language models. arXiv preprint arXiv:2210007183, 2022.
Muhammad Akhtar Munir, Muhammad Haris Khan, M Sarfraz, and Mohsen Ali. Towards improving calibration in object detection under domain shift. Advances in Neural Information Processing Systems, 35:38706-38716, 2022.
Muhammad Akhtar Munir, Muhammad Haris Khan, Salman Khan, and Fahad Shahbaz Khan. Bridg- ing precision and confidence: train-time loss for calibrating object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11474-1 11483, 2023.
Balamurali
Murugesan, Julio Silva-Rodríguez, Ismail Ben Ayed, and Jose Dolz. Robust calibration of large lision-language adapters. In European Conference on Computer Vision, pp. 147-165. Springer, 2024.
Mahdi Pakdaman Naeini, Gregory Cooper, and Milos Hauskrecht. Obtaining well calibrated proba- bilities using bayesian binning. In Proceedings of the AAAI conference on artificial intelligence, volume 29, 2015.
Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian conference on computer vision, graphics & image processing, pp. 722-729. IEEE, 2008.
Jeremy Nixon, Michael Dusenberry, Linchuan Zhang, Ghassen Jerfel, and Dustin Tran. Measuring calibration in deep learning. In CVPR workshops, volume 2, 2019.
Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In 2012 IEEE conference on computer vision and patlern recognition, pp. 3498-3505. IEEE, 2012.
John Platt et al. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. Advances in large margin classifiers, 10(3):61-74, 1999.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. International conference on machine learning, pp. 8748-8763. PmLR, 2021.
Benjamin Recht, Rebecca Roclofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? International conference on machine learning, pp. 5389-5400. PMLR, 2019.
Ashshak Sharifdeen, Muhammad Akhtar Munir, Sanoojan Baliah, Salman Khan, and Muham- mad Haris Khan. O-tpt: Orthogonality constraints for calibrating test-time prompt tuning in vision-language models. arXiv preprint arXiv:2503.12009 2025.
Manli Shu, Weili Nie, De-An Huang, Zhiding Yu, Tom Goldstein, Anima Anandkumar, and Chaowei Xiao. Test-time prompt tuning for zero-shot generalization in vision-language models. Advances in Neural Information Processing Systems, 35:14274-14289, 2022.
Richard Socher, Milind Ganjoo, Christopher D Manning, and Andrew Ng. Zero-shot learning through cross-modal transfer. Advances in neural information processing systems, 26, 2013.
Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: dataset of 101 human actions classes from videos in the wild. arXiv preprint 12/2.:2 2012.
Pieter Merkus Lambertus Tammes. On the origin of number and arrangement of the places of exit on the surface of prrlen-grains. Recueil des travaux botaniques néerlandais, 27(1):1-84 14, 1930.
Vladimir Vovk, Alexander Gammerman, and Glenn Shafer. Algorithmic learning random world, volume 29, Springer, 2005.
Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global representations by penalizing local predictive power. Advances in neural information processing systems, 32, 2019.
13

---

Published as a conference paper at ICLR 2026
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through align- ment and uniformity on the hypersphere. In International conference on machine learning, pp- 9939. PMLR, 2020.
Zifeng Wang, Zhenbang Wu, Dinesh Agarwal, and Jimeng Sun. Medclip: Contrastive learning from unpaired medical images and text. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. Conference Empirical Methods in Natural Language Processing, volume 2022, pp. 3876, 2022.
Jianxiong Xiao, James Hays, Krista Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In 2010 IEEE computer society conference computer vision and pattern recognition, pp. 3485-3492. IEEE, 2010.
Hantao Yao, Rui Zhang, and Changsheng Xu. Visual-language prompt tuning with knowledge -ggged context optimization. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 6757-6767, 2023.
Lewei Yao, Runhui Huang, Lu Hou, Guansong Lu, Minzhe Niu, Hang Xu, Xiaodan Liang, Zhenguo Li, Xin Jiang, and Chunjing Xu. Filip: Fine-grained interactive language-image pre- pre-training. arXiv preprint wxXiv:211..07780 11.07783, 2021.
Hee Suk Yoon, Joshua Tian Jin Tee, Eunseop Yoon, Sunjae Yoon, Gwangsu Kim, Yingzhen Li, and Chang D Yoo. Esd: Expected squared difference as a tuning-free trainable calibration measure. arXiv preprint arXiv:2303. 024722 2023.
Hee Suk Yoon, Eunseop Yoon, Joshua Tian Jin Tee, Mark Hasegawa-Johnson, Yingzhen Li, and Chang D Yoo. C-tpt: Calibrated test-time prompt tuning for vision-language models via text feature dispersion. arXiv preprint 1xXx::243..44199 2024.
Junwei You, Haotian Shi, Zhuoyu Jiang, Zilin Huang, Rui Gan, Keshu Wu, Xi Cheng, Xiaopeng Li, and Bin Ran. V2x-vlm: End-to-end v2x cooperative autonomous driving through large vision-language models. arXiv preprint arXiv:2408.. 0925/, 2024.
Sheng Zhang, Yanbo Xu, Naoto Usuyama, Jaspreet Bagga, Robert Tinn, Sam Preston, Rajesh Rao, Mu Wei, Naveen Valluri, Cliff Wong, et al. Large-scale sppeiffc pretraining for biomedical vision-language processing, arXiv preprint 000Xiv:2303000015, 2(3):6, 2023.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for langual models. In Proceedings of the IEEE/CVF conference on computer vision and paltern recognition, pp. 168 16-258,5, 2022a.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision- language models. International Journal of Computer Vision, 130(9):2337-234,, 2022b.
Xingcheng Zhou, Mingyu Liu, Ekim Yurtsever, Bare Luka Zagar, Walter Zimmer, Hu Cao, and Alois C Knoll. Vision language models in autonomous driving: A survey and outlook. IEEE Transactions on Intelliennt Vehicles, 2024.
A APPENDIX
A.1 THE USE OF LARGE LANGUAGE MODELS (LLMs)
LLM Usage Statement: We made limited use of large language models to enhance the clarity and readability of the text. They were not involved the conception of ideas, experiment design, analysis, or the production of results.
A.2 ETHICS STATEMENT
We confirm that our research adheres to the highest standards of ethical considerations. All work presented in this paper is original, and any external contributions sources have been appropri- ately cited. Our study does not introduce new datasets, nor does it involve experiments utilizing demographic identity characteristics.
14

---

Published as a conference paper at ICLR 2026
A.3 REPRODUCIBILITY STATEMENT
To ensure the reproducibility of our work, have detailed the comprehensive training procedures, hyperparameters, and experimental settings in Sections and A.9 of the paper.
A.4 LIMITATIONS
A notable limitation of the proposed angular diversity-based calibration method is slight reduction in classification accuracy observed in certain scenarios. Although empirical results suggest that overall accuracy is not significantly affected, some minor degradations in few cases like semantically overlapping datasets - typically under 1-2% - have been noted in some instances. This trade-off' aligns with findings from prior test-time prompt calibration-focusee approaches (Yoon al. 2024] Sharifdeen et al. 2025), which similarly reduce the expected calibration error (ECE) at the expense of minor compromise in accuracy. Such trade-offs are generally considered acceptable, particularly in sigh-stakes applications where better model calibration, reliability, and trustworthy uncertainty estimation are of critical importance.
Another limitation arises from the method's reliance on test-time adaptation, which limits access to labeled training or validation data, making hyperparameter tuning difficult and limiting optimization during adaptation. Within the A-TPT framework, the trade-off between accuracy and calibration is governed by regularization hyperparameter (as defined in Eq. 3), which is fixed at 80.0 for all test instances based on insights from ablation studies. While this fixed value has shown better performance across various settings, we acknowledge that it may not be optimal for each data sample; serves practical starting point. Future research could benefit from exploring dynamic, every data sample adaptation of À to further better calibration performance, without compromising accuracy. Notably, any such method does not depend on labeled data to preserve the real-world applicability of test-time prompt tuning in label-scarce settings.
A.5 GRADIENT ANALYSES COMPARISON TO O-TPT
We analyze and compare the gradients of loss functions for A-TPT with O-TPT. To simplify the derivation, we only consider the norm of the gradient of the objective function, composing the loss function, w.r.t. the corresponding text feature matrix E. For intuitive comparison, the analysis results are presented in Fig. 7
Corresponding the O-TPT, the gradient norm is derived as follows:
of oj lle [|]
0 ÊĚ² de;
II(I M.,ee ||: |eyll
||e
|sin
0gll
||sin 0:yll
e.e," T
=
de;
lec ||el||
le
Mc.
|e.
(3)
where Me, represents the projection matrix of e; From the above derivation and Fig. 7 we can see that the gradient norm very small when the pairwise angle is close to zero. That is why the orthogonality constraint is hard to converge for the case that e; and ej are close to each other in both
N D N |D] categories. Next, we derive the gradient norm corresponding to the A-TPT:
80gg ae:
80gg
8 ÉÉ
sin O.j 1
(4)
a ÉET de:
sin O:j
|es
|e.
Compared to the gradient norm corresponding to the orthogonality constraints, as referred to Equa- tion 3 the gradient norm corresponding to the angular diversity is independent of the pairwise angle Ozj. so it would not encounter the very small gradient even though O., is zero. Fig, 7 shows that the gradient norm corresponding to angular diversity empirically proves that the A-TPT is better than the О-TРТ, primarily due to its more stable and consistent gradient during test-time prompt tuning. Therefore, directly optimizing angular distance rather than using cosine similarity, and getting near-optimal solutions for the Tammes problem. sufficiently justifies the design choice for both N [D] & N > DI categories (Fig. 4). especially for N > [D].
15

---

Published as a conference paper at ICLR 2026
A.6 COMPUTATIONAL COMPLEXITY
As shown in the Tab. wc compare C-TPT, O-TPT, and A-TPT on the larger, higher-dimensional embeddings natural distribution shift dataset (Recht et al. 2019) for the N > [D] case from the perspective of asymptotic complexity, calculating time per batch, occupied memory, and ECE. Compared to L2 Regularization, the orthogonality constraints and angular diversity slightly increase the asymptotic complexity, calculation time, and occupied memory, while reducing ECE.
However, the orthogonality constraints greatly increase that due the computational overhead of all the pairwise matrix operations. In terms of ECE, the angular diversity reduces over the 1.2 regularization by substantial margin. The angular diversity also the most effective enlarge the minimal pairwise angular distance of textual features to sharpening class boundaries, which would increase the inter-class separability and better calibration performance. Besides, as a simple plug-in regularizer with negligible computational overhead, it is shown to be pre-trained backbone-agnose and produces better calibration on fine-grained classífication tasks and natural distribution shifts. Therefore, the key advantage of angular diversity highlighted in this paper is not only significant but also scalable.
Method
Regularization
Asymptotic
Time (S)
Memory (MiB)
ECE
Complexity
/Batch
C-TPT
L2 Regularization
O(N [P]
1,055
21838
6.23
Orthogonality
O-TPT
O(N2.100
1.064
23740
3.97
Constraints
A-TPT
Angular
O(N2 ID))
1.058
21840
2.96
(Ours)
Diversity
Table 8: Comparison of different regularization methods on ImageNet-V2 V2 diversity achieves lower ECE with negligible computational overhead.
Recht et al. 2019).
The angular
A.7 DETAILS ON THE DATASETS
We evaluate approach multiple datasets encompassing both fine-grained classification and natural distribution shift scenarios (suppl. carries details). For fine-grained classification, we conduct experiments using the ImageNet (Deng et al., 2009) and Caltech 101 (Fei-Fei al., 2004) along with diverse datasets across various domains: DTD (Cimpoi et al. (0014) for texture identification, Flower102 (Nilsback Zisserman 2008) and OxfordPets (Parkhi al. 2012) for plants and animal categories, Food101 (Bossard et al. [014] for food classification, StanfordCars (Krause et al.) 2013) and Aireraft (Maji 2013) for transportation classification, SUN397 (Xiao et al. (010) for scene categorization, UCF101 (Soomro et al. 2012) for human action recognition, and EuroSAT (Helber) (et al. 2018) for satellite imagery environmental categorization. For natural distribution shifts, we utilize several ImageNet variants: ImageNet-V V2 (Recht et al. 2019) (natural images), ImageNet-A (Hendrycks et al.) 2021b) (natural adversarial examples), ImageNet-R (Hendrycks et al. (021a) (artistic renditions), and ImageNet-Sketch (Wang et al. 2019) (black and white sketches) datasets, as benchmarks for dutti--distribution (OOD) performance evaluation.
Tab. (left) summarizes the details of each dataset, such the number of classes, test size, and the corresponding task descriptions.
A.8 CLIP'S EMBEDDING DIMENSION
Tab. TO (right) presents the embedding dimensions associated with various CLIP backbone archi- tectures. For the ResNet-based models, CLIP RN50 and CLIP RN101 produce 1024-dimensional and 512-dimennional embeddings, respectively. contrast, the Vision Transformer (ViT) variants, including ViT-B/16. ViT-B/32, and ViT-L/14, generate embeddings of 512, 512, and 768 dimensions, respectively. These embedding sizes reflect the representational capacity of the model and play a crucial role in angular diversity for zero-shot performance, across diverse tasks where the relationship between the number of classes N and the embedding dimension [D] influences calibration (N < [ and N > |D)) via feature angular separation.
16

---

Published as a conference paper at ICLR 2026
Datasel
# Classes
Test size
Text feature matrix (E)
CLIP ViT-B/16
CLIP RN50
fine-grained classification datasets
|N||P|]
ImageNet
1000
50,000
1000. 512)
11000, 1024
Caltech101
2,465
512]
[100, 1024
OxfordPets
3,669
[37,512
[37. 1024
CLIP pre-trained
Embedding dimension
StanfordCars
196
8,041
196, 512
196, 1024
backbone
Flowers102
102
2,463
102, 512
102, 1024
RN50
1024-d
Food101
101
30,300
101,
101, 1024
RNIOI
512-d
FGVCAircraft
100
3,333
100,512
100, 1024
SUN397
397
19,850
397, 512]
397, 1024
ViT-B/16
512-d
DTD
47
1,692
512
[47, 1024
ViT-B/32
512-d
EuroSAT
10
8,100
[10,
[10, 1024
ViT-L/14
768-d
UCF101
101
3,783
101, 512]
[01,, 1024
Table 10:
CLIP ResNet and
natural distribution shift datasets
ViT's embedding dimension.
ImageNet-A
200
7,500
2200, 512]
[200, 1024
ImageNetV2
1000
10,000
10000, 512
10000, 1021
ImageNet-R
200
30,000
2200, 512]
[200, 1024
ImageNet-Sketch
1000
50,889
11000, 512]
11000, 1024
Table 9: The detailed statistics of datasets used in the experiments.
A.9 IMPLEMENTATION DETAILS
We employ two CLIP backbones: CLIP VIT-B/16 and CLIP RN50. For all experimental setups, we use test-time prompt tuning (TPT) (Shu al., 2022) as the primary objective maximize accuracy while incorporating A-TPT as auxiliary objective to better the calibration performance described Eq. 2 We fix the À as 80.0 for all cases unless otherwise specified. We perform prompt optimization for single-step update with the Adamm (Loshchilov & Hutter 2017) optimizer and set the learning rate to 5e-3. We initialize the prompt embeddings with hard prompts, following C-TPT (Yoon et al., 2024) and all other settings following the standard TPT (Shu et al. 2022) configurations. We conduct all experiments with batch size of 64 on a single NVIDIA Quadro RTX 6000 GPU (24GB memory).
A.10 WEIGHTED AVERAGE COMPARISON
The formula for the weighted average metric is:
(Test Set Size; x Metric,)
Weighted Average Metric
(5)
('Tss Set Size;)
In Tab. 11 method based on the test set size, and we observe better calibration performance with both CLIP ViT-B/16 and CLIP RN50 backbones. Our method (A-TPT) significantly outperforms C-TPT as well as O-TPT in terms of calibration performance, resulting in a lower ECE without compromising accuracy. Specifically, with the CLIP ViT-B/16 backbone, the average ECE drops from 4.74 ( (C-TPT) and 3.91 (O-TPT) to 2.73 with А-TPT. Similarly, for the CLIP RN50 backbone, ECE reduces from 6.11 (C-TPT) and 4.88 (O-TPT) to 3.32 and outperforms C-TPT and O-TPT with substantial improvement in both settings.
we present the weighted average accuracy and ECE results the proposed A-TPT
A.11 EXPANDED STUDY
We have expanded this study by collecting another prompt templates within 80 different hard prompt styles (Radford al. 2021). For illustration, consider the following examples from the natural distribution shift ImageNet dataset (Deng et al, 2009) using CLIP ViT-B/16, categorized into well-calibrated and poor-calibrated prompts:
17

---

Published as a conference paper at ICLR 2026
Method
Metric
CLIP VIT-B/16
CLIP RN50
Baseline
Acc.
62.99
51.75
ECE
3.93
4.04
Acc.
64.84
54.00
TPT
ECE
10.23
11.48
Acc.
64.60
53.64
С:ТРT
ECE
4.74
О-TPT
Acc.
63,54
52.51
ECE
3.91
4.88
A-IPT (Ours)
Acc.
63.89
52.40
ECE
2.73
3.32
Table 11: Comparison of Weighted Average Accuracy and Expected Calibration Error (ECE) with CLIP VIT-B/16 and CLIP RN50 backbones.
well-calibrated prompts (similar accuracy, higher poor-calibrated prompts (lower AD, higher ECE):
AD, lower ECE):
sculpture of [CLS] Acc: 61.8. ECE: 6.08, AD: 0.54
photo of [CLS] Ace: 66.8, ECE:
2.12, AD: 0.65
graffiti of [CLS] Ace: 62.4, ECE: 4.19, AD: 0.57
good photo 66.7, BCE: 3.46, AD: 0.62
the [CLS] Ace:
cartoon [CLS] Ace: 62.1, ECE: 2.24, AD: 0.59
photo of the weird [CLS] Acc:
67.1, BCE; 5.38, AD:
As shown, these examples provide important insight into the relationship between AD and calibration error (ECE) within the same accuracy group, aligning with the empirical findings of our paper. We agree that needs lot of prompt templates, datasets, and models support the conclusion of Figure 4 further. However, finding prompt templates within the same accuracy group across different hard prompt styles (Radford et al. 2021 a challenging task.
Lastly, while different prompt template certainly affects calibration error, our empirical results suggest that accuracy not a sufficient indicator calibration, and directly optimizing for angular diversity leads to improved ECE across different settings.
A.12 RESULTS ON STANDARD DEVIATION ACROSS RANDOM SEEDS
Tab. 12 presents the mean and standard deviation of accuracy and ECE over three random runs with different seeds for A-TPT across five datasets with CLIP V backbone. Compared C-TPT and О-TPT, A-TPT shows lower mean standard deviation in both accuracy and ECE, indicating more stable performance randomness in prompt initialization and greater consistency in calibration. This consistency highlights A-TPT's reliability in maintaining stable performance across runs, a critical property for practical deployment scenarios that demand reproducibility and calibration stability.
Metoo
Metric
DTD
Flovers
Foodfii
Caltech
Stanlford
Car
Areratee
Pre-trained Buclhone:
Embodding
512-d
4:21
=:19
D-TPT
Sid. ECB
1.17
1:25
1.14
1.20
1.10
A-IFT (Ours)
Std. ECE
Std. Acc.
#.12
1.15
4:15
1.00
4:10
1.04
4.00
1.08
2000
-0.11
Table 12: Standard deviation of three random runs with different seeds with CLIP VIT-B/16 backbone.
A.13 RESULTS WITH OTHER CALIBRATION METRICS: SCE CALIBRATION PERFORMANCE COMPARISON
Since the ECE metric could suffer from bias, In addition to evaluating model calibration through Expected Calibration Error (ECE), also examine calibration using Static Calibration Error (SCE) (Nixon et al.) 2019), which serves as a class-wiss wise variant ECE. Tab, [3] compares the SCE results across ten thne-graing classification datasets with the CLIP V and CL.P RN50 backbones. The
18

---

Published as a conference paper at ICLR 2026
performance of our proposed A-TPT method is benchmarked against several alternative approaches, including the Baseline, TPT, C-TPT, and О-TPT. A-TPT consistently achieves superior calibration performance, demonstrating substantial reduction in SCE across all datasets. For the CLIP-ViT-B// backbone, our method outperforms the alternatives, with an overall average SCE reduction of up to 0.89 compared 1.06 for Baseline, 1.15 for TPT, 1.11 for C-TPT, and 1.07 for O-TPT. For the CLIP RN50 1.03 compared to 1.22 for Baseline, 1.30 for TPT, 1.27 for C-TPT, and 1.24 for О-TPT. This substantial improvement highlights the efficacy of angular diversity better prompt calibration performance.
Method
DTD
Flowers 102
Food101
SUN399
Aircratts
OxtardPels
Caltechin1
UCF101
EaruAAT
Standford
Cara
Average
Pre-trained Backbone:
CLIP VIF-B/16 Metrie: SCE
Embedding dimension: 512-d
Baseline
1.33
0.59
0.20
0.12
0.52
0.68
0,25
0.52
6.18
0.21
1.06
CTTPT
1.31
0.52
0.22
0.14
0.56
0,58
0,22
0.52
6.81
0.22
1.11
0.56
6.58
A-TPT
111
0.51
0.15
0.11
0,53
0,56
0.13
0.49
5.04
0.22
0.89
Pre-trained Backbone:
CLIP RN50
Metrie: SCE
Embedding dimension: 1024-d
Baseline
1.31
0.66
0.29
0.12
0.54
0.73
0.35
0.54
7.39
0.23
TPT
1.52
0.63
0.25
0.11
0.60
0.54
0.38
0.51
8.23
0.24
1.30
C-TPT
1.43
0.62
0.26
0.11
0.53
0.67
0.32
0.51
8.07
0.23
1.27
О:TPT
1.34
0.50
0.27
0.12
0.51
0.69
0.30
0.50
7.85
0.22
1.24
A-TPT
1.66
0.59
0.24
0.10
0.49
0.58
0.28
0.49
6.20
0.21
1.03
Table 13: Static Calibration Error (SCE) (10e-2) performance comparison across fine-grained datasets
with CLIP ViT-B/16 and CLIP RN50 backbone. The overall best-performing result is in bold.
A.14 RELIABILITY PLOTS AND CONFIDENCE HISTOGRAM
Figs. and | present the reliability diagrams for the CLIP ViT-B/16 and CLIP RN50 backbones, respectively, comparing the calibration performance of C-TPT, O-TPT, and A-TPT across the Aircraft, UCF101, StandfordCars. and SUN397 datasets. For the CLIP ViT-B/16 backbone (Fig, 8). A-TPT exhibits marked improvement in addressing the overconfidence problem, outperforming both О-TPT and C-TPT as evidenced in the reliability diagrams presented in the first, second and third rows of Fig. 8 Similarly. the results corresponding the CLIP RN50 backbone, (Fig. 9 show that A-TPT yields substantially better calibration compared to О-TPT and C-TPT, particularly in reducing the prevalence of overconfident predictions. Additional reliability diagrams classification tasks and natural distribution shifts evaluated on EuroSAT (N < [DI), ImageNet-Ve V2, (N > (D] datasets with both backbones - are provided in Figs. potency of A-TPT under varying conditions.
fine- enainnd
further validating the
C:IP:: Air
C.IPT: UCF
(2) C-TPE Car
(d) C:TT:: SUN
(6) CTIPT: (b)
(e) C-TP: C-TPE:
0-TPP:::AMM Air
0-TPP: UCF
(g) O/TPP: Car
(b) SUN
(0) O-TPT: Air
( O-TPT: UCF
O-TPT: Car
(b) O-TPT: SUN
@ A-T.PT. Air
A-T-T: UCF
A Car
()) A-TP:: SUN A Air
(j) TPT:-PU
( Car
A.P::::: SUN
Figure 8: Reliability diagrams for CLIP ViT-B/16
Figure 9: Reliability diagrams for CLIP RN50 back- bone.
backbone.
In addition to the reliability diagrams on EuroSAT (Fig T0a, confidence histogram diagrams (as shown in Fig. 10b) also included to provide complementary perspective the distribution of
19

---

Published as a conference paper at ICLR 2026
Ertaddiy ATPeams
CHefevenn Reweove A EPT Foei
CLIP VIT-B/16 (a) Reliability diagrams (A-TPT EuroSAT).
CLIP RN50
CLIP VIT-B/16
CLIP RN50
(b) Confidence histograms (A- (P-TP:: EuroSAT).
Figure 10: Reliability and confidence histogram diagrams for EuroSAT on fine- ann-igra classification tasks with CLIP ViT-B/16 (N < (D], left) and CLIP RN50 (N [D], right) backbone.
Suppon
Daggeen AAPUU
Olagues
А-ТРТ: ImageNet-V2 V2
А-TPT: ImageNet-K
А-TPT: ImageNet-V2
А-TРТ: ImageNet-K
(a) CLIP VIT-B/16
(b) CLIP RN50
Figure 11: Reliability diagrams for (maggNNetV2.KK natural distribution shifts with CLIP ViT- B/16 (N > D)) and CLIP RN50 (N < [D] backbone, respectively.
model confidence scores. These histograms illustrate how frequently different confidence levels are assigned to predictions, thereby offering deeper insight into the calibration characteristics of each method. A well-calibrated model is expected to produce a confidence distribution that aligns closely with the true likelihood of correctness, and the confidence histograms further highlight the extent to which A-TPT mitigates overconfident predictions.
A.15
WHY HYPERSPHERE OFFERS OPTIMAL TEXTUAL FEATURE SEPARATION IN A-TPT?
Existing test-time prompt tuning techniques Yoon et al. 2024 Sharifdeen et al. 2025 may not guarantee optimal angular separation among textual features distributed across the hypersphere, which can often result in inconsistent fluctuating, but slightly lower (for > [D], where angular separation (orthogonalization) fails), and showing higher (for N < |D], underutilize hyperspherical space, which available to achieve even greater angular separation (even lower cosine similarities) in cases where TPT fails (these challenging points), illustrated in Fig. 2 cosine similarities between textual features, thereby leading to suboptimal model calibration performance. For better calibration, we hypothesize that mere feature dispersion and orthogonalization are insufficient; instead, promoting uniform angular separation across textual features is more beneficial to achieving optimal calibration performance.
Figs. 12 and 13 present comparative analysis of the cosine similarity distribution among textual features generated by the T-TPT and our proposed A-TPT method under varying dataset regimes. Fig. 12 examines the natural distribution shifts (ImageNet-V2 (Recht et al.] 2019)) with the CL.IP ViT-B/16 backhone, which is shown for the case where the number of classes exceeds the embedding dimensionality (N > [D]. Under this regime, A-TPT exhibits a slightly higher but markedly more consistent mean cosine similarity score across each data sample relative to О-TР.. The corresponding histograms further illustrate that A-TPT produces narrower distribution (lower variance) of cosine similarities, indicating greater angular uniformity and stability in the angular relationships among textual feature embeddings.
Fig. 13 present a similar analysis on the fine- tiee-graind classification tasks (EuroSAT (Helber et al. 2018)) with the same backbone, representing the scenario where the number of classes is lesser
20

---

Published as a conference paper at ICLR 2026
than the embedding dimensionality (N < [D]). In this setting, A-TPT yields substantially lower and more consistent mean cosine similarity score and narrower distribution (lower variance) around the mean relative to O-TPT, as evidenced by both the sample-wise cosine similarity plots and corresponding histograms. This suggests that A-TPT is particularly effective at promoting greater angular diversity and increasing feature dispersion when the hyperspherical space is underutilized.
Overall, these findings highlight A-TPT's capability to enforce stable and uniform angular separation among textual features, regardless of the relationship between the number of classes and embedding dimensionality. These empirical findings validate that angular diversity enables optimal angular separation, better utilization of the hyperspherical space, and increased dispersion among features are indicative of clearer class boundaries and improved calibration performance.
Heas cot de be Hel
da bodyy Hillogana cary stard
124
Corte
О-TPT: ImageNct-V2 V2
A-TPT: V2
(a) Cosine similarity between features.
(b) Histogram of cosine similaritics.
Comparison of mean cosine similarity changes on a natural distribution dataset (ImageNet- V2) (Recht et al., 2019) with CLIP ViT-B/16 backbone. When N > |D] our A-TPT offers slightly higher but more consistent cosine similarity values among text features for all the data points.
Figure 12:
Muun cop bo dite by Uhitgrn OUPY
Plunn cod to dir Bur Histogeer Cpra
122
12
X12
6/e
O-TPT: EuroSAT
А-TPT: EuroSAT
(a) Cosine similarity between features.
(b) Histogram cosine similarities.
Figure 13: Comparison of mean cosine similarity changes on a fine-grained classification dataset (EuroSAT (Helber al., )018)) with CLIP V backbone. When N < |D] our A-TPT offers much lower and more consistent cosine similarity values among text features for all the data points.
A.16 SOME THEORETICAL ASPECTS ON A-TPT OVER O-TPT.
Firstly, based on the Tammes problem TTamess 1930), the uniform distribution (best-packing) means the distance between two points is maximized; therefore, while A-TPT builds upon the TPT framework, it introduces a novel angular diversity that is boosted to the utmost extent. Angular diversity (AD) focuses on the uniform distribution - maximizing the angular distance between textual features corresponding to their actual class labels, thereby maximizing the inter-class feature separability for better prompt calibration. Prior work (Wang & Isola 2020) has shown that uniformity (uniformly distributed feature points on the unit hypersphere) preserves maximal information, closely associated with strong zero-shot CLIP performance. Therefore, we argue that uniformly distributed textual features are the way to ensure better class separation and improve VL.M calibration. Unlike orthogonality constraints pursue orthogonality for all the pairwise textual features and suffer from poor calibration when N D). our numerical optimization (A-TPT) is robust with negligible computational overhead across both N > D and N < [D] settings. Secondly, inspired by insights from ArcFace (Deng et al. 2019), Figs. 2 and Appendix Af of the main paper demonstrate that AD gets the greatest minimum pairwise angular distance across all N [D] & N < [D) cases, and therefore the most diverse prompt vectors. Thirdly, as discussed in the paper, orthogonal regularization tends group textual features closer, especially when the number of classes is greater
21

---

Published as a conference paper at ICLR 2026
Method
Metric
DTD
Floverst02
Foodi01
SUN397
Aircrafs
OxfordPets
Caltedh101
UCFI0I
EarsAA
Standford
Cars
Average
Pre-trained Backbone: CLIP VIT-8/16
Embedding dimension; 512-d
Baeline
Acc.
38.40
64.50
81.40
62.40
22.70
86.20
88.10
67.60
34.60
66.50
61.24
1.10
14.10
14.10
TPT
Acc.
45,50
67.90
84.90
65.90
24.50
87.40
91.50
66.40
43.50
67.20
64.45
13.30
6.34
14.10
18.20
12.09
C-TPT
Acc.
BCE
46.30
18.00
69.60
10.60
84.10
2.43
63.50
10:70
24.70
10.50
88.80
1.59
91.70
1.89
67.00
7.42
43.00
8.73
66,90
1.64
64.76
7.35
63.92
0-TPT
ECE
12.85
4.67
1.85
2.67
6.37
3.59
3.00
4.08
8.33
2:71
5.01
A-TPT
Acc.
40.84
69.27
82.84
62.59
23,58
87.54
92.13
67.38
43.90
66.54
63 66
Pre-trained Backbome: CLIP RN50
Embeddiing dimension: 1024-d
Baseline
Acc. BCE
39.60
57.70
73.00
56.50
16.10
79.80
80.90
56.30
21.90
56,90
60.22
6.94
5.14
1.49
3.33
6.42
3.30
4.79
3,76
13.90
4.83
5.39
39.20
59.70
56.81
IPT
PCB
24.80
17.00
7.93
11:40
17.50
7.31
6.02
14.40
15.70
4.49
12.65
C-TPT
39.10
67.00
76.00
60.30
17.40
83.50
87.10
59.60
26.10
$7.20
57.33
BCE
18.00
6.34
3,70
8.28
13.50
1.75
2.85
8.82
11.20
1.65
7.61
O-TPT
Acc.
BCF
40.54
12.42
65.49
3.03
75.51
1.32
58.98
3.35
15.99
8:36
83.78
4.47
86.98
3.53
58.79
3.27
26.89
7.21
56.77
2.74
$6.97
A-TPT
ECE
Acc.
39.79
6.61
64.32
2.97
74.21
1.39
58.36
2.79
16.03
6.11
83.44
3.53
86.89
2.89
58.33
2.83
26.37
3.34
56.37
2.19
56.41
3.46
Table 14: Comparison of calibration performance with CLIP ViT-B/16 and CLIP RN50 backbone with the prompt of "a photo of the cool [CLS]".
Method
Metric
DTD
Flowers
F0od101
SUN399
Aircrafis
OdardPes
Caltechton
UCF101
EurSAT
Standford
Cars
Average
Pre-truined Backhone:
CLIP VIT-B/16
Embedding dimention: 512-d
Baseline
Ace.
42.40
64.70
83.90
61,40
22.30
82.50
90:90
64.80
38.80
64.60
61.63
13.40
TPT
Ace.
45.80
69:40
84:80
65.30
22.90
83.00
93.00
67.10
40.70
67.30
63.93
BCE
20.50
12.20
5.05
7.94
16.20
7.30
2.91
11.60
20.80
6.26
11.07
C-TPT
BCB
15.50
71.50
4.49
1.36
66.00
3.54
9.05
86.90
2,89
1.62
3.87
$1.50
5.18
1.75
4.93
Acc.
45.45
70.32
84.79
64.50
22.77
87.76
93.35
65.40
51.01
66.25
65.16
BCE
11.79
3.22
2.92
4.62
7.92
3.29
3.24
2.63
5.08
1.92
4.66
A-TPT
ECE
Acc.
43.44
70.53
3.13
84.80
1.64
65.37
3.44
22.42
86.64
2.44
93.32
65.50
2.54
46.10
3:20
65.12
64.32
Pre-trained Backhone: CLIP RN50
Fmbbeddng dimonsion: 1024-a
Bascline
Ace. BCE
41.10
58.10
75.20
56.20
16.10
75.70
80.30
56.30
25.50
55.80
48.45
5.20
3.04
3.31
3.68
4.80
2.52
7.91
3.76
9.43
4.80
4.85
41.20
76.10
77.20
56.77
BCE
20.20
12.20
4.83
8.19
15.20
6.98
5.12
15.30
11.10
5.52
10.46
C:TPT
Acc.
41.20
65.40
75,80
61.40
17.60
78.00
88.40
58.40
30.40
57.10
57.37
15.60
1.90
O-TPT
BCE
Acc.
41.19
13,59
65.49
2.49
75.62
1.47
60.97
3.38
16.71
6.60
77.79
2.55
2,56
36
$7.94
6.20
33.32
5.07
56.73
2.69
$7.41
4.66
A-TPT
41.09
65.24
75:36
59.85
16.43
64
87.70
64
31.82
56.51
57.03
ECE
7.05
2.43
1.22
3.19
6.15
2.52
2.18
5.57
2.76
2.52
3,56
Table 15: Comparison of calibration performance with CLIP VIT-B/16 and CLIP RN50 backbone with the prompt of "an example of [CLS]".
than the embedding dimension. Fourthly, we argue that the improvement over other methods is significant.
A.17 CALIBRATION PERFORMANCE OF DIFFERENT PROMPT INITIALIZATIONS
This section evaluates the calibration performance of the proposed A-TPT approach when ini- tialized with different prompt templates, such as "a photo of the cool [CLS]" and "an example of [CLS]* ]", evaluated across CLIP ViT-B/16 and CLIP RN50 backbones. Tab. 14 ports the calibration results of A-TPT with the prompt "a photo of the cool [CLS] with tunable context tokens. For CLIP ViT-B/16 backbone, A-TPT achieves an overall reduced Expected Calibration Error (ECE) of 3.05, compared 5.01 for O-TPT and 7.35 for C-TPT. Similarly, with
22

---

Published as a conference paper at ICLR 2026
the CLIP RN50 backbone, attains calibration error of 3.46, compared to 4.97 for O-TPT and 7.61 for C-TPT.
Similarly, Tab. 15 presents the calibration results for the prompt "an example of [CLS]" with 4 tunable context tokens. In this setting, A-TPT again outperforms C-TPT, achieving a reduced calibration error of 3.30 (CLIP VIT-B/16) compared to 4.66 for O-TPT, and 4.93 for C-TPT. For (CLIP RN50) backbone, A-TPT attains an ECE of 3.56 compared to 4.66 for O-TPT and 5.48 for C-TPT. These results consistently demonstrate that A-TPT maintains strong calibration capabilities (reduces calibration errors) across different prompt initializations, showcasing its robustness and adaptability diverse settings and affirming its potency better prompt-base based VLM calibration.
A.18 CALIBRATION PERFORMANCE WITH COMBINED C-TPT AND A-TPT
Tab. 16 presents the calibration results of a combined approach that integrates C-TPT and A-TPT. The findings indicate that combining A- APTPT with C-TPT leads to superior calibration performance and can outcompete A-TPT alone. This enhancement reveals the generalizability of A-TPT over stronger baseline.
Method
Metrie
DTD
Flowers102
UCF101
C-TPT
Acc.
46.00
69.80
65.70
ECE
11.90
5.04
2.54
Acc.
45.51
69.22
66.16
A-TPT
ECE
4.76
3.61
2.12
Acc. ECE
45.15
69.51
66.23
C-TPT + A-TPT
4.11
3,49
1.96
Table 16: C-TPT A-TPT on DTD, Flowers102 and UCF101.
A.19 PARETO FRONT: VISUALIZING THE EFFECT OF A-TPT
Fig. I4 presents the Pareto frontier analysis on the Flowers 102 and Food101 datasets, highlighting the trade-off between classification accuracy and Expected Calibration Error (ECE) across varying values of X's. The proposed A-TPT method does not merely achieve higher ECE at the expense of lower accuracy. Instead, it seeks optimal balance between these metrics than TPT, C-TPT, and O-TPT across a wide range of X settings in two datasets. This suggests that A-TPT effectively optimizes the trade-off, achieving superior model calibration performance without compromising predictive accuracy and providing more detailed picture of the performance characteristics of A-TPT, ensuring that the increased performance is not solely due the better hyperparameter
The difference in calibration behaviour of the Flowers 102 and Food101 datasets when increasing À is due dataset characteristies and how model confidence (over- vs. under-confidence) distributed. Refer to Fig. 5, the reliability diagrams of C-TPT and O-TPT show overconfidence on Flowers102 and underconfidence on 1o1.
Flowers102. likely contains more fine-grained, visually similar classes, where the baseline model often makes incorrect predictions with high confidence (overconfidence). To correct overconfidence, the model must lower its confidence in ambiguous samples. This necessary softening of the probability distribution can occasionally shift high-confidence incorrect prediction low-confidence correct incorrect state. This results in the observed trade-off with increased trading, some accuracy as the model becomes more conservative to improve ECE.
Food101: features more visually distinct classes, where the model makes correct predictions with uncertain or unreliable confidence estimates (under-confidencce. Increasing maximizes the margin between these distinct features. This pushes the correct class probabilities higher (mitigating under-confidence) without introducing ambiguity. This helps A-TPT to lower ECE without sacrificing accuracy (or even gaining a little by sharpening the decision boundary).
That's why A-TPT shows trend trading some accuracy for improved ECE Flowers while on F00d101, A-TPT shows a contrasting pattern, lower ECE without sacrificing accuracy (or even gaining little) for better calibration.
23

---

Published as a conference paper at ICLR 2026
As shown in the Fig. 14 A-TPT seeks better balance overall rather than sacrificing accuracy for better calibration, exactly as observed on Food101, while the accuracy-EC3 trade-off on Flower102 across X, confirms that increased performance could not be a result of better A.
The reported curves are averaged over three seed runs. In Tab. 12 we have reported 3-seed variability for A-TPT (ViT-B/16), with low standard deviations (Flowers Std Acc. 0.009 Std ECE =0.15; Food101: Std Ace. +0.04, Std ECE =0.10), confirming the curve shapes are stable across runs.
Flowers102
Food101
59
22
05
68
55
23
15
Lambdd MM
Lamboda (A]
13
13
Acouncc
Figure 14: Pareto front analysis on Flowers102 and Food101. The represents A-TPT (Ours). compared against zero-shot baseline, TPT, C-TPT, and О-TPT.
A.20 CAN ANGULAR DIVERSITY LOSS DEGRADE ACCURACY FOR SEMANTICALLY OVERLAPPING CLASSES?
We distinguish A-TPT results on fine-grained, semantically overlapping datasets like Oxford Pets (37 breeds), Stanford Cars (196 models), FGVC Aircraft (100 variants), Flowers- 102, and Food-101 across CLIP-ViT-B/16 and RN50 backbone.
OxfordPeis se
Staeford
Aircratts
Flowers toz
Food o1
Overali
Method
Metric
Currs
VIT-IV16
Embeedding
312
BCB
5.77
5.16
16.30
13.50
3.98
9.04
C-FFT
BCB
Acc.
$8.20
1.90
65.30
1.59
24.85
4.36
69.80
5.04
$3.70
3.43
66.47
3.26
O-TPT
BCB
Acc.
87.95
1.90
61.53
1.78
23.64
3.68
70.07
3.87
$4.13
1.46
66.06
2.54
A-TPT (Ours)
Acc. ECE
88.33
65.78
23.76
69.22
3.61
83.64
66.15
2,08
Pre-trained Backbone: CLIP RN50
Ambeddiing dimouston:: 1024-d
TPT
Asc. BCB
84.50
3.65
58.00
3.76
17.00
16.10
62.50
13.40
74,90
5.25
59.38
8,43
C-IPT
Acc.
BCB
$4.10
2.77
36.30
17.00
10.70
65.20
4.14
74.70
1.86
59.50
4.28
0-TPT
56.44
16.77
65.61
74,22
3.41
A-TFT(QQur)) (Qurs)
ECE
2.70
Table 17: Accuracy and ECE on semantically overlapping datasets. A-TPT preserve accuracy comparable C-TPT/O-TPT while consistently lowering ECE.
CLIP Overall Ace. remains stable (66.10 - 15), while ECE drops significantly (9.04 - 2.08). Notably, accuracy actually improves datasets like OxfordPets (87.10- 88.33) and Aircrafts (23.40 + 23.76)
CLIP RN50: We observe minor trade-off in overall accuracy (59.38 58.83), while ECE drops significantly (8.43 + 2.70). Notably, accuracy actually improves on datasets like Flowers 102 (62.50- 64.89).
24

---

Published as a conference paper at ICLR 2026
Unlike orthogonality constraints (O-TPT) that foree features to be 90" apart, potentially breaking semantic clusters, A-TPT solves the Tammes best-packing problem. maximizes the minimum angular distance available. This allows semantically overlapping classes remain relatively closer to each other than to dissimilar classes, provided they utilize the hypersphere effectively, rather than collapsing into identical vectors.
Empirically, in our runs, we observe small accuracy drops (typically < 1-2) in a few cases, for example, Food101 (ViT-B/16), and Aireraft (RN50), while substantially lowering ECE, Overall accuracy remains comparable, and making the trade-off favorable for improved calibration. A- TPT actually improves accuracy over TPT, suggesting that angular diversity can help disambiguate fine-grained features rather than degrade them.
In principle, any text dispersion regularizer (L2, orthogonality, or angular diversity) can degrade top-1 accuracy if over-weighted on semantically overlapping classes syar--yynnymous labels or fine-grained siblings). The stability of any regularizer in these scenarios due its formulation as an auxiliary loss. In A-TPT, angular diversity is auxiliary term added to the standard TPT objective (entropy minimization). TPT still boosts accuracy, and angular diversity improves calibration. In Appendix A.4, limitations, we acknowledge this trade-off and fix moderate to improve calibration while avoiding reeraaaauliinnnoon on near-synonyms.
25


Reviews:
[{"review_id": "cSvBPbM3QT", "reviewer": "Reviewer_Aoux", "summary": "The paper proposes Angular Diversity for Test-time Prompt Tuning (A-TPT), instead of pushing text prompts to be dispersed by l2 distance or cosine similarity, it maximizes the minimum pairwise angle between normalized class-wise prompt vectors (a maximin objective using θ=arccos of cosine similarity). This directly spreads prompts on the unit hypersphere to promote more uniform coverage and better calibration during inference. The authors also argue why this angle-based objective has stable gradients even when vectors are very close—unlike the orthogonality loss whose gradient vanishes as angles go to 0.", "questions": "See weakness section", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Rather than optimizing L2 distance or cosine similarity, the paper optimizes the **angle itself,** which better captures geometric separation on the unit sphere and compensates for the shortcomings of previous work. This paper shows the limitation of previous work well.\n- The paper includes extensive analyses that illuminate the method’s behavior from multiple perspectives, aiding interpretation and practical use.\n- It explicitly examines the calibration differences between N > |D| and N ≤ |D|, a case prior work largely overlooks, and clarifies where the proposed method offers the biggest gains over O-TPT.", "weaknesses": "- When we increase λ, we understand this as trading some accuracy for improved ECE (better calibration). This trend aligns with Flowers102, but Food101 shows a contrasting pattern. Could you provide insight into why the two datasets behave differently? Also, are these curves averaged over multiple seeds, and how large is the variance across runs?\n- How did you choose the λ term?\n- In the main performance table, could you report results separately or make them explicitly distinguishabl for the N>|D| and N≤|D| regimes? This would help isolate where your method provides the most benefit over O-TPT.\n- How do you ensure numerical stability when computing arccos?\n- While increasing the minimun θ can encourage dispersion, it doesn’t seem sufficient to prevent localized density (clustering) in certain regions. Do you have any guarantees or empirical evidence that your method avoids such partial clustering?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4SlUXtLP7k", "reviewer": "Reviewer_r5c7", "summary": "This paper targets the calibration problems that arise when doing test-time prompt tuning (TPT) on vision-language models and argues that existing fixes (like text feature dispersion (C-TPT) and orthogonality constraints (O-TPT)) don’t actually guarantee that class-wise text features are well separated, especially when the number of classes exceeds the embedding dimension. To address this, the authors propose A-TPT (Angular Test-time Prompt Tuning), which adds an angular diversity regularizer that, for each class embedding, maximizes its minimum angular distance to any other class, encouraging a more uniform packing of text features on the unit hypersphere. According to experiments on fine-grained, distribution-shifted the proposed method reduces expected calibration error (ECE) while largely preserving TPT’s accuracy gains over zero-shot CLIP and prior TPT variants.", "questions": "See weaknesses above.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* The paper clearly identifies the shortcomings of prior text-feature dispersion approaches and uses Figure 2 to illustrate them effectively.\n\n* The reported ECE gains over baselines such as C-TPT and O-TPT are also encouraging.\n\n* The authors show the method also works not only on standard benchmarks used to evaluate CLIP performance, but also on 'calibration critical applications' such as medical domain in Table 4.", "weaknesses": "* Although the paper proposes angular diversity regularization as a new metric, the method still operates within the existing C-TPT and O-TPT test-time adaptation paradigm, so the contribution feels more incremental than fundamentally novel in terms of theory or technique\n\n* Could the proposed method be a complementary to previous methods (e.g., C-TPT or O-TPT). That is, could we for example enforce the proposed angular diversity on top of textual dispersion proposed by C-TPT or the orthogonality constraints of O-TPT. It would be interesting to see such an ablation.\n\n* Since the ECE metric could suffer from bias, could the authors report calibration metrics other than ECE as well?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wk9k2Rhkn7", "reviewer": "Reviewer_RJNf", "summary": "This paper describes a new criterion for test-time prompt tuning in order to reduce calibration error.  It has previously been noted that calibration error can be reduced by designing a prompt template so that, when the different class labels are inserted into the template, the resulting prompts have maximally dispersed text vectors.  C-TPT measured dispersion using mean L2 distance, which does not guarantee pairwise separation; O-TPT guaranteed pairwise dispersion of 90 degrees if the number of classes is less than twice the number of embedding dimensions, but not otherwise.  The proposed A-TPT minimizes the maximum pairwise cosine similarity of classes, thus maximizing pairwise dispersion.", "questions": "On p. 4, what does it mean when the same prompt appears in both the list \"Hard prompts\" and the list \"Tuned prompts,\" but with different Accuracy, ECE and AD?\n\nEq. (1) min_{j,j\\ne i} should be min_{i,j\\ne i}", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 2, "contribution": 3, "strengths": "Derivations are interesting and clear. \n\nEquations and derivations seem correct. The point about \\arccos normalizing gradient magnitudes is quite interesting.  I find multi-letter variable names aesthetically displeasing in general, but the use of \"Cos\" as a variable name does not impair legibility or correctness in this case.\n\nResults show significant consistent reduction in calibration error, with small and inconsistent changes in accuracy, across 15 datasets, in comparison to TPT, C-TPT, and O-TPT.", "weaknesses": "Minor: Fig. 3 clearly shows that the prompts with the highest ECE (\"the nearest shape in this image is\" and TPT) are clustered in the center, while other prompts are distributed.  This does not show, however, that the prompts with high ECE have low angular diversity, because t-SNE does not show the angles of vectors: it only shows their cluster structure.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1EPkPzOZEq", "reviewer": "Reviewer_5gVD", "summary": "normalized textual features on the unit hypersphere, ensuring uniform distribution an A-TPT enhances test-time prompt tuning (TPT) by introducing angular diversity calibration. Instead of maximizing dispersion or orthogonality alone, it maximizes the minimum pairwise angular distance between d better", "questions": "1.\tHow sensitive is A-TPT to initialization of textual prompts?\n2.\tCan angular diversity loss degrade accuracy for semantically overlapping classes?\n3.\tIs there any theoretical bound on ECE improvement from uniform angular spacing?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1.\tElegant mathematical framing grounded in Tammes best-packing problem.\n2.\tClearly articulated motivation: poor calibration under low angular diversity.\n3.\tDemonstrates lower Expected Calibration Error (ECE) with negligible accuracy loss.\n4.\tTheoretically principled and easy to integrate.", "weaknesses": "1.\tIncremental relative to O-TPT and C-TPT; lacks major conceptual leap.\n2.\tNo runtime or convergence analysis for numerical optimization.\n3.\tExperiments mostly on classification; unclear utility for generative tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)