CARPRT: Class-Aware Zero-Shot Prompt Reweighting for Vision-Language Model
Authors: Ruijiang Dong
Zesheng Ye
Jianzhong Qi
Lei Feng
Feng Liu
Gang Niu
Masashi Sugiyama
Venue: ICLR 2026 Poster (2026)
Keywords: Prompt Weighting
Prompt Ensembling
Pre-trained Models
Vision-Language Models
Primary area: foundation or frontier models, including LLMs

Abstract: Pre-trained vision-language models (VLMs) enable zero-shot image classification by computing the similarity score between an image and textual descriptions, typically formed by inserting a class label (e.g., "cat") into a prompt (e.g., "a photo of a").
Since the score for a given image-class pair is sensitive to the choice of prompt, existing studies ensemble multiple prompts using a weighting vector to aggregate scores across different prompts.
Yet, in current strategies, the weighting vector assigned to each prompt is shared across all classes, implicitly assuming that prompts are conditionally independent of classes, which often does not hold in practice, as a prompt like "an aerial view of" might be apt for "airport" but ill-suited for "apple".
To address this, we propose class-aware zero-shot prompt reweighting (CARPRT). 
This scoring scheme adjusts the weighting vector for each class label by capturing the class-specific relevance of different prompts in a training-free manner. 
For each class label and every available prompt, we quantify their class-specific relevance by averaging image–text relevance scores over images predicted to that class under the given prompt. These estimates are then normalized to derive class-specific weights.
Evaluations on standard image classification benchmarks show that CARPRT outperforms existing class-independent reweighting methods, confirming that modeling prompt-class dependencies is crucial for effective zero-shot prediction and even broader VLM-based application settings that rely on prompt ensembling. Our code is available at https://github.com/tmlr-group/CARPRT.

Published as a conference paper at ICLR 2026
CARPRT: CLASS- AWARE ZERO-SHOT PROMPT REWEIGHTING FOR BLACK-BOX VISION-LANGUAGE MODELS
Ruijiang Dong 1; Zesheng Ye:; Jianzhong Qi', Lei Fenga.,, Feng Liu Gang Niu Masashi Sugiyama Univvrsiity of Melbourne, *Southeast University,
'RIKEN Center for Advanced Intelligence Project, "The University of Tokyo rui j suuett.... unimell 1b. edu. au, fenglei @seu. edu. en {zesheng. ye, jianzhong.qr gi, leng.liull Junnimelleeelllee edu. au gang . miu. ml @gmaill 1 com, sugi@k. a-tokyo. ac. ip
ABSTRACT
Pre-trained vision-language models (VLMs) enable zero-shot image classification by computing the similarity score between an image and textual descriptions, typi- cally formed by inserting a class label (e.g., "cat"" into prompt (e.g., "a photo of a"). Since the score for given image-class pair is sensitive to the choice of prompt, existing studies ensemble multiple prompts using a weighting vector to aggregate scores across different prompts. Yet, in current strategies, the weighting vector signed to each prompt is shared across all classes, implicitly assuming that prompts are conditionally independent of classes, which often does not hold in practice, prompt like "an aerial view of"" might be apt for "airport" but ill-suited for "apple". To address this, propose class-aware zero-shot prompt reweighting (CARPRT). This scoring scheme adjusts the weighting vector for each class label by capturing the class saass-specifie relevance of different prompts in a training-free manner. For each class label and every available prompt, we quantify their class-specific relevance by averaging image-text relevance scores over images predicted to that class under the given prompt. These estimates are then normalized to derive class-specific weights. Evaluations on standard image classification benchmarks show that CARPRT outperforms existing inaseinndeenndet reweighting methods, confirming that mod- eling prompt-class dependencies is crucial for effective zero-shot prediction and even broader VLM-baseed application settings that rely prompt ensembling. Our code is available at https: / /ttp//ggttt/b//////////////htbht com/ tml m grouup CARPRT.
INTRODUCTION
Vision-language models (VLMs) have transformed how machine learning models interpret visual content by jointly leveraging visual and textual modalities. Models like CLIP (Radford et al., 2021) and DeCLIP (Li et al., 2022) enable zero-shot image classification by computing similarity scores between image and textual descriptions of class labels, then predicting the label with the highest score. By forming textual descriptions of labels (e.g.. "a photo of a [label]"), this approach- known prompting promoeing-remooes the need for task-specific training to recognize visual concepts.
removes
However, these models' zero-shot performance is sensitive to the precise wording of prompts, as subtle phrasing changes significantly alter the perceived relevance of visual features, leading to different similarity scores and classification outcomes (Radford et al., 2021), Identifying phrasings that remain effective across diverse visual concepts is challenging and often yields inconsistent results across datasets (Allingham et al., 2023). This sensitivity means that manually crafting optimal prompts for each class or dataset, while helpful for performance, becomes laborious and unreliable in large-scale settings. Recent work has explored using large language models (LLMs) to generate
Equaa contribution. Corresponding Author.

---

Published as a conference paper at ICLR 2026
Accuracy Comparison by Class
Prompt Weight Comparison
Accuracy (%)
100%
Uniform weight
0.005
WPE
Accurary drop
Marigoid
Weight
0.004
Lotus
60%
40%
0.002
0%
Marigold
Petunia
Latus
Columbie
Tiger Lily
0.000
(a) Accuracy comparison between class- agnostic WPE and class-specific WPF.
(b) Optimal weight comparison between class- specific WPE and class-agnostic WPE.
Figure I: Empirical motivation for class-specific weighting Flower102 (Nilsback & Zisserman, 2008). We showease the results of five classes by shifting from class-agnostic WPE to class-specific WPE (using ground-truth labels), and the estimated optimal weights under two weighting schemes, confirming that optimal prompt weights are class-dependent.
richer class descriptions, but this introduces heavy computational overhead, reducing the efficiency that makes zero-shot methods attractive the first place.
This paper focuses on a more prevalent question:
improving zero-shot classification when only fixed set of predefined prompts and unlabeled images are available at inference, under black-box access, which requires methods that leverage only the inference data optimize prompt utilization. A common strategy is prompt ensembling, which averages embeddings of multiple prompts to produce stable class representations (Radford et al., 2021). However, this approach assumes equal prompt contributions- simplification that harms downstream performance when misaligned templates are included. Allingham et al. (2023) advanced this concept by determining prompt-specific weights using unlabeled data, depending on how compatible each prompt is with the downstream task. This method achieves results comparable to manually selected templates. Still, while such methods vary weights across prompts, they assign the same weight across all classes to each prompt.
We argue that this class-agnostic reweighting suboptimal. Intuitively, different semantic classes vary in their affinity to different prompts. For example, a prompt like "This is photo of a [label). type of fruit" is more relevant to class "strawberry." but ill-suited for class "lamb", which would better match "This is photo of [label], type of animal" instead. This implies that optimal prompt utilization may require class-specific considerations. To validate this intuition, we conduct controlled proof-of- concept experiments on the Flower 102 dataset (Nilsback & Zisserman, 2008) (Fig. 1). By applying Weighted Prompt Ensembling (WPE) (Allingham et al., 2023) independently to images of each class (thus simulating "perfect" class-specific knowledge for weight estimation), we observe accuracy gains compared to global WPE that estimates a single set of class-agnostic weights (Fig. (a)). Moreover, the optimal prompt weights vary across classes' (Fig. 1(b)), rather than being globally shared.
We further study this observation theoretically and present a probabilistic framework (Sec. 3) to clarify the underlying mechanism of prompt ensembling. We show that class-agnostic weighting schemes, such as WPE, indeed implicitly assume conditional independence between the class label and the prompt weights given an image. This assumption, however, may not always reflect real- world data characteristics and limit the expressivity of such weighting schemes as a result.
Building on insights, we introduce Class-A ware Zero-shol Prompt ReweighTing (CARPRT), training-free method to infer class-specific prompt weights using only unlabeled images. Unlike our controlled proo--of-concept experiment, CARPRT does not require ground-truth labels for weight estimation. Instead (Sec. 4), for each image, CARPRT first calculates similarity scores against all possible prompt-class combinations using a pre-trained VLM (e.g., CLIP (Radford al., 2021)) via forward inference only. It then assigns a pseudo-class label to the image based on the combination yielding the highest score. These pseudo-labels are then used to aggregate information for class- specific weight derivation: for each class, the weight for given prompt is determined by the maximum similarity that prompt achieves in conjunction with that (pseudo-)ed -cllss across the reference images. This scheme helps tailor the prompt ensemble to the semantic content of each category.
'The prompt templates denoted in Fig. 1(b) are: P1 = =a photo of a, a type of flower.", P2 "satellite photo of " P3 "a close-up photo of the P4 = "a drawing of a."
2

---

Published as a conference paper at ICLR 2026
We empirically evaluate CARPRT on ten fine-grained zero-shot classification benchmarks (Sec. 5), ImageNet (Russakovsky et al., 2015) (and its variants), and explore its utility in broader VLM-based adaptation scenarios such as prompt tuning (App. G). Our results show that CARPRT consistently outperforms existing prompt ensembling/eewiighingg schemes across VI.M architectures and back- bones, highlighting that incorporating class- awareness is way maximize the potential of prompt ensembling for zero-shot classification, with potential benefits for a wide range of VLM applications.
2 PROBLEM SETTING AND RELATED WORK
Zero-Shot Prediction with VLM. VLMs such as CLIP (Radford et al., 2021) achieve visual-text aligmment through large-scale scale contrastive pre-training. It consists of an image encoder f X - Z and text encoder T + Z, mapping images from space X' and texts from space y into shared embedding space 2. The alignment is driven by maximizing the cosine similarity between the embeddings of matched image-text pairs while minimizing it for non-matched pairs.
This alignment enables zero-shot image classification. For a set of C classes y/ {u),--. DC each class Ve is mapped to text description te via prompt template p: y T, such as to ="A photo of 3". The text encoder g(-) then produces class embeddings 2² = =[2] 2 zt. where x = g(x)) for E .1, C). Given an image æ E X with its embedding 2 = ((x) the predicted class given by 9 = arg max.e (I.....C} sim (zl >x=) i.e., one whose text embedding =! has the highest cosine similarity with z1. This allows for zero-shot classification based on semantic alignment without task-specifi fine-tuning. Yet, the classification performance is highly sensitive to the choice of prompt template p. An ill-suited template can lead to misaligned class embeddings.
This work focuses mitigating this sensitivity by ensembling multiple predefined templates IP = {PI..... Pa particularly when IP is fixed, without relying on additional labeled data. That is, in the zero-shot classification setting, we consider the following problem?
Problem 1 (Prompt Ensembling). Given pre-trained VLM with an image encoder f and a text encoder g, a label space y with C classes, a fixed prompt template set P with [PI = n, and an unlabeled image dataset D {Ei,..., 3mh ). construct the class embeddings z" using prompt weight matrix W E RaxC, where each row W = [Wi.e:... .... Wr,el" refers to weights of n prompts for class Ve E V subject Ws,c 2 0 and Σ Wi,- 1. The text embeddings for class ye thus
2
22,1
2n.1
W1.1
W1,2
W1.C
(1)
n
11,C
22,
Wn.1
Wn,2
Wn,C
where 2T, = ((P((U))) is the text embedding for class Ve under prompt pa. The objective is then to find the set of all such weight vectors W WW (We) that would (ideally) minimize the empirical zero-shot classification error over the unlabeled dataset TD, ie., correctly predict the (unknown) ground-truth label yj by Új for each æ E D..
Existing prompt ensembling schemes can be viewed constrained versions of the general formulation in Problem 1, differing primarily in how they determine the prompt weights W.
Mean Prompt Ensembling (MPE) as a Solution. The most straightforward approach, MPE (Rad- ford et al., 2021), averages text embeddings from multiple prompts, equivalently setting wie = 1 for all prompts p: and classes Ye in Eq. such that W reduces an all-ones matrix. MPE seeks to improve robustness over single-prompe usage by diversifying textual inputs. Yet, treating all prompts equally can impair the efficacy if semantically misaligned with the downstream task
Weighted Prompt Ensembling (WPE) Solution. To mitigate the impact of mask-irreeevan prompts, WPE (Allingham et al., 2023) (originally termed ZPF) extends MPE by assigning data- driven weights to the prompts. WPE assesses whether prompt p. yields generally high similarity scores over all classes with samples of I, and up-weights more relevant ones. Each prompt p; is
We note that there are some VL.M adaptation settings, e.g., prompt tuning (Zhou al., 2022a;b), which are the focus of this work. To clarify, App. B details the relationship between Problem I and other settings.
3

---

Published as a conference paper at ICLR 2026
assigned weight via w,,: وド= c (I.... sim (0m 2,,e), which, after normalization, is Wi,C- While WPE can down- wwinh unhelpful
applied uniformly across classes W :1 prompts, still assumes: a prompt deemed useful (or not) is considered so for all classes equally.
Can We Bridge the Gap? As Fig. shows, prompt's efficacy often depends the specific class it describes. Both MPE and WPE largely neglect this class class-prompt prompt interaction, nor attempt understand why class specificity is necessary determine prompt relevance and how statistical tools help to address it. To bridge this gap, we next present a prohabilistic framework, establishing a principled connection between class-aware prompt reweighting and zero-shot classification.
UNDERSTANDING PROMPT REWEIGHTING: A PROBABILISTIC VIEWPOINT
Zero-shot classification with VLMs can be framed as estimating the conditional probability Pr(y x=,, IP, D) of a label y" given a query image æ', set of prompts P. and an unlabeled dataset IC, To understand how prompt reweighting influences this process, develop probabilistic framework that reveals why class-ave aware reweighting is necessary.
Let W W be weight matrix. We begin by marginalizing over the weight space W as
Pr(y"|a" PD J,, Pr(y"|æ", PP, W) Pr(W x" P, D)dW,
where Pr Pr(WW=",, D) can further simplify to Pr(W[ D), since in zero-shot settings, W deter- mined before access the new query image x". This decomposition suggests essential tasks in zero-shot classification: (i) modeling prompt weights Pr(W [P, D) and (ii) making aggregated predictions Pr(y" ae" IP, D, W) weighted by Pr(W IP, D). As such, we will continue to explore how further expansions can inform and align with practical implementations..
Modeling Weight Pr(W[P,D)) Using Bayes' theorem and considering m i.i.d. samples æy E DD,
Pr(W[P,D) x Pr(WIP) Pr (P(W II. T 1=1 Pr(zy W.P),
where Pr(W P) is the prior over weights (details are deferred to App. H) and the data (image) likelihood Pr(x;) [ P) is obtained by marginalizing over classes We further:
Pr(x;/ = Σ
NcEY Pr(xzlue, W. W.P),
which describes how it depends on class priors and class-conditional likelihood.
Modeling Class Prior Pr(ye] P). For zero-shot classification where D is large enough, the class prior Pr(ye) W, IP) can be estimated from pseudo-labels (i.e., predictions from a pre-trained VLM). Proposition 1. Let D = H be unlabeled dataset with unobserved classes y {ye) and Pr(ye) be the true class probability for class Ye- As grows, the empirical class distribution Pr(ye] W, P') from pseudo-labels converges Pr(ye) with exponentially decreasing error probability Specifically, for any > O, we have: (Pr(Pr( W,P) Pr(ya)l 2 c) I 2 cexp(-2mél This implies that we can approximate true distributions by
Pr(yel F,
The Ey De
Vye E y,,
(5)
where =Ep= 1 1. counts the images pseudo-labete as class over all samples in ID)
Modeling Likelihood Pr(æz/ue. lve. W.P. Given that images x, often lie in nigh-dimensional spaces, directly modeling the class-conditional likelihood can be challenging. We therefore adopt Energy- based Models (EBMs) (LeCun et al., 2006) that excel at modeling high-dimensional distributions by defining an unnormalized energy function, normalized by a partition function. Interpreting sim(z), zf²) the negative energy (lower energy means more likely), have
1
Pr(æz|Vo, W,P) = Z(yo, W, P) exp {sim( 2/. x-
(6)
4

---

Published as a conference paper at ICLR 2026
where =/ = f(x,) is the image embedding, x!- (y)( is weighted text embedding for class Ye using W (from W). While the partition function Z (ye, W. .)) = - exp(sim(2*+ =!") )dz makes exact computation intractable, for classification we only need relative likelihoods of different classes.
Lemma (Relative Likelihood). Assume sim(a, a, b) = aa' TT (for Ez-normalized embeddings), then:
Pr(azlue: W,P) x exp (sim(z), (2), ) exp
(wiee T.
2 1
This proportion relationship shows that class-specific weights wi. (for (1....,C) ,.)) indeed determine the influence of each prompt Pa (via its embedding zT,z) on the likelihood for class ye-
Why Class-Specific Weighting Matters. It is easy to check that Lemma (proof in App. 1) aligns with the most general form of prompt ensembling (Eq. 1). Crucially, class-agnostic weighting (i.e., independent) schemes, such as WPE, deviate from this form by unnecessarily imposing shared wi.e for all classes yes which fundamentally limits model expressivity.
Proposition 2. Let X be the image space and y be the class space. Given prompt set P. for any prompt reweighting scheme S, define the representable likelihood set Fg as:
Fs x
E Ws, PP, S.I. f(x, yo) x Pr(x) Yes W.P.
where We is the weight space under scheme S. Let Fai and Fos be the representable likelihood sets induced from coss-independeet weighting (i.e., WPE) and class-specific weighting (ef. Eq. 1) schemes, respectively. Then, we have: 3f Fes such that fft Fct, 3x X. yo E y where (" Be) + fc1( We). That is, Fa is a strict subset of Fcs-
Remark 1. Proposition 2 formally states that class- class-specific weighting allows for capturing a richer set of image-texi relationships than class-agnostic ones. To maximize potential expressivity, prompt weights Wi, must be class-specific to ensure that each class benefits from the most relevant prompts.
Modeling Predictive Probability Pr(y" x""""=="_" D.,,..D. W). We now come to predicting the label ÿ. for the query image T... As zero-shot classification is training-free, a practical way is to approximate full Pr(y"|æ* P. ID, W) with Pr(y" IP, W), where W is point estimate derived from unlabeled data D, per our discussion in Eq. and Eq. 7. By considering each prompt E P. we have
exp (E (Wi, =7,e) z!)
Pr. W) Pr(y"|æ",P, = Σ Pr(y"< (P.,,.
(8)
EVEL, exp Er (Wi.e 2T 2!!
P:EP
By now, we have framed VLM- based zero-shot classification in a probabilistic framework (Eq. 2), justified class-aware prompt reweighting (Propositions 1 and 2), and interpreted how class prediction for a query image can be performed (Eq. 8) under this understanding.
CLASS-AWARE PROMPT REWEIGHTING FOR VLMS
Guided by the probabilistic principles from Sec. 3, we next introduce CARPRT, a minimalistic training-free method designed compute class-specific weights for prompt ensembling in VLMs.
Overview. Given an unlabeled dataset D = {x> 1m an unknown class space y = [v1 Vc a fixed prompt set = {pr ) and a pre-trained VLM accessed via forward inference only, CARPRT aims to find the optimal weight matrix W e = where each column W = [wi.c...., w denotes the relative importance of different prompts for particular class ye and specifies the contribution of each prompt to the class representation, as with Problem Recall that the key insight driving CARPRT is that optimal prompt weights should reflect the semantic alignment between prompts and class concepts. As depicted in Fig. 2, CARPRT implements this insight through two steps: Score Calculation and Weight Calculation (the algorithmic outline can be found in App. D).
5

---

Published as a conference paper at ICLR 2026
Before CARPRT: VLM Forward Pass
CARPRT Stage Score Calculation
Tabel space
descriptions
text embeddings
score tensor
28
Thai pua Jult
Biot Fine
SJA siznt (amn
1/1
M2
VC
Abal COnTERN
Inxt uncuder
Anyaal Parrs
prompt pool P
Downstream Task P
image embeddings
class-apacciic resseighting
This photo
t
a type of food
tor commuting
112 unisbeled Images
enooder
image
Weight Calculation unique optimal weights per class
CARPRT Stage 2:
Winccal MA.1E
Figure 2: The CARPRT pipeline. First, the encoder g and image encoder f yield textual class embeddings (from C' classes and prompts) and image embeddings (from m unlabeled images). Then, compute the score tensor from image-text embedding similarities, cach entry $,,.c measures the relevance between the i-th prompt and the j-th image for the e-th class. Extract pseudo-labels from the score tensor, and derive the class-aware prompt reweighting matrix W, which assigns class-specific weights for each prompt based on the scores.
Stage 1: Prompt Relevance Score Calculation. Eq. and Eq. 4 suggest that estimating weight distribution Pr( (rI, ID) hinges on the individual data likelihood Pr(æe]le, W, W.P.. As Lemma established, Pr(æ, Mee W, IP) is proportional to the VLM's similarity score, which is thus leveraged by CARPRT via forward similarity evaluations to compute similarity scores between all image embeddings and all prompt-derived text class embeddings. For an image zy E D, a prompt template ps E IP, and class Ye E V/ the relevance score 85,i.c is:
$J,.c = sim( zi,c)
where =/} f(xx) ) is the image embedding and z (p:()))) is the text embedding for class ye under prompt p.. This yields score tensor, wherein each entry $j,dc is an unnormalized estimate of Pr(an,lue (y, W, P). The score tensor captures the semantic compatibility among all images D. prompts P, and classes )/ providing the foundation for reweighting prompt-template combinations.
Remark. Eq. expressed using embeddings for convenience, but in practice we only query the VLM for the similarity scores $j.,c. Hence CARPRT works with a black-box CLIP interface and does not require explicit access the emheddings.
Stage 2: Class-Specific Weight Calculation. The second stage transforms unnormalized similarity
scores into normalized class-specific prompt weights through process that mirrors our probabilistic analysis in Sec. 3. By empirically quantifying each prompt's relevance specific classes, the resulting weights ensure that prompts primarily contribute to the aggregated representation of their most semantically aligned classes.
m. ) without any parameter update, by identifying. for each image-prompt pair, the class with the highest similarity score Új.i arg maxy.ey $j Then, we calculate intermediate weight w/ for each prompt-class pair by aggregating the scores $j across all images æ, predicted to class ye under prompt p,. This can be expressed as:
First, we create pseudo-label set V (U)
E A.
W
(10)
ΣØ= 1 13.
Here, =g is the indicator function. Eq. 10 implements an empirical estimate of the class priors. w/, reflects the average strength of association prompt p; shows for class y. across D, when p; itself identifies ye as the best match. Finally, these intermediate weights are normalized via
exp (w(,o/7)
(1))
Ey=1 exp (w',e/7)
The temperature controls the sharpness of the distribution. This normalization ensures weights sum to one for each class, preserving their probabilistic validity. By constructing wt in this way, integrate empirical class distributions into the reweighting scheme, ensuring that wf, reflects

---

Published as a conference paper at ICLR 2026
Table Accuracy (%) comparison hetween baselines and our method % various fine-g fine-grained classification
datasets using CLIP and DeCLIP backbones. Bold values indicate the highest accuracy, while underlined values
represent the second highest in each column. "Human Selection" uses handcrafted prompts recommended by
CLIP authors and introduces externl knowledge. Results are not directly comparable to automated methods.
Caltech101
Food101
Flower002
UCP101
ImageNec
Average
CLIP-VIT-BI6
MPE
92.50
46.88
51.86
21.49
$5.34
64.21
79.46
65.21
64.92
67.41
67.59
64.26
Majority Vote
93.10
46.75
$2.07
22.93
60
67.20
81,27
64,93
65.75
68,30
67.98
65,08
WPB
CARPRT (Ouns)
94.16
48.90
33.56
24.49
86.31
71.36
89.13
66.14
66.93
70.41
68.59
67.45
Human Selection
92.94
44,39
47.60
24.72
86.06
71.23
88.91
65.32
62.50
66.75
68.31
05.34
CLIP.ResNer50
Majority Vote
$6.79
42.14
28.86
16.29
76.00
60.06
77.29
56.01
50.40
60.87
59.24
56.72
WPR
86.65
40.89
30.65
16.11
76.15
58.82
78.43
56.02
59.71
61.53
59.78
56.79
CARPRT (Our))
88.46
41.31
36.84
16.88
76.88
68.56
85.69
56.44
61.28
63,66
$9.98
59.36
Homan Selection
40.32
29.36
17.28
75.31
66.14
85.77
55.61
58.52
61.46
59.71
$7.82
DeCLIP-NIT-BG22
MPE
94.04
41.63
28.05
7.10
71.71
77.26
76.75
52.22
62.08
57.87
67.01
57.84
Majority Vote
94.26
40.29
27.68
2.20
72.34
78.19
77.75
51.87
62.86
58.20
67.24
58,03
40.97
27.92
7.54
73.15
81.32
63.23
58.91
67.97
CARPRT (Ouns)
94.37
43.31
33.14
8.76
74.15
82.42
83.28
52.23
64.12
59.57
68.08
60.31
Human Selection
93.97
42.55
30.07
9.05
73.59
83:41
83.14
50.77
63.14
$8.70
67.85
06.66
both the relevance scores (Eq. 4) and the estimated class priors (Eq. 5), thus providing a principled inference time approach achieve class-aware prompt reweighting.
Inference with Estimated Weights. After calculating class-specific prompt weights wi, we perform zero-shot inference by aggregating prompt relevance scores for each class. Given a test image x² with embedding z! f(=") and prompt text embeddings z²,c ((pr()e)), we first compute St,ie sim (2,, =T))) following Eq. 9. We then define the class score and prediction as
((x") arg max cE(I....C))
sc(x"),
se(x")
w St,
(12)
This inference step only requires the similarity scores returned by the VLM (score-only black-box queries) and does not require access to model parameters, gradients, or internal embeddings.
Iterative Refinement. While the single-pass pipeline deseribed above forms the core of our approach, CARPRT can naturally be extended to refine both pseudo-labels and weights, by following the procedure iteratively: (i). Use current weight estimates combine predictions from all prompts into refined pseudo-labels; (ii). Update class-specific weights based on these refined pseudo-labels. Importantly, this refinement procedure is gradient-free and thus does not require access to ground tround-truth labels. This alternating refinement process allows CARPRT to sharpen its weight estimates as pseudo-label quality improves. Full details are in App. E.1.
(Optional):
EXPERIMENTS
We evaluate how CARPRT performs on zero-shot classification with ten fine-grained benchmarks, compared to existing prompt ensembling methods. Our investigation centers on three questions: (RQ1) Does class-awar prompt reweighting outperform class-agnostic ones; if so, does it generalize across different VL. architectures and backbones? (RQ2) What factors contribute to CARPRT's effectiveness? (RQ3) Can CARPRT's benefit extend beyond zero-shot classification?
5.1 EXPERIMENTAL SETUP
Dataset. We evaluate eleven classification benchmarks spanning diverse visual domains: Cal- tech101, DTD, EuroSAT, Aircraft, Food101, Flowers 102, Pets, Cars, Sun397, UCF101 and ImageNet (details in App. C. We follow the evaluation protocol established by Zhou et al. (2022b).
7

---

Published as a conference paper at ICLR 2026
Models and Prompts. We test CARPRT with three configurations: CLIP (Radford et al., 2021) with ViT-B/16 and ResNet50 backbones, and DeCLIP (Li et al., 2022) with the ViT-B/32, to validate CARPRT generalizes across both CNN-based (He et al., 2016) and transformer- based (Dosovitskiy et al., 2021) backhones, and different VLM architectures. For all experiments, we use the same fixed set of 247 prompt templates from Allingham et al. (2023) ensure fair comparisons.
Baselines. We compare CARPRT against three automated PE baselines: (1) MPE (Radford et al., 2021): Uniformly averages embeddings from all prompis. (2) Majority Vote (Allingham al.,
2023): Final prediction is based on the most frequent class predicted by individual prompts. (3) WPE (Allingham et al., 2023): Estimates a class-agnostic set of prompt weights from unlabeled test data. As an upper-bound reference. we also report "Human Selection" which uses subset of prompts manually filtered for each dataset by human experts. This helps to benchmark automated methods against careful prompt engineering. See App. C.2 for details.
Implementation. We follow the publicly available code of baselines, with two adjustments noted. We use a smaller batch size for weight estimation due to resource limitations, and we omit its original frequency normalization step, which requires the external LAION-400M dataset (Schuhmann et al., 2021), since this step is not the focus of this study (See App. G.6 for the analysis of the impact). Moreover, this omission ensures all methods align with our problem setting of using only unlabeled test data without external resources, for fair comparison. Details and code are in App. C.3.
5.2 RESULTS OF ZERO-SHOT CLASSIFICATION
Overall Comparison. Tab. shows that CARPRT consistently achieves the best accuracy across both fine-grained benchmarks and large-scale real-world datasets, such as ImageNet (with further evaluations its variants provided in App. G.3). Gains are pronounced datasets like Flower102 102 and Pets, highlighting the substantial impact of class-specific prompt relevance. Notably, CARPRT also surpasses Human Selection, where tesk-relevant prompts are manually filtered. This confirms that capturing class-specific weights can effectively compensate for irrelevant prompts in generic prompt pools and potentially outperform -ataset-specific manual prompt engineering.
Generalization and Robustness.
Across architectures. CARPRT's performance ben- efits are consistent across different VLM architec- tures and backbones. With Reet5,, de- spite its lower capacity than ViT-B/16, CARPRT
Accuracy Comparison: Class Aware vs. Uniform Weights
Closs:/waree
+1 +159%
0..99%
+7.52%
Accurncy (N)
800%
17.00%
12.38%
60%
+2.70%
still achieves clear and measurable gains. When ap- plied to DeCLIP-ViT-B/32, which adopts distinct pre- rre-training strategy, CARPRT likewise maintains its strong lead. Overall, performance across diverse model configurations suggests that CARPRT can effectively capture semantic relationships, rather than exploiting a particular setup. Moreover, be- training-free, black- man- ner, CARPRT is readily applicable whenever a VLM exposes only forward similarity queries.
11.50%
40%
12.51%
20%
0N
DTD
EurosAT
Aimaft
Calteshdel
Fondial
Flawerin2
Pata
Cara
SUNUAR
UCFIEL
Figure 3: CARPRT-Uniform.
Accuracy gains of CARPRT over
cause it operates
Under distribution shifts. We further evaluate robustness under distribution shifts on ImageNet and four variants: ImageNet-A. ImageNet-R, ImageNet-Sketch, and ImageNet-Ve V2. In this setting, prompt weights are estimated once using only unlabeled samples from the in-distribution ImageNet test set, and the same weights are directly transferred to all variants for evaluation, without access to their target distributions during estimation. As shown in Tab. 2, CARPRT consistently surpasses MPE and WPE across all variants. These results suggest that CARPRT's reweighting strategy generalizes well under distribution shifts. We hypothesize that this robustness stems from estimating prompt-cle class relevance primarily via image-ter text score statistics tied to class semantics, and that ImageNet's larger sample size yields more stable weight estimates that transfer effectively across distributions.
Overall, these results indicate that CARPRT generalizes across both model architectures and data distributions, making it a practical plug-in for zero-shot classification in training- free, score-only
8

---

Published as a conference paper at ICLR 2026
Table 2: Accuracy (%) under distribution shifts ImageNet and its variants using CLIP VIT-B/16. Prompt weights are estimated once on in-distribution ImageNet and directly transferred ImageNet-A, ImageNet-R, ImageNet-Sketch, and ImageNet-V2. V2. Bold values indicate the highest accuracy in each column.
Method
ImageNet
ImageNet-A
ImageNet-R
ImageNet-Sketch
ImageNet-V2
Average
MPR
67.59
49.35
77.33
46.92
61.37
60.51
Majority Vote
67.
49.47
77.54
47.18
61.55
60.74
WPE
68.28
50.34
77.34
47.50
61.96
61.08
CARPRT (Ours)
68.59
51.96
77.69
47.91
62.51
61.73
black-box settings, where only forward similarity queries are available and no access to model parameters, gradients, or internal embeddings is required.
Dataset-Specific Patterns. The extent of CARPRT's improvement varies by dataset, showing larger gains on datasets with well-separated semantic categories (e.g., Flowers Pets). On highly specialized domains like Aircraft, the gains are modest, likely due to (i) the quality of the initial pseudo-labels generated by base VLMs, which impact both WPE and CARPRT. (ii) the suitability of generic prompt pool for highly specialized visual distinctions. Nonetheless, CARPRT consistently improves performance, highlighting the broad value of class-specific weighting.
5.3 ABLATION STUDY AND HYPERPARAMETER ANALYSIS
Role of Class-specific Weights. To isolate the benefit of class-specificity, we compare CARPRT to "CARPRT-Uniform". This variant first computes CARPRT's class-specific weights, then av- erages them across classes to yield a global wh t E. Wr.c for each prompt p;- This vari- ant retains CARPRT's prompt scoring mechanism but discards class-level adaptation (it still dif- fers from WPE; see App. G.2). As Fig. 3 shows, CARPRT consistently outperforms CARPRT- Uniform, with an average gain of 2.39%. Considerable improvements datasets like Pets and Flowers affirm that tailoring prompt weights to individual classes is key to performance.
Accuracy Temperature
Temperature Sensitivity. CARPRT uses a temper- ature 7 (Eq. 11) adjust prompt weight distribu- tions. As shown in Fig. 4, t 1.0 balances relevance and diversity, emphasizing useful prompts while pre- serving ensemble variety for generalization. Lower 7 < 1.0 concentrates weights on dominant prompts but reduces diversity, whereas higher values flatten the distribution. Although finding a single best hyper- parameter for all zero-shot tasks is difficult, 7 = 1.0 is a stable choice across tasks, showing that calibrated reweighting helps without extensive per-task tuning. See App. G.2 for details.
Standard Deviation Average Accuracy
Accuracy (%)
67.5%
67.34%
67.0%
66.5%
Temperature
Figure 4: The variation of inference accuracy as the temperature 7 changes, using CLIP-ViT VLTP-VVT-B/16.
5.4 EXTENDED EVALUATIONS AND CLASS-SPECCS SPECIFIC WEIGHT VISUALIZATIONS
We explore CARPRT's versatility further with additional experiments (detailed in App. E,F.G).
Refined Pseudo-Labels and Weight Estimation. CARPRT's gains vary by dataset, partly due the quality of initial pseudo-labels from the base VI.M. Motivated by this observation, we further examine filtering low-confidence lseudo-labels Cconfidencee/entropy-maaee but observe only marginal and inconsistent gains, suggesting that explicit filtering is unnecessary in practice (App. G.I). Instead, iterative refinement yields steady improvements by progressively leveraging increasingly accurate class information (App. E.2).
to improve pseudo-label quality,
Does Prompt Quality Matter? While CARPRT designed for generic prompt pools, it could further benefit from higher-quality, potentially domain-specifis prompt templates. Preliminary tests with LLM-ggnnrated prompts showed improved CARPRT performance compared to using only dataset- agnostic templates from Allingham et al. (2023) (App. G.5), suggesting that CARPRT effectively

---

Published as a conference paper at ICLR 2026
Dalmatian
Pizza
example of
0.0130
a photo of D. type of fond.
0.0138
type of pel
0.0127
a pholo a nice 0
0.0098
photo of (). type of pat.
10000
0.0124
a rendering of the 0
Aagg D
0.
0.0092
a street sign with the number
0.0002
itap of the 0
0.0001
a photo took while visiting
0.0002
a 1 review of movie,
0.0001
aerial view of -00000
an outdoor number writlen sign
0.0000
0.00
0.01
0.00
0.01
Weight
Figure 5: Visualization of class-specific prompt weights on Caltech101. For dalmatian and pizza, CARPRT assigns high weights to class-relevant prompts while suppressing irrelevant ones.
leverages the information in any given prompt set. While it is difficult to evaluate the "prompt quality", we argue that investing in careful prompt engineering is likely to be beneficial.
CARPRT beyond Zero-shot Classification as General-Purpose Plug-In. We lastly show CARPRT's versatility as a component to enhance various VLM adaptation settings: (i) with test-time adaptation (Karmanov et al., 2024), CARPRT offers improved weight initialization (App. F.1); (ii) with image-feature focused zero-shot methods (Qian et al., 2024a), CARPRT enhances pseudo-labels for visual proxy learning (App. F.3); (iii) with soft prompt tuning (Lu et al., 2022), class aware reweighting of learned prompts can boost performance further (App. F.2); (iv) with LLM-empowered prompt augmentation (Shtedritski et al., 2023; Mirza et al., 2024), the utility of high-quality generated prompts still be improved via class-aware reweighting (App. F.4). All these results confirm CARPRT'S flexibility as a general-purpose plug-in for broader VLM adaptation scenarios.
Visualization of Class-Specific Prompt Weights. To provide qualitative insight into CARPRT's mechanism, we visualize class-specific prompt weights on Caltech101. Fig. 5 shows weights estimated for two representative classes, dalmatian and pizza. For dalmatian, CARPRT assigns higher weights to semantically relevant prompts (e.g., example, pet, photo) while suppressing mismatched ones (e.g., aerial, visiting, number). Similarly, for pizza, food-related prompts (c.g., food, photo, rendering) are prioritized, whereas unrelated terms (e.g., sign, movie, itap) are dewn-weighted. Overall, these visualizations corroborate our quantitative results and illustrate that CARPRT learns class-dependent prompt preferences; additional examples are provided in App. J.
6 DISCUSSION AND FUTURE OUTLOOK
Broader Related Works. The performance of VLM adaptation in downstream classification tasks relevant to the text prompt, motivating research on improving prompt effectiveness in different directions. Prompt tuning (Zhou et al., 2022b; Khattak et al., 2023a) optimizes task-specific soft prompts through training, but departing from zero-shot settings. Unsupervised transfer learning methods (Qian et al., 2024a) aim to bridge domain gaps between visual and textual embeddings without labels; they do not focus on combining multiple prompts. Augmentation-baus instead relies on large-scale data augmentation, such using LLMs to generate task-specific prompts or building partial image views, then assigning weights to augmented prompts or views (Zhu et al., 2024; Li et al., 2024); while powerful, they necessitate the availability of external computing resources. In contrast, CARPRT explicitly addresses the setting of prompt ensembling with fixed, potentially iask-irrelevant prompt pool. is entirely training-fiee frre,, relies on neither label supervision nor LLM- generated prompts, and focuses reweighting existing prompts to capture class-specific relevance. This makes CARPRT orthogonal to the above directions, while also complementary to them, offering unique perspective on VLM adaptation. We discuss these related works in detail in App. A.
weighting
Summary. This study focused on prompt ensembling and confirmed that class-aware prompt reweighting is not only beneficial but essential for improving the efficacy of VLMs across a variety of downstream classification tasks. By moving beyond uniform weighting, we showed that adapting weights to better reflect the class-specifi characteristics leads to measurable gains in performance. We hope this study encourages further exploration of integrating awareness with other VLM adaptation techniques enhance across wider range of applications.
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
All authors have read and agree to abide by the ICLR Code of Ethics and Code of Conduct. This work does not involve sensitive personal data experiments with human subjects. We have taken care to ensure that the datasets used are publicly available and widely adopted in prior research, and that the proposed method does not raise foreseeable ethical concerns. All claims and findings are reported honestly and transparently.
REPRODUCIBILITY STATEMENT
We are committed to ensuring the reproducibility of our results. Detailed descriptions of the setup, training and evaluation protocols, implementation details, and hyperparameter settings are provided in Sec. and App. C.3. All experiments are conducted on publicly available datasets, which are listed in App. C.I. An anonymous code link is supplied to facilitate replication.
AUTHOR CONTRIBUTIONS
All authors contributed the research and the writing of this paper.
ACKNOWLEDGMENTS
We thank the anonymous reviewers and area chairs for their constructive feedback. MS was sup- ported by JST ASPIRE Grant Number JPMIAP25B1. Jianzhong Qi is supported in part by the Australian Research Council (ARC) via Discovery Project 0P240101000 and Future Fellowship FT240100170. Feng Liu supported by the Australian Research Council (ARC) with the grant number DE240101082 089, LP240100101, DP230101544 and the NSF&CSIRO Responsible Al program with grant number 2303037.
REFERENCES
James Urquhart Allingham, Jie Ren, Michael W Dusenberr, Xiuye Gu, Yin Cui, Dustin Tran, Jeremiah Zhe Liu, and Balaji Lakshminarayanan. A simple zero-shot prompt weighting technique improve prompt ensembling in text-image models. In ICMI,
Anthropic. Claude 3.5 sonnet model card addendum. com/ fed9cc1 93a1 2111188112222263888880022222222. 3 3_Addendum, pdf, 2024.
cwwpwwwwwwwcnnncwwc///o//ood/ anthropic.
Lukas Bossard, Matthieu Guillaumin, and Lue Van Gool. Food-101 -mining discriminative compo- nents with random forests. In ECCV, 2014.
Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. Learning imbalanced datasets with label-distribution-aaaa margin loss. In NeurIPS, 2019,
Dian Chen, Dequan Wang, Trevor Darrell, and Sayna Ebrahimi. Contrastive test-time adaptation. In CVPR, 2022.
Mircea Cimpoi. Subhransu Maji, lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. De- scribing textures in the wild. In CVPR, 2014.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image worth 16x words: Transformers for image recognition at scale. In ICLR, 2021.
Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In CVPR, 2004.
Chun-Mei Feng. Kai Yu, Yong Liu, Salman Khan, and Wangmeng Zuo. Diverse data augmentation with diffusions for effective test-time prompt tuning. In CVPR, 2023.
11

---

Published as a conference paper at ICLR 2026
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario March, and Victor Lempitsky. Domain-adversarial training of neural networks. Journal of machine learning research, 2016.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2019.
Dan Hendryeks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: A critical analysis of out-of-distribution generalization. In ICCV, 2021a.
Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. In CVPR, 2021b.
Adilbek Karmanov, Dayan Guan, Shijian Lu, Abdulmotaleb El Saddik, and Eric Xing. Efficient test-time adaptation of vision-language models. In CVPR, 2024.
Muhammad Uzair Khattak, Hanoona Rasheed, Muhammad Maaz, Salman Khan, and Fahad Shahbaz Khan. Maple: Multi-modal prompt learning. In CVPR, 2023a.
Muhammad Uzair Khattak, Syed Talal Wasim, Muzammal Naseer, Salman Khan, Ming Hsuan Yang, and Fahad Shahbaz Khan. Self-regulating prompts: Foundational model adaptation without forgetting. In ICCV, 2023b.
Soomro Khurram. Ucf101: A dataset 101 human actions classes from videos the wild. arXiv preprint arXiv: 1212.0402, 2012.
Jonathan Krause, Michael Stark, Jia Deng, and Fei-Fei. 3d object representations for fine-grained categorization. In ICCV, 2013.
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. 2009.
Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. Technical Report, 2015.
Yann LeCun, Sumit Chopra, Raia Hadsell, M Ranzato, Fujie Huang, al. A tutorial on energy- based learning. Predicting structured data, 100), 2006.
Jinhao Li, Haopeng Li. Sarah Monazam Erfani, Lei Feng. James Bailey, and Feng Liu. Visual- text cross alignment: Refining the similarity score in vision- laigunla models. In International Conference Machine Learning, 2024.
Yangguang Li, Feng Liang, Lichen Zhao, Yufeng Cui, Wanli Ouyang. Jing Shao, Fengwei Yu, and Junjie Yan. Supervision exists everywhere: A data efficient contrastive language-image pre-training paradigm. In ICLR, 2022.
Yanghao Li, Naiyan Wang, Jianping Shi, Jiaying Liu, and Xiaodi Hou. Revisiting batch normalization for practical domain adaptation. arXiv preprint arXiv: 04779, 2016.
Mingsheng Long, Yue Cao, Jianmin Wang, and Michael Jordan. Learning transferable features with deep adaptation networks. In ICML, 2015.
Yuning Lu, Jianzhuang Liu, Yonggang Zhang, Yajing Liu, and Xinmei Tian. Prompt distribution learning. CVPR, 2022.
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained visual classification of aircraft. In arXiv preprint arXiv:1306.575/ 2013.
Aditya Menon and Carl Vondrick. Visual classification via description from large language models. In ICLR, 2023a.
12

---

Published as a conference paper at ICLR 2026
Sachit Menon and Carl Vondrick. Visual classification via description from large language models. In ICLR, 2023b.
Microsoft. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404. 14219, 2024.
M Jehanzeb Mirza, Leonid Karlinsky, Wei Lin, Sivan Doveh, Jakub Micorek, Mateusz Kozinski, Hilde Kuehne, and Horst Possegger. Meta-prompting for automating zero-shot visual recognition with 1lms. In ECCV, 2024.
Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In Indian conference on computer vision, graphics & image processing, 2008.
Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. CVPR, 2012.
Qi Qian, Yuanhong Xu, and Juhua Hu. Intra-modal proxy learning for zero-shot visual categorization with clip. In NeurlPS, 2024a.
Qiyuan Qian, Yifan Xu, and Jie Hu. Intra-modal proxy learning for zero-shot visual categorization with clip. In NeurIPS, 2024b.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, 2021.
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In ICML, 2019.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. IJCV, 2015.
Christoph
Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarezyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400ms Open dataset of clip-fittred 400 million image-text pairs. arXiv preprint 1rXiv:2/11.02/1 14, 2021.
Aleksandar Shtedritski, Christian Rupprecht, and Andrea Vedaldi. What does clip know about a red circle? visual prompt engineering for vlms. In ICCV, 2023.
Manli Shu, Weili Nie, De-An Huang, Zhiding Yu, Tom Goldstein, Anima Anandkumar, and Chaowei Xiao. Test-time prompt tuning for zero-shot generalization in vision-language models. In NeurIPS, 2022.
Rui Shu, Hung H Bui, Hirokazu Narui, and Stefano Ermon. A dirt-t approach to unsupervised domain adaptation. In ICLR, 2018.
Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. Tent: Fully test-time adaptation by entropy minimization. In ICLR, 2021.
Haohan Wang. Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global representations by penalizing local predictive power. NeurlPS, 2019.
Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In CVPR, 2010.
Marvin Zhang, Sergey Levine, and Chelsea Finn. Memo: Test time robustness via adaptation and augmentation. In NeurIPS, 2022.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. In CVPR, 2022a.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision- language models. In IJCV, 2022b.
Yuhan Zhu, Yuyang Ji, Zhiyu Zhao, Gangshan Wu, and Limin Wang. Awt: Transferring vision- language models via augmentation, weighting, and transportation. In NeurlPS, 2024.
13

---

Published as a conference paper at ICLR 2026
A DETAILED DISCUSSION ON RELATED WORKS
Prompt Tuning Methods. Prompt tuning adapts pre-trained model by introducing learnable embeddings (prompt tokens) at the imput stage. These tokens can be instantiated as textual prompts or visual prompts, enabling task-specific adaptation through the model's 's input interface. CoOp first applied prompt tuning to CLIP by optimizing learnable prompts in the text branch for few-shot recognition (Zhou et al., 2022b). To address CoOp's limited generalization, CoCoOp conditionally generates prompts from visual features (Zhou et al., 2022a). MaPL.e further extends prompt tuning to both the vision and text branches improve transferability (Khattak et al., 2023a). Building on MaPLe, PromptSRC enhances prompt learning by leveraging descriptive text generated by large language models (LLMss), e.g., GPT-4 (Khattak al., 2023b). However, these tuning-basc based approaches require optimizing learnable variables (and typically labeled downstream data, even in the few-shot regime), which falls outside our strict zero-shot, training-fret ffreng-freeeee setting. We therefore do not include them as baselines for CARPRT.
Unsupervised Transfer Methods for VLMs. Unsupervised transfer for VLMs aims to adapt pre-trained models (e.g., CLIP) to downstream tasks without ground-truth labels. Existing methods can be broadly grouped into two directions. The first direction, exemplified by Weighted Prompt Ensembling (WPE) (Allingham et al., 2023), focuses on automatically reweighting prompts from a given template pool. By assigning datasct-level importance weights to different templates, such methods help identify which prompts are more compatible with a downstream task and can offer a degree of interpretability (Allingham et al., 2023). The second direction relies on transductive learning using image features to construct classifiers, such as InMaP (Qian et al., 2024a). These methods often achieve stronger accuracy by exploiting structure in the unlabeled images, but are typically less interpretable since decisions are driven primarily by image-feature learning rather than explicit prompt contributions. Our work follows the first direction by retaining the prompt-based formulation while improving accuracy via class-speciff reweighting, Moreover, the -auuds produced by CARPRT can also benefit image-centric transductive pipelines; see App. F.3 for detailed results.
View-aware Weighting Approaches. View-aware methods improve zero-shot transfer by aggregat- ing evidence from multiple augmented visual or textual views and weighting them by confidence or alignment. WCA performs local visual prompting by pooling similarities between cropped regions and fine-grained textual descriptions using weighted aggregation (Li et al., 2024). AWT combines diverse image augmentations with LLM-generated prompts and computes weights across views, followed by optimal transport for cross modal alignment (Zhu et al., 2024). While effective, these approaches may rely on external resources (e.g.. LLMs) and/or costly augmentations at inference. In contrast, CARPRT derives class-specific weights directly from image- text similarity scores under fixed prompt pool, without external models or additional augmentations, yielding substantially lower inference overhead while remaining complementary view-aware techniques.
Test-time Adaptation. Test-time adaptation (TTA) aims to adapt models to unlabeled test data (Ganin et al., 2016; Long et al., 2015; Zhang et al., 2022). Broadly, TTA methods can be categorized into training-based and training- free approaches. Training based methods update model parameters or prompts using test-time objectives (c.g., entropy minimization), as in TENT (Wang et al., 2021), or incorporate additional regularization to preserve alignment, as in CoTTA (Chen et al., 2022). For VLMs. prompt-centric TTA methods such as TPT fine-tune learnable prompt at test time (Shu et al., 2022), and DiffTPT further leverages diffusion models to enrich test-time augmentations (Feng et al., 2023). Training-free TTA methods instead rely on adjusting normalization statistics or test-time augmentation without updating model parameters (I.i et al., 2016; Karmanov al.. 2024). Since CARPRT is also training-free free and uses only unlabeled test data, we study its relationship with training-fre free TTA in App. F.I.
B DIFFERENT PROBLEM SETUP FOR VLMS ADAPTATION
Prompt ensembling, as formalized in Problem 1, targets a strictly zero-shot inference setting where the only available resources are fixed prompt template set P and unlabeled test set ID. No
14

---

Published as a conference paper at ICLR 2026
learnable parameters, task-specific fine-tuning, external supervision permitted. This setting is entirely inference-time, model-free, and tuning-free.
In contrast, other VLM adaptation paradigms operate under more relaxed assumptions, either by enabling trainable components, leveraging supervision, or utilizing additional knowledge sources. We outline the key differences follows:
Prompt Tuning relaxes the "no training" constraint by introducing learnable prompt tokens, typically optimized using downstream supervision. Formally, the prompt becomes a learnable function pe (ye) with parameters 0, where 0 is optimized on labeled data {(xz, My)). CoOp Zhou et al. (2022b) learns global soft prompt, while CoCoOp Zhou et al. (2022a) further conditions image embeddings f fz) to improve generalization. These methods trade interpretability for adaptability and require supervision at training time.
LLM-Geneaated Descriptions expand the prompt space P using external generative models. Rather than fixing P priori, large language model 9LLM generates class descriptions (r( = 9LLM (ye) that are often more expressive and context-award aware Menon & Vondrick (2023a). While such prompts can improve alignment, this introduces non-negligible computational overhead and reduces repro- ducibility, especially when prompts are generated on-the-fly.
Image-Centric Adaptation bypasses prompt usage entirely by constructing classifiers purely from image features. Methods like InMaP Qian et al. (2024b) rely on clustering method to construct label assignment function h X + y without accessing any textual information. These methods often outperform prompt-based approaches in raw accuracy but offer limited interpretability and are incompatible with text-conditioned decision-making
CARPRT operates strictly within the constraints of Problem 1. Unlike the above paradigms, it does not rely any learnable components, L.LM-generate text, image-only inference. Instead, it focuses on exploiting the class-specific aligmment between P and y in training-fiee interpretable, and modular fashion. As demonstrated in App. F, its output (pseudo-labels and weights) can directly benefit and enhance downstream methods in both prompt tuning and image-centric learning pipelines.
C DATASETS, BASELINE METHODS, AND IMPLEMENTATION
C.I DATASETS
Fine-grained Datasets. Following Zhou et al. (2022b), we evaluate our method on 10 fine-grained classification benchmarks: Caltech10) (Fei-Fei et al., 2004) (101 object categories); DTT (Cimpoi et al., 2014) (47 describable texture attributes such as "bumpy" and "scaly"); EuroSAT (Helber et al., 2019) (10 land-use classes from satellite imagery, e.g., residential, forest, and river); FGVC- Aircraft (Maji et al., 2013) (100 aircraft variants); Food101 (Bossard et al., 2014) (101 food cate- gories); Flowers 102 (Nilsback & Zisserman, 2008) (102 flower species); Oxford Pets (Parkhi et al., 2012) (37 pet breeds); Cars (Krause et al., 2013) (196 car models); SUN397 (Xiao et al., 2010) (397 scene categories); and UCF101 (Khurram, 2012) (101 human action classes).
ImageNet and its Variants. Following Allingham et al. (2023), we additionally evaluate on ImageNet and its commonly used robustness variants: ImageNet (Russakovsky et al., 2015) (1,000 classes): Imag-eea (Le & Yang, 2015) (a 200-class subset); ImageNet-A (Hendrycks et al., 2021b) (naturally adversarial examples); ImageNet-R (Hendrycks al., 2021a) (renditions such as paintings and cartoons); ImageNet- Sketch ( (Wang et al., 2019) (sketch-style images); and ImageNet- V2 (Recht al., 2019) (a re-collected test set).
C.2 BASELINES
To evaluate our method under a consistent setting, we compare CARPRT with representative baselines that operate within the same zero-shot classification protocol and fixed prompt set (Problem l).
Mean Prompt Ensembling (MPE). MPE averages predictions across prompts with uniform weights. For each class, the model encodes all prompted class texts and averages them form class prototype.
15

---

Published as a conference paper at ICLR 2026
Algorithm I Class-Aware Prompt Reweighting (CARPRT)
Input: A pre-trained VLM model (x, pi (Ve)) that returns a cosine similarity score', prompt set IP, an unlabeled dataset ID, candidate label space V, the temperature parameter 7 and the normalization scale X.
Construct prompted- texts Pi(y.). Vp, E Py E y;: 2: Obtain the relevance score set $ = (8p by querying the scorer: $5,ie =
(())):(())));
3: Obtain the pscudo-label set: Y (0,,))=1..es
4:
Derive the weight matrix W by Eq. 10 and Eq. 11;
Output: class- aware prompt weight matrix W
At test time, an image classified by cosine similarity to these averaged embeddings. This baseline assumes prompts contribute equally, regardless of class semantics.
Majority Vote. Majority Vote treats each prompt as an independent voter. For each prompt, the model predicts the most similar class for an image, and the final prediction is determined by majority voting across prompts. This method ignores prediction confidence and assumes all prompt votes earry equal importance.
Weighted Prompt Ensembling (WPE) (Allingham et al., 2023). WPE estimates a global prompt- weight vector from the unlabeled test set and aggregates prompt-conditioned class embeddings accordingly (e.g., via an unsupervised objective such as entropy minimization). However, WPE uses single weight vector shared across all classes and thus cannot capture class-specific variations in prompt relevance.
C.3 DETAILS REGARDING EXPERIMENTS
Implementation Details. We implement all methods using PyTorrch 1.7.1 and Python 3.7.6, and run experiments on a single NVIDIA A100 Tensor Core GPU. We use OpenAl CLIP (Radford et al., 2021) and DeCLIP (Li et al., 2022) as the underlying VLM backbones. Our code is available for reproducibility.
at https:://// /ttthbbttttttgggt//g com/ tmlr
Hyperparameter Settings. Unless otherwise specified, we set 1.0 for fine-grained datasets and t 1.5 for ImageNet (Russakovsky et al., 2015) and its variants, and use batch size of 512 for all experiments.
D MORE DETAILS OF CARPRT
D.1 CARPRT ALGORITHM
We summarize the overall procedure of our proposed Class-Aware Prompt Reweighting (CARPRT) in Algorithm 1. As shown in the algorithm, CARPRT begins by encoding both image and text embeddings using a pre-trainee trained CLIP- model. It then computes the relevance score between image features and prompt-cone coompt-coniiiioned text features, followed by pseudo-label assignment. Finally, class-aware weight matrix is derived based on the computed scores, enabling the construction of refined prompt weight matrix that improves zero-shot classification performance.
D.2 CONNECTING CARPRT FORMULATION WITH THE PROBABILISTIC FRAMEWORK
We now detail the correspondence between the CARPRT formulation (Section 4) and the probabilistic framework established in Section 3.
We write CARPRT using black-boo box scoring function & since the method only requires access to similarity scores (or logits) sj,ie from forward pass. Explicit access to the image/text encoders (and embeddings) is only needed if one wishes to optimize altemattve objectives beyond such score queries (e.g., non-cosine similarity losses representation-leve regularizers).
16

---

Published as a conference paper at ICLR 2026
Algorithm 2 Iterative Class-Aware Prompt Reweighting (iCARPRT)
Input: A pre-traines VLM model (2x, (( pi(ee) that returns a cosine similarity score, prompt set P, an unlabeled dataset TD, a candidate label space ), the maximum iterations Tmax, the temperature parameter 7 and the normalization scale X.
1: Generate prompted-class texts (yo),, Vp E P. Vy. 3
2: Obtain the relevance score set $ [$g.4.e] m.m.C j=,,,,,, by querying the scorer: 8j,4, (b (v()))):
3: Initialize the class-aware weights wi,e uniforrmyy;
for = to Tinx do
4: Obtain the pseudo-labee set: Y f using Eq. 13;
Derive the weight matrix W Eq. 14 and Eq. 11;
end
Output: a class-awar aware prompt weight matrix W W Tmaxx
Concretely, the practical implementation Eq. 9- Eq. 11 align with Eq. 3- Eq. 7 in the following manner.
Score Calculation. Eq. 9 implements the likelihood term Pr(æ,/8e |Be, W,P) from Eq. 7 by defining
exp(aj.i.e/A) 8j,i yey exp( This formulation aligns with the EBM in Eq. 7 by using cosine similarity aj,i. as the negative energy term and normalizing through softmax to obtain proper probabilities.
Weight Calculation. Eq. 10- Eq. 11 correspond to estimating Pr(W P. D) from Eq. 4 through two- step process. Eq. 10 first obtains the pseudo-labels for samples as the empirical estimates Pr(ye W, IP) (i.e., Eq. 5). It then estimates intermediate weights by aggregating scores across pseudo-labeled samples by multiplying the scores Pr(ar, [Be, W, IP) (i.e., 8já.e) with Pr( W, IP). Eq. 11 applies softmax to ensure the resulting weights form a valid probability distribution over prompts for each class, which satisfies the simplex constraint implied by our probabilistic framework.
E DETAILS OF CARPRT WITH ITERATIVE REFINEMENT (ICARPRT)
E. METHODS
In this section, we introduce iterative fass-aware prompt reweighting (iCARPRT). Unlike the single- pass approach described in the main text, iCARPRT refines pseudo-labels and class-aware prompt weights through multiple rounds of alternating updates. The procedure consists of the following two main steps: pseudo-label generation and class-aware weight estimation.
In lseudo-label generation, the pseudo-label Ø3 of the image æ, is computed using the prompt weights estimated in the previous iteration, W as:
3 arg max
w/o
$3.80c
(13)
B. B.E
where $y the relevance score computed in Eq. Once the pseudo-labels ú are updated, the intermediate weights w/e are estimated by:
En 1 $
who
(14)
Ej 1 18
where =g is indicator function that is 1 if ÚJ Yes and 0 otherwise. Then the final weights W,,, are computed from the intermediate weights w/,- using Eq. 11.
These two steps repeat until predefined maximum number of iterations is reached. By alternating between pseudo-label prediction and weight ce-smmmatiin, iCARPRT creates a reinforcing cycle that continuously improves both the pseudo-labels and the class-aware prompt weights.
17

---

Published as a conference paper at ICLR 2026
Table 3: Accuracy (%) comparison between CARPRT and iCARPRT various fine-grained clas-
sification datasets using CLIP-ViT-B/ and CLIP-ResNet50 backbones. Bold values indicate the highest accuracy.
Caltech101
DTD
EuroSAT
Aireraft
Food101
Flower102
Pets
Cars
SUN397
UCF101
Average
VTT--V
CARPRT
94.16
48.90
55,56
24.49
86.31
71.36
89.13
66.14
66.93
70.41
67.34
ICARPRT
94.27
48.14
54.79
23.71
87.25
72.01
89.64
67.19
67.28
70.53
67.48
CLIP-ResNet50
CARPRT
88.46
41.31
36.84
16.88
76.88
65.56
85.69
56.44
61.28
63.66
59.30
iCARPRT
35.65
15.42
77.96
86.09
57.28
59.53
Table Accuracy (%) comparison between baselines and CARPRT when combined with TDA, using CLIP-ViT-B/16 and
backbones. Bold values represent the highest accuracy in
each column.
Caleeh101
DTD
EuroSAT
Aircraft
Food101
Flower102
Pets
Cars
SUN397
UCFI01
Average
CLJP.VTT-B/1
WPE
93.49
47.02
62.48
23.09
86.21
68. 10.10
84.12
67.23
66.98
71.23
67.00
CARPRT (Ours)
94,62
48,52
63.95
24.05
86.50
70:36
84.50
67.83
68.06
71.85
68.02
Selection (TDA)
23.91
CLIPP ResNee0
MPΕ
92.03
41.77
54.56
19.77
83.41
62.50
80.65
63.55
64.14
68.80
63.12
WPE
CARPRT (Ours)
91.75
42,71
$7.68
19.98
83.61
62.66
81.38
68.98
68.98
68.63
63.76
Human Selection (TDA)
91.42
41,00
56.97
20.55
83.34
62.75
83.62
64.14
65,86
68.52
63.82
E.2 EXPERIMENTAL RESULTS
We evaluate the performance of iCARPRT against the single-pass version, CARPRT. As shown in Tab. 3, the results demonstrate that iCARPRT achieves improvements in mean accuracy across different backbones. This suggests that the iterative refinement process effectively enhances class- aware prompt weighting by progressively improving pseudo-label quality and weight estimation.
Quality of Pseudo Labels Matters. In datasets such as EuroSAT and Aircraft, iCARPRT does not outperform CARPRT. A possible reason is the relatively low initial pscudo-label accuracy in these datasets. Since iCARPRT updates prompt weights based on pseudo-labels in each iteration, a poor starting point may lead to reinforcement of incorrect labels rather than improvement. In such cases, the iterative updates fail enhance pseudo-label quality, limiting the effectiveness of the approach.
F COMBINING CARPRT WITH OTHER VISION-LANGUAGE METHODS
While CARPRT focuses strict zero-shot image classification with fixed set of handcrafted prompts and unlabeled data (Problem 1), it inherently modular and can be integrated into a wide range of existing vision-language pipelines. Although direct comparison is not meaningful due to differing problem assumptions, we show that CARPRT can function as a complementary component rather than a competing method.
Specifically, we conduct case studies three representative scenarios. We first combine CARPRT with test-time adaptation method, then apply it to augment soft prompt tuning, and finally integrate it with a recent zero-shot method that leverages LLM-generated prompts. Details and results for each case are presented in the following subsections.
F.1 COMBINING CARPRT WITH TEST-TIME ADAPTATION METHOD
CARPRT can be integrated with TDA, a state-of-the-art training- free test-time adaptation (TTA) method for CLIP that enables efficient adaptation without backpropagation (Karmanov et al., 2024).
Our approach is not in conflict with TDA but is orthogonal to it. While TDA uses a human- selected prompt pool for each task, method can serve complementary module that replaces
18

---

Published as a conference paper at ICLR 2026
this human selection pool, providing alternative way of selecting prompts without requiring human intervention. This allows our method to work alongside TDA, enhancing the adaptability of lision-languuge models in more automated manner. We conduct the experiment to compare the performance of our method with several baselines, including the human-selected prompts, the equal weight prompt selection, WPE, all combined with the TDA method. The results evaluated using both CLIP-ViT-B/16 and CLIP-ResNete backbones across ten fine-grained datasets, as shown in Tab. 4.
From the result, we can observe that our method outperforms the other baselines in several datasets, achieving the highest average accuracy of 67.96% for L and 63.76% for CLIP-ResNet50. Specifically, for datasets like EuroSAT, Food101, and Flower102, our method shows significant improvements over the human-selected and WPE baselines. These improvements demonstrate that our approach effectively enhances the performance of TTA methods, by offering more efficient prompt selection strategy. However, there are cases where falls short compared to human-selected prompts. This may be caused by the limited diversity and smaller size of the template pool, where automatic reweighting methods may not perform as well as direct human selection. However, the automated approach significantly reduces the human labor cost. This experiment demonstrates the promising future of our method- not only in prompt reweighting but also as a technique that can be integrated into other vision-language model (VLM) transfer learning approaches. The ability to automatically adjust prompts a computationally efficient manner paves the way for broader applications and adaptability various VLM-based tasks.
Posterior Update with TTA. When prompt weights can be updated continuously, such as in TTA settings, different priors (e.g., uniform, global Dirichlet, or class-specific Dirichlet) define initial beliefs about weight distributions before observing test data. In the TTA scenario, test data arrives as }. Based on Eq. 4, we have a general form of posterior
a stream: (o) ((),, (((+1),... ...)),
p( W (+), P) x p(æ W, W,P)p(W[FF)
where PWIIP) is the prior, p(x WPP) is the likelihood from test data, and p (W) P) is the posterior that guides weight updates sample-by-sampe The posterior updating process follows:
For first test sample (%):
Prior: :p( (PPP)
Likelihood p(x (0) W P))
Posterior p( ( (9) P) p(x (") W. W, (WW(P)
Then, as we observe the second test sample (1), we have
Prior: p( [[z IP) (previous posterior)
Likelihood p(z(1) (1) W.P)
Posterior p( W (zx (1) PP) α p(x:() (///////// W, P)p(P) (1) ().P))
This leads to the sequential update scheme, formulated as
p ( ae (0)
(t), P) x ((x)() W, PWp((WW
(t
Thus, in TTA settings, these priors can be (1) initialized based on initial test samples; and (2) updated sequentially as new test samples arrive.
More specifically, choosing different prior distributions would lead to different updating computations. Uniform Prior. Recall the uniform prior is defined as
if E W otherwise
PW
IP)
W 0
By taking log to both LHS and RHS. we will have
- log [W]
W E W otherwise
log Dog((W]p)
(PP)
00
19

---

Published as a conference paper at ICLR 2026
which then leads to the log posterior to be expressed as
log W (1) (P) .P) x log [ + log
Σ
p(2
Yer
W. W.P)
Y.EY
1 Ay==oo
lolog [W] + log
Σ M.Ey
exp
(ws,c2i,c)
Eg Loyi=ve
Global Dirichlet Prior. The global Dirichlet prior treats all weights across classes as a single vector:
p(W (P)
where vec(W) RDC is the vectorization of weight matrix W (here we denote C = | as the cardinality of label space) Similarly, we will have the log prior and posterior as
log p( W PP) log Dir(vec( W an, anc)
cC Σ
nC
nC Σ
= log ((o))
log [(ax) +
Σ (op
log wk
(00
oKk)
k=1
k=1
k=1
nC
C
= log l'
ax)
log (((((c)))tt))
+
ΣV
(o 1Jce-Innt+ - 1) log wio
k=1
c=1
1=1
=1
and
logp(W (" x log p( (WPP) P) + log (1) |W,)) P) (og
nC
C
logg((av)
log ((x)) - Σ Σ
(a
1)nti 1) log wi.
k=
c=1 i=1
+ log Σ
p(rlye. W. P)p(y W.P)
KcE
n
log [(an)
Σ log [(az) ΣΣ6 (a(c-1)n+t 1) log wi.
+ log Σ exp
(wiezi(e)
18
Eg 1 A
Class-specific Dirichlet Prior. We again start from the prior definition
II
then turn into the log prior and posterior
log Papp(WW WIP)
log Dir(DirWWWW.) (Ka,1, (i. C...)
log [(a Σ log (acs) Σ (ae, 1) log wic
(ac. = Σ acs)
20

---

Published as a conference paper at ICLR 2026
and log posterior
"
log pl W
x
P)
log (( Σ
log ((ass +
Σ
(ac 1) log wi.
log
p(a ye. W.P)p(ue W.
NcEY
log ((a,o)
Σ log (( +
Σ
(oot - 1) log wi.
1=1
10
log Σ
Σ (wi.c2TTe) b,e)
exp
Σj 18
MeEy
However, since Dirichlet priors would introduce additional steps (e.g., estimating concentration parameters a),, in our preliminary investigation, we used uniform prior to keep simplicity. Despite this simplest setup, our CARPRT prompt reweighting strategy effectively facilitated TTA methods. We leave more systematic explorations of alternative priors (e.g., Dirichlet) into future work.
F.2 COMBINING CARPRT WITH SOFT PROMPT TUNING
Seft prompt tuning has recently become powerful technique for adapting CLIP and other pre-trained vision- Lanou-la models to downstream tasks. By learning optimal prompts that guide the model's understanding of new data, prompt tuming has shown remarkable effectiveness (Zhou et al., 2022b;a; Khattak et al., 2023b). ProDA optimizes prompt distributions to improve few-shot performance by training set of learnable invisible prompt embeddings. While CARPRT is primarily designed to reweight visible prompt templates, our approach is not restricted to visible prompts. In this section, also apply class-aware reweighting to the invisible prompts trained by ProDA, making our method capable of enbancing performance in various prompt tuning scenarios.
Our CARPRT method could enhance the ProDA framework by introducing class-aware reweighting technique that adjusts the influence of each prompt based on the underlying class structure. Specif- ically, before each iteration of ProDA's prompt distribution learning, we use CARPRT update the weights, which then guide the model's logit outputs for training the prompts. As the problem setting transitions from zero-shot to few-shot, our approach adapts by refining the weight estimation. Specifically, we use ground truth labels instead of the pseudo-labels for weight estimation, as shown in the following replacement for Eq. 10:
E Sy..c y
w/,0
(15)
m 2j=1 1y5=
where yy is the ground truth label of the sample j. The results shown in Table F.2 demonstrate that our method provides notable improvements in most datasets, highlighting the effectiveness of our class-aware prompt reweighting mechanism.
F.3 COMBINING CARPRT WITH MODERN ZERO-SHOT METHODS
Recent pero approaches often rely on large language models (LLMs) generate class descrip- tions or prompts. While these methods have shown strong performance, they typically introduce external information and lack mechanisms to calibrate prompt relevance across classes. CARPRT can be applied top of such methods to reweight their prompt pools in class-aware manner, enhancing prediction quality without modifying the model or relying on additional supervision.
Beyond prompt-based methods, CARPRT is also compatible with image-centric approaches that construct classifiers directly from visual features, such as InMaP (Qian et al., 2024a). These two method provides
strategies complementary: while InMaP builds vision proxy via clustering, high-quality aseudo-labels that can guide its optimization. As shown in Tab. 6, integrating CARPRT with InMaP consistently improves performance. In particular, refining pseudo-labels using Sinkhorn
21

---

Published as a conference paper at ICLR 2026
Table 5: Accuracy (%) comparison between method and the prompt tuning baseline fine- grained datasets using the CLIP-ViT-B/16 backbone. Bold values represent the highest accuracy in each row.
ProDA
ProDA CARPRT
10
91.3
95.4
DTD
70.1
69.6
EuroSAT
84.3
83.4
Aircraft
36.6
36.9
Food101
82.4
88.1
Flower102
95.5
95.6
Pets
90.0
93.7
Cars
75.5
78.6
Average
78.2
80.2
distance leads to further gains, validating that better pseudo-labels directly reduce the theoretical gap between recovered and optimal vision proxies. These results highlight that CARPRT not only improves zero-shot inference on own, also serves as a valuable component within broader vision-language learning frameworks.
Table 6: Accuracy (%) comparison between our method and the baseline on ImageNet using the CLIP-ViT-B/16 and CL NL.P-ReeNet5Ohackhonee Bold values represent the highest accuracy in each row.
InMaP
InMaP CARPRT
CLIP-ViT-B/LE
w/o Skinhorn
70.14
71.09
Skinhorn
72.55
72.57
CLIP-ResNet50
w/o Skinhorn
60.83
60.95
Skinhorn
63.74
63.14
22

---

Published as a conference paper at ICLR 2026
Table 7: Details for the datasets in experiments.
Dataset
Classes
Test Size
ImageNet
1000
50,000
Tiny- Tin-ImageNeet
200
10,000
ImageNet-R
200)
30,000
ImageNet-A
200
6862
ImageNet-Sketeh
1000
50,889
ImageNet-V2
1000
10,000
Caltech101
100
2465
DTD
47
1692
EuroSAT
10
8100
Aircraft
100
3333
Food101
101
30,300
Flowers 102
102
2463
Oxford Pets
37
3669
Cars 96
196
8041
Sun397
397
19,850
UCF101
101
3783
Table 8: Accuracy (%) comparison between LLM-based prompt generation baselines and their combinations with our method on fine-grained datasets using the CLIP-ViT-B/16 backbone. Bold
values represent the highest accuracy in each row.
Method Caltech101
DTD
FuroSAT
Aircraft
Food101
Flower102
Pets
Cars
SUN397
UCFI01
Average
CuPL
93.68
50.27
$2.69
25.57
86.71
71.31
10
65.31
65.13
70.33
67.01
CuPL+Ours
94.27
56.67
89.24
67.91
MPVR 93.98
50.12
55.47
26.18
86.89
72.14
89.07
66.97
65.24
70.42
67.65
MPVR+Ours
94.23
50.46
56.82
86.87
72.25
89.24
67.13
67.32
71.37
68.18
VisDese
94.52
50.59
56.12
25.16
85.75
71.89
88.87
67.28
67.87
70.37
67.84
F.4 COMBINING CARPRT WITH LLM-EMPOWERED PROMPT AUGMENTATION METHODS
Although CARPRT and LLM-empowered prompt augmentation methods are conceptually different, they can be combined in complementary way. CARPRT is training- fhining-free and inference-only method, relying solely on fixed prompt template pool and without using any external knowledge such as LLMs. By contrast, CuPL (Shtedritski et al., 2023). MPVR (Mirza et al., 2024), and VisDesc (Menon & Vondrick, 2023b) generate class-specific prompts/descriptoss via large language models and thus address different setting. Importantly, these approaches are orthogonal to ours: while direct comparison is not the focus, CARPRT can reweight LLM-generated prompts, and combining them consistently brings further gains.
As shown in Tab. integrating CARPRT with LLM-based prompt generation methods consistently improves their performance across datasets. This demonstrates that class- reweighting is plementary to LLM-generated prompts, enhancing their effectiveness without altering the underlying generation process. While VisDese be competitive stronger in some cases, it requires more complex pipeline and additional resources, whereas CARPRT provides a lightweight plug-in alternative.
G ADDITIONAL EXPERIMENTS
G.1 FILTERING LOW-CONHDENCE PSEUDO-LABELS
We evaluate whether explicitly filtering low-confidence pseudo-labels improves performance. We consider two heuristics: (i) confidence- basedd filtering by thresholding the maximum similarity score, and (ii) entropy-bard based filtering by thresholding class-wise prediction entropy. As shown in Tab. explicit filtering yields only marginal and inconsistent gains across datasets, sensitive the choice
23

---

Published as a conference paper at ICLR 2026
Table 9: Accuracy (%) comparison hetween confidence-/cetronyy-aass pseudo-label filtering variants and their combinations with CARPRT on fine-grained datasets. Bold values represent the highest accuracy in each row.
Thresh.
Method
Aircraft
DTD
BuroSAT
Food101
Pcts
Caltech101
Avcrage
Confidence-based filtering (max score)
WPE
22.28
47.18
52.37
85.49
81.46
92.62
63.57
0.30
CARPRT
24.10
47.45
58.31
85.16
90.06
94.01
66.52
Filter Ratio
0.51
0.15
0.16
0.53
0.60
0.34
WPE
22.11
47.18
51.88
85.34
80.92
94.24
63.61
0.25
CARPRT
24.73
47.45
54.87
86.29
89.86
94.60
66.30
Filter Ratio
0.98
0.77
0.77
0.97
0.98
0.81
WPE
23.28
47.18
49.60
86. 14
82.38
93.09
63.61
0.00 (orig.)
CARPRT
24.49
48.90
55.56
86.31
89.13
94.16
66.43
Entropy-beed -base filtering (prediction entropy)
WPE
21.90
44.80
51.92
85.41
92.57
63.11
63.11
2.0
CARPRT
23.21
47.63
53.54
86.31
94.36
65.82
65.82
Filter Ratio
0.18
0.41
0.88
0.91
0.88
WPE
21.95
45.85
49.60
85.33
92.61
62.81
62.81
2.5
CARPRT
24.20
48.13
55.56
86.27
94.69
66.39
66.39
Filter Ratio
0.38
0.62
1.00
0.97
0.92
WPE
23.28
47.18
49.60
86.14
93.09
63.61
63.61
max (orig.)
CARPRT
24.49
48.90
55.56
86.31
96.16
66.43
66.43
of threshold, and can occasionally lead to performance drops. Overall, these results suggest that CARPRT is relatively robust to noisy pseudo-labels, and additional filtering heuristics provide limited practical benefit.
G.2 DETAILED RESULTS FOR HYPERPARAMETER ANALYSIS
In this section, analyze the impact of key hyperparameters across all fine-grained datasets, focusing on the temperature parameter т. In zero-shot classification, where only test data is available, conventional hyperparameter selection is inherently challenging due to the absence of training validation data. Following Shu et al. (2018), we aim to identify hyperparameters that exhibit robust and consistent performance across diverse datasets.
As shown in Tab. 10, accuracy peaks at r = 1.0 and remains stable across a broad range, with a slight decline at higher values. A lower temperature, such 0.5, sharpens focus the most probable prompts but reduces distribution spread, limiting the ensemble effect of 247 prompt templates. This effect is crucial for capturing diverse information cues, and excessive concentration prompts may lead to performance degradation. While - = 1.0 may not be optimal for every dataset, serves as a practical and generalizable choice under zero-shot constraints.
dominant
G.3 RESULTS ON IMAGENET'S VARIANTS DATASETS
We also evaluate the performance of our method across Tiny-ImageNet and its variant datasets (ImageNet-A, ImageNet-R, ImageNet-Sketch, and ImageNet-V2D V2), as shown in Tab. 11. The im- provements on ImageNet and its variants datasets are smaller compared those observed on the fine-graines datasets (shown in Tab. I), for the following reasons. First, frequency bias is likely more pronounced in ImageNet and its variants. Given our use of relatively small batch size of 512 and the exclusion of larger datasets such as LAION-400M for debiasing, the skewed class distribution may have negatively impacted the results. Second, the quality of the template pool plays a crucial role in model performance. According to (Allingham et al., 2023), the template pool was constructed by combining templates from 10 fine-grained datasets and ImageNet and its variants datasets. Fine- sine-grained datasets benefit more from the pool, they contrast, the more diverse categories in ImageNet and its variants find less relevant information in the fine-grained templates, deriving less benefit from these templates. This mismatch reduces
exploit class olass-specific templates. In
24

---

Published as a conference paper at ICLR 2026
Table 10: Accuracy($)) results for varying temperature settings across fine-grained datasets using CLIP-ViT-B/16 and CLIP-ResNet50 backbones, Bold value represents the highest accuracy in each
column.
Temperature
Caltech101
DID BuroSAT
Aireraft
Food101
Hlower102
Pets
Cars
SUN397
UCHI0L
Average
CLIP VIT-B/16
0.5
93.45
49.13
53.29
23.97
87.26
71.82
88.69
64.66
66.32
69.68
66.83
(selected)
94.16
55.56
86.31
66.93
67.34
2.0
94.07
48.54
$5.19
24.17
85.87
71.12
88.69
65.67
66.07
70.11
66.95
88.39
4.0
93.87
48.16
$5.07
23.96
85.69
70.93
88.36
65.21
65.91
69.95
66.71
CLIP-ResNet50
0.5
88.67
38.92
34.31
16.61
77.11
66.05
86.40
56.56
60.47
62.43
58.75
(sellcted)
88.46
41.31
36.84
16.88
76.88
65. 56
56.44
61,28
63.66
$9.30
2.0
88.64
41.13
35.00
16.54
76.43
64.26
84.07
56.51
61.04
64.09
58.77
3.0
88.29
41.41
32.41
16.50
76.20
64.31
83.41
56.35
60.88
63.70
58.35
4.0
88. 18
41.30
31.78
16.48
76.08
64.36
82.94
56,34
60.65
63.64
58.17
5.0
88.07
41.20
31.14
16.46
75.96
64.40
82.46
56.33
60.64
63.17
57.98
Table 11: Accuracy (%) comparison between baselines and method ImageNet and its variants using
and CLIP-ResNet50 backbones. Bold value represents the highest accuracy on each column. Standard deviations are shown inline using +.
ImageNet
Tiny-Imagen Ima-emegeNet
-R
-Sketch
Average
CLIP- V/T-B/16
MPR 67.59
62.12
49.35
77.33
46.92
61.37
60.51
WPE
68.28:10.01
62. 19:0055
50.07
77.25
47.14
61.8110.1
61.12::0066
CARPRT (Ours)
68.59
62.71 20004
51.60
77.48=2004
47.53=0.02
62.11:000
61.67=0005
CLIP-ResNet50
MPR
59.12
43.32
46.25
69.05
39.05
$4.05
$3.50
WPE
59.78=2001
46.37
69.27:001
14.1077
34.07:100
$3.72::0.06
CARPRT (Ours)
59.98=2002
43.45
69.594001
39.34=0004
54.26:000
53.90:00066
method's effectiveness ImageNet datasets, it depends template-provide information. These limitations suggest that mitigating frequency bias and enhancing template relevance for broader datasets could further improve CARPRT's performance.
G.4 EXPERIMENTS ON IMBALANCED DATASETS
In this section, we evaluate the performance of CARPRT on datasets with class imbalances. Following Cao et al. (2019), we manually construct an imbalanced CIFAR-10 (Krizhevsky et al., 2009) dataset using exponential decay strategy to create various degrees of class imbalance. We use imbalance factor to describe the severity of the long-taile distribution, defined as the ratio between the number of training samples in the most frequent class and the least frequent class. Specifically, B is given by:
Nmax Nmin
8
where Nmax and Nmin represent the number of training samples the most frequent and least frequent classes, respectively. We conduct experiments with different imbalance ratios, setting 10, 50, and B 100, using the CLIP-ViT-B/16 backbone.
The results shown in Tab. 12 demonstrate that CARPRT significantly outperforms the average baseline for degrees of class imbalance. Specifically, CARPRT provides a consistent improvement in performance over WPE, though the gain decreases as the imbalance factor 3 increases. This decreasing gain may be attributed to the global nature of the WPE weight estimation, which remains effective even under higher imbalance. WPE calculates single weight for the entire dataset, capturing the overall distribution and maintaining reasonable performance, even when certain classes underrepresented.
25

---

Published as a conference paper at ICLR 2026
Table 12: Accuracy (%) comparison between method and baselines CIFAR-10 using the CLIP-ViT-B/16 backbone, Bold values represent the highest accuracy in each column.
Balanced Datasets
3=
B = 50
B 100
MPE
89.56
89.58
89.57
89.56
WPE
89.55
90.02
90.78
91.07
CARPRT (Ours)
90.82
91.07
91.36
91.70
uses per-class weighting strategy, which allows better adaptation to individual class characteristics, which is highly effective in balanced or moderately imbalanced settings. How- ever, when the class imbalance becomes severe, the challenge arises for classes with very few samples (e.g., only 10 samples). In these cases, the reliability of CARPRT's weight estimates decreases as a result of insufficient data, impacting performance.
In contrast, CARPRT
G.5 IMPACT OF TEMPLATE QUALITY
In this section, we investigate the impact of template quality on ImageNet classification tasks. Specifically, we explore how different prompt template pools influence performance by evaluating two newly generated template pools alongside the original templates the ImageNet datasets. Specifically, Pooll was generated using Claude 3.5 (Anthropic, 2024) to produce 300 templates tailored to the ImageNet label space. Each category in Pooll consists of 100 prompt templates structured descriptive formats, such as "A photo of a ", "A photo of a ", "The rype of ". These templates aim incorporate task-specific context and improve the alignment between the prompts and ImageNet categories. Pool2, on the other hand, was constructed using Phi 3.1 (Microsoft, 2024) create highly descriptive templates. For each ImageNet category, Phi 3.1 generated five detailed prompts, resulting in a total of 5,000 templates across all categories. These templates focus on providing class-specific descriptive information, enabling more precise and nuanced interaction with the underlying vision-language model. These additional template pools were evaluated on ImageNet dataset compared the original templates (Pool0), as shown Tab. 13.
Table 13: Accuracy (%) comparison across different template pools using WPE and CARPRT
methods on ImageNet classification.
Pool
Method
ImageNet Acc. (%)
Perf. Comparison
WPE
68.28
PoolO
CARPRT
68.59
+0.31
WPE
68.35
Pooll
CARPRT
68.61
+0.26
WPE
68.34
Pool2
CARPRT
68.97
+0.63
Pooll targets more task-specific information by generating templates with respect to the ImageNet label space. This leads performance improvements for both WPE and CARPRT prompt reweighting strategies compared to PoolO. On the other hand, the generated templates in Pool2 incorporate more class-specific descriptive information. CARPRT benefits significantly from these templates, achieving greater performance gains compared to WPE. This highlights the effectiveness of class-aware prompt reweighting in leveraging descriptive templates.
Future Work. Results in App. G.5 show that high-quality prompt template pool significantly improves performance. Building on these results and the previously discussed limitations, a key direction for future work is enhancing the quality and diversity of the prompt template pool, which ex- isting methods often overlook. Future research could focus on cost-effective strategies for generating and evaluating diverse, representative prompts. This may include developing metrics to assess how well prompts capture class-specific characteristics and enhancing inter-class distinctions to improve the model's ability differentiate closely related categories.
26

---

Published as a conference paper at ICLR 2026
Table 14: Comparison of normalization schemes under WPE and CARPRT. Accuracy (%) is reported on Fine-Graine Grained, ImageNet, and Variant subsets, along with the average across them.
Method Normalization Schemes
Fine-Grained
ImageNet
Variant
Average
none
64.82
68,28
59.69
64.26
64.93
68.45
59.72
64.37
WPE
pre-train
65.01
68.64
59.57
64.41
both
65.00
68.56
59.74
64.43
none
67.34
68.59
60.39
65.44
test
67.12
68.27
60.18
65.19
CARPRT
pre-train
67.45
68.72
60.55
65.57
both
67.44
68.77
60.53
65.58
G.6 ANALYSIS FREQUENCY BIAS CORRECTION
To correct potential biases introduced by the class frequency distribution in the pre-training or test- time datasets, Allingham et al. (2023) applies normalization to the score matrix before computing the prompt weights. This step ensures that the scale and distribution of class-prompt scores are consistent across categories and prompts, thereby mitigating dataset-specific artifacts that could affect final predictions. The scores 8j,i.c across all images æ; predicted to class ye under prompt p; are normalized as follows:
$J = S3,1.c 14,
(16)
where y denotes the mean of the scores, computed differently depending the normalization scheme:
(1) none: No normalization is applied and we set μ 0: (2) test: i computed by the test data scores:
Nita Σj=1 $j,i.ii (3) pre-train: f is computed by the data drawn from LAION-4000
14 11 pest
Ntwu
Npr
(Schuhmann et al., 2021), following Allingham al. (2023): μ1 p Nere Zj=1
$y,.ci (4)
both: Combine the two sources by interpolation: = (f1 tes + / These normalized scores are then used compute prompt weights.
As shown in Tab. 14, the WPE method benefits noticeably from normalization. All normalization schemes improve over the unnormalized baseline, with the both setting achieving the best overall performance. This suggests that WPE is sensitive to distributional bias and gains from explicitly correcting both pre-training and test-time frequency effects.
By contrast, CARPRT performs robustly across all settings. Even without normalization, CARPRT outperforms WPE, and gains only slight improvements from applying are-train both nor- malization. Interestingly, test-only normalization slightly reduces performance, indicating that test-derived statistics may inject noise rather than correct meaningful bias. This robustness likely stems from the class-aware formulation of CARPRT, which captures prompt-class dependencies more explicitly.
In summary, while WPE requires normalization to mitigate its reliance on biased score distributions, CARPRT consistently maintains strong performance, demonstrating its effectiveness as a prompt reweighting method.
H DISCUSSION OF PRIOR DISTRIBUTION OF THE PROMPT WEIGHTS Pr(W[[P)
We extend the discussion of the proposed probabilistic interpretation (Sec. 3) to the weights prior Pr. In the current zero-shot classification scenario addressed by CARPRT, there is no optimization-
process for "estimating" the weights, and as such, the weight prior PP) does not play role in the methodology. Nevertheless, our probabilistie framework is flexible enough to accommodate more general trainable settings, such as active learning and few-shot estimation, where the probabilistic formulation becomes particularly beneficial. In these cases, discussion of
27

---

Published as a conference paper at ICLR 2026
the weight prior would provide valuable insights and contribute to complete understanding of the framework's advantages.
Suppose there is label space y' with size | C. Let = {pr be pool of independent prompt templates. Let W WW be our weight matrix. Recall that W E A is the (n - -iddimnnsion probability simplex, representing the weights for class ye across all prompts.
We consider three choices of priors: uniform prior, global Dirichlet prior, and class-specific Dirichlet priors
Uniform Prior. The uniform prior assumes all valid weight configurations are equally likely priori.
W E W
p PP)
W
otherwise
where W = (W E RaxC; W. E An11 for all e (1....
The uniform prior is the easiest setup to implement and does not introduce bias towards any particular weight configuration. However, the uniform prior does not leverage any prior knowledge about the prompts, which prone to overfitting with limited data (when adapted to trainable setting).
Global Dirichlet Prior. This defines a single Dirichlet distribution over all weights, treating them as a single vector.
(Br(ve AnC) p W IP)
where vee( W) is the vectorization of W, and a; 0 are concentration parameters of the Dirichlet distribution.
Compared to uniform prior, Dirichlet prior can encode varying degrees of certainty about different weights. Moreover, is conjugate to multinomial likelihood, allowing for closed-form posterior updates for certain model setup. This also align with WPE-like However, it ignores the class structure and treats all weights as part of a single distribution, potentially missing class-specific patterns.
strategies.
Class-specific Dirichlet Prior. This strategy sets an independent Dirichlet distribution for each class's weight, and stacks a product of C' classes' Dirichlet distributions.
C II Dir(
p
PP)
(.,..)
where C'e > 0 are class and prompt-specific contenration parameters.
Currently, this setup best suits our class-aware prompt reweighting mechanism, it allows for different prior beliefs about weight distributions for each class, class-specific modeling. Compared with global Dirichlet, it reduces dimensionality each Dirichlet distribution is over n parameters, not n anymore. More importantly, it aligns with the per-class simplex constraint of the weight space.
Entropy Analysis. Different prior choices lead to different entropy results. The uniform prior has an associated entropy as
(PP(W[P)) unitorm log [W].
where [ is the volume of the weight space.
As for global Dirichlet prior, we have
nC - (a)(1)
H[p(W[P) = log B(a) (0o nC)
where B(-) is the multivariate beta function, and vb) is the digamma function.
The entropy for class-specific Dirichlet priors is
Hp(WP)) Σ (log B(ac) + (acp - n)b((cc)) - Σ
(ac,s - 1)p((cc)))
1=1
28

---

Published as a conference paper at ICLR 2026
where 0e (Ac,1, ... Gc,n) and dc,D Ei=1 dcii for each class
When we are setting the equal concentration parameters, such that a; = a for all in the global Dirichlet, and de,i for all i in the class-specific Dirichlets, and let 1, the uniform prior has the highest entropy (uninformative), while the class -pecifi Dirichlets having the lowest entropy. This is because the class-specific Dirichlets with 1 equivalent to independent uniform distributions over smaller simplices, further concentrating the probability.
I DETAILED PROOFS
Lemma (Relative Likelihood cf. Lemma 1). The likelihood of an image æ, given class c, prompt weights W and a prompt pool IP, following the EBM defined in Eq. 6, is proportional to:
Pr(mz]le: W. P) x exp (sim(z), x exp
(wie
2
(17)
where 2) = f(x,)) and 2 g(pi(y))) are image embeddings of sample æ; and text embeddings of class Ye under prompt pa respectively.
Proof. Similarity as Negative Energy. As with (LeCun et al., 2006), a general form of EBMs is given by (x)x (xp(-BEg(z)))2((0)) which enables us define unnormalized energy function with a partition function for normalization. Therefore, in our zero-shot classification context, we define the energy function with respect to the score function of the CLIP.
Exx Vc: W,P) = sim(2), x²""
This score function measures the compatibility between the image embedding 2] and the text embedding embedding x class Ye. higher compatibility corresponds to lower energy, aligning with the EBM principle that more likely configurations (of model) have lower energy.
Intractable Partition Function. Computing the partition function is intractable since we need to marginalize over the image space. However, what we about is the relative relation between Pr(æ,/ue: W,P) and Pr(x, Be, W, W).), we can safely drop off the partition function in our relative likelihood.
Similarity Computation. Consider a general linear combination of similarities for a prompt ensem- ble:
sim(2', =..) = he (fsim(z', +
he he(1s)
Σ ai,0 + 8
where 12d IR is function that linearly combines the similarities over prompts pt for specific class Ve. di,c E IR and Be E R are weights and bias terms. Substituting s; sim =x.0) = ZTT I!, we get:
sim 3)
T 21.e)
Σ
an. (T.T)) 2/ + Be
We can then absorb the bias term 3e into the exponential function,
Pr(xz/Ue. W.P) x exp(sim( 23J. xTT.))
= cxpl
Cs (2TTe) T 2 + Be)
1=1
cxp( exp
ai, (2 TT 2/)
(a TT) 2/).
expl
By setting Wi.c 0iges arrive the formulation in Lemma
29

---

Published as a conference paper at ICLR 2026
Proposition 3 (cf. Proposition 2). Let X be the image space, y be the class space. Given of prompts F. for any prompt weighting scheme S (cf. Eq. 1), define the representable likelihood set Fs
Fs xf=
E Ws,P, s.t. f(x, Mc) Pr(x/yo: W.P)),
where Ws is the weight space under the scheme S. Let Fai and Fcs be the representable likelihood from
set induced from class-inde iasseeindeeennent weighting and class-aware weighting (cf. Eq. 1) schemes. Then,
have: 3f E Fcs such that Vfcr Fa, lx X, Me where (x, Me) fcr(æ, Mc).
Proof. We prove this by constructing a specific function in Fcs and showing it cannot be represented by any function in Fer. For simplicity, consider toy setting with three classes y {yn, y/, ya3 and two prompts P = {pi.p2). For any E X, the function under class-aware weighting for ye {81, y2, yay takes the form:
(0, Me)
Σ wie Pr(ae ((ollyyePP))
Wi.c Pr(x yes P1) + W2,c Pr( Ve.P2).
where Wij R.+ class-aware weights for prompt and class For ease of notation, denote the prompt-conditional likelihood by ai,c & Pr(x Mes pı). This way f Fes can be expressed
f'(x, V1) = W1,101,1 + 22,102,1
f" 322 11,201,2 + 22,202,
f" ys) = W1,301,3 + W2,302,3
We then consider specific instance" of this function by choosing:
W1,1 2, W221
11,2 = 1, W2,2 =
W1,3 3, W2,3
This leads
f' (x, v1) = 201,1 + 02
f" 322 = a1,2 + 202,2
f' ys) = --31,3 + 302,3
Now, suppose for contradiction that 3fa E Fai such that = fci. By definition Fei. fai takes the form [ar( ye) wi 01,e + W2002,0 where W1, W2 R.+ class-independent weights.
f" = fa. then for all classes Ye E {y), 12, }3 we must have the following equations to hold simultaneously:
20 + 02,1 11 + w202.1
(for y1)
01.2 + 02, = 11 12 + W202,2
(for 1/2)
3 + 332,3 W111,3 W202,3
(for 1/a)
From these equations, deduce that Wi 2 and Wy 1 must hold for any 01,1, 02,1 >> 0 (for y1)
Wi = 1 and W2 = 2 must hold for any 01,2, 02, > 0 (for y2)
Wh 3 and W12 3 must hold for any 01,3, 02,3 >> 0 (for yn)
Thus, need wi 2 for yn while wn for y2, immediately leading contradiction wi cannot simultaneously equal and 2.
Therefore, class-independent weighting scheme can represent the function f constructed. We have proven that 3 E Fes such that ffa E Fcr, l E X' Be E gY where f" Ue) + far ve).
uunnormalized weights, just for illustration
30

---

Published as a conference paper at ICLR 2026
2.00
Senche Wemp
5004
2.001
Seledtdd Clrss index
-208 0.00
60
(a) Truncated Class- Weights Heatmap
(b) Per-Class Weight Distribution for "a low resolution photo of ()."
Figure 6: Visualization of class-aware prompt weights estimated by CARPRT on Caltech101 dataset. (a) The heatmap shows prompt weights across subset of classes and prompts, revealing diverse weight patterns and confirming class-specific preferences. (b) The bar plot displays the distribution of prompt weights assigned to the prompt "a low resolution photo of (}" across all classes.
J ADDITIONAL VISUALIZATIONS OF PROMPT WEIGHTS
To provide qualitative insight into CARPRT's mechanism, we first visualize the learned class-specific prompt weights on Caltech101 dataset. Fig. 6(a) shows the truncated weight matrix for a subset of prompts (n' < columns) and classes (C" < C' rows) from the full matrix W E IR= where clear differences the weights assigned to the same prompt across different classes are evident. Fig. 6(b) further illustrates this class-dependency by plotting the weights of single prompt template-"s low resolution photo of a {}" - across classes, demonstrating that the contribution this prompt is tailored each class. These visualizations corroborate quantitative results, confirming that CARPRT prioritizes prompts differently for each class.
In addition, include additional visualizations of the CARPRT all ten fine-grained datasets in the supplementary material (due to file size, these figures are not embedded in the main PDF). Each visualization is presented as heatmap. where the vertical axis corresponds to the prompt index and the horizontal axis to the class index.
prompt weights across
These heatmaps consistently reveal the class-specific nature of the learned weights: the columns exhibit noticeable variation across prompts rather than remaining uniform, indicating that different prompts are emphasized for different classes. Moreover, for most fine tin-grained datasets, only a small subset of prompts receive high weights across classes, while the majority are down-weighted this sparsity manifests visually as a few strong horizontal lines. This trend is particularly evident on Food 01, where the semantic homogeneity of the dataset leads to more consistent prompt preferences across classes.
Nevertheless, even within Food101, the highest-weighted prompt still varies across classes, demon- strating that class-aware prompt weighting remains essential. These results collectively support the effectiveness of WPE (Allingham et al., 2023) in highlighting useful prompts for the dataset, while also confirming the necessity of CARPRT's Class-awar weighting to fully capture intra-dataset variation.
K USE LARGE LANGUAGE MODELS (LLMs)
In preparing this submission, used LLMss solely writing aids improve readability. Specifically, LLMs were employed to correct grammar errors and polish the text. No part of the scientific content- content-including problem formulation, method design, experiments, analysis-is generated by LLMs. All technical contributions and claims were conceived, implemented, and evaluated by the authors.
31


Reviews:
[{"review_id": "XXdOP7e4x7", "reviewer": "Reviewer_xSa5", "summary": "This paper proposes a new scheme for prompt ensembling. The weights of each prompt for a given class $c$ are based on the average similarity score between the pseudo-label related to class $c$ and the image embedding. Experiments show improvement in accuracy and weights related to meaningful concepts.", "questions": "How does the proposed approach compare to few-shot prompt learning VLM adaptation methods both in term of performances and complexity?", "limitations": "", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The method is simple and interpretable by design. The improvements are consistent across multiple datasets and backbones.", "weaknesses": "The contribution is relatively incremental, as it can be summed up as a class-wise weighting of prompts based on their cosine similarity with the input image. Furthermore, recent works also explore the prompt ensembling strategy and should be discussed and compared [1,2]\n\nEnsembling is known for its inference overhead, which might be prohibitive on many applications. Ablations showing the complexity and inference time would be required to evaluate the practicability of the approach.\n\nThe Bayesian framework is relatively artificial, as the core of the approach does not rely on the predictive distribution.\n\n[1] Liao, Ning et al. “Rethinking Visual Prompt Learning as Masked Visual Token Modeling.” Artif. Intell. 348 (2023): 104417.\n[2] Huang, Chen, et al. \"Aggregate-and-adapt natural language prompts for downstream generalization of clip.\" Advances in Neural Information Processing Systems (2024)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nABgSZFKHp", "reviewer": "Reviewer_jRZ1", "summary": "This paper introduces CARPRT, a training-free method for zero-shot prompt ensembling in vision–language models. Unlike previous approaches that assign global weights to prompts, CARPRT estimates class-specific prompt weights from unlabeled data. The method computes image–text similarities for all prompt–class combinations, assigns pseudo-labels based on these similarities, and averages the scores for images pseudo-assigned to each class to derive class-specific prompt weights. These weights are then used to construct text embeddings for zero-shot classification. Experiments on fine-grained datasets and ImageNet variants show consistent improvements over mean and weighted prompt ensembling baselines.", "questions": "1. What is the size of the unlabeled dataset used to estimate prompt weights, and how does performance scale with this size?\n\n2. How does the approach behave under noisy or imbalanced pseudo-labels?\n\n3. Have the authors considered filtering low-confidence pseudo-labels to reduce the influence of uncertain assignments?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "- **Practical simplicity**\n\n    The proposed approach is conceptually simple yet effective. It operates entirely at inference time, without additional training or fine-tuning, and requires no architectural modification to the underlying vision–language model. This makes CARPRT easy to integrate into existing zero-shot pipelines.\n\n- **Clear motivation**\n\n    The paper addresses a well-identified limitation of class-agnostic prompt weighting by explicitly modeling the dependence between classes and prompts. The intuition is sound and well justified: prompts that work well for one class may not be equally suitable for another, and accounting for this improves prediction reliability.\n\n- **Good empirical results**\n\n    The empirical results show consistent gains across a range of architectures and datasets.", "weaknesses": "- Theoretical analysis does not provide substantial insight and formalizes already intuitive facts.\nThe theoretical material feels disproportionate to the simplicity of the idea, and several results restate expected behavior rather than offering explanatory or predictive value.\n\n    - Proposition 1 is a standard Chernoff-Hoeffding bound on the convergence of the empirical distribution to the  true distribution. It does not hold when using pseudo-labels.  To see this, consider a case when the VLM systematically under-predicts or doesn't predicts at all a certain classe which can happen on very specialized downstream dataset (e.g. Aircraft). Then the pseudo-label frequency for this class (zero) do not converge to the true class probability.\n    - Proposition 2: Shows that class-specific weighting has greater representational capacity than class-agnostic weighting. This result is largely self-evident: allowing class-dependent weights simply provides more degrees of freedom in the model, and formalizing this point does not substantially deepen understanding of the method or its behavior.\n    -  EBM formulation & Lemma 1: The EBM framing does not lead to algorithmic consequences, and Lemma 1 effectively restates that prompt weights scale text embeddings and thus linearly affect log-likelihood. This follows directly from the modeling assumptions and does not yield new theoretical insight.\n\n- No analysis of robustness to pseudo-label errors. This limitation is evident in the iterative extension, which underperforms when the VLM is less accurate, suggesting sensitivity to noisy pseudo-labels.\n\n- The contribution, while useful and well-executed, feels incremental. Moving from global to class-specific prompt weighting is a modest conceptual step.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qcf32GoL2h", "reviewer": "Reviewer_pFNT", "summary": "This paper proposes Class-Aware Zero-Shot Prompt Reweighting (CARPRT), which leverages unlabeled test data to enhance existing zero-shot prompt reweighting methods. Specifically, it estimates a distinct weight for each template–class pair based on the unlabeled test images and utilizes this weight matrix to construct zero-shot text classifiers.", "questions": "I believe that incorporating the following experiments would further strengthen this paper.\n\n## Unsupervised baselines:\n\n\n1. Pseudo-labeling approach: Generate pseudo-labels for the unlabeled samples, obtain a visual classifier based on these pseudo-labels (via class mean), and then combine it with the original CLIP text classifier in a 50-50 manner to predict the remaining samples.\n\n\n2. Reference [1]: This work presents a published unsupervised method that can serve as a suitable baseline.\n\n\n3. Reference [2]: Although this is a transductive approach, it constructs a parametric model for classification and can therefore be adapted to the current setting.\n\n\n## Sampling settings:\n\n\n1. N-shot sampling: Evaluate performance with varying numbers of labeled samples per class (e.g., 1-shot, few-shot, and larger-N settings).\n\n\n2. Partial class coverage: Use only a subset of downstream classes to assess generalization ability to unseen classes.\n\n\n3. Random sampling: Conduct multiple runs of random sampling with a specified proportion of the test dataset. Note that the resulting class distribution may vary significantly across runs.\n\n\nIt is recommended that the authors include experiments comparing their proposed methods against the aforementioned baselines under these different sampling configurations.\n\n\n## References:\n[1] Liang, Jian, et al. “Realistic Unsupervised CLIP Fine-tuning with Universal Entropy Optimization.” Proceedings of the 41st International Conference on Machine Learning, 2024.\n\n[2] Zanella, Maxime, Benoît Gérin, and Ismail Ayed. “Boosting Vision-Language Models with Transduction.” Advances in Neural Information Processing Systems 37 (2024): 62223–62256.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "This paper is readable, and I enjoyed its presentation. It introduces a clear and well-motivated method for prompt reweighting using unlabeled test images. The proposed approach is grounded in probabilistic principles.", "weaknesses": "The paper introduces unlabeled test images to aid prompt reweighting. As far as I know, this should not be considered a zero-shot method. In fact, it should be regarded as unsupervised learning. This distinction raises two issues. First, it contradicts the zero-shot claim made in the paper. Second, it makes the comparison against the three baselines unfair, as those are genuine zero-shot methods.\n\nAnother critical point is the absence of discussion regarding the sampling methods used in the experiments. It is crucial to address this aspect, as the sampling techniques, as well as the number and distribution of samples, can have a substantial impact on the final performance.\n\nFurthermore, the assumption that we have access to unlabeled test images is somewhat unconventional, as it implies a data collection phase prior to the testing stage. If such a phase exists, one might reasonably ask why the labels are not collected as well. I also notice that the authors discuss transductive methods and methods using external resources in the appendix; however, these operate under different—and arguably more practical—settings than the one adopted in this paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3erWsnWOjX", "reviewer": "Reviewer_kLQi", "summary": "This paper addresses the limitations of class-agnostic prompt reweighting in zero-shot image classification with vision-language models (VLMs). The authors propose CARPRT, a training-free method that estimates class-specific prompt weights using only unlabeled data and VLM similarity scores. CARPRT introduces a principled probabilistic framework that justifies the need for class-aware weighting and provides an efficient implementation by aggregating maximum image-text similarity scores per prompt-class pair. Experiments show that CARPRT consistently outperforms class-agnostic reweighting baselines on various datasets, confirming the crucial role of class-aware prompt reweighting for improved zero-shot performance.", "questions": "* Why is the performance gap on ImageNet so small?\nGiven that ImageNet is closer to real-world image distributions, why does CARPRT offer minimal gains over WPE in this setting? Could this be due to prompt-template mismatch, semantic ambiguity, or dataset scale?\n\n* Is CARPRT effective in real-world, low-resource domains?\nCan the authors comment on whether CARPRT can be effective in domains like medical imaging or satellite imagery, where labels are scarce and prompts may carry domain-specific semantics?\n\n* Can CARPRT be combined with lightweight prompt tuning?\nWould it be beneficial to use CARPRT-derived weights as an initialization or prior for prompt tuning frameworks? Could this combination yield further performance gains in narrow domains?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "* The paper is well-organized and clearly written. The motivation is compelling, and the methodology is both conceptually sound and practically simple.\n\n* The probabilistic framework offers strong theoretical justification for class-specific prompt weighting, addressing assumptions made by previous works (e.g., WPE).\n\n* The method is training-free, requiring only inference-time access to unlabeled images, making it broadly applicable and computationally efficient.\n\n* Empirical results show consistent gains across datasets, validating the method's generality.", "weaknesses": "* Limited Performance Gain on ImageNet\nWhile CARPRT shows consistent improvements on fine-grained datasets, its advantage over WPE on the general-purpose ImageNet benchmark is marginal. This raises concerns about the method’s scalability or effectiveness in broader, more diverse domains.\n\n* Lack of Real-World Use Case Discussion\nThe paper focuses primarily on academic benchmarks. Without examples or discussions related to real-world scenarios (e.g., medical or industrial domains), it is hard to evaluate the method's broader applicability where labeled data is truly scarce.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)