Prompt-Robust Vision-Language Models via Meta-Finetuning
Authors: Haohui Liang
Runlin Huang
Yingjun Du
Yujia Hu
Weifeng Su
Cees G. M. Snoek
Venue: ICLR 2026 Poster (2026)
Keywords: Vision-Language Models
Prompt Learning
Meta-learning
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: To address this prompt sensitivity, we propose Promise, a meta-learning framework for prompt-Robust vision-language models via meta-finetuning, which explicitly learns to generalize across diverse prompt formulations.

Abstract: Vision-language models (VLMs) have demonstrated remarkable generalization across diverse tasks by leveraging large-scale image-text pretraining. However, their performance is notoriously unstable under variations in natural language prompts, posing a considerable challenge for reliable real-world deployment. To address this prompt sensitivity, we propose Promise, a meta-learning framework for prompt-Robust vision-language models via meta-finetuning, which explicitly learns to generalize across diverse prompt formulations. Our method operates in a dual-loop meta-finetuning setting: the inner loop adapts token embeddings based on a set of varied prompts, while the outer loop optimizes for generalization on unseen prompt variants. To further improve robustness, we introduce an adaptive prompt weighting mechanism that dynamically emphasizes more generalizable prompts and a token-specific learning rate module that fine-tunes individual prompt tokens based on contextual importance. We further establish that Promise’s weighted and preconditioned inner update provably (i) yields a one-step decrease of the outer empirical risk together with a contraction of across-prompt sensitivity, and (ii) tightens a data-dependent generalization bound evaluated at the post-inner initialization. Across 15 benchmarks spanning base-to-novel generalization, cross-dataset transfer, and domain shift, our approach consistently reduces prompt sensitivity and improves performance stability over existing prompt learning methods.

Published as a conference paper at ICLR 2026
PROMISE: PROMPT-ROBUST VISION-LANGUAG MOD- ELS VIA META- FINETUNING
Haohui Liang', Runlin Huang",, Yingjun Du², Yujia Hu', Weifeng Su', Cees G. M. Snoek? 21
Beijing Normal-Hove -ong Kong Baptist University
2 University of Amsterdam
* Equal contribution.
Corresponding author.
ABSTRACT
Vision-language models (VL.Ms) have demonstrated remarkable generalization across diverse tasks by leveraging large-scale image-text pretraining. However, their performance is notoriously unstable under variations in natural language prompts, posing considerable challenge for reliable real-world deployment. To address this prompt sensitivity, we propose Promise, a metaa-eernnng framework for prompt-robust vision-language models via meta-finetuning, which explicitly learns to generalize across diverse prompt formulations, Our method operates in a dual-loop mneta-finetuning setting: the inner loop adapts token embeddings based on a set of varied prompts, while the outer loop optimizes for generalization on unseen prompt variants. To further improve robustness, we introduce an adaptive prompt weighting mechanism that dynamically emphasizes more generalizable prompts and a token-specific learning rate module that fine-tunes individual prompt tokens based on contextual importance. We further establish that Promise's 's weighted and preconditioned inner update provably (i) yields a one-step decrease the outer em- pirical risk together with contraction of across prompt sensitivity, and (ii) tightens a tata-dependen generalization bound evaluated at the post-inner initialization. Across 15 benchmarks spanning base-to-nove generalization, cross-dataset trans- fer, and domain shift, our approach consistently reduces prompt sensitivity and improves performance stability over existing prompt learning methods.
INTRODUCTION
Vision-language models (VLMs) have achieved impressive generalization by aligning image and text representations through large-scale pretraining (Radford et al., 22021: Jia et al. [021] Li et al. 2022). A common zero-shot inference strategy in these models involves filling handerafted templates like "a photo of [CLASS]* and comparing encoded text features against image embeddings. However, their predictions surprisingly brittle: small variations in prompt phrasing can cause large performance fluctuations (Zhou et al., (2026)). This phenomenon, often referred to as prompr sensitivity, presents a serious obstacle when deploying VLMs reliably in real-world applications.
To mitigate manual prompt engineering, recent efforts in prompt learning introduce learnable tokens replace or augment natural language prompts (Lester et al. 2021 Zhou et al. 2022a Jia al. 2022). CoOp (Zhou et al. 2022b) and CoCoOp (Zhou et al.) (022a)) optimize textual prompts with supervision from few-shot examples but tend to overht on base classes and generalize poorly to novel ones. Khattak et al. (2023b) propose MaPLe, multi-modal approach that injects visual and textual prompts across network layers improve transferability. More recently, Guo & Gu (2025) enhance cross moosl representations via dynamic feature routing. While these approaches improve downstream adaptation, they still rely on a fixed set of prompt templates during finetuning and inference. As shown in Figure they remain vulnerable to prompt rewording and offer limited robustness to natural linguistic variability.
This paper addresses the underexplored challenge of building prompt- vision- vision-language models, models whose predictions remain stable across diverse prompt formulations. While prior meta- learning-based prompt tuning methods (Li et al. 2023 Park al. 2024 Zhao al. 2024) have

---

Published as a conference paper at ICLR 2026
CLIP
MaPLe
Promise
CLIP
MaPLe
Promise
CLIP
MaPLe
Promise
CLIP
MaPLe
Promise
photo
Itep
photo
origami
Figure Prompt sensitivity and performance comparison in VLMs. We visualize the predicted confidence scores five prompt formulations for the input image. CLIP and MaPI.e exhibit high sensitivity to prompt phrasing, with both larger fluctuations and lower overall scores. contrast, our method not only produces more consistent predictions across prompts, but also achieves higher confidence values, reflecting both improved robustness and better overall performance.
improved generalization by learning better prompt initializations, domain-invarian representations, task-level regularization, they typically assume fixed prompt templates and focus cross-task cross-domain transfer. In contrast, we target intra-task robustness by explicitly modeling variation semantically equivalent prompts within the same task. Our method addresses this gap through a dual-loop meta-learning framework that learns prompt-invariant token embeddings and dynamically adapts prompt-level variability.
To address prompt sensitivity, we introduce Promise, a meta-learning framework for prompt-robust lssion-anguuage models via meta- veta--inetuning. The model learns adapt token embeddings under varying prompt views and generalizes to unseen phrasings. Inspired by prior work on meta-gradient adaptation, we build a framework that enables both prompt-level and token-level adaptation. Our approach makes four key contributions. First, we design dual-loop finetuning strategy: the inner loop adapts learnable token embeddings using a subset of prompt templates, while the outer loop evaluates them on disjoint prompts to encourage prompt-invariant representations. Second, we incorporate an adaptive prompt weighting mechanism that dynamically prioritizes templates with higher generalization utility. Third, we develop token-specifie learning rate module, which fine- tunes each prompt token based on its contextual importance. These components work jointly to produce stable, generalizable predictions across prompt variations. Finally, prove that Promise weighted, preconditioned inner step both decreases the outer empirical risk in one move and contracts across-prompt sensitivity, thereby tightening data-dependent generalization bound at the post-inner initialization.
Extensive experiments across 15 datasets demonstrate that framework substantially reduces prompt sensitivity while enhancing generalization to novel categories. It consistently outperforms prior multi-modal prompt learning methods on base-to-new transfer, cross-dataset adaptation, and domain generalization benchmarks. Ablation studies further validate the effectiveness of each component, confirming that meta-learning. adaptive prompt weighting, and token-specing learning rates jointly contribute improved robustness and reduced sensitivity to prompt variation in vision- language models.
RELATED WORK
Learning the Prompt Template. This line of work treats the textual prompt as learnalee sentence or soft vector sequence. CoOp (Zhou et al. 2022b) and CoCoOp (Zhou et al.) 2022a) pioneered this approach by optimizing continuous embeddings of the prompt template in the language branch of CLIP (Radford et al.) 2021), enabling few-shot generalization
classes. Variants such as Bayesian prompt learning (Derakhshani et al. (023) further model uncertainty over soft prompts to improve generalization. Subsequent works explore different strategies to enhance prompt robust-
2

---

Published as a conference paper at ICLR 2026
ness and generalizability: GRAM (Li et al. 2023) and MetaPrompt (Zhao al. 2024) introduce meta-learning- based schemes to improve adaptation across domains, while DePT (Zhang et al. 2024) alleviates overfitting by decoupling base-specific and task-shared knowledge during prompt optimiza tion. These methods directly modify the textual input space, typically focusing on adapting the entire prompt string. In contrast, our work targets different axis of eeceraiizatonnrrouuttie
to prompt phrasing variation within the same task. Rather than optimizing fixed prompt template, we simulate prompt distribution shifts during training and meta-learn prompt-invarian token embeddings. This shift in focus-from task-level or domain-levell generalization to within- task prompt robustness is central to our design and has been underexplored in prior prompt tuning literature.
Learning the Prompt Tokens. An alternative and increasingly prevalent direction is to append learnable tokens to the input embeddings without altering the prompt text itself. This paradigm, often referred to as prefix tuning. has been applied to either the text encoder (Khattak et al. 2025) the
vision encoder (Jia et al. 2022 Bahng al., 2022), or both (Li et al. 2024 Khattak et al.) 2023b Roy
& Etemad| 2024; Guo & Gu 2025 Park et al.) 2024). These methods introduce soft tokens actoss one or more layers to improve model adaptation without modifying the underlying language prompt. Despite their effectiveness, they still rely fixed prompt templates during training and evaluation, making them vulnerable to prompt phrasing variations. Our work belongs to this category but goes beyond conventional prompt tokens by incorporating meta-learning explicitly learn from prompt variations. Instead of optimizing prompt tokens for a single fixed formulation, our framework learns to generalize across diverse prompt templates, thereby improving robustness to natural language variation.
METHOD
We propose prompt-robust vision-language models via Meta-finetuning (Promise), a meta-learning framework that enhances the robustness of vision-language models (VLMs) to variations in prompt phrasing. Our method Iearns prompt-invarian token representations by simulating prompt distribution shifts during finetuning. Specifically, it adopts dual-loop structure: the inner loop updates token embeddings using a subset of prompt templates, while the outer loop optimizes for generalization to unseen prompts. Additionally, our method introduces two key modules- (1) an adaptive prompt weighting mechanism to prioritize generalizable prompts, and (2) a token-specific learning rate sched- uler for fine-grained token adaptation. These components jointly improve stability and performance under prompt variation.
3.1 DUAL-LOOP META-FINETUSE FINETUNING
Let Pin and Pour denote the subsets of prompt templates used in the inner and outer loops, respectively, where Pin n Pou = 0. For each prompt T, Pin. we associate a learnable token e,, The meta- parameters 0 represent the initial shared parameters for all prompt-specific tokens, where 0g denotes the parameters associated with text tokens, and O, denotes those associated with visual features. For simplicity of notation, we refer these collectively as 0 from now on. In our instantiation, the "tasks" in the meta-learning sense correspond to different natural-language prompt variants within the same dataset, rather than to different datasets or label spaces. The doop-loop structure is therefore used to explicitly simulate prompt distribution shifts within fixed label space.
Inner-Loop Finetuning, In the inner loop, we adapt the text token embeddings e, for each prompt template T. Pin. Given an input image x and its corresponding label y. the model adapts to minimize the loss for each prompt Pu. This process can described as follows:
i -=0 0 a a Σ
VoLin x),)),
T.EP
where Lin(-) represents the loss computed with prompts from Pm, is the learning rate, and f(e; x) denotes the model's output given the adapted prompt embedding e; and input x. This prompt-specific adaptation helps the model capture the variations within the prompts in P.n.
Outer-Loop Finetuning, In the outer loop. we optimize the meta pera-merrmeters @ with a distinct set of prompt templates P.... For each prompt template T2 E Poun: the corresponding token embedding e,
3

---

Published as a conference paper at ICLR 2026
Inner-Loop Finetuning
Outer-Loop Finetuning
w. II) (6., 3V SVL.
8 aV.. fin
apaD strar
Viuua
elea
Pia
Puse <GLASI
dELASIO
top of CCLASON
ari of CELASON
good COLA
0
0g
VV Coi
(6, Mr, H)
(9, My,
updates
W1
Learnable weight prompt T,
Learning rate prompt
Figure Overview of Promise. Inner-loop finetuning adapts the prompt tokens using subset of prompts in Pn to minimize the loss. Outer-loop finetuning updates the meta-parameters 0 across the disjoint prompt subset
Pou to promote prompl-agnostic generalization. Adaptive prompt weighting and token-specific learning rates further stabilize performance under prompt variation.
utilized. Leveraging the adapted parameters i from the inner loop, refine 0 by minimizing the aggregate loss over all prompts in Pout
0-0-5 3 Σ ((e,,
TyEP
where (-a((-) is the loss calculated with prompts from Pou and / is the outer-loop learning rate. This process ensures that the meta-parameters are optimized generalize across both Pn, and Pour. achieving the intended prompt-agnostic performance. In practice, this inner/outer separation allows the model to adapt on one subset of prompt templates and be explicitly optimized for stable performance on disjoint, unseen phrasings of the same labels, which directly targets intra-task prompt robustness.
While MAML (Finn et al. 2017) provides general framework for meta mnta-grrdieet adaptation through an inner- and looe structure, it typically assumes uniform treatment of input samples during meta-training. Our formulation is MAML-inspired in that it uses similar dual-loop meta-finetuning structure, but it instantiated specifically for prompt robustness vision-language models by treating different prompt templates tasks and by explicitly optimizing for consistency across Pnn and Pout- Inspired by this dual-loop formulation, we adopt a similar meta-learning structure to simulate prompt variation: the inner loop performs prompt-specific adaptation, while the outer loop promotes generalization across prompt distributions. However, unlike MAML, it applies a uniform update across all inputs and lacks the flexibility to differentiate between prompts of varying generalization quality. This limitation is particularly problematic in prompt tuning, where some prompt templates are inherently more transferable than others.
3.2 ADAPTIVE PROMPT WEIGHTING
To further enhance robustness across varied prompt templates, Promise incorporates an adaptive prompt weighting mechanism. In this approach, each text prompt template T; is assigned learnable weight w; within the inner loop, which scales the loss corresponding to each prompt template. This allows the model learn the importance of each template dynamically, similar to mixture of expert models. The weights {w;] J are optimized in the outer loop, where represents the number of prompts in the inner loop. This optimization enables the model prioritize templates that contribute more effectively to generalization.
In the inner loop, for each prompt template T, Pin. we compute the prompt-specific loss Ci. weighted by the corresponding parameter wi. The adapted parameter 6 for each prompt T; is updated as:
Ô-0-a a Σ wi ()) T.EP.
(3)
4

---

Published as a conference paper at ICLR 2026
where (-n(-) represents the inner-loop loss for prompt T,, is the learning rate, and w; modulates the impact of each prompt-specifis loss in the inner loop. This weighted adaptation enables the model to learn varying levels of emphasis for different prompt templates during the inner-loop optimization.
After the inner-loop updates, the outer loop optimizes the meta-parameterd 0 together with a set of unnormalized prompt scores { on P To aggregate across prompts while controlling variance, we constrain the resulting weights w = [w...... : w to lie on the probability simplex 0: the outer step
A [w 0, Σ; w; = 1} via an entropic mirror map with a temperature performs a gradient update {s+] followed by mirror descent normalization onto AN-1, The temperature (learned or annealed) trades off exploration and concentration: larger T spreads mass across templates stabilize gradients, while smaller concentrates mass on high-utility prompts to reduce estimator variance. To further prevent degenerate collapse and encourage useful sparsity, we include an optional entropy regularizer À (H(W) (or its sparse alternative via entmax), which keeps few high-utility prompts active while pruning noisy ones. Intuitively, this simplex-conse contte--onsttrined, temperature-contret weighting focuses learning signal on prompts that best transfer to unseen phrasings, without hand-tuning per-template coefficients.
Finally, the outer-loop update for both the meta-parameters 0 and the weights {w;) defined as: 0 = 0 - VV Woot w, w, VV. Cout where 3 is the learning rate for the outer-loop optimization. This update process enables the model to adaptively assign higher importance to prompts that contribute more effectively to generalization, achieving more robust prompt-agnostic performance.
3.3 TOKEN-SPECIFI ADAPTIVE LEARNING RATE
In Promise, we implement a ooken-specifie adaptive learning rate module using a neural network that generates learning rates for each token embedding in the inner loop. Our design is inspired by MetaSGD (Li et al. 2017), which learns a task-specific learning rate for each parameter. We extend this idea by introducing token-wise, data-driven learning rate module. Instead of treating learning rates as fixed or per-task scalars, we use a neural network to generate wouumpttooken-speciie rates conditioned on input features. This enables fine-graine -eeegrraddd, context-aware adaptation for each token within the inner loop. Specifically, given set of prompt templates T; and their corresponding features extracted by the text encoder, (-) generates a unique learning rate ag for each learnable token e, in prompt template T..
learning
In the inner loop, the token-specific learning rate a; is used to update each token embedding e; in a targeted way. The adaptation process for each token embedding e; with the generated learning rate is as follows:
0 0= 04 Σ w; (o T.EP
where (.) represents the inner-loop loss for prompt template Tú and (e;)((;;) is the data-driven, token-specific learning rate generated by 11 (-). This token-specific adjustment enables the model to fine-tune each token based on its importance in the prompt, improving adaptation to variations in prompt templates. H (-) takes the encoded features of each prompt template T, as input and outputs a unique learning rate a; for each token embedding e;. This data-driven approach allows the learning rates to be conditioned on the specific characteristics of each prompt template, rather than using a uniform rate across all tokens. Formally, the token-specific learning rate for each token e; is defined as o,=HH(e,) where e; denotes the encoded features of prompt T, derived from the text encoder. This feature-drived learning rate allows the model to adapt each token's 's learning rate based on the semantic and contextual information within the prompt template.
In the outer loop, we update both the meta- parammearrsssss 0 and the parameters of H[(-) to optimize the model's generalization across different prompt templates. The outer-loop objective thus becomes: 0 - BV Cout, H = H VV Lout, where Loui is the cumulative outer-loop loss across prompts in Pone and 33 is the lute learning rate. This process ensures that both the meta meta- parameters and (.) optimized to enhance prompt-agnostic performance across diverse prompt templates.
5

---

Published as a conference paper at ICLR 2026
4 ANALYSIS
The core idea our Promise is to turn a single, globally shared SGD initialization into a prompt- aware initialization via weighted and preconditioned inner step. The outer optimization starts from a better-conditioned and lower-variance point by tailoring the inner update to the prompt distribution-tthruuggh adaptive prompt weights and token-wise diagonal preconditioner. In this section, we theoretically analyze the advantage of this adaptation: we show (i) guaranteed one-step decrease the outer empirical risk together with a contraction of across across-prompt sensitivity, and (ii) tightened, data-dependent generalization bound evaluated at the post-inner initialization.
Setup. For prompt template T, let Dr denote data distribution over samples = (x, y). and let ((0; T") be the per-sample loss. We define the population outer risk
Rpop (0) == EToP EenD, [ z, T)].
Given a finite set of samples {21 nh j=1
for each T, the per-prompt empirical loss is
L(0:T)
T).
Pone (0) = Pout
L(0;T),
Rout :=
Σ nT. TEPout
TUT
TePou
In each meta-episode we draw independent and disjoint prompt subsets (Pin, Pour) with Piu Pout Promise performs a weighted & preconditioned inner update
- - PE WT (oL(0,
= a.)) =0,
Σ WT = 1, WT 2 0,
TEPx
and then evaluates R Pon at 9 to update (0, w, H). For later use, denote the aggregated inner gradient and its variance by
G(O) =
Σ wT VoL(O;T), (()) = B(G(O), En == VarlG(O)). TER
Assumptions. (i) L(:TT) is L-smooth; (ii) stochastic gradients are unbiased with bounded second moment; (iii) P., and Pout independent draws from P; (iv) the weighted inner gradient aligns with the outer descent direction (Def. B.I.
Theorem 4.1 (tne outer-risk descent sensitivity contraction). Under the assumptions above, there exist constants E (0,1), k 2 1 determined by the alignment of the weighted inner gradient and by gradient heterogeneity across prompts, such that
E Rrema (0)
Re (0) (7 Aumin (P) 1xx211P113 [R)r -1111
where Exp = Varl Saa[ETER, wT VoL(O; []]] In particular, if ||P||2 small enough so that Y Aminn (P) > L 1 [P12, then the expected outer risk strictly decreases. Moreover, defining the across-prompt sensitivity S(A) Varr [fo (2; T)) (or its Jacobian surrogate), have
E [() < (1-Mail S(0) + C Tr(Ew),
for some Peft > depending on the local Lipschitz constants of fo and the same alignment, i.e., the inner step contracts prompt sensitivity up variance term that shrinks with Tr(E.).
The adaptive weights wy concentrate the inner update on prompts whose gradients are aligned with the outer descent direction (variance reduction), while the token-wise diagonal P rescales ill-conditioned coordinates (preconditioning). Their combination enlarges the one-step decrease of the outer objective and suppresses across ppoompt drift. Intuitively, the resulting inner update
0+ = P PV (0), (0) = Σ WT L(O,T),
TEP
acts as a cautious, biased gradient step that moves more strongly along directions supported by prompts which generalize well to Pout. while dampening directions that unstable across templates.

---

Published as a conference paper at ICLR 2026
Under the conditions of Theorem 4.1 this weighted and preconditioned step tends to shrink the prompt-averaged outer risk, which is what we refer to as a "risk contraction" effect.
Empirically, validate this behavior by counting how often single inner update decreases the outer empirical loss on Paut. On a representative subset of datasets, the fraction of such "risk-dccreasing"" inner steps increases from about 61% for unweighted, unpreconditioned update about 78% when using the full Promise update with both w and P. This matches the qualitative picture suggested by Theorem 4. even though the smoothness and alignment assumptions used in the analysis are only approximate for large CLIP-scale networks in practice.
Theorem 4.2 (Data-dependent generalization at the post-inner initialization). Let Ô 0 P ETer,, wr WrLooL( T)) Conditioned on 0, the samples in Pout are i.i.d.; hence for any 5 E (0, (0.
with probability al least 1 1- 5,
In(1/6) + ei T TI(E P).
Ppop (0) S Re (0) + e2
Nout
Here nout is the number of samples used by Rp I upper-bounds the 8- 8-Lisschitz constant of L(:; T). and the last term quantifies the initialization variability induced by the inner-step gradient noise. Consequently, any choice of (wr, P) that reduces both Rp.... (0) and (r(PE,, tightens the bound on (0).
The complete proof is provided in Appendix
EXPERIMENTS
5.1 EXPERIMENTAL SETUP
15 Datasets. To evaluate base-to-new generalization and cross-dataset generalization, we adopt diverse set of 11 image classification datasets, following prior work such as CLIP (Radford et al.)
202T) and CoOp (Zhou al. 2022b). These datasets cover wide range of visual recognition tasks: ImageNet (Deng et al. 2009) and Caltech101 (Fei-Fei et al.) 2004) are used for generic object classi- fication; OxfordPets (arkki al. 2012), StanfordCars (Krause et al. 2013) Flowers 102 NNilsbak
& Zisserman 2008). Food 101 (Bossard et al.) 2014). and FGVCAircraft Maji et al. 2013) are
included for fine-grained image recognition; EuroSAT (Helber al. 2019 employed for satellite image classification; UCF101 (Soomro et al. (012) for action recognition; DTD (Cimpoi et al.)
2014) for texture classification; and SUN397 (Xiao et al., 2010) for scene recognition. For domain generalization experiments, we follow CoOp (Zhou et al. 2022b) and use ImageNet as the source domain, with four distinct ImageNet variants serving as the target domains: ImageNetV2 (Recht) ct al. 2019), ImageNet-SSktcch (Wang et al. 2019). ImageNet-A (Hendrycks et al. 2021b). and ImageNet-R (Hendrycks et al. 2021a).
the CLIP-VIT-B/16 architecture the base model across all methods, consistent with previous work like CoCoOp (Zhou et al.) 2022a) and VPT (Ji et al. 2022). In line with MaPLe (Khattak et al. 2023a). set the prompt depth to 9 and used prompt lengths of 2 for both language and vision prompts. All models are trained for 10 epochs with batch size of 8 and learning A6000 GPU. For training on the full 1000 classes of ImageNet as the source model, we set the prompt depth and trained for 5 epochs with learning rate of 0.0035. Following PromptSRC (Khattak) ett al., 2023b), we adopt the 60 hand-crafted prompt templates originally provided in the PromptSRC appendix (Khattak et al. 2023b). each training episode, we randomly sample 30 templates for inner-loop adaptation and 30 disjoint templates for outer-loop generalization. Our network H consists of a 3-layer MLP. We perform three iterations for the inner loop and adopt Reptile (Nichol et al.) 2018)'s, using first-order derivatives to approximate the outer-loop loss. For consistency, all the results of the learning- based methods are computed as an average over three random seeds. All code will be made available. Additional results are in the Appendix: more robustness visualizations (App. (C), training time analysis (App. $G.2), results of structured proompt effects (App. $D) detailed full prompts (App. $ and algorithmic details (App. $
Implementation Details. To ensure fair comparison,
of 0.0035, utilizing the SGD optimizer on single NVIDIA
7

---

Published as a conference paper at ICLR 2026
UCF101 <CLASSE>
Caltech101
<ELASE>
CLASS
CCLASS
<CLASS
ACLASS
photo <CLASSS
HCLASS>
CLIP
MaPLe
PrompeSRC
MMRL
Promise
Figure Sensitivity models to various prompts UCF101 Caltech While other methods show considerable variability across templates, our method maintains consistent high performance, demonstrating robustness to prompt variation.
5.2
RESULTS
Prompt Sensitivity of Promise. To assess robustness prompt variation, we MaPLe e as the base prompt learner and apply our method on top, resulting in the combined model Promise, we compare CLIP, MaPLe. PromptSRC, and MMRL under diverse prompt templates. This analysis highlights the ability to maintain both high accuracy and stable predictions across formulations, achieving prompt-agnostic behavior. We begin with a quantitative study on the UCF101 and Caltech101 datasets, evaluating the consistency of predictions across different templates. As shown in Figure 3 while other methods exhibit notable performance fluctuations across prompts, our method yields consistently high accuracy regardless of template choice. This stability underscores the effectiveness of our approach mitigating prompt sensitivity and ensuring reliable performance under natural prompt shifts.
Enhancing Existing Prompt Learners. Table shows that our meta-learning framework con- sistently improves the performance of existing prompt learning methods across 11 datasets.
We apply our training strategy on top of MaPLe, IVLP, and MMRL by keeping their base architectures unchanged while replacing their original optimization with our dual-loop meta-learning procedure. Notably, this yields considerable gains in novel class accuracy- +2.09 on IVLP, +2.12 on MaPLe, and +1.68 on MMRI- maintaining slightly improving base class performance. These gains result in higher harmonic means across all methods, demon- strating that our framework not only enhances generalization but also integrates well with diverse prompting strategies.
Table Effeet of Promise.
Method
Base
New
H
77.66
Promise
MaPLe
82.28
75.14
78.55
Promise
83.57
77.26
80.29
MMRL
85.68
77.16
81.20
Promise
86.14
78.84
82.33
Benefit of Adaptive Prompt Weighting. We assess the impact of adaptive prompt weighting by comparing Promise variants with and without this component, using MaPLe and MMRL as backbones. As shown Table incorporating adaptive weighting consistently improves performance, particularly on novel classes and in harmonic mean, across both settings. This mechanism complements our meta meta-finetuning framework by enabling the model to dynamically prioritize more transferable prompts during adaptation, thereby enhancing optimization efficiency and generalization performance.
Table 2: Impact of adaptive prompt weighting.
Base model
Promise variant
Base
New
H
w/o adaptive weighting
82.93
75.87
79.24
MaPLe
w/ adaptive weighting
83.57
77.26
80.29
wlo adaptive weighting
85.97
77.53
81,53
MMRL.
adaprive weighting
86.14
78.84
82.33
Benefit of Token-Specific Adaptive Learning Rate. To isolate the effect of token-level adap-
tation, we compare three independent variants of our framework: (1) using fixed learning rate for all tokens, (2) adopting a task-specific adaptive learning rate following MetaSGD (Li al., 2017), and (3) our proposed token-specific learning rate module that assigns distinet learning rates to each prompt token. As shown in Table 3 the token-specific variant consis- tently yields better performance on both base and novel classes across MaPLe and MMRL.
8

---

Published as a conference paper at ICLR 2026
Table 4: Base-to-new generalization across 11 datasets. Our method achieves the highest harmonic mean, demonstrating robust performance and superior novel class generalization compared to previous methods. Blue numbers indicate the best performance in each column, cyan highlights the second-best, and the values in parentheses represent the difference between our method and the best previous method in cach column.
93.47
71.08
15.03
CaCAT
78.00
33.45
17.10
CuC
07.80
01.81
$5.81
86.30
75:00
14.77
UNKGRAM
76.0
33.00
71.53
UNICRAM
6000
06.11
66.57
6130
73:11
81:08
11
31.00
24.33
88.77
04.60
66.55
66.30
26.99
81:00
13.56
3175
74.86
18.1
04
KNNi (01-001111
77.16
11.00
NNNI
20
04.95
14.30
Oun
$6.14
1934
$2:31
18:00
74.85
16:11
90.08
$7.41
(a) Average over 11 datasets.
(b) ImageNet
(c) Caltech101
Bxcc
Nerd
W.BGo
Mlxd
Nevel
KUP
11.37
07:33
54.12
14.91
66.00
72884
18.55
11.13
15.35
17130
13.31
34.97
19.38
$3.35
35.35
0736
2014
18.34
73/1
12.48
97.06
74.45
$4.32
Colagg
33.87
0733
(2
35.0
12.68
02.27
78.65
$5.11
25.37
17.66
07.0
1874
76.00
5434
74.56
187T
02.50
$5.85
25.00
07.6
15.34
MDRRL
60.97
11.27
96.13
0924
70.44
80.58
96.99
14.
(d) OxfordPets
(c) StanfordCars
(l) Flowers102
R11
Rate
Navel
Bose
Neail
New
01.9
30.00
01.38
23:40
53.71
55:31
24.05
71.56
58.37
Propestr
40077
01:13
(3.53
17.87
10.15
08.53
UMICRAM
4020
0135
$1.51
07.30
14.04
$3.00
DeFt
01.33
Bar
40.73
04.27
$1:36
18m 00.
01.00
01.0
$1.01
40:0
42.09
02.81
11.11
Ouex
Ckar
(g) Food101
(h) FGVCAireraft
(i) SUN397
7741
16.10
14.55
$1.00
01.00
71:31
$2.35
13.43
17.54
U836RAS
7562
6231
$3.50
UNKERAN
18:25
71.35
78-12
UNLLRAM
$2.91
13.00
13.95
$5.13
66:13
12.9
p120
04.00
14.37
$2.34
$6.00
13.07
$3.37
13/02
64.00
13.31
Tuld
04.04
17.46
65.97
71.84
$3.35
17:07
65:0
1112
05.08
$1.17
MHRRI
$1.99
Oun
$6.25
03.57
76:3
05:41
77.66
$5.59
75.05
$3.52
(i) DTD
(k) EuroSAT
(1) UCF101
Table 3: Effect of token-speeific adaptive learning rate.
These results confirm the benefit of learning fine-grained, data-driven adaptation rates within the inner loop, providing more precise control over token optimization than task-level adaptation alone.
Base model
Promise variant
New
wio token- specific adaptive learning rate
$2.45
76.01
79.08
MaPLe
wi taskk specific adaptive learning rate
82.93
76.29
79.25
wi token specilic adaptive learning rale
83.57
77.26
80.29
token specific adaptive leorning rate
77.42
$1.39
MMRL
w' task speciic alaptive learning rale
85.99
77
81.48
w/ saken- specifi adapiive learning rate
66.14
78.84
82.33
Robustness cost trade-off. Promise is most useful in scenarios where robustness to prompt variation matters, not just raw accuracy. In many VLM applications, a model that behaves stably under different user phrasings can be more valuable than one that gains small amount of accuracy but remains highly sensitive to wording. In terms of efficiency, Promise only updates the soft prompt parameters and the small weighting/ / preconditioning modules; the vision and text encoders remain frozen. At inference time, the computation graph is identical to the underlying prompt learner, so the runtime and memory cost at deployment are unchanged. The additional cost appears only during training. In our implementation, meta-finent fenttfinntuning increases wall-clock training time by about 8822%% compared with standard prompt tuning for the same number of epochs, mainly due to the extra outer-loop gradient and token- wise preconditioning. For example, on ImageNet with MaPLe on single A6000, vanilla training takes about 3.4 hours, while Promise takes about 4.0 hours; on smaller datasets, the absolute overhead is correspondingly smaller. We also experimented with lighter variant that removes the token-wiss learning-rai rate module and keeps only adaptive weighting; this reduces the overhead by roughly 8-10%, but also weakens the robustness gains. Since meta-finetuning is one-time offline stage and inference cost is unchanged, we consider this training overhead acceptable in settings where prompts are reused extensively and stable behavior under prompt shifts is important.
Base-to-New Generalization. To evaluate the generalization performance of our method on novel classes, we conduct base-to-new experiments aeross 11 diverse datasets. As shown in Table 4

---

Published as a conference paper at ICLR 2026
Table 5: Cross-dataset generalization. Our method consistently improves cross-dataset generalization. Blue numbers indicate the best performance in each column, cyan highlights the second-best. and the values in parentheses represent the difference between our method and the best previous method in each column.
Source
Target
ImsgeNet
Calrech101
OxfontPiits
SunffrdCaas
Plovers02
Fosat0l
Airoaft
SUN0397
DTD
EureAAT
UCPI11
Avotage
CLIP Radford
71.51
64.15
41.92
CoOp
Ziooee RMAR
71.51
93.70
89.14
64.51
68.71
85:30
18.47
68.15
41.92
46.39
66.35
63.88
Gscoup p Zimrrsii 23222
71.02
94.45
90.1
65.32
71 71.88
86
22.94
67.3 36
45.73
45.57
68.21
65.74
PrompiSKC
Kibuel cchi 002
71.27
93.60
90.25
65.70
70.25
86. 15
23 23.90
67.10
46.87
45.50
68.
65.81
MetaPrompt Zillore R024
71.27
93.63
90.25
65.70
70.25
86.
23.90 90
67.10
46.37
45.50
08.75
05.81
DePT Zhupi 21 2014
71.60
93.80
90.13
66.00
70.92
86.27
24.30
67.23
46.60
45.83
69.10
66.02
ProMaox
2014
90.59
86,
47.08
09.
(K
AlPrompe
70.69
94.04
91.03
66.06
71.99
86-33
67:05
45.21
69:15
66.75
CoPrompt
Joyy Mempo 2024
70.80
94.50
90.73
65.87
72.30
86.43
24.00
67.57
47.07
51,90
69.73
67.00
FddMVP
Smnnare Bi 2008
70.87
95.31
89.27
65.85
72.300
87.06
25. 94
68.19
29.77
50.84
718.51
67.57
MMRI
Con X 1ir Farn]
72.03
94.67
91
66. 10
72.77
86 40
26. 30
67.57
45.90
53.10
68 27
67.15
HCCroF
Zinogg & die 2023
70.84
94,48
90.13
65.68
72.03
96.
26, 26.58
08.75
53.19
49.19
70.31
67.68
1PT++
71.81
91.02
92.16
65.55
72.43
86
28.60
08.75
51.02
50,76
70.53
Ours
72.77
95.72
92.73
67.09
73.56
47.92
method is directly compared with the strongest existing prompt learning approaches. Across all datasets, our method consistently achieves the highest harmonic mean, indicating balanced performance on both base and novel classes. On average, surpasses the best-performing baseline (MMRL) by 1.13 points in H. In challenging fine-grained benchmarks such as FGVCAircraft and Flowers 102, our method shows clear compared to the strongest alternative. Overall, these results indicate that our prompt-robust meta- finetuning framework offers modest but consistent gains accuracy, and, together with the reduced sensitivity to prompt phrasing observed in our robustness analysis, leads to more stable base-to-new performance in both coarse- and fine-grained domains.
by 2.79 and 3.81 points respectively
Generalization. To evaluate the robustness of our method under domain shift, we conduct cross-dataset generalization experiments following the standard protocol. As shown in Table our method outperforms all competing approaches across different datasets. These results highlight the ability of our prompt-robust meta-learnnng framework to generalize effectively across unseen distributions and confirm its superiority in cross-domain prompt transfer.
Domain Generalization. We evaluate the domain generalization ability of our method using the standard ImageNet robustness bench- mark, where models trained on Image- Net are tested four shifted do- mains: ImageNet-V2, -S, -A, and -R. our method out-
Table 6: Domain generalization.
Source
Tangr
ImegeNee
-V2
-5
-R
Avg
CLIP
65.73
60.53
46.15
47.77
73.96
57.17
As shown in Table 6
0s000
Co0p
77nn 11 230201
71.51
21.07
64.20
48.75
47:99
50.63
49:11
76.18
15:31
59.28
performs all competing prompt learn- ing methods, achieving the highest av- erage accuracy. Ours yields strong improvements across all domain vari- ants, notably outperforming HPT++ on the more challenging shifts such as ImageNet-A A and RR These results confirm that our prompt-robust design not only improves in-domain
MaP4
70.23
49.15
50.90
76.08
70.69
77.11
CoPromet
KWW Dienad
77.51
HiCm/)
77.15
77.50
ProMeus
51.25
77.50
PedMVv
17.25
60.91
HPT+-
51.18
17.52
Ours
72.77
14:.0
14.55
53.31
(-1.6
78.52
62.15
1.1.5
generalization but also significantly enhances resilience to distributional shifts.
6 CONCLUSION
We presented Promise, a meta-learning framework for prompt-robust vision- languaee modeling that learns to generalize across diverse prompt formulations. Promise combines dual-loop adap- tation scheme with adaptive prompt weighting and token-specific learning rates, enabling fine- grained, context-aware prompt optimization. Beyond empirical gains, our analysis shows that the weighted--reeconttionne inner update induces a single-step decrease of the outer empirical risk while contracting across-prompt sensitivity and tightens data-dependent generalization bound at the post-inner initialization. Experiments on base-to-new new generalization, cross- transfer, and domain-shiff benchmarks corroborate these guarantees, yielding consistent improvements over ffte--of-the-rr prompt tuning methods.
10

---

Published as a conference paper at ICLR 2026
Ethics Statement This work introduces Promise, meta-fiintuniig framework improve prompt robustness of vision-language models. We do not collect new human data or annotate personal information; all experiments and standard cross-daaset/dommin-shii benchmarks). Our method does not attempt to infer demo- graphics, identities, or other sensitive attributes. Nevertheless, adapted models may inherit societal biases present in pretraining/evaluttio data and could be misused for privacy-invasive applications (e.g., surveillance on personal images) or for processing proprictary content without authorization. We discourage such uses and recommend adherence to tata-governance policies, license terms, and applicable laws. We include dataset/usage licenses. prompt lists, and evaluation details support responsible replication; we also report compute profiles to increase transparency about environmental impact.
public datasets under their original licenses (e.g., ImageNet variants
Reproducibility Statement All models, datasets, and baselines used in this paper are publicly accessible. We specify training/validation splits, optimization settings, and evaluation protocols in the main text, with additional implementation details in the appendix. To facilitate replication: (i) we provide the complete prompt templates and puuututed-prompl variants; (ii) we include an algorithmic description with pseudocode and complexity notes; (iii) report hardware, wall-clock training time, and memory usage; and (iv) we add qualitative visualizations to verify robustness behaviors. We release configuration files, random seeds, and checkpoints together with evaluation scripts exactly reproduce the reported numbers.
Acknowledgments Our work was supported by the Research Start-up fund from the Department of Statistics and Data Science, Beijing Normal-Hong Kong Baptist University; the Guangdong Provincial Key Laboratory of IRADS (2022B 121201 0006); the Guangdong Higher Education Upgrading Plan (2021-2025); and the Guangdong and Hong Kong Universities "1+1+1" Joint Research Collaboration Scheme (20550050000004),
REFERENCES
Hyojin Bahng, Ali Jahanian, Swami Sankaranarayanan, and Phillip Isola. Exploring visual prompts for adapting large-scale models. arXiv preprint arXiv:2203.17774 2022.
Lukas Bossard, Matthieu Guillaumin, and Lue Van Gool. Food-101-minin discriminative compo- nents with random forests. In ECCV, pp. 446-461, 2014.
Mircea Cimpoi. Subhransu Maji, lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. De- scribing textures in the wild. In CVPR, pp. 3606-3613, 2014.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In CVPR, pp. 248-255, 2009.
Mohammad Mahdi Derakhshani, Enrique Sanchez, Adrian Bulat, Victor G Turrisi Costa, Cees GM Snoek, Georgios Tzimiropoulos, and Brais Martinez. Bayesian prompt learning for image-language model generalization. In ICCV, pp. 15237-15246, 2023.
Yingjun Du, Wenfang Sun, and Cees G. M. Snoek. IPO: Interpretable prompt optimization for vision-language models. NeurIPS, volume 37, pp. 126725 2024.
Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In CVPR, pp. 178-178, 2004.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agmostic meta-learnig for fast adaptation of deep networks. ICML, pp. 1-26-1135, 2017.
Yuncheng Guo and Xiaodong Gu. MMRL: Multi-modal representation learning for vision-language models. In CVPR, pp. 25015-25025, 2025.
Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. EuroSAT: novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7):2217-2266, 2019.
11

---

Published as a conference paper at ICLR 2026
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: A critical analysis of out-of-distribution generalization. In ICCV, pp. 1440-8349, 2021a.
Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. In CVPR, pp- 15262-15271, 2021b.
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision- -siounlanguage representation learning with noisy text supervision. In ICML, pp. 4904-4916 4916, 2021.
Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In ECCV, pp. 709-727,2022
Muhammad Uzair Khattak, Hanoona Rasheed, Muhammad Maaz, Salman Khan, and Fahad Shahbaz Khan. MaPLe: Multi-modal prompt learning. In CVPR, pp. 19113-19122, 2023a.
Muhammad Uzair Khattak, Syed Talal Wasim, Muzammal Naseer, Salman Khan, Ming-Hsure Yang, and Fahad Shahbaz Khan. Self-regulating prompts: Foundational model adaptation without forgetting. ICCV, pp. 15190-15200, 2023b.
Muhammad Uzair Khattak, Muhammad Ferjad Naeem, Muzammal Naseer, Luc Van Gool, and Federico Tombari. Learning to prompt with text only supervision for vision-language models. In AAAI, pp. 4230-4238, 2025.
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3D object representations for fine-grained categorization. In ICCV, pp. 554-561, 2013.
Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. In EMNLP. pp. 3045-3059, 2021.
Juncheng Li, XIN HE, Longhui Wei, Long Qian, Linchao Zhu, Lingxi Xie, Yueting Zhuang, Qi Tian, and Siliang Tang. Fine-gras Fine-grained semantically aligned vision-langue language pre-training. In NeurIPS, volume 35, pp. 7290-7303, 2022.
Juncheng Li, Minghe Gao, Longhui Wei, Siliang Tang, Wenqiao Zhang, Mengze Li, Wei Ji, Qi Tian, Tat-Seng Chua, and Yueting Zhuang. Grauient-regulated meta-prompt learning for generalizable vision-language models. ICCV, pp. 2551-2562. 2023.
Zheng Li, Xiang Li, Xinyi Fu, Xin Zhang, Weiqiang Wang. Shuo Chen, and Jian Yang. PromptKD: Unsupervised prompt distillation for vision-language models. In CVPR, pp. 26617-26626 2024.
Zheng Li, Yibing Song, Ming-Ming Cheng, Xiang Li, and Jian Yang. Advancing textual prompt learning with anchored attributes. In ICCV, 2025.
Zhenguo Li, Fengwei Zhou, Fei Chen, and Hang Li. Meta-SGD: Learning learn quickly for few-shot learning. arXiv preprint arXiv: 110.:0:777009833 2017.
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained visual classification of aircraft. arXiv preprint 176.51130..515, 2013.
Alex Nichol, Joshua Achiam, and John Schulman. On first-order meta-learning algorithms. arXiv preprint arXiv: arXiv:1803.02999) 2018
Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In ICVGIP, pp. 722-729, 2008.
Jinyoung Park, Juyeon Ko, and Hyunwoo Kim. Prompt learning via meta-regullarization. In CVPR, pp. 20940-2699588 2024.
Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar, Cats and dogs. CVPR, pp. 3498-3505, 2012.
12

---

Published as a conference paper at ICLR 2026
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning transferable visual models from natural language supervision. In ICML, pp. 8748-8763 2021.
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In ICML. pp- 5389-5400, 2019.
Shuvendu Roy and Ali Etemad. Consistency-guid guided prompt learning for vision-language models. In ICLR, 2024.
Mainak Singha, Subhankar Roy, Sarthak Mehrotra, Ankit Jha, Moloud Abdar, Biplab Banerjee, and Elisa Ricci. Fedmvp: Federated multi-modal visual prompt tuning for vision-language models. In ICCV, 2025.
Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. UCFI01: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv: 2012.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-SNE. JMLR, 9(11), 2008.
Haohan Wang. Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global representations by penalizing local predictive power. NeurlPS, volume 32, 2019.
Yubin Wang, Xinyang Jiang, De Cheng, Wenli Sun, Dongsheng Li, and Cairong Zhao. HPT++ Hierarchically prompting vision-language models with multi caauliigggaulaariyy knowledge generation and improved structure modeling. arXiv preprint arXiv:2408. 14812, 2024.
Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. SUN database: Large-scale scene recognition from abbey to zoo. CVPR, pp. 3485-3492, 2010.
Zhang, Shihan Wu, Lianli Gao, Heng Tao Shen, and Jingkuan Song. DePT: Decoupled prompt tuning. In CVPR, pp. 1924-12933, 2024.
Cairong Zhao, Yubin Wang, Xinyang Jiang, Yifei Shen, Kaitao Song, Dongsheng Li, and Duoqian Miao. Learning domain invariant prompt for vision rision-language models. TIP, 33:1348-136 1360, 2024.
Hao Zheng, Shunzhi Yang. Zhuoxin He, Jinfeng Yang, and Zhenhua Huang. Hierarchical cross-modal pompteerrringg for vision-language models. ICCV, 2025.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. In CVPR, pp- 18816-16823 2022a.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning prompt for vision- language models. IJCV, 130(9))2337-2348, 2022b.
13

---

Published as a conference paper at ICLR 2026
A LLM USAGE STATEMENT
We used a large language model (ChatGPT) solely for grammar checking and language polishing of the manuscript text. It did not contribute to research ideation, method design, experiments, data analysis, result generation; all technical content authored and verified by the authors.
B PROOFS AND AUXILIARY RESULTS
Notation. Write the outer empirical risk
R(O)
Rpo (0)
Σ
L(0:T)
Pout
TeP
and the (weighted) inner aggregated gradient
G(Ø) = Σ
WT VDL(B:T).
m(0) E[G(O),,
En == Var[G(O)]]
TEP
The post-inner update is 1 0 PG()) with diagonal P diag(on,..... 0a) 0.
B.1
ASSUMPTIONS
AO: Disjoint and independent draws. Pin and Pout are drawn independently from P and Pin 0 Pout
0.
A1: Smoothness. For every prompt T. L(:T)) is L-smooth; hence R(-) is also L-smooth: [VL(0:T) ; s
-0110-00) 0'1l, [|R((0) VR(O') s L||e 0'|].
A2; Unbiased inner gradient with bounded second moment. B[G((0) u(0) and E|G( (0)ll² Tr(E.) 0²
A3: Preconditioner. P PT ž 0 is diagonal, with spectral bounds 0 s Amin P) - Pl12.
A4: Alignment and bounded amplification. There exist y E (0, 1] and 2 such that
(VR(O), Pp(()) Amia [V)(()))
||x())] I k [|
(5)
A5: Lipschitz properties. (i) (Model Lipschitz for sensitivity) There exists 1 such that [Jo 7" (x: 7' < y |0 0'll for all (xr, T). (ii) (Loss Lipschitz in parameters) There exists Tr. such that [T((0; T) T((O'TT) < Tr ||0 0'| for all T.
A6: Prompt-coherence for sensitivity. Let S(0) = Varr (fo(:: 7)] (or its Jacobian surrogate).
There exist Meff (0, 1] and Co 2 such that
S(0 (P s (1- pott) S(0) Co 1P
(6)
B.2 AUXILIARY LEMMAS
Lemma B.1 (Smoothness descent). For any E Rd, R(0-Pu) < (10))(VR(O)) Pb)+ 21 Pull2.
Proof. By I-smoothness, R(y) < R(x) (VR(x),y 2) + flly 2112. Set and y
0 Pv.
Lemma B.2 (Second moment under preconditioning). B|P)( < ||PI3(1|)0)) Tr(Eww))
Proof. Write = + with E[{] = 0, Cov(E) = En. Then PGI| = |P + PEI | | + | Tr(
14

---

Published as a conference paper at ICLR 2026
B.3 PROOF OF THEOREM 4.1
Recall 0 0=0-
0- - P())). Applying Lemma B.1
with c w= G(0) and taking expectation over Pin.
R(0) - R(0) (VR(O), P Pp())
Σ R(0) ? Amin (P) (+)
+ T |P1| [|((0)])++TT(
< R(0) (7 Ammin (P) - 1 1 |VR())] + 11
where the second line uses Lemma B.2 and the alignment bound in equation 5 and the third line uses [|x(0)] S R|]VR( 7 Amin( (P) PI the expectation strictly decreases.
Sensitivity contraction. Let f = 0 - ( be the noise-free step. By equation 6 S( S (1 Pett (S)) + Coll P u(0)]12. Note that â f PG p.). By the Lipschitz property of So A5(i) and the definition of variance,
ES))) s S())
TEE]P(GG-N] s (1 -
(enr Heit ( + Coll 2 +T
Using Pμ| & |P| R [V(())] and absorbing constants into C. yields F.S(() Σ (1 - Mes ((()) C Tr(E..)-
Remark. If one subsequently takes an outer gradient step A+ â ,)) with 7 (0, Pl(2)) then by L-smoothness, R(0+) I R(0) - v| + [ which further decreases the risk for suitably small 1).
B.4 PROOF OF THEOREM 4.2
Decompose the error into (i) initialization noise from the inner step, and (ii) sampling error from Pout-
Step 1: Initialization noise bound. Write Å P(u + {) with E[g] 0 and Cov(f) Ew-
By A5(ii) (loss Lipschitz in 0) and the triangle inequality,
Ppop (0) Rpop (0 pμ) S Tr TT,
110 - (0 (0- P1)ll = Tr. PE||-
Taking expectation and using Jensen, Rpop (0) S Rpop(0 (f- - Pи) + v / PE|12 S Rpop
Ри) + I'' PT.
Step Concentration on Pout. Conditioned on 0, by AO the samples in Pout are i.i.d. If L E [o, 1]
(or is sub-Gaus Gaossaansinn after scaling), Hoeffding's 's inequality gives, for any d E (0, 1), with probability at least 1 - 5,
In(1/8)
Reop (0) S R(O) + ez
Nout
where nour the number of samples in Rpom. and 02 > 0 an absolute constant. Combining Step and the concentration of R(() around Ppop(() (0), we obtain
In(1/5)
Rpop (0) < R(Ō) + ez
+ e1 I' / PT(
Nout
after absorbing universal constants into e₁ 0. This yields the stated bound.
B.5 COROLLARY AND PRACTICAL DIAGNOSTICS
Corollary B.3 (Improvement over uniform inner update). Let 0.mi - 0 - T.nl ETEr. VoL(O;T).
If there exist (wr, P) such that the alignment and amplification equation 5 holl with y Amia (P) > 1RP1 and Tr(PE,, < Tr(PE,m;P) then E R((0) < E R( (m)))))) and the variance term in Theorem 4.2 is smaller for (WT, P), resulfing in strictly tighter bound.
15

---

Published as a conference paper at ICLR 2026
-1
-1
(a) UCF101 MaPLe
(b) UCFI01 Promise
(c) Caltech101
MMPPee (d) Caltech101 Promise
Figure Visualization of prompt sensitivity and robustness with unseen prompts. Each color denotes a class. Stars are image embeddings; circles are text embeddings from ten unseen but semantically equivalent prompts generated by GPT-4o. MaPLe shows high prompt sensitivity (large star-circle gaps), while our method (Promise) produces consistent, tightly clustered embeddings.
Table 7: Effect of structures prompt Promise.
Base
New
H
MaPLe (Khattak et al.
2023a
82.28
75.14
78.55
Promise (CLIP prompts)
83.57
77.26
80.29
Promise (LLM prompts)
84.21
79.42
81.74
Diagnostics. (i) Alignment. Track cos (VV(()), PG()) an empirical lower bound for 7; (ii)
Variance term. Report TrPPE,,)) with batch-level estimates En across training; (iii) Sensitivity.
Monitor S(0) (APV or Jacobian surrogate) and contraction to validate equation 6
C VISUALIZATION OF PROMPT ROBUSTNESS
To further examine the robustness our method prompt variations, we visualize the feature distributions obtained from different prompt templates using t-SNE (Van der Maaten & Hinton (2008), as shown in F Each class represented by a unique color; stars denote image embeddings, and circles correspond to text embeddings generated from ten semantically equivalent but unseen prompts using GPT-4o.
Compared to MaPLe, which exhibits high sensitivity to prompt variations-evidee spread of circles around each star-our method, Promise, produces tightly clustered embeddings. This indicates that our method yields more stable and consistent text representations across diverse prompt formulations. Such consistency demonstrates Promise's effectiveness in mitigating prompt sensitivity and achieving prompt-agnostic behavior.
by the wide
D EFFECT OF STRUCTURED PROMPTS.
While our main experiments employ handcrafted CLIP-style prompt templates to simulate realistic prompt variation, recent advances suggest that prompts generated by LLMs can provide greater semantic diversity and syntactic richness. To evaluate whether our proposal can benefit from such structured prompt generation, we adopt the IPO framework (Du et al. 2024), which uses LLMs to automatically generate 80 diverse and interpretable prompt templates. We retrain Promise using the IPO-generated prompts, keeping all other experimental settings unchanged. As shown in Table 7 incorporating LLM--generater prompts leads consistent performance improvements across both base and novel classes. These results confirm that Promise not only maintains compatibility with structured prompts but also benefits from their expanded distributional coverage.
E DETAILED PROMPTS
To implement the textual diversity technique, randomly select 60 prompt templates from the full list templates provided in Radford et al.) (202T). Specifically, the following prompt templates are utilized in textual diversity component.
16

---

Published as a conference paper at ICLR 2026
"a photo of (CLASS) ."
"a bad photo of a {CLASS)."
"a photo of many (CLASS)."
"a sculpture of a [CLASS]."
"a photo of the hard to see (CLASS)." "
"a low resolution photo of the (CLASS}." "a rendering of a (CLASS)."
"graffiti of a {CLASS}."
"a bad photo of the (CLASS)."
"a cropped photo of the (CLASS)." "a tattoo of a [CLASS]."
"the embroidered (CLASS)."
"a photo of a hard to see [CLASS]. "
"a bright photo of {CLASS)."
"a photo of a clean (CLASS)."
"a photo of a dirty (CLASS)."
"a dark photo of the (CLASS)." "
"a drawing of a (CLASS)." "
"a photo of my (CLASS}."
"the plastic (CLASS)."
"a photo of the cool (CLASS)." "a close-up photo
a [CLASS]."
"a black and white photo of the (CLASS)." "a painting of the (CLASS]."
"a painting of (CLASS)."
"a pixelated photo of the [CLASS}." "a sculpture of the (CLASS)."
"a bright photo of the [CLASS}."
"a cropped photo of (CLASS)." "a plastic [CIASS]."
"a photo of the dirty (CLASS}."
"a jpeg corrupted photo of a {CLASS}." "" "a blurry photo of the [CLASS]."
"a photo of the (CLASS)."
"a good photo of the (CLASS)."
"a rendering of the (CLASS)."
"a (CLASS)
a video game." "a photo of one [CLASS}."
"a doodle of a [CLASS]."
"a close-up photo of the (CLASS)
"the origami (CLASs)."
"the {CLASS} in a video game."
"a sketch of a {CLASS]."
"a doodle of the (CLASS)." "an origami (CLASS}."
"a low resolution photo of a (CLASS}."
"the toy (CLASS)." "a rendition of the (CLASS)."
"a photo of the clean [CLASS}."
"a photo of a large (CLASS)."
"a rendition of a (CLASS}."
"a photo of a nice [CLASS]."
"a photo of a weird {CLASS)."
"a blurry photo of (CLASS).
"a cartoon (CLASS)."
"art of a (CLASS)."
"a sketch of the (CLASS)."
aa embroidered (CLASS}."
17

---

Published as a conference paper at ICLR 2026
"a pixelated photo of {CLASS)." "itap of the {CLASS}."
F ALGORITHM DESCRIPTION
The Promise algorithm, outlined Algorithm[] I designed improve prompt learning for zero-shot generalization. It employs meta-learning approach with two loops: the inner loop for task-specific adaptation and the outer loop for optimizing meta-paran parammerssssers for generalization across diverse prompts. Key components of the algorithm include: (1) Inner Loop: The inner loop adapts the parameters of the CLIP model using a sub- set of prompts Pmn and data Din- A hypernet- work H dynamically generates learning rates for prompt-specifie updates, enabling efficient optimization. The adapted parameters 9 are ob- tained after several gradient steps. (2) Outer Loop: The outer loop aims improve the gen- eralization capability of the meta- parameters A using a distinct subset of prompts Pout and data Dont The peta-parameters 0, hypernetwork H, and prompt weights [wy} are updated by min- imizing the loss over Pours ensuring robustness across varied prompts. (3) Prompt Weight Ad- justment: To balance the contribution of differ- ent prompts, the weights ]w;} are updated using an exponential normalization mechanism based on the gradient of the outer-loop loss. This it- erative process ensures that Promise learns to generalize across diverse prompt templates, en- hancing the zero-shot capabilities the underly- ing vision-language model. By leveraging first- order approximations in the outer loop, inspired by Reptile, the algorithm avoids computational overhead from second-order gradients, making it more time-efficient.
Algorithm Promise
Input: Dataset D, CI.IP parameters 0 = [0:, 0g Prompt set P== (T,, ....... -ax) ].
o, Require: Inacr-loop data Da. Pa. Outer-loop data Dan, Pan. Hy- pernetwork (-).wehthtt {wi]. Ensure Pa, n Par U. Da, Duu D.
for epoch to do
Inner Loop: Initialize 6 9.
Sample batch {[x, y}} C. Dn, Pi. for k to do
for T, E Pa, do
Compute embedding e, (Tr,T, 6z).
Generate +- H(e.).
Compute Ci = (f(, v)-
Update 0 8 - E. Vo
Outer Loop:
Sample batch {(v, y/} C Dons Pen.
Compute Con 2ij LeE x). v). Update:
0 t 0 3V
H. +- BJ Cunt
We exp(w: 1V Lun)
E, exp(w1s)
Output: Updated 9, hypernetwork H. and weights (wy).
G ADDITIONAL ABLATION STUDIES
G.1 INNER VS. OUTER LOOPS IN PROMISE
To better separate the roles of the inner and outer loops, we compare the base prompt learner, an inner-loop-only variant, outer-loop-only variant, and the full dual-loop Promise. Concretely, the inner-only variant applies a single inner update on P., and is evaluated directly, while the outer-only variant optimizes directly Pau without any inner adaptation step. The full Promise uses both loops as in the main method. Results, averaged over 11 datasets in the base-to-new setting, are summarized in Table g
As shown in Table 8 inner-only and outer-only variants bring only small gains over the base methods, whereas the full dual-loop Promise consistently achieves the best harmonic mean H. In partieular, the inner-only variant tends to overfit the templates in Pm. while the outer-only variant loses the benefit of fast adaptation from shared initialization. The combination of both loops therefore important for robust performance unseen prompts in Pout-
G.2 ADDITIONAL RESULTS ON SIGLIP AND BLIP-2 BACKBONES
To verify that textbfPromise is not restricted to CLIP, we also evaluate it on more recent frozen multimodal encoders, namely SigLIP and BLIP-2, using MaPLe and MMRL as underlying prompt learners in the base-to-new setting. We apply exactly the same meta-finetuning protocol as in the CLIP experiments and report averages over 11 datasets in Table 9 As in the CLIP setting, adding
18

---

Published as a conference paper at ICLR 2026
Table 8: Ablation inner-only, outer-only, and dual-loop variants of Promise top of MaPLe and MMRL in the base-to-new setting (averaged over datasets). Promise consistently achieves the best harmonic mean
Base model
Variant
Base
New
H
w/o Promise
82.28
75.14
78.55
inner-only
82.95
75.55
79.08
MaPLe
outer-only
82.90
75.45
79.00
dual-loop (Ours)
83.57
77.26
80.29
w/o Promise
85.68
77.16
81.20
inner-only
86.05
77.85
81.75
MMRL
outer-only
86.00
77.75
81.67
dual-loop (Ours)
86.14
78.84
82.33
Table 9: Base-to-new results (averaged over 11 datasets) on SigLIP and BLIP-2 backbones with MaPLe and MMRL as base prompt learners. "Base PTT denotes the original prompt-tuning baseline without meta- veta-fincttning. textb/Promise consistently improves the harmonic mean H.
Backbone
Base model
Method
Base
New
H
Base PT Promise
83.10
76.20
79.50
MaPLe
83.90
78.00
80.84
SigLIP
Base PT Promise
86.00
78.30
81.97
MMRL
86.60
79.40
82.84
Base PT
82.10
75.60
78.72
MaPLe
Promise
82.90
76.80
79.73
BLIP-2
Base PT 85.10
77.40
81.07
MMRL
Promise
85.97
78.94
82.30
textbfPromise top of MaPLe and MMRL yields consistent gains in the harmonic mean H on both SigLIP and BLIP-2 backbones.
G.3 COMPONENT- WISE ABLATION AND REGULARIZATION BASELINE
To analyze how each component of Promise contributes to performance and how it compares to simpler regularization-bas baseline, we conduct a component- ablation using MMRL as base prompt learner. We compare: plain MMRL, MMRL with simple variance regularization term across prompts, MMRL with adaptive weighting only, MMRL with token-wise learning rate only, and the full Promise (adaptive weighting + token-wise learning rates). Averaged over 11 datasets in the base-to-nee new setting, the results are summarized in Table T0
The regularization baseline provides only small improvement over plain MMRL, while learning ex- plicit adaptive weights and token-wise preconditioning each yields additional gains on H. Combining both components in Promise gives the most consistent improvement, indicating that the robustness and accuracy gains come from the full combination of adaptive weighting and token-wise learning rates rather than from simple regularization trick.
For completeness, we also consider a randomized outer-loop baseline in which the outer objective replaced by a Gaussian random signal that is independent of the data (the outer update observes random pseudo-loss values). In this case, dual-loop training either matches or slightly degrades the base model, with H typically dropping by about 0.3-0.5 on average. This highlights that the improve- ments of Promise rely on meaningful outer objective and the structured weighing//recondittonnng design, rather than on the mere presence of second optimization loop.
G.4 SENSITIVITY TO INNER/OUTER PROMPT SPLITS
PROMISE samples Pmn and Pan from fixed pool of human-writte templates per dataset. In the main experiments, random 50-50 split with overlap, sampled once per dataset. To assess
19

---

Published as a conference paper at ICLR 2026
Table 10: Component- wise ablation of Promise top of MMRL in the base-to-new setting (averaged 11 datasets). The regularization baseline brings only small improvement, while adaptive weighting and token- wise learning rates each yield additional gains. Combining both components gives the best harmonic mean H.
Variant
Base
New
MMRL (base prompt learner)
85.68
77.16
81.20
variance regularization
85.80
77.50
81.44
adaptive weighting only
86.00
78.00
81.80
token-wise learning rate only
85.95
78.10
81.84
full Promise (weighting LR)
86.14
78.84
82.33
Table 11: Sensitivity of Promise to different choices of inner/outer prompt splits (averaged over 11 datasets in the base-lo-new setting). Different disjoint splits yield very similar performance, while allowing small overlap slightly weakens but does not remove the robustness effect.
Split type
H (f), mean + std Prompt Std (L), mean + std
disjoint, seed
80.31 0.00
1.41 0.00
disjoint, seed 2
80.12 12 = 0.10
1.52 + 0.10
disjoint, seed 3 20% overlap
80.23 = 0.10
1.50 0.10
80.07 0.10
1.61 + 0.10
sensitivity to this choice, we repeat the experiments on a subset datasets with three different random disjoint splits and also consider partially overlapping split that allows 20% shared templates between Pin and Poon Averaged over 11 datasets in the nase-to-new setting, the results are summarized in Table
We observe that different disjoint splits lead to very similar performance, with fluctuations within about 0.2 points H. Allowing overlap slightly weakens the robustness effect (as expected, since the outer loop sees some inner-loop templates), but the degradation is small. Overall, Promise is not highly sensitive the exact split, as long as the inner and outer sets reasonably diverse and there is some non-trivial disjointness.
G.5 TRAINING TIME ANALYSIS
PROMISE meta-finetuning only updates the soft prompt parameters and the small weighting/pre- conditioning modules, while the vision and text encoders remain frozen. At inference time, the computation graph identical the underlying prompt learner, so the runtime and memory cost at deployment do not change. The additional overhead appears only during training. To quantify this cost more systematically, we measure the total wall-clock time to train across all 11 datasets on single NVIDIA A6000 GPU, using the same number of epochs for the base prompt learners (MaPLe MMRL) and for Promise. The results are summarized Table 12
Overall, Promise increases training time by about 18-19% compared to the corresponding single-loop baselines, mainly due to the extra outer-loop gradient and losen-wise preconditioning. Since meta- finetuning is one-time offline stage and inference cost is unchanged, this overhead is acceptable in settings where prompts are reused many times and robustness to prompt wording important.
G.6 EFFECT OF TEMPLATE POOL SIZE AND DIVERSITY
In the main experiments, we use 60 human-designed templates from PromptSRC (Khattak et al.) 2023b), which already cover a range of natural phrasings per dataset. The base MMRL model follows the standard protocol and uses single CLIP-style template, while Promise meta-finetunes over pool of templates. To study how Promise behaves under different degrees of syntactic and semantic diversity, vary the pool size and also add LLM-generater generated paraphrases and more structured prompts. Table reports results averaged over all 11 datasets the base-to-ned new setting, using MMRL as the base learner.
20

---

Published as a conference paper at ICLR 2026
Table 12: Total training time 11 datasets for MaPLe and MMRL with and without Promise. measured on a single A6000 GPU. Promise increases wall-clock time by about 18-19% relative to the corresponding single-loop baselines.
Base model
Method
Total train time (h)
Relative cost
MaPLe
base PT Promise
6.10
1.00x
MaPLe
7.25
1.19x
MMRL
base PT
6.30
1.00x
MMRL
Promise
7.45
1.18x
Table 13: Effect of template pool size and diversity MMRL with Promise in the base-to-new setting (averaged over 11 datasets). The base MMRL model uses a single CLIP-style template, while Promise meta-finetune over pools of 30, 60, 80, or 100 templates (human-designel plus L.M-ggenrrttd variants).
Template pool
Method
Base
New
single CLIP prompt
MMRL (base)
85.68
77.16
81.20
30 templates
+ Promise
85.95
78.20
81.89
60 templates
+ Promise
86.14
78.84
82.33
templates
+ Promise
86.17
78.93
82.39
100 templates
+ Promise
86.10
78.80
82.29
Using relatively small pool (30 templates) leads to slightly weaker gains, likely because the model sees fewer distinct phrasings during meta- meea--inetuning. Once the pool size reaches 60, further increasing to 80 or 100 templates has only a minor effect, and the harmonic mean H remains very similar. Overall, Promise consistently improves effective when prompts become more syntactically and semantically varied, as long as the template pool is reasonably diverse.
the single-template MMRL baseline and remains
21


Reviews:
[{"review_id": "9BAqzMvvcj", "reviewer": "Reviewer_hpup", "summary": "This paper introduces PROMISE, a meta-learning framework designed for prompt-robust vision-language models through meta-finetuning. It achieves performance improvements via dual-loop fine-tuning and provides comprehensive experiments in both in-domain and cross-domain settings.", "questions": "Please refer to the weaknesses section for additional questions and queries!", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ Decent results have been achieved in both in-domain and cross-domain experiments.\n\n+ A dual-loop fine-tuning method has been proposed, achieving enhanced robustness and generalization.", "weaknesses": "+ The ablation experiments in this paper investigate the impact of adaptive prompt weighting but do not explore the effects of standalone Inner-Loop Finetuning or Outer-Loop Finetuning.\n\n+ The current description in the paper gives the impression that the method merely introduces an adaptive prompt weighting mechanism based on MAML, and without this component, it seemingly reduces to the standard MAML approach.\n\n+ In Table 6 of the paper, the best result in the \"-S\" category is actually 50.93% from FepMVP, not the 50.28% reported in this paper. A similar issue appears at line 450 of the paper, where the improvement on the FGVCAircraft dataset should be 2.79% for H rather than the reported 2.34%. There are numerous such issues in the paper, and it is recommended that the authors carefully review them. Additionally, all tables in the paper should highlight the second-best results in a different color, similar to the best results, and include a separate row explicitly stating the percentage improvement to avoid such errors and facilitate better observation. There are minor formatting issues with the tables; table captions should be placed above the tables rather than below.\n\n+ While the proposed method achieves notable improvements on some datasets, the overall enhancement is limited. Moreover, the performance on certain datasets lags significantly behind prior methods, such as on EuroSAT, where it is comprehensively lower than previous approaches.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Oc1IB82DkA", "reviewer": "Reviewer_1bz8", "summary": "This paper presents PROMISE, a meta-finetuning approach that aims to enhance the prompt robustness of vision-language models (VLMs), particularly CLIP-based ones. The work addresses the well-known issue that such models are often sensitive to small variations in natural language prompts. PROMISE introduces a dual-loop meta-learning strategy: the inner loop fine-tunes soft prompt embeddings using one set of prompt templates, while the outer loop updates meta-parameters based on a separate, unseen set. This setup is intended to simulate prompt distribution shifts and encourage cross-template consistency.\n\nIn addition, the framework includes adaptive prompt weighting, which learns to emphasize more generalizable templates, and token-specific adaptive learning rates, inspired by MetaSGD, to refine updates on individual prompt tokens. The paper provides a theoretical argument that the weighted, preconditioned inner updates can reduce empirical risk and improve consistency across prompts.\n\nExperiments on fifteen benchmark datasets show modest but consistent gains over prior prompt-learning methods such as CoOp and CoCoOp, especially under unseen prompt templates. The results indicate improved stability rather than large absolute performance jumps. Overall, the paper is technically sound and carefully executed, though its contribution feels somewhat incremental given the maturity of CLIP-style prompt learning and the field’s current shift toward large multimodal foundation models.", "questions": "Could the authors clarify how sensitive PROMISE is to the choice of the base prompt set used in the inner and outer loops? For example, would the performance change noticeably if both sets were constructed differently or partially overlapped?\n\nHow well does PROMISE generalize to newer or trainable backbones beyond CLIP (e.g., BLIP-2, SigLIP, or LLaVA)? Given that CLIP is becoming less central in recent multimodal research, a discussion or small-scale experiment on this point would help clarify the method’s broader applicability.\n\nThe paper claims improved prompt robustness, but do these benefits persist under other distribution shifts? In other words, is PROMISE’s robustness specific to language variation, or does it extend to multimodal shifts more generally?\n\nRegarding the theoretical analysis, could the authors provide more intuition or empirical evidence that the “risk contraction” property holds beyond the simplified assumptions made in Section 3.3?\n\nHave the authors considered or tested lighter-weight alternatives that might yield similar robustness with lower training cost? It would be helpful to understand the efficiency trade-off.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper targets an identifiable and practically relevant problem — the sensitivity of CLIP-like vision-language models to prompt wording. The motivation is easy to follow, and the setup is consistent with prior work in prompt learning.\n\nThe proposed dual-loop meta-finetuning structure is coherent and well-explained. The integration of adaptive prompt weighting and token-level learning rates forms a complete framework rather than an isolated trick.\n\nEvaluations cover 15 datasets, with results that are consistent and reproducible. The paper also explicitly tests under unseen prompt templates, which directly supports the stated goal of prompt robustness.\n\nThe writing is clear, diagrams are intuitive, and the ablation studies are reasonably thorough. It is easy to reproduce the method and interpret the findings.\n\nAlthough the absolute accuracy improvement is limited, the model demonstrates stable behavior under prompt shifts, which may have practical value in applications relying on frozen CLIP backbones.", "weaknesses": "Conceptually incremental despite a solid formulation.\nThe meta-learning setup is reasonable, but the idea of using a dual-loop structure to optimize different objectives (inner for adaptation, outer for generalization) has already appeared in several prompt-tuning variants. Many recent works also explore meta-learning–based prompt optimization. As a result, the conceptual novelty of PROMISE is somewhat limited, even if the execution is careful.\n\nDependence on an outdated backbone.\nAll experiments are based on frozen CLIP encoders. While CLIP remains a convenient benchmark, it is no longer representative of the current generation of multimodal models. Evaluating PROMISE on stronger or trainable backbones (e.g., SigLIP, BLIP-2, or LLaVA) would make the contribution more convincing and show whether the method still holds under modern architectures.\n\nMarginal quantitative improvement.\nAlthough PROMISE reduces performance variance across prompts, its absolute accuracy gains are small (typically within 1–2%). Considering that the framework adds a meta-finetuning phase, the computational cost may outweigh the benefit in practical settings.\n\nAblation and analysis depth.\nThe paper does include component-wise ablations, but it remains unclear how each part (e.g., adaptive weighting, token-specific learning rate) contributes to robustness. The method could be further analyzed against simpler baselines, such as regularization-based approaches, to isolate where the gain truly comes from.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CCE24LdMhV", "reviewer": "Reviewer_67Ri", "summary": "This paper introduces PROMISE, a meta-learning framework designed to improve prompt robustness in VLMs such as CLIP. \nPROMISE explicitly learns to generalize across diverse prompt formulations through a dual-loop meta-finetuning strategy. \nThe framework integrates two novel components: adaptive prompt weighting, which emphasizes more generalizable templates; and\ntoken-specific adaptive learning rates, providing fine-grained control over token adaptation.\nThe authors provide theoretical guarantees that the weighted, preconditioned inner update decreases the outer empirical risk and contracts across-prompt sensitivity. \nExperiments on 15 benchmarks show consistent improvements over state-of-the-art prompt tuning methods.", "questions": "1. How sensitive is PROMISE to the choice of prompt template pool size (e.g., 30/60/80 templates)?\n2. Which template is used in table 4, CLIP template or LLM template？", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The dual-loop meta-finetuning framework is well-motivated and effectively bridges meta-learning and prompt optimization.\n2. Evaluations covering base-to-novel, cross-dataset, and domain generalization tasks demonstrate consistent performance improvements and robustness.", "weaknesses": "1. Limited analysis of computational cost: while the dual-loop setup is theoretically appealing, its efficiency relative to single-loop baselines could be quantified more precisely. Appendix D mentions training time, but more systematic comparisons would help.\n2. Prompt distribution diversity: The paper uses 60 pre-defined prompt templates from PromptSRC. It would be valuable to analyze whether PROMISE remains effective under more syntactically or semantically diverse prompt distributions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JHEOncf6tu", "reviewer": "Reviewer_BmLg", "summary": "This paper proposes a meta learning framework Promise, which explicitly improves the robustness of VLM to natural language prompt changes through inner and outer dual loop fine-tuning. This method introduces adaptive prompt weights and token level learning rates, significantly reducing prompt sensitivity and stabilizing performance on 15 benchmarks. Theoretical proof of the reliability of its inner loop and tightening of the data dependency generalization boundary.", "questions": "See [Weaknesses].", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. Explore the robustness of VLM to prompt and analyze the shortcomings of existing methods, which rely on prompt templates in finetuning and inference processes.\n\n2. Propose the meta learning framework Promise, which includes two parts: inner and outer loops, to help VLM models adapt to different prompt templates.\n\n3. This paper verifies the reliability of the Promise framework from both experimental and theoretical perspectives.", "weaknesses": "1. The main body of the Promise framework is inner and outer loops, but the motivation behind this design is not clear. Further explanation (or explicit display) is needed on which issues correspond to the inner and outer loop design in the process of VLM adapting to different prompt templates.\n\n2. The improvement brought by the Promise framework is limited, with most indicators improving by 1 to 2 points, and some indicators not exceeding state-of-the-art model accuracy.\n\n3. Further research is needed on ablation of inner and outer circulation, such as (1) comparing the accuracy of the original model with the accuracy after removing adaptive prompt weighting and adaptive learning rate; (2) Simplify the outer loop to the overall accuracy of Gaussian distributed random numbers.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)