Title: VaMP: Variational Multi-Modal Prompt Learning for Vision-Language Models
Venue: NeurIPS 2025
Accepted: yes
Keywords: Prompt Learning; Vision-Language Models; Multi-Modal Representation Learning
Abstract: Vision-language models (VLMs), such as CLIP, have shown strong generalization under zero-shot settings, yet adapting them to downstream tasks with limited supervision remains a significant challenge. Existing multi-modal prompt learning methods typically rely on fixed, shared prompts and deterministic parameters, which limits their ability to capture instance-level variation or model uncertainty across diverse tasks and domains. To tackle this issue, we propose a novel Variational Multi-Modal Prompt Learning (VaMP) framework that enables sample-specific, uncertainty-aware prompt tuning in multi-modal representation learning. VaMP generates instance-conditioned prompts by sampling from a learned posterior distribution, allowing the model to personalize its behavior based on input content.
To further enhance the integration of local and global semantics, we introduce a class-aware prior derived from the instance representation and class prototype. Building upon these, we formulate prompt tuning as variational inference over latent prompt representations and train the entire framework end-to-end through reparameterized sampling. Experiments on few-shot and domain generalization benchmarks show that VaMP achieves state-of-the-art performance, highlighting the benefits of modeling both uncertainty and task structure in our method. Project page: https://visual-ai.github.io/vamp

VaMP: Variational Multi-Modal Prompt Learning for Vision-Language Models

Abstract

Vision-language models (VLMs), such as CLIP, have shown strong generalization under zero-shot settings, yet adapting them to downstream tasks with limited supervision remains a significant challenge. Existing multi-modal prompt learning methods typically rely on fixed, shared prompts and deterministic parameters, which limits their ability to capture instance-level variation or model uncertainty across diverse tasks and domains. To tackle this issue, we propose a novel Va riational M ulti-Modal P rompt Learning (VaMP) framework that enables sample-specific, uncertainty-aware prompt tuning in multi-modal representation learning. VaMP generates instance-conditioned prompts by sampling from a learned posterior distribution, allowing the model to personalize its behavior based on input content. To further enhance the integration of local and global semantics, we introduce a class-aware prior derived from the instance representation and class prototype. Building upon these, we formulate prompt tuning as variational inference over latent prompt representations and train the entire framework end-to-end through reparameterized sampling. Experiments on few-shot and domain generalization benchmarks show that VaMP achieves state-of-the-art performance, highlighting the benefits of modeling both uncertainty and task structure in our method. Project page: https://visual-ai.github.io/vamp

1 Introduction

Vision-Language Models (VLMs), such as CLIP [1], have achieved impressive performance across a wide range of visual recognition tasks through multi-modal representation learning. Their ability to align images and texts in a shared embedding space enables strong zero-shot transfer. However, their large-scale parameters and the scarcity of training data, particularly in few-shot settings, make full model fine-tuning computationally expensive and prone to overfitting on downstream tasks.

To address this, prompt learning has emerged as a parameter-efficient alternative, where a small number of learnable tokens are prepended to the input to steer the frozen model toward task-specific behavior [2, 3, 4, 5, 6, 7, 8]. While effective, existing multi-modal prompt tuning methods typically rely on fixed, shared prompts that are applied uniformly across all samples. Such methods are inherently deterministic and lack the flexibility to adapt to instance-level variations or model uncertainty, limiting their generalization to unseen tasks and domains [9, 10].

While recent work has explored uncertainty-aware prompt tuning, most existing approaches remain limited in scope. Bayesian Prompt Learning introduces uncertainty modeling in text-only prompts, and Any-Shift Prompting leverages variational inference to enhance robustness across distribution shifts. However, these methods suffer from key limitations. First, by restricting variational modeling to input-level prompts with global latent variables, they fail to capture hierarchical feature interactions

or fine-grained, token-level semantic variations. Second, their text-only prompt tuning overlooks valuable visual information that could enhance cross-modal alignment. Finally, the standard Gaussian prior, shared across all classes, fails to capture inter-class variations, resulting in less discriminative prompt distributions. Consequently, these models are inadequate for capturing fine-grained, inputspecific variations, particularly in vision-language tasks where precise alignment is critical.

To overcome these limitations, we introduce a novel Va riational M ulti-Modal P rompt Learning (VaMP) framework that enables sample-specific, uncertainty-aware prompt tuning for vision-language models. We make three key contributions: First, we introduce token-wise variational modeling across multiple intermediate network layers. This approach treats individual prompt tokens as latent variables, enabling the model to capture fine-grained semantic relationships at multiple abstraction levels and improve generalization in low-data and out-of-distribution scenarios. Second, our multimodal design incorporates both visual and textual signals when inferring posterior distributions, creating more aligned cross-modal representations. Third, by employing class-aware priors instead of standard Gaussian distributions, VaMP generates more discriminative prompts that better capture category-specific features and decision boundaries.

We evaluate VaMP on three challenging adaptation settings: base-to-new generalization, domain generalization and cross-dataset generalization. Our method consistently outperforms strong multimodal prompt baselines while maintaining high parameter efficiency. Ablation studies further confirm the effectiveness of each component, including the variational modeling and task-aware prior.

2 Related Work

Pre-trained Vision-Language Models. Pre-trained vision-language models (VLMs) [1, 11, 12, 13] have gained significant attention for their strong performance across diverse vision-language tasks. These models typically follow four training paradigms: 1) masked language modeling [14, 15]; 2) masked region prediction [16, 17]; 3) image-text matching [16, 14]; and 4) contrastive learning [1, 11, 18, 19]. While VLMs provide robust, generalized representations, adapting them to downstream tasks remains challenging. Recent studies show that tailored approaches significantly improve performance in specific domains, such as few-shot image recognition [20, 21], object detection [22, 23, 24, 25, 26, 27], semantic segmentation [28, 29, 30, 31, 32] and visual grounding [33, 34]. In this work, we focus on adapting vision-language models for few-shot and zero-shot visual recognition tasks.

Prompt Tuning. Instructions provided to language models as text prompts enable task-specific understanding and performance in VLMs. These prompts, either manually designed or automatically optimized through "Prompt Learning" (originally from NLP [35, 36, 37]), have been adapted for computer vision in three primary forms: textual prompt leanring [2, 3, 38, 39, 40, 41, 42, 43, 44, 45, 46] that fine-tune CLIP's [1] by optimizing continuous prompt vectors in its language branch; visual prompt learning [4, 47, 48, 49, 50] that optimize task-specific learnable inputs in the visual input space while keeping pre-trained backbones frozen; and multi-modal prompt learning [5, 51, 52, 53, 54, 55, 6] that enhance alignment by applying prompts to both vision and language branches. Our work advances this research by introducing a variational framework for multi-modal prompt tuning, enabling samplespecific, uncertainty-aware prompt tuning with structured guidance from both visual inputs and class-level semantics.

Variational Inference. Variational inference has been widely applied to computer vision tasks, such as image generation [56, 57, 58, 59], action recognition [60], instance segmentation [61], anomaly detection [62], depth estimation [63], few-shot learning [64, 65, 66, 67], and domain generalization [68, 69]. Recently, variational inference has been applied to prompt learning to mitigate overfitting in low-shot settings and improve generalization. For example, Bayesian Prompt Learning [9] captures uncertainty by sampling prompts from learned distributions, while Any-Shift Prompting [10] uses a hierarchical probabilistic framework to model distribution shifts and generate adaptive prompts without test-time optimization. However, both methods are limited to the text modality and rely on globally shared prompts, which restrict their ability to capture fine-grained, input-specific variations. To overcome these limitations, we propose a probabilistic framework that combines multi-modal prompt tuning with variational modeling and sample-specific adaptation.

3 Preliminary

3.1 Revisiting CLIP

Our work builds upon the pre-trained vision-language model, CLIP [1], which comprises both a text encoder and a vision encoder. Following previous prompt-learning methods [2, 3, 5, 8], we adopt a ViT-based CLIP model that encodes both images I ∈ R H × W × 3 and text descriptions.

Encoding Image. The image encoder V consists of K transformer layers, denoted as { V i } K -1 i =0 . It first divides the input image I into B non-overlapping patches. These patches are then projected into embeddings e 0 ∈ R B × d v , where d v represents the embedding dimension. Subsequently, patch embeddings, along with a class token c i , are sequentially processed through the transformer blocks:

The final image representation x is obtained by applying a linear projection to the last layer's class token, mapping it into the shared vision-language embedding space:

Encoding Text. The text encoder converts tokenized words into embeddings w 0 = [ w 1 0 , w 2 0 , · · · , w N 0 ] ∈ R N × d l , which are processed through K transformer layers:

Similarly, the text representation t is obtained through a linear projection of the final embedding of the last token:

Zero-shot Classification. For classification, hand-crafted text prompts ( e.g. , ' a photo of a <category> ') with class labels y ∈ { 1 , 2 , . . . C } are used. The prediction ˆ y for image I is determined by the highest cosine similarity:

where τ is the temperature coefficient.

3.2 Multi-Modal Prompt Learning

Multi-modal prompt learning methods [5, 70, 8] extend text-only prompt learning approaches [2, 3] by jointly tuning the text and image prompts to achieve improved alignment for downstream tasks. These methods typically specify H consecutive transformer layers, beginning from the J -th layer, for prompt tuning, while leaving the remaining transformer layers fixed (where J ≥ 0 and H < K -J +1 ). For example, MaPLe [5] focuses on prompt tuning the shallow layers ( J = 0 , H = 9 ), whereas MMRL[8] applies prompt tuning to deeper layers ( J = 5 , H = 7 ).

Deep Language Prompting. In the text branch, we introduce learnable prompts z i ∈ R M × d l , consisting of M tokens, into the i -th transformer layer for prompt tuning. For each layer i from J to J + H -1 , these tokens are concatenated with the original token embeddings and fed into the i -th transformer layer to generate inputs for the next layer:

while layers outside this range (from 0 to J -1 and from J + H to K -1 ) remain unchanged:

Finally, the text representation is derived via Eq. 4.

Meanwhile, layers outside this range (from 0 to J -1 and from J + H to K -1 ) remain unchanged:

The final text representation is obtained through Eq. 4.

...

...

...

...

...

...

Word Embedding

...

...

...

Image

Encoder

Inference

Network

Inference

Network

(a) Class-Aware Prior Construction

...

Cat

Dog

Bird

(b) Variational Multi-Modal Prompt Adaptation (VMPA)

VMPA

VMPA

...

...

Image

Encoder

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

(c) Training pipeline

Prototype embedding

Instance embedding

...

...

...

...

...

Sampling

...

"A photo of a

cat"

...

...

Deep Vision Prompting. In the visual branch, M learnable tokens ˜ z i ∈ R M × d v are inserted into the i -th transformer layer for prompt tuning. The exact computation of these visual tokens depends on the specific multi-modal prompt learning method being used. For instance, in MaPLe [5], these visual tokens are generated from the language prompts via a linear transformation implemented with an MLP. In contrast, MMRL [8] obtains visual tokens from a shared latent space-a set of learnable tokens initialized by sampling from a Gaussian distribution-and then uses separate linear projection functions (also implemented with MLPs) to generate modality-specific prompts.

For each layer i from J to J + H -1 , these generated tokens are concatenated with the original patch token embeddings and fed into the i -th transformer layer to produce inputs for the next layer.

For the remaining layers, the original operation without prompts is preserved:

The final image representation follows the same projection process outlined in Eq. 2. During inference, predictions follow standard classification procedures based on similarity.

4 Method

We propose VaMP-a variational multi-modal prompt learning framework that enables samplespecific, uncertainty-aware, and structured tuning within vision-language models. Our method

consists of three key components: (i) sample-specific multi-modal prompt generation , where imageconditioned prompts are injected across multiple transformer layers; (ii) variational prompt adaptation for multi-modal representation learning , which models the text-side prompts as latent variables to capture instance-level uncertainty; and (iii) class-aware prior construction , which regularizes the latent space using semantic information from both the input instance and its class prototype. An overview of the full framework is illustrated in Figure 1, which highlights the generation of text prompts from image features, the variational posterior, and the class-conditioned prior used for regularization.

4.1 Sample-specific Multi-Modal Prompt Generation

In multi-modal prompt learning, prior work typically learns a fixed set of prompt tokens shared across all input samples [5, 8]. While effective, such fixed prompts cannot adapt to instance-specific variations, which are crucial for robust downstream performance under distribution shifts.

To overcome this limitation, we propose sample-specific prompt generation , where prompts in the text encoder are dynamically generated based on the input image. Unlike previous methods that insert fixed prompts at a single layer, we generate and inject prompts across multiple transformer layers, enabling hierarchical and fine-grained modulation of the language stream.

Specifically, given an input image x , we first extract its global visual representation f x using the frozen CLIP image encoder. To generate text-side prompts, we define a set of H layer-specific prompt generators { Φ i } J + H -1 i = J , where each Φ i is a lightweight MLP that maps the image feature to a set of M prompt tokens for the i -th transformer layer:

where d is the token embedding dimension. Although all Φ i share the same architecture, they have independent parameters to allow for layer-specific adaptation.

The resulting prompts z i are concatenated with the frozen text token embeddings W i at each layer i of the CLIP text encoder, as formalized in Eq. 8, following the multi-layer prompt insertion strategy used in MaPLe [5] and MMRL [8].

In parallel, we also adopt deep vision prompting in the CLIP image encoder, where a set of learnable vision-side prompt tokens ˜ z i -1 ∈ R M × d is inserted at each selected transformer layer i . These vision prompts are shared across samples and optimized independently from the input x . They are not generated dynamically, nor modeled as latent variables.

In summary, our method introduces dynamic, sample-specific text prompts z i conditioned on image features and static, shared vision prompts ˜ z i , aligning both modalities through a structured and hierarchical prompting design. Only the text-side prompts z i are modeled probabilistically in our variational framework.

4.2 Variational Multi-Modal Prompt Adaptation

While our sample-specific prompts z i increase flexibility, they remain deterministic and lack the ability to capture uncertainty-a key factor in few-shot and distribution-shift settings. To address this, we formulate prompt adaptation as a probabilistic latent variable model, replacing deterministic prompt tokens with latent prompt tokens learned via variational inference.

For each input image x and fixed text template t , we define z = { z i } J + H -1 i = J , where each z i ∈ R M × d is a latent variable representing the prompt tokens inserted at layer i of the text encoder. To model the posterior distribution over these latent variables, we introduce a set of layer-specific MLPs { ϕ i } J + H -1 i = J , where each ϕ i predicts the parameters of a Gaussian distribution:

where f x is the frozen CLIP image embedding. Using these predicted parameters, the posterior distribution is formulated as a product of layer-wise Gaussians, conditioned solely on the image x :

Given label y , we aim to maximize the marginal likelihood:

As the integral is intractable, we maximize the variational evidence lower bound (ELBO):

Here, p ( z ) represents the prior distribution over the latent prompts, which we initially set to a standard Gaussian N ( 0 , I ) . In implementation, we apply the reparameterization trick [71] to enable gradient-based optimization:

Each sampled z i replaces the deterministic z i , and is concatenated with the frozen tokens w i to form the layer input, as formalized in Eq. 8. This variational formulation introduces a structured, uncertainty-aware distribution over prompts and allows the model to adaptively control prompt behavior per input.

4.3 Class-Aware Prior Construction

In variational inference, the prior distribution p ( z ) serves as a crucial regularizer for the learned posterior q ϕ ( z | x ) . While a standard choice p ( z ) = N ( 0 , I ) provides a generic reference, it lacks semantic structure and offers no task-specific guidance. To incorporate class-level semantics into the latent prompt space, we introduce a class-aware prior that conditions on a prototype representation for each class.

During training, we assume access to the class label y for each sample. We compute a class prototype o y by averaging the posterior means of training samples in class y :

where D y is the set of training instances labeled y .

To model layer-wise latent prompts z = { z i } J + H -1 i = J , we introduce a set of layer-specific prior networks { ψ i } J + H -1 i = J . Each prior network maps the class prototype c y to the parameters of a Gaussian prior at layer i :

The resulting ELBO objective now sums across layers:

This class-aware prior construction provides global semantic anchoring for each layer's prompt distribution. It encourages prompts from the same class to lie in nearby regions of the latent space, improving intra-class consistency and few-shot generalization. At test time, when y is unavailable, we revert to the standard prior p ( z i ) = N ( 0 , I ) for all layers.

4.4 Inference Procedure

During inference, our method follows a single forward pass through the VaMP framework. Given an input image x and a fixed text template t ( e.g. , ' A photo of a [CLASS] '), we first extract the frozen CLIP image feature f x .

For probabilistic inference, we leverage Monte Carlo sampling over the latent prompt distribution. Specifically, given input image x , we draw S samples { z i,s } S s =1 from the learned posterior q ϕ ( z i | x ) for each layer i :

Each sampled latent prompt z i,s is concatenated with the frozen text tokens w i at each transformer layer. Simultaneously, a set of shared vision-side prompts { ˜ z i } J + H -1 i = J is injected into the corresponding layers of the CLIP image encoder. The model then computes the image and text features using the frozen CLIP encoders with inserted prompts, and generates a prediction p s ( y | x, t, { z i,s } J + H -1 i = J ) based on the similarity between the projected image and text representations.

Finally, the predictions are averaged across samples to form the final output:

In our experiments, we use S = 10 samples for all evaluations. This inference-time ensembling preserves the expressiveness of the variational model while improving robustness and stability.

5 Experiments

5.1 Experiments Setup

We evaluate the performance of VaMP under three different settings: base-to-novel generalization, cross-dataset evaluation, domain generalization, and few-shot learning. All conducted under a 16-shot setting, where each category has only 16 training examples.

Base-to-Novel Generalization. In this setting, dataset classes are split into base and novel classes. The model is trained exclusively on base classes and tested on both base and novel classes, enabling an assessment of its transfer learning performance on base classes and its ability to preserve the inherent generalization and zero-shot capabilities of pre-trained VLMs for novel classes. This evaluation is conducted across 11 diverse classification datasets: ImageNet [72], Caltech101 [73], OxfordPets [74], StanfordCars [75], Flowers102 [76], Food101 [77], FGVCAircraft [78], SUN397 [79], UCF101 [80], DTD [81], and EuroSAT [82].

Cross-Dataset Evaluation. This evaluation examines the model's ability to generalize to unseen datasets. Following CoCoOp [3], the model is trained on all 1000 ImageNet classes in a few-shot setting and directly tested, without further fine-tuning, on the same datasets used for base-to-novel generalization, allowing us to assess cross-dataset transferability.

Domain Generalization. To evaluate the model's robustness to domain shifts and its generalization to out-of-distribution data, we train it on ImageNet and test it on four domain-variant datasets: ImageNetV2 [83], ImageNet-Sketch [84], ImageNet-A [85], and ImageNet-R [86], each introducing distinct types of domain variation.

Implementation Details. We follow prior studies [5, 8] and adopt a 16-shot training setting for all experiments unless otherwise noted. We build on the ViT-B/16 variant of CLIP [89] as the visual backbone and apply multi-layer prompt tuning on the text and vision encoders starting from the J -th transformer layer. For MMRL-style settings, we set J = 5 , H = 7 and insert M = 5 learnable representation tokens per layer. For MaPLe-style configurations, we adopt J = 0 , H = 9 with prompt length M = 2 for both modalities. All experiments are conducted on a single NVIDIA V100 GPU.

5.2 Main Results

Base-to-Novel Generalization. We evaluate VaMP against recent prompt tuning methods across 11 diverse datasets under the base-to-new generalization protocol. As shown in Table 1, VaMP achieves competitive performance on base classes while consistently outperforming all baselines on novel classes. In particular, VaMP attains the highest average novel accuracy of 78.67%, outperforming the best previous method (MMRL) by 1.51% and demonstrating strong generalization to unseen categories. The advantage is especially pronounced on challenging datasets with large domain shifts. For example, on DTD-a dataset characterized by fine-grained textures rather than semantic categories-VaMP achieves a novel accuracy of 75.50%, surpassing MMRL by 2.67%. These results highlight the strength of our structured, variational modeling in adapting to unfamiliar domains without sacrificing base class performance.

Domain Generalization. We further evaluate VaMP in a domain generalization setting,

where prompts are optimized on ImageNet and directly applied to its variants (-V2, -S, -A, -R) without any target supervision. As shown in Table 2, VaMP achieves the highest accuracy across all four target domains, with an average accuracy of 61.73%, outperforming the best baseline (MMRL) by 1.20%. The gain is particularly significant on Sketch, a challenging domain due to its abstract and textureless nature. On this subset, VaMP achieves 49.69%, improving over MMRL by 0.52%. These results vali-

Source

Target

ImageNet

-V2

-S

-A

-R

CLIP

[

1

]

66.73

60.83

46.15

47.77

73.96

CoOp

[

2

]

71.51

64.20

47.99

49.71

75.21

CoOpOp

[

3

]

71.02

64.07

48.75

50.63

76.18

MaPLe

[

5

]

70.72

64.07

49.15

50.90

76.98

PromptSRC

[

6

]

71.27

64.35

49.55

50.90

77.80

MMA

[

70

]

71.00

64.33

49.13

51.12

77.32

MMRL

[

8

]

72.03

64.47

49.17

51.20

77.53

72.83

64.96

49.69

51.97

78.01

date the effectiveness of our variational prompt modeling and class-aware regularization in enhancing out-of-distribution generalization, even in low-texture, cross-modality scenarios.

Cross-Dataset Generalization. To assess robustness under domain shift, we evaluate VaMP on cross-dataset transfer, where prompts are tuned on one source dataset (ImageNet) and evaluated directly on unseen target datasets. As shown in Table 3, VaMP achieves the best average performance across 10 diverse target datasets, with an average accuracy of 67.74%, outperforming the strongest baseline (MMRL) by 0.49%. The improvement is particularly notable on challenging datasets with large domain gaps. For example, on EuroSAT-a remote sensing dataset with significant visual discrepancy from natural images-VaMP achieves a target accuracy of 53.82%, improving over

Source

Target

ImageNet

Average

Caltech101

OxfordPets

StanfordCars

Flowers101

Food101

FGVCAircraft

SUN397

DTD

EuroSAT

UCF101

CoOp

[

2

]

71.51

63.88

93.70

89.14

64.51

68.71

85.30

18.47

64.15

41.92

46.39

66.55

CoOpOp

[

3

]

71.02

65.74

94.43

90.14

65.32

71.88

86.06

22.94

67.36

45.73

45.37

68.21

MaPLe

[

5

]

70.72

66.30

93.53

90.49

65.57

72.23

86.20

24.74

67.01

46.49

48.06

68.69

PromptSRC

[

6

]

71.27

65.81

93.60

90.25

65.70

70.25

86.15

23.90

67.10

46.87

45.50

68.75

TCP

[

42

]

71.40

66.29

93.97

91.25

64.69

71.21

86.69

23.45

67.15

44.35

51.45

68.73

MMA

[

70

]

71.00

66.61

93.80

90.30

66.13

72.07

86.12

25.33

68.17

46.57

49.24

68.32

MMRL

[

8

]

72.03

67.25

94.67

91.43

66.10

72.77

86.40

26.30

67.57

45.90

53.10

68.27

72.83

67.74

94.96

91.79

66.10

73.18

86.97

26.76

68.04

46.82

53.82

68.93

MMRLby 0.72%. These results demonstrate that our structured and uncertainty-aware adaptation generalizes well across domains, even when the target distribution differs substantially from the source.

5.3 Ablation Study

Effects of Sample-specific Multi-Modal Prompt Generation. We begin by assessing the effect of sample-specific multi-modal prompt generation. As shown in Table 4a, previous methods such as MaPLe and MMRL use task-specific prompts shared across all inputs. In contrast, our samplespecific design generates prompts conditioned on each input instance and injected across multiple layers. This structured, instance-aware formulation consistently improves generalization to novel categories, demonstrating the advantage of personalized multi-modal adaptation over fixed prompt configurations.

Effects of Variational Prompt Adaptation. Next, we evaluate the effect of variational modeling on the prompt tokens. While prior work adopts deterministic prompt embeddings, our approach treats the text-side prompts as latent variables inferred per instance. This formulation introduces uncertainty modeling into the prompt space, enabling better adaptation to ambiguous or out-of-distribution inputs. As shown in Table 4b, applying variational prompt learning improves generalization across both MaPLe and MMRL backbones, validating the benefit of latent, sample-specific modeling.

Effects of Class-Aware Prior. We further assess the role of structured priors in regularizing the latent prompt space. Unlike standard variational methods that use an isotropic Gaussian prior, our method constructs a class-aware prior from prototype representations computed over training data. This provides global semantic guidance for latent prompt inference. As shown in Table 4c, replacing the normal prior with a class-aware one consistently improves performance, highlighting the importance of semantic regularization for class-consistent prompt adaptation.

(a) ImageNet

(b) Flowers102

Layer 6

Layer 9

Layer 12

Layer 6

Layer 9

Layer 12

Golden Retriever

Persian Cat

African Elephant

Alaskan Malamute

sunflower

5.4 Qualitative Analysis

To gain intuitive insights into the learned latent prompt space, we visualize the posterior distributions across text encoder layers. Figure 2 illustrates this process for representative samples from ImageNet and Oxford Flowers datasets. For each input image x , we extract the image-conditioned posterior q ϕ ( z i | x ) at layers 6, 9, and 12. Since VaMP generates M prompt tokens per layer, we aggregate their means and covariances as µ agg ,i ( x ) = 1 M ∑ M j =1 µ j i ( x ) and Σ agg ,i ( x ) = 1 M ∑ M j =1 Σ j i ( x ) , then project these high-dimensional vectors to 2D using PCA. Contours represent uncertainty regions at 1.5 σ and 2.5 σ standard deviations.

These visualizations provide insights into how VaMP achieves uncertainty-aware prompt learning through its variational design. First, the varying distributional characteristics across layers and samples-reflected in the spatial positions, orientations, and covariance structures of the posterior distributions-demonstrate that VaMP successfully models input-dependent uncertainty through its variational framework, moving beyond deterministic point estimates. Second, the layer-wise evolution reveals hierarchical uncertainty refinement: early layers (Layer 6) exhibit broader posterior distributions with larger variance to accommodate diverse prompt adaptations, while deeper layers (Layer 12) manifest tighter, more concentrated distributions with reduced uncertainty reflecting task-specific specialization. This progressive variance reduction enables adaptive representation learning at different semantic abstraction levels. Third, the consistent inter-class separation of posterior distributions across all depths demonstrates the effectiveness of the class-aware prior p θ ( z | c ) , which regularizes the variational posterior toward discriminative regions of the latent space. This class-conditional guidance ensures that prompts encode category-specific information from early feature extraction stages. Finally, VaMP's stochastic sampling mechanism, manifested through the distributional spread and overlap patterns, provides inherent robustness against overfitting to singular prompt configurations. By parameterizing prompts as probability distributions rather than fixed embeddings, VaMP balances representation flexibility with discriminative capacity throughout the hierarchical architecture.

6 Conclusion

We presented VaMP, a variational framework for prompt adaptation in multi-modal representation learning. Our approach addresses the limitations of fixed, shared prompts by introducing a structured and uncertainty-aware mechanism that adapts to individual input instances. VaMP comprises three key components: ( i ) sample-specific multi-modal prompt generation, where visual features condition prompt tokens across multiple transformer layers; ( ii ) variational modeling of text-side prompts as latent variables, enabling instance-specific and probabilistic adaptation; and ( iii ) a class-aware prior constructed from semantic prototypes, which regularizes the latent space with global class-level information. Through extensive experiments on few-shot and domain generalization benchmarks, we demonstrate that VaMP achieves state-of-the-art performance while maintaining high parameter efficiency. Our findings highlight the importance of modeling both instance-level variability and task structure in prompt-based adaptation for vision-language models.

References

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, and Jack Clark. Learning transferable visual models from natural language supervision. In ICML , 2021.

Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. IJCV , 2022.

Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. In CVPR , 2022.

Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In ECCV , 2022.

Muhammad Uzair Khattak, Hanoona Rasheed, Muhammad Maaz, Salman Khan, and Fahad Shahbaz Khan. Maple: Multi-modal prompt learning. In CVPR , 2023.

Muhammad Uzair Khattak, Syed Talal Wasim, Muzammal Naseer, Salman Khan, Ming-Hsuan Yang, and Fahad Shahbaz Khan. Self-regulating prompts: Foundational model adaptation without forgetting. In ICCV , 2023.

Yingjun Du, Gaowen Liu, Yuzhang Shang, Yuguang Yao, Ramana Kompella, and Cees GM Snoek. Prompt diffusion robustifies any-modality prompt learning. arXiv preprint arXiv:2410.20164 , 2024.

Yuncheng Guo and Xiaodong Gu. Mmrl: Multi-modal representation learning for visionlanguage models. In CVPR , 2025.

Mohammad Mahdi Derakhshani, Enrique Sanchez, Adrian Bulat, Victor G Turrisi da Costa, Cees GM Snoek, Georgios Tzimiropoulos, and Brais Martinez. Bayesian prompt learning for image-language model generalization. In ICCV , 2023.

Zehao Xiao, Jiayi Shen, Mohammad Mahdi Derakhshani, Shengcai Liao, and Cees GM Snoek. Any-shift prompting for generalization over distributions. In CVPR , 2024.

Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In ICML , 2021.

Lewei Yao, Runhui Huang, Lu Hou, Guansong Lu, Minzhe Niu, Hang Xu, Xiaodan Liang, Zhenguo Li, Xin Jiang, and Chunjing Xu. Filip: Fine-grained interactive language-image pre-training. In ICLR , 2022.

Juncheng Li, Xin He, Longhui Wei, Long Qian, Linchao Zhu, Lingxi Xie, Yueting Zhuang, Qi Tian, and Siliang Tang. Fine-grained semantically aligned vision-language pre-training. In NeurIPS , 2022.

Wonjae Kim, Bokyung Son, and Ildoo Kim. Vilt: Vision-and-language transformer without convolution or region supervision. In ICML , 2021.

Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. In NeurIPS , 2019.

Hao Tan and Mohit Bansal. Lxmert: Learning cross-modality encoder representations from transformers. In EMNLP-IJCNLP , 2019.

Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. Vl-bert: Pre-training of generic visual-linguistic representations. In ICLR , 2019.

Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. Align before fuse: Vision and language representation learning with momentum distillation. In NeurIPS , 2021.

Yuqi Huo, Manli Zhang, Guangzhen Liu, Haoyu Lu, Yizhao Gao, Guoxing Yang, Jingyuan Wen, Heng Zhang, Baogui Xu, and Weihao Zheng. Wenlan: Bridging vision and language by large-scale multi-modal pre-training. arXiv preprint arXiv:2103.06561 , 2021.

Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma, Rongyao Fang, Yongfeng Zhang, Hongsheng Li, and Yu Qiao. Clip-adapter: Better vision-language models with feature adapters. IJCV , 2024.

Renrui Zhang, Rongyao Fang, Peng Gao, Wei Zhang, Kunchang Li, Jifeng Dai, Yu Qiao, and Hongsheng Li. Tip-adapter: Training-free clip-adapter for better vision-language modeling. In ECCV , 2022.

Jiaming Li, Jiacheng Zhang, Jichang Li, Ge Li, Si Liu, Liang Lin, and Guanbin Li. Learning background prompts to discover implicit knowledge for open vocabulary object detection. In CVPR , 2024.

Muhammad Maaz, Hanoona Rasheed, Salman Khan, Fahad Shahbaz Khan, Rao Muhammad Anwer, and Ming-Hsuan Yang. Class-agnostic object detection with multi-modal transformer. In ECCV , 2022.

Xingyi Zhou, Rohit Girdhar, Armand Joulin, Philipp Krähenbühl, and Ishan Misra. Detecting twenty-thousand classes using image-level supervision. In ECCV , 2022.

Xiuye Gu, Tsung-Yi Lin, Weicheng Kuo, and Yin Cui. Open-vocabulary object detection via vision and language knowledge distillation. In ICLR , 2022.

Yuhang Zang, Wei Li, Kaiyang Zhou, Chen Huang, and Chen Change Loy. Open-vocabulary detr with conditional matching. In ECCV , 2022.

Tianheng Cheng, Lin Song, Yixiao Ge, Wenyu Liu, Xinggang Wang, and Ying Shan. Yolo-world: Real-time open-vocabulary object detection. In CVPR , 2024.

Xiangtai Li, Haobo Yuan, Wei Li, Henghui Ding, Size Wu, Wenwei Zhang, Yining Li, Kai Chen, and Chen Change Loy. Omg-seg: Is one model good enough for all segmentation? In CVPR , 2024.

Yongming Rao, Wenliang Zhao, Guangyi Chen, Yansong Tang, Zheng Zhu, Guan Huang, Jie Zhou, and Jiwen Lu. Denseclip: Language-guided dense prediction with context-aware prompting. In CVPR , 2022.

Xiangtai Li, Henghui Ding, Haobo Yuan, Wenwei Zhang, Jiangmiao Pang, Guangliang Cheng, Kai Chen, Ziwei Liu, and Chen Change Loy. Transformer-based visual segmentation: A survey. IEEE TPAMI , 2024.

Timo Lüddecke and Alexander Ecker. Image segmentation using text and image prompts. In CVPR , 2022.

Wenfang Sun, Yingjun Du, Gaowen Liu, Ramana Kompella, and Cees GM Snoek. Training-free semantic segmentation via llm-supervision. arXiv preprint arXiv:2404.00701 , 2024.

Zeyu Han, Fangrui Zhu, Qianru Lao, and Huaizu Jiang. Zero-shot referring expression comprehension via structural similarity between images and captions. In CVPR , 2024.

Silin Cheng, Yang Liu, Xinwei He, Sebastien Ourselin, Lei Tan, and Gen Luo. Weakmcn: Multi-task collaborative network for weakly supervised referring expression comprehension and segmentation. In CVPR , 2025.

Taylor Shin, Yasaman Razeghi, Robert L Logan IV, Eric Wallace, and Sameer Singh. Autoprompt: Eliciting knowledge from language models with automatically generated prompts. In EMNLP , 2020.

Zhengbao Jiang, Frank F Xu, Jun Araki, and Graham Neubig. How can we know what language models know? TACL , 2020.

Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys , 2023.

Yuning Lu, Jianzhuang Liu, Yonggang Zhang, Yajing Liu, and Xinmei Tian. Prompt distribution learning. In CVPR , 2022.

Beier Zhu, Yulei Niu, Yucheng Han, Yue Wu, and Hanwang Zhang. Prompt-aligned gradient for prompt tuning. In ICCV , 2023.

Xinyang Liu, Dongsheng Wang, Miaoge Li, Zhibin Duan, Yishi Xu, Bo Chen, and Mingyuan Zhou. Patch-token aligned bayesian prompt learning for vision-language models. arXiv preprint arXiv:2303.09100 , 2023.

Hantao Yao, Rui Zhang, and Changsheng Xu. Visual-language prompt tuning with knowledgeguided context optimization. In CVPR , 2023.

Hantao Yao, Rui Zhang, and Changsheng Xu. Tcp: Textual-based class-aware prompt tuning for visual-language model. In CVPR , 2024.

Jinyoung Park, Juyeon Ko, and Hyunwoo J Kim. Prompt learning via meta-regularization. In CVPR , 2024.

Xinyu Tian, Shu Zou, Zhaoyuan Yang, and Jing Zhang. Argue: Attribute-guided prompt tuning for vision-language models. In CVPR , 2024.

Adrian Bulat and Georgios Tzimiropoulos. Lasp: Text-to-text optimization for language-aware soft prompting of vision & language models. In CVPR , 2023.

Yingjun Du, Wenfang Sun, and Cees Snoek. Ipo: Interpretable prompt optimization for vision-language models. In NeurIPS , 2024.

Zifeng Wang, Zizhao Zhang, Chen-Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, and Tomas Pfister. Learning to prompt for continual learning. In CVPR , 2022.

Hyojin Bahng, Ali Jahanian, Swami Sankaranarayanan, and Phillip Isola. Exploring visual prompts for adapting large-scale models. arXiv preprint arXiv:2203.17274 , 2022.

Feng Li, Qing Jiang, Hao Zhang, Tianhe Ren, Shilong Liu, Xueyan Zou, Huaizhe Xu, Hongyang Li, Jianwei Yang, Chunyuan Li, et al. Visual in-context prompting. In CVPR , 2024.

Lingfeng Yang, Yueze Wang, Xiang Li, Xinlong Wang, and Jian Yang. Fine-grained visual prompting. In NeurIPS , 2024.

Yi-Lun Lee, Yi-Hsuan Tsai, Wei-Chen Chiu, and Chen-Yu Lee. Multimodal prompting with missing modalities for visual recognition. In CVPR , 2023.

Yaowei Li, Ruijie Quan, Linchao Zhu, and Yi Yang. Efficient multimodal fusion via interactive prompting. In CVPR , 2023.

Shuvendu Roy and Ali Etemad. Consistency-guided prompt learning for vision-language models. In ICLR , 2024.

Zheng Li, Xiang Li, Xinyi Fu, Xin Zhang, Weiqiang Wang, Shuo Chen, and Jian Yang. Promptkd: Unsupervised prompt distillation for vision-language models. In CVPR , 2024.

Haoyang Li, Liang Wang, Chao Wang, Jing Jiang, Yan Peng, and Guodong Long. Dpc: Dual-prompt collaboration for tuning vision-language models. In CVPR , 2025.

Ming Ding, Zhuoyi Yang, Wenyi Hong, Wendi Zheng, Chang Zhou, Da Yin, Junyang Lin, Xu Zou, Zhou Shao, Hongxia Yang, et al. Cogview: Mastering text-to-image generation via transformers. In NeurIPS , 2021.

Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In ICML , 2021.

Zhengyi Wang, Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu. Prolificdreamer: High-fidelity and diverse text-to-3d generation with variational score distillation. In NeurIPS , 2023.

Peihao Wang, Dejia Xu, Zhiwen Fan, Dilin Wang, Sreyas Mohan, Forrest Iandola, Rakesh Ranjan, Yilei Li, Qiang Liu, Zhangyang Wang, et al. Taming mode collapse in score distillation for text-to-3d generation. In CVPR , 2024.

Hyung-gun Chi, Myoung Hoon Ha, Seunggeun Chi, Sang Wan Lee, Qixing Huang, and Karthik Ramani. Infogcn: Representation learning for human skeleton-based action recognition. In CVPR , 2022.

Namdar Homayounfar, Yuwen Xiong, Justin Liang, Wei-Chiu Ma, and Raquel Urtasun. Levelset r-cnn: A deep variational method for instance segmentation. In ECCV , 2020.

Mingyu Lee and Jongwon Choi. Text-guided variational image generation for industrial anomaly detection and segmentation. In CVPR , 2024.

Ziyao Zeng, Daniel Wang, Fengyu Yang, Hyoungseob Park, Stefano Soatto, Dong Lao, and Alex Wong. Wordepth: Variational language prior for monocular depth estimation. In CVPR , 2024.

Jian Zhang, Chenglong Zhao, Bingbing Ni, Minghao Xu, and Xiaokang Yang. Variational few-shot learning. In CVPR , 2019.

Jingyi Xu and Hieu Le. Generating representative samples for few-shot classification. In CVPR , 2022.

Wenfang Sun, Yingjun Du, Xiantong Zhen, Fan Wang, Ling Wang, and Cees GM Snoek. Metamodulation: Learning variational feature hierarchies for few-shot learning with fewer tasks. In ICML , 2023.

Jie Liu, Yingjun Du, Zehao Xiao, Cees GM Snoek, Jan-Jakob Sonke, and Efstratios Gavves. Memory-augmented variational adaptation for online few-shot segmentation. In CVPR , 2023.

Yingjun Du, Jun Xu, Huan Xiong, Qiang Qiu, Xiantong Zhen, Cees GM Snoek, and Ling Shao. Learning to learn with variational information bottleneck for domain generalization. In ECCV , 2020.

Pan Li, Da Li, Wei Li, Shaogang Gong, Yanwei Fu, and Timothy M Hospedales. A simple feature augmentation for domain generalization. In ICCV , 2021.

Lingxiao Yang, Ru-Yuan Zhang, Yanchen Wang, and Xiaohua Xie. Mma: Multi-modal adapter for vision-language models. In CVPR , 2024.

Diederik P Kingma, Max Welling, et al. Auto-encoding variational bayes. In ICLR , 2014.

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR , 2009.

Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In CVPR workshop , 2004.

Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In CVPR , 2012.

Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In ICCV , 2013.

Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In ICVGIP , 2008.

Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101-mining discriminative components with random forests. In ECCV , 2014.

Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Finegrained visual classification of aircraft. arXiv preprint arXiv:1306.5151 , 2013.

Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In CVPR , 2010.

Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:1212.0402 , 2012.

Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In CVPR , 2014.

Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing , 2019.

Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In ICML , 2019.

Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global representations by penalizing local predictive power. In NeurIPS , 2019.

Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. In CVPR , 2021.

Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, and Mike Guo. The many faces of robustness: A critical analysis of out-of-distribution generalization. In ICCV , 2021.

Matteo Farina, Massimiliano Mancini, Giovanni Iacca, and Elisa Ricci. Rethinking few-shot adaptation of vision-language models in two stages. In CVPR , 2025.

Shihan Wu, Ji Zhang, Pengpeng Zeng, Lianli Gao, Jingkuan Song, and Heng Tao Shen. Skip tuning: Pre-trained vision-language models are effective and efficient adapters themselves. In CVPR , 2025.

Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAI blog , 2019.

Seokju Cho, Heeseong Shin, Sunghwan Hong, Anurag Arnab, Paul Hongsuck Seo, and Seungryong Kim. Cat-seg: Cost aggregation for open-vocabulary semantic segmentation. In CVPR , 2024.

Xiaohu Huang, Hao Zhou, Kun Yao, and Kai Han. Froster: Frozen clip is a strong teacher for open-vocabulary action recognition. In ICLR , 2024.

Jian Ding, Nan Xue, Gui-Song Xia, and Dengxin Dai. Decoupling zero-shot semantic segmentation. In CVPR , 2022.

Feng Liang, Bichen Wu, Xiaoliang Dai, Kunpeng Li, Yinan Zhao, Hang Zhang, Peizhao Zhang, Peter Vajda, and Diana Marculescu. Open-vocabulary semantic segmentation with mask-adapted clip. In CVPR , 2023.

Mengde Xu, Zheng Zhang, Fangyun Wei, Han Hu, and Xiang Bai. Side adapter network for open-vocabulary semantic segmentation. In CVPR , 2023.

Hanoona Rasheed, Muhammad Uzair Khattak, Muhammad Maaz, Salman Khan, and Fahad Shahbaz Khan. Fine-tuned clip models are efficient video learners. In CVPR , 2023.

Zejia Weng, Xitong Yang, Ang Li, Zuxuan Wu, and Yu-Gang Jiang. Open-vclip: Transforming clip to an open-vocabulary video model via interpolated weight optimization. In ICML , 2023.

Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao. Eva-clip: Improved training techniques for clip at scale. arXiv preprint arXiv:2303.15389 , 2023.

Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In ICCV , 2023.

Michael Tschannen, Alexey Gritsenko, Xiao Wang, Muhammad Ferjad Naeem, Ibrahim Alabdulmohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, et al. Siglip 2: Multilingual vision-language encoders with improved semantic understanding, localization, and dense features. arXiv preprint arXiv:2502.14786 , 2025.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We clearly state our contributions in Abstract and also Section 1. These contributions are well validated by our experimental results in Section 5.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We provide a "limitation" subsection in the supplementary material.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We have provided the theoretical proof in the appendix.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We have specified all the training details in Section 5.1 and supplemental materials.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: (1) Data: All the datasets we used in this paper are publicly available online, and all the readers are free to download them. We list the statistics of all the used datasets in the supplementary material. (2) Code: Code will be available after paper got accepted.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification:

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: Following the standard experimental setup, we repeat each experiment over 3 random seeds and report the mean of the results.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We provide the computing resources in experiments 5.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We reviewed and followed the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We provide the potential broader impacts in supplemental materials.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The data and models pose no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We cite the original papers that produced the code package and datasets.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: Details of the datasets/model are provided in the supplemental materials. The code will be released after the paper get accepted.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Appendix

A Dataset Details

Details of 14 datasets are shown in Table A1.

B More Implementation Details

All models are trained using the AdamW optimizer with a learning rate of 0.001 and a weight decay of 0.01. The batch size is set to 32 for ImageNet and 4 for all other datasets. We apply automatic mixed-precision training throughout to improve efficiency. For base-to-novel generalization on ImageNet, we train for 5 epochs; for other datasets we train for 10 epochs. For cross-dataset and domain generalization tasks, we train on ImageNet for a single epoch. Few-shot learning tasks use 5 training epochs on ImageNet and 50 epochs on target datasets. All reported results are averaged over three independent runs. All prompts and representation tokens are initialized from a zero-mean Gaussian distribution with a standard deviation of 0.02. For EuroSAT, we follow MMRL [8] and set the representation token dimension d r = 2048 ; for all other datasets, we use d r = 512 . The fusion parameter α in MMRL-style classifiers is fixed to 0.7. The average accuracy is reported over three independent runs. For variational modeling, we use a two-layer MLP with GELU activation to parameterize both the posterior network ϕ and the prior network ψ , outputting mean and log-variance vectors per layer. The latent variables z are sampled using the reparameterization trick, and we perform S = 10 Monte Carlo samples at inference time. Class prototypes o y are computed offline at the start of training.

C Efficiency Analysis

We analyze the efficiency of our variational multi-modal prompt framework, focusing on the tradeoff between inference cost and generalization ability. As detailed in Table A2, VaMP introduces only a modest overhead compared to the strong MMRL baseline. For instance, with S = 10 , our method improves the Harmonic Mean (HM) from 81.20 to 82.37 with a minimal latency increase of just 0.8ms per image on an NVIDIA V100 GPU. The performance gains saturate quickly as S increases, demonstrating that VaMP achieves an effective balance between accuracy and efficiency even with limited sampling. Furthermore, our approach is parameter-efficient, increasing the number of learnable parameters by less than 6% when integrated into MMRL (from 4.992M to 5.132M). This balance demonstrates that VaMP can be deployed with low overhead while still leveraging uncertainty modeling for improved generalization.

D More Ablation Studies

We conduct ablation studies by integrating our variational prompt adaptation into the MMRL [8] framework, analyzing the impact of key hyperparameters such as the prompt depth, and prompt length. All results are reported on the base-to-novel generalization benchmark (11 datasets), using average accuracy across base and novel splits.

Effect of Prompt Insertion Depth J and H . We vary the transformer layer index J where prompt tuning begins, and the number of consecutive layers H to which prompts are added. As shown in Table A3, inserting prompts deeper into the encoder (e.g., starting from layer J = 5 ) and extending them to more layers (e.g., H = 7 ) leads to better performance on both base and novel classes. This confirms the benefit of hierarchical prompt injection for deeper vision-language alignment.

Effect of Prompt Token Length M . We analyze the sensitivity to the number of prompt tokens M injected per layer. Table A4 shows that increasing M from 1 to 5 improves accuracy, as the model benefits from higher representational capacity. However, further increasing to M = 8 slightly reduces generalization, likely due to redundancy and overfitting. Hence, we set M = 5 as the default in our main experiments.

E Detailed Derivation of the Variational Lower Bound (ELBO)

We derive the evidence lower bound (ELBO) used in our variational prompt learning framework. Our goal is to estimate the conditional likelihood of the class label y given an image x and a text prompt template t , where the latent variables z represent the sample-specific text prompt tokens injected across multiple layers of the language encoder.

The marginal likelihood is obtained by integrating out the latent variables z :

Since this integral is generally intractable, we introduce a variational posterior q ϕ ( z | x ) and apply Jensen's inequality:

We model z as a collection of layer-wise latent prompt embeddings { z i } J + H -1 i = J , one for each of the H transformer layers in the text encoder. We assume the posterior and prior factorize across layers:

This leads to the following form of the ELBO:

where z i is injected at layer i of the frozen text encoder, modulating the token representations via concatenation.

During training, we replace p ( z i | x ) with a class-aware prior:

where the class prototype o y is the mean of posterior means ˆ µ i over all training samples in class y .

The final training objective maximizes the ELBO in Eq. 29.

This variational formulation enables our model to learn expressive, uncertainty-aware, sample-specific prompts while regularizing the latent space with class-level semantic structure.

F Extension to Other Tasks

To assess the generalizability of VaMP beyond standard image classification, we extended our evaluation to the more complex domains of open-vocabulary segmentation and action recognition. Our approach was integrated into two state-of-the-art frameworks: CAT-Seg [90] for segmentation and FROSTER [91] for action recognition, both of which utilize a CLIP ViT-B/16 backbone. This compatibility allowed for a seamless and direct evaluation of VaMP's effectiveness in these diverse tasks.

For the open-vocabulary segmentation task, we adhered to the established CAT-Seg protocol. The model was trained on the COCO-Stuff dataset (118K images, 171 categories) and subsequently evaluated on several challenging benchmarks, including ADE20K, PASCAL-Context, and PASCAL VOC, which feature a wide range of category scales (59-847 classes). The results are detailed in Table A5.

For open-vocabulary action recognition, we adopted the base-to-novel evaluation protocol from FROSTER. This setup involves training the model exclusively on the base classes from two widely used video benchmarks, Kinetics-400 and UCF-101. The primary metric is the model's ability to generalize to novel, unseen action categories during testing. Table A6 summarizes these results.

The consistent performance gains observed across both segmentation and action recognition tasks underscore the scalability and robust generalization capabilities of our proposed framework.

G Scalability to Other VLM Architectures

To demonstrate the architecture-agnostic nature of our method, we validated VaMP on several stateof-the-art VLMs that represent the latest advances in CLIP-style architectures. Our method can be readily integrated into more complex VLMs, as it only requires access to intermediate transformer layers and the ability to inject prompt tokens. The variational adaptation module is lightweight (MLPs per layer) and does not assume any specific backbone structure. We therefore integrated VaMP into several prominent models, including EVA-CLIP [97], SigLIP [98], and SigLIP 2 [99]. These models share similar structural designs with CLIP, which enabled their rapid adaptation for our experiments. As shown in Table A7, VaMP obtains clear and consistent performance improvements across all VLMs, demonstrating its strong generalization capability to unseen categories-a critical challenge in prompt learning. The consistent improvements across different VLMs further substantiate the effectiveness and generalizability of our approach to different architectures.

H Broader Impact and Limitations

Our work presents a variational framework for sample-specific, uncertainty-aware prompt adaptation in vision-language models, aiming to improve robustness under distribution shifts and limited supervision. The proposed method has the potential to benefit a wide range of downstream applications where multi-modal understanding and generalization are essential, such as assistive AI systems, open-world recognition, or low-resource domain transfer. The probabilistic modeling component can further inspire future efforts in calibrated and interpretable multi-modal adaptation. At present, we have not identified any ethical concerns associated with the real-world applications of this technology. However, we strongly recommend continuous monitoring and evaluation to ensure its responsible

and ethical deployment. Our approach also has certain limitations. First, our class-aware prior construction relies on access to class prototypes computed from training data, which may not be available in zero-label scenarios. Extending our method to work under fully unsupervised or few-label conditions remains an open direction. Second, our experiments focus on classification tasks; the extension to generative or structured prediction settings ( e.g. , image captioning, VQA) requires further investigation and architectural adaptation. Despite these challenges, we believe our framework takes an important step toward principled and efficient multi-modal prompt learning, and hope it provides useful insights for future research in uncertainty-aware adaptation, vision-language alignment, and lightweight tuning strategies.

Peer reviews:
comments: SUMMARY:
This paper transforms deterministic multimodal prompts into statistical ones to capture instance-level variations and model uncertainty, achieving performance that surpasses state-of-the-art methods across multiple settings and benchmarks.

STRENGTHS AND WEAKNESSES:
Strengths:

1.The writing is clear and easy to follow.

2.The mathematical formulations are well-defined, and the reasoning is sound.

3.The method demonstrates superior performance, surpassing state-of-the-art prompting strategies.

Weaknesses:

1.I feel that uncertainty-based modeling may increase computational overhead. Could the authors compare the models and computational complexity of deterministic and uncertainty-based prompting strategies?

2.The current benchmarks and settings are relatively simple. Can the proposed method be extended to dense prediction or video-based benchmarks?

3.There are many strategies for uncertainty modeling, such as Monte Carlo sampling and evidence learning. Why did the authors choose to model the Gaussian mean and variance using MLPs?

4.According to Figure 1, the prompts only operate at the encoding stage. Why not consider integrating them at the decoding stage as well?

QUESTIONS:
Please refer to the Weaknesses section above.

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-06-04

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a variational prompt learning technique designed to overcome the limitations of data-scarce fine-tuning in downstream vision-language tasks (e.g., CLIP). The proposed method enables sample-specific prompt learning by modeling uncertainty through posterior sampling, making the model more adaptable to input content.\nThe key contributions of the paper include:\n1.\tIntroduction of a sample-specific, multi-modal prompt generation strategy.\n2.\tA mechanism for prompt adaptation.\n3.\tA class-aware prior, which integrates both instance-level representations and class prototypes.\nThe method is framed as a variational inference-based prompt adaptation (VMPA) approach, enabling uncertainty-aware learning for vision-language models.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\no\tThe paper builds upon recent developments in prompt learning and variational inference in the context of vision-language models.\no\tThe proposed architecture appears technically sound and is consistent with methods in prior work (e.g., MaPLe [24], MMRL [15]).\n\nWeaknesses:\no\tThe implementation details are insufficiently explained, particularly regarding the computation of $\\tilde{z}_{i-1}$, which seems critical for reproducibility.\t\no\tThere is no discussion of the method's limitations in the main paper, although it is stated to be present in the supplementary material.\no\tThe performance improvement over the state-of-the-art is marginal, raising concerns about the impact.\no\tOverlap with prior work (notably [15] and [24]) is there, but after the rebuttal I agree that: one key novelty of VaMP lies in the reformulation of prompt learning as a variational inference problem for multi-modal prompt learning. \n\n\nSignificance\n•\tWhile the idea of integrating uncertainty into prompt learning is interesting, the actual improvements are limited and lack justification.\n•\tThe method is only evaluated on a narrow range of models (e.g., CLIP), with no extension to more recent large-scale vision-language models.\n•\tThe contribution feels incremental, as it adapts known techniques with minor modifications.\n \nOriginality\n•\tThe proposed method borrows heavily from prior work:\n   o\tSimilar prompt insertion strategy as MaPLe [24] and MMRL [15].\n   o\tThe variational adaptation idea has been previously explored in the literature.\n•\tThere is no clear justification of why this method is novel or more effective than the prior works.\n•\tThe paper does not sufficiently distinguish itself from existing methods.\n\nQUESTIONS:\nMuch of the paper is well-written. However, some parts are confusing:\no\tThe role of $\\tilde{z}_{i-1}$ is unclear: is it computed from the residual stream of the image encoder? Is it passed through a trainable layer and concatenated with transformer inputs? \n\n1.\tUnclear implementation: How exactly is $\\tilde{z}_{i-1}$ calculated? What input is used to compute it? The process is vague and could hinder reproducibility.\n2.\tOverlaps with prior work: Please clarify the key technical differences from [24] and [15], beyond what has already been published.\n3.\tLimitations not stated: The paper should explicitly discuss its limitations in the main body, not only in the supplementary material.\n\nLIMITATIONS:\nThe paper shows technical competence and incremental improvements, but lacks novelty, clear explanation of core mechanisms, and sufficient performance gains to justify acceptance.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes VaMP, a novel framework for multi-modal prompt learning with vision-language models. It introduces sample-specific prompt generation conditioned on visual input,  a variational inference formulation where prompt tokens are treated as latent variables, and  a class-aware prior to guide prompt regularization. The method is evaluated under various settings including base-to-novel generalization, cross-dataset generalization, and domain generalization. VaMP outperforms strong baselines like MMRL and PromptSRC across all settings.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n- The paper is well-motivated, addressing a clear gap in current multi-modal prompt tuning methods by enabling instance-level adaptation and uncertainty modeling. \n\n- The proposed framework is conceptually elegant, formulating prompt tuning as variational inference with a class-aware prior, which is both novel and intuitive. \n\n- The writing is clear and well-structured, making the method easy to understand despite its technical depth. \n\n- The authors provide solid empirical results across a wide range of benchmarks, consistently outperforming strong baselines. The ablation studies are particularly thorough and directly support the value of each design component.\n\nWeaknesses\n\nWhile the paper is novel and interesting, there are a couple of points where a bit more discussion could make the work even more complete. \n\n- First, the class-aware prior is only used during training, and replaced by a standard Gaussian at test time—some reflection on the potential impact of this train-test mismatch would be helpful. \n\n- Second, although the variational approach introduces uncertainty into prompt learning, the paper doesn’t delve into how this uncertainty might be utilized in practice, such as for calibration or confidence-aware decision making, which could be an interesting extension.\n\nQUESTIONS:\n- The current experiments use a fixed 16-shot setting. Have the authors explored how VaMP performs under more limited (e.g., 1-shot) or more generous supervision?\n\n- Since the posterior over prompt tokens is conditioned only on the image, have the authors considered whether adding textual input (e.g., class name or prompt template) could further inform the distribution? \n\n- In Section 4.4, the authors mention drawing 10 Monte Carlo samples from the posterior to compute the final prediction. Could the authors clarify whether the same set of sampled prompts is shared across all classes for computing similarities, or whether separate prompts are sampled for each class template (e.g., “a photo of a [CLASS]”)?\n\n- In Section 4.3, class prototypes are computed by averaging the posterior means of image embeddings within each class (Eq. 18). Are these computed once before training (e.g., with frozen CLIP features), or are they updated dynamically during training as the posterior evolves?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel framework for adapting vision-language models (VLMs), like CLIP, to new tasks with limited supervision. It introduces sample-specific, uncertainty-aware prompt tuning by modeling text-side prompts as latent variables conditioned on image inputs, and regularizing them with a class-aware prior that encodes semantic structure. Experiments on few-shot, cross-dataset, and domain generalization tasks demonstrate that VaMP outperforms state-of-the-art baselines while maintaining high parameter efficiency.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n1. VaMP generates prompts dynamically based on each input image, which allows the model to adapt better to individual samples and handle uncertainty—especially beneficial in few-shot or out-of-distribution scenarios.\n2. By incorporating a class-aware prior based on semantic prototypes, the method enhances intra-class consistency and improves generalization, especially across domains.\n3. Extensive experiments and good results.\n\nWeakness:\n1. Constructing the class-aware prior depends on class labels to compute prototypes, which may not generalize well to unsupervised or label-scarce settings.\n2. Only classification tasks, need to show the potential to extend this method to other tasks, to show the impact.\n3. The use of variational sampling introduces additional computational overhead during inference. Although there's efficiency analysis in appendix, the trade-off between improved generalization and increased latency has not been fully quantified or explored in depth.\n4. Scalability to more complex architectures is not clearly stated.\n\nQUESTIONS:\n1. Can you provide more insight into the computational overhead introduced by variational sampling during inference, and whether there is a trade-off between accuracy and efficiency?\n2. Would it be better if put efficiency analysis in the main paper rather than being placed in the appendix? The current experiments primarily highlight the effectiveness of the method, and incorporating efficiency results would provide a more practical perspective.\n3. Can you provide qualitative examples to illustrate the generated prompts' structure or diversity?\n4. Can VaMP be scaled or adapted to newer multi-modal models like GPT-4V, which use more complex fusion mechanisms?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-04"}