Prompt Diffusion Robustifies Any-Modality Prompt Learning

## Abstract

Abstract Foundation models enable prompt-based classifiers for zero-shot and few-shot learning. Nonetheless, the conventional method of employing fixed prompts suffers from distributional shifts that negatively impact generalizability to unseen samples. This paper introduces prompt diffusion , which uses a diffusion model to gradually refine the prompts to obtain a customized prompt for each sample.
Specifically, we first optimize a collection of prompts to obtain over-fitted prompts per sample. Then, we propose a prompt diffusion model within the prompt space, enabling the training of a generative transition process from a random prompt to its overfitted prompt. As we cannot access the label of a test image during inference, our model gradually generates customized prompts solely from random prompts using our trained, prompt diffusion. Our prompt diffusion is generic, flexible, and modality-agnostic, making it a simple plug-and-play module seamlessly embedded into existing prompt learning methods for textual, visual, or multi-modal prompt learning.
Our diffusion model uses a fast ODE-based sampling strategy to optimize test sample prompts in just five steps, offering a good trade-off between performance improvement and computational efficiency.
For all prompt learning methods tested, adding prompt diffusion yields more robust results for base-to-new generalization, cross-dataset generalization, and domain generalization in classification tasks tested over 15 diverse datasets.

## 1 Introduction

Foundation models trained on a diverse set of image-text pairs that encapsulate a virtually limitless vocabulary of real-world concepts (Radford et al., ; Jia et al., ; Li et al., ), have demonstrated remarkable adaptability across various downstream tasks (Lin et al., ; Li et al., ; ; Zhang et al., ; ). These models perform zero-shot image classification by filling in a predefined prompt template (*e.g.*, “a photo of a [CLASS]”) with specific class names for the text encoder.
Despite their effectiveness in generalizing to new tasks, performance can be affected by minor alterations in the wording of prompt templates, (Zhou et al., ).
Rather than manually creating hand-made prompts, several new prompt learning techniques in natural language processing (Lester et al., ; Liu et al., ) and computer vision (Zhou et al., ; ; Jia et al., ; Khattak et al., ; Roy & Etemad, ; Li et al., ) have been suggested, which focus on learning a set of soft prompts with the aid of a small amount of labeled data. However, training a model with such deterministic prompts often results in overfitting, causing the model to focus too much on the training data, which affects its ability to generalize. These methods usually fail when a considerable distribution shift between training and test data leads to suboptimal generalization performance.
We propose generating a distribution of prompts for each sample, employing a probabilistic approach that effectively incorporates visual (domain) information in a manner capable of learning and adaptation.

We are inspired by diffusion models (Song et al., ; Zhou et al., ) that have emerged as a powerful
generative technique with broad applicability for tasks as diverse as image generation (Ho et al., ), video processing (Ho et al., ), and text generation (Gong et al., ). The core principle
behind diffusion involves an iterative refinement of the data distributions, transitioning from
a simple initial distribution to the desired target distribution. This iterative improvement process transforms the simple initial distribution into a series of sub-transformations, making it a versatile tool suitable for various tasks.

Figure: Figure 1: Prompt diffusion enhances traditional prompt learning methods such as CoCoOp (Zhou et al., ) by introducing a diffusion process within the prompt space (colored arrows). Unlike deterministic prompt learning methods (black arrows), we employ a diffusion transformer to refine the prompts gradually. This process creates tailored prompts for each sample, complementing and augmenting existing prompting methods to achieve higher prediction accuracy through stronger generalization.
Refer to caption: x1.png

To the best of our knowledge, we are the first to introduce diffusion models into prompt learning. Our process of generating prompts through a diffusion model is depicted in Figure 1. Our prompt diffusion involves gradually refining the prompts with a diffusion transformer, which leads to the development of custom prompts tailored to each sample, thereby enhancing the accuracy of predictions and robustifying their generalization across downstream tasks.

In this paper, we make three contributions.
First, we propose a prompt diffusion method based on the
transformer within the prompt space, enabling the learning of a generative pathway that seamlessly transitions from a random prompt to its personalized prompt.
Rather than relying on a single static prompt acquired for the entire dataset, our prompt diffusion can learn and evolve from noise to the example prompt throughout the training process. These personalized prompts are adept at generalizing the unique domain characteristics inherent in each sample, thus enhancing the model’s ability to generalize.
Second, to better deploy prompt diffusion, we propose a per-sample overfitting strategy to obtain “optima” prompts for each data sample, allowing our diffusion transformer to effectively navigate the transition from general to highly personalized prompts within the training phase.
Third, our prompt diffusion approach is versatile, adaptable, and modality-agnostic, which makes it easily integrated as a plug-and-play module within existing prompt learning techniques. This includes methods specialized for text-based prompts (*e.g.*, CoCoOP (Zhou et al., )), visual prompts (*e.g.*, VPT (Jia et al., )), as well as three approaches that combine both text and visual inputs (*e.g.*, MaPLe (Khattak et al., ), PromptSRC (Khattak et al., ), and CoPrompt (Roy & Etemad, )).
Our diffusion model leverages a state-of-the-art fast ODE-based sampling strategy (Zhou et al., ) that optimizes test sample prompts in just five steps, achieving an effective balance between performance enhancement and computational efficiency. To validate the effectiveness of our method, we conduct extensive testing across three common prompt learning experimental setups over 15 datasets: base-to-new generalization, cross-dataset generalization, and domain generalization. Adding prompt diffusion yields more robust results for all prompt learning methods tested.

## 2 Related Work

Foundation models. Foundation models developed through training on a wide and varied collection of image-text pairs, capture a nearly boundless array of concepts from the real world (Radford et al., ; Jia et al., ; Li et al., ; Schneider et al., ; Xu et al., ), and have exhibited exceptional versatility in numerous downstream tasks (Lin et al., ; Li et al., ; ; Zhang et al., ; ). Foundation models can be categorized into four types:
1) Masked language modeling, as investigated in studies such as (Kim et al., ; Lu et al., ), 2) Masked region prediction exemplified by (Tan & Bansal, ; Su et al., ), 3) Image-text matching addressed by works like (Tan & Bansal, ; Kim et al., ), and 4) Contrastive learning, with notable references including (Radford et al., ; Jia et al., ; Li et al., ; Huo et al., ).
Numerous studies have demonstrated improved performance in tasks such as few-shot image recognition (Gao et al., ; Zhang et al., ; Kim et al., ), object detection (Li et al., ; Maaz et al., ; Zhou et al., ; Gu et al., ; Zang et al., ; Cheng et al., ), and segmentation (Li et al., ; Rao et al., ; Li et al., ; Lüddecke & Ecker, ) using tailored methods. In this paper, we introduce a novel plugin designed to unify different prompt-learning approaches
to address the issues of prompt engineering in traditional foundation models, aimed at solving the base-to-new, cross-dataset, and domain generalization of visual recognition problems.

Prompt learning. Prompt learning, originally introduced in the natural language processing community (Shin et al., ; Jiang et al., ; Liu et al., ), involves applying a fixed function to input tokens to provide task instructions to the model. In the computer vision community, prompt learning has been explored in various forms, including textual prompts (Zhou et al., ; ; Derakhshani et al., ; Lu et al., ; Zhu et al., ), visual prompts (Jia et al., ; Ge et al., ; Wang et al., ; Bahng et al., ; Li et al., ; Yang et al., ), and multi-modal prompts (Khattak et al., ; Lee et al., ; Li et al., ; Roy & Etemad, ; Li et al., ).
1) Textual prompt learning, as pioneered by CoOp (Zhou et al., ) and CoCoOp (Zhou et al., ), fine-tunes a CLIP vision-language model (Radford et al., ) for few-shot transfer by optimizing a continuous set of prompt vectors within its language branch. Bayesian prompt learning (Derakhshani et al., ) formulated prompt learning as a variational inference problem and demonstrated its ability to generalize unseen classes at the expense of base class accuracy.
2) Visual prompt tuning (Jia et al., ) introduces task-specific learnable prompts in the input visual space while keeping the pre-trained backbone fixed, optimizing them using the downstream task’s label.
3) Multi-modal prompt learning (Khattak et al., ; ; Li et al., ; Xiao et al., ) applied prompt learning in both vision and language branches to improve the alignment between the vision and language representations.
In contrast to previous prompt learning methods, this paper introduces modality-agnostic prompt diffusion, which leverages a diffusion model to generate prompts gradually. Our method serves as a simple plug-and-play module that seamlessly integrates with existing prompt learning methods, whether textual, visual, or multi-modal.

## 3 Preliminaries

Before detailing our prompt diffusion, we first present the technical background on the CLIP foundation model, prompt-based learning, and diffusion models.

Contrastive Language-Image Pre-Training (CLIP). The objective of CLIP (Radford et al., ) is to train an image encoder $f_{I}$ and a text encoder $g_{T}$ through contrastive pre-training using a large set of paired images and texts. This encourages the encoders to align corresponding image-text pairs in a shared semantic space. After pre-training, CLIP exhibits the capacity for zero-shot visual recognition by casting classification as an image-text matching task. Specifically, the term “[CLASS]” is utilized as a placeholder within a prompt template (*e.g.*, “a photo of a [CLASS]”) for the text encoder $g_{T}$. If we let $g_{T}(\mathbf{T}_{i})$ represent text features extended for class $i$, the classification probability for class $i$ given an image $\mathbf{I}$ is:

$$ $p(y{=}i|\mathbf{I}){=}\frac{\exp(\langle g_{T}(\mathbf{T}_{i}),f_{I}(\mathbf{I })\rangle/\tau)}{\sum_{j{=}1}^{K}\exp(\langle g_{T}(\mathbf{T}_{j}),f_{I}( \mathbf{I})\rangle/\tau)},$ (1) $$

where $\langle g_{T}(\mathbf{T}_{i}),f_{I}(\mathbf{I})\rangle$ denotes the cosine similarity between the image feature $f_{I}(\mathbf{I})$ and the class-specific text feature $g_{T}(\mathbf{T}_{i})$ for the $i$-th class, $K$ the total number of classes, and $\tau$ the temperature parameter optimized during training.

Prompt-based learning enhances the transferability of the CLIP model by avoiding the need for prompt engineering. Instead, it enables automatic learning of prompts with a few samples from a downstream task. CoOp (Zhou et al., ) introduces and refines a set of $M$ continuous context vectors $\bm{V}{{=}}\{\bm{v}_{1},\bm{v}_{2},\ldots,\bm{v}_{M}\}$ as the learnable prompt. The prompt $\bm{T}_{i}{{=}}\{\bm{v}_{1},\bm{v}_{2},\ldots,\bm{v}_{M},\bm{c}_{i}\}$ is a concatenation of the learnable context vectors $\bm{V}$ and the class token embedding $\bm{c}_{i}$, which is then inputted to the text encoder $g_{T}(\cdot)$. CoOp tailors the static context vectors $\bm{V}$ by minimizing the negative log-likelihood for the correct class token:

$$ $\mathcal{L}_{\text{CE}}(\bm{V}){{=}}-\sum_{i}\bm{y}_{i}\log p(\bm{T}_{i}|\bm{I }),$ (2) $$

Here, $\bm{y}_{i}$ denotes the one-hot ground-truth label for class $i$. In downstream tasks, the pre-trained model parameters remain frozen, allowing the learnable prompt vectors $\bm{V}$ to be efficiently optimized through the minimization of the cross-entropy loss with only a limited number of samples.

Diffusion model.
In denoising diffusion probabilistic models (Ho et al., ), $q(\mathbf{x}_{t}|\mathbf{x}_{t-1})$, is characterized as a Markov chain that progressively introduces Gaussian noise at each time step $t$, beginning with a clean image $\mathbf{x}_{0}\sim q(\mathbf{x}_{0})$.
The forward diffusion process is formulated as:

$$ $\displaystyle q(\mathbf{x}_{T}|\mathbf{x}_{0}):{=}\prod_{t{=}1}^{T}q(\mathbf{x }_{t}|\mathbf{x}_{t-1}),$ (3) $$

where $q(\mathbf{x}_{t}|\mathbf{x}_{t-1}):{=}\mathcal{N}(\mathbf{x}_{t};\sqrt{1-\beta
_{t}}\mathbf{x}_{t-1},\beta_{t}{\bm{I}})$, $\{\beta\}_{t{=}0}^{T}$ is a variance schedule.
By defining $\alpha_{t}{:{=}}1{-}\beta_{t}$ and $\bar{\alpha_{t}}{:{=}}\prod_{s{=}1}^{t}\alpha_{s}$, the forward diffused sample at time step $t$, denoted as $\bm{x}_{t}$, can be generated in a single step as $\mathbf{x}_{t}{=}\sqrt{\bar{\alpha}_{t}}\mathbf{x}_{0}+\sqrt{1-\bar{\alpha}_{t
}}\bm{\epsilon}$.

The reverse process of the diffusion model learns to maximize the variational lower bound using parameterized Gaussian transitions, $p_{\theta}(\mathbf{x}_{t-1}|\mathbf{x}_{t})$. Consequently, the reverse process is approximated as a Markov chain with the learned mean and fixed variance, starting from random noise $\mathbf{x}_{T}\sim\mathcal{N}(\mathbf{x}_{T};\textbf{0},\bm{I})$.
The diffusion model is trained by optimizing the following objective function:

$$ $\displaystyle\mathcal{L}_{\theta}{=}\mathbb{E}_{t,\mathbf{x}_{0},\bm{\epsilon} }\Big{[}\|\bm{\epsilon}-\bm{\epsilon}_{\theta}(\sqrt{\bar{\alpha}_{t}}\mathbf{ x}_{0}+\sqrt{1-\bar{\alpha}_{t}}\bm{\epsilon},t)\|^{2}\Big{]}.$ (4) $$

In the sampling phase of diffusion, to sample from $p_{\theta}(\mathbf{x}_{t-1}|\mathbf{x}_{t})$, one can perform the following:

$$ $\displaystyle\mathbf{x}_{t-1}{=}\frac{1}{\sqrt{\alpha_{t}}}\Big{(}\mathbf{x}_{ t}-\frac{1-\alpha_{t}}{\sqrt{1-\bar{\alpha}_{t}}}\bm{\epsilon}_{\theta}( \mathbf{x}_{t},t)\Big{)}+\sigma_{t}\bm{\epsilon}.$ (5) $$

Based on the geometric property that each sampling trajectory approximately resides within a two-dimensional subspace embedded in a high-dimensional space, Zhou et al. () introduce the Approximate MEan-Direction Solver (AMED-Solver), a single-step ODE solver that predicts the mean direction at each sampling step. By appropriately selecting $s_{n}$ and $c_{n}$, the AMED-Solver achieves an approximation given by:

$$ $\mathbf{x}_{t_{n}}\approx\mathbf{x}_{t_{n+1}}+c_{n}(t_{n}-t_{n+1})\mathbf{ \epsilon}_{\theta}(\mathbf{x}_{s_{n}},s_{n}).$ (6) $$

This formulation provides a single-step ODE solver, and the DPM-Solver-2 (Lu et al., ) can be derived by setting $s_{n}=\sqrt{t_{n}t_{n+1}}$ and $c_{n}=1$. Unlike typical approaches that operate on images, our prompt diffusion model directly optimizes prompts. Given that prompt learning in vision-language tasks aims for faster and more accurate image classification, our proposed prompt diffusion, built upon the AMED-Solver, enables more rapid image classification during inference time.

## 4 Methodology

This section outlines our approach to training prompt learning via our proposed prompt diffusion model. We begin by explaining how to generate sample-specific overfitted prompts in Section 4.1. Next, we introduce prompt diffusion during both the training and testing phases to obtain diffused prompts in Section 4.2.

### 4.1 Per-sample prompt overfitting

Our approach begins by fine-tuning various prompts to achieve individualized overfitting for each data sample. This ensures the precise generation of prompts that are tailored to specific instances. Specifically, when dealing with an image represented as $\bm{x}$, we aim to obtain a set of prompts, denoted as $\bm{V}^{*}$, which have been explicitly overfitted to that sample. We feed both the image $\bm{x}$ and the initial prompts $\bm{V}{=}\{v_{1},v_{2},\ldots,v_{M}\}$ into various prompt learning models and then employ iterative gradient descent on Eq. (2) to optimize the set of prompts, resulting in $\bm{V}^{*}{=}\{v_{1}^{*},v_{2}^{*},\ldots,v_{M}^{*}\}$. These optimized prompts can be considered as the “optima” prompts for each sample. Note that the intermediate loss is solely adjusted to achieve overfitted prompts in this process. Afterward, the gradient information for the learnable prompts will be discarded without optimization incorporated into the final loss.
We illustrate this per-sample prompt overfitting for textual prompt learning with CoCoOp (Zhou et al., ) in Figure 2.

Once we obtain the overfitted prompts, our objective is to train the model using random prompts about these overfitted prompts. This is necessary because we cannot access the overfitted prompts during the testing stage. Therefore, in the following section, we use the diffusion model to learn the generative process of sample-specific prompts, thus robustifying the generalizability of the prompts for each sample.

Figure: Figure 2: Per-sample prompt overfitting for textual prompt learning. Through a minimal number of iterations $I$ using gradient descent, we successfully derive overfitted prompts for each sample in the dataset. These overfitted prompts act as a “ground truth” for the prompts of each sample, enabling our proposed diffusion transformer to grasp the transition from generic prompts to highly personalized overfitted prompts.
Refer to caption: x2.png

### 4.2 Prompt diffusion

We leverage the diffusion model (Song et al., ) to model textual, visual, or multi-modal prompts. In our implementation, we adapt the diffusion process to incrementally denoise and refine overfitted prompts, thereby enhancing the generative quality and coherence of the prompts.
Consequently, we introduce the concept of modality-agnostic prompt diffusion, a novel method that incrementally crafts sample-specific prompts for each instance. This methodical generation of prompts enhances their overall quality, ensuring that each prompt is optimally tuned to the nuances of its corresponding sample. This adaptive approach is designed to fine-tune the diffusion process, allowing for a more targeted and effective prompt generation that elevates the efficacy of the model’s performance.

Training phase. During the initial training stage, we obtain the overfitted prompts $\bm{V}^{*}$ of individual samples via our proposed per-sample prompt overfitting. Then, the diffusion model is used to progressively approximate the overfitted prompts, from a Gaussian noise vector $\tilde{\bm{V}}_{T}\sim\mathcal{N}(0,\bm{I})$, which possesses the exact dimensions as $\bm{V}^{*}$.
The approximation process iterates through the noise vectors $\tilde{\bm{V}}_{t}^{*}$, with $t$ representing the diffusion step from $T$ to 0. This process leads to the reconstruction of $\tilde{\bm{V}}_{0}$, which is expected to closely mirror the overfitted prompt associated with the particular sample being analyzed.

Specifically, throughout the forward diffusion phase at an increment in time $t$, we
derive the overfitted prompts $\bm{V}_{t}^{*}$.
Subsequently, the noised prompts, denoted as $\tilde{\bm{V}}_{t}$, and the training image feature $\pi$ - extracted through a lightweight neural network, Meta-Net $\pi(\theta)$ (Zhou et al., ) - are utilized to create a conditional token for each input and the temporal timestep $t$. These are then inputted into the diffusion transformer.
This process yields the interim diffused prompts $\tilde{\bm{V}}_{t}$. These prompts
then, the token $[\texttt{CLASS}]$ is synergized and integrated into the text encoder to generate the corresponding text features. The prediction of the final classification outcome for the training image is then conducted by Eq. (1). For each sample, our diffusion model encapsulates a dual-component objective comprising the variational lower bound $\mathcal{L}_{\rm{diff}}$ for the diffusion model and the cross-entropy loss $\mathcal{L}_{\rm{CE}}$. The overarching schema of our training scheme is depicted at the top of Figure 3.

The objective function, the simplified variational lower bound, aims to predict the denoised overfitted prompts accurately. Formally, the loss function is given by:

$$ $\mathcal{L}_{\text{diff}}=\left\lVert{\bm{V}}^{*}-\tilde{\bm{V}}_{\theta}\left (\sqrt{\bar{\alpha}_{t}}{\bm{V}}^{*}+\sqrt{1-\bar{\alpha}_{t}}\bm{\epsilon}, \pi,t\right)\right\rVert^{2},$ (7) $$

where $\tilde{\bm{V}}_{\theta}(\cdot,\cdot,\cdot)$ denotes the function parameterized by the transformer architecture (Vaswani et al., ). This function processes the input comprising the original overfitted prompts ${\bm{V}}^{*}$, image feature $\pi$, and the diffusion time step $t$. The efficacy of our model is measured by its ability to minimize this loss, thus accurately reconstructing the overfitted prompts from their noised counterparts.
By utilizing Eq. (2), we derive the final prediction $\hat{\mathbf{y}}$ using diffused prompts $\tilde{\bm{V}}_{t}$. The final objective is:

$$ $\displaystyle\mathcal{L}_{\rm{final}}$ $\displaystyle=\sum_{(x,y)}\Big{[}-\mathbb{E}{q_{(\tilde{\bm{V}}_{t}|\tilde{\bm {V}}_{t+1},\pi)}}\big{[}\log p(\mathbf{y}|\mathbf{x},\tilde{\bm{V}}_{t})\big{]}$ (8) $\displaystyle+\beta\left\lVert{\bm{V}}^{*}-\tilde{\bm{V}}_{\theta}\left(\sqrt{ \bar{\alpha}_{t}}{\bm{V}}^{*}+\sqrt{1-\bar{\alpha}_{t}}\bm{\epsilon},\pi,t \right)\right\rVert^{2},$ $$

where $\beta$ represents a hyperparameter.

In the supplemental materials, we provide the computational graph, which showcases the sequential steps of the forward and inverse diffusion processes on the prompts. Our method balances adaptability and informativeness by incorporating probabilistic prompts with the diffusion model.
Our model has also been applied to visual prompt tuning (VPT) (Jia et al., ) and multi-modal prompt learning (MaPLe, PromptSRC, and CoPrompt) (Khattak et al., ; ; Roy & Etemad, ), generating visual prompts through a process identical to that used for generating text prompts.

Figure: Figure 3: Prompt diffusion. (1) Training by generating prompts that are initially overfitted using per-sample overfitting. These prompts are then subjected to a noise injection before entering the forward diffusion process. The inputs for diffusion include noisy prompts $\tilde{\bm{V}_{t}^{*}}$, the image features $\pi$, and a randomly chosen time step $t$, which leads to the generation of diffused prompts $\tilde{\bm{V}_{t}}$. After training, the diffusion transformer can convert generic prompts into their overfitted counterparts for each sample. (2) During testing, the sampling process begins with an initial random noise $\tilde{\bm{V}}_{T}$, which is gradually refined into diffused prompts $\tilde{\bm{V}}_{t}$. At each time step $t$, the sampling process incorporates the previous state $\tilde{\bm{V}}_{t-1}$, test image features $\pi$, and current time step $t$ as inputs. The resulting diffused prompts $\tilde{\bm{V}_{0}}$ are then employed to make test sample predictions. Throughout $T$ with our diffusion transformer, the vanilla prompts are adapted into customized prompts that contain more specific information about the test sample, thereby enhancing prediction accuracy.
Refer to caption: extracted/5955913/images/training_testing.png

Testing phase.
During the testing phase, the generation of overfitted prompts is infeasible due to the unavailability of test sample labels. Consequently, the diffusion sampling process begins with the introduction of Gaussian noise $\tilde{\bm{V}}_{T}$ alongside the computed image feature set $\pi$, followed by a systematic denoising procedure. To address an unseen test instance $x$, initial image feature computations $\pi$ are performed. After this, the noise vector $\epsilon$ is drawn from a standard normal distribution $\mathcal{N}(\textbf{0},\textbf{I})$ for each model and data/set.
This ensures a diverse starting point for each prompt without using multiple models, training
multiple times, or employing different checkpoints.
These elements, comprising $\tilde{\bm{V}}_{T}$, $\pi$, and $\epsilon$, are then supplied to the trained prompt diffusion model to derive intermediate diffused prompts $\tilde{\bm{V}}_{T-1}$, represented by $\tilde{\bm{V}}_{\theta}(\tilde{\bm{V}}_{T},\pi,T)$. This
iterative process unfolds over $T$ steps, culminating in the acquisition of the terminal diffused prompts $\tilde{\bm{V}}_{0}{=}\tilde{\bm{V}}_{\theta}(\tilde{\bm{V}}_{1},\pi,t_{0})$. Upon retrieval of $\tilde{\bm{V}}_{0}$, integration with the text encoder occurs, facilitating the generation of relevant text features. The final stage involves the deployment of these features to predict the classification result for the test image, as delineated by Eq. (1). The diffusion sampling framework throughout the testing phase is shown at the bottom of Figure 3.

## 5 Experiments

We validate the effectiveness of our approach across three widely adopted scenarios for evaluating prompt learning in vision-language models: (1) base-to-new generalization, (2) cross-dataset generalization, and (3) domain generalization.

### 5.1 Experimental setup

15 diverse datasets. For base-to-new generalization and cross-dataset gneralization, we follow CLIP (Radford et al., ) and CoOp (Zhou et al., ) to use 11 image classification datasets, *i.e.*, ImageNet (Deng et al., ) and Caltech101 (Fei-Fei et al., ) for generic object classification, OxfordPets (Parkhi et al., ), StanfordCars (Krause et al., ), Flowers102 (Nilsback & Zisserman, ), Food101 (Bossard et al., ) and FGVCAircraft (Maji et al., ) for fine-grained image recognition, EuroSAT (Helber et al., ) for satellite image classification, UCF101 (Soomro et al., ) for action classification, DTD (Cimpoi et al., ) for texture classification, and SUN397 (Xiao et al., ) for scene recognition.
For domain generalization, we
follow CoOp (Zhou et al., ) with ImageNet as the source dataset, and we select four variants of ImageNet: ImageNetV2 (Recht et al., ), ImageNet-Sketch (Wang et al., ), ImageNet-A (Hendrycks et al., ) and ImageNet-R (Hendrycks et al., ) as the target datasets.

5 prompt learning baselines. For comparative evaluation, we employ several established baselines: (1) Textual prompt learning CoCoOp (Zhou et al., ); (2) Visual prompt tuning (VPT) (Jia et al., ), representing the visual prompt learning method; (3) Multi-modal prompt learning (MaPLe (Khattak et al., ), PromptSRC (Khattak et al., )), and CoPrompt (Roy & Etemad, ) employing prompt learning in both the visual and textual domains. Note that our method acts as a plugin that is easily integrated into each of these methods.

**Table 1: Base-to-new generalization. Prompts are derived from base class examples. The harmonic mean (H) underscores the trade-off in generalization. Top-performing results are emphasized in blue. By integrating our plugin within five different prompt learning methods, we consistently improve their average accuracy across 11 datasets, demonstrating enhanced performance over approaches without our plugin.**
|  | Base | New | H |
| --- | --- | --- | --- |
| VPT (Jia et al., ) | 72.53 | 72.34 | 72.43 |
| + Prompt Diffusion | 74.98 | 74.97 | 74.97 |
| CoCoOp (Zhou et al., ) | 80.47 | 71.69 | 75.83 |
| + Prompt Diffusion | 81.35 | 74.97 | 78.02 |
| MaPLe (Khattak et al., ) | 82.28 | 75.14 | 78.55 |
| + Prompt Diffusion | 83.39 | 77.32 | 80.24 |
| PromptSRC (Khattak et al., ) | 84.26 | 76.10 | 79.97 |
| + Prompt Diffusion | 85.74 | 78.97 | 82.22 |
| CoPrompt (Roy & Etemad, ) | 84.00 | 77.23 | 80.48 |
| + Prompt Diffusion | 86.14 | 80.01 | 82.96 |

Training details. To ensure a fair comparison, we utilize the CLIP-ViT-B/16 as the base pre-training model for CoCoOp (Zhou et al., ), and VPT (Jia et al., ), setting the prompt token count to 4. This configuration is based on recommendations in (Zhou et al., ), indicating optimal performance with a more concise context length. For MaPLe (Khattak et al., ), PromptSRC (Khattak et al., ) and CoPrompt (Roy & Etemad, ), the prompt depth $M$ is adjusted to 9, and we configure the language and vision prompt lengths at two tokens each. In the diffusion preprocessing stage, we adapt the strategy of positional token assignment (Dosovitskiy et al., ) to the input prompts $\tilde{\bm{V}}^{*}$ and the image features $\pi$. Furthermore, the diffusion time step $t$ is encoded as a series of individual tokens, adopting a frequency-based vector representation scheme (Mildenhall et al., ). We set the diffusion time step $t$ as 100 for our experiments. Our transformer-based model architecture is the same as the GPT-2 framework (Radford et al., ). This includes a 12-layer transformer, a linear transformation, and an attention mechanism with 16 heads. The batch size is 32 for all prompt-based models, except for CoCoOp, which is trained with a batch size of 4. Each model leverages a learning rate 0.0035 applied through the SGD optimizer on a single NVIDIA A100 GPU for execution. Code will be made available.

Evaluation setting.
Across all experiments, we benchmark the models’ performance in a 16-shot setting, standardizing the number of training epochs to 50 for each baseline and dataset. The appendix presents a 4-shot experiment, compares outcomes across different epochs, and evaluates various parameter-efficient approaches. For consistency, all results from learning-based methods are computed as an average over three random seeds.

### 5.2 Comparative experiments

Base-to-new generalization.
Table 1(l) shows that various prompting methods, when combined with our prompt diffusion approach, consistently surpass the average performance across all datasets.
Regarding base class accuracy averaged across 11 datasets, our approach advances VPT, CoCoOp, MaPle, PromptSRC, and CoPrompt by 2.54%, 1.08%, 1.11%, 2.25% and 2.48%, respectively, showcasing that our approach strengthens the adaptation of existing methods. When it comes to recognizing new classes, our approach also shows good improvement, with gains of 2.60% for VPT, 1.26% for CoCoOp, 2.78% for MaPle 1.81% for PromptSRC, and 2.87% for CoPrompt, emphasizing its effectiveness in dealing with unseen samples.
Regarding the harmonic mean, which considers both base and new classes, our method retains a superior few-shot generalization capacity across all datasets compared to baseline models. Notably, CoPrompt, with our prompt diffusion, consistently outperforms all other methods across most datasets, demonstrating the advantages of using both modalities in prompt learning.
Our prompt diffusion, applied to different prompt learning models, consistently improves the outcomes by generating more informative and precise prompts.

**Table 2: Cross-dataset generalization. Accuracy (%) evaluation for prompts learned from the source dataset. Our plugin consistently enhances existing prompt learning methods, whether textual, visual, or multi-modal.**
|  | Source | Target |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | ImageNet | Caltech101 | OxfordPets | StanfordCars | Flowers102 | Food101 | Aircraft | SUN397 | DTD | EuroSAT | UCF101 | Average |
| VPT (Jia et al., ) | 68.92 | 93.07 | 89.44 | 64.77 | 67.79 | 84.91 | 23.72 | 66.16 | 45.02 | 37.74 | 67.00 | 63.96 |
| + Prompt diffusion | 70.23 | 94.71 | 90.93 | 65.53 | 68.93 | 85.71 | 24.81 | 66.98 | 46.16 | 39.67 | 67.91 | 65.11 |
| CoCoOp (Zhou et al., ) | 71.02 | 94.43 | 90.14 | 65.32 | 71.88 | 86.06 | 22.94 | 67.36 | 45.73 | 45.37 | 68.21 | 65.74 |
| + Prompt diffusion | 71.98 | 95.07 | 91.11 | 66.73 | 73.52 | 87.18 | 22.23 | 68.25 | 46.84 | 47.13 | 69.53 | 66.76 |
| MaPLe (Khattak et al., ) | 70.72 | 93.53 | 90.49 | 65.57 | 72.23 | 86.20 | 24.74 | 67.01 | 46.49 | 48.06 | 68.69 | 66.30 |
| + Prompt diffusion | 71.23 | 95.98 | 92.49 | 67.17 | 74.13 | 88.24 | 26.23 | 69.43 | 47.95 | 49.73 | 69.53 | 68.09 |
| PromptSRC (Khattak et al., ) | 71.27 | 93.60 | 90.25 | 65.70 | 70.25 | 86.15 | 23.90 | 67.10 | 46.87 | 45.50 | 68.75 | 65.81 |
| + Prompt diffusion | 71.73 | 96.01 | 93.13 | 68.12 | 73.71 | 88.31 | 26.14 | 70.21 | 48.35 | 48.15 | 70.24 | 68.23 |
| CoPrompt (Roy & Etemad, ) | 70.80 | 94.50 | 90.73 | 65.67 | 72.30 | 86.43 | 24.00 | 67.57 | 47.07 | 51.90 | 69.73 | 67.00 |
| + Prompt diffusion | 71.46 | 96.12 | 93.94 | 68.81 | 74.98 | 88.11 | 26.31 | 71.73 | 49.15 | 54.41 | 71.14 | 69.47 |

Cross-dataset generalization.
Our study assesses how well models can adapt prompt learning from one dataset and apply it effectively to different datasets for cross-dataset generalization. We test the zero-shot transfer capabilities of the models on a wide range of 10 datasets. As shown in Table 2, our prompt diffusion substantially improves the average transfer performance of models like VPT, CoCoOp, MaPLe, PromptSRC and CoPrompt, with respective increases of1.15%, 1.02%, 1.79%, 2.43%, and 2.47%.
These results not only confirm the effectiveness of our method in enhancing cross-dataset generalization but also highlight its versatility across various prompt learning methods.

Domain generalization.
The performance of various ImageNet
variants, which have a domain shift compared with the source dataset, is evaluated.
Table 3 summarizes these findings, highlighting not only the improvement in performance across VPT, CoCoOp, MaPLe, PromptSRC, and CoPrompt but also the maintenance of performance on the source dataset itself.
Interestingly, using prompt diffusion with CoCoOp performs better than all multi-modal prompt learning methods on the ImageNet-A dataset. This could be due
to CoCoOp’s emphasis on textual prompt learning, which may be more suited to these types of datasets. ImageNet-A images frequently display anomalies or atypical features, whereas
visual prompts could highlight deceptive or overly intricate aspects, making classification less accurate. Thus, when addressing
real-world datasets like ImageNet-A,
it is advantageous to use
textual prompting
with our prompt diffusion.

**Table 3: Domain generalization. Accuracy (%) evaluation on target datasets using prompts learned from a source dataset. Our method delivers consistent, prompt learning improvements across all datasets.**
|  | Source | Target |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
|  | ImageNet | -V2 | -S | -A | -R | Average |
| VPT (Jia et al., ) | 68.92 | 61.84 | 47.64 | 46.50 | 75.86 | 57.96 |
| + Prompt diffusion | 70.23 | 62.97 | 48.77 | 47.25 | 77.06 | 59.01 |
| CoCoOp (Zhou et al., ) | 71.02 | 64.07 | 48.75 | 50.63 | 76.18 | 59.91 |
| + Prompt diffusion | 71.98 | 65.28 | 50.11 | 52.23 | 77.50 | 61.25 |
| MaPLe (Khattak et al., ) | 70.72 | 64.07 | 49.15 | 50.90 | 76.98 | 60.83 |
| + Prompt diffusion | 71.23 | 65.49 | 50.46 | 52.18 | 78.31 | 62.36 |
| PromptSRC (Khattak et al., ) | 71.27 | 64.35 | 49.55 | 50.90 | 77.80 | 60.65 |
| + Prompt diffusion | 71.73 | 66.33 | 51.21 | 52.02 | 79.86 | 62.88 |
| CoPrompt (Roy & Etemad, ) | 70.80 | 64.25 | 49.43 | 50.50 | 77.51 | 60.42 |
| + Prompt diffusion | 71.46 | 66.01 | 50.71 | 51.75 | 80.76 | 62.30 |

Since such datasets often contain natural images, textual prompts can
leverage the semantic context effectively. On the other hand, in scenarios involving a clear distribution shift (*e.g.* sketch, cartoon), employing a multi-modality prompt with our prompt diffusion is more effective.
From the results of these experiments, our method fosters a level of adaptability that allows models to maintain their initial generalizability even after being fine-tuned to limited datasets.

### 5.3 Ablation experiments

Benefit of the diffusion model.
To confirm that the performance gain of our model can be attributed to the diffusion model, we first conducted the experiments using MLP and transformers as non-generative models, using overfitted prompts as supervision. We also compared it with three widely
used generative
models: generative adversarial networks (GAN) (Goodfellow et al., ), variational auto-encoders (VAE) (Kingma & Welling, ), and normalizing
flows (Rezende & Mohamed, ).
First, we obtain the overfitted prompts with per-sample prompt overfitting.
In the case of
non-generative models, the process involves solely using image features
$\pi$, and then employing overfitted prompts, $\bm{V}^{*}$, for supervising the training of MLP and transformer models.
The input is extended for GANs, VAE and normalizing flows to include image
features $\pi$ and a variable $\epsilon$ sampled from a standard normal distribution $\mathcal{N}(0,I)$.
These inputs, along with the overfitted

**Table 4: Benefit of diffusion model in the base-to-new generalization.**
|  | Base | New | H |
| --- | --- | --- | --- |
| CoCoOp (Zhou et al., ) | 80.47 | 71.69 | 75.83 |
| w/ MLP | 79.18 | 71.98 | 75.41 |
| w/ Transformer | 80.17 | 72.03 | 75.88 |
| w/ GAN | 81.15 | 71.44 | 75.99 |
| w/ VAE | 80.73 | 72.09 | 76.17 |
| w/ Normalizing flows | 80.65 | 72.43 | 76.32 |
| w/ Diffusion | 81.35 | 74.97 | 78.02 |

prompts $\bm{V}^{*}$, are used to supervise and train these three types of generative models.
Table 4 shows the diffusion model
outperforms all variants in accuracy. Specifically, our proposed per-sample overfitting integration with MLP and transformer architectures shows a slight harmonic mean improvement over the CoCoOp
baseline, validating the effectiveness of our per-sample prompt overfitting.
Notably, our diffusion model presents a good increase in accuracy, surpassing the GAN, VAE and normalizing flows models by 2.01%, 1.85% and 1.70%, respectively.

Figure: Figure 4: Effect of number of function evaluation on base-to-new generalization.
Refer to caption: extracted/5955913/images/infer_time.png

Figure: Figure 6: Visualization of generated prompts by ControlNet (Zhang et al., ). We generate diverse images by utilizing three distinct Monte Carlo prompt samples, each derived from our prompt distribution and based on varying random noise. When using ground truth names (red names), images produced by CoCoOp, MaPLe, PromptSRC, and CoPrompt are more realistic, while prompt diffusion incorporates domain-specific details from the test image. Regarding the other classes (blue names), CoCoOp, MaPLe, PromptSRC, and CoPrompt blend true class features with others, potentially leading to confusion, but our plugin using these methods can generate a stylized version of the specified class. This suggests that our plugin enables the distilling of unique domain details from the test image without conflating them with class labels.
Refer to caption: extracted/5955913/images/generated_image.png

Effect of the number of function evaluation.
Our prompt diffusion utilizes the fast ODE-based
sampling strategy introduced by (Zhou et al., )
enabling efficient sampling with a reduced number of timesteps during testing. In Figure 5, we analyze the effect of different numbers of function evaluations (NFE) on both final performance and inference time. Our findings indicate that at an NFE of 5, our method achieves the best balance between performance and prediction time. In comparison to the original CoCoOp, our approach results in only a 0.045-second increase in prediction time while delivering a substantial performance improvement. This highlights the effectiveness of our method in balancing accuracy and computational efficiency.

Impact of iterations on per-sample prompt overfitting. In our prompt diffusion method, per-sample prompt overfitting is crucial to generate optimal prompts during training. Figure 5 shows that as iterations increase, accuracy on novel classes improves for all methods, peaking at iteration 5. This shows that the quality of the optimal prompt directly influences the final performance. Moreover, our prompt diffusion effectively learns the transformation from a vanilla prompt to an optimal prompt throughout training using a diffusion transformer. As a result, during testing, our method can generate a sample-specific prompt for each test sample, thereby improving accuracy.

Visualization of generated prompts. We also visualize the generated per-sample prompts during inference in Figure 6, demonstrating our diffusion prompting method effectively distills unique domain details from the test image without mixing them with class labels.
This shows the better capability of the diffusion model in refining the prompt learning process for vision-language tasks.

## 6 Conclusion

Our approach addresses the limitations of fixed prompts by introducing a method that crafts customized prompts for individual test samples, enhancing model robustness against distributional shifts. The diffusion model serves as the backbone of this method, enabling a generative process that refines prompts from a random initialization to an optimized state, tailored to each specific instance. The versatility and modality-agnostic nature of prompt diffusion mark it as an universally applicable solution that integrates smoothly with existing prompt learning methods, regardless of the data type. The empirical results across a wide range of datasets validate the efficacy of our method, demonstrating its increased robustness in generalization tasks.

## Appendix A Computational graph of diffusion prompting

In this section, we illustrate the computational graph of diffusion prompting in Figure 7. The figure is divided into two parts: the top diagram displays the overall computational structure of the method, while the bottom left part presents a graphical representation of the method’s diffused prompt. In this process, the diffused prompt $\mathbf{V}_{t-1}$ is created through a fusion of $\mathbf{V}_{t}$ and the image descriptor $x$ using diffusion sampling techniques. This results in the prediction $y$, which is generated by combining the diffused prompt, expanded as the "[CLASS]" label $\mathbf{c}$: $\{\mathbf{V}_{0},\mathbf{c}\}$, with the image descriptor $x$. The shaded rectangle in the diagram helps to visually differentiate the components of the process, where dashed arrows indicate the diffusion steps, and solid arrows represent the sampling stages. This figure provides a clear and concise visual representation of the complex processes involved in diffused prompting, highlighting the intricate interactions between different components of the computational model.

## Appendix B Additional related works

Diffusion models. This class of neural generative models is characterized by the employment of stochastic diffusion processes akin to those observed in thermodynamic systems (Sohl-Dickstein et al., ; Song et al., ). The operational principle of these models involves a sequential noise addition to data samples, followed by a learned neural network’s effort to reverse this process. This is achieved by gradually denoising the noise-saturated sample to retrieve data reflecting the trained data distribution. Significant strides in the realm of image generation have been accredited to the works of Ho et al. (Ho et al., ) and Song et al. (Song et al., ), while Dhariwal and Nichol (Dhariwal & Nichol, ) have been pivotal in pioneering classifier-guided diffusion for generation under specific conditions. Building on this foundation, GLIDE (Nichol et al., ) has further refined the methodology by incorporating conditioning on textual representations derived from CLIP. The concept of classifier-free guidance introduced by Ho et al. (Ho & Salimans, ) has brought forward a method of conditioning that judiciously balances fidelity and diversity, leading to notable enhancements in model performance (Nichol et al., ). However, guided diffusion models typically necessitate an extensive corpus of image-annotation pairs for effective training, prompting Hu et al. (Hu et al., ) to suggest the novel concept of self-guided diffusion models. More contemporary developments include Hyperdiffusion (Lutati & Wolf, ; Erkoç et al., ), which targets the generation of implicit neural representations and 3D reconstruction through diffusion in weight space.
To the best of our knowledge, we are the first to introduce diffusion models into the realm of prompt learning. Our diffusion prompting involves gradually refining prompts with a diffusion transformer, which leads to the development of custom prompts tailored to each sample, thereby enhancing the accuracy of predictions and their generalization across downstream tasks.

## Appendix C Hyperparameter sensitivity and few-shot.

Table 5 presents a comparison of our model’s performance over different epochs relative to CoOp’s training duration of 200 epochs. Our model reaches convergence around the 50-epoch mark and surpasses the performance of CoOp after 200 epochs. Additionally, we also conduct a few-shot learning experiment (4-shot) similar to those conducted with CoOp and CoCoOp, as shown in Table 6. In these comparisons, our model consistently achieves improved performance across a range of datasets.

## Appendix D Parameter-efficient comparison.

Table 7 contrasts our approach with four other parameter-efficient fine-tuning techniques. Our integration with MaPLe (Khattak et al., ) showcases superior average performance, underscoring its superior ability to generalize in comparison to other parameter-efficient fine-tuning approaches. Furthermore, we have applied our plugin in conjunction with LLU (Ibing et al., ) in a base-to-new setting, where it also exhibits enhanced performance relative to LLU alone.

**Table 5: Comparison with CoOp for various epochs.**
| Epochs | Base | New | H |
| --- | --- | --- | --- |
| 10 | 80.29 | 73.51 | 76.26 |
| 50 | 81.35 | 74.97 | 77.72 |
| 100 | 81.47 | 74.88 | 77.70 |
| 200 | 81.78 | 74.24 | 77.65 |

**Table 6: Comparison with 4-shots on domain generalization. Our results are competitive for all domains.**
|  | Source | Target |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
|  | ImageNet | -V2 | -S | -A | -R | Average |
| VPT (Jia et al., ) | 69.24 | 62.13 | 45.78 | 48.16 | 72.91 | 57.37 |
| + Prompt Diffusion | 70.27 | 63.83 | 48.15 | 50.97 | 76.15 | 60.12 |
| CoCoOp (Zhou et al., ) | 70.13 | 63.05 | 46.48 | 49.36 | 73.80 | 58.17 |
| + Prompt Diffusion | 70.96 | 64.12 | 48.92 | 51.47 | 76.93 | 60.75 |
| MaPLe (Khattak et al., ) | 70.72 | 64.07 | 49.15 | 50.90 | 76.98 | 60.83 |
| + Prompt Diffusion | 71.23 | 65.24 | 50.21 | 51.93 | 78.06 | 62.11 |

**Table 7: Comparison with parameter-efficient fine-tuning methods in the base-to-new setting across 11 datasets.**
|  | Venues | Base | New | H |
| --- | --- | --- | --- | --- |
| ProGrad (Zhu et al., ) | ICCV 23 | 82.79 | 68.55 | 74.46 |
| CLIP Adapter (Gao et al., ) | IJCV | 82.62 | 70.97 | 76.02 |
| LLU (Ibing et al., ) | CVPR 23 | 83.48 | 74.47 | 78.46 |
| MaPLe (Khattak et al., ) | CVPR 23 | 82.28 | 75.14 | 78.55 |
| LLU + Diffusion Prompt |  | 84.45 | 75.99 | 79.17 |
| MaPLe + Diffusion Prompt |  | 83.39 | 77.12 | 79.96 |

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: 1.  Is the author's approach a two-stage process, starting with a prompt study followed by a prompt proliferation.
2. Diffusion models incorporate randomness in the generation process, which may lead to uncontrollable fluctuations in the generated prompts and thus affect the robustness of the model. How to cope with the randomness of the generated prompts and avoid the instability of prediction caused by it?
3. The authors' approach seems to be applicable only to VPT-shallow prompt types, and whether the authors' approach can be migrated to the VPT-deep prompt learning paradigm.
soundness: 3
strengths: 1. The method in this paper generates customized prompts for each sample by gradually optimizing the prompts through diffusion, which enhances the accuracy of prediction and generalization across downstream tasks.
2. The diffusion prompting method in this paper is a plug-and-play module that can be seamlessly integrated into existing textual, visual, or multimodal prompt learning methods.
3. The method in this paper improves the prompt learning process by efficiently extracting unique domain details from test images without mixing them with class labels.
summary: This paper proposes a method called Prompt Diffusion, which employs a diffusion model to progressively refine prompts, enabling customized prompts for each sample. By introducing a technique for creating tailored prompts for individual test samples, this method addresses the limitations of fixed prompts, enhancing the model's robustness to distribution shifts. Empirical results on extensive datasets validate the effectiveness of this approach, demonstrating its robustness in generalization tasks.
weaknesses: 1.  The authors' method requires stepwise optimization of the prompts and may require several iterations to obtain optimal results, in addition, the introduction of a diffusion model increases the complexity of the system, and therefore whether the training time is likely to be relatively long.
2.  Whether the authors' approach is a two-stage process, where prompt learning is performed first, followed by diffusion of the prompts, and the final model performance relies on the goodness of the previously learned prompts. In addition, the diffusion process relies on random noise vectors to generate the prompts and therefore may be sensitive to noise, which may affect the stability of the final performance.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: 1. I am curious about the setting of the two loss weights β in Equation (8). Can further experimental analysis be provided?
soundness: 3
strengths: 1. Experiments have shown that the proposed method outperforms baseline methods.​

2. The overall idea is intuitive and straightforward, addressing the limitations of fixed prompts by leveraging diffusion models to generate over-fitted prompts per sample, which enhances model robustness against distribution shifts.
summary: In this paper, the authors introduce prompt diffusion, which utilizes a diffusion model to refine prompts for each input image, thereby enhancing the model's ability to generalize across different distributions. The proposed prompt diffusion is a straightforward plug-and-play module that can be seamlessly integrated into existing prompt learning frameworks.
weaknesses: 1. Considering that the proposed method is conducted on per sample. during training, does it introduce a significantly larger computational load compared to conventional prompt learning methods? Can a comparative analysis be provided to address this concern?

2. While the proposed method is plug-and-play and the pipeline figure demonstrations are based on CoCoOp, it would be beneficial to include sections addressing visual prompt tuning and multi-modal prompt tuning.  Additionally, the method emphasizes the meta-net π within CoCoOp, but it is unclear how it handles other prompt learning methods that do not involve π, such as VPT and MaPLe.

3. The length of prompts in prompt learning methods can affect the final performance.  Does the proposed method also encounter similar situations?  It is encouraged for the authors to supplement relevant ablation studies to address this concern.

4. There are also some works in the field of prompt learning that address the limitations of fixed prompts by generating instance-level prompts (e.g. [1]).  It is recommended that the authors supplement the related work to make the paper more comprehensive.

[1] Xinyang Liu ,et al. Patch-Token Aligned Bayesian Prompt Learning for Vision-Language Models. UAI 2024

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: 1. What are the key motivations behind using diffusion models for prompt learning, and how does it address the limitations of fixed prompts?
2. How does Prompt Diffusion leverage the diffusion model to gradually transition from a random to a sample-specific prompt?
3. In what ways does Prompt Diffusion enhance generalization capabilities across base-to-new, cross-dataset, and domain generalization tasks?
4. How does Prompt Diffusion ensure compatibility with existing prompt learning models across textual, visual, and multimodal prompts?
soundness: 2
strengths: 1.	Introduces an innovative, modality-agnostic diffusion process that significantly enhances robustness in prompt-based learning.
2.	Demonstrates consistent empirical improvements across various prompt learning tasks, supporting the efficacy of diffusion models.
3.	Efficient design reduces inference time, making it suitable for diverse real-world applications.
summary: This paper introduces a novel framework, Prompt Diffusion, which aims to improve the generalizability and robustness of prompt-based learning across various modalities (e.g., visual, textual, multimodal). In prompt-based learning, especially for foundation models in zero-shot and few-shot learning, fixed prompts often suffer from distributional shifts, impacting performance on unseen data. Prompt Diffusion leverages a diffusion model to refine prompts gradually, transforming them from a generic to a sample-specific prompt. This process enhances the robustness and adaptability of prompts across datasets with distinct distributions, providing a plug-and-play solution compatible with existing prompt-learning methods.
weaknesses: 1.	The paper does not fully articulate the specific limitations of the SOTA prompt mehthods in adapting to distributional shifts in data, which creates ambiguity around the critical nature of these issues within broader prompt-learning applications. To make this critique more actionable, the authors could quantify the performance degradation caused by these shifts in existing methods to better contextualize the importance of their contribution.  Specific examples are not enough to illustrate the problem.
2.	Although the diffusion model is proposed to generate sample-specific, customized prompts, the paper does not clearly explain why diffusion was chosen over other, potentially simpler methods. This raises questions about the model's unique contributions and practical effectiveness. For instance, if simpler statistical methods like ProDA[1] are available, what advantages does the complex diffusion model offer? Moreover, there are already several statistical approaches for prompt learning, such as Bayesian Prompt Learning[2], which the authors could consider referencing.
3.	The approach has limited empirical exploration outside the image-text domain, raising questions about its generalizability to other modalities. To strengthen this point, the authors could discuss the potential challenges and adaptations needed to apply their method to other modalities, such as audio or video. 
4.	The high resource demands of diffusion models, including substantial GPU and training time requirements, make them impractical for parameter-efficient methods such as prompt learning. The complexity and cost of implementing diffusion models in this context undermine their accessibility and practicality. 

[1] Lu, Yuning, et al. "Prompt distribution learning." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.

[2] Derakhshani, Mohammad Mahdi, et al. "Bayesian prompt learning for image-language model generalization." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

Meta-review: The main idea of this paper is to apply diffusion modeling to refining instance-specific prompts, and the method is tested on multiple image classification benchmarks. The reviewers generally liked the idea of prompt diffusion but did not find the motivation strong enough. In particular, the paper does not explain clearly why diffusion is the right choice compared to other designs. After reading the paper, the AC has the same doubt as the reviewers. Moreover, the reviewers pointed out that the method has a much higher complexity than other baselines due to the diffusion process, thus diverging from the principle of parameter-efficient fine-tuning. The AC also agrees with this comment and feels that due to this heavy design the method is less likely to be adopted in practice and therefore has limited value to the community.

Decision: Reject