Diff-Prompt: Diffusion-Driven Prompt Generator with Mask Supervision
Authors: Weicai Yan
Wang Lin
Zirun Guo
Ye Wang
Fangming Feng
Xiaoda Yang
Zehan Wang
Tao Jin
Venue: ICLR 2025 Poster (2025)
Keywords: prompt learning
diffusion model
multimodal learning
Primary area: transfer learning, meta learning, and lifelong learning

Abstract: Prompt learning has demonstrated promising results in fine-tuning pre-trained multimodal models. However, the performance improvement is limited when applied to more complex and fine-grained tasks. The reason is that most existing methods directly optimize the parameters involved in the prompt generation process through loss backpropagation, which constrains the richness and specificity of the prompt representations. In this paper, we propose Diffusion-Driven Prompt Generator (Diff-Prompt), aiming to use the diffusion model to generate rich and fine-grained prompt information for complex downstream tasks. Specifically, our approach consists of three stages. In the first stage, we train a Mask-VAE to compress the masks into latent space. In the second stage, we leverage an improved Diffusion Transformer (DiT) to train a prompt generator in the latent space, using the masks for supervision. In the third stage, we align the denoising process of the prompt generator with the pre-trained model in the semantic space, and use the generated prompts to fine-tune the model. We conduct experiments on a complex pixel-level downstream task, referring expression comprehension, and compare our method with various parameter-efficient fine-tuning approaches. Diff-Prompt achieves a maximum improvement of 8.87 in R@1 and 14.05 in R@5 compared to the foundation model and also outperforms other state-of-the-art methods across multiple metrics. The experimental results validate the effectiveness of our approach and highlight the potential of using generative models for prompt generation. Code is available at https://github.com/Kelvin-ywc/diff-prompt.

Published as a conference paper at ICLR 2025
DIFF-PROMPT: DIFFUSION-DRIVEN PROMPT GENERATOR WITH MASK SUPERVISION
Weicai Yan, Wang Lin, Zirun Guo, Ye Wang, Fangming Feng, Xiaoda Yang, Zehan Wang, Tao Jin Zhejiang University
(yanweicai, linwangiw, ger, yeu. fangming xi xiaodayang, wangzehan01, jin ijjiinnnneezjuuuu www.j/u.uuuuu..c....mm edu.cn
ABSTRACT
Prompt learning has demonstrated promising results in fine-tuning pre-trained multimodal models. However, the performance improvement is limited when applied to more complex and fine-grained tasks. The reason is that most exist- ing methods directly optimize the parameters involved in the prompt generation process through loss backpropagation, which constrains the richness and speci- ficity of the prompt representations. In this paper, we propose Diffusion-Driven Prompt Generator Priff-Prompt) aiming to use the diffusion model to generate rich and fine- mme-grained prompt information for complex downstream tasks. Specif- ically, our approach consists of three stages. In the first stage, train Mask- VAE to compress the masks into latent space. In the second stage, we lever- age an improved Diffusion Transformer (DiT) train prompt generator in the latent space, using the masks for supervision. denoising process of the prompt generator with the pre-trained model in the se- mantic space, and use the generated prompts to fine-tune the model. We con- duct experiments on a complex pixel-level downstream task, referring expres- sion comprehension, and compare our method with various parameter-efficient fine-tuning approaches. Diff-Prompt achieves maximum improvement of 8.87 in R@] I and 14.05 in R@5 compared to the foundation model and also out- performs other state-of-the-ee mental results validate the effectiveness of our approach and highlight the po- tential of using generative models for prompt generation. Code available at https:/ tht.....huu.. www./eelvin-yww/dieeeooo
the third stage, we align the
methods across multiple metrics. The experi-
INTRODUCTION
Pre-trained multimodal models (Radford et al., 2021; Jia et al., 2021; Yuan et al., 2021; Pham et al., 2023: Li et al., 2022b; Zhang et al., 2022a; Li* et al., 2022) have received widespread attention due to their strong generalization capabilities. Taking the Contrastive Language-Iman (CLIP) (Radford et al., 2021) as an example, it is pre-trained on web-scale data, which enables it to learn joint vision-language representations. Fine-tuning techniques enable these models to be effectively applied to downstream tasks. Early approaches utilized full fine-tuning; however, these methods demands considerable computational resources and compromise the generalization capabilities of the pre-trained model.
Pretraining
Prompt learning (Lester et al., 2021; Jia et al., 2022; Zhou et al., 2022b; Khattak al., 2023a; Zhang et al., 2024a), as an efficient fine-tuning method, has garnered extensive research interest. It involves designing prompts either manually automatically for fine-tuning pre-trained models. The advan- tages include significantly reducing training resources while preserving the original generalization capabilities. As shown in Fig. (a)), most current prompt learning methods follow the first two paradigms. For the first paradigm (Jia et al., 2022; Zhou et al., 2022b; Wang et al., 2022a; MA et al., 2023; Fang et al., 2023), learnable prompts are added to the encoder input of the pre-trained model. These approaches have certain limitations: first, prompts for different modalities are learned inde- pendently, preventing the establishment of inter-modal connections. Second, only global prompts
"Corresponding author.

---

Published as a conference paper at ICLR 2025
Trusd:
Teldy
Larguge
Tidkl
Lanpunge
Espoder
VIKM
tegulaboticn
siocal
00.
Viioo brusder
Vicder
Prompte
Interaction
Genistate
AVC000 bocill
(a)
(b)
Figure 1: (a) Comparison between mainstream prompt learning methods (the first two paradigms) our
Diff-prompt paradigm. (h) Comparison of different efficient fine-tuning methods on the RefCOCO dataset, with the x-axis representing R@ I, the y-axis representing @@5, and bubble size indicating the total model parameters. Diff- hiff-Prompt achieves higher performance at the cost of using partial parameters.
can be learned for all training data, which restricts the prompting capability. Subsequent works fol- low the second paradigm for improvements (Khattak al., 2023a; Shi et al., 2024; Qiu et al., 2024; Roy & Etemad, 2024), using networks or regularization methods to establish connections between prompts of different modalities. However, we believe that the above methods update prompts the prompt generation process goal-driven manner, which significantly limits the richness of the prompts. When applied complex and fine-grained downstream tasks, their prompting capability is limited. As shown in Fig. 1(b), for a multimodal localization task that requires consideration of the complex relationships between modalities and multimodal understanding. VPT (Jia al., 2022) adds prompts only on the visual modality side, and its performance is even inferior to that of the foundation model. Other prompt learning methods also show limited performance improvement.
To address the above issues, we consider how to generate rich prompts that can provide sufficient in- formation to the pre-trained model, even for fine-grained downstream tasks. Inspired by the powerful feature extraction and generation capabilities of diffusion models, this paper proposes Diff-Prompp which uses the diffusion model to generate rich prompt information. The training process of the dif- fusion model employs masks supervision to inform the model which parts of the input image need to be emphasized for a given caption. Specifically, Diff- feff-Prompt consists of three stages. In the first stage, map the masks to latent space, extracting dense information while reducing computational load in the later stages. In the second stage, we train a prompt generator with mask supervision in latent space using improved DIT model, conditioned emphasized parts of the image. In the third stage, we align the prompt generator's 's output with the pre-trained model semantically better integrate the generated prompts into the pre-trained model. Finally, we concatenate the generated prompts with a few learnable global prompts to supplement universial knowledge. We conduct experiments on a fine- fine-grained multimodal task, specifically the referring expression comprehension, and evaluate on multiple metrics. The results demonstrate that our method outperforms other existing efficient fine-tuning methods, validating its effectiveness.
the image and caption to generate
Our main contributions are as follows: (1) We train Mask-VAF and diffusion-driven prompt generator to generate rich prompt information in the mask latent space. (2) We align the gener- ated prompts with the pre-trained model in the semantic space effectively guide the pre-trained model. (3) We conduct experiments on a complex fine- mne-grained mutli-modal downstream task, and the experimental results demonstrate the effectiveness of our method.
2 RELATED WORK
2.1 VISION-LANGUAGE MODELS
Vision- models trained on large-scale data exhibit strong feature extraction and generaliza- tion capabilities. These models include CLIP (Radford al., 2021), ALIGN (Jia et al., 2021), Flo- rence (Yuan et al., 2021), BASIC (Pham et al., 2023), and OpenCLIP (Ilharco et al., 2021). When addressing downstream tasks, they considered ideal choices. Additionally, some works pro-
2

---

Published as a conference paper at ICLR 2025
pose pre-trained models for specific tasks, allowing for easy transfer to particular data distribution. LSeg (Li et al., 2022), and CLIPSeg (Lüddecke & Ecker, 2022) are used for segmentation, BLIP (Li et al., 2022a) is used for visual question answering, while GLIP (Li et al., 2022b), PPMN (Ding et al., 2022) and Grounding DINO (Liu et al., 2023) are used for localization tasks.
2.2 PROMPT TUNING
Prompt learning is initially applied in the field of natural language processing (NLP)(Petroni et al., 2019; Brown et al., 2020; Wallace et al., 2019; Shin al., 2020; Li & Liang, 2021; Lester al., 2021), where it achieves excellent performance. The core idea is to design manually crafted or auto- matically learned prompts to fine-tune pre-trained models. This approach allows pre-trained models to adapt to downstream tasks while avoiding the excessive resource consumption that comes with fully fine-tuning the models. Subsequently, prompt learning has been widely applied to the fields of computer vision (CV) (Jia et al., 2022; Bahng et al., 2022) and multimodal learning (Zhou et al., 2022b;a; Zang et al., 2022; Khattak et al., 2023a; Cao et al., 2023; Guo et al., 2024a; Fu et al., 2024; Qiu et al., 2024; Guo et al., 2024b; Yang et al., 2024b; Yan et al., 2024; Jin et al., 2024; Li et al., 2024a; Shi et al., 2024; Roy & Etemad, 2024). VPT (Jia et al., 2022) concatenates learnable prompts to the imput of the vision encoder layer, while CoOp (Zhou et al., 2022b) concatenates learn- able prompts to the input of the language encoder. These works incorporate prompts only within single modality. To enable communication between modalities, MaPLe (Khattak et al., 2023a) and UPT (Zang et al., 2022) introduce prompts for different modalities and establish connections between the prompts of these modalities. Recent work attempts to generate input-specific prompts. QNet (Shi et al., 2024) generates prompts using Quaternion Networks. Additionally, more work explore broader application scenarios for prompt learning. For example, L2P (Wang al., 2022c) and S prompts (Wang et al., 2022a) investigate the performance of prompt learning in continual learning. TPT (Shu et al., 2022) and PromptAlign (Hassan et al., 2023) applies prompt learning in the context of test-time adaptation.
2.3 DIFFUSION MODELS
Diffusion Models are a type of generative model that generates new data by simulating a gradual reverse process of data distribution. DDPM (Ho et al., 2020) introduced a method for generating data through the stepwise addition and removal of noise. IDDPM (Nichol & Dhariwal, 2021) im- proved upon DDPM by employing more efficient training strategies and finer denoising steps. To enhance the generation efficiency of diffusion models, DDIM (Song al., 2020) is non-Markov diffusion model that allows skipping certain steps during inference, while LDM (Rombach et al., 2022) performs diffusion by mapping data into latent space. The subsequent work, DIT (Peebles & Xie, 2023a), combines diffusion models with transformer architecture, leveraging the strong rep- resentational capabilities of transformers ControlNet (Zhang et al., 2023) is type of controllable generative model that introduces additional conditional information to regulate the generation process. Works (Ruiz et al., 2023; Gal et al., 2022; Lin et al., 2024a;b) investigate the customization of diffusion models.
improve generation quality. In terms of specific tasks,
3 PRELIMINARY
3.1 PROMBLEM FORMULATION AND FOUNDATION MODEL
Given an image u and caption q, the objective of the task is to predict the location o of the described object within the image. GLIP (Li et al., 2022b) is used as the foundation model, which primarily consists of a vision encoder Enc,, a language encoder Ency(.), and a downstream head Head( ) The image is first divided into multiple patches, which are then embedded into Eg. The caption ş is tokenized and embedded into EE/. These embeddings are subsequently fed into the modality encoder to generate the corresponding modality features. These features are then passed to the downstream head to predict bounding boxes ò for referring objects. For GLIP, the downstream head is a region proposal network (RPN). RPN uses sliding window to generate multiple candidate regions and then adjusts the positions and sizes of these anchor boxes to generate high-quality candidate regions. The GLIP training loss Lrase is the sum of the classification loss Lats and the localization loss Lioc-
3

---

Published as a conference paper at ICLR 2025
Encoder Layer
Endoder Layer
Vision Encoder Layer
Viion
Vision Adapter
Prompt
Downstream Head
Generator
Language Adapter
Language Encoder Layer
Language Layar
Language Encoder Layer
Teddy bear red dresss
GP
Global Prompts
Visual Embeddings
Embecdings
Input-spocific
Generaned Prompts
Figure 2: The framework of Diffusion-Driven Prompt Generator (Diff-Promph We fully utilize diffusion model as the prompt generator, which generates prompts conditioned on given image and caption. The generated prompts are then mapped into input-specific prompts through modality- specific adapters. These input-specific prompts are concatenated with global prompts of equal length to form the final prompts, which are used to fine-tune the pre-trained model.
3.2 DEEP PROMPTING
For an encoder Enc(:) composed of Nj stacked attention layers [L.) N the deep prompting technique introduces prompts into the first D attention layers. For the ith attention layer, the prompt P. E R. N xd, is concatenated with the embedding E, as the input, where N, denotes the prompt length and dp is the dimension size.
3.3 DIFFUSION MODELS
For computational efficiency, typically VAE encoder compress the target, then train a diffusion model in the latent space, and finally use VAE decoder 'D to reconstruct the generated target. Our training objective to obtain diffusion model сœ that can predict noise based on given condition C. During inference, Gaussian noise is randomly sampled, and multi-step denoising applied to obtain the generated result. Additionally, the skip-step strategy from DDIM is used to accelerate the generation process.
DIFF-PROMPT: DIFFUSION-DRIVEN PROMPT GENERATOR
We propose the Diff-Prompt, which aims efficiently fine-tune pre-trained foundation models using prompts generated by diffusion model. Diff-Prompp consists of three stages. In the first stage, train a Mask- VAE to compress the mask into low-dimensional space. In the second stage, we enhanced DIT (Peebles & Xie, 2023b) the prompt generator, generating prompts (denoted as generated prompts) given an image and caption. In the third stage, we freeze the backbone network, Mask-VAE. and the prompt generator trained in the first two stage, and design modality- specific adapters to align the latent features generated by prompt generator with the foundation model, mapping the generated prompts the representations of the foundation model. We then introduce small number of learnable global prompts to complement universal knowledge, thus generating more expressive features.
4.1 MASK- VAE TRAINING
In the first stage, we aim to use the DiT model with mask supervision to generate visual prompts that locate the approximate position of the object referenced by the caption in the image, thereby
4

---

Published as a conference paper at ICLR 2025
Forward Process
XT
Sample Process
21
By
2
2
Z
Prompt Generator
Prompt Generator
N
Prompt Generator
N
Teddy
Condition
VAE Eneader
VAE Decodor
N
Random Caussian noise
N Predicted Noise
Element-wite Subtraction
Genarated Prompts
Figure 3: Forward and Sample Process of the Prompt Generator.
aiding the foundation model in reasoning. To reduce computational complexity, follow the LDM approach by first training Mask-VAn VAE to compress the masks into the latent space. Given mask IR. 1 HXW where H denotes the height and W denotes the width, train encoder & and decoder D. The encoder E encodes m into mean and variance vector in the latent space, and then the latent feature sampled from the Gaussian distribution:
μt a {mm) ~~ ~~,
(1)
The decoder then reconstructs the latent feature back to the original mask r
m = D(2).
(2)
To train the Mask- VAE, we use the following loss function:
Lvae ||m
ml (W
(3)
where À is the scale parameter.
4.2 VISUAL PROMPT GENERATION USING DIFFUSION MODEL
In the second stage, given input image and caption we trained prompt generator using the mask as guidance. For the diffusion process, the mask m first compressed into a latent feature 2o by the encoder E. We continuously add noise to z,repeating T forward times until it completely becomes Gaussian noise:
/ 20 + VI aget,
E ~O0,I.
(4)
The following loss is used to train the prompt generator Ep with condition C:
Lo []c C) 2 C = [Emb,(w). (mb, Emb,
(5)
where E0(2,, C`) is the predicted noise, Emb,(.) is the image embedding layer, Emb, is the lan- guage embedding layer, (mb, (.) the timestep embedding layer, and is the timestep.
For the generation process, prevent the prompt generation from taking too much time, choose to use DDIM for accelerated sampling. The reverse process goes through Teample timesteps:
1 2 (0(2,, C)-
(6)
As the diffusion steps of the prompt generator increase, the fusion of text and image informa- tion deepens. The latent features at the intermediate steps can effectively capture the fusion be- tween different modalities, which is consistent with the encoding process. By aligning the dif- fusion process of the prompt generator with the encoding process of the pre-trained model's en- coder, we can effectively inject modality fusion information into the encoder. Throughout the dif- fusion process of the prompt generator, we retain D latent features z = [λ ..,...., 2ip_11 where (io, io-1)
{0, 1.... Tsample): ). These generated prompts represent the degree of modal- ity information fusion, and are subsequently mapped to input-specific prompts and integrated into the pre-trained model's encoder.
5

---

Published as a conference paper at ICLR 2025
4.3 VISUAL PROMPT TUNING WITH FOUNDATIONAL MODELS
In the second stage, we retain the latent features from the denoising process of the prompt generator as prompts. Considering that the prompt generator and GLIP are in different semantic spaces, we first use the Mask-VAE decoder in the second stage to reconstruct the latent prompts, generating prompts of the same size as the image. These generated prompts form a saliency map. which in- forms the foundation model about which parts of the image require more attention. Furthermore, to integrate the generated prompts into the pre-trained model, we design an adapter for each modality, namely Adapter, and Adapter, The modality-specifie adapter aligns the generated prompts with the space of the modality encoder. This design not only enables cross-modality prompting over time but also facilitates communication between different modalities:
Pj Addpter,(((E,,))
Pj= = (D(3,),, j j= 0,,,..... D -
(7)
The input-specific prompts are designed to tailor the prompts the input data. At the same time, we added the same number of learnable global prompts, namely global visual prompts, {GPj 1 and global textual prompts, {GP]) P-1 The input-specific prompts, global prompts, and embed- dings are concatenated and fed into the encoder, which ultimately produces the model's output:
j+1 I.3 ((P)), GP,, BT),, == ... 11
(8)
E j+i (EE"))
j D..... Ni 1,
(9)
Ő o= Head EN,, E,,,,
(10)
where represents v(ision) or I(anguage) modality.
Throughout the entire third stage, we only train the parameters in the adapter and global prompts. To train our model, we select the same loss function the one used in the foundation model.
EXPERIMENT
In this section, we first introduce the experiment setup, followed by quantitative analysis. Next, we conduct qualitative analysis, ablation studies, and in-depth analysis on the RefCOCO val dataset. In Appendix D, we explore the zero-shot capabilities of Diff-Prompl In Appendix E, we provide more visualization results compared with other methods. In Appendix C, We conduct an ablation experi- ment on prompt selection, and in Appendix F, we carry out further in-depth analysis on Flickr30k. Finally, we discuss the limitations in Appendix H.
5.1 EXPERIMENT SETUP
Dataset. We conducted experiments on two vision-language understanding datasets, RefCOCO (Kazemzadeh et al., 2014) and Flickr30k (Plummer et al., 2016). RefCOCO includes training set, two test sets (testA and testB), and a validation set (val). TestA contains multiple people, while testß contains multiple non-human objects. The Flickr30k dataset includes the train, test, and val set.
Evaluation Metrics. We use Recall at K (R@K) and Upper Bound (UB) as the evaluation metric. R@K indicates the proportion of times the model correctly identifies the target within the top K retrieval results, reflecting the model's ability to find the correct match within a given ranking range. UB evaluates the proportion of target presence among all prediction results. Specifically, we chose R@I 1, R 5 and UB as the evaluation standards. R@I I measures the model's ability for precise retrieval, R@S reflects the model's overall recall ability, and UB indicates the model's potential.
Baseline. we conduct a quantitative analysis, selecting GLIP-T(A) (Li et al., 2022b) as the foun- dation model. We compare our model with two efficient parameter tuning methods: adapter and prompt tuning. For the adapter method, we choose Tip-adapter (Zhang et al., 2022b), Meta- adapter (Song et al., 2023), CLIP-Adapter(Gao et al., 2024), MMA(Yang et al., 2024a) for com- parison. For MMA, we introduce the adapter in the last Transformer layer of the encoder. For Tip-Adapte Meta-adapter, and Edapter we introduce the adapter at the encoder's output. For prompt tuning, select VPT(Jia al., 2022), VFPT (Zeng et al., 2024), CoOp(Zhou et al., 2022b), S-Prompts(Wang et al., 2022b), MaPLe(Khattak et al., 2023a), and FedTPG(Qiuu et al,, 2024).

---

Published as a conference paper at ICLR 2025
Table 1: Performance Evaluation on RefCOCO and Flickr30k datasets. Bold: best results, underline: second
best results.
RefCOCO (testA)
RefCOCO (lesiB)
RefCOCO (val)
Flickr30k (test)
Flickr30k (val)
Method
R@1
R@5
UB
R@1 1
R@55
UB
R@1 1
R@5
UB
R
R@5
UB
R@1 1
R@5
UB
GLIP-TIA)
30.21
80.66
93.44
32.93
88,30
31.82
80.00
91.42
45.57
63.72
70.55
44.87
70.17
Tip-adapter
34.68
86.24
97.25
32.83
78.39
94.02
34.56
83.06
95.47
50.16
74.89
84.53
48.
73.54
85,
CLIP CLIP-Adapter
34,08
85.81
98.18
32.76
76.72
93.39
33.91
81.93
96.24
49.30
73.12
84.78
47.15
71.72
83,24
Meta-adapter
35.02
87.96
98.64
33.29
78.67
95. 14.14
36.54
85.09
96.34
51.32
75.36
85.16
49.18
74.87
88.14
MMA
36.68
89.03
99.13
34.67
79:06
95.88
35.28
86.46
97.18
52.60
77.04
85.77
51.43
76.28
89.46
$0.40
81.04
84.56
44.27
VEPT
37.24
91.45
98.23
31.98
79.36
97.74
34.92
87.93
98.41
55.82
76.16
88.67
51.53
75.94
88.31
CoOp
36.89
93.18
99.61
32.29
82.89
97.21
35.31
88.62
98.01
51.29
74.85
88.52
50.95
74.79
87.96
S-Prompes
93.11
97.21
32.80 12.84
81.86
90:25
35.32
87.99
98.65
53.09
76.58
88.92
52.15
76.03
88.57
MaPLe
37.72
91.97
99.13
32.70
82.22
98.88
34.81
87.54
98.36
55.69
80.24
90.50
54,96
79.91
90.49
FedTPG
37.65
93.78
99.61
33.22 25
82.81
97.68
35,29
88.34
99.03
74.32
87.98
51.48
74,07
87.54
FedTPGss
32.76
90.98
99.58
29.9
75.66
97.94
33.73
83.46
98.90
57.95
80.62
90.17
56.08
79.96
90.13
Dift- Diff-Prompt
39.08
94.71
99.63
36,09
85.67
99.00
37.94
90.55
99.37
59.53
81.85
90.46
57.39
81,20
90,54
Experiment Detail. For the Diff-Prompt, in the first stage, we train Mask- VAE on the RefCOCO dataset for 200 epochs, setting the batch size to 128, the learning rate to 0.05, and 0.0003. In the second stage, we train the prompt generator. During the training phase, we set T forward = 100 and use squaredcos, Satreddoosccpp__e__ee cap_v2 v2 as the noise scheduler. In the sampling phase, we use DDIM and set the number of sampling timesteps Tsample to 25, with the batch size set to 128 and the number of epochs to 100. In the third stage, for the input of the vth attention layer, we select the latent features at step 25- 2i the generated prompts. The visual embedding size is set to 96, and the language embedding size is set to 768. The learning rate is set to 0.0001, and Adam is used as the optimizer. In Appendix C, we discuss the rationale behind this choice. The specific architectures of Mask- Mas,-VAE, the prompt generator and the adapters are detailed in the Appendix A. Additional experimental details are provided in Appendix B.
5.2 QUANTITATIVE ANALYSIS
The experimental results are shown in Tab. 1. What we can see from the results is that Diff-Prroppt surpasses other adapter and prompt tuning methods across all metrics. Specifically, for the Ref- COCO dataset, Diff Piff-Prompt shows performance improvements across all three subsets. Compared to the GLIP-T(A) model, Diff-Prompt achieves a maximum increase of 8.87% in R@I and 14.05% in @@S in testA dataset. Furthermore, we observed that the more interaction between prompts, the more significant the performance improvement. VPT and VFPT add prompt information only on the visual side, resulting in less performance improvement compared to methods that incorporate prompts in both modalities, such as MaPLe and FedTPG. Additionally, is found that CLIPAdapter, CoOp, S-Prompts, MaPLe, and FedTPG all show improvements across metrics on the testA subset, but there is slight decrease in performance on testB for some metrics. Based on the distribution of data in testA and testB, we can infer that the model overfits images in the person class during training. leading to slight decline in performance for other classes. Notably, the CLip-Adapter method outperforms VPT but falls short of CoOp. This is because CLIP-Adapter maps modality features but fails to provide additional auxiliary information to the pre-trained model, thus limiting the performance enhancement. The performance on Flickr30k is similar to that on RefCOCO. As the interaction between different modality prompts deepens, the richer the content of the prompts, the more significant the performance improvement.
Overall, prompt tuning methods generally achieve greater improvements accuracy compared to adapter tuning. This is because adapter tuning typically requires adjusting the learned network to the entire data distribution, which may compromise the generalization ability of the original back- bone network, making training more challenging. Consequently, the accuracy improvement is rel- atively limited. The performance boost of Diff-Prompi can be attributed to its ability to provide spput-specific rich prompt information to the pre- pre-trained model, leveraging the strong generative capabilities of the generative model based on image and caption content. In contrast, this is difficult to achieve with random initialization.
7

---

Published as a conference paper at ICLR 2025
(x) Caption: row of chairs slong wall middle one
(b) Castion: yok 9856
Caption:: bigbear
Captiore first dog
Figure 4: Qualitative Analysis RefCOCO: Ground Truth (left), GLIP-T(A) (middle), Diff- Pfff-Prompt (right). The results show the top three bounding boxes with the highest confidence, represented by green, blue, and purple from highest to lowest confidence, respectively. In the caption, the red content indicates positive tokens.
5.3 QUALITATIVE ANALYSIS
Visualization result is shown Fig. we can observe that: (1) Compared to the foundation model, Diff-Prompt is more sensitive to location; in figure (a), its attention is focused on the top right cor- ner. (2) Diff-Prompt exhibits stronger language understanding;
figure (b), the caption refers to the motorcycle's license plate, but it understands that the license plate refers to the motorcycle itself. (3) Diff-Promp has superior object recognition capabilities; as shown in figure (c), can aceu- rately identify occluded objects. (4) Diff- Piff-Prommt demonstrates stronger multimodal understanding, accurately identifying the referred object when multiple similar objects are present in the image.
5.4 ABLATION STUDY
We first conduct ablation studies to evaluate the generalization capability of our model. We fol- low prior work and conduct experiments two benchmarks: the cross-dataset benchmark and the cross-domain benchmark. CoOp, MaPLe, PromptKD (Li et al., 2024b), PromptSRC (Khattak al., 2023b), CoPrompt (Roy & Etemad, 2023), CPL (Zhang et al., 2024c), and CoCoLe (Zhang et al., 2024b) are chosen for comparison. The cross-datase benchmark evaluates the model's general- ization ability on shifted data. The cross-domain benchmark is illustrated in Appendix. D. Then, we ablate on the generalization ability across downstream tasks and backbones. Subsequently, we conduct ablation experiments the effectiveness of prompts and prompt depth.
Table 2: Comparison with state-of-the-an methods cross-dataset evaluation.
StanfordCars
Cattech101
OxfordPete
Flovers122
ImugeNet
Yood101
Alreraft
SUN297
EurSAT
UCFR8
Average
DTD
Source
CoOp
68.71
85.30
MaPLe
70.72
93.53
90.49
65.57
72.23
86:20
24.74
67.01
46.49
48.06
68.69
66.30
PromptKD)
93.61
91.59
73,93
75.33
88.84
26.24
68.57
55.08
63.74
76.39
71.33
PrompiSRC
71.27
93.60
90.25
65.70
70.25
86.15
23.90
45.50
68.75
65,81
CaPrompe
70.80
90.73
65.67
72.30
86.43
24.00
67.57
47.07
$1,90
69.73
CPL
73.53
95.52
91.64
66.17
73.35
87.68
27.36
68.24
18.96
$1.25
70.52
68.07
CoCoLe
73.88
95.88
91.93
67.79
74.17
87.97
28.83
68.75
49.26
51.75
72.78
68.91
DMI-Proopt
72.06
91.08
75.57
87.26
29.07
48.87
$2.83
73.32
68.85
Generalization. We consider the following 11 datasets to evaluate cross-domain performance: Aircraft (Maji et al., 2013), Caltech101 (Fei-Fei al., 2004), Cars (Krause et al., 2013), DTD (Cimpoi et al., 2014), EuroSAT (Helber et al., 2019), Flower102 (Nilsback & Zis- serman, 2008), Food101 (Bossard et al., 2014), Pets (Parkhi et al., 2012), SUN397 (Xiao et al., 2010), and UCF101 (Soomro, 2012). These datasets cover wide range of categories, allowing us to assess the model's ability across diverse classes. The experimental results are shown in the Tab. 2. The conclusions for cross dotssetttsset generalization and cross-domain generalization simi- lar. Although biff-Prompt does not achieve sfte-oof-the-a performance, it performs notably well certain datasets, such Flowers 102, Aircraft, and SUN397.
8

---

Published as a conference paper at ICLR 2025
Method
VQA
NIVR-
Method
mloU
Iol/RG
AP
test-dev
testostd
dev
test-P
CLIPSegIPC)
46.1
56.2
78.2
CLIPSeggPC. D=128)
48.2
78.2
BLIP
78.24
78.17
82.48
83.08
BLIPC
78.25
78.32
82.15
82.24
CIIPPee(PC)) Prompt
47.8
56.4
78.2
CLIPSeg(PC, D=128y+Dif-Promp
49.6
57.0
78.7
BLIP+Dil--Promp
78.59
78.88
82.94
83.76
BLIP p +Dii Prompi
78.92
79.24
83.09
84.06
Table Generalization Using CLIPSeg on the RES Task.
Table 4: Generalization Using BLIP on the GQA Task.
Generalization Across Downstream Tasks and Backbones. To validate the generality of Diff- Prompt, we conducted experiments using two different backbones on two new downstream tasks. Specifically, we employed the CLIPSeg (Lüddecke & Ecker, 2022) model on the PhaseCut dataset for the Referring Expression Segmentation task and the BLIP (Li et al., 2022a) model on the VQA- v2 and NLVR datasets for the VQA task. Detailed experimental settings are provided in Ap- pendix B.2, and the results are shown in Tab. and 4. The results demonstrate that Diff-Prompt is equally effective with new backbones and downstream tasks. This effectiveness stems from method's ability to guide the model to focus on relevant parts of the image based on captions, making highly applicable various visual understanding tasks.
Effectiveness of Prompts. We investigated the im- pact of different prompts by removing them one at time: the visual prompt (w/o P"), global visual prompt (w/o GP"), textual prompt (w/o Pt), and global textual prompt (w/o GP')). Results Tab. 5 show that all prompts enhance accuracy. Remov- ing task-specific prompts significantly reduces @@ 1 and @@5, highlighting their role in guiding the pre- trained model. The textual prompt, derived from vi- sual information, has the most substantial impact on accuracy when removed. This is because it integrates visual data into the language encoder, facili- tating cross-modal interaction.
Table 5: Effectiveness of Prompts.
Method
R@1
R@5
UB
Diff- Piff-Prompt
37.94
w/o P
36.91;-1.00
89.84(0071
99.15, 99.15
GP
37.01, 0.98)
89.61, 0.94)
99.16 0.213
wlo p
35.61;-258)
(-.24)
98, (8.82
wlo Gp
36.95,
89.81
09.11; (9.11[-0.20) (-220)
Prompt Depth. This section investigates the impact of prompt depth. We selected prompt depths of 1, 3, 6, 9, and 12. Specifically, when the prompt depth set to 1, prompts are added only at the encoder in- put, while for depth of 12, prompts are added to the input of each transformer layer in the encoder. As shown in the Fig. 5, both R@ and R@ R@5 steadily increase the prompt depth increases. When the prompt depth is shallow, the accuracy improvement relatively slow, and there may even be down- ward trend. However, as the depth increases, the im- provement becomes more significant. This indicates that deeper prompts are more effective, likely be- cause the deeper layers of the encoder encode richer information, facilitating easier information interac- tion. However, increasing the depth also leads to an increase in parameters and computational complexity, which is discussed in the complexity analysis.
RgK
nga
0.01
R,05
039
0373-
0330
Fromot Desth
Figure 5: Metrics at Different Prompt Depths.
5.5 IN-DEPTH ANALYSIS
Visual Prompt Visualization. We visualize the prompts of the first 9 layers, which is shown in Fig. Through progressive denoising, the visual and textual information is fully integrated. These prompts can provide information to the pre-trained model. Additionally, early stages of denoising, the prompts can already perceive the approximate location of the referent object. As the denoising process deepens, the prompts p these prompts are not absolutely precise, they can provide fine-grained information and help filter out the approximate contours.
observe that in the
become increasingly informative. Although

---

Published as a conference paper at ICLR 2025
Caption: the laptop nearest
Caption: zebra on the right
Caption: top left clock
Figure 6: Visualization of Prompt Generation.
Computation Complexity. In this section, explore the parameter introduction and compu- tational complexity of different methods. We calculate the learnable parameters introduced by different models (# Tunable), the percent- age of learnable parameters in the total model parameters (# Tunable %), computational com- plexity (Comp. Complex.) and inference time(Infer. Time). The results are shown in the Tab. 6. Regarding the introduction of param- eters, VPT and CoOp only introduce a small number of parameters, which appended the input of the attention layer. CLIp-Adapter and MaPLe require the introduction of additional network modules, leading to slightly more parameters compared VPT and CoOp. For FedTPG and Diff- Diff-Prompt, these methods involve designing net- works to generate prompts. To generate effective prompts, the network architecture is more complex than that of CLIP Adapter and MaPLe. For Diff-Prompt, Mask VAE takes up 368kB, the Prompt Generator 309MB, while the foundation model GLIP-T(A) occupies 2.43GB. The additional param- eters introduced by these prompt generators are still acceptable compared to the base model.
Table 6: Parameter and computational complexity analysis.
Methed
Tunable
TTiablle S
Coopp. Comples, (GFLOPS)
Idfer. Tinels)
CLIPAdaper
6.98
Cudp
$3.38
0.037
27.1
1.85
&-horpp
61.7K
0041
21-3
1881
Malte
0.7M
0.473
27.1
1.78
4.3
Dett-roopi
5.5M
In terms computational complexity, the complexity other methods is roughly the same, while Diff-Prompt is much higher. This is because Diff- hiff-Prompt requires multi-step generation of prompts using a diffusion model. We optimized the model size and sampling speed as much as possible, resulting in time complexity of 28.2 GFLOPs for the final model, plus 7.7 GFLOPs per sampling step. However, we found that in actual inference, the inference time of Diff' Diff-Prompt does not increase significantly, taking only 2.29 seconds. This thanks the transformer architecture of the diffusion model, which allows for high-speed parallel computation, and the diffusion model's I's size is an order of magnitude smaller than GLIP, thus causing a significant increase in time.
CONCLUSION
This paper presents new method for prompt generation based on a diffusion model. We find that, with appropriate supervision. the diffusion model can generate fine- ain-igrined prompts, achieving cross-modal information fusion and understanding. These generated prompts provide rich informa- tion that can help fine-tune pre-trained models for complex multimodal downstream tasks.
7 ACKNOWLEDGMENTS
This work was supported in part by Public Welfare Research Program of Ningbo under Grant No. 20245062 and Yongjiang Talent Project of Ningbo under Grant No. 2024A-161-G.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Hyojin Bahng, Ali Jahanian, Swami Sankaranarayanan, and Phillip Isola. Exploring visual prompts for adapting large-scale models. arXiv preprint crXiv:2203.17274, 2022.
Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. (rod-101-mining discriminative compo- nents with random forests. In Computer vision-EcCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, proceedings, part VI 13, pp. 446-461. Springer, 2014.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Hemighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners, 2020. URL https: / aavivv org/ alss 2005. 14165.
Qinglong Cao, Zhengqin Xu, Yuntian Chen, Chao Ma, and Xiaokang Yang. Domain prompt learning with quaternion networks, 2023. URL
org. abs/ 2312. 08878.
Mircea Cimpoi, Subhransu Maji, lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. De- scribing textures in the wild. In Proceedings of the IEEE conference on computer vision and paltern recognition, pp. 3606-3613, 2014.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hi- erarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. leee, 2009.
Zihan Ding, Zi-han Ding, Tianrui Hui, Junshi Huang, Xiaoming Wei, Xiaolin Wei, and Liu. Ppmn: Pixel-phrase matching network for one-stage panoptic narrative grounding. In Proceedings of the 30th ACM International Conference on Multimedia, pp. 5537-5546, 2022.
Taoran Fang, Yunchao Zhang, YANG YANG, Chunping Wang. and Lei Chen. Univer- sal prompt tuning for graph neural networks. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Pro- cessing Systems, volume 36, pp. 52464-52489. Curran Associates, Inc., 2023. URL https: pttpss:/proceediiee
neurips cc www..ps.cccppppee.eeeeeee.ee files//aper/2223/ www.ppppppeocccccceeeeeeee.
Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested 101 object categories. In 2004 conference on computer vision and pattern recognition workshop, pp. 178-178. IEEE, 2004.
Dongjie Fu, Xize Cheng, Xiaoda Yang. Wang Hanting, Zhou Zhao, and Tao Jin. Boosting speech with contrast-augmennt prompts. In Proceedings
recognition robustness to of the 32nd ACM International Conference on Multimedia, pp. 3838-3847, 2024.
Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or. An image is worth one word: Personalizing text-to-image generation using textual inversion, arXiv preprint arXiv:2208. 0/608 2022.
Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma, Rongyao Fang. Yongfeng Zhang, Hongsheng Li, and Yu Qiao. Clip-adapter: Better vision-language models with feature adapters. International Journal of Computer Vision, 132(2):581-555, 2024.
Zirun Guo, Xize Cheng. Yangyang Wu, and Tao Jin. A wander through the multimodal land- scape: Efficient transfer learning via low-rank sequence multimodal adapter. arXiv preprint arXiv:2412.08999, 2024a.
Zirun Guo, Tao Jin, and Zhou Zhao. Multimodal prompt learning with missing modalities for sentiment analysis and emotion recognition. arXiv preprint arXiv:2407.05374 2024b.
11

---

Published as a conference paper at ICLR 2025
Jameel Hassan, Hanan Gani, Noor Hussein, Muhammad Uzair Khattak, Muzammal Naseer, Fa- had Shahbaz Khan, and Salman Khan. Align your prompts: Test-time prompting with distribution alignment for zero-shot generalization. arXiv preprint arXhv:2311.01499 2023.
Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remole Sensing, 12(7):2217-2226, 2019.
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: A criti- cal analysis of -ftto--distribution generalization. Proceedings of the IEEE/CVF international conference on computer vision, pp. 8340-8349, 2021a.
Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. In Proceedings of the IEEE/CVF conference on computer vision and pattern recogni- tion, pp. 15262-152711 2021b.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neu- ral Information Processing Systems, volume 33, pp. 6840-6851. Curran Associates, Inc., cc fappe__fleesppper/2222
2020. URL ptttps://proccedingg..oo Papper
Drew A Hudson and Christopher Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 6700-6709 67000,679,400 2019.
Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannanch Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Openclip. July 2021. URL https ht////c..orr 10. 5281/ zenodo. 5143773. If you this software, please cite it as below.
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoe Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and lsioonlanguuae representation learning with noisy text supervision. In International conference on machine learning, pp. 4904-4916. PMLR, 2021.
Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. European Conference on Computer Vision, pp. 709-727. Springer, 2022.
Tao Jin, Weicai Yan, Ye Wang, Sihang Cai, Qifan Shuai, and Zhou Zhao. Calibrating prompt from history for continual vision-language retrieval and grounding. In Proceedings of the 32nd ACM International Conference Multimedia, pp. 4302-4311, 2024.
Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. ReferltGame: Referring to objects in photographs of natural scenes. In Alessandro Moschitti, Bo Pang, and Walter Daele- mans (eds.). Proceedings of the 2014 Conference Empirical Methods in Natural Language Processing (EMNLP),, pp- 787-798, Doha, Qatar, October 2014. Association for Computational Linguistics. doi: 10.3115/v1/D14-1000 1086. URL https: htt///aaaaanntoooo.oaoeo. orgg/14 1086.
Muhammad Uzair Khattak, Hanoona Rasheed, Muhammad Maaz, Salman Khan, and Fahad Shah- baz Khan. Maple: Multi-modal prompt learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19113-19122, 2023a.
Muhammad Uzair Khattak, Syed Talal Wasim, Muzammal Naseer, Salman Khan, Ming-Hsuan Yang. and Fahad Shahbaz Khan. Self-regulating prompts: Foundational model adaptation without forgetting. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 11190-15200, 2023b.
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine grained categorization. In Proceedings of the IEEE international conference on computer vision work- shops, pp. 554-561, 2013.
12

---

Published as a conference paper at ICLR 2025
Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efffcient prompt tuning. arXiv preprint arXiv:2104.08691, 2021.
Boyi Li, Kilian Q. Weinberger, Serge Belongie, Vladlen Koltun, and René Ranftl. Language-driven http://..rr///////ee///e
semantic segmentation, 2022. URL https:
03546.
Chunyuan Li Haotian Liu", Liunian Harold Li, Pengchuan Zhang, Jyoti Aneja, Jianwei Yang. Ping Jin, Yong Jae Lee, Houdong Hu, Zicheng Liu, et al. Elevater: A benchmark and toolkit for evaluating imguage-auumenter visual models. arXiv preprint arXiv:2204.0879)) 2022.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-img -mage pre- training for unified rision-languag understanding and generation. In International conference on machine learning, pp. 12888-12900 PMLR, 2022a.
Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang, Chunyuan Li, Yiwu Zhong, Li- juan Wang, Lu Yuan, Lei Zhang, Jenq-Nensn Hwang, et al. Grounded language-image pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 10965-10975, 2022b.
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101.000 0000000 2021.
Xin Li, Dongze Lian, Zhihe Lu, Jiawang Bai, Zhibo Chen, and Xinchao Wang. Graphadapter: Tuning vision-language models with dual knowledge graph. Advances in Neural Information Processing Systems, 36, 2024a.
Zheng Li, Xiang Li, Xinyi Fu, Xin Zhang, Weigiang Wang, Shuo Chen, and Jian Yang. Promptkd: Unsupervised prompt distillation for vision-language models. In Proceedings of IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 26617-26626, 2024b.
Wang Lin, Jingyuan Chen, Jiaxin Shi, Zirum Guo, Yichen Zhu, Zehan Wang, Tao Jin, Zhou Zhao, Fei Wu, YAN Shuicheng, et al. Action imitation in common action space for customized action image synthesis. In The Thirty-eighth Annual Conference Neural Information Processing Systems, 2024a.
Wang Lin, Jingyuan Chen, Jiaxin Shi, Yichen Zhu, Chen Liang, Junzhong Miao, Tao Jin, Zhou Zhao, Fei Wu, Shuicheng Yan, al. Non-confusing generation of customized concepts diffusion models. arXiv preprint arXi::2405.06914 2024b.
Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang. Jie Yang. Chunyuan Li, Jianwei Yang, Hang Su. Jun Zhu, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint vxXiv22333055499 2023.
Timo Lüddecke and Alexander Ecker. Image segmentation using text and image prompts. Pro- ceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 7086- 7096, 2022.
XIAOSONG MA, Jie ZHANG, Song Guo, and Wenchao Xu. Swapprompt: Test-time prompt adaptation for vision-language models. In A. Oh. T. Naumann, A. Glober- son, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Pro- cessing Systems, volume 36, pp. 65252-6524 Curran Associates, Inc., 2023. URL.
https: http:://ttttttt http:///pocceeen..ee.e neurips cc ccppaper_ppppeeeeeeeeeeeeeeeeeee httppp/eeeepppeeisssepppeeeeeeee fttps///ppp/e//////////////// otaaa//99222222222222222-0----e0
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained visual classification of aircraft. arXiv preprint 1306.53300.5/555 2013.
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffuusion probabilistic models. In International conference on machine learning, pp. 8162-8171. PMLR, 2021.
Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification large number of classes. In 2008 Sixth Indian conference on computer vision, graphics image processing, pp. 722-729. IEBE, 2008
13

---

Published as a conference paper at ICLR 2025
Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In 2012 IEEE conference on computer vision and pattern recognition, pp. 3498-3505 IEEE, 2012.
William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4195 4205, 2023a.
William Peebles and Saining Xie. Scalable diffusion models with transformers, 2023b. URL
https: /ttp//aavvvvv//v org/ /bss2212.09948
Fabio Petroni, Tim Rocktäschel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, Alexander H. Miller, and Sebastian Riedel. Language models as knowledge bases?, 2019. URL https://arxiv. org/ abs/ 1909. 01066.
Hieu Pham, Zihang Dai, Golnaz Ghiasi, Kenji Kawaguchi, Hanxiao Liu, Adams Wei Yu, Jiahui Yu, Yi-Ting Chen, Minh-Thang Luong, Yonghui Wu, et al. Combined scaling for zero-shot transfer learning. Neurocomputing,
2023.
Bryan A. Plummer, Liwei Wang, Chris M. Cervantes, Juan C. Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models, 2016. URL
wwtp://///aa/a//aoooa/aa /155/ 04870.
Chen Qiu, Xingyu Li, Chaithanya Kumar Mummadi, Madan Ravi Ganesh, Zhenzhen Li, Lu Peng, and Wan-Yi Yi Lin. Federated text-driven prompt generation for vision- lasounlanguuge models. In The Twelfth International Conference Learning Representations, 2024.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision, 2021. URL pettps://arxie wwtrxiiv..ggg/e abs www./////000 2103 00020.
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In International conference on machine learning, pp. 5389-54800 5400. PMLR, 2019.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 10684-10695, 2022.
Shuvendu Roy and Ali Etemad. Consistency-guidn prompt learning for vision-language models. arXiv preprint arXiv:2306.011990 2023.
Shuvendu Roy and Ali Etemad. Consistency- prompt learning for vision-language models.
In The Twelfth International Conference Learning Representations, 2024. URL https:
/ / //opnneveieww...mmm net/ / /o///oormm/// idwwsiwwwiiii 4x4w.
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman.
Dreambooth: Fine tuning text-lo-image diffusion models for subject-driven generation. In Pro- ceedings of the IEEE/CVF conference on computer vision and paltern recognition, pp. 22500- 22510, 2023.
Boya Shi, Zhengqin Xu, Shuai Jia, and Chao Ma. Prompt learning with quaternion networks. In The Twelfth International Conference on Learning Representations, 2024. URL https: / //openzeview. net http://ooommmmddkklladode
Taylor Shin, Yasaman Razeghi, Robert Logan IV, Eric Wallace, and Sameer Singh. Autoprompt: Eliciting knowledge from language models with automatically generated prompts. arXiv preprint arXiv:2010. 15980, 2020.
Manli Shu, Weili Nic. De-An Huang, Zhiding Yu, Tom Goldstein, Anima Anandkumar, and
Chaowei Xiao. Test-time prompt tuning for zero-shot generalization in vision-language models. Advances in Neural Information Processing Systems, 35:14274-14289 2022.
Jiaming Song. Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502 2020.
14

---

Published as a conference paper at ICLR 2025
Lin Song, Ruoyi Xue, Hang Wang, Hongbin Sun, Yixiao Ge, Ying Shan, et al. Meta-adapter: An online few-shot learner for vision-language model. Advances in Neural Information Processing Systems, 36:55361-55374, 2023.
K Soomro. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint ar2:0:12/2.0402,, 2012.
Eric Wallace, Shi Feng, Nikhil Kandpal, Matt Gardner, and Sameer Singh. Universal adversarial triggers for attacking and analyzing nip. arXiv preprint arXiv:/908.07/22. 2019.
Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global representa-
tions by penalizing local predictive power. Advances in Neural Information Processing Systems,
32, 2019.
Yabin Wang, Zhiwu Huang, and Xiaopeng Hong. S-prompts learning with pre-trained trans- formers: An occam's 's razor for domain incremental learning. In S. Koyejo, S. Mo- hamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp- 5682-5695. Curran Associates, Inc., 2022a. URL attps://procedings neurips ce httppaprrfiiee/eeeeeeeeeeeeeeeee wwwp:f/epeeeeeeee//e/ee Paperr----pp-----e Cerrr-Cooeeenneeeee pdf.
Yabin Wang, Zhiwu Huang, and Xiaopeng Hong. S-prompts learning with pre-trained transformers: An occam's razor for domain incremental learning. Advances in Neural Information Processing Systems, 35:5682-5695, 2022b.
Zifeng Wang, Zizhao Zhang, Chen-Yu Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vin- cent Perot, Jennifer Dy, and Tomas Pfister. Learning to prompt for continual learning. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 139-149, June 2022c.
Jianxiong Xiao, James Hays, Krista Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In 2010 IEEE computer society conference on computer vision and pattern recognition, pp. 3485-3492. IEEE, 2010.
Weicai Yan, Ye Wang. Wang Lin, Zinun Guo, Zhou Zhao, and Tao Jin. Low-rank prompt interaction for continual vision-language retrieval. In Proceedings of the ACM International Conference on Multimedia, pp. 8257-8266, 2024.
I Lingxiao Yang, Ru- Yuan Zhang, Yanchen Wang, and Xiaohua Xie. Mma: Multi-modal adapter for vision-language models. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 13826-23837, 2024a.
Xiaoda Yang, Xize Cheng, Jiaqi Duan, Hongshun Qiu, Minjie Hong, Minghui Fang, Shengpeng Ji, Jialong Zuo, Zhiqing Hong, Zhimeng Zhang, al. Audiovsr: Enhancing video speech recogni- tion with audio data. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 15352-15361, 2024b.
Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao. Houdong Hu, Xuedong Huang, Boxin Li, Chumyuan Li, et al. Florence: A new foundation model for computer vision. arXiv preprint arXiv:2/11.11
2021.
Yuhang Zang, Wei Li, Kaiyang Zhou, Chen Huang, and Chen Change Loy. Unified vision and language prompt learning, 2022. URL https: //tpxaaavvv/ooo///a/.../.aaa org/ abs /obs/22100 07225.
Runjia Zeng, Cheng Han, Qifan Wang, Chunshu Wu, Tong Geng, Lifu Huang, Ying Nian Wu, and Dongfang Liu. Visual fourier prompt tuning. arXiv preprint arXiv:241 1.01327, 2024.
H Zhang, Pengchuan* Zhang, Xiaowei Hu, Yen- Yen-Chun Chen, Liunian Harold Li, Xiyang Dai, Lijuan Wang. Lu Yuan, Jenq-Neng Hwang, and Jianfeng Gao. Glipv2: Unifying localization and vision-language understanding. arXiv preprint arXiv:2206. 05836, 2022a.
Ji Zhang, Shihan Wu, Lianli Gao, Heng Tao Shen, and Jingkuan Song. Dept: Decoupled prompt tuning. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 00924-129333 June 2024a.
15

---

Published as a conference paper at ICLR 2025
Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3836-3847, 2023.
Renrui Zhang, Wei Zhang, Rongyao Fang, Peng Gao. Kunchang Li, Jifeng Dai, Yu Qiao, and Hong- sheng Li. Tip-adapter: Training-free adaption of clip for few-shot classification. In European conference on computer vision, pp. 493-510. Springer, 2022b.
Yi Zhang, Ke Yu, Siqi Wu, and Zhihai He. Conceptual codebook learning for vision-language models. In European Conference on Computer Vision, pp. 235-251. Springer, 2024b.
Yi Zhang, Ce Zhang, Ke Yu, Yushun Tang, and Zhihai He. Concept-guided prompt learning for generalization in vision-language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp- 7377-7386, 2024c.
Kaiyang Zhou, Jingkang Yang. Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022a.
Kaiyang Zhou, Jingkang Yang. Chen Change Loy, and Ziwei Liu. Learning to prompt for vision- language models. International Journal of Computer Vision, 130(9):2337-2344 2022b.
16

---

Published as a conference paper at ICLR 2025
A MODEL DESIGN
A.I MASK-VAE DESIGN
We use the AutoencoderKL class from the Python diffusers library to train our Mask- VAE, setting the in channel parameter to 1. The Mask-VAE reads masks of size 224 x 224 and compresses them into x 28 x 28. The final trained Mask- VAE occupies only 368kB in safetensors format.
A.2 PROMPT GENERATOR ARCHITECTURE
The architecture of the prompt generator is generally the same as DiT, with only two differences. (1) the condition needs to incorporate both image and text information. We begin by using the same encoders as that in GLIP-T(A) to encode image and text, resulting in visual embedding, and textual embedding. The visual embedding and textual embedding then concatenated to form the condition, which is fed into the DiT block. (2) For parameter selection, aim to minimize the number of parameters while maintaining model performance. The number of DiT blocks is set to 12, the hidden size is set to 512, the patch size is 2, and the number of attention heads is 8. The final trained prompt generator occupies only 309MB when using float32 precision.
each added to timestep embedding.
A.3 ADAPTER DESIGN
C,AW by 2.1
[hr 2rr
[Bs,,,,,1
[bs.,,PP]
Convid, SILU
Convad, SRLU
Conväd, SiLu
Flatten
Linear Layer
Reshape
Figure Adapter Architecture.
As shown in Fig. the vision adapter and language adapter share the network architecture. The latent features are first decoded through the mask- VAF decoder and then is mapped into input- specific prompts through the corresponding modality encoders. To avoid the significant resource waste that could result from direct mapping, we first reduce the dimension of the latent prompts using a few convolutional layers, and then perform the mapping the dineeinnnio space.
B EXPERIMENT DETAIL
B. EXPERIMENTAL DETAILS OF COMPARATIVE EXPERIMENTS
VPT and VFPT introduces prompts into the visual encoder, while CoOp introduces prompts into the language encoder. S- S-Prompts, MaPLe, and FedTPG introduce prompts into both modalities simul- taneously. It should be noted that S-Prompts are used in continual learning scenarios, while FedTPG is used in multiple remote clients scenarios. In this work, we only use their network architectures. The prompt length is set to for all methods, while Diff- Diff-Prompt sets 4 input-specific prompts and global prompts for each modality. Except for the FedTPG method, other prompt learning methods add prompts to the first attention layers. Since FedTPG originally designed to only add prompts at the encoder input, introduce FedTPG.gs. which adds prompts to the first 9 layers of the encoder.
B.2 EXPERIMENTAL DETAILS GENERALIZATION ABLATION EXPERIMENTS
We explore the generalization ability of Diff-Prompt across different backbones and tasks. Specif- ically, we select CLIPSeg for the Referring Expression Segmentation task and the BLIP model for the Visual Question Answering task. For the CLIPSeg model, its encoder is CLIP, with the original CLIP weights kept unchanged. Therefore, we adopt the same settings as the comparative experi- ments, introducing prompts both the visual and textual encoder ends. For each modality, use
17

---

Published as a conference paper at ICLR 2025
Table 7: Results of different prompt strategies on the RefCOCO dataset.
testA
testß
val
Strategy @@
R@5
UB
R @
R@5
UB
R@l
R@5
UB
sequential
35.09
86.18
92.98
29.28
75.70
87.16
32.27
80.61
90.36
reverse
39.08
94.71
99.63
36.09
85.67
99.00
37.94
90.55
99.37
input-specific prompts and 4 global prompts. During training, only the adapter is trained, ensuring that the remaining network parameters remain unchanged.
For the BLIP model, which follows an encoder-decoder architecture, we introduce prompts only in self-attention module of the encoder. During the training phase, we train only one adapter while keeping all other parameters unchanged.
C PROMPT SELECTION
Vision Adapter
Adapter
Franue
Promer
Frompe
(a) Sequential Prompting.
(b) Reverse Prompting.
Figure 8: Different strategies for introducing prompts.
The prompt generator uses the image and caption as conditions, and the final result is obtained by denoising the random Gaussian noise 25 times. As the denoising process progresses, the image and text information gradually merge; that is, the denoising process can be seen an interaction process between the image and text information. Therefore, as increases, the interaction between image and text increases. We choose T sampple minimizing the number of sampling steps and aligning with the pre-trained model. Here, we can align the sampling process with the encoding process either sequentially in reverse order. In the sequential process Fig. 8(a), a small amount of interaction information is provided in the early stages of encoding, while in the reverse process Fig. 8(b), richer interaction information is provided in the shallow layers of the encoder. We conducted ablation experiments, and the results are shown in the Tab. From the figure, we can see that introducing prompts in reverse order results in better performance. This improvement is likely due to incorporating more interactive information in the shallow layers of the encoder, which may better assist the encoding process.
25 to ensure the quality of the final result while
D ZERO-SHOT EVALUATION
This section explores the zero-shot capabilities of Diff-Prompt. We compare Diff-Promp with GLIP-T(A) and GLIP-I.. Compared to GLIP-T(A), GLIP-I has a larger model size, more training data, and stronger generalization abilities. We selected 11 representative datasets from ODinW (Li et al., 2022b) for testing: AmericanSSinLLaagugeLLtttt. (Letters), BCCD, brackishUnderwater (Underwater), CottontailRabbits (Rabbits). NorthAmeriiaaMMssrronn. (Mushrooms), Packages, pis- tols, Raccoon, ShellfishOpenlnages (Shellfish), thermalDogssAndPeople (DogsPeople), and Vehi- clesOpenlmages (Vehicles). We Average Precision (AP) @ [I0U=0.5:0.95] the metric.
Zero-Shot Evaluation for Object Detection. As shown in Tab. and 9, the zero-shot capabilities of Diff-Prompt and foundation models compared. From the figures,
that Diff- Diff-Prommt
18

---

Published as a conference paper at ICLR 2025
Table 8: Zero-shot Evaluation of Diff- Diff-Prompt and Foundation models. (Part 1)
Letters
BCCD
Underwater
Rabbits
Mushrooms
Puckage
GLIP-T(A) 0.07
29.72
32.43
GLIP-L
1.56
2.78
0.98
78.25
57.41
51.13
Diff- Pfff-Prompt
2.25
8.42
1.21
71.27
54.18
41.74
Table 9: Zero-shot Evaluation of Diff- Dift-Prompt and Foundation models. (Part
pistols
Raccoon
Shellfish
DogsPeople
Vehicles
Average
GLIP-T(A)
32.28
15.43
15.34
40.82
45.35
25.73
GLIP-L
71.5
47.96
46.93
64.82
55.75
43.55
Diff- Prompt
12.82
45.63
11.3
34.27
20.22
27.57
retains the generalization ability of GLIP-T(A), and even outperforms GLIP-T(A) on portion of the datasets. This is because Diff-Promph provides additional auxiliary information for the original GLIP-T(A) without making any changes to the model itself. However, compared to GLIP-L, which has larger parameter size and more training data, Diff-Prompt and GLIP-L still have significant gap. This indicates that prompt learning's ' improvement to performance is still limited. Notably, for the Letters and Underwater datasets, both GLIP-T(A) and GLIP-L perform particularly poorly. In contrast, Diff-Prompt shows subtle improvement, suggesting that when the pretrained model fails extract feature information effectively, prompt information can play a significant role in enhancing performance.
Table 10: Comparison with -fatt-of-thhe-aee methods on cross cross-domain evaluation.
Source
ImageNet
-V2
-8
-A
-R
Avg.
CLIP
66.73
60.83
46.15
47.77
73.96
18
CoOp
71.51
64.20
47.99
49.71
75.21
59.28
MaPLe
70.72
64.07
49.15
50.90
76.98
60.27
PromptKD
PromptSRC
71.27
64.35
49.55
50.90
77.80
60.65
CoPrompt
70.80
64.25
49.43
50.50
77.51
60.42
CPL.
73.53
65.18
49.92
50.73
77.38
61180
CoCoLe
73.88
65,86
50.89
51.75
78,89
61.85
Drof-Prompt
72.06
51,06
50.97
Cross-Domain Generalization. Ablation study for cross-domain generalization assesses the model's 's capability across different categories. We select ImageNet (Deng et al., 2009) and its four variations: ImageNet-A (Hendrycks et al., 2021b), ImageNet-V V2 (Recht et al., 2019), ImageNet- R (Hendrycks et al., 2021a) and ImageNet-S (Wang et al., 2019), as the evaluation datasets. Specif- ically, we can consider that the CLIP model well-fitted to the ImageNet dataset, while the data from the other four datasets are treated as -ut-of-distribution. As shown in the Tab. 10, Diff-Prompt achieves competitive accuracy with CPL and outperforms most prompt-learning methods, though still lags behind CoCoLe. Notably, it achieves the best performance ImageNet-S, highlighting its robustness against overfitting due to its controlled, generated prompts, unlike directly learned prompts, which are more prone overfitting.
19

---

Published as a conference paper at ICLR 2025
E
QUALITATIVE ANALYSIS
Ground Truth
GLIP-T(A)
S- Prompts
MaPLe
FedTPG
Diff-Pre hiff-Prompt
Caption: black and white striped animal
Caption: bigbear
Caption: first hot dog toward front in left column
Caption: in row of chairs along wall the middle one
Lite
DJ03
200
Caption: red shirt
Caption: cow cut off on far left
Caption: yrk 9856
Caption: apples middle
Caption: green
Figure 9: Qualitative Analysis on RefCOCo.
This section provides additional visualization results. We visualize Ground Truth, GLIP-T(A), S-
Prompts, MaPLe, FedTPG, and Diff Diff-prompt. From the Fig. 9, it can be seen that Diff- Profppr outperforms the other methods most of the situation.
20

---

Published as a conference paper at ICLR 2025
F IN-DEPTH ANALYSIS OF THE FLICKR30K
Method
Animals
Bodyyarts
Clothing
Instruments
R:0:0
UB
R.61
R:910
R:055
R1
R0061
13.55
UB
CLIP-Adapter
50.32
65.81
78.
82.
VPT
91.20
11.46
32:77
46.38
74.48
57.42
CaOp
78.39
95.79
7.76
18.48
27.17
$7.04
67.88
79.79
87.
S-Prosages
77.63
94.65
95.98
96.37
8.50
20.89
29.21
45.10
50.54
53.96
66.90)
79.74
43.23
72. 26
85.81
89: 08
MaPLe
75.53
94.35 26
96.94
98.47
10.72
28.47
38.45
55.27
37.56
67.884
77.90
85.48
38 38.06
74.
$4.52
92.90
FedTFG
28.97
94.05
95.90 58
96.56
7,38
19.04
31.24
49.35
35.07
59.25
08.74
79.53
43
76,
84.52
88.39
DiT Prompt
81,45
92.35
96 18
97.71
14,42
36,97
49,35
61.92
42.91
72.38
80.13
85.74
50.97
81.94
83.23
86.45
Table 11: Recall Across Different Categories on the Flickr30k val Dataset. (Part 1)
Methud
Otter
People
Velicles
R@1
R:B.5
R@:10
UB
R:61
R@:
R:0-10
RG
R:95
R:6:0
UB
R@1
R
R@10
UB
Adapter
32.45
57.76
65.68
73.72
68.66
92.77
94.87
96.16
29.16
65.17
77.63
$4.61
66.27
$4.62
39.64
91.42
VPT
30.09
57.83
65.83
73.84
38
89.52
93.58
95.56
35.
68.04
76.97
53.37
51.48
81.
89. $9.35
92,
CoUp
38,00
64.86
71.99
80.54
72.80
94.87
96.46
97.73
25.
36.95
69.67
39,24
67.75
$9.64
94,38
97.04
S.Proupes
37.45
64. 16
72.36
81.39
72.65
94.87
96.87
98.31
44.20
75.41
83.82
92.24
61.83
56, 39
90:53
93.
MaPLe
41.44
67
75.18
83.07
72.84
94.53
96.55 56
97.76
51.47
81.08
88.19
94.65
61.54
86,
92. 00.90
96.75
FoITPG
71.41
50.29
92.
95.27
Dift- Prompi
75.67
73,90
$8.
93.79
96.15
Table 12: Recall Across Different Categories on the Flickr30k val Dataset. (Part 2)
Method
Animals
Bodyyprts
Clothing
Instruments
R@1
R:B5
R@10
UB
R@
R@85
R 10
UB
R
R:85
R:810
UB
R@
R.85
R& L0
UB
CT.P.Aclpter
73.94
91.31
92.08
94.02
4.78
14.72
20.27
34,80
12.22
55.68
64.61
76.95
50.62
80.86
85.80
88.27
VPT
63.32
86.10
90.35
93.44
5.54
19.12
28.68
42.26
26.84
52.52
62.53
76.11
33.95
57.41
67.26
70.37
CoOp
93.05
93.32
6.12
18.36
22:75
60.75
72.33
82.87
73.46
80.25
89.51
S-Prompes
25.48
1.27
57.24
$2.09
75.31
$5.80
MaPL
76.83
89.77
94.31
8.99
41.87
71.12
74.07
77.16
FoITTG
76.83
93.24
94.40
93.56
7.07
21.80
27.53
46.27
37.21
61.88
72.59
$3.09
56.79
75.93
$1.48
89.51
Deff-Prompt
80.50
90.35
92.28
94.02
17.21
36.14
42.83
55.64
47.35
74.28
82.18
87.60
54.32
77.16
81. 18
92.59
Table 13: Recall Across Different Categories on the Flickr30k test Dataset. (Part 1)
Method
Other
People
Scene
Vehicles
R:0:10
UR
R
R:6:10
R
RPP55
R:P:10
R:00
R
R
1:B
CLIP-Adueh Aclppee
60:14
76.38
73.56
95.
VPT
64.07
63.25
13.75
50.11
$7.50
CoOp
39.45
61.34
72.97
81.51
73,80
95.42
97.26
98.36
22:24
$2.99
67.76
35.61
81:00
92.50
94.25
96.75
S-Proeages
37.82
63 (33.99
72.32
$1.49
73.16
95.40
97.56
98.33
45.95
75.34
83.01
90.85
72.00
91.50
95. 95.00
99.00
MaPl
40.46
66.51
75.79
83.31
74.54
95.51
97.53
98.66
49
79.12
86.10
92.00
75. 73.50
91.3 05.25
93.25
96. 00
FoITPC
39.77
63.40
72.36
$1.59
74. 35
95.58
97.14
98.21
19.15
46.57
61.21
80.91
81.75
92.50
94.55 50
96.75
PETT-Prompt
44.61
68.02
74.39
82.34
75.21
96.06
97.68
98.44
55.65
81.04
87,03
91.66
79.75
94,25
95.75
97.25
Table 14: Recall Across Different Categories on the Flickr30k test Dataset. (Part 2)
In this part, we conduct a further analysis of the results of diffeernt methods on the Flickr30K test and validation datasets. Tab. 11, 12, 13 and 14 presents the R@) 1, R@ 5, @ 10 and UB scores for different categories, which include Animals, Bodyparts, Clothing, Instruments, Other, People, Scene, and Vehicles. From the figure, we can conclude that Diff-Prompp performs well across all cat- egories, indicating more stable training. It steadily improves performance across different categories without significantly increasing performance in some at the expense of others.
G CATEGORY-WISE ACCURACY
Category- wise Accuracy. In this section, we conduct a further analysis of the results from the quan- titative analysis. Specifically, we divide the RefCOCO test dataset into 12 categories based on the "super category" field the COCO annotations: person, vehicle, outdoor, animal, accessory, sports, kitchen, food, furniture, electronic, appliance, and indoor. We compared the R@ and R @@5 metrics of GLIP-T(A). S- S-Pomppss, MaPLe, FedTPG, and Diff-Prompt across these categories. Metrics for Different Categories for Flickr30k is provided in Appendix F.
21

---

Published as a conference paper at ICLR 2025
0.5
0.4
0.3
Model
GLIP-T(A)
0.2
S-prompts
0.1
MaPLe
FedTPG
0.0
Diff- Pio-Prromp
Person
vehicle
outdsor
animal
accestaory
sports
kitchen
food
furniture
electronic
appliance
indoco
Figure 10: Category-wins wise R@ for RefCOCO Val Dataset.
1.0
0.8
Model
GLIP-T(A)
0.6
prompts
MaPLe
FedTPG
0.4
Diff-Prompt
perazn
veticie
outtioor
animal
accasaory
sporta
kitchen
food
fumitura
eleciranic
appliance
indioar
Figure 11: Category-wise R@5 for RefCOCO Val Dataset.
The results are shown in Fig. 10 and 11. From the figures, we can see that, compared to the founda- tion model, Diff-Prompt shows a more balanced improvement across all categories. This is because the prompt generator, during the second phase of training, can provide general range for the ob- jects, and the generated prompts do not significantly affect the backbone network. The process of first training the prompt generator and then aligning it helps to effectively prevent overfitting during training. FedTPG generates prompts using attention layers, but R@ performance in the outdoor and electronic categories is worse than that of GLIP-T(A). This is because the training results are biased towards certain data, making it unable to achieve improvements across all categories. For the S-Prompts and MaPLe methods, there is notable performance increase in the person cate- gory, while in other categories, their performance shows slight increase or decrease compared to GLIP--((), indicating that they mainly fit the data in the person category.
H LIMITATION
In this section, we discuss the limitations of our proposed method. First, due to the constraints of the DiT model, our model can only process images with an input size of 224x224, which limits the diversity of the image inputs. A solution is to perform some downsampling and interpolation oper- ations on the image. Second, Diff-Prompt uses a diffusion model to generate prompts in multiple steps, requiring the pretrained VAE and DiT to have strong generalization capabilities; otherwise, performance specific data may be particularly poor. Additionally, the multi-step generation pro- cess of the diffusion model consumes a large amount of time and computational resources. Overall, while Diff-Prompt generates rich, fine-grained prompt information through the diffusion model, is also influenced by the model itself, leading to high computational complexity.
For the prompt generator, only concatenate visual and textual information. For future work, believe more in-depth research on the diffusion model could explore controlled mask generation to produce more fine-grained prompts. As for the design of the prompt generator, exploring other one-step, lightweight generation models to produce prompts could help address the limitations of this paper. Meanwhile, we will explore the capabilities of Diff-Prompt on more downstream tasks, such as PNG (Ding et al., 2022) and GQA (Hudson & Manning, 2019).
22


Reviews:
[{"review_id": "HcyZ4mVVDP", "reviewer": "Reviewer_PtHg", "summary": "To address the limitation of prompt learning in applying to complex tasks requiring high granularity, the authors propose a three-stage training prompt generator named Diff-Prompt. This method aims to generate rich prompts and provide sufficient information for pretrained models in fine-grained downstream tasks. Through experiments on two vision-language understanding datasets and comparisons with other methods across visual aspects and various metrics, they demonstrate the superiority of their approach.", "questions": "The authors mentioned that Diff-Prompt may suffer from poor generalization. Based on this, have they proposed any improvement plans to mitigate this issue and enable reproduction by other researchers?\nHow does the strategy of Diff-Prompt differ from independently training a conditional mask generator and then performing information interaction with a multimodal encoder?\nDuring prompt generation, did the authors consider how to maintain a balance between visual and textual information to avoid information imbalance caused by modality bias?", "limitations": "", "rating": 3, "confidence": 5, "soundness": 2, "presentation": 1, "contribution": 1, "strengths": "The paper's comparative experiments include multiple models and provide statistical results, offering guidance for other researchers interested in prompt learning.\nThe proposed Diff-Prompt surpasses other existing efficient fine-tuning methods on multiple evaluation metrics, which may stimulate further thinking among researchers.\nThe authors train the prompt generator through mask supervision, controlling the generation of fine edge information via mask variations, which is a novel idea.", "weaknesses": "Diff-Prompt is compared with two different types of efficient parameter tuning methods, but the number of baselines for the two methods is seriously imbalanced. It would be preferable to include more comparative experiments with adapter methods.\nCompared with previous work, this paper seems focused on training masks through diffusion models to generate masks with fine edges for boundary control, but the contribution beyond this aspect is insufficient.\nThe authors claim to generate rich prompts, but the masks generated during the diffusion process do not exhibit finely controllable generation. This approach may not necessarily adapt to highly fine-grained tasks, which is evidently inconsistent with the problem the authors intend to solve.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lm3oNPpXAg", "reviewer": "Reviewer_cYSP", "summary": "The paper introduces Diff-Prompt, a diffusion-based prompt generation method aimed at enhancing fine-tuning for complex multimodal tasks. Unlike conventional prompt tuning methods, which struggle with fine-grained tasks due to limited prompt richness, Diff-Prompt uses a multi-stage process with a Mask-VAE and an improved Diffusion Transformer to generate detailed, task-specific prompts. This approach outperforms other fine-tuning methods, achieving notable improvements in recall and accuracy on tasks requiring nuanced multimodal understanding.", "questions": "1. Generalizability needs further verification: Why was testing only done on GLIP? How would it perform with other models? Why was the evaluation limited to the referring expression comprehension task? What about tasks like RES and PNG [1] for segmentation? Would it also be effective for other more complex tasks? If it only works with this model and task, then the contribution is quite limited, and the paper's title should be adjusted to specify that it is focused on the REC task.\n\n[1] PPMN: Pixel-Phrase Matching Network for One-Stage Panoptic Narrative Grounding.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "-The motivation of this paper is very clear and interesting, making it an engaging topic worth exploring in depth.\n\n-The illustrations in the paper are very clear, and the validation on the visual grounding task is quite comprehensive.\n\n-There is a significant improvement in some metrics for the referring expression comprehension task.\n\n-The code has been released, which I believe will be very helpful for further study.", "weaknesses": "1. Generalizability needs further verification: Why was testing only done on GLIP? How would it perform with other models? Why was the evaluation limited to the referring expression comprehension task? What about tasks like RES and PNG for segmentation? Would it also be effective for other more complex tasks? If it only works with this model and task, then the contribution is quite limited, and the paper's title should be adjusted to specify that it is focused on the REC task.\n\n2. The introduction of diffusion slows inference speed by twofold, which is a significant cost. Could this be optimized?\n\n3. In Tables 1 and 2, the improvement in the UB metric compared to MaPLe is minimal, despite MaPLe being a much simpler method. Why?\n\n4. The effectiveness of using diffusion in this paper warrants deeper analysis. Since the core contribution of the method itself isn’t very substantial, the primary value lies in using diffusion to generate prompts. Exploring the underlying mechanism further could provide more profound insights.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vr5EwJBddc", "reviewer": "Reviewer_3hUM", "summary": "The paper introduces Diff-Prompt, a diffusion-driven prompt generator designed to enhance fine-tuning of pre-trained multimodal models. Traditional prompt learning methods often fail on complex tasks due to limited prompt richness and specificity. Diff-Prompt addresses this by using a three-stage process: (1) compressing mask information into latent space with a Mask-VAE, (2) employing an enhanced Diffusion Transformer (DiT) to generate prompts in the latent space with mask supervision, and (3) aligning these generated prompts with a pre-trained model's semantic space for fine-tuning. Experimental results show that Diff-Prompt significantly outperforms other methods on pixel-level tasks, achieving notable improvements in recall metrics (R@1 and R@5).", "questions": "1. The vision and language adaptors adopt the same architecture. How to reasonably exploit the advantages of vision and language cues to ensure consistent fine-tuning of the pre-trained model?\n2. What potential strategies could be employed to address the high computational complexity and image input size limitations of the Diff-Prompt model?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The use of a Mask-VAE and diffusion-driven prompt generator enables the creation of rich prompts, which improves the model's ability to effectively capture information in the latent space.\n2. The alignment of generated prompts with the pre-trained model in the semantic space ensures better guidance for the pre-trained model, enhancing performance during fine-tuning.\n3. The experimental results on a complex fine-grained multimodal downstream task validate the effectiveness of the proposed method, showcasing its applicability to challenging scenarios.", "weaknesses": "Compared to other methods, Diff-Prompt has slightly more learnable parameters (5.5M). More parameters may require more memory and lead to longer inference time.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TKxWfH12Lg", "reviewer": "Reviewer_LEnk", "summary": "This paper introduces Diff-Prompt, a novel prompt learning approach that leverages diffusion models to generate rich prompts for multimodal tasks. Diff-Prompt employs a three-stage process: first training a Mask-VAE to compress masks into latent space, then using an improved Diffusion Transformer to generate prompts in this latent space with mask supervision, and finally aligning the denoising process with the pre-trained model in semantic space. The method addresses limitations of existing prompt learning approaches by generating more informative and task-specific prompts.", "questions": "Please refer to the weaknesses. In general I am concerned with evaluation settings which are different compared to other major works in prompt learning, as well as missing comparisons to recent works.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "I find the approach to be innovative and a departure from other methods in prompt learning for improved generalization. The paper is well presented, and easy to follow. The method achieves strong performance on two vision-language understanding datasets, RefCOCO and Flickr30k compared to existing methods.", "weaknesses": "I have two major concerns regarding this work.\n\n1. The authors do not report on other generalization benchmarks such as the ones reported by prompt learning techniques like CoOP and Maple, which the authors compare with. These are representative tasks in generalization to novel classes, new target datasets and unseen domain shifts. Why were these benchmarks not considered, since methods reporting on these benchmarks are considered in the evaluation?\n\n2. The authors are missing comparisons with multiple recent works related to prompt learning for generalization such as [1,2,3,4,5,6]. Why were these methods not considered during evaluation and comparison?\n\n[1] Z. Li et al., \"PromptKD: Unsupervised Prompt Distillation for Vision-Language Models\", in CVPR, 2024.\n\n[2] M. U. Khattak et al., \"Self-regulating Prompts: Foundational Model Adaptation without Forgetting,\" ICCV, 2023.\n\n[3] J. Hasan et al., \"Align Your Prompts: Test-Time Prompting with Distribution Alignment for Zero-Shot Generalization,\" NeurIPS, 2023.\n\n[4] S. Roy and A. Etemad, ‘Consistency-guided Prompt Learning for Vision-Language Models’, in ICLR, 2024.\n\n[5] Y. Zhang, C. Zhang, K. Yu, Y. Tang, and Z. He, ‘Concept-Guided Prompt Learning for Generalization in Vision-Language Models’, in AAAI, 2024.\n\n[6] Y. Zhang, K. Yu, S. Wu, and Z. He, ‘Conceptual Codebook Learning for Vision-Language Models’, in ECCV, 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TV0RfTBLLA", "reviewer": "Reviewer_A4tj", "summary": "This paper introduces the Diffusion-driven Prompt Generator (Diff-Prompt), which leverages the diffusion model to generate detailed and comprehensive prompt information for intricate downstream tasks.", "questions": "see above-mentioned issues", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "This paper is well organized\nThe performances are competitive", "weaknesses": "(1)\tThe definition in DEEP PROMPTING is confused. What is the definition of E_{i}\n(2)\tWhat mechanism do you use to the D latent features of generated prompt? It is not clear\n(3)\tIn subsection 3.1, this study is linked to referring expression comprehension. Nonetheless, there is a lack of information on how the proposed method generates a position to accurately locate the target object described in the expression. \n(4)\tThe advantage of using the generated prompt is not clear. Why can we replace the generated prompts with learnable prompts", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)