Diff-Prompt: Diffusion-Driven Prompt Generator with Mask Supervision

Authors: Weicai Yan, Wang Lin, Zirun Guo, Ye Wang, Fangming Feng, Xiaoda Yang, Zehan Wang, Tao Jin
Primary area: transfer learning, meta learning, and lifelong learning
Keywords: prompt learning, diffusion model, multimodal learning

Abstract:
Prompt learning has demonstrated promising results in fine-tuning pre-trained multimodal models. However, the performance improvement is limited when applied to more complex and fine-grained tasks. The reason is that most existing methods directly optimize the parameters involved in the prompt generation process through loss backpropagation, which constrains the richness and specificity of the prompt representations. In this paper, we propose Diffusion-Driven Prompt Generator (Diff-Prompt), aiming to use the diffusion model to generate rich and fine-grained prompt information for complex downstream tasks. Specifically, our approach consists of three stages. In the first stage, we train a Mask-VAE to compress the masks into latent space. In the second stage, we leverage an improved Diffusion Transformer (DiT) to train a prompt generator in the latent space, using the masks for supervision. In the third stage, we align the denoising process of the prompt generator with the pre-trained model in the semantic space, and use the generated prompts to fine-tune the model. We conduct experiments on a complex pixel-level downstream task, referring expression comprehension, and compare our method with various parameter-efficient fine-tuning approaches. Diff-Prompt achieves a maximum improvement of 8.87 in R@1 and 14.05 in R@5 compared to the foundation model and also outperforms other state-of-the-art methods across multiple metrics. The experimental results validate the effectiveness of our approach and highlight the potential of using generative models for prompt generation. Code is available at https://github.com/Kelvin-ywc/diff-prompt.

Decision: accept (poster)
Reviewer scores: 3, 6, 6, 8, 6

Reviews and discussion:
<thread>
creation date:
1730521477524

thread_type:
official_review

<thread_object>
creator: reviewer_a4tj

<content>
summary:
This paper introduces the Diffusion-driven Prompt Generator (Diff-Prompt), which leverages the diffusion model to generate detailed and comprehensive prompt information for intricate downstream tasks.

strengths:
This paper is well organized
The performances are competitive

weaknesses:
(1)	The definition in DEEP PROMPTING is confused. What is the definition of E_{i}
(2)	What mechanism do you use to the D latent features of generated prompt? It is not clear
(3)	In subsection 3.1, this study is linked to referring expression comprehension. Nonetheless, there is a lack of information on how the proposed method generates a position to accurately locate the target object described in the expression. 
(4)	The advantage of using the generated prompt is not clear. Why can we replace the generated prompts with learnable prompts

questions:
see above-mentioned issues
</content>
</thread_object>

</thread>
<thread>
creation date:
1730554920667

thread_type:
official_review

<thread_object>
creator: reviewer_lenk

<content>
summary:
This paper introduces Diff-Prompt, a novel prompt learning approach that leverages diffusion models to generate rich prompts for multimodal tasks. Diff-Prompt employs a three-stage process: first training a Mask-VAE to compress masks into latent space, then using an improved Diffusion Transformer to generate prompts in this latent space with mask supervision, and finally aligning the denoising process with the pre-trained model in semantic space. The method addresses limitations of existing prompt learning approaches by generating more informative and task-specific prompts.

strengths:
I find the approach to be innovative and a departure from other methods in prompt learning for improved generalization. The paper is well presented, and easy to follow. The method achieves strong performance on two vision-language understanding datasets, RefCOCO and Flickr30k compared to existing methods.

weaknesses:
I have two major concerns regarding this work.

1. The authors do not report on other generalization benchmarks such as the ones reported by prompt learning techniques like CoOP and Maple, which the authors compare with. These are representative tasks in generalization to novel classes, new target datasets and unseen domain shifts. Why were these benchmarks not considered, since methods reporting on these benchmarks are considered in the evaluation?

2. The authors are missing comparisons with multiple recent works related to prompt learning for generalization such as [1,2,3,4,5,6]. Why were these methods not considered during evaluation and comparison?

[1] Z. Li et al., "PromptKD: Unsupervised Prompt Distillation for Vision-Language Models", in CVPR, 2024.

[2] M. U. Khattak et al., "Self-regulating Prompts: Foundational Model Adaptation without Forgetting," ICCV, 2023.

[3] J. Hasan et al., "Align Your Prompts: Test-Time Prompting with Distribution Alignment for Zero-Shot Generalization," NeurIPS, 2023.

[4] S. Roy and A. Etemad, ‘Consistency-guided Prompt Learning for Vision-Language Models’, in ICLR, 2024.

[5] Y. Zhang, C. Zhang, K. Yu, Y. Tang, and Z. He, ‘Concept-Guided Prompt Learning for Generalization in Vision-Language Models’, in AAAI, 2024.

[6] Y. Zhang, K. Yu, S. Wu, and Z. He, ‘Conceptual Codebook Learning for Vision-Language Models’, in ECCV, 2024.

questions:
Please refer to the weaknesses. In general I am concerned with evaluation settings which are different compared to other major works in prompt learning, as well as missing comparisons to recent works.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730601766025

thread_type:
official_review

<thread_object>
creator: reviewer_3hum

<content>
summary:
The paper introduces Diff-Prompt, a diffusion-driven prompt generator designed to enhance fine-tuning of pre-trained multimodal models. Traditional prompt learning methods often fail on complex tasks due to limited prompt richness and specificity. Diff-Prompt addresses this by using a three-stage process: (1) compressing mask information into latent space with a Mask-VAE, (2) employing an enhanced Diffusion Transformer (DiT) to generate prompts in the latent space with mask supervision, and (3) aligning these generated prompts with a pre-trained model's semantic space for fine-tuning. Experimental results show that Diff-Prompt significantly outperforms other methods on pixel-level tasks, achieving notable improvements in recall metrics (R@1 and R@5).

strengths:
1. The use of a Mask-VAE and diffusion-driven prompt generator enables the creation of rich prompts, which improves the model's ability to effectively capture information in the latent space.
2. The alignment of generated prompts with the pre-trained model in the semantic space ensures better guidance for the pre-trained model, enhancing performance during fine-tuning.
3. The experimental results on a complex fine-grained multimodal downstream task validate the effectiveness of the proposed method, showcasing its applicability to challenging scenarios.

weaknesses:
Compared to other methods, Diff-Prompt has slightly more learnable parameters (5.5M). More parameters may require more memory and lead to longer inference time.

questions:
1. The vision and language adaptors adopt the same architecture. How to reasonably exploit the advantages of vision and language cues to ensure consistent fine-tuning of the pre-trained model?
2. What potential strategies could be employed to address the high computational complexity and image input size limitations of the Diff-Prompt model?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730624919493

thread_type:
official_review

<thread_object>
creator: reviewer_cysp

<content>
summary:
The paper introduces Diff-Prompt, a diffusion-based prompt generation method aimed at enhancing fine-tuning for complex multimodal tasks. Unlike conventional prompt tuning methods, which struggle with fine-grained tasks due to limited prompt richness, Diff-Prompt uses a multi-stage process with a Mask-VAE and an improved Diffusion Transformer to generate detailed, task-specific prompts. This approach outperforms other fine-tuning methods, achieving notable improvements in recall and accuracy on tasks requiring nuanced multimodal understanding.

strengths:
-The motivation of this paper is very clear and interesting, making it an engaging topic worth exploring in depth.

-The illustrations in the paper are very clear, and the validation on the visual grounding task is quite comprehensive.

-There is a significant improvement in some metrics for the referring expression comprehension task.

-The code has been released, which I believe will be very helpful for further study.

weaknesses:
1. Generalizability needs further verification: Why was testing only done on GLIP? How would it perform with other models? Why was the evaluation limited to the referring expression comprehension task? What about tasks like RES and PNG for segmentation? Would it also be effective for other more complex tasks? If it only works with this model and task, then the contribution is quite limited, and the paper's title should be adjusted to specify that it is focused on the REC task.

2. The introduction of diffusion slows inference speed by twofold, which is a significant cost. Could this be optimized?

3. In Tables 1 and 2, the improvement in the UB metric compared to MaPLe is minimal, despite MaPLe being a much simpler method. Why?

4. The effectiveness of using diffusion in this paper warrants deeper analysis. Since the core contribution of the method itself isn’t very substantial, the primary value lies in using diffusion to generate prompts. Exploring the underlying mechanism further could provide more profound insights.

questions:
1. Generalizability needs further verification: Why was testing only done on GLIP? How would it perform with other models? Why was the evaluation limited to the referring expression comprehension task? What about tasks like RES and PNG [1] for segmentation? Would it also be effective for other more complex tasks? If it only works with this model and task, then the contribution is quite limited, and the paper's title should be adjusted to specify that it is focused on the REC task.

[1] PPMN: Pixel-Phrase Matching Network for One-Stage Panoptic Narrative Grounding.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731053894271

thread_type:
official_review

<thread_object>
creator: reviewer_pthg

<content>
summary:
To address the limitation of prompt learning in applying to complex tasks requiring high granularity, the authors propose a three-stage training prompt generator named Diff-Prompt. This method aims to generate rich prompts and provide sufficient information for pretrained models in fine-grained downstream tasks. Through experiments on two vision-language understanding datasets and comparisons with other methods across visual aspects and various metrics, they demonstrate the superiority of their approach.

strengths:
The paper's comparative experiments include multiple models and provide statistical results, offering guidance for other researchers interested in prompt learning.
The proposed Diff-Prompt surpasses other existing efficient fine-tuning methods on multiple evaluation metrics, which may stimulate further thinking among researchers.
The authors train the prompt generator through mask supervision, controlling the generation of fine edge information via mask variations, which is a novel idea.

weaknesses:
Diff-Prompt is compared with two different types of efficient parameter tuning methods, but the number of baselines for the two methods is seriously imbalanced. It would be preferable to include more comparative experiments with adapter methods.
Compared with previous work, this paper seems focused on training masks through diffusion models to generate masks with fine edges for boundary control, but the contribution beyond this aspect is insufficient.
The authors claim to generate rich prompts, but the masks generated during the diffusion process do not exhibit finely controllable generation. This approach may not necessarily adapt to highly fine-grained tasks, which is evidently inconsistent with the problem the authors intend to solve.

questions:
The authors mentioned that Diff-Prompt may suffer from poor generalization. Based on this, have they proposed any improvement plans to mitigate this issue and enable reproduction by other researchers?
How does the strategy of Diff-Prompt differ from independently training a conditional mask generator and then performing information interaction with a multimodal encoder?
During prompt generation, did the authors consider how to maintain a balance between visual and textual information to avoid information imbalance caused by modality bias?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732189976573

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of Changes in the Rebuttal Revision

comment:
We appreciate the reviewers' time and valuable feedback. In response to their comments and suggestions, we have made several revisions to the paper. All changes have been highlighted in blue in the updated version. A summary of the updates is provided below:
- In Sections 3.1 and 3.2, we provide further explanations of the preliminaries of the method.
- In Sections 4.2 and 4.3, we provide a more detailed description of the method design and a more in-depth explanation of the principles behind these design choices.
- In Sections 5.1 and 5.2, we **supplement several state-of-the-art efficient fine-tuning methods**, including three adapter tuning methods: Tip-adapter, Meta-adapter, and MMA. Additionally, we introduce one prompt tuning method, VFPT, and analyzed the corresponding experimental results.
- In Section 5.4, we conduct an in-depth exploration of the model's **generalization capabilities** through three experiments: **Cross-dataset Generalization**, **Cross-domain Generalization**, and **Generalization Across Downstream Tasks and Backbones**. The first two experiments are commonly used generalization benchmarks, while the third experiment examines the performance of Diff-Prompt across different backbones and downstream tasks.
- In Appendix E, we add an experiment analyzing the variation of **Recall across different IoU intervals**.
- We move the category-wise accuracy from the in-depth analysis to the Appendix H.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732209235690

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
For All Reviewers

comment:
We sincerely thank all the reviewers for their insightful comments and unanimous recognition of the contributions of our work:
- **Novel and interesting motivation**: We appreciate the Reviewer  **PtHg**, **cYSP**, **LEnk** recognition of the novelty of our work and their acknowledgment that it is an intriguing and worthwhile direction for further exploration.
- **Effective method and superior performance**: We express our gratitude to **ALL** for their positive feedback, acknowledging the effectiveness of our method and the strong experimental results.

We will now address the key arguments raised in the reviews and provide detailed responses to each reviewer. We kindly encourage the reviewers to refer to the **updated version** of our paper.

Once again, we sincerely thank the reviewers for their time and effort in evaluating our work. We look forward to any additional insights or questions that may arise from our research.
</content>
</thread_object>

</thread>


Metareview:
metareview:
In this work, the authors present Diff-Prompt, a novel three-stage prompt generation method based on diffusion models, designed to enhance the fine-tuning of pretrained multimodal models by providing richer and more fine-grained information. Extensive experiments on referring expression comprehension demonstrate that the proposed method outperforms other parameter-efficient fine-tuning approaches, highlighting its effectiveness. The majority of reviewers agree that the paper is well-written, and the proposed method is novel and interesting (PtHg, cYSP, LEnk). They acknowledge the strong experimental results and the effectiveness of the proposed approach. However, some reviewers raise concerns, requesting additional experiments and analyses with more baselines and tasks to further validate the method’s effectiveness and generalization capabilities (PtHg, cYSP, LEnk). Other concerns include the slow inference speed of the diffusion model (cYSP, 3hUM) and the need for clearer explanations of the mathematical notations and contributions (A4tj). The authors provided detailed responses, addressing most of the reviewers’ concerns and significantly revising the manuscript. As a result, reviewers LEnk, cYSP, and A4tj increased their ratings from negative to positive, while reviewers PtHg and 3hUM did not participate in the final discussion. The paper ultimately received an average score of 5.8, with one accept, three borderline accepts, and one reject. I also agree that the proposed method has its own merits and recommend the paper for acceptance. Moreover,  the authors should follow the suggestions from reviewers LEnk, cYSP, and A4tj by incorporating all materials discussed during the rebuttal and including additional comparisons with learnable prompts in the final version.