Prompt Diffusion Robustifies Any-Modality Prompt Learning

Authors: Yingjun Du, Gaowen Liu, Yuzhang Shang, Yuguang Yao, Ramana Rao Kompella, Cees G. M. Snoek
Primary area: generative models
Keywords: Prompt learning, Diffusion model, Vision-language models

Abstract:
Foundation models enable prompt-based classifiers for zero-shot and few-shot learning. Nonetheless, the conventional method of employing fixed prompts suffers from distributional shifts that negatively impact generalizability to unseen samples. This paper introduces prompt diffusion, which uses a diffusion model to gradually refine prompts to obtain a customized prompt for each sample. 
Specifically, we first optimize a collection of prompts to obtain over-fitted prompts per sample. Then, we propose a prompt diffusion model within the prompt space, enabling the training of a generative transition process from a random prompt to its overfitted prompt. As we cannot access the label of a test image during inference, our model gradually generates customized prompts solely from random prompts using our trained, prompt diffusion. Our prompt diffusion is generic, ﬂexible, and modality-agnostic, making it a simple plug-and-play module seamlessly embedded into existing prompt learning methods for textual, visual, or multi-modal prompt learning.
Our diffusion model uses a fast ODE-based sampling strategy to optimize test sample prompts in just five steps, offering a good trade-off between performance improvement and computational efficiency.
For all prompt learning methods tested, adding prompt diffusion yields more robust results for base-to-new generalization, cross-dataset generalization, and domain generalization in classification tasks tested over 15 diverse datasets.

Decision: reject
Reviewer scores: 6, 6, 3

Reviews and discussion:
<thread>
creation date:
1730082175957

thread_type:
official_review

<thread_object>
creator: reviewer_xqzt

<content>
summary:
This paper introduces a novel framework, Prompt Diffusion, which aims to improve the generalizability and robustness of prompt-based learning across various modalities (e.g., visual, textual, multimodal). In prompt-based learning, especially for foundation models in zero-shot and few-shot learning, fixed prompts often suffer from distributional shifts, impacting performance on unseen data. Prompt Diffusion leverages a diffusion model to refine prompts gradually, transforming them from a generic to a sample-specific prompt. This process enhances the robustness and adaptability of prompts across datasets with distinct distributions, providing a plug-and-play solution compatible with existing prompt-learning methods.

strengths:
1.	Introduces an innovative, modality-agnostic diffusion process that significantly enhances robustness in prompt-based learning.
2.	Demonstrates consistent empirical improvements across various prompt learning tasks, supporting the efficacy of diffusion models.
3.	Efficient design reduces inference time, making it suitable for diverse real-world applications.

weaknesses:
1.	The paper does not fully articulate the specific limitations of the SOTA prompt mehthods in adapting to distributional shifts in data, which creates ambiguity around the critical nature of these issues within broader prompt-learning applications. To make this critique more actionable, the authors could quantify the performance degradation caused by these shifts in existing methods to better contextualize the importance of their contribution.  Specific examples are not enough to illustrate the problem.
2.	Although the diffusion model is proposed to generate sample-specific, customized prompts, the paper does not clearly explain why diffusion was chosen over other, potentially simpler methods. This raises questions about the model's unique contributions and practical effectiveness. For instance, if simpler statistical methods like ProDA[1] are available, what advantages does the complex diffusion model offer? Moreover, there are already several statistical approaches for prompt learning, such as Bayesian Prompt Learning[2], which the authors could consider referencing.
3.	The approach has limited empirical exploration outside the image-text domain, raising questions about its generalizability to other modalities. To strengthen this point, the authors could discuss the potential challenges and adaptations needed to apply their method to other modalities, such as audio or video. 
4.	The high resource demands of diffusion models, including substantial GPU and training time requirements, make them impractical for parameter-efficient methods such as prompt learning. The complexity and cost of implementing diffusion models in this context undermine their accessibility and practicality. 

[1] Lu, Yuning, et al. "Prompt distribution learning." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.

[2] Derakhshani, Mohammad Mahdi, et al. "Bayesian prompt learning for image-language model generalization." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

questions:
1. What are the key motivations behind using diffusion models for prompt learning, and how does it address the limitations of fixed prompts?
2. How does Prompt Diffusion leverage the diffusion model to gradually transition from a random to a sample-specific prompt?
3. In what ways does Prompt Diffusion enhance generalization capabilities across base-to-new, cross-dataset, and domain generalization tasks?
4. How does Prompt Diffusion ensure compatibility with existing prompt learning models across textual, visual, and multimodal prompts?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730454951633

thread_type:
official_review

<thread_object>
creator: reviewer_kjln

<content>
summary:
In this paper, the authors introduce prompt diffusion, which utilizes a diffusion model to refine prompts for each input image, thereby enhancing the model's ability to generalize across different distributions. The proposed prompt diffusion is a straightforward plug-and-play module that can be seamlessly integrated into existing prompt learning frameworks.

strengths:
1. Experiments have shown that the proposed method outperforms baseline methods.​

2. The overall idea is intuitive and straightforward, addressing the limitations of fixed prompts by leveraging diffusion models to generate over-fitted prompts per sample, which enhances model robustness against distribution shifts.

weaknesses:
1. Considering that the proposed method is conducted on per sample. during training, does it introduce a significantly larger computational load compared to conventional prompt learning methods? Can a comparative analysis be provided to address this concern?

2. While the proposed method is plug-and-play and the pipeline figure demonstrations are based on CoCoOp, it would be beneficial to include sections addressing visual prompt tuning and multi-modal prompt tuning.  Additionally, the method emphasizes the meta-net π within CoCoOp, but it is unclear how it handles other prompt learning methods that do not involve π, such as VPT and MaPLe.

3. The length of prompts in prompt learning methods can affect the final performance.  Does the proposed method also encounter similar situations?  It is encouraged for the authors to supplement relevant ablation studies to address this concern.

4. There are also some works in the field of prompt learning that address the limitations of fixed prompts by generating instance-level prompts (e.g. [1]).  It is recommended that the authors supplement the related work to make the paper more comprehensive.

[1] Xinyang Liu ,et al. Patch-Token Aligned Bayesian Prompt Learning for Vision-Language Models. UAI 2024

questions:
1. I am curious about the setting of the two loss weights β in Equation (8). Can further experimental analysis be provided?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730535755550

thread_type:
official_review

<thread_object>
creator: reviewer_14m8

<content>
summary:
This paper proposes a method called Prompt Diffusion, which employs a diffusion model to progressively refine prompts, enabling customized prompts for each sample. By introducing a technique for creating tailored prompts for individual test samples, this method addresses the limitations of fixed prompts, enhancing the model's robustness to distribution shifts. Empirical results on extensive datasets validate the effectiveness of this approach, demonstrating its robustness in generalization tasks.

strengths:
1. The method in this paper generates customized prompts for each sample by gradually optimizing the prompts through diffusion, which enhances the accuracy of prediction and generalization across downstream tasks.
2. The diffusion prompting method in this paper is a plug-and-play module that can be seamlessly integrated into existing textual, visual, or multimodal prompt learning methods.
3. The method in this paper improves the prompt learning process by efficiently extracting unique domain details from test images without mixing them with class labels.

weaknesses:
1.  The authors' method requires stepwise optimization of the prompts and may require several iterations to obtain optimal results, in addition, the introduction of a diffusion model increases the complexity of the system, and therefore whether the training time is likely to be relatively long.
2.  Whether the authors' approach is a two-stage process, where prompt learning is performed first, followed by diffusion of the prompts, and the final model performance relies on the goodness of the previously learned prompts. In addition, the diffusion process relies on random noise vectors to generate the prompts and therefore may be sensitive to noise, which may affect the stability of the final performance.

questions:
1.  Is the author's approach a two-stage process, starting with a prompt study followed by a prompt proliferation.
2. Diffusion models incorporate randomness in the generation process, which may lead to uncontrollable fluctuations in the generated prompts and thus affect the robustness of the model. How to cope with the randomness of the generated prompts and avoid the instability of prediction caused by it?
3. The authors' approach seems to be applicable only to VPT-shallow prompt types, and whether the authors' approach can be migrated to the VPT-deep prompt learning paradigm.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732208755719

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of Revisions

comment:
We thank all reviewers for their constructive feedback and recognition of the strengths of our work. In response to the insightful suggestions and concerns raised, we have made several significant improvements and additions to the revised manuscript:

1. Included the effect of prompt length on performance (Appendix E).
2. Added a comparative analysis of training time and inference efficiency (Appendix F).
3. Included new experiments on VPT-deep to demonstrate the versatility of our method (Appendix G).
3. Expanded discussions on the adaptability of our method to VPT and MaPLe without relying on the meta-net (Appendix H).
4. Applied our method to video understanding tasks, such as HMDB-51, UCF-101, K-400, and K-700, to evaluate generalizability beyond the image-text domain (Appendix I).
5. Conducted additional evaluations on distributional shifts, including covariate, label, concept, conditional, and multiple shifts, in combination with "Any-Shift Prompting for Generalization over Distributions" (Xiao et al., CVPR 2024) (Appendix J).
6. Performed ablation study of loss weight $\beta$ to validate parameter choices (Appendix K).

We believe these updates comprehensively address the reviewers' concerns and enhance the clarity, robustness, and scope of our work. We thank the reviewers again for their valuable feedback, which has been instrumental in improving this submission.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The main idea of this paper is to apply diffusion modeling to refining instance-specific prompts, and the method is tested on multiple image classification benchmarks. The reviewers generally liked the idea of prompt diffusion but did not find the motivation strong enough. In particular, the paper does not explain clearly why diffusion is the right choice compared to other designs. After reading the paper, the AC has the same doubt as the reviewers. Moreover, the reviewers pointed out that the method has a much higher complexity than other baselines due to the diffusion process, thus diverging from the principle of parameter-efficient fine-tuning. The AC also agrees with this comment and feels that due to this heavy design the method is less likely to be adopted in practice and therefore has limited value to the community.