Title: value: PromptCoT: Align Prompt Distribution via Adapted Chain of Thought
Authors: value: Junyi Yao
Yijiang Liu
Zhen Dong
Mingfei Guo
Jiashi Feng
Kurt Keutzer
Li Du
Daquan Zhou
Shanghang Zhang
Venue: NeurIPS 2023
Keywords: value: text-to-image generative models
prompt engineering
Chain of Thought
parameter efficient adaptation
Large Language Models

Abstract: value: Diffusion-based generative models have exhibited remarkable capability in the production of high-fidelity visual content such as images and videos. However, their performance is significantly contingent upon the quality of textual inputs, commonly referred to as "prompts".  The process of traditional prompt engineering, while effective, necessitates empirical expertise and poses challenges for inexperienced users.  In this paper, we introduce PromptCoT, an innovative enhancer that autonomously refines prompts for users.  The design of PromptCoT is based on the observation that, prompts resembling textual information corresponding to high-quality images within the training set tend to yield superior generation performance.  As such, we fine-tune the pre-trained Large Language Models (LLM) using a curated text dataset comprising solely of high-quality visual content descriptions. By doing so, the LLM becomes capable of capturing the distribution of high-quality training texts, enabling it to generate aligned continuations and revisions to boost the original texts.  Nonetheless, one drawback of pre-trained LLMs is their tendency to generate extraneous or irrelevant information. To enhance the alignment between the original text prompts and the refined counterparts, we leverage the Chain-of-Thought (CoT) mechanism. CoT can extract and amalgamate crucial information from the aligned continuation and revision, enabling reasonable inferences based on the contextual cues to produce a more comprehensive and nuanced final output.  Considering computational efficiency, instead of allocating a dedicated LLM for prompt enhancement to each individual model or dataset, we integrate adapters that facilitate dataset-specific adaptation, leveraging a shared pre-trained LLM as the foundation for this process.   By fine-tuning these adapters independently, we can adapt PromptCoT to new datasets with minimal increase in training cost and memory usage.  We assess the performance of PromptCoT on widely-used latent diffusion models for image and video generation to validate the effectiveness. The results demonstrate significant improvements in key performance metrics. 


Review discussion:
1) This paper shows an insight from the observation that prompts resembling high-quality image descriptions from the training set lead to better generation performance.  It makes sense from my perspective.  Current text-guided image generation is still with limited generalization ability.  The points near the training support samples are transferred better.
2) The idea to utilize LLM to adapt the original prompt to the one that is more aligned with training samples makes sense. It leverages the ability of LLMs to align the distributions.
3) Three training methods are proposed to implement the ideas including the continuation, revision, and CoT.
4) This paper builds the corresponding datasets to help the fine-tuning, which can benefit the following research.
5) The method using the GPT-3 to build datasets is smart.
1) The finetuning of LLaMa is time-consuming.  Could it be replaced by prompt learning or LORA?
2) Please add a discussion (e.g. in related work) with ''Visual Chain-of-Thought Diffusion Models'', though these two methods are clearly different.
3) If the datasets with neural captions can be built.  How about adding these sentences to the training set of the diffusion model?
Please refer to the weaknesses part.
It can be found in the F part of the appendix.
No ethics review needed.
We sincerely thank you for participating in the review of our work and providing meticulous reviews.2
Regarding your questions, we strive to provide as comprehensive answers as possible.
## Q1: Replace finetuning by prompt lerning or LoRA.
Indeed, the fine-tuning process for full-parameter LLaMA can be time-intensive if done individually for each distinct task, rendering it inefficient and impractical. In pursuit of streamlined multi-task adaptation, we have embraced the employment of an adapter-based architecture known as LoRA.
Experimental results demonstrate that LoRA-based adapters achieve comparable performance.
| booster | aes-score | CLIP-score|
|t2t-blip-30b | 5.90 | 0.282|
|GPT-4| 5.91| 0.280|
|cot-30b| 5.93| 0.274|
Also provided  in pdf file tabel 1.
## Q2: Add a discussion with ``Visual Chain-of-Thought Diffusion Models''.
We have thoroughly examined this study, in which the authors enhance the expressive capability of the text condition by generating a new CLIP embedding through a pre-trained diffusion model. This enhancement leads to an improvement in the quality of generated images. It's worth noting that this work is quite intriguing, showcasing methods in bridging text and image generation. We will certainly cite and discuss the work in our main paper.
In contrast, our approach begins with the textual prompt and explicitly focuses on optimizing the alignment between the prompt and the training dataset. By employing the CoT (Chain-of-Thought) method, we achieve a more effective fusion of text continuation and revision abilities. Our approach not only enhances the visual quality of the images but also ensures consistency between the textual and visual concept.
## Q3: Add neural captions to the training set of the diffusion model.
Our core objective in the study was to enhance the capabilities of the generation model while maintaining its inherent structure. This was accomplished through prompt alignment, rather than resorting to modifications in the training dataset or pursuing further fine-tuning.
The rationale behind utilizing automatic captioners to construct the dataset was to establish pairs of expressions, with one resembling human-like expressions and the other mirroring training-like expressions. In this context, the outputs generated by the automatic captioners serve as representations of human-like expressions. It enables us to utilize LLM (Large Language Model) as an aligner, facilitating the conversion of human-like expressions into training-like expressions.
I appreciate the authors taking the time to answer the questions during the rebuttal. I also read the comments of other reviewers. I still think this is good work with a clear contribution.
Thus, I keep my positive score.
1. The proposed PromptCoT is easy to follow.
2. The qualitative results is promising.
1. Introducing prompting technique to improve the quality of the textual inputs for generative models lacks novelty, since intuitively, providing more detailed information indeed improves the generation quality of the generative models. The authors could clarify the novelty, e.g., how the prompting technique or the CoT technique different from other works, or why the generated text is better than human-refined counterparts.
2. As shown in Figure 4, the proposed text revision technique may generate some uncontrollable additional information (e.g., time, weather and place). I think the prompted results may be influenced by the inherent bias of the language model or the prompting dataset. The authors could provide some analysis to show that the enhanced textual inputs are unbiased with given CoT technique, or show that the enhanced textual inputs are controllable.
See the weakness for detail. If the authors could address my conerns 1 and 2, I will raise my rating.
The authors have stated the limitation of the proposed PromptCoT, i.e., relying on the capability of the generative models and the quality of inital textual inputs. These limitations can be seen as the future direction of this work.
Ethics review needed: Inadequate Data and Algorithm Evaluation
We sincerely thank you for participating in the review of our work and providing meticulous reviews.
Regarding your questions, we strive to provide as comprehensive answers as possible.
## Q1: Intuitively, providing details improves the generation quality. Clarify the novelty.
Adding details to the text directly for performance enhancement is uncontrollable. Our "tcontinue" experiment serves as the most direct evidence. By adding details without altering the original content, the improvements it brings, as observed from various evaluation metrics in Tab.2 in the main paper, remain quite limited.
We extensively surveyed numerous relevant works and found that our approach possesses a significant level of uniqueness. To illustrate the novelty of adopting the CoT method, we provide the following comparisons.
* **[1]** employs Large Language Models (LLMs) to align with manually optimized prompts, and subsequently integrating reinforcement learning techniques to enhance performance.
* **[2]** utilizes Large Language Models (LLMs) to generate an additional set of conditional embeddings, which collaboratively interact with text embeddings to enhance performance.
* **[3]** explores the usage of models from the GPT series to refine prompts.
However, our experiments have demonstrated that the CoT approach significantly outperforms GPT-3.5 and is on par with GPT-4 in terms of effectiveness.
* **[4]** empirically summarizes various methods for optimizing prompts.
None of published studies formally discuss the problem. Our PromptCoT features a tailored 5-step CoT design that takes into account the trade-off between introducing new details and maintaining visual-textual consistency, which not only stands out as distinctive but has also proven successful.
In contrast to human refinement, our approach provides enhanced stability. This is attributed to the reliance of human refinement on intuition and experience, which introduces a greater degree of randomness in the introduced words.
For instance, in the pursuit of clearer images, the inclusion of terms such as "4k," "8k," and "high-resolution" seems logical, but they might not constitute the most optimal selections.
Conversely, our methodology concentrates on identifying the optimal refinements, with the goal of maximizing the diffusion model's generation capabilities by achieving optimal alignment within the combined prompt.
In practice, we conducted experiments shown in Tab.5 in the supplementary material, which revealed that manual refinement did not demonstrate strong performance.
On the contrary, our method proved to be more effective.
## Q2: Analyze the enhanced prompts are unbiased or controllable.
According to the design of the CoT pipeline, during the refinement process, we extract key information from the original prompt. In the expansion phase, we continue to expand based on the key information, thereby filtering out irrelevant details and ensuring semantic consistency.
Besides, in pdf file, we provide some examples which show that CoT aligner can better control refined prompts by alleviating missing crucial information and extending irrelevant information.
We evaluate that the refined prompts do not deviate through cross-CLIPScore between origin prompt and image generated by refined prompt, between refined prompt and image generated by origin prompt.
|booster| CLIP-score origin prompt| CLIP-score refined prompt|
|N/A| 0.26| 0.26|
|t2t-blip-30b| 0.23| 0.21|
|cot-30b| 0.24|0.25|
Also provided  in pdf file table 2.
## Reference
**[1]** Hao, Yaru, et al. "Optimizing prompts for text-to-image generation." arXiv preprint arXiv:2212.09611 (2022).
**[2]** Zhong, Shanshan, et al. "Sur-adapter: Enhancing text-to-image pre-trained diffusion models with large language models." arXiv preprint arXiv:2305.05189 (2023).
**[3]** Zhu, Wanrong, et al. "Collaborative Generative AI: Integrating GPT-k for Efficient Editing in Text-to-Image Generation." arXiv preprint arXiv:2305.11317 (2023).
**[4]** Oppenlaender, Jonas. "A taxonomy of prompt modifiers for text-to-image generation." arXiv preprint arXiv:2204.13988 2 (2022).
Thanks for your response. Most of my concern has been solved. I will raise my rating.
1.	The overall presentation is good, and the paper is simple to follow.
2.	The visualization is very interesting.
For example, Figure 1 is very impressive.
3.	The overall pipeline is well encapsulated in Figure2-4. It is simple for the reader to understand the high-level idea of the work by looking at the figures.
Major
1.	Novelty is a big concern in this work and there is no technical contribution in this work. The author basically uses many off-the-shelf techniques to augment/modify the input text.  Text-continuation is also commonly used for pretraining large LLM. The adaptation techniques in section 3.4 are also not novel and the author simply leverages the existing techniques. In general, it is unclear what is the technical contribution of this work.
2.	L258 Is there any reason why t2t-blip booster demonstrates the best performance? Any theoretical reason behind this?
Minor
1.	L224 gpt to “GPT”
Overall, there is no big drawback in this work.
However, there is not much technical contribution in this work as well. This work might be useful for industrial applications, but its publication might not benefit the community.
Limitation is discussed in appendix
No ethics review needed.
We sincerely thank you for participating in the review of our work and providing meticulous reviews.
Regarding your questions, we strive to provide as comprehensive answers as possible.
## Q1: Technical contribution is unclear.
Expanding the text input does not guarantee a definite improvement in performance. Our investigation has unveiled a notable insight: even when individuals possessing academic expertise engage in the process of refinement, it can paradoxically result in a reduction in overall performance (as depicted in Table 5 in the supplementary materials).
Conversely, certain automated methodologies, as elaborated in our paper, provide evidence that the potential for performance enhancement through prompt refinement remains largely untapped. Our motivation lies in the pursuit of identifying the most optimal refinement approach. Through an in-depth analysis presented in our paper, we have discovered a pronounced correlation between the refined prompts and the distribution of textual content within the training dataset.
Then we fine-tune the LLM on the textual data extracted from the training set of the pre-trained diffusion model. Leveraging the model's inherent tendency to overfit, we ingeniously harnessed its capacity to automatically discern and select the most fitting synonymous word candidates from a myriad of options. These choices are tailored to harmonize seamlessly with the underlying pre-trained SD model. This motivation diverges fundamentally from the conventional application of LLM in direct prompt refinement, embodying a unique and distinctive approach.
Furthermore, we extensively surveyed numerous relevant works and found that our approach possesses a significant level of uniqueness. To illustrate the novelty of adopting the CoT method, we provide the following comparisons.
* **[1]** employs Large Language Models (LLMs) to align with manually optimized prompts, and subsequently integrating reinforcement learning techniques to enhance performance.
* **[2]** utilizes Large Language Models (LLMs) to generate an additional set of conditional embeddings, which collaboratively interact with text embeddings to enhance performance.
* **[3]** explores the usage of models from the GPT series to refine prompts.
However, our experiments have demonstrated that the CoT approach significantly outperforms GPT-3.5 and is on par with GPT-4 in terms of effectiveness.
* **[4]** empirically summarizes various methods for optimizing prompts.
None of published studies formally discuss the problem. Our PromptCoT features a tailored 5-step CoT design that takes into account the trade-off between introducing new details and maintaining visual-textual consistency, which not only stands out as distinctive but has also proven successful.
## Q2: Explain why t2t-blip booster demonstrates the best performance.
Both t2t-blip and t2t-inter utilize the same revision approach;
however, t2t-blip exhibits superior performance compared to t2t-inter. Our analysis revealed that the captions generated by blip demonstrated greater consistency with the text descriptions in our validation set (COCO). This enhanced the ability of the LLM to effectively learn the alignment between COCO and LAION, thereby resulting in superior performance. To harness the benefits of both continuation and revision advantages within the CoT method, we employed a more robust pre-trained model, llama-30B, which yielded the most promising outcomes in our latest experimental evaluations.
| booster | aes-score | CLIP-score|
|t2t-blip-30b | 5.90 | 0.282|
|GPT-4| 5.91| 0.280|
|cot-30b| 5.93| 0.274|
Also provided  in pdf file tabel 1.
## Q3: L224 gpt to “GPT”
Thank you for the correction. We will carefully review the revision.
## Reference
**[1]** Hao, Yaru, et al. "Optimizing prompts for text-to-image generation." arXiv preprint arXiv:2212.09611 (2022).
**[2]** Zhong, Shanshan, et al. "Sur-adapter: Enhancing text-to-image pre-trained diffusion models with large language models." arXiv preprint arXiv:2305.05189 (2023).
**[3]** Zhu, Wanrong, et al. "Collaborative Generative AI: Integrating GPT-k for Efficient Editing in Text-to-Image Generation." arXiv preprint arXiv:2305.11317 (2023).
**[4]** Oppenlaender, Jonas. "A taxonomy of prompt modifiers for text-to-image generation." arXiv preprint arXiv:2204.13988 2 (2022).
The rebuttal has addressed my concerns and the score is raised.
Application of LLMs to image generation for better text alignment is reasonable. The authors prepared datasets and conducted curated pipeline to achieve this goal. The presentation is clear.
1. the observation that prompts aligned with high-quality images withing the training set are more inclined to yield visually superior outputs needs more demonstration. The authors verified this on  LAION dataset.
But it does not mean this is always the truth on all datasets. Whether it works on Flickr, Pinterest images?
2. The application of LLMs to refine texts is widely used now. Prompt engineering, Cot, adapters are also widely explored. Only the Application of these techniques seems not novel enough.
3. The pipeline may needs more explanation on why we need text continuation first then text revision.
4. the experiments are quite confusing. what are t2t-inter, cot_d, cot? the scores in Table 2 shows the effectiveness of each booster. How do they work together?  Which booster works the best  according to these experiments. what are Table 3 and Table 4 used for? It's really hard to summarize a conclusion that can match the design of the whole method.
questions are listed in weakness.
limitations are presented in the paper.
No ethics review needed.
We sincerely thank you for participating in the review of our work and providing meticulous reviews.
Regarding your questions, we strive to provide as comprehensive answers as possible.
## Q1: Evaluate effectiveness of CoT aligner datasets besides LAION.
The most prominent open-source generative models, such as SD, VQ-diffusion, Imagen, etc., have achieved remarkable performance through training on the LAION dataset. Our objective is to refine the textual prompt to align to the training text, without altering the generative model itself.
Consequently, we utilize the most popular training data of the leading generative models.
Furthermore, we have extended our validation to text-to-video models in the main paper, which yielded results affirming the effectiveness of our aligner. This substantiates not only the efficacy of our aligner approach but also its capability to generalize across different modalities.
## Q2: The application of LLMs to refine texts is widely used now. Prompt engineering, Cot, adapters are also widely explored. Only the Application of these techniques seems not novel enough.
Conventional wisdom held that expanding input text was believed to enhance performance in text generation.
However, our investigation challenges this notion, revealing that involving academic experts in prompt refinement may paradoxically diminish overall performance (illustrated in Table 5 in the supplementary materials). Our approach demonstrates that the latent potential for performance improvement through prompt refinement remains untapped. We uncover a profound correlation between finely honed prompts and the distribution of textual nuances within the training dataset. Our methodology strategically fine-tunes LLM, empowering them to select fitting synonyms and seamlessly integrate them into the output. This approach diverges from conventional prompt enhancement methods. Our primary motivation is to identify the most optimal approach for prompt refinement. In our paper, we present a comprehensive analysis that reveals a strong correlation between refined prompts and the distribution of textual content within the training dataset.
To achieve this, we employ a fine-tuning process on the LLM using the textual data extracted from the training set of the pre-trained diffusion model. Leveraging the LLM's inherent tendency to overfit, we creatively utilize its capacity to automatically discern and select the most suitable synonymous word candidates from a wide range of options. These carefully chosen words are seamlessly integrated into the underlying pre-trained SD model, resulting in a harmonious combination. It is important to note that our motivation significantly deviates from the conventional approach of using LLM for direct prompt refinement, making our methodology unique and distinctive.
## Q3: The pipeline may needs more explanation on why we need text continuation first then text revision.
In our experiments, we separately employed two methods, revision and continuation. From the results of text refinement, we found that continuation can bring both useful and irrelevant information, while revision can greatly improve alignment.
However, revision also runs the risk of altering the main content and omitting important information.
Therefore, we constructed the CoT to complement the strengths and weaknesses of each method.
In addition to the sequential order of performing continuation before revision in our CoT approach, we have also carefully designed methods for extracting the main content and adding relevant details. This ensures that the modified text not only contains rich and nuanced information but also avoids ambiguity or conflicting interpretations compared to the original text.
## Q4: Results in table is not explained clearly enough.
We have provided descriptions of the aligners in both the main paper (line 240-248) and the supplementary materials (line 67-73). We will incorporate additional descriptions in the revised version. Here we provide a brief overview of the roles of each aligner:
* t-continue: represents the continuation method.
* t2t-blip: represents the rewriting method with BLIP incorporation for captioning during the training process.
* t2t-inter: represents the rewriting method with the inclusion of clip-interrogator for captioning during training.
* CoT_d: refers to the approach that exclusively employs the CoT dataset for fine-tuning.
* CoT: Signifies the aligner that employs all five types of datasets as outlined in Figure 5 in the main paper for fine-tuning.
These aligners function separately. During the initial phases of our investigation, we conducted separate explorations of the continuation and revision methodologies. Substantive enhancements in performance were discerned with each respective approach, accompanied by discernible variances in outcomes.
Consequently, the prospect of harnessing the strengths of both approaches through a CoT was entertained. Subsequent empirical investigations involving both CoT_d and CoT substantiated that the direct revision methodology yielded markedly subpar outcomes in contrast to the incremental CoT procedure.
In order to harness the full potential of the CoT methodology, we took a progressive step by incorporating a substantially larger-scale Language Model (LLM) with an expansive parameter count of 30 billion. This augmentation was accompanied by the implementation of the QLoRA technique, which facilitated efficient fine-tuning operations. The outcome of this strategic augmentation culminated in the development of our CoT_30B aligner, which notably exhibited unparalleled levels of performance within the context of our study.
Hi Reviewer 2oA6,
Would you please respond to the Authors' rebuttal? Thanks!
Best,
the rebuttal addressed some of my concerns. I agree that the the prompt refinement method in this paper is different from previous works. I raised my rating.
But I still think it lacks technical contribution. The combination of off-the-shell techniques is more like a good, successful practice to utilize LLM for better image generation on specific models in industry, rather than a novel technique contribution.
# To ALL
We sincerely thank all reviewers for your comments. After reading all theomments carefully, we add more corresponding results and image examples to te rebuttal PDF file. We hope this rebuttal can address your concerns. If you have other concerns, we will reply as soon as possible.
We sincerely thank all reviewers for your comments. After reading all comments carefully, we add more corresponding results and image examples to the rebuttal PDF file. Below we will reaffirm our contribution based on the latest experiments, illustrate novelty through more extensive investigation, and explain the image examples in a PDF file. We hope this rebuttal can address your concerns. If you have other concerns, we will reply as soon as possible.
## Contribution
* We propose PromptCoT, an innovative prompt refiner that aligns input prompts with the text distribution employed during the training of diffusion models, along with our unique dataset for finetuning. By accomplishing this alignment, PromptCoT effectively activates generative models and enhances their performance.
* The latest CoT aligner based on llama-30b LLM shows significant ability for refining prompts. It surpasses GPT-4 when simply asking GPT-4 to regenerate a better prompt.
Besides, our CoT aligner performs better than GPT-4 even when asking GPT-4 to refine following our 5 refine steps.
* We evaluate the performance of replacing full-finetune by adapters and LoRA which are both of which are comparable to full-finetune.
Thus, our CoT method can use such technics easily adapting to different generation models for both image and video.
## Novelty comparison
Our investigation has revealed a significant insight: even when individuals possessing academic expertise engage in the process of refinement, it can paradoxically result in a reduction in overall performance (as shown in Table 5 in the supplementary materials).
Conversely, certain automated methodologies, as detailed in our paper, provide evidence that the potential for enhancing performance through prompt refinement remains largely untapped. Our motivation lies in the pursuit of identifying the most optimal refinement approach. Through a comprehensive analysis presented in our paper, we have uncovered a pronounced correlation between the refined prompts and the distribution of textual content within the training dataset.
Then we fine-tune the LLM on the textual data extracted from the training set of the pre-trained diffusion model. Leveraging the model's inherent tendency to overfit, we ingeniously harnessed its capacity to automatically discern and select the most fitting synonymous word candidates from a myriad of options. These choices are tailored to harmonize seamlessly with the underlying pre-trained SD model. This motivation diverges fundamentally from the conventional application of LLM in direct prompt refinement, embodying a unique and distinctive approach.
We make a comparison with extensive related work and it illustrates that PromptCoT possesses a significant level of uniqueness.
* **[1]** employs Large Language Models (LLMs) to align with manually optimized prompts, and subsequently integrating reinforcement learning techniques to enhance performance.
* **[2]** utilizes Large Language Models (LLMs) to generate an additional set of conditional embeddings, which collaboratively interact with text embeddings to enhance performance.
* **[3]** explores the usage of models from the GPT series to refine prompts.
However, our experiments have demonstrated that the CoT approach significantly outperforms GPT-3.5 and is on par with GPT-4 in terms of effectiveness.
* **[4]** empirically summarizes various methods for optimizing prompts.
None of published studies formally discuss the problem. Our PromptCoT features a tailored 5-step CoT design that takes into account the trade-off between introducing new details and maintaining visual-textual consistency, which not only stands out as distinctive but has also proven successful.
## Image Examples
To show the CoT aligner's ability on generating controllable prompts, the 3 sets of 4 images from left to right are generated by the origin prompt, t-continue aligner refined prompt, t2t-blip aligner refined prompt, and CoT aligner refined prompt. For the first set, the t-continue aligner adds information about 'mask' which makes the diffusion model ignore 'tie', t2t-blip only focuses on "man" ignoring 'tie', CoT aligner maintains both and emphasizes the action 'showing off'. The second set's missing main concept is the wall while the third set's is three pillows.
## References
**[1]** Hao, Yaru, et al. "Optimizing prompts for text-to-image generation." arXiv preprint arXiv:2212.09611 (2022).
**[2]** Zhong, Shanshan, et al. "Sur-adapter: Enhancing text-to-image pre-trained diffusion models with large language models." arXiv preprint arXiv:2305.05189 (2023).
**[3]** Zhu, Wanrong, et al. "Collaborative Generative AI: Integrating GPT-k for Efficient Editing in Text-to-Image Generation." arXiv preprint arXiv:2305.11317 (2023).
**[4]** Oppenlaender, Jonas. "A taxonomy of prompt modifiers for text-to-image generation." arXiv preprint arXiv:2204.13988 2 (2022).

Meta-review:
role: PC
data: comment: value: The paper received 5/4/7/5 ratings. Reviewers were concerned about the significance of technical contribution of the paper. After the rebuttal discussion, two reviewers raised the score, while others kept the ratings. After reading the paper, the reviews and the rebuttal discussion, the AC tends to agree with reviews that the paper’s technical significance is less competitive compared to other submissions. The AC recommends rejection.

Decision: Reject