Title: PromptCoT: Align Prompt Distribution via Adapted Chain of Thought
Venue: NeurIPS 2023
Accepted: no
Keywords: text-to-image generative models

prompt engineering

Chain of Thought

parameter efficient adaptation

Large Language Models
Abstract: Diffusion-based generative models have exhibited remarkable capability in the production of high-fidelity visual content such as images and videos. However, their performance is significantly contingent upon the quality of textual inputs, commonly referred to as "prompts".  The process of traditional prompt engineering, while effective, necessitates empirical expertise and poses challenges for inexperienced users.  In this paper, we introduce PromptCoT, an innovative enhancer that autonomously refines prompts for users.  The design of PromptCoT is based on the observation that, prompts resembling textual information corresponding to high-quality images within the training set tend to yield superior generation performance.  As such, we fine-tune the pre-trained Large Language Models (LLM) using a curated text dataset comprising solely of high-quality visual content descriptions. By doing so, the LLM becomes capable of capturing the distribution of high-quality training texts, enabling it to generate aligned continuations and revisions to boost the original texts.  Nonetheless, one drawback of pre-trained LLMs is their tendency to generate extraneous or irrelevant information. To enhance the alignment between the original text prompts and the refined counterparts, we leverage the Chain-of-Thought (CoT) mechanism. CoT can extract and amalgamate crucial information from the aligned continuation and revision, enabling reasonable inferences based on the contextual cues to produce a more comprehensive and nuanced final output.  Considering computational efficiency, instead of allocating a dedicated LLM for prompt enhancement to each individual model or dataset, we integrate adapters that facilitate dataset-specific adaptation, leveraging a shared pre-trained LLM as the foundation for this process.   By fine-tuning these adapters independently, we can adapt PromptCoT to new datasets with minimal increase in training cost and memory usage.  We assess the performance of PromptCoT on widely-used latent diffusion models for image and video generation to validate the effectiveness. The results demonstrate significant improvements in key performance metrics. 

Abstract

Diffusion-based generative models have exhibited remarkable capability in the production of high-fidelity visual content such as images and videos. However, their performance is significantly contingent upon the quality of textual inputs, commonly referred to as "prompts". The process of traditional prompt engineering, while effective, necessitates empirical expertise and poses challenges for inexperienced users. In this paper, we introduce PromptCoT, an innovative enhancer that autonomously refines prompts for users. The design of PromptCoT is based on the observation that, prompts resembling textual information corresponding to high-quality images within the training set tend to yield superior generation performance. As such, we fine-tune the pre-trained Large Language Models (LLM) using a curated text dataset comprising solely of high-quality visual content descriptions. By doing so, the LLM becomes capable of capturing the distribution of high-quality training texts, enabling it to generate aligned continuations and revisions to boost the original texts. Nonetheless, one drawback of pre-trained LLMs is their tendency to generate extraneous or irrelevant information. To enhance the alignment between the original text prompts and the refined counterparts, we leverage the Chain-ofThought (CoT) mechanism. CoT can extract and amalgamate crucial information from the aligned continuation and revision, enabling reasonable inferences based on the contextual cues to produce a more comprehensive and nuanced final output. Considering computational efficiency, instead of allocating a dedicated LLM for prompt enhancement to each individual model or dataset, we integrate adapters that facilitate dataset-specific adaptation, leveraging a shared pre-trained LLM as the foundation for this process. By fine-tuning these adapters independently, we can adapt PromptCoT to new datasets with minimal increase in training cost and memory usage. We assess the performance of PromptCoT on widely-used latent diffusion models for image and video generation to validate the effectiveness. The results demonstrate significant improvements in key performance metrics.

1 Introduction

In recent years, deep generative models have made notable advancements, specifically with the introduction of diffusion probabilistic models (DPMs). These models have exhibited exceptional capabilities in generating a wide range of visually compelling and high-fidelity visual contents, such as images and videos, as evidenced by notable contributions in the literature [37, 12, 38, 36, 7, 28, 32, 30].

By harnessing textual inputs as conditional guidance, diffusion models have the ability to generate 34

visual outputs that align with the corresponding input text, utilizing an iterative denoising procedure. 35

This technological advancement has paved the way for revolutionary applications, including notable 36

examples such as DALL-E 2 [28], Stable Diffusion [30], MagicVideo [50], among others. 37

(a)

(b)

(c)

(d)

Nevertheless, the quality of the generated content is intricately tied to the caliber of the textual 38

39

prompts provided to the generative model. Human inputs tend to be informal and straightforward,

40

41

42

43

44

45

46

47

48

49

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

which may impede the expression of the desired scene with the desired level of depth. Additionally, the text encoder within the generative model may not fully comprehend the semantic nuances present

in the human-generated text, resulting in notable disparities between the encoded textual guidance and the user's intended meaning. Diffusion probabilistic models (DPMs) are commonly trained on

extensive text-vision pairs acquired through web-scraping techniques [35]. Our observation reveals that the distribution of the text dataset might not be congruent with the linguistic style employed by

layman users. Furthermore, even in cases where the training text data aligns with the desired style, the quality can exhibit substantial variations due to the presence of meaningless words or extraneous

information within the text data. This intricacy further complicates the establishment of a clear and unambiguous mapping between the text and the corresponding image.

As a result, there is an immediate imperative to develop a methodology that can effectively align prompts, consequently augmenting the image generation performance in generative models. Although data cleaning and model fine-tuning have been considered potential solutions, these methods often entail drawbacks such as high costs, instability, and time intensiveness. Another alternative is manual prompt engineering, which involves refining prompts to optimize generation performance. However, this empirical task traditionally demands the expertise of experienced professionals, thereby posing a significant challenge for individuals lacking relevant experience.

In our study, we observe a noticeable trend that prompts, which resemble those found in the training set, usually lead to superior generative performance. Stemming from this observation, we propose PromptCoT, a novel prompt booster that leverages the power of pre-trained Large Language Models (LLMs) and incorporates the Chain-of-Thought (CoT) mechanism to learn high-quality prompt expressions from the training texts of generative models. Specifically, we carry out the fine-tuning of LLaMA [40], a widely-used pre-trained Large Language Model, on two distinct datasets we've prepared. With a text-continuation dataset that appends aligned details to original prompts, and a text-revision dataset that rewrites original prompts to aligned prompts, we enable LLaMA to refine prompts that better match the distribution of the text data used for training the diffusion models. To further enhance the performance of LLMs by combining the advantages of both text-continuation and text-revision, we construct a dataset using the CoT mechanism assisted by ChatGPT. This CoT dataset is designed to enable LLMs to reason and generate text that follows a logical and coherent flow. By fine-tuning LLMs on this CoT dataset, we can enhance their reasoning ability and augments their capacity to generate high-quality text that is both contextually relevant and logically coherent.

71 efficient adaptation design into the training pipeline of PromptCoT, augmenting a pre-trained base 72

booster with specific lightweight adapters that are capable of aligning text distributions for various 73 generative models across multiple tasks. We demonstrate the effectiveness of PromptCoT through 74 extensive experiments on widely-used latent diffusion models for image and video generation, 75 showing significant improvements in key performance metrics such as Fréchet Inception Distance, 76 aesthetic score, and CLIP-similarity. 77

78

79

80

81

82

83

84

85

86

87

88

89

90

91

92

93

94

95

96

97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

Our main contributions are:

We demonstrate that allocating a dedicated Large Language Model (LLM) for each diffusion model is not a requirement. Instead, we propose an innovative scheme where a set of lightweight adapter weights suffices for each dedicated diffusion model. These adapters can share a shared base pre-trained LLM, resulting in a considerable reduction in memory footprint.

2 Related Work

2.1 Text-to-Image Generative Models

Text-to-Image Generative Models operate by taking natural language descriptions as input and generating corresponding images as output. One of the recent popular model is DALL·E 2 [29]. It utilize CLIP [26] to align the text and image embeddings. By conditioning the diffusion probabilistic generator on the textual embedding, DALL·E 2 is able to produce photorealistic images that correspond to the given textual description. Later, Google's Imagen [32] and Parti [46] were proposed by gradually simulating the spread of noise into the original image to reveal the desired image. Specifically, both Parti and Imagen combine autoregressive and diffusion. The application of diffusion probabilistic models has also been extended to the domain of video generation. The Video Diffusion Model [13], built upon the foundations of diffusion models, enables the sequential generation of high-quality video frames. To address the substantial computational requirements associated with video generation, MagicVideo [51] was introduced, combining latent diffusion and attention models. MagicVideo utilizes a frame-wise lightweight adapter and an attention module to effectively adjust the image-to-video distribution and capture temporal dependencies across frames.

2.2 Large Language Models

Large Language Models (LLMs) are powerful deep learning models for various natural language processing tasks. The most popular LLMs are the GPT [27, 5] series models developed by OpenAI, which are based on the decoder component of the transformer architecture. Another LLM is Meta's OPT [49], which is open-sourced and performs similarly in performance to GPT-3. However, GPT-3's massive size of 175B parameters requires significant computing power and resources, which makes it challenging for researchers to explore. In contrast, LLaMA [40, 41], StableLM [2], as well as the instruction-following Alpaca model [39] are smaller and more performant, achieve comparable results to ChatGPT with far fewer parameters (7B). For specific tasks like conversational applications, ChatGLM [47, 9] can generate coherent and contextually relevant responses in dialogue systems.

2.3 Parameter-Efficient Fine-Tuning

The goal of parameter-efficient fine-tuning is to attain comparable performance to fine-tuning on a specific downstream task while using the fewest trainable parameters possible. According to [1], common pre-trained models generally have a very low intrinsic dimension, and LoRA [15] learns low-rank parameterizations to enhance tuning efficiency based on that. Except reducing the number of parameters needed for fine-tuning, other approaches try to attach pre-trained parameters to reduce training time. Adapter training [14, 24] utilizes dynamic pre-trained adapters for different tasks and languages to reduce adaptation time. Compacter [21] combines both concepts and builds on top of adapters, low-rank optimization, and parameterized hypercomplex multiplication layers.

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

2.4 Prompt Engineering

Prompt Engineering is to optimize the outputs of language models with specific input prompts [4, 33, 20, 8]. Discrete text prompts [16] serve as starting points for the model's language generation, and are used to generate responses in dialogue systems. Beyond discrete prompts, [17, 43] explores prompt tuning to learn soft prompts to perform specific downstream tasks, which provide more context-aware guidance to the model. [25] extends the idea of learning soft prompts and demonstrates that the implicit factual knowledge in language models was underestimated. Given that manually designing prompts can be cumbersome, automatically generating prompts gives a chance avoid intensive labor and enhance efficiency [33, 34]. [10] proposes to generate all prompt candidates and selectively incorporate them into each context using a refined strategy. [11] introduces a more efficient method to construct prompts with several sub-prompts that employs prompt tuning with rules without searching. Overall, prompt engineering is an efficient approach that helps bridge the gap between pre-training and fine-tuning.

2.5 Chain-of-Thought

Chain-of-Thought is a specialized tool designed for the task of multi-step reasoning and decisionmaking [44]. The traditional prompting method [4] performs poorly when it comes to tasks that require reasoning abilities. Inspired by the concept of using intermediate steps to solve reasoning problems [19, 6], the chain of thought method mimics a step-by-step thinking process and breaks down multi-step problems into intermediate steps, enabling the model to deduce more accurate results [23]. Additionally, [52] address the challenge of dealing with tasks that are more complex than example prompts, and proposes the least-to-most prompting approach which breaks down complex problems into smaller and easier subproblems. Moreover, [42] introduces self-consistency as a replacement for the greedy decoding algorithm, which samples and selects the most consistent reasoning paths to replace the greedy set.

3 Method

Inputprompts

Continuationresults

A black Honda motorcycle

A black Honda motorcycle

parked in front of a garage

Language

parked in front of a garage,

with a Honda logo on the

garage door.A Honda

Model

motorcycleparked in front of

Instruction

a garage, with a Honda logo

on the garage door. honda

pattern

Adapter 1:

logo stock pictures,royalty-

CoT aligner for SDv-1

freephotos&images

Textcontinuation

Below is a text imitation

Revisionresults

task-

Honda Valkyrie motorcycle

###Input:A black·

ndno###

Adapter 2:

in the garage wallpaper

Generative

CoT aligner for MagicVideo

modal

Text-revision

CoTresults

Below is a text extending

Black Honda CXC off-road

### Input: A black..

task...

diagonally in front of rustic

motorcycle parked

dno###

earthy tones with pops of

wooden garage. Warm,

black and red,brightnatural

Chain-of-thought

Adapter n:

Realistic style with attention

sunlight casting shadows.

finish it stepby step

Below is a text rewriting task

### Input: A black ...

guidedgenerationmodel

Specific type of aligner for a text-

freedom,motorcycleculture,

to detailand texture.

Keywords:Adventure,

ndno###

outdoor lifestyle.

3.1 Overview 151

Text-to-image diffusion models serve as an illustrative example for showcasing the functionality of 152 PromptCoT. However, it is important to note that the same methodology can be extended and applied 153 to other diffusion-based generative models, including text-to-video and various other domains. In 154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

the context of training text-to-image diffusion-based models, which involve image-text pairs and employ an iterative denoising process to reconstruct images based on corresponding prompts, our hypothesis posits that prompts aligned with high-quality images within the training set are more inclined to yield visually superior outputs. We randomly select 5 sets of 50 prompts corresponding to images with varying levels of quality from the Stable Diffusion training set, LAION [35], for image generation. The aesthetic score, an image quality metric introduced by [31], is used to represent the quality of individual images. As shown in Table 1, the generation performance is highly related to the prompts corresponding to the original image quality. For convenience, we refer to them as ' high-quality prompts '. In the following sections, we explain the key components of PromptCoT,

which is a prompt booster that can align input prompts with high-quality prompts in the training set, and in turn, improve generation performance.

3.2 Aligning Prompt Distribution with LLM

LLMs are extremely powerful tools that are capable of generating human-like language and completing tasks such as translation, summarization, question answering, etc. They are trained on massive amounts of text data and can learn from unstructured data to generalize to new tasks and domains. LLMs can also be fine-tuned on specific tasks with relatively small amounts of task-specific data, making them highly versatile. In this paper, we leverage this ability to align the distribution of high-quality prompts via fine-tuning a popular LLM LLaMA [40], on text continuation and revision tasks. To fine-tune LLaMA on text continuation, we use an instruction tuning template that includes incomplete text descriptions and a goal to provide a compelling continuation. The instruction tuning template is shown in Figure 3. We feed truncated text prompts placed in the input field to the LLM, supervised by the complete prompts. This enables the LLM to generate continuations containing more details.

Belowisatextcontinuationtask.

Youwillbegiven an incompletetextdescription.

Yourgoal istoprovideacompellingcontinuationof it,whilemaintaining the

distinctivestylenamed!LAlON-6plus-style!.

###Input:{input}

###Output:

Original Prompt=>A group of horses are grazing in the field.

Continuation=>A groupof horsesaregrazing inthefield.Alonetreestandsin

thecenterofthefield.Stormcloudsareenteringfromtheleftofthepicture.

For text revision, we train the LLM to map human-like input texts to high-quality prompts. However, 178 acquiring a large amount of human-written input text can be costly. Therefore, we leverage image 179 captions from BLIP as a low-cost source of "human-like" input texts. The details of collecting 180 and filtering data pairs are described in the later section. For training, we construct the instruction 181 tuning template in Figure 4. The training pipeline is similar to continuation, but with the input being 182 human-like prompts. As a result, we obtain a booster capable of performing revision tasks. 183

184

185

186

187

3.3 Enhancement with CoT

Instruction tuning enables the LLM to add details and align text distribution, however, it tends to generate extraneous information that degrades performance. As such, we introduce the Chain-ofThought (CoT) mechanism in the pipeline to address this issue. We set up five steps to make the

Belowisatextrevisiontask.

Youwillbegivenatextdescription.

Tryyourbest torewrite it inadistinctivestylenamed!LAlON-6plus-style!.

###Input:{input}

:ndno###

OriginalPrompt=>Peoplewalking towards aparked bus on the street

Revision=> In this picture taken onJanuary 6, 2016 a man walks past a bus snowfallinMoscow.(PhotobyKirillKudryavtsev/AFPPhoto)

LLM yield the expected production: (i) Extract key information from the original prompt, such as 188 visual medium and main elements, (ii) Leverage the text-continuation model to append reasonable 189 details, (iii) Extract additional concepts (for example, the color scheme) from the extended prompt 190 and emphasize crucial concepts, (iv) With improved key information and crucial concepts, the LLM 191 can generate a fluent prompt, remaining to be aligned, (v) Leverage the text-revision model to align 192 prompts to the specific distribution. This mechanism extracts and amalgamates crucial information 193 from the aligned continuation and revision, enabling reasonable inferences based on the contextual 194 cues. As a result, a more comprehensive and nuanced final output is produced. 195

196

197

198

199

200

201

202

203

204

205

206

3.4 Multi-task Adaptation

As the training set of different generative models can vary greatly, one approach to adapt to these new datasets is to fine-tune the entire LLM on the task-specific dataset. However, LLMs are typically models with billions of parameters, and allocating a dedicated LLM to each individual model proves impractical due to computational constraints. Moreover, there are plenty of text-to-image generative models trained on different datasets, and a single LLM cannot cover a diverse distribution of these datasets. As an alternative, we integrate adapters that facilitate dataset-specific adaptation, leveraging a shared pre-trained LLM as the foundation for this process. Adapters are lightweight modules that can be independently fine-tuned and subsequently added to the base model. Keeping adapters instead of the whole model significantly reduces memory usage, while enabling the adaptation of the LLM to different datasets.

Finetuning Datasets

alpaca

Text continuation

CoT

Aesthetic score >6.0

CLIP similarity >0.33

Summarization

Extraction

Text length >150

Separate to 0.3~0.7

Generation

400kfor t_continue aligner,47k forCoT aligner

Extension

Identification

Supplementary

Rewriting

Text revision

Combination

Examplification

Aesthetic score >6.0

CLIP similarity >0.33

Imitation

BLIP

CLIP-interrogator

text similarity >0.4

text similarity >0.5

52k for CoT aligner

92k for t2t_blip aligner

26k for CoT aligner

85k for t2t_inter aligner

26k for CoT aligner

52k for CoT aligner

·Aesthetic score: a scorer used by LAlON officially to evaluate whether human like this image

CLIP similarity: a score to semantically evaluate a description is aligned with image

Textlength:acountofthenumberofcharacters

Text similarity: a score to evaluate how semantically similar between two sentences

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

3.5 Dataset Preparation

We build three types of datasets: text-continuation, text-revision, and text-CoT.

Text-continuation dataset. To create this dataset, we filter high-quality prompts from the training data of existing generative models, using criteria such as high CLIP similarity and proper length. In the case of the LAION dataset, we also consider aesthetic scores to ensure a higher quality of prompts. Once high-quality prompts are identified, we truncate a portion of the text, with the remaining front part assigned as input data. The LLM is then trained to generate the missing information and complete the text. This process enables the LLM to learn how to effectively continue text prompts in a manner that is consistent with the style and context of the original text.

Text-revision dataset. The dataset consists of human-like texts and corresponding high-quality prompts which are described in the text-continuation dataset. To acquire human-like prompts, we leverage BLIP and CLIP-interrogator for image captioning. Furthermore, we calculate the text distance with the text encoder of CLIP, ensuring a score greater than 0.4 to guarantee semantic relevance between the two prompts.

Text-CoT dataset. We use GPT-3.5-Turbo to build a task-specific dataset. Initially, we design a step-by-step interaction with GPT-3.5-Turbo to extract and guide the prompt booster to finish the alignment task, due to the fact that CoT is still difficult for alpaca with a simple finetuning on datasets above. Following the alpaca's thought, 52k pairs are all generated from gpt-3.5-turbo.

4 Experimental Results

In this section, we first introduce the details on the datasets, pre-trained models, and the training hyperparameters used for all our experiments in Section 4.1. Then we demonstrate the results of applying PromptCoT to text-to-image and text-to-video pre-trained generative models in Section 4.2 and Section 4.3 respectively.

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

(j)

(k)

(l)

(m)

(n)

万

4.1 Setup

Dataset. For training, we build Text-revision and Text-continuation dataset from LAIONaes6plus [35], and Text-CoT dataset with the help of GPT-3.5-turbo. LAION-aes6plus is the subset of LAION, containing 12M image-text pairs with predicted aesthetics scores of 6 or higher. As a supplement, we also train with Text-revision, Text-continuation, and Text-CoT datasets from the WebVid-10M dataset [3] for video generation. For evaluation, we conduct experiments on COCO [18] validation set and MSR-VTT [45] for FID, FVD, aesthetic score, CLIP score, and PickScore.

Models. The pre-trained LLaMA-7B is used as the base model and we employ the adapter design outlined in [48] to facilitate multi-task adaptation. Two versions of Stable Diffusion [31], v1.4 and v2.1, are used for image generation. MagicVideo [50] is used for video generation.

Implementation Details. We finetune the LLaMA following alpaca's [39] strategy and instruction pattern, which has been verified powerful for text generation tasks. We validate the viability of

242

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

our two initial ideas by finetuning three task-specific LLaMA for prompt refining works shown in experiments 2. One is trained on the self-constructed text-continuation dataset while the other two are trained on two types of text-revision dataset. While combining such basic methods by CoT, we include a dataset from alpaca, a subset of the text-continuation dataset, and the text-revision dataset with higher text similarity and the CoT dataset as a whole. We evaluate our alignment work on three diffusion models and on different parameters. Furthermore, we evaluate the portability of promptCoT through an adapter by comparing its performance with the fully-finetuned model.

4.2 Text-to-image Evaluation

The COCO [18] validation set is the standard benchmark for evaluating text-to-image models. The key automated performance metrics used are FID to measure image fidelity, CLIP score, PickScore to measure image-text alignment, aesthetic score [22] to predict the aesthetic quality, and Inception Score (IS) to evaluate the diversity. We utilize two versions of Stable Diffusion for image generation with prompts from COCO and our PromptCoT. Table 2 presents the evaluation results for each metric with different single-function boosters including t-continue, t2t-blip, and t2t-inter, as well as a baseline. The results show that incorporating the alignment method proposed in our paper consistently improved the generated image quality across all metrics compared to the baseline. Among the single-function boosters, the t2t-blip booster demonstrates the best performance, as it is able to achieve alignment to a greater extent. For example, it transfers 'Boxes of fruit displayed at an open-air market' to 'A view of stalls selling fruit at the Harare International Market in Harare, Zimbabwe' by rephrasing

261

262

263

264

265

266

267

268

269

270

271

272

273

274

275

276

277

278

279

280

281

282

283

284

285

286

287

288

289

290

291

292

the expression and adding reasonable details. In contrast, the t2t-inter booster, which has a similar function to t2t-blip, shows inferior performance, although it still outperforms the baseline. This could be due to the CLIP-interrogator used to create the text-revision dataset introducing irrelevant entities. Furthermore, we test with different factors of classifier-free guidance to prove the generality of our PromptCoT. Varying the scale of classifier-free guidance results in consistent performance.

4.3 Text-to-video Evaluation

In addition, we experiment with the text-to-video evaluation task to demonstrate the effectiveness of our approach. We employ two single-function boosters, t-continue, and t2t-blip on the WebVid-10M dataset [3]. For t2t-blip, we uniformly sample the video and randomly select five frames, which serve as input for the blip model and be used to generate the revision result. Then, we finetune the LLaMA model following alpaca's [39] strategy and build prompts from MSR-VTT with the fine-tuned model. We use MagicVideo [50] as the base model to test the effectiveness of our prompts. The results are shown in Table 5. The results indicate that the boosters are effective in enhancing the quality of the generated videos compared to the baseline, at least they "do no harm". Among the boosters, the booster better aligns the prompts and achieves the best performance overall. For cot_d, we generate 21k data with the help of GPT-3.5-turbo. Similar to text, we utilize a chain of five questions to generate the expected production, but with subtle differences to encourage GPT-3.5-turbo to generate more video-related features, e.g., movement. Similar to text generation, we adopt a chain of five questions to generate the expected production for video prompts. However, there are subtle differences in the question prompts to encourage GPT-3.5-turbo to incorporate more video-related features, such as movement, into its generated content. For example, "a large passenger jet flying in the sky at sunset" can be refined to "Boeing 747 flying across a vibrant sunset backdrop in a captivating, cinematic 4K video. Slowly gaining altitude with wings tilting slightly, this footage captures the plane's majesty". The scores of cot_d will be included in the supplementary material.

5 Conclusion

In this paper, we present PromptCoT, an innovative system designed to autonomously enhance the quality of prompts used in diffusion-based generative models, which are critical for high-fidelity visual content generation. PromptCoT leverages pre-trained Large Language Models (LLMs) and a unique Chain-of-Thought (CoT) mechanism to refine prompts, thereby improving the alignment between the original and refined prompts. To balance computational efficiency, we employ adapters to allow for efficient adaptation to new datasets or models. Our evaluations demonstrate that PromptCoT can achieve superior performance compared to the baselines.

293

294

295

296

297

298

299

300

301

302

303

304

305

306

307

308

309

310

311

312

313

314

315

316

317

318

319

320

321

322

323

324

325

326

327

328

329

330

331

332

333

334

335

336

337

338

339

340

341

342

References

Armen Aghajanyan, Luke Zettlemoyer, and Sonal Gupta. Intrinsic dimensionality explains the effectiveness of language model fine-tuning, 2020.

Alex Andonian, Quentin Anthony, Stella Biderman, Sid Black, Preetham Gali, Leo Gao, Eric Hallahan, Josh Levy-Kramer, Connor Leahy, Lucas Nestler, Kip Parker, Michael Pieler, Shivanshu Purohit, Tri Songz, Wang Phil, and Samuel Weinbach. GPT-NeoX: Large Scale Autoregressive Language Modeling in PyTorch, 8 2021.

Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisserman. Frozen in time: A joint video and image encoder for end-to-end retrieval, 2022.

Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, T. J. Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeff Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. ArXiv , abs/2005.14165, 2020.

Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. 2020.

Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems, 2021.

Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems , 34, 2021.

Ning Ding, Yujia Qin, Guang Yang, Fu Wei, Zonghan Yang, Yusheng Su, Shengding Hu, Yulin Chen, Chi-Min Chan, Weize Chen, Jing Yi, Weilin Zhao, Xiaozhi Wang, Zhiyuan Liu, Haitao Zheng, Jianfei Chen, Yang Liu, Jie Tang, Juan Li, and Maosong Sun. Delta tuning: A comprehensive study of parameter efficient methods for pre-trained language models. ArXiv , abs/2203.06904, 2022.

Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. Glm: General language model pretraining with autoregressive blank infilling. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 320-335, 2022.

Tianyu Gao, Adam Fisch, and Danqi Chen. Making pre-trained language models better few-shot learners. ArXiv , abs/2012.15723, 2021.

Xu Han, Weilin Zhao, Ning Ding, Zhiyuan Liu, and Maosong Sun. Ptr: Prompt tuning with rules for text classification. ArXiv , abs/2105.11259, 2021.

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems , 33:6840-6851, 2020.

Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J. Fleet. Video diffusion models, 2022.

Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for NLP. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning , volume 97 of Proceedings of Machine Learning Research , pages 2790-2799. PMLR, 09-15 Jun 2019.

Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models, 2021.

Zhiting Hu, Zichao Yang, Xiaodan Liang, Ruslan Salakhutdinov, and Eric P. Xing. Toward controlled generation of text. In International Conference on Machine Learning , 2017.

Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning, 343 2021. 344

Tsung-Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, 345 and C. Lawrence Zitnick. Microsoft coco: Common objects in context. In European Conference on 346 Computer Vision , 2014. 347

348

349

Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blunsom. Program induction by rationale generation: Learning to solve and explain algebraic word problems. arXiv preprint arXiv:1705.04146 , 2017.

350

351

352

353

354

355

356

357

358

359

360

361

362

363

364

365

366

367

368

369

370

371

372

373

374

375

376

377

378

379

380

381

382

383

384

385

386

387

388

389

390

391

392

393

Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys , 55:1 - 35, 2021.

Rabeeh Karimi Mahabadi, James Henderson, and Sebastian Ruder. Compacter: Efficient low-rank hypercomplex adapter layers, 2021.

Naila Murray, Luca Marchesotti, and Florent Perronnin. Ava: A large-scale database for aesthetic visual analysis. In 2012 IEEE conference on computer vision and pattern recognition , pages 2408-2415. IEEE, 2012.

Sharan Narang, Colin Raffel, Katherine Lee, Adam Roberts, Noah Fiedel, and Karishma Malkan. Wt5?!

training text-to-text models to explain their predictions, 2020.

Jonas Pfeiffer, Andreas Rücklé, Clifton Poth, Aishwarya Kamath, Ivan Vuli´ c, Sebastian Ruder, Kyunghyun Cho, and Iryna Gurevych. Adapterhub: A framework for adapting transformers, 2020.

Guanghui Qin and Jas' Eisner. Learning how to ask: Querying lms with mixtures of soft prompts. ArXiv , abs/2104.06599, 2021.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. 2021.

Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019.

Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125 , 2022.

Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Dall·e 2: Exploring cross-modal transformers for image generation. OpenAI Blog , 2021.

Robin Rombach, A. Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 10674-10685, 2021.

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models, 2022.

Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S. Sara Mahdavi, Rapha Gontijo Lopes, Tim Salimans, Jonathan Ho, David J Fleet, and Mohammad Norouzi. Photorealistic text-to-image diffusion models with deep language understanding, 2022.

Timo Schick, Helmut Schmid, and Hinrich Schütze. Automatically identifying words that can serve as labels for few-shot text classification. In International Conference on Computational Linguistics , 2020.

Timo Schick and Hinrich Schütze. Exploiting cloze-questions for few-shot text classification and natural language inference. In Conference of the European Chapter of the Association for Computational Linguistics , 2020.

Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. arXiv preprint arXiv:2210.08402 , 2022.

Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502 , 2020.

Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in Neural Information Processing Systems , 32, 2019.

Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben 394 Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint 395 arXiv:2011.13456 , 2020. 396

397

398

399

Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-following llama model. https://github. com/tatsu-lab/stanford_alpaca , 2023.

400

401

402

403

404

405

406

407

408

409

410

411

412

413

414

415

416

417

418

419

420

421

422

423

424

425

426

427

428

429

430

431

432

433

434

435

436

437

438

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models, 2023.

Leandro von Werra, Alex Havrilla, Max reciprocated, Jonathan Tow, Aman cat state, Duy V. Phung, Louis Castricato, Shahbuland Matiana, Alan, Ayush Thakur, Alexey Bukhtiyarov, aaronrmm, Fabrizio Milo, Daniel, Daniel King, Dong Shin, Ethan Kim, Justin Wei, Manuel Romero, Nicky Pochinkov, Omar Sanseviero, Reshinth Adithyan, Sherman Siu, Thomas Simonini, Vladimir Blagojevic, Xu Song, Zack Witten, alexandremuzio, and crumb. CarperAI/trlx: v0.6.0: LLaMa (Alpaca), Benchmark Util, T5 ILQL, Tests, March 2023.

Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Huai hsin Chi, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. ArXiv , abs/2203.11171, 2022.

Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. Finetuned language models are zero-shot learners. ArXiv , abs/2109.01652, 2021.

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models, 2023.

Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: A large video description dataset for bridging video and language. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 5288-5296, 2016.

Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, Ben Hutchinson, Wei Han, Zarana Parekh, Xin Li, Han Zhang, Jason Baldridge, and Yonghui Wu. Scaling autoregressive models for content-rich text-to-image generation, 2022.

Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, et al. Glm-130b: An open bilingual pre-trained model. arXiv preprint arXiv:2210.02414 , 2022.

Renrui Zhang, Jiaming Han, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, Peng Gao, and Yu Qiao. Llama-adapter: Efficient fine-tuning of language models with zero-init attention. arXiv preprint arXiv:2303.16199 , 2023.

Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. Opt: Open pre-trained transformer language models, 2022.

Daquan Zhou, Weimin Wang, Hanshu Yan, Weiwei Lv, Yizhe Zhu, and Jiashi Feng. Magicvideo: Efficient video generation with latent diffusion models. arXiv preprint arXiv:2211.11018 , 2022.

Daquan Zhou, Weimin Wang, Hanshu Yan, Weiwei Lv, Yizhe Zhu, and Jiashi Feng. Magicvideo: Efficient video generation with latent diffusion models, 2023.

Denny Zhou, Nathanael Scharli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Olivier Bousquet, Quoc Le, and Ed Huai hsin Chi. Least-to-most prompting enables complex reasoning in large language models. ArXiv , abs/2205.10625, 2022.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces PromptCoT, an enhancer that automatically refines text prompts for diffusion-based generative models, improving their capability to produce high-quality visual content. The system is based on the idea that prompts that resemble high-quality image descriptions from the training set lead to better generation performance. Pre-trained Large Language Models (LLM) are fine-tuned using a dataset of such high-quality descriptions, allowing them to generate improved prompts. To mitigate the tendency of LLMs to generate irrelevant information, contamination, transfer, and even a Chain-of-Thought (CoT) mechanism are used to improve alignment between original and refined prompts. Additionally, to maintain computational efficiency, the system employs adapters for dataset-specific adaptations, leveraging a shared pre-trained LLM. When tested on popular latent diffusion models for image and video generation, PromptCoT showed significant performance improvements.\n\nSTRENGTHS:\n1) This paper shows an insight from the observation that prompts resembling high-quality image descriptions from the training set lead to better generation performance.  It makes sense from my perspective.  Current text-guided image generation is still with limited generalization ability.  The points near the training support samples are transferred better. \n2) The idea to utilize LLM to adapt the original prompt to the one that is more aligned with training samples makes sense. It leverages the ability of LLMs to align the distributions. \n3) Three training methods are proposed to implement the ideas including the continuation, revision, and CoT.  \n4) This paper builds the corresponding datasets to help the fine-tuning, which can benefit the following research.\n5) The method using the GPT-3 to build datasets is smart.\n\nWEAKNESSES:\n1) The finetuning of LLaMa is time-consuming.  Could it be replaced by prompt learning or LORA? \n2) Please add a discussion (e.g. in related work) with ''Visual Chain-of-Thought Diffusion Models'', though these two methods are clearly different. \n3) If the datasets with neural captions can be built.  How about adding these sentences to the training set of the diffusion model?\n\nQUESTIONS:\nPlease refer to the weaknesses part.\n\nLIMITATIONS:\nIt can be found in the F part of the appendix.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-06-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this manuscript, the authors proposed a simple yet effective framework to improve the generation quality of pretrained generative models. Generally, to align the prompt distribution with large language models, the authors present three individual solutions to align and enhance the original textual inputs, i.e., providing a compelling text continuation with given initial inputs, revising the initial inputs, and using chain-of-thought to enhance the initial inputs by 5 steps. Moreover, the authors also introduce multi-task adaptation to improve the quality of generated textual desctiptions for each dataset. The experimental results demonstrate that the generative models with PromptCoT can achieve better qualitative performance than the baseline method.\n\nSTRENGTHS:\n1. The proposed PromptCoT is easy to follow.\n\n2. The qualitative results is promising.\n\nWEAKNESSES:\n1. Introducing prompting technique to improve the quality of the textual inputs for generative models lacks novelty, since intuitively, providing more detailed information indeed improves the generation quality of the generative models. The authors could clarify the novelty, e.g., how the prompting technique or the CoT technique different from other works, or why the generated text is better than human-refined counterparts.\n\n2. As shown in Figure 4, the proposed text revision technique may generate some uncontrollable additional information (e.g., time, weather and place). I think the prompted results may be influenced by the inherent bias of the language model or the prompting dataset. The authors could provide some analysis to show that the enhanced textual inputs are unbiased with given CoT technique, or show that the enhanced textual inputs are controllable.\n\nQUESTIONS:\nSee the weakness for detail. If the authors could address my conerns 1 and 2, I will raise my rating.\n\nLIMITATIONS:\nThe authors have stated the limitation of the proposed PromptCoT, i.e., relying on the capability of the generative models and the quality of inital textual inputs. These limitations can be seen as the future direction of this work.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-02"}

comments: SUMMARY:
This paper aims to improve the images generated by the off-the-shelf diffusion model, such as Stable Diffusion. This is done by fine-tuning a large language model (LLaMA) using text continuation on more high-quality prompts, which are collected by hand-crafted rules on, for example high CLIP similarity and text length. Furthermore, off-the-shelf image caption models (like BLIP) are also used to curate high-quality prompts. While the proposed method seems effective, the lack of technical novelty is a concern of this work.

STRENGTHS:
1.	The overall presentation is good, and the paper is simple to follow.

2.	The visualization is very interesting. For example, Figure 1 is very impressive.

3.	The overall pipeline is well encapsulated in Figure2-4. It is simple for the reader to understand the high-level idea of the work by looking at the figures.

WEAKNESSES:
Major 
1.	Novelty is a big concern in this work and there is no technical contribution in this work. The author basically uses many off-the-shelf techniques to augment/modify the input text.  Text-continuation is also commonly used for pretraining large LLM. The adaptation techniques in section 3.4 are also not novel and the author simply leverages the existing techniques. In general, it is unclear what is the technical contribution of this work. 

2.	L258 Is there any reason why t2t-blip booster demonstrates the best performance? Any theoretical reason behind this?

Minor
1.	L224 gpt to “GPT”

QUESTIONS:
Overall, there is no big drawback in this work. However, there is not much technical contribution in this work as well. This work might be useful for industrial applications, but its publication might not benefit the community.

LIMITATIONS:
Limitation is discussed in appendix
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

comments: SUMMARY:
The main motivation of this paper is to better align prompts to the textual information of high-quality images within the training set. The authors propose datasets, instruction templates and use CoT to finetune LLM to achieve this goal. Adapters are also use adapters to facilitate dataset-specific adaptation.

STRENGTHS:
Application of LLMs to image generation for better text alignment is reasonable. The authors prepared datasets and conducted curated pipeline to achieve this goal. The presentation is clear.

WEAKNESSES:
1. the observation that prompts aligned with high-quality images withing the training set are more inclined to yield visually superior outputs needs more demonstration. The authors verified this on  LAION dataset. But it does not mean this is always the truth on all datasets. Whether it works on Flickr, Pinterest images?
2. The application of LLMs to refine texts is widely used now. Prompt engineering, Cot, adapters are also widely explored. Only the Application of these techniques seems not novel enough. 
3. The pipeline may needs more explanation on why we need text continuation first then text revision. 
4. the experiments are quite confusing. what are t2t-inter, cot_d, cot? the scores in Table 2 shows the effectiveness of each booster. How do they work together?  Which booster works the best  according to these experiments. what are Table 3 and Table 4 used for? It's really hard to summarize a conclusion that can match the design of the whole method.

QUESTIONS:
questions are listed in weakness.

LIMITATIONS:
limitations are presented in the paper.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-06