Optimizing Prompts for Text-to-Image Generation
Authors: Yaru Hao
Zewen Chi
Li Dong
Furu Wei
Venue: NeurIPS 2023 spotlight (2023)
Keywords: prompt adaptation
automatic prompt engineering
text-to-image generation
TL;DR: We propose to automatically optimize prompts for text-to-image models so that the user input and model-preferred prompts can be well aligned.

Abstract: Well-designed prompts can guide text-to-image models to generate amazing images. However, the performant prompts are often model-specific and misaligned with user input. Instead of laborious human engineering, we propose prompt adaptation, a general framework that automatically adapts original user input to model-preferred prompts. Specifically, we first perform supervised fine-tuning with a pretrained language model on a small collection of manually engineered prompts. Then we use reinforcement learning to explore better prompts. We define a reward function that encourages the policy to generate more aesthetically pleasing images while preserving the original user intentions. Experimental results on Stable Diffusion show that our method outperforms manual prompt engineering in terms of both automatic metrics and human preference ratings. Moreover, reinforcement learning further boosts performance, especially on out-of-domain prompts.

Optimizing Prompts for Text-to-Image Generation
Yaru Hao; Zewen Chi, Li Dong, Furu Wei
Microsoft Research
Abstract
Well-designed prompts can guide text-to- -to-imaee models to generate amazing images. However, the performant prompts are often model-specific and misaligned with user input. Instead of laborious human engineering, we propose prompt adaptation, a general framework that automatically adapts original user input to model-preferred prompts. Specifically, we first perform supervised fine-tuning with a pretrained language model on small collection of manually engineered prompts. Then we use reinforcement learning to explore better prompts. We define a reward function that encourages the policy generate more aesthetically pleasing images while preserving the original user intentions. Experimental results on Stable Diffusion show that method outperforms manual prompt engineering in terms of both automatic metrics and human preference ratings. Moreover, reinforcement learning further boosts performance, especially out-of-domain prompts. The pretrained checkpoints are available at /ttph:tt///aa //ap::://aaaa.. http://aaaaaaaaaappaa/// wwtpammpp/poottttttmmmmmmmmm The demo can be found https://aakk ma/prooppisst-demo
Introduction
Generative foundation models can be prompted to follow user instructions, including language models [Brown et al.) 2020 Chowdhery et al. 2022 |mith et al.] 2022]. and text-to-imaae mod- els [Ramesh et al. 2021 2022 Saharia et al. 2022 Rombach et al. 2022]. has been recognized that prompt design plays an essential role in the generation quality. We need to adjust the prompt to make the model better understand our intentions and produce higher-quality results [Reynolds] and McDonell 2021 Zhou et al.) 2022b]. The problem is severe in text-to-imen -iaag models because the capacity of their text encoders, such as CLIP text encoder [Radford et al., [021]] in Stable Diffu- sion [Rombach et al. 2022 is relatively small. Empirical observations also confirm that common user input is often insufficient to produce aesthetically pleasing images with current models.
Prior efforts implement manual prompt engineering towards specific text-to-iman image models [Liu and Chilton 202T, Oppenlaender 2022 Parsons [022]. typically adding some modifiers the original input. However, it is laborious and sometimes infeasible to conduct manual prompt engineering. Besides, the manually engineered prompts often cannot be transferred between various model versions. Therefore, it is necessary to find systematic way to automatically align user intentions and various moeel-preferred prompts.
In this work, we propose a prompt adaptation framework for automatic prompt engineering via reinforcement learning. Specifically, we first perform supervised fine-tuning with a pretrained language model (e.g., GPT) on small collection of manually engineered prompts. The finetuned model used initialize the prompt policy network for reinforcement learning. Next, the model is trained by exploring optimized prompts of user inputs, where diverse beam search [Vijayakumar al. 2016) used to ensure generation quality and diversity. The training objective maximize
Equal contribution.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Stage 1: Supervised Fine-Tuning
Human Input
Optimized Prompt
LM as Promptist
Stage 2: Reinforcement Learning
Train with PPO
Text-to-lmage
Optimized Prompt
as Promptist
Input
Aesthetic
Score
Reward
Text-to-lmage
Relevance
Score
Figure 1: Overview of PROMPTIST training: (I) supervised fine-tuning (SFT) on manually engineered prompts; (2) reinforcement learning (RL) to increase the rewards of generated images after prompt optimization.
the reward, which defined combination of relevance scores and aesthetic scores of generated images. The relevance score reflects how much the original user intentions are retained after prompt adaptation. The aesthetic score indicates what degree the generated images aesthetically pleasing.
We conduct experiments with the publicly available Stable Diffusion models [Rombach et al., 2022] We evaluate our method using both the automatic reward metric and human preference ratings. Experimental results show that the optimized prompts outperform human-engineered ones and the original inputs. Human preference ratings also show consistent improvements across in-domain and out-of-domain prompts. Moreover, we find that reinforcement learning is more favorable than supervised fine-tumin fine-tuning, especially oft-oo-domain models can serve as prompt interface that optimizes user input into model-preferred prompts.
inputs. Overall, we show that language
Our contributions are as follows:
We propose general prompt optimization framework that adapts user input to model- preferred prompts.
We collect user queries and conduct extensive experiments on text-to-image generation.
Experimental results show that method outperforms manual prompt engineering in terms of both automatic metrics and human preference ratings.
2 Methods
The goal of our prompt adaptation framework is automatically perform prompt engineering. Given user input of the text-to-image generator, our model learns to generate model-preferred prompts that obtain better output images while preserving their original intentions. Figure [ presents the overview of our method. The prompt optimization model is named PROMPTIST, which is built upon a pretrained language model, such as GPT [Brown et al. 2020). We first collect a set of human-engineered examples and use them to conduct supervised fine-tuning (Section 2.1). Next, we perform reinforcement learning (Section 2.3 maximize the target reward (Section 2.2 which
improves both relevance and quality of generated images.

---

2.1 Supervised fine-tuning
Initialized with a pretrained generative language model, the policy model is first finetuned on a set of prompt pairs before reinforcement learning. A parallel prompt corpus D = {( y)} contains prompt
pairs of original user imputs æ and manually engineered examples y. The training objective is to maximize the log-likelihood with teacher forcing:
SEFT = -B y)-1 log (ypp(yy))
where the finetuned weights are used to initialize the policy network in reinforcement learning.
Collect human demonstrations We collect human-engineered prompts from Lexica Most prompts are composed of two parts, i.e., main content that describes the user's intention, and some modifiers that customize the art style, such as artist names, and popular elements. We use the crawled human-enge eennneeeg prompts as targets. In order to obtain parallel data, we use three methods to construct their source inputs. First, we extract the main contents by trimming the modifiers and regard them as original user inputs. Second, we randomly remove or shuffle some modifiers and use the remaining texts as source inputs. Third, we use the OpenAI API text -waviiiiiiio--- to rephrase the main contents and the human- -aggannnniieeeeed prompts, respectively. We find that the template " [[Input] Rephrase:" " works well in practice and translates input to more user-ffiendly version. As shown in Table given a target prompt "dieselpunk blue wolf with fuzzy tail, concept art, dramatic, fantasy, , pixiv", there are four source prompts collected.
Table I: An example of a human-engineered prompt and four types of our constructed source prompts.
Human-engineered target prompt
dieselpunk blue wolf with fuzzy tail, concept art, dramatic, fantasy, pixiv
Main content
dieselpunk blue wolf with fuzzy tail
Main content with random modifiers
dieselpunk blue wolf with fuzzy tail, dramatic
Rephrasing of main content
A blue wolf with a fuzzy tail that looks like belongs in a dieselpunk setting.
Rephrasing of target prompt
This is dicselpunk-style blue wolf with fuzzy tail. looks like it could be from fantasy or dramatic piece of artwork.
2.2 Reward definition
We measure the quality of optimized prompts from two aspects, namely relevance and aesthetics. The goal motivates us to define the reward function R(-)) from the above two perspectives.
First, we measure whether the generated images are relevant to the original input prompt after prompt adaptation. To be specific, we first sample images by the text-to-image model conditioned on the optimized prompt, respectively. Then, we compute CLIP [Radford et al. 2021] similarity scores to measure how relevant the generated images and the original input prompts are. The resulting relevance score is defined as:
fee(( = Ei (z, (2,2,,))) 1y))
fre (n)) =min (20 * ICLI 1n) - 5)
where in ~ G(y) means sampling images iy from the text-to-image model G with as input prompt, and (CLr((,)) -) stands for the CLIP similarity function. Notice that we always compute the similarity between the generated images and the original input prompt, which ensures the relevance score reflects the user preferences. We determine the specific form of the relevance score according to the approximate range of the clip score. Experiments show that this form works well in reinforcement learning. If the relevance score relatively reasonable (larger than 0.28), we encourage the model to generate more aesthetically pleasing images.
Second, employ the aesthetic predictof quantify aesthetic preferences. The predictor builds linear estimator on top of a frozen CLIP model, which is trained by human ratings in the Aesthetic
hattpe://exxcaa.rr
3

---

Visual Analysis Murray et al. 2012 dataset. The aesthetic score is defined as:
( y) = (i
(ines(ty) (tyn)
Jacs (1x)
where (-es(-) denotes the aesthetic predictor, and by. &u are the images generated by the prompts y and æ, respectively. Notice that both 9CLIP()) and Jaes (-) require the CLIP model, so we can share the CLIP forward pass during reward computation.
Finally, we define the overall reward by combining the above scores with an additional KL penalty, which is between the policy model Th and the supervised finetuned model TSFT with coefficient
R(x, y)
facs (æ, y) + fiel y)
Te (2) (z)
(5)
log
TSFT x
The KL term is added to mitigate the overoptimization issue [uyyng et al. 2022
2.3 Reinforcement learning
Starting from the supervised fine-tuning, we further finetune model with reinforcement learning. We employ proximal policy optimization (PPO) (Schulman et al.] 2017], which is empirically data- efficient and of reliable performance. As text generation problem, prompt optimization can be viewed as a Markov decision process (MDP) Ar, fa,?) with a finite state space S, action space r.
A, reward function r, ptate-transitonn probability function fst. and discount term 7. In an episode of prompt adaptation, the initial state E S is the input prompt with n tokens æ = (I1,... ...... Tn) where each token is from a finite vocabulary V. At t-th time step, the agent selects an action Ye E V according to the current policy model Ye π (z, yerr). With deterministic state transition, the next state is (xr, (<(+1) = (31 ........ In .1..... Me). The episode ends when the agent selects an und-of-sentence action. The goal of the agent is to maximize the accumulated expected reward Eu.y E, (Ix y<+) = R y).
Let Ta denote the policy model to be trained, we maximize the accumulated expected reward over a training set D (
J = yo
yoara [R(x, y)
(6)
We implement both the policy model and the value function model as generative language models, with the language modeling head and the regression head, respectively. The parameters of the two models are initialized from the supervised finetuned policy model TSFT and are optimized during reinforcement learning. The supervised finetuned model TSFT and the score function model are frozen during training. Besides, we employ the clipped probability ratios [Schulman et al. 2017 to avoid large policy updates.
3 Experiments
We conduct experiments on public text-to-image model Stable Diffusion v and v1.5.. We use the DPM solver [Lu et al. 2022 to accelerate image sampling and set the denoising steps to 20.
3.1 Data collection
For supervised fine-tuning, we collect 90k target prompts from Lexica website and construct four types of source prompts as described in Section 2.1 obtaining 360k paired data in total. At the reinforcement learning stage, we only require source prompts and the policy can explore better rephrasings itself. We use three types of data: (1) in-domain prompts from DiffusionDB [Wang et al., 2022], which is a gallery of prompts specified by real users. We use the user input (main content) for exploration and the manually engineered prompt (with modifiers) for comparison, (2) out-of-domain image captions from COCO dataset [Chen et al., 2013). (3) image labels from ImageNet-211 (Deng al. 2009], the sizes of which are 600k, 600k and 40k respectively. We empirically observe that

---

0.2
0.0
Reward
-02
-0.4
-0.6
MC
MCM
RMC
RTP
In-Domain (DiffusionDB)
Out-of-Domain
In-Domain (Lexica)
(COCO)
Input
Promptist (Ours)
Supervised Fine-Tuning
Human Engineered Prompt
Figure 2: Reward comparison of optimized prompts with other baselines in-domain and out-of- domain data. For in-domain Lexica prompts, we evaluate on four augmentations: main content (MC), main content with random modifiers (MCM), rephrasing of main content (RMC) and rephrasing of target prompt (RTP). Results indicate that the text-to-image model benefits a lot from our method.
Table Absolute reward improvements of supervised fine-tuning and reinforcement learning. is observed that RL generally outperforms the SFT-only model.
In In-Domain (Lexica)
In In-Domain (DiffusionDBB
Out-of-Domain (COCO)
MC
MCM
RMC
RTP
SFT
0.36
0.16
0.44
0.11
0.29
0.28
RL
0.47
0.17
0.63
0.25
0.36
0.48
Gain
+31%
+6%
+43%
+127%
+24%
+71%
human-engineered prompts from Lexica perform better than those from DiffusionDB we use the former in supervised fine-tuning. To improve the data diversity, we add image caption data and image label data in reinforcement learning. To avoid model bias in certain data formats, we randomize the capitalization of the first letter of each prompt and randomly add periods at the end of it.
3.2 Settings
For the policy model, we use GPT-2 Radford et al. 2019 with 117M parameters, which multi-layer Transformer /Vaswani et al. 2017 decoder pretrained with causal language modeling.
Supervised fine-tuning We finetune GPT-2 to predict the target prompt conditioned on the source prompt with teacher forcing. The input format is [Source] Rephrase: [Target]. We use a batch size of 256, a learning rate of 5e-5, and max length of 512. We finetune the model for 15k steps and choose slightly underfitting checkpoint according the validation loss which aims to avoid overfitting and provide a proper exploration space for the policy.
Reinforcement learning We train the policy with Proximal Policy Optimization (Shulman et al. 2017] PPOJ. The value and policy network are initialized from the supervised finetuned model. The parameters of the value function are separated from the policy avoid excessive competition between two objectives. To guarantee the quality and diversity of exploration, we adopt diverse beam search (Vijayakumar al. 2016 with beam size of 8 and diversity penalty of 1.0. We find that having too long rephrasings occasionally produces aesthetically pleasing but misleading results, especially for short user input like image labels. In order to prevent the model from only exploring long completions, the maximum generation length at each step is set to random value from 15 to 75 so that the policy can learn to adjust the generation length for each prompt. We randomly choose one of the returned completions after diverse beam search to update the policy. We generate three images per prompt and compute the average reward reduce variance. We train the policy for 12k episodes, four PPO epochs per batch with one minibatch each, with batch size of 256 and constant
5

---

learning rate of 5e-5. The value loss coefficient and the KL reward coefficient are kept at 2.3 and 0.2 respectively. We do not cherry-pick checkpoints and directly use the final checkpoint for evaluation. Please refer to the Appendix A for more training details and Appendix B for computational resources.
Evaluation In order to evaluate how text-to- text-to-image models benefit from the prompt adaptation. compare the reward value computed by two automatic predictors (Section 2.2p. Moreover, we use human preference ratings to demonstrate real user feedback. We adopt beam search with a beam size of and length penalty of 1.0. We evaluate method on held-out data from training distribution, including in-domain data from Lexica with four augmentations, in-domain data from DiffusionDB, and out-of-domain COCO data. Each category contains 256 prompts. In-domain data has corresponding manually engineered prompts for comparison, and the out-of-domain data is used to verify whether our method can generalize new domains. Except for the user input and manually engineered baseline, we also consider the supervised finetuned model as a baseline that can reflect the importance of reinforcement learning.
3.3 Results
Optimized prompts obtain higher reward improvements than manual engineering. We evaluate optimized prompts on held-out data by generating three images for each prompt and computing the average reward value. Figure 2 shows that the reward value can be improved regardless of the engineering method, which suggests the misalignment problem between user-friendly prompts and model-preferred prompts is serious. Compared with the strong baseline of manually engineered prompts, optimized prompts can still achieve considerable reward improvements. Furthermore, optimized prompts perform even better on rephrased versions (i.e., RMC, and RTP), and out-of- domain data. These prompts are more user-friendly but cause more significant reward drops on generation results, especially on the rephrasing of the main content. Benefiting from automatic prompt engincering, optimized prompts can align well between two different domains from users and exxtto-imagg models respectively.
We also present the evaluation results of the aesthetic score and relevance score respectively in Table We empirically found that the generated images are relevant enough to the input prompt the relevance score (CL.IP score) is around 0.26. As mentioned at Section 2.2 we design the reward function which encourages the model to generate more aesthetically pleasing images if the relevance score is good enough. On the DiffusionDB dataset, our RL method improves the SFT baseline in terms of relevance score from 0.25 to 0.26, and the human-engineered baseline also obtains a relevance score of 0.26. Moreover, the aesthetic score of our model is improved significantly over both the Haman-engineers prompts and the supervised tne-ttned model. It demonstrates that our method generates images with good relevance and much better aesthetic scores.
We provide some images generated by user input and its corresponding optimized prompt in Table 3 Each group consists of three images generated by different random seeds. We observe that images generated by user input are intuitively uninspiring while optimized prompts can not only retain the original intentions but also induce the model to produce more remarkable results. For example, generated images are crude when prompted with "A rabbit is wearing a space suit". After prompt optimization, generated images become more bright and more expressive.
Reinforcement learning can further boost the reward value. Reinforcement learning in our method is supposed perform better on out-of-domain data through explorations. To quantify its effect, we compute the ratio of reward improvements after fine-tuning and reinforcement learning. As shown in Table 2 reinforcement learning brings 1., 24%. and 71% average improvements on in-domain main content from Lexica, DiffusionDB, and out-of-domain COCO data. In-domain prompts are very similar the data we used in supervised fine-tuning, so reward improvements are relatively saturated in the first stage and improvements of reinforcement learning on them are correspondingly smaller. Oppositely, oft-oi-domain data such as COCO captions are more similar to user input and unseen during the first stage. The policy must learn to adapt better to new domains through exploration, so their improvements on these prompts are more prominent. Surprisingly, although in-domain Lexica prompts and their augmentations are not used, reinforcement learning still exhibits better generalization capability on them. The boost is remarkable on those prompts that fine-tuning cannot optimize well (43% on rephrasings of main content and 127% rephrasings of

---

Table 3: Images generated by input and optimized prompts using Stable Diffusion v1.4. Each group contains three images generated with three different random seeds. We observe that optimized prompts can generate more aesthetically pleasing images than original user input.
User Input
Optimized Prompt
A rabbit is wearing a space suit, digital Art, Greg rutkowski, Trending cinematographic artstation
rabbit is wearing a space suit
several railroad tracks with one train passing by, hyperdetailed, artstation, egsocicty, 8 k
Several railroad tracks with one train passing
by
the roof is wet from the rain, intricate, elegant, highly detailed, digital painting, artstation, con- cept art, smooth, sharp focus, illustration,
The roof is wet from the rain
Cats dancing in a space club, digital painting, artstation, concept art, soft light, hdri, smooth, sharp focus, illustration, fantasy,
Cats dancing in space club
target prompt). These results suggest that given appropriate human queries, reinforcement learning can optimize them to adapt to different domains and boost reward improvements.
To further demonstrate the effectiveness of our framework, we also present the results of our model on Stable Diffusion v1.5 in Appendix C comparisons with the heuristic baseline in Appendix and the results on different categories and lengths of prompts in Appendix E
3.4 Human evaluation
The reward function of our model is defined by two automatic metrics, aesthetic score and relevance score predicted by neural networks, which may have some discrepancies from real human feedback. Therefore, we additionally evaluate whether optimized prompts actually make humans more satisfied. We generate two images for each user input and the optimized prompt. Afterward, three held-out annotators are asked to rank the two groups of images in preference order and we compute the average preference distribution. Results are shown Table We observe that annotators generally prefer images generated by optimized prompts over their original input. Compared with manually
7

---

Table Evaluation of the aesthetic score and relevance score on DiffusionDB.
Aesthetic
Relevance
User Input
5.47
0.28
Human Engineered Prompt
5.87
0.26
Supervised Fine-tuning
6.15
0.25
PROMPTIST(OUrS)
6.26
0.26
Table 5: Human evaluation results. The different colors represent how many images generated by corresponding prompts considered more aesthetically pleasing. The orange block means that both prompts produce equally pleasing images.
Optimized User Input
Optimized vs. Manually Engineered
In-Domain
72%
13%
15%
49%
21%
3.0%
Out-of-Domain
67%
12%
21%
engineered prompts, optimized prompts yield less gain over user input. It suggests that the aesthetic score can measure the quality of generated images to some extent, would better if direct human feedback is included in the reward function.
3.5 Ablation of source prompt augmentation
As described in Section 2.1 we crawl human- remman-nggieered prompts as target prompts and use the main content without any modifiers as source prompts. To enable the supervised fine-tuned model to generalize better on unseen domains, we propose different augmentation methods that improve the diversity of source prompts. We compare the fine-tuning performance with and without the augmentation strategy and results are shown in Figure 3 We observe that fine-tuning with source prompt augmentation brings consistent improvements on both in-domain held-out data and out-of- domain data. From the results MCM, adding some random modifiers to the user input slightly obtains reward improvement but not as distinct as improvement brought by fine-tuning, indicating that we should customize modifiers for each individual prompt and automatic prompt engineering is a promising way to tackle it. Compared with other data, prompts rephrased by text-ce toxt-davin sert-davinci--00 from reinforcement learning. Overall, source prompt augmentation makes the fine-tuned model generalize better and is important in our prompt adaptation framework.
more difficult optimize the fine-tuning stage and they benefit more
Related work
Prompt engineering. Manual prompt engineering is natural way to optimize prompts. Manually designed cloze-style prompts have been used to probe knowledge from pre-trained language mod-
els [Petroni al. 2019 Dai al. 2022 In addition knowledge probing, models are also prompted
to handle NIP tasks with manually designed prefix prompts [Brown et al.) 2020 Du et al. 2021)
Recent work has explored how to write prompts to improve performance (Wei al. 2022). Despite the success of manually-cratie 202T) and can be sub-optimal (Tiang al. 2020]. In particular, when using text-to-image models. users have to carefully select and compose sentences to achieve certain visual style [Liu and Chilton 2021 Oppenlaender [0022] Parsons 2022). Thus, various methods focus on automatically searching prompts by mining Jiang et al.; 2020). paraphrasing (Haviv et al. 2021], and text generation [Gao ct al. 2021]. Besides, continuous prompt methods treat the prompts as additional continuous pa- rameters of pre-trained models and directly optimize the parameters on downstream tasks [Li and Liang 202T I'simpoukelli et al. [02T]] Zhou et al.) [022a]. However, continuous prompt methods require access to manipulating the model, and the learned prompts lack interpretability. In contrast,
prompts, designing prompts takes time and experience [Shin et al.)
8

---

0.0
Reward
-0.2
-0.4
-0.6
MC
MCM
RMC
RTP
In-Domain Oft-an-Dommin (COCO)
In-Domain (Lexica)
(DiffusionDB)
User Input
SFT w/o Source Prompt Augmentation
SFT w/ Source Prompt Augmentation
Figure 3: Reward comparison supervised fine-tuning with or without source prompt augmentation. For in-domain Lexica prompts, we evaluate on four augmentations: main content (MC), main content with random modifiers (MCM), rephrasing of main content (RMC) and rephrasing of target prompt (RTP). It is observed that source prompt augmentation in supervised fine-tuning can boost performance on both in-domain data and out-of-domain data.
our methods directly optimize prompts in text format, which can fit in black-box downstream systems such as text-to-image models.
Learning from human feedback. Our work is related to research on learning from human feedback, which has been widely studied machine learning problems. Several studies propose to continually improve dialogue systems by collecting human feedback after deployment [Hancock et al., 2019]
Shuster et al., 2020 Xu al. 2022). Besides, human feedback has also been also applied to human-in- toop methods for entity linking (Klie et al. 2020]. semantic parsing Yao et al. 2019) etc. Recent research on reinforcement learning from human feedback (RLHF) has shown promising results on machine learning problems, ranging from classical RL tasks [Christiano et al., 2017 Ibarz et 2018 to wide range of natural language processing tasks, including text summarization [Stiennon et al. 2020 Ziegler et al. 2019]. dialogue (Jaques et al. [2019]. J and general text generation tasks [Ouyang ct al. 2022 Differently, our goal is to automatically optimize prompts for text-to-image models.
Text-to-image models. Text-to-image synthesis models are typically trained to generate images conditioned on text. Text-to-image synthesis has been widely studied using GANs [Reed et al. 2016a b Tao et al. 2022]. More recently, text-to-image models are further improved with large-scale auto regressveee models Ramesh et al., 2021b Ding et al. 202T or diffusion- basood models [Rombach)
al. 2022 Gu 2022
5
Conclusion
We propose to automatically optimize prompts for text-to-image models so that the user input and prefee-peff prompts can be well aligned. We evaluate method with Stable Diffusion. Experimental results show that prompt adaptation outperforms human prompt engineering and supervised fine-tuning, in terms of automatic metrics and human evaluation. The exploration nature of reinforcement learning enables the model to go beyond teacher forcing, which improves generalization over out-of-domain examples. The proposed method is flexible align human intentions and model- favored languages. Although our experiments are conducted on text-to- text-to-image models, the framework can be easily applied other tasks for prompt adaptation. Rather than using automatic score functions as rewards, we can directly use human feedback as supervision to train a reward model [Ouyang let al. 22022. Moreover, using a larger-size language model as the prompt interface tends to improve the optimization quality.
6 Limitations
We crawl human-engineered prompts from the Lexica website as golden prompts to guide the supervised fine-tuning process. The crawled prompts contain some biases. For example, we observe

---

that they tend to generate more artwork instead of realistic photographs because most of them contain one or more artist names. Besides, the proportion of prompts about portraits is relatively higher than those about other categories. Although the reinforcement learning stage can mitigate these issues, would be better to balance the art styles and objects at the beginning. Moreover, we currently only apply our framework to text-to-image models. As the proposed framework is general to prompt-guided generation, we will apply to other generative models like text-only models and text-to-video models for future work.
Acknowledgments
We would like to thank Tan Yan for the helpful discussions.
References
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc., 2020.
Xinlei Chen, Hao Fang, Tsung-Yi Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO captions: Data collection and evaluation server. 2015.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez. Abhishek B Rao, Parker Barnes, Yi Tay, Noam M. Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Benton C. Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier García, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Oliveira Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Díaz, Orhan Firat, Michele Catasta, Jason Wei, Kathleen S. Meier- Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. PaLM: Scaling language modeling with pathways. ArXiv, abs/2204.02311. 2022.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017
Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. Knowledge neurons in pretrained transformers. In Proceedings of the 60th Annual Meeting of the Asvociation for Computational Linguistics (Volume 1: Long Papers), pages 2022. Association for Computational Linguistics. doi: 10.116653w122002.add-boooo
Dublin, Ireland, May
URL
.wcpogg...rrrg/e wwg. act- long. 581
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li. Kai Li, and Li Fei-Fei. Imagenet: large-scale hier- archical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248-255, 2009.
Ming Ding, Zhuoyi Yang, Wenyi Hong, Wendi Zheng, Chang Zhou, Da Yin, Junyang Lin, Xu Zou, Zhou Shao, Hongxia Yang, et al. Cogview: Mastering text-to-image generation via transformers. Advances in Neural Information Processing Systems, 34: 19822-9933,, 2021.
Nan Du Yanping Huang, Andrew M. Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, Barret Zoph, Liam Fedus, Maarten Bosma, Zongwei Zhou, Tao Wang, Yu Emma Wang, Kellie Webster, Marie Pellat, Kevin Robinson,
10

---

Toju Duke, Lucas Dixon, Kun Zhang, Quoe V Le. Yonghui Wu, Zhifeng Chen, and Claire Cui. Glam: Efficient scaling of language models with nixture-of-experts, 2021.
Kathleen
Tianyu Gao, Adam Fisch, and Danqi Chen. Making pre-trained language models better few-shot learners. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 11th International Joint Conference on Natural Language Processing ( (Volume I: Long Pupers), pages 381 6-3380, Online, August 2021. Association for Computational Linguisties. doi: 10.18653/v1/2011 acl-lllgg.299999 295. URL
acl-long
295
Shuyang Gu, Dong Chen, Jianmin Bao, Fang Wen, Bo Zhang, Dongdong Chen, Lu Yuan, and Baining Guo. Vector quantized diffusion model for text-to- -ontttoo-amge synthesis. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 10696-10706, 2022.
Braden Hancock, Antoine Bordes, Eanrnnel Mazare, and Jason Weston. Learning from dialogue after deployment: Feed yourself, chatbot! In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3667-3684, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10/19655 URL ftttps://aclantoology wrgg 1358
Adi Haviv, Jonathan Berant, and Amir Globerson. BERTese: Learning to speak to BERT. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume. pages 3618-3623, Online, April 2021. Association for Computational Linguistics. doi: 10. 10.18653/w122221.ccaa http:////22....12aaliaa macl-main.316 URL eacl 316
org/2021 2021
Borja Ibarz, Jan Leike, Tobias Pohlen, Geoffrey Irving. Shane Legg, and Dario Amodei. Reward learn- ing from human preferences and demonstrations in atari. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances Neural Information Processing http:::::///p/o/ppppoae
wwwnnng.nnuuirpr. new. sperrpps.om cc/ Systems, volume 31. Curran Associates, Inc., 2018 8. Paper. pdff]
Natasha Jaques, Asma Ghandeharioun, Judy Hanwen Shen, Craig Ferguson, Agata Lapedriza, Noah Jones, Shixiang Gu, and Rosalind Picard. Way off-policy batch deep reinforcement learning of implicit human preferences in dialog. arXiv preprint 1907 0046600456, 2019.
Zhengbao Jiang, Frank F. Xu, Jun Araki, and Graham Neubig. How Can We Know What Language Models Know? Transactions of the Association for Computational Linguistics, 8:423-438 07 2020. ISSN 2307-387X. doi: 10.1162/tacl a 00324. URL htthstts::/// //d ooooiioorg11... 1162/tacl_ a_ 00324
Christtooo Klie, Richard Eckart de Castilho, and Iryna Gurevych. From Zero to Hero: Human- In-The-Loop Entity Linking in Low Resource Domains. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6982 6993, Online, July 2020. Association for Computational Linguistics. doi: 10. 1/218 mai- URL hhtps
//aclanthology. . org/ 2020 acl- 624
Xiang Lisa Li and Perey Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 11th International Joint Conference on Natural Language Processing (Volume I: Long Papers), pages 4582-4597, Online, August 2021. Association for Computational Linguistics. doi: 1..18653/v1/20211 wtw.:/2220..aaoonoo.o... acl--ong.333.
.or.llyy..ooo/oooooooo 2021 1e02.acc--a...... 1o1g.11on.3300 353
Vivian Liu and Lydia B. Chilton. Design guidelines for prompt engineering text-to-image generative models, 2021.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa: A robustly optimized bert pretraining approach. arXiv preprint arXiv: 1692, 2019.
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. arXiv preprint arXiv:2206.0000 00927, 2022.
11

---

Naila Murray, Luca Marchesotti, and Florent Perronnin. Ava: A large-scale database for aesthetic visual analysis. In 2012 IEEE conference on computer vision and pattern recognition. pages 2408-2415 IEEE, 2012.
Jonas Oppenlaender. A taxonomy of prompt modifiers for sot-to--mage generation, 2022.
Long Ouyang. Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, al. Training language models to follow instructions with human feedback. arXiv preprint arXhv:2003.02/55: 2022.
Guy Parsons. The dall-e 2 prompt book, 2022.
Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. Language models knowledge bases? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2463-2473, Hong Kong. China, November 2019. Association for Computational Linguisties. doi: 10. 10.18653/11/D19-11 URL https://aclantholloo .rg/ 1250
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 7768.8763, PMLR, 18-24 Jul 2021. URL httpps://proceedingg. http:/ediiinnm......rreeeeeeee/e
www.ppressseeee wttass//v33///adfodd// html
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. arXiv, abs/2102. 12092, 2021a.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International Conference on Machine Learning, pages 8821-8831 PMLR, 2021b.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents, 2022.
Scott Reed, Zeynep Akata, Xinchen Yan, Lajanugen Logeswaran, Bernt Schiele, and Honglak Lee. Generative adversarial text to image synthesis. In International conference on machine learning, pages 1060-1069 PMLR, 2016a.
Scott E Reed, Zeynep Akata, Santosh Mohan, Samuel Tenka, Bernt Schiele, and Honglak Lee. Learning what and where to draw. Advances in neural information processing systems, 29, 2016b.
Laria Reynolds and Kyle McDonell. Prompt programming for large language models: Beyond the few-shot paradigm, 2021.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition (CVPR), pages 10684-10695, June 2022.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S. Sara Mahdavi, Rapha Gontijo Lopes, Tim Salimans, Jonathan Ho. David J Fleet, and Mohammad Norouzi. Photorealistic text-to-image diffusion models with deep language understanding, 2022.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. ArXiv, (ts//1007.6377,200
12

---

Richard Shin, Christopher Lin. Sam Thomson, Charles Chen, Subhro Roy, Emmanouil Antonios Platanios, Adam Pauls, Dan Klein, Jason Eisner, and Benjamin Van Durme. Constrained language models yield few-shot semantic parsers. In Proceedings of the 202/ Conference on Empirical Methods in Natural Language Processing, pages 7699-7715, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021. emnlp-main.60% 608. URL
wtt.e//annnnoooooooooo//// emnlp- mapp..minne. 608
Kurt Shuster, Jack Urbanek, Emily Dinan, Arthur Szlam, and Jason Weston. Deploying lifelong open-domain dialogue learning. arXiv preprint arXiv:2008.08076 2020.
Shaden Smith, Mostofa Patwary, Brandon Norick, Patrick LeGresley, Samyam Rajbhandari, Jared Casper, Zhun Liu, Shrimai Prabhumoye, George Zerveas, Vijay Korthikanti, Elton Zhang, Rewon Child, Reza Yazdani Aminabadi, Julie Bernauer, Xia Song, Mohammad Shoeybi, Yuxiong He, Michael Houston, Saurabh Tiwary, and Bryan Catanzaro. Using DeepSpeed and Megatron to train Megatron- Turing NLG 530B, large-scale scale generative language model, 2022.
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F. Christiano. Learning to summarize with human feedback. In Advances in Neural Information P Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020.
Ming Tao, Hao Tang, Fei Wu, Xiao Yuan Jing, Bing-Kun Kun Bao, and Changsheng Xu. Df-gan: A simple and effective baseline for text-to-image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16515-16525, 2022.
Maria Tsimpoukelli, Jacob L Menick, Serkan Cabi, SM Eslami, Oriol Vinyals, and Felix Hill. Multimodal few-shot learning with frozen language models. Advances in Neural Information Processing Systems, 34:200-212, 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, pages 5998-6008. Curran Associates, Inc., 2017. URL http::/paperre nips.cc.
all-you-need pdf
Ashwin K. Vijayakumar, Michael Cogswell, Ramprasaath R. Selvaraju, Qing Sun, Stefan Lee, David J. Crandall, and Dhruv Batra. Diverse beam search: Decoding diverse solutions from neural sequence models. ArXiv, abs/ 111.0444, 2016.
Zijie J. Wang, Evan Montoya, David Munechika, Haoyang Yang, Benjamin Hoover, and Duen Horng Chau. DiffusionDB: A large-scale prompt gallery dataset for text-to-image generative models. arXiv:2210.144 14899 [cs], 2022. URL Inttps///axxi wwwpaxii..org/aab.e a////////22111 14896
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoc V Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022. URL httpt::////peeeeeeeee nwt.:nuumtfffff..ec.mm VjQ[MeSB_J
Jing Xu, Megan Ung, Mojtaba Komeili, Kushal Arora, Y-Lan Boureau, and Jason Weston. Learn- ing skills after deployment: Improving open-domain internet-driven dialogue with human feedback. arXiv preprint arXiv:2208.033 03270, 2022.
Ziyu Yao, Yu Su, Huan Sun, and Wen-tau Yih. Model-based interactive semantic parsing: A unified framework and a text-to-SQL case study. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP- EEMNLP-IJCNPP,, pages 5447-5458, Hong Kong. China, November 2019. Association for Computational Linguistics. doi: 10.19655/11/D19-15477 URL
org/D19- DD9 1547
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision- language models. International Journal of Computer Vision, 13009))2337-2344 2022a.
13

---

Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. Large language models are human-level prompt engineers, 2022b.
Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv:199.008593, 2019.
14

---

Appendix
A Hyperparameter settings
Table 6: Hyperparameter settings supervised fine-tuning (SFT) and reinforcement learning (RL).
Hyperparameters
SFT
RL
Batch Size
256
256
Learning Rate
5e-5
5c-5
Training Steps
15000
12000
Max Length
512
512
Dropout
0.0
0.0
Optimizer
Adam
Adam
Adam
le-6
le-6
Adam B
(0.9, 0.999)
(0.9, 0.95)
Weight Decay
0.1
le-6
B
Computational budget
Our experiments implemented on V100 (32GB) GPU.
Table 7: Computational budget of supervised fine-tuning (SFT) and reinforcement learning (RL).
SFT
RL
The Number of GPUs
4
32
GPU Hours
3 hours
2.5 days
C
Results on Stable Diffusion v1.5.
Table 8: Results on Stable Diffusion v1.5
Lexica
DiffusionDB
COCO
User Input
-0.31
-0.32
-0.37
Human Engineered Prompt
-0.05
-0.18
Supervised Fine-tuning
-0.04
-0.16
-0.1
(OOM
0.05
0.06
0.11
15

---

D Comparisons with heuristic baseline
Table Combinations of common tags.
Tag
Content
artstation, highly detailed, elegant trending on artstation, concept art digital painting, intricate, fantasy illustration, smooth, octane render digital art, 8k, intricate highly detailed, elegant, smooth
Table 10: Comparisons with the heuristic baseline.
Data
User
Tag1
Tag2
Tag3
Tag4
Tag5
Tag6
Human
SFT
Ours
Lexica
-0.32
0.07
-0.06
0.06
-0.17
-0.05
-0.28
-0.02
0.03
0.14
DiffusionDB
-0.3
0
-0.07
-0.16
-0.1
-0.17
-0.31
-0.21
-0.01
0.06
COCO -0.38
-0.24
-0.29
-0.2
-0.33
-0.32
-0.41
-0.1
0.1
To compare the performance of our proposed framework with the heuristic baseline, select the top 15 frequent tags from human-engineered prompts and randomly combine them to create six groups of common tags. The specific tags presented in Table g We concatenate the user input with these common tags and compute their reward. The results are in Table [0]
While using these common tags can improve the reward to some extent, found that their perfor- mance varies significantly across different domains. For instance, tag3 performs well on COCO and Lexica but poorly on DiffusionDB. It suggests that relying tags may not be practical in real-world scenarios. In contrast, our proposed framework can perform well across domains and improve lot over the common tags.
handful of common selectd
16

---

E Results on different categories and lengths of prompts
We aim to validate the effectiveness of our method on different categories and different lengths. In Figure 2 we divide the prompts into several categories according to the prompt pattern, there are MC, MCM, RMC, RTP, in-domain DiffusionDB and out-of-domain COCO. Results show that optimized prompts are generally effective for all these categories. For semantic categories, these prompts have no clear boundaries. Therefore, we use RoBERTa-Large Liu et al. [2019) to get the sentence embedding of each prompt and perform K-means clustering on these prompts and divide list their proportion and their reward in Table. For length ablation, also classify them into five categories according to the length of input tokens. The results are in I2
them into five categories.
We observe that the performance of different lengths and semantic categories varies slightly but our model can improve the reward generally. When conducting reinforcement learning, we build large-scale prompts from both in in-domain data and -f-mma-n data, which cover a wide range of prompts with different lengths and semantic categories.
Table 11: Results on different semantic categories of prompts.
Cluster
1
2
3
4
Proportion
0.18
0.08
0.17
0.21
0.36
User Input
-0.39
-0.29
-0.37
-0.34
-0.31
PROMPTIST (Ours)
-0.01
0.04
0.13
0.06
0.1
Table 12: Results on different lengths of prompts.
Length
0~~10
10~20
20~~30
30~40
>40
Proportion
0.21
0.48
0.2
0.07
0.04
User Input
-0.48
-0.33
-0.18
-0.28
-0.22
PROMPTIST (Ours)
-0.02
0.11
0.08
0.05
0.06
17


Reviews:
[{"review_id": "IwFlsFcdz2", "reviewer": "Reviewer_zvr2", "summary": "This paper proposes a method for improving user-provided prompts for text-to-image models to make them more intent-aligned and aesthetically appealing. They use the following method:\n- Collect a dataset of paired basic and optimized user prompt.\n- Finetune a LM to take in a basic prompt and produce an optimized prompt.\n- Define a reward function based on CLIP text-image alignment and aesthetic appeal. (The aesthetic appeal is a new head trained on top of CLIP embeddings using a dataset of human judgements.)\n- Use PPO with diverse beam search sampling to finetune the prompt-improvement LM to train against this reward.\nThey show that the optimized prompts score better than even human-engineered prompts on aesthetic quality according to human evaluations.", "questions": "* What does it mean \"If the relevance score is relatively reasonable (larger than 0.28), we encourage the model to generate more aesthetically pleasing images\".  Does this mean you use a binary reward thresholded at .28?\n* In eqn 3, what are the constants 20 and -5.6 for? These seem arbitrary. (If it's just hyperparameters from a sweep, mention that.)\n\n* I wish there a comparison with a heuristic baseline which just includes a handful of common hand-selected tags which can apply across most inputs (e.g. \"artstation, hyperdetailed, concept art\"). Basically, I'm wondering how well Promptist is able to target the generated tags to the particular user prompt or whether it's has mostly just learned a few common tags which produce aesthetic images of all kinds. Alternatively, a supplement with a couple pages of non-cherrypicked optimized prompts would also answer this for me.\n* Relatedly, how does the diversity of the styles in Promptist prompts compare to the human-optimized prompts? (i.e. is it possible that human prompts are preferred less because humans ask for a more diverse set of image styles, not all of which score highly on the aesthetic score?)", "limitations": "The authors point out the issue that the training dataset is biased towards certain artistic styles. This analysis seems adequate.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "Originality: There is plenty of work on finetuning text-to-image models for text-image alignment and aesthetic scores, as well as on designing good hard prompts. I have not seen a paper which specifically optimizes prompts through RL.\nQuality: The paper's experiments and evaluation seem thorough.\nClarity: the paper is well written.\nSignificance: It's a known issue that there are specific keywords which can significantly improve image quality and that casual users of text-to-image models might not know to use these. This seems like a meaningful use case.", "weaknesses": "- I wish there were comparisons to other methods of improving text-to-image alignment and aesthetics. The paper does not compare against alternate methods (such as learning soft prompts or finetuning the model directly), claiming these are not preferred because they require you to backprop through the model. They also yield less-interpretable outputs. These are reasonable points, so I think this is a minor weakness. \n- The related work section also mentions methods for automatically searching for prompts. I'd either like to see comparisons so some of these methods or an explanation of why these comparisons aren't relevant.\n\n- There is no analysis on how reliable the CLIP score and the aesthetic score are.  (Though the results are backed up by human evaluation.)\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "*Q1: Does this mean you use a binary reward thresholded at .28? In eq 3, what are the constants 20 and -5.6 for?*\n\nA1: Yes, the relevance reward has a binary threshold at 0.28. We empirically observe that the image and the text are sufficiently relevant when the CLIP score is 0.28. Eq3 can be also written as $20 * min(g_{clip} - 0.28, 0)$, where 20 is a hyperparameter to balance the scale of the relevance score and the aesthetic score. When the CLIP score is larger than 0.28, we do not consider its further gain and encourage the model to generate more aesthetic images. When it is smaller than 0.28, the reward will decrease with a slope of 20.\nWe will state them in a clearer way in the camera-ready version.\n\\\n\\\n*Q2: I'd either like to see comparisons for automatically searching for prompts in the related work or an explanation of why these comparisons aren't relevant.*\n\nA2: Automatically searching for prompts [1,2,3] also aim to rewrite prompts to improve the downstream performance. [1] and [2] focus on knowledge extraction from BERT and [3] focus on few-shot learning of BERT. The main difference between our method and them is that they directly use downstream task labels as supervision to search for better prompts and cannot utilize the unsupervised data. They are more like the first stage of our framework, while our method can utilize the large-scale unsupervised data and the free-form exploration in reinforcement learning can improve further over supervised fine-tuning. \n\n[1] How Can We Know What Language Models Know? Jiang et al., 2020.\n\\\n[2] BERTese: Learning to Speak to BERT. Haviv et al., 2021.\n\\\n[3] Making Pre-trained Language Models Better Few-shot Learners. Gao et al., 2021.\n\\\n\\\n*Q3: Comparison with the heuristic baseline which just includes a handful of common hand-selected tags.*\n\nA3: To compare the performance of our proposed framework with the heuristic baseline, we select the top15 frequent tags from human-engineered prompts and randomly combine them to create six groups of common tags. The specific tags are as follows.\n\n|     tag    |     text                                          |\n|------------|---------------------------------------------------|\n|     1      |     artstation, highly detailed, elegant          |\n|     2      |     8 k, trending on artstation, concept   art    |\n|     3      |     digital painting, intricate, fantasy          |\n|     4      |     illustration, smooth, octane render           |\n|     5      |     digital art, 8k, intricate                    |\n|     6      |     highly detailed, elegant, smooth              |\n\nWe concatenate the user input with these common tags and compute their reward. The results are as follows.\n\n|     Data           |     user input    |     tag1     |     tag2     |     tag3     |     tag4     |     tag5     |     tag6     |     Human    |     SFT      |     Promptist    |\n|--------------------|-------------------|--------------|--------------|--------------|--------------|--------------|--------------|--------------|--------------|------------------|\n|     COCO           |     -0.38         |     -0.24    |     -0.29    |     -0.2     |     -0.33    |     -0.32    |     -0.41    |     N/A      |     -0.1     |     0.1          |\n|     DiffusionDB    |     -0.3          |     0        |     -0.07    |     -0.16    |     -0.1     |     -0.17    |     -0.31    |     -0.21    |     -0.01    |     0.06         |\n|     Lexica         |     -0.32         |     0.07     |     -0.06    |     0.06     |     -0.17    |     -0.05    |     -0.28    |     -0.02    |     0.03     |     0.14         |\n\nWhile using these common tags can improve the reward to some extent, we found that their performance varies significantly across different domains. For instance, tag3 performs well on COCO and Lexica but poorly on DiffusionDB. It suggests that relying on a handful of common hand-selected tags may not be practical in real-world scenarios. In contrast, our proposed framework can perform well across domains and improve a lot over the common tags. \n\\\n\\\n*Q4: The diversity of the optimized prompts compared with the human-optimized prompts.*\n\nA4: We propose two strategies to promote diversity:\n\\\n(1) (Line 72) When constructing data, we randomly remove or shuffle some modifiers of the human-engineered prompts as source prompt augmentations. It corresponds to the scenario that the human input contains specific styles based on their requirements. We encourage the model to maintain them while adding further improvements or choosing a better modifier order.\n\\\n(2) (Line 150) We observe that both top-p sampling and beam search limit the exploration space make the optimization prompts less diverse. To prevent the model from learning a trivial solution of adding same modifiers for all prompts, we use diverse beam search to explore optimized prompts to ensure both the quality and the diversity. \n\\\n\\\n*Q5: There is no analysis on how reliable the CLIP score and the aesthetic score are.*\n\nA5: (1) CLIP score: Both TorchMetric (https://torchmetrics.readthedocs.io/en/stable/multimodal/clip_score.html) and HuggingFace (https://huggingface.co/docs/diffusers/main/en/conceptual/evaluation) state that “CLIP score was found to have high correlation with human judgement”. \n\\\n(2) Aesthetic score: The aesthetic score predictor is trained on AVA [4] dataset, which contains over 250,000 images along with a rich variety of annotations. It has been widely used for aesthetic analysis.\n\\\nTherefore, they are reliable for evaluation. We did not conduct quantitative reliability analysis because it requires much human effort. Moreover, our main goal is to validate the effectiveness of prompt optimization with a reasonable reward, so we directly use them for evaluation.\n\n[4] Ava: A large-scale database for aesthetic visual analysis. Murray et al., 2012.\n"}, {"review_id": "PguSnNPos3", "reviewer": "Reviewer_brRy", "summary": "The paper proposes Promptist a prompt engine that leverages a pre-trained LLM (specifically GPT-2) to learn to generate prompts preferred by the model based on original prompt from the user. The approach is simple and very effective.\n\nThe policy network is first fine-tuned with supervision based on pairs of user inputs and manually engineered prompts, obtained from Lexica.art (90k manually engineered prompts). Synthetic prompts were generated using few different approaches, and then paired with the original engineered prompt from Lexica.art to create paired samples. This initial policy network is used as part of the reward function along with aesthetic and relevance scores. Prompts generated by Promptist performs even better than heavily manually engineered prompts. Experiments section also includes human evaluation comparing against main prompt and manually engineered prompt. Detailed experiments section provides useful insights.\n", "questions": "Q1) Would the prompts for the training set need to come from prompts used with SD 1.4+ family? Perhaps just requires the T2I model to be trained based on LAION-2B-en subset?", "limitations": "Yes", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 4, "contribution": 2, "strengths": "S1) Clarity of writing\n\nS2) Detailed experiments section providing useful insights\n\nS3) The proposed approach produces better results (aesthetics) than heavily manually engineered prompts\n\nS4) Simple and very effective approach\n", "weaknesses": "W1) Generalization to other architectures is not clear. Results are shown only for SD 1.4. \n\nW2) Generalization to the LM in Promptist is also not clear. Current experiments use GPT-2\n\nW3) CLIPScore of 0.28 seems to be a magic number. See Line 91 and equation 3. It is not clear how to adapt this to new data sets\n\nW4) Effectiveness of Promptist for different categories and lengths of prompts is also not clear\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "*Q1: Generalization to other architectures. (Results are shown only for SD 1.4)*\n\nA1: We also conduct experiments on Stable Diffusion v1.5. The results are as follows.\n\n|     Data           |     User input    |     Human    |     SFT      |     Promptist    |\n|--------------------|-------------------|--------------|--------------|------------------|\n|     COCO           |     -0.37         |     N/A      |     -0.1     |     0.11         |\n|     DiffusionDB    |     -0.32         |     -0.18    |     -0.16    |     0.06         |\n|     Lexica         |     -0.31         |     -0.05    |     -0.04    |     0.05         |\n\nThe results show that the optimized prompts are effective on SD 1.5 as well, i.e., consistently outperform human-engineered prompts and supervised-fine-tuning. Our framework is not limited to a specific text-to-image model version and the training data is designed independently of the model version. Therefore, our approach can easily generalize to other text-to-image models. \n\\\n\\\n*Q2: Generalization to other LMs. (Current experiments use GPT-2)*\n\nA2: Our proposed framework is agnostic to the choice of language model, and we use GPT-2 in our experiments to verify the effectiveness of optimized prompts. There are other more capable language models like LLAMA and MPT that can potentially yield better results. However, using these models will bring a higher training cost. We plan to explore the use of these language models in the future. \n\\\nIn summary, the proposed framework is general and can be applied to various text-to-image models and language models.\n\\\n\\\n*Q3: CLIPScore of 0.28 seems to be a magic number. It is not clear how to adapt this to new data sets.*\n\nA3: We studied the CLIP scores of generated images with the input text and found that the generated images are highly relevant to the input text when the CLIP scores are greater than 0.28. Therefore, we designed the reward function with a binary reward threshold at 0.28. We can directly use it on new datasets, because the number 0.28 depends on which CLIP model we choose.\n\\\n\\\n*Q4: Effectiveness of Promptist for different categories and lengths of prompts is also not clear.*\n\nA4: We divide the prompts into several categories according to the prompt pattern (Figure 2), there are MC, MCM, RMC, RTP, in-domain DiffusionDB and out-of-domain COCO. Results show that optimized prompts are generally effective for all these categories. For semantic categories, these prompts have no clear boundaries. Therefore, we use RoBERTa-Large to get the sentence embedding of each prompt and perform K-means clustering on these prompts and divide them into five categories. We list their proportion and their reward in the following table.\n\n|     Cluster       |     1        |     2        |     3        |     4        |     5        |\n|-------------------|--------------|--------------|--------------|--------------|--------------|\n|     Proportion    |     0.18     |     0.08     |     0.17     |     0.21     |     0.36     |\n|     User Input    |     -0.39    |     -0.29    |     -0.37    |     -0.34    |     -0.31    |\n|     Promptist     |     -0.01    |     0.04     |     0.13     |     0.06     |     0.1      |\n\nFor length ablation, we also classify them into 5 categories according to the length of input tokens. The results are as follows.\n\n|     Length        |     0~10     |     10~20    |     20~30    |     30~40    |     >40      |\n|-------------------|--------------|--------------|--------------|--------------|--------------|\n|     Proportion    |     0.21     |     0.48     |     0.2      |     0.07     |     0.04     |\n|     User Input    |     -0.48    |     -0.33    |     -0.18    |     -0.28    |     -0.22    |\n|     Promptist     |     -0.02    |     0.11     |     0.08     |     0.05     |     0.06     |\n\nWe observe that the performance of different lengths and semantic categories varies slightly but our model can improve the reward generally. When conducting reinforcement learning, we build large-scale prompts from both in-domain data and out-of-domain data, which cover a wide range of prompts with different lengths and semantic categories.\n\\\n\\\n*Q5: Would the prompts for the training set need to come from prompts used with SD 1.4+ family? Perhaps just requires the T2I model to be trained based on LAION-2B-en subset?*\n\nA5: The human-engineered prompts are collected from the Lexica website, where the prompts are designed for not only SD 1.4 but also many other text-to-image models. The proposed framework focuses on adapting user prompts to model-preferred ones. It does not matter whether the text-to-image model is trained on LAION-2B or not.\n"}, {"review_id": "r4KJI38ne3", "reviewer": "Reviewer_Kg4J", "summary": "The paper presents a novel framework for automatically optimizing prompts in text-to-image generation models. The authors combine supervised fine-tuning with a pretrained language model and reinforcement learning to explore better prompts. The goal is to generate more aesthetically pleasing images while preserving the original user intentions. Experimental results show that the proposed method outperforms manual prompt engineering in terms of both automatic metrics and human preference ratings. This work contributes significantly to the field of text-to-image generation by providing a systematic way to align user intentions and model-preferred prompts.", "questions": "1. How does the proposed framework handle complex prompts that require the generation of images with multiple objects or scenes?\n2. Could the authors provide more details on how the reinforcement learning stage mitigates the biases in the prompts used for supervised fine-tuning?", "limitations": "The main limitation of this study is the bias in the prompts used for supervised fine-tuning. The authors have acknowledged this and suggested that balancing the art styles and objects at the beginning could help address this issue. However, they do not provide a clear strategy for achieving this balance.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. **Originality**: The paper presents a novel framework, \"Prompt Adaptation\", for automatically optimizing prompts in text-to-image generation models. This approach is innovative as it combines supervised fine-tuning with a pretrained language model and reinforcement learning to explore better prompts. This is a significant contribution to the field of text-to-image generation.\n2. **Quality**: The paper is well-structured and provides a clear and detailed description of the proposed framework. The authors have also provided a comprehensive analysis of the results, demonstrating that their method outperforms manual prompt engineering in terms of both automatic metrics and human preference ratings.\n3. **Clarity**: The paper is well-written and easy to understand. The authors have done a good job of explaining complex concepts in a clear and concise manner.\n4. **Significance**: The proposed framework has the potential to significantly improve the quality of images generated from text prompts, making it a valuable tool for a wide range of applications in areas such as digital art, advertising, and content creation.", "weaknesses": "1. **Data Bias**: The authors acknowledge that the prompts used for supervised fine-tuning, which were crawled from the Lexica website, contain biases. For example, they tend to generate more artwork instead of realistic photographs and have a higher proportion of prompts about portraits than other categories. This could limit the generalizability of the proposed framework.\n2. **Limited Application**: Currently, the proposed framework is only applied to text-to-image models. While the authors mention plans to extend it to other generative models like text-only models and text-to-video models, this limitation could restrict the current applicability of the framework.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "*Q1: Prompts crawled from Lexica website contain data bias. This could limit the generalizability of the proposed framework.*\n\nA1: There are several approaches to mitigate the data bias such as data cleaning, semantic deduplication and directly writing high-quality prompt pairs. However, these approaches require much human labor cost. We leave utilizing these data cleaning methods for future work. \n\\\n\\\n*Q2: Limited Application: Currently, the proposed framework is only applied to text-to-image models.*\n\nA2: The misalignment of human prompts and model-preferred prompts is more severe in text-to-image models because the capacity of their text encoders is limited. Thus, we currently apply the method to text-to-image models. The proposed framework is general to other models of which different prompts can affect the generation quality. We leave them for future work.\n\\\n\\\n*Q3: How does the proposed framework handle complex prompts that require the generation of images with multiple objects or scenes?*\n\nA3: The training data we used is large and contains those prompts with many tokens (75 at most for Stable Diffusion), especially in rephrasing augmented prompts and prompts from DiffusionDB. They ensure that the framework can handle complex prompts and generate accurate and high-quality images.\n\\\n\\\n*Q4: Could the authors provide more details on how the reinforcement learning stage mitigates the biases in the prompts used for supervised fine-tuning?*\n\nA4: For supervised fine-tuning, the training data is the human-engineered prompts crawled from the website, which have some bias. When conducting reinforcement learning, we only need the source prompt, so we use many out-of-domain data which has nearly no bias compared to the supervised data. Moreover, we also use diverse beam search for prompt exploration in reinforcement learning. It means that the optimized prompts we explore are diverse every time. In summary, the first strategy can mitigate the bias of source prompt distribution and the second strategy can mitigate the bias of modifier distribution. \n\\\nAdditionally, the talk (https://eecs.berkeley.edu/research/colloquium/230419) illustrates the benefits of reinforcement learning over supervised fine-tuning, which is also applicable to our work.\n"}, {"review_id": "0j0EeJzxrx", "reviewer": "Reviewer_51Hz", "summary": "The author proposes a new general framework to optimize prompts for text-to-image generation. Specifically, the framework goes through two stages: a supervised fine-tuning stage on a small sets of human engineered prompts and a reinforcement learning process to explore better prompts.\n\nThe supervised data were web-crawled from Lexica and is augmented with different methods for diversity. During the reinforcement learning process, the model is optimized with two losses. A relevance loss that is obtained with a frozen CLIP encoder to determine if the image obtained from the modified prompt matches with the original text description; an aesthetic loss that is evaluated against an aesthetic predictor trained on human ratings.\n\nOverall, the paper is technically sound and introduces a new framework that can be easily adapted to use.", "questions": "In equation (4), the aesthetic score is the averaged difference in the aesthetic predictor score of the modified prompt and the original prompt. I wonder why the image generation from the original prompt is used.  \nWhat are the expected behaviour when using different choices of image generation model or language model? \n\nCould a smaller LM reach the same level of performance / generated quality with a larger image generation model?", "limitations": "As already identified by the authors, the generated results are highly biased towards artistic generation as opposed to photo-realistic images. It would be better if the promptist model could handle a wide range of art styles in generated images. The problem could be mitigated when bigger and better image generation model or language model are used.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "The paper is nicely written and easy to follow.\n\nThe introduced a two-stage pipeline that trains a LM to modify human prompts for better text-to-image generation. The qualitative and quantitative results are solid.", "weaknesses": "The experiment section could use more work to further validate the robustness of the general framework.\n\nThe choice of the model architecture for image generation (Stable Diffusion V1.4) and the LM (GPT-2) seem a bit arbitrary, especially given that these model were iterated at a very fast pace. Given new iteration on the development of image generation model, it could very well be the case that the limitations are addressed with more advanced image generation model.\n\nNo experiments were done on the different choices of model architecture.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "*Q1: Why is the image generated from the original prompt used in the aesthetic score?*\n\nA1: There are two possible reasons for the high aesthetic score of an optimized prompt. One is the optimized prompt has high aesthetic improvements over the original prompt, the other is the original prompt itself has a high aesthetic score. Intuitively, the former should have a higher reward than the latter. Therefore, we use the aesthetic improvement over the original prompt instead of the absolute aesthetic score as the reward.\n\\\n\\\n*Q2: No experiments were done on the different choices of model architecture. What are the expected behaviour when using different choices of image generation model or language model?*\n\nA2: (1) Different image generation model. We also conduct experiments on Stable Diffusion v1.5. The results are as follows.\n\n|     Data           |     User input    |     Human    |     SFT      |     Promptist    |\n|--------------------|-------------------|--------------|--------------|------------------|\n|     COCO           |     -0.37         |     N/A      |     -0.1     |     0.11         |\n|     DiffusionDB    |     -0.32         |     -0.18    |     -0.16    |     0.06         |\n|     Lexica         |     -0.31         |     -0.05    |     -0.04    |     0.05         |\n\n(2) There are other more capable language models like LLAMA and MPT that can potentially yield better results. However, using these models will bring a higher training cost. \n\nIn general, using more capable text-to-image models and language models can potentially improve the generation quality further. The proposed framework is general to various choices of models and we will explore the use of these models in the future.\n\\\n\\\n*Q3: Could a smaller LM reach the same level of performance / generated quality with a larger image generation model?*\n\nA3: Compared with smaller image generation models, the larger models usually have better baseline generation quality and are relatively more robust to prompt patterns. Therefore, we can use a relatively smaller LM to optimize prompts.\n\\\n\\\n*Q4: Given new iteration on the development of image generation model, it could very well be the case that the limitations are addressed with more advanced image generation model.*\n\nA4: It is because of the fast iteration of models that we require an automatic framework to optimize prompts. The old optimized prompts could be less effective on the new models and it is laborious to conduct human-engineering for all versions of models. The proposed framework can generally optimize prompts automatically for them through reinforcement learning. Furthermore, more high-quality training data and larger model scale can improve the generation quality of text-to-image models, but there is still a gap between model-preferred prompts and human-preferred prompts. It is challenging for current text-to-image models (890M for SD) to achieve the strong enough capability of text understanding and be totally robust to the prompt change. Therefore, the framework is effective because better prompts can improve the generation quality further.\n"}]

Author rebuttal:

Meta-review: This paper proposes a prompt adaptation method to generate model-preferred prompts from original user inputs for text-to-image generation models. The proposed method first performs supervised fine-tuning with a pre-trained language model on manually engineered prompts, and then explores improved prompts via reinforcement learning with a reward function based on the aesthetic appeal and text-image alignment. 

Initial concerns include the lack of comparisons with other methods, the reliability of the CLIP score and the aesthetic score, the generalization to other architecture and data, and limited application. Reviewers acknowledged that the rebuttal addressed most of these concerns. Finally, all reviewers gave positive final ratings. The AC agrees with the reviewers that the idea is novel, the proposed RL-based method is simple yet effective, and the paper is generally well-written. Therefore, the AC recommends accept. Reviewers did raise valuable concerns that should be addressed. The authors are encouraged to make the necessary changes in the camera-ready version.
Decision: Accept (spotlight)