Improving Text-to-Image Consistency via Automatic Prompt Optimization
Authors: Oscar Mañas
Pietro Astolfi
Melissa Hall
Candace Ross
Jack Urbanek
Adina Williams
Aishwarya Agrawal
Adriana Romero-Soriano
Michal Drozdzal
Venue: Accepted by TMLR ()

Abstract: Impressive advances in text-to-image (T2I) generative models have yielded a plethora of high performing models which are able to generate aesthetically appealing, photorealistic images. Despite the progress, these models still struggle to produce images that are consistent with the input prompt, oftentimes failing to capture object quantities, relations and attributes properly. Existing solutions to improve prompt-image consistency suffer from the following challenges: (1) they oftentimes require model fine-tuning, (2) they only focus on nearby prompt samples, and (3) they are affected by unfavorable trade-offs among image quality, representation diversity, and prompt-image consistency. In this paper, we address these challenges and introduce a T2I optimization-by-prompting framework, OPT2I, which leverages a large language model (LLM) to improve prompt-image consistency in T2I models. Our framework starts from a user prompt and iteratively generates revised prompts with the goal of maximizing a consistency score. Our extensive validation on two datasets, MSCOCO and PartiPrompts, shows that OPT2I can boost the initial consistency score by up to 24.9% in terms of DSG score while preserving the FID and increasing the recall between generated and real data. Our work paves the way toward building more reliable and robust T2I systems by harnessing the power of LLMs.

Published in Transactions on Machine Learning Research (11/2024)
Improving Text-to-lmage Consistency via Automatic Prompt Optimization
Oscar Mañas*
www.rmmmnommmmla..oomo quebec
Mila, Université de Montréal, Meta FAIR
Pictro Astolfi
Meta FAIR
Melissa Hall
Meta FATR
Candace Ross
Meta FAIR
Jack Urbanek
Meta FAIR
Adina Williams Meta FAIR
Aishwarya Agrawal
Mila, Université de Montréal, Canada CIFAR AI Chair
Adriana Romero-Soriano
Mila, McCall Universily, Meta FAIR, Canada CIFAR AI Chair
Michal Drozdzal
Meta FATR
"Contributed equally
Reviewed on OpenReview: https: openreview. net/ pormm id= g12Gdl6aDi
Abstract
Impressive advances in text-to-image (T21) generative models have yielded plethora of high performing models which are able to generate aesthetically appealing, photorealistic images. Despite the progress, these models still struggle to produce images that are consistent with the input prompt, oftentimes failing to capture object quantities, relations and attributes properly. Existing solutions to improve prompt-inage consistency suffer from the following challenges: (1) they oftentimes require model fine-tuning, (2) they only focus on nearby prompt samples, and (3) they are affected by unfavorable trade-offs among image quality, representation diversity, and prompt-image consistency. In this paper, we address these challenges and introduce a T2I optimization optinizzttoonby-porpttinngng framework, OPT2I, which leverages large language model (LLM) improve prompt-image consistency in T21 models. Our framework starts from user prompt and iteratively generates revised prompts with the goal of maximizing a consistency score. Our extensive validation on two datasets, MSCOCÓ and PartiPrompts, shows that ОРT21 can boost the initial consistency score by up 24.9% in terms of DSG score while preserving the FID and increasing the recall between generated and real data. Our work paves the way toward building more reliable and robust. T2I systems by harnessing the power of L.Mss.

---

Published in Transactions on Machine Learning Research (11/2024)
Optimized prompt
User prompt
cultured tophat, proudly holes cane, while nearby garbage beg is artist toxl ly included in the scene. The painting reflects the vibrent molors textures Váncent begh paintangs.
femal
A raccoon wearing formal clothes, wearing tophat and ho holeing a cane. The racenen is doldding gertage bag. painting in the style of Vinoent Bogg
T21
T21 prompt aptimizer
T21
T2I
Scorer
LLM
Scorer
Scorer
1. there a racecon? Yes
Is there
recnnon solding came?
2. Ia the raccoun holidang came? You
receoon vearing tophac? Yes bag?
3. Ia the ruccoun near ang cophat? You
4. I6 there garbage bag? Yes
the rectsen holdiing gurbagee bug?
5. the
holdding garbage bag?
Score: 1. nenn
Figure 1: Overview of our backpropagatton--fre texl-to-image optimization by prompting approach that rewrites user prompts with the goal of improving prompt-image consistency. Our framework is composed of text-to-image generative model (T2I), a large language model (LLM) and consistency objective (Scorer). The LLM iteratively leverages history of prompt-score pairs to suggest revised prompts. In the depicted example, our system improves the consistency score by over 30% in terms of Davidsonian Scene Graph score.
1 Introduction
In recent years, we have witnessed remarkable progress in text-to-image (T2I) generative models (Ramesh
et al. 2022 Saharia et al., 2022 Rombach et al. 2022b Podell et al. 2023 Dai et al. 2023b). The photore- alistic quality and aesthetics of generated images has positioned T21 generative models at the center of the current AI revolution. However, progress in image quality has come at the expense of model representation diversity and prompt-image consistency (Hall et al. 2023). From a user's perspective, the fact that not all the elements of the input text prompt are properly represented in the generated image is particularly prob- lematic as it induces ledious and time-consuming trial-and-error process with the T2I model to refine the initial prompt in order to generate the originally intended image. Common consistency failure modes of the T2I generative models include missing objects, wrong object cardinality, missing mixed object attributes, and non-compliance with requested spatial relationships among objects in the image (Wu et al. 2023a
To improve prompt-image consistency, researchers have explored avenues such as adjusting the sampling guidance scale (Ho & Salimans 2022), modifying cross-altention operations (Feng et al. 2022 Epstein
et al. 2023 Liu et al. 2022 Chefer et al.) 2023 Wu et al., 2023a),, fine-tmnigg models <Lee et al. 2023
Wu et al. 2023c Sun et al. 2023), leveraging additional input modalities such as layouts (Cho et al.
2023b Lian et al. 2023), and selecting images post-hoc (Karthik et al. (023). Most of these approaches require access to the model's weights and are not applicable when models only accessible through API interface, e.g., (Betker et al. 2023). The only two approaches that applicable to APP-accessible models are guidance scale modification and post-hoc image selection. However, these approaches rely on a single lext prompt - the one provided by the user so their ability to generate diverse images is limited to resampling of the imput noise, í.e., changing the random seed. Perhaps more importantly, both these approaches have unfavorable trado-offs, as they improve prompt-image consistency at the expense of image quality and diversity e.g., high guidance scales lead reduced image quality and diversity, while post-hoc selecting the most consistent images decreases significantly the representation diversity.
2

---

Published in Transactions on Machine Learning Research (11/2024)
At the same time, the natural language processing (NLP) community has explored large language models
(LLMss) as prompt optimizers for NL.P tasks (Pryzant et al. 2023 Yang et al. 2023 Guo et al. 2023 Fernando
et al. 2023), showing that LLMs can relieve humans from the manual and tedious task of prompt-engineering
One particularly interesting approach is in-context learning (ICL) (Dong et al.
2023
where an LLM can
learn to solve a new task from just a handful of in-contexi examples provided in the input prompt. Notably, LLMs can accomplish this without requiring parameter updates - e.g., LLMs can solve simple regression tasks when provided with a few input-output examples (Mirchandani et al. 2023). ICL enables rapid task adapta- tion by changing the problem definition directly in the prompt. However, ICL commonly relies on predefined datasets of examples (input-output pairs), which might be challenging to obtain. To the best of our knowl- edge, ICL has not yet been explored in the context of T21 generative models, which pose unique challenges: (1) the in-conlext examples of prompls and the associated scores are not readily available, and (2) the LLM needs to ground the prompt in the generated images in order to understand how to refine it. Recently, and once again the NLP domain, optimization-by prnpriaaatoooo----rptiinnnnn prompting (OPRO) (Yang et al. 2023) has extended previous ICL approaches by iteratively optimizing instruction prompls to maximize task accuracy based on a dataset of input-output examples, leveraging feedback from past instructions and their respective task accuracies.
In this work, we propose the first ICL-based method to improve prompt-image consistency in T2I mod- els. In particular, we leverage optimization-by. srwniiitttioo-b--oooppp by-promptig and introduce novel inference-time optimization framework for T2I prompts, which constructs a dataset of in-context examples on-the-fly. Our framework, OPtimization for T21 generative models (OPT2I), involves a pre-trained T21 model, an LLM, and an auto- matic prompt-image consistency score e.g., CLIPScore (Hessel et al.) 2021) or Davidsonian Scene Graph score (DSG) (Cho et al., 2023a)) Through ICT., the L.M iteratively improves a user-provided text prompt by suggesting alternative prompts that lead to images that are more aligned with the user's intention, measured by the consistency score. At each optimization iteration, the in-context examples are updated to include the best solutions found so far. Intuitively, we expect our method to explore the space of possible prompt paraphrases and gradually discover the patterns in its previously suggested prompts that lead to highly-cousistent images. Crucially, the optimized prompts will produce more consistent images on expec- tation, across multiple input noise samples. OPT2T is designed to be a versatile approach that works as a plug-and and- solution with diverse T21 models, LLMs, and scoring functions since it does not require any parameter updates. An overview of our framework is presented in Figure
Through extensive experiments, we show that OPT21 consistently outperforms paraphrasing baselines (e.g., random paraphrasing and Promptist (Hao et al. 2022)), and boosts the prompt-image consistency by up to 12.2% and 24.9% on MSCOCO (Lin et al. 2014 and PartiPrompts (Yu et al.) 2022 datascts, respectively. Notably, we achieve this improvement while preserving the Fréchet Inception Distance (FID) (Heusel el al., 2017) and increasing the recall between generated and real data. Moreover, we observe that OPT2I achieves consistent improvements for diverse T21 models and is robust to the choice of LLM. Our qualitative results reveal that the optimized prompts oftentimes emphasize the elements of the initial prompt that do not appear in the generated images by either providing additional details about those or reordering elements in the prompt to place the ignored elements at the beginning. In summary, our contributions are:
We propose OPT2I, a training-free T2I hotiiiiizatiio-----o-oo-o prompts for a T21 model that improve crompt-image consistency.
framework that provides refined
OPT2T a versatile framework as it not tied to any particular T2I model and is robust to both the choice of LLM as well as consistency metric.
We show that OPT2I consistently outperforms paraphrasing baselines and can boost the prompt- image consistency by up to 24.9%.
2 OPT21: Optimization by prompting for T21
Figure 2 depicts our T2I optimizattion-by-rroee prompiigg framework, which is composed of a pro-trained T2I model and an LLM that work together to optimize a prompt-image consistency score. Our framework starts from user prompt and iteratively generates revised prompls with the goal of maximizing the chosen
3

---

Published in Transactions on Machine Learning Research (11/2024)
consistency score. More concretely, we start by feeding the user prompt to the T2I model to generate multiple images. Next, we compute the consistency score between each generated image and the user prompt, and average the scores. We then initialize the meta-prompll history with the user prompt and its associated consistency score. Finally, we feed the meta-prompt to the LLM, which proposes a set of revised prompls. To start a new optimization step, we feed the revised prompts back to the T2I model that generate new images. Note that the consistency score always computed w.r.t. the user prompt. At each optimization step, the meta-prompt history updated to include the top-k most consistent prompts (among revised prompts and user prompt), along with their consistency score. The prompt optimization process terminates when maximum number of iterations is reached, or when perfect /target consistency score is achieved. Finally, the best prompt is selected, namely, the optimized prompt.
2.1 Problem formulation
We assume access to LLM, and pre- trained T2I generative model, g. Given text prompt, P, we can generate an image 1 condi- tioned on the prompt with our T2I generator, I = g(p). Let us define the set all possible paraphrases from p that can be obtained with an LLM as P {p:}, and let us introduce a prompl-image consistency score, S(p, Our objective is to find prompt paraphrase P P that maximizos the expected consistency score of sampled images:
User prompt
T21
Generated image
Revised prompt
Consistency Metric
arggax E P:~P Ims(p))
[S(po: [),,
Meta-prompt
LLM
Task description
Prompt history
where Po denotes the user prompt. We ap- proach the optimization problem with ICL by iteratively searching for revised prompts gen- erated by the LLM,
Prompt optimization
Our prompt optimization framework, OPT2I, composed of (1) text-to-image (T2I) generative model that generates images from text prompts, (2) a consistency met- ric that evaluates the fidelity between the generated images and the user prompt., and (3) a large language model (LLM) that leverages task description and history of prompt- score tuples to provide revised prompts. At the beginning, the revised prompt is initialized with the user prompt.
Figure 2:
Pt = [(C(tp))UP U Pt-)),
where P. is the set of prompts generated at iteration i, t is the current iteration, and C' is a function that defines the context of prompt-score pairs fed to the LLM.
2.2 Meta-prompt design
We adapt L.MSs for T2T prompt optimization via ICL. We denote meta- prommp the prompt which instructs the LLM to optimize prompts for T2I models. Our mela-prompt is composed of a lask instruction and history of past revised prompt-score pairs. The list of meta-proon prompts used can be found in Appendix A
The peta-p provides context about T2I models, the consistency metric and the optimization problem. Additionally, it contains a history of prompt-score pairs that provides context about which paraphrases worked best in the past for a particular T2I model, encouraging the LLM to build upon the most successful prompts and removing the need for explicitly specifying how modify the T2I prompt. The consistency score is normalized to an integer between and 100, and we only keep in the history the top-k scoring prompts found so far, sorted by increasing score.
4

---

Published in Transactions on Machine Learning Research (11/2024)
2.3 Optimization objective
A critical part of our framework is feeding visual feedback the LLM. The visual feedback is captured by the consistency score, which determines how good the candidate prompt is at generating consistent images. Although OPT2I can work with any even non-differentiable consistency score, argue that the consistency score must be detailed enough for the LLM to infer how to improve the candidate prompt. While CLIPScore (Hessel et al., 2021) is arguably the most popular metric for measuring prompt-image consistency, in our initial experiments we found that scoring a prompt with a single scalar is too coarse for our purposes. Thus, we opt for two metrics that provide finer- finer-graine grained information about the prompt-image consistency: (1) Davidsonian Scene Graph (DSG) score (Cho et al.. 2023a), and (2) our proposed decomposed CLIPScore.
DSG assesses prompt-image consistency based on a question generation and answering approach, similar to TIFA (Hu et al. 2023). In particular, DSG generates atomic and unique binary questions from the user prompt that are organized into semantic dependency graphs. For example, "a bike lying on the ground, covered in snow" is decomposed into: (1) "Is there a bike?"; (2) "Is the bike lying on the ground?"; (3) "Is the bike covered in snow?" In this case, questions (2) and (3) depend on (1), and so (1) is used to validate (2) and (3). These questions are then answered by an off thefhff- VQA model based on the generated image. We include the resulting question-answer pairs in our meta-prompt. A global score per prompt-iman -imape pair is computed by averaging across answer scores.
Decomposed CLIPScore computes partial consistency score for each noun phrase present in the user prompt. For example, "a bike lying on the ground, covered in snow" is decomposed into "a bike", "the ground" and "snow". Each noun phrase is then scored against. the generated image using CLIPScore, resulting in a list of pairs of noun phrases and their associated scores, which are included in our meta-prompt. A global score per prompt-image pair computed by averaging across subscores. We provide examples of decomposed CLIPScore and DSG outputs in Appendix
2.4 Exploration-exploitttior trade-off
During the optinization process, OPT2I requires controllability over the LLM's exploration-exploiaaiii trade-off, as the LLM could either focus on exploring possible revised prompts or on exploiting the context provided in the meta-prompt history. On the one hand, too much exploration would hamper the optimiza- tion as it could be hard to find a high quality solution. On the other hand, too much exploitation would limit the exploration to prompts that are very similar the ones already presented in the meta-prompt history. We control this balance by adjusting the number of generated revised prompts per iteration and the LLM sampling temperature. Moreover, T21 imput noise samples, generate multiple images per prompt at each iteration.
objective is to find prompts that work well across different
3 Experiments
We first introduce our experimental setting. Next, we validate the effectiveness of OPT21 in improving prompt-image consistency, compare it to paraphrasing basclines (random paraphrasing and Promptist), and show some qualitative results. Then, we explore the trade-offss with image quality and diversity. And finally, we ablate OPT2I components and touch on post-hoc image selection.
3.1 Experimental setting
Benchmarks. We run experiments using prompts from MSCOCO Lin et al. 2014 and PartiPrompts (P2) Yu ct al.) 2022). For MSCOCO, we use the 2000 captions from the validation set as in (Hu et al. 2023). These captions represent real world scenes containing common objects. PartiPrompts, instead, is a collection of 1600 artificial prompts, often unrealistic, divided into categories to stress different capabilities of T2I generative models. We select our PartiPrompts subset by merging the first 50 prompts from the most challenging categories: "Properties & "ositioning", "Quantity", "Fino-grained Detail", and "Complex". This results in set of 185 complex prompts.
5

---

Published in Transactions on Machine Learning Research (11/2024)
Baselines. We compare OPT2I against a random paraphrasing baseline, where the LLM is asked to generate diverse paraphrases of the user prompt, without any context about the consistency of the images generated from it. The meta- meta-prompt prompt used to obtain paraphrases is provided in Appendix We also compare to Promp- tist Hao et al. (2022). which relics on a dataset of initial and target prompts to finetune an LLM to rewrite user prompls with the goal of improving image aesthelics, while trying to maintain prompl-image consistency.
Evaluation metrics. We measure the quality of T21 prompt by averaging prompt-image consistency scores across multiple image generations (f.e., multiple random seeds for the initial noise). For each generated image, we compute its consistency with the user prompt. We consider our proposed decomposed CLIPScore (dCS) and the recent DSG score (Cho et al. 2023a) as consistency metrics (sce Section 2.3 for more details). For DSG score, we use Instruct- BLIP (Dai et al. 2023a) as the VQA model. To assess the trade-off between prompt-image consistency, image quality and diversity, we additionally compute FID score (Heusel et al. 2017), precision and recall metrics (Nacem et al. 2020).
LLMs and T2I models. For the T2I model, we consider (1) a state-of-the-ar latent diffusion model, namely LDM-2.1 (Rombach et al. 2022a), which uses CLIP text encoder for conditioning, and (2) a cascaded pixel-based diffusion model, CDM-M, which instead relies on the conditioning from a large language model, T5-XXL (Raffel et al. 2020). : similarly to (Saharia et al.] 2022]. For the LLM, we experiment with the open source Llama-2-700 chat (Llama-2) Toovron et al. 2023 and with GPT-3. 5-Tuboo-0-- (GPT-3.5) (Brown et al. 2020
Implementation details. Unless stated otherwise, OPT2I runs for at most 30 iterations generating 5 new revised prompts per iteration, while the random paraphrasing baseline generates 150 prompts at once (see Section 3.4 for a discussion about the relationship between #iters and #prompts/iten LLM to generate revised prompts by enumerating them in a single response to prevent duplication, rather than making multiple calls with a sampling temperature greater than O. In the optimization meta-promp we set the history length to 5. To speed up image generation, we use DDIM (Song et al. (020) sampling. We perform 50 inference steps with LDM-2. 1, while for CDM-M we perform 100 steps with the low-resolution generalor and 50 steps with the super-resolution network following the default parameters in both cases. The guidance scale for both T21 models is kept to the suggested value of 7.5, Finally, in our experiments, we fix the initial random seeds across iterations and prompts wherever possible, i.e. we fix 4 random sceds for sampling different prior/noise vectors to generate 4 images from the same prompt. However, we note that CDM-M does not allow batched generation with fixed seeds. Experiments without seed beed-fixing are reported in Appendix B as we observe no substantial differences.
We instruct the
3.2 Main results
T2I optimization by prompting. In Figure B we plot the prompt optimization curves with LDM-2. LDM-2.1/CDM0 1/CDM-M as T2I models, Llama-2/GPT-33 as LLM, and decomposed CLIPscore (dCS)/DSG as the scorer for prompts from MSCOCO and PartiPrompts. Fach data point corresponds to the mean/ / rel- ative improvement in consistency score (w.r.t. the user prompt) achieved by revised prompts generated in that iteration, averaged across the full dataset of prompts. The optimization curves show an overall upward trend, which confirms that the LLM in OPT2T is capable of optimizing T2T prompts. These improvements are especially noticeable in the mar consistency score. The initial dip in mean consistency score is expected due to the initial exploration, since the LLM has limited context provided only by the user prompt (1-shot ICL). As the optimization progresses, the LLM generates more consistent revised prompts at each iteration, as its context increasingly enriched with the performance of previous revised prompts. Notably, achieving positive mean relative consistency starting from a single in-context example is very challenging task (Wei et al. 2023), and OPT2I achieves this goal for all configurations except for Llama-2 2 and GPT-3.5 in the dCS( P2) setting (Figure 3b), where it only get close to 0. As mentioned Section 3.1 PartiPrompts is a hard benchmark containing highly detailed and complex prompts, so it is perhaps unsurprising that decom- posed CLIPScore falls short (with improvements < 7% in the max case, and < 2% in the mean case) given the imperfect decomposition into noun mounsehrsses. We also explored a hierarchical version of decomposed CLIPscore leveraging constituency trees, which did not show any improvement over our noun-phrase basod decomposition, further reinforcing the criticism that CLIP behaves as a bag-of-words and is unable to prop-
explored
erly capture object attributes and relations (Yuksekgonul et al. 2022 Yamada al. 2022). Instead, using
6

---

Published in Transactions on Machine Learning Research (11/2024)
Paraphr w/ Llama-2 LDM-2.1
Paraphr w/ GPT-3.5, LDM-2.1
Paraphr w/ Liama CDM-M
OPT2I w/ Liama-2, LDM-2.1
OPT2I w/ GPT-3.5, LDM-2.1
OPT2I w/ Liama-2, CDM-M
Relative consistency (%)
max
max
max
max
2
Relative consistency (%)
2
/mean
mean
mean
mean
-2
-2
-4
10 20
30
-6
10
30
20
10 iteration
30
iteration
iteration
iteration
(a) dCS (MSCOCO)
(b) dCS (P2)
(c) DSG (MSCOCO)
(d) DSG (P2)
Figure 3: OPT2T curves with different consistency objectives (dCS vs. DSG), LLMs, and T2I models. Each plot track either the mar or the mean relative improvement in consistency across revised prompts per iteration.
more detailed consistency score during the prompt optimization process, such DSG, results in more significant improvements (< 17% in the max case, and < 5% in the mean case).
Comparison to paraphrasing baselines. Table shows our proposed OPT2I framework is robust to the choice of LLM, T2I model and optimization cacliion//vlluatiin /evaluation objective. In particular, for dCS, we report relative improvement as scorebest/ scoreimit - 1. For DSG score, since the initial score can be zero and it already percentage, we instead report scoresest scoreinit- We observe that OPT2I consistently outperforms the random paraphrasing baseline across different LLMs and T2T models. Additionally, both paraphrasing and optimization get around 10% boost in consistency improvement when using DSG optimization objective instead of dCS for P2 but not for MSCOCO. This highlights again that more complex prompts, such as those from PartiPrompts, benefit from more accurate consistency metric. We note that some prompts might already have fairly high initial consistency score (sce App. B.2), so there is little room for improvementt. For instance, prompts from MSCOCO evaluated with DSG have an average initial score of 86.54%, which means that the improvement in this setting has an upper bound of 13.46%.
that
In addition to random paraphrasing, prompts by generating images from initial/best prompts (4 images/prompt with SD-1.4 (Promptist's 's ref- erence model) and LDM-2.1, 1, evaluating consistency with DSG score. We observe Promptist decreases the consistency score by 33.56%/-3.200 on SD-1. 10-1.4/LDM-2.1 1, while OPT21 (Llama-2) improves consistency by +44.6%//11.21% This aligns with the results reported in (Hao et al. 2022), which show that optimizing prompts primarily for aesthetics actually decreases prompt-image consistency.
compare OPT2I to Promptist (Hao et al. 2022 MSCOCO
Qualitative results. In Figure 4 we provide examples of images generated from user and optimized prompts with OPT2I for different LLMs and T2I models. We observe OPT2I is capable of finding para- phrases of the user prompt which considerably improve the consistency between the generated images and the initial, user-provided prompt, measured by DSG in this case. These examples suggest the optimized prompls are capable of steering the T2I model towards generating visual elements that. were ignored with the initial phrasing. From our qualitative analysis, we observed the LLM uses several strategies to emphasize the missing visual elements, such as providing a more detailed description of those elements (e.g., "a flower" "a vibrant flower arrangement", "a vase filled with fresh blossoms") or placing them the beginning of the sentence (e-g., "four teacups surrounding kettle" "surround kettle placed at the center with four

---

Published in Transactions on Machine Learning Research (11/2024)
harse several. covs feed (0. 4043)
horse several colo feed (2.5000)
6,3744
C.2522
2,220
9.6571
atack hay, surrounded cowa tang "estive 6573)
horse muttiple feed
8629)
6.6572
5.0466
bowl full of tosa toes sitting next to flower, (6, 6657)
bool Nil of tonatoes sitting next to flaver, (8.6660 6667)
3.30
8,0663
corondc overfluding Flover arrangeeent. treating visunlly sspealing contraxt. (1. 2000)
bovl.
beautsfull
ri lile with Fresh blober. (3 10.0000))
(a) LLM: Llama-2, T2I: LDM-2.
(b) LLM: Llama-2, T2I: CDM-M
roccoun woriig tomat clattes, weardng toohat and holaing cane. The garbage 19. 47221
reccoon veardng formal clothes, wes riin toprot and holding a care. The (6.4722)
style
spghiffched attire did poses" Siti
ans pansche, heising cane and ga rargee bas. against stutnang soan laam- style suintirg's siotorate details. 18. (3.7)00)
An sil painting the style of pointil Lise depácting a raccoon elegantly
holdinn gartage
(0, 6944)
surounding
surourding
5.394
5.3111
E-1593
E.799
E.393
Four teacuss ancircle lettle, fomsing catesive and picturesque sctup.
alaced center teacups. (L.29201
1.3000
L/C8EE
1.0000
1.300
1.0083
1.030
2.0000
(c) LLM: Llama-2, T2I: LDM-2.1
(d) LLM: GPT-3.5, T2I: LDM-2.1
Figure 4: Selected qualitative results for prompts from MSCOCO (a-b) and P2 (c-d) datascts, using DSG as consistency metric. For each setup, we display four rows (from the lop): initial prompt #1, optimized prompt #1, initial prompt #2, and optimized prompt #2. Each column corresponds to a different T2r model random seed. We report average consistency score across seeds between parenthesis.
8

---

Published in Transactions on Machine Learning Research (11/2024)
Table Relative improvement in T21 consistency between the user prompt and the best prompt found, averaged across all prompts in the dataset. Every method generates the same total number of prompts (and images).
Table 2: Distributional metrics on the MSCOCO dataset.
Prompt
Obj.
(1)
IV3
1V3
CLIP
L00-2.3
34.6
16.0
81.6
86.3
47.2
22.8
OPT2
dCS
Liara-2
LDK-2.1
34.6
14.9
70.7
81.0
55.6
30.0
Method
Objective
LLM
T21
Dataset
OPT2
D8G
LbN-2.1
2.1
53.5
15.4
70.0
64.3
64A
270
MSC000
P2
OrTn
dCs
OPT-3.6
LDW-2.1
34.1
74.9
84.1
63.9
27.3
Paraphnssng
19.96
8.00
DPT2
DSG
691-3.5
1DN-2.1
53.4
90.3
65.4
60.5
20.7"
OPT21
dCS
Liasa-2
LDK-2.1
+11.88
+10.34
lnntal
CD3--5
41.2
15.2
82.2
85.6
08.8
26.0
Paraphrasing
DSG
Clans-2
LDN-2.1
10.667
+91.22
OFTH
dCs
Liara-2
39.8
29.5
OPT21
+11.21
+22.24
OPTN
DSG
Llara-2
CDN-N
39.9
25.0
OPT21
Paraphrusing
dCS
GPT-3.6
1DK-2.1
+10.35
+9.81
+9.33
Table 3: Meta-prompt ablations.
OPT21
Paraphnwing
DSG
GPT-3.5
LDK-2.1
+10.85
-10.34
+19.77
+19.00
Paraphrasing
dCs
Liana-2
CDN-M
-11.00
+10..9
Conciseness
Prioritize
Reasoning
Structure
dCS (%)
OPT21
+12.21
+12.13
+9.68
Paraphrusing
10.53
22.24
+10.34
OPT21
DSG
Llana-2
CDK-11
+11.07
+24.93
/
/
x
+10.23
/
x
+99.9
leacups"). We note perfect consistency score does not ensure perfectly aligned images (e.g., for the user prompt "four teacups surrounding a kettle", all optimized prompts reach a DSG score of 100% while the car- dinality of teacups remains incorrect), which highlights the limitations of current prompt-inage consistency scores. We also observe that prompts optimized by Llama-2 tend to be longer than those from GPT-3. 5 (see App. B.5,, and that images generated by CDM-M from user prompts are generally more consistent than those generated by LDM-2.1, 1, which we attribute to the use of a stronger text encoder (T5-XXL instead of CLIP).
3.3 Trade-offs with image quality and diversity
Following common practice in the T2I community, we evaluate the quality OPT2I generations by comput- ing image generation metrics such as FTD, precision (P), and recall (R). We use the 2000 prompts from the MSCOCO validation set that included in the TIFAv1 benchmark (Hu al. 2023), and generate 4 images for each initial and best prompt. To ensure robust conclusions, Inception-v3 (IV3) (Szegedy et al. 2016) and CLIP (Radford et al. 2021 Results in Table show that the FID of prompts optimized with OPT2I is either on-par or better compared to that of initial prompts, vali- dating that our method does not trade image quality for consistency. Hence, we conclude FID is not affected by our optimization strategy. However, in terms of precision and recall, we observe that optimized prompts reach higher recall at the expense of lower precision compared to the user prompt. This can be explained by the fact that rephrasing the input prompt allows generate more diverse images (higher recall), which may occasionally fall outside of the manifold of natural images (lower precision); e-g., in Fig. 12 (Appendix B optimizing for consistency leads to change of artistic style. The observed trade-off between realism and diversity aligns with the results reported by Astolfi et al. (2024).
two feature extractors in metrics:
3.4 Ablations
We perform ablations with Llama-2 and LDM-2. 1 on PartiPrompts using default parameters unless otherwise specified. Figure illustrates the trade-off between exploration and exploitation, implemented as the number of revised prompts per iteration (#pomppss/ (roopts//ter) and the number of optimization itcrations (#itcrations), re- spectively. Generating more prompts at each iteration increases the erploration of multiple solutions given the same context, while by increasing the number of iterations, the LLM can exploit more frequent feedback from the T2I model and the consistency score. We observe that increasing number of iterations leads to higher consistency improvement. In other words, more exploitation beneficial with a fixed budget of 150 prompt generations. However, pushing exploitation too much, i.e., #it. 150 and #p/it 1, becomes harmful.
9

---

Published in Transactions on Machine Learning Research (11/2024)
10
95
90
Average D5 G score
Relative dCs (%)
85
80
75
4
Paraphr, it=1, pft==50
70
OPTZI, It=1, p/t=110
OPT21, it=5, p/
65
OPT2I, It=15, pri==10
Initial
OPT2I, it=30, prit-5
60
Paraphr.
OPT2I, it=1500, p/it=1
OPT2I
55
0
30
60
90
120
150
4
10
25
50
100
200
400
600
# #revised prompts
Figure 5: Cumulative mar relative dCS as a function of #revised prompts = #iterations #prompts/iter.
Figure 6: Average DSG score for the top-k most consistent. images among 600.
In table 3 we report relative consisteney (dCS) when ablating for different task instructions in the mela- prompt. We explore four instruction additions to be combined with our base meta veeta-prompt. Conciseness encourages to explore paraphrases beyond just adding details/ details/adjectiins adjectives; Prioritize encourages focusing on missing/low-soone elements: Reasoning encourages reason about the in-context examples; and Structure asks for simple vocabulary informative of the structure of images, e-g-, foreground and background (full meta-prompts provided in Appendix Ap. We observe that Prioritize achieves slightly better performance over Reasoning and Structure, yet the LLM remains fairly robust to specific meta-prona prompt phrasing.
3.5 Post-hoc image selection
We emphasize OPT2I aims to optimize prompts to generate more consistent images on erpectation. However, for the sake of completeness, we also evaluate the setting where we generate the same amount of images from the initial prompt and select the most consistent ones. In particular, we generate 600 images from PartiPrompts using either random image sampling from the initial prompt, paraphrasing, or OPT2I, and select the top-k most consistent images based on DSG score. In Fig. 6 we observe that OPT2I consistently outperforms both baselines. Interestingly, sampling from the initial prompl outperforms paraphrasing, which might be due to random paraphrases deviating too much from the user's intent.
4 Related work
Improving consistency in T2I models. Several recent works propose extensions to T2I models to improve their faithfulness to user prompts. Some studies focus on improving the guidance with cross-attention (Feng
et al.
2022
Epstein et al. 2023: Liu et al. 2022 Chefer et al. 2023 Wu et al.) 2023a). Other studies first
convert a textual prompt into a layout before feeding to a iorouu-to-imagee generative model (Cho et al.
2023b Lian et al. 2023 Recent works also finetune T2T models on human (Lee al., 2023 Wu et al. 2023c
Wallace et al. 2023) or AI model (Sun al., 2023) feedback, or perform post-hoc image selection (Karthik et al. 2023). In contrast, OPT2I acts exclusively at. the level of input prompt in text space, without accessing model weights, making it applicable to wider range of T21 models, including those only accessible through API.
LLMs as prompt optimizers. Several recent works explore the role of LLMs as prompl optimizers for NLP tasks. Some use LLMs to directly optimize the task instruction for ICL (Zhou et al. 2022 Pryzant et al.
2023
Yang et al. 2023 Other studics use I.LMs mutate prompts for evolutionary algorithms (Guo al.
2023
Fernando et al. 2022). crucial difference between these works and our method is that they optimize a
task instruction prompt. by using training set, which is subsequently applied across test examples, while
10

---

Published in Transactions on Machine Learning Research (11/2024)
perform multimodal inference-time optimization on individual T21 prompts. More similar to our work, other studies rewrite prompts for T2T models using an L.M.M (Hao et al. 2022 finetunes an LLM with reinforcement learning to improve image aesthetics, while (Valerio et al, 2023) focuses on filtering out non-visual prompt clements. In contrast, OPT2I aims to improve prompt-image consistency via optimization-by- wrw.iiioonb---roppiii..omo
Evaluating prompt-image consistency. Several metries have been proposed to evaluate prompt-image consistency. CLIPScore (Hessel et al. 2021] is the de facto standard for measuring the compatibility of image-caption pairs, used both for image captioning and text-conditioned image generation. However, CLIPScore provides single global score, which can be too coarse to understand failures in the generated images. Consequently, subsequent metrics such as TIFA (Hu et al., (023), VQ² (Yarom et al.) 2023) or DSG (Cho et al. 2023a) propose generating pairs of questions and answers from T21 prompts and using off th-the-shelf VQA models to evaluate each of them on the generated images, providing a fine- Ane-grained score. Other recent studies suggest directly learning a prompt-image consistency metric from human feedback (Xul ct al. 2023 Wu et al., 2023b Kirstain et al., 2023). However, none of these metrics are without flaws and human judgment remains the most. reliable way of evaluating prompt-image consistency.
5 Conclusions
In this paper, we introduced the first T2I optimization-by. -ytprottttoo-----pptinnnnn framework improve prompt-image consistency. Through extensive evaluations, we showed that OPT2T can be effectively applied to different combinations of LLM, T21 models and consistency metrics, consistently outperforming paraphrasing baselines and yielding prompl-image consistency improvements of up to 24.9% over the user prompl, while maintain- ing the FID between generated and real images. By contrasting MSCOCO and PartiPrompts results, we highlighted the importance of the choice of consistency score: complex prompts in PartiPrompts appear to significantly benefit from more detailed scores such as DSG. Qualitatively, we observed that optimizing prompts for prompt-image consistency oftentimes translates into emphasizing initially ignored elements in the generated images, by either providing additional details about those rewording the prompt such that the ignored elements appear at the beginning. Interestingly, such prompt modifications steer the generated images away from the learned modes, resulting in higher recall w.r.l. the real data distribution.
Limitations. One limitation of our method is that it expects prompt-image consistency scores to work reasonably well. However, this assumption might not hold in some cases. For instance, it has been shown that CLIP (used for CLIPScore) sometimes behaves like a bag-of-words (Yuksekgonul et al. 2022] Yamada al.] (022).. VQA-based prompt-image consistency metrics such as TIFA or DSG also suffer from limitations in generating questions (e.g., the question "Is the horse the hay?" is generated from the prompt "A horse and several cows feed on hay:") in answering them with VQA model (e.g., for the prompt "A bowl full of tomatoes sitting next to flower.", the VQA model answers that there is flower when it is in fact a bouquet made of tomatoes). Moreover, using these metrics as optimization objectives might exacerbate their failure modes by finding prompts which generate images that fulfill the requirements for a high score in an adversarial way. This highlights the need for further research in developing more robust prompt-image consistency metrics which can be used as optimization objectives addition to evaluation.
Another limitation of our approach is its runtime, which is consequence of performing inference-time optimization. For instance, running the optimization process with Llama-2, LDM-2.1 and DSG score, generating 5 prompt paraphrases per iteration and 4 imagos per prompt with 50 diffusion steps, takes 7.34/20.27 iterations on average for JDCCOPartiPromptt which translates to ~ 10/28 minutes when using NVIDIA V100 GPUs However, we emphasize that (1) OPT2I is designed to be a versatile approach that works as a plug-and-pl play solution with diverse T2I models and LLMs since does not require any parameter updates nor training data, and (2) optimizing T2I prompts with our automatic framework relieves humans from the manual and tedious task of prompt-engineering.
11

---

Published in Transactions on Machine Learning Research (11/2024)
References
Pietro Astolfi, Marlene Careil, Melissa Hall, Oscar Mañas, Matthew Muckley, Jakob Verbeek, Adri- ana Romero Soriano, and Michal Drozdzal. saggisteenny-dvvvrstyyrrlaa pareto fronts of conditional image generative models. arviv preprinl arXiv:2406. 10429, 2024.
James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, et al. Improving image generation with better captions. Computer Science. https://cdn. openai. hom/papers/dall-st pdf, 2(3):8, 2023.
Lucas Beyer, Andreas Steiner, André Susano Pinto, Alexander Kolesnikov, Xiao Wang, Danicl Salz, Maxim Neumann, Ibrahim Alabdulmohsin, Michael Tschannen, Emanuele Bugliarello, et al. Paligemma:: A versatile 3b vim for transfer. arXiv preprint arXiv:2407.07726, 2024.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Hila Chefer, Yuval Alaluf, Yael Vinker, Lior Wolf, and Daniel Cohen-Or. Attend-and-excite: Attention-based semantic guidance for poxt-to-image diffusion models. ACM Transactions on Graphics (TOG), 42(4):1-10, 2023.
Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E Gonzalez, al. Chatbot arena: An open platform for evaluating Ilms by human preference. arXiv preprint ar WxXi:2403.04433. 2024.
Jaemin Cho, Yushi Hu, Roopal Garg, Peter Anderson, Ranjay Krishna, Jason Baldridge, Mohit Bansal,
Jordi Pont-Tuset, and Su Wang. Davidsonian scene graph: Improving reliability in fine-grained evaluation for text-image generation. arXiv preprint arXiv:2310.1823: 2023a.
Jaemin Cho, Abhay Zala, and Mohit Bansal. Visual programming for text-to-inne image generation and evalua- tion. arXiv preprint or Xiv:2300 15328, 2023b.
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general- purpose vision-language models with instruction tuning, 2023a.
Xiaoliang Dai, Ji Hou, Chih- Yao Ma, Sam Tsai, Jialiang Wang, Rui Wang, Peizhao Zhang, Simon Van- denhende, Xiaofang Wang, Abhimanyu Dubey, et al. Emu: Enhancing image generation models using photogenic noedles in a haystack. arXiv preprint arXiv:2309. 15807, 2023b.
Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, Lei Li, and Zhilang Sui. A survey on in-context learning, 2023.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llaaa herd of models. arXiv preprinl vrXiv:2407.2178. 2024.
Dave Epstein, Allan Jabri, Ben Poole, Alexci A Efros, and Aleksander Holynski. Diffusion self-guidance for controllable image generation. arXiv preprint ar arXiv:2306.0080 2023.
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, al. Scaling rectified flow transformers for high-resolution image synthesis. In forty-first International Conference on Machine Learning, 2024.
Weixi Feng, Xuehai He, Tsu-Jui Fu, Varun Jampani, Arjun Reddy Akula, Pradyumna Narayana, Sugato Basu, Xin Eric Wang, and William Yang Wang. Training-free structured diffusion guidance for compo- sitional text-to-image synthesis. In The Eleventh International Conference on Learning Representations, 2022.
12

---

Published in Transactions on Machine Learning Research (11/2024)
Chrisantha Fernando, Dylan Banarse, Henryk Michalewski, Simon Osindero, and Tim Rocktäschel. Prompt- breeder: Self-referential self-improvement via prompt evolution. arXiv preprint or arX::2330.. 16797, 2023.
Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedingss of the IEEE conference on compuler vision and pattern recogrition, pp. 6904 6913, 2017.
Qingyan Guo, Rui Wang, Junliang Guo, Bei Li, Kaitao Song, Xu Tan, Guoqing Liu, Jiang Bian, and Yujiu Yang. Connecting large language models with evolutionary algorithms yields powerful prompt optimizers. arXiv preprint Xix:2990 08532, 2023.
Melissa Hall, Candace Ross, Adina Williams, Nicolas Carion, Michal Drozdzal, and Adriana Romero Soriano. Dig in: Evaluating disparities in image generations with indicators for geographic diversity, 2023.
Yaru Hao, Zewen Chi, Li Dong, and Furu Wei. Optimizing prompts for text-to-image generation. arXiv preprint XXf:22912.09611, 2022.
Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation metric for image captioning. arXiv preprint arXiv:2144.08718 2021.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural infor- mation processing systems, 30, 2017.
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance, 2022.
Yushi Hu, Benlin Liu, Jungo Kasai, Yizhong Wang, Mari Ostendorf, Ranjay Krishna, and Noah A. Smith. Tifa: Accurate and interpretable text-to-image faithfulness evaluation with question answering. In Proceed- ings of the IEEE/CVE International Conference on Compuler Vision (ICCV), pp. 20406-241 20417, October 2023.
Shyamgopal Karthik, Karsten Roth, Massimiliano Mancini, and Zeynop Akata. If at first you don't 't succeed, try, try again: Faithful diffusion- based text-to-image generation by selection. arXiv preprint wwX3:2305511308, 2023.
Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, and Omer Levy. Pick-a-pic: An open dataset of user preferences for text-to-image generation. arXiv preprint amxiv:2305.01509 2023.
Kimin Lee, Hao Liu, Moonkyung Ryu, Olivia Watkins, Yuqing Du, Craig Boutilier, Pieter Abbeel, Moham- mad Ghavamzadeh, and Shixiang Shane Gu. Aligning text-to-image models using human feodback. arXiv preprinl ar Xiv:22302. 12192, 2023.
Long Lian, Boyi Li, Adam Yala, and Trevor Darrell. Lim-ga grounded diffusion: Enhancing prompt under- standing of text-to-image diffusion models with large language models. arXiv preprint arXiv:2005. 13655, 2023.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrencc Zitnick. Microsoft coco: Common objects in context. In Computer European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part 13, pp. 740 755. Springer, 2014.
2014: 13th
Nan Liu, Shuang Li, Yilun Du, Antonio Torralba, and Joshua B Tenenbaum. Compositional visual generation with composable diffusion models. In European Conference on Computer Vision, pp. 423-439. Springer, 2022.
Suvir Mirchandani, Fei Xia, Pete Florence, Danny Dricss, Montserrat Gonzalez Arenas, Kanishka Rao, Dorsa Sadigh, Andy Zeng, et al. Large language models as general pattern machines. In 7th Annual Conference on Robot Learning, 2023.
13

---

Published in Transactions on Machine Learning Research (11/2024)
Muhammad Ferjad Nacem, Scong Joon Oh, Youngjung Uh, Yunjey Choi, and Jacjun Yoo. Reliable fidelity and diversity metrics for generative models. In International Conference on Machine Learning, pp. 7176- 7185. PMLR, 2020.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-rcsolution image synthesis. arXiv preprinl trXiv:2307011522 2023.
Reid Pryzant, Dan Iter, Jerry Li, Yin Tat Lee, Chenguang Zhu, and Michael Zeng. Automatic prompt oplimization with' gradient descent" and beam search. arXiv preprinl (XXv:2305.03495, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramosh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkinn Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8700 PMT.R, 2021.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 21(1):5485 5551, 2020.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents, 2022.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 10684-10695, 2022a.
Robin Rombach, Andreas Blattmann, Dominík Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models, 2022b.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Bureu Karagol Ayan, S. Sara Mahdavi, Rapha Gontijo Lopes, Tim Salimans, Jonathan Ho, David J Fleet, and Mohammad Norouzi. Photorealistic text-to-image diffusion models with deep language understanding, 2022.
Axel Sauer, Dominik Lorenz, Andreas Blattmann, and Robin Rombach. Adversarial diffusion distillation. arXiv preprint ar arXtt:2311.10023 2023.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXie preprint (:Xx::2100.22500 2020.
Jiao Sun, Deqing Fu, Yushi Hu, Su Wang, Royi Rassin, Da-Cheng Juan, Dana Alon, Charles Herrmann, Sjoerd van Steenkiste, Ranjay Krishna, et al. Dreamsyne: Aligning text-to-image generation with image understanding feedback. arXiv preprint (tt::2311.199... 2023.
Christian Szegedy, Vincent Vanhoucke, Sergey Toffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pallern recognition, pp. 2818 2826, 2016.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bash- lykov, Sounya Batra, Prajjwal Bhargava, Shruti Bhosale, al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 2023.
Rodrigo Valerio, Joao Bordalo, Michal Yarom, Yonattan Bitton, Idan Sxpektor, and Joao Magalhaes. Transferring visual attributes from natural language to verified image generation. arXiv preprint arXi::3305.10026 2023.
Bram Wallace, Meihua Dang, Rafael Rafailov, Linqi Zhou, Aaron Lou, Senthil Purushwalkam, Stefano Ermon, Caiming Xiong, Shafiq Joty, and Nikhil Naik. Diffusion model alignment using direct preference optimization. arXiv preprint ar Xim:2311 12908, 2023.
14

---

Published in Transactions on Machine Learning Research (11/2024)
Jerry Wei, Jason Wei, Yi Tay, Dustin Tran, Albert Webson, Yifeng Lu, Xinyun Chen, Hanxiao Liu, Da Huang, Denny Zhou, et al. Larger language models do in-context learning differently. orxiv preprinl arXiv:2303.038 03846, 2023.
Qiucheng Wu, Yujian Liu, Handong Zhao, Trung Bui, Zhe Lin, Yang Zhang, and Shiyu Chang. Harnessing the spatial-temporal attention of diffusion models for high-fidelity text-to-image synthesis. In Proceedings of the IEEE/CVF International Conference on Compuler Vision, pp. 7766 7776, 2023a.
Xiaoshi Wu, Yiming Hao, Keqiang Sun, Yixiong Chen, Feng Zhu, Rui Zhao, and Hongsheng Li. Human preference score v2: A solid benehmark for evaluating human preferences of text-to-image synthesis. arXiv preprint ar xrx::233060000 09341, 2023b.
Xiaoshi Wu, Keqiang Sun, Feng Zhu, Rui Zhao, and Hongsheng Li. Better aligning text-to-imes tottgeegeee models with human preference. arXiv preprint Xix:22000 140:22003,440,, 2023c.
Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong, Imagereward: Learning and evaluating human preferences for text-to-image generation. arXiv preprinl AXXiv:2344005977 2023.
Yutaro Yamada, Yingtian Tang, and Ilker Yildirim. When are lemons purple? the concept association bias of clip, arViv preprint Xr::::2212. 12048, 2022.
Chengrum Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoe V Le, Denny Zhou, and Xinyun Chen. Large language models optimizers. arXiv preprint arXin:2309 09409, 2023.
Michal Yarom, Yonatan Bitton, Soravit Changpinyo, Roee Aharoni, Jonathan Herzig, Oran Lang, Eran Ofek, and Idan Szpektor. What you see is what you read? improving text-image alignment evaluation. arXiv preprint arXio:2305.10400 2023.
Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexan- der Ku, Yinfei Yang, Burcu Karagol Ayan, et al. Scaling autoregressive models for content-rich text-to- image generation. arXiv preprint or Xix:22200. 10789, 2022.
Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou. When and why vision-languaae models behave like lags-offwords. and what to do about it? In The Eleventh International Conference on Learning Representations, 2022.
Yongchao Zhou, Andrei Ioan Muresamu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. Large language models are human-level prompt engineers. In The Eleventh International Conference on Learning Representations, 2022.
15

---

Published in Transactions on Machine Learning Research (11/2024)
A Additional method details
A.1 Meta-prompts
Tables include all the prompts used in our work. The teal text is fed to the LLM as a system prompt, and the purple lext denotes placeholders to be dynamically filled.
Table 4: Meta-prompt used for decomposing prompts into noun phrases for dCS.
Decompose the folloving sentence into individual noun phrases. Ignore prefixes auch as 'a photo of ta picture of "a portrait of', etc. Your response should only be 1ist of comma separated values, baz"
{pronpt)
Table 5: Meta-prompt used for the paraphrasing baseline.
Generate (oun solutions] paraphrases following inage description while keeping senantic meaning: "(user_proupt)", Respond with each new pronpt in between <PROИPТ> and </PRONPT>, eg:
1. <РROиPT> parappaaar 1</PROXPT> pPRoMstsaarapaase 2</PROMPT>
(nun_solutions). <P80KPT>paraphrase (aun_solutions) </PRONPT>
16

---

Published in Transactions on Machine Learning Research (11/2024)
Table 6: Nleta-prompe used for OPT2I with dCS (top) or DSG (bottom) as scorer.
You expert prompt optimizer for text-to-inage nodels. Text-to- models input and generate inages depicting the prompt as output. You translate prompta gritten by humana into better prompta for the -to--gg models. Your ansvers should be concise and affactive.
prompt
decompasition initial proupt visual elamenta. Each elament patrad its presence in the generated image. The pronpts are arranged in asconding order based on their scores, which range fron 0 100. Higher scores indicate higher likelihood of presence.
"(user, "unerr_rompt]". Belov
prempts
score indicating
1. frevised_pee frevised_proapt..j
acore: favg_acore__) visual elementa:
(aubpronpt__1-) {clip_acore_ {aubprompt_1_2 $elip_score___2 (... nore questions ...)
(... exanplea
Generate (ous_solutin solutions] paraphrases of the initial prompt which keep the semantic neaning and that have higher scores than all the prompta above. Prioritize optinizing for object with lovest acorea. Favor subetitutione reorderinge PROMPT>, eg:
additione. Respond with pronpt <PROMPT>
*PROMPT>paraphret 1</PROXPT>
2. pPRaMat>parapbase 2</PROMPT>
{num_ solutioitio <PR0NTT>paraphreee { </PROMPT>
You expert prompt optinizer for text-to-insgs nodels. Text Tex--to-inage inage models take text prempt prenpt output. translate prompts vritten by input generate inages depicting nodels. Your snekers should be concise and effective. better pronpts for the tezt-to- to--ttgo
Your task preapts The prompts are arranged in ascending order based on their overall consistency score, which questions.
optinize this initial pronpt coneistency
hunan: "(user_prompt)" Belov generated inage
previous binary
each preapt 's
elements
ranges from 0 to 100 (higher is better).
overall score: (dsg.score_1) evalustion questicns: (vgs_score_1_1.
(question_1.2) (vqs_score__1, (... more questions ...)
(... examples ...)
Generate cuu_sroutioos] paraphrases the initial preapt which keep the senantic moaning and that have higher scores than all the pronpts above. Focus on optinizing for the visual elements that are not consistent. Favor substitutions and reorderings additions. Respond with each prompt in betusen PROMPT> and </PADNPT>, eg:
1. <PROMPT>paraphrase 1</PROMPT
2. <PROMPT>paraphase 2</PROMPT>
{num_ cunn_colutios). <P80NPT>paraphruse <aun_solutions] C/Pp:tt </PROMPT>
17

---

Published in Transactions on Machine Learning Research (11/2024)
Table 7: Meta-prompt ablations. Modifications w.r.t. the base meta- veeta-pommp are denoted blue.
Conciseness:
(... nore exanplea ...)
Generate (oun_ solutions] paraphrases initial prompt which keep semantic neaning higher scores than all the prompta above. Favor aubatitutions and reorderings over additions. Respond with each nav prompt in batvean <PROMPT> and </PROMPT>, eg:
Conciseness prioritize:
(... more examples ...)
Generate {oun solutions) paraphrases of the initial prompt which keep the semantic meaning and that have higher than all the prompts abore. Prioritize optinizing for object with lovest Favor aubatitutions and reorderinga over additions. Respond with each neu pronpt in betseen <PROMPT> PROMPT>, eg:
Conciseness + prioritize reasoning:
(... more exanples ...)
Briafly reason (nax two sentences) about the pronpts above to understand why cartain objects have higher certain prompts. Then, based this reasaning, generate coun_solutions) paraphrases of the initial proupt which keep aemantic neaning that have higher acorea than all preapta above. Prioritize optimizing for objects with lavest scores while keeping high acores for tha other objacts. Favor substitutions and reorderings over additions. Respand vith each new pronpt in batreen PRUMPT> </PRDNPT>, eg:
lover
Conciseness + prioritize + structure:
(... more exanplea ...)
rolution))) paraphrases higher scores than all the prompts above. PRIORITIZE optinizing for abjects with lavest acores whila keeping high scores for the other objacts. FAVOR substitutions and reorderings ovar additions. USE simple zorda/coned concepta, underatable Respond
initial prempt
keep semantic meaning
inage e.g., distinguish foreground background. </PROMPT>, eg:
pronpt in betuean
18

---

Published in Transactions on Machine Learning Research (11/2024)
A.2 Examples of prompt decompositions
Table shows lew examples of outpuls generaled by dCS (noun phrases) and DSG (binary questions) from the imput prompt. These outputs are then used to compute a fine-grained consistency score w.r.t. the gencrated image, cither with a multimodal encoder (dCS) or a VQA model (DSG).
Table 8: Prompt decompositions into noun phrases (dCS) and binary questions (DGS).
Prompt
dCS
DSG
"A ginger sleeping the window."
"ginger cat", "window"
"Is there cat?", "Is the ginger?", "Is the sleep- ing?", "Is there window?", "Is the cat next the win- dow?"
"Many electronic wires pass over the road with few cars on it."
"electronic
wires",
"road"
"Are there many electronic wires?", "Is there road?", "Are there few cars?", "Do the electronic wires pass over the road?", "Are the cars on the road?"
"cars"
"there is long hot dog that has toppings it"
"long hot dog", "toppings"
"Is there hot dog?", "Is the hot dog long?", "Is the hot dog hot?", "Are there toppings the hot dog?"
"the mona lisa wearing a cow- boy hat and screaming a punk song into microphone"
"the mona lisa", "a cowboy hat", "a punk song", "a micro- phone"
"Is there the Mona Lisa?", "Is the Mona Lisa wearing a cowbay hat?", "Is the Mona Lisa holding microphone?", "Is the hat cowbay hat?", "Is the Mona Lisa scream- ing?", "Is song the Mona Lisa singing punk?", "Is the Mona Lisa screaming into the microphone?"
"a photograph of a bird wearing headphones and speaking into microphone in recording stu- dio"
"photograph", "bird", "head- phones", "microphone"
"Is this photograph?", "Is there bird?", "Doss the bird have headphones?", "Does the bird have microphone?", "Is there recording studio?", "Is the bird speaking into the microphone?", "Is the bird wearing headphones?", "Is the bird in the recording studio?"
"recording studio"
"concentric squares fading from yellow outside to deep or- ange the inside"
"concentric squares", "yellow", "outside", "deep orange", "in- side"
the squares concentric?", "Is the square yellow?", , "Is the inner square deep ange?", "Is the square inside nuter square?", "Is the outer square the outside?", "Is the inner square the inside?"
there squares?",
19

---

Published in Transactions on Machine Learning Research (11/2024)
B
Additional results
Table 9: Comparison with 1-shot in-context learning (ICL). We report relative improvement (%) in prompt- image consistency between the user prompt and the best prompt found, averaged across all prompts in the dataset.
Method
Objective
LLM
T21
Dataset
MSCOCO
P2
Paraphrasing
+9.86
+8.00
I-shot ICL
dCs
Llena-2
LDK-2.
+10.67
+8.74
OPT21
+11.68
+10.34
Paraphrasing
+9.92
+19.22
1-shot ICL
DSG
Llana-2
LDK-2.
+10.14
+19.69
OPT21
+11.02
+22.24
Paraphrasing
+9.64
+8.06
1-shot ICL
dcs
GPT-3.5
LDK-2.
+9.21*
+8.72
OPT21
+10.56
+9.33
Paraphrasing
+10.21
+19.09
1-shot ICL
DSG
GPT-3.5
LDK-2.
+10.09+
OPT21
+11.19
+19.77
Paraphrasing 1-shot ICL
+11.38
+10,29
dCs
Llana-2
CDN-M
+12.19
+11.34
OPT21
+1266
+1213
Paraphrosing
+9.86
+22.24
L-shot ICL
DSG
Llana-2
CDN-M
+10.10
+22.25
OPT21
+10.15
+24.93
B.1 1-shot in-context learning as baseline
In this experiment, compare OPT2I with 1-shot in-context learning (ICL), which implement by running OPT21 with #iter = 1 and #prompts/ feedback about the performance of the user prompt. We maintain the same experimental setting described in Section 3 except we use 200 prompts for MSCOCO, and report the results in Table g First, we notice that 1-shot ICL achieves higher prompt-image consistency than random paraphrasing in all settings except when using GPT-3.5 5, which performs on-par or slightly worse (marked with * in Table D see also the discussion Section B.5 Second, and more importantly, we observe that OPT2I outperforms the 1-shot ICL baseline regardless of the consistency objective, LLM, or T2I model adopted. These results reinforce claims: (1) the iterative procedure allows OPT2I to keep improving revised prompts over time, and (2) 1-shot ICL is challenging due to the limited feedback provided to the LLM about how to improve the user prompt. and thus only minor improvements in prompt-image consistency can be obtained over random paraphrasing.
150. Note that, in this setting, the LLM only receives
previous
Table 10: Relative prompt-image consistency improvement between the user prompt and the best prompt found, averaged across prompts.
MSCOCO
P2
Method
LLM
T21
Spec (Po. ((P)) <1
All
Spsa (Po. g(Po)) < 1
All
Paraphrasing
+17.74
+10.67
+21.95
+19.22
OPT21
Llana-2
LDM-2.1
+18.63
+11.21
+25.39
+22.24
Paraphrasing
+17.52
+10.53
+21.80
+19.09
OPT2I
GPT-3.5
LDM-2.1
+18.05
+10.85
+22.58
+19.77
Paraphrasing
+18.65
+10.53
+226.54
+22.24
OPT21
Llana-2
CDN-M
+19.61
+11.07
+29.19
+24.93
B.2 Filtering out already consistent user prompts
When adopting the DSG objective, user prompts might already achieve a perfect consistency score initially,
ie., Spse (po, ((p)) 1. We observe this phenomenon happens with higher frequency the simpler
20

---

Published in Transactions on Machine Learning Research (11/2024)
** Parapphrainng
** Paraphrasing
Paraphrasing
Paraphrasing
Paraphrasing
Paraphrasing
** Llaama-2 Llama-22_S2.11 2_D2.11
@GPT-3.5_SD2.1
*#lama-2_IFF IF-M
Llama-2_SD24
GPT-3.5_5D2.3
Llama-2_IF-M
Relative consistency (%)
15 max
20 max
15
10
10
5
5
0
Relative consistency (%)
mean
mean
5
5
-5
-5
10
20
30
0
10
20
30
iteration
iteration
(a) DSG (MSCOCO)
(b) DSG (P2)
Figure 7: OPT21 optimization curves obtained with prompts having SpSG (po) < 1, marked by *** and full colors. In contrast, faded-color curves consider all prompts. Each plot tracks either the max or the mean relative improvement in consistency across revised prompts per iteration.
MSCOCO prompts (~40%), and with less frequency on the more complex PartiPrompts prompts (~10%). Since Spsc (po))(po)) can be computed beforehand, we can avoid optimizing for those user prompts that have already a perfect initial SpsG and better showcase the optimization performance OPT2I. We provide the updated optimization curves in Figure 7 and report the final results Table 10 Tn both cases, we highlight results obtained by filtering out "perfeet" user prompts with full colors, and contrast them against results obtained with all prompts in faded colors (equivalent to Figure
In Table 10 observe higher relative improvement for both MSCOCO and PartiPrompts in all configura- tions when filtering out "perfect" user prompts, which is more prominent for MSCOCO because the number of excluded prompts is higher. In Figure 7 we observe similar consistent and considerable increases of all optimization curves when considering both mean and mar consistency improvement. In the mean case, we remark reduction the initial dip in relative consistency, especially MSCOCO, where OPT2I reaches positive relative consistency much earlier, i.e., it [6.2.2] vs. it [23, 8, 5] with Llama-2, GPT-3.5, and CDM-M, respectively.
21

---

Published in Transactions on Machine Learning Research (11/2024)
OPT2I, #imgs/p=1, seed=0
OPT21, #imgs/p=1, No seed
OPT2I, #imgs/p=4, seed=0
OPT21, #imgs/p=4, No seed
OPT2I, # #imgs/p=10, seed=0
OPT21, #imgs/p=10, No seed
Relative consistency (%)
max
max
10
10
8
8
6
6
4
4
2
2
0
Relative consistency (%)
mean
mean
2
2
1
1
0
0
-1
-1
-2
-2
0
10
20
30
0
10
20
30
iteration
iteration
(a) fixed seed
(b) non-fixed seed
Figure ОРТ2I optimization curves with (a) fixed or (b) non-fixed seed. Each curve optimizes a different number of images per prompt. Y-axis is aligned between (a) and (b). Curves obtained with Llama-2 and LDM-2.1 200 out of the 2000 prompts from MSCOCO.
B.3 Impact of seed-fixing and #images/prompt
In this experiment, ablate the impact of fixing the random seed of the initial noise for the diffusion model throughout the imiizatat process when optimizing different numbers of images/prompl. We use our default configuration with Llama-2 and LDM-2. 1 on MSCOCO. In Figure Sal we show the optimization curves obtained when optimizing 1, (default), and 10 images/ pprompttotptttoom with fixed image seed. As expected, we observe no meaningful differences in mean consistency improvement. In contrast, the max consistency improvement shows clear distinction between optimizing a single image (single seed) and optimizing 10, with the former achieving more substantial improvements. We argue that when optimizing single image seed, OPT2I is more sensitive to changes in the prompts, é.e., there revised prompts. We then contrast the optimization curves with fixed seed (8a) against the non-fixed seed ones (8b). Our hypothesis is that optimizing, when not fixing the scod, generating too few images images/prompt prrompt leads to an unstable/ /uneeliablee feedback for the LLM due lo the high variance of the generations. Indeed, looking at the optimization curves, we notice that optimizing a single image without fixing the seed is more difficult for OPT2I, which results in noisy and less steop trajectory, espocially in the mean case. In contrast, when OPT2I optimizes 4 or 10 images/promp with no fixed seed, both the mar and mean curve remain similar w.r.t. using fixed seed. This supports choice of generating 4 images/
higher variance among the scores of
it provides
22

---

Published in Transactions on Machine Learning Research (11/2024)
Llama-2, LDM-2.1
GPT-3.5, LDM-2.1
Llama-2, CDM-M
14
31
13
29
12
27
Relative dCs
Relative DSG
11
25
10
23
9
21
8
19
7
17
6
15
Overall
Complex
Fine-grained Detail
Quantity
Properties &
Overall
Complex
fine-grained
Quantity
Properties & Positioning
Positioning
Detail
Figure 9: Relative improvement in comptt-imae consistency between the user prompt and the best prompt found, averaged across PartiPrompts prompts and broken down by challenge aspect.
enough diversity in the generations while being substantially more computationally efficient than generating 10.
B.4 Stratified PartiPrompts results
Figure 9 shows the relative improvement in consistency score (dCS or DSG) on prompts from PartiPrompts (P2), broken down by challenge aspect. Note that we only sampled prompls from four the most difficult dimensions in P2: "Complex", "Fine-grained Detail", "Quantity", and "Properties & Positioning". Intu- itively, this plot shows what kinds of prompts casier to optimize for OPT2I when using different LLMs, T2I models and consistency scores.
The most significant improvement in consistency is observed for prompts related to "Properties & Posi- tioning" when using Llama-2 in conjunction with CDM-M and dCS. Similarly, the combination of Llama-2, CDM-M, and DSG yields the best results for prompts about "Quantity". For other challenges, CDM-M continues provide the most substantial consistency improvement, although the margin narrower compared LDM-2.1. 1. Interestingly, GPT-3.5 shows the smallest improvement in consistency for prompts about "Quan- tity", regardless of whether dCS DGS metrics used. Consistency improvements for prompts from the "Complex" and "Fine-grained Detail" challenges comparable, which is expected due to their inherent similaritics.
B.5 Why GPT-3.5 not as good as Llama-2?
In Figure 8 and Table we observe that OPT2I achieves worse results when using GPT-3.5 as the LLM. Notably, the optimization curves with GPT-3.5 are flatter than when using Llama-2. This result is rather surprising, as current leaderboards (Chiang al. 2024 indicate that GPT-3.5 5 generally outperforms Llama-2 wide variety of NLP tasks. So, in this experiment, aim to shed light the possible causes. Given the closed (and expensive) access to GPT-3.5, was based on Llama-2, and later on we used the exact same prompt with 5. Hence, one hypothesis for the observed phenomenon is that our mela-prompt is better optimized for Llama-2. Another hypothesis is that each LLM has a different balance point between exploration and exploitation for the same sampling temperature of 1.0. In particular, given the flatter optimization curves drawn by GPT-3.5, we conjecture that it explores less diverse prompts than Llama-2. To verify this, we analyze some text properties of the revised prompts generated by both LLMss.
initial exploration of the meta-promp prompt structure and phrasing
23

---

Published in Transactions on Machine Learning Research (11/2024)
1.00
175
llama-2
0.95
gpt-3.5
length (chars)
150
similarity
0.90
125
0.85
100
0.80
75
llama-2
gpt-3.5
0.75
50
10
20
30
0
10
20
30
iterations
iterations
(a) Prompt length
(b) Text similarity w/ prompt
Figure 10: Text analysis of revised prompts generated by Llama-2 2 and GPT-3.5.
Figure 10a tracks the length (in number of characters) of the revised prompts at each iteration, and Figure 10b tracks CLIP text similarity between revised prompts and the user prompt along the optimization process, both averaged over the revised prompts generated at the same iterations and over all prompts in the dataset. We observe that when using Llama-2 for OPT2I, the revised prompts generated at each iteration are longer and more semantically dissimilar to the user prompt compared to those generated by GPT-3.5. This means that OPT2I benefits from greater prompt diversity to find the best T2I prompts that lead to more consistent images, which is better achieved with Llama-2. Additionally, we note that both the prompt length and the semantic similarity with the user prompt start platcauing around the maximum number of iterations set, which further validales our selected value of 30. We leave as future work ablating for the sampling temperature with both LLMs.
B.6 Additional qualitative examples
Figures 11 and 12 show some additional selected examples of user prompt and revised prompts throughout the optimization process, along with the generated images and consistency scores. In particular, we select revised prompts such that the consistency score of the generated images (w.r.t. the user prompt) is strictly higher than the previous best score found so far, i.e., the leaps prompt-image consistency.
Figure I1 shows revised prompts generated with DSG the scorer. Since DSG is computed an average of binary scores, it is more coarse than CLIPScore and thus there fewer lcaps in consistency. Overall, observe that the intermediate revised prompt manages to increase the consistency score in some of the generated images but not for all of them. The best prompt, however, usually manages to improve all 4 generated images.
Figure shows revised prompts generated with dCS as the scorer. In this case, we can see a gradual increase in average dCS, which visually translates generated images which are more consistent with the user prompt on average. The strong effect of the initial latent noise in the image structure is evident, yet substantial modifications in the format of the input prompt used to condition the generation lead to significant changes in how the T21 model interprets the structure determined by the initial noise (e.g., between rows 2-3 and 4-5 in the squirrel example). We also note that dCS (CLIPScoree averaged over subprompts) can occasionally fall short as an evaluation metric for image-text consistency. This is primarily because dCS tends to focus on the presence of visual elements, overlooking other aspects such as spatial relationships. In the toilet example, for instance, we observe how the generated images progressively become more consistent up a certain point (around row 6). Beyond this point, the revised prompts and the generated images start degenerating (e.g., overemphasizing certain elements), while dCS continues to improve. This highlights that, while dCS may
24

---

Published in Transactions on Machine Learning Research (11/2024)
useful fine-grained consistency metric to provide visual feedback for T2I prompt optimization with LLM, it may not. be as effective for evaluation purposes.
cute wieden sel statue holding large globe of the Earth above its tead 19 (8.3929)
There cutting boars (9.3200)
status made ueed, charning holds large Earth glate
above ats head, Lang presiaton-c tec sur (D./857)
1000
3.3553
1.335
0.3535
A outting board helds oelicious pieza. (0.0133)
8.5553
1:999
ploke STewcaolig the Earty's 's Eivense terrain, It sits pedestal, ensuring striking
rapression,
deaa cutting sotthetering sticing.
resting
2,3060
1,302
1,666
L,033
L,423
microphote
statcing on stunp (8.1820)
traffic Light and signpost crossreads Intersection near a veterwyy. (9, 30191
5,3350
resellious sevirret
traffic Light and sigrpost stanting croseraads intersection, surrounted by
holes a nacroptons. shoutatg isto the mind witt sum rach renvor.. 19287
9989
5809
8,4686
nicrmpton weleddin squi tetl, drased leather jacket sturk,
traftic light and linne intersectios, besutuful, satervay flaving searby, producáng a straking vással. (8:3167)
wind while standing epitonizing the othos and ran energy of park rock. (9.2575)
Figure 11: Selected examples of initial prompts from MSCOCO (left) and PartiPrompts (right) and revised prompts across the optimization, along with the generated images. The optimizer refines prompts for LDM-2. 1 using Llama-2 2 as LLM and DSG as scorer. We report DSG score averagod across images.
25

---

Published in Transactions on Machine Learning Research (11/2024)
park ratk stutenel In stucded leather jacket eccuting into vicrophore white
cleon wilte tollet odamndd with sisheils tid. (8.1955)
Micropions wille standing (6.1913)
punt-nackcnn nacken souirrel stards stop 1ily ped and belts inte a microphtee, wearing
pristire sorseells. (6.2341)
stuzesd jacket cozss attitods. 38581
einiature white tollet graces sosshell rovered pitestal. (8.2489)
into #scrophune with conniction, pisitiored 2top lily cod. (6.1928)
niniature tollet, flavleslyy white. odarned with ar of vibrart seastelts, stevatting chars.
cossionttely sings into nécrostons wille stonzing 1ily ouuding rick
A rebel Lous squirrel, asorned in blace Leather Jacket texturing matal studs, oudinn
white, civered array seashells, elevating Lts thara and creating Cittivating display. (6.252))
passionatlly sings intte microshone starit. 2.129111
squirrel, nicroffore with Intensity, starding 1ily rod, enbraciin the Bunk style.
featuring sings
swalt, inwocolate white tollet, covered in diverse arra of seashells, perches acdestal. eye
rebel1iz0z squtrret, attired in black testher Jacket featuring estat studs, forcefutly pus 2150)
aferred seastells, petestal, bousting untorgettable eshisitimn. 10,2575)
tity pas. cranneling
Figure 12: Selected examples of initial prompts from MSCOCO (Ieft) and PartiPrompte (right) and revised prompts across the optimization, along with the generated images. The optimizer refines prompts for LDM-2. using Llama-2 LLM and dCS as scorer. We report dCS score averaged across images.
26

---

Published in Transactions on Machine Learning Research (11/2024)
B.7 Results with updated models
In this section, report results with more recent (and better) T2T models and LLMs. We updated the T2r model to JDM-XL-Tube (Sauer et al. 2023) and LDM-3 (Esser et al. 2024) (which is based on a diffusion transformer), and the LLM to the instruction-finttund Llama-3.1 (8B and 70B) (Dubey ct al. 2024).
addition, we have upgraded the VQA model that powers the DSG metric to PaliGemma- 224 (3B) Beyer et al. 2024) finetuned on VQAv2 (Goyal et al. 2017).
Optimization curves and comparison with baselines We report the results in Figure 13 a-b) and Table 11 Our results show that, while the initial consistency score is higher, OPT2T still manages con- siderably improve consistency. For instance, optimizing PartiPrompts' prompts for 30 iterations generating 5 solutions per iteration, Llama-3.1 (70B) with LDM-3 has an initial average DSG score of 84.7%; OPT2I is able to find a rephrase of the initial prompts achieving an average DSG score of 98.8%, yielding a total absolute improvement 14%. Note that the final DSG score already very close to 100%, so there's not much more room for improvement. Meanwhile, a (compute-mattched) paraphrasing baseline using the same models only achieves an average DSG score of around 97.7%. For LDM-XL-Turbo the initial DSG score is 72.8% and OPT21 is able to find prompt rephrases with average DSG score of 93.7% (absolute improve- ment of 20.9%), while the paraphrasing baseline is only able to improve up to 91.5%. In every case, OPT2I consistently out performs the paraphrasing baseline, which is in line with our results for older models (see first two rows of Table I1 more recent T2I models.
Therefore, we conclude that our framework increases the consistency even for
Success rate In Figure 13 c), we report the success rate throughout the optimization process. At a given iteration, we define success rate as the percentage of generated paraphrascs that achieve a strictly higher consistency score than the initial prompl. Because of this, here we only consider prompls that can be improved, i.e., that do not already have an initial DSG score of 100%. We observe that, when optimizing prompts with OPT2I, the success rate surpasses 50% after 10 iterations, reaching up to 60-70% after 30 iterations. Instead, the paraphrasing baseline only achieves a score between 30-40%.
Table 11: Absolute (DSG) and relative (A DSG) improvement in T21 consistency between the user prompt and the best prompt found, averaged across all prompts in the dataset.
Method
DSG
DSG
OPT21
Paraphrasing
Llama-2 (708)
CDM-N
88.48
86.32
+22.24
244.93
Paraphrasing
+17.65
Llana-3.1 (8B)
LDM-XI-Turbo
OPT21
93.02
+00.18
Paraphrasing
91.50
+18.70
OPT21
Llama-3.1 (708)
LDM-XL-Turbo
93.68
+ 20..85
Paraphrasing
97.67
+12.99
OPT21
Llana-3.1 (708)
LDM-3
99.04
+14.10
B.8 Complete optimization example
Here we provide a detailed example list of prompt paraphrases along with their fne-grained DSG scores. We consider the user prompt "A dignified beaver wearing glasses, vest, and colorful neck tie. He stands next to tall stack of books in library." " (from PartiPromptss). We use Llama-3.1 (70B), LDM-XL-Turbo and DSG-PaliGemma.. We generale 3 new prompts per ileration, images per prompl, and we early-stop when a perfect score of 100% is reached, which takes 7 itteations and 2 minutes 12 seconds in this case. In Figures 14/ 15 we show a prompt paraphrase from each iteration (including the initial prompt at iteration 0) and its corresponding DSG score, including the evaluation questions and the average VQA scores for the 4 images.
27

---

Published in Transactions on Machine Learning Research (11/2024)
max relative consistency (%)
15.0
avg relative consistency (%)
5.0
2.5
12.5
0.0
10.0
-2.5
7.5
-5.0
5.0
-7.5
2.5
-10.0
0.0
-12.5
0
5
10
15
20
25
30
5
10
15
20
25
30
iteration
iteration
(a)
(b)
0.7
0.6
0.5
opt2i jliillmmm-3..100bbb Idm-xl-turbo turr
avg success
opt2i llama-3.1-8b 1tt.3---bbdmm-lltuubbbbb Idm-xl-turbo
0.4
opt2i llama-3.1-70b 11.--700lldm-3 Idm-3
0.3
paraphrasing Batpaesiinnlllmaa333 _ll.ma-..m1100 Jdm-jddm-l-iio -l-txbo
llama-3.1-88 Idm-xl-turbo
0.2
(nraphrasing_lamm-33112200 _dm33
0.1
0.0
0
5
10
15 iteration
20
25
30
(c)
Figure 13: OPT21 curves with updated T2T models and LLMss We report maximum (a) and average (b) DSG scores, and also average success rate (c) across the revised prompts at each iteration.
28

---

Published in Transactions on Machine Learning Research (11/2024)
dignified searing glasses, vest, and colorful neck tie. stands next to tall stack of booke in library."
overall score: 90
evaluation quastions:
Is there a beaver? 100
glassee?
Does the beavar have vest? 190
Does beaver have neck tie? 100
Is the beaver dignified? 100
Is the neck tie colorful? 100
Is the beaver standing? 75
Are there books? 100
Is there Library? 100 Is the beaver next the booke? Is the beaver the library? 100 Are the books in the library? 100
1. "A dignified beaver, vearing glasses perched the ita nose, veet, and vibrant neck tie, stande upright next tall stack of booke quiet library."
acore:
evalustion questions:
Is there a beaver? 100
glasses? 50
Does the beaver have vest? 75
beaver Is the bearer dignifind? 100
Is the nack tie colorful7 100
Is the beaver standing? 25
Are there books? 100
Is there library? 100 Is the beaver in the library? 100
next
the books the library? 100
2. "A dignified beaver, vearing glasses perched on ite nose, veat, and vibrant
upright confidence beside stack neatly arranged within arn's reach library booksbelf."
overall score: 88
evaluation questions:
Is there beaver? 100
Does the beaver have glasses? 25
Does the beaver have vest? 75
Does the heaver have neck tie? 100
dignified?
Is the neck tie colorful? 75
standing?
Are there books? 100
Are the booke Is there library? 100
tall stack? 100
Is the beaver next to the books?
poise elegance, sell-dressed high-rise bookshelf in library, wearing glassea that sit confortably on ita nose, along with neat rest and bright, oye-catching neck tic."
directly adjacent to
overall acore: 85
evaluation questions:
Is there a beaver? 100
glasses? 50 Does the beavar
Does the beaver have vest? 100
Is the beaver dignified? 100
Is the neck tie colorful? 75
Is the beaver standing? 75
Are there books? 100
books Is there a Library? 160
tall stack? 75
Is the beaver in the library? 100
Are the booke the library? 100
Figure Example of complete optimization. Part 1/2
29

---

Published in Transactions on Machine Learning Research (11/2024)
4. "In library, rafined beaver stands proudly adjacent to tall, orderly stack of books, searing a pair of glasses, veat, colerful neck tie, exuding sir of dignity."
overall score: 94
evaluation questions:
Is there beaver? 100
Does the beaver have glasses? 50
vest?
Doee the beaver have neck tie? 100
dignified?
Is the nack tie colorful? 100
the Are there books? 100 Are the booka in tall stack? 100
standing?
Library?
Is the beaver next to the booke? 100
library?
Are the books in the library? 100
"In library, refined beaver stands confidently its hind legs, showcaring its dignified demeanor, shile wearing glasses that sit confortably the bridge its rest, and colorful nack tovering stack of books."
overall acore: 94
evaluation questions:
Is there a beaver? 100
glasses?
Does the beaver have vest? 100
beavar Is the beaver dignified? 100
Is the neck tie colorful? 100
Is the beaver standing? 100
Are there books? 100
stack?
Is there library? 100 Is the beaver next Is the beaver the library? 100
books?
Are the books the library? 100
pair epectacles resting confortably on its nase, stands upright besida tovering stack of books in library, dresaed vest and vibrant neck tie, exuding an air of refinement.""
dignified beaver,
overall score:
eralustion questions:
Is there beaver? 100
Does the beaver have glasses? 50
Does the heaver have vest? 100
Doea
Is the beaver dignified? 100
tie colorful? 100
Is the beaver standing? 75
Are there books? 100
the books Is there a library? 100 the Is the beaver in the library? 100
tall stack? 100
booke? 100
library?
dignified beaver, sporting pair vire-rinned glasses lensea that sparkle the 1ibrary light, etande proudly next to tovering stack books, adorned vith colorful neck tie and a vest that complenents its sir of sophistication."
overall acore: 100
evaluation questions:
there beaver?
Does the beaver have glasses? 100
beaver
Does the beaver have neck tie? 100
dignified?
Is the neck tie colorful? 100
Is the beaver standing? 100
100
Are the booke in tall stack? 100
library?
Is the beazer next to the books? 100
Is the beaver the library?
the booke the library? 100
Figure 15: Example of complete optimization. Part 2/2
30


Reviews:
[{"review_id": "hoYPETTNho", "reviewer": "Reviewer_3vv4", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n+ The idea of improving prompt-image consistency is well-motivated, where the proposed iterative optimization is reasonable for this scenario.\n+ They provide comprehensive ablation studies and qualitative visualization, as well as the used prompts in the system, which is valuable to future research in this community.\n\n**Weaknesses**\n- The idea of prompt paraphrasing to improve T2I is not new. There are already several works [1, 2] studying this idea, where [3] also investigate the iterative paraphrasing.\n- Though Figure 2 can show the overview of OPT2I, it still needs to provide a detailed process for the entire pipeline—for example, a complete example list of prompt paraphrasings with the related DSG scores.\n- According to Figure 10, with more iterations, the prompts are getting longer but less similar to the original ones. Will the leveraged LLM produce hallucinations as well? How will it hurt the performance? How can such error propagations to the end T2I be alleviated?\n- According to Figure 7, more iterations can keep leading to better final consistency. Will it be converged after even more iterations? Or the performance may drop with too excessive iterations.\n\n**Reference**\n- [1] Improving Image Generation with Better Captions\n- [2] (NeurIPS'23) Optimizing Prompts for Text-to-Image Generation\n- [3] (EMNLP'23) Collaborative Generative AI: Integrating GPT-k for Efficient Editing in Text-to-Image Generation", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper leverages an external Larage Language Model (LLM) to provide prompt-image consistency feedback and paraphrase the input prompt to improve the synthesized image from the target text-to-image model (T2I). Through this iterative optimization, the proposed OPT2I can significantly boost the consistency score yet maintain a competitive FID score.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Add a discussion to compare the differences with previous works on this similar prompt paraphrasing method.\n2. Add a discussion of prompt paraphrasing examples and their quality.\n3. Add experiments of more/excessive paraphrasing iterations.", "broader_impact_concerns": "As the used LLMs may produce hallucinations when paraphrasing the prompt, this error may propagate to the final T2I result.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "bBKKNUEiXL", "reviewer": "Reviewer_GmPJ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1: The introduced OPT2I is a training-free framework, making it easy to transfer to other advanced text-to-image models and LLMs.\n\n2: The newly proposed metric is effective in evaluating the consistency between input text and generated images.\n\n3: Text-based in-context learning for improving consistency is interesting and heavily reduces the trivial attempts on prompt design, which will be generally used practically.\n\nWeaknesses:\n\n1: The rationality of decomposed CLIPScore (dCS) to measure the consistency. As the shown examples in this paper, the generated images always contain **multiple objects** and the CLIPScore is originally used to measure the global or overall similarity between text and image. Therefore, with the decomposed text like \"a bike,\" if we compute the similarity between this decomposed text and the whole image, can it reflect the consistency used for meta-prompt construction? If so, could the authors provide more illustrations?\n\n2: Experiments with more text-to-image generation models. Besides the used latent diffusion model (LDM), will the proposed framework be in other generation architectures such as text-to-image diffusion transformers or API-based DALLE?\n\n3: The question design/generation in DSG or potential measurement limitations in the current consistency metric. From the 3rd image in the 4th row of Figure 4(d), the text contains the description \"**four** teacups,\" and the generated images may have **five** or **three**, which means the inconsistency still exists. Meanwhile, the consistency scores are 1.0. Could the authors provide more analysis of similar situations or analyze the current limitations of the proposed metric for consistency evaluation? \n\n4: (Not a weakness but more like a suggestion or discussion) More detailed analysis of potential sub-metrics for different failure modes. The authors emphasize several failure modes in the Introduction, such as missing objects, the wrong amount of objects, object attribute errors, and non-compliance. I haven't seen the metrics for each mentioned mode to evaluate the text-to-image generation. Can the proposed framework obtain apparent performance gain on these specific modes (with corresponding metrics)? \n\nMinor:\n\nThe sentence \"a plethora of\" in the abstract. This sounds like it has an excessive quantity.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors propose OPT2I, a training-free optimization-by-prompting framework designed to enhance prompt-image consistency in Text-to-Image (T2I) models. The proposed method refines prompts without requiring any parameter updates, making it versatile and compatible with various T2I models, Large Language Models (LLMs), and consistency metrics. By leveraging a pre-trained T2I model, an LLM, and an automatic prompt-image consistency score (e.g., CLIPScore or DSG), OPT2I iteratively improves user-provided prompts by suggesting alternatives that lead to more consistent images. Extensive experiments demonstrate the effectiveness of OPT2I on the MSCOCO dataset and 24.9% on the PartiPrompts while preserving the Fréchet Inception Distance (FID).", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "More analysis of the DSG and potential limitations of the current framework is necessary. Wether the dCS is rational remains unclear. Additional experiments on more text-to-image models can strengthen the paper.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "61yLYwBvl9", "reviewer": "Reviewer_X6VC", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n- The method is training-free and independent of specific T2I images.\n- The method is well-motivated and described in detail (e.g., Sec.2.3. Optimization objective)\n- Extensive quantitative and qualitative results are provided to support the claims.\n\n\n### Weaknesses\n- The DSG questions and dCS decompositions are also generated by the LLM. This may lead to a situation: if the LLM does not notice or care about some points of the generation's requirement, it will always be ignored in the following optimization processes.\n- The \"optimization\" of the desired metrics is not tailored. It depends on the decision of LLM and luck to achieve successful editing.\n- The tested T2I models cannot perfectly support the claims.\n    - The T2I models are not the state-of-the-art at this point. For example, the most powerful version of Stable Diffusion is SDXL or SD3, which has a larger capacity and more powerful text encoders\n    - On the other hand, only considering state-of-the-art T2I models may lead to incomplete conclusions. For example, a weaker model like SD1.5 may have a lower ability to reason the text prompts and, therefore, a more challenging process to optimize through the proposed framework.\n    - (Minor) The LLMs are also not state-of-the-art. This is not a problem since a weaker LLM can better support the claims.\n- (Minor) The proposed model has some efficiency issues.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies text-to-image generative models. Aiming to improve prompt-image consistency, they propose a novel framework, OPT2I, using LLM for such improvement. The method is training-free and expected to be generalizable to any T2I models. The authors claimed that the experiment results show the effectiveness of the proposed method.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- I would like to see the successful rate and/or standard variance of the quantitative results to understand how robust and likely the LLM can improve the results.\n- If possible, I would like to see the results on more T2I models, e.g., SD1.5, SDXL, SD3.\n- Some non-T2I generative models, like Instruct-Pix2Pix, which is an image editing model and is IT2I (image+text to image), can also be regarded as T2I when the image is fixed.\n    - i.e., the original model can be defined as$ f(I, t) \\to I'$; for a given image $I_1$, let $g(t) = f(I_1, t)$, then $g(t)$ can be regarded as a T2I model.\n    - I wonder if the proposed model can also work in this scenario to optimize the best editing prompt. Successfully achieving this will make the model more general and powerful.", "broader_impact_concerns": "There are no such concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces a new training-free optimization-by-prompting framework designed to enhance prompt-image consistency in Text-to-Image (T2I) models. After author rebuttal, it received 3 Accept recommendations. All the reviewers are happy about the paper, commenting that (1) the model is well-motivated and its performance is verified by extensive experiments; (2) the paper is high quality, and the reply mostly addresses the reviewers' concerns; (3) the pipeline is elegant, easy to follow  and has the potential to benefit general visual generation. Therefore, the Action Editor would also like to recommend acceptance of the paper.
Decision: Accept (poster)