Venue: NAACL 2025

# DreamSync: Aligning Text-to-Image Generation with

Image Understanding Feedback

Jiao Sun\({}^{*\dagger\alpha}\) &Deqing Fu\({}^{*\beta\psi}\)Yushi Hu\({}^{*\beta\xi}\)Su Wang\({}^{\beta}\)Royi Rassin\({}^{\delta}\)Da-Cheng Juan\({}^{\beta}\)Dana Alon\({}^{\alpha}\)Charles Herrmann\({}^{\alpha}\)Sjoerd van Steenkiste\({}^{\beta}\)Ranjay Krishna\({}^{\epsilon}\)Cyrus Rashtchian\({}^{\beta}\)Google DeepMind\({}^{\beta}\)Google

\({}^{\omega}\)University of Southern California University of Washington Bar-Ilan University Jiaosun@google.com, deqingfu@usc.edu, yushihu@uw.edu

###### Abstract

Despite their widespread success, Text-to-Image models (T2I) still struggle to produce images that are both aesthetically pleasing and faithful to the user's input text. We introduce **DreamSync**, a simple yet effective training algorithm that improves T2I models to be faithful to the text input. DreamSync utilizes large vision-language models (VLMs) to effectively identify the fine-grained discrepancies between generated images and the text inputs and enable T2I models to self-improve without labeled data. First, it prompts the model to generate several candidate images for a given input text. Then, it uses two VLMs to select the best generation: a Visual Question Answering model that measures the alignment of generated images to the text, and another that measures the generation' aesthetic quality. After selection, we use LoRA to iteratively finetune the T2I model to guide its generation towards the selected best generations. DreamSync does not need any additional human annotation, model architecture changes, or reinforcement learning. Despite its simplicity, DreamSync improves both the semantic alignment and aesthetic appeal of two diffusion-based T2I models, evidenced by multiple benchmarks (+1.7% on TIFA, +2.9% on DSG1K, +3.4% on VLA aesthetic) and human evaluation shows that DreamSync improves text rendering compared to SDXL by 18.5% on DSG1K benchmark.

## 1 Introduction

Although we invite creative liberty when we commission art, we expect an artist to follow our instructions. Despite the advances in text-to-image (T2I) generation models [6, 10, 12, 13], it remains challenging to obtain images that meticulously conform to users' intentions [16, 17, 18, 19, 20, 21, 22]. Current models often fail to compose multiple objects [16, 17, 18, 19]. In fact, the difficulty of finding effective textual prompts has led to a myriad of websites and forums dedicated to collecting and sharing useful prompts (e.g. PromptHero, Arthub.ai, Reddit/StableDiffusion). There are also online marketplaces for purchasing and selling useful such commands (e.g. PromptBase). The ouns to generate aesthetic images that are faithful to a user's desires should lie with the model and _not_ the user.

Today, there are efforts to address these challenges. For example, it is possible to manipulate attention maps based on linguistic structure to improve attribute-object binding [17, 18]; or train reward models using human feedback to better align generations with user intent [14, 15]. Unfortunately, these methods either operate on a specific model architecture [17, 18] or require expensive labeled human data [14, 15]. Worse, most of these methods sacrifice aesthetic appeal when optimizing for faithfulness, which we confirm in our experiments.

We introduce **DreamSync, a simple yet effective framework that improves T2I generation faithfulness while maintaining aesthetic appeal**. Our approach extends work on fine-tuning T2I models for alignment, but does not require any human feedback. The key insight behind DreamSync is in leveraging the advances in vision-language models (VLMs), which can identify fine-grained discrepancies between the generated image and the user's input text [19, 20]. Intuitively at a high level, our method can be thought of as a scalable version of reinforcement

learning with human feedback (RLHF); just as LLaMA2Touvron et al. (2023) was iteratively refined using human feedback, DreamSync improves T2I models using feedback from VLMs, except without the need for reinforcement learning.

Given a set of textual prompts, T2I models first generate multiple candidate images per prompt. DreamSync automatically evaluates these generated images using two VLMs. The first one measures the generation's faithfulness to the text Hu et al. (2023); Cho et al. (2023), while the second one measures aesthetic quality Ke et al. (2023). The best generations are collected and used to finetune the T2I model using parameter-efficient LoRA fine-tuning Hu et al. (2022). With the new finetuned T2I model, we repeat the entire process for multiple iterations: generate images, curate a new finetuning set, and finetune again.

We conduct extensive experiments with latest benchmarks and human evaluation. We experiment DreamSync with two T2I models, SDXL Podell et al. (2023) and SD v1.4 Radford et al. (2021). Results on both models show that **DreamSync enhances the alignment of images to user inputs and retains their aesthetic quality.** Specifically, human evaluation demonstrates that DreamSync yield consistent and statistically significant improvements in all categories of text-to-image faithfulness on SDXL. Quantitative results on FIFA Huang et al. (2023), DSG Cho et al. (2023), and ImageReward Xu et al. (2023) demonstrate that **DreamSync is more effective than all baseline alignment methods on SD v1.4, and can yield even bigger improvements on SDXL.** While our study primarily focuses on boosting faithfulness and aesthetic quality, DreamSync has broader applications: it can be used to improve other characteristics of an image as long as there is a model that can measure that characteristic.

## 2 Related Work

**T2I Evaluation with VLMs.** Several prior works have proposed to use VQA models to evaluate text-to-image generation. The TIFA benchmark, which pioneered this approach for evaluation, consists of 4K prompts and 25K questions across 12 categories (e.g., object, count, material), enabling T2I model evaluation by using VQA models to answer questions about the generated images Hu et al. (2023). TIFA prompts come from various resources, including DrawBench used in Imagen Saharia et al. (2022), PartiPrompt used in Parti Yu et al. (2022), PaintSkill Cho et al. (2022) used in Dall-Eval, etc. DSG Cho et al. (2023) further improves TIFA's realiability by examining their evaluation questions

Figure 1: **DreamSync**. Given a prompt, a text-to-image generation model generates multiple candidate images, which are evaluated by two VLM models: one VQA model that provides feedback on text faithfulness and the other on image aesthetics. The best image chosen by the VLMs are collected to fine tune the T2I model. This process can repeat indefinitely until convergence on feedback is achieved.

 

carefully. Another related benchmark is SeeTrue, which also uses VQA models to measure alignment (Yarom et al., 2023). Before the VQA evaluation era, several other evaluation benchmarks were proposed focusing primarily on compositional text prompts for attribute binding (e.g., color, texture, shape) and object relationships (e.g., spatial). Examples include T2I-CompBench (Huang et al., 2023), C-Flowers (Park et al., 2021), CC-500 and ABC-6K benchmarks (Feng et al., 2023). Aside from automated benchmarks, human evaluation for text-to-image generation is widely used in the community, although such annotations are notoriously costly to collect. In response, (Xu et al., 2023) propose ImageReward, the first general purpose text-to-image human preference reward model to encode human preferences automatically. In our work, we use a collection of three evaluation methods to evaluate DreamSync: VQA evaluation for generated images on both TIFA and DSG benchmarks, human evaluation, and ImageNet for automatic human preference prediction.

Improving General T2I AlignmentWe roughly categorize the alignment methods for improving T2I alignment into two classes depending on if they involve training. For training-involved methods, several works use Reinforcement Learning from Human Feedback (RLHF) based on human rankings to maximize a reward and improve faithful generation Fan et al. (2023); Karthik et al. (2023); Lee et al. (2023). In a similar vein, Pick-a-Pic is a dataset of prompts and preferences that is used to train a CLIP-based scoring function (Kirstain et al., 2023). StyleDrop trains adapters to synthesize of images that follow a specific style (Sohn et al., 2023), and T2I-Adapter trains adapters to improve the control for the color and structure of the generation results (Mou et al., 2023). DreamBooth and HyperDreamBooth improve personalized generation Ruiz et al. (2023); b), and they have inspired more efficient methods such as SVDiff Han et al. (2023). Being orthogonal to training-involved methods, there is a body of work on training-free methods that make inference time adjustments to the model to improve alignment, such as SynGen and StructureDiffusion Hong et al. (2022); Epstein et al. (2023); Feng et al. (2023); Rassin et al. (2023); Chefer et al. (2023); Agarwal et al. (2023). DreamSync leverages training but does not involve reinforcement learning. We compare DreamSync with two RL-based methods and two learning-free methods in our experiments. We find that DreamSync outperform all the baselines in terms of text-image alignment on both DSG and TIFA.

Iterative BootstrappingIterative Bootstrapping, also known as model self-training, is a semi-supervised learning approach that utilizes a teacher model to assign labels to unlabelled data, which is then used to train a student model Yarowsky (1995); McClosky et al. (2006); Kumar et al. (2020); Fu et al. (2023). In our work, we adopt a self-training scheme where the teacher model are the VLMs and the student model is the T2I model we aim to improve. During training, the VLMs (teacher) are used to annotate and select aligned examples for the next batch finetuning (student).

## 3 DreamSync

Our method improves alignment and aesthetics in four steps (see Figure 1): Sample, Evaluate, Filter, and Finetune. The high-level idea is that T2I models are capable of generating interesting and varied samples. These examples are further judged by VLMs to pass qualification as faithful and aesthetic candidates for further finetuning T2I models. We

Figure 2: Qualitative examples of DreamSync improving image-text alignment after each iteration. LoRA fine-tuning on generated and filtered prompt-image pairs can steer the model to gradually capture more components of the text inputs.

 

next dive into each component more formally.

**Sample.** Given a text prompt \(T\), the text-to-image generation model \(G\) generates an image \(I=G(T)\). Generation models are randomized, and running \(G\) multiple times on the same prompt \(T\) can produce different images, which we index as \(\{I^{(k)}\}_{k=1}^{K}\). To improve the model's faithfulness to text guidance, our method collects faithful examples generated by \(G\). We use \(G\) to generate \(K\) samples of the same prompt \(T\), so that with some probability \(\delta>0\), a generated image \(I\) is faithful. Note that we need \(K=\Omega(1/\delta)\) samples for each prompt \(T\), and DreamSync is not expected to improve totally unaligned models (with \(\delta\to 0\)). Prior work [23] estimates that 5-10 samples can yield a good image, and hence, \(\delta\) can be thought of as roughly 0.1 to 0.2.

**Evaluate.** For each text prompt \(T\), we derive a set of \(\mathcal{N}_{T}\) question-answer pairs \(\{\mathcal{Q}(T),\mathcal{A}(T)\}\) that can be used to test whether a generated image \(I\) is faithful to \(T\). We use an LLM to generate these pairs, only using the prompt \(T\) as input (with no images). Typically \(\mathcal{N}_{T}\approx 10\). We use VQA models to evaluate the faithfulness of the generation model, \(F_{j}(T,I)=\mathbbm{1}\{\text{VQA}(I,\mathcal{Q}_{j}(T))=\mathcal{A}_{j}(T)\}\), for \(j\in\{1,\ldots,\mathcal{N}_{T}\}\). We measure the faithfulness of a caption-image pair \((T,I)\) given all questions and answers, using two metrics. Intuitively, we can average the number of correct answers, or we can be more strict, and only count an image as a success if all the answers are correct. Formally, the _Mean_ score is the expected success rate

\[\mathcal{S}_{\mathrm{M}}(T,I)=\frac{1}{\mathcal{N}_{T}}\sum_{j=1}^{\mathcal{ N}_{T}}F_{j}(T,I),\]

and the _Absolute_ score is the absolute success rate

\[\mathcal{S}_{\mathrm{A}}(T,I)=\prod_{j=1}^{\mathcal{N}_{T}}F_{j}(T,I).\]

Filter.We combine text faithfulness and visual appeal (given by \(\mathcal{V}(\cdot)\)) as rewards for filtering. For a text prompt \(T\) and its corresponding synthetic image set \(\{I_{k}\}_{k=1}^{K}\), we select samples that pass both VQA and aesthetic filters:

\[C(T)=\{(T,I_{k}): \mathcal{S}_{\mathrm{M}}(T,I_{k})\geq\theta_{\mathrm{Faithful}},\] \[\mathcal{V}(I_{k})\geq\theta_{\mathrm{Aesthetic}}\}.\]

To avoid an imbalanced distribution where easy prompts have more samples, which could cause adversely affected image quality, we select one representative image (denoted as \(\hat{I}_{T}\)) having the highest visual appeal for each \(T\):

\[(T,\hat{I}_{T})=\operatorname*{argmax}_{\mathcal{V}(I_{k})}C(T).\]

We apply this procedure to all text prompts in our finetuning prompt set \(\{T_{i}\}_{i=1}^{N}\) with \(T_{i}\sim\mathcal{D}\), where \(\mathcal{D}\) is a prompt distribution. After filtering, we collect a subset of examples, \(D(G):=\bigcup_{i\in\{j|C(T_{j})\neq\varnothing\}}\{(T_{i},\hat{I}_{T_{i}})\}\), that meet our aesthetic and faithfulness criteria. Note that it is possible for \(C(T_{i})\) to be empty, and we empirically show what fraction of the training data is selected in Figure 5. We ablate other aspects of the selection procedure in SS 5.5.

**Finetune.** After obtaining a new subset of faithful and aesthetic text-image pairs, we fine-tune our generative model \(G\) on this set. We denote the generative model after \(s\) iterations of DreamSync as \(G_{s}\), such that \(G_{0}\) denotes the baseline model. To obtain \(G_{s+1}\) we fine-tune on data generated by \(G_{s}\) after applying our filtering procedure as outlined above. We follow the same loss objective and fine-tuning dynamics as LoRA [46]. Let \(\Theta(\cdot)\) denote all parameters of a model, then the hypothesis class at iteration \(s\) is:

\[\mathcal{G}_{s}=\left\{G\mid\operatorname{rank}\Bigl{(}\Theta(G)-\Theta(G_{ s})\Bigr{)}\leq R\right\}.\]

where \(R\) denotes the rank of weight updates and in practice we choose \(R=128\) to balance efficiency and image quality. Overall, the iterative training procedure is as follows:

\[G_{s+1}=\operatorname*{argmin}_{G\in\mathcal{G}_{s}}\frac{1}{|D(G_{s})|}\sum_ {(T_{j},f_{j})\in D(G_{s})}\ell(G(T_{j}),I_{j}). \tag{1}\]

The self-training process eq. (1) can in principle be executed indefinitely. In practice, it repeats for three iterations at which point we observe diminishing returns.

## 4 Generating Diverse Prompts with Llms

The success of DreamSync is dependent on the diversity of the prompts used during self-training. We need a large amount of diverse training prompts to ensure that DreamSync models have good generalization ability, rather than overfitting to a few entities. To obtain such prompts and corresponding question-answer pairs without human-in-the-loop, 

we utilize Large Language Models (LLM) to create our prompt source. We choose PaLM 2Anil et al. (2023) and proceed as follows:

1. _Prompt Generation._ We provide five handcrafted seed prompts as examples and then ask PaLM 2 to generate similar textual prompts. We include additional instructions that specify the prompt length, a category (randomly drawn from twelve desired categories as in Hu et al. (2023), e.g., spatial, counting, food, animal/human, activity), no repetition, etc.1 We change the seed prompts and repeat the prompt generation three times. Footnote 1: In Appendix A.1, we show the complete instruction used to probe LLM for prompt generation and QA generation.
2. _QA Generation._ Given prompts, we then use PaLM 2 to generate question and answer pairs that we will use as input for VQA models as in TIFA Hu et al. (2023). See the example in Table 4.
3. _Filtering._ We finally use PaLM 2 once more to filter out unanswerable QA pairs. Here our instruction aims to identify three scenarios: the question has multiple answers (e.g., "black and white panda" where the object has multiple colors, each color could be the answer), the answer is ambiguous (e.g., "a lot of people") or the answer is not valid to the question.

We showcase the diversity of PaLM 2 generated prompts in Figure 3 using qualitative examples and quantitive statistics of our generated prompts in Appendix A.2. We use this set of generated prompts for all our experiments.

## 5 Experiments

In this section, we conduct a series of experiments, with different evaluation strategies, to measure the effectiveness of DreamSync. We explain our training setup in SS5.1. We conduct a human study with over 1000 images in SS5.2 to understand the performance of DreamSync compared to the most state-of-the-art open-sourced model, SDXL. Then, we report the results with automatic evaluation metrics and compare DreamSync against other methods in SS5.3. We also evaluate our models with reward models trained with human preferences (SS5.4). Finally, we conduct analyses on the training process and the effect of choosing different VQA models in SS5.5.

### Training set-up

**Fine-grained VLM Feedback.** We use feedback from two VLM models to decide what text-image pairs to keep for finetuning. We use BLIP-2 Li et al. (2023) as the VQA model to measure the faithfulness of generated images to textual input and and VIIA Ke et al. (2023) to measure the aesthetics measurement score. Empirically, we keep the text-image pairs whose VQA scores are greater than \(\theta_{\mathrm{Faithful}}=0.9\) and aesthetics score greater than \(\theta_{\mathrm{Acsthatics}}=0.6\). If there are multiple generated images passing the threshold, we keep the one with the highest VILA score. Starting from 28,250 prompts, we find that more than 25% prompts are kept for \(D(G_{0})\) (for both T2I models), which we will use for finetuning. We later show that this percentage increases further as we perform additional DreamSync iterations.

**Baselines.** We evaluate DreamSync with both Stable Diffusion v1.4 Radford et al. (2021) and SDXL as bakcbones, because most related works build on SD v1.4 and SDXL Podell et al. (2023) is the current state-of-the-art open-sourced T2I model. For each prompt, we generate eight images per prompt, i.e., \(K=8\). We compare DreamSync with two types of methods that improve the faithfulness of T2I models: two training-free methods (StructureDiffusion Feng et al. (2023) and SynGen Rassin et al. (2023) and two RL-based methods (DPOK Fan et al. (2023) and DDPO Black et al. (2023)). As the baselines use SD v1.4 as their backbone, we also use it with DreamSync for a fair comparison.

Figure 3: PaLM-2 generated training prompts and their corresponding images generated via DreamSync. Prompt acquisition requires no human effort. It enables us to train on more complex and diversified prompt-image pairs than found in typical datasets.

 



### Human Evaluation

We conduct a large-scale human study to understand the capability of DreamSync, where we use the full set of 1,060 images in DSG Cho et al. (2023). During the study, we ask approximately 8 fine-grained questions for each of 1060 images to external raters. These questions are divided into categories (entity, attribute, relation, global). In Figure 4, we observe consistent and statistically significant improvements comparing DreamSync to SDXL. The greatest gain (+18.52%) comes from the text rendering category. The details and findings of both studies are in Appendix C.

### Automatic Evaluation with VLMs

Faithfulness evaluation setupsThe most recent text-to-image faithfulness evaluations are based on VLMs. We choose two representative benchmarks for automatic evaluation: (1) **TIFA.** To evaluate the faithfulness of the generated images to the textual input, TIFA Hu et al. (2023) uses VQA models to check whether, given a generated image, questions about its content are answered correctly. There are 4k diverse prompts and 25k questions spread across 12 categories in the TIFA benchmark. Although there is no overlap between our training data and TIFA, we use the TIFA attributes to constrain our LLM-based prompt generation. Therefore, we use TIFA to test DreamSync on in-distribution prompts. We follow TIFA and use BLIP-2 as the VQA model for evaluation. (2) **Davidsonian Scene Graph (DSG).** DSG cho et al. (2023) exhibits the same VQA-as-evaluator insight as TIFA's and further improves its reliability. Specifically, DSG ensures that all questions are atomic, distinct, unambiguous, and valid. To comprehensively evaluate T2I images, DSG provides 1,060 prompts covering many concepts and writing styles from different datasets that are completely independent of DreamSync's training data acquisition stage. Not only is DSG a strong T2I benchmark, but it also enables further analysis of DreamSync with out-of-distribution prompts. Furthermore, DSG uses PaLI as the VQA model for evaluation, which is different from the VQA model that we use in training (_i.e._, BLIP-2) and lifts the concern of VQA model bias in evaluation. We use DSG QA both automatically (with PaLI) and with human raters (details in Appendix C).

Out-of-domain prompts and different VQA models for evaluationWe carefully design our evaluation to test the generalizability of DreamSync. While our training prompts, generated by PaLM-2, are mainly single sentences, our testing set DSG contains paragraphs, real user prompts, and commonsense-defying prompts. Thus, it is out-of-domain compared with training prompts. To avoid "overfitting" to one VQA model, we use PaLI Chen et al. (2022) for DSG evaluation, which is a different VQA model compared to the BLIP-2 model used for training.

DreamSync improves the alignment of both SDXL and SD v1.4In Table 1 we compare DreamSync to various state-of-the-art approaches with four random seeds. For SDXL Podell et al. (2023), we show how three iterations of DreamSync improve the generation faithfulness by 1.7 points of the mean score and 3.7 points of the absolute score on TIFA. The visual aesthetic scores after performing DreamSync improved by 3.4 points. It is always straightforward to apply DreamSync to different T2I models. We also apply DreamSync to SD V1.4 Radford et al. (2021). DreamSync improves faithfulness by 1.0 points of mean score and 1.7 points of absolute score on TIFA. Most prominently on DSG1K, DreamSync improve text faithfulness of SDXL by 2.9 points. We report fine-grained results for DSG in Appendix C. In Appendix D we show more qualitative comparisons.

DreamSync yields the best text-to-image faithfulness while not sacrificing aestheticsSome prior works report that they observed some degradation of visual aesthetics after applying their approaches to improve text-to-image faithfulness. Feng et al. (2023); Rassin et al. (2023) To quantitatively measure the visual appeal of synthesized images, we use VILA Ke et al. (2023) to

Figure 4: **Human study on 1060 DSG prompts.** We ensure each image is rated by 3 human raters. Altogether we have 25 human annotators and they are distributed to rate different images. We show that DreamSync improves SDXL across **all** categories on DSG, with the greatest gain 18.52% for text rendering.



compute the average aesthetic scores for all the images generated from TIFA and DSG prompts. The results are in Table 1. DreamSync improves VILA score by 0.3 on SDv1.4, and 3.4 on SDXL. The results demonstrate that DreamSync, while yielding the best text-to-image faithfulness, has not sacrificed the aesthetics. We highlight several qualitative examples in Figure 2 for DreamSync applied on SDXL and qualitative examples in Figure 7 for DreamSync applied on SD v1.4.

### Evaluation with Human-Preference Reward Models

We next test whether DreamSync yields an improvement on human preference reward models, even though DreamSync is _not_ trained to optimize them. We use ImageReward Xu et al. (2023) as an off-the-shelf human preference model for generated images. ImageReward is trained with binary human preference data in Pick-a-pic Kirstain et al. (2023) and is very different from our VLM evaluation scheme. Table 2 shows that DreamSync plus either SD v1.4 or SDXL increases ImageNetward scores on images based on both TIFA and DSG1K. Tuning with VLM-based feedback helps align the generated images with human preferences, at least according to ImageReward.

### Analysis & Ablations

**Evolution of faithfulness and aesthetics through each DreamSync iteration.** In Figure 5 we report TIFA and aesthetics scores for each iteration, where we observe how DreamSync gradually improves the alignment and aesthetics of the generated images. Also, more images pass through the filter for each iteration. One interesting question to ask is what will happen if the base generation model is really weak. As we observe from the SDv1.4 experiments, compared with SDXL, we have to lower the filtering threshold and sample more images per prompt, since the filter passing rate is too low. Nevertheless, we can still observe increases in faithfulness and aesthetics scores per iteration, similar to that of Figure 5.

**Impact of VQA model on evaluation.** We ann

\begin{table}
\begin{tabular}{l l c c c c c} \hline \multirow{2}{*}{**Model**} & \multirow{2}{*}{**Alignment**} & \multicolumn{6}{c}{**Text Faithfulness**} \\ \cline{3-8}  & & & TIFA & & DSGIK & \\ \cline{3-8}  & & & _Mean_ & _Absolute_ & & \\ \hline \multirow{9}{*}{SD v1.4 Radford et al. (2021)} & \multirow{3}{*}{Training-Free} & No alignment & 76.6 & 33.6 & 72.0 & 44.6 \\  & & SynGen Rassin et al. (2023) & 76.8 (+0.2) & 34.1 (+0.5) & 71.2 (-0.8) & 42.4 (-2.2) \\  & & StructureDiffusion Feng et al. (2023) & 76.5 (–0.1) & 33.6 (+0.0) & 71.9 (-0.1) & 41.5 (-3.1) \\ \cline{2-8}  & \multirow{3}{*}{RL} & DPOK Fan et al. (2023) & 76.4 (–0.2) & 33.8 (+0.2) & 70.3 (-1.7) & **46.5** (+1.9) \\  & & DDPO Black et al. (2023) & 76.7 (+0.1) & 34.4 (+0.8) & 70.0 (-2.0) & 43.5 (-1.1) \\ \cline{2-8}  & & \multirow{3}{*}{DreamSync (ours)} & **77.6** (+1.0) & **35.3** (+1.7) & **73.2** (+1.2) & 44.9 (+0.3) \\ \cline{2-8}  & & No alignment & 83.5 & 45.5 & 83.4 & 60.9 \\ \cline{2-8}  & & \multirow{3}{*}{DreamSync (ours)} & **85.2** (+1.7) & **49.2** (+3.7) & **86.3** (+2.9) & **64.3** (+3.4) \\ \hline \end{tabular}
\end{table}
Table 1: **Benchmark on Text Faithfulness and Visual Appeal. All models are sampled with the same set of four seeds, i.e. \(K=4\). Best scores under each backbone T2I model are highlighted in bold; gain and loss compared to base models are highlighted accordingly. DreamSync significantly improve SD-XL and SD v1.4 in alignment and visual appeal across all benchmark. Additionally, DreamSync does not sacrifice image quality when improving faithfulness.**

Figure 5: DreamSync improves faithfulness and aesthetics iteratively. More examples pass the filters with additional iterations.

 

alyze whether using BLIP-2 as a VQA model for finetuning and for evaluation in TIFA might be the reason for the improvement by DreamSync that we have observed. To test this we use PaLI Chen et al. (2022) to replace the BLIP-2 as the VQA in TIFA. Using SDXL as the backbone, DreamSync improves the mean score from 90.09 to 92.02 on TIFA compared to the vanilla SDXL model. These results confirm that DreamSync is in fact able to improve the textual faithfulness of T2I models.

**Albating the Reward Models** In Table 3, we present the results for an ablation study where we remove one of the VLMs during filtering and evaluate SDXL after applying one iteration of DreamSync. It can be seen how training with a single pillar mainly leads to an improvement in the corresponding metric, while the combination of the two VLM models leads to strong performance for both text faithfulness and visual aesthetics, justifying our approach. One interesting finding is that training with both rewards, rather than VLA only, gives the highest visual appeal score. Our possible explanation is that images that align with user inputs may have higher visual appeal.

## 6 Discussion

A key design choice behind DreamSync is to maintain simplicity and automation throughout each step of the pipeline. Despite this feature, our experimental results show that DreamSync can improve both SD v1.4 and SDXL on TIFA, DSG, and visual appeal. In the case of SD v1.4, this improvement holds true compared to four different baseline models (two training-free and two RL-based). For SDXL, even though the base model achieves SoTA among open-source models, DreamSync can still substantially improve alignment and aesthetics.

The effectiveness of DreamSync's self-training methodology opens the door for a new paradigm of parameter-efficient finetuning. Indeed, the DreamSync pipeline is easily generalizable. For the training prompts, we can construct a set with complex and non-conventional examples compared to standard web-scraped data. On the filtering and fine-tuning side, our framework shows that VLMs can provide effective feedback for T2I models. Together, these steps do not require human annotations, yet they can tailor a generative model toward desirable criteria.

### Limitations

Like prior methods, the performance of DreamSync is limited by the pre-trained model it starts with. As exemplified in "the eye of the planet Jupiter" in Figure 2, SDXL generates a human's eye rather than Jupiter's. DreamSync adds more features of the Jupiter in each iteration. Nevertheless, it did not manage to produce an image that is perfectly faithful to the prompt. This is also exemplified by the quantitative results in SS5.3. Despite outperforming the baselines using SD v1.4 on TIFA and DSG, SD v1.4 + DreamSync still falls behind SDXL. Similarly, our human studies on DSG in SS5.2 indicate that DreamSync improves SDXL from 70.9% accuracy to 74.3%. Nonetheless, there is still a 25.7% headroom to improve. We identify several common failure modes (e.g., attribute-binding) and conduct a detailed analysis in Appendix B. Future works may investigate if these challenges can be addressed by further scaling up DreamSync, or mixing it with large-scale pre-training.

## 7 Conclusion

We introduce DreamSync, a versatile framework to improve text-to-image (T2I) synthesis with feedback from image understanding models. Our dual VLM feedback mechanism helps in both the alignment of images with textual input and the aesthetic quality of the generated images. Through evaluations on two challenging T2I benchmarks (with over five thousand prompts), we demonstrate that DreamSync can improve both SD v1.4 and SDXL for both alignment and visual appeal. The benchmarks also show that DreamSync performs well in both in-distribution and out-of-distribution settings. Furthermore, human ratings and a human preference prediction model largely agree with DreamSync's improvement on benchmark datasets.

For future work, one direction is to ground the feedback mechanism to give fine-grained annotations (e.g., bounding boxes to point out where in the image the misalignment lies). Another direc

\begin{table}
\begin{tabular}{r c r r} \hline \hline  & Rewards & Text Faithfulness & Visual Appeal \\ \hline VQA & VILA & Fainthfulness & Appeal \\ \hline - & - & 83.5 & 60.9 \\ \hline ✓ & & **84.8** & 61.9 \\  & ✓ & 83.8 & 61.7 \\ ✓ & ✓ & 84.7 & **62.8** \\ \hline \hline \end{tabular}
\end{table}
Table 3: **Ablation of different VLM rewards. Models are evaluated after _one DreamSync iteration_.

 

tion is to tailor the prompts used at each iteration of DreamSync to target different improvements: backpropagating VLM feedback to the prompt acquisition pipelines for continual learning.

## References

* A. Agarwal, S. Karanam, K. Joseph, A. Saxena, K. Goswami, and B. Vasan Srinivasan (2023)A-star: test-time attention segregation and retention for text-to-image synthesis. arXiv preprint arXiv:2306.14544. Cited by: SS1.
* R. Anil, A. M. Dai, O. Firat, M. Johnson, D. Lepikhin, A. Passos, S. Shakeri, E. Taropa, P. Bailey, Z. Chen, E. Chu, J. H. Clark, L. El Shafey, Y. Huang, K. Meier-Hellstern, G. Mishra, E. Moreira, M. Omrenicka, K. Robinson, S. Ruder, Y. Tay, K. Xiao, Y. Xu, Y. Zhang, G. Hernandez Abrego, J. Ahn, J. Austin, P. Barham, J. Botha, J. Bradbury, S. Brahma, K. Brooks, M. Catasta, Y. Cheng, C. Cherry, C. A. Choquette-Choo, A. Chowdrey, C. Crepy, S. Dave, M. Dehghani, S. Dev, J. Devlin, M. Diaz, N. Du, E. Dyer, V. Feinberg, F. Feng, V. Fienber, M. Freitag, X. Garcia, S. Gehrmann, L. Gonzalez, G. Gur- Ari, S. Hand, H. Hashemi, L. Hou, J. Howland, A. Hu, J. Hui, J. Hurwitz, M. Isard, A. Ittycherian, M. Jagielski, Wenhao Jia, K. Kenealy, M. Krikun, S. Kudujunta, C. Lan, K. Lee, B. Lee, E. Li, M. Li, W. Li, Y. Li, J. Li, H. Lim, H. Lin, Z. Liu, F. Liu, M. Maggioni, A. Mahendru, J. Maynez, V. Misra, M. Moussalem, Z. Nadq, J. Nham, E. Ni, A. Nystrom, A. Parrish, M. Pellatt, M. Polacek, A. Polozov, R. Pope, S. Qiao, E. Reif, B. Richter, P. Riley, A. Castro Ros, A. Roy, B. Saeta, R. Samuel, R. Shelby, A. Slone, D. Smilkov, D. So, D. Sohn, S. Tokumine, D. Valter, Vijay Vasudevan, Kiran Vodrahalli, Xuezhi Wang, P. Wang, Z. Wang, T. Wang, J. Wieting, Yuhuai Wu, Kelvin Xu, Yunhan Xu, Linting Xue, Pengcheng Yin, Jiahui Yu, Qiao Zhang, Steven Zheng, Ce Zheng, Weikang Zhou, Denny Zhou, Slav Petrov, and Yonghui Wu (2023)Palm 2 technical report. PreprintarXiv:2305.10403. External Links: Link Cited by: SS1.
* J. Betker, G. Goh, L. Jing, T. Brooks, J. Wang, L. Li, L. Ouyang, J. Zhuang, J. Lee, Y. Guo, W. Manassara, P. Dhariwal, C. Chu, Y. Jiao, and A. Ramos (2023)Improving image generation with better captions. Vol. , pp. . Note: Improving image generation with better captions External Links: Document, Link Cited by: SS1.
* N. Bitton-Guetta, Y. Bitton, J. Hessel, L. Schmidt, Y. Elovici, G. Stanovsky, and R. Schwartz (2023)Breaking common sense: whoops! a vision-and-language benchmark of synthetic and compositional images. In ICCV, Cited by: SS1.
* K. Black, M. Janner, Y. Du, I. Kostrikov, and S. Levine (2023)Training diffusion models with reinforcement learning. PreprintarXiv:2305.13301. Cited by: SS1.
* H. Chefer, Y. Alaluf, Y. Vinker, L. Wolf, and D. Cohen-Or (2023)Attention-based semantic guidance for text-to-image diffusion models. ACM Transactions on Graphics (TOG)42 (4), pp. 1-10. Cited by: SS1.
* X. Chen, X. Wang, S. Changpinyo, A. J. Piergioanni, P. Padlewski, D. M. Salz, S. Goodman, A. Grysencr, B. Mustafa, L. Boyer, A. Kolesnikov, J. Puigcerver, N. Ding, K. Kong, H. Akbari, G. Mishra, L. Xue, A. V. Thapilyad, J. Bradbury, W. Kuo, M. Seyedhosseini, C. Jia, B. Karagol Ayan, C. Riquelme, A. Steiner, A. Angelova, X. Zhai, N. Houlsby, and R. Soricut (2022)Pali: a jointly-scaled multilingual language-image model. ArXivabs/2209.06794. Cited by: SS1.
* J. Cho, Y. Hu, R. Garg, P. Anderson, R. Krishna, J. Baldridge, M. Bansal, J. Pont-Tuset, and S. Wang (2023)Davidsonian scene graph: improving reliability in fine-grained evaluation for text-to-image generation. PreprintarXiv:2310.18235. Cited by: SS1.
* J. Cho, A. Zala, and M. Bansal (2022)Dall-eval: probing the reasoning skills and social biases of text-to-image generative transformers. ArXivabs/2202.04053. Cited by: SS1.
* Donald Davidson (1965)The theories of meaning and learnable languages. In In Tejoshua Bar-Hillel (ed.) Proceedings of the 1964 International Congress for Logic, Methodology, and Philosophy of Science. Amsterdam: North-Holland. pp. 383-394. Cited by: SS1.
* Donald Davidson (1967)Truth and meaning. In In Intu Prutth and Interpretation; Soames, chapter 12 of PATC, Cited by: SS1.
* G. Delmas, P. Weinzaepfel, T. Lucas, F. Moreno-Noguer, and G. Rogez (2022)PoseScript: 3D human poses from natural language.. In ECCV, Cited by: SS1.
* D. Epstein, A. Jabri, B. Poole, A. A. Efros, and A. Holynski (2023)Diffusion self-guidance for controllable image generation. arXiv preprint arXiv:2306.00986.

 

* Fan et al. (2023) Ying Fan, Olivia Watkins, Yuqing Du, Hao Liu, Moonkyung Ryu, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, Kangwook Lee, and Kimin Lee. 2023. Dpok: Reinforcement learning for fine-tuning text-to-image diffusion models. _arXiv preprint arXiv:2305.16381_.
* Feng et al. (2023) Weixi Feng, Xuehai He, Tsu-Jui Fu, Varun Jampani, Arjun Akula, Pradyumna Narayana, Sugato Basu, Xin Eric Wang, and William Yang Wang. 2023. Training-free structured diffusion guidance for compositional text-to-image synthesis. _Preprint_, arXiv:2212.05032.
* Feng et al. (2022) Weixi Feng, Xuehai He, Tsu-Jui Fu, Varun Jampani, Arjun Reddy Akula, P. Narayana, Sugato Basu, Xin Eric Wang, and William Yang Wang. 2022. Training-free structured diffusion guidance for compositional text-to-image synthesis. _ArXiv_, abs/2212.05032.
* Fu et al. (2023) Deqing Fu, Ameya Godbole, and Robin Jia. 2023. Scene: Self-labeled counterfactuals for extrapolating to negative examples. _ArXiv_, abs/2305.07984.
* Han et al. (2023) Ligong Han, Yinxiao Li, Han Zhang, Peyman Milanfar, Dimitris Metaxas, and Feng Yang. 2023. Svdiff: Compact parameter space for diffusion fine-tuning. _arXiv preprint arXiv:2303.11305_.
* Hong et al. (2022) Susung Hong, Gyuseong Lee, Wooseok Jang, and Seungryong Kim. 2022. Improving sample quality of diffusion models using self-attention guidance. _arXiv preprint arXiv:2210.00939_.
* Hu et al. (2022) Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-rank adaptation of large language models. In _International Conference on Learning Representations_.
* Hu et al. (2023) Yushi Hu, Benlin Liu, Jungo Kasai, Yizhong Wang, Mari Ostendorf, Ranjay Krishna, and Noah A Smith. 2023. Tifa: Accurate and interpretable text-to-image faithfulness evaluation with question answering. _arXiv preprint arXiv:2303.11897_.
* Huang et al. (2023) Kaiyi Huang, Kaiyue Sun, Enze Xie, Zhenguo Li, and Xihui Liu. 2023. T2i-compbench: A comprehensive benchmark for open-world compositional text-to-image generation. _arXiv preprint arXiv:2307.06350_.
* Karthik et al. (2023) Shyamgopal Karthik, Karsten Roth, Massimiliano Mancini, and Zeynep Akata. 2023. If at first you don't succeed, try, try again: Faithful diffusion-based text-to-image generation by selection. _arXiv preprint arXiv:2305.13308_.
* Ke et al. (2023) Junjie Ke, Keren Ye, Jiahui Yu, Yonghui Wu, Peyman Milanfar, and Feng Yang. 2023. Vila: Learning image aesthetics from user comments with vision-language pretraining. _Preprint_, arXiv:2303.14302.
* Kirstain et al. (2023) Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, and Omer Levy. 2023. Pick-a-pic: An open dataset of user preferences for text-to-image generation. _arXiv preprint arXiv:2305.01569_.
* Krause et al. (2017) Jonathan Krause, Justin Johnson, Ranjay Krishna, and Li Fei-Fei. 2017. A hierarchical approach for generating descriptive image paragraphs. In _CVPR_.
* Kumar et al. (2020) Ananya Kumar, Tengyu Ma, and Percy Liang. 2020. Understanding self-training for gradual domain adaptation. In _Proceedings of the 37th International Conference on Machine Learning_, volume 119 of _Proceedings of Machine Learning Research_, pages 5468-5479. PMLR.
* Lee et al. (2023) Kimin Lee, Hao Liu, Moonkyung Ryu, Olivia Watkins, Yuqing Du, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, and Shixiang Shane Gu. 2023. Aligning text-to-image models using human feedback. _arXiv preprint arXiv:2302.12192_.
* Li et al. (2023) Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. _ArXiv_, abs/2301.12597.
* Liu et al. (2022a) Nan Liu, Shuang Li, Yilun Du, Antonio Torralba, and Joshua B. Tenenbaum. 2022a. Compositional visual generation with composable diffusion models. _ArXiv_, abs/2206.01714.
* Liu et al. (2022b) Rosanne Liu, Daniel H Garrette, Chitwan Saharia, William Chan, Adam Roberts, Sharan Narang, Irina Blok, R. J. Mical, Mohammad Norouzi, and Noah Constant. 2022b. Character-aware models improve visual text rendering. _ArXiv_, abs/2212.10562.
* Lu et al. (2016) Cewu Lu, Ranjay Krishna, Michael Bernstein, and Li Fei-Fei. 2016. Visual relationship detection with language priors. In _ECCV_.
* McClosky et al. (2006) David McClosky, Eugene Charniak, and Mark Johnson. 2006. Effective self-training for parsing. In _Proceedings of the Human Language Technology Conference of the NAACL, Main Conference_, pages 152-159, New York City, USA. Association for Computational Linguistics.
* Mou et al. (2023) Chong Mou, Xintao Wang, Liangbin Xie, Jian Zhang, Zhongang Qi, Ying Shan, and Xiaohu Qie. 2023. T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models. _arXiv preprint arXiv:2302.08453_.
* Paiss et al. (2023) Roni Paiss, Ariel Ephrat, Omer Tov, Shiran Zada, Inbar Mosseri, Michal Irani, and Tali Dekel. 2023. Teaching clip to count to ten. In _ICCV_.
* Park et al. (2021) Dong Huk Park, Samaneh Azadi, Xihui Liu, Trevor Darrell, and Anna Rohrbach. 2021. Benchmark for compositional text-to-image synthesis. In _Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1)_.
* Petsiuk et al. (2022) Vitali Petsiuk, Alexander E Siemenn, Saisamrit Surbhehera, Zad Chin, Keith Tyser, Gregory Hunter, Arvind Raghavan, Yann Hecke, Bryan A Plummer, Ori Kerret, et al. 2022. Human evaluation of text-to-image models on a multi-task benchmark. _arXiv preprint arXiv:2211.12112_.

 

* Podell et al. (2023) Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Muller, Joe Penna, and Robin Rombach. 2023. Sdxl: Improving latent diffusion models for high-resolution image synthesis. _Preprint_, arXiv:2307.01952.
* Pont-Tuset et al. (2020) Jordi Pont-Tuset, Jasper Uijlings, Soravit Changpinyo, Radu Soricut, and Vittorio Ferrari. 2020. Connecting vision and language with localized narratives. In _ECCV_.
* Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learning transferable visual models from natural language supervision. _Preprint_, arXiv:2103.00020.
* Ramesh et al. (2022) Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. 2022. Hierarchical text-conditional image generation with clip latents. _ArXiv_, abs/2204.06125.
* Ramesh et al. (2021) Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. 2021. Zero-shot text-to-image generation. _ArXiv_, abs/2102.12092.
* Kassin et al. (2023) Royi Rassin, Eran Hirsch, Daniel Glickman, Shauli Ravfogel, Yoay Goldberg, and Gal Chechik. 2023. Linguistic binding in diffusion models: Enhancing attribute correspondence through attention map alignment. _Preprint_, arXiv:2306.08877.
* Rossin et al. (2022) Royi Rassin, Shaudil Ravfogel, and Yoav Goldberg. 2022. Dalle-2 is seeing double: Flaws in word-to-concept mapping in text?image models. _Preprint_, arXiv:2210.10606.
* Rombach et al. (2021) Robin Rombach, A. Blattmann, Dominik Lorenz, Patrick Esser, and Bjorn Ommer. 2021. High-resolution image synthesis with latent diffusion models. _2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_, pages 10674-10685.
* Ruiz et al. (2023a) Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. 2023a. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_, pages 22500-22510.
* Ruiz et al. (2023b) Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Wei Wei, Tingbo Hou, Yael Pritch, Neal Wadhwa, Michael Rubinstein, and Kfir Aberman. 2023b. Hyperredream-booth: Hypernetworks for fast personalization of text-to-image models. _arXiv preprint arXiv:2307.06949_.
* Saharia et al. (2022) Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L. Denton, Seyed Kamyr Seedy Ghascimpour, Burcu Karagol Ayan, Seyeda Sara Mahdavi, Raphael Gontijo Lopes, Tim Salimans, Jonathan Ho, David J. Fleet, and Mohammad Norouzi. 2022. Photorealistic text-to-image diffusion models with deep language understanding. _ArXiv_, abs/2205.11487.
* Sohn et al. (2023) Kihyuk Sohn, Nataniel Ruiz, Kimin Lee, Daniel Castro Chin, Irina Blok, Huiwen Chang, Jarred Barber, Lu Jiang, Glenn Entis, Yuanzhen Li, et al. 2023. Styledrop: Text-to-image generation in any style. _arXiv preprint arXiv:2306.00983_.
* Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin R. Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajawal Bhargava, Shruti Bhasale, Daniel M. Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenvin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Namam Goyal, Anthony S. Hartshorn, Saghar Hosseini, Riui Hou, Hakan Inau, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel M. Kloutaman, A. V. Korenev, Punt Singh Koruta, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihalov, Pushkar Mishra, Igor Molbyog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashii Rungta, Kalyan Saladi, Alan Schelenton, Ruan Silva, Eric Michael Smith, R. Subramanian, Xia Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zhengxu Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023. Llama 2: Open foundation and fine-tuned chat models. _ArXiv_, abs/2307.09288.
* Turc and Nemade (2022) Iulia Turc and Gaurav Nemade. 2022. Midjourney user prompts & generated images (250k).
* Wang et al. (2023) Zijie J. Wang, Evan Montoya, David Munechika, Haoyang Yang, Benjamin Hoover, and Duen Horng Chau. 2023. Diffusiondb: A large-scale prompt gallery dataset for text-to-image generative models. In _ACL_.
* Xu et al. (2023) Jiaxheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. 2023. Imagenreward: Learning and evaluating human preferences for text-to-image generation. _arXiv preprint arXiv:2304.05977_.
* Yarom et al. (2023) Michal Yarom, Yonatan Bitton, Soravit Changpinyo, Roee Aharoni, Jonathan Herzig, Oran Lang, Eran Ofek, and Idan Szpektor. 2023. What you see is what you read? improving text-image alignment evaluation. _arXiv preprint arXiv:2305.10400_.
* Yarowsky (1995) David Yarowsky. 1995. Unsupervised word sense disambiguation rivaling supervised methods. In _33rd Annual Meeting of the Association for Computational Linguistics_, pages 189-196, Cambridge, Massachusetts, USA. Association for Computational Linguistics.
* Yu et al. (2022) Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Said, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, Benton C. Hutchinson, Wei Han, Zarana Parekh, Xin Li, Han Zhang, Jason Baldridge, and Yonghui Wu. 2022. Scaling autoregressive models for content-rich text-to-image generation. _ArXiv_, abs/2206.10789 

Training Data Acquisition

### LLM Instructions

Training Data Acquisition is the first step and the foundation of DreamSync as discussed in Section 4. We use PaLM 2 for each step of the training data acquisition, including prompt generation, QA generation, and filtering. Here are the complete instructions that we use.

**Instruction for Prompt Generation.** You are a large language model, trained on a massive dataset of text.

You can generate texts from given examples.

You are asked to generate similar examples to the provided ones and follow these rules:

1. Your generation will be served as prompts for Text-to-Image models. So your prompt should be as visual as possible.
2. Do NOT generate scary prompts.
3. Do NOT repeat any existing examples.
4. Your generated examples should be as creative as possible.
5. Your generated examples should not have repetition.
6. Your generated examples should be as diverse as possible.
7. Do NOT include extra texts such as greetings.

**Instruction for QA Generation.** Given an image description, generate one or two multiple-choice questions that verify if the image description is correct. Classify each concept into a type (object, human, animal, food, activity, attribute, counting, color, material, spatial, location, shape, other), and then generate a question for each type.

We then provide fifteen prompts together with about ten question-answer pairs as a demonstration for PaLM 2. Table 4 shows an example of PaLM2-generated _prompt_ and _QA. Answer source_ and _Answer Type_ are also automatically generated altogether, making it possible for us to get statistics of our training set below.

### Statistics

Table 5 shows the statistics of the prompts and questions we obtained, and we list a few prompts from our training set and DreamSync's generation in Figure 3. Prior work (e.g., TIFA, DSG) identifies that T2I models do not perform equally well for depicting different attribute categories; we verify the variety of attributes in our prompts by counting unique words (i.e., _Answer Source_ in Table 4) in these categories (i.e., _Answer Type_ in Table 4): counting (4179), object (3638), shape (973), human (945), location (1047), activity (2984), attribute (2925), color (3259), food (1086), spatial (1009), animal (645), material (1610), existence (3072), and other (878)





\begin{table}
\begin{tabular}{l|p{142.3pt} p{56.9pt}|c|c}
**Prompt** & **Question and Choices** & **Answer Source** & **Answer Type** \\  & question : what is in the field? choices: [“flowers”, “grass”, “trees", “rocks”] & flowers & object \\  & is there afield? choices: [“yes”, ”no”] & field & location \\  & are there flowers? choices: [“yes”, ”no”] & flowers & object \\  & that type of place is this? choices: [“field”, “park”, “forest”, “mountain”] & field & location \\  & are the baseball players holding sheep? choices: [“yes”, ”no”] & holding & activity \\  & are there sheep? choices: [“yes”, ”no”] & sheep & animal \\  & are there baseball players? choices: [“yes”, ”no”] & baseball players & human \\  & how many baseball players are three? choices: [“1”, “2”, “3”, “4”, ”5”, “g”] & 6 & human \\  & how many sheep are there? choices: [“1”, “2”, “3”, “4”, ”5”, “g”] & 6 & animal \\ \end{tabular}
\end{table}
Table 4: One example of PaLM-2 generated _prompt_ and _QA_. _Answer source_ and _Answer Type_ are also generated by PaLM 2, making it possible for us to get detailed statistics of our training set (see Table 5 in Appendix A). We highlight correct answers in underlines here.

\begin{table}
\begin{tabular}{l r} \hline \hline \# of prompts & 28,250 \\ \hline \# of questions & 239,310 \\ - \# of binary questions & 125,094 \\ - \# of multiple-choice questions & 114,214 \\ \hline avg. \# of questions per prompt & 8.5 \\ avg. \# of words per prompt & 16.7 \\ avg. \# of elements per prompt & 1.9 \\ \hline \hline \end{tabular}
\end{table}
Table 5: Statistics of Training Set for DreamSync.





### Images Generated by DreamSync for Finetuning Exhibit High Quality

Figure 6: Prompts and Images Generated via DreamSync for Finetuning. Prompts generated by PaLM-2 exhibit high diversity and corresponding synthetic images exhibit high quality and alignment.





### Improvement over Alternative Alignment Methods

Figure 7: **Qualitative Comparison with all models mentioned in Table 1 with base model _SD v1.4_. Images are generated with the same seed. DreamSync improves the base model’s alignment to prompts. Unlike RL-based method (e.g. DDPO), DreamSync does not introduce biases to cartoon. Unlike training-free methods (e.g. SynGen and StructureDiffusion), DreamSync does not degrade image aesthetics.



Failure Modes Analysis

Figure 8 presents several side-by-side examples showcasing common failure modes of DreamSync. For each example, we show the image generated by SDXL on the left, and the image of SDXL + DreamSync on the right. We also indicate some key directions for improvements.

* **Composing multiple objects and attributes is still challenging.** As shown in (a), (b), (c), and (d), SDXL + DreamSync struggles to produce an image that is faithful to the prompt. In (a), DreamSync adds a bench in the image. However, the attributes of chairs and benches are mixed. In (b), DreamSync removes the extra glass in the background, but neither model is able to place the lemon wedge in the rim of the bottom. In (c), DreamSync adds purple fish in the image, but the counting is not correct. In (d), DreamSync produces four objects but they are cloud-keychain combinations.
* **We observe decline of texture details and shadows on some images.** In (e), the alignment between the text and the bus significantly improves. However, the quality of the bus shadow declines. In (f), both images align well with the text. The main difference is in the details of the temple facade. Notice that for most images we observe DreamSync yields images with high quality and visual appeal, as illustrated in Appendix A.3.

Future work may explore if these challenges can be addressed by following extensions to DreamSync: (1) DreamSync could be used in tandem with RL-based method and training-free method to further improve text-to-image faithfulness; (2) prompt engineering methods in DALL-E 3 [4] may help rewriting challenging prompts into simpler ones for models to synthesize; (3) scaling up DreamSync with a more diverse set of prompts and reward models; (4) mixing DreamSync with large-scale pre-training on real images. In summary, as discussed in SS6.1, there is still plenty of headroom to improve.

 


Figure 8: **Failure modes. We present qualitative examples of DreamSync failures. First, it remains challenging to compose multiple objects and bind the attributes correctly, as shown in (a), (b), (c), and (d). Second, we observe that the quality of details and shadows decline on some images, as illustrated in (e) and (f). Overall, SDXL + DreamSync still has room for improvement in terms of text-to-image faithfulness and quality.**



DSG and Human Rating Evaluation

Details of DSG-1k benchmark table 6 presents the data sources, quantity, and examples for DSG-1k. fig.9 summarizes the 4 broad and 14 detailed semantic categories covered in the benchmark.

Like TIFA, DSG (Cho et al., 2023) falls into the Question Generation / Answering (QG/A) alignment evaluation framework. Unlike TIFA, DSG introduces a linguistically motivated (Davidson, 1965, 1967a,b) question generation module to ensure the questions generated to hold 4 reliability traits: a) _atomic_: only queries about 1 semantic detail, for unambiguous interpretation; b) _unique_: no duplicated questions; c) _dependency-aware_: prevent invalid queries to VQA/human answerers, e.g. if the answer to a parent question "is there a bike?" is negative, then the child question "is the bike blue?" will not be queried; d) _full semantic coverage_: dovetailing the semantic content of a prompt, no more no less. DSG is powered by a large variant of PaLM 2 (Anil et al., 2023) for QG and the SoTA VQA module PaLI (Chen et al., 2022) for QA. For our evaluation task, we adopt DSG-1k (DSG's 1,060 benchmark prompt set) which covers a balanced set of diverse semantic categories and writing styles - including 4 broad categories (e.g. entity/attribute/etc.) and 14 detailed categories (e.g. color/counting/texture/etc.).

Human QA protocolFor human evaluation, we elicit 3 rating responses per prompt/question set (with \(\sim\)8 questions per set on average, and a total of 8183 questions). fig.10 exemplifies the UI the human raters see. fig.11 presents the annotation instructions used to guide the raters. The inner-annotator agreement for this study is 0.684. While the raters respond with YES/NO/UNSURE, we find it to be practically useful to numerically convert the answers - 1.0 point for YES, 0 for NO, and 0.5 for UNSURE as partial credit, with the justification that if a semantic detail _can potentially_ be grounded in an image yet not necessarily so (e.g. "does this man dress like an engineer?" image: a male in a plain shirt; "is this a cat" image: a blob that _may_ be interpreted as a cat), partial credit is fair for not completely failing.

 

* [18] M. C. Bader, J. A. Harvey, and R. H. Sohn, "A new approach to the design of a multi-objective optimization problem in a multi-objective optimization framework," _Journal of Machine Learning Research_**20**(4):357-369, 2000.
* [19] D. K. Lee, P. L. Schuster, and T. W. Teh, "Optimizing the objective function for solving problems with applications to machine learning," _IEEE Transactions on Signal Processing Systems_**20**(1):105-118, 2002.
* [20] G. E. Hinton, "Deep learning with deep neural networks," _arXiv preprint arXiv:1412.6090_**, 2014.
* [21] Y. Bengio et al., "Generative adversarial nets for large data sets," _Advances in Neural Information Processing Systems_**20**(4):1-10, 2015.
* [22] I. Goodfellow, J. Pouget-Abadie, and J. Schmidhuber, "Learning from tiny images by minimizing gradients," _arXiv preprint arXiv:1412.6090_**, 2014.

[MISSING_PAGE_POST]

 



\begin{table}
\begin{tabular}{l l c p{142.3pt}}
**Feature** & **Source** & **Sample** & **Example** \\ \hline Assorted categories & TIFA160 (Hu et al., 2023) & 160 & “A Christmas tree with lights and teddy bear” \\ \hline Paragraph-type captions & Stanford paragraphs (Krause et al., 2017) & 100 & “There is a cat in the shelf. Under the shelf are two small silver barbels. On the shelf are also DVD players and radio. Beside the shelf is a big bottle of white in a wooden case.” In this picture I can see food items on the plate, which is on the surface. At the top right corner of the image those are looking like fingers of a person.” \\ \hline Counting & CountBench (Paiss et al., 2023) & 100 & “The view of the nine leftmost moai at Ahu Tongariki on Easter Island" \\ \hline Relations & VRD (Lu et al., 2016) & 100 & “person at table. person has face. person wear shirt. person wear shirt. chair next to table. shirt on person. person wear glasses. person hold phone” \\ \hline Written by TIZI _real_ users & DiffusionDB (Wang et al., 2023) & 100 & “a painting of a huangshan, a matte painting by marc simonetti, deviantart, fantasy art, apocalypse landscape, matte painting, apocalypse ant” \\  & Midjourney-prompts (Turc and Nemade, 2022) & 100 & “furry caterpillar, puppa, screaming evil face, demon, fangs, red hands, horror, 3 dimensional, delicate, sharp, lifelike, photorealistic, deformed, wet, shiny, slimy” \\ \hline Human poses & PoseScript (Delmas et al., 2022) & 100 & “subject is squatting, torso is leaning to the left, left arm is holding up subject, right arm is straight forward, head is leaning left looking forward” \\ \hline Commonsense-defying & Whoops (Bitton-Guetta et al., 2023) & 100 & “A man riding a jet ski through the desert” \\ \hline Text rendering & DrawText-Creative (Liu et al., 2022b) & 100 & “a painting of a landscape, with a handwritten note that says ‘this painting was not painted by me” \\ \end{tabular}
\end{table}
Table 6: **DSG-1k overview. To comprehensively evaluate T2I models, DSG-1k provides 1,060 prompts covering diverse skills and writing styles sampled from different datasets.**

Detailed Human Evaluation Results on DSG-1K.

We present detailed evaluations on DSG-1K by semantic categories listed in Figure 9. The results are shown in Figure 12. By applying DreamSync upon SDXL, the human evaluation on alignments improved on all categories.

Single-Question Human EvaluationBesides the large-scale human annotation, we also did a lightweight single-question human evaluation for text prompt alignment. This study was completed by three of the paper's authors. Although this study yields a quite low inter-annotator agreement, we hope it would provide valuable insights on how to set up human evaluation for measuring textual faithfulness of generated images. For this study, we generated one image with SDXL and DreamSync. See Figure 13 for an example rating screen. We randomized the order of the images and prompts. Three authors were asked "Which image better aligns with the prompt?" They could choose Image 1 is better, Image 2 is better, or that they cannot tell (indicating a tie). We use 200 prompts in total with 100 prompts from TIFA and another 100 from DSG.

As mentioned, the inner-annotator agreement was quite low for this study. Only for 42.5% of the 200

Figure 12: Detailed Human Evaluation Results on DSG-1K. By applying DreamSync upon SDXL, the human evaluation of alignments improved on all categories.

Figure 13: Example human rater screen for the author raters. We display two image side-by-side, where we randomize which is Image 1 vs. Image 2. We ask a single question to the raters, referencing the prompt that is displayed below the images. The raters were given that instructions “Rate images based on how many ‘components’ of the prompt are captured in the image. If both images depict every part of the prompt, then you should choose ”I can’t tell (tie).” Otherwise, the image with more correct components is better.” The raters were also shown four examples, with desired ratings and an explanation for the choices.

 

prompts did the human raters all agree in their answers. This is likely due to the fact that it is hard to judge overall prompt alignment directly when given two side-by-side images. Indeed, the majority of prompts led to the raters choosing that they cannot tell which image is better. Using the scoring rules from the DSG study described above (with 1 point going to the model with a direct vote, and with 0.5 going to each model for a tie vote), then we have that DreamSync scores 50.08 while SDXL scores 49.92.

**Key Takeaway from Human Evaluation.** Comparing the fine-grained large-scale human evaluation and the single-question human evaluation, we encourage researchers who are interested in evaluating the text-image alignment to ask annotators detailed and fine-grained questions. It yields significantly better inter-annotator agreement than asking a general single question about alignment. Our large-scale human evaluation with a better agreement suggests that DreamSync improves the textual faithfulness of SDXL on DSG-1k, resonating with our automatic evaluation.

 

Randomly-Sampled SDXL+DreamSync Images

Aside from the failure cases discussed in Figure 8, we would like to showcase more randomly-sampled examples of SDXL and DreamSync. We sample 100 prompts. Among these prompts, Figure 14 shows the examples where the VQA scores of applying DreamSync are significantly different from the base model, SDXL, i.e. the absolute difference of mean score are significantly different: \(\left|{\mathcal{S}_{M}(T,G^{\mathrm{DreamSync}}(T))-\mathcal{S}_{M}(T,G^{ \mathrm{SDXL}}(T))}\right|>0.5\). Meanwhile, Figure 15 presents examples where the DreamSync does not improve the VQA scores upon SDXL.

Figure 14: Random samples where DreamSync are significantly different from SDXL. Both models are sampled with the same seed.

 

* [14] M. C. Bader, A. D. Krizhevsky, and S. Ghemawat, _Deep learning for image recognition_, volume 30 of _IEEE Computer Vision Magazine_, pages 257-268, 2009.
* [15] J. L. Friedman, R. E. Hinton, and P. H. Zisserman, _A review on machine learning_. MIT press, 2002.
* [16] T. Hastie, N. I. Jordan, and Y. Bengio, _Deep learning: An introduction to neural networks_. MIT Press, 2002.

[MISSING_PAGE_POST]

 

Qualitative Comparison with SD v1.4-based Methods in Table 1

Among the 6 examples shown in Figure 16, DreamSync has 3 absolute successes, wheres SynGen, DDPO and StructureDiffusion each has 2, DPOK has 1 and the base model SD v1.4 has 0 absolute success. These results match well with Table 1.

Figure 16: Qualitative Comparison with all models mentioned in Table 1 with base model SD v1.4. Images are generated with the same seed. DreamSync improves the base model’s alignment to prompts. Unlike RL-based method (e.g. DDPO), DreamSync does not introduce biases to cartoon. Unlike training-free methods (e.g. SynGen and StructureDiffusion), DreamSync does not degrade image aesthetics.

 

Technical Details for Reproducing DreamSync

\begin{table}
\begin{tabular}{r|c} \hline \hline Sampling & \\ \hline Number of Inference Steps & 50 \\ LoRA \(\alpha\) & 0.5 \\ Prompts per Iteration & 10,000 \\ Images per Prompt & 8 \\ Sampling Precision & FP16 \\ \hline \hline Filtering & \\ \hline \(\theta_{\mathrm{VQA}}\) & 0.9 \\ \(\theta_{\mathrm{Aesthetics}}\) & 0.6 \\ Percentage of Prompt-Image Pairs Passing the Filters & 20\% \(\sim\) 30\% (see Figure 5) \\ Selected Prompt-Image Pairs for Fine-tuning & 2,000 \(\sim\) 3,000 \\ \hline \hline LoRA Fine-tuning & \\ \hline LoRA Rank & 128 \\ Initial Learning Rate & 0.0001 \\ Learning Rate Scheduler & Cosine \\ LR Warmup Steps & 0 \\ Batch Size & 8 \\ Gradient Accumulation Steps & 2 \\ Total Steps & 2,500 \\ Resolution & 1024 \(\times\) 1024 \\ Random Flip & Yes \\ Mixed Precision & No (i.e. FP32) \\ GPUs for Training & 4 \(\times\) NVIDIA A6000 \\ Finetuning Time & \(\sim\) 4 Hours \\ \hline \hline \end{tabular}
\end{table}
Table 7: Technical Details for Reproducing DreamSync with Base Model SDXL.

\begin{table}
\begin{tabular}{r|c} \hline \hline Filtering & \\ \hline \(\theta_{\mathrm{VQA}}\) & 0.85 \\ \(\theta_{\mathrm{Aesthetics}}\) & 0.5 \\ \hline \hline LoRA Fine-tuning & \\ \hline Finetuning Time & \(\sim\) 1 Hours \\ \hline \hline \end{tabular}
\end{table}
Table 8: Technical Details for Reproducing DreamSync with Base Model SD v1.4. Same Hyper-parameters as Table 7 are omitted.