SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Baselines. We compare the performance of the Decompositional-Alignment Score with prior works on evaluating text-to-image alignment in a reference-free manner. In particular, we show comparisons with CLIP [9], BLIP [10] and BLIP2 [11] scores where the text-to-image alignment score is computed using the cosine similarity between the corresponding image and text embeddings. We also include comparisons with BLIP-ITM and BLIP2-ITM which directly predict a binary image-text matching score (between 0 and 1) for input prompt and output image. Finally, we report results on the recently proposed text-to-text (T2T) similarity metric [7] which computes image-text similarity as the average cosine similarity between input prompt and captions generated (using BLIP) from the input image.
Prompt: A lion playing a guitar
Attend-and-Excite
(a) Object Relationship: Eval-and-Refine helps better capture both presence and relationship between the objects.
Prompt: A person in a spacesuit riding a bicycle by the lake
Attend-and-Excite
Ours  $(\mathrm{PW} + \mathrm{CA})$
(b) Overlapping entities: Proposed approach can better handle cases with overlapping entities (spacesuit, person).
Prompt: a snowman wearing sunglasses holding an umbrella on a beach on a sunny day
Attend-and-Excite
Ours  $(\mathrm{PW} + \mathrm{CA})$
Prompt Complexity: Eval-and-Refine shows better alignment as number of subjects in input prompt increase.
Figure 7: Additional comparisons with Attend-and-Excite. We analyse three main ways in which the proposed iterative-refinement improves over Attend-and-Excite [7] (refer Sec. 4.2 for details).
Quantitative Results. Fig. 5 shows the correlation between human annotations and predicted text-to-image alignment scores across different metrics on the Decomposable-Captions dataset. We observe that the DA-Score shows a significantly higher correlation with human evaluation ratings as opposed to prior works across varying number of subjects  $N \in [2, 5]$  in the input prompt. We also note that
Figure 8: Variation of alignment accuracy with prompt difficulty. We observe that while the accuracy of all methods decreases with increasing difficulty in prompt realism (refer Sec. 4), the proposed iterative refinement approach consistently performs better than prior works.
while the recently proposed T2T similarity score [7] shows comparable correlation with ours for  $N = 2$ , its performance significantly drops as the number of subjects in the input prompt increases.
4.2 Improving Text-to-Image Alignment
In this section, we compare the performance of our iterative refinement approach with prior works on improving text-to-image alignment in a training-free manner. In particular, we show comparisons with 1) Stable Diffusion [1], 2) Composable Diffusion [6] 3) StructureDiffusion [8] and 4) Attend-and-Excite [7]. All images are generated using the same seed across all methods.
Qualitative Results. Results are shown in Fig. 6. As shown, we observe that Composable Diffusion [6] struggles to generate photorealistic combinations of objects especially as number of subjects in the prompt increase. StructureDiffusion [8] helps in addressing some missing objects e.g., telescope in example-1, but the generated images tend to be semantically similar to those produced by the original Stable Diffusion model, and thus does not significantly improve text-to-image alignment.
Attend-and-Excite [7] shows much better performance in addressing missing objects (e.g., telescope in example-1 and umbrella in example-4). However, as sumamrized in Fig. 7 we observe that it suffers from 3 main challenges: 1) Object Relationship (Fig. 7a): we observe that despite having desired objects, generated images may sometimes fail to convey relationship between them. For e.g., in row-1 Fig. 7 while output images show both a lion and guitar, the lion does not seem to be playing the guitar. In contrast, Eval-and-Refine is able to describe both presence and relation between objects in a better manner. 2) Overlapping Entities (Fig. 7b): For images with overlapping entities (e.g., person and spacesuit), we observe that Attend-and-Excite [7] typically spends most of gradient updates balancing between the overlapping entities, as both entities (person and spacesuit) occupy the same cross-attention region. This can lead to outputs where a) other important aspects (e.g., lake in Col-3) or b) one of the two entities (e.g., spacesuit) are ignored. 3) Prompt Complexity (Fig. 7c): Finally, we note that since Attend-and-Excite [7] is limited to applying the cross-attention update w.r.t the least dominant subject, as the complexity of input prompt  $\mathcal{P}$  increases, it may miss some objects (e.g., umbrella, beach, sunny day) during the generation process. In contrast, the iterative nature of our approach allows it to keep refining the output image  $\mathcal{I}$  until a desirable threshold for the overall image-text alignment score  $\Omega(\mathcal{I}, \mathcal{P})$  is reached.
Quantitative Results. In addition to qualitative experiments, we also evaluate the efficacy of our approach using human evaluations. In this regard, we report three metrics: 1) normalized human score: which refers to the average human rating (normalized between 0-1) for images generated on the Decomposable-Captions-4k dataset. 2) accuracy: indicating the percentage of generated images which are considered as an accurate match (rating: 5) for the input text prompt by a human subject. 3) pairwise-preference: where human subjects are shown pair of images generated using our method and prior work, and are supposed to classify each image-pair as a win, loss or tie (win meaning our method is preferred). For our approach we consider two variants 1) Ours  $(PW)$  which performs iterative refinement using only prompt-weighting, and 2) Ours  $(PW + CA)$  where iterative refinement is performed using both prompt weighting and introducing cross-attention updates (Sec. 3.2). Pairwise preference scores are reported while using Ours  $(PW + CA)$  while comparing with prior works.
Method	Norm. Human Score (%)	Alignment Accuracy (%)	Pairwise Win ↑	Comparison % Tie	Lose ↓	Inference Time (s)
Stable-Diffusion [1]	72.98	43.66	41.7	50.1	8.2	3.54 s
Composable-Diffusion [6]	70.28	37.72	57.1	38.5	4.4	10.89 s
Structure-Diffusion [8]	74.93	45.23	37.5	54.6	7.9	11.51 s
Attend-and-Excite [7]	85.94	65.50	23.6	62.3	14.1	8.59 s
Ours (PW)	89.53	70.28	N/A	N/A	N/A	10.32 s
Ours (PW + CA)	90.25	74.16	N/A	N/A	N/A	12.24 s
Table 1: Quantitative Results. We report text-to-image alignment comparisons w.r.t normalized human rating score (Col:2), average alignment accuracy evaluated by human subjects (Col:3) and pairwise user-preference scores (ours vs prior work) (Col:4-6). Finally, we also report average inference time per image for different methods in Col:7. We observe that our approach shows better text-to-image alignment performance while on average using marginally higher inference time.
Results are shown in Fig. 8 and Tab. 1. We observe that while the text-to-image alignment accuracy for all methods decreases with an increased difficulty in input text prompts (Fig. 8), we find that the our approach with only prompt-weighting is able to consistently perform on-par or better than Attend-and-Excite [7]. Further introduction of cross-attention updates (Sec. 3.2), allows our approach to exhibit even better performance, which outperforms Attend-and-Excite [7] by  $8.67\%$  in terms of overall alignment accuracy of the generated images. These improvements are also reflected in the pairwise comparisons where human subjects tend to prefer our approach over prior works [6-8].
Inference time comparison. Tab. 1 shows comparison for the average inference time (per image) for our approach with prior works [6-8]. We observe that despite the use of an iterative process for our approach, the overall inference time is comparable with prior works. This occurs because prior works themselves often include additional steps. For instance, Composable-Diffusion [6] requires the computation of separate denoising latents for each statement in the conjunction/disjunction operation, thereby increasing the overall inference time almost linearly with number of subjects. Similarly, Attend-and-Excite [7] includes additional gradient descent steps for modifying cross-attention maps. Moreover, such an increase is accumulated even if the baseline Stable-Diffusion [1] model already generates accurate images. In contrast, the proposed iterative refinement approach is able to adaptively adjust the number of iterations required for the generation process by monitoring the proposed DA-Score for evaluating whether the generation outputs are already good enough.
In this paper, we explore a simple yet effective decompositional approach for both evaluation and improvement of text-to-image alignment with latent diffusion models. To this end, we first propose a Decompositional-Alignment Score which given a complex prompt breaks it down into a set of disjoint assertions. The alignment of each of these assertions with the generated image is then measured using a VQA model. The assertion-based alignment scores are finally combined to give an overall text-to-image alignment score. Experimental results show that proposed metric shows significantly higher correlation with human subject ratings over traditional CLIP, BLIP based image-text matching scores. Finally, we propose a simple iterative refinement approach which uses the decompositional-alignment scores as feedback to gradually improve the quality of the generated images. Despite its simplicity, we find that the proposed approach is able to surpass previous state-of-the-art on text-to-image alignment accuracy while on average using only marginally higher inference times. We hope that our research can open new avenues for robust deployment of text-to-image models for practical applications.
[38] Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. Align before fuse: Vision and language representation learning with momentum distillation. Advances in neural information processing systems, 34:9694-9705, 2021. 3
[39] Kimin Lee, Hao Liu, Moonkyung Ryu, Olivia Watkins, Yuqing Du, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, and Shixiang Shane Gu. Aligning text-to-image models using human feedback. arXiv preprint arXiv:2302.12192, 2023. 3
[40] Xiaoshi Wu, Keqiang Sun, Feng Zhu, Rui Zhao, and Hongsheng Li. Better aligning text-to-image models with human preference, 2023. 3
[41] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020. 4
[42] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Roziere, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023. 4
[43] Yiheng Liu, Tianle Han, Siyuan Ma, Jiayue Zhang, Yuanyuan Yang, Jiaming Tian, Hao He, Antong Li, Mengshen He, Zhengliang Liu, et al. Summary of chatgpt/gpt-4 research and perspective towards the future of large language models. arXiv preprint arXiv:2304.01852, 2023. 4
[44] AUTOMATIC1111. Stable-diffusion-webui. https://github.com/AUTOMATIC1111/stable-diffusion-webui, 2022.5