Divide, Evaluate, and Refine: Evaluating and Improving Text-to-Image Alignment with Iterative VQA Feedback
Authors: Jaskirat Singh
Liang Zheng
Venue: NeurIPS 2023 poster (2023)
Keywords: Text-to-Image Generation

Abstract: The field of text-conditioned image generation has made unparalleled progress with the recent advent of latent diffusion models. While revolutionary, as the complexity of given text input increases, the current state of art diffusion models may still fail in generating images that accurately convey the semantics of the given prompt. Furthermore, such misalignments are often left undetected by pretrained multi-modal models such as CLIP.  To address these problems, in this paper, we explore a simple yet effective decompositional approach towards both evaluation and improvement of text-to-image alignment. In particular,  we first introduce a Decompositional-Alignment-Score which given a complex caption decomposes it into a set of disjoint assertions. The alignment of each assertion with generated images is then measured using a VQA model. Finally, alignment scores for different assertions are combined aposteriori to give the final text-to-image alignment score. Experimental analysis reveals that the proposed alignment metric shows a significantly higher correlation with human ratings as opposed to traditional CLIP, BLIP scores. Furthermore, we also find that the assertion level alignment scores also provide useful feedback which can then be used in a simple iterative procedure to gradually increase the expressivity of different assertions in the final image outputs. Human user studies indicate that the proposed approach surpasses previous state-of-the-art by 8.7% in overall text-to-image alignment accuracy.

Divide, Evaluate, and Refine: Evaluating and Improving Text-to-lmage Alignment with Iterative VQA Feedback
Jaskirat Singh'
Liang Zheng 1.2
The Australian National University https://1jsingh, .ithhhgggthhbu...g
Australiian Centre for Robotic Vision -nd--andd-eeeee
Abstract
The field of text-conditioned image generation has made unparalleled progress with the recent advent of latent diffusion models. While remarkable, as the complexity given text input increases, the state-of-the-arr diffusion models may still fail in generating images which accurately convey the semantics of the given prompt. Furthermore, it has been observed that such misalignments are often left undetected by pretrained multi-modal models such CLIP. To address these problems, in this paper we explore a simple yet effective decompositional approach towards both evaluation and improvement of text-to-image alignment. In particular, we first introduce a DecompasitionallAAigee Sawaal-AAlgmmee-SSSee -Sooe which given a complex prompt decomposes it into set of disjoint assertions. The alignment of each assertion with generated images is then measured using a VQA model. Finally, alignment scores for different assertions are combined aposteriori to give the final rext-to-image alignment score. Experimental analysis reveals that the proposed alignment metric shows significantly higher correlation with human ratings as opposed to traditional CLIP, BLIP scores. Furthermore, we also find that the assertion level alignment scores provide useful feedback which can then be used in simple iterative procedure to gradually increase the expression of different assertions in the final image outputs. Human user studies indicate that the proposed approach surpasses previous state-of- offt---f-he-art by 8.7% in overall text-to-image alignment accuracy.
Introduction
The field of text-to-image generation has made significant advancements with the recent advent of large-scale language-image (LLI) models [1-5]. In particular, text-conditioned latent diffusion models have shown unparalleled success in generating creative imagery corresponding to diverse range of free-form textual descriptions. However, while remarkable, it has been observed [6-8] that as the complexity of the input text increases, the generated images do not always accurately align with the semantic meaning of the textual prompt.
To facilitate the reliable use of current text-to-image generation models for practical applications, is essential to answer two key questions: 1) Can we detect such fine- misalignments between the input text and the generated output in robust manner? and 2) Once detected, can we improve the tik-tt--mmae alignment for failure cases? While several metrics for evaluating text-to-image alignment (e.g., CLIP [9], BLIP [10], BLIP2 [11]) exist, it has been observed [7, 12] that a high score with these metrics can be achieved even if the image does not fully correspond with input prompt. For instance, in Fig. an output image (containing only pink trees) shows high CLIP/BLIP scores with the text "pink trees and yellow car" even if yellow car is not present. Evaluating text-to-image matching using the image-text-matching (lTM) head of BLIP models has also been recently explored [10, 11]. However, the generated scores also show similar tendency to favor the main subject
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

CLIF BLIII-ITM:: 98.7%
CLIP 0.32
BLIP: 0:13
BLIP2-ITM: 96.3%
Human: 5/5
Disjoint Assertion Set
Dijooint Assertion Set
The image shows pink trees:
The image shows woman: 0,91
image shows yellow car: 0.02
The woman doing yoga: 0.89
The setting mountains:
Woman midelee of lake: 0.79
DA-Score: 0.53
DA-Score: 0.87
Prompt: trees yellow car going through mountains
Prompt: practicing yogn middlee of lake
9 waae wearing saube slaii selle uiderwster with fish Swiimaniingg Prounna.
vaccooss wearing ghofss patparipiinng a gounneet metal ixt loitoleu..
DiffllDii
Attend-and caiind--nd--ae Excite
Ours
Diffle-iifuuiion
Attend-and Excite
Ours
Figure I: Overview. Top: Traditional methods for evaluating text-to-image alignment e.g., CLIP [9]. BLIP-2 [10] and BLIP2-ITM (which provides binary image-text matching score between 0 and 1) often fail to distinguish between good (right) and bad (left) image outputs and can give high scores even if the generated image not an accurate match for input prompt (missing yellow car). In contrast, by breaking down the prompt into a set of disjoint assertions and then evaluating their alignment with the generated image using a VQA model [10], the proposed Decompositional-Nignnes (DA-score) shows much better correlation with human ratings (refer Sec. 4.1). Bottom: Furthermore, we show that the assertion-leve alignment scores can be used along with simple iterative refinement strategy to reliably improve the alignment of generated image outputs (refer Sec. 4.2).
Score
of input prompt. Furthermore, even if such misalignments are detected, it is not clear how such information can be used for improving the quality of generated image outputs in reliable manner.
To address these problems, in this paper we explore simple yet effective decompositional approach towards both evaluation and improvement of fine-grain text-to-image alignment. In particular, we wwtpossitiona--llimmmeeee Score (DA-Score) which given complex text prompt, first decomposes it into a set of disjoint assertions about the content of the prompt. The alignment of each of these assertions with the generated image is then measured using a VQA model [10, 13]. Finally, the alignment scores for diffferent assertions are combined to give an overall text-to-imant -maag aligmment score. Our experiments reveal that the proposed evaluation score shows significantly higher correlation with human ratings over prior evaluation metrics (e.g., CLIP, BLIP, BLIP2) (Sec. 4.
propose
Furthermore, we also find that the assertion-level alignment scores provide useful and explainable feedback for determining which parts of the input prompt are not being accurately described in the output image. We show that this feedback can then be used to gradually improve the alignment the generated images with the input text prompt. To this end, we propose a simple iterative refinement procedure (Fig. 2), wherein at each iteration the expressivity of the least-aligned assertion improved by increasing the weightage/cross-attenee attenntio strength (refer Sec. 3.2) of corresponding prompt tokens during the reverse diffusion process. Through both qualitative and quantitative analysis, we find that the proposed iterative refinement process allows for generation of better aligned image outputs over prior works [6-8] while on average showing comparable inference times (Sec. 4.2).
2 Related Work
Text to Image Generation Models. Text conditional image synthesis is a topic of keen interest in the vision community. For instance, [14-18] use GANs to perform text guided image generation. Similarly, [5, 19] explore the use of autoregressive models for zero-shot text to image generation. Recently, diffussion-baed-mmoole
[1-5, 20, 21) have emerged as a powerful class of methods for performing text-conditional image synthesis over diverse range of target domains.

---

scuplle wearing souda part hRving Lee parryy deddenae with nokane of fick
risting skottlrnadd dewn comuattiio vnaul white holoting comcrrila and wearing giogles..
Figure 2: Iterative refinement (Col: (C34:1-3;466) for improving text-to-image alignment. We propose simple iterative refinement approach which uses the decompositional alignment scores (refer Sec. 3.1) as feedback gradually improve the alignment of the generated images with the input text-prompt.
While remarkable, generating images which align perfectly with the input text-prompp remains a challenging problem [6-8, 22]. To enforce, heavier reliance of generated outputs the provided text, classifier-free guidance methods [2, 3, 23] have been proposed. Similarly, use an additional guidance input to improve controllability of text-to-image generation have recently been extensively explored [24-35]. However, even with their application, the generated images are often observed to exhibit fine-grain misalignments such missing secondary objects [6, 7] with the input text prompt.
Evaluating Image-Text Alignment. Various protocols for evaluating text-image alignment in efference-free manner have been proposed [9-11]. Most prior works [2, 3, 5, 9] typically use the cosine similarity between the text and image embedding from large-scale multi-modal models [9, 36- 38] such CLIP [9], BLIP [10], BLIP-2 [11] for evaluating the alignment scores. Recently, [10, 11] also show the application of BLIP/BLIP-2 models for image-text matching using image retrieval. However, as shown in Fig. these scores can give very high scores even if the generated images do not full align with the input text prompt. Furthermore, unlike our approach image-text alignment is often represented through single scalar value which does not provide explainable measure which can be used to identify/improve weaknesses of the image generation process.
Improving Image-Texx Alignment. Recently several works [6-8] have been proposed to explore the problem of improving image-text alignment in training free manner. Liu et [6] propose to modify the reverse diffusion process by composing denoising vectors for different image components. However, has been observed [7] that it struggles while generating photorealistic compositions of diverse objects. Feng et al. [8] use scene graphs to split the input sentence into several noun phrases and then assign a designed attention map to the output of the cross- cross-attention operation. another recent work, Chefer el al. [7] extend the idea of cross-altention map modification to minimize missing objects but instead do so by modifying the noise latents during the reverse diffusion process. While effective at reducing missing objects, find that the performance quality of output images can suffer as the number of subjects in the input prompt increases (refer Sec. 4.2).
Besides training-free methods, recent contemporary work [39, 40] has also explored the possibility of improving image-text alignment using human feedback to finetune existing latent diffusion models. However this often requires the collection of large-scale human evaluation scores and finetuning the diffusion model across a range of diverse data modalities which can be expensive. In contrast, we explore training free approach for improvement of fine-grain text-to-image alignment.
3 Our Method
Given the image generation output I corresponding to a text prompt P, we wish to develop mechanism for evaluation and improvement of fine-grain text-to-image alignment. The core idea of our approach is to take a decompositional strategy for both these tasks. To this end, we first generate a set of disjoint assertions regarding the content of the input prompt. The alignment of the output image I with each of these assertions is then calculated using VQA model. Finally, use the
3

---

Text Prompe P souba civer playing undorwater
Prompt
Displint Assentinn Set {
Acsertion uefee.
Overall Score
Decompositinn Model
brange slomes enitts
90% P)
ΣA ozj ay(i,.
The wate dierisplannn gaiar.
Scnte-derr in playing toiing
Brenge setting analemotion
argmaa RP It
Paramatrized Diffusion Model D(P, (Ra))
VQA
Update Assertion Weights
Model
argrain (e P.)
Figure 3: Method Overview. Given text prompt P and an initially generated output Io, first generate a set of disjoint assertions a, regarding the content of the caption. The alignment of the output image To with each of these assertions then calculated using a VQA model. Finally, use the assertion- assertion-based--e aligmmmnt scores u, uz(Zo.7 P) as feedback to increase the weightage w; (of the assertion with least alignment score) in parameterized diffusion model formulation 'D (Sec. 3.2). This process can then be performed in an iterative manner to gradually improve the quality of the generated outputs until desirable threshold for the overall alignment score (T( P) is reached.
bai--eedddddlgmeenttt scores as feedback to improve the expressiveness of the assertion with the least alignment score. This process can then be performed in an iterative manner to gradually improve the quality of generated outputs until desired value for the overall alignment score is attained.
In the next sections, we discuss each of these steps in detail. In Sec. 3.1 we first discuss the process for evaluating decompositionallaliie aaignmmenn scores. We then discuss the iterative refinement process for improving text-to-image alignment in Sec. 3.2. Fig. 3 provides an overview for the overall approach.
3.1 Evaluating Text-to-Image Alignment
Prompt Decomposition Model. Given an input prompt P. we first decompose its textual information into a set of disjoint assertions (and corresponding questions) which exhaustively cover the contents of the input prompt. Instead of relying on auman-inputs as in [6. 71', we leverage the in-context learning capability [41] of large-language models [42, 43] for predicting such decompositions in an autonomous manner. In particular, given an input prompt P and large-language model M, the prompt decomposition is performed using in-context learning as,
x { Ti.... xn = M(x P, Pezempplers T),
(I)
where x is the model output, n is the number of decompositions, D Dexemplar is the in-context learning dataset consisting 4-5 human generated examples for prompt decomposition, and task description. Please refer supp. material for further details on exemplar-date dataset and descriietnn design.
The model output is predicted to contain tuples x; {a,, where each tuple is formatted to contain assertions O; and the sub-part p; of the original prompt P corresponding to the generated assertion. For instance, given P 'a and dog' the prompt decomposition can be written as,
M(x P: : 'a cat and a dog', Dezempler: T) = = 'there is a cat', cat'), { ('there a dog', 'a dog ')]
Alignment Scores. We next compute the alignment of the generated image I with each of the disjoint assertions using In particular, given image I, assertions a,, i = 1,... n, their rephrasing in question format af and VQA-model V. the assertion-level alignment scores u; a;) are computed as,
Computing Assertion-basee
(VQA) model [10].
exp (a4/T)
u, (z) = exp (os/T) + exp (B:/T)' where 0; = V("yes' | a,a)), B = V("no' I, I,a?),
where as, 8 refer to the logit-scores of VQA-model V for input tuple (image I, question a!) corresponding to output tokens 'yes', 'no' respectively. Hyperparameter controls the temperature of the softmax operation and controls the confidence of the alignment predictions.
Prior works on improving image-text alignment often rely on human- user inputs for expressing contents of the input prompt into its simpler constituents. For instance, Feng et al. [6] require the user to describe the prompt a conjunction/disjunetioo of simpler statements. Similarly, Chefer et al. [7] require the user to provide set of entities / subjects in the prompt, which their optimization should be performed.

---

Provzz
LA soubba playiig guittrr underwsterr
Sub-prompts:
{p:)
Disjoint Assertion Set: {a:)
Question Rephrasing: {a?
a souder aluee
The twage airows soucda diiver.
Doees the lmaae sinow acsdba alver?
playjang gustar
Tht souda slowe is playing il gidtar.
is scubbe alive playfing guitar?
wasterwater
The setting of scone ia wndlerwater.
setting of sotne inderwweert
Provupp P
/AA kopse piu beach wearing suouglasiss.
Sub-orompte {p;}
Disjoint Assertion Set: {a:}
Question Rephrasing: {a}
lnpse
The lesage shows Herss.
poss the imagg shop korse?
beachn
The horre is a bsack.
is tie lovse pus beeek?
Netringg sunglosses
The horse is wetring singglasses
Lits lipesoo wetting sunglusses?
Prowet P
: slotphii wearing it paitt kot at post party
Sub--poomps:
{p+}
Disjoint Assertion Set. {a:}
Question Rephrasing: {a.)
polpuion
The lowage ploppiiss
(owage deliolnla>?
wabringg a party
vilphin is wearing a party kat.
dolphiis wentlag parly hat??
pack
The settling Umage is poot party.
sotting of inuage pack party?
Figure 4: Visualizing the prompt decomposition process. By dividing complex prompt P into set of disjoint assertions 0g we are able identify the sub-prompts p; (circled) which are not expressed in the image output using VQA, and thereby address them using iterative refinement (Sec. 3.2).
Combining Alignment Scores. Finally, the assertion level alignment-scores u, u,(Z, a.) are combined to give the overall text-to- text-to-image aligmment score (II, P) between image I and prompt P as,
SZZ, P) E, (P,, a.) u, (Tx. a:)
AP,(P,
where weights (,, a;) refer the importance of assertion a, in capturing the overall content of the input prompt P. and allows the user to control the relative importance of different assertions in generating the final image output". Please refer Fig. for the overall implementation.
3.2 Improving Text to Image Aligmment
In addition to predicting overall eext-to-mmage alignment score, we find that assertion-leve alignment scores (I, a,) also provide a useful and explainable way for determining which parts of the input prompt are not being accurately described in the output image This feedback can then be used in an iterative manner to improve the expressivity of the assertion with least alignment score u; (I, 4k), until a desired threshold for the overall text-image alignment score (II, obtained.
Parameterized Diffusion Model. We first modify the image generation process of standard diffusion models in order control the expressiveness of different assertions a; in parametric manner. In particular, we modify the reverse diffusion process to also receive inputs weights w;, where each w; controls the relative importance of assertion a, during the image generation process, In this paper, we mainly consider the following two methods for obtaining such parametric control.
Prompt Weighting. Instead of computing the CLIP [36] features from original prompt P we use prompt-weighting [44] to modify the input CLIPP embeddings to the diffusion model as,
CLIP(P) W(P, {CLIP(p:), )))=))
(3)
where W refers to the prompt-weightiee function from [1, 44], P: refers to the sub-prou sub-prompt (Sec. 3.1) corresponding to assertion a,, and weights w; control the relative weight of different sub-prompts p: in computing the overall CLIP embedding for prompt P.
Cross-Attention Control. Similar [7], we also explore the idea of modifying the noise latents 2, during the reverse diffusion process, to increase the cross-attention strength of the main noun-
FFor simplicity reasons, we mainly use X; Ivi in the main paper. Further analysis on variable A; to account for variable information content visual verifiability of assertion provided in supp. material.
5

---

of Subjects
Number Subjects =
Number of Subjects =
Number of Subjects
Corclation Coefliciat
0.6
0.5
0.4
02
0.1
0.9
DA-Score
CLIP
BLIP
BLIP2
BLIP-ITM
BLIP2-ITM
T2T-Score
Figure 5: Method comparisons w.E. correlation with human ratings. We compare the correlation of different sotttoo-mmae alignment scores with those obtained from human subjects, as the number of subjects in the input prompt (refer Sec. 4) is varied. We observe that the proposed alignment score (DA-score) provides a better match for human human-ratings over traditional text-to-image alignment scores.
subject for each sub-assertion aq. However, instead of only applying the gradient update for the least dominant subject [7], we modify the loss for the latent update in parametric form as,
= V f(2,, (w;);=))), where C(z,, (w;) = Σ w:(1 max G(A))), (4)
where is the step-size, A refer the attention map corresponding the main noun-subject in assertion ag G' is a smoothing function and weights wy control the extent to which the expression of different noun-subjects in the prompt (for each assertion) will be increased in the next iteration.
Iterative Refinement. Given the above parametric formulation for controlling expression different assertions, we next propose simple yet effective iterative refinement approach towards improving text-to-image alignment. In particular, at any iteration E 5] during the refinement process, we first compute both overall text-image similarity score (?[k,, P) and assertion-level alignment scores u, (Ix, P). The image generation output I for the next iteration is then computed as,
{ [wt + if i= argmin, (I w
Ik+1 D(P, (w,+ (uuft11p 1 )==1)); where
w R++ =
(5)
otherwise
where D refers to the parametrized diffusion model and A a hhper--aramette This iterative process is then performed until desirable threshold for the overall alignment score (TI,, P) is reached. The image generation output I* at the end of the refinement process then computed as,
I argmax,, P(I,, P).
(6)
4 Experiments
Dataset. Since there are no openly available datasets addressing semantic challenges in text-based image generation with human annotations, we introduce new benchmark dataset Decomposable- Captions-4k for method comparison. The dataset consists overall of 24960 human annotations on images generated using all methods [1, 6, 7] (including ours) across diverse set of 4160 input prompts. Each image is given rating between and 5 (where I represents that 'image is irrelevant to the prompt' and represents that 'image is an accurate match for the prompt").
Furthermore, unlike prior works [7] which predominantly analyse the performance on relatively simple prompts with two subjects (e.g. object and object b), we construct systematically diverse pool of input prompts for better understanding text-to-image alignment across varying complexities the text prompt. particular, the prompts for the dataset are designed to encapsulate two axis of complexity: number of subjects and realism. The number of subjects refers to the number of main objects described in the input prompt and varies from (e.g., a cat with a ball) walking her dog on leash by the beach during sunser). Similarly, the realism of prompt is defined as the degree to which different concepts naturally co-oceur together and varies as easy, medium, hard and very hard. easy typically refers to prompts where concepts are naturally co-occurring together (e.g., a dog park) while very hard refers to prompts where concept combination very rare (e.g., dog playing piano). Further details regarding the dataset are provided in supplementary material.
(e.g., woman

---

EOCCONAN aaring tolonoggh
mone secoring orner rating anigrol
mon wearing aeaba mit plannn celly undeeaaneer
man riding shatebward dearw mouninin holding
undrrella and weuringg gogples
Slable- Diffusion
Compesiible Diffusion
Structure Dillesionn
Altend and
Figure 6: Qualitative comparison W.E. text-to-imann to-image alignment. We compare the outputs of our iterative refinement approach with prior works [1, 6-8] on improving quality of generated images with changing number of subjects (underlined) from 2 to 5. Please zoom-in for best comparisons.
4.1 Evaluating Text-to-Image Alignment
Baselines. We compare the performance of the DecompositionalAiignene Score with prior works on evaluating text-to-imen to-image alignment in a reference- -fee manner. In particular, we show comparisons with CLIP [9], BLIP [10] and BLIP2 [11] scores where the text-to-image alignment score is computed using the cosine similarity between the corresponding image and text embeddings. We also include comparisons with BLIP-ITM and BLIP2-ITM which directly predict a binary image-text matching score (between 0 and I) for input prompt and output image. Finally, report results on the recently proposed text-to-text (T2T) similarity metric [7] which computes image-text similarity as the average cosine similarity between input prompt and captions generated (using BLIP) from the input image.
7

---

Prompt: A Lien playing auitar
Artend-and--aan Exccit
Ours (PW + CA)
41 TIRRED
(a) Object Relationship: Eval-and- Rnfe-ine-e helps better capture both presence and relationship between the objects.
Prompt:: person in Ø. SpRcesvit riding a bioycle by the Lekee
Attend-anc--an Excite
Ours (PW + CA)
(b) Overlapping entities: Proposed approach can better handle cases with overlapping entities (spacesuit, person).
Prompt:: & sиaиman wearing sunglasses holding unmbrella beacht Dus sunmuy day
Artend-and-Exxt
Ours (PW + CA)
(c) Prompt Complexity: Eval-and Seal-and-Refine shows better alignment number of subjects in input prompt increase.
Figure 7: Additional comparisons with Attend-and-Exciee We analyse three main ways in which the proposed iterative-refinemenn improves over Attend-and and-Excite [7] (refer Sec. 4.2 for details).
Quantitative Results. Fig. shows correlation between human annotations and predicted text-to- image alignment scores across different metrics the Decomposable- Caplionn dataset. We observe that the DA DA-Score shows significantly higher correlation with human evaluation ratings as opposed to prior works across varying number of subjects N e [2,5] in the input prompt. We also note that

---

Prosutt Difficulty: Easy
Prompt Difficulty: Medium
Prompt Difficulty: Hard
Promptt Difficulty; Very Hard
Alignneat Aocurity
0.8
0.6
0.4
0.0
Ours (PW- CA))
Ours (PW)
Aneed-andd-Esitee
Scructure Dilfusion.
Composabie Diflusion
Stable Diffusion
Figure 8: Variation of alignment accuracy with prompt difficulty. We observe that while the accuracy of all methods decreases with increasing difficulty in prompt realism (refer Sec. 4), the proposed iterative refinement approach consistently performs better than prior works.
while the recently proposed T2T similarity score [7] shows comparable correlation with ours for N 2, its performance significantly drops as the number of subjects in the input prompt increases.
4.2 Improving Text-to-lmage Alignment
In this section, we compare the performance of our iterative refinement approach with prior works on improving text-to-image alignment in training- free manner. In particular, we show comparisons with 1) Stable Diffusion [1], 2) Composable Diffusion [6] 3) StructureDiffusion [8] and 4) Attend- and-Excite [7]. All images are generated using the same seed across all methods.
Results are shown in Fig. 6. As shown, we observe that Composable Diffusion [6] struggles to generate photorealistic combinations of objects especially as number of subjects in the prompt increase. StructureDiffusion [8] helps in addressing some missing objects e.g., telescope in example-1, 1, but the generated images tend to be semantically similar to those produced by the original Stable Diffusion model, and thus does not significantly improve cext-to-image alignment.
Qualitative Results.
Lound-and-Exeitte [7] shows much better performance in addressing missing objects (e.g., telescope in example-1 1 and umbrella in example-4). However, as sumamrized in Fig. 7 we observe that it suffers from 3 main challenges: 1) Object Relationship (Fig. 7a): we observe that despite having desired objects, generated images may sometimes fail to convey relationship between them. For egg., in row-1 Fig. while output images show both a lion and guitar, the lion does not seem to be playing the guitar. In contrast, Eval-and-Reeine is able to describe both presence and relation between objects in a better manner. 2) Overlapping Entities (Fig. 7b): For images with overlapping entities (e.g.. person and spacesuir), we observe that Attend-and-Excits [7] typically spends most of gradient updates balancing between the overlapping entities, as both entities (person and spacesuir) occupy the same cross-attention region. This can lead to outputs where a) other important aspects (e.g., lake in Col-3) or b) one of the two entities (e.g., spacesuit) are ignored. 3) Prompt Complexity (Fig. 7c): Finally, we note that since Attend-and and-Excite [7] is limited to applying the cross-attentie update w.r. the least dominant subject, as the complexity of input prompt P increases, it may miss some objects (e.g., unbrella, beach, sunny day) during the generation process. contrast, the iterative nature of our approach allows it to keep refining the output image I until a desirable threshold for the overall image-text alignment score QI P) reached.
Quantitative Results. In addition to qualitative experiments, we also evaluate the efficacy of our approach using human evaluations. In this regard, we report three metrics: normalized human score: which refers to the average human rating (normalized between 0-1) for images generated on the Decomposable-Cappe Cappbootttt dataset. 2) accuracy: indicating the percentage of generated images which are considered as an accurate match (rating: 5) for the input text prompt by human subject. 3) where human subjects are shown pair of images generated using our method and prior work, and are supposed to classify each image-pair as a win, loss or tie (win meaning our method preferred). For our approach we consider two variants 1) Ours (PW) which performs iterative refinement using only prompt-weighting. and 2) Ours (PW + CA) where iterative refinement is performed using both prompt weighting and introducing cross-attention updates (Sec. 3.2). Pairwise preference scores are reported while using Ours (PW CA) while comparing with prior works.

---

Norm. Human Score (%)
Alignment Accuracy (%)
Pairwise Comparison %
Inference
Method
Win t
Tie
Lose
Time (s)
Sihbl-Diffussin [1]
72.98
43.66
41.7
50.1
8.2
3.54 s
Composable- Diffusion [6]
70.28
37.72
57.1
38.5
10.89 s
Structure- Difffuuionn [8]
74.93
45.23
37.5
54.6
7.9
11,51
Atlend-and-Exeee [7]
85.94
65.50
23.6
62.3
14.1
8,59 s
Ours (PW) Ours (PW CA)
89.53
70.28
N/A
N/A
N/A
10.32
90.25
74.16
N/A
N/A
N/A
12.24
Table 1: Quantitative Results. We report text-to-image alignment comparisons
normalized human rating score (Col:2), average alignment accuracy evaluated by human subjects (Col:3) and pairwise seer-preference scores (ours vs prior work) (Col:4-6). Finally, we also report average inference time per image for different methods in Col:7. We observe that our approach shows better text-to-image alignment performance while on average using marginally higher inference time.
Results are shown in Fig. 8 and Tab. 1. We observe that while the text-to-image alignment accuracy for all methods decreases with an increased difficulty in input text prompts (Fig. 8), we find that the our approach with only prompt-weighting is able to consistently perform on-par or better than Attend-and -and--n----eiieee [7]. Further introduction to exhibit even better performance, which outperforms
pross-attention updates (Sec. 3.2), allows our approach
Beeend-and-Exee [7] by 8.67 % in terms of overall aligmment accuracy of the generated images. These improvements are also reflected in the pairwise comparisons where human subjects tend to prefer our approach over prior works [6-8].
Inference time comparison. Tab. 1 shows comparison for the average inference time (per image) for our approach with prior works [6-8]. We observe that despite the use of an iterative process for our approach, the overall inference time is comparable with prior works. This occurs because prior works themselves often include additional steps. For instance, Composable-Diffusss computation of separate denoising latents for each statement in the confinetiio//dijunntiioi operation, thereby increasing the overall inference time almost linearly with number of subjects. Similarly, Attend-and lall-and--xiiiie Excite [7] includes additional gradient descent steps for modifying cross-attention maps. Moreover, such an increase is accumulated even if the baseline Siffl--iffusion [1] model already generates accurate images. In contrast, the proposed iterative refinement approach is able adaptively adjust the number of iterations required for the generation process by monitoring the proposed DA- Score for evaluating whether the generation outputs are already good enough.
[6] requires the
Conclusion
In this paper, explore simple yet effective decompositional approach for both evaluation and improvement text-to-inn to-ag alignment with latent diffusion models. To this end, we first propose a Decompositional-Niigneee Score which given complex prompt breaks it down into set of disjoint assertions. The alignment of each of these assertions with the generated image is then measured using VQA model. The assertion- based alignment scores finally combined to give overall text-to- image alignment score. Experimental results show that proposed metric shows significantly higher correlation with human subject ratings over traditional CLIP, BLIP based image-text matching scores. Finally, we propose simple iterative refinement approach which uses the iitomppoitionnllllinnne.. scores feedback gradually improve the quality of the generated images. Despite its simplicity, find that the proposed approach is able to surpass previous state-of-the-aa on text-to-iie text-to-image alignment accuracy while on average using only marginally higher inference times. We hope that our research can open new avenues for robust deployment of text-to-image io-image models for practical applications.
10

---

References
[1] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models, 2021. 1, 5, 6, 7, 9, 10
[2] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam. Pamela Mishkin, Bob McGrew, llya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2 12.../..//4)) /0741, 2021,
[3] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, et al. Photorealistic text- to-image diffusion models with deep language understanding. arXiv preprint arXi::2205./1487 2022.
[4] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204, 06125, 2022.
[5] Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang. Bureu Karagol Ayan, al. Scaling autoregressive models for content-rich text-to-image generation. arXiv preprint
200::2226.0889 2022. 2,
[6] Nan Liu, Shuang Li. Yilun Du, Antonio Torralba, and Joshua B Tenenbaum. Compositional visual generation with composable diffusion models. In Computer Vision-ECCC 2022: 17th European Conference, Tel Aviv, Isvael, October 23-27, 27, 2022, Proceedings, Part XVII, pages 423-439. Springer, 2022. 1, 2, 3, 6,
7, 9, 10
[7] Hila Chefer. Yuval Alaluf. Yael Vinker, Lior Wolf, and Daniel Cohen-Or. Attend-and-exciter Attention- based semantic guidance for text-to-image diffusion models. arXiv preprint arXiv:2301. 13826, 2023. 3, 4, 5, 6, 7, 8, 9, 10
[8] Weixi Feng. Xuehai He, Tsu-Jui Fu, Varun Jampani, Arjun Akula, Pradyumna Narayana, Sugato Basu, Xin Eric Wang. and William Yang Wang. Training- free structured diffuusinn guidance for compositional text-to-image synthesis, arXiv preprint arXiv222/2.00022 2022. 1, 2, 3, 7, 9, 10
[9] Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation metric for image captioning. arXiv preprint arXiv:2104.087/1 2021. 1. 2,7,
[10] Jumnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-inage pre-training for unified languaee understanding and generation. In International Conference Machine Learning, pages 18888 12900. PMLR, 2022.
4,7
[11] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301. 12597, 2023. 1, 3,
[12] Roni Paiss, Hila Chefer, and Lior Wolf. No token left behind: Fxplainability-aide image classification and generation. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Ismael, October 23-27, 2022, Proceedings, Part XII. pages 350-350. Springer, 2022. 1
[13] Wonjae Kim, Bokyung Son, and Ildoo Kim. Vilt: Vision-and- Iision-and-language transformer without convolution or region supervision, 2021. 2
[14] Tao Xu, Pengehuan Zhang, Qiuyuan Huang, Han Zhang, Zhe Gan, Xiaolci Huang, and Xiaodong He. Attngan: Fine-grainee text to image generation with attentional generative adversarial networks. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 1316-1324, 2018.
[15] Ming Tao, Hao Tang, Fei Wu, Xiao-Yuan Jing, King-Kun Bao, and Changsheng Xu. Df-gan: A simple and effective baseline for text-to-image synthesis. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 16515-16525, 2022.
[16] Hui Ye, Xiulong Yang, Martin Takac, Rajshekhar Sunderraman, and Shihao Ji. Improving text-to-image synthesis using contrastive learning. arXiv preprint
2021.
[17] Han Zhang. Jing Yu Koh, Jason Baldridge, Honglak Lee, and Yinfei Yang. Cross-modal contrastive learning for text-to-image generation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 833-842, 2021.
[18] Minfeng Zhu, Pingbo Pan, Wei Chen, and Yi Yang. Dm-gan: Dynamic memory generative adversarial networks for text-to- to-iiaie synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5802 5810, 2019. 2

---

[19] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International Conference on Machine Learning. pages 8821-8831, PMLR, 2021.
[20] Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Karsten Kreis, Miika Aittala, Timo Aila, Samuli Laine, Bryan Catanzaro, et al. ediffi: Text-to-image diffusion models with an ensemble of expert denoisers. arXiv preprint (0XXx:2211.013222 2022. 2
[21]
Chenshuang Zhang, Chaoning Zhang, Mengehun Zhang, and In So Kwcon. Text-to-image diffusion model in generative ai: A survey. arXiv preprint arXiv:2303.07909 2023.
[22] Rosanne Liu, Dan Garrette, Chitwan Saharia, William Chan, Adam Roberts, Sharan Narang, Irina Blok, RJ Mical, Mohammad Norouzi, and Noah Constant, Character-aware models improve visual text rendering. arXiv preprint arXiv:2212. 10562, 2022. 3
[23]
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12590 2022.
[24] Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon. SDEdit: Guided image synthesis and editing with stochastic differential equations. International Conference on Learning Representations, 2022. 3
[25] Jooyoung Choi, Sungwon Kim, Yonghyun Jeong. Youngjune Gwon, and Sungroh Yoon. Ilvr: Conditioning method for denoising diffusion probabilistic models. arXiv preprint 1XX::2008002938 2021.
[26] Jaskirat Singh, Stephen Gould, and Liang Zheng. High-fidelity guided image synthesis with latent diffusion models. arXiv preprint 2rXiv:2211.10884. 2022.
[27] Omri Avrahami, Thomas Hayes, Oran Gafni, Sonal Gupta, Yaniv Taigman, Devi Parikh, Dani Lischinski, Ohad Fried, and Xi Yin. Spatext: Spatio-textual representation for controllable image generation. arXiv preprint axXiv:2211.14300 2022.
[28] Omer Tal- Lior Yariv, Yaron Lipman, and Tali Dekel. Multidiffusion: Fusing diffusion paths for controlled image generation. arXiv preprint arXiv:2302.08113 2, 2023.
[29] Vishnu Sarukkai, Linden Li, Arden Ma. Christopher Ré, and Kayvon Fatahalian. Collage diffusion. arXiv preprint 2303::2233300222 2023.
[30] Zeng, Zhe Lin, Jianming Zhang, Qing Liu, John Collomosse, Jason Kuen, and Vishal M Patel. Scenecomposer Any-level semantic image synthesis. arXiv preprint arXiv:2211.11742. 2022.
[31] Zhengyuan Yang, Jianfeng Wang, Zhe Gan, Linjie Li, Kevin Lin, Chenfei Wu, Nan Duan, Zicheng Liu, Ce Liu, Michael Zeng, et al. Reco: Region-controlle text-to-image generation. arXiv preprint arXiv:221..15518 2022.
[32] Lvmin Zhang and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. arXiv preprint arXiv:2302.00 055.05543, 2023.
[33] Chong Mou, Xintao Wang, Lianghin Xie, Jian Zhang, Zhongang Qi, Ying Shan, and Xiaohu Qie. T2i- adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models. arXiv preprint arXi:: 08453, 2023.
[34] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yacl Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. arXiv preprint arXiv:2208.1 12242, 2022.
[35] Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit Bermano, Gal Chechik, and Daniel Cohen-Or. An image is worth one word: Personalizing text-to-image generation using textual inversion. arXiv preprint arXiv:2208.01611 2022. 3
[36] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pages $748-8763. PMLR, 2021. 3,5 5
[37] Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. arXiv preprint XXix:2205009911 2022.
12

---

[38] Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. Align before fuse: Vision and language representation learning with momentum distillation. Advances in nearul information processing systems, 34:9694-9700 -9705, 2021.
[39] Kimin Lee. Hao Liu. Moonkyung Ryu, Olivia Watkins, Yuqing Du Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, and Shixiang Shane Gu. Aligning text-to-image models using human feedback. arXiv preprint arXiv22302.21992 2023. 3
[40] Xiaoshi Wu, Kegiang Sun, Feng Zhu, Rui Zhao, and Hongsheng Li, Better aligning text-to-image models with human preference, 2023. 3
[41] Tom Brown, Benjamin Mann. Nick Ryder. Melanie Subbiah. Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877 13:1877-1901, 2020. 4
[42] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie Anne Lachaux, Timothée Lacroix, Baptiste Rozière. Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2300. 13391, 2023.
[43] Yiheng Liu, Tianle Han, Siyuan Ma, Jiayue Zhang. Yuanyuan Yang. Jiaming Tian, Hao He, Antong Li, Mengshen He, Zhengliang Liu, Summary of chatgpt/gpt-4 research perspective towards the future large language models. arXiv preprint arXiv:2304.0/882, 2023.
[44] AUTOMATIC1111.
stablediffusion-webbl stable-diffuse webui, 2022.
https:// http://gtttuu.ccc com AUTOMATIC1111/
13


Reviews:
[{"review_id": "RD6HND4VzH", "reviewer": "Reviewer_GK7B", "summary": "The paper proposes a new approach to evaluate text-image alignment of generated images and how to use that feedback to improve the image quality. For this, a LLM is first used to parse the text description into individual assertions and the alignment of those individual assertions with the generated image is then measured via a VQA model. The resulting metric is shown to better correlate with human judgement than other metrics such as CLIP/BLIP score. Furthermore, the feedback of the VQA model can be used to increase the impact of specific parts of the promp during the denoising process which improves the text-image alignment of the generated image without requiring additional model training.", "questions": "While the approach works I wonder how much slower it is (as an evaluation metric) compared to other approaches such as CLIP. CLIP only requires a forward pass through the text and image encoder, whereas the new approach requires interactions with an LLM (to get the assertions) and then also needs to query a VQA model for each assertion.", "limitations": "Overall this is a good paper and the evaluation metric reminds me of an updated and improved version of the SOA score (Semantic object accuracy for generative text-to-image synthesis, TPAMI, 2020).", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper is well written and presented. The novel evaluation process is intuitive and automated, and shows better alignment with human judgement than other metrics. The use of the metric to improve tet-image alignment at test time is also shown and leads to better results than other approaches. The evaluation includes several human user studies highlighting the merits of the new approach.", "weaknesses": "The approach introduces the need for two additional networks (LLM and VQA model) into the pipeline which will incur additional cost and compute requirements. The overall quality and the resulting scores are likely also heavily dependent on the quality of the LLM and VQA model.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their positive feedback and are pleased that they find our work well written, intuitive and comprehensive with several user studies.\n\n\n\n>  The overall quality and the resulting scores are likely also heavily dependent on the quality of the LLM and VQA model.\n\n Yes, as discussed as limitation in the supplementary material,  we do note that the performance of our approach depends on assertion-parsing from the LLM and question-answering from the VQA model. Nevertheless, as shown through the quantitative results,   competitive results across a diverse range of input prompts are obtained in our current framework by using BLIP model for VQA and GPT-3.5-Turbo for text-prompt parsing.\n\n\n\nFurthermore, as noted in SM and response to Q1@sTZE, we find that while the use of a more stronger VQA model (e.g., BLIP2-VQA) can help improve overall question-answering performance, the use of relatively lightweight BLIP-VQA model (used in the paper) still allows us to achieve competitive performance without sacrificing runtime performance during the iterative refinement process.\n\n\n\n>   While the approach works I wonder how much slower it is (as an evaluation metric) compared to other approaches such as CLIP. \n\n Please see Table below for a detailed comparison of the average inference times when evaluating image-text alignment scores across a random sample of 1000 image-prompt pairs from the Decomposable-Captions-4k dataset. \n\nSince the overall inference time would also depend on the nature of LLM used and the API speed, average inference times with DA-Score are reported in 2 forms 1) DA-Score (with precomputed assertions): which reportes the average inference time when using precomputed assertions, and 2) DA-Score (Overall): where both assertions and VQA based scores are computed at inference time. We use parallel API requests to the \"gpt-3.5-turbo\" model with 64 processes to reduce the average overhead required for obtaining assertions from the LLM model.\n\n| Evaluation Method            | Avg. Inference Time (s) |\n| ---------------------------- | ----------------------- |\n| CLIP-Score [9]               | 0.061 sec               |\n| BLIP-Score [10]              | 0.079 sec               |\n| BLIP2-Score [11]             | 0.112 sec               |\n| BLIP-R ITM Score [10]        | 0.082 sec               |\n| BLIP2-R ITM Score [11]       | 0.116 sec               |\n| T2T-Score [7]                | 0.438 sec               |\n| DA-Score (Precomputed Ques.) | 0.329 sec               |\n| DA-Score (Overall)           | 0.457 sec               |\n\n\n\nWe observe that:\n\n1. While the proposed evaluation metric (DA-Score) is slower when compared to single model metrics (e.g. CLIP, BLIP, BLIP2), DA-Score with precomputed assertions is faster than T2T-Score (which first generates multiple captions for the generated image using BLIP-captioning model and then computes an average similarity score using CLIP), while exhibiting significantly higher correlation with human preference ratings (please refer Fig. 4 main paper).\n\n2. Also in context of iterative refinement, we note that the question generation needs to be done only once per text-prompt. Furthermore, the average time required for DA-Score computation is significantly lower than the runtime for a standard reverse diffusion process. This allows DA-Score to serve as a quick evaluation metric when performing iterative refinement, thereby retaining its usability for practical applications.\n\n   "}, {"review_id": "KlREH6sEMy", "reviewer": "Reviewer_sTZE", "summary": "The paper proposes a method called Decompositional-Alignment-Score (DA-Score) to assess and enhance the alignment between text and generated images during training. The authors accomplish this by breaking down each prompt or text into a series of assertions using a large language model (LLM). They then calculate correlation scores between each assertion and the generated image, employing a pre-trained VQA model. The final score, obtained by combining all the assertion scores, serves as an evaluation metric for text-image alignment. It further guides the generative models to produce more aligned images iteratively.", "questions": "*Method\n1. The performance of the proposed alignment score heavily relies on the accuracy of the VQA's answering results (also mentioned in the limitations by the authors). It raises the question of whether the used VQA model is sufficiently robust to handle all the given questions. If available, it would be helpful to provide quantitative results or an assessment of the accuracy of the VQA model in addressing the given questions on the dataset in this paper.\n2. It seems there is a lack of definition for the meaning of the variable $l$ in Eq.(5).\n\n*Experiment and Dataset\n1. How about the quality of the generated images? Based on the visual results presented in the paper (e.g., Figure 5), it appears that focusing solely on text-image alignment may lead to a reduction in image quality. It would be beneficial to provide additional results that consider image quality evaluation metrics such as FID and FID-CLIP.\n2. It seems that the generated images of the baseline generative models (e.g., stable diffusion) are not satisfied enough. I think it would be more realistic if use a more recent version like stable diffusion 2.1. Besides, it raises the question of whether the proposed alignment method remains effective or necessary when applied to more powerful text-to-image generation models, such as Imagen [1] or Parti [2].\n3. The paper does not provide clarity on the number of annotators involved in rating the samples within the proposed dataset. If only a small number of annotators, the obtained human annotations may lack sufficient credibility and be susceptible to subjective biases.\n\n[1] Saharia, Chitwan, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Information Processing Systems 35 (2022): 36479-36494.\n\n[2] Yu, Jiahui, et al. \"Scaling autoregressive models for content-rich text-to-image generation.\" arXiv preprint arXiv:2206.10789 (2022).\n", "limitations": "The authors discuss three limitations of the current version and provide the corresponding proposal for a solution.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper seeks to address a significant challenge in the text-to-image generation task, i.e., text-image alignment. The proposed framework adopts LLM and pre-trained multi-modal models, which is reasonable.", "weaknesses": "1. There is a concern regarding the performance of the VQA model, as it plays a crucial role in determining the accuracy of the proposed score. The effectiveness and reliability of the evaluation process heavily rely on the capabilities and performance of the VQA model.\n2. The experiments are not convincing enough and there and there is a lack of some specific details regarding the design of the dataset.\n\nPlease refer to more details in the following.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">   If available, it would be helpful to provide quantitative results or an assessment of the accuracy of the VQA model in addressing the given questions on the dataset in this paper. \n\n To analyse the accuracy of the VQA model's answering results, we perform an additional user-study wherein for each question-image pair, the human subject is asked to answer as yes, no or maybe (when answer is unclear). The accuracy of different VQA models is then predicted using the human labels as ground truths. Results are shown below.\n\n| VQA Model      | Accuracy % | Avg. Inference Time with DA-Score (s) |\n| -------------- | ---------- | ------------------------------------- |\n| VILT-VQA [13]  | 76.8%      | 0.224 s                               |\n| BLIP-VQA [10]  | 81.2%      | 0.329 s                               |\n| BLIP2-VQA [11] | 83.4%      | 0.512 s                               |\n\nWe observe that while using a better VQA model (e.g. BLIP-2) obtains a better answering accuracy, using a relatively lightweight BLIP-VQA model helps us reduce the overall inference time during evaluation while still maintaining competitive performance on the human-correlation scores (Fig. 4, main paper).\n\n>  It seems there is a lack of definition for the meaning of the variable $l$ in Eq.(5).\n\nApologies for the typo. As described in Alg. 2 (SM), Eq. 5 should be as follows (to increase the weight of assertion with lowest alignment score):\n\n$$w_i^{k+1} = \n    \\begin{cases}\n    w_i^k + \\Delta, \\ \\text{if} \\quad  i = \\text{argmin}_{l}  \\ u_l(\\mathcal{I},\\mathcal{P})\\\\\\\\\n    w_i^k \\quad \\text{otherwise}\n    \\end{cases},$$\n\n>  It would be beneficial to provide additional results that consider image quality evaluation metrics.\n\nSince the ground truth real images (required for FID) corresponding to typically creative and challenging prompts (e.g. a lion playing a piano) on the decomposable-captions dataset are not available, we use the following results to evaluate overall image quality:\n\n1. Quantitative Results: Similar to [8] we perform an additional user study for image fidelity, wherein given a pair of images (ours vs prior work), the human subject is asked to select the image with higher visual quality. The user-study was conducted across 1000 input-prompts randomly sampled from decomposable-captions dataset. Results are reported in Table below.\n\n2. Qualitative Results: We also provide additional qualitative results comparing the output image quality across different methods. Results are shown in Fig. 1 (global response).\n\n \n| Method                   | Win (Ours is better)   | Tie  | Loss (Ours is worse) |\n| ------------------------ | :--: | :--: | :--: |\n| Stable-Diffusion [1]     | 6.9  | 78.4 | 14.7 |\n| Composable-Diffusion [6] | 38.8 | 52.9 | 8.3  |\n| Attend-and-Excite [7]    | 23.6 | 69.7 | 6.7  |\n\nAs shown in Table above and Fig. 1 (global response), we observe that while the proposed iterative refinement approach does lead to a slight reduction in image quality when compared to Stable-Diffusion (6.9 % win, 78.4 % tie, 14.7 % lose), it still shows better visual quality than prior works like Attend-and-Excite and Composable-Diffusion. \n\nWe find that the reduction in visual quality typically occurs when performing large number of modifications (e.g. cross-attention updates) to the reverse diffusion process. In contrast with [6,7], our approach is able to adaptively adjust the number of such modifications by monitoring the proposed DA-Score. This helps us achieve better alignment results while still maintaining superior image quality as compared with prior works.\n\n\n>  It seems that the generated images of the baseline generative models (e.g., stable diffusion) are not satisfied enough. I think it would be more realistic if use a more recent version like stable diffusion 2.1. \n\nPlease note that in order to maintain consistency (implementation hyperparams) with prior works [Attend-Excite, Structure-Diffusion], we use the open-source Stable-Diffusion v1.4 model for our experiments. \n\nNevertheless, we note that the proposed approach is model-agnostic and easily applicable to other diffusion models. For instance, results with use of proposed iterative refinement on more recent Stable Diffusion models (v1.5 and v2.1) are shown in Fig. 2  (global response). We observe that while more recent versions of Stable-Diffusion show better performance, the generated images often have inconsistencies (e.g., pool and beach chair in example-1, bowtie and surfboard in example-2) with the input prompt. The proposed approach is able to detect such inconsistencies (with DA-Score) and gradually improve the final image output through iterative refinement.\n\n>  The paper does not provide clarity on the number of annotators involved in rating the samples within the proposed dataset.\n\nIn order to ensure the quality of collected fine-grain human ratings (scale: 1-5 between synthetic images and complex prompts with multiple subjects), the data annotations were collected from a group of 20 *\"expert\"* annotators (similar to contemporary work [a]) with experience in image-text rating tasks. An unlimited time is provided for each annotation.\n\nWe also verify the quality of the collected finegrain annotations by performing a secondary user-study for the simpler binary task of evaluating alignment accuracy (i.e. whether an image accurately matches the text prompt?) on a larger pool of 100 annotators. Results are shown below. We observe that the results collected among the larger group (with binary task of evaluating alignment accuracy) shows similar results to those obtained from the fine-grain expert human ratings (rating=5).\n\n|Method|Alignment Acc. [Expert Rating=5]|Alignment Acc. [Binary Task with 100 Subjects]|\n|---|---|---|\n|Stable-Diffusion [1] |43.66|45.73|\n|Attend-and-Excite [7] |65.50|65.05|\n|Ours (PW+CA)|74.16|73.24|\n\n[a] TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answering, Hu et al., 2023."}, {"review_id": "2di4MOcm3i", "reviewer": "Reviewer_bVCd", "summary": "This paper makes a twofold contribution: namely introducing a new metric for evaluating text-to-image alignment, as well as a method that builds upon this insight to improve the process of text-to-image generation. \nFor the first contribution of evaluating text-to-image alignment, the paper employs the strategy of using a Large Language Model (LLM) to decompose  the textual prompt into multiple components, and then using a Visual Question Answering (VQA) model to answer the individual components/assertions to obtain a final score for the alignment between the generated image and the textual prompt. \nTo improve the image generation process, the paper builds upon the existing approach Attend-and-Excite, that uses the cross-attention maps to focus on different tokens in the textual prompt. Here, the weights applied to the sub-prompts, as well as the strength of the cross-attention are used as parameters to be optimized in an iterative refinement procedure (while optimizing the decompositional metric) to obtain more faithful generated images.", "questions": "The only question I have is that all the experiments in this work are done on the Decomposable Prompts dataset. While it's large enough (with 4k prompts), I wonder if it is sufficient in terms of variety, of if it makes sense to also benchmark on other datasets such as DrawBench.", "limitations": "Yes, the limitations section in the supplementary material provides a fairly comprehensive overview of the possible limitations of the proposed method in the paper.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1) The paper tackles an important problem (of both evaluating text-image alignment, and improving text-to-image generation), and proposed interesting ideas for the same.\n2) The Prompt Decompositional Model intuitively makes a lot of sense, and quantitative results indicate that it seems to be significantly better than prior methods (i.e CLIPScore)\n3) Using the improved evaluation metric to improve the generation is a good idea. While in terms of novelty, it is almost entirely based off Attend-and-Excite and prior work that uses prompt weighting, optimizing it during test-time is new, and seems to bring good improvements. \n4) The experiments seems to be reasonably thorough. There are user studies that validate the key claims (of improved text-image alignment evaluation, and improved generation), as well as a new dataset of prompts.", "weaknesses": "One major concern I have with respect to the evaluation metric proposed in the paper is its similarity to [a]. While [a] is only an Arxiv preprint, and can definitely be considered as contemporary work, I would think that it is a good idea to atleast acknowledge it, since the idea behind breaking down a prompt into multiple Question-Answer pairs and then using a VQA model to evaluate it has been first introduced there. \n\nIn terms of the quantitative results, my worries are that the paper mostly relies on user studies/human evaluations to compare the proposed method against existing methods. While this is understandable (and quantitative results on the DAScore metric would be unfair, since the model is explicitly optimizing it), it might make sense to have quantitative results on other metrics (such as T2TScore) to demonstrate that the improvements are visible on automatic quantitative metrics. This would also allow benchmarking on other datasets beyond the Decomposable Prompts. \n\n[a] Hu et al. TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answering", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their positive feedback and are pleased that they find our work interesting, intuitive, tackling an important problem and with thorough experiments.\n\n\n\n>  While [a] is only an Arxiv preprint, and can definitely be considered as contemporary work, I would think that it is a good idea to acknowledge it.\n>\n>  [a] Hu et al. TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answering\n\n\n\n Thanks! We will update Sec. 2 (Related Work) to include the same as *contemporary* work.\n\nPlease also note that [a] only uses the idea of VQA for evaluation alone. The proposed approach further demonstrates how the interpretability of computed VQA scores can then be used to reliably improve the performance of text-to-image generation models.\n\n\n\n>  While this is understandable (and quantitative results on the DAScore metric would be unfair, since the model is explicitly optimizing it), it might make sense to have quantitative results on other metrics (such as T2TScore) to demonstrate that the improvements are visible on automatic quantitative metrics.\n\n\n\nWe predominantly use human-annotations/user-studies to avoid erroneous evaluation due to the weak-correlation between prior image-text alignment metrics and human preferences (Fig. 4, main paper).  Nevertheless, as seen in Table below, we do find that increases in DA-Score are also accompanied by increase in other evaluation metrics such as T2T Score during the iterative refinement process.\n\n\n\n| Method                   | T2T-Score |\n| ------------------------ | :-------- |\n| Stable-Diffusion [1]     | 72.11     |\n| Composable-Diffusion [6] | 71.19     |\n| Structure-Diffusion [8]  | 72.84     |\n| Attend-and-Excite [7]    | 74.23     |\n| Ours (PW + CA)           | 76.09     |\n\n\n\n\n\n>  While Decomposable-Captions dataset is large enough (with 4k prompts), I wonder if it is sufficient in terms of variety, or if it makes sense to also benchmark on other datasets such as DrawBench. \n\n\n\n**Diversity and challenging nature of Decomposable Captions Dataset** \n\nPlease note that as mentioned in Sec. 3 (main paper) and dataset details in App. C (Fig. 9, 10, 11 from SM), the decomposable-captions-4k dataset consists of prompts across a diverse range of complexities in both **1)** number of subjects, and, **2)** the realism of the input prompts. This allows us to evaluate the efficacy of our approach over a much wider data distribution as compared to prior works [e.g., Attend-and-Excite], which predominantly analyse the performance on relatively simple prompts with two subjects (e.g. object a and object b).\n\n\n\nThe diversity and challenging nature of the Decomposable-Captions dataset is also revealed by variation of the overall alignment performance on different subsets of the introduced dataset. For instance, while the alignment accuracy for Stable-Diffusion on the *easy* realism subset is as high as 73.4%,  as the *realism* difficulty increases the alignment accuracy drops down to 21.6%. This value further drops down to 9.94% when considering only prompts with *number of subjects = 5* and *realism difficulty = 'very hard'*. \n\n\n\nThe above variation (along with dataset details in App. C - Fig. 9, 10, 11), not only point to the diversity of the overall dataset but also provide an idea of the challenging nature of the introduced dataset.\n\n\n\n**Additional evaluation on DrawBench**\n\nIn addition to above, we also perform additional experiments evaluating the proposed approach on Drawbench prompts.\n\n\n\n**Results for DA-Score**:  We use the Stable Diffusion v1.5 model to generate images corresponding to the prompts from the DrawBench dataset (excluding rare words and misspellings as the same are not usable with VQA). Each prompt-image pair is then annotated by human experts on a scale of 1-5 (5: image is an accurate match for the prompt). The correlation results across different evaluation metrics are shown below. We observe that similar to decomposable-captions dataset, the DA-Score shows significantly higher correlation with human preferences as compared to prior evaluation metrics.\n\n| Evaluation Method              | Spearman's $\\rho$ (%) | Kendall's $\\tau$ (%) |\n| ------------------------------ | --------------------- | -------------------- |\n| BLIP-Score [10]                | 19.02                 | 14.07                |\n| BLIP2-Score [11]               | 21.49                 | 16.20                |\n| BLIP-R ITM Score [10]          | 20.14                 | 14.90                |\n| BLIP2-R ITM Score [11]         | 22.64                 | 16.99                |\n| T2T-Score [7]                  | 24.33                 | 18.13                |\n| DA-Score (with BLIP-VQA) | 55.01                 | 43.21                |\n\n\n\n**Results for Iterative Refinement**: \n\nWhile fullscale quantitative evaluation with human evaluation across all methods was not feasible during the response period, we provide detailed qualitative comparisons with prompts from the Drawbench dataset in Fig. 3 (global response pdf).  Similar to observations on the Decomposable-Captions-4k dataset, we find that the use of iterative refinement helps improve the text-to-image alignment across a diverse range of input prompts. \n\n\n\nPlease also note that while the proposed approach performs better than prior work, the improvements noticed with iterative refinement are dependent on the ability of the VQA model to detect such inconsistencies. Thus, prompts with inconsistencies not detectable by a VQA model e.g., misspellings, rare words etc, do not observe significant/noticeable improvement through the use of iterative refinement. \n\n"}, {"review_id": "gWhD8qWppq", "reviewer": "Reviewer_5iW3", "summary": "This paper proposes a decompositional approach to evaluate and improve a pre-trained text-to-image diffusion model given complex input prompts describing multiple objects and novel combinations. To that end, a LLM decomposes a difficult caption into disjoint assertions that can be individually evaluated using a VQA model. The assertion scores can be used to a) evaluate fine-grained image-text alignment and b) provide feedback for iterative refinement of generated images. The proposed metric correlates stronger with human ratings and the iterative refinement procedure increases the fidelity of generated images on complex input prompts.", "questions": "- TIFA is a recent, similar looking approach that uses VQA for evaluation. How does it differ?", "limitations": "- Yes, authors have extensively discussed limitations of the proposed method in the supplementary material.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- The paper is very well written and the method tackles an important problem\n- The proposed DA-score has clear motivation and correlates much better with human ratings\n- Using a LLM to produce a diverse pool of input prompts across varying complexity in terms of participating objects and realism is novel\n- Quantitative and visual results are strong. The resulting model produces images that align a lot better on unreal/creative input captions.", "weaknesses": "I don't have any substantial weaknesses but including early approaches to combining text-to-image with VQA models would enhance the comprehensiveness of the related work section.\n- https://dl.acm.org/doi/abs/10.1145/3372278.3390684\n- https://aclanthology.org/2020.lantern-1.2/", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their positive feedback and are pleased that they find our work well-written, novel, with clear motivation and having strong quantitative and visual results.\n\n\n\n>  I don’t have substantial weaknesses but including early approaches to combining text-to-image with VQA models would enhance the comprehensiveness of the related work section.\n\n\n\n Thanks! We will update the related work section to include the suggested references in Sec. 2.\n\n\n\n>TIFA is a recent, similar looking approach that uses VQA for evaluation. How does it differ?\n\n\n\n**Please note that as also mentioned by Reviewer bVCd, TIFA is a recent arXiv preprint which is concurrent to our work.**\n\n\n\nAdditionally, we note the following key differences between TIFA and the proposed approach:\n\n1. In contrast with the proposed approach, TIFA mainly uses the idea of VQA for evaluation alone. The proposed approach further demonstrates how the interpretability of theses VQA scores can then be used to reliably improve the performance of text-to-image generation models.\n\n2. TIFA directly uses the LLM model to generate questions with options. In contrast, we first break down the caption into a set of **disjoint** assertions, before rephrasing them into questions. In our experiments, we found the above approach to lead to less overlapping and redundant questions, which is especially critical for the use of DA-Score in the iterative refinement process. \n\n  \n\nFor instance, consider the following examples of question-generation outputs by TIFA and our approach: \n\n\n\n**Input Prompt1: \"a woman making a documentary in a coral reef with a camera and a school of fish\"**\n\nTIFA Questions: \n\n1. is there a woman?\n2. who is making a documentary?\n3. is the woman making a documentary?\n4. what is the woman doing?\n5. is there a coral reef?\n6. is there a camera?\n7. what is the woman using?\n8. is there a school of fish?\n9. what is in the water?\n\n*Total LLM Tokens: 3681*\n\n\n\n\nDA-Score Questions:\n\n1.  is there a woman in the image?\n2.  is the woman making a documentary?\n3.  is the image set in a coral reef?\n4.  does the woman have a camera?\n5.  is there a school of fish in the image?\n\n*Total LLM Tokens : 607*\n\n\n\n**Input Prompt2: \"a crab using a sewing machine\"**\n\nTIFA Questions: \n\n1. is this a crab?\n2. what animal is using the sewing machine?\n3. is there a sewing machine?\n4. what type of machine is this?\n5. is the crab using the sewing machine?\n\n*Total LLM Tokens: 3563*\n\n\n\nDA-Score Questions:\n\n1. is there a crab in the image?\n2. is there a sewing machine in the image?\n3. is the crab using the sewing machine?\n\n*Total LLM Tokens: 546*\n\n\n\nAs seen above, we find that TIFA typically has multiple questions pertaining to the same concept (e.g., woman in example-1, crab in example-2). It also uses a secondary question filtering model to filter through the initial list of questions. This increases the number of LLM tokens required during inference (~4-6x compared with DAScore) and also incurs additional runtime cost. In contrast, we first break down an input prompt into a set of **disjoint** assertions and then rephrase them as questions. This allows DA-Score to generate a more targeted set of questions with a faster runtime which is highly critical for using DA-Score for iterative refinement with T2I models.\n\n"}]

Author rebuttal:
We thank all the reviewers for their positive feedback and are pleased that they find our work well written [GK7B, 5iW3], intuitive [GK7B, bVCd],  tackling an important problem [sTZE, bVCd, 5iW3] and comprehensive with several user studies [GK7B,bVCd] and strong quantitative, visual results [5iW3].



We have tried our best to answer all the queries raised during the rebuttal through individual responses to each reviewer. Additionally, we also provide a global response pdf containing the requested results as summarised below:

* Additional image quality comparison results [sTZE]: Fig. 1
* Generalizability to more recent Stable-Diffusion versions [sTZE]: Fig. 2
* Additional results on DrawBench prompts [bVCd]: Fig. 3


We hope that the additional results / clarifications answer the questions raised during the discussion period.


Meta-review: This paper focuses on an important and challenging problem - the evaluation of text-to-image generation. All reviewers are positive about this submission. Three voted for Accept and one voted for a Borderline Accept. The strengths of this paper are clear and confirmed by all the reviewers, including well-written and presented, novel techniques and solid experiments. The authors also did a good rebuttal to solve reviewers' concerns on speed and similarity to some existing works. These valuable responses and discussions should be included in the final camera-ready version. Based on the reviewer's comments and discussion, the AC voted for acceptance of this submission.
Decision: Accept (poster)