SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Jaskirat Singh<sup>1</sup>
<sup>1</sup>The Australian National University
Liang Zheng $^{1,2}$
2Australian Centre for Robotic Vision
https://1jsingh.github.io/divide-evaluate-and-refine
The field of text-conditioned image generation has made unparalleled progress with the recent advent of latent diffusion models. While remarkable, as the complexity of given text input increases, the state-of-the-art diffusion models may still fail in generating images which accurately convey the semantics of the given prompt. Furthermore, it has been observed that such misalignments are often left undetected by pretrained multi-modal models such as  $CLIP$ . To address these problems, in this paper we explore a simple yet effective decompositional approach towards both evaluation and improvement of text-to-image alignment. In particular, we first introduce a Decompositional-Alignment-Score which given a complex prompt decomposes it into a set of disjoint assertions. The alignment of each assertion with generated images is then measured using a VQA model. Finally, alignment scores for different assertions are combined a posteriori to give the final text-to-image alignment score. Experimental analysis reveals that the proposed alignment metric shows significantly higher correlation with human ratings as opposed to traditional  $CLIP$ ,  $BLIP$  scores. Furthermore, we also find that the assertion level alignment scores provide a useful feedback which can then be used in a simple iterative procedure to gradually increase the expression of different assertions in the final image outputs. Human user studies indicate that the proposed approach surpasses previous state-of-the-art by  $8.7\%$  in overall text-to-image alignment accuracy.
The field of text-to-image generation has made significant advancements with the recent advent of large-scale language-image (LLI) models [1-5]. In particular, text-conditioned latent diffusion models have shown unparalleled success in generating creative imagery corresponding to a diverse range of free-form textual descriptions. However, while remarkable, it has been observed [6-8] that as the complexity of the input text increases, the generated images do not always accurately align with the semantic meaning of the textual prompt.
To facilitate the reliable use of current text-to-image generation models for practical applications, it is essential to answer two key questions: 1) Can we detect such fine-grain misalignments between the input text and the generated output in a robust manner? and 2) Once detected, can we improve the text-to-image alignment for failure cases? While several metrics for evaluating text-to-image alignment (e.g., CLIP [9], BLIP [10], BLIP2 [11]) exist, it has been observed [7, 12] that a high score with these metrics can be achieved even if the image does not fully correspond with input prompt. For instance, in Fig. 1, an output image (containing only pink trees) shows high CLIP/BLIP scores with the text "pink trees and yellow car" even if yellow car is not present. Evaluating text-to-image matching using the image-text-matching (ITM) head of BLIP models has also been recently explored [10, 11]. However, the generated scores also show a similar tendency to favor the main subject
Prompt: Pink trees and yellow car going through the mountains
Prompt: A woman practicing yoga in middle of a lake
Stable-Diffusion
a man wearing a scuba suit playing a eello underwater with fish swimming around.
Attend-and-Excite
a raccoon wearing a chef's hat preparing a gourmet meal in a kitchen.
Stable-Diffusion
Figure 1: Overview. Top: Traditional methods for evaluating text-to-image alignment e.g., CLIP [9], BLIP-2 [10] and BLIP2-ITM (which provides a binary image-text matching score between 0 and 1) often fail to distinguish between good (right) and bad (left) image outputs and can give high scores even if the generated image is not an accurate match for input prompt (missing yellow car). In contrast, by breaking down the prompt into a set of disjoint assertions and then evaluating their alignment with the generated image using a VQA model [10], the proposed Decompositional-Alignment Score (DA-score) shows much better correlation with human ratings (refer Sec. 4.1). Bottom: Furthermore, we show that the assertion-level alignment scores can be used along with a simple iterative refinement strategy to reliably improve the alignment of generated image outputs (refer Sec. 4.2).
Attend-and-Excite
of input prompt. Furthermore, even if such misalignments are detected, it is not clear how such information can be used for improving the quality of generated image outputs in a reliable manner.
To address these problems, in this paper we explore a simple yet effective decompositional approach towards both evaluation and improvement of fine-grain text-to-image alignment. In particular, we propose a Decompositional-Alignment-Score (DA-Score) which given a complex text prompt, first decomposes it into a set of disjoint assertions about the content of the prompt. The alignment of each of these assertions with the generated image is then measured using a VQA model [10, 13]. Finally, the alignment scores for different assertions are combined to give an overall text-to-image alignment score. Our experiments reveal that the proposed evaluation score shows significantly higher correlation with human ratings over prior evaluation metrics (e.g., CLIP, BLIP, BLIP2) (Sec. 4.1).
Furthermore, we also find that the assertion-level alignment scores provide a useful and explainable feedback for determining which parts of the input prompt are not being accurately described in the output image. We show that this feedback can then be used to gradually improve the alignment of the generated images with the input text prompt. To this end, we propose a simple iterative refinement procedure (Fig. 2), wherein at each iteration the expressivity of the least-aligned assertion is improved by increasing the weightage/cross-attention strength (refer Sec. 3.2) of corresponding prompt tokens during the reverse diffusion process. Through both qualitative and quantitative analysis, we find that the proposed iterative refinement process allows for generation of better aligned image outputs over prior works [6-8] while on average showing comparable inference times (Sec. 4.2).
Text to Image Generation Models. Text conditional image synthesis is a topic of keen interest in the vision community. For instance, [14-18] use GANs to perform text guided image generation. Similarly, [5, 19] explore the use of autoregressive models for zero-shot text to image generation. Recently, diffusion-based-models [1-5, 20, 21] have emerged as a powerful class of methods for performing text-conditional image synthesis over diverse range of target domains.
Figure 2: Iterative refinement (Col:1-3;4-6) for improving text-to-image alignment. We propose a simple iterative refinement approach which uses the decompositional alignment scores (refer Sec. 3.1) as feedback to gradually improve the alignment of the generated images with the input text-prompt.
While remarkable, generating images which align perfectly with the input text-prompt remains a challenging problem [6-8, 22]. To enforce, heavier reliance of generated outputs on the provided text, classifier-free guidance methods [2, 3, 23] have been proposed. Similarly, use of an additional guidance input to improve controllability of text-to-image generation have recently been extensively explored [24-35]. However, even with their application, the generated images are often observed to exhibit fine-grain misalignments such as missing secondary objects [6, 7] with the input text prompt.
Evaluating Image-Text Alignment. Various protocols for evaluating text-image alignment in a reference-free manner have been proposed [9-11]. Most prior works [2, 3, 5, 9] typically use the cosine similarity between the text and image embedding from large-scale multi-modal models [9, 36-38] such as CLIP [9], BLIP [10], BLIP-2 [11] for evaluating the alignment scores. Recently, [10, 11] also show the application of BLIP/BLIP-2 models for image-text matching using image retrieval. However, as shown in Fig. 1, these scores can give very high scores even if the generated images do not full align with the input text prompt. Furthermore, unlike our approach image-text alignment is often represented through a single scalar value which does not provide an explainable measure which can be used to identify/improve weaknesses of the image generation process.
Improving Image-Text Alignment. Recently several works [6-8] have been proposed to explore the problem of improving image-text alignment in a training free manner. Liu et al. [6] propose to modify the reverse diffusion process by composing denoising vectors for different image components. However, it has been observed [7] that it struggles while generating photorealistic compositions of diverse objects. Feng et al. [8] use scene graphs to split the input sentence into several noun phrases and then assign a designed attention map to the output of the cross-attention operation. In another recent work, Chefer et al. [7] extend the idea of cross-attention map modification to minimize missing objects but instead do so by modifying the noise latents during the reverse diffusion process. While effective at reducing missing objects, we find that the performance / quality of output images can suffer as the number of subjects in the input prompt increases (refer Sec. 4.2).
Besides training-free methods, recent contemporary work [39, 40] has also explored the possibility of improving image-text alignment using human feedback to finetune existing latent diffusion models. However this often requires the collection of large-scale human evaluation scores and finetuning the diffusion model across a range of diverse data modalities which can be expensive. In contrast, we explore a training free approach for improvement of fine-grain text-to-image alignment.
Given the image generation output  $\mathcal{I}$  corresponding to a text prompt  $\mathcal{P}$ , we wish to develop a mechanism for evaluation and improvement of fine-grain text-to-image alignment. The core idea of our approach is to take a decompositional strategy for both these tasks. To this end, we first generate a set of disjoint assertions regarding the content of the input prompt. The alignment of the output image  $\mathcal{I}$  with each of these assertions is then calculated using a VQA model. Finally, we use the
a couple wearing scuba gear having a tea party underwater with a school of fish
Figure 3: Method Overview. Given a text prompt  $\mathcal{P}$  and an initially generated output  $\mathcal{I}_0$ , we first generate a set of disjoint assertions  $a_i$  regarding the content of the caption. The alignment of the output image  $\mathcal{I}_0$  with each of these assertions is then calculated using a VQA model. Finally, we use the assertion-based-alignment scores  $u_i(\mathcal{I}_0, \mathcal{P})$  as feedback to increase the weightage  $w_i$  (of the assertion with least alignment score) in a parameterized diffusion model formulation  $\mathcal{D}$  (Sec. 3.2). This process can then be performed in an iterative manner to gradually improve the quality of the generated outputs until a desirable threshold for the overall alignment score  $\Omega(\mathcal{I}_k, \mathcal{P})$  is reached.
assertion-based-alignment scores as feedback to improve the expressiveness of the assertion with the least alignment score. This process can then be performed in an iterative manner to gradually improve the quality of generated outputs until a desired value for the overall alignment score is attained.
In the next sections, we discuss each of these steps in detail. In Sec. 3.1 we first discuss the process for evaluating decompositional-alignment scores. We then discuss the iterative refinement process for improving text-to-image alignment in Sec. 3.2. Fig. 3 provides an overview for the overall approach.
3.1 Evaluating Text-to-Image Alignment
Prompt Decomposition Model. Given an input prompt  $\mathcal{P}$ , we first decompose its textual information into a set of disjoint assertions (and corresponding questions) which exhaustively cover the contents of the input prompt. Instead of relying on human-inputs as in [6, 7], we leverage the in-context learning capability [41] of large-language models [42, 43] for predicting such decompositions in an autonomous manner. In particular, given an input prompt  $\mathcal{P}$  and large-language model  $\mathcal{M}$ , the prompt decomposition is performed using in-context learning as,
$$
\mathbf {x} = \left\{x _ {0}, x _ {1}, \dots x _ {n} \right\} = \mathcal {M} (\mathbf {x} \mid \mathcal {P}, D _ {\text {e x e m p l e r}}, \mathcal {T}), \tag {1}
$$
where  $\mathbf{x}$  is the model output,  $n$  is the number of decompositions,  $D_{exemplar}$  is the in-context learning dataset consisting 4-5 human generated examples for prompt decomposition, and  $\mathcal{T}$  is task description. Please refer supp. material for further details on exemplar-dataset and task-description design.
The model output  $\mathbf{x}$  is predicted to contain tuples  $x_{i} = \{a_{i},p_{i}\}$ , where each tuple is formatted to contain assertions  $a_{i}$  and the sub-part  $p_i$  of the original prompt  $\mathcal{P}$  corresponding to the generated assertion. For instance, given  $\mathcal{P} : \text{a cat and a dog}$  the prompt decomposition can be written as,
$\mathcal{M}(\mathbf{x}\mid \mathcal{P}:a$  cat and a dog',  $D_{exempler},\mathcal{T}) = [\{\textit{there is a cat},\textit{a cat}\} ,\{\textit{there is a dog},\textit{a dog}\} ]$
Computing Assertion-based Alignment Scores. We next compute the alignment of the generated image  $\mathcal{I}$  with each of the disjoint assertions using a Visual-Question-Answering (VQA) model [10]. In particular, given image  $\mathcal{I}$ , assertions  $a_i$ ,  $i = 1, \ldots, n$ , their rephrasing in question format  $a_i^q$  and VQA-model  $\mathcal{V}$ , the assertion-level alignment scores  $u_i(\mathcal{I}, a_i)$  are computed as,
$$
u _ {i} (\mathcal {I}, a _ {i}) = \frac {\exp \left(\alpha_ {i} / \tau\right)}{\exp \left(\alpha_ {i} / \tau\right) + \exp \left(\beta_ {i} / \tau\right)}, w h e r e \quad \alpha_ {i} = \mathcal {V} (\mathrm {` y e s `} | \mathcal {I}, a _ {i} ^ {q}), \quad \beta_ {i} = \mathcal {V} (\mathrm {` n o `} | \mathcal {I}, a _ {i} ^ {q}),
$$
where  $\alpha_{i},\beta_{i}$  refer to the logit-scores of VQA-model  $\nu$  for input tuple (image  $\mathcal{I}$ , question  $a_i^q$ ) corresponding to output tokens 'yes', 'no' respectively. Hyperparameter  $\tau$  controls the temperature of the softmax operation and controls the confidence of the alignment predictions.
Prior works on improving image-text alignment often rely on human-user inputs for expressing contents of the input prompt into its simpler constituents. For instance, Feng et al. [6] require the user to describe the prompt as a conjunction/disjunction of simpler statements. Similarly, Chefer et al. [7] require the user to provide a set of entities / subjects in the prompt, over which their optimization should be performed.
Prompt  $\mathcal{P}$  : A scuba diver playing a guitar underwater
Sub-prompts:  $\{p_t\}$
Disjoint Assertion Set:  $\{a_i\}$
Question Rephrasing:  $\{a_i^q\}$
a scuba diver.
The image shows a scuba diver.
Does the image show a scuba diver?
Playing a guitar
The scuba diver is playing a guitar.
Is scuba diver playing a guitar?
The setting of scene is underwater.
Is setting of scene underwater?
Prompt P : A horse on a beach wearing sunglasses
Sub-prompts:  $\{p_i\}$
The image shows a horse.
Does the image show a horse?
n a beach
The horse is on a beach
Is the horse on a beach?
Wearing sunglasses
The horse is wearing sunglasses.
Is the horse wearing sunglasses?
Figure 4: Visualizing the prompt decomposition process. By dividing a complex prompt  $\mathcal{P}$  into a set of disjoint assertions  $a_{i}$ , we are able to identify the sub-prompts  $p_i$  (circled) which are not expressed in the image output using VQA, and thereby address them using iterative refinement (Sec. 3.2).
Prompt P : A dolphin wearing a party hat at a pool party
adolphin
The image shows a dolphin.
wearing a party h
The dolphin is wearing a party hat.
Does the image show a dolphin?
at a pool party
The setting of image is a pool party.
Is dolphin wearing a party hat?
Is setting of image pool party?
Combining Alignment Scores. Finally, the assertion level alignment-scores  $u_{i}(\mathcal{I},a_{i})$  are combined to give the overall text-to-image alignment score  $\Omega (\mathcal{I},\mathcal{P})$  between image  $\mathcal{I}$  and prompt  $\mathcal{P}$  as,
$$
\Omega (\mathcal {I}, \mathcal {P}) = \frac {\sum_ {i} \lambda_ {i} (\mathcal {P} , a _ {i}) u _ {i} (\mathcal {I} _ {k} , a _ {i})}{\sum_ {i} \lambda_ {i} (\mathcal {P} , a _ {i})}, \tag {2}
$$
where weights  $\lambda_{i}(\mathcal{P},a_{i})$  refer to the importance of assertion  $a_{i}$  in capturing the overall content of the input prompt  $\mathcal{P}$ , and allows the user to control the relative importance of different assertions in generating the final image output<sup>2</sup>. Please refer Fig. 3 for the overall implementation.
3.2 Improving Text to Image Alignment
In addition to predicting overall text-to-image alignment score, we find that assertion-level alignment scores  $u_{i}(\mathcal{I},a_{i})$  also provide a useful and explainable way for determining which parts of the input prompt  $\mathcal{P}$  are not being accurately described in the output image  $\mathcal{I}$ . This feedback can then be used in an iterative manner to improve the expressivity of the assertion with least alignment score  $u_{i}(\mathcal{I},q_{i})$ , until a desired threshold for the overall text-image alignment score  $\Omega (\mathcal{I},\mathcal{P})$  is obtained.
Parameterized Diffusion Model. We first modify the image generation process of standard diffusion models in order to control the expressiveness of different assertions  $a_{i}$  in parametric manner. In particular, we modify the reverse diffusion process to also receive inputs weights  $w_{i}$ , where each  $w_{i}$  controls the relative importance of assertion  $a_{i}$  during the image generation process. In this paper, we mainly consider the following two methods for obtaining such parametric control.
Prompt Weighting. Instead of computing the CLIP [36] features from original prompt  $\mathcal{P}$  we use prompt-weighting [44] to modify the input CLIP embeddings to the diffusion model as,
$$
\operatorname {C L I P} (\mathcal {P}) = \mathcal {W} (\mathcal {P}, \left\{\operatorname {C L I P} \left(p _ {i}\right), w _ {i} \right\} _ {i = 1} ^ {n})) \tag {3}
$$
where  $\mathcal{W}$  refers to the prompt-weighting function from [1, 44],  $p_i$  refers to the sub-prompt (Sec. 3.1) corresponding to assertion  $a_i$ , and weights  $w_i$  control the relative weight of different sub-prompts  $p_i$  in computing the overall CLIP embedding for prompt  $\mathcal{P}$ .
Cross-Attention Control. Similar to [7], we also explore the idea of modifying the noise latents  $z_{t}$  during the reverse diffusion process, to increase the cross-attention strength of the main noun
For simplicity reasons, we mainly use  $\lambda_{i} = 1\forall i$  in the main paper. Further analysis on variable  $\lambda_{i}$  to account for variable information content or visual verifiability of an assertion are provided in supp. material.
Figure 5: Method comparisons w.r.t correlation with human ratings. We compare the correlation of different text-to-image alignment scores with those obtained from human subjects, as the number of subjects in the input prompt (refer Sec. 4) is varied. We observe that the proposed alignment score (DA-score) provides a better match for human-ratings over traditional text-to-image alignment scores.
subject for each sub-assertion  $a_i$ . However, instead of only applying the gradient update for the least dominant subject [7], we modify the loss for the latent update in parametric form as,
$$
z _ {t} = z _ {t} - \alpha \nabla_ {z _ {t}} \mathcal {L} \left(z _ {t}, \left\{w _ {i} \right\} _ {i = 1} ^ {n}\right)), \text {w h e r e} \quad \mathcal {L} \left(z _ {t}, \left\{w _ {i} \right\} _ {i = 1} ^ {n}\right) = \sum_ {i} w _ {i} \left(1 - \max  G \left(\mathcal {A} _ {i} ^ {t}\right)\right), \tag {4}
$$
where  $\alpha$  is the step-size,  $\mathcal{A}_i^t$  refers to the attention map corresponding to the main noun-subject in assertion  $a_i$ ,  $G$  is a smoothing function and weights  $w_{i}$  control the extent to which the expression of different noun-subjects in the prompt (for each assertion) will be increased in the next iteration.
Iterative Refinement. Given the above parametric formulation for controlling expression of different assertions, we next propose a simple yet effective iterative refinement approach towards improving text-to-image alignment. In particular, at any iteration  $k \in [1,5]$  during the refinement process, we first compute both overall text-image similarity score  $\Omega(\mathcal{I}_k, \mathcal{P})$  and assertion-level alignment scores  $u_i(\mathcal{I}_k, \mathcal{P})$ . The image generation output  $\mathcal{I}_{k+1}$  for the next iteration is then computed as,
$$
\mathcal {I} _ {k + 1} = \mathcal {D} \left(\mathcal {P}, \left\{w _ {i} ^ {k + 1} \right\} _ {i = 1} ^ {n}\right); \text {w h e r e} w _ {i} ^ {k + 1} = \left\{ \begin{array}{l l} w _ {i} ^ {k} + \Delta , & \text {i f} i = \operatorname {a r g m i n} _ {l} u _ {l} (\mathcal {I}, \mathcal {P}) \\ w _ {i} ^ {k} & \text {o t h e r w i s e} \end{array} \right., \tag {5}
$$
where  $\mathcal{D}$  refers to the parametrized diffusion model and  $\Delta$  is a hyper-parameter. This iterative process is then performed until a desirable threshold for the overall alignment score  $\Omega(\mathcal{I}_k, \mathcal{P})$  is reached. The image generation output  $\mathcal{I}^{\star}$  at the end of the refinement process is then computed as,
$$
\mathcal {I} ^ {\star} = \operatorname {a r g m a x} _ {\mathcal {I} _ {k}} \Omega (\mathcal {I} _ {k}, \mathcal {P}). \tag {6}
$$
Dataset. Since there are no openly available datasets addressing semantic challenges in text-based image generation with human annotations, we introduce a new benchmark dataset DecomposableCaptions-4k for method comparison. The dataset consists of 24960 human annotations on images generated using all methods [1, 6, 7] (including ours) across a diverse set of 4160 input prompts. Each image is a given rating between 1 and 5 (where 1 represents that 'image is irrelevant to the prompt' and 5 represents that 'image is an accurate match for the prompt').
Furthermore, unlike prior works [7] which predominantly analyse the performance on relatively simple prompts with two subjects (e.g. object a and object b), we construct a systematically diverse pool of input prompts for better understanding text-to-image alignment across varying complexities in the text prompt. In particular, the prompts for the dataset are designed to encapsulate two axis of complexity: number of subjects and realism. The number of subjects refers to the number of main objects described in the input prompt and varies from 2 (e.g., a cat with a ball) to 5 (e.g., a woman walking her dog on a leash by the beach during sunset). Similarly, the realism of a prompt is defined as the degree to which different concepts naturally co-occur together and varies as easy, medium, hard and very hard. easy typically refers to prompts where concepts are naturally co-occurring together (e.g., a dog in a park) while very hard refers to prompts where concept combination is very rare (e.g., a dog playing a piano). Further details regarding the dataset are provided in supplementary material.
Figure 6: Qualitative comparison w.r.t text-to-image alignment. We compare the outputs of our iterative refinement approach with prior works [1, 6-8] on improving quality of generated images with changing number of subjects (underlined) from 2 to 5. Please zoom-in for best comparisons.
4.1 Evaluating Text-to-Image Alignment