SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Royi Rassin
Bar-Ilan University, Israel  
rassinroyi@gmail.com
Eran Hirsch
Bar-Ilan University, Israel eran.hirsch@biu.ac.il
Daniel Glickman
Bar-Ilan University, Israel
danielglickman1@gmail.com
Shauli Ravfogel
Bar-Ilan University, Israel Allen Institute for AI, Israel shauli.ravfogel@gmail.com
Yoav Goldberg
Bar-Ilan University, Israel Allen Institute for AI, Israel yoav.goldberg@gmail.com
Gal Chechik
Bar-Ilan University, Israel  
NVIDIA, Israel  
gal.chechik@biu.ac.il
Text-conditioned image generation models often generate incorrect associations between entities and their visual attributes. This reflects an impaired mapping between linguistic binding of entities and modifiers in the prompt and visual binding of the corresponding elements in the generated image. As one example, a query like "a pink sunflower and a yellow flamingo" may incorrectly produce an image of a yellow sunflower and a pink flamingo. To remedy this issue, we propose SynGen, an approach which first syntactically analyses the prompt to identify entities and their modifiers, and then uses a novel loss function that encourages the cross-attention maps to agree with the linguistic binding reflected by the syntax. Specifically, we encourage large overlap between attention maps of entities and their modifiers, and small overlap with other entities and modifier words. The loss is optimized during inference, without retraining or fine-tuning the model. Human evaluation on three datasets, including one new and challenging set, demonstrate significant improvements of SynGen compared with current state of the art methods. This work highlights how making use of sentence structure during inference can efficiently and substantially improve the faithfulness of text-to-image generation.<sup>1</sup>
Diffusion models for text-conditioned image generation produce impressive realistic images [1, 2, 3, 4]. Users control the generated content through natural-language text prompts that can be rich and complex. Unfortunately, in many cases the generated images are not faithful to the text prompt [5, 6]. Specifically, one very common failure mode results from improper binding, where modifier words fail to influence the visual attributes of the entity-nouns to which they are grammatically related.
<sup>1</sup>We make our code publicly available https://github.com/RoyiRa/Syntax-Guided-Generation
Figure 1: Visual bindings of objects and their attributes may fail to match the linguistic bindings between entities and their modifiers. Our approach, SynGen, corrects these errors by matching the cross-attention maps of entities and their modifiers.
As an illustration, consider the prompt "a pink sunflower and a yellow flamingo". Given this prompt, current models often confuse the modifiers of the two entity-nouns, and generate an image of a yellow sunflower and a pink flamingo (Fig. 1, bottom left, semantic leak in prompt). In other cases, the attribute may semantically leak to areas in the image that are not even mentioned in the prompt (Fig. 1, bottom center, semantic leak outside prompt) or the attribute may be completely neglected and missed from the generated image (Fig. 1, bottom right, attribute neglect). Such mismatch can be addressed by providing non-textual control like visual examples [7, 8], but the problem of correctly controlling generated images using text remains open.
A possible reason for these failures is that diffusion models use text encoders like CLIP [9], which are known to fail to encode linguistic structures [10]. This makes the diffusion process "blind" to the linguistic bindings, and as a result, generate objects that do not match their attributes. Building on this intuition, we propose to make the generation process aware of the linguistic structure of the prompt. Specifically, we suggest to intervene with the generation process by steering the cross-attention maps of the diffusion model. These cross-attention map serve as a link between prompt terms and the set of image pixels that correspond to these terms. Our linguistics-based approach therefore aims to generate an image where the visual binding between objects and their visual attributes adheres to the syntactic binding between entity-nouns and their modifiers in the prompt.
Several previous work devised solutions to improve the relations between prompt terms and visual components, with some success [11, 12, 13]. They did not focus on the problem of modifier-entity binding. Our approach specifically addresses this issue, by constructing a novel loss function that quantifies the distance between the attention patterns of grammatically-related (modifier, entity-noun) pairs, and the distance between pairs of unrelated words in the prompt. We then optimize the latent denoised image in the direction that separates the attention map of a given modifier from unrelated tokens and bring it closer to its grammatically-related noun. We show that by intervening in the latent code, we markedly improve the pairing between attributes and objects in the generated image while at the same time not compromising the quality of the generated image.
We evaluate our method on three datasets. (1) For a natural-language setting, we use the natural compositional prompts in the ABC-6K benchmark [13]; (2) To provide direct comparison with previous state-of-the-art in [11], we replicate prompts from their setting; (3) Finally, to evaluate binding in a challenging setting, we design a set of prompts that includes a variety of modifiers and entity-nouns. On all datasets, we find that SynGen shows significant improvement in performance based on human evaluation, sometimes doubling the accuracy. Overall, our work highlights the effectiveness of incorporating linguistic information into text-conditioned image generation models and demonstrates a promising direction for future research in this area.
(a) Entity-Modifier Identification
Figure 2: The SynGen workflow and architecture. (a) The text prompt is analyzed to extract entity-nouns and their modifiers. (b) SynGen adds intermediates steps to the diffusion denoising process. In that step, we update the latent representation to minimize a loss over the cross attention maps of entity-nouns and their modifiers (Eq 3).
The main contributions of this paper are as follows: (1) A novel method to enrich the diffusion process with syntactic information, using inference-time optimization with a loss over cross-attention maps; (2) A new challenge set of prompts containing a rich number and types of modifiers and entities.
2 Syntax-Guided Generation
Our approach, which we call SynGen, builds on two key ideas. First, it is easy to analyze the syntactic structure of natural language prompts to identify bindings of entity-nouns and their modifiers. Second, one can steer the generation of images to adhere to these bindings by designing an appropriate loss over the cross-attention maps of the diffusion model. We describe the two steps of our approach: extracting syntactic bindings and then using them to control generation.
2.1 Identifying entity-nouns and their modifiers
To identify entity-nouns and their corresponding modifiers, we traverse the syntactic dependency graph, which defines the syntactic relation between words in the sentence. Concretely, we parse the prompt using spaCy's transformer-based dependency parser [14] and identify all entity-nouns (either proper-nouns or common-nouns) that are not serving as direct modifiers of other nouns.
These are the nouns that correspond to objects in the generated image. We then recursively collect all modifiers<sup>2</sup> of the noun into its modifier set. The set of modifier-labels includes a range of syntactic relations between nouns and their modifiers, such adjectivial modification (amod; “the regal dog”), compounds (compound; “the treasure map”), nominal modification through an intervening marker, adverbial modifiers (npadvmod; “A watermelon-styled chair”), adjectivial complement (acomp; “The apple is blue”), and coordination between modifiers (conj; “A black and white dog”).
2.2 Controlling generation with language-driven cross-attention losses
Consider a pair of a noun and its modifier. We expect the cross-attention map of the modifier to largely overlap with the cross-attention map of the noun, while remaining largely disjoint with the maps corresponding to other nouns and modifiers. To encourage the denoising process to obey these spatial relations between the attention maps, we design a loss that operates on all cross-attention maps. We then use this loss with a pretrained diffusion model during inference. Specifically, we
2We consider modifiers from the set  $\{\text{amod},\text{nmod},\text{compound},\text{npadvmod},\text{acomp},\text{conj}\}$ . We exclude conj when determining the top-level nouns.
Figure 3: Evolution of cross-attention maps and latent representation along denoising steps, for the prompt "a red crown and a golden strawberry". At first, the attention maps of all modifiers and entity-nouns are intertwined, regardless of the expected binding. During denoising, attention maps gradually become separated, adhering the syntactic bindings. The vertical line indicates that after 25 steps intervention stops, but the attention maps remain separated.
optimize the noised latents by taking a gradient step to reduce that loss. See illustration in Fig. 2. Fig. 3 illustrates the effect of the loss over the cross-attention maps.
Loss functions: Consider a text prompt with  $N$  tokens, for which our analysis extracted  $k$  noun-modifier sets  $\{S_1, S_2, \ldots, S_k\}$ . Let  $P(S_i)$  represent all pairs  $(m, n)$  of tokens between the noun root  $n$  and its modifier descendants  $m$  in the  $i$ -th set  $S_i$ . For illustration, the set of "A black striped dog" contains two pairs ("black", "dog") and ("striped", "dog"). Next, denote by  $\{A_1, A_2, \ldots, A_N\}$  the attention maps of all  $N$  tokens in the prompt, and denote by  $\text{dist}(A_m, A_n)$  a measure of distance (lack of overlap) between attention maps  $A_m$  and  $A_n$ .
Our first loss aims to minimize that distance (maximize the overlap) over all pairs of modifiers and their corresponding entity-nouns  $(m,n)$ ,
$$
\mathcal {L} _ {p o s} (A, S) = \sum_ {i = 1} ^ {k} \sum_ {(m, n) \in P \left(S _ {i}\right)} \operatorname {d i s t} \left(A _ {m}, A _ {n}\right). \tag {1}
$$
We also construct a loss that compares pairs of modifiers and entity-nouns with the remaining words in the prompt, which are grammatically unrelated to these pairs. In other words, this loss is defined between words within the (modifiers, entity-nouns) set and words outside of it. Formally, let  $U(S_{i})$  represent the set of unmatched words obtained by excluding the words in  $S_{i}$  from the full set of words and  $A_{u}$  is the corresponding attention map for a given unrelated word  $u$ . The following loss encourages moving apart grammatically-unrealized pairs of words:
$$
\mathcal {L} _ {n e g} = - \sum_ {i = 1} ^ {k} \frac {1}{| U (S _ {i}) |} \sum_ {(m, n) \in P (S _ {i})} \sum_ {u \in U (S _ {i})} \frac {1}{2} \left(d i s t \left(A _ {m}, A _ {u}\right) + d i s t \left(A _ {u}, A _ {n}\right)\right). \tag {2}
$$
Our final loss combines the two loss terms:
$$
\mathcal {L} = \mathcal {L} _ {\text {p o s}} + \mathcal {L} _ {\text {n e g}}. \tag {3}
$$
For a measure of distance between attention maps we use a symmetric Kullback-Leibler divergence  $dist(A_i, A_j) = \frac{1}{2} D_{KL}(A_i || A_j) + \frac{1}{2} D_{KL}(A_j || A_i)$ , where  $A_i, A_j$  are attention maps normalized to a sum of 1, i and  $j$  are generic indices, and  $D_{KL}(A_i || A_j) = \sum_{\text{pixels}} A_i \log (A_i / A_j)$ .
Our test-time optimization approach resembles the one of [11], which defined a loss over the cross-attention maps to update the latents at generation time. However, their loss aims to maximize the presence of the smallest attention map at a given timestep to guarantee a set of selected tokens is included in the generated image, and our loss depends on pairwise relations of linguistically-related words and aims to align the diffusion process to the linguistic-structure of the prompt.
2.3 The workflow
We use the loss of Eqs 1-3 to intervene in the first 25 out of 50 denoising steps. Empirically, using a smaller number of steps did not correct well improper binding, and using a larger number generated blurred images, as detailed in Appendix B. In each of the first 25 steps, a pretrained denoiser (U-Net) was first used to denoise the latent variable  $z_{t}$ . Then, we obtained the cross-attention maps as in [15]. Next, we used the loss  $\mathcal{L}$  to update the latent representation  $z_{t}$  with a gradient step  $z_{t}^{\prime} = z_{t} - \alpha \cdot \nabla_{z_{t}}\mathcal{L}$ . Finally, the U-Net architecture denoises the updated latent variable  $z_{t}^{\prime}$  for the next timestep.
3.1 Compared baseline methods
We compare SynGen with three baseline methods. (1) Stable Diffusion 1.4 (SD) [1]; (2) Structured Diffusion [13], extracts noun- phrases from the prompt and embeds them separately, to improve the mapping of the semantics in the cross-attention maps; and (3) Attend-and-Excite (A&E) [11], a method that given a predetermined set of tokens, updates the latent a certain number of timesteps, to eventually incorporate these tokens in the generated image. To automate token selection in A&E, we follow the recommendation by the authors to select the nouns using a part-of-speech tagger.
We evaluate our approach using two existing benchmark datasets, and one new dataset that we designed to challenge methods in this area.
(1) ABC-6K [13]. This benchmark consists of 3.2K natural compositional prompts from MSCOCO [16], which were manually written by humans, using natural language and contain at least two color words modifying different noun-entities. In addition, the dataset contains 3.2K counterparts, where the position of modifiers in the original prompts are swapped. (e.g., "a white bench in front of a green bush" and "a green bench in front of a white bush"). We randomly sample 600 prompts.
(2) Data from Attend-and-Excite [11]. Originally introduced to evaluate the A&E method which focuses on entity-neglect, this dataset also showed that A&E improved over previous work in terms of improper binding.
Prompts in this dataset belong to three categories: (1) "a {color} {in-animate object} and a {color} {in-animate object}"; (2) "a {color} {in-animate object} and an {animal}"; (3) "an {animal} and an {animal}". Following the split in A&E, we sample 33 prompts from type (1) and 144 prompts from type (2), but exclude type (3), as it does not contain modifiers. This is a very simple dataset, which we use to facilitate direct comparison with previous work.
(3) Diverse Visual Modifier Prompts (DVMP). The above two datasets are limited in terms of number and types of modifiers, and the number of entity-nouns per prompt. To challenge our model, we design a dataset consisting of coordination sentences, in similar fashion to the dataset from A&E, but with strong emphasis on the number and types of modifiers per prompt. Specifically, we aim to compare the models with prompts that contain numerous and uncommon modifiers, creating sentences that would not usually be found in natural language or training data, such as "a pink spotted panda". DVMP was designed with two key aspects in mind:
Expanding the set of modifiers: We have extended the number of modifiers referring to an entity-noun from one to up to three. For instance, "a blue furry spotted bird". We also added types of modifiers besides colors, including material patterns ("a metal chair"), design patterns ("a checkered shoe"), and even nouns modifying other noun-entities ("a baby zebra").
Visually verifiable and semantically coherent: The modifiers selected for DVMP are visually verifiable, with a deliberate avoidance of nuanced modifiers. For instance, "big" is a relative modifier dependent on its spatial context, and emotional states, such as in the prompt "an excited dog", are largely excluded due to their subjective visual interpretation. Simultaneously, DVMP maintains semantic coherence by appropriately matching modifiers to noun-entities, thereby preventing the creation of nonsensical prompts like "a sliced bowl" or "a curved zebra".
In total, we have generated 600 prompts through random sampling. For a comprehensive description of the dataset's creation, see Appendix F.
3.3 Human Evaluation
We evaluate image quality using Amazon Mechanical Turk (AMT). Raters were provided with a multiple-choice task, consisting of a single text prompt and four images, each generated by the baselines and SynGen. Raters could also indicate that all images are "equally good" or "equally bad". We provided each prompt and its corresponding generations to three raters, and report the majority decision. In cases where there is no majority model winner, we count it toward "no majority winner".
We evaluate generated images in two main aspects: (1) concept separation (sometimes known as editability [17]) and (2) visual appeal. Concept separation refers to the ability of the model to distinctly depict different concepts or objects in the generated image. The effectiveness of concept separation is assessed by asking raters, "Which image best matches the given description?". To assess visual quality, raters were asked "Which image is more visually appealing?". To maintain fairness and reduce biases, the order of images was randomized in each task. Full rater instructions and further details are provided in Appendix G.1 of the supplemental materials.
We also experimented automatic evaluation, but find its quality subpar. For standardized evaluation purposes, it is detailed in Appendix G.2.
Fine-grained evaluation. In addition to a multiple-choice task, we evaluate concept separation using the following key metrics: (1) Proper Binding, quantifying how well the model associates attributes with their corresponding objects; (2) Improper Binding, measuring the instances where attributes are incorrectly linked to unrelated objects; and (3) Entity Neglect, capturing the frequency with which the model omits entities specified in the prompt.
To this end, we randomly select 200 prompts each from the DVMP and ABC-6K datasets, while using all 177 prompts available in the A&E dataset. Human evaluators were asked to mark if instances have correct or incorrect attribute-object mapping. Importantly, incorrect mappings are counted on a per-attribute basis—multiple incorrect mappings of a single attribute are considered one violation. For example, in the prompt "the white dog chased the cat up the tree", if the modifier "white" is incorrectly mapped to both "cat" and "tree", it is counted as one instance of violation. Evaluators also identify the number of entities mentioned in the prompt that are subsequently depicted in the generated image.
Based on these counts, we define the metric of Proper Binding as the ratio of correctly mapped attributes to the total number of attributes. Similarly, Improper Binding is defined as the ratio of incorrectly mapped attributes to the total number of attributes, while Entity Neglect is the complement of the ratio of mentioned entities that are depicted in the generated image to the total number of entities in the prompt. Rater instructions are provided in Appendix G.1.
4.1 Quantitative Results
Table 1 provides results of the comparative experiment. SynGen is consistently ranked first in all three datasets, and by a large margin, sometimes double the approval rate of the second ranked method, A&E. These results are observed for concept separation, which measures directly the semantic leak, and for visual appeal.
The high number of "no winner" cases reflects the large difficulty of some of the prompts, for which no method provides good enough generated images. Population results before majority aggregation are given in Appendix G.1 of the supplemental material. Comparisons with StableDiffusion are given in Fig. 19 of the supplemental.
Table 2 provides results of the individual experiment. We find that SynGen outperforms all models by a landslide in both proper and improper binding and is on par with state-of-the-art on entity neglect [11], despite not directly tackling this problem.
Dataset	Model	Concept Separation	Visual Appeal
A&E	SynGen (ours)	38.42	37.85
A&E	18.08	18.65
Structured Diffusion	04.52	04.52
Stable Diffusion	01.69	02.26
No majority winner	37.29	36.72
DVMP (challenge set)	SynGen (ours)	24.84	16.00
A&E	13.33	12.17
Structured Diffusion	04.33	07.83
Stable Diffusion	03.83	07.17
No majority winner	53.67	56.83
ABC-6K	SynGen (ours)	28.00	18.34
A&E	11.17	10.00
Structured Diffusion	05.83	06.33
Stable Diffusion	04.83	07.83
No majority winner	50.17	57.50
Table 1: Human evaluation of all methods on the three datasets. The table reports scores for concept separation (how well the image matches the prompt) and visual appeal. Values are the fraction of majority vote of three raters, normalized to sum to 100.
Table 2: Results of the fine-grained concept separation experiment. Proper Binding should be maximized to 100, while Improper Binding and Entity Neglect should be minimized to 0.
Dataset	Model	Proper Binding ↑ Improper Binding ↓ Entity Neglect ↓
A&E	SynGen (ours)	94.76	23.81	02.82
A&E	81.90	63.81	01.41
Structured Diffusion	55.71	67.62	21.13
Stable Diffusion	59.05	68.57	20.56
DVMP (challenge set)	SynGen (ours)	74.90	19.49	16.26
A&E	52.47	31.64	10.77
Structured Diffusion	48.73	30.57	28.46
Stable Diffusion	47.80	30.44	26.22
ABC-6K	SynGen (ours)	63.68	14.37	34.41
A&E	56.26	26.43	33.18
Structured Diffusion	51.47	29.52	34.57
Stable Diffusion	52.70	27.20	36.57
Figures 4-6 provide qualitative examples from the three datasets, comparing SynGen with the two strongest baselines.
The qualitative examples illustrate several failure modes of our baselines. First, semantic leak in prompt, occurs when a modifier of an entity-noun "leaks" onto a different entity-noun in the prompt, as shown in Fig. 4, for the prompt "a pink clock and a brown chair", in columns 3 and 4. In this case, all baselines incorrectly apply pink hues to the chair, despite the prompt explicitly defining it as brown. A more nuanced variant of this issue is semantic leak out of prompt, when a modifier is assigned to an entity-noun that is not mentioned in the prompt. For instance, the "spiky" attribute in "a spiky bowl and a green cat" leaks to a plant, which is not in the prompt, or the green coloration in the background of the images generated by the baselines, as seen in columns 5 and 6 in Fig. 5.
Attribute neglect occurs when a modifier from the prompt is absent from the generated image. As exhibited in Fig. 4, for "a frog and a brown apple", both baselines do not include a brown color at all.
Entity casting is another failure type where a modifier is treated as a standalone entity, a phenomenon commonly observed with noun modifiers. For example, the prompt "a wooden crown and a furry baby
Figure 4: Qualitative comparison for prompts from the Attend-and-Excite dataset. For every prompt, the same three seeds are used for all methods.
Figure 5: Qualitative comparison for prompts from the DVMP dataset. For every prompt, the same three seeds are used for all methods.
rabbit" (column 1 in Fig. 5) has all methods, apart from ours, generate human infants. Presumably, this occurs because "baby" is interpreted as a noun rather than as a modifier, leading other methods to treat it as a separate object due to the lack of syntactic context. Conversely, SynGen correctly interprets "baby" as a modifier and accurately binds it to the rabbit. Similarly, in the prompt "a white fire hydrant sitting in a field next to a red building" (column 6 in Fig. 6), "fire" is wrongly interpreted as an entity-noun, which leads to the unwarranted inclusion of a fire in the scene.