SlowGuess/NeurIPS_2023_OCR huggingface dataset:
All methods, barring SynGen, grapple with entity entanglement [18, 19, 20, 21, 22], where some objects tend to strongly associate with their most common attribute (e.g., tomatoes are typically red). This is evident in columns 3 and 4 in Fig. 6, where other methods fail to visually associate the blue attribute with the dog in "a blue and white dog sleeps in front of a black door". Instead, they resort to typical attributes of the objects, generating a black and white dog.
Further qualitative analysis is provided in Appendix D.1.
Figure 6: Qualitative examples for ABC-6K prompts. For every prompt, all methods use the same three seeds.
The importance of using both positive and negative losses. We evaluated the relative importance of the two terms in our loss Eq. (3). The positive term  $\mathcal{L}_{pos}$ , which encourages alignment of the attention map of an object and its modifiers, and the negative loss term,  $\mathcal{L}_{neg}$ , which discourages alignment with other modifiers and objects. We sampled 100 prompts from the DVMP dataset and generated images with and without each of the two loss terms. See example in Fig. 7. Then, raters were asked to select the best of four variants. Table 3 shows that raters preferred the variant that combined both the positive and the negative terms. More examples are given in the supplemental Appendix B.
Loss	Concept Separation	Visual Appeal
Both losses Lpos + Lneg	27	22
Positive only Lpos	0	11
Negative only Lneg	3	35
Stable Diffusion	4	28
No majority winner	66	4
Table 3: Ablation of loss components. Values are percent preferred by human raters.
Figure 7: Ablation of loss components. Removing  $\mathcal{L}_{neg}$  results in semantic leakage (the bird is white) and entity neglect (there is no crown). Removing  $\mathcal{L}_{pos}$  also leads to semantic leakage (generating a bird and background with white parts), and failed attribution binding (generating a crown that is not white).
Semantic leakage. [2] pointed out cases of semantic leakage in diffusion models, where properties of one entity-noun influence the depiction of another. [23] attributed this issue to a lack of understanding of syntax, specifically noting failures when processing texts requiring subtle syntactic binding comprehension. [6] identified semantic leakage issues in DALL-E, where properties of one entity-noun influence how other entity nouns are depicted. In this work, we pinpoint semantic leakage as a consequence of improper mapping between syntactic and visual binding.
Attention-based interventions. [15] demonstrated that the cross-attention mechanism determines the spatial layout of entities in generated images. This result suggested that cross-attention is causally involved in the aforementioned issues. A&E [11] addresses the problem of entity omission, where certain entities mentioned in the prompt do not appear in the generated image. They propose a loss function that encourages each noun token in the image to significantly attend to a corresponding image patch, thereby preventing its omission. Our approach is similar to [11] in that it updates the latent representation through a loss function over attention maps, during image generation.
Syntax-based generation was also explored in [13], proposing the Structured Diffusion method. It aims to address the problem of missing entities and semantic leakage of attributes. This is achieved by parsing the prompt, extracting phrases corresponding to nouns and modifiers, and encoding them separately. They also intervene in the attention patterns, ensuring that each individual phrase influences the attention patterns. Our experiments show that it is better to implicitly influence the attention patterns through our loss which we dynamically optimize. In contrast, their intervention remains fixed.
Concurrent to this work, [24] proposed an alternative approach to combine syntactic control and attention-based optimization. They extract nouns from prompts and train a layout predictor to identify the corresponding pixels for each noun. Then, they optimize the latents by encouraging the pixels corresponding to the objects to attend to CLIP representations of phrases containing those objects. While similar in spirit, the current paper demonstrates intervention in the generation process solely based on syntax, without explicitly learning the correspondence between image entities and tokens.
Like previous methods, the performance of SynGen degrades with the number of attributes to be depicted (see supplemental Fig. 12). However, its decline is remarkably less pronounced compared to other methods. This decay in performance can be attributed to two primary factors: (1) an image begins to lose its visual appeal when the negative loss term becomes excessively large; (2) an overly cluttered image poses challenges in crafting a cohesive "narrative" for all the concepts. We expect that some of these issues can be addressed with more hyper-parameter tuning.
Naturally, the effectiveness of our method is intrinsically tied to the quality of the parser. When the parser fails to extract the stipulated syntactic relations, our method essentially operates akin to SD.
Finally, SynGen takes longer to generate images with modifiers in the prompt than SD and slightly slower than than A&E (see Appendix A).
In this work, we target the improper binding problem, a common failure mode of text-conditioned diffusion models, where objects and their attributes incorrectly correspond to the entity-nouns and their modifiers in the prompt. To address it, we propose SynGen, an inference-time intervention method, with a loss function that encourages syntax-related modifiers and entity-nouns to have overlapping cross-attention maps, and discourages an overlap from cross-attention maps of other words in the prompt. We challenge our method with three datasets, including DVMP – a new dataset that is specially-designed to draw out hard cases of improper-binding problem. Our method demonstrates improvement of over  $100\%$  across all three datasets over the previous state-of-the-art. Finally, our work highlights the importance of linguistic structure during denoising for attaining faithful text-to-image generation, suggesting promising avenues for future research.
This study was funded by a grant to GC from the Israel Science Foundation (ISF 737/2018) and an equipment grant to GC and Bar-Ilan University from the Israel Science Foundation (ISF 2332/18). This project has also received funding from the European Research Council (ERC) under the European Union's Horizon 2020 research and innovation programme, grant agreement No. 802774 (iEXTRACT). Shauli Ravfogel is grateful to be supported by the Bloomberg Data Science PhD Fellowship.
[22] Ishan Misra, Abhinav Gupta, and Martial Hebert. From red wine to red tomato: Composition with context. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1160-1169, 2017.
[23] Evelina Leivada, Elliot Murphy, and Gary Marcus. Dall-e 2 fails to reliably capture common syntactic processes. arXiv preprint arXiv:2210.12889, 2022.
[24] Qiucheng Wu, Yujuan Liu, Handong Zhao, Trung Bui, Zhe Lin, Yang Zhang, and Shiyu Chang. Harnessing the spatial-temporal attention of diffusion models for high-fidelity text-to-image synthesis. arXiv preprint arXiv:2304.03869, 2023.
Computing resources. Experiments were run on an NVIDIA DGX Station with four v100-SXM2-32GB GPUs. The overall duration of all experiments in the paper was about two weeks.
Efficiency. SynGen takes  $\sim 144\%$  longer than SD and  $\sim 10.9\%$  longer than A&E to generate images with modifiers in the prompt. To arrive at these numbers, we randomly sampled a set of 50 images from the A&E dataset, the DVMP set, and ABC-6K and timed the generations for each method. On average, SD needs 4 seconds to generate an image, A&E 8.8 seconds, and SynGen 9.76 seconds.
Hyperparameters. The hyperparameters we used consist of 50 diffusion steps, a guidance scale of 7.5, a scale-factor of 20, and 25 latent update steps. The choices of scale factor and latent update steps are described in Appendix B.2 and Appendix B.3 respectively.
Parser. Throughout this project, we use the spacy parser with the out-of-the-box en_core_web_trf model.
Attending word pieces. When a relatively uncommon word is encountered by the tokenizer of the text encoder, it is split to sub-words (i.e., word pieces). In the context of our loss function, when an entity (or modifier) is split into word pieces, we compute our distance function (the Symmetric-KL) for each word piece. Then, only the word piece that maximizes the distance is added to the loss.
Cross-attention maps. We describe more formally the cross-attention maps on which we intervene. Let  $N$  be the number of tokens in the prompt, and let  $D^2$  be the dimensionality of the latent feature map in an intermediate denoising step. The denoising network defines a cross-attention map  $A^{patches\rightarrow tokens}\in \mathbb{R}^{D^2\times N}$  between each of  $D^2$  patches in the latent feature map and each token. Intuitively, the attention maps designate which tokens are relevant for generating each patch.
Our goal is to derive an attention distribution  $A^{tokens \to patches} \in R^{N \times D^2}$  such that its  $i$ -th row  $A_i^{tokens \to patches}$  contains the attention distribution of token  $i$  over patches. For this goal, we define a score matrix  $S$  to be the transpose of  $A^{patches \to tokens}$ , i.e., a matrix whose  $i^{th}$  row contains the attention scores from each patch to token  $i$ . Since  $S$  is not normalized, we divide each row by its sum to get a distribution over patches. Unless stated otherwise, across the paper, we refer to  $A^{tokens \to patches} \in R^{N \times D^2}$  when mentioning the "cross-attention maps"  $A$  and its  $i^{th}$  row  $A_i$  corresponding to the attention map from the  $i^{th}$  token.
B Additional Ablation Experiments
B.1 Further Investigation of the Positive and Negative Loss Terms
In Section 4.3, we investigate the importance of the positive and negative loss function terms using a human rater. Here, we accompany the rating with a qualitative analysis, to examine the effect of each term. To this end, we generate images for 15 randomly selected prompts, five from each dataset. Fig. 8 depicts a sample of the generated prompts.
We find that proper binding necessitates both the positive and negative terms: excluding the negative term from the loss function results in two noteworthy observations. First, the number of missing objects increase, evident by the missing crown, cat, metal chair, and tomato, in columns 1, 2, 4, and 5 in Fig. 8. One consequence of missing objects is the apparent improper binding, indicated by the red backpack and black shirt in columns 1 and 3.
On the other hand, excluding the positive term results in fuzzier separation between objects. For instance, the cat is not completely formed, and is "merged" with the pillow; and while it appears that there is some green residue on the dog, it is not colored green. Moreover, the grass is green, which indicates a semantic leakage.
Figure 8: We examine the effect of employing only one of the two losses instead of both. All images were generated using the same random seed.
Putting these insights together, we observe that to some extent, the effect the loss terms is complementary. In addition to the increase of objects and proper binding, the images are more coherent (less cases of objects mixed into each other, such as the cat in the only-negative loss or the elephant in the only-positive loss).
B.2 Number of Timesteps for Intervention
Recall that our method intervenes in latent denoising generation. In this appendix, we study the effect of the hyperparameters determining the number of steps in which we intervene.
To identify an ideal number of timesteps to intervene, we experiment with 100 randomly selected prompts from the DVMP dataset, a fixed random seed, and a number of update steps from 5 to 50, in increments of 5. Examples of this experiment are shown in Fig. 9.
We observe that when intervening in a small number of timesteps, our method failed to adequately mitigate semantic leakage or that images are not completely formed. For instance, the apple in column 1 in the 15-steps example is cartoon-ish, while the dog is not. Conversely, intervening for the full 50 timesteps resulted in an increase rate of blurred images (potentially due to the significant modification of the latent, which shifts it away from the learned distribution). We conclude that the optimal number of timesteps for intervention is 25, as this allows for effective mitigation of improper binding, while still generating visually appealing images.
B.3 Setting the Scale Factor
The scale factor affects the update step size. Recall the update step stated in Section 2  $z^{\prime}t = z_{t} - \alpha \cdot \nabla z_{t}\mathcal{L}$ . Here,  $\alpha$  is the scale-factor.
To determine a good selection for the scale-factor, we generate 100 randomly sampled prompts from the DVMP dataset, with a scale-factor value from 1 to 40, in increments of 10.
As can be seen in Fig. 10, we observe that merely updating the latent using a scale-factor of 1 yields relatively good results in terms of improper binding, which confirms the utility of our loss function. However, such a low scale-factor also consistently leads to missing objects.
Figure 9: We experiment with varying number of diffusion steps and examine the effect of changing the number of diffusion steps for which we intervene with the cross attention maps. All images were generated using the same random seed.
Interestingly, for greater scale-factor values, the generations become alike in their overall look, but are nonetheless very different. As an example, for both values, 10 and 30, the sliced banana is missing from the image in column 2, but the 30-value does result in a spotted teal skateboard. In column 3, values below 20 lead to images that contain two pandas (none of which are spotted), which indicates the proper binding process, and that the latent was not updated enough. On the other hand, a value greater than 20 leads to an image of a striped rabbit, instead of a spotted rabbit.
One interesting conclusion from this experiment is that the greater the scale-factor, the stronger the concept separation. However, this is only true to a point. For a great enough value, generations become too blurred or simply lose their visual appeal.
C Additional Quantitative Analyses
To study the efficacy of SynGen relative to the baselines in improper binding setting, we analyze the results under three perspectives. (1) as a function of repeating entities and modifiers; (2) as a function of the number of modifiers; and (3) degree of entanglement. Samples of generations are shown in Fig. 14.
Number of repeating modifiers and entities. In this analysis, we examine the performance of all methods for prompts containing recurring modifiers (e.g., "a sliced strawberry and a sliced tomato) or entities (e.g., "a sliced tomato and a skewered tomato"). Aggregated results are illustrated in Fig. 11. Our observations reveal a decrease in performance across all methods when modifiers are repeated. However, the relative success between SynGen and the baselines in performance remains the same. Moreover, there is no substantial decline in results when entities are repeated.
Number of modifiers in prompt. We hypothesize that since our method is specifically designed to tackle improper binding, it handles prompts containing many modifiers with more success. This is
Figure 10: Qualitative comparison between scale factor values for SynGen. For every prompt, the same seeds are applied. We anecdotally show our scale-factor value (we use the value 20) provides superior results.
confirmed in Fig. 12, which shows the gap between SynGen and the baselines widens as the number of modifiers in a prompt increases.
Entangled entities. As we describe in Section 4.2, entangled entities are strongly associated with their most common attribute. For instance, a tomato is typically red, and thus, it is common for images to depict red tomatoes.
We categorize the prompts into three groups: (1) entangled prompts, which contain entangled entities with a modifier that overrides a common modifier (e.g., a purple strawberry); (2) common entangled prompts, which contain entangled entities with their common modifiers; and (3) neutral prompts, which do not contain entangled entities at all. Performance as a function of these groups is demonstrated in Fig. 13.
D Additional Qualitative Results
D.1 Qualitative analysis by number of modifiers
In Fig. 15, examples from the DVMP challenge set include 2 to 6 modifiers.
While errors of all types are prevalent regardless of the number of modifiers, their frequency tends to rise as more modifiers are added.
As for SynGen, although it does not display semantic leakages at an increased rate compared to the baselines (as quantitatively demonstrated in Fig. 12), it does show a tendency to generate more than the specified number of entities as the modifier count increases. This behavior is observable in rows 8 and 10 for SynGen, and in rows 7 through 10 for the baselines.
(b)
Figure 11: The performance of SynGen and the baselines in concept separation on prompts containing (a) repeating modifiers; and (b) repeating entities in the DVMP dataset.
Figure 12: Concept Separation as a function of number of modifiers in a prompt in the DVMP dataset, introduced in Section 3.2. Only the top-competing method (Attend-and-Excite) is plotted for readability.
D.2 Comparison to Spatial-Temporal Diffusion
As described in Section 5, concurrent to this work, [24] developed a method to optimize the latents. While they primarily attend spatial and temporal relations, they too report on improper binding, namely, attribute mismatch. Thus, we extend the tables from Section 4, to include Spatial-Temporal Diffusion, see Fig. 16, Fig. 17, Fig. 18.
Based on these 18 images, we observe that Spatial-Temporal Diffusion consistently misses at least one entity from the prompt. As an example, see Fig. 16. The images in columns 1 and 2 miss a crown (but include "wooden" objects), and columns 3 and 4 miss a lion and exhibit semantic leakage.
In other cases, we note many cases of semantic leakage in and out of the prompt. For instance, in Fig. 18, in column 2 the clock is brown and the wall is pink, and in column 3, the chair is pink.
Figure 13: The performance of SynGen and the baselines in concept separation when grouping the prompts with respect to entangled modifiers in the DVMP dataset.
Figure 14: Samples from the analyses in Appendix C. (a) a case of recurring entity (strawberry); (b) a recurring modifier (black) and entity (apple); (c) and (d) contain entangled entities (a blue bear and a purple strawberry); (e), (f), (g) are examples of prompts with more than two modifiers.
Figure 16: Extended qualitative comparison for prompts from the DMVP dataset. SynGen and Spatial-Temporal Diffusion [24].
Attend-and Structured SynGen
(ours)
2 modifiers
3 modifiers
4 modifiers
5 modifiers
6 modifiers
Figure 15: Extended qualitative comparison for prompts from the DVMP challenge set.
a pink crown and a furry elephant
a white bicycle and a spiky bow
a green bicycle and a red baby bird
a spotted gorilla and
a beige modern car
a sliced strawberry and
a wooden bench and
a baby furry rabbit
a red spiky spotted clock and
a baby horse
a spiky wooden clock and
a modern checkered camera and
a purple apple
a baby lion and
a green sliced banana and
a white skewered apple
a purple metal wooden car and
a sliced tomato and
a blue sliced apple
an orange wooden spiky camera and
a gray checkered spotted hat
D.3 Stable Diffusion and Structured Diffusion Comparison
A comparison between Stable Diffusion and Structured Diffusion is depicted in Fig. 19. The findings from the study by [11] suggest that the generated images from Structured Diffusion are often similar to those generated by Stable Diffusion, with limited improvements in addressing semantic flaws and enhancing image quality. This is further supported by the comparable results presented in our findings Table 1. Therefore, while we include all baselines in our evaluations, our qualitative analysis only showcases images produced by the slightly superior Structured Diffusion.
Figure 17: Extended qualitative comparison for prompts from the ABC6K dataset. SynGen and Spatial-Temporal Diffusion [24].
Figure 18: Extended qualitative comparison for prompts from the Attend-and-Excite dataset. SynGen and Spatial-Temporal Diffusion [24].
Figure 19: Side-by-side generations of StableDiffusion and StructureDiffusion.
E SynGen Failures
We observe three recurring types of failure SynGen displays Fig. 20. First, when there are many modifiers and entities in the prompt, despite the results in Fig. 12, we note that sometimes the negative loss component becomes exceedingly large, and thus, pushes the latent out of the distribution the decoder was trained on. Consequently, images become blurred, or contain concepts which are successfully separated, but are incoherent. This is likely because our method over-fixates on incorporating all elements described in the prompt.
Second, while SynGen typically successfully addresses the possible error cases described in Section 4.2, at times it can neglect generating all objects, unify separate entities, or neglect generating attributes. We conjecture that it is because the cross-attention maps of the modifier and its corresponding entity do not overlap enough. We note it usually occurs when there are many modifiers that refer to the same entity.
Finally, as common with many diffusion models, we report a recurring issue with faithfulness to the number of units specified in the prompt, for a certain entity. For instance, upon receiving prompts containing "a strawberry", SynGen generates images with multiple strawberries, instead of just one. One explanation to this problem is that the representation of a certain entity begins "scattered", and is never quite formed into a single cluster. Interestingly, the opposite problem, where multiple units are "merged" into one, occurs far less in the generations of SynGen. Possibly, because of the inherent objective function of our loss, which "pushes away" foreign concepts from one another.
a red spotted dog and a blue striped cat are playing with a yellow spiky ball
(a)
Figure 20: Frequent failure modes in SynGen. (a) depicts a case of blurred image, (b) incoherent image which maintains concept separation. Both are a result of excessive updates to the latent, resulting from a large negative loss term. In example (c), the zebra and lion are merged into a single entity and (d) omits the sleepy lion. We conjecture (c) and (d) are a result of too little updates. (e) and (f) exhibit the well-known issue of flawed mapping between the number of units an entity is mentioned in the prompt to the generated image.
"abrown skewered tomato and a baby furry zebra and a yellow modern skateboard
(b)
"apink baby zebra and a brown lion and a checkered wooden camera
(c)
"a baby sleepy penguin and a furry gorilla and a sleepy lion"
(d)
"a sliced strawberry and a yellow furry zebra"
(e)
"acurved modern car and an orange baby sleepy elephant and a brown bench
(f)
F The Diverse Multiple Modifiers Prompts (DVMP) dataset
In Section 3.2 we describe DVMP, a new dataset containing rich and challenging combinations, for the purpose of evaluating improper binding.
In total, DVMP has 18 types of objects, 16 types of animals, and 4 types of fruit. There are four animal modifiers, 7 object modifiers, two fruit modifiers, and 13 colors. A comprehensive account of the entities and their possible modifiers is shown in Table 4.
G Extended Evaluation
G.1 Additional Details on Human Evaluation Experiments
In the manual evaluation procedure detailed in Section 3.3 the evaluator is tasked with comparing various image generations and selecting the optimal image based on multiple criteria. The guidelines and examples given to the evaluators are presented in Fig. 21 and Fig. 22. Fig. 23 provides a screenshot of the rating interface. The full results of the human evaluation are given in Table 5