Linguistic Binding in Diffusion Models: Enhancing Attribute Correspondence through Attention Map Alignment
Authors: Royi Rassin
Eran Hirsch
Daniel Glickman
Shauli Ravfogel
Yoav Goldberg
Gal Chechik
Venue: NeurIPS 2023 oral (2023)
Keywords: syntax
diffusion
stable diffusion
attribute
attention

Abstract: Text-conditioned image generation models often generate incorrect associations between entities and their visual attributes. This reflects an impaired mapping between linguistic binding of entities and modifiers in the prompt and visual binding of the corresponding elements in the generated image. As one example, a query like ``a pink sunflower and a yellow flamingo'' may incorrectly produce an image of a yellow sunflower and a pink flamingo. To remedy this issue, we propose SynGen, an approach which first syntactically analyses the prompt to identify entities and their modifiers, and then uses a novel loss function that encourages the cross-attention maps to agree with the linguistic binding reflected by the syntax. Specifically, we encourage large overlap between attention maps of entities and their modifiers, and small overlap with other entities and modifier words. The loss is optimized during inference, without retraining or fine-tuning the model. Human evaluation on three datasets, including one new and challenging set, demonstrate significant improvements of SynGen compared with current state of the art methods. This work highlights how making use of sentence structure during inference can efficiently and substantially improve the faithfulness of text-to-image generation.

Linguistic Binding in Diffusion Models: Enhancing Attribute Correspondence through Attention Map Alignment
Royi Rassin
Eran Hirsch
Daniel Glickman
Bar-Ilan University, Israel rassinroyl wwwinnooo@@gamal...m..mm @gmail. com
Bar-Ilan University, Israel eran. hww.chebiu..c.. ac. i1
Bar-Ilan University, Israel daniel elglilgllioo clmmannmnn @gmail com
Shauli Ravfogel
Yoav Goldberg
Bar-Ilan University, Israel
Bar-Ilan University, Israel
Allen Institute for AI, Israel shauli.. rwv.oaalle@geaeeee @gmail. com
Allen Institute for AI, Isracl yoav. goldbergggaiil.......m. com
Gal Chechik
Bar-Ilan University, Israel
NVIDIA, Israel
gal. www.hhhhhhhibiuuu ac.
Abstract
Text-conditioned image generation models often generate incorrect associations between entities and their visual attributes. This reflects an impaired mapping between linguistic binding of entities and modifiers the prompt and visual binding of the corresponding elements in the generated image. As one example, a query like pink sunflower and a yellow flamingo" may incorrectly produce an image of yellow sunflower and pink flamingo. To remedy this issue, propose SynGen, an approach which first syntactically analyses the prompt to identify entities and their modifiers, and then uses novel loss function that encourages the cross- aotss-attention maps to agree with the linguistic binding reflected by the syntax. Specifically, we encourage large overlap between attention maps of entities and their modifiers, and small overlap with other entities and modifier words. The loss is optimized during inference, without retraining fine-tuning the model. Human evaluation on three datasets, including one new and challenging set, demonstrate significant improvements of SynGen compared with current state of the methods. This work highlights how making use sentence structure during inference can efficiently and substantially improve the faithfulness of text-to-image generation
1 Introduction
Diffusion models for text- cext-conditioned image generation produce impressive realistic images 2
3
4. Users control the generated content through latural-language text prompts that can rich and complex. Unfortunately, in many cases the generated images are not faithful to the text prompt [5] b] Specifically, one very common failure mode results from improper binding, where modifier words fail to influence the visual attributes of the entity-nouns to which they are grammatically related.
We make our code publicly available http:://ggsattt.oooooooaalasasss ieeitaa---uiidddd-eeeeriiiii.ei
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

(a) Semantic Leak Prompt
(b) Semantic Leak out Prompt
(c) Attribute Neglect
"a pink sunflower and a yellow flamingo"
"a checkered bowl in cluttered room
horned lion and spolted monkey"
SynGen (ours)
Stable Diffusion
Figure 1: Visual bindings of objects and their attributes may fail to match the linguistic bindings between entities and their modifiers. Our approach, SynGen, corrects these errors by matching the cross-attention maps of entities and their modifiers.
As an illustration, consider the prompt "a pink sunflower and a yellow flamingo". Given this prompt, current models often confuse the modifiers of the two entity-nouns, and generate an image of yellow sunflower and a pink flamingo (Fig. | bottom left, semantic leak in prompt). In other cases, the attribute may semantically leak to areas in the image that are not even mentioned in the prompt (Fig. []] bottom center, semantic leak outside prompt) or the attribute may be completely neglected and missed from the generated image (Fig. [ bottom right, attribute neglect). Such mismatch can be addressed by providing non-textual control like visual examples 7 8] but the problem of correctly controlling generated images using text remains open.
possible reason for these failures is that diffusion models use text encoders like CLIP g. which are known to fail to encode linguistic structures [10]. This makes the diffusion process "blind" to the linguistic bindings, and as a result, generate objects that do not match their attributes. Building on this intuition, we propose to make the generation process aware of the linguistic structure of the prompt. Specifically, we suggest to intervene with the generation process by steering the cross-attention maps of the diffusion model. These cross-attention map serve as link between prompt terms and the set of image pixels that correspond these terms. Our linguistics- based approach therefore aims to generate image where the visual binding between objects and their visual attributes adheres to the syntactic binding between entity-noums and their modifiers in the prompt.
Several previous work devised solutions to improve the relations between prompt terms and visual components, with some success [T] 12 3
They did not focus on the problem of modifier-entity binding. Our approach specifically addresses this issue, by constructing a novel loss function that quantifies the distance between the attention patterns of grammatically-relkat pairs, and the distance between pairs of unrelated words in the prompt. We then optimize the latent denoised image in the direction that separates the attention map of a given modifier from unrelated tokens and bring it closer to its grammatically-rehiee code, we markedly improve the pairing between attributes and objects in the generated image while at the same time not compromising the quality of the generated image.
(modifier, entity-noun
noun. We show that by intervening in the latent
We evaluate our method on three datasets. (1) For a natural-Languue
setting, we use the natural compositional prompts in the ABC-6K benchmark [13] (2) To provide direct comparison with previous state-of of- in [II. we replicate prompts from their setting: (3) Finally, to evaluate binding in a challenging setting, we design set of prompts that includes a variety of modifiers and entity-nouns. On all datasets, we find that SynGen shows significant improvement in performance based on human evaluation, sometimes doubling the accuracy. Overall, our work highlights the effectiveness of incorporating linguistic information into text-conditioned image generation models and demonstrates promising direction for future research in this area.

---

(a)
Entity-Modifier Identification
strawberry
amod
Sannd
lmmod
red
and
golden
strawberry
red
golden
(b)
Diffusion Process
SynGen Step
Extract cross artention maps
U-Net
Language driven cross attention losses
U-Nett
Loss
Zz.
2t
20
Fed
goldon strguberry
Nagallin
Loss
Figure 2: The SynGen workflow and architecture. (a) The text prompt is analyzed to extract entity- nouns and their modifiers. (b) SynGen adds intermediates steps the diffusion denoising process. In that step, we update the latent representation to minimize loss over the cross attention maps of entity-nouns and their modifiers (Eq 3).
The main contributions this paper are as follows: (1) A novel method to enrich the diffusion process with syntactic information, using inference-time optimization with loss over cross- cooss-attnttion maps; (2) A new challenge set of prompts containing rich number and types of modifiers and entities.
Syndax-Guided Generation
Our approach, which we call SynGen, builds on two key ideas. First, is easy to analyze syntactic structure of natural language prompts to identify bindings of entity-nouns and their modifiers. Second, one can steer the generation of images to adhere to these bindings by designing an appropriate loss over the crossaatenntion maps of the diffusion model. We describe the two steps of our approach: extracting syntactic bindings and then using them to control generation.
2.1 Identifying entity-noume and their modifiers
To identify entity-nouns and their corresponding modifiers, we traverse the syntactic dependency graph, which defines the syntactic relation between words in the sentence. Concretely, we parse the prompt using spaCy's transformer-base dependency parser [14] and identify all entity-nouns (either proper-nouns or common- )ouns) that are not serving as direct modifiers of other nouns.
These are the nouns that correspond to objects in the generated image. We then recursively collect all modifiers"jof of the noun into its modifier set. The set of modifier-labels includes range of syntactic relations between nouns and their modifiers, such adjectivial modification (amod; "the regal dog"), compounds (compound; "the freasure map"), nominal modification through intervening marker, adverbial modifiers (npadvmod; "A watermelon-styledd chair"), adjectivial complement (acomp; "The apple is blue"), and coordination between modifiers (conj: "A black and white dog").
2.2 Controlling generation with language-driven cross-attention losses
Consider a pair of noun and its modifier. We expect the cross-attention map of the modifier to largely overlap with the cross-altention map of the noun, while remaining largely disjoint with the maps corresponding to other nouns and modifiers. To encourage the denoising process to obey these spatial relations between the attention maps, we design a loss that operates on all cross-attention maps. We then use this loss with a pretrained diffusion model during inference. Specifically, we
We consider modifiers from the set amod,, nmod, compound, npadvmod, acomp, conj): J. We exclude conj when determining the top-level nouns.

---

attention maps
Figure 3: Evolution of cross-attention maps and latent representation along denoising steps, for the prompt "a red crown and a golden strawberry". At first, the attention maps of all modifiers and entity-nouns are inter- twined, regardless of the expected binding, During denoising, attention maps gradually becomes separated, adhering the syntactic bindings. The vertical line indicates that after 25 steps intervention stops, but the atten- tion maps remain separated.
red
crown
goldon
staawbeny
t= T
= 0
optimize the noised latents by taking a gradient step to reduce that loss. See illustration in Fig. Fig. iillustrates the effect of the loss over the cross-ateention maps.
Loss functions: Consider a text prompt with N tokens, for which our analysis extracted & noun- modifier sets S2,, S. P(S.) represent all pairs (m, n) of tokens between noun root and its modifier descendants in the i-th set S,. For illustration, the set of "A black striped dog" contains two pairs ("black". "dog") ") and ("striped", "dog"). Next, denote by {2,, AN } the attention maps of all N tokens in the prompt, and denote by dist An) measure of distance (lack of overlap) between attention maps Am and A,,.
Our first loss aims to minimize that distance (maximize the overlap) over all pairs of modifiers and their corresponding entity-nouns (m,
Cpos (A,
Σ
dist (A An)-
1=1 (m.n)EP(S.)
We also construct loss that compares pairs of modifiers and entity-nouns with the remaining words in the prompt, which are grammatically unrelated these pairs. other words, this loss is defined between words within the (modifiers, entity- set and words outside of it. Formally, let (S,)) represent the set unmatched words obtained by excluding the words in S from the full set of words and A, is the corresponding attention map for given unrelated word u. The following loss encourages moving apart vaaammtiicall--unealleeee pairs of words:
Σ (m.n)E (EU(S.)
Σ
dist(Am Au) dist dist(Au; An))
Eueg
(2)
==1 USE))
Our final loss combines the two loss terms:
C Lpos + Lneg-
For measure of distance between attention maps
symmetric Kullback-Leible divergence dist( A;, A2) = 1 D (Aall A₂) + 1 D (A, where A,, A, are attention maps normalized to sum of and are generic indices, and DRz(A:114) = -Epizets A,
(A
Our rest-time optimization approach resembles the one of [TT], which defined loss over the cross- attention maps to update the latents at generation time. However, their loss aims to maximize the presence of the smallest attention map at a given timestep to guarantee set of selected tokens is included the generated image, and our loss depends on pairwise relations of linguistically-reltte words and aims to align the diffusion process to the iingussic-structur of the prompt.

---

2.3 The workflow
We use the loss of Eqs 1-3 to intervene in the first 25 out of 50 denoising steps. Empirically, using smaller number of steps did not correct well improper binding, and using larger number generated blurred images, as detailed Appendix B. In each of the first 25 steps, a pretrained denoiser (U-Net) was first used to denoise the latent variable ž₁. Then, we obtained the cross-attention maps in [15] Next, we used the loss to update the latent representation 2, with a gradient step z/ = 24 - a Va, C. Finally, the U-Net architecture denoises the updated latent variable of for the next timestep.
3 Experiments
3.1 Compared baseline methods
We compare SynGen with three baseline methods. (1) Stable Diffusion 1.4 (SD) M: (2) Structured Diffusson [13.. extracts noun-phrases from the prompt and embeds them separately, to improve the mapping of the semantics in the cross-attention maps; and (3) Attend-and-Exxt method that given a predetermined set of tokens, updates the latent a certain number of timesteps, to eventually incorporate these tokens in the generated image. To automate token selection in A&E, we follow the recommendation by the authors to select the nouns using a part-of-speech tagger.
(A&E) [I].
3.2 Datasets
We evaluate our approach using two existing benchmark datasets, and one new dataset that we designed to challenge methods in this area.
This benchmark consists of 3.2K natural compositional prompts from MSCOCO [16. which were manually written by humans, using natural language and contain at least two color words modifying different noun-entities. In addition, the dataset contains 3.2K counterparts, where the position of modifiers in the original prompts swapped. (e.g.. "a white bench in front of green bush" and "a green bench front of a white bush"). We randomly sample 600 prompts.
(1) ABC-6K 13).
(2) Data from -a--ud---nn--xiiit [TI. Originally introduced to evaluate the A&E method which focuses on entity-neglect, this dataset also showed that A&E improved over previous work in terms of improper binding.
Prompts in this dataset belong to three categories: (1) "a (color) (in-animate object) and (color) fin-animate object]"; (2) "a (color) (in-animate object) and an [animal]"; (3) "an (animal) and an (animal)".. Following the split in A&E, we sample 33 prompts from type (1) and 144 prompts from type (2), but exclude type (3), as it does not contain modifiers. This is very simple dataset, which we use to facilitate direct comparison with previous work.
(3) Diverse Visual Modifier Prompts (DVMP). The above two datasets are limited in terms of number and types of modifiers, and the number of entity-noums per prompt. To challenge our model, design dataset consisting of coordination sentences, in similar fashion to the dataset from A&E, but with strong emphasis on the number and types of modifiers per prompt. Specifically, we aim to compare the models with prompts that contain numerous and uncommon modifiers, creating sentences that would not usually be found in natural language or training data, such as "a pink spotted panda".. DVMP was designed with two key aspects in mind:
Expanding the set of modifiers: We have extended the number of modifiers referring to an entity- noun from one up to three. For instance, "a blue furry spotted bird". We also added types of modifiers besides colors, including material patterns ("a metal chair"), design patterns ("a checkered shoe"), and even nouns modifying other noun-entities ("a baby zebra").
Visually verifiable and semantically coherent:
The modifiers selected for DVMP are visually verifiable, with a deliberate avoidance of nuanced modifiers. For instance, "big" is a relative modifier dependent on its spatial context, and emotional states, such as the prompt "an excited dog", are largely excluded due to their subjective visual interpretation. Simultaneously, DVMP maintains semantic coherence by appropriately matching modifiers to noun-entities, thereby preventing the creation of nonsensical prompts like "a sliced bowl" or "a curved zebra".
5

---

In total, we have generated 600 prompts through random sampling, For a comprehensive description of the dataset's creation, see Appendix F
3.3 Human Evaluation
We evaluate image quality using Amazon Mechanical Turk (AMT). Raters were provided with a multiple-choice task, consisting of single text prompt and four images, each generated by the baselines and SynGen. Raters could also indicate that all images are "equally good" or "equally bad". We provided each prompt and its corresponding generations three raters, and report the majority decision. In cases where there is no majority model winner, we count it toward "no majority winner".
We evaluate generated images in two main aspects: (1) concept separation (sometimes known as editability [17]) and (2) visual appeal. Concept separation refers to the ability of the model to distinctly depict different concepts objects in the generated image. The effectiveness of concept separation is assessed by asking raters, "Which image best matches the given description?". To asses visual quality, raters were asked "Which image is more visually appealing?". To maintain fairness and reduce biases, the order of images was randomized in each task. Full rater instructions and further details are provided in Appendix G.T of the supplemental materials.
We also experimented automatic evaluation, but find its quality subpar. For standardized evaluation purposes, it is detailed in Appendix G.2
Fine-grained evaluation. In addition to multiple-choice task, we evaluate concept separation using the following key metries: (1) Proper Binding, quantifying how well the model associates attributes with their corresponding objects; (2) Improper Binding. measuring the instances where attributes are incorrectly linked to unrelated objects; and (3) Entity Neglect, capturing the frequency with which the model omits entities specified in the prompt.
To this end, we randomly select 200 prompts each from the DVMP and ABC-6K datasets, while using all 177 prompts available in the A&E dataset. Human evaluators were asked to mark if instances have correct or incorrect attribute-object mapping. Importantly, incorrect mappings are counted on a per-attribute basis multiple incorrect mappings of single attribute are considered one violation. For example, in the prompt "the white dog chased the cat up the tree", if the modifier "white" is incorrectly mapped to both "cat" and "tree", it is counted as one instance of violation. Evaluators also identify the number of entities mentioned in the prompt that are subsequently depicted in the generated image.
Based on these counts, we define the metric of Proper Binding as the ratio of correctly mapped attributes to the total number of attributes. Similarly, Improper Binding is defined as the ratio of incorrectly mapped attributes to the total number of attributes. while Entity Neglect the complement of the ratio of mentioned entities that are depicted in the generated image to the total number of entities in the prompt. Rater instructions are provided Appendix G.1
4 Results
4.1 Quantitative Results
Table provides provides results of the comparative experiment. SynGen is consistently ranked first in all three datasets, and by a large margin, sometimes double the approval rate of the second ranked method, A&E. These results are observed for concept separation, which measures directly the semantic leak, and for visual appeal.
The high number of "no winner" cases reflects the large difficulty of some of the prompts, for which no method provides good enough generated images. Population results before majority aggregation given in Appendix of the supplemental material. Comparisons with StableDiffusion are given in Fig. of the supplemental.
Table 2 provides results of the individual experiment. We find that SynGen outperforms all models by a landslide ide both proper and improper binding and on par with state-of-the-ae on entity neglect despite not directly tackling this problem.

---

Table I: Human evaluation of all meth- ods on the three datasets. The table re- ports scores for concept separation (how well the image matches the prompt) and visual appeal. Values are the fraction of
Concept Visual Separation Appeal
Dataset
Model
SynGen (ours)
38.42
37.85
A&E
18.08
18.65
majority vote of three raters, normalized
A&E
Structured Diffusion
04.52
04.52
to sum 100.
Stable Diffusion
01.69
02.26
No majority winner
37.29
36.72
SynGen (ours) A&E
24.84
16.00
13.33
12.17
DVMP
Structured Diffusion
04.33
07.83
(challenge set)
Stable Diffusion
03.83
07.17
No majority winner
53.67
56.83
SynGen (ours) A&E
28.00
18.34
11.17
10.00
ABC-6K
Structured Diffusion
05.83
06.33
Stable Diffusion
04.83
07.83
No majority winner
50.17
57.50
Table 2: Results of the fine- ane-grained concept separation experiment. Proper Binding should be maximized to 100, while Improper Binding and Entity Neglect should be minimized to 0.
Proper Binding 1 Improper Binding + Entity Neglect
Dataset
Model
SynGen (ours)
94.76
23.81
02.82
A&E
81.90
63.81
01.41
A&E
Structured Diffusion
55.71
67.62
21.13
Stable Diffusion
59.05
68.57
20.56
SynGen (ours)
74.90
19.49
16.26
DVMP
A&E
52.47
31.64
10.77
(challenge set) Structured Diffusion Stable Diffusion
48.73
30.57
28.46
47.80
30.44
26.22
SynGen (ours)
63.68
14.37
34.41
A&E Structured Diffusion
56.26
26.43
33.18
ABC-6K
51.47
29.52
34.57
Stable Diffusion
52.70
27.20
36.57
4.2
Qualitative Analysis
Figures 4 provide qualitative examples from the three datasets, comparing SynGen with the two strongest baselines.
The qualitative examples illustrate several failure modes of our baselines. First, semantic leak in prompt, occurs when modifier of an entity-noun "leaks" onto different entity-noun in the prompt, as shown in Fig. for the prompt "a pink clock and a brown chair", in columns 3 and 4. In this case, all baselines incorrectly apply pink hues to the chair, despite the prompt explicitly defining as brown. A more nuanced variant of this issue is semantic leak out of prompt, when a modifier is assigned an entity-noun that "a spiky bowl and green cat" leaks to plant, which is not in the prompt, the green coloration in the background of the images generated by the baselines, as seen in columns and in Fig.
mentioned in the prompt. For instance, the "spiky" attribute in
Attribute neglect occurs when modifier from the prompt is absent from the generated image. As exhibited in Fig. 4 for "a frog and brown apple". both baselines do not include brown color at all.
Entity easting is another failure type where modifier treated as standalone entity, phenomenon commonly observed with noun modifiers. For example, the prompt "a wooden crown and furry baby

---

monkey and black bow"
"a pink clock and brown chair"
"a frog and brown apple"
SynGen (ours)
Structured Diffusion
Attend -nte Excite
Figure 4: Qualitative comparison for prompts from the Attend-and-Excit dataset. For every prompt, the same three seeds are used for all methods.
wooden crown and furry baby rabbit"
"a chair purple camera and a baby lion"
spiky bowl and green car"
SynGen (ours)
Structured Diffusion
Attend-and
Excite
Figure 5: Qualitative comparison for prompts from the DVMP dataset. For every prompt, the same three seeds are used for all methods.
rabbit" (column in Fig. 5) has all methods, apart from ours, generate human infants. Presumably, noun rather than modifier, leading other methods
this occurs because "baby" is interpreted to treat it as a separate object due to the lack of syntactic context. Conversely, SynGen correctly interprets "baby" as a modifier and accurately binds it to the rabbit. Similarly, in the prompt "a white fire hydrant sitting in field next to red building" (column 6 in Fig. (6). "fire" is wrongly interpreted as an noun, which leads the unwarranted inclusion of a fire in the scene.
All methods, barring SynGen, grapple with entity entanglement [8 objects tend to strongly associate with their most common attribute (e.g., tomatoes typically red). This is evident in columns 3 and 4 in Fig. 6 where other methods fail to visually associate the blue attribute with the dog "a blue and white dog sleeps in front of black door". Instead, they resort typical attributes of the objects, generating a black and white dog.
20 21 22, where some
Further qualitative analysis is provided in Appendix D.I

---

"a brown brush glides through beautiful blue hair"
blue and white dog sleeps front of black dour"
"a white fire bydrant sitting in
field next to red building"
SynGen (ours)
Structured Diffusion
Attend-ane andedd-and Excite
Figure 6: Qualitative examples for ABC-6K prompts. For every prompt, all methods use the same three seeds.
4.3 Ablation study
The importance of using both positive and negative lasses. We evaluated the relative importance of the two terms our loss Eq. (3). The positive term Lyoss which encourages alignment of attention map of an object and its modifiers, and the negative loss term, Leeg. which discourages alignment with other modifiers and objects. We sampled 100 prompts from the DVMP dataset and generated images with and without each of the two loss terms. See example in Fig. 7 Then, raters were asked to select the best four variants. Table 3 shows that raters preferred the variant that combined both the positive and the negative terms. More examples are given in the supplemental Appendix
Both Losses
Only Positive Loss
Only Negative Loss
No Intervention
Concept Separation
Visual Appeal
Loss
Both losses pps Lang
27
22
Positive only Eyou
0
11
Negative only fineg
35
"A bird and a white crown"
Stable Diffusion No majority winner
28
66
Figure 7: Ablation of loss components. Removing Gneg results in semantic leakage (the bird white) and entity neglect (there is no crown). Removing Cpos also leads to semantic leakage (generating bird and background with white parts), and failed attribution binding (generating crown that not white).
Table 3: Ablation of loss components. Val- ues are percent preferred by human raters.
Related Work
Semantic leakage. pointed out cases of semantic leakage in diffusion models, where properties of one entity-noun influence the depiction of another. [23] attributed this issue to a lack of understanding of syntax, specifically noting failures when processing texts requiring subtle syntactic binding comprehension. 66] identified semantic leakage issues in DALL-E, where properties of one entity- noun influence how other entity nouns are depicted. In this work, we pinpoint semantic leakage as consequence of improper mapping between syntactic and visual binding.

---

Attention-based interventions. [15] demonstrated that the cross- aoss-attentionn mechanism determines the spatial layout of entities in generated images. This result suggested that cross-attention is causally involved in the aforementioned issues. A&E [1] addresses the problem of entity omission, where certain entities mentioned in the prompt do not appear the generated image. They propose a loss function that encourages each noun token in the image to significantly attend to a corresponding image patch, thereby preventing its omission. Our approach similar to [ in that it updates the latent representation through a loss function over attention maps, during image generation.
Syntax-baue based generation was also explored in 13 proposing the Structured Diffusion method. aims to address the problem of missing entities and semantic leakage of attributes. This is achieved by parsing the prompt, extracting phrases corresponding to nouns and modifiers, and encoding them separately. They also intervene in the attention patterns, ensuring that each individual phrase influences the attention patterns, Our experiments show that is better to implicitly influence the attention patterns through our loss which we dynamically optimize. In contrast, their intervention remains fixed.
Concurrent to this work, [24] proposed an alternative approach to combine syntactic control and aitention-based optimization. They extract nouns from prompts and train layout predictor identify the corresponding pixels for each noun. Then, they optimize the latents by encouraging the pixels corresponding to the objects to attend to CLIP representations of phrases containing those objects. While similar in spirit, the current paper demonstrates intervention in the generation process solely based on syntax, without explicitly learning the correspondence between image entities and tokens.
Limitations
Like previous methods, the performance of SynGen degrades with the number of attributes to be depicted (see supplemental Fig. 12). However, its decline is remarkably less pronounced compared to other methods. This decay in performance can be attributed to two primary factors: (1) an image begins to lose its visual appeal when the negative loss term becomes excessively large; (2) an overly
cluttered image poses challenges in crafting cohesive "narrative" for all the concepts. We expect that some of these issues can be addressed with more syper-paramedt tuning.
Naturally, the effectiveness of our method is intrinsically tied to the quality of the parser. When the parser fails to extract the stipulated syntactic relations, our method essentially operates akin to SD.
Finally, SynGen takes longer to generate images with modifiers in the prompt than SD and slightly slower than than A&E (see Appendix A).
7 Conclusions
In this work, we target the improper binding problem, common failure mode of text-conditioned diffusion models, where objects and their attributes incorrectly correspond to the entity-nouns and their modifiers in the prompt. To address it, we propose SynGen, an inference-time intervention method, with a loss function that encourages syntax-related modifiers and entity-nouns to have overlapping crossaatentiion maps, and discourages words in the prompt. We challenge our method with three datasets, including DVMP a new dataset that is -pesially-designed to draw out hard cases of paproper-bindingg problem. Our method demonstrates improvement of over 100% across all three datasets over the previous state-of -otite-ff-hh-aar the-art. Finally, our work highlights the importance of linguistic structure during denoising for attaining faithful text-to-image generation, suggesting promising avenues for future research.
overlap from cross aoss-attention maps of other
Acknowledgements
This study was funded by a grant GC from the Israel Science Foundation 737/201) and an equipment grant to GC and Bar-Ilan University from the Israel Science Foundation (ISF 2332/18). This project has also received funding from the European Research Council (ERC) under the European Union's 's Horizon 2020 research and innovation programme, grant agreement No. 802774 (iEXTRACT). Shauli Ravfogel is grateful be supported by the Bloomberg Data Science PhD Fellowship.
10

---

References
[1] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10684-10695, 2022.
[2] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06122. 2022.
[3] Chitwan Saharia, William Chan, Saurabh Saxena, Lala I.i, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S. Sara Mahdavi, Rapha Gontijo Lopes. Tim Salimans, Jonathan Ho, David J Fleet. and Mohammad Norouzi. Photorealistic text- to-inage diffusion models with deep language understanding. 2022.
[4] Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Karsten Kreis, Miika Aittala, Timo Aila, Samuli Laine, Bryan Catanzaro, ediffi: Text-to-image diffusion models with an ensemble of expert denoisers. arXiv preprint arXiv:2211 1..0344 2022.
[5] Colin Conwell and Tomer Ullman. Testing relational understanding in text-guided image generation. arXiv preprint arXiv:2208. 00005, 2022.
[6] Royi Rassin, Shauli Ravfogel, and Yoav Goldberg. DALLE-2 is seeing double: Flaws in word-to-concept mapping in Text2Image models. In Proceedings of the Fifth BlackboxNLP Workshop on Analyzing and Interpreting Nearal Networks for NLP, pages 335-345, Abu Dhabi, United Arab Emirates (Hybrid), December 2022. Association for Computational Linguistics.
[7] Lvmin Zhang and Maneesh Agrawala, Adding conditional control to text-to-image diffusion models, 2023.
[8] Yoad Tewel, Rinon Gal, Gal Chechik, and Yuval Atzmon. Key-locked rank one editing for text-to-image personalization. In ACM SIGGRAPH 2023 Conference Proceedings, SIGGRAPH '23, 2023.
[9] Alee Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning transferable visual models from natural language supervision, 2021.
[10] Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou. When and why vision-language models behave like bags-of-wordd words, and what to do about it? In The Eleventh International Conference on Learning Representations, 2023.
[11] Hila Chefer. Yuval Alaluf, Yael Vinker, Lior Wolf, and Daniel Cohen-Or. Attend-and-excite. Attention- based semantic guidance for text-to-image diffusion models. arXiv preprint arXiv:2301. 13826, 2023.
[12] Nan Liu, Shuang Li, Yilun Du, Antonio Torralba, and Joshua B Tenenbaum. Compositional visual generation with composable diffusion models. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23 27, 2022, Proceedings, Part XVII, pages 423-439. Springer, 2022.
[13] Weixi Feng. Xuehai He, Tuu-Jui Fu. Varun Jampani, Arjun Akula, Pradyumna Narayana, Sugato Basu, Xin Eric Wang. and William Yang Wang. Training- free structured diffusion guidance for compositional text-to-iman -mage synthesis. arXiv preprint (tti::2222.000333 2022.
[14] Matthew Honnibal and Ines Montani. spaCy 2: Natural language understanding with Bloom embeddings. convolutional neural networks and incremental parsing. To appear. 2017.
[15] Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Prompt-to- prompt image editing with cross attention control. arXiv preprint arXiv:2208.016: 2022.
[16] Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays, Pietro Perona, Deva Ramanan, C. Lawrence Zitnick, and Piotr Dollár. Microsoft coco: Common objects in context, 2015.
[17] Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H. Bermano, Gal Chechik, and Daniel Cohen-Or. An image worth one word: Personalizing text-to-image generation using textual inversion, 2022.
[18] Yuval Atzmon, Felix Kreuk, Uri Shalit, and Gal Chechik. A causal view of compositional zero-shot recognition. Advances in Neural Information Processing Systems, 33:1462-1443, 2020.
[19] Massimiliano Mancini, Muhammad Ferjad Nacem, Yongqin Xian, and Zeynep Akata. Open world compositional zero-shot learning, In Proceedings of the IEEE//VF conference on computer vision and pattern recognition, pages 5222-5230, 2021.
[20] Yuval Atzmon, Jonathan Berant, Vahid Kezami, Amir Globerson, and Gal Chechik. Learning to generalize to new compositions in image understanding, arXiv preprint arXiv: 1et..:008..00 07639, 2016.
[21] Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C. Lawrence Zitnick, and Ross B. Girshick. CLEVR: A diagnostic dataset for compositional language and elementary visual reasoning. CoRR, abs/1612. 2016.
11

---

[22] Ishan Misra, Abhinav Gupta, and Martial Hebert. From red wine to red tomato: Composition with context.
In 2017 IEEE Conference Computer Vision and Pattern Recognition (CVPR), pages 1160-1169, 2017.
[23] Evelina Leivada, Elliot Murphy, and Gary Marcus. Dall-e 2 fails to reliably capture common syntactic processes. arXiv preprint arXiv:22 12889, 2022.
[24] Qiucheng Wu, Yujian Liu, Handong Zhao, Trung Bui, Zhe Lin, Yang Zhang, Shiyu Chang. Harnessing the spatial-temporal attention of diffusion models for high-fidelity text-to-imag synthesis. arXiv preprint arXiv:2.304003866 2023.
12

---

Supplementary Material
A Implementation Details
Computing resources. Experiments were run on an NVIDIA DGX Station with four v100-SXM2- 32GB GPUs. The overall duration of all experiments in the paper was about two weeks.
Efficiency. SynGen takes -144% longer than SD and 10.9% longer than A&E to generate images with modifiers in the prompt. To arrive to these numbers, we randomly sampled a set of 50 images from the A&E dataset, the DVMP set, and ABC-6K and timed the generations for each method. On average, SD needs 4 seconds generate an image, A&E 8.8 seconds, and SynGen 9.76 seconds.
Hyperparameters. The hyperparameters we used consist of 50 diffusion steps, a guidance scale of 7.5, a scale-factor of 20, and 25 latent update steps. The choices of scale factor and latent update steps are described in Appendix [.22and and Appendix B.3 respectively.
Parser. Throughout this project, ceccoreeee_bat trf model.
we
use the spacy parser with the out-of-the-box
Attending word pieces. When a relatively uncommon word is encountered by the tokenizer of the text encoder, split to sub-words (i.e., word pieces). the context of our loss function, when an entity (or modifier) is split into word pieces, we compute distance function (the Symmetric-KI for each word piece. Then, only the word piece that maximizes the distance is added to the loss.
Cross-attention maps. We describe more formally the cross-attention maps on which we intervene. Let N be the number of tokens in the prompt, and let D be the dimensionality of the latent feature map an intermediate denoising step. The denoising network defines a cross-attention map Hooeenss E T D x between each of D2 patches in the latent feature map and each token. Intuitively, the attention maps designates which tokens relevant for generating each patch.
Our goal it derive an attention distribution Alokens-optthhn A pooeens-pputcee contains the attention distribution of token over patches. For this goal, we de- fine a score matrix S to be the transpose of Apatcher-ttolen: i.e.. a matrix whose 7th row contains the attention scores from each patch to token i. Since S is not normalized, we divide each row by its sum get distribution over patches. Unless stated otherwise, across the paper, we refer to Annkens-sputth E N when mentioning the "cross-attention maps" A and its ith row A, corresponding to the attention map from the 7t token.
N such that its i-th row
B
Additional Ablation Experiments
B.1
Further Investigation of the Positive and Negative Loss Terms
In Section 4.3 we investigate the importance of the positive and negative loss function terms using a human rater. Here, we accompany the rating with a qualitative analysis, to examine the effect of each term. To this end, we generate images for 15 randomly selected prompts, five from each dataset. Fig. & depicts sample of the generated prompts.
We find that proper binding necessitates both the positive and negative terms: excluding the negative term from the loss function results in two noteworthy observations. First, the number of missing objects increase, evident by the missing crown, cat, metal chair, and tomato, in columns 1, 2, 4, and in Fig. & One consequence missing objects is the apparent improper binding, indicated by the red backpack and black shirt in columns 1 and 3.
the other hand, excluding the positive term results in fuzzier separation between objects. For instance, the cat is not completely formed, and is "merged" with the pillow; and while it appears that there is some green residue on the dog, it not colored green. Moreover, the grass is green, which indicates semantic leakage.
13

---

"an prance crown and red backpack
black white iny on preen pillow top of desk
reaches down black and green dog as they stand on white grass"
lecie spotied a metal
sliced tomato und smotied clophant"
Both Losses
Only Positive Loss
Only Negative Loss
No Intervention
Figure 8: We examine the effect of employing only one of the two losses instead of both. All images generated using the same random seed.
Putting these insights together, we observe that to some extent, the effect the loss terms is comple- mentary. In addition to the increase of objects and proper binding, the images more coherent (less cases of objects mixed into each other, such as the cat in the only-negative loss or the elephant in the only-positive loss).
B.2 Number of Timesteps for Intervention
Recall that our method intervenes in latent denoising generation. In this appendix, study the effect of the hyperparameters determining the number of steps in which we intervene.
To identify an ideal number of timesteps to intervene, we experiment with 100 randomly selected prompts from the DVMP dataset, a fixed random seed, and number of update steps from to 50, in increments of 5. Examples of this experiment are shown in Fig. 9
We observe that when intervening in small number of timesteps, our method failed adequately mitigate semantic leakage or that images are not completely formed. For instance, the apple in column 1 in the 15-steps example cartoon-ish, while the dog is not. Conversely, intervening for the full 50 timesteps resulted in an increase rate of blurred images (potentially due to the significant modification of the latent, which shifts it away from the learned distribution). We conclude that the optimal number of timesteps for intervention is 25, as this allows for effective mitigation of improper binding, while still generating visually appealing images.
B.3 Setting the Scale Factor
The scale factor affects the update step size. Recall the update step stated in Section z't a L2 Here, a is the scale-factor.
To determine good selection for the scale-factor, generate 100 randomly sampled prompts from the DVMP dataset, with a scale-factor value from to 40, in increments of 10.
we observe that merely updating the latent using scale-facti of 1 yields relatively good results in terms of improper binding, which confirms the utility of our loss function. However, such a low scale-factor also consistently leads to missing objects.
As can be seen in Fig. 10
14

---

"a furry spotted dog and a black apple
"a sliced banana and a Leal spotted skateboard and an orange elephant"
"a spotted baby rabbit and hahy panda"
"a spotted sleeny horse and a sliced apple"
"a red turile and a pink apple"
No intervention
15 steps
25 steps (SynGen)
Every step
Figure 9: We experiment with varying number of diffusion steps and examine the effect of changing the number of diffusion steps for which we intervene with the cross attention maps. All images were generated using the same random seed.
Interestingly, for greater scale-factor values, the generations become alike in their overall look, but are nonetheless very different. As an example, for both values, 10 and 30, the sliced banana is missing from the image in column 2, but the 30-value does result in a spotted teal skateboard. In column 3, values below 20 lead to images that contain two pandas (none of which are spotted), which indicates the proper binding process, and that the latent not updated enough. On the other hand, value greater than 20 leads to an image of a striped rabbit, instead spotted rabbit.
One interesting conclusion from this experiment is that the greater the scale-factor, the stronger the concept separation. However, this is only true to a point. For great enough value, generations become too blurred or simply lose their visual appeal.
C Additional Quantitative Analyses
To study the efficacy of SynGen relative to the baselines in improper binding setting, we analyze the results under three perspectives. (1) function repeating entities and modifiers; (2) as function of the number of modifiers; and (3) degree of entanglement. Samples of generations are shown in Fig. 14
Number of repeating modifiers and entities. In this analysis, we examine the performance of all methods for prompts containing recurring modifiers (e.g., "a sliced strawberry and a sliced tomato) or entities (e.g., "a sliced tomato and skewered tomato"), Aggregated results illustrated in Fig. 11 Our observations reveal a decrease in performance across all methods when modifiers are repeated. However, the relative success between SynGen and the baselines in performance remains the same. Moreover, there is no substantial decline in results when entities are repeated.
Number of modifiers prompt. We hypothesize that since our method is specifically designed to tackle improper binding, it handles prompts containing many modifiers with more success. This is
15

---

"a furry spotted dog and black apple
sliced banana and teal spotted skateboard orange elephant"
"a spotted baby rabbit and baby panda"
"a spotted sleepy horse and slieed apple"
"a red turtle and pink apple"
Scale Foctor=]
Foctor = 10
Scole
Scole Foctor = 20
Scole Factor 30
Scale Foctor = 40
Figure 10: Qualitative comparison between scale factor values for SynGen. For every prompt, the same seeds are applied. We anecdotally show our scale-factor value (we use the value 20) provides superior results.
confirmed in Fig. 12 which shows the gap between SynGen and the baselines widens the number of modifiers in a prompt increases.
entangled entities are strongly associated with their most common attribute. For instance, tomato is typically red, and thus, it is common for images to depict red tomatoes.
Entangled entities. As describe in Section 4.2
We categorize the prompts into three groups: (1) entangled prompts, which contain entangled entities with a modifier that overrides a common modifier (e.g., a purple strawberry); (2) common entangled prompts, which contain entangled entities with their common modifiers; and (3) neutral prompts, which do not contain entangled entities all. Performance as a function of these groups demonstrated in Fig. 13
D
Additional Qualitative Results
D.1 Qualitative analysis by number of modifiers
In Fig. examples from the DVMP challenge include 2 modifiers.
While errors all types are prevalent regardless of the number of modifiers, their frequency tends to rise more modifiers are added.
As for SynGen, although does not display semantic leakages at an increased rate compared baselines (as quantitatively demonstrated in Fig. 12). it does show tendency to generate more than the specified number of entities as the modifier count increases. This behavior observable in rows and 10 for SynGen, and in rows 7 through 10 for the baselines.
16

---

Distribution of Mothod by Max Repeating Modifie's in Prompt
Distribtiion Mothod by Max Repeating Entlies in Prompt
Number of Timace Modea Salected as Baxt
SynCen (ours)
Numbar of Timas Model Selected as Bazz
25%
SynCen (ours)
Abwnd-ard and--a
Attnn--and and-Exiilee
25%
Strustured
Strusture
Ver illl-lS
Verilla-SD
38%
40%
15%
47%
10%
-625
.81%
04%
81.3
07%
078
Reppeling Prompt
Repeating
(a)
(b)
Figure The performance of SynGen and the baselines in concept separation on prompts containing (a) repeating modifiers; and (b) repeating entities in the DVMP dataset.
Distribution of Method by Modifiers
Percent of Times Model Salected as Best
35%
SynGen (ours)
Attend-and-Exed
30%
25%
20%
46%%
42%%
-38%
15%
.9%
10%
68%
5%
0%
2
Number Modifiers
Figure 12: Concept Separation as function of number of modifiers in a prompt the DVMP dataset, introduced in Section 3.2 Only the top-competing method (Attend-and and-anccte)) is plotted for readability.
D.2 Comparison to TentiialTemmorall Diffusion
As described in Section concurrent this work. 244 developed method to optimize the latents. While they primarily attend spatial and temporal relations, they too report on improper binding, namely, attribute mismatch. Thus, we extend the tables from Section include Opatial-Temporal
Diffusion, see Fig. 16 Fig. 17
Fig. 18
Based on these 18 images, observe that Spatial-Temppral Diffusion consistently misses least one entity from the prompt. As an example, see Fig. [6 The images in columns and miss a crown (but include "wooden" objects), and columns 3 and 4 miss lion and exhibit semantic leakage.
In other cases, we note many cases of semantic leakage and out of the prompt. For instance, in Fig. 18 in column 2 the clock is brown and the wall is pink, and in column 3, the chair is pink.
17

---

Distribution of Method by Entangled Group
Number of Times Model Selected as Best
SynGen (ours)
Attend-and-Bn Eand-Exiite
50%
Structured
Vanilla SD
40%
-60%
45%
80%
-84%-64%
-0%%-80%
10%
-91%
0%
entangled
entangled with
neutral
without common modifiers
common modifiers Entangled Group
Figure 13: The performance of SynGen and the baselines in concept separation when grouping the prompts with respect entangled modifiers in the DVMP dataset.
"a skewered strawberry and yellow strawberry"
"a black apple and black shteed apple
"a blue bear and beige
"a heige modern balloon and purple strawberry"
"a furry zebra and a spotted metal bicycle"
"a skewered apple and a spiky wooden chait"
"a pink cat and gray wooden camera"
SynGen (ours)
Structure Diffusion
Attend and Excite
(a)
(b)
(c)
(d)
(c)
(f)
Figure 14: Samples from the analyses in Appendix (a) a case of recurring entity (strawberry): (b) recurring modifier (black) and entity (apple): (c) and (d) contain entangled entities (a blue bear and a purple strawberry); (e), (f), (g) are examples of prompts with more than two modifiers.
wooden crown and furry baby rabbit"
"a red chair purple camera and baby lion
spiky bowl and green cat"
SynGen (ours)
Spatial- Temporal Diffusion
Figure 16: Extended qualitative comparison for prompts from the DMVP dataset. SynGen and Spatial-Temporal Diffusion 24
18

---

Attend-and Structured SynGen
Excite
Diffusion
(ours)
a pink crown and furry elephant
2 modifiers
white bicycle and spiky bow
a green bicycle and a red baby bird
3 modifiers
spotted gorilla and
a beige modern car
a sliced strawberry and a wooden bench and a baby furry rabbit
4 modifiers
a red spiky spotted clock and a baby horse
a spiky wooden clock and modern checkered camera and a purple apple
5 modifiers
baby lion and a green sliced banana and white skewered apple
a purple metal wooden car and a sliced tomato and a blue sliced apple
6 modifiers
an orange wooden spiky camera and a gray checkered spotted hat
Figure 15: Extended qualitative comparison for prompts from the DVMP challenge set.
D.3 Stable Diffusion and Structured Diffusion Comparison
comparison between Stable Diffusion and Structured Diffusion is depicted in Fig. 19 from the study by [II] suggest that the generated images from Structured Diffusion are often similar to those generated by Stable Diffusion, with limited improvements addressing semantic flaws and enhancing image quality. This is further supported by the comparable results presented in our findings T Therefore, while we include baselines our evaluations, our qualitative analysis only showcases images produced by the slightly superior Structured Diffusion.
The findings
19

---

"a brewn brush plides through beautiful blue hair"
"a blus and ychite dog sleeps in front of black door
whit fire hydrant sitting in field next red building"
SynGen (ours)
Temporal
Spatial- Opatial-Tempont Diffusion
Figure 17: Extended qualitative comparison for prompts from the ABC6K dataset. Opatial-Temporae Diffusion (24)
SynGen and
"a monkey and a black blow
"a pink clock and brown chair"
"a frog and a brown apple"
Syn (ours)
Spatial- Temporal Diffusion
RASH
Figure 18: Extended qualitative comparison for prompts from the Attend-and and--anddttt dataset. SynGen and Spatial-Tempont Diffusion 24
"a wooden crown and furry bahy rabbit"
"a red chair and pumpl camera a bahy lion"
"a spiky a green cal"
"a brown brush glides through beantiful blue hair"
"a blue and while dog sleeps in front of black door"
"a white firs hydrani sitting in a field next a red building"
monkey and black bow"
Stable Diffusion
Structure Diffusion
Figure 19: Side-b generations of
StableDiffusion and StructureDiffusion.
E SynGen Failures
We observe three recurring types of failure SynGen displays Fig. 20 First, when there are many modifiers and entities in the prompt, despite the results in Fig. 12 we note that sometimes the negative loss component becomes exceedingly large, and thus, pushes the latent out of the distribution the decoder was trained on. Consequently, images become blurred, contain concepts which are successfully separated, but are incoherent. This is likely because our method over-fixate on incorporating all elements described in the prompt.
20

---

Second, while SynGen typically successfully addresses the possible enror cases described in Sec- tion 4.2 at times it can neglect generating all objects, unify separate entities, or neglect generating attriiutes. We conjecture that it is because the cross-attention maps of the modifier and its corre- sponding entity do not overlap enough. We note it usually occurs when there are many modifiers that refer to the same entity.
Finally, as common with many diffusion models, we report recurring issue with laithfulness the number of units specified in the prompt, for a certain entity. For instance, upon receiving prompts containing "a strawberry", SynGen generates images with multiple strawberries, instead of just one. One explanation to this problem is that the representation of certain entity begins "scattered", and is never quite formed into a single cluster. Interestingly, the opposite problem, where multiple units are "merged" into one, occurs far less in the generations of SynGen. Possibly, because of the inherent objective function of our loss, which "pushes away" foreign concepts from one another.
red spotted dog and a blug striged cat are playing with
brown skewered tomato and baby furry zebra and a vellow modem skateboard"
"a pink baby zebra and a brown lion and a checkered wooleg camera
"a baby sleepy penguin fuamy porilla and a alyay lion
curved modenn and an orange baby sleemy elephant and boown bench
"a aliced crawberry yellow furr zebra
yollow spilky ball
(8)
(b)
(c)
Figure 20: Frequent failure modes in SynGen. (a) depicts a case of blurred image, (b) incoherent image which maintains concept separation. Both are result of excessive updates to the latent, resulting from a large negative loss term. In example (c), the zebra and lion are merged into single entity and (d) omits the sleepy lion. We conjecture (c) and (d) are a result of too little updates. (e) and (f) exhibit the well-known issue of flawed mapping between the number of units an entity is mentioned in the prompt to the generated image.
F The Diverse Multiple Modifiers Prompts (DVMP) dataset
In Section 3.2 we describe DVMP, new dataset containing rich and challenging combinations, for the purpose evaluating improper binding.
In total, DVMP has 18 types of objects, 16 types of animals, and 4 types of fruit. There are four animal modifiers, 7 object modifiers, two fruit modifiers, and 13 colors. A comprehensive account of the entities and their possible modifiers is shown in Table
G Extended Evaluation
G.1 Additional Details on Human Evaluation Experiments
In the manual evaluation procedure detailed in Section 3.3 the evaluator is tasked with comparing
various image generations and selecting the optimal image based on multiple criteria. The guidelines and examples given to the evaluators are presented in Fig. 21 and Fig. 22 Fig. 23 provides a
screenshot of the rating interface. The full results of the human evaluation are given in Table
Rater Compensation. Raters were selected based on their performance history, requiring mini- mum of 5,000 approved HITs with an approval rate exceeding 98%. They were required to pass qualification exam with a perfect score before given access to the task. The hourly compensation was $10, ensuring fair renumeration for their contributions.
21

---

Table 4: List of entities and their modifiers in the DVMP dataset. Colors are not restricted to categories.
Category
Entities
Modifiers
General
backpack, crown, suitcase, chair, balloon, bow, car, bowl, bench, clock, camera, umbrella, guitar, shoe, hat, surfboard, skateboard, bicycle
modern, spotted, wooden, metal, curved, spiky, checkered
Fruit
apple, tomato, banana, strawberry
sliced, skewered
Animals
cat, dog, bird, bear, lion, horse, elephant, monkey, frog, turtle, rabbit, mouse, panda, zebra, gorilla, penguin
furry, baby, spotted, sleepy
Color Modifiers
red, orange, yellow, green, blue, purple, pink, brown, gray, black, white, beige, teal
Instructions
follcadoa instuctions
In Ih's task, will given ceseritiin and two images, Your is evaluste the images besed two oriteraa
1. Concept Separation: How well does the image match the given description
2 Visual Appeal: Which image looks ovenal beterr or more natural:
Concept Saparation: wach irage, aok yoursal
you
from the descritionn
Choose the
matichee desorition
choose
bad
lyguilly bac?.
Vauel Appeat: After evakatinng Conoept Sippratton, decade which imaga tooka setter ratural you Plaas choose the images based on Concept Separation firs, and then consider their Mouul Appeol.
Figure 21: The instructions that were given to the raters.
Table 5: The population vote of three raters was normalized to sum to 100 and the standard error mean was added. The table reports the scores for concept separation (how well the image matches the prompt) and visual appeal for different models on each dataset.
Concept Separation
Visual Appeal
Dataset
Model
SynGen (ours) A&E
38.80 0.48
40.11 0.49
22.60 + 0.41
21.47 - 0.41
A&E
Structured Diffusion
09.98 0.29
11.87 + 0.32
Stable Diffusion
08.85 + 0.28
09.79 0.29
No majority winner
19.77 0.39
16.76 0.37
SynGen (ours)
29.22 0.45
23,55 + 0.42
A&E
19.83 0.39
19.00 0.39
DVMP
Structured Diffusion
09.00 + 0.28
3.56+0.33
(challenge set)
Stable Diffusion
09.89 0.29
15.56 0.36
No majority winner
32.06 + 0.46
26.33 0.44
SynGen (ours)
33.00 + 0.47
25.72 +5.72+0.44
A&E
17.84 0.38
17.28 + 0.37
ABC-6K
Structured Diffusion
13.44 + 0.34
14.50 0.35
Stable Diffusion
11.72 + 0.32
14.50 - 0.35
No majority winner
24.00 0.42
28.00 - 0.44
22

---

"a blue balloon and an orange bench"
"a blue suitcase and orange car'
Good match
This match clearly shows BLUE balloon OPANGE bench (even though we only see part of the benon), which matches the descr pptionioo
image
This is good match because the the cleatly shows ORAANGE image would shaw cleaser orange car than would not have an crange fruit would be better)
Bed match
is
theress
This is bad mutthh becsuse the image showa BLUE bench, which does match desteription.
orange object it does not look more like car than the first option.
Figure 22: Examples given to raters in their instructions. Each example consists of a prompt and two images: A good match (top) and a bad match (bottom) for the concept separation criterion. These examples were accompanied by text explaining why the images are considered a good (or bad) match to the prompt.
Description:
mage
Map
Roady
Raydly
cectppaant
Figure 23: A screenshot of the AMT task. The order of images randomized per HIT. "equally good" and "equally bad" were merged during post-processing into "no winner", to simplify presentation of results.
G.2 Phrases-to-Image Similarity
common approach to automatically assess text-based image generation is by computing the cosine similarity between an image and prompt, using a vision-language model like CLIP [9]. However, the very challenge we tackle here is rooted in CLIP's failure establishing correct mapping between syntactic bindings and visual bindings, functioning like a bag-of-words model [10]. As an example, suppose CLIP is prompted with "a blue room with a yellow window". If we present CLIP with an image of a yellow room with a blue window, it may yield a similar score to an image that accurately depicts a blue room with a yellow window.
In an attempt to address this flaw, we segment prompts to phrases containing entity-nouns and their corresponding modifiers (e.g.. "a blue room" and "a yellow window"), and compute the similarity between these segmented phrases and the image. We then aggregate the result to a single score by computing the mean. With this approach, we expect CLIP to properly associate the modifiers (e.g.. "blue" and "yellow") with the correct entity-noun (i.c.. "room" and "window") as there is only one entity-noun in each segment. Unfortunately, this metric achieves relatively low agreement with the majority selection of human evaluation, only 43. 43.5% 5% of the time, where 25% is random selection. Despite the low agreement, we note the overall trend of selections of this automatic metric is very similar to the human majority selection. Table o shows the results of our automatic evaluation.
23

---

Took
"a white bicycle and a spiky bow*
How many onjects are described in the text?
How many objects which
appwar imaga?
Looking only objects aspcar image. many correctly assigned thair object?
manyy mapped somethinn object they aesigned prompt?
confictent my answer
(optiona)
Figure 24: A screenshot of the fine-grained AMT task.
Table 6: Automatic evaluation of all methods on the three datasets. The table reports scores for concept separation (how well the image matches the prompt) and visual appeal. Values are the fraction of majority vote of three raters, normalized to to 100.
Method
DVMP (ours)
ABC-6K
A&E
SynGen (ours)
47.33
41.33
44.63
A&E
27.66
24.33
27.11
Structured Diffusion
12.84
17.84
11.87
Stable Diffusson
12.17
16.50
16.39
24


Reviews:
[{"review_id": "Y5zY2AGzAO", "reviewer": "Reviewer_GWNd", "summary": "This paper approaches the binding problem in text to image diffusion models that is marked by the inability of the models to appropriately identify which modifiers are attached to which nouns in the input text. This often results either in images that mix up the attributes of the different nouns that are mentioned in the text or in images that completely disregard some of the attributes or fall back to statistically likely combinations that are not mentioned in the text. The paper proposes an inference time optimization based on first identifying which modifiers are associated with which nouns, and then optimizing a loss function that explicitly enforces the cross attention matrices of corresponding (i.e. related by modifier-noun relationship) tokens in the text to be similar, while those of every other pair in the sentence to be dissimilar. This results in a diffusion model that adheres more faithfully to the input text and produces images that are deemed better more often than competing baseline approaches that attack the same problem.", "questions": "## Clarifications\n* The metric used in the human evaluation in Table 1 is not sufficiently clear. Aren't the values in each column supposed to sum to 100? \n* What is the Complex Concepts Prompts dataset mentioned in Fig 5?\n\n## Suggestions\n* It would be easier for the reader to follow if the examples in Figure 4,5,6 are reorganized such that the references to them in the text are chronological. Currently it requires the reader to keep going back and forth across the figures. ", "limitations": "Yes", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "## Originality, Quality, Significance\n* The paper provides a novel lightweight method using off the shelf syntactic parsers to enforce better binding of modifiers to nouns in text to image diffusion models. The qualitative results seem promising and the human evaluation results suggest that the proposed method outperforms previous approaches to mitigate the binding issue.  \n\n## Clarity\nThe paper is easy to follow. The qualitative examples demonstrate the different kind of issues faced by the baselines and the improvements brought by the proposed approach. ", "weaknesses": "* Some qualitative examples on the collected DVMP dataset with some categorization on failure cases based on having uncommon modifiers vs different number (performance on 1,2,3 and more modifiers) of modifiers would have been good to see. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Provide qualitative examples having uncommon modifiers vs different number:** Following this suggestion. We will show qualitative examples of SynGen and our baselines on prompts with a varying number of modifiers. Specifically, with 2, 3, 4, 5, and 6 modifiers. Note that the DVMP dataset does not contain prompts with just a single modifier. In this context, we note that figure 12 in the appendix quantitatively compares SynGen with A&E over several numbers of modifiers in the DVMP dataset. \n\n**Q1: Should values in Table 1 sum to 100%?** Yes. The current sum of ~ 99.8 was due to a rounding error. We’ll fix it in the final version. \n\n**Q2: What is the Complex Concepts Prompts dataset mentioned in Fig 5?** \nThis was an earlier name we considered for our DVMP data. Thank you for noting this mistake.\n\n**Suggestions 1: Reorganize Figures 4, 5, 6 for chronological reference in the text:**\nWe agree with your suggestion and will rearrange the figures to be congruent with the text describing them. Thank you for helping us improve the paper.  "}, {"review_id": "9htTfOk3qD", "reviewer": "Reviewer_RH2g", "summary": "The paper focuses on attribute binding/leak/neglect problems in text-to-image models. The authors propose, SynGen, a method that utilizes dependency trees combined with cross-attention map optimization to achieve better attribute binding results. Extensive experiments are conducted to show the effectiveness of SynGen compared to previous methods. ", "questions": "Apart from the question in weaknesses:\n\n1. Would SynGen still work for prompts like \"an apple that is blue\" or \"a red apple on the left and another on the right\"?\n\n2. I may have missed this part but how do you deal with the padding tokens?\n\n3. Table 2 shows that positive only and positive+negative have lower visual appeal percentages than negative only. Does that imply that the positive loss will harm the visual appealing of the images?", "limitations": "See weaknesses. ", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper addresses an important compositional problem in text-to-image generation. \n\n- The method is intuitive and effective. SynGen can obtain stronger attribute-object association using dependency trees compared to Structure Diffusion, the positive loss design is well-motivated for solving the problem, and the negative loss design enforces the cross-attention maps over A&E.  \n\n- The experiment looks comprehensive by considering all sources of datasets and including even more challenging prompts DVMP. The prompts in DVMP are more challenging and realistic than previous \"A and B\" format prompts.", "weaknesses": "- I am concerned with the efficiency of incorporating so many negative losses in the diffusion process. What's the speed of SynGen compared to the original SD?\n\n- Using tree-based methods for binding may fail for more complicated and practical prompts. In reality, SD users write much longer prompts that describe the components at different levels. However, I think the contribution of SynGen still matters a lot as the community needs time to develop from methods for short prompts to more generalized methods. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1: What is the speed SynGen compared to the original SD?** Syngen is about the same speed as A&E (~10% slower). This is about twice slower than vanilla SD. We did not invest in performance tuning, so we assume speed can be improved considerably once we do. We will add this information to the Limitations and appendix.\n\n**W2: \"Using tree-based methods for binding may fail.... However, the contribution of SynGen matters a lot as the community … develops more generalized methods.\":**  We agree. Importantly, once better binding is available, it can be easily integrated into the generation process using our SynGen approach.\n\n\n**Q1:Would SynGen work for prompts like \"an apple that is blue\" or \"a red apple on the left and another on the right\"?** (1) “An apple that is blue” was not supported in the submitted code, but we already have it working well in the most recent version. Same for “An apple that’s extremely blue” and “An apple that is red and yellow in appearance”. (2) “A red apple on the left and another on the right”: Unfortunately not. There are two issues here. First, regarding \"left of”: Spatial relations are not well handled by SD, and SynGen is not designed to fix that problem. Several recent papers proposed ways to improve spatial relations in SD and we assume that combining them with SynGen may help. Second, regarding \"another”: SynGen will attribute ‘red’ to ‘apple’, but ‘another’ is an implicit mention of a second red apple, which requires commonsense to identify, and is beyond the scope of our work. More generally, we work with a dependency graph, a syntactic scheme which captures some useful aspects of the structure of the prompt, but is limited to syntactic relations. More elaborate semantic graph schemes can be easily incorporated in our pipeline in the future.\n\n**Q2: How do you deal with the padding tokens?** We only extract the cross-attention maps of tokens that are in the prompt. That is, we ignore the start, end, and padding tokens. \n\n**Q3: Does Table 2 imply that positive loss harms visual appeal of images?** The table demonstrates that to address the binding problem we need both, the positive and the negative loss. If we were to only use the negative term, the images would have been more visually appealing, but like the table shows, it would not address improper binding."}, {"review_id": "dGU5Xoi92Z", "reviewer": "Reviewer_EHJQ", "summary": "A frequent issue in text-prompted image generation (and in many other grounded language scenarios) is that a model will treat text akin to a bag of words and ignore syntactic relationships, such as which adjective attaches to which noun -- this is referred to as a problem with lingusitic binding. The paper proposes addressing this problem in diffusion models by adding an extra step to the diffusion process that nudges the image in the process of being generated so as to respect modifier relationships extracted from a syntactic parse of the prompt. This intervention operates in terms of cross-attention maps (i.e. attention relationships established between tokens and parts of the image), with the intuition that related tokens should map to the same region of the image, and unrelated tokens to different regions.\n\nThe paper also introduces a new challenge dataset aimed at diagnosing problems with linguistic binding in image generation (the proposed method performs well on this challenge set).", "questions": "none", "limitations": "Limitations are well addressed in the paper.", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "The issue of improper linguistic binding is one that plagues not only attempts at image generation, but many other model uses of grounded language. It's an important problem that has not been adequately resolved by prior work, whereas this paper has demonstrated substantial steps to overcoming this issue without sacrificing on generated image quality. All of this speaks to the significance of this work, which has the potential to see broad and immediate application without the need to undertake costly new model training efforts.\n\nThe paper is clearly written, and makes effective use of figures to illustrate the problem and how the proposed method resolves it.\n\nThere is thorough and convincing human evaluation on both existing data, and a new challenge dataset designed specifically to reveal instances of problems with binding.", "weaknesses": "There are not many weaknesses to point to. This is not a weakness of the paper per se, but when the method my main remaining (aesthetic) dissatisfaction is the reliance on an external parser, and the possibility of cascading failure that any such pipeline-based system entails. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your supportive review of this work. The reviewer raises an important point in designing future systems using SynGen. Future work will need to take into account failures of the parser and develop ways to handle them."}, {"review_id": "lUFExUMYUV", "reviewer": "Reviewer_LWw2", "summary": "This paper presents an approach called “SynGen” to improve attribute binding between nouns and their modifiers for text-to-image latent diffusion models, specifically Stable Diffusion. They propose a two-part loss on attention maps that (1) encourages attention maps of nouns and their modifiers to be similar, and (2) encourages attention maps between the noun/modifier to be different from those of other tokens in the prompt. \n\nDatasets and baselines: They run experiments on three datasets: the attribute binding contrast set (ABC-6K) from the Structured Diffusion (baseline 1) paper, data from the Attend-and-Excite (baseline 2), and a newly proposed challenge set called Diverse Visual Modifier Prompts.\n\nEvaluation:  \n(1) Human evals for concept separation and visual appeal. They find that SynGen outperforms baselines.  \n(2) Qualitative analysis with sample prompts and generations for Syngen and the two baselines, demonstrating that SynGen overcomes failure cases of the other two models.  \n(3) Ablations on the loss components and loss weighting.", "questions": "1. Was there a reason to not include numerical modifiers in the study? \n2. Are there unexpected side-effects of such constraints on model generalization? ", "limitations": "One limitation that was perhaps overlooked is the lack of standardized evaluation. Having to rely on different crowdsourcing tasks and qualitative comparisons makes it much harder to track progress.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. This paper tackles an important and challenging drawback of text-to-image diffusion models struggling to faithfully generate objects with the right attributes. \n2. This paper proposes an interesting two-part loss that imposes localized pairwise constraints on the attention maps between image patches and token embeddings. The first constraint enforces that the same image patches attend to both the noun and its modifiers. The second constraint enforces that image patches attending to these nouns/modifiers do not attend to any other tokens in the prompt. \n3. This paper attempts to evaluate on a large set of prompts including those from prior work as well as proposing their own challenge set.\n4. This paper attempts to better understand the effects and side-effects of the proposed losses via ablations. \n", "weaknesses": "1. The proposed method appears to be strongly grounded in the framework of the Attend-and-Excite paper. The writing doesn’t highlight this connection while introducing the approach in Section 2, beyond a minor footnote. Section 2 would probably need some revision to better draw this connection.\n2. Considering that the paper is tackling a very specific problem, the general “content separation” rating collected in the human eval seems pretty weak. Why not collect finer-grained scores for: number of objects in the prompt, number of objects from prompt generated, number of objects from prompt generated with the correct attributes? Wouldn’t this give a much better sense of task success and could potentially allow computing recall?\n3. The loss function ablation mentions that sometimes objects are omitted from the generated image. Is combining the proposed losses with global constraint defined in the Attend-and-Excite paper a viable option? Since the framework is identical, wouldn’t combining the losses have mitigated this issue? The Attend-and-Excite paper talks about attribute binding as well, so this appears to be a variant to try. \n\nNits:\n1. Maybe more explicitly state that this is an inference-only guidance procedure?\n2. There’s a typo on line 112 for the equation: $\\\\nabla_{z_t}\\\\mathcal{L}$ not $\\\\nabla z_t\\\\mathcal{L}$. Also $z’_t$ not $z’t$.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1: Writing doesn’t highlight the connection to A&E. Especially section 2:** Thank you for pointing out that the connection to A&E did not come across clearly. We will revise section 2 to give just credit and clarify the relation to A&E.\n\n**W2: \"Concept Separation\" is too general. Consider collecting finer-grained scores for: number of objects in the prompt, number of objects from its corresponding generated image, number of objects with correct attribute in the corresponding generated image. This provides more sense of success and allows to compute recall.**  Thank you for this important suggestion. Following this suggestion, we collected finer-grained ratings as requested for 120 images for all four baselines in the DVMP dataset. In the table below, the top row shows the fraction of images where raters stated that an entity was missing from the image (sum of number of objects in the image divided by the actual number of objects in the text). The bottom row shows the fraction of images where raters found the attribute was missing or incorrectly bound (sum of number of attribute errors divided by the actual number of attributes in the text). In this dataset, SynGen is on par with A&E in terms of entity neglect, and better in terms of improper binding. We will provide more analysis in the final version.\n\n|                               | SynGen | A&E  | Structured | Standard SD |\n|-------------------------------|--------|------|------------|-------------|\n| Entity Neglect Rate (lower is better) | **20.45** | 22.94 | 31.67     | 33.78      |\n| Improper Binding (lower is better)    | **46.27** | 57.51 | 66.84     | 64.14      |\n\n\n\n**W3: Is combining the proposed losses with A&E’s global constraint a viable option?**\nYes, we believe that should work well. We note that A&E uses Gaussian Smoothing and ‘Iterative Latent Refinement’ that we find to hurt for our task, but both losses can be used.  We will discuss in the paper together with the response to W1.\n\n**Nits:** We will address the typo noted as well as state more explicitly that no training is needed.\n\n\n**Q1 Why didn’t you include numerical modifiers in the study:**\n\nWe agree that this is an interesting future direction for SynGen. Our preliminary work showed that controlling the number of instances of an object has some fundamental differences with the task discussed here. In terms of attention maps, a numerical modifier behaves very differently than modifiers like color, because it induces different properties of the attention maps. For instance, in an image with “two trees” you expect the attention map to usually have two non-overlapping blobs. \nDue to these differences, we decided to leave numerical modifiers for future work. \n\n**Q2: Are there unexpected side-effects of such constraints on model generalization?**\nAs stated in the Limitations section, we observe that the visual appeal of images generated degrades with the number of modifiers in the prompt. However, SynGen’s decline is remarkably less pronounced compared to existing models (see figure 12 in the appendix). \n\n**The lack of standardized evaluation makes it difficult to track progress:**\nWe agree with the reviewer that this is an important topic. \n\nIn fact, we spent significant effort to develop automated evaluation metrics and our best attempt is recorded in section G and table 4 in the appendix. However, multimodal models notoriously fail in groundedness, and human agreement is low. One such example can be seen in the evaluation of the StructureDiffusion paper. There, in table 1, the automatic measure only agrees with human evaluation ~ 47% of the time, where 33% is random. The automatic measure we devise reaches better human agreement (43.5%, where 25% is random), but it is still low. We thus opted to keep it in the appendix, to provide a way of tracking progress, while not over-emphasizing these results.  We believe that given these limitations, it is futile, at the current state of the research, to rely on automatic evaluation in this task. Rather, we opted for high quality and fine-grained human evaluation.\n\nFollowing this comment, we will discuss the need for automated metrics in the paper, and encourage the community to develop some. We will share the raw data of our experiments so future papers can compute future metrics on our data. "}]

Author rebuttal:

Meta-review: This paper proposes an approach to improve diffusion-based image generation models with respect to image–text alignment, in particular, improving the correspondence between entity-noun and its modifier (in the text prompt) and the generated object and its attributes (in the image). The reviewers find the experiments comprehensive and the paper is clear. The concern around the design of the human annotation task is discussed and addressed during the rebuttal, and the authors will provide this additional analysis to the final paper. Overall I agree with the reviewers that the paper addresses an interesting problem in a simple but effective way (which is shown through the experiments).
Decision: Accept (oral)