Training-Free Structured Diffusion Guidance for Compositional Text-to-Image Synthesis
Authors: Weixi Feng
Xuehai He
Tsu-Jui Fu
Varun Jampani
Arjun Reddy Akula
Pradyumna Narayana
Sugato Basu
Xin Eric Wang
William Yang Wang
Venue: ICLR 2023 poster (2023)
Keywords: Text-to-Image Synthesis
Diffusion Models
Compositional Generation

Abstract: Large-scale diffusion models have achieved state-of-the-art results on text-to-image synthesis (T2I) tasks. Despite their ability to generate high-quality yet creative images, we observe that attribution-binding and compositional capabilities are still considered major challenging issues, especially when involving multiple objects. Attribute-binding requires the model to associate objects with the correct attribute descriptions, and compositional skills require the model to combine and generate multiple concepts into a single image. In this work, we improve these two aspects of T2I models to achieve more accurate image compositions. To do this, we incorporate linguistic structures with the diffusion guidance process based on the controllable properties of manipulating cross-attention layers in diffusion-based T2I models. We observe that keys and values in cross-attention layers have strong semantic meanings associated with object layouts and content. Therefore, by manipulating the cross-attention representations based on linguistic insights, we can better preserve the compositional semantics in the generated image. Built upon Stable Diffusion, a SOTA T2I model, our structured cross-attention design is efficient that requires no additional training samples. We achieve better compositional skills in qualitative and quantitative results, leading to a significant 5-8\% advantage in head-to-head user comparison studies. Lastly, we conduct an in-depth analysis to reveal potential causes of incorrect image compositions and justify the properties of cross-attention layers in the generation process. 

Published as a conference paper at ICLR 2023
TRAINING-FREE STRUCTURED DIFFUSION GUIDANCE FOR COMPOSITIONAL TEXT-TO-IMAGE S SYNTHESIS
Weixi Feng',
Xuehai He?, , Pradyumna Narayana', v Sugato Basu', Xin Eric Wang", William Yang Wang' University of California, Santa Barbara, University of California, Santa Cruz, Google
Tsu-jui Fu', Varun Jampani", Arjun Akula?,
ABSTRACT
Large-scale diffusion models have achieved state-of-the-an results on text-to-image to-image synthesis (T2I) tasks. Despite their ability generate high-quality yet creative images, we observe that attribution-binding and compositional capabilities are still considered major challenging issues, especially when involving multiple objects. Attribute-inding requires the model to associate objects with the correct attribute descriptions, and compositional skills require the model to combine and generate multiple concepts into a single image. In this work, we improve these two aspects of T2I models achieve more accurate image compositions. To do this, we incorporate linguistic structures with the diffusion guidance process based on the controllable properties of manipulating cross-attention layers diffusion-basee based T2I models. We observe that keys and values in cross-attention layers have strong semantic meanings associated with object layouts and content. Therefore, by manipulating the cross-attention representations based linguistic insights, we can better preserve the compositional semantics in the generated image. Built upon Stable Diffusion, SOTA T2I model, structured cross-attentis design is efficient that requires no additional training samples. We achieve better compositional skills in qualitative and quantitative results, leading to significant 5-8% advantage in tted user comparison studies. Lastly, we conduct an in-depth analysis reveal potential causes of incorrect image compositions and justify the properties of cross-attention layers in the generation process.
INTRODUCTION
Text-to-Image Synthesis (T2I) is to generate natural and faithful images given a text prompt as input.
Recently, there has been significant advancement in the quality of generated images by extremely large-scale lision-language models, such as DALL-E 2 (Ramesh et al.) 2022), Imagen (Saharia al. (202). and Parti (Yu et al. 2022). In particular, Stable Diffusion (Rombach al.) (022) the state-of-the-art open-source implementation showing superior evaluation metric gains after training billions of text-image pairs.
In addition to generating high-fidelity images, the ability to compose multiple objects into a coherent scene also essential. Given a text prompt from the user end, T2I models need to generate an image that contains all necessary visual concepts as mentioned in the text. Achieving such ability requires the model to understand both the full prompt and individual linguistic concepts from the prompt. As result, the model should be able to combine multiple concepts and generate novel objects that have never been included in the training data. In this work, we mainly focus on improving the compositionality of the generation process, as it is essential to achieve controllable and generalized text-to-image synthesis with multiple objects a complex scene.
Antriute binding is critical compositionality challenge (Ramesh et al.) 2022 Saharia et al. 2022) to existing large-scale diffusion-based models. Despite the improvements in generating multiple objects in the same scene, existing models still fail when given prompt such "a brown bench in front of white building" (see Fig. D.. The output images contains "a white bench" and "a brown building" instead, potentially due to strong training set bias imprecise language understanding. From practical perspective, explaining and solving such a two-object binding challenge is a primary step to understanding more complex prompts with multiple objects. Therefore, how to bind the attributes

---

Published as a conference paper at ICLR 2023
Stable Diffusion
Ours
A red car and white sheep.
A brown bench sits in front an old white building Interchanged attributes
A blue backpack and brown elephant Missing objects
Attribute leakagee
Figure Three challenging phenomena in the compositional generation. Attribute leakage: The attribute of one object is (partially) observable in another object. Interchanged attributes: the attributes two or more objects are interchanged. Missing objects: one or more objects are missing. With slight abuse of attribute binding definitions, we aim to address all three problems in this work.
to the correct objects a fundamental problem for a more complicated and reliable compositional generation. While previous work has addressed compositional T2I (Park et al. (021), work tackles open-domain foreground objects with counterfactual attributes, such as color and materials.
Even though state-of-the-art (SOTA) T2I models are trained on large-scale text-image datasets, they can still suffer from inaccurate results for simple prompts similar to the example above. Hence, motivated to seek alternative, data-efficient method to improve the compositionality. We observe that the attribute-object relation pairs can be obtained as text spans for free from the parsing propose combine the structured representations of prompts,
tree of the sentence. Therefore, such as a constituency tree or a scene graph, with the diffusion guidance process. Text spans only depict limited regions of the whole image. Conventionally, coordinates (Yang et al. 2022) as input to map their semantics into corresponding images. However, coordinate inputs cannot be interpreted by T2I models. Instead, we make use of the observations that attention maps provide free token-region associations in trained T2I models (Hertz et al., (002)). By modifying key-value pairs cross-attention layers, we manage to map the encoding of each text span into attended regions in 2D image space.
need spatial information such as
In this work, we discover similar observations in Stable Diffusion (Rombach et 2022) and utilize the property build structured cross-altention guidance. Specifically, we use language parsers to obtain hierarchical structures from the prompts. We extract text spans across all levels, including visual concepts entities, and encode them separately to disentangle the attribute-object pairs from each other. Compared to using a single sequence of text embedding for guidance, we improve the compositionality by multiple sequences where each emphasizes entity union of entities from multiple hierarchies in the structured language representations. We refer to our method as Structured Diffusion Guidance (SiructureDifftssion) Our contributions be summarized three-fold:
We propose an intuitive and effective method to improve compositional poxt-to-image synthesis by utilizing structured representations of language inputs. Our method is efficient and training-free that requires no additional training samples.
Experimental results show that our method achieves more accurate attribute binding and compositionality in the generated images. We also propose benchmark named Attribute Binding Contrast set (ABC-6K) to measure the compositional skills of T2I models.
We conduct extensive experiments and analysis to identify the causes of incorrect attribute binding, which points out future directions in improving the faithfulness and compositional- ity of text-to-image synthesis.
2

---

Published as a conference paper at ICLR 2023
unflatten
token-region associations
black
sitting...
red
car
hxw
X
X
X
X
Value
Value from
Attention map M₁ L
taken
Value from token "cat" ("car") contributes most (least) spatial location corresponds "cat"
(M,
Figure 2: An illustration of cross-attention operations and the token-region associations from attention
maps. We omit some tokens for simplicity.
2 DIFFUSION MODELS & STRUCTURED GUIDANCE
In this section, we propose simple yet effective approach incorporating structured language repre- sentations into the cross-attention layers. We briefly introduce the Stable Diffusion model and its critical components in Sec. 2.1 Then, we present our method in detail in Sec. 2.2
2.1 BACKGROUND
Stable Diffusion We implement our approach and experiments on the state-of-the-art T2I model, Stable Diffusion (Rombach al., 2022). It is two-stage method that consists of an autoencoder and diffusion model. The pre-trained autoencoder encodes images as lower-resolution latent maps for diffusion training. During inference, it decodes generated outputs from the diffusion model into images. The diffusion model generates lower-resolution latent maps based on random Gaussian noise input _T Given 2f, outputs a noise estimation e at each step and subtracts it from z'. The final noise-free latent map prediction 2" is fed into the autoencoder to generate images. Stable Diffusion adopts a modified UNet (Ronneberger et al. 2015) for noise estimation and a frozen CLIP text encoder (Radford et al.) 2021) to encode text inputs as embedding sequences. The interactions between the image space and the textual embeddings are achieved through multiple cross-attention layers in both downsampling and upsampling blocks.
CLIP Text Encoder Given an input prompt the CLIP encoder encodes a sequence of embeddings W, CLIP (eXlPexa(P)) (P) where cp is the embedding dimension and I is the sequence length. Our key observation is that the contextualization CLIP embeddings is a potential cause of incorrect attribute binding. Due to the causal attention masks, tokens in the later part of a sequence are blended with the token semantics before them. For example, When the user indicates some rare color for the second object (c.g. "a yellow apple and red bananas"), Stable Diffusion tends to generate "banana" in "yellow", as the embeddings of "yellow" is attended by token "banana".
Cross Attention Layers The cross-attention layers take the embedding sequences from the CLIP text encoder and fuse them with latent feature maps achieve classificr-free guidance. Denote a 2D feature map X it is projected into queries by linear layer f-(-) and reshaped Q' E R(n.hxw.d) where n denotes the number attention heads, d is the feature dimension. Similarly W, is projected as keys and values Kp. Vp E R(n..d) by linear layers fx((), f.). The attention maps refer to the product between queries and keys, denoted as a function fM(()
Q'KT Vd
M = SM(QQ' K,) = Softmax(
), M E R(n.kxw.))
(I)
3

---

Published as a conference paper at ICLR 2023
Quay Canfarundd for darde
Prompt blue walls end a white sink.
ClII
18 Aror
Purser
ROOT
Attention maps AP"
Structured Representations
Constiturncy
NP
MB
A
0000
1/
Scone Graph
<beas tal wall caad
M.
CI. Text
14
Condition Encoding Stage
Sequence Alignment
Cross Attention Layers
Figure 3: An illustration of our cross- cusss-attention design with structured representations. We unflatten the query and attention maps and omit the feature dimension d of all query, key, and value tensors for demonstration purposes. Note that noun phrases at multiple hierarchies are extracted and encoded through the frozen CLIP text encoder and projected to value vectors.
Cross Attention Controls Hertz et al. 2022)) observes that the spatial layouts depend on the cross attention maps in Imagen Saharia et al. (2022). These maps control the layout and structure of generated images, while the values contain rich semantics mapped into attended regions. Therefore, we assume that the image layout and content can be disentangled by controlling attention maps and values separately.
2.2 STRUCTURED DIFFUSION GUIDANCE
Given the challenging prompts in Fig. the attribute-object pairs are available for free in many structured representations, such as a constituency tree or scene graph. seek an implicit way of combining language structures with the eross- cats-attention layers. As is shown in Fig. multiple noun phrases (NPs) and map their semantics into corresponding regions. Since Mj provides natural token-region associations (see Fig. 2), we can apply it multiple values from different NPs to achieve region-wise semantic guidance.
extract
Specifically, given parser E(.), first extract collection of concepts from all hierarchical levels as C {c1, C j. For constituency parsing, we extract all NPs from the tree structure (see 3 For the scene graphs, extract objects and their relations with another object as text segments. We encode each NP separately:
W Wi, W Wa], W; CLIP (c;), i I....
The embedding sequence W; is realigned with W, as shown in the middle of Fig. 3 Embeddings between (bos) and (pad) are inserted into Wp to create new sequence, denoted as W,, We use
Wp to obtain K, and M as in Eq. [ assuming that the full-prompt key able to generate layouts without missing objects. We obtain set of values from W and multiply each with M achieve conjunction of k NPs in C:
= Jv ,vV.., = V. Vi.... V&].
(3)
O = (k: 1) (M.,) p. 1,2
Compared to using fv (W₂) only. Eq. does not modify the image layout composition since A still calculated from Q K,. Empincalll, we justify the claim by a series of visualizations of M,
For free that the extra computational introduced here is trivial in the whole diffusion process.
4

---

Published as a conference paper at ICLR 2023
Algorithm 1 StructureDiffusion Guidance.
Require:
Input:
Prompt P, Parser Ç, decoder tb, trained diffusion model øb
Output: Generated image I.
Retrieve concept set
(ci...., Gq] by traversing E(P);
2: Wp f- CLIP uex (P), W: CLIP (c:);
3: for = - 1, ..... 1 do
4: for each cross attention layer in do
5:
Obtain previous layer's output X 6: Q' fo("", Kp (WP). V, Jv(w);;
i p. 1
7:
Obtain attention maps M from Q² Kp:
Eqq
8:
Obtain Of from M [V). and feed following layers;
Eq.
9:
end for
end for
11: Feed 2 to decoder vb))) to generate x.
(see Appendix C) However, Stable Diffusion tends to omit objects in generated images (Fig, D especially for concept conjunctions that connect two objects with the word "and". We devise a variant of our method that computes set of attention maps M [Mp. Mf, Mi,---} from C and multiply them to V:
K [fx(W.)) M (fM(Q', K,)): p. 1.2......
1
0
(k: + (M.V).. p. 2.
(6)
Of is the output of certain cross- cross-attention layer and the input into downstream layers to generate final image æ. Our algorithm can be summarized as which requires no training or additional data.
EXPERIMENT
3.1 EXPERIMENT SETTINGS
Datasets To address attribute binding and compositional generation, we propose new benchmark, Attribute Binding Contrast set (ABC-6K). It consists of natural prompts from MSCOCO where each contains at least two color words modifying different objects. We also switch the position of two color words to create contrast caption (Gardner et al.) 2020). We end up with 6.4K captions or 3.2K contrastive pairs. In addition to natural compositional prompts, we challenge detailed prompts that conjunct two concepts together. These prompts follow the sentence pattern of "a red apple and yellow banana" and conjunet objects with their attribute deseriptions. We refer to this set of prompts as Concept Conjunction 500 (CC-500). We also evaluate our method on 10K randomly sampled captions from MSCOCO (Lin et al. 2014). We show that our method generalizes beyond attribute binding and introduces no quality degradation for general prompts.
method with less
Evaluation Metrics We mainly rely human evaluations for compositional prompts and concept conjunction (ABC-6K & CC-500), We ask annotators to compare two generated images, from Stable Diffusion and our method respectively, and indicate which image demonstrates better image-text alignment image fidelity. For image fidelity, we ask the annotators "Regardless of the text, which image is more realistic and natural?". We also investigate an automatic evaluation metric for image compositions, i.e., using a SOTA phrase grounding model GLIP (Li et al. (022) to match phrase- object pairs. As for system-level evaluation, we follow previous work to utilize Inception Score (IS) (Salimans et al., 2016), Frechet Inception Distance (FID) (Heusel et al. (017)) and CLIP R-precision (R-prec.) (Park et al. 202I). IS and FID mainly measure the image bank's 's systematic quality and diversity, while R-prec measures image-level alignment.
3.2 COMPOSITIONAL PROMPTS
Here we show the quantitative and qualitative evaluation results on ABC-6K. We observe that our method sometimes generates very similar images to Stable Diffusion. Hence, first generate
5

---

Published as a conference paper at ICLR 2023
StructureDiffusion (ours)
Alignment
Fidelity
Benchmark
V.8.
Win (t)
Lose (4)
Tic
Win (t)
Lose (L)
Tic
ABC-6K
Stable Diffusion
42.2
35.6
22.2
48.3
39.1
12.6
Stable Diffusion
31.8
27.7
38.9
37.8
30.6
31.6
CC-500
Composable Diffusion
46.5
30.1
22.8
61.4
19.8
18.8
Table I: Percentage of generated images of StractureDiffusion that are better than (win), tied with, or worse than (lose) the compared model in terms of text-image alignment and image fidelity. We filtered out 20% most similar image pairs for comparison (See Sec. E Composable Diffusion cannot be applied to ABC-6K as those prompts may not contain explicit "and" words that separate concepts.
Stable Diffusion
Stable Diffusion
Stable Diffusion
while and brown bird with besk and
Two ripe sported bananas are sittigg insidle boof countirr
A black standing next twn white
orange legs standing ng
a green goat standing
(a) Object-lovel composttonnyy
[b) Scene-level compositionality Attribute Binding
(cl Scene-level compositionality Counting
Figure 4: Qualitative results ABC-6K. Our method improves both object-level and scene-level
compositionality.
two images per prompt for our method and Stable Diffusion, involving around 12K image pairs to compare. Then, we filter out 20% of the most similar pairs and then randomly sampled 1500 pairs for human evaluations. As shown in Table [ annotators indicate around 42% chance of our method winning the comparison, 7% higher than losing the comparison. There is still a 22% chance that images tied with images from Stable Diffusion.
We show qualitative examples characterizing three different perspectives in Fig. Our method fills in the correct color for different parts of object different objects, shown in the first two examples. The third example demonstrates that our method can mitigate the issue of "missing objects". Among the 42% winning cases, there 31 % for "fewer missing objects", 14.1% % for "better-matched colors", and 54.8% for "other attributes or details" as indicated by annotators. The results certify that the improvement goes beyond colors to component completeness and fine-grained details. More qualitative examples characterizing all three aspects can be found in Fig. [4 in the Appendix.
3.3 CONCEPT CONJUNCTION
Here we address challenging concept conjunction prompts and evaluate our method on CC-500, Apart from Stable Diffuusin, we also compare Composable Diffusion (Liu et al. 2022) implemented on top of Stable Diffusion. For Composable Diffusion, we separate the prompts into text segments by the keyword "and" and feed each span into independent diffusion process. We generate three images per prompt and use all images for human evaluation for Stable Diffusion. randomly sampled 600 images for comparison to Composable Diffusion.

---

Published as a conference paper at ICLR 2023
CC-500 (Prompt format: "a [object]] and [color8] [objectB1")
Human Annotations
GLIP
Human GLIP
Zero/One
Two obj. w/ correct colors
Zero/One obj. (J)
Methods
Two obj.
Two obj.
Consistency
obj. (t)
Stable Diffusion
65.5
34.5
19.2
69.0
31.0
46.4
Composable Diffusion
69.7
30.3
20.6
74.2
25.8
48.9
StructureDiffusion (Ours)
62.0
38.0
22.7
68.8
31.2
47.6
Table 2: Fine-grained human and automatic evaluation results on CC-500. Recall that each prompt is a conjunction of two different objects with different colors. "Zero/One obj." means that the model reffitt
fails to generate all desired objects in the image. "Human-GLit consistency" reflects the percentage of images where human annotations align with GLIP detection results.
Stable Diffusion
Composable
Ours
Stable Diffusion
Composable
Ours
Diffusion
Diffusion
red bird and green apple
green apple and blue bow!
green apple
apple green
[a] Color leakage
(b) Missing abjects
Figure 5: Qualitative results CC-500 prompts that emphasize two aspects. (a) Color leakage: method prevents the green color from invading the bird or apple. (b) Missing objects: our method completes the "blue bowl" and improves the quality of the "blue apple".
As shown in Table 1 our method outperforms Stable Diffusion by around 4. % and Composable Diffusion by 16.4% terms of image-text alignment. We also observe that our method enhances some eme-ggrined details in the generated images, leading 7.2% improvement image fidelity when compared with Stable Diffusion. We observe that images from composable diffusion can be oversaturated with unnatural visual textures and layouts, which could be the reason for StructureDif- fusion to have high win rate in image fidelity. As shown in Fig. 5 and Fig. [3] Our approach prevents color bleeding (left), missing objects (right) and strengthens details (right).
To further quantify the text-image alignment, we consider both human annotations and automatic evaluations. For each object mentioned in the prompt, ask annotators whether the object exists in the image and whether it is in the correct color. We also apply a state-of-the-ar detection model GLIP (Li al., (0222) ground each "a [color] [objecr]" phrase into bounding boxes. We report the percentage of images that contain incomplete objects/ complete objects/ / complete objects with correct colors in Table 2 StructureDiffusion improves the compositionality by 3.5% based on human annotations while only 0.2% based on GLIP. We discover that humans disagree with GLIP for more than 50% of the images, as entailed by the low consistency rate. Previous work also suggests the deficiency of large pre-trained models in compositional understanding (Thrush et al. 2022).
3.4
OTHER PROMPTS
We show that our StruetureDiffsiin maintain the overall image quality and diversity on general prompts. We follow the standard evaluation process and generate 10,000 images from randomly
7

---

Published as a conference paper at ICLR 2023
ong shanding
large craeling
bodkyborin boarie Chistres Not oftinng nen la plos, con tna
on surfooard on the beach
throught lenchcape Includes show and bore trees
mude
Ieility
Surf. bourd
standingg
Chuteroc
beach
Lenkeape
edlad
.com
Figure 6: Qualitative results of using scene graph parser to generate structured representations.
(bos), by fo tas (pad, (nadz), -. (pudei=z)
Text encoding stage
CLIP
20.5
38
Eaue Eg. Egy- -, Eiues Paasi... pas
37
20.0
Full: Edon Ep, Epaa*
Seq. Length: 77
36
Inception Sco
One pad: Enoy, En Epan
Seq. Length: k 2
19.5
FID
No pad: Eboss Ei..., Ex
/ Seq. Length:
-A-
15
34
One pad only Fauss Fran
/Seq. Length: 2
19.0
33
18.5
32
18.0
31
full
pad
no pad one pad only
Full
One pad
No pad
only
Figure 7: Ablation study on the text sequence embeddings. find that the padding embeddings are fully contextualized, representing the prompt's high-level semantics. However, not all padding tokens are necessary to maintain high-fidelit sege-fidlitty output from Stable Diffusion.
sampled MSCOCO captions. Stable Diffusion obtains 39.9 IS, 18.0 FID and 72.2 R-Precision. Our method achieves 40.9 IS, 17.9 FID and 72.3 R-Precision. StructureDiffusson maintains the image fidelity and diversity as indicated in the comparable S/FII/R-Pree scores.
3.5 SCENE GRAPH INPUT
We show that our method is not limited to constituency parsing but can also be extended to other structured representations, such as scene graphs. As shown Fig. 6 we first adopt the scene graph parser (Wu et al. 2019) and obtain graph like the ones next to each image from the input prompt. The parser returns basic entities and their relations between. We extract text spans of basic entities with their attributes attached and text spans that include two related entities. We provide examples in Appendix 3 and make comparison to the constituency parser. Similarly, we encode these spans separately and re-align each with the entire prompt encoding sequence. On MS MS-COCO, the scene graph parser setting maintains the image quality with 39.2 IS, 17.9 FID, and 72.0 R-Precision. When compared Stable Diffusion ABC-6K, the scene graph parser achieves Win-Lose- Tie in image fidelity. As
22%-2.9%%-32.2
Win-Lose- Tie in image-text alignment and for CC-500, the scene graph parser leads the same output images due the same text spans. We refer to Table 3 and Fig. 12 for more results and comparison.
4 ABLATION STUDY
4.1 RE-ALIGNING SEQUENCE
In Section 2 we describe method to realign the encoding of text span back into the sequence of the full prompt. Since the noun-phrase text spans are shorter than the full sequence, re-alignment ensures that each token's 's value vector corresponds to the correct attention map. On the other hand, naively expanding the span to the length of the full sequence degrades the image quality by ~ IS FID (37.5 IS, 19.8 FID) compared images with re-alignment Stable Diffusion.
8

---

Published as a conference paper at ICLR 2023
4.2 CONTEXTUALIZED TEXT EMBEDDINGS
One limitation brought by our StructureDiffusion is that the cross-attention computation costs increase by the number of noun phrases. Yet we noticed that most of the attention maps are computed from padding embeddings, as Stable Diffusion adopts CLIP text encoders and automatically pads the sequence to 77 tokens. We conjecture that not all padding tokens are necessary for generating high-quality images. As is shown in Fig. 7 study four different patterns of token embeddings. We discover that leaving the nearest padding embeddings maintains a similar IS/ / FID score as the full sequence. Further removing this padding embedding results in apparent degradation. While only using the nearest padding embedding results in the worst image quality, we find that the high-level image layout and semantics are preserved (see bottom right of Fig. 77) This phenomenon indicates that the padding embeddings are fully contextualized with the full prompt semantics. This also justifies our re-alignment operation that preserves padding embeddings of the main sequence Whall
RELATED WORK
Text-to-Image Synthesis The diffusion model is an emerging type of model that generate high-
quality images with a much more stable training process (Song & Ermon 2019: Ho al. 2020). Rombach et al. (2022) proposes to encode image with an autoencoder and then leverage diffusion model to generate continuous feature maps in the latent space. Stable Diffusion Rombach et al. (2022) adopts similar architecture but is trained large-scale image-text datasets with fixed CLIP text encoder. Imagen (Saharia et al., 2022) addresses the importance of language understanding by using a frozen T5 encoder (Raffel et al. (020)), dedicated large language model. We mainly focus on diffusion models and conduct our experiments on Stable Diffusion (Rombach et al. 2022). the SOTA open-sourced T2I model.
Compositional Generation The compositional or controllable generation has been an essential direction for T2I models to understand and disentangle basic concepts in the generation process. As text inputs are relatively weak conditions. previous work leverage layout or scene graph to enhance compositionality (Johnson et al. 2018: Hong et al. 2018 Yang et al. 2022 Gafni et al., 2022). More recently. [Liu et al. (2022) proposes an approach where the concept conjunctions are achieved by adding estimated scores from a parallel set of diffusion processes. contrast, our method can be directly merged into the cross-attention layers with much less computational overhead.
Diffusion Guidance Ho & Salimans 2022)
develops classifier free guidance where a single
diffusion model is jointly trained under conditional and unconditional inputs. Most large-scale SOTA models, including autoregressive ones, adopt this technique for flexible and improved conditional
synthesis results (Rombach et al.) 2022 Ramesh al. 2022] Gafni al.] 2222)] Yu et al. 2022] Saharia et al. 2022). Hertz et al. (2022) discovers unique properties of cross attention maps on Imagen (Saharia al. 2022) and achieves structure-preservvngg image editing by manipulating these
maps. We observe similar properties in Stable Diffusion (Rombach et al. 2022) but propose a
different algorithm for fine-graine -na-graided, compositional text-to-image generation.
CONCLUSION
In this work, we propose training-free method for compositional text-to-image generation. First, we observe that existing large-scale T2I diffusion models can still struggle in compositional image synthesis. We address this challenge by explicitly focusing on binding objects with the correct attributes. Second, we propose structured diffusion guidance incorporating language structures into the cross-attention layers. We propose two simple techniques to align the structured encoding with the attention maps. Using our structured guidance on Stable Diffusson,, attributes can be bound more accurately while maintaining the overall image quality and diversity. In addition, we justify our approach by conducting an in-depth analysis of the frozen language encoder and attention maps. Future work may explore explicit approaches to generate plausible image layouts without missing components. We hope that our approach accelerates the development of interpretable and efficient methods for diffusion-based text-to-image models.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGEMENT
We would like to thank the Robert N. Noyce Trust for their generous gift to the University of California via the Noyce Initiative. The work also partially funded by an unrestricted gift from Google and by the National Science Foundation award #2048 122. The writers' opinions and conclusions in this publication are their own and should not be construed as representing the sponsors' official policy, expressed or inferred.
REPRODUCIBILITY STATEMENT
We release codebase containing the methodology implementation, settings, benchmarks containing compositional prompts under supplementary materials.
ETHICAL STATEMENT
As for the data collection and verification, we use the Amazon Mechanical Turk platform and form the comparison task as batches of HITs. We select workers from the US, CA, UK, AU, and NZ, since the task require understanding the English input prompt. Each HIT takes around 15-30 seconds on average accomplish, and we pay each submitted HIT with 0.15 US dollars, resulting in an hourly payment of 18 US dollars.
countries, including
REFERENCES
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans image synthesis. Advances in Neural Information Processing Systems, 34:8780-8794, 2021.
Ming Ding, Wendi Zheng, Wenyi Hong, and Jie Tang. Cogview2: Faster and better text-to-image generation via hierarchical transformers. arXiv preprint arXiv:2204.14211 2022.
Alaaeldin El-Nouby, Shikhar Sharma, Hannes Schulz, Devon Hjelm, Layla El Asri, Samira Ebrahimi Kahou, Yoshua Bengio, and Graham W.Taylor. Tell, Draw, and Repeat: Generating and Modifying Images Based on Continual Linguistic Instruction. In ICCV, 2019.
Tsu-Jui Fu, Xin Eric Wang, Scott Grafton, Miguel Eckstein, and William Yang Wang. SSCR: Iterative Language-Basee Based Image Editing via Supervipd Counterfactual Reasoning. In EMNLP, 2020.
Oran Gafni, Adam Polyak, Oron Ashual, Shelly Sheynin, Devi Parikh, and Yaniv Taigman. Make-a- scene: Scene-based text-to-image generation with human priors. arXiv preprint arXiv:2203. 13131, 2022.
Matt Gardner, Yoav Artzi, Victoria Basmov, Jonathan Berant, Ben Bogin, Sihao Chen, Pradeep Dasigi. Dheeru Dua. Yanai Elazar, Ananth Gottumukkala,, et al. Evaluating models' local decision boundaries via contrast sets. Findings of Empirical Methods in Natural Language Processing. 2020.
Shuyang Gu, Dong Chen, Jianmin Bao, Fang Wen, Bo Zhang, Dongdong Chen, Lu Yuan, and Baining Guo. Vector quantized diffusion model for text-to-image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10796, 2022a.
Shuyang Gu, Dong Chen, Jianmin Bao, Fang Wen, Bo Zhang, Dongdong Chen, Lu Yuan, and Baining Guo. Vector quantized diffusion model for text-to-image synthesis. Proceedings of the IEFF/CVF Conference Computer Vision and Pattern Recognition, pp- 10696-10706, 2022b.
Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Prompt- to-prompt image editing with cross attention control. arXiv preprint arXiv:2208.0163 2022.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
10

---

Published as a conference paper at ICLR 2023
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arX1v:2207.12598 2022.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6885 2020.
Seunghoon Hong, Dingdong Yang, Jongwook Choi, and Honglak Lee. Inferring semantic layout for hierarchical text-to-image synthesis. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1994,7994, 2018.
Justin Johnson, Agrim Gupta, and Li Fei-Fei. Image generation from scene graphs. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1219-1228 1228, 2018.
Doyup Lee, Chiheon Kim, Saehoon Kim, Minsu Cho, and Wook-Shin Han. Autoregressive image generation using residual quantization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11523-11532, 2022.
Bowen Li, Xiaojuan Qi, Thomas Lukasiewicz, and Philip Torr. Controllable text-to-image generation. Advances in Neural Information Processing Systems, 32, 2019.
Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang. Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang.Jenq-Nenn Hwang, et al. Grounded language-image pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp.
10965 10975, 2022.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Luping Liu, Yi Ren, Zhijie Lin, and Zhou Zhao. Pseudo numerical methods for diffusion models on manifolds. In International Conference on Learning Representationss 202
Nan Liu, Shuang Li, Yilun Du, Antonio Torralba, and Joshua B Tenenbaum, Compositional visual generation with composable diffusion models. arXiv preprint 1oxiv:2206.017144, 2022.
Xihui Liu, Dong Huk Park, Samaneh Azadi, Gong Zhang, Arman Chopikyan, Yuxiao Hu, Humphrey Shi, Anna Rohrbach, and Trevor Darrell. More control for free! image synthesis with semantic diffusion guidance. arXiv preprint wxxi:2212.00574. 2021b.
Chao Lou, Wenjuan Han, Yuhuan Lin, and Zilong Zheng. Unsupervised vision-langusee language parsing: Seamlessly bridging visual scene graphs with language structures via dependency relationships. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 15607-15616, June 2022.
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. ICML, 2021.
Dong Huk Park, Samaneh Azadi, Xihui Liu, Trevor Darrell, and Anna Rohrbach. Benchmark for compositional text-to- rext-to-image synthesis. In Thirty-fftth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1), 2021.
Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D. Manning. Stanza: Python natural language processing toolkit for many human languages. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, 2020.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference Machine Learning, pp. $748-8763, PMLR, 2021.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang. Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-67, 2020. URL http://jm
http::///m...or//pppprrs. www.//ppgr......oooooooo. pwtpg//aaaar///22/220.... 07..htm.. htm!
11

---

Published as a conference paper at ICLR 2023
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot eext-to-image generation. In International Conference on Machine Learning, pp. 8821-8831. PMLR, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06115, 2022.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pp. 10684-10692 2022.
Olaf Ronneberger, Philipp
Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer- assisted intervention, pp. 234-241. Springer, 2015.
Nataniel Ruiz, Yuanzhen Li, Varun Jampani. Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. arXiv preprint arXiv:2208. 12242, 2022.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, et al. Photorealistic text-to-image diffusion models with deep language understanding. arXiv preprint arXiv:2205.11487, 2022.
Tim Salimans, Ian Goodfellow, Wojeiech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans, Advances in neural information processing systems, 29, 2016.
Sebastian Schuster, Ranjay Krishna, Angel Chang, Li Fei-Fei, and Christopher Manning. Generat- ing semantically precise scene graphs from textual descriptions for improved image retrieval. In Proceedings the fourth workshop on vision and language, pp. 70-80, 2015.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in Neural Information Processing Systems, 32, 2019.
Ming Tao, Hao Tang. Fei Wu, Xiao Yuao Jing, Bing-Kun Kun Bao, and Changsheng Xu. Df-gan: A simple and effective baseline for text-to-image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 16515-16525, 2022.
Tristan Thrush, Ryan Jiang, Max Bartolo, Amanpreet Singh, Adina Williams, Douwe Kiela, and Can- dace Ross. Winoground: Probing vision and language models for lisouling compositionality. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pp. 1288-5248,77 2022.
Bo Wan, Wenjuan Han, Zilong Zheng, and Tinne Tuytelaars. Unsupervised vision-language grammar induction with shared structure modeling. In International Conference on Learning Representa- tions, 2021.
Hao WU, Jiayuan Mao, Yufeng Zhang, Yuning Jiang, Lei Li, Weiwei Sun, and Wei-Ying Ma. Unified visual-semantic embeddings: Bridging vision and language with structured meaning representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1609-6618, 2019.
Zuopeng Yang, Daqing Liu, Chaoyue Wang, Jie Yang. and Dacheng Tao. Modeling image compo- sition for complex scene generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7764 7773, 2022.
Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfci Yang, Burcu Karagol Ayan, et al. Scaling autoregressive models for content- rich text-to-image generation. arXiv preprint arXiv:2206. 10789, 2022.
Han Zhang, Jing Yu Koh, Jason Baldridge, Honglak Lee, and Yinfei Yang. Cross-modal contrastive learning for text-to-inn -iattto-image generation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 833-842, 2021.
12

---

Published as a conference paper at ICLR 2023
Yiwu Zhong, Liwei Wang. Jianshu Chen, Dong Yu, and Yin Li. Comprehensive image captioning via scene graph decomposition. In European Conference on Computer Vision, pp. 211-229. Springer, 2020.
Yufan Zhou, Ruiyi Zhang, Changyou Chen, Chunyuan Li, Chris Tensmeyer, Tong Yu, Jiuxiang Gu, Jinhui Xu, and Tong Sun. Towards language-fice free training for text-to-image generation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp.
17907-17917, 2022.
Minfeng Zhu, Pingbo Pan, Wei Chen, and Yi Yang. Dm-gan: Dynamic memory generative adversarial networks for text-to-image synthesis. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 5802-5810, 2019.
A RELATED WORK
Text-to-Image Synthesis
There are mainly three types of models for text-to-image synthesis:
GAN-based (Tao et al. 2022 Zhu al. 2019: Li et al.; 2019 Fu et al. 2020 El-Nouby et al. 2019
autoregressive (Gu et al. [022b]] Lee et al. [2022] Ding et al. 2022) and diffusion models (Liu et al.
2021b Nichol et al. 2022 Ruiz et al. (022). Zhang et al. (021) proposes XMC-GAN, one-stage
GAN that employs multiple contrastive losses between image-image image-text, and region-token pairs. More recently, LAFITE (Zhou et al. 2022) enables language- free training by constructing pscudo image-text feature pairs using CLIP (Radford et al.) 2021). As for autoregressive models, DALL-E adopts VQ- to quantize image patches into tokens and then uses a transformer to generate discrete tokens sequentially (Ramesh al. 2021). Parti Yuu et al., 2022 and Make-A-Scene (Gafni et al. 2022) both leverage classifier-free guidance to improve controllability. As for diffusion models, Gu al. (2022a) concatenates VQ-VAE with the diffusion model and shows that the diffusion process can operate in discrete latent space. DALL-E 2 adopts the CLIP text encoder so that the diffusion process inverts the textual features into images (Ramesh et al., 2022).
Structured Representations for Vision and Language Inferring shared structures across language and vision has been long-ten pursuit in unifying these modalities (Schuster et al., 2015 Johnson al. 2018 Zhong et al. 2020: Lou et al. 2022). Wu et al. (2019) utilizes the structure from semantic parsing visual-semantic embedding framework to facilitate embedding learning. [Wan al. (202T) proposes a new task in which the goal is to learn joint structure between semantic parsing and image regions. To the best of our knowledge, our work is the first attempt in T2I to incorporate language structures into the image synthesizing process.
Diffusion Guidance To convert an unconditional diffusion model into a class-conditional one, Dhariwal & Nichol (2021) input the noisy image from each step into classifier and calculate the classification loss. The loss can be back-propagated to the image space to provide a gradient that marginalizes the score estimation from the log of conditional probability. Similarly, in the T2I subdomain, Liu et al. (b021b) and Nichol et al. (202I) apply noisy CLIP model to measure the cosine similarity between text prompts and noisy images.
B IMPLEMENTATION DETAILS
Throughout the experiments, we implement method upon Stable Diffusion v1.4. For all compar- isons between our method and Stable Diffusion, we fix the seed to generate the same initial Gaussian map and use 50 diffusion steps with PLMS sampling (Liu al. 2021a). We fix the guidance scale to 7.5 and equally weight the key-value matrices in cross-attention layers if not otherwise specified. We do add hand-crafted prompts such as "a photo of" to the text input. We use the Stanza Library Q et al. 2020) for constituency parsing and obtain noun phrases if not otherwise specified.
C VISUALIZATION OF ATTENTION MAPS
In this section, we demonstrate the visualization of cross-attention maps to support our assumptions and claims in Sec. 2 As is shown in Fig. 8 the attention maps of Stable Diffusion and method
13

---

Published as a conference paper at ICLR 2023
"A long, narrow yellow kitchen with black and white floor tiles"
t=
>t=
Token = "yellow"
input blocks.al (16, 16)
Stable Diffusion
Stable Diffusion
Ours
Token
Stable Diffusion
Ours
Ours
T Token
"white"
Stable Diffusion
"floor"
Stable
Ours
Figure 8: Visualization of cross attention maps of Stable Diffusion and our method. We compare maps multiple tokens throughout the whole diffusion process with equal intervals.
have similar spatial distribution and highlights throughout the diffusion process. This phenomenon supports our assumption in Sec. that the attention map M, is unchanged even with multiple values in each cross-attention layer. We can observe similar phenomenon in Fig. excepp that our method accelerates the formation of interpretable attentions for both "green" and "clock" tokens.
Fig. 8 also justify our claim that values represent rich textual semanties mapped the image space as contents. For instance, our method parses the prompt in Fig. 8 into "A long narrow yellow kitchen" and "black and white floor tiles", encodes and aligns them separately form V. Empirically, these operations enhance the semantics of "yellow" and "black and white" separately and mitigate "yellow" being blended into "black and "hite". This explains the disappearance of color leakage in our image compared to Stable Diffusion. Though one may attribute the leakage to incorrect attention distribution the "yellow" token, we argue that this is not the critical reason. Despite the attention maps of "yellow" from our method slightly highlighting the "floor tile" regions, we cannot observe any yellow in our generated image. This proves that inaccurate attention distributions contribute little to the final image content. In addition, also show in Fig. that that using multiple Keys is able to rectify the image layouts to mitigate missing object issues. The sheep-like like attention maps the third verify the proposed variants of method for concept conjunctions.
14

---

Published as a conference paper at ICLR 2023
"A brown building tall green tower behind"
Stable Diffusion
Token "green"
input_ blocks. B.1, (16. 16)
Stable Diffusion
Ours
Ours
Token "clock"
Stable
Diffusion
Figure 9: Visualization of cross attention maps corresponding to token "green" and "clock" across the full diffusion timestamps from step 50 to step in equal intervals. Red boxes highlight steps where our method accelcrates the formation correct attention on the clock region. The evolution of the token "green" is also interpretable in method. Although the image composition is imperfect, the visualization still supports our assumptions and claims Sec.
Attention maps of token "sheep" in "a red car and white sheep"
input_bles bww.eblloccssso. 8.1, (16, 16)
Stable Diffusion M,
Ours, Mp
Ours, Mp
Ours, M;
Ours, M2
Visualization of attention maps for token "sheep" of different methods. Our method with multiple Keys successfully rectify image layouts.
Figure 10:
D ABLATION STUDY
D.I A CASE STUDY OF ATTRIBUTE BINDING
Here, we present a case study to show evidence of two root causes of incorrect attribute binding. The first one is the contextualized token embeddings due causal attention masks. As is shown the left side of Fig. [| we first encode two different prompts with a shared component, e.g. "a red apple" the naive and "a green bag and red apple". Using the encoding sequence of the naive prompt,
15

---

Published as a conference paper at ICLR 2023
red apple
<hescs A gremn hag and a red apple spud>
Prompt furry bear watching bird
Attention maps of token "bird"
CLIPtest
CLIPteu
Input block
Input block 7
Input block
apple <pads
green bog redapple <pads
Intialiiation
Output block
Output block
Final Image
Figure Examples showing the potential root causes of incorrect attribute binding. Left: The large green regions in the second image prove that the hidden state's 's output of token "red" is contextualized with token "green" before Right: Visualization of attention maps showing that the semanties from the token "bird" is mistakenly attended the mouth region of the bear. The final image shows the unnatural beak-like shape of the bear.
we are able to get an image of red apple only. It reasonable to assume that the yellow green regions are natural results of learning from authentic apple images. Then, replace the tokens of the naive prompt with embeddings of the same token from the more complicated prompt. We use the same gaussian noise initialization and generate unnatural image with a solid green region (in the yellow bounding box). This result proves that the token "red" is contaminated with the semantics of ""green" before it and explains some images with color leakage problems (e.g.. Fig. D.
The second reason attributes to inaccurate attention maps. In Fig. II (right), we visualize five cross- attention maps (averaged across attention heads) from both downsample and upsampling blocks. The attention maps show the salient regions corresponding to the token "bird". These maps demonstrate highlighted regions in the bottom left corner where the bird is located in the final image. Despite the interpretable structures, the maps also show saliency around the mouth region of the bear across all five layers. Thus, the inaccurate attention maps lead beak-like mouth of the bear in the image.
D.2 COMPARISON OF PARSERS
In this subsection, we compare the difference between using a constituency parser and scene graph parser to obtain text spans and generate images. Table 3 compares the extracted text spans using constituency parser and scene graph parser. Example shows that both parsers end up with the same results for CC-500 prompts. For Example 1-4, the scene graph parser generates more spans than the constituency parser. We notice that concepts the middle of the sentence appear more often in these spans than other noun tokens, like "egg" or "red sauce" in Example 3. This imbalance potentially explains why the "egg" looks more highlighted in Fig. 12 (bottom left). On the other hand, "orange slices" appear more often in constituency parsing results, leading better "orange" textures in the generated image. Similar observations can be made in Example 2, where "green pole" emphasized more often by the constituency parser.
E LIMITATIONS & FUTURE WORK
There are several limitations of our work. First all, our method depends on an external parsing function that may not be perfect. We adopt the commonly used Stanza Library Qi et al. (2020) for constituency parsing. The parsing function can be replaced with a more advanced learning based method for improvement. Secondly, our method mainly focuses compositional T2I neglecting any
16

---

Published as a conference paper at ICLR 2023
Constituency Parser
Scene Graph Parser
CC-500 Prompt: A white sheep and red car
Example
"A white sheep", "a red car"
"A white sheep", red car"
Prompt: A silver car with black cat sleeping on top of
Example 1
"A silver car", "a black cat",
"A silver car",, "a black cat", "top of' "a black cat sleeping on top of it"
"A silver car with black cat"
Prompt: horse running white field next to a black and green pole
"A horse", "a white field".
"A horse", "a white field".
Example 2
"a black and green pole".
"a black and green pole". "A horse running in a white field"
"a white field next to a black and green pole"
Prompt: Rice with red sauce with eggs over the top and orange slices on the side
"red sauce", "the side", "the top and orange slices".
"red sauce", "the side". "the top and orange slices". "the top and orange slices the side"
Example 3
"Rice with red sauce", "red sauce with eggs". "the top and orange slices the side". "red sauce with eggs over the top and orange slices"
Prompt: A pink scooter with black seat next to a blue car
Example 4
"A pink scooter", "a black seat", "a blue car", "a pink scnoter with black seat", "a black seat next to a blue car"
"A pink scnoter", "a black seat", "a blue car"
Table 3: Comparison between the constituency parser and scene graph parser. For CC-500 prompts, both parsers end up with the same results. As for general prompts, scene graph parser tends to generate more text spans with middle concepts appearing multiple times across different spans.
Stable Diffusion
Ours (Constituency)
Ours (Scene Graph)
Stable
Ours (Constituency)
Ours (Scene Graph)
Diffusion
A silver with black cat sleeping top of
horse running in white next to a black and green pole
Rice with red sauce with eggs over the top and orange slices on the side
pink scooter seat next to a blue car
DOD
Figure 12: Synthesized images corresponding to prompts in Table Yellow boxes annotate composi- tions that are improved using different parsers.
style descriptions. The parsing mechanism may categorize a style description, e.g. "in Van Gogh style" as a separate noun phrase that cannot be grounded in the image space. In addition, we discover that StructureDiffusion tends to generate similar images as Stable Diffusion. Thus we filtered out 20% of most similar image pairs in Table considering the efficiency of human evaluation. Therefore, the improvement could be compromised when evaluated on the full set of generated images. Future work may focus on devising explicit methods to associate attributes to objects using spatial information as input. For example, how to make a text-to-image synthesis model interpret coordinate information with limited fine-tuning or prompt tuning steps would be an appealing direction.
F ADDITIONAL RESULTS
17

---

Published as a conference paper at ICLR 2023
Stable Diffusion A green apple and red banana
Ours
Stable Diffusion
Ours
A red bird and a green banana
A gold clock and green bench
A blue bird and brown bowl
A red cake and blue suitcase
A blue backpack and brown bear
brown dog and blue suitcase
blue backpack and a brown cow
A yellow bowl and blue cat
A brown dog and blue horse
Figure 13: Qualitative results on CC-500
18

---

Published as a conference paper at ICLR 2023
Stable Diffusion
Stable
Stable
Ours
Ours
Ours
Diffusion
Diffusion
purple with orange hat on its head
cat sits on a rug with black cord
A yellow cat is wearing blue plastic baseball hat.
Two elephants walking by green wall with tan palm trees painted on
red helmet is yellow toilet in the dirt
A red stop sign above white walk across road sign
STOP
AOID
A spacious kitchen has white walls, red countertops, and large stove
large white bed sitting in a hotel room next red couch
bathroom with red tile and green shower curtain
white toilet bowl with a purple rug front
A pink towel stands out greatly in the white bathroom
A large pizza white plate sitting on blue table
black and white curtain hanging in that is decorated in black, white and
A spoon and bowl of red pea soup and green beans with onions
A cow standing outside of white building with blue entrance
Figure 14: Qualitative results on ABC-6K
19

---

Published as a conference paper at ICLR 2023
Stable Diffusion
Structure Diffusion
Stable Diffusion
Structure Diffusion
Stable Diffusion
Structure Diffusion
two back to back wooden benches next shrub stone walkway
small wooden laptop sitting of silver computer desk
small silver laptop sitting top of wooden computer desk
a skateboarder jumps over silver dome with stone domes front and bicycler behind
shiny silver metal microwave near wooden cabinets
wooden
stone building with wooden bench placed near
wooden building with stone bench placed near
cat is perched upon stone bench that sits on wooden patio
wooden building with stone bench placed near it
a red sphere and yellow cube
a brown sphere and cyan cube
sphere a cube
sphere
green sphere a
Figure 15: Qualitative results characterizing attributes beyond colors, including shape, size and materials.
20

---

Published as a conference paper at ICLR 2023
fantasy anime
portrait
base
ours
base
ours
sharp focus
kawail, Japanese anime ours
sharp focus, thick lines burs
fantasy anime, concept art base
base
ours
matic studig lighting, kyoto animation style, sharp base
cinematic studio lighting, kyoto animation style base
ours
ours
kawaii, concept art, fantasy anime base
concept art, japanese anime, thick lines base
ours
ours
Figure 16: A prompt "an astronaut riding horse" appended with different (combinations of) style descriptions. Our method has negative effects on the image style. "base" refers to Stable Diffusion.
21


Reviews:
[{"review_id": "vWyXY5pRZXJ", "reviewer": "Reviewer_QzN2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new diffusion-based T2I models to improve the compositional skills. The linguistic structures are incorporated with the diffusion guidance process based on the controllable properties of manipulating cross-attention layers in diffusion-based T2I models. The model is built based on the stable diffusion, and the proposed cross-attention design requires no additional training samples. The model can achieve SOTA compositional skills in qualitative and quantitative results.", "summary_of_the_review": "I think the proposed linguistic insights strategy is a useful trick for T2I diffusion models. However, the experimental results are not convincing enough.", "strength_and_weaknesses": "Strength:\nThis paper proposes to manipulate the cross-attention representations based on linguistic insights. And the compositional performance can be obviously improved.\n\nWeakness:\n\n1. As shown in Figs. 4 and 5, the proposed diffusion model's effects are mainly observed with the color attributes. The authors should provide the comparison with more types of attributes to demonstrate the improvement of compositional skills.\n\n2. Why the quantitative comparisons between Stable Diffusion and Composable Diffusion are not conducted with the metrics of IS, FID, and R-prec?\n\n3. The authors have claimed that their method can also take the scene graphs as the inputs. They should also compare the results with existing scene-graph to image approaches. And the visual cases in Fig. 6 have some errors. For example, there is not beach in the middle column of Fig. 6, while beach exists in the scene graph.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "tLsCU_3Ojx", "reviewer": "Reviewer_mgR1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a method for improving the attribute binding and compositional ability of denoising diffusion probabilistic models for image generation. The authors utilize the fact  that the attribute-object relation pairs can be obtained as text spans for free from the parsing tree of the sentence used for conditioning image generation. Using this free source of information, the authors propose  a method to combine the structured representations of prompts with the diffusion guidance process. Specifically, using the text spans from the parsing tree, the method obtains a cross-attention output that is a weighted average of the attention map of the full prompt multiplied by the value tensor of the full prompt and the value tensors of individual noun-pairs obtained by the parsing tree. To evaluate the method, the authors create two datasets. The first dataset, Attribute Binding Contrast set (ABC-6K) evaluates attribute binding ability of the method, by creating prompts with common and uncommon colors binding to objects in the COCO dataset. The second dataset, Concept Conjunction 500 (CC-500), tests the model’s ability to compose together objects that rarely appear together in reality. Pairwise comparisons in human evaluation study are used to compare the method to the baseline model (Stable diffusion) and to Composable diffusion for the CC-500 dataset. The proposed approach, StructureDiffusion, is preferred by users over Stable Diffusion on ABC-6K, while it performs worse on CC-500. However, StructureDiffusion outperforms Composable Diffusion on CC-500. FID scores on COCO are comparable to original Stable Diffusion. ", "summary_of_the_review": "I am on the fence about how effective the proposed approach is in solving the problems of attribute binding and especially compositionality. It is also not clear to me why and how the proposed approach is working in different cases. So currently, I consider the paper to be borderline, but I am happy to re-evaluate after the rebuttal stage if I get clarification on the issues raised above.  ", "strength_and_weaknesses": "Strengths: \n1. The proposed idea of using the free supervision from structure in the prompts is interesting and worth investigating. \n2. The problems of attribute-binding and composition in image generation are important and retain scope for improvement. \n3. The authors propose two well-crafted datasets for evaluating the attribute-binding and compositional ability of image generation models and perform a comprehensive comparison of the proposed approach with baseline using a human study.\n4. The paper is well written and organized, which makes it easy to follow and reproducible. \n\nWeaknesses/suggestions:\n1. It was not clear to me how human evaluators assess “image fidelity” in table 1. Please clarify this in the rebuttal.   \n2. It was not clear to me why the authors claim that  “[On CC-500] our method outperforms Stable Diffusion by around 5-8% in terms of both image-text alignment and image fidelity”, when the reported numbers in Table 1 show that StructureDiffusion wins against Stable Diffusion (SD) only 31.8% of the time, while SD wins 38.9%. These numbers make me conclude that SD is actually superior to StructureDiffusion for compositionality. Please clarify this in the rebuttal.  \n3. If I am interpreting the results in Table 1 correctly, then  StructureDiffusion is 20% better than SD in alignment, but 8% worse in compositionality. These results are not discussed in detail in the paper, which could help us understand how the proposed approach is helping in these tasks. \n4. No qualitative experiments are performed to show how the weighed cross-attention maps affect image formation (see the Hertz et al. paper for examples, which is used as an inspiration for the proposed approach) during the diffusion process. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "8Gs60p7Luv", "reviewer": "Reviewer_5ZP1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a modification of stable diffusion to better reflect per-object attributes in a sentence to the corresponding objects in the image and prevent missing objects.\nFor example, feeding \"a red car and a white sheep\" into stable diffusion produces and image with a red car and a red sheep.\nFor experiments, the authors introduce a new dataset for evaluation.", "summary_of_the_review": "I value the colors being correct in the resulting images. However, contribution of this paper is limited as mentioned in weaknesses. I consider the bar for ICLR is much higher.", "strength_and_weaknesses": "Strengths\n\n(+) The proposed method does not require training samples.\n\n(+) This paper introduces a benchmark dataset to evaluate attribute binding.\n\n(+) The experiments visualizes the causes of incorrect attribute binding (Figure 8).\n\n(+) Sec. 2.1 provides thorough background.\n\nWeaknesses\n\n(-) Missing analysis: How do the attention maps from NPs and sentence evolve throughout the timetsteps?\n\n(-) The contributions are limited.\n- The attributes are limited to colors.\n- The authors do not provide comparisons in the scene graph setting.\n- Evaluation of the compositionality is limited to user study.\n- Preventing objects from missing is not evaluated.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "7_A7vgqZc5", "reviewer": "Reviewer_ZtBV", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a way to improve compositional text-to-image in large, pre-trained diffusion models without additional data or training requirements. To achieve this, language parsers are used to extract high-level information from a given caption (e.g. noun phrases) and use those to obtain additional text embeddings. These text embeddings are then used in combination with the original ones to scale the outputs of cross-attention layers. Experiments show that the approach can improve attribute binding and compositionality in generated images.", "summary_of_the_review": "Overall I think the approach is novel, may increase the performance in some specifc settings, and does not need any additional training or data. I'm not entirely sure about the evaluation with the human user study and how well the approach will scale to more complex/longer captions or the kinds of captions that users seem to use in reality.\nHowever, I see value in the approach and its simplicity and easy applicability are advantageous.", "strength_and_weaknesses": "Previous work has shown that the cross-attention layers have a lot of influence on the final content of generated images and, therefore, it makes sense to look into those more closely in this respect. The proposed approach does not need any additional training data or training of the model, making it easy to use in practice. The qualitative evaluation shows some improvement over the baseline model (Stable Diffusion) and human user studies indicate that the generated images are indeed better.\n\nSome questions and concerns:\n- I am not sure about the human user study; the authors say that they generate images from both approaches (their approach and the baseline Stable Diffusion model), then remove 20% of the most similar images, and only then run the user study; I understand that a lot of the images are similar since the same seed is used, but given that the performance of both approaches is probably similar on those 20% of the images this will affect the final results; in reality, the performance of the two models may be much closer to each other than the human user study indicates\n- Table 1, CC-500, comparison to SD seems that SD is actually better, but the text seems to indicate otherwise?!\n- Does it scale to longer captions with many constituents?\n- Also, looking at real-life use people seem to append a lot of words for specific styles, how would behavior like that affect the performance? Captions like *a photograph of an astronaut riding a horse portrait pixiv art soft lines fantasy anime smooth illustration murata range 8 k studio ghibli sharp focus wlop kyoto animation style art viewer anime art close sharp detailed pixiv kawaii concept art high quality cute sunset exile intricate hd elegant 4 k cute face artgerm madhouse stunning ufotable valley alphonse mucha detailed background anime portrait yoshinari yoh artgem detailed face japanese anime cinematic studio lighting pink hue white background thick lines large eyes*", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster