Scene Graph Disentanglement and Composition for Generalizable Complex Image Generation
Authors: Yunnan Wang
Ziqiang Li
Wenyao Zhang
Zequn Zhang
Baao Xie
Xihui Liu
Wenjun Zeng
Xin Jin
Venue: NeurIPS 2024 spotlight (2024)
Keywords: Scene Graph; Disentanglement; Diffusion Model; Compositional Image Generation
Primary area: generative_models
TL;DR: In this paper, we leverage the scene graph, a powerful structured representation, for complex image generation.

Abstract: There has been exciting progress in generating images from natural language or layout conditions. However, these methods struggle to faithfully reproduce complex scenes due to the insufficient modeling of multiple objects and their relationships. To address this issue, we leverage the scene graph, a powerful structured representation, for complex image generation. Different from the previous works that directly use scene graphs for generation, we employ the generative capabilities of variational autoencoders and diffusion models in a generalizable manner, compositing diverse disentangled visual clues from scene graphs. Specifically, we first propose a Semantics-Layout Variational AutoEncoder (SL-VAE) to jointly derive (layouts, semantics) from the input scene graph, which allows a more diverse and reasonable generation in a one-to-many mapping. We then develop a Compositional Masked Attention (CMA) integrated with a diffusion model, incorporating (layouts, semantics) with fine-grained attributes as generation guidance. To further achieve graph manipulation while keeping the visual content consistent, we introduce a Multi-Layered Sampler (MLS) for an "isolated" image editing effect. Extensive experiments demonstrate that our method outperforms recent competitors based on text, layout, or scene graph, in terms of generation rationality and controllability.

Scene Graph Disentanglement and Composition for Generalizable Complex Image Generation
Yunnan Wang!.2 Ziqiang Li Wenyao Zhang
Zequn Zhang Baao Xie Xihui Liu" Wenjun Zeng* Xin J
Shanghai Jiao Tong University, Shanghai. China
2 Ningbo Institute of Digital Twin, Eastern Institute of Technology. Ningbo, China
3 University of Science and Technology of China, Hefei, China
"The University of Hong Kong, Hong Kong, China
(wangyunnan, [igiangli]@s]] edu. on jinxingeitech. edu. cn
Abstract
There has been exciting progress in generating images from natural language or lay- out conditions. However, these methods struggle to faithfully reproduce complex scenes due to the insufficient modeling of multiple objects and their relationships. To address this issue, we leverage the scene graph, powerful structured representa- tion, for complex image generation. Different from the previous works that directly scene graphs for generation, tional autoencoders and diffusion models in a generalizable manner, compositing diverse disentangled visual clues from scene graphs. Specifically, first propose a Semantics- Layout Variational AutoEncoder (SL-VAF) to jointly derive (layouts, semantics) from the input scene graph, which allows more diverse and reasonable generation in one-to-many mapping. We then develop a Compositional Masked Attention (CMA) integrated with diffusion model, incorporating (layouts, se- mantics) with fine-grained attributes as generation guidance. To further achieve graph manipulation while keeping the visual content consistent, we introduce a Multi-Layered Sampler (MLS) for an "isolated" image editing effect. Extensive experiments demonstrate that our method outperforms recent competitors based on text, layout, or scene graph, in terms of generation rationality and controllability. Code available at https: //
employ the generative capabilities of varia-
bttts:://gihhut github......... com/ wwwgwannannnnnnn /DisCo.
Introduction
Text-to-image (T2I) generation with diffusion models (DMs) [1, 2] has yielded remarkable advance- ments [3, 4, 5] in recent years, benefiting from the developments of Dision-language foundation models [6, 7, 9]. However, textual conditions with linear structure struggle to delineate the intri- cacies of complex scenes precisely. For example, as shown in the failure cases of DALL-E 3 [3] in Figure (a), given the intricate text prompt "A sheep by another sheep ... boat on the grass.", ", the T2I model may have difficulty accurately generating object relationships or quantities. Consequently. some studies [10, 11. 12, 13] strive to improve spatial relationship (c.g.. "by" and "on") control by incorporating additional layout conditions. Nevertheless, as illustrated in the failure cases of LayoutDiffusion [10] Figure (b), layout-to-image (L2I) methods inevitably encounter challenges in representing certain non-spatial interactions, such as depicting "playing" within spatial topology.
To efficiently depict complex scenes for guiding generative models, recent methods [14, 15, 16] utilize structured scene graphs as conditions instead of text or layout prompts. Scene graphs [17]
Xin Jin is the corresponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Relationship and Quantity Confusion of T2I
Non-Spatial Interaction Dilemma of L2I
sneep
unother
on the grass wilh the under
the stye
the dcean by Bries
bast Bie DroLs
(a) DALL-E3 (middle) and Ours (right).
(h) LayoutDiffusion (middle) and Ours (right).
Independent Nodes Missing of Semantics-based SG2I
Attribute Control of our SG2I method
bunlding
Dlamoo
BRODDN
Ensn abeve
tost R0
wall
Window)
(c) R3CD (middle) and Ours (right).
(d) Results w/o (middle) and w/ (right) AC.
Figure 1: Failure cases generated by (a) text-to-image (T21) (DALL-E [3], (b) layout-to-image
(1.2I) (I (LayoutDiffusion [10]), and (c) semantics-based scene-gras scene-graph-tee graph-to-image (SG21) (R3CD [14])
methods. (d) Generalizable object Attribute Control (AC) under consistency achieved by our DisCo.
represent scenes with a structured graph format, where objects within the scene are denoted as nodes and the relationships between objects represented edges. Seene-Graph-to-Ime (SG2I) generation is challenging task due to the frequent ambiguous alignment between graph edges and relationshipss/nteeaciion among visual objects. To address this issue, layout-based SG2I methods [15, 17, 18, 19] explicitly predict the spatial arrangements of objects in scenes by additional layout predictors, followed by L2I synthesis according the layout (as demonstrated in Figure 2 (a)). These methods commonly employ one-to-one mapping, i.c., a single scene graph only corresponds to one layout, which severely limits the generation diversity. Besides, they also inherit the limitation of the L2I approach in modeling non-spatial interactions, whereby each object is typically generated independently. In contrast, as shown Figure (b), semantic- sastantee-bsedddd SG2I methods implicitly encode graph edges into node embeddings by graph convolutional networks (GCNs), which effectively aligns object semantics with non-spatial interactions. Nonetheless, these methods are weak in logically determining the spatial positions of independent nodes, which might cause the absence of independent nodes (e.g., the "lamp" and "stone" shown in Figure (c)) the generated image.
In this paper, we propose DisCo, Compositional image generation framework that integrates the Disentangled layout and semantics derived from scene graph representations (as depicted in Figure 2 (c)). To boost the representational capacity of scene graphs for complex scenes, we augment the node and edge representations with CLIP [6] text embeddings, and incorporate extra spatial information (i.e., bounding box embeddings) for nodes during training. Once the textual scene graph constructed, we propose a Semantics Layout Variational AutoEncoder (SL-VAE) based on triplet- GCN [18] to jointly model the spatial relationships and non-spatial interactions in the scene. SL-VAE allows the one-to-many disentanglement for spatial layout and inteructive semantics that match the
GCN bascd VAE
Object faston
Difhedoon
Lapost Decoder
(a) Layout based SG2I
Viiies Ennonder
Laket Spice
Lapouts
Diliska
piree
Semantic Decoder
Diftusen
Nataalie tiabeddings
(b) bementic-based SG21
(c) Our method
Figure 2: Comparison between the previous SG21 architectures and ours. (a) Layout-basee SG2I model [15] generate a spatial arrangement with an object layout; (b) Semantic-based SG2I models [14, 16] build interactive semantic embedding between objects; (c) Our method leverages scene graph representation by jointly deriving the disentangled layout and semantics with the proposed SL- VAE.

---

input scene graph. This is achieved by sampling from Gaussian distribution, offering object-level (layouts, semantics) conditions for the diffusion process [20]. Given that the layout and semantics encapsulate global and local relational information, we further introduce the Compositional Masked Autention (CMA) mechanism to inject object-level graph information with fine-grained attributes into the diffusion model, thereby preventing relational confusion and attribute present a Multi-Layere Sampler (MLS) technique that leverages the diverse conditions generated by SL-VAE, achieving generalizable generation for object-level graph manipulation (i.e., node addition and attribute control) in the SG2I task, as depicted by the color change of two "sheep" in Figure I (d).
we
In summary, our key contributions are as follows: (i) We apply the textual scene graph structured scene representation and introduce the Semantics-Layon Variational AutoEncoder (SL- (SL-VAE) to disentangle diverse spatial layouts and interactive semantics from the scene graph; (ii) We present the Compositional Masked Attention (CMA) to inject extracted object-level graph information with fine- grained attributes into the diffusion model, which avoids relational confusion and attribute leakage; (iii) We introduce the Multi-Layered Sampler (MLS), a technique that leverages the diverse conditions produced by SL-VAE to implement object-level graph manipulation while keeping the visual content consistent; (iv) Our method outperforms current text/layout-bart and achieves significantly superior generation performance compared to state-of-the-at SG2I models, thus showcasing the generalization textual scene graphs in depicting complex scenes.
methods in relationship generation
2 Preliminary
2.1 Text-to-Image Diffusion Models
Diffusion models (DMs) [1] are generative models that Iearn the data distribution p(x) by gradually performing 7'step noise reduction from the variables 2r sampled from the Gaussian distribution (00,1), 1). Thus the training process of DMs can be regarded as the reverse process of Markov chain with a fixed length T. To generate high-resolution images with less computational resources, Latent Diffusion Models (LDMs) [20] encode the image into the latent space = with the pre-trained Vector Quantized Variational AutoEncoder (VQ-VAB) Subsequently, the LDMs aim to predict the distribution p(z) rather than p(æ)). For the text-to-image LDMs, the text is encoded with the CLIP [6] text encoder ECLIP. Then the objective function of a text-guided LDM can be formulated as follows:
LLDM = (0,.e (z, ECLIP ()))
where (CIIP(C) is the text embedding of the text condition c, t is the diffusion step, Ep is a model for estimating the noise e, and 2, a; + o,E is the 1-step noised latent code from the ground-truth Zo. During inference, the model 6g with various samplers [1, 21] gradually denoises the initial noise zT (0, 1). Finally, the predicted latent code is decoded into the image space.
Graph E Encoding for Semantics-Lane Layout VAE
Graph Decoding for Semantics-Lay out VAE
IV. Details of CMA Machanism
Infonstpe
Semambic Decoder
Stating Point
O
Graph Union
Layout
Object levd Pusion
Masto-ttttion
Fokis Fostan:
ILL. V Visuaa Diffusion Prucess with Sumantic and Layout Condition
Visual V Eccoder
Diéfusion Process
MI P
Self thention
Visuual V
Zig
Res embedking / sebadding
2 Learaale ambadiing
CMA Mechanlemm
Figure Framework overview. (I) We parameterize the node embeddings into the Gaussian distribution with the Graph Union Encoder, which jointly models the spatial relationships and non- spatial interactions scene graphs; and interactive semantics sampled from Gaussian distribution, respectively: (III) A diffusion model with the proposed Compositional Masked Attention (CMA) incorporates object-level conditions to generate visual images following the scene graph description; (IV) Detailed structure of CMA Layer.
(II) The Semantic and Layout Decoders generate spatial layouts
3

---

2.2 Scene Graph Representation
The scene graph G = (0, E) [17] presents a structured scene representation. Nodes O= {o:) denotes No objects within the scene. while edges E {er, Jic.jss denotes relationships between objects. All nodes and edges come with a semantic label, denoted as c E C and cj E C', where Co and C are category vocabularies of nodes and edges, respectively. In practice, the nodes Eij; 0j) N representing connections from or to o, serve as inputs for graph convolutional networks (GCNs). Moreover, nodes and edges are typically converted into learnable embeddings using embedding layers denoted as Eemb and E emb
O Jo and the triples T = {hy (i,,
3 Methodology
As illustrated in Figure 3, we present novel SG2I synthesis framework known as DisCo. The DisCo comprises three primary components: (1) Semantics- Layout Variational AutoEncoder (SL-VAE) that disentangle diverse spatial layouts and interactive semantics from the scene graph (Section 3.1); (2) Compositional Masked Attention (CMA) that injects object-level (layouts, semantics) with fine- mne-grained attributes into the diffusion model (Section 3.2); and (3) Multi-Laayrred Sampler (MLSS that implements generalizable generation for object-level graph manipulation (Section 3.3).
3.1 Semantics-Layout Variational AutoEncoder
Textual Scene Graph Construction. For constructing the scene graph representation, we employ the visual-language model to fully leverage the inherent disentangled semantics of the language, while sinultaneously facilitating the alignment between images and scene graphs. Specifically, we augment the node and edge embeddings of the scene graph with CLIP [6] text embeddings. During training, we also incorporate spatial information for node embeddings by including bounding box coordinates (i.e., top-left corner and box size denoted as 6; (It, Vs: Wi, h4)). Then the node embeddings O and edge embeddings & can be formulated as follows:
O = {E (c?) @ (o,) & (b.))
{
@ Eclr((yy)hs.ss.w..t (2)
where ECLIP denotes the frozen pre-trained text encoder, Ebor is the spatial encoder for bounding box coordinates using Multi-Layer Perceptions (MLPs), and @ denotes concatenate operation.
Graph Union Encoding. Although baseut-based SG2I methods are superior in modeling spatial topology compared to the mnnantics-based method, they fall short in capturing object interactions (i.e., non-spatial relationships) within the scene. Accordingly, after obtaining the node and edge embeddings mentioned above, we apply Conditional Variational Autoencoder (CVAE) [22] based on triplet-GCN [18] to jointly model the layout and semantics information. As shown in Figure 3.I, the L-layer Graph Union Encoder E,, takes node and edge embeddings as inputs:
(of 1+1
ij . + GCN.( oly, o)), ( E L0. L-1 1}
where denotes the layer index of Graph Union Encoder, and 6 denotes intermediate features. Here we initialize (6) Φ0 09) (O,, Eyy, O,). Please refer to the Appendix for more details about the triplet- GCN. Given that the last node embedding of integrates both topology and interaction information, we conduct layout-semantic modeling by parameterizing it into Gaussian spaces Z N(1,, σ). In this context, the means μ E RD and variances a E IR Dx are estimated individually by two supplementary MLPs, where D2 denotes the dimensional of latent space for node embedding. Hence, we jointly model the layout and semantics through the following minimization:
(4) where KL denotes the Kullback-Liebler divergence, y denotes condition and the prior p( p(uly) is the standard Gaussian distribution (/u 0, 1). Specifically, we condition the latent space of the graph structure using the edge embedding following Equation 2 alongside the updated node embedding {Eemas (cf) ECLiP (o.) @ u, V.N... where u, is random vector sampled from Z. This architecture ensures that layout is solely necessary for training, with no need for hand-crafted layout in inference.
Lunion KL (uly, O,E) || p(ulz)),
Disentangled Semantics-Layon Decoding. As illustrated in Figure 3.11, we disentangle the explicit spatial layout and implicit interactive semantics from the latent space using two separate triplet-GCN- based decoders, i.c., layout decoder D; and semantic decoder D.. The proposed Semantics-Layon

---

Variational AutoEncode (SL-VAE) comprises these two decoders and the graph union encoder mentioned above, which derives the spatial topology and object interactions from the scene graph representation. During training, the layout decoder is optimized by the following objective function:
No
1
Llayout
16 1=1
6,11,
(5)
where b, denotes the predicted coordinates. We only incorporate the ground truth layout B [b: (b:);= )
during training, while generating N₁ diverse layouts {B, = (-1.)) by sampling Gaussian noise at inference time. For simplicity, we omit the superscript in the following description. The semantic decoder Dg generates semantics embeddings S = {s; } W facilitate subsequent diffusion processes, and its parameters are iteratively updated with the diffusion loss in the next Section 3.2.
3.2 Diffusion with Compositional Masked Attention
[br) and interactive semantics S = { at object level, as illustrated in Figure 3.III. The single-object embeddings (c) = {&s & F(b:)) No are acquired by directly applying semantic embeddings, while encoding box information using Fourier mapping F [23]. We define learnable null embedding to pad the embedding length to N Nmar, thereby accommodating varying numbers of objects:
Object-level Fusion Tokenizer. We integrate the spatial layout B
8 & F(b;),
isNo
Ci
66)
Creulis
otherwise
where Cnall denotes the learnable null embedding for padding. We optionally add attribute embedding A {a, N. to construct updated C fc₂ a, ] J. where c; and a; separately processed by two MLPs before concatenation. Note that a, is obtained similarly to edge embedding in Equation 2. We also define Iearnable null embedding anull for cases where no attribute is specified.
Compositional Masked Attention. The cross-attention mechanism in diffusion bridges the visual and textual information, while self- attention captures self-related information within visual tokens [24]. Therefore, we insert our proposed Compositional Masked Atten- tion (CMA) between self-attention and cross-attention layers. This technique effectively injects graph information into the diffusion process at the object level, preventing semantic confusion and at- tribute leakage through the attention mask. Specifically, we denote the visual token output by the vanilla self-attention as V E R N, D., where N, and D, represent the number and dimensions of tokens, respectively. Then the CMA layer can be expressed as:
visual
objert embedding
(a) Compositional masked attention
visu bakce
object enteriting
-inf
1
-Inf
-inf
1
-inf
-inf
-inf
-inf
-Inf
v SAmast (V @ Ć, M)[: Nol.
where é C= =0 ( Nima denotes object embeddings whose dimensions aligned with the visual token V using MLPs. The matrix M (N,, +Nmea) ( (N, +Nuus) denotes the attention mask that depends on layout B, which can be constructed as follows:
(b) tlention mask
Figure 4: Toy example of (a) compositional masked atten- tion, and (b) its correspond- ing attention mask. We use vi- sual tokens and object embed- dings of objects A and for demonstration. A and B have and visual tokens, respec- tively, whose attribution is de- termined by bounding boxes.
M.j 1, if j fall into the same object
inf, otherwise
where "i.j fall into the same object" means that i and j index the visual tokens or object embeddings of the same object. Figure illustrates the mechanism of CMA through toy example. In con- trast to vanilla self-attention, the proposed CMA prevents relational confusion and attribute leakage between different objects through the well-designed object-level masks mentioned above. As shown in Figure 3.IV, we forward the output of the CMA layer into the subsequent layers, serving the updated visual token.
Diffusion Loss. Based on the object semantics output from the proposed SA,, we optimize the evidence lower bound between sampling noise and prediction noise conditioned on object-level
5

---

Table 1: Performance comparison on COCO-Stuff and Visual Genome datasets using Inception Score (IS) and Fréchet Inception Distance (FID) metrics. We report the results of methods with two generator structures, namely GAN- and Diffusion-based. The architecture of these methods based on the layout (L. or semantics (S), while our approach includes both. The best results are bolded.
COCO [26]
Visual Genome [27]
Method
Type
IS f
FID μ
IS f
FID
Real Image
30.7
27.3
SG2Im [17]
8.2
99.1
90.5
PasteGAN [28]
L
12.3
79.1
8.1
66.5
GAN-based
SOAP [18]
L
14.5
81.0
WSGC [19]
L
6.5
121.7
9.8
84.1
KCGM [29]
S
11.6
27.4
LDM [20]
S
22.2
63.8
16.5
45.7
SGDiff [16]
17.8
36.2
26.0
Diffusion-basse
SceneGenie [15]
L
22.2
63.3
20.3
42.2
R3CD [14]
S
19.5
32.9
18.9
23.4
DisCo (ours)
L+S
23.1
30.8
22.3
21.9
information (i.e., ground truth spatial layout and generated interactive semantics). Then the training loss of the diffusion model equipped with CMA can be summarized as follows:
LLDM = [Ez,e~
Ea(2,, ECLP A,
Finally, we employ end-to-end joint training pipeline for the whole proposed DisCo framework. The total objective function is presented as follows:
Liotal = CLDM + AzCunion + A3Clayouts
(10)
where Å1 X2, and Ag hyperparameters, which typically 1.0, 0.1 and 1.0, respectively.
3.3 Multi-Layered Sampler
Manipulations the input scene graph, such as node addition and attribute adjustment, pose chal- lenges for maintaining visual consistency in the generated images, ultimately compromising general- izability. To achieve an "isolated" image editing effect, we also provide the Multi-Layered Sampler (MLS) motivated by SceneDiffusion [25]. The scheme defines each object layer, thus allowing independent object-level Gaussian sampling. In contrast to SceneDiffusion which scrambles the reference layouts randomly, sample additional N; (layouts, semantics) by the SL-VAE. Note that N₁ fixed seeds exist for the same scene. Then we aggregate latent codes from various layers into 2, NN and utilize layout-converted non-overlapping masks [M, = [ i=1 n== for locally conditioned diffusion. During the inference, the noise estimation for N, scenes is calculated follows:
No
Mn,i C co(2!1) ECLIY(O)). Dn,ss 8n,is ai
(11)
Subsequently, the latent code for each object is computed the weighted average of the Ni cropped denoised views. Please refer to the Appendix for more details about MLS.
Experiments
Dataset. We conduct scene- ccene-graph-ee graph-to-imsn to-imaae (SG2I) generation experiments on the Visual Genome (VG) [27] and COCO-Stufi (COCO) [26] datasets. The VG dataset comprises 108, 077 image-scene graph pairs, accompanied by the bounding box coordinates and object attributes. Following previous work [17], we select objects and relationships that appear at least 2, 000 and 500 times respectively VG, resulting 178 objects and 45 unique relationship types. Also, we ignore small objects and images containing 5 to 30 objects along with minimum of 3 relationships. Based the above

---

filtering, we have 62, 565 images available for training, each containing an average of 10 objects and 5 relationships. While the original COCO-Stuff dataset [26] lacks scene graph annotations, it consists of 40, 000 images annotated with bounding box coordinates and captions, essential for synthesizing geometric scene graphs [15, 16]. All images in the COCO-Stuff dataset are labeled as 80 item categories and 91 stuff categories.
Implementation Details. We fine-tune the pre-trained Stable-Diffusion 1.5) with the modified Attention module on 4 NVIDIA A100 GPUs, each with 80GB of memory. We apply the CLIP text encoder lar-larrrepchh-
to construct the textual scene graph. We train the model with a batch size of 64 using the Adam W optimizer [30] with an initial learning rate of 1.0 x 10 which is adjusted linearly over 50, 000 steps. During inference, we use the 50-step PNDMScheduler [21] with a classifiers- free scale [3 of 7.5. The sample number N₂ in the multi-layered sampler is set to 5.
Evaluation Metries. Following previous works [14, 15, 16], we evaluate the performance of our method with the Inception Score (IS) [32] and the Fréchet Inception Distance (FID) [33]. The IS score derived from a pre-trained Inception Net [34], assessing both the quality and diversity of synthesized images. The FID score quantifies the dissimilarity between the generated image and the real image distribution, which evaluates the fidelity of the generated images. To measure the effectiveness of compositional generation, we further evaluate our method on the T21-CompBench [35]. Besides, we apply CLIP for zero-shot attribute classification of the controlled object cropped by the bounding box. and subsequently evaluate the attribute control performance by the classification accuracy ACCattr-
Quantitative Comparisons. To demonstrate the effectiveness of the proposed DisCo, we compare it with current state-of-the-art SG2I meth- ods on the COCO-Stuff and Visual Genome datasets, which are summa- rized in Table 1. Our DisCo outper- forms other methods in both IS and FID scores, revealing its superior per- formance in both fidelity and diver- sity of image generation. Compared with previous methods, the primary architectural advantage of DisCo is its innovative approach of simultancously integrating disentangled layout and semantics extracted from scene graph representations. Moreover, the proposed SL- SAEV achieves the diverse generation of layouts and semantics from a single scene graph through Gaussian distribution sampling. Therefore, our DisCo integrates the benefits of both layout-based and semantic-based methods, which is further ablated in detail in Table 4 of the ablation study. We proceed to assess the compositional generation on the De-CComBencch [35], as shown in Table 2. The benchmark evaluates the competency of the text-to-image model in responding and 3-in-1 scores for measuring the generation of spatial/non-ssaaial relationships, attributes, and complex scenes, respectively. Following [35], we the UniDet [39], CLIP [6], and BLIP [7] to evaluate these results. Our DisCo surpasses all compared T2I methods, confirming the efficacy of scene graphs in depicting complex scenes.
Table 2: Relationship and attribute generation compared with text-to-image methods on T21-CompBench [35].
Method
UniDet
CLIP
B-VQA
3-in-1
SD-v1.4 [20]
0.1246
0.3079
0.3765
0.3080
SD-v2 [20]
0.1342
0.3127
0.5065
0.3386
Composable [36]
0.0800
0.2980
0.4063
0.2898
Structured [37]
0.1386
0.3111
0.4990
0.3355
Attn Etee-Exc|F [38]
0,1455
0.3109
0,6400
0,3401
GORS [35]
0.1815
0.3193
0.6603
0.3328
DisCo (ours)
0.2376
0.3217
0.6959
0.4143
compositional prompts. We report UniDet, CLIP, B- B-VQA,
Table 3: User study. The score quantifies the user evaluation (i.e., relationships, quantities, and generation quality) of the alignment between the given prompt and the generated image.
SD-XL
DALL-E3
Imagen 2
GLIGEN
L.D
MIGC
SG2Im
SGDiff R3CD DisCo
Method
[40]
[3]
[41]
[12]
[10]
[13]
[17]
[16]
[14]
(ours)
Score
0.6684
0.5944
0.5637
0.6549
0.6200
0.7055 0,3783
0.4717 0.6928
0.8533
User Study. We conduct user study by recruiting 50 participants from Amazon Mechanical Turk. We randomly select 8 prompts for each method, resulting in 80 generated images. We ask participants alignment. The worker can choose a score from {1, 2.3, 4, 5) and we normalize the scores by dividing them by 5. We then compute the average score across all images and all workers. The results are presented in the Table 3. Our method is favored by most participants in terms of generation rationality and controllability.
to score each generated image independently based on the image-pronen
intcps://hugginggacc httpr//www////aalleeeae diffusion v1-5
7

---

sheep another sheep the with the ocean Ca under the shy; the ocean brea: boat on the grass
Two men standing on beach playing kite
A building with window on side of to bus has bine
A dog Cchasingj cat on the grass
bop the pasture
building with window side of conaana
SD-XL
DALL-E3
Imagen
Ours
GLIGEN
LayoutDifftsion
MIGC
Quss
(a) Comparison with T2I methods
(b) Comparison with 1.21 methods in non-spatial interactions and rationality.
spatial relationships and object quantities.
building
llamp stone:
s sky
ground
tree
shadow?
has
has
has
side of road
wall
window
door
sky grass
wood reflection
dirt ishadow
road
cars
building
street
near
behind
top of
in front of
behind
with
has
bench
tree
window
door
SG21m
SCDiff
R3CD
Our
SG2Im
SGDiff
R3CD
Ous
(c) Comparison with SG21 methods in independent node inference and generation quality.
Qualitative Comparisons with (a) sot-to-image (T2I) (SableDiffusion-XX XL [40], DALL-E 3 [3], and Imagen 2 [41]). (b) layout-to-inas im-ae (L21) (GLIGEN [12], LayouDiffusiio [10], and MIGC [13]), and (c) scene-graph-tee grraph-to-iaage (SG21) (SG2lm [17]. SGDiff [16], and R3CD [14]) methods.
Figure 5:
Table 4: Ablation study for overall architecture. Table 5: Ablation study for attention mechanism.
SL-VAE w/o D, means independent of O.
Vanilla attention means off-the-shelf T2I attention.
Method
G2I-ACC 12G-ACC
Attention Type
t
FID 1
Layout (D))
70.3
70.5
Vanilla attention
29.1
Semantics (D.)
71.1
71.5
CMA (w/o mask M)
17.9
28.4
VAE (w/o Dz)
72.9
72.8
CMA (union MLP)
19.8
22.0
VAE (D; D.)
73.9
74.3
CMA (separate MLP)
22.3
21.9
Qualitative Comparisons. Figure 5 visualizes the results of the methods conditioned by text, layout, scene graph, showcasing our advantages generating rationality and controllability: (i) Comparison with the text-to-image (T2I) methods. In Figure 5 (a), we present the superiority of

---

the disentangled structured scene graph over linear text for repre- senting complex scenes. Firstly, we resolve ambiguity in textual relationships and semantics by employing layout and semantic disentanglement within the scene graph. For example, our DisCo clarifies the relationship between "boat" and "grass" in the first line, as well the semantics of "bus" and "building" in the following line. Additionally, the samples in the first and last lines also showcase our capacity generate the specified quantities of objects precisely. (ii) Comparison with the layout-to-imass (1.21) methods. We also demonstrate that DisCo outperforms the diffusion methods relying on manually crafted scene layout representations, as illustrated in Figure (b). While the L2I method struggles to model non-spatial interactions (such as "playing" in the first line and "chasing" in the second line), our DisCo addresses this challenge using disentangled object interactive semantics. Furthermore, by establishing semantics among objects derived from their relationships, we prevent independent gen- eration instances that rely solely on the layout, exemplified by the "bus" and "tire" in the last line. gowpg--taaimaaaee
(a) unmodified
(b) w/ NA
(c) w/ AC (blue)
(d) w/ AC (red)
Figure 6: Illustration of object-level Node Addition (NA) and
Attribute Control (AC) in the scene. From left to right: (a) the image generated by the unmodified scene graph;
(b) the chair addition; (c) the blue-colored wall; and (d) the red-colored wall.
(SG21) methods. The SG2I visualization results of different methods are showcased in Figure 5 (c). Our DisCo significantly improves the quality of SG2I generation, particularly for independent nodes. The proposed layout and semantics disentan- glement technique effectively capture both the spatial and interactive information of independent nodes. Taking the "lamp" and "stone" in the first image and the "shadow" in the second image as illustrations, these entities are neglected by previous methods, whereas our DisCo not only retains their semantic relevance but also infers their appropriate spatial placement within the scene. We also demonstrate the generalizable generation under consistency for graph manipulation (i.e., node addition and attribute control) in SG2I tasks, as shown in Figure 6.
(iii) Comparison with the
Ablation Study. Table 4 explores the overall architecture by evaluating the alignment of the generated image with the objects and relationships depicted in the input scene graph. Following SGDiff [16], we conduct this analysis by graph-to-iman
Table 6: Ablation study for Multi-Layer Sampler (MLS).
50
Method
1S 1
FID
Baseline (w/o ML.S)
20.5
23.0
ACC.i
w/ LSD [25]
21.1
22.7
(G2I) and image- to-graph (12G) retrieval experiments.
w/ MLS (Ours)
22.3
21.9
45-
Note that "w/o D." " means processing each node embedding by MLP independently.instead of obtaining object interactive semantics through D.. We observe that the spatial layout and interactive semantics collaborate to boost both retrieval tasks. These results demonstrate the effectiivneess of integrating explicit spatial relations with implicit interactive semantics. In Table 5, we study the impact of different attention mechanisms. We inject graph conditions using different mechanisms: (a) Vanilla attention mechanism in the T2I diffusion model without our CMA; (b) CMA without attention mask M: (c) CMA with union MLP after concatenating object and attribute embeddings; and (d) CMA with two separate MLPs before concatenating object and attribute embeddings. We found that CMA, which fuses separate encoding of object and attribute embeddings, significantly enhances the overall generation performance. Table 6 presents the Multi-Layer Sampler (MLS) ablation results, confirming its enhancement over the baseline and LSD [29]. In contrast to LSD, which randomly scrambles layouts, the proposed MLS naturally leverages variety of coherent layouts and semantics produced by SL-VAE. Moreover, the increase in ACC scores also indicates that MLS facilitates controllability, especially in attribute control, while ensuring generation quality.
Related Works
Diffusion Models. Diffusion models (DMs) [2, 20, 31, have achieved great success in high-quality image generation. The essence of DMs lies in estimating image distributions by iterative denoising noise-corrupted image, showcasing the superiority over VAEs [43, 44) and GANs [45] in training stability and likelihood estimation. To further explore the controllability of DMs, considerable efforts

---

have been devoted to conditional generation based on DMs. Benefiting from the naturalness of language [6] and the advancements of Vision-language foundation models [6, 7, 9], numerous text- to-image DMs [40, 41, 46] are beginning to emerge, facilitating explicit control of the corresponding semanties and style. However, the expressive capacity of linear text is limited. Therefore, many studies also endeavor to bolster global control through supplementary conditions, such as depth [46, 47],
layout [10, 11, 12, 13), segmentation map [46, 48], and scene graph [14, 15].
Image Generation from Scene Graphs. Scene graphs are structured scene representations, where nodes represent objects and edges represent relationships between objects [17. 27]. Given the superiority of scene graphs over linear text in delineating multiple objects and their intricate relation- ships [27.49, 49. 50], many studies investigate image generation from scene graphs. These approaches typically fall into two categories: layout- layout-based and semantics- baauun methods. Layout- sayout-based meth- ods [17, 18, 19, 28, 36, 51J initially map scene graphs to coarse scene layouts comprising multiple bounding boxes and further refine these layouts to images with a layout-to-image model (e.g., Layout- Diffusion [10]. While the layout depicts spatial relationships, fails to capture abstract relationships within the scene, leading to lack of object interaction. Another branch is semantics- based methods [14, 16, 29, 52, 53], which focus on graph understanding by directly encoding semantic information from the scene graph. Nevertheless, these methods have limitations in addressing independent scene nodes, leading to issues like entity loss and unreasonable placement. In this paper, we propose a compositional image generation that leverages the layout and semantics derived from the scene graph representation. We complement explicit layout and implicit semantics to enhance the understanding of the diffusion model for scene graphs. Additionally, to improve the controllability in the scene- graph-to-image imag task, we also attain generalizable generation for object-level graph manipulation (i.e., node addition and attribute control).
6 Limitations
The proposed CMA injects object-level information into the diffusion model via masks from the layout, effectively miti- gating semantic ambiguity and limiting attribute leakage. In scenarios involving object overlap. the proposed CMA inhibits direct interaction between the visual token and the object em- bedding along with its attributes. Nonetheless, the attribute information from the visual token inadvertently leaks into the overlapping region in subsequent layers. Hence, there may be attribute leakage among the objects, as shown in Figure 7.
Figure 7: Qualitative limitations on attribute leakage of overlapping.
7 Conclusion
In this study, we leverage the disentangled textual scene graph representation to condition the diffusion process for generating complex scene images. The innovation of our framework lies in utilizing the VAE for scene relationship modeling and diffusion model (DM) for the composite visual generation. To comprehensively capture spatial relationships and non-spatial interactions within scenes, we introduce the Homantics-Layou Variational AutoEncoder (SL-VAE) for deriving diverse layouts and semantics from single scene graph. Building upon them, we propose the Compositional Masked Attention (CMA) integrated with DM, which guides the de-noising trajectory by compositing extracted object-level graph information with fine-grained attributes. We also introduce Multi-Layer Sampler (MLS) to preserve the main visual content while modifying the input scene graph. Extensive experiments demonstrate that our framework outperforms current methods conditioned by text, layout, or scene graph in relationship modeling and controllability.
Acknowledgments
This research supported by the National Natural Science Foundation of China [Grant 62302246] and the Zhejiang Provincial Natural Science Foundation of China [Grant L.Q23F010088.. We also express our sincere gratitude to the Al Computing Center the Eastern Institute of Technology for their valuable support and assistance.
10

---

References
[1] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems (Neur/PS), 33:6840-6851, 2020. 1,
[2] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems (NeurlPS), 878880887994 2021. 1,9
[3] James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, et al. Improving image generation with better captions. Computer Science. https://edn. openai. con/papers/dallee-. pdf, 2:8, 2023. 2, 8
[4] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint 2r 2021.
[5] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06/23. 2022.
[6] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In Proceedings of the International Conference on
Machine Learning (ICML), pages 8748-87088 8763, 2021. 2, 3, 4, 7, 10
[7] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In Proceedings of the International Conference on Machine Learning (ICML), pages 12888-12900, 2022. 1, 7, 10
[8] Jumnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: 2: Bootstrapping Language-image pre- pre-training with frozen image encoders and large language models. In Proceedings of the International Conference on Machine Learning (ICML), pages 19730-19742, 2023. 1, 10
[9] Kecheng Zheng, Yifei Zhang, Wei Wu, Fan Lu, Shuailei Ma, Xin Jin, Wei Chen, and Yujun Shen. Dreamlip: Language-image pre-training with long captions. arXiv preprint arXiv:2403. 17007, 2024. 1, 10
[10] Guangcong Zheng, Xianpan Zhou, Xuewei Li, Zhongang Qi, Ying Shan, and Xi Li. Lay- outdiffusion: Controllable diffusion model for layout-to-image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 22490-22499, 2023. 1, 2, 7, 7,8,17 10
[11] Jiaxin Cheng, Xiao Liang, Xingjian Shi, Tong He, Tianjun Xiao, and Mu Li. Layoutdif- fuse: Adapting foundational diffusion models for layout-to-image generation. arXiv preprint arXiv:2302. 089008 2023. 1, 10
[12] Yuheng Li, Haotian Liu, Qingyang Wu, Fangzhou Mu, Jianwei Yang. Jianfeng Gao, Chunyuan Li, and Yong Jae Lee. Gligen: Open-set grounded text-to-image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages
22511 1 125211 2023. 1, 7, 8, 10
[13] Dewei Zhou, You Li, Fan Ma, Zongxin Yang, and Yi Yang. Mige: Multi-instance generation controller for text-to-image synthesis. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024. 1, 7, 8, 10
[14] Jinxiu Liu and Qi Liu. R3ed: Scene graph to image generation with relation-aware compositional contrastive control diffusion. In Proceedings of the AAAI Conference on Artificial Intelligence
(AAAI), volume 38, pages 3657-3665, 2024. 2, 6, 7, 8, 10
[15] Azade Farshad, Yousef Yeganeh, Yu Chi, Chengzhi Shen, Böjrn Ommer, and Nassir Navab. Scenegenie: Scene graph guided diffusion models for image synthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 88-98, 2023. 1, 2, 6, 7, 10
[16] Ling Yang. Zhilin Huang, Yang Song, Shenda Hong, Guohao Li, Wentao Zhang. Bin Cui, Bernard Ghanem, and Ming Hing- Yang. Diffusion- scene graph to image generation with masked contrastive pre- -raiirinn arXiv preprint arXiv:2211.111.38 2022. 1, 7, 8, 9, 10
[17] Justin Johnson, Agrim Gupta, and Li Fei-Fei. Image generation from scene graphs. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1228, 2018. 1, 2, 4, 6, 7,8, 10
11

---

[18] Oron Ashual and Lior Wolf. Specifying object attributes and relations in interactive scene generation. In Proceedings of the IEEE/CVE International Conference on Computer Vision (ICCV), pages 4561-4569, 2019. 2, 4, 6, 10
[19] Roei Herzig, Amir Bar, Huijuan Xu, Gal Chechik, Trevor Darrell, and Amir Globerson. Learning canonical representations for scene graph to image generation. In Proceedings of the European Conference Computer Vision (ECCV), pages 210-227. Springer, 2020. 2, 6, 10
[20] Robin Rombach, Andreas Blatmaann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages
2022.
3, 6, 7, 14, 16
[21] Luping Liu, Yi Ren, Zhijie Lin, and Zhou Zhao. Pseudo numerical methods for diffusion models on manifolds. arXiv preprint arXiv:2202.09777. 2022. 3,7
[22] Andrew Luo, Zhoutong Zhang. Jiajun Wu, and Joshua Tenenbaum. End-to-ent optimization of scene layout. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pages 3754-3763, 2020. 4
[23] Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich-Keil Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. Advances in Neural Information Processing Systems (NeurIPS), 33:7537-7547, 2020. 5
[24] Andrey Voynov, Qinghao Chu, Daniel Cohen-Or, and Kfir Aberman. p+: Extended textual conditioning in text-to-image generation. arXiv preprint 23X 2023. 5
[25] Jiawei Ren, Mengmeng Xu, Jui-Chieh Wu, Ziwei Liu, Tao Xiang, and Antoine Toisoul. Move anything with layered scene diffusion. arXiv preprint arXiv:2404.077 07178, 2024. 6, 9, 14, 15
[26] Holger Caesar, Jasper Uijlings, and Vittorio Ferrari. Coco-stuff: Thing and stuff classes in context. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1209-1218, 8, 2018. 6,7
[27] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International Journal of Computer Vision (ICCV), 123:32-73, 2017. 6, 10
[28] Yikang Li, Tao Ma, Yeqi Bai, Nan Duan, Sining Wei, and Xiaogang Wang. Pastegan: A semi-parametric method to generate image from scene graph. Advances in Neural Information Processing Systems (NeurIPS), 32, 2019. 6, 10
[29] Yang Wu, Pengxu Wei, and Liang Lin. Scene graph to image synthesis via knowledge consensus. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), volume 37, pages 2856 08662286,,222 2023. 6, 9, 10
[30] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint
arXiv: 1. 2.600 2014.7 7
[31] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint
arXiv:2207. / /298, 2022. 7,9
[32] Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. Advances in Neural Information Processing Systems (NeurIPS), 29, 2016. 7
[33] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge a local nash equilibrium. Advances in Neural Information Processing Systems (NeurIPS), 30, 2017. 7
[34] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethink- ing the inception architecture for computer vision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1818-2826. 2016. 7
[35] Kaiyi Huang, Kaiyue Sun, Enze Xie, Zhenguo Li, and Xihui Liu. T2i-comphench: A compre- hensive benchmark for open-world compositional text-to-image generation. Advances in Neural Information Processing Systems (NeurIPS), 36:78733-788877 2023. 7
12

---

[36] Nan Liu,, Shuang Li, Yilun Du, Antonio Torralba, and Joshua B Tenenbaum. Compositional visual generation with composable diffusion models. In Proceedings of the European Conference
on Computer Vision (ECCV), pages 143.439. Springer, 2022. 7, 10
[37] Weixi Feng, Xuchai He, Tsu-Jui Fu, Varun Jampani, Arjun Akula, Pradyumna Narayana, Sugato Basu, Xin Eric Wang, and William Yang Wang. Training free structured diffusion guidance for compositional text-to-image synthesis. arXiv preprint arXiv:22 050022, 2022. 7
[38] Hila Chefer, Yuval Alaluf, Yael Vinker, Lior Wolf, and Daniel Cohen- Or. Attend-and exxeite Attention- based semantic guidance for text-to-image diffusion models. ACM Transactions on Graphics (TOG), 42(4):1-10, 2023. 7
[39] Xingyi Zhou, Vladlen Koltun, and Philipp Krähenbühl. Simple multi-dataset detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR),
pages 7571-75881 2022. 7
[40] Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952, 2023. 7, 10
[41] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Information Processing Systems (NeurIPS), 35:36479- -3494,, 2022. 7, 8, 10
[42] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-hased generative modeling through stochastic differential equations. arXiv preprint arXiv:201 13466, 2020. 9
[43] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv: /3/2.61 14, 2013. 9
[44] Kihyuk Sohn, Honglak Lee, and Xinchen Yan. Learning structured output representation using deep conditional generative models. Advances in Neural Information Processing Systems (NeurIPS), 28, 2015. 9
[45] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, 63(11):139-14)
2020.
[46] Lvmin Zhang, Anyi Rao. and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEF/CVF International Conference on Computer
Vision (ICCV), pages 3836-3847, 2023. 10
[47] Lezhong Wang, Jeppe Revall Frisvad, Mark Bo Jensen, and Siavash Arjomand Bigdeli. Stere- odiffusion: Training-free stereo image generation using latent diffusion models. arXiv preprint arXiv:2403.04965. 2024. 10
[48] Ziqi Huang, Kelvin CK Chan, Yuming Jiang, and Ziwei Liu. Collaborative diffusion for multi- modal face generation and editing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pages 6080-6090, 2023. 10
[49] Justin Johnson, Ranjay Krishna, Michael Stark, Li-Jia Li. David Shamma, Michael Bernstein, and Li Fei-Fei. Image retrieval using scene graphs. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3668-3678, 2015. 10
[50] Subarna Tripathi, Anahita Bhiwandiwalla. Alexei Bastidas, and Hanlin Tang. Using scene graph context to improve image generation. arXiv preprint arXiv:1901.03762 2019. 10
[51] Yilun Du, Conor Durkan, Robin Strudel, Joshua B Tenenbaum, Sander Dieleman, Rob Fergus, Arnud Doucet, and Will Sussman Grathwohl. Reduce, reuse, recycle: Compositional generation with based diffusion models and meme. In Proceedings of the International Conference on Machine Learning (ICML), pages 8489-8510, 2023. 10
Jascha
[52] Ruijun Li, Weihua Li, Yi Yang, Hanyu Wei, Jianhua Jiang, and Quan Bai. Swiny2-imagen: Hi- erarchical vision transformer diffusion models for text-to-image -moggaagee generation. Neural Computing and Applications, pages 1-16. 2023. 10
[53] Ruichen Wang. Zekang Chen, Chen Chen, Jian Ma, Haonan Lu, and Xiaodong Lin. Composi- tional text-to-image synthesis with attention map control of diffusion models. Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), volume 38, pages 5544-5552, 2024. 10
13

---

A Appendix
This appendix is organized follows: Section A.I provides more details about Semantics-Layout Variation AutoEncoder; Section A.2 introduces the Stable Diffusion and its attention mechanism; Section A.3 describes the implementation of the Multi-Layer Sampler in detail; Section A.4 covers more ablation studies; Section A.5 presents more qualitative results, including comparison visualiza- tion and graph manipulation; Section A.6 delves into the broader societal impacts of this work. The core script is zipped and attached to the supplementary material.
A.1 Semantics- Layouu Variation AutoEncoder
Recall that we apply the triplet-GCN based CVAE architecture in Section 3.1. Each triplet-GCN layer in the encoder and decoder takes the node and edge embeddings. Specifically, the GCN, mentioned in the paper uses two cascading MLPs {mlp1, mlp2) } to deal with node and edge embeddings:
1++1 (w))
by ( mlp, (o). dy o)), E 10...
(12)
0 [+1 v mlpy 1 E (N(o))))
(13)
where denotes the layer index of encoder or decoder, .eg denotes the neighbor index set for each node, avg denotes the average pooling operation, 6 and & denote intermediate features. Hence, mlp; conducts message passing among interconnected nodes and updates the edge features, while mlp aggregates features from all neighboring nodes and updates its features. For graph union encoder, we let (0% S..- (.))))
(O;, O,). The last embedding of is parameterized to the Gaussian distribution ~ (N,,o) σ), where μe, a E R:D: output by two additional MLPs and D2 denotes the dimensional of latent space for node embedding.
A.2 Diffusion with Compositional Masked Attention
Stable Diffusion [20] is one of most popular text-to-image model. As described in Section 2.1, Stable Diffusion uses a U-Net Eg composed of convolution and transformer to estimate noise. The transformer includes two attention mechanisms, namely Cross-Attention and Self-Attention.
Cross- Attentonnn Layer. Text prompts mapped to sequence embeddings by CLIP text encoder and integrated into UNet via Cross-Attention to guide the de-noising trajectory:
Attention(Quisuua Ktext, Vtext) = softmax Quisnal KT tent) Vtext
(14)
Vd
where Quisuat denotes the Query from the visual token of the UNet, Ktezt and Vieut denotes Key and Value from text embeddings, all of which are projected by linear layers, denotes the dimension of Quisuats Krewt, and Vient.
Sell-Attention Layer Self-Attentio captures self-related information within visual tokens:
Attention (Qvisual, Kassual, Veiswat) = softm Quienal KT esssual
Veisual
(15)
Vd
where Quisual, Krisual, and Voisual separately represent the Query, Key, and Value in self-attention layers, which are projected by linear layers. The self-attention mechanism isolates the information flow between specific tokens by multiplying mask M the Quisual KT Visuaat- Since M is applied before sof tmax, the value of the isolated position is set to negative infinity -inf.
Compositional Masked Attention Layer. Based on the attention mask M that depends layout B, the Compositional Masked Attention can be expressed as:
Attention(QCMAr KCMA, VCMA) = so softmaz QcMAKTMA M
VeMA
(16)
Vd
where QoMA, KCMA, and VeMA individually represent the Query, Key, and Value derived from V & Ć, achieved through linear layer projections. We insert our proposed Compositional Masked Attention (CMA) between self-attention and cross-attention layers.
A.3 Multi-Layer Sampler
Layered Scene Representation. We decompose a controllable scene containing N, objects into N, layers. Different from SceneDiffusion [25], our approach involves each layer incorporating not only
14

---

separate latent code and spatial layout b;, but also integrating the interactive semantics produced by the VAE. Here we convert the layout parameter b; two parts: (1) a fixed object-centric binary mask m, E {0, 10 jexwxh to solely show the geometric property of the object, and (2) a two-element offset p: = {4i, 24 to solely indicate its spatial locations, with μi; and vs defining the horizontal and vertical movement range. We sample Gaussian noise individually for the initial latent code of each
layer, i.e., Z {2 (
N((0, 1) Then we utilize the layout-converted non-overlapping masks
{4) ( M
to derive the aggregated latent code z from various layers:
4 4 shift(z,
.p.s) .p))
(17)
N
4 = shift(m,, Pa)
II (1 - (1-shfft(m,,p))) J=1
(18)
where denotes element-wise multiplication, and shi,t(x.,)) denotes spatially shifting the values of x in the direction of p.
Multi-Layer Generation. We introduce the Multi-Layer Sampler that matches our diverse layout and semantic simulation. In contrast to Scene [25] which scrambles the reference layouts randomly, sample additional N₁ layouts and semantics by the proposed SL-VAE. On the one hand, the SL-VAE ensures that the generated scene layout is reasonable. On the other hand, we take full advantage of the paired object-level (layouts, semantics). Specifically, the denoising scheme consists of four steps:
(a) Sampling additional Ni layouts (B,,
N. i= J- and semantics {S,, = {sn. | N by the proposed SL-VAR Note that N₁ fixed seeds exist for the same scene graph. According to the description of the layered representation, we convert the layout get offset (P,, {pn,)) i=1 1n
(bn,,)
i=1
N;
(b) Aggregating latent codes from various layers in each scene:
No
(0)
4. shift(zz .P.))
(19)
(c) Estimating the noise el from each aggregated latent code 20 and gets denoised aggregated (2(t-1) s(t-1)),
latent code s!{-1) {2(
No
(
Σ Mass @ (o(21)) Focar(O:), bn,s, Sn,i, ai, 1);
(20)
where mn, is the non-overlapping mask converted by the layout b
(d) Updating the latent code of each layer by computing the weighted average of the Nj aggregated latent code
Nr 1 shift(l @ 1(111) -Pn.))
(21)
ΣNI shift(l,, Pn.s)
where shift(x,- -p) denotes spatially shifting the values of æ in the reverse direction of p.
A.4 More Ablation Studies
Graph Construction. We conduct ablation for graph construction in Table 7. We investigate the impact of different graph components (i.e., CLIP, Box, and Learnable Embeddings) by turning off each independently. We observe that each compo- nent improves the performance, all of which are crucial components presented in our DisCo.
Table 7: Ablation study for graph construction.
Graph Type
IS 1
FID
No CLIP Emb.
20.6
23.9
No Box Emb.
21.7
22.5
No Learnable Emb.
21.9
22.2
Computing Consumption. We demonstrate the impact of our proposed CMA on the computational complexity of the U-Net within the Stable Diffusion, presented in Table 8, We use Floating Point
15

---

Table 8: Ablation study for computing consumption.
Method
FLOPs (G)
Params (M)
Time (ms)
SD-v1.5 [20]
677.5
859.4
37.9
DisCo
724.1
875.8
108.3
Operations (FLOPs), the number of parameters (Params), and inference time (Time) to measure computing consumption. The FLOPS and Time metries are conducted by processing the tensor with resolution of 2 x 4 x 64 x 64 on an NVIDIA A100 GPU. Our proposed DisCo significantly improves the controllability of the Stable Diffusion with tolerable increase in computational cost.
A.5
More Visualization Results
Figure showcases more generalizable generation results under consistency for graph manipulation (i.e., node addition and attribute control) in SG2I task. In Figure 9, 10, and 11, present more visualization comparisons with the methods conditioned by text, layout, or scene graph, which demonstrates the superiority of DisCo terms of generation rationality and controllability.
A.6 Broader Impacts
We demonstrate the superiority of our DisCo over existing generation methods based on text, layout, and scene graphs, suggesting potential beneficial influence the realms of creation and data synthesis. Nevertheless, there remains a concern regarding the possibility of generating malicious images infringing copyright.
16

---

Graph Manipulation (Node Addition and Attilbute Control)
Landar Bky abore
Byy SlSoo
Bksy
boat
boot
boot
Shova
D gress
wilh
malli
calling
Belling
cedligg
sebro beloww
Iii Iron,
below
frout
below
house
Shove
top
top
grass
BrOSS
gross
on
Figure 8: Generalizable Generation Samples under Consistency Graph Manipulation.
17

---

building in front of mountain; freeiin the building
saadinn on the grass; two near the bree; gone front the housel
cow on top of grass; one near the tree; sky above the grass; building near the trea
The street by the building; the car next to the bruck!
teddy bear sit on the ground another Ewo Eeddy bear basides the table
SD-XL
DALL E
Imagen 2
Ours
Figure 9: Qualitative Comparison with Text- methods.
18

---

man twakking the sidewalk
A man Mankooiin abithe cloud in sky
booligiigg apple on the grass
A boy triding dog on the grass
child with the hairton the grass
GLIGEN
LayoutDiffusion
MIGC
Ours
Figure 10: Qualitative Comparison with Layout-to-Image methods.
19

---

building with
light
Line
window side of
road
bus
pole
building
near
street
dog
chase
cat
grass
Scene Graph
SGDiff
R3CD
Ours
Sky
surlboard
Watie
Back
head
inflection
bolcing
Bogi
patuccie mal
Scene Graph Figure 11: Qualitative Comparison with Scene- Scene-Graph-to-o Graph-to-Image methods.
SG2Im
SceneGenie
Ours
20

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We claim in the abstract and introduction that this paper studies generating complex images with structured scene graphs.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The limitations are addressed in the Appendix, which will be included in the camera-ready version.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
21

---

Answer: [Yes]
Justification: All the theorems and formulas are clearly stated in Section and 3 of the main paper, and complemented in Section A. A.2 and A.3 of the Supplemental Material. Besides, all theorems and assumptions related to this work are properly referenced.
Guidelines:
The answer NA means that the paper does not include theoretical results, All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide the implementation details for reproducibility in Section 4, and the project will be open-soureed.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
22

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Question:
Answer: [Yes]
Justification: We attach the core script and data preparation to the Supplemental Material. The complete code will be released in the camera-ready version, accompanied by detailed instructions for reproducibility.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nipp httpss///iis...o cc/
oonylccyl)) for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to rum to reproduce the results. See the NeurIPS code and data submission guidelines (https: //n //ipps.coo catss.ccpppoo cccpublic//./ccccceccccccccc...e wttp//gwieeeeeeeeeeeee http://ooeSSuuiiiseeee wwwsssoosllioooooooo for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We provide the training and test details in Section 4.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification: We report error bars suitably and correctly to achieve statistical significance in the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
23

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide sufficient information on the computer resources in Section 4.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: /ttee///neeuueeeeee /terrps.....p// cc ccc/publcc/Baeeeeeeeeeeeeeeeee/e wwwideiiseeeeeeneeennennnnee
Answer: [Yes]
Justification: Our work conforms with the NeurIPS Code of Ethics in every respect.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- cration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We discuss both potential positive and negative socictal impacts in Section A.6 of the Appendix.
Guidelines:
The answer NA means that there is no societal impact of the work performed.

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: Our model is heavily based on existing StableDiffusion, so our safeguards are the same theirs.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: In Section 4, we properly credit all the public baselines and datasets utilized in this paper.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
25

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
If assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
The assets introduced in the paper are well documented.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
This paper does involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
26

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
27


Reviews:
[{"review_id": "UScqubBWfc", "reviewer": "Reviewer_3rPJ", "summary": "This paper proposes a method that uses a scene graph and integrates variational autoencoders (VAEs) and diffusion models to address complex scene generation. Specifically, a Semantics-Layout Variational AutoEncoder (SL-VAE) is used to derive diverse layouts and semantics from the scene graph, while a Compositional Masked Attention (CMA) combined with a diffusion model incorporates these elements as guidance. Additionally, a Multi-Layered Sampler (MLS) is introduced for isolated image editing. Experiments show that this approach outperforms recent methods in generation rationality and controllability.", "questions": "Please see the weakness", "limitations": "Please see the weakness", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. This paper considers an important issue in text-to-image generation realm.\n2. The structure design in Section 3 makes sense.\n3. The experimental results shown in Table 1 and 2 show the effectiveness of this method.", "weaknesses": "1. My main concern is the practical application of this method. As we all known, scene graph building is not a trivial task, but you don't explain detail in the paper how to construct an exact scene graph. In addition, during inference, the prompt proposed by uses may be non-standard so that building a scene graph may be more difficult. \n2. Besides, recent SOTA models, e.g. DALLE3, stable diffusion 3 try to solve the complex generation task by large-scale fine-grained dataset construction. How do you compare your methods with these data-centric methods. The authors should spend more space discussing the issues.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate all of your valuable suggestions, which play a pivotal role in enhancing the quality of our paper. Below we address all your concerns.\n\n**Q1: Details of scene graph construction.**\n\n**R1**: Thank you for the questions raised by the reviewer, and we will add the details of scene graph construction from text in the revision. Actually, our DisCo focuses on scene-graph-to-image (SG2I) task, which typically assumes that the scene graph is ready [14, 17]. However, to make our method more practical and allow comparison with the popular text-to-image (T2I) methods, we also provide a text-to-scene graph conversion method (it is not the focus of our paper, so we didn't add it in the original manuscript), which is feasible and convenient with the assistance of the recent powerful LLM such as GPT-4o or Llama.\n\nSpecifically, as mentioned in Section 2.2, we use LLM to extract **instances (subject and object)** and **relationships (predicates)** from the prompt. Given a text describing the scene, the LLM outputs a structured format that contains **a node list** and **a triple list**. To standardize the output format of the LLM and facilitate inferring scene graph from non-standard prompt, we here provide a simplified template of our instructions and in-context examples for scene graph building:\n\n```1. Task instruction``` **```(pre-provided)```**\n\n    You are a master of composition who excels at extracting key objects and their relationships from input text. You should reorganize the input text as a scene graph format, which consist of a node list representing the objects and a triple list representing the relationships between the objects.\n\n```2. Scene graph construction tutorial``` **```(pre-provided)```**\n\n    There are a few rules to follow:\n        # Output format specification \n        # Numbering rules of the same category\n        # Extract relationships from non-standard text based on context\n\n```3. In-context examples``` **```(pre-provided)```**\n\n    **Prompt Example**:\n        Two men standing on the beach, both of them playing a kite.\n    \n    **Output Example**:\n        Objects: [man1, man2, beach, kite]\n        Relationships: [(man1, standing on, beach), (man2, standing on, beach), (man1, playing, kite), (man2, playing, kite)]\n \n    **More examples using non-standard prompt**\n\n```4. Trigger reasoning ability of LLM``` **```(user-provided)```**\n\n    **User Prompt**\n        A sheep by another sheep on the grass with the ocean under the sky; the ocean by a tree; a boat on the grass\n        \n    **Output**:\n        Objects: [sheep1, sheep2, grass, ocean, sky, tree, boat]\n        Relationships: [(sheep1, by, sheep2), (sheep1, on, grass), (sheep2, on, grass), (grass, with, ocean), (ocean, under, sky), (ocean, by, tree), (boat, on, grass)]\n\nNote that such scene graph built by LLM only applies to the inference phase for practicality. For training, we use specialized scene graph-to-image datasets (Visual Genome [27] and COCO-Stuff [26]) where the required information like box labeling is all well covered.\n\n**Q2: Comparison with recent SOTA data-centric text-to-image (T2I) methods.**\n\n**R2**: Actually, we have discussed and compared with these methods in Figures 1 (a) and 6 (a) of the manuscript. These data-centric text-to-image (T2I) methods like DALLE$\\cdot$3 typically benefit from large-scale text-image pairs that can be automatically collected in high quality. However, even with large-scale fine-grained text annotation, they still suffer from deficiencies in aspects such as **quantity generation** and **relationship binding** due to the **linear structure** of the text. \n\nIn summary, we point out the advantages of our **structured scene graphs** over these data-centric text-only methods in representing complex scenes from the following aspects: \n\n- Based on scene graph data, which is more compact and efficient than text, our DisCo demonstrates advantages in generation **rationality** and **controllability**, particularly for quantity generation and relationship binding. The qualitative  comparisons with the T2I models are shown in Figures 1 (a) and 6 (a) of the manuscript.\n\n- We also conduct evaluation with these T2I methods on T2I-CompBench [35], which quantifies our improvement in **spatial/non-spatial relationships**, **attributes**, and **complex scenes**. The results are shown in the table below (**bold** for 1st, *italic* for 2nd).\n\n    | **Method**|**UniDet**|**CLIP**|**B-VQA**|**3-in-1**|\n    |--------------|:----:|:----:|:----:|:----:|\n    | SD-v1.4|0.1246|0.3079|0.3765|0.3080|\n    | SD-v2| 0.1342 |0.3127|0.5065|0.3386|\n    | SD-XL| 0.2032 |0.3110|0.6369|0.4091|\n    | Pixart-$\\alpha$|0.2082|*0.3179*|0.6886|*0.4117*|\n    | DALL$\\cdot$E 2|0.1283|0.3043|0.5750|0.3696|\n    | DALL$\\cdot$E 3|*0.2265*|0.3003|**0.7785**|0.3773|\n    | **DisCo (ours)**|**0.2376**|**0.3217** |*0.6959*|**0.4143**|\n\n- Besides, we conduct a user study by recruiting 50 participants from Amazon Mechanical Turk. We randomly select 8 prompts for each method, and ask participants to score each generated image independently based on the **image-prompt alignment**. The worker can choose a score from {1, 2, 3, 4, 5} and we normalize the scores by dividing them by 5. We then compute the average score across all images and all workers. The results of user study are presented in the table below, and we can see our DisCo achieves the best alignment score.\n    ||||||\n    |:----:|:----:|:----:|:----:|:----:|\n    |**Method**|SD-XL|DALL$\\cdot$E 3|Imagen 2|**DisCo (ours)**|\n    |**Alignment Score**|0.6684|0.5944|0.5637|**0.8533**|\n\n- Moreover, the SL-VAE designed in our DisCo disentangles diverse conditions from the scene graph, which allows multi-round “**separate object-level manipulation while keeping the other content unchanged**” editing effect as a by-product that is more practical than the general T2I models. The generalizable generation results are illustrated in Figures 5 and 8 of the manuscript.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "o6rsgqdca4", "reviewer": "Reviewer_anRy", "summary": "This paper presents \"DisCo,\" a novel framework for generating complex images from structured scene graphs. Unlike traditional text-to-image or layout-to-image methods, DisCo utilizes a Semantics-Layout Variational AutoEncoder (SL-VAE) to disentangle and generate diverse spatial layouts and interactive semantics from scene graphs. It incorporates these elements using a Compositional Masked Attention (CMA) mechanism within a diffusion model, enhancing generation control and rationality. The framework also introduces a Multi-Layered Sampler (MLS) for flexible, graph-based image editing, preserving visual consistency while manipulating object attributes and positions.", "questions": "1. The paper presents results on standard benchmarks. However, can you provide insights or preliminary results on how the model performs across datasets with higher variability in object complexity and scene density? \n2. Why were certain closely related baselines omitted from quantitative comparisons? Could inclusion of these baselines provide a more comprehensive evaluation?", "limitations": "The authors address the challenge of attribute leakage in overlapping objects, which affects image fidelity in scenes with dense object interactions. While mitigation strategies are discussed via the CMA mechanism, further refinement is required to eliminate this issue completely. Additional exploration into the computational efficiency and scalability of the proposed methods would also benefit the paper, providing a more comprehensive view of their practical applications and limitations.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Introduces innovative methods for disentangling and integrating spatial and semantic information from scene graphs, which is a novel approach in image generation. \n2. Offers significant improvements in image generation from complex scene graphs, enhancing both the fidelity and controllability of generated image", "weaknesses": "1. The paper lacks quantitative comparisons with closely related baselines, such as R3CD, which could provide a more comprehensive evaluation of the model's performance. Inclusion of these comparisons could help validate the proposed advantages of DisCo over existing methods, particularly in handling complex scene graph-to-image generation tasks.\n2. Some generated images, particularly those highlighted in Figure 10, exhibit unnatural aspect ratios and stretched elements, suggesting issues with the model’s handling of object proportions and spatial embeddings. \n3. It would be great to discuss the scalability aspects, particularly how the proposed model handles graph sizes that exceed typical training configurations.\n4. how the model performs with imperfect or noisy scene graphs, which are common in automatically extracted data.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the affirmation from the reviewer for our work. It serves as a strong motivation for us! Bellow we address your concerns sequentially.\n\n**Q1: More quantitative comparisons with related baselines, such as R3CD.**\n\n**R1**: Actually, in Table 1 of the manuscript, we have already provided the quantitative comparisons with the related SG2I baselines, including diffusion-based (e.g., R3CD) and GAN-based methods (e.g., SG2Im). We can see that our DisCo achieves the best in both IS and FID scores.\n\nBesides, we conduct a user study by recruiting 50 participants from Amazon Mechanical Turk.  We randomly select 8 prompts for each method, and ask participants to score each generated image independently based on the image-prompt alignment. The worker can choose a score from {1, 2, 3, 4, 5} and we normalize the scores by dividing them by 5. We then compute the average score across all images and all workers. The results of user study are presented in the table below.\n|||||||\n|:----:|:----:|:----:|:----:|:----:|:----:| \n|**Method**|SG2Im|SGDiff|SceneGenie|R3CD|**DisCo (ours)**|\n|**Alignment Score**|0.3783|0.4717|0.5032|0.6928|**0.8533**|\n\n**Q2: Unnatural aspect ratios issue in Figure 10.**\n\n**R2**:  Thank you for the questions raised by the reviewer. Actually, this is due to typographical error of the manuscript, and we will fix it in revison. \n\nNevertheless, we can still easily see from Figure 10 that our DisCo has a superior ability in modeling **non-spatial relationships**, such as human \"holding\" and \"looking at\" in Figure 10. \n\n**Q3: Scalability aspect of graph size, object complexity and scene density.**\n\n**R3**: Great point! We sincerely appreciate the valuable suggestion, and we will discuss the scalability aspects in the revision. Actually, during the training of our SL-VAE, the way we used that forms **a batch of graphs with different node quantities** is to merge them into one larger graph, where each subgraph is not connected to each other (similar to independent nodes). Therefore, the GCN-based encoder can effectively learn relationships independent of the number of nodes, and thus handle graph sizes that exceed typical training configurations.\n\nBesides, we also provide the generation results with higher variability in object complexity and scene density. Please refer to **Figure A1 of the newly added PDF file** for more information, which demonstrates that our proposed model could handle larger graph sizes well.\n\n**Q4: How the model performs with imperfect or noisy scene graphs?**\n\n**R4**: Thank you for the questions raised by the reviewer. Actually, our method performs robust enough and is not sensitive to the imperfect or noisy scene graphs. For example, we consider two cases of imperfect or noise scene graphs, i.e., the anti-logical and the missing relationships.\n\n- **Anti-logical relationship**. There may occur anti-logical relationship in the provided scene graph, such as “*a dog on the sky*” and “*a sheep on top of a tree*”. As stated in the manuscript, our DisCo disentangles spatial relationships and interactive semantics. The layouts that represent the spatial relationships can output relative positions that follow the description even with an anti-logical input. Besides, the embeddings that represent the interactive semantics influence visual semantics with the given anti-logical relationship while ensuring semantic correctness. Thus, our method could still generate reasonable outputs matching the descriptions even if they are anti-logical. We present the anti-logical generation sample in **Figure A2 the newly added PDF file**.\n\n- **Missing relationships**. In case of node relationship missing (as we discussed in Figure 6 (c) of the manuscript), we deliberately define a special node \"__#image#__\" and a special relationship \"__#in#__\", which represent the whole scene and the relationship of objects to the scene, respectively. All regular nodes are connected to the \"__#image#__\" node with the \"__#in#__\" edge. Thus, independent nodes (i.e., missing relationship) could still interact with other nodes in the same scene, and infer their placement based on mutual semantics. In addition, our explicit spatial layout further ensures that there are no node missing problems. We present the results of the missing relationships in **Figure A3 of the newly added PDF file**. We will add these details in revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Yp80tB6uTe", "reviewer": "Reviewer_SBq5", "summary": "The paper proposes DisCo (Disentangled Compositional image generation), which integrates both layout and semantic information derived from scene graphs to improve the quality and controllability of generated images. In particular, DisCo has three main components: Semantics-Layout Variational AutoEncoder (SL-VAE) for disentangling spatial layouts and semantics, Compositional Masked Attention (CMA) for fine-grained attribute guidance, and Multi-Layered Sampler (MLS) for object-level graph manipulation. Extensive experiments demonstrate that DisCo outperforms state-of-the-art methods in generating rational and controllable images from text, layout, and scene graph conditions.", "questions": "See the weakness.", "limitations": "Yes, authors have stated their limitations on attribute leakage of overlapping in Section A.6, as well as a short discussion on the broader impact in Section A.7. \n\nBeside that, authors are also encouraged to add a few discussions on the efficiency of their proposed pipeline. Even though these overheads are inevitable (they are quite common in most researches), a clearer trade-off between the improved image quality and the increased model complexity would help to better assess the value of this work.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The motivation is clear. The idea of disentangling layout and semantics from scene graphs is novel.\n2. DisCo outperforms recent methods in both fidelity and diversity of image generation, as evidenced by the IS and FID scores. Overall, it enhances the generation diversity and controllability.\n3. Extensive experiments and ablation studies have demonstrated the effectiveness and the contribution of each component.", "weaknesses": "1. The increased inference cost of DisCo (Table 7). In particular, CMA mechanism might increase the computational cost, which may limit the method's scalability and efficiency, especially for large-scale applications. Moreover, since diffusion models are already quite large, the additional AutoEncoders (Lines 129-130) may result in more parameter and memory overhead.\n2. DisCo requires expensive training, e.g. 4 A100 GPUs, as indicated in Lines 202-203. With more models releasing recently, this technique might be not scalable.\n3. The image quality looks better with this proposed method. However, as metrics today cannot always reflect the real image quality, it would be more convincing to conduct a user study, e.g. votes, to quantify the advantage of DisCo compared to previous works.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate all of your valuable suggestions, which play a pivotal role in enhancing the quality of our paper. Bellow we address all your concerns.\n\n**Q1: Discussion about the complexity and quality.**\n\n**R1**: Thanks for your valuable suggestions. To comprehensively evaluate the complexity and efficiency of our DisCo, we here provide a clearer trade-off between the improved image quality and the increased model complexity to help readers better assess the value of our work. We evaluate the image quality using the T2I-CompBench [35] and measure model complexity using Floating Point Operations (FLOPs) and the number of parameters (Params). The results are shown in the following table.\n\n|||||||\n|--------------|:----:|:----:|:----:|:----:|:----:|\n|**Method**|**UniDet**|**B-VQA**|**3-in-1**|**FLOPs (G)**|**Params (M)**|\n|SD-v1.5|0.1246|0.3765|0.3080|677.5|859.4|37.9|0.3080|677.5|859.4|\n|**DisCo**|**0.2376**|**0.6959**|**0.4143**|732.9|894.6|\n|**(ours)**|**(+85.9%)**|**(+84.8%)**|**(+34.5%)**|(+8.2%)|(+4.1%)|\n\nAs we can see that our proposed DisCo achieves significant image quality improvements with a tolerable increase in computational cost.\n\nBesides, for the additional AutoEncoders (Lines 129-130) you mentioned, we rectify that this design would not bring more parameter and memory overhead, just negligible increase:\n\n||||\n|--------------|:----:|:----:|\n|**Method**|**FLOPs (G)**|**Params (M)**|\n|**DisCo (w/o SL-VAE)**|724.1|875.8|\n|**DisCo (w/ SL-VAE)**|732.9 **(+1.2%)**|894.6 **(+2.1%)**|\n\n\n**Q2: About training cost and scalability.**\n\n**R2**: We present the training cost between different methods in the following table. We use the A100-80G GPU hours as the metric.\n\n||||||\n|:----:|:----:|:----:|:----:|:----:| \n|**Method**|GLIGEN|MIGC|R3CD|**DisCo (ours)**|\n|**GPU hours**|~120|~300|~180|**~90**|\n\nIt can be found that that our proposed DisCo has lower training cost compared to other methods. Actually, our DisCo can be treated as a plug-in controller for other models, once we release the weights of SL-VAE and CMA upon paper acceptance, users could directly fine-tune models on GPUs like **3090-24G** (no need training from scratch) to make our DisCo scalable to more scenarios.\n\n**Q3: The metrics to reflect the real image quality, and the required user study.**\n\n**R3**: We sincerely appreciate the valuable suggestion. Actually, we evaluate our method on T2I-CompBench, where the metrics (i.e., the spatial/non-spatial relationships, attributes, and complex scenes) are validated to be consistent with human assessments [35]. \n\nHowever, we firmly believe that the user study you mentioned is more convincing. Therefore, we conduct a user study by recruiting 50 participants from Amazon Mechanical Turk. We randomly select 8 prompts for each method, resulting in 80 generated images. We ask participants to score each generated image independently based on the image-prompt alignment. The worker can choose a score from {1, 2, 3, 4, 5} and we normalize the scores by dividing them by 5. We then compute the average score across all images and all workers. The results of user study are presented in the table below.\n\n||||||||||||\n|:----:|:----:|:----:|:----:|:----:|:----:|:----:|:----:|:----:|:----:|:----:| \n|**Method**|SD-XL|DALL$\\cdot$E 3|Imagen 2|GLIGEN|LayoutDiffusion|MIGC|SG2Im|SGDiff|R3CD|**DisCo (ours)**|\n|**Alignment Score**|0.6684|0.5944|0.5637|0.6549|0.6200|0.7055|0.3783|0.4717|0.6928|**0.8533**|", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6yzbDS3EEG", "reviewer": "Reviewer_Knts", "summary": "This paper employs scene graph for image generation. Different from the previous methods, they employ the generative capabilities of variational autoencoders and diffusion models in a generalizable manner, compositing diverse disentangled visual clues from scene graphs. The authors propose a semantics-Layout Variational AutoEncoder to jointly derive layouts and semantics from scene graph. Then they develop CMA integrated with a diffusion model. They also introduce the multi-layered sampler for achieving graph manipulation. Experiments show that the method outperforms existing methods.", "questions": "No.", "limitations": "Yes.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. The paper address the problems in existing methods well. Existing methods in the field of scene graph to image generation mainly depends on the layout or semantics. Using one of them may cause some problems. Inspired by these phenomenons, the authors propose the method to jointly considering the layout and the semantics. What's more, the techniques used in the framework are novel enough. \n2. The authors conduct plenty of experiments. The ablation studies support the motivations.", "weaknesses": "The only weakness I found is that the authors should reorganize the paper carefully. The writings is not so clear in some sections. For example, the multi-layered sampler section is too abstract to be understood.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive comments and valuable feedback on our work! We are excited and encouraged by your support! Below we address your concern.\n\n**Q1: About the clarification of the multi-layered sampler section.**\n\n**R1**: We sincerely appreciate the valuable suggestion, and we will make our method more understandable in the revision. For the multi-layered sampler, we define each object as a layer, thus allowing independent object-level Gaussian sampling. Besides, we leverage diverse scene conditions ($N_l=5$) disentangled by SL-VAE for locally conditioned diffusion and aggregate them into the latent code for each object. In summary, it can be simply understood as calculating a more **robust layered representation** for each object in the scene, which achieves an “isolated” image editing effect.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear reviewers,

We thank all reviewers for their time and efforts in reviewing our paper. These constructive reviews can bring multiple improvements to our manuscript. We are encouraged that the reviewers appreciate our method, including:

 - structure design that makes sense *[Reviewer 3rPJ]*
 - innovative method *[Reviewer anRy, SBq5, Knts]*
 - clear motivation *[Reviewer SBq5, Knts]*
 - outperform prior methods in rationality and controllability *[Reviewer 3rPJ, anRy, SBq5]*

We have also made diligent efforts to address all the concerns raised point by point. Please see separate responses for details.

We are open to discussions and addressing any issues from reviewers. Your constructive comments can further help us to improve our method.

Sincerely yours,

Authors

Meta-review: This paper received generally positive reviews during the initial review period. Some concerns were raised during that period, such as the potential overhead in application, comparison with data-centric models, and comparison with related baselines. The author's response effectively addressed many of these concerns. I agree with the reviewers and recommend acceptance. The authors are encouraged to include their responses from the rebuttal in the final version for completeness.
Decision: Accept (spotlight)