Scene Graph Disentanglement and Composition for Generalizable Complex Image Generation

Authors: yunnan  wang
ziqiang  li
wenyao  zhang
zequn  zhang
baao  xie
xihui  liu
wenjun  zeng
xin  jin
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-07 03:51:09

Abstract: There has been exciting progress in generating images from natural language or layout conditions. However, these methods struggle to faithfully reproduce complex scenes due to the insufficient modeling of multiple objects and their relationships. To address this issue, we leverage the scene graph, a powerful structured representation, for complex image generation. Different from the previous works that directly use scene graphs for generation, we employ the generative capabilities of variational autoencoders and diffusion models in a generalizable manner, compositing diverse disentangled visual clues from scene graphs. Specifically, we first propose a Semantics-Layout Variational AutoEncoder (SL-VAE) to jointly derive (layouts, semantics) from the input scene graph, which allows a more diverse and reasonable generation in a one-to-many mapping. We then develop a Compositional Masked Attention (CMA) integrated with a diffusion model, incorporating (layouts, semantics) with fine-grained attributes as generation guidance. To further achieve graph manipulation while keeping the visual content consistent, we introduce a Multi-Layered Sampler (MLS) for an "isolated" image editing effect. Extensive experiments demonstrate that our method outperforms recent competitors based on text, layout, or scene graph, in terms of generation rationality and controllability.

Main paper: 
 introduction: 
 Text-to-image (T2I) generation with diffusion models (DMs) [1, 2] has yielded remarkable advancements [3, 4, 5] in recent years, benefiting from the developments of vision-language foundation models [6, 7, 8, 9] . However, textual conditions with linear structure struggle to delineate the intricacies of complex scenes precisely. For example, as shown in the failure cases of DALL•E 3 [3] in Figure 1 (a), given the intricate text prompt "A sheep by another sheep ... a boat on the grass.", the T2I model may have difficulty accurately generating object relationships or quantities. Consequently, some studies [10, 11, 12, 13] strive to improve spatial relationship (e.g., "by" and "on") control by incorporating additional layout conditions. Nevertheless, as illustrated in the failure cases of LayoutDiffusion [10] in Figure 1 (b), layout-to-image (L2I) methods inevitably encounter challenges in representing certain non-spatial interactions, such as depicting "playing" within spatial topology.To efficiently depict complex scenes for guiding generative models, recent methods [14, 15, 16] utilize structured scene graphs as conditions instead of text or layout prompts. Scene graphs [17] A sheep by another sheep on the grass with the ocean under the sky; the ocean by a tree; a boat on the grass represent scenes with a structured graph format, where objects within the scene are denoted as nodes and the relationships between objects are represented as edges. Scene-Graph-to-Image (SG2I) generation is a challenging task due to the frequent ambiguous alignment between graph edges and relationships/interactions among visual objects. To address this issue, layout-based SG2I methods [15, 17, 18, 19] explicitly predict the spatial arrangements of objects in scenes by additional layout predictors, followed by L2I synthesis according to the layout (as demonstrated in Figure 2 (a) ). These methods commonly employ one-to-one mapping, i.e., a single scene graph only corresponds to one layout, which severely limits the generation diversity. Besides, they also inherit the limitation of the L2I approach in modeling non-spatial interactions, whereby each object is typically generated independently. In contrast, as shown in Figure 2 (b), semantic-based SG2I methods implicitly encode graph edges into node embeddings by graph convolutional networks (GCNs), which effectively aligns object semantics with non-spatial interactions. Nonetheless, these methods are weak in logically determining the spatial positions of independent nodes, which might cause the absence of independent nodes (e.g., the "lamp" and "stone" shown in Figure 1 (c )) in the generated image.In this paper, we propose DisCo, a Compositional image generation framework that integrates the Disentangled layout and semantics derived from scene graph representations (as depicted in Figure 2 (c)). To boost the representational capacity of scene graphs for complex scenes, we augment the node and edge representations with CLIP [6] text embeddings, and incorporate extra spatial information (i.e., bounding box embeddings) for nodes during training. Once the textual scene graph is constructed, we propose a Semantics-Layout Variational AutoEncoder (SL-VAE) based on triplet-GCN [18] to jointly model the spatial relationships and non-spatial interactions in the scene. SL-VAE allows the one-to-many disentanglement for spatial layout and interactive semantics that match the model [15] generate a spatial arrangement with an object layout; (b) Semantic-based SG2I models [14, 16] build interactive semantic embedding between objects; (c) Our method leverages scene graph representation by jointly deriving the disentangled layout and semantics with the proposed SL-VAE.input scene graph. This is achieved by sampling from a Gaussian distribution, offering object-level (layouts, semantics) conditions for the diffusion process [20] . Given that the layout and semantics encapsulate global and local relational information, we further introduce the Compositional Masked Attention (CMA) mechanism to inject object-level graph information with fine-grained attributes into the diffusion model, thereby preventing relational confusion and attribute leakage.Finally, we present a Multi-Layered Sampler (MLS) technique that leverages the diverse conditions generated by SL-VAE, achieving generalizable generation for object-level graph manipulation (i.e., node addition and attribute control) in the SG2I task, as depicted by the color change of two "sheep" in Figure 1 (d) .In summary, our key contributions are as follows: (i) We apply the textual scene graph as a structured scene representation and introduce the Semantics-Layout Variational AutoEncoder (SL-VAE) to disentangle diverse spatial layouts and interactive semantics from the scene graph; (ii) We present the Compositional Masked Attention (CMA) to inject extracted object-level graph information with finegrained attributes into the diffusion model, which avoids relational confusion and attribute leakage;(iii) We introduce the Multi-Layered Sampler (MLS), a technique that leverages the diverse conditions produced by SL-VAE to implement object-level graph manipulation while keeping the visual content consistent; (iv) Our method outperforms current text/layout-based methods in relationship generation and achieves significantly superior generation performance compared to state-of-the-art SG2I models, thus showcasing the generalization of textual scene graphs in depicting complex scenes. 
text-to-image diffusion models: 
 Diffusion models (DMs) [1] are generative models that learn the data distribution p(x) by gradually performing T -step noise reduction from the variables x T sampled from the Gaussian distribution N (0, 1). Thus the training process of DMs can be regarded as the reverse process of a Markov chain with a fixed length T . To generate high-resolution images with less computational resources, Latent Diffusion Models (LDMs) [20] encode the image x into the latent space z with the pre-trained Vector Quantized Variational AutoEncoder (VQ-VAE). Subsequently, the LDMs aim to predict the distribution p(z) rather than p(x). For the text-to-image LDMs, the text is encoded with the CLIP [6] text encoder E CLIP . Then the objective function of a text-guided LDM can be formulated as follows:EQUATIONwhere E CLIP (c) is the text embedding of the text condition c, t is the diffusion step, ϵ θ is a model for estimating the noise ϵ, and z t = α t z t-1 + σ t ϵ is the t-step noised latent code from the ground-truth z 0 . During inference, the model ϵ θ with various samplers [1, 21] gradually denoises the initial noise z T ∼ N (0, 1). Finally, the predicted latent code is decoded into the image space. The scene graph G = (O, E) [17] Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: The limitations are addressed in the Appendix, which will be included in the camera-ready version.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
methodology: 
 As illustrated in Figure 3 , we present a novel SG2I synthesis framework known as DisCo. The DisCo comprises three primary components: ( O = {E o emb (c o i ) ⊗ E CLIP (o i ) ⊗ E box (b i )} No i=1 , E = {E e emb (c e ij ) ⊗ E CLIP (t ij )} 1≤i,j≤No,i̸ =j (2)where E CLIP denotes the frozen pre-trained text encoder, E box is the spatial encoder for bounding box coordinates using Multi-Layer Perceptions (MLPs), and ⊗ denotes concatenate operation.Graph Union Encoding. Although layout-based SG2I methods are superior in modeling spatial topology compared to the semantics-based method, they fall short in capturing object interactions (i.e., non-spatial relationships) within the scene. Accordingly, after obtaining the node and edge embeddings mentioned above, we apply a Conditional Variational Autoencoder (CVAE) [22] based on triplet-GCN [18] to jointly model the layout and semantics information. As shown in Figure 3 .I, the L-layer Graph Union Encoder E u takes node and edge embeddings as inputs:EQUATIONwhere l denotes the layer index of Graph Union Encoder, and ϕ denotes intermediate features. Here we initialize(ϕ 0 i , ϕ 0 ij , ϕ 0 j ) = (O i , E ij , O j ).Please refer to the Appendix for more details about the triplet-GCN. Given that the last node embedding ϕ L i integrates both topology and interaction information, we conduct layout-semantic modeling by parameterizing it into Gaussian spaces Z ∼ N (µ, σ). In this context, the means µ ∈ R Dz and variances σ ∈ R Dz are estimated individually by two supplementary MLPs, where D z denotes the dimensional of latent space for node embedding. Hence, we jointly model the layout and semantics through the following minimization:EQUATIONwhere KL denotes the Kullback-Liebler divergence, y denotes condition and the prior p(u|y) is the standard Gaussian distribution N (u | 0, 1). Specifically, we condition the latent space of the graph structure using the edge embedding following Equation 2 alongside the updated node embedding{E o emb (c o i ) ⊗ E CLIP (o i ) ⊗ u i } No i=1, where u i is a random vector sampled from Z. This architecture ensures that layout is solely necessary for training, with no need for hand-crafted layout in inference.Disentangled Semantics-Layout Decoding. As illustrated in Figure 3 .II, we disentangle the explicit spatial layout and implicit interactive semantics from the latent space using two separate triplet-GCNbased decoders, i.e., layout decoder D l and semantic decoder D s . The proposed Semantics-Layout Variational AutoEncode (SL-VAE) comprises these two decoders and the graph union encoder mentioned above, which derives the spatial topology and object interactions from the scene graph representation. During training, the layout decoder is optimized by the following objective function:EQUATIONwhere bi denotes the predicted coordinates. We only incorporate the ground truth layoutB = {b i } No i=1during training, while generating N l diverse layouts { Bn = { bn,i } No i=1 } N l n=1 by sampling Gaussian noise at inference time. For simplicity, we omit the superscriptˆin the following description. The semantic decoder D s generates semantics embeddings S = {s i } No i=1 to facilitate subsequent diffusion processes, and its parameters are iteratively updated with the diffusion loss in the next Section 3.2.Object-level Fusion Tokenizer. We integrate the spatial layout B = {b i } No i=1 and interactive semantics S = {s i } No i=1 at object level, as illustrated in Figure 3 .III. The single-object embeddingsC = {c i } No i=1 = {s i ⊗ F(b i )} No i=1are acquired by directly applying semantic embeddings, while encoding box information using a Fourier mapping F [23] . We define a learnable null embedding to pad the embedding length to N max , thereby accommodating varying numbers of objects:EQUATIONwhere c null denotes the learnable null embedding for padding. We optionally add attribute embeddingA = {a i } Nmax i=1 to construct updated C = {c i ⊗ a i } Nmax i=1, where c i and a i are separately processed by two MLPs before concatenation. Note that a i is obtained similarly to edge embedding in Equation 2. We also define a learnable null embedding a null for cases where no attribute is specified. object embedding visual token Compositional Masked Attention. The cross-attention mechanism in diffusion bridges the visual and textual information, while selfattention captures self-related information within visual tokens [24] . Therefore, we insert our proposed Compositional Masked Attention (CMA) between self-attention and cross-attention layers. This technique effectively injects graph information into the diffusion process at the object level, preventing semantic confusion and attribute leakage through the attention mask. Specifically, we denote the visual token output by the vanilla self-attention as V ∈ R Nv×Dv , where N v and D v represent the number and dimensions of tokens, respectively. Then the CMA layer can be expressed as:A B A1 A2 B1 A B (b) Attention mask A1 1 1 -inf -inf 1 1 1 -inf -inf 1 1 1 -inf -inf 1 1 1 -inf -inf 1 -inf -inf 1 1 -inf -inf -inf 1 1 -inf 1 1 -inf -inf 1 1 1 -inf -inf 1 -inf -inf 1 1 -inf A2 B B1 A B1 A object embedding visual token visual token object embedding A1 1 1 -inf -inf 1 1 1 -inf -inf 1 -inf -inf 1 1 -inf 1 1 -inf -inf 1 -inf -inf 1 1 -inf A2 B B1 A objectEQUATIONwhereĈ = {ĉ i } Nmax i=1denotes object embeddings whose dimensions are aligned with the visual token V using MLPs. The matrix M ∈ R (Nv+Nmax)×(Nv+Nmax) denotes the attention mask that depends on layout B, which can be constructed as follows:EQUATIONwhere "i, j fall into the same object" means that i and j index the visual tokens or object embeddings of the same object. Figure 4 illustrates the mechanism of CMA through a toy example. In contrast to vanilla self-attention, the proposed CMA prevents relational confusion and attribute leakage between different objects through the well-designed object-level masks mentioned above. As shown in Figure 3 .IV, we forward the output of the CMA layer into the subsequent layers, serving as the updated visual token.Diffusion Loss. Based on the object semantics output from the proposed SL-VAE, we optimize the evidence lower bound between sampling noise and prediction noise conditioned on object-level EQUATIONFinally, we employ an end-to-end joint training pipeline for the whole proposed DisCo framework. The total objective function is presented as follows:EQUATIONwhere λ 1 , λ 2 , and λ 3 are hyperparameters, which are typically set to 1.0, 0.1 and 1.0, respectively.Manipulations in the input scene graph, such as node addition and attribute adjustment, pose challenges for maintaining visual consistency in the generated images, ultimately compromising generalizability. To achieve an "isolated" image editing effect, we also provide the Multi-Layered Sampler (MLS) motivated by SceneDiffusion [25] . The scheme defines each object as a layer, thus allowing independent object-level Gaussian sampling. In contrast to SceneDiffusion which scrambles the reference layouts randomly, we sample additional N l (layouts, semantics) by the SL-VAE. Note that N l fixed seeds exist for the same scene. Then we aggregate latent codes from various layers into z n and utilize layout-converted non-overlapping masks {M n = {m n,i } No i=1 } N l n=1 for locally conditioned diffusion. During the inference, the noise estimation for N l scenes is calculated as follows:EQUATIONSubsequently, the latent code for each object is computed as the weighted average of the N l cropped denoised views. Please refer to the Appendix for more details about MLS.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Justification: We provide sufficient information on the computer resources in Section 4.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
experiments: 
 Dataset. We conduct scene-graph-to-image (SG2I) generation experiments on the Visual Genome (VG) [27] and COCO-Stuff (COCO) [26] datasets. The VG dataset comprises 108, 077 image-scene graph pairs, accompanied by the bounding box coordinates and object attributes. Following previous work [17] , we select objects and relationships that appear at least 2, 000 and 500 times respectively in VG, resulting in 178 objects and 45 unique relationship types. Also, we ignore small objects and use images containing 5 to 30 objects along with a minimum of 3 relationships. Based on the above filtering, we have 62, 565 images available for training, each containing an average of 10 objects and 5 relationships. While the original COCO-Stuff dataset [26] lacks scene graph annotations, it consists of 40, 000 images annotated with bounding box coordinates and captions, essential for synthesizing geometric scene graphs [15, 16] . All images in the COCO-Stuff dataset are labeled as 80 item categories and 91 stuff categories.Implementation Details. We fine-tune the pre-trained Stable-Diffusion 1.5 1 with the modified Attention module on 4 NVIDIA A100 GPUs, each with 80GB of memory. We apply the CLIP text encoder (vit-large-patch14 ) to construct the textual scene graph. We train the model with a batch size of 64 using the AdamW optimizer [30] with an initial learning rate of 1.0 × 10 -4 , which is adjusted linearly over 50, 000 steps. During inference, we use the 50-step PNDMScheduler [21] with a classifiers-free scale [31] of 7.5. The sample number N l in the multi-layered sampler is set to 5.Evaluation Metrics. Following previous works [14, 15, 16] , we evaluate the performance of our method with the Inception Score (IS) [32] and the Fréchet Inception Distance (FID) [33] . The IS score is derived from a pre-trained Inception Net [34] , assessing both the quality and diversity of synthesized images. The FID score quantifies the dissimilarity between the generated image and the real image distribution, which evaluates the fidelity of the generated images. To measure the effectiveness of compositional generation, we further evaluate our method on the T2I-CompBench [35] . Besides, we apply CLIP for zero-shot attribute classification of the controlled object cropped by the bounding box, and subsequently evaluate the attribute control performance by the classification accuracy ACC attr . 4 of the ablation study. We proceed to assess the compositional generation on the T2I-CompBench [35] , as shown in Table 2 . The benchmark evaluates the competency of the text-to-image model in responding to compositional prompts. We report UniDet, CLIP, B-VQA, and 3-in-1 scores for measuring the generation of spatial/non-spatial relationships, attributes, and complex scenes, respectively. Following [35] , we use the UniDet [39] , CLIP [6] , and BLIP [7] to evaluate these results. Our DisCo surpasses all compared T2I methods, confirming the efficacy of scene graphs in depicting complex scenes. User Study. We conduct a user study by recruiting 50 participants from Amazon Mechanical Turk. We randomly select 8 prompts for each method, resulting in 80 generated images. We ask participants to score each generated image independently based on the image-prompt alignment. The worker can choose a score from {1, 2, 3, 4, 5} and we normalize the scores by dividing them by 5. We then compute the average score across all images and all workers. The results are presented in the Table 3 .Our method is favored by most participants in terms of generation rationality and controllability. , LayoutDiffusion [10] , and MIGC [13] ), and (c) scene-graph-to-image (SG2I) (SG2Im [17] , SGDiff [16] , and R3CD [14] ) methods. Qualitative Comparisons. Figure 5 visualizes the results of the methods conditioned by text, layout, or scene graph, showcasing our advantages in generating rationality and controllability: (i)Comparison with the text-to-image (T2I) methods. In Figure 5 (a), we present the superiority of the disentangled structured scene graph over linear text for representing complex scenes. Firstly, we resolve ambiguity in textual relationships and semantics by employing layout and semantic disentanglement within the scene graph. For example, our DisCo clarifies the relationship between "boat" and "grass" in the first line, as well as the semantics of "bus" and "building" in the following line. Additionally, the samples in the first and last lines also showcase our capacity to generate the specified quantities of objects precisely. (ii) Comparison with the layout-to-image (L2I) methods. We also demonstrate that our DisCo outperforms the diffusion methods relying on manually crafted scene layout representations, as illustrated in Figure 5 (b) . While the L2I method struggles to model non-spatial interactions (such as "playing" in the first line and "chasing" in the second line), our DisCo addresses this challenge using disentangled object interactive semantics. Furthermore, by establishing semantics among objects derived from their relationships, we prevent independent generation instances that rely solely on the layout, exemplified by the "bus" and "tire" in the last line. (iii) Comparison with the scene-graph-to-image (SG2I) methods. The SG2I visualization results of different methods are showcased in Figure 5 (c). Our DisCo significantly improves the quality of SG2I generation, particularly for independent nodes. The proposed layout and semantics disentanglement technique effectively capture both the spatial and interactive information of independent nodes. Taking the "lamp" and "stone" in the first image and the "shadow" in the second image as illustrations, these entities are neglected by previous methods, whereas our DisCo not only retains their semantic relevance but also infers their appropriate spatial placement within the scene. We also demonstrate the generalizable generation under consistency for graph manipulation (i.e., node addition and attribute control) in SG2I tasks, as shown in Figure 6 . Ablation Study. Table 4 explores the overall architecture by evaluating the alignment of the generated image with the objects and relationships depicted in the input scene graph. Following SGDiff [16] , we conduct this analysis by graph-to-image (G2I) and imageto-graph (I2G) retrieval experiments. Note that "w/o D s " means processing each node embedding by MLP independently,instead of obtaining object interactive semantics through D s . We observe that the spatial layout and interactive semantics collaborate to boost both retrieval tasks. These results demonstrate the effectiveness of integrating explicit spatial relations with implicit interactive semantics. In Table 5 , we study the impact of different attention mechanisms. We inject graph conditions using different mechanisms: (a) Vanilla attention mechanism in the T2I diffusion model without our CMA; (b) CMA without attention mask M; (c) CMA with a union MLP after concatenating object and attribute embeddings; and (d) CMA with two separate MLPs before concatenating object and attribute embeddings. We found that CMA, which fuses separate encoding of object and attribute embeddings, significantly enhances the overall generation performance. Table 6 presents the Multi-Layer Sampler (MLS) ablation results, confirming its enhancement over the baseline and LSD [29] . In contrast to LSD, which randomly scrambles layouts, the proposed MLS naturally leverages a variety of coherent layouts and semantics produced by SL-VAE. Moreover, the increase in ACC attr scores also indicates that MLS facilitates controllability, especially in attribute control, while ensuring generation quality. 
related works: 
 Diffusion Models. Diffusion models (DMs) [2, 20, 31, 42] have achieved great success in high-quality image generation. The essence of DMs lies in estimating image distributions by iterative denoising noise-corrupted image, showcasing the superiority over VAEs [43, 44] and GANs [45] in training stability and likelihood estimation. To further explore the controllability of DMs, considerable efforts have been devoted to conditional generation based on DMs. Benefiting from the naturalness of language [6] and the advancements of vision-language foundation models [6, 7, 8, 9] , numerous textto-image DMs [40, 41, 46] are beginning to emerge, facilitating explicit control of the corresponding semantics and style. However, the expressive capacity of linear text is limited. Therefore, many studies also endeavor to bolster global control through supplementary conditions, such as depth [46, 47] , layout [10, 11, 12, 13] , segmentation map [46, 48] , and scene graph [14, 15] .Image Generation from Scene Graphs. Scene graphs are structured scene representations, where nodes represent objects and edges represent relationships between objects [17, 27] . Given the superiority of scene graphs over linear text in delineating multiple objects and their intricate relationships [27, 49, 50] , many studies investigate image generation from scene graphs. These approaches typically fall into two categories: layout-based and semantics-based methods. Layout-based methods [17, 18, 19, 28, 36, 51] initially map scene graphs to coarse scene layouts comprising multiple bounding boxes and further refine these layouts to images with a layout-to-image model (e.g., Layout-Diffusion [10] ). While the layout depicts spatial relationships, it fails to capture abstract relationships within the scene, leading to a lack of object interaction. Another branch is semantics-based methods [14, 16, 29, 52, 53] , which focus on graph understanding by directly encoding semantic information from the scene graph. Nevertheless, these methods have limitations in addressing independent scene nodes, leading to issues like entity loss and unreasonable placement. In this paper, we propose a compositional image generation that leverages the layout and semantics derived from the scene graph representation. We complement explicit layout and implicit semantics to enhance the understanding of the diffusion model for scene graphs. Additionally, to improve the controllability in the scenegraph-to-image task, we also attain generalizable generation for object-level graph manipulation (i.e., node addition and attribute control).6 Limitations The proposed CMA injects object-level information into the diffusion model via masks from the layout, effectively mitigating semantic ambiguity and limiting attribute leakage. In scenarios involving object overlap, the proposed CMA inhibits direct interaction between the visual token and the object embedding along with its attributes. Nonetheless, the attribute information from the visual token inadvertently leaks into the overlapping region in subsequent layers. Hence, there may be attribute leakage among the objects, as shown in Figure 7 . 
conclusion: 
 In this study, we leverage the disentangled textual scene graph representation to condition the diffusion process for generating complex scene images. The innovation of our framework lies in utilizing the VAE for scene relationship modeling and diffusion model (DM) for the composite visual generation. To comprehensively capture spatial relationships and non-spatial interactions within scenes, we introduce the Semantics-Layout Variational AutoEncoder (SL-VAE) for deriving diverse layouts and semantics from a single scene graph. Building upon them, we propose the Compositional Masked Attention (CMA) integrated with DM, which guides the de-noising trajectory by compositing extracted object-level graph information with fine-grained attributes. We also introduce a Multi-Layer Sampler (MLS) to preserve the main visual content while modifying the input scene graph. Extensive experiments demonstrate that our framework outperforms current methods conditioned by text, layout, or scene graph in relationship modeling and controllability. 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes]Justification: We discuss both potential positive and negative societal impacts in Section A.6 of the Appendix.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [Yes] Justification: Our model is heavily based on existing StableDiffusion, so our safeguards are the same as theirs.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: In Section 4, we properly credit all the public baselines and datasets utilized in this paper.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [Yes] Justification: The assets introduced in the paper are well documented.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] Justification: This paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA] Justification: This paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.• Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: This appendix is organized as follows: Section A.1 provides more details about Semantics-Layout Variation AutoEncoder; Section A.2 introduces the Stable Diffusion and its attention mechanism; Section A.3 describes the implementation of the Multi-Layer Sampler in detail; Section A.4 covers more ablation studies; Section A.5 presents more qualitative results, including comparison visualization and graph manipulation; Section A.6 delves into the broader societal impacts of this work. The core script is zipped and attached to the supplementary material.A.1 Semantics-Layout Variation AutoEncoder Recall that we apply the triplet-GCN-based CVAE architecture in Section 3.1. Each triplet-GCN layer in the encoder and decoder takes the node and edge embeddings. Specifically, the GCN l mentioned in the paper uses two cascading MLPs {mlp 1 , mlp 2 } to deal with node and edge embeddings:EQUATIONEQUATIONwhere l denotes the layer index of encoder or decoder, N E denotes the neighbor index set for each node, avg denotes the average pooling operation, ϕ and ψ denote intermediate features. Hence, mlp 1 conducts message passing among interconnected nodes and updates the edge features, while mlp 2 aggregates features from all neighboring nodes and updates its features. For graph union encoder, we let(ϕ 0 i , ϕ 0 ij , ϕ 0 j ) = (O i , E ij , O j ).The last embedding ϕ L i is parameterized to the Gaussian distribution Z ∼ N (µ, σ), where µ, σ ∈ R Dz output by two additional MLPs and D z denotes the dimensional of latent space for node embedding. [20] is one of most popular text-to-image model. As described in Section 2.1, Stable Diffusion uses a U-Net ϵ θ composed of convolution and transformer to estimate noise. The transformer includes two attention mechanisms, namely Cross-Attention, and Self-Attention.Cross-Attention Layer. Text prompts are mapped to sequence embeddings by CLIP text encoder and integrated into UNet via Cross-Attention to guide the de-noising trajectory:EQUATIONwhere Q visual denotes the Query from the visual token of the UNet, K text and V text denotes Key and Value from text embeddings, all of which are projected by linear layers, d denotes the dimension of Q visual , K text , and V text .Self-Attention Layer. Self-Attention captures self-related information within visual tokens:EQUATIONwhere Q visual , K visual , and V visual separately represent the Query, Key, and Value in self-attention layers, which are projected by linear layers. The self-attention mechanism isolates the information flow between specific tokens by multiplying a mask M to the Q visual K T visual . Since M is applied before sof tmax, the value of the isolated position is set to negative infinity -inf .Compositional Masked Attention Layer. Based on the attention mask M that depends on layout B, the Compositional Masked Attention can be expressed as:EQUATIONwhere Q CM A , K CM A , and V CM A individually represent the Query, Key, and Value derived from V ⊗ Ĉ, achieved through linear layer projections. We insert our proposed Compositional Masked Attention (CMA) between self-attention and cross-attention layers.A.3 Multi-Layer Sampler Layered Scene Representation. We decompose a controllable scene containing N o objects into N o layers. Different from SceneDiffusion [25] , our approach involves each layer incorporating not only Operations (FLOPs), the number of parameters (Params), and inference time (Time) to measure computing consumption. The FLOPS and Time metrics are conducted by processing the tensor with a resolution of 2 × 4 × 64 × 64 on an NVIDIA A100 GPU. Our proposed DisCo significantly improves the controllability of the Stable Diffusion with a tolerable increase in computational cost.Figure 8 showcases more generalizable generation results under consistency for graph manipulation (i.e., node addition and attribute control) in SG2I task. In Figure 9 , 10, and 11, we present more visualization comparisons with the methods conditioned by text, layout, or scene graph, which demonstrates the superiority of our DisCo in terms of generation rationality and controllability.We demonstrate the superiority of our DisCo over existing generation methods based on text, layout, and scene graphs, suggesting a potential beneficial influence on the realms of art creation and data synthesis. Nevertheless, there remains a concern regarding the possibility of generating malicious images or infringing copyright.Graph Manipulation (Node Addition and Attribute Control) A teddy bear sit on the ground; another two teddy bear besides the tableThe street by the building; the car next to the truck Two cow on top of grass; one near the tree; sky above the grass; a building near the tree Tree zebras standing on the grass; two near the tree; one in front of the houseA building in front of a mountain; a tree in front of the building Justification: We claim in the abstract and introduction that this paper studies generating complex images with structured scene graphs.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.Justification: Our work conforms with the NeurIPS Code of Ethics in every respect.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).https://huggingface.co/runwayml/stable-diffusion-v1-5 


Decision: This paper received generally positive reviews during the initial review period. Some concerns were raised during that period, such as the potential overhead in application, comparison with data-centric models, and comparison with related baselines. The author's response effectively addressed many of these concerns. I agree with the reviewers and recommend acceptance. The authors are encouraged to include their responses from the rebuttal in the final version for completeness.

Reviews: review: paper_summary: paper_summary: This paper proposes a method that uses a scene graph and integrates variational autoencoders (VAEs) and diffusion models to address complex scene generation. Specifically, a Semantics-Layout Variational AutoEncoder (SL-VAE) is used to derive diverse layouts and semantics from the scene graph, while a Compositional Masked Attention (CMA) combined with a diffusion model incorporates these elements as guidance. Additionally, a Multi-Layered Sampler (MLS) is introduced for isolated image editing. Experiments show that this approach outperforms recent methods in generation rationality and controllability.
strength_weakness: strength_weakness: 1. My main concern is the practical application of this method. As we all known, scene graph building is not a trivial task, but you don't explain detail in the paper how to construct an exact scene graph. In addition, during inference, the prompt proposed by uses may be non-standard so that building a scene graph may be more difficult. 
2. Besides, recent SOTA models, e.g. DALLE3, stable diffusion 3 try to solve the complex generation task by large-scale fine-grained dataset construction. How do you compare your methods with these data-centric methods. The authors should spend more space discussing the issues.
questions: questions: Please see the weakness
limitations: limitations: Please see the weakness
paper_summary: paper_summary: This paper presents "DisCo," a novel framework for generating complex images from structured scene graphs. Unlike traditional text-to-image or layout-to-image methods, DisCo utilizes a Semantics-Layout Variational AutoEncoder (SL-VAE) to disentangle and generate diverse spatial layouts and interactive semantics from scene graphs. It incorporates these elements using a Compositional Masked Attention (CMA) mechanism within a diffusion model, enhancing generation control and rationality. The framework also introduces a Multi-Layered Sampler (MLS) for flexible, graph-based image editing, preserving visual consistency while manipulating object attributes and positions.
strength_weakness: strength_weakness: 1. The paper lacks quantitative comparisons with closely related baselines, such as R3CD, which could provide a more comprehensive evaluation of the model's performance. Inclusion of these comparisons could help validate the proposed advantages of DisCo over existing methods, particularly in handling complex scene graph-to-image generation tasks.
2. Some generated images, particularly those highlighted in Figure 10, exhibit unnatural aspect ratios and stretched elements, suggesting issues with the model’s handling of object proportions and spatial embeddings. 
3. It would be great to discuss the scalability aspects, particularly how the proposed model handles graph sizes that exceed typical training configurations.
4. how the model performs with imperfect or noisy scene graphs, which are common in automatically extracted data.
questions: questions: 1. The paper presents results on standard benchmarks. However, can you provide insights or preliminary results on how the model performs across datasets with higher variability in object complexity and scene density? 
2. Why were certain closely related baselines omitted from quantitative comparisons? Could inclusion of these baselines provide a more comprehensive evaluation?
limitations: limitations: The authors address the challenge of attribute leakage in overlapping objects, which affects image fidelity in scenes with dense object interactions. While mitigation strategies are discussed via the CMA mechanism, further refinement is required to eliminate this issue completely. Additional exploration into the computational efficiency and scalability of the proposed methods would also benefit the paper, providing a more comprehensive view of their practical applications and limitations.
paper_summary: paper_summary: The paper proposes DisCo (Disentangled Compositional image generation), which integrates both layout and semantic information derived from scene graphs to improve the quality and controllability of generated images. In particular, DisCo has three main components: Semantics-Layout Variational AutoEncoder (SL-VAE) for disentangling spatial layouts and semantics, Compositional Masked Attention (CMA) for fine-grained attribute guidance, and Multi-Layered Sampler (MLS) for object-level graph manipulation. Extensive experiments demonstrate that DisCo outperforms state-of-the-art methods in generating rational and controllable images from text, layout, and scene graph conditions.
strength_weakness: strength_weakness: 1. The increased inference cost of DisCo (Table 7). In particular, CMA mechanism might increase the computational cost, which may limit the method's scalability and efficiency, especially for large-scale applications. Moreover, since diffusion models are already quite large, the additional AutoEncoders (Lines 129-130) may result in more parameter and memory overhead.
2. DisCo requires expensive training, e.g. 4 A100 GPUs, as indicated in Lines 202-203. With more models releasing recently, this technique might be not scalable.
3. The image quality looks better with this proposed method. However, as metrics today cannot always reflect the real image quality, it would be more convincing to conduct a user study, e.g. votes, to quantify the advantage of DisCo compared to previous works.
questions: questions: See the weakness.
limitations: limitations: Yes, authors have stated their limitations on attribute leakage of overlapping in Section A.6, as well as a short discussion on the broader impact in Section A.7. 

Beside that, authors are also encouraged to add a few discussions on the efficiency of their proposed pipeline. Even though these overheads are inevitable (they are quite common in most researches), a clearer trade-off between the improved image quality and the increased model complexity would help to better assess the value of this work.
paper_summary: paper_summary: This paper employs scene graph for image generation. Different from the previous methods, they employ the generative capabilities of variational autoencoders and diffusion models in a generalizable manner, compositing diverse disentangled visual clues from scene graphs. The authors propose a semantics-Layout Variational AutoEncoder to jointly derive layouts and semantics from scene graph. Then they develop CMA integrated with a diffusion model. They also introduce the multi-layered sampler for achieving graph manipulation. Experiments show that the method outperforms existing methods.
strength_weakness: strength_weakness: The only weakness I found is that the authors should reorganize the paper carefully. The writings is not so clear in some sections. For example, the multi-layered sampler section is too abstract to be understood.
questions: questions: No.
limitations: limitations: Yes.

Comments: title: Official Comment from Authors
Thanks for your kind explanation
comment: Thank you for your continued efforts. We have provided comprehensive rebuttals and tried to address the concerns raised in your reviews. Please take the time to review if possible, if you have any further questions or require additional clarification, please let us know and we welcome discussions in any format. Thanks again.
Thanks for the authors' rebuttal. They have addressed my concerns. Thus, I would like to raise my score to 6.
After reading the responses and comments from other reviewers, most concerns have been addressed. I am especially interested   of the anti-logical relationships generation. I keep my rating considering that this paper deserves being accepted. I also hope the authors could release the codes as soon as possible.
After thoroughly reading your rebuttal and other reviewers' comments, I will raise my score to 5