CommonScenes: Generating Commonsense 3D Indoor Scenes with Scene Graph Diffusion
Authors: Guangyao Zhai
Evin Pinar Örnek
Shun-Cheng Wu
Yan Di
Federico Tombari
Nassir Navab
Benjamin Busam
Venue: NeurIPS 2023 poster (2023)
Keywords: Scene Graph
Scene Synthesis
Diffusion Model
Graph Convolution Network
TL;DR: Generate indoor scenes using scene graph-conditioned diffusion models.

Abstract: Controllable scene synthesis aims to create interactive environments for numerous industrial use cases. Scene graphs provide a highly suitable interface to facilitate these applications by abstracting the scene context in a compact manner. Existing methods, reliant on retrieval from extensive databases or pre-trained shape embeddings, often overlook scene-object and object-object relationships, leading to inconsistent results due to their limited generation capacity. To address this issue, we present CommonScenes, a fully generative model that converts scene graphs into corresponding controllable 3D scenes, which are semantically realistic and conform to commonsense. Our pipeline consists of two branches, one predicting the overall scene layout via a variational auto-encoder and the other generating compatible shapes via latent diffusion, capturing global scene-object and local inter-object relationships in the scene graph while preserving shape diversity. The generated scenes can be manipulated by editing the input scene graph and sampling the noise in the diffusion model. Due to the lack of a scene graph dataset offering high-quality object-level meshes with relations, we also construct SG-FRONT, enriching the off-the-shelf indoor dataset 3D-FRONT with additional scene graph labels. Extensive experiments are conducted on SG-FRONT, where CommonScenes shows clear advantages over other methods regarding generation consistency, quality, and diversity. Codes and the dataset are available on the website.

CommonScenes: Generating Commonsense 3D Indoor Scenes with Scene Graph Diffusion
Guangyao Zhai
Evin Pinar Örnek!.2- 1.2
Shun-Cheng Wu²
Yan Di
Federico Tombari 1.3
Nassir Navab Benjamin Busam
Iguangyao.. zhai,
yan, di dijeea (étum.
Technical University of Munich
Munich Center for Machine Learning
*Google
https://sites.
httess// con//view/comoonsceee
Abstract
Controllable scene synthesis aims to create interactive environments for numerous industrial use cases. Scene graphs provide highly suitable interface to facilitate these applications by abstracting the scene context in compact manner. Existing methods, reliant on retrieval from extensive databases or pre-trained shape embeddings, often overlook scene-object and object-object relationships, leading to inconsistent results due to their limited generation capacity. To address this issue, we present CommonScenes, a fully generative model that converts scene graphs into corresponding controllable 3D scenes, which are semantically realistic and conform to commonsense. Our pipeline consists of two branches, one predicting the overall scene layout via variational auto-encoder and the other generating compatible shapes via latent diffusion, capturing global scene-object and local inter-object relationships in the scene graph while preserving shape diversity. The generated scenes can be manipulated by editing the input scene graph and sampling the noise in the diffusion model. Due to the lack of scene graph dataset offering high- quality object-level meshes with relations, we also construct SG-FRONT, enriching the off-the-shelf indoor dataset 3D-FRONT with additional scene graph labels. Extensive experiments are conducted on SG-FRONT, where CommonScenes shows clear advantages other methods regarding generation consistency, quality, and diversity. Codes and the dataset are available on the website.
Introduction
Controllable Scene Synthesis (CSS) refers to the process of generating or synthesizing scenes in way that allows for specific entities of the scene to be controlled or manipulated. Existing methods operate on images [62] or 3D scenes [34] varying by controlling mechanisms from input scene graphs [24] or text prompts [5]. Along with the development of deep learning techniques, CSS demonstrates great potential in applications like film and video game industry [7], augmented and virtual reality [65], and robotics [72, 68]. For these applications, scene graphs provide a powerful tool to abstract scene content, including scene context and object relationships.
This paper investigates scene graph- based CSS for generating coherent 3D scenes characterized by layouts and object shapes consistent with the input scene graph. To achieve this goal, recent methods propose two lines of solutions. The first line of works optimizes scene layouts [35] and retrieves objects [16] from given database (see Figure (a)). Such retrieval-basee approaches are
"The first two authors contributed equally. 'Corresponding author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Shape Database
Code Datahase
Lavous
Clear
Tublle
CVAB
Clai
Trlle
Uaitel
Database
nuil
CVAR
Code CNNE
Codes
Bedl
Deooders n
Dillision
a) Retrieval
Semi-Generatiie
Generative (Ours)
Compared with previous methods, our fully generative model requires neither databases nor multiple category-leve decoders. Performance Comparison (Bottom Row): We demonstrate the effectiveness of encapsulating scene-object and object-object relationships. The semantic information from the scene graph is 'a table is surrounded by three chairs'. As highlighted in the rounded rectangles, through the scene-ohject relationship. our network outperforms other methods by generating a round table and three evenly distributed chairs. Through the object-ohject relationship, the three chairs are consistent in style. Moreover, our method still preserves the object diversity (blue dashed rectangle).
Figure Architecture Comparison (Upper Row):
inherently sub-optimal [57] in generation quality due to performance limitations through the size of the database. The second solution, e.g., Graph-to-3D [12], regresses both the layout and shape of the objects for synthesis. However, the shape generation relies on pre-trained shape codes from category-wisg auto-decoders, e.g., DeepSDF [42]. This cemi-generative design (Figure 1 (b)) results in reduced shape diversity in the generated outputs. To enhance generation diversity without relying vast databases, one possible solution is concurrently predict scene layouts and generate object shapes with text-driven 3D diffusion models[8, 32], where the textual information is obtained from input scene graphs. Yet, in our experiments, observe that such intuitive algorithm works poorly since it does not exploit global and local relationship cues among objects encompassed by the graph.
In this work, our approach CommonScenes exploits global and local scene-object relationships and demonstrates that fully generative approach can effectively encapsulate and generate plausible 3D scenes without prior databases. Given a scene graph, during training, we first enhance it with pre- trained -isual-language model features, c.g., CLIP [46], and bounding box embeddings, incorporating coarse local inter-object relationships into the feature of each node. Then, leverage triplet- GCN [35] based framework to propagate information among objects, learning layouts through global cues and fine local inter-object relationships, which condition the diffusion process [47] to model the shape distribution as well. During inference, each node in the scene graph is enriched with the learned local-to-global context and sequentially fed into the latent diffusion model for each shape generation. Figure (c) illustrates that our method effectively leverages relationship encoding generate commonsense scenes, i.e., arranged plausibly and realistically, exhibiting scene-level consistency while preserving object shape diversity. Furthermore, to facilitate the benchmarking of CSS, we curate novel indoor scene graph dataset, SG-FRONT. upon a synthetic dataset 3D- FRONT [18], since no existing indoor scene graph datasets provide high-quality meshes. SG-FRONT comprises around 45K 3D samples with annotated semantic and instance segmentation labels and corresponding scene graph describing each scene.
Our contributions be summarized into three points. First, present CommonScenes, fully generative model that converts scene graphs into corresponding 3D scenes using a diffusion model. be intuitively manipulated through graph editing. Second, CommonScenes concurrently models scene layout and shape distribution. It thereby encapsulates both global inter-object relationships and local shape cues. Third, contribute SG-FRONT, synthetic indoor dataset extending 3D-FRONT by scene graphs, thereby contributing graph-conditional scene generation benchmarks.
2 Related Work
Scene Graph Scene graphs provide rich symbolic and semantic representation of the scene using nodes and relationships [25]. They are useful in many 2D-related topics such as image

---

generation [24, 67]. image manipulation [12], caption generation [31], visual question answering [55], and camera localization [27]. Quickly after this progress, they are used in following areas: 3D scene understanding [59, 1, 64, 29], dynamic modeling [49], robotic grounding [22, 50, 68), spatio-temporal 4D [75, 30, 74J, and controllable scene synthesis [33, 73, 60, 41, 13].
Indoor 3D Scene Synthesis Controllable scene synthesis is extensively explored in the computer graphics community, varying from text-based scene generation [19] to segmentation map [43] or spatial layout-based image-to-imagee translation tasks [71]. Likewise in 3D, several methods generated scenes from images [56, 39, 2, 69, 14, 40, 38], text [36], probabilistic grammars [4. 23, 11, 45], layouts [26, 54], in an autoregressive manner [44, 63], or through learning deep priors [61]. Another line of work closer to ours is based on graph conditioning [33, 73, 60, 41]. Luo et al. [35] proposed generative scene synthesis through variational modeling coupled with differentiable rendering. However, their method relies on shape retrieval, which depends on an existing database. Graph- to-3D [13] proposes a joint approach learn both scene layout and shapes with scene graph condition. Nevertheless, their object generation relies on a pre-trained shape decoder, limiting the generalizability. Unlike previous work, our method generates 3D scenes with the condition over a scene graph, which is trained end-to-end along with content awareness, resulting in higher variety and coherency.
Denoising Diffusion Models A diffusion probabilistic model is trained Markov chain that generates samples matching data by reversing a process that gradually adds noise until the signal vanishes [52]. Diffusion models have quickly gained popularity due to their unbounded, realistic, and flexible generation capacity [21, 53, 37, 28, 47, 15]. However, studies have identified that the diffusion models lack compositional understanding of the input text [51]. Several advancements have been introduced to address these limitations. Techniques such as the introduction of generalizable conditioning and instruction mechanisms have emerged [S, 70J. Moreover, optimizing attention channels during testing has also been explored [6, 17J. Recently, a latent diffusion model using a Signed Distance Field (SDF) to represent 3D shapes was proposed contemporaneouslly works [8, 32], which can be conditioned on a text or a single view image. For the contextual generation conditioned on scene graphs, methods have converted triplets into the text to condition the model [17, 67], where Yang et al. [67] proposed a graph conditional image generation based on masked contrastive graph training. To the best of our knowledge, we are the first leverage both areas of scene graph and latent diffusion for end-to-end 3D scene generation.
multiple
3 Preliminaries
Scene Graph A scene graph, represented as 9 = (V,E), is a structured representation of a visual scene where V (w i (1....,NN) denotes the set of vertices (object nodes) and E = {ei-s j.j E (1,...,N), N), * j} represents the set of directed edge from node U; to fj Each vertex v/ is categorized through an object class chode E Cnode, where Cnode denotes the set of object classes. The directed edges in capture the relationships between objects in terms of both semantic and geometric information. Each edge ei-+; has predicate class edgg E Cedge, where Cedge denotes the set of edge predicates. These relationships can incorporate various aspects, such as spatial locations (e.g., left/ /right, cl ose by) object properties (b i ggger/ sma eer). To facilitate subsequent processing and analysis, each node and edge ei-J are typically transformed into learnable vectors o and Ti-sj. respectively, through embedding layers, as shown in Figure 2.A.
Conditional Latent Diffusson Model Diffusion models learn a target distribution by reversing
a progressive noise diffusion process modeled by a fixed Markov Chain of length T [21, 53]. Fundamentally, given a sample from the latent space, gradual Gaussian Noise is added with a predefined scheduler x,,6 L E (1...., T} [21]. Then, the denoiser Ea, typically a UN] is trained to recover denoising from those samples. The recently introduced Latent Diffusion Models (LDMs) [47] reduce the computational requirements by learning this distribution a latent space established by a pre-trained VAE [58] instead of directly processing the full-size input. The popular usage of LDM is conditional LDM, which allows the generation to obey the input cue c; [47]. The training objective can be simplified
LLDM ((.1) [||= Ea c.)
(I)
3

---

Graph Representations
Contextial Cimpph Og
Rus-eole Groph
Bed
Bea
"Bed"
f
f
MLP
New trshcking Lugene
Beal
Front
"Bella in box: of T'Ta
Enoole
3% oj
Th-
fr.
How Parme de
Exlldding Lagur
Edge
Oy
Encoding
Figure 2: Scene Graph Evolution. Take the features of two nodes Bed (ox), Table (0,) and the linked edge In front (7i+s) feature, respectively. We enhance the node and edge features with CLIP feature Pt, Pi-+ to obtain B. Contextual Graph. Then, we parameterize the ground truth bounding box b, to the node to further build C. Box- Fox--Enhnneed Contextual Graph with node and edge feature represented as p {p., Os b fe {pi- Te-s).
example. where (os,0;) Ti-+j embedded learnable node features and the edge
where c, denotes a conditioning vector corresponding to the input sample fed into Eg.
4 Method
Overview Given semantic scene graph, our approach endeavors to generate corresponding 3D scenes conforming to commonsense. We employ a dual-branch network starting with contextual encoder Ee, as shown in Figure 3. The two branches, referred to as the Layout Branch and the Shape Branch, function simultaneously for layout regression and shape generation. In Sec. 4.1, first illustrate how a scene graph evolves to a Box-en Box-enhanced Contextual Graph (BCG) with features from pre-tree pre-trained visual- lasgul-an model CLIP [46] and bounding box parameters (Figure 2). Then, we show how BCG is encoded by E. and manipulated by the graph manipulator (Figure 3.A, B and C). In Sec. 4.2, we introduce the layout branch for layout decoding, and in Sec. 4.3, we introduce shape branch for shape generation. Finally, we explain the joint optimization in Sec. 4.4.
4.1 Scene Graph Evolution
Contextual Graph As shown in Figure 2.A and B, we incorporate readily available prompt features from CLIP [46] as semantic anchors, capturing coarse inter-object information, into each node and edge of the input graph to conceptualize it as Contextual Graph Go with p; = Ecup (cquo(c for objects and Pi++ (Ecui(cpode H dedge B Bejode) for edges. Here ECLIP is the pre-trained and frozen text encoder [46], denotes the aggregation operation on prompts including subject class eqode, edge predicate (-tj and object class ey ezode, Thereby, the embeddings of Ge (Vor Ee) formalized as,
Fv.. (ft (pi, E (1, (I...., NH), Fs. =f--"' (pi+i, T--5) j E (1,...,NJ} (2)
where Fv. represents the set of object features and Fs. the edge features.
Box-Enhanced Contextual Graph In training, we enrich each node in the contextual graph by using ground truth bounding boxes parameterized by box sizes 8, locations I, and the angular rotations along the vertical axis a, yielding BCG, shown in Figure 2.C. Thereby, the node embeddings of BCG are represented by F(() = {f = (p., o, by) | f1,.... , .N), where b, is obtained by encoding (st, t,, ax) with MLPs. Note that BCG is only used during training, i.c., bounding box information not needed during inference.
Graph Encoding BCG is encoded by the subsequent Caplet-GGC--aae contextual encoder E., which together with the layout decoder D; in Sec. 4.2 construct Conditional Variational Autoencoder (CVAB) [35]. The encoding procedure shown in Figure 3.A. and C. Given a BCG, during training, we input the embeddings F()) and Fe, into E. to obtain an Updated Contextual Graph with node
and edge features represented as (2) FF.), which is also the beginning point of the inference route. Each layer of E, consists of two sequential MLPs (91, ). where 91 performs message passing between connected nodes and updates the edge features, G2 aggregates features from all connected neighbors of each node and update its features, as shown in the follows:
(uly dlu+1 o5-45" : why 91(0'y Ly
lg O....,L-
(3)
$1" = -//g + 92 (VV by E No(u:)))

---

Blox-Eeharcced Contexiual Graph
Shape Branch
Pr
0x
b
(8)4
Relation Emmzedinngs Nodes
Bed
Denoiser to
fe
Relation
Encoder
Noos
C. Updazed Contextual Graph
D. Layout Branch
Only
Comeestial Enceder
Lak
Pronoon
Layour Deseder D)
Medalex
Pa-id
f
Inference
Manipulator
B. Shared Madilees
Scene Layouts
Scene Generation
Figure 3: Overview of CommonSceness. Our pipeline consists shared modules two collaborative branches I Layout Branch and Shape Branch. Given a BCG (Figure 2.C), we first feed into Fe,, yielding joint layout-shape distribution Z. We sample 2; from Z for each node, obtaining concatenated feature (2i, Pi, 0.) with CLIP feature pt and self-updated feature o;. graph manipulator then optionally adopted manipulate the graph for data augmentation. Next, updated contextual graph fed into layout branch shape branch for layout regression and shape generation respectively. In the shape branch, we leverage E, to encapsulate global scene-object and local object-object relationships into graph nodes, which are then conditioned to Ee in LDM via roos-attentionn mechanism generate Xe back in T steps. Finally, frozen shape decoder (VQ-VAE reconstructs S' using Xg- The final scene is generated by fitting S' layouts.
where lg denotes a single layer in E. and Ng (v;) includes all the connected neighbors of u, AVG refers to average pooling. We initialize the input embeddings of layer as the features from the Updated Contextual Graph, (6%, &e 8% ( , p(b)))) The final embedding oi is used to model a joint layout-shape distribution Z, parameterized with the Na-dimensional Gaussian distribution Z [u,, o). where ,4, R.N: predicted by two separated MLP heads. Thereby, Z is modeled through minimizing:
LKL= = DRE (2 FE.) F1.) (1
where DEL is the Kullback- posterior distribution p(= .r) chosen to be standard N( 0, 1). We sample random vector 2; from Z for each node, update node feature F( by concatenating (2i, Pi, 0. keep edge features unchanged.
divergence measuring the discrepancy between Z and the
4.2 Layout Branch
As shown in Figure 3.D, decoder Dr, which another triplet-GCN, generates 3D layout predictions upon updated embeddings. In this branch, E,, and D; are jointly optimized by Eq. (4) and a bounding box reconstruction loss:
N Σ (1s - 8:l1 + +11 G2/1 N
Liagout
Σ a log á²);
(
1=1
where S; f, di denote the predictions of bounding box size locations and angular rotations a, respectively. À is the rotation classification label. We partition the rotation space into bins, transforming the rotation regression problem into classification problem.
4.3 Shape Branch
As shown in Figure 3.E, in parallel the Layout Branch, we introduce Shape Branch to generate shapes for each node in the given graph, and represent them by SDF.
Relation Encoding The core idea our method to exploit global scene-object and local object- object relationships guide the generation process. Hence, besides incorporating CLIP features for coarse local object-object relationships, as introduced in Section 4.1, we further design relation encoder E,, based triplet-GCN well, to encapsulate global semantic cues of the graph into
5

---

each node and propagate local shape cues between connected nodes. Specifically, E,- operates on the learned embeddings (F Fe.) of the Updated Contextual Graph, and updates the feature of each node with local-to-global semantic cues, yielding node-relation embeddings for subsequent diffusion-based shape generation.
Shape Decoding We use an LDM conditioned on node-relation embedding to model the shape- generation process. In terms of shape representation, we opt for truncated SDF (TSDF) [10] around the surface of the target object within voxelized space S E DDDxD Following the LDM, the dimension D of TSDF can be reduced by training VQ-VAE [58] as shape compressor to encode the built upon a discretized codebook.
3D shape into latent dimensions x E Rdxdxd, where << Then forward diffusion process adds random noise to the input shape X transferring to xy, upon which we deploy a 3D-UNet [9] Eq (Figure 3.E) to denoise the latent code back to X according to DDPM model by Ho et al. [21]. The denoiser is conditioned on node-relation embeddings to intermediate features of 3D UNet via the cross-attention mechanism. Finally, the decoder of VQ- V-A generates the shape S' from the reconstructed Xo. For the denoising process at the timestep the training objective is to minimize:
(2) Lshape Fx (O..).. [|le - Ex E,, (F..) Fe.M1
(6)
where the evidence lower bound between the sampled noise E and the prediction conditioned on the contextual relation embedding extracted from E,- is optimized. At test time, latent vector is randomly sampled from (0,, 1) and progressively denoised by 3D-Unet to generate the final shape. Each shape within the layout is populated based on per-node conditioning, ultimately producing plausible scene. Compared to prior work, our design can bring more diverse shape generation by taking advantage of the diffusion model architecture, shown in experiments.
4.4 Layout-Shape Training
Our pipeline trained jointly in an end-to-end fashion, allowing the Layout Branch and Shape Branch to optimize each other by sharing latent embeddings (F(²) FF.). coming out of Eg. The final optimization loss is the combination scene distribution modeling, layout generation, and shape generation:
L = CKL A2hlayout XaCahapes
where λ1. A2 and As are weighting factors. Further insights on accomplishing the batched training are provided in Supplementary Material.
5 Experiments
SG-FRONT Dataset Due to the lack of scene graph datasets also providing high-quality object meshes, we construct SG-FRONT, a set of well-annotated scene graph labels, based on a 3D synthetic dataset 3D-FRONT [18] that offers professionally decorated household scenarios. The annotation labels can be grouped into three categories spatial/preximiity, support, and style. The spatial relationships are based series of relationship checks between the objects, which control the object bounding box location, e.g., left/right, the comparative volumes, e.g., bi giger/////. r sma 1 ler, /shorerrr The support relationships include directed structural support, e.g. cl close by, above, and st standing on. Thresholds for these labels are iterated and set by the annotators. Finally, style relationships include the object attributes to assign labels, namely same material as, same shape as, and s same super ceter-cctegoryy as. SG-FRONT contains around 45K samples from three different indoor scene types, covering 15 relationship types densely annotating scenes. More details are provided in the Supplementary Material.
and heights, e.g.,
Implementation Details We conduct the training, evaluation, and visualization of CommonScenes on single NVIDIA 1000 GPU with 40GB memory. We adopt the AdamW optimizer with an initial learning rate of le-4 to train the network in an end-to-end manner. We set (X1, 22, da} = (1.0, 1.0, 1.0} in all our experiments. N₂ in distribution set to 128 and TSDF size D set as 64. We provide more details in the Supplementary Material.

---

Dining Room
Bedroom
Living Room
Cheelty
Seft
Table
Ulivety
Abenqu.e
Biaadd
Nigrrand
Fed
Close Seatiir
Al Zapae Dellad
Stadler
Above Rihidd
Abbre
Prie
Preet Leck
Rignt
Les
Sadu
Chai
Take
Aere Righi
Char-
Castr
Aleers
Kaft
Right Chee by
Highic
Scene Graph Input
Graph-to-Bol
Graph-to-3D
CommonScenes (Ours)
Figure 4: Qualitative comparison The orientations of l.eft/ /it tht and Front/Beh nd in the graph align with the top-down view. Both scene-ohject and object-ohject inconsistencies are highlighted rectangles. Green rectangles emphasize the commonsense consistency our method produces.
Evaluation Metrics To measure the fidelity and diversity of generated scenes, we employ commonly adopted Fréchet Inception Distance (FID) [20] Kernel Inception Distance (KID) [3] metrics [44]. We project scenes onto bird's-eye view, excluding lamps to prevent occlusion and using object colors as semantic indicators. To measure the scene graph consistency, follow the scene graph constraints [13], which measure the accuracy of set

---

Shape
Bedroom
Living room
Dining room
All
Method
Representation
FID
KID
FID
KID
FID
KID
FID
KID
3D-SLN [35]
57.90
3.85
77.82
3.65
69.13
6.23
44.77
3.32
Progressive [13]
58.01
7.36
79.84
4.24
71.35
6.21
46.36
4.57
Retrieval
Graph-to-Box [13]
54.61
2.93
78.53
3.32
67.80
6.30
43.51
3.07
Ours w/o SB
52.69
2.82
76.52
2.08
65.10
6.11
42.07
2.23
Graph-to-3D [13]
DeepSDF [42]
63.72
17.02
82,96
11.07
72,51
12.74
50.29
7.96
Layout=tX22shpp
SDFusion [8]
68.08
18.64
85,38
10.04
64.02
5.08
50.58
8.33
Ours
rel2shupe
$7.68
6.59
80.99
6.39
65.71
5.47
45.70
3.84
Table 1: Scene generation realism as measured by FID and KID (x0.001) scores at 256 pixels between the top-down rendering of generated and real scenes (lower is better). Two main rows separated with respect the reliance on an external shape database for retrieval. "Ours w/o SB" refers to ours without the shape branch.
Nicoree
Coraiorri
Concieann
Consistency
Diversity
Method
Chair
Table
Chair
Table
Graph-to- [13]
10.42
50.66
0.53
0.89
Graph-to-3D [13]
2.49
11.74
1.43
0.93
Ours
1.96
9.04
30,04
10.53
Scal
Scad
Table 2: Consistency and diversity in the dining
Figure 5: Consistency co-exists with diversity in different rounds. Our generated objects show diversity when activated twice while preserving the shape consistency within the scene (chairs in a suit).
rooms. The object shapes related with same within a scene are consistent as indicated by low values (x0.001), whereas the shapes across different runs have high diversity, per high KL divergence.
of relations on a generated layout. We calculate the spatial left/right, front /behind, sma 1 llerrrrla 1 arger, taller/ shorter metries, well as more diffieult proximity constraints To measure the shape consistency, we test dining rooms, a typical
close by and syymmtrrcaa scenario in which tables and chairs should be in suits according the commonsense decoration. We identify matching dining chairs and tables in 3D-FRONT using the same CAD models and note their instance IDs. This helps us determine which entities belong to the same sets. We calculate Chamfer Distance (CD) of each of the two objects in each set after generation. To measure the shape diversity, we follow [13] to generate each scene 10 times and evaluate the change of corresponding shapes using CD. We illustrate the concepts of diversity and consistency in Figure 5, We also report MMD, COV, and I-NNA following [66] for the object-level evaluation in the Supplementary Materials.
Compared Baselines We include three types of baseline methods. First, three retrieval-based methods, i.e., layout generation network 3D-SLN [35], a progressive method add objects one- by-one designed in [13]. and Graph-to-Bar from [13]. Second, a semi semi-generative SOTA method Graph-to-3D 3D. Third, an intuitive method called exyout+xx2shhap that we design according to the instruction in Sec. 1, stacking a layout network and a text-to-shapp generation model in series, and it a fully generative approach with text only. All baseline methods are trained on the SG-FRONT dataset following the public implementation and training details.
5.1 Graph Conditioned Scene Generation
Qualitative results The generated scenes from different baselines shown in Figure 4: (a) Graph- to-Box retrieval, (b) Graph-to-3ID, (c) layout= and (d) ours. It can be seen that our method has better shape consistency and diversity. Specifically, in terms of object shape quality, retrieval baseline scenes look optimal since they are based on real CAD models. Yet, the layout configuration poor, and the retrieved object styles are not plausible, e.g., dining chairs and nightstands are not in suit, respectively. in the same rooms. Graph-to-3D improves this by learning coherent scenes. However, the object styles are not conserved within scene styles, providing unrealistic scene-object inconsistency. e.g., in the bedroom, the height of the chair does not match the style and the height of the table. Subsequently, (c) improves the layout, which can be seen in the living room area, but again falls back on the generated shape quality and shares the same problem with Graph-to-Bor on the poor object-object consistency. In contrast, CommonScenes can capture diverse environments considering both generation consistency and stylistic differences with respect to each scene. In the
8

---

Shape Representation
Easy
Hard+
Method
left right
front behind
smaller larger
taller/ shorter
close
symmetrical
3D-SLN [35]
0.97
0.99
0.95
0.91
0.72
0.47
Progressive [13]
Retrieval
0.97
0.99
0.95
0.82
0.69
0.46
Graph-to-Bor [13]
0.98
0.99
0,96
0,95
0.72
0.45
Ours wño SB
0.98
0.99
0.97
0.95
0.74
0.63
Graph-to-3D [13]
DeepSDF (42)
0.98
0.99
0.97
0.95
0.74
0.57
Ours
rel2shape
0.98
1,00)
0.97
0.95
0.77
0.60
Table 3: Scene graph constrains on the generation task (higher is better). The total accuracy is computed as the mean over the individual edge class accuracy to minimize class imbalance bias.
Shape Representation
Lasy
Hards
Method
Mode
fright
smailler larger
by+
sy symmetricul-
3D-SLN [35]
0.89
0.90
0.55
0.58
0.10
0.09
Progressive [13]
Retrieval
0.89
0.52
Graph-to-Box [13]
0.91
0.91
0.86
0.91
0.66
0.53
Ours w/o SB
change
0,91
0.92
0.86
0.92
0.70
0.53
Graph-to-3D [13]
DeepSDF [42]
0.91
0.92
0.86
0.89
0.69
0.46
Ours
rel2shape
0,91
0.92
0.86
0.91
0.69
0.59
3D-SLN [35]
0.92
0.92
0.56
0.58
0.05
0,05
Progressive Gruph to Box [13]
Retrieval
0.92
0.91
0.53
0,54
0.02
0.06
0.94
0.93
0.90
0.94
0.67
0.58
Ours
addition
0.95
0.95
0.90
Graph-to-3D [13]
DeepSDF [+2]
0.94
0.95
0.91
0.93
0.63
0.47
Ours
rel2shape
0.95
0.95
0.91
0.70
0.61
Table 4: Scene graph constraints on the manipulation task (higher is better). The total accuracy computed as the mean over the individual edge class accuracy to minimize class imbalance bias. Top: Relationship change mode. Bottom: Node addition mode.
living room, the scene graph input requires that Table No.3 and Table No.4 should stand close each other. Graph-to-Box and Graph-to-3D fail to achieve the goal, which is also reflected in Table 3 and Table 4 that these kinds of methods cannot handle close by relation. We show more results in Supplementary Material.
Quantitative results We provide the FID/KID scores in Table 1 separated as average over three room types. Our method establishes the best results among the free-shape representation methods by improving the Graph-to-3D by approximately 10% on FID and 40% on KID on average. On the other hand, most of the layout+Xr2shapp results are worse than the previous state-of-the-art methods, proving the requirement of learning layout and shape jointly. Notably, the retrieval-basee methods overall have better scores than those of free shapes since the retrieved objects align well with the test database. Further on, among the retrieval basclines, (Ours w/o SB) surpasses the prior works, indicating the benefit of contextual graph representation. Additionally, on the generation consistency, the edge relationship in the generated shapes directly, indicated by significant improvement in the consistency score. By leveraging the latent diffusion model, our diversity is significantly improved compared with other methods. We provide more results on diversity and also show perceptual study on the evaluation of consistency and diversity in the Supplementary Material.
without the diffusion shape branch
observe in Table 2 that our method can reflect
In terms scene graph constraints, we present the results as easy and hard metrics in Table 3. On easy metrics, methods (Ours, Ours w/o SB) are either better than While on hard metrics, they are superior to others. shows that the spatial relationships could be learned easier than the proximity (close by) and commonsense (synmet rical) based ones. The improvement over the s symmet t rical constraint is particularly evident, as our method surpasses the state-of-the-an (SOTA) by 10%. This demonstrates that the integrated layout and shape cues crucial for learning these contextual configurations.
par with other methods.
5.2 Scene Manipulation
We demonstrate the downstream application of scene manipulation our method and compare with the aforementioned methods on scene graph constraints Table 4. Our methods have the highest total score in both relation change and object addition modes. In both modes, the results

---

on with Graph-to-3) 3D on easy relations. whereas there is major improvement in the hard ones. Further, within the retrieval methods. Ours w/o SB improves over the baselines in the addition mode and is par in the change mode. It could be argued that changing could be more difficult since it could alter multiple objects, whereas object insertion has less effect on the overall scene. We show some qualitative manipulation examples in the Supplementary Material.
5.3 Ablations
We ablate the primary components of CommonScenes in Table 5, including (1) scene generation from the original scene graph without contextual information brought by CLIP features, (2) scene generation from the contextual graph without the participation of the relational encoder E,, (3) conditioning with concatenation on diffusion latent code instead of cross-attention, and (4) our final proposed method. We provide the mean FID scores over the scenes, as well as the mean over the hard scene graph constraints (mSG). We observe the benefit of both the context and E,, indicated by FID/KID scores, as well as the choice of our conditioning mechanism.
Condition
FID
KID
mSG
Ours w/o context
50.24
5.63
0.59
Ours w/o E,
51.62
5.41
0.73
Ours with concat
48.57
4.25
0.71
Ours
45.70
3.84
0.74
Soone Gruph lepul
Failure Case
Figure 6: An interpenetrating phenomenon.
Table 5: Ablations under three circumstances.
6 Limitations
We address the main aspects of the dataset and the limitations of our method and discuss more in the Supplementary Materials. First, the 3D-FRONT dataset used in our research contains significant noise, which we have mitigated through post-processing step. Despite this effort, small proportion of noisy data, specifically interpenetrating furniture instances, remains in the training dataset. Consequently, our proposed method and the baseline approaches reflect this during inference, rarely resulting in occurrences of scenes with collided objects. We show some cases in Figure 6. While our method outperforms others by effectively combining shape and layout information, it is essential to note that minor collision issues may still arise.
Conclusion
Scene graph-based CSS designs interactive environments suitable for wide range of usages. Current methods heavily depend on object retrieval or pre-trained shape models, neglecting inter-object relationships and resulting in inconsistent synthesis. To tackle this problem, we introduce CommonScenes, a fully generative model that transforms scene graphs into corresponding commonsense 3D scenes. Our model regresses scene layouts via variational auto-encoder, while generating satisfying shapes through latent diffusion, gaining higher shape diversity and capturing the global scene-object relationships and local object-object relationships. Additionally, we annotate a scene graph dataset, SG-FRONT, providing object relationships compatible with high-quality object-level meshes. Extensive experiments SG-FRONT show that our method outperforms other methods in terms of generation consistency, quality, and diversity.
Acknowledgements This research is supported by Google unrestricted gift, the China Scholarship Council (CSC), and the Munich Center for Machine Learning (MCML). We are grateful to Google University Relationship GCP Credit Program for supporting this work by providing computational resources. Further, we thank all participants of the perceptual study.
10

---

References
[1] Iro Armeni, Zhi- Yang He, Young Gwak, Amir R Zamir, Martin Fischer, Jitendra Malik, and Silvio Savarese. 3d scene graph: A structure for unified semantics, 3d space, and camera. In ICCV, 2019. Joon Park, Despoina Paschalidou, Xingguang Yan, Gordon Wetzstein, Leonidas
[2] Sherwin Bahmani, Guibas, and Andrea Tagliasacchi. Cc3d: Layout-conditioned generation of compositional 3d scenes. In
ICCV, 2023, 3
[3] Mikotaj Binkowski, Dougal J. Sutherland, Michael Arbel, and Arthur Gretton. Demystifying MMD GANs.
In ICLR, 2018. 7
[4] Martin Bokeloh, Michael Wand, Hans-Peter Seidel, and Vladlen Koltun. An algebraic model for parameterized shape editing. ACM Transactions on Graphics (TOG), 31(4):1-10, 2012. 3
[5] Tim Brooks, Aleksander Holynski, and Alexci A Efros. Instr Learning to follow image editing
instructions. In CVPR, 2023. 1,
[6] Hila Chefer, Yuval Alaluf, Yael Vinker, Lior Wolf, and Daniel Cohen-Or. Attend-and and-excite: Attention-
based semantic guidance for text-to-image diffusion models, 2023. 3
[7] Mo Jiao Chen, Bin Zhang, and San Xing Cao. A study of dynamic scene automatic generation system for micro-film studio. In Applied Mechanics and Materials, volume 411, pages 993-996, 2013. I
[8] Yen-Chi Cheng. Hsin-Yinn Lee, Sergey Tuyakov, Alex Schwing, and Liangyan Gui. SDFusion: Multimodal 3d shape compiction, reconstruction, and generation. Özgun Çişek, Ahmed Abdulkadir, Soeren Lienkamp. Thomas Brox, and Olaf Ronneherger. 3d u-net: Learning dense volumetric segmentation from sparse annotation. In MICCAI, 2016. 6
CVPR, 2023. 2, 3, 8
[10] Brian Curless and Mare Levoy. A volumetric method for building complex models from range images. In SIGGRAPH, 1996.
[11] Jeevan Devaranjan, Amlan Kar, and Sanja Fidler. Meta-sim2: Unsupervised learning scene structure for synthetic data generation. In ECCV, 2020. 3
[12] Helisa Dhamo, Azade Farshad, Iro Laina, Nassir Navab, Gregory D. Hager, Federico Tombari, and Christian Rupprecht. Semantic image manipulation using scene graphs. In CVPR, 2020. 2, 3
[13] Helisa Dhamo, Fabian Manhardt, Nassir Navab, and Federico Tombari. Graph-to-3d: End-to-end generation
and manipulation of 3d scenes using scene graphs. In ICCV, 2021. 3, 7, 8,
[14] Yan Di, Henrique Morimitsu, Zhiqiang Lou, and Xiangyang Ji. A unified framework for piccewise semantic reconstruction in dynamic scenes via exploiting superpixel relations. In ICRA, 2020.
[15] Yan Di, Chenyangguang Zhang, Pengyuan Wang, Guangyao Zhai, Ruida Zhang, Fabian Manhardt, Benjamin Busam, Xiangyang Ji, and Federico Tombari. Ced-3dr: Consistent conditioning in diffusion for single-image 3d reconstruction, 2023.
[16] Yan Di, Chenyangguang Zhang, Ruida Zhang, Fabian Manhardt, Yongzhi Su, Jason Rambach, Didier Stricker, Xiangyang Ji, and Federico Tombari. U-red: Unsupervised 3d shape retrieval and deformation for partial point clouds. In ICCV, 2023.
[17] Weixi Feng. Xuchai Hc, Tsu-Jui Fu, Varun Jampani, Arjun Reddy Akula, Pradyumna Narayana, Sugato Basu, Xin Eric Wang, and William Yang Wang. Training-free free structured diffusion guidance for compositional text-to- -oxmace- synthesis. In ICLR, 2023. 3
[18] Huan Fu, Bowen Cai. Lin Gao, Ling-Xiao Xiao Zhang. Jiaming Wang, Cao Li, Qixun Zeng, Chengyue Sun, Rongfei Jia. Binqiang Zhao, al. 3d-front: 3d furnished rooms with layouts and semantics. In ICCV,
2021. 2. 6
[19] Oran Gafni, Adam Polyak, Oron Ashual, Shelly Sheynin, Devi Parikh, and Yaniv Taigman. Make-a-scene: Scene-based text-to-image generation with human priors. In ECCV, 2022.
[20] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two smm-sccae update rule converge to a local nash equilibrium. In NewrlPS, 2017, 7
[21] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In NeurlPS, 2020. 3,
[22] N. Hughes, Y. Chang, and L. Carlone. Hydra: A real-time spatial perception system for 3D scene graph construction and optimization. In RSS, 2022. 3
[23] Chenfanfu Jiang, Siyuan Qi, Yixin Zhu, Siyuan Huang, Jenny Lin, Lap-Fai Yu, Demetri Terzopoulos, and Song-Chun Zhu. Configurable 3d scene synthesis and 2d image rendering with per-pixel ground truth using stochastic grammars. International Journal of Computer Vision, 126:920 14009941,20 2018 3
[24] Justin Johnson, Agrim Gupta, and Li Fei-Fei. Image generation from scene graphs. In CVPR, 2018. 1,
[25] Justin Johnson, Ranjay Krishna, Michael Stark. Li-Jia Li, David A. Shamma, Michael S. Bernstein, and Li
Fei-Fei. Image retrieval using scene graphs. In CVPR, 2015. 2
[26] Akash Abdu Jyothi. Thibaut Durand, Jiawei He, Leonid Sigal, and Greg Mori. Layoutvae: Stochastic
scene layout generation from a label set. In ICCV, 2019. 3
[27] Julia Kabalar, Shun- -Chnng Wu, Johanna Wald, Keisuke Tateno. Nassir Navab, and Federico Tombari.
Towards long-term betsee visual localization in indoor environments with changes. IEEE Robotics
and Automation Letters, (99::1975-9988 2023. 3
[28] Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-basse generative models. In NewrlPS, 2022.
[29] Hwan Kim, Jin-Man Park, Taek-jin Song, and Jong- Hwan Kim. 3-d scene graph: A sparse and semantic representation of physical environments for intelligent agents. IEEE Transactions on Cybernetics. 50(12):4921 4933349993399------ 2020. 3
11

---

[30] Xin Kong, Xuemeng Yang. Guangyao Zhai, Xiangrui Zhao. Xianfang Zeng, Mengmeng Wang. Yong Liu, Wanlong Li, and Feng Wen. Semantic graph based place recognition for 3d point clouds. In IROS, 2020. [31] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanic Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. Interaational journal of computer vision, 123(3)332-3,
2017. 3
[32]
Muheng Li, Yueqi Duan, Jie Zhou, and Jiwen Lu. Diffusion-sdf: rext-to-shape via voxelized diffusion. In
CVPR, 2023. 2,3
[33] Manyi Li, Akshay Gadi Patil, Kai Xu, Siddhartha Chaudhuri, Owais Khan, Ariel Shamir, Changhe Tu, Baoquan Chen, Daniel Cohen-Or, and Hao Zhang. Grains: Generative recursive autoencoders for indoor
scenes. ACM Transactions on Graphics (TOG), 38(2):1-16, 2019. 3
[34] Yiyi Liao, Katja Schwarz, Lars Mescheder, and Andreas Geiger. Towards unsupervised learning of
generative models for 3d controllable image synthesis. In CVPR, 2020. 1
[35] Andrew Luo. Zhoutong Zhang. Jiajun Wu, and Joshua B. Tenenbaum. End-to-end optimization of scene
layout. In CVPR, 2020. 2. 3. 4, 8,
[36] Rui Ma, Akshay Gadi Patil, Matthew Fisher, Manyi Li, Sören Pirk, Binh- Hua, Sai-Kit Young, Xin Tong. Leonidas Guibas, and Hao Zhang. Language-driven synthesis of 3d scenes from scene databases.
ACM Transactions on Graphics (TOG), 37(6);116, 2018. 3
Chenlin Meng. Yutong He. Yang Song. Jiaming Song. Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon. SDEdit: Guided image synthesis and editing with stochastic differential equations. In ICLR, 2022. 3 [38] Yinyu Nic, Angela Dai, Xiaoguang Han, and Matthias Nießnerr Learning 3d scene priors with 2d supervision. In CVPR, 2023. 3
[37]
[39] Yinyu Nie. Xiaoguang Han, Shihui Guo, Yujian Zheng, Jian Chang, and Jian Jun Zhang. TotdaBdunderstanddngg Joint layout, object pose and mesh reconstruction for indoor scenes from single imgee In CVPR, 2020. 3
[40] Evin Pinar Örnek, Shristi Mudgal, Johanna Wald, Yida Wang, Nassir Navab, and Federico Tombari. From 2d to 3d: Re-thinking benchmarking of monocular depth prediction. 2022. 3
[41] Wamiq Para, Paul Guerrero, Tom Kelly, Leonidas J Guibas, and Peter Wonka. Generative layout modeling
using constraint graphs. In ICCV, 2021. 3
[42] Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In CVPR, 2019. 2, 8,9 9
[43] Taesung Park, Ming- Yu Liu, Ting-Chun Wang, and Jun-Yan Zhu. Semantic image synthesis with spatially-
adaptive normalization. In CVPR, 2019. 3
[44] Despoina Paschalidou, Amlan Kar, Maria Shugrina, Karsten Kreis, Andreas Geiger, and Sanja Fidler.
Atiss; Autoregressive transformers for indoor seene synthesis. In NeurIPS, 2021. 3.7
[45]
Pulak Purkait, Christopher Zach, and Ian Reid. Sg-vae: Scene grammar variational autoencoder to generate
new indoor scenes. In ECCV, 2020. 3
[46] Alec Radford, Jong Wook Kim. Chris Hallacy. Aditya Ramesh, Gabriel Goh. Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In /CML,, 2021. 2,
[47]
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution
image synthesis with latent diffusion models. CVPR, 2022. 2,
[48]
Olaf Ronneberger. Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In MICCAI, 2015.
[49] Antoni
Arjun Gupta, Marcus Abate, Jingnan Shi, and Luca Carlone, 3d dynamic scene graphs:
Actionable spatial perception with places, objects. and humans. In RSS, 2020. 3
[50] Antoni Rosinol, Andrew Violette, Marcus Abate, Nathan Hughcs, Yun Chang, Jingnan Shi, Arjun Gupta, and Luca Carlone. Kimera: From slam to spatial perception with 3d dynamic seene graphs. The International Journal of Robotics Research,
14:12-144:111-111
2021. 3
[51] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li. Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes. Burcu Karagol Ayan, Tim Salimans, Jonathan Ho. David J Fleet, and Mohammad Norouzi. Photoreafistic text-to-image diffusion models with deep language understanding.
In NeurlPS, 2022. 3
[52] Jascha fiek-Dicsttin, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning
using nonequilibrium thermodynamics. In /CML, 2015. 3
[53] Yang Song, Jascha Sohl-Dicksterd Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole.
Score-hased generative modeling through stochastic differential equations. In /CLR, 2021. 3
[54] Jiapeng Tang, Yinyu Nie, Lev Markhasin, Angela Dai, Justus Thies, and Matthias Nießner. Diffuscene:
Scene graph denoising diffusion probabilistic model for generative indoor scene synthesis. 2023. 3
[55] Damien Teney, Lingqiao Liu, and Anton van Den Hengel. Graph-structure, representations for visual question answering. In CVPR, 2017. 3
[56] Shubham Tulsiani, Saurabh Gupta, David Foubey, Alexei A. Efros, and Jitendra Malik. Factoring shape. pose, and layout from the 2d image of 3d scene. In CVPR, 2018. 3
[57] Mikacla Angelina Uy. Vladimir G Kim, Minhyuk Sung, Noam Aigerman, Siddhartha Chaudhuri, and Lconidas J Guibas. Joint learning of 3d shape retrieval and deformation. In CVPR, 2021. 2
[58] Aaron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete representation learning. In NewrIPS, 2017. 6
12

---

[59]
Johanna Wald, Helisa Dhamo, Nassir Navab. and Federico Tombari. Learning 3d semantic scene graphs from 3d indoor reconstructions. In CVPR, 2020. 3
[60] Kai Wang. Yu-An Lin, Ben Weissmann, Manolis Savva, Angel X Chang. and Daniel Ritchic. Planit: Planning and instantiating indoor scenes with relation graph and spatial prior networks. ACM Transactions on Graphicx (TOG), 38(4):1-15, 2019.
[61]
Kai Wang, Manolis Savva, Angel Chang. and Daniel Ritchie. Deep convolutional priors indoor scene synthesis. ACM Transactions on Graphics (TOG), 37(4):70, 2018.
[62]
Xiaolong Wang and Abhinav Gupta. Generative image modeling using style and structure adversarial networks. In ECCV, 2016.
[63]
Xinpeng Wang. Chandan Yeshwanth, and Matthias Nießner. Sceneformer: Indoor scene generation with
transformers. In 3DV, 2021. 3
[64]
Shun-Cheng Wu, Johamna Wald, Keisuke Tateno, Nassir Navah, and Federico Tombari. Scenegraphfusion:
Incremental 3d scene graph prediction from rgb-d sequences. In CVPR, 2021.
[65]
Jianghao Xiong, En-Lin Hsiang, Ziqian He, Tao Zhan, and Shin-T'son Wu. Augmented reality and virtual reality displays: emerging technologics and future perspectives. Light: Science & Applications, 10(1):216, 2021.
[66]
Guandao Yang, Xun Huang. Zekun Hao, Ming- Yu Liu, Serge Belongie, and Bharath Hariharan. Pointflow:
point cloud generation with continuous normalizing flows. ICCV, 2019.
[67]
Ling Yang. Zhilin Huang, Yang Song, Shenda Hong, Guohao Li, Wentao Zhang, Bin Cui, Bernard Ghanem, and Ming-Hsure Yang. Diffusion- basiid scene graph to image generation with masked contrastive pre-training, 2022. 3
[68]
Guangyao Zhai, Xiaoni Cai, Dianye Huang, Yan Di, Fabian Manhardt, Federico Tombari, Nassir Navab, and Benjamin Busam. Sg-bot: Object rearrangement via coarse-to-fine to-fine robotic imagination on scene
graphs, 2022. 1.
Chenyangguang Zhang. Zhiqiang Lou, Yan Di, Federico Tombari, and Xiangyang Ji. Sst: Real-time end-to-end monocular 3d reconstruction via sparse patialltemporal guidance. In ICME, 2023. 3 [70] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In ICCV, 2023.
[69]
[71]
Bo Zhao, Lili Meng. Weidong Yin, and Leonid Sigal. Image generation from layout. In CVPR, 2019. 3
[72] Yizhou Zhao, Kaixiang Lin. Zhiwei Jia. Qiaozi Gao, Govind Thattai, Jesse Thomason, and Gaurav S. Sukhatme. Luminous: Indoor seene generation for embodied ai challenges. In NeurlPS Workshop, 2021.
[73] Yang Zhou, Zachary While, and Evangelos Kalogerakis. Scenegraphnet: Neural message passing for 3d indoor scene augmentation. In ICCV, 2019.
[74] Ege Özsoy, Tobias Czempiel, Evin Pinar Örnek, Ulrich Eck, Federico Tombari, and Nassir Navab. Holistic or domain modeling: a semantic scene graph approach. International Journal of Computer Assisted Radiology and Surgery, pages 1-9, 2023. 3
[75] Ege Ozsoy, Evin Pinar Örnek, Ulrich Eck, Tobias Czempiel, Federico Tombari, and Nassir Navab, 4d-or: Semantic scene graphs for or domain modeling. In MICCAI, 2022. 3
13


Reviews:
[{"review_id": "KFCZlJXyFS", "reviewer": "Reviewer_Wi3K", "summary": "Gist:\nThe paper presents a framework, called CommonScenes, for generating 3D indoor scenes given scene graphs as inputs. CommonScenes is a dual-branch framework where one branch generates the scene layout using a VAE and the second one generates what the authors call \"compatible\" 3D shapes using latent diffusion. The claim is that having this second branch (where compatible 3D shapes are generated for populating the generated layout from the first branch) allows capturing global scene-object and local inter-object relationships, something that prior works cannot capture (I am not convinced about this, but more on this later). \n\nThe generated scenes can be manipulated by editing the input scene graph, as well as sampling noise in the diffusion process. \n\nThe paper also constructs a scene graph dataset using an off-the-shelf 3D scene dataset.\n\nDataset Used:\n3D-FRONT is the base dataset used, which is augmented with scene graph labels and this augmented dataset is termed in the paper as \"SG-FRONT\" dataset.\n\nTraining Mechanism:\nSupervised in the form of triplet network setting and latent diffusion models\n\n\nEvaluation Metrics:\nTo measure the fidelity and diversity of generated scenes, FID, KID scores at 256x256 pixel resolution between the top-down rendering of the generated and real scenes is used.\n\nTo measure shape diversity, each scene is generated 10 times, and the changes in shapes is evaluated using CD.\n\n\nBaselines and Comparisons:\nThree kinds of baselines are compared against:\n\n1) First, a retireval-based method, namely, 3D-SLN from CVPR 2020\n\n2) Second, a semi-generative SOTA method, Graph-to-3D from ICCV 2021, and\n\n3) Third, a text-to-shape generation methods that follows layout generation (this is not cited, so I am not informed by the paper if this was implemented by the authors on their own or if any specific algorithm was re-implemented)\n\n", "questions": "Please see the Weaknesses section above", "limitations": "The paper does not discuss the limitations of the proposed approach.\n\nThere exist many questions (please see the Weaknesses section above) that can critically limit the application of the proposed approach, starting from the way the input scene graphs are obtained. At the least, a discussion on how this work can address or alleviate such challenges using additional processing should have been discussed.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "+ Conceptualizing layout generation using graphs is a nice concept, although this is not the first time it has been addressed. A structured input modality gives rise to many applications, such as scene editing and modification, as demonstrated in the paper.\n\n+ As seen from Figure 4, the proposed method seems to produce plausible outputs given a scene graph as input. This is also validated quantitatively, although I would only consider Table 1 to be more representative of such quantification than other tables.", "weaknesses": "- Not really a concern but this is something that people will find about this paper: the paper is trying to do too many things at once. While this may also be a positive aspect in the era of today's models, a reader cannot clearly discern what design aspect leads to layout improvement and what leads to shape improvements. One may even argue: why not use the shape generation scheme employed as an independent approach and submit a paper if enough novelty exists? You get my point.\n\n\n- L96-97: How are the edge predicates (like spatial relations \"left/right\", \"closeby\" etc.) obtained? Is the dataset manually annotated with semantic scene graph information? If that is the case, then, the problem formulation is weak. What would have been interesting is to automatically extract meaningful semantic scene graphs (especially that ground spatial relations to a reasonable extent) and then use these graphs to generate a 3D scene.\n\n- L3-6: I do not understand the message in these lines. Do you mean to say that existing methods use retrieval-based mechanisms to populate the generated layout, because of which scene-object and object-object relationships are inconsistent? It is not true. So, first, I think it is important to rephrase this sentence. It is conveying an altogether different meaning.\n\n- There is mention of a triplet graph network, triplet-GCN in L46, 129, 148, 159. However, there is not much detail about how the positive and negative examples to train this triplet network are obtained. This is quite important since the training data plays a key role in obtaining meaningful and discriminative embedding spaces in the context of contrastive learning setups.\n\n- I fail to understand why the initial node embeddings, c_i^{node}, which are obtained from one-hot semantic encodings (as per info in L93), should be passed through a pre-trained and frozen CLIP text encoder. It makes sense to pass the initial __edge__ embeddings through it as spatial information needs to be captured and the CLIP text encoder does a good job of mapping the initial English text to a meaningful embedding space. But I cannot understand why the node embeddings c_i^{node} need to be passed through the text encoder from CLIP.\n\n- L37: It is not trivial to obtain text information from input scene graphs. This alternative solution is not so straightforward, unlike what is mentioned in this line.\n\n- L65-66, the sentence is incomplete\n\n- L87-88: LEGO-Net from CVPR 2023 is the first work to leverage diffusion models for scene generation. Even though LEGO-Net is designed for scene rearrangement, I would still place it in the generative model category since it is inherently doing denoising to provide a plausible output. So, these two lines are not correct.\n\n- Figure 4: Again, my question is how are the input scene graphs obtained? I am interested in knowing how the spatial relations on the graph edges are obtained. If this is done in a heuristic manner, it is prone to errors, and I do not think this is trivial to obtain in the presence of multiple objects. Obtaining such spatial relations is a challenge, as widely acknowledged in the community (see pre-LLM works on 3D indoor scene generation using language/text input, such as from Chang 2014, 2015, Ma 2018).\n\n- There are quite a few typos and punctuation errors in the paper (one such example is pointed out below). Need to be corrected.\nOne Missing period/full-stop symbol in line 24.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## 1. Text-to-shape baseline\nWe have explained this baseline, named Layout+txt2shape\", more in detail in Supp. Mat. Section 7. This baseline solely considers the shape generation from text input, which we establish through the text-to-shape SDFusion [8]. \n## 2. Complexity, contribution, and motivation\nNotably, our main contribution is the proposal of a method to enable a fully generative model of the entire 3D scene from scene graphs, encompassing its layout and 3D geometries holistically. In our experiments, we systematically show that using only layout and retrieval-based shapes (3D-SLN [31]) underperforms and that a semi-generative shape model of Graph-to-3D [13] also fares poorly (p.8, Table 1). We show that it is possible to generate shapes together with scene layouts in the proposed framework. Contextual information improves the coarse inter-object relationships (p. 9, Table 5 row1 vs. row 4), and the GCN further propagates information among objects and learning global cues, and finer local-object relationships (p.9, Table 5 row 2 vs. row 4). Besides, the optimization from the diffusion-based shape branch not only assists layout generation during training but also brings better shape generation quality than previous work (p. 7 Figure 4).\n## 3. Dataset annotation and scene graph sources\nWe obtain the edge predicates (e.g., \"left/right\", \"front/behind\", “above/standing on'') in a semi-automatic manner, similar to prior work (3DSSG [53], 4D-OR [66]), as explained in the main paper L190-192, and in Supp. Mat. Section 3 L75-78. Essentially, we use three methods: 1) For spatial relations (e.g., \"left/right\", \"bigger/smaller\"), we initiate the process by applying relationship checks on the bounding box extents, followed by collision checks; 2) For support relations (e.g., \"standing on\", \"above\"), we apply a set of thresholds for each object type identified by human annotators; 3) For stylistic cues, we refer to 3D-FRONT object annotations [15] to identify meaningful information, generating semantic edge labels. Since we use the synthetic and high-quality 3D-FRONT [15], any potential errors occur systematically, in contrast to real scenes obtained by depth cameras [66,67]. These can then be easily corrected. \n\nThe input scene graphs in Figure 4 are the annotated scene graphs from the test scenes of 3D-FRONT. For general acquisition of scene graphs, there are various ways, such as from text (Chang et al. [68, 69], Ma et al. [32]), single image (Chen et al. [70], Dhamo et al. [12]), or video (Wu et al. [58]). However, our work aligns with 3D-SLN [31] and Graph-to-3D [13] in using the scene graph to enable controllable scene analysis. Therefore, our scene graphs are obtained in a manner akin to previous approaches [13]. We recognize the challenge in this task and have added the related work [68,69], to our paper, further discussing this aspect in the Supp. Mat. \n## 4. The explanation and methodology of retrieval-based methods\nRetrieval-based methods, like 3D-SLN [31], Graph-to-Box [13] and ATISS [39], typically determine retrievals based on the sizes of bounding boxes, as shown in Figure 1. In this case, even a slight variation in the estimated bounding box size can lead to significant and undesired variations in the generated shapes, resulting in the generation of inconsistent scenes (p.8, Table 2). In contrast, our generated shapes and appearances are conditioned on the relations among the objects. This leads to more consistent shape generation (p.8 Figure 5, L244-248). \n\n## 5. The meaning of triplet-GCN\nIn this context, the triplet graph network uses a triplet of “subject-predicate-object” [31]. We don’t use any contrastive learning or triplet loss. We will clarify this aspect for readers more familiar with triplet loss concepts. \n\n## 6. CLIP embeddings for graph nodes\nWe do not pass one-hot embeddings to the CLIP text encoder. Instead, we input the object class names for nodes (e.g., “Bed” and “Table”), as well as the edge class names (e.g., “left”, “right”) as illustrated in Figure 2, exactly because of the intuition explained by the reviewer. \n\n## 5. Text information acquisition in the scene graph\nIn L36-39, we explained that simply replacing the shape retrieval baseline with a text-to-shape generator, where the text is acquired from the scene graph, does not yield good results (i.e., the baseline \"Layout+txt2shape\"). **The textual input from the scene graph is obtained by means of class names**. We will add a cross-reference to this experiment on L39 to prevent misunderstanding. \n\n## 8. LEGO-Net from CVPR 2023\nWe thank the reviewer for pointing out the very recent work [71]. We will cite this work and refine our statement in related work accordingly.\n\n## 9. Limitation Discussion\nWe discussed the limitations in the Supp. Mat., on Section Discussion and Limitations L131-149. We will carry the essential parts to the main manuscript.\n\n[67] Wald et al. \"Rio: 3d object instance re-localization in changing indoor environments.\" ICCV 2019.\n\n[68] Chang et al. \"Learning spatial knowledge for text to 3D scene generation.\" EMNLP 2014.\n\n[69] Chang et al. \"Text to 3D Scene Generation with Rich Lexical Grounding.\" ACL 2015.\n\n[70] Chen et al. \"Scene graph prediction with limited labels.\" ICCV 2019.\n\n[71] Qiuhong et al. \"LEGO-Net: Learning Regular Rearrangements of Objects in Rooms\". CVPR 2023."}, {"review_id": "FWc7O9Jc62", "reviewer": "Reviewer_ANPx", "summary": "This paper addresses the task of controllable scene synthesis of indoor rooms, conditioned on a semantic scene graph that captures spatial, style and support relationships between objects in a scene. In particular, they introduce CommonScenes, a generative model capable of converting scene graphs into 3D scenes using diffusion models. Given a scene graph of the scene to be generated, they first enhance it using CLIP features computed on the inter-object relationships, as well as using the ground-truth bounding box annotations. Next, they leverage a triplet-GCN based relation network to propagate information among the objects in the scene and produce the Box-enhanced Contextual Graph (BCG). Given a BCG, they then utilize a dual-branch network that generates the final scene. The first branch (layout branch) is another triplet-GCN that generates 3D layout predictions and the second branch (shape branch) generates shapes for each node represented as SDF. The shape branch is simply a latent diffusion model. During training, CommonScenes considers supervision in the form of per-object bounding-box annotations (size, location, rotation), as well as per-object 3D meshes in the form of SDFs. To evaluate their model, the authors enrich the 3D-FRONT dataset with scene graph labels, by annotating per-object spatial, style and support relationships. The authors compare their model to several baselines and show that their approach yields more realistic scenes in terms of FID and KID scores.\n\nOverall, I think this is a nice work that alleviates the need for relying on a library of assets to replace the generated bounding boxes with 3D objects. The proposed architecture is novel and seems to be able to consistently produce plausible scenes. That being said I think that the proposed pipeline is quite complex, as it consists of multiple submodules, and it requires conditioning in the form of semantic graphs that are relatively hard to acquire, hence the authors had to enrich 3D-FRONT. However, since the authors show that their model outperforms prior research I am in favor of accepting this paper.\n", "questions": "1. For the Ablations in Sec 6.3, for the model variant that is trained without context, do the authors omit both the CLIP features, as well as the bound-box features or only the former. I think that they should be ablated separately. Personally, I am not 100% convinced about the significance of the CLIP features. Can the authors please clarify?\n\n2. I am wondering whether the authors tried to condition their scene generation also on the floor plan together with the semantic scene graph? Unless I am missing out something, this can be easily done e.g. by extracting features from the floor plan and concatenating with the per-object features while created the BCG? Do the authors think that something as simple as that would work?\n\n3. One thing is that a bit unclear to me is whether the first component of the pipeline that generates the BCG using the triplet-GCN is really necessary. Have the authors tried to directly use the scene graph enhanced with the CLIP features and the bounding box features? This might be an interesting ablation to provide for the final version of the paper.\n\n4. One concurrent work that I think the authors should add in their reference list is the Learning 3D Scene Priors with 2D Supervision, CVPR 2023. I think this work could also be an interesting baseline as they can generate both the scene layout as well as the 3D shapes\n\n5. A minor comment/suggestion: In the caption of Figure 1 in the supplementary material the authors state \"shows a huge diversity\". I think huge might be slightly overclaiming. It might be better to tone it down a bit.\n", "limitations": "The authors discuss the limitations of their work and the potential negative impacts in the society in their supplementary material. I think they could have also provided some qualitative examples of the failure case of their model to help the reader better understand the limits of this work.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. To the best of my knowledge, the proposed model is novel and the authors clearly demonstrate that their model consistently produces plausible scenes. I think that this work is an important step towards alleviating the need for large libraries of assets when generating a novel scenes. Relying on such libraries naturally restricts the diversity of the generated scenes to the diversity of the objects in the library. Although, the proposed model is a relatively complex, I believe it is a valuable work that could potential inspire other works in this direction.\n\n2. I think that the development of the SG-FRONT dataset, that extends 3D-FRONT with scene graph labels is an important contribution of this work that can potentially facilitate other research projects, therefore I strongly encourage the authors to make this dataset variant available upon the paper's acceptance.\n\n3. I really appreciated the supplementary video that the authors provided. I particularly liked the intuitive explanation of the proposed method as well as the additional results. In the future, I hope that more authors will provide such high quality supplementary videos alongside with the paper submission.\n\n4. Although the proposed model is quite complex, I think the paper is nicely written and easy to follow. I really liked the provided figures that provide a more high-level pictorial overview of the various components of the proposed pipeline.\n", "weaknesses": "1. The main weakness of this work is that it is relatively complex, as it consists of multiple sub-modules. One thing that was not 100% clear from the text was whether they perform a two-stage training, namely first produce the BCG, as discussed in Sec 4.1 and then jointly train the shape and layout branch of CommonScenes, as discussed in Sec 4.2. It might be good to clarify this for the final version of the paper. In addition, for the Shape Decoding module (L164-179), I am not 100% sure whether the authors use some sort of class conditioning to train one LDM for all object types or they use separately trained LDMs per-object class. This might be good to clarify.\n\n2. Although, the paper focuses on scene synthesis, I think that an important evaluation that is missing is measuring the quality of the generated shapes. In particular, the authors could generate a couple of rooms and then take the beds, chairs, nightstands etc. and compute the COV and MMD of the generated objects w.r.t the ground-truth objects. Although, object generation is not the main focus of this work, I think this analysis would be valuable, since this is an integral component of the proposed model. I noticed that the authors tried to do this type of analysis in Sec. 1.1 in their supplementary, but I found it a bit weird that they report Chamfer Distance, instead of MMD/COV. Is there any reason for this?\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## 1. Training details\nWe perform one-stage training. As mentioned in the L129, the BCG is created and encoded on-the-fly by the contextual encoder $E_c$. We agree that the overview needs to indicate one-stage training clearly. As for LDM, we train a single LDM for all object types conditioned on the learned relation embeddings. We will clarify both aspects in the final version.\n## 2. More object-level evaluation\nSince our objective is scene generation, we use FID/KID as the main metrics for evaluating the scene-level generation quality. **On the other hand, we reported the CD as a metric to evaluate generation diversity, following the previous state-of-the-art method Graph-to-3D to enable a fair comparison**  (more details in Supp. Mat. L16-17). \nAs requested, we report the MMD (x0.01) and COV (%) for evaluating per-object generation. We collect ground truth objects in each category within the test set and use the evaluation script from PointFlow [72].\n\n**Table 1.** MMD(↓) comparison\n\n|      Method     |   Bed  | Nightstand | Wardrobe |  Chair |  Table | Cabinet |  Lamp  |  Shelf |  Sofa  | TV stand |\n|:---------------:|:------:|:----------:|:--------:|:------:|:------:|:-------:|:------:|:------:|:------:|:--------:|\n| Graph-to-3D | 1.56 | 3.91     | 1.66   | 2.68 | 5.77 | 3.67  | 6.53 | 6.66 | 1.30 | 1.08 |\n| Ours    | **0.49** | **0.92**     | **0.54**   | **0.99** | **1.91** | **0.96**  | **1.50** | **2.73** | **0.57** | **0.29** |\n\n**Table 2.** COV(↑) comparison\n\n|      Method     |   Bed  | Nightstand | Wardrobe |  Chair |  Table | Cabinet |  Lamp  |  Shelf |  Sofa  | TV stand |\n|:---------------:|:------:|:----------:|:--------:|:------:|:------:|:-------:|:------:|:------:|:------:|:--------:|\n| Graph-to-3D | 4.32 | 1.42 | 5.04   | 6.90 | 6.03 | 3.45  | 2.59 | 13.33 | 0.86 | 1.86 |\n| Ours    | **24.07** | **24.17** | **26.62** | **26.72** | **40.52** | **28.45**  | **36.21** | **40.00** | **28.45** | **33.62** |\n\nAs shown in Tables 1 and 2, our method shows better performance in both MMD and COV, which highlights the object-level shape generation ability of CommonScenes.\nWe also calculate 1-nearest neighbor accuracy (1-NNA, %), which directly measures distributional similarity on diversity and quality. This measurement has motivated us to include it in this rebuttal as well for reviewers’ and readers’ reference. The closer the 1-NNA is to 50%, the better the shape distribution is captured. \n\n**Table 3.** 1-NNA(↓) comparison\n\n|      Method      |     Bed    | Nightstand |  Wardrobe  |    Chair   |    Table   |   Cabinet  |    Lamp    |    Shelf   |    Sofa    |  TV stand  |\n|:----------------:|:----------:|:----------:|:----------:|:----------:|:----------:|:----------:|:----------:|:----------:|:----------:|:----------:|\n| Graph-to-3D   | 98.15     | 99.76     | 98.20     | 97.84     | 98.28     | 98.71     | 99.14     | 93.33     | 99.14     | 99.57     |\n| Ours | **85.49** | **95.26** | **88.13** | **86.21** | **75.00** | **80.17** | **71.55** | **66.67** | **85.34** | **78.88** |\n\nIt can be observed that our method surpasses Graph-to-3D in the evaluation of distributional similarity. Coupled with the results in Tables 1 and 2, CommonScenes exhibits more plausible object-level generation than the previous state-of-the-art. We will extend our Supp. Mat. with these additional experiments. Please also check our summarized table in the attached PDF.\n## 3. Ablations on the CLIP features\nExactly as the reviewer indicated, we ablate the CLIP features only in Section 6.3, *“Ours w/o context”*. By omitting the features, the BCG degrades to a box-enhanced scene graph, lacking strong semantic cues to guide the shape and consistent layout generation. Table 5 shows this is vital for the quality of 3D scene generation results, both in terms of synthesis quality (FID/KID) and the relation correctness (mSG). Additionally, compared to layout generation in Tables 1, 3, and 4, our method without a shape branch (*\"Ours w/o SB\"*) highlights the effectiveness of BCG. On the other hand, the ground truth bounding box parameterization process cannot be ablated since it is essential to the VAE modeling, which also serves as supervision labels for the layout. \n## 4. BCG encoding\nWe did not use the triplet-GCN to \"generate\" a BCG, Instead the BCG is encoded by the triplet-GCN-based $E_c$ during the training, and we actually followed the suggested approach as mentioned in L128-131. For even more clarity, we will rephrase the relevant part in the final version. \n## 5. Floor plan involvement\nNo, we have not tried that. Our motivation is to use as simple as possible conditions, i.e., only objects and their relationships, to condition a scene generation. However, the approach mentioned by the reviewer may mean removing the floor node since it would be used in enhancing the per-object features, i.e., by concatenation. Instead, replacing the floor feature from CLIP with a floor plan embedding and treating it as a node can potentially generate user-defined floor shapes. We will investigate this direction as our further work.\n## 6. A related paper from CVPR 2023\nIndeed this is a relevant work that learns 3D shape priors from 2D images and generates shapes and layouts from a hypersphere space. We now cite this paper published after our work's submission. However, it is worth noting that this method is trained and conditioned on RGB-related information, e.g., 2D bounding boxes and masks. Therefore, a direct comparison between our method and this would be inappropriate since our inputs are in different modalities, where we use the form of a scene graph.\n## 7. Qualitatives for limitations\nWe will append Supp. Mat. Section 6 Discussion and Limitations with illustrations. We provide some examples of these in the attached PDF. \n\n[72] Yang et al. \"Pointflow: 3d point cloud generation with continuous normalizing flows.\" ICCV 2019.\n\n[73] Nie et al. \"Learning 3D Scene Priors with 2D Supervision\". CVPR 2023."}, {"review_id": "r2l0zMrFaG", "reviewer": "Reviewer_NT6V", "summary": "Summary: This paper presents a generative model to generate 3D scenes from scene graphs. Their model is fully generative without the need of any shape database or embeddings. The 3D scenes generation model is pipelined into finding the scene layout and the construction the shapes of the nodes using a diffusion model. The authors constructed a scene graph dataset SG-FRONT from 3D-FRONT dataset and trained end to end to generate 3D scenes from the scene graph input. The paper reported qualitative, quantitative results with comparison to other sota models on the SG-FRONT dataset.", "questions": "1. In section 6, Compared Baselines, 'a fully generative method a text-to-shape generation model that follows a layout generation' does not have citations. Is it something the authors modeled for experimentation? ", "limitations": "no negative societal impact.\nNo limitation addressed.\nAll the 3D scene generation is done on synthetic dataset. How would the whole model perform on the real scene? ", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. the paper claims to have curated a dataset(enriching an existing dataset with scene graph annotations) for 3D scenes construction from scene graphs. \n2. Codes and dataset will be publicly available\n3. The generated 3D scenes and the quantitative numbers show potential of the scene graph based approach for 3D scenes reconstruction using a diffusion model", "weaknesses": "1. In section 6, Compared Baselines, 'a fully generative method a text-to-shape generation model that follows a layout generation' does not have citations. Is it something the authors modeled for experimentation? \n2. In Table 1, need to specify which of the methods use fully generative approach with text only. Also, calrifying in the caption of table 1 on how you segmented table 1 into two main rows and what is 'Ours 'w/o SB''\n3. Sharing more details on how the authors curated the dataset from 3D-FRONT in the main paper might be helpful for the readers\n4. Some possible typos:\nline 47 : 'cues and fine local inter-object relationships.'\nLine 65 and 66: Incomplete lines : 'Quickly after this progress, the 3D [53, 1, 58, 26], dynamic [44], robotic\n66 grounding [19, 45], spatio-temporal 4D [66], and controllable scene synthesis [29, 65, 54, 36, 13].'", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## 1. The baseline based on the text-to-shape model\nIt is a method we modeled for experimentation. The motivation is introduced in L36-40, the main paper. We have also further explained the implementation of this baseline in Supp. Mat. Section 7 in detail. This baseline consists of a layout generator (the same as ours) and a single text-to-shape generator (SDFusion [8]). Given a scene graph describing the target scene, this baseline uses the layout branch to generate the bounding boxes, while concurrently feeding the individual category name of each node in the scene graph into the text-to-shape generator to generate object shapes, e.g., using the word “bed” with SDFusion to generate a plausible bed. Finally, the entire scene is synthesized by populating each shape within its corresponding sounding box. Here, we train the layout branch using the same settings as ours and train the text-to-shape generator following SDFusion. \n## 2. Clarification on Table 1\nAmong the baselines evaluated, only “Layout+txt2shape” is the “a fully generative approach with text only”. Two main rows are separated with respect to the reliance on an external shape database for retrieval. “Ours w/o SB” refers to ours without the shape branch. We will clarify these aspects in the main paper to ensure that readers can fully understand the distinctions.\n## 3. More dataset collection details in the main paper\nWe plan to extend Section 3 of the main manuscript through the details from Supp. Mat. Section 3 Dataset Details (L82-86) upon acceptance. In core, we adopt a semi-automatic approach for detecting spatial layout through bounding box extensions followed by human annotator inspection (as in prior work 3DSSG [53], 4D-OR [66]). For annotating semantic-level edges, we resort to 3D-FRONT [15] annotations where we extract the object-level features (e.g., “same material as”, “same style as”).\n## 4. Limitations, societal impact, and real-world performance\nIn the Supp. Mat. Sec. 6 Discussion and Limitations, we addressed the negative societal impact (L145-149) as well as the limitations (L132-144). We plan to carry the essential parts to the main manuscript.\nFurthermore, for real-scene performance, we also provided a discussion and analysis in Supp. Mat. Section 4 Results on 3DSSG dataset [53]."}, {"review_id": "FDgsWIL4L2", "reviewer": "Reviewer_zttF", "summary": "The paper looks at the problem of learning a generative model for sampling 3D environments from a description based on a graph and natural language. The graph does correspond to objects that participate in the scene, and textual descriptions attached to nodes and edges further specify the scene. Textual descriptions are encoded using CLIP. A variational auto encoder conditioned on these inputs is trained to generate the bounding boxes of the 3D objects. A latent diffusion model is trained in parallel to generate the 3D shape of the object in each bounding box. The existing FRONT-3D dataset is augmented with textual descriptions for training this model. Experiments show the advantages and limitations compared to ablations and prior works.", "questions": "* Please clarify how the vectors o in Figure 2 and line 99 are obtained. The paper only states that they are \"learnable\".\n\n* The two shape and layout branches are said to be trained jointly (Section 4.4). However, the learning dynamics of auto encoder and diffusion processes (one per object) seem to me to be very different. Is it really that trivial to mix them?\n\n* Line 172: the shape diffusion model connects to the *entire* graph, and so to all objects, via cross-attention. How does each instantiation of the diffusion process know *which* of the several objects it is meant to generate?\n\n* Line 214: I don't know what it means to share the same CAD model source. Nor I do understand why it is is sensible to compare a generated sample to a specific CAD instance -- I would not expect the two to match.\n\n\n", "limitations": "There is sufficient discussion of limitations, but no discussion of ethics. The latter, however, is not particularly relevant for this paper. It may be worth at least discussing the licensing of 3D-FRONT.\n", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "* Scene generation is an important problem and this paper addresses it with a responsible model using modern components in a competent manner.\n\n* The paper contributes a large amount of additional labels for 3D-FRONT.", "weaknesses": "* The  presentation, especially of the technical parts, can be improved. It was difficult to understand how the layout auto-encoder is setup. It would be useful to give a high-level overview first of where the method is trying to achieve (e.g., encoder Z = E(P, O, B), decoder \\hat B = E(P, O, Z), Z Gaussian). Section 4.1 presents only half of the story, and we need to get to section 4.2 before the auto-encoder materialises. Line 143 literally states the the encoder is trained by minimising Eq. (4), which is clearly not enough -- Eq. (5) is also needed.\n\n* You man need to revisit the formalism at Ines 90 to 99. $c_i^{node}$ and the same terms for the edges are very poorly defined. Furthermore, it also seems that you can attach more than one such attributes to each node or edge. The formalism does not allow for that.\n\n* Some aspects of the model are unclear. See questions below.\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## 1. Overview paragraph on optimization\nThanks for the useful advice! We now extend our overview section (L111-116) by explicitly cross-referencing figures and the subsections. Further, we rephrase the sentence in L143 as: *“We guide the distribution of embedding space through employing the KL loss.”*. Note that in Section 4.4, we already explained that we use the combination of all these losses to optimize the network, but we will clarify the necessity of this up-front by cross reference.\n## 2. The formalism in the preliminary\nWe have fixed L93 thusly to clarify the formalism:\n*Each vertex $v_{i}$ is categorized  through an object class $c^{node}_{i} \\in \\mathcal{C}^{node}$, where $\\mathcal{C}^{node}$ denotes the set of object classes.* Note that in terms of notations, we follow the prior work [58].\n## 3. Clarification about vectors $o$\nThe vectors $o$ in Figure 2 and L99 are learnable object embeddings, where each object class is identified with one fixed-size embedding initiated randomly and optimized over the training. It is in the same vein as how Large Language Models use learnable tokens per word in a dictionary.\n## 4. Joint training of two branches\nJoint training can be achieved. We train the layout VAE and the latent shape-diffusion branches together with the combination of losses (Eq. 7). It was indeed not trivial to mix them, which is why we discussed potential implications in the Supp. Mat. (L155-170), and we ensured synchronized training between the two branches with a uniform sampling (Supp. Mat. Figure 9).\n## 5. Cross-attention feature conditioning\nThe shape branch is conditioned on per-node relation embedding via cross-attention to generate per-node shapes, not the entire graph. We then run the diffusion for every node in the graph to populate the scene. We will clarify this in the final version.\n## 6. CAD model comparison\nThe intention is to measure the generation consistency in the dining rooms, where dining chairs usually appear together in a suit and dining tables as well (p.8 Figure 8). To collect the consistency ground truth, for example, in one dining room, we collect the chairs that are decorated using the same textured CAD model. This shows us which chairs should be in a suit after the generation. We then calculate the CD between these chairs, the smaller the CD the better. This metric is mainly used to test object-object consistency (p.8 Table 2). \n## 7. License discussion\nWe have discussed the ethics in Supp. Mat. L145-149. In terms of licensing, we will follow CC BY-NC-SA."}]

Author rebuttal:
# Thank you for your insightful comments!
We would like to thank all reviewers for their insightful and valuable comments. In summary, they highlighted the significance of the work (*“scene generation is an important problem”* (xttF), *“structured input modality gives rise to many applications”* (wi3K), and *“this is a nice work”* (ANPx)), indicated that *“architecture is novel”* (ANPx), *“this paper addresses it with a responsible model using modern components in a competent manner”* (xttF), *“the generated scenes seem to produce plausible outputs”* (wi3K). They recognized the value of SG-FRONT *“enriching an existing dataset with scene graph annotations”* (NT6V), *“large amount of additional labels for 3D-FRONT”* (xttF), and acknowledged our code and dataset publicity (NT6V). Furthermore, they appreciated our supplementary video, *“particularly liked the intuitive explanation”* (ANPx). 

This rebuttal addresses each reviewer’s concerns. **We also appreciate the pointers for the grammar errors and typos, which we have since corrected. We further attach a PDF document in this rebuttal for reference.**

Meta-review: The work received primarily positive reviews, with many reviewers recommending acceptance. All reviewers acknowledged the contributions made by the work in alleviating the need for large libraries of 3D assets. Several important questions were raised during the review period, including ablations, limitations, and claims made in the paper. The rebuttal seems to have addressed many of these concerns effectively. The authors are highly encouraged to incorporate the results and discussions from the rebuttal into the final version for completeness.
Decision: Accept (poster)