Title: CommonScenes: Generating Commonsense 3D Indoor Scenes with Scene Graph Diffusion
Venue: NeurIPS 2023
Accepted: yes
Keywords: Scene Graph

Scene Synthesis

Diffusion Model

Graph Convolution Network
Abstract: Controllable scene synthesis aims to create interactive environments for numerous industrial use cases. Scene graphs provide a highly suitable interface to facilitate these applications by abstracting the scene context in a compact manner. Existing methods, reliant on retrieval from extensive databases or pre-trained shape embeddings, often overlook scene-object and object-object relationships, leading to inconsistent results due to their limited generation capacity. To address this issue, we present CommonScenes, a fully generative model that converts scene graphs into corresponding controllable 3D scenes, which are semantically realistic and conform to commonsense. Our pipeline consists of two branches, one predicting the overall scene layout via a variational auto-encoder and the other generating compatible shapes via latent diffusion, capturing global scene-object and local inter-object relationships in the scene graph while preserving shape diversity. The generated scenes can be manipulated by editing the input scene graph and sampling the noise in the diffusion model. Due to the lack of a scene graph dataset offering high-quality object-level meshes with relations, we also construct SG-FRONT, enriching the off-the-shelf indoor dataset 3D-FRONT with additional scene graph labels. Extensive experiments are conducted on SG-FRONT, where CommonScenes shows clear advantages over other methods regarding generation consistency, quality, and diversity. Codes and the dataset are available on the website.

CommonScenes: Generating Commonsense 3D Indoor Scenes with Scene Graph Diffusion

Abstract

Controllable scene synthesis aims to create interactive environments for numerous industrial use cases. Scene graphs provide a highly suitable interface to facilitate these applications by abstracting the scene context in a compact manner. Existing methods, reliant on retrieval from extensive databases or pre-trained shape embeddings, often overlook scene-object and object-object relationships, leading to inconsistent results due to their limited generation capacity. To address this issue, we present CommonScenes , a fully generative model that converts scene graphs into corresponding controllable 3D scenes, which are semantically realistic and conform to commonsense. Our pipeline consists of two branches, one predicting the overall scene layout via a variational auto-encoder and the other generating compatible shapes via latent diffusion, capturing global scene-object and local inter-object relationships in the scene graph while preserving shape diversity. The generated scenes can be manipulated by editing the input scene graph and sampling the noise in the diffusion model. Due to the lack of a scene graph dataset offering highquality object-level meshes with relations, we also construct SG-FRONT , enriching the off-the-shelf indoor dataset 3D-FRONT with additional scene graph labels. Extensive experiments are conducted on SG-FRONT, where CommonScenes shows clear advantages over other methods regarding generation consistency, quality, and diversity. Codes and the dataset are available on the website.

1 Introduction

C ontrollable S cene S ynthesis ( CSS ) refers to the process of generating or synthesizing scenes in a way that allows for specific entities of the scene to be controlled or manipulated. Existing methods operate on images [62] or 3D scenes [34] varying by controlling mechanisms from input scene graphs [24] or text prompts [5]. Along with the development of deep learning techniques, CSS demonstrates great potential in applications like the film and video game industry [7], augmented and virtual reality [65], and robotics [72, 68]. For these applications, scene graphs provide a powerful tool to abstract scene content, including scene context and object relationships.

This paper investigates scene graph-based CSS for generating coherent 3D scenes characterized by layouts and object shapes consistent with the input scene graph. To achieve this goal, recent methods propose two lines of solutions. The first line of works optimizes scene layouts [35] and retrieves objects [16] from a given database (see Figure 1 (a)). Such retrieval-based approaches are

Chair

Table

···

Bed

Layout

CVAE

Layouts

Scenes

a) Retrieval

Layouts

Scenes

Codes

Code Database

Chair

Table

···

Bed

Decoders × n

···

···

···

Code

CVAE

Layout

CVAE

Latent Diffusion Model

Unified

CVAE

Decoder

×

1

Scenes

Layouts

b) Semi-Generative

c) Generative (Ours)

No Database

···

···

···

inherently sub-optimal [57] in generation quality due to performance limitations through the size of the database. The second solution, e.g., Graph-to-3D [12], regresses both the layout and shape of the objects for synthesis. However, the shape generation relies on pre-trained shape codes from category-wise auto-decoders, e.g., DeepSDF [42]. This semi-generative design (Figure 1 (b)) results in reduced shape diversity in the generated outputs. To enhance generation diversity without relying on vast databases, one possible solution is to concurrently predict scene layouts and generate object shapes with text-driven 3D diffusion models[8, 32], where the textual information is obtained from input scene graphs. Yet, in our experiments, we observe that such an intuitive algorithm works poorly since it does not exploit global and local relationship cues among objects encompassed by the graph.

In this work, our approach CommonScenes exploits global and local scene-object relationships and demonstrates that a fully generative approach can effectively encapsulate and generate plausible 3D scenes without prior databases. Given a scene graph, during training, we first enhance it with pretrained visual-language model features, e.g., CLIP [46], and bounding box embeddings, incorporating coarse local inter-object relationships into the feature of each node. Then, we leverage a tripletGCN [35] based framework to propagate information among objects, learning layouts through global cues and fine local inter-object relationships, which condition the diffusion process [47] to model the shape distribution as well. During inference, each node in the scene graph is enriched with the learned local-to-global context and sequentially fed into the latent diffusion model for each shape generation. Figure 1 (c) illustrates that our method effectively leverages relationship encoding to generate commonsense scenes, i.e., arranged plausibly and realistically, exhibiting scene-level consistency while preserving object shape diversity. Furthermore, to facilitate the benchmarking of CSS, we curate a novel indoor scene graph dataset, SG-FRONT , upon a synthetic dataset 3DFRONT [18], since no existing indoor scene graph datasets provide high-quality meshes. SG-FRONT comprises around 45K 3D samples with annotated semantic and instance segmentation labels and a corresponding scene graph describing each scene.

Our contributions can be summarized into three points. First , we present CommonScenes , a fully generative model that converts scene graphs into corresponding 3D scenes using a diffusion model. It can be intuitively manipulated through graph editing. Second , CommonScenes concurrently models scene layout and shape distribution. It thereby encapsulates both global inter-object relationships and local shape cues. Third , we contribute SG-FRONT , a synthetic indoor dataset extending 3D-FRONT by scene graphs, thereby contributing graph-conditional scene generation benchmarks.

2 Related Work

Scene Graph Scene graphs provide a rich symbolic and semantic representation of the scene using nodes and relationships [25]. They are useful in many 2D-related topics such as image

generation [24, 67], image manipulation [12], caption generation [31], visual question answering [55], and camera localization [27]. Quickly after this progress, they are used in following areas: 3D scene understanding [59, 1, 64, 29], dynamic modeling [49], robotic grounding [22, 50, 68], spatio-temporal 4D [75, 30, 74], and controllable scene synthesis [33, 73, 60, 41, 13].

Indoor 3D Scene Synthesis Controllable scene synthesis is extensively explored in the computer graphics community, varying from text-based scene generation [19] to segmentation map [43] or spatial layout-based image-to-image translation tasks [71]. Likewise in 3D, several methods generated scenes from images [56, 39, 2, 69, 14, 40, 38], text [36], probabilistic grammars [4, 23, 11, 45], layouts [26, 54], in an autoregressive manner [44, 63], or through learning deep priors [61]. Another line of work closer to ours is based on graph conditioning [33, 73, 60, 41]. Luo et al. [35] proposed a generative scene synthesis through variational modeling coupled with differentiable rendering. However, their method relies on shape retrieval, which depends on an existing database. Graphto-3D [13] proposes a joint approach to learn both scene layout and shapes with a scene graph condition. Nevertheless, their object generation relies on a pre-trained shape decoder, limiting the generalizability. Unlike previous work, our method generates 3D scenes with the condition over a scene graph, which is trained end-to-end along with content awareness, resulting in higher variety and coherency.

Denoising Diffusion Models A diffusion probabilistic model is a trained Markov chain that generates samples matching data by reversing a process that gradually adds noise until the signal vanishes [52]. Diffusion models have quickly gained popularity due to their unbounded, realistic, and flexible generation capacity [21, 53, 37, 28, 47, 15]. However, studies have identified that the diffusion models lack compositional understanding of the input text [51]. Several advancements have been introduced to address these limitations. Techniques such as the introduction of generalizable conditioning and instruction mechanisms have emerged [5, 70]. Moreover, optimizing attention channels during testing has also been explored [6, 17]. Recently, a latent diffusion model using a Signed Distance Field (SDF) to represent 3D shapes was proposed contemporaneously at multiple works [8, 32], which can be conditioned on a text or a single view image. For the contextual generation conditioned on scene graphs, methods have converted triplets into the text to condition the model [17, 67], where Yang et al. [67] proposed a graph conditional image generation based on masked contrastive graph training. To the best of our knowledge, we are the first to leverage both areas of scene graph and latent diffusion for end-to-end 3D scene generation.

3 Preliminaries

Scene Graph A scene graph, represented as G = ( V , E ) , is a structured representation of a visual scene where V = { v i | i ∈ { 1 , . . . , N }} denotes the set of vertices (object nodes) and E = { e i → j | i, j ∈ { 1 , . . . , N } , i = j } represents the set of directed edge from node v i to v j . Each vertex v i is categorized through an object class c node i ∈ C node , where C node denotes the set of object classes. The directed edges in E capture the relationships between objects in terms of both semantic and geometric information. Each edge e i → j has a predicate class c edge i → j ∈ C edge , where C edge denotes the set of edge predicates. These relationships can incorporate various aspects, such as spatial locations (e.g., left/right, close by ) or object properties ( bigger/smaller ). To facilitate subsequent processing and analysis, each node v i and edge e i → j are typically transformed into learnable vectors o i and τ i → j , respectively, through embedding layers, as shown in Figure 2.A.

̸

Conditional Latent Diffusion Model Diffusion models learn a target distribution by reversing a progressive noise diffusion process modeled by a fixed Markov Chain of length T [21, 53]. Fundamentally, given a sample x t from the latent space, gradual Gaussian Noise is added with a predefined scheduler x t , t ∈ { 1 , . . . , T } [21]. Then, the denoiser ε θ , typically a UNet[48], is trained to recover denoising from those samples. The recently introduced Latent Diffusion Models (LDMs) [47] reduce the computational requirements by learning this distribution in a latent space established by a pre-trained VQ-VAE [58] instead of directly processing the full-size input. The popular usage of LDM is conditional LDM, which allows the generation to obey the input cue c i [47]. The training objective can be simplified to

Bed

MLP

Bed

Table

Front of

Node

Embedding Layers

Edge

Embedding Layers

A. Previous Scene

Graph Feature Representations

Bed

Table

Bed

Contextual Encoding

"Bed"

"Bed in front

of Table"

s

i

, t

i

, α

i

Box Parametrization

Text

Encoder

B. Contextual Graph

where c i denotes a conditioning vector corresponding to the input sample i fed into ε θ .

4 Method

Overview Given a semantic scene graph, our approach endeavors to generate corresponding 3D scenes conforming to commonsense. We employ a dual-branch network starting with a contextual encoder E c , as shown in Figure 3. The two branches, referred to as the Layout Branch and the Shape Branch , function simultaneously for layout regression and shape generation. In Sec. 4.1, we first illustrate how a scene graph evolves to a B ox-enhanced C ontextual G raph (BCG) with features from pre-trained visual-language model CLIP [46] and bounding box parameters (Figure 2). Then, we show how BCG is encoded by E c and manipulated by the graph manipulator (Figure 3.A, B and C). In Sec. 4.2, we introduce the layout branch for layout decoding, and in Sec. 4.3, we introduce the shape branch for shape generation. Finally, we explain the joint optimization in Sec. 4.4.

4.1 Scene Graph Evolution

Contextual Graph As shown in Figure 2.A and B, we incorporate readily available prompt features from CLIP [46] as semantic anchors, capturing coarse inter-object information, into each node and edge of the input graph to conceptualize it as Contextual Graph G c with p i = E CLIP ( c node i ) for objects and p i → j = E CLIP ( c node i ⊞ c edge i → j ⊞ c node j ) for edges. Here E CLIP is the pre-trained and frozen text encoder in [46], ⊞ denotes the aggregation operation on prompts including subject class c node i , predicate c edge i → j , and object class c node j . Thereby, the embeddings of G c = ( V c , E c ) is formalized as,

F V c = { f i v c = ( p i , o i ) | i ∈ { 1 , . . . , N }} , F E c = { f i → j e c = ( p i → j , τ i → j ) | i, j ∈ { 1 , . . . , N }} , (2) where F V c represents the set of object features and F E c the edge features.

Box-Enhanced Contextual Graph In training, we enrich each node in the contextual graph by using ground truth bounding boxes parameterized by box sizes s , locations t , and the angular rotations along the vertical axis α , yielding a BCG, as shown in Figure 2.C. Thereby, the node embeddings of BCG are represented by F ( b ) V c = { f ( b ) i v c = ( p i , o i , b i ) | i ∈ { 1 , . . . , N }} , where b i is obtained by encoding ( s i , t i , α i ) with MLPs. Note that BCG is only used during training, i.e., bounding box information is not needed during inference.

Graph Encoding BCG is encoded by the subsequent triplet-GCN-based contextual encoder E c , which together with the layout decoder D l in Sec. 4.2 construct a Conditional Variational Autoencoder (CVAE) [35]. The encoding procedure is shown in Figure 3.A, B and C. Given a BCG, during training, we input the embeddings F ( b ) V c and F E c into E c to obtain an Updated Contextual Graph with node and edge features represented as ( F ( z ) V c , F E c ) , which is also the beginning point of the inference route. Each layer of E c consists of two sequential MLPs { g 1 , g 2 } , where g 1 performs message passing between connected nodes and updates the edge features, g 2 aggregates features from all connected neighbors of each node and update its features, as shown in the follows:

...

D. Layout Branch

Relation Embeddings of Nodes

Layout

Decoder

D

l

Denoiser

ε

θ

C. Updated Contextual Graph

Contextual

Encoder

E

c

Relation

Encoder

E

r

VQ-VAE

Encoder

S

Bed

Scene Generation

Z

X

t

X

t-1

S'

X

0

X

T

···

···

···

···

VQ-VAE

Decoder

Noise

Training

Only

Fronzen

Modules

Inference

Route

Graph

Manipulator

B. Shared Modules

where l g denotes a single layer in E c and N G ( v i ) includes all the connected neighbors of v i . AVG refers to average pooling. We initialize the input embeddings of layer 0 as the features from the Updated Contextual Graph , ( ϕ 0 v i , ϕ 0 e i → j , ϕ 0 v j ) = ( f ( b ) i v c , f i → j e c , f ( b ) j v c ) . The final embedding ϕ L v i is used to model a joint layout-shape distribution Z , parameterized with the N c -dimensional Gaussian distribution Z ∼ N ( µ, σ ) , where µ, σ ∈ R N c , predicted by two separated MLP heads. Thereby, Z is modeled through minimizing:

where D KL is the Kullback-Liebler divergence measuring the discrepancy between Z and the posterior distribution p ( z | x ) chosen to be standard N ( z | 0 , 1) . We sample a random vector z i from Z for each node, update node feature F ( z ) V c by concatenating { z i , p i , o i } , keep edge features unchanged.

4.2 Layout Branch

As shown in Figure 3.D, a decoder D l , which is another triplet-GCN, generates 3D layout predictions upon updated embeddings. In this branch, E c and D l are jointly optimized by Eq. (4) and a bounding box reconstruction loss:

where ˆ s i , ˆ t i , ˆ α i denote the predictions of bounding box size s , locations t and angular rotations α , respectively. λ is the rotation classification label. We partition the rotation space into Λ bins, transforming the rotation regression problem into a classification problem.

4.3 Shape Branch

As shown in Figure 3.E, in parallel to the Layout Branch , we introduce Shape Branch to generate shapes for each node in the given graph, and represent them by SDF.

Relation Encoding The core idea of our method is to exploit global scene-object and local objectobject relationships to guide the generation process. Hence, besides incorporating CLIP features for coarse local object-object relationships, as introduced in Section 4.1, we further design a relation encoder E r , based on triplet-GCN as well, to encapsulate global semantic cues of the graph into

each node and propagate local shape cues between connected nodes. Specifically, E r operates on the learned embeddings ( F ( z ) V c , F E c ) of the Updated Contextual Graph , and updates the feature of each node with local-to-global semantic cues, yielding node-relation embeddings for subsequent diffusion-based shape generation.

Shape Decoding We use an LDM conditioned on node-relation embedding to model the shapegeneration process. In terms of shape representation, we opt for truncated SDF (TSDF) [10] around the surface of the target object within a voxelized space S ∈ R D × D × D . Following the LDM, the dimension D of TSDF can be reduced by training a VQ-VAE [58] as a shape compressor to encode the 3D shape into latent dimensions x ∈ R d × d × d , where d << D is built upon a discretized codebook. Then a forward diffusion process adds random noise to the input shape x 0 transferring to x T , upon which we deploy a 3D-UNet [9] ε θ (Figure 3.E) to denoise the latent code back to x 0 according to DDPM model by Ho et al. [21]. The denoiser is conditioned on node-relation embeddings to intermediate features of 3D UNet via the cross-attention mechanism. Finally, the decoder of VQ-V AE generates the shape S ′ from the reconstructed x 0 . For the denoising process at the timestep t , the training objective is to minimize:

where the evidence lower bound between the sampled noise ε and the prediction conditioned on the contextual relation embedding extracted from E r is optimized. At test time, a latent vector is randomly sampled from N (0 , 1) and progressively denoised by 3D-Unet to generate the final shape. Each shape within the layout is populated based on per-node conditioning, ultimately producing a plausible scene. Compared to prior work, our design can bring more diverse shape generation by taking advantage of the diffusion model architecture, as shown in experiments.

4.4 Layout-Shape Training

Our pipeline is trained jointly in an end-to-end fashion, allowing the Layout Branch and Shape Branch to optimize each other by sharing latent embeddings ( F ( z ) V c , F E c ) , coming out of E c . The final optimization loss is the combination of scene distribution modeling, layout generation, and shape generation:

where λ 1 , λ 2 and λ 3 are weighting factors. Further insights on accomplishing the batched training are provided in Supplementary Material.

5 Experiments

SG-FRONT Dataset Due to the lack of scene graph datasets also providing high-quality object meshes, we construct SG-FRONT , a set of well-annotated scene graph labels, based on a 3D synthetic dataset 3D-FRONT [18] that offers professionally decorated household scenarios. The annotation labels can be grouped into three categories spatial/proximity, support, and style . The spatial relationships are based on a series of relationship checks between the objects, which control the object bounding box location, e.g., left/right , the comparative volumes, e.g., bigger/smaller , and heights, e.g., taller/shorter . The support relationships include directed structural support, e.g., close by , above , and standing on . Thresholds for these labels are iterated and set by the annotators. Finally, style relationships include the object attributes to assign labels, namely same material as , same shape as , and same super-category as . SG-FRONT contains around 45K samples from three different indoor scene types, covering 15 relationship types densely annotating scenes. More details are provided in the Supplementary Material.

Implementation Details We conduct the training, evaluation, and visualization of CommonScenes on a single NVIDIA A100 GPU with 40GB memory. We adopt the AdamW optimizer with an initial learning rate of 1e-4 to train the network in an end-to-end manner. We set { λ 1 , λ 2 , λ 3 } = { 1 . 0 , 1 . 0 , 1 . 0 } in all our experiments. N c in distribution Z is set to 128 and TSDF size D is set as 64. We provide more details in the Supplementary Material.

Bedroom

Dining Room

Graph-to-3D

Layout+Txt2shape

CommonScenes (Ours)

Graph-to-Box

Living Room

Closeby

Left;

Smaller

Bed

Bigger

Closeby

Table 2

Right

Right

Table3

Closeby

Left;

1

↑

Behind

Close by;

Chair 1

Smaller

Above;→

Behind

Above

Lamp

Above;

Front

Chair 2

Front

Bigger

Nightstand 2

Above;Right, Behind

个

Smaller

Smaller

Above;

Lamp

Smaller

Left;

Above;

I ps

Front; Left

Smaller

Front,

Table 1

Above;Behind;

Sofa2

Right

Above;Behind;

Right

Above

Above; Behind

Sofa1

Lamp

Above; Behind;

Above

Left

Above; Behind;

Chair

Left

Table 4

Smaller

Front;

Table

Wardrobe

Above,Right;

Chair

Chair3

Chair 4

Smaller

Above;

Left;

Above; Front

Left;

Smaller

Right;

Behind; Right;

Smaller

Front;

Table 5

Closeby

Smaller

Table

山

Evaluation Metrics To measure the fidelity and diversity of generated scenes, we employ the commonly adopted Fréchet Inception Distance (FID) [20] & Kernel Inception Distance (KID) [3] metrics [44]. We project scenes onto a bird's-eye view, excluding lamps to prevent occlusion and using object colors as semantic indicators. To measure the scene graph consistency , we follow the scene graph constraints [13], which measure the accuracy of a set

Diverse

Consistent

Consistent

Seed 1

Seed 2

of relations on a generated layout. We calculate the spatial left/right, front/behind, smaller/larger, taller/shorter metrics, as well as more difficult proximity constraints close by and symmetrical . To measure the shape consistency , we test dining rooms, a typical scenario in which tables and chairs should be in suits according to the commonsense decoration. We identify matching dining chairs and tables in 3D-FRONT using the same CAD models and note their instance IDs. This helps us determine which entities belong to the same sets. We calculate Chamfer Distance (CD) of each of the two objects in each set after generation. To measure the shape diversity , we follow [13] to generate each scene 10 times and evaluate the change of corresponding shapes using CD. We illustrate the concepts of diversity and consistency in Figure 5. We also report MMD, COV, and 1-NNA following [66] for the object-level evaluation in the Supplementary Materials.

Compared Baselines We include three types of baseline methods. First , three retrieval-based methods, i.e., a layout generation network 3D-SLN [35], a progressive method to add objects oneby-one designed in [13], and Graph-to-Box from [13]. Second , a semi-generative SOTA method Graph-to-3D . Third , an intuitive method called layout+txt2shape that we design according to the instruction in Sec. 1, stacking a layout network and a text-to-shape generation model in series, and it is a fully generative approach with text only. All baseline methods are trained on the SG-FRONT dataset following the public implementation and training details.

5.1 Graph Conditioned Scene Generation

Qualitative results The generated scenes from different baselines are shown in Figure 4: (a) Graphto-Box retrieval, (b) Graph-to-3D, (c) layout+txt2shape, and (d) ours. It can be seen that our method has better shape consistency and diversity. Specifically, in terms of object shape quality, retrieval baseline scenes look optimal since they are based on real CAD models. Yet, the layout configuration is poor, and the retrieved object styles are not plausible, e.g., dining chairs and nightstands are not in a suit, respectively, in the same rooms. Graph-to-3D improves this by learning coherent scenes. However, the object styles are not conserved within scene styles, providing unrealistic scene-object inconsistency, e.g., in the bedroom, the height of the chair does not match the style and the height of the table. Subsequently, (c) improves the layout, which can be seen in the living room area, but again falls back on the generated shape quality and shares the same problem with Graph-to-Box on the poor object-object consistency. In contrast, CommonScenes can capture diverse environments considering both generation consistency and stylistic differences with respect to each scene. In the

living room, the scene graph input requires that Table No.3 and Table No.4 should stand close to each other. Graph-to-Box and Graph-to-3D fail to achieve the goal, which is also reflected in Table 3 and Table 4 that these kinds of methods cannot handle close by relation. We show more results in Supplementary Material.

Quantitative results We provide the FID/KID scores in Table 1 separated as average over three room types. Our method establishes the best results among the free-shape representation methods by improving the Graph-to-3D by approximately 10% on FID and 40% on KID on average. On the other hand, most of the layout+txt2shape results are worse than the previous state-of-the-art methods, proving the requirement of learning layout and shape jointly. Notably, the retrieval-based methods overall have better scores than those of free shapes since the retrieved objects align well with the test database. Further on, among the retrieval baselines, ours without the diffusion shape branch ( Ours w/o SB ) surpasses the prior works, indicating the benefit of contextual graph representation. Additionally, on the generation consistency, we can observe in Table 2 that our method can reflect the edge relationship in the generated shapes directly, indicated by significant improvement in the consistency score. By leveraging the latent diffusion model, our diversity is significantly improved compared with other methods. We provide more results on diversity and also show perceptual study on the evaluation of consistency and diversity in the Supplementary Material.

In terms of scene graph constraints, we present the results as easy and hard metrics in Table 3. On easy metrics, our methods (Ours, Ours w/o SB) are either better than or on par with other methods. While on hard metrics, they are superior to others. It shows that the spatial relationships could be learned easier than the proximity ( close by ) and commonsense ( symmetrical ) based ones. The improvement over the symmetrical constraint is particularly evident, as our method surpasses the state-of-the-art (SOTA) by 10% . This demonstrates that the integrated layout and shape cues are crucial for learning these contextual configurations.

5.2 Scene Manipulation

We demonstrate the downstream application of scene manipulation on our method and compare it with the aforementioned methods on scene graph constraints in Table 4. Our methods have the highest total score in both relation change and object addition modes. In both modes, the results are

on par with Graph-to-3D on easy relations, whereas there is a major improvement in the hard ones. Further, within the retrieval methods, Ours w/o SB improves over the baselines in the addition mode and is on par in the change mode. It could be argued that changing could be more difficult since it could alter multiple objects, whereas object insertion has less effect on the overall scene. We show some qualitative manipulation examples in the Supplementary Material.

5.3 Ablations

We ablate the primary components of CommonScenes in Table 5, including (1) scene generation from the original scene graph without contextual information brought by CLIP features, (2) scene generation from the contextual graph without the participation of the relational encoder E r , (3) conditioning with concatenation on diffusion latent code instead of cross-attention, and (4) our final proposed method. We provide the mean FID scores over the scenes, as well as the mean over the hard scene graph constraints (mSG). We observe the benefit of both the context and E r indicated by FID/KID scores, as well as the choice of our conditioning mechanism.

Failure Case

Bed

Left

Bigger

Smaller

Above;

Above;

I psn

Nightstand 2

Le

Bigger

Above;

Behind

Lamp

Smaller

Above;

Above;

Lef

Wardrobe

Fronit; Left

Right;

Bigger

6 Limitations

We address the main aspects of the dataset and the limitations of our method and discuss more in the Supplementary Materials. First, the 3D-FRONT dataset used in our research contains significant noise, which we have mitigated through a post-processing step. Despite this effort, a small proportion of noisy data, specifically interpenetrating furniture instances, remains in the training dataset. Consequently, our proposed method and the baseline approaches reflect this during inference, rarely resulting in occurrences of scenes with collided objects. We show some cases in Figure 6. While our method outperforms others by effectively combining shape and layout information, it is essential to note that minor collision issues may still arise.

7 Conclusion

Scene graph-based CSS designs interactive environments suitable for a wide range of usages. Current methods heavily depend on object retrieval or pre-trained shape models, neglecting inter-object relationships and resulting in inconsistent synthesis. To tackle this problem, we introduce CommonScenes , a fully generative model that transforms scene graphs into corresponding commonsense 3D scenes. Our model regresses scene layouts via a variational auto-encoder, while generating satisfying shapes through latent diffusion, gaining higher shape diversity and capturing the global scene-object relationships and local object-object relationships. Additionally, we annotate a scene graph dataset, SG-FRONT , providing object relationships compatible with high-quality object-level meshes. Extensive experiments on SG-FRONT show that our method outperforms other methods in terms of generation consistency, quality, and diversity.

Acknowledgements This research is supported by Google unrestricted gift, the China Scholarship Council (CSC), and the Munich Center for Machine Learning (MCML). We are grateful to Google University Relationship GCP Credit Program for supporting this work by providing computational resources. Further, we thank all participants of the perceptual study.

References

Iro Armeni, Zhi-Yang He, JunYoung Gwak, Amir R Zamir, Martin Fischer, Jitendra Malik, and Silvio Savarese. 3d scene graph: A structure for unified semantics, 3d space, and camera. In ICCV , 2019. 3

Sherwin Bahmani, Jeong Joon Park, Despoina Paschalidou, Xingguang Yan, Gordon Wetzstein, Leonidas Guibas, and Andrea Tagliasacchi. Cc3d: Layout-conditioned generation of compositional 3d scenes. In ICCV , 2023. 3

Mikołaj Bi´ nkowski, Dougal J. Sutherland, Michael Arbel, and Arthur Gretton. Demystifying MMD GANs. In ICLR , 2018. 7

Martin Bokeloh, Michael Wand, Hans-Peter Seidel, and Vladlen Koltun. An algebraic model for parameterized shape editing. ACM Transactions on Graphics (TOG) , 31(4):1-10, 2012. 3

Tim Brooks, Aleksander Holynski, and Alexei A Efros. Instructpix2pix: Learning to follow image editing instructions. In CVPR , 2023. 1, 3

Hila Chefer, Yuval Alaluf, Yael Vinker, Lior Wolf, and Daniel Cohen-Or. Attend-and-excite: Attentionbased semantic guidance for text-to-image diffusion models, 2023. 3

Mo Jiao Chen, Bin Zhang, and San Xing Cao. A study of dynamic scene automatic generation system for micro-film studio. In Applied Mechanics and Materials , volume 411, pages 993-996, 2013. 1

Yen-Chi Cheng, Hsin-Ying Lee, Sergey Tuyakov, Alex Schwing, and Liangyan Gui. SDFusion: Multimodal 3d shape completion, reconstruction, and generation. In CVPR , 2023. 2, 3, 8

Özgun Çiçek, Ahmed Abdulkadir, Soeren Lienkamp, Thomas Brox, and Olaf Ronneberger. 3d u-net: Learning dense volumetric segmentation from sparse annotation. In MICCAI , 2016. 6

Brian Curless and Marc Levoy. A volumetric method for building complex models from range images. In SIGGRAPH , 1996. 6

Jeevan Devaranjan, Amlan Kar, and Sanja Fidler. Meta-sim2: Unsupervised learning of scene structure for synthetic data generation. In ECCV , 2020. 3

Helisa Dhamo, Azade Farshad, Iro Laina, Nassir Navab, Gregory D. Hager, Federico Tombari, and Christian Rupprecht. Semantic image manipulation using scene graphs. In CVPR , 2020. 2, 3

Helisa Dhamo, Fabian Manhardt, Nassir Navab, and Federico Tombari. Graph-to-3d: End-to-end generation and manipulation of 3d scenes using scene graphs. In ICCV , 2021. 3, 7, 8, 9

Yan Di, Henrique Morimitsu, Zhiqiang Lou, and Xiangyang Ji. A unified framework for piecewise semantic reconstruction in dynamic scenes via exploiting superpixel relations. In ICRA , 2020. 3

Yan Di, Chenyangguang Zhang, Pengyuan Wang, Guangyao Zhai, Ruida Zhang, Fabian Manhardt, Benjamin Busam, Xiangyang Ji, and Federico Tombari. Ccd-3dr: Consistent conditioning in diffusion for single-image 3d reconstruction, 2023. 3

Yan Di, Chenyangguang Zhang, Ruida Zhang, Fabian Manhardt, Yongzhi Su, Jason Rambach, Didier Stricker, Xiangyang Ji, and Federico Tombari. U-red: Unsupervised 3d shape retrieval and deformation for partial point clouds. In ICCV , 2023. 1

Weixi Feng, Xuehai He, Tsu-Jui Fu, Varun Jampani, Arjun Reddy Akula, Pradyumna Narayana, Sugato Basu, Xin Eric Wang, and William Yang Wang. Training-free structured diffusion guidance for compositional text-to-image synthesis. In ICLR , 2023. 3

Huan Fu, Bowen Cai, Lin Gao, Ling-Xiao Zhang, Jiaming Wang, Cao Li, Qixun Zeng, Chengyue Sun, Rongfei Jia, Binqiang Zhao, et al. 3d-front: 3d furnished rooms with layouts and semantics. In ICCV , 2021. 2, 6

Oran Gafni, Adam Polyak, Oron Ashual, Shelly Sheynin, Devi Parikh, and Yaniv Taigman. Make-a-scene: Scene-based text-to-image generation with human priors. In ECCV , 2022. 3

Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In NeurIPS , 2017. 7

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In NeurIPS , 2020. 3, 6

N. Hughes, Y. Chang, and L. Carlone. Hydra: A real-time spatial perception system for 3D scene graph construction and optimization. In RSS , 2022. 3

Chenfanfu Jiang, Siyuan Qi, Yixin Zhu, Siyuan Huang, Jenny Lin, Lap-Fai Yu, Demetri Terzopoulos, and Song-Chun Zhu. Configurable 3d scene synthesis and 2d image rendering with per-pixel ground truth using stochastic grammars. International Journal of Computer Vision , 126:920-941, 2018. 3

Justin Johnson, Agrim Gupta, and Li Fei-Fei. Image generation from scene graphs. In CVPR , 2018. 1, 3

Justin Johnson, Ranjay Krishna, Michael Stark, Li-Jia Li, David A. Shamma, Michael S. Bernstein, and Li Fei-Fei. Image retrieval using scene graphs. In CVPR , 2015. 2

Akash Abdu Jyothi, Thibaut Durand, Jiawei He, Leonid Sigal, and Greg Mori. Layoutvae: Stochastic scene layout generation from a label set. In ICCV , 2019. 3

Julia Kabalar, Shun-Cheng Wu, Johanna Wald, Keisuke Tateno, Nassir Navab, and Federico Tombari. Towards long-term retrieval-based visual localization in indoor environments with changes. IEEE Robotics and Automation Letters , 8(4):1975-1982, 2023. 3

Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. In NeurIPS , 2022. 3

Ue-Hwan Kim, Jin-Man Park, Taek-jin Song, and Jong-Hwan Kim. 3-d scene graph: A sparse and semantic representation of physical environments for intelligent agents. IEEE Transactions on Cybernetics , 50(12):4921-4933, 2020. 3

Xin Kong, Xuemeng Yang, Guangyao Zhai, Xiangrui Zhao, Xianfang Zeng, Mengmeng Wang, Yong Liu, Wanlong Li, and Feng Wen. Semantic graph based place recognition for 3d point clouds. In IROS , 2020. 3

Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International journal of computer vision , 123(1):32-73, 2017. 3

Muheng Li, Yueqi Duan, Jie Zhou, and Jiwen Lu. Diffusion-sdf: Text-to-shape via voxelized diffusion. In CVPR , 2023. 2, 3

Manyi Li, Akshay Gadi Patil, Kai Xu, Siddhartha Chaudhuri, Owais Khan, Ariel Shamir, Changhe Tu, Baoquan Chen, Daniel Cohen-Or, and Hao Zhang. Grains: Generative recursive autoencoders for indoor scenes. ACM Transactions on Graphics (TOG) , 38(2):1-16, 2019. 3

Yiyi Liao, Katja Schwarz, Lars Mescheder, and Andreas Geiger. Towards unsupervised learning of generative models for 3d controllable image synthesis. In CVPR , 2020. 1

Andrew Luo, Zhoutong Zhang, Jiajun Wu, and Joshua B. Tenenbaum. End-to-end optimization of scene layout. In CVPR , 2020. 1, 2, 3, 4, 8, 9

Rui Ma, Akshay Gadi Patil, Matthew Fisher, Manyi Li, Sören Pirk, Binh-Son Hua, Sai-Kit Yeung, Xin Tong, Leonidas Guibas, and Hao Zhang. Language-driven synthesis of 3d scenes from scene databases. ACM Transactions on Graphics (TOG) , 37(6):1-16, 2018. 3

Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon. SDEdit: Guided image synthesis and editing with stochastic differential equations. In ICLR , 2022. 3

Yinyu Nie, Angela Dai, Xiaoguang Han, and Matthias Nießner. Learning 3d scene priors with 2d supervision. In CVPR , 2023. 3

Yinyu Nie, Xiaoguang Han, Shihui Guo, Yujian Zheng, Jian Chang, and Jian Jun Zhang. Total3dunderstanding: Joint layout, object pose and mesh reconstruction for indoor scenes from a single image. In CVPR , 2020. 3

Evin Pınar Örnek, Shristi Mudgal, Johanna Wald, Yida Wang, Nassir Navab, and Federico Tombari. From 2d to 3d: Re-thinking benchmarking of monocular depth prediction, 2022. 3

Wamiq Para, Paul Guerrero, Tom Kelly, Leonidas J Guibas, and Peter Wonka. Generative layout modeling using constraint graphs. In ICCV , 2021. 3

Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In CVPR , 2019. 2, 8, 9

Taesung Park, Ming-Yu Liu, Ting-Chun Wang, and Jun-Yan Zhu. Semantic image synthesis with spatiallyadaptive normalization. In CVPR , 2019. 3

Despoina Paschalidou, Amlan Kar, Maria Shugrina, Karsten Kreis, Andreas Geiger, and Sanja Fidler. Atiss: Autoregressive transformers for indoor scene synthesis. In NeurIPS , 2021. 3, 7

Pulak Purkait, Christopher Zach, and Ian Reid. Sg-vae: Scene grammar variational autoencoder to generate new indoor scenes. In ECCV , 2020. 3

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML , 2021. 2, 4

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In CVPR , 2022. 2, 3

Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In MICCAI , 2015. 3

Antoni Rosinol, Arjun Gupta, Marcus Abate, Jingnan Shi, and Luca Carlone. 3d dynamic scene graphs: Actionable spatial perception with places, objects, and humans. In RSS , 2020. 3

Antoni Rosinol, Andrew Violette, Marcus Abate, Nathan Hughes, Yun Chang, Jingnan Shi, Arjun Gupta, and Luca Carlone. Kimera: From slam to spatial perception with 3d dynamic scene graphs. The International Journal of Robotics Research , 40(12-14):1510-1546, 2021. 3

Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, Jonathan Ho, David J Fleet, and Mohammad Norouzi. Photorealistic text-to-image diffusion models with deep language understanding. In NeurIPS , 2022. 3

Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In ICML , 2015. 3

[53]

Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole.

Score-based generative modeling through stochastic differential equations. In

ICLR

, 2021. 3

Jiapeng Tang, Yinyu Nie, Lev Markhasin, Angela Dai, Justus Thies, and Matthias Nießner. Diffuscene: Scene graph denoising diffusion probabilistic model for generative indoor scene synthesis, 2023. 3

Damien Teney, Lingqiao Liu, and Anton van Den Hengel. Graph-structured representations for visual question answering. In CVPR , 2017. 3

Shubham Tulsiani, Saurabh Gupta, David Fouhey, Alexei A. Efros, and Jitendra Malik. Factoring shape, pose, and layout from the 2d image of a 3d scene. In CVPR , 2018. 3

Mikaela Angelina Uy, Vladimir G Kim, Minhyuk Sung, Noam Aigerman, Siddhartha Chaudhuri, and Leonidas J Guibas. Joint learning of 3d shape retrieval and deformation. In CVPR , 2021. 2

Aaron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete representation learning. In NeurIPS , 2017. 3, 6

Johanna Wald, Helisa Dhamo, Nassir Navab, and Federico Tombari. Learning 3d semantic scene graphs from 3d indoor reconstructions. In CVPR , 2020. 3

Kai Wang, Yu-An Lin, Ben Weissmann, Manolis Savva, Angel X Chang, and Daniel Ritchie. Planit: Planning and instantiating indoor scenes with relation graph and spatial prior networks. ACM Transactions on Graphics (TOG) , 38(4):1-15, 2019. 3

Kai Wang, Manolis Savva, Angel X Chang, and Daniel Ritchie. Deep convolutional priors for indoor scene synthesis. ACM Transactions on Graphics (TOG) , 37(4):70, 2018. 3

Xiaolong Wang and Abhinav Gupta. Generative image modeling using style and structure adversarial networks. In ECCV , 2016. 1

Xinpeng Wang, Chandan Yeshwanth, and Matthias Nießner. Sceneformer: Indoor scene generation with transformers. In 3DV , 2021. 3

Shun-Cheng Wu, Johanna Wald, Keisuke Tateno, Nassir Navab, and Federico Tombari. Scenegraphfusion: Incremental 3d scene graph prediction from rgb-d sequences. In CVPR , 2021. 3

Jianghao Xiong, En-Lin Hsiang, Ziqian He, Tao Zhan, and Shin-Tson Wu. Augmented reality and virtual reality displays: emerging technologies and future perspectives. Light: Science & Applications , 10(1):216, 2021. 1

Guandao Yang, Xun Huang, Zekun Hao, Ming-Yu Liu, Serge Belongie, and Bharath Hariharan. Pointflow: 3d point cloud generation with continuous normalizing flows. In ICCV , 2019. 8

Ling Yang, Zhilin Huang, Yang Song, Shenda Hong, Guohao Li, Wentao Zhang, Bin Cui, Bernard Ghanem, and Ming-Hsuan Yang. Diffusion-based scene graph to image generation with masked contrastive pre-training, 2022. 3

Guangyao Zhai, Xiaoni Cai, Dianye Huang, Yan Di, Fabian Manhardt, Federico Tombari, Nassir Navab, and Benjamin Busam. Sg-bot: Object rearrangement via coarse-to-fine robotic imagination on scene graphs, 2022. 1, 3

Chenyangguang Zhang, Zhiqiang Lou, Yan Di, Federico Tombari, and Xiangyang Ji. Sst: Real-time end-to-end monocular 3d reconstruction via sparse spatial-temporal guidance. In ICME , 2023. 3

Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In ICCV , 2023. 3

Bo Zhao, Lili Meng, Weidong Yin, and Leonid Sigal. Image generation from layout. In CVPR , 2019. 3

Yizhou Zhao, Kaixiang Lin, Zhiwei Jia, Qiaozi Gao, Govind Thattai, Jesse Thomason, and Gaurav S. Sukhatme. Luminous: Indoor scene generation for embodied ai challenges. In NeurIPS Workshop , 2021. 1

Yang Zhou, Zachary While, and Evangelos Kalogerakis. Scenegraphnet: Neural message passing for 3d indoor scene augmentation. In ICCV , 2019. 3

Ege Özsoy, Tobias Czempiel, Evin Pınar Örnek, Ulrich Eck, Federico Tombari, and Nassir Navab. Holistic or domain modeling: a semantic scene graph approach. International Journal of Computer Assisted Radiology and Surgery , pages 1-9, 2023. 3

Ege Özsoy, Evin Pınar Örnek, Ulrich Eck, Tobias Czempiel, Federico Tombari, and Nassir Navab. 4d-or: Semantic scene graphs for or domain modeling. In MICCAI , 2022. 3

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper looks at the problem of learning a generative model for sampling 3D environments from a description based on a graph and natural language. The graph does correspond to objects that participate in the scene, and textual descriptions attached to nodes and edges further specify the scene. Textual descriptions are encoded using CLIP. A variational auto encoder conditioned on these inputs is trained to generate the bounding boxes of the 3D objects. A latent diffusion model is trained in parallel to generate the 3D shape of the object in each bounding box. The existing FRONT-3D dataset is augmented with textual descriptions for training this model. Experiments show the advantages and limitations compared to ablations and prior works.\n\nSTRENGTHS:\n* Scene generation is an important problem and this paper addresses it with a responsible model using modern components in a competent manner.\n\n* The paper contributes a large amount of additional labels for 3D-FRONT.\n\nWEAKNESSES:\n* The  presentation, especially of the technical parts, can be improved. It was difficult to understand how the layout auto-encoder is setup. It would be useful to give a high-level overview first of where the method is trying to achieve (e.g., encoder Z = E(P, O, B), decoder \\hat B = E(P, O, Z), Z Gaussian). Section 4.1 presents only half of the story, and we need to get to section 4.2 before the auto-encoder materialises. Line 143 literally states the the encoder is trained by minimising Eq. (4), which is clearly not enough -- Eq. (5) is also needed.\n\n* You man need to revisit the formalism at Ines 90 to 99. $c_i^{node}$ and the same terms for the edges are very poorly defined. Furthermore, it also seems that you can attach more than one such attributes to each node or edge. The formalism does not allow for that.\n\n* Some aspects of the model are unclear. See questions below.\n\nQUESTIONS:\n* Please clarify how the vectors o in Figure 2 and line 99 are obtained. The paper only states that they are \"learnable\".\n\n* The two shape and layout branches are said to be trained jointly (Section 4.4). However, the learning dynamics of auto encoder and diffusion processes (one per object) seem to me to be very different. Is it really that trivial to mix them?\n\n* Line 172: the shape diffusion model connects to the *entire* graph, and so to all objects, via cross-attention. How does each instantiation of the diffusion process know *which* of the several objects it is meant to generate?\n\n* Line 214: I don't know what it means to share the same CAD model source. Nor I do understand why it is is sensible to compare a generated sample to a specific CAD instance -- I would not expect the two to match.\n\nLIMITATIONS:\nThere is sufficient discussion of limitations, but no discussion of ethics. The latter, however, is not particularly relevant for this paper. It may be worth at least discussing the licensing of 3D-FRONT.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nSummary: This paper presents a generative model to generate 3D scenes from scene graphs. Their model is fully generative without the need of any shape database or embeddings. The 3D scenes generation model is pipelined into finding the scene layout and the construction the shapes of the nodes using a diffusion model. The authors constructed a scene graph dataset SG-FRONT from 3D-FRONT dataset and trained end to end to generate 3D scenes from the scene graph input. The paper reported qualitative, quantitative results with comparison to other sota models on the SG-FRONT dataset.\n\nSTRENGTHS:\n1. the paper claims to have curated a dataset(enriching an existing dataset with scene graph annotations) for 3D scenes construction from scene graphs. \n2. Codes and dataset will be publicly available\n3. The generated 3D scenes and the quantitative numbers show potential of the scene graph based approach for 3D scenes reconstruction using a diffusion model\n\nWEAKNESSES:\n1. In section 6, Compared Baselines, 'a fully generative method a text-to-shape generation model that follows a layout generation' does not have citations. Is it something the authors modeled for experimentation? \n2. In Table 1, need to specify which of the methods use fully generative approach with text only. Also, calrifying in the caption of table 1 on how you segmented table 1 into two main rows and what is 'Ours 'w/o SB''\n3. Sharing more details on how the authors curated the dataset from 3D-FRONT in the main paper might be helpful for the readers\n4. Some possible typos:\nline 47 : 'cues and fine local inter-object relationships.'\nLine 65 and 66: Incomplete lines : 'Quickly after this progress, the 3D [53, 1, 58, 26], dynamic [44], robotic\n66 grounding [19, 45], spatio-temporal 4D [66], and controllable scene synthesis [29, 65, 54, 36, 13].'\n\nQUESTIONS:\n1. In section 6, Compared Baselines, 'a fully generative method a text-to-shape generation model that follows a layout generation' does not have citations. Is it something the authors modeled for experimentation?\n\nLIMITATIONS:\nno negative societal impact.\nNo limitation addressed.\nAll the 3D scene generation is done on synthetic dataset. How would the whole model perform on the real scene?", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the task of controllable scene synthesis of indoor rooms, conditioned on a semantic scene graph that captures spatial, style and support relationships between objects in a scene. In particular, they introduce CommonScenes, a generative model capable of converting scene graphs into 3D scenes using diffusion models. Given a scene graph of the scene to be generated, they first enhance it using CLIP features computed on the inter-object relationships, as well as using the ground-truth bounding box annotations. Next, they leverage a triplet-GCN based relation network to propagate information among the objects in the scene and produce the Box-enhanced Contextual Graph (BCG). Given a BCG, they then utilize a dual-branch network that generates the final scene. The first branch (layout branch) is another triplet-GCN that generates 3D layout predictions and the second branch (shape branch) generates shapes for each node represented as SDF. The shape branch is simply a latent diffusion model. During training, CommonScenes considers supervision in the form of per-object bounding-box annotations (size, location, rotation), as well as per-object 3D meshes in the form of SDFs. To evaluate their model, the authors enrich the 3D-FRONT dataset with scene graph labels, by annotating per-object spatial, style and support relationships. The authors compare their model to several baselines and show that their approach yields more realistic scenes in terms of FID and KID scores.\n\nOverall, I think this is a nice work that alleviates the need for relying on a library of assets to replace the generated bounding boxes with 3D objects. The proposed architecture is novel and seems to be able to consistently produce plausible scenes. That being said I think that the proposed pipeline is quite complex, as it consists of multiple submodules, and it requires conditioning in the form of semantic graphs that are relatively hard to acquire, hence the authors had to enrich 3D-FRONT. However, since the authors show that their model outperforms prior research I am in favor of accepting this paper.\n\nSTRENGTHS:\n1. To the best of my knowledge, the proposed model is novel and the authors clearly demonstrate that their model consistently produces plausible scenes. I think that this work is an important step towards alleviating the need for large libraries of assets when generating a novel scenes. Relying on such libraries naturally restricts the diversity of the generated scenes to the diversity of the objects in the library. Although, the proposed model is a relatively complex, I believe it is a valuable work that could potential inspire other works in this direction.\n\n2. I think that the development of the SG-FRONT dataset, that extends 3D-FRONT with scene graph labels is an important contribution of this work that can potentially facilitate other research projects, therefore I strongly encourage the authors to make this dataset variant available upon the paper's acceptance.\n\n3. I really appreciated the supplementary video that the authors provided. I particularly liked the intuitive explanation of the proposed method as well as the additional results. In the future, I hope that more authors will provide such high quality supplementary videos alongside with the paper submission.\n\n4. Although the proposed model is quite complex, I think the paper is nicely written and easy to follow. I really liked the provided figures that provide a more high-level pictorial overview of the various components of the proposed pipeline.\n\nWEAKNESSES:\n1. The main weakness of this work is that it is relatively complex, as it consists of multiple sub-modules. One thing that was not 100% clear from the text was whether they perform a two-stage training, namely first produce the BCG, as discussed in Sec 4.1 and then jointly train the shape and layout branch of CommonScenes, as discussed in Sec 4.2. It might be good to clarify this for the final version of the paper. In addition, for the Shape Decoding module (L164-179), I am not 100% sure whether the authors use some sort of class conditioning to train one LDM for all object types or they use separately trained LDMs per-object class. This might be good to clarify.\n\n2. Although, the paper focuses on scene synthesis, I think that an important evaluation that is missing is measuring the quality of the generated shapes. In particular, the authors could generate a couple of rooms and then take the beds, chairs, nightstands etc. and compute the COV and MMD of the generated objects w.r.t the ground-truth objects. Although, object generation is not the main focus of this work, I think this analysis would be valuable, since this is an integral component of the proposed model. I noticed that the authors tried to do this type of analysis in Sec. 1.1 in their supplementary, but I found it a bit weird that they report Chamfer Distance, instead of MMD/COV. Is there any reason for this?\n\nQUESTIONS:\n1. For the Ablations in Sec 6.3, for the model variant that is trained without context, do the authors omit both the CLIP features, as well as the bound-box features or only the former. I think that they should be ablated separately. Personally, I am not 100% convinced about the significance of the CLIP features. Can the authors please clarify?\n\n2. I am wondering whether the authors tried to condition their scene generation also on the floor plan together with the semantic scene graph? Unless I am missing out something, this can be easily done e.g. by extracting features from the floor plan and concatenating with the per-object features while created the BCG? Do the authors think that something as simple as that would work?\n\n3. One thing is that a bit unclear to me is whether the first component of the pipeline that generates the BCG using the triplet-GCN is really necessary. Have the authors tried to directly use the scene graph enhanced with the CLIP features and the bounding box features? This might be an interesting ablation to provide for the final version of the paper.\n\n4. One concurrent work that I think the authors should add in their reference list is the Learning 3D Scene Priors with 2D Supervision, CVPR 2023. I think this work could also be an interesting baseline as they can generate both the scene layout as well as the 3D shapes\n\n5. A minor comment/suggestion: In the caption of Figure 1 in the supplementary material the authors state \"shows a huge diversity\". I think huge might be slightly overclaiming. It might be better to tone it down a bit.\n\nLIMITATIONS:\nThe authors discuss the limitations of their work and the potential negative impacts in the society in their supplementary material. I think they could have also provided some qualitative examples of the failure case of their model to help the reader better understand the limits of this work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nGist:\nThe paper presents a framework, called CommonScenes, for generating 3D indoor scenes given scene graphs as inputs. CommonScenes is a dual-branch framework where one branch generates the scene layout using a VAE and the second one generates what the authors call \"compatible\" 3D shapes using latent diffusion. The claim is that having this second branch (where compatible 3D shapes are generated for populating the generated layout from the first branch) allows capturing global scene-object and local inter-object relationships, something that prior works cannot capture (I am not convinced about this, but more on this later). \n\nThe generated scenes can be manipulated by editing the input scene graph, as well as sampling noise in the diffusion process. \n\nThe paper also constructs a scene graph dataset using an off-the-shelf 3D scene dataset.\n\nDataset Used:\n3D-FRONT is the base dataset used, which is augmented with scene graph labels and this augmented dataset is termed in the paper as \"SG-FRONT\" dataset.\n\nTraining Mechanism:\nSupervised in the form of triplet network setting and latent diffusion models\n\n\nEvaluation Metrics:\nTo measure the fidelity and diversity of generated scenes, FID, KID scores at 256x256 pixel resolution between the top-down rendering of the generated and real scenes is used.\n\nTo measure shape diversity, each scene is generated 10 times, and the changes in shapes is evaluated using CD.\n\n\nBaselines and Comparisons:\nThree kinds of baselines are compared against:\n\n1) First, a retireval-based method, namely, 3D-SLN from CVPR 2020\n\n2) Second, a semi-generative SOTA method, Graph-to-3D from ICCV 2021, and\n\n3) Third, a text-to-shape generation methods that follows layout generation (this is not cited, so I am not informed by the paper if this was implemented by the authors on their own or if any specific algorithm was re-implemented)\n\nSTRENGTHS:\n+ Conceptualizing layout generation using graphs is a nice concept, although this is not the first time it has been addressed. A structured input modality gives rise to many applications, such as scene editing and modification, as demonstrated in the paper.\n\n+ As seen from Figure 4, the proposed method seems to produce plausible outputs given a scene graph as input. This is also validated quantitatively, although I would only consider Table 1 to be more representative of such quantification than other tables.\n\nWEAKNESSES:\n- Not really a concern but this is something that people will find about this paper: the paper is trying to do too many things at once. While this may also be a positive aspect in the era of today's models, a reader cannot clearly discern what design aspect leads to layout improvement and what leads to shape improvements. One may even argue: why not use the shape generation scheme employed as an independent approach and submit a paper if enough novelty exists? You get my point.\n\n\n- L96-97: How are the edge predicates (like spatial relations \"left/right\", \"closeby\" etc.) obtained? Is the dataset manually annotated with semantic scene graph information? If that is the case, then, the problem formulation is weak. What would have been interesting is to automatically extract meaningful semantic scene graphs (especially that ground spatial relations to a reasonable extent) and then use these graphs to generate a 3D scene.\n\n- L3-6: I do not understand the message in these lines. Do you mean to say that existing methods use retrieval-based mechanisms to populate the generated layout, because of which scene-object and object-object relationships are inconsistent? It is not true. So, first, I think it is important to rephrase this sentence. It is conveying an altogether different meaning.\n\n- There is mention of a triplet graph network, triplet-GCN in L46, 129, 148, 159. However, there is not much detail about how the positive and negative examples to train this triplet network are obtained. This is quite important since the training data plays a key role in obtaining meaningful and discriminative embedding spaces in the context of contrastive learning setups.\n\n- I fail to understand why the initial node embeddings, c_i^{node}, which are obtained from one-hot semantic encodings (as per info in L93), should be passed through a pre-trained and frozen CLIP text encoder. It makes sense to pass the initial __edge__ embeddings through it as spatial information needs to be captured and the CLIP text encoder does a good job of mapping the initial English text to a meaningful embedding space. But I cannot understand why the node embeddings c_i^{node} need to be passed through the text encoder from CLIP.\n\n- L37: It is not trivial to obtain text information from input scene graphs. This alternative solution is not so straightforward, unlike what is mentioned in this line.\n\n- L65-66, the sentence is incomplete\n\n- L87-88: LEGO-Net from CVPR 2023 is the first work to leverage diffusion models for scene generation. Even though LEGO-Net is designed for scene rearrangement, I would still place it in the generative model category since it is inherently doing denoising to provide a plausible output. So, these two lines are not correct.\n\n- Figure 4: Again, my question is how are the input scene graphs obtained? I am interested in knowing how the spatial relations on the graph edges are obtained. If this is done in a heuristic manner, it is prone to errors, and I do not think this is trivial to obtain in the presence of multiple objects. Obtaining such spatial relations is a challenge, as widely acknowledged in the community (see pre-LLM works on 3D indoor scene generation using language/text input, such as from Chang 2014, 2015, Ma 2018).\n\n- There are quite a few typos and punctuation errors in the paper (one such example is pointed out below). Need to be corrected.\nOne Missing period/full-stop symbol in line 24.\n\nQUESTIONS:\nPlease see the Weaknesses section above\n\nLIMITATIONS:\nThe paper does not discuss the limitations of the proposed approach.\n\nThere exist many questions (please see the Weaknesses section above) that can critically limit the application of the proposed approach, starting from the way the input scene graphs are obtained. At the least, a discussion on how this work can address or alleviate such challenges using additional processing should have been discussed.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-15"}