SlowGuess/NeurIPS_2023_OCR huggingface dataset:
CommonScenes: Generating Commonsense 3D Indoor Scenes with Scene Graph Diffusion
Guangyao Zhai $^{1,2*}$  Evin Pinar Örnek $^{1,2*}$  Shun-Cheng Wu $^{1}$  Yan Di $^{1\dagger}$  Federico Tombari $^{1,3}$  Nassir Navab $^{1,2}$  Benjamin Busam $^{1,2}$
{guangyao.zhai,evin.oernek,yan.di}@tum.de
$^{1}$ Technical University of Munich  $^{2}$ Munich Center for Machine Learning  $^{3}$ Google
https://sites.google.com/view/commonscenes
Controllable scene synthesis aims to create interactive environments for numerous industrial use cases. Scene graphs provide a highly suitable interface to facilitate these applications by abstracting the scene context in a compact manner. Existing methods, reliant on retrieval from extensive databases or pre-trained shape embeddings, often overlook scene-object and object-object relationships, leading to inconsistent results due to their limited generation capacity. To address this issue, we present CommonScenes, a fully generative model that converts scene graphs into corresponding controllable 3D scenes, which are semantically realistic and conform to commonsense. Our pipeline consists of two branches, one predicting the overall scene layout via a variational auto-encoder and the other generating compatible shapes via latent diffusion, capturing global scene-object and local inter-object relationships in the scene graph while preserving shape diversity. The generated scenes can be manipulated by editing the input scene graph and sampling the noise in the diffusion model. Due to the lack of a scene graph dataset offering high-quality object-level meshes with relations, we also construct SG-FRONT, enriching the off-the-shelf indoor dataset 3D-FRONT with additional scene graph labels. Extensive experiments are conducted on SG-FRONT, where CommonScenes shows clear advantages over other methods regarding generation consistency, quality, and diversity. Codes and the dataset are available on the website.
Controllable Scene Synthesis (CSS) refers to the process of generating or synthesizing scenes in a way that allows for specific entities of the scene to be controlled or manipulated. Existing methods operate on images [62] or 3D scenes [34] varying by controlling mechanisms from input scene graphs [24] or text prompts [5]. Along with the development of deep learning techniques, CSS demonstrates great potential in applications like the film and video game industry [7], augmented and virtual reality [65], and robotics [72, 68]. For these applications, scene graphs provide a powerful tool to abstract scene content, including scene context and object relationships.
This paper investigates scene graph-based CSS for generating coherent 3D scenes characterized by layouts and object shapes consistent with the input scene graph. To achieve this goal, recent methods propose two lines of solutions. The first line of works optimizes scene layouts [35] and retrieves objects [16] from a given database (see Figure 1 (a)). Such retrieval-based approaches are
*The first two authors contributed equally. † Corresponding author.
Figure 1: Architecture Comparison (Upper Row): Compared with previous methods, our fully generative model requires neither databases nor multiple category-level decoders. Performance Comparison (Bottom Row): We demonstrate the effectiveness of encapsulating scene-object and object-object relationships. The semantic information from the scene graph is 'a table is surrounded by three chairs'. As highlighted in the rounded rectangles, through the scene-object relationship, our network outperforms other methods by generating a round table and three evenly distributed chairs. Through the object-object relationship, the three chairs are consistent in style. Moreover, our method still preserves the object diversity (blue dashed rectangle).
inherently sub-optimal [57] in generation quality due to performance limitations through the size of the database. The second solution, e.g., Graph-to-3D [12], regresses both the layout and shape of the objects for synthesis. However, the shape generation relies on pre-trained shape codes from category-wise auto-decoders, e.g., DeepSDF [42]. This semi-generative design (Figure 1 (b)) results in reduced shape diversity in the generated outputs. To enhance generation diversity without relying on vast databases, one possible solution is to concurrently predict scene layouts and generate object shapes with text-driven 3D diffusion models[8, 32], where the textual information is obtained from input scene graphs. Yet, in our experiments, we observe that such an intuitive algorithm works poorly since it does not exploit global and local relationship cues among objects encompassed by the graph.
In this work, our approach CommonScenes exploits global and local scene-object relationships and demonstrates that a fully generative approach can effectively encapsulate and generate plausible 3D scenes without prior databases. Given a scene graph, during training, we first enhance it with pretrained visual-language model features, e.g., CLIP [46], and bounding box embeddings, incorporating coarse local inter-object relationships into the feature of each node. Then, we leverage a triplet-GCN [35] based framework to propagate information among objects, learning layouts through global cues and fine local inter-object relationships, which condition the diffusion process [47] to model the shape distribution as well. During inference, each node in the scene graph is enriched with the learned local-to-global context and sequentially fed into the latent diffusion model for each shape generation. Figure 1 (c) illustrates that our method effectively leverages relationship encoding to generate commonsense scenes, i.e., arranged plausibly and realistically, exhibiting scene-level consistency while preserving object shape diversity. Furthermore, to facilitate the benchmarking of CSS, we curate a novel indoor scene graph dataset, SG-FRONT, upon a synthetic dataset 3D-FRONT [18], since no existing indoor scene graph datasets provide high-quality meshes. SG-FRONT comprises around 45K 3D samples with annotated semantic and instance segmentation labels and a corresponding scene graph describing each scene.
Our contributions can be summarized into three points. First, we present CommonScenes, a fully generative model that converts scene graphs into corresponding 3D scenes using a diffusion model. It can be intuitively manipulated through graph editing. Second, CommonScenes concurrently models scene layout and shape distribution. It thereby encapsulates both global inter-object relationships and local shape cues. Third, we contribute SG-FRONT, a synthetic indoor dataset extending 3D-FRONT by scene graphs, thereby contributing graph-conditional scene generation benchmarks.
Scene Graph Scene graphs provide a rich symbolic and semantic representation of the scene using nodes and relationships [25]. They are useful in many 2D-related topics such as image
generation [24, 67], image manipulation [12], caption generation [31], visual question answering [55], and camera localization [27]. Quickly after this progress, they are used in following areas: 3D scene understanding [59, 1, 64, 29], dynamic modeling [49], robotic grounding [22, 50, 68], spatio-temporal 4D [75, 30, 74], and controllable scene synthesis [33, 73, 60, 41, 13].
Indoor 3D Scene Synthesis Controllable scene synthesis is extensively explored in the computer graphics community, varying from text-based scene generation [19] to segmentation map [43] or spatial layout-based image-to-image translation tasks [71]. Likewise in 3D, several methods generated scenes from images [56, 39, 2, 69, 14, 40, 38], text [36], probabilistic grammars [4, 23, 11, 45], layouts [26, 54], in an autoregressive manner [44, 63], or through learning deep priors [61]. Another line of work closer to ours is based on graph conditioning [33, 73, 60, 41]. Luo et al. [35] proposed a generative scene synthesis through variational modeling coupled with differentiable rendering. However, their method relies on shape retrieval, which depends on an existing database. Graph-to-3D [13] proposes a joint approach to learn both scene layout and shapes with a scene graph condition. Nevertheless, their object generation relies on a pre-trained shape decoder, limiting the generalizability. Unlike previous work, our method generates 3D scenes with the condition over a scene graph, which is trained end-to-end along with content awareness, resulting in higher variety and coherency.
**Denoising Diffusion Models** A diffusion probabilistic model is a trained Markov chain that generates samples matching data by reversing a process that gradually adds noise until the signal vanishes [52]. Diffusion models have quickly gained popularity due to their unbounded, realistic, and flexible generation capacity [21, 53, 37, 28, 47, 15]. However, studies have identified that the diffusion models lack compositional understanding of the input text [51]. Several advancements have been introduced to address these limitations. Techniques such as the introduction of generalizable conditioning and instruction mechanisms have emerged [5, 70]. Moreover, optimizing attention channels during testing has also been explored [6, 17]. Recently, a latent diffusion model using a Signed Distance Field (SDF) to represent 3D shapes was proposed contemporaneously at multiple works [8, 32], which can be conditioned on a text or a single view image. For the contextual generation conditioned on scene graphs, methods have converted triplets into the text to condition the model [17, 67], where Yang et al. [67] proposed a graph conditional image generation based on masked contrastive graph training. To the best of our knowledge, we are the first to leverage both areas of scene graph and latent diffusion for end-to-end 3D scene generation.
Scene Graph A scene graph, represented as  $\mathcal{G} = (\mathcal{V},\mathcal{E})$ , is a structured representation of a visual scene where  $\mathcal{V} = \{v_{i}\mid i\in \{1,\ldots ,N\} \}$  denotes the set of vertices (object nodes) and  $\mathcal{E} = \{e_{i\rightarrow j}\mid i,j\in \{1,\dots ,N\} ,i\neq j\}$  represents the set of directed edge from node  $v_{i}$  to  $v_{j}$ . Each vertex  $v_{i}$  is categorized through an object class  $c_{i}^{node}\in \mathcal{C}^{node}$ , where  $\mathcal{C}^{node}$  denotes the set of object classes. The directed edges in  $\mathcal{E}$  capture the relationships between objects in terms of both semantic and geometric information. Each edge  $e_{i\rightarrow j}$  has a predicate class  $c_{i\rightarrow j}^{edge}\in \mathcal{C}^{edge}$ , where  $\mathcal{C}^{edge}$  denotes the set of edge predicates. These relationships can incorporate various aspects, such as spatial locations (e.g., left/right, close by) or object properties (bigger/smaller). To facilitate subsequent processing and analysis, each node  $v_{i}$  and edge  $e_{i\rightarrow j}$  are typically transformed into learnable vectors  $o_i$  and  $\tau_{i\rightarrow j}$ , respectively, through embedding layers, as shown in Figure 2.A.
Conditional Latent Diffusion Model Diffusion models learn a target distribution by reversing a progressive noise diffusion process modeled by a fixed Markov Chain of length  $T$  [21, 53]. Fundamentally, given a sample  $\mathbf{x}_t$  from the latent space, gradual Gaussian Noise is added with a predefined scheduler  $\mathbf{x}_t$ ,  $t \in \{1, \dots, T\}$  [21]. Then, the denoiser  $\varepsilon_{\theta}$ , typically a UNet[48], is trained to recover denoising from those samples. The recently introduced Latent Diffusion Models (LDMs) [47] reduce the computational requirements by learning this distribution in a latent space established by a pre-trained VQ-VAE [58] instead of directly processing the full-size input. The popular usage of LDM is conditional LDM, which allows the generation to obey the input cue  $\mathbf{c}_i$  [47]. The training objective can be simplified to
$$
\mathcal {L} _ {L D M} = \mathbb {E} _ {\mathbf {x}, \varepsilon \sim \mathcal {N} (0, 1), t} \left[ \left| \left| \varepsilon - \varepsilon_ {\theta} \left(\mathbf {x} _ {t}, t, \mathbf {c} _ {i}\right) \right| \right| _ {2} ^ {2} \right], \tag {1}
$$
Figure 2: Scene Graph Evolution. Take the features of two nodes  $Bed(o_i)$ ,  $Table(o_j)$  and the linked edge  $In \, front \, of \, (\tau_{i\rightarrow j})$  as an example, where  $(o_i,o_j), \tau_{i\rightarrow j}$  are embedded learnable node features and the edge feature, respectively. We enhance the node and edge features with CLIP feature  $p_i, p_{i\rightarrow j}$  to obtain  $B$ . Contextual Graph. Then, we parameterize the ground truth bounding box  $b_i$  to the node to further build  $C$ . Box-Enhanced Contextual Graph with node and edge feature represented as  $f_{vc}^{(b)i} = \{p_i,o_i,b_i\}$ ,  $f_{ec}^{i\rightarrow j} = \{p_{i\rightarrow j},\tau_{i\rightarrow j}\}$ .
where  $\mathbf{c}_i$  denotes a conditioning vector corresponding to the input sample  $i$  fed into  $\varepsilon_{\theta}$ .
Overview Given a semantic scene graph, our approach endeavors to generate corresponding 3D scenes conforming to commonsense. We employ a dual-branch network starting with a contextual encoder  $E_{c}$ , as shown in Figure 3. The two branches, referred to as the Layout Branch and the Shape Branch, function simultaneously for layout regression and shape generation. In Sec. 4.1, we first illustrate how a scene graph evolves to a Box-enhanced Contextual Graph (BCG) with features from pre-trained visual-language model CLIP [46] and bounding box parameters (Figure 2). Then, we show how BCG is encoded by  $E_{c}$  and manipulated by the graph manipulator (Figure 3.A, B and C). In Sec. 4.2, we introduce the layout branch for layout decoding, and in Sec. 4.3, we introduce the shape branch for shape generation. Finally, we explain the joint optimization in Sec. 4.4.
4.1 Scene Graph Evolution
Contextual Graph As shown in Figure 2.A and B, we incorporate readily available prompt features from CLIP [46] as semantic anchors, capturing coarse inter-object information, into each node and edge of the input graph to conceptualize it as Contextual Graph  $\mathcal{G}_c$  with  $p_i = E_{\mathrm{CLIP}}(c_i^{node})$  for objects and  $p_{i\rightarrow j} = E_{\mathrm{CLIP}}(c_i^{node}\boxplus c_i^{edge}\boxplus c_j^{node})$  for edges. Here  $E_{\mathrm{CLIP}}$  is the pre-trained and frozen text encoder in [46],  $\boxplus$  denotes the aggregation operation on prompts including subject class  $c_i^{node}$ , predicate  $c_{i\rightarrow j}^{edge}$ , and object class  $c_j^{node}$ . Thereby, the embeddings of  $\mathcal{G}_c = (\mathcal{V}_c,\mathcal{E}_c)$  is formalized as,
$$
\mathcal {F} _ {\mathcal {V} _ {c}} = \left\{f _ {v _ {c}} ^ {i} = \left(p _ {i}, o _ {i}\right) \mid i \in \{1, \dots , N \} \right\}, \mathcal {F} _ {\mathcal {E} _ {c}} = \left\{f _ {e _ {c}} ^ {i \rightarrow j} = \left(p _ {i \rightarrow j}, \tau_ {i \rightarrow j}\right) \mid i, j \in \{1, \dots , N \} \right\}, \tag {2}
$$
where  $\mathcal{F}_{\mathcal{V}_c}$  represents the set of object features and  $\mathcal{F}_{\mathcal{E}_c}$  the edge features.
Box-Enhanced Contextual Graph In training, we enrich each node in the contextual graph by using ground truth bounding boxes parameterized by box sizes  $s$ , locations  $t$ , and the angular rotations along the vertical axis  $\alpha$ , yielding a BCG, as shown in Figure 2.C. Thereby, the node embeddings of BCG are represented by  $\mathcal{F}_{\mathcal{V}_c}^{(b)} = \{f_{v_c}^{(b)i} = (p_i,o_i,b_i) \mid i \in \{1,\dots,N\}\}$ , where  $b_i$  is obtained by encoding  $(s_i,t_i,\alpha_i)$  with MLPs. Note that BCG is only used during training, i.e., bounding box information is not needed during inference.
Graph Encoding BCG is encoded by the subsequent triplet-GCN-based contextual encoder  $E_{c}$ , which together with the layout decoder  $D_{l}$  in Sec. 4.2 construct a Conditional Variational Autoencoder (CVAE) [35]. The encoding procedure is shown in Figure 3.A, B and C. Given a BCG, during training, we input the embeddings  $\mathcal{F}_{\mathcal{V}_c}^{(b)}$  and  $\mathcal{F}_{\mathcal{E}_c}$  into  $E_{c}$  to obtain an Updated Contextual Graph with node and edge features represented as  $(\mathcal{F}_{\mathcal{V}_c}^{(z)},\mathcal{F}_{\mathcal{E}_c})$ , which is also the beginning point of the inference route. Each layer of  $E_{c}$  consists of two sequential MLPs  $\{g_1,g_2\}$ , where  $g_{1}$  performs message passing between connected nodes and updates the edge features,  $g_{2}$  aggregates features from all connected neighbors of each node and update its features, as shown in the follows:
$$
\begin{array}{l} \left(\psi_ {v _ {i}} ^ {l _ {g}}, \phi_ {e _ {i \rightarrow j}} ^ {l _ {g} + 1}, \psi_ {v _ {j}} ^ {l _ {g}}\right) = g _ {1} \left(\phi_ {v _ {i}} ^ {l _ {g}}, \phi_ {e _ {i \rightarrow j}} ^ {l _ {g}}, \phi_ {v _ {j}} ^ {l _ {g}}\right), \quad l _ {g} = 0, \ldots , L - 1, \\ \phi_ {v _ {i}} ^ {l _ {g} + 1} = \psi_ {v _ {i}} ^ {l _ {g}} + g _ {2} \left(\operatorname {A V G} \left(\psi_ {v _ {j}} ^ {l _ {g}} \mid v _ {j} \in N _ {\mathcal {G}} (v _ {i})\right)\right), \tag {3} \\ \end{array}
$$
Figure 3: Overview of CommonScenes. Our pipeline consists of shared modules and two collaborative branches Layout Branch and Shape Branch. Given a BCG (Figure 2.C), we first feed it into  $E_{c}$ , yielding a joint layout-shape distribution  $Z$ . We sample  $z_{i}$  from  $Z$  for each node, obtaining concatenated feature  $\{z_{i}, p_{i}, o_{i}\}$  with CLIP feature  $p_{i}$  and self-updated feature  $o_{i}$ . A graph manipulator is then optionally adopted to manipulate the graph for data augmentation. Next, the updated contextual graph is fed into the layout branch and shape branch for layout regression and shape generation respectively. In the shape branch, we leverage  $E_{r}$  to encapsulate global scene-object and local object-object relationships into graph nodes, which are then conditioned to  $\varepsilon_{\theta}$  in LDM via cross-attention mechanism to generate  $\mathbf{x}_0$  back in  $T$  steps. Finally, a frozen shape decoder (VQ-VAE) reconstructs  $S'$  using  $\mathbf{x}_0$ . The final scene is generated by fitting  $S'$  to layouts.
where  $l_{g}$  denotes a single layer in  $E_{c}$  and  $N_{\mathcal{G}}(v_i)$  includes all the connected neighbors of  $v_{i}$ . AVG refers to average pooling. We initialize the input embeddings of layer 0 as the features from the Updated Contextual Graph,  $(\phi_{v_i}^0,\phi_{e_{i\rightarrow j}}^0,\phi_{v_j}^0) = (f_{v_c}^{(b)i},f_{e_c}^{i\rightarrow j},f_{v_c}^{(b)j})$ . The final embedding  $\phi_{v_i}^L$  is used to model a joint layout-shape distribution  $Z$ , parameterized with the  $N_{c}$ -dimensional Gaussian distribution  $Z\sim N(\mu ,\sigma)$ , where  $\mu ,\sigma \in \mathbb{R}^{N_c}$ , predicted by two separated MLP heads. Thereby,  $Z$  is modeled through minimizing:
$$
\mathcal {L} _ {K L} = D _ {K L} \left(E _ {c} ^ {\theta} (z | x, \mathcal {F} _ {\mathcal {V} _ {c}} ^ {(b)}, \mathcal {F} _ {\mathcal {E} _ {c}}) | | p (z | x)\right), \tag {4}
$$
where  $D_{KL}$  is the Kullback-Liebler divergence measuring the discrepancy between  $Z$  and the posterior distribution  $p(z|x)$  chosen to be standard  $N(z\mid 0,1)$ . We sample a random vector  $z_{i}$  from  $Z$  for each node, update node feature  $\mathcal{F}_{\mathcal{V}_c}^{(z)}$  by concatenating  $\{z_i,p_i,o_i\}$ , keep edge features unchanged.
4.2 Layout Branch
As shown in Figure 3.D, a decoder  $D_{l}$ , which is another triplet-GCN, generates 3D layout predictions upon updated embeddings. In this branch,  $E_{c}$  and  $D_{l}$  are jointly optimized by Eq. (4) and a bounding box reconstruction loss:
$$
\mathcal {L} _ {\text {l a y o u t}} = \frac {1}{N} \sum_ {i = 1} ^ {N} \left(\left| s _ {i} - \hat {s} _ {i} \right| _ {1} + \left| t _ {i} - \hat {t} _ {i} \right| _ {1} - \sum_ {\lambda = 1} ^ {\Lambda} \alpha_ {i} ^ {\lambda} \log \hat {\alpha} _ {i} ^ {\lambda}\right), \tag {5}
$$
where  $\hat{s}_i, \hat{t}_i, \hat{\alpha}_i$  denote the predictions of bounding box size  $s$ , locations  $t$  and angular rotations  $\alpha$ , respectively.  $\lambda$  is the rotation classification label. We partition the rotation space into  $\Lambda$  bins, transforming the rotation regression problem into a classification problem.
4.3 Shape Branch
As shown in Figure 3.E, in parallel to the Layout Branch, we introduce Shape Branch to generate shapes for each node in the given graph, and represent them by SDF.
Relation Encoding The core idea of our method is to exploit global scene-object and local object-object relationships to guide the generation process. Hence, besides incorporating CLIP features for coarse local object-object relationships, as introduced in Section 4.1, we further design a relation encoder  $E_{r}$ , based on triplet-GCN as well, to encapsulate global semantic cues of the graph into
each node and propagate local shape cues between connected nodes. Specifically,  $E_{r}$  operates on the learned embeddings  $(\mathcal{F}_{\mathcal{V}_c}^{(z)},\mathcal{F}_{\mathcal{E}_c})$  of the Updated Contextual Graph, and updates the feature of each node with local-to-global semantic cues, yielding node-relation embeddings for subsequent diffusion-based shape generation.
Shape Decoding We use an LDM conditioned on node-relation embedding to model the shape-generation process. In terms of shape representation, we opt for truncated SDF (TSDF) [10] around the surface of the target object within a voxelized space  $S \in \mathbb{R}^{D \times D \times D}$ . Following the LDM, the dimension  $D$  of TSDF can be reduced by training a VQ-VAE [58] as a shape compressor to encode the 3D shape into latent dimensions  $\mathbf{x} \in \mathbb{R}^{d \times d \times d}$ , where  $d << D$  is built upon a discretized codebook. Then a forward diffusion process adds random noise to the input shape  $\mathbf{x}_0$  transferring to  $\mathbf{x}_T$ , upon which we deploy a 3D-UNet [9]  $\varepsilon_{\theta}$  (Figure 3.E) to denoise the latent code back to  $\mathbf{x}_0$  according to DDPM model by Ho et al. [21]. The denoiser is conditioned on node-relation embeddings to intermediate features of 3D UNet via the cross-attention mechanism. Finally, the decoder of VQ-VAE generates the shape  $S'$  from the reconstructed  $\mathbf{x}_0$ . For the denoising process at the timestep  $t$ , the training objective is to minimize:
$$
\mathcal {L} _ {\text {s h a p e}} = \mathbb {E} _ {\mathbf {x}, \varepsilon \sim \mathcal {N} (0, 1), t} \left[ | | \varepsilon - \varepsilon_ {\theta} (\mathbf {x} _ {t}, t, E _ {r} \left(\mathcal {F} _ {\mathcal {V} _ {c}} ^ {(z)}, \mathcal {F} _ {\mathcal {E} _ {c}}\right) | | _ {2} ^ {2} \right], \tag {6}
$$
where the evidence lower bound between the sampled noise  $\varepsilon$  and the prediction conditioned on the contextual relation embedding extracted from  $E_{r}$  is optimized. At test time, a latent vector is randomly sampled from  $\mathcal{N}(0,1)$  and progressively denoised by 3D-Unet to generate the final shape. Each shape within the layout is populated based on per-node conditioning, ultimately producing a plausible scene. Compared to prior work, our design can bring more diverse shape generation by taking advantage of the diffusion model architecture, as shown in experiments.
4.4 Layout-Shape Training
Our pipeline is trained jointly in an end-to-end fashion, allowing the Layout Branch and Shape Branch to optimize each other by sharing latent embeddings  $(\mathcal{F}_{\mathcal{V}}^{(z)},\mathcal{F}_{\mathcal{E}_c})$ , coming out of  $E_{c}$ . The final optimization loss is the combination of scene distribution modeling, layout generation, and shape generation:
$$
\mathcal {L} = \lambda_ {1} \mathcal {L} _ {K L} + \lambda_ {2} \mathcal {L} _ {\text {l a y o u t}} + \lambda_ {3} \mathcal {L} _ {\text {s h a p e}}, \tag {7}
$$
where  $\lambda_{1},\lambda_{2}$  and  $\lambda_{3}$  are weighting factors. Further insights on accomplishing the batched training are provided in Supplementary Material.