SlowGuess/NeurIPS_2023_OCR huggingface dataset:
SG-FRONT Dataset Due to the lack of scene graph datasets also providing high-quality object meshes, we construct SG-FRONT, a set of well-annotated scene graph labels, based on a 3D synthetic dataset 3D-FRONT [18] that offers professionally decorated household scenarios. The annotation labels can be grouped into three categories spatial/proximity, support, and style. The spatial relationships are based on a series of relationship checks between the objects, which control the object bounding box location, e.g., left/right, the comparative volumes, e.g., bigger/smaller, and heights, e.g., taller/shorter. The support relationships include directed structural support, e.g., close by, above, and standing on. Thresholds for these labels are iterated and set by the annotators. Finally, style relationships include the object attributes to assign labels, namely same material as, same shape as, and same super-category as. SG-FRONT contains around 45K samples from three different indoor scene types, covering 15 relationship types densely annotating scenes. More details are provided in the Supplementary Material.
Implementation Details We conduct the training, evaluation, and visualization of CommonScenes on a single NVIDIA A100 GPU with 40GB memory. We adopt the AdamW optimizer with an initial learning rate of 1e-4 to train the network in an end-to-end manner. We set  $\{\lambda_1,\lambda_2,\lambda_3\} = \{1.0,1.0,1.0\}$  in all our experiments.  $N_{c}$  in distribution  $Z$  is set to 128 and TSDF size  $D$  is set as 64. We provide more details in the Supplementary Material.
Figure 4: Qualitative comparison The orientations of Left/Right and Front/Behind in the scene graph align with the top-down view. Both scene-object and object-object inconsistencies are highlighted in red rectangles. Green rectangles emphasize the commonsense consistency our method produces.
Evaluation Metrics To measure the fidelity and diversity of generated scenes, we employ the commonly adopted Fréchet Inception Distance (FID) [20] & Kernel Inception Distance (KID) [3] metrics [44]. We project scenes onto a bird's-eye view, excluding lamps to prevent occlusion and using object colors as semantic indicators. To measure the scene graph consistency, we follow the scene graph constraints [13], which measure the accuracy of a set
Method	Shape Representation	Bedroom	Living room	Dining room	All
FID	KID	FID	KID	FID	KID	FID	KID
3D-SLN [35]	Retrieval	57.90	3.85	77.82	3.65	69.13	6.23	44.77	3.32
Progressive [13]	58.01	7.36	79.84	4.24	71.35	6.21	46.36	4.57
Graph-to-Box [13]	54.61	2.93	78.53	3.32	67.80	6.30	43.51	3.07
Ours w/o SB	52.69	2.82	76.52	2.08	65.10	6.11	42.07	2.23
Graph-to-3D [13]	DeepSDF [42]	63.72	17.02	82.96	11.07	72.51	12.74	50.29	7.96
Layout+txt2shape	SDFusion [8]	68.08	18.64	85.38	10.04	64.02	5.08	50.58	8.33
Ours	rel2shape	57.68	6.59	80.99	6.39	65.71	5.47	45.70	3.84
Figure 5: Consistency co-exists with diversity in different rounds. Our generated objects show diversity when activated twice while preserving the shape consistency within the scene (chairs in a suit).
Table 1: Scene generation realism as measured by FID and KID ( $\times 0.001$ ) scores at  $256^2$  pixels between the top-down rendering of generated and real scenes (lower is better). Two main rows are separated with respect to the reliance on an external shape database for retrieval. "Ours w/o SB" refers to ours without the shape branch.
Method	Consistency ↓	Diversity ↑
Chair	Table	Chair	Table
Graph-to-Box [13]	10.42	50.66	0.53	0.89
Graph-to-3D [13]	2.49	11.74	1.43	0.93
Ours	1.96	9.04	30.04	10.53
Table 2: Consistency and diversity in the dining rooms. The object shapes related with same as within a scene are consistent as indicated by low CD values  $(\times 0.001)$ , whereas the shapes across different runs have high diversity, per high KL divergence.
of relations on a generated layout. We calculate the spatial left/right, front/behind, smaller/larger, taller/shorter metrics, as well as more difficult proximity constraints close by and symmetrical. To measure the shape consistency, we test dining rooms, a typical scenario in which tables and chairs should be in suits according to the commonsense decoration. We identify matching dining chairs and tables in 3D-FRONT using the same CAD models and note their instance IDs. This helps us determine which entities belong to the same sets. We calculate Chamfer Distance (CD) of each of the two objects in each set after generation. To measure the shape diversity, we follow [13] to generate each scene 10 times and evaluate the change of corresponding shapes using CD. We illustrate the concepts of diversity and consistency in Figure 5. We also report MMD, COV, and 1-NNA following [66] for the object-level evaluation in the Supplementary Materials.
Compared Baselines We include three types of baseline methods. First, three retrieval-based methods, i.e., a layout generation network  $3D$ -SLN [35], a progressive method to add objects one-by-one designed in [13], and Graph-to-Box from [13]. Second, a semi-generative SOTA method Graph-to-3D. Third, an intuitive method called layout+txt2shape that we design according to the instruction in Sec. 1, stacking a layout network and a text-to-shape generation model in series, and it is a fully generative approach with text only. All baseline methods are trained on the SG-FRONT dataset following the public implementation and training details.
5.1 Graph Conditioned Scene Generation
Qualitative results The generated scenes from different baselines are shown in Figure 4: (a) Graph-to-Box retrieval, (b) Graph-to-3D, (c) layout+txt2shape, and (d) ours. It can be seen that our method has better shape consistency and diversity. Specifically, in terms of object shape quality, retrieval baseline scenes look optimal since they are based on real CAD models. Yet, the layout configuration is poor, and the retrieved object styles are not plausible, e.g., dining chairs and nightstands are not in a suit, respectively, in the same rooms. Graph-to-3D improves this by learning coherent scenes. However, the object styles are not conserved within scene styles, providing unrealistic scene-object inconsistency, e.g., in the bedroom, the height of the chair does not match the style and the height of the table. Subsequently, (c) improves the layout, which can be seen in the living room area, but again falls back on the generated shape quality and shares the same problem with Graph-to-Box on the poor object-object consistency. In contrast, CommonScenes can capture diverse environments considering both generation consistency and stylistic differences with respect to each scene. In the
Method	Shape Representation	Easy	Hard*
left / right	front / behind	smaller / larger	taller / shorter	close by*	symmetrical*
3D-SLN [35]	0.97	0.99	0.95	0.91	0.72	0.47
Progressive [13]	0.97	0.99	0.95	0.82	0.69	0.46
Graph-to-Box [13]	Retrieval	0.98	0.99	0.96	0.95	0.72	0.45
Ours w/o SB	0.98	0.99	0.97	0.95	0.74	0.63
Graph-to-3D [13]	DeepSDF [42]	0.98	0.99	0.97	0.95	0.74	0.57
Ours	rel2shape	0.98	1.00	0.97	0.95	0.77	0.60
Table 3: Scene graph constrains on the generation task (higher is better). The total accuracy is computed as the mean over the individual edge class accuracy to minimize class imbalance bias.
Method	Shape Representation	Mode	Easy	Hard*
left /right	front / behind	smaller / larger	taller / shorter	close by*	symmetrical*
3D-SLN [35]	0.89	0.90	0.55	0.58	0.10	0.09
Progressive [13]	Retrieval	0.89	0.89	0.52	0.55	0.08	0.09
Graph-to-Box [13]	change	0.91	0.91	0.86	0.91	0.66	0.53
Ours w/o SB	0.91	0.92	0.86	0.92	0.70	0.53
Graph-to-3D [13]	DeepSDF [42]	0.91	0.92	0.86	0.89	0.69	0.46
Ours	rel2shape	0.91	0.92	0.86	0.91	0.69	0.59
3D-SLN [35]	0.92	0.92	0.56	0.58	0.05	0.05
Progressive	Retrieval	0.92	0.91	0.53	0.54	0.02	0.06
Graph-to-Box [13]	addition	0.94	0.93	0.90	0.94	0.67	0.58
Ours w/o SB	0.95	0.95	0.90	0.94	0.73	0.63
Graph-to-3D [13]	DeepSDF [42]	0.94	0.95	0.91	0.93	0.63	0.47
Ours	rel2shape	0.95	0.95	0.91	0.95	0.70	0.61
Table 4: Scene graph constraints on the manipulation task (higher is better). The total accuracy is computed as the mean over the individual edge class accuracy to minimize class imbalance bias. Top: Relationship change mode. Bottom: Node addition mode.
living room, the scene graph input requires that Table No.3 and Table No.4 should stand close to each other. Graph-to-Box and Graph-to-3D fail to achieve the goal, which is also reflected in Table 3 and Table 4 that these kinds of methods cannot handle close by relation. We show more results in Supplementary Material.
Quantitative results We provide the FID/KID scores in Table 1 separated as average over three room types. Our method establishes the best results among the free-shape representation methods by improving the Graph-to-3D by approximately  $10\%$  on FID and  $40\%$  on KID on average. On the other hand, most of the layout+txt2shape results are worse than the previous state-of-the-art methods, proving the requirement of learning layout and shape jointly. Notably, the retrieval-based methods overall have better scores than those of free shapes since the retrieved objects align well with the test database. Further on, among the retrieval baselines, ours without the diffusion shape branch (Ours w/o SB) surpasses the prior works, indicating the benefit of contextual graph representation. Additionally, on the generation consistency, we can observe in Table 2 that our method can reflect the edge relationship in the generated shapes directly, indicated by significant improvement in the consistency score. By leveraging the latent diffusion model, our diversity is significantly improved compared with other methods. We provide more results on diversity and also show perceptual study on the evaluation of consistency and diversity in the Supplementary Material.
In terms of scene graph constraints, we present the results as easy and hard metrics in Table 3. On easy metrics, our methods (Ours, Ours w/o SB) are either better than or on par with other methods. While on hard metrics, they are superior to others. It shows that the spatial relationships could be learned easier than the proximity (close by) and commonsense (symmetrical) based ones. The improvement over the symmetrical constraint is particularly evident, as our method surpasses the state-of-the-art (SOTA) by  $10\%$ . This demonstrates that the integrated layout and shape cues are crucial for learning these contextual configurations.
5.2 Scene Manipulation
We demonstrate the downstream application of scene manipulation on our method and compare it with the aforementioned methods on scene graph constraints in Table 4. Our methods have the highest total score in both relation change and object addition modes. In both modes, the results are
on par with Graph-to-3D on easy relations, whereas there is a major improvement in the hard ones. Further, within the retrieval methods, Ours w/o SB improves over the baselines in the addition mode and is on par in the change mode. It could be argued that changing could be more difficult since it could alter multiple objects, whereas object insertion has less effect on the overall scene. We show some qualitative manipulation examples in the Supplementary Material.
5.3 Ablations
We ablate the primary components of CommonScenes in Table 5, including (1) scene generation from the original scene graph without contextual information brought by CLIP features, (2) scene generation from the contextual graph without the participation of the relational encoder  $E_{r}$ , (3) conditioning with concatenation on diffusion latent code instead of cross-attention, and (4) our final proposed method. We provide the mean FID scores over the scenes, as well as the mean over the hard scene graph constraints (mSG). We observe the benefit of both the context and  $E_{r}$  indicated by FID/KID scores, as well as the choice of our conditioning mechanism.
Figure 6: An interpenetrating phenomenon.
Condition	FID	KID	mSG
Ours w/o context	50.24	5.63	0.59
Ours w/o Er	51.62	5.41	0.73
Ours with concat	48.57	4.25	0.71
Ours	45.70	3.84	0.74
Table 5: Ablations under three circumstances.
We address the main aspects of the dataset and the limitations of our method and discuss more in the Supplementary Materials. First, the 3D-FRONT dataset used in our research contains significant noise, which we have mitigated through a post-processing step. Despite this effort, a small proportion of noisy data, specifically interpenetrating furniture instances, remains in the training dataset. Consequently, our proposed method and the baseline approaches reflect this during inference, rarely resulting in occurrences of scenes with collided objects. We show some cases in Figure 6. While our method outperforms others by effectively combining shape and layout information, it is essential to note that minor collision issues may still arise.
Scene graph-based CSS designs interactive environments suitable for a wide range of usages. Current methods heavily depend on object retrieval or pre-trained shape models, neglecting inter-object relationships and resulting in inconsistent synthesis. To tackle this problem, we introduce CommonScenes, a fully generative model that transforms scene graphs into corresponding commonsense 3D scenes. Our model regresses scene layouts via a variational auto-encoder, while generating satisfying shapes through latent diffusion, gaining higher shape diversity and capturing the global scene-object relationships and local object-object relationships. Additionally, we annotate a scene graph dataset, SG-FRONT, providing object relationships compatible with high-quality object-level meshes. Extensive experiments on SG-FRONT show that our method outperforms other methods in terms of generation consistency, quality, and diversity.
Acknowledgements This research is supported by Google unrestricted gift, the China Scholarship Council (CSC), and the Munich Center for Machine Learning (MCML). We are grateful to Google University Relationship GCP Credit Program for supporting this work by providing computational resources. Further, we thank all participants of the perceptual study.
[59] Johanna Wald, Helisa Dhamo, Nassir Navab, and Federico Tombari. Learning 3d semantic scene graphs from 3d indoor reconstructions. In CVPR, 2020. 3
[60] Kai Wang, Yu-An Lin, Ben Weissmann, Manolis Savva, Angel X Chang, and Daniel Ritchie. Planit: Planning and instantiating indoor scenes with relation graph and spatial prior networks. ACM Transactions on Graphics (TOG), 38(4):1-15, 2019. 3
[61] Kai Wang, Manolis Savva, Angel X Chang, and Daniel Ritchie. Deep convolutional priors for indoor scene synthesis. ACM Transactions on Graphics (TOG), 37(4):70, 2018. 3
[62] Xiaolong Wang and Abhinav Gupta. Generative image modeling using style and structure adversarial networks. In ECCV, 2016. 1
[63] Xinpeng Wang, Chandan Yeshwanth, and Matthias Nießner. Sceneformer: Indoor scene generation with transformers. In 3DV, 2021. 3
[64] Shun-Cheng Wu, Johanna Wald, Keisuke Tateno, Nassir Navab, and Federico Tombari. Scenegraphfusion: Incremental 3d scene graph prediction from rgb-d sequences. In CVPR, 2021. 3
[65] Jianghao Xiong, En-Lin Hsiang, Ziqian He, Tao Zhan, and Shin-Tson Wu. Augmented reality and virtual reality displays: emerging technologies and future perspectives. Light: Science & Applications, 10(1):216, 2021. 1
[66] Guandao Yang, Xun Huang, Zekun Hao, Ming-Yu Liu, Serge Belongie, and Bharath Hariharan. Pointflow: 3d point cloud generation with continuous normalizing flows. In ICCV, 2019. 8
[67] Ling Yang, Zhilin Huang, Yang Song, Shenda Hong, Guohao Li, Wentao Zhang, Bin Cui, Bernard Ghanem, and Ming-Hsuan Yang. Diffusion-based scene graph to image generation with masked contrastive pre-training, 2022. 3
[68] Guangyao Zhai, Xiaoni Cai, Dianye Huang, Yan Di, Fabian Manhardt, Federico Tombari, Nassir Navab, and Benjamin Busam. Sg-bot: Object rearrangement via coarse-to-fine robotic imagination on scene graphs, 2022. 1, 3
[69] Chenyangguang Zhang, Zhiqiang Lou, Yan Di, Federico Tombari, and Xiangyang Ji. Sst: Real-time end-to-end monocular 3d reconstruction via sparse spatial-temporal guidance. In ICME, 2023. 3
[70] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In ICCV, 2023. 3
[71] Bo Zhao, Lili Meng, Weidong Yin, and Leonid Signal. Image generation from layout. In CVPR, 2019. 3
[72] Yizhou Zhao, Kaixiang Lin, Zhiwei Jia, Qiaozi Gao, Govind Thattai, Jesse Thomason, and Gaurav S. Sukhatme. Luminous: Indoor scene generation for embodied ai challenges. In NeurIPS Workshop, 2021. 1
[73] Yang Zhou, Zachary While, and Evangelos Kalogerakis. Scenegraphnet: Neural message passing for 3d indoor scene augmentation. In ICCV, 2019. 3
[74] Ege Özsoy, Tobias Czempiel, Evin Pinar Örnek, Ulrich Eck, Federico Tombari, and Nassir Navab. Holistic or domain modeling: a semantic scene graph approach. International Journal of Computer Assisted Radiology and Surgery, pages 1-9, 2023. 3
[75] Ege Özsoy, Evin Pinar Örnek, Ulrich Eck, Tobias Czempiel, Federico Tombari, and Nassir Navab. 4d-or: Semantic scene graphs for or domain modeling. In MICCAI, 2022. 3