InstructScene: Instruction-Driven 3D Indoor Scene Synthesis with Semantic Graph Prior
Authors: Chenguo Lin
Yadong MU
Venue: ICLR 2024 spotlight (2024)
Keywords: 3D indoor scene synthesis
controllable generative models
graph diffusion models
Primary area: generative models
TL;DR: We propose to synthesize 3D indoor scenes from instructions by integrating a semantic graph prior and a layout decoder, significantly improving generation controllability and fidelity.

Abstract: Comprehending natural language instructions is a charming property for 3D indoor scene synthesis systems. Existing methods directly model object joint distributions and express object relations implicitly within a scene, thereby hindering the controllability of generation. We introduce InstructScene, a novel generative framework that integrates a semantic graph prior and a layout decoder to improve controllability and fidelity for 3D scene synthesis. The proposed semantic graph prior jointly learns scene appearances and layout distributions, exhibiting versatility across various downstream tasks in a zero-shot manner. To facilitate the benchmarking for text-driven 3D scene synthesis, we curate a high-quality dataset of scene-instruction pairs with large language and multimodal models. Extensive experimental results reveal that the proposed method surpasses existing state-of-the-art approaches by a large margin. Thorough ablation studies confirm the efficacy of crucial design components. Project page: https://chenguolin.github.io/projects/InstructScene.

Published as a conference paper at ICLR 2024
INSTRUCTSCENE: INSTRUCTION-DRIVEN 3D INDOOR SCENE S SYNTHESIS WITH SEMANTIC GRAPH PRIOR
Chenguo Lin, Yadong Mu" Peking University
chenguol inêstu. www.ssuuup.u......... .du.cn, en, mydēpku.. edu. cn
ABSTRACT
Comprehending natural language instructions is charming property for 3D in- door scene synthesis systems. Existing methods directly model object joint dis- tributions and express object relations implicitly within a scene, thereby hinder- ing the controllability of generation. We introduce INSTRUCTSCENE, a novel generative framework that integrates a semantic graph prior and a layout de- coder to improve controllability and fidelity for 3D scene synthesis. The pro- posed semantic graph prior jointly learns scene appearances and layout distribu- tions, exhibiting versatility across various downstream tasks zero-shot man- ner. To facilitate the benchmarking for text-driven 3D scene synthesis, we curate a high-quality dataset of scene- instruccinn pairs with large language and multi- modal models. Extensive experimental results reveal that the proposed method surpasses existing state-of-the-art approaches by a large margin. Thorough ab- lation studies confirm the efficacy of crucial design components. Project page: https: https////ccehguoiiinc gigggithhhhu io/ httppoojctss//////cooooooocsssss Inst Scene.
INTRODUCTION
Automatically synthesizing controllable and realistic 3D indoor scenes has been a persistent chal- lenge for computer vision and graphics (Merrell et al., 2011; I; Fisher et al., 2015; Qi et al., 2018;
Wang et al., 2018; Ritchie et al., 2019; Zhang et al., 2020; Yang et al., 1b;a; Höllein et al., 2023; Song al., 2023; Cohen-Bar et al., 2023; Lin et al., 2023; Feng et al., 2023; Patil et al., 2023). An ideal indoor scene synthesis system should fulfill at least three objectives: (1) comprehending instructions in natural languages, thus providing ing object compositions that exhibit aesthetic appeal and thematic harmony; (3) placing objects in appropriate positions and orientations adhering to their functions and regular arrangements.
intuitive and user-friendly interface; (2) design-
Natural instructions for interior design often rely on abstract object relationships, posing significant challenges for recent advancements 3D scene synthesis (Wang et al., 2021; Paschalidou et al., 2021: Liu et al., 2023a; Tang et al., 2023) due to the implicit modeling of relationships through indi- vidual object attributes. Other studies (Luo al., 2020; Dhamo et al., 2021; Zhai al., 2023) utilize relation graphs to provide explicit control over object interactions, which are however too compli- cated and fussy for human users to specify. Moreover, previous works primarily represent objects by only categories (Luo et al., 2020; Paschalidou et al., 2021) or low-dimensional features (Wang et al., 2019; Tang et al., 2023) which lack visual appearance details, resulting in style inconsistency and constraining customization options in scene synthesis.
To address these issues, we present INSTRUCTSCENE, a novel generative framework for 3D in- door scene synthesis with natural language instructions. The overview of the proposed method is illustrated in Figure 1. INSTRUCTSCENE comprises two parts: a semantic graph prior and a lay- out decoder. In the first stage, it takes instructions about partial interior arrangement and object appearances, and learns the conditional distribution of semantic graphs for holistic scenes. In the second stage, hamessing the well-structured and informative graph latents, the layout decoder can easily embody scenes that exhibit semantic consistency while closely adhering to the provided in- structions. With the learned semantic graph prior, INSTRUCTSCENE also achieves a wide range of instruction- generative tasks in zero-shot manner.
"Corresponding author.

---

Published as a conference paper at ICLR 2024
User Instructions
Semantic Graph Prior
Decoded 3D Scene
Position grey nighsstand the loft of wooden wardrobe with dours. Hang white lamp with a wooden circle direcily black double bed.
Ahooe
left
Design
Embooly
rigtro
Object Feature Codebook
Figure 1: Method overview. (1) INSTRUCTSCENE first designs holistic semantic graph based on user instruction. Within this graph, each node is an object endowed with semantic features and each edge represents a spatial relationship between objects. (2) It proceeds to place objects in scene by decoding precise 7 degrees--offrreedom attributes for each object from the informative graph prior.
Specific conditional diffusion models are devised for both parts of INSTRUCTSCENE. Benefitting from the two-stage scheme, it can separately handle discrete and continuous attributes of indoor scenes, drastically reducing the burden of network optimization. To enhance the capability of aes- thetic design, INSTRUCTSCENE also leverages object geometrics and appearances by quantizing semantic features from multimodal-aligned model (Radford al., 2021; Liu et al., 2023b).
To fit practical scenarios and promote the benchmarking of instruction-drive scene synthesis, we curate a high-quality dataset containing paired scenes and instructions with the help of large lan- guage and multimodal models (Li et al., 2022; Ouyang et al., 2022; OpenAI, 2023). Comprehensive quantitative evaluations reveal that INSTRUCTSCENE surpasses previous state-of-the-arr methods by a large margin in terms of both generation controllability and fidelity. Each essential component of our method is carefully verified through ablation studies.
Our contributions can be summarized as follows:
We present an instruction-ddrivn generative framework that integrates semantic graph prior and a layout decoder to improve the controllability and fidelity for 3D scene synthesis.
The proposed general semantic graph prior jointly models appearance and layout distribu- tions, facilitating various downstream applications in zero-shot manner.
We curate a high-quality dataset to promote the benchmarking of instruction-driven 3D scene synthesis. and quantitative experiments demonstrate that the proposed method sig- nificantly outperforms existing state-of-the -ftth--rrt techniques.
2 RELATED WORK
Graph-driven 3D Scene Synthesis Graphs have been used to guide complex scene synthesis in the form of scene bierarchies (Li et al., 2019; Gao et al., 2023), parse trees (Purkait et al., 2020), scene graphs (Zhou et al., 2019a; Para et al., 2021), etc. Wang et al. (2019) utilize an image-based module and condition its outputs on the edges of relation graph within each non-differentiable step. They also adopt autoregressive model (Li al., 2018) to generate relation graphs, which are however unconditional and with limited object attributes. Other works (Luo et al., 2020; Dhamo et al., 2021; Zhai et al., 2023) adopt conditional VAEs (Kingma & Welling, 2014; Sohn et al., 2015) with graph convolutional networks (Johnson et al., 2018) to generate layouts. While offering high controllability, these methods demand the specification of elaborate graph conditions, which are notably more intricate than those driven by natural languages.
Language-driven 3D Scene Synthesis Early studies on language-driven scene synthesis are con- ducted through procedural modeling, resulting in a semi-automatic process (Chang et al., 2014; 2015a; 2017; Ma et al., 2018). With the advent of attention mechanisms (Vaswani et al., 2017), recent approaches (Wang et al., 2021; Paschalidou et al., 2021; Liu et al., 2023a; Tang et al., 2023) can implicitly acquire object relations by self-attention and condition scene synthesis with texts by cross-attention. However, text prompts in these works tend to be relatively simple, containing only object categories or lacking layout descriptions, limiting the expressiveness and customization. Implicit relation modeling also significantly hinders their controllability.
2

---

Published as a conference paper at ICLR 2024
Generative Models for Graphs There have been lots of endeavors on generative models for undi- rected graphs, molecules and scene graphs by autoregressive models (You et al., 2018; Garg et al., 2021), VAEs (Simonovsky & Komodakis, 2018; Verma et al., 2022), GANs (De Cao & Kipf, 2018; Martinkus et al., 2022) and diffusion models (Niu et al., 2020; Jo et al., 2022; Vignac et al., 2023; Kong al., 2023). Longland et al. (2022) and Lo et al. (2023) employ VAE and GAN respectively for text-driven undirected simple graph generation without any semantics. In contrast, we present pioneering effort to generate holistic semantic graphs with expressive instructions.
3 PRELIMINARY: DIFFUSION MODELS
Diffusion generative models (Sohl-Dickstein et al., 2015) consist of a non-parametric forward pro- cess and a learnable reverse process. The forward process progressively corrupts a data point from ((xo) to sequence of increasingly noisy latent variables: q(x1:T|x xo) = I q(x: x,-1)) A neural network is trained to reverse the process by denoising them iteratively: Pe (Xo.ree)
p(xr) IL Pyl (x Xr, c), where is an optional condition guide the reverse process as needed. These two processes are supposed to admit p(XT) q(x (x)) for sufficiently large T. The gen- erative model is optimized by minimizing a variational upper bound on (xx())) [- log (xo)]:
T
Cvb == -B((x)) DKL (xx (x0))))(xx)
Σ 2
C
-Eg(x (x) Po (Xo x1, c)] ].
(1)
ET
where Lt-1 DKLLg(x4 Xg, X (x 1x,, c)] and Cr is constant during training can be ignored. DKLl-Il-1 indicates the KL divergence between two distributions.
METHOD
4.1
PROBLEM STATEMENT
Denote S = (S SM as a collection of indoor scenes. Each scene S; is composed of multiple objects O = 1 with distinct attributes of == {cj, si.. rj. ff.f including category cf (1...., Ke), where K2 is the number of object classes in S, location tj E R33 axis-aligned size 8) E R3, orientation rf E R and semantic feature fÿ E Rdd, where is the dimension of the feature vector. To set up 3D scene,
can generate each 3D object retrieve it from database by and They are then resized and transformed to the same scene coordinate by corresponding t. s and To simplify the process, we opt to retrieve 3D objects from high-quality dataset, and leave the generative models of each object conditioned on c and f for future work.
Given instructions y. our goal is to learn the conditional scene distribution y(Sly). Rather than direct modeling (Paschalidou et al., 2021; Tang et al., 2023), we employ well-structured and informative graphs to serve as general and semantic latents. Each graph G; contains a node set V; == { and directed edge set & == L Ivy, ví V.). A node vý functions as a high-level representation of object of. and directed edge e/e explicitly conveys the relations between objects.
To this end, propose generative framework, INSTRUCTSCENC that consists of two compo- nents: (1) semantic graph prior Po (gly) (Sec. 4.2) that jointly models high-level object and re- lation distributions conditioned y: (2) layout decoder po 9) (Sec. 4.3) that produces precise layout configurations with semantic graphs. Since G is deterministic by corresponding S, the two networks together yield an instruction-driven generative model for 3D indoor scenes:
Ps.0 (Sly) = Po. = (o ) (S(g).
(2)
4.2 SEMANTIC GRAPH PRIOR
The spatial relations are defined based on distances and relative orientations, such as "left", "closely in front of". "above" and "too far away (none)". Details about the definitions are provided in Ap- pendeix A. Layout configurations including t, and can be derived from spatial relations, so we leave them to the decoder po(S)g) Denote v == {c, f} and e E {l...., 1.....KK, K,), where K, is the number of relation classes.
3

---

Published as a conference paper at ICLR 2024
(a) Feature Quantization
(b) Discrete Semantic Graph Diffision
SUO Gy
Ge
Gr-1
Learabie Token
Recomitttial Foxure
Allimaleed Gragle
Pot0r-166..9
Ocginal Gragh
<(G.::1G,
pink
Texl Encoder
Graph Trunsformer
OpenStupe
caltnet Doural
Gp
CLIPelgedd Fattc
3D BAI Port Ulead
Desswing
Repeoanderiaiie
Reveameesd Gloph
Figure 2: Semantic Graph Prior. (a) Feature Quantization. Semantic features for 3D objects are extracted from frozen multimodal-aliin
point cloud encoder and then quantized by codebook entries. (b) Discrete Semantic Graph Diffusion. Three categorical variables in Gg are indepen- dently diffused; Empty states not depicted for concision; A graph Transformer with frozen text encoder learns the semantic graph prior by iteratively denoising corrupted graphs.
4.2.1 FEATURE QUANTIZATION
Hiig-dimmensionll features, such those with d = 1280 in OpenCLIP ViT-biig (Liu et al., 2023b), are too complicated to model. We circumvent this drawback by introducing a vector- quantized variational autoencoder for feature vectors, coined as fVQ-VAE The intuition behind is that there are general intrinsic characteristics shared among objects, encompassing attributes like colors, materials and basic geometric shapes. Indexing semantic features from codebook could dramatically reduce the cost of operating in a continuous space.
Formally, VAE contains pair of encoder E and decoder D, along with codebook Z RKyxdz, where K and dz are its size and dimension respectively. To concurrently capture object employ a multimodal-aligned point cloud encoder,
visual appearances and geometric shapes, OpenShape (Liu et al., 2023b), to extract object semantic features. The diagram for /VQ-VAE is presented in Figure ((a). is trained to maximize the evidence lower bound (ELBO) for log p(f):
Ex-pe(a(f) [log
3DK (pæ [p()))
where z E xd2 consists of ny vectors indexed by a sequence of scalars f= 1z where each scalar fm E (1...., ...... Ky). Since the quantization operation is non-differentiable, gumbel- softmax relaxation (Jang et al., 2016; Ramesh et al., 2021) adopted to optimize the ELBO.
4.2.2 DISCRETE SEMANTIC GRAPH DIFFUSION
After the feature quantization, all attributes in semantic graph are categorical, Gr (Ci, E,), where C; := {1,..., .......,, K.) & == {1,..., K. K. NNXN; and F; == {l,..., K N,xn,, While it is possible embed discrete variables in continuous spaces using one-hot encodings, diminishes the sparsity inherent in the original data and imposes a substantial burden on network optimization. Instead, we propose to model the semantic graph prior through discrete diffusion models.
For scalar discrete random variable with K categories x E {l...., K}. diffusion noise is defined by series of transition matrices Q R k The forward process timestep is expressed as ((Ix|2:(1) = x Q(x--1 where E IR is the column one-hor encoding for 2; and QQ1m = q(x; m Ft-1 n) the probability that 2t-1 transits the category m from n. The probabilistic distribution of 3, can be directly derived from g(Zx (mn) == x, Q.xo. where Q == Q. Q
Instead of commonly used Gaussian uniform transitions for graph generation (Niu et al., 2020; Hoogeboom al., 2021; et al., 2022: Vignac al., 2023), we propose to diffuse semantic graphs by independently masking graph attributes (i.e., object class c, quantized feature indices f and re- lation e) by introducing an absorbing state [MASK] (Austin et al., 2021; Gu et al., 2022) to each uniform transition matrix. For object class e, its transition matrix is defined as:
of 88
B
8
ai
BE
85
0
Q
82
BC
B
ai
B
2
4

---

Published as a conference paper at ICLR 2024
by which с₁ has a probability of to be masked, a probability of of to maintain the same, leaving chance - Yf a; for uniform sampling. [MASK] will always stay in its own state. Transition matrices for f and e, denoted as Qf and Q respectively, exhibit analogous structures. Schedules of (ar, 3e, 7t) are designed to admit that the initial states for semantic graphs are all masked.
Since the number of objects varies across different scenes, semantic graphs are padded by empty states to maintain consistent number of N objects. One-hot encodings for scalar variables c, and e in a scene are denoted as C E RxNx(K+2)), F E (=N and E E (NxxN((.+2)) respectively. Here "+2" accounts for the two extra states (i.e., empty state and mask state) for each variable. A one-hot encoded semantic graph Go (Co, Fo. Eo) at timestep t is formulated
G(G = C
Co. O.
Q.En)
(5)
The process for learning the graph prior is illustrated in Figure 2(b). The independent diffusion with mask states offers two significant advantages:
Perturbed states for one variable (e.g.. C) could be recovered by incorporating information from uncorrupted portions of the other variables (e.g.. F and E). compelling the semantic graph prior learning from the interactions among different scene attributes.
The introduction of mask states facilitates the distinction between corrupted states and clean ones, thus simplifying the denoising task.
These benefits are critical especially for intricate semantic graphs and diverse downstream generative tasks, compared with simple graph generative tasks (Niu et al., 2020; Jo et al., 2022; Vignac et al., 2023). Ablation study on the choice of is provided in Sec. 5.5.2.
Output of the graph prior network is parameterized to produce the clean scene graphs Go. which then diffused to get the predicted posterior for computing the variational bound in Equation I: Bo(G,-i/Gt,3 & Eg, g(G,-1/G,, (GolP.G Training objective for pp is a weighted summation of variational bounds for three random variables conditioned on y:
c == Ly + >5
FFly + de -
(6)
where de, Ay E R are hyperparameters to adjust the relative importance of three components in the semantic graph.
4.3 3D LAYOUT DECODER
Instantiating 3D scenes becomes easy with semantic graph prior. Denote layout configurations of S as {1:))= where 1j == of -v = {t), st. rf) SO(2) rotation parameterize by [cos(r), sin(r)) to continuously represent (Zhou et al., 2019b). Consequently, the layout of S; can be expressed as 2D matrices L, E R x Note that S (L, 9). generating indoor scenes (S (G) is equivalent to learning the conditional distributions of layout configurations pa (L)
diffusion model with variance- prrearce-preeeviigg Gaussian kernels (Ho et al., 2020; Song et al., 2020) is adopted to learn po (L) Its forward process is q(L:[oo) reverse process is modeled as PoLL,iILL 9) = _L (Le, t, 9); Eg (L,, t, G)). Following Ho et al. (2020), the variational bound in Equation 1 for the decoder po (L is reweighted and simplified:
/ ([, VoLLo, (1 T.) The
Lximple == ELO.te [||e co co,
ELO.t [||c- colo,,o vI o.e t,,,,,)))11
where t is sampled from uniform distribution (1(1, 7) and e is sampled from standard normal distribution AO I). Diagram of the layout decoder is depicted in Figure 3(a). Intuitively, the network is trained to predict noise e in the corrupted data L,,
4.4 MODEL. ARCHITECTURE
We the general-purpose Transformer (Vaswani al., 2017) for all models across tasks.
5

---

Published as a conference paper at ICLR 2024
3D Layout Decoder
(b) Graph Transformer
Geroin Nore
OOoCCo
Apertee
MLP
Robieve and Transoute
Graph
Transformer
Py(kg)
Hanstivly Penoising
Dacadee 10 Soone
Figure 3: (a) 3D Layout Decoder. Gaussian noises are sampled to attach at every node of semantic graphs; A graph Transformer processes these graphs iteratively to remove noises and generate layout configurations. (b) Graph Transformer. A graph Transformer consists of a stack of M blocks, each comprising graph attention, MIP and optional cross-attention modules; AdaLN and multi- head scheme are not depicted for concision.
Vanilla Transformer As illustrated in Figure 2(a), ny learnable tokens are employed with stack of cross-attentions to extract information from object features f in the encoder E in f'VQ-VAE. Regarding the decoder D, ny vectors retrieved from the codebook 2 are fed to another Transformer, and an average pooling on the top of it is applied to aggregate information.
Graph Transformer The prior and decoder share the same model architecture as shown in Fig- ure 3(b). Since relation Ejk can be determined by ekj. only the upper triangular part of the lation matrix is necessary. Object categories and features together form input tokens for Trans- formers. Message passing on graphs is operated via node self-attention and node-edge fusion with FiLM (Perez et al., 2018), which linearly modulates edge embeddings and node attention matrices before softmax (Dwivedi & Bresson, 2021; Vignac et al., 2023). Timestep for diffusion / is injected by AdaLN (Ba et al., 2016; Dhariwal & Nichol, 2021). In the prior pe (9ly), instructions are embed- ded by a frozen text encoder and consistently influence network outputs by rross-ateention mecha- nisms. Layout decoder pe (Slg) is conditioned on semantic graphs by appending sampled Gaussian noises on node embeddings, which are then iteratively denoised to produce layout attributes.
Permutation Non-invariance Although 9 should ideally remain invariant node permutations, invariant diffusion models could encounter learning challenges for multi-mode modeling. Thus, each node feature is added with positional encodings (Vaswani et al., 2017; Tang et al., 2023; Lei et al., 2023) before the permuration-eqqiiaaiinn Transformer. Exchangeability for graph prior distri- butions is strived by random permutation augmentation during the training process. Ablation on the permutation non-invariance is provided in Sec. 5.5.2.
EXPERIMENTS
5.1 SCENE-INSTRUCTION PAIR DATASET
All experiments are conducted on 3D-FRONT al., 2021a), a professionally designed collection of synthetic indoor scenes. However, it does not contain any descriptions of room layouts or object appearances. To construct a high-quality ceneeinstruuctio paired dataset, we initially extract view- dependent spatial relations with predefined rules. The dataset is further enhanced by captioning objects with BLIP (Li et al., 2022). To ensure the accuracy of descriptions, the generated captions filtered by ChatGPT (Ouyang et al., 2022; OpenAI, 2023) with object ground-truth categories. The final instructions are derived from randomly selected relation triplets. Details on dataset curation can be found in Appendix A.
5.2 EXPERIMENTAL SETTINGS
Baselines We compare our method with two state-of-the-art approaches for 3D scene generative tasks: (1) ATISS (Paschalidou et al., 2021), a Transformer-based auto-regressive network that gards scenes as sets of unordered objects, and generates objects and their attributes sequentially. (2) DiffuScene (Tang et al., 2023), diffusion model with Gaussian kernels that treats object at- tributes in one scene as a 2D matrix after padding them to a fixed size. Both of these methods can be conditioned on text prompts by cross-attention with pretrained text encoder. Our preliminary

---

Published as a conference paper at ICLR 2024
Table 1: Quantitive evaluations for instruction-driven synthesis by ATISS (Paschalidou al., 2021), DiffuScene (Tang et al., 2023) and our method on three room types. Higher iRecall, lower FID, FIDCLIP and KID indicate better synthesis quality. For SCA, score closer to 50% is better. Standard deviation values are provided subscripts.
cneruction-driverr Synthesis
iRecally
FID
FIDCUIP
KID xle3
SCAg
ATISS
48.13.2.5
1.55
6,95
0.39
59.17
Bedroom
DiffuScene
56.43:1207
123.0910.79
7.13:10.16
0.39
60.49:12.96
Ours
73.64+1..7
114.78#1.19
6.65++0118
0.32=0.03
56,02#1143
ATISS
29.50:13.67
117.67::2.32
6.08
17.60
69.38
Living room
DiffuScene
31.15++2.99
122.20#1.09
6.10=+0.1
16.49
12.92++129
Ours
56.81
110.39
5.07
8,16
65.42:12.52
ATISS
17.58:
13+.30
8,49+
23.60=2.52
67. +3.23
Dining room
DiffuScene
37.87:22766
145.48
8.631031
24.08 a:
70.57
Ours
61.23
129.76.1.61
7.67
13.24
64.20 1..0
experiments suggest that both baselines encounter difficulties in modeling high high-dimensional seman- tic feature distributions, consequently impacting their performance in generating other attributes. Therefore, we augment them to generate quantized features. Further implementation details about baselines and method are provided in Appendix B.1 and B.2.
Evaluation Metrics To assess the controllability of layouts, use metric named "instruction recall" (iRecall), which quantifies the proportion of the required triplets "(subject, relation, object)" occurring in synthesized scenes to all provided in instructions. It is stringent metric that takes into account all three elements in a layout relation simultaneously. Following previous works (Paschali- dou al., 2021; Liu et al., 2023a; Tang et al., 2023), we also report Fréchet Inception Distance (FID) (Heusel et al., 2017), CLP (Kynkåänniemi et al., 2022), which computes FID scores by CLIP features (Radford et al., 2021), 1), Kernel Inception Distance (KID) (Bińkowski et al., 2018), scene classification accuracy (SCA). These metrics evaluate the overall quality of synthesized scenes and rely on rendered images. We use Blender (Community, 2018) to produce high-qualite images for both synthesized and real scenes. For more details on evaluation, please refer to Appendix B.3.
5.3 INSTRUCTIIO-DDIVER SCENE SYNTHESIS
Table presents the quantitive evaluations for synthesizing 3D scenes with instructions. We report the average scores of five runs with different random seeds. As demonstrated, even with the en- hancement of quantized semantic features, two baseline methods continue to demonstrate inferior performance compared to ours. ATISS outperforms DiffuScene in terms of generation fidelity, ow- ing to its capacity to model in discrete spaces. DiffuScene shows better controllability ATISS because affords global visibility of samples during generation. Our proposed INSTRUCTŠCENE exhibits the best of both worlds. Remarkably, achieve substantial advancement in controllabil- ity, measured iRecall, for scene generative models, surpassing current state-of-the the-art approaches by about 15 % 25 % across various room types, all while maintaining high fidelity. It is noteworthy that INSRRUCTSCENE excels in handling more complex scenes, such as living and dining rooms, which typically comprise average of 20 objects, in contrast to bedrooms, which have only ob- jects on average, revealing the benefits of modeling intricate 3D scenes associated with the semantic graph prior. Qualitative visualizations are provided in Appendix C.I.
5.4 ZERO-SHOT APPLICATIONS
Thanks the discrete design and mask modeling, the learned semantic graph prior is capable of diverse downstream tasks without any fine-tuning. We investigate four zero-shot tasks, including stylization, re-arrangement, completion, and unconditional generation. The first three tasks can be regarded as conditional synthesis guided by both instructions and partial scene attributes.
Stylization and re-arrangement task can be formulated as p (flc, 1, y) and (t. s, rlc, f.y) respectively. In the completion task, we intend to add new objects (of) to partial scene S, with

---

Published as a conference paper at ICLR 2024
Table Quantitive evaluations for zero-shot generative applications on three room types. "Uncond." stands for unconditional scene synthesis.
Zero-shot
Stylization
Re-arrangement
Completion
Uncond.
Applications
1Ax Xxle-3
FID
1 iRecallx
FID
iRecallx
1 FID
FID
ATISS
3,44
123.91
61.22
107.67
64.90
89.77
134.51
Bedroom
DiffuScene
1.08
127.35
68.57
106.15
48.57
96.28
135.46
Ours
6.34
122.73
79.59
105.27
69.80
82.98
124.97
ATISS
-3.57
110.85
31.97
117.97
43,20
106.48
129.23
Living room
DiffuSeene
-2.69
112.80
41.50
115.30
19.73
95.94
129.75
Ours
0.28
109.39
56.12
106.85
46.94
92.52
117.62
ATISS
-1.11
131.14
36.06
134.54
57.99
122.44
147.52
Dining room
DiffuScene
-2.98
135.20
46.84
133.73
32.34
115.08
150.81
Ours
1.69
128.78
62.08
125.07
60.59
107.86
137.52
instructions. By filling the partial scene attributes with [MASK] tokens, we treat them as intermedi- ate states during discrete graph denoising, allowing for a straightforward adaptation of the learned semantic graph prior these tasks in zero-shot manner. Unconditional synthesis is implemented by simply setting text features as zeros. To assess controllability in the stylization task, we define A == N Σi= CosSim(f,, d CosSim(f;, dglasss, where de represents the CLIP text fea- ture of object class name with the desired style, and di:luss is the CLIP text feature with only class information. CosSim(:.)) calculates the cosine similarity between two vectors.
Evaluations zero-shot applications reported in Table 2. Our method consistently outperforms two strong baselines in both controllability and fidelity. While ATISS, as an auto-regressiv model, natural fit for the completion task, its unidirectional dependency chain limits its effectiveness for tasks requiring global scene modeling, such as re-arrangement. DiffuScene can adapt to these tasks by replacing the known parts with the noised corresponding scene attributes during sampling, similar to image in-painting (Meng et al., 2021; Nichol et al., 2022). However, the known attributes greatly corrupted in the early steps, which could misguide the denoising direction, and therefore necessitate fine-tuning, Additionally. DiffuScene also faces challenges in searching for semantic features in continuous space for stylization. contrast, INSTR globally models scene attributes and treats partial scene attributes as intermediate discrete states during training. These designs effectively eliminate the training-test gap, rendering highly versatile for a wide range of downstream tasks. Visualizations of zero-shot applications are available in Appendix C.2.
5.5
ABLATION STUDIES
5.5.1
CONFIGURATIONS FOR DIFFUSION MODELS
Diffusion Timesteps Although containing two diffusion models, our method could achieve bet- ter efficiency by reducing the steps of reverse processes without noticeable decline in perfor- mance. This stems from the fact that each stage in INSTRUCISCENE tackles an casier denoising task compared the single-stage DiffuScene. Following the original setting of Tang et al. (2023), the timestep value (T) for DiffuScene is set to 1000. While for INSTRUCTSCENE, we find T =100 and 10 is sufficient for (o(gly) and (S(G) respectively. Evaluation results with different timesteps are presented in Figure 4(a), with values averaged on three room types. The plotted timesteps for our method are "100+ 1000". "100+4000, "100+100", "100+10", "50+10" and "25+10". where the first number represents T for the prior and the latter is for the decoder.
Classifier-Free Guidance Classifier-fre free guidance (CFG) (Ho & Salimans, 2021) is widely used technique to trade off controllability with diversity. We not adopt it in previous experiments for fair comparison, as the sequential attribute decoding hinders ATISS from realizing the benefits offered by CFG. To assess its effectiveness for diffusion models, we randomly remove instruction conditions 20% of samples during training, inducing unconditional generation. At inference, CFG is implemented by adjusting conditional ligeliiellsoods away from unconditional counterparts:
Do (oo = (1 5) Poo y) po((Go).)),
(8)
8

---

Published as a conference paper at ICLR 2024
Timestaps
(3) Classifie-Feee
1 Rieat th
-35
126
130
DiftScenne
D/PNdoore
defuzzere
124
-25
:20
1:0
CTG Scale
CTG Scale
Figure 4: Ablation studies configurations for diffusion models, including diffusion timesteps and classifier-free guidance scales.
Table 3:
Ablation studies on different strategies to learn semantic graph prior poly) "Perm. Invar." means permutation-invariin graph modeling.
Graph Prior
Ours
Gaussian
Joint Mask
Uniform
Perm. Invar.
1 iRecally
73.64+1337
34.18+2255
34.71
69.22+3.25
70,49=2.50
+ FID
114.78:1.19
128.98
130.86:1.766
139.61
116.53
FIDCLIP
6.65 1:0.88
7.30
7.59
8.82.1024
6.69 1..16
4 KID
0.32
2..73
4.82
10.55
0.37
SCA%
56.02:10.9
57.1013.22
60.37
76.7913.14
58.64
where s is hyperparameter to control the scale of CFG. Performance for diffusion-based models with different CFG scales are plotted in Figure 4(b), where values averaged over three room types. Within an appropriate range of scales, CFG can effectively enhance the controllability for catructive-driverr 3D scene synthesis, while large scales lead to performance decline. Though DiffuScene also benefits from CFG, our method still significantly outperforms it in both metrics.
5.5.2 LEARNING SEMANTIC GRAPH PRIOR
We explore different strategies learn the proposed semantic graph prior. All experiments conducted on the bedroom dataset. Quantitative results are presented in Table 3.
Transition Matrices for Learning Graph Prior We investigate the effects of different transition matrices for learning the proposed semantic graph prior, including: (1) Embed all categorical vari- ables into their one-hot encodings and diffuse them by Gaussian kernels, which is similar Niu et al. (2020) and Jo et al. (2022); (2) Jointly masking and E along with nodes C in graph, so only the attributes of other objects can be utilized for recovery; (3) Adopt uniform transition matri- ces without mask states, which is similar to Vignac et al. (2023). Evaluations on both controllability and fidelity reveal the advantages our independent mask strategy.
Permutation Non-invariance Unlike previous studies on graph generation (Niu et al., 2020; Jo al., 2022; Vignac al., 2023), we depart from the convention of permutation-invarian modeling to ease the learning process of semantic graph prior. We strive to preserve exchangeable graph distributions by randomly shuffling object orders during training. Performance for invariant graph prior is provided in the last column of Table 3. Its performance declines due to the unnecessary imposition of invariance in scene synthesis.
CONCLUSION
By integrating a semantic graph prior and a layout decoder, we propose a novel generative frame- work, INSTRUCTSCENE, that significantly improves the controllability and fidelity of 3D indoor scene synthesis, providing a user-friendly interface through instructions in natural languages. Bene- fits from the design of semantic graph prior, method can also apply to diverse applications with- out any fine-tuning. The controllability and versatility positions INSTRUCTSCENE as promising tool. We hope this work could help in practical scenarios, such facilitating interior design, deliv- ering immersive metaverse experiences, simulations for embodied agents, developing outting-edge VR/AR applications, etc. We discuss the limitations of our method and future work in Appendix D.

---

Published as a conference paper at ICLR 2024
ETHICS STATEMENT
Several large pretrained models are incorporated in this work, including OpenShape (Liu et al., 2023b) for object semantic feature extraction, CLIP (Radford al., 2021) for text feature extraction, BLIP (Li et al., 2022) for object captioning and ChatGPT (Ouyang et al., 2022; OpenAl, 2023) for caption refinement. Most of these models are trained on large-scale datasets collected from the web, lacking rigorous content filtering, thereby potentially encompassing harmful material. We curate the dataset and train our method based on these models, thus may inherit these imperfections. Given that our generative framework is trained only indoor scene datasets, it exhibits less probability of propagating negative consequences compared to the synthesis and editing methods on human faces and natural images. Nevertheless, will still explicitly specify permissible applications of our system through appropriate licensing to mitigate potential adverse societal impacts.
REPRODUCIBILITY STATEMENT
To ensure the reproducibility of our method, we include the details of dataset processing in Appen- dex A, including the rule-based spatial relation definitions (A.1) and the used prompt and hyper- parameters for ChatGPT to refine object descriptions (A.2). Implementation details also pro- vided in Appdex B, including baseline reproductions (B.1), model hyperparameter disclosure (B.2) and evaluation metric computations (B.3). Our instruction-ssene pair dataset and code for both training and evaluation can be found in InstructScene.
in gi thub. io projjetss
ACKNOWLAGEMENT
This work is supported by National Key R&D Program of China (2022ZD1160305).
REFERENCES
Jacob Austin, Daniel D Johnson, Jonathan Ho, Daniel Tarlow, and Rianne Van Den Berg. Structured denoising diffusion models in discrete state-spaces. Advances in Neural Information Processing Systems (NeurIPS), 34:17981-17999 2021.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. 2016.
arXiv preprint
Mikolaj Binkowski, Danica J Sutherland, Michael Arbel, and Arthur Gretton, Demystifying mmd gans. In International Conference on Learning Representations (ICLR), 2018.
Angel Chang, Manolis Savva, and Christopher D Manning. Learning spatial knowledge for text to 3d scene generation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 2028-2038, 2014.
Angel Chang. Will Monroe, Manolis Savva, Christopher Potts, and Christopher D Manning. Text to 3d scene generation with rich lexical grounding. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics (ACL), pp. 53-62, 2015a.
Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al. Shapenet: An information-rict 3d model repository. arXiv preprint arXiv: 2015b.
Angel Chang, Mihail Eric, Manolis Savva, and Christopher D Manning. Sceneseer: scene design with natural language. arXiv preprint 1rx3::1703.00050 2017.
Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gor- don, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-image learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2818-2829, 2023.
10

---

Published as a conference paper at ICLR 2024
Dana Cohen-Ban Bar, Elad Richardson, Gal Metzer, Raja Giryes, and Daniel Cohen-Or. Set-the-scene: Global-local training for generating controllable nerf scenes. In IEEE/CVF International Confer- on Computer Vision Workshops (ICCVW), 2023.
Blender Online Community, Blender - a 3D modelling and rendering package. Blender Foundation, Stichting Blender Foundation, Amsterdam, 2018. URL http:/ /wwwwwwww.........////..... blender. org.
Nicola De Cao and Thomas Kipf. MolGAN: An implicit generative model for small molecular graphs. ICML 2018 workshop Theoretical Foundations and Applications of Deep Generative Models, 2018.
Matt Deitke, Ruoshi Liu, Matthew Wallingford, Huong Ngo, Oscar Michel, Aditya Kusupati, Alan Fan, Christian Laforte, Vikram Voleti, Samir Yitzhak Gadre, et al. Objaverse- xl: A universe of 10m+ 3d objects. Advances in Neural Information Processing Systems (NeurIPS), 2023.
Helisa Dhamo, Fabian Manhardt, Nassir Navab, and Federico Tombari. Graph-to-3d: End-to-end generation and manipulation of 3d scenes using scene graphs. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 16352-16361, 2021.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans image synthesis. Advances in Neural Information Processing Systems (NeurIPS), 34:8780-8799 8794, 2021.
Vijay Prakash Dwivedi and Xavier Bresson. A generalization of transformer networks to graphs. In AAAI Workshop on Deep Learning on Graphs: Methods and Applications, 2021.
Weixi Feng, Wanrong Zhu, Tsu-Jui Fu, Varun Jampani, Arjun Reddy Akula, Xuehai He, S Basu, Xin Eric Wang, and William Yang Wang. LayoutGPT: Compositional visual planning and gen- cration with large language models. In Advances in Neural Information Processing Systems (NeurIPS), 2023.
Matthew Fisher, Manolis Savva, Yangyan Li, Pat Hanrahan, and Matthias Nicßner. Activity-centric scene synthesis for functional 3d scene modeling. ACM Transactions on Graphics (TOG), 34(6); 1-13, 2015.
Huan Fu, Bowen Cai, Lin Gao, Ling-Xiao Zhang, Jiaming Wang, Cao Li, Qixun Zeng. Chengyue Sun, Rongfei Jia, Binqiang Zhao, et al. 3d-front: 3d furnished rooms with layouts and seman- tics. In Proceedings of the JEFE/CVF International Conference on Computer Vision (ICCV), pp- 00933-10942. 2021a.
Huan Fu, Rongfei Jia, Lin Gao, Mingming Gong, Binqiang Zhao, Steve Maybank, and Dacheng Tao. 3d-future: 3d furniture shape with texture. International Journal of Computer Vision (IJCV), 129:3313-3333 2021b.
Lin Gao, Jia-Mu Sun, Kaichun Mo, Yu-Kun Lai, Leonidas J Guibas, and Jie Yang. Scenehgn: Hierarchical graph networks for 3d indoor scene generation with fine ane-grained geometry. JEEE Transactions Pattern Analysis and Machine Intelligence (T-PAMI), 2023.
Sarthak Garg, Helisa Dhamo, Azade Farshad, Sabrina Musatian, Nassir Navab, and Federico Tombari. Unconditional scene graph generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 16362-16371, 2021.
Shuyang Gu, Dong Chen, Jianmin Bao, Fang Wen, Bo Zhang, Dongdong Chen, Lu Yuan, and Baining Guo. Vector quantized diffusion model for text-to- text-to-image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition (CVPR), pp. 10696-10706. 2022.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in Neural Information Processing Systems (NeurIPS), 30, 2017.
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In NewrIPS 2021 Workshop on Deep Generative Models and Downstream Applications, 2021.

---

Published as a conference paper at ICLR 2024
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems (NeurIPS), 53:6840-6853 2020.
Lukas Höllein, Ang Cao, Andrew Owens, Justin Johnson, and Matthias Nießner. Text2room: Ex- tracting textured 3d meshes from 2d text-to-image models. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), pp. 7909-7920, 2023.
Emiel Hoogeboom, Didrik Nielsen, Priyank Jaini, Patrick Forré, and Max Welling, Argmax flows and multinomial diffusion: Learning categorical distributions. Advances in Neural Information Processing Systems (NewrIPS), 34: 2454-12465, 2021.
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterizatiion with gumbel-softmax. In International Conference on Learning Representations (ICIR), 2016.
Jachyeong Jo, Seul Lee, and Sung Ju Hwang. Score-based generative modeling of graphs via the system of stochastic differential equations. International Conference on Machine Learning (ICML), pp. 10362-10383. PMLR, 2022.
Justin Johnson, Agrim Gupta, and Li Fei-Fei. Image generation from scene graphs. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1229-1228, 2018.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. In International Conference on Learning Representations (ICLR), 2014.
Lingkai Kong, Jiaming Cui, Haotian Sun, Yuchen Zhuang, B. Aditya Prakash, and Chao Zhang. Autoregressive diffusion model for graph generation. In International Conference on Machine Learning (ICML), volume 202. pp. 17391-17408. PMLR, 2023.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton.
Imagenet classification with deep con- volutional neural networks. Advances in Neural Information Processing Systems (NeurIPS), 25, 2012.
Tuomas Kynkäänniemi, Tero Karras, Miika Aittala, Timo Aila, and Jaakko Lehtinen. The role of imagenet classes in fréchet inception distance. In International Conference on Learning Repre- sentations (ICLR), 2022.
Jiahui Lei, Congyue Deng, Bokui Shen, Leonidas Guibas, and Kostas Danillidis. Nap: Neural 3d articulation prior. Advances Neural Information Processing Systems (NeurIPS), 2023.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre- training for unified vision-language understanding and generation. In International Conference on Machine Learning (ICML), pp. 12888-12900. PMLR, 2022.
Manyi Li, Akshay Gadi Patil, Kai Xu, Siddhartha Chaudhuri, Owais Khan, Ariel Shamir, Changhe Tu, Baoquan Chen, Daniel Cohen-Or, and Hao Zhang, Grains: Generative recursive autoencoders for indoor scenes. ACM Trunsactions on Graphics (TOG), 3862):1-16, 2019.
Yujia Li, Oriol Vinyals, Chris Dyer, Razvan Pascanu, and Peter Battaglia. Learning deep generative models of graphs. arXiv preprint arXiv:1803.03322 2018.
Yiqi Lin, Haotian Bai, Sijia Li, Haonan Lu, Xiaodong Lin, Hui Xiong, and Lin Wang. Compon- erf: Text-guided multi-object compositional nerf with editable 3d scene layout. arXiv preprint arXiv:2303.13843 2023.
Jingyu Liu, Wenhan Xiong, Ian Jones, Yixin Nie, Anchit Gupta, and Barlas Oguz. Clip-layout: Style-consistent indoor scene synthesis with semantic furniture embedding. arXiv preprint arXivv2303.03355 2023a.
Minghua Liu, Ruoxi Shi. Kaiming Kuang, Yinhao Zhu, Xuanlin Li, Shizhong Han, Hong Cai, Fatih Porikli, and Hao Su. Openshape: Scaling up 3d shape representation towards open-world understanding. Advances in Neural Information Processing Systems (NeurIPS). 2023b.
12

---

Published as a conference paper at ICLR 2024
Robert Lo, Arnhav Datar, and Abishek Sridhar. Lic-gan: Language information conditioned graph generative gan model. arXiv preprint 2xiv:22306.01937, 2023.
Michael Longland, David Liebowitz, Kristen Moore, and Salil S Kanhere. Text-conditioned graph generation using discrete graph variational autoencoders, 2022.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Representations (ICLR), 2018.
Andrew Luo, Zhoutong Zhang, Jiajun Wu, and Joshua B Tenenbaum. End-to-end optimization of scene layout. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3754-3763, 2020.
Rui Ma, Akshay Gadi Patil, Matthew Fisher, Manyi Li, Sören Pirk, Binh-Son Hua, Sai-Kit Yeung, Xin Tong, Leonidas Guibas, and Hao Zhang. Language-driven synthesis of 3d scenes from scene databases. ACM Transactions Graphics (TOG), 37(6): 2018.
Karolis Martinkus, Andreas Loukas, Nathanaël Perraudin, and Roger Wattenhofer. Spectre: Spectral conditioning helps to overcome the expressivity limits of one-shot graph generators. Interna- lional Conference on Machine Learning (ICML), pp. 15159-15179, PMLR, 2022.
Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun- Zhu, and Stefano Ermon. Sdedit: Guided image synthesis and editing with stochastic differential equations. International Conference Learning Representations (ICLR), 2021.
Paul Merrell, Eric Schkufza, Zeyang Li, Maneesh Agrawala, and Vladlen Koltun. Interactive furni- ture layout using interior design guidelines. ACM Transactions on Graphics (TOG), 30,,):1-10, 2011.
Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob Megrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. In International Conference on Machine Learning (ICML), pp. 11784-18004, PMLR, 2022.
Chenhao Niu, Yang Song, Jiaming Song, Shengjia Zhao, Aditya Grover, and Stefano Ermon. Per- mutation invariant graph generation via score-based generative modeling. In International Con- ference Artificial Intelligence and Statistics (AISTATS), pp. 4474-4484, PMLR, 2020.
OpenAl. Gpt-4 technical report. arXiv preprini 2rXiv:2303.08774, 2023.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to fol- low instructions with human feedback. Advances in Neural Information Processing Systems (NeurIPS), 35:27730-27744, 2022.
Wamiq Para, Paul Guerrero, Tom Kelly, Leonidas Guibas, and Peter Wonka. Generative layout modeling using constraint graphs. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 6690-6700, 2021.
Gaurav Parmar, Richard Zhang, and Jun-Yan Yan Zhu. On aliased resizing and surprising subtleties in gan evaluation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 11410-11420 2022.
Despoina Paschalidou, Amlan Kar, Maria Shugrina, Karsten Kreis, Andreas Geiger, and Sanja Fi- dler. Atiss: Autoregressive transformers for indoor scene synthesis. Advances Neural Informa- fion Processing Systems (NeurIPS), 34:12013-12022 2021.
Akshay Gadi Patil, Supriya Gadi Patil, Manyi Li, Matthew Fisher, Manolis Savva, and Hao Zhang. Advances in data-driven analysis and synthesis of 3d indoor scenes. Computer Graphics Forum, 2023.
Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. Film: Visual reasoning with a general conditioning layer. In Proceedings of the AAAI conference on Artificial Intelligence (AAAI), volume 32, 2018.
13

---

Published as a conference paper at ICLR 2024
Boris T Polyak and Anatoli B Juditsky. Acceleration of stochastic approximation by averaging. SIAM Journal on Control and Optimization (SICON), 30(4))838- 855, 1992.
Pulak Purkait, Christopher Zach, and Ian Reid. Sg-vae: Scene grammar variational autoencoder to generate new indoor scenes. In European Conference on Computer Vision (ECCV), pp. 155-171. Springer, 2020.
Siyuan Qi, Yixin Zhu, Siyuan Huang, Chenfanfu Jiang, and Song-Chun Zhu. Human-centric indoor scene synthesis using stochastic grammar. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5899-5908, 2018.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference Machine Learning PMLR, 2021.
(ICML), pp.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International Conference on Machine Learning (ICML), pp. 8821-8831 PMLR, 2021.
Daniel Ritchic, Kai Wang, and Yu-an Lin. Fast and flcxible indoor scene synthesis via deep con- volutional generative models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6182-6190, 2019.
Martin Simonovsky and Nikos Komodakis. Graphvae: Towards generation of small graphs using variational autoencoders. In International Conference on Artificial Neural Networks (ICANN), pp. Springer, 2018
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learn- ing (ICML), pp. 2256-2265. PMLR, 2015.
Kihyuk Sohn, Honglak Lee, and Xinchen Yan. Learning structured output representation using deep conditional generative models. Advances in Neural Information Processing Systems (NeurIPS), 28, 2015.
Liangchen Song, Liangliang Cao, Hongyu Xu, Kai Kang, Feng Tang, Junsong Yuan, and Yang Zhao. Roomdreamer: Text-driven 3d indoor scene synthesis with coherent geometry and texture. arXiv preprint arXiv:2305. 11337, 2023.
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In Interna- tional Conference on Learning Representations (ICLR), 2020.
Jiapeng Tang, Yinyu Nie, Lev Markhasin, Angela Dai, Justus Thies, and Matthias Nießner. Dif- fuscene: Scene graph denoising diffusion probabilistic model for generative indoor scene synthe- sis. arXiv preprint arXiv:2303. 14207, 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neurul Informa- fion Processing Systems (NeurIPS), 30, 2017.
Tathagat Verma, Abir De, Yateesh Agrawal, Vishwa Vinay, and Soumen Chakrabarti. Varscene: A deep generative model for realistic scene graph synthesis. In International Conference on Machine Learning (ICMI., pp. 22168-22183. PMLR, 2022.
Clement Vignac. Igor Krawczuk, Antoine Siraudin, Bohan Wang, Volkan Cevher, and Pascal Frossard. Digress: Discrete denoising diffusion for graph generation. In International Conference Learning Representations (ICLR), 2023.
Kai Wang, Manolis Savva, Angel X Chang, and Daniel Ritchie. Deep convolutional priors for indoor scene synthesis. ACM Transactions on Graphies (TOG), 37(4): 2018.
14

---

Published as a conference paper at ICLR 2024
Kai Wang, Yu-An Lin, Ben Weissmann, Manolis Savva, Angel X Chang, and Daniel Ritchie. Planit: Planning and instantiating indoor scenes with relation graph and spatial prior networks. ACM Transactions Graphics (TOG), 38(4):1-15, 2019.
Xinpeng Wang, Chandan Yeshwanth, and Matthias Nießner. Sceneformer: Indoor scene generation with transformers. In International Conference on 3D Vision (3DV), pp. 106-115. IEEE, 2021.
Haitao Yang, Zaiwei Zhang, Siming Yan, Haibin Huang, Chongyang Ma, Yi Zheng, Chandrajit Bajaj. and Qixing Huang. Scene synthesis via uncertainty attribute synchronization. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 5630- 5640, 2021a.
Ming-Jia Yang, Yu-Xiao Guo, Bin Zhou, and Xin Tong. Indoor scene generation from a collection of semantic-segmented depth images. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 15203-15212, 2021b.
Jiaxuan You, Rex Ying, Xiang Ren, William Hamilton, and Jure Leskovec. Graphrnn: Generat- ing realistic graphs with deep auto-regressive models. In International Conference on Machine Learning (ICML), pp. 5708 5717. PMLR, 2018.
Guangyao Zhai, Evin Pinar Örnck, Shun-Cheng Wu, Yan Di, Federico Tombari, Nassir Navab, and Benjamin Busam. Commonscenes: Generating commonsense 3d indoor scenes with scene graphs. Advances in Neural Information Processing Systems (NeurIPS), 2023.
Zaiwei Zhang, Zhenpei Yang, Chongyang Ma, Linjie Luo, Alexander Huth, Etienne Vouga, and Qixing Huang. Deep generative modeling for scene synthesis via hybrid representations. ACM Transactions on Graphics (TOG), 39(2):1-21, 2020.
Yang Zhou, Zachary While, and Evangelos Kalogerakis. Scenegraphnet: Neural message passing for 3d indoor scene augmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 7384-7392, 2019a.
Yi Zhou, Connelly Barnes, Jingwan Lu, Jimei Yang, and Hao Li. On the continuity of rotation representations in neural networks. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 5745-5753, 2019b.
15

---

Published as a conference paper at ICLR 2024
A DATASET PREPARATION
Following previous works (Paschalidou et al., 2021; Tang et al., 2023; Liu et al., 2023a), we use three types of indoor rooms in 3D-FRONT (Fu et al., 2021a) and preprocess the dataset by filtering some problematic samples, resulting in 4041 bedrooms, 813 living rooms and 900 dining rooms. The number of objects N; in the valid bedrooms between and 12 with 21 object categories, i.e., K. = 21. While for living and dining rooms, N, varies from to 21 and Ke 24. We use the same data split for training and evaluation as ATISS (Paschalidou al., 2021).
The original 3D-FRONT dataset does not contain any descriptions of room layout or object appear- ance details. order to advance research in the field of text-conditional indoor scene generation, we carefully curate high-quality dataset with paired scenes and instructions for interior design through a multi-step process:
1. Spatial Relation Extraction: View-dependent spatial relations are initially extracted from the 3D-FRONT dataset using predefined rules similar to Johnson et al. (2018) and Luo et al. (2020), which are listed in Appendix A.
2. Object Captioning: We further enhance the dataset by providing captions objects using BLIP (Li et al., 2022), a powerful model pretrained for vision-language understanding, given furniture 2D thumbnail images from the original dataset (Fu et al., 2021b).
3. Caption Refinement: As generated captions may not always be accurate, we filter them with corresponding ground-truth categories using ChatGPT (Ouyang et al., 2022; Ope- nAI, 2023), large language model fine-tuned for instruction- based tasks. This results in accurate and expressive descriptions of each object in the scene, The prompt and hyperpa- rameters for ChatGPT to filter captions provided in Appendix A.2.
4. Instruction Generation:
The final instructions for scene synthesis are derived from ~ 2 randomly selected "(subject, relation, object)" triplets obtained during the first extraction process. Verbs and conjunctions within sentences are also randomly picked to maintain diversity and fluency.
To facilitate future research and replication, the processing scripts and the processed dataset can be found in https: / http://cenngnnnnnncmmc g github. io/ pro ject rnct Scene.
A.1 RELATION DEFINATIONS
3D space as listed in Table 4. Assume and Y span the ground plane, and Z is the vertical axis. We use Center represent the coordinates of 3D bounding box's center. Height is the Z-axis size of a bounding box. Relative orientation is computed as Osn aYan2(Y,, Yo, X, - Xx). where and respectively refer "subject" and "object" in a relationship. d(s. o) the ground distance between s and o. Inside(s, indicates whether the subject center is inside the ground bounding box of the object.
We define 11 relationships
A.2 CAPTION REFINEMENT BY CHATGPT
3...t...-ttttrtbbbbbb. 5-tuubb),, Our prompt The generated object captions from BLIP are refined by ChatGPT 3 to ChatGPT is provided in Table We set the hyperparameter temperature and cop-p for text generation to 0.2 and 0. respectively, encouraging more deterministic and focused outputs.
B IMPLEMENTAION DETAILS
B.1 BASELINE DETAILS
We choose two prominent methods for comparison: (1) ATISS (Paschalidou et al., 2021)', an au- toregressive model that sequentially generates unordered object sets; (2) DiffuScene (Tang et al., 2023)². a Gaussian diffusion model that treats scene attributes as continuous 2D matrices.
https:: http:///ggtttttteooeee http://thho...cmmmmmtaaaaaa//// Phttps://g www..tc.....aaaaeeoooooo hhob.ccom/tagggiaapengg /Di FfuScene
16

---

Published as a conference paper at ICLR 2024
Table 4: Rules determine the spatial relationships between objects.
Relationship
Rule
Left of
(0sn 2 3r or A0 < 3x
4 and < d((s 3
Right of
< 00 < and
and 1 < d(s,o) 53
In front of
T S 4so
< 1 and < d(8,0) 53
Behind Closely left of
3n
T < As < - and 1 < <((,,)) 5
(0 2 3 or s0 <
3m
1 and d(s,0) 1
Closely right of
< 00
< and d(s,o) <
Closely front of Closely bebind Above
S 80
and d(s, 1
8T < 00 < < and d(s, 0) 1
T
(Centerz, Centerz.) > (Height, + Height.)/2 and (Inside o) or Inside
Below
(Centerz, Centerz,) (Height, H2/
and (Inside 8, o) or Inside 8)) d(s, o) > 3
None
Table 5: Prompt for ChatGPT to refine raw object descriptions.
Given a description of furniture from captioning model and its ground truth category, please combine their information and generate a new short description one line. The provided category must be the descriptive subject of the new description. The new description should be as short and concise as possible, encoded in ASCII. Do not describe the background and counting numbers. Do not describe size like 'small', 'large', etc. Do not include descriptions like 'a 3D model', 'a 3D image'. 'a 3D printed', etc. Descriptions such as color, shape and material are very important, you should include them. If the old description already good enough, you can just copy If the old description mcaningless, you can just only include the category. For example: Given 'a 3D image of brown sofa with four wooden legs' and 'multi-scat sofa', , you should return: a brown multi-seat sofa with wooden legs. Given 'a pendant lamp with six hanging balls on the white background" and 'pendant lamp', you should return: pendant lamp with hanging balls. Given 'a black and brown chair with floral pattern' and 'armchair', you should return: black and brown floral armchair. The above examples indicate that you should delete the redundant words in the old description, such '3D image', 'four', 'six" and 'white background', and you must include the category name as the subject in the new description. The old descriptions (BI.IP capt i on)", its category is "(ground- category)' the new descriptions should be:
We re-implement and augment these methods based on their official GitHub repositories to support instruction-driven scene synthesis and quantized semantic feature generation. In the case of ATISS, replace the [START] token, which originally is the room mask feature, with learnable token, as we condition scene synthesis on instruction prompts rather than room masks. The augmented ATISS predicts quantized feature indices after class label sampling, and they are subsequently uti- lized to predict the remaining scene attributes. Instead, quantized features are one-hot encoded in DiffuScene, allowing them to be diffused and denoised in continuous space alongside other at- tributes.
To maintain fair comparison, network architectures, training hyperparameters, retievvl
the experimental settings across all methods, including
retrieval procedures, rendering schemes, etc.
B.2 MODEL DETAILS
We use 5-layer and 8-head Transformers with 512 attention dimensions and a dropout rate of 0.1 for all generative models in this work. They are trained by the AdamW optimizer (Loshchilov & Hutter, 2018) for 500, 000 iterations with batch size of 128, a learning rate of 1e-4, and weight decay of 0. 02. Exponentially moving average (EMA) technique (Polyak & Juditsky, 1992; Ho et al., 2020) with decay factor of 0. 9999 is utilized in the model parameters.
17

---

Published as a conference paper at ICLR 2024
(Liu al., 2023b)* extract 3D object seman- tie features E 280 It is a recently introduced 3D RGB point cloud encoder that aligms with the pretrained CLIP ViI BalT-bigG/ / 14 multi-modal features (Cherti et al., 2023), enabling the simultane- ous representation visual appearances and geometric shapes. The codebook 2 has a size of 64 and dimension of 512 2. We
We adopt OpenShape
ordered indices quantize OpenShape features. f'VQ (Q-VAR is trained on over 4, 000 3D objects found in the filtered 3D-FRONT scenes (Fu et al., 2021abb), We use the frozen text encoder in CLIP ViT-B/32 (Radford al., 2021)* extract instruction fea- tures for all experiments. Regarding the loss weights R and de in Equation 6, we do not tune and simply fix them as and 10 respectively to ensure that the three terms in the loss are of comparable numerical magnitudes.
Code for both training and evaluation can be found in pro projectt//o Ceacss/Inst.o InnstructSSceee
io/
B.3 EVALUATION DETAILS
Blender Rendering After retrieving objects from a 3D database (Fu et al., 2021b), we use Blender (Community, 2018) with the CYCLES engine render high-quality images for 3D scenes. Our rendering script is adapted from the one available at https: //tpt::///tthbb.../ com/ allenai////////////aa ob javersee render r ng/ higg//loo blob/moaammmmm/ htt//maannn//cc//aa/ httaaiissssiisseee /blender seript......... py. The images for evaluation are rendered from top-down perspective in 256 256 resolutions. We maintain a camera distance of 1. filter width of 0 1, and use the RGB color mode. Other hyperparameters are set in accordance with the referenced script. Sizes of floor plans are adapted across scenes to include all objects, and their textures fixed ensure the choice does introduce any bias in evaluations.
Computation of Metrics FID, FIDCLIP and KID scores computed by the e clean-fid li- brary (Parmar et al., 2022);. Lower scores derived from these metrics indicate a higher degree of similarity between the learned distributions and real ones, Following Paschalidou al. (2021), we fine-tuned an AlexNet (Krizhevsky et al., 2012) that had been pretrained on ImageNet to classify the rendered images of synthesized scenes as well as those of ground-truth scenes. The scene clas- sification accuracy (SCA) that approaches 50% signifies better generation performance.
C ADDITIONAL RESULTS
C.I INSTRUCTION-DRIVEN SCENE SYNTHESIS
We present visualizations of instruction-driver synthesized bedrooms, living rooms, and dining rooms in Figure 5, 6 and 7. Besides the quantitative evaluations shown in Table 1, these qualitative visualizations also evident the superiority of method over previous state-of-the-art approaches in terms of adherence to instructions and generative quality.
C.2 ZERO-SHOT APPLICATIONS
We present visualizations illustrating various zero-shot instruction-driven applications, including stylization, re-arraneement, completion, and unconditional 3D scene synthesis in Figure 8, 9, 10 and 11 respectively. We find that the autoregressive model ATISS tends to generate redundant objects, resulting in chaotic synthesized scenes. DiffuScene encounters challenges in accurately modeling object semantic features, often yielding objects that lack coherence terms of style or pairing, thereby diminishing the aesthetic appeal of the synthesized scenes. Moreover, both of these baseline models frequently struggle to follow the provided instructions during conditional generation. In contrast, approach demonstrates notable capability generate highly realistic 3D scenes that concurrently adhere to the provided instructions.
'https: :/mp:s:///iae tttp///tthhhooooo comp:////ooop/////ccccccc htttop/s//tttbbbcco///////////pp code
'https://g thub. http://hhhb..cmaaaaaaaaaaaaaaaae fid
18

---

Published as a conference paper at ICLR 2024
C.3 FEATURE RECOVERY
We conduct two additional experiments to further validate our method: (1) masking the semantic fea- ture of one object and utilizing pretrained semantic graph prior for recovery: pa (f(fff/ e, t, s,,,)): (2) masking semantic features of all objects except one and again using the pretrained semantic graph prior for recovery: ( c, s, f/s means semantic features of all objects except the i-th one. Instructions for both experiments are set to none. Visualization results are presented in Figure 12 and 13 respectively.
These results indicate the diversity of our method and highlight that semantic graph prior could ef- fectively capture stylistic information and object co-occurrences from the training data. Our method trends to generate style consistent and thematic harmonious scenes, e.g., chairs and nightstands in suit, and matched color palettes and cohesive artistic style.
C.4 DIVERSITY
We provide examples of diverse set of scenes generated from a single prompt and the same seman- tic graph in Figure 14 and 15 respectively, showcasing the diversity of our generative method.
C.5 INSTRUCTSCENS WITHOUT SEMANTIC FEATURES
We observed a significant decline the appearance controllability and style consistency of generated scenes when semantic features were omitted. We include these degraded visualization results in Figure 16 and 17.
It arises from the fact that, without semantic features, the generative models solely focus mod- eling the distributions of layout attributes, i.e., categories, translations, rotations, and scales. This exclusion of semantic features results in generated objects whose occurrences and combinations lack awareness of object style and appearance, which are crucial elements in scene design.
C.6 RUNTIME COMPARISON
In the default settings (T 100 + 10), our method takes about 1 2 seconds to generate a batch of 128 living rooms by our method on single A40 GPU. In comparison, ATISS (Paschalidou et al., 2021) takes 3 seconds, and DiffuScene (Tang et al., 2023) requires 22 seconds.
It's noteworthy that our method can be significantly accelerated by reducing the number of diffusion time steps. For instance, setting = 20 + 5 reduces the runtime to seconds without noticeable decline in performance. The impact of diffusion time steps is investigated in Sec. 5.5.1. We believe with more advanced diffusion techniques, diffusion models can be more effective and efficient than autoregressive models, especially for complex scenes.
D LIMITATIONS AND FUTURE WORK
Although method significantly enhances the controllability and fidelity of 3D indoor scene syn- thesis, still has some limitations. First, despite our best efforts to ensure the accuracy of the pro- posed instruct-scene pair dataset, 3D-FRONT contains problematic object arrangements and mis- classifications even after filtering, as discussed in previous works (Paschalidou et al., 2021; Tang al., 2023). Our learned prior may consequently inherit these erroneous cases. Meanwhile, the scale of the current 3D scene dataset remains small, with only hundreds of scenes, in contrast to 3D object datasets containing thousands or even millions of samples (Chang et al., 2015b; Deitke et al., 2023). A promising avenue for future research is to expand the scale of the 3D scene dataset or leverage large-scale and well-annotated datasets for 3D objects to establish a new benchmark for 3D scene synthesis. In this work, we only focus on indoor scene synthesis. However, the proposed semantic graph prior, which encapsulates high-level object interactions within a scene, also offers the potential for modeling more intricate outdoor scenes. Furthermore, achieving fully genera- tive synthesis pipeline is feasible by substituting the object retrieval step with 3D object generative models conditioned on categories and semantic features provided by our graph prior. Lastly, in light the rapid development of large language models (LLMs), the integration of an LLM into our instruction-driven pipeline holds significant promise for further enhancing generation controllability.
19

---

Published as a conference paper at ICLR 2024
E DISCUSSION ON DATASET
While the curated instructions in our proposed dataset are derived from predefined rules, we believe that our model exhibits generalizability broader range of instructions. For example, in the stylization task, we utilize instructions in different sentence patterns with training, such as "Let the room be wooden style" and "Make objects the room black", as illustrated Figure 8, We also experiment with instructions containing vague location words, like "Put a chair next to double bed". wherein our method generates corresponding objects in all possible spatial relations (e.g., "left", "right", "front", and "behind").
Nevertheless, INSTRUCTŠCENE still faces limitations in comprehending complex text instructions and abstract concepts that do not occur in the curated instructions. For instance, (1) handling in- structions with more required triplets, like or 5, poses a challenge. (2) Additionally, identifying the same object within one instruction, such "Put table left to sofa. Then add a chair to the table mentioned before" also a difficult task. (3) Furthermore, struggles with abstract concepts such artistic style, occupants, and functionalities that do not occur the curated instructions. These limitations are attributed to the CL.P text encoder, which is contrastively trained with image features and tends capture global semantic information in sentences. Given the rapid develop- ment of large language models, we believe the integration of LLMs into the proposed pipeline is promising research topic.
A viable approach to improve the quality of current instructions involves employing LLMs to refine entire sentences in the proposed dataset using crowdsourcing make the dataset curation pipeline semi-superse superviperv..ed. We hope the proposed dataset and creation pipeline could serve as a good starting point for creating high-quality instruction datasets.
20

---

Published as a conference paper at ICLR 2024
comer side round tog to the left of black silver pendant lamp lights.
Add grey stool with black legs behind'a wooden bookshelf with shelves, Set up bookshelf left of a black and white double bed.
Add black and double bed front of black wardrobe with hanging clothes, black pendant lamp with a in of wardrobe.
Add double hed left of nightstand with drawers. Place gray nightstand with drawers to the right of a gray wardrobe
wooden wardrobe with drawers behind a brown and black double bed. Additionally, install a pendant lamp in front a wooden drawers.
Arrange green and gray double bed below pendant lamp.
(a) Instructions
(b) ATISS
(c) DiffuScene
(d) Ours
Figure 5: Visualizations for instruction-diie synthesized 3D bedrooms by ATISS (Paschalidou al., 2021), DiffuScene (Tang et al., 2023) and our method.
21

---

Published as a conference paper at ICLR 2024
Place dining table the right of coffee table. Put black dining thair right a coffee table.
Set up floral multi seat sofa left of
Put pendant lamp with white shades ohove wooden coffee table with carved legs.
Set up brown dining table metal legs (eft of lounge
chair. Then, set red pendant lamp behindd' black bookshelf with shelves and a drewer.
handle
Add blue dining table with waoden top behind blue plsid upholstered dining chair. Place metal corner side table to the left of
aut black and white marble coffee table right of multi-seat sofa with pillows.
(a) Instructions
(b) ATISS
(c) DiffuScene
(d) Ours
Figure 6: Visualizations for instruction-drive synthesized 3D living rooms by ATISS (Paschalidou al., 2021), DiffuScene (Tang et al., 2023) and our method.
22

---

Published as a conference paper at ICLR 2024
dining chair closely front af dining table. Put dining table below pendant lamp.
Place burgundy dining chair right of dining
Posiitin brown dining table legs closely behind grrey fabric dining chair.
Additionally, arrange triangle coffee table the left a black multi-seat sofa with pillows.
Set up dining table closely
right of dining
Place dining table in front of silver wine
door. Then, arrange metal pendant lamp with hanging rod to the a multi seat sofa.
Arrange brass pendant lamp balls closely left black leather dining chair.
(a) Instructions
(b) ATISS
(c) DiffuScene
(d) Ours
Figure 7: Visualizations for instruction drive synthesized 3D dining rooms by ATISS (Paschalidou al., 2021), DiffuScene (Tang et al., 2023) and our method.
23

---

Published as a conference paper at ICLR 2024
Make the room brown diye.
romm style browes
Mare the room browns.
Jet the roam uneden style.
Make the room black
objects
(a) Instructions
(b) Original Scenes
(c) ATISS
(d) DiffuScene
(c) Ours
Figure 8: Visualizations for instruction-drive scenes stylization by ATISS (Paschalidou al., 2021), 1), DiffuScene (Tang et al., 2023) and our method.
24

---

Published as a conference paper at ICLR 2024
Plass bike wardrobs No buu doutle bod.
Putls chareers
Set
pillews betand brack pendoen bomp with
the
Position patterned dining
thair ngit multi sear sota
pillws Addi unally put Mock TV stand in front of muth-ssat sofn with pilltws
of
toble conved log,
armcharr carved logs.
Position murt seat sofa of driing table with weoden Then, mosher pandlnt lamp with metal frant brown stand
Arrange white dining table to the right of'a blans consale tubli
with frome bohlted brown
(a) Instructions
(b) Messy Scenes
(c) ATISS
(d) DiffuScene
(c) Ours
Figure 9: Visualizations for instruction-drive 3D scenes re-arrangement by ATISS (Paschalidou et al., 2021), DiffuScene (Tang et al., 2023) and our method.
25

---

Published as a conference paper at ICLR 2024
Add a secedten werthohe with drawers bebord a becen and black double bed Additionlly, instal pendant lame fiont o waedon wardroce with drowers
sirader parsdant lamp sibeer rightssand drower
weoden
up stand in frant of brown Insesseet sofu with weoden frame.
armichali muttiseat
siver coffee ofa soft.
Put eecdden diring table delow parsient lamp.
Put bock multi seot sota with glllws the or blans plaste dining table.
(a) Instructions
(b) Partial Scenes
(c) ATISS
(d) DiffuScene
(c) Ours
Figure 10: Visualizations for instruction-drive 3D scenes completion by ATISS (Paschalidou al., 2021) 1), DiffuScene (Tang et al., 2023) and our method.
26

---

Published as a conference paper at ICLR 2024
(a) ATISS
(b) DiffuScene
(c) Ours
Figure 11: Visualizations for unconditional 3D scenes stylization by ATISS (Paschalidou et al., 2021), DiffuScene (Tang et al., 2023) and our method.
27

---

Published as a conference paper at ICLR 2024
(a) Original Scenes
(b) Ours (seed: 0)
(c) Ours (seed:
G, Figure 12: Generating the semantic feature of one object without instructions (f s,
28

---

Published as a conference paper at ICLR 2024
(a) Original Scenes
(b) Ours (seed: 0)
(c) Ours (seed: 1)
Figure 13: Generating semantic features of all objects in scene except one without instructions
Po (f y s,
29

---

Published as a conference paper at ICLR 2024
(a) Put dressing table with mirror to the left of double bed.
(b) Add lounge chair in front of a multi-seat sofa.
Figure 14: Examples of diverse set of scenes generated from a single prompt.
(a) Semantic Graph 1.
(b) Semantic Graph 2.
Figure 15: Examples of a diverse set of scenes generated from the same semantic graph.
30

---

Published as a conference paper at ICLR 2024
Add corner side table with round top the left of a black and silver pendant lamp with lights.
Place L-shaped sofa behind black marble desk. Then, position lounge chair the right af black marble desk.
Put a TV stand In front of a grey sofa. Finally, add green upholstered dining chair to the left grey stand.
(a) Instructions
(b) Ours w/o semantic features
(c) Ours
Figure 16: Instruction-driven scene synthesis results of INSTRUCTŠCENE and its degraded version, which is not encoded with semantic features.
31

---

Published as a conference paper at ICLR 2024
Figure 17: Unconditional scene synthesis results of a degraded version of INSTRUCTSCENE, which not encoded with semantic features.
32


Reviews:
[{"review_id": "s3FJwXQjsQ", "reviewer": "Reviewer_Peja", "summary": "This paper proposes a framework that generates 3D indoor scenes from natural language prompts. The framework takes two stages. The first stage generates a scene graph from the language prompt. The scene graph contains category and vector quantized feature representation of objects and relations (as directed edges) between them. The second stage generates the scene layout from the scene graph, predicting the location, size and orientation of the objects. Both stage are learned with transformer-based diffusion models. Various techniques are adopted to allow using diffusion to learn discrete attributes and graphs. \nEvaluation show that the proposed method outperforms previous state of the art methods (on slightly different tasks) convincingly. Qualitative examples appear to be of good quality as well.", "questions": "I think this is a good paper: it is clearly very solid technically, allows useful applications, and clearly outperforms prior works. I don't think there is anything that I need to see in the author responses. However, as state in the weakness section above, I do have issues here and there with the generalizability with the method, and with how the evaluations are carried out. If the authors could provide more insight regarding these concerns, then I would be much more inclined to champion this paper (which I currently am hesitant to do).", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- Very solid design choices. The authors clearly have good knowledge of techniques relevant to this problem, and the proposed solution make a lot of sense to me e.g. discretizing object features with VQVAE, discrete diffusion models, the use of MASK for diffusion-based graph generation, various learning tricks, etc.\n- Good effort in creating a dataset suitable for this task (as existing scene datasets lack language descriptions). The pipeline for creating such text descriptions synthetically is reasonable.\n- Clearly outperforms prior SOTAs over many metrics commonly used for this task. \n- Qualitative results look good --- matches my impression of gt 3DFRONT scenes. Samples provided in the supp are realistic - exhibits many common minor issues, showing that they are clearly not cherry picked results (which is important considering the questionable reproducibility of many prior works).", "weaknesses": "- More of a system paper: combines many ideas from previous works on 3D scene generation and works on diffusion-based generation model. Seems that there is not too much truly new ideas here (and clarifications on where these happens would be great). I'm fine with it, to be honest, because this paper ticks my checkboxes for what a good system paper should be: clever/novel combination of ideas/components, solid execution, exceptional results. However, this is technically still a \"weakness\" I guess, as it limits the further applicability of the proposed method in other domains.\n- Questionable choice of baselines. I get that previous graph-based/text-based works are not trained on 3D-FRONT, but it should be possible to compare with them in some capacities, or at least discussing the issues with them in the evaluation section,\n- Despite the good discussions of issues of the way prior works handle latent object features (my impression was that they don't help much), I am left unconvinced that the proposed method actually make use of such features e.g. 1. would manually changing semantic feature f of an object result in notable difference in the final scene? 2. Can the model pick up stylistic information from the training data? Do objects that frequently occur together in the training data behave similarly in the generated scenes? 3. Is there any noticeable degrade in scene layout quality if semantic features are not encoded (i.e. just sample CAD models by category and size, which is a common practice for prior works) or if they are encoded with a smaller feature space?\n- A user study would have been useful as it seems that humans, although not always consistent at rating the quality of the scenes, would be very good at checking if the generated scenes matches the text description.\n- More discussions on how the qualitatively the proposed method works better than prior works would be very helpful. I can clearly that this is the case from the supplemental figures, and it would be great to highlight this in the main paper as well with a paragraph or so, as the proposed method clearly avoids many issues with prior works, especially on larger rooms. The set of quantitative metrics, while comprehensive, are known to be not the most sensitive metrics when it comes to evaluating indoor scenes, especially when it comes to layout/appearance details. More in depth analysis of the qualitative examples would make it a much stronger case that the proposed method is clearly superior.\n- Lack of examples on how the proposed method can generalize & generate a diverse set of rooms from a single prompt. I think there can be many valid graphs for a single prompt and many valid layouts given a fixed semantic graph. It would be great to show examples that the proposed method can indeed achieve this.\n- Most of the instructions are very specific, which makes sense consider how they are generated. However, in real applications, fuzzier instructions are also very important e.g. a room in the style of {certain style}, a room for {certain type of occupants}, a room that supports {certain functionalities}. Discussion on how to incorporating these instructions would be good.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "64I8zGq3nt", "reviewer": "Reviewer_jsgt", "summary": "The paper proposes a new scene generation framework with two stages. The first stage models the semantic graph, which consists of object categories, relations and appearances. With feature quantization, all these attributes become discrete variables, which can be learned through a discrete diffusion model. Conditioned on the semantic graph prior, the second stage uses a continuous diffusion model to learn the remaining scene attributes (rotations, translations and sizes). By modeling the semantic graph separately, the method enables various zero-shot downstream tasks. Experiments show that the proposed method achieves better scene generation results. The paper also creates a new dataset of text-scene pair for instruction-driven 3D scene synthesis.", "questions": "- More details about the dataset creation can be provided. It would be better if the author can provide some examples of the training data.\n- In the semantic graph, relation e_{jk} can be determined by e_{kj}. Are the both directions of the instructions needed to be created? If so, how to determine which direction to use during training? \n- What is the runtime of the proposed method compared to baselines?\n- The instructions are generated mainly from rules. Do you think these rules can cover the majority of the instructions compared to a human-annotated dataset? What are the limitations of the instructions in the current dataset?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- One main challenge of scene generation is the discrete and continuous nature of scene representation. The paper provides a new scene generation pipeline by disentangling the generation of discrete attributes and continuous attributes. Though the architecture designs of both the discrete and continuous diffusion model largely follow existing methods, the new formulation of the scene generation pipeline is novel. In addition, the disentanglement enables various zero-shot downstream tasks, improving the controllability of the generative model.\n- The newly proposed dataset of scene-instruction pairs is a timely contribution that aligns well with the ongoing research of scene generation. \n- The paper is clear and well-written, both detailed quantitative and qualitative results are provided.", "weaknesses": "Generally I think the paper is good. There are some questions in the part Questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ptncrMPgDW", "reviewer": "Reviewer_qrjz", "summary": "The paper proposes a semantic graph prior plus a layout decoder system to generate indoor scenes based on natural language instructions. Instruction-scene pairs are generated as a new dataset for the task.", "questions": "- What is the CLIP-aligned feature in Fig. 2? How is this feature vector applied to the VQ-VAE encoder?\n\n- Though the relations are view-dependent, the generated instructions do not describe the viewpoint of the camera. Is there an assumed viewpoint for all scenes? \n\n- Minor: in Fig. 5 row 4, the proposed methods generate a single bed while the instruction mentions a double bed. \n\nMissing reference: LayoutGPT: Compositional Visual Planning and Generation with Large Language Models. Neurips 2023", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 1, "contribution": 2, "strengths": "- The paper collects a dataset that supplements existing indoor scene datasets with natural language instructions.\n- The paper proposes a two-stage method that is shown to be more effective than previous methods in quantitative and qualitative evaluations. \n- The proposed method demonstrates some abilities in zero-shot indoor scene editing.", "weaknesses": "- I am concerned about the novelty of the proposed method. The semantic graph prior is a combination of existing VQ-VAE and discrete diffusion models. The layout decoder is a plain diffusion model. \n- Some claims or arguments about using VQ-VAE are not convincing and require further substantiation. The authors claim that VQ-VAE is used because objects share common characteristics like colors or materials. But I don't see why OpenCLIP features cannot encode these characteristics. And it's not well grounded to say that pre-trained OpenCLIP features are too complicated to model. As the author also mentioned the efficiency of discrete tokens, it's better to write the motivation clearly for using VQ-VAE. \n- The writing could be improved:\n    - the diffusion preliminary section is too abbreviated to serve as a separate section and is disjoined from Sec. 4.2.2 where the paper starts to relate to diffusion models. Then the semantic graph diffusion model employs a discrete diffusion model, that was proposed in previous work (Austin, et al., 2021). It seems that a large portion of Sec. 4.2.2 is not novel and could be treated as a preliminary study as well. \n    - The annotation is overcomplicated and hard to follow. There are new definitions of previous variables throughout the method section, like $c, f$ in the first paragraph of Sec. 4.2, $\\mathcal{C}_i$ and $\\mathcal{F}_i$ in the first two lines of Sec. 4.2.2. $f$VQ-VAE is not a standard notation. There are also variables with omitted super/sub-scripts that are not explained clearly.\n\nSome of the technical concerns might be entangled with writing issues of the current version. I am open to adjusting the rating if my concerns are resolved later.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "US9lD1Mozz", "reviewer": "Reviewer_rQD7", "summary": "The paper proposes InstructScene, which does 3D indoor scene synthesis with semantic graph prior. It first quantizes CLIP features using a transformer with learnable tokens, use a frozen text encoder and graph transformer generate the semantic graph, and finally uses a graph transformer to decoder the final layout. Experiments are done on 3D-Front following the prior works although instructions are re-generated. With the semantic graph prior,  InstructScene outperforms state-of-the-art methods significantly on controllability. It also shows zero-shot generalization to a few other tasks.", "questions": "- Can you confirm how iRecall is computed?\n- Can you comment on what will happen if we give \"arbitrary instructions\" (11 pre-defined relationships are not available)?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- The problem is meaningful. Improving controllability critical to 3D scene generation, as we have already seen a lot of 3D scene generation models.\n- The method is well-designed. \n- Experiments are extensive. \n- The paper is very well-written and easy to follow. It provides enough mathematical details, as well as comprehensive qualitative results. I enjoy reading the paper.", "weaknesses": "InstructScene is proposed to handle natural language instructions. However, the instruction does not seem to support arbitrary instructions. In qualitative results, the instruction always explicitly mentions one of these relationships. \n\n>  (Section 5.2) we use a metric named “instruction recall” (iRecall), which quantifies the proportion of the required triplets “(subject, relation, object)” occurring in synthesized scenes to all provided in instructions\n\nIt's not clear to me how iRecall is computed. To the best of my understanding, iRecall computes the recall, where the prediction is generated scene graph triplets and the ground truth is the instruction triplets. However, how do you get instruction triplets? Let's say the instruction is \"Add a corner side table with a round top to the left of a black and silver pendant lamp with lights\".  It looks non-trivial to extract triplets. Or do you use relationship keywords to extract?\n\nAdditional comments:\n- Table 4: bebind -> behind", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)