EditRoom: LLM-parameterized Graph Diffusion for Composable 3D Room Layout Editing
Authors: Kaizhi Zheng
Xiaotong Chen
Xuehai He
Jing Gu
Linjie Li
Zhengyuan Yang
Kevin Lin
Jianfeng Wang
Lijuan Wang
Xin Eric Wang
Venue: ICLR 2025 Poster (2025)
Keywords: 3D Scene Editing
Large Language Model
Diffusion-based Models
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: We propose a graph diffusion-based method fo language-guided 3D scene layout editing

Abstract: Given the steep learning curve of professional 3D software and the time-
consuming process of managing large 3D assets, language-guided 3D scene editing has significant potential in fields such as virtual reality, augmented reality, and
gaming. However, recent approaches to language-guided 3D scene editing either
require manual interventions or focus only on appearance modifications without
supporting comprehensive scene layout changes. In response, we propose EditRoom, a unified framework capable of executing a variety of layout edits through
natural language commands, without requiring manual intervention. Specifically,
EditRoom leverages Large Language Models (LLMs) for command planning and
generates target scenes using a diffusion-based method, enabling six types of edits: rotate, translate, scale, replace, add, and remove. To address
the lack of data for language-guided 3D scene editing, we have developed an automatic pipeline to augment existing 3D scene synthesis datasets and introduced
EditRoom-DB, a large-scale dataset with 83k editing pairs, for training and evaluation. Our experiments demonstrate that our approach consistently outperforms
other baselines across all metrics, indicating higher accuracy and coherence in
language-guided scene layout editing.

Published as a conference paper at ICLR 2025
EDITROOM: LLM-PARAMETERIZED GRAPH DIFFU- SION FOR COMPOSABLE 3D ROOM LAYOUT EDITING
Kaizhi Zheng'
Xiaotong Chen
Xuehai He
Jing Gu'
Linjie Li
Zhengyuan Yang²
Kevin Lin
Jianfeng Wang
Lijuan Wang² Xin Eric Wang!
UC Santa Cruz
University of Michigan, Ann Arbor
Microsoft
[kzheng31, xwang366) (uccscc..............o...o......
@ucsc. edu
"Sure, I can do that. will first use remove function remove the side tables and then use replace function to change the sofa larger one....
"I want to remove the side tables so that can have larger sofe.""
172: [Rammvv, table, (leftiright of, muiti-seat sofa)]
Slep [Replace, mulli-seal sofa, the large wooden bed]
Command Parameterizer
Scene confguraatio:
Step Command Source Scane Graph Layout
Graph Dillusion
Target Scane Graph w/ Layout
Object I: dlssssofs...
Scane
Object 2: casss sice table... Object cladeetv stand...
Figure I: Editing Pipeline with EditRoom. EditRoom is a unified language-guided 3D scene lay- out editing framework that can automatically execute all layout editing types with natural language commands, which includes the command parameterizer for natural language comprehension and the scene editor for editing execution. Given source scene and natural language commands, it can generate a coherent and appropriate target scene.
ABSTRACT
Given the steep learning curve of professional 3D software and the time- consuming process of managing large 3D assets, language-guided 3D scene edit- ing has significant potential in fields such as virtual reality, augmented reality, and gaming. However, recent approaches to language-guided 3D scene editing either require manual interventions or focus only on appearance modifications without supporting comprehensive scene layout changes. In response. pose EditRoom, a unified framework capable of executing a variety of layout edits through natural language commands, without requiring manual intervention. Specifically, EditRoom leverages Large Language Models (LLMs) for command planning and generates target scenes using a diffusion-baat based method, enabling six types of edits: rotate, translate scale, replace, add, and remove. To address the lack of data for aingugg--uidee 3D scene editing, we have de- veloped an automatic pipeline to augment existing 3D scene synthesis datasets and introduced EditRoom-DB, a large-scale dataset with 83k editing pairs, for training and evaluation. Our experiments demonstrate that our approach con- sistently outperforms other baselines across all metrics, indicating higher accu- racy and coherence in language-guided scene layout editing. Project website: -iw.aallaaaaae..ae github.ittbbb.... wow wtt.::ttoooeittteeeeeee room. github.
pro-

---

Published as a conference paper at ICLR 2025
INTRODUCTION
Traditionally, editing 3D scenes requires manual intervention through specialized software like Blender (Community) 2024). which demands substantial expertise and considerable time for re- source management. As result, Linuagg-gguides 3D scene editing has emerged as promising technology for next-genenne next-generation 3D software. To build an automated system capable of interpreting natural language and manipulating scenes, the system must be able to align complex, diverse, and often ambiguous language commands with various editing actions while also comprehending the global spatial structure of the scene. Additionally, the relatively small size of available 3D scene datasets presents a challenge for developing large-scale pretrained models necessary for fully auto- mated, end-to-end bunguage-guide scene editing.
Recently, there have been some works (Zhuang et al. 2023 Bartrum et al. 2024| Karim et al. 2023)
focusing on leveraging pretrained image generation models to edit single object appearance inside the scene, but they fail to modify the layout of original scenes. Meanwhile, other approaches (Chen et al. 2023 Ye al., 2023) further incorporate pretrained segmentation models enable individual object manipulation. However, they require manual intervention to determine the edited object and editing type for any layout adjustments. like adding a new object or changing the object pose. Furthermore, these methods are all limited to executing a single editing step and are hard to deal with commands with multiple potential steps.
Therefore, we propose EditRoom, unified framework that can execute all editing types by com- plex natural language commands without intermediate manual interventions. consists two main modules: the command parameterizer and the scene editor. The command parameterizer employs an pretrained LLM,, specifically GPT-4o (OpenAl| 2024), to transform natural language commands into sequences of breakdown commands for six basic editing types single object: add ng, removing, replacing, translat ing, rotating, and scaling. These breakdown com- mands, along with the source scenes, are then fed sequentially into the scene editor for execution. To unify all editing types, we convert scenes into graph representations and construct scene editor as conditional graph generation models, where take source scenes and text commands as conditions and train diffusion models to generate the target scene graphs with the layout. By conditioning on the user's natural language prompts, our model generates reasonable editing results that serve foundation for further refinement. Importantly, the scenes remain fully editable, allowing users to provide additional prompts for iterative improvements to achieve their desired outcome.
Another challenge is the lack of language-guided scene editing datasets, which constrains both train- ing and evaluation of scene editing models. To enable the scene editor accurately execute every basic editing type, we construct an automatic data generation pipeline and collect a synthetic scene editing dataset named EditRoom-DB for both training and evaluation, which includes approxi- mately 83k editing pairs with commands. We implement several functions to simulate the single editing process on the existing 3D scene synthetic dataset, 3D FRONT (Fu et al. 2021a), which contains 16k indoor scene designs equipped with high-quality object models, and generate corre- sponding language commands using predefined templates. To mimic the human inputs, we employ LLMs to transform the templated commands into natural language forms, serving both as training material for our baselines and as test cases for single- opperatin evaluations.
In the experiments, we quantitatively assess the performance of EditRoom in scenarios with single- operation commands. From the results, we find that EditRoom outperforms other baselines in all metrics across different room types and editing types, which indicates higher precision and co- herence in single-operation editing. Furthermore, we qualitatively evaluate EditRoom in scenarios involving multi-operation commands. We find that the model can successfully generalize these scenarios even though we do not train the model on multi-operation data.
Our contributions are summarized follows:
We propose new framework, named EditRoom, consisting of the command parameterizer and scene editor, which accepts scene inputs and can edit scenes using natural language commands by leveraging LLM for planning.
We propose unified graph diffusion- ble of executing every basic editing type, including adding, removing, replacing, trans lataslatiin in, rotating and scal 1 i ing.
module that serves the scene editor, capa-
2

---

Published as a conference paper at ICLR 2025
We introduce the EditRoom-De dataset with 83k editing pairs for the first language sanguage-guided guided 3D scene layout editing dataset by augmenting the existing 3D scene synthesis dataset.
From the experiments, we demonstrate that EditRoom outperforms other baselines across all editing types and room types on single operation commands, and it can generalize to complex operation commands without further training.
2
RELATED WORK
Language-Guided 3D Scene Generation Language-guided 3D scene generation works claiming editing capabilities can be categorized into two main approaches. The first approach utilizes large language models to generate scene configurations and perform edits (Vilesov et al. 2023 Zhou et al.) 2024b] Aguina--anne et al. 2024). However, these methods are limited to editing scenes they gen- erate and cannot accept external source scenes. The second approach trains diffusion- bbasdd models for text-conditional scene generation (Haque et al.) 2023] Tang et al. 2023 Zhai et al. 2024). These methods often require manual masking for edits and have limited editing functionalities. For ample, InstructScene (Haque et al. (2023) lacks support for operations like translation, rotation, and scaling, while EchoScene (Zhai al. (024), which directly take target scene graphs as input, only modifies object relations manually and does not process textual instructions. In contrast, EditRoom accepts existing scenes and supports free-form editing commands for 31) layouts. By leveraging dif- fusion models trained on clean data, EditRoom enables comprehensive edits, including translating, rotating, scaling, adding, and removing objects, without manual intervention.
Language-guided 3D Scene Editing Previously, some works (Ma et al. 2018) have explored the rule-based methods for language- -uided 3D scene editing. Compared to these works, our diffusion- based method can generate diverse editing results, generalize across editing types. Recent language- guided 3D scene editing works mostly incoprate neural field representations and pretrained image generation models for object appearance editing. Some works are mainly focusing on replacing
single object (Zhuang et al.) 2023 Bartrum et al. 2024 Karim et al. 2023} but failing to edit the layout. Other approaches (Chen et al.) 2023 Ye et al. 2023) leverage pretrained segmantation models to obtain the individual object representation so that they can remove the objects by manually selecting the target objects. Besides, all these methods can only take one single-operation command at each interaction. Compared to these previous works, EditRoom can leverage LLM to deal with multi-operation natural language commands and automatically execute all editing types through unified graph diffusion-basee based module.
LLMs for 3D Scene Understanding Recent works demonstrate that existing LLMS can facilitate 3D spatial reasoning. These works usually leverage the pretrained image caption models to convert
3D scenes into text descriptions and ask the LLM to generate navigation steps (Zhou et al.) 2023
2024a), provide room layout (Feng et al. 2024), or ground 3D objects (Yang et al.) 2023 Hong al.] 2023) Huang et al., 2023). In our work, we are the first work leverage LLM for natural language- guided 3D layout editing, where LLM takes source scenes in text format and breaks the natural language commands into basic editing operations.
Diffusion Models for Graph In recent years, denoising diffusion models have shown impressive generative capability in the graph generation (Liu et al., 2023a Kong et al., 2023), Vignac et al,, 2022). Compared to the previous VAE-based Verma al. (2022) GAN-based (Martinkus et al. 2022 models, diffusion-basee models have advantages like stable training processes and generaliz- ability to various graph structures. Some works have presented that graph diffusion-based models can be used in molecule generation (Guo et al., 2022 Akhmetshin (2023), protein modeling (Zhang) al. 2 3D scene generation (Haque al. 2023 /hai al. 2024). and etc. In this work, we first propose to use graph diffusion -hased models for language-guider 3D scene layout editing.
THE EDITROOM METHOD
In this section, we introduce EditRoom, shown in Figure [ comprising two primary modules: Com- mand Parameterizer and the Scene Editor. Given natural language command C and source scene S, we aim to estimate the target scene with conditional distribution (T)S, C). Our command pa- rameterizer takes the source scene S and natural command C to generate the breakdown commands
3

---

Published as a conference paper at ICLR 2025
Source Scene Graph
All masked Graph
Target Graph Generation
Source Scene
Targel Scene
Add object: wooden wardrobe; location: on the right of the hed,
FIRIE
Template Command
1 Target Scene Graph (wl random layout)
Source Soene Graph (w/ layout)
wooden wardrobe
Target Scene Graph (w/ layout)
Text Encoder
Target Layout Generation
Object Node with Category Semantic Fealure
Directed Edge for Object Spatial Relations
Scene layout with posilions, rotations sizes
Frozen
Trainable
Figure 2: Scene Editor Overview. Scene Editor aims to provide accurate, coherent editing results according to the given source scene and language commands. It consists two graph transformer- based conditional diffuusion models. One diffusion model generates semantic target scene graphs. Another diffusion model can estimate accurate poses and size information for each object inside the generated target graphs. All diffusion processes conditioned on the source scene and breakdown command.
L. Then, the scene editor conditions breakdown commands L to obtain the final target scene T, where the whole pipeline can be written as q( S, C*) = q(L]S, C) x (T L). All objects inside source scenes and target scenes retrieved from high-quality 3D furniture dataset (Fu et al., 20216).
3.1 LLM AS COMMAND PARAMETERIZER
order to process open natural language commands, we use GPT-4o (OpenAT, 2024] to convert natural language command C into set of combinations of basic editing types with breakdown commands L := {4 W where Nr is the number of breakdown commands. To cover the general manipulations on the scene, we design six basic editing operations:
Rotate an object: [Rotate, Target Object Description, Angle]
Translate object: [Translate, Target Object Description, Direction, Distance)
Scale an object: [Scale, Target Object Description, Scale Factor]
Replace an object: [Replace, Source Object Description, Target Object Description]
Add an object: [Add, Target Object Description, Target Object Location]
Remove an object: [Remove, Target Object Description]
When the target object is not unique, we ask the LLM to use another unique object as a reference to describe the spatial relation. During the inference phase, we prompt the LLM with attributes of objects within the source scene along with the natural language command, tasking the model to analyze the scene and delineate basic editing operations through breakdown commands in specified formats. The attributes include categories, locations, sizes, rotations, and object captions. Detailed descriptions of the full prompt and examples provided in Figure the appendix.

---

Published as a conference paper at ICLR 2025
3.2 GRAPH DIFFUSION AS 3D SCENE EDITOR
Given the breakdown command | and source scene S, our objective is to determine the conditional target scene distribution ( l). Drawing inspiration from recent advancements in language- guided 3D scene synthesis (Lin & Yadong 2023), we transform scenes into semantic graphs and employ a graph wransformer-basee
conditional diffusion model learn the conditional target scene graph distribution, as depicted in Figure 2 Our approach involves two key graph transformer- based diffusion models: the Target Graph Diffusion, which generates object shapes and their spatial relations as graphs, and the Target Layout Diffiusion, which computes the final layout of the target scene. To reduce the alignment challenges between the 3D scene distribution and language, all commands are encoded using the text encoder of CLIP-ViT-B-32 (Radford et al. 2021).
Scene Graph Representation Each scene is represented as a combination of layout B and a scene graph G (Lin & Yadong, 2023). The layout B encapsulates the position, size, and orientation of each object, while the scene graph G encodes additional high-level semantic information. Formally, a semantic scene graph G == (V, E¹) comprises nodes V/ E V, where each e; corresponds to an object or with high-level attributes. Directed edges ejj E E represent spatial relationships: ["in front of", "behind", "right of", "left of", "closely in front of", "closely behind", "closely right of", "closely left of", "above", "below"], connecting the i-th object to the j-th object, where "closely" means the distance between two object centers are less than 1 meter. Each node U; is characterized by discrete category o and continuous semantic features fü. derived from a pretrained multimodal- aligned point cloud encoder, OpenShape (Liu al.] 2024c), which features 1280-dimensional representation space.
Target Graph Diffusion In this stage, we aim to learn target scene graphs Sig by giving source scene graphs G. and language commands through discrete graph diffusion model Eg. where Sig includes category Ctg and semantic features Fzg for each node and the edges E, for object relative relations. Since high-dimensional object semantic features (d = 1280) are too complicated to learn from limited data, we use a VQ-VAE model (Lin & Yadong 2023 Wang et al. 2019) to compress them into low-dimensional features z Ra1x0,, which consists of ny vectors extracted from learned codebook Z E g by sequence of feature indices fude := ..., ..., Ky }rv, where K₁ and dz are the size and dimension of codebook. Then, we use the feature indices to replace the orig- inal object semantic features as targets for training, denoted as F. Therefore, Grg = (Crg. Frg. Eng) and g. (C., F.. E,), and our goal is to learn the conditional distribution (( 1). During the training process, at timestep f, the gaussian noises are added to the Gıg to get Gig and the model Eg aims to reconstruct G by conditioning on G. and 1. To add the conditions, we concatenate each element of source scene graphs into noisy target scene graphs as context and use cross-attention layers to incorporate language features. The loss function can be written as:
}
Lg E.
Σ 4111 1ot 11,t,,,,,
I, D=z 10p 2% IIp,, (0t--111 9s, 1)]
(2)
where DEL indicates the KL divergence.
Target Layout Diffusion In this stage, aim to estimate the target scene layout BIg using another graph diffusion model Es. conditioning on target scene graph Gty. source scene graph Gz, source layout B., and language command L. The target scene layout Btg E R consists of position Tre E Mx3 size Sig E MMx and rotation Reg E RMxx During the training process, gaussian noises will be added to the target layout, and the layouts are encoded into the node features by MLP layers. Similar to the Target Graph Diffusiin, we concatenate the source scene graph and source layout to the target scene graph context and corrupted target layout. The language features are incorporated through cross-attention layers. The objective target is to estimate the added noises at each time step. The loss function can be written as:
Lg == E Eg ..e lele Eb(Big, t, Geg. B..
Inference Process During the inference phase, the first step consists of transforming the source scene into a scene graph G. and a corresponding layout B,. Subsequently, the Target Graph Gener- ation model predicts the target scene graph Gig. conditioned the source scene graph G, and the
5

---

Published as a conference paper at ICLR 2025
Table 1: EditRoom-DDB dataset statisties. We collect around 83k training data across all room types and 500 test data for each room type.
Train
Test
Types
Bedroom
Dining room
Living room
Bedroom
Dining room
Living room
Translate
8.6k
3.2k
2.7k
808
253
250
Rotate
4.0k
1.3k
1.3k
804
250
252
Scale
12.7k
4.5k
3.9k
805
251
252
Add
8.9k
3,4k
2.8k
747
232
240
Remove
8.8k
3.3k
2.8k
816
260
253
Replace
6.8k
2.2k
2.1k
820
254
253
Total
49.8k
17.9k
15.6k
4800
1500
1500
language command 1. This followed by the Target Layoul Generation model, which computes the target layout Big. leveraging all available variables inputs. The final step in constructing the tar- get scene, denoted as := (Gy. Big). involves retrieving the object meshes based on the estimated object features and arranging them according to the generated layout. This systematic approach enables the dynamic generation of scenes that are aligned with verbal instructions, ensuring that the resulting scenes accurately represent the specified conditions.
4 THE EDITROOM-DB DATASET
To facilitate comprehensive training and evaluation of our framework, we present the EditRoom-
DB dataset, designed to support wide range of basic 3D scene editing operations. The dataset is generated through an automated data augmentation pipeline that produces editing pairs based on object-level modifications applied to scenes from the 3D-FRONT dataset (Fu et al. 2021a). We utilize scenes from the bedroom, dining room, and living room categories and enhance them with high-quality object models from the 3D-FUTURE dataset (Fu et al. 2021c) to simulate real-world editing workflows.
Our data generation pipeline supports a variety of editing operations, including Add and Remove Ob j jects,, Pose and Si ze Changes, and Ob Rep I a acement. Each modification produces a new scene paired with detailed textual description of the changes made, following a predefined template. For each scene, objects are selected randomly and modified iteratively through basic editing operations. In the case of Add and Remove Ob jbjects, the modified scene serves the target for the operation, with the original scene acting the source. For Pose and Size Changes, random adjustments are applied to selected objects, and collision checking ensures that the final scenes are free from object overlap. Similarly, in Ob ject Repl cement, new objects from the same category are substituted for existing ones, with collision checks ensuring high-quality outputs. More details about automatic pipelines can be found in Appendix C
To create diverse and realistic language instructions for the editing tasks, we first employ the mul- timodal understanding model LLAVA-1.6 (Liu al.) 2024b 2023b), which captions front-view images of the objects in the scenes. Then, these object captions are used to construct the template- based commands. The template-basee based commands are then transformed into natural language com- mands using GPT-4o, making the dataset suitable for training and testing language-guide scene editing models. Detailed prompts and additional examples are shown Figure 7 in the appendix.
The resulting dataset consists of approximately 83,000 training samples and 7,800 test samples (randomly sampled across editing types for each room type). Table provides a detailed breakdown of the dataset statistics across the three scene categories. Each sample includes source scene, an edited target scene, and corresponding language commands. This comprehensive dataset enables the development and evaluation of robust models capable of performing various scene editing tasks. By simulating realistic workflows and providing detailed text commands, EditRoom DB serves as a valuable resource for advancing language-guided 3D scene editing.

---

Published as a conference paper at ICLR 2025
Performance on single operation with different room types. From the table, we can find EditRoom outperforms basclines among all room types, which indicates that our methods can provide more accurate and coherent editing across room types.
Table 2:
Dedroom
Dining room
Living room
Model
IOU (*)
S-1OL (1)
LPIPS (1)
CLIP(1)
JOU
S1OU(()
LPIPS (1)
CLIP (1)
IOU (7)
S-OOU
LPIPS
CLIP(1>
Diftuboene N
I.G220
0.6125
0.1421
0.9415
1.4473
0.47221
0.2922
0,9254
11,3332
0.4161
0.1811
11.1236
SceneEditor
0.7055
0.0942
0.1261
0.9520
0,5252
0.5091
0.1557
0.3366
0.4618
0.4198
0.1738
0,9829
EditRoces
0.7236
0.1090
0,9897
0.1460
0,4710
0,9446
5 EXPERIMENTS
5.1 EXPERIMENTAL SETUP
Baselines To the best of the authors' knowledge, EditRoom is the first work that can automatically execute all editing types with natural language commands. Therefore, we construct two baselines by modifying language-guided 3D scene synthesis methods for comparisons: DiffuScene-N and SceneFditor-N:
DiffuScene-N: DiffuScene-N is modified from the language-guided 3D scene synthesis work, DiffuScene [Tng al., 2023), which includes UNet-based diffusion model to generate scene layout. To enable it with language- gggudge-guided scene editing ability, we lever- age their scene completion pipeline by incorporating the source scene as context for the diffusion process. During the training and testing, the model directly conditions natural commands for target scene layout generation.
SceneEditor-N: To test where the scene editor directly trains on the natural commands got from the GPT-4o. Dur- ing the inference time, the model conditions the natural commands and generates the final scenes.
generalization ability, experiment with another setting,
We would like to emphasize that not all language-guided 3D scene synthesis methods can be mod- ified 3D scene layout editing methods. There
least two prerequisites: (1) there is a way to casily convert the source scene into the required intermediate representations: (2) the scene repre- sentations have the capability to execute all editing manipulations. Some L.M-relevant 3D scene generation models (Vilesov et al.] 2023) Zhou et al. 2002bb) Eggiin-Kaag et al.] [024] Hu et al.] 2024) can generate 3D scenes based on descriptions, but source scenes hard convert into their
intermediate representations, like program language (Aguina-Kang et al. 2024) and blender codes
(Hu et al. 2024).
Metries To evaluate the models' performance, we utilize four metrics: IOU, S-IOU, LPIPS (Zhang al. 2018), and CLIP (Radford al. 2021) scores. The IOU scores are calculated by determining the 3D Intersection Over Union (IOU) between each object in the generated and target scenes, selecting pairs with the highest 3D IOU values. The S-IOU represents the cemantic-weighted 3D IOU, where semantic similarities between matching objects are calculated using Sentence BERT (S-BERT) (Reimers Gurevych, 2019) based on their captions. For visual evaluation, we render both the generated and target scenes from 24 fixed camera views. Visual similarity is assessed using the LPIPS metric for pixel similarity, and semantic similarity is evaluated using the CLIP image encoder
5.2 RESULTS
Single-operation Commands To assess model performance on single operations, we test our model and baselines using the EditRoom-DD test set, which contains 500 samples per room type, with lan- guage commands generated by GPT-4o. Quantitative results are depicted in Tables 2 and 3 and qualitative outcomes are illustrated in Figure 3 Table indicates that EditRoom consistently out- performs other baselines across all room types, with notably superior performance in bedrooms. According to Table 3 EditRoom also excels across all editing types. Comparisons between Edit- Room and SceneEditor-N reveal that template- based instructions by more effectively aligning language commands with scene changes. Moreover, the LLM (GPT-4o) demonstrates successful bridge between natural language and template commands. SceneEditor-
simplify the learning process

---

Published as a conference paper at ICLR 2025
Table 3: Performance on single operations with different editing types. From the table, we can
notice EditRoom provides better editing results across all basic editing types.
Translate
Rotate
Model
IOU (*)
S-IOU (1)
I.PPS (L)
CLIP (1)
IOU (1)
S-IOU (t)
LPIPS
CIJIP (?)
DiffuScene-N
0.5363
0.5279
0.1715
0.9484
0.6010
0.5913
0.1307
0.9511
SceneEditor-N
0.5983
0.5905
0.1534
0.9501
0.6655
0.6556
0.1202
0,9560
EditRoom
0.6148
0.6078
0.1388
0.9584
0.6764
0.6673
0.1067
0,9610
Scale
Replace
Model
IOU (*)
S-IOU (t)
LPIPS ())
CLIP (1)
IOU (1)
S-IOU (t)
LPIPS
CLIP (?)
DiffuScene-N
0.6057
0.5947
0.1250
0.9566
0,5994
0.5754
0.1488
0,9305
SceneEditor-N
0.6754
0.6650
0.1105
0.9603
0.6451
0.6193
0.1418
0,9389
EditRoom
0.6881
0.6788
0.1008
0.9647
0.6549
0.6305
0.1384
0,9419
Add
Remove
Model
IOU (?)
S-IOU
LPIPS ())
CLIP (1)
IOU (1)
S-IOU (1)
LPIPS
CLIP
DiffuScene N
0.5312
0.5198
0.1799
0.9263
0.4626
0.4511
0.1877
0.9281
SceneEditor-N
0.5863
0.5745
0.1620
0.9355
0,6160
0.6065
0.1290
0.9489
EditRoom
0.6001
0.5887
0.1588
0.9410
0,6399
0.6316
0.1150
0.9583
Table 4: Ablation on different condition types on the bedroom. From the table, we can show that incorporating source information as context with the self-attention (our design) instead of the cross-altention mechanism can significantly improve model performance.
Model
IOU (t)
S-IOU (t)
LPIPS (L)
CLIP (t)
EditRoom (Concat-Text) 0.5855
0.5746
0.1432
0.9488
EditRoom (Original)
0.7342
0.7236
0.1090
0.9597
outperforms DiffScene-E across all metrics and editing types, suggesting that our graph-based diffusion method yields more coherent and accurate editing results compared to the UNet-based ap- proach. Thus, EditRoom provides more precise and coherent atomic editing operations from natural language commands than its counterparts.
Analysis across different room types shows that all models perform better as the average number of objects in rooms decreases, highlighting potential for improvements in larger, more complex scenes. Evaluating different editing operations reveals that translating, adding, and removing opera- tions score lower on IOU, demanding stronger spatial reasoning. Meanwhile, replacing and adding operations yield lower CLIP scores, indicating a need for better aligmment between object descrip- tions and their semantic features. This underscores the potential for further enhancement of models' spatial reasoning and object alignment capabilities.
Multi-operation Commands To demonstrate the generalization capabilities of EditRoom, man- ually designed several test prompts that combine multiple atomic operations, and we assessed each model's performance qualitatively. Figure 4 shows that EditRoom provides more coherent and ap- propriate responses than the baseline models. For instance, the command in the first row requests a bed replacement and the addition of a wardrobe. EditRoom successfully interprets the natural lan- guage command and translates it into the corresponding atomic operations using an LLM, whereas other baseline models misinterpret the command and perform incorrect operations such trans- lation. These outcomes highlight the challenges of directly training models on natural language commands for compositional editing tasks. EditRoom, by contrast, effectively executes complex editing operations through strategic LLM planning.
To validate our model design, we experimented with an alternative conditioning approach, where graph transformer encodes the source scene into sequence of vec- tors that are then concatenated with text features. These combined features are incorporated into the cross-attention layers of our graph diffusion process. We specifically tested this method the bed- room scene type, with results shown Table 4 The table indicates a significant decrease in model performance, both in terms of layout accuracy and visual coherence. This outcome suggests that
Ablation on Condition Types
8

---

Published as a conference paper at ICLR 2025
headboard cogrece
move the double lowards direction a litte di distance
gr een bofa to size
remove the anay coffee tabie with matte finich
catiret dining table
the
base
chain
one geametric dasign and ceerr glass bulb
Language Command
Source Soene
EditRoom
SceneEditor
DiffuScene-N
Figure 3: Qualitative results single-operation commands. The left column is the source scene with single operation commands for each basic editing type. From the examples, we can find that EditRoom provide more coherent and appropriate editing operations across all editing types.
Table 5: Ablation different text encoders the bedroom. Due to the limited size of training data, we find using the larger text encoder with high- nigh-dimensional features induces decreasing per- formance editing, which indicates further exploration with 3DD editing data generation.
Model
IOU (t) S-IOU (t) LPIPS (1) CLIP (t)
EditRoom
0.6882
0.6743
0.1305
0.9488
EditRoom (Original)
0.7342
0.7236
0.1090
0.9597
utilizing source scene information as the context for self-attention layers, rather than conditions for cross-attention, yields better results.
In an exploration of text encoder options, we replaced the CLIP-VIT- B32 text encoder (512 feature dimensions) with a larger pretrained encoder, OpenCLiP- (1280 feature dimensions), used by OpenShape to align with object semantic with the object features in our models. We conducted tests on the bedroom test set, with outcomes detailed in Table 5 The results indicate that the model equipped with the larger text encoder derperforms compared to the one using the original encoder. We attribute this decrease in perfor- mance to the limited size of our training dataset. Given that diffusion models trained from
Ablation on Text Encoders

---

Published as a conference paper at ICLR 2025
Chenge wooden wardrabe
Remove the lef nightstandd and rotae the bed cegrees make fase the cabinet.
Make luger
Maki the room to be modern style.
Change the table belling tomp more vintage
Spread fumitturs space walkwayy
Language
Command
Source Scene
EditRoom
SceneEditor-N
DiffuSceneNN
Figure 4: Qualitative results on multi-operation commands. The left column is the source scene with multi-operation operation commands. From the figure, we can find the EditRoom can success- fully generalize to complex natural language commands with multiple operations without further training on the multi-operation operation data, while baselines fail execute coherent editing.
scratch, they require more data to effectively align with higher-dimensional features (d = 512 vs 1280). This finding underscores the need further exploration into constructing larger scene editing datasets.
CONCLUSION
In this work, we introduce EditRoom, language-guided 3D room layout editing method. EditRoom incorporates a graph diffusion- based scene editor that facilitates unified basic editing operations. and it utilizes an LLM for natural language planning. Our experiments demonstrate that EditRoom effectively execute appropriate edits for both single and complex operations. We believe this work will inspire further research into language-guided 3D scene layout editing.
10

---

Published as a conference paper at ICLR 2025
Limitation Since EditRoom leverages the LLM for the command parameterizer, its performance contingent upon the LLM's capability 3D scene understanding and natural command com- prehension. This dependency may lead to the generation of erroneous commands that prompt the scene editor to execute potentially problematic operations, such as collisions. However, because the training data predominantly consist of collision-free samples, there is an inherent trade-off between adhering strictly to the commands and avoiding collisions. If the commands deviate significantly from typical scenarios-such moving an object 100 meters away-the model might instead per- form a similar action that falls within the observed training distributions.
Ethics Statement Our work presents a framework for automated 3D scene editing guided by natural language. The primary focus of this research is to advance technical capabilities in scene manipulation. We do not anticipate any ethical concerns or negative societal impacts arising from this work. All datasets used in our research are synthetic and publicly available, and personally identifiable information or human-related data were involved.
Reproducibility Statement All language commands encoded through the pretrained CLIP- ViT-B32 text encoder. Each graph diffusion model includes a five-layer graph transformer model with 512 hidden dimensions and 8 attention heads. Training is conducted using the AdamW op- timizer over 300 epochs, with a batch size of 512 and a learning rate of 2 x 10-4 All models individually trained and tested on each room type. For EditRoom, template commands em- ployed for the scene editor during training. whereas other baseline models utilize natural language commands generated by GPT-4o. During testing, all models receive natural language commands as input. We set the node number of denoise graphs to the maximum object number for different room types (bedroom 12, living room = dining room 21). During training, we pad the objects with zeros to facilitate the batch process.
REFERENCES
Rio Aguina-Kanp Maxim Gumin, Do Heon Han, Stewart Morris, Seung Jean Yoo, Aditya Gane- shan, R Kenny Jones, Qiuhong Anna Wei, Kailiang Fu, and Daniel Ritchie. Open-universe indoor scene generation using IIm program synthesis and uncurated object databases. arXiv preprint arXiv:2403.09675, 2024.
Tagir Akhmetshin. Graph-bused neural networks for generation of synthetically accessible molec- ular structures. PhD thesis, Université de Strasbourg; Kazanskij gosudarstvennyj universitet im.
VI... 2023.
Edward Bartrum, Thu Nguyen-Phuoe Chris Xie, Zhengqin Li, Numair Khan, Armen Avetisyan, Douglas Lanman, and Lei Xiao. Replaceanything3dt Text-guided 3d scene editing with compo- sitional neural radiance fields. arXiv preprint 2xXiv:2401.1789) 2024.
Yiwen Chen, Zilong Chen, Chi Zhang, Feng Wang, Xiaofeng Yang, Yikai Wang, Zhongang Cai, Lei Yang. Huaping Liu, and Guosheng Lin. Gaussianeditor: Swift and controllable 3d editing with gaussian splatting. arXiv preprint arXiv:2311.14521, 2023.
Blender Online Community, Blender - a 3D modelling and rendering penderinggaccaage. Blender Foundation, Stichting Blender Foundation, Amsterdam, 2024. BRLhhtpp:// httt:://wwww.. www wwwpwwwwww.........ww org
Weixi Feng, Wanrong Zhu, Tsu-jui Fu, Varun Jampani, Arjun Akula, Xuehai He, Sugato Basu, Xin Eric Wang, and William Yang Wang. Layoutgpt: Compositional visual planning and gen- cration with large language models. Advances in Neural Information Processing Systems, 36, 2024.
Huan Fu, Bowen Cai, Lin Gao, Ling-Xias Xiao Zhang, Jiaming Wang, Cao Li, Qixun Zeng, Chengyue Sun, Rongfei Jia, Bingiang Zhao, et al. 3d-front 3d furnished rooms with layouts and seman- tics. In Proceedings of the IEEE/CF International Conference on Computer Vision, pp. 10933- 10942, 2021a.
Huan Fu, Rongfei Jia, Lin Gao, Mingming Gong, Binqiang Zhao, Steve Maybank, and Dacheng Tao. 3d-future: 3d furniture shape with texture. International Journal of Computer Vision, 129: 3313-3337, b021b.
11

---

Published as a conference paper at ICLR 2025
Huan Fu, Rongfei Jia, Lin Gao, Mingming Gong, Binqiang Zhao, Steve Maybank, and Dacheng Tao. 3d-future: 3d furniture shape with texture. International Journal of Computer Vision, pp- 1-25, 2021c.
Zhichun Guo, Kehan Guo, Bozhao Nan, Yijun Tian, Roshni G Iyer, Yihong Ma, Olaf Wiest, Xian- gliang Zhang, Wei Wang, Chuxu Zhang, et al. Graph-based molecular representation learning. arXiv preprint arXiv:2207.04865 2022.
Ayaan Haque, Matthew Tancik, Alexci A Efros, Aleksander Holynski, and Angjoo Kanazawa. Instruct-nerf2nefff: Editing 3d scenes with instructions. In Proceedings of the IEEE/CVF In- ternational Conference on Computer Vision, pp. 19740-19750, 2023.
Yining Hong, Haoyu Zhen, Peihao Chen, Shuhong Zheng, Yilun Du, Zhenfang Chen, and Chuang Gan. 3d-llm: Injecting the 3d world into large language models. Advances in Neural Information Processing Systems, 36:20482 224944, 2023.
Ziniu Hu, Ahmet Iscen, Aashi Jain, Thomas Kipf, Yisong Yue, David A Ross, Cordelia Schmid, and Alireza Fathi. Scenecraft: An IIm agent for synthesizing 3d scenes as blender code. In Forty-first International Conference Machine Learning, 2024.
Jiangyong Huang, Silong Yong, Xiaojian Ma, Xiongkun Linghu, Puhao Li, Yan Wang, Qing Li, Song-Chun Zhu, Baoxiong Jia, and Siyuan Huang. An embodied generalist agent 3d world. arXiv preprint arXiv:2311.12877 2023.
Johnny Huynh. Separating axis theorem for oriented bounding boxes. URL.: 2009.
Axis Theorem for Oriented Bounding Boxes.pdf,
Nazmul Karim, Umar Khalid, Hasan Iqbal, Jing Hua, and Chen Chen. Free-editor: Zero-shot text- driven 3d scene editing. arXiv preprint arXiv:2312.1366. 2023.
Lingkai Kong, Jiaming Cui, Haotian Sun, Yuchen Zhuang. B Aditya Prakash, and Chao Zhang- Autoregressive diffusion model for graph generation. In International conference machine learning, pp. 17391-17408. PMLR, 2023.
Chenguo Lin and MU Yadong. Instructscene: Instruction-dtiven 3d indoor scene synthesis with semantic graph prior. In The Twelfth International Conference on Learning Representations, 2023.
Chengyi Liu, Wenqi Fan, Yunqing Liu, Jiatong Li, Hang Li, Hui Liu, Jiliang Tang, and Qing Li. Gen- erative diffusion models on graphs: Methods and applications. arXiv preprint urXiv:2302.0259) 2023a.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. arXiv
preprint arXiv:2304.0844 08485, 2023b.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion, pp. 26296-26306, 2024a.
Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024b. URL httpp:://
llava- vl. wel.githin hub. io/blog//024 30- llava -nxtta
Minghua Liu, Ruoxi Shi, Kaiming Kuang, Yinhao Zhu, Xuanlin Li, Shizhong Han, Hong Cai, Fatih Porikli, and Hao Su. Openshape: Scaling up 3d shape representation towards open-workd understanding. Advances in Neural Information Processing Systems, 36, 2024c.
Rui Ma, Akshay Gadi Patil, Matthew Fisher, Manyi Li, Sören Pirk, Binh-Son Hua, Sii-Kit Yeung, Xin Tong, Leonidas Guibas, and Hao Zhang. Language-driven synthesis of 3d scenes from scene databases. ACM Transactions Graphics (TOG), 3 2018.
12

---

Published as a conference paper at ICLR 2025
Karolis Martinkus, Andreas Loukas, Nathanaël Perraudin, and Roger Wattenhofer. Spectre: Spectral conditioning helps to overcome the expressivity limits of one-shot graph generators. In Interna- tional Conference on Machine Learming, pp. 15179999-999, PMLR, 2022.
OpenAl. Hello gpt-4o. httppp//gggaa....co////aae///lla 2024-07-29.
2024. Accessed:
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748 8748-8763. PMLR, 2021.
Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert- networks. arXiv preprint arXiv:/90k 10084, 2019.
Jiapeng Tang, Yinyu Nie, Lev Markhasin, Angela Dai, Justus Thies, and Matthias Nießner. Dif- fuscene: Scene graph denoising diffusion probabilistic model for generative indoor scene synthe- sis. arXiv preprint arXiv:2303.14207 2023.
Tathagat Verma, Abir De, Yateesh Agrawal, Vishwa Vinay, and Soumen Chakrabarti. Varscene: A deep generative model for realistic scene graph synthesis. In International Conference on Machine Learming, pp. 22168 -211222.1. PMLR, 2022.
Clement Vignac, Igor Krawezuk, Antoine Siraudin, Bohan Wang, Volkan Cevher, and Pas- cal Frossard. Digress: Discrete denoising diffusion for graph generation. arXiv preprint arXiv:2209. 14734, 2022.
Alexander Vilesov, Pradyumna Chari, and Achuta Kadambi. Cg3d: Compositional generation for text-to-3d via gaussian splatting, arXiv preprint arXiv:2311.17900 2023.
Xin Wang. Shinji Takaki, Junichi Yamagishi, Simon King, and Keiich Tokuda. A vector quantized variational autoencoder (vq-vae) autoregressive neural f f.0 model for statistical parametric speech synthesis. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 28:157-170 2019.
Jianing Yang. Xuweiyi Chen, Shengyi Qian, Nikhil Madaan, Madhavan Iyengar, David Fouhey, and Joyce Chai. LIm-grounder. as an agent. arXiv preprint arXiv:2309. 12311, 2023.
3d visual grounding with large language model
Mingqiao Ye, Martin Danelljan, Fisher Yu, and Lei Ke. Gaussian grouping: Segment and edit anything in 3d scenes. arXiv preprint arXiv:2312.00732, 2023.
Guangyao Zhai, Evin Pinar Ömek, Dave Zhenyu Chen, Ruotong Liao, Yan Di, Nassir Navab, Fed- erico Tombari, and Benjamin Busam. Echoscene: Indoor scene generation via information echo over scene graph diffusion. arXiv preprint 55X1 2024.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In CVPR, 2018.
Zuobai Zhang, Minghao Xu, Arian Jamasb, Vijil Chenthamarakshan, Aurelie Lozano, Payel Das, and Jian Tang. Protein representation learning by geometric structure pretraining. arXiv preprint arXiv:2203.06/25, 2022.
Gengze Zhou, Yicong Hong, and Qi Wu. Navgpt: Explicit reasoning in vision-and-languaae naviga- tion with large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, pp. 7641-7649, 2024a.
Kaiwen Zhou, Kaizhi Zheng, Connor Pryor, Yilin Shen, Hongxia Jin, Lise Getoor, and Xin Eric Wang. Esc: Exploration with soft commonsense constraints for zero-shot object navigation. In International Conference Machine Learning, pp. 42829-42842, PMLR, 2023.
Xiaoyu Zhou, Xingjian Ran, Yajiao Xiong, Jinlin He, Zhiwei Lin, Yongtao Wang. Deqing Sun, and Ming-Hsuan Yang. Gala3d: Towards text-to-3d complex scene generation via layout-guided generative gaussian splatting. arXiv preprint arX 2024h.
13

---

Published as a conference paper at ICLR 2025
Jingyu Zhuang, Chen Wang. Liang Lin, Lingjie Liu, and Guanbin Li. Dreameditor: Text-driven 3d scene editing with neural fields. In SIGGRAPH Asia 2023 Conference Papers, pp. 1-10, 2023.
14

---

Published as a conference paper at ICLR 2025
Source Sone (Frozen)
Target Scne (Ditlusion Denoising)
1+77
Figure 5: Visualization of layout diffusion denoising process. The whole diffusion process is conditioned on both source scene and language commands. At the beginning of the process, the target scene layout starts from random noises. After the iterative diffusion denoising process, the target scene layout becomes coherent to source scene and command.
A LLM AS COMMAND PLANNER
(Referred by Section 3.1
A detailed dialog between user and LLM (GPT-4o) is shown in Figure
B DIFFUSION DENOISING PROCESS
In order for better illustration of target scene layout generation process, visualize intermediate steps during the diffusion denoising process for an adding operation example, shown in Fig. [ At the start of the layout diffusion process, the poses and scales of all objects inside the target scene begin as random noise. As the diffusion process progresses, each object's properties are iteratively refined and placed into an appropriate configuration that aligns with the source scene and the given commands. This iterative refinement highlights the strength of the diffusion model in handling scene edits cohesively. By adopting unified approach, the diffusion model is capable of jointly generating the entire scene layout for all editing types, ensuring consistency and coherence across diverse operations.
C EDITROOM-DB PIPELINE DETAILS
A detailed example of using LLM to generate natural language description from template command is shown in Figure
(Referred by Section 4
Add and Remove Objects Removing each object in the scene separately could generate the mod- ified scenes as the result after removal compared to the original scene. Conversely, the original scene could be treated the result after object addition. The formatted editing description will be Addd/Remove [object description]'.
order to consider the location of the addition and potential multiple objects in the scene, will add the relative location description with the closest unique object in the scene, like 'location: [relative description] [reference object description]..
Pose and Size Changes We can similarly repeat the pose change operation for every object in the scene as add/remove. Specifically, we design three operations: translation, rotation, and scaling. For translation, we create random translations as the mix of distances, sampled from 0.1 meters to 1.5 meters with step 0. and directions, sampled along the two axes directions (front/back and left/right). Then, collision checking is done for every translated object until we find a collision-fii sample. The translation will be skipped if all the samples fail in collision checking. The formatted editing description will be 'Move object towards the fromtbback/lettriiht direction for [distance] (object description]"
15

---

Published as a conference paper at ICLR 2025
Similarly, create random rotation angles the mix of uniform direction samples, clockwise counterclockwise, and random values between 15 180 degrees with the step degrees, and check collision for each sample. The check stops when we find collision-fii free sample or all samples fail the checking. The formatted editing description will be 'Rotate object [angle] degrees: [object description]"
For scaling, we separate it as shrinking and enlarging. The scaling factor is randomly generated be- tween 0.5-0.8 1.2-1.5. The scaling factor uniformly applies three dimensions. Since shrinking won't cause a collision with other objects, it can always result in a successfully modified scene. For enlarging, collision checking fails on all trials, the enlarging is skipped. Otherwise, we save the largest collision-fiee scaling factor. The formatted editing description will be 'Shrink/Enlarge object by (scale factor] times [object description]'
Object Replacement For the replace operation, we access object dataset with semantic class labels and 3D meshes. The system will retrieve several objects from the dataset with the same class label as the replaced object, and check their collision with other existing objects in the scene. If none of the objects could be placed without collision, we randomly select one object and shrink its bounding box be equal smaller than the replaced object avoid collision. The formatted description is 'Replace source with target: [source object description] to [target object description]..
Collision Detection Module The objects are abstracted as 3D bounding boxes and further decom- posed into 2D bounding boxes on horizontal plane and vertical range, as the objects can only rotate about the vertical axis. Then, the two objects are only in collision if their 2D bounding boxes overlap and their vertical ranges overlap. For 2D bounding box collision detection, we apply the Separating Axis Theorem Huynh (2009) to determine if the boxes intersect.
16

---

Published as a conference paper at ICLR 2025
System Prompt
Imiiii e you are ndoor rcom designer and you are using provided API to contrtt the 50 models in the soonc, sporee configuriioo in command od scone. you ahoule
planiin zorlove the provided The dimonsion sccuence (uuad Vertical angles the along Stacs ore the hal longhhs the boun ding box llong the y, and axes whin the vorkaa arggl la zoro.
controlds
We dofine wula os the fight directio, +yo as the up upldown dtroctin, and ealiz de frontbock direction. Poollive argles dounterclodwwoo. and negativ ang dadcewis.
API::
Rutate object (Rosst, Target Ofject Desoristion Angle (degroes@
Tronslate object
Desoripton, Direction (wlytt, Distance (meters) Ooject Desoripiion,
Focter
Rep ooe object Replach Source Ofject Desombiton, Targel Orject Descripton)
5. Asd an etject (Aad. Target Ciject Descripton))
Nemoro object (Pamono, larget Objeci Descrition)
Matters noeding attention:
If thors are mullppp sams exjects the soono and the command is rolated to the déject you should reler to the atject locations. When
you
obused locations, you
format Deseriptoo, Relaaive Object Deser plint.
Rolstve Desoription: Jol, right, Font of behind, above, beloww closdly closely right closcty in foot of, dlssety beh Vlesety mians the distanoe botween too object controlds loss than metors 1-4 planc.
examppii plllll wonk
chair front sable you
the Iaas, Chair, flont Tablcy.
scolo,
Considor the prosential coll sion bobecon oijects when you cacoute Tie comn nannands.
Whon you poale object, the ofject shou il be scoled unifomlty Scale foolor should one fooo nu number.
Houp otpect well only was the otject with same casa. Deppbaoe command will only change the sttject apporraaion, not the oipect poess and targei,
provicedd coremnncc image
undersland the scanw.
Attempt to uaw the menamum rumber commantte to actarne the targer.
a you woant to nemrvee convarted but
the dépect withen the same class, you ahourd use the reptace command.
upra shoold
numbers, which diructy presessed poon loucha))
examppee:
2. to wad chair tront of the sabie, should use the format: (Aoe) lchal", f'n front of," "atrin".
3. you want lo nemove chair, you ahooudd use the format [Reemove" (thar). you repiace mittal
woodinn one and
badt you ahould use the formal (Roplacee 'mental chai"; woooen chaif, (dett, boc'..
Think abbout step. Summarow the used the
The find formut abouudd "map Toep:t....
User Input
Ctject (aaass MMbubll bes", Naaee (1.01, 0.29, 1 "verical angle" -00. "controid" 1.22, U.D. -2,48)
Wasee plaon The douple bed moders minialao deskgn with white coler scheme and simple, disan apocorance.)
"ightsland" "iae"" 2828, 0,23), vatical angle" "o
Object 2: ("elass" sightsland" "Sioe' 12.33, 0.59 0.23), vertial angle" "oortrad" 56. 0.0.-3300)
"apser plaon righto bound is skmple, wiile lao draner ploce with smooth Frish end small round krob on each drowe:
("llass. wardross" "Maz" 11.04, 1.2 0.50 "vertaal angle 900 Controld"11..5
-245)
Clbyed A: ("iaeer" "wak"" "ere" "ceecr plaen The dook aia modemn menaral al deeign with a slwec. rezteggrar shape andia mode gnay bean ?
0.3) vertcal wngle" -20 "cantroih" (5.K 0.O, -2.4%)
[Cammennd] two raghteeend neer the catirel
earra model benhhernghittrrd rwar black tedre
symeretrs
replace the nightstand near the white cabinet with the same model another nightstand near the black table, then move it closer to the bed corner to be symmetric with another one.
LLM Response
bat command replading righnstand near the while
other rightstand near and moving to nymmetii rign rightaaed
hollow Identify the nightrtnce::
a Niggisstenn near the oxtiiee (wendrobe) the divkvr wnos nightsteed centrodd PP p2.05, o.n. -4.87]
Repinon deeker girrutand with the whih rightesend
Mous this replooed nighratard closer the the to symmercc with the othwr rightrtennt
to menitain symmmeryy. the righaroedd aeould an poed onved doww the corner, the right diwcttoo.
langet
"Spppaac the dank wood righ segee
rightstand
pusn
[Teeetaa modeem, chark wood nightdunnt himple, while twnedran now nighteeend" (twar) 'wh catioee]]
TTannsttee replaced rightstanc
dircctio
symmerrii."
Considoring ri there is another in rightstand near the bod (on the right skbo), the new one should move doosely to the same distanoe as the first one but the nemiie pos llon tirt the bedi
"pon
ee 'Simple white
nightstand. X..
PhaalAPl commance usedt
""Replace righesaadd
[Replaos" Tmodem, dars wood reyndoow "timpte, wh neo-drawer rightrtana" ("heatt "wilite sabiner]]
rightsann
symmery
rightrondi
"Tecn
(Tranduas", 'smppk,, while two drawer rightstand", 'X. (222)
Contbiiing these together, the ou
""["Replace", "modem, dark wood nightstand", "simple, white two swo-drawer rightstand", ("near", "white cabinet'l]:
["Translate" "simple, white pro-drawer nightssand". "x", 1.22)""
Figure 6:
Dialog between user and LLM to generate edit plan to rearrange the bedroom. The room image in 'User Input" is not given to LLM, just for illustration purpose.
17

---

Published as a conference paper at ICLR 2025
System Prompt
Given scaee comiguun pout and bemplstec commanti you should wrthe newe commrande uxing rel rut language and sparts neteneen seteenoo Terppiited commantee be aethon: target sojec! fermot the location pros the tempisted commentes, can conaidered barget onjeet location centrcite
maters. anglen delined degrees The dimeraton
Mertical
slong
Steee ane the hall wingtee of the bounding box skong the y, and mooee when the versor angle seea.
daline as rightset direction, 1yly us upildoon direction and sake trontibcck dedcioo
Wian thesign commannds please
camedraess, con corchareee,
You anould atarmpt to make your command rewd resscring.
ane cuprica larget copedta scams, should odpect location by netative asated neiad ore with one unique sbgeet scaree ammiar obgect should ixticla
Il tampleted commannaa indicate
oiject where
addingg new object your command
Entagge and strins er the commant und ahould unilorm.
You can soject descristion accordting to the scone corrigurations trmm commancc, and image proviced).
Example::
[lumplaoed commancai move object lowards the "msutrr rection for meterss while bed with red and while plaid comforter and red and white pla pi ow. ungth bed meters, you wiil:: trove the witl and white lowards sabi
satie ona sable rosks scenct
Exumpiec
[Tomplazed commancs) mooe object lewards the "mgatt rection for 8 metors, o woodon rightslandd para wild iighstand morinng to make the rightsland
cord you wiite: trove rig itssand deser the hoadboard by
Exampics:
(Tomplaned commancs]] ricpaoe source with sanget: I (uuroo) white bed (Torget bown bed.
You dan soe while
[Tooppaged commancs)] I rodd object a white bed. loorioon: rrrgghrt wordndoo]]
ther wardrote the sceni, you con whoho add while bos on the right side the wardrobe."
you can design comn con be processed ast Iienal eeveltt or Jcade).
fouuniatt fioal format indiinotuaal commandi
comreands, can supplementt command. shiings
User Input
[Scone confguratioon::
Object 6: Puass" diiing lidke" "sibe 10 55 0.38, 223 "vorkoal angle": 0. "conbeid" 10 C.O. -3.43.46
"sie"" méniiaalitt design Vveiica angle"
marble top and a metal frmee.)
Object Polass "wesest
(-378,
Mescc (ption" "he oveseat sot brown with modem dodign and has variey of pottemed throw pllows.)
Object 2: Poass" coffee tile. "Mlo" P 65, E. 0.47) "verical angle eogietB0p "coriii" (244) 0.0.
design
"vertoll geomenic
combination
lighter penels
Objent 3: ("aass"
thair. "deaciption iptio" The caar a modem, mimimollst dedin with dark wood fram and oustionn featuring geomeeri patwm.))
0737.3
163, "ceemold"" [12.4,
Orject t: ("ass" comer side moae". "Now" 22. 20 2.22| "vertor ongie 90, "comtoic"" [3.98, -6.2%)
Otject &o ("ias"" daving
"row" 6.45 0.3) "eriol anglet 1000 "canimaf" (0:47,
"deec iption the mar blok with modwn dedign fenninng high back and amrowe-)) -
Orject in ("asse" dainn thei"
6.45 0.30 Twitil angle Toaaima 11100,0 myy
Orject 7: ("dase" come side model "oww'
0.22)
Towmrich (3.9 3.13.
"deecrppion" mund bark mottle ratee with while teen
Objectt in ("isse" daving "Row" 1131 0.45 0.3) Twetic angle: foustoot on. -4.17
Oijjett in ("daer" caring "Now" (131. 0.45 (.3) "wiiio angle" towshooh (0.40) oz, "deectplion" The caer black with modwen dedign henuinn hight back and aemoees))
Cijeett ('daae "tmeed bew "eaa" (17,
1.1 "wertad angae "orrmach (3.35 o. 4,59]
"deectptoo""
Cijject ("theer" "catewe" "aza" (8.50 1055 222
angle" "centrof" 94, oi, 2000)
"deectptosr" The on th cacesel is reccem meneral design with light wose trmm and a white cator schems reaturnn playfal surny mobil on the doom..y Objeett (idaas pwerdtt baro". "ese" 18,
"ventee ample: "certrca" p2.r1 438]
"deactptor""
geometre patten.")
Oijject ('dass" "pendaet
"aaa"' 0.22. "veriaa angle: "centrad" (2.20 1.55, 40.)
"Seacrpteur" The pendart lamp is brack with woven design and whtee stanto
(Tamplated commands]:
object degroes: the pendant black design interiar.)
white interior",
"obviausly object degress pandant lamp black
target object Object_
Response
"*MM[[Rotate black pendant lamp woven design by degrees.
Figure 7: Dialog between user and LLM to generate natural language description from the template command. The room image in 'User Input' is not given to LLM, just for illustration purposes.
18


Reviews:
[{"review_id": "FA48ogOihX", "reviewer": "Reviewer_RttZ", "summary": "Given a source 3D scene and a natural language command describing the scene edits, the paper presents a method to generate the  corresponding 3D edited scene.\n\nThe existing gaps that motivate this paper are three folds: (a) Requiring manual intervention to perform 3D scene edits, (b) limited number of edits and a lack of support for multi-operational textual commands, and (c) lack of a language-guided scene editing dataset, making it difficult to train and evaluate scene-editing models.\n\nTo this end, the contributions of the paper include proposing a new approach that combines an LLM (to encode edit commands) with a graph diffusion model (to incorporate scene edits from the input language command) to effectively serve the high-level goal of synthesizing a 3D scene based on natural-language edits/commands. The framework is termed “EditRoom”. In addition to the proposed approach, the paper also introduces a new synthetic scene editing dataset named “EditRoom-DB” with 83K pairs of language+edited 3D scene, spanning 16K initial 3D scenes.\n\nTo summarize, EditRoom takes as input a source scene and a natural language command describing scene edits, and outputs a corresponding edited scene. The dataset used is “EditRoom-DB”, which is a new dataset of scene editing pairs created from 3D-FRONT  dataset. GPT-4o is the LLM used, and there is a graph-transformer based diffusion model. The learning setup is strongly supervised. The loss functions involved are KL divergence loss and diffusion loss for graph diffusion model and layout diffusion model. In terms of evaluations, the paper looks at the following four quantitative metrics: (1) IoU score: 3D IoU between objects in pre-edit and post-edit scene, and selecting pairs with highest 3D IoU value (Q: how many pairs do you use?) (2) S-IoU score: semantic similarity between captions of matching objects as measured by Sentence BERT (Q: how do you find caption for generated objects?) (3) LPIPS: determines pixel similarity between 24 views of pre-edit and post-edit scenes, and (4) CLIP score: Using CLIP-ViT-B32 to find semantic similarity between 24 views. As for comparisons, the paper compares against two works: DiffuScene-N (modified version of existing 3D scene synthesis model DiffuScene, from CVPR 2024), and  ScenEditor-N. (Spelling mistake: Line 323, the name of the baseline should be DiffuScene-N, instead of DiffuScene-E)", "questions": "1) Is the scene edit one-shot (i.e., take all edit commands at once and generate a scene) or progressive (i.e., edit the 3D scene one command at a time)?\n2) Motivation behind using a diffusion model is unclear. If the edits allowed are simply adding,\nremoving, replacing, scaling, translating and rotating an object (even the relations\nbetween objects are pre-defined and limited), why do you need to train a diffusion\nmodel? You could perform these tasks directly on the source scene graph by\nadding/removing/replacing a node and then retrieving an object from the given object\ndatabase that fits the text description (see Language-Driven Editing of 3D Scenes from Scene Databases, SIGGRAPH 2018; missing this reference). Using diffusion model makes sense when one\nwants to generate an entirely new scene/new object from scratch. In the scene editing\ntask, we are given with an initial scene and our task is to modify it.\n3) How does the diffusion process look like for, say, rotation operation? It would be\nhelpful to have visualizations of intermediate diffusion steps. Applying rotation to\nexisting object can be done analytically. Why would we need a diffusion model? I would consider this a weakness of the paper as it precludes some important visualizations pertaining to the proposed approach and something that is relevant to the end results. \n4) Regarding the proposed dataset “EditRoom-DB”, what is the reason behind having only\n500 examples as test data? This number is too small compared to the number of\ntraining samples i.e. 83k. The standard practice is to have a certain percentage (say,\n10%) of total samples as test data. Looking at Table-1, dataset in its current form is\nbiased towards “bedroom” category because it has ~50k training and only 500 test\nsamples, as compared to ~18k train and 500 test samples for “dining room” category.\nThe ratio of train-test data is high for bedroom scenes compared to the other two categories, and its\neffect is seen in the results Table-2 (high performance for bedroom than dining and\nliving room). It would be a good experiment to run when you keep the train:test ratio\nequal for all categories and see if the model gives comparable results.\n\nOverall, the paper proposes a novel 3D indoor scene editing model that uses LLM to encode\nscene editing instruction and a graph-diffusion model to perform scene edits. Moreover, it introduces a \nnew dataset for training and evaluating 3D indoor scene editing task.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1) Well-written paper\n2) The idea of breaking down command into multiple steps is interesting\n3) The proposed model can be used for multi-operation editing without explicitly training\non such commands. This is a useful feature for interior designers\n4) The newly introduces dataset “EditRoom-DB” is quite useful to the 3D scene research community", "weaknesses": "1) The motivation to use diffusion model is unclear.\n2) Out of the two compared methods, only one, DiffuScene-N, is based on an existing work (viz., DiffuScene, from CVPR 2024, Tang et. al.). The other one is a modified version of the proposed approach. This is good, But, why not use a modified version of InstructScene (Haque et. al., ICLR 2024) and also of EchoScene (Zhai et.al. , ECCV 2024) as well? This would be tell a good story on how effective the proposed approach is. This is a weakness, something I believe can be addressed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lHnpGdHYUR", "reviewer": "Reviewer_SwhT", "summary": "This paper introduces EditRoom, a two-module scene editing framework based on natural language prompts. First, the command parameterizer converts natural language prompts into sequences of breakdown commands for six editing types using an LLM. Then, the scene editor constructs the target scene from the source one using the proposed conditional graph generation models based on diffusion. Quantitative and qualitative results prove the effectiveness and generalization capability of the proposed approach.", "questions": "All of my questions are listed in the weaknesses section, and I may adjust the rating if they are well addressed.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "a) The problem studied in the paper is important for real-world applications. b) This paper introduces a synthetic scene editing dataset called EditRoom-DB, contributing to the field. c) Support for multi-operation commands enhances the practicality of the proposed approach.", "weaknesses": "a) Comparison with baselines: The authors mention the difficulty of adapting scene synthesis methods for layout editing, and thus compare with only two baselines. However, several LLM-based methods using in-context learning (e.g. LayoutGPT) could potentially be adapted to support editing by defining the task for the LLM and conditioning on in-context learning samples from EditRoom-DB. Including such a comparison with in-context learning-based methods would enhance the evaluation. b) Missing complexity analysis: In this work, the authors train two graph diffusion models, however the complexity analysis is missing. How much time do the training and inference take?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ACykyTLPa2", "reviewer": "Reviewer_d67d", "summary": "This paper proposes EditRoom for 3D layout editing of an existing room. It first uses GPT-4o to parse the natural language instruction into a sequence of breakdown commands, and then trains a diffusion model to generate the edited layout based on the commands. The framework is reasonable and utilize the state-of-the-art LLM and generative diffusion model. The proposed approach generates good results and outputs the other variants of itself, validating the effectiveness of using GPT-4o to infer the breakdown commands and the design of the graph diffusion model as 3D scene editor.", "questions": "Overall, this paper presents a novel framework for language-based 3D room layout editing. The main weaknesses lie in the motivation and evaluation of the proposed approach. The current evaluation only validate the key designs under the novel framework (first LLM and then a diffusion model), but lacks the explanation on why this framework is essential (especially the diffusion model as a scene editor).", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The strengths include:\n\n1. It proposes the 3D layout editing pipeline, which uses GPT-4o as the command parameterizer and a graph diffusion model as the scene editor.\n\n2. The graph diffusion is tailored to deal with the layout editing problem and outperforms the adapted DiffuScene approach.\n\n3. It constructs a large dataset composed of the paired scene layouts and the corresponding editing instructions, which facilitates the following language-based 3D layout editing research.", "weaknesses": "**Firstly, it lacks the discussion of the motivation for the target problem: why and when a user would prefer a language-based 3D layout editing rather than interactively modify the one or two object placements by himself?** This relates with the design of evaluation experiments. Intuitively, the problem setting should be a complicated editing and a very rough instruction. For example, the user might not explicitly specify all the atom editing, but rather describe the key intent only. Or it requires a series of operations to make enough space to enable the target editing. The single-operation and multi-operation cases shown in the paper seem straightforward. Actually, the second row  in Fig.4 gave a good example, since a user might express “rotate the bed” but desire to have the bed s well as the accompanied nightstand rotated together. But it seems that the proposed approach only edit the objects that are explicitly specified, and produces an unreliable layout result.\n\n**Secondly, it lacks the comparison with heuristic methods.** The proposed approach trains a graph diffusion model as the scene editor. But the paper only compare with DiffuScene (another diffusion model for 3D room layouts) and its own variants, only to validate the key designs such as taking breakdown commands as conditions and the condition types of the network. In fact, training neural networks for 3D layout problem often aims to deal with high-level conditions and efficiently generate plausible layouts. Given that the GPT-4o has already inferred the breakdown commands and the generative process of diffusion model is slow, do we really need such a network to produce the edited layout? What if directly parse the breakdown commands to modify the specified object? What if using a simulated annealing algorithm as a post-processing step to deal with the possible collisions? At least it alleviates the demand for constructing a training dataset.\n\n**Thirdly, it lacks a detailed discussion of the command parameterizer.** Currently, there are different opinions on whether the LLM has satisfactory spatial reasoning ability. Fig.5 is a good example as the goal is to make the two nightstands symmetric with each other. But it seems the LLM response presented in the figure is incomplete. It is important to give a in-depth and detailed analysis on the LLM performance, e.g. when the LLM would fail and how to promote its ability?\n\nThere are some minor issues. For example, are the point cloud of each object (line 227) only point locations or rgb color as well? How to “concatenate each element of source scene graphs into noisy target scene graphs” (line 241)? It means to concatenate the attributes of corresponding elements, or merge the node lists?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)