EditRoom: LLM-parameterized Graph Diffusion for Composable 3D Room Layout Editing

## Abstract

Abstract Given the steep learning curve of professional 3D software and the time-consuming process of managing large 3D assets, language-guided 3D scene editing has significant potential in fields such as virtual reality, augmented reality, and gaming. However, recent approaches to language-guided 3D scene editing either require manual interventions or focus only on appearance modifications without supporting comprehensive scene layout changes. In response, we propose EditRoom , a unified framework capable of executing a variety of layout edits through natural language commands, without requiring manual intervention. Specifically, EditRoom leverages Large Language Models (LLMs) for command planning and generates target scenes using a diffusion-based method, enabling six types of edits: rotate , translate , scale , replace , add , and remove . To address the lack of data for language-guided 3D scene editing, we have developed an automatic pipeline to augment existing 3D scene synthesis datasets and introduced EditRoom-DB , a large-scale dataset with 83k editing pairs, for training and evaluation. Our experiments demonstrate that our approach consistently outperforms other baselines across all metrics, indicating higher accuracy and coherence in language-guided scene layout editing.

## 1 Introduction

Traditionally, editing 3D scenes requires manual intervention through specialized software like Blender (Community, ), which demands substantial expertise and considerable time for resource management. As a result, language-guided 3D scene editing has emerged as a promising technology for next-generation 3D software. To build an automated system capable of interpreting natural language and manipulating scenes, the system must be able to align complex, diverse, and often ambiguous language commands with various editing actions while also comprehending the global spatial structure of the scene. Additionally, the relatively small size of available 3D scene datasets presents a challenge for developing large-scale pretrained models necessary for fully automated, end-to-end language-guided scene editing.

Recently, there have been some works (Zhuang et al., ; Bartrum et al., ; Karim et al., ) focusing on leveraging pretrained image generation models to edit single object appearance inside the scene, but they fail to modify the layout of original scenes. Meanwhile, other approaches (Chen et al., ; Ye et al., ) further incorporate pretrained segmentation models to enable individual object manipulation. However, they require manual intervention to determine the edited object and editing type for any layout adjustments, like adding a new object or changing the object pose. Furthermore, these methods are all limited to executing a single editing step and are hard to deal with commands with multiple potential steps.

Therefore, we propose EditRoom, a unified framework that can execute all editing types by complex natural language commands without intermediate manual interventions.
It consists of two main modules: the command parameterizer and the scene editor. The command parameterizer employs an pretrained LLM, specifically GPT-4o (OpenAI, ), to transform natural language commands into sequences of breakdown commands for six basic editing types on single object: adding, removing, replacing, translating, rotating, and scaling. These breakdown commands, along with the source scenes, are then fed sequentially into the scene editor for execution. To unify all editing types, we convert scenes into graph representations and construct scene editor as conditional graph generation models, where we take source scenes and text commands as conditions and train diffusion models to generate the target scene graphs with the layout.

Another challenge is the lack of language-guided scene editing datasets, which constrains both training and evaluation of scene editing models. To enable the scene editor to accurately execute every basic editing type, we construct an automatic data generation pipeline and collect a synthetic scene editing dataset named EditRoom-DB for both training and evaluation, which includes approximately 83k editing pairs with commands. We implement several functions to simulate the single editing process on the existing 3D scene synthetic dataset, 3D FRONT (Fu et al., ), which contains 16k indoor scene designs equipped with high-quality object models, and generate corresponding language commands using predefined templates. To mimic the human inputs, we employ LLMs to transform the templated commands into natural language forms, serving both as training material for our baselines and as test cases for single-operation evaluations.

In the experiments, we quantitatively assess the performance of EditRoom in scenarios with single-operation commands. From the results, we find that EditRoom outperforms other baselines in all metrics across different room types and editing types, which indicates higher precision and coherence in single-operation editing. Furthermore, we qualitatively evaluate EditRoom in scenarios involving multi-operation commands. We find that the model can successfully generalize to these scenarios even though we do not train the model on multi-operation data.

Our contributions are summarized as follows:

- •
We propose a new framework, named EditRoom, consisting of the command parameterizer and scene editor, which accepts scene inputs and can edit scenes using natural language commands by leveraging LLM for planning.
- •
We propose a unified graph diffusion-based module that serves as the scene editor, capable of executing every basic editing type, including adding, removing, replacing, translating, rotating, and scaling.
- •
We introduce the EditRoom-DB dataset with 83k editing pairs for the first language-guided 3D scene layout editing dataset by augmenting the existing 3D scene synthesis dataset.
- •
From the experiments, we demonstrate that EditRoom outperforms other baselines across all editing types and room types on single operation commands, and it can generalize to complex operation commands without further training.

## 2 Related Work

Language-guided 3D Scene Generation
Currently, some language-guided 3D Scene generation works that claim for language-guided scene editing capablility can be abstractly categorized into two main approaches. The first approach starts with generating the scene configurations by utilizing large language models, and prompt large language model for following editing (Vilesov et al., ; Zhou et al., ; Aguina-Kang et al., ). However, they are limited to editing its own generated scenes and do not accept source scenes as input. The second approach is about training diffusion-based model to learn text-conditional scene generation (Haque et al., ; Tang et al., ; Zhai et al., ). During the editing, they needs to manually mask the editing part and leverage the completion ability of diffusion models. Besides, their editing types are limited. InstructScene (Haque et al., ) does not add language conditions to the scene layout, so translating, rotating, and scaling cannot be applied to the scenes. EchoScene’s manipulations only apply to object relations so that they can only add an object or change the relative positions of objects (Zhai et al., ). In contrast to these works, EditRoom can accept an existing scene as input and apply free-form editing commands for 3D scene layout without manual interventions.

Language-guided 3D Scene Editing
Recent language-guided 3D scene editing works mostly incoprate neural field representations and pretrained image generation models for object appearance editing. Some works are mainly focusing on replacing single object (Zhuang et al., ; Bartrum et al., ; Karim et al., ) but failing to edit the layout. Other approaches (Chen et al., ; Ye et al., ) leverage pretrained segmantation models to obtain the individual object representation so that they can remove the objects by manually selecting the target objects. Besides, all these methods can only take one single-operation command at each interaction. Compared to these previous works, EditRoom can leverage LLM to deal with multi-operation natural language commands and automatically execute all editing types through a unified graph diffusion-based module.

LLMs for 3D Scene Understanding
Recent works demonstrate that existing LLMs can facilitate 3D spatial reasoning. These works usually leverage the pretrained image caption models to convert 3D scenes into text descriptions and ask the LLM to generate navigation steps (Zhou et al., ; ), provide room layout (Feng et al., ), or ground 3D objects (Yang et al., ; Hong et al., ; Huang et al., ). In our work, we are the first work to leverage LLM for natural language-guided 3D layout editing, where LLM takes source scenes in text format and breaks the natural language commands into basic editing operations.

Diffusion Models for Graph
In recent years, denoising diffusion models have shown impressive generative capability in the graph generation (Liu et al., ; Kong et al., ; Vignac et al., ). Compared to the previous VAE-based Verma et al. () or GAN-based (Martinkus et al., ) models, diffusion-based models have advantages like stable training processes and generalizability to various graph structures. Some works have presented that graph diffusion-based models can be used in molecule generation (Guo et al., ; Akhmetshin, ), protein modeling (Zhang et al., ), 3D scene generation (Haque et al., ; Zhai et al., ), and etc. In this work, we first propose to use graph diffusion-based models for language-guided 3D scene layout editing.

## 3 The EditRoom Method

In this section, we introduce EditRoom, shown in Figure 1, comprising two primary modules: Command Parameterizer and the Scene Editor. Given a natural language command $\mathit{C}$ and source scene $\mathit{S}$, we aim to estimate the target scene $\mathit{T}$ with conditional distribution $q(\mathit{T}|\mathit{S},\mathit{C})$. Our command parameterizer takes the source scene $\mathit{S}$ and natural command $\mathit{C}$ to generate the breakdown commands $\mathit{L}$. Then, the scene editor conditions on breakdown commands $\mathit{L}$ to obtain the final target scene $\mathit{T}$, where the whole pipeline can be written as $q(\mathit{T}|\mathit{S},\mathit{C})=q(\mathit{L}|\mathit{S},\mathit{C})\times q
(\mathit{T}|\mathit{S},\mathit{L})$. All objects inside source scenes and target scenes are retrieved from a high-quality 3D furniture dataset (Fu et al., ).

Figure: Figure 2: Scene Editor Overview. Scene Editor aims to provide accurate, coherent editing results according to the given source scene and language commands. It consists of two graph transformer-based conditional diffusion models. One diffusion model generates semantic target scene graphs. Another diffusion model can estimate accurate poses and size information for each object inside the generated target scene graphs. All diffusion processes are conditioned on the source scene and breakdown command.
Refer to caption: x2.png

### 3.1 LLM as Command Parameterizer

In order to process open natural language commands, we use GPT-4o (OpenAI, ) to convert natural language command $C$ into a set of combinations of basic editing types with breakdown commands $L:=\{l_{j}\}_{j=1}^{N_{L}}$, where $N_{L}$ is the number of breakdown commands.
To cover the general manipulations on the scene, we design six basic editing operations:

- •
Rotate an object: [Rotate, Target Object Description, Angle]
- •
Translate an object: [Translate, Target Object Description, Direction, Distance]
- •
Scale an object: [Scale, Target Object Description, Scale Factor]
- •
Replace an object: [Replace, Source Object Description, Target Object Description]
- •
Add an object: [Add, Target Object Description, Target Object Location]
- •
Remove an object: [Remove, Target Object Description]

When the target object is not unique, we ask the LLM to use another unique object as a reference to describe the spatial relation. During the inference phase, we prompt the LLM with attributes of objects within the source scene along with the natural language command, tasking the model to analyze the scene and delineate basic editing operations through breakdown commands in specified formats. The attributes include categories, locations, sizes, rotations, and object captions. Detailed descriptions of the full prompt and examples are provided in Figure 5 of the appendix.

### 3.2 Graph Diffusion as 3D Scene Editor

Given the breakdown command $l$ and source scene $S$, our objective is to determine the conditional target scene distribution $q(T|S,l)$. Drawing inspiration from recent advancements in language-guided 3D scene synthesis (Lin & Yadong, ), we transform scenes into semantic graphs and employ a graph transformer-based conditional diffusion model to learn the conditional target scene graph distribution, as depicted in Figure 2. Our approach involves two key graph transformer-based diffusion models: the Target Graph Diffusion, which generates object shapes and their spatial relations as graphs, and the Target Layout Diffusion, which computes the final layout of the target scene.
To reduce the alignment challenges between the 3D scene distribution and language, all commands are encoded using the text encoder of CLIP-ViT-B-32 (Radford et al., ).

Scene Graph Representation
Each scene is represented as a combination of a layout $B$ and a scene graph $\mathcal{G}$ (Lin & Yadong, ). The layout $B$ encapsulates the position, size, and orientation of each object, while the scene graph $\mathcal{G}$ encodes additional high-level semantic information. Formally, a semantic scene graph $\mathcal{G}:=(V,E)$ comprises nodes $v_{i}\in V$, where each $v_{i}$ corresponds to an object $o_{i}$ with high-level attributes. Directed edges $e_{ij}\in E$ represent spatial relationships: [“in front of”, “behind”, “right of”, “left of”, “closely in front of”, “closely behind”, “closely right of”, “closely left of”, “above”, “below”], connecting the $i$-th object to the $j$-th object, where “closely” means the distance between two object centers are less than 1 meter. Each node $v_{i}$ is characterized by a discrete category $c_{i}$ and continuous semantic features $f_{i}$, derived from a pretrained multimodal-aligned point cloud encoder, OpenShape (Liu et al., ), which features a 1280-dimensional representation space.

Target Graph Diffusion
In this stage, we aim to learn target scene graphs $\mathcal{G}_{tg}$ by giving source scene graphs $\mathcal{G}_{s}$ and language commands $l$ through a discrete graph diffusion model $\varepsilon_{g}$, where $\mathcal{G}_{tg}$ includes category $C_{tg}$ and semantic features $F_{tg}$ for each node and the edges $E_{tg}$ for object relative relations. Since high-dimensional object semantic features ($d=1280$) are too complicated to learn from limited data, we use a VQ-VAE model (Lin & Yadong, ; Wang et al., ) to compress them into low-dimensional features $z\in\mathbb{R}^{n_{f}\times d_{\mathit{Z}}}$, which consists of $n_{f}$ vectors extracted from a learned codebook $Z\in\mathbb{R}^{K_{f}\times d_{\mathit{Z}}}$ by a sequence of feature indices $f_{idx}:=\{1,...,K_{f}\}^{n_{f}}$, where $K_{f}$ and $d_{\mathit{Z}}$ are the size and dimension of codebook. Then, we use the feature indices to replace the original object semantic features as targets for training, denoted as $\hat{F}$. Therefore, $\mathcal{G}_{tg}=(C_{tg},\hat{F}_{tg},E_{tg})$ and $\mathcal{G}_{s}=(C_{s},\hat{F}_{s},E_{s})$, and our goal is to learn the conditional distribution $q(\mathcal{G}_{tg}|\mathcal{G}_{s},l)$. During the training process, at timestep $t$, the gaussian noises are added to the $\mathcal{G}_{tg}$ to get $\mathcal{G}_{tg}^{t}$, and the model $\varepsilon_{g}$ aims to reconstruct $\mathcal{G}_{tg}^{0}$ by conditioning on $\mathcal{G}_{s}$ and $l$. To add the conditions, we concatenate each element of source scene graphs into noisy target scene graphs as context and use cross-attention layers to incorporate language features.
The loss function can be written as:

$$ $\displaystyle L_{g}:=$ $\displaystyle\mathbb{E}_{q(\mathcal{G}_{tg}^{0})}[\sum_{t=2}^{T}L_{t-1}- \mathbb{E}_{q(\mathcal{G}_{tg}^{1}|\mathcal{G}_{tg}^{0})}[\log p_{\varepsilon_ {g}}(\mathcal{G}_{tg}^{0}|\mathcal{G}_{tg}^{1},\mathcal{G}_{s},l)]]$ (1) $\displaystyle L_{t-1}:=$ $\displaystyle D_{KL}[q(\mathcal{G}_{tg}^{t-1}|\mathcal{G}_{tg}^{t},\mathcal{G} _{tg}^{0})||p_{\varepsilon_{g}}(\mathcal{G}_{tg}^{t-1}|\mathcal{G}_{tg}^{t}, \mathcal{G}_{s},l)]$ (2) $$

where $D_{KL}$ indicates the KL divergence.

Target Layout Diffusion
In this stage, we aim to estimate the target scene layout $B_{tg}$ using another graph diffusion model $\varepsilon_{b}$, conditioning on target scene graph $\mathcal{G}_{tg}$, source scene graph $\mathcal{G}_{s}$, source layout $B_{s}$, and language command $l$. The target scene layout $B_{tg}\in\mathbb{R}^{M\times 8}$ consists of position $T_{tg}\in\mathbb{R}^{M\times 3}$, size $S_{tg}\in\mathbb{R}^{M\times 3}$, and rotation $R_{tg}\in\mathbb{R}^{M\times 2}$. During the training process, gaussian noises $\epsilon$ will be added to the target layout, and the layouts are encoded into the node features by MLP layers. Similar to the Target Graph Diffusion, we concatenate the source scene graph and source layout to the target scene graph as context and corrupted target layout. The language features are incorporated through cross-attention layers. The objective target is to estimate the added noises at each time step. The loss function can be written as:

$$ $\displaystyle L_{b}:=\mathbb{E}_{B_{tg}^{0},t,\epsilon}[||\epsilon-\varepsilon _{b}(B_{tg}^{t},t,\mathcal{G}_{tg},\mathcal{G}_{s},B_{s},l)]$ (3) $$

Inference Process
During the inference phase, the first step consists of transforming the source scene into a scene graph $\mathcal{G}_{s}$ and a corresponding layout $B_{s}$. Subsequently, the Target Graph Generation model predicts the target scene graph $\mathcal{G}_{tg}$, conditioned on the source scene graph $\mathcal{G}_{s}$ and the language command $l$. This is followed by the Target Layout Generation model, which computes the target layout $B_{tg}$, leveraging all available variables as inputs. The final step in constructing the target scene, denoted as $T:=(\mathcal{G}_{tg},B_{tg})$, involves retrieving the object meshes based on the estimated object features and arranging them according to the generated layout. This systematic approach enables the dynamic generation of scenes that are aligned with verbal instructions, ensuring that the resulting scenes accurately represent the specified conditions.

## 4 The EditRoom-DB Dataset

To facilitate comprehensive training and evaluation of our framework, we present the EditRoom-DB dataset, designed to support a wide range of basic 3D scene editing operations. The dataset is generated through an automated data augmentation pipeline that produces editing pairs based on object-level modifications applied to scenes from the 3D-FRONT dataset (Fu et al., ). We utilize scenes from the bedroom, dining room, and living room categories and enhance them with high-quality object models from the 3D-FUTURE dataset (Fu et al., ) to simulate real-world editing workflows.

Our data generation pipeline supports a variety of editing operations, including Add and Remove Objects, Pose and Size Changes, and Object Replacement. Each modification produces a new scene paired with a detailed textual description of the changes made, following a predefined template.
For each scene, objects are selected randomly and modified iteratively through basic editing operations. In the case of Add and Remove Objects, the modified scene serves as the target for the operation, with the original scene acting as the source. For Pose and Size Changes, random adjustments are applied to selected objects, and collision checking ensures that the final scenes are free from object overlap. Similarly, in Object Replacement, new objects from the same category are substituted for existing ones, with collision checks ensuring high-quality outputs. More details about automatic pipelines can be found in Appendix B.

To create diverse and realistic language instructions for the editing tasks, we first employ the multimodal understanding model LLAVA-1.6 (Liu et al., ; ; ), which captions front-view images of the objects in the scenes. Then, these object captions are used to construct the template-based commands. The template-based commands are then transformed into natural language commands using GPT-4o, making the dataset suitable for training and testing language-guided scene editing models. Detailed prompts and additional examples are shown in Figure 6 in the appendix.

The resulting dataset consists of approximately 83,000 training samples and 1,500 test samples (randomly sampled 500 for each room type). Table 1 provides a detailed breakdown of the dataset statistics across the three scene categories. Each sample includes a source scene, an edited target scene, and corresponding language commands. This comprehensive dataset enables the development and evaluation of robust models capable of performing various scene editing tasks. By simulating realistic workflows and providing detailed text commands, EditRoom-DB serves as a valuable resource for advancing language-guided 3D scene editing.

**Table 1: EditRoom-DB dataset statistics. We collect around 83k training data across all room types and 500 test data for each room type.**
|  | Train | Test |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Types | Bedroom | Dining room | Living room | Bedroom | Dining room | Living room |
| Translate | 8.6k | 3.2k | 2.7k | 61 | 58 | 74 |
| Rotate | 4.0k | 1.3k | 1.3k | 38 | 35 | 27 |
| Scale | 12.7k | 4.5k | 3.9k | 146 | 144 | 162 |
| Add | 8.9k | 3.4k | 2.8k | 75 | 79 | 57 |
| Remove | 8.8k | 3.3k | 2.8k | 129 | 142 | 127 |
| Replace | 6.8k | 2.2k | 2.1k | 51 | 42 | 53 |
| Total | 49.8k | 17.9k | 15.6k | 500 | 500 | 500 |

## 5 Experiments

### 5.1 Experimental Setup

Baselines
To the best of the authors’ knowledge, EditRoom is the first work that can automatically execute all editing types with natural language commands. Therefore, we construct two baselines by modifying language-guided 3D scene synthesis methods for comparisons: DiffuScene-E and SceneEditor-N:

- •
DiffuScene-N: DiffuScene-N is modified from the language-guided 3D scene synthesis work, DiffuScene (Tang et al., ), which includes a UNet-based diffusion model to generate scene layout. To enable it with language-guided scene editing ability, we leverage their scene completion pipeline by incorporating the source scene as context for the diffusion process. During the training and testing, the model directly conditions natural commands for target scene layout generation.
- •
SceneEditor-N: To test our generalization ability, we experiment with another setting, where the scene editor directly trains on the natural commands got from the GPT-4o. During the inference time, the model conditions the natural commands and generates the final scenes.

We would like to emphasize that not all language-guided 3D scene synthesis methods can be modified as 3D scene layout editing methods. There are at least two prerequisites: (1) there is a way to easily convert the source scene into the required intermediate representations; (2) the scene representations have the capability to execute all editing manipulations. Some LLM-relevant 3D scene generation models (Vilesov et al., ; Zhou et al., ; Aguina-Kang et al., ; Hu et al., ) can generate 3D scenes based on descriptions, but source scenes are hard to convert into their intermediate representations, like program language (Aguina-Kang et al., ) and blender codes  (Hu et al., ).

Metrics
To evaluate the models’ performance, we utilize four metrics: IOU, S-IOU, LPIPS (Zhang et al., ), and CLIP (Radford et al., ) scores. The IOU scores are calculated by determining the 3D Intersection Over Union (IOU) between each object in the generated and target scenes, selecting pairs with the highest 3D IOU values. The S-IOU represents the semantic-weighted 3D IOU, where semantic similarities between matching objects are calculated using Sentence BERT (S-BERT) (Reimers & Gurevych, ) based on their captions. For visual evaluation, we render both the generated and target scenes from 24 fixed camera views. Visual similarity is assessed using the LPIPS metric for pixel similarity, and semantic similarity is evaluated using the CLIP image encoder (CLIP-ViT-B32).

### 5.2 Results

**Table 2: Performance on single operation with different room types. From the table, we can find EditRoom outperforms baselines among all room types, which indicates that our methods can provide more accurate and coherent editing across room types.**
|  | Bedroom | Dining room | Living room |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Model | IOU ($\uparrow$) | S-IOU ($\uparrow$) | LPIPS ($\downarrow$) | CLIP ($\uparrow$) | IOU ($\uparrow$) | S-IOU ($\uparrow$) | LPIPS ($\downarrow$) | CLIP ($\uparrow$) | IOU ($\uparrow$) | S-IOU ($\uparrow$) | LPIPS ($\downarrow$) | CLIP ($\uparrow$) |
| DiffuScene-N | $0.6213$ | $0.6122$ | $0.1374$ | $0.9550$ | $0.4484$ | $0.4338$ | $0.1984$ | $0.9247$ | $0.4693$ | $0.4507$ | $0.1748$ | $0.9328$ |
| SceneEditor-N | $0.7254$ | $0.7150$ | $0.1071$ | $0.9601$ | $0.5189$ | $0.5033$ | $0.1572$ | $0.9356$ | $0.4797$ | $0.4667$ | $0.1638$ | $0.9385$ |
| EditRoom | 0.7435 | 0.7344 | 0.0967 | 0.9644 | 0.5246 | 0.5095 | 0.1489 | 0.9450 | 0.4801 | 0.4724 | 0.1564 | 0.9463 |

**Table 3: Performance on single operations with different editing types. From the table, we can notice EditRoom provides better editing results across all basic editing types.**
|  | Translate | Rotate |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Model | IOU ($\uparrow$) | S-IOU ($\uparrow$) | LPIPS ($\downarrow$) | CLIP ($\uparrow$) | IOU ($\uparrow$) | S-IOU ($\uparrow$) | LPIPS ($\downarrow$) | CLIP ($\uparrow$) |
| DiffuScene-N | $0.5237$ | $0.5115$ | $0.1691$ | $0.9488$ | $0.5902$ | $0.5770$ | $0.1372$ | $0.9510$ |
| SceneEditor-N | $0.5611$ | $0.5491$ | $0.1488$ | $0.9511$ | $0.6269$ | $0.6146$ | $0.1313$ | $0.9526$ |
| EditRoom | 0.5782 | 0.5673 | 0.1432 | 0.9553 | 0.6277 | 0.6158 | 0.1290 | 0.9538 |
|  | Scale | Replace |  |  |  |  |  |  |
| Model | IOU ($\uparrow$) | S-IOU ($\uparrow$) | LPIPS ($\downarrow$) | CLIP ($\uparrow$) | IOU ($\uparrow$) | S-IOU ($\uparrow$) | LPIPS ($\downarrow$) | CLIP ($\uparrow$) |
| DiffuScene-N | $0.5816$ | $0.5691$ | $0.1248$ | $0.9510$ | $0.5662$ | $0.5435$ | $0.1439$ | $0.9336$ |
| SceneEditor-N | $0.6191$ | $0.6083$ | $0.1150$ | $0.9573$ | $0.6002$ | $0.5746$ | $0.1398$ | $0.9390$ |
| EditRoom | 0.6309 | 0.6216 | 0.1083 | 0.9610 | 0.6114 | 0.5837 | 0.1369 | 0.9427 |
|  | Add | Remove |  |  |  |  |  |  |
| Model | IOU ($\uparrow$) | S-IOU ($\uparrow$) | LPIPS ($\downarrow$) | CLIP ($\uparrow$) | IOU ($\uparrow$) | S-IOU ($\uparrow$) | LPIPS ($\downarrow$) | CLIP ($\uparrow$) |
| DiffuScene-N | $0.5075$ | $0.4946$ | $0.1826$ | $0.9381$ | $0.4291$ | $0.4144$ | $0.2060$ | $0.9325$ |
| SceneEditor-N | $0.5595$ | $0.5472$ | $0.1619$ | $0.9341$ | $0.5545$ | $0.5412$ | $0.1442$ | $0.9496$ |
| EditRoom | 0.5657 | 0.5542 | 0.1571 | 0.9430 | 0.5556 | 0.5453 | 0.1363 | 0.9517 |

Figure: Figure 3: Qualitative results on single-operation commands. The left column is the source scene with single operation commands for each basic editing type. From the examples, we can find that EditRoom can provide more coherent and appropriate editing operations across all editing types.
Refer to caption: x3.png

Figure: Figure 4: Qualitative results on multi-operation commands. The left column is the source scene with multi-operation operation commands. From the figure, we can find the EditRoom can successfully generalize to complex natural language commands with multiple operations without further training on the multi-operation operation data, while baselines fail to execute coherent editing.
Refer to caption: x4.png

Single-operation Commands
To assess model performance on single operations, we test our model and baselines using the EditRoom-DB test set, which contains 500 samples per room type, with language commands generated by GPT-4o. Quantitative results are depicted in Tables 2 and 3, and qualitative outcomes are illustrated in Figure 3. Table 2 indicates that EditRoom consistently outperforms other baselines across all room types, with notably superior performance in bedrooms. According to Table 3, EditRoom also excels across all editing types. Comparisons between EditRoom and SceneEditor-N reveal that template-based instructions can simplify the learning process by more effectively aligning language commands with scene changes. Moreover, the LLM (GPT-4o) demonstrates a successful bridge between natural language and template commands. SceneEditor-N outperforms DiffScene-E across all metrics and editing types, suggesting that our graph-based diffusion method yields more coherent and accurate editing results compared to the UNet-based approach. Thus, EditRoom provides more precise and coherent atomic editing operations from natural language commands than its counterparts.

Analysis across different room types shows that all models perform better as the average number of objects in rooms decreases, highlighting potential for improvements in larger, more complex scenes. Evaluating different editing operations reveals that translating, adding, and removing operations score lower on IOU, demanding stronger spatial reasoning. Meanwhile, replacing and adding operations yield lower CLIP scores, indicating a need for better alignment between object descriptions and their semantic features. This underscores the potential for further enhancement of models’ spatial reasoning and object alignment capabilities.

Multi-operation Commands
To demonstrate the generalization capabilities of EditRoom, we manually designed several test prompts that combine multiple atomic operations, and we assessed each model’s performance qualitatively. Figure 4, shows that EditRoom provides more coherent and appropriate responses than the baseline models. For instance, the command in the first row requests a bed replacement and the addition of a wardrobe. EditRoom successfully interprets the natural language command and translates it into the corresponding atomic operations using an LLM, whereas other baseline models misinterpret the command and perform incorrect operations such as translation. These outcomes highlight the challenges of directly training models on natural language commands for compositional editing tasks. EditRoom, by contrast, effectively executes complex editing operations through strategic LLM planning.

**Table 4: Ablation on different condition types on the bedroom. From the table, we can show that incorporating source information as context with the self-attention (our design) instead of the cross-attention mechanism can significantly improve model performance.**
| Model | IOU ($\uparrow$) | S-IOU ($\uparrow$) | LPIPS ($\downarrow$) | CLIP ($\uparrow$) |
| --- | --- | --- | --- | --- |
| EditRoom (Concat-Text) | 0.5992 | 0.5835 | 0.1325 | 0.9547 |
| EditRoom (Original) | 0.7435 | 0.7344 | 0.0967 | 0.9644 |

Ablation on Condition Types
To validate our model design, we experimented with an alternative conditioning approach, where a graph transformer encodes the source scene into a sequence of vectors that are then concatenated with text features. These combined features are incorporated into the cross-attention layers of our graph diffusion process. We specifically tested this method on the bedroom scene type, with results shown in Table 4. The table indicates a significant decrease in model performance, both in terms of layout accuracy and visual coherence. This outcome suggests that utilizing source scene information as the context for self-attention layers, rather than as conditions for cross-attention, yields better results.

**Table 5: Ablation on different text encoders on the bedroom. Due to the limited size of training data, we find using the larger text encoder with high-dimensional features induces decreasing performance on editing, which indicates further exploration with 3D editing data generation.**
| Model | IOU ($\uparrow$) | S-IOU ($\uparrow$) | LPIPS ($\downarrow$) | CLIP ($\uparrow$) |
| --- | --- | --- | --- | --- |
| EditRoom (OpenCLIP-ViT-bigG-14) | 0.6970 | 0.6788 | 0.1271 | 0.9490 |
| EditRoom (Original) | 0.7435 | 0.7344 | 0.0967 | 0.9644 |

Ablation on Text Encoders
In an exploration of text encoder options, we replaced the CLIP-ViT-B32 text encoder (512 feature dimensions) with a larger pretrained encoder, OpenCLIP-ViT-bigG-14 (1280 feature dimensions), used by OpenShape to align with object semantic features—consistent with the object features in our models. We conducted tests on the bedroom test set, with outcomes detailed in Table 5. The results indicate that the model equipped with the larger text encoder underperforms compared to the one using the original encoder. We attribute this decrease in performance to the limited size of our training dataset. Given that our diffusion models are trained from scratch, they require more data to effectively align with higher-dimensional features ($d=512$ vs $d=1280$). This finding underscores the need for further exploration into constructing larger scene editing datasets.

## 6 Conclusion

In this work, we introduce EditRoom, a language-guided 3D room layout editing method. EditRoom incorporates a graph diffusion-based scene editor that facilitates unified basic editing operations, and it utilizes an LLM for natural language planning. Our experiments demonstrate that EditRoom can effectively execute appropriate edits for both single and complex operations. We believe this work will inspire further research into language-guided 3D scene layout editing.

#### Limitation

Since EditRoom leverages the LLM for the command parameterizer, its performance is contingent upon the LLM’s capability in 3D scene understanding and natural command comprehension. This dependency may lead to the generation of erroneous commands that prompt the scene editor to execute potentially problematic operations, such as collisions. However, because the training data predominantly consist of collision-free samples, there is an inherent trade-off between adhering strictly to the commands and avoiding collisions. If the commands deviate significantly from typical scenarios—such as moving an object 100 meters away—the model might instead perform a similar action that falls within the observed training distributions.

#### Ethics Statement

Our work presents a framework for automated 3D scene editing guided by natural language. The primary focus of this research is to advance technical capabilities in scene manipulation. We do not anticipate any ethical concerns or negative societal impacts arising from this work. All datasets used in our research are synthetic and publicly available, and no personally identifiable information or human-related data were involved.

#### Reproducibility Statement

All language commands are encoded through the pretrained CLIP-ViT-B32 text encoder. Each graph diffusion model includes a five-layer graph transformer model with 512 hidden dimensions and 8 attention heads. Training is conducted using the AdamW optimizer over 300 epochs, with a batch size of 512 and a learning rate of $2\times 10^{-4}$. All models are individually trained and tested on each room type. For EditRoom, template commands are employed for the scene editor during training, whereas other baseline models utilize natural language commands generated by GPT-4o. During testing, all models receive natural language commands as input. We set the node number of denoise graphs to the maximum object number for different room types (bedroom = 12, living room = dining room = 21). During training, we pad the objects with zeros to facilitate the batch process.

## Appendix A LLM as Command Planner

(Referred by Section 3.1) A detailed dialog between user and LLM (GPT-4o) is shown in Figure 5.

## Appendix B EditRoom-DB pipeline details

(Referred by Section 4) A detailed example of using LLM to generate natural language description from template command is shown in Figure 6.

#### Add and Remove Objects

Removing each object in the scene separately could generate the modified scenes as the result after removal compared to the original scene. Conversely, the original scene could be treated as the result after object addition.
The formatted editing description will be ‘Add/Remove [object description]’. In order to consider the location of the addition and potential multiple objects in the scene, we will add the relative location description with the closest unique object in the scene, like ‘location: [relative description] [reference object description]’.

#### Pose and Size Changes

We can similarly repeat the pose change operation for every object in the scene as add/remove. Specifically, we design three operations: translation, rotation, and scaling.
For translation, we create random translations as the mix of distances, sampled from 0.1 meters to 1.5 meters with step 0.1, and directions, sampled along the two axes directions (front/back and left/right). Then, collision checking is done for every translated object until we find a collision-free sample. The translation will be skipped if all the samples fail in collision checking. The formatted editing description will be ‘Move object towards the front/back/left/right direction for [distance] : [object description]’

Similarly, we create random rotation angles as the mix of uniform direction samples, clockwise or counterclockwise, and random values between $15-180$ degrees with the step of 15 degrees, and check collision for each sample. The check stops when we find a collision-free sample or all samples fail the checking. The formatted editing description will be ‘Rotate object [angle] degrees : [object description]’

For scaling, we separate it as shrinking and enlarging. The scaling factor is randomly generated between 0.5-0.8 or 1.2-1.5. The scaling factor uniformly applies to three dimensions. Since shrinking won’t cause a collision with other objects, it can always result in a successfully modified scene. For enlarging, if collision checking fails on all trials, the enlarging is skipped. Otherwise, we save the largest collision-free scaling factor. The formatted editing description will be ‘Shrink/Enlarge object by [scale_factor] times : [object description]’

#### Object Replacement

For the replace operation, we access an object dataset with semantic class labels and 3D meshes. The system will retrieve several objects from the dataset with the same class label as the replaced object, and check their collision with other existing objects in the scene. If none of the objects could be placed without collision, we randomly select one object and shrink its bounding box to be equal or smaller than the replaced object to avoid collision. The formatted description is ‘Replace source with target : [source object description] to [target object description]’.

#### Collision Detection Module

The objects are abstracted as 3D bounding boxes and further decomposed into 2D bounding boxes on a horizontal plane and vertical range, as the objects can only rotate about the vertical axis. Then, the two objects are only in collision if their 2D bounding boxes overlap and their vertical ranges overlap. For 2D bounding box collision detection, we apply the Separating Axis Theorem Huynh () to determine if the boxes intersect.

Figure: Figure 5: Dialog between user and LLM to generate an edit plan to rearrange the bedroom. The room image in ’User Input’ is not given to LLM, just for illustration purpose.
Refer to caption: x5.png

Figure: Figure 6: Dialog between user and LLM to generate natural language description from the template command. The room image in ’User Input’ is not given to LLM, just for illustration purposes.
Refer to caption: x6.png

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1) Is the scene edit one-shot (i.e., take all edit commands at once and generate a scene) or progressive (i.e., edit the 3D scene one command at a time)?
2) Motivation behind using a diffusion model is unclear. If the edits allowed are simply adding,
removing, replacing, scaling, translating and rotating an object (even the relations
between objects are pre-defined and limited), why do you need to train a diffusion
model? You could perform these tasks directly on the source scene graph by
adding/removing/replacing a node and then retrieving an object from the given object
database that fits the text description (see Language-Driven Editing of 3D Scenes from Scene Databases, SIGGRAPH 2018; missing this reference). Using diffusion model makes sense when one
wants to generate an entirely new scene/new object from scratch. In the scene editing
task, we are given with an initial scene and our task is to modify it.
3) How does the diffusion process look like for, say, rotation operation? It would be
helpful to have visualizations of intermediate diffusion steps. Applying rotation to
existing object can be done analytically. Why would we need a diffusion model? I would consider this a weakness of the paper as it precludes some important visualizations pertaining to the proposed approach and something that is relevant to the end results. 
4) Regarding the proposed dataset “EditRoom-DB”, what is the reason behind having only
500 examples as test data? This number is too small compared to the number of
training samples i.e. 83k. The standard practice is to have a certain percentage (say,
10%) of total samples as test data. Looking at Table-1, dataset in its current form is
biased towards “bedroom” category because it has ~50k training and only 500 test
samples, as compared to ~18k train and 500 test samples for “dining room” category.
The ratio of train-test data is high for bedroom scenes compared to the other two categories, and its
effect is seen in the results Table-2 (high performance for bedroom than dining and
living room). It would be a good experiment to run when you keep the train:test ratio
equal for all categories and see if the model gives comparable results.

Overall, the paper proposes a novel 3D indoor scene editing model that uses LLM to encode
scene editing instruction and a graph-diffusion model to perform scene edits. Moreover, it introduces a 
new dataset for training and evaluating 3D indoor scene editing task.
soundness: 3
strengths: 1) Well-written paper
2) The idea of breaking down command into multiple steps is interesting
3) The proposed model can be used for multi-operation editing without explicitly training
on such commands. This is a useful feature for interior designers
4) The newly introduces dataset “EditRoom-DB” is quite useful to the 3D scene research community
summary: Given a source 3D scene and a natural language command describing the scene edits, the paper presents a method to generate the  corresponding 3D edited scene.

The existing gaps that motivate this paper are three folds: (a) Requiring manual intervention to perform 3D scene edits, (b) limited number of edits and a lack of support for multi-operational textual commands, and (c) lack of a language-guided scene editing dataset, making it difficult to train and evaluate scene-editing models.

To this end, the contributions of the paper include proposing a new approach that combines an LLM (to encode edit commands) with a graph diffusion model (to incorporate scene edits from the input language command) to effectively serve the high-level goal of synthesizing a 3D scene based on natural-language edits/commands. The framework is termed “EditRoom”. In addition to the proposed approach, the paper also introduces a new synthetic scene editing dataset named “EditRoom-DB” with 83K pairs of language+edited 3D scene, spanning 16K initial 3D scenes.

To summarize, EditRoom takes as input a source scene and a natural language command describing scene edits, and outputs a corresponding edited scene. The dataset used is “EditRoom-DB”, which is a new dataset of scene editing pairs created from 3D-FRONT  dataset. GPT-4o is the LLM used, and there is a graph-transformer based diffusion model. The learning setup is strongly supervised. The loss functions involved are KL divergence loss and diffusion loss for graph diffusion model and layout diffusion model. In terms of evaluations, the paper looks at the following four quantitative metrics: (1) IoU score: 3D IoU between objects in pre-edit and post-edit scene, and selecting pairs with highest 3D IoU value (Q: how many pairs do you use?) (2) S-IoU score: semantic similarity between captions of matching objects as measured by Sentence BERT (Q: how do you find caption for generated objects?) (3) LPIPS: determines pixel similarity between 24 views of pre-edit and post-edit scenes, and (4) CLIP score: Using CLIP-ViT-B32 to find semantic similarity between 24 views. As for comparisons, the paper compares against two works: DiffuScene-N (modified version of existing 3D scene synthesis model DiffuScene, from CVPR 2024), and  ScenEditor-N. (Spelling mistake: Line 323, the name of the baseline should be DiffuScene-N, instead of DiffuScene-E)
weaknesses: 1) The motivation to use diffusion model is unclear.
2) Out of the two compared methods, only one, DiffuScene-N, is based on an existing work (viz., DiffuScene, from CVPR 2024, Tang et. al.). The other one is a modified version of the proposed approach. This is good, But, why not use a modified version of InstructScene (Haque et. al., ICLR 2024) and also of EchoScene (Zhai et.al. , ECCV 2024) as well? This would be tell a good story on how effective the proposed approach is. This is a weakness, something I believe can be addressed.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 4
questions: All of my questions are listed in the weaknesses section, and I may adjust the rating if they are well addressed.
soundness: 3
strengths: a) The problem studied in the paper is important for real-world applications. b) This paper introduces a synthetic scene editing dataset called EditRoom-DB, contributing to the field. c) Support for multi-operation commands enhances the practicality of the proposed approach.
summary: This paper introduces EditRoom, a two-module scene editing framework based on natural language prompts. First, the command parameterizer converts natural language prompts into sequences of breakdown commands for six editing types using an LLM. Then, the scene editor constructs the target scene from the source one using the proposed conditional graph generation models based on diffusion. Quantitative and qualitative results prove the effectiveness and generalization capability of the proposed approach.
weaknesses: a) Comparison with baselines: The authors mention the difficulty of adapting scene synthesis methods for layout editing, and thus compare with only two baselines. However, several LLM-based methods using in-context learning (e.g. LayoutGPT) could potentially be adapted to support editing by defining the task for the LLM and conditioning on in-context learning samples from EditRoom-DB. Including such a comparison with in-context learning-based methods would enhance the evaluation. b) Missing complexity analysis: In this work, the authors train two graph diffusion models, however the complexity analysis is missing. How much time do the training and inference take?

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: Overall, this paper presents a novel framework for language-based 3D room layout editing. The main weaknesses lie in the motivation and evaluation of the proposed approach. The current evaluation only validate the key designs under the novel framework (first LLM and then a diffusion model), but lacks the explanation on why this framework is essential (especially the diffusion model as a scene editor).
soundness: 2
strengths: The strengths include:

1. It proposes the 3D layout editing pipeline, which uses GPT-4o as the command parameterizer and a graph diffusion model as the scene editor.

2. The graph diffusion is tailored to deal with the layout editing problem and outperforms the adapted DiffuScene approach.

3. It constructs a large dataset composed of the paired scene layouts and the corresponding editing instructions, which facilitates the following language-based 3D layout editing research.
summary: This paper proposes EditRoom for 3D layout editing of an existing room. It first uses GPT-4o to parse the natural language instruction into a sequence of breakdown commands, and then trains a diffusion model to generate the edited layout based on the commands. The framework is reasonable and utilize the state-of-the-art LLM and generative diffusion model. The proposed approach generates good results and outputs the other variants of itself, validating the effectiveness of using GPT-4o to infer the breakdown commands and the design of the graph diffusion model as 3D scene editor.
weaknesses: **Firstly, it lacks the discussion of the motivation for the target problem: why and when a user would prefer a language-based 3D layout editing rather than interactively modify the one or two object placements by himself?** This relates with the design of evaluation experiments. Intuitively, the problem setting should be a complicated editing and a very rough instruction. For example, the user might not explicitly specify all the atom editing, but rather describe the key intent only. Or it requires a series of operations to make enough space to enable the target editing. The single-operation and multi-operation cases shown in the paper seem straightforward. Actually, the second row  in Fig.4 gave a good example, since a user might express “rotate the bed” but desire to have the bed s well as the accompanied nightstand rotated together. But it seems that the proposed approach only edit the objects that are explicitly specified, and produces an unreliable layout result.

**Secondly, it lacks the comparison with heuristic methods.** The proposed approach trains a graph diffusion model as the scene editor. But the paper only compare with DiffuScene (another diffusion model for 3D room layouts) and its own variants, only to validate the key designs such as taking breakdown commands as conditions and the condition types of the network. In fact, training neural networks for 3D layout problem often aims to deal with high-level conditions and efficiently generate plausible layouts. Given that the GPT-4o has already inferred the breakdown commands and the generative process of diffusion model is slow, do we really need such a network to produce the edited layout? What if directly parse the breakdown commands to modify the specified object? What if using a simulated annealing algorithm as a post-processing step to deal with the possible collisions? At least it alleviates the demand for constructing a training dataset.

**Thirdly, it lacks a detailed discussion of the command parameterizer.** Currently, there are different opinions on whether the LLM has satisfactory spatial reasoning ability. Fig.5 is a good example as the goal is to make the two nightstands symmetric with each other. But it seems the LLM response presented in the figure is incomplete. It is important to give a in-depth and detailed analysis on the LLM performance, e.g. when the LLM would fail and how to promote its ability?

There are some minor issues. For example, are the point cloud of each object (line 227) only point locations or rgb color as well? How to “concatenate each element of source scene graphs into noisy target scene graphs” (line 241)? It means to concatenate the attributes of corresponding elements, or merge the node lists?

Meta-review: This paper proposes a method that provides a unified framework for language-guided 3D scene layout editing.The main idea is to leverage a LLM and a graph diffusion-based model to edit 3D scenes represented as semantic graphs and layouts. Other contributions include a new 3D dataset for the editing task. 

The proposed method resolves an important and practical task with an interesting solution. During rebuttal, the reviewers raised issues including motivation, method, datasets, missing comparisons and efficiency performance. After rebuttal, all reviewers agree to accept this paper.

Decision: Accept (Poster)