SceneCraft: Layout-Guided 3D Scene Generation
Authors: Xiuyu Yang
Yunze Man
Jun-Kun Chen
Yu-Xiong Wang
Venue: NeurIPS 2024 poster (2024)
Keywords: 3D Scene Generation; 3D Content Generation
Primary area: diffusion_based_models
TL;DR: We propose a novel layout-guided 3D scene generation pipeline to generate complicated indoor scenes adhering to user specifications.

Abstract: The creation of complex 3D scenes tailored to user specifications has been a tedious and challenging task with traditional 3D modeling tools. Although some pioneering methods have achieved automatic text-to-3D generation, they are generally limited to small-scale scenes with restricted control over the shape and texture. We introduce SceneCraft, a novel method for generating detailed indoor scenes that adhere to textual descriptions and spatial layout preferences provided by users. Central to our method is a rendering-based technique, which converts 3D semantic layouts into multi-view 2D proxy maps. Furthermore, we design a semantic and depth conditioned diffusion model to generate multi-view images, which are used to learn a neural radiance field (NeRF) as the final scene representation. Without the constraints of panorama image generation, we surpass previous methods in supporting complicated indoor space generation beyond a single room, even as complicated as a whole multi-bedroom apartment with irregular shapes and layouts. Through experimental analysis, we demonstrate that our method significantly outperforms existing approaches in complex indoor scene generation with diverse textures, consistent geometry, and realistic visual quality.

SceneCraft: Layout-Guided 3D Scene Generation
Xiuyu Yang"l
Yunze Man
Jun-Kun Chen?
Yu-Xiong Wang?
Shanghai Jiao Tong University
University of Illinois Urbana- Chaapa-Champaign
https://orangesodanue . wec.ub.io/ccennCrre
Abstract
The creation of complex 3D scenes tailored to user specifications has been tedious and challenging task with traditional 3D modeling tools. Although some pioneer- ing methods have achieved automatic text-to-3D generation, they are generally limited to small-scale scenes with restricted control over the shape and texture. We introduce SceneCraft, novel method for generating detailed indoor scenes that adhere to textual descriptions and spatial layout preferences provided by users. Central our method is rendering-basee -batn technique, which converts 3D semantic layouts into multi-view 2D proxy maps. Furthermore, we design a semantic and depth conditioned diffusion model to generate multi-view images, which are used to learn neural radiance field (NeRF) as the final scene representation. Without the constraints of panorama image generation, we surpass previous methods in supporting complicated indoor space generation beyond single room, even complicated as whole multi-bedroom apartment with irregular shapes and lay- outs. Through experimental analysis, we demonstrate that our method significantly outperforms existing approaches in complex indoor scene generation with diverse textures, consistent geometry, and realistic visual quality.
Introduction
The generation of diverse and complex 3D scenes plays critical role in enhancing virtual and augmented reality (VR/AR) experiences, video game development, and the advancement of human- centric embodied AI. However, manually creating these complex 3D scenes is a tedious procedure that requires extensive knowledge and proficiency 3D modeling tools [13, 14]. The recent success of 2D generative models [23, 50, 55] fuels the development of a line of text-to-3D work [31, 46, 63, 66). Although these methods have achieved impressive object generation performance, scaling from object- level to scene-level generation presents significant challenges. It involves managing considerably larger space with complicated semantics while ensuring 3D consistency (in terms of shape, texture, occlusion, etc.) across various camera perspectives.
Recent advances scene-level 3D generation [17, 24, 37, 60, 75] have opened new pathways for creating larger-scale virtual environments. Most work leverages image inpainting [17, 24, 75] or multi-view diffusion methods [37, 60] optimize text-guided 3D scene. While generating locally convincing textured meshes, these methods share two common drawbacks: (1) Focusing local coherence, they often struggle to accurately depict geometrically consistent rooms with plausible layouts and rich semantic details. (2) Conditioned only on textual prompts, these methods fall short in terms offering precise control over the entire scene's composition and arrangement. Although some concurrent research [16, 45, 53] has explored the generation of an indoor environment conditioned on user-defined 3D layouts, it is restricted to creating small-scale compositions involving multiple
"Equal contribution.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

SceneCraft2D Distillation
User-Specified Layout as "Bounding Box Secne"
Figure 1: Our novel method generates complex and detailed indoor scenes from 3D spatial layouts and textual descriptions. Given perrsppeiffid layouts represented as "Bounding Box Scene (BBS)," our method renders batches of 2D layouts and coarse depth maps and then transforms them into high-quality 3D scenes.
objects [12, 45], or lacks the ability to generate multiple rooms with complex layouts, shapes, and free camera viewpoints [16, 53] due to the use of panoramic representation.
In this paper, we introduce SceneCraft, a novel method designed to generate high-quality indoor scenes conditioned on user-specified free-form layouts. A high-level illustration of our work shown in Figure 1. Our method features two key innovative designs:
Layout Control. Central to our approach is the utilization of 3D bounding boxes to guide the layouts of the target space, namely "bounding-box scene (BBS)." which allows users design complex and free-form room arrangements with simple bounding boxes. With this layout format, users can easily define both the spatial arrangement and the placement of objects within room, constructing building the Minecraft game. And SceneCraft leverages this preliminary design to generate a detailed and realistic scene. We surpass previous methods in supporting complicated indoor layouts beyond single room, three-story house with multiple layers and irregular rooms.
User-Friendly Semantic-Awan
complicated whole
High-Quality Complex Scene Generation with 2D Diffusion Model. Our framework excels in creating 3D scenes by leveraging the advanced generation capabilities of pre-trained 2D diffusion model, SceneCraft2D, SceneCraft2D takes the "bounding-bow box images (BBI)" rendered from BBS as a condition through ControlNets [76] generate high-fidelity views of the room that follow the given simple prompt like "This is one view of a [style description] room." By obtaining high-quality multi-view images through SceneCraft2D, we successfully distill high-resolution 31> representation [56] of the generated indoor scene.
Trained with multi-view indoor scene datasets [49, 72], work achieves state-of-the-an 3D indoor scene generation performance, both quantitatively and qualitatively. We present the first effective framework generate complex text- and layout-guided 3D-consistent scenes with free camera trajectories and diverse semantics. In summary, our technical contributions are threefold:
We propose a novel -ouiddd 3D scene generation framework to create complicated indoor scenes adhering to user specifications, being the first to operate on free multi-view trajectories and free from the constraints of using panoramas.
We introduce the "bounding-bor scene" as a user-friendly format to scratch a desired room easy as building homes in the Minecraft game, which provides accurate geometry control.
We design high-quality 2D diffusion model, SceneCraft2D, to generate high-fidelit and high-quality rooms following the rendered "bounding-bow box image" from the "bounding-boo scene," and to support the generation of various styles via text conditioning.
With all these contributions, our SceneCraft achieves high-quality generation of various fine-grained and complicated indoor scenes that have not been supported by previous work.
2 Related Work
Learnable Scene Representation. Traditional scene representations [1, 22, 28, 40, 44, 54, 64] directly model the 3D geometry information of the scene and thus suffer from limited flexibility

---

or low rendering quality. The neural radiance field (NeRF) [41] pioneers neural setwork-based scene representation, providing the ability to reconstruct a complete and precise 3D scene from only multi-view images and corresponding camera parameters. Follow-up variants of NeRF [2, 6, 7, 20, 56, 62, 65, 67, 69, 73, 74] aim either to improve the original framework in different aspects, e-g., rendering quality and training efficiency, or to support additional tasks, e.g., relighting ability and editing ability. Recently 3D Gaussian Splatting [27] outperforms the NeRF-family representation with high rendering quality and efficiency. In our work, we use a learnable scene representation as the backbone to model the output. As any representation can be used in our framework, we choose Nerfacto [56] for its high-quality rendering of complicated large-scate scage scenes.
Diffusion-Guider Text-to-3D Generation. The recent successful 2D diffusion-based generative models [3. 4, 23, 50, 51, 55] have inspired a series of innovative text-to-3D methods [10, 31, 43, 46, 59, 66, 75] to distill powerful 2D pre-trained models for 3D content creation. DrcamFusion [46] proposes the score distillation sampling (SDS) module to optimize scene representations, e.g., NeRF [41] Gaussian Splatting [27]. of objects by denoising their rendered views. Building on top of DreamFusion, SJC [63], Magic3D [31], and PreaificDreamer [66] alleviate the over- saturation problem and improve the generation quality. Despite impressive results, these methods restricted in generating small-scale objects without complex semantic composition. More recently, Text2Room [24], SceneScape [17], and Text2NeRF [75] propose extend object generation scene generation with off-the-shelf text-image inpainting models [39, 50], where they iteratively inpaint unseen parts of the scene from novel camera perspectives. Another work [37] proposes progressively distilling a text-conditioned indoor panorama generation model [60] using different groups of camera views to optimize a scene representation. However, all of these methods lack semantic control over the generation output other than a simple text prompt. Hence, they cannot be used in the creation of 3D scene models where users want to specify the structure and layouts of the environment. In comparison, our work learns to generate 3D scenes that adhere to user-specified room layouts and textual descriptions, allowing precise control over the environment.
Scene Generation with Semantic Guidance. A recent line of work has studied 2D generation with semantic guidance for better controllability [9, 15, 18, 30, 50, 71]. Attempting to extend image generation to 3D creation, prior work has studied single image to 3D object reconstruction [33, 34, 36, 38, 47, 57, 68] and single image to video reconstruction [5, 61]. These methods face great challenges in 3D consistency, due to the lack of large scene-level datasets for training and the use of the auto-regressive generation paradigm. Meanwhile, Set-the-scene [12], CompoNeRR [32], and Compo3D [45] learn to generate object compositions from semantic layouts with the SDS method [46]. Discoscene [70] aims to disentangle the scene and then perform object-level scene editing leveraging the layout priors. DiffuScene [58] and GraphDreamer [19] utilize scene graphs together with textual descriptions conditions to generate compositional 3D scenes. However, these methods are restricted to generating small-scale scenes composed of only several objects. They also neglect representations of walls, doors, ceilings, and ground, which are essential in defining indoor scenes but difficult to control in generation. Close to our work are three concurrent methods, CommrolRoom3DD [53], Ctrl-Room [16), and UrhanArchitect [35]. The first two methods generate 3D room meshes from user-defined or estimated layouts with multi-view diffusion followed by a monocular depth estimation process. While achieving outstanding generation performance, they rely on panorama images [60] as their preliminary results, which not only simplifies the scene generation problem, but also limits the complexity of their room layouts and diversity of their camera viewpoints. Our method, by learning a 3D-consistent multi-view generator and a scene renderer without viewpoint constraints, is able to generate more complex and consistent scenes with diverse camera trajectories. UrbanArchited [35] focuses on street-view scene generation with semantic-aware layout controls. However, it allows for greater geometric approximation due to simpler conditions: fewer object categories, sparser and non-overlapping object placement, and more predictable camera trajectories. In contrast, indoor scenes feature dense objects that overlap with more fine- ane-grained categories, which are all effectively addressed in our method.
3 CreneCraft: Methodology
Our SceneCraft is a novel method for text- and layout-guided scene generation. As illustrated in Figure 2, the input to SceneCraft consists of (1) prompt as a coarse description of the target scene's
3

---

Sec. 3.1 Layout Format: Bounding Box Sease (BBS)
Scene Represontation
Dataset
Image
Iungee During Trmnning
Readet Semantic Stap
Render Depth Mup
Pre-trained SeeneCraft2D
"This of room"
SceneCrafi2D
Refined Images
Denoised Imaage
Supervise
(a) Sec. 3.2 SceneCraft2D: Diffusion Pre-training
(b) Sec. 3.3 SceneCraft: Distillation
provide the layout as a bounding-bos scene (BBS, See. 3.1), user-friendly layout format that guides the generation. Our framework contains two stages: (a) pre-training of 2D diffusion model, SceneCrafi2D, to solve the 2D version of the layout-gu guiddd scene generation task (Sec. 3.2), and (b) distillation of the SceneCraft2D to learn scene representation of the generated seene (Sec. 3.3).
Figure 2: SceneCraft novel framework for layout-guided scene generation. which allows
style and content, (2) a "bounding-bow box scene" (BBS) serving as the layout guidance of the target scene, and (3) camera trajectory defined in the space BBS. SceneCraft renders the BBS in the camera trajectory to construct "bounding-box images" (BBI) the layout condition for a pre- trained 2D diffusion model "SceneCraft2D" generate high-quality 2D images of the scene. With the high-quality images generated by SceneCraft2D, SceneCraft is able use SDS-equivalent paradigm [46] to aggregate them into scene representation (e.g., NeRF [41] or 3D Gaussian splatting [27]) of the generated 3D scene. Notably, our SceneCraft dues not require punoramic view. Instead, our camera view can move freely in the 3D space, enabling the generation of much more complicated indoor layouts consisting of multiple rooms, unlike prior work which only supports single-room scenes.
3.1 Bounding-Bos Scene (BBS): A User-Friendly Layout Interface
To provide a user-friendly format for free-form indoor layouts, we design the bounding -ox scene (BBS) representation. As shown in Figure 2, BBS is similar to the "Proxy Room" of Control- Room3D [53], but each object in the scene can represented by a union of several intersecting bounding boxes in BBS with category label, to indicate the coarse shape and category of an object. This provides users with the ability to indicate the shape of the object, egg., an L-shaped or even an S S-shaped desk, while still maintaining the freedom of using single bounding box for generation.
3.2 SceneCraft2D: Layout-Guided Image Generation
BBS can be regarded as draft or a coarse version of the scene. In order to generate the actual room accurately conditioned on BBS, scene corresponds to a 2D generation task, conditioned on the "bounding- -bo image (BBI)" of the same view in BBS, where each pixel of BBI contains both the semantic category and the depth the pixel in BBS. By rendering BBS into BBI on the projected camera trajectory provided, we decompose the layout-guided 3D scene generation task into set of layout- guided 2D image generation tasks, with BBI as conditions. To solve these tasks, we propose SceneCraft2D, 2D diffusion model for high-quality layout-guided 2D image generation.
distillation-guidee framework. Each view of the generated
Augmented SD for BBI Conditions. Our SceneCraft2D is augmented from Stable Diffusion [50], with additional BBI condition at the current viewpoint, which contains both the semantic category

---

map and the BBS depth map. The semantic map (converted to one hot vectors based on the category) and depth map are injected into the model, as conditions via two separate ControlNets [76].
Finetuning. We finetune the augmented Stable Diffusion with scenes in indoor datasets like ScanNet++ [72] and Hypersim [49]. Each scene is converted to a generation task by generating the prompt, converting its semantic point cloud into a BBS, and using the camera trajectory provided by the dataset. We split the generation task into several 2D generation tasks at each view in the dataset, and train the SD model with these tasks. During the finetuning process, instead of using existing caption tools such as BLIP [29] to generate prompts, training samples. During inference-time generation, our model supports more specific and customized scene-specific prompts produce the results that users desire. Note that the base prompt does not need to contain any information describing the image content, it merely serves as a placeholder to avoid the model overfitting any partieular word sentence. Specifically, of room." as the base prompt and user-desired target prompts like "This is one view of a bedroom in Van Gogh painting style." " for generation. The results showed that this method effectively controls the style of the generated outputs via prompts while maintaining good layout-conditioned generation. After finetuning, SceneCraft2D can generate high-quality images according to the given BBI and text prompt.
a single base prompt for all
use "This is one view
3.3 Distillation-Guidee Scene Generation
Distillation Process with Annealing. To generate 3D scenes, we distill the generation ability of our pre-trained SceneScraft2D model a Score Distillation Sampling (SDS) [46, 633-equivalent pipeline. Unlike the vanilla SDS [46] that works in the latent space and directly works with gradients, our pipeline applies an IN2N [2t]-st which is proven SDS-equivalent by HiFA [78]. In this pipeline, maintain multi-view dataset for continual scene representation training while simultancously and iteratively replacing the multi-view dataset with newly generated images by SceneCrafi2D. Through this process, the multi-view dataset will be gradually replaced with views of the generated scenes, which are used to fit the scene representation towards the generation.
Within this pipeline, we also propose an annealing- based distillation strategy inspired by [39, 78], for a more efficient and high-quality distillation. Leveraging the SDEdit method [39] to control the similarity of generated images with the currently modeled scene, we gradually decrease this similarity along with the entire distillation procedure. In other words, at an early stage of distillation, SceneCraft2D can freely generate the room to satisfy the BBS and the prompt; while at later stage, by generating similar but higher-quality images, SceneCraft2I can also serve as a refiner of the scene representation refine the rendering result and improve the scene representation. With this pipeline, our SceneCraft is able to generate high-quality scenes.
Layout-Aware Depth Constraint. When generating a complex indoor scene based on free camera trajectories, learning a reasonable geometry of the scene from scratch is both crucial and challenging. However, we have prior knowledge of the BBS input, which allows the model to quickly capture the geometry of the scene through the layout-aware depth constraint. Specifically, at the initial stage of distillation, we add a normalized depth loss Edepth. where the secud--uupervision signal comes from our BBS input. We set a soft threshold 8 that allows the pixel depths Drender modeled by the scene representation to fluctuate within a reasonable range around the pseudo-groum truth depths D Duayaut- This ensures that the model quickly converges to initial coarse geometry. This loss is modeled in the following form:
Edepth [max(] (Deadderreuerrr Diayout| - 8, 0))².
(1)
Later in the distillation process, disable this loss term to allow the model to learn more fine-grained geometry.
Floc Removal with Periodical Migration. The images generated at the initial steps of the distillation process have a lower consistency, which can result in blurry floes close to the surface and in the air when "averaging" inconsistent multi-view images on the scene representation side. At a later stage, even when the diffusion's 's output is relatively 3D-consistent with annealing, the floes, with condensed volume density, are still hard to remove and may result in Janus problems. Therefore, instead of "fixing" flocs issues in the original scene representation, we propose a method migrate the current relatively coarse scene to another scene from scratch, to obtain finer version. After the
5

---

floer
bed
pileer
flocmnst
shrives
blindi
BRS-A
BAS-R
dresse
BXR Saunatie
BIIT Scnonic
BBS--
Scund mn
BBI Senanii
BBS-C
SemoCot
Figure 3: Generation results of SceneCraft on Hypersim [49] provided room layouts. For cach sample, we demonstrate the 3D BBS and BBI semantic maps and the generated scene RGB images and rendered depth map. Our method is able to generate complex and free-form scenes from challenging room layouts.
first several iterations as early-stage training, we begin to maintain two scene representations, S, and Sy, to indicate the previous coarse representation and the mitigated fine representation, respectively. We freeze S. and generate new images to supervise Sy by generating images similar to S. rendering results (by only applying T noise adding steps), to refine S. and store into Sy with the diffusion model's generation. We also periodically update Sy with S. (with a smaller interval of training iterations) synchronize the latest information in both scene representations. With the periodical migration method, we achieve more and more fine-grained and clear scenes during the training procedure.
The generation of high-quality images by our SceneCraft2D ensures that the scene representation can converge accurately to the intended scene geometry. This advancement negates the necessity for explicit mesh exportation from scene representation as commonly required in previous work. To assign the modeled scene with sharp and clear textures, incorporate the use
Texture Consolidation.

---

SC Ai------aa sccss
TeatBooc
SectoClaatt
Figure 4: Qualitative comparisons of SceneCraft and baseline approaches. We show our generated color and depth renderings under two common layout conditions (a bedroom and living room) alongside three other baselines. SceneCraft demonstrates higher credibility in following the layout conditions and is capable of handling more complex scenarios.
of VGG [25] perceptual and stylization loss during the distillation process. This strategy allows the scene representation produce rendered images that share semantic meaning and stylistic elements with SceneCraff2D-ggeenaae
images, rather than striving for pixel-perfect replication, which often leads blurred results. By employing this loss, SceneCraft framework emerges unified model to generate scenes in sharp and clear manner, thereby climinating the need for labor-intensive processes of mesh exportation and optimization.
Experiments
In this section, we focus on demonstrating the quality of SceneCraft generation under various layout conditions and prompts, and compare our performance with publicly available methods quantitatively and qualitatively. Then we present more challenging generation that is beyond the scope of the previous methods.
Implementation and Datasets. For the development of our SceneCraft2D diffusion model, finetune Stable Diffusion [50] with our produced layout data. We use multi-view images from ScanNet++ [72] and HyperSim [49] to construct BBI data. In the distillation process, we choose Nerfacto from NeRFStudio [56] as our backbone for scene representation. During distillation, we use a dual-GPU pipeline parallelize diffusion generation and NeRF training. More details are provided in Appendix Sec. A.
BBS Sources. For efficiency and effectiveness, we employ two distinct approaches to leverage bounding-box scenes (BBS), one of which utilizes original 3D bounding boxes (axis-aligned or oriented) by directly rendering them into 2D images. This straightforward method is already sufficient for the generation in our experiment, as we applied on Hypersim [49] data. Another approach enhances traditional bounding boxes by voxelizing them into more detailed collection of smaller,
7

---

fine- fine-grained voxels. This method is particularly adept at capturing the nuances of more complex geometries and arrangements within a scene, such as L-shaped tables or S-shaped desks, which often pose challenges for more simplistic modeling techniques. We find that this significantly improves the model's s ability to accurately represent and understand the spatial dynamics and intricate designs of various objects within scene. We use this strategy for realistic and challenging scenes [72].
4.1 Layout-Guided Scene Generation
Baselines. Most existing work does not generate scenes conditioned on user-specified layouts [17, 24, 37]. The only two concurrent scene generation methods that support layout guidance have not released their codebases [16, 53] for comparison. Hence, we make our best effort to create fair comparison with open-sourced scene generation methods and demonstrate the effectiveness of our method through ablation study (Sec. 4.2). Text2Room [24] uses text-conditioned inpainting model to construct the scene frame by frame. Following their original instructions, we change the text prompt along the trajectory reflect which objects visible in the current frame. Similarly, for MVDiffusion [60], we construct different prompts for each of the eight views that make up the panorama image. For Set-the-scene [12], follow their official guidelines, using 3D modeling software (e.g., Blender) to create the same layout input for training and set the same prompts as SceneCraft.
demonstrate qualitative generation results of SceneCraft Hypersim [49] provided room layouts. These illustrations vividly demonstrate the model's proficiency in crafting detailed, complex, and free-form scenes, showcasing its application across both the realistic and synthetic datasets. Not only does it highlight the technical prowess of SceneCraft in navigating the intricacies of scene generation, but also its adaptability the diverse requirements of real-world and artificially constructed environments. In Sec. 4.3, we demonstrate more challenging generation, which includes extremely challenging cases for panorama-based methods, but naturally supported by our framework.
Qualitative Results. In Figure
Quantitative Results. We present quantitative comparisons with base- line methods [12, 24, 60] using both 2D and 3D metrics in Tab. 1. For 2D metrics, we compute the CLIP Score (CS) [48] and the Inception Score (IS) which do not require ground truth scenes from the dataset, and therefore agnostic to the dataset used in training. We also measure 3D quality by conducting user study with 32 participants, who scored 3D) consisteney (3DC) and overall visual quality (VQ) of rooms generated by different methods on scale of 1 to Our experimental design follows previous work [51]. The quantitative results highlight that our method consistently outperforms prior approaches in terms of the CLIP Score. 3D consistency, and visual quality. Regarding the Inception Score, we anticipate that our diffusion model's finetuning with fixed categories slightly limits generation diversity. However, this is not major concern for our task, as previous work has struggled to achieve both high consis- tency and visual quality while being controlled by layout prompts. Additionally, we did not provide other common metrics on generative tasks, e.g., Fréchet Inception Distance (FID) Score, since dependent on the ground truth dataset and would result in unfair and inaccurate comparison if applied to our experiments.
Table 1: Quantitative comparisons of SceneCraft against baselines.
2D Metrics
3D Quality
Method
Cst
IS
3DC1
VQf
Text2Room [24]
22.98
4.20
3.11
3.06
MVDiffusion [60]
23.85
4.36
3.20
3.35
Set-the-scene [12]
21.32
2.98
3.53
2.41
SceneCraft (Ours)
24.34
3.54
3.71
3.56
Comparison with Existing Methods. In Figure 4, we present our results compared with three baselines under two common layout conditions. SceneCraft significantly outperforms previous methods. For panorama-bane
methods (MVDiffusion). the biggest limitation lies the inability to model rooms with complex shapes, such as L- or S-shaped structures. When using prompts to describe layout conditions, MVDiffusion fails to generate the desired results accurately. For inpainting-basee based methods (Text2Room), although they support free camera trajectories, their iterative
8

---

G1
00
00
Scene
Scene
18
Scene
Figure 5: Generation results SceneCraft in complex scenes. We demonstrate SceneCraft's 's ability to generate more complex indoor scenes leveraging arbitrary camera trajectories. Such non-regular shape of rooms cannot be naturally achieved by previous work.
generation nature often results in repetitive contradictory frames. In the example shown in Figure Text2Room generates four heds in that room simply because the prompt contains the word "bedromm," completely failing to adhere to the specified layout conditions. For NeRF-composition methods (Set-the-scene), the main drawback is the inability to generate objects with significant size differences. Set-the-scene trains and combines different objects within the unified NeRF space. In Figure Set-the-scene fails to generate objects hanging on walls, such as blinds or televisions. Our model, however, addresses all these issues: can generate scenes of any scale and complexity following the given layout conditions and can also be adjusted via prompts.
4.2 Ablation Study
We conduct various ablation studies validate our methods. Specifically, we test the effect of the base prompt used in finetuning, the layout-aware depth constraint, and the texture consolidation. The appendix section offers a comprehensive introduction to our evaluated models and additional experimental details, and includes further visualization and ablation experiments. We also elaborate on the limitations, failure cases, broader impacts, and future directions of our work. Please refer to Appendix Sec. B.l for more details.

---

"An ley Winter Children's Bedroom."
"A Vampire's bloody bedroom..
Ours
BLIP2
Figure 6: Effect of Base Prompt. Using our base prompt successfully avoids overfitting and maintains
inherent power of pre-trained Stable Diffusion, while using BLIP2 captions leads to control failure.
(A)
(9)
Style variants on the fixed layouts of SceneCraft. We show three variants A/B/C with different appearances while the geometries remain unchanged.
Figure 7:
Effect of Base Prompt. To verify the effectiveness of using our base prompt, we test different prompt settings: e.g., generating image captions from BLIP2 [29] with user-defined specific prompts. In Figure 6, we show that our method successfully achieves the control of the generation style through prompts, while maintaining good layout-follon foolow-fo ability. Considering the failure of the BLIP2 prompt and the complexity of our layout conditions, we believe that the more complex the condition, the more general the prompt we should take. our case, we use "This one view of a room." " to generally guide the model to fit the entire dataset of the indoor scene, rather than focusing on particular class or object.
4.3 More Generation Results
Generation on Irregular Shape. Figure 5, we showcase the results of more complex scene generation with fully customized layout on the free-camera trajectory. In the first example (Scene A), we customize an indoor layouts input where a bedroom connected to a living room, along with the corresponding arbitrary camera trajectory. Theoretically, any scale, for example, complex indoor room systems composed of multiple interconnected small rooms (Scenes B-D of Figure 5). Such tasks are not well-supported by methods based panorama generation [60] or NeRF composition [12]. Although some other work [24] supports arbitrary camera geometry and controlling the scene
generate indoor scenes of
trajectories, performs poorly in establishing reasonable content.
Style Variants Generation with Fixed Layouts. In Figure 7, we show three variants of generation with the same room layout and different appearance, simply achieved by using different prompts. The results demonstrate the various control abilities of SceneCraft, allowing us to accurately define the shape and appearance generation.
Conclusion
This work has introduced SceneCraft, an innovative method for generating complex and detailed indoor scenes from textual descriptions and spatial layouts. By leveraging a rendering- based operation, and conott-codd diffusion model, our work effectively converts 3D semantic layouts into multi-view 2D images and learns final scene representation that is not only consistent and realistic but also adheres closely to user specifications. Experimental results show the superiority of our model over existing state-of-the-ar methods, highlighting its ability to generate diverse textures and maintain geometric consistency across complex indoor scenes.
10

---

Acknowledgments
This work was supported in part by NSF Grant 2106825, NIFA Award 2020-67021-32799 the IBM-IIlinois Discovery Accelerator Institute, the Toyota Research Institute, and the Jump ARCHES endowment through the Health Care Engineering Systems Center Illinois and the OSF Founda- tion. This work used computational resources on NCSA Delta through allocations CIS220014 and CIS230012 from the Advanced Cyberinfrastructure Coordination Ecosystem: Services & Support (ACCESS) program, and on TACC Frontera through the National Artificial Intelligence Research Resource (NAIRR) Pilot.
References
[1] Achlioptas, O. Diamanti, Mitliagkas, and L. Guibas. Learning representations and generative models for 3D point clouds. In ICML, 2018. 2
[2] J. T. Barron, B. Mildenhall, M. Tancik, Hedman, R. Martin Brrtin- and P. P. Srinivasan. Mip-NeRF: A multiscale representation for anti-aliasing neural radiance fields. In ICCV, 2021.
3
[3] Betker, G. Goh, L. Jing, Brooks, Wang. Li, L. Ouyang. J. Zhuang, J. Lee, Y. Guo, W. Manassra, P. Dhariwal, C. Chu, and Y. Jiao. Emu: Enhancing image generation models using photogenic needles in a haystack. arXiv preprint (xXi:22309.15807 2023. 3
[4] Betker, G. Goh, Jing, T. Brooks, Wang, L. Li, L. Ouyang, J. Zhuang, J. Lee, Y. Guo, WesamManassra, PrafullaDhariwal, CaseyChu, Y. Jiao, and A. Ramesh. Improving image generation with better captions. Computer Science, https://edn. openai. comfpapers/dall-e-3 pdf, 2(3):8, 2023.
[5] E. R. Chan, K. Nagano, M. A. Chan, A. W. Bergman, J.J. Park, A. Levy, M. Aittala, S. De Mello, Karras, and G. Wetzstein. Generative novel view synthesis with 3D-aware diffusion models.
In ICCV, 2023. 3
[6] A. Chen, Z. Xu, F. Zhao, X. Zhang, F. Xiang, J. Yu, and H. Su. MVSNeRF: Fast generalizable radiance field reconstruction from multi-view stereo. In ICCV, 2021. 3
[7] J.-K. Chen, J. Lyu, and Y.-X. Wang. NeuralEditor: Editing neural radiance fields via manipulat-
ing point clouds. CVPR, 2023.
[8] J.-K. Chen, S. R. Buld, N. Müller, L.. Porzi, P. Kontschieder, and Y.-X. Wang. ConsistDreammer: 3D-consistent 2D diffusion for high-fidelity scene editing. In CVPR, 2024. 15
[9] M. Chen, I. Laina, and A. Vedaldi. Training-free layout control with cross-attention guidance.
In WACV, 2024. 3
[10] R. Chen, Y. Chen, N. and K. Jia. Fantasia3D: high-quality text-to-3D content creation. In ICCV, 2023.
Disentangling geometry and appearance for
[11] 7. Chen and H. Zhang. Learning implicit fields for generative shape modeling. In CVPR, 2019.
2
[12] D. Cohen-Bas E. Richardson, G. Metzer, R. Giryes, and D. Cohen-Or Or. sec-the-scene: Global-
local training for generating controllable NeRF scenes. In ICCV Workshop, 2023. 2, 3, 8,
10
[13] B. O. Community. Blender a 3D modelling and rendering package. Blender Foundation, Stichting Blender Foundation, Amsterdam, 2018. 8. URL htttp://wwy/ blender. org.
[14] Epic Games. Unreal engine. URL https: //www/ unreal wnw.allnnniii.e......ee com.
[15] Esser, R. Rombach, and B. Ommer. Taming transformers for high-resolution image synthesis.
In CVPR, 2021.
[16] C. Fang, X. Hu, K. Luo, and P. Tan. Ctrl-Room: Controllable text-to-3D room meshes generation with layout constraints. arXiv preprint arXiv:2310.0360,, 2023. 1, 2, 3, 17
[17] R. Fridman, A. Abecasis, Y. Kasten, and T. Dekel. SceneScape: Text-driven consistent scene
generation. In NewrlPS, 2023. 3, 8 8
[18] O. Gafni, A. Polyak, O. Ashual, S. Sheynin, Parikh, and Y. Taigman. Make-a-scene: Scene-based text-to-image generation with human priors. In ECCV, 2022.

---

[19] G. Gao. W. Liu, A. Chen, A. Geiger, and B. Schölkopf. GraphDreamer: Compositional 3D scene synthesis from scene graphs. In CVPR, 2024.
[20] Y.-C. Guo, D. Kang, L. Bao, Y. He, and S.-H. Zhang. NeRFReN: Neural radiance fields with reflections. In CVPR, 2022. 3
[21] A. Haque, M. Tancik, A. Efros, A. Holynski, and A. Kanazawa. Instruct-NeRF2NeRF: Editing
3D scenes with instructions. In ICCV, 2023,
[22] P. Hedman, T. Ritschel, G. Drettakis, and G. Brostow. Scalable inside-out image-based rendering.
ACM Transactions Graphics, 35(6):231:1 (11::::33:1-221:111 1:1111111:1, 2016. 2
[23] J. Ho, A. Jain, and P. Abbeel. Denoising diffusion probabilistic models. In NeurlPS, 2020. 1, 3 [24] L. Höllein, A. Cao, A. Owens, J. Johnson, and M. NieBner. Text2room: Extracting textured 3D meshes from 2D text-to-image models. In ICCV, 2023. 1, 10
[25] J. Johnson, A. Alahi, and L. Fei-Fei. Perceptual losses for real-time style transfer and super- resolution. ECCV, 2016. 7
[26] L. Kay and J. T. Kajiya. Ray tracing complex scenes. SIGGRAPH Computer Graphics, 20 (4):269-278, Aug. 1986. 15
[27] B. Kerbl, G. Kopanas, T. Leimküüllr, and G. Drettakis. 3D Gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics, 42(4), July 2023. 3,
[28] M. Levoy and P. Hanrahan. Light field rendering. In SIGGRAPH, 1996. 2
[29] J. Li, D. Li, C. Xiong, and S. Hoi. BLIP: Bootstrapping language-image pre-training for unified
vision-language understanding and generation. In ICML, 2022. 5, 10
[30] Y. Li, H. Liu, Q. Wu, F. Mu, J. Yang. J. Gao, C. Li, and Y. Lee. GLIGEN: Open-set grounded
text-to-image generation. In CVPR, 2023. 3
[31] C.-H. Lin, J. Gao, L. Tang, T. Takikawa, X. Zeng, X. Huang, K. Kreis, S. Fidler, M.-Y. Liu, and
T.-.. Lin. Magic3D: High-resolution text-to-3D content creation. In CVPR, 2023. 1,
[32] Y. Lin, H. Bai, S. Li, H. Lu, X. Lin, H. Xiong, and L. Wang. CompoNeRF: Text-guided multi- object compositional NeRF with editable 3D scene layout. arXiv preprint arXiv:2303. 13843, 2023. 3
[33] M. Liu, C. Xu, H. Jin, L. Chen, M. V. T, Xu, and H. Su. One-2-3-45: Any single image to 3D mesh in 45 seconds without per-shape optimization. NeurlPS, 2023.
[34] R. Liu, R. Wu, B. Van Hoorick, P. Tokmakov, S. Zakharov, and C. Vondrick. Zero-1-to-3: Zero-shot one image to 3D object. In ICCV, 2023.
[35] Lu, K.-Y. Lin, Y. Xu, H. Li, G. Chen, and C. Jiang. Urban architect: Steerable 3D urban scene generation with layout prior. arXiv preprint urXiv:2404.06780 2024. 3, 15
[36] Y. Man, Y. Sheng, Zhang, L.-Y. Gui, and Y.-X. Wang. Floating No More: Object-ground reconstruction from single image. arXiv preprint arXiv:2407. 18914, 2024.
[37] W. Mao, Y.-.. Cao, J. Liu, Z. Xu, and M. Z. Shou. ShowRoom3D: Text to high-quality 3D room generation using 3D priors. arXiv preprint arXiv:2312.13324, 2023. 3, 8, 15
[38] L. Melas-Kyriazi, C. Rupprecht, I. Laina, and A. Vedaldi. RealFusion: 360 reconstruction of any object from a single image. CVPR, 2023.
[39] C. Meng, Y. He, Song, Song, J. Wu, J.-Y. Zhu, and S. Ermon. SDEdit: Guided image synthesis and editing with stochastic differential equations. In ICLR, 2022. 3,5, 18
[40] L. Mescheder, M. Oechsle, M. Niemeyer, S. Nowozin, and A. Geiger. Occupancy networks: Learning 3D reconstruction in function space. In CVPR, 2019. 2
[41] B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng. NeRF; Representing scenes as neural radiance fields for view synthesis. In ECCV, 2020. 4
[42] I.. Mou, J.-K. Chen, and Y.-X. Wang. Instruct 4DDtt-4DD; Editing 4)) scenes pseudo-31) scenes using 2D diffusion. In CVPR, 2024. 15
[43] N. Müller, Y. Siddiqui, I.. Porzi, S. R. Bulo, P. Kontschieder, and M. Nießner. DiftRF; Rendering- guided 3D radiance field diffusion. In CVPR, 2023. 3
[44] M. Niemeyer, Mescheder, M. Oechsle, and A. Geiger. Differentiable volumetric rendering:
Learning implicit 3D representations without 3D supervision. In CVPR, 2020. 2
12

---

[45] R. Po and G. Wetzstein. Compositional 3D scene generation using locally conditioned diffusion.
In 3DV, 2024. 1,2.3
[46] B. Poole, A. Jain, J. T. Barron, and B. Mildenhall, DreamFusion: Text-to-3D using 2D diffusion.
In ICLR, 2022. 4,5
[47] G. Qian, J. Mai, A. Hamdi, J. Ren, A. Siarohin, B. Li, H.-Y. Lee, I. Skorokhodov, P. Wonka, S. Tuulykkv, and G. Bernard. Magic123: One image to high-quality 3D object generation using both 2D and 3D diffusion priors. In ICLR, 2024. 3
[48] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, Mishkin, J. Clark, G. Krueger, and I. Sutskever. Learning transferable visual models from natural language supervision. In ICML, 2021. 8
[49] M. Roberts, J. Ramapuram, A. Ranjan, A. Kumar, M. A. Bautista, N. Paczan, R. Webb, and J. M. Susskind. HyperSim: A photorealistic synthetic dataset for holistic indoor scene understanding. In ICCV, 2021. 2, 5, 6, 7, 8, 15, 18
[50] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer. High-resolution image synthesis with latent diffusion models. In CVPR, 2022. 1, 4, 7,18 18
[51] C. Saharia, W. Chan, S. Saxena, L. Li, J. Whang, E. L. Denton, K. Ghasemipour, R. Gon- tijo Lopes, B. Karagol Ayan, T. Salimans, H. Jonathan, J. F. David, and N. Mohammad. Photorealistic text-to-image diffusion models with deep language understanding. In NeurIPS, 2022. 3.8
[52] T. Salimans, I. Goodfellow, W. Zaremba. V. Cheung, A. Radford, and X. Chen. Improved techniques for training GANs. In NeurIPS, 2016. 8
[53] J. Schult, S. Tsai, L. Höllein, B. Wu, J. Wang, C.-Y. Ma, K. Li, X. Wang, F. Wimbauer, Z. He, Z. Peizhao, L. Bastian, V. Peter, and H. Ji. Controlroom3D: Room generation using semantic proxy rooms. In CVPR, 2024. 1, 2, 3, 4, 8, 15
[54] N. Snavely, S. M. Seitz, and R. Szeliski. Photo tourism: Exploring photo collections in 3D. ACM Transactions on Graphics, 25(3):835-844 2006.
[55] Sohl-Dickstein, E. Weiss, N. Maheswaranathan, and S. Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In ICML, 2015, 1,3
[56] M. Tancik, E. Weber, E. Ng. R. Li, B. Yi, Kerr, T. Wang, A. Kristoffersen. J. Austin, K. Salahi, A. Ahuja, D. McAllister, and A. Kanazawa. NeRFStudio: A modular framework for neural radiance field development. In SIGGRAPH, 2023. 3,
[57] J. Tang, T. Wang, B. Zhang, T. Zhang, R. Yi, L. Ma, and D. Chen. Make-it-3D: High-fidelity 3D creation from a single image with diffusion prior. In ICCV, 2023.
[58] J. Tang. Y. Nic, L. Markhasin, A. Dai, J. Thies, and M. NicBner. DiffuScene: Scene graph denoising diffusion probabilistic model for generative indoor scene synthesis. In CVPR, 2024.
[59] J. Tang, J. Ren, H. Zhou, Z. Liu, and G. Zeng. DreamGaussian: Generative Gaussian splatting
for efficient 3D content creation. In ICLR, 2024. 3
[60] S. Tang, F. Zhang, J. Chen, P. Wang, and Y. Furukawa. MVDiffusion: Enabling holistic multi-view image generation with correspondence-wware diffusion. In NeurlPS, 2023. 1, 3,
10
[61] H.-Y. Tseng, Q. Li, C. Kim, S. Alsisan, J.-B. Huang, and J. Kopf. Consistent view synthesis with pose-guided diffusion models. In CVPR, 2023. 3
[62] D. Verbin, Hedman, B. Mildenhall, T. Zickler, T. Barron, and P. P. Srinivasan. Ref-NeRF:
Structured view-dependen appearance for neural radiance fields. In CVPR, 2022. 3
[63] H. Wang, X. Du, J. Li, R. A. Yeh, and G. Shakhnarovich. Score Jacobian Chaining: Lifting pretrained 2D diffusion models for 3D generation. In CVPR, 2023. 3,5
[64] N. Wang. Y. Zhang, Z. Li, Y. Fu, W. Liu, and Y.-G. Jiang. Pixel2Mesh: Generating 3D mesh
models from single RGB images. In ECCV, 2018. 2
[65] Q. Wang, Z. Wang. K. Genova, P. Srinivasan, H. Zhou, J. T. Barron, R. Martin-Brualla,
N. Snavely, and T. Funkhouser. IBRNet: Learning multi-view image-based rendering. In CVPR,
2021. 3
13

---

[66] Z. Wang, C. Lu, Y. Wang, F. Bao, Li, Su, and J. Zhu. ProlificDreamer. Sigh-fidelity and diverse text-to-3D generation with variational score distillation. NeurIPS, 2023. 1,3
[67] L.. Wu, J. Lee, A. Bhattad, Y.-.. Wang, and D. Forsyth. DIVeR: Real-time and accurate neural radiance fields with deterministic integration for volume rendering. In CVPR, 2022. 3
[68] D. Xu, Y. Jiang, P. Wang, Z. Fan, Y. Wang, and Z. Wang. NeuralLift-360; Lifting an in-the-will
2D photo 3D object with 360 views. In CVPR, 2023. 3
[69] Q. Xu, Z. Xu, J. Philip. S. Bi, Z. Shu, K. Sunkavalli, and U. Neumann. Point-NeRF: Point-based
neural radiance fields. In CVPR, 2021.
[70] Y. Xu, M. Chai, Z. Shi, S. Peng. I. Skorokhodov, A. Siarohin, C. Yang, Y. Shen, H.-Y. Lee,
B. Zhou, and S. Tulyakov. DisCoScene: Spatially disentangled generative radiance fields for controllable 3D-aware scene synthesis. In CVPR, 2023. 3
[71] Z. Yang. J. Wang, Z. Gan, L. Li, K. Lin, C. Wu, N. Duan. Z. Liu, C. Liu, M. Zeng. and L. Wang. RECO: Reegion-controlln text-to-image generation. In CVPR, 2023. 3
[72] C. Yeshwanth, Y.-C. Liu, M. Nießner, and A, Dai. ScanNet++: A high-fidelity dataset of indoor scenes. In ICCV, 2023. 2, 5, 7, 8, 15, 16, 18
[73] A. Yu, R. Li, M. Tancik, H. Li, R. Ng. and A. Kanazawa. PlenOctrees for real-time rendering of neural radiance fields. In ICCV, 2021.
[74] A. Yu, V. Ye, M. Tancik, and A. Kanazawa. PixeINeRF: Neural radiance fields from or few images. In CVPR, 2021. 3
[75] Zhang, X. Li, Z. Wan, C. Wang, and Liao. Text2NeRF: Text-driven 3D scene generation with neural radiance fields. IEEE Transactions Visualization and Computer Graphics, 2024. 1.3
[76] L. Zhang, A. Rao, and M. Agrawala. Adding conditional control to text-to-image diffusion models. In ICCV, 2023. 5,
[77] Q Zhang, C. Wang. A. Siarohin, P. Zhuang, Y. Xu, C. Yang, D. Lin, B. Zhou, Tulyakov, and H.-Y. Lee. SceneWiz3ID: Towards text-guided 3D composition. arXiv preprini
2023. 17
[78] Zhu and P. Zhuang. HiFA; High-fidelity Text-to-3D with Advanced Diffusion Guidance. In ICLR, 2024. 5
14

---

SceneCraft: Layout-Guided 3D Scene Generation
Appendix
online tminer
offline render
Nerfacto
hold
Diffusion Pipeline
standard noise
added noise
Figure A: An illustration of dual-GPU training scheduling.
A
Additional Details
Data Processing. As mentioned in Sec. 4, we use two datasets: HyperSim [49] and ScanNet++ [72], which are synthetic data and real-world data, respectively. We mainly use the HyperSim dataset in our experiments. For HyperSim, the original data provide approximately 77400 images of 461 indoor scenes, with the corresponding camera parameters and semantic bounding boxes. However, some of them do not meet the quality requirements of our training. Following prior work [53], we examine the entire dataset and filter out the lower-quality portions, such rooms containing extremely complex and irregularly shaped objects, unbounded outdoor space, and rooms with excessively large scales (e.g., churches, restaurants), ultimately retaining approximately half of the original data, amounting to around 24k pairs. ScanNet++ shows more complicated scenes (450+ scenes) compared to HyperSim. We voxelize them with unit size equal to 0.2m make the trade-off between rendering cost and data quality. Our processed data are publicly available 2, We leverage the Ray-OBB model (extended from the Ray-AABB model [26]) complete the rasterization process.
DPU Training Scheduling. Inspired by [8, 42], have implemented dual-GPU scheduler (see Figure A) for efficient diffusion and NeRF generation. Specifically, the second (or any GPU other than the first) GPU continuously generates new images to update the dataset, while the first GPU continuously trains Nerfacto with the current dataset. Once the diffusion procedure needs images from the first GPU refine, the first GPU will switch to an offline renderer. This configuration effectively decouples the diffusion generation process, which is inherently more time-intensive, from the comparatively rapid NeRF training, thus streamlining the overall distillation workflow without compromising quality or efficiency.
Training Details and Cost. For finetuning the diffusion model, 2 NVIDIA A6000 GPUs with a constant learning rate of 5e-5, training for around 10k iterations. For the scene generation task, we use 2 A6000 GPUs to perform all our experiments. Generally, each well-generated scene takes around 3-4 hours with -150 frames, 5-6 hours with -300 frames (which handles camera trajectories of arbitrary length and complexity), the first GPU is responsible for the diffusion model and the memory cost is -6GB in FP16 mode with an image size of 512x768. The other GPU is responsible for the scene representation process and the memory cost is -28GB (with Nerfacto). Note that some concurrent methods cost more time: ShowRoom3D [37] costs approximately 10 hours to produce single scene (Tab.3 in [37]), UrbanArchitect [35] costs -12 hours and 32GB to produce a single scene (Sec. 4.1 in [35]). For NeRF training, we use a constant learning rate of le-2 for proposal networks and 1e-3 for fields.
total batch size of 16 on
B Additional Experiments
In this section, we first discuss additional aspects of ablation studies, which show the superiority of our method. Then we showcase additional generations on more complex layouts via our SceneCraft2D.
I Layout Scannet44: http://hwggiiggeee...vvaaaaaaayy http://dfiffiiiieeeepeeeoeveeose scannetpp_voxel0.2 Layout http:///wggggggeeecooosseseeyyyy fit diffusson _iffusion_hypersim
15

---

Rendered RGB
Rendered Depth
Figure C: Effect of Texture Consolidation. The generated renderings are blurry without the texture consolidation strategy. Our SceneCraft produces more detailed and textured results.
B.1 Ablation Study
Effect of Texture Consolidation. We demonstrate the effectiveness of our texture consoidation shown in Figure C. Without texture consolidation, the loss function only includes the latent image loss and RGB image loss, similar conventional SDS methods. this case, the model fails to capture the high-frequency components of the scene from 2D images, resulting in very blurry generated scenes.
Effect of Layout-Aware Depth Con- straint. We conduct experiments to validate the effectiveness of the layout- aware depth constraint. As depicted in Figure B, without this strategy, the model is completely unable to learn the correct geometry of the scene from 2D guidance. Although a reasonable appearance achieved, it is mainly due to the fully flexible camera trajec- tories and complex layout conditions. Notably, using fixed camera poses as in panorama generation can partially alleviate this issue. With the layout-aware depth constraint, the scene geometry rapidly converges to the ground truth during the very initial training stage. Although some areas may find incorrect positions (red box), they are corrected during subsequent training after intermediately disabling this strategy, eventually leading to final convergence (green box). Once geometry converges well, color and texture begin to emerge.
Renadized RCB Resdoed Doyth
Condition muput
w/o depth constraint. stsp-1930
w/ depti: constratit. 1t0p-1905 to stop-8400
Figure B: Layout-Aware Depth Constraint. With depth constraint strategy. our SceneCraft can effectively learn scene geometry from prior input, which is crucial ultimate 3D consistency.
B.2 Complex Room Generation with Irregular Object Geometry and Free Camera Trajectory
In Figure D, take the complex room layouts from the ScanNet++ [72] dataset as example. The first two rows represent the input layout condition, and the last row shows the SceneCraft2D output. Note that for rooms with irregular object geometries, convert voxelized 3D bounding boxes into more fine-grained voxels. This method particularly captures layouts with irregular geometries, such as L-shaped tables or S-shaped desks. which pose challenges more simplistic modeling techniques. The results show that SceneCraft2D performs well in such complex room layouts.
C Limitations and Potential Future Direction
Although we have achieved promising results in complex scene generation from user prompts, 3D generation remains very challenging task with many unsolved problems, and our method limited in some aspects. This section provides a detailed discussion of the limitations and outlines potential future directions.
Discussion on Failure Cases. Despite our promising performance, some failure cases also exist: (I) Extremely Complicated Scenes. Our method may struggle to reason the layouts and generate rooms when layouts are excessively complex, containing many closely placed objects highly overlapped bounding boxes (see Figure E). In this case, optimized method using voxelization
16

---

Layour Condition Generatioon
Figure D: Performance of SceneCraft2D on complex ScanNet++ provided room layouts.
does not provide clear and accurate representation of object layouts, which also reflects the difference between indoor and outdoor (street-view) scenes; (2) Mismatched Layout and Prompt Inputs. When the prompt does not align with the actual room layouts (e.g., a bedroom layout with a "kitchen" prompt), our method may fail generate appropriate room contents or achieve good convergence (see Figure F). Hence, users may need to adjust the corresponding prompts when generating large complex scene with long-term trajectory. All of these failure cases reflect the limitations of our method.
Quality of Images Requires Further Improvement. The quality of our generated 3D scenes still requires further improvement. When dealing with more challenging objects that typically have irregular geometries, such as hollowed-out chairs, lamps, or blinds, our results tend to be somewhat blurry. Moreover, complex layout condition inputs still limit the control ability of the prompt. For instance, are unable to generate objects vivid and richly detailed those produced by the original diffusion model.
Extension to Generation of Outdoor Scenes. Most existing literature separates the task of generating indoor and outdoor scenes and focuses generation of indoor and outdoor scenarios features different challenges, with indoor scenes having denser and more complex layouts, and outdoor scenes having more dynamic objects and larger space to cover. Although it is valid choice to start from indoor scene generation because of its relatively smaller scale and complex layouts, it is not comprehensive. The generation of outdoor scenes will likely lead to unique challenges and insights, and we consider this direct future direction.
of the scenarios in their work. The
Other Future Directions. There are also challenges in defining fair, accurate, and comprehensive metrics for evaluating 3D scene generation methods in aspects. The development of such metrics remains an open challenge and serves as potential direction for exploration in the generative AI community. Another promising direction is flexible and controllable scene editing using decomposed 3D representations, given that layout input is freely defined and adjustable. For complex scene generation, creating layouts by hand could be time-consuming and laborious, hence integrating some methods of automatic scene layouts and camera trajectories creation will also be future topic, such as LLM (large language model)- based [77] and transformer- banssdd approaches [16]. Extending our framework to incorporate user feedback loops for iterative refinement of generated scenes offers another promising direction.
D Societal Impact
We anticipate a potential positive social impact from our work. Being able to generate high-quality 3D digital content based on user queries, SceneCraft has the potential to revolutionize various industries. from VR/AR to architectural design and gaming. By significantly reducing the time and expertise required to create detailed 3D scenes, our work also leads to more accessible and fair content creation.
Potential Negative Societal Impact. We do not see a direct negative societal impact of our work. Indirect potential negative impact involves misusing scene generation models for digital content creation. We believe that it is crucial for researchers to proactively consider these concerns and establish guidelines to ensure responsible usage of these models.
17

---

generation
Figure E: Failure case #1: Extremely complicated scene that contains many closely placed objects and even highly overlapped bounding boxes.
layout
matched
bed
pillow
unmatehed
picture
bliads
cabinet
dresser
Figure F: Failure case #2: Generations of a bedroom with a matched prompt "Bedroom" and mismatched prompt "Kitchen."
E License of Dataset Used
In this section, we list the licenses of all the datasets we have used during our evaluation.
ScanNet++ [72]: The ScanNet++ data are released under the ScanNet++ Terms of Use
HyperSim [49]: Creative Commons Autibutiin-ShareAAli 3,0 Unported License.
In addition, we utilize a number of public foundation model checkpoints pre-trained on various data sources in our paper. Please refer to their original papers [39, 50] for the license of datasets they have used pre-training their models.
hnttps://kaldir in, http:///www....aaaa...e.eeeeeeea htt.dd///ccmmmppp/tttttttttttttt pdf
18

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility. transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and precede the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justifiiaton right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]", ", it is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS paper checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: We make sure the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
19

---

Justification:
We discuss the limitations of the work in the paper.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approxinations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: We disclose all the details required to reproduce all experimental results in our paper.
Guidelines:
20

---

The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open- source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome deseribe the particular way they provide for reproducibilityy In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
We promise that will open-source the data and code after paper acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips cc/
Seeiiicguuddes/ooeeeuee wwi.bmiisssooooiiiiiooocco for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: //n ctt.pss.cc/pubic/uuuuuuu gesds//////eeeeeee/e /CodeSubmian ss cassoonPoll. onoolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
21

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We introduce all the training and evaluation details necessary to understand the results.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Due to the large amount of experiments required to be run in this paper, we do have enough computational resource and time to generate error bars for all our experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We disclose sufficient information on the computer resources used to train and evaluate all experiments.
Guidelines:
22

---

The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: httpp//pccappppppppppppic/////// httppp/ii//ttiiiattteeeeeeeeeeee ines?
Answer:
[Yes]
Justification:
Our research conform with the NeurlPS Code of Ethics in every respect.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We have discussed both the potential positive and negative societal impacts.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (c.g., disinformation, generating fake profiles, surveillance), fairness considerations deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
23

---

Justification:
We identify our paper as having no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We have given the creators original owners of assets used in the paper proper credits.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ comddatasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: We have not released new assets at the submission time. We will carefully document our data and model when we release the code and data.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/moodl as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects

---

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
25


Reviews:
[{"review_id": "ngN3GrOdww", "reviewer": "Reviewer_DRMD", "summary": "This paper addresses the task of 3D-consistent indoor scene generation using 2D diffusion models. As input the user provides a 3D bounding-box layout of the scene and a text prompt coarsely describing the scene. The proposed method contains a two-stage training phase. In the first stage, a 2D diffusion model is fine-tuned to synthesize a rendered view of a scene based on its rendered semantic layout and a depth map from the same view. In the second stage, the fine-tuned diffusion model is distilled into a NeRF representation with an SDS-based approach. The authors qualitative evaluate their method and compare with existing baselines, showcasing the effectiveness of their method.", "questions": "See the weaknesses", "limitations": "Adequately discussed.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper is well-written and easy to follow.\n- The proposed method is straightforward.\n- The provided results show promising results compared to the baselines.\n- The authors provide ablation study on their proposed components", "weaknesses": "- The proposed method might not be very novel. The contribution of the paper seems mainly to be fine-tuning the diffusion model for semantic-guided image synthesis. The distillation stage is mostly similar to the existing methods.\n\n- The authors only evaluate their method qualitatively. Although the provided results seem very promising in terms of the 3D consistency of the scenes, a quantitate evaluation could provide more certainty about the advantage of the proposed method.\n\n- The authors could have provided video visualizations in the supplementary, as assessing the 3D consistency of the method from multi-view images is challenging", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your comments, and address your concerns as follows:\n\n***\n1. *Q: The proposed method might not be very novel. The contribution of the paper seems mainly to be fine-tuning the diffusion model for semantic-guided image synthesis. The distillation stage is mostly similar to the existing methods.*\n\nWe would like to highlight the novelty and contributions of our work:\n* **Pioneering Free Multi-View Trajectories**: Our primary contribution lies in being the first to operate on free multi-view trajectories during scene generation. This approach contrasts sharply with baseline models [16,48,55], which rely on panoramas as intermediate states. As a result, we can now generate complex non-square rooms (e.g., L-shaped rooms, as demonstrated in supplementary Figure B and rebuttal PDF Figure.1), a capability beyond the reach of prior work.\n* Other than the novel non-panorama-based framework, our DreamScene has the following **technical contributions**, as also summarized in L58~L64:\n    * Bounding-Box Scene (BBS) Layout Condition. We introduce this novel layout condition, which offers greater versatility compared to layout conditions in previous work, inpainting-based approaches, and composition-based methods.\n    * We support text-conditioned generation to generate rooms with different styles, with our fine-tuned diffusion model which produces multiview semantic-rich guidance, which is not different from prior works using panorama guidance.\n    * We propose a novel NeRF distillation procedure that could remove the flocs via periodical migration and consolidate textures through unique loss design, as described in Sec. 3.3 Line 165-184. Additionally, we introduce multi-timestep sampling to enhance both semantic guidance and generation consistency.\n* Unlike existing layout-guided methods, we commit to releasing our code upon acceptance, fostering transparency and reproducibility in the research community.\n\nThese contributions, particularly our non-panorama-based framework, represent significant advancements in the field of semantic-guided image synthesis and scene generation.\n\n***\n2. *Q: Evaluation: authors should provide quantitative results.*\n\nWe appreciate the reviewer's suggestion for quantitative results.\n* Our initial decision to prioritize qualitative evaluation was based on the challenges in defining fair, accurate, and comprehensive metrics for evaluating 3D scene generation methods across all aspects. The development of such metrics remains an open challenge in the generation community. \n* However, to address the reviewer's concern, we include quantitative comparison with baseline methods [11,24,55] on CLIP Score (CS), Inception Score (IS), and a user study on 3D scene consistency and overall visual quality. The experiment design follows previous work [48]. The detailed quantitative evaluation results are provided in **General Response**. \n\n***\n3. *Q: Visualization: authors should provide videos to better show the 3d consistency in addition to multiview images.*\n\nThank the reviewer for the suggestion. Please refer to **General Response** for details about our provided video visualization.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "86Arpv00oZ", "reviewer": "Reviewer_Nerf", "summary": "DreamScene proposes a method for 3D indoor scene generation with text and layout as input. For this, they propose using a 3D bounding box scene representation as the means to provide layout guidance. This is then used as an input, along with text prompt, to a 2D diffusion model, dreamscene2D, which is capable of generating high-fidelity and high-quality views of rooms following the rendered “bounding-box image” (containing 2D semantic maps and depth) from the bounding box representation. Dreamscene2D is fine-tuned from stable diffusion using LoRA adapters. This is then used for training a NeRF using Score Distillation Sampling. Additional measures, such as a depth constraint, floc removal and texture consolidation with backbone models helps generate the desired scene. In comparison with presented baselines, the results show the ability to model complex shapes in rooms, greater consistency across views, as well as the ability to generate objects of varying sizes. Ablation experiments are provided for the effect of the base prompt, the effect of the depth constraint and the effect of the texture consolidation using backbone models.", "questions": "- Runtime: I believe the time taken for 3D asset generation through such text-guided approaches is an essential metric for comparison. However I am unable to find this quantification in this work. \n- Quantitative metrics: I think it is important to quantify performance of the model (using maybe something like a CLIP score or FID), especially since this is mentioned in the introduction of the paper.\n- It would be good to get an understanding of failure cases of the proposed method, or settings where the proposed method is unable to generate a favorable output.\n\nFor now, I would like to give a weak accept score and wait for the authors' responses to the mentioned questions and weaknesses.", "limitations": "The authors have addressed limitations and future directions in the appendix which is appreciated. I have some questions with regards to failure cases that I have addressed in previous sections.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "+ The paper is structured in a manner that is easy to understand\n+ The method is reasonable, and the various components are well motivated in the text and through ablations.\n+ In terms of the qualitative comparison with the shown baseline methods, clear differences can be seen with the proposed method leading to more detailed and accurate scenes\n+ The authors promise to open-source the code post acceptance", "weaknesses": "- Lack of quantitative metrics: The paper claims superiority over baselines both qualitatively and quantitatively (L55). However, I am unable to find any quantitative metrics in the paper.\n- Generation quality: While the proposed method seems to perform better than the chosen baselines, the quality of the generated scenes seems to be relatively lower overall. For example, in the last row of Figure 4, the scene lacks texture and shows a degeneracy in color generation.\n- Lack of failure cases: In such works as scene generation, and especially since the scope of this work is limited to indoor scenes, I believe it is necessary to contextualize the kind of scenes where the proposed method is successful and where it fails.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your comments, and address your concerns as follows:\n\n***\n1. *Q: Evaluation: Authors should provide quantitative results.*\n\nWe appreciate the reviewer's suggestion for quantitative results.\n* Our initial decision to prioritize qualitative evaluation was based on the challenges in defining fair, accurate, and comprehensive metrics for evaluating 3D scene generation methods across all aspects. The development of such metrics remains an open challenge in the generative AI community. \n* However, to address the reviewer's concern, we include quantitative comparison with baseline methods [11,24,55] on CLIP Score (CS), Inception Score (IS), and a user study on 3D scene consistency and overall visual quality. The experiment design follows previous work [48]. The detailed quantitative evaluation results are provided in **General Response**. \n\n***\n2. *Q: Overall quality: While the proposed method seems to perform better than the chosen baselines, the quality of the generated scenes seems to be relatively lower overall (especially Fig.4 second example).*\n\nWe appreciate the reviewer's observation regarding the quality of generated scenes. We'd like to address this concern and highlight the strengths of our approach:\n* **Pioneering Complex Scene Generation**: Our primary contribution lies in achieving 3D-consistent complex scene generation with flexible layouts and free camera trajectories. We are the first to generate non-regular room shapes with complex layouts, as demonstrated by the L-shaped rooms in our supplementary Figure B and the new T-shaped 1-bedroom apartment scene provided in the rebuttal (PDF and video). This capability surpasses baseline models [16,48,55] that rely on panoramas as intermediate states and cannot achieve such non-regular room generation with precise layout control.\n* **Geometric Consistency**: While the second example in Figure 4 may have some limitations in color and texture, it maintains reasonable and consistent geometry that accurately follows the layout condition. This geometric fidelity is a crucial aspect where our method outperforms others (e.g., Text2Room, Set-the-Scene, MVDiffusion) that struggle to achieve comparable geometry quality or adhere to layout constraints.\n* **Improved results**: We have found that the quality of generated scenes can be significantly enhanced by using more stylish and thematic descriptions. In the rebuttal PDF, we've included two variants of the same layout with improved text prompts. Our experiments show that using stylish descriptions like \"ocean-themed\" or \"Van Gogh style\" yields much better results compared to using specific words like colors to define the scene. \n\nThese improvements demonstrate the potential of our method to generate high-quality scenes when provided with appropriate prompts, while maintaining its unique capabilities in complex layout generation and geometric consistency. \n\n***\n3. *Q: Lack of failure cases: successful and failure cases should be provided and discussed.*\n\nWe thank the reviewer for the suggestion. Here, we provide a summary over the common failure cases of our model. Analysis shows that two failure cases of our model are *Extremely Complicated Layout* and *Unmatched Prompts*. For more details and visualization please refer to the **General Response** and figures in our rebuttal PDF.\n\n***\n4. *Q: Training details: Lack of runtime data.*\n\nWe thank the reviewer for the suggestion, and will clarify these points in our revised manuscript.\n* As described in suppl. Sec. A, we use a duo-GPU training scheduling.\n    * We use 2 A6000 GPU to conduct all our experiments and generally, each well-generated scene costs around 3-4 hours, the first GPU holds the process of diffusion model and the memory cost is ~6GB in FP16 with an image size of 512×768, another GPU holds the process of scene representation, and the memory cost is ~28GB (used Nerfacto from NerfStudio).\n    * For NeRF training, we use a constant learning rate of 1e-2 for proposal networks and 1e-3 for fields. We will add those to the main paper.\n* Note that some concurrent works cost more time, ShowRoom3D costs ~10h to produce a single scene (Tab.3 of their paper), UrbanArchitect costs ~12h and 32GB to produce a single scene (Sec.4.1 of their paper). Therefore, our DreamScene has higher efficiency than the baselines.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "14WBbAVuML", "reviewer": "Reviewer_Gzhh", "summary": "This work tackles the problem of 3D scene generation from bounding box scene, text prompt and camera trajectory. The core contribution of the work stems from a 3D scene generation method where the method is able to produce a complex scene, which the previous works could not generate due to using a panoramic representation. The method generates 2D images from multiview images using a diffusion model and then generates 3D. The results show that method is able to generate realistic 3D scenes from the user input.", "questions": "- I’m not fully understanding why camera trajectory is required to generate a 3D scene. If the users no the bounding box, couldn’t the method just use random camera trajectory to make the method work?\n- In line 134, the authors mentioned “.glet@tokeneonedot”, what does this mean?\n- What is the approximate run time of the method? Since the method generates 2D images and generate 3D, it maybe slow.", "limitations": "The method is limited to generation of a small scene. It would be nice to provide results on a large outdoor scene, which does not anyway diminish the work's contribution.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The method makes sense. Conditioned on the semantic mask, depth map, and text from a viewpoint in the 3D scene, the work uses diffusion model to generate scenes and uses NeRF to reconstruct 3D scenes. To increase 3D consistency the authors use floc removal with periodical migration.\n- Using the bbox layout seems a nice way to control the 3D scene generation, since the users usually have a specific layout in mind.\n- The figures show a nice portrait of the method.", "weaknesses": "- The scenes look more synthetic compared to the baselines such as MVDiffusion and the text2room.\n- More qualitative results are required to be convincing. What fascinates me about this work is that the method is able to produce a complex layout scene. However, there is only a single example in the appendix, which the panoramic scene methods cannot generate. The authors should provide more qualitative examples to show that the method can indeed generate complex scenes. Also, the authors claimed that the method can generate various scenes, but only few of the visual examples were provided. \n- In line 55, the authors mentioned that the method achieves SOTA performance quantitatively. Could the authors provide the quantitative results to back the claim? Also, I would like to see what the “qualitative” here refers to.\n- In Line 66, the authors mentioned “our DreamScene achieves high-quality generation in various fine-grained and complicated indoor scenes, which are never supported by previous work”. To me this sounds a little bit confusing. Previous works such as ControlRoom3D were able to generate various high-quality scenes, yet they were not able to generate complicated indoor scenes. I think the authors should clearly mention that the completed indoor scenes is the key difference between the previous works.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your comments, and address your concerns as follows:\n\n***\n1. *Q: Overall, the scenes look more synthetic compared to MVDiffusion and text2room.*\n\nWe'd like to address this concern of the reviewer by providing context for our approach:\n* **Prioritizing 3D Consistency**: Our primary focus in this work is achieving 3D consistency in generated rooms, which we consider the most crucial and fundamental factor in 3D generation tasks, especially for complex room layouts. While this approach may result in some loss of detail or texture simplification, it consistently produces rooms with reasonable geometry that accurately matches the specified layout.\n* **Differences in Scene Representation**:\n    * The two methods mentioned have different scene representations from ours. MVDiffusion and Text2Room use RGB images from diffusion models directly and reconstruct scenes using depth estimators.\n    * Our method employs Neural Radiance Fields (NeRFs), offering a different approach to scene representation.\n* **Dataset Choice and Impact**:\n    * Our diffusion model is primarily fine-tuned on the synthetic Hypersim [45] dataset, chosen for its high-quality layout data. This choice aligns with other recent works like ControlRoom3D [48].\n    * In contrast, MVDiffusion uses a version of Stable Diffusion fine-tuned on the real-world ScanNet dataset, while Text2Room uses the original Stable Diffusion model.\n* **Flexibility for Improvement**: The perceived quality of our results can be significantly enhanced by adjusting prompt instructions, distillation parameters, and camera trajectories. One example is provided in the **General Response**.\n\nIn summary, while our current results may appear more synthetic in some aspects, this is not an inherent limitation of our method. Rather, it reflects our current focus on geometric consistency and layout fidelity. Our approach offers significant flexibility for quality improvements while maintaining its core strengths in 3D consistency for complex scenes. \n\n***\n2. *Q: In order to really demonstrate the uniqueness of this work, authors should provide more challenging qualitative results (especially complex scenes).*\n\nIn our main paper, we aimed to show the diversity (i.e., different room types and styles) and layout-controllability of our model with the results. And considering the results shown in our paper, we believe those adequately demonstrate the superiority over the sota baselines [11,24,55].\n\nFollowing your suggestion, we provide more results on complex scene generation in the rebuttal PDF and the linked video, as detailed in the **General Response**. \n\n***\n3. *Q: Evaluation: authors should provide quantitative results.*\n\nWe appreciate the reviewer's suggestion for quantitative results. We have provided the results in the general response.\n\n***\n4. *Q: “our DreamScene achieves … and complicated indoor scenes, …”. This sentence seems confusing. Authors should clearly mention that (being able to generate) complicated indoor scenes is the key difference to previous works.*\n\nWe thank the reviewer for pointing this out. We will revise and clarify this in our revision.\n\n***\n5. *Q: Why couldn't we just use a random camera trajectory?*\n\nIt is important to note that generating a reasonable camera trajectory, particularly for non-square rooms with complex layouts, is far from trivial. There are several key considerations:\n* **Complexity of Trajectory Selection**: In complex room layouts, a random camera trajectory may intersect with objects or fail to cover all scene elements adequately. Even in 2D scenarios, collision-free trajectory generation remains a challenging research problem in robotics. There are dedicated papers investigating this problem in the problem of text-to-3D generation [A, B], suggesting its complexity.\n    * *[A] E.T. the Exceptional Trajectories: Text-to-camera-trajectory generation with character awareness, ECCV 2024*\n    * *[B] CameraCtrl: Enabling Camera Control for Video Diffusion Models, arXiv 2024*\n* **Consistency with Prior Work**: Our approach aligns with existing methods in the field. For instance, Text2Room also requires user-provided camera trajectories. Other baselines that use panoramas have a built-in camera trajectory (rotation around a center point), but this limits them to simple square rooms with minimal occlusions.\n* **Advantages of current setting**: By allowing for specified camera trajectories, we enable more precise control over scene exploration and visualization, which is particularly valuable in complex room layouts. This approach provides flexibility for future integration with advanced trajectory generation methods as they develop.\n\n***\n6. *Q: Typo at line 134?*\n\nWe will revise the manuscript and correct this typo.\n\n***\n7. *Q: Training details: Lack of runtime data?*\n\nWe have provided the training details in the general response.\n\n***\n8. *Q: The method is limited to small scenes. It would be nice to provide results on a large outdoor scene, which does not anyway diminish the work's contribution.*\n\nWe appreciate the reviewer's observation. Regarding the generation of large outdoor scenes:\n* This is not a limitation of our method. While large outdoor scenes might intuitively seem more challenging due to their scale, our work prioritizes 3D-consistent, complex layout-controlled generation — a challenging aspect independent of scene size.\n* Outdoor scenes generally present simpler conditions: fewer object categories, sparser object placement, and more predictable camera trajectories. For instance, most outdoor scene views follow a common paradigm (road and sky in the center, vehicles and buildings on the sides), simplifying the diffusion process.\n* Our focus on dense indoor scenes aligns with the problem setting of several prior methods [11,24,55]. We consider outdoor scene generation beyond the scope of our current model, but we acknowledge it as an intriguing direction for future research.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Gh7pIT8Yrv", "reviewer": "Reviewer_DHDm", "summary": "This paper proposes a layout-guided method for room generation. The proposed method uses 3D scene layout as a prior and leverages semantic and depth maps as 2D conditions. Specifically, the authors start by training a 2D conditional diffusion model named DreamScene2D, which utilizes ControlNet to incorporate projected semantic and depth maps and conditions. Then the 3D scene can be distillated from the conditional model. The proposed method achieves good rendering quality in complex room scenes.", "questions": "1. 3DGS has been widely used in 3D generation models due to its high speed and good quality. Have you tried to use 3DGS to replace NeRF?\n2. Is there a limit to the complexity of the scenarios a method can handle?\n3. Text2Room uses SD for missing parts inpainting. Have you tried to replace SD model in Text2room with your finetuned SD model for inpainting for controllability?", "limitations": "Please refer to weaknesses.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Using 3D layout as a prior for 3D scene generation is reasonable, which is a user-friendly format for scene control.\n2. The proposed method achieves high rendering quality and good controllability. The generated results can be controlled using both text prompts and scene layouts.\n3. The overall writing is easy to follow.", "weaknesses": "1. The key insight of the proposed method is similar to recently released paper UrbanArchitect [1], which also uses layout as prior to generate urban scenes (more complex and larger scale). DreamScene uses a pipeline based on InstructNerf2Nerf and UrbanArchitect directly uses a layout-guided SDS. Although the two works can be seen as concurrent works, I hope the authors can include a discussion.\n2. As a 3D generation model, it would be better to provide video results for better illustration. The authors can also provide 3D results for better visualization (e.g., generated 3D mesh like text2room).\n3. Quantitative results are missing. The authors should at least perform User Study regarding the visual quality and the 3D consistency. Besides, since the ControlNet is trained on specific datasets, metrics like FID can also be used for better evaluation.\n4. Some training details are missing, e.g., requirements for memory and time.\n5. Since the layout prior is used, it would be desired to use decomposed 3D representations for further editing. \n6. It would be better to include a method for automatic scene layout creation. For example, Ctrl-Room utilizes a scene code diffusion mode to generate scene layouts [2]. LLM can also be used to generate scene layouts [3].\n\n[1] Lu, Fan, et al. \"Urban Architect: Steerable 3D Urban Scene Generation with Layout Prior.\" arXiv preprint arXiv:2404.06780 (2024).\n\n[2] Fang, Chuan, et al. \"Ctrl-room: Controllable text-to-3d room meshes generation with layout constraints.\" arXiv preprint arXiv:2310.03602 (2023).\n\n[3] Hu, Ziniu, et al. \"SceneCraft: An LLM Agent for Synthesizing 3D Scenes as Blender Code.\" Forty-first International Conference on Machine Learning. 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your comments, and address your concerns as follows:\n\n***\n1. *Q: The key insight of the proposed method is similar to the recently released paper UrbanArchitect [A]. Although two works can be seen as concurrent works, I hope the authors can include a discussion.*\n\nWe appreciate the reviewer highlighting this related work. We were unaware of UrbanArchitect [A], which was released just before the submission deadline. We will include a discussion of this concurrent work in our revision.\n\nWe would like to clarify that outdoor scenes are not necessarily more complex than indoor scenes.\n* Indoor scenes feature dense, overlapping bounding boxes with fine-grained categories, while outdoor scenes typically have sparser, non-overlapping boxes with broader categories. Indoor scene generation methods must therefore handle object overlap and reason about relationships among numerous objects of varied categories.\n* Outdoor scene generation allows for greater geometric approximation. As evidenced in Fig.17 of [A], while street views are reasonably generated, peripheral elements like houses and cars lack precise construction. Indoor scenes, conversely, demand accurate geometry for all objects within a closed room.\n* Our distillation pipeline is optimized for efficiency and reduced memory usage. In contrast, [A] employs VSD and a two-stage generation process, incurring higher time and memory costs.\n* As noted by HiFA [71] and mentioned at L124, the SDS used by [A] is essentially equivalent to the IN2N we employ in DreamScene, with only slight differences in loss calculation. Building on IN2N, we introduce a novel distillation pipeline that removes artifacts through periodic migration and enhances texture consistency via specialized loss design (Sec. 3.3, L165-184), achieving superior 3D consistency.\n\nIn conclusion, indoor and outdoor scene generation present distinct challenges and cannot be directly compared. Methods developed for outdoor scenes may not be directly applicable to indoor scene generation tasks.\n\n***\n2. *Q: Visualization: it would be better to provide video results for better illustration.*\n\nThank the reviewer for the suggestion. Please refer to the general response for our provided video visualization. \n\n ***\n3. *Q: Evaluation: authors should provide quantitative results.*\n\nWe appreciate the reviewer's suggestion. We have provided the results in the general response.\n\n***\n4. *Q: Training details: Some training details are missing (requirements for memory and time)*\n\nWe thank the reviewer for the suggestion. We have provided the training details in the general response.\n\n***\n5. *Q: Since the layout prior is used, would it be desired to use decomposed 3D representations for further editing?*\n\nWhile layout-controlled scene editing is indeed an intriguing concept, we believe it falls outside the scope of our current manuscript. Our focus, in line with existing indoor scene generation methods [11,24,55], has been primarily on the generation task. Given the time constraints of the rebuttal phase, we have not explored this avenue thoroughly. However, we acknowledge the potential of this idea and consider it a promising direction for future research.\n\n***\n6. *Q: It would be better to include a method for automatic scene layout creation. E.g., Ctrl-Room utilizes a diffusion model [A]. LLM can also be used [B].*\n\nWe would like to clarify that our method is designed to be layout-agnostic, accepting scene layouts as input regardless of their source. This flexibility allows for layouts to be pre-defined, user-generated, or even LLM-generated. Hence, our framework is compatible with various layout generation methods, including LLM-based approaches like Ctrl-Room [16]. This means that a text-guided layout generation module could be seamlessly integrated into our system to support fully text-guided scene generation.\n\nGiven the time constraints of the rebuttal period, we have not implemented this extension. However, we recognize its potential and plan to include it in our final revision. This addition would further enhance the versatility and user-friendliness of our system without altering its core functionality.\n\n***\n7. *Q: 3DGS has been widely used in 3D generation models. Have you tried to use 3DGS to replace NeRF?*\n\nAs mentioned in our introduction, our method is designed to be agnostic to the underlying scene representation, allowing for compatibility with various approaches. Our choice of NeRF was based on its proven stability in handling large-scale indoor scenes. However, we concur that integrating 3DGS could potentially offer comparable performance with improved convergence time. Due to the limited time in the rebuttal phase, we will try 3DGS in our revision.\n\n***\n8. *Q: Is there a limit to the complexity of the scenarios a method can handle?*\n\nThank the reviewer for considering this problem. We provide a discussion around failure cases and in the **general response**. \n\n***\n9. *Q: Text2Room uses SD for missing parts inpainting. Have you tried to replace the SD model in Text2room with your finetuned SD model for inpainting for controllability?*\n\nWe appreciate the reviewer's insightful suggestion. \n* Indeed, our diffusion model could potentially replace SD in Text2Room for inpainting tasks, as it is specifically designed for indoor scene generation. Moreover, this substitution could introduce layout conditioning capabilities to the Text2Room framework, potentially enhancing its controllability.\n* However, it's important to note that Text2Room relies on a depth-based naive 3D reconstruction approach. This fundamental aspect of its architecture would likely still result in challenges related to inappropriate or incomplete geometry, regardless of the diffusion model used for inpainting. Consequently, we anticipate that the results may not achieve the same level of quality as our current method.\n* Due to the time limitation, we consider this investigation as future work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
# General Response

We greatly appreciate the thoughtful feedback and suggestions from all reviewers. We are pleased that the reviewers recognize the strengths of our approach, including more detailed and accurate scene generation (DRMD, Nerf), excellent controllability and user-friendly design (Gzhh, DHDm), and a well-structured, clearly written manuscript (DRMD, Nerf, DHDm).  

**New Experiments and Clarifications**: We have addressed each reviewer's concerns individually. Below, we summarize the key experiments, visualizations, and clarifications added during the rebuttal phase. 
* **Please refer to our attached PDF for all visualization figures.**
* **Video Visualization** (DRMD, DHDm): To demonstrate that our method generates scenes with complex shapes and layouts while maintaining strong 3D consistency, we provide videos of multiple generated scenes. We have emailed program chairs and got permission to provide links to anonymous videos in the rebuttal. **In compliance with the conference rules, we have sent the video to AC via a private link. Once AC has reviewed the video, it will be shared with all the reviewers.**
* **Additional Challenging Qualitative Results**(Gzhh): In addition to the video, we present more qualitative results of complex scenes in rebuttal Figure [1]. These results clearly illustrate our method's superior performance in terms of 3D consistency and complex scene generation compared to previous approaches.

* **Quantitative Results** (DRMD, Nerf, Gzhh, DHDm): In our original submission, we prioritized qualitative evaluation for various types and styles of generation, as we found it challenging to define fair, accurate, and comprehensive metrics for quantitative evaluation of different 3D scene generation methods.  
    * However, to address the reviewers' concerns, we now include quantitative comparisons with baseline methods [11,24,55] using CLIP Score (CS) and Inception Score (IS). We also conducted a user study with 37 participants, who scored the 3D consistency and overall visual quality of rooms generated by different methods on a scale of 1 to 5. Our experimental design follows previous work [48].
    * The quantitative results are presented in the table below. Our method consistently outperforms prior approaches in terms of CLIP Score, 3D Consistency, and Visual Quality. Regarding the Inception Score, our diffusion model's finetuning with fixed categories somewhat limits generation diversity. However, this is not a major concern for our task, as previous works struggle to achieve both high consistency and visual quality while being controlled by layout prompts.

| **Method**        | CS ↑      | IS↑      | Consistency ↑ | Visual Quality ↑ |
|:-----------------:|:---------:|:--------:|:-------------:|:----------------:|
| Text2Room         | 22.98     | 4.20     | 3.12          | 3.08             |
| MVDiffusion       | 23.85     | **4.36** | 3.20          | 3.47             |
| Set-the-scene     | 21.32     | 2.98     | 3.58          | 2.45             |
| **DreamScene** (Ours) | **24.34** | 3.54     | **3.91**      | **3.68**         |


* **Discussion on Failure Cases** (Nerf, DHDm): We discuss and clarify the failure cases and limitations of our method.
    * **Extremely Complicated Scenes**: Our method may struggle to reason the layout and generate rooms when the layout is excessively complex, containing many closely-placed objects or highly overlapped bounding boxes (see Figure [2] in our rebuttal PDF). 
    * **Mismatched Prompts**: When the prompt does not align with the actual room layout (e.g., a bedroom layout with a "kitchen" prompt), our method may fail to generate appropriate room contents or achieve convergence (see Figure [3] in our rebuttal PDF).
* **Improved Visual Quality** (Nerf, Gzhh): We demonstrate that generation quality is closely tied to prompt control. Our experiments reveal that using stylistic descriptions (e.g., "ocean-themed", "Van Gogh style") leads to greater quality improvements compared to specific descriptors like colors. We can further optimize appearance quality by refining textual prompts, as illustrated in rebuttal Figure [4].
* **Training Details** (Nerf, Gzhh, DHDm): We provide comprehensive training details for our model, including runtime, memory usage, and other relevant parameters used in our experiments. Our code will be made publicly available upon acceptance of the paper.
    * As described in suppl. Sec. A, we use a duo-GPU training scheduling.
        * We use 2 A6000 GPU to conduct all our experiments and generally, each well-generated scene costs around 3-4 hours, the first GPU holds the process of diffusion model and the memory cost is ~6GB in FP16 with an image size of 512×768, another GPU holds the process of scene representation, and the memory cost is ~28GB (used Nerfacto from NerfStudio).
        * For NeRF training, we use a constant learning rate of 1e-2 for proposal networks and 1e-3 for fields. We will add those to the main paper.
    * Note that some concurrent works cost more time, ShowRoom3D costs ~10h to produce a single scene (Tab.3 of their paper), UrbanArchitect costs ~12h and 32GB to produce a single scene (Sec.4.1 of their paper). Therefore, our DreamScene has higher efficiency than the baselines.

Meta-review: This paper introduces an interesting approach to generating scenes given the 3D layout. The reviewers questioned technical novelty, evaluation of the 3D consistency, more visualization, training details, and limitation analysis during the review phase. The authors provided solid feedback to relieve the concerns, and most of the reviewers mentioned that the rebuttal helps to understand the methods better, and the provided additional results are acceptable. After the rebuttal phase, reviewer DRMD raised the score to 'weak accept', reviewer Nerf keep the 'weak accept' score, reviewer Gzhh increased the score to weak accept, and reviewer DHDM keep the score 'borderline accept'. As a result, no reviewers opposed the paper's rejection. Although there is a concern about the similarity of the concurrent work published in ArXiv, AC confirmed that that issue could not affect paper judgment, given the NeurIPS paper submission policy. AC also confirms that the paper has several merits worth presenting in the field.
Decision: Accept (poster)