HouseCrafter: Lifting Floorplans to 3D Scenes with 2D Diffusion Model

Authors: Hieu Trung Nguyen, YIWEN CHEN, Vikram Voleti, Varun Jampani, Huaizu Jiang
Primary area: generative models
Keywords: diffusion model, indoor scene generation

Abstract:
We introduce HouseCrafter, a novel approach that can lift a floorplan into a complete large 3D indoor scene (e.g., a house). Our key insight is to adapt a 2D diffusion model, which is trained on web-scale images, to generate consistent multi-view color (RGB) and depth (D) images across different locations of the scene. Specifically, the RGB-D images are generated autoregressively in a batch-wise manner along sampled locations based on the floorplan, where previously generated images are used as condition to the diffusion model to produce images at nearby locations. The global floorplan and attention design in the diffusion model ensures the consistency of the generated images, from which a 3D scene can be reconstructed. Through extensive evaluation on the 3D-Front dataset, we demonstrate that HouseCraft can generate high-quality house-scale 3D scenes. Ablation studies also validate the effectiveness of different design choices. We will release our code and model weights.

Decision: reject
Reviewer scores: 6, 5, 6

Reviews and discussion:
<thread>
creation date:
1730645286008

thread_type:
official_review

<thread_object>
creator: reviewer_9saq

<content>
summary:
This paper presents an approach for generating house-scale 3D indoor scenes based on 2D floorplans.
Starting from a floorplan, camera poses are uniformly sampled throughout the scene.
Then, these camera poses are traversed, and RGB-D images are synthesized in batches in an auto-regressive manner, where
each batch is conditioned on the existing nearby RGB-D images (if they exist) and an embedding of the floorplan. 
By incorporating depth into the synthesis process, the generated RGB-D images exhibit better visual and depth consistency than generating RGB images and using a standalone monocular depth estimation. 
Experiments on the 3D-Front dataset show that this approach can generate large, consistent, high-quality scenes that accurately align with the provided floorplans.

strengths:
- The paper addresses a compelling problem with potential applications in architectural design and game development.
- The paper covers the literature of novel-view synthesis well.
- The method is clear and well explained.
- The floorplan embedding method is novel and clever.
- The generated results are geometrically and visually consistent.

weaknesses:
- The paper does not clarify why generating "house-scale" scenes is preferable to generating "room-scale" scenes. Why is consistency across the entire house necessary, rather than generating individual rooms and combining them? In reality, different rooms within a house may vary in style and are not necessarily consistent with one another. This needs to be clarified as the authors used it as grounds to exclude comparison against Controlroom3d and Ctrl-Room at L207.

- The authors argue at L203 that they can not compare against BlockFusion as it generates 3D meshes without texture. However, the BlockFusion paper provides some interesting results for textured room generation where Meshy was used to texturize the scene. I believe that BlockFusion is highly relevant and should be considered for comparison.

- The quality of the generated scene (textures and geometry) is relatively low compared to existing room-scale approaches mentioned in the paper. Is this caused by the use of an old reconstruction method TSDF? If so, why are more recent approaches, such as depth-aware Nerfs/Gaussian splatting, not considered?

- The improvement from generating the depth map with the RGB images has not been demonstrated! To be able to judge if this approach is beneficial over the standard RGB synthesis followed by a monocular depth estimation, additional experiments are needed! For example, how would the scene reconstruction be affected if only the RGB images were predicted and then the depth was estimated using an off-shelf monocular depth estimation model?

questions:
- Can the proposed method used for outdoor scenes similar to BlockFusion? In that case, how can the depth range in section 3.2.1 be extended to longer distances?

- What is the motivation for the proposed layout embedding approach? Were other embedding approaches tested?

- Why do you choose to add $\text{PosEnc}(p_K)$ to $v_K$ in (4)? 

- What is the inference time for generating a complete floorplan?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730646999486

thread_type:
official_review

<thread_object>
creator: reviewer_fdmd

<content>
summary:
This paper proposes a method for generating house-level 3D scenes conditioned on 2D floorplans. Given a 2D floorplan, the approach first samples camera positions and rotations within free space. Next, it adapts a large 2D diffusion model, trained on internet images, to generate RGBD scans. Finally, it applies TSDF Fusion to reconstruct a 3D mesh with textured information derived from all the generated dense RGBD point clouds. Experimental results show that the generated novel views maintain consistency with both the input view and the floorplan. Furthermore, the resulting 3D scenes exhibit improved global coherence compared to existing methods, such as CC3D and Text2Room.

strengths:
1. Using 2D floorplans as a conditioning method requires less human intervention and manual effort, yet produces globally coherent 3D scenes. In contrast, methods like Text2Room are relatively simple but do not ensure a plausible layout in generated 3D scenes. This highlights the advantages of using 2D floorplans as a basis for 3D scene generation.
2. The proposed approach of floorplan conditioning and multi-view RGB-D conditioning may inspire new methods for extending 2D generation to 3D at the scene level.
3. The paper is well-written and easy to follow, offering a detailed overview of related work. The motivation to generate 3D scenes by creating multi-view 2D observations is also well-justified and logical.

weaknesses:
1. The current comparisons are not entirely convincing. For example, CC3D focuses on novel view rendering at the scene level rather than geometry, and its renderings qualitatively appear superior to those in this work. Additionally, Text2Room, which generates scene layouts based on text input alone, may not provide an appropriate basis for comparison in this context.
2. Since this paper focuses on floorplan-based 3D scene generation, a more relevant comparison would be with BlockFusion [1], which operates directly in the 3D domain to generate scenes based on input scene layouts (or unconditionally), making it a closely related baseline.
3. The generated 3D scenes exhibit relatively coarse and blocky geometry; for example, most chairs and tables lack legs. This may be due to the camera sampling strategy or the resolution of the generated images.
4. Additionally, the paper should report the number of RGB views used and the time required to generate a complete 3D scene for a more thorough evaluation.

Reference:

[1] BlockFusion: Expandable 3D Scene Generation using Latent Tri-plane Extrapolation

questions:
1. What is the bottleneck in the current pipeline for generating high-resolution 3D geometry? The paper reports that the generated 2D images have a resolution of only 256, which may be insufficient for reconstructing high-quality 3D geometry.
2. Can the pipeline support interactive editing, such as moving objects, without requiring retraining?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730696646880

thread_type:
official_review

<thread_object>
creator: reviewer_zzrx

<content>
summary:
This work proposes a novel approach for transforming 2D floorplans into detailed 3D indoor scenes, such as houses. The authors introduce HouseCrafter, an autoregressive pipeline that leverages a pre-trained 2D diffusion model to generate consistent multi-view RGB and depth images. This method addresses the challenges of generating large-scale scenes by sampling camera poses based on the floorplan and producing images in batches, ensuring spatial coherence across the generated views.

strengths:
(1) This is a well-written paper.
(2) This method generates high-resolution depth images for larges-scale scene reconstruction, which is more practicala and meaningful in real-world scenerios.
(3) The proposed method is compared with various methods. The experiments are complete and convincing
(4) Some visualizations are helpful to understand.

weaknesses:
(1) Lack of inference time comparison.

questions:
(1) In line 1080, what do you think are the advantages and disadvantages of sampling pose and sampling point cloud.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732484873074

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General response to Reviewers

comment:
We sincerely thank all the reviewers for their valuable feedback, constructive suggestions, and insightful comments on our work. We are happy to see the proposed method recognized as “**practical and meaningful**” (Zzrx), “**inspire new methods for extending 2D generation to 3D … well-justified and logical**” (FdMd), “**compelling problem with potential applications**” (9Saq).

We have revised the manuscripts with changes marked in blue:
- Adding comparison with BlockFusion+Meshy (Table 1, Figure 6, Figure 14)
- Adding comparison with Monocular depth estimation baseline (Appendix D) 
- Adding comparison with baseline floorplan encoding method (Appendix B)
- Adding time benchmark (Appendix C)

Some common questions asked in the reviews are addressed below, while the individual questions are responded directly to the reviews.

**Q1: Inference time comparison with other methods (Reviewers Zzrx, FdMd, and 9Saq).**

- We provide the inference time comparison in the table below. We note the average number of images/blocks per scene (#images/Block), and the total time to generate a scene. All models were run on an A6000 GPU. 

- _Note that while Text2Room, BlockFusion, and HouseCrafter produce meshes as final output, CC3D generates volumetric latent as scene representation and requires neural rendering to get any view. Hence we followed CC3D’s codebase to generate a room then render 40 images and report the total time of generation and rendering._
| Method          | #Images/Blocks      | Total time (min)  |
| ------------------ | ------------------------- | --------------------- |
| Text2Room    | $217_{\\pm  5}$      | $50_{\\pm  1}$   |
| CC3D             | 40                           | < 1                     |
| BlockFusion   | *$23_{\\pm 10}$     | $30_{\\pm 12}$   |
| HouseCrafter | $1000_{\\pm 400}$ | $24_{\\pm 10}$  |

_*denotes the number of blocks_

 **Q2: Comparison with BlockFusion (Reviewers FdMd and 9SaQ).**
- This is a good point. As suggested by **Reviewer 9Saq**, while BlockFusion [1] only generates the geometry, Meshy [2] can be used to texturize the generated scene given text prompt. Hence, we add the quantitative comparison with BlockFusion here and the qualitative comparison in **Figure 6 and Figure 14** in the revised paper.
| Method                      | Visual (IS) &#8593; | Floorplan -  mAP@25 &#8593; |
| --------------------------- | -------------------------- | ----------------- |
| Text2Room               | **5.35**                    | -                    |
| CC3D                       | 4.02                          | 25.60            |
| BlockFusion             | 5.01                          | 0.81              |
| HouseCrafter           | 4.24                          | **46.48**       |
| 3DFront GT images | 4.50                          | 54.51             |


- In terms of texture diversity, Meshy provides a better score as its generated textures are more diverse in style due to the additional controls from the text prompts. 
- In terms of floorplan-compliant evaluation via 3D object detection (mAP), the generated textures are not detailed and accurate enough to distinguish different object instances in the scene. Hence the object detection model failed to recognize them, leading to low mAP while the geometry is reasonable. Please refer to **Figure 14** for the visualization of the texture and geometry of the scenes. 
- It is worth noting that texturizing the generated meshes of BlockFusion using Meshy is not trivial. The main challenge is that Meshy restricts the input meshes to be smaller than 50MB, so we had to simply the mesh using Quadric Error Metric Decimation method [3] to limit the number of triangles to 1,000,000. While the mesh precision is reduced, the objects in the scenes are still recognizable as shown in **Figure 14

[1] BlockFusion: Expandable 3D Scene Generation using Latent Tri-plane Extrapolation, Siggraph 2024

[2] https://www.meshy.ai/

[3] Surface Simplification Using Quadric Error Metrics, Siggraph 1997
</content>
</thread_object>

</thread>


Metareview:
metareview:
In this paper, the authors have proposed HouseCrafter by generating house-scale 3D scenes from 2D floorplans. The key idea is to leverage a pre-trained 2D diffusion model to generate multi-view images of the scene in an autoregressive manner, so that the TSDF fusion can be used to reconstruct the scene with textured meshes. The setting of using 2D floorplans as a condition to generate 3D scenes is interesting. However, there are still some concerns of the paper. There should still be more comprehensive comparisons with BlockFusion, and the pose sampling and the TSDF volumetric fusion modules should be further improved to obtain better performance. The quality is still unsatisfied. Due to the reasons above, I recommend a decision of rejection of this paper.