Title: DeBaRA: Denoising-Based 3D Room Arrangement Generation
Venue: NeurIPS 2024
Accepted: yes
Keywords: Indoor 3D Scene Synthesis

Layout Generation

Score-based Generative Models

Diffusion Models

Conditional Generation
Abstract: Generating realistic and diverse layouts of furnished indoor 3D scenes unlocks multiple interactive applications impacting a wide range of industries. The inherent complexity of object interactions, the limited amount of available data and the requirement to fulfill spatial constraints all make generative modeling for 3D scene synthesis and arrangement challenging. Current methods address these challenges autoregressively or by using off-the-shelf diffusion objectives by simultaneously predicting all attributes without 3D reasoning considerations. In this paper, we introduce DeBaRA, a score-based model specifically tailored for precise, controllable and flexible arrangement generation in a bounded environment. We argue that the most critical component of a scene synthesis system is to accurately establish the size and position of various objects within a restricted area. Based on this insight, we propose a lightweight conditional score-based model designed with 3D spatial awareness at its core. We demonstrate that by focusing on spatial attributes of objects, a single trained DeBaRA model can be leveraged at test time to perform several downstream applications such as scene synthesis, completion and re-arrangement. Further, we introduce a novel Self Score Evaluation procedure so it can be optimally employed alongside external LLM models. We evaluate our approach through extensive experiments and demonstrate significant improvement upon state-of-the-art approaches in a range of scenarios.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors present DeBaRa, a diffusion-based framework for indoor scene generation, and a self-score evaluation strategy to select conditioning input. They demonstrate the effectiveness of their approach on scene synthesis and several downstream tasks.\n\nSTRENGTHS:\n+ The empirical results are promising.\n\nWEAKNESSES:\n+ Insignificant Architectural Difference: The architectural differences between DeBaRa and other diffusion-based baselines (e.g., DiffuScene) are not substantial. The results benefited architecture difference is not sufficiently supported by experimental evidence. To be honest, we **do not** care about final performance, but the impact of the differences.\n+ Lack of Implementation Details and Ablation Studies: The paper does not provide sufficient implementation details, making it difficult to reproduce the results. Additionally, ablation studies are missing, which are crucial to understanding the contributions of individual components of the model.\n+ Missing Experiments: The paper claims advantages in downstream tasks like rearrangement and completion but lacks comparative experiments with baseline methods (ATISS, LEGO-Net, and DiffuScene) both qualitatively and quantitatively.\n+ Unclear Writing and Missing Sections: The writing is unclear in several parts of the paper. Additionally, **Section 4.4 (\"Additional Results\") is missing**, which further reduces the clarity and completeness of the work.\n+ Typing Errors: There are numerous typing errors throughout the paper, such as \"biaises\" instead of \"biases\" in Line 39. These errors affect the readability and professionalism of the manuscript.\n\nQUESTIONS:\n+ The inference time is reported to be significantly faster than DiffuScene (about 100 times faster). Are these advantages due to the use of EDM over DDPM? Please elaborate on how the choice of EDM contributes to the improved inference time.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-06-16"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies 3D room arrangement/layout generation. It proposes DeBaRA, a diffusion-based generative model, which can generate layouts given the list of furniture and the floor map of the room. The proposed method provides good results and is able to work in various scenarios: layout generation, LLM-guided text-to-scene generation, scene completion, etc.\n\nSTRENGTHS:\n- The proposed method is a novel diffusion-based generative model, which is effective in the aimed task.\n- The results generated are high-quality both qualitatively and quantitatively.\n- An LLM-guided alternative pipeline is provided to simplify the input format, which adds more functions to the model.\n\nWEAKNESSES:\n- There is no ablation study of the design choices. It is unclear whether and how each design choice and component helps the results.\n    - It would be better if the ablation results could be provided for the following designs: EDM v.s. DDPM/DDIM, designed 3D spatial objective v.s. simple MSE, different classifier-free guidance scales, with v.s. without SSE.\n- Some of the designs are not clearly described in the paper. Besides, some of the assumptions are not very clear.\n    - I wonder whether adding noise to the scene will cause the noisy layout to contain overlapped objects, objects of unreasonable sizes/rotations, etc. I did not see any of these in Fig.1. It might be better if a visualization of the process of scene being denoised could be provided.\n    - I wonder how the size is determined by the generative model. For example, a sofa can be long or short, a table can be large or small, and a cabinet can be as tall as the wall or as small as a bedside table. How will the model decide the sizes of these objects? Can the user indicate the rough size of the object?\n    - I wonder if it is assumed that most objects should be located in a rigid way that each edge is parallel to one wall? These are observed in most of the results of the proposed model in Figs. 3~6. If so, will these limit the diversity of generation results?\n- It seems that all the floor maps are relatively simple. I wonder how the model can work in more complicated floor maps, e.g., a floor with many rooms, a round room, or a triangle room.\n- In contrast to the claim in Q4 of the Checklist, many implementation details are not revealed, e.g., training settings (e.g., lr and iterations) and classifier-free guidance scales.\n- (Minor) The reason why this diffusion model can work might be that it can be regarded as an extension from some point cloud diffusion models, which also directly apply denoising on coordinates or some explicit presentations. However, this direction was not discussed in the related work.\n- (Minor) There are some minor math presentation issues. For example, multiple $min$s should be corrected to $\\min$s (e.g., L166, L178).\n\nQUESTIONS:\nPlease see \"Weaknesses\".\n\nLIMITATIONS:\nThe authors mentioned some limitations in the paper and did not indicate the societal impacts.\n\nOne possible societal impact might be that \"the generated layouts may lead to unsafe constructions, and therefore the model should given warnings about actually using it for constructing real-word rooms\".", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

comments: SUMMARY:
This paper proposes a method for room layout generation given objects and a floor plan using score-based EDM. First, three encoders are used to encode objects, the floor plan and the noise respectively. These encoded latents are then given as input to a noise based scene encoder which decodes into the output latents. The output latents are then finally decoded to their respective categories and the decoded output is pushed to be closer to the input as measures by a proposed semantic aware Chamfer distance loss. Once trained, the model is able to generate novel layouts and scene in addition to being able to perform completion, rearrangement and retrieval

STRENGTHS:
1) Qualitatively, the layouts generated seem far more coherent and spatially aligned than prior work.

2) Quantitative results provide further evidence that this model performs well.

3) The paper is well written and easy to follow.

WEAKNESSES:
1) While using a permutation invariant Chamfer loss is intuitive, I believe it should still be ablated w.r.t just the standard chamfer loss

QUESTIONS:
N/A

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-07-14