Layout-your-3D: Controllable and Precise 3D Generation with 2D Blueprint

Authors: Junwei Zhou, Xueting Li, Lu Qi, Ming-Hsuan Yang
Primary area: generative models
Keywords: 3D generation, gaussian splatting, Text-to-3D, compositional asset generation

Abstract:
We present Layout-Your-3D, a framework that allows controllable and compositional 3D generation from text prompts. Existing text-to-3D methods often struggle to generate assets with plausible object interactions or require tedious optimization processes. To address these challenges, our approach leverages 2D layouts as a blueprint to facilitate precise and plausible control over 3D generation. Starting with a 2D layout provided by a user or generated from a text description, we first create a coarse 3D scene using a carefully designed initialization process based on efficient reconstruction models. To enforce coherent global 3D layouts and enhance the quality of instance appearances, we propose a collision-aware layout optimization process followed by instance-wise refinement. Experimental results demonstrate that Layout-Your-3D yields more reasonable and visually appealing compositional 3D assets while significantly reducing the time required for each prompt. Additionally, Layout-Your-3D can be easily applicable to downstream tasks, such as 3D editing and object insertion.

Decision: accept (poster)
Reviewer scores: 6, 6, 5, 5

Reviews and discussion:
<thread>
creation date:
1729064850440

thread_type:
official_review

<thread_object>
creator: reviewer_ypwr

<content>
summary:
This paper uses a 2D layout as guidance and proposes a Collision-aware Layout Refinement to optimize 3D layouts, effectively achieving compositional 3D generation with plausible object interactions.

strengths:
- The paper is rich in content, and the proposed method successfully accomplishes the task set by its motivation.
- The writing is clear, making it easy to understand the authors' research methods.

weaknesses:
- Over-reliance on existing methods: The engineering aspect of the method is too pronounced. Aside from the newly introduced loss functions, much of the technique is a stack of existing methods, including SAM, ControlNet-tile, LGM, GeoWizard, DINO, etc. The use of LLM to assist in generating the 2D layout is also an existing strategy. Such extensive reliance on pre-existing methods can overshadow the key contribution of the paper. Additionally, I would like to ask: since LLMs are already in use, why not have the LLM or MLLM directly predict the 3D layout and then apply your Collision-aware Layout Refinement for 3D layout refinement? This would simplify the initial stage, reduce reliance on existing methods, and highlight the contribution of your technique, making the overall framework more elegant and concise.

- Limited contribution: The task goal overlaps significantly with many existing compositional 3D scene generation methods (GALA3D, graphdreamer, dreamscene， progress3D...), making the problem being addressed less novel. Furthermore, as mentioned earlier, many critical components are stacked from existing methods. Although the Feature-level Reference Loss and Tolerant Collision Loss are novel, their impact seems less substantial compared to the other parts, as seen in the visual results in Figure 8. As a suggestion for improvement, it might be better to focus on complex single-image-to-3D reconstruction instead of compositional 2D layout + text-to-3D generation. You could use multimodal LLMs to decompose objects from a complex, unclear image for reconstruction and then refine and combine them into a 3D scene. This shift could substantially enhance the novelty and contribution of the paper.

- Insufficient comparison in experiments: In the supplementary materials, methods like GALA3D and DreamScene are mentioned. Why aren’t there comparisons with compositional 3D scene generation methods in the main text, such as GALA3D, GraphDreamer, DreamScene, Progressive3D, or more recent works like The Scene Language? Even if comparing with the very recent papers isn't necessary, it would be more reasonable to compare your work with three or so methods that are highly relevant to your contribution, such as GALA3D, GraphDreamer, and DreamScene. This could also help improve the previous point about novelty. Additionally, based on my own experience, DreamFusion should be faster than LucidDreamer—on my computational resources, the timing is approximately 30 minutes vs. 50 minutes, differing from the 1 hour vs. 1 hour stated in your paper. Although a minor issue, it raises some doubts about the accuracy of the reported experimental results.

questions:
- My main concerns are discussed in the Weaknesses section. I can see from the paper that the authors have put a lot of effort into creating a valuable contribution, and despite my doubts about the contribution, experimental results, and design of the method, I still rate the paper as borderline reject (5). I believe the improvements needed cannot realistically be achieved during the rebuttal stage, and this may be due to time constraints in preparing the submission. I hope the authors can improve the method’s effectiveness and rewrite the contribution section to better highlight their approach, so that the paper can be successfully accepted at top conferences like **CVPR, ICML, or ICCV**.

- One final note: while I don't mean to discourage the authors, I have limited expectations for changes in the rebuttal, as the scope of improvement I am suggesting is quite large. Therefore, **it might not be necessary to spend too much time adding experiments during rebuttal for me**. I wish the authors all the best with their research.
</content>
</thread_object>

</thread>
<thread>
creation date:
1729260989423

thread_type:
official_review

<thread_object>
creator: reviewer_pade

<content>
summary:
the work trying to generate a 3D complex scene based on a textual prompt. they start by generating a 2D image of the scene using a 2D text-to-image generator, they extract 2D boxes for each instance. 
Next, they create each object in a 3D instance and position them according to the boxes.
the final step is SDS which fine-tune the positions for natural placement, specifically in a collision-free manner.

[1] Yang, Yue, et al. "Holodeck: Language guided generation of 3d embodied ai environments." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024
[2] Rahamim, Ohad, et al. "Lay-A-Scene: Personalized 3D Object Arrangement Using Text-to-Image Priors." arXiv preprint arXiv:2406.00687 (2024).

strengths:
generating complex scenes is a difficult task today.
this work tries to solve this task, into stages to deal with each part separately.

weaknesses:
1. no novelty in the approach, you concatenated known strategies while trying to solve this problem.
2. can it be solved using previous methods like [1]
3. Besides the bear and the book, I can't see any interaction between objects. can you propose how to create interactions? (like the bird drinking the beer)

questions:
1. why do the bird, bear, and Bagle look different in each example?
2. what happens if the generated 2D images don't contain all the specified objects? it does happen a lot with the generation of complex scenes. like what is shown in [2]
</content>
</thread_object>

</thread>
<thread>
creation date:
1730510871063

thread_type:
official_review

<thread_object>
creator: reviewer_miqw

<content>
summary:
This paper tries to address the problem of lay-out controlled 3D generation task. 
Interestingly, it relies on 2D layout instead of 3D layouts, and the 2D layout could be generated by LLM given a text prompt. 
With this 2D Layout, it first generate a reference image with an off-the-shelf 2D-layout image generation model, and then try to reconstruct the 3D model and layout from the reference image. 
This generated 3D model is coarse and may not very accurate, and this paper refines the generated model with SDS and feature loss, as well as collision-tolerant loss.
This paper proposes a very interesting pipeline using many off-the-shelf components, and it achieves good results

strengths:
1. This paper proposes an interesting pipeline by combining several (off-the-shelf) components in a novel way. Getting 2D reference images from layout and trying to reconstructing 3D models from them leads a interesting 3D reconstruction task, and this paper provides a reasonable solution for it, by utilizing the inpainting, DINO feature distance and off-the-shelf image generation model.
2. This paper involves several intesting losses like collision-tolerant loss, feature-level loss, and SDS with adjusted timesteps. 
3. Results are good. and the paper is easy to read and follow.

weaknesses:
1. For baselines, it would be more convincing to seriously compare against models with 3D layout control like ComboVerse or [Compositional 3D Scene Generation using Locally Conditioned Diffusion]. The current paper only shows a single scene comparison against ComboVerse in Figure 7, which is insufficient to demonstrate the advantages of using 2D layout and the proposed method. More rigorous evaluation, such as those shown in Tables 1 and 2, along with a user study, would be needed for such comparisons.

2. After reconstructing individual 3D models from 2D images, it is critical to align the pose (6D pose) of the reconstructed 3D model with the image appearance. The proposed method achieves this by randomly sampling many views of the 3D model and comparing them against the reference image by computing the DINO feature distance. However, I am curious about the accuracy of this method, as the rendered image could differ significantly from the reference image even with perfect poses if the distance (object scale) is not accurate. It would be more convincing to show results demonstrating the accuracy of this method.

3. Regarding the reconstruction of objects from reference images, how is the mask/segmentation obtained for each object? Is this process automated, or does it require manual intervention? This question also appears in the following inpainting and pose alignment sections. Do we need to manually specify which object to inpaint or segment during this process or not?

questions:
1. For clarification, when inputing the text prompts, what's the prompt for LLM to generate 2D layouts? Also, do you mark the object in bracket like text prompts in figure 5 or LLM is supposed to decide which objects should be created?

2. The proposed method uses many off-the-shelf algorithm and components like LLM generating 2D layouts, reference image generation, pose alignment. I am curious about the successful ratio of the whole process, and some potential failure cases.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730612044084

thread_type:
official_review

<thread_object>
creator: reviewer_tchq

<content>
summary:
- They enforce structural control of a 3D scene using a 2D layout. Their optimization process avoids collisions.
- Their method works by first generating a 2D image based on text description, then use SAM to segment each isntnace in the reference iamge, with in-painting done on occluded parts.
- They then use a large mulit-view gausian model to generate coarse 3D instances of objects within the scene. 
- Each 3D instance is then arranged into 3D space by lifting the bounding boxes into 3D, done by optimizing the rotation according to similarity in the DINO space of the rendered images of the rotated instances.
- final two stages refine the layout according to collisions and refines the instance representations.

strengths:
The papers shows a potential way to use many existing models together to generate compositional 3D scenes. The results show that the method produces reasonable visual reconstructions and geometries. The method seems well motivated, and the paper is decently written.

weaknesses:
My concern is that this work lacks fundamental novelty, in that prior works like Compositional 3D Scene Generation using Locally Conditioned Diffusion (https://arxiv.org/abs/2303.12218), Towards Text-guided 3D Scene Composition (https://arxiv.org/pdf/2312.08885) and CG3D: Compositional Generation for Text-to-3D via Gaussian Splatting (https://arxiv.org/pdf/2311.17907) all present various ways to create 3D scenes using implicit 3D represntations in a compositional way.

There does not seem to be any comparisons to these works, or mention of them in the related works (though CG3D was mentioned in the collisions section, but not compared with in the experiments section). A comparison to contextualize this work seems very important, but lacking.

questions:
Within the collision loss, is there any way to take into account the shape (covariance) of each gaussian in 3D space? My understanding is that it's only using the positions of the the Gaussians right now. Have you observed this to be a limitation?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732078870930

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General comments to all the reviewers

comment:
Dear reviewers,

We thank you for the valuable reviews and suggestions. We first address some common questions raised by reviewers. 

## 1. Experiments and comparison with other compositional Text-to-3D generation methods (Reviewer tCHQ, miqw, YpWR)
Reviewers suggest to include comparisons with other Compositional 3D generation works, including Locally Conditioned Diffusion, SceneWiz3D, CG3D, ComboVerse, GALA3D, DreamScene, and GraphDreamer. Note that among all of these works, only GALA3D, DreamScene and GraphDreamer have released their official implementations. As a result, we may not be able to compare to other methods mentioned by reviewers. We added all references suggested by reviewers in **Line144-146** in the revised manuscript. Next, we compare the proposed method with GALA3D, DreamScene and GraphDreamer.

For **GALA3D**, the current publicly available code does not include the finetuned ControlNet. As a result, we cannot carry out the layout fine-tuning stage. To make the comparison as fair as possible, we reached out to the authors of GALA3D, they explained that the ControlNet is specially finetuned for indoor room layout, thus it will not affect the final results of common input prompts used in our paper. Therefore, we use the publicly available code of GALA3D and show the comparison in **Appendix D.2** and **Fig. 20**.

**DreamScene** aims at generating individual 3D objects and fitting them into fixed 3D layouts. However, we **do not** compare with DreamScene since it does not include layout refinement and cannot produce reasonable results that contain certain interactions between objects.

As suggested, we have included comparisons with **GraphDreamer** in the revised manuscript (**Fig. 5** and **Tab. 1**). 

&nbsp;
## 2. Novelty and Contributions (Reviewer tCHQ, PaDe, YpWR)
We would like to clarify the novelty and contribution of this work. 

- First, we exploit easily-accessible and accurate 2D layouts as blueprint for controllable and precise 3D generation. As stated in **L081-L082** in the manuscript (marked in orange color), we leverage 2D layout to provide an easily-accessible and accurate guidance for controllable and precise 3D generation. This setting differs from existing compositional text-to-3D generation methods, which rely on Large Language Models (LLMs) to generate imprecise 3D layouts that lead to subpar outcomes and time-consuming optimization processes (**Fig. 18** in **Appendix D.1**). This is our main motivation to leverage the strength and performance of 2D layout methods and lift the results to 3D. 

- Second, we emphasize that generating 3D assets from 2D layout is a challenging task. To address this, we meticulously design a method that incorporates several novel modules, including: **a)** a 3D layout initialization module that predicts plausible object geometry, appearance, location as well as rotation, **b)** a refinement stage that improves not only individual object appearance but also their interactions, using a novel collision-aware constraint. Consequently, our method not only generates better results, but also significantly reduces processing time (minutes vs. hours) compared to existing approaches.

- Third, our work extends beyond text-to-3D generation. As illustrated in our paper (**Fig. 10, 11**), our pipeline also supports additional applications such as instance-wise customization and editing, offering greater control and flexibility in the generation process.

&nbsp;
## 3. Explain why not use LLM/MLLM to infer 3D layouts instead of 2D layouts? (Reviewer YpWR)

We note that state-of-the-art text-to-3D models (GraphDreamer, GALA3D) cannot generate reasonable results when given poorly initialized 3D layouts (**Fig. 5, 20** in our paper).

We use 2D layouts as guidance for the following reasons:
- Large language models (LLMs) are trained for general-purpose tasks and often struggle with reasoning, leading to inaccurate spatial relationships. Specifically, the 3D bounding boxes generated by LLMs tend to be imprecise and of low quality, which creates challenges in the subsequent 3D generation process. For visualization, we provide examples of 3D bounding boxes generated by LLMs in the rebuttal, please refer to **Fig. 18** and **Appendix D.1** in the manuscript.
- In contrast, 2D layout generation models are specifically designed for this task and produce much more accurate initial layouts. By first generating a 2D layout, we break down the complex problem into two more manageable subproblems: *2D layout generation and 3D lifting*. As a result, as demonstrated by the outcomes in the main paper, our framework generates higher-quality 3D scenes in a shorter amount of time.
- Finally, 2D layouts are much easier for applications like customization and interactive editing by users compared to 3D layouts.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper tackles the problem of controllable 3D scene generation. The proposed pipeline starts with 2D layout generation and then progresses through 3D instance reconstruction and arrangement. Their key insight is to decompose the original complex problem into a series of simpler and modular tasks. Then, by exploiting state-of-the-art modules for each stage, the proposed method is able to produce plausible 3D reconstructions with reasonable scene layouts. On the negative side, the reviewers are concerned that similar pipelines have already existed in the past literature and there are not sufficient and comprehensive evaluations/comparisons. The reviews were on the fence. During the discussion phase, the authors have attempted (and managed) to address most of the concerns. While there is still a debate going on regarding whether the novelty is enough and whether the work is incremental (ie, switching from generating 3D layout from LLM to producing 2D layout and lift it to 3D), considering the fact the authors incorporated quite a bit of additional results and showcased superior performance during the rebuttal, the ACs agree that the paper is marginally above the acceptance bar and decide to recommend acceptance. The authors are strongly encouraged to incorporate the feedback from the reviewers into their final camera ready version.