Layout-your-3D: Controllable and Precise 3D Generation with 2D Blueprint

## Abstract

Abstract We present Layout-Your-3D, a framework that allows controllable and compositional 3D generation from text prompts.
Existing text-to-3D methods often struggle to generate assets with plausible object interactions or require tedious optimization processes.
To address these challenges, our approach leverages 2D layouts as a blueprint to facilitate precise and plausible control over 3D generation.
Starting with a 2D layout provided by a user or generated from a text description, we first create a coarse 3D scene using a carefully designed initialization process based on efficient reconstruction models.
To enforce coherent global 3D layouts and enhance the quality of instance appearances, we propose a collision-aware layout optimization process followed by instance-wise refinement.
Experimental results demonstrate that
Layout-Your-3D yields more reasonable and visually appealing compositional 3D assets while significantly reducing the time required for each prompt.
Additionally, Layout-Your-3D can be easily applicable to downstream tasks, such as 3D editing and object insertion.
Video and results can be viewed on our website: https://colezwhy.github.io/layoutyour3d/

## 1 Introduction

The recent years have witnessed
significant advances in 3D content creation.
By leveraging advanced diffusion models through optimization (Poole et al., ) or training efficient reconstruction models (Hong et al., ), numerous methods successfully synthesize high-quality 3D objects from text prompts or images.
These approaches have facilitated various applications such as virtual reality (Chen et al., ; Liu et al., ), robotics (Huang et al., ; Wan et al., ), etc.

Nevertheless, existing text-to-3D methods often struggle to synthesize plausible object interactions in compositional 3D asset generation.
For instance, given the text prompt ”a brown soft with a cushion, a teddy bear and a basketball on it”, none of the state-of-the-art methods (Wang et al., ; Liang et al., ; Yi et al., ) shown in Fig. 5 (last row) correctly predict the plausible location and scale of the objects.
This is because these optimization-based models create 3D assets by distilling prior knowledge from 2D diffusion models (Rombach et al., ).
Consequently, they require extensive optimization time for each prompt and suffer from the same issues such as semantic confusion or missing objects as the diffusion models. Meanwhile, feed-forward methods (Tang et al., ) produce 3D shapes instantly but often perform poorly in creating complex compositional cases due to the lack of compositional 3D objects in their training data (Deitke et al., ).
Furthermore, none of these methods provide any user control over the 3D generation process.
These limitations significantly restrict the application of text-to-3D methods in real-world scenarios.
Two lines of work aim to resolve these issues and generate plausible compositional 3D assets.
First, several methods use various attention mechanisms (Li et al., ) to enhance the alignment of synthesized 3D objects or scenes with compositional text descriptions.
However, these methods do not offer controllable signals such as layout.
Another line of approaches focuses on text-to-3D scene generation (Zhou et al., ) by first synthesizing each 3D instance within the scene, followed by a global layout refinement through Score Distillation techniques (Poole et al., ).
While these schemes aim to improve scene coherence, they require significantly more computational resources and generate less realistic 3D results than conventional text-to-3D object synthesis methods.

In this work, we present Layout-Your-3D, an efficient text-to-3D generation method for compositional 3D scene synthesis with precise control.
Specifically, given a text prompt describing few objects and their spatial relationships, we aim to generate these objects and their natural interactions. We note that this is similar to the setting in (Sun et al., ) but differs from  (Zhou et al., ; Li et al., ), which focuses on synthesizing complete indoor/outdoor 3D scenes.
Our key idea is to utilize a 2D layout as guidance to produce target reference images for 3D scene generation, which not only allows precise user control but also produces more plausible object interactions.
We start by collecting a user-provided or LLM-generated 2D layout and predict a reference image depicting the desired instances and their interactions.
Using this reference, we propose a carefully designed process to create coarse 3D scenes with roughly reasonable layouts using reconstruction models (Tang et al., ).
While these feed-forward reconstruction models are computationally lightweight, they often yield inferior 3D geometry and appearance.
To address these limitations, we introduce a two-step disentangled refinement stage to enhance the quality of 3D instances and global layout.
First, we develop collision-aware layout optimization to adjust the layouts of the objects to create a globally coherent and visually appealing composition.
By incorporating collision awareness, we prevent objects from severe intersections, resulting in a realistic compositional layout.
Second, we carry out instance-wise refinement to further improve the geometry and texture of individual objects, which also allows the customization of different instances as an added benefit.

We evaluate our method with a proposed validation set, extensive experiments, and ablations to showcase the ability of Layout-Your-3D in rapid and accurate 3D generation.
In addition, we conduct comparisons with current state-of-the-art text-to-3D methods, demonstrating the potential of Layout-Your-3D in compositional 3D generation, as well as customized design and editing.
The contributions of this work are threefold:

- •
We propose Layout-Your-3D, a practical approach for highly controllable compositional 3D generation, demonstrating superior performance compared to the baseline methods.
- •
We present a reconstruction-based initialization process that efficiently produces reasonable coarse 3D scenes. We further enhance the texture, geometry, and global layout of the 3D scenes using a carefully designed two-step disentangled refinement stage.
- •
We conduct comprehensive experiments to assess the effectiveness of our proposed method against state-of-the-art baselines. Our approach demonstrates superior qualitative and quantitative results while significantly reducing the time for generation (i.e., 12 minutes).

## 2 Related Work

Text-to-3D generation.
Significant advances have been made in this field thanks to the rapid development of foundational diffusion models (Rombach et al., ).
The DreamFusion (Poole et al., ) model enables the generation of 3D objects using powerful 2D diffusion models by introducing the Score Distillation Sampling (SDS).
Numerous works
further improve text-to-3D generation performance by developing various SDS methods (Yu et al., ; Ma et al., ), leveraging advanced 3D representations such as DMTet (Chen et al., ) and 3D Gaussian Splatting (3DGS) (Chen et al., ; Liang et al., ), or leveraging pre-trained normal and depth diffusion models (Qiu et al., ).
For instance, ProlificDreamer (Wang et al., ) proposes a particle-based variational framework that promotes the generation of more diverse and less saturated 3D objects, while GaussianDreamer (Yi et al., ) accelerates optimization by leveraging 3DGS.
Since 3DGS enables fast generation speed with various initialization methods and state-of-the-art generation performance, we are building on it for 3D generation from text prompts.

Image-to-3D Reconstruction. Instead of using inherently ambiguous text descriptions, one line of works reconstructs 3D objects from one or more input images.
Numerous methods (Liu et al., ; Long et al., ; Shi et al., ) finetune 2D diffusion models with 3D-aware data to generate highly consistent multi-view images.
Subsequent approaches, such as LRM (Hong et al., ) and LGM (Tang et al., ), further accelerate 3D reconstruction from single-view images through an efficient feed-forward process.
As these reconstruction models are trained on synthetic, single-object-centric datasets, they often struggle to reconstruct plausible 3D assets that include multiple objects or produce realistic appearances for out-of-distribution input images.
In this work, we leverage the efficient reconstruction models to produce initial coarse 3D scenes while addressing their limitations in compositional generation and the issue of unappealing appearances.

Compositional 2D and 3D generation. Generating or reconstructing scenes with multiple objects presents significant challenges in 2D and 3D contexts.
In the 2D domain, numerous models (Li et al., ; Xie et al., ; Zhou et al., ) use 2D bounding box layouts as conditions to guide the generation process.
These works perform well in controlling the number of instances and their attributes.
Due to the additional depth dimension, generating compositional assets in the 3D domain faces even more significant challenges.
Several image-to-3D reconstruction methods, such as that by (Han et al., ; Chen et al., ), reconstruct each object in the scene independently and subsequently optimize the global layouts by aligning them with reference images or utilizing additional guidance from diffusion models.
Another line of work focuses on compositional text-to-3D generation.
For example, COMOGen (Sun et al., ) adopts 2D layouts to guide the SDS process to provide accurate control over the generation process.
GroundedDreamer (Li et al., ) integrates the core idea of Attend-and-Excite (Chefer et al., ) into MVDream (Shi et al., ) to facilitate the generation of compositional multi-view images from complex text prompts.
Moreover, text-to-3D scene generation methods (Zhou et al., ; Li et al., ) that rely on the score distillation technique often utilize 3D layouts provided by users (Bai et al., ) or generated from LLMs to provide coarse compositional spatial information to construct the scene.
However, these methods often require hours for generation and may produce unreasonable object interactions or unrealistic appearances.
In this work, we leverage efficient reconstruction models to achieve high quality and efficiency, significantly advancing high-fidelity compositional 3D scene generation.

## 3 Method

Given a 2D layout and a text prompt, we aim to synthesize a 3D scene, including multiple objects and specific interactions aligned with the layout and text.
Unlike existing text-to-3D methods, our approach aims to generate reasonable 3D geometry and texture for each object and produce a plausible layout with all objects interacting naturally (i.e., with reasonable location, scales, etc.).

As shown in Fig. 2, we represent the 2D layout as a set of bounding boxes and their corresponding instance names $B=\{(b_{1},y_{1}),(b_{2},y_{2}),...,(b_{N},y_{N})\}$, which can be extracted from the overall text prompt $Y_{B}$.
Given these bounding boxes and instance names, we design a coarse 3D generation stage (see Sec. 3.1) that creates all 3D objects along with a roughly reasonable 3D layout.
To enhance the visual quality and consistency of the layout, we introduce a disentangled refinement stage (see Sec. 3.2), which includes a collision-aware layout refinement, followed by an instance-wise refinement.
Additionally, our method allows users to interactively edit each instance and the global layout as a by-product.
In the following, we provide detailed discussions of these components.

Figure: Figure 2: Overview of Layout-Your-3D. Given a 2D layout and text prompt, our coarse 3D generation stage (green box, see Sec. 3.1) generates coarse 3D instances along with roughly reasonable layouts. The disentangled refinement stage (see Sec. 3.2) then refines the 3D layout and enhances individual instance quality by leveraging a collision-aware layout refinement (blue box) followed by an instance-wise refinement (yellow box).
Refer to caption: x2.png

### 3.1 Coarse 3D Generation Stage

Given a set of bounding boxes with corresponding object names, the coarse 3D generation stage synthesizes each 3D object and arranges them in a roughly reasonable layout.
To achieve this, we begin by creating a reference image based on the 2D layout. Next, we reconstruct each 3D object (instance) using an efficient reconstruction model. We adopt 3DGS as our 3D representation, considering its flexibility and geometry attributes.
For layout initialization, naively converting the 2D layout into 3D space using depth information can lead to unrealistic overlaps and collisions between objects. To address this issue, we develop an initialization process to produce plausible 3D layouts by incorporating both geometry and semantic knowledge.

Reference Image Generation. Instead of lifting 2D bounding boxes to 3D space and generating 3D objects from scratch (Sun et al., ), our method first synthesizes a 2D reference image based on the given layout and text prompt. This reference image significantly narrows the solution space and provides crucial guidance for the subsequent generation process.
Specifically, we use the state-of-the-art method MIGC (Zhou et al., ) for layout-guided 2D image generation, defined as:

$$ $\displaystyle I_{ref}=\texttt{MIGC}(Y_{B},B),$ (1) $$

where $I_{ref}$ represents the generated reference image.

Individual 3D Object Reconstruction. Next, we reconstruct each instance from the generated 2D reference image.
To alleviate the influence of occlusion among instances, we first segment each instance in the reference image $I_{ref}$ using Segment-Anything Model (SAM) (Kirillov et al., ), and inpaint them as follows:

$$ $\displaystyle I_{i},M_{i}=\texttt{SAM}(I_{ref},b_{i});\ \hat{I}_{i}=\texttt{SD }(I_{i},(\sim{M_{i}})\cap b_{i},y_{i}),$ (2) $$

where $I_{i}$ and $M_{i}$ are the RGB values and the foreground mask of the $i_{th}$ instance. $SD$ is the Stable Diffusion inpainting model, $(\sim{M_{i}})\cap{b_{i}}$ represents the region to be inpainted and $\hat{I}_{i}$ is the $i_{th}$ completed 2D instance.
Note that to make the inpainted image compatible with the object-centric reconstruction models, we also perform post-processing to remove the background and center the object within the image $\hat{I}_{i}$.
Fig. 3 illustrates this segmentation, inpainting and post-processing process.
Since the reconstruction models are sensitive to input image quality, we optionally enhance the details of the inpainted image $\hat{I}_{i}$ using a ControlNet-Tile (Zhang et al., ) model.
Finally, $\hat{I}_{i}$ is fed into the Large Multi-View Gaussian Model (LGM) (Tang et al., ) to generate the coarse 3D instance $A_{i}$.
Thanks to the explicit geometry priors from LGM and attributes of gaussian splatting, we could primarily eliminate the Janus Problem and add fine-grained details to the original instances in the subsequent refinement step.

Figure: Figure 3: Example of the instance segmentation, inpainting and post-process.
Refer to caption: x3.png

3D layout initialization. Given the synthesized 3D instances, we arrange them in the 3D space by lifting the 2D bounding boxes $\{b_{i}\}$ discussed above into 3D.
Each 3D bounding box is defined by its scale, rotation, and translation (denoted as $\{s_{i},r_{i},t_{i}\}$).
From the given 2D layout, the scale $s_{i}$ of each 3D box can be intuitively set to $W_{b_{i}}/{\overline{W_{b_{i}}}}$, in which $W_{b_{i}}$ and $\overline{W_{b_{i}}}$ are the width of original and post-processed 2D box $b_{i}$ (see Fig. 3).
For the translation $t_{i}$, we can directly get the first two elements (i.e., $x_{i},y_{i}$) of the transition vector $t_{i}=\{x_{i},y_{i},z_{i}\}$ as the center coordinates of $b_{i}$.
Additionally, $z_{i}$ is initialized as the average of the foreground depth value predicted by an off-the-shelf depth estimation model (i.e., GeoWizard (Fu et al., )), which is computed as:

$$ $\displaystyle z_{i}=\text{mean}\left(M_{i}\odot\texttt{Depth}(I_{ref})\right).$ (3) $$

Rotation of each instance is crucial in accurate layout estimation. Unlike existing works that overlook rotation initialization, we propose to estimate rotation based on feature similarity between rendered images and the 2D instance $\hat{I}_{i}$ in the reference image.
Specifically, we render the coarse 3D instance $A_{i}$ from a set of $n$ camera poses with uniformly sampled azimuth and elevation, denoted as $\pi\in\mathbb{R}^{n\times 2}=\{(e_{j},a_{j})\}_{j=1}^{n}$.
We then extract the features of 2D instance in the reference images and rendered images as $f_{\hat{I}_{i}}$ and $F=\{f_{1},f_{2},...,f_{n}\}$ from DINOv2 (Oquab et al., ), respectively.
Finally, we calculate the cosine similarity and choose the best rotation $r_{i}$:

$$ $\displaystyle(e_{b},a_{b})=\arg\max_{(e,a)}(cos(f_{\hat{I}_{i}},F)),$ (4) $$

where $(e_{b},a_{b})$ is elevation and azimuth of the best rotation $r_{i}$.
As a result, our reconstruction-based initialization strategy provides a more accurate initial state for the final compositional 3D, which can benefit the subsequent refinement stage.

### 3.2 Disentangled Refinement Stage

Previous works (Zhou et al., ; Li et al., ) have attempted to optimize 3D scenes through local and global guidance. However, they often suffer from excessively long optimization times and the need for additional guidance.
Instead, we disentangle the refinement process
into two consecutive steps: a collision-aware layout refinement step and an instance-wise refinement step.

#### 3.2.1 Collision-aware Layout Refinement

The Coarse 3D Generation Stage (Sec. 3.1) has provided a relatively precise initial 3D layout, however, we empirically observe that
it is still insufficient to create a visually appealing composition.
Thus we propose a layout refinement step for further improvement.
First, we adopt the spatial-aware SDS Loss (SSDS) as proposed in  (Chen et al., ):

$$ $\displaystyle\nabla_{\theta}\mathcal{L}_{ssds}(\phi^{\ast},x)=\mathbb{E}_{t, \epsilon}[w(t)(\hat{\epsilon}_{\phi^{\ast}}(x_{t};Y_{B},t)-\epsilon)\frac{ \partial x}{\partial\theta}],$ (5) $$

In which $\theta,x$, $\phi^{\ast}$, $\hat{\epsilon}_{\phi^{\ast}}(x_{t};Y_{B},t)$ are the 3D representation, rendered image, attention map strengthened on spatial words, and the score function that predicts the sampled noise $\epsilon$ from the noised image $x_{t}$ with noise-level $t$ and text prompt $Y_{B}$.
However, the SSDS loss is inadequate for addressing significant collisions caused by occlusion. For example, in Fig. 8, the dog obscures the blue tie and consequently does not receive sufficient gradients during optimization.
To resolve this issue, we propose a feature-level reference loss that leverages the 2D reference image and a tolerant collision loss to handle object collisions more effectively.

Feature-level Reference Loss. To align the geometry information in the reference image with our 3D scene, the most intuitive method is to apply a pixel-level RGB loss to enforce the alignment.
Nevertheless, we notice that the post-processing and the reconstruction model will unavoidably introduce some distortions and artifacts, making the RGB loss less robust and functional.
Therefore, we propose a feature-level reference loss to stabilize the refinement process.
Specifically, we only keep the foreground instances in the reference image and obtain its feature from higher layers of the DINOv2 (Oquab et al., ) model:

$$ $\displaystyle f_{ref}=\texttt{DINO}(I_{ref}\odot(M_{1}\cup M_{2}\cup...\cup M_ {N})).$ (6) $$

The feature-level reference loss can be computed as the L2 distance of $f_{ref}$ and $f_{render}$, here $f_{render}$ is the extracted feature of the rendered image at the reference view:

$$ $\displaystyle\mathcal{L}_{feat}=\lambda_{f}\sum\left\|f_{ref}-f_{render}\right \|_{2}.$ (7) $$

Figure: Figure 4: A simple illustration of how to calculate the collision loss $\mathcal{L}_{col}$.
Refer to caption: x4.png

Tolerant Collision Loss. Inspired by physics-based simulation (Santesteban et al., ) and CG3D (Vilesov et al., ), we propose a tolerant collision loss to separate incorrectly overlapping instances while simultaneously allowing a certain degree of collision to model natural object interactions.
As illustrated in Fig. 4, gaussians for two interacting instances are denoted as $P_{1}=\{p^{1}_{1},p^{1}_{2},...,p^{1}_{K_{1}}\}$ and $P_{2}=\{p^{2}_{1},p^{2}_{2},...,p^{2}_{K_{2}}\}$, where $p_{k}$ is the coordinate of the $k_{th}$ gaussian.
The $K_{1}$ and $K_{2}$ are the numbers of gaussians in the two instances, respectively.
In order to get a rough estimation of each instance,
we first compute the mean coordinate of each instance denoted as $p_{mean}^{1}$ and $p_{mean}^{2}$.
Then, we define mean sparsity $R$ as the mean distance of each point in $P_{1}$ to $p^{1}_{mean}$, which can also be interpreted as calculating the mass distribution of each instance:

$$ $R=\frac{1}{K}\sum_{k=1}^{K}\left\|p^{1}_{k}-p^{1}_{mean}\right\|_{2}.$ (8) $$

Next, we calculate distances from all the points in $P_{2}$ to $p_{mean}^{1}$ and subtract mean sparsity $R$, which can be regarded as the average distance between the distributions of two instances. The tolerant collision loss $\mathcal{L}_{col}$ is computed as:

$$ $\displaystyle\mathcal{L}_{col}=\lambda_{c}\sum_{k=1}^{K_{2}}ReLU(\mathcal{D}) \quad\mbox{where}\ \mathcal{D}=\{R-\left\|p^{2}_{k}-p_{mean}^{1}\right\|_{2},k =1,2,...,K_{2}\}.$ (9) $$

Here $ReLU$ is the rectified linear unit.
Intuitively, this loss penalizes gaussians that are too close to the centers of other instances, while allowing specific distances to model intersection between instances, making the optimization more robust and effectively addressing the overlapping problem.

Our full collision-aware layout loss is defined as a weighted combination of the three loss terms:

$$ $\displaystyle\mathcal{L}_{layout}=\mathcal{L}_{ssds}+\lambda_{f}\mathcal{L}_{ feat}+\lambda_{c}\mathcal{L}_{col}.$ (10) $$

#### 3.2.2 Instance-wise Refinement

Since the LGM can only produce fixed number of 3D gaussians, the reconstructed 3D instances are often in low quality with holes and broken textures.
Next, we discuss refining these coarse 3D instances to enhance its geometry and texture quality.
The conventional refinement strategies adopted by other optimization-based methods (Wang et al., ; Yi et al., ) are not efficient enough and often result in complete deformations of the shape of coarse 3D instances due to the excessive noise added to the rendered view, which aggravates the degeneration of geometry and results in drastic changes.
To address this issue, we propose an adjustable timestep sampling strategy to preserve the geometry details and retain the primary interactions of the coarse 3D instances.
Specifically, we lower the upper bound of timestep sampling’s range in the earlier phase and increase it after specific iterations when the geometry details have been well formed.
Formally, the gradient of instance refinement SDS loss $\mathcal{\hat{L}}_{sds}$ is computed as follows:

$$ $\displaystyle\nabla_{\theta}\mathcal{\hat{L}}_{sds}(\phi,x)=\mathbb{E}_{\hat{t },\epsilon}[w(\hat{t})(\hat{\epsilon}_{\phi}(x_{\hat{t}};y_{i},\hat{t})- \epsilon)\frac{\partial x}{\partial\theta}],$ (11) $$

where $\hat{t}$ is the adjusted timestep.
Intuitively, geometry is most likely to deviate during earlier refinement phase but should remain mostly unchanged in the rest of the process. As a result, our minor modification prevents the basic geometry from deviating significantly from its initial state.
Moreover, we noticed that the normal of coarse 3D instances might be rough and contain artifacts, which can harm their visual quality. So we combine an additional normal smooth loss $\mathcal{L}_{smooth}$ and total variation (Rudin & Osher, ) (TV) regularization terms (denoted as $\mathcal{L}_{TV}^{d}$ and $\mathcal{L}_{TV}^{n}$) to encourage better geometry of 3D instances. The full loss for instance-wise refinement can be represented as:

$$ $\displaystyle\mathcal{L}_{instance}=\mathcal{\hat{L}}_{sds}+\lambda_{s} \mathcal{L}_{smooth}+\lambda_{tv}(\mathcal{L}_{TV}^{d}+\mathcal{L}_{TV}^{n}).$ (12) $$

It is worth noting that when we extend the iterations for refinement (marked as extended refinement), we can produce comparable or even better single-object generation results compared to other optimization-based methods (Liang et al., ; Wang et al., ), as shown in Fig. 6.

Interactive Editing.
Besides 3D scene generation, users can customize the per-instance text prompt $\hat{y}_{i}$ in the instance-wise refinement step for interactive editing.
Specifically, we first generate a 3D scene with our coarse 3D generation stage and then stylize each 3D instance in the refinement step, with users’ custom text prompts $\hat{y}_{i}$.
This editing application can open up new possibilities for generating extra-complex compositional 3D scenes, with each instance having its attribute and style.

## 4 Experiments

### 4.1 Implementation Details

In the Coarse 3D Generation Stage, we render the 3D instances at 10-degree intervals when estimating the coarse rotation $r_{i}$. For the Disentangled Refinement Stage, in the collision-aware layout refinement step, we use the original hyper-parameter settings for the SSDS loss $\mathcal{L}{ssds}$ and optimize the layout for 400 iterations, with $\lambda_{f}$ and $\lambda_{c}$ set to 10.0 and 0.2, respectively. In the instance-wise refinement step, we utilize DeepFloyd (Shonenko et al., ) guidance with a total of 1500 iterations following the short refinement strategy (see Fig. 5). We set the timestep range to [0.10, 0.50] from step 0 to 800, and [0.02, 0.75] from step 800 to 1500. The parameters $\lambda_{s}$ and $\lambda_{tv}$ are set to 1.0 and 0.2, respectively. The extended instance-wise refinement strategy and additional implementation details are provided in Appendix B.

Figure: Figure 5: Main results and comparison with other text-to-3D methods on our Compo20 validation set. Layout-Your-3D can generate compositional 3D scenes with higher quality and more reasonable 3D layouts. Note that the first two rows of our results are generated with LLM-grounded 2D layouts, and the last two are generated with user-given 2D layouts.
Refer to caption: x5.png

### 4.2 Main Results

Validation Set. To better validate Layout-Your-3D’s ability in compositional 3D generation, we construct a validation set naming Compo20 for evaluation. The Compo20 consists of 20 compositional text prompts, each containing two or more instances with specific interactions.
For each text prompt, we have a user to manually provide a 2D layout and also automatically generate a 2D layout using LLM-grounded Diffusion (Lian et al., ).

Comparison methods. We compare Layout-Your-3D with several text-to-3D generation methods, i.e., DreamFusion (Poole et al., ), ProlificDreamer (Wang et al., ), GaussianDreamer (Yi et al., ) and LucidDreamer (Liang et al., ).
Since our method utilizes an additional 2D layout as a condition,
we provide results by both the user-given and LLM-grounded 2D layout. Moreover, we present a comparison of single 3D object generation in Fig. 6.

Qualitative Comparison.
As shown in Fig. 5, our method can generate compositional 3D scenes with higher quality and better understand the text prompts. In contrast, other methods often struggle to comprehend spatial relationships and semantic attributes in text prompts,
leading to poor quality
and the Janus Problem. These results
highlight
Layout-Your-3D’s strength in compositional 3D generation. More visualization results are
provided
in the appendix  C.

**Table 1: Quantitative results of Layout-Your-3D and other text-to-3d methods on our proposed Compo20 validation set. Note that we adopt the short instance-wise refinement setting here for fair comparison. The best and second best results are bold and underlined, respectively.**
| Method | Time cost $\downarrow$ | CLIP-Score $\uparrow$ | BLIP-VQA Score $\uparrow$ | mGPT-CoT $\uparrow$ |
| --- | --- | --- | --- | --- |
| DreamFusion | 1 hour | 17.98% | 23.44% | 31.08% |
| ProlificDreamer | 4 hours | 24.93% | 42.14% | 21.36% |
| GaussianDreamer | 15 mins | 22.21% | 28.74% | 25.91% |
| LucidDreamer | 1 hour | 22.36% | 34.11% | 34.84% |
| Layout-Your-3D | 12 mins | 23.26% | 53.51% | 50.44% |
| Layout-Your-3D (LLM-grounded) | 22.83% | 52.26% | 48.80% |  |

**Table 2: User study on different text-to-3D methods in terms of text-alignment, quality and rationality. The best and second best results are shown in bold and underlined.**
| Method | Text-Alignment | Quality | Rationality |
| --- | --- | --- | --- |
| DreamFusion | 3.36 | 5.48 | 3.79 |
| ProlificDreamer | 7.34 | 7.26 | 6.91 |
| GaussianDreamer | 4.22 | 6.61 | 4.72 |
| LucidDreamer | 5.61 | 7.72 | 6.38 |
| Ours | 9.08 | 8.44 | 8.72 |
| Ours(LLM-grounded) | 8.68 | 7.93 | 8.14 |

Quantitative Comparison. We conduct quantitative comparisons on the $Compo20$ validation set, presenting results adopted by the short refinement strategy. Since CLIP-Score (Radford et al., ) is not capable of accurately measuring the semantic correspondences (Huang et al., ), we also employ other metrics, BLIP-VQA and mGPT-CoT (Huang et al., ), to make a fine-grained assessment of the text-3D alignment. As shown in Tab. 1, our Layout-Your-3D outperforms other methods, demonstrating the superiority of our method on quantitative evaluation.

Figure: Figure 6: Comparison on the quality of single 3D instance generation. We provide both the short and extended strategy. When lengthening the optimization process, our Layout-Your-3D can generate comparable or even better results than SOTA text-to-3D methods.
Refer to caption: x6.png

Figure: Figure 7: Illustration of user given and LLM-grounded 2D layout generated reference images. We also visualized our coarse 3D generation and compared it with ComboVerse.
Refer to caption: x7.png

User Study. In addition to qualitative and quantitative comparisons, we conduct a user study to evaluate the proposed method further.
Participants
are asked to
rate text alignment, quality, and rationality on a scale ranging from 1 to 10.
Concretely,
users are shown 3D results and corresponding text prompts in a shuffled order and then asked to give ratings. We collect 500 responses from 100 users.
In Tab. 2, the overall ratings indicate a clear preference of users for our method.

### 4.3 Ablation Studies

3D layout initialization. We conduct an ablation study on the 3D layout initialization strategy by comparing the generated coarse scenes. As illustrated in Fig. 8, compared to ComboVerse (Chen et al., ), our initialization provides a more accurate starting point for subsequent refinement. These results demonstrate the effectiveness of our initialization strategy.

Collision-aware Layout Refinement. We analyze the function of various loss terms in the layout refinement step by using a toy example. When different 3D instances have major intersections, the overlapped instances tend to be concealed in the rendered images, thus hindering the optimization of diffusion-guided loss.
Instead,
our tolerant collision loss $\mathcal{L}_{col}$ further optimizes the geometry itself and $\mathcal{L}_{feat}$ encourages the alignment of spatial and high-level semantic information between the reference image and rendered reference view. As shown in Fig. 8 and Tab. 9, our layout loss $\mathcal{L}_{layout}$ can effectively address this issue, thus resulting in
more coherent 3D layouts.

Instance-wise Refinement. We conduct an ablation study on the influence of our timestep adjustment given examples with different settings. As shown on the left side of Fig. 9, timestep adjustment strategies
applied by other methods often result in inferior performances and would introduce
artifacts in the appearance.
On the right side of Fig. 9,
we visualize the effects of the TV losses $\mathcal{L}_{TV}^{d},\mathcal{L}_{TV}^{n}$ and normal smooth loss $\mathcal{L}_{smooth}$
By incorporating these two additional loss terms, Layout-Your-3D can
achieve better geometry consistency and quality.
We note that all the analyses above are based on a short instance-wise refinement strategy.

Figure: Figure 9: Ablation studies on timestep sampling and additional loss items in instance-wise refinement step.
Refer to caption: x9.png

### 4.4 Downstream Applications

Figure: (a) The extended refinement strategy.
Refer to caption: x10.png

Figure: Figure 11: A simple illustration of the object insertion application.
Refer to caption: x12.png

Instance customization. We present 3D instances refined with custom text prompts in Fig. 10. Our method can guide the refinement process to
generate plausible and coherent results that are more aligned with text prompts.
Thus, Layout-Your-3D allows for the initial generation of a coarse 3D scene, followed by the controlled adjustment of specific attributes across different instances. As a result, our method can produce stylized and complex outputs with better flexibility.

Object insertion.
Since 3D results are closely related to the reference image, incorporating object insertion into our pipeline would be straightforward.
As illustrated in Fig. 11, the first and second rows
show the edited reference images and corresponding
3D results.
We can see that
our method can insert objects with iterative editing, helping users customize their desired 3D content.

## 5 Conclusion

In this paper, we propose Layout-Your-3D, a method designed for highly controllable and precise compositional 3D generation. Our method addresses the issue of compositional 3D generation in a coarse-to-fine manner. First, we initialize the rough asset with our Coarse 3D Generation Stage. Then, we propose a Disentangled Refinement Stage. By refining the layout and instances separately, Layout-Your-3D can achieve more flexible and high-quality compositional 3D generation with a minimal cost of time. Through massive experiments and comparisons, we believe Layout-Your-3D indicates an essential step toward more comprehensive and mature 3D generation.

## Appendix A Discussions

### A.1 Limitations & Future work

Since Layout-Your-3D relies on reference images to generate compositional 3D assets, it would need deliberately designed 2D layouts to generate extra-complex 3D scene, e.g., having more than 10 instances in the asset. What’s more, as we observed, the SSDS loss, proposed in ComboVerse, is not capable of harmonizing interactions with too many instances involved in.

Thus, in our future work, we will work on deliberately designed modules to achieve more mature text-to-3d layout refinement and generation, without constraints on number of instances or interactions. Also, the application in scene generation of our method is a promising direction, we can extend our method in order to improve the efficiency and controllability of Text-to-3D scene generation.

Figure: Figure 12: A simple illustration of failure cases.
Refer to caption: x13.png

### A.2 Failure cases

We provide a simple failure case in Fig 12. The text prompt is ’A lego man riding a horse, the horse is wearing a blue hat’, in which we add uncommon interactions between instances, i.e., ’blue hat’ and ’horse’. As illustrated in the figure, for text prompts that are extremely complicated or counter-intuitive, the diffusion models are not capable of comprehending the spatial relationships correctly and hence misdirect the optimization process.

### A.3 More downstream applications

Except for instance-wise customization and object insertion, our Layout-Your-3D can also support the Conversational Iterative Editing as proposed in GALA3D. Once obtained the final 3D layouts of the resulting compositional 3D, we can easily edit the position and rotation of each instance, together with its attributes and style, through iterative conversation. Since the generated 3D asset is essentially comprised of different 3D instances, it has a higher degree of freedom and can achieve better quality. This advantage helps Layout-Your-3D stand out among different Text-to-3D generation methods.

### A.4 Discussion on concurrent methods

GroundedDreamer (Li et al., ) and COMOGen (Sun et al., ) are the two most related and concurrent works. Here we give a comprehensive analysis and a comparison between these two methods and Layout-Your-3D. Since they have not released official implementations yet, we simply use the results from the original paper for comparison, and analyze the possible advantages and disadvantages.

Figure: Figure 13: Comparison with COMOGen, results are generated with 2D layouts provided in the original paper of COMOGen.
Refer to caption: x14.png

GroundedDreamer’s core concept is to marry Attend-and-Excite with MVDream (Shi et al., ) to generate compositional and consistent multi-view images, then lift them to 3D assets through an optimization-based method.
Though it shows some surprising results and relatively high quality, but due to the limited compositional 3D data used in the training process, the limitations of MVDream still exists, i.e., lacking in prior knowledge for the generation of multiple instances. Also, the pure optimization-based paradigm makes GroundedDreamer low in efficiency, taking nearly 2 hours to generate one single asset. In comparison, Layout-Your-3D needs only about ten minutes (short refinement) or about one hour (extended refinement) to generate one asset. If given more GPU cards, the time cost could be reduced to several minutes and half an hour. Furthermore, since Layout-Your-3D primarily generates individual 3D instances and subsequently combines them, which mitigates the domain gap during the generation process, it effectively helps addressing compositional 3D generation task at its source.

COMOGen takes the same 2D layout setting as adopted in Layout-Your-3D to indicate ideal regions of different instances in the text prompts.
However, different from Layout-Your-3D, COMOGen first adopted a Layout-SDS loss to distill the spatial layout knowledge, then uses a Multi-view Control module to ensure multi-view consistency and alleviate Janus Problem, finally proposed a 3D content enhancement module to generate realistic and diverse 3D samples. The Layout-SDS loss could provide a rough estimation of the ideal 3D content, in the following Multi-view Control module, COMOGen adopted a Zero-123 (Liu et al., ) multi-view diffusion model to ensure the geometry consistency. But as we mentioned above, current multi-view diffusion models are still subject to the domain gap, thus making it hard to generalize to compositional scenarios, damaging the effectiveness of COMOGen. Moreover, though not presented, the time cost for COMOGen might not be comparable to our Layout-Your-3D, which only costs about 10 minutes to generate a reasonable and plausible compositional 3D scene, as displayed in Fig. 13. Additionally, COMOGen did not show detailed and high quality single object generation results, which constrains the application scenarios of it.

Another line of works, i.e., GALA3D and DreamScene also have similar setting as our Layout-Your-3D.
Nonetheless, these works have their own disadvantages. DreamScene is essentially composing different objects into one predefined scene with no specific interaction modeling process during generation, thus will make the resulting scene less realistic and consistent.
GALA3D addresses this issue by incorporating 3D layout-conditioned diffusion models as an implicit control over the interactions between objects. However, the produced 3D scene has no background and the quality of objects are relatively inferior, though processed through tedious optimization process.
Since the input settings and generated 3D assets are not strictly aligned with ours, we do not make direct comparisons between these methods and our Layout-Your-3D.

### A.5 Other reconstruction models

In the coarse 3D generation stage, we could exchange for any 3DGS-based Image-to-3D model for the reconstruction of each instance, e.g., GRM (Xu et al., ), Point-to-Gaussian (Lu et al., ). What’s more, it is also applicable to replace 3DGS with other 3D representations, like mesh or tetrahedron for advanced or specific usage. Though it will be slightly different to optimize coarse 3D instances in other representation, there are substitutions for the refinement strategies in our pipeline. Also, our final 3DGS-based asset could be converted into mesh with the efficient mesh extraction method proposed in DreamGaussian (Tang et al., ) or advanced mesh extraction methods like SuGaR (Guédon & Lepetit, ).

## Appendix B Experiment Settings

### B.1 Validation Set

We analyze the core design of our proposed $Comp20$ validation set in this part.
In order to provide a comprehensive measurement for the compositional 3D generation task, we divide the text prompts in our $Comp20$ validation set into several scenarios, covering a wide range of cases that can meet the requirements by users and real world applications:

- •
Multiple instance without interactions,
e.g., ’A black bottle, with a white bottle on one side and a blue bottle on the other’.
- •
Multiple instances with interactions,
e.g., ’An astronaut sitting on a sofa’.
- •
Attribute control of instances,
e.g., ’A blue toy robot and a red toy robot playing basketball’.
- •
Small components in a larger main body,
e.g., ’A furry fox’s head, with a cupcake on its top, wearing red sunglasses and have a lego block on the nose’.

In accordance with the requirements for compositional 3D generation, each text prompt contains 2 to 5 instances in the compositional 3D asset. This intentional design of our $Comp20$ validation set allows for a comprehensive evaluation on text-to-3D generation methods. The complete prompt list, together with the corresponding 2D layouts (including both user-provided and LLM-generated layouts), are presented in Tab. C.3. Note that in the default experiments we generate reference images with a 512 * 512 resolution, and the 2D layouts are presented in the form of $[x_{1},y_{1},x_{2},y_{2}]$, with each element has a value from range 0 to 512.

### B.2 More implementation details

We give a more detailed illustration of our experiment settings in this section.
We implement Layout-Your-3D based on threestudio (Guo et al., ) for a more integrated and readable system. We conduct all of our experiments based on NVIDIA RTX6000 GPUs. It is worth noting that the baseline methods, i.e., DreamFusion and ProlificDreamer are implemented on threestudio, thus might result in minor differences with the original implementation results.

**Table 3: The detailed list of the short (efficient) instance-wise refinement step.**
| Hyper-parameters | value |
| --- | --- |
| batch_size | 1 |
| resolution | [256, 512] |
| resolution milestones | 800 |
| position_lr | [0, 0.0005, 0.00005, 500] |
| scale_lr | 0.005 |
| feature_lr | 0.01 |
| opacity_lr | 0.01 |
| rotation_lr | 0.001 |
| densify/prune interval | 100 |
| densify/prune start iter | 300 |
| densify/prune until iter | 900 |
| total_iter | 1500 |

**Table 4: The detailed list of the extended (high quality) instance-wise refinement step.**
| Hyper-parameters | value |
| --- | --- |
| batch_size | 4 |
| resolution | 512 |
| position_lr | [0, 0.0005, 0.00002, 1000] |
| scale_lr | 0.005 |
| feature_lr | [0, 0.01, 0.005, 2000] |
| opacity_lr | 0.05 |
| rotation_lr | 0.005 |
| densify/prune interval | 200 |
| densify/prune start iter | 400 |
| densify/prune until iter | 1600 |
| total_iter | 2000 |

In the coarse 3D generation stage, to guide the layout-conditioned text-to-image generation to generate canonical views, we add an additional text prompt ’3d scene, front view’ to the overall text prompt $Y_{B}$. This simple measure helps Image-to-3D reconstruction models obtain better results in the coarse 3D generation stage since multi-view diffusion models, e.g., MVDream, perform better when given images with canonical views. When performing segment & inpaint, we prompt the SAM (Kirillov et al., ) with 2D bounding box of each instance to indicate the foreground region and the center points of all other instance’s boxes are treated as negative prompts to avoid segmenting multiple irrelevant instances all at once. Moreover, for the depth estimation part, since the output prediction from GeoWizard is not metric depth, we normalize the predicted depth values and centralize the 3D location of the final 3D assets. In the instance-wise refinement step, we provide a more concrete setting for both the extended and short refinement and layout refinement steps.
All the hyper-parameters are listed in Tab. 3, Tab. 4, and Tab. 5, including learning rates for attributes of the gaussian splatting representation and different elements in layouts. Note that for the extended instance-wise refinement strategy we use Stable Diffusion (Rombach et al., ) guidance instead of Deep Floyd, to boost the quality of each instance.
For the layout learning part, we assume that the input 2D layouts are accurate and reasonable enough. Since two of the spatial coordinates in the transition $t_{i}$ depend mainly on the given 2D layouts, we only give a very small learning rate to these two elements.

**Table 5: The detailed hyper-parameter list of the collision-aware layout refinement step.**
| Hyper-parameters | value |
| --- | --- |
| quaternion_lr | 0.0001 |
| transition_x_lr | 0.00002 |
| transition_y_lr | 0.00002 |
| transition_z_lr | 0.02 |
| total_iter | 400 |

### B.3 Additional Ablation studies

We conduct more ablation studies to validate the effectiveness of our model selection.
First, we compare the visual foundation model used when extracting features for coarse rotation estimation. We choose DINOv2 (Oquab et al., ) to estimate the rotation $r_{i}$ since DINOv2 can provide higher-level geometry and semantic information. In comparison, CLIP (Radford et al., ) is another model we adopted, but since it is not specifically designed to capture geometry information in images, it causes visible errors in the rotation estimation.
Also, we give a detailed time cost of Layout-Your-3D in Tab. 6, including the time cost for each part and different settings, measured on one single RTX6000 GPU.

**Table 6: Time cost for each part of Layout-Your-3D, including both the short and extended refinement strategy.**
| Designed module | Time cost |
| --- | --- |
| Coarse 3D gen. | + $\sim$1 minute |
| Layout ref. | + $\sim$2 minutes |
| Ins. ref. (short/extended) | + $\sim$3 / 35 minutes |
| Overall (short) | $\sim$12 minutes (3 instances) |

Figure: Figure 14: 3D assets generated with out coarse 3d generation stage.
Refer to caption: x15.png

Moreover, to prove that it is essential to process the layout refinement step and instance-wise refinement step, we present some toy results after the Coarse 3D Generation stage in Fig. 14. These examples, though show relatively reasonable geometry shapes and spatial relationships, are still not satisfying enough for further usage.
Thus we claim that it is necessary for us to develop the Disentangled Refinement Stage to obtain better results.

## Appendix C More Qualitative results

### C.1 Single Object generation

In Fig. 15, we visualize more single object generation results produced by our extended instance-wise refinement step. These results demonstrate the capability of our Layout-Your-3D in the field of text-to-single 3D object generation.
The figure shows that our refinement process could retain the primary geometry shapes and add fine-grained details. In about 25 to 30 minutes, we can generate 3D object of extra high quality with our Layout-Your-3D, comparable, or even much better than previous or current SOTA methods. These results proves the versatility of our method.

Figure: Figure 15: More results on the single object generation with our extended refinement strategy.
Refer to caption: x16.png

### C.2 customization

To validate the effectiveness and generalizability of our method, we give more examples in Fig. 16, both the short and extended refinement strategy can bring customized attributes and easily deform the style of 3D assets, thus enabling flexible and controllable generation.

### C.3 Compositional generation

We present more generated compositional 3D assets by our Layout-Your-3D in Fig. 17, here we only show results generated with our short instance-wise refinement setting ($\sim$12 mins in all). We can see that Layout-Your-3D can generate compositional assets, in which 3D instances can be in all ’granularity’, from smaller ones like ’blue sunglasses’, to bigger ones like ’A box’, ’A toy pyramid’. This can be another strong evidence, showcasing that our method has great advantages in the aspect of controlling 3D generation.

Figure: (a) The extended refinement strategy.
Refer to caption: x17.png

Figure: Figure 17: More visualization results generated with our $Comp20$ text prompts and layouts.
Refer to caption: x19.png

## Peer Reviews

confidence: 2
contribution: 3
initial_rating: 5
presentation: 2
questions: Within the collision loss, is there any way to take into account the shape (covariance) of each gaussian in 3D space? My understanding is that it's only using the positions of the the Gaussians right now. Have you observed this to be a limitation?
soundness: 3
strengths: The papers shows a potential way to use many existing models together to generate compositional 3D scenes. The results show that the method produces reasonable visual reconstructions and geometries. The method seems well motivated, and the paper is decently written.
summary: - They enforce structural control of a 3D scene using a 2D layout. Their optimization process avoids collisions.
- Their method works by first generating a 2D image based on text description, then use SAM to segment each isntnace in the reference iamge, with in-painting done on occluded parts.
- They then use a large mulit-view gausian model to generate coarse 3D instances of objects within the scene. 
- Each 3D instance is then arranged into 3D space by lifting the bounding boxes into 3D, done by optimizing the rotation according to similarity in the DINO space of the rendered images of the rotated instances.
- final two stages refine the layout according to collisions and refines the instance representations.
weaknesses: My concern is that this work lacks fundamental novelty, in that prior works like Compositional 3D Scene Generation using Locally Conditioned Diffusion (https://arxiv.org/abs/2303.12218), Towards Text-guided 3D Scene Composition (https://arxiv.org/pdf/2312.08885) and CG3D: Compositional Generation for Text-to-3D via Gaussian Splatting (https://arxiv.org/pdf/2311.17907) all present various ways to create 3D scenes using implicit 3D represntations in a compositional way.

There does not seem to be any comparisons to these works, or mention of them in the related works (though CG3D was mentioned in the collisions section, but not compared with in the experiments section). A comparison to contextualize this work seems very important, but lacking.

---

confidence: 3
contribution: 2
initial_rating: 6
presentation: 3
questions: 1. For clarification, when inputing the text prompts, what's the prompt for LLM to generate 2D layouts? Also, do you mark the object in bracket like text prompts in figure 5 or LLM is supposed to decide which objects should be created?

2. The proposed method uses many off-the-shelf algorithm and components like LLM generating 2D layouts, reference image generation, pose alignment. I am curious about the successful ratio of the whole process, and some potential failure cases.
soundness: 3
strengths: 1. This paper proposes an interesting pipeline by combining several (off-the-shelf) components in a novel way. Getting 2D reference images from layout and trying to reconstructing 3D models from them leads a interesting 3D reconstruction task, and this paper provides a reasonable solution for it, by utilizing the inpainting, DINO feature distance and off-the-shelf image generation model.
2. This paper involves several intesting losses like collision-tolerant loss, feature-level loss, and SDS with adjusted timesteps. 
3. Results are good. and the paper is easy to read and follow.
summary: This paper tries to address the problem of lay-out controlled 3D generation task. 
Interestingly, it relies on 2D layout instead of 3D layouts, and the 2D layout could be generated by LLM given a text prompt. 
With this 2D Layout, it first generate a reference image with an off-the-shelf 2D-layout image generation model, and then try to reconstruct the 3D model and layout from the reference image. 
This generated 3D model is coarse and may not very accurate, and this paper refines the generated model with SDS and feature loss, as well as collision-tolerant loss.
This paper proposes a very interesting pipeline using many off-the-shelf components, and it achieves good results
weaknesses: 1. For baselines, it would be more convincing to seriously compare against models with 3D layout control like ComboVerse or [Compositional 3D Scene Generation using Locally Conditioned Diffusion]. The current paper only shows a single scene comparison against ComboVerse in Figure 7, which is insufficient to demonstrate the advantages of using 2D layout and the proposed method. More rigorous evaluation, such as those shown in Tables 1 and 2, along with a user study, would be needed for such comparisons.

2. After reconstructing individual 3D models from 2D images, it is critical to align the pose (6D pose) of the reconstructed 3D model with the image appearance. The proposed method achieves this by randomly sampling many views of the 3D model and comparing them against the reference image by computing the DINO feature distance. However, I am curious about the accuracy of this method, as the rendered image could differ significantly from the reference image even with perfect poses if the distance (object scale) is not accurate. It would be more convincing to show results demonstrating the accuracy of this method.

3. Regarding the reconstruction of objects from reference images, how is the mask/segmentation obtained for each object? Is this process automated, or does it require manual intervention? This question also appears in the following inpainting and pose alignment sections. Do we need to manually specify which object to inpaint or segment during this process or not?

---

confidence: 3
contribution: 2
initial_rating: 3
presentation: 2
questions: 1. why do the bird, bear, and Bagle look different in each example?
2. what happens if the generated 2D images don't contain all the specified objects? it does happen a lot with the generation of complex scenes. like what is shown in [2]
soundness: 3
strengths: generating complex scenes is a difficult task today.
this work tries to solve this task, into stages to deal with each part separately.
summary: the work trying to generate a 3D complex scene based on a textual prompt. they start by generating a 2D image of the scene using a 2D text-to-image generator, they extract 2D boxes for each instance. 
Next, they create each object in a 3D instance and position them according to the boxes.
the final step is SDS which fine-tune the positions for natural placement, specifically in a collision-free manner.

[1] Yang, Yue, et al. "Holodeck: Language guided generation of 3d embodied ai environments." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024
[2] Rahamim, Ohad, et al. "Lay-A-Scene: Personalized 3D Object Arrangement Using Text-to-Image Priors." arXiv preprint arXiv:2406.00687 (2024).
weaknesses: 1. no novelty in the approach, you concatenated known strategies while trying to solve this problem.
2. can it be solved using previous methods like [1]
3. Besides the bear and the book, I can't see any interaction between objects. can you propose how to create interactions? (like the bird drinking the beer)

---

confidence: 3
contribution: 2
initial_rating: 3
presentation: 2
questions: - My main concerns are discussed in the Weaknesses section. I can see from the paper that the authors have put a lot of effort into creating a valuable contribution, and despite my doubts about the contribution, experimental results, and design of the method, I still rate the paper as borderline reject (5). I believe the improvements needed cannot realistically be achieved during the rebuttal stage, and this may be due to time constraints in preparing the submission. I hope the authors can improve the method’s effectiveness and rewrite the contribution section to better highlight their approach, so that the paper can be successfully accepted at top conferences like **CVPR, ICML, or ICCV**.

- One final note: while I don't mean to discourage the authors, I have limited expectations for changes in the rebuttal, as the scope of improvement I am suggesting is quite large. Therefore, **it might not be necessary to spend too much time adding experiments during rebuttal for me**. I wish the authors all the best with their research.
soundness: 3
strengths: - The paper is rich in content, and the proposed method successfully accomplishes the task set by its motivation.
- The writing is clear, making it easy to understand the authors' research methods.
summary: This paper uses a 2D layout as guidance and proposes a Collision-aware Layout Refinement to optimize 3D layouts, effectively achieving compositional 3D generation with plausible object interactions.
weaknesses: - Over-reliance on existing methods: The engineering aspect of the method is too pronounced. Aside from the newly introduced loss functions, much of the technique is a stack of existing methods, including SAM, ControlNet-tile, LGM, GeoWizard, DINO, etc. The use of LLM to assist in generating the 2D layout is also an existing strategy. Such extensive reliance on pre-existing methods can overshadow the key contribution of the paper. Additionally, I would like to ask: since LLMs are already in use, why not have the LLM or MLLM directly predict the 3D layout and then apply your Collision-aware Layout Refinement for 3D layout refinement? This would simplify the initial stage, reduce reliance on existing methods, and highlight the contribution of your technique, making the overall framework more elegant and concise.

- Limited contribution: The task goal overlaps significantly with many existing compositional 3D scene generation methods (GALA3D, graphdreamer, dreamscene， progress3D...), making the problem being addressed less novel. Furthermore, as mentioned earlier, many critical components are stacked from existing methods. Although the Feature-level Reference Loss and Tolerant Collision Loss are novel, their impact seems less substantial compared to the other parts, as seen in the visual results in Figure 8. As a suggestion for improvement, it might be better to focus on complex single-image-to-3D reconstruction instead of compositional 2D layout + text-to-3D generation. You could use multimodal LLMs to decompose objects from a complex, unclear image for reconstruction and then refine and combine them into a 3D scene. This shift could substantially enhance the novelty and contribution of the paper.

- Insufficient comparison in experiments: In the supplementary materials, methods like GALA3D and DreamScene are mentioned. Why aren’t there comparisons with compositional 3D scene generation methods in the main text, such as GALA3D, GraphDreamer, DreamScene, Progressive3D, or more recent works like The Scene Language? Even if comparing with the very recent papers isn't necessary, it would be more reasonable to compare your work with three or so methods that are highly relevant to your contribution, such as GALA3D, GraphDreamer, and DreamScene. This could also help improve the previous point about novelty. Additionally, based on my own experience, DreamFusion should be faster than LucidDreamer—on my computational resources, the timing is approximately 30 minutes vs. 50 minutes, differing from the 1 hour vs. 1 hour stated in your paper. Although a minor issue, it raises some doubts about the accuracy of the reported experimental results.

Meta-review: The paper tackles the problem of controllable 3D scene generation. The proposed pipeline starts with 2D layout generation and then progresses through 3D instance reconstruction and arrangement. Their key insight is to decompose the original complex problem into a series of simpler and modular tasks. Then, by exploiting state-of-the-art modules for each stage, the proposed method is able to produce plausible 3D reconstructions with reasonable scene layouts. On the negative side, the reviewers are concerned that similar pipelines have already existed in the past literature and there are not sufficient and comprehensive evaluations/comparisons. The reviews were on the fence. During the discussion phase, the authors have attempted (and managed) to address most of the concerns. While there is still a debate going on regarding whether the novelty is enough and whether the work is incremental (ie, switching from generating 3D layout from LLM to producing 2D layout and lift it to 3D), considering the fact the authors incorporated quite a bit of additional results and showcased superior performance during the rebuttal, the ACs agree that the paper is marginally above the acceptance bar and decide to recommend acceptance. The authors are strongly encouraged to incorporate the feedback from the reviewers into their final camera ready version.

Decision: Accept (Poster)