Layout-your-3D: Controllable and Precise 3D Generation with 2D Blueprint
Authors: Junwei Zhou
Xueting Li
Lu Qi
Ming-Hsuan Yang
Venue: ICLR 2025 Poster (2025)
Keywords: 3D generation
gaussian splatting
Text-to-3D
compositional asset generation
Primary area: generative models
TL;DR: Given a text prompt describing multiple objects and their spatial relationships, our method generates a 3D scene depicting these objects naturally interacting with one another.

Abstract: We present Layout-Your-3D, a framework that allows controllable and compositional 3D generation from text prompts. Existing text-to-3D methods often struggle to generate assets with plausible object interactions or require tedious optimization processes. To address these challenges, our approach leverages 2D layouts as a blueprint to facilitate precise and plausible control over 3D generation. Starting with a 2D layout provided by a user or generated from a text description, we first create a coarse 3D scene using a carefully designed initialization process based on efficient reconstruction models. To enforce coherent global 3D layouts and enhance the quality of instance appearances, we propose a collision-aware layout optimization process followed by instance-wise refinement. Experimental results demonstrate that Layout-Your-3D yields more reasonable and visually appealing compositional 3D assets while significantly reducing the time required for each prompt. Additionally, Layout-Your-3D can be easily applicable to downstream tasks, such as 3D editing and object insertion.

Published as a conference paper at ICLR 2025
LAYOUT-YOUR-3D: CONTROLLABLE AND PRECISE 3D GENERATION WITH 2D BLUEPRINT
Junwei Zhou'
Xueting Li
Lu Qi
Ming-Hsuan Yang
Huazhong University Science and Technology 2NVIDI
Wuhan University
IIntta360 Research
UC Merced
Yonsei University
Text prompt
3D instances (multi-view)
3D layout & compositional 3D
[A pigeon] having (a bagel] and [a beer]'
LLM generated
User given
J
2D layout
pigeon
beer
A bagel
Custom text prompts
blue origami pigeon'
bagel made of grass!
beer fire'
Figure 1: Layout-Your-3D (top). Layout-Your-3D Your-3D further enables editing each instance in the 3D scene with custom text prompts, achieving controllable and precise 3D generation (bottom).
Your-3D generates high-quality compositional 3D scenes with given 2D layouts
ABSTRACT
We present Layout- Your-3D, framework that allows controllable and composi- tional 3D generation from text prompts. Existing text-to-3D methods often strug- gle to generate assets with plausible object interactions or require tedious opti- mization processes. To address these challenges, approach leverages 2D lay- outs as a blueprint to facilitate precise and plausible control over 3D generation. Starting with 2D layout provided by tion, we first create a coarse 3D scene using a carefully designed initialization process based efficient reconstruction models. To enforce coherent global 3D layouts and enhance the quality of instance appearances, we propose a collision- aware layout optimization process followed by instance-wise refinement. Exper- imental results demonstrate that Layout-Your-31 yields more reasonable and vi- sually appealing compositional 3D assets while significantly reducing the time required for each prompt. Additionally, Layout-Your-31 Your-3D can ble downstream tasks, such as 3D editing and object insertion. Video and results can be viewed on our website: https:/ / /ccl ezwhy. gi thub. io/ layoutyour3d/
generated from text descrip-
be easily applica-
"Corresponding author

---

Published as a conference paper at ICLR 2025
1 INTRODUCTION
The recent years have witnessed significant advances in 3D content creation. By leveraging ad- vanced diffusion models through optimization (Poole et al., 2022) or training efficient reconstruc- tion models (Hong et al., 2024), numerous methods successfully synthesize high-quality 3D objects from text prompts or images. These approaches have facilitated various applications such as virtual reality (Chen et al., 2024a; Liu et al., 2024b), robotics (Huang et al., 2023b; Wan et al., 2024), etc.
Nevertheless, existing text-to-3D methods often struggle to synthesize plausible object interactions in compositional 3D asset generation. For instance, given the text prompt "a brown soft with a cush- ion, a teddy bear and a basketball on it", none of the state-of-the-an methods (Wang et al., 2023; Liang et al., 2024; Yi et al., 2024) shown in Fig. 5 (last row) correctly predict the plausible loca- tion and scale of the objects. This is because these optimization- based models create 3D assets by distilling prior knowledge from 2D diffusion models (Rombach et al., 2022). Consequently, they require extensive optimization time for each prompt and suffer from the same issues such as seman- tic confusion missing objects as the diffusion models. Meanwhile, ferd-forward methods (Tang et al., 2024a) produce 3D shapes instantly but often perform poorly in creating complex composi- tional cases due to the lack of compositional 3D objects in their training data (Deitke et al., 2023). Furthermore, none of these methods provide any user control over the 3D generation process. These limitations significantly restrict the application of text-to-3D methods real-world scenarios. Two lines of work aim to resolve these issues and generate plausible compositional 3D assets. First, several methods use various attention mechanisms (Li et al., 2024b) to enhance the alignment of synthesized 3D objects scenes with compositional text descriptions. However, these methods do not offer controllable signals such as layout. Another line of approaches focuses on text-to-3D scene generation (Zhou et al., 2024b) by first synthesizing each 3D instance within the scene, followed by a global layout refinement through Score Distillation techniques (Poole et al., 2022). While these schemes aim improve scene coherence, they require significantly more computational resources and generate less realistic 3D results than conventional text-to-31) object synthesis methods.
In this work, present Layout-Yourr Your-3D, efficient text-to-3D generation method for compo- sitional 3D scene synthesis with precise control. Specifically, given a text prompt describing few objects and their spatial relationships, we aim generate these objects and their natural interac- tions. We note that this is similar to the setting in (Sun et al., 2024) but differs from (Zhou et al., 2024b; Li et al., 2024a: Gao al., 2024), which focuses on synthesizing complete indoor/outdoor 3D scenes. Our key idea is to utilize 2D layout as guidance to produce target reference images for 3D scene generation, which not only allows precise user control but also produces more plausible object interactions. We start by collecting user- esor--poodded or L..M-generated 2D layout and predict a reference image depicting the desired instances and their interactions. Using this reference, we propose carefully designed process to create coarse 3D scenes with roughly reasonable layouts using reconstruction models (Tang et al., 2024a). While these feed-forward reconstruction models computationally lightweight, they often yield inferior 3D geometry and appearance. To address these limitations, we introduce a two-step disentangled refinement stage to enhance the quality of 3D instances and global layout. First, we develop collision-aware layout optimization to adjust the layouts of the objects to create a globally coherent and visually appealing composition. By incor- porating collision awareness, we prevent objects from severe intersections, resulting in realistic compositional layout. Second, we carry out instance- wise refinement to further improve the geom- etry and texture of individual objects, which also allows the customization of different instances as an added benefit.
We evaluate our method with a proposed validation set, extensive experiments, and ablations to showcase the ability of Your-3D in rapid and accurate 3D generation. In addition, we con- duct comparisons with current state-of-the-an text-to-3D methods, demonstrating the potential of Yoyout-Your-3D
Your-3D in compositional 3D generation, as well as customized design and editing. The contributions of this work are threefold:
We propose Layout- Your-D., practical approach for highly controllable compositional 3D generation, demonstrating superior performance compared to the baseline methods. We present reconstruction-brit based initialization process that efficiently produces reason- able coarse 3D scenes. We further enhance the texture, geometry, and global layout of the 3D scenes using carefully designed two-step disentangled refinement stage.
2

---

Published as a conference paper at ICLR 2025
We conduct comprehensive experiments assess the effectiveness of our proposed method against baselines. Our approach demonstrates superior qualitative and quantitative results while significantly reducing the time for generation (i.e., 12 minutes).
RELATED WORK
Text-to-3D generation. Significant advances have been made in this field thanks the rapid de- velopment of foundational diffusion models (Rombach et al., 2022). The DreamFusion (Poole al.. 2022) model enables the generation of 3D objects using powerful 2D diffusion models by introducing the Score Distillation Sampling (SDS). Numerous works further improve text-to-3D generation performance by developing various SDS methods (Yu et al., 2023; Ma al., 2024), leveraging advanced 3D representations such as DMTet (Chen et al., 2023) and 3D Gaussian Splat- ting (3DGS) (Chen et al., 2024e; Liang et al., 2024), or leveraging pre-trained normal and depth diffusion models (Qiu et al., 2024). For instance, ProlificDreamer (Wang et al., 2023) proposes particle-based variational framework that promotes the generation of more diverse and less saturated 3D objects, while GaussianDreamer (Yi et al., 2024) accelerates optimization by leveraging 3DGS. Since 3DGS enables fast generation speed with various initialization methods and tate-of-the-aat the-art generation performance, are building on it for 3D generation from text prompts.
Image-to-3D Reconstruction. Instead of using inherently ambiguous text descriptions, one line of works reconstructs 3D objects from or more input images. Numerous methods (Liu al., 2024a; Long et al., 2024; Shi et al., 2024) finetune 2D diffusion models with 3D-aware data to generate highly consistent multi-view images. Subsequent approaches, such as LRM (Hong al., 2024) and LGM (Tang et al., 2024a), further accelerate 3D reconstruction from single-view images through an efficient feed-forward process. As these reconstruction models are trained on synthetic, bongle-object-cemee
datasets, they often struggle to reconstruct plausible 3D assets that include multiple objects or produce realistic appearances for aat-of-diitribution input images. In this work, we leverage the efficient reconstruction models to produce initial coarse 3D scenes while addressing their limitations in compositional generation and the issue of unappealing appearances.
Compositional 2D and 3D generation. Generating reconstructing scenes with multiple objects presents significant challenges in 2D and 3D contexts. In the 2D domain, numerous models (Li et al., 2023; Xie et al., 2023; Zhou et al., 2024a) use 2D bounding box layouts as conditions to guide the generation process. These works perform well in controlling the number of instances and their attributes. Due to the additional depth dimension, generating compositional assets in the 3D domain faces even more significant challenges. Several image-to-3D reconstruction methods, such that by (Han et al., 2024; Chen et al., 2024b), reconstruct each object in the scene independently and subsequently optimize the global layouts by aligning them with reference images or utilizing additional guidance from diffusion models. Another line of work focuses on compositional text- to-3D generation. For example, COMOGen (Sun et al., 2024) adopts 2D layouts to guide the SDS process to provide accurate control over the generation process. GraphDreamer (Gao et al., 2024) establishes a graph to represent objects and their relations. GroundedDreamer (Li et al., 2024b) in- tegrates the core idea of Attend-and- Excite (Chefer et al., 2023) into MVDream (Shi et al., 2024) to facilitate the generation of compositional multi-view images from complex text prompts. Moreover, text-to-3D 3D scene generation methods (Zhou et al., 2024b; Li et al., 2024a; Po & Wetzstein, 2023; Zhang et al., 2024) that rely the score distillation technique often utilize 3D layouts provided by users (Bai et al., 2024) or generated from L.MSs to provide coarse compositional spatial information to construct the scene. However, these methods often require hours for generation and may pro- duce unreasonable object interactions or unrealistic appearances. In this work, we leverage efficient reconstruction models achieve high quality and efficiency, significantly advancing high-fidelity compositional 3D scene generation.
3 METHOD
Given a 2D layout and a text prompt. we aim to synthesize 3D scene, including multiple ob- jects and specific interactions aligned with the layout and text. Unlike existing text-to-3D methods, our approach aims to generate reasonable 3D geometry and texture for each object and produce a plausible layout with all objects interacting naturally (i.e., with reasonable location, scales, etc.).
3

---

Published as a conference paper at ICLR 2025
Coorse 3D Generation Stage
Compositional text prompt Yg 14 Figen] havveg boyel neery
aggoo
Individual Object Reconstruuttion
Layout guied 2D imge Generation
LLM User
10u. (t (% Instonce-wise iis Leyout
3D Layout Initialization
6 frorslation rotation
Disentongled Refinement Stage
Layout Refinemont
Refirement
(+N)
Nivol
Nosst
Diffusion Guidance
y
Luss
Lymosch
Reot
Iraftanee y/ 8t
Coans comprsed SD)
Relined 3D Layout
Tameet 3D asst
Figure 2: Overview of Layout-Your-3D. ation stage (green box, Sec. 3.1) generates coarse 3D instances along with roughly reasonable layouts. The disentangled refinement stage (see Sec. 3.2) then refines the 3D layout and enhances individual instance quality by leveraging collision-aware layout refinement (blue box) followed by an instance- wise refinement (yellow box).
Your-3D. Given a 2D layout and text prompt, our coarse 3D gener-
As shown in Fig. 2, we represent the 2D layout as a set of bounding boxes and their corresponding
instance names B = {(b1.11) V1), (b2, y2), (bN, uN)}, which can be extracted from the overall text prompt YB. Given these bounding boxes and instance names, we design a coarse 3D generation stage (see Sec. 3.1) that creates all 3D objects along with roughly reasonable 3D layout. To en- hance the visual quality and consistency of the layout, we introduce a disentangled refinement stage (see Sec. 3.2), which includes collision-aware layout refinement, followed by instance- wise refinement. Additionally, our method allows users to interactively edit each instance and the global layout by-product. In the following, we provide detailed discussions of these components.
3.1 COARSE 3D GENERATION STAGE
Given a set of bounding boxes with corresponding object names, the coarse 3D generation stage synthesizes each 3D object and arranges them in a roughly reasonable layout. To achieve this, we begin by creating reference image based on the 2D layout. Next, we reconstruct each 3D object (instance) using an efficient reconstruction model. We adopt 3DGS our 3D representation, considering its flexibility and geometry attributes. For layout initialization, naively converting the 2D layout into 3D space using depth information can lead to unrealistic overlaps and collisions between objects. To address this issue, develop initialization process to produce plausible 3D layouts by incorporating both geometry and semantic knowledge.
Reference Image Generation. Instead of lifting 2D bounding boxes to 3D space and generating 3D objects from scratch (Sun et al., 2024), our method first synthesizes a 2D reference image based on the given layout and text prompt. This reference image significantly narrows the solution space and provides crucial guidance for the subsequent generation process. Specifically, we use the state- of-the-art method MIGC (Zhou et al., 2024a) for layout-guided 2D image generation, defined as:
Lref MIGC(YB, B3)
where Leef represents the generated reference image.
Individual 3D Object Reconstruction. Next, we reconstruct each instance from the generated 2D reference image. To alleviate the influence of occlusion among instances, we first segment each instance in the reference image Iref using Segment- Anything Model (SAM) (Kirillov et al., 2023), and inpaint them as follows:
(2) where 4; and M, are the RGB values and the foreground mask of the ins instance. SD is the Stable Diffusion inpainting model, (~ M,) n b, represents the region to be inpainted and i, is the inh completed 2D instance. Note that to make the inpainted image compatible with the object- centric reconstruction models, also perform proc-pnnes remove the background and center
1,, M, SAM(Tref by); f, SD(T, (~ (b,,

---

Published as a conference paper at ICLR 2025
the object within the image Í, Fig. 3 illustrates this segmentation, inpainting and post-processing process. Since the reconstruction models are sensitive to imput image quality, we optionally enhance the details of the inpainted image I using a ControlNet- (Zhang et al., 2023) model. Finally, I, is fed into the Large Multi- View Gaussian Model (LGM) (Tang et al., 2024a) to generate the coarse 3D instance A, Thanks the explicit geometry priors from LGM and attributes of gaussian splatting, we could primarily eliminate the Janus Problem and add fine- une-grained details to the original instances in the subsequent refinement step.
3D layout initialization. Given the syn- thesized 3D instances, we arrange them in the 3D space by lifting the 2D bound- ing boxes }bi} discussed above into 3D. Each 3D bounding box is defined by its scale, rotation, and translation (denoted as {ss, ru, Ts J. From the given 2D layout, the scale s; of each 3D box can be intuitively set to Wo,WWs the width of original and post-processed 2D box b, (see Fig. 3). For the transla- tion li, we can directly get the first two el- ements (i.e., Ze. 3r) of the transition vector t; = {zz, 3t. } as the center coordinates of b;. Additionally, z; is initialized as the average of the foreground depth value predicted by an off-the-shelf depth estimation model (i.e., GeoWizard (Fu et al., 2024)), which is computed as:
Hosk of inshancs
piil Segmented instance
Final instonce
Referrnce
Mask of instorce? 2
doar Signented insterce? 2
Fieel instance
in which W., and Wi, are
resid graw out of to convos shoe!
Figure 3: Example of the instance segmentation, inpainting and post-process.
mean (M; @ Depthh4ref))
Rotation of each instance crucial in accurate layout estimation. Unlike existing works that over- look rotation initialization, propose to estimate rotation based feature similarity between ren- dered images and the 2D instance Í, in the reference image. Specifically, instance A, from a set of n camera poses with uniformly sampled azimuth and elevation, denoted as R=x2 = {(e,, )= We then extract the features of 2D instance in the reference images and rendered images as fi, and F {fi, f2, J2.,.. fu) from DINOv2 (Oquab et al., 2023), respectively. Finally, we calculate the cosine similarity and choose the best rotation r;:
render the coarse 3D
(es, .b) = arg max coo f1, F)),
(4)
(e.a)
where (eb, as) is elevation and azimuth of the best rotation As result, our reconstruction-basse initialization strategy provides a more accurate initial state for the final compositional 3D, which can benefit the subsequent refinement stage.
3.2 DISENTANGLED REFNNEMEN STAGE
Previous works (Zhou et al., 2024b; Li et al., 2024a) have attempted to optimize 31) scenes through local and global guidance. However, they often suffer from excessively long optimization times and the need for additional guidance. Instead, we disentangle the refinement process into two consecu- tive steps: a collision-aware layout refinement step and instance-wise refinement step.
3.2.1 COLLISTION LAYOUT REFINEMENT
The Coarse 3D Generation Stage (Sec. 3.1) has provided a relatively precise initial 3D layout, how- ever, we empirically observe that it is still insufficient to create a visually appealing composition. Thus propose layout refinement step for further improvement. First, adopt the spatial-aware SDS Loss (SSDS) as proposed in (Chen et al., 2024b):
VoLssds (0", x) = E,, (w)() (x,; Yn, 1) -)) Ox 00
(5)
In which 0, x. o², Eg- (xy; Yn. t) are the 3D representation, rendered image, attention map strength- ened spatial words, and the score function that predicts the sampled noise from the noised
5

---

Published as a conference paper at ICLR 2025
image with noise-level t and text prompt Yg. However, the SSDS loss is inadequate for address- ing sigmificant collisions caused by occlusion. For example, in Fig. 8, the dog obscures the blue tie and consequently does not receive sufficient gradients during optimization. To resolve this issue, propose a feature-level reference loss that leverages the 2D reference image and a tolerant collision loss handle object collisions more effectively.
Feature-level Reference Loss. To align the geometry information in the reference image with our apply pixel-level RGB loss enforce the alignment.
3D scene, the most intuitive method Nevertheless, we notice that the post-processing and the reconstruction model will unavoidably introduce some distortions and artifacts, making the RGB loss less robust and functional. Therefore, we propose a feature-level reference loss to stabilize the refinement process. Specifically, we only keep the foreground instances in the reference image and obtaín feature from higher layers of the DINOv2 (Oquab et al., 2023) model:
fref = DINO @ (M1 U M2 U...UUU UN)).
(6)
The feature-level reference loss be computed the 1.2 distance of fref and frender, here frender is the extracted feature of the rendered image at the reference view:
Ljeat X3 Σ
lifres frenderl12.
Tolerant Collision Loss. Inspired by physics-based sim- ulation (Santesteban et al., 2021) and CG3D (Vilesov et al., 2023), we propose a tolerant collision loss to sep- arate incorrectly overlapping instances while simultane- ously allowing a certain degree of collision to model nat- ural object interactions. As illustrated in Fig. 4, gaus- sians for two interacting instances are denoted (pi, .p...2..... Pk. }o and P2 the coordinate of the kes gaussian. The K₁ and K2 are the numbers tively. In order to get rough estimation of each instance, we first compute the mean coordinate of each instance de- noted as P'nea and p pmean. Then, we define mean sparsity R as the mean distance of each point in Pj to Prncan, which can also be interpreted as calculating the mass distribution of each instance:
Precan
Pi ........... pk,), where ys is
3DGSA
3DGSP:
gaussians in the two instances, respec-
Figure 4: A simple illustration of how calculate the collision loss Loot-
K
1 K
R =
Pk
Pineaa
(8)
Next, we calculate distances from all the points in P2 to Pleea and subtract mean sparsity Rt which can be regarded as the average distance between the distributions two instances. The tolerant collision loss Loot is computed as:
K2 Lest de Σ ReLU(D) where D (R 11p2 Pmeaa 2,,,..., K2).
Here ReLU is the rectified linear unit. Intuitively, this loss penalizes gaussians that are too close to the centers of other instances, while allowing specific distances to model intersection between instances, making the optimization more robust and effectively addressing the overlapping problem.
Our full collision-avee aware layout loss is defined as a weighted combination the three loss terms:
Llayout Lsads + >x XoCoot-
(10)
3.2.2 INSTANCE- WISE REFINEMENT
Since the LGM can only produce fixed number of 3D gaussians, the reconstructed 3D instances are often low quality with holes and broken textures. Next, we discuss refining these coarse 3D instances enhance its geometry and texture quality. The conventional refinement strategies adopted by other optimization based methods (Wang et al., 2023; Yi et al., 2024) are not efficient enough and often result in complete deformations of the shape of coarse 3D instances due to the

---

Published as a conference paper at ICLR 2025
excessive noise added to the rendered view, which aggravates the degeneration of geometry and results in drastic changes. To address this issue, we propose an adjustable timestep sampling strategy to preserve the geometry details and retain the primary interactions of the coarse 3D instances. Specifically, we lower the upper bound of timestep sampling's range in the earlier phase and increase it after specifie iterations when the geometry details have been well formed. Formally, the gradient of instance refinement SDS loss Lods is computed as follows:
dr 2g)
VeL x) E,,, (,,,
(11)
where f is the adjusted timestep. Intuitively, geometry most likely to deviate during earlier refine- ment phase but should remain mostly unchanged in the rest of the process. As a result, our minor modification prevents the basic geometry from deviating significantly from its initial state. More- over, we noticed that the normal of coarse 3D instances might be rough and contain artifacts, which can harm their visual quality. So we combine an additional normal smooth loss Lsmothh and total variation (Rudin & Osher, 1994) (TV) regularization terms (denoted as Liv and Liv) to encourage better geometry of 3D instances. The full loss for instance- mnssence-wise refinement can be represented as:
Linstance = Lods + CeCooooth + >eo + C7v).
(12)
worth noting that when we extend the iterations for refinement (marked as extended refine ment), we can produce comparable even better single-object generation results compared to other optimization- methods (Liang et al., 2024; Wang et al., 2023), as shown in Fig. 6.
Interactive Editing. Besides 3D scene generation, users can customize the per-instance text prompt y in the wiste refinement step for interactive editing. Specifically, we first generate a 3D scene with our coarse 3D generation stage and then stylize each 3D instance in the refinement step, with users' custom text prompts 9;- This editing application opens up new possibilities for generating extra-complex compositional 3D scenes, with each instance having its attribute and style.
4 EXPERIMENTS
4.1 IMPLEMENTATION DETAILS
In the Coarse 3D Generation Stage, we render the 3D instances at 10-degree intervals when esti- mating the coarse rotation For the Disentangled Refinement Stage, in the collision-aware layout refinement step, we use the original hyper-parameter settings for the SSDS loss Lasds and optimize the layout for 400 iterations, with N and A set to 10.0 and 0.2, respectively. In the instance-wise re- finement step, we utilize DeepFloyd (Shonenko et al., 2023) guidance with a total of 1500 iterations following the short refinement strategy (see Fig. 5). We set the timestep range to [0.10, 0.50] from step 0 to 800, and [0.02, 0.75] from step 800 to 1500. The parameters A, and λtv are set to 1.0 and 0.2, respectively. The extended instance-wist refinement strategy and additional implementation details are provided in Appendix B.
4.2 MAIN RESULTS
Validation Set. To better validate Layout-Your-3D' Your-3D's ability in compositional 3D generation, we construct a validation set naming Compo20 for evaluation. The Compo20 consists of 20 composi- tional text prompts, each containing two or more instances with specific interactions. For each text prompt, we have a user to manually provide a 2D layout and also automatically generate 2D layout using LLM-grounded Diffusion (Lian et al., 2023).
Comparison methods. We compare Layout- Layout-You-33] Your=3D with several text-to-3D methods, i.e., Dream- Fusion (Poole al., 2022), ProlificDreames (Wang et al., 2023), Dreasrr (Yi al., 2024), LucidDreamer (Liang al., 2024) and GraphDreamer (Gao et al., 2024). Since our method utilizes an additional 2D layout as a condition, we provide results by both the user-given and -groudddd 2D layout. Moreover, we present a comparison of single 3D object generation in Fig. 6. Note that do not compare with works that haven't 't released complete or official implementations (e.g., Locally Conditioned Diffusion (Po & Wetzstein, 2023), CG3D (Vilesov et al., 2023)).
Qualitative Comparison. As shown in Fig. 5, our method can generate compositional 3D scenes with higher quality and better understand the text prompts. In contrast, other methods often struggle
7

---

Published as a conference paper at ICLR 2025
Dreamf usion
ProlificDreamer
LucidD reamer
GaussianD reamer
GraphD reaeer
Ours
To bengle) weaninng (a blue. tik""
TA small blue toy bax] riding (a lien?
cute stuf cracadile] and (a small stuf stuffed tay rabbit] sitting [a skateboard]
[A brown with cushion), [a feddy bear] and (a basketbali) on
Figure 5: Main results and comparison with other cext-to-3D methods on our Compo20 validation Sayouu-YYour311 can generate compositional 3D scenes with higher quality and more reasonable 3D layouts. Note that the first two rows of our results are generated with L-M--goudded 2D layouts, and the last two are generated with user-given 2D layouts.
Table 1: Quantitative results of Layout-Your-3D and other text-to-3d methods on our proposed Compo20 validation set. Note that we adopt the short instance-wise refinement setting here for fair comparison. The best and second best results bold and underlined, respectively.
Method
Time cost
CLIP Ccore t
BLIP VQA Score + MGPT-CoTT
DreamFusion
45 mins hours
17.98%
23.44%
31.08%
ProlificDreamer
24.93% %
42. 12%
21.36%
Greaser
15 mins
22.21%
28.74%
25.91%
IncidDreamer
35 mins
22.36%
34.11%
34.84%
GraphDreamer
hours
24.78%
31.18%
44.28%
Layout-Your-3D Your-3D Yourut- (LLM- -..MM--gouddd)
23.26%
53.51
50.44%
12 mins
22.83%
52.26%
48.80%
comprehend spatial relationships and semantic attributes in text prompts, leading poor quality and the Janus Problem. These results highlight Layout- Your--or- 's strength in compositional 3D generation. More visualization results are provided in the appendix C.
Table User study on different text-to-3D methods in terms of alignmnin,mnt, quality and rational- ity. The best and second best results shown in bold and underlined.
Method
Text- Alignmettn
Quality
Rationality
DreamFusion
3.36
5.48
3.79
ProaifieDreamer
7.34
7.26
6.91
Dreaerrr
4.22
6.61
4.72
LueidDreamer
5.61
7.72
6.38
Ours
9.08
8.44
8.72
Mars(LLM-groundee)
8.68
7.93
8.14
Quantitative Comparison. We conduct quantitative comparisons on the Compo20 validation set, presenting results adopted by the short refinement strategy. Since CLIP-Score (Radford et al., 2021) not capable of accurately measuring the semantic correspondences (Huang et al., 2023a), we also employ other metrics, BLIP-VQA and mGPT-CoT (Huang al., 2023a), to make fine-graine assessment of the text-3D alignment. As shown in Tab. 1, our Layout- Your-3D outperforms other methods, demonstrating the superiority of our method quantitative evaluation.
8

---

Published as a conference paper at ICLR 2025
Drolificrreammee (-Htes)
LucidD ruamer (-35mirs
Caussian (-15mins)
Ours (-Jama))
Ours extended 1-2amass
Text Frompt: aa beoglé'
Text Prompt: squirref
Text Prompt: la pigsor'
Figure 6: Comparison on the quality of single 3D instance generation. We provide both the short and extended strategy. When lengthening the optimization process, our Layout- Your-3D can generate comparable or even better results than SOTA text-to-3D methods.
10 Legonti
(Dun) CombaaVerse
10 User gaven
Losts only (w/l Syean boutl
Lasts Lyut (w// Laa)
Lyse Lyeas besa (Full Aupour)
Illustration of user given and LLM- L.M-ggruuded 2D lay- out generated reference images. We also visualized our coarse 3D generation and compared it with Combo
Figure 7:
Figure 8: Ablation study on the effective- ness of different loss items in our layout loss Llagout.
User Study, In addition to qualitative and quantitative comparisons, we conduct user study to evaluate the proposed method further. Participants are asked to rate text alignment, quality, and rationality on a scale ranging from to 10. Concretely, users are shown 3D results and corresponding text prompts in shuffled order and then asked give ratings. We collect 500 responses from 100 users. In Tab. 2, the overall ratings indicate clear preference of users for our method.
4.3 ABLATION STUDIES
3D layout initialization. We conduct an ablation study on the 3D layout initialization strategy by comparing the generated coarse scenes. As illustrated in Fig. 7, compared Combo (Chen et al., 2024b), our initialization provides a more accurate starting point for subsequent refinement. These results demonstrate the effectiveness of initialization strategy.
Collision-aware Layout Refinement. We analyze the function of various loss terms in the layout refinement step by using a toy example. When different 3D instances have major intersections, the overlapped instances tend to be concealed in the rendered images, thus hindering the optimization of diffusion -uiided loss. Instead, our tolerant collision loss Lew further optimizes the geometry itself and feat eat encourages the alignment of spatial and high-level semantic information between the reference image and rendered reference view. As shown in Fig. and Tab. 3. our layout loss Llayout can effectively address this issue, thus resulting in more coherent 3D layouts.
Instance-wise Refinement. We conduct an ablation study on the influence of our timestep ad- justment given examples with different settings. As shown the left side of Fig. 9, timestep adjustment strategies applied by other methods often result in inferior performances and would introduce artifacts in the appearance. On the right side of Fig. 9, visualize the effects of the

---

Published as a conference paper at ICLR 2025
Ouis
Dream usonn Timestop
Prolificirreamr Timestop
Table Ablation study on different loss items of our Llayout, measured in BLIP-VQA score.
Loss term
BLIP-VQA
Ours
Emoott
Full (Llayout)
53.51%
-Cool
52.04%
-asds
52.71 %
feat
53.41 %
Figure 9: Ablation studies on timestep sampling and ad- ditional loss items in instance-wise refinement step.
feat, Leot
53.09%
A beogle
beogle creased the aupemoo
beagls wouring aechorea"
A becpl
bogle eecriin colert petters!
Wes morble of boagl
pigeee
Volenc Hoc spal ogeen,
M gemn secerart
pgen!
anigane plen
orlgrar aggal
(a) extended refinement strategy.
(b) short refinement strategy.
Figure 10: Examples the 3D instances refined with custom text prompts. We conduct experiments on both the longer and shorter refinement strategies to validate the effectiveness of customization.
TV losses Lty. City and normal smooth loss Lemooth By incorporating these two additional loss terms, Soyout-Your-3] can achieve better geometry consistency and quality. We that all the analyses above are based on short instance- wise refinement strategy.
4.4 DOWNSTREAM APPLICATIONS
Instance customization. We present 3D instances fined with custom text prompts in Fig. 10. Our method can guide the refinement process generate plausible and coherent results that are more aligned with text prompts. Thus, Layout-Youur Your-3D allows for the initial generation of coarse 3D scene, followed by the controlled adjust- ment of specific attributes across different instances. As result, method can produce stylized and complex outputs with better flexibility.
Figure 11: A simple illustration of the ob- ject insertion application.
Object insertion. Since 3D results are closely related to reference image, incorporating object insertion into pipeline would be straightforward. As illustrated in Fig. 11, the first and second rows show the edited reference images and corresponding 3D results. We can see that our method can insert objects with iterative editing, helping customize their desired 3D content.
CONCLUSION
In this paper, propose Layout-Your-3D. method designed for highly controllable and precise compositional 3D generation. Our method addresses the issue compositional 3D generation in coarse-to-fine fine manner. First, we initialize the rough asset with our Coarse 3D Generation Stage. Then, we propose a Disentangled Refinement Stage. By refining the layout and instances separately, Layout-Your-3D Your-3D achieve more flexible and high-quality compositional 3D generation with minimal cost time. Through massive experiments and comparisons, we believe Layout-Your-3D Your-3D indicates essential step toward more comprehensive and mature 3D generation.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Haotian Bai, Yuanhuiyi Lyu, Lutao Jiang, Sijia Li, Haonan Lu, Xiaodong Lin, and Lin Wang. Com- ponerf: Text-guided multi-object compositional nerf with editable 3d scene layout. In arxiv, 2024.
Hila Chefer, Yuval Alaluf, Yael Vinker, Lior Wolf, and Daniel Cohen-Or. Attend-and-excite Attention-based semantic guidance for text-to-image diffusion models. In ACM Transactions on Graphics (TOG), ACM New York, NY, USA, 2023. 3
Rui Chen, Yongwei Chen, Ningxin Jiao, and Kui Jia. Fantasia3d: Disentangling geometry and appearance for high-quality text-to-3d content creation. ICCV, 2023.
Tianrun Chen, Chaotao Ding, Shangzhan Zhang, Chunan Yu, Ying Zang, Zejian Li, Sida Peng, and Lingyun Sun. Rapid 3d model generation with intuitive 3d input. In CVPR, 2024a.
Yongwei Chen, Tengfei Wang, Tong Wu, Xingang Pan, Kui Jia, and Ziwei Liu. Comboverse: Compositional 3d assets creation using spatially-aware diffusion guidance. In ECCV, 2024b. 3, 5,9
Zilong Chen, Feng Wang, Yikai Wang, and Huaping Liu. Text-to-3d using gaussian splatting. In CVPR, 2024c. 3
Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A universe of anno- tated 3d objects. In CVPR, 2023.
Xiao Fu, Wei Yin, Mu Hu, Kaixuan Wang, Yuexin Ma, Ping Tan, Shaojie Shen, Dahua Lin, and Xiaoxiao Long. Geowizard: Unleashing the diffusion priors for 3d geometry estimation from single image. In ECCV, 2024.
Gege Gao, Weiyang Liu, Anpei Chen, Andreas Geiger, and Bernhard Schölkopf. Graphdreamer: Compositional 3d scene synthesis from scene graphs. In CVPR, 2024. 2, 7,
Antoine Guédon and Vincent Lepetit. Sugar: Surface-aligned gaussian splatting for efficient 3d mesh reconstruction and high-quality mesh rendering. In CVPR, 2024. 16
Yuan-Chen Guo, Ying-Tian Liu, Ruizhi Shao, Christian Laforte, Vikram Voleti, Guan Luo, Chia- Hao Chen, Zi-Xin Zou, Chen Wang, Yan-Pei Cao, and Song-Hai Zhang. threestudio: unified framework for 3d content generation.
project/
threestudio, oi 2023. 16
Haonan Han, Rui Yang, Huan Liao, Jiankai Xing, Zunnan Xu, Xiaoming Yu, Junwei Zha, Xiu Li, and Wanhua Li. Reparo: Compositional 3d assets generation with differentiable 3d layout alignment. In arXiv, 2024.
Yicong Hong, Kai Zhang, Jiuxiang Gu, Sai Bi, Yang Zhou, Difan Liu, Feng Liu, Kalyan Sunkavalli, Trung Bui, and Hao Tan. Lrm: Large reconstruction model for single image 3d. ICLR, 2024. 3
Kaiyi Huang, Kaiyue Sun, Enze Xie, Zhenguo Li. and Xihui Liu. T2i-compbench: A comprehensive benchmark for worn-world compositional text-to-image generation. In NeurlPS, 2023a.
Siyuan Huang, Zan Wang, Puhao Li, Baoxiong Jia, Tengyu Liu, Yixin Zhu, Wei Liang, and Song- Chun Zhu. Diffusion-batt generation, optimization, and planning in 3d scenes. In CVPR, 2023b.
2
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson. Tete Xiao, Spencer Whitehead, Alexander Berg, Wan- Yen-Yen Lo, et Segment anything. In ICCV, 2023. 4, 17
Haoran Li, Haolin Shi, Wenli Zhang, Wenjun Wu, Yong Liao, Lin Wang, Lik-hang Lee, and Pengyuan Zhou. Dreamscene: 3d gaussian- based text-to-3d scene generation via formation pat-
tern sampling. In ECCV, 2024a. 2, 3
11

---

Published as a conference paper at ICLR 2025
Xiaolong Li, Jiawei Mo, Ying Wang, Chethan Parameshwara, Xiaohan Fei, Ashwin Swaminathan, CJ Taylor, Zhuowen Tu, Paolo Favaro, and Stefano Soatto. Grounded compositional and diverse text-to-3d with pretrained multi-view diffusion model. In arXiv, 2024b. 2, 14
Yuheng Li, Haotian Liu, Qingyang Wu, Fangzhou Mu, Jianwei Yang, Jianfeng Gao, Chunyuan Li, and Yong Jae Lee. Gligen: Open-set grounded text-to-image generation. In CVPR, 2023.
Long Lian, Boyi Li, Adam Yala, and Trevor Darrell. LIm-grounded diffusion: Enhancing prompt understanding of text-to-image diffusion models with large language models. arXiv, 2023.
Yixun Liang, Xin Yang, Jiantao Lin, Haodong Li, Xiaogang Xu, and Yingcong Chen. Luciddreamer: Towards high-fidelity text-to-3d generation via interval score matching. In CVPR, 2024. 3,
Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick. 7ero-1-to-3: zero-shot one image to 3d object. ICCV, 2023. 15
Yuan Liu, Cheng Lin, Zijiao Zeng. Xiaoxiao Long, Lingjie Liu, Taku Komura, and Wenping Wang. Syncdreamer: Generating multiview-consisten images from single-view image. In ICLR, 2024a. 3
Yuheng Liu, Xinke Li, Xucting Li, Lu Qi, Chongshou Li, and Ming-Hsuaa Yang. Pyramid diffusion for fine 3d large scene generation. In ECCV, 2024b. 2
Xiaoxiao Long, Yuan-Chen Guo, Cheng Lin, Yuan Liu, Zhiyang Dou, Lingjie Liu, Yuexin Ma, Song-Hai Zhang, Mare Habermann, Christian Theobalt, et al. Wonder3d: Single image to 3d using cross-domain diffusion. In CVPR, 2024.
Longfei Lu, Huachen Gao, Tao Dai, Yaohua Zha, Zhi Hou, Junta Wu, and Shu-Tao Xia. Large point-to-gaussian model for image-to-3d generation. In ACM MM, 2024. 15
Zhiyuan Ma, Yuxiang Wei, Yabin Zhang, Xiangyu Zhu, Zhen Lei, and Lei Zhang. Scaledreamer: Scalable text-to-3d synthesis with asynchronous score distillation. In ECCV, 2024. 3
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel HAZIZA, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learn- ing robust visual features without supervision. TMLR, 2023. 5, 6, 17, 22
Ryan Po and Gordon Wetzstein. Compositional 3d scene generation using locally conditioned dif- fusion. In arxiv, 2023. 3,
Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. ICLR, 2022. 3,7
Lingteng Qiu, Guanying Chen, Xiaodong Gu, Qi Zuo, Mutian Xu, Yushuang Wu, Weihao Yuan, Zilong Dong, Liefeng Bo, and Xiaoguang Han. Richdreamer: A generalizable normal-depth diffusion model for detail richness in text-to-3d. In CVPR, 2024. 3
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In ICML, 2021. 8, 17, 22
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High- resolution image synthesis with latent diffusion models. In CVPR, 2022. 2, 3, 17
L.I. Rudin and S. Osher. Total variation based image restoration with free local constraints. In ICIP. 1994. 7
Igor Santesteban, Nils Thuerey, Miguel A Otaduy, and Dan Casas. Self-supervised collision han- dling via generative 3d garment models for virtual try-on. In CVPR, 2021.
Yichun Shi, Peng Wang, Jianglong Ye, Long Mai, Kejic Li, and Xiao Yang. Mvdream: Multi-view diffusion for 3d generation. In ICLR, 2024. 3, 14
12

---

Published as a conference paper at ICLR 2025
Alex Shonenko, Misha Konstantinov, Daria Bakshandaeva, Christoph Schuhmann, Ksenia Ivanova, and Nadiaa Klokova. DeepFloyd IF: a novel state-of ofe--of-thhtrrrre open-source text-to-image model with high degree of photorealism and language understanding. https:/ httpss//www// /www. deepfloyd. ai/ deepfl oyd- f, 2023. 7
Shaorong Sun, Shuchao Pang. Yazhou Yao, and Xiaoshui Huang. Comogen: A controllable text-to- 3d multi-object generation framework. In arXiv, 2024. 2, 3, 4, 14
Jiaxiang Tang, Zhaoxi Chen, Xiaokang Chen, Tengfei Wang, Gang Zeng, and Ziwei Liu. Lgm: Large multi-view gaussian model for high-resolution 3d content creation. In ECCV, 2024a. 2, 3,
Jiaxiang Tang, Jiawei Ren, Hang Zhou, Ziwei Liu, and Gang Zeng. Dreamgaussian: Generative
gaussian splatting for efficient 3d content creation. In ICLR, 2024b. 16
Alexander Vilesov, Pradyumna Chari, and Achuta Kadambi. Cg3d: Compositional generation for text-to-3d via gaussian splatting. In arXiv, 2023. 6, 7
Zhaoliang Wan, Yonggen Ling, Senlin Yi, Lu Qi. Wang Wei Lee, Minglei Lu, Sicheng Yang, Xiao Teng, Peng Lu, Xu Yang, et al. Vint-6d: A large-scale object-in-hand dataset from vision, touch and proprioception. In JCML, 2024. 2
Zhengyi Wang, Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu. Prolific- dreamer: High-fidelity and diverse text-to-3d generation with variational score distillation. In NeurIPS, 2023. 2, 3, 6,
Jinheng Xie, Yuexiang Li, Yawen Huang, Haozhe Liu, Wentian Zhang, Yefeng Zheng, and Mike Zheng Shou. Boxdiff: Text-to-image synthesis with training-free free box-constrained diffu- sion. In ICCV, 2023. 3
Yinghao Xu, Zifan Shi, Wang Yifan, Sida Peng, Ceyuan Yang, Yujun Shen, and Wetzstein Gordon. Grm: Large gaussian reconstruction model for efficient 3d reconstruction and generation. In arxiv, 2024. 15
Taoran Yi, Jiemin Fang, Junjie Wang, Guanjun Wu, Lingxi Xie, Xiaopeng Zhang, Wenyu Liu, Qi Tian, and Xinggang Wang. Gaussiandreamer: Fast generation from text to 3d gaussians by bridging 2d and 3d diffusion models. In CVPR, 2024. 2, 3. 6,
Xin Yu, Yuan-Chen Guo, Yangguang Li, Ding Liang, Song-Hai Zhang, and XIAOJUAN QI. Text- to-3d with classifier score distillation. ICLR, 2023.
Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. ICCV, 2023. 5
Qihang Zhang, Chaoyang Wang, Aliaksandr Siarohin, Peiye Zhuang, Yinghao Xu, Ceyuan Yang, Dahua Lin, Bolei Zhou, Sergey Tulyakov, and Hsin-Ying Lee. Towards text-guided 3d scene composition. In CVPR, 2024.
Dewei Zhou, You Li, Fan Ma, Xiaoting Zhang, and Yi Yang. Mige: Multi-instance generation controller for text-to-image synthesis. In CVPR, 2024a. 3,
Xiaoyu Zhou, Xingjian Ran, Yajiao Xiong, Jinlin He, Zhiwei Lin, Yongtao Wang, Deqing Sun, and Ming-Hsuan Yang. Gala3d: Towards text-to-3d complex scene generation via layout-guided generative gaussian splatting. In ICML, 2024b. 2, 3,
A DISCUSSIONS
A.1 LIMITATIONS & FUTURE WORK
Since Layout- Your-3D relies on reference images to generate compositional 3D assets, it would need deliberately designed 2D layouts to generate extra-complex 3D scene, e.g., having more than
13

---

Published as a conference paper at ICLR 2025
10 instances in the asset. What's more, observed, the SSDS loss, proposed in Verse, is not capable of harmonizing interactions with too many instances involved.
Thus, in future work, will work deliberately designed modules to achieve mature text-to-3d layout refinement and generation, without constraints on the number of instances or in- teractions. Also, the application in scene generation of our method is promising direction, can extend our method in order to improve the efficiency and controllability Text-to-3D scene generation.
lego man
horoe
Figure 12: A simple illustration of failure cases.
A.2 FAILURE CASES
We provide a simple failure case in Fig 12. The text prompt is 'A lego man riding a horse, the horse wearing blue hat', in which add uncommon interactions between instances, i.e., 'blue hat' and 'horse'. As illustrated in the figure, for text prompts that are extremely complicated or counter- intuitive, the diffusion models are not capable of comprehending the spatial relationships correctly and hence misdirect the optimization process.
A.3 MORE DOWNSTREAM APPLICATIONS
Except for instance-wise customization and object insertion, our Layout-Your-3D Your-3D can also support the Conversational Iterative Editing as proposed in GALA3D, Once obtained the final 3D layouts of the resulting compositional 3D, we can easily edit the position and rotation each instance, together with its attributes and style, through iterative conversation. Since the generated 3D set is essentially comprised of different 3D instances, has higher degree of freedom and can achieve better quality. This advantage helps Layout- Your-3D stand out among different Text-to-3D generation methods.
A.4 DISCUSSION ON CONCURRENT METHODS
GroundedDreamer (Li al., 2024b) and COMOGen (Sun al., 2024) are the two related and concurrent works. Here we give comprehensive analysis and a comparison between these two methods and Layout- Your-3D. Since they have released official implementations yet, we simply use the results from the original paper for comparison, and analyze the possible advantages and disadvantages.
G GreameedDreamer's core concept is to marry thend-and-Exeiit with MVDream (Shi et al., 2024) to generate compositional and consistent multi-view images, then lift them to 3D assets through an optimization-basee method. Though it shows some surprising results and relatively high quality, due to the limited compositional 3D data used in the training process, the limitations MVDream still exist, i.e., lack of prior knowledge for generating multiple instances. Also, the pure optimization- based paradigm makes GroundedDreamer low in efficiency, taking nearly hours to generate one single asset. In comparison, Layout-Your-31 Your-3D needs only about ten minutes (short refinement)
14

---

Published as a conference paper at ICLR 2025
COMOGen
Ours
COMOGen
Ours
squirrel
retra rodio
antique dock
A fire hydrant
A dusty shelf
'A squirrel and fire hydrant'
antique retro radio compete for attention dusty sheif
COMOGen
Ours
blue bird
hamburger
'A blue bird standing hamburger'
Comparison with COMOGen, results are generated with 2D layouts provided in the original paper of COMOGen.
Figure 13:
about one hour (extended refinement) to generate one asset. given more GPU cards, the time cost could be reduced to several minutes and half an hour. Furthermore, since Layout- Your-3D primarily generates individual 3D instances and subsequently combines them, which mitigates the domain gap during the generation process, it effectively helps address compositional 3D generation task at its source.
COMOGen takes the same 2D layout setting as adopted in Layout-Your-al to indicate ideal regions of different instances in the text prompts. However, different from Layout-Your-3D, COMOGen first adopted Layout-SDS loss to distill the spatial layout knowledge, then uses Multi-view Control module to ensure multi-view consistency and alleviate Janus Problem, finally proposed a 3D con- tent enhancement module generate realistic and diverse 3D samples. The Layout-SDS loss could provide a rough estimation of the ideal 3D content, in the following Multi-view Control module, COMOGen adopted Zero-12 123 (Liu et al., 2023) multi-view diffusion model ensure the geome- try consistency. But as we mentioned above, current multi-view diffusion models are still subject to the domain gap, thus making it hard to generalize compositional scenarios, damaging the effec- tiveness of COMOGen. Moreover, though not presented, the time cost for COMOGen might not be comparable our Layout- Your-3D, which only costs about 10 minutes generate a reasonable and plausible compositional 3D scene, as displayed in Fig. 13. Additionally, COMOGen did not show detailed and high-quality single object generation generatinn results, which constrains the application scenarios of it.
Another line of works, i.c., GALA3D and DreamSeene also have similar settings as our Layout- Your-3D. Nonetheless, these works have their own disadvantages. DreamScene is essentially com- posing different objects into one predefined scene with no specific interaction modeling process during generation, thus making the resulting scene less realistic and consistent. GALA3D addresses this issue by incorporating 3D layout-conditioned diffusion models as an implicit control over the interactions between objects. However, the produced 3D) scene has background and the quality of objects is relatively inferior, though processed through a tedious optimization process. Since the input settings and generated 3D assets are not strictly aligned with ours, we do not make direct comparisons between these methods and our You
A.5 OTHER RECONSTRUCTION MODELS
In the coarse 3D generation stage, we could exchange for any 3DGS based Image-to-3D model for the reconstruction of each instance, e.g., GRM (Xu et al., 2024), Coint-to-Gaussian (Lu al., 2024). What's more, it is also applicable to replace 3DGS with other 3D representations, like mesh or tetrahedron for advanced or specific usage. Though will be slightly different to optimize coarse
15

---

Published as a conference paper at ICLR 2025
3D instances in other representations, there substitutions for the refinement strategies in pipeline. Also, our final 3DGS- based asset could be converted into mesh with the efficient mesh extraction method proposed in DreamGaussian (Tang et al., 2024b) or advanced mesh extraction methods like SuGaR (Guédon & Lepetit, 2024).
B EXPERIMENT SETTINGS B.1 VALIDATION SET
We analyze the core design of our proposed Comp20 validation set in this part. In order to provide a comprehensive measurement for the compositional 3D generation task, we divide the text prompts in our Comp20 validation set into several scenarios, covering wide range of cases that can meet requirements by users and real world applications:
Multiple instance without interactions,
e.g., 'A black bottle, with a white bottle on one side and blue bottle on the other".
Multiple instances with interactions, e.g., 'An astronaut sitting on sofa'.
Attribute control of instances,
e.g., 'A blue toy robot and red toy robot playing basketball'.
Small components in larger main body,
e.g., 'A furry fox's head, with a cupcake on its top, wearing red sunglasses and have lego block on the nose'.
In accordance with the requirements for compositional 3D generation, each text prompt contains 2 to 5 instances in the compositional 3D asset. This intentional design of our Comp20 validation allows for comprehensive evaluation on text-to-3D generation methods. The complete prompt list, together with the corresponding 2D layouts (including both eser-provided and L.LM-generated layouts), are presented Tab. 9. Note that in the default experiments we generate reference images with 512 512 resolution, and the 2D layouts presented in the form of [x1, v1, 22, y/], with each element has a value from range 0 to 512.
B.2 MORE IMPLEMENTATION DETAILS
We give a more detailed illustration of our experiment settings in this section. We implement Layout- Your-3D based on threestudio (Guo et al., 2023) for more integrated and readable system. We con- duct all of our experiments based on NVIDIA RTX6000 GPUs. It is worth noting that the baseline methods, i.e., DreamFusion and PremificDreamer are implemented threestudio, thus might result in minor differences with the original implementation results.
Table 4: The detailed list of the short (efficient) instance wise refinement step.
Hlyper-parameterr
value
batch_size
resolution resolution milestones
[256, 512]
800
position_Ir
[0. 0.0005, 0.00005, 500]
scale_Ir
0.005
feature Ir
0.01
opacity
0.01
rotation_Ir
0.001
densify/prune interval
100
densify/prune start iter
300
densify/prune until iter
900
total iter
1500
16

---

Published as a conference paper at ICLR 2025
Table 5: The detailed list of the extended (high quality) instance-wise refinement step.
Hyper-parameters
value
batch size
resolution
512
position_ir
[0. 0.0005, 0.00002, 1000]
scale_Ir
0.005
feature_Ir Ir
[0. 0.01, 0.005, 2000]
opacity _lr
0.05
rotation Ir
0,005
densify/prune interval
200
densify/prune start iter densify/prune until iter total_iter
400
1600
2000
Table 6: The detailed hyper-parametee list of the collision-aware layout refinement step.
Hypeer-aarameters
value
quaternion Ir
0.0001
transition x Ir
0.00002
transition transition.z.rr
0.00002
0.02
total iter
400
In the coarse 3D generation stage, guide the layout-conditioned text-to-image generation gen- erate canonical views, we add an additional text prompt '3d scene, front view' to the overall text prompt YB. This simple measure helps Image-to-3D reconstruction models obtain better results in the coarse 3D generation stage since multi-view diffusion models, e.g., MVDream, perform bet- ter when given images with canonical views. When performing segment inpaint, we prompt the SAM (Kirillov et al., 2023) with 2D bounding box of each instance to indicate the foreground region and the center points of all other instance's s boxes are treated as negative prompts to avoid segment- ing multiple irrelevant instances all at once. Moreover, for the depth estimation part, since the output prediction from Wizard not metric depth, we normalize the predicted depth values and cen- tralize the 3D location of the final 3D assets. In the instance-wise refinement step, provide more concrete setting for both the extended and short refinement and layout refinement steps. All listed in Tab. 4, Tab. 5, and Tab. 6, including learning rates for attributes of the gaussian splatting representation and different elements in layouts. Note that for the extended instance-wiss refinement strategy
the hyper-parameters
Stable Diffusion (Rombach al., 2022) guidance instead of Deep Floyd, to boost the quality of each instance. For the layout learning part, we assume that the input 2D layouts are accurate and reasonable enough. Since two of the spatial coordinates in the transition t depend mainly on the given 2D layouts, we only give a very small learning rate to these two elements.
B.3 ADDITIONAL ABLATION STUDIES
We conduct more ablation studies to validate the effectiveness our model selection. First, we compare the visual foundation model used when extracting features for coarse rotation estimation. We choose DINOv2 (Oquab et al., 2023) to estimate the rotation r; since DINOv2 can provide higher-level geometry and semantic information. In comparison, CLIP (Radford et al., 2021) is another model we adopted, but since it is not specifically designed to capture geometry information in images, it causes visible errors in the rotation estimation.
Also, we give a detailed time cost Layout-Your-31 Your-3D Tab. 7, including the time cost for each part and different settings, measured
single RTX6000 GPU.
Moreover, to prove that essential to process the layout refinement step and instance-win wiie re- finement step, present some toy results after the Coarse 3D Generation stage in Fig. 14. These
17

---

Published as a conference paper at ICLR 2025
Table 7: Time cost for each part of Layout-Your-3D, Your-3D, including both the short and extended refine- ment strategy.
Designed module
Time cost
Coarse 3D gen. Layout ref.
~ minute
+ ~~2 minutes
Ins. ref. (short/extended) Overall (short)
~ 35 minutes
~ 12 minutes (3 instances)
beagle] wearing [o tie]
TA squireel] standing
brown (s calition]
haddy Sear!
an
7 foy heart] and (a white fenther) on balonce scale]
Figure 14: 3D assets generated without coarse 3D generation stage.
examples, though show relatively reasonable geometry shapes and spatial relationships, are still not satisfying enough for further usage. Thus we claim that it is necessary for us to develop the Disentangled Refinement Stage to obtain better results.
C MORE QUALITATIVE RESULTS
C.1 SINGLE OBJECT GENERATION
In Fig. 15, we visualize more single object generation results produced by our extended instance- wise refinement step. These results demonstrate the capability of our Layout- Your-3D in the field of text-to-single 3D object generation. The figure shows that our refinement process could retain the primary geometry shapes and add fine-grained details. In about 25 to 30 minutes, we can generate 3D object of extra high quality with our Layout- Your-3D, comparable, or even much better than previous current SOTA methods. These results proves the versatility of our method.
C.2 CUSTOMIZZTIION
To validate the effectiveness and generalizability of method, give more examples in Fig. 16, both the short and extended refinement strategies can bring customized attributes and easily deform the style 3D assets, thus enabling flexible and controllable generation.
C.3 COMPOSITIONAL GENERATION
We present more generated compositional 3D assets by our Layout-Your-3D in Fig. 17, here we only show results generated with our short instance-wise refinement setting (~12 mins in all). We can see that Layout- Your-3D can generate compositional assets, in which 3D instances can in all granularity' from smaller ones like 'blue sunglasses', to bigger ones like 'A box', 'A toy pyramid'. This can be another strong evidence, showcasing that our method has great advantages the aspect of controlling 3D generation.
18

---

Published as a conference paper at ICLR 2025
'A bagel filled with cream cheese and lox
'A blue tie with flora patterns'
Gundam style small robot'
'An astronaut'
'A stuffed toy rabbit'
'Golden beer with frothy white head'
Figure 15: More results on the single object generation with our extended refinement strategy.
morble of lion, oncient articie
Ainecnoft lew
gold
Minecroft
Iecteriod
dressed
The extended refinement strategy.
(b) The short refinement strategy.
Figure 16: Additional visualization results on instance generation with custom text prompts. We show results generated with both the short and extended instance-wise refinement strategies. White images on the top left corner of each figure are the original coarse 3D instances produced by LGM.
19

---

Published as a conference paper at ICLR 2025
2D layouts
3D results
TA teddy bear] reading [a book] wearing (blue sunglasses]
[[A kitten] lying next to [a flower]
Findl mint
blue toy robot] and ned toy robot] playing [basketball]
[A tray] on top of (a toy pyramid], on the tray [a red bottle], [a toy pandaj [an applej
'[A squirrel] standing on la box]
"lA pigeon] having [a bagel] and [a beery
Figure 17: More visualization results generated with our Comp20 text prompts and layouts.
D ADDITIONAL RESULTS & COMPARISONS
D.1 LAYOUT INITIALIZATION
In this section, we provide more visualization results on the effectiveness our 3D layout initial- ization. We compare different 3D layouts generated with LLM inference (Gao et al., 2024; Zhou et al., 2024b) and our strategy in Fig. 18. LLM-generated 3D layouts cannot align with the input text prompts well, thus posing significant optimization problems in the following refinement pro-
20

---

Published as a conference paper at ICLR 2025
In comparison, our 3D layout initialization strategy benefits from deployed 2D models, and can generate both precise and reasonable layouts. These comparisons reflect the motivation using 2D blueprint for 3D layout initialization, instead of generating directly from LLMs.
Inpul lext prompl
LLM generaled layout Input 2D layoul lext prompt
Our layout initialization
bluc soy box riling loon'
koy
blze toy box ridig Hon'
lion
binng
"A kitton lying next to fowwe
kitten
Eoverr
kitten
"A squiree standing box"
square
squiral
squirel standing on
box
box
pushing big Pi Hbbit
'A smill robut pushing a big Godhall
smalll mieet
big footbell
small nabok
Figure 18: Visualization results of our initialized 3D layout, together with a comparison with LLM- generated coarse 3D layout boxes.
D.2 COMPARISON WITH OTHER WORKS
Since Combo Verse has not released the official implementations, compare Layout- Noyouut-Your333 Your-3Dwill the unofficial version in our experiments. For the unofficial implementation, we use the same ini- tialization strategy
Coarse 3D Generation Stage, instead of the presented in Combo Verse. The comparisons are shown in Fig. 19. The left images for each prompt are 3D scenes generated by our implemented Combo Verse and the right ones results generated with Yayout-Your-3D.
Moreover, we compare the results generated with the same prompt by GALA3D and our Layout- Your-3D in Fig. 20. The figure shows that GALA3D tends to produce unreasonable 3D scenes when given poorly initialized 3D layout boxes. It worth mentioning that GALA3D takes about 3 hours to generate one scene with instances, which can be inefficient compared our Layout-Your-3h Your-3D (12 minutes).
Ya beogle] wearing ing (a blue tie]
Io suuirell standin on a bak
[4 pigsen] and esfo) cattion
bedwetollf on
Figure 19: Comparison with Combo Verse.
21

---

Published as a conference paper at ICLR 2025
GALA3D
Layout-Youe Your- 3D
beagle] wearing blue tieJ
TA small blue toy box] riding [a lion]'
"AA cute stuffed toy crocodile] and [a small stuffed toy rabbit) sitting on (a skateboard]
'[A brown sofa] with [a cushion]. [a teddy bear] and [a basketball] on it'
Figure 20: Comparison with GALA3D.
D.3 ABLATION ON ROTATION ESTIMATION
We conduct ablation studies the effect of visual foundation models used when extracting features for coarse rotation estimation in Tab. 8. The 'Average error' term in the table represents the average deviation degrees for all instances in the validation set, calculate and estimate the error in de- grees manually. As shown in the table, the deployed visual foundation models significantly reduce rotation error, compared scenarios where rotation estimation not applied, thus validating the effectiveness of our coarse rotation estimation strategy.
Table 8: Ablation study on the foundation models used when extracting features for coarse rotation estimation.
Method
No estimation
CLIP (Radford al., 2021)
DINOv2 (Oquab et al., 2023)
Average error
18.9
10.8
5.5
22

---

Published as a conference paper at ICLR 2025
Table Detailed prompt list of our proposed Comp20 validation set, including overall text prompt
YB and 2D layout set B. We provide both the user-given 2D layout set and the LLM- -LM-generated onc.
Index
Text Prompt
[A black bottle], with [a white bottle] on one side and [a blue bottle] on the other
User
[24. 136, 168. 424]. [152. 336, 424J. [344, 104, 504, 424]
[200. 200, 280, 360J. [100, 200, 180, 360]. [300. 200, 380, 360]
IA teddy bear] reading [a book] wearing [blue sunglasses]
User
188, 40, 168, 192], [144, 24, 120, 128), [168, 96, 112, 168]
LLM
[104, 64, 168, 200]. [152, 0, 128, 168), [182, 112, 88, 184]
squirrel] standing
box]
User
[204, 51, 51, 0), [153, 0, 102, 179]
LLM
[190, 150, 222, 330]. [172, 330, 340, 442]
4
[A small blue toy box] riding [a lion]
User
153, 102, 0j. [102. 204, 153, 153]
[0. 140, 10). [120, 200, 168. 168]
[A small robot) pushing (a big football]
User
[51, 153, 179, 102]. [194, 102, 204, 128]
LLM
175, 250, 225, 2001. [225, 275, 200, 200]
pigeon] having [a bagel] and [a beer]
User
[102, 102. 358, 358). [230, 307. 358, 384). [358. 204, 435, 384]
LLM
[40. 88, 232, 416J. (240, 360, 368, 416), [376, 168, 488, 416]
[An astronaut] sitting on sofa]
User
[153, 102. 384, 409]. [51, 153. 384]
LLM
[165, 130, 347, 392], [80, 100, 432, 372)
[A kitten] lying next to [a flower]
User
[51. 153, 307, 358), [307, 204, 460, 358)
LLM
195, 218, 269, 492], [297, 224, 417, 382]
beagle] wearing blue tie]
User
[117, 51, 394, 460), [230, 204, 281, 435]
I.L
[88, 96, 424, 464], [208, 288, 312, 464] [A teddy bear] driving (a toy car]
10
User
[40. 184, 472. 384]. [215. 119, 335, 326]
[160, 150, 352, 406). [140, 320, 372, 470]
11
[A lego man] riding [a horse], the horse is wearing [a blue hat]
User
96, 64, 288, 3521, [39, 181, 481, 4631, [367, 152, 463, 240]
LLM
[150, 200, 280, 400), [120, 250, 390, 450], [220, 150, 290, 220] toy robot] playing [basketball]
blue toy robot] and
User
[39. 64, 240, 416], [256, 48, 488, 416]. [199. 167, 304, 272]
LLM
[50, 200, 190, 440), [320, 200, 460, 440), [225, 280, 285, 340]
13
[A brown sofa] with cushion], teddy hear] and basketball] on
User
32. 256. 480, 432]. [200. 120, 328, 344], [72. 224, 168, 360), [360, 256, 456, 352] [20, 180, 492, 412], [140, 230. 240, 310], [250, 220, 340, 330), [380, 250. 460. 330]
14
[A dog] taking [a boat]
User
[24. 256, 488, 384). [160, 104, 336, 304]
I.M
[186, 1500, 326, 290], [50, 250, 462, 780]
15
furry fox's head]
it, wearing [red sunglasses] have Ja lego block] the nose cupcake]
User
[40, 64, 480, 512], [168, 72, 336, 216], [64, 256, 440, 376), [168, 264, 256, 4071
LLM
[176, 128. 336, 288). [224, 88, 288. 152). [200, 160. 312, 200). [240. 208, 272, 240]
16
[A cute stuffed
crocodile] and [a small stuffed rabbit] on [a skuteboard]
User
[40. 360, 472, 432]. [72, 72, 296, 376). [296. 200, 424, 376]
[40. 304, 480, 432]. [80, 112, 280. 344), (296. 176, 440. 336]
17
[A toy dragon) reaching for [a cola] on [a wooden cabinet]
User
[256. 192, 456, 416), [32, 104, 248, 416), [312, 80, 392, 200]
LLM
[101. 179, 289, 400), [316, 233, 394, 471]. [40, 370, 472. 500]
18
the tray [a bottle], [a toy panda] [an apple] tray] top of toy pyramid],
User
196, 208, 424, 471], [56, 56, 136, 184], [408, 120, 480, 184], [16, 192, 504, 240), [192, 16, 336, 192]
LIM
[150, 300, 350, 500]. [140, 260, 360, 320], [160, 100, 220, 260], [240, 130, 320, 260], [330, 120, 390, 180]
19
[A toy heart] and [a white feuther]
bulance scale]
User
[56. 136. 168. 248]. [336. 192, 456. 240), [40. 224, 480. 384]
[140, 220, 240. 320), [272, 235, 352, 305]. [100. 150. 412, 362)
20
[A green pumpkin] and [a big round orunge] under [an umbrella] [48, 200, 288, 424], [280, 272, 448, 416), [8, 120, 504, 224]
User
LLM
[112, 290, 240, 4181, [272, 290, 400, 4181, [110, 50, 402, 290]
23


Reviews:
[{"review_id": "VDPNKirETv", "reviewer": "Reviewer_tCHQ", "summary": "- They enforce structural control of a 3D scene using a 2D layout. Their optimization process avoids collisions.\n- Their method works by first generating a 2D image based on text description, then use SAM to segment each isntnace in the reference iamge, with in-painting done on occluded parts.\n- They then use a large mulit-view gausian model to generate coarse 3D instances of objects within the scene. \n- Each 3D instance is then arranged into 3D space by lifting the bounding boxes into 3D, done by optimizing the rotation according to similarity in the DINO space of the rendered images of the rotated instances.\n- final two stages refine the layout according to collisions and refines the instance representations.", "questions": "Within the collision loss, is there any way to take into account the shape (covariance) of each gaussian in 3D space? My understanding is that it's only using the positions of the the Gaussians right now. Have you observed this to be a limitation?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The papers shows a potential way to use many existing models together to generate compositional 3D scenes. The results show that the method produces reasonable visual reconstructions and geometries. The method seems well motivated, and the paper is decently written.", "weaknesses": "My concern is that this work lacks fundamental novelty, in that prior works like Compositional 3D Scene Generation using Locally Conditioned Diffusion (https://arxiv.org/abs/2303.12218), Towards Text-guided 3D Scene Composition (https://arxiv.org/pdf/2312.08885) and CG3D: Compositional Generation for Text-to-3D via Gaussian Splatting (https://arxiv.org/pdf/2311.17907) all present various ways to create 3D scenes using implicit 3D represntations in a compositional way.\n\nThere does not seem to be any comparisons to these works, or mention of them in the related works (though CG3D was mentioned in the collisions section, but not compared with in the experiments section). A comparison to contextualize this work seems very important, but lacking.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "A9lwPkYzYl", "reviewer": "Reviewer_miqw", "summary": "This paper tries to address the problem of lay-out controlled 3D generation task. \nInterestingly, it relies on 2D layout instead of 3D layouts, and the 2D layout could be generated by LLM given a text prompt. \nWith this 2D Layout, it first generate a reference image with an off-the-shelf 2D-layout image generation model, and then try to reconstruct the 3D model and layout from the reference image. \nThis generated 3D model is coarse and may not very accurate, and this paper refines the generated model with SDS and feature loss, as well as collision-tolerant loss.\nThis paper proposes a very interesting pipeline using many off-the-shelf components, and it achieves good results", "questions": "1. For clarification, when inputing the text prompts, what's the prompt for LLM to generate 2D layouts? Also, do you mark the object in bracket like text prompts in figure 5 or LLM is supposed to decide which objects should be created?\n\n2. The proposed method uses many off-the-shelf algorithm and components like LLM generating 2D layouts, reference image generation, pose alignment. I am curious about the successful ratio of the whole process, and some potential failure cases.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. This paper proposes an interesting pipeline by combining several (off-the-shelf) components in a novel way. Getting 2D reference images from layout and trying to reconstructing 3D models from them leads a interesting 3D reconstruction task, and this paper provides a reasonable solution for it, by utilizing the inpainting, DINO feature distance and off-the-shelf image generation model.\n2. This paper involves several intesting losses like collision-tolerant loss, feature-level loss, and SDS with adjusted timesteps. \n3. Results are good. and the paper is easy to read and follow.", "weaknesses": "1. For baselines, it would be more convincing to seriously compare against models with 3D layout control like ComboVerse or [Compositional 3D Scene Generation using Locally Conditioned Diffusion]. The current paper only shows a single scene comparison against ComboVerse in Figure 7, which is insufficient to demonstrate the advantages of using 2D layout and the proposed method. More rigorous evaluation, such as those shown in Tables 1 and 2, along with a user study, would be needed for such comparisons.\n\n2. After reconstructing individual 3D models from 2D images, it is critical to align the pose (6D pose) of the reconstructed 3D model with the image appearance. The proposed method achieves this by randomly sampling many views of the 3D model and comparing them against the reference image by computing the DINO feature distance. However, I am curious about the accuracy of this method, as the rendered image could differ significantly from the reference image even with perfect poses if the distance (object scale) is not accurate. It would be more convincing to show results demonstrating the accuracy of this method.\n\n3. Regarding the reconstruction of objects from reference images, how is the mask/segmentation obtained for each object? Is this process automated, or does it require manual intervention? This question also appears in the following inpainting and pose alignment sections. Do we need to manually specify which object to inpaint or segment during this process or not?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vWDz8Uob7P", "reviewer": "Reviewer_PaDe", "summary": "the work trying to generate a 3D complex scene based on a textual prompt. they start by generating a 2D image of the scene using a 2D text-to-image generator, they extract 2D boxes for each instance. \nNext, they create each object in a 3D instance and position them according to the boxes.\nthe final step is SDS which fine-tune the positions for natural placement, specifically in a collision-free manner.\n\n[1] Yang, Yue, et al. \"Holodeck: Language guided generation of 3d embodied ai environments.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024\n[2] Rahamim, Ohad, et al. \"Lay-A-Scene: Personalized 3D Object Arrangement Using Text-to-Image Priors.\" arXiv preprint arXiv:2406.00687 (2024).", "questions": "1. why do the bird, bear, and Bagle look different in each example?\n2. what happens if the generated 2D images don't contain all the specified objects? it does happen a lot with the generation of complex scenes. like what is shown in [2]", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "generating complex scenes is a difficult task today.\nthis work tries to solve this task, into stages to deal with each part separately.", "weaknesses": "1. no novelty in the approach, you concatenated known strategies while trying to solve this problem.\n2. can it be solved using previous methods like [1]\n3. Besides the bear and the book, I can't see any interaction between objects. can you propose how to create interactions? (like the bird drinking the beer)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "N2GyNmgOyn", "reviewer": "Reviewer_YpWR", "summary": "This paper uses a 2D layout as guidance and proposes a Collision-aware Layout Refinement to optimize 3D layouts, effectively achieving compositional 3D generation with plausible object interactions.", "questions": "- My main concerns are discussed in the Weaknesses section. I can see from the paper that the authors have put a lot of effort into creating a valuable contribution, and despite my doubts about the contribution, experimental results, and design of the method, I still rate the paper as borderline reject (5). I believe the improvements needed cannot realistically be achieved during the rebuttal stage, and this may be due to time constraints in preparing the submission. I hope the authors can improve the method’s effectiveness and rewrite the contribution section to better highlight their approach, so that the paper can be successfully accepted at top conferences like **CVPR, ICML, or ICCV**.\n\n- One final note: while I don't mean to discourage the authors, I have limited expectations for changes in the rebuttal, as the scope of improvement I am suggesting is quite large. Therefore, **it might not be necessary to spend too much time adding experiments during rebuttal for me**. I wish the authors all the best with their research.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The paper is rich in content, and the proposed method successfully accomplishes the task set by its motivation.\n- The writing is clear, making it easy to understand the authors' research methods.", "weaknesses": "- Over-reliance on existing methods: The engineering aspect of the method is too pronounced. Aside from the newly introduced loss functions, much of the technique is a stack of existing methods, including SAM, ControlNet-tile, LGM, GeoWizard, DINO, etc. The use of LLM to assist in generating the 2D layout is also an existing strategy. Such extensive reliance on pre-existing methods can overshadow the key contribution of the paper. Additionally, I would like to ask: since LLMs are already in use, why not have the LLM or MLLM directly predict the 3D layout and then apply your Collision-aware Layout Refinement for 3D layout refinement? This would simplify the initial stage, reduce reliance on existing methods, and highlight the contribution of your technique, making the overall framework more elegant and concise.\n\n- Limited contribution: The task goal overlaps significantly with many existing compositional 3D scene generation methods (GALA3D, graphdreamer, dreamscene， progress3D...), making the problem being addressed less novel. Furthermore, as mentioned earlier, many critical components are stacked from existing methods. Although the Feature-level Reference Loss and Tolerant Collision Loss are novel, their impact seems less substantial compared to the other parts, as seen in the visual results in Figure 8. As a suggestion for improvement, it might be better to focus on complex single-image-to-3D reconstruction instead of compositional 2D layout + text-to-3D generation. You could use multimodal LLMs to decompose objects from a complex, unclear image for reconstruction and then refine and combine them into a 3D scene. This shift could substantially enhance the novelty and contribution of the paper.\n\n- Insufficient comparison in experiments: In the supplementary materials, methods like GALA3D and DreamScene are mentioned. Why aren’t there comparisons with compositional 3D scene generation methods in the main text, such as GALA3D, GraphDreamer, DreamScene, Progressive3D, or more recent works like The Scene Language? Even if comparing with the very recent papers isn't necessary, it would be more reasonable to compare your work with three or so methods that are highly relevant to your contribution, such as GALA3D, GraphDreamer, and DreamScene. This could also help improve the previous point about novelty. Additionally, based on my own experience, DreamFusion should be faster than LucidDreamer—on my computational resources, the timing is approximately 30 minutes vs. 50 minutes, differing from the 1 hour vs. 1 hour stated in your paper. Although a minor issue, it raises some doubts about the accuracy of the reported experimental results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)