Build-A-Scene: Interactive 3D Layout Control for Diffusion-Based Image Generation
Authors: Abdelrahman Eldesokey
Peter Wonka
Venue: ICLR 2025 Poster (2025)
Keywords: Diffusion Models
Text-to-Image
Layout Control
Primary area: generative models
TL;DR: We propose a diffusion-based approach for Text-to-Image (T2I) generation with interactive 3D layout control.

Abstract: We propose a diffusion-based approach for Text-to-Image (T2I) generation with interactive 3D layout control.
Layout control has been widely studied to alleviate the shortcomings of T2I diffusion models in understanding objects' placement and relationships from text descriptions.
Nevertheless, existing approaches for layout control are limited to 2D layouts, require the user to provide a static layout beforehand, and fail to preserve generated images under layout changes.
This makes these approaches unsuitable for applications that require 3D object-wise control and iterative refinements, e.g., interior design and complex scene generation. 
To this end, we leverage the recent advancements in depth-conditioned T2I models and propose a novel approach for interactive 3D layout control.
We replace the traditional 2D boxes used in layout control with 3D boxes.
Furthermore, we revamp the T2I task as a multi-stage generation process, where at each stage, the user can insert, change, and move an object in 3D while preserving objects from earlier stages.
We achieve this through a novel Dynamic Self-Attention (DSA) module and a consistent 3D object translation strategy.
To evaluate our approach, we establish a benchmark and an evaluation protocol for interactive 3D layout control.
Experiments show that our approach can generate complicated scenes based on 3D layouts, outperforming the standard depth-conditioned T2I methods by two-folds on object generation success rate.
Moreover, it outperforms all methods in comparison on preserving objects under layout changes.
Project Page: https://abdo-eldesokey.github.io/build-a-scene/

Published as a conference paper at ICLR 2025
BUILD-A-SCENE: INTERACTIVE 3D LAYOUT CON- TROL FOR CARFUSION-BASEE IMAGE GENERATION
Abdelrahman Eldesokey & Peter Wonka King Abdullah University of Science and Technology (KAUST)
Thuwal, Saudi Arabia (first. last l @kaust.. edu.se sa
Dazy 1oom
whnt sofa
white soda
books
Wood table
malion
plant
Figure 1: Build-A-Scene is interactive diffusion- approach for image generation based on a user-provided 3D layout. At each generation stage, the user can control object type, location, and orientation (in-plane rotation) in 3D. Build-A-Scene ensures that objects are seamlessly integrated into the scene (see shadows and reflections) and preserve their identity under layout changes.
ABSTRACT
We propose a diffusion-based approach for Text-to-Image (T2I) generation with inteructive 3D layout control. Layout control has been widely studied to alleviate the shortcomings T2I diffusion models understanding objects' placement and relationships from text descriptions. Nevertheless, existing approaches for layout control are limited to 2D layouts, require the user to provide a static layout be- forehand, and fail preserve generated images under layout changes. This makes these approaches unsuitable for applications that require 3D object-wise control and iterative refinements, e.g., interior design and complex scene generation. To this end, we leverage the recent advancements in depth-conditioned T2I models and propose novel approach for interactive 3D layout control. We replace the traditional 2D boxes used in layout control with 3D boxes. Furthermore, we re- vamp the T2I task as multi-stage generation process, where at each stage, the user can insert, change, and move object in 3D while preserving objects from earlier stages. We achieve this through a novel Dynamic Self-Attention (DSA) module and consistent 3D object translation strategy. To evaluate approach, we establish a benchmark and an evaluation protocol for interactive 3D layout control. Experiments show that approach can generate complicated scenes based on 3D layouts, outperforming the standard depth-conditioned T2I meth- ods by swo-folds on object generation success rate. Moreover, it outperforms all methods in comparison on preserving objects under layout changes. Project Page: Ittps:: /tad:s://addeeee eldokeey girokkeeyggitbub.... www....hhuu....gg oo/buu 1 a- scene/

---

Published as a conference paper at ICLR 2025
20 Layour Guidance
X
3D Boxes
Interactive
Layout control
"A Windows and beige cartaiinaad
"Auur room wtth windon und bege
"Acase room with wendono be anda ficenllnna
Conse
3D Boxes
Interactive
x
Layout control
floor
Build A Scene (Ours)
3D Boxes
Interactive
Layout control
Figure 2: Existing 2D layout control approaches, i.e., Layout Guidance Chen al. (2024) accept static 2D layouts with no mechanism for 3D control or interactively changing the layout while preserving the image. LooseControl Bhat et al. (2024) can generate images conditioned on 3D boxes but cannot interactively move objects or handle layout with diverse objects. Build-A-Scens the first approach to support 3D layouts and allows users manipulate them interactively.
INTRODUCTION
Recent advancements in diffusion models (Rombach et al. 2022 Ramesh al. 2022 Saharia al. 2022 Podell et al. 2024) have profoundly revolutionized image generation, making it a pivotal task various creative domains, including design, art, and media production. Image diffusion models excel at generating high-quality images that adhere to a given textual prompt, effectively describing the image contents. This enhances the capability and efficiency of creators by enabling them to generate concepts and designs solely by describing their thoughts in text. However, multiple studies (Chen al., 2024] Lian et al., 2023) Feng et al.) 2023) have shown that diffusion models struggle to follow textual prompts accurately. More specifically, they encounter difficulties following object count, comprehending object placement, and understanding the relationship between objects.
Several approaches (Chen et al. 2024; Lian et al. 2023
Feng et al., 2023 Couairon et al. 2023
Xie al., 2023) Gani al., 2024) attempted to alleviate these shortcomings by investigating 2D layout control. These approaches require the user to provide a layout describing each object's 's size, shape, and location in the image alongside their respective textual description. Nonetheless, existing approaches for layout control adopt 2D inputs such points, bounding boxes, segmentation maps with no mechanism to position objects in 3D. This limits the creators' controllability in applications that require 3D control over the location and orientation (in-plane rotations) of objects, such as interior design and complex scene generation. Moreover, existing approaches for layout control require the user provide static layout beforehand and would fail to preserve the generated image under any layout changes, e.g., moving or scaling an object. Figure 2 shows an example of this scenario for the 2D layout approach, GuiduneG Chen et al. (2024). where moving the box that contains the sofa leads to changes in the sofa itself and the rest of the scene (the door).
To control the position of objects in 3D, several depth-conditioned diffusion models (Zhang al.,
2023 Mou et al. 2023) have been proposed to generate images for given depth maps. Furthermore, LooseControl (LC) (Bhat et al. 2024) introduced the use of rendered 3D boxes and planes as conditioning signal for T2I models, enabling control over the location and orientation of objects in 3D. However, LooseControl was not designed to follow a layout with diverse set of objects and relies solely on textual prompts to describe the image's contents. It is often observed that when dealing with multiple types of objects, some objects are either omitted or placed in an incorrect location, demonstrated in Figure 2 In addition, any changes to the 3D boxes used guidance alter the generated objects and might cause artifacts. This limits the usability of LC generating complex scenes with diverse objects.
2

---

Published as a conference paper at ICLR 2025
We introduce Build-A-Scena an interactive training-free approach for T2I with 3D layout control. Our approach formulates the image generation process as a multi-stage building process where the user starts with empty scene and populates it using interactive 3D layout. We achieve this by leveraging the existing depth conditioning model I.C to replace the 2D bounding boxes in layout control with 3D boxes. Furthermore, allows seamlessly adding objects to scene while preserving the existing contents, Additionally, we introduce a strategy for consistent 3D translation that preserves the identity of objects under layout changes. To evaluate our approach, we establish benchmark and an evaluation protocol for the task of interactive 3D layout control. Experiments show that our approach can generate complicated images and outperforms LC by factor of 2 object generation success rate and even on adherence to the user-provided 3D boxes. It even outperforms the 2D Layout-Guidance by ~ 15% on object generation success rate despite being training and guidance-free. Moreover, it outperforms Layout- Guidance and LC in preserving objects under layout changes on all metrics.
propose Dynamic Self-Attention (DSA) module that
2 RELATED WORK
In this section, we give a brief overview of existing approaches for 2D layout control. Since we introduce new strategy for preserving object identity under layout change, deseribe existing approaches for the task of consistent object generation in diffusion- based Text-to-Image (T2I).
2.1 LAYOUT CONTROL IN T21 DIFFUSION MODELS
The objective of layout control is to allow the user to explicitly specify where each element of the generated image should be placed. Existing approaches employ 2D layouts with various types of annotations, including points, scribbles, bounding boxes, and segmentation masks. These layouts are incorporated into the image generation process either by fine-tuning the pre-trained diffusing models to incorporate them as additional conditions or in a training-free manner. (Yang et al. 2023b) Zheng et al. 2023) trained additional modules to incorporate the layout as coordinates into pre-trained diffusion model. (Li al. 2023 Zhou al., 2024 Nie al. 2024; Avrahami al. 2023b) train different modules to condition the image generation process on bounding boxes, dense blobs, and other types of grounding data. SceneComposer (Zeng et al. 2023) introduced different levels of semantic layouts ranging from text to fine segmentation maps by fine-tuning pre-trained diffusion model on a richly annotated dataset. This category of approaches requires fine-tuning of pre-trained diffusion models, which comes computational and data annotation cost.
The training-free approaches attempted to solve the problem in a zero-shot manner to avoid com- plexities associated with fine-tuning diffusion models. (Xie al.) 2023 Chen et al. 2024 Liu et al. 2024) employed guidance strategies over the cross-attention responses to steer the denoising process direction that fulfills the layout specification. For finer level of layout control, Zest (Couairon et al.) 2023) employed layout segmentation maps and utilized a segmentation model to guide the diffusion process align with the segmentation maps. None of these aforementioned approaches are capable of controlling object location and orientation in a 3D scene, i.e., 3D layout control. Moreover, they expect the user to provide a static layout beforehand and do not offer any mechanisms to change layout elements while preserving the rest of the image. We attempt to tackle these shortcomings by introducing the first interactive 3D layout control approach for T2I.
2.2 CONSISTENT OBJECT GENERATION IN T2I
Several approaches have studied the problem of consistent object generation, i.e., personalized gen- eration, in diffusion models to generate consistent variations of specific object. (Ruiz et al. 2023] Hu et al., 2021] Wang et al.] 2023) fine-tunes a pre-trained diffusion model on set of images of specific subject, which allows for generating consistent images of the subject as specified by the text prompt. (Ye et al. 2023 Ma et al. 2024 Wang et al.) 2024 Song et al. 2023) followed a different approach and trained adapters to condition pre- pre-ted diffusion models on a single image of the subject. (Yuan et al. 2023) finetuned a diffusion model to be conditioned on a rigid transformation matrix describing the pose of the object. These aforementioned approaches are focused on person- alized generation given user-provided images and textual prompts. In contrast, our approach aims
3

---

Published as a conference paper at ICLR 2025
cocy living noom
STAGEE
1/a
RAS
RONDOR
PENDER
D
StAce
Self- tantion
14 Ki......... A2
(9f Of-1....000 01
P. PR
C
Poin wise multi mulippicaiionn 2 Concaterate Merne multplixation 5M Sofonax ① Summation
Cartesion Coo riloates
BAS
Dynamic Self-Attention (0SA)
Blending
Warp Object
Kis
M
4F
DDD Eae essiO
x
STAGG
1o .P...... A5
BAS
Segment Object
Consistent Transiation
ROVDER
A white sofa
To the next STAGB
Figure 3: An overview of Build-A A-Scene (BAS). We formulate the Text-to-Imant Image (T2I) task as a multi-stage generation process. We illustrate 3-stage scene. At STAGE O, the user defines empty scene with full control over scene size and camera parameters. At STAGE 1, the user adds an object (a white sofa) by defining box and its corresponding prompt. Our proposed Dynamic Self-Attention (DSA) module, coupled with latent blending, ensures that the object is seamlessly integrated into the scene while preserving the existing contents from STAGE 0. STAGE
illustrate our Consistent 3D Translation strategy that allows moving the object in 3D while preserving its identity. Note: DDIM inversion the reverse sampling process of the diffusion model Song al. 2021). and the Object Warping operation is explained in Appendix
to generate image based on a 3D layout and textual prompts. However, aim allow users to change the layout while preserving the generated image, similar to personalized generation.
Another category of training-free frree approaches for consistent object generation manipulates self- attention to preserve image consistency (Cao et al. 2023 Khachatryan et al.) 2023 Qi et al.) 2023). The image style be preserved by injecting the keys and values from reference image into the self-attention layers of the generated image. However, these approaches are designed to preserve the overall style, but the details of every individual object are fully preserved (see Figure We propose novel self-attention module that allows seamlessly inserting objects in an existing scene without altering the existing image contents.
METHOD
In Text-to- Image (T2I) diffusion models, the objective is generate image I given provided textual prompt P. In our work, a layout is specified by the user in the form of 3D bounding boxes B = B B² ,,,...,,B B²) and their corresponding prompts setting, the goal is to generate an image where the content enclosed within each bounding box ad- heres its respective prompt. More specifically, have two objectives: Establish interactive 3D layout pipeline. 2) Ensure object consistency under layout changes.
{P1, p2 :.... P*). Under this
3.1
FROM 2D TO 3D LAYOUT CONTROL
The first step to establishing 3D layout control is finding an appropriate form of 3D annotations that the user can easily create. We leverage LooseControl (Bhat et al. 2024) (LC) for this purpose as it accepts rendered 3D boxes and planes as conditioning signal in addition to a text prompt P. We define the 3D layout as empty 3D cuboid where the user can add planes to define the boundaries and 3D boxes to define the elements of the scene. We refer to the set of 3D boxes as B and their corresponding prompts P as explained above. Unlike existing 2D layout approaches that require the user to provide the entire layout beforehand, we propose a novel paradigm for layout control by revamping image generation sequential process. The user starts with empty scene and
4

---

Published as a conference paper at ICLR 2025
Xeference Image
Standard A ttenton
Croes Prame ttention
Exteended tbention
DSA (Ouns)
Rafanaso Dusth
Toryel
Figure 4: comparison between existing self-attention mechanisms and our proposed Dynamic Self-Attention (DSA).
interactively adds objects to it through multiple generation stages i E n]. At each stage, the user has control over single object and can change its type, size, 3D location, and orientation. This greatly enhances user controllability and customizability over the standard layout control. Figure illustrates our pipeline.
the first stage, i.e., STAGE 0, the background is generated based on an initial prompt that we denote p0 and rendered depth map of the layout D with only planes to define the boundaries. At this stage, the user has full control over the camera model, intrinsic parameters, and viewpoint, providing enhanced 3D control over the generated scene, unlike the existing 2D layout control approaches that lack any control of the camera. At the following generation stages, STAGE i O, the user adds 3D box B' and specifies its corresponding prompt P The 3D scene is rendered to obtain a depth map D' background and foreground masks for the box being generated at this stage, which we denote as Ga and Mr During the diffusion process at STAGE i, an initial latent code zy drawn from random Gaussian noise distribution. DDIM (Song al. 202T) is used iteratively denoise the latent code through multiple denoising steps t 0:
r/-1 vaa- $0
+ a-11 of c0 + oie.
(1)
at c D")
(2)
var
where Oee oi are the parameters of noise scheduler, c) is the noise prediction from the diffusion model, and e, is random Gaussian noise.
3.2 INTERACTIVE 3D LAYOUT CONTROL
At STAGE >> O, we aim generate a new object based on the 3D box B and the textual prompt pi At the same time, the object is desired to be seamlessly integrated into the scene while preserving the existing contents from previous stages. This can typically be done through inpainting or blended diffusion (Avrahami et al., 2023a). However, they require the user to provide free-form inpainting mask per object, which is laborious, and they are not directly compatible with the depth-conditioned LC. We propose novel technique for this purpose that is based on manipulating the self-attention maps and the latent codes.
In standard image diffusion models with UNet backbone, i.e., Stable Diffusion 1.5 (Romaach et al. (002),, each residual block has self-attention modules, which were found to encode the style and the structure of the generated image (Tumanyan et al. 2023). Self-attention for given block and timestep / at STAGE is computed as:
Qt Ki
A = Softmax
V
Vdns
Qi ft Wo:
Ki WK.
V V²= f Wv.
where fi are the intermediate UNet features, and W are trainable projection matrices.
A widely used approach transfer the style of reference image to target image is cross-frame attention (Coo et al. 2023 Khachatryan et al. 2023). which replaces K and V in Equation (3) with those of the reference image, i.e., RT-1 and 7-- This suggests that the target image will query

---

Published as a conference paper at ICLR 2025
the reference image for style, resulting in a consistent style between the two images. This approach was adopted in LC, but it was found to be incapable of generating new objects with a different style, as shown in Figure This is intuitive as limit the target image to copy the style exclusively from the source image. An alternative approach is the extended attention adopted in Qi et al. (2023) for performing consistent video edits, where the target image queries style from multiple images. Figure 4 shows that this strategy can generate objects with a new style but deviates from the overall style of the reference image as different styles are mixed in an uncontrolled manner.
We propose a Dynamic Self-Attention (DSA) technique, which is able to freely generate an object with new style while preserving the existing elements of the image. We achieve this by augmenting the attention keys to include the keys of STAGE - and masked window of the keys of STAGE i:
K;- K
2 [ T
@ Mbal],
1/1
[Vi-1]
@ [V @ MMalll
(4)
Qi
K
A Softmax
V
where @ is the concatenation operator, and @ point-wise product. This enforces the diffusion model to copy the overall style of the previous stage and allows it to generate a new style within the box the current stage. is noteworthy that our approach is plug-and-play into the pre-trained dif- fusion model and does not require any finetuming. Moreover, it does not rely guidance similar to (Chen et al., 2024], which requires backpropagating through the diffusion model at some iterations, adding computation overhead.
To enhance the preservation of the background of the previous stage (especially when the latent distribution changes between stages), we blend the latent codes as follows:
(5) Similarly, the features can be blended for more preservation. Finally, to harmonize the colors of the scene, we follow (Yang et al.) (223a)), and we optionally perform AdaIN (Huang & Belongie| 2017) between x;- and zi. We obtain the final image I' for STAGE at the end of the denoising process.
x Mh x,- + MG x;
3.3 CONSISTENT 3D TRANSLATION
A major limitation of existing layout control approaches their inability to preserve objects under layout changes, iee, scaling and translation. LCC also suffers from the same problem as demonstrated in Figure? This was attributed to the distributional shift of the latents that are aligned with the object before and after the layout change (Eldesokey & Wonka 2024). Therefore, we propose a strategy for preserving objects under layout changes, i.e., 3D translation.
To translate object at STAGE i, we start by segmenting the object out of the generated image in the previous stage 7t-1, To achieve this without any user intervention, we first obtain a coarse segmen- tation by accumulating the cross-attention maps that correspond to the object token in pi-1 similar to (Cao et al. 2023 Hertz et al. (022). Then, we fit a bounding box to this coarse segmentation and use it as input to SAM (Kirillov et al. 2023) to obtain fine segmentation map 5 After segmenting the object, we construct a warped image I" of the object after layout change by pasting the segmented object on the generated image from STAGE i 2. To simulate the 3D translation accurately in the image plane, we use the 3D Cartesian coordinates map of the object box before and after the translation warp the object. More specifically, we compute correspondences for the 4 corners of the objects between the two Cartesian maps and use them to warp the object in the im- age plane. We also use the same technique to warp the segmentation map S to the new location producing St. A detailed illustration is provided in the Appendix.
By inverting the warped image through DDIM inversion, we obtain an approximate trajectory 2T, 27--1.... 2z of the latents corresponding to the object after changing the layout. Finally, we blend the latents between z:"' and = as follows:
x S' x. + +1 (1--5 )'
This blending allows the diffusion model to regenerate the object of interest at the new location while preserving the background. We perform this blending for number of timesteps T <= T.
6

---

Published as a conference paper at ICLR 2025
Layour Conrol
Object Consistoncy
RT
CLiPra
0A 1
mIOU
CLiPaai
SSIM
PSNR
3D-Dir 1
3D-Rot
Layout-Uuidance
0,323
48.2
0.425
0.838
0.189
28.35
0.48
29'
12
LooseControl
0.302
24.3
0.635
0.924
0.367
29.12
0.65
27
2.
Ours
0,321
53.5
0.772
0.924
0.463
29.3
0.88
22"
65
Table I: A quantitative comparison between our proposed approach, the 2D layout control method Layout- Guidance Chen al. (2024). and LooseControl [Bhat al. (2024). RT refers to runtime.
4 EXPERIMENTS
In this section, provide qualitative and quantitative evaluation of our proposed approach. For clearer insights, we split the evaluation into two sub-tasks: (1) 3D layout control and (2) object consistency under layout change.
4.1 EXPERIMENTAL SETUP
Comparison We compare against the baseline LooseControl (LC) Bhat et al. (2024) that we employ for depth conditioning in our pipeline. To show where our approach stands with respect to 2D layout approaches, also compare against Layout-Guidanee Chen al. (2024) that accepts 2D bounding boxes. To map the 3D boxes to 2D, we fit bounding box to box masks Myc.
Implementation Details LC is based ControlNet with Stable Diffusion v1.5 Rombach et al. (022) that is fine-tuned through LoRA adaptation Hu et al. (2021). We keep all the original settings DDIM sampler with a linear schedule noticed that it is more stable. We perform T = 20 denoising steps in the quantitative comparison for efficiency and T' 40 for the qualitative results for better quality. For Layout-Guidance we use the official implementation with the default parameters. The source code and the evaluation protocol are publicly available.
of LC except for the sampler that changed
4.2 3D LAYOUT CONTROL EVALUATION
In this task, given a set of 3D bounding boxes and their corresponding prompts, the goal to generate an image that conforms to these inputs.
Evaluation Strategy Since there exist no criteria for evaluating 3D layout control, define new evaluation protocol for 3D layout control inspired by its 2D counterpart (Chen et al.) 2024 Xie al. 2023). We define a set of 16 objects from the MS COCO dataset (Lin al. 2014) and their corresponding aspect ratios. In addition, we define 10 different prompts for diverse scenes such as desert, snow, and room to use for the initial prompt p0 We start by sampling a random scene, object, and create 3D box that matches the object's aspect ratio at random z-coordinate. Then, sample another object, and we randomly select one the three placements ['left", "right", "above] relative to the first object. We place the second object into the scene similarly, ensuring that it does not occlude the first object or out of bounds.
We sampled 100 random layouts and each layout with 5 different seeds for fairness. Since the baseline L.C does not accept layouts, we automatically create textual description pa of the scene in the form:
p "[P"] with [P*] on the left/right and [P*]on the right/left"
Similarly, create the textual description for the relation "above" well.
Evaluation Metrics We are interested in evaluating three aspects: how the generated image con- fronts to the textual description of the scene, whether all specified objects in the layout have been generated, and how well each object fits within its box.
CLJPr21: We compute the CLIP score (Radford et al. 2021 between the final generated image and the textual description of the image P*.
https://git https:://gttuub.com ctt.://oooooa/oooo/oao//oo/o/// dasokey/bui scene

---

Published as a conference paper at ICLR 2025
Leyout
Depth
Our (staca
(Staaee
Layorr-Guldane
LooseConteal
Moad
Mustung Our
Steeet
Rane
Bowl
Gloss
Tabbl
Grovelzne
read
Reg
Tidle
Sada
mome
Figure 5: A qualitative comparison on the task 3D layout control.
Object Accuracy (OA): We use general object detector, YOLOv8 (Reis et al. 2023 to check if objects specified by the layout are detected the image.
Mean Intersection-over- Union (mloU): We compute the intersection between the bounding box pre- dicted by YOLOv8 and a bounding box fitted to the 3D box the image plane. This tells how well the object is enclosed within the layout box.
Quantitative Results Table summarizes the quantitative comparison. Our approach scores two times higher than LC and 15% higher than Layout-Guudance Guidance Object Accuracy (OA), demonstrat- ing its effectiveness in executing the layout. Our approach also outperforms LC and on par with Layout-Guidance CLIP demonstrating that the generated image conforms better to the textual description. For the mIOU metric, our approach outperforms Layout-Guidance by a huge margin despite not incorporating any quidance-based techniques. Surprisingly, it also outperforms LC, de- spite the fact that we have not tuned it. We believe this improvement is caused by our dynamic self-attention that forces the generated objects lie within their respective boxes.
The figure shows that our approach is more faithful the layout compared Layout-Guidance, Guidance, while LC struggles gen- erate all objects. We show both STAGE 0 and STAGE of our approach to highlight how objects are seamlessly integrated into the scene between stages (notice reflections and shadows). We also provide some examples with advanced 3D control of camera and objects in Figure
Qualitative Results We provide a qualitative comparison in Figure 5
4.3 OBJECT CONSISTENCY UNDER LAYOUT CHANGE EVALUUATION
This task aims move scale object in the provided layout while preserving its identity.
8

---

Published as a conference paper at ICLR 2025
Low Camera, Zoom-n
High Camera, Zoom
Law Camera Zoom
High-Camera, zoom-out
Figure 6: Examples of the advanced 3D control over camera and object provided by our approach.
Layout
MOVE RIGHT
Figure 7: A qualitative comparison for object consistency under layout change.
Evaluation Criteria We randomly sampled one of the 16 MS COCO objects and placed it layout. Then we randomly selected one of the actions ["move left", "move right", zoom-in", "zoom-out"). We ensure that applying the action does not cause the object to out of bounds. Finally, compare the object's visual appearance before and after applying the action.
Evaluation Metrics Our goal is to evaluate how similar the object is before and after applying some layout action. We crop the object from the images and resize the cropped images to ensure they match. We compute the CLIP score CLIP Structural Similarity (SSIM), and Peak-Signe Soak-Signal-to- Noise-Ration Similarity (PSNR) between cropped image of the object before and after the action. We also propose 3D Adherence metrics that evaluate if the generated image adheres to the 3D layout change (see Appendix for details).
Quantitative Results Table [ summarizes the results averaged over 5 seeds. Our method outper- forms other methods all metries, demonstrating better preservation of objects and 3D adherence to the layout changes.
Qualitative Results We show some qualitative examples for layout changes in Figure For the scaling layout changes, iee. zoom-in and zoom-out, our approach applies them successfully and seamlessly inserts the object at the new location. LC either distorts the object does not apply the layout change, while Layout-Guidance changes the object completely. When moving the main object to the left or the right, our approach successfully applies the changes and preserves the object. On the other hand, LC distorts the object, while Aayout-Guidand changes the object pose.
4.4 ABLATION ANALYSIS
Figure 8 provides ablation analysis for the impact of different components of our pipeline. When the Dynamic Self-Attention (DSA) is disabled, the model is not capable of inserting new object into the reference image. Skipping the latent blending in Equation (5) causes some details of the background to change (the paintings on the wall). Using AdaIN (Huang & Belongie 2017) con- tributes to harmonizing the colors of the generated object with the background.

---

Published as a conference paper at ICLR 2025
Beferezce Image
Vil DSA
w/ Laber: Blending
MiS ddiN
Ours
SD Lapood Ooetea
Reforszee Imsge
Warped Image
to 0.47
up to 1.5
up 00
Oitect ConnAnryy
Figure 8: Ablation analysis for different parts our pipeline.
Prorpt: lorp post"
Prorpit
"The Jokeg"
"A Cararo
Figure 9: Limitations of our approach.
We also experiment with varying T in Section 3.3 for blending the latents. The warped image lacks realism, and the sofa appears to be floating. By applying Equation (6) for T 0.4T, the sofa is seamlessly integrated into the scene, but some of the details (the cushions) constructed. When T = 0.8T, the sofa seamlessly blended into the scene, and the fine details are well constructed. With T T', some artifacts start appear at the boundaries.
not perfectly
LIMITATIONS AND FUTURE WORK
Figure 9 shows some of the limitations of our approach. First, our approach is sensitive the aspect ratio of the box. If provided with a box wider than the actual width of the object, it can generate two instances the object (left figure). When the aspect ratio of the box is not suitable for the specified prompt, it can generate distorted object or a photo of the object (middle figure) We believe that this is natural behavior of our approach as tries fulfill the layout and the prompt requirements concurrently. Secondly, if a large object is placed in small space, i.e. box intersecting with the boundaries, such as a car room, the out-of-boundary parts of the car are distorted (right figure). In general, the definition of boxes needs to be reasonable to obtain the desired results.
For the Consistent 3D Translation strategy, if the object segmentation part fails, it becomes infeasi- ble preserve the objects. Finally, one might argue that the multi-stage generation pipeline adds computational overhead to the generation process. However, this is a fair price in return for the en- hanced control over scene elements that helps the user reach the desired output faster and eventually save time. Moreover, our approach takes seconds per stage and can generate a layout of objects in the same time Layout-Guidanne (see Table
Future Work We would like to investigate automated layout generation through large-language model (LLM) in (Feng et al. 2023 Gani al. 2024). Another direction is supporting in in-plane rotations while preserving object identity, similar to (Yuan et al. 2023
CONCLUSION
We presented first approach for interactive 3D layout control based on a pre-trained T2I diffusion model. Our approach reformulated image generation as multi-stage process, providing users with enhanced control over individual objects 3D. Moreover, we provided the first strategy to preserve objects under layout changes. Experiments show that our approach outperformed both LooseControl and the recent 2D layout control, both quantitatively and qualitatively. We hope that our approach will establish research direction for 3D layout control and consistency in layout control.
10

---

Published as a conference paper at ICLR 2025
7 ACKNOWLEDGEMENT
The work is supported by funding from KAUST Center of Excellence for Generative AI, under award number 5940, and the NTGC-AI program.
REFERENCES
Omri Avrahami, Ohad Fried, and Dani Lischinski. Blended latent diffusion. ACM transactions graphics (TOG), 42(4):1-11, 2023a.
Omri Avrahami, Thomas Hayes, Oran Gafni, Sonal Gupta, Yaniv Taigman, Devi Parikh, Dani Lischinski, Ohad Fried, and Xi Yin. Spatext: Spatio-textual representation for controllable im- age generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13770-1380,, 2023b.
Shariq Farooq Bhat, Niloy J Mitra, and Peter Wonka. Loosecontrol: Lifting controlnet for general- ized depth conditioning. In ACM Siggraph Conference, 2024.
Garrick Brazil, Abhinav Kumar, Julian Straub, Nikhila Ravi, Justin Johnson, and Georgia Gkioxari. Omni3D: large benchmark and model for 3D object detection in the wild. In CVPR, Vancouver, Canada, June 2023. IEEE.
Mingdeng Cao, Xintao Wang. Zhongang Qi, Ying Shan, Xiaohu Qie, and Yinqiang Zheng. Mas- actrl: Tuning- free mutual self-attention control for consistent image synthesis and editing. arXiv 2023.
preprint arXiv:2304.0844
Minghao Chen, Iro Laina, and Andrea Vedaldi. Training-free layout control with cross-attention guidance. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 5343-5353, 2024.
Guillaume Couairon, Marlène Careil, Matthieu Cord, Stéphane Lathuilière, and Jakob Verbeek. Zero-shot spatial layout conditioning for text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2174-2183, 2023.
Abdelrahman Eldesokey and Peter Wonka. Latentman: Generating consistent animated characters using image diffusion models. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 7510-7519, 2024.
Weixi Feng, Wanrong Zhu, Tsu-jui Fu, Varun Jampani, Arjun Akula, Xuehai He, Sugato Basu, Xin Eric Wang, and William Yang Wang. Layoutgpt: Compositional visual planning and genera- tion with large language models. arXiv preprint arXiv:2305.15393, 2023.
Hanan Gani, Shariq Farooq Bhat, Muzammal Naseer, Salman Khan, and Peter Wonka. Llm blueprint: Enabling text-to-image generation with complex and detailed prompts. In ICLR, 2024.
Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfrr Aberman, Yael Pritch, and Daniel Cohen-Or. Prompt-to-promp image editing with cross attention control. arXiv preprint wrXiv:2208001220 2022.
Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09635 2021.
Xun Huang and Serge Belongie. Arbitrary style transfer in real-time with adaptive instance normal- ization. In Proceedings of the IEEE international conference on computer vision, pp. 1501-1510, 2017.
Levon Khachatryan, Andranik Movsisyan, Vahram Tadevosyan, Roberto Henschel, Zhangyang Wang, Shant Navasardyan, and Humphrey Shi. Text2vieeo zero: Text-to-image diffusion models zero-shot video generators. arXiv preprint arXiv:2303. 13439, 2023.

---

Published as a conference paper at ICLR 2025
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg. Wan- Yen Lo, et al. Segment anything. arXiv preprint ar wwXiv:2304.02644 2023.
Yuheng Li, Haotian Liu, Qingyang Wu, Fangzhou Mu, Jianwei Yang, Jianfeng Gao, Chunyuan Li, and Yong Jae Lee. Gligen: Open-set grounded text-to-image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 222122222522 2023.
Long Lian, Boyi Li, Adam Yala, and Trevor Darrell. Llm-grounded diffusion: Enhancing prompt understanding of text-to-image diffusion models with large language models. arXiv preprint arXiv:2305. 13655, 2023.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pp. 740-755. Springer, 2014.
Jiaqi Liu, Tao Huang, and Chang Xu, Training-free composite scene generation for layout-to-image synthesis. 2024. URL BtLhttts::// /ttp//aaivvv....///i//ea./aa.av/ org/ abs/a407..oo 13609
Jian Ma, Junhao Liang, Chen Chen, and Haonan Lu. Subject-diffusion Open domain personalized text-to-image generation without test-time fine-tuning. In ACM SIGGRAPH 2024 Conference Papers, pp. 1-12, 2024.
Chong Mou, Xintao Wang, Liangbin Xie, Yanze Wu, Jian Zhang, Zhongang Qi. Ying Shan, and Xiaohu Qie. T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models. arXiv preprint arXiv:2300..0 022300000844,,. 2023.
Weili Nie, Sifei Liu, Morteza Mardani, Chao Liu, Benjamin Eckart, and Arash Vahdat. Composi- tional text-to-image generation with dense blob representations. In International Conference on Machine Learning (ICML), 2024.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. SDXL: Improving latent diffusion models for higl-resolution image synthesis. In The Twelfth International Conference on Learning Representations, 2024.
Chenyang Qi Xiaodong Cun, Yong Zhang, Chenyang Lei. Xintao Wang, Ying Shan, and Qifeng Chen. Fatezero: Fusing attentions for zero-shot text-based video editing. 23Xiv:2303.09535, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp- 8748-8763. PMLR. 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06122 1(2):3, 2022.
Dillon Reis, Jordan Kupec, Jacqueline Hong, and Ahmad Daoudi. Real-time flying object detection with yolov8. arXiv preprint arXiv:2305.0997,. 2023.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings the IEEF/CVF confer- ence on computer vision and pattern recognition, pp. 10684-10695, 2022.
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In Pro- ceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 22500- 22510, 2023.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Bureu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Informa- fion Processing Systems, 35:36479-36494 2022.
12

---

Published as a conference paper at ICLR 2025
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In Interna- tional Conference on Learning Representations, 2021. URL lattps://ppeerreiieeeee net/
Eorum?]d=StlgiarrHH
Yizhi Song, Zhifei Zhang, Zhe Lin, Scott Cohen, Brian Price, Jianming Zhang. Soo Ye Kim, and Daniel Aliaga. Objectstitch: Object compositing with diffusion model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18310-18319, 2023.
Narek Tumanyan, Michal Geyer, Shai Bagon, and Tali Dekel. Plug-and-play diffusion features for text-driven image-to-image translation. In Proceedings of the IEEE/CVF Conference Com- puter Vision and Puttern Recognition (CVPR), pp. 1921-1930, June 2023.
Qixun Wang, Xu Bai, Haofan Wang, Zekui Qin, and Anthony Chen. Instantid: Zero-shot identity- preserving generation in seconds. arXiv preprint arXiv:240..07519 2024.
Zhonghao Wang, Wei Wei, Yang Zhao, Zhisheng Xiao, Mark Hasegawa-Johnson Humphrey Shi, and Tingbo Hou. Hifi tuner: High-fidelity subject-driven fine-tuning for diffusion models. arXiv preprint arXiv:23/2.00073 /2.00079, 2023.
Jinheng Xie, Yuexiang Li, Yawen Huang, Haozhe Liu, Wentian Zhang, Yefeng Zheng, and Mike Zheng Shou. Boxdiff: Text-to-image synthesis with training- free box-constrained diffusion. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 7452-7461, 2023.
Lihe Yang, Bingyi Kang, Zilong Huang, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything: Unleashing the power of large-scale unlabeled data. In CVPR, 2024.
Shuai Yang, Yifan Zhou, Ziwei Liu, , and Chen Change Loy. Rerender video: Zero-shot text- guided video-to-videe video translation. In ACM SIGGRAPH Asia Conference Proceedings, 2023a.
Zhengyuan Yang, Jianfeng Wang, Zhe Gan, Linjie Li, Kevin Lin, Chenfei Wu, Nan Duan, Zicheng Liu, Ce Liu, Michael Zeng, et al. Reco: Region-controlles text-to-image generation. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14246- 14255, 2023b.
Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang. Ip-adapter: Text compatible image prompt adapter for text-to- image diffusion models. arXiv preprint arXiv:2308.0672) 06721,, 2023.
Ziyang Yuan, Mingdeng Cao, Xintao Wang. Zhongang Qi, Chun Yuan, and Ying Shan. Customnet: sero-shot object customization with variable-viewpoints in text-to-image diffusion models, 2023.
Yu Zeng, Zhe Lin, Jianming Zhang, Qing Liu, John Collomosse, Jason Kuen, and Vishal M Patel. Scenecomposer: Any-level semantic image synthesis. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pp. 22468 -2448,, 2023.
Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models, 2023.
Guangcong Zheng, Xianpan Zhou, Xuewei Li, Zhongang Qi, Ying Shan, and Xi Li. Layoutdiffusion: Controllable diffusion model for layout-to-image generation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 22490-2222222 22499,, 2023.
Dewei Zhou, You Li, Fan Ma, Xiaoting Zhang, and Yi Yang. Mige: Multi-instance generation controller for text-to-image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 681 18888888,, 2024.
13

---

Published as a conference paper at ICLR 2025
LocseCortral
LooneContral
Curs
LoosaCantrol
ZOOM
XOOM OUT
200M DUT
ZOIC OUT
Figure 10: Additional results for our consistent layout change under zoom-in/out layout changes.
A APPENDIX
B ADDITIONAL QUALITATIVE RESULTS
Video results of the interactive generation process and more qualitative results are provided in the attached supplementary.
We also provide additional qualitative results for moving objects towards and away from the observer in Figure 10 (zoom-in and zoom-out).
C 3D ADHERENCE METRICS
We propose two metrics to assess the adherence of different approaches to the requested 3D layout changes. The first metric, which term 3D-Dir, evaluates whether the moved box is relocated in the correct direction. Given a layout change to box in image /1, we store the origin of the box that corresponds that object before and after applying the layout change on and 02, respectively. After generating new image 1n that corresponds to the layout change. we use a monocular depth estimation model, i.e. Depth-Anything Yang al. (2024), obtain depth maps for both images that we denote as Dj and D2. Then, based on the depth maps, we obtain predicted center point of the object before and after the layout change that we denote as P1 and P2. We compute this with the help of the 3D box masks M/ and Mf. before and after the layout change. Afterward, we define two directional vectors x ol o2, and = p1 p2 as reference and predicted trajectories. Finally, compute the cosine similarity between these two vectors to obtain the 3D Adherence score:
I
3D Adherence =
|vll
high score indicates that the reference and predicted vectors have a similar direction and magni- tude, meaning the desired layout change was executed correctly.
The second metric is 3D-Rot, which examines if the orientation of the generated object before and after the layout change aligns with the 3D box. For this purpose, employ the Omni3D Brazil et al. 2023) detector that predicts 3D bounding boxes for objects in images I1 and Iz. The D detector also predicts the pose the box in 6D representation that we transform Euler angles for comparison. We compute the difference between the Euler angles of the 3D box and the predicted 3D bounding box from / and /2 to obtain differences 0,a0, Opich Orell and 0% Ofiich Oroll Finally, we average over all these angle differences to obtain an average deviation in degrees. Our method scores the best these metries, and provide the results Table [ It is worth noting that Omni3D is less accurate than 2D object detectors, which leads to inflated error rates across all methods. Our approach achieves the lowest error relative to the other methods in comparison.
14

---

Published as a conference paper at ICLR 2025
Ara Segnentaton
Warp Object
DDIM Lay esion
x
Labet Bleeding
BAS
The Cartasian Coordinntes
Ca
S*
D
U2
%)
Figure 11: A detailed illustration of our Consistent 3D Translation strategy.
D OBJECT WARPING
We provide more detailed explanation of our 3D Consistent 3D Translation in Figure 11 To translate an object at STAGE with vector Л, we start by segmenting out the object based on the cross-attention maps of the object prompt pi-1 to provide coarse segmentation map (shown in yellow). We fit bounding box (shown in blue) the coarse segmentation map, and input prompt to the SAM segmentation model to provide a fine segmentation map 5i-1. Given the segmented object, since we apply the translations in 3D, there is a perspective change that can not be captured by directly cutting and pasting the object. Instead, we use the Cartesian coordinates of the rendered 3D box to warp the segmented object and the fine segmentation map to the new location in the warped image. To achieve this, We render the object box using ray-tracing before and after the 3D translation to obtain depth maps D D and Cartesian coordinates maps C C for the point- of-hit of the rays. We align C with Cx-1 by subtracting the translation vector T from the former. Finally, we warp the 4 corners of the segmented object based on the Cartesian coordinate maps to produce the warped image _ and warped mask S², This strategy ensures that the object follows the perspective change the 3D box and scales the object according to the translation in 3D, not the image plane.
it as
aim to paste the image from STAGE i 2 at the location. However,
Given the warped image I", we invert using DDIM to obtain a trajectory of latents x²". Then, we generate the final translated image I using latent blending based on Equation 6
E MULTI-STAGE GENERATION
To illustrate how our approach maintains scene consistency across different stages, we visualize individual stages for several examples in Fig. 12 The figure clearly showcases the effectiveness of our proposed Dynamic Self-Attention (DSA) module in seamlessly integrating new objects into the scene while preserving the original elements.
F EVALUATION PROTOCOL
We present the details for the benchmark and evaluation protocol that was used to evaluate approach Figure 13 First, we randomly select a scene from the 'scenes' list. For every scene, there is pre-defined list of objects that naturally appear in this scene. Based this pre-defined list, we select objects to added to the scene. To create a reasonable 3D box for every object, the aspect ratios for different objects generated by prompting ChatGPT, and then manually refining them.
15

---

Published as a conference paper at ICLR 2025
Build-A-Scere (Ours]
ttae
Stage 2
Staee
LocseContral
Dog
Elepbant
jungle
Treesure
SONG
Trach caz.
Coe
Nien stom
Chair
Bucket
Bail
beach
Rock
The
Batnoblle
Snowy mad
Stomtrooper
Darthaader
Specesaip
Figure 12: A demonstration for the effectiveness of our proposed Dynamic Self-Attention (DSA) in seamlessly inserting objects while preserving existing ones.
16

---

Published as a conference paper at ICLR 2025
object categories
"aninals": cat", dog", horse", elephant", "a grizzly bear"], "indoor": teddy bear", nicroweve", backpack", led tv", sofa", chair", table", bed"], "outdoor": car", notorcycle", backpack", bench", sofa"l,
(*An expty desert uith cloudy sky", ["antnals", "outdoor")),
(*An erpty roon with windows and curtains", ["indoor"]),
(*An erpty street", ["outdoor"]),
(*An expty Jungle", ["antnals"]),
(*An arpty road" ("aninals", ""ouddoo"]]
(*An enpty studio", ["indoor"]]
(*An expty beach", ("aninals"]),
(*A snowy landscape", ["autdoor"] }),
(*An empty apartment", ("indoor" }),
aspect ratios
width, Depth, Height
"a cat": (9.2, 8.2, 8.4),
"a dog": (8.3, 0.2, 8.5),
"a horse": (9.7, 0.3, 8.5),
"an elephant"; (8.9, 0.3, 8.6),
"a grizzly bear": (8.8, 0.3, 8.5),
"a teddy bear": (0.3, B.2, 8.4),
"a nicrovave"; (8.4, 8.2, 8,25),
"a backpack": (9.3, 8.1, 8.4),
"a car": (a.6, 1.2, B.4),
"a notoreycle"; (0.8, 8.2, 8.35),
"an led tv": (8.5, 8.85, 8.3),
"a sofa": (0.9, 8.3, 8.4),
"a chain": (0.25, 0.25,
"a table": (0.5, 0.5, 8.4),
"a bench": (0.7, 8.3, 8.3), "a bod": (0.6, 8.7, 0.25),
relations
"a cat": ('I", "r" "s"),
"a dog": ("1", "r', *s"),
"a horse": ("1", 'r"),
"an elephant": ("1", "r"),
"a grizzly bear": ("2", "r")
(r"",
"a teddy bear": (*1", "6", "s),
"a nicrowave": ("1", ("1", "r', ta),
"a backpack"; ("1", "6",
"a car"s ("1", "p"),
"a notorcycle": (?1", "e"),
"an led tv": ('1", 765 (8"),
"a sofa": ("1%, "p").
"a chain": (*1", tr").
"a table": (*1", "e"),
"a bench": (*1", "r"),
"a bed": ("1", "r"),
Objects and scenes used for the evaluation Protocol, TT, "T", and "a" in relations stand Figure 13:
for left, right, and above respectively.
The relationships between objects were selected based how they usually co-occur in nature. For instance, an LCD TV can be on top of another object, but an elephant can not.
G USER INTERFACE
To enable interactive scene building with our approach, we developed Gradio- web interface that is illustrated in Figure 14 At each stage, users create box and manipulate its scale,
17

---

Published as a conference paper at ICLR 2025
Build-A-Scene
Leyout
A Awhite sofa
42
Scale
20
65
20
Generate
Translation
Image
Ls
0
-00
20
180
180
-380
15
Next Stage
Start Over
Figure 14: The user interface for Build-A-Scene (BAS)
translation, and rotation in space. The box's contents are defined through a text prompt, and clicking the "Generate" button produces the corresponding image. Users then iteratively refine the box, modify the prompt, and adjust the seed to achieve the desired outcome. Once satistied with the current stage, users click the "Next Stage" button save the scene's state and proceed. A "Start Over" button is also provided to create a new scene from scratch.
18


Reviews:
[{"review_id": "7RNVksDJHT", "reviewer": "Reviewer_9MBn", "summary": "Based on the recent advancements in depth-conditioned T2I, this work presents a novel approach for interactive 3D layout control.     Further, the authors propose a Dynamic Self-Attention (DSA) module and a consistent 3D object translation strategy, to preserve the existing contents and consistent 3D translation.", "questions": "See Weakness.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The task is well-defined.\n2. The model is designed reasonably.\n3. The paper is well-written and clearly states the contribution.\n4. The results are well-organized, and the experiments are comprehensive.", "weaknesses": "1. 3D Awareness. Existing results are mainly about the 3D layout conditioned generation. However, the 3D information is not adequately used. There are several other degrees of freedom in 3D spaces, such as the camera view, the rotation of objects, and the zoom-in/out. These results (even parts) could strengthen this work a lot. \n\n2. User interface. What exactly is the interface? Can you describe the tools for users to use? The interface for the creator is critical, especially for 3D editing. A reasonably designed interface can make this work for real-world applications. Just a demo-based (actually controlled by codes) can not make it practical.\n\n3. The motivation. The motivations of the design of the pipeline is not very clear. The key feature of this work is that it can implement 3D layout-based generation. However, what are the challenges in this task, and why authors must use the proposed pipelines and modules to deal with this challenge?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FTcztVqNOx", "reviewer": "Reviewer_oNnn", "summary": "BAS introduces a pipeline which allows users to guide the image synthesis through interacting with boxes in the 3D space and using the text prompt. The boxes reflect the position of objects in the image. To keep the image consistency throughout interacting, the author propose the dynamic self attention and object wrapping in order to maintain the background as much as possible. As a new work to introduce 3D interacting into 2D image synthesis, it proposes several benchmarks to evaluate its performance and compare to some image synthesis works.", "questions": "No", "limitations": "", "rating": 3, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "Firstly, using inpainting to preserve the background during the interacting is a simple but sound idea, and the manipulations in 3D world make the masks’ calculation fully under control. Furthermore, 3D guidance is common and useful when we want to show the relative transformation of objects to the diffusion model, as we usually want it to give image of the real 3D world.", "weaknesses": "(1) The technical contribution seems not to be so sufficient. As the T2I techniques and pre-trained models are mature, would organizing the masks according to the boxes 3D transformation being sufficient for a paper? Well, the dynamic self-attention block is surely a contribution.\n(2) However, in the ablation study, it says “when DSA is disabled, the model isn’t capable of inserting new object”, but in the method, DSA is proposed for maintaining the background, which is different from what may happen in the ablation study. The technical contribution turns out to be confusing and doubtful. \n(3) Another consideration is that your run time is 3 times of “loose control”. Would that be harmful to the interaction? Could you please discuss more on that?\n(4) Some expression in the “METHOD” still tells why other methods aren’t suitable for this “attention”. That should not appear here.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "t8Og82vIPb", "reviewer": "Reviewer_JbbB", "summary": "This paper proposes a interactive 3D layout control approach for T2I based on diffusion approach.  Beginning with an generated scene images without objects inside, the proposed approach allows uses to add 3D box and its text description in the bounding box of the scene to specify the 3D layout of the scene.  The depth map of the 3D layout and the text description are used as the condition to guide diffusion model, and the keys are combined according to the mask of the box to maintain the style of the image in each stage, denoted as dynamic self-attention. \n\nThe video demo is nice, which demonstrates the 3D layout control process clearly.", "questions": "In the DSA model,  why do you only combine the key tokes between stage i-1 and stage i?  I am wondering the effect of combining V tokens instead.  Discussions on why chose key tokens in DSA is necessary for the deep undstanding the design of DSA.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. A straightforward yet effective approach to control the scene layout and the styles of objects placed in the scene.  With the 3D interface, the user can control how to put the objects on other objects or the relative position between objects, more natural than 2D interface.  The ability to move the object closer or farther away from the camera is also attractive. \n\n2. The DSA module designed to merge the keys at different stage is easy to implement, and it allows to generate new object of a different style. Also, the combination of latent codes according to the mask  and AdaIN operation can effectively maintain the image styles surrounding the newly inserted object.", "weaknesses": "The composes scene is relatively simple.  All objects are put on the flat surface, such as floor or desk top. It might be partially attributed to the box-based 3D layout control interface.  For example, can we put an object on the sofa if we use freeform surface as the control interface?  If freeform surface, how much effort does the user should pay to place a new object such that the diffusion model can generate the image correctly?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XU4k3AQpv1", "reviewer": "Reviewer_X2dB", "summary": "This work is trying to generate an image based on the layout of 3D boxes for each object we desire in the image. It builds upon a depth-conditioned stable diffusion generator, and they make it an iterable fashion where they focus on different object placement in each step to enhance control of each separate object. to keep the information from the last iteration to the next, they presented the Dynamic Self-Attention (DSA) module which blends between attention maps from the previous iteration and the new one, specifically on the mask where the new object is placed.", "questions": "from your pipeline figure (Fig.3) it seems like you didn't talk about some key parts of the method, specifically in stage 2 (warping, DDIM inversion) it might not be novel, but helps to grasp the notion of the method.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "they present an interesting 3D layout control over generated images, Furthermore, the multi-stage to have more control over each object sounds good.", "weaknesses": "1. you presented in most of the paper examples of moving the object aside (Fig.2), however, with 3D boxes, it is much more interesting to see examples of moving toward me or away from the observer. you provided only two examples like this in Fig.6, I would love to see it as the focus of the paper.\n2. your evaluation metrics (Table. 1) have no comparison to 3D information, which is the focus of the paper. i would love o see a new metric that can evaluate the 3D placement, like with monocular depth estimation, or 3D object detection networks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)