Build-A-Scene: Interactive 3D Layout Control for Diffusion-Based Image Generation

## Abstract

Abstract We propose a diffusion-based approach for Text-to-Image (T2I) generation with interactive 3D layout control .
Layout control has been widely studied to alleviate the shortcomings of T2I diffusion models in understanding objects’ placement and relationships from text descriptions.
Nevertheless, existing approaches for layout control are limited to 2D layouts, require the user to provide a static layout beforehand, and fail to preserve generated images under layout changes.
This makes these approaches unsuitable for applications that require 3D object-wise control and iterative refinements, e.g. , interior design and complex scene generation.
To this end, we leverage the recent advancements in depth-conditioned T2I models and propose a novel approach for interactive 3D layout control.
We replace the traditional 2D boxes used in layout control with 3D boxes.
Furthermore, we revamp the T2I task as a multi-stage generation process, where at each stage, the user can insert, change, and move an object in 3D while preserving objects from earlier stages.
We achieve this through our proposed Dynamic Self-Attention (DSA) module and consistent 3D object translation strategy.
Experiments show that our approach can generate complicated scenes based on 3D layouts, boosting the object generation success rate over the standard depth-conditioned T2I methods by 2 × 2\times 2 × .
Moreover, it outperforms other methods in comparison in preserving objects under layout changes.
Project Page: https://abdo-eldesokey.github.io/build-a-scene/

## 1 Introduction

Figure: Figure 2: Existing 2D layout control approaches, i.e., Layout Guidance Chen et al. () accept static 2D layouts with no mechanism for 3D control or interactively changing the layout while preserving the image. LooseControl Bhat et al. () can generate images conditioned on 3D boxes but cannot interactively move objects or handle a layout with diverse objects. *Build-A-Scene* is the first approach to support 3D layouts and allows users to manipulate them interactively.
Refer to caption: x2.png

Recent advancements in diffusion models Rombach et al. (); Ramesh et al. (); Saharia et al. (); Podell et al. () have profoundly revolutionized image generation, making it a pivotal task in various creative domains, including design, art, and media production.
Image diffusion models excel at generating high-quality images that adhere to a given textual prompt, effectively describing the image contents.
This enhances the potency and efficiency of creators by enabling them to generate concepts and designs solely by describing their thoughts in text.
However, multiple studies Chen et al. (); Lian et al. (); Feng et al. () have shown that diffusion models struggle to follow textual prompts accurately.
More specifically, they encounter difficulties following object count, comprehending object placement, and understanding the relationship between objects.

Several approaches Chen et al. (); Lian et al. (); Feng et al. (); Couairon et al. (); Xie et al. (); Gani et al. () attempted to alleviate these shortcomings by investigating *2D layout control*.
These approaches require the user to provide a layout describing each object’s size, shape, and location in the image alongside their respective textual description.
Nonetheless, existing approaches for layout control adopt 2D inputs such as points, bounding boxes, or segmentation maps with no mechanism to position objects in 3D.
This limits the creators’ controllability in applications that require 3D control over the location and orientation (in-plane rotations) of objects, such as interior design and complex scene generation.
Moreover, existing approaches for layout control require the user to provide a static layout beforehand and would fail to preserve the generated image under any layout changes, e.g., moving or scaling an object.
Figure 2 shows an example of this scenario for the 2D layout approach, Layout-Guidance Chen et al. (), where moving the box that contains the sofa leads to changes in the sofa itself and the rest of the scene (the door).

To control the position of objects in 3D, several depth-conditioned diffusion models Zhang et al. (); Mou et al. () have been proposed to generate images for given depth maps.
Furthermore, LooseControl (LC) Bhat et al. () introduced the use of rendered 3D boxes and planes as a conditioning signal for T2I models, enabling control over the location and orientation of objects in 3D.
However, LooseControl was not designed to follow a layout with a diverse set of objects and relies solely on textual prompts to describe the image’s contents.
It is often observed that when dealing with multiple types of objects, some objects are either omitted or placed in an incorrect location, as demonstrated in Figure 2.
In addition, any changes to the 3D boxes used as guidance alter the generated objects and might cause artifacts.
This limits the usability of LC in generating complex scenes with diverse objects.

We introduce *Build-A-Scene*, an *interactive* training-free approach for T2I with *3D layout control*.
Our approach formulates the image generation process as a sequential building process where the user starts with an empty scene and populates it using an interactive 3D layout.
We achieve this by leveraging the existing depth conditioning model LC to replace the 2D bounding boxes in layout control with 3D boxes.
Furthermore, we propose a Dynamic Self-Attention (DSA) technique that allows seamlessly adding objects to a scene while preserving the existing contents.
Additionally, we introduce a strategy for consistent 3D translation that preserves the identity of objects under layout changes.
Experiments show that our approach can generate complicated images and outperforms LC by a factor of 2 on object generation success rate and even on adherence to the user-provided 3D boxes.
It even outperforms Layout-Guidance by $\sim 15\%$ on object generation success rate despite being training and guidance-free.
Moreover, it outperforms Layout-Guidance and LC in preserving objects under layout changes on all metrics.

## 2 Related Work

In this section, we give a brief overview of existing approaches for 2D layout control.
Since we introduce a new strategy for preserving object identity under layout change, we describe existing approaches for the relevant task of consistent object generation in diffusion-based Text-to-Image (T2I).

### 2.1 Layout Control in T2I Diffusion Models

The objective of layout control is to allow the user to explicitly specify where each element of the generated image should be placed.
Existing approaches employ 2D layouts with various types of annotations, including points, scribbles, bounding boxes, and segmentation masks.
These layouts are incorporated into the image generation process either by fine-tuning the pre-trained diffusing models to incorporate them as additional conditions or in a training-free manner.
Yang et al. (); Zheng et al. () trained additional modules to incorporate the layout as coordinates into a pre-trained diffusion model.
Li et al. (); Zhou et al. (); Nie et al. () train different attention modules to condition the image generation process on bounding boxes, dense blobs, and other types of grounding data.
SceneComposer Zeng et al. () introduced different levels of semantic layouts ranging from text to fine segmentation maps by fine-tuning a pre-trained diffusion model on a richly annotated dataset.
This category of approaches requires fine-tuning of pre-trained diffusion models, which comes at a computational and data annotation cost.

The training-free approaches attempted to solve the problem in a zero-shot manner to avoid complexities associated with fine-tuning diffusion models.
Xie et al. (); Chen et al. (); Liu et al. () employed guidance strategies over the cross-attention responses to steer the denoising process in a direction that fulfills the layout specification.
For a finer level of layout control, Zest Couairon et al. () employed layout segmentation maps and utilized a segmentation model to guide the diffusion process to align with the segmentation maps.
None of these aforementioned approaches are capable of controlling object location and orientation in a 3D scene, i.e., 3D layout control.
Moreover, they expect the user to provide a static layout beforehand and do not offer any mechanisms to change layout elements while preserving the rest of the image.
We attempt to tackle these shortcomings by introducing the first interactive 3D layout control approach for T2I.

### 2.2 Consistent Object Generation in T2I

Several approaches have studied the problem of consistent object generation, i.e., personalized generation, in diffusion models to generate consistent variations of a specific object.
Ruiz et al. (); Hu et al. (); Wang et al. () fine-tunes a pre-trained diffusion model on a set of images of a specific subject, which allows for generating consistent images of the subject as specified by the text prompt.
Ye et al. (); Ma et al. (); Wang et al. (); Song et al. () followed a different approach and trained adapters to condition pre-trained diffusion models on a single image of the subject.
Yuan et al. () finetuned a diffusion model to be conditioned on a rigid transformation matrix describing the pose of the object.
These aforementioned approaches are focused on personalized generation given *user-provided images* and textual prompts.
In contrast, our approach aims to generate an image based on a *3D layout* and textual prompts.
However, we aim to allow users to change the layout while preserving the generated image, similar to personalized generation.

Another category of training-free approaches for consistent object generation manipulates self-attention to preserve image consistency Cao et al. (); Khachatryan et al. (); Qi et al. ().
The image style can be preserved by injecting the keys and values from a reference image into the self-attention layers of the generated image.
However, these approaches are designed to preserve the overall style, but the details of every individual object are not fully preserved (see Figure 4).
We propose a novel self-attention module that allows seamlessly inserting objects in an existing scene without altering the existing image contents.

## 3 Method

In Text-to-Image (T2I) diffusion models, the objective is to generate an image ${I}$ given a user-provided textual prompt $P$.
In our work, a layout is specified by the user in the form of 3D bounding boxes $\mathcal{B}=\{B^{1},B^{2},\dots,B^{n}\}$ and their corresponding prompts $\mathcal{P}=\{P^{1},P^{2},\dots,P^{n}\}$.
Under this setting, the goal is to generate an image where the content enclosed within each bounding box adheres to its respective prompt.
More specifically, we have two objectives: 1) Establish an interactive 3D layout pipeline. 2) Ensure object consistency under layout changes.

Figure: Figure 3: An overview of our pipeline *Build-A-Scene* (BAS). We formulate the Text-to-Image (T2I) task as a multi-stage generation process. Here, we illustrate a 3-stage scene. At stage 0 0, the user defines an empty scene with full control over scene size and camera parameters. At stage $1$, the user adds an object (a white sofa) by adding a box and its corresponding prompt to the layout. Our proposed *Dynamic Self-Attention (DSA)* module, coupled with latent blending, ensures that the object is seamlessly integrated into the scene while preserving the existing contents. At stage $2$, we illustrate our *Consistent 3D Translation* strategy that allows moving the object in 3D while preserving its identity.
Refer to caption: x3.png

### 3.1 From 2D to 3D Layout Control

The first step to establishing 3D layout control is finding an appropriate form of 3D annotations that the user can easily create.
We leverage LooseControl Bhat et al. () (LC) that accepts rendered 3D boxes and planes as a conditioning signal in addition to a single text prompt $P$.
We define the 3D layout as an empty 3D cuboid where the user can add planes to define the boundaries and 3D boxes to define the contents of the scene.
We refer to the set of 3D boxes as $\mathcal{B}$ and their corresponding prompts as $\mathcal{P}$ as explained above.
Unlike existing 2D layout approaches that require the user to provide the entire layout beforehand, we propose a novel strategy for layout control by revamping image generation as a sequential process.
The user starts with an empty scene and interactively adds objects to it through multiple generation stages $i\in[0,n]$.
At each stage, the user has control over a single object and can change its type, size, 3D location, and orientation.
This greatly enhances user controllability and customizability over the standard layout control.
Figure 3 illustrates our pipeline.

At the first stage, i.e., stage 0 0, the background is generated based on an initial prompt that we denote $P^{0}$ and a rendered depth map of the layout $D^{0}$ with only planes to define the boundaries.
At the following generation stages, stage $i>0$, the user adds a 3D box $B^{i}$ and specifies its corresponding prompt $P^{i}$.
The 3D scene is rendered to obtain a depth map $D^{i}$, a background and foreground masks for the box being generated at this stage, which we denote as $M_{BG}^{i}$ and $M_{FG}^{i}$.
During the diffusion process at stage $i$, an initial latent code $x_{T}^{i}$ is drawn from a random Gaussian noise distribution.
DDIM Song et al. () is used to iteratively denoise the latent code through multiple denoising steps $t:T\rightarrow 0$:

$$ $x_{t-1}^{i}=\sqrt{\alpha_{t-1}}\ \hat{x}_{0}^{i,t}+\sqrt{1-\alpha_{t-1}-\sigma ^{2}_{t}}\ \epsilon_{\theta}^{t}(x_{t}^{i},P^{i},D^{i})+\sigma_{t}\epsilon_{t}\enspace,$ (1) $$

$$ $\hat{x}_{0}^{i,t}=\dfrac{x_{t}^{i}-\sqrt{1-\alpha_{t}}\ \epsilon_{\theta}^{t}( x_{t}^{i},P^{i},D^{i})}{\sqrt{\alpha_{t}}}\enspace.$ (2) $$

where $\alpha_{t},\sigma_{t}$ are the parameters of a noise scheduler, $\epsilon_{\theta}^{t}$ is the noise prediction from the diffusion model, and $\epsilon_{t}$ is random Gaussian noise.

### 3.2 Interactive 3D Layout Control

At stage $i>0$, we aim to generate a new object based on the 3D box $B^{i}$ and the textual prompt $P^{i}$.
At the same time, the object is desired to be seamlessly integrated into the scene while preserving the existing contents from previous stages.
This can typically be done through inpainting or blended diffusion Avrahami et al. ().
However, they require the user to provide a free-form inpainting mask per object, which is laborious, and they are not directly compatible with the depth-conditioned LC.
We propose a novel technique for this purpose that is based on manipulating the self-attention maps and the latent codes.

In standard image diffusion models with a UNet backbone, i.e., Stable Diffusion 1.5 Rombach et al. (), each residual block has self-attention modules, which were found to encode the style and the structure of the generated image Tumanyan et al. ().
Self-attention for a given block and timestep $t$ at stage $i$ is computed as:

$$ $\displaystyle A^{i}_{t}=\text{{Softmax}}$ $\displaystyle\left(\dfrac{Q^{i}_{t}\ {K^{i}_{t}}^{\top}}{\sqrt{{d_{k_{t}^{i}}} }}\right)\ V^{i}_{t}\enspace,$ (3) $\displaystyle Q^{i}_{t}=f^{i}_{t}\ W_{Q},\qquad K^{i}_{t}=$ $\displaystyle f^{i}_{t}\ W_{K},\qquad V^{i}=f^{i}_{t}\ W_{V}.$ $$

where $f^{i}_{t}$ are the intermediate UNet features, and $W$ are trainable projection matrices.

A widely used approach to transfer the style of a reference image to a target image is *cross-frame attention* Cao et al. (); Khachatryan et al. (), which replaces $K^{i}$ and $V^{i}$ in Equation 3 with those of the reference image, i.e., $K^{i-1}$ and $V^{i-1}$.
This suggests that the target image will query the reference image for style, resulting in a consistent style between the two images.
This approach was adopted in LC, but it was found to be incapable of generating new objects with a different style, as shown in Figure 4.
This is intuitive as we limit the target image to copy the style exclusively from the source image.
An alternative approach is the extended attention adopted in Qi et al. () for performing consistent video edits, where the target image queries style from multiple images.
Figure 4 shows that this strategy can generate objects with a new style but deviates from the overall style of the reference image as different styles are mixed in an uncontrolled manner.

Figure: Figure 4: A comparison between existing self-attention mechanisms and our proposed Dynamic Self-Attention (DSA).
Refer to caption: x4.png

We propose a Dynamic Self-Attention (DSA) technique, which is able to freely generate an object with a new style while preserving the existing elements of the image.
We achieve this by augmenting the attention keys to include the keys of stage $i-1$ and a masked window of the keys of stage $i$:

$$ $\displaystyle\hat{K}^{i}_{t}$ $\displaystyle=[{K^{i-1}_{t}}^{\top}\oslash[{K^{i}_{t}}^{\top}\odot M^{i}_{FG}]\ ]$ (4) $\displaystyle A^{i}_{t}$ $\displaystyle=\text{{Softmax}}\left(\dfrac{Q^{i}_{t}\ \hat{K}^{i}_{t}}{\sqrt{{ d_{{k}^{i}_{t}}}}}\right)\ {V}^{i}_{t}\enspace,$ $$

where $\oslash$ is the concatenation operator, and $\odot$ is point-wise product.
This enforces the diffusion model to copy the overall style of the previous stage and allows it to generate a new style within the box of the current stage.
It is noteworthy that our approach is plug-and-play into the pre-trained diffusion model and does not require any finetuning.
Moreover, it does not rely on guidance similar to Chen et al. (), which requires backpropagating through the diffusion model at some iterations, adding a computation overhead.

To enhance the preservation of the background of the previous stage (especially when the latent distribution changes between stages), we blend the latent codes as follows:

$$ $x_{t}^{i}=M_{BG}^{i}\ x_{t}^{i-1}+M_{FG}^{i}\ x_{t}^{i}\enspace,$ (5) $$

Finally, to harmonize the colors of the scene, we follow Yang et al. (), and we perform AdaIN Huang & Belongie () between $x_{t}^{i-1}$ and $x_{t}^{i}$.
Finally, we obtain the final image ${I}^{i}$ for stage $i$ at the end of the denoising process.

### 3.3 Consistent 3D Translation

A major limitation of existing layout control approaches is their inability to preserve objects under layout changes, i.e., scaling and translation.
LC also suffers from the same problem as demonstrated in Figure 2.
This was attributed to the distributional shift of the latents that are aligned with the object before and after the layout change
Eldesokey & Wonka ().
Therefore, we propose a strategy for preserving objects under layout changes, i.e., 3D translation.

To translate an object at stage $i$, we start by segmenting the object out of the generated image in the previous stage $I^{i-1}$.
To achieve this without any user intervention, we first obtain a coarse segmentation by accumulating the cross-attention maps that corresponds to the object token in $P^{i-1}$ similar to Cao et al. ().
Then, we fit a bounding box to this coarse segmentation and use it as an input to SAM Kirillov et al. () to obtain a fine segmentation map $S^{i-1}$
After segmenting the object, we construct a warped image $I^{w}$ of the object after layout change by pasting the segmented object on the generated image from stage $i-2$.
To simulate the 3D translation accurately in the image plane, we use the 3D Cartesian coordinates map of the object box before and after the translation to warp the object.
More specifically, we compute correspondences for the 4 corners of the objects between the two Cartesian maps and use them to warp the object in the image plane.

By inverting the warped image through DDIM inversion, we obtain an approximate trajectory $x^{w}_{T},x^{w}_{T-1},\dots,x^{w}_{0}$ of the latents corresponding to the object after changing the layout.
Finally, we blend the latents between $x^{w}_{t}$ and $x^{i-2}_{t}$ as follows:

$$ ${x}_{t}^{i}=S^{i-1}\ x_{t}^{w}+(1-S^{i-1})\ x_{t}^{i-2}\enspace.$ (6) $$

This blending allows the diffusion model to regenerate the object of interest at the new location while preserving the background.
We perform this blending for a number of timesteps $\mathcal{T}<=T$.

## 4 Experiments

In this section, we provide a qualitative and quantitative evaluation of our proposed approach.
For clearer insights, we split the evaluation into two sub-tasks: (1) *3D layout control* and (2) *object consistency under layout change*.

### 4.1 Experimental Setup

#### Comparison

We compare against the baseline LooseControl (LC) Bhat et al. () that we employ for depth conditioning in our pipeline.
To show where our approach stands with respect to 2D layout approaches, we also compare against Layout-Guidance Chen et al. () that accepts 2D bounding boxes.
To map the 3D boxes to 2D, we fit a bounding box to box masks $M^{i}_{FG}$.

**Table 1: A quantitative comparison between our proposed approach, the 2D layout control method Layout-Guidance Chen et al. (), and LooseControl Bhat et al. ().**
|  | *3D Layout Control* | *Object Consistency* | Runtime |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
|  | CLIP${}_{\text{T2I}}\uparrow$ | OA (%) $\uparrow$ | mIOU $\uparrow$ | CLIP${}_{\text{I2I}}\uparrow$ | SSIM $\uparrow$ | PSNR $\uparrow$ |  |
| Layout-Guidance Chen et al. () | 0.323 | 48.2 | 0.425 | 0.838 | 0.189 | 28.35 | 12 s |
| LooseControl Bhat et al. () | 0.302 | 24.3 | 0.633 | 0.924 | 0.367 | 29.12 | 2 s |
| Ours | 0.321 | 55.3 | 0.772 | 0.940 | 0.476 | 29.5 | 6 s |

#### Implementation Details

LC is based on ControlNet with Stable Diffusion v1.5 Rombach et al. () that is fine-tuned through LoRA adaptation Hu et al. ().
We keep all the original settings of LC except for the sampler that we changed to a DDIM sampler with a linear schedule as we noticed that it is more stable.
We perform $T=20$ denoising steps in the quantitative comparison for efficiency and $T=40$ for the qualitative results for better quality.
For Layout-Guidance, we use the official implementation with the default parameters.
The code for our approach and the evaluation protocol will be made publicly available to facilitate future development and comparisons.

### 4.2 3D Layout Control Results

In this task, given a set of 3D bounding boxes and their corresponding prompts, the goal is to generate an image that conforms to these inputs.

#### Evaluation Strategy

Since there exist no criteria for evaluating 3D layout control, we define a new evaluation protocol for 3D layout control inspired by its 2D counterpart Chen et al. (); Xie et al. ().
We define a set of 16 objects from the MS COCO dataset Lin et al. () and their corresponding aspect ratios.
In addition, we define 10 different prompts for diverse scenes such as desert, snow, and room to use for the initial prompt $P^{0}$.
We start by sampling a random scene and object and create a 3D box that matches the object’s aspect ratio at a random z-coordinate.
Then, we sample another object, and we randomly select one of the three placements [”left”, ”right”, ”above] relative to the first object.
We place the second object into the scene similarly, ensuring that it does not occlude the first object or go out of bounds.

We sampled 100 random layouts and ran each layout with 5 different seeds for fairness.
Since the baseline LC does not accept layouts, we automatically create a textual description $P^{*}$ of the scene in the form:

$$ $P^{*}:``[P^{0}]\texttt{ with }[P^{1}]\texttt{ on the left/right and }[P^{2}] \texttt{on the right/left}"$ $$

Similarly, we create the textual description for the relation ”above” as well, where only the second object is allowed to be on top of the first object.

#### Evaluation Metrics

We are interested in evaluating three aspects: how the generated image confronts to the textual description of the scene, whether all specified objects in the layout have been generated, and how well each object fits within its box.

*CLIP${}_{T}2I$:* We compute the CLIP score Radford et al. () between the final generated image and the textual description of the image $P^{*}$.

*Object Accuracy (OA):* We use a general object detector, YOLOv8 Reis et al. (), to check if objects specified by the layout are detected in the image.

*Mean Intersection-over-Union (mIoU):* We compute the intersection between the bounding box predicted by YOLOv8 and a bounding box fitted to the 3D box in the image plane.
This tells how well the object is enclosed within the layout box.

#### Quantitative Results

Table 1 summarizes the quantitative comparison.
Our approach scores two times higher than LC and 15% higher than Layout-Guidance on Object Accuracy (OA), demonstrating its effectiveness in executing the layout.
Our approach also outperforms LC and is on par with Layout-Guidance on CLIP${}_{\text{T2I}}$, demonstrating that the generated image conforms better to the textual description.
For the mIOU metric, our approach outperforms Layout-Guidance by a huge margin despite not incorporating any guidance-based techniques.
Surprisingly, it also outperforms LC, despite the fact that we have not tuned it.
We believe this improvement is caused by our dynamic self-attention that forces the generated objects to lie within their respective boxes.

#### Qualitative Results

We provide a qualitative comparison in Figure 5.
The figure shows that our approach is more faithful to the layout compared to Layout-Guidance, while LC struggles to generate all objects.
We show both stage 0 0 and stage $n$ of our approach to highlight how objects are seamlessly integrated into the scene between stages (notice reflections and shadows).

Figure: Figure 5: A qualitative comparison on the task 3D layout control.
Refer to caption: x5.png

### 4.3 Object Consistency under Layout Change Results

This task aims to move or scale an object in the provided layout while preserving its identity.

#### Evaluation Criteria

We randomly sampled one of the 16 MS COCO objects and placed it in a layout.
Then we randomly selected one of the actions [”move left”, ”move right”, ” zoom-in”, ”zoom-out”].
We ensure that applying the action does not cause the object to be out of bounds.
Finally, we compare the object’s visual appearance before and after applying the action.

#### Evaluation Metrics

Our goal is to evaluate how similar the object is before and after applying some layout action.
We crop the object from the images and resize the cropped images to ensure they match.
We compute the CLIP score *CLIP${}_{I}2I$*, *Structural Similarity (SSIM)*, and *Peak-Signal-to-Noise-Ration Similarity (PSNR)* between a cropped image of the object before and after applying the action.

Figure: Figure 6: A qualitative comparison for object consistency under layout change.
Refer to caption: x6.png

#### Quantitative Results

Table 1 summarizes the results averaged over 5 seeds.
Our method and LC have a comparable CLIP${}_{\text{I2I}}$ as it measures the global similarity of the two images.
However, our method performs significantly better in terms of SSIM and PSNR.
Layout-Guidance performs the worst in all metrics as it does not have any mechanism to preserve the style.

#### Qualitative Results

We show some qualitative examples for different layout changes in Figure 6.
When moving the main object to the left or the right, our approach successfully applies the changes and preserves the object.
On the other hand, LC distorts the object, while Layout-Guidance changes the image drastically.
For the scaling layout changes, i.e., zoom-in and zoom-out, our approach applies them successfully and seamlessly inserts the object at the new location.
LC fails to apply them, and the object is subtly changed, while Layout-Guidance changes the object pose completely.

### 4.4 Ablation Study

Figure 7 provides an ablation analysis for the impact of different components of our pipeline.
When the Dynamic Self-Attention (DSA) is disabled, the model is not capable of inserting a new object into the reference image.
Skipping the latent blending in Equation 5 causes some details of the background to change (the paintings on the wall).
Using AdaIN Huang & Belongie () contributes to harmonizing the colors of the generated object with the background.

We also experiment with varying $\mathcal{T}$ in Section 3.3 for blending the latents in the Consistent 3D Translation strategy.
Initially, the warped image lacks realism, and the sofa appears to be floating.
By applying Equation 6 for $\mathcal{T}=0.4T$, the sofa is seamlessly integrated into the scene, but some of the details (the cushions) are not perfectly constructed.
When $\mathcal{T}=0.8T$, the sofa is seamlessly blended into the scene, and the fine details are well constructed.
With $\mathcal{T}=T$, some artifacts start to appear at the boundaries.

Figure: Figure 7: Ablation analysis for different parts of our pipeline.
Refer to caption: x7.png

## 5 Limitations and Future Work

Figure 8 shows some of the limitations of our approach.
First, our approach is sensitive to the aspect ratio of the box.
If provided with a box wider than the actual width of the object, it can generate two instances of the object (left figure).
When the aspect ratio of the box is not suitable for the specified prompt, it can generate a distorted object or a photo of the object (middle figure)
We believe that this is a natural behavior of our approach as it tries to fulfill the layout and the prompt requirements concurrently.
Secondly, if a large object is placed in a small space, i.e. a box intersecting with the boundaries, such as a car in a room, the out-of-boundary parts of the car are distorted (right figure).
In general, the definition of boxes needs to be reasonable to obtain the desired results.

For the Consistent 3D Translation strategy, if the object segmentation part fails, it becomes infeasible to preserve the objects.
Finally, one might argue that the multi-stage generation pipeline adds a computational overhead to the generation process.
However, this is a fair price in return for the enhanced control over scene elements that helps the user reach the desired output faster and eventually save time.
Moreover, our approach takes 2 seconds per stage and can generate a layout of 5 objects in the same time as Layout-Guidance (see Table 1).

Figure: Figure 8: Limitations of our approach.
Refer to caption: x8.png

#### Future Work

We would like to investigate automated layout generation through a large-language model (LLM) as in Feng et al. (); Gani et al. ().
Another direction is supporting in-plane rotations while preserving object identity, similar to Yuan et al. ().

## 6 Conclusion

We presented a first approach for interactive 3D layout control based on a pre-trained T2I diffusion model.
Our approach reformulated image generation as a multi-stage process, providing users with enhanced control over individual objects in 3D.
Moreover, we provided the first strategy to preserve objects under layout changes.
Experiments show that our approach outperformed both LooseControl and the recent 2D layout control, both quantitatively and qualitatively.
We hope that our approach will establish a new research direction for 3D layout control and consistency in layout control.

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 4
questions: See Weakness.
soundness: 3
strengths: 1. The task is well-defined.
2. The model is designed reasonably.
3. The paper is well-written and clearly states the contribution.
4. The results are well-organized, and the experiments are comprehensive.
summary: Based on the recent advancements in depth-conditioned T2I, this work presents a novel approach for interactive 3D layout control.     Further, the authors propose a Dynamic Self-Attention (DSA) module and a consistent 3D object translation strategy, to preserve the existing contents and consistent 3D translation.
weaknesses: 1. 3D Awareness. Existing results are mainly about the 3D layout conditioned generation. However, the 3D information is not adequately used. There are several other degrees of freedom in 3D spaces, such as the camera view, the rotation of objects, and the zoom-in/out. These results (even parts) could strengthen this work a lot. 

2. User interface. What exactly is the interface? Can you describe the tools for users to use? The interface for the creator is critical, especially for 3D editing. A reasonably designed interface can make this work for real-world applications. Just a demo-based (actually controlled by codes) can not make it practical.

3. The motivation. The motivations of the design of the pipeline is not very clear. The key feature of this work is that it can implement 3D layout-based generation. However, what are the challenges in this task, and why authors must use the proposed pipelines and modules to deal with this challenge?

---

confidence: 5
contribution: 2
initial_rating: 3
presentation: 2
questions: No
soundness: 3
strengths: Firstly, using inpainting to preserve the background during the interacting is a simple but sound idea, and the manipulations in 3D world make the masks’ calculation fully under control. Furthermore, 3D guidance is common and useful when we want to show the relative transformation of objects to the diffusion model, as we usually want it to give image of the real 3D world.
summary: BAS introduces a pipeline which allows users to guide the image synthesis through interacting with boxes in the 3D space and using the text prompt. The boxes reflect the position of objects in the image. To keep the image consistency throughout interacting, the author propose the dynamic self attention and object wrapping in order to maintain the background as much as possible. As a new work to introduce 3D interacting into 2D image synthesis, it proposes several benchmarks to evaluate its performance and compare to some image synthesis works.
weaknesses: (1) The technical contribution seems not to be so sufficient. As the T2I techniques and pre-trained models are mature, would organizing the masks according to the boxes 3D transformation being sufficient for a paper? Well, the dynamic self-attention block is surely a contribution.
(2) However, in the ablation study, it says “when DSA is disabled, the model isn’t capable of inserting new object”, but in the method, DSA is proposed for maintaining the background, which is different from what may happen in the ablation study. The technical contribution turns out to be confusing and doubtful. 
(3) Another consideration is that your run time is 3 times of “loose control”. Would that be harmful to the interaction? Could you please discuss more on that?
(4) Some expression in the “METHOD” still tells why other methods aren’t suitable for this “attention”. That should not appear here.

---

confidence: 3
contribution: 2
initial_rating: 8
presentation: 3
questions: In the DSA model,  why do you only combine the key tokes between stage i-1 and stage i?  I am wondering the effect of combining V tokens instead.  Discussions on why chose key tokens in DSA is necessary for the deep undstanding the design of DSA.
soundness: 3
strengths: 1. A straightforward yet effective approach to control the scene layout and the styles of objects placed in the scene.  With the 3D interface, the user can control how to put the objects on other objects or the relative position between objects, more natural than 2D interface.  The ability to move the object closer or farther away from the camera is also attractive. 

2. The DSA module designed to merge the keys at different stage is easy to implement, and it allows to generate new object of a different style. Also, the combination of latent codes according to the mask  and AdaIN operation can effectively maintain the image styles surrounding the newly inserted object.
summary: This paper proposes a interactive 3D layout control approach for T2I based on diffusion approach.  Beginning with an generated scene images without objects inside, the proposed approach allows uses to add 3D box and its text description in the bounding box of the scene to specify the 3D layout of the scene.  The depth map of the 3D layout and the text description are used as the condition to guide diffusion model, and the keys are combined according to the mask of the box to maintain the style of the image in each stage, denoted as dynamic self-attention. 

The video demo is nice, which demonstrates the 3D layout control process clearly.
weaknesses: The composes scene is relatively simple.  All objects are put on the flat surface, such as floor or desk top. It might be partially attributed to the box-based 3D layout control interface.  For example, can we put an object on the sofa if we use freeform surface as the control interface?  If freeform surface, how much effort does the user should pay to place a new object such that the diffusion model can generate the image correctly?

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 2
questions: from your pipeline figure (Fig.3) it seems like you didn't talk about some key parts of the method, specifically in stage 2 (warping, DDIM inversion) it might not be novel, but helps to grasp the notion of the method.
soundness: 2
strengths: they present an interesting 3D layout control over generated images, Furthermore, the multi-stage to have more control over each object sounds good.
summary: This work is trying to generate an image based on the layout of 3D boxes for each object we desire in the image. It builds upon a depth-conditioned stable diffusion generator, and they make it an iterable fashion where they focus on different object placement in each step to enhance control of each separate object. to keep the information from the last iteration to the next, they presented the Dynamic Self-Attention (DSA) module which blends between attention maps from the previous iteration and the new one, specifically on the mask where the new object is placed.
weaknesses: 1. you presented in most of the paper examples of moving the object aside (Fig.2), however, with 3D boxes, it is much more interesting to see examples of moving toward me or away from the observer. you provided only two examples like this in Fig.6, I would love to see it as the focus of the paper.
2. your evaluation metrics (Table. 1) have no comparison to 3D information, which is the focus of the paper. i would love o see a new metric that can evaluate the 3D placement, like with monocular depth estimation, or 3D object detection networks.

Meta-review: This paper proposes a T2I model that supports interactive 3D layout editing. The main idea is to divide the scene synthesis process in multiple stages, with each stage generating one object. To make the newly generated object has novel style while preserving other parts of the scene, the paper develops a dynamic self-attention mechanism that blends keys and values from previous stage with the ones in current stage. Furthermore, the paper proposes 3D translation to incorporate scene scale change.

The paper is well written and provides effective solution to an interesting task. 

During rebuttal, reviewers raised questions about the motivation, limitation (e.g., 3D awareness), generalization to free-form surfaces and questions about quantitative metrics. The authors have actively replied to all questions, addressing most of these. After rebuttal, 3 out of 4 reviewers agree to accept the paper, while one reviewer does not respond. AC agrees that this paper proposes an interesting task and simple yet effective solution, recommending the paper for acceptance. Meanwhile, AC also agrees that the paper is limited, since the main benefit of 3D layout control is free-form surfaces, the authors are encouraged to explore more in the future.

Decision: Accept (Poster)