3DIS: Depth-Driven Decoupled Instance Synthesis for Text-to-Image Generation

## Abstract

Abstract The increasing demand for controllable outputs in text-to-image generation has spurred advancements in multi-instance generation (MIG), allowing users to define both instance layouts and attributes. However, unlike image-conditional generation methods such as ControlNet, MIG techniques have not been widely adopted in state-of-the-art models like SD2 and SDXL, primarily due to the challenge of building robust renderers that simultaneously handle instance positioning and attribute rendering. In this paper, we introduce D epth- D riven D ecoupled I nstance S ynthesis (3DIS), a novel framework that decouples the MIG process into two stages: (i) generating a coarse scene depth map for accurate instance positioning and scene composition, and (ii) rendering fine-grained attributes using pre-trained ControlNet on any foundational model, without additional training. Our 3DIS framework integrates a custom adapter into LDM3D for precise depth-based layouts and employs a finetuning-free method for enhanced instance-level attribute rendering. Extensive experiments on COCO-Position and COCO-MIG benchmarks demonstrate that 3DIS significantly outperforms existing methods in both layout precision and attribute rendering. Notably, 3DIS offers seamless compatibility with diverse foundational models, providing a robust, adaptable solution for advanced multi-instance generation. The code is available at: https://github.com/limuloo/3DIS.

## 1 Introduction

With the rapid advancement of text-to-image generation technologies, there is a growing interest in achieving more controllable outputs, which are now widely utilized in artistic creation (Wang et al., ; Li et al., ): (i) Image-conditional generation techniques, e.g., ControlNet (Zhang et al., ), allow users to generate images based on inputs like depth maps or sketches. (ii) Multi-instance generation (MIG) methods, e.g., GLIGEN (Li et al., ) and MIGC (Zhou et al., ), enable users to define layouts and detailed attributes for each instance within the generated images.

However, despite the importance of MIG in controllable generation, these methods have not been widely adopted across popular foundational models like SD2 (Rombach et al., ) and SDXL (Podell et al., ), unlike the more widely integrated ControlNet. Current state-of-the-art MIG methods mainly rely on the less capable SD1.5 (Rombach et al., ) model.

We argue that the limited adoption of MIG methods is not merely due to resource constraints but also stems from a more fundamental challenge, i.e., unified adapter challenge. Current MIG approaches train a single adapter to handle both instance positioning and attribute rendering. This unified structure complicates the development of robust renderers for fine-grained attribute details, as it requires large amounts of high-quality instance-level annotations. These detailed annotations are more challenging to collect compared to the types of controls used in image-conditional generation, such as depth maps or sketches.

To address the unified adapter challenge and enable the use of a broader range of foundational models for MIG, we propose a novel framework called Depth-Driven Decoupled Instance Synthesis (3DIS). 3DIS tackles this challenge by decoupling the image generation process into two distinct stages, as shown in Fig. 2. (i) Generating a coarse scene depth map: During this stage, the MIG adapter ensures accurate instance positioning, coarse attribute alignment, and overall scene harmony without the complexity of fine attribute rendering. (ii) Rendering a fine-grained RGB image: Based on the generated scene depth map, we design a finetuning-free method that leverages any popular foundational model with pretrained ControlNet to guide the overall image generation, focusing on detailed instance rendering. This approach requires only a single training process for the adapter at stage (i), enabling seamless integration with different foundational models without needing retraining for each new model.

The 3DIS architecture comprises three key components:
(i) Scene Depth Map Generation:
We developed the first layout-controllable text-to-depth generation model by integrating a well-designed adapter into LDM3D (Stan et al., ). This integration facilitates the generation of precise, depth-informed layouts based on instance conditions.
(ii) Layout Control:
We introduce a method to leverage pretrained ControlNet for seamless integration of the generated scene depth map into the generative process. By filtering out high-frequency information from ControlNet’s feature maps, we enhance the integration of low-frequency global scene semantics, thereby improving the coherence and visual appeal of the generated images.
(iii) Detail Rendering:
Our method performs Cross-Attention operations separately for each instance to achieve precise rendering of specific attributes (e.g., category, color, texture) while avoiding attribute leakage. Additionally, we use SAM for semantic segmentation on the scene depth map, optimizing instance localization and resolving conflicts from overlapping bounding boxes. This advanced approach significantly improves the rendering of detailed and accurate multi-instance images.

We conducted extensive experiments on two benchmarks to evaluate the performance of 3DIS: (i) COCO-Position (Lin et al., ; Zhou et al., ): Evaluated the layout accuracy and coarse-grained category attributes of the scene depth maps. (ii) COCO-MIG (Zhou et al., ): Assessed the fine-grained rendering capabilities. The results indicate that 3DIS excels in creating superior scenes while preserving the accuracy of fine-grained attributes during detailed rendering.
On the COCO-Position benchmark, 3DIS achieved a 16.3% improvement in AP75 compared to the previous state-of-the-art method, MIGC. On the COCO-MIG benchmark, our training-free detail rendering approach improved the Instance Attribute Success Ratio by 35% over the training-free method Multi-Diffusion (Bar-Tal et al., ) and by 5.5% over the adapter-based method InstanceDiffusion (Wang et al., ).
Furthermore, the 3DIS framework can be seamlessly integrated with off-the-shelf adapters like GLIGEN and MIGC, thereby enhancing their rendering capabilities.

Figure: Figure 2: The overview of 3DIS. 3DIS decouples image generation into two stages: creating a scene depth map and rendering high-quality RGB images with various generative models. It first trains a Layout-to-Depth model to generate a scene depth map. Then, it uses a pre-trained ControlNet to inject depth information into various generative models, controlling scene representation. Finally, a training-free detail renderer renders the fine-grained attributes of each instance.
Refer to caption: x2.png

In summary, the key contributions of this paper are as follows:

- •
We propose a novel 3DIS framework that decouples multi-instance generation into two stages: adapter-controlled scene depth map generation and training-free fine-grained attribute rendering, enabling integration with various foundational models.
- •
We introduce the first layout-to-depth model for multi-instance generation, which improves scene composition and instance positioning compared to traditional layout-to-RGB methods.
- •
Our training-free detail renderer enhances fine-grained instance rendering without additional training, significantly outperforming state-of-the-art methods while maintaining compatibility with pretrained models and adapters.

## 2 Related Work

Controllable Text-to-Image Generation. With the rapid advancements in diffusion models (Zhou et al., ; Lu et al., ; Zhao et al., ; ; Xie et al., ), text-to-image technologies have reached a level where they can produce high-quality images (Rombach et al., ; ; Podell et al., ; Lu et al., ). Researchers are now increasingly focused on enhancing their control over the generated content. Numerous approaches have been developed to improve this control. ControlNet (Zhang et al., ) incorporates user inputs such as depth maps and edge maps by training an additional side network, allowing for precise layout control in image generation. Methods like IPAdapter (Ye et al., ) and PhotoMaker (Li et al., ) generate corresponding images based on user-provided portraits. Techniques such as ELITE (Wei et al., ) and SSR-Encoder (Zhang et al., ) enable networks to accept specific conceptual image inputs for better customization. Additionally, MIGC (Zhou et al., ; ) and InstanceDiffusion (Wang et al., ) allow networks to generate images based on user-specified layouts and instance attribute descriptions, defining this task as Multi-Instance Generation (MIG), which is the focal point of this paper.

Multi-Instance Generation (MIG). MIG involves generating each instance based on a given layout and detailed attribute descriptions, while maintaining overall image harmony. Current MIG methods primarily use Stable Diffusion (SD) architectures, classified into three categories: 1) Training-free methods: Techniques like BoxDiffusion (Xie et al., ) and RB (Xiao et al., ) apply energy functions to attention maps, enabling zero-shot layout control by converting spatial guidance into gradient inputs. Similarly, Multi-Diffusion (Bar-Tal et al., ) generates instances separately and then combines them according to user-defined spatial cues, enhancing control over orientation and arrangement. 2) Adapter methods: Approaches like GLIGEN (Li et al., ) and InstanceDiffusion (Wang et al., ) integrate trainable gated self-attention layers into the U-Net (Ronneberger et al., ), improving layout assimilation and instance fidelity. MIGC (Zhou et al., ) further divides the task, using an enhanced attention mechanism to generate each instance precisely before integration. 3) SD-tuning methods: Reco (Yang et al., ) and Ranni (Feng et al., ) add instance position data to text inputs and fine-tune both CLIP and U-Net, allowing the network to utilize positional cues for more precise image synthesis.
Previous methods entangled instance positioning with attribute rendering, complicating the training of a robust instance renderer. Our approach decouples this process into adapter-controlled scene depth map generation and training-free detail rendering. This separation allows the adapter to only handle instance positioning and coarse attributes, while leveraging the generative priors of pre-trained models, enhancing both flexibility and performance.

## 3 Method

### 3.1 Preliminaries

Latent Diffusion Models (LDMs) are among the most widely used text-to-image models today. They significantly enhance generation speed by placing the diffusion process for image synthesis within a compressed variational autoencoder (VAE) latent space. To ensure that the generated images align with user-provided text descriptions, LDMs typically employ a Cross Attention mechanism, which integrates textual information into the image features of the network.
In mathematical terms, the Cross Attention operation can be expressed as follows:

$$ $\operatorname{Attention}(\mathbf{Q},\mathbf{K},\mathbf{V})=\operatorname{ Softmax}\left(\frac{\mathbf{QK}^{\top}}{\sqrt{d_{k}}}\right)\mathbf{V},$ (1) $$

where $\mathbf{Q}$, $\mathbf{K}$, and $\mathbf{V}$ represent the query, key, and value matrices derived from the image and text features, respectively, while $d_{k}$ denotes the dimension of the key vectors.

### 3.2 Overview

Fig. 2 illustrates the overview framework of the proposed 3DIS, which decouples image generation into coarse-grained scene construction and fine-grained detail rendering. The specific implementation of 3DIS consists of three steps: 1) Scene Depth Map Generation (§ 3.3), which produces a corresponding scene depth map based on the user-provided layout; 2) Global Scene Control (§ 3.4), which ensures that the generated images align with the scene maps, guaranteeing that each instance is represented; 3) Detail Rendering (§3.5), which ensures that each generated instance adheres to the fine-grained attributes described by the user.

### 3.3 Scene Depth Map Generation

In this section, we discuss how to generate a corresponding depth map based on the user-provided layout, creating a coherent and well-structured scene while accurately placing each instance.

Choosing the text-to-depth model. Upon investigation, we identified RichDreamer (Qiu et al., ) and LDM3D (Stan et al., ) as the primary models for text-to-depth generation. RichDreamer fine-tunes the pretrained RGB Stable Diffusion (SD) model to generate 3D information, specifically depth and normal maps, while LDM3D enables SD to produce both RGB images and depth maps simultaneously. Experimental comparisons show LDM3D outperforms RichDreamer in complex scenes, likely due to its concurrent RGB and depth map generation. This dual capability preserves RGB image quality while enhancing depth map generation, making LDM3D our preferred model for text-to-depth generation.

Fine-tuning the text-to-depth model. In contrast to RGB images, depth maps typically prioritize the restoration of low-frequency components over high-frequency details. For instance, while a texture-rich skirt requires intricate details for RGB image generation, its corresponding depth map remains relatively smooth. Therefore, we aim to enhance the model’s ability to recover low-frequency content. Low-frequency components often indicate significant redundancy among adjacent pixels. To simulate this characteristic, we implemented an augmented pyramid noise strategy (Kasiopy, ), which involves downsampling and then upsampling randomly sampled noise $\epsilon$ to create patterns with high redundancy between adjacent pixels. We used the original SD training loss (Rombach et al., ) to fine-tune our text-to-depth model $\theta$, but adjusted the model to predict this patterned noise $\epsilon_{\text{pyramid}}$ with the text prompt $\bm{c}$:

$$ $\min_{\theta}\mathcal{L}_{\text{text}}=\mathbb{E}_{z,\epsilon\sim\mathcal{N}(0 ,I),t}\left[\left\|\epsilon_{\text{pyramid}}-f_{\theta}(z_{t},t,c)\right\|^{2} _{2}\right].$ (2) $$

Training the Layout-to-depth adapter.
Similar to previous methodologies (Zhou et al., ; Li et al., ; Wang et al., ), we incorporated an adapter into our fine-tuned text-to-depth model, enabling layout-to-depth generation, specifically leveraging the state-of-the-art MIGC (Zhou et al., ) model. Unlike earlier approaches, our method for generating depth maps does not rely on detailed descriptions of specific instance attributes, such as material or color. Consequently, we have augmented the dataset used for MIGC by eliminating fine-grained attribute descriptions from the instance data, thus focusing more on the structural properties of individual instances and the overall scene composition. The training process for the adapter $\theta^{\prime}$ can be expressed as:

$$ $\min_{\theta^{\prime}}\mathcal{L}_{\text{layout}}=\mathbb{E}_{z,\epsilon\sim \mathcal{N}(0,I),t}\left[\left\|\epsilon_{\text{pyramid}}-f_{\theta,\theta^{ \prime}}(z_{t},t,c,l)\right\|_{2}^{2}\right],$ (3) $$

where the base text-to-depth model $\theta$ is frozen, and the $\bm{l}$ is the input layout.

### 3.4 Global Scene Control

In this section, we will describe how to control the generated images to align with the layout of the generated scene depth map, ensuring that each instance appears in its designated position.

Injecting depth maps with ControlNet. After generating scene depth maps with our layout-to-depth models, we employed the widely adopted ControlNet (Zhang et al., ) model to incorporate global scene information. Scene depth maps focus on overall scene structure, without requiring fine-grained detail. Thus, although the base model produces 512x512 resolution maps, they can be upsampled to 768x768, 1024x1024, or higher (see Fig. 3 and Fig. 4, e.g., SD2 and SDXL). Since most generative models have depth ControlNet versions, these maps can be applied across various models, ensuring accurate instance placement and mitigating omission issues.

Removing high-frequency noise in depth maps. In our framework, the injected depth maps are designed to manage the low-frequency components of the constructed scene, while the generation of high-frequency details is handled by advanced grounded text-to-image models. To enhance the integration of these components, we implement a filtering process to remove high-frequency noise from the feature maps generated by ControlNet before injecting them into the image generation network. Specifically, the scene condition feature output from ControlNet, denoted as $F$, is added to the generation network. Prior to this addition, we transform $F$ into the frequency domain via the Fast Fourier Transform (FFT) and apply a filter to attenuate the high-frequency components:

$$ $F_{\text{filtered}}=\mathcal{F}^{-1}\left(H_{\text{low}}\cdot\mathcal{F}(F) \right),$ (4) $$

where $\mathcal{F}$ and $\mathcal{F}^{-1}$ denote the FFT and inverse FFT, respectively, and $H_{\text{low}}$ represents a low-pass filter applied in the frequency domain. This approach has been shown to reduce the occurrence of artifacts and improve the overall quality of the generated images without reducing performance.

### 3.5 Details rendering

Through the control provided by ControlNet, we can ensure that the output images align with our generated scene depth maps, thus guaranteeing that each instance appears at its designated location. However, we still lack assurance regarding the accuracy of attributes such as category, color, and material for each instance. To render each instance with correct attributes, we propose a training-free detail renderer to replace the original Cross-Attention Layers for this purpose. The process of rendering an entire scene using a detail renderer can be broken down into the following three steps.

Rendering each instance separately. For an instance $\bm{i}$, ControlNet ensures that a shape satisfying its descriptive criteria is positioned within the designated bounding box $\bm{b}_{i}$. By applying Cross Attention using the text description of the instance $\bm{i}$, we can ensure that the attention maps generate significant response values within the $\bm{b}_{i}$ region, accurately rendering the attributes aligned with the instance’s textual description. For each Cross-Attention layer in the foundation models, we independently render each instance $\bm{i}$ with their text descriptions to obtain the rendered result $\mathbf{r}_{i}$, while similarly applying the global image description to yield rendering background $\mathbf{r}_{c}$. Our next step is to merge the obtained feature maps $\{\mathbf{r}_{1},\cdots,\mathbf{r}_{n},\mathbf{r}_{c}\}$ into a single feature map, aligning with the forward pass of the original Cross-Attention layers.

SAM-Enhancing Instance Location. While mering rendering results, acquiring precise instance locations helps prevent attribute leakage between overlapping bounding boxes and maintains structural consistency with the instances in the scene depth maps. Consequently, we employ the SAM (Kirillov et al., ) model to ascertain the exact position of each instance. For an instance $\bm{i}$, by utilizing our generated scene depth map $\mathbf{m}_{scene}$ alongside its corresponding bounding box $\bm{b}_{i}$, we can segment the specific shape mask $\mathbf{m}_{i}$ of this instance, thereby facilitating subsequent merging:

$$ $\mathbf{m}_{i}=\operatorname{SAM}(\mathbf{m}_{scene},\mathbf{b}_{i})$ (5) $$

Merging rendering results. We employ the precise mask $\mathbf{m}_{i}$ obtained from SAM to constrain the rendering results of instance $\bm{i}$ to its own region, ensuring no influence on other instances. Specifically, we construct a new mask $\mathbf{m}_{i}^{\prime}$ by assigning a value of $\alpha$ to the areas where $\mathbf{m}_{i}$ equals $1$, while setting all other regions to $-\infty$. Simultaneously, we assign a background value of $\beta$ to the global rendering $\mathbf{r}_{c}$ through a mask $\mathbf{m}_{c}^{\prime}$. By applying the softmax function to the set $\{\mathbf{m}_{1}^{\prime},\mathbf{m}_{2}^{\prime},\ldots,\mathbf{m}_{n}^{
\prime},\mathbf{m}_{c}^{\prime}\}$, we derive the spatial weights $\{\mathbf{m}_{1}^{\prime\prime},\mathbf{m}_{2}^{\prime\prime},\ldots,\mathbf{m
}_{n}^{\prime\prime},\mathbf{m}_{c}^{\prime\prime}\}$ for each rendering instance. At each Cross Attention layer, the output can be expressed as follows to render the whole scene:

$$ $\mathbf{r}=\mathbf{m}_{1}^{\prime\prime}\cdot\mathbf{r}_{1}+\mathbf{m}_{2}^{ \prime\prime}\cdot\mathbf{r}_{2}+\ldots+\mathbf{m}_{n}^{\prime\prime}\cdot \mathbf{r}_{n}+\mathbf{m}_{c}^{\prime\prime}\cdot\mathbf{r}_{c}$ (6) $$

## 4 Experiment

### 4.1 Implement Details

**Table 1: Quantitative results on COCO-Position (§4.3). We only utilize complex layouts that contain at least five instances, resulting in significant overlap.**
|  | Layout Accuracy | Instance Accuracy | Image Quality |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Method | $AP\uparrow$ | $AP_{75}\uparrow$ | $AP_{50}\uparrow$ | $SR_{inst}\uparrow$ | MIoU | $CLIP\uparrow$ | $SR_{img}\uparrow$ | $FID\downarrow$ |
| BoxDiff  [ICCV23] | 3.15 | 2.12 | 10.92 | 22.74 | 27.28 | 18.82 | 0.53 | 25.15 |
| MultiDiff  [ICML23] | 6.37 | 4.24 | 13.22 | 28.75 | 34.17 | 20.12 | 0.80 | 33.20 |
| GLIGEN [CVPR23] | 38.49 | 40.75 | 63.79 | 83.31 | 70.14 | 19.61 | 40.13 | 26.80 |
| MIGC [CVPR24] | 45.03 | 46.15 | 80.09 | 83.37 | 71.92 | 20.07 | 43.25 | 24.52 |
| 3DIS | 56.83 | 62.40 | 82.29 | 84.71 | 73.32 | 20.84 | 46.50 | 23.24 |
| vs. prev. SoTA | +11.8 | +16.3 | +2.2 | +1.3 | +1.4 | +0.8 | +3.3 | +1.3 |

Tuning of text-to-depth models. We utilized a training set comprising 5,878 images from the LAION-art dataset (Schuhmann et al., ), selecting only those with a resolution exceeding 512x512 pixels and an aesthetic score of $\geq{8.0}$. Depth maps for each image were generated using Depth Anything V2 (Yang et al., ). Given the substantial noise present in the text descriptions associated with the images in LAION-art, we chose to produce corresponding image captions using BLIP2 (Li et al., ). We employed pyramid noise (Kasiopy, ) to fine-tune the LDM3D model for 2,000 steps, utilizing the AdamW (Kingma & Ba, ) optimizer with a constant learning rate of $1e^{-4}$, a weight decay of $1e^{-2}$, and a batch size of 320.

Training of the layout-to-depth adapter. We adopted the MIGC (Zhou et al., ) architecture as the adapter for layout control. In alignment with this approach, we utilized the COCO dataset (Lin et al., ) for training. We employed Stanza (Qi et al., ) to extract each instance description from the corresponding text for every image and used Grounding-DINO (Liu et al., ) to obtain the image layout. Furthermore, we augmented each instance’s description by incorporating modified versions that omitted adjectives, allowing our layout-to-depth adapter to focus more on global scene construction and the coarse-grained categories and structural properties of instances. We maintain the same batch size, learning rate, and other parameters as the previous work.

Figure: Figure 3: Qualitative results on the COCO-Position (§4.3).
Refer to caption: x3.png

### 4.2 Experiment Setup

Baselines.
We compared our proposed 3DIS method with state-of-the-art Multi-Instance Generation approaches. The methods involved in the comparison include training-free methods: BoxDiffusion (Xie et al., ) and MultiDiffusion (Bar-Tal et al., ); and adapter-based methods: GLIGEN (Li et al., ), InstanceDiffusion (Wang et al., ), and MIGC (Zhou et al., ).

Evaluation Benchmarks.
We conducted experiments using two widely adopted benchmarks, COCO-position (Lin et al., ) and COCO-MIG (Zhou et al., ), to assess the performance of models in different aspects of instance generation. The COCO-position benchmark emphasizes the evaluation of a model’s capacity to control the spatial arrangement of instances, as well as their high-level categorical attributes. In contrast, the COCO-MIG benchmark is designed to test a model’s ability to precisely render fine-grained attributes for each generated instance. To rigorously compare the models’ performance in handling complex scene layouts, we concentrated our analysis on the COCO-position benchmark, specifically focusing on layouts containing five or more instances. For a comprehensive evaluation, each model generated 750 images across both benchmarks.

Evaluation Metrics. We used the following metrics to evaluate the model: 1) Mean Intersection over Union (MIoU), measuring the overlap between the generated instance positions and the target positions; 2) Local CLIP score, assessing the visual consistency of the generated instances with their corresponding textual descriptions; 3) Average Precision (AP), evaluating the overlap between the generated image layout and the target layout; 4) Instance Attribute Success Ratio (IASR), calculating the proportion of correctly generated instance attributes; 5) Image Success Ratio (ISR), measuring the proportion of images in which all instances are correctly generated.

**Table 2: Quantitative results on proposed COCO-MIG-BOX (§4.3). $\mathcal{L}_{i}$ means that the count of instances needed to generate in the image is i.**
|  | Instance Attribute Success Ratio$\uparrow$ | Mean Intersection over Union$\uparrow$ |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Method | $\mathcal{L}2$ | $\mathcal{L}3$ | $\mathcal{L}4$ | $\mathcal{L}5$ | $\mathcal{L}6$ | $\mathcal{AVG}$ | $\mathcal{L}2$ | $\mathcal{L}3$ | $\mathcal{L}4$ | $\mathcal{L}5$ | $\mathcal{L}6$ | $\mathcal{AVG}$ |
| Adapter rendering methods |  |  |  |  |  |  |  |  |  |  |  |  |
| GLIGEN  [CVPR23] | 41.3 | 33.8 | 31.8 | 27.0 | 29.5 | 31.3 | 33.7 | 27.6 | 25.5 | 21.9 | 23.6 | 25.2 |
| InstanceDiff  [CVPR24] | 61.0 | 52.8 | 52.4 | 45.2 | 48.7 | 50.5 | 53.8 | 45.8 | 44.9 | 37.7 | 40.6 | 43.0 |
| MIGC  [CVPR24] | 74.8 | 66.2 | 67.4 | 65.3 | 66.1 | 67.1 | 63.0 | 54.7 | 55.3 | 52.4 | 53.2 | 54.7 |
| training-free rendering |  |  |  |  |  |  |  |  |  |  |  |  |
| TFLCG [WACV24] | 17.2 | 13.5 | 7.9 | 6.1 | 4.5 | 8.3 | 10.9 | 8.7 | 5.1 | 3.9 | 2.8 | 5.3 |
| BoxDiff   [ICCV23] | 28.4 | 21.4 | 14.0 | 11.9 | 12.8 | 15.7 | 19.1 | 14.6 | 9.4 | 7.9 | 8.5 | 10.6 |
| MultiDiff   [ICML23] | 30.6 | 25.3 | 24.5 | 18.3 | 19.8 | 22.3 | 21.9 | 18.1 | 17.3 | 12.9 | 13.9 | 15.8 |
| 3DIS    (SD1.5) | 65.9 | 56.1 | 55.3 | 45.3 | 47.6 | 53.0 | 56.8 | 48.4 | 49.4 | 40.2 | 41.7 | 44.7 |
| 3DIS    (SD2.1) | 66.1 | 57.5 | 55.1 | 51.7 | 52.9 | 54.7 | 57.1 | 48.6 | 46.8 | 42.9 | 43.4 | 45.7 |
| 3DIS    (SDXL) | 66.1 | 59.3 | 56.2 | 51.7 | 54.1 | 56.0 | 57.0 | 50.0 | 47.8 | 43.1 | 44.6 | 47.0 |
| vs. MultiDiff | +35 | +34 | +31 | +33 | +34 | +33 | +35 | +31 | +30 | +30 | +30 | +31 |
| rendering w/ off-the-shelf adapters |  |  |  |  |  |  |  |  |  |  |  |  |
| 3DIS+GLIGEN | 49.4 | 39.7 | 34.5 | 29.6 | 29.9 | 34.1 | 43.0 | 33.8 | 29.2 | 24.6 | 24.5 | 28.8 |
| vs. GLIGEN | +8.1 | +5.9 | +2.7 | +2.6 | +0.4 | +2.8 | +9.3 | +6.2 | +3.7 | +2.7 | +0.9 | +3.6 |
| 3DIS+MIGC | 76.8 | 70.2 | 72.3 | 66.4 | 68.0 | 69.7 | 68.0 | 60.7 | 62.0 | 55.8 | 57.3 | 59.5 |
| vs. MIGC | +2.0 | +4.0 | +4.9 | +1.1 | +1.9 | +2.6 | +5.0 | +6.0 | +6.7 | +3.4 | +4.1 | +4.8 |

Figure: Figure 4: Qualitative results on the COCO-MIG (§4.3).
Refer to caption: x4.png

### 4.3 Comparison

Scene Construction. The results in Tab. 1 demonstrate the superior scene construction capabilities of the proposed 3DIS method compared to previous state-of-the-art approaches. Notably, 3DIS surpasses MIGC with an 11.8% improvement in AP and a 16.3% increase in AP75, highlighting a closer alignment between the generated layouts and the user input. As shown by the visualizations in Fig. 3, 3DIS achieves marked improvements in scenarios with significant overlap, effectively addressing challenges such as object merging and loss in complex layouts. This results in the generation of a more accurate scene depth map, capturing the global scene structure with greater fidelity.

Detail Rendering.
The results presented in Tab. 2 demonstrate that the proposed 3DIS method exhibits robust detail-rendering capabilities. Notably, the entire process of rendering instance attributes is training-free for 3DIS. Compared to the previous state-of-the-art (SOTA) training-free method, MultiDiffusion, 3DIS achieves a 30% improvement in the Instance Attribute Success Ratio (IASR). Additionally, when compared with the SOTA adapter-based method, Instance Diffusion, which requires training for rendering, 3DIS shows a 5% increase in IASR, while also allowing the use of higher-quality models, such as SD2 and SDXL, to generate more visually appealing results. Importantly, the proposed 3DIS approach is not mutually exclusive with existing adapter methods. For instance, combinations like 3DIS+GLIGEN and 3DIS+MIGC outperform the use of adapter methods alone, delivering superior performance. Fig. 4 offers a visual comparison between 3DIS and other SOTA methods, where it is evident that 3DIS not only excels in scene construction but also demonstrates strong capabilities in instance detail rendering. Furthermore, 3DIS is compatible with a variety of base models, offering broader applicability compared to previous methods.

### 4.4 Ablation Study

Constructing scenes with depth maps. Tab. 3 demonstrates that generating scenes in the form of depth maps, rather than directly producing RGB images, enables the model to focus more effectively on coarse-grained categories, structural attributes, and the overall scene composition. This approach leads to a 3.32% improvement in AP and a 4.1% increase in AP75.

Tuning of the Text-to-depth model. Tab. 3 demonstrates that, compared to using LDM3D directly, fine-tuning LDM3D with pyramid diffusion as our base text-to-depth generation model results in a 1.3% improvement in AP and a 2.97% increase in AP75. These improvements stem from the fine-tuning process, which encourages the depth generation model to focus more on recovering low-frequency components, benefiting the global scene construction.

**Table 3: Ablation study on scene generation (§4.4).**
| method | AP | AP50 | AP75 | MIOU |
| --- | --- | --- | --- | --- |
| w/o using depth | 53.51 | 81.75 | 58.30 | 72.17 |
| w/o aug data | 54.04 | 78.35 | 59.43 | 73.30 |
| w/o tuning LDM3D | 55.53 | 81.93 | 60.23 | 72.82 |
| w/ all | 56.83 | 82.29 | 62.40 | 73.32 |

Augmenting instance descriptions by removing adjectives.
The data presented in Tab. 3 indicate that during the training of layout-to-depth adapters, augmenting instance descriptions by removing fine-grained attribute descriptions allows the model to focus more on the structural of the instances and the overall scene construction. This approach ultimately results in a 2.79% improvement in AP and a 2.97% increase in AP75.

Figure: Figure 5: Visualization of the Impact of Low-Pass Filtering on ControlNet (§4.4).
Refer to caption: x5.png

Figure: Figure 6: Visualization of the Impact of SAM-Enhancing Instance Location (§4.4).
Refer to caption: x6.png

**Table 4: Ablation study on rendering (§4.4).**
| method | IASR $\uparrow$ | MIOU $\uparrow$ |
| --- | --- | --- |
| w/o Low-Pass Filter | 55.87 | 46.93 |
| w/o SAM-Enhancing | 52.42 | 45.17 |
| w/ all | 56.01 | 47.01 |

Low-Pass Filtering on the ControlNet. Fig. 5 shows that filtering out high-frequency noise from ControlNet’s feature maps improves the overall quality of the generated images, resulting in more accurate scene representation. Moreover, as indicated in Tab. 4, this process does not affect the Instance Attribute Success Ratio (IASR) and MIoU when rendering fine details.

SAM-Enhancing Instance Location. Fig. 6 illustrates that utilizing SAM for more precise instance location effectively prevents rendering conflicts caused by layout overlaps, ensuring accurate rendering of each instance’s fine-grained attributes. As shown in Tab. 4, enhancing instance localization with SAM improves the Instance Attribute Success Ratio (IASR) by 3.19% during rendering.

### 4.5 Universal rendering capabilities of 3DIS

Figure: Figure 7: Rendering results based on different-style models (§4.5).
Refer to caption: x7.png

Figure: Figure 8: Rendering results on specific concepts (§4.5).
Refer to caption: x8.png

Rendering based on different-architecture models. Fig. 1, 3, and 4 present the results of 3DIS rendering details using SD2 and SDXL without additional training. The results demonstrate that 3DIS not only leverages the enhanced rendering capabilities of these more advanced base models, compared to SD1.5, but also preserves the accuracy of fine-grained instance attributes.

Rendering based on different-style models. Fig. 7 presents the results of 3DIS rendering using various stylistic model variants (based on the SDXL architecture). As shown, 3DIS can incorporate scene depth maps to render images in diverse styles while preserving the overall structure and key instance integrity. Furthermore, across different styles, 3DIS consistently enables precise control over complex, fine-grained attributes, as illustrated by the third example in Fig. 7, where “Dotted colorful wildflowers, some are red, some are purple” are accurately represented.

Rendering Specific Concepts.
3DIS renders details leveraging pre-trained large models, such as SD2 and SDXL, which have been trained on extensive corpora. This capability allows users to render specific concepts. As demonstrated in Fig. 8, 3DIS precisely renders human details associated with specific concepts while preserving control over the overall scene.

## 5 Conclusion

We propose a novel 3DIS method that decouples image generation into two distinct phases: coarse-grained scene depth map generation and fine-grained detail rendering. In the scene depth map phase, 3DIS trains a Layout-to-Depth network that focuses solely on global scene construction and the coarse-grained attributes of instances, thus simplifying the training process. In the detail rendering phase, 3DIS leverages widely pre-trained ControlNet models to generate images based on the scene depth map, controlling the scene and ensuring that each instance is positioned accurately. Finally, our proposed detail renderer guarantees the correct rendering of each instance’s details. Due to the training-free nature of the detail rendering phase, our 3DIS framework utilizes the generative priors of various foundational models for precise rendering. Experiments on the COCO-Position benchmark demonstrate that the scene depth maps generated by 3DIS create superior scenes, accurately placing each instance in its designated location. Additionally, results from the COCO-MIG benchmark show that 3DIS significantly outperforms previous training-free rendering methods and rivals state-of-the-art adapter-based approaches. We envision that 3DIS will enable users to apply a wider range of foundational models for multi-instance generation and be extended to more applications. In the future, we will continue to explore the integration of 3DIS with DIT-based foundational models.

## 6 Appendix

### 6.1 Comparison of LDM3D and RichDreamer

Figure: Figure 9: Comparison of LDM3D and RichDreamer.
Refer to caption: x9.png

Upon investigation, we identified RichDreamer (Qiu et al., ) and LDM3D (Stan et al., ) as the primary models employed for text-to-depth generation. To compare their performance, we utilized prompts from the COCO2014 dataset as input for both models, with the corresponding results illustrated in Fig. 9. Our analysis indicates that LDM3D demonstrates a superior ability to preserve the original SD1.4 priors, resulting in enhanced text comprehension and more precise control over scene generation. In contrast, RichDreamer exhibits certain shortcomings: (i) it often misses semantic details or omits entire objects in the depth maps, as seen in cases where essential elements like the cot and man are entirely absent; (ii) the depth maps produced by RichDreamer frequently suffer from artifacts such as blotches or thread-like distortions, particularly when used in conjunction with ControlNet. Therefore, after a thorough comparison, we selected LDM3D as the base model for text-to-depth generation in our 3DIS system.

### 6.2 Visualization on the impact of the LDM3D Fine-Tuning

Figure: Figure 10: Comparison of original LDM3D and finetuned LDM3D.
Refer to caption: x10.png

Although LDM3D is capable of generating relatively good depth maps, several issues remain:
(i) Since LDM3D was trained using depth maps extracted from the DPT-Large Model (Ranftl et al., ) , the resulting image quality is relatively poor.
(ii) As a diffusion model trained by Gaussian noise, LDM3D exhibits limited ability to recover low-frequency content (Guttenberg, ). This is clearly illustrated in Fig. 10, where the generated depth maps struggle to produce large uniform color blocks. Moreover, the average color value of the depth maps tends to converge towards the initial noise, whose mean value is close to 0. This constraint places a harmful limitation on text-to-depth generation.

To address (i), we fine-tuned the model using depth maps extracted from the latest Depth-Anything V2 model. For (ii), we adopted pyramid noise instead of Gaussian noise, which helps mitigate the constraints on text-to-depth generation. As shown in Fig. 10, the fine-tuned LDM3D model is capable of generating depth maps with higher contrast and improved overall quality.

### 6.3 More results of the generated scene depth map

Figure: Figure 11: More results of the generated scene depth map.
Refer to caption: x11.png

Fig. 11 presents additional results of scene depth map generation using our 3DIS system. The results demonstrate that, even with complex layouts, 3DIS effectively understands and generates cohesive scenes, harmoniously placing all objects within them. Furthermore, even in cases of significant overlap, such as the five suitcases in the fifth row, 3DIS handles the arrangement with precision, maintaining clear object separation and preventing blending.

### 6.4 More results on COCO-MIG benchmark

Fig. 12 presents additional results of 3DIS on the COCO-MIG dataset, revealing several key advantages over the previous state-of-the-art model, MIGC. First, 3DIS demonstrates superior scene construction capabilities, as seen in the first and second rows, where it constructs more coherent scenes that appropriately place all specified instances—such as rendering an indoor environment when prompted with “refrigerator.” Second, 3DIS exhibits enhanced detail rendering, as shown in the fourth to sixth rows. By leveraging the more advanced SDXL model in a training-free manner, 3DIS outperforms MIGC, which primarily relies on SD1.5, producing more visually appealing and structurally refined results. Third, 3DIS handles smaller instances better, as demonstrated in the third row with the “red bird” and “yellow dog.” Its ability to render at higher resolutions using SDXL leads to clearer and more accurate depictions of these smaller objects. Finally, 3DIS excels in managing overlapping objects, as illustrated in the seventh row, where it avoids object merging while generating the scene’s depth map.

Figure: Figure 12: More qualitative results on the COCO-MIG.
Refer to caption: x12.png

### 6.5 Limitation

Although 3DIS leverages various foundation models for rendering fine instance details, its scene construction continues to rely on the less advanced SD1.5 model. This dependency limits 3DIS’s capacity to accurately generate complex structures, particularly those that SD1.5 struggles with, such as intricate shapes or highly detailed spatial configurations. As a result, 3DIS may occasionally fail in scenarios that demand more sophisticated generative capabilities. Addressing this limitation in future work could involve the development of specialized datasets aimed at enhancing the model’s proficiency in handling complex geometries, thereby improving the overall robustness and versatility of 3DIS in a broader range of applications.

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 8
presentation: 3
questions: 1. How does the layout adapter connect to the text-to-depth model? Does it work like ControlNet?
2. Now that depth information is introduced, can this method control the front-back relationship of overlapping instances?
soundness: 4
strengths: 1. This paper has a very clear motivation and solves the pointed problems very well. Although the idea of ​​using depth map as a coarse-grained scene guidance is simple, it effectively solves the limited adaptability challenge in existing MIG methods. Extensive qualitative and quantitative results (e.g., Table 2, Figure 3, and Figure 4) demonstrate that the proposed method has strong flexibility and can be applied to a variety of foundational models.
2. The presented results are promising, achieving state-of-the-art performance on instance attributes and locations. In particular, the proposed method can be further combined with other controllable image generation methods such as GLIGEN and MIGC to improve their performance for multi-instance generation..
3. The paper is well written and clearly structured.
summary: This work focuses on controllable image generation and points out the unified adapter challenge of multi-instance generation (MIG) methods: current MIG methods uses a single adapter to simultaneously handle instance positioning and attribute rendering. Such a unified structure complicates the development of detail renderers because it requires a large number of high-quality instance-level annotations. To this end, this work proposes a two-stage generation paradigm: (1) generating a coarse-grained depth map from layout; (2) rendering fine-grained instance details from depth map. This design enables the MIG adapter to be seamlessly integrated into various foundational models such as SD2 and SDXL without specific training. Extensive experiments demonstrate the effectiveness and flexibility of the proposed method.
weaknesses: 1. The proposed framework depends on many existing models. For example, a) the text-to-depth model is obtained by fine-tuning LDM3D; b) a pretrained depth-conditioned ControlNet is required for depth layout injection; c) the detailed renderer relies on SAM to segment instances from depth. These dependencies weaken the originality of the paper and may compromise the robustness of the proposed framework.
2. The paper does not discuss depth ambiguity when multiple bounding boxes overlap. For example, given two partially overlapping instances, how can the model tell which is in front and which is behind?

---

confidence: 4
contribution: 4
initial_rating: 6
presentation: 3
questions: 1. How does the proposed 3DIS framework handle the challenges of noisy annotations in the input data?

2. The training procedure relied on the LAION-art dataset, which may not fully represent the diversity of real-world images or scenarios.

3. What strategies could be employed further to improve the robustness of the generated depth maps? 

4. Given the dependence on pre-trained models for the training-free detail rendering process, how might variability in these models' performance impact the overall effectiveness of the 3DIS framework?
soundness: 3
strengths: - **Innovative Framework:** The 3DIS framework effectively decouples the multi-instance generation process, allowing for improved accuracy in scene composition and instance positioning.

- **Training-Free Approach:** The ability to render fine-grained attributes without additional training is a significant advantage, making the model more accessible and easier to integrate with existing systems.

- **Robust Performance:** Extensive experiments on benchmarks such as COCO-Position and COCO-MIG show that 3DIS consistently outperforms state-of-the-art methods in both layout precision and attribute rendering, indicating its effectiveness in practical applications.
summary: The paper introduces Depth-Driven Decoupled Instance Synthesis (3DIS), a novel framework aimed at enhancing multi-instance generation (MIG) in text-to-image generation. 3DIS addresses the limitations of existing MIG methods by decoupling the generation process into two distinct stages: generating a coarse scene depth map for accurate instance positioning and rendering fine-grained attributes without additional training. This two-stage approach allows for greater control over both layout and attribute details, leading to improved scene composition and integration with various foundational models. The extensive experimental results demonstrate that 3DIS significantly outperforms existing techniques in layout accuracy and fine-grained attribute rendering across established benchmarks.
weaknesses: - **Limited Dataset Scope**: The experiments relied on the LAION-art dataset and COCO benchmarks, which may not fully represent the diversity of real-world images or scenarios. Expanding the evaluation to include a broader range of datasets could provide a more comprehensive assessment of the model's generalizability.

- **Evaluation Metrics Limitations**: The selected evaluation metrics, while informative, may not capture all aspects of image quality or user satisfaction. Additional metrics could enhance the evaluation of the generated images' visual appeal and usability.

---

confidence: 3
contribution: 2
initial_rating: 6
presentation: 3
questions: 1. How does the occlusion of the layout impact the depth map generation? Since there is no clue of which one is nearer. It is better to evaluate the generation quality of the depth map, and how the depth map impact the final generation.
2. The parameters of the low pass filter, and how to choose it are missing.
soundness: 2
strengths: 1. The two stage framework for MIG seems to be novel, and the findings that depth map can be utilized as an intermediate product for  layout conditioned generation. 
2. Some practical techniques are proposed, which may be useful in practical  applications.
summary: The paper proposes a two-stage framework for multiple instance generation, which is 1)  generating a depth map conditioned on the layout map, 2) generating the image conditioned on the generated depth map with pretrained ControlNet.  Some techniques are also proposed to improve the results, such as using low-pass filter on the depthmap, using segment model for detail renderer.  The claimed improvements are on layout accuracy, instance accuracy and image quality.
weaknesses: 1. Compared to one-stage methods, the proposed one can be computionally inefficient, the extra computation cost should be clearly evaluated and presented. 
2. The paper heavily based on previous works, the novelty can be limited. 
3. It seems that the key contribution to the improvement of MIOU is the depth map, especially as shown in Fig.3 and Fig. 4, the depth map already demonstrats advantage compared to MIGC results. Further discussion on why the depth map can better align with the layout can be added.

---

confidence: 3
contribution: 2
initial_rating: 6
presentation: 3
questions: 1. Can the layout-to-depth model generate diverse depth maps for the same layout? Like changing the pose, shape, and other attributes of instances?

2. It seems that sometimes the background scene can be blurry or unpleasant. Does that have something to do with the instance-wise rendering step?

3. Are there any failure cases that can make analyses on the Layout-to-depth model?

4. How long will it take to generate a single image?
soundness: 2
strengths: 1. Proposed a novel approach for MIG task with Depth control, the proposed method shows good generalizability and performance on different MIG benchmarks.

2. The proposed 3DIS can be seamlessly integrated into various pre-trained diffusion models, without requiring fine-tuning processes for each pretrained diffusion model.

3. The incorporation of 3D depth information can provide a better understanding of instances' attributes, thus benefiting the generation process of 2D images.
summary: This paper introduces 3DIS for multiple instance generation(MIG) tasks in text-to-image generation. 3DIS decouples the image generation process into two stages. First, 3DIS generates a coarse scene depth map that accurately positions instances and aligns their coarse attributes. Second, it renders fine-grained instance attributes using pre-trained ControlNet on any foundation diffusion model without additional training. This approach enables seamless integration with diverse foundational models with a Depth-Controlled method, providing a robust and adaptable solution for advanced multi-instance generation.
weaknesses: 1. This pipeline is largely dependent on the fine-tuned Layout-to-Depth generation model, and the final output uses the foundation diffusion model to render instances in a strictly confined region(depth). There might be inconsistencies between the layout-to-depth generation model and the renderer model, thus hindering the final performance.

2. The slightest artifacts in the depth map might cause great changes in the final generation results, like unwanted additional objects or background depth.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: Please see the weakness.
soundness: 3
strengths: 1. The proposed method is flexible and can combine many existing generation models.
2. The location improvement is pretty obvious compared to previous SoTA methods.
summary: In this paper, the authors propose a novel multi-instance generation method, which uses a depth map as the intermediate generation step to decouple the whole process into two stages. Based on the depth map, the following rendering process only applies existing models, such as SAM, ControlNet, SD, and so on. Experiments show that the generated image is more in line with the given layout and description.
weaknesses: 1. The locations of the cat and lamp are not correct in Figure 1.
2. The details of the comparison should be clarified. For example, it mentions that MIGC only supports SD1.5 in Figure 1, but the visual results of the proposed method in Figures 3 and 4 are based on SD2 and SDXL. In addition, it's unclear which version of SD is used in Table 1.
3. It's better to provide some inference efficiency analysis since the proposed method contains multiple steps.
4. The visual quality is not very satisfactory. It's better to apply user study to evaluate user preferences.
5. It's better to add the image quality metrics to the ablation study in Tables 3 and 4.

Meta-review: - **Strengths**:
  - Clear and novel decoupling of the MIG pipeline.
  - Demonstrated improved spatial control, flexibility, and integration with pre-trained models.
  - Robust validation through metrics, qualitative examples, and user studies.

- **Weaknesses**:
  - Dependency on existing models limits perceived originality.
  - Persistent concerns about generalizability and scalability.
  - Incremental improvements over prior art.

The overall decision was to **accept (spotlight)**, with reviewers acknowledging the significant technical contributions and improvements made during the rebuttal period. The work is deemed impactful, with potential for further refinements.

Decision: Accept (Spotlight)