3DIS: Depth-Driven Decoupled Image Synthesis for Universal Multi-Instance Generation
Authors: Dewei Zhou
Ji Xie
Zongxin Yang
Yi Yang
Venue: ICLR 2025 Spotlight (2025)
Keywords: Image Generation; Diffusion Models
Primary area: generative models

Abstract: The increasing demand for controllable outputs in text-to-image generation has spurred advancements in multi-instance generation (MIG), allowing users to define both instance layouts and attributes. However, unlike image-conditional generation methods such as ControlNet, MIG techniques have not been widely adopted in state-of-the-art models like SD2 and SDXL, primarily due to the challenge of building robust renderers that simultaneously handle instance positioning and attribute rendering. In this paper, we introduce Depth-Driven Decoupled Image Synthesis (3DIS), a novel framework that decouples the MIG process into two stages: (i) generating a coarse scene depth map for accurate instance positioning and scene composition, and (ii) rendering fine-grained attributes using pre-trained ControlNet on any foundational model, without additional training. Our 3DIS framework integrates a custom adapter into LDM3D for precise depth-based layouts and employs a finetuning-free method for enhanced instance-level attribute rendering. Extensive experiments on COCO-Position and COCO-MIG benchmarks demonstrate that 3DIS significantly outperforms existing methods in both layout precision and attribute rendering. Notably, 3DIS offers seamless compatibility with diverse foundational models, providing a robust, adaptable solution for advanced multi-instance generation. The code is available at: https://github.com/limuloo/3DIS.

Published as a conference paper at ICLR 2025
3DIS: DEPTH-DRIVEN DECOUPLED IMAGE SYNTHE- SIS FOR UNIVERSAL MULTI-INSTANCE GENERATION
Dewei Zhou J Xie Zongxin Yang??, Yi Yang
'RELER, CCAI, Zhejiang University DDMI, HMS, Harvard University {zaw1 999, sanaka87, yangyics] edu. edu. cn [Zongxin Maangxin.Yang/ae Equal contribution * Corresponding author
harvard.
Layout
MIGC
3DIS
only suppont SDL.
generated scene
rendering w
fendering WY SDAL
Labie
plaza
penersted scene
adj gill
A beautiful Altican girl with bruided deep brown
A beautiful Asian with blckk lair and blue eyes. 41 An oul painting isspired by Van Gogh's Suarry Night, with swirling, vihmutt
1) beausiful European girl wath blonde hair light blue lasten-ike lamp. hast
adi. lamp
Gonal 2) clegant crystal tradi- pierce- ng grean eyos. 4) An oil painting of alm- ospheric
eycs. paper Pessiun
caditiona
3) aff
English gnay fur, yellow eyes. An painting inspicall by Mona's Impression, Sunrise, with soft, diffused capturing carly moming Brick wall.
4) adi painning
3) ginger
5) odj wall
with drmmatic clouds, sailing ship
yellow stookes filling sky. Purely colored
vihmet glowing
battling wave:
Figure 1: Images generated using our 3DIS. Based on the user- esoviprovided layout, 3DIS generates a scene depth map that precisely positions each instance and renders their fine-grained attributes without the need for additional training, using a variety of foundational models.
ABSTRACT
The increasing demand for controllable outputs in text-to-image generation has spurred advancements in multi-instance generation (MIG), allowing users to de- fine both instance layouts and attributes. However, unlike image-conditional generation methods such ControlNet, MIG techniques have not been widely adopted in state-of-the-art models like SD2 and SDXL, primarily due to the chal- lenge of building robust renderers that simultaneously handle instance position- ing and attribute rendering. In this paper, we introduce Depth-Driven Decoupled Image Synthesis (3DIS), novel framework that decouples the MIG process into two stages: (i) generating a coarse scene depth map for accurate instance po- sitioning and scene composition, and (ii) rendering fine-grained attributes using pre-trained ControlNet on any foundational model, without additional training. Our 3DIS framework integrates custom adapter into LDM3D for precise depth- based layouts and employs fietuning-free method for enhanced instance-level attribute rendering. Extensive experiments on COCO-Position and COCO-MIG benchmarks demonstrate that 3DIS significantly outperforms existing methods in both layout precision and attribute rendering. Notably, 3DIS offers seam- less compatibility with diverse foundational models, providing robust, adapt- able solution for advanced multi-instance generation. The code is available at: https//github.oo//immloo///////

---

Published as a conference paper at ICLR 2025
INTRODUCTION
With the rapid advancement of text-to-image generation technologies, there is growing interest in achieving more controllable outputs, which are now widely utilized in artistic creation (Zhou et al.
2024c; Zhuo et al. 2024): (i) mnage-conditional generation techniques, e.g., ControlNet (Zhagg al. 2023), allow users to generate images based on inputs like depth maps sketches. (ii) Multi- instance generation (MIG) methods, e.g., GLIGEN (Li et al. 2023c) and MIGC (Zhou et al. 2024a), enable users define layouts and detailed attributes for each instance within the generated images.
However, despite the importance of MIG in controllable generation, these methods have not been widely adopted across popular foundational models like SD>2 (Rombach et al., (023) and SDXI. (Podell et al., 2023), unlike the more widely integrated ControlNet. Current state-of-the- art MIG methods mainly rely on the less capable SD1.5 RRombach et al. 2022) model.
We argue that the limited adoption of MIG methods is not merely due to resource constraints but also stems from a more fundamental challenge, i.e., unified adapter challenge. Current MIG approaches train single adapter to handle both instance positioning and attribute rendering. This unified structure complicates the development of robust renderers for fine aine-graind attribute details, as it requires large amounts of high-quality instance-level annotations. These detailed annotations are more challenging to collect compared to the types of controls used in image-conditional generation, such depth maps sketches.
To address the unified adapter challenge and enable the use of a broader range of foundational models for MIG, we propose a novel framework called Depth-Driven Decoupled Image Synthesis (3DIS). 3DIS tackles this challenge by decoupling the image generation process into two distinct stages, shown in Fig. (i) Generating coarse scene depth map: During this stage, the MIG adapter ensures accurate issancce positioning, coarse attribute alignment, and overall scene harmony without the complexity of fine attribute rendering.
(ii) Rendering a fine-grained RGB image: Based on the generated scene depth map, we design a finetuning-fret method that leverages any popular foundational model with pretrained ControlNet to guide the overall image generation, focusing on detailed instance rendering. This approach requires only a single training process for the adapter at stage (i), enabling seamless integration with different foundational models without needing retraining for each new model.
The 3DIS architecture comprises three key components: (i) Scene Depth Map Generation: We de- veloped the first layout-controllable text-to-depth generation model by integrating a well-designed adapter into LDM3ID (Stan et al. 2023). This integration facilitates the generation of precise, depth- informed layouts based on instance conditions. (ii) Layout Control: We introduce a method to leverage pretrained ControlNet for seamless integration of the generated scene depth map into the generative process. By filtering out high-frequency information from ControlNet's 's feature maps, we enhance the integration of low-frequency global scene semantics, thereby improving the co- herence and visual appeal of the generated images. (iii) Detail Rendering: Our method performs Cross-Attention operations separately for each instance to achieve precise rendering of specific at- tributes (e.g., category, color, texture) while avoiding attribute leakage. Additionally, use SAM for semantic segmentation on the scene depth map, optimizing instance localization and resolving conflicts from overlapping bounding boxes. This advanced approach significantly improves the ren- dering of detailed and accurate multi-instance images.
We conducted extensive experiments on two benchmarks to evaluate the performance of 3DIS: (i) POCO-Position (Lin et al., 2013) Zhou et al., 2024a): Evaluated the layout accuracy and coarse- grained category attributes of the scene depth maps. (ti) COCO-MIG (Zhou et al. 2024a): Assessed the fine-grained rendering capabilities. The results indicate that 3DIS excels in creating superior scenes while preserving the accuracy of fine-grained attributes during detailed rendering. On the COCO-Position benchmark, 3DIS achieved 16.3% improvement in AP75 compared to the previ- ous state-of-the-art method, MIGC. On the COCO-MIG benchmark, our training- detail render- ing approach improved the Instance Attribute Success Ratio by 35% over the training-free method Multi-Diffusion (Bar- Tal et al.) (023) and by 5.5% over the adapter-bac based method InstanceDif- fusion (Wang al. 2024). Furthermore. the 3DIS framework can be seamlessly integrated with off-the-shelf adapters like GLIGEN and MIGC, thereby enhancing their rendering capabilities.
In summary, the key contributions of this paper are as follows:
2

---

Published as a conference paper at ICLR 2025
Stage Generate a coarse-grained scene depth map
Layout
1) A red knife
Adapter
Korange cup
A yellow cup
4) A blue
A purple sandwich
Fine-tuned
A black sandwich
Text-to-Depth Model
Scene Depth Map
Stage Rendering fine-grained instance details
Pretrained ControlNet
Low-Pass Filter
sample
SDL.5
Segment
Detail Renderer
Model
SD2
...
High Quality Image
Off-the-shelf Controllers
(Optional)
(MIGC, GLIGEN....)
SDXL
Trainable
Frozen
The overview of 3DIS. 3DIS decouples image generation into two stages: creating scene depth map and rendering high-quality RGB images with various generative models. It first trains Layout-to-Depth model generate scene depth map. Then, it uses pre-trained ControlNet to inject depth information into various generative models, controlling scene representation. Finally, a training-free free detail renderer renders the fine- grained attributes of each instance.
Figure 2:
We propose novel 3DIS framework that decouples multi-instance generation into two stages: adapter-controlle scene depth map generation and training-free free fine-graid fine-grained attribute rendering, enabling integration with various foundational models.
We introduce the first layout-fo-depth model for multi-instance generation, which improves scene composition and instance positioning compared to traditional layout-to- tooutttt-RGBB methods.
Our training-free detail renderer enhances fine-graines instance rendering without additional training, significantly outperforming state-of-the-ee methods while maintaining compatibility with pretrained models and adapters.
2 RELATED WORK
Controllable Text-to-Image Generation. With the rapid advancements in diffusion models (Zhou
al. 2023 2025 Lu et al.) 2023 2024a b) Zhao et al., 2024a by Gao et al., 2024 [ie et al.) 2024h
and multimodal learning (Wang al. 2225 Yang et al. 2021
all 2023b Yang al. 2024b).
text-to-image technologies (Rombach et al. 2022 Podell et al. 2023) have reached level where
now increasingly focused on enhancing their control over the generated content. Numerous approaches have been developed to improve this con- trol. ControlNet (Zhang al. 2023) incorporates user inputs such depth maps and edge maps by training additional side network, allowing for precise layout control in image generation. Methods like IPAdapter (Ye et al., 2023) and PhotoMaker (Li et al. 2024) generate corresponding images based uscv-provided portraits. Techniques such ELITE (Wei et al. 2023) and SSR- Encoder (Zhng et al. 2024) enable networks to accept specific conceptual image inputs for better customization. Additionally, MIGC (Zhou al.) 2024b) and DistanceDiffusion (Wang et al. 2024) allow networks to generate images based on user-specified layouts and instance attribute descrip- tions, defining this task Multi-Instance Generation (MIG), which is the focal point of this paper.
they can produce high-quality images. Researchers
Instanen Generation (MIG). MIG involves generating each instance based on a given layout and detailed attribute descriptions, while maintaining overall image harmony. Current MIG methods

---

Published as a conference paper at ICLR 2025
primarily Stable Diffusion (SD) architectures, classified into three categories: 1) Training-fre free methods: Techniques like BoxDiffusion (Xie et al. 2023 and RB (Xiao et al. 2023 apply energy functions to attention maps, enabling zero-shot layout control by converting spatial guidance into gradient inputs. Similarly, Multi-Diffusion (Bar-Tal et al. 2023) generates instances separately and then combines them according user-defined spatial cues, enhancing control over orientation and
arrangement. 2) Adapter methods: Approaches like GLIGEN (Li et al.) 2023c) and InstanceD-
iffusion (Wang al. 2024 integrate trainable gated self-attention layers (Vaswani al. 2017
Yang et al. [024c] 2025) into the U-Net (Ronneberger et al. 2013), improving layout assimilation and instance fidelity. MIGC (Zhou al. 2024a bp further divides the task, using an enhanced at- tention mechanism to generate each instance precisely before integration. 3) SD-tuning methods: Reco Yang et al,, 2023) and Ranni (Feng et al. 2024) add instance position data to text inputs and fine-tune both CLIP and U-Net, allowing the network to utilize positional cues for more pre- cise image synthesis. Previous methods entangled instance positioning with attribute rendering, complicating the training of robust instance renderer. Our approach decouples this process into udapter-controlle scene depth map generation and training-free detail rendering. This separation allows the adapter to only handle instance positioning and coarse attributes, while leveraging the generative priors of pre-trained models, enhancing both flexibility and performance.
METHOD
3.1 PRELIMINARIES
Latent Diffusion Models (LDMs) are among the most widely used text-to-image models today. They significantly enhance generation speed by placing the diffusion process for image synthesis within compressed variational autoencoder (VAE) latent space. To ensure that the generated images align with user-proveee seer-provided text descriptions, LDMs typically employ Cross Attention mechanism, which integrates textual information into the image features of the network. In mathematical terms, the Cross Attention operation can be expressed as follows:
QK VdLe
Attention Q, K, V) = Softmax
V
(1)
where Q, K, and V represent the query, key, and value matrices derived from the image and text features, respectively, while dę denotes the dimension of the key vectors.
3.2 OVERVIEW
Fig. 2 illustrates the overview framework of the proposed 3DIS, which decouples image generation into coarse-grained scene construction and fine-grained detail rendering. The specifie implementa- tion of 3DIS consists of three steps: 1) Scene Depth Map Generation ($ 3.3). which produces a corresponding scene depth map based on the user-provided layout; 2) Global Scene Control 3.4). which ensures that the generated images align with the scene maps, guaranteeing that each instance represented: 3) Detail Rendering ($3.5), which ensures that each generated instance adheres to the fine-grained attributes described by the user.
3.3 SCENE DEPTII MAP GENERATION
In this section, we discuss how to generate a corresponding depth map based on the user-provine eser-provided layout, creating a coherent and well-structured scene while accurately placing each instance.
Choosing the text-to-depth model. Upon investigation, we identified RichDreamer (Qiu et al.) 2024) and LDM3D (Stan et al. 2023) as the primary models for text-to-depth generation. Rich- Dreamer fine-tunes the pretrained RGB Stable Diffusion (SD) model to generate 3D information, specifically depth and normal maps, while LDM3D enables SD to produce both RGB images and depth maps simultancously. Experimental comparisons show LDM3D outperforms RichDreamer in complex scenes, likely due to its concurrent RGB and depth map generation. This dual capability preserves RGB image quality while enhancing depth map generation, making LDM3D our preferred model for text-to-depth generation.
Fine-tuning the text-to-depth model. In contrast to RGB images, depth maps typically priori- tize the restoration of frequency components over high-frequency details. For instance, while
4

---

Published as a conference paper at ICLR 2025
texture-rich skirt requires intricate details for RGB image generation, its corresponding depth map remains relatively smooth. Therefore, we aim to enhance the model's ability to recover low- frequency content. now-frequency components often indicate significant redundancy among ad- jacent pixels. To simulate this characteristic, we implemented an augmented pyramid noise strat- egy (Kastopy 2.22)) which involves downsampling and then upsampling randomly sampled noise e to create patterns with high redundancy between adjacent pixels. We used the original SD training loss (Rombach al., 2022) fine-tune our texi-to-depth model 0, but adjusted the model predict this patterned noise Gpyramid with the text prompt C:
min List = (0,. [yyramid - fo(,,, c01b3
(2)
Training the Layout-to-depth adapter. Similar previous methodologies (Zhou al.
2024a Li
et al. 2023c Wang et al. 2024), we incorporated an adapter into our fine-tuned text-to-depth model, enabling layout-to-depth generation, specifically leveraging the state-of-the-ar MIGC (Zhou al. 2024a) model. Unlike earlier approaches, our method for generating depth maps does not rely on detailed descriptions of specific instance attributes, such as material or color. Consequently, we have augmented the dataset used for MIGC by eliminating fine-gre grained attribute descriptions from the instance data, thus focusing more on the structural properties of individual instances and the overall scene composition. The training process for the adapter 0' can be expressed as:
min Layon EE,.=N((.D..
[|l ||
Saler (Za, t, 1113]
(3)
where the base text-to-depth model is frozen, and the is the input layout.
3.4 GLOBAL SCENE CONTROL
In this section, we will describe how to control the generated images to align with the layout of the generated scene depth map, ensuring that each instance appears in its designated position.
Injecting depth maps with ControlNet. After generating scene depth maps with our layout-to- depth models, we employed the widely adopted ControlNet (Zhang al. (023) model to incorporate global scene information. Scene depth maps focus on overall scene structure, without requiring fine-grained detail. Thus, although the base model produces 5 512x512 resolution maps, they can be upsampled 768x768, 1024x 1024, higher (see Fig. S and Fig. + e.g., SD2 and SDXL). Since most generative models have depth ControlNet versions, these maps can be applied across various models, ensuring accurate instance placement and mitigating omission issues.
Removing high-frequency noise in depth maps. In our framework, the injected depth maps are designed to manage the low-frequency components of the constructed scene, while the generation of high-frequency details is handled by advanced grounded text-to-image models. To enhance the integration of these components, we implement a filtering process remove high-frequency noise from the feature maps generated by ControlNet before injecting them into the image generation network. Specifically, the scene condition feature output from ControlNet, denoted as F, is added to the generation network. Prior to this addition, we transform F into the frequency domain via the Fast Fourier Transform (FFT) and apply a filter to attenuate the high-frequency components:
Filtered FF (How F(F)),
where FF and F-1 denote the FFT and inverse FFT, respectively, and How represents a low-pass filter applied in the frequency domain. This approach has been shown to reduce the occurrence of artifacts and improve the overall quality of the generated images without reducing performance.
3.5 DETAILS RENDERING
Through the control provided by ControlNet, we can ensure that the output images align with our generated scene depth maps, thus guarantecing that each instance appears at its designated location. However, we still lack assurance regarding the accuracy of attributes such as category, color, and material for each instance. To render each instance with correct attributes, free detail renderer to replace the original Cross-Attention Layers for this purpose. The process of rendering entire scene using a detail renderer can be broken down into the following three steps.
propose training-
5

---

Published as a conference paper at ICLR 2025
Table 1:
Quantitative results on COCO-Position 4.3
We only utilize complex layouts that
contain at least five instances, resulting in significant overlap.
Layout Accuracy
Instance Accuracy
Image Quality
Method
AP1 f
APrs
APso f
SRina f
MloU
CLIP t
SRong
FTD 4
BoxDiff [CCV22)
3.15
2.12
10.92
22.74
27.28
18.82
0.53
25.15
MultDiff (CML223
6.37
4.24
13.22
28.75
34.17
20.12
0.80
33.20
GLIGEN (CVPR233
38.49
40.75
63.79
83.31
70.14
19.61
40.13
26.80
MIGC (CIPPR24)
45.03
46.15
80.09
83.37
71.92
20.07
43.25
24.52
3DIS (5m1.5)
56.83
62.40
82.29
84.71
73.32
20.84
46.50
23.24
prev. SoTA
+11.8
+16.3
+2.2
+1.3
+1,4
+0.8
+3.3
+1.3
Rendering each instance separately. For instance i. ControlNet ensures that shape satisfying its descriptive criteria is positioned within the designated bounding box b,. By applying Cross Attention using the text description of the instance i, significant response values within the b, region, accurately rendering the attributes aligned with the instance's textual description. For each Cross-Attention layer in the foundation models, independently render each instance with their text descriptions to obtain the rendered result r; while similarly applying the global image description yield rendering background re- Our next step is to merge the obtained feature maps {rı,-.. Fr, Fef into single feature map, aligning with the forward pass of the original Cross-Attention layers.
ensure that the attention maps generate
SAM-Enhancing Instance Location. While mering rendering results, acquiring precise instance locations helps prevent attribute leakage between overlapping bounding boxes and maintains struc- tural consistency with the instances in the scene depth maps. Consequently. we employ the SAM (Kirillov al. (023) model to ascertain the exact position of each instance. For an instance i, by utilizing our generated scene depth map Mscene alongside its corresponding bounding box b; we can segment the specific shape mask m, of this instance, thereby facilitating subsequent merging:
m, SAM b.)
(5)
Merging rendering results. We employ the precise mask m; obtained from SAM constrain the rendering results of instance i to its own region, ensuring no influence on other instances. Specifi- cally, we construct a new mask m by assigning a value of a to the areas where m; equals 1, while setting all other regions to -oc. Simultaneously, we assign background value of 3 to the global ren- dering r, through mask m'. By applying the softmax function to the set (m/ m/, m'...... .. m m ) we derive the spatial weights (m". my.... m" m"} for each rendering instance. At each Cross Attention layer, the output can be expressed as follows to render the whole scene:
r m" ri + m² r2 +... + m" Fn + +mm
(6)
4 EXPERIMENT
4.1 IMPLEMENT DETAILS
Tuning of text-to-depth models. We utilized training set comprising 5,878 images from the
LAION-art dataset (Schuhmann et al. 202I). selecting only those with a resolution exceeding 512x512 pixels and an aesthetic score of
8.0. Depth maps for each image were generated using
Depth Anything V2 (Yang et al. 2024a). Given the substantial noise present in the text descriptions associated with the images in LAION-an art, we chose to produce corresponding image captions using BLIP2 (Li et al. 2023a). We employed pyramid noise (Kasiopy 2023) to fine-tune the LDM3D model for 2,000 steps, utilizing the AdamW (Kingma Bal 2017
optimizer with a constant learn-
ing rate of 14-4, weight decay of 1e-2 and batch size of 320.
Training of the layout-to-depth adapter. We adopted the MIGC Zhou et al. 2024a) architecture as the adapter for layout control. In alignment with this approach, we utilized the COCO dataset (Lin et al. 2013) for training. We employed Stanza (Qi et al. 2020 to extract each instance description grounding-DINO
from the corresponding text for every image and used Grounding DINO (Liu et al. 2023) to obtain the image layout. Furthermore, we augmented each instance's 's description by incorporating modified versions that omitted adjectives, allowing our layout-to-depth adapter to focus more on global scene

---

Published as a conference paper at ICLR 2025
Layout
MIGC
Ours (depth, SD1.5)
Ours (RGB, SI2)
Ours (RGB, SDXL)
dining table
pizza
pizza
bottlk
table
Figure 3: Qualitative results on the COCO-Position
4.3
construction and the coarse- the same batch size, learning rate, and other parameters as the previous work.
coarse-grained grained categories and structural properties of instances. We maintain
4.2 EXPERIMENT SETUP
Baselines. We compared our proposed 3DIS method with state-of-the-arr Multi-Instance Genera- tion approaches. The methods involved in the comparison include training-free free methods: BoxD-
iffusson (Xie al. 2023) and MultiDiffusion (Bar- Tal et al. 2023); and adapter-based methods:
GLIGEN (Li et al.
2023c InstanceDiiffusio (Wang et al. 2024), and MIGC (Zhou et al. 2024a).
widely adopted benchmarks, COCO-position (Lin et al. (213)) and COCO-MIG (Zhou et al. 2024a). to assess the performance of models in different aspects of instance generation. The COCO COcO-position benchmark emphasizes the evaluation of model's 's capacity to control the spatial arrangement of instances, as well as their high- level categorical attributes. In contrast, the COCO-MIG benchmark is designed to test model's ability to precisely render fine-grained attributes for each generated instance. To rigorously compare the models' performance in handling complex scene layouts, we concentrated our analysis on the COCO-position benchmark, specifically focusing on layouts containing five or more instances. For a comprehensive evaluation, each model generated 750 images across both benchmarks.
Evaluation Benchmarks. We conducted experiments using
Evaluation Metrics. We used the following metrics to evaluate the model: 1) Mean Intersection over Union (MIoU), measuring the overlap between the generated instance positions and the target positions; 2) Local CLJP score, assessing the visual consistency of the generated instances with their corresponding textual descriptions; 3) Average Precision (AP), evaluating the overlap between the generated image layout and the target layout; 4) Instance Attribute Success Ratio (IASR), calculating the proportion of correctly generated instance attributes; 5) Image Success Ratio (ISR), measuring the proportion of images in which all instances correctly generated.
4.3 COMPARISON
Scene Construction. The results in Tab. demonstrate the superior scene construction capabili- ties of the proposed 3DIS method compared to previous sfate-of-the art approaches. Notably, 3DIS surpasses MIGC with an 11.8% improvement in AP and a 16.3% increase in AP7s highlighting closer alignment between the generated layouts and the user input. As shown by the visualizations in Fig. g 3DIS achieves marked improvements in scenarios with significant overlap, effectively addressing challenges such object merging and loss in complex layouts. This results in the gener- ation of a more accurate scene depth map, capturing the global scene structure with greater fidelity.
Detail Rendering. The results presented in Tab. demonstrate that the proposed 3DIS method exhibits robust detail-rendering capabilities. Notably, the entire process rendering instance at- tributes is training-free for 3DIS. Compared to the previous ttate-of-the-as (SOTA) training- free method, MultDiffusion, 3DIS achieves a 30% improvement in the Instance Attribute Success Ratio (IASR). Additionally, when compared with the SOTA adapter-based method, Instance Diffusion,
7

---

Published as a conference paper at ICLR 2025
Table 2: Quantitative results on proposed COCO-MIG-BOX
4.3
means that the count of
instances needed to generate in the image is
Instance Auribute Success Ratio
Mean Intersection over Union?
Method
C2
C3
L6
AVG
C2
C3
AVG
Adapter rendering methods
GLIGEN [CVPR233
41.3
33.8
31.8
27.0
29.5
31,3
33.7
27.6
25.5
21.9
23.6
25.2
InstanceDiff (CVPR24)
61.0
52.4
50.5
43.0
MIGC (CVPR24]
74.8
66.2
67.4
65.3
66.1
67.1
63.0
54.7
55.3
52.4
53,2
54.7
training-free rendering
TFLCG [WACV24]
17.2
13.5
8.3
10.9
5.3
BixDiff [CCV23]
28.4
21.4
14.0
11.9
12.8
15.7
19.1
14.6
9.4
7.9
8.5
10.6
MultiDiff [CML23]
30.6
25.3
24.5
18.3
19.8
22.3
21.9
18.1
17.3
12.9
13.9
15.8
3DIS
(SD1.5)
65.9
56.1
55.3
45.3
47.6
53.0
56.8
48.4
49.4
40.2
41.7
44.7
3DIS
(SD2.)
66.1
57.5
55.1
51.7
52.9
54.7
57.1
48.6
46.8
42.9
43.4
45.7
3DIS
(SDX1)
66.1
59.3
56.2
51.7
54.1
56.0
57.0
50.0
47.8
43.1
44.6
47.0
vs. MultiDiff
+35
+34
+31
433
+34
+33
+35
+31
+30
+30
+30
+31
rendering w/ off-the-shelf adapters
3DIS+GLIGEN
49.4
39.7
34.5
29.6
29.9
34.1
43.0
33.8
29.2
24.6
24.5
28.8
vs. GLIGEN
+8.1
+5.9
+2.7
+2.6
+0.4
+2.8
+9.3
+6.2
+3.7
+2.7
+0,9
+3.6
3DIS+MIGC
70.2
72.3
66.4
68.0
69.7
68.0
60.7
62.0
59.5
MIGC
+2.0
+4.0
+4,9
+1.1
+1.9
+2.6
+5.0
+6.0
+6.7
+3,4
+4.1
+4.8
Layout
MIGC
Ours (Depth, SD11.5
Ours (RGB, S122)
Ours (RGB, SDXX.)
brown
uebrella
ulbralla
Layout
Instance Diffusion
Ours (depth, SD1.5)
Ours (RGB, SD2)
(RGB, SDXL)
blue
Figure 4: Qualitative results on the COCO-MIG ($4.3)
which requires training for rendering, 3DIS shows 5% increase in IASR, while also allowing the of higher-quality models, such SD2 and SDXL, to generate Importantly, the proposed 3DIS approach is not mutually exclusive with existing adapter methods. For instance, combinations like 3DIS+GLIGEN and 3DIS+MIGC outperform the of adapter methods alone, delivering superior performance. Fig. offers a visual comparison between 3DIS and other SOTA methods, where is evident that 3DIS not only excels in scene construction but also demonstrates strong capabilities in instance detail rendering. Furthermore, 3DIS is compatible with variety of base models, offering broader applicability compared to previous methods.
visually appealing results.
4.4 ABLATION STUDY
Constructing scenes with depth maps. Tab. demonstrates that generating scenes the form of depth maps, rather than directly producing RGB images, enables the model to focus more effectively on coarse-graines categories, structural attributes, and the overall scene composition. This approach leads to 3.3% improvement in AP and 4.1% increase in AP75-
Tuning of the Text-to-depth model. Tab. demonstrates that, compared to using LDM3D di- rectly, fine-tuning LDM3D with pyramid diffusion base text-to-depth generation model
8

---

Published as a conference paper at ICLR 2025
Scene Depth
w: Filter
w/o Filter
Scene Depth
w/ Filter
wio Filter
Figure 5: Visualization of the Impact of Low-Pass Filtering on ControlNet ($4.4),
Layout
w/ SAM
w/o SAM
Layout
w/ SAM
w/o SAM
yellow orange blue fork
black bed
green teddy bear
white table
Figure 6:
Visualization of the Impact of SAM-Enhancing Instance Location ($..4)
results in a 1.3% improvement in AP and 2.2% increase in AP75- These improve- ments stem from the fine-tuning process, which encourages the depth generation model to fo- cus more on recovering low-frequency components, benefiting the global scene construction.
Table 3: Ablation study on scene generation 4.4
Augmenting instance descriptions by removing adjectives. The data presented in Tab. B indicate that during the training of layout-to-depth adapters, augmenting instance descriptions by removing fine- grained attribute descriptions allows the model to focus more on the structural of the instances and the overall scene construction. approach ultimately results in a 2.8% improvement in AP and 3.0% increase in AP75-
method
AP/APta/APrs f
MloU
FID |
w/o using depth w/o aug data w/o tuning LDM3D w/ all
53.5/818/583
72.2
24.1
54.0/ 74.0/78.4/9.44
73.3
23.5
55.5 81.9 60.2
72.8
25.2
56.8 56.8/82.3 82.3 /62.4
73.3
23.2
This
Low-Pass Filtering the ControlNet. Fig. S shows that filtering out high-frequency noise from ControlNet's feature maps improves the overall quality of the generated images, re- sulting in more accurate scene representation. Moreover, as indicated in Tab. this process does not affect the Instance Attribute Success Ratio (LASR) and MIoU when rendering fine details.
Table 4: Ablation study rendering (4.44)
method
TASR 1
MIOU
FID
Low-Pass Filter
55.87
46,93
24.50
w/o SAha-nnhncing
52.42
45.17
23.67
w/ all
56.01
47.01
23.24
MAM-ninhnninng Instance Location. Fig. G iillustrates that utilizing SAM for more precise instance location effectively prevents rendering conflicts caused by layout overlaps. ensuring accurate ren- dering of each instance's 's fine-grained attributes. As shown in Tab. g enhancing instance localization with SAM improves the Instance Attribute Success Ratio (IASR) by 3.19% during rendering.
4.5 UNIVERSAL RENDERING CAPABILITIES OF 3DIS
Rendering based different-architettie models. Fig. and present the results of 3DIS rendering details using SD2 and SDXL without additional tramingg The results demonstrate that 3DIS not only leverages the enhanced rendering capabilities of these more advanced base models, compared to SD1.5, but also preserves the accuracy of fine-graie fineigrained instance attributes.
Rendering based on different-style models. Fig. presents the results of 3DIS rendering using various stylistic model variants (based on the SDXL architecture). As shown, 3DIS can incorporate scene depth maps to render images in diverse styles while preserving the overall structure and key instance integrity. Furthermore, across different styles, 3DIS consistently enables precise control over complex, fine-grained attributes, as illustrated by the third example in Fig, 7 where "Dotted colorful wildflowers, some red, some are purple" are accurately represented.
Rendering Specifie Concepts. 3DIS renders details leveraging pre-trained large models, such as SD2 and SDXL. which have been trained on extensive corpora. This capability allows users to render specific concepts. As demonstrated in Fig. g 3DIS precisely renders human details associated with specific concepts while preserving control over the overall scene.

---

Published as a conference paper at ICLR 2025
Layout
Scene Depth map
SDXL
AutismMix
Watercolor (LoRA) Pixel Art (L.RA)
A luxurious chair armrest crafted from polished gold, with smooth and reflective surface. Chair legs, carved from solid, high-quality wood, with natural, rich grain. 3) A chair seat, crafted from luxurious white velvet, soft to the touch and with smooth, velvety finish that exudes elegance. 4) A chair buckrest features an exquisile, tiger- like pallern of orange-y elloow stripes.
1) A cute girl, black hair, brown smiling sweetly, looking at viewer, soft expression. 2) Glossy black hair, smooth and slightly wavy. 3) A jewey carring, embedded with emerald stones, 4) A Santa-style bat, with red body and white flully shaped sticker cheek, shine.
1) A lowering snow-capped mountain with sprawling glacier. 2) A crystal-clear blue river, geatly flowing with soft ripples. 3) dense forest dark green tall trees with lush foliage. Soft, warm-toned clouds illuminated by the colors of the sunsel, blemding orange, pink, purple. Dotted colorful wildflowers,
red, purple. Figure Rendering results based on different-style models ($4.5)
Layout
Scene Depth map
SDXL.
SDXI.
SDXL.
Three man.
Albest Finstein: John Biden, 3) Donald Trump
Superman. Lantem. 3) Halk.
Mesel. 2) Cristiono Ronaldo. 3) Kobe Bryant.
Figure
Rendering results on specific concepts ($4.5)
5 CONCLUSION
We propose novel 3DIS method that decouples image generation into two distinct phases: coarse- grained scene depth map generation and fine- grained detail rendering. In the scene depth map phase, 3DIS trains Layout-to- coarse-grained attributes instances, thus simplifying the training process. In the detail rendering phase, 3DIS leverages widely pre-trained ControlNet models to generate images based on the scene depth map, controlling the scene and ensuring that each instance is positioned accurately. Finally, our proposed detail renderer guarantees the correct rendering of each instance's details. Due the training- free nature of the detail rendering phase, our 3DIS framework utilizes the generative priors of various foundational models for precise rendering. Experiments on the COCO-Position bench- mark demonstrate that the scene depth maps generated by 3DIS create superior scenes, accurately placing each instance in its designated location. Additionally, results from the COCO-MIG bench- mark show that 3DIS significantly outperforms previous training- free rendering methods and rivals statc-of-ther -otte-of-hhe-at adapter-based approaches. envision that 3DIS will enable users to apply wider range of foundational models for the future, we will continue to explore the integration of 3DIS with baTeb foundational models.
network that focuses solely global scene construction and the
generation and be extended to more applications. In
Acknowledgements. This work was supported in part by the Natural Science Foundation of Zhe- jiang Province (LDT23F02023F02) and Fundamental Research Funds for the Zhejiang Provincial Universities (22088024-00208)
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Omer Bar-Tal, Lior Yariv, Yaron Lipman, and Tali Dekel. Multidiffusion: Fusing diffusion paths for controlled image generation. arXiv preprint wwXiv:2302.08113. 2023.
Jingye Chen, Yupan Huang, Tengchao Lv, Lei Cui, Qifeng Chen, and Furu Wei. Textdiffuser: Diffusion models as text painters, 2023. URL https: hetps://arxiv.org/ wttp:////b//bbbb..11.omm 10855
Yutong Feng, Biao Gong, Di Chen, Yujun Shen, Yu Liu, and Jingren Zhou. Ranni: Taming text-to- image diffusion for accurate instruction following. Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pp. 4744-4753, 2024.
Daiheng Gao, Shilin Lu, Shaw Walters, Wenbo Zhou, Jiaming Chu, Jie Zhang, Bang Zhang, Mengxi Jia, Jian Zhao, Zhaoxin Fan, et al. traseanything: Enabling concept erasure rectified flow transformers. arXiv preprint arXiv:2412.2000
2024.
Nocholas Guttenberg. Diffusion with offset noise, 2023, URL
Jonathan Ho. Classifier-free diffusion guidance. ArXiv, abs/2207. 2022.
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- 00364
based generative models, 2022.
Kasiopy. Multi-resollution noise for diffusion model training, 2023. URL https: http://wandb../ al/
jotnowhittaerr/mmlt
noise/reports/ /ttp:///eeeeeeee.mi..ee.ee.ee//r Last accessed 17 Nov 2023.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2017.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan- Yen Lo, Piotr Dollár, and Ross Girshick. Segment anything. arXiv:2334.026643 2023.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-iman pre-training with frozen image encoders and large language models. In International conference machine learning, pp. 19730-19742. PMLR, 2023a.
Kexin Li, Zongxin Yang. Lei Chen, Yi Yang, and Jun Xiao. Catr: Combinatorial-dependdee audio- queried transformer for audio-visual video segmentation. In Proceedings of the 31st ACM inter- national conference multimedia, pp. 1485- 1494, 2023b.
Yuheng Li, Haotian Liu, Qingyang Wu, Fangzhou Mu, Jianwei Yang, Jianfeng Gao, Chunyuan Li, and Yong Jae Lee. Gligen: Open-set grounded text-to-image generation. CVPR, 2023c.
Zhen Li, Mingdeng Cao, Xintao Wang, Zhongang Qi, Ming-Ming Cheng, and Ying Shan. Pho- tomaker: Customizing realistic human photos via stacked id embedding. In Proceedingy of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 8640-8650, 2024.
Tsung- Yi Lin, Michael Maire, Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays, Pietro Perona, Deva Ramanan, Lawrence Ziticck, and Piotr Dollár. Microsoft coco: Common objects in context, 2015.
Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint urXiv:2303.05499 2023.
Shilin Lu, Yanzhu Liu, and Adams Wii-Kin Kong. Tf-icon: Diffusion-based training-free cross- domain image composition. In ICCV, 2023.
Shilin Lu, Zilan Wang, Leyang Li. Yanzhu Liu, and Adams Wai-Kin Kong. Mace: Mass concept crasure in diffusion models. CVPR, 2024a.
Shilin Lu, Zihan Zhou, Jiayou Lu, Yuanzhi Zhu, and Adams Wii-Kin Kong. Robust watermarking using generative priors against image editing: From benchmarking to advances. arXiv preprint arXiv:2410. 187775 2024b.
11

---

Published as a conference paper at ICLR 2025
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhom, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXin:2307.01195, 2023.
Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D. Manning. Stanza: A Python natural language processing toolkit for many human languages. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguisties: System Demonstrations, 2020.
Lingteng Qiu, Guanying Chen, Xiaodong Gu, Qi Zuo, Mutian Xu, Yushuang Wu, Weihao Yuan, Zilong Dong, Liefeng Bo, and Xiaoguang Han. Richdreamer: A generalizable normal-depth diffusion model for detail richness in text-to-3d. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pultern Recognition, pp. 9914 2024.
René Ranftl, Alexey Bochkovskiy, and Vladlen Koltun. Vision transformers for dense prediction. ICCV, 2021.
Robin Rombach, Andreas Blattmann, Dominík Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models, 2022.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Om-
mer. Stable diffusion version 2, 2023. URI.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedi- cal image segmentation. MICCAI, awss/5055044. 2015.
Christoph
Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarezyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki, Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint arXiv:21 11.22/111.1114... 2021.
Gabriela Ben Melech Stan, Diana Wofk, Scottie Fox, Alex Redden, Will Saxton, Jean Yu, Estelle Aflalo, Shao- Yen Tseng, Fabio Nonato, Matthias Muller, et al. Ldm3d: Latent diffusion model for 3d. arXiv preprint 2:X.::2305.10853 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informa- tion processing systems, 30, 2017.
Wenguan Wang, Yi Yang, and Yunhe Pan. Visual knowledge in the big model era: Retrospect and prospect. Frontiers of Information Technology & Electronic Engineering, 26(1):1-19, 2025.
Xudong Wang, Trevor Darrell, Sai Saketh Rambhatla, Rohit Girdhar, and Ishan Misra. Instancedif- fusion: Instance-level control for image generation, 2024.
Yuxiang Wei, Yabo Zhang, Zhilong Ji, Jinfeng Bai, Lei Zhang, and Wangmeng Zuo. Elite: Encoding visual concepts into textual embeddings for customized text-to-image generation. arXiv preprint arXiv:2302. 13848, 2023.
Jayy Xiao, Liang Li, Henglei Lv, Shuhui Wang, and Qingming Huang. R&b: Region and boundary aware zero-shot grounded text-to-image generation. arXiv preprint arXiv22310..8872, 2023.
Jinheng Xie, Yuexiang Li, Yawen Huang, Haozhe Liu, Wentian Zhang, Yefeng Zheng, and Mike Zheng Shou. Boxdiff: Text-to-image synthesis with training-free box-constrained diffu- sion. ICCV, 2023.
Rui Xie, Ying Tai, Chen Zhao, Kai Zhang, Zhenyu Zhang, Jun Zhou, Xiaoqian Ye, Qian Wang, and Jian Yang. Addsr: Accelerating diffusion-based blind super-resolution with adversarial diffusion distillation. arXiv preprint arXiv:244.001717, 2024.
Lihe Yang, Bingyi Kang, Zilong Huang, Zhen Zhao, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything v2. arXiv preprint arXiv:2406. 2024a.
Xiangpeng Yang, Linchao Zhu, Hehe Fan, and Yi Yang. Videograin: Modulating space-time atten- tion for multi-grained video editing. arXiv preprint arXiv:2502. /7258, 2025.
12

---

Published as a conference paper at ICLR 2025
Yi Yang, Yueting Zhuang. and Yunhe Pan. Multiple knowledge representation for big data artificial intelligence: framework, applications, and case studies. Frontiers of Information Technology & Electronic Engineering, 22(12):1511-1555 2021.
Zhengyuan Yang, Jianfeng Wang. Zhe Gan, Linjie Li, Kevin Lin, Chenfei Wu, Nan Duan, Zicheng Liu, Ce Liu, Michael Zeng, and Lijuan Wang. Reco: Region-controlled text-to-image generation. In CVPR, 2023.
Zongxin Yang. Guikun Chen, Xiaodi Li, Wenguan Wang. and Yi Yang. Doraemongpt: Toward understanding dynamic scenes with large language models. ICML, 2024b.
Zongxin Yang, Jiaxu Miao, Yunchao Wei, Wenguan Wang. Xiaohan Wang, and Yi Yang. Scalable video object segmentation with identification mechanism. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024c.
Hu Ye, Jun Zhang. Sibo Liu. Xiao Han, and Wei Yang. Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models. arXiv preprint arxiv:2308.065 067221, 2023.
Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. ICCV, pp. 3836-3847, 2023.
Yuxuan Zhang, Jiaming Liu, Yiren Song, Rui Wang, Hao Tang, Jinpeng Yu, Huaxia Li, Xu Tang, Yao Hu, Han Pan, al. Ssr-encoder: Encoding selective subject representation for subject-driven generation. CVPR, 2024.
Chen Zhao, Weiling Cai, Chenyu Dong, and Chengwei Hu. Wavelet-based fourier information interaction with frequency diffusion adjustment for underwater image restoration. CVPR, 2024a.
Chen Zhao, Chenyu Dong, and Weiling Cai. Learning a physical-aware diffusion model based on transformer for underwater image enhancement. arXiv preprint arXiv:2403.01497, 2024b.
Wenliang Zhao, Lujia Bai, Yongming Rao, Jie Zhou, and Jiwen Lu. Unipe: unified predictor- corrector framework for fast sampling of diffusion models. NeurlPS, 2023.
Dewei Zhou, Zongxin Yang, and Yi Yang. Pyramid diffusion models for low-light image enhance- ment. In IJCAI, 2023.
Dewei Zhou, You Li, Fan Ma, Zongxin Yang, and Yi Yang. Mige: Multi-instance generation con- troller for text-to-image synthesis. CVPR, 2024a.
Dewei Zhou, You Li, Fan Ma, Zongxin Yang, and Yi Yang. Mige++: Advanced
generation controller for image synthesis. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024b.
Dewei Zhou, Ji Xie, Zongxin Yang, and Yi Yang. 3dis-flux: simple and efficient multi-instance generation with dit rendering. arXiv preprint #xxi:2501.00533 2025.
Zhenglin Zhou, Fan Ma, Hehe Fan, Zongxin Yang, and Yi Yang. Headstudio: Text to animatable head avatars with 3d gaussian splatting. In European Conference on Computer Vision, pp. 145- 163. Springer, 2024c.
Wenjie Zhuo, Fan Ma, Hehe Fan, and Yi Yang. Vividdreamer: invariant score distillation for hyper-realistic text-to-3d generation. In European Conference on Computer Vision, pp. 122-139. Springer, 2024.
13


Reviews:
[{"review_id": "8QkzCq03G1", "reviewer": "Reviewer_m1R4", "summary": "This work focuses on controllable image generation and points out the unified adapter challenge of multi-instance generation (MIG) methods: current MIG methods uses a single adapter to simultaneously handle instance positioning and attribute rendering. Such a unified structure complicates the development of detail renderers because it requires a large number of high-quality instance-level annotations. To this end, this work proposes a two-stage generation paradigm: (1) generating a coarse-grained depth map from layout; (2) rendering fine-grained instance details from depth map. This design enables the MIG adapter to be seamlessly integrated into various foundational models such as SD2 and SDXL without specific training. Extensive experiments demonstrate the effectiveness and flexibility of the proposed method.", "questions": "1. How does the layout adapter connect to the text-to-depth model? Does it work like ControlNet?\n2. Now that depth information is introduced, can this method control the front-back relationship of overlapping instances?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. This paper has a very clear motivation and solves the pointed problems very well. Although the idea of ​​using depth map as a coarse-grained scene guidance is simple, it effectively solves the limited adaptability challenge in existing MIG methods. Extensive qualitative and quantitative results (e.g., Table 2, Figure 3, and Figure 4) demonstrate that the proposed method has strong flexibility and can be applied to a variety of foundational models.\n2. The presented results are promising, achieving state-of-the-art performance on instance attributes and locations. In particular, the proposed method can be further combined with other controllable image generation methods such as GLIGEN and MIGC to improve their performance for multi-instance generation..\n3. The paper is well written and clearly structured.", "weaknesses": "1. The proposed framework depends on many existing models. For example, a) the text-to-depth model is obtained by fine-tuning LDM3D; b) a pretrained depth-conditioned ControlNet is required for depth layout injection; c) the detailed renderer relies on SAM to segment instances from depth. These dependencies weaken the originality of the paper and may compromise the robustness of the proposed framework.\n2. The paper does not discuss depth ambiguity when multiple bounding boxes overlap. For example, given two partially overlapping instances, how can the model tell which is in front and which is behind?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kVyTnY6Xy8", "reviewer": "Reviewer_sPv1", "summary": "The paper introduces Depth-Driven Decoupled Instance Synthesis (3DIS), a novel framework aimed at enhancing multi-instance generation (MIG) in text-to-image generation. 3DIS addresses the limitations of existing MIG methods by decoupling the generation process into two distinct stages: generating a coarse scene depth map for accurate instance positioning and rendering fine-grained attributes without additional training. This two-stage approach allows for greater control over both layout and attribute details, leading to improved scene composition and integration with various foundational models. The extensive experimental results demonstrate that 3DIS significantly outperforms existing techniques in layout accuracy and fine-grained attribute rendering across established benchmarks.", "questions": "1. How does the proposed 3DIS framework handle the challenges of noisy annotations in the input data?\n\n2. The training procedure relied on the LAION-art dataset, which may not fully represent the diversity of real-world images or scenarios.\n\n3. What strategies could be employed further to improve the robustness of the generated depth maps? \n\n4. Given the dependence on pre-trained models for the training-free detail rendering process, how might variability in these models' performance impact the overall effectiveness of the 3DIS framework?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "- **Innovative Framework:** The 3DIS framework effectively decouples the multi-instance generation process, allowing for improved accuracy in scene composition and instance positioning.\n\n- **Training-Free Approach:** The ability to render fine-grained attributes without additional training is a significant advantage, making the model more accessible and easier to integrate with existing systems.\n\n- **Robust Performance:** Extensive experiments on benchmarks such as COCO-Position and COCO-MIG show that 3DIS consistently outperforms state-of-the-art methods in both layout precision and attribute rendering, indicating its effectiveness in practical applications.", "weaknesses": "- **Limited Dataset Scope**: The experiments relied on the LAION-art dataset and COCO benchmarks, which may not fully represent the diversity of real-world images or scenarios. Expanding the evaluation to include a broader range of datasets could provide a more comprehensive assessment of the model's generalizability.\n\n- **Evaluation Metrics Limitations**: The selected evaluation metrics, while informative, may not capture all aspects of image quality or user satisfaction. Additional metrics could enhance the evaluation of the generated images' visual appeal and usability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IaX8DEdys6", "reviewer": "Reviewer_LTry", "summary": "The paper proposes a two-stage framework for multiple instance generation, which is 1)  generating a depth map conditioned on the layout map, 2) generating the image conditioned on the generated depth map with pretrained ControlNet.  Some techniques are also proposed to improve the results, such as using low-pass filter on the depthmap, using segment model for detail renderer.  The claimed improvements are on layout accuracy, instance accuracy and image quality.", "questions": "1. How does the occlusion of the layout impact the depth map generation? Since there is no clue of which one is nearer. It is better to evaluate the generation quality of the depth map, and how the depth map impact the final generation.\n2. The parameters of the low pass filter, and how to choose it are missing.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The two stage framework for MIG seems to be novel, and the findings that depth map can be utilized as an intermediate product for  layout conditioned generation. \n2. Some practical techniques are proposed, which may be useful in practical  applications.", "weaknesses": "1. Compared to one-stage methods, the proposed one can be computionally inefficient, the extra computation cost should be clearly evaluated and presented. \n2. The paper heavily based on previous works, the novelty can be limited. \n3. It seems that the key contribution to the improvement of MIOU is the depth map, especially as shown in Fig.3 and Fig. 4, the depth map already demonstrats advantage compared to MIGC results. Further discussion on why the depth map can better align with the layout can be added.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UANRAv1Ig2", "reviewer": "Reviewer_Pjgj", "summary": "This paper introduces 3DIS for multiple instance generation(MIG) tasks in text-to-image generation. 3DIS decouples the image generation process into two stages. First, 3DIS generates a coarse scene depth map that accurately positions instances and aligns their coarse attributes. Second, it renders fine-grained instance attributes using pre-trained ControlNet on any foundation diffusion model without additional training. This approach enables seamless integration with diverse foundational models with a Depth-Controlled method, providing a robust and adaptable solution for advanced multi-instance generation.", "questions": "1. Can the layout-to-depth model generate diverse depth maps for the same layout? Like changing the pose, shape, and other attributes of instances?\n\n2. It seems that sometimes the background scene can be blurry or unpleasant. Does that have something to do with the instance-wise rendering step?\n\n3. Are there any failure cases that can make analyses on the Layout-to-depth model?\n\n4. How long will it take to generate a single image?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. Proposed a novel approach for MIG task with Depth control, the proposed method shows good generalizability and performance on different MIG benchmarks.\n\n2. The proposed 3DIS can be seamlessly integrated into various pre-trained diffusion models, without requiring fine-tuning processes for each pretrained diffusion model.\n\n3. The incorporation of 3D depth information can provide a better understanding of instances' attributes, thus benefiting the generation process of 2D images.", "weaknesses": "1. This pipeline is largely dependent on the fine-tuned Layout-to-Depth generation model, and the final output uses the foundation diffusion model to render instances in a strictly confined region(depth). There might be inconsistencies between the layout-to-depth generation model and the renderer model, thus hindering the final performance.\n\n2. The slightest artifacts in the depth map might cause great changes in the final generation results, like unwanted additional objects or background depth.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "keElNLM34p", "reviewer": "Reviewer_dKuo", "summary": "In this paper, the authors propose a novel multi-instance generation method, which uses a depth map as the intermediate generation step to decouple the whole process into two stages. Based on the depth map, the following rendering process only applies existing models, such as SAM, ControlNet, SD, and so on. Experiments show that the generated image is more in line with the given layout and description.", "questions": "Please see the weakness.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed method is flexible and can combine many existing generation models.\n2. The location improvement is pretty obvious compared to previous SoTA methods.", "weaknesses": "1. The locations of the cat and lamp are not correct in Figure 1.\n2. The details of the comparison should be clarified. For example, it mentions that MIGC only supports SD1.5 in Figure 1, but the visual results of the proposed method in Figures 3 and 4 are based on SD2 and SDXL. In addition, it's unclear which version of SD is used in Table 1.\n3. It's better to provide some inference efficiency analysis since the proposed method contains multiple steps.\n4. The visual quality is not very satisfactory. It's better to apply user study to evaluate user preferences.\n5. It's better to add the image quality metrics to the ablation study in Tables 3 and 4.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)