InstanceAssemble: Layout-Aware Image Generation via Instance Assembling Attention
Authors: Qiang Xiang
Shuang Sun
Binglei Li
Dejia Song
Huaxia Li
Nemo Chen
Xu Tang
Yao Hu
Junping Zhang
Venue: NeurIPS 2025 poster ()
Keywords: Text to Image
Latent Diffusion Model
Layout to Image
Primary area: deep_learning

Abstract: Diffusion models have demonstrated remarkable capabilities in generating high-quality images. Recent advancements in Layout-to-Image (L2I) generation have leveraged positional conditions and textual descriptions to facilitate precise and controllable image synthesis. Despite overall progress, current L2I methods still exhibit suboptimal performance.
Therefore, we propose InstanceAssemble, a novel architecture that incorporates layout conditions via instance-assembling attention, enabling position control with bounding boxes (bbox) and multimodal content control including texts and additional visual content. Our method achieves flexible adaption to existing DiT-based T2I models through light-weighted LoRA modules. 
Additionally, we propose a Layout-to-Image benchmark, Denselayout, a comprehensive benchmark for layout-to-image generation, containing 5k images with 90k instances in total. We further introduce Layout Grounding Score (LGS), an interpretable evaluation metric to more precisely assess the accuracy of L2I generation. 
Experiments demonstrate that our InstanceAssemble method achieves state-of-the-art performance under complex layout conditions, while exhibiting strong compatibility with diverse style LoRA modules.
The code and pretrained models are publicly available at \url{https://github.com/FireRedTeam/InstanceAssemble}.

InstanceAssemin Layout-Aware Image Generation via Instance Assembling Attention
Qiang Xiang -Shangg Sun?, Binglei Li
Dejia Song", Huaxia Li", Yibo Chen², Xu Tang", Yao Hu², Junping Zhang Shanghai Key Laboratory of Intelligent Information Processing, College of Computer Science and Artificial Intelligence, Fudan University
Shanghai Innovation Institute fqxiang24, b11i24}0m. fudan edu. en, jpzhangdfudan. edu cn, (sunshuangl, dejiasong, lihuaxia, zhaohaibo, tangshen, xi xiahou)@xiaoohongshhe com
EXiaohongshu Inc.
INSTANCEE ASSEMBLE
Figure Layout-aware image generation result by InstanceAseemble. We show image generation result under precise layout control, ranging from simple to intricate, sparse to dense layouts.
Abstract
Diffusion models have demonstrated remarkable capabilities in generating high- quality images. Recent advancements in Layout-to-Image (L2I) generation have leveraged positional conditions and textual descriptions to facilitate precise and controllable image synthesis. Despite overall progress, current L21 methods still exhibit suboptimal performance. Therefore, we propose InstanceAssemble, novel attention,
architecture that incorporates layout conditions via anstance-assemble enabling position control with bounding boxes (bbox) and multimodal content control including texts and additional visual content. Our method achieves flexible adaption to existing DiT-based T2I models through light-weighted LoRA modules. Additionally, we propose a Layout-to-Iman benchmark, Denselayout, a compre- hensive benchmark for tuyout-to-imag generation, containing 5k images with 90k instances in total. We further introduce Layout Grounding Score (LGS), an inter- pretable evaluation metric to more precisely assess the accuracy of L21 generation. Experiments demonstrate that our InstanceAssemble method achieves state-of-the art performance under complex layout conditions, while exhibiting strong compati- bility with diverse style LoRA modules. The code and pretrained models are pub- licly available at httph::// htts::///tttbbb.t. www.eoooeeeeeeeaaaaa.eeeeee hat.c///rr.eeeeeeeeeeaaaaaeaaaaa Insaaaceecccaceeeeeeeeee
*Corresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Introduction
Diffusion models [22 have revolutionized image generation task, with architectures like Diffusion Transformer (DiT) 40 offering superior quality over traditional UNet-based frameworks. Recent implementations such as Stable Diffusion 3/3.5 [SS 49 and Flux.1 4 further enhance text-to-image aligmment, paving the way for advancements in layout-controlle (L2I) generation is task that focuses creating images under layout conditions, allowing users to define spatial positions and semantic content cach instance explicitly. This task faces several significant challenges: (i) ensuring precise layout alignment while maintaining high image quality, (ii) preserving object positions and semantic attributes accurately during the iterative denoising process of diffusion models, and (iit) supporting various types of reference conditions, such texts, images and structure information. These challenges highlight the complexity of achieving robust and flexible layout-controlled image generation.
generation. Layout-to-Iman Imaage
Existing 1.21 methods can broadly categorized into training-free and training-based approaches, both possessing distinct advantages and limitations. Training-free methods [55] [3]
g 5 27 rely on heuristic techniques without modifying the base model. However, these methods often exhibit degraded performance in complex layouts, demonstrate high sensitivity to hyperparameter tuning, and suffer from slow inference speed, which make them less practical for real-world applications. 61 involve training specific layout modules to improve layout alignment, which introduces a significant amount of extra parameters and increases training complexity and resource requirements. Additionally, existing L21 evaluation metrics ex- hibit inaccuracies, such as false acceptance and localization errors. These identified shortcomings necessitate algorithm innovation for effective and efficient layout-controlled image generation.
In contrast, training- -baee methods 63 53
64
29
Therefore, we propose IsseanceAssemble, a novel framework that systematically tackles these issues through innovative design and efficient implementation. Our approach introduces cascaded InstanccAssemble structure, which employs a multimodal interaction paradigm to process global prompts and instance-wise layout conditions sequentially. By leveraging the Assemble-MMDDD architecture, we apply an independent attention mechanism to the semantic content of each instance, thus enabling effective handling of dense and complex layouts. Furthermore, for lightweight adaptation, adding only 71M parameters to SD3-Medium (2B) and 102M to Flux.J (11 (11.8B). Our method enables position control with bounding boxes and multimodal content control including texts and additional visual content. This lightweight design preserves the capabilities of the base model while enhaneing flexibility and efficiency. We also introduce novel metric called Layout Grounding Score (L.GS) to ensure accurate evaluation for 1.21 generation, alongside a test dataset DenseLayout. This metric provides a consistent benchmark for assessing layout alignment.
adopt LoRA (23)
Our method achieves state-of-the-art performance across benchmarks and demonstrates robust layout alignment under wide variety of scenarios, ranging from simple to intricate, sparse to dense layouts. Notably, despite being trained on sparse layouts (< 10 instances), our approach maintains robust generalization capability dense layouts (2 10 instances), confirming the effectiveness of our proposed InstanceAssemble. The main contributions are listed below.
1. We propose cascaded Instance structure that processes global text prompts and layout conditions sequentially, enabling robust handling of complex layouts through an independent attention mechanism.
2. By leveraging LoRA (23 we achieve efficient adaptation with minimal extra parameters (3.46% on SD3-Medium and 0.84% on Flux.1), supporting position control with multimodal content control while preserving capabilities of base model.
3. We propose a new test dataset DenseLayout and a novel metric Layout Grounding Score (LGG)) for Layout-to-Imaad evaluation. Experimental results demonstrate that our approach achieves state-of-the--rt performance and robust capability under complex and dense layout conditions.
2 Related Work
Text-to-Image Image Generation Text-to-image synthesis 47
45
40
30
17
6
3 has witnessed
rapid progress with the development of diffusion models. Initial works as the denoising architecture, leveraging cross-attention mechanisms to inject text-conditioning
45
utilize UNet [46]

---

mixdal Pwope
planal Proept
RESSH
Jeage
IPOOT
Srage
Laage
Nolse
Layout
Pramblle-
IOLT
Toates Irofance Contiens
merl the
Fourder
roef
www.beeeeeeeeee Altry
Iegend
C Concatmnnttion
Frozen
TrainableeLood)
Figure 2: The proposed InstanceAsseniee pipeline. Various layout conditions are processed by the Layout Encoder to obtain instance tokens, which guide the image generation via Assemble-MMDTT. In Assemble-MMDiT, the instance tokens interact with image tokens through the Assembling-Are
signals. Recently, researches 6 15
have used the Multimodal Diffusion Transformer (MMDiT) architecture, marking significant improvement.
34
Layout-to-Image Generation Layout-to- lo-ae generation enables image generation under layout conditions, which is defined as spatial positions with textual descriptions. Existing approaches can be broadly categorized into training-free and training-based paradigms.
Training-free methods leverage pretrained text-to-image diffusion models without additional train- ing. A common strategy involves gradient- based guidance during denoising align with layout conditions [2,
[12 41 13
[8]
[0]] Also, there are methods that directly manipulate latents through
well-defined replacing or merging operations S 48 2 or enforce layout alignment via spatially
constrained attention masks [5 20). GrounDiT 27 exploits semantic sharing in DIT: cropped noisy
patch and the full image become semantic clones when denoised together, enabling mayout-to-image generation by jointly denoising instance regions with their corresponding image context. Other approaches generate each instance separately and employ inpainting techniques to compose the final image (37]. However, these methods demonstrate decent performance primarily simple and sparse layouts, while their accuracy decreases more complex layouts. Some methods require hyperparam- eter tuning specific different layout conditions, reducing their adaptability. Furthermore, additional gradient computations or latent manipulations result slow inference speed, thus limiting their applicability real-world scenarios.
Training- based methods explicitly incorporate layout conditioning through architectural modifica- tions. Most approaches inject spatial constraints via cross cots-attention [63] 57 36.
25
59
[6]
54
self-attention [10 53 28) Some works propose dedicated layout encoding modules 64
65
58
62
adopt two-stage pipeline that generates images after predicting depth map with layout condi- tions 14 (6). Other works leverage autoregressive image generation models [9] These methods
suffer high computational costs due excessive parameters.
Method
Preliminaries Recent state-of-the-an text-to-image models such as SD3 [1] and Flux 4 adopt the Multimodal Diffusion Transformer (MMDiT) as the backbone for generation. Unlike traditional UNet-based cross-attention approaches, MMDiTs treat image and text modalities in a symmetric manner, which leads to stronger prompt alignment and controllability. These models are trained under the flow matching framework [3), which formulates generation as learning a continuous velocity field that transports noise to data. Given clean latent x and Gaussian noise (O I), an

---

interpolated latent is defined as
Zt t)x + te, E [0,1].
(I)
The training objective minimizes the squared error between the predicted velocity and the target velocity (E (-xx) -);
FIM = EE-,((. [lvo(2e.
- (E- -
(2)
where ug is implemented with an MMDiT backbone.
Problem Definition -yyout-to-Image generation aims to synthesize images with precise control through global prompt p and instance-wise layout conditions L. The layout conditions comprise N instances {4:)= where each instance 4 is defined by its spatial position b, and content c;:
(1, .,,,..,,N IN), where = (cr,
In our framework, spatial positions represented as bounding boxes, while instance content be specified through multiple modalities: textual instance content and additional visual instance content, including reference images, depth maps and edge maps.
propose Instance framework with Layout Encoder to encode the layout conditions and Assemble-MMDiT effectively integrate the encoded layout conditions with image features.
3.1 Layout Encoder
We use Layout Encoder (Fig. left-bottom panel) to encode each instance 1, and the tokens are denoted h² = [h', ....,...... h which represents the layout information of each instance. Given the spatial position of the instance (bounding box), we first enhance the spatial representation through DenseSample, Given bounding box b; = (#1, y1, w,h) E [o, 1]4 with top-left coordinates (I1, y1) and size (w, h),, we generate K²2 uniformly spaced points:
P. 21 Kiz-
{
{
K 1n Kig
w
1 K
)
Ka, ky E (O....,K-11
K- 11}
(4)
Then, following GLIGEN we compute the textual instance tokens as:
hf MLP ([T(c), Fourier(P:)),,
(5)
where represents the text encoder, Fourier(-) denotes Fourier embedding (51). [. .] denotes concatenation along the feature dimension, and MLP a multi-layer perception.
Additionally, we can use additional visual instance content to better improve performance. Given the visual instance content, we first extract features using the VAE encoder of the base model, then project them to the unified instance token space through a MLP:
h MLP (VAE(c:))
3.2 Assemble-MMDiTT
We observe that applying attention between all image tokens and instance tokens results in suboptimal performance under complex layout conditions (e.g.. overlapping, tiny objects). To address this, we introduce Assemble-MMDiT (Fig. 2 right-bottom panel), which enhances the location of each instance while maintaining compositional coherence with other instances. Our method processes each instance independently through attention modules with its associated image tokens, followed by weighted feature assembling.
Formally, given image tokens h E R Cx (where C' denotes the latent channel size and [W, H] the latent size) and instance tokens h E RCCN
apply AdyLayerNorm (56), followed by our as shown in Fig. 2 (right-bottom panel). panel), We crop the image tokens h by the bbox b, of each instance and get hi = h b. Cx Then, we project the cropped image tokens hf, and their corresponding instance tokens I; into queries (Q"). Q"). keys (K K*)). ) and
proposed Assembling-Atue
values (V=r, V4), and then apply attention:
h h² Attention "00"",""))11 [K**,, K[]], v'()).

---

where [., 1 denotes concatenation along the token dimension. The updated tokens are assembled across instances. Let M E NW represent the instance density map, calculating the counts of instances, The assembled image tokens h and instance tokens h" are computed as:
ht h [.1,5] M
Σ h i. where [0, (0,W- 11. E 10, H
(8)
h" h [c h%
As illustrated in Fig. the top row demonstrates that assembling mechanism ensures instance tokens attend only to relevant image regions, where unrelated regions are left in black. The middle row reveals that the mechanism effectively guides global prompt tokens to focus on their correct spatial positions. In contrast, generation without explicit layout control (bottom row) results in localization errors ("British Shorthair" in wrong location) semantic inconsistencies ("dog" missing).
Furthermore, preserve the generation capability of the original model and mitigate conditional conflicts between global prompt and layout conditions, caded mechanism as shown in Fig. 2 (right-above panel). In design, the global text prompt and image latents are passed through original MMDiT first, then the image tokens along with instance tokens processed by our Assemble-MMDii module. The first step captures global context and ensures generation quality, while the second step ensures instance layout alignment. Besides, train Assemble-MMDi with LoRA, significantly reducing both the training cost and inference costs.
Figure 3: (Top) instance- image attention map w/ layout. (Middle) global prompt- image attention map w/ layout. (Bottom) global prompt-image attention map w/o layout.
employ cas-
(4) Rally Car? CropyqA: Yes
(b) slack shoulden bag? CropygA: DetectieU: 0.54
MEndow? (c)
(d) Tower?
SANCoU:
8.77
BinaryIoU: 1
Detoctiou: 9.69
DetoctioU:
9.69
DetectIeU: 6.51
Figure 4: Failure cases of other metrics. (a) false acceptance in CrApVQA.(b) false rejection in CropVQA, (c) localization error in SAMIoU, and (d) discontinuous in BinaryloU.
3.3 Benchmark: DenseLayout and Layout Grounding Score
The Layout-to-Image task aims to generate images that align precisely with provided layouts, eval- uating both spatial accuracy and semantic consistency (e.g., color, texture, and shape, if provided).
The existing metrics (AP/AR) for object detection [10 28
35 53 are suboptimal. They assume fixed category set and rely on inappropriate precision/recall for binary layout outcomes. VLM-based cropped VQA methods [6]] suffer false acceptance (Fig. 4 (a)) and false rejection (Fig.E(b) While spatial-only metrics like SAMIoU [T] ignore appearance consistency(Fig. (c)), Grounding
based (35] binary IoU thresholds 664 (4 fail to capture continuous layout precision(Fig. (d). Thus, we propose Layout Grounding Score (LGS), which integrates both spatial accuracy and semantic accuracy:
1. Spatial Accuracy (DetectloU): we detect all instances via an off-the-shelf detector [55] compute the loU against condition bbox, and report the global mean loU across all instances for equal weighting.
2. Semantic Accuracy: for instances with loU>0.5, we crop the predicted region and assess the semantic accuracy by its attribute consistency (color, texture, shape) via VLM-based VQA (60).
LGS supports open-set evaluation, uses DetectIoU to evaluate spatial accuracy and decouples the spatial and semantic check to avoid CropVQA [61 failures (shown in Fig. Furthermore,
5

---

Table 1: Quantitative comparison between our SD3-based InstanceAssemble and other L2I methods on LayoutSAM-Eval. 'he Crop VQA score is proposed in Creatilayout [6]] and the score InstanceDiff, MIGC and Creatilayout borrowed from Layout [61
CropVQA
Layout Grounding Soore
Global Quality
LayoutAMMEvaal
spatial" color?
texture shapet
mloUt colort texturef stapet
VQAT
Pickf CLIPT
Real Images(Upper Bound)
98.45
98.90
98.80
88.07
88.71
InstanceDiit (SD1.5)
87.99
69.16
72.78
71.08
78.16 16
63.14
66.82
65.86
86.42
21.16
11.73
MIGC (SD1.4)
71.24
50.70
52.99
20.69
12.56
HICO (realistiVisson\551)
90.92
69.82
73.25
71.69
70.68
53.16
55.71
54.61
86.53
21.77
9.47
CyeotiLayout (SD3-M)
92.67
74.45
77.21
75.93
45.82
38.44
39.68
39.24
92.74
21.71
13.82
InstanccAssemmblxouuss) (SD3-M)
94,97
77.53
80.72
80.11
78.88
63.89
66.27
65.86
93.12
21.79
12.76
Table 2: Quantitative comparison between our InstanceAssemblle and other L21 methods on DenseLayout.
Layout Grounding mloUf colort texturef shapet
Global Quality
Densel.ayout
VQAT Pickt CLIP
Real Images(Upper Bound)
92.35
76.52
80.78
79.78
InstanceDiff (SD1.5)
47.31
29.48
33.36
32.43
83.79
20.87
11.73
MIGC (SD1.4)
34.39
22.10
23.99
23.45
91.18
20.74
12.81
IIICO VraalistieVisioNVS55)
22.42
10.52
11.69
11.46
74.42
20.51
8.16
Creatil ayout (SD3-Medium) Instance-ssemblcccuss)) (SD3-Medium)
15.54
11.69
12.34
12.17
93.42
21.88
12.89
52.07
33.77
36.21
35.81
93.54
21.68
12.58
Regional- Flux (Flux.I-Dev)
14.06
11.34
11.91
11.84
92.94
22.67
10.66
RAG (Flux. 1-Dev)
17.23
14.22
14,62
14.55
22.28
11.01
(ssunncessemmleionnn) (Flux.I-Dev DeDe))
43.42
27.60
29.50
29. 14
93.36
21.98
11.38
(nstaaneeAseemb(ccourr)) (Flux. I-Schnell)
45.33
27.73
30.06
29.62
93.52
21.72
10.78
introduce DenseLayout, dense evaluation dataset for 1.21, which consists of 5k images with 90k instances (18.1 per image). The images in DenseLayout are generated by Flux. 1-Dev, tagged by RAM++ (24], detected by GroundingDINO (35), recaptioned by Qwen2.5-VL. 43 and filtered to retain those with 2 15 instances, thus providing dense layout conditions.
3.4 Training and Inference
During training, we freeze the parameters of the base model and only update the proposed Layout Encoder and Assemble-MMDiT module. We denote the adding parameters by The training objective is given by
C E...
[1 t, (c) x)|
(9)
where z, = (1- t)x + During inference, layout-conditioned denoising applied during the first 30% of diffusson steps, as the layout primarily forms in early stages [28] 64].
Experiments
4.1 Experimental Setup
Implementation Details The textual-only InstanceAssemble trained on SD3-Medium 15 and I-uv. 4 and the version with additional visual instance content only trained on SD3-Medium. We freeze the pretrained MMDiT backbone and only adapt the Layout Encoder and LoRA modules of Aasemble-MMDiT. Assemble-MMDii is initialized from pretrained weights, and LoRA with rank=4 is applied. In the SD3-based model all Assemble-MMii Flux-based model we adapt eight blocks (seven double-blocks and one single block) due resource constraints. During inference, the LoRA-based Assemble-MMDii is activated for the first 30% of denoising steps, while the global prompt-image phase uses the frozen backbone. This design yields 71M (SD3-M) and 102M (Flux. I-Dev) additional parameters for the textual-only setting; the variant with additional visual instance content (SD3-M) introduces 85M parameters. All models are trained on LayoutSAM [6] at 1024x x 1024 with Prodigy, for 380K iterations (batch size 2) SD3-M and
blocks are adapted, while in the

---

Table Parameter addition and time efficiency under sparse and dense layout conditions. We evaluated on 10% of the LayoutSAM-Ei and DenseLayout datasets at 1024x1024 resolution. are
optimized for 512x512 12 resolution, so their results are reported this scale.
Parameter Addition
Time Efficiency(s) (relative runtime incrcasc(5)) ()))
(relative parameter addition!
Sparse Layout
Layout
InstaneeDiff" (SD1.5)
369M (43%)
14.37 (+771%)
44.81 (+2754%)
MIGCYSD1.4)
57M (6.64%)
14.41 (+25,4%)
21.58 (+87.5%)
HICO (realisticVisionS55)
361M(33.99%)
4.11 (+92. (92.9
9.93 (+320%)
SreatLayyuuSDD-MM
1.2B (64.0%%
4.37 (+14.4%)
4.42 (+14.8%
Regional-Fllxx Flux(Flux. 1-Dev) Dev)
15.29 (+1139)
37.47 (+4188%)
RAG(Flux. I-Dev)
15.69 (+1
(+ 1925)
InstaneeAseemblccooss/SSS34
7IM (3 (3.46%)
7.19 (+88.2%)
13. (+248%)
lastaseeAscembbllousss..... I-Dev)
102M (0.84%)
8.21 (+14.3%)
10.28 (+419%)
MasianeeeAseembbbecossstttxxxxxx Schnelll)
102M (0.84%)
1.41 (+8.46%)
1.70 (+28.8%)
300K iterations (batch size 1) on Flux.1-Dew, 1-Dev, using HxH800 GPUs (7 days for SD3-M; 5 days for Flux.1-Dev)
Evaluation Dataset We use LayoutSAM-Evaa 66 to evaluate performance on fine-grained open-set sparse L2I dataset, containing 5k images and 19k instances in total (3.8 instances per image). To assess performance on fine-grained open-set dense L2I evaluation dataset, we use the proposed Denselayout, which consists of 5k images and 90k instances in total (18.1 instances per image). Following conventional practice, we also evaluate on coarse-gras coarse-grained close-set 1.21 evaluation dataset COCO [1. We combine COCO-Stuff and COCO-Instance annotations to create COCO-Layout evaluation dataset, containing 5k images and 57k instances in total (11.5 instances per image).
Evaluation Metric We evaluate the accuracy of L2I generation using our proposed LGS metric along with VQA proposed by Creatil [61], measuring spatial and semantic accuracy. We also employ multiple established metrics to measure overall image quality and global prompt alignment, including VQA Score [32), PickScore [26] and CLIPScore (21].
4.2 Evaluation on L2I with Textual-Only Content
Fine-Grained Open-Set Sparse L2I Generation Tab. presents the quantitative results of Instance- Assemble on LayoutSAM-Eva 61], reporting results using our proposed LGS, CropVQA [61] and global quality metrics. Our proposed mestanceAssemblee not only achieves SOTA in spatial and semantic accuracy of each instance, but also demonstrates superior global quality.
Fine-Grained Open-Set Dense 1211 Gener- ation Tab. 2 presents results on DenseLay- out. With the same SD3-Medium backbone, InstanceAseemble significantly outperforms CreatiLayout (mloU: 52.07 vs. 15.54) while maintaining comparable global quality. On Flux.1, it also yields large gains over Regional- Flux and RAG (e.g., mloU: 43.42 vs. 17.23 for RAG). showing that our cascaded Assemble- Attn design generalizes well across backbones. Compared to earlier UNet-based approaches such as InstanceDiff and MIGC, our method achieves higher spatial and semantic accuracy (mIoU: 52.07 vs. 47.31) without sacrificing re- alism. Overall, InstanceAssemble establishes consistent improvements in layout alignment while ensuring high image quality under chal- lenging dense layouts.
Table 3: Comparison between our SD3-based InstanceAssemble and other L21 methods on COCO-Layout. Since COCO don"t have detailed description for each instance, we cannot evaluate the attribute accuracy and only report the spatial accu- racy mloU.
LGS
Global Quality
COCO-Layout
mloUt
VQAt
Pickt
CLIP
Real Imagest Bound)
49.14
InstanceDiff (SDI.5) MIGC (SD1.4)
30.39
75.77
20.75
24.41
27.36
70.32
20.20
23.58
HICO ScalistioVisionNSS
18.88
50.61
20.38
20.72
CreatiLayout (SD3-M)
7.12
87.79
21.22
25.59
InstanceAsseemllle (SD3-M) M)
27.85
89.066
21.58
25.68
Coarse-Graines Closed-Set 121 Generation Tab. presents the quantitative result of Instance-
Assemble on COCO-Layour. Our proposed InstanceAssemble surpasses previous methods overall image quality but lags slightly behind InstanceDiff [53] in layout precision. We attribute this gap to:
7

---

Layout
MTCC
HTCO
CreatiLaynut
R Fux
BAG
Ours
Figure 5: Qualitative comparison of InstanceAssemble with other methods.
(i) InstanceDiff's fine-grained COCO training data with per-entity attribute annotations, and (ii) its entity-wis, generation strategy, which improves precision at significant computational cost (Tab.
Qualitative Comparison The comparative results in Fig. demonstrate that our proposed InstanceAssemble method achieves superior spatial precision and instance-caption alignment compared to baseline methods. For example, the third row, both InstanceDiff (33 and MIGC [44 gen- erate more than one shoes; HICO [9] fails to generate the specified New Balance shoe; Regional-Flux (5) does not ad- here the layout conditions; and the shoe generated by RAG (8) is not properly fused with the background. In con- trast, our method generates the correct instance, accurately placed and seamlessly integrated with the scene.
Neture
Xel
40 Todor
shand
10
38.5
InoU
117
Time Efficcincy and Parameter Addition We compare time efficiency and parameter addition with other L2I meth- ods, as shown in Tab. 4 Our method achieves SOTA perfor- mance on layout alignment with acceptable time efficiency and minimal parameter addition.
Figure 6: Quantitative results of In- stanceAssemble with additional vi- sual instance content.
4.3 Evaluation on 1.2I with Additional Visual Content
We evaluate three additional visual instance content: image, depth, and edge (see Fig. (). Unsurprisingly, using image as additional instance content yields the best performance, as it provides rich visual information. Although depth and edge capture texture and shape features, their performance remains inferior compared to image instance content. Neverthcless, visual modalities outperform textual-only instance

---

Table Ablation study on our proposed components on DenseLayout. "Assemble" refers the presence of the Assemble Attn module (architectural design). "Cascaded" indicates the interaction order: (V) means global prompt-image interaction followed by instance-imaae interaction (cascaded structure), while (X) means both are applied in parallel. "LoRA" specifies the training strategy for the AsMen module: (V) indicates training with LoRA, while (x) indicates full fine-tuning. "DenseSample" denotes whether the DenseSample spatial encoding used.
Layout Grounding Score
Assemble
Cascuded
LoRA
DenseSample
mloUf
colort
texture shupet
VQA
x
11.69
9.16
9.68
9.56
93.75
43.98
24.19
26.95
26.75
84.57
x
$1.28
32.68
34.94
34,58
93.33
52.07
33.77
36.21
38.81
93.54
content. Qualitative comparisons (Fig. further demonstrate that visual instance content leads to superior texture and shape alignment compared to text.
Lagout
Enuge
TestsCapth
Tege
Figure 7: Qualitative results of InstanceAssemble with additional visual instance content.
4.4 Ablation Study
We evaluated the contribution of each proposed component on SD3-M based InstanceAssemble in Tab. (5) The base model (SD3-Medium without additional modules) yields very low Layout Grounding Score, indicating poor layout and content control when instance information is not explicitly modeled. The introduction of Assemble-Attn module elevates spatial accuracy (mloU to 43.98) and boosts semantic metrics (color/texture/shape to 29/19/26..99 14.19/26.952267555 The cascaded design (V: prompt-image followed by Instance-image. X: parallel) resolves global quality degradation while maintaining layout alignment. Using LoRA to train Assemble-MMDiTT improves performance for two reasons: (1) retains the base model's capabilities compared to the fully fine-tuned version, and (2) it enables effective layout control with far fewer trainable parameters by introducing only lightweight low-rank matrices on attention projections. Finally, DenseSample further enhances spatial accuracy, instance semantic accuracy and image quality. Together, these refinements progressively collectively optimize layout to image modeling without compromising generation ability.
4.5 Applications
We demonstrate that InstanceAssemmlee is versatile and applicable to various tasks. It seamlessly integrates with Nomain-sppecife LoRA modules for multi-domain style transfer while maintaining layout consistency, as shown Fig. g Our proposed InstanceAssemble can cooperate with distilled models such as Flux. 1-Schnell detailed synthesis. Our approach demonstrates both style adaptability and computational efficiency, making it well-suited for controllable generative design applications.
as illustrated in Tab. Tab.2 achieving geometric layout control and

---

FRE
Figure 8: The adaption of Cute3D (52/ Oil Painting /39// Ghibli 38) LoRA with our methods.
The proposed InstanceAssemble successfully adapts diverse style lora and maintaining superior layout alignment.
5 Conclusion
We present InstanceAssemble, a novel approach for Layout-to-Image generation. Our method achieves state-of-the-an layout alignment while maintaining high-quality generation capabilities of DiT-based architectures. We validate Instance across textual instance content and additional visual instance content, demonstrating its versatility and robustness. Our layout control scheme also successfully adapts diverse style LoRAs while maintaining superior layout alignment, demonstrating cross-domain generalization capability. Futhermore, we introduce Layout Grounding Score metric and DenseLayour evaluation dataset to validate performance under complex layout conditions.
Limitations and Future Work While work advances controllable generation by unifying precise layout control with the expressive power of diffusion models, several limitations remain. First, our design currently requires sequential Assemble-MMDD parallelization strategies is an important direction. Second, although our approach is effective under wide range of layouts, image fidelity degrade in extremely dense or highly complex cases.
calls, which may incur inefficiency; exploring
Broader Impacts InstanceAssemble expands the frontier of structured visual synthesis by providing fine- ame-ggained layout control and high-quality multimodal generation. However, its powerful generative capabilities may also introduce risks. In particular, malicious use could enable the creation of misleading deceptive layouts, exacerbating the spread of disinformation. The model may also raise privacy concerns if applied to sensitive data, and like many generative systems, inherits and may amplify societal biases present in training corpora. We encourage responsible deployment and continued investigation into safeguards that mitigate these risks while enabling beneficial applications in design, education, and accessibility.
10

---

Acknowledgments and Disclosure of Funding
This research was supported by the National Natural Science Foundation of China (NSFC 62576103, 62176059). The computations were conducted using the CFFF platform at Fudan University. Part of this work was carried out during an internship Xiaohongshu.
References
[1] Bao, S. Nie, K. Xue, Cao, Li, H. Su, and J. Zhu, All are Worth Words: A ViT Backbone for Diffusion Models. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pages 22669-22679, IEEE, 2023.
[2] O. Bal, L. Yariv, Y. Lipman, and T. Dekel. MultiDiffusion: Fusing Diffusson Paths for Controlled Image Generation. In Proceedings of the International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 1737-1752. PMLR, 2023.
[3] J. Betker, G. Goh, L. Jing, T. Brooks, J. Wang, L. Li, L. Ouyang, J. Zhuang, J. Lee, Y. Guo, al. Improving image generation with better captions, 2023.
[4] Black Forest Labs. Flux, https::/github... com www.claccbbbc..ooo..aaaaaaaacca. foreft--tttoo---- labesst-abb/flus 2024.
[5] A. Chen, J. Xu, W. Zheng, G. Dai, Y. Wang, R. Zhang, H. Wang, and S. Zhang. Training-free Regional Prompting for Diffusion Transformers, 2024. UKK.attp::/arxiiv.or///// 02395
[6] J. Chen, J. Yu, C. Ge, I.. Yao, E. Xie, Z. Wang, J. T. Kwok, P. Luo, H. Lu, and 7. Li. PixArt- a: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesis. In Proceedings of the International Conference Learning Representations, OpenReview.net, 2024.
[7] M. Chen, I. Laina, and A. Vedaldi. Training- Layout Control with Cross-Attention Guidance. In Winter Conference on Applications of Computer Vision, pages 5331-5341. IEEE, 2024.
[8] Z. Chen, Li, H. Wang. Z. Chen, Z. Jiang, J. Li, Q. Wang, J. Yang, and Y. Tai. Region- Aware Text-to- Text-to-Image Generation via Hard Binding and Soft Refinement, 2024. URL https
//arxiv. org/ hts///b///// /a////bss/////////////// 2411. 06558
[9] B. Cheng, Y. Ma, L. Wu, S. Liu, A. Ma, X. Wu, D. Leng, and Y. Yin. HiCo: Hierarchical Con- trollable Diffusion Model for Layout-to-imaa Generation. In Advances Neural Information Processing Systems, 2024.
[10] Cheng, X. Liang, X. Shi, T. He, T. Xiao, and M. Li. LayoutDiffuus: Adapting Foundational Diffusion Models for Layout-to-Image Generation, 2023. httpsstss://artt org/ /bs/ 7302.08908
[11] Cheng, Zhao, T. He, Xiao, Zhang, and Zhou. Rethinking The Training And Evaluation of Coch-Context Layout-to-Image Generation. In Advances in Neural Information Processing Systems, 2024.
[12] G. Couairon, M. Careil, M. Cord, S. Lathuilière, and J. Verbeek. Zero-shot spatial layout conditioning for text-to-image diffusion models. In Proceedings the IEEE/CVF International Conference on Computer Vision, pages 2174-2183. IEEE, 2023.
[13] O. Dahary, O. Patashnik, K. Aberman, and D. Cohen-Or. Be Yourself: Bounded Attention for Multi-subject Text-to-Image Generation. Proceedings of the European Conference on Computer Vision, volume 15072 of Lecture Notes in Computer Science, pages 432-448 Springer, 2024.
[14] dewei Zhou, J. Xie, Z. Yang, and Y. Yang. 3DIS: Depth-Driven Decoupled Image Synthesis for Universal Multi-Instance Generation. In Proceedings of the International Conference on Learning Representations, 2025. URL https: h hwtp//veeeeeeeeeeeeeeeee aoss//forumm"ddddMaawooddd
11

---

[15] P. Esser, S. Kulal, A. Blattmann, R. Entezari, J. Müller, H. Saini, Y. Levi, D. Lorenz. A. Sauer, Boesel, D. Podell, T. Dockhorn, Z. English, and R. Rombach. Scaling Rectified Flow Trans- formers for High-Resolutio Image Synthesis. In Proceedings of the International Conference on Machine Learning. OpenKeview.net, 2024.
[16] Y. Feng, B. Gong, D. Chen, Y. Shen, Y. Liu, and J. Zhou. Ranni: Taming Text-to-lmage Diffusion for Accurate Instruction Following. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pages 4744-4753 IEEE, 2024.
[17] P. Gao, L. Zhuo, D. Liu, R. Du, X. Luo, L. Qiu, Y. Zhang, C. Lin, R. Huang, S. Geng, R. Zhang, J. Xi, W. Shao, Z. Jiang, T. Yang, W. Ye, H. Tong, J. He, Y. Qiao, and H. Li. Lumina-T2X. Transforming Text into Any Modality, Resolution, and Duration via Flow-based Large Diffusion Transformers, 2024. httpttttp///rrtt
org/ at//a/////// 2005/ 059455.
[18] B. Gong, S. Huang, Y. Feng, S. Zhang, Y. Li, and Y. Liu. Check, Locate, Rectify: A Training- Free Layout Calibration System for Text- to- Image Generation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 6624 6634. IEEE, 2024.
[19] R. He, B. Cheng, Y. Ma, Q. Jia, S. Liu, A. Ma, X. Wu, L. Wu, D. Leng, and Y. Yin. PlanGen: Towards Unified Layout Planning and Image Generation in Auto- Reghe-sseeesvvv Vision Language Models, 2025. bttps: arxiv org/ /
[20] Y. He, R. Salakhutdinov, and J. Z. Kolter Localized text-to- generation for free via cross attention control, 2023. URL httththts:/// http:/ttts//aaxi/ii //ap://///xxxi org/ 2306. 14636
[21] J. Hessel, A. Holtzman, M. Forbes, R. L. Bras, and Y. Choi. CLIPScore: A Reference-fre Evaluation Metrie for Image Captioning. In Proceedings of the Empirical Methods in Natural Language Processing, pages 7514-7528. Association for Computational I Linguistics, 2021.
[22] J. Ho, A. Jain, and P. Abbeel. Denoising Diffusion Probabilistic Models. In NeurlPS, 2020.
[23] E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, I.. Wang, and W. Chen. LoRA: Low- Rank Adaptation of Large Language Models. In Proceedings of the International Conference on Learning Representations. OpenReview.net, 2022.
[24] X. Huang, Y.-. Huang, Y. Zhang, W. Tian, R. Feng, Y. Zhang, Y. Xie, Y. Li, and L. Zhang. Open-Set Image Tagging with Multi-Grained Text Supervision, 2023. URL https: //arxiv org/abs/222 2210. 15200
[25] C. Jia, M. Luo, Z. Dang, G. Dai. X. Chang, M. Wang, and J. Wang. SSMG: Spatial-Semantic Map Guided Diffusion Model for Free-Form Layout-to-Image Generation. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 2480-2488. AAAI Press, 2024.
[26] Y. Kirstain, A. Polyak, U. Singer, S. Matiana, J. Penna, and O. Levy. Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generation. In Advances in Neural Information Processing Systems, 2023.
[27] Y. Lee, T. Yoon, and M. Sung. GrounDiT: Grounding Diffusion Transformers via Noisy Patch Transplantation. In Advances in Neural Information Processing Systems, 2024.
[28] Y. Li, H. Liu, Q. Wu, F. Mu. Yang, J. Gao, C. Li, and J. Lee. GLIGEN: Open-Set Grounded Text-to-Image Generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22511-22521, IEEE, 2023.
[29] Y. Li, M. Keuper, D. Zhang, and A. Khoreva. Adversarial Supervision Makes Layout-to- Image Diffusion Models Thrive. In Proceedings of the International Conference on Learning Representations. OpenReview.net, 2024.
[30] Z. Li, J. Zhang, Q. Lin, Xiong, Y. Long, X. Deng, Y. Zhang, X. Liu, M. Huang, Z. Xiao, D. Chen, J. He, J. Li, W. Li, C. Zhang, R. Quan, J. Lu, J. Huang, X. Yuan, X. Zheng, Y. Li, J. Zhang, C. Zhang, M. Chen, Liu, Z. Fang, W. Wang, Xue, Y. Tao, J. Zhu, K. Liu, S. Lin, Y. Sun, Y. Li, D. Wang, M. Chen, Z. Hu, X. Xiao, Y. Chen, Y. Liu, W. Liu, D. Wang, Y. Yang, J. Jiang, and Q. Lu. Hunyuan-DiT: A Powerful Multi-Resolltion Diffusion Transformer with GinenGrained Chinese Understanding, 2024. https::s:// /ttp/v//iii//..////v//ea////aavv v. org/
12

---

[31] T. Lin, M. Maire, J. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick. Microsoft COCO: Common Objects in Context. Proceedings of the European Conference on Computer Vision, volume 8693 of Lecture Notes in Computer Science, pages 740-755. Springer, 2014.
[32] 7. Lin, D. Pathak, B. J. X. Xia, G. Neubig, P. Zhang, and D. Ramanan. Evaluating Text-to- Visual Generation with Image-to-Texx Generation. Proceedings of the European Conference on Computer Vision, volume 15067 of Lecture Notes in Computer Science, pages 366-384, Springer, 2024.
[33] Y. Lipman, R. T. Q. Chen, H. Ben-Hamu, M. Nickel, and M. Le. Flow matching for genera- tive modeling. In Proceedings of the International Conference on Learning Representations. OpenReview.net, 2023.
[34] B. Liu, Akhgari, A. Visheratin, Kamko, L. Xu, S. Shrirao, C. Lambert, J. Souza, S. Doshi, and D. Li. Playground v3: Improving Text-to-Iman Image Alignment with Deep-Fusion Large Language Models, 2024. URL (tRps:/s:://axx/ org/ abaaaaas///// 22// 10695
[35] S. Liu, Z. Zeng. T. Ren, F. Li, H. Zhang. J. Yang, Q. Jiang, C. J. J. Yang, H. Su, J. Zhu, and L. Zhang. Grounding DINO: Marrying DINO with Grounded Pre-training for Open-Set Object Detection. In Proceedings of the European Conference Computer Vision, volume 15 of Lecture Notes in Computer Science, pages 38-55. Springer, 2024.
[36] Z. Lv, Y. Wei, W. Zuo, and K. K. Wong. PLACE: Adaptive Layout-Semantic Fusion for Semantic Image Synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9264-9274, IEEE, 2024.
[37] M. Ohanyan, H. Manukyan, Z. Wang. S. Navasardyan, and H. Shi. Zero-Painter: Training-Free Layout Control for Text-to-lmage Synthesis. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pages 8764-8774. IEEE, 2024.
[38] openfree.
flux-chatgpt-hhibil-oraa flux- wwwttpttghibliiiiti. -hihliiiiblrrrr 2025.
[39] PatrickStamrr:
FLUX - Oil painting.
chatgpt-4o www.eeeeeeeeemeeeeeeeeeeeeeeeeee
2024.
[40] W. Peebles and S. Xie. Scalable Diffusion Models with Transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4172-4182, IEEE, 2023.
[41] Q Phung, S. Ge, and J. Huang. Grounded Text-to-lmage Synthesis with Attention Refocusing. Proceedings of the IEEE/CVE Conference on Computer Vision and Puttern Recognition, pages 7932-7942. IEEE, 2024.
[42] D. Podell, Z. English, K. Lacey, A. Blattmann, T. Dockhomn, Müller, J. Penna, and R. Rom- bach. SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis. In Proceedings of the International Conference on Learning Representations. OpenReview.net, 2024.
[43] Qwen Team. Qwen2.5-VL., January 2025. URL itstt:::/ggeellgggiiuuu....o/. hwaimmmmm.ittuuiiiie quee2 5-v1/]
[44] A. Ramesh, P. Dhariwal, A. Nichol, C. Chu, and M. Chen. Hierarchical Text-Conditional Image Generation with CLIP Latents, 2022. https://axxx.. .rgg abs/ ht..b//224001122.. 06125
[45] R. Rombach, A. Blattmano, D. Lorenz, P. Esser, and B. Ommer. High-Resolution Image Synthesis with Latent Diffusion Models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10674-10685. IEEF, 2022.
[46] O. Ronneberger, P. Fischer, and T. Brox. U-Net: Convolutional Networks for Biomedical Image Segmentation. In Medical Image Computing and Computer-Assisted Intervention, volume 9351 of Lecture Notes in Computer Science, pages 234-241. Springer, 2015.
13

---

[47] C. Saharia, W. Chan, S. Saxena, L. Li, J. Whang, E. L. Denton, S. K. S. Ghasemipour, R. G. Lopes, B. K. Ayan, T. Salimans, J. Ho, D. J. Fleet, and M. Norouzi. Photorealistic text-to-Image Diffusion Models with Deep Language Understanding. In Advances in Neural Information Processing Systems, 2022.
[48] T. Shirakawa and S. Uchida. NoiseCollage: A Layout-Aware Text-to-lmage Diffusion Model Based on Noise Cropping and Merging. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 8921-8930. IEEE, 2024.
[49]
stability.ai.
Stable Diffusion 3.5.
https://stability aww.ttt...///wws///..m
Nov 2024.
[50] A. Taghipour, M. Ghahremani, M. Bennamoun, A. M. Rekavandi, H. Laga, and F. Boussaid. Box It to Bind It: Unified Layout Control and Attribute Binding in T21 Diffusion Models, 2024. URL https: //arxiss://xxv..../// org/ abs/ 17910
[51] M. Tancik, P. P. Srinivasan, B. Mildenhall, S. Fridovich-Keil, N. Raghavan, U. Singhal, R. Ra- mamoorthi, T. Barron, and R. Ng. Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains. In Advances in Neural Information Processing Systems, 2020.
[52] vjleoliu. ChatGPPT-40 Renderer. https: htaiiaaa..com/modees//445555111 chatgnt-4o-renderes wwwrrrddderrrmmoeeeeerrrrraeeeee
2025.
[53] X. Wang, T. Darrell, S. S. Rambhatla, R. Girdhar, and I. Misra. InstanceDifffusion: Instance- Level Control for Image Generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6232-6242. IEEE, 2024.
[54] Y. Wu, X. Zhou, B. Ma, X. Su, Ma, and X. Wang. IFAdapter: Instance Feature Control for org/
Grounded Text-to-lmage Generation, 2024.
002440
[55] Xie, Y. Li, Y. Huang, Liu, W. Zhang, Y. Zheng, and M. Z. Shou. BoxDiff: Text-to-Image Synthesis with Training-Free Free Box-Constrainet Diffusion. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7418-7427. IEEE, 2023.
[56] J. Xu, X. Sun, Z. Zhang, G. Zhao, and J. Lin. Understanding and Improving Layer Normalization. In Advances in Neural Information Processing Systems, pages 4383 4393,, 2019.
[57] H. Xue, Z. Huang, Q. Sun, L. Song. and W. Zhang. Freestyle ayyoutto-Immage Synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14256-14266. IEEE, 2023.
[58] B. Yang, Y. Luo, Z. Chen, G. Wang, Liang, and L. Lin. LAW-Diffusio:: Complex Scene Gen- eration by Diffusion with Layouts. In Proceedings of the IEEE/CVF International Conference on Computer Vision pages 22612-22622. IEEE, 2023.
[59] Z. Yang, J. Wang, Z. Gan, L. Li, K. Lin, C. Wu, N. Duan, Z. Liu, C. Liu, M. Zeng, and Controlled Text-to-Image Generation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 14246-14255. IEEE, 2023.
L. Wang. ReCo:
[60] Y. Yao, T. Yu, A. Zhang, C. Wang, J. Cui, H. Zhu, T. Cai, H. Li, W. Zhao, Z. He, Q. Chen, H. Zhou, Z. Zou, H. Zhang, S. Hu, Z. Zheng, J. Zhou, J. Cai, X. Han, G. Zeng, D. Li, Z. Liu, and M. Sun. MiniCPM-V: GPT-4V Level MILM on Your Phone, 2024. URL
https://arxir org/ abs/ 01800
[61] H. Zhang, D. Hong, Y. Wang, Shao, X. Wu, Z. Wu, and Y.-G. Jiang. CreatiLayout: Siamese Multimodal Diffusion Transformer for Creative Layout-to-Imaae Generation, 2025. URL https:
org/ abtp///2...00 03859
[62] I.. Zhang, A. Rao, and M. Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEF/VV International Conference on Computer Vision, pages 2813-3824. IEEE, 2023.
14

---

[63] G. Zheng, X. Zhou, X. Li, Z. Qi, Y. Shan, and X. Li. LayoutDiffusion: Controllable Diffusion Model for Layout-to-Image Generation. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 22490-22499. IEFE, 2023.
[64] D. Zhou, Y. Li, F. Ma, X. Zhang, and Y. Yang. MIGC: Multi-Instance Generation Controller for Text-to-lmage Synthesis. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6818-6888 18-6822. IEEE, 2024.
[65] D. Zhou, Y. Li, F. Ma, Z. Yang, and Y. Yang. MIGC++: Advanced Multi-Instance Gener- ation Controller for Image Synthesis. IEEE Transactions on Pattern Analysis and Machine Intelligence, 1714:1714--728. 2025.
[66] D. Zhou, J. Xie, Z. Yang, and Y. Yang. 3DIS-FLUX: simple and efficient multi-instance generation with DiT rendering, 2025. URL https::/arxxi...... org/ abs/2501.0512
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We introduce our InstanceAssemble method in the abstract and introduction, provide a detailed description in Section and present experimental results Section
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of this work in Section 5
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
16

---

Justification:
This paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The architecture of our module is introduced Section and the implementa- tion details are presented at the beginning of Section 4
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
17

---

Answer:
[No]
Justification:
We will release our code upon acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (httpss:/niis.cc.
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification:
The implementation details are presented the beginning of Section
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Although we report multiple evaluation metrics (e.g., mIoU, VQA, CLIP) showing significant improvements over baselines, we did not include error bars, standard deviations, confidence intervals, or statistical significance tests across multiple runs because it would be too computationally expensive. As such, our results do not fully meet the criteria outlined in this question for statistical rigor and uncertainty quantification.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
18

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma enror bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We present the computer resources needed for the experiments at the begining of Section
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics htttp:://eeuriii wwwpuuriiss.ccppuuuupe httpp//clcccccccccccccaaaaaaaaaa
Answer: [Yes]
Justification: We conducted the research in the paper conform, in every respect, with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discussed the broader impacts in the last part of the article.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
19

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [Yes]
Justification: We use the existing excellent SD3, Flux. I and their safeguards. Our model will not bring more risk than SD3 or Flux.I.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We work with open-source models that are publicly available, and we cited them properly.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
20

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/dataseta
latasets.papersstt
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Question:
Answer: [Yes]
Justification: The dataset defined for our benchmark will be made publicly available, in case acceptance, together with the documentation required for reproducing the experiments. Moreover, in case of acceptance, we will also release the source code of our model.
Guidelines:
The answer NA means that the paper does release assets.
Researchers should communicate the details the dataset/code/mooell as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification:
This paper does involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
21

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does not involve LLMS as any important, original, non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy (https://neurips: hett:s///eerrrps.ccc//ooeeereeee cc///onfereeeeee/222////... for what should or should not be described.
22

---

Supplementary Material
A Early-Stage Layout Control in Assemble-MMDiT
We apply the Bheemble-MMDDD layout control module exclusively during the initial 30% of the denoising trajectory and deactivate for the remaining 70%. This two-stage strategy provides robust low-frequency structural guidance in the early stages to facilitate layout alignment, while allowing subsequent unconstrained refinement of high-frequency details during later denoising phases.
Ablation Control Ratio
-x- Time (5)
0.9
10
0.9
0.7
0.6
Time
0.5
0.4
mloU (LG55]
Texture (LGS)
Shape (LGS)
VQA
0.2
Control Ratio
Figure 9: Impact of the proportion of diffusion steps incorporating layout conditioning generation quality.
As illustrated in Figure 9 restricting layout control to less than 30% of the diffusion process results in insufficient layout alignment with the target bounding boxes. contrast, extending control beyond this optimal threshold leads to decline in output quality. Furthermore, increasing the proportion of guydut-guided steps results in significant additional computational cost.
B
Additional Ablation Studies
B.1
Effect of Bbox Encoding and DenseSample
To clarify the role of bounding box encoding and DenseSample, we further ablated the SD3-M based Assemlle model on DenseLayout. Bounding box embeddings guide correct object placement, while DenseSample provides additional improvements in spatial accuracy and instance- level semantics. The results in Table demonstrate that both components contribute to the overall performance.
Table 6: Ablation on bounding box encoding and DenseSample.
Sctting
mloUt
coloct
texturet
shapet
VQAt
bbox encoding. w/o SemplS $1.22
32.15
33.53
w/ bbux encoding, wlo DenseSample bhox encoding, DenseSample
$1.28
32.68
34.94
34.58
93.33
$2.07
33.77
B.2 Comparison with Attention Mask-based Region Injection
We also compare our Assemble-Attn design with attention mask-based region injection. While both can be viewed region-wise attention mechanisms, attention masks operate globally and may cause
23

---

semantic leakage in overlapping regions. Our method instead applies instance-wise self-attention on cropped latent regions and then fuses the updated features via the Assemble step, which is more effective in dense layouts. As shown in Table 7 our design achieves superior instance attribute consistency and higher VQA score compared to the attention mask baseline.
Table 7: Comparison between attention mask- base injection and our Assemble
Method
spatialf
colort
texture
shapet
VQMT
SD3 Medium modell)
60.28
Attention mask (SD3-M)
94.11
74,28
77.58
76.54
91.53
InstanceAssemblee (ours, SD3-M)
94.97
77.53
80.72
80.11
93.12
C Underlying data for radar-chart visualizations
In Sections 4.3 we utilized radar chart to depict each quantitative variable along equi-angular axes, providing an intuitive comparison. This visualization highlights the multifaceted superiority of our method. Here, present the corresponding raw evaluation results in tabular form. Specifically, Tab. [ corresponds to Fig. thus ensuring a clear mapping between each radar-chart subfigure and its underlying data.
Table 8: Quantitative results of additional visual content on DenseLayout.
Layour Grounding Score
Global Quality VQAT Pickt CLIPT
DenseLayout
mloU? colort texturef shupet
Real Imaper(Upper Bound)
92.35
76.52
80.78
79.78
text
43.72
26.57
28.56
28.39
93.37
21.63
12.45
text+image
55.29
42.15
44.50
44.24
91.66
22.05
12.95
text+depth
49.64
28.25
31,82
31.62
92.83
21.28
13.25
texitedge
50.73
29.45
33.92
33.84
90.13
21.26
13.55
D
More Details on DenseLayout Evaluation Dataset
D.1
Construction Pipeline DenseLayout Dataset
The DenseLayout dataset is constructed through multi-stage pipeline designed to extract high- density and semantically-rice layout information from synthetic images. The pipeline includes following steps:
1. Image Generation using Flux.1-Dev 4
A diverse set of synthetic images generated using Flux. 1-Dev, text-to-image model. The input prompts are generic textual descriptions, sampled from the LayoutS dataset, which is based on SA-IB. The images are resized maintain the same aspect ratio as the original SA-1B images, with the longer edge set to 1024 pixels. This step provides a visually complex base for extracting layout structures.
2. Multi-label Tagging using RAM++ 24
The generated images are tagged using RAM++, the next-generation model of RAM, which supports open-set recognition. These tags offer high-level semantic guidance for subsequent grounding.
3. Object Detection via GroundingDINO 35
Using the image and its predicted tags as input, GroundingDINO performs open-set object detection. It outputs bounding boxes and class labels for all detected entities. The detection is configured with box_ box_bbreeeeeeeeeeeee threshold of 0.35 and a tex__threshere bounding box is treated as instance bounding box. and the corresponding predicted label is recorded as the instance description.
of 0.25. Each detected
4. Detailed Captioning with Qwen2.5-VI. 43
Each bounding box region is cropped from the original image and fed into Qwen2.5-Vi 5L
24

---

to generate a fine- grained caption. These region-level captions are stored as the detailed description for each instance, enriching the semantic information beyond category labels.
5. Density Filtering
To ensure high layout complexity, only images with 15 or more detected instances (as output by GroundingDiNO) retained. This results in dense layout distribution suitable for layout-conditioned generation tasks. The distribution of instance count is shown in Fig. 10
Finally, the LenseLaayot dataset contains 5,000 images and 90,339 instances, with an average of 18.1 instances per image.
DenseLayout: Instance
Number Inscanceia per tma
Figure 10: Instance count distribution per image DenseLayout.
Annotation Format. The annotation for each image consists of:
global_caption: the original prompt used for image generation. image_infe info metadata of the image, including height and width.
instance_info: list of instances, each with:
bbox: the bounding box of the instance, formatting as [æ1, V2,32,320 32). description: the category label predicted by GroundingDINO. detail_descrile description: fine-grained caption generated by Qwen2.5-VL for the cropped region.
D.2 Samples of DenseLayout Dataset
"instanca_info".
"bbox": 289.2)33,)).
'description"' "nightatand".
[astall_dwserigee dexeription *The nighsatand is dark brown, "detail
conpact, with drazer."
"abox"; [306,170, .30,,430,3391
ddserrppiion "picture frasa".
frane containing vatercolor painting of green leaves on white background."
description"
"bbax": [803, ,175, 740.))
"description": "picture frame".
detatl dentrip. simple brovn wooden frane holds botanical print
stone
,,
Figure 11: A sample of DenseLayout and its annotation.
25

---

with not the romils pnd mose butte
with caadia or woaring
envosting intrrment car tho ragerny wite poleecteen walliing cheee
wiwe oete now ootaedt wour ritess sigh, onerinting
Niu ENOOA rasl The fow Fattittd
vdla Bil who' pou
this Larking aitcnt
baociee Necla Clyck
Falige whili tor
Dhe
ThE detoo shomasing
1000
vacler coend Netlitayy ceriig treggel sha enarge ooe
actatin of plriteet and style aeenoael battings
vilil. clank doy
andlag atoccder like
Figure 12: Representative samples from DenseLayout dataset demonstrating: (a) High-density scene with > 15 instances, (b) Complex instance relationships with precise attribute specifications.
26

---

E More results with textual-only content
E.1
InstanceAssemble based on SD3-Medium
Figure 13: More results InstanceAssemble based on 3D3-Medium with textual -nll content.
E.2
InstanceAssemble based on F-Dx.
Figure 14: More results of InstanceAssemble based on Flux. 1-Dev with textual-only content.
E.3 InstanceAssemble based on Flux.1-Schma
Figure 15: More results of InstanceAssemble based Flux.1 1 Schhnell with textual-only content.
27

---

More results with additional visual content
E1 Additional Image
Figure 16: More results with additional image.
F2
Additional Depth
Figure 17: More results with additional depth.
F33
Additional Edge
Figure 18: More results with additional edge.
28


Reviews:
[{"review_id": "WoCpHIQFOL", "reviewer": "Reviewer_PzZh", "summary": "In this paper, the authors proposed a new model for layout-aware image generation. In particular, they design LoRA modules that can be attached to modern DiT based text-to-image generative models such as Stable Diffusion 3 and Flux.1. They also proposed a new benchmark and a new evaluation metric for the layout-aware image generation task.", "questions": "1. During inference the layout condition is only applied in the first 30% of the denoising steps, then is it still essential to train with all time steps? What is going to happen if you apply the layout condition for all time steps at inference time? Will the image quality degrade significantly?\n2. Related to weakness 1, what exactly is the “image” input to the networks and what is the output of the network exactly? Are the outputs epsilon predictions, x0 predictions or v predictions? Are the “image” input the actual denoised latent, noisy latent, or epsilon?\n3. Can the authors explain the oversaturation problem?\n4. Which training dataset did the authors use for the image/depth/edge input?\n5. Does the Assemble-MMDiT have a similar size as the frozen MMDiT? If so, is the reason why it only adds 8% more inference time the fact that you only apply it in the first 30% of the denoising steps?\n6. How would the native multimodal image generation models like GPT-4o or Gemini 2.0 Flash compare to InstanceAssemble in this task?\n7. Does this model only work for bounding boxes? The authors should consider adding (demo) experiments with semantic maps if possible.", "limitations": "Yes, but the authors can potentially extend the paragraph to include more discussion on potential negative societal impact.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strength:**\n1. The proposed model has very impressive empirical results\n2. The experiment section is solid and thorough\n3. The addition of the new benchmark and metric is a good plus\n4. All figures in this paper are very high quality\n\n**Weakness:**\n1. The modeling details are a bit unclear from the writing. For example, what exactly is the network’s “image” input and output? I.e. which parametrization did the authors consider? Or is it different based on different backbone models?\n2. Related to weakness 1, if the training objective in this paper is the same as the backbone models, then the description of the training objective is not very accurate. Stable Diffusion 3 and Flux.1-dev both use the flow matching objective and it should not be described as the same as the one in LDM (i.e. velocity prediction with conditional optimal transport path is not the same as epsilon prediction with Gaussian probability path).\n3. The generated images from InstanceAssemble look very saturated, and with depth and edge inputs this problem is exacerbated.\n4. The authors should consider including the discussions of the following related works:\n\n        [1] Zhang, et al. “Adding Conditional Control to Text-to-Image Diffusion Models”. ICCV 2023.\n\n        [2] Hertz, et al. “Prompt-to-Prompt Image Editing with Cross-Attention Control”. arXiv 2022.\n\n        [3] He, et al. “Localized text-to-image generation for free via cross attention control”. arXiv 2023.", "was_revised": true, "final_justification": "The authors have addressed all my concerns and therefore I would like to maintain my rating as 5 accept.", "rebuttal": "We sincerely thank the reviewer for your positive feedback and high evaluation of our work. We greatly appreciate your recognition of our contributions, and your thoughtful comments have further enhanced the clarity and quality of our paper. Below, we provide point-by-point responses to each concern.\n\n> **Weakness 1:** The modeling details are a bit unclear from the writing. ...\n\nIn Section 4.1 (\"Experimental Setup\"), we explored two input variants:\n\n- **Prompt + layout:** The inputs consist of a global prompt describing the overall content and layout condition consist of paired bounding boxes and instance captions. No additional visual content is provided.\n\n- **+ additional visual instance content:** This extends the previous variant by optionally incorporating per-instance reference images, edge maps, depth maps, etc.\n\nFor the layout encoder, we train a **MLP from scratch** to encode layout features. In the **Assemble-MMDiT module**, we fine-tune through **LoRA** on top of the frozen backbone. Overall, compared with the SD3 or FLUX.1-dev backbones, we add a layout encoder trained from scratch and the Assemble-MMDiT module as a LoRA adaptation.\n\n> **Weakness 2:** Related to weakness 1, if the training objective in this paper is the same as the backbone models, ...\n\nWe sincerely thank the reviewer for pointing out this issue. We apologize for the imprecise description of the training objective. To clarify, we use the **flow-matching loss**, consistent with Stable Diffusion 3 and Flux 1-dev, we will correct this in the revised version to ensure accuracy.\n\n> **Weakness 3:** The generated images from InstanceAssemble look very saturated, ...\n\nWe trained two variants of the model:\n\n- The **prompt + layout** variant shows negligible oversaturation.\n\n- The **additional visual instance content** variant, however, results in slight oversaturation, especially when conditioning on depth or edge maps. This is likely due to the different data distributions in these modalities, which exceed the representational capacity of a single LoRA.\n\nWe acknowledge this limitation and plan to address it in future work by (a) training separate LoRA models for each modality and (b) exploring normalization or modality-specific injection strategies to mitigate oversaturation.\n\n> **W4:** The authors should consider including the discussions of the following related works:...\n\nWe sincerely appreciate the reviewer’s recommendation to include these important related works, all of which contribute valuable insights. We will incorporate discussions of the papers in our final revision.\n\n---\n\n> **Q1:** During inference the layout condition is only applied in the first 30% of the denoising steps, ...\n\nTraining across all timesteps **is crucial** for generalization, as the timestep conditions the denoising network. This ensures that the model dynamically adapts its predictions to the specific noise level at each diffusion step, which is fundamental to the noise estimation process. During the inference, extending beyond 30% offers no significant gains but increases runtime overhead. Therefore, we choose the 30% trade-off to balance quality and efficiency as shown in Supplementary A and Figure 10.\n\n> **Q2:** Related to weakness 1, ...\n\nSince our backbone uses the flow‑matching formulation (as in SD3 and Flux), our network outputs v‑predictions. During training, inputs are noised latents from the VAE encoder of real images; at inference, inputs are Gaussian-initialized noisy latents.\n\n> **Q3:** Can the authors explain the oversaturation problem?\n\nWe have addressed the oversaturation issue in response to **Weakness 3**.\n\n> **Q4:** Which training dataset did the authors use for the image/depth/edge input?\n\nWe use the **LayoutSAM** dataset (proposed by CreatiLayout[1]) for training. For additional visual content, we crop the original images and compute depth and Canny edge maps. During training, we randomly select one modality (image, depth, or edge) or none for each instance. We will clarify these details in the revised version.\n\n> **Q5:** Does the Assemble-MMDiT have a similar size as the frozen MMDiT? ...\n\nThe **Assemble-MMDiT** has the same block architecture as the original MMDiT backbone, with a **LoRA adapters** (not the full parameters). The approximately 8% additional inference time (reported on the Flux model) is due to applying the LoRA adaptation only during the first 30% of the denoising steps and only on 8 blocks (seven double-blocks and one single-block). We will clarify these details in the revised version.\n\n> **Q6:** How would the native multimodal image generation models like GPT-4o or Gemini 2.0 Flash compare to InstanceAssemble in this task?\n\nWe explored layout control in GPT-4o by embedding bounding box positions into the text prompt or by providing a reference image with bounding boxes. While GPT-4o demonstrates some basic layout awareness, the positional accuracy is not satisfactory. We attribute this limitation to its lack of dedicated layout conditioning mechanisms.\n\n> **Q7:** Does this model only work for bounding boxes? ...\n\nWe appreciate the suggestion to explore semantic maps. To demonstrate this, we conducted experiments with our model originally trained with bounding boxes. We modified the model at inference time to crop based on semantic maps instead of bounding boxes. Results show that semantic-map-based inference aligns instance shapes more closely with segmentation. However, dedicated training with semantic maps is still necessary to achieve robust results. This is a promising direction for future work.\n\n---\n\n> **Limitations**: Yes, but the authors can potentially extend the paragraph to include more discussion on potential negative societal impact.\n\nWe agree that a more comprehensive discussion of potential negative impacts is warranted. We will expand the *Limitations* section to address possible risks such as misuse in generating misleading or deceptive layouts, privacy concerns, and the potential for bias amplification in conditioned generation.\n\nWe sincerely thank the reviewer again for the insightful feedback, which we will incorporate into the final revision.\n\n[1] H. Zhang, D. Hong, Y. Wang, J. Shao, X. Wu, Z. Wu, and Y.-G. Jiang. CreatiLayout: Siamese Multimodal Diffusion Transformer for Creative Layout-to-Image Generation, 2025."}, {"review_id": "H4R0R8KRbb", "reviewer": "Reviewer_FDqA", "summary": "The paper proposes a new method for layout-to-image (L2I) generation based on a new Instance Assembling Attention mechanism. The proposed mechanism processes layout tokens and the matching image tokens with an additional attention block, and subsequently assembles them back into image and layout tokens. In addition, the paper also introduces a new benchmark for the L2I task based on spatial accuracy, determined by DetectIoU, and semantic accuracy, which is calculated using a VLM-based VQA protocol. Overall, the proposed method outperforms related approaches while being significantly more efficient.", "questions": "- Which weights are trained using LoRA? How are these weights initialized? Why would LoRA improve the performance? What is the \"base model\" referred to in the ablation study section?\n- In Figure 2, it looks like there is one large attention block that processes all projected tokens simultaneously, but in the text, it sounds like every instance is processed separately, i.e., image tokens do not attend across instances. Which one is it?\n- Why is training performed for all time steps, but during inference, the proposed method is only applied to the first 30% of timesteps?\n- Why are the LayoutGrounding scores for DenseLayout so much lower than for LayoutSAM-Eval?", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strengths\n- **Straightforward but effective idea**\nThe authors leverage the attention mechanisms in MMDiT to guide the generation process to place objects accordingly. The integration of layout tokens in a cascaded way through Assmeble-MMDiT is straightforward and effective.\n\n- **Thorough and convincing ablation study**\nThe authors perform a thorough ablation study that considers multiple design choices, making a compelling case for the proposed pipeline. Specifically, the A\n\n- **Good results and great improvements in efficiency**\nBoth the quantitative and qualitative results show tremendous improvements in efficiency while also improving on most quality metrics. Table 3, in particular, demonstrates the impressive efficiency improvement clearly.\n\n### Weaknesses\n- **Bounding boxes are rectangular**\nDue to the rectangular nature of bounding boxes, fine-grained controllability is limited. While having a rough way to control the generation process might be desired in some cases, greater control might be desirable in other settings. Nonetheless, the proposed framework seems to be a good option for bounding box level of control.\n\n- **Unclear role of LoRA**\nThe authors mention the use of LoRA but do not specify for which weights LoRA is applied. See below for more specific questions regarding LoRA.\n\n- **Extensive additional training**\nThe proposed approach requires training on 8 H800 GPUs for a week. So it seems that the method moves the required computational effort from the inference step to the training step, allowing for the added efficiency. While this is not necessarily a weakness, it might be important to consider depending on the available computational budget.\n\n- **Scales with the number of instances, handling overlapping areas**\nIt looks like image tokens might be reused in Assemble-MMDiT if bounding boxes overlap, which could be inefficient. There seems to be an almost 2x increase in efficiency between sparse and dense layouts.", "was_revised": true, "final_justification": "Given that the rebuttal has addressed all my concerns convincingly, I have decided to keep my score and recommend acceptance.", "rebuttal": "We sincerely thank the reviewer for the positive recognition and thoughtful suggestions. Your encouraging assessment affirms our design choices and motivates future improvements. Below, we provide point-by-point responses to each concern.\n\n> **Weakness 1:** Bounding boxes are rectangular Due to the rectangular nature of bounding boxes, ...\n\nWe agree that bounding boxes provide only coarse spatial control. Therefore, we conducted experiments with more precise control condition like depth map, canny map, or reference image which showed better alignment with object shapes (Section 4.3). As for finer control, we will consider this an exciting direction for future work.\n\n> **Weakness 2:** Unclear role of LoRA The authors mention the use of LoRA but do not specify for which weights LoRA is applied. ...\n\nThank you for raising this point. We provide a detailed explanation of how LoRA is applied in **Q1** below.\n\n> **Weakness 3:** Extensive additional training The proposed approach requires training on 8 H800 GPUs for a week. ...\n\nThe main computational burden comes from our reliance on the base models (e.g., SD3, FLUX). Additionally, training on a dataset of over 200M+ images was essential for the model to learn effective layout control.\n\nWhile our approach does require notable training resources to achieve robust layout control, this upfront cost enables fast and flexible inference. Compared to related methods like CreatiLayout[1] (7 days on 8×A800 GPUs) or InstanceDiffusion[2] (64×A100 GPUs), our training cost is comparable.\n\n> **Weakness 4:** Scales with the number of instances, ...\n\nToken reuse in overlapping regions is a deliberate design choice to ensure that each instance’s semantics are properly injected. While this introduces some overhead in dense layouts, it preserves fidelity in layout-grounded generation. We acknowledge the trade-off and will explore optimizations in future work.\n\n---\n\n> **Q1:** Which weights are trained using LoRA? ...\n\nWe apply **LoRA to the Assemble-MMDiT module**, initialized from the base model (SD3-Medium or Flux.1-dev). The **Layout Encoder** is trained from scratch. In Table 4 :\n \"Assemble\" refers to the architecture design of Assemble-MMDiT, while \"LoRA\" refers to its training method:\n\n- The rows **without** \"LoRA\" indicate full fine-tuning of Assemble-MMDiT modules.\n\n- The rows **with** \"LoRA\" indicate fine-tuning using LoRA.\n\nThe **LoRA** module improves performance for two reasons: 1) it retains the base model's capabilities compared to the fully fine-tuned version, and 2) it enables effective layout control with fewer trainable parameters. The **base model** in the ablation study refers to the SD3-M model.\n\n> **Q2:** In Figure 2, it looks like there is one large attention block that processes all projected tokens simultaneously, ...\n\nThank you for pointing out the ambiguity in Figure 2. We apologize for this confusion. The reviewer’s understanding is correct: each instance is processed separately, and image tokens only attend to tokens within their corresponding bounding boxes. We will revise the figure to make this clearer.\n\n> **Q3:** Why is training performed for all time steps, ...\n\nTraining across all timesteps **is crucial** for generalization, as the timestep conditions the denoising network. This ensures that the model dynamically adapts its predictions to the specific noise level at each diffusion step, which is fundamental to the noise estimation process. However, during inference, layout guidance is most effective in the first 30% of timesteps, where the global spatial layout is established, as demonstrated in prior works like GLIGEN [3].\n\nAs shown in Supplementary A and Figure 10, extending beyond 30% offers no significant gains but increases runtime overhead. Therefore, we choose the 30% trade-off to balance quality and efficiency.\n\n> **Q4:** Why are the LayoutGrounding scores for DenseLayout so much lower than for LayoutSAM-Eval?\n\nThe difference in LayoutGrounding scores between DenseLayout and LayoutSAM-Eval is primarily due to the data complexity. DenseLayout contains an average of 18.1 instances per case (with at least 15 per image), making it much more challenging than LayoutSAM-Eval, which averages only 3.8 instances. As a result, DenseLayout sets a higher bar for layout adherence, leading to lower scores across all methods. However, our model maintains state-of-the-art performance on both benchmarks.\n\nOnce again, we greatly appreciate the reviewers’ insightful feedback. We hope these clarifications address your concerns, and we are committed to advancing layout-controlled synthesis by incorporating the suggested refinements.\n\n[1] H. Zhang, D. Hong, Y. Wang, J. Shao, X. Wu, Z. Wu, and Y.-G. Jiang. CreatiLayout: Siamese Multimodal Diffusion Transformer for Creative Layout-to-Image Generation, 2025.\n\n[2] X. Wang, T. Darrell, S. S. Rambhatla, R. Girdhar, and I. Misra. InstanceDiffusion: Instance-Level Control for Image Generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6232–6242. IEEE, 2024.\n\n[3] Y. Li, H. Liu, Q. Wu, F. Mu, J. Yang, J. Gao, C. Li, and Y. J. Lee. GLIGEN: Open-Set Grounded Text-to-Image Generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22511–22521. IEEE, 2023."}, {"review_id": "hr1XR2BYpy", "reviewer": "Reviewer_BQLG", "summary": "This paper introduces InstanceAssemble, a novel framework for accurate layout-based image generation. By using Assemble-MMDiT architecture, this method can robustly handle each instance, and by using LoRA, it is more training-efficient compared to previous training-based methods that require heavy fine-tuning. Moreover, the authors introduce a new benchmark and evaluation metric to enable more consistent assessment of the generated image with respect to the given layout. As a result, InstanceAssemble achieves state-of-the-art performance for complex layout conditions.", "questions": "Questions\n* For the quantitative comparisons in both Table 1 and Figure 5, only CreatiLayout is based on SD3-M and all other baselines use older text-to-image models. GIven that InstanceAssemble uses more recent models including SD3-M and Flux.1, I'm conerned whether the advantage of InstanceAssemble is coming from the novel method itself or from the better choice of the backbone model.\n\n* Moreover, I'm curious whether the previous layout-to-image methods such as GLIGEN [1] and InstanceDiffusion [2] are not applicable to MMDiT architectures, and whether there have been similar discussions on efficient ways to train them. Such discussions would provide a more clear motivation for the need of an efficient training framework like InstanceAssemble.\n\n[1] GLIGEN: Open-Set Grounded Text-to-Image Generation, CVPR 2023\n\n[2] InstanceDiffusion: Instance-level Control for Image Generation, CVPR 2024", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strength**\n\n* Effective engineering: The paper introduces a well-engineered framework, which seems to effectively address the limited robustness of of training-free methods and the computation  \n\n* Strong results: Clear improvements in the quantitative comparisons with the baseline methods (e.g. CreatiLayout [1], HICO [2])\n\n* New benchmark: The introduction of a new benchmark (DenseLayout) and an evaluation metric (LGS) are solid contributions in the literature of layout-to-image generation.\n\n**Weakness**\n\n* Unclear points on technical novelty: While InstanceAssemble shows strong results, there are some critical questions regarding the main idea of the proposed method, as listed below:\n\n   * **Details on Assemble-MMDiT:** I find it unclear about how Assemble-MMDiT is implemented. Are the attention modules trained from scratch, or are they initialized from the original MMDiT weights? As the full pipeline in Figure 2 also uses the original MMDiT for the global prompt, does it use the same weights and only activate the LoRA parameters during inference? Moreover, to which part of the MMDiT is the LoRA applied? Please correct me if the details are already included and I have missed it.\n\n   * **Rationale behind Assembling-Attn:** Section 3.2 doesn't seem to sufficiently explain why and how the design choice of Assembling-Attn, which is to apply attention between indivisual patches and the corresponding instance tokens, would lead to more accurate object locations. Specifically, is it the case that the attention in MMDiT already has such property and the LoRA training enlarges it? (Similar to the \"semantic sharing\" property of DiT observed in GrounDiT [3]) Or does the attention module newly learn to locate each object during the training phase of LoRA?\n\n   * **DenseSample**: The design of DenseSample in Section 3.1 is also confusing. It seems that the xy coordinates of each bounding box is encoded into each instance token. However, in Assembling-Attn only a cropped subset of image tokens are attended with each instance token. Given this, why is the xy coordinate information required in the instance token? Is this identical to the **grounding token** in GLIGEN [4]? \n\nAlthough InstanceAssemble seems to produce high-quality outputs, the paper's main contributions - Layout Encoder and Assemble-MMDiT - still contain multiple unclear points regarding its technical details and novelty compared to previous works.\n\n[1] CreatiLayout: Siamese Multimodal Diffusion Transformer for Creative Layout-to-Image Generation, 2024\n\n[2] HiCo: Hierarchical Controllable Diffusion Model for Layout-to-image Generation, NeurIPS 2024\n\n[3] GrounDiT: Grounding Diffusion Transformers via Noisy Patch Transplantation, NeurIPS 2024\n\n[4] GLIGEN: Open-Set Grounded Text-to-Image Generation, CVPR 2023", "was_revised": true, "final_justification": "The authors' rebuttal provides a clear clarification on most of the concerns raised regarding the main technical contribution and justification of the method design. Specifically, providing technical details on Assemble-MMDiT, and discussing the exact rationale behind Assembling-Attn were helpful for understanding the significance of the proposed method.", "rebuttal": "We sincerely thank the reviewer for the detailed and constructive feedback. Your questions have helped us clarify and refine several technical aspects of our work. Below, we provide detailed responses to the main concerns.\n\n> **Weakness 1:** **Details on Assemble-MMDiT:**...\n\nIn our final model:\n\n- **Initialization and Training:** Assemble-MMDiT is initialized from the pretrained MMDiT backbone weights, with **only the LoRA modules trained** during fine-tuning. The original MMDiT weights are **frozen**.\n\n- **Pipeline:**\n  \n  - The \"Global Prompt → Image\" step uses the unmodified, pretrained MMDiT backbone (e.g., SD3-M or FLUX), where no additional LoRA training.\n  \n  - The \"Instances → Image\" step utilizes LoRA-based Assemble-MMDiT, which is only activated during first 30% steps during inference.\n\n- **LoRA Application:** In the SD3-based version, we apply LoRA to all Assemble-MMDiT blocks, whereas in the Flux-based version, due to resource constraints, LoRA is applied to only eight blocks (seven double-blocks and one single block).\n\nWe will make these implementation details clearer in the revised version to ensure better readability and transparency.\n\n> **Weakness 2:** **Rationale behind Assembling-Attn:**...\n\nWe are grateful for the reviewer's feedback regarding the unclear presentation of our design choices. We will take this opportunity to elaborate on our design rationale and will make sure to articulate it more effectively in the revised version. \n\nOur method directly links each instance token with the corresponding image tokens within its bounding box, using Assemble-Attn. This approach aims to improve object localization accuracy by **ensuring that each instance's attention is focused on its specific region**. Moreover, as shown in our ablation study (Section 4.4), layout-aware generation **is primarily enabled by our architectural design**, rather than by any inherent property of the attention in MMDiT (see the comparison between Row 1 and Row 2).\n\nWe would also like to acknowledge **GrounDiT**[1] as a valuable and insightful work that explores important properties in DiT models. However, the design and objectives of **GrounDiT** differ significantly from ours:\n\n- **GrounDiT** introduced the concept of **semantic sharing**: when a smaller noisy patch is jointly denoised with the full noisy image, the two gradually become semantic clones. GrounDiT leverages this property to perform layout-to-image generation by cropping each instance's image tokens and denoising them together with **their corresponding instance images**.\n\n- In contrast, **InstanceAssemble** directly crops the noisy latent according to instance bounding boxes, applies **instance-wise self-attention**, followed by our \"Assemble\" process to aggregate the features, and reassembles the modified latent back into the full image before proceeding with denoising.\n\nWe believe that both approaches are complementary in advancing the understanding and improvement of DiT-based layout-aware generation. We will include a more detailed discussion of **GrounDiT** in our revised version to highlight these distinctions and further emphasize the uniqueness of our approach.\n\n> **Weakness 3:** **DenseSample**:...\n\nWe thank the reviewer for pointing out this important question regarding **DenseSample**. We are happy to clarify its design and motivation:\n\n- While each instance token attends only to a **cropped subset of image tokens** within its bounding box (as part of the Assemble-Attn mechanism), the coordinate embedding remains **essential** for several reasons. It **injects positional awareness** into the instance token and helps the model **disambiguate between visually similar instances** (e.g., two differently colored dogs in nearby but distinct locations).\n\n- As demonstrated in our ablation study (Section 4.4), **DenseSample** provides richer geometric context, leading to improved performance in layout-aware generation.\n\n- We note that this is **not entirely identical to the grounding token in GLIGEN** [2]. In GLIGEN, grounding tokens are global prompts that attend over the full image, whereas in our method, instance tokens are localized in both representation and attention scope.\n\nFinally, we have initiated preliminary ablation studies suggesting that removing coordinate encoding significantly degrades layout understanding and instance disambiguation, particularly in dense scenes. However, due to time constraints, we were unable to complete these experiments during the rebuttal period. We plan to include the results in the revised version or discuss them as a direction for future work.\n\n---\n\n> **Q1:** For the quantitative comparisons in both Table 1 and Figure 5, ...\n\nWe select InstanceDiff[3], MIGC[4] and HICO[5] for comparison, as they are strong and influential baselines in layout-to-image generation. Our primary focus, however, is on the comparison with CreatiLayout[6] in Table 1 and Figure 5a, which uses the same SD3 based architecture. In this comparison, our method achieves SOTA results, a point further validated by our comparison with Flux-based method (in Figure 5b). Also, the ablation study in section 4.4 highlights the impact of each component on the overall performance. We will clarify these points in the revised manuscript and highlight that our contribution lies in enabling effective layout-aware generation for modern DiT-based architectures.\n\n> **Q2:** Moreover, I'm curious whether the previous layout-to-image methods ...\n\nWe thank the reviewer for this crucial question, as it allows us to more precisely situate our contribution in the field. The reviewer’s intuition is correct: to our knowledge, classic layout-to-image methods like GLIGEN [2] and InstanceDiffusion[3], designed for U-Net architectures, have **not** been directly extended to MMDiTs. With recent approaches on MMDiT such as CreatiLayout[6] (based on SD3) and several training-free methods on FLUX[7,8], we provide direct comparisons in Table 1 and Figure 5 , and show our SOTA performance. \n\nDue to the lack of discussion on efficient layout control in MMDiT models, our work introduces:\n\n- A novel attention mechanism (**Assemble-Attn**) that enables layout-aware generation and is fully compatible with DiT-based architectures,\n\n- A **cascaded** structure that processes the global prompt and instance conditions sequentially, ensuring both high global image quality and precise layout control,\n\n- Instance-wise conditioning strategies that support flexible and explicit **multi-conditional** generation, and\n\n- A **lightweight** design that supports efficient training and inference.\n\nthereby enabling an efficient, flexible, and practical layout control method in modern DiT-based models.\n\nWe greatly appreciate your thoughtful feedback, which has substantially strengthened the clarity, rigor, and overall structure of our paper. We will carefully incorporate these changes in the final revision and hope that you will take them into consideration during your final review.\n\n[1] Y. Lee, T. Yoon, and M. Sung. GrounDiT: Grounding Diffusion Transformers via Noisy Patch Transplantation. In Advances in Neural Information Processing Systems, 2024.\n\n[2] Y. Li, H. Liu, Q. Wu, F. Mu, J. Yang, J. Gao, C. Li, and Y. J. Lee. GLIGEN: Open-Set Grounded Text-to-Image Generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22511–22521. IEEE, 2023.\n\n[3] X. Wang, T. Darrell, S. S. Rambhatla, R. Girdhar, and I. Misra. InstanceDiffusion: Instance-Level Control for Image Generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6232–6242. IEEE, 2024.\n\n[4] D. Zhou, Y. Li, F. Ma, X. Zhang, and Y. Yang. MIGC: Multi-Instance Generation Controller for Text-to-Image Synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6818–6828. IEEE, 2024.\n\n[5] B. Cheng, Y. Ma, L. Wu, S. Liu, A. Ma, X. Wu, D. Leng, and Y. Yin. HiCo: Hierarchical Controllable Diffusion Model for Layout-to-image Generation. In Advances in Neural Information Processing Systems, 2024.\n\n[6] H. Zhang, D. Hong, Y. Wang, J. Shao, X. Wu, Z. Wu, and Y.-G. Jiang. CreatiLayout: Siamese Multimodal Diffusion Transformer for Creative Layout-to-Image Generation, 2025.\n\n[7] A. Chen, J. Xu, W. Zheng, G. Dai, Y. Wang, R. Zhang, H. Wang, and S. Zhang. Training-free Regional Prompting for Diffusion Transformers, 2024.\n\n[8] Z. Chen, Y. Li, H. Wang, Z. Chen, Z. Jiang, J. Li, Q. Wang, J. Yang, and Y. Tai. Region300 Aware Text-to-Image Generation via Hard Binding and Soft Refinement, 2024."}, {"review_id": "6ADuDJ1L0I", "reviewer": "Reviewer_KyXo", "summary": "This work studies the multimodal layout-to-image generation with the generative diffusion models. The multimodal condition here can be either visual condition or text condition. This work uses a regional cross-attention between the image features (as query) and condition tokens (as key and value) to apply the conditions to the given layout position. The positional information is added to the condition tokens via a layout encoder with “DenseSample”. The DenseSample turns the bounding box to a series of tokens representing position. For evaluation, this works proposed Layout Grounding Score, which uses a visual language model to judge if the generated instance aligns with the given prompt.", "questions": "Please see my weaknesses regarding experimental setting and technical novelty.", "limitations": "While there is the limitation and broader impacts section, the author's claim of limitation and broader impacts seems pretty general. e.g., the limitation is the designed assemble module needs to be sequentially called and the broader impacts is general for all generative models. In reviewer's view, the same limitation and broader impacts can be applied in the same way to other layout-to-image methods and therefore is not strongly related to this work.", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengthens\n* This work proposes an evaluation protocol by using visual LLM to assess the layout-to-image generation. \n\n\nWeaknesses\n\n* The comparison of the proposed method against most of baselines are using different backbones. The InstanceDiff, MIGC and HICO are all using the SD1.5 or its variations (e.g., 1.4 or realistic SD 1.5). Though these variations might have different generation capabilities, they still share the same architecture and model capacity (e.g, number of parameters and pretrained data scale). However, using the SD3 model can cause a significant change in the backbone model capacity change. If one checks the Layout Grounding Score in the Table 1 and 2, the InstanceDiff’s performance is very close to this work. Given this observation, if the propose work uses the same backbone model (SD 1.5), it is unsure if it can still outperform the SD1.5 based methods.\n* The reviewer has some questions regarding the number of additional parameters. In the Table 4, the assemble attention and LoRA are in two different columns, so the reviewer assumes they are two different components in the networks and can be added individually. While in the Table 3, the proposed method introduces only 3.46% or 0.84% of new parameters. Such ratios aligns with the reviewer’s knowledge of the number of additional parameters of LoRA. However, it seems that such number is the total number of new params of both assemble attention and LoRA. The reviewers wonders how the author achieved to make the number of new parameters in the assemble attention to be such small. (when I read the Fig 2, the assemble-MMDiT module is repeated N times and I assume N stands for number of layers in the DiT model. In such case, the total number of parameters in the attention layers should be relatively large)\n* I also have some concerns regarding the technical novelty regarding the assemble attention layer. This module seems to be the core module that apply the layout condition to the image. The realization in the 3.2 seems to be a region-wise cross-attention between visual token and condition tokens. However, such strategy has also been InstanceDiff (where they use an attention mask to achieve regional cross-attn) and [11], which applies the regional cross-attention for layout conditioning. The only fundamental difference that reviewer can found is the encoding of bounding box is different in this work, where the author uses a DenseSample strategy. Nevertheless, this seems to be a nice engineering strategy to improve performance but show little technical novelty.", "was_revised": true, "final_justification": "Please see my last comment", "rebuttal": "We sincerely thank the reviewer for your thoughtful comments. We greatly appreciate the opportunity to clarify our work and have carefully considered each point raised. Below, we respond to the reviewer's main concerns:\n\n> **Weakness 1:** The comparison of the proposed method against most of baselines are using different backbones. ...\n\nWe select InstanceDiff[1], MIGC[2] and HICO[3] for comparison, as they are strong and influential baselines in layout-to-image generation. Our primary focus, however, lies in the comparison with CreatiLayout[4] in Table 1 and Figure 5a, which uses the same SD3-medium based architecture. In this comparison, our method achieves SOTA results, a point further validated by our comparison with Flux (in Figure 5b). Also, the ablation study in Section 4.4 highlights the impact of each component on the overall performance.\n\nAs for the LGS in Table 2, our performance is comparable to InstanceDiff, which is noteworthy given that we did not perform any additional COCO-specific training or fine-tuning as InstanceDiff did. We believe this highlights the strong generalization of our approach without dataset-specific optimization. The effectiveness of our method is not solely attributed to the enhanced capabilities of the base model, but also to our novel architectural design, which we elaborate on in our response to Weakness 3.\n\n> **Weakness 2:** The reviewer has some questions regarding the number of additional parameters. ...\n\nWe apologize for the ambiguity in our description in Table 4. To clarify, in this table, \"Assemble\" refers to the architecture design of Assemble-MMDiT, while \"LoRA\" refers to its training method:\n\n- The rows **without** \"LoRA\" indicate full fine-tuning of Assemble-MMDiT modules.\n\n- The rows **with** \"LoRA\" indicate fine-tuning using LoRA.\n\nTherefore, the additional parameters in Table 3 correspond to the final lora-based version used in our work. Thanks a lot for pointing this out, we will revise and clarify this point in the revised version of our manuscript.\n\n> **Weakness 3:** I also have some concerns regarding the technical novelty regarding the assemble attention layer. ...\n\nWe acknowledge that our approach is indeed a form of region-wise self-attention strategy. However, our novelty lies in a series of design choices that distinguish our work from previous methods. We will now detail the distinctions and re-emphasize our core innovation in model design.\n\n- **Layout injection mechanism:** Instead of relying on global attention masks (e.g., InstanceDiff [1]), we introduce an instance-wise self-attention mechanism. We argue that the use of global attention masks can lead to semantic leakage in overlapping regions. In contrast, our approach, which involves separating instance injection by processing cropped latent regions followed by a feature fusion step, is specifically designed to better handle dense layout scenarios. This claim is empirically supported by the experiment below, which demonstrates that our method achieves superior instance attribute consistency and a higher VQA score compared to the global attention mask-based approach.\n\n| LayoutSAM-EVAL | spatial   | color     | texture   | shape     | VQA       |\n| :-------------- |---------:|---------:| ---------:| ---------:| ---------:|\n| SD3-medium (base model) | 77.49     | 60.28     | 62.55     | 60.38     | 93.30     |\n| attention mask  (SD3-M) | 94.11     | 74.28     | 77.58     | 76.54     | 91.53     |\n| InstanceAssemble(ours) (SD3-M) | **94.97** | **77.53** | **80.72** | **80.11** | **93.12** |\n\n- **Instance token encoding:** Unlike the method in [5], which directly adds position and text embeddings, we employ a learnable fusion module to fuse positional and text embeddings for more expressive instance-level representations. This fusion module improves alignment and generalization across diverse layouts.\n\n- **Overlapping region handling:** Unlike the previous method [5], which handles overlapping regions by isolating them into non-interacting segments, our **\"Assemble\"** process directly aggregates features in overlapping regions. The **\"Assemble\"** design enables the modeling of interactions between instances, proving highly effective in complex and dense layouts.\n\nTogether, our design presents a distinct, efficient, and precise layout injection paradigm, validated by strong empirical results in challenging scenarios. We will sharpen these points in the revised manuscript. Thanks for prompting this important clarification.\n\n---\n\n> **Limitations**: While there is the limitation and broader impacts section,...\n\nThank you for your valuable feedback on our limitations. We have since re-examined our method and its boundaries. While our method addresses several practical and performance challenges in layout-conditioned generation, we acknowledge that certain limitations specific to our proposed approach remain open:\n\n- **Image fidelity degradation** may still occur under extremely dense or complex layouts. Mitigating these effects remains a key direction for future improvement.\n\n- **Extending controllability** to support additional conditioning signals (e.g., style, segmentation, or scene attributes) within a unified framework is another active research frontier, which we aim to explore in future work.\n\nOnce again, we thank the reviewer for the thoughtful and constructive feedback, which has prompted deeper reflection on the scope and impact of our contributions. We hope our responses clarify our design choices, technical novelties, and experimental decisions, and that they have addressed the reviewer’s concerns. We would greatly appreciate it if you would consider these points in your final assessment of our paper.\n\n[1] X. Wang, T. Darrell, S. S. Rambhatla, R. Girdhar, and I. Misra. InstanceDiffusion: Instance-Level Control for Image Generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6232–6242. IEEE, 2024.\n\n[2] D. Zhou, Y. Li, F. Ma, X. Zhang, and Y. Yang. MIGC: Multi-Instance Generation Controller for Text-to-Image Synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6818–6828. IEEE, 2024.\n\n[3] B. Cheng, Y. Ma, L. Wu, S. Liu, A. Ma, X. Wu, D. Leng, and Y. Yin. HiCo: Hierarchical Controllable Diffusion Model for Layout-to-image Generation. In Advances in Neural Information Processing Systems, 2024.\n\n[4] H. Zhang, D. Hong, Y. Wang, J. Shao, X. Wu, Z. Wu, and Y.-G. Jiang. CreatiLayout: Siamese Multimodal Diffusion Transformer for Creative Layout-to-Image Generation, 2025.\n\n[5] J. Cheng, Z. Zhao, T. He, T. Xiao, Z. Zhang, and Y. Zhou. Rethinking The Training And Evaluation of Rich-Context Layout-to-Image Generation. In Advances in Neural Information Processing Systems, 2024. ([11] in our paper)"}]

Author rebuttal:

Meta-review: This paper introduces an instance-assembling attention mechanism for layout-to-image generation. It also introudces a new benchmark and metric for the task. State-of-the-art performance is reported. The reviewers generally agreed that the proposed solution is effective, the results are strong, and the new benchmark and metric are solid contributions. They raised some questions related to implementation details, technical novelty, design justification, over-saturation of the results, comparison with other models, and scaling issue. The authors addressed these question adequately in the rebuttal and discussions. The final ratings are 2 "accept" and 2 "borderline accept". It is recommended to accept this paper for its solid technical contributions.
Decision: Accept (poster)