Title: InstanceAssemble: Layout-Aware Image Generation via Instance Assembling Attention
Venue: NeurIPS 2025
Accepted: yes
Keywords: Text to Image

Latent Diffusion Model

Layout to Image
Abstract: Diffusion models have demonstrated remarkable capabilities in generating high-quality images. Recent advancements in Layout-to-Image (L2I) generation have leveraged positional conditions and textual descriptions to facilitate precise and controllable image synthesis. Despite overall progress, current L2I methods still exhibit suboptimal performance.
Therefore, we propose InstanceAssemble, a novel architecture that incorporates layout conditions via instance-assembling attention, enabling position control with bounding boxes (bbox) and multimodal content control including texts and additional visual content. Our method achieves flexible adaption to existing DiT-based T2I models through light-weighted LoRA modules. 
Additionally, we propose a Layout-to-Image benchmark, Denselayout, a comprehensive benchmark for layout-to-image generation, containing 5k images with 90k instances in total. We further introduce Layout Grounding Score (LGS), an interpretable evaluation metric to more precisely assess the accuracy of L2I generation. 
Experiments demonstrate that our InstanceAssemble method achieves state-of-the-art performance under complex layout conditions, while exhibiting strong compatibility with diverse style LoRA modules.
The code and pretrained models are publicly available at \url{https://github.com/FireRedTeam/InstanceAssemble}.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work studies the multimodal layout-to-image generation with the generative diffusion models. The multimodal condition here can be either visual condition or text condition. This work uses a regional cross-attention between the image features (as query) and condition tokens (as key and value) to apply the conditions to the given layout position. The positional information is added to the condition tokens via a layout encoder with “DenseSample”. The DenseSample turns the bounding box to a series of tokens representing position. For evaluation, this works proposed Layout Grounding Score, which uses a visual language model to judge if the generated instance aligns with the given prompt.\n\nSTRENGTHS AND WEAKNESSES:\nStrengthens\n* This work proposes an evaluation protocol by using visual LLM to assess the layout-to-image generation. \n\n\nWeaknesses\n\n* The comparison of the proposed method against most of baselines are using different backbones. The InstanceDiff, MIGC and HICO are all using the SD1.5 or its variations (e.g., 1.4 or realistic SD 1.5). Though these variations might have different generation capabilities, they still share the same architecture and model capacity (e.g, number of parameters and pretrained data scale). However, using the SD3 model can cause a significant change in the backbone model capacity change. If one checks the Layout Grounding Score in the Table 1 and 2, the InstanceDiff’s performance is very close to this work. Given this observation, if the propose work uses the same backbone model (SD 1.5), it is unsure if it can still outperform the SD1.5 based methods.\n* The reviewer has some questions regarding the number of additional parameters. In the Table 4, the assemble attention and LoRA are in two different columns, so the reviewer assumes they are two different components in the networks and can be added individually. While in the Table 3, the proposed method introduces only 3.46% or 0.84% of new parameters. Such ratios aligns with the reviewer’s knowledge of the number of additional parameters of LoRA. However, it seems that such number is the total number of new params of both assemble attention and LoRA. The reviewers wonders how the author achieved to make the number of new parameters in the assemble attention to be such small. (when I read the Fig 2, the assemble-MMDiT module is repeated N times and I assume N stands for number of layers in the DiT model. In such case, the total number of parameters in the attention layers should be relatively large)\n* I also have some concerns regarding the technical novelty regarding the assemble attention layer. This module seems to be the core module that apply the layout condition to the image. The realization in the 3.2 seems to be a region-wise cross-attention between visual token and condition tokens. However, such strategy has also been InstanceDiff (where they use an attention mask to achieve regional cross-attn) and [11], which applies the regional cross-attention for layout conditioning. The only fundamental difference that reviewer can found is the encoding of bounding box is different in this work, where the author uses a DenseSample strategy. Nevertheless, this seems to be a nice engineering strategy to improve performance but show little technical novelty.\n\nQUESTIONS:\nPlease see my weaknesses regarding experimental setting and technical novelty.\n\nLIMITATIONS:\nWhile there is the limitation and broader impacts section, the author's claim of limitation and broader impacts seems pretty general. e.g., the limitation is the designed assemble module needs to be sequentially called and the broader impacts is general for all generative models. In reviewer's view, the same limitation and broader impacts can be applied in the same way to other layout-to-image methods and therefore is not strongly related to this work.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces InstanceAssemble, a novel framework for accurate layout-based image generation. By using Assemble-MMDiT architecture, this method can robustly handle each instance, and by using LoRA, it is more training-efficient compared to previous training-based methods that require heavy fine-tuning. Moreover, the authors introduce a new benchmark and evaluation metric to enable more consistent assessment of the generated image with respect to the given layout. As a result, InstanceAssemble achieves state-of-the-art performance for complex layout conditions.\n\nSTRENGTHS AND WEAKNESSES:\n**Strength**\n\n* Effective engineering: The paper introduces a well-engineered framework, which seems to effectively address the limited robustness of of training-free methods and the computation  \n\n* Strong results: Clear improvements in the quantitative comparisons with the baseline methods (e.g. CreatiLayout [1], HICO [2])\n\n* New benchmark: The introduction of a new benchmark (DenseLayout) and an evaluation metric (LGS) are solid contributions in the literature of layout-to-image generation.\n\n**Weakness**\n\n* Unclear points on technical novelty: While InstanceAssemble shows strong results, there are some critical questions regarding the main idea of the proposed method, as listed below:\n\n   * **Details on Assemble-MMDiT:** I find it unclear about how Assemble-MMDiT is implemented. Are the attention modules trained from scratch, or are they initialized from the original MMDiT weights? As the full pipeline in Figure 2 also uses the original MMDiT for the global prompt, does it use the same weights and only activate the LoRA parameters during inference? Moreover, to which part of the MMDiT is the LoRA applied? Please correct me if the details are already included and I have missed it.\n\n   * **Rationale behind Assembling-Attn:** Section 3.2 doesn't seem to sufficiently explain why and how the design choice of Assembling-Attn, which is to apply attention between indivisual patches and the corresponding instance tokens, would lead to more accurate object locations. Specifically, is it the case that the attention in MMDiT already has such property and the LoRA training enlarges it? (Similar to the \"semantic sharing\" property of DiT observed in GrounDiT [3]) Or does the attention module newly learn to locate each object during the training phase of LoRA?\n\n   * **DenseSample**: The design of DenseSample in Section 3.1 is also confusing. It seems that the xy coordinates of each bounding box is encoded into each instance token. However, in Assembling-Attn only a cropped subset of image tokens are attended with each instance token. Given this, why is the xy coordinate information required in the instance token? Is this identical to the **grounding token** in GLIGEN [4]? \n\nAlthough InstanceAssemble seems to produce high-quality outputs, the paper's main contributions - Layout Encoder and Assemble-MMDiT - still contain multiple unclear points regarding its technical details and novelty compared to previous works.\n\n[1] CreatiLayout: Siamese Multimodal Diffusion Transformer for Creative Layout-to-Image Generation, 2024\n\n[2] HiCo: Hierarchical Controllable Diffusion Model for Layout-to-image Generation, NeurIPS 2024\n\n[3] GrounDiT: Grounding Diffusion Transformers via Noisy Patch Transplantation, NeurIPS 2024\n\n[4] GLIGEN: Open-Set Grounded Text-to-Image Generation, CVPR 2023\n\nQUESTIONS:\nQuestions\n* For the quantitative comparisons in both Table 1 and Figure 5, only CreatiLayout is based on SD3-M and all other baselines use older text-to-image models. GIven that InstanceAssemble uses more recent models including SD3-M and Flux.1, I'm conerned whether the advantage of InstanceAssemble is coming from the novel method itself or from the better choice of the backbone model.\n\n* Moreover, I'm curious whether the previous layout-to-image methods such as GLIGEN [1] and InstanceDiffusion [2] are not applicable to MMDiT architectures, and whether there have been similar discussions on efficient ways to train them. Such discussions would provide a more clear motivation for the need of an efficient training framework like InstanceAssemble.\n\n[1] GLIGEN: Open-Set Grounded Text-to-Image Generation, CVPR 2023\n\n[2] InstanceDiffusion: Instance-level Control for Image Generation, CVPR 2024\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a new method for layout-to-image (L2I) generation based on a new Instance Assembling Attention mechanism. The proposed mechanism processes layout tokens and the matching image tokens with an additional attention block, and subsequently assembles them back into image and layout tokens. In addition, the paper also introduces a new benchmark for the L2I task based on spatial accuracy, determined by DetectIoU, and semantic accuracy, which is calculated using a VLM-based VQA protocol. Overall, the proposed method outperforms related approaches while being significantly more efficient.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n- **Straightforward but effective idea**\nThe authors leverage the attention mechanisms in MMDiT to guide the generation process to place objects accordingly. The integration of layout tokens in a cascaded way through Assmeble-MMDiT is straightforward and effective.\n\n- **Thorough and convincing ablation study**\nThe authors perform a thorough ablation study that considers multiple design choices, making a compelling case for the proposed pipeline. Specifically, the A\n\n- **Good results and great improvements in efficiency**\nBoth the quantitative and qualitative results show tremendous improvements in efficiency while also improving on most quality metrics. Table 3, in particular, demonstrates the impressive efficiency improvement clearly.\n\n### Weaknesses\n- **Bounding boxes are rectangular**\nDue to the rectangular nature of bounding boxes, fine-grained controllability is limited. While having a rough way to control the generation process might be desired in some cases, greater control might be desirable in other settings. Nonetheless, the proposed framework seems to be a good option for bounding box level of control.\n\n- **Unclear role of LoRA**\nThe authors mention the use of LoRA but do not specify for which weights LoRA is applied. See below for more specific questions regarding LoRA.\n\n- **Extensive additional training**\nThe proposed approach requires training on 8 H800 GPUs for a week. So it seems that the method moves the required computational effort from the inference step to the training step, allowing for the added efficiency. While this is not necessarily a weakness, it might be important to consider depending on the available computational budget.\n\n- **Scales with the number of instances, handling overlapping areas**\nIt looks like image tokens might be reused in Assemble-MMDiT if bounding boxes overlap, which could be inefficient. There seems to be an almost 2x increase in efficiency between sparse and dense layouts.\n\nQUESTIONS:\n- Which weights are trained using LoRA? How are these weights initialized? Why would LoRA improve the performance? What is the \"base model\" referred to in the ablation study section?\n- In Figure 2, it looks like there is one large attention block that processes all projected tokens simultaneously, but in the text, it sounds like every instance is processed separately, i.e., image tokens do not attend across instances. Which one is it?\n- Why is training performed for all time steps, but during inference, the proposed method is only applied to the first 30% of timesteps?\n- Why are the LayoutGrounding scores for DenseLayout so much lower than for LayoutSAM-Eval?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors proposed a new model for layout-aware image generation. In particular, they design LoRA modules that can be attached to modern DiT based text-to-image generative models such as Stable Diffusion 3 and Flux.1. They also proposed a new benchmark and a new evaluation metric for the layout-aware image generation task.\n\nSTRENGTHS AND WEAKNESSES:\n**Strength:**\n1. The proposed model has very impressive empirical results\n2. The experiment section is solid and thorough\n3. The addition of the new benchmark and metric is a good plus\n4. All figures in this paper are very high quality\n\n**Weakness:**\n1. The modeling details are a bit unclear from the writing. For example, what exactly is the network’s “image” input and output? I.e. which parametrization did the authors consider? Or is it different based on different backbone models?\n2. Related to weakness 1, if the training objective in this paper is the same as the backbone models, then the description of the training objective is not very accurate. Stable Diffusion 3 and Flux.1-dev both use the flow matching objective and it should not be described as the same as the one in LDM (i.e. velocity prediction with conditional optimal transport path is not the same as epsilon prediction with Gaussian probability path).\n3. The generated images from InstanceAssemble look very saturated, and with depth and edge inputs this problem is exacerbated.\n4. The authors should consider including the discussions of the following related works:\n\n        [1] Zhang, et al. “Adding Conditional Control to Text-to-Image Diffusion Models”. ICCV 2023.\n\n        [2] Hertz, et al. “Prompt-to-Prompt Image Editing with Cross-Attention Control”. arXiv 2022.\n\n        [3] He, et al. “Localized text-to-image generation for free via cross attention control”. arXiv 2023.\n\nQUESTIONS:\n1. During inference the layout condition is only applied in the first 30% of the denoising steps, then is it still essential to train with all time steps? What is going to happen if you apply the layout condition for all time steps at inference time? Will the image quality degrade significantly?\n2. Related to weakness 1, what exactly is the “image” input to the networks and what is the output of the network exactly? Are the outputs epsilon predictions, x0 predictions or v predictions? Are the “image” input the actual denoised latent, noisy latent, or epsilon?\n3. Can the authors explain the oversaturation problem?\n4. Which training dataset did the authors use for the image/depth/edge input?\n5. Does the Assemble-MMDiT have a similar size as the frozen MMDiT? If so, is the reason why it only adds 8% more inference time the fact that you only apply it in the first 30% of the denoising steps?\n6. How would the native multimodal image generation models like GPT-4o or Gemini 2.0 Flash compare to InstanceAssemble in this task?\n7. Does this model only work for bounding boxes? The authors should consider adding (demo) experiments with semantic maps if possible.\n\nLIMITATIONS:\nYes, but the authors can potentially extend the paragraph to include more discussion on potential negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}