I-DRUID: Layout to image generation via instance-disentangled representation and unpaired data
Authors: Fengxiang Yang
Tianyi Zheng
Bangjie Yin
Shice Liu
Jinwei Chen
Peng-Tao Jiang
Bo Li
Venue: ICLR 2026 Poster (2026)
Keywords: diffusion models;reinforcement learning
Primary area: generative models

Abstract: Layout-to-Image (L2I) generation, aiming at coherently generating multiple instances conditioned on the given layouts and instance captions, has raised substantial attention in the recent research. The primary challenges of L2I stem from 1) attribute leakage due to the entangled instance features within attention and 2) limited generalization to novel scenes caused by insufficient image-text paired data. To address these issues, we propose I-DRUID, a novel framework that leverages instance-disentanglement representations (IDR) and unpaired data (UID) to improve L2I generation. IDR are extracted with our instance disentanglement modules, which utilizes information among instances to obtain semantic-related features while suppressing spurious parts. To facilitate disentangling, we require semantic-related features to trigger more accurate attention maps than spurious ones, formulating the instance-disentangled constraint to avoid attribute leakage. Moreover, to improve L2I generalization, we adapt L2I with unpaired, prompt-only data (UID) to novel scenes via reinforcement learning. Specifically, we enforce L2I model to learn from unpaired, prompt-only data by encouraging / rejecting the rational / implausible generation trajectories based on AI feedback, avoiding the need for paired data collection. Finally, our empirical observations show that IDM and RL cooperate synergistically to further enhance L2I accuracies. Extensive experiments  demonstrate the efficacy of our method.

Published as a conference paper at ICLR 2026
I-DRUID: LAYOUT TO IMAGE GENERATION VIA INSTANCE-DISENTANGLEE REPRESENTATION AND UNPAIRED DATA
Fengxiang Yang, Tianyi Zheng, Bangjie Yin, Shice Liu, Peng- Jiang, Jinwei Chen, Bo Li' vivo Bluelmage Lab, vivo Mobile Communication Co., Ltd. fyangfengxiang, zhengtianyi, libra]evivo...
ABSTRACT
Layout-to-Image (I.2I) generation, aiming at coherently generating multiple in- stances conditioned on the given layouts and instance captions, has raised sub- stantial attention in the recent research. The primary challenges of L2I stem from 1) attribute leakage due the entangled instance features within attention and 2) limited generalization to novel scenes caused by insufficient image-text paired data. To address these issues, we propose I-DRUID, novel framework that lever- ages instance- disetaaddnettnnllemeent representations (IDR) and unpaired data (UID) to improve 1.21 generation. IDR are extracted with our instance disentanglement modules, which utilizes information among instances to obtain semantic-related features while suppressing spurious parts. To facilitate disentangling, we require semantic-related features trigger more accurate attention maps than spurious ones, formulating the instance -isennceeddsttnngeed constraint to avoid attribute leakage. Moreover, to improve L2I generalization, adapt L2I with unpaired, prompt-only data (UID) to novel scenes via reinforcement learning. Specifically, we enforce 1.21 model learn from unpaired, prompt-only data by encouraging / rejecting the rational / implausible generation trajectories based on AI feedback, avoiding the need for paired data collection. Finally, our empirical observations show that IDM and RI cooperate synergistically to further enhance 1.21 accuracies. Extensive experiments demonstrate the efficacy of our method.
INTRODUCTION
Recent advances in text-to-image (T2I) generation have achieved remarkable success, primarily driven by diffusion models (Rombach et al., 2022; Ho et al., 2020; Esser et al., 2024; Ramesh et al., 2021). By employing UNet (Ronneberger et al., 2015) multi-modal diffusion transformers (MM-DiT) (Li et al., 2024; Esser et al., 2024) for noise velocity prediction, these models learn to generate high-quality images with given prompts. To achieve finer control over the generation process, recent works have explored various spatial controls, such as semantic masks (Couairon et al., 2023; Kim et al., 2023; Zhang et al., 2025a), sketches (Voynov et al., 2023; Zhang al., 2023), or bounding boxes (Zhang et al., 2025b; Zhou et al., 2024a; Wang et al., 2024). Among these methods, bounding box-based control has become partieularly prevalent controlling factor (Li et al., 2023; Wang et al., 2024; Xie et al., 2023; Zhang et al., 2025b; Zhou et al., 2024b) due to its compatibility with downstream vision tasks. This has spurred the development of layouttoo-mmaee (L21) generation (Zhou et al., 2024a; Wang et al., 2024; Xie et al., 2023; Li et al., 2023; Zhang et al., 2025b), which aims to synthesize multiple objects coherently based on given spatial layout and corresponding captions.
Concurrent studies show promising results in L2I (Zhou al., 2024a; Zhang et al., 2025b; Wang et al., 2024), but two key challenges remain. (1) The information fusion within attention layer hinders instance rendering (Dahary et al., 2024; Wang et al., 2024), leading to the persistence of attribute leakage. Previous works attempt to address attribute leakage through attention map manipulation (Da- hary et al., 2024; Zhou et al., 2024a; Wang et al., 2024), but the inherent difficulty of CLIP (Radford
Corresponning author.

---

Published as a conference paper at ICLR 2026
(a) Overall Framework
(b) 1.21 Results
Layont Captions
Ours
Training Dataa Globa Prwwpt
Thrro dlrecssttg the Foe image....
Bes (UA 1151, Lans
Ben joHk coA
Adaplor
Bou JERR 19% Há, 11E2
UNet Cross-Attn
RL Datts
IDM
I. Acchnccolicnaa malliiakiingg 2: Urzes Iade dram aetiars coven grang brachee
Glibal
Nake
MM-DIT Joint-Alin
sfodee Bos 383,
related
28, 431
IDC
Traising Dasta (laugetest Pair I
(Proage
Traj:
Fahaaned Pastures
Triinate
Sample Trajecuey
Fredback
Noise Velucity Prodiction
Glabal Presage gloro dowingg hh colund I: SSuahese Aukas nucker Colord boolscrr Pirk kengokev Tradtionl lat. Coeen top.
IBC
Diccttrngknncll
ID9I Ins. Discstonpkrncat Modalc
Figure 1: (a) Overall framework of I-DRUID. I-DRUID jointly considers attribute leakage and improves model generalization via Al feedback. (b) Visual results from method.
et al., 2021) in differentiating singular attributes from complex prompts remains bottleneck (Feng et al., 2023). Moreover, few studies have considered 1.21 under MM-DiT scenario (Zhang et al., 2025b). Therefore, it is necessary explore precise instance-level representation control for both UNet- and MM-DiT-based architectures. (2) 1.21 models may encounter performance degradation when being deployed novel scenes. e.g., models trained on training set with long captions (Wang et al., 2024; Zhang et al., 2025b) get poor performance when deployed under testing set with short coarse captions (Zhou et al., 2024b), demonstrating poor generalization when training with limited pair-wise training data. Collecting more wair-wise data is an empirical solution, but suffers from huge time cost. Inspired by the success of RL in scaling large language models to complex tasks (Guo et al., 2025; Jaech et al., 2024), we would like to advance 1.21 with RL by using prompt-only unpaired data, letting the novel-scene adaptation to be exempted from pair-wise data collection.
As shown in Fig. I-(a), present I-DRUID, which is comprised of instance disentanglement learning, coupled with reinforcement learning. (1) We first introduce adapters (Ye et al., 2023) to inject layout information into training process, then decompose attention features into "semantic- related features" " and "spurious part" with our instance disentanglement modules (IDM). The core of disentanglement through IDM is based our instance disentangling constraint (IDC). Our key insight of designing IDC is that semantic-related features must elicit more precise attention maps than spurious features, thereby facilitating the disentanglement process during 1.21 optimization. (2) To generalize our L2I model to OOD prompts, we further introduce novel reinforcement learning (RL) framework (Agarwal al., 2019; Liu et al., 2025b), encouraging L2I model learn from unpaired novel prompts based on AI feedback. Specifically, we conduct trajectory sampling from Gaussian noise and the given novel prompt, producing images for the assessment of visual language model (VLM) in terms of spatial accuracy and instance consistency in an online manner. The feedback used as guidance in analyzing best generation policy, thus encouraging L21 capability. (3) We also demonstrate IDC and RL could mutually benefit each other as IDM provides more accurate generation policy for RL, advancing L2I results as shown in Fig. I-(b). Our method also has a high flexibility, which could be easily utilized for both UNet-based (e.g., SD 1.5) and MMaDeE-based architectures (e.g., SD3).
To sum up, contributions three-fold:
We seek diseenanled representation, which is achieved with our IDC-supervised IDM. IDC leverages collective instance information to extract semantic-related features, which triggers more precise attention maps and thus avoids attribute leakage.
We equip our L2I model with novel Reinforcement Learning (RL) strategy to improve its generalization. RL enables L2I to learn from unpaired data, letting novel-scene adaptation to be exempted from wise data collection.
2

---

Published as a conference paper at ICLR 2026
We demonstrate that these two components can be synergistically cooperated to further enhance L2I accuracy under both UNet and MM-DiTT-based architectures. Our approach achieves state-of- the-art results on multiple L2I benchmarks, demonstrating its efficacy and flexibility.
RELATED WORK
Layout to Image Generation. Diffusion models like SD (Ho ct al., 2020; Rombach et al., 2022), SD3 (Esser et al., 2024), and FLUX (Labs, 2024) are powerful tools to achieve text to image generation. To achieve fine-level control, additional spatial control like bounding boxes (Wang et al., 2024; Li et al., 2023; Zhou et al., 2024b;a; Zhang et al., 2025b; Lee al., 2024) are introduced into generation process to craft entities within the given location and instance-level prompts, termed as layout image generation. Generally speaking, these methods can be categorized as training- based (Zhou et al., 2024b; Li et al., 2023) and training- free approach (Xie et al., 2023; Lee et al., 2024). The former introduce adapters (Ye et al., 2023; Mou et al., 2023) and additional modules to encode locations for subsequent attention. The latter focus manipulating attention map at test-time to achieve location control for each entity. Although effective, most of them are conducted based UNet architecture, while ignoring modern MM-DiT architectures like SD3 (Esser et al., 2024). Recently, Creati-Layout (Zhang et al., 2025b) first devise SD3-based L2I scheme, but ignores explicit constraint to alleviate attribute leakage ( (Wang et al., 2024). Different from previous methods, our approach devise disentanglement modules to advance L2I task under both UNet-based and MM-Di'T-based scenarios.
Reinforcement Learning. Reinforcement learning (RL) (Schulman et al., 2017) is a widely-used strategy to align model's response to human preference, both for large language models (Rafailov et al., 2023; Wang et al., 2025; Yan et al., 2024) and diffusion models (Chen et al., 2024; Fan et al., 2023). The core idea of RL is training model interact with environment, typically reward model, that provides feedback for the model's ' responses. This feedback guides the model in exploring and learning an optimal policy. Early RL methods for alignment often relied on either separate reward model (Ramamurthy et al., 2022; Peng et al., 2019) or extensive human annotation of samples (Rafailov et al., 2023; Yuan al., 2024) to obtain preference data. However, the recent development of large-scale models has led to a paradigm shift. Learning from Al feedback has emerged as a promising and efficient alternative for scaling the alignment process (Bai et al., 2022; Fan al., 2023). The RL under diffusion scenario offline scheme. DPOK (Fan et al., 2023) and DDPO (Black et al., 2023) are pioneering works that introduce RL into image generation by Iewge (Xu et al., 2024) in an online manner. DifIDPO (Wallace et al., 2024) proposes the first DPO-based (Rafailov et al., 2023) RL method to fine-tune diffusion in offline manner. Although these methods show promising results, most of them relies on the randomness of generation trajectory to achieve environmental exploration, which is not compatible with SD3 (Esser al., 2024) or FLUX (Labs, 2024) with deterministic ODE forward process. Recently, the advent of SDE-based forwarding equivalence (Albergo et al., 2023; Liu al., 2025a) enables efficient exploration during the forwarding process of now-matching methods. However, the application of RL under L2I scenario still exper-explored, inspiring us to scale L21 model with the help of AI feedback and RL.
be categorized into online and
generation policy with feedback from AI like
3 METHODOLOGY
3.1 OVERVIEW
Problem Formulation. Layout to image generation (I.2I) assumes the users to give the following
information during inference. (1) global prompt PG defining the overall semantic information for the generated image; (2) n instance captions Pt, = {pI, ,2, ...: )n which describes detailed information for each instance; (3) layout B = {b), b2 ..., bn J. where b; {zi0, y:o, Zi1, yi1 )a (1 < < n) containing lep-left coordinate (z \/oo and bottom-right coordinate (I,1, Va) for each instance. The goal of L2I is generating images that follows the above instructions. Although of L21 for both UNet- based (e.g., SD-1.5) and AM-DiT-ber architectures (e.g., SD3), we mainly introduce our method with SI33 and the SD- variant could be easily derived.
method is capable
3

---

Published as a conference paper at ICLR 2026
Gloted Prozeet
(a) Instance Disentanglement Learning
(b) Reinforcement Learning
Eacoder
Test
Teat Pratuna
Comvinsny
Reward-1.00 ..0 Kevand-d, 42
MM-Attention
Image
Accanacy
Fezalures
GDINO
Image Eacoder
MM DIT
S
lafs
CASOR'S
CASIR)
Cross-Atratiion
IDM
Instance Encoder
MM-DDI
Caption
MIM-DIT
Canser da MEL
@
Leyour
Trajectory Sumpling
Figure 2: Overall Training Process. Our method comprised of: (a) Instance Disentanglement Learning and (b) Reinforcement Learning Stage. At (a), our instance disentanglemer module (IDM) is supervised by instance disentanglement constraint (IDC) for extracting semantic-related representations R= while discarding spurious part R. based on the proposed metric CAS. Moreover, we sample various generation trajectories with unpaired data and adopt GDINO to evaluate generated images. Better results will be encouraged at (b), adapting L21 model novel scenes.
Overall Pipeline. As shown in Fig. 2-(a), Our L21 solution is comprised of (a) instance disentangle- ment learning and (b) reinforcement learning. The former is built upon adapter-based L2I (Li et al., 2023; Wang et al., 2024), further refining features based on our IDM and IDC. The latter enforces 1.21 model to seek for the optimal generation policy based the given prompt and feedback from reward, thus generalizing the model to novel scenes without using image-text paired data. These two contributions mutual benefit each other and further improves 1.21 capability. Next, introduce solution in detail.
3.2 INSTANCE DISENTANGLEMENT STAGE
Instance Disentanglement Module. Previous L21 methods (Li et al., 2023; Wang et al., 2024; Zhang et al., 2025b) achieve layout control by injecting layout-caption coupled features with cross- attention for optimization. However, Zhou et al., 2024b;a), attention tends to fuse instance-level features, leading to the attribute leakage. We therefore devise IDMM as feature refiner alleviate this. The IDM specifications are shown in Fig. 2-(a), which receives n + 1 enhanced features and layout masks as inputs. IDM learns to extract "semantic-related features" RR while discarding "spurious part" R for subsequent generation. Specifically, we compute the given n instances' enhanced features with corresponding layout and captions, coupled with global patpall--mmmttteeaanae
demonstrated in previous literature (Dahary et al., 2024;
features to formulate n + inputs {el, e,, e,+1 [Rx++)xCCXW xH These features formulate the input of feature branch. For mask branch, we obtain layout masks for instance with its bounding box b, = {I yio, Tai, ( Pixels within bounding box b, will be assigned with while others are set to O. The mask for global prompt is set to all-one matrix, formulating n + 1 masks M = {m,, m2...., m,+1). IDM jointly considers and M get the optimal channel-wise weighting scheme:
a = IDM(E,M) E R=*1
(I)
where a is the weighting scores assigned to each enhanced feature in &. We thus obtain "semantic- related features" R+ while discarding "spurious part" R through simple multiplication:
R = O E; R = (1 (1 &
(2)
where R and R are "semantic-relle -reatted features"' and "spurious part" with the size of (n + 1) x C x W x H.
Instance Disentanglement Constraint. To facilitate the disentangling process, we further devise instance disentanglement constraint (IDC) by measuring R rs ability in triggering accurate attention map. Specifically, R will be used to obtain cross-attention map R R We then adopt layout mask M to extract attention value at background and evaluate overall attention accuracy with the following
4

---

Published as a conference paper at ICLR 2026
eriterion:
CAS(R CA
Σ R R AVG(R o (1 o (M,)
(3)
where M, and CA,i are the mask and triggered attention map for instance i, respectively. (- the averaging operation. In our design, AVG (1 (1-M))) extracts instance i's averaged background attention values. Eq. 3 subsequently quantifies the total absolute deviation beyond instance i's scope. Higher CAS indicates potential bounding box mis-alignment for the generated instance as the instance will lead high dispersion at background. Based on CAS, it obvious that R should trigger lower CAS value than R formulating an inequality CASR+ < inequality could be further transformed into our "Instance Disentanglement Constraint":
CASR-. The
Lais (RCA) CAA = Softplus CAS(R - CAS(R
(4)
where Softplus In(1 + exp(-)) ()) is a monotonically increasing function. Minimizing Eq. 4 is equivalent encourage lower CAS score for RR and higher CAS for R thus enabling our disentanglement to heuristically learn semantic-related features for L2I generation.
3.3 REINFORCHMENT LEARNING STAGE
RL Formulation. We build our algorithm based on classical PPO (Schulman et al., 2017). The elements of conventional RL could be summarized into four essentials: (1) state (2) action (3) policy (4) reward. We then introduce these elements in the context of flow matching model SD3. For the given time and condition y = pb (b and p are bounding box and their corresponding instance captions), "state" is S, - f, "action" denotes denoised latent at - i,i., a, _ X,-1- "Policy" denotes the transition probability between two time steps T Tayy inferring from T to 0, the model generate final output Xp. which could be subsequently decoded by VAE to obtain image Xo, and evaluated with reward model r(.) in terms of spatial accuracies and instance consistency. r(-) assigns score as guidance supervise the overall generation process, thus promoting the L2I accuracy when novel prompts emerges. For UNet-based diffusion model, policy derived based on DDPM (Ho al., 2020), which contains randomness to achieve exploration. However, MM-DiT-B based diffusion is built upon flow matching (Esser et al., 2024), which relies on deterministic ODE and hinders environmental exploration. We thus divert deterministic ODE sampling to SDE to formulate our policy. Specifically, we follow (Domingo-Ennich et al., 2024; Albergo et al., 2023; Liu et al., 2025a) and change the policy as follow:
(s;) - -x (x. By
of
X = X + (x): 4,) +
(x+ + (1- 1)va Xt, 42)) Al oLVAle
(5)
21.
where X, is the partially denoised image latent at time i, is the predicted velocity. is the random term sampled from standard Gaussian distribution NO, 1). at = =ay - and a = 0.7 is hyper-parameter defined in (Liu al., 2025a).
Reward Definition. Considering the layout-control nature of L2I task, we choose Grounding-DINY DINO (GDINO) (Liu et al., 2024) as the reward model r(-) to assess the overall spatial accuracy and instance consistency for generated images. GDINO receives instance captions and X, to detect the location of objects within the input, then returns detected bounding box and confidence score Opred pred, Cpred). ). We thus define the reward based Opred and ground-truth o:
r(o, Opred) = Σ [IOU by) + Cpred,
(6)
where bpred,; is the ground-truth bounding box for instance i, Cpred, is the confidence score in detecting instance i based instance caption Ppred, IoU is the function to compute IoU score between ground-truie bounding box and GDINO ppredceped counterpart. Based on these formulations, require L21 model to perform actions and interact with environment, seeking the optimal action by optimizing underlying policy. Following standard RL formulation, we term our diffusion model as "actor net".
5

---

Published as a conference paper at ICLR 2026
Critic-Net. During RL fine-tuning, also introduce critic-net e as collaborator further improve actor-net's generation capability. The critic-net net a lightwweighter MLP, which receives model state 8, and embedded user inputs predict final scalar reward value. Its training goal is minimizing the discrepancy between predicted and actual reward given by GDINO to learn the average reward:
Leritic O, Opred) = ((ss) r(o, 0 0prea)
With critic-net, could further define advantage score by subtracting critic-net's prediction from reward:
A o, Opred) = r(o, Opred) d(s)),
the advantage function measures the expected reward over the average performance, which facilitates actor-critic collaboration.
Actor-Critic Collaboration via PPO. The goal of RL is fine-tuning actor-net to obtain high advantage scores for generated images. By following classical PPO (Schulman al., 2017) algorithm, we use importance sampling to optimize actor net, Specifically, we compute probability ratio Pr at t-th step by comparing the policy for current and old model:
TTae (8t. π (apld) yy)
where agdd and st are action and state for old actor net, which is commonly the EMA model (Klinker, 2011). The final loss for actor-net could be formulated as:
L = max Ac. clip(e: 1 - C, + 4. + KT.(T(a y) (aq (sold yy),
(10)
where clip(-) is the clipping function, limiting the probability ratio in the range of [I-Ç, I+[] to ensure stable training (Schulman et al., 2017), (T(-II-) is the KI. divergence to avoid training collapse. It should be noted that L, instructs and optimizes actor-net through Pt to achieve the enhancement of advantage scores A, A, does not directly supervise actor-net's training.
Joint Optimization. We incorporate flow matching loss (Esser et al., 2024) to to optimize our actor-net:
Laam = Ee..V || ve(x,, 4,)x,
(11)
where is the ground-truth velocity, obtained through encoded image and Gaussian noise. For UNet-based diffusion model like SD 1.5 (Rombach et al., 2022), the objective of training should be noise prediction (Ho et al., 2020). We will introduce our UNet-based solution in the supplementary. The final loss for actor-net's optimization are formulated as follow:
Lact Liam AdieLaii + Xpt
(12)
where Adis and 'r are balancing factors. practice, we utilize model pool MP to collect old model's 's policies. MP is a memory pool, storing old model's policies to facilitate RL. When computing Eq. 12, we randomly sample an old policy for image generation and optimization. The overall training pipeline is shown in our appendix, at Alg. 1.
RI. Speed-up. Our original RL scheme requires trajectory sampling for reward computation and action-based fine-tuning, consuming large amount of time due to the inference of model with full time-steps. Recently, the finding that optimization at early time steps (Zhou et al., 2024c; Kang et al., 2025; Zheng al., 2025) yields similar or even better results inspiring us conduct RL at early time steps. We therefore conduct RL only at the first 20% time steps and save corresponding actions to avoid reference model's action computation. Moreover, computational advantage over the popular GRPO-base based methods (Liu et al., 2025a) as GRPO requires sampling several trajectories at the same time for group-level advantage computation.
PPO-based RL naturally has
4 EXPERIMENT
4.1 EXPERIMENTAL. SETTING
Training Sets. We train our model with Lay (Zhang et al., 2025b) and COCO-2014-
MIG (Zhou al., 2024b). LayoutSAM contains 2.7 million image-text pairs and 10.7 million

---

Published as a conference paper at ICLR 2026
Table 1: Comparison with state-of-the-ar methods on COCO-MIG. We compare our method with state-of-the-art generation methods, including BoxDiff (Xie et al.. 2023), Reco (Yang et al., 2023), GLIGEN (Li et al., 2023), RichContext Cheng et al. (2024b), InstanceDiff (Wang et al., 2024), MIGC (Zhou et al., 2024b), and Creati-Layout (Zhang et al., 2025b). *; Evaluated based on MIGC's code. Red and Green are best and second best results.
Instance Success Rate t
mloU
f
Methods
1.2
1.3
1.4
T.5
T.6
Avg
1.2
1,3
1.4
1.5
1.6
Avg
BoxDiff
24.61 19.22 14.20
11.92
9.31
15.85
32.54
29.88
25.39
23.81
21.19
26.56
GLIGEN
42.30 35.55 32.66 28.18
30.84
33.89
37.58
32.34
29.95
26.60
27.70 30.83
RichContext
40,31 30.83 30.78 26.50 25.42 30.76
37.88
31.43 30.35 28.42 26.59 30.93
InstanceDiff
58.00 52.16 55.03 47.59 47.12 51.98
52.14
48.64 50.36
42.64
42.86 47.33
MIGC
67.70 59.61 58.09 56.16 56.88 59.68
59.39
52.73 51.45 49.52 49.89 52.60
Reco
65,50 56. 10 52.30 52.40 58.30 56.90
55.70
46.70 47.20 43.30
48.80 47.60
Creati-Layout
65.93 65.41 56.40 50.62 50.00 57.67
56.61
56.29 50.30 45.85 45.66 50.94
Ours (SD-1.55)
79.10 70.24 65.48 63.87 66.97 69.13
70.61
62.10 58.63
56.04
58.18 68.18
Ours (SD-3)
76.87 69.16 62.96 52.37 52.39 62.75
63.35
59.01 53,80 51.32 50.54 55.60
fine-gre iiee-grinedd instance-level captions derived from SAM (Kirillov et al., 2023) dataset. Different from LayouSAA,, COCO-2014-MIG (Zhou et al., 2024b) adopts stanza (Qi al., 2020) recognize entities, while obtaining instance-level bounding boxes with Grounding-DNie DINO.. Therefore COCO- 2014-MIG has coarse instance captions while LayoutSAM has detailed insatance captions. We only use the prompt (layout and instance captions) from COCO-2014-MIG for RL while adopt image-text pairs from Lay for diffusion training.
Evaluation Protocol. We evaluate our method on two large-scale benchmarks: COCO-MIG (Zhou AM-eval et al., 2024b) and A.M-evll (Zhang et al., 2025b).COCO-MC COCO-MIG consists of 800 multi-instance prompts from COCO-2014, spanning five complexity levels (L2-L6) based on instance counts. Following (Zhou et al., 2024b), we use Grounded-SAN (Ren et al., 2024) to assess spatial correctness via Instance Success Rate (ISR), which measures the percentage of successfully detected instances, and mloU, which reflects the average maximum IoU across all instances. LayoutSAM-cval comprises 5, 000 prompts for comprehensive L2I evaluation. We adopt the MiniCPM-based (Yao et al., 2024) inquiry protocol to evaluate spatial, color, texture, and shape attributes. Furthermore, we report FID, PickScore (Kirstain et al., 2023), and IS to measure the general image quality.
Implementation Details. We validate our method on both SD-1.5 (Rombach et al., 2022) and SD3-mid (Esser et al., 2024). Conducting paired image-text training with LayoutS while un- paired RL. with COCO-2014-MIG, The optimized model is evaluated on COCO-MIG to check OOD generalization while assessed on LayoutSA.M-ew -vaal to check basic 1.11 capability. For SD-11. variant, please check our supplementary for details. For SD3-based variant, we deploy our IDM and IDC on every joint-attention transformer layers. During training, we set training batch size 128, RL batch size as 16, and optimize the model for 20 epochs with a learning rate of 1 x 10-4, which will take days on 8 NVIDIA H20 GPUs Note that RL will not be included into training for the first 10 epochs. During RL, the critic-net and actor-net are optimized in a GAN-like manner. Critic-net will be optimized for times before the collaboration with actor-net. During the inference, we set classifier free guidance scale as 7.5. We follow the inference-time trick used in (Zhang et al., 2025b), activating IDM for the first 30% time steps during inference, while conduct vanilla joint attention for the rest 70%. We set X Av = I and (= 1 x 10-4 4 We set training resolution to 1024 x 1024 for SD3. We set maximum number of instances as 10 for each training sample, i.e., only the first 10 instances will be kept during training. Samples with less than 10 instances will be padded with zero for both text and layout embeddings.
4.2 COMPARISON WITH STATE- OF-THE-ARTS
Results on COCO-MIG. We first report the results on COCO-MIG to check L21 generalization. As shown in Tab. 1, we have two conclusions. (1) Previous methods are struggling in adapting to novel L2I scenes. Specifically, InstanceDiff (Wang et al., 2024) and Creati-Layout (Zhang et al., 2025b) are trained with detailed instance captions, which is different from MIGC (Zhou et al., 2024b) that is trained with short captions. As COCO-MIG assigns short captions for each instance, both
7

---

Published as a conference paper at ICLR 2026
Table Comparison on LayoutSA.-evaal. We compare our method with InstanceDiff (Wang et al., 2024), Ranni (Feng et al., 2023), Be Yourself (Dahary et al., 2024), MIGC (Zhou et al., 2024b), HiCo (Cheng al., 2024a), and CreatiLayour (Zhang al., 2025b).
Methods
Spatial f
Color
Texture
Shape
FID
PickScore f
IS 1
Ranni
41.38
24.10
25.57
23.35
27.24
20.49
19.81
Be Yourself
53.99
31.73
35.26
32.75
28.
20.20
17.98
MIGC 85.66
66.97
71.24
69.06
21.19
20.71
19.65
InstanceDiff
87.99
69.16
72.78
71.08
19.67
21.01
20.02
HiCo
87.04
69.19
72.36
71.10
22.61
21.70
20.15
Creati-Layout
92.67
74.45
77.21
75.93
19.10
22.02
22.04
(SD-1.5)
86.95
70.49
73.56
72.30
22.92
21.98
17.95
Ours (SD-3)
93.14
75.37
78.35
77.20
17.21
23.16
22.45
Boctiiff
Instanceliff
MIGC
Creati-Layout
E-DRUID
photo show cesing Southeestt Asian ket, with the ficus oa two womenn The left woman wenring #colorfall headseart Parrec lortg sleeved top, while the right weenan is wearing Prraditional hat and "pink long sleeved top.
SROTDN
NOTDSROMM
NORDSTROM
Promineut "NORDSTROM" displayed top of buildiing with brckkround being sky several Mogeen palles trees
WIPA
Phillio 66
PHILIPS
66
66
?Phillips
black text. Tie smaller
signs
corragated Sonedle
This photograph of city park scene takes under autural light. Through the furches sne can sec the Puus frees and Plarge fountais in the park. The foregroand the image frotume veveral Sprodestrian.
Figure 3: Visualization of generated samples with I-DRUID and state-of-the-arts Dashed red box: wrongly generated. Dashed yellow box: Distorted text.
InstanceDiff and Creati-Layout fail to outperform MIGC, demonstrating the necessity of improving L21 model's generalization. (2) Our method outperforms all previous works in terms of averaged ISR and mloU under both SD-1.5 and SD3 scenario, showing its strong flexibility and L21 generalization. By introducing unpaired data from OOCO-2014-MIG. 1-DRUID learns generate with short captions under the guidance from GDINO and thus enhances its L2I generalization in novel COCO-MIG,
Results on LayoutSAM-eval. We also evaluate our method on LayoutSAM-evas in terms of spatial, color, texture, and shape to show its basic L2I capability. As LayoutSAM-ewt provides image-text pair, we also report the general image quality metrics like FID Tab. counterpart outperforms other methods. SD3 provides better image quality, enabling our method to achieve lower FID scores on LayoutS of-the-arts in all aspects (spatial, color, texture, and shape), demonstrating its efficacy on large-scale L21 benchmark. It should be noted that MIGC (Zhou et al., 2024b) fails to achieve better results than InstanceDiff (Wang et al., 2024) and Creati-Layout (Zhang et al., 2025b), demonstrating the significance improving L2I's generalization to different scenes, especially when there is large domain gap between training and testing sets.
obvious our SD3
Moreover, our method also outperforms other state-
8

---

Published as a conference paper at ICLR 2026
Table 3: Ablation study. We report the averaged ISR on COCO-MIG and four main criteria on
Layou to check each component's efficacy, Green and red are best and second best results.
Attributes
COCO-MIG
LayoutSAM--on eval
No.
IDM RL-PPO RL-GRPO SFT
Avg ISR
Spatial
Color
Texture
Shape
1
X
X
56.82
86.96
71.08
73.20
72.44
2
X
X
57.64
88.53
73.12
74.93
75.62
3
V
X
X
66.92
89.75
72.21
74.33
73.19
4
V
X
V
61.64
92.86
74.67
78.65
75.80
5
V
60.23
91.47
72.45
74.79
72.93
6
V
X
X
62.75
93.14
75.37
78.35
77.20
4.3 VISUAL COMPARISON
visualize some L2I results and compare them with state-of-the-arts for an intuitive evaluation. As shown in Fig. 3, we conclude that (1) Our method achieves better image quality for L2I. Specifically, our method and SD3-based Creati-Layout generate human face and characters with high quality, while other methods like MIGC fail achieve this. (2) Our method achieves better detail interpretation. e.g., at the third row of Fig. 3, I-DRUID correctly rendered the given instance prompt "Phillips brand", while Creati-layout additionally generates "66", which is potentially caused by the attribute leakage problem. By explicitly considering the instance-disentarg alleviated. Therefore, we demonstrate that our method show competitive capability in L2I.
representation for L2I, the problem is
4.4 ABLATION STUDY
Ablated Visual Comparison. We gradually add each component our proposed method and check the direct influence of each component the final gen- erated images through visualization. The results are shown in Fig. 4. From the visualization, attribute leak- age could be easily found when solely using Liam for optimization (at the column of "Lidm") "). Specifically, at the first row of Fig. 4, "a brown hot dog" is mistak- enly generated as red hot dog due to nearby instance prompt "a red hot dog". Similar results could also be noted the second row, where the "white chair" is mistakenly generated as "a brown chair". After introducing Ldias the attribute leakage problem alleviated as all instances are generated in their corre-
Lig-
tha
Ajulor and sakrel and ckitechurr
Figure Ablated Visual Comparison. sponding locations. Moreover, after adding Lyt, the L2I model achieves fine-level attribute control for instances. e.g., at the second row, the instance "a brown chair" adjust its color align with the attribute "brown".
Effectiveness of using IDM. To check the efficacy of each component, also conduct ablation study on COCO-MIG and LayoutSAM-eva and report the results in Tab. We mainly check four attributes of our method. "IDM": using our IDM+IDC to disentangle features during L2I; "RL-PPO": PPO fine-tuning: "RL-GRPO": replacing PPO with GRPO. GRPO does not require critic net for advantage estimation, but needs sample multiple trajectories the same time; "SFT": We annotate COCO-20 2014-MIG with stanza (Qi et al., 2020) and GDINO in an offline manner while integrating the annotated data into training. Note that SFT introduces images from COCO-2014-00 MIG for joint optimization and thus cannot be conducted with unpaired data. By comparing "No. 1" and "No. 2",, we found that introducing IDC beneficial improving 1.21 accuracies on both benchmarks, yielding improvement L21 accuracies all metrics. The results indicate IDC's effectiveness in generating multiple samples with accurate attribute and layout.
Effectiveness of RL. By further comparing the results between "No. 1", "No. 3", "No. 4", "No. 5" and "No. 6", we conclude RL is beneficial to improving L2I accuracies. Specifically, we only adopt unpaired prompt-only data from 20CO-2014-MIE in our RL-based experiments, but the results on COCO-MIC demonstrate RL's efficacy in improving L2I model's generalization with variant in "No. 3" is optimized with paired data unpaired data. It should also be noted that variant

---

Published as a conference paper at ICLR 2026
65
Averapd (58
60
Reward
0.5
-33
0.5
0.8
10
12
13
14
15
16
17
18
19
20
Ada
Art
Epoch
Figure 5: Sensitivity to Xais and A, Adis 0.
"Blue
Figure 6: Reward stability.
dashed line": Result when Art
from COCO-2014-MIG and violates our experimental setting that requires unpaired data for training. Although "SFT" achieves better results on COCO-MIG, it fails to enhance L21 accuracies under Maveell Therefore, SFT may lead performance degradation due to the domain gap between different training sets. Moreover, introducing RL does not lead to performance degradation, which further demonstrates RL's efficacy.
Effectiveness of using Different RL scheme. We also compare paseb RL with GRPO- based (Shao al., 2024; Liu et al., 2025a) RL. By comparing "No. 4" and "No. 6", note that using different RL strategies yields similar improvement over baseline. The results demonstrate the efficacy of RL even when different RL strategies. Moreover, GRPO-based counterpart requires sampling several trajectories at the same time to obtain group-level advantage, significantly increasing the training time cost. As advantage in our solution obtained through weightee critic-net, training under our scheme is much faster than GRPO-based method. Our method achieves training speed of nearly 15s iter on NVIDIA H20, while GRPO-based method needs more than 60s / iter when setting group size to 4 and conducting RL on full time steps.
4.5 SENSITIVITY ANALYSIS
Sensitivity to Adis and Ayt. We first check model's sensitivity to hyper-parameters Adis and X with averaged ISR over all instance-levels. At this experiment, we change Adia / Xxt from 0.5 to while keeping A / Aais at 1. The results are shown in Fig. 5, we visualize baseline method's ss averaged ISR (results of "No.1"
Tab. 3) with blue dashed line. From the results, we draw two conclusions. (1) Specifically, we note in Fig. 5 that changing
Our method is not sensitive to both these two parameters does not bring significant degradation on performance and the best results achieved when setting Adis Xx1 1. (2) Adis plays important role in 1.21 algorithm. Specifically, we note performance degradation than baseline when setting Adis = 0.5 and X = 1. However, setting Adi = 1 and Xnt 0.5 will not cause such problem. We speculate disentangling process brings better representation for L21, generalizing model in a more efficient manner as correct samples' trajectory could be easily collected than models without disentangling.
Reward stability. We also check the reward stability during the RL process. As RL is introduced after the 10-th training epoch, we compute the averaged reward for sampled trajectories and visualize them in Fig. 6. From the figure, we conclude that although RL may encounter fluctuation during training, the training process is generally steady.
5 CONCLUSION
This paper advances layout to image (I.21) generation by addressing attribute leakages and generaliza- tion. For the first challenge, we devise instance disentanglement module (IDM) and utilize instance disentanglement constraint (IDC) for disentangling remantic-related features. IDC requires semantic- related features to yield more accurate attention maps than spurious ones, and thus avoids attribute leakage. For the second challenge, we formulate reinforcement learning (RL) framework, enabling our method to learn with unpaired prompt-only data to improve model's generalization. These two contributions formulates our I-DRUID and achieves high accuracies several benchmarks.
Acknowledgments. We acknowledge the support from Shanghai Municipal Commission of Economy and Informatization, under Grant No. 2024-G71--G22--00000
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Alekh Agarwal, Nan Jiang. Sham M Kakade, and Wen Sun. Reinforcement learning: Theory and algorithms. CS Dept. UW Seattle, Seattle, WA, USA, Tech. Rep, 32:96, 2019.
Michael Albergo, Nicholas M Boffi, and Eric Vanden-Eijnden. Stochastic interpolants: unifying framework for flows and diffusions. arXiv preprint arXiv:2303.0879) 2023.
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, al. Constitutional Harmlessness from ai feedback. arXiv preprini arXiv:2212.080773 2022.
Kevin Black, Michael Janner, Yilun Du, Ilya Kostrikov, and Sergey Levine. Training diffussion models with reinforcement learning. arXiv preprint arXiv:2305. 13300, 2023.
Chaofeng Chen, Annan Wang, Haoning Wu, Liang Liao, Wenxiu Sun, Qiong Yan, and Weisi Lin. Enhancing diffusion models with text-encoder reinforcement learning. In Proceedings of the European Conference on Computer Vision, 2024.
Bo Cheng, Yuhang Ma, Liebucha Wu, Shanyuan I. Liu, Ao Ma, Xiaoyu Wu, Dawei Leng, and Yuhui Yin. Hico: Hierarchical controllable diffusion model for dayout-toimmae generation. In Advances in Neural Information Processing Systems, 2024a.
Jiaxin Cheng. Zixu Zhao, Tong He. Tianjun Xiao, Zheng Zhang, and Yicong Zhou. Rethinking the training and evaluation of rich-context layout-to-image generation. Advances in Neural Information Processing Systems, 2024b.
Guillaume Couairon, Marlene Careil, Matthieu Cord, Stéphane Lathuiliere, and Jakob Verbeek. Zero-shot spatial layout conditioning for text-to-image diffusion models. In Proceedings of the International Conference Computer Vision, pp. 2174-2183, 2023.
Omer Dahary, Or Patashnik, Kfir Aberman, and Daniel Cohen-Or. Be yourself: Bounded attention for subltc--uubjettttt text-to-inn eext-to-image generation. In Proceedings of European Conference on Computer Vision, pp. 432-448, 2024.
Carles Domingo-Enten Enrich, Michal Drozdzal, Brian Karrer, and Ricky TQ Chen. Adjoint matching: Fine-tuning flow and diffusion generative models with memoryless stochastic optimal control. arXiv preprint XXiv:2409.0886), 2024.
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. In Proceedings of International Conference on Machine Learning, 2024.
Ying Fan, Olivia Watkins, Yuqing Du, Hao Liu, Moonkyung Ryu, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, Kangwook Lee, and Kimin Lee. Dpok: Reinforcement learning for fine-tuning text-to-image diffusion models. Advances in Neural Information Processing Systems, 2023.
Weixi Feng, Xuehai He, Tsu-Jui Fu, Varun Jampani, Arjun Akula, Pradyumna Narayana, Sugato Basu, Xin Eric Wang, and William Yang Wang. Training-free structured diffusion guidance for compositional text-to-image synthesis. In Proceedings of International Conference in Learning Representation, 2023.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in lims via reinforcement learning. arXiv preprint arXiv:2501.12945 2025.
Jonathan Ho, Ajay Jain, and Picter Abbeel. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems, 2020.
Aaron Jacch, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El- El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai ol system card. arXiv preprint arXiv:2412.1677 16720, 2024.
11

---

Published as a conference paper at ICLR 2026
Junyong Kang, Seohyun Lim, Kyungjune Back, and Hyunjung Shim. Rethinking direct preference optimization in diffusion models. arXiv preprint arXiv:2505.187. 18736, 2025.
Yunji Kim, Jiyoung Lee, Jin-Hwa Kim, Jung-Woo Woo Ha, and Jun- Yan Zhu. Dense text-to-image generation with attention modulation. In Proceedings of the International Conference Computer Vision, pp. 7701-7711, 2023.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, al. Segment anything. In Proceedings of the International Conference on Computer Vision, pp. 4015-4026, 2023.
Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, and Omer Levy. Pick- a-pic: An open dataset of user preferences for text-to-image generation. Advances in Neural Information Processing Systems, 2023.
Frank Klinker. Exponential moving average versus moving exponential average. Mathematische Semesterberichte, pp. 97-107, 2011.
Black Forest Labs. Flux, https: / //gg//tthhhhht com/black serrestlabb/fleee 2024.
Phillip Y. Lee, Tachoon Yoon, and Minhyuk Sung. Groundit: Grounding diffusion transformers via
noisy patch transplantation. Advances in Neural Information Processing Systems, 2024.
Yuheng Li, Haotian Liu, Qingyang Wu, Fangzhou Mu, Jianwei Yang, Jianfeng Gao, Chunyuan Li, and Yong Jae Lee. GLIGEN: open-set grounded text-to-image generation. In Proceedings of the Conference on Computer Vision and Pattern Recognition, pp. 22511-225211 2023.
Zhimin Li, Jianwei Zhang, Qin Lin, Jiangfeng Xiong, Yanxin Long, Xinchi Deng, Yingfang Zhang, Xingchao Liu, Minbin Huang, Zedong Xiao, et al. Hunyuan-dit: A powerful multi-resolution diffusion transformer with fine- gne-grained chinese understanding. arXiv preprint arXiv:2405.08748. 2024.
Jie Liu, Gongye Liu, Jiajun Liang, Yangguang Li, Jiaheng Liu, Xintao Wang, Pengfei Wan, Di Zhang, and Wanli Ouyang. Flow-grpo: Training flow matching models via online reinforcement learning. arXiv preprint arXiv:2505.05470. 2025a.
Jie Liu, Gongye Liu, Jiajun Liang, Ziyang Yuan, Xiaokun Liu, Mingwu Zheng, Xiele Wu, Qiulin Wang, Wenyu Qin, Menghan Xia, et al. Improving video generation with human feedback. arXiv preprint arXiv:2500 13918, 2025b.
Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Qing Jiang, Chunyuan Li, Jianwei Yang. Hang Su, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. In Proceedings of European Conference on Computer Vision, pp. 38-55, 2024.
Zhengyao Lv, Tianlin Pan, Chenyang Si, Zhaoxi Chen, Wangmeng Zuo, Ziwei Liu, and Kwan- Yee Wong. Rethinking cross-modal interaction in multimodal diffusion transformers. In Proceedings of the International Conference on Computer Vision, 2025.
Chong Mou, Xintao Wang, Liangbin Xie, Yanze Wu, Jian Zhang, Zhongang Qi, Ying Shan, and Xiaohu Qie T2i-adapter: Learning adapters dig out more controllable ability for text-to-imaae diffusion models. In AAAI Conference on Artificial Intelligence, 2023.
Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advintage-weightte regresssioi regression: Simple and scalable off- off-policy reinforcement learning. arXiv preprint 19..0:1910.000777 2019.
Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher Manning. Stanza: python natural language processing toolkit for many human languages. arXiv preprint arXiv:2003.07082, 2020.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In Proceedings of International Conference on Machine Learning, pp. 8748-8763. PMLR, 2021.
12

---

Published as a conference paper at ICLR 2026
Rafael Rafailov, Archit Sharma, Erie Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 2023.
Rajkumar Ramamurthy, Prithviraj Ammanabrolu, Kianté Brantley, Jack Hessel, Rafet Sifa, Christian Bauckhage, Hannanch Hajishirzi, and Yejin Choi. reinforcement learning (not) for natural language processing: Benchmarks, baselines, and building blocks for natural language policy optimization. arXiv preprint arXiv:2210.01241, 2022.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ily Sutskever. Zero-shot text-to-image generation. In Proceedings of International Conference Machine Learning, pp.
Pmlr, 2021.
Tianhe Ren, Shilong Liu, Ailing Zeng, Jing Lin, Kunchang Li, He Cao, Jiayu Chen, Xinyu Huang, Yukang Chen, Feng Yan, Zhaoyang Zeng, Hao Zhang, Feng Li, Jie Yang, Hongyang Li, Qing Jiang, and Lei Zhang. Grounded sam: Assembling open- open-world models for diverse visual tasks. arXiv preprint arXiv:2306.0355 2024.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the Conference on Computer Vision and Pattern Recognition, pp. 10684-10695, 2022.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-assisted Intervention, pp- 234-241, 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 1707006677 2017.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, al. Deepseekmath: Pushing the limits of mathemat- ical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024.
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. In Advances in Neural Information Processing Systems, 2020.
Andrey Voynov, Kfir Aberman, and Daniel Cohen-Or. Sketch-guided text-to-image diffusion models. In ACM SIGGRAPH Conference Proceedings, pp. 1-11, 2023.
Bram Wallace, Meihua Dang, Rafael Rafailov, Linqi Zhou, Aaron Lou, Senthil Purushwalkam, Stefano Ermon, Caiming Xiong, Shafiq Joty, and Nikhil Naik. Diffusion model alignment using direct preference optimization. In Proceedings of the Conference on Computer Vision and Pattern Recognition, pp.
2024.
Xudong Wang, Trevor Darrell, Sai Saketh Rambhatla, Rohit Girdhar, and Ishan Misra. Instancediffu- sion: Instance-level control for image generation. In Proceedings of the Conference on Computer Vision and Pattern Recognition, pp. 6232-6242, 2024.
Yiping Wang, Qing Yang. Zhiyuan Zeng, Liliang Ren, Liyuan Liu, Baolin Peng. Hao Cheng, Xuehai He, Kuan Wang, Jianfeng Gao, et al. Reinforcement learning for reasoning in large language models with one training example. arXiv preprint drXiv:2504.0571,220 2025.
Jinheng Xie, Yuexiang Li, Yawen Huang, Haozhe Liu, Wentian Zhang, Yefeng Zheng, and Mike Zheng Shou. Boxdiff: Text-to-image synthesis with training free box-constrained dif- fusion. In Proceedings of the International Conference on Computer Vision, pp. 7452-7461, 2023.
Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong. Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. Imagereward: Learning and evaluating human preferences for text-to-image generation. In Advances in Neural Information Processing Systems, 2024.
13

---

Published as a conference paper at ICLR 2026
Xue Yan, Yan Song, Xidong Feng, Mengyue Yang, Haifeng Zhang, Haitham Bou Ammar, and Jun Wang. Efficient reinforcement learning with large language model priors. In International Conference in Learning Represenation, 2024.
Zhengyuan Yang, Jianfeng Wang, Zhe Gan, Linjie Li, Kevin Lin, Chenfei Wu, Nan Duan, Zicheng Liu, Ce Liu, Michael Zeng, et al. Reco: Region- text-to-image generation. In Proceedings of the Conference on Computer Vision and Pattern Recognition, pp. 14246-14255, 2023.
Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, et al. Minicpm-v: A gpt-4v level milm on your phone. arXiv preprint arXiv:2408. 01 01800, 2024.
Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang. Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models. arXiv preprint arXiv:2308.007211 2023.
Huizhuo Yuan, Zixiang Chen, Kaixuan Ji, and Quanquan Gu. Self-play fine-tuning of diffusion models for text-to-image generation. In Advances in Neural Information Processing Systems, 2024.
Hong Zhang, Zhongjie Duan, Xingjun Wang, Yingda Chen, and Yu Zhang. Eligen: Entity-level controlled image generation with regional attention. AAAI conference on artificial intelligence, 2025a.
Hui Zhang, Dexiang Hong, Tingwei Gao, Yitong Wang. Jie Shao, Xinglong Wu, Zuxuan Wu, and Yu- Gang Jiang. Creatilayout: Siamese multimodal diffusion transformer for creative layout-to-image generation. In Proceedings of International Conference in Computer Vision, 2025b.
Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control text-to-image diffusion models. In Proceedings of International Conference on Computer Vision, 2023.
Tianyi Zheng, Jiayang Zou, Peng-Tao Jiang, Hao Zhang, Jinwei Chen, Jia Wang, and Bo Li. Bidirec- tional beta-tuned diffusion model. IEEE Trunsactions on Pattern Analysis and Machine Intelligence, pp. 1-15, 2025. doi: 10. 1.0/TTTA...222..6000000
Dewei Zhou, You Li, Fan Ma, Zongxin Yang, and Yi Yang. Migc++: Advanced generation controller for image synthesis. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024a.
Dewei Zhou, You Li, Fan Ma, Xiaoting Zhang, and Yi Yang. Mige: Multi-instance generation controller for text-to-image synthesis. In Proceedings of the Conference on Computer Vision and Pattern Recognition, pp. 6818-6828, 2024b.
Zikai Zhou, Shitong Shao, Lichen Bai, Zhiqiang Xu, Bo Han, and Zeke Xie. Golden noise for diffusion models: A learning framework. arXiv preprint arXiv:241 02024/1.09502 2024c.
14

---

Published as a conference paper at ICLR 2026
APPENDIX
Algorithm The Process of I-DRUID.
Inputs:
Sample with b bounding boxes, global prompt Pg and instance captions p. Reward model r(-). critic-net, actor-net, net, hyper-paramenn Adis, Art, training epochs E, action pool MP, unpaired prompt (Pa, bu, Pg.a
1: // Disentangling Stage.
2: Initialize L21 model with SD3-mid;
3: for in E do
4: Sample a batch of training data (bes Pe. Pg):
5: Sample time t:
6: Obtain R with IDM and compute Eq. 4;
7: Compute Eq. 11 with R+ and time l;
8:
// RI.
9:
i 2 10 them;
10:
Sampling trajectory with unpaired data {Pus bus Po.ut and enqueue actions into MP
(only enqueue actions for the first 20% time steps);
11:
Obtaining generated images Xo:
12:
Obtaining reward with Eq. 6;
Computing Eq. 7 and optimize critic-net; Computing advantage scores with Eq. 8;
14:
Sampling old actions of the same input from MP;
16:
Obtaining policy for old and current actor; Computing Eq. 10 obtain L,,:
18:
else
19:
Setting L,j to 0; end if
20:
21:
Optimizing L21 model through backpropagation with Eq. 12;
22:
end for
LLM Statement. use LLM to polish the writing, such as correcting grammar and other errors.
A OVERALL TRAINING PIPELINE
We demonstrate overall training pipeline in Alg. 1. Our training contains two parts, i.e., disen- tangling stage and RL stage. The former will be conducted for the whole training process to ensure basic L2I capability while the latter will be utilize only for the last 10 epochs to improve L21 model's generalization with unpaired data. It should be noted that action pool MP will be activated only for the last 10 epochs and will only save the actions for the first 20% time steps for faster RL. Moreover, during RL, actor- and critic-net will be optimized in a GAN-like manner. critic-net will be optimized 5 times before joining actor-net's optimization.
B IMPLEMENTATION DETAILS FOR UNET-BASED METHOD
Our method is also compatible with UNet-based diffusion models like SD-1.5. When transferring to SD-1.5, (1) IDM's location for deployment, (2) the objective for noise prediction, and (3) RL actions should be modified. For (1), we deploy IDM at the x 8 mid-layer and 16 x 16 decoder layers of UNet to conduct feature disentanglement and subsequent cross attention. For other cross-attention layers without IDM, we optimize them with global prompt. For (2), as SD-1.5's training objective is noise prediction, Lidm should be changed (Rombach et al., 2022) accordingly. Finally, for (3), the actions for SD-1.5 follows original transitional probability formula (Ho al., 2020).
C FAILURE CASE STUDY
We visualize some failed cases in Fig. to show its limitation. Specifically, our method is liable to generate flawed cases when small characters and human body emerges. However, the generation with
15

---

Published as a conference paper at ICLR 2026
KNITING BOITIQUE
Cueck
"wommmm walking on 'cobblestone strcet, Behind her is bluc shop front, with Preoon that reads KNITTING BOUTIQUE.
Trafle signs that displays the street names CaCuumbb Circle' and Breadwayy,
"ONE
Figure 7: Visualization of generated failure cases.
Bescline
IoUl Only
IoU Conf
IoU Conf
Instancelhir
black potted plant and white TV
blue cal and red couch
Figure 8; Justification of Reward Design.
small-scale text is known issue in original stable diffusion (Esser al., 2024), which will bring similar negative effects on all SD-based AIGC algorithms (Zhou et al., 2024b). To mitigate these negative effects, users could collect more data with text and human bodies to finetune the model.
We also note that previous SD1.5-based MIGC Zhou et al. (2024b) achieves better results than SD3- based Creati-Layouu Zhang et al. (2025b) COCO-MIG in terms of "Avg ISR". COCO-MIG is benchmark highly focused on evaluating attribute leakage and rendering of each instance, suggesting weaker instance-level rendering capability for SD3-based L2I. We speculate the phenomenon caused by the inherent problem in AMMattention, where text tokens are diluted in MM-attention Lv al. (2025). Therefore, as MM-attention heavily used SD3, improving SD3-based L2I models generation capability in correctly rendering instance attributes becomes challenging. The problem further leads the relatively lower 1.21 accuracies on COCO-MIG.
D JUSTIFICATION OF REWARD DESIGN
As demonstrated in our experiments,
Table 4: Quantitative Results for Reward Justification.
we set the reward as "IoU Confi-
ISR
1.2
L.3
1.4
1.5
1.6
Avg
dence" to jointly consider spatial accu-
Baseline
63.30
61.04
57.46
52.20
50.10
56.82
racies and instances' generation qual-
+ loU
70.37
65.48
59.21
51.26
51.14
59.49
ity. To explore the optimal reward de-
Conf
76.87
69.16
62.96
52.37
52.39
62.75
sign, we conduct experiments by set- ting different weights to "Confidence"
5Conf
72.50
67.29
61.09
51.25
52.79
60.98
term, and visualize generated images
for different variants. The results shown in Fig. 8, also visualize the results from InstanceD- iff Wang et al. (2024) for comparison. From the result, we draw two conclusions. (1) IoU term RL enables better spatial control. Specifically, the "blue cat" second row does not follow the given
16

---

Published as a conference paper at ICLR 2026
Table RL for initial states,
Table Robustness False Reward on COCO-MIG.
Method
Avg ISR
ISR
L2
L3
L4
L5
L6
Full-Time Sampling
63.06
w/o False Reward
76.87
69.16
62.96
52.37
52.39
Ours
62.75
w/ False Reward
67.18
66.25
57.34
51.20
51.92
2
2
the foreground, figore dreesee exquisite traditional attire stands by the water. In the background, the Iconic Venetian buildings and Mgoadolas dearty visible, with the shimmering under the sunlight.
This is photo of woman sitting on the reof vehicde in desert environment, The dressed black belge wearing pair grey hiking sieting vehichs.
The tempple located in tropical eavironment, with trees growing and walls, the covered with paved
The man dressed a black suit, while the woman is wearing blue evening with design. They sitting table, with background.
The officer the left wearing black hat. The officer the right also They standing yellow pollice van, Big Ben surrounding buildings backgrouad
child middle holding hand, young focused on her dancing.
oversized right
Figure 9: Visualization of more cases (Part-I).
layout when RL is not applied. After using "loU-based" RI., the spatial control further improved. (2) Introducing confidence score into RL further improves instance-level generation, but may lead to low image quality, Specifically, ignoring "confidence term" will lead to the unnatural generation of some instances (e.g., green cat with black tail, or blue cat with multiple legs). These artifacts are alleviated after taking "confidence term" into RL. However, when changing the reward function to "IoU 5 confidence", note verrsaturation for some generated instances. We thus adopt "loU + confidence" as our final reward. The guantitative results on COCO-MIG at Tab. 4 demonstrate the same conclusion.
17

---

Published as a conference paper at ICLR 2026
The foreground features winting river, lined with hare trees, In the bockground, mosatains forum
The man stands in front of pedium, wearing dark many uniform with badges. the backgrousd. painting of sailbout long on the wall, next several decorative paintings and
white
STAR WARS
M
Docolde Ti
background
bage Wars
The bedream equipped with deuble bod, eovered with blue blanket. Beside desk There blue
carpet.
The center of the image is bust road, with vehlcdes vartous colors and modeds driving on it, Ou both sides the road are medern lighorise buildings The sky filled with thick cloads.
centerplece image grey carpet of the photo features a window.
The background
Figure 10: Visualization of more cases (Part-II).
E RL AT EARLY TIME STEPS
We only conduct RL on the first 20% time steps speed up the training process, raising the concern whether RL on initial states yields similar improvement on L2I trajectory sampling with full time steps (we term this variant full-time sampling). We conduct experiment with full time trajectory sampling and evaluate 1.2I model on COCO-MIG to demonstrate this. As shown in Tab. 5, using full-time sampling yields slight improvement but requires higher computational cost as optimization will be conducted on the whole trajectory. Specifically, with RL training at early time steps, our method achieves training speed of nearly 15.4s iter NVIDIA H20, while the training speed with full trajectory sampling is nearly 82.6 s iter. We thus propose to optimize initial states for faster RL.
F MORE VISUALIZATION
We visualize results in Fig. 9 and Fig. 10 to show method's efficacy in L2I. Specifically, method achieves both spatial correctness and high instance image quality, demonstrating its potential in 1.1 field.
18

---

Published as a conference paper at ICLR 2026
G ROBUSTNESS TO UNRELIABLE REWARD
We also conduct experiment to check our method's robustness when encountering false reward by deliberately assigning false reward (minus of the reward). The results shown Tab. 6.
From the result, we conclude that our RL part suffers from performance degradation when wrong rewards are given, but the performance degradation does not cause catastrophic false generation. This is due to two reasons:
(I) Joint optimization with paired data and KL terms. This is commonly used strategy to overcome training instability during RL Stiennon et al. (2020); Fan et al. (2023); Black et al. (2023). By using KL term and keeping original pair-wise training loss, our method mitigates the negative effects brought by RL. Moreover, we only conduct RL at the last 10 epochs of training, where our model already strong capability for L21 generation. During trajectory sampling, the generated images will not introduce samples with completely false layout, alleviating the negative effects brought by false reward.
(2) Feature Disentanglement. As demonstrated in Tab. 3, after using IDM for RL, the final L2I accuracies are improved. Therefore, our IDM provides better representation for subsequent RL.
To further alleviate the negative effects brought by false reward, one combine multiple reward functions (combining multiple detectors, etc.) to improve reward robustness, or carefully adjusting KL weights strike balance between aggressive exploration or stable training.
H PSEUDO CODE OF OUR I-DRUID
We provide the pytorch-like pseudo code for our disentangling module and RI training. Specifically, "CAS score computation" is the function to compute CAS score. "Compute Lais" is the function compute Lais. "Compute L,," is the function compute Lyt, while "SDE-ODE Conversion" is conducted on original "FlowWaachingSShhedlee"" of SD3 to implement Eq. 5.
CAS score computation
def _get_bg_attn_lose (self, vorse_attn, bg_mask)
BPN, heads, HW, ori ori_attnmap. B shapek bg mask = bg mask reshape (-1, 1, HW)
get ori bg attn may ori_attmmap = torch. sum (ori_attnmapl:: ;, ;, 1:1, dim=-1)
ori ori_at = ori ori_attnmap reshapo (B, -1, HW)
bg_mask bg_mask. reshape 1, HW)
get general bg info bg_attn_meat meatn__mean__ = (ori_attnmap: * bg_mask). .um
sum (dim--1) /
(bbg_mask) sum (dim=-1) + 1e-6) # (B, PN-heads)
loss attn (abs (ori attnmap bg_attn mean Nonel detach bg mas_m sum (dim=-1) (bg mask. sum (dim=-1) + 1e-6)
return loss_attn. mean ()
Compute Ldis
def _ost__diicn_oosseee (self, worse_attn, better_attn, masks):
Lanking_loss torch. H_ori = t www.mm.ottarrgnlsssslll...
(worse_attn, masks)
H_better = self la_______________________ loss (better_attn) masks) y torch. like ori))
return ranking Joss www.orriieeeter...eeee y)
Compute Tue
19

---

Published as a conference paper at ICLR 2026
def
ri (sel,,,,, state_dict, chosen_ pid, height, width, clip_range=le-44:
cur_glob Istate_dict ["caption"] [pid] for pid in chosen_pid]
sub_prts Istate_dict (rrp__ccccccsuuu__prmmttsss/s [pid]
[pid] for pid in chosen_pid]
sub bbox [cate__dct("Loo_prad"]]] (chosen_______pppp [chosen _id, :]
for pid in chosen_pid)
reg_logp = state_
cur pur_legp, cur_reward, infer_ing = self se.f. pipe.traing traie..rain_get_lgpp
cur_glob, height, width, nun_inferenne_step==ee max_objs 10, timesteps None,
bbox_phtaaco_euibbrtt,,, but_aawwwbbbbo.. ooward_funn-ssslff.eerrrr___ load_image=self.1oddd
topk-10
topk = ref teip.ogg..she shapelll
our adv = cur cur_ to
to Legp.device) device) vvex (-1,1) -\
self. funa (infer
ratio torch. exceheexp(cr (xurr__ogc :topk]
ref ref_logp.tor to unclipped_los loss adv ratio
clipped_ _oss = -cur_adt _dv terch.clang
ratic, 1.0 elip_range, 1.0 clip_range,
return torch. mean (torch. @oorcch.mximmmmuneee... tttechh...iiiiuuuu.ipppe....... cosss) +\
self.kl_loss (cur_ ref_logp)
ODE-SDE Conversion
def add_sttp_witth_loprro
self, model_oottpln timestep, sample 0.7, prev_sample None, generater None,
sample=sample. ()
if prev_sample is not None: seey_ampleeprew.eemmee....
step_index
hetnnddeffo_ttimtteppppppe (t). for in timestep] prev_step_index [step+1 for step in step_index]
sigma self, sigmas [stes_indeetlel_e_____ee indexiie.ii... .vie(
-1, ([1] (len (sample. 1))
sigma_pros prev self.signee (wigmassproysstop_inneeeooo view
-1, ([1] (len @aaampll.shhppe 1))
sigma max self.sigme [1].item[] dt si sigma_prev - sigma
std_dev_t torch. sgrt (si gma /
torch. where (sigma 1, sigma max, sigma) knoise
sce
prev_sample_mean - sample* (evstt_dev_tt++) (2+si (mma) ddt) +\
model_output= (1+std_dev__tat (11=*****11s11100011111 (1-sigma) / s gma) +dt
if prev_sample is None:
variance_noiie noise randn_tensor.
sodel_output.sharee
generator-gemeratoor device-modelLoutput..doo device, dispe-modelLotpptt.dyy
prev_sample prev_sample_mean +
std_des torch. torch.sqrt variance_noise
20

---

Published as a conference paper at ICLR 2026
log_prcb
- -(pprov_sampee.dee detach () - prov_sample__mean 2) /
(2 ((std_dev_t torch. (-=
torch.log(std_ dev_t tarch.sqrt
terch. (torch. sqrt torch.as_tensor (math.pi))
prcb = wog.prob.meat mean (dim=tuple (range (1,
_og_prop pww.proo.mee ())m).
return prev_sample, log_prob, prev_sample_nent std_dev_t
def compute_log_prob (self, noise_pred, sample, time):
prev prev_ see__s log prob, prev_sample mean, std dev self.sde_step witp_wwthh_ae 1ogprobb
noise noise_pred.float float (), time, (mmple.loaa(() prov_eampic-None, noisc_levcl=0.7,
return prev_sample, log_breb, prov_sample_nean, std_dev_t
21


Reviews:
[{"review_id": "hTbNIUvEpr", "reviewer": "Reviewer_WrbM", "summary": "This paper introduces I-DRUID, a novel framework for Layout-to-Image (L2I) generation that addresses the critical challenges of attribute leakage and limited generalization. To prevent attribute leakage between instances, it proposes an Instance Disentanglement Module (IDM) that isolates semantic features for more precise attention control. To enhance generalization to novel scenes, the framework uniquely leverages reinforcement learning (RL) with unpaired, prompt-only data, enabling the model to learn from diverse AI feedback without requiring expensive paired image-text datasets. The method demonstrates state-of-the-art performance across multiple benchmarks and shows high flexibility by successfully adapting to both UNet and MM-DiT architectures.", "questions": "See Weakness.", "limitations": "", "rating": 4, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. The paper introduces a creative combination of instance-level disentanglement and reinforcement learning (RL) to address challenges in layout-to-image (L2I) generation.\n2. The paper is logically structured and the language is fluent.\n3. A comprehensive performance comparison was conducted against several recent SOTA methods, achieving leading results on multiple metrics and fully demonstrating the superiority of the proposed approach.", "weaknesses": "1. Using Grounding-DINO as a fixed reward model introduces bias and limits generalization. The paper does not examine robustness when GDINO fails to detect objects or misinterprets captions.\n2. The framework is too complex and maks this work hard to follow. The I-DRUID framework integrates multiple complex components. While its performance is superior, its high complexity may pose challenges for researchers attempting to reproduce and build upon this work.\n3. While synergy is claimed, the mechanism of mutual benefit (how disentangled features improve RL stability and vice versa) is discussed qualitatively but not empirically isolated. Table 3 does not show results with only RL applied (without IDM). It's better to provide these results to support the qualitatively analysis.\n\n4. Minor issue: Some notations (e.g., R+, R−, CAS) could be made clearer.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "g8U4zMjaBs", "reviewer": "Reviewer_h5RF", "summary": "The paper proposes I-DRUID, a framework for layout-to-image (L2I) generation that combines reinforcement learning and instance disentanglement learning to enhance generalization and spatial consistency. The proposed method eliminates attribute leakage between instances by introducing the Instance Disentanglement Module (IDM) and Instance Disentanglement Constraint (IDC), which separate semantically relevant and irrelevant features. A Reinforcement Learning (RL) module is applied on top of a diffusion-based generator, trained using PPO optimization on unpaired prompt-only data and guided by feedback from an external grounding model (Grounding-DINO).", "questions": "1. Inconsistency between SD-1.5 and SD-3 results across Tables 1 and 2. The reported results are not consistent between the two backbones. On COCO-MIG (Table 1), SD-1.5 clearly outperforms SD-3 with higher Instance Success Rate (69.13 vs 62.75) and mIoU (68.18 vs 55.60). However, on LayoutSAM-eval (Table 2), the trend reverses, SD-3 achieves markedly better scores (Spatial 93.14 vs 86.95, Color 75.37 vs 70.49, Texture 78.35 vs 73.56, Shape 77.20 vs 72.30, FID 17.21 vs 22.92, IS 22.45 vs 17.95). Could the authors clarify why?\n2. Grounding-DINO is primarily a detection model trained on real images, focusing on spatial grounding rather than perceptual or stylistic quality. When used as a reward function for diffusion RL, its outputs may be biased toward positional correctness while ignoring visual realism and may also be unreliable on synthetic generations. How do the authors mitigate these biases or ensure that the reward signal remains stable and meaningful during RL training?\n3. Could the authors provide runtime or FLOPs comparison to quantify efficiency versus baselines?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ Original combination of disentanglement and RL in diffusion-based L2I generation.\n+ Experimental results demonstrated improved spatial alignment and instance fidelity.\n+ The use of unpaired data is practical and reduces annotation costs.\n+ Ablation studies show evidence of reduced attribute leakage.", "weaknesses": "- It is questionable to employ Grounding-DINO as a reward evaluator. Although it excels at object detection, it was trained for spatial grounding on real images rather than perceptual or stylistic quality. As a result, the reward may be biased toward positional correctness, unstable on synthetic generations, and insensitive to visual realism, leading to inconsistent RL behavior across datasets.\n- No quantitative analysis of computational cost, which is critical given the RL overhead.\n- Evaluation metrics mostly focus on geometric accuracy (mIoU, ISR); perceptual or semantic quality is not deeply discussed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CNzYJLMmCj", "reviewer": "Reviewer_6mc3", "summary": "This work proposes to use the feature disentanglement and reinforcement learning to improve the performance of layout-to-image generation task. The author designs an instance disentanglement module to separate the image feature and encourages the model to concentrate semantic-related information into R+ while “spurious parts” into R-. Afterwards, this work uses the reinforcement learning with PPO to train the model. The final model is optimized jointly with the diffusion loss, disentanglement loss and RL loss.", "questions": "Please see my weaknesses", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "* The motivation to separate the semantic-related features and spurious features is interesting.", "weaknesses": "* From the abstract, it seems that the purpose of feature disentanglement is for avoiding attribute leakage, is there any evidence that support that attribute leakage has been mitigated after using the feature disentanglement?\n* The reviewer would like the author to clarify  the novelty in the RL part. From the current manuscript, it seems that the method just adopts the PPO and uses a new reward for layout-to-image task. Also I would like the author to quantitatively justify their choice of reward, e.g., why use IoU + confidence, what if change the importance ratio between them. \n* The reviewer has several questions for instance disentanglement part.\n    * ln 196, what does “global-prompt-enhanced features” mean?\n    * if the reviewer understands correctly, the purpose of the disentanglement is to concentrate the semantic-related features in cross-attention region, i.e., increasing the response in R+ while decreasing the response in R-. While from the illustration in Fig 2, it seems that in both building and background regions, the R- has higher response. The reviewer wonders if this observation suggests that the disentanglement is not work as expected.\n    * the reviewer wonders the design rational of Eq 3. the (1-M) part is understandable as noted in ln 218 (evaluates the degree of attention value beyond instance i’s scope.) but why use R+ - AVG(R+)? What is the purpose of computing the difference between each instance response and average response? And what is the point to have the average response?\n    * ln 225-227 mentioned that “Minimizing the above disentanglement loss Ldis enables IDM to accurately separate each instance and thus avoid attribute leakage explicitly.”. why minimizing Eq 4 can lead to instance separation and avoid attribute leakage?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IfgovjCyFk", "reviewer": "Reviewer_uTfS", "summary": "This paper targets for solving two issues in L2I generation, including instance overlapping and poor generation to new scenes. The method it proposed has two components, 1/ the instance-disentanglement module to disentangle semantic-related features and spurious features; 2/ RL module to generate unpaired data for model training. The evaluation on COCO_MIG and LayoutSAM-eval demonstrates that the proposed method achieves the SOTA performance compared with prior work.", "questions": "- Why SD-1.5 has better performance than SD-3 on COCO-MIG and is it because IDM helps one backbone more than the other?\n\n- Is there any overlap or duplicate risk for testing on COCO-MIG since RL uses COCO-2014-MIG prompts for generating unpaired data?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The way of incorporating RL on unpaired data into L2I model training is impressive, which indeed solves the generation limitation of current L2I approaches. The conversion to SDE solves an emerging pain point of exploring flow-matching models.\n\n- The ablation and sensitivity analyses make the paper solid, especially the illustration of module effectiveness.\n\n- The visualization is also good to follow with both good cases and failure cases provided.", "weaknesses": "- IDC is heuristic and does not stand out compared to previous attention ideas. The novelty in CAS-based Softplus inequality is incremental to existing attention regularization in L2I, such as region-aware attention, for example, Rethinking The Training And Evaluation of Rich-Context Layout-to-Image Generation (NeuIPS 2024), which should also be compared in the result tables.\n\n- The reward design (GDINO IoU plus confidence) mainly measures the spatial grounding, ignoring the attribute binding such as color and texture, and text fidelity. This should be improved in the future.\n\n- Comparison fairness: the results mix SD-1.5 and SD-3 while some baselines are UNET-based only. It would be better to use stronger base model such as SD-3 for the baselines if possible.\n\n- The contents are too long. The introduction of experimental settings should be included in the original paper rather than appendix.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)