Control and Realism: Best of Both Worlds in Layout-to-Image without Training
Authors: Bonan Li
Yinhan Hu
Songhua Liu
Xinchao Wang
Venue: ICML 2025 poster (2025)
Keywords: Layout-to-Image generation; Training-free
Primary area: applications->computer_vision
TL;DR: In this paper, a novel training-free method based on the diffusion model, WinWinLay, is proposed by revisiting attention backward guidance and introducing modifications to tackle existing drawbacks.

Abstract: Layout-to-Image generation aims to create complex scenes with precise control over the placement and arrangement of subjects. Existing works have demonstrated that pre-trained Text-to-Image diffusion models can achieve this goal without training on any specific data; however, they often face challenges with imprecise localization and unrealistic artifacts. Focusing on these drawbacks, we propose a novel training-free method, WinWinLay. At its core, WinWinLay presents two key strategies—Non-local Attention Energy Function and Adaptive Update—that collaboratively enhance control precision and realism. On one hand, we theoretically demonstrate that the commonly used attention energy function introduces inherent spatial distribution biases, hindering objects from being uniformly aligned with layout instructions. To overcome this issue, non-local attention prior is explored to redistribute attention scores, facilitating objects to better conform to the specified spatial conditions. On the other hand, we identify that the vanilla backpropagation update rule can cause deviations from the pre-trained domain, leading to out-of-distribution artifacts. We accordingly introduce a Langevin dynamics-based adaptive update scheme as a remedy that promotes in-domain updating while respecting layout constraints. Extensive experiments demonstrate that WinWinLay excels in controlling element placement and achieving photorealistic visual fidelity, outperforming the current state-of-the-art methods.

Control and Realism: Best of Both Worlds in Layout-to-Imagg without Training
Bonan 12
Yinhan Hu
Songhua Liu Xinchao Wang? 2
Abstract
1. Introduction
Recent advances in Text-to-Image (T2I) generation (Rom- bach et al., 2022; Podell al., 2023) have profoundly rev- olutionized the vision landscape, facilitating the synthesis of highly authentic assets from textual prompts, e.g., text- driven Image-to-Image translation (Tumanyan et al., 2023; Parmar et al., 2023; Ruiz et al., 2023; Shi et al., 2024; Tosi et al., 2025) and video generation (Wu et al., 2023; Zhang et 2023; Jiang et al., 2024; Qing et al., 2024; Kwon et al., 2025). Nevertheless, designing comprehensive prompts to meticulously control every aspect of an image can be both intor-intensive and time-consuming, posing challenges for efficient generation workflows. As remedy, Layout-to- Image (L21) models (Xue et al., 2023; Zheng et al., 2023; Chen et al., 2024b; Liu et al., 2024a) have been developed, guiding the process in a desired direction by incorporating user-provided inputs, such as bounding boxes.
Layout-to- -magee generation aims to create com- plex scenes with precise control over the place- ment and arrangement of subjects. Existing works have demonstrated that pre-trained Text-to-Image diffusion models can achieve this goal without training on any specific data; however, they often face challenges with imprecise localization and unrealistic artifacts. Focusing these drawbacks, we propose a novel training-free method, Win- WinLay. At its core, WinWinLay presents two key strategies-NNn-locaa Attention Energy Func- tion and Adaptive Update- that collaboratively enhance control precision and realism. On one hand, we theoretically demonstrate that the com- monly used attention energy function introduces inherent spatial distribution biases, hindering ob- jects from being uniformly aligned with layout instructions. To overcome this issue, non-local attention prior is explored to redistribute attention scores, facilitating objects to better conform to the specified spatial conditions. On the other hand, we identify that the vanilla backpropagation up- date rule can cause deviations from the pre-trained domain, leading to out-of-dist distriibutiion artifacts. We accordingly introduce a Langevin dynamics- based adaptive update scheme as a remedy that promotes in-domain updating while respecting layout constraints. Extensive experiments demon- strate that Win WinWinLay excels in controlling ele- ment placement and achieving photorealistic vi- sual fidelity, outperforming the current state-of- the-art methods.
To acquire L21 models, an intuitive framework is to fine- tune powerful T2I models with with spatial conditioning. However, these approaches (Li et al., 2023; Wu et al., 2024) incurs expensive training cost and faces challenges when collecting resource-intensive paired data. To overcome the aforementioned limitations, existing methods (Couairon et al., 2023; Xie al., 2023) have explored the incorporation of layout guidance during the sampling phase, establishing the training- ffree paradigm for L2I. Among them, backward guidance (Chen et al., 2024d), which combines attention redistribution and backpropagation update rules. has been demonstrated as a promising scheme (Liu et al., 2024a; Chen et al., 2024b). Specifically, attention redistribution encourages cross-attention activation at designated positions via the energy function, while the backpropagation rule di- rectly updates feature maps using corresponding gradients to match the specified layout. However, while these works offer higher efficiency, they fall short of training craneig-bbssed ap- proaches in terms of spatial fidelity and image quality.
To bridge this gap, we initiate our exploration with an in- depth analysis of existing mechanisms and demonstrate that, although effective, they are still prone to suboptimal results in terms of control capabilities and visual realism. Firstly, we theoretically verify that the attention energy function tends to favor patches with higher initial attention within the bounding box during the optimization process, lead-
'Equal contribution 'University of Chinese Academy of Sciences, Beijing, China Natiooal University of Singa- pore, Singapore. Correspondence to: Xinchao Wang <xin- chao@nus.edu.se edu.sg>.
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Control and Realism: Best of Both Worlds in Layout-to-Image without Training
A beggar sitting ow the chair unller glowing Lampppest in vtoppia The beautifull scenery soattered behinal hime Mocoubaiins and Lakess, studio ghibli styls, by Hayao Miyazaki, vévéd colors, sharp angles, playfull SK.
Pilazchu and dog wearing a pineapple are playing skantshoard, hyper realistio, kighty detailed, best quality. high resoluttoon, 8K, HD.
A forke, a kenife and a plate with the broccolí and apple.
A teddy bear and hello kitty sit in front the Eiffel Tower.
Figure 1: Given user-pree user-provided bounding boxes and prompts of subjects, our Win Wan generates controllable and realistic images with pre-trained diffusion model, such as SDXI. (Podell al., 2023), without any finetuning with paired data.
ing the final generated objects being confined smaller regions rather than evenly distributed across the specified box. Secondly, the backpropagation update rule fails to account for the pre-trained distribution when biasing fea- tures, resulting in a trade-off between control and image quality, where stronger control often comes at the cost of poorer visual appearance. To the best of our knowledge, this the first study that theoretically analyze these two core components of backward guidance in Layout-to-Image task.
to encourage attention to uniformly cover the specified re- gion. Additionally, to avoid constraining irregularly shaped objects into rigid box-like form (e.g., coconut trees usually have broad leaves and slender trunks), we introduce a decay- ing schedule that gradually decreases the strength of prior along the denoising step that facilitates natural structure. On the other hand, focusing on trade-off between con- trollability and realism, design the update rule based on Langevin dynamies, which brings the best of two worlds by simultancously incorporating the updating directions given by both layout controls and the original T2I model. Specifi- cally, we introduce an adaptive weighting strategy to balance the two directions across different sampling steps, eliminat- ing the need for cumbersome hyperparameter search.. Ex- periments demonstrate that the proposed approach mitigates
In this paper, we propose novel model named WinWinl.ay, which generates high-quality training-free L2I results by explicitly accounting for above challenges. On one hand, to better align the given spatial guidance, we augment the attention energy function with the non-local attention prior
2

---

Control and Realism: Best of Both Worlds in Layout-to- without Training
the above issues successfully and generates satisfactory L.1 results (see Figure 1) in training-free manner.
A rabbit standing on a panoake.
In summary, our contributions can be summarized as: (i) We provide the first theoretical analysis of previous backward guidance methods to the best of our knowledge. Inspired by the theoretical insights, we present an advanced approach, Win WinWa,,yy, for Layout-to-Image generation which exhibits significantly control and realistic quality. (ii) We propose novel Non-local Attention Energy Function that guides the model to better adhere to spatial constraints while preserving the natural structure of objects. (iit) We explore a Langevin dynamics-baan -based Adaptive Update scheme to eliminate the trade-off between layout instruction and realistic appearance while maintaining efficiency. (iv) We conduct extensive experiments highlight the effectiveness of WinWinLay for both controllability and quality, thereby advancing the practical application of 1.21 generation.
Nor-local Attention Phio
Attention Map Cross-attention Layer
Output
Figure 2: Visualization of cross-attention between text to- ken "rabbit" and intermediate features of the denoiser. It can be observed that the attention energy function leads to attention focusing on local region. Conversely, after in- corporating non-local attention prior, the attention attempts to encompass the entire bounding box. To demonstrate the robustness of this prior, "pancake" is consistently equipped with non-local attention prior in this context.
2. Related Work
2.1. Text-to-Image Generation
Diffusion models (Ho et al., 2020) have recently disrupted the longstanding dominance of generative adversarial net- works (GANs) (Goodfellow et al., 2014) in image syn- thesis (Dhariwal& & Nichol, 2021: Song et al., 2021a; la; Ho et al., 2022), further accelerating advancements in Text-to- Image (T2I) generation (Saharia et al., 2022; Rombach et al., 2022; Podell et al., 2023; Peebles & Xie, 2023; Chen et al., 2024c). Benefitting from training on large-scale image-text datasets (Schuhmann et al., 2022b), they exhibited remark- able ability to generate diverse, creative images controlled by text prompts. Morcover, recent developments also unlock the potential of T2I to tackle challenging vision tasks, such as image editing (Brooks et al., 2023; Kawar et al., 2023; Ruiz et al., 2023; Mokady et al., 2023; Xu et al., 2024), style transfer (Sohn et al., 2024; Chen et al., 2024a; Ahn et al., 2024) and 3D generation (Chen et al., 2024e; Li et al., 2024; Wang et al., 2024b). Despite substantial progress, existing works still struggle to precisely control image details, such as layout, which significantly hampers their applicability in real-world scenarios. In this paper, we focus on control- ling subject synthesis within the complex scene through user-specified layout condition in a training-fre manner.
Image models with paired data. However, collecting such extensive labeled images is non-trivial and high-cost overheads also limit the usage of 1.21 in practice. To over- come the aforementioned challenges, recent studies (Bar-Tal et al., 2023; Kim et al., 2023; Singh et al., 2023; Couairon et al., 2023; Chen et al., 2024d:b; Liu et al., 2024a) explore training-free frreengg--reeeeee approaches that utilize forward or backward guidance mechanisms within the denoising process. Despite this success, the generated results often deviate from the predefined positions and exhibit severe unrealistic artifacts. Here, theoretically analyze the backward guidance and propose improved strategies to alleviate these problems.
3. Preliminaries
3.1. Latent Diffusion Model
Our work recruits the Latent Diffusion Model (LDM) (Rom- bach al., 2022) as prior, which defines a generative process that gradually transforms noise latent and a text prompt p an image Specifically, LDM first encodes the image into in a low-dimensional latent space using pre-trained encoder E, i.e., zo E(ro) and operates the diffusion pro- cess. Then, the representation 2n are decoded back into the image To using a pre-trained decoder D. Specifically, given noised latent 2, at step and text tokens y = <(p) where is frozen encoder (Radford et al., 2021), the denoiser eg is optimized to remove the noise e added to the latent code 2o:
2.2. Layout-to-Image Generation
Layout-to-Image (L2I) (Xue et al., 2023; Zheng et al., 2023; Xie et al., 2023; Jia et al., 2024; Liu et al., 2024b;a) fo- cus on generating images that simultaneously adhere to the textual prompt and corresponding layout instructions, e.g., bounding boxes and scribble. To achieve this, existing works (Li et al., 2023; Yang al., 2023; Wang et al., 2024a; Zhou et al., 2024) propose to finetune the powerful Text-to-
min Exo.e.elee t. 0
(1)
3

---

Control and Realism: Best of Both Worlds in Sayouutto-Image without Training
Here, learnable parameters 0 is typically integrated into U-Net (Ronneherger al., 2015) architecture, which comprises convolutional layers, self-altention and cross- attention mechanisms.
intuitive analysis:
Eves (a, m) = = -1--
E du mw),
(4)
where a. = E. an denotes the normalization of du which denotes attention value of 4 patch in attention map a. Notably, for simplicity and clarity of presentation, the notation for subjects and cross- cross-attention layer is omitted. Given max my = 1, so we have max; En, 0g my = and then obtain following equation:
3.2. Backward Guidance for Layout-to-Iman to-Image
To precisely position the subject at the specified location, backward guidance (Chen et al., 2024d) aims to sample desired images from the distribution p(=ly, b, i), where the bounding box b, corresponding to the text token yi of target subject. Specifically, backward guidance begins by design- ing an optimizable object, e.g., attention energy function Euef, redistribute attention map in feature 2, encour- aging the attention values of the 7t token to focus within region b,:
Eaef(a, m) = -mmxx Σ C.e ma - - ā. m,)². (5)
Here, is evident that minimizing Eaef is equivalent to max- imizing m Specifically, when ä attains maximum value, the support set of â is guaranteed to be contained within support set of m. Building on this insight, we first observe that the optimal solution to the attention energy function is not unique. Actually, when the support set of entirely contained within m a m can be maximized. However, this non- con-uniqueness may lead to the support of concentrating in localized regions of m. thereby compromis- ing effective control over the spatial layout. Meanwhile, notice that VaEnef (a, m) || m, causing all patches within the masked region to receive identical gradient magnitudes. This gives patches with larger initial values a significant ad- vantage during the optimization process, thereby amplifying localized effects (see Figure 2). To support this perspective, we start by considering a simple yet universal optimization objective follows:
Eaeg(al)), m, (1 En Mui (7) aui
En (7)
where d quantifies the strength of the association between each location in rros-attentionn layer y and token yi. m; denotes the binary mask which is transformed from b; with pixels inside the box region marked as I and those outside as O. To bias attention maps, the gradient of the Equation (2) is computed via backpropagation to update the initial latent feature = z:
2 2
ΣΣ (al)
(3)
where 1) is a hyperparameter used to control the strength.
max f(v) = m-softmax(t) where E ER" m E 13",
(6)
4. Method
and we denote g = sofumax(v) for simplicity.
In this section, we introduce WinWinLay, a training-free Layout-to-Iman generation framework. First, provide detailed introduction to Non-local Attention Energy Func- tion (Section 4),,, which is used to enhance layout constrain. Subsequently, we shift focus to explore Adaptive Update (Section 4.2) to eliminate the trade-off between control and quality.
Theorem 4.1. Assume that during the optimization process at a certain step, there exist mi -m and qj > Qk- After a single gradient update with step size > 0, the updated values d/y de satisfy d; > 4 and o au
Proof. First, Jacobian matrix of q with respect to can be computed as follows:
4.1. Non-local Attention Energy Function
diag(g) qq"
Attention energy function (Chen et al., 2024d) is a widely used loss term for guiding attention redistribution, but it often leads to objects occupying local region of the bounding box, hindering precise control. To address this, non-local attention prior is introduced to encourage attention to be
According to the chain rule, we have the gradient of v by:
q1 (mı - m)) : In (m g= m)
vef(u)= = JJ
(8)
smoothly distributed across the specified location.
Revisiting of Attention Energy Function. Based on the overview of attention energy function (Section 3.2), we can straightforwardly reformulate Equation (2) as following for
Then, w and t are updated as:
= + 3(m, 9" m)) = U, + 18'9
(9)

---

Control and Realism: Best of Both Worlds in Layout-to-Iman without Training
b/2 = uk 3mm - 9- mm)m) = Ukk + 88 gk:
(10)
Naturally, combining the above equations gives:
Varkedo Diyribition
Sugar Lolked
Lf = explo/ - -v/))= = =xp(u, - uk + 3 (13 9k))
adiisiedl riding homs to the leevil avd in het sir balizen twittn sig.
= exp(11 -mx) eep((' (q} -4:)
(11)
Li exp(B (4) -0x)) 4)
musher Lapest
[r] Backgropaphhoo Update
180 Acceptine Update
qk
qk
Figure 3: Given the initial feature E in a pre-trained dis- tribution, our goal is to iteratively update it within layout constraints to achieve a target latent vector 2. Here, (a) backpropagation update struggles to effectively maintain spatial control with limited number of updates, while an increased number of updates often leads to deviations from the pre-trate pre-trained distribution, leading to a trade-off between control and generation quality. In contrast, (b) adaptive up- date strategies simultancously account for the pre-trained distribution and layout constraints, thereby consistently im- proving overall performance. Zoom in for more details.
Through the analysis of the above problem, it can be con- cluded that patches within the masked region with larger initial values tend to amplify their relative prominence dur- ing the optimization process, thereby suppressing the growth of other regions. This implies that the attention map redis- tributed by energy function exhibits an implicit bias, fa- voring regions with larger initial values. Consequently, it becomes challenging to evenly cover the entire box.
Non-local Attention Prior. In this regard, simple and effec- tive non-local attention prior is introduced to facilitate global attention responses. Different from intuitively conceivable uniform constrain, this prior promotes the placement of objects near the center of the bounding box while encour- aging maximal coverage of the entire region. Concretely, given the bounding box (width as W, height as H) and its center point (Cz: Cy), the normalized distance from point = (Ua, Myy within the masked region S to the center can be calculated as du (u.-c,)² + (My -. H Accord-
4.2. Adaptive Update
Although backpropagation update is simple, it fails to bal- ance layout constraints and image quality. Therefore, we propose Adaptive Update to consistently enhance the appear- ance of outputs based on Langevin dynamics and adaptive distribution construction.
W ingly, we construct the prior distribution Tu IX exp(- - -Xdu). where À 2 0 is used to control the variance of the distribu- tion. This design ensures that points farther from the center are assigned smaller probability values. Subsequently, local bias is alleviated by maximizing the KL divergence between the distribution of the attention within S and the prior
Revisiting of Backpropagation Update. Given latent feature St+1 as timestep t + 1, conditional probability p(2; (x of pretrained diffusion model to obtain an initial estimation ž, Subsequently, the gradient update step refines & to mini- mize the non-local attention energy function Snaef (replace E for consistent description) obtained optimized latent variable 2, arg min Enoe (&,, m) via Equation (3) where inteentio__map(3
Rnap Σ WES
au an log
(12)
However, update in this man- ner fails to adequately account for the constraints imposed by the latent variable distribution p(zz), which may lead to a trade-off between generation control and output qual- ity. Specifically, when the gradient updates are insufficient, the optimization of E remains suboptimal, resulting in poor layout control. Conversely, when the gradient updates are overly exhaustive, the resulting 2 may deviate signif- icantly from the initial estimate ž, leading to a reduced likelihood p(2)). This misalignment adversely affects sub- sequent denoising steps, ultimately degrading the quality of the generated images. Moreover, the visualization results in Figure 3 under varying levels of control intensity further experimental substantiate this conclusion.
Tu
where au aa/ Eves au and denotes attention value.
Total Loss. Here, non-local attention energy function is formulated as the summation of Eaef and Rnap across all subject i and layer y:
En Mys aue )2
(7)
Enwess = >' [11
En aus
(7)
7
(13)
+ pp>
Σ S
á
log
us
Tui
To account for the irregular shape of objects in real-world scenarios, we introduce a hyperparameter y that decreases linearly with the denoising timesteps, enabling objects to adapt to naturally structure. Similar to existing work (Chen et al., 2024d), we only reallocate cross-ates atteenattntsoons with cor- responding tokens in the middle and first up layers.
Langevin Dynamics Updating. To eliminate the stubborn trade-off, we propose that both the attention redistribu- tion and p(2,) should be concurrently considered during the update process. For attention redistribution function, the corresponding Gibbs distribution can be constructed as
5

---

Control and Realism: Best of Both Worlds in Imyout without Training
p(m]zz) exp((-vnne m)), where is a hyperpa- rameter that controls the shape of the distribution. Given timestep f, our ultimate goal is to sample ž, from p(z)
[gf R2 S < K} represent the gradients of K tasks, the optimal gradient combination coefficients R sat- isfy GO Ga 4. where G is the matrix whose columns are the gradients Jj- Nash-M'L uses optimization to approxi- mate the solution for o, and we find that when K = 2, this equation has a simple analytical solution:
Based on Bayes' Theorem, we have:
p(z mm) (m))))))))))))
(14)
then the score function of pm=+/m) can be derived as:
Corollary 4.2. Given G = (g1)92) E Rdx2 and a =
(01, E R2, G Ga a. then we have
dn 02
= 1
V v,, (o))o(((())) V log(p( (2t)) V V=g(o((p())zz))
lm
= V log(p(z;)) (=x)) v as E (2,, m.)
Proof. According to Ga _ we have:
GT
(15)
Here, Vz, Vz,(og(p(z))) represents the unconditional score function which is approximated by pre-trained diffusion model. According to (Song& & Ermon, 2019), we can use Langevin dynamics to sample from any distribution with known score function. Specifically, given step size >> 0 and an initial value = Langevin dynamics iteratively up- dates as follows:
afllgı11 + 2222 92 = 1,
(17)
01 010292 91 + a || || =
(18)
By subtracting Equation (18) from Equation (17):
lellgn 19 11 = g=11
(19)
we can derive the conclusion as en | Telll
2 ((++1) = 2/ +5V (00 log(p(z(
(1) mm) + v 2Eck. (16)
Based on the above proof, we propose apative update rule by formalize v as an adaptive parameter for each iteration:
where I) and 0 < k S O. f 0 and O + 0o, the distribution of 2 will converge to p(2, m). Note that for step size & > 0 and finite O, the sampling process can be corrected using the Metropolis-Hastinne convert it into a strict MCMC sampling procedure. However, this correction step is typically omitted for convenience in practice. Here, similar to (Song et al., 2021b), we determine where r is
V[ log(p(=)))]]
(20)
method to
= / (zt, m))l'
This design enables us to effectively mitigate the trade-off at negligible cost, making more suitable for practical applications.
step size ç = 2(r 0<l1//1 V (og(((E the signal-to-noise ratio.
5. Experiments
Adaptive Distribution Construction. Although Langevin dy- namies effectively mitigates the trade-off, the introduction of the additional hyperparameter v > 0 of distribution conse- quently reduces generation efficiency. From Equation (15), it follows that adjusts the weight of v,,Enaef (Ze, m) in score function V., (=log(p(2((m)) Intuitively, a larger v re- sults in a steeper distribution, where the optimization pro- cess focuses more on minimizing the non-local attention energy function, leading to faster convergence (smaller O) but requiring larger step sizes to accelerate the optimiza- tion process of log(p(z))), which can increase the error of Langevin dynamics and then degrade image quality. Con- versely, a smaller /' produces a flatter distribution, prioritiz- ing image quality preservation, which slows the optimiza- tion process (larger O) and requires smaller & but leads to more iterations, reducing sampling efficiency. Therefore, selecting the appropriate is critical to balance image qual- ity and generation efficiency. Here, we propose treating Equation (16) as a multi-task optimization problem to plore the optimal v, where one task minimizes the attention energy and the other maximizes the distribution probabil- ity. Inspired by Nash-MTL (Navon et al., 2022), we model the gradient combination of these two tasks as a bargain- ing game, solving for the Nash Bargaining Solution. Let
In this section, we first provide the experimental setup and then conduct both qualitative and quantitative experiments to compare our method with previous SOTA Layout-to- Image methods. Additionally, we perform an ablation study to demonstrate the effectiveness of the proposed approaches.
5.1. Experimental setup
Evaluation Benchmarks. Akin to prior work (Chen et al., 2024d), we quantitatively evaluate our WinWinlay on COC02014 (Lin et al., 2014) and Flickr30K (Plummer et al., 2015). For performance evaluation, we leverage YOLOv7 (Wang et al., 2023) for object detection, employing metrics such as AP (Li et al., 2021) to assess the effective- ness of our method accurately locating and generating objects. Furthermore, the CLIP-s (Radford et al., 2021) is utilized quantitatively evaluate image-text compatibility, thereby measuring the semantic accuracy of the synthesized images. Additionally, we also use the advantage metric FID (Kynkäänniemi et al., 2023), PickScore (Kirstain et al., 2023) and ImageReward (Xu et al., 2023) to evaluate image quality. Here, we set text template as "A photo of [prompt]" to acquire more realistic results.
6

---

Control and Realism: Best of Both Worlds in Layout-to-Image without Training
Layout-Control
AttRe
R&B
CSG
WinWinLay (Ours)
A polay bear wearing the white het, black glasses, tie and suit.
A castle, trees and blacke car in the warsh.
IICH
THE IS
pillow on the gray sofa olock hanging on wall.
An astrow.Rut riding a horsé on the beach, and A hot Rür balloon in the sky.
As the lights up the skyy, tiger wanders the grassy meadoww, admiring the breathtakking view, Lakes quistlt refleats the maggiifient display. and the distance, fantasy. gle, highly dettilee.
Figure Qualitative comparison of our WinWinLay and state-of-the-arr methods. Zoom for more details.
CO005014
Flicker30K
User Study
Model
AP1
CLIP-st
FID.
PickScore]
ImageReward!
APt
CLIP-s?
FID]
PickScore?
ImageRowardh
Controlability?
Quality?
Layout- Coatrol (Chen al., 202/d)
8:42
0.310
29.79
21.09
0.7038
14.19
0.288
29.83
20.39
0.7016
12.1
7.5
AuRe (Phing al., 2024)
15.51
0.296
27.51
21.23
0.7109
15.26
0.277
27.72
20164
0.7055
18.7
RRB (Xiao al. 2024)
17.63
0.306
28.22
21.16
0.7071
14,80
0.291
28.18
20.58
0.114
20.6
19.4
al., 2024a)
Ours
19.74
0.327
26.85
21.41
0.7218
17.28
0.309
27.04
20.85
0.7002
27.7
29.8
Table 1: Quantitative comparison of our WinWinLay and state-of-the-art methods.

---

Control and Realism: Best of Both Worlds in Layout-to- Image without Training
Staiee Difiusien 15
Nansiona ALL Dop
Adl Uppt
Alt Furt dask Uppd
p=1
5
20
reality.
volobit wotring sunglasss,, Strious expression.
Figure 6: Ablation on hyperparameter of non-local attention prior.
highly sensitive to gradient strength, where higher accuracy often leads to a severe decline in image quality, especially when generating large number of objects. In contrast, our method consistently outperforms across multiple datasets and evaluation metrics, demonstrating more robust im- provement. Additionally, we resort to user studies to eval- uate which method generates results that are most favored by humans. We conducted two user studies on the results in terms of the Controllability and Quality. In the first study, participants are asked to select the images that best align with the given layout. In the second study, participants are tasked with identifying the images that exhibit the most realistic appearance. To ensure clarity and reproducibility, we conducted the user study on Wenjuanxing, a platform similar to Amazon Mechanical Turk. 150 participants eval- uated 50 image pairs, yielding 7500 responses per study, Images were shown side-by-side with layout prompts, and both question order and image positions were randomized to avoid bias. As shown in Table 1, over 27.7% of our results are selected as the best in both two metrics, which proves a significant advantage in generation.
Lights akyg. tigur winnllos grassy autrisiig breatkeskingg vieu, a strend Lokuz quittly reflacts the magenifieent slapping..
Figure 5: Qualitative ablation on proposed Non-local Atten- tion Energy Function and Adaptive Update. Zoom in for more details.
COCO2014
Model
APso
APt
CLIP-st
Att. Fun, Back. Upd.
25.8
8.4
0.310
Non-local Att.Eng.Fun. + Back.
44.1
17.4
0.318
Att. Ent.Eng.Fun Fun. Upd.Upd.
36.7
14.9
0.324
Non-local Att.Eng.Fun. + Add.UUp.
49.2
19.7
0.327
Table 2: Quantitative ablation on proposed Non-local Atten- tion Energy Function and Adaptive Update on COCO2014. Implementation Details. We adopt the Stable Diffusson 1.5 (Rombach et al., 2022), pre-trained on the LAION- 5B (Schuhmann et al., 2022a), as our base Text-to-Image model. During generation, we employ the DDIM sampler with 50 steps and set the scale guidance to 7.5 for generation. Since layout construction typically occurs during the early stages of denoising, we apply the layout constraint only within the initial 10 steps. The hyperparameters p of non- local attention prior is set to 5/0 for max/min, respectively. For adaptive update, we set steps O of Langevin dynamics is set as 4 and signal-to-noiss ratio as 0.06. We observe that these parameters generally work well in most cases, proving the generalizability of WinWinLay. We also point that better results may be obtained with a customized setting, e.g., a larger p or more iterations for Langevin dynamics.
Qualitative Comparison. To present a more detailed and visual comparison of our model, we carry out experiments smaller hand-crafted dataset with 3-5 objects. For fair comparison, we generate 10 images for each method under the same random seed and subsequently select the optimal image based on the AP for display. To demonstrate the effecttiveness of WinWinLay, images are presented for each case. From the results Figure 4, we can draw the following conclusions: (t) Our method effectively places the target object within the given region while filling the entire bounding box without compromising the natural structure of the object, representing significant improvement over existing methods. In contrast, other approaches often fail to generate images that faithfully adhere to the layout (e.g., th row), and parts of the object may collapse into localized regions of the bounding box (e.g., 4th row); (ii) Our method successfully eliminates the trade-off between control and quality, without compromising the generative capability of the underlying model despite the additional layout con- straints. However, existing works focus on layout adherence while neglecting the realism of the generated objects (e.g., tth row). Furthermore, multiple distinct results generated under the same prompt and spatial constraints demonstrate
5.2. Comparison With SOTA Methods
We compare our approach with four representative state- of-the-arts of Layout-Contre (Chen et al., 2024d), At- tRe (Phung et al., 2024), R&B (Xiao et al., 2024) and CSG (Liu et al., 2024a) show the advantage of Win- WinLay. All methods are implemented by official codes.
Quantitative Comparison. As presented in Table 1, we first quantitatively evaluate generated images for our test and AttRe,
dataset. Compared to methods Layout-Conte CSG shows a significant improvement in object placement accuracy. However, we observed in our experiments that is
8

---

Control and Realism: Best of Both Worlds in Layout-to-Image without Training
AP=17.5
However, when p becomes too large, it may lead unnat- ural object placements within the bounding box, such as "rabbit" appearing on square canvas. In our experiments, we find that setting to 5 generally yields optimal results, which used across all experiments.
0.20
Signal-to-noise Ratio
0.15
AP=17.8
0.30
AP=19.7
Hyperparameter of of Adaptive Update. Adaptive parame- ter v plays critical role in the effectiivnesss of the proposed Adapative Update and impact the overall performance of WinWinLay. In Section 4.2, analyzed the impact of different on efficiency and performance, proposing an adaptive strategy to significantly reduce the complexity of hyperparameter tuning. To validate its effectiveness, we introduce coefficients of varying magnitudes to the adap- tive parameter and conduct grid searches to determine the signal-to-nofe ratio of optimal step size and the number of update steps O for each coefficient. As shown in Fig- ure 7, larger generally require larger step sizes and fewer update steps, which substantially degrade both accuracy and quality. Conversely, smaller v has less pronounced effects on performance but significantly increase generation time.
1P=19.4
0.00
AP=18.4
0
20
Step Number 0
Cs=0.321
15
10
Cs=0.323
Cs=0.327
0
0s=0.312
Cs=0.306
0.05
0.2
10
Coefficient of V
Figure 7: Ablation on hyperparameter of Adaptive Update on COCO2014. Cs denotes CLIP-s metric.
6. Conclusion
The paper introduces WinWinLay, novel training-fiee framework for Layouut-t-Image generation, which achieves significant improvements in layout precision and visual fi- delity. Addressing limitations in existing methods, WinWin- Lay incorporates two innovative components: Non-local Attention Energy Function, which ensures uniform attention distribution within specified layouts while preserving natu- ral object structures, and Adaptive Update, which leverages Langevin dynamics to effectively balance layout control and image quality. Extensive experiments on standard bench- marks demonstrate that Win WinLay outperforms state-of- the-art approaches both controllability and photorealism, making it a robust and efficient solution for 1.21 tasks.
the robustness of WinWinLay, thereby further advancing the progress of Layout-to-Iman in practical applications.
5.3. Ablation Study
Effect of Proposed Strategies. To further demonstrate the efficacy of the proposed method, we incrementally introduce Non-local Attention Energy Function and Adaptive Update the baseline model and observe the resulting changes. As shown in Figure 5, Non-local Attention Energy Function significantly enhances the control over the layout, while also ensuring an accurate representation of all target objects. On the other hand, Adaptive Update not only strengthens the spatial placement accuracy but also improves the over- all image quality (e.g., "giraffe" appears more realistic). Furthermore, the quantitative results provided in Table 2 align with the visual observations, with Non-local Atten- tion Energy Function achieving substantial increase in AP and AP while Adaptive Update further refines the spatial positioning and enhances image quality.
Impact Statement
This project provides a training-free method for layout- controlled image synthesis, enhancing controllability while preserving generative strength of base models; however, like other generative techniques, may be misused for disinfor- mation, highlighting the need for future research to address ethical risks associated with layout-guided generation.
Hyperparameter of Non-local Attention Prior. Previous Attention Energy Function often suffers from the problem of attention collapse to local regions. Hence, non-local at- tention prior is introduced to constrain the attention to focus on the center of the bounding box and gradually expand to cover the entire box. Here, p serves as the hyperparameter that controls the strength of the non-local attention prior. As shown in Figure 6, with the gradual increase of p. the objects in the image progressively align with the edges of the bounding box, allowing for more precise layout control.
A Acknowledgements
This paper is supported by National Natural Science Foun- dation of China under Grants (U23B2012, 12471308), Bei- jing Natural Science Foundation (1254050), Fundamental Research Funds for the Central Universities, and National Research Foundation, Singapore, under its Medium Sized Center for Advanced Robotics Technology Innovation.

---

Control and Realism: Best of Both Worlds in Layout-to-Image without Training
References
Jiang. Y., Wu, T., Yang, S., Si, C., Lin, D., Qiao, Y., Loy, C. C., and Liu, Z. Videobooth: Diffusion-based video generation with image prompts. In CVPR, pp. 6689-6700, 2024.
Ahn, N., Lee, J., Lee, C., Kim, K., Kim, D., Nam, S.-H., and Hong, K. Dreamstyler: Paint by style inversion with text-to-image diffusion models. In AAAI, pp. 674-681, 2024.
Kawar, B., Zada, S., Lang, O., Tov, O., Chang, H., Dekel, T., Mosseri, I., and Irani, M. Imagic: Text-based real image editing with diffusion models. In CVPR, pp. 6007-6017, 2023.
Tal, O., Yariv, L., Lipman, Y., and Dekel, T. Multi- diffusion: Fusing diffusion paths for controlled image generation. ICML, pp. 1737-1752, 2023.
Brooks, T., Holynski, A., and Efros, A. A. Instructpix2pix: Learning to follow image editing instructions. CVPR, pp. 18392-18402, 2023.
Kim, Y., Lee, J., Kim, J.-H., Ha, J.-W., and Zhu, J.-Y. Dense text-to-image generation with attention modulation. In ICCV, pp. 7701-7711, 2023.
Chen, D.-Y., Tennent, H., and Hsu, C.-W. Artadapter: Text- to-image style transfer using multi-level style encoder and explicit adaptation. In CVPR, pp. 6619-8628, 2024a.
Kirstain, Y., Polyak, A., Singer, U., Matiana, S., Penna, J., and Levy, O. Pick-a-pic: An open dataset user preferences for text-to-image generation. NeurlPS, pp. 36652-36663, 2023.
Chen, H., J., Zhuang. W., Vikalo, H., and Lyu, L. Bound- ary attention constrained zero-shot layout-to coouuttt-imaaee gen- eration. arXiv preprint arXiv:2411.10499 2024b.
Kwon, M., Oh, S. W. Zhou, Y., Liu, D., Lee, J.-Y., Cai, H., Liu, B., Liu, F,, and Uh, Y. Harivo: Harnessing text-to- image models for video generation. ECCV, pp. 19-36, 2025.
Chen, J., Jincheng, Y., Chongjian, G., Yao, L., Xie, E., Wang, Z., Kwok, J., Luo, P., Lu, H., and Li, Pixart- alpha: Fast training of diffusion transformer for photore- alistic text-to-image synthesis. In ICLR, 2024c.
Kynkäääniemi, T., Karras, T., Aittala, M., Aila, T., and Lehtinen, The role of imagenet classes in fréchet in- ception distance. In ICLR, 2023.
Chen, M., Laina, I., and Vedaldi, Training-free layout control with atts-nattntionn guidance. In WACV, pp. 5343 5353, 2024d.
Li, M., Zhou, P., Liu, J.-.., Keppo, J., Lin, M., Yan, S., and Xu, X. Instant3d: Instant text-to-3d -d-- generation. Interna- tional Journal of Computer Vision, pp. 1-17, 2024.
Chen, Y., Pan, Y., Yang, H., Yao, T., and Mei, T. Vp3d: Unleashing 2d visual prompt for text-to-3d generation. In CVPR, pp. 4896-4905, 2024e.
Li, Y., Liu, H., Wu, Q. Mu, F,, Yang, J., Gao, J., Li, C., and Lee, Y. J. Gligen: Open-set grounded text-to- text-to-image generation. In CVPR, pp. 22511-22522 2023.
Couairon, G., Careil, M., Cord, M., Lathuiliere, S., and Verbeek, J. Zero-shot spatial layout conditioning for text- to-image diffusion models. In ICCV, pp. 2174-2183, 2023.
Li, Z., Wu, J., Koh, I,, Tang, Y., and Sun, L. Image synthesis from layout with locality-aware mask adaption. In ICCV, pp. 138 13-28888, 2021.
Dhariwal, P. and Nichol, A. Diffusion models beat gans image synthesis. In NeurlPS, pp. 8780-878 8794, 2021.
Lin, T.-Y,, Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., and Zitnick, C. L. Microsoft coco: Common objects in context. In ECCV, pp. 740-755, 2014.
Goodfellow, I. J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley Farley, D., Ozair, S., Courville, A. C., and Bengio, Y. Generative adversarial nets. In NewrIPS, 2014.
Ho, J., Jain, A., and Abbeel, P. Denoising diffusion proba- bilistic models. In NeurlPS, pp. 6840-6851 2020.
Liu, J., Huang, T., and Xu, C. Training-free composite scene generation for layout-to-image synthesis. In ECCV, pp. 37-53, 2024a.
Ho, J.. Saharia, C., Chan, W., Fleet, D. J., Norouzi, M., and Salimans, T. Cascaded diffusion models for high fidelity image generation. Journal of Machine Learning Research, 23(4)): 2022.
Liu, S., Ma, A., Wu, X., Leng, D., Yin, Y., et al. Hico: Hierarchical controllable diffusion model for layout-to- image generation. NeurlPS, 2024b.
Jia, C., Luo, M., Dang, Z., Dai, G., Chang, X., Wang, M., and Wang, J. Ssmg: Spatial-semantic map guided diffusion model for free-form layout-ur coutt-to-ime to-image generation. In AAAL, pp. 2480 2488, 2024.
Mokady, R., Hertz, A., Aberman, K., Pritch, Y., and Cohen- Or, D. Null-text inversion for editing real images using guided diffusion models. In CVPR, pp. 6038-6044 2023.
10

---

Control and Realism: Best of Both Worlds in Layout-to-Image without Training
Navon, A., Shamsian, A., Achituve, I., Maron, H., Kawaguchi, K., Chechik, G., and Fetaya, E. Multi-task learning bargaining game, ArXiv, abs/2202.01017, 2022.
C., Wortsman, M., Schramowski, P., Kundurthy, S., Crow- son, K., Schmidt, L.., Kaczmarczyk, R., and Jitsev, J. Laion-5b: An open large-scale dataset for training next generation image-text models. ArXiv, abs/2210.08402, 2022a.
Parmar, G., Kumar Singh, K., Zhang, R., Li, Y., Lu, J., and Zhu, J.-Y. Zero-shot amage-to-image translation. In SIGGRAPH, pp. 1-11, 2023.
Schuhmann, C., Beaumont, R., Vencu, R., Gordon, C., Wightman, R., Cherti, M., Coombes, T., Katta, A., Mullis, C., Wortsman, M., et al. Laion-5b: An open large-scale dataset for training next generation image-texi models. In NeurIPS, pp. 25278 252944, 2022b.
Peebles, W. and Xie, S. Scalable diffusion models with transformers. In ICCV, pp. 4195-4205, 2023.
Phung, Q., Ge, S., and Huang, J.-B. Grounded text-to- image synthesis with attention refocusing. In CVPR, pp. 7932-7942, 2024.
Shi, J., Xiong, W., Lin, Z., and Jung, H. J. Instantbooth: Personalized ext-to-immage generation without test-time finetuning. In CVPR, pp. 8543-8552, 2024.
Plummer, B. A., Wang, I.., Cervantes, M., Caicedo, J. C.,, Hockenmaier, J., and Lazebnik, Flickr30k entities: Collecting region-to-phrase correspondences for richer Image-to-sentence models. In ICCV, pp. 2641-2649, 2015.
Singh, J., Gould, S., and Zheng. L. High-fidelity guided image synthesis with latent diffusion models. In CVPR, pp. 5997-6006, 2023.
Sohn, K., Jiang, L., Barber, J., Lee, K., Ruiz, N., Krishnan, D., Chang, H., Li, Y.. Essa, I., Rubinstein, M., al. Style- drop: Text-to-image synthesis of any style. In NeurIPS, 2024.
Podell, D., English, Z., Lacey, K., Blattmann, A., Dockhorn, T., Müller, J., Penna, J., and Rombach, R. Sdxl: Im- proving latent diffusion models for resouuiono image synthesis. arXiv preprint arXiv:2307.01952, 2023.
Song, J., Meng. C., and Ermon, S. Denoising diffusion implicit models. In ICLR, 2021a.
Qing, Z., Zhang, S., Wang, J., Wang. X., Wei, Y., Zhang, Y., Gao, C., and Sang, N. Hierarchical spatio-temporal decoupling for text-to-video generation. In CVPR, pp. 6635-6645 2024.
Song, Y. and Ermon, S. Generative modeling by estimating gradients of the data distribution. In NeurlPS, 2019.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In ICML, pp. 8748-8763. 2021.
Song, Y., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Er- mon, S., and Poole, B. Score-based generative modeling through stochastic differential equations. In ICLR, 2021b.
Tosi, F., Ramirez, P. Z., and Poggi. M. Diffusion models for monocular depth estimation: Overcoming challenging conditions. In ECCV, pp. 236-257, 2025.
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. High-resolution image synthesis with latent diffuusinn models. In CVPR, pp. 10684-10695, 2022.
Tumanyan, N., Geyer, M., Bagon, S., and Dekel, T. Plug- and-play diffusion features for text-driven image-to- image translation. In CVPR, pp. 1921-1930, 2023.
Ronneherger, O., Fischer, P., and Brox, T. U-net: Convolu- tional networks for biomedical image segmentation. In MICCAI, pp. 234-241, 2015.
Wang, C.-Y., Bochkovskiy, A., and Liao, H.-Y. M. Yolov7: Trainable bag-of- freebbefreebies sets new state-of-the-an for real-time object detectors. In CVPR, pp. 7464-7475, 2023.
Ruiz, N., Li, Y., Jampani, V., Pritch. Y., Rubinstein, M., and Aberman, K. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In CVPR, pp. 1,500-22510, 2023.
Saharia, C., Chan, W., Saxena, S., Li, L., Whang, J., Denton, E. I.,, Ghasemipour, K., Gontijo Lopes, R., Karagol Ayan, B., Salimans, T. et al. Photorealistic text-to-image dif- fusion models with deep language understanding. In NeurlPS, pp. 36479-36494, 2022.
Wang, X., Darrell, T., Rambhatla, S. S., Girdhar, R., and
Misra, I. Instancediffusion: Instance-level control for image generation. In CVPR, pp.
2024a.
Wang, Z., Lu, C., Wang, Y., Bao, F., Li, C., Su, H., and Zhu, J. Prolificdreamer: High-fidelity and diverse text-to-3d generation with variational score distillation. In NeurIPS, 2024b.
Schuhmann, C., Beaumont, R., Vencu, R., Gordon, C., Wightman, R., Cherti, M., Coombes, T., Katta, A., Mullis,

---

Control and Realism: Best of Both Worlds in Sevout-to-Image without Training
Wu, J. Z.,, Ge, Y., Wang, X., Lei, S. W., Gu, Y., Shi, Y., Hsu, W., Shan, Y., Qie, X., and Shou, M. Z. Tune-a- video: One-shot tuning of image diffusion models for text-to-video generation. In ICCV, pp. 7623-7633, 2023.
Wu, Y., Zhou, X., Ma. B., Su, X., Ma, K., and Wang, X. Ifadapter: Instance feature control for grounded text- to-image generation. arXiv preprint 0004020009082240, 2024.
Xiao, J., Lv, H., Li, L., Wang, S., and Huang, Q. R&b: Region and boundary aware zero-shot grounded text-to- image generation. In ICLR, 2024.
Xie, J., Li, Y., Huang, Y., Liu, H., Zhang, W., Zheng, Y., and Shou, M. Boxdiff: Text-to-image synthesis with training-free box-constrained diffusion. In ICCV, pp. 7452-7461, 2023.
Xu, J., Liu, X., Wu, Y., Tong, Y., Li, Q., Ding, M., Tang, J.. and Dong, Y. Imagereward: Learning and evaluat- ing human preferences for texi-to-image generation. In NeurlPS, pp. 15903-15923 2023.
Xu, S., Huang. Y., Pan, J., Ma, Z., and Chai, J. Inversion- free image editing with language-gui guided diffusion models. In CVPR, pp. 9452-9461, 2024.
Xue, H., Huang, Z., Sun, Q.. Song, L.,, and Zhang, W. Freestyle mayout-to-image synthesis. In CVPR, pp. 14256- 14266, 2023.
Yang, Z., Wang, J., Gan, Z., Li, L., Lin, K., Wu, C., Duan, N., Liu, Z., Liu, C., Zeng, M., et al. Reco: Region- controlled text-to-image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14246-14255, 2023.
Zhang, Z., Li, B., Nie, X., Han, C., Guo, T., and Liu, L. Towards consistent video editing with text-to-image dif- fusion models. In NeurlPS, pp. 18508-58519, 2023.
Zheng, G., Zhou, X., Li, X., Qi, 7., Shan, Y., and Li, X. Layoutdiffusion: Controllable diffusion model for layout- to-image generation. In CVPR, pp. 22490-22499, 2023.
Zhou, D., Li, Y., Ma, F., Zhang, X., and Yang. Y. Mige: Multi-instance generation controller for text-to-image synthesis. In CVPR, 6818-6828 2024.
12


Reviews:
[{"review_id": "eizEDIZxLg", "reviewer": "Reviewer_z1Kv", "summary": "This paper is a new method in the layout to image domain. \n\nTwo key problems this paper tries to address: \n* Imprecise location \n* Unrealistic artifacts \n\nThe core contribution of this paper: \n* A non-local attention energy functino \n* An adaptive updating mechanism to balance the spatial control and image quality  \n\nThe paper has a well-organized structure: \n- Basic context, introduction, related work \n- Problem analysis, theoretical analysis \n- Method details \n- Evaluation \n\nThis structure is easy to follow. And the theoretical analysis part is the most interesting and insightful part.\n\n## Update after rebuttal \nThe rebuttal addressed my concerns. So I increased the rating.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to sincerely thank Reviewer z1Kv for the thorough and constructive feedback on our manuscript. We are more than happy that the reviewer finds our writing well-organized, theoretical analysis illustrative, and quantitative improvement promising. We would like to address the concerns as below.\n\n***W1. Object in the center region.***\n\n**A1.** We would like to clarify that generating objects near the center of the bounding box represents a \"positive bias\", which is a deliberate design choice intended to enhance generation stability and user controllability. Specifically, placing objects at the center of the bounding box offers finer-grained and more predictable spatial control. In contrast, generating objects near the periphery of the box can introduce several undesirable effects:\n\n- Objects placed near the boundaries are more susceptible to spatial overflow, often leading to leakage beyond the given region;\n\n- Non-central placements inherently introduce positional ambiguity, making it difficult to predict or control which subregion of the box will be emphasized during generation.\n\nSuch ambiguity increases uncertainty during optimization and often results in instability or inconsistent outcomes. By encouraging central placement within the bounding box, our method achieves a more robust trade-off between layout adherence and generation reliability. This design also enables users to maintain precise control, especially when dealing with dense or complex scenes. To avoid confusion, we have provided additional clarification in both the Method and Limitation sections of the revised manuscript. We emphasize that this center-focused behavior arises from the prior formulated in Equ.13, where the centrality bias is smoothly modulated by the hyperparameter $\\rho$ and decayed over generation. This mechanism is not rigid; it merely guides the model toward stable, interpretable spatial alignment without strictly constraining the center of the object to remain fixed at the center *(As discussed in lines 80-84)*.\n\n\n\n***W2. Adaptive update rule.***\n\n**A2.** While we acknowledge the use of adaptive updates in prior works such as \"Taming Transformers for High-Resolution Image Synthesis\", we would like to emphasize that adaptive update rule proposed in our method is conceptually and technically distinct, as elaborated below:\n\n- Theoretical Foundation: The adaptive strategies employed in prior works are largely heuristic, lacking a principled formulation. In contrast, our approach is grounded in multi-task optimization theory. Specifically, we formulate the adaptive update as a Nash Bargaining problem between two competing objectives—layout controllability and visual fidelity—leading to a theoretically justified and analytically solvable solution. This formulation offers a deeper understanding of the trade-off and provides a rigorous basis for gradient balancing.\n\n- Gradient Computation: To mitigate computational burden, prior works typically approximate gradients by restricting updates to the final layer or a limited subset of parameters. Our method, however, naturally yields full gradients for each task without introducing additional overhead, as both objectives are defined over the same latent variables in the denoising process. This makes the adaptive update mechanism not only theoretically sound but also computationally efficient and seamlessly compatible with our training-free setting.\n\n\nTaken together, while the high-level motivation is shared, the underlying methodology, implementation, and theoretical framing of our adaptive update rule differ substantially. We have further clarified this distinction in the revised manuscript.\n\n***W3. Other Limitaion.***\n\n**A3.** Compared to Text-to-Image methods, our Layout-to-Image approach typically requires more time—for instance, SD1.5 takes 3.17s, while WinWinLay takes 15.01s. Nevertheless, our method remains more efficient than the current SOTA method, CSG, which takes 19.88s. Hence, improving the generation efficiency of our model will be an important focus in future work.\n\n***W4. Use of Vspace.***\n\n**A4.** We have reorganized the content layout in the revised version to ensure improved visual clarity.\n\nThanks again for the insightful review. We are happy to discuss any aspects of the manuscript that may require further clarification.", "claims_and_evidence": "The claims have been supported by theoretical anaysis, proof and comprehensive evaluations, including quantitative and qualitative experiments. \nThe overal claim is solid.", "methods_and_evaluation_criteria": "This method follows existing evaluation critera. These results make sense to illustrate the effectiveness of the propoesd method.", "theoretical_claims": "There are two theoretical claims: \n- Non-local attention energy function helps overcome the spatial distributino biases \n- The adaptive update promotes in-domain updating. \n\nBoth of the theoretical claims look correct.", "experimental_designs_or_analyses": "- Quantitative\n  - Layout accuracy has been evaluated using AP_50/AP\n  - Image-text alignment\n- Qualitative\n  - User study\n  - Some qualitative comparing results", "relation_to_broader_scientific_literature": "This paper introduces some improvements on the layout to image generation problem. Potentially the idea could also helps other training-free methods in T2I domain, e.g. some training-free image editing tasks.", "essential_references_not_discussed": "This paper has discussed relevant papers.", "other_strengths_and_weaknesses": "Main advantages: \n- The paper writing is well-organized and easy to follow. \n- The problems of existing work have been well illustrated in the theoretical analysis sections \n- The quantitative evaluation results show promising improvement over existing works.  \n\nThere are two main weaknesses of this paper: \n- The non-local attention prior seems also introduces a new bias: the object seems to be prioritized into the center region.\n- The adaptive update rule has already been proposed in several places, e.g. (Taming Transformers for High-Resolution Image Synthesis)", "other_comments_or_suggestions": "- The paper figure uses a lot of vspace, affecting the readability.", "questions_for_authors": "The improved non-local attention energy function is designed to solve the spatial distribution biases problem. However, the non-local attention prior seems also introduces a new bias: the object seems to be prioritized into the center region. This may not be a critical problem. But it seems that it's better included in the limitation or discussion sections. \n\nAny other limitations of the proposed method?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dyjMcybrsq", "reviewer": "Reviewer_hxyx", "summary": "Layout-to-image generation faces two significant challenges: a.) imprecise object localization and b.) the presence of unrealistic artefacts in the final output. To address these issues, the authors propose Win-Winlay, a novel method that incorporates:\n1. **Non-Local Attention Energy Function**: This function refines attention scores across regions within each layout box, improving localization accuracy.\n2. **Adaptive Update Scheme**: A Langevin dynamics-based adaptive update scheme that generates more realistic outputs.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to sincerely thank Reviewer hxyx for the thorough and constructive feedback on our manuscript. We are more than happy that the reviewer finds our paper is well-structured, theoretical analysis illustrative, and exhaustive ablation studies. We would like to address the concerns as below.\n\n***W1. More Baselines.***\n\n**A1.**  As our proposed WinWinLay framework operates in a training-free paradigm for Layout-to-Image generation, all methods selected for comparison in our initial experiments,e.g., BoxDiff and CSG, were likewise training-free approaches. In contrast, methods such as [1] and [2] are training-based, requiring supervised learning on paired layout-image data. These methods operate under fundamentally different assumptions and objectives, making a direct comparison less meaningful within the scope of our setting.\n\nRegarding [3], we acknowledge its relevance. Due to strict page limitations, we initially prioritized comparisons with widely recognized baselines and the most recent SOTA method, CSG (ECCV 2024). Here, to address the concern and to offer the more comprehensive evaluation, we have now included additional experiments comparing our method with [3] and [4], both of which follow training-free paradigms more aligned with our setting.\n\nThe extended results demonstrate that WinWinLay consistently outperforms these methods across multiple evaluation metrics, highlighting its superior controllability and visual realism. Visual results are visible on this [link](https://anonymous.4open.science/r/WinWinLay/Compared%20with%20more%20SOTAs/Compared%20with%20more%20SOTAs.png). These findings further substantiate the robustness and effectiveness of our proposed method.\n\n| Model(On COCO2014) | AP$\\uparrow$ | CLIP-s$\\uparrow$ | FID$\\downarrow$ | PickScore$\\uparrow$[5] | ImageReward$\\uparrow$[6] |\n| :------------------: | :----------: | :--------------: | :-------------: | :-----------------: | :-------------------: |\n|    Layout-Control    |    14.19     |      0.288       |      29.83      |        20.39        |        0.7016         |\n|       AttRe[3]       |    15.26     |      0.277       |      27.72      |        20.64        |        0.7095         |\n|        R&B[4]        |    14.80     |      0.291       |      28.18      |        20.58        |        0.7114         |\n|         CSG          |    15.11     |      0.282       |      27.90      |        20.51        |        0.7049         |\n|         Ours         |  **17.28**   |    **0.309**     |    **27.04**    |      **20.85**      |      **0.7202**       |\n\n\n| Model(On Flicker30K) | AP($\\uparrow$) | CLIP-s$\\uparrow$ | FID$\\downarrow$ | PickScore$\\uparrow$[5] | ImageReward$\\uparrow$[6] |\n| :----------------: | :------------: | :--------------: | :-------------: | :-----------------: | :-------------------: |\n|   Layout-Control   |      8.42      |      0.310       |      29.79      |        21.09        |        0.7038         |\n|      AttRe[3]      |     15.51      |      0.296       |      27.51      |        21.23        |        0.7109         |\n|       R&B[4]       |     17.63      |      0.306       |      28.22      |        21.16        |        0.7071         |\n|        CSG         |     17.58      |      0.299       |      27.64      |        21.22        |        0.7027         |\n|        Ours        |   **19.74**    |    **0.327**     |    **26.85**    |      **21.41**      |      **0.7218**       |\n\n[1] Instancediffusion: Instance-level control for image generation. CVPR2024.\n\n[2] Migc: Multi-instance generation controller for text-to-image synthesis. CVPR2024. \n\n[3] Grounded text-to-image synthesis with attention refocusing. CVPR2024.\n\n[4] R&B: Region and Boundary Aware Zero-shot Grounded Text-to-image Generation. ICLR2024.\n\n[5] Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generation. NeurIPS2023\n\n[6] ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generation. NeurIPS2023\n\n\n\n***W2. Multiple instances.***\n\n**A2.** We would like to clarify that our method is capable of handling  textual prompts and layouts involving multiple instances of the same object. This can be achieved by jointly mapping quantifiers and object names (e.g., \"two\" + \"apples\") to multiple bounding boxes, and then enforcing center-based generation constraints within each individual bounding box. Visual results are visible on this [link](https://anonymous.4open.science/r/WinWinLay/Multiple%20instances%20generation/Multiple%20instances%20generation.png) and have been added in the revised version.\n\n***W3. Typos.***\n\n**A3.** We have carefully checked the grammar of the entire manuscript and made corrections.\n\nThanks again for the insightful review. We are happy to discuss any aspects of the manuscript that may require further clarification.", "claims_and_evidence": "Yes, the claims are well-supported by thorough evidence in the manuscript.\n- Figures 2 and 3 clearly illustrate the shortcomings of existing methods. \n- The ablation studies in Section 5.3 provide strong evidence justifying the key design choices.", "methods_and_evaluation_criteria": "Yes, the proposed method is well-justified and effectively addresses the issues of imprecise localization and unrealistic artefacts in a training-free approach for the Layout-to-Image generation task.\n\nThe method is evaluated on the COCO2014 and Flickr30 datasets. YOLOv7 is used for object detection, with AP metrics assessing the accuracy of generated objects. Additionally, CLIP-s evaluates alignment with the conditioning text, and a user study further supports the findings.", "theoretical_claims": "Yes, I checked the correctness of the theoretical claims presented in Theorem 4.1, Corollary 4.2 and other Equations in the paper. All the proofs and equations are correct.", "experimental_designs_or_analyses": "Yes, the experimental design for ablation studies is sound and is analyzed well in the manuscript.", "relation_to_broader_scientific_literature": "The key insight of this paper is that the widely used attention energy function introduces spatial distribution biases, preventing objects from aligning uniformly with layout boxes. To address this, the authors propose a non-local attention prior that redistributes attention scores for better alignment with layout boxes. \n\nHence, it is helpful for broader scientific literature where the attention scores need to be redistributed based on a spatial region.", "essential_references_not_discussed": "All the major works are cited. Although, following work is not discussed: \n1. Phung, Q., Ge, S., & Huang, J. B. (2024). Grounded text-to-image synthesis with attention refocusing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 7932-7942).   \n        This work uses attention refocusing mechanism and spatial layouts for controlled generation.", "other_strengths_and_weaknesses": "**Strengths**\n\n- **[S1] Paper Writing**: The paper is well-structured, with clear figures that effectively explain the key ideas.\n- **[S2] Theoretical Proofs**: The proposed strategies are strongly supported by rigorous theoretical proofs. All the proofs and equations appear to be correct.\n- **[S3] Necessity of the Problem Statement**: Guided generation without fine-tuning in diffusion models is an important research area. The proposed method addresses significant limitations in existing work, such as imprecise localization.\n- **[S4] Exhaustive Ablation Studies**: The manuscript thoroughly discusses all key design choices. It provides exhaustive analysis on non-local attention and adaptive updates (Figure 5, Table 2), the effect of $\\rho$ (Figure 6), and hyperparameters for adaptive updates (Figure 7).\n\n\n**Weaknesses**\n\n- **[W1] Missing Baselines**: The proposed method does not include comparisons with Attention-Refocusing [R1], InstanceDiffusion [R2], and MIGC [R3]. The authors must compare their method with these approaches for complete evaluation.\n\n- **[W2] Generation of multiple objects**: Can the proposed method generate images for complex text and layouts, such as \"Two apples on a plate\" or \"Three rabbits wearing sunglasses\"? If not, the authors should discuss this limitation in the manuscript.\n\n[R1] Phung, Q., Ge, S., & Huang, J. B. (2024). Grounded text-to-image synthesis with attention refocusing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 7932-7942).\n\n[R2] Wang, X., Darrell, T., Rambhatla, S. S., Girdhar, R., & Misra, I. (2024). Instancediffusion: Instance-level control for image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 6232-6242).\n\n[R3] Zhou, D., Li, Y., Ma, F., Zhang, X., & Yang, Y. (2024). Migc: Multi-instance generation controller for text-to-image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 6818-6828).", "other_comments_or_suggestions": "**Typos**:\n\n- L436: \"of\" is repeated twice in \"Hyperparameter of of Adaptive Update\".", "questions_for_authors": "- What is the rationale for not comparing the proposed method with Attention-Refocusing [R1], InstanceDiffusion [R2], and MIGC? (Refer to W1 in Strengths and Weaknesses.)\n- Can the proposed method generate multiple objects of the same category? (Refer to W2 in Strengths and Weaknesses.)\n\nIf these concerns are addressed, I am willing to increase my score.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nrqlaSa63L", "reviewer": "Reviewer_QbsF", "summary": "This work aims to achieve high-quality generation for the Layout-to-Image generation task without requiring any training data. It begins by providing a theoretical analysis of existing backward guidance methods and introduces a novel Non-local Attention Energy Function, which enables the model to better respect spatial constraints while maintaining the natural structure of objects. Furthermore, the authors identify that the standard backpropagation update rule can lead to deviations from the pre-trained domain, resulting in visual artifacts. To address this issue, the work proposes a Langevin dynamics-based Adaptive Update scheme, effectively balancing layout adherence and visual realism. Extensive experiments demonstrate the method’s superior performance in both controllability and realism.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to sincerely thank Reviewer QbsF for the thorough and constructive feedback on our manuscript. We are more than happy that the reviewer finds our paper is clearly structured, comprehensive and insightful analysis, and superior performance. We would like to address the concerns as below.\n\n***W1. Other base model.***\n\n**A1.**  In line with prior works, we adopt SD1.5 as the default base model in our experiments to ensure a fair and direct comparison with existing training-free Layout-to-Image methods. This choice is consistent with the evaluation protocols followed by representative baselines such as BoxDiff, Layout-Control, and CSG. To further demonstrate the generality and scalability of our approach, we also extend WinWinLay to more recent and powerful SDXL model. As shown in the teaser figure and in additional visualizations provided in the Appendix, our method consistently delivers superior performance under the SDXL setting as well—achieving strong controllability and realism.\n\n***W2. Small objects.***\n\n**A2.** In fact, both the COCO2014 and Flickr30K datasets used in our experiments naturally contain a significant number of small object instances, such as mouse, coffee cups, and similar items. Therefore, the quantitative results already reported in Tab.1 and Tab.2 implicitly reflect the model’s effectiveness in handling small bounding boxes. To more directly address the reviewer’s concern, we have conducted an additional focused evaluation specifically targeting small-object scenarios. In this experiment, we filtered the datasets to retain only those bounding boxes that occupy no more than 1/8 of the total image area, following a standard definition of small object regions. As expected, all methods exhibit some performance degradation under these more challenging conditions, due to the intrinsic difficulty of generating fine-grained details in small spatial regions. However, our method consistently outperforms all baselines across multiple metrics in this setting as well, demonstrating its robustness, precise spatial control, and superior fine-detail generation capabilities. Specifically, we observe that when the bounding boxes of multiple objects exhibit unnatural relationships—for instance, when a cat and an avocado are assigned bounding boxes of the same size—most existing algorithms suffer from significant performance degradation. In contrast, our method consistently maintains strong performance under such conditions. Visual results are visible on this [link](https://anonymous.4open.science/r/WinWinLay/Small%20objects/Small%20objects.png).\n\n|            Model             | Multidiffusion | BoxDiff | Layout-Control |  CSG  | **Ours**  |\n| :--------------------------: | :------------: | :-----: | :------------: | :---: | :-------: |\n|    COCO2014(AP$\\uparrow$)    |      12.2      |   8.1   |      6.9       | 15.7  | **18.2**  |\n|  COCO2014(CLIP-s$\\uparrow$)  |     0.267      |  0.281  |     0.279      | 0.274 | **0.296** |\n|   Flicker30K(AP$\\uparrow$)   |      10.7      |  13.3   |      13.6      | 14.1  | **16.4**  |\n| Flicker30K(CLIP-s$\\uparrow$) |     0.262      |  0.272  |     0.277      | 0.270 | **0.289** |\n\nThanks again for the insightful review. We are happy to discuss any aspects of the manuscript that may require further clarification.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "Yes", "experimental_designs_or_analyses": "Yes", "relation_to_broader_scientific_literature": "Layout-to-Image generation has emerged as a prominent research area. This work offers a theoretical analysis of existing backward guidance methods and proposes novel algorithms to address their limitations.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "Strength:\n- The paper is clearly structured and easy to follow, providing a smooth and coherent flow of ideas.\n- The analysis presented in this work is both comprehensive and insightful. It provides a deep investigation into the limitations of existing methods and introduces effective solutions that significantly improve generation quality and controllability, including Non-local Attention Energy Function and Langevin dynamics-based Adaptive Update scheme. Together, these innovations lead to more controllable and higher-quality generation outcomes.\n- Experimental results demonstrate that the proposed method achieves superior performance compared to state-of-the-art techniques, both qualitatively and quantitatively, highlighting its effectiveness.\n\nWeakness:\n- The proposed method is evaluated using Stable Diffusion 1.5, which is relatively outdated. Have the authors considered implementing their approach on more recent models, such as Stable Diffusion 3/3.5 or FLUX?", "other_comments_or_suggestions": "No.", "questions_for_authors": "The examples shown in the visualizations involve relatively large bounding boxes, which may provide more contextual cues to the model. However, it remains unclear how the method performs with smaller bounding boxes corresponding to small objects (e.g., a mug on a table). Have the authors evaluated the model’s generation quality and spatial controllability under such scenarios?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dH7GRaBGJj", "reviewer": "Reviewer_YGgi", "summary": "This work introduces WinWinLay, a novel training-free optimization strategy for layout-to-image generation using text-to-image diffusion models. The paper tackles two main drawback of previous approaches for layout-to-image generation: (1) that the generated objects are often not precisely places within the given bounding boxes, and (2) that there is clear trade-off between the controllability from the layout and the overall quality of the image. First, the authors provide a theoretical analysis on the backward guidance, and propose a non-local attention energy function for better adherence to the bounding boxes. Then, to mitigate the quality-accuracy trade-off, the authors introduce the adaptive update based on Langevin-dynamics. The experimental results show that WinWinLay sets the state-of-the-art on training-free layout-to-image generation.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to sincerely thank Reviewer Reviewer YGgi for the thorough and constructive feedback on our manuscript. We are more than happy that the reviewer finds our problem definition is reasonabl, theoretical analysis illustrative, and superior performance. We would like to address the concerns as below.\n\n***W1. Non-local attention prior.***\n\n**A1.** We acknowledge that for certain object categories—such as coconut trees *(As discussed in lines 80-84)* and cars —the center of mass may deviate from the geometric center of the bounding box due to their asymmetric or elongated structures, making strict centralization suboptimal. To address this, we adopt a decaying schedule for weight $\\rho$ of non-local attention prior *(Lines 225–229)*, which applies stronger centralizing guidance during early denoising and gradually relaxes it as generation progresses. This design allows the model to first stabilize attention and then adapt to the object’s natural shape and pose. *As shown in Fig.4*, our method successfully generates well-aligned objects like cars and astronauts without rigidly adhering to box center. Moreover, our *ablation in Fig.6* shows that maintaining a high prior weight throughout can degrade results, confirming the effectiveness of our decay strategy. In summary, the non-local attention prior functions as a soft, adaptive constraint, enabling both precise spatial control and natural object generation across diverse categories.\n\n***W2. Ablation of Nash-MTL.*** \n\n**A2.** Adaptive update is a key component of WinWinLay, designed to balance layout controllability and visual quality with minimal computational overhead. *As shown in Tab.2*, our ablation study demonstrates that incorporating this strategy consistently improves both spatial fidelity and semantic alignment. Moreover, *Fig.7* shows that it reduces the number of update steps by dynamically adjusting optimization directions, leading to faster convergence. We further report actual inference times *in Tab.3*, confirming that adaptive update significantly improves generation efficiency.\n\n***W3. Example in Fig.2.***\n\n**A3.** Adequate object coverage within the bounding box is crucial for precise and reliable user control. In our evaluation, metrics like AP depend on the IoU between generated objects and target boxes—if an object covers only part of the box, IoU drops, leading to lower AP. This not only affects quantitative results but also indicates a mismatch with user-defined spatial intent, particularly in scenes with dense object layouts requiring fine-grained control.\n\n***W4. Compared with R&B.***\n\n**A4.**  To clarify the distinctions and advantages of WinWinLay, we provide a focused analysis of the two core components of R&B: Region-aware Loss (RAL) and Boundary-aware Loss (BAL).\n\n(1) RAL leverages the IoU between the predicted box $\\hat{\\mathcal{B}}$ and the ground-truth box $\\mathcal{B}$ to guide object placement. However, this mechanism introduces two critical limitations:\n\n- The non-differentiability of bounding box operations makes gradient-based updates less reliable and harder to tune.\n- Similar to traditional attention energy functions, RAL does not mitigate local bias, leading to incomplete or inaccurate coverage within the target region.\n\n(2) BAL employs a Sobel filter to enhance attention contrast at object boundaries within the box, with the goal of improving object realism. However, it too adopts an energy function-based formulation, and as such, suffers from similar drawbacks:\n\n- Due to the non-differentiability of IoU, BAL reduces to a heuristic energy-based form, which continues to suffer from local bias and incomplete box filling.\n- BAL does not account for the underlying latent distribution $z_{t}$, and therefore cannot effectively navigate the trade-off between layout adherence and image fidelity.\n\nTogether, these differences underscore the principled, efficient, and robust nature of WinWinLay in addressing the limitations that R&B is inherently constrained by. We have now included this discussion in revised manuscript.\n\n***W5. More baselines and metrics.***\n\n**A5.** Please see A1 for Reviewer hxyx.\n\n***W6. Details of user study.***\n\n**A6.** To ensure clarity and reproducibility, we conducted the user study on Wenjuanxing, a platform similar to Amazon Mechanical Turk. 150 participants evaluated 50 image pairs, yielding 7500 responses per study. For each pair, users answered two questions:\n\n- Which image better matches the bounding box layout?\n- Which image has higher visual quality?\n\nImages were shown side-by-side with layout prompts, and both question order and image positions were randomized to avoid bias. These details have been included in revised manuscript for completeness.\n\nThanks again for the insightful review. We are happy to discuss any aspects of the manuscript that may require further clarification.", "claims_and_evidence": "1. The problem definition of the paper is reasonable. Training-free layout-to-image generation is a practically important application, and there have been limitations in the layout adherence and the image quality, and a trade-off between the two.\n\n2. The authors provide a thorough analysis on the optimization behavior of the attention energy function and why it becomes difficult for the generated object to cover the whole bounding box region. The proposed solution, non-local attention prior, is a intuitive choice to mitigate this. However, to claim its novelty and clear effectiveness compared to previous approaches, further justications on the design of the prior distribution $\\tau_u$ and comparisons with previous works is required. For further details on this, please refer to the below section.\n\n3. The authors identify the critical issue of the trade-off between generation (layout) control and image quality. Applying the Langevin dynamics-based update to alleviate this trade-off seems to be a clear and valid approach.\n\n4. While bringing the idea of Nash-MTL seems novel and acts as a key technical detail of WinWinLay, it is hard to understand its effectiveness clearly. Perhaps an ablation study with and without applying this idea could help undertand better.", "methods_and_evaluation_criteria": "1. Introducing the \"non-local attention prior\" for encouraging the object to cover a larger region within the bounding box seems as a reasonable choice. However, defining the prior distribution to be centered at the \"centroid\" of the bounding box would need a better justification. It seems to be quite a hard constraint when we want an object to be generated within the box. For some objects, it is more intuitive to think that its center of mass is at the lower region of the box (e.g. a car) rather than the center.  Have the authors encountered any failure cases in which enforcing the center to be at the centroid of the box led to degradation of the objects?\n\n2. While one of the key objectives of \"non-local attention prior\" is to maximize the coverage of the bounding box, there is no discussion and comparison to R&B [1], which also addresses a similar issue by introducing a boundary-aware loss using the Sobel operator. Since R&B is the state-of-the-art method on training-free layout-to-image generation, it is crucial to discuss which aspect WinWinLay has enabled it to obtain results that R&B had failed to achieve.\n\n3. The paper lacks critical baselines for comparison. Please refer to the \"Essential References\" section for this.\n\n4. For the user study to have validify, the authors should provide more details on its setup. For instance, how were the participants gathered (e.g. Amazon Mechanical Turk), and how many people participated? What was the question format and how many images were presented at each question?\n\n5. In addition to user study, there are multiple metrics that can be used to measure the image quality. FID, PickScore [2] and ImageReward [3] are widely used metrics for this. User study alone seems insufficient to claim the clear advantage of WinWinLay compared to the baselines.\n\n[1] R&B: Region and Boundary Aware Zero-shot Grounded Text-to-image Generation, Xiao et al., ICLR 2024\n\n[2] Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generation, Kirstain et al., NeurIPS 2023\n\n[3] ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generation, Xu et al., NeurIPS 2023", "theoretical_claims": "Each theoretical claim in the paper is supported by a proof, which seem to be valid.", "experimental_designs_or_analyses": "1. The paper doesn't present sufficient exploration on the previous state-of-the-art methods, and misses out important baselines. Please see the \"Essential References\" section for this.\n\n2. Since enhancing the image quality is a key objective of WinWinLay, it is expected to show more concrete comparisons on image quality compared to the baselines. Other than the user study, there are multiple widely used metrics to measure the image quality and diversity. I have discussed this in the above section.", "relation_to_broader_scientific_literature": "In the broader field of image generation, enhancing user control is important. Suggesting accurate layout-based controls while preserving the image quality would extend the applicability of text-to-image model to practical use cases in which the users requires precise region control.", "essential_references_not_discussed": "The paper lacks critical references to the previous works on the topic of \"training-free layout-to-image generation\", which should have also been considered as baselines for quantitative/qualititative comparison. Notably, Attention-Refocusing [1] and R&B [2] were previous works on the same topic, both published at a conference in 2024. These works also aim to address the problem of limited adherence to the bounding box inputs, and the degrated image quality due to the introduction of the backward guidance.\n\nTherefore, without discussions on these prior works and comparisons between the outputs of WinWinLay and these baselines, it would be difficult to claim the novelty and the effectiveness of the proposed approach.\n\n[1] Grounded Text-to-Image Synthesis with Attention Refocusing, Phung et al., CVPR 2024 \n\n[2] R&B: Region and Boundary Aware Zero-shot Grounded Text-to-image Generation, Xiao et al., ICLR 2024", "other_strengths_and_weaknesses": "I have discusses the main points in the above sections.", "other_comments_or_suggestions": "The example in Fig. 2 may not be the best choice for claiming that it is important for each object to covered the whole bounding box. Even if the rabbit does not fully fit the box, it is likely to satisfy the user's intentions. If there is an example in which the situation or the story in the image may change when the object doesn't fit the box, it could be a better example showing the importance of maximum coverage.", "questions_for_authors": "Please refer to the above sections.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper presents an interesting contribution to the field of training-free layout-to-image generation. The proposed WinWinLay method effectively addresses the limitations of existing approaches from its novel non-local attention energy function and update mechanism. The authors provide theoretical grounding for their method, and demonstrate its effectiveness through comprehensive experimental evaluations. While the reviewers initially expressed some concerns regarding novelty, missing baselines, and evaluation details, they were mostly satisfied with the authors' responses and clarifications provided in the rebuttal. 

Based on the reviewers' feedback and the several round discussions, it is a clear case for acceptance. I’d recommend the authors to address the reviewers’ comments in the revision.
Decision: Accept (poster)