RealCompo: Balancing Realism and Compositionality Improves Text-to-Image Diffusion Models
Authors: Xinchen Zhang
Ling Yang
YaQi Cai
Zhaochen Yu
Kai-Ni Wang
xie jiake
Ye Tian
Minkai Xu
Yong Tang
Yujiu Yang
Bin CUI
Venue: NeurIPS 2024 poster (2024)
Keywords: Text-to-Image Diffusion
Layout-guided Image Diffusion
Primary area: diffusion_based_models
TL;DR: We propose a new training-free and transferred-friendly text-to-image generation framework to enhance the realism and compositionality of the generated images.

Abstract: Diffusion models have achieved remarkable advancements in text-to-image generation. However, existing models still have many difficulties when faced with multiple-object compositional generation. In this paper, we propose ***RealCompo***, a new *training-free* and *transferred-friendly* text-to-image generation framework, which aims to leverage the respective advantages of text-to-image models and spatial-aware image diffusion models (e.g., layout, keypoints and segmentation maps) to enhance both realism and compositionality of the generated images. An intuitive and novel *balancer* is proposed to dynamically balance the strengths of the two models in denoising process, allowing plug-and-play use of any model without extra training. Extensive experiments show that our RealCompo consistently outperforms state-of-the-art text-to-image models and spatial-aware image diffusion models in multiple-object compositional generation while keeping satisfactory realism and compositionality of the generated images. Notably, our RealCompo can be seamlessly extended with a wide range of spatial-aware image diffusion models and stylized diffusion models. Code is available at: https://github.com/YangLing0818/RealCompo

Real Compo: Balancing Realism and Compositionality
Improves Text-to-Image Diffusion Models
Xinchen Zhang!*
Ling Yang** Yagi Cai
Zhaochen Yu² Kai-Ni Wang
Jiake Xie
Ye Tian Minkai Xu Yong Tang Yujiu Yang't Bin Cui
Tsinghua University 2 Peking University 3 University of Science and Technology of China
Southeast University LibAI Lab Stanford University
Abstract
Diffusion models have achieved remarkable advancements in text-to-image gen- eration. However, existing models still have many difficulties when faced with multiple-object compositional generation. In this paper, we propose RealCompo, a new training-free and which aims leverage the respective advantages of text-to-image models and spatial-aware image diffusion models (e.g.. layout, keypoints and segmentation maps) to enhance both realism and compositionality of the generated images. An intuitive and novel balancer is proposed to dynamically balance the strengths of the two models denoising process, allowing plug-and and-play use of any model without extra training. Extensive experiments show that our RealCompo consistently out- performs state-of-the-art text-to-image models and spatial-aware image diffusion models in multiple-object compositional generation while keeping satisfactory realism and compositionality of the generated images. Notably, our RealCompo can be seamlessly extended with a wide range of spatial- image diffusion models and stylized diffusion models.
text-to-image generation framework,
Introduction
The field of diffusion models has witnessed exciting developments and significant advancements recently[65, 46, 19, 45, 40, 73]. Among various generative tasks, text-to-image (T2I) generation [33, 20, 64) has gained considerable interest within the community. T2I diffusion models such as Stable Diffusion [41], Imagen [42] and DALL-E 2/3 [39, 4] have exhibited powerful capabilities in generating images with high aesthetic quality and realism [4, 36). However, they often struggle to align accurately with the compositional prompt when it involves multiple objects complex relationships [28. 3, 34], which requires the model to have strong spatial-awand ability.
One potential solution to optimize the compositionality of generated images is providing spatial- aware condition control diffusion models [12, 66, 58], such as layout/boxes [35, 14], keypoint/pose [72] and segmentation map [22]. These spatial-aware conditions are fundamentally similar in functioning, thus we mainly focus our analysis on layout-to-image (L2I) models for simplicity. With the control of layout, 1.21 models [27, 8, 59] improve compositionality by generating objects at specified locations. For instance, GLIGEN [27] designs trainable gated self-attention layers to incorporate layout input and controls the strength of its incorporation by changing parameter B. Although L2I models improve the weaknesses compositional text-to-image generation, their generated images exhibit a significant decline in realism compared to T2I models [27, 78).
"Contributed equally. 'Corresponding authors.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Prompt: On a wooden table, there white vase with pink flower inserted it, and sitting on the table the right the
(R)
Table
0.0
0.2
0.4
0.6
0.8
1.0
Tlover
(b)
nelc
Figure I: Motivations of RealCompo. (a) and (c) The realism and aesthetic quality of generated images become poor as more layout incorporated. (b) Even if layout is incorporated only in the early denoising stages, the control of text alone still fails to alleviate the poor realism issue. More results are shown in Appendix B.
We conducted experiments to analyze why a significant decrease in image realism exists. We analyze the layout injection mechanism in GLIGEN [27] by controlling the density of layout through parameter B. As shown in Fig. (a) and (c), our experiments indicate that the density of layout directly influences the realism of generated images. As the control of layout gradually increases, the generated images become less aesthetic and more unstable. This demonstrates that layout and text, different control conditions, guide the model towards different generation directions, with the former emphasizing compositionality and the latter emphasizing realism. To alleviate this issue, some models [28, 27] leverage the early-stage localization capability of diffusion models [71, 49] and incorporate layouts only during the initial denoising phase. In the later denoising stage, only use text balance image realism. However, we found this approach yielded minimal effectiveness. We assumed B 1 in the first denoising steps and B = 0 the subsequent denoising steps. As shown in Fig. (b), the object's position is already determined around 20 steps. However, it is common that the generated images exhibit almost no difference between / = 20 and = 50. This suggests that even when the injection of layout is stopped in the later denoising stages, the control of text alone still fails to alleviate the poor realism issue. The trade-off between realism and compositionality in T2I and L2I models is challenging yet necessary.
To this end, we introduce a general training-free and transferred-friendee text-to-image generation framework RealCompo, which utilizes a novel balancer achieve dynamic equilibrium between realism and compositionality in generated images. We first utilize LLMs to generate scene layouts from text prompt through in-context learning [32]. Then we propose innovative balancer to dynamically compose pre-trained fidelity-aware (T21, stylized T2I) and spatial-aware (e.g.. layout, keypoint, segmentation map) image diffusion models. This balancer automatically adjusts the coefficient of the predicted noise for each model by analyzing their cross-attention maps during the denoising stage. By combining the respective strengths of the models, it achieves trade-off between realism and compositionality. Finally, we extend RealCompo to various spatial-aware conditions through a general compositional denoising process. Moreover, by changing the T2I model to stylized T21 model, Realcompo can seamlessly achieve compositional generation specified with a particular style. These dramatically demonstrate the great generalization ability of RealCompo. A Although there exist methods [61, 2] for composing multiple diffusion models, their application lacks flexibility because they require additional training and cannot be generalized other conditionss and models. Our method effectively composes two models in a training-fiee free manner, allowing for seamless transition between various models.
To the best of our knowledge, RealCompo effectively achieves a trade-off between realism and compositionality in text-to-image generation. Choosing one (stylized) T2I model and one spatial- aware (c.g., layout, keypoint, segmentation map) image diffusion model, RealCompo automatically balances their fidelity and spatial-awareness realize collaborative generation. We expands the family of model ensembling/checkpoin merging techniques, which are extensively used in the diffusion community. We believe RealCompo opens up new research perspective in controllable and compositional image generation.
Our main contributions are summarized as the following:
We introduce a new training-free and transferred text-to-image generation frame- work RealCompo, which enhances compositional text-to-image generation by balancing the realism and compositionality of generated images.

---

We design a novel balancer to dynamically combine the predict noise from T2I model and spatial-aware (e.g., layout, keypoint, segmentation map) image diffusion model.
RealCompo has strong flexibility, can be generalized balance various (stylized) T2I models and spatial-aware image diffusion models and can achieve high-quality compositional stylized generation. It provides fresh perspective for compositional image generation.
Extensive qualitative and quantitative comparisons with previous outstanding methods demonstrate that RealCompo has significantly improved the performance in generating multiple objects and complex relationships.
2 Related Work
Text-to-Image Generation In recent years, the field of text-to-image generation has made remark- able progress [47, 60, 36, 18, 11, 74, 63], largely attributed to breakthroughs in diffusion models. By training on large-scale image-text paired datasets, T21 models such Stable Diffusion (SD) [41], DALL-E 2/3 [39, 4], MDM [17], and Pixart-o [7], have demonstrated remarkable generative capabil- for improvement in compositional generation when text prompts include multiple objects and complex relationships [58]. Many studies have attempted to address this issue through controllable generation [72] by providing additional conditions such as segmentation map [22]. scene graph [62], layout [77], etc., to constrain the model's generative direc- tion ensure the accuracy of the number and position of objects in the generated images. However, due to the constraints of the additional conditions, image realism may decrease [27]. Furthermore, several works [37, 9, 68, 65, 30] have attempted to bridge the language understanding gap in models by pre-processing prompts with Large Language Models (LLMs) [1, 48]. It is challenging for T2I models to achieve trade-off between realism and compositionality [65] of generated images.
ities. However, there is still significant
Compositional Text-to-Image Generation Recently, numerous methods have been introduced to improve compositional text-to-image generation [53, 78, 69, 55, 25, 29]. These methods enhance diffusion models in attribute binding, object relationship, numeracy, and complex prompts. Recent studies can generally be divided into two types [52]: one primarily uses cross-attention maps for compositional generation [31, 24, 76], while the other provides more conditions (e.g., layout, keypoint, segmentation map) to achieve controllable generation [16, 78]. The first methods delve into a detailed analysis of pross-attention maps, particularly emphasizing their correspondence with the text prompt. Attend-and -a-d--xd-dtetee [6] dynamically intervenes in the generation process to improve the model's generation results in terms of attribute binding (such as color). Most of the second methods offer layout constraint, enabling the model to generate images that meet this condition. This approach directly defines the area where objects are located, making more straightforward and observable compared to the first type of methods [27]. LMD [28] provides an additional layout as input with LLMs. Afterward, a controller is designed to predict the masked latent for each object's bounding box and combine them in the denoising process. However, these algorithms are unsatisfactory in the realism generated images. A recent powerful framework RPG [65] utilizes Multimodal LLMs to decompose complex generation tasks into simpler subtasks to obtain satisfactory realism and compositionallit of generated images. Orthogonal to this work, we achieve dynamic equilibrium between realism and compositionality by combining T2I and spatial-aware image diffusion models.
3 Method
In this section, we introduce our method, RealCompo, which designs a novel balancer to achieve dynamic equilibrium between realism and compositionality of generated images. We initially focus on the layout-to- to-image models. In Section 3.1, we analyze the necessity of incorporating influence for the predictive noise of each model and provide a method for calculating coefficients. In Section 3.2, we provide a detailed explanation of the update rules employed by the balancer, which utilizes a training-free approach to update coefficients dynamically.
Section 3.3, we provide a universal formula and denoising procedure that enable the balance of T2I models with any spatial-aware image diffusion model, such as keypoint or segmentation-to -i-imag models based on ControlNet [72]. We also extend RealCompo to stylized compositional generation by stylized T2I models.
3

---

Prompt: womsn with long goldian sitting front is stone table, which right a orange
burningg
sitting
Fidelity
Teat
Rediam
LLMs
(Styitzed
Test-t--mma
SD v1. S0
50XL
Sty food Models...
f(C)
Layout -KKypoitt -to En-Enn
SSggestation
GLEGEN LaCazto
Costroleet
Loyou?
Spatial-aware
Comporttinniig
Figure 2: An overview of RealCompo framework for text-to-image generation. We first use LLMs transfer function obtain the corresponding layout. Next, the balancer dynamically updates the influence two models, which enhances realism by focusing on contours and colors in the fidelity branch, and improves compositionality by manipulating object positions in the spatial-aware branch.
3.1 Combination of Fidelity and Spatial-Awarenes
LLM-based Layout Generation. Since spatial-aware conditions are similar essentially, we first choose layout the representative of spatial-aware condition for introduction. As shown in Fig. 2, we leverage the powerful in-context learning [57, 79] capability of Large Language Models (LLMs) to analyze the imput text prompt and generate accurate layout to achieve "pre-binding" between objects and attributes. The layout is then used as input for the L2I model. In this paper, we choose GPT-4 for layout generation. Please refer Appendix C.I for detailed explanation.
Combination of Two Types Noise. In diffusion models, the model's predicted noise 6, directly affects the direction of the generated images. In T2I models, c, exhibits more directive toward realism [41], whereas in L2I models, dayyuu demonstrates more directive toward compositionality [27]. To achieve the trade-off between realism and compositionality, a feasible but untapped solution compose the predicted noise of two models. However, the predicted noise from different models has its own generative direction, contributing differently to the generated results at different timesteps and positions. Based on this, we design novel balancer that achieves dynamic equilibrium between the two models' strengths at every position i in the noise for timestep f. This is achieved by analyzing the influence of each model's predicted noise. Specifically, we first the same coefficient for the predicted noise of each model to represent their influence before the first denoising step:
Coer
text
Coe,
layout
In order regularize the influence of each model, we perform softmax operation on the coefficients to get the final coefficients:
exp(Coei)
c:
exp(Coee(()) + exp (oe layoul
where c E {text, layout).
The balanced noise can be derived according to the coefficient of each model:
St
text
Ex
text
Se
layout
E
layout
where 2 denotes wixel-wise multiplication.
Once the predicted noise Ef and the coefficient Coe of each model are provided, the balanced noise can be derived from Eq. 2 and Eq. 3. At timestep I, the balancer dynamically updates coefficients as described in Section 3.2.

---

3.2 Influence Estimation with Dynamic Balancer
The alignment between the generated images and the input prompts is largely influenced by model's cross-attention maps, which encapsulate wealth of matching information between visual and textual elements, such as location and shape. Specifically, given the intermediate feature co(z)) and the text embeddings To (y), cross-attention maps can be derived in the following manner.
(2(KOT
A = Softmax
,cE ftext, layout)
(4)
vag.
= = o0 (2t), k= WK To(y)
(5)
where Q and К are respectively the dot product results of the intermediate feature (z(z), text embeddings (y)). and two learnable matrices Wo and WK. Ai defines the weight of the value of the j-th token on the i-th pixel. Here, E 21,2..... N (T+(y)) and N (T+(y)) denotes the number of tokens in (y)). The dimension of K is represented by dg-
Update Rule of Dynamic Balancer. We designed novel balancer that dynamically balances two models according to their cross-attention maps timestep Specifically, we represent layout as which is composed of bounding boxes b. Each bounding box b corresponds
B = (br, be...., bb),, ) binary mask Mr. where the value inside the box is and the value outside the box is Given the predicted noise ex and the coefficient Coe, of each model, the balanced noise e, and denoised latent 2 can derived from Eq. and Eq. 12. By feeding 2t-1 into two models, we obtain the cross-attention maps Af_1 output by the two models at timestep / 1, which indicates the denoising quality feedback after the noise ef of the model at time is weighted by E. Based on A4-1, we define the loss function follows:
A Σ,
(M6,t-1) ② M.
C(A layout ΣΣ
(6)
A. (166.111)
where e {text, aayout).j. Js denotes the token corresponding the object in bounding box b. Since two models are controlled by different conditions, averaging the predicted noise equally will lead to instability in the generated images. This is because the T2I model breaks the layout constraints of the L21 model, reducing the compositionality of the generated images, as we have demonstrated in experiments in Fig. 9. Therefore, we designed this loss function to measure the alignment between the cross-attention maps and layout for each model. A smaller loss indicates better compositionality. The following rule used to update Coef:
Coef = =Coe - Pe Coce ((A
layout
where is the updating rate. This update rule continuously strengthens the constraints on both models by assessing the positional alignment of the layout within the cross-attention maps, ensuring the maintenance of the localization capability of L.21 model while injecting fidelity information of T21 model. It is worth noting that previous methods [6, 59, 28] for parameter updates based on function gradients were primarily using energy functions to update latent z,. We are the first to update the influence of predicted noise based the gradient of the loss function, which is novel and stable method well-suited to our task. The complete denoising process is detailed in Appendix C.3.
3.3 Extend RealCompo to any Spatial-Aware Conditions in a General Form
Other spatial-aware text-to-image diffusion models are essentially similar to L21 models. Keypoint- to-image (K21) models generate specified actions or poses within each group of keypoints region, and segmentation-to-image (S2I) models fill indicated objects within each segmented region. The concept of "region" always present, which transforms T2I generation from a macro perspective to utilizing region-based control for T2I generation from micro perspective. This concept is also the core of enhancing image compositionality. Compared with lasout-based T2I generation, the only difference is that keypoints and segmentation maps have stronger control over the model based on regions, requiring that the pose is maintained and the object is correct and unique.
5

---

Keypoint
ControlNet
RealCompo(Ours)
Segmentation
SD
ControlNet
RealCompo((urs)
gin ongelic aspconance
getteer
A house with dark red walls surrounded by flowers and frees.
creating ohmosphare.
group of four friends, girls and two beys wearing cosual clathing, loughing together they walk clong flower lined path beneath a chenry blossom friee full bleon.
In the living room there are Two white sofes and white coffee toble, and the living room has two oversized floor colling windows.
Figure 3: Extend RealCompo to keypoint- and segmentation-based image generation.
General Form for Extension to Other Spatial-
Fidelity
Spatial- aware
Aware Conditions We rethink Eq. 6, which is
zero convolution
RealCompo's core approach combining T2I and 1.21 models, where the only layout-related variable is the binary masks M. Considering that spatial-aware controllable T2I generation inherently focus on the concept of "region con- trol", we introduce a transfer function:
neural network block (locked)
neural network block (locked)
neural network (locked)
convolution
RealCompo
M = f(C
(8)
ControlNet
where represents other spatial-aware condi- tions such as keypoint and segmentation map. f(+) represents calculation of the minimum and maximum values of the horizontal and ver- tical coordinates occupied by each set of keypoints or a segmentation block within the entire image coordinate system, which can be transformed into layout and binary mask M. Therefore, for any T2I models with spatial- aware control, the general loss function of RealCompo is:
Composs
Figure 4: RealCompo constructed Nen
E: Ar
(Njo.t-1) AiAisoot-1 C fc(()
C(A A--1
spatial,
ΣΣ
E; AiA.see0-)
where E (text, spatial]. Similarly, Coe is dynamically updated using Eq. ControlNet [72] enables controllable T21 generation based on various spatial-awars conditions. In this work, the spatial-awar branches besides layout are all based on ControlNet, which illustrated in Fig. 4. The generated images of keypoint- and segmentation-bun based RealCompo are shown in Fig. 3.
Extend RealCompo to Stylized Image Generation As an essential indicator fidelity, image style [50, 67] guides us to expand the application potential of RealCommo. Since RealCompo mainly leverages T2I models enhance and guide the realism and aesthetic quality of generated images. By replacing the T2I model with various stylized T2I models and combining it with spatial- aware image diffusion model, we can achieve outstanding compositional generation under this style. The experiments are shown in Fig 8.
4 Experiments
4.1 Experimental Setup
Implementation Details Our RealCompo is a generic, scalable framework that can achieve the
complementary advantages of the model with any chosen (stylized) T2I models and spatial-aware image diffusion models. We selected GPT-4 [1] as the layout generator in our experiments, the detailed rules are described in Appendix C.I. For layout-based RealCompo, we chose SD v1.5 [41] and GLIGEN [27] the backbone. For keypoint-based RealCompo, chose SDXL [4] and

---

Table 1: Evaluation results about compositionality on T21-CompBench [21]. RealCompo consistently demonstrates the best performance regarding attribute binding, object relationships, numeracy and complex compositions. We denote the best score in blue and the second-best score in green The baseline data is quoted from PixArt-a [7].
Attribute Binding
Object Relationship
Model
Numeracyt
Complext
Color
Shapet
Texture
Spatial
Non-Spatial
Stable Diffusion v1.4 [41]
0.3765
0,3576
0,4156
0.1246
0,3079
0.4461
0,3080
Stable Diffusion v2 [41]
0.5065
0.4221
0.4922
0.1342
0.3096
0.4579
0.3386
Structured Diffusion [13]
0.4990
0.4218
0.4900
0.1386
0,3111
0.4550
0.3355
Attn-Exct v2 [6]
0.6400
0.4517
0.5963
0.1455
0.3109
0.4767
0.3401
DALL-E [39]
0.5750
0.5464
0.6374
0.1283
0.3043
0.4873
0,3696
Stable Diffusion XL [4]
0.6369
0.5408
0.5637
0.2032
0,3110
0,4988
0,4091
PixArt-a [7]
0.6886
0.5582
0.7044
0.2082
0.3179
0.5058
0.4117
GLIGEN[27]
0,4288
0.3998
0.3904
0.2632
0.3036
0,4970
0.3420
LMD+[28]
0.4814
0.4865
0.5699
0.2537
0,2828
0.5762
0.3323
RealCompo (Ours)
0.7741
0.6032
0.7427
0.3173
0.3294
0.6592
0.4657
Layout
GLIGEN
LMD-
RealGomp:(Ours)
v1.5
GLIGEN
LondScape with frees growing both sides small river, and snowy distonce.
robbit wearing Sunglasses surbathing hing on beach by the lake.
his wife holding
looking stained glass window and marveling
table yellow purple flowers.
yellow, white,
Three parked front houses.
Figure 5: Qualitative comparison between our RealCompo and the outstanding text-to-image model Stable Diffusion v1.5 [41], as well as the layout-to-iman models, GLIGEN [27] and LMD+ [28]. Colored text denotes the advantages of RealCompo generated images.
ControlNet [72] as the backhone. For segmentation-baset RealCompo, we chose SD v2.1 [41] and ControlNet [72] as the backbone. For style-based RealCompo, we chose two stylized T2I models: Coloring Page Diffusion and Cute YukiMix as the backbone, and chose GLIGEN [27] as the backbone of L2I model. All our experiments are conducted under NVIDIA 80G-A100 GPU.
Baselines and Benchmark To evaluate compositionality, we compare our RealCompo with the outstanding T2I and L2I models on T21-CompBench [21]. This benchmark test models across aspects of attribute binding, object relationship. numeracy and complexity. To evaluate realism, we randomly select 3K text prompts from the COCO validation set we utilize ViT-B-32 [10] to calculate the CLIP score and LAION aesthetic predictor calculate aesthetic score, reflecting the degree of match between generated images and prompts as well as the aesthetic quality, respectively. In addition to objective evaluations, we conducted a user study to evaluate RealCompo and stylized RealCompo in terms of realism, compositionality, and comprehensive evaluation.
4.2 Main Results
Results of Compositionality: T2I-CompBess BenCchh We conducted tests on T21-CompBeneh [21] to evaluate the compositionality of RealCompo compared to the outstanding T2I and L.1 models. As demonstrated in Table 1, RealCompo achieved tate--of-the-att performance
all seven evaluation
tasks. It is clear that RealCompo and 1.21 models GLIGEN [27] and LMD+ [28] show significant improvements in spatial-aware tasks such as spatial and numeracy. These improvements are largely
7

---

T21
Styliaed
Spattal aware model
60.0%
Stylized L21 model
60.0%
ReaCCompo
RealCompo
Percentage
Percentage
40.0%
40.0%
20.0%
20.0%
0.0%
0.0%
Realism
Compasitionality
Comprehansive Evaluation
Realism
Compasitionality
Comprehensive Evaluation
(Style Preservation)
Figure 6: Results of user study.
Layout
T2I model
(eokCompo(Ours)
Loyout
T27 model
RealCompo(Curs)
Keypsint
T2E model
(err)Comps(Ohrs)))
siptop
PMC
(s)
(6)
(6]
Figure 7: Text-to-image models often generate unrealistic images due to unreasonable object positions. Our method improves image authenticity through conditional control while maintaining detail and aesthetic quality.
attributed to the guidance provided by the additional conditions, which greatly enhances the model's compositional performance. RealCompo employs balancer for better control positioning, boosting its advantages in these aspects. However, the L2I models exhibit noticeable decline in performance on tasks like texture and non-spatial. This decline is due the injection of layout embed- dings, which dilute the density of text embeddings, leading to suboptimal semantic understanding by the model. By composing additional T2I models, RealCompo provides sufficient textual information during the denoising process and achieves outstanding results in tasks that reflect realism, such as texture, non-spatial and complex tasks. As shown in Fig. 5, compared with the current outstanding L2I models GLIGEN and LMD+, RealCompo achieves high level of realism while keeping the attributes of the objects matched and the number of positions generated correctly.
Quantitative Comparison As shown in Table 2, our model significantly outperforms ex-
Results of Realism:
Table 2: Evaluation results on image realism.
isting outstanding T2I and L2I models in both CLIP score and aesthetic score. We attribute this to the dynamic bal- ancer, which enhances image realism and aesthetic quality while maintaining high compositionality.
Model
CLIP Scoret
Aesthetic Scoref
Stable Diffusion vi [41]
0.307
5.326
TokenCompose v2.1 [54]
0.323
5.067
Diffusion [41]
Stable Diffusion XL [4]
0.322
5.531
Layont Guidance[8]
0.294
4,947
GLIGEN[27]
0.301
4,892
LMD+(28)
User Study In addition objective evaluations, we designed user study to subjectively assess the practical per- formance of various methods. We ran-
RealCompo (Ours)
0.334
5,742
domly selected 15 prompts, including 5 for stylization experiments. Comparative tests were conducted using T2I models, spatial-aware image diffusion models, and RealCompo. We invited 39 users from diverse backgrounds to vote on image realism, image compositionality, and comprehensive evaluation, resulting in total of 1755 votes. As illustrated in Fig. 6, RealCompo received widespread user approval in terms of realism and compositionality.
Reasonable Composition Improves Realism We provide examples from the user study Fig. 7, which demonstrates the advantages of RealCompo over the T2I model in realism. As shown in Fig. 7(a), T2I model generates a teapot that visibly suspended in the air, which doesn't 't conform to the physical laws of real-worl world scenes. In contrast, RealCompo generates objects within reasonable

---

Layout
InstontStyyle
RealCompo(Ours)
Style: Coloring Page Diffusion (SD v1.5)
house
car
Coloring page park in front of house.
red moon
Style: CuteYukiMix (5D v1.5)
vampire girl
black wing
black coat
vampire girl with black wing and black coat, moon, castle-background.
Figure 8: Extend RealCompo stylized compositional generation.
bounds through layout constraints, ensuring both the aesthetic quality and positional reasonableness. In Fig. 7(b), the red chair generated by the T2I model is unnaturally placed on top of the table, and in too close each other. These examples illustrate
Fig. 7(c), two people generated by the T2I model that although T2I model outperforms in detail and visual refinement, its positional reasonableness needs improvement. Our method utilizes LLM generate conditions that comply with physical laws, guiding the model to generate images with both high positional reasonableness and aesthetic quality. Therefore, under similar detail and aesthetic quality, RealCompo's more reasonable composition gives it an advantage over the T2I models in termss of realism.
Results of Extend Applications: More Spatial-Aware Conditions We extend RealCompo to more spatial-aware controlled image generation. As shown in Fig. 3, keypoint- and segmentation- based RealCompo achieves outstanding performance in both realism and compositionality. This promising result reveals that spatial-aware conditions, layout, keypoint, and segmentation map fundamentally similar, RealCompo focuses on these similarities and achieves a general generative paradigm for compositional generation.
Results of Extend Applications: Stylized Generation Image style is an essential indicator of fidelity. We experiment with generalizing RealCompo to various pre-trained stylized T2I models. We selected the Coloring Page Diffusion and Cutyukimix as the foundational stylized models, focusing on the coloring page style and adorable style, respectively. As shown in Fig. 8, RealCompo perfectly inherits the style of the T2I models and, with the help of 1.21 model, achieves powerful compositional generation under these styles, which is currently difficult for stylized diffusion models to accomplish. We found it difficult for LMD to strictly maintain the style by simply replacing the backbone with stylized model, often leading text leakage [13]. For example, terms like "crayon" frequently appear in the coloring page style, indicating that the layout control disrupts the style or text control, making it challenging for L2I models to achieve stylized compositional generation. In contrast, by maintaining image realism and style, RealCompo demonstrates strong compositionality while better preserving the style compared to currently outstanding stylized models like InstantStyle [50].
4.3 Ablation Study
Importance of Dynamic Balancer As shown in Fig. conducted experiments the im- portance of the dynamic balancer. It is clear that without the use of the dynamic balancer, the generated images do not align with the layout. This is because the predicted noise in T2I model is not constrained by the layout, leading to the model generating the object at any position, and the quantity is uncontrollable. Although the image realism is high, the predicted noise of T2I model disrupts the object distribution of the predicted noise of L21 model, leading to poor compositionality of the generated images and uncontrollable in the generation process.
Generalizing to Different Backbones To explore the generalizability of RealCompo for various models, we choose two T2I models, SD v1.5 [41] and TokenCompose [54], and two 1.21 models, GLIGEN [27] and LayGuide (Layout Guidance) [8]. We combine them two by two, yielding four

---

Compo weo Dynomic
RealCompo vl IH6LIGEN
RealCompo v2 TowanCompose-SLTTe
RealCompo
RealCompo TekenCompuseel.oy
Layout
LMD
cute
cutte long
poptorn popcont
2 HOM
small congi sitting
popcorns them.
light
a cruise ship
cruise ship and lighthouse the right.
a bureerfly
sinfloaor
sanffoner
floverset
Two sunflowers are growing in blue flowerpot, and a butterfly dances gracefully.
Figure 9: Ablation study on the significance of the dynamic balancer and qualitative comparison of RealCompo's ss generalization different models. We demonstrate that dynamic balancer is important to compositional generation and RealCompo has strong generalization and generality to different models, achieving remarkable level both fidelity and precision in aligning with text prompts.
versions RealCompo v1-v4. The experimental results are shown in Fig. The four versions of RealCompo all have high degree of realism in generating images and achieving desirable results regarding instance composition. This is attributed to the dynamic balancer combining the strengths of T2I and L2I models, and it can scamlessly switch between models because is simple and requires no training. We also found that RealCompo, when using GLIGEN as the L21 model, performs better than when using LayGuide in generating objects that match the layout. For instance, in the images generated by RealCompo v4 in the first and third rows, "popcorns" and "sunflowers" do not fill up the bounding box, which can be attributed to the superior performance of the base model GLIGEN compared to LayGuide. Therefore, when combined with more powerful T2I and L21 models, RealCompo expected to yield more satisfactory results.
Conclusion
In this paper, to solve the challenge of complex or compositional cext-to-image generation, we propose the SOTA training-free and transferred-friendd) framework RealCompo. In RealCompo, we propose novel balancer that dynamically combines the advantages of various (stylized) '21 and spatial-aware (e.g., layout, keypoint, segmentation map) image diffusion models to achieve the trade-off between realism and compositionality in generated images. In future work, will continue to improve this framework by using a more powerful backbone and extend it to more realistic applications.
Acknowledgement
This work is supported by National Natural Science Foundation of China (U23B2048, U22B2037), Beijing Municipal Science and Technology Project (Z231 1100000032000 00011033302), research grant No. SH- 2024JK29, Alibaba Cloud, and high-pperformncce Computing Platform Peking University. This work is also supported by the National Natural Science Foundation of China (Grant No.U1903213) and the Shenzhen Science and Technology Program (ISGG20220831093000))
10

---

References
[1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Igee Akkaya, Florencia Lconi Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprini arXiv:2303.08774 2023.
[2] Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Karsten Kreis, Miika Aittala, Timo Aila, Samuli Laine, Bryan Catanzaro, et al. ediffi: Text-to-image diffusion models with ensemble of expert denoisers. arXiv preprint arXiv:2211.01322 2022.
[3] Omer Bar-Tal, Lior Yariv, Yaron Lipman, and Tali Dekel. Multidiffusion: Fusing diffusion paths for controlled image generation. arXiv preprint arXiv:2302.08/ 2023.
[4] James Betker, Gabriel Goh, Li Jing. Tim Brooks, Jianfeng Wang. Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufci Guo, et al. Improving image generation with better captions. Computer Science. https://cdn. openai. com/papers/dall-et pdf. 2:3, 2023.
[5] Rui- Yang Cai, Hua-Cheng Zhou, and Chun-Hai Kou. Active disturbance rejection control for fractional reaction-diffusion equations with spatially varying diffusivity and time delay. Science China. Information Sciences, 65(2):129203. 2022.
[6] Hila Chefer, Yuval Alaluf, Yael Vinker, Lior Wolf, and Daniel Cohen-Or. Attend-and-exciie Attention- based semantic guidance for text-to-image diffusion models. ACM Transactions Graphics (TOG), 42 (4):1-10, 2023.
[7] Junsong Chen, Jincheng Yu, Chongjian Ge, Lewei Yao, Eaze Xie, Yue Wu, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, et al. Pixart-o: Fast training of diffusion transformer for photorealistic text-to-image synthesis, arXiv preprint a0Xiv22110.00226, 2023.
[8] Minghao Chen, Iro Laina, and Andrea Vedaldi. Training-fre frree layout control with cross-attention guidance. In Proceedings of the IEEE/CVF Winter Conference Applications of Computer Vision, pp. 5343 5353, 2024.
[9] Xiaohui Chen, Yongfei Liu, Yingxiang Yang. Jianbo Yuan, Quanzeng You, Li-Ping Liu, and Hongxia Yang. Reason out your layout: Evoking the layout master from large language models for text-to-image synthesis. arXiv preprint arXiv:2311.17110 2023.
[10] Alexey Dosovitskiy. Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010.11922 2020.
[11] Chenghin Du, Yanxi Li, Zhongwei Qiu, and Chang Xu. Stable diffusion is unstable. Advances in Neural Information Processing Systems, 36, 2024.
[12] Wan- Wan-Cyuan Fan, Yen-Chun Chen, DongDong Chen, Yu Cheng, Lu Yuan, and Yu-Chiang Frank Wang. Frido: Feature pyramid diffusion for complex scene image synthesis. In Proceedings of the AAAI Conference on Artificial Intelligence, pp. 579-587,20 2023.
[13] Weixi Feng. Xuchai He, Tsu-Jui Fu, Varun Jampani, Arjun Reddy Akula, Pradyumna Narayana. Sugato Basu, Xin Eric Wang, and William Yang Wang. Training free structured diffusion guidance for composi- tional text- text-to-image synthesis. In The Eleventh International Conference on Learning Representations. 2023.
[14] Weixi Feng, Wanrong Zhu, Tsu-jui Fu, Varun Jampani, Arjun Akula, Xuehai He, Sugato Basu, Xin Eric Wang, and William Yang Wang. Layoutgpt: Compositional visual planning and generation with large language models. Advances in Neural Information Processing Systems, 36, 2024.
[15] Myles Foley, Ambrish Rawat, Taesung Lee, Yufang Hou, Gabriele Picco, and Giulio Zizzo. Matching pairs: Autriuttin fine-tuned models to their pre-trained large language models. arXiv preprint 2.066 2023.
[16] Hanan Gani, Shariq Farooq Bhat, Muzammal Nascer, Salman Khan, and Peter Wonka. Llm blucprint: Enabling text-to-image generation with complex and detailed prompts, arXiv preprint arXiv:2310. 10640, 2023.
[17] Jiatao Gu, Shuangfei Zhai, Yizhe Zhang, Josh Susskind, and Navdeep Jaitly. Matryoshka diffusion models. arXiv preprint 21Xi::2300.5111, 2023.
11

---

[18] Yaru Hao, Zewen Chi, Li Dong, and Furu Wei. Optimizing prompts for text-to-image generation. Advances in Neural Information Processing Systems, 36, 2024.
[19] Jonathan Ho, Ajay Jain, and Pieter Ahheel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-688 6851, 2020.
[20] Hexiang Hu, Kelvin CK Chan, Yu-Chuan Su, Wenhu Chen, Yandong Li, Kihyuk Sohn, Yang Zhao, Xue Ben, Boqing Gong, William Cohen, et al. Instruct-imagen: Image generation with multi-modal instruction. arXiv preprint arXiv:2401.00 019522 2024.
[21] Kaiyi Huang, Kaiyue Sun, Enze Xie, Zhenguo Li, and Xihui Liu. Tži-comphench: comprehensive benchmark for open-world compositional text-to-image generation. arXiv preprint arXiv22307003350 2023.
[22] Ziqi Huang, Kelvin CK Chan, Yuming Jiang, and Ziwei Liu. Collaborative diffusion for multi-modal face generation and editing. In Proceedings of the IEEE/CVE Conference on Computer Vision and Puttern Recognition, pp.
2023,
[23] Mehran Kazemi, Najoung Kim. Deepti Bhatia, Xin Xu, and Deepak Ramachandran, Lambada: Backward chaining for automated reasoning in natural language. arXiv preprint arXiv:2212.13888 2022.
[24] Yunji Kim, Jiyoung Lee, Jin-Hwa Kim, Jung-Wow Woo Ha, and Jun-Yan Zhu. Dense text-to-image generation with attention modulation. In Proceedings of the IEEE/CVF International Conference on Computer Vision.
pp. 7701-7711, 2023.
[25] Sen Li, Ruochen Wang. Cho-Jui Hsieh, Minhao Cheng, and Tianyi Zhou. Mulan: Multimodal-Ihh agent for progressive multi-object diffusion. arXiv preprint arXiv:2402.122 12741, 2024.
[26] Xiaonan Li, Kai Lv, Hang Yan, Tianyang Lin, Wei Zhu, Yuan Ni, Guotong Xie, Xiaoling Wang, and Xipeng Qiu. Unified demonstration retriever for in-context learning, arXiv preprint arXiv:2305.04320, 2023.
[27] Yuheng Li, Haotian Liu, Qingyang Wu, Fangzhou Mu, Jianwei Yang, Jianfeng Gao, Chunyuan Li, and Yong Jae Lee. Gligen: Open-set grounded text-to-image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22511-22522 2023.
[28] Long Lian, Boyi Li, Adam Yala, and Trevor Darrell. Lim-grounded diffusion: Enhancing prompt under- standing of text-to-image diffusion models with large language models. arXiv preprint arXiv:2305. 13655. 2023.
[29] Zhiheng Liu. Ruili Feng. Kai Zhu, Yifei Zhang, Kecheng Zheng, Yu Liu, Deli Zhao, Jingren Zhou, and Yang Cao. Cones: Concept neurons in diffusion models for customized generation. arXiv preprint arXiv:2303.05125. 2023.
[30] Yujie Lu, Xianjun Yang. Xiujun Li, Xin Eric Wang, and William Yang Wang. Llmscore: Unveiling the power of large language models in text-to-image synthesis evaluation. Advances in Neural Information Processing Systems, 36, 2024.
[31] Tuna Han Salih Meral, Enis Simsar, Federico Tombari, and Pinar Yanardag. Conform: Contrast is all you need for high-fideling fidelity text-to-image -i--tt--maggee diffusion models. arXiv preprint 21X002231..06059, 2023.
[32] Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. Rethinking the role of demonstrations: What makes in-context learning work? arXiv preprint 1rX2v:2202.12837, 2022.
[33] Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob Megrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. In International Conference Machine Learning, pp. 16804. PMLR, 2022.
[34] Geon Yeong Park, Jeongsol Kim, Beomsu Kim, Sang Wan Lee, and Jong Chul Ye. Energy- based cross attention for bayesian context update in text-to-image diffusion models. Advances in Neural Information Processing Systems, 36, 2024.
[35] Quynh Phung, Songwei Ge. and Jia- Bin Huang. Grounded text-to-image synthesis with attention refocusing. arXiv preprint 2.30xiv223000000 05427, 2023.
[36] Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhom, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for righ-resouution image synthesis. arXiv preprint 2013.
12

---

[37] Leigang Qu, Shenggiong Wu, Hao Fei, Liqiang Nie. and Tat-Seng Chua. Layoutllm- Eliciting layout guidance from IIm for text-to-image generation. In H Proceedings the 31st ACM International Conference on Multimediaa, pp. 643-654, 2023.
[38] Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763, PMLR. 2021.
[39] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint urXiv:2204.06122 1(2)3, 2022.
[40] Royi Rassin, Eran Hirsch, Daniel Glickman, Shauli Ravfogel, Yoav Goldberg, and Gal Chechik. Linguis- tic binding in diffusion models: Enhancing attribute correspondence through attention map alignment. Advances in Neural Information Processing Systems, 36, 2024.
[41] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10684-10695, 2022.
[42] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to- image diffusion models with deep language understanding. Advances in Neural Information Processing Systems, 35:36479 33:44479-36494, 2022.
[43] Chenglei Si, Dan Fricdman, Nitish Joshi, Shi Feng, Danqi Chen, and He He. Measuring inductive biases of in-context learning with underspecified demonstrations, arXiv preprint arXiv:2305. 13299, 2023.
[44] Jascha Sohl-Dickstein, Eric Weiss. Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learning, pp. 2256-2265. PMLRR, 2015.
[45] Jiaming Song. Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.025502 2020.
[46] Yang Song. Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456. 2020.
[47] Jiao Sun, Deqing Fu, Yushi Hu, Su Wang. Royi Rassin, Da- Cheng Juan, Dana Alon, Charles Herrmann, Sjoerd van Steenkiste, Ranjay Krishna, et al. Dreamsyne: Aligning text-to-image generation with image understanding feedback. arXiv preprint wxxi:23311.19444 2023.
[48] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Maric Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. / 13971, 2023.
[49] Narek Tumanyan, Michal Geyer, Shai Bagon, and Tali Dekel. Plug-and-play diffusion features for text- driven image-to-image translation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1921-1930, 2023.
[50] Haofan Wang, Qixun Wang, Xu Bai, Zekui Qin, and Anthony Chen. Instantstyle: Free lunch towards style-preserving in text-to-image generation. arXiv preprint arXiv:2404.02333 2024.
[51] Meng Wang, Yinghui Shi, Han Yang, Ziheng Zhang, Zhenxi Lin, and Yefeng Zheng. Probing the impacts of visual context in multimodal entity alignment. Data Science and Engineering, 8(2):124 134, 2023.
[52] Ruichen Wang, Zekang Chen, Chen Chen, Jian Ma, Haonan Lu, and Xiaodong Lin. Compositional text-to-image synthesis with attention map control of diffusion models. arXiv preprint arXiv:2305.13921 2023.
[53] Xudong Wang, Trevor Darrell, Sai Saketh Rambhatla, Rohit Girdhar, and Ishan Misra. Instancediffusion: Instance-level control for image generation. arXiv preprint vxXiv:2402.03290 2024.
[54] Zirui Wang, Zhizhou Sha, Zheng Ding, Yilin Wang, and Zhuowen Tu. Tokencompose: Grounding diffusson with token-level supervision. arXiv preprint 23 2023.
[55] Song Wen, Guian Fang, Renrui Zhang, Peng Gao, Hao Dong, and Dimitris Metaxas. Improving compo- sitional coxt-to-iman toattto-imggeeeee generation with large vision-languag models. arXiv preprint 1rXiv.2310.06311. 2023.
13

---

[56] Haibin Wu, Kai-Wei Chang. Yuan- Kuei Wu, and Hung-yi Lee. Speechgen: Unlocking the generative power of speech language models with prompts. arXiv preprint wrXiv:230600227, 2023.
[57] Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Shaokun Zhang, Erkang Zhu, Beibin Li, Li Jiang, Xiaoyun Zhang, and Chi Wang. Autogen: Enabling next-gen Ilm applications via multi-agent conversation framework, arXiv preprint arXiv:2308.08/.5. 2023.
[58] Tsung-Han Wu, Long Lian, Joseph E Gonzalez, Boyi Li, and Trevor Darrell. Self-correcting Ilm-controlled diffusion models. arXiv preprint arXiv:2311. 2023.
[59] Jinheng Xie, Yuexiang Li, Yawen Huang, Hanzhe Liu, Wentian Zhang, Yefeng Zheng, and Mike Zheng Shou. Boxdiff: Text-to-image synthesis with training-free box-constrained diffusion. In Proceedings of the IEFF/CVF International Conference on Computer Vision, pp. 7452-7461, 2023.
[60] Yanwu Xu, Yang Zhao, Zhisheng Xiao, and Tingbo Hou. Ufogen: You forward once large scale text-to- image generation via diffusion gans, arXiv preprint arXiv:2311.0923 2023.
[61] Zeyue Xue, Guanglu Song, Qiushan Guo, Boxiso I Liu, Zhuofan Zong, Yu Liu, and Ping Luo. Raphael: Text-to-image generation via large mixture of diffusion paths. arXiv preprint arXiv:2305. /8295, 2023.
[62] Ling Yang. Zhilin Huang, Yang Song, Shenda Hong. Guohao Li, Wentao Zhang. Bin Cui, Bernard Ghanem, and Ming- Hsuan Yang. Diffusion-basee scene graph to image generation with masked contrastive pre- pre-training. arXiv preprint arXiv:2211.11133 2022.
[63] Ling Yang, Zhilong Zhang, Yang Song, Shenda Hong, Runsheng Xu, Yue Zhao, Wentao Zhang, Bin Cui, and Ming-Hsuan Yang. Diffiusion models: A comprehensive survey of methods and applications. ACM Computing Surveyx, 56(4):1-39, 2023.
[64] Ling Yang. Jingwei Liu, Shenda Hong, Zhilong Zhang. Zhilin Huang, Zheming Cai, Wentao Zhang, and Bin Cui. Improving diffusion-based image synthesis with context prediction. Advances in Neural Information Processing Systems, 36, 2024.
[65] Ling Yang, Zhaochen Yu, Chenlin Meng, Minkai Xu, Stefano Ermon, and Bin Cui. Mastering text-to-image diffusion: Recaptioning, planning, and generating with multimodal llms. arXiv preprint arXiv:2401. 2024.
[66] Zhengyuan Yang, Jianfeng Wang, Zhe Gan, Linjie Li, Kevin Lin, Chenfei Wu, Nan Duan, Zicheng Liu, Ce Liu, Michael Zeng, and Lijuan Wang. Reco: Region-controlled text-to-image generation. In CVPR, 2023.
[67] Hu Ye, Jun Zhang. Sibo Liu, Xiao Han, and Wei Yang. Ip-adapter: Text compatible image prompt adapter for exx-to--mage diffusion models. arXiv preprint arXiv:2308.0677 2023.
[68] YuTeng Ye, Jiale Cai, Hang Zhou, Guanwen Li, Youjia Zhang, Zikai Song, Chenxing Gao, Junqing Yu, and Wei Yang. Progressive text-to-image diffusion with soft latent direction. arXiv preprint arXiv:2309.094. 09466. 2023.
[69] Chun-Hsiat Yeh, Ta- Ying Cheng, He- Yen Hsieh, Chuan Lin, Yi Ma, Andrew Markham, Niki Trigoni, HT Kung, and Yubei Chen. Gen4gen: Generative data pipeline for generative multi-concept composition. arXiv preprint arXiv:2402. 15504, 2024.
[70] Hao Yu and Jianxin Wu. A unified pruning framework for vision transformers. Science China Information Sciences, 6697):19100, 2023.
[71] Jiwen Yu, Yinhuai Wang, Chen Zhao, Bernard Ghanem, and Jian Zhang. Freedom: Training-free free energy- guided conditional diffusion model. Proceedings of the IEFF/CVF International Conference on Computer Vision (ICCV), 2023.
[72] Lymin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3836 3847, 2023.
[73] Xinchen Zhang, Ling Yang, Guohao Li, Yaqi Cai, Jiake Xie, Yong Tang, Yujiu Yang, Mengdi Wang, and Bin Cui. Itercomp: Iterative composition-aware feedback learning from model gallery for text-to-image generation. arXiv preprint 21Xiv:2410.07771, 2024.
[74] Yuechen Zhang, Jinbo Xing, Eric Lo, and Jiay Jia. Real-world image variation by aligning diffusion inversion chain. Advances in Neural Information Processing Systems, 36, 2024.
14

---

[75] Xiang Zhao, Weixin Zeng, Jiuyang Tang, Xinyi Li, Minnan Luo, and Qinghua Zheng. Toward entity alignment in the open world: an unsupervised approach with confidence modeling. Data Science and Engineering, 7(1):16-29, 2022.
[76] Yibo Zhao, Liang Peng. Yang Yang. Zekai Luo, Hengjia Li, Yao Chen, Wei Zhao, Wei Liu, Boxi Wu, al. Local conditional controlling for rext-to-image diffusion models. arXiv preprint arXiv:2312.08768, 2023.
[77] Guangeong Zheng, Xianpan Zhou, Xuewei Li, Zhongang Qi, Ying Shan, and Xi Li. Layoutdiffusion: Controllable diffusion model for sayout-o--mmag generation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12490-22499 2023.
[78] Dewei Zhou, You Fan Ma, Zongxin Yang, Yi Yang. Mige: Multi-instance generation controller for text-to-image synthesis. arXiv preprint arXiv:2402.05400 2024.
[79] Yiyu Zhuang, Yuxiao He, Jiawei Zhang. Yanwen Wang, Jiahe Zhu, Yao Yao, Siyu Zhu, Cao, and Hao Zhu. Towards native generative model for 3d head avatar. arXiv preprint arXiv:2410.01220 2024.
15

---

This supplementary material is structured into several sections that provide additional details and analysis related our work on RealCompo. Specifically, will cover the following topics:
In Appendix A, we provide preliminary about Stable Diffusion.
In Appendix B, we rovide more visualized results to verify the generality of the phenomenon we discovered in our motivation.
In Appendix C.I, provide detailed pipeline about how to get layout through in in-context learning of LLMs.
Appendix C.2, we provide a detailed proof of the existence of the gradient in Eq.
In Appendix C.3, we provide the pseudocode for RealCompo thoroughly demonstrate its denoising process.
In Appendix C.4, we conduct a detailed analysis of the gradient changes of the two models in Eq. during the denoising process.
In Appendix C.5, we analysis the limitations and future work of RealCompo.
In Appendix C.6, we analysis the broader impact of RealCompo.
In Appendix D, we provide more additional visualized results.
Preliminary
Diffusion models [19, 44, are probabilistic generative models. They can perform multi-step denoising on random noise æy ~ (O, I) to generate clean images through training. Specifically, gaussian noise is gradually added to the clean image æn the forward process:
x Va vI ā,e
(10)
where IO, and a, is the noise schedule.
Training is performed by minimizing the squared error loss:
min L == F..,erW(.. || - |o
(11)
The parameters of the estimated noise Eæ are updated step by step by calculating the loss between the real noise and the estimated noise ca( t).
The reverse process aims to start from the noise xr, and denoise according to the predicted noise col ( t) at each step. DDIM [45] is a deterministic sampler with denoising steps:
VI āgED (It, 1) + 11-1 ac (I1,
Tz-1 Va--1
(12)
Va
Stable Diffusion [41] is significant advancement in this field, which conducts noise addition and removal the latent space. Specifically, SD uses a pre-trained autoencoder that consists of an encoder E and decoder D. Given an image æ. the encoder È maps to the latent space, and the decoder D can reconstruct this image, i.e., z = E(x), = D(z). Moreover, Stable Diffusion supports an additional text prompt y for conditional generation. y is transformed into text embeddings Ta(y) through the pre-trained CLIP [38] text encoder. Eg is trained via:
min L= E= 0
||e ||e- (z (a
(13)
In the inference process, noise Zr A (0, is sampled from the latent space. By applying Eq. 12, we perform step-by-step denoising to obtain a clean latent 2o. The generative image then reconstructed through the decoder D.
16

---

B More Visualized Results on Motivation
Model: GLIGEN
Layout
f 0-0.0
G - 0.2
0 - 0.4
3 - 0.6
3 3 0.8
e -= 1-1.0
Realism
Layout
f
to
Realism
moeiio toblo
Prompt: cute rabbit sitting wooden table, and there reddy bear and its side.
Figure 10: A more intuitive and clearer example to showcase our discoveries and motivation, using GLIGEN [27].
Model: InstanceDiffusion
Loyout
f 0.0
f 0.2
0 0.4
Ø 0.6
0 0.8
0 1.0
Sharther
Realism
Layout
f
onay
Unreal
Prompt: A yellow American robin, brown Maltipoo dog, gray British Sharthair in stream, alongside with trees and rocks.
Figure 11: A more intuitive and clearer example to showcase our discoveries and motivation, using InstanceDiffusiin [53].
To further verify the generality of the phenomenon we discovered in our motivation. As shown in Fig. 10, we first conducted more experiments on GLIGEN [27]. We observed that as the layout control increased (with a higher 3) or the number of layout control steps increased (with higher to). the realism of the generated images declined. There is noticeable degradation in both detail richness and aesthetic quality. For instance, the legs of the teddy bear appear unrealistic, as is facing backward with strange distortions, and the overall details of the rabbit become blurred and unappealing.
Similarly, as shown in Fig. 11, we performed experiments using InstanceDiffusion [53], where we also define parameter to control the strength the layout control. is evident that there significant quality degradation in the dog's facial and body details. Additionally, the cat's eyes are different sizes, and the bird's legs are abnormally thin, indicating reduced realism in the generated images under the influence of layout control. This suggests that achieving balance between realism and compositionality in generated images is generally unattainable.
C
Additional Analysis
C.1 LLM-based Layout Generation
Large Language Models (LLMs) have witnessed remarkable advancements recent years [48, 23, 51, 75, 70]. Due to their robust language comprehension, induction, reasoning, and summarization capabilities, L.MMs have made significant strides in the Natural Language Processing (NIP) tasks [15, 56). In the context of multiple-object compositional generation, text-to-image diffusion models
17

---

exhibit relatively weaker understanding of language, as reflected in the poor compositionality of the generated images. Consequently, exploring ways harness the inferential and imaginative capacities of LLMSs to facilitate their collaboration with text-to-image diffusion models, thereby producing images that adhere to the prompt, offers substantial research potential.
In our task, we leverage LLMs to directly infer the layout of all objects based on the user's input prompt through in-context learning (ICL) [26, 43]. This layout is used for the layout-to-image model of RealCompo, eliminating the need to manually provide a layout for each prompt and achieve pre-binding multiple objects and attributes. Specifically, as shown Fig. 12, we construct prompt templates, which include descriptions of task rules (instruction), in-context examples (demonstration) and the user's input prompt (test). Through imitation reasoning based on the instruction. LLM generate layout for each object, where each layout represents the coordinates of the top-left and hottom-right corners of a respective box. We selected the highly capable GPT-4 [1] as layout generator.
A man and his wife is holding dog.
Embed Prompt into Templated
Instructions
ou are an intelligent bounding box generator. I will provide you with caption for photo, image, or painting. Y our task generate the bounding boxes for the objects mentioned in the caption, along with background prompt describing the scene. The images are of size 1x1. The top-left comer has coordinate [0, 0]. The bottom- rightt comer has coordinate [1, 1J. The bounding boxes should not overlap or go beyond the image boundaries...
Demonstration
Input: teddy bear sits next to bird
Output:
[('a teddy bear', [0.0, 0.1, 0.3, 0.7]), ('a bird', [0.5, 0.1, 1.0, 0.8))] Imput: book on the right and bowl the left
Output:
[('a book', [0.6, 0.4, 1.0, 0.8)), ('a bowl', [0.0, 0.4, 0.4, 0.8))
Test
Input: A man and his wife is holding dog.
Request
Parsing
Output:
mon wife
[('a man', [0.1, 0.1, 0.6, 1.0)). ('wife', [0.5, 0.2, 0.9, 1.0]). ('a dog', [0.4, 0.45, 0.8, 1.0)]
GPT-4
dddg
Figure 12: Firstly, the user's input text is embedded into the prompt template. The template is then parsed using GPT-4 with frozen parameters, which yields descriptions of the objects in the prompt as well as their corresponding layout.
18

---

C.2
Analysis of the Existence of Gradient in Eq. 7
Here we set:
C(A,-) 1-1 A
Luyoul
Σ Lo(AA A dayoul
1-1
Alext E,
(+j..t-1) Mb
E, A Abyrur (1btt-1) 2 Mo)
)
)
(14)
text (156,1-1)
E; (1
the loss function given by Eq. 6, the gradient in Eq. can be derived as follows:
ac
text
A
layout
aCoei
0 E. Lb (Ap-1
layout 1-1
&Coe
OCb (Alt-) Inyour
OCoe
TLD (ApEo) 1,
layout
(u 0zc-1
de, de;
Σ
0A (1.1-11)
0z.-1 DE, 05; aCoe;
A
Layoul 8-1
exp Coe text + Coe, layout
( 021-1 dc:
(15)
Σ
0AC
θz1-1
DEc dei (exp (Coel lext) + exp (Coe, layrmut)
(o.t-1)
U.D A Layoul
exp Coee text + Coe. layou)
0AC. ( 0z:-1
0Ac (o.t-1)
OE: 0z1-1
(exp (Coe, + exp Coe,
layour
O.C Alea 11 A layout
A-1
0AC. 0AC...11 (c.t-11)
04-1 σ2 vI ai
Σ
0Ac.
dzi-1
Vai
(b,l-1)
cf cxp (ooetot + Coel Layoul
exp (Coeleu) + exp (Coe/yyou
Inyout
For any T2I and L2I models, we have the following:
0hg (Abor 1
glayour
layour JE, O Mb) -Mb E, Aforn---1
1,
(16)
0A (
2
where is matrix with elements equal 1, All variables in Eq. 15 are known, indicating the existence of the gradient in Eq. 7.
19

---

When using the loss function given by Eq. under any apatial-awars conditions, the gradient in Eq. 7 can be derived as follows:
OC (A
ppotial
aCoe;
OLb (Alm)) A,
( 0zc-1 de: og: dz1-1 dec de; aCoe
0.4C 0 (1)
Uhb (A)))) A-11
spotial
0A (in,t-1)
v1 az
(17)
Σ
V ac-1 - 0²
0.4 (bit-11
0z.-1
Vot.
exp (Coetal Coeppalial spatial
exp
x
exp (Coe) + exp (ooe; spatial)
dLb
spatial
JE, (Aiss 2 sb(c)) Sb(C) E, A A(06s-11)
+-1
(18)
0A 0 (j),-11
where c ftext, spatial]. Therefore, the gradient in Eq. 7 exists for the selection of different loss functions.
C.3 Inference details
We provide detailed compositional denoising process for RealCompo, which achieves comple- mentary balance between the advantages of the T2I model and the spatial-awar diffusion model by combining their predicted noise during the denoising stage. We provide the pscudocode for the compositional denoising process of the layout-based RealCompo as followed, we have highlighted the innovations of our method in blue.
Algorithm 1 Compositional denoising procedure of fayout-based RealCompo
Input: text prompt PP set of layout B, a pretrained T21 model and a pretrained 1.21 model Output: A clear latent zo
1: ZT 1)
2: Coer text = Coe, (0,1)
layour
3: for T,...,
4: if t >> to then
5:
Et. L21 (zt, P,B,t)
6:
else
7:
text T21 (2,, P.t)
layout = 1.21 P.
8:
9:
Get the balanced noise 6, from Eq. and Eq.
10:
Get the denoised latent 2;-1 from Eq. 12
claal
1.-1, T2I (2,-1, P,t)
12:
E2-1
layout
A = L22 12, B,t)
Compute L(Atr A layout from Eq.
14:
Update Coe, according to Eq. 7
Get the balanced noise e, from Eq. and Eq.
16:
end if
Get the denoised latent 2;-1 from Eq.
18:
end for
19:
return 20
20

---

C.4
Gradient Analysis
Gradient Analysis We selected RealCompo v3 and to analyze the gradient changes in Eq. across all denoising stages. As shown in Fig. 13, we use the same prompt and random seed to visualize the gradient magnitude changes corresponding T2I and L2I for each model version. We observe that the gradient magnitude change of RealCompo v4 fluctuated more in the early denoising stages. We argue that TokenCompose, which enhances the composition capability of multiple-object generation by fine-tuning the model using segmentation masks, may overlap in functionality with the baseed multiple-object generation, and TokenCompose's positioning of objects may not consistently align with the bounding box. Therefore, RealCompo must focus balancing the positioning of TokenCompose and layout in the early denoising stages, leading to less stable gradients compared to RealCompo v3. Additionally, due to LayGuide's ' weaker positioning ability compared to GLIGEN, RealCompo v4 may occasionally generate objects with less coverage of the bounding box, mentioned in the ablation experiment in Section 4.3.
1e-5
Compo 121
1.0
Roal Compe v3 T21
Roal Compo v4 L21
Real Compo v4 T21
Gradient Magnitude
0.8
0.4
0.2
20
30
40
50
Step
Figure 13: Changes of gradient magnitude in Eq. 7 across all denoising process for the T2I and L2I models of RealCompo v3 and v4.
C.5 Limitations and Future Work
Limitations While RealCompo enhances both realism and compositionality in training-free manner, it should be noted that the computational cost of our method slightly higher compared that of single T2I model or a single spatial-aware image diffusion model, due the need to combine two models and compute loss and gradients. However, by adjusting the combination stage of RealCompo, we can keep the computational cost within an acceptable range.
Future Work In future work, we aim to explore more efficient computational methods to improve the calculation efficiency of RealCompo while maintaining high-quality results and we plan to extend its application to more challenging tasks such as text-to-video and text-to-3D generation. Furthermore, given that the exceptional classifier-free guidance strategy employs fixed weights, we aim to explore the potential of using fixed coefficients to further enhance the capabilities of RelCompo.
C.6 Broader Impact
Recent significant advancements in text-to-image diffusion models have opened up new possibilities for creative design, autonomous media, and various other sectors. However, the dual-use nature of this technology raises concerns about its social impact. Image diffusion models carry the risk of misuse, particularly in the realm of impersonating humans. For example, in today's society, malicious applications such as "deepfakes" have been employed in inappropriate contexts fabricate attacks on specific public figures. It is crucial to clarify that our algorithm is designed to enhance the quality of image generation, and we do not endorse or facilitate such malicious applications.
21

---

D
More Generation Results
Layout
SD1.5
GLIGEN
LMD+
RealCompo(Ours)
brown
teddy bear
blue
water Cup
A brown teddy bear holding blue water cup sits the sofa.
blue
TW0 Nelrew
a brunch
blossom
blue bird standing branch with yellow blossoms the right.
fireworks display
group of people
group of people watching fireworks display.
cat
a COT
plate of food
table
Three cats are sitting on the table, with plate of food in front of them.
a tree
Blooo
child is reading book under a tree.
comera
person
the city
person holding camera and taking photos of the city.
Figure 14: More generation results about layout-basc based RealCompo.
22

---

Keypoint
SDXL
ControlNet
RealCompo(Ours)
Cinematic photo shot of Leonardo teenage turtle ninja, weapon, and dirty background
girl, Elsa and Anna, sparks of magic between them, princess dress, background with sparkles, black purple red color schemes.
Two standing the moen, behind them white planet amidst the vast Figure 15: More generation results about keypoint- based RealCompo.
Segmentation map
SD v2.1
ControlNet
RealCompo(Ours)
An airplane parked the runway the sets behind
Two cats sitting on the windowsill looking at each other.
Five stand together in line, serious expression.
Figure 16: More generation results about segmentation- based RealCompo.
23

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main claims made the abstract and introduction accurately reflect ours contributions and scope lie in proposing a training-free and transferred-frieme text- to-image generation framework, namely RealCompo, which aims to achieve the trade-offs between realism and compositionality of the generated images.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We disscuss the limitications of the work in Appendix C.5
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
24

---

Answer:
[NA]
Justification:
This paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We have released our code for others to reproduct the results in paper. we have also give detailed instructions about experiment sctup in Section 4.1.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
25

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We have provided open access to code, with sufficient instructions to faithfully reproduce the main experimental results.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
oo icy for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: / //nips. cc/ wewiic/ggiee//eoeee
ss cettimissssoooooiiiiooco onnoliccy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We have also give detailed instructions about experiment setup in Section 4.1.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [NA]
Justification: Error bars are not reported because it would be too computationally expensive.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
26

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We have give detailed information about experiment setup in Section 4.1.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't I make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics atttp:///eeurie httpp////eeuripsccppppiieeeeeeee httppubiic/Bthcsauiii.eo
Answer: [Yes]
Justification: Our research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We disscuss the limitications of the work in Appendix C.6
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
27

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper poses such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
The creators or original owners of code used in the paper properly credited, and the license and terms of use are explicitly mentioned and properly respected.
Justification:
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
28

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/dataseta has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New Assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: New assets introduced in the paper well documented. We provide them as supplementary material.
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
29

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
30


Reviews:
[{"review_id": "SxPjXrbMMZ", "reviewer": "Reviewer_XVVX", "summary": "The paper introduces a training-free and flexible text-to-image generation framework called RealCompo, which enhances compositional text-to-image generation by balancing the realism and compositionality of generated images. It features a novel balancer that dynamically combines the predicted noise from T2I models and spatial-aware image diffusion models (such as layout, keypoint, segmentation map). This framework provides a fresh perspective for compositional image generation.", "questions": "I will consider increasing the score or maintaining it if the authors can address the following issues.\n1. The paper uses a balancer to balance two diffusion models, which is very similar to the concept of \"checkpoint merge\" or \"model ensemble\". It is necessary to clarify the differences between these two approaches.\n2. Please explain the necessity of the proposed method. If condition-guided models like ControlNet use a stronger backbone, higher quality, larger quantity, and more diverse data and annotations during training, is it still necessary to balance with non-condition-guided text-to-image models?", "limitations": "Yes.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed method is training-free but achieves notable results.\n2. The proposed method mainly applies a balance between two pretrained diffusion models, which is easy to implement and quite flexible.\n3. The paper’s explanation of realism and control sounds reasonable.", "weaknesses": "1. The biggest problem is novelty/contribution. The paper uses a balancer to balance two diffusion models, which is very similar to the concept of \"checkpoint merge\" / \"model ensemble\". It is necessary to clarify the differences between these two approaches.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "*We sincerely thank you for your time and efforts in reviewing our paper, and your valuable feekback. We are glad to see that the proposed method is flexible, the experimental results are promising, and the explanation of the important concepts is reasonable. Please see below for our responses to your comments.*\n\n**Q1: The paper uses a balancer to balance two diffusion models, which is very similar to the concept of \"checkpoint merge\" or \"model ensemble\". It is necessary to clarify the differences between these two approaches.**\n\nA1: Thank you for the comment. We here clarify the differences between our method and checkpoints merge and model ensemble. \n\n Checkpoint merge aims to combine parameters from multiple models; however, this method presents a problem in compositional generation. Checkpoint merge is **typically used to blend the realism or stylistic features of two models**, requires the model to possess a high level of spatial awareness and precise localization capabilities. Simply merging the parameters of two models, such as SD and GLIGEN, may **compromise the strength of layout control, leading to a decline in compositionality**. Our method merges the predicted noise from both models, as **each contains unique strengths in either rich semantic information or spatial features**. This fusion method more effectively leverages the strengths of both models.\n\nModel ensemble, on the other hand, involves a straightforward weighted combination of the output noises from two models. We have discussed the limitations of this approach in our paper. As seen **in Section 4.3 and Figure 8**,  although simply weighting the predicted noise from T2I and L2I models can result in a generated image with higher realism, **it disturbs the object positions of the L2I model**. The lack of conditioning in the T2I model leads to arbitrary object placements, causing conflicts with the layout's object positions and resulting in uncontrollable composition. Thus, model ensemble has limitations in compositional generation, and **dynamic balancer is a crucial bridge** for balancing the predicted noise from the models. \n\nTherefore, in compositional generation tasks, both checkpoint merge and model ensemble methods have their own shortcomings when it comes to achieving a trade-off between realism and compositionality.\n\n**Q2: Please explain the necessity of the proposed method. If condition-guided models like ControlNet use a stronger backbone, higher quality, larger quantity, and more diverse data and annotations during training, is it still necessary to balance with non-condition-guided text-to-image models?**\n\nA2: Even though condition-guided models use more powerful backbones with larger sizes and higher performance, **the challenge of achieving a balance between realism and compositionality in generated images still exists**. Therefore, our approach is necessary and  widely needed in the field of controllable generation. In the **supplementary PDF**, we provide additional experimental results to validate the necessity of our method. As shown in Figure 1, we conducted experiments using layout-based models (GLIGEN) of different sizes, varying the parameter $\\beta$ to control the strength of condition guidance. It is evident that the aesthetic quality of the generated results significantly decreases as the condition guidance strength increases, **even with a larger and stronger backbone**. This indicates that improving compositionality comes at the expense of realism. Similarly, in Figure 2, we conducted experiments using keypoint-based models (ControlNet) of different sizes, varying the parameter *control_scale* to control the strength of condition guidance. Again, it is apparent that larger and stronger backbones, trained with more and better data, exhibit superior realism under the same conditions. **However, as the strength of the condition control increases, a notable decline in realism is observed**. Therefore, **the trade-off between realism and compositionality is a common issue**.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VcCcXThBgR", "reviewer": "Reviewer_juPc", "summary": "This paper presents a method, named RealCompo, for combining multiple diffusion models, such as a text-to-image model and a spatial-aware one, to achieve the best of both worlds: superior image realism and compositionality. It merges predicted noise from both models during each denoising step, and balances them based on two sets of coefficients that are dynamically updated through a loss that forces cross attention maps to follow provided spatial constraints. Experiments show that RealCompo improves both realism and compositionality compared to using either model alone, and it outperforms SOTA spatial-aware models in compositional generation.", "questions": "- Why can RealCompo improve image realism (Table 2, Figure 6 left) and style preservation (Figure 6 right) beyond the capabilities of the T2I model alone? This is difficult to understand because the loss function (Eq. 6) employed for balancing the two models is designed solely to encourage better adherence to the provided spatial constraints. I will reconsider my rating if this aspect can be properly addressed.\n- What is the purpose of initializing the coefficients as random values (Eq. 1)? This seems unnecessary and introduces random bias towards pixels.", "limitations": "The authors touched on the computational cost in Appendix B.5. However, a more detailed comparison with T2I and L2I models concerning computational overhead and memory overhead is crucial to determine the feasibility of the proposed method.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- The paper is well written and easy to follow.\n- The proposed method is straightforward and compatible across various models.\n- The idea of fusing multiple diffusion models during denoising is interesting. This may have applications beyond compositional text-to-image generation.\n- Extensive experiments are conducted on various T2I models, spatial-aware models, stylized models.", "weaknesses": "- Since the proposed method runs multiple models concurrently and requires gradient-based updates, a discussion on computational efficiency is crucial for its practical application.\n- Figure 1 does not illustrate the trade-off between realism and compositionality well. I don't see an apparent drop of realism/aesthetics as $\\beta$ (control of layout) increases.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "*We sincerely thank you for your time and efforts in reviewing our paper, and your valuable feekback. We are glad to see that the proposed method is straightforward and flexible, the whole paper is well written, the idea is interesting and promising, and the experiments are extensive. Please see below for our responses to your comments.*\n\n**Q1: Since the proposed method runs multiple models concurrently and requires gradient-based updates, a discussion on computational efficiency is crucial for its practical application.**\n\nA1: Thank you for your suggestion. We compare our method with other T2I models and spatial-aware diffusion models regarding inference time, VRAM usage and complex performance in the table below:\n\n||Training-free|Inference Time (/Img)|VRAM Usage|Complex$\\uparrow$|\n|-|-|-|-|-|\n|SDXL|×|10.8s|14.7G|0.4091|\n|SD 3|×|10.4s|22.3G|0.3924|\n|Attn-Exct|√|14.3s|16.3G|0.3401|\n|ControlNet|×|11.2s|20.4G|-|\n|GLIGEN|×|16.8s|13.4G|0.3420|\n|LMD+ (SD 1.5)|√|26.6s|14.5G|0.3323|\n|RealCompo (SD 1.5 + GLIGEN)|√|23.8s|20.8G|**0.4657**|\n\nWe observed that our method achieves better compositional generation results compared to other training-free approaches, with only marginal increases in inference time and VRAM usage. Our method can be flexibly applied to any T2I and spatial-aware diffusion models without the need for training.\n\n**Q2：Fig1 doesn't illustrate the trade-off between realism and compositionality well. I don't see an apparent drop of realism/aesthetics as $\\beta$ increases.**\n\nA2: We are grateful for your feedback and apologize for any potential confusion caused. We have provided two more intuitive examples in **supplementary PDF**. As shown in **Fig3**, we conducted experiments using GLIGEN. We observed that as the layout control increased (with a higher $\\beta$) or the number of layout control steps increased (with a higher $t_0$), the realism of the generated images declined. There is a noticeable degradation in both detail richness and aesthetic quality. For instance, the legs of the teddy bear appear unrealistic, as if it is facing backward with strange distortions, and the overall details of the rabbit become blurred and unappealing.\n\nSimilarly, as shown in **Fig4**, we performed experiments using InstanceDiffusion, where we also define a parameter $\\beta$ to control the strength of the layout control. It is evident that there is significant quality degradation in the dog's facial and body details. Additionally, the cat's eyes are different sizes, and the bird's legs are abnormally thin, indicating reduced realism in the generated images under the influence of layout control. **This suggests that achieving a balance between realism and compositionality in generated images is generally unattainable**. \n\nThank you again for your feedback. We will update the manuscript to make Fig1 clearer and more intuitive.\n\n**Q3: Why does RealCompo improve image realism (Table 2, Fig6 left) and style preservation (Fig6 right) beyond the T2I model's capabilities? This is unclear, as the loss function (Eq6) aims only to enforce spatial constraints.**\n\nA3: First, we provide a detailed explanation of the concept of realism. Realism refers to the fidelity of details, aesthetic quality, and positional reasonableness of an image. Here, positional reasonableness is different from compositionality. Compositionality refers to the number of objects, spatial relationships, and attribute bindings in the generated images, **while positional reasonableness is an important metric for assessing the image realism**, specifically whether the placement of each object in the image is reasonable. **When the details and aesthetic quality of an image are similar, positional reasonableness becomes a key factor in user selection**. In Fig5 of the supplementary PDF, we provide examples from the user study, which demonstrates the advantages of RealCompo over the T2I model in realism. As shown in Fig5 (a), T2I model generates a teapot that is visibly suspended in the air, which doesn't conform to the physical laws of real-world scenes. In contrast, RealCompo generates objects within reasonable bounds through layout constraints, ensuring both the aesthetic quality and positional reasonableness. In Fig5 (b), the red chair generated by the T2I model is unnaturally placed on top of the table, and in Fig5 (c), two people generated by the T2I model are too close to each other. These examples illustrate that although T2I model outperforms in detail and aesthetics, its positional reasonableness needs improvement. Our method utilizes LLM to generate conditions that comply with physical laws, guiding the model to generate images with both high positional reasonableness and aesthetic quality. **Therefore, under similar detail and aesthetic quality, RealCompo's more reasonable and realistic composition gives it an advantage over the T2I model in terms of realism**.\n\nSecond, the design of our loss function (Eq6) is crucial. As discussed in Sec 4.3 and illustrated in Fig8 of our paper, simply weighting the predicted noise from T2I and L2I models disrupts the object positions controlled by the L2I model, despite enhancing the detail and aesthetic quality of the generated images. This disruption occurs because T2I model lacks conditional controlled, leading to arbitrary positioning that interferes with layout control and results in uncontrollable compositionality. **Therefore, our loss function is essential to control the T2I model, ensuring it enhances image realism without compromising layout control**.\n\n**Q4: What is the purpose of initializing the coefficients as random values (Eq. 1)? This seems unnecessary and introduces random bias towards pixels.**\n\nA4: Thank you for your comment. Our fundamental goal is to ensure the initialization coefficients of the two models are the same, so it is indeed unnecessary to initialize them with random values. This is not the focus of our method, we will update and optimize this part in our paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wyNYOnqHb5", "reviewer": "Reviewer_ZHfz", "summary": "This paper proposes to balance the realism and compositionality of the generated images by means of different diffusion models, such as pre-trained T2I models and models based on spatial perceptual control. The authors develop a balancer that optimizes the two coefficients through mask-based attention manipulation to enable dynamic combination of the different models. They also demonstrate the effectiveness of the method through extensive experiments.", "questions": "1. Fig-1 lacks a detailed explanation.\n2. Sub-sec 3.3 contains not much valuable information, as the only difference between Eq.9 and Eq.6 is how to obtain a mask/spatial condition. \n\nI will revise my rating according to the author's feedback and the reviewer's discussion.", "limitations": "N/A", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 1, "strengths": "Pros:\n1. The proposed design could be used to bridge different models and  support better image generation.\n2. SOTA quantitative and qualitative results obtained by the proposed method.\n3. The method can be extended to stylized image generation in a training-free manner.", "weaknesses": "Cons:\n1. Unclear Task definition. First, even different pre-trained T2I models can result in different so-called REALISM due to different training strategies and data. Secondly, Control-based models, e.g., Controlnet and GLIGEN, all add controllability while inheriting the realism generated by the original T2I models. So I'm not sure why the authors need to propose another method to do the so-called balancing of realism and composability.In addition, how can you guarantee that the combination of T2I and L2I can generate real diagrams when the images generated by T2I are all more fake?\n2. Unconvincing observation. In Figure 1 and paragraphs 2 to 4, the authors present their motivation only through an existing method, GLIGEN. I argue that this is inappropriate. After all, there are many methods based on layout or control, no matter what conditions these methods use, such as layout, segmentation map, and pose. Motivation is the key to a paper and cannot be deduced from just one method. So I argue that this observation is not very convincing to me.\n3. Insufficient contribution. It is common to use optimization to produce better images, but the difference lies in the design of the optimization. The use of mask to combine with attention to do optimization is more common way, such as in [1,2,3,4,5]. The only difference is that the existing methods may optimize zt, this paper optimizes two coefficients. So I argue that the contribution of this paper is not very sufficient.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "*We sincerely thank you for your time and efforts in reviewing our paper, and your valuable feekback. We are glad to see that the proposed method can be generalized to different models, achieve SOTA generation results, and expand to various application scenarios in a training-free manner. Please see below for our responses to your comments.*\n\n**Q1: Why propose another method for balancing realism and composability? T2I models differ in realism due to training variations, and control-based models enhance controllability while retaining original realism.**\n\nA1: Thank you for your comment. As you mentioned, control-based models enhance controllability while retaining the realism of original T2I models. However, **our extensive experiments show that increasing the strength of condition control significantly reduces detail richness and overall aesthetics.** Fig1 in our paper and Figs3 and 4 in supplementary PDF illustrate this decline of realism with increased control strength ($\\beta$ increase). Additionally, Fig1 and 2 in supplementary PDF demonstrate that even with larger, more powerful backbones, increasing control strength degrades detail and aesthetic quality. **Thus, control-based models must sacrifice realism to improve compositionality**. The imbalance between realism and compositionality in control-based models is a critical and unresolved issue.\n\nIt's correct that different pre-trained T2I models produce varying levels of realism due to different training strategies and data. Hence, **by using different stylized T2I models, our method easily achieves stylized compositional generation**. As shown in Fig7 in our paper, our method maintains high-quality style preservation with different stylized T2I models. \n\n**Q2：How can you ensure that combining T2I and L2I generates real diagrams when T2I images are mostly fake?**\n\nA2: **One important criterion for determining whether an image appears real or fake is the reasonableness of its composition, specifically whether object placement aligns with real-world physical scenarios**. Fig5 of supplementary PDF shows examples where T2I-generated images fail in this regard.  For instance, in Fig5(a), the teapot generated by T2I model is visibly suspended in the air, defying physical laws. Our method uses layout constraints to ensure objects are generated within reasonable bounds, maintaining both aesthetic quality and compositional reasonableness. Similarly, the image generated by T2I model in Fig5(b) shows a red chair unnaturally placed on a table, and in Fig5(c) depicts two people too close to each other. **These examples indicate that while T2I model excels in aesthetics, it lacks compositional reasonableness. Our method uses LLM to generate conditions that comply with physical laws, guiding the model to generate images with high compositional and aesthetic quality.** Thus, our method generates more realistic images compared to T2I models.\n\n**Q3: In Fig1 and paragraphs 2-4, the authors present their motivation solely through GLIGEN, which is inappropriate.**\n\nA3: We are grateful for your feedback and apologize for our oversight. In supplementary PDF, we provided additional experiments to validate our motivation. Fig3 shows a more apparent trend of changes in image realism using GLIGEN. Fig4 uses InstanceDiffusion to further validate this, with a parameter $\\beta$ to control the strength of layout guidance. As $\\beta$ increases, the realism of generated images decreases, evidenced by loss of details, decline in aesthetic quality, and unrealistic content. For example, the dog's facial and body details degrade, the cat's eyes differ in size, and the bird has abnormally thin legs.\n\nAdditionally, we set the parameter $t_0$ to control the number of steps for layout control during generation. Specifically, when denoising steps are fewer than $t_0$, layout is used for control, and when the steps exceed $t_0$, only text is used. Similar to GLIGEN, when $t_0$ exceeds 20, the generated images of InstanceDiffusion show minimal differences, indicating that layout control's influence is mainly in the early denoising stages. Even if layout guidance is discarded in the later stages of denoising and only text is used, it is still challenging to recover the realism of the images.\n\nFig1 and 2 in supplementary PDF show our analysis of the aesthetic quality of images generated by GLIGEN and ControlNet at different sizes. The figures demonstrate that as conditional control strength increases, image aesthetic quality declines at any size. Thank you again for your feedback. We will expand on this part of the experiments in the manuscript.\n\n**Q4:  Existing methods may optimize $z_t$, while this paper optimizes two coefficients. Thus, the contribution is insufficient.**\n\nA4: Here we would like to emphasize two points that distinguish us from previous work and add to the significance of our findings in this paper, **we provide detailed clarifications in the General Response**.\n\nFirst, we are the first to discover the imbalance between realism and compositionality in generated images, providing experimental analysis and conclusions.\n\nSecond, we offer a new perspective on optimization-based generation methods, achieving satisfactory results and generalization using coefficient updates for the first time.\n\nWe will clarify these two points further in our manuscript.\n\n**Q5: Fig1 lacks detailed explanation.**\n\nA5: Thank you for your feedback, and we apologize for any confusion caused. Details about Fig1 can be found in **A3**. We will update our manuscript to provide more comprehensive and accessible explanations regarding Fig1.\n\n**Q6: Sub-sec 3.3 contains limited valuable information since difference between Eq9 and Eq6 is how to obtain a mask/spatial condition.**\n\nA6: Thank you for your suggestion. Sub-sec 3.3 is an extended part of our paper, aimed at explaining the generalizability of spatial-aware conditions. We will condense and refine this section in future revisions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely thank all the reviewers for the thorough reviews and valuable feedback. We are glad to hear that the idea is interesting, promising and flexible (Reviewer juPc), the paper is well written and easy to follow (Reviewer XVVX, juPc),  the experiments are extensive and performance improvements are promising (all reviewers).

To address the reviewers' concerns and misunderstandings, we would like to emphasize the contributions and novelty of our method by highlighting two key points that distinguish our work from previous research and underscore the significance of our findings.

***First, we discovered for the first time the imbalance between realism and compositionality of generated images in compositional generation. Our study provides experimental analysis and conclusions on this issue.***

Previous controllable generation methods often overlooked the decline in image realism while focusing on enhancing compositionality. We introduce a critical new challenge to the field of controllable generation: achieving a balance between the realism and compositionality of generated images. In Figure 1 of the paper and Figures 1, 2, 3, and 4 of the supplementary PDF, we conduct detailed analyses using multiple models and experiments to identify the root causes of this imbalance.

***Second, we provide a new perspective on optimization-based generation methods, achieving satisfactory results and generalization by using coefficient updates for the first time.***

Previous optimization-based generation methods have focused on optimizing the latent $z_t$ based on the loss function. However, this approach is unsuitable for our task, which requires dynamically combining the strengths of two models. Therefore, we introduce a novel perspective to address this challenge by dynamically optimizing the coefficients of the models. This approach effectively leverages the advantages of each model, achieving a balance between realism and compositionality. Additionally, due to the flexibility and generalizability of our method, we can seamlessly integrate any stylized T2I models and spatial-aware diffusion models to achieve high-quality stylized compositional generation.

We here summarize and highlight our responses to the reviewers:

- We thoroughly emphasize our contributions and novelty (Reviewer XVVX, juPc and ZHfz), and clarify the differences between our method and "checkpoints merge" or "model ensemble (Reviewer XVVX)."
- We conduct extensive experiments on different sizes of different backbones to verify that the issue of imbalance between realism and compositionality in controllable generation is widespread (Reviewer XVVX, ZHfz). Additionally, we provided more clear and intuitive examples to explain the motivation of the paper as illustrated in Fig1 in detail (Reviewer juPc, ZHfz).
- We provided a more detailed explanation of realism and offered a nexperimental analysis of why our method can prevent the generation of unrealistic images (Reviewer juPc and ZHfz).
- We compare our method with other diffusion models regarding inference time, VRAM usage and complex performance, achieving a better trade-off than previous diffusion models (Reviewer juPc).


We reply to each reviewer's concerns in detail below their reviews. Please kindly check out them. Thank you and please feel free to ask any further question.

Meta-review: All the reviewers recommend acceptance of the work. Reviewers appreciated the simple yet effective technique with good results. Reviewers raised several concerns related to novelty and clarification questions, several of which are addressed in the rebuttal. Reviewers did raise valid concerns and authors are encouraged to do the best of their abilities to include the rebuttal discussions and address all the concerns in the final version.
Decision: Accept (poster)