IterComp: Iterative Composition-Aware Feedback Learning from Model Gallery for Text-to-Image Generation
Authors: Xinchen Zhang
Ling Yang
Guohao Li
YaQi Cai
xie jiake
Yong Tang
Yujiu Yang
Mengdi Wang
Bin CUI
Venue: ICLR 2025 Poster (2025)
Keywords: Compositional text-to-image generation
Feedback learning for diffusion model
Primary area: generative models

Abstract: Advanced diffusion models like Stable Diffusion 3, Omost, and FLUX have made notable strides in compositional text-to-image generation. However, these methods typically exhibit distinct strengths for compositional generation, with some excelling in handling attribute binding and others in spatial relationships. This disparity highlights the need for an approach that can leverage the complementary strengths of various models to comprehensively improve the composition capability. To this end, we introduce IterComp, a novel framework that aggregates composition-aware model preferences from multiple models and employs an iterative feedback learning approach to enhance compositional generation. Specifically, we curate a gallery of six powerful open-source diffusion models and evaluate their three key compositional metrics: attribute binding, spatial relationships, and non-spatial relationships. Based on these metrics, we develop a composition-aware model preference dataset comprising numerous image-rank pairs to train composition-aware reward models. Then, we propose an iterative feedback learning method to enhance compositionality in a closed-loop manner, enabling the progressive self-refinement of both the base diffusion model and reward models over multiple iterations. Detailed theoretical proof demonstrates the effectiveness of this method. Extensive experiments demonstrate our significant superiority over previous methods, particularly in multi-category object composition and complex semantic alignment. IterComp opens new research avenues in reward feedback learning for diffusion models and compositional generation. Code: https://github.com/YangLing0818/IterComp

Published as a conference paper at ICLR 2025
ITERCOMP: ITERATIVE COMPOSITION-AWARE FEEDBACK LEARNING FROM MODEL GALLERY FOR
TEXI-TO-IMAGE GENERATION
Xinchen Zhang Ling Yang"*
Guohao Li
Yaqi Cai
Jiake Xie
Yong Tang*
Yujiu Yang't Mengdi Wang
Bin Cui
TTinnhua University PPeking University "LibAI 4USTC
University Oxford "Princeton University
https://githai www.coo////o//o www./o/agggiinnnnitteeeeeeeeeeee
ABSTRACT
Advanced diffusion models like Stable Diffusion 3, Omost, and FLUX have made notable strides in compositional text-to-image generation. However, these meth- ods typically exhibit distinct strengths for compositional generation, with some excelling in handling attribute binding and others in spatial relationships. This disparity highlights the need for an approach that can leverage the complementary strengths of various models to comprehensively improve the composition capa- bility. To this end, we introduce IterComp, a novel framework that aggregates composition-aware model preferences from multiple models and employs iter- ative feedback learning approach to enhance compositional generation. Specifi- cally, we curate gallery of six powerful open-source diffusion models and eval- uate their three key compositional metrics: attribute binding, spatial relationships, and non-spatial relationships. Based on these metrics, we develop composition- aware model preference dataset comprising numerous image-rank pairs to train composition-aw reward models. Then, we propose an iterative feedback learn- ing method to enhance compositionality in closed-loop manner, enabling the progressive self-refinement of both the base diffusion model and reward models over multiple iterations. Detailed theoretical proof demonstrates the effective- ness of this method. Extensive experiments demonstrate our significant superior- ity over previous methods, particularly in multi-category object composition and complex semantic alignment. IterComp opens new research avenues reward feedback learning for diffusion models and compositional generation.
INTRODUCTION
The rapid advancement of diffusion models (Sohl-Dickstein et al., 2015; Ho et al., 2020; Song et al., 2020; Peebles & Xie, 2023) has recently brought unprecedented progress to the field of text-to-image generation, with powerful models like DALL-E (Betker et al., 2023), Stable Diffusion 3, (Fiser et al., 2024) and FLUX (BlackForest, 2024) demonstrating remarkable capabilities in generating aesthetic and diverse images. However, these models often struggle to follow complex prompts to achieve precise compositional generation (Omost-Team, 2024; Yang et al., 2024b; Zhang et al., 2024b), which requires the model possess robust, comprehensive capabilities in various aspects, such as attribute binding, spatial relationships, and non-spatial relationships (Huang et al., 2023).
To enhance compositional generation, some works introduce additional conditions such lay- outs/boxes (Li et al., 2023; Zhou et al., 2024; Wang et al., 2024a; Zhang et al., 2024b). InstanceDif- fusion (Wang al., 2024a) controls the generation process using layouts, masks, other conditions through trainable instance masked attention layers. Although these layout-based methods demon- strate strong spatial awareness, they struggle with image realism, especially in generating non-spatial relationships and preserving aesthetic quality (Zhang et al., 2024b). Another potential solution lever-
"Contributed equally. Contact: yangling08188 @ 163.com 'Coresponding authors.

---

Published as a conference paper at ICLR 2025
Artributt Binding
Spatial Relationship
Non Spatial Relationship
Others
A magical night scene of the Hegwarts Express, it races through mocnlit landscape. The iconic red steam engine glows under silvery moonlight, casting golden light from its windows as it cresses arched stone viaduct. An owl flies alongside. Hogwarts Castle towers majestically, with dork lake below.
A stream, alorgside with trees and on the rocks, from left to right, is a gray British Sharthair a yellow American robin, brown Maltipoo dog. The Maltipoo dog sit quietly, showing on adorable smile.
FLUX-dev
InstanceDiffusion
FLUX-dev
RPG
InstanceDiffusion
right
night
Shorthoi
Britis Shor
Irther Shorthei
red engine:
red engine.
red engine
brewn dog mihht
brown deg
brown dog right
golden light
golzen light
goiden light
right
on
on awl
owl
en the reckot
on rocks
on the rocks
lastle
at quietty adoorable smil
quastly
quastly
lake below
lakes Ielooy
liste bailon
estorable smile
adonable snile
Figure I: Motivation of IterComp. We select three types of compositional generation methods.
The results show that different models exhibit distinct strengths across various aspects of composi- tional generation. fig. 3 further demonstrated these distinct strengths quantitatively.
ages the impressive reasoning abilities of Large Language Models (LLMs) to decompose complex generation tasks into simpler subtasks (Yang et al., 2024b; Smost-Team, 2024; Wang et al., 2024b). RPG (Yang et al., 2024b) employs MILMSs as the global planner to transform the process of gener- ating complex images into multiple simpler generation tasks within subregions. However, it requires designing complex prompts for LLMs, and it is challenging to achieve precise generation results due to their intricate outputs (Yang et al., 2024b).
We conducted extensive experiments to explore the unique strengths of different models in compo- sitional generation. As shown in the left example in fig. 1, text-to-image model FLUX (BlackForest, 2024) demonstrates impressive performance in attribute binding and aesthetic quality due to its ad- vanced training techniques and model architecture. In contrast, layout-to-image model InstanceDif- fusion (Wang et al., 2024a) struggles to capture fine-grained visual details, such as 'night scene' or "gllenn light.' In the right example of fig. 1, where the text prompt involves complex spatial relation- ships between multiple objects, FLUX (BlackForest, 2024) exhibits limitations in spatial awareness. In contrast, InstanceDifflsiin (Wang al., 2024a) excels in handling spatial relationships through layout guidance. This demonstrates that different models exhibit distinct strengths across various aspects of compositional generation. Moreover, fig. 3 further demonstrated these distinct strengths quantitatively. Naturally, a pertinent question arises: Is there a method capable of excelling in all aspects of compositional generation?
In order to enable the diffusion model to improve compositional generation comprehensively, we present a new framework, IterComp, which collects composition-on awaae model preferences from var- ious models, and then employs novel yet simple iterative feedback learning framework to achieve comprehensive improvements in compositional generation. Firstly, we select six open-sourced mod- els excelling in different aspects of compositionality form our model gallery. We focus three essential compositional metrics: attribute binding, spatial relationships, and non-spatial relationships to curate new composition-awanl model preference dataset, which consists of large number of image-rank pairs. Next, to comprehensively capture diverse composition-awanl train reward models to provide fine- fine-grained compositional guidance during the finetuning of the base diffusion model. Finally, given that compositional generation is difficult to optimize, we pro- pose iterative feedback learning. This approach enhances compositionality in a closed-loop manner, allowing for the progressive self-refinement of both the base diffusion model and reward models in multiple iterations. We theoretically and experimentally demonstrate the effectiveness of our method and its significant improvement in compositional generation.
model preferences,
Our contributions are summarized as follows:
We propose the first iterative composition-awaan reward-controlled framework herComp, to comprehensively enhance the compositionality of the base diffusion model.
We curate a model gallery and develop a high-quality composition- conposition-aware model preference dataset comprising numerous image-rank pairs.
2

---

Published as a conference paper at ICLR 2025
We utilize new iterative feedback feedback learning framework to progressively enhance both the reward models and the base diffusion model.
Extensive qualitative and quantitative comparisons with previous SOTA methods demon- strate the superior compositional generation capabilities of our approach.
2 RELATED WORK
Compositional Text-to-lmage Generation Compositional text-to-image generation is a complex and challenging task that requires a model with comprehensive capabilities, including the under- standing of complex prompts and spatial awareness (Yang et al., 2024b; Zhang et al., 2024b). Some methods enhance prompt comprehension by using more powerful text encoders or architectures (Esser et al., 2024; Betker et al., 2023; Hu et al., 2024; Dai et al., 2023). Stable Diffusion 3 (Esser et al., 2024) utilizes three different-sizee text encoders to enhance prompt comprehension. DALL-E 3 (Betker et al., 2023) enhances the understanding of rich textual details by expanding image cap- tions through recaptioning. However, compositional capability such as spatial awareness remains a limitation of these models (Li al., 2023; Chen al., 2024a). Other methods attempt enhance spatial awareness by the control of additional conditions (e.g.. layouts) (Yang et al., 2023; Dahary al., 2024). BoxDiff (Xie et al., 2023) and LMD (Lian et al.. 2023b) guide the generated objects to strictly adhere to the layout by designing energy functions based on cross-attention maps. Con- trolNet (Zhang et al., 2023) and T2I-Adapter (Mou al., 2024) specify high-level image features to control semantic structures. Although these methods enhance spatial awareness, they often com- promise image realism (Zhang et al., 2024b). Additionally, some approaches leverage the powerful reasoning capabilities of LLMs to assist in the generation process (Yang et al., 2024b; Omost-Tean Team, 2024; Wang et al., 2024b). RPG (Yang et al., 2024b) employs MILMM to decompose complex compositional generation tasks into simpler subtasks. However, these methods require designing complex prompts as inputs to the LLM, and the diffusion model struggles to produce precise results due to the LLM's intricate outputs (Yang et al., 2024b). In contrast, method extracts these prefer- ences from different models in model gallery and trains composition-awee -warre reward models to refine the base diffusion model iteratively, achieving robust compositionality across multiple aspects.
Diffusion Model Alignment Building on the success of reinforcement learning from human feed- back (RLHF) in Large Language Models (LLMs) (Ouyang et al., 2022; Bai et al., 2022), numerous methods in diffusion models have attempted to use similar approaches for model alignment (Lee al., 2023; Fan et al., 2024; Sun al., 2023). Some methods use pretrained reward model train a new one to guide the generation
Process(Zhang et al., 2024a; Black et al., 2023; Deng et al., 2024; Clark et al., 2023; Prabhudesai et al., 2023). For instance, ImageReward (Xu al., 2024) manu- ally annotated large dataset of human- preffrred images and trained a reward model to assess the alignment between images and human preferences. Reward Feedback Learning (ReFL) is proposed for tuning diffusion models with the ImageReward model. RAHF (Liang et al., 2024a) is trained on RichHF-18K, a high-quality dataset rich human feedback, and is capable of predicting the unreasonable parts in generated images. Some methods bypass the training of reward model and directly finetune diffusion models on human preference datasets ( (Yang et al., 2024a; Liang et al., 2024b; Yang et al., 2024c). Diffusion-DPO (Wallace et al., 2024) reformulates Direct Preference Optimization (DPO) to account for a diffusion model's notion of likelihood, utilizing the evidence lower bound to derive differentiable objective. The potential for alignment in diffusion models goes beyond this. We iteratively align the base model with composition-awann model preferences from the model gallery, effectively enhancing its performance on compositional generation.
3 METHOD
In this section, we present our method, IterComp, which collects composition-aware model prefer- ences from the model gallery and utilizes iterative feedback learning to enhance the comprehensive capability of the base diffusion model in compositional generation. An overview of IterComp is illustrated in fig. 2. In section 3.1, introduce the method for collecting the composition- model preference dataset from the model gallery. In section 3.2, we describe the training process for the composition-aware reward models and multi-reward feedback learning. In section 3.2,
3

---

Published as a conference paper at ICLR 2025
Model Gallery
Inerative Feedback Learning
SDXL
Zngl
pars
Attribute Binding
FLUX
Tng?
Emg5
Not-spatial
Relationship
5D
Por wide
Compos Award Reward Hiddals
Itenstion Compositionality
[omposition-Auurre
SD
1.5
Ing
Tnga
Traning
Liil
Model Preference
Ranking
Imgp
Inçó
Looping
Spatial Relationship
Instance. fusiin
Tngs
Collect Compositiat-AAuure Model Preferee
Expond Marel Golley
Sing
Iteration
Prompt: Farrustiod world,
adorned with greasery nee mojset caly from the opean.
Rewards
comnected by finur bridgns
the uir Sallools navigate the
Iter Comp
prietin wchere
Multi-Reward Feedbock Learning
Feedback Optinization
Figure 2: Overview of IterComp. We collect composition-awann model preferences from multiple models and employ an iterative feedback learning approach to enable the progressive self-refinement of both the base diffusion model and reward models.
propose the iterative feedback learning framework to enable the self-refinement of both the base diffusion model and reward models, progressively enhancing compositional generation.
3.1 COLLECTING HUMAN PREFERENCES COMPOSITIONALITY
Compositional Metric and Model Gallery We focus on three key aspects of compositionality: attribute binding, spatial relationships, and non-spatial relationships (Huang al., 2023), collect composition-aware model preferences. We initially select six open-sourced models excel in different aspects of compositional generation as our model gallery: FLUUX-ev (BlackForest, 2024), Stable Diffusion (Esser et al., 2024), SDXL (Podell al., 2023), Stable Diffusion 1.5 (Rombach et al., 2022), RPG (Yang et al., 2024b), and IistanceDiffusion (Wang et al., 2024a).
Human Ranking Attribute Binding For attribute binding, randomly select 500 prompts from each of the following categories: color, shape, and texture in the T2I-CompBench (Huang al., 2023), resulting in total of 1,500 prompts. Three professional experts ranked the images generated by the six models for each prompt, and their rankings were weighted to determine the final result. The primary criterion is whether the attributes mentioned in the prompt were accurately reflected in the generated images, especially the correct representation and binding of attributes to the corresponding objects.
Human Ranking on Complex Relationships For spatial and non-spatial relationships, select 1,000 prompts for each category from the T21-CompBench (Huang et al., 2023) and apply the same manual annotation method to obtain the rankings. For spatial relationships, the primary ranking criterion is whether the objects are correctly generated and whether their spatial positioning matches the prompt. For non-spatial relationships, the focus is whether the objects display natural and realistic actions.
Analysis of Composition-aware Model Preference Dataset For each prompt, we obtain 6 im- ages and (*) 15 image-rank pairs. As shown table in total, we collected dataset with 22,500 image-rank pairs for model preference in attribute binding, 15,000 for spatial relationships, and 15,000 for non-spatial relationships. We visualize the proportion of generated images ranked first for each model in fig. 3. The results demonstrate that different models exhibit distinct strengths across various aspects of compositional generation, and this dataset effectively captures diverse range of composition-awaat model preferences.
3.2 COMPOSITION-AWAR MULTI- FEEDBACK LEARNING
Composition-awwre Reward Model Training To achieve comprehensive improvements in com- positional generation, utilize three types of composition-aware datasets descrihed in section 3.1,
4

---

Published as a conference paper at ICLR 2025
Table I: Statistics on the composition-aware model preference dataset. The dataset con- sists of 3,500 text prompts, 27,500 images, and 52,500 image-rank pairs.
Binding
Spatial Relationship
Non-spatial Relatonship
Canegoey
Texts
Images
Image ranl pairs
1,500
Spatial Relationship
1,300
6,000
15,000
20
40
60
80
100
Non-spatial Relationship
1,000
6,000
15,000
FLUX
RPC
50
50
1.5
InstanceDirfusion
Total
3,500
21,000
$2,500
Figure 3: The proportion of each model ranked first.
decomposing compositionality into three subtasks and training a specific reward model for each. Specifically, the reward model Rg, (c, xo) is trained using the input format æp > zh c, where 2.0 and o denoting the "winning" and "losing" images, denoting the text prompt. We select two images corresponding to the same prompt from the composition-awe aware model preference datasets to form an input image-rank pair, and trained the reward model using the following loss function:
((0.) (c,m wb)-D, [log (a (Ro, (c, x0))) Re, (c.x))))]
(1)
where denotes the composition-awan model preference dataset, o() is the sigmoid function.
The three composition-aware reward models apply BLIP (Li et al., 2022; Xu et al., 2024) as feature extractors. We combine the extracted image and text features with cross attention mechanism, and learnable MLP to generate
scalar for preference comparison.
Multi-Reward Feedback Learning Due to the multi-step denoising process in diffusion models, yielding likelihoods for their generations is impossible, making the RLHF approach used in language models unsuitable for diffusion models. Some existing methods (Xu et al., 2024; Zhang et al., 2024a) finetune diffusion models directly by treating the scores of the reward model as the human preference loss. To optimize the base diffusion model using multiple composition-awee aware reward models, we design the loss function as follows:
(()) ABe Σ (eb (Ri (cj,Po (cz))))
where C = (c1, C2..... denotes the prompt set, (o(c) denotes the generate image of diffusion model with parameter 9 under the condition of prompt We calculate the loss for each reward model R.(-) and sum them to obtain the multi-reward feedback loss.
3.3 ITERATIVE OPTIMIZATION OF COMPOSITION-WWAR FEEDBACK LEARNING
Compositional generation challenging to optimize due to its inherent complexity and multifaceted nature, requiring both reward models and base diffusion model excel in aspects such com- plex text comprehension and the generation of complex relationships. To ensure more thorough optimization, propose iterative feedhack learning framework that progressively refines both the reward models and the base diffusion model over multiple iterations.
At the (k+ 1)-th iteration of the optimization described in section 3.2, we denote the reward models and the base diffusion model from the previous iteration as R+)-) (-) and (-). respectively. For each prompt in the datasets & we sample an image zj pă (c) and expand the composition-awes aware model preference dataset D with the sampled image. The image rankings for each prompt are updated using the trained reward model R (-), while preserving the relative ranks of the initial six images. Following this process, we update the composition-awwree model preference dataset to more comprehensive version, denoted as D Using this dataset, we finetune both the reward models and the base diffusion model get R (-) and Pg &+1 (-). The detailed process of iterative feedback learning can be found in algorithm 1.
Effectiveness of Iterative Feedback Learning Through this iterative feedback learning frame- work, the reward models become more effective understanding complex compositional prompts,

---

Published as a conference paper at ICLR 2025
Algorithm Iterative Composition-awarr Feedback Learning
Dataset: Composition-aware model preference dataset Do [[(e), , m. æb),.... ................. (Cn, wo ab)
Prompt set C = {c1, C2..... C
Input: Base model with pretrained parameters pe. reward model R. reward-to-loss map func- tion 6, reward re-weight scale X, iterative optimization iterations iter Initialization: Number noise scheduler time steps T, time step range for finetuning [Tr.7b]
1: for k -...... iter do for (ci, v. 2b) E Dr do
2:
3:
+ log (or (Rá, (c, xi))
R2. (c. // Reward model loss
4:
Rb t -R (c,, a. x/)) // Update the reward models
5:
end for // Get Rk+1 after training
6:
for c; E C do
7:
t - rand(Ti, T2) PPick a random timestep t [Ti]
8:
(O,
9:
for j = -T....,t +......+....... do
10:
no grad: žパ-1 + Pg. (zz) end for
11:
12:
with grad: 21-1 Pb. (zt)
13:
To VaeDec(22) (20) 21-1 // Predict image from original latent
14:
L A01o R RØt1 (ci, an)) // Multi-reward feedback learning loss
po.., + Pb. // Update the base diffusion model
16:
end for 1/ Get phtl after training
for (c,, xii 2g) Dr do
18: T. + pk (c:)
// Sample images from optimi aed base model
end for
20:
DR+11 + ronk(Da U Uzū) // Expand the dataset and update ranking
21: end for
providing more comprehensive guidance to the base diffusion model for compositional generation. The optimization objective the iterative feedback learning process is formalized in the following lemma (proof provided in the appendix A.2):
Lemma 1. The unified optimization framework of iterative feedbacck learning can be formulated as:
pd
(xh.r
c)
Pe (x'aT
I lc)
max (0) = (oc.
log
Blog
Blog
Prof (xber o))
(3)
Prot (x
where p-() denotes the optimized base diffusion model. We simplify the bilevel problem of iterative feedback learning into single-level objective. Based this, present the following theorem regarding the gradient of this objective:
Theorem 1. Assume that Fa(c, xi xb)) log (8 (Blgg
Pi(mnrle) Prot (Tū)le) 3 log pāl (((x...re))) the gra- Pout (c)
dient of optimization object can written as the sum of two terms: VoJ = Ti + T2, where:
Ti = E [[V log po (x c) V lo (# c)) Fa (c, zb)1
(4)
T2 = (- m/) [oo xi ab)1
(5)
It is evident that T2 represents the gradient form of direct preference optimization. In addition, we have another term Ti, which guides the gradient of optimization objective. As shown in eq. (4), the gradient directs the generation of ax and æi to optimize the implicit reward function Fo(c, x z(). The gradient term T; helps the model better distinguish between winning and losing samples, in- creasing the probability of generating high-quality images while reducing the probability of gener- ating low-quality images. This improves the model's 's alignment with the reward model's preferences during generation, thereby enhancing the comprehensive capabilities of compositional generation.

---

Published as a conference paper at ICLR 2025
Superiority over Diffusion-DPO and ImageReward Here clarify superiorities of Iter- Comp over Diffusion-DPP (Wallace et al., 2024) and ImageReward (Xu et al., 2024). Our IterComp first focuses composition-awes awarre rewards to optimize T2I models for realistic complex generation scenarios, and constructs a powerful model gallery to collect multiple composition-awan model preferences. Then novel iterative feedback learning framework effectively achieve progres- sive sell-refinement of both base diffusion model and reward models over multiple iterations.
Text-controlled
LLM-controlled
Layout- controlled
Reward-controlled
23,02 s/Img
15.57 s/ s/Img
s/Img IterComp (Ours)
s/Img
FLUX- dev
RPG
InstanceDiffusion
colossal, free with made of ice towers mystical castle. 6reen line both sides, while cascading waterfalls and on ethereal glow adonn the scone, The bockdrap features towering mountains and vibrant, colorful sky.
rooftap skyscroper in bustling cyberpunk city, a figure a trench stands amidst garden of bio-luminescent plants, overlocking the maze of flying cars and towering ho holograms. Robotic birds among foliage, digital billboards flesh advertisements the distance.
visor
magical seascape, majestic ship through crystal waters surrounded by vibrant and soaring birds. Towering c cliffs frame the scone, while stunning rainbow arches across the sky, blending clouds. enchanting journey captures beauty wonders.
Under luminous moon, serene Japonese garden with traditional poyodes a tronquil pond creates a magical night scone. The soft glow from the lontern lit buildings reflects on the water, blending and architecture harmony. moonlight
landscape, enhancing peaceful ambiance.
Figure 4: Qualitative comparison between our IterComp and three types of compositional genera- tion methods: text-controlled, I.LM-controlled, and layout-controlled approaches. IterComp the first reward-controlled method for compositional generation. utilizing an iterative feedback learning framework enhance the compositionality generated images. Colored text denotes the advan- tages of IterComp in generated images.
4 EXPERIMENTS
Datasets and Training Setting The reward models are trained on the composition-awe aware model preference dataset, consisting of 3,500 prompts and 52,500 image-rank pairs. For training the three reward models, we finetune BLIP and the learnable MLP with a learning rate of 1e 5 and a batch size of 64. During the iterative feedback learning process, randomly select 10,000 prompts from

---

Published as a conference paper at ICLR 2025
Table 2: Evaluation results about compositionality on T21-CompBench (Huang et al., 2023). Iter- Comp consistently demonstrates the best performance regarding attribute binding, object relation- ships, and complex compositions. We denote the best score in blue and the second-best score in green The baseline data is quoted from GenTron (Chen et al., 2024b).
Attribute Binding
Object Relationship
Model
Complext
Color 1
Shapet
Texturef
Spatialf
Non-Spatial"
Stable Diflusion 1,4 (Rombuch et al., 2022)
0.3765
0.3576
0.4156
0.1246
0,3079
0.3080
Stable Diffusion (Rombach et al., 2022)
0.5065
0.4221
0.4922
0.1342
0.3096
0.3386
Attn-Fxct v2 (Chefer et al., 2023)
0.6400
0.4517
0.5963
0.1455
0.3109
0.3401
Stable Diffusion XL (Betker et al., 2023)
0.6369
0,5408
0,5637
0.2032
0,3110
0.4091
PixArt-a (Chen al., 2023)
0.6886
0.5582
0.7044
0,2082
0.3179
0.4117
ECLIPSE (Patel et al., 2024)
0.6119
0,5429
0.6165
0.1903
0.3139
Dimba-G (Fei et al., 2024)
0.6921
0,5707
0,6821
0.2105
0.3298
0.4312
GenTron (Chen ct al., 2024b)
0.7674
0,5700
0.7150
0,2098
0.3202
0,4167
GLIGEN (Li et al., 2023)
0.4288
0.3998
0.3904
0.2632
0,3036
0.3420
LMD+ (Lian et al., 202.3u
0.4814
0.4865
0,5699
0.2537
0.2828
0,3323
InstanceDiffusion (Wang et al., 2024a)
0,5433
0.4472
0.5293
0.2791
0.2947
0,3602
IterComp (Ours)
0.7982
0.6217
0.7683
0.3196
0.3371
0.4873
DiffusionDB (Wang et al., 2022) and use SDXL (Betker et al., 2023) as the base diffusion model, finetuning it with a learning rate of le - 5 and a batch size of 4. We set T = 40, [Ti, Tz] = 10]. ReLU, and À le 3. All experiments
conducted on 4 NVIDIA A100 GPUs.
Baseline Models We curate a model gallery of six open-source models, each excelling in different aspects of compositional generation: FLUX (BlackForest, 2024). Stable Diffusion 3 (Esser al., 2024). SDXL (Betker et al., 2023), Stable Diffusion 1.5 (Rombach et al., 2022), RPG (Yang et al., 2024b), and InstanceDifffsion (Wang et al., 2024a). To ensure the base diffusion model thoroughly and comprehensively learns composition-awann model preferences, we progressively expand the model gallery by incorporating new models (e.g., Omost (Omost-Team, 2024). Stable Cascade (Per- nias et al., 2023), PixArt-a: (Chen et al., 2023)) at each iteration. For performance comparison in compositional generation, we select several tfte-off-the-rt methods, including FLUX (BlackForest, 2024), SDXL (Betker et al., 2023), and RPG (Yang et al., 2024b) to compare with approach. We use GPT-4o (OpenAI, 2024) for the L.MM-controledd methods. Additionally, GPT-44 is also employed to infer the layout from the prompt for the layout-controlled methods.
4.1 MAIN RESULTS
Qualitative Comparison As shown in fig. 4, IterComp achieves superior compositional genera- tion results compared to the three main types of compositional generation methods: text-controlled, LL.M-controlled, and layout-controlled approaches. In comparison to text-controlled methods FLUX (BlackForest, 2024), IterComp excels in handling spatial relationships, significantly reduc- ing errors such as object omissions and inaccuracies in numeracy and positioning. When compared to LLM--ontrolled methods like RPG (Yang et al., 2024b), IterComp produces more reasonable object placements, avoiding the unrealistic positioning caused by LLM hallucinations. Compared to layout methods like InstanceDiffusion (Wang et al., 2024a), IterComp demonstrates clear advantage in both semantic aesthetics and compositionality, particularly when generating under complex prompts.
Quantitative Comparison We compare IterComp with previous outstanding compositional text/layout-to- models on the 121-CompBench (Huang et al., 2023) in six key compositional scenarios. As shown in table 2, IterComp demonstrates remarkable preference across all evalu- ation tasks. Layout-contooledd methods such as LMD+ (Lian et al., 2023a) and InstanceDifflusion (Wang et al., 2024a) excel in generating accurate spatial relationships, while text-to-image mod- els like SDXL (Betker et al., 2023) and Trnn (Chen et al., 2024b) exhibit particular strengths in attribute binding and non-spatial relationships. In contrast, IterComp achieves comprehensive improvement in compositional generation. It obtains the strengths of various models by collecting composition-awe aware model preferences, and employs novel iterative feedback learning to enable self-refine refferefinement of both the base diffusion model and reward models in closed-loop manner.
IterComp achieves a high level of compositionality while simultaneously enhancing the realism and aesthetics of the generated images. As shown in table 3, we evaluate the improvement in image
8

---

Published as a conference paper at ICLR 2025
Table 3: Evaluation on image realism.
Table 4: Evaluation on inference time.
Model
CLIP Score" Aesthetle Seore? ImageReward!
Time.
Stable Diffusion 1.4 (Rombach et al., 2022)
0.307
5.326
-0.655
Stable Diflusion 2.1 (Rombach 2022)
0.521
5458
0:216
FLUX
s/img
Stable Diflision XI. (Betker
0.780
Stable Diffusice XL. (Betker al., 2023)
5.63 s/lmg
Omsost (Omost: Team, 2024)
21.08 s/img
2023)
-0.077
RPCi
20244)
15.57 s/ing
LMD+ [Lian et al., 2023a)
0.298
4,964
-0.072
DisiaieDillosoon (Wang al., 2024a)
9.88 s/1mg
InstanoDiifusion (Wang 2024a)
0.302
5,012
-0.035
IterComp (Ours)
0.357
5.96
1457
IteCComp (Ours)
5.67 sllmg
Table 5: Comparison between IterComp and other diffusion alignment methods.
Model
Average Result on T21-CB*
CLIP Score"
Aesthetic Score?
Stable Diftusion XL (Betker al., 2023)
0.4441
0.322
5.531
Diffasion-DPO) (Wallace et al., 2024)
0.4417
0.326
S.572
ImageReward (Xu et 2024)
0.4639
0.323
5.613
IterComp (Ours)
0.5554
0.337
5,936
realism by calculating the CLIP Score, Aesthetic Score, and ImageReward. IterComp significantly outperforms previous models across all three scenarios, demonstrating remarkable fidelity and pre- cision in alignment with the complex text prompt. These promising results highlight the versatility quantitative comparison results
of IterComp in both compositionality and fidelity. We provide between IterComp and other diffusion aligmment methods in appendix A.7.
IterComp requires less time generate high-quality images. In table we compare the inference time of IterComp with other outstanding models, such as FLUX (BlackForest, 2024), RPG (Yang al., 2024b) generating single image. Using the same text prompts and fixing the denois- ing steps to 40, IterComp demonstrates faster generation, because it avoids the complex attention computations in RPG and Omost. Our method can incorporate composition-awee different models without adding any computational overhead. This efficiency highlights its potential for various applications and offers a new perspective on handling complex generation tasks.
knowledge from
We compare IterComp with state-of-the -frte--rrr diffusion alignment methods, Diffusion- -PO (Wallace et al., 2024) and Rewae (Xu al., 2024). As demonstrated in table 5, IterComp significantly outperforms previous diffusion alignment methods across all three scenarios. Iterative feedback learning allows models to achieve self-refinement over multiple iterations, resulting in comprehen- sive improvements in compositionality and realism.
4.2
ABLATION STUDY
0.338
5.95
1.45
0.336g0.330
5.933
5,936
CLIP Score
0.336
0.3358
Aesthetic Score
5.90-
5.894
Imagereward
1.40-
1.420
1.422
1.381
0.334
5.85
1.35
0.332
0.330
4
5.80
3
5
6
1.30
2
3
Size of Model Gallery
Size of Model Gallery
Size of Model Gallery
(a) Impact on CLIP Score.
(b) Impact on Aesthetic Score.
(c) Impact on ImageReward.
Figure 5: Ablation study on the model gallery size.
In the ablation study model gallery size, as shown in fig. 5, we observe that increasing the size of the model gallery leads to improved performance for Iter- Comp across various evaluation tasks. To leverage this finding and provide more fine-grained reward guidance, we progressively expand the model gallery over multiple iterations by incorporating the optimized base diffusion model and new models such Omost (Omost-Ters -eaa,, 2024).
Effect of Model Gallery Size
Effeet of Composition-awar Feedback Learning We conducted ablation study (see fig, 6) to evaluate the impact of composition-aware iterative feedback learning. The results show that this approach significantly improves both the accuracy of compositional generation and the aesthetic quality of the generated images. As the number of iterations increases, the model's preferences gradually converge. Based on this observation, set the number of iterations to 3 in IterComp.

---

Published as a conference paper at ICLR 2025
SDXL
Iteration
Iteration
Iteration
SDXL
Iteration
Iteration
Ix the heart of bustling city coloasa treehouse complex rises, blending rature with urban life. Elevated weoden homes nestle among ush branches, creating serene, green oesis amidst modern skyscropers and streets below.
Majestic sailing ships navigate waves, vibront underwcter world heams with marine life and ancient ruins. sea turtle glides through the crystol-clesr waters, bridging reains aceanic mystery and marihime history.
seas dbove, while beneath the
Figure 6: Ablation study on the iterations of feedback learning.
Omost
Omost with IterComp
Omost
Omost with IterComp
picfuresque tage, and sits peacefully along
climbing
cobblestone path, Warm embrace of the ofternoon sun. RPG
RPG with IterComp
Amidst stormy, apocalyptic skyline, in intricate armer and a flowing cape. Lightring illumirates the dark clouds behind him, highlighting his steely determination. With futunistic city in ruins at his back and red sword hand, he embodies the fusion of oncient volor and odvanced techno egy, ready face the chacs anead.
valley towering red caryons
scene framed by green vegetation and cacti, under a blue sky with fluffy clouds.
RPG
RPG with IterComp
Futuristic and prehistonic worlds collide: Dinosours room near a medieval costle, flying cors and advanced skyscrapers dominate the skeyline. winds through lush greenery. blending ancient and modern civilizations surreal landscope.
Figure 7: The generation performance of integrating IterComp into RPG and Omost.
Generalization Study IterComp can serve as powerful backbone for various compositional gen- eration tasks, leveraging its strengths in spatial awareness, complex prompt comprehension, and faster inference. As shown in fig. 7, we integrate IterComp into Omost (Omost-Teane 2024) and RPG (Yang al., 2024b). The results demonstrate that equipped with the more powerful IterComp backbone, both Omost and RPG achieve excellent compositional generation performance, highlight- ing IterComp's 's strong generalization ability and potential for broader applications.
5 CONCLUSION
In this paper, we propose a novel framework, IterComp, to address the challenges of complex and compositional text-to-image generation. IterComp aggregates composition-aware model pref- erences from a model gallery and employs an iterative feedback learning approach to progressively refine both the reward models and the base diffusion models over multiple iterations. For future work, we plan to further enhance this framework by incorporating more complex modalities as in- put conditions and extending more practical applications.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENT
This work is supported by National Natural Science Foundation of China (U23B2048, U22B2037), Beijing Municipal Science and Technology Project (Z231 (2211100010323000) research grant No. SH2024JK29 and High-performance Computing Platform of Peking University. This work also supported by the National Key Research and Development Program of China (No. 2024YFB2808903), the research fund of Tsinghua University Tencent Joint Laboratory for In- ternet Innovation Technology.
The author team would like to deliver sincere thanks to Ruihang Chu from Tsinghua University for his significant suggestions for the refinement of this paper.
REFERENCES
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.00 05862, 2022.
James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang. Joyce Lee, Yufei Guo, et al. Improving image generation with better captions. Computer Science. https://edn. openai. com/papers/dall---3 pdf. 2(3):8, 2023.
Kevin Black, Michael Janner, Yilun Du, Ilya Kostrikov, and Sergey Levine. Training diffusion models with reinforcement learning. arXiv preprint 2rXiv:2305.13330 2023.
BlackForest. Black forest labs; frontier lab, 2024. URL https: http://///laaaccaftaaaaaaa ai/.
Hila Chefer, Yuval Alaluf, Yael Vinker, Lior Wolf, and Daniel Cohen-Or. Attend-and-excite: Attention-basee semantic guidance for text-to-image diffusion models. ACM Transactions on Graphics (TOG), 42(4):1-10, 2023.
Junsong Chen, Jincheng Yu, Chongjian Ge. Lewei Yao, Enze Xie, Yue Wu, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, et al. Pixart-a: Fast training of diffusion transformer for photore- alistic text-to-image synthesis. arXiv preprint arXiv:2310.00426, 2023.
Minghao Chen, Iro Laina, and Andrea Vedaldi. Training-free free layout control with cross-attention guidance. In Proceedings of the IEEE/CVF Winter Conference Applications of Computer Vision, pp. 5343-5353 2024a.
Shoufa Chen, Mengmeng Xu, Jiawei Ren, Yuren Cong, Sen He, Yanping Xie, Animesh Sinha, Ping Luo, Tao Xiang, and Juan-Manuel Perez-Rua. Gentron: Diffusion transformers for image and video generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6441-6451, 2024b.
Kevin Clark, Paul Vicol, Kevin Swersky, and David Fleet. Directly fine-tuning diffusion models on differentiable rewards. arXiv preprint arXiv:2309. /7400, 2023.
Omer Dahary, Or Patashnik, Kfir Aberman, and Daniel Cohen-Or. Be yourself: Bounded attention for multi-subject text-to-image generation. arXiv preprint arXiv:2403. /6990, 2024.
Xiaoliang Dai, Hou, Chih-Yar Ma, Sam Tsai, Jialiang Wang, Rui Wang, Peizhao Zhang, Simon Vandenhende, Xiaofang Wang, Abhimanyu Dubey, et al. Emu: Enhancing image generation models using photogenic needles a haystack. arXiv preprint arXiv:2309. 15807, 2023.
Fei Deng, Qifei Wang, Wei Wei, Tingbo Hou, and Matthias Grundmann. Prdp: Proximal reward difference prediction for large-scale reward finetuning of diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7423-7433, 2024.
Mucong Ding, Souradip Chakraborty, Vibhu Agrawal, Zora Che, Alec Koppel, Mengdi Wang, Am- rit Bedi, and Furong Huang. Sail: Self improving efficient online alignment of large language models. arXiv preprint arXiv:2406.1 /5567, 2024.

---

Published as a conference paper at ICLR 2025
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederie Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. In Forty-first International Conference Machine Learning, 2024.
Ying Fan, Olivia Watkins, Yuqing Du, Hao Liu, Moonkyung Ryu, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, Kangwook Lee, and Kimin Lee. Reinforcement learning for fine- tuning text-to-image diffusion models. Advances in Neural Information Processing Systems, 36, 2024.
Zhengcong Fei, Mingyuan Fan, Changqian Yu, Debang Li, Yougiang Zhang, and Junshi Huang. Dimba: Transformer-mant mamba diffusion models. arXiv preprint arXiv:2406.00 159, 2024.
Dhruba Ghosh, Hannaneh Hajishirzi, and Ludwig Schmidt. Geneval: An object- framework for evaluating sat-to-image alignment. Advances in Neural Information Processing Systems, 36, 2024.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.
Xiwei Hu, Rui Wang, Yixiao Fang, Bin Fu, Pei Cheng, and Gang Yu. Ella: Equip diffusion models with I1m for enhanced semantic alignment. arXiv preprini urXiv:2403.05133. 2024.
Kaiyi Huang, Kaiyue Sun, Enze Xie, Zhenguo Li, and Xihui Liu. T2i-compbench: compre- hensive benchmark for open-world compositional text-to-image generation. Advances in Neural Information Processing Systems, 36:78723-78777, 2023.
Kimin Lee, Hao Liu, Moonkyung Ryu, Olivia Watkins, Yuqing Du, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, and Shixiang Shane Gu. Aligning text-to-image models using human feedback. arXiv preprint arXiv: 12192, 2023.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-iman pre- training for unified languae understanding and generation. In International conference on machine learning, pp. 12888-12900. PMLR, 2022.
Yuheng Li, Haotian Liu, Qingyang Wu, Fangzhou Mu, Jianwei Yang, Jianfeng Gao, Chunyuan Li, and Yong Jae Lee. Gligen: Open-set grounded text-to-image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22511-22521, 2023.
Long Lian, Boyi Li, Adam Yala, and Trevor Darrell. Lim-grounded diffusion: Enhancing prompt understanding of text-to-image diffusion models with large language models. arXiv preprint 1.Xi::2305.13655 2023a.
Long Lian, Boyi Li, Adam Yala, and Trevor Darrell. Llm- Alm-grounded diffusion: Enhancing prompt understanding of text-to-image diffusion models with large language models. arXiv preprint arXiv:2305. 13655, 2023b.
Youwei Liang, Junfeng He, Gang Li, Peizhao Li, Arseniy Klimovskiy, Nicholas Carolan, Jiao Sun, Jordi Pont-Tuset, Sarah Young, Feng Yang, et al. Rich human feedback for text-to-image genera- tion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19401-19411, 2024a.
Zhanhao Liang, Yuhui Yuan, Shuyang Gu, Bohan Chen, Tiankai Hang, Ji Li, and Liang Zheng. Step-aware preference optimization: Aligning preference with denoising performance at each step. arXiv preprint w.Xiv:2406.043144 2024b.
Chong Mou, Xintao Wang, Liangbin Xie, Yanze Wu, Jian Zhang, Zhongang Qi, and Ying Shan. T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models. In Proceedings of the AAAI Conference on Artificial Intelligence, pp. 4296-4304, 2024.
Omost-Team. Omost github page, 2024.
OpenAl. Hello gpt-4o, 2024. URL Utt.htcpps//ppenniicc////eeeeee/ http::///wweeeelllloo-o--ooo/
12

---

Published as a conference paper at ICLR 2025
Long Ouyang, Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to fol- low instructions with human feedback. Advances in neural information processing systems, 35: 27730-27744, 2022.
Maitreya Patel, Changhoon Kim, Sheng Cheng, Chitta Baral, and Yezhou Yang. Eclipse: A ressurce-efficient text-to-image prior for image generations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 9069-9078, 2024.
William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4195-4205, 2023.
Pablo Pernias, Dominic Rampas, Mats L Richter, Christopher Pal. and Mare Aubreville. Würstchen: An efficient architecture for large-scale text-to-image diffusion models. arXiv preprint arXiv:2306.000 00637, 2023.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952, 2023.
Mihir Prabhudesai, Anirudh Goyal, Deepak Pathak, and Katerina Fragkiadaki. Aligning text-to- image diffusion models with reward backpropagation. arXiv preprint arXiv:2310.03736 2023.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the CEBEE/VFFccofeee ence on computer vision and pattern recognition, pp. 10684-10695, 2022.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathhn, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamies. In International conference on machine learn- ing. pp. 2256-2265, PMLR, 2015.
Yang Song. Jascha Doch-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. basre-based generative modeling through stochastic differential equations. arXiv preprint arXiv:201 /. 13456, 2020.
Jiao Sun, Deqing Fu, Yushi Hu, Su Wang, Royi Rassin, Da-Cheng Juan, Dana Alon, Charles Her- rmann, Sjoerd van Steenkiste, Ranjay Krishna, al. Dreamsyne: Aligming text-to-image gen- eration with image understanding feedback. In Synthetic Data for Computer Vision Workshop@r CVPR 2024, 2023.
Bram Wallace, Meihua Dang. Rafael Rafailov, Linqi Zhou, Aaron Lou, Senthil Purushwalkam, Stefano Ermon, Caiming Xiong, Shafiq Joty, and Nikhil Naik. Diffusion model alignment using direct preference optimization. Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 8228-8238, 2024.
Xudong Wang, Trevor Darrell, Sai Saketh Rambhatla, Rohit Girdhar, and Ishan Misra. Instancedif- fusion: Instance-level control for image generation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 6232 6222, 2024a.
Zhenyu Wang, Aoxue Li, Zhenguo Li, and Xihui Liu. Genartist: Multimodal Ilm as an agent for unified image generation and editing. arXiv preprint arXiv:4407.56600, 2024b.
Zijie Wang, Evan Montoya, David Munechika, Haoyang Yang, Benjamin Hoover, and Duen Horng Chau. Diffusiondb: A large-scale prompt gallery dataset for text-to-image generative models. arXiv preprint arXiv:2210. 12220..4499 14896, 2022.
Jinheng Xie, Yuexiang Li, Yawen Huang, Haozhe Liu, Wentian Zhang, Yefeng Zheng, and Mike Zheng Shou. Boxdiff: Text-to-image synthesis with training-free free box-constrained diffusion. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 7452-7461, 2023.
Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li. Ming Ding, Jie Tang, and Yuxiao Dong. Imagereward: Learning and evaluating human preferences for text-to-image generation. Advances in Neural Information Processing Systems, 36, 2024.
13

---

Published as a conference paper at ICLR 2025
Kai Yang. Jian Tao, Jiafei Lyu, Chunjiang Ge, Jiaxin Chen, Weihan Shen, Xiaolong Zhu, and Xiu Li. Using human feedback to fine-tune diffusion models without any reward model. In Proceedings of the IEEE/CVF Conference Computer Viston and Pattern Recognition, pp. 8941-8995 2024a.
Ling Yang, Zhaochen Yu, Chenlin Meng, Minkai Xu, Stefano Ermon, and CUI Bin. Mastering text- to-image diffusion: Recaptioning, planning, and generating with multimodal Ilms. In Forty-first International Conference on Machine Learning, 2024b.
Shentao Yang, Tianqi Chen, and Mingyuan Zhou. A dense reward view on aligning text-to-image diffusion with preference. arXiv preprint arXiv:2402.08225. 2024c.
Zhengyuan Yang. Jianfeng Wang, Zhe Gan, Linjie Li, Kevin Lin, Chenfei Wu, Nan Duan, Zicheng Liu, Ce Liu, Michael Zeng, et al. Reco: Region-controlled text-to-image generation. In Pro- ceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 14246 14255, 2023.
Jiacheng Zhang, Jie Wu, Yuxi Ren, Xin Xia, Huafeng Kuang, Pan Xie, Jiashi Li, Xuefeng Xiao, Weilin Huang, Min Zheng, et al. Unifl: Improve stable diffusion via unified feedback learning- arXiv preprint arXiv:2404.05595, 2024a.
Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3836-3847, 2023.
Xinchen Zhang, Ling Yang. Yaqi Cai, Zhaochen Yu, Jiake Xie, Ye Tian, Minkai Xu, Yong Tang, Yu- jiu Yang, and Bin Cui. Realcompo: Dynamic equilibrium between realism and compositionality improves text-to-image diffusion models. arXiv preprint wxXi:2240.112905 2024b.
Dewei Zhou, You Li, Fan Ma, Xiaoting Zhang, and Yi Yang. Mige: Multi-instance generation controller for text-to-image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1818-6828, 2024.
14

---

Published as a conference paper at ICLR 2025
A APPENDIX
This supplementary material is structured into several sections that provide additional details and analysis related IterComp. Specifically, it will cover the following topics:
In appendix A. we provide preliminary about Stable Diffusion (SD) and Reward Feed- back Learning (ReFL).
In appendix A.2, we provide detailed theoretical proof of the effectiveness of iterative feed- back learning.
In appendix A.3, we conduct an experimental analysis to assess model stability.
In appendix A.4, provide the results of user study.
In appendix A.5, we present the quantitative comparison between IterComp and RPG.
In appendix A.6, we present the quantitative comparison between IterComp and two layout- based models: InstancelDiffusion and MIGC.
In appendix A.7, we provide more visualization results for IterComp.
A.1 PRELIMINARY
Stable Diffusion Stable Diffusion (SD) (Rombach et al., 2022) performs multi-step denoising on random noise 27 (0, I) to generate clear latent 2o in the latent space under the guidance of text prompt c. During the training, an input image 2o is processed by pretrained autoencoder to obtain its latent representation zo. random noise I)) injected into zo in the forward process as follow:
2, = 20 + v1 āfe
where a2 is the noise schedule. The UNet eg trained to predict the added noise with the optimiza- tion objective:
min C()) Blpo--((m) [||c cs ca t, (()))
where E() denote the preteained encoder of VAE, r--) denotes the pretrained text encoder.
Reward Feedback Learning Reward Feedback Learning (ReFL) (Xu al., 2024) is proposed to align diffusion models with human preferences. The reward model serves as the preference guidance during the finetuning of the diffusion model. ReFL begins with an input prompt and random noise 2T (N I). The noise zy is progressively denoised until it reaches a randomly selected timestep The latent Zu directly predicted from 2,, and the decoder from pretrained VAE is used to generate the predicted image Zu. The pretrained reward model R(-) provides reward score as feedback, which is used to finetune the diffusion model as follows:
min C()) Ee-c (R (c, zo))
(8)
where the prompt e is randomly selected from the prompt dataset C.
A.2 THEORETICAL PROOF OF THE EPFECTIVENESS OF ITERATIVE FEEDBACK LEARNING
A.2.1 PROOF OF LEMMA
Proof of Lemma 1. Considering the general form of RLHF, we change the optimization problem of iterative feedback learning to bilevel optimization (Wallace et al., 2024; Ding et al., 2024):
min - Beeeee (ag sb) (-lei] [log (R (c, x R xb)] R
(9)
s.t. PR= = arg max E [Exo~p())) e) R(c, ar)] (1... [p (No:T c)) ||rret (Norr c)]
where PR denotes the optimized base models under the guidance of reward model R. We have the reparameterization of the reward model (also shown in previous works by (Wallace al., 2024)):
PR (Fo:T
c)
R(c. xo) = BRar (#e (2)
log
+ +,3 log 7(c)
(10)
Prof (To.T c)
15

---

Published as a conference paper at ICLR 2025
Z(c)
Σ Prot (Farr c) exp (R(c, 2))
(11)
Substituting this reward reparameterization into eq. (9), get the new optimization objective as:
PR ( c)
Pi (zigzr' |c)
min
- 1e- mb)) (-ee)
log
Blog
log
(12)
Pie
Prof (o c)
Pref
(z/:2 c))
This optimization objective is denoted as J(pk), ) get:
PR. (2 c)
PRe (mb:r 1 c)
max J(DR) -Bjecc,
[e-C.(nn .ag mbbrpp (-1es] log B Blog
Peet (TT c) Prot ( e)
-llog
(13)
Ps
We po to parameterize the policy and formulate the final optimization objective as:
Pb (=ör c). Pref (20 [c) Prof (ar )
pb (mber (c)
max J(0) (o-C,(05 mn (m))1)))))) logo
[
-B Blog
(14)
A.2.2 PROOF OF THEOREM
Proof of Theorem I. The gradient of the optimization objective in eq. (14) can be written as:
Pd (x c) Prof (x oer [c)
Pe (ah.r c)
V VoJ - V Σ
pa( (c) Pa pa (c)
log a BBlo
Glog
Pret (z) (c)
e. m
(15)
Assume that:
Pe oor c)
Pà (#byr c)
Fo xb, x²)) = logo
Blog
B log
(16)
Prot (w c)
Pref (#b:T c)
po (xxT Zi. c) po (x)rr
c) po(2b:r c)
(17)
The gradient can be decomposed into two terms:
To) Σ Do (wr..., ab.r c) Fo(e, ar" ab)
c,m6 2i mnd
Vope p. a l Fo xh x/)+ + (B (my (x) [V (o x ab)]
c,00 20
(18)
By expanding the distribution Po Ti, a more specific form is obtained:
T Σ
Vopo (a c)) Fo 2."" ab))
c c E (Velogpa (xū c) + Ve log (#b. c)) Fè x,, æb)]
(19)
A.3 ANALYSIS ON MODEL STABILITY
To evaluate the model stability, we selected five methods for comparison: SD1.5 (Rombach et al., 2022), SDXL (Podell et al., 2023), InstanceDiffusion (Wang et al., 2024a), Diffusion-DPO (Wal- lace et al., 2024), and FLUX (BlackForest, 2024), along with two evaluation metrics: Complex and CLIP-score. Using the same 50 seeds, calculated the mean and variance of the models' per- formance for these metrics. To facilitate visualization, we used the variance of each method as the radius and scaled it uniformly by common factor (104) for stability analysis.
16

---

Published as a conference paper at ICLR 2025
T2l.CompBench Complex
0.50
0.45
CLIP-Score
0.33
0.40
0.32
0.35
0.31
0.30
501.5
SDXL
Instaniceite Diff
DIfF-DPO
FLUX
RerComp
501.5
SDXL
InstanceDiff
Diff-DPO
FLUX
IterComp
(a) T21- T21-CompBench. Complex.
(b) CLIP Score.
Figure 8: Analysis on model stability.
Regarding the stability of compositionality, shown in fig. 8a, we found that IterComp not only achieved the best overall performance but also demonstrated superior stability. This can be attributed to the iterative feedback learning paradigm enable the model to analyze and refine its output each optimization step, effectively self-correcting and self-improving. The iterative training approach generated samples rather than
enables the model to perform feedback training based on its solely relying on external data, this enables the model to steadily improve over multiple iterations based its own foundation. This enables the model steadily improve over multiple iterations, building on its existing foundation, which significantly enhances its stability.
For the stability of realism or generation quality, as shown in fig. 8b, our method also exhibited the highest stability. Therefore, the iterative training approach not only improves the model's perfor- mance but also substantially enhances its stability across different dimensions.
A.4
USER STUDY
100%
Attribute Binding
Non Spatil Relationship
Spatial Relationship
80%
76%
74%
72%.
76%
66%
64%
Winrate [%)
60%
58% 60%
5.6
40%
20%
0%
InstanceDiff Ours Vs RPG
Ours vs Omost
Ours VS SDXL
FLUX
Figure 9; Results of user study.
We conducted comprehensive user study evaluate the effectiveness of IterComp in composi- tional generation. The study involved 41 randomly selected participants from diverse backgrounds. We compared IterComp with five other methods across four aspects: attribute binding, spatial rela- tionships, non-spatial relationships and overall performance. Each comparison involved 25 prompts, culminating in final survey 125 prompts and generating 20,500 votes. From the win rate distribu- tion of IterComp shown in the fig. 9, it is evident that IterComp demonstrates significant advantages across all three aspects of compositional generation.
Specifically, compared to the layout-based model InstanceDiffusion (Wang et al., 2024a), IterComp shows an absolute advantage attribute binding. For text- bext-based models SDXL (Podell et al., 2023) and FLUX Forect 2024), IterComp leads significantly in spatial relationships. This high- lights that the model gallery design effectively collects composition-awan model preferences and enhances performance across different compositional aspects through iterative feedback learning.
17

---

Published as a conference paper at ICLR 2025
Table 6: Comparison between IterComp and RPG on DPG-Bench
Model
Global
Entity
Attribute
Relation
Other
Average
IterComp
89,91
88,64
86.73
84.77
89.74
81.17
RPG
91.01
87.39
84.53
87.92
89.84
81.28
RPG+IterComp
92.74
91.33
89.10
92.38
90.13
84.72
Table 7: Comparison between IterComp and RPG on Genval.
Model
Single Obj.
Two Obj.
Counting
Colors
Position
Color Attri
Overall
IterComp
0.97
0.85
0.63
0.86
0.33
0.41
0.675
RPG
0.97
0.86
0.66
0.79
0.30
0.38
0.660
RPG+IterComp
0.99
0.90
0.72
0.90
0.35
0.48
0.723
A.5 COMPARISON BETWEEN ITERCOMP AND RPG
We employed two up-to-date benchmarks: DPG-Bench (Hu et al., 2024) and GenEval (Ghosh et al., 2024) for testing to evaluate the capabilities of IterComp and RPG (Yang et al., 2024b) compo- sitional generation. As demonstrated in table 6 and table 7, IterComp outperforms RPG in metrics like attributes and colors. This due to our training of a specific reward model for attribute bind- ing, which iteratively enhances IterComp over multiple iterations. Leveraging the strong planning and reasoning capabilities of LLMs, RPG excels areas such as relations, counting, and position- ing. When IterComp is used the backbone for RPG, the model exhibits remarkable performance across all aspects. This highlights IterComp's superiority in compositional generation. It's important note that IterComp is simple SDXL-like model that doesn't require complex computations dur- ing inference. As a result, under the same conditions such prompts and inference steps, IterComp nearly three times faster than RPG.
A.6 COMPARISON BETWEEN ITERCOMP AND LAYOUT-BASED METHOSS
Layout
InstanceDiffussin
MISC
IterComp
Layout
InstonceDiffusion
MIGC
IterComp
bafri
clossic violin nesting elegant velvet choir, accomparied by table the right holding of
elegent fountain rests handwritten parchment, accomparied by three designs.
charming bicycle with basket of vibrant ned flowers leans house.
a lakeside table, blue roses rest an open book beside a steaming of tea. The majestic mountains and pine trees in the background atmasphere, reflection reloxation.
against a cobblestone wall near doorway, number plate, capturing the essence of quaint European village.
enhance
Figure 10: Qualitative comparison between IterComp and two layout-to-imen image methods: Instancel. iffusion and MIGC.
We provide additional experiments between IterComp. InstanceDiffusion (Wang et al., 2024a), and MIGC (Zhou et al., 2024). As shown in fig. 10, these examples clearly show that while MIGC and InstanceDifflssin can accurately generate objects in the specified positions of the layout, there is notable gap in generation quality compared to IterComp. such as aesthetics and details. Moreover, the images generated by these two methods often appear visually unrealistic, with significant flaws such as incomplete violins or mismatches between bicycle and its basket. This highlights the clear superiority of our IterComp compositional generaton.
18

---

Published as a conference paper at ICLR 2025
A.7
MORE VISUALIZATION RESULTS
Text-controlled
LLM-controlled
Layout-controlled
Reward-controlled
23,02 s/Img
15.57 s/Img
9.88 s/Img
5.63 s/Img
FLUX-dev
RPG
InstanceDiffusion
IterComp (Ours)
In cozy library nook, A young woman with expressive, natural eyes and gentle smile, soft oil brushwork adding wormth and depth to her skin, two curious cats sit beside open books, capturing a moment of quiet companionship. Shelves filled with colorful volumes surround them, and warm sunlight streams through the window, textured chalk pastel for sub tle highlights
tiny hedgehog wrapped snugly in miniature blanket, holding a red cup with both paws, with fireplace behind, lit candle on the right, and some scattered pebbles nearby.
An artist captures the serene beauty a tranquil lake surrounded by majestic mountains. Four swans glide gracefully across the water, mirraring the peaceful scene on canvas. The vibrant colors nature and the artist's focused dedication create a harmonious blend of art and reality this pic turesque setting.
family gathers by the fireplace, basking in its warm glow. Three children play on the rug while an elderly grandparent watches lovingly. Red scarves and cozy sweaters add to the warmth of the room, complementing the flickering flames of the fire.
Qualitative comparison between IterComp and three types of compositional generation methods: text-controlled, I.LM-controlled, and layout-controlled approaches. We GPT-40 to infer the layout from the prompt for InstanceDiffusion. Colored text denotes the advantages of IterComp in generated images.
Figure 11:
19

---

Published as a conference paper at ICLR 2025
Text-controlled
LLM-controlled
Layout-controlled
Reward-controlled
23,02 s/Img
15.57 s/Img
9.88 s/Img
63 s/Img
FLUX-dev
RPG
InstanceDiffusion
IterComp (Ours)
glass sphere sculpture, concealed inside the sphere is a large Pirate Ship in Lightning storm, large waves, in the dark, moonlight filters through a nearby window, casting serene glow.
The little prince standing small earth in starry sky. With bright red scorf and golden hair. he gazes at the stars, copturing the essence of adventure. by saint exupery. crocheted style.
wise and intelligent little girl, her face illuminated by the lights of the night, embodies the universe's ss mysteries. Created from constellations and galaxy nebuloe, she holds the endless power of quasar. Her expression fnsightful, as if she understands the depths of black hole.
In the heart of enchanted forest, majestic tree stands illuminated by glowing mushrooms and tiny lights. Its thick roots form staircase leading to cozy door, suggesting hidden world within. The scene vibrant and magical, inviting wonder and exploration in this mystical woodland realm.
Qualitative comparison between IterComp and three types of compositional generation methods: text-controlled, cLL--oonrlllldd and layout-controlled approaches. We use GPT-4o to infer the layout from the prompt for InstanceDiffusion. Colored text denotes the advantages of IterComp in generated images.
Figure 12:
20

---

Published as a conference paper at ICLR 2025
SDXL
IterComp (Ours)
SDXL
IterComp (Ours)
SAA NAIL
Gon PEA LNNOMA
IPAPAGAN
RITIL NERREDE
Figure 13: More visualization results for IterComp and its base diffusion model, SDXL.
21


Reviews:
[{"review_id": "ENKeelceX2", "reviewer": "Reviewer_Vtxw", "summary": "This paper proposes IterComp, an iterative composition-aware reward-controlled framework. It introduces a model gallery and constructs a high-quality composition-aware model preference dataset. Utilizing a new iterative feedback learning framework, IterComp progressively enhances both the reward models and the base diffusion model.", "questions": "Please see the weakness.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This work claims to be the first work to introduce a reward-controlled framework in the concept composition generation, which is somehow novel in this field.\n2. This work is presented well with complete theoretical details and proof.\n3. The quantitative experimental results show better performance compared to SOTAs.", "weaknesses": "1. The qualitative comparison in Fig. 4 is confused. There seems to be a marginal improvement in Line 3 and Line 4. The authors should make the difference between the qualitative examples clear to be recognized.\n2. The qualitative examples are not enough to evaluate the model performance since the cases in the paper are somehow complex. The authors can provide more examples for a single case. Also, there is a need to evaluate the stability of the proposed model.\n3. The comparison with InstanceDiffusion is confusing. As a layout-guided method, InstanceDiffusion needs detailed layout inputs. It is not fair to provide only one box if the case includes two or more instances, as indicated in the third line of Fig. 4. As the authors attempt to compare with layout-to-image methods, a SOTA method named MIGC [1] is also not included.\n\n[1] Zhou, Dewei, et al. \"Migc: Multi-instance generation controller for text-to-image synthesis.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5o5RV8qVME", "reviewer": "Reviewer_52Cf", "summary": "This paper introduces IterComp, a novel framework that enhances compositional text-to-image generation by aggregating preferences from multiple diffusion models through iterative feedback learning. The approach demonstrates superior performance in both compositional accuracy and image quality, while maintaining efficient inference speed. The main strength lies in its ability to combine different models' advantages without adding computational overhead, though the long-term stability of the iterative learning process could be further explored.", "questions": "See weakness", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "Novel and practical approach: The paper presents a simple yet effective way to combine multiple models' strengths for compositional generation without increasing computational complexity.\n\nStrong empirical results: The method shows clear improvements over existing approaches, with comprehensive evaluations on both compositional accuracy and image quality metrics.\n\nWell-structured technical contribution: The paper provides clear theoretical analysis with detailed proofs, and the iterative feedback learning framework is well-designed and easy to implement.", "weaknesses": "The paper mentioned that RPG is challenging to achieve precise generation, but Tab2 did not compare with RPG, and I checked that RPG's performance on T2I-Compbench is better than that of the paper.\n\nIt is necessary to test the results of FLUX-dev directly on the t2i compbench to see how much improvement the method proposed in this paper has. I currently suspect that the improvement may not be very significant.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fhbaP9qgC7", "reviewer": "Reviewer_HHze", "summary": "This paper propose a framework that aggregates composition-aware model preferences from multiple models and employs an iterative feedback learning approach to enhance T2I compositionality and general performance. The qualitative and quantitative results show their SOTA compositional generation capabilities compared to previous works.", "questions": "1. As mentioned in W.1, how long does the training loop take (including the iterative feedback learning)?\n2. Could I use this method to improve a specific concept generation (e.g., a human-object interaction)? How much time does it take from collecting synthetic data to finalizing the model training?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper presents a novel framework combining preferences from multiple diffusion models to enhance compositional text-to-image generation and address the relationship understanding in diffusion models.\n2. The qualitative/quantitative results show comparable improvements in compositionality.\n3. A composition-aware dataset is collected which provide diverse preferences that inform the reward models. (will it be released in the future?)", "weaknesses": "1. There is limited discussion of the computation resources required to manage multiple reward models, which may affect the scalability in large-scale applications. Although the authors claim that their model has fast inference speed, the cost of model training and data collection is not clear. This makes me feel less likely than DPO to be widely used in practice.\n2. The user study only demonstrates the user preferences lacking the deep analysis of attribute binding and object relationship, which are critical to model performance. 16 samples is also too small to evaluate such a complex task.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tBBby97ctz", "reviewer": "Reviewer_bbqi", "summary": "This study addresses improving the compositional text-to-image generation capability of less powerful diffusion models. The authors contribute in two main areas. First, they decompose the capabilities of various diffusion models into three domains—attribute binding, spatial relationships, and non-spatial relationships—and rank model outputs accordingly to guide reinforcement learning with reward models specific to each domain. Second, they introduce an iterative training process that utilizes the fine-tuned diffusion model outputs to progressively enhance the reward models.\n\nThrough multiple experiments, the study demonstrates the proposed method’s effectiveness, enabling early-stage models to achieve comparable generative abilities with reduced inference time. The authors also verify the effectiveness and general applicability of each design component across different models.", "questions": "In addition to the weakness mentioned above, I have a question regarding stability. \n\nBased on my experience, RLHF in diffusion models can often be unstable. I’m curious whether your method consistently produces stable results or if there’s a risk of occasionally poorer outcomes. I’m concerned that the iterative training process might lead to overfitting on biases present in the reward models, potentially reducing overall robustness.\n\nI hope the authors can make up for the weaknesses mentioned and address these questions.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well-structured, making it accessible and easy for readers to follow.\n2. Clear formulas are provided for each component, effectively removing any ambiguity.\n3. Mathematical proofs substantiate the validity of the proposed method.\n4. The authors conduct detailed, extensive experiments to support their approach.\n5. Illustrative images are included, enhancing clarity and understanding.", "weaknesses": "1. About the number of texts used for attribute binding, \"500 prompts\" in Line 185 is inconsistent with \"1500\" in Table 1. Which is correct?\n2. Although the experiments are detailed, some comparisons appear incomplete. The reinforcement learning from human feedback (RLHF) approach leverages outputs from advanced models like FLUX and SD3 for training, yet direct comparisons with these models are not provided. Including these comparisons would better highlight the method's effectiveness.\n3. An additional experiment focusing on the first-round fine-tuned model—trained solely with human-annotated data—would be valuable. This would clarify the necessity and impact of the iterative training approach.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nvB67hq7hk", "reviewer": "Reviewer_ocHL", "summary": "The paper presents a new dataset for compositional generation where experts evaluate the outputs of multiple pre-trained models. The dataset consists of three aspects of compositionality: attribute binding, spatial relationships, and non-spatial relationships, along with 52.5K image-rank pairs, which can be used for training feedback, ranking, or reward models.\n\nThe second contribution of the paper is using the collected dataset to train reward models for each of the three key aspects. A multimodal model (BLIP) is used as a feature extractor for both the prompts and the generated images, and the extracted features are projected by MLPs to output the reward. The goal is to predict how good the given image-prompt pairs are by training the model similar to contrastive learning (moving toward winning examples and away from losing examples).\n\nThe third contribution is improving the compositional ability of a base model (SDXL is selected but it should not be limited to that) by optimizing it using the trained reward models. The base model is trained to maximize the reward model's output so that its outputs are better aligned with the reward models, which are trying to enforce compositionality. Furthermore, the paper proposes an iterative update mechanism for both reward models and the base model. Reward models are updated to predict the rankings generated by experts while the base model is updated to maximize the outputs of reward models. Through this process, both the base model and reward models are improved for their specific tasks.\n\n**Important note:** This review's technical content and analysis are my original work. Large Language Models were used solely to improve grammar and writing style, without altering any meanings or substantive feedback.", "questions": "* Training time considerations:\n   - Diffusion-DPO operates in the latent space without requiring image decoding. In contrast, IterComp requires image decoding for the reward models (though this could be avoided by training the reward model with latent space inputs), likely resulting in slower training. Additional commentary on the training scheme would be valuable.\n* Potential code and paper discrepancy:\n   - The paper describes an iterative feedback mechanism that optimizes both reward models and the base model. However, examination of `feedback_train.py` reveals that only unet parameters (base model) are passed to the optimizer. This suggests that only the base model is being optimized, while reward models remain static. This difference requires clarification.\n* Question regarding test-time adaptation:\n   - Could the iterative feedback mechanism be applied as test-time adaptation of the base model? Similar to Slot-TTA [1], the base model could be optimized using reward models to improve compositional quality. The process would work as follows: for a given prompt, the base model generates an image, which is then evaluated by reward models. The base model's parameters would be updated to maximize these rewards. This process could be repeated for several iterations. This approach would eliminate the need for training the base model, allowing it to adapt to any prompt at test time through multiple iterations. Comments on this possibility would be valuable.\n\n[1] Test-time Adaptation with Slot-Centric Models, Prabhudesai et al., ICML 2023\n\nEDIT: After the rebuttal and discussion with the authors, I raise my score to 8 from 6.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The collected dataset can lead to better models and new research directions. Furthermore, researchers can follow a similar approach to collect their own data. The data can also be used for other RLHF methods, such as diffusion-DPO.\n* The iterative feedback mechanism seems to be a novel way to optimize both reward models and the base model.\n* The code is shared, which allows reviewers to follow the details of the proposed method.\n* The performance gain from IterComp appears significant, as evaluated through user studies, quantitative analysis, and qualitative assessment.", "weaknesses": "* Table 5 should be in the main part instead of the Appendix, as it simply demonstrates that the proposed method outperforms previous methods.\n* Several experiments are missing:\n   - The paper combines all reward models simultaneously, likely leading to improved compositional performance. However, reviewers would benefit from seeing the individual effect of each reward model.\n   - While SDXL is chosen as the base model, testing other models would help reviewers understand how reward models affect different base architectures.\n   - It would be valuable to examine Diffusion-DPO's performance when trained on the collected dataset. Currently, Diffusion-DPO is trained on the pick-a-pic dataset, which is larger but lacks compositional details. These results would be necessary to evaluate the proposed method's effectiveness using consistent standards.\n* Expert ranking may inadvertently include aesthetic information [This observation is meant to prompt discussion rather than highlight a weakness, as the underlying cause remains unclear]:\n   - When IterComp is applied, the aesthetic score improves, suggesting that reward models can interpret image aesthetics, since reward maximization leads to better aesthetic scores. This could be because either expert rankings are influenced by image aesthetics (beyond compositional attributes) or because models with better composition naturally generate more aesthetic images (for instance, FLUX, being the best compositional model, likely produces more aesthetically pleasing outputs).\n* Some experimental conditions may be misleading:\n   - The paper uses 40 inference steps for all models to ensure fairness. However, some models can generate samples with fewer steps; for example, FLUX-dev uses 28 steps by default.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)