IterComp: Iterative Composition-Aware Feedback Learning from Model Gallery for Text-to-Image Generation

## Abstract

Abstract Advanced diffusion models like RPG, Stable Diffusion 3 and FLUX have made notable strides in compositional text-to-image generation. However, these methods typically exhibit distinct strengths for compositional generation, with some excelling in handling attribute binding and others in spatial relationships. This disparity highlights the need for an approach that can leverage the complementary strengths of various models to comprehensively improve the composition capability.
To this end, we introduce IterComp , a novel framework that aggregates composition-aware model preferences from multiple models and employs an iterative feedback learning approach to enhance compositional generation. Specifically, we curate a gallery of six powerful open-source diffusion models and evaluate their three key compositional metrics: attribute binding, spatial relationships, and non-spatial relationships. Based on these metrics, we develop a composition-aware model preference dataset comprising numerous image-rank pairs to train composition-aware reward models.
Then, we propose an iterative feedback learning method to enhance compositionality in a closed-loop manner, enabling the progressive self-refinement of both the base diffusion model and reward models over multiple iterations. Theoretical proof demonstrates the effectiveness and extensive experiments show our significant superiority over previous SOTA methods (e.g., Omost and FLUX), particularly in multi-category object composition and complex semantic alignment. IterComp opens new research avenues in reward feedback learning for diffusion models and compositional generation.

## 1 Introduction

The rapid advancement of diffusion models (Sohl-Dickstein et al., ; Ho et al., ; Song et al., ; Peebles & Xie, ) has recently brought unprecedented progress to the field of text-to-image generation, with powerful models like DALL-E 3 (Betker et al., ), Stable Diffusion 3, (Esser et al., ) and FLUX (BlackForest, ) demonstrating remarkable capabilities in generating aesthetic and diverse images. However, these models often struggle to follow complex prompts to achieve precise compositional generation (Omost-Team, ; Yang et al., ; Zhang et al., ), which requires the model to possess robust, comprehensive capabilities in various aspects, such as attribute binding, spatial relationships, and non-spatial relationships (Huang et al., ).

To enhance compositional generation, some works introduce additional conditions such as layouts/boxes (Li et al., ; Zhou et al., ; Wang et al., ; Zhang et al., ). InstanceDiffusion (Wang et al., ) controls the generation process using layouts, masks, or other conditions through trainable instance masked attention layers. Although these layout-based methods demonstrate strong spatial awareness, they struggle with image realism, especially in generating non-spatial relationships and preserving aesthetic quality (Zhang et al., ). Another potential solution leverages the impressive reasoning abilities of Large Language Models (LLMs) to decompose complex generation tasks into simpler subtasks (Yang et al., ; Omost-Team, ; Wang et al., ). RPG (Yang et al., ) employs MLLMs as the global planner to transform the process of generating complex images into multiple simpler generation tasks within subregions. However, it requires designing complex prompts for LLMs, and it is challenging to achieve precise generation results due to their intricate outputs (Yang et al., ).

We conducted extensive experiments to explore the unique strengths of different models in compositional generation. As shown in the left example in fig. 1, text-to-image model FLUX (BlackForest, ) demonstrates impressive performance in attribute binding and aesthetic quality due to its advanced training techniques and model architecture. In contrast, layout-to-image model InstanceDiffusion (Wang et al., ) struggles to capture fine-grained visual details, such as ’night scene’ or ’golden light.’ In the right example of fig. 1, where the text prompt involves complex spatial relationships between multiple objects, FLUX (BlackForest, ) exhibits limitations in spatial awareness. In contrast, InstanceDiffusion (Wang et al., ) excels in handling spatial relationships through layout guidance. This demonstrates that different models exhibit distinct strengths across various aspects of compositional generation. Moreover, fig. 3 further demonstrated these distinct strengths quantitatively. Naturally, a pertinent question arises:
Is there a method capable of excelling in all aspects of compositional generation?

In order to enable the diffusion model to improve compositional generation comprehensively, we present a new framework, IterComp, which collects composition-aware model preferences from various models, and then employs a novel yet simple iterative feedback learning framework to achieve comprehensive improvements in compositional generation. Firstly, we select six open-sourced models excelling in different aspects of compositionality to form our model gallery. We focus on three essential compositional metrics: attribute binding, spatial relationships, and non-spatial relationships to curate a new composition-aware model preference dataset, which consists of a large number of image-rank pairs. Next, to comprehensively capture diverse composition-aware model preferences, we train reward models to provide fine-grained compositional guidance during the finetuning of the base diffusion model. Finally, given that compositional generation is difficult to optimize, we propose iterative feedback learning. This approach enhances compositionality in a closed-loop manner, allowing for the progressive self-refinement of both the base diffusion model and reward models in multiple iterations. We theoretically and experimentally demonstrate the effectiveness of our method and its significant improvement in compositional generation.

Figure: Figure 1: Motivation of IterComp. We select three types of compositional generation methods. The results show that different models exhibit distinct strengths across various aspects of compositional generation. fig. 3 further demonstrated these distinct strengths quantitatively.
Refer to caption: x1.png

Our contributions are summarized as follows:

- •
We propose the first iterative composition-aware reward-controlled framework IterComp, to comprehensively enhance the compositionality of the base diffusion model.
- •
We curate a model gallery and develop a high-quality composition-aware model preference dataset comprising numerous image-rank pairs.
- •
We utilize a new iterative feedback learning framework to progressively enhance both the reward models and the base diffusion model.
- •
Extensive qualitative and quantitative comparisons with previous SOTA methods demonstrate the superior compositional generation capabilities of our approach.

## 2 Related Work

##### Compositional Text-to-Image Generation

Compositional text-to-image generation is a complex and challenging task that requires a model with comprehensive capabilities, including the understanding of complex prompts and spatial awareness (Yang et al., ; Zhang et al., ). Some methods enhance prompt comprehension by using more powerful text encoders or architectures (Esser et al., ; Betker et al., ; Hu et al., ; Dai et al., ). Stable Diffusion 3 (Esser et al., ) utilizes three different-sized text encoders to enhance prompt comprehension. DALL-E 3 (Betker et al., ) enhances the understanding of rich textual details by expanding image captions through recaptioning. However, compositional capability such as spatial awareness remains a limitation of these models (Li et al., ; Chen et al., ). Other methods attempt to enhance spatial awareness by the control of additional conditions (e.g., layouts) (Yang et al., ; Dahary et al., ). BoxDiff (Xie et al., ) and LMD (Lian et al., ) guide the generated objects to strictly adhere to the layout by designing energy functions based on cross-attention maps. ControlNet (Zhang et al., ) and T2I-Adapter (Mou et al., ) specify high-level image features to control semantic structures. Although these methods enhance spatial awareness, they often compromise image realism (Zhang et al., ). Additionally, some approaches leverage the powerful reasoning capabilities of LLMs to assist in the generation process (Yang et al., ; Omost-Team, ; Wang et al., ). RPG (Yang et al., ) employs MLLM to decompose complex compositional generation tasks into simpler subtasks. However, these methods require designing complex prompts as inputs to the LLM, and the diffusion model struggles to produce precise results due to the LLM’s intricate outputs (Yang et al., ). In contrast, our method extracts these preferences from different models in model gallery and trains composition-aware reward models to refine the base diffusion model iteratively, achieving robust compositionality across multiple aspects.

##### Diffusion Model Alignment

Building on the success of reinforcement learning from human feedback (RLHF) in Large Language Models (LLMs) (Ouyang et al., ; Bai et al., ), numerous methods in diffusion models have attempted to use similar approaches for model alignment (Lee et al., ; Fan et al., ; Sun et al., ). Some methods use a pretrained reward model or train a new one to guide the generation process(Zhang et al., ; Black et al., ; Deng et al., ; Clark et al., ; Prabhudesai et al., ). For instance, ImageReward (Xu et al., ) manually annotated a large dataset of human-preferred images and trained a reward model to assess the alignment between images and human preferences. Reward Feedback Learning (ReFL) is proposed for tuning diffusion models with the ImageReward model. RAHF (Liang et al., ) is trained on RichHF-18K, a high-quality dataset rich in human feedback, and is capable of predicting the unreasonable parts in generated images. Some methods bypass the training of a reward model and directly finetune diffusion models on human preference datasets (Yang et al., ; Liang et al., ; Yang et al., ). Diffusion-DPO (Wallace et al., ) reformulates Direct Preference Optimization (DPO) to account for a diffusion model’s notion of likelihood, utilizing the evidence lower bound to derive a differentiable objective. The potential for alignment in diffusion models goes beyond this. We iteratively align the base model with composition-aware model preferences from the model gallery, effectively enhancing its performance on compositional generation.

## 3 Method

In this section, we present our method, IterComp, which collects composition-aware model preferences from the model gallery and utilizes iterative feedback learning to enhance the comprehensive capability of the base diffusion model in compositional generation. An overview of IterComp is illustrated in fig. 2. In section 3.1, we introduce the method for collecting the composition-aware model preference dataset from the model gallery. In section 3.2, we describe the training process for the composition-aware reward models and multi-reward feedback learning. In fig. 3, we propose the iterative feedback learning framework to enable the self-refinement of both the base diffusion model and reward models, progressively enhancing compositional generation.

Figure: Figure 2: Overview of IterComp. We collect composition-aware model preferences from multiple models and employ an iterative feedback learning approach to enable the progressive self-refinement of both the base diffusion model and reward models.
Refer to caption: x2.png

### 3.1 Collecting Human Preferences of Compositionality

##### Compositional Metric and Model Gallery

We focus on three key aspects of compositionality: attribute binding, spatial relationships, and non-spatial relationships (Huang et al., ), to collect composition-aware model preferences. We initially select six open-sourced models excel in different aspects of compositional generation as our model gallery: FLUX-dev (BlackForest, ), Stable Diffusion 3 (Esser et al., ), SDXL (Podell et al., ), Stable Diffusion 1.5 (Rombach et al., ), RPG (Yang et al., ), and InstanceDiffusion (Wang et al., ).

##### Human Ranking on Attribute Binding

For attribute binding, we randomly select 500 prompts from each of the following categories: color, shape, and texture in the T2I-CompBench (Huang et al., ). Three professional experts ranked the images generated by the six models for each prompt, and their rankings were weighted to determine the final result. The primary criterion is whether the attributes mentioned in the prompt were accurately reflected in the generated images, especially the correct representation and binding of attributes to the corresponding objects.

##### Human Ranking on Complex Relationships

For spatial and non-spatial relationships, we select 1,000 prompts for each category from the T2I-CompBench (Huang et al., ) and apply the same manual annotation method to obtain the rankings. For spatial relationships, the primary ranking criterion is whether the objects are correctly generated and whether their spatial positioning matches the prompt. For non-spatial relationships, the focus is on whether the objects display natural and realistic actions.

##### Analysis of Composition-aware Model Preference Dataset

For each prompt, we obtain 6 images and $\binom{6}{2}=15$ image-rank pairs. As shown in table 1, in total, we collected a dataset with 22,500 image-rank pairs for model preference in attribute binding, 15,000 for spatial relationships, and 15,000 for non-spatial relationships. We visualize the proportion of generated images ranked first for each model in fig. 3. The results demonstrate that different models exhibit distinct strengths across various aspects of compositional generation, and this dataset effectively captures a diverse range of composition-aware model preferences.

### 3.2 Composition-aware Multi-Reward Feedback Learning

##### Composition-aware Reward Model Training

To achieve comprehensive improvements in compositional generation, we utilize three types of composition-aware datasets described in section 3.1, decomposing compositionality into three subtasks and training a specific reward model for each. Specifically, the reward model $\mathcal{R}_{\theta_{i}}(\bm{c},\bm{x}_{0})$ is trained using the input format $\bm{x}_{0}^{w}\succ\bm{x}_{0}^{l}\mid\bm{c}$, where $\bm{x}_{0}^{w}$ and $\bm{x}_{0}^{l}$ denoting the ”winning” and ”losing” images, $\bm{c}$ denoting the text prompt. We select two images corresponding to the same prompt from the composition-aware model preference datasets to form an input image-rank pair, and trained the reward model using the following loss function:

$$ $\mathcal{L}(\theta_{i})=-\mathbb{E}_{\left(\bm{c},\bm{x}_{0}^{w},\bm{x}_{0}^{l }\right)\sim\mathcal{D}_{i}}\left[\log\left(\sigma\left(\mathcal{R}_{\theta_{i }}\left(\bm{c},\bm{x}_{0}^{w}\right)-\mathcal{R}_{\theta_{i}}\left(\bm{c},\bm{ x}_{0}^{l}\right)\right)\right)\right]$ (1) $$

where $\mathcal{D}$ denotes the composition-aware model preference dataset, $\sigma(\cdot)$ is the sigmoid function.

The three composition-aware reward models apply BLIP (Li et al., ; Xu et al., ) as feature extractors. We combine the extracted image and text features with cross attention mechanism, and use a learnable MLP to generate a score scalar for preference comparison.

Figure: Table 1: Statistics on the composition-aware model preference dataset. The dataset consists of 3,500 text prompts, 27,500 images, and 52,500 image-rank pairs.
Refer to caption: x3.png

##### Multi-Reward Feedback Learning

Due to the multi-step denoising process in diffusion models, yielding likelihoods for their generations is impossible, making the RLHF approach used in language models unsuitable for diffusion models. Some existing methods (Xu et al., ; Zhang et al., ) finetune diffusion models directly by treating the scores of the reward model as the human preference loss. To optimize the base diffusion model using multiple composition-aware reward models, we design the loss function as follows:

$$ $\mathcal{L}(\theta)=\lambda\mathbb{E}_{\bm{c}_{j}\sim\mathcal{C}}\sum_{i}\left (\phi\left(\mathcal{R}_{i}\left(\bm{c}_{j},p_{\theta}\left(\bm{c}_{j}\right) \right)\right)\right)$ (2) $$

where $\mathcal{C}=\{\bm{c}_{1},\bm{c}_{2},\dots,\bm{c}_{n}\}$ denotes the prompt set, $p_{\theta}(\bm{c})$ denotes the generate image of diffusion model with parameter $\theta$ under the condition of prompt $\bm{c}$. We calculate the loss for each reward model $\mathcal{R}_{i}(\cdot)$ and sum them to obtain the multi-reward feedback loss.

### 3.3 Iterative Optimization of Composition-aware Feedback Learning

Compositional generation is challenging to optimize due to its inherent complexity and multifaceted nature, requiring both our reward models and base diffusion model to excel in aspects such as complex text comprehension and the generation of complex relationships. To ensure more thorough optimization, we propose an iterative feedback learning framework that progressively refines both the reward models and the base diffusion model over multiple iterations.

Figure: Algorithm 1 Iterative Composition-aware Feedback Learning

At the $(k+1)$-th iteration of the optimization described in section 3.2, we denote the reward models and the base diffusion model from the previous iteration as $\mathcal{R}^{k}(\cdot)$ and $p_{\theta}^{k}(\cdot)$, respectively. For each prompt $\bm{c}$ in the datasets $\mathcal{D}^{k}$, we sample an image $\bm{x}_{0}^{*}=p_{\theta}^{k}(\bm{c})$ and expand the composition-aware model preference dataset $\mathcal{D}^{k}$ with the sampled image. The image rankings for each prompt are updated using the trained reward model $\mathcal{R}_{\theta}^{k}(\cdot)$, while preserving the relative ranks of the initial six images. Following this process, we update the composition-aware model preference dataset to a more comprehensive version, denoted as $\mathcal{D}^{k+1}$. Using this dataset, we finetune both the reward models and the base diffusion model to get $\mathcal{R}^{k+1}(\cdot)$ and $p_{\theta}^{k+1}(\cdot)$. The detailed process of iterative feedback learning can be found in algorithm 1.

##### Effectiveness of Iterative Feedback Learning

Through this iterative feedback learning framework, the reward models become more effective at understanding complex compositional prompts, providing more comprehensive guidance to the base diffusion model for compositional generation. The optimization objective of the iterative feedback learning process is formalized in the following lemma (proof provided in the section A.2):

###### Lemma 1 .

The unified optimization framework of iterative feedback learning can be formulated as:

$$ $\max_{\theta}\ J(\theta)\!=\!\mathbb{E}_{\left[\bm{c}\sim\mathcal{C},(\bm{x}_{ 0}^{w},\bm{x}_{0}^{l})\sim p_{\theta}^{*}(\cdot\mid\bm{c})\right]}\left[\log \sigma\left(\!\beta\log\frac{p_{\theta}^{*}\left(\bm{x}_{0:T}^{w}\mid\bm{c} \right)}{p_{\mathrm{ref}}\left(\bm{x}_{0:T}^{w}\mid\bm{c}\right)}-\beta\log \frac{p_{\theta}^{*}\left(\bm{x}_{0:T}^{l}\mid\bm{c}\right)}{p_{\mathrm{ref}} \left(\bm{x}_{0:T}^{l}\mid\bm{c}\right)}\!\right)\right]$ (3) $$

where $p^{*}(\cdot)$ denotes the optimized base diffusion model. We simplify the bilevel problem of iterative feedback learning into a single-level objective. Based on this, we present the following theorem regarding the gradient of this objective:

###### Theorem 1 .

Assume that $F_{\theta}(\bm{c},\bm{x}_{0}^{w},\bm{x}_{0}^{l})=\log\sigma\left(\!\beta\log
\frac{p_{\theta}^{*}\left(\bm{x}_{0:T}^{w}\mid\bm{c}\right)}{p_{\mathrm{ref}}
\left(\bm{x}_{0:T}^{w}\mid\bm{c}\right)}-\beta\log\frac{p_{\theta}^{*}\left(
\bm{x}_{0:T}^{l}\mid\bm{c}\right)}{p_{\mathrm{ref}}\left(\bm{x}_{0:T}^{l}\mid
\bm{c}\right)}\!\right)$, the gradient of optimization object can be written as the sum of two terms: $\nabla_{\theta}J(\theta)=T_{1}+T_{2}$, where:

$$ $T_{1}=\mathbb{E}\left[\left(\nabla_{\theta}\log p_{\theta}\left(\bm{x}_{0:T}^{ w}\mid\bm{c}\right)+\nabla_{\theta}\log p_{\theta}\left(\bm{x}_{0:T}^{l}\mid \bm{c}\right)\right)F_{\theta}\left(\bm{c},\bm{x}_{0}^{w},\bm{x}_{0}^{l}\right )\right]$ (4) $$

$$ $T_{2}=\mathbb{E}_{\left[\bm{c}\sim\mathcal{C},(\bm{x}_{0}^{w},\bm{x}_{0}^{l}) \sim p_{\theta}^{*}(\cdot\mid\bm{c})\right]}[\nabla_{\theta}[F_{\theta}(\bm{c} ,\bm{x}_{0}^{w},\bm{x}_{0}^{l})]]$ (5) $$

It is evident that $T_{2}$ represents the gradient form of direct preference optimization. In addition, we have another term $T_{1}$, which guides the gradient of optimization objective. As shown in eq. 4, the gradient directs the generation of $\bm{x}_{0}^{w}$ and $\bm{x}_{0}^{w}$ to optimize the implicit reward function $F_{\theta}(\bm{c},\bm{x}_{0}^{w},\bm{x}_{0}^{l})$. The gradient term $T_{1}$ helps the model better distinguish between winning and losing samples, increasing the probability of generating high-quality images while reducing the probability of generating low-quality images. This improves the model’s alignment with the reward model’s preferences during generation, thereby enhancing the comprehensive capabilities of compositional generation.

###### Lemma 1 .

###### Theorem 1 .

##### Superiority over Diffusion-DPO and ImageReward

Here we clarify some superiorities of IterComp over Diffusion-DPO (Wallace et al., ) and ImageReward (Xu et al., ).
Our IterComp first focuses on composition-aware rewards to optimize T2I models for realistic complex generation scenarios, and constructs a powerful model gallery to collect multiple composition-aware model preferences. Then our novel iterative feedback learning framework can effectively achieve progressive self-refinement of both base diffusion model and reward models over multiple iterations.

Figure: Figure 4: Qualitative comparison between our IterComp and three types of compositional generation methods: text-controlled, LLM-controlled, and layout-controlled approaches. IterComp is the first reward-controlled method for compositional generation, utilizing an iterative feedback learning framework to enhance the compositionality of generated images. Colored text denotes the advantages of IterComp in generated images.
Refer to caption: x4.png

## 4 Experiments

### 4.1 Experimental Setup

##### Datasets and Training Setting

The reward models are trained on the composition-aware model preference dataset, consisting of 3,500 prompts and 52,500 image-rank pairs. For training the three reward models, we finetune BLIP and the learnable MLP with a learning rate of $1e-5$ and a batch size of 64. During the iterative feedback learning process, we randomly select 10,000 prompts from DiffusionDB (Wang et al., ) and use SDXL (Betker et al., ) as the base diffusion model, finetuning it with a learning rate of $1e-5$ and a batch size of 4. We set $T=40$, $[T_{1},T_{2}]=[1,10]$, $\phi=\text{ReLU}$, and $\lambda=1e-3$. All experiments are conducted on 4 NVIDIA A100 GPUs.

##### Baseline Models

We curate a model gallery of six open-source models, each excelling in different aspects of compositional generation: FLUX (BlackForest, ), Stable Diffusion 3 (Esser et al., ), SDXL (Betker et al., ), Stable Diffusion 1.5 (Rombach et al., ), RPG (Yang et al., ), and InstanceDiffusion (Wang et al., ). To ensure the base diffusion model thoroughly and comprehensively learns composition-aware model preferences, we progressively expand the model gallery by incorporating new models (e.g., Omost (Omost-Team, ), Stable Cascade (Pernias et al., ), PixArt-$\alpha$ (Chen et al., )) at each iteration. For performance comparison in compositional generation, we select several state-of-the-art methods, including FLUX (BlackForest, ), SDXL (Betker et al., ), and RPG (Yang et al., ) to compare with our approach. We use GPT-4o (OpenAI, ) for the LLM-controlled methods and to infer the layout from the prompt for the layout-controlled methods.

### 4.2 Main Results

**Table 2: Evaluation results about compositionality on T2I-CompBench (Huang et al., ). IterComp consistently demonstrates the best performance regarding attribute binding, object relationships, and complex compositions. We denote the best score in blue and the second-best score in green. The baseline data is quoted from GenTron (Chen et al., ).**
| Model | Attribute Binding | Object Relationship | Complex$\uparrow$ |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Color $\uparrow$ | Shape$\uparrow$ | Texture$\uparrow$ | Spatial$\uparrow$ | Non-Spatial$\uparrow$ |  |  |
| Stable Diffusion 1.4 (Rombach et al., ) | 0.3765 | 0.3576 | 0.4156 | 0.1246 | 0.3079 | 0.3080 |
| Stable Diffusion 2 (Rombach et al., ) | 0.5065 | 0.4221 | 0.4922 | 0.1342 | 0.3096 | 0.3386 |
| Attn-Exct v2 (Chefer et al., ) | 0.6400 | 0.4517 | 0.5963 | 0.1455 | 0.3109 | 0.3401 |
| Stable Diffusion XL (Betker et al., ) | 0.6369 | 0.5408 | 0.5637 | 0.2032 | 0.3110 | 0.4091 |
| PixArt-$\alpha$ (Chen et al., ) | 0.6886 | 0.5582 | 0.7044 | 0.2082 | 0.3179 | 0.4117 |
| ECLIPSE (Patel et al., ) | 0.6119 | 0.5429 | 0.6165 | 0.1903 | 0.3139 | - |
| Dimba-G (Fei et al., ) | 0.6921 | 0.5707 | 0.6821 | 0.2105 | 0.3298 | 0.4312 |
| GenTron (Chen et al., ) | 0.7674 | 0.5700 | 0.7150 | 0.2098 | 0.3202 | 0.4167 |
| GLIGEN (Li et al., ) | 0.4288 | 0.3998 | 0.3904 | 0.2632 | 0.3036 | 0.3420 |
| LMD+ (Lian et al., ) | 0.4814 | 0.4865 | 0.5699 | 0.2537 | 0.2828 | 0.3323 |
| InstanceDiffusion (Wang et al., ) | 0.5433 | 0.4472 | 0.5293 | 0.2791 | 0.2947 | 0.3602 |
| IterComp (Ours) | 0.7982 | 0.6217 | 0.7683 | 0.3196 | 0.3371 | 0.4873 |

##### Qualitative Comparison

As shown in fig. 4, IterComp achieves superior compositional generation results compared to the three main types of compositional generation methods: text-controlled, LLM-controlled, and layout-controlled approaches. In comparison to text-controlled methods FLUX (BlackForest, ), IterComp excels in handling spatial relationships, significantly reducing errors such as object omissions and inaccuracies in numeracy and positioning. When compared to LLM-controlled methods like RPG (Yang et al., ), IterComp produces more reasonable object placements, avoiding the unrealistic positioning caused by LLM hallucinations. Compared to layout-controlled methods like InstanceDiffusion (Wang et al., ), IterComp demonstrates a clear advantage in both semantic aesthetics and compositionality, particularly when generating under complex prompts.

##### Quantitative Comparison

We compare IterComp with previous outstanding compositional text/layout-to-image models on the T2I-CompBench (Huang et al., ) in six key compositional scenarios. As shown in table 2, IterComp demonstrates a remarkable preference across all evaluation tasks. Layout-controlled methods such as LMD+ (Lian et al., ) and InstanceDiffusion (Wang et al., ) excel in generating accurate spatial relationships, while text-to-image models like SDXL (Betker et al., ) and GenTron (Chen et al., ) exhibit particular strengths in attribute binding and non-spatial relationships. In contrast, IterComp achieves comprehensive improvement in compositional generation. It obtains the strengths of various models by collecting composition-aware model preferences, and employs a novel iterative feedback learning to enable self-refinement of both the base diffusion model and reward models in a closed-loop manner.

IterComp achieves a high level of compositionality while simultaneously enhancing the realism and aesthetics of the generated images. As shown in table 4, we evaluate the improvement in image realism by calculating the CLIP Score, Aesthetic Score, and ImageReward. IterComp significantly outperforms previous models across all three scenarios, demonstrating remarkable fidelity and precision in alignment with the complex text prompt. These promising results highlight the versatility of IterComp in both compositionality and fidelity. We provide more quantitative comparison results between IterComp and other diffusion alignment methods in section A.3.

IterComp requires less time to generate high-quality images. In table 4, we compare the inference time of IterComp with other outstanding models, such as FLUX (BlackForest, ), RPG (Yang et al., ) in generating a single image. Using the same text prompts and fixing the denoising steps to 40, IterComp demonstrates faster generation, because it avoids the complex attention computations in RPG and Omost. Our method can incorporate composition-aware knowledge from different models without adding any computational overhead. This efficiency highlights its potential for various applications and offers a new perspective on handling complex generation tasks.

Figure: Figure 5: Results of user study.
Refer to caption: x5.png

##### User Study

We conducted a comprehensive user study to evaluate the effectiveness of IterComp in compositional generation. As illustrated in fig. 5, we randomly selected 16 prompts for each comparison, and invited 23 users from diverse backgrounds to vote on image compositionality, resulting in a total of 1,840 votes. The results show that IterComp received widespread user approval in compositional generation.

**Table 3: Evaluation on image realism.**
| Model | CLIP Score$\uparrow$ | Aesthetic Score$\uparrow$ | ImageReward$\uparrow$ |
| --- | --- | --- | --- |
| Stable Diffusion 1.4 (Rombach et al., ) | 0.307 | 5.326 | -0.065 |
| Stable Diffusion 2.1 (Rombach et al., ) | 0.321 | 5.458 | 0.216 |
| Stable Diffusion XL (Betker et al., ) | 0.322 | 5.531 | 0.780 |
| GLIGEN (Li et al., ) | 0.301 | 4.892 | -0.077 |
| LMD+ (Lian et al., ) | 0.298 | 4.964 | -0.072 |
| InstanceDiffusion (Wang et al., ) | 0.302 | 5.042 | -0.035 |
| IterComp (Ours) | 0.337 | 5.936 | 1.437 |

### 4.3 Ablation Study

Figure: (a) Impact on CLIP Score.
Refer to caption: x6.png

##### Effect of Model Gallery Size

In the ablation study on model gallery size, as shown in fig. 6, we observe that increasing the size of the model gallery leads to improved performance for IterComp across various evaluation tasks. To leverage this finding and provide more fine-grained reward guidance, we progressively expand the model gallery over multiple iterations by incorporating the optimized base diffusion model and new models such as Omost (Omost-Team, ).

##### Effect of composition-aware iterative feedback learning

We conducted an ablation study (see fig. 7) to evaluate the impact of composition-aware iterative feedback learning. The results show that this approach significantly improves both the accuracy of compositional generation and the aesthetic quality of the generated images. As the number of iterations increases, the model’s preferences gradually converge. Based on this observation, we set the number of iterations to 3 in IterComp.

Figure: Figure 7: Ablation study on the iterations of feedback learning.
Refer to caption: x9.png

Figure: Figure 8: The generation performance of integrating IterComp into RPG and Omost.
Refer to caption: x10.png

### 4.4 Generalization Study

IterComp can serve as a powerful backbone for various compositional generation tasks, leveraging its strengths in spatial awareness, complex prompt comprehension, and faster inference. As shown in fig. 8, we integrate IterComp into Omost (Omost-Team, ) and RPG (Yang et al., ). The results demonstrate that equipped with the more powerful IterComp backbone, both Omost and RPG achieve excellent compositional generation performance, highlighting IterComp’s strong generalization ability and potential for broader applications.

## 5 Conclusion

In this paper, we propose a novel framework, IterComp, to address the challenges of complex and compositional text-to-image generation. IterComp aggregates composition-aware model preferences from a model gallery and employs an iterative feedback learning approach to progressively refine both the reward models and the base diffusion models over multiple iterations. For future work, we plan to further enhance this framework by incorporating more complex modalities as input conditions and extending it to more practical applications.

## Acknowledgement

The author team would like to deliver sincere thanks to Ruihang Chu from Tsinghua University for his significant suggestions for the refinement of this paper.

## Appendix A Appendix

This supplementary material is structured into several sections that provide additional details and analysis related to IterComp. Specifically, it will cover the following topics:

- •
In section A.1, we provide a preliminary about Stable Diffusion (SD) and Reward Feedback Learning (ReFL).
- •
In section A.2, we provide detailed theoretical proof of the effectiveness of iterative feedback learning.
- •
In section A.3, we present the quantitative comparison results between IterComp and other diffusion alignment methods.
- •
In section A.4, we provide more visualization results for IterComp and its base diffusion model, SDXL.

### A.1 Preliminary

##### Stable Diffusion

Stable Diffusion (SD) (Rombach et al., ) performs multi-step denoising on random noise $\bm{z}_{T}\sim\mathcal{N}(\mathbf{0},\mathbf{I})$ to generate a clear latent $\bm{z}_{0}$ in the latent space under the guidance of text prompt $\bm{c}$. During the training, an input image $\bm{x}_{0}$ is processed by a pretrained autoencoder to obtain its latent representation $\bm{z}_{0}$. A random noise $\epsilon\sim\mathcal{N}(\mathbf{0},\mathbf{I})$ is injected into $\bm{z}_{0}$ in the forward process as follow:

$$ $\bm{z}_{t}=\sqrt{\bar{\alpha}_{t}}\bm{z}_{0}+\sqrt{1-\bar{\alpha}_{t}}\epsilon$ (6) $$

where $\alpha_{t}$ is the noise schedule. The UNet $\epsilon_{\theta}$ is trained to predict the added noise with the optimization objective:

$$ $\min_{\theta}\ \mathcal{L}(\theta)=\mathbb{E}_{[\bm{z}_{0}\sim\mathcal{E}(\bm{ x}_{0}),\epsilon\sim\mathcal{N}(\mathbf{0},\mathbf{I}),t]}\left[\left\| \epsilon-\epsilon_{\theta}(\bm{z}_{t},t,\tau(\bm{c}))\right\|_{2}^{2}\right]$ (7) $$

where $\mathcal{E}(\cdot)$ denote the preteained encoder of VAE, $\tau(\cdot)$ denotes the pretrained text encoder.

##### Reward Feedback Learning

Reward Feedback Learning (ReFL) (Xu et al., ) is proposed to align diffusion models with human preferences. The reward model serves as the preference guidance during the finetuning of the diffusion model. ReFL begins with an input prompt $\bm{c}$ and a random noise $\bm{z}_{T}\sim\mathcal{N}(\mathbf{0},\mathbf{I})$. The noise $\bm{z}_{T}$ is progressively denoised until it reaches a randomly selected timestep $t$. The latent $\bm{z}_{0}$ is directly predicted from $\bm{z}_{t}$, and the decoder from a pretrained VAE is used to generate the predicted image $\bm{x}_{0}$. The pretrained reward model $\mathcal{R}(\cdot)$ provides a reward score as feedback, which is used to finetune the diffusion model as follows:

$$ $\min_{\theta}\ \mathcal{L}(\theta)=-\mathbb{E}_{\bm{c}\sim\mathcal{C}}\left( \mathcal{R}\left(\bm{c},\bm{x}_{0}\right)\right)$ (8) $$

where the prompt $\bm{c}$ is randomly selected from the prompt dataset $\mathcal{C}$.

### A.2 Theoretical Proof of the Effectiveness of Iterative Feedback Learning

#### A.2.1 Proof of Lemma 1

###### Proof of Lemma 1 .

Considering the general form of RLHF, we change the optimization problem of iterative feedback learning to a bilevel optimization (Wallace et al., ; Ding et al., ):

$$ $\displaystyle\min_{\mathcal{R}}\ \ \ -\mathbb{E}_{\left[\bm{c}\sim\mathcal{C}, (\bm{x}_{0}^{w},\bm{x}_{0}^{l})\sim p_{\mathcal{R}}^{*}(\cdot\mid\bm{c})\right ]}\left[\log\sigma\left(\mathcal{R}\left(\bm{c},\bm{x}_{0}^{w}\right)-\mathcal {R}\left(\bm{c},\bm{x}_{0}^{l}\right)\right)\right]$ (9) $\displaystyle\text{ s.t. }p_{\mathcal{R}}^{*}:=\arg\max_{p}\mathbb{E}_{\bm{c} \sim\mathcal{C}}\left[\mathbb{E}_{\bm{x}_{0}\sim p(\cdot\mid\bm{c})}\mathcal{R }(\bm{c},\bm{x}_{0})\right]-\beta\mathbb{D}_{\mathrm{KL}}[p\left(\bm{x}_{0:T} \mid\bm{c}\right)||p_{\mathrm{ref}}\left(\bm{x}_{0:T}\mid\bm{c}\right)]$ $$

where $p_{\mathcal{R}}^{*}$ denotes the optimized base models under the guidance of reward model $\mathcal{R}$. We have the reparameterization of the reward model (also shown in previous works by (Wallace et al., )):

$$ $\mathcal{R}(\bm{c},\bm{x}_{0})=\beta\mathbb{E}_{p_{\mathcal{R}}\left(\bm{x}_{1 :T}\mid\bm{x}_{0},\bm{c}\right)}\left[\log\frac{p_{\mathcal{R}}^{*}\left(\bm{x }_{0:T}\mid\bm{c}\right)}{p_{\mathrm{ref}}\left(\bm{x}_{0:T}\mid\bm{c}\right)} \right]+\beta\log Z(\bm{c})$ (10) $$

$$ $Z(\bm{c})=\sum_{\bm{x}}p_{\mathrm{ref}}\left(\bm{x}_{0:T}\mid\bm{c}\right)\exp \left(\mathcal{R}(\bm{c},\bm{x}_{0})/\beta\right)$ (11) $$

Substituting this reward reparameterization into eq. 9, we get the new optimization objective as:

$$ $\min_{p_{\mathcal{R}}^{*}}\ -\mathbb{E}_{\left[\bm{c}\sim\mathcal{C},(\bm{x}_{ 0}^{w},\bm{x}_{0}^{l})\sim p_{\mathcal{R}}^{*}(\cdot\mid\bm{c})\right]}\left[ \log\sigma\left(\beta\log\frac{p_{\mathcal{R}}^{*}\left(\bm{x}_{0:T}^{w}\mid \bm{c}\right)}{p_{\mathrm{ref}}\left(\bm{x}_{0:T}^{w}\mid\bm{c}\right)}-\beta \log\frac{p_{\mathcal{R}}^{*}\left(\bm{x}_{0:T}^{l}\mid\bm{c}\right)}{p_{ \mathrm{ref}}\left(\bm{x}_{0:T}^{l}\mid\bm{c}\right)}\right)\right]$ (12) $$

This new optimization objective is denoted as $J(p_{\mathcal{R}}^{*})$, we get:

$$ $\max_{p_{\mathcal{R}}^{*}}\ J(p_{\mathcal{R}}^{*})\!=\!\mathbb{E}_{\left[\bm{c }\sim\mathcal{C},(\bm{x}_{0}^{w},\bm{x}_{0}^{l})\sim p_{\mathcal{R}}^{*}(\cdot \mid\bm{c})\right]}\!\left[\log\sigma\!\left(\!\beta\log\frac{p_{\mathcal{R}}^ {*}\left(\bm{x}_{0:T}^{w}\mid\bm{c}\right)}{p_{\mathrm{ref}}\left(\bm{x}_{0:T} ^{w}\mid\bm{c}\right)}\!-\!\beta\log\frac{p_{\mathcal{R}}^{*}\left(\bm{x}_{0:T }^{l}\mid\bm{c}\right)}{p_{\mathrm{ref}}\left(\bm{x}_{0:T}^{l}\mid\bm{c}\right )}\!\right)\right]$ (13) $$

We use $p_{\theta}$ to parameterize the policy and formulate the final optimization objective as:

$$ $\max_{\theta}\ J(\theta)\!=\!\mathbb{E}_{\left[\bm{c}\sim\mathcal{C},(\bm{x}_{ 0}^{w},\bm{x}_{0}^{l})\sim p_{\theta}^{*}(\cdot\mid\bm{c})\right]}\left[\log \sigma\left(\!\beta\log\frac{p_{\theta}^{*}\left(\bm{x}_{0:T}^{w}\mid\bm{c} \right)}{p_{\mathrm{ref}}\left(\bm{x}_{0:T}^{w}\mid\bm{c}\right)}-\beta\log \frac{p_{\theta}^{*}\left(\bm{x}_{0:T}^{l}\mid\bm{c}\right)}{p_{\mathrm{ref}} \left(\bm{x}_{0:T}^{l}\mid\bm{c}\right)}\!\right)\right]$ (14) $$

∎

###### Proof of Lemma 1 .

#### A.2.2 Proof of Theorem 1

###### Proof of Theorem 1 .

The gradient of the optimization objective in eq. 14 can be written as:

$$ $\nabla_{\theta}J(\theta)\!=\!\nabla_{\theta}\!\!\!\sum_{\bm{c},\bm{x}_{0}^{w}, \bm{x}_{0}^{l}}\!\!p_{\theta}\!\left(\bm{x}_{0:T}^{w}\!\mid\!\bm{c}\right)p_{ \theta}(\bm{x}_{0:T}^{l}\!\mid\!\bm{c})\!\left[\log\sigma\!\left(\!\beta\log \frac{p_{\theta}^{*}\left(\bm{x}_{0:T}^{w}\!\mid\!\bm{c}\right)}{p_{\mathrm{ ref}}\left(\bm{x}_{0:T}^{w}\!\mid\!\bm{c}\right)}-\beta\log\frac{p_{\theta}^{* }\left(\bm{x}_{0:T}^{l}\!\mid\!\bm{c}\right)}{p_{\mathrm{ref}}\left(\bm{x}_{0: T}^{l}\!\mid\!\bm{c}\right)}\!\right)\right]$ (15) $$

Assume that:

$$ $F_{\theta}(\bm{c},\bm{x}_{0}^{w},\bm{x}_{0}^{l})=\log\sigma\left(\!\beta\log \frac{p_{\theta}^{*}\left(\bm{x}_{0:T}^{w}\mid\bm{c}\right)}{p_{\mathrm{ref}} \left(\bm{x}_{0:T}^{w}\mid\bm{c}\right)}-\beta\log\frac{p_{\theta}^{*}\left( \bm{x}_{0:T}^{l}\mid\bm{c}\right)}{p_{\mathrm{ref}}\left(\bm{x}_{0:T}^{l}\mid \bm{c}\right)}\!\right)$ (16) $$

$$ $\hat{p}_{\theta}\left(\bm{x}_{0:T}^{w},\bm{x}_{0:T}^{l}\mid\bm{c}\right)=p_{ \theta}\left(\bm{x}_{0:T}^{w}\mid\bm{c}\right)p_{\theta}(\bm{x}_{0:T}^{l}\mid \bm{c})$ (17) $$

The gradient can be decomposed into two terms:

$$ $\displaystyle\nabla_{\theta}J(\theta)$ $\displaystyle\!=\!\nabla_{\theta}\!\!\sum_{\bm{c},\bm{x}_{0}^{w},\bm{x}_{0}^{l }}\hat{p}_{\theta}\left(\bm{x}_{0:T}^{w},\bm{x}_{0:T}^{l}\mid\bm{c}\right)F_{ \theta}(\bm{c},\bm{x}_{0}^{w},\bm{x}_{0}^{l})$ (18) $\displaystyle\!=\!\!\!\!\underbrace{\sum_{\bm{c},\bm{x}_{0}^{w},\bm{x}_{0}^{l} }\!\!\nabla_{\theta}\hat{p}_{\theta}\!\left(\bm{x}_{0:T}^{w},\!\bm{x}_{0:T}^{l }\!\mid\!\bm{c}\right)\!F_{\theta}(\bm{c},\bm{x}_{0}^{w},\bm{x}_{0}^{l})}_{T_{ 1}}\!+\!\underbrace{\mathbb{E}_{\left[\bm{c}\sim\mathcal{C},(\bm{x}_{0}^{w}, \bm{x}_{0}^{l})\sim p_{\theta}^{*}(\cdot\mid\bm{c})\right]}[\nabla_{\theta}[F_ {\theta}(\bm{c},\bm{x}_{0}^{w},\bm{x}_{0}^{l})]]}_{T_{2}}$ $$

By expanding the distribution $\hat{p}_{\theta}$ in $T_{1}$, a more specific form is obtained:

$$ $\displaystyle T_{1}$ $\displaystyle=\sum_{\bm{c},\bm{x}_{0}^{w},\bm{x}_{0}^{l}}\nabla_{\theta}\hat{p }_{\theta}\left(\bm{x}_{0:T}^{w},\!\bm{x}_{0:T}^{l}\mid\bm{c}\right)F_{\theta} (\bm{c},\bm{x}_{0}^{w},\bm{x}_{0}^{l})$ (19) $\displaystyle=\mathbb{E}\left[\left(\nabla_{\theta}\log p_{\theta}\left(\bm{x} _{0:T}^{w}\mid\bm{c}\right)+\nabla_{\theta}\log p_{\theta}\left(\bm{x}_{0:T}^{ l}\mid\bm{c}\right)\right)F_{\theta}\left(\bm{c},\bm{x}_{0}^{w},\bm{x}_{0}^{l} \right)\right]$ $$

∎

###### Proof of Theorem 1 .

### A.3 Quantitative Comparison with Other Diffusion Alignment Methods.

We compare IterComp with state-of-the-art diffusion alignment methods, Diffusion-DPO (Wallace et al., ) and ImageReward (Xu et al., ) in terms of image compositionality and realism. We calculate the average results of these models on T2I-CompBench (Huang et al., ), and evaluate image realism via CLIP Score and Aesthetic Score. As demonstrated in table 5, IterComp significantly outperforms previous diffusion alignment methods across all three scenarios. IterComp aggregates composition-aware model preferences from multiple models, which are used to train reward models. Guided by these composition-aware reward models, it achieves comprehensive improvements in compositional generation. Its superior performance in image realism is attributed to the effectiveness of iterative feedback learning, where the self-refinement of both the base diffusion model and reward models across multiple iterations drives significant gains in both compositionality and realism.

**Table 5: Comparison between IterComp and other diffusion alignment methods.**
| Model | Average Result on T2I-CB$\uparrow$ | CLIP Score$\uparrow$ | Aesthetic Score$\uparrow$ |
| --- | --- | --- | --- |
| Stable Diffusion XL (Betker et al., ) | 0.4441 | 0.322 | 5.531 |
| Diffusion-DPO (Wallace et al., ) | 0.4417 | 0.326 | 5.572 |
| ImageReward (Xu et al., ) | 0.4639 | 0.323 | 5.613 |
| IterComp (Ours) | 0.5554 | 0.337 | 5.936 |

### A.4 More Visualization Results

Figure: Figure 9: More visualization results for IterComp and its base diffusion model, SDXL.
Refer to caption: x11.png

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: Please see the weakness.
soundness: 3
strengths: 1. This work claims to be the first work to introduce a reward-controlled framework in the concept composition generation, which is somehow novel in this field.
2. This work is presented well with complete theoretical details and proof.
3. The quantitative experimental results show better performance compared to SOTAs.
summary: This paper proposes IterComp, an iterative composition-aware reward-controlled framework. It introduces a model gallery and constructs a high-quality composition-aware model preference dataset. Utilizing a new iterative feedback learning framework, IterComp progressively enhances both the reward models and the base diffusion model.
weaknesses: 1. The qualitative comparison in Fig. 4 is confused. There seems to be a marginal improvement in Line 3 and Line 4. The authors should make the difference between the qualitative examples clear to be recognized.
2. The qualitative examples are not enough to evaluate the model performance since the cases in the paper are somehow complex. The authors can provide more examples for a single case. Also, there is a need to evaluate the stability of the proposed model.
3. The comparison with InstanceDiffusion is confusing. As a layout-guided method, InstanceDiffusion needs detailed layout inputs. It is not fair to provide only one box if the case includes two or more instances, as indicated in the third line of Fig. 4. As the authors attempt to compare with layout-to-image methods, a SOTA method named MIGC [1] is also not included.

[1] Zhou, Dewei, et al. "Migc: Multi-instance generation controller for text-to-image synthesis." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: See weakness
soundness: 2
strengths: Novel and practical approach: The paper presents a simple yet effective way to combine multiple models' strengths for compositional generation without increasing computational complexity.

Strong empirical results: The method shows clear improvements over existing approaches, with comprehensive evaluations on both compositional accuracy and image quality metrics.

Well-structured technical contribution: The paper provides clear theoretical analysis with detailed proofs, and the iterative feedback learning framework is well-designed and easy to implement.
summary: This paper introduces IterComp, a novel framework that enhances compositional text-to-image generation by aggregating preferences from multiple diffusion models through iterative feedback learning. The approach demonstrates superior performance in both compositional accuracy and image quality, while maintaining efficient inference speed. The main strength lies in its ability to combine different models' advantages without adding computational overhead, though the long-term stability of the iterative learning process could be further explored.
weaknesses: The paper mentioned that RPG is challenging to achieve precise generation, but Tab2 did not compare with RPG, and I checked that RPG's performance on T2I-Compbench is better than that of the paper.

It is necessary to test the results of FLUX-dev directly on the t2i compbench to see how much improvement the method proposed in this paper has. I currently suspect that the improvement may not be very significant.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. As mentioned in W.1, how long does the training loop take (including the iterative feedback learning)?
2. Could I use this method to improve a specific concept generation (e.g., a human-object interaction)? How much time does it take from collecting synthetic data to finalizing the model training?
soundness: 3
strengths: 1. This paper presents a novel framework combining preferences from multiple diffusion models to enhance compositional text-to-image generation and address the relationship understanding in diffusion models.
2. The qualitative/quantitative results show comparable improvements in compositionality.
3. A composition-aware dataset is collected which provide diverse preferences that inform the reward models. (will it be released in the future?)
summary: This paper propose a framework that aggregates composition-aware model preferences from multiple models and employs an iterative feedback learning approach to enhance T2I compositionality and general performance. The qualitative and quantitative results show their SOTA compositional generation capabilities compared to previous works.
weaknesses: 1. There is limited discussion of the computation resources required to manage multiple reward models, which may affect the scalability in large-scale applications. Although the authors claim that their model has fast inference speed, the cost of model training and data collection is not clear. This makes me feel less likely than DPO to be widely used in practice.
2. The user study only demonstrates the user preferences lacking the deep analysis of attribute binding and object relationship, which are critical to model performance. 16 samples is also too small to evaluate such a complex task.

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 3
questions: In addition to the weakness mentioned above, I have a question regarding stability. 

Based on my experience, RLHF in diffusion models can often be unstable. I’m curious whether your method consistently produces stable results or if there’s a risk of occasionally poorer outcomes. I’m concerned that the iterative training process might lead to overfitting on biases present in the reward models, potentially reducing overall robustness.

I hope the authors can make up for the weaknesses mentioned and address these questions.
soundness: 3
strengths: 1. The paper is well-structured, making it accessible and easy for readers to follow.
2. Clear formulas are provided for each component, effectively removing any ambiguity.
3. Mathematical proofs substantiate the validity of the proposed method.
4. The authors conduct detailed, extensive experiments to support their approach.
5. Illustrative images are included, enhancing clarity and understanding.
summary: This study addresses improving the compositional text-to-image generation capability of less powerful diffusion models. The authors contribute in two main areas. First, they decompose the capabilities of various diffusion models into three domains—attribute binding, spatial relationships, and non-spatial relationships—and rank model outputs accordingly to guide reinforcement learning with reward models specific to each domain. Second, they introduce an iterative training process that utilizes the fine-tuned diffusion model outputs to progressively enhance the reward models.

Through multiple experiments, the study demonstrates the proposed method’s effectiveness, enabling early-stage models to achieve comparable generative abilities with reduced inference time. The authors also verify the effectiveness and general applicability of each design component across different models.
weaknesses: 1. About the number of texts used for attribute binding, "500 prompts" in Line 185 is inconsistent with "1500" in Table 1. Which is correct?
2. Although the experiments are detailed, some comparisons appear incomplete. The reinforcement learning from human feedback (RLHF) approach leverages outputs from advanced models like FLUX and SD3 for training, yet direct comparisons with these models are not provided. Including these comparisons would better highlight the method's effectiveness.
3. An additional experiment focusing on the first-round fine-tuned model—trained solely with human-annotated data—would be valuable. This would clarify the necessity and impact of the iterative training approach.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: * Training time considerations:
   - Diffusion-DPO operates in the latent space without requiring image decoding. In contrast, IterComp requires image decoding for the reward models (though this could be avoided by training the reward model with latent space inputs), likely resulting in slower training. Additional commentary on the training scheme would be valuable.
* Potential code and paper discrepancy:
   - The paper describes an iterative feedback mechanism that optimizes both reward models and the base model. However, examination of `feedback_train.py` reveals that only unet parameters (base model) are passed to the optimizer. This suggests that only the base model is being optimized, while reward models remain static. This difference requires clarification.
* Question regarding test-time adaptation:
   - Could the iterative feedback mechanism be applied as test-time adaptation of the base model? Similar to Slot-TTA [1], the base model could be optimized using reward models to improve compositional quality. The process would work as follows: for a given prompt, the base model generates an image, which is then evaluated by reward models. The base model's parameters would be updated to maximize these rewards. This process could be repeated for several iterations. This approach would eliminate the need for training the base model, allowing it to adapt to any prompt at test time through multiple iterations. Comments on this possibility would be valuable.

[1] Test-time Adaptation with Slot-Centric Models, Prabhudesai et al., ICML 2023

EDIT: After the rebuttal and discussion with the authors, I raise my score to 8 from 6.
soundness: 3
strengths: * The collected dataset can lead to better models and new research directions. Furthermore, researchers can follow a similar approach to collect their own data. The data can also be used for other RLHF methods, such as diffusion-DPO.
* The iterative feedback mechanism seems to be a novel way to optimize both reward models and the base model.
* The code is shared, which allows reviewers to follow the details of the proposed method.
* The performance gain from IterComp appears significant, as evaluated through user studies, quantitative analysis, and qualitative assessment.
summary: The paper presents a new dataset for compositional generation where experts evaluate the outputs of multiple pre-trained models. The dataset consists of three aspects of compositionality: attribute binding, spatial relationships, and non-spatial relationships, along with 52.5K image-rank pairs, which can be used for training feedback, ranking, or reward models.

The second contribution of the paper is using the collected dataset to train reward models for each of the three key aspects. A multimodal model (BLIP) is used as a feature extractor for both the prompts and the generated images, and the extracted features are projected by MLPs to output the reward. The goal is to predict how good the given image-prompt pairs are by training the model similar to contrastive learning (moving toward winning examples and away from losing examples).

The third contribution is improving the compositional ability of a base model (SDXL is selected but it should not be limited to that) by optimizing it using the trained reward models. The base model is trained to maximize the reward model's output so that its outputs are better aligned with the reward models, which are trying to enforce compositionality. Furthermore, the paper proposes an iterative update mechanism for both reward models and the base model. Reward models are updated to predict the rankings generated by experts while the base model is updated to maximize the outputs of reward models. Through this process, both the base model and reward models are improved for their specific tasks.

**Important note:** This review's technical content and analysis are my original work. Large Language Models were used solely to improve grammar and writing style, without altering any meanings or substantive feedback.
weaknesses: * Table 5 should be in the main part instead of the Appendix, as it simply demonstrates that the proposed method outperforms previous methods.
* Several experiments are missing:
   - The paper combines all reward models simultaneously, likely leading to improved compositional performance. However, reviewers would benefit from seeing the individual effect of each reward model.
   - While SDXL is chosen as the base model, testing other models would help reviewers understand how reward models affect different base architectures.
   - It would be valuable to examine Diffusion-DPO's performance when trained on the collected dataset. Currently, Diffusion-DPO is trained on the pick-a-pic dataset, which is larger but lacks compositional details. These results would be necessary to evaluate the proposed method's effectiveness using consistent standards.
* Expert ranking may inadvertently include aesthetic information [This observation is meant to prompt discussion rather than highlight a weakness, as the underlying cause remains unclear]:
   - When IterComp is applied, the aesthetic score improves, suggesting that reward models can interpret image aesthetics, since reward maximization leads to better aesthetic scores. This could be because either expert rankings are influenced by image aesthetics (beyond compositional attributes) or because models with better composition naturally generate more aesthetic images (for instance, FLUX, being the best compositional model, likely produces more aesthetically pleasing outputs).
* Some experimental conditions may be misleading:
   - The paper uses 40 inference steps for all models to ensure fairness. However, some models can generate samples with fewer steps; for example, FLUX-dev uses 28 steps by default.

Meta-review: All reviewers agree to accept the paper. Reviewers appreciate the novel framework and significant performance improvement. Please be sure to address the reviewers' comments in the final version.

Decision: Accept (Poster)