Progressive Compositionality in Text-to-Image Generative Models
Authors: Xu Han
Linghao Jin
Xiaofeng Liu
Paul Pu Liang
Venue: ICLR 2025 Spotlight (2025)
Keywords: compositional text-to-image generation
contrastive learning
compositional understanding
T2I generation
Primary area: generative models
TL;DR: Contrastively improving compositional text-to-image generation in diffusion models via generating fine-grained hard negative images.

Abstract: Despite the impressive text-to-image (T2I) synthesis capabilities of diffusion models, they often struggle to understand compositional relationships between objects and attributes, especially in complex settings. Existing approaches through building compositional architectures or generating difficult negative captions often assume a fixed prespecified compositional structure, which limits generalization to new distributions. In this paper, we argue that curriculum training is crucial to equipping generative models with a fundamental understanding of compositionality. To achieve this, we leverage large-language models (LLMs) to automatically compose complex scenarios and harness Visual-Question Answering (VQA) checkers to automatically curate a contrastive dataset, ConPair, consisting of 15k pairs of high-quality contrastive images. These pairs feature minimal visual discrepancies and cover a wide range of attribute categories, especially complex and natural scenarios. To learn effectively from these error cases (i.e., hard negative images), we propose EvoGen, a new multi-stage curriculum for contrastive learning of diffusion models. Through extensive experiments across a wide range of compositional scenarios, we showcase the effectiveness of our proposed framework on compositional T2I benchmarks.

Published as a conference paper at ICLR 2025
PROGRESSIVE COMPOSITIONALITY IN SOKT-TO-IMAN GENERATIVE MODELS
Evans Xu Han' Linghao Jin Xiaofeng Liu' Paul Pu Liang*
Yale University, University of Southern California, MMassacuseetts Institute of Technology {xu. fxu.han xh365, xiaofeng. Liu]@yale edu
edu ppliangemit.em edu
ABSTRACT
Despite the impressive text-to-image (T2I) synthesis capabilities of diffusion mod- els, they often struggle to understand compositional relationships between objects and attributes, especially in complex settings. Existing approaches through build- ing compositional architectures or generating difficult negative captions often as- sume fixed prespecified compositional structure, which limits generalization new distributions. In this paper, we argue that curriculum training is crucial to equipping generative models with a fundamental understanding of composition- ality. To achieve this, we leverage lange-anguuae models (LLMs) to automati- cally compose complex scenarios and harness Visual- Visual-Quester Question Answering (VQA) checkers to automatically curate contrastive dataset, CONPATR, consisting of 15k pairs of high-quality contrastive images. These pairs feature minimal visual discrepancies and cover wide range of attribute categories, especially complex and natural scenarios. To learn effectively from these error cases (i.e., hard nega- tive images), propose EVOGEN, a new multi-stage curriculum for contrastive learning of diffusion models. Through extensive experiments across a wide range of compositional scenarios, showcase the effectiivness of our proposed frame- work on compositional T21 benchmarks. The project page with data, code, and de- be found at https: /ew.naaeee.....eee g thub. o PvvoGen_Pagee
INTRODUCTION
The rapid advancement of text-to-image generative models (Saharia et al., 2022; Ramesh et al., 2022) has revolutionized the field of image synthesis, driving sigmificant progress in various ap- plications such image editing (Brooks et al., 2023; Zhang et al., 2024; Gu et al., 2024), video generation (Brooks et al., 2024), and medical imaging (Han et al., 2024a). Despite their remarkable capabilities, state-of-the-art models such Stable Diffusion (Rombach et al., 2022) and DALL- 3 (Betker et al., 2023) still face challenges with composing multiple objects into a coherent scene (Huang al., 2023; Liang al., 2024b; Majumdar al., 2024). Common issues include incorrect attribute binding, miscounting, and flawed object relationships as shown in Figure 1. For example, when given the prompt red motorcycle and a yellow door", the model might incorrectly bind the colors to the objects, resulting in a yellow motorcycle.
Recent progress focuses on optimizing the attention mechanism within diffusion models to better capture the semantic information conveyed by input text prompts (Agarwal et al., 2023; Chefer al., 2023; Pandey al., 2023). For example, Meral et al. (2023) proposes manipulating the attention on objects and attributes as contrastive samples during the test time to the optimize model performance. While more focused, the practical application of these methods still falls short of fully addressing attribute binding and object relationships. Other works develop compositional generative models to improve compositional performance, as each constituent model captures the distributions of an independent domain (Du & Kaelbling, 2024). However, these approaches assume fixed prespecified structure to compose models, limiting generalization to new distributions.
In this paper, we argue that curriculum training is crucial equip diffusion models with fun- damental understanding of compositionality. Given that existing models often struggle with even basic tasks (e.g., generating three when prompted with "Two caty are playing") ") (Wang al.,

---

Published as a conference paper at ICLR 2025
A woman standing front of three peintings on the sol.. (Counshig)
white cut walks towards brown ring. (Attoo Spaci!))
A yellow boel. blue mug and pink plate on the table. (Ooj, Counting, Colod)
Two cats are playingg un der trem in anowy city (Boane, Accion, Counting)
A dog is feeding a girl. (1 ngiis Action)
X SDKL
303
X SDXL
x DALLE
30XL
Figure I: Limited Compositionality Understanding Diffusion Models. Existing SOTA models such as SDXL, DALL-E 3 often fail to correctly compose objects and attributes. The bottom are images generated by our EVOGEN.
2024), we progressively introduce more complex compositional scenarios during fine-tuning. This staged training strategy helps models build solid foundation before tackling intricate cases for best performance.
With the increasing demand for large-scale data both model pre-training and fine-tuning, high- quality data generation plays crucial role in this process (Peng et al., 2025; Ye et al., 2024; Peng al., 2024a). Although many datasets exist for compositional generation (Wang et al., 2023; Feng et al., 2023a), there remains significant gap in datasets that offer a clear progression from simple to complex samples within natural and reasonable contexts. Moreover, creating high-quality con- trastive image datasets is both costly and labor-intensive, especially given the current limitations of generative models in handling compositional tasks. To address this, we propose an automatic pipeline to generate faithful contrastive image pairs, which we find crucial for guiding models to focus on compositional discrepancies. In summary, our work can be summarized as follows:
Contrastive compositional dataset. We introduce CONPAIR, meticulously crafted composi- tional dataset consisting of high-quality contrastive images with minimal visual representation dif- ferences, covering wide range of attribute categories. By leveraging LLMs, we scale up the com- plexity of compositional prompts while maintaining a natural context design. Our dataset features faithful images generated by diffusion models, assisted by VQA checker to ensure accurate align- ment with the text prompts.
EVOGEN: Curriculum contrastive learning. We also incorporate curriculum contrastive learn- ing into diffusion model improve compositional understanding. This curriculum is designed with three sub-tasks: (1) learning single object-attribute composition, (2) mastering attribute bind- ing between objects, and (3) handling complex scenes with multiple objects. We conduct ex- tensive experiments using the latest benchmarks and demonstrate that EvOGEN significantly boosts the model's compositional understanding. outperforming most baseline generative methods.
PRELIMINARY BACKGROUND
2.1 DIFFUSION MODELS
We implement our method on top the state-of-the-art text-to-image (T2I) model, Stable Diffu- sion (SD) (Rombach et al., 2022). In this framework, encoder E maps given image E X into a spatial latent code z = E(x), while a decoder reconstructs the original image, ensuring D(E(x)) x
2

---

Published as a conference paper at ICLR 2025
Contra. Imuge
Samples
Categories
Complex
DRAWBENCH (Saharia CC-500 (Feng 20136)
(oolor, spasial action) (oslor)
ATTN-AND-EAN (Chefer et at. 2023)
210
2 (oolor, animal obj.
121 Boc-CorrB (Hanng et 2023)
6000
5 (oolor, counting. texture, shape, tace-ispatil, comples) (seme, aitriutt: relarion. couminng. comparison, differentiation, legic)
GEN- AT (Li et al,, 2024a)
1600
X
V
ABC 6K (Peng al. 2025a)
6000
X
I (color)
X
WINOGROUNTT21 (7ho 2023)
22k
20 (action,, spatial, Gircttinn, eolor, number size, testare shape, weight, sentimens. procedure, speei,
COMP. SPLITS (Purk 2021)
31k
3 (color, shupe)
(Tbrush 2012)
4001
jobject, celation, symbolic, catribute,
prageaatics)
FOREN (Wang 2033)
2501
object. coum)) (relations, aurthusss)
ARO (Viksekgomll 2023)
(ours)
countng. shape, testure, (non Jsputial relations, scene, complex)
Table I: The comparison of compositional T2I datasets. Contra. is the abbreviation of Contrastive. Complex refers the samples that have multiple objects and complicated attributes and relationships.
pre-trained denoising diffusion probabilistic model (DDPM) (Sohl-Dickstein et al., 2015; Ho et al., 2020) for noise estimation and pre-trained CLIP text encoder (Radford al., 2021) to process text prompts into conditioning vectors c(y). The DDPM model e(0) is trained to minimize the difference between the added noise and the model's estimate at each timestep
t, L = E==- [[|= +
During inference, latent is sampled from (0, 1) and is iteratively denoised to produce a latent 2o. The denoised latent 2g is then passed to the decoder to obtain the image = D(zo).
2.2 COMPOSITIONAL DATASETS AND BENCHMARKS
The most commonly used data sets for object-attribute binding, including DRAWBENCII (Saharia et al., 2022), CC-500 (Feng et al., 2023a) and ATTEND-AND-EXCITE (Chefer et al., 2023) con- struct text prompts by conjunctions of objects and a few of common attributes like color and shape. To more carefully examine how generative models work on each compositional category, recent work explores the disentanglement of different aspects of text-to-image compositionality. Huang et al. (2023) introduces T21-COMPBENCH that constructing prompts by LLMs which covers six categories including color, shape, textual, (non-Jspation relationships and complex compositions; Recently, GEN-AI AI (Li et al., 2024a) collects prompts from professional designers which captures more enhanced reasoning aspects such as differentiation, logic and comparison.
Another line of work proposes contrastive textual benchmarks to evaluate the compositional capa- bility of generative models. ABC-6K (Feng et al., 2023a) contains contrast pairs by either swapping the order of objects attributes while they focus negative text prompts with minimal changes. WINOGROUND (Zhu et al., 2023) contains 11K complex, high-quality contrastive sentence pairs spanning 20 categories. However, such benchmarks focus text perturbations but do not have im- ages, which have become realistic with the ad vancement of generative models.
Several benchmarks featuring contrastive image pairs have also been introduced. COMPOSITIONAL SPLITS C-CUB AND C-FLOWERS (Park et al., 2021) mainly focused on the color and shape at- tributes of birds and flowers, sourcing from Caltech-UCSS) Birds (Wah al., 2011), Oxford-102 (Flowers) (Nilsback & Zisserman, 2008). Thrush et al. (2022) curated WINOGROUND consists of 400 high-quality contrastive text-image examples. EQBEN (Wang et al., 2023) an early effort to use Stable Diffusion to synthesize images to evaluate the equivariance of VLMMs similarity, but it lacks more complex scenarios. Yuksekgonul et (2023) emphasizes the importance of hard nega- tive samples and constructs negative text prompts in ARO by swapping different linguistic elements in the captions sourced from COCO and sampling negative images by the nearest- neiiest algo- rithm. However, it is not guaranteed that the negative images found in the datasets truly match the semantic meaning of the prompts.
3 DATA CONSTRUCTION: CONPAIR
To address attribute binding and compositional generation, we propose a new high-quality con- trastive dataset, CONPATR. Next, introduce our design principle for constructing CONPAIR.
3

---

Published as a conference paper at ICLR 2025
Category
Stage-1
Stage-11
An american football. A volleyball. (0)
american football and volleyball.
Shape
A badminton ball and Frisbee
Ablue backpack.
A blue backpack and yellow purse. A yellow purse and blue backpack.
Color
A red backpack
Three birds.
Two cats and one dog.
Counting
Two birds.
Two dogs and one cat.
A plastic toy.
rubber tire and glass mirror.
Texture
A fluffy toy.
A ruhber mirror and glass tire
plate the right of bee.
Spatial
bee the right of place.
busketball player eating dinner. basketball player is dancing. A snowy night. rainy night.
A woman passing tennis to man.
Non-spatial
A man is passing a lennis ball to woman.
a serene lake during a thunderstorm.
Scene
In screne lake on a sunny day.
Two fluffy dogs are cating apples to the right of a brown cat.
Complex
A brown dog are cating pears to the left of two fluffy cats.
Stage-IIII
Two green birds standing
orange birds willow tree.
An orange bird standing to three green birds the grass.
wcaring blue hat is throwing an american foothall from the left the right wearing green pants the playground during snowy day. A woman wearing green hat is throwing tennis ball from the right the left to woman wearing blue hat on the playground during rainy night.
Complex
Table 2: Examples of text prompts. Each sample has positive (top) and negative prompt (hottom).
Each sample in CONPAIR consists of pair of images (a associated with a positive caption et.
3.1 GENERATING TEXT PROMPTS
Our text prompts cover eight categories of compositionality: color, shape, texture, counting, spa- tial relationship, non-spatial relationship, scene, and complex. To obtain prompts, utilize the in-context learning capability of LLMs. We provide hand-crafted seed prompts as examples and predefined templates (e.g., "A {color) {object} and {color) fobjecr).") and then ask GPT-4 to generate similar textual prompts. We include additional instructions that specify the prompt length, repetition, etc. In total, we generate 15400 positive text prompts. More information the text prompt generation is provided in the appendix A.
To generate a negative text prompt t use GPT-4 to perturb the specified attributes or rela- tionships of the objects for Stage-I data. In Stage-II, we either swap the objects or the attributes, depending on which option makes more sense the given context. For complex sentences, we prompt GPT-4 to construct contrastive samples by altering the attributes or relationships within the sentences. Table presents our example contrastive text prompts.
3.2 GENERATING CONTRASTIVE IMAGES
Minimal Visual Differences. Our key idea is to generate contrastive images that are mini- mally different in visual representations. By "minimal," we mean that, aside from the altered at- tribute/relation, other elements in the images remain consistent similar. In practice, we source negative image samples in two ways: 1) generate negative images by prompting negative prompts diffusion models; 2) edit the positive image by providing instructions (e.g., change motorcycle color to red) using MagicBrush (Zhang et al., 2024), as shown at the left of Figure 2.
Text-Image Alignment. The high-level objective of CONPAIR is to generate positive images that faithfully adhere to the positive text guidance, while the corresponding negative images do not align with the positive text, despite having minimal visual differences from the positive images. As the quality of images generated by diffusion-based T2I generative models varies significantly (Karthik
4

---

Published as a conference paper at ICLR 2025
Curriculum Contrastive Learning
Image Genoration (103)
Obj. Aftr. Rel.
Select
Nog Promptt
Instruction
(Magicfrush)
Image Editing
Text- Image Similority
Stagesi Stage-ll
Minimal Visual
Stage -Il
400
GPT4
LLOVA
DIfF.
Multi-Shage Fine-uuning
Yoa
Beat
Better
S. Stable 3iffusion
LLaVA
Two
Aligmmett
bowl
Yea
LLaVA
Revised
2. What ooive ln boad?
Red
Frompt
3. How mary bowls are there?
One
mag?
Figure 2: EvOGEN Framework. Data generation pipeline (left) and curriculum contrastive learning (right). Quality control of image generation (bottom): Given prompt. SD3 generates multiple candidate images, which are evaluated by LLaVA. We select the best image by alignment and CLIPScore. If the alignment score is low, we prompt LLaVV describe the image a newly revised caption based on the generated image.
al., 2023), first generate 10-20 candidate images per prompt. However, selecting the most faithful image is difficult. Existing automatic metrics like CLIPScore are not always effective at comparing the faithfulness of images when they visually similar. To address this, propose decomposing each text prompt into a set of questions using an LLM and leverage the capabilities of VQA models rank candidate images by their alignment score, as illustrated in Figure 2 (bottom) ! Note that the correct answers can be directly extracted from the prompts. Intuitively, we consider an image a success if all the answers are correct or if the alignment is greater than Balign for certain categories, such Complex. After getting aligned images, select the best image by automatic metric (e.g., CLIPScore).
Empirically, we find this procedure fails to generate faithful images particularly when the prompts become complex, as limited by the compositionality understanding existing generative models, which aligns with the observations of Sun et al. (2023). In response such cases-i.e., the align- ment scores for all candidate images are low-we introduce an innovative reverse- eeverse-alignmen strategy. Instead of simply discarding low-alignment images, leverage VLM to dynamically revise the text prompts based on the content of the generated images. By doing so, we generate new captions that correct the previous inaccuracies while preserving the original descriptions, thereby improving the alignment between the text and image.
Image-Image Similarity. Given each positive sample, we generate 20 negative images and select the one with the highest similarity to the corresponding positive image, ensuring that the changes between the positive and negative image pairs are minimal. In the case of color and texture, we use image editing rather than generation, as it delivers better performance for these attributes. Han et al. (2024b) proposes that human feedback plays vital role in enhancing model performance. For quality assurance, 3 annotators randomly manually reviewed the pairs in the dataset and filtered 647 pairs that were obviously invalid.
EVOGEN: CURRICULUM CONTRASTIVE FINE-TUNING
common challenge in training models with data of mixed difficulty is that it can overwhelm the model and lead suboptimal learning (Bengio et al., 2009). Therefore, we divide the dataset into
Examples of decomposed questions provided in the Appendix A.3
5

---

Published as a conference paper at ICLR 2025
Attribute Binding
Color
Counting, Missing Object Two cats, one dog. and one rabbit are on the grass.
Color
A red motorcycle in front of yellow door
Two blood moons hang in the night sky. and a flock of bats flies medieval-style castle
Object Relationships
Action
Spatial
Action
white caf chasing little girl a yellow floral dress on grass
black dog is the left of pig
man yellow T-shirt crying
Complex
In the British Museum, dinosaur fossil fighting with four caveman specimens circular platform
Three differently colored apples (yellow, green, red from left to right) with transparent water bottle placed behind the middle apple.
A fully armored knight wearing blue cape and small golden dragon perched on their shoulder, is staring red evil dragon.
Contrastive dataset examples. Each pair includes a positive image generated from the given prompt (left) and negative image that is semantically inconsistent with the prompt (right), differing only minimally from the positive image.
Figure 3:
three stages and introduce simple but effective multi-stage fine-tuning paradigm, allowing the model to gradually progress from simpler compositional tasks to more complex ones.
Stage-I: Single object. In the first stage, the samples consist of a single object with either specific attribute (e.g., shape, color, quantity, or texture), a specific action, or a simple static scene. The differences between the corresponding negative and positive images are designed to be clear and noticeable. For instance, "A man is walking" vs. "A man eating", where the actions differ significantly, allowing the model to easily learn distinguish between them.
Object compositions. We compose two objects with specified interactions and spatial relationships. An example of non-spatial relationship is "A woman chases dog" vs. "A yellow dog chases a woman." " This setup helps the models learn to differentiate the relationships between two objects.
Stage-II:
Stage-III: Complex compositions. To further complicate the scenarios, we propose prompts with complex compositions of attributes, objects, and scenes. Data in this stage can be: 1) contain more than two objects; 2) assign more than two attributes to each object, or 3) involve intricate relation- ships between objects.
Ultimately, our goal to equip the model with the capability to inherently tackle challenges in compositional generation. Next, we discuss how to design the contrastive loss during fine-tuning at each stage. Given a positive text prompt t, a generated positive image at and corresponding negative image æ the framework comprises the following three major components:

---

Published as a conference paper at ICLR 2025
Attribute Binding
Object Relationship
Complex
Model
Color
Shape
Texture
Spatial
Non-Spatial
STABLE 11.4 (Rombach et al., 2022)
37.65
35.76
41.56
12.46
30.79
30.80
STABLE V2 (Rombach al., 2022)
50.65
42.21
49.22
13.42
30.96
33,86
DALL-E 2 (Ramesh al., 2022)
57.00
55.00
63.74
13.00
30.00
37.00
SDXL (Podell et al., 2023)
54.00
36.45
20.00
31.00
41.00
COMPOSABLE (Liu et al., 2023)
40.63
32.99
36.45
8.00
29.80
28.98
STRUCTURED v2 (Feng et al., 2023a)
42.18
49.00
31.11
33.55
ATTN-EXCT Chefer
(2023)
64.00
45.17
59.63
14.55
31.09
34.01
GORs (Huang al., 2023)
66.03
47.85
62.87
18.15
31,93
33.28
PIXART-a (Chen al., 2023)
68.86
55.82
70.44
20.82
31.79
41.17
MARS (He al., 2024)
69.13
54.31
71.23
19.24
32.10
40.49
EvoGEN (Ours)
71.04..13
54.57
72.34
21.76
33.08
42.52
Table 3:
Alignment evaluation on T21-CompBench. We report average and standard deviations three runs. The best results are in bold.
Diffusion Model. The autoencoder converts the positive image and negative image latent space as 20 and zn The noisy latent at timestep f is represented as z and 2 The encoder of the noise estimator is used to extract feature maps =et and Zet respectively.
Projection head. We apply small neural network projection head g(-) that maps image repre- sentations to the space where contrastive loss is applied. We use a MLP with one hidden layer to obtain h1 = g((e)) = w(2) o (W())
Contrastive loss. For the contrastive objective, we utilize a variant of the InfoNCE loss (van den Oord et al., 2019), which is widely used in contrastive learning frameworks. This loss function is designed to maximize the similarity between the positive image and its corresponding text prompt while minimizing the similarity between the negative image and the same text prompt. The loss for a positive-negative image pair is expressed as follows:
(xp(si((tt..())/7)) f(t)/T)
C - log
exp(sim(hT ((1))/T) exp(sim (())//>)
where r is temperature parameter, f(:) CLIP text encoder, sim function represents cosine simi- larity:
4
sim(u, v)
(3)
1 |0
This encourages the model to distinguish between positive and negative image-text pairs.
EXPERIMENTS AND DISCUSSIONS
5.1 IMPLEMENTATION DETAILS
Experimental Setup In an attempt to evaluate the faithfulness of generated images, GPT-4 to decompose a text prompt into a pair of questions and answers, which serve as the input of our VQA model, LLaVA vI.5 (Liu et al., 2024a). Following previous work (Huang al., 2023; Feng et al., 2023a), we evaluate EVOGEN on Stable Diffusion v2 (Rombach et al., 2022).
Baselines We compare our results with several state-of-the-an methods, including trending open- sourced T2I models that trained on large training data, Stable Diffusion v1.4 and Stable Diffusion v2 (Rombach et al., 2022), DALL-E 2 (Ramesh et al., 2022) and SDXL. (Podell et al., 2023). Com- posableDiffusion v2 (Liu et al., 2023) is designed for conjunction and negation of concepts for pretrained diffusion models. StructureDiffusion v2 (Feng et al., 2023a), Divide-Bind (Li et al., 2024b) and Attn-Exct v2 (Chefer et al., 2023) are designed for attribute binding for pretrained dif- fusion models. GORs (Huang et al., 2023) finetunes Stable Diffusion v2 with selected samples and rewards. PixArt-a (Chen et al., 2023) incorporates cross-attention modules into the Diffusion Trans- former. MARS (He et al., 2024) adapts from auto-regressive pre-trained trained LLMs for T2I generation tasks.
7

---

Published as a conference paper at ICLR 2025
Animal Object
Objest-
0.36
0.38
0.38
0.27
0.285
0.285
0.18
0.19
0.19
0.09
0.095
0.095
0
Full Prompt
Minimum Object
Full Prompt
Minimum Object
Full Promot
Minimum Object
Stable Diffusion v1.4 Attn & Exct
Composalbee
Structured
COMFORM
EvoGen-SD v2.1
Figure 4: Average CLIP image-text similarities between the text prompts and the images generated by different models. The Full Prompt Similarity considers full-text prompt. Minimum Object rep- resents the minimum of the similarities between the generated image and each of the two object prompts. An example of this benchmark is in subsection C.3.
Evaluation Metries To quantitatively assess the efficacy of our approach, we comprehensively evaluate our method via two primary metrics: compositionality on 121-CompBench (Huang al., 2023) and 2) color-object compositionality prompts (Chefer et al., 2023).
5.2 PERFORMANCE COMPARISON AND ANALYSIS
Alignment Assessment. To examine the quality of CONPAIR, we measure the align- ment of the positive image and texts using CLIP similarity. Figure 5 compares directly select- ing the best image based on CLIPScore with our pipeline, which leverages a VQA model to guide image generation. These results confirm that our approach consistently improves image faithfulness across all categories with VQA as- sistance during image generation and demon- strate CONPAIR contains high-quality image- text pairs.
CLIPScore
VQA-GUPScore
0.343
0.505
Color
Shape
Counting
Spotial
Figure 5: Average CLIP similarity of image-text pairs in CONPAIR. Applying VQA checker con- sistently improves text-image alignment.
Benchmark Results Beyond the above evaluation, we also assess the alignment between the gen- erated images using EvOGEN and text condition on T2I-Compbench. As depicted in Table evaluate several crucial aspects, including attribute binding, object relationships, and complex com- positions. EVOGEN exhibits outstanding performance across 5/6 evaluation metrics. The remark- able improvement in Complex performance is primarily attributed to Stage-Ill training, where high- quality contrastive samples with complicated compositional components are leveraged achieve superior alignment capabilities.
Figure 4 presents the average image-text similarity on the benchmark proposed by Chefer et al. (2023), which evaluates the composition of objects, animals, and color attributes. Compared to other diffusion-based models, our method consistently outperforms in both full and minimum similarities across three categories, except for the minimum similarity Object-Objer prompts. These results demonstrate the effectiveness our approach.
We conduct ablation studies on T21-CompBench by exploring three key design choices. First, we assess the effectiveness of our constructed dataset, CONPAIR, by fine-tuning Sta- ble Diffusion v2 directly using CONPAIR. As shown in Table 4, our results consistently outperform the baseline evaluation Stable Diffusion v2 across all categories, demonstrating that our data generation pipeline is effective. Next, we validate the impact our contrastive loss by comparing
Ablation Study
Mooe details about specific metries used in T21-CompBench are in the Appendix.
8

---

Published as a conference paper at ICLR 2025
Attribute Binding
Object Relationship
Complex
Model
Color
Shape
Texture
Spatial
Non-Spatial
STABLE v2 (Rombach al., 2022)
50.65
42.21
49.22
13.42
30.96
33,86
CONPAIR
63.63
47.64
61.64
17.77
31.21
35.02
CONPATR Contra. Loss
69.45
54.39
67.72
20.21
32.09
38.14
CONPAIR + Contra. Multi-stage
71.04
54.57
72.34
21.76
33.08
42.52
Table 4: Ablation on T21-CompBench. CONPAIR refers to directly finetune SDv2 on CONPAIR.
SDXL A brown bear and white cat, both wearing spacesuits, are playing frisbee Mars.
DALL-E 3
SD3.0
PixArt-Alpha
Ours
Three spaceships are sailing through space, and the top spaceship, octagonal clock is hanging.
Two little bears swimming the pool on the terrace through the window of skyscraper on the rainy day.
left statue, men running talking-
Figure 6: Qualitative comparison between EVOGEN and other SOTA T2I models. EvoGEN shows consistent capabilities in following compositional instructions to generate images.
with fine-tuning without this loss. The contrastive loss improves performance the attribute binding category, though has less impact on object relationships and complex scenes. We hypothesize this because attribute discrepancies are casier for the model to detect, while relationship differences more complex. Finally, applying the multi-stage fine-tuning strategy leads to further improve- ments, particularly in the Complex category, suggesting that building foundational understanding of simpler cases better equips the model to handle more intricate scenarios.
Qualitative Evaluation Figure presents a side-by-side comparison between EVOGEN and other state-of-the-art T2I models, including SDXL, DALL-E 3, SD v3 and PixArt-a. EvoGEN con- sistently outperforms the other models in generating accurate images based on the given prompts. SDXL frequently generates incorrect actions and binds attributes to the wrong objects. DALL-E fails to correctly count objects in two examples and misses attributes in the first case. SD v3 struggles with counting and attribute binding but performs well in generating actions. PixArt-a unable to handle attributes and spatial relationships and fails to count objects accurately in the second prompt.

---

Published as a conference paper at ICLR 2025
On the stest a waiing tlue boseball sap standing 01 skrebeers
stwal, was ing a senyy day, a slontocadd
away
hystant
thos
Two
Three
Thee with callio the fgrt of white cats
Under the pross, with the teo wiite cas walding the cal.
Easier, less compositionally
More complex, more compositionally
Figure 7: Examples of EVOGEN for complex compositionality.
Ours
Stable Diffusion v3
SUXL
DALL-E2
PixArt-Alpha
70
35
Aesthetic
Alignment
Aesthetio
Alignment
Aesthetic
Alignment
Aesthetic
Alignment
Figure 8:
User study on 100 randomly selected prompts from Feng al. (2023a). The ratio values indicate the percentages of participants preferring the corresponding model.
Next, we evaluate how our approach handles complex compositionality, as shown in Figure 7. Using the same object, "bear" and "cat," we gradually increase the complexity by introducing variations in attributes, counting, scene settings, interactions between objects, and spatial relationships. The generated results indicate that our model effectively mitigates the attribute binding issues present in existing models, demonstrating significant improvement in maintaining accurate compositional relationships.
User Study We conducted user study to complement our evaluation and provide a more intuitive assessment of EVOGEN'S performance. Due to the time-intensive nature of user studies involving human evaluators, we selected pop-performin comparable PixArt-a- all accessible through APIs and capable of generating images. As shown in Figure 8, the results demonstrate EvOGEN's superior performance in alignment, though the aesthetic quality may be slightly lower compared to other models.
Mideels-DALLE-2 SD v3, SDXL, and
CONCLUSION
In this work, we present EVOGEN, a curriculum contrastive framework to overcome the limitations of diffusion models in compositional text-to-image generation, such as incorrect attribute binding and object relationships. By leveraging a curated dataset of positive-negail multi-stage fine-tuning process, EVOGEN progressively improves model compositionality, particu- larly complex scenarios. Our experiments demonstrate the effectiveness of this method, paving the way for more robust and accurate generative models.
image pairs and
10

---

Published as a conference paper at ICLR 2025
7 LIMITATION
Despite the effectiveness of current approach, there are few limitations that can be addressed in future work. First, our dataset, while comprehensive, could be further expanded to cover an even broader range of compositional scenarios and object-atrribute relationships. This would enhance the model's generalization capabilities. Additionally, although we employ a VQA-guided image generation process, there is still room for improvement in ensuring the faithfulness of the generated images to their corresponding prompts, particularly more complex settings. Refining this process and incorporating more advanced techniques could further boost the alignment between the text and image.
REPRODUCIBILITY
We have made efforts to ensure that our method reproducible. Appendix A provides description of how we construct our dataset. Especially, Appndix A.1 and A.2 presents how we prompt GPT- and use predefined template to generate text prompts of our dataset. Appendix A.3 provides an example how utilize VQA system to decompose prompt into Appendix B provides the details of implementation, to make sure the fine-tuning is reproducible.
of questions, and answers.
REFERENCES
Aishwarya Agarwal, Srikrishna Karanam, K. Joseph, Apoorv Saxena, Koustava Goswami, and Balaji Vasan Srinivasan. A-star: Test-time attention segregation and retention for text-to-image synthesis. 2023 JEEE/CVF International Conference on Computer Vision (ICCV), pp. 2283-2293, 2023. URL wwwpthtpt//cc.....m / semant cscholar .rg wwr.oo/////ooooo Corp/Corpussst ID: 259252450.
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. ICML '09, pp. 41-48, New York, NY, USA, 2009. Association for Computing Machinery. ISBN 11..111..111//1
9781605585161. doi: 11144//153334.1555388 URL https: 1553374 1553380.
James Betker, Gabriel Goh, Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang. Joyce Lee, Yufei Guo, et al. Improving image generation with better captions. Computer Science. https://cdn. openai. com/papers/dallle-., pdf, 2(3):8, 2023.
Tim Brooks, Aleksander Holynski, and Alexei Efros. Instructpix2pix: Learning follow image editing instructions, 2023. URL https: : / arxiv. www.ii..g//i...eo....oooooooooor abs 09800.
Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh. Video generation models as world simulators. 2024. URL /tttps://opeaa..... com/ rete/rche www.rrddsiulator.....
video-geeeerationnmooeee.
Hila Chefer, Yuval Alaluf, Yael Vinker, Lior Wolf, and Daniel Cohen-Or. Attend-and-excite: Attention-based semantic guidance for text-to-image diffusion models, 2023. URL https: 13825.
//arxiv.org/abee..... 1382..
Junsong Chen, Jincheng Yu, Chongjian Ge, Lewei Yao, Enze Xie, Yue Wu, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, and Zhenguo Li. Pixart-o: Fast training of diffusion transformer for photorealistic text-to-image synthesis, 2023. URL https: arxiv. www./as//////eoooooo..oo/.eeeeee 00426.
Andrew Drozdov, Nathanael Schärli, Ekin Akyürek, Nathan Scales, Xinying Song, Xinyun Chen, Olivier Bousquet, and Denny Zhou. Compositional semantic parsing with large language models, 2022. URL https: /ttp//aaavvv////////////////aav/ http://wwwwwgg////////.... /
15003.
Yilun Du and Leslie Kaelbling. Compositional generative modeling: A single model not all you need, 2024. https:: / http://axivvvo.// abss///bb///////// 0ttp:/22222.00....... 01103.

---

Published as a conference paper at ICLR 2025
Weixi Feng, Xuehai He, Tsu-Jui Fu, Varun Jampani, Arjun Akula, Pradyumna Narayana, Sug- ato Basu, Xin Eric Wang, and William Yang Wang. Training-free structured diffusion guidance for compositional text-to-image synthesis, 2023a. URL MtLLttt://xxiiv..oo /ttgg//bs/2122 05032.
Weixi Feng, Wanrong Zhu, Tsu jui Fu, Varun Jampani, Arjun Akula, Xuehai He, Sugato Basu, Xin Eric Wang, and William Yang Wang. Layoutgpt: Compositional visual planning and genera- tion with large language models,
Xin Gu, Ming Li, Libo Zhang, Fan Chen, Longyin Wen, Tiejian Luo, and Sijie Zhu. Multi-reward as condition for instruction-based image editing. CoRR, abs/2411.04713, 2024, doi: 10.48550/ 2411. 04713.
ARXIV.2411.044 .07713. URL https::////. aooiioogg/000 ///./1...4 485508550//00 wtt.i.orrg1..44555000
Xu Han, Fangfang Fan, Jingzhao Rong, and Xiaofeng Liu. Fair text to medical image diffusion model with subgroup distribution aligned tuning, 2024a. URL attps://arxiv... org/abs/ 2406. 14847.
Xu Han, Felix Yu, Joao Sedoc, and Benjamin Van Durme. Baby bear: Seeking just right rating scale for scalar annotations, 2024b. URL https: / arwiv.iiivv....ii http//b//22300 09765.
Xu Han, Linghao Jin, Xuezhe Ma, and Xiaofeng Liu. Light-weight fine-tuning method for de- fending adversarial noise in pre-trained medical vision-language models, 2025. URL https: //arxiv.orgg abs/ /2407 02716.
Wanggui He. Siming Fu, Mushui Liu, Xierui Wang, Wenyi Xiao, Fangxun Shu, Yi Wang, Lei Zhang, Zhelun Yu, Haoyuan Li, Ziwei Huang, LeiLei Gan, and Hao Jiang. Mars: Mixture of auto-regressive models for fine teee-rraned text-to-image synthesis, 2024. URL https:://aaaat org/abs/ 2407. 07614.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840 6851, 2020.
Hexiang Hu, Ishan Misra, and Laurens van der Maaten. Evaluating text-to-image matching using binary image selection (bison). In 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW), pp. 1887-1890, 2019. doi: 10. 10911//C....2019..000
Kaiyi Huang, Kaiyue Sun, Enze Xie, Zhenguo Li, and Xihui Liu. T2i-compbench: A com- prehensive benchmark for open-world compositional text-to-image generation, 2023. URL. https: http://axvvvvo///e/ee/eee aa/////s///0 06350.
Shyamgopal Karthik, Karsten Roth, Massimiliano Mancini, and Zeynep Akata. If at first you don't succeed, try, try again: Faithful diffusion-based text-to-image generation by selection, 2023. URL https:/ //ts:///rxiii http///awwvrgg.//e/....... alos 2305 13308.
Dongwon Kim, Ju He, Qihang Yu, Chenglin Yang, Xiaohui Shen, Suha Kwak, and Liang-Chics Chen. Democratizing text-to-image masked generative models with compact text-aware one- dimensional tokens, 2025. URL https: //tp://aaavv.///aa//ea/eeaeeaaaa wwwpxxiv.rggab////// /2501 07730.
Baiqi Li, Zhiqiu Lin, Deepak Pathak, Jiayao Li, Yixin Fei, Kewen Wu, Tiffany Ling, Xide Xia, Pengchuan Zhang, Graham Neubig, and Deva Ramanan. Genai-bench: Evaluating and improving compositional text-to-visual generation, 2024a. URL aittps://arxiaa org/ abs/2466. 13743.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre- training with frozen image encoders and large language models, 2023, URL https: //attss://rxiv.. org/abs/ 2301. 12597.
Yumeng Li. Margret Keuper, Dan Zhang, and Anna Khoreva. Divide & bind your attention for improved generative semantic nursing, 2024b. URL attps://arxits org/ abs/2307. 10864.
Paul Pu Liang, Akshay Goindani, Talha Chafekar, Leena Mathur, Haofei Yu, Ruslan Salakhutdi- nov, and Louis-Philippe Morency. Hemm: Holistic evaluation of multimodal foundation models, 2024a. URL https::////// htappps:///xiii////// www.aiioorg/abaaaaaaaeaa. 2407. 03415 8.
12

---

Published as a conference paper at ICLR 2025
Paul Pu Liang, Amir Zadeh, and Louis-Philippe Morency. Foundations & trends in multimodal
machine learning: Principles, challenges, and open questions. ACM Computing Surveys, 56(10); 1-42, 2024b.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee, Improved baselines with visual instruction tuning, 2024a. URL https: /ttp/aaxxvvv..//../...... org/ /as///bb///// /2110 03744.
Nan Liu, Shuang Li, Yilun Du, Antonio Torralba, and Joshua B. Tenenbaum. Compositional vi- http///awi.....gabbb...beo
sual generation with composable diffusion models, 2023. URL 2206. 01714.
Qihao Liu, Zhanpeng Zeng, Ju He, Qihang Yu, Xiaohui Shen, and Liang- Chigg-Chhhhhh Chen. Alleviating distortion image generation via multi-resolution diffusion models and time-dependent layer normalization, 2024b. URL https: www.a.iii...oo. org/ als 22002440000000 1.4066000 0941 6.
Arjun Majumdar, Anurag Ajay. Xiaohan Zhang, Pranav Putta, Sriram Yenamandra, Mikael Henaff, Sneha Silwal, Paul Movay, Oleksandr Maksymets. Sergio Arnaud, Karmesh Yadav, Qiyang Li, Ben Newman, Mohit Sharma, Vincent-Pierre Berges, Shiqi Zhang, Pulkit Agrawal, Yonatan Bisk, Dhruv Batra, Mrinal Kalakrishnan, Franziska Meier, Chris Paxton, Alexander Sax, and Aravind Rajeswaran. Openeqa: Embodied question answering in the era of foundation models. 2024 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 16488-16498, 2024. URL https:: /apppppp.////// semant www.annicccchholloo org
Tuna Han Salih Meral, Enis Simsar, Federico Tombari, and Pinar Yanardag. Conform: Contrast is all you need for high-fidelity text-to-image diffusion models, 2023. URL attps://arxiss
org/ab /rg//bb/// 2212..1.. 06059.
Chancharik Mitra, Brandon Huang, Trevor Darrell, and Roei Herzig. Compositional chain-of- thought prompting for large multimodal models, 2024. URL 2311. 17076.
abs/
Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing, pp- 722-729, 2008. URL https: /api. semanticscholatt .tt.///orppssS::1111133333
Rohan Pandey, Rulin Shao, Paul Pu Liang, Ruslan Salakhutdinov, and Louis-Philippe Morency. mrosll attention congruence regularization for vision-language relation alignment, 2023. URL https:ttp////
//tp/vaa.vv//////////eeeeeawwa.a htttt//s///axxx org/ abs/ 2212 10549.
Dong Huk Park, Samaneh Azadi, Xihui Liu, Trevor Darrell, and Anna Rohrbach. Benchmark for compositional text-to-image synthesis. In NeurlPS Datasets and Benchmarks, 2021. URL
https: //api. semant wwa.nticcchhlaan
Eric Pasewark, Kyle Montgomery, Kefei Duan, Dawn Song, and Chenguang Wang. Re-tuning: Overcoming the compositionality limits of large language models with recursive tuning, 2024.
URL httph::////// /atp/v/axvvvv/vv/////////////wa/ org/ abs 20bb/24070 2407 04787.
Long Peng, Yang Cao, Renjing Pei, Wenbo Li, Jiaming Guo, Xueyang Fu, Yang Wang, and Zheng- Jun Zha. Efficient real-world image super-resolution via adaptive directional gradient convolution. arXiv preprint arXiv:2405.0702. 2024a.
Long Peng, Yang Cao, Yuejin Sun, and Yang Wang. Lightweight adaptive feature de-drifting for compressed image classification. IEEE Transactions on Multimedia, 2024b.
Long Peng, Wenbo Li, Renjing Pei, Jingjing Ren, Jiaqi Xu, Yang Wang, Yang Cao, and Zheng- Jun Zha. Towards realistic data generation for real-world super-resolution. The Thinteenth International Conference on Learning Representations, 2025. URL https: httth::://peeeeeiieee /oppneevie. n fert/fo det.rummmddooKC.CNNUNUCC
Wujian Peng, Sicheng Xie, Zuyao You, Shiyi Lan, and Zuxuan Wu. Synthesize, diagnose, and op- timize: Towards inne-grained vision- lision-language understanding, 2024c. URL https: : //rxii..
argg/abs/2312 00081.
13

---

Published as a conference paper at ICLR 2025
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhom, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. 2023. URL ptatttss://axiiiii org/ abs 230.. 01952.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning. pp. 8748-8763. PMLR, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents, 2022. URL https://arxin org/ abs/ 2204. 06125.
Arijit Ray, Filip Radenovic, Abhimanyu Dubey, Bryan A. Plummer, Ranjay Krishna, and Kate Saenko. Cola: A benchmark for compositional text-to-image retrieval, 2023. URL https:
//arxiv.org/aab. 2305. 03689.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pp. 10684-10695, 2022.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Informa- tion Processing Systems, 35:36479-3649/ 2022.
Jascha Sohl-Doe Sohl-Diccstten, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrimm thermodynamics. In International Conference on Machine Learn- ing. pp. 2256-2265. PMLR, 2015.
Jiao Sun, Deqing Fu, Yushi Hu, Su Wang, Royi Rassin, Da-Cheng Juan, Dana Alon, Charles Her- rmann, Sjoerd van Steenkiste, Ranjay Krishna, and Cyrus Rashtchian. Dreamsyne: Aligning text-to-image generation with image understanding feedback, 2023. URL https://arxiv. org/abs/22 /2311 17946.
Tristan Thrush, Ryan Jiang, Max Bartolo, Amanpreet Singh, Adina Williams, Douwe Kiela, and Candace Ross. Winoground: Probing vision and language models for visio-linguistic composi- tionality, 2022. URL https: h/tp://aaavvvv./// wwwpiv.org//aaaa ab// 2204. 03162.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding, 2019. URL https: /ttp:///aaaiviviii org/ http///b///111/77/11 03748.
C. Wah, S. Branson, P. Welinder, Perona, and S. Belongie. The caltech-uesd birds-200-2011 dataset. Technical Report ENS-TR-2011-001 California Institute of Technology, 2011.
Tan Wang, Kevin Lin, Linjie Li, Chung-Chine Lin, Zhengyuan Yang, Hanwang Zhang, Zicheng Liu, and Lijuan Wang. Equivariant similarity for vision-language foundation models, 2023. URL https: / arxiv. org/ abss 22033 14465.
Zhenyu Wang, Enze Xie, Aoxue Li, Zhongdao Wang, Xihui Liu, and Zhenguo Li. Divide and conquer: Language models can plan and self-correct for compositional text-to-image generation, 2024. URL https: http//xxvvvvro...... abs wta.////111..
Mark Weber, Lijun Yu, Qihang Yu, Xueqing Deng, Xiaohui Shen, Daniel Cremers, and Liang- Chich Chen. Maskbit: Embedding free image generation via bit tokens, 2024. URL https:
/ //arxivv.cao .rr.ivorrrr/oo abs/ 22409. 6621.
Zihan Ye, Shreyank N Gowda, Xiaobo Jin, Xiaowei Huang, Haotian Xu, Yaochu Jin, and Kaizhu Huang. Exploring data efficiency in zero-shot learning with diffusion models. arXiv preprint arXiv:2406.0222 02999, 2024.
Qihang Yu, Mark Weber, Xueqing Deng, Xiaohui Shen, Daniel Cremers, and Liang-Chiel Chen. An image is worth 32 tokens for reconstruction and generation, 2024. URL https: : /arxiv. org/ h/gg/ab//200 6..6.. 07550.
14

---

Published as a conference paper at ICLR 2025
Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou. When and why vision-language models behave like ofgs-of-words, and what to do about it?, 2023. URL https: /atass///axivva wwwp/otgg///ooooooottt../t...ttt /tnp//22100 01936.
Arman Zarei, Keivan Rezaei, Samyadeep Basu, Mehrdad Saberi, Mazda Moayeri, Priyatham Kat- takinda, and Soheil Feizi. Understanding and mitigating compositional issues in text-to-image generative models, 2024. URL https: http://axvvv..///
org org. /ad.b///46... 07844.
Kai Zhang, Lingbo Mo, Wenhu Chen, Huan Sun, and Yu Su. Magicbrush: A manually annotated dataset for instruction-guided image editing, 2024. URL htttps::/aaxii.rrgg////p 10012.
Xiangru Zhu, Penglei Sun, Chengyu Wang, Jingping Liu, Zhixu Li, Yanghua Xiao, and Jun Huang. A contrastive compositional benchmark for text-to-image synthesis: A study with unified text-to- image fidelity metrics, 2023, URL https: / aarxiiiiiv.i/.ii. org/ abs/2312 02338.
15

---

Published as a conference paper at ICLR 2025
A CONPAIR DATA CONSTRUCTION
A.1 TEXT PROMPTS GENERATION
Here, we design the template and rules generate text prompts by GPT-4 as follows:
Color: Current state-of- tfrtrr-rrtttt text-to-image models often confuse the colors of objects when there multiple objects. Color prompts in Stage-I follow fixed sentence template "A {color} {object)." " and "A {color} {object) and a {color} {object)." for Stage-II.
Following Huang et al. (2023), we emphasize in the GPT-4 instructions to require valid combinations of an object and textural attribute. The texture prompts follows the template "A {texture) (object)." for Stage-I and {texture) {object) and {texture) {object)." for Stage-II.
Texture:
Shape: We first generate objects with common geometric shapes using fixed template "A {shape} {object)." for Stage-I and "A [shape} {object) and a {shape} {object)." for Stage-II. Moreover, we ask GPT-4 to generate objects in the same category but with differ- ent shapes, e.g., American football vs. Volleyball, as contrastive samples.
Counting:
Counting prompts in Stage-I follows fixed sentence template "[count) {object)." and "[count) {object) and {count) {object)." for Stage-II.
Spatial Relationship: Given predefined spatial relationship such as next to, on the left, etc, we prompt GPT-4 to generate a sentence in fixed template as "(object) {spatial) {object)." for Stage-II.
Non-spatial Relationship: Non-spatial relationships usually describe the interactions be- tween two objects. We prompt GPT-4 to generate text prompts with non-spatial relation- ships (e.g., actions) and arbitrary nouns. We guarantee there is only one object in the sentence for Stage-I, and two objects in Stage-II. We also find generative models fails to understand texts like "A woman passing ball to a man". It's hard for the model to correctly generate the directions of actions. We specially design prompts like this.
Scene: We ask GPT-4 to generate scenes such as weather, place and background. For Stage-I, the scene simple, less than words (e.g., rainy night.); For Stage-II, scenes combine weather and background or location (e.g.. in a serene lake during a thunderstorm.).
Complex: Here, we refer prompts that either contain more than two objects or assign more than two attributes to each object, involve intricate relationships between objects. We first manually curate 10 such complex prompts, each involving multiple objects bound various attributes. These manually generated prompts generate additional natural prompts that emphasize compositionality. The complex cases in Stage-Il will be objects with more attributes; Stage-III involves more objects.
context for GPT-4 to
Note that when constructing our prompts, we consciously avoided using the same ones as those in T21-Compbench, especially considering some prompts from T21-CompBench are empirically difficult to generate aligned image (e.g.. "a pentagonal warning sign and pyramidal bookend" as shown in Figure 9), which are not well-suited for our dataset. We have filtered out similar prompts from our dataset using LLMs to identify uncommon combinations of objects and attributes.
Figure 9: Example image that is hard to generate to align the prompt from T21-CompBench.
16

---

Published as a conference paper at ICLR 2025
A.2 NEGATIVE TEXT PROMPTS GENERATION
We apply in-context learning and prompt GPT-4 to generate negative cases, we give 5-10 10 example test prompts each time, and make sure the generation not repetitive, within certain lengths.
In Stage-I, we prompt GPT-4 to change the attribute of the object such as color, shape, texture, counting, action, scene, with instruction the differences should be noticeable.
In Stage-II, we either swap the objects or attributes and let GPT-4 validate the swapped text prompts. For complex cases, we generate negative text by asking GPT-4 to change the atrribueeseelatooniiissssee.
Stage-III, we carefully curate complicated examples with 3-6 objects, each object has 1-3 attributes, with negative prompts change attributes, actions and spatial relationships, and scenes. We also prompt GPT-4 with such examples.
A.3 VQA ASSISTANCE
Instruction for QA Generation. Given an image description, generate one or two multiple-choice questions that verify if the image description is correct. Table 5 shows example of generated prompt and QA.
Prompt
Question
Answer
Is there a bear?
Yes
Is there cat?
Yes
What color is the bear? What color is the cat?
Brown
White
A brown bear and a white cat, both wearing spacesuits, are playing frishee on Mars
Does the bear wear spacesuit? Does the cat wear a spacesuit? Is the bear playing the frisbee? Is cat playing frishee? Where are they playing?
Yes
Yes
Yes
Yes
Mars
Table 5: VQA generated questions from a prompt.
Modifying Caption to Align Image. Next, we illustrate how we prompt VQA to revise the caption when alignment scores of all candidate images are low. Given generated image and original text prompt, we prompt the VQA model with the following instruction:
Instruction: "Given the original text prompt describing the image, identify any parts that inaccu- rately reflect the image. Then, generate revised text prompt with correct descriptions, making minimal semantic changes. Focusing on counting. color, shape, texture, scene, spatial relationship, and non-spatial relationship.". At the same time, we will provide examples of revised captions for in-context learning.
For example, given the following image (Figure 10) and the original text prompt, the modified prompt generated by the VQA model is follows:
Original text prompt: Three puppies are playing on the sandy field on sunny day, with two black ones walking toward a brown one.
Modified prompt: Four puppies standing on a sandy field on sunny day, with three black puppies and one brown puppy facing forward.
Figure 10: Image applies reverse- alignmen.
Note that the instruction "Focusing on the counting, color, shape, texture, scene, spatial
17

---

Published as a conference paper at ICLR 2025
relationship, non-spatial relationship" plays crucial role in guiding the VQA model to provide answers that accurately correspond to the specific attributes and categories we are interested in. Without this directive, the model may occasionally fail to generate precise captions that correctly describe the image.
A.4 DATA STATISTICS
The dataset is organized into three stages, each progressively increasing in complex- ity. In Stage-I, the dataset includes sim- pler tasks such as Shape (500 samples), Color (800), Counting (800), Texture (800), Non- spatial relationships (800), and Scene (800), totaling 4,500 samples. Stage-II introduces more complex compositions, with each cate- gory-including Shape, Color, Counting, Tex- ture, Spatial relationships, Non-spatial relation- ships, and ceene-containigg 1,000 samples, for a total of 7,500 samples. Stage-III repre- sents the most complex scenarios, with fewer but more intricate samples. We also include some simple cases like Stage-I and II, each contain 200 samples, while the Complex category includes 2,000 samples, totaling 3,400 samples, Across all stages, the dataset contains 15,400 samples, pro- viding wide range of compositional tasks for model training and evaluation. Figure 11 show more examples of images in our dataset.
Stage-I
Stage-I1
Stage-III
Total
Shape
500
1000
200
1700
Color
800
1000
200
2000
Counting
800
1000
200
2000
Texture
800
1000
200
2000
Spatial
1000
200
1200
Non-spatial
800
1000
200
2000
Scene
800
1000
200
2000
Complex
500
2000
2500
Table 6: Corpus Statistics.
A.5 COMPARISON WITH REAL CONTRASTIVE DATASET
To evaluate how our model would fare with real hard-negative dataset, we include the results of fine-tuning our model with COLA (Ray et al., 2023), BISON (Hu et al., 2019) evaluated by T2I- CompBench Table (randomly sampled consistent number samples across datasets).
Although COLA and BISON try to construct semantically hard-negative queries, the majority of the retrieved image pairs are quite differrntt practice, often introducing a lot of noisy ob- jects/background elements in the real images, due to the nature of retrieval from an existing dataset. We hypothesize this makes it hard for the model to focus on specific attributes/relationships in compositionality. In addition, they don't have complex prompts with multiple attributes and don't involve action, or scene.
In contrast, our dataset ensures the generated image pairs are contrastive with minimal visual changes, enforcing the model to learn subtle differences in the pair, focusing on a certain cate- gory. To the best of our knowledge, real contrastive image dataset only differs minimal visual characteristics.
Dataset
Color
Shape
Texture
Spatial
Non-Spatial
Complex
COLA
62.20
48.98
53.73
15.21
30.87
33.15
BISON
59.49
49.36
48.77
14.64
31.25
32.91
Ours
71.04
54.57
72.34
21.76
33.08
42.52
Table 7: Performance of fine-tuning EVOGEN T21-CompBench across different dataset.
A.6 QUALITY CONTROL.
Coverage of LLM-generated QA Pairs We conducted human evaluations on Amazon Mechan- ical Turk (AMT). We sampled 1500 prompt-image pairs (about 10% of the dataset, proportionally across stages) to perform the following user-study experiments. Each sample is annotated by human annotators.
18

---

Published as a conference paper at ICLR 2025
12
10
3
3
4
yellow dog running a woman
A square clock hanging on the wall
green upple and yellow pear
Four hobbits preparing hexagonal time tunnel in underground voleanic cave
golden vase and clear glass
Cupid is playing with pink rubbit on white clouds with cotton candy texture
cat is chasing woman
A pyramid in the desert under the sky
A man walking on the street
A garden with various flowers, five garden gnomes three wearing hats and wearing green hats birdbatt in the middle, and wooden bench the left, with two butterflies flying above
Three wolves in the foggy weather and a bright moon in the sky.
green car parked between two person wearing jacket standing to the reur left of the holding briefcase.
blue molorcycles,
Figure 11: Example contrastive Image pairs in CONPAIR
To analyze if the generated question cresstio-aanser answer pairs by GPT-4 cover all the elements in the prompt, we performed user study wherein for each qquestion-proppe pair, the human subject is asked answer if the question-set covers all the objects in the prompt. The interface is presented in Figure 13.
Empirically, we find about 96% of the questions generated by GPT-4 cover all the objects, 94% cover all the atrribueesreeaaiooshipss
Accuracy of Question-Answerinn of VQA Models To analyze the accuracy of the VQA model's answering results, performed additional user-study wherein for each question- -mmae pair, the human subject is asked to answer the same question. The accuracy of the VQA model is then predicted using the human labels as ground truths. Results displayed in Table 8.
Image Stage
VQA Accuracy % Annotation Time Image (s)
Stage-I
93.1%
8.7s
Stage-II
91.4%
15.3s
Stage-III 88.9%
22.6s
Table 8: VQA accuracy and annotation time for sampled images across different stages.
19

---

Published as a conference paper at ICLR 2025
COLA EXAMPLES
Peotfee
Negative
Negative
bullding blue dioudy
clear round
builing
Image Differences
Imprectie Logicaal Relationship
COM- -DFFF EXAMPLES
Poothe
Negithv
Poutine
Mogathen
A yellow bowl blue mug and a pink plate a gray table
Under the blue dky, two dogs are ractnn on the grass, right.
BISON EXAM PLES
0.12
Postfes
Negative
Posties
Negethe
Plates filed with carros and boets on white table, elements covered prompt
Yellow shirted tonnis player lcoking incoming ball.
Image Differences
COM DIFF EXAMPLES
A bag of potatoes and two watermelons are casually deoed the muddy floor the warehouse.
coddy bear baseball player getting ready catch fling posato.
Figure 12: Comparison with Real Contrastive Dataset: COLA and BISON.
We observe that the VQA model is effective at measuring image-text alignment for the majority of questions even as the complexity the text prompt increases, attesting the effectiveness of pipeline.
Alignment of Revised Caption with Images To further validate the effectiveness of revising cap- tions by VQA, we randomly sampled 500 images that are obtained by revising caption and per- formed an additional user-study for those samples that obtain low alignment score from VQA an- swering, but the reverse-alinmment strategy. Specifically, for each revised caption-image pair, the human subject is asked to answer how accurately the caption describes the image. The interface presented in Figure 14. Note we have 5 annotators, each is assigned 100 caption-image pairs.
Empirically, we found that 4% the samples show that the revised caption similarly describes the image the original caption. 94.6% of the samples show the revised caption better deseribes the image. Overall,win with the following settings, the average rating the alignment between revised caption and image is 4.66, attesting that revised caption accurately describes the image.
20

---

Published as a conference paper at ICLR 2025
Prompt
A beer and white cet, both wearing spacesults, are playing frisbee on Marks.
Question
Is there bear?
there
What color bear?
What color the cat?
Does the bear
spacesuit?
Does
spacesuit?
is the bear playing the frisbee?
the cat playing the frisbee? Where they playing?
Does the above question set covers all the objects and relationships/ attributes in the prompt?
the objects and relationshipsiattrieee are covered.
All the objects are covered, some relationships/atrribuee are missing.
Some objects missing, all the relaiionshipssatrribtess missing.
objects missing, some relationships/strribuess missing.
Figure 13: Interface for User Study: Coverage of LLM -ug-eeeertted QA Pairs
Caption 1:
Three pupples playing on the sandy field sunny day, with two ones walking
Caption Four puppies are standing on sandy field on a sunny day, with three black puppies and brown puppy facing forward.
Q1. Which of the above caption better describes the image?
Caption
Caption 2
Both similarly describes the image
How well does the caption2 describes the image?
caption perfectly
image.
4 Has few minor discrepancies.
Has discrepancles.
Has significant discrepancies.
Figure 14: Interface for User Study: Alignment of Revised Caption with Images
Similarity of Contrastive Image Pairs We have 3 annotators in total, each annotator is assigned 2550 images (about 50% samples) to check if the positive and negative image pairs align with its text prompt and are similar with small visual changes specific attributeshellationsiips We filtered 647 images from the randomly selected 7650 images, which is 8.45%, attesting the quality of most images in the dataset.
21

---

Published as a conference paper at ICLR 2025
Method
Attribute
Scene
Relation
Avg
Count
Differ
Compare
Logical
Avg
Spotial
Action
Part
Negale
Universal
SD v2.1
0.75
0..77
00.72
0.77
0.66
0.74
0.66
0.63
0.61
0.50
0.57
0.58
TURRO
0.79
01.82
0.778
0.65
0.64
0..51
0.60
DEaFFLOYD.IP
SD-XL
0.82
0.84
0.80
0.81
0.81
0.82
0.71
0.67
0.54
0.49
0.57
0.60
MIDIOURNEY v6
0.85
0.88
0.86
0.86
0.85
0.85
0.75
0.73
0.70
0:49
0.64
0.65
SD3 101-MEDIUM
0.86
0.85
0.87
0.86
0.88
0.86
0.74
0.77
0.72
0.50
0.73
0.68
DALL-E3
0,91
0.89
0.89
0.78
0.76
0.70
0.46
0.65
SD3-MEDUL (OuRs)
Table 9: Gen-AI Benchmark Results.
B TRAINING IMPLEMENTATION DETAILS
We implement our approach upon Stable Diffuion v2.1 and Stable Diffusion -3-meuuum. We We employ the pre-trained text encoder from the CLIP ViT-L/14 model. The VAE encoder is frozen during training. The resolution is 768, the batch size 16, and the learning rate 3e-5 with linear decay.
C QUANTITATIVE RESULTS
C.I T2I-COMPBENCH EVALUATION METRICS
Following 121-CompBench, we use DisentangledBLIP-VV VQA for color, shape, texture, UniDet for spatial, CLIP for non-spatial and 1-in-1 for complex categories.
C.2 GEN-AI BENCHMARK
We further evaluate EVOGEN the Gen-AI (Li al., 2024a) benchmark. For fair comparison with DALL-E 3, we finetune our model on Stable Diffusion v3 medium. As indicated Table 9, EVOGEN performs best all the Advanced prompts, although it exhibits relatively weaker perfor- mance in some of the basic categories compared to DALL-ES.
C.3 ATTN & EXCT BENCHMARK PROMPT EXAMPLES
The benchmark protocol we follow comprises structured prompts 'a [animalA] and [an- imalBJ', 'a [animal] and a [color][obeect], 'a [color]][objectA]]] and a [coler
Model
Animal-COl
Obi--hi
STABLE (Rambach
COMPOSABLE 2023)
0.77
2023c)
0.74
ArTI-ExCT (Chefer
Table 10 demonstrate the results of aver- age CLIP similarities between text prompts and captions generated by BLIP for Stable Diffusion-basee methods on this benchmark. EVOGEN outperforms those models in three categories.
Table 10: Attn-Exct benchmark Results.
D QUALITATIVE RESULTS
Figure 15 presents comparison hetween EvoGEN and other state-of-the-art T21 models, in-
cluding SDXL, DALL-E 3, SD v3 and PixArt-a.
E RELATED WORK
With the rapid development of multimodal learning (Li et al., 2023; Liang et al., 2024b;a; Han al., 2025) and image generation (Yu et al., 2024; Liu et al., 2024b; Weber et al., 2024; Peng et al., 2024b; Kim et al., 2025), understanding and addressing compositional challenges in text-to-image generative models has been a growing focus in the field (Thrush et al., 2022: Huang et al., 2023; Chefer et al., 2023; Peng et al., 2024c). In particular, Zarei et al. (2024) identifies key compositional
22

---

Published as a conference paper at ICLR 2025
SDXL
DALL-E
SD3.0
PixArt-Alpha
Ours
On the table in the living room, there hexagonal clock and three glass
Three westies are chasing two birds on the gross, one of which yellow and the other is green.
On the man in red throwing ball to woman wearing blue hat.
wolf and three yellow puppies.
There two chairs to the right of house.
On rainy day, detective wants to out but doesn't want draw any atteiion..
Figure 15: Qualitative Results.
challenges in text-to-image diffusion models and proposes strategies to enhance attribute binding and object relationships. Leveraging the power of large-lagguage models (LLMs) for compositional generation is another area of active research (Drozdov et al., 2022; Mitra al., 2024; Pasewark et al., 2024). For instance, Feng et al. (2023b) leverages large language models (LLMs) to generate visually coherent layouts and improve compositional reasoning in visual generation tasks.
23


Reviews:
[{"review_id": "6DYFywoKFq", "reviewer": "Reviewer_PM7r", "summary": "This paper addresses a complex text-to-image generation approach that aims to achieve accurate and high quality generations of complex samples.  The authors develop a data generation pipeline where they use a LLM and a VQA model to create fine-grained differences in images to help train their mode.  The authors then use curriculum learning to train a diffusion model using their data.  The authors evaluate their approach using compositional T2I benchmarks and provide a user study of their approach.", "questions": "I proposed some specific questions in the weaknesses.  Generally I would mostly like to hear about W1 and W2", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The approach is well motivated and addresses a critical shortcoming of T2I models\n2. The paper is relatively well written\n3. The experiments cover a number of key comparisons, including a users study, to help validate the effectiveness of their approach", "weaknesses": "1. The data pipeline relies on VQA method's ability to measure alignment.  However, just as T2I models struggle with this, VLMs also struggle, e.g., [A,B].  As such, it seems quite questionable that this would provide a high quality dataset. The statement on L259 on quality control is also ambiguous. How many total annotators were used? How many samples?  How were the samples selected (i.e., how did you ensure coverage of all the elements of the dataset?). The sentence there is vague and gives no details.\n\n2. The authors motivate the need to use generated data due to the cost of collecting pairs.  While I tend to agree with this generally, in practice it isn't clear if this is true.  Some benchmarks like [B] show that fine-grained negatives can be extracted from existing datasets.  There are also datasets with fine-grained pairs of images [C].  What would make this stronger is if the authors compared directly against trying to use some of the competing datasets in their model (including balancing the number of samples).  This would help demonstrate that the differences are important, or the effect of using generated vs. real pairs\n\n3. The user study in figure 8 is not convincing that the proposed approach is better.  The alignment has improved, but at the cost of the generation quality.  I don't have a specific recommendation here- it seems to be simply a weakness of the approach.  That said- the discussion could be more forthcoming and perhaps discuss why despite the drop in aesthetics this is still important.\n\n[A] See, Say, and Segment: Teaching LMMs to Overcome False Premises. CVPR 2024\n\n[B] Cola: A Benchmark for Compositional Text-to-image Retrieval. NeurIPS 2023\n\n[C] Evaluating Text-to-Image Matching using Binary Image Selection (BISON). 2019", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pXGTJMje3c", "reviewer": "Reviewer_drFx", "summary": "The authors make a dataset of images with hard negatives for a text prompt. They use LLMs to generate a hard negative prompt, stable diffusion3, and an image editing method (MagicBrush) to make pairs of positive and hard negative samples. Finally, they train a Stable diffusion model with an additional contrastive loss using the negative images on the denoising encoder representation. This leads to better compositional generation.", "questions": "See weaknesses.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- Introducing a contrastive loss in the denoising encoder representation is an interesting idea. \n- The experimental setups are clear, and there are good comparisons to baselines and ablations on their design choices.", "weaknesses": "- A bit more detail on how the contrastive loss is incorporated would be nice. Is the contrastive loss of the denoising encoder representation at any arbitrary time step t? I am assuming while training, the denosing encoder just predicts the noise to be subtracted for an arbitrary time step t. Hence, the losses would be the MSE on the noise prediction and the InfoNCE loss? What proportions are the added in?  \n\n- A few other datasets with hard-negative compositional images exist, albeit small. Two examples are COLA (https://arxiv.org/abs/2305.03689, they have a hard negative attribute-object composition image for a caption) and Winoground (which the authors already note).  A nice baseline to show would have been comparing using the author's synthetic dataset for tuning over using a combination of the real hard negative image datasets.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2mxSLSjcA8", "reviewer": "Reviewer_cqxf", "summary": "The paper introduces ContraFusion, a framework designed to enhance the compositional understanding of diffusion models through a combination of contrastive learning and multi-stage fine-tuning. This approach addresses the prevalent challenges faced by text-to-image (T2I) synthesis models, particularly in managing complex relationships between objects and attributes. It introduces the Com-Diff dataset, which comprises 15,000 carefully curated contrastive image pairs, tailored to refine the model's performance.", "questions": "- Do you have evaluation results for training on the Com-Diff dataset using multi-stage fine-tuning without incorporating contrastive loss? \n- Regarding the multi-stage fine-tuning process, have you considered reintroducing a small number of samples from earlier stages to maintain performance on simpler compositional prompts? Observations from Table 4 suggest a larger performance gap in complex scenarios (excluding texture); thus, recycling simpler samples might help bridge this gap and enhance overall model robustness.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper addresses a critical issue in text-to-image generative models, focusing on the challenge of compositionality. Despite significant advancements in this field, even the most sophisticated models often struggle with accurately rendering complex compositional details. This work provides an approach to mitigate these limitations.\n- While datasets for compositionality already exist, this paper makes a valuable contribution by introducing a new dataset that features contrastive pairs of images generated from positive and negative prompts. This dataset is specifically designed to enhance model performance on compositional tasks, making it a valuable resource for further research.\n- The paper applies two techniques—contrastive loss and multi-stage training—for fine-tuning Stable Diffusion on compositional data. It demonstrates the effectiveness of each technique through quantitative analysis, providing evidence of their impact on improving model accuracy and handling of compositional elements.", "weaknesses": "- The ContraFusion model is compared against other methods using the T2I-CompBench dataset. However, it is trained on the Com-Diff dataset, which likely overlaps noticeably with the T2I-CompBench test set. To address this, I recommend that the authors provide a detailed analysis of any overlap between Com-Diff and T2I-CompBench and discuss its potential impact on the reported results. Furthermore, I suggest evaluating the model on a completely separate test set to ensure the robustness and generalizability of the findings.\n- For improved clarity, it would be beneficial if the paper explicitly stated the metrics used in Table 3 and other relevant sections. For instance, the BLIP VQA score is employed for evaluation in Table 3, but this is not directly mentioned. Clearly indicating this metric would help readers better understand the basis of the model’s evaluation and the reported results.\n- The paper would benefit from more comprehensive details about the experimental setup. For example, it should specify the number of samples generated for calculating the BLIP VQA score, discuss the potential variability in results by indicating the standard deviation across different seeds, and provide a more detailed description of the user study, including the number of participants involved.\n- Incorporating a dedicated subsection on relevant background research would provide a richer context for understanding the current contributions. Specifically, discussing recent methodologies like those in [1] and [2], which offer new perspectives on addressing compositionality issues, would demonstrate awareness of and engagement with the latest developments in the field.\n\n[1] LayoutGPT: Compositional Visual Planning and Generation with Large Language Models, Feng et al, 2023.\n\n[2] Understanding and Mitigating Compositional Issues in Text-to-Image Generative Models. Zarei et al, 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2Hb9Ox755u", "reviewer": "Reviewer_Amfr", "summary": "This paper introduces CONTRAFUSION, a novel framework designed to improve the compositional understanding of text-to-image (T2I) diffusion models. The authors address these models' common issues, such as incorrect attribute binding and flawed object relationships, by introducing COM-DIFF, a high-quality contrastive dataset consisting of minimally differing image pairs across diverse attribute categories. Using a curriculum-based contrastive learning approach, CONTRAFUSION progressively fine-tunes models from simple to complex compositional scenarios. By leveraging large-language models (LLMs) and Visual-Question Answering (VQA) systems for enhanced image-text alignment, the framework significantly boosts model performance on compositional benchmarks, setting a new standard for T2I generation.", "questions": "1. How do you ensure that when generating negative images, only the specified attribute or relationship described in the prompt was changed? It’s unclear how this was rigorously verified.\n\n2. In Figure 3, isn't the dog's head in the second image of the first row incorrectly generated? It looks like there might be an issue with the image.\n\n3. What is the training cost of the proposed approach? Considering the computational expense, it’s not entirely clear if the performance improvement is significant enough to justify the cost.\n\n4. Are there any adverse effects, such as a decrease in the model's ability to generate diverse images, after fine-tuning?\n\n5. Given that CLIPScore has several limitations as an evaluation metric, such as being inaccurate in counting objects, isn't it contradictory to use CLIPScore for measuring alignment in a paper that aims to overcome these limitations?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper introduces a novel methodology targeting the major shortcomings of image generation models, such as the lack of expressive capability for multiple objects or attributes. It explains the creation of a benchmark to address these issues and the application of contrastive learning for model fine-tuning to achieve improvement, demonstrating superior performance across various models.", "weaknesses": "1. The argument that three different sub-tasks were resolved solely using contrastive learning is not very convincing, especially since the paper does not propose a novel loss function but rather uses an existing one. Given that the main contribution of this work is the creation of a benchmark, more justification is needed to support the claim of achieving multiple improvements solely from this contribution.\n\n2. With the text prompts structured into eight categories, there seems to be potential for generating a higher number of combinations than the 15,000 currently provided, which feels somewhat limited.\n\n3. The method used to measure text-image alignment has already been employed in several papers, such as TIFA and other works evaluating text-to-image generation using VQA and LLMs. These approaches have limitations, such as not being able to account for all elements in the generated images using a limited QA set, and issues like hallucination from LLMs and VLMs. It appears that these concerns have not been adequately addressed.\n\n4. The idea of modifying text prompts to match generated images, as mentioned around line 250, is unclear. It’s difficult to imagine how such prompts would be presented, especially given that generated images often have subtle and complex characteristics that are hard to describe accurately, as exemplified by the top-right part of Figure 2.\n\n5. The description of the three stages appears for the first time on page 5, but sufficient explanation should be provided earlier, such as in Table 2 or preceding sections.\n\n6. The claim that three annotators randomly assessed image-image similarity is questionable, as it would have been better to review all images comprehensively. There are likely cases where the differences were not properly captured. Alternatively, some automated method should have been used to ensure accuracy.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)