Evaluating Semantic Variation in Text-to-Image Synthesis: A Causal Perspective
Authors: Xiangru Zhu
Penglei Sun
Yaoxian Song
Yanghua Xiao
Zhixu Li
Chengyu Wang
Jun Huang
Bei Yang
Xiaoxiao Xu
Venue: ICLR 2025 Poster (2025)
Keywords: text-to-image synthesis
semantics
evaluation
metric
Primary area: datasets and benchmarks
TL;DR: Introducing SemVarEffect and SemVarBench, a novel framework for evaluating how well Text-to-Image models capture semantic variations in human instructions.

Abstract: Accurate interpretation and visualization of human instructions are crucial for text-to-image (T2I) synthesis. 
However, current models struggle to capture semantic variations from word order changes, and existing evaluations, relying on indirect metrics like text-image similarity, fail to reliably assess these challenges. This often obscures poor performance on complex or uncommon linguistic patterns by the focus on frequent word combinations.
To address these deficiencies, we propose a novel metric called SemVarEffect and a benchmark named SemVarBench, designed to evaluate the causality between semantic variations in inputs and outputs in T2I synthesis. 
Semantic variations are achieved through two types of linguistic permutations, while avoiding easily predictable literal variations.
Experiments reveal that the CogView-3-Plus and Ideogram 2 performed the best, achieving a score of 0.2/1. Semantic variations in object relations are less understood than attributes, scoring 0.07/1 compared to 0.17-0.19/1. 
We found that cross-modal alignment in UNet or Transformers plays a crucial role in handling semantic variations, a factor previously overlooked by a focus on textual encoders. 
Our work establishes an effective evaluation framework that advances the T2I synthesis community's exploration of human instruction understanding. Our benchmark and code are available at https://github.com/zhuxiangru/SemVarBench.

Published as a conference paper at ICLR 2025
EVALUATING SEMANTIC VARIATION IN TEXT-TO- IMAGE SYNTHESIS: A CAUSAL PERSPECTIVE
Xiangru Zhu Penglei Sun? Yaoxian Song' Yanghua Xiao Zhixu Li*
Chengyu Wang² Jun Huang" Bei Yang*
Xiaoxiao Xu
Shhnngaai Key Laboratory of Data Science, School of Computer Science, Fudan University "Hong Kong University of Science and Technology (Guangzhou) *Zhejiang University "School of Information, Renmin University of China; School of Smart Governance, Renmin University of China 5 AAlibaba Group
ABSTRACT
Accurate interpretation and visualization of human instructions are crucial for text-to-image (T2I) synthesis. However, current models struggle to capture se- mantic variations from word order changes, and existing evaluations, relying on indirect metrics like text-image similarity, fail to reliably assess these challenges. This often obscures poor performance on complex or uncommon linguistic pat- terns by the focus on frequent word combinations. To address these deficien- cies, we propose a novel metric called Sem VarEffect and a benchmark named Sem VarBench, designed to evaluate the causality between semantic variations in inputs and outputs T2I synthesis. Semantic variations are achieved through two types of linguistic permutations, while avoiding easily predictable literal varia- tions. Experiments reveal that the Cog View-3-Plus and Ideogram 2 performed the best, achieving score of 0.2/1. Semantic variations object relations are less understood than attributes, scoring 0.07/1 compared to 0.17-0.19/1. We found that cross-modal alignment UNet or Transformers plays a crucial role in han- dling semantic variations, a factor previously overlooked by focus on textual encoders. Our work establishes an effective evaluation framework that advances the T2I synthesis community's exploration of human instruction understanding.
Input Prompt: A cat chasing a mouse.
Input Prompt: A mouse chasing cat.
DALL-E
Midjourmey V6
Ideogram
Stable Diffacion
FLUX.1
CogView---PPls
Figure 1: Failed state-of-the-an (SOTA) T2I model examples: different permutations of the same words, different textual semanties, yet similar visual semantics.
INTRODUCTION
Accurately interpreting and visually depicting human instructions is essential for text-to-image (T2I) synthesis (Cao et al., 2024). Despite advancements in alignment (Lee et al., 2023a; Wu et al., 2023;
"Emal:: xrzhu @9@ fudan.edu.cn 1 'Corresooning author. Email: wwwpwy@@fudancddue
OOur benchmark and code available at https: thub. http://tww..........oc////////aa

---

Published as a conference paper at ICLR 2025
Topet
Variationn
Vissal Beaantic Evaluation
Effect
Interveation
Test
Image
Allgament Score
Visual Semantics Variation
Traditional Average Aligument
Under Textual Semantie Change do[T T.)
Seems Good
mouse chasing cat.
((T 0.65
S (IT, T 1x I (0
S0T,4))
s(r..,,)
Permunatios: Fyr (Vartanog)
S(T,
0.13
SemVarEtfest
A cut chasing mouse.
50%,4,) 0.98
Under Iifferent Intervention Conditious:
5(T,,)) 0.993 0.93
Under Testual Semantic Maintained dolf - T,)
Eyy Idee + 7.)) EEy
Mainteila
Semanticss
(TT (T (
A mouse being chased
s(T, - 0.95
0, 09
cat.
s(rp::4,)=099
0.04
Porracatio:: Fai
But not understand
(lavariaac))
Figure 2: Framework for measuring semantic variation causality in T2I models. Our evaluation con- sists of three components: (I) Input Variations with semantic change/maintenance interventions, (II) Visual Semantic Evaluation under both interventions (blue for semantic change, pink for se- mantic maintenance), and (III) Causal Effect Calculation where Sem VarEffect (purple) quantifies the difference between intervention outcomes. For Comparison, traditional alignment scores (gray) only measure surface similarity, as shown in the cat-mouse example where high alignment coexists with poor semantic consistency. See Section for mathematical details.
Kirstain et al., 2023), composition (Liu et al., 2022; Wang al., 2024; Li et al., 2024; Feng al., 2023b), and long instructions (Yang et al., 2024; Gani et al., 2024), these models still treat text prompts as bags of words. failing depict the semantic variations in human instructions (Yu al., 2024; Mo et al., 2024). As shown in Fig. I, existing T2I models generate images with identical semantics, even when the inputs differ semantically (e.g.. "a mouse chasing cat" vs. "a cat chasing mouse"). This indicates that existing T2I models struggle to accurately capture the semantic variations caused by word orders changes.
There is lack of direct metric to evaluate T2I model's ability to understand semantic variations caused by word order changes. Existing NIPP research typically evaluates semantic variation indi- rectly through downstream tasks. For example, in language generation (Gordon et al., 2020), the in- put sequences with different word orders are used as the actions in a navigation game and the model is evaluated based on the game's accuracy. Similarly, in visual-language understanding (Thrush et al., 2022; Diwan et al., 2022; Yüksekgönüül et al., 2023; Wang et al., 2023; Burapacheep et al., 2024), models are evaluated via cross modal retrieval and image-text matching, focusing text- image similarity. In T2I synthesis, the text-image alignment score offers an indirect performance measure but may not fully capture a model's sensitivity and robustness word order. For example, as shown in the upper-right of Fig. 2, an average alignment score of 85, as evaluated by GPT-4, might seem satisfying, but it may conceal the model's proficiency with common word combinations while masking its inadequacy with less frequent or more complex linguistic patterns.
We propose novel metric, called Sem VarEffect, to evaluate the causality of semantic variations between inputs and outputs of T2I models. Our approach uses inputs' semantics as the only in- tervention to evaluate the average causal effect (ACE) of this intervention on outputs' semantic variations, that is, the contribution of inputs to outputs. A significant ACE would indicate that the T2I model can effectively capture and reflect input semantic variations. On the contrary, small ACE, such the 0.09 shown in Fig. 2, exposes considerable weakness in the T2I model's ability to understand and respond to sentence semantics.
To facilitate the evaluation, we present new benchmark, called Sem VerVerrnnch. To avoid overt literal differences, semantic variations are achieved through two types of linguistic permuta- tions (Gerner, 2012): permutation-variann ings, and permutation-inwe -nvaatiionnivvrinnneee where the meaning remains unchanged regardless of word orders. Utilizing pre-defined templates and rules the guidance in the generation stage, followed by large
where different word orders result in different mean-
2

---

Published as a conference paper at ICLR 2025
amount of annotation and hard sample selection in the validation stage, constructed benchmark comprising 1 11,454 samples, where 10,806 are in the training set and 648 are in the test set. We experimented with variety of T2I models using our proposed benchmark and metric. The results show that even SOTA models like Cog View-3-Plus and Ideogram 2 struggle, achieving scores far from the ideal, which highlights the need for further advancements in handling semantic variations.
Our key contributions are: (1) A first systematic study of semantic variations in T2I synthesis, investigating the causal relationship between input text variations and output images. (2) Sem- VarEffect: A metric quantifying how semantic variations in input text affect T2I output quality. (3) Sem VarBench: An expert-annotated benchmark evaluating semantic variations in T2I synthe- sis through permutation-varie varriacc and permutation-invarian invariiace tests. (4) Comprehensive evaluation of SOTA T21 models, revealing significant limitations in handling semantic variations and distinct challenges posed by different variation types, while identifying specific areas for improvement.
2 SEMANTIC VARIATION EVALUATION FOR TEXT-TO-IMAGE SYNTHESIS
2.1 PRELIMINARY
The T2I model / generates images for each input sentence T, represented as 1 F(T). S(T, T) is the text-image alignment score, measuring text-image similarity. S(-) represents the scoring method.
Linguistic Permutation. Linguistic permutation refers changes in word order. Given an anchor sentence T., Tp and Tpe are two permutations of Ta- Tpe exemplifies permutation-variance,, which shows a change in meaning, while Tμ: exemplifies mains unchanged. The expected Ipe is a permutation of objects or relations from La, while Ipt is semantically equivalent to La. preserving the same visual objects and relations after transformation.
cermutation-invariin
where the meaning re-
2.2 DEFINITION OF VISUAL SEMANTIC VARIATIONS
First, we define the visual semantic variations observed in a single sentence T. We decompose complex semantic variation into minimal discrete steps, called localized changes. For each image I, the visual semantic variation of a single minimal discrete step I + AT, denoted as μ (T, I), is the difference in alignment scores: pr(T, S A S(T, When the anchor image In transforms permutation image Ip- through these minimal discrete steps, the integrated visual semantic variation can be measured by initial and final states: Σ (T, S(T, S(T,I))
Second, we sum the visual semantic variations from multiple text-image pairs to comprehensively measure these variations. For the sentence Ta. the visual semantic variations Σ/. p(Ta, /) demon- strate a shift from a matched a mismatched image-text pair, indicating negative change. For the sentence Tp.. the visual semantic variations Σ/." p(Tp..,) I) demonstrate shift from mismatched to matched image-text pair, indicating positive change. To measure the total magnitude of these variations regardless of direction, we use the absolute values. Therefore, the summation of visual semantic variations is defined as ;
-IEETA Ei M(T, n) |S(Tay 4p-) S(Ta, 12)l ()) (S(( Ip.)) - S(Tp., 1a)l
(I)
2.3 THE CAUSALITY BETWEEN TEXTUAL AND VISUAL SEMANTIC VARIATIONS
F0
Fig. 3 illustrates the causal relationship between input and output semantic variations. T' is the text input, serving as the input vari- able, while I is the generated image, acting mediator. S is the text-image alignment score, influenced by both T and I, and serves as an intermediate result variable. 7 denotes visual se- mantic variation and is the final comparison result variable. ʃ(() is exogenous variable representing T2I model that maps T to I. S(-) is an exogenous variable representing a scoring func- tion that maps T' and I to S. The dashed line between S and y indicates their derived relationship: x measures visual semantic variation by summing the absolute differences in alignment scores S between original and permuted text-image pairs.
S
r
Figure 3: Causal relationship between the input and the output semantic variations.
3

---

Published as a conference paper at ICLR 2025
Template pm
Template T.
Template Tp
Change the Semanties
Maintain Semanties
[Noun3] [Verb] [Noun2] and [Noun3] behind [Noun!]
T, [Noun1] [Verb] [Noun2] and [Nount] behind [Noun3]
Tx [Noun1] bahind [Noun3] and [Nount]
T, pui [Noun - [Noun3]
[]oun1] [Verb] [Noun2] [Noun1] behind [Noun3]
T, -T
[Verb] [Noun2]
Permutation 7
Permutation Fe
AncharT,
and the kid is flying giant
: kid is flying a giant butterfly kite and the kid bahind
flying glant butterfly kite and the cat is behind a kid.
butterfly kite.
(T glant butterfly
fieed by kid and the kid behind cat.) }
The data collection process of Sem VarBench. Top: Templates. Bottom: tences, The templates are extracted from the seed pair dog using wheelchair and the dog next to person"/'a person is using wheelchair and the person is next to a dog".
Figure 4:
Generated Sen-
According to causal inference theory, we define the average causal effect (ACE) of textual semantic variations on visual semantic variations as the Sem VarEffect score. It quantifies the influence of input semantic variations on output semantic variations. As shown in Fig. 3, the sentence T serves as an independent variable that influences the generated image /. The visual semantics variations jointly influenced by T, and 5(.). Let do(T * T.) and do(T T.) represent two types of interven- tions. do(T * T.) represents an intervention where differs in meaning from the anchor sentence Ta. The visual semantic variation caused by this intervention is denoted
// E/y do T.)) = E(T' TT - - S(T.. a [.]l +S(SSppp S(Tpu: Ipo) S(T [a)l (2) do T.) represents an intervention where T match the meaning of the anchor sentence Ta. The visual semantic variation caused by this intervention is denoted as:
%w/o = B/y do(T = T.)) = Ely | T SS(T.. [4) S(T,, Iw)l + +SS(T Inx) S(T,,, Ja)l (3)
By comparing visual variations under semantic change interventions and semantic maintenance in- terventions, we determine the ACE of textual semantic variations:
= E'y' do(T & T.)] EnT do Te) = Tiu. Yu/o
SS( 4p)) S(T,, 1.)1 - S(Tp.) 1pm) S(T la)l
[S(Ta.4pp) - S(2. 1.)1 [S(Tpi 4pi) S(T la)l:
The Sem VarEffeet score к. provides theoretically justified bounds (0.5-1.0) for the evaluation. The alignment score S(-) consists object and relation (triple) components, each contributing up to 0.5 to the total score. ranges from 0 to under ideal conditions, where f(-) accurately represents the text through images and S(.) faithfully measures text-image alignment. x > 0.5 indicates semanti- cally meaningful variations; random variations. When no triple remains identical under two interventions, K is maximized, with yuy reaching its upper bound and 74.o reaching its lower bound. More analysis are in Appendix B.
< 0.5 indicates variations insensitive to semanties; < 0 indicates
SEMANTIC VARIATION DATASET FOR TEXT-TO-IMAGE SYNTHESIS
We create semantic variation dataset for T2I synthesis through two types of linguistic permutations. In this Section, we first describe the data characteristics and then introduce collection pipeline.
3.1 CHARACTERISTICS OF DATA
Each sample (Ta, Ta::Twu Tpoy Tp:) consists of three sentences: anchor sentence T, and permutations Tpe and Tps. They should adhere to the following characteristics:
Literal Similarity: Ta. Toe and Tp are literally similar, differing only in word order.
Distinct Semantics: T, and Tpm have distinct semantics. T. and Tp share the same semantics.
Reasonability: Ta. Tp and Tp semantically reasonable in either the real fictional world.
4

---

Published as a conference paper at ICLR 2025
Visualizability: Ta. pe and Tp. describe something humans visualize. If text is unimaginable to humans, it cannot be meaningfully visualized by T2I models.
Discrimination: The images evoked by Ta and Tyu present distinguishable differences. The images cvoked by Ta and Tp appear similar.
Recognizability: The image evoked by Ta. Tp and Tpt maintain key elements necessary for recog- nizing typical scenes and characters.
3.2 DATA COLLECTION
We use LLMs (GPT-3.5) generate anchor sentences and their permutations, guided by templates. However, LLMS tend to produce patterns common in their training data, which leads to the neglect of less common combinations specified by templates and rules. To address this issue, we employ different process for generating To. Tp. and Tpi.
Template Acquisition. We choose all 171 sentence pairs suitable for T2I synthesis from Winoground (Thrush et al., 2022; Diwan et al., 2022) as seed pairs. These pairs are used to ex- tract templates and rules for T. and Tpu, while those for Tps are extended manually. To increase diversity, we change the word orders according to the part of speech, including number, adjective, adjective phrase, noun, noun with adjective, noun with clause, noun with verb, noun with preposi- tional phrase, verb, verb with adverb, adverb, prepositional and prepositional phrase. In Fig. 4, the top left shows an example of templates for Ta and Te derived from extraction, while the top right shows the corresponding templates for Ta and Tp: derived from manual completion.
Template-guided Generation for Ta. We use LLMs to generate anchor sentences by filling tem- plate slots based on prior knowledge and maximum likelihood estimation. In Fig. 4, the bottom middle sentence Ta is generated using the template for Ta as a guide.
Rule-guided Permutation for Tpe- Tp is generated by swapping rearranging words in Ta based on predefined rules, ensuring that Tpu introduces semantic variation. This method avoids a ran- dom generation autonomous generation by LLMs. By following these rules, Tpo includes many rare combinations commonly found in existing NLP corpora. In Fig. 4, Tp is generated by swapping [Noun]] and [Noun3] in T. (shown in the top left).
semantically equivalent passive structure to Ta, which common pitfall in
Paraphrasing-guide Permutation for Tp-- Tp can be generated by following rules, such changing phrases connected by coordinating conjunctions. However, not all sentences contain coor- dinating conjunctions. so we also allow other synonymous transformations, including passive voice and slight rephrasing. Both T'i examples in Fig. 4 are acceptable.
3.3 DATA ANNOTATION AND STATISTICS
LLM and Human Annotation. We establish 14 specific criteria to define what constitutes a "valid" input sample. LLMs check each sample against these criteria, labeling them as "yes" or "no" with confidence scores. Samples la- beled "no" with confidence scores above 0.8 are removed. Then, 15 annotators and 3 experts manually verify the re- maining samples. Each sample is independently reviewed by two annotators, with expert resolving any disagree- ments. This process produced 1,,454 valid samples, from which 684 challenging cases are selected for testing based on thresholds and voting. Details are in Appendix C.2.
ksmpersturt:
aboilae Iscadea: Si relutiv lscatie: SI
upc:
ventinunt:
quati-teapernl:
axtive
muterll:: 53
Virtee
Ieslare 50
fany counting: St
slape 50 opperrence
h5ighr:58
wright
Scale and Split. VerBenhh comprises 11,454 samples of (Ta. T Tpe, Tpi), divided into training set and test set. The training set contains 10,806 samples, while the test set consists of 648 samples. All evaluations are on the test set.
Figure 5: Distribution of semantic variations by category in the sem Var- Bench test set.
Category. In Sem VarBench, samples are divided into 20 categories based on their types of semantic variation. These categories are further classified into three aspects: Relation, Attribute Comparison, and Attribute Values. Fig. 5 shows the distribution of the test set in Sem VarBench.
5

---

Published as a conference paper at ICLR 2025
Model
Abbc:
Type
ADMM
Text
Image Genemitee
Image Decnder
RLP
Open-source Models
(Rombach al. 2022)
Stable Diflusion v1.3
SD 1.5
Diffusion
768
CLIP VIT-L
123.06M
UNet
$59.5IM
VAE
83.65M
1,07B
(Rombach al, 2022)
Stable Diftusion v2.1
SD
Diflasiae
1034
OpenCTIP VIT-H
340,59M1
UNet
$65.9IM
VAE
83.65
1.29R
Stable Diffusion XL v1.0
(Podell
Diffusion
2148
OpeCLIPP WiT-bgG
CLIP VIL
694.6M
123.06M
UNet
4.83
VAE
83.6M
6,518
Stble Cascade
(Pernis et 20124)
SD
1250
UNet
VQGAN
(Saharia 2022)
DoepFlayd
DeepFloyd
Diffusion
4096
T5-XXL
4,76B
UNet
6.02B
VAEE
55.338
11.18
(Chm et al., 1024a)
PoxAnt-alpha XL.
PixAn
Diffasian
4036
FXx-TS-XXI
4,768
Transformer
611.35M
VAE
83.65M
5.46R
Koloes (Team, 20124)
Kolors
Diffusioe
4856
CharGiLM
0,248
UNet
2,58
VAE
33.65M
891B
Stable Diffusion Mmneiium) (Esser 20124)
SD
Diffusion
2048
OpeaCLIP WiT-bugt
CLIP WTHL
662.43M
117.92M
Transformer
2,08
VAE
83.82M
7.09
TS-XXI.
4,76B
BLUX.][dev]
T5-XXL
Transformer
16.87B
API-bused Models
Midjouney VB DALL-E3 (Betker ot al., 20233
Midl V6
Diffasion
DALL D33
Ditfusion
T5 XXL
4,76B
UNet
VAE
CogView Plus
CogV3 Plus
Diffusion
T5 XXL
4,76B'
Transformer
VAE
Mengerm
Mangram
Diffasion
the eoit nuteia nosvia govidad
Table I: Mainstream T21 models. #DIM: pooled dimension of text encoders' outputs. #TEP, #IGP, #IDP, #ToP: parameters of text encoders, image generators, image decoders and whole models.
EXPERIMENTS
4.1 EXPERIMENTAL SETUP
T21 Synthesis Models.
We evaluate 13 mainstream T2I models as shown in Tab. 1. For each sen- tence, we generate one image, resulting in a total of 684 3 x 13 images. Each input prompt the sentence itself, without any negative prompts or additional details expanded by prompt generators.
Evaluators. We use MILMS as automatic evaluators to calculate text-image alignment scores: Gemini 1.5 Pro, Claude 3.5 Sonnet, GPT-4o and GPT-4 Turbo. Each evaluator is given sentence and an image and asked to assign two scores: object accuracy (0-50 points) and relation accuracy (0-50 points). The sum of these two scores is taken as the total score, which is then normalized to [0, 1]. To validate MLLMs' effectiveness, we conducted human evaluation using three raters on 80 samples (20 each from Midjourney v6. DALL-E 3, Cog View3Plus. and Ideogram2) with the same scoring protocol. Then we measure the correlation coefficients of MLLMs with human preferences.
Metrics. We use 4 metries: text-image alignment score (S,.), proposed Sem Var-Effect (s),, visual semantic variation under semantic change (:4,) and maintenance (o/)))) For each sample, Sa [ ΣigR S(T,, L)), where
(a. pi).. High Sa. The and k. coupled with low Thyo indicate model's strong causality of input to output. For brevity, we denote Sa. Thy- and 74,o as S.7 7u. and Two-
Evaluation Dataset. We evaluate T2I models on the test set in a zero-shot manner. To demonstrate the improvements from fine-tuning, we collected sentences and their generated images from the training set, selecting only those with high quality, high discrimination, and consistent variations as the training data. Details about the selection of the training data are provided in Appendix D.3.
4.2 RESULTS
The results of the influence of inputs semantic variations on outputs semantic variations in T21 synthesis are shown in Tab. 2. The scores for S range between 0.6 and 0.8. Despite the alignment score reaching up to 0.8. this does not imply strong grasp of semantics. The following three metrics provide more comprehensive view of the model's ability handle semantic variations.
Visual Semantic Variation with Changed Textual Semantics. As shown in Tab. 2, the values of Yhe all below 0.52 for all evaluators, significantly lower than the optimal value of 1. This indicates that none of the T2I models perform at an acceptable level. These models are highly insensitive to semantic variations. This finding aligns with the widely accepted notion that T2I models tend to

---

Published as a conference paper at ICLR 2025
Models
1.5 Pro
Claudo Sommet
GPT-4o
GPT-
S(t)
7x(1)
| (tx()) M(T)
S(t)
(ux(C)
x(f)
8(t) | Pwx())
x(t)
S(1))
| 7T
Open-source Models
SD 1.5
0.55
0.43
0.46
-0.03
0.64
0.19
0.30
-0.01
0.63
0.34
0.33
0.01
0.65
0.32
0.32
0,00
SD 2.1
0.58
0.45
0.46
0.01
0.66
0.21
0.20
0:01
0.65
0.33
0.02
0.68
0.35
0.34
0.01
0.39
0.69
0.19
0.18
0.00
0.28
0.35
0.03
0.41
0.69
0.19
0.33
1011
0.67
4000
0.31
0.01
DagpFloyd
0.44
0.00
071
0.20
0.19
0:01
0.69
0.33
0.30
0:03
0.74
0.33
0.28
0.05
PixAat
0.60
0.35
0.32
0.02
0.69
0.17
0.15
0.02
0.70
0.29
0.26
0:03
0.71
0.29
0.27
0.02
Kuluns
0.60
0.41
0:42
-0.1
0.69
0.22
0.22
-0.01
0.69
0.31
0.30
0.01
0.69
0.33
0.30
0.02
SD
0.67
0.45
0.40
1:015
0.76
0.22
0.19
0:04
0.75
0.35
0.29
01:07
0.76
0..33
0.28
0.05
FLUXX.1
0.72
0.43
0.35
0.08
0.75
0.25
0.17
0.06
0.72
0.42
0.33
0.10
0.75
0.40
0.30
0.10
API-based Models
0:21
0.25
0.38
0.22
DALI-R 3
0.75
0.46
18
0.82
0.36
0.23
0.83
0.35
0.30
0.10
OogV3-Plus Plus
0.79
0.52
0.35
0.17
0.80
0.28
0.18
0.10
0.81
0.49
0.28
0.20
0.82
0.43
0.26
0.17
Mrogram 2
0.80
0.47
0.29
0.18
0.79
0.26
0.17
0.09
0.81
0:46
0.27
0.20
0.81
0.40
0.24
0.15
Table 2: Evaluations on T2I models (left column) using semantic variations under multiple MLLM
scores (top row). S,(()): text-image alignment score. x(t): Sem Vam---fffeet,, measuring input-to- output variation contribution. (t)) and (L)) visual semantic variation under semantic change and maintenance. Bold and underline indicate 1st and 2nd optimal cases. Blue and green indicates average Sem VarEffect scores between 0.05 and 0010, and above 0.10, respectively.
treat input text as collection of isolated words leading them to interpret sentences with minor changes in word order as having the same meaning.
Visual Semantic Variation with Unchanged Textual Semanties. The values of Two in Tab. are unexpectedly much higher than the optimal value of O. Only the models highlighted in blue and green demonstrate slightly better performance, with Two scores consistently lower than Yw- These T2I models illustrate potential semantic variations caused by word order through images, yet struggle to differentiate between meaning-variant and meaning-invarian inputs. These models primarily understand language based on word order rather than the underlying semantics.
Influence of Textual Semantics on Visual Semantic Variations. In Tab. 2. the k values for all evaluators are below 0.20, indicating considerable room for improvement in T2I models' under- standing of semantic variations. Models with higher alignment scores are more sensitive to semantic variations caused by word orders. However, models highlighted in blue overreact permutations maintaining the meanings, resulting in higher Thoo values and subsequently lower e values. These models excel at capturing common alignments but struggles to handle semantic variations.
Human Evaluation. We observe consistent performance trends between human raters and the four MLLMS across all evaluated models. Correlation analysis on Cog View3-Plus reveals moderate (up to 0.54) correlation coefficients between machine and human scores, suggesting our selected MLLMs can serve as a reliable proxy for human evaluation. Detais are shown in Appendix E.2.
Comparison of Metrics. Sem VarEffect k focuses on the consistency of variations, while the align- ment score S focuses on the overall output quality. Thus, a model may score high on K even if consistently generates incorrect attributes (such as believing oranges are blue and bananas are green). Sem VarEffect validates the reliability of high aligmment scores: high S with low x indicates limited semantic understanding, while high S with high k suggests effective semantic understanding.
4.3 ANALYSIS
Is a superior text encoder the exclusive solution for T2I models to grasp semantic variations? Given differences in text encoders' ability discriminate semantic variations, we examine whether two metrics, alignment scores 5 and Sem VarBffect K, offer insights beyond what text encoders cap- ture. We use text similarity² to measure the text encoder's ability, and use visual semantic variation scores Tw and "wo to measure Sem VarEffect, as illustrated in Fig. 6. PixArt and Kolors, utilizing T5 and ChatGLM as text encoders, fail transfer the results of distinguishing semantic variations to image generators, as shown by permutation-variane (indicated by squares). However, FLUX.1, utilizing weaker CLIP-T5 hybrid models as text encoders, achieves higher 5 and greater differenti- ation in To and Ywo despite showing minimal changes in text similarity. It indicates that model's
Sentences for changed textual semantics unexpectedly show higher text similarity than those for unchanged textual semanties, likely due the edit distance between our sentences. For further analysis, see Appendix F.

---

Published as a conference paper at ICLR 2025
Visual Semantic Varlation Score
CJP Series Text Encoder
0.65
CUPTS SUPTSI Hybrid Text Encoder
FLUX
Alignment Score
0.00
DALL
6.40
ChasSLM Encoder
Encoder
FLUX.1
Yo Permutation Vanance
0.75
Purmutation--variiin
0.70
Drepfloyd
50.37
Deepricyd
8:65
Kolbrs
1.0
Kolors
5D
50
1.0
0.00
21 330
PlAAr:
8.55
DALUE3
0.70
90
1.00
0.70
0.00
0
0.60
0.95
1.00
Cosine Similarity
Cosine Similarity
Figure 6: Illustration of the text embedding similarity between the anchor text and the permuted text. Squares represent permutation-variane results (with changed textual semantics), while triangles represent permutation-invaaiince results (with unchanged textual semantics). The evaluator is GPT- 4o. (a) The alignment score between the anchor image La and permutation Tp. decreases as the text similarity between Ta and Tp. increases. (b) The semanttc variation score y increases as the text similarity between Ta and Tp. increases. The cosine similarity for DALL-E 3, API-driven model, is deduced using T--XXI,, indicated by hollow shapes.
ability to distinguish semantic variations not only dependent on text encoders, and further efforts are needed in cross-modal alignment to effectively transfer these differences to the image generators.
Altribute Comparison
Attribute Value
Relation
Re ation
Propertion
0.60
2:1
SD XI 1.0
60.0%
50 CA
0.30
Deepfleyd
Comsarison
Value
5D 3
28
FLUX 1
DALL-E
Mic] V6-
EAEES
28.5%
CogV3 Plus.
Meagram
Absolute Location
Relative Locatinn
Action Direction
Interaction
of
Size Weigitt Vague Amount
Height
Color
Matarial
Appearane
Temperaton
Age
Shape
Texture
Manner
Counting
Figure 8: The distribution of Sem SemVarfffed scores for the SOTA model Ideogram across different aspects of the samples. The evaluator is GPT-4 Turbo.
Spatio
Figure 7: The distribution categories across different T2I models based on Sem VarEffect scores The evaluator is GPT-4 Turbo.
Does the influence of input semantic variations on output semantic variations vary by cate- gory? As shown in Fig. 7, the Sem VarEffect scores in Color consistently exceed 0.4 in many models, while those in other categories are mostly below 0.1. This suggests that T2I models understand se- mantic variations well only in the case of Color. We found that the Sem VarEffect scores of Ideogram 2 in Relation, Attribute Comparison, and Attribute Value are 0.07, 0.13, and 0.19. To compare the distribution of Sem VarEffect scores across different aspects, we set 0.2 and 0.5 thresholds. As shown in Fig. 8, the proportion of high scores in Attribute Values is significantly higher than those in Relation and Attribute Comparison. T2I models lack the capability discriminate semantic vari- ations, particularly in aspects emphasizing relations and comparisons. Fig. 9 shows failed examples Relation and Comparison. Although T2I models can generate correct images for common rela- when semantic variations occur,
tions, they tend to rigidly adhere these common relations leading to incorrect images. More examples are provided in Appendix G.
Does fine-tuning improve T21 model performance on semantic variations? We examine im- provements from fine-tuning text encoders and image generators. We use samples in the training set to generated images and select text-images pairs with high alignment scores and high discrim- inability as training data, details shown in Appendix D.3. As shown Tab. 3, for categories with sufficient high-quality data, such as Color, supervised fine-tuning (SFT) enhanced the performance
8

---

Published as a conference paper at ICLR 2025
of the T2I model. However, in categories with insufficient high-quality data, such as Direction, SFT led to decline in performance. Additionally, direct preference optimization (DPO) resulted in invariance, evidenced by the increased 7 Two
performance drops due to failures in
GPPT-4
Category
Models
Category
Models
S(t)
7+(()
A(?))
S(t)
%u(1)
<(t)
Color
+ *lll-unet
+ sll--ext
0.7%-)
0.78(1)
0.4X))
0.38())
0,27())
0.20(()
0.13())
0.18())
Height
+ sft-test
=sll-unct
0.77(-)
0.73(1)
0.39c1)
0.33(1)
0.34(1)
0.24(1)
0485()
0.09LL)
upo-umet
0.69(1)
0.3(t)
0.27(1)
0.12(1)
dpo-unet
0.71(
0.31(-)
0.33(1)
0.02(1)
dpp-ttedt
0,68(L)
0.47(1)
0.29())
-pp--tett
0.66(
0.40(1)
0.53(1)
40,13())
SD
0.64
0.29
0.34
.0.05
SD XL
0.79
0.20
0.15
0.05
Location
Absolute
+ stt--unet
0.64(
(L65(T)
0.31())
0.34(T)
0.36c))
0.32())
-0.05c
0,02(1)
Direction
stt-umer
0.77(
0.77(
0.23(1)
0.24())
0.21(1)
0.23(1)
0.1())
0.03))
-ggo-ueet
0.6013)
0.29())
0.31
dpo-unes
0.65(L)
0.23(1)
0.26(()
.0.03(.)
dpo-text
0.57(J)
0.33T)
0339
-000%
4po-tces
0.700
0.29K1)
0.27(1)
0.01()
Table 3: Performance of fine-tuned models measured by the Sem VarEffect score. Training candi- dates for Color, Absolute Location, Height, and Direction are 4.4k, 1.7k, 0.2k, and 0.3k.
crucial to find a balance between sensitivity and robustness to semantic changes, as this determines whether performance can be enhanced. However, fine-tuning tends to improve sensitivity at the expense of robustness. While T2I models become more sensitive to permutations with different meanings, this discrim- ination is quickly disrupted by over-sensitivity permutations with similar meanings, weakening the model's 's ability to discern differences. This phenomenon may be attributed to two poten- tial limitations in Diffusion- based T2I Models. First, its cross- attention mechanism only maps tokens to visual regions without capturing inter-toker token relationships. Second, the training process lacks semantie-level supervision. Fine-tuning improves token- region correspondence in Tab. 4, but cannot enhance the under- standing of semantic relationships between tokens. Thus, these permutations, which differ only in word order but contain identi- cal tokens, confuse the models and lead to performance declines, especially during DPO. More validation is in Appendix F.4.
Token Accuracy
Canegory
Model
T,
Tje
Ty²
Absolute
0.716
Location
+sfl-unet
0.718
0.654
0.716
0,861
Height
+sf1 +sfl-unet
0.886
0.662
0.866
Table 4: Performance of fine-
tuned models measured by token accuracy on Absolute_Location and Height. Token accuracy refers to the ratio of tokens suc-
cessfully generated that corre- spond to images. We filter out meaningful tokens and verify their visual representation in the image using GPT-4o.
Anchar Text
Permutation-Va cirmuttttion-Varinnce
Permutation- Commutation-Variance
SemVarEffect Score
The baby cranss and the panent wallo.
The and panens cravls.
The parent malis babl crewts.
GPT-4
Macched
U344
045
Action
pars
0:98
0.25
507
30. L-.) 0%
$(7.kk) 0.95
The elder teacher's hand young student's shoulder.
The young student's hand shoulder.
The young student's under
teacher's
teacher's
Relative Location
B
AfTu.)) 055
5(5y-le) 0.95
50 L.) 0.85
U.2
Milmatccee para
50T fa.)
0.95
giraffe is
The than
aobra shorier the
Marched 5(T,= fie) 0.55
.0t.A) 020
0.79
Hoight
5(%x-4e) 0,85
D05
0.50
Mumattced
pars
5(a.kp))
-0.9
-010
Figure 9: Failed examples of DALL-E3 Relation and Attribute Comparison.

---

Published as a conference paper at ICLR 2025
Do T21 models' struggles with semantic relationships stem from training data imbalance? We conducted experiments testing performance of fine-tuned SDXL trained with balanced training data. We used human-crafted balanced dataset of cat+>dog chasing interactions (80 images per direction) for fine-tuning, and tested on two unseen prompt pairs involving the same "chasing" re- lationship between common objects. The model consistently showed poor generalization: accuracy remained low for both anti-commonsense scenarios (mouse+>cat, 3-3/30) and plausible scenarios (bull<-man, 4-5/30). Failure analysis revealed three main categories: (1) Relationship Understand- ing Failures, where objects appear either without interaction or with incorrect interactions, indicating the model's 's inability to comprehend the "chasing" concept; (2) Reversed Roles, where the model fails to properly assign who chases whom; and (3) Missing Objects, where the model fails to gen- erate all required objects. Even among the generated images, correct relationships occurred less frequently than these failure cases, suggesting random performance rather than true understanding. Thus, the model's struggles persist even with perfectly balanced training data, suggesting the core issue lies in relationship understanding rather than data imbalance. Details are in Appendix F.3.4.
5 RELATED WORK
Evaluation of T21 synthesis. Benchmarks of T2I synthesis primarily focus on general align- ment (Saharia et al., 2022; Yu et al., 2022; Cho ct al., 2023a), composition (Park et al., 2021: Feng et al., 2023a; Park et al., 2021; Hu et al., 2023; Cho et al., 2023b; Li et al., 2024), bias and fair- ness (Lee et al., 2023b; Luo et al., 2024b;a), common sense (Fu et al., 2024) and creativity (Lee et al., 2023b). In these evaluations, the quality of images is measured by caaeetion-based or alignment- based metrics. Recent research on T2I synthesis has explored samples involving semantic variations caused by word orders, typically using them evaluate reasoning abilities with alignment- metrics (Marcus et al., 2022; Lee et al., 2023b; Li et al., 2024), However, a significant gap in this re- search is the underexplored area of whether the generated images consistently represent fundamental semantic variations within the input text.
Semantic Variation Evaluation in VLMs. In VLMs, semantic variations caused by word order has been evaluated by benchmarks like Winoground (Thrush et al., 2022) and its expansion in specific domain (Burapacheep et al., 2024). Winoground is designed to challenge models with visio-linguistic compositional reasoning. It requires models to accurately match two images to their respective captions, where the two captions are different permutations of the same words, resulting in different meanings. To enhance performance training datasets with negative samples and optimizing training strategies to handle the resulting semantic variations (Yüksekgönüi et al., 2023; Hsieh al., 2023; Burapacheep et al., 2024).
Winoground, studies have focused expanding
The application of Winoground to T2I synthesis faces several limitations due to the variety and quantity of its permutations. First, the dataset, with 400 sentence pairs, provides only 171 suitable for text-image composition analysis (Diwan et al., 2022), where samples are classified into three cat- object, relation, and both. This limited variety is insufficient for comprehensive evaluation of T21 models. Second, the suitability of certain samples for T21 model evaluation is problematic. Winoground primarily focuses on semantic distinctiveness for cross-modal retrieval (Yüksekgönül et al., 2023; Ma et al., 2023; Cascante- -oniila et al., 2023). It overlooks the criteria essential for T21 synthesis, such as sentence completeness, clarity of expression, unambiguity, and specificity in referencing image elements. All of these factors have been carefully considered in the quality control of our benchmark annotations.
egories:
6 CONCLUSION
We comprehensively study the challenge of semantic variations in T2I synthesis, specifically fo- cusing causality between semantic variations of inputs and outputs. We propose new metric, Sem VarEffect, to quantify the influence of input semantic variations on model outputs, and a novel benchmark, Sem VarBench, designed to examine T2I models' understanding of semantic variations. Our experiments reveal that SOTA T2I models struggle with semantic variations, scoring below 0.2 on our benchmark. Fine-tuning shows limited improvement, improving sensitivity but at the cost of robustness. These findings underscore the need for better tross-modal understanding of relation in semantics, particularly for capturing inter-token dependencies in T2I synthesis.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work is supported by Alibaba Group through Alibaba Innovative Research Program, the Post- doctoral Fellowship Program of CPSF (GZC20232292), the Suzhou Key Laboratory of Artificial Intelligence and Social Governance Technologies (SZS2023007) and the Smart Social Governance Technology and Innovative Application Platform (YZCXPT2023 101). We also thank Yunpeng Liu, Yuanyi Xu, and Kejun Xue for their assistance in supplementing the human evaluation experiments during the rebuttal phase.
REFERENCES
Anthropic. Claude 3.5 sonnet, 2024. URL
claude- 3--ie-3-5-oonee sonnet.
Eslam Mohamed Bakr, Pengzhan Sun, Xiaoqian Shen, Faizan Farooq Khan, Li Erran Li, and Mo- hamed Elhoseiny. Hrs-bench: Holistic, reliable and scalable benchmark for text-to-image mod- els. In IEEE/CVF International Conference Computer Vision, ICCV 2023, Paris, France, October 1-6, 2023, pp. 19984 19996. IEEE, 2023. doi: 10.1 10011990CCV11000.22...0001 https: //t:///oii org/ 10. 1101111110111 1T/9///CCC1100000 2023. 01834.
URL
James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, Wesam Manassra, Prafulla Dhariwal, Casey Chu, Yunxin Jiao, and Aditya Ramesh. Improving image generation with better captions. 2023. URL https: / //cán. openai. com/papers / dall-er 3.pdf.
Jirayu Burapacheep, Ishan Gaur, Agam Bhatia, and Tristan Thrush. Colorswap: A color and word order dataset for multimodal evaluation. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.). Findings of the Association for Computational Linguistics, ACL 2024, Bangkok, Thailand and virtual meeting, August 11-16, 2024, pp. 1716-1726. Association for Computational Linguis- tics, 2024. URL https: htwpp:/h/////althhhhcccccccccc logy. 2024 f Finding aaciingsscl....
Pu Cao, Feng Zhou, Qing Song, and Lu Yang. Controllable generation with text-to-image diffusion models: A survey. CoRR, abs/2403.04279, 2024. doi: 1044550/ARXIV.2400..60
URL
https: //oii org/l0. 48540 arwiv. 2403. 04279.
Paola Cascante-Bonilla, Khaled Shehada, James Seale Smith, Sivan Doveh, Donghyun Kim, Rameswar Panda, Gül Varol, Aude Oliva, Vicente Ordonez, Rogério Feris, and Leonid Karlin- sky. Going beyond nouns with vision & language models using synthetic data. In IEEE/CVF International Conference on Computer Vision, ICCV 2023, Paris, France, October 1-6, 2023, pp. 20098 20108 IEEE, 2023. doi: 10.1 111.///CC11100.222..... 10. 1109/ ICCV5 070. . 2023. 044.
htttps:///oo. orrg/
Junsong Chen, Jincheng Yu, Chongjian Ge, Lewei Yao, Enze Xie, Zhongdao Wang. James T. Kwok, Ping Luo, Huchuan Lu, and Zhenguo Li. Pixart-o: Fast training of diffusion transformer for photorealistic text-to-image synthesis. In The Twelfth International Conference Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024a. URL net/ wwwprmmmideeAA@ePP
Zhaorun Chen, Yichao Du, Zichen Wen, Yiyang Zhou, Chenhang Cui, Zhenzhen Weng, Haoqin Tu, Chaoqi Wang, Zhengwei Tong, Qinglan Huang, Canyu Chen, Qinghao Ye, Zhihong Zhu, Yuqing Zhang, Jiawei Zhou, Zhuokai Zhao, Rafael Rafailov, Chelsea Finn, and Huaxiu Yao. Mj- bench: your multimodal reward model really a good judge for text-to-image generation? CoRR, abs/2407.048.. 2024b. doi: 10.48 00448500/RRIVV24000 044444 04842. URL https: ott./.00......... 48550/arXiv 2407. 04842.
Jaemin Cho, Abhay Zala, and Mohit Bansal. DALL-EVAL: probing the reasoning skills and social biases of text-to-image to-image generation models. In /EEE/CVF International Conference on Computer Vision, ICCV 2023, Paris. France, October 1-6, 2023, pp. 3020-3031 IEEE, 2023a. doi: 10. L109ICCCV5000..202.... 000233. URL https: /ttp://o..oo/////o/oooooooo wtt.://.....././//1111.......... 00283.
1109/ http///CCV1110.000000 2023.
11

---

Published as a conference paper at ICLR 2025
Jaemin Cho, Abhay Zala, and Mohit Bansal. Visual programming for text-to-image generation and evaluation. CoRR, abs/2305. 15328, 2023b. doi: 10.4850/ARXIV.2005.5555 URL https: //doi.org/10. 1155//500011./ 2305.
Jaemin Cho, Yushi Hu, Jason M. Baldridge, Roopal Garg, Peter Anderson, Ranjay Krishna, Mo- hit Bansal, Jordi Pont-Tuset, and Su Wang. Davidsonian scene graph: Improving reliability in fine ane-grained evaluation for mext-to-image generation. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL https://p//.......t... wwwperppeeiieeeeeee ew net/ forum? 1d=TT447RU44:
Anuj Diwan, Layne Berry, Eunsol Choi, David Harwath, and Kyle Mahowald. Why is winoground hard? investigating failures in visuolinguistic compositionality, In Yoav Goldberg, Zomitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference on Empirical Meth- ods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, De- cember 7-11, 2022, pp. 2236-2250. Association for Computational Linguistics, 2022. doi: WTW.:///11222.EEL---MMMME emn mammmaiin......
URL https: /tw.. wrt.:/..g...o.. 2ttg//0..165//1//22222
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, Dustin Podell, Tim Dockhorn, Zion En- glish, and Robin Rombach. Scaling rectified flow transformers for high-resolution image synthe- sis. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024. URL https: wttpp//pwww.ew.ceopppp/pp net forum?id- FPr.hsQJJB.
Weixi Feng, Xuehai He, Tsu-Jui Fu, Varun Jampani, Arjun R. Akula, Pradyumna Narayana, Sugato Basu, Xin Eric Wang, and William Yang Wang. Training-free structured diffusion guidance for compositional text-to-image synthesis. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023a. URL net/ Eorum?id-PUiqiie
pettss://penne
Weixi Feng, Wanrong Zhu, Tsu-Jui Fu, Varun Jampani, Arjun R. Akula, Xuehai He, Sug- ato Basu, Xin Eric Wang. and William Yang Wang. Layoutgpt: Compositional vi- sual planning and generation with large language models. In Alice Oh, Tristan Nau- mann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Annual Conference Neural Informa- tion Processing Systems 2023. NeurIPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023b. URL http: /ttt::/pppptt papersaprsssseespa iappeerssippps. nips cc/paper (777f9445845daaa -at.1377771441a0---aaaaa----oee-
html.
Xingyu Fu, Muyu He, Yujie Lu, William Yang Wang, and Dan Roth. Commonsense- challenge: Can text-to-image generation models understand commonsense? In First Conference on Lan- guage Modeling, 2024. URL https:: / wwwpeerreeiww..ee/eeeeeeeee
Hanan Gani, Shariq Farooq Bhat, Muzammal Naseer, Salman Khan, and Peter Wonka. LLM blueprint: Enabling text-to-image generation with complex and detailed prompts. The Twelfth International Conference Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL mNYFOIHbRY.
net met/forum?id
Matthias Gerner. Predicate-induced permutation groups J. Semant., 29(1):009-144, 2012. doi: 10. 10.1093//O//FFR00.. URL https::////te..t. //to:.///ooooe/ corpiioorg/1000 1093/ fr007.
Tejas Gokhale, Hamid Palangi, Besmira Nushi, Vibhav Vineet, Eric Horvitz, Ece Kamar, Chitta Baral, and Yezhou Yang. Benchmarking spatial relationships in text-to-image generation. CoRR, 10015,, 2022. doi: 1044850/ARXIV.2212..1 10015. URL https: http:///ooooo...../////oo// 48550/arxiv 2212. 10015.
Jonathan Gordon, David Lopez-Paz, Marco Baroni, and Diane Bouchacourt. Permutation equiv- ariant models for compositional generalization in language. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia. April 26-30, 2020. OpenRe- view.net, 2020. URL htttps://pperreee net wwtte//ooommmddlllleeeo Fer.
12

---

Published as a conference paper at ICLR 2025
Cheng- Yu Hsieh, Jieyu Zhang, Zixian Ma, Aniruddha Kembhavi, and Ranjay Krishna. Sug- arcrepe: Fixing hackable benchmarks for vision-language compositionality. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural In- formation Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023. URL http: /ttp:r/papprssee nips cc seaper_fiess/ppeer hash/ wew.eeeeecc766ff4455 Benchmarks html.
-bbssrrat Datasets_an and
Abstract
Yushi Hu, Benlin Liu, Jungo Kasai, Yizhong Wang, Mari Ostendorf, Ranjay Krishna, and Noah A. Smith. TIFA:
accurate and interpretable text-to-image faithfulness evaluation with question swering. In /EEE/CVF International Conference on Computer Vision, ICCV 2023, Paris, France, October 1-6, 2023, pp. 10t.011111///111111111 atttps://doi. http://dooooog.....11/// 1109 TCCV51 070 2.23. 01866.
IEEE, 2023. doi: 10.1 10010000010 11////CCV1100000...1111111
URL
Kaiyi Huang, Kaiyue Sun, Enze Xie, Zhenguo L.,, and Xihui Liu. T2i-comphench: A com- prehensive benchmark for open-world compositional text-to-image generation. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Annual Conference formation Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023. URL http: /ppperspesssssesee nips cc cc/paper halpppprfileee//llllllalaaapeeee files/ wwtps//ppppee//....... hash/ fach/f8aad0 f8ad01 Berccmarke rks. html.
Neural In-
Diederik Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. Ad- vances neural information processing systems, 34:21696-21707, 2021.
Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, and Omer Levy. Pick-a-pic: An open dataset of user preferences for text-to-image generation. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural In- formation Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023. URL http: wwwppaappppppppapeeeeeeeaeaeeeep nips wwtppppeeeillleeeeeee htwplee//pppe//22/2222 2tt3//2222/////////// 3baacd@b3b05b4bb555 (w..555335553100525552533
Abstract- -orere-rtttatttttoeeeeeeeeee
html.
Kimin Lee, Hao Liu, Moonkyung Ryu, Olivia Watkins, Yuqing Du, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, and Shixiang Shane Gu. Aligning text-to-image models using human feedback. CoRR, abs/2302.12192. 2023a. doi: (044855/ARKXI.2300.2DD0 URL https: //doi. www..rrg//l1 10.0 48550
2302. 12192.
Tony Lec, Michihiro Yasunaga, Chenlin Meng. Yifan Mai, Joon Sung Park, Agrim Gupta, Yun- zhi Zhang, Deepak Narayanan, Hannah Benita Teufel, Marco Bellagente, Minguk Kang, Taesung Park, Jure Leskovec, Jun-Yan Zhu, Li Fei-Fei, Jiajun Wu, Stefano Ermon, and Percy Liang. Holis- tic evaluation of text-to-image models. CoRR, 0ass23311.44282 2023b. doi: 10.48550/ARXIV. 2311 1,04877 URL hntpps://oa
www.ooo.//.........o 4855./855//400 wwwiv. 2311. 04287.
Baiqi Li, Zhiqiu Lin, Deepak Pathak, Jiayao Li, Yixin Fei, Kewen Wu, Xide Xia, Pengchuan Zhang, Graham Neubig, and Deva Ramanan. Evaluating and improving compositional text- to-visual generation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2024 Workshops. Seattle, WA, USA, June 17-18, 2024. pp. 5290-5301. IEEE, 2024. doi: URL https: / /wo.oooooooooooooooooooo org. /1rgg11000 1109 /000///3333332....
10. 11w.0//PRW6322222000000 0024.00550
Nan Liu, Shuang Li, Yilun Du, Antonio Torralba, and Joshua B. Tenenbaum. Compositional visual generation with composable diffusion models. Shai Avidan, Gabriel Brostow, Moustapha Cissé, Giovanni Maria Farinella, and Tal Hassner (eds.), Computer Vision ECCV 2022 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XVII, volume 13677 of Lecture Notes in Computer Science, pp- 423 439. Springer, 2022. doi: 10.1007/ 978-3-031- 19790- 26. URL https: bttps:://oi... /doi org 1000000.0000 10. 001- 0-00---33311-------011 266
13

---

Published as a conference paper at ICLR 2025
Hanjun Luo, Ziye Deng, Ruizhe Chen, and Zuozhu Liu. Faintbench: A holistic and precise bench- mark for bias evaluation in text-to-image models. CoRR, abs/2405. 178144 2024a. doi: 10.48550/ ARXIV 17814. URL https: http://:///o..ogg/////e//o/ 481111..55//22//
Hanjun Luo. Haoyu Huang, Ziye Deng, Xuecheng Liu, Ruizhe Chen, and Zuozhu Liu. Bigbench: A unified benchmark for social bias in texi-to-image generative models based on multi-modal LLM. CoRR, abs/2407.15240 2024b.
doi: 10.48550/ARXII.2200.1522 URL https:// doi. ort. 10. 40w54//02iv..... 2ttp:/aawai...2.15222.2
Zixian Ma, Jerry Hong, Mustafa Omer Gul, Mona Gandhi, Irena Gao, and Ranjay Krishna. @ CREPE: can vision-language foundation models reason compositionally? IEEE/CVE Con- ference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver BC, Canada, June 17-24, 2023, pp. 10910-10921, IEEE, 2023, doi: 10. 00.110//CVS5777222230000063 URL https: / doi org/ 10 wwt111110////222200
Gary Marcus, Ernest Davis, and Scott Aaronson. very preliminary analysis of DALL-E CoRR, 0bs/2204.13807,, 2022. doi: 10.48550/ARXIV.220.. 13880 URL 48550/arXiva 2204 13807.
org/10..
Wenyi Mo, Tianyu Zhang, Yalong Bai, Bing Su, Ji-Rong Wen, and Qing Yang. Dynamic prompt op- timizing for text-to-image generation. In IEEE/CVF Conference Computer Vision and Pattern Recognition, CVPR 2024, Seattle, WA, USA, June 16-22, 2024, pp. 26617-26626. IEEE, 2024. doi: 10. N..109/CVVRR2223.224..0033 URL httss:/ /toi//oo.oo. 1011////0001110/////222 2024 02514.
Dong Huk Park, Samaneh Azadi, Xihui Liu, Trevor Darrell, and Anna Rohrbach. Benchmark for compositional text-to-image synthesis. In Joaquin Vanschoren and Sai-Kit Yeung (eds.), Proceedings of the Neural Information Processing Systems Track on Datasets and Bench- marks 1, NeurIPS Datasets and Benchmarks 2021, December 2021, virtual, 2021. URL https: / hwwpsstttebeecccaaa---peoeeeeeee wtw.rrooooiiinnniiiiiiieeeeeeeee neurips hww.iins...erris...peeeeeeeeee wttppppppp/p.......... hash/ httaa///....c4eeee00022220002110
www.:oo--bbbttatt.t.c.o roundl, html.
Pablo Pernias, Dominic Rampas, Mats Richter, Christopher Pal, and Mare Aubreville. Würstchen: An efficient architecture for large-scale text-to-image diffusion models. In The Twelfth Interna- tional Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL https: /ttp:::s/////peeeeeeeeee/ http://ww.eeetoooo/ddeeoo/o// -USBsdee...
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhom,, Jonas Müller, Joe Penna, and Robin Rombach. SDXL: improving latent diffusion models for high-resolution image synthesis. CoRR, 020/200700192, 2023. doi: 10.48550/ARXIV22307.00
URL https:
48550 00/55000500
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, CVPR 2022, New Orleans, IA, USA, June 18-24, 2022, pp. 10674-10685. IEEE, 2022. doi: (LL09//CVPR2268.22222000 org/10 1111111/022/66600 2022. 01042.
URL https:
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L. Den- ton, Seyed Kamyar Seyed Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, Jonathan Ho, David J. Fleet, and Mohammad Norouzi. Photorealis- tic text-to-image diffusion models with deep language understanding. In NeurlPS, 2022. ec795
URL http://papers http://ppppss.ip..ccppppapeepeee wtteeeefiilleeepppeeeeeee html.
Yang Song, Conor Durkan, Tain Murray, and Stefano Ermon. Maximum likelihood training of score-based diffusion models. In Marc' Aurelio Ranzato, Alina Beygelz- imer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (eds.), Advances in Neural Information Processing Systems 34: Anmual Conference on Neural Infor- mation Processing Systems 2021, NeurlPS 2021, December 6-14, 2021, virtual, pp. 1415-1428, 2021. URL https://prooeeiiieeeeee httpsniss.ppppppppeeeeeeeea.eeee 0a9fdbb1 feebcccb T -bsst222222-bb-baccccaaaa
Abst html.
14

---

Published as a conference paper at ICLR 2025
Kolors Team. Kolors: Effective training of diffusion model for photorealistic text-to-image synthe- sis. arXiv preprint, 2024. URL https:: //g1 //ttttbb...///////////////..hhtt com/ Kwai Mamai-Koloree Kooaaooo///////////bbboo master/ /mgg/////////// Kolors paper. pdf.
Tristan Thrush, Ryan Jiang, Max Bartolo, Amanpreet Singh, Adina Williams, Douwe Kiela, and Candace Ross. Winoground: Probing vision and language models for visio-linguistic com- positionality. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pp. 5228 5238. IEEE, 2022. doi: 10. (U09CCVPR5226822222.001771 URL https: http//doiooooooo 11t.:/....10/////28..8222 00517.
Bram Wallace, Mcihua Dang, Rafael Rafailov, Linqi Zhou, Aaron Lou, Senthil Purushwalkam, Stefano Ermon, Caiming Xiong, Shafiq Joty, and Nikhil Naik. Diffusion model alignment us- ing direct preference optimization. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2024, Seattle, WA, USA, June 16-22, 2024, pp. 8228 8238. IEEE, 2024. doi: 10.1 10.110//CV227333000000777 URL https: //w.//iooooo org. 1109 CVPR52733. 202 60786.
Tan Wang, Kevin Lin, Linjie Li, Chung Ching Lin, Zhengyuan Yang, Hanwang Zhang, Zicheng Liu, and Lijuan Wang. Equivariant similarity for assionllaguaan foundation models. In IEEE/CVF International Conference Computer Vision, ICCV 2023, Paris, Frunce, October 1-6, 2023, pp- 11964-11974. IEEE, 2023. doi: 10. 00.11090CCV11100222...0000 10. 10.1109/IcCV5100 2023 01102.
URL https::///deec org/
Zhenyu Wang, Enze Xie, Aoxue Li, Zhongdao Wang, Xihui Liu, and Zhenguo Li. Divide and conquer: Language models can plan and self-correct for compositional text-to-image generation. CoRR, 0tls/2401..1688, 2024. doi: 10.4850/ARXI.24001156888 URL https: http://d.ooo///// org// 10. 485508550// arxiv 2401. 15688.
Xiaoshi Wu, Keqiang Sun, Feng Zhu, Rui Zhao, and Hongsheng Li. Human preference score: Better aligning text-to-image models with human preference. In IEEE/CVF International Conference on Computer Vision, ICCV 2023, Paris, France, October 1-6, 2023, pp. doi: D0.100//CCV110000000000 202 00200.
IEEE, 2023. URL https:tt////...... //t::///oo..o org/ 10. 1109 ICCV51070.
Ling Yang, Zhaochen Yu, Chenlin Meng, Minkai Xu, Stefano Ermon, and Bin Cui. Mastering text- to-image diffusion: Recaptioning, planning, and generating with multimodal 1lms. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024. URL https: openreview. net bete/forrae
Chang Yu, Junran Peng, Xiangyu Zhu, Zhaoxiang Zhang, Qi Tian, and Zhen Lei. Seek for incan- tations: Towards accurate cot--to-maaee diffusion synthesis through prompt engineering. CoRR, abs/2401.006355, 2024. doi: 1048550/ARXXV.2400.0033
URL https: //t:///dooo //w.////////////////// org/10.
18550/arke 20...000023..1 06345.
Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Va- sudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, Ben Hutchinson, Wei Han, Zarana Parekh, Xin Li, Han Zhang, Jason Baldridge, and Yonghui Wu. Scaling autoregressive mod- els for content-rich text-to-image generation. Trans. Mach. Learn. Res., 2022, 2022. URL https: /ttps:///ppereeveww net/ (oorumidd-ADDyUUEE..
Mert Yüksekgönül, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou. When and why vision-language models behave like bags-of-words, and what to do about it? In The Eleventh International Conference Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL https:/ https:///ppeeevieee net forum? umdid= KRLUvxhBuaX.
Xinlu Zhang. Yujie Lu, Weizhi Wang, An Yan, Jun Yan, Lianke Qin, Heng Wang, Xifeng Yan, William Yang Wang, and Linda Ruth Petzold. Gpt-4v(ision) generalist evaluator for vision- language tasks. CoRR, abs/2311.01361. 2023, doi: 10.4850/ARXX/22111.0033 URL https: / //doi.org/// 48550 2311 01361.
15

---

Published as a conference paper at ICLR 2025
Xiangru Zhu, Penglei Sun, Chengyu Wang, Jingping Liu, Zhixu Li, Yanghua Xiao, and Jun Huang. A contrastive compositional benchmark for eet-to-imaae synthesis: A study with unified text-to- image fidelity metrics. CoRR, abs/23 2023. doi: 1044550/ARXXVV2212223338 URL
https: / /doii. .org/10..... 0tt//10.855500 48550/ aww.warr...v.... 231.. 02338.
16

---

Published as a conference paper at ICLR 2025
The Appendix is organized follows:
Section A provides detailed illustrations of four types of semantic variation results in T21 synthesis.
Section describes the data requirements, as well as the properties of the text-image align- ment function and the Sem VarEffect score.
Section C details the construction process of the benchmark.
Section D presents the implementation details of the evaluation.
Section presents more experimental results.
Section F provides further analysis of the results.
Section G visualizes more successful and failed examples in the evaluation.
Section H discusses the limitations of our evaluation and benchmark.
A FOUR TYPES OF SEMANTIC VARIATIONS RESULTS IN T21 SYNTHESIS
The results of semantic variations in T2I synthesis, both in text and images, can be divided into four types, as shown in Fig. 10.
Image Changing Semantics with Text Changing Semanties: The consistency between the input and output in the first quadrant suggests that the model tends to understand the different meanings introduced by linguistic permutations. In this case, the value of semantic variation (Ta/) tends to approach 1.
Image Maintaining Semantics with Text Changing Semantics: The inconsistency be- tween the input and output in the fourth quadrant suggests that the model does not under- stand the different meanings introduced by linguistic permutations. In this case, the value
semantic variation (Twy) tends to approach
Image Changing Semantics with Text Maintaining Semantics: The consistency between the input and output in the third quadrant suggests that the model tends to understand the similar meanings introduced by linguistic permutations. In this case, the value of semantic variation (Nw/a) tends to approach O.
Image Maintaining Semantics with Text Maintaining Semantics: The inconsistency between the input and output in the second quadrant suggests that the model does not understand the similar meanings introduced by linguistic permutations. In this case, the value of semantic variation (Twoo) tends to approach 1.
B PROPERTIES OF SEMVAREFFECT
B. PRELIMINARY
T2I generation model / consists one or more text encoders, image generators, and an image decoder. The T2I generation model f generates images I E I for each input textual prompt T E T. represents the textual space, and I represents the visual space. S(T, I) denotes the alignment score between 7 and T.
Let T. be an anchor textual prompt. Let T,. represent a permutation Ta, where Tp is a permuta- tion with meaning different from T., and Tpi is permutation with the same meaning T.. Let I,, Ips and Ipi be the resulting images generated by a T2I model from T., Tp and Tpi respectively. We expect that I,. will be rearrangement of the objects or relations found within Ia.
B.2 TEXTUAL VS. VISUAL SEMANTIC VARIATIONS
The measurement of semantic variations in the transition from (Ta, Ia) to (Tpe, can be defined from two perspectives: (1) textual semantic variations ,., which refer the semantic changes in
17

---

Published as a conference paper at ICLR 2025
Image Semantic Changes
An emerald top hat with blue gems
A cat with black fur is chasing red ball.
T21 model
T21 model
permute
permute
Blue gems
A cat with red fur chasing black ball.
T21 model
T21 model
emerald
top_ha hat.
Text Semantic Remains
Text Semantic Changes
The adult is frowning and the child is laughing.
cat is chasing mouse.
T21 model
T21 model
permute
permute
The child is laughing
mouse is chasing a cat.
the adult is frowning.
model
model
Image Semantic Remains
Figure 10: Four types of semantic variation results in T2I synthesis. The images with red borders represent incorrect outputs, while those with green borders represent correct outputs.
texts, observed through images La and Ip=, and (2) visual semantic variations r', which refer to the semantic changes in images, observed through texts Te and Tp-.
Specifically, define the textual semantic variations observed in single image J. We decompose complex semantic variation into minimal discrete steps, called localized changes. For each sentence T, the textual semantic variation of single minimal discrete step T + AT, denoted pr I), is the difference in alignment scores: Jr((T, I) = S(T + AT, 1) S(T, 7). When the anchor sentence To transforms permutation sentence Tp- through these minimal discrete steps, the integrated
textual semantic variation is: ΣTr pr (T S(T 1) S(T,, I). Therefore, the summation of textual semantic variations is defined as 7" EieLL. Σ n[
Similarly, define the visual semantic variation observed in single sentence T. For each image I, the visual semantic variation of a single minimal discrete step I + AI, denoted as μur (T, I), is the difference in alignment scores: 4T I) S(T, A/) S(T. I). When the anchor image I, transforms to a permutation image Ip- through these minimal discrete steps, the integrated visual semantic variation is: Er Mr I) S(T, IT.)) S(T,I,) Therefore, the summation of visual semantic variations is defined as - ETE(T, Σt. ((T,1)
We have not evaluated the influence by measuring the synchronicity of semantic changes between images and text, which has been applied VLM (Wang et al., 2023). This because semantic variations introduce unique challenge in evaluating T2I synthesis: images not independent; they are influenced by both the input textual prompt and the inherent characteristics the model, complicating the independent assessment of semantic changes.
18

---

Published as a conference paper at ICLR 2025
Therefore, conduct the evaluation by measuring the influence of external interventions seman- tie variations within the corresponding generated images from T2I synthesis models, while avoiding the direct imposition of interventions the images.
B.3 PROPERTIES OF ALIGNMENT SCORES S
Definition of Text-Image Alignment Score. To facilitate semantic analysis, we structured these permutations by objects and triples. Changing the word order affects the arrangement of objects or relations and leads to changes in syntactic dependencies and semantics. Let Tp. represent any permutation of Ta- Тa and Tp- share the same set of objects, denoted as V, and the same set of relations, denoted as R. The triple set E in Ta is a subset of V x R x V. Some triples in T,. may differ from those in To. but they have the same number of triples. For example, the initial triple set of To contains (apple, on, box), (girl, touch, apple) and (girl, NULL, box). After swapping box and apple, the triple set of Tp= contains (box, on, apple), (girl, touch, box) and (girl, NULL, apple).
We define the alignment score S between and as the sum of fine fine-grained alignment scores for objects and triples:
S(T 1)
Σ Satj (T, Stri, (T;
(5)
where [V] is the number of objects in T and [E| is the number of triples in T. The components of the alignment score defined as piecewise functions:
Sonj. (T,1) =
Wos
if the i-th object matches, if the i-th object does not match,
0
(6)
Wey
the j-th triple matches, the j-th triple does not match.
Serig (T,1)
where Wey and Wey are the weighted matching scores for the i-th object and j-th triple, respectively. We obtain the aligmment function S that satisfies the constraints in Eq. 10. Consequently, the alignment score of a matched text-image pair calculated as follows:
S(Tp,, Ip.) = laT
Σ We.
Σ
Weyt
..EVMA
cyCEMA
where VMA and EMA represent the exactly matched objects and triples between a text prompt and its generated image, with VMrll= = |V] and [EMal= |E), respectively. The alignment score for mismatched text-image pair is calculated as follows:
S(Tp.. Ip.)
Σ eiEVMr
Σ
Wey
(8)
cyEEMI
where VMI and EAI represent the partially matched objects and triples between text prompt and a mismatched image, with VMrl= = |V| and < [EMrl < [E].
Range of S. If accurately depicts the text through images and S faithfully measures the semantic changes between text space and the image space, then any alignment score S(T, I) is bounded by:
Σ VVV
We S S(T,T)
Σ Wv, +
Σ Wey-
CSEE
In our implementation, we set the value of S(T, 7) as an integer between O and 100, where the object accuracy ranges from 0 and 50 and the triple accuracy ranges from 0 and 50. We then normalize into a real number within the range [0, 1]. Based on the assumption of mentioned above, we have 0.5 < S(T, < 1.
However, limitations in the capabilities of the model f(-) and the alignment function S(.). often prevent the alignment score values from achieving the property outlined in Eq. 10. For example, if a model generates a low-quality image I, it may fail to accurately depict all target objects (V), leading to Vl < V| and [Vrrl < [V.. This results in object accuracy below 0.5 (as illustrated in the bottom case of Fig. 25) and inconsistent relation accuracy (see cases in Fig. 20 and Fig.
19

---

Published as a conference paper at ICLR 2025
21). Furthermore, inaccuracies in the scoring approach S(.) may incorrectly assess the similarity between text prompts and generated images, causing unpredictable fluctuations in semantic variation measurements (as illustrated in Fig. 27).
Identity Relation for S. Under ideal conditions where f accurately transforms all semantic varia- tions from text space image space, the alignment scores would satisfy the following constraints:
S(T., 1a) = (T 4p-) and L.) = S(I.,
(10)
Eq. 10 also demonstrated under the assumption that the aligmment function an equivariant map in the continuous textual feature space T and visual feature space I, as detailed in (Wang al., 2023). This assumption ensures that the alignment scores vary consistently with the semantic changes in images or text. This property is crucial for determining the characteristics of the data in VerBeehench and for designing the alignment functions.
However, the luw-quality image 1, resulting from poorly performing model f. and the inaccura- cies the scoring approach S, often prevent the alignment scores from meeting the criteria in Eq. 10. These limitations make the direct comparison of textual and visual semantic variation scores unreliable in T2I synthesis. This approach was previously explored in (Wang et al., 2023).
B.4 PROPERTY OF VISUAL SEMANTIC VARIATIONS 7
We analyze the theoretical relationship between visual semantic variations and model performance. According to Eqs. 1, 7 and 8, we derive the visual semantic variations as follows:
+
Wes
(11)
I [VM VArr)
CJE(EMA-ENM EMr)
For both permutation-variance and permutation-invarianee the value of Ev. E(VMA -Vax) We; re- mains constant, which we denote as C1. As result, the visual semantic variation can be simplified to 7 2 C1 + Ee,e( (
Wey primarily depending the size of the set EMA- EMI- However, the size of the set varies dramatically between the two settings:
In permutation-variane
settings (Ta, T,o), the optimal value for the set EMA EMP is its maximum set E, resulting in an obvious positive correlation between visual semantic variation /// and model performance.
permutation-invarianee settings (Ta: Tp.); the optimal value for the set EMA - E its minimum set 0, resulting in obvious negative correlation between visual semantic variation Twjo and model performance.
Therefore, we conclude that visual semantic variations in permutation-variance and permutation- invariance settings differ significantly.
A higher Tw/ value indicates that the model effectively captures and reflects the intended semantic variations in the input text.
A lower Mafo value indicates that the model maintains semantic consistency in the images despite variations in the input text.
B.5 PROPERTY OF SEMVAREFFECT SCORE K.
The Sem VarEffect score K, is defined as the difference between two types of visual semantic vari- ations score Tu, and Tur quantifies the model's ability to discriminate between significant and negligible semantic changes the text. Under ideal conditions, k ranges from 0 to 1. x max- imized when /u, reaches its upper bound of 1, which occurs in extreme cases where no relation
between objects are identical, and I Wwjo reaches its optimal value of 0.
If x is large, it suggests that the model is sensitive to semantic changes, recognizing vari- ations in meaning. However, this does not necessarily imply strong alignment. The model might detect changes in semantics but still struggle to fully capture all objects and relation- ships described in the text, indicating gap between sensitivity and complete alignment.
20

---

Published as a conference paper at ICLR 2025
If A is small close to zero, it suggests that the model either fails to reflect meaningful semantic changes or overreacts to minor text variations. Regardless of the overall alignment score, the model may generate similar images even in the presence of significant semantic differences in the input text.
C CONSTRUCTION DETAILS
C.I DATA COLLECTION
Seed Sentence Pairs from Winoground
Templates & Rule
Ta: [Noun]] [Verb (vt) [Noun2]
caption.00 OO: a cats snake caption.1: a snake eats bird
T [Noun2] [Verb (vt) [Noun]]
T. Tpu: [Noun]] [] <- [Noun2]
Ta: [Noun]] [erb (vi)] Prepositional Phrasel (location))
which is in [Prepositional Phrasc2 (location)]
caption.0: OO: a person is in helicopter which is in a car
Tpv: [Nounl] I] [Verb (vi) [Prepositional Phrase2 (location)
caption 1: a person is in car which is in helicopter
which is [Prepositional Phrase I (location)]
Ta Tpe: [Prepositional Phrasel (location)
<> [Prepositional Phrase2 (location)
Ta ([Prepositional Phrasel (location)). >There [Noun]
caption O: there are some pineapples in boxes, and far more pincapples thun boxes caption. 1: there are some boxes containing pineapples,
llocate in] [Noun2]. and far more [Nounl] than [Noun2]
Tp Tpv: ((Prepositional Phrasel I (location)), XThere [Nou2]
[contain] [Noun and far more [Noun2] than [Noun]]
and far more boxes than pincapples
T. Tpr: [Noun]] <>> [Noun2]
Ta: [Nounl] (which) [Verbl (vi) (eebb (vt) [Noun2]
cappioo_.. the person sitting down supporting the person standing up caption. 1: the person standing up supporting the person sitting down
(which) [Verb2 (vi)]
Tpu: [Noun]] (which) [Verb2 (vi)] [Verb (vt) [Noun2]
(which) [Verb] (vi)]
T. Tps: [Verbl (vi)] ++ [Verb2 (vi)]
T.: [Noun]] PPrepssition Phrase I/Relatvve Clausel
(appearance)) |erbll (vi)] slowly and [Noun2] (Prepositional
caption.0: O:: the person with green legs is running quite slowly and the red legged one runs faster caption.1: 1: the person with green legs is running faster
Phrase2/Relative Clause2 (appearance)] [Verh2 (vi)] faster
T pp: [Nounl] [Prepositional Phrasel/Relatie Clausel (appearance)) [Verbl (vi)] faster and [Noun2] [Prepositional Phrase2/Relative Clause2 (appcarance)] [Verb2 (vi)) slowly
and the red legged one runs quite slowly
To -> Tpu: slowly <+> faster
Table 5: Examples of extracted templates and transformation rules between templates of (Ta, Tpu).
Template Acquisition We designate 171 compositional cases in Winoground (Thrush et al., 2022), labeled as "no-tag" in subsequent research (Diwan al., 2022), and refer to them as SEED, and SEED, The template of Tp. the permutation with semantic changes, is extracted from each pair of seeds by human annotators. Then, we define the rule of Tpi. which is the permutation without semantic changes, as the original template of T Tpi- An example is illustrated as follows.
Ta: [Noun]] [Verb] [Noun2] and [Noun]] | behind [Noun3]
Ty.: [Noun3] [Verb] [Noun2] and [Noun3] behind [Nounl] Ty:: [Nounl] behind [Noun3] and [Nounl] [Verb] [Noun2]
T. Tp:: [Nounl] ++ [Noun3]
T. + Tps: [Noun]] ] [Verb] [Noun2]=+[Nounl] behind [Noun3]
If there is no coordinating conjunction such as and or while in the template of Tp the template can be set to NULL., In this case, the permutation Tps will be generated based on the LLM according to other solutions.
Ta: [] [Verbl (vi)] [Verb (vt)) [Noun2] [Verb2 (vi)) Tpo: [Noun]] [Verb2 (vi)) [Verb (vt) [Noun2] [Verbl (vi)]
Tp:: NULL
T, + Tpu: [Verbl (vi) <> [Verb2 (vi)]
T,
T.s: NULL
21

---

Published as a conference paper at ICLR 2025
Template-guided Generation for Ta- The prompt for generating Ta which is guided by the tem- plates and seed pairs, is as follows:
Assuming you are a linguist, you have the ability to create a similar sentence following the structure of given sentences.
The given sentences (SEEDo) and (SEED:). The structure of them are both "(Template Ta Please create similar "[ "[Template Ta sentence "TEXTO", and diversify your sentence much as possible by using different themes, scenes, objects, predicate, verbs, and modifiers.
Output a list containing (NUM) json objects that contain the following keys: TEXTO. Use double quotes instead single quotes for key and value. Now, let's start. The output json object list:
Rule-guided Permutation for Tpu. The prompt for generating Tpo based on Ta and its correspond- ing rule, is:
Assuming you are linguist, you have the ability to judge the structure of existing sentences and imitate more new sentences with similar structure but varied content.
Step I: Input some sentences structured by {Templlte T.} and (Template T. We call each sentence "TEXTO".
Step For each "TEXTO", perform the change which is "(RULE of Ta + Tpo and keep the other words unchanged as "TEXT1".
For example, TEXTO= (TEXTO). Only swap/move {RULE of To > Tpu and keep the other words unchanged generate (TETI=-
Output a list containing (NUM) json objects that contain the following keys: TEXTO, TEXTI. Use double quotes instead of single quotes for key and value. Now, let's start. The input is: (TEXTO). The output json object list:
Paraphrasing-guidee Permutation for Tp-- The prompt for generating Tpi based Ta and its corresponding rule, is:
[Instruction]
Please generate sentence that has similar length and meaning in the following six ways:
1. Change the word order: For example, "a red and yellow dog" be changed to "a yellow and red dog." In some languages, adjusting the order of words sentence the meaning. For instance, like you" can adjusted to "You are the person like".
new sentence form without changing
Passive voice: For example, "a kid flying yellow kite" can changed to "a yellow kite being flown by a kid."
For example, "a boy is playing with a girl" can he changed hy paraphrasing and altering the sentence structure to "a boy is playing. He is near girl."
3. Change the deseription:
Use synonyms: Replace words in the sentence with their synonyms. For example. "happy" can be replaced with "joyful".
Use infinitive or gerund forms: example, likes run" can be changed "He enjoys rumning". Simplify or expand: You can either simplify the sentence structure or add additional information to create a new sentence. For example, "The quick, brown fox jumps over the lazy dog" can be simplified to "The fox jumps over the dog". or expanded to "The fox, which is quick and brown, jumps over the lazy dog".
Now, please generate similar sentence for input prompt given the end. Provide sentence for each of the six methods. If sentence cannot generated using a particular method, please output "None". Add the results as a list of JSON objects, containing JSON objects. Each object should include the keys: number, modification method, and sentence.
[Prompt] "(TEXTO3"
C.2
DATA ANNOTATION
Criteria for Valid Samples. The primary challenge in annotation lies defining the criteria for what qualifies "valid". For T2I synthesis models, we define "valid" input text based 14 specific criteria. First, we illustrate these criteria through examples T, and Tp.. Second, for Tpi we require it apply one of the six synonymous transformations defined in the prompt for generating
22

---

Published as a conference paper at ICLR 2025
Type
Valid Criteria
Example
<xx
Tx: Swinging the swing and off the metal chains.
Complete Expression
T pu: Swinging off the swing and on the metal chains.
x Tp A dimmer sun is shining on brighter object.
Basic
n: Swinging
the metal chains on the swing.
T. A brighter suil shining
dimmer object.
Clear and Concrete Objects
X
pe dimmer object is shined on by brighter sun.
T. An engineer builds a bridge.
Reasonable Semantics
T pui A bridge huilds an engineer.
pi: bridge built by an engineer.
T. There are more salads than burgers on the menu.
Visuilyy Depicted Elements
T Tpu There are more burgers than salads on the menu.
ps: There are less burgers than salads on menu.
Visualizable
7. The wave is moving laster and the fish is swimming slowly.
Static Scene or Multiple Exposure Scene
Tpu: The fish is swimming faster and the wave is moving slowly.
Tp The fish swimming slowly and the wave moving faster.
T. In the library, there are stack of books and some more magazines.
x
Moderate Details
Tyv: In the library. there are a stack of magazine and some more books.
Tp.: the library, there
magarines and stack of books.
T.: There are more ants than becs in the garden.
Quantifiable Comparison
pu: There are more hees than ants in the garden.
T Tp. There are less bees than ants in the garden.
Ta: A sharp knife is on a dull cutting board.
x
Modification Rules
Tpu A dull cutting board under shurp knife.
Discriminative
T Tp dull cutting board is under sharp knife.
Tu: The boat on the dock and the fisherman is on the pier.
Distinct Textual Semantics
Tpu: The boat on pier the fisherman is the dock.
Tp: The fisherman on the pier and the boat is the dock
T:: There's a delicious chocolate cake with a bitter coffee frosting,
x
Visually Distinguishable
Tpu There's bitter chocolate cake with delicious coffce frosting.
T Tp:: There's 's a bitter coffee frosting with delicious chocolate cake.
T:: There are more books than shelves in this library.
Item-Specific Scene
Tuu There
more shelves than books this library.
Tx:: There are less shelves than books in this librury.
Recognizable
Ta: A photographer wearing a camera strap with his lens the air and videogrupher wearing a tripod.
Tpu A photographer wearing tripod with his lens in the air and a
Iee--Spccfiic Character
videographer wearing camera strap.
Tx:: A videographer wearing tripod and
photographer wearing camera strap with his lens in the air.
X
T.: The soldier in the barracks cleaning equipment and the officer in the office is reviewing reports.
Tpu: The soldier in the barracks is reviewing reports and the officer
Attire-based Character
the office cleaning equipment.
Tp: The officer in the office is reviewing reports and the soldier
in the barracks is cleaning equipment.
Ta: The businessman wearing navy suit and red
Action-based Character
T Tpu: The businessman is wearing red suit and navy tic.
Tp The businessman is wearing red tie and navy suit.
Table 6:
Error Examples of LLM-generated permutation--asse sentences (Ta. Tpu Tpp)) and the
criteria they violate.
Tp. From semantic perspective, Ty must be strictly consistent with Ta. ensuring the consistency and accuracy of the entire dataset. We list these criteria and examples in the following.
Basic
Complete Expression: Both sentences should be complete and free of obvious linguis- tic errors.
- Clear and Concrete Objects: Both sentences must be clear and unambiguous, either contextually or inherently, and specifically describe tangible objects, avoiding abstract concepts.
Meaningful Sentence: Both sentences must maintain logical coherence within their respective contexts. The reasonable definition includes real-world plausibility or sce- narios typically seen as implausible in virtual or imaginative settings (like children's literature, animations, or science fiction), like flying pigs or dinosaurs piloting planes. For example, "a shorter person can reach a higher shelf while taller one cannot" is not reasonable in any world.
Visualizable
23

---

Published as a conference paper at ICLR 2025
Visually Depicted Element: Both sentences must convey visual elements, including objects, scenes, actions, and attributes, ensuring that the text prompts are visually depictable and that the image content is identifiable during evaluation.
Static Scene or Multiple Exposure Scene: Both sentences should be visually repre- sentable through images alone, without the need for video, audio, or other sensory inputs such touch and smell. Temporal aspects, procedures, and comparisons in test cases must be conveyable within the scope of single image.
A Moderate Level of Detail: Sentences should maintain a moderate level of detail with similar scales for objects and scenes. Excessive or mismatched scales can result in sentences that are challenging to depict. For example, comparing the quantity of books and magazines "in library" less suitable than comparing them "on table". Quantifiable Comparison: Comparisons in both sentences should be quantifiable, us- ing measures like counts, areas, or volumes. For example, "There are more students in the classroom than words on the blackboard" is difficult to compare quantitatively.
Discriminative
Following Permutation Rules: Generated samples Tpo must strictly follow the desig- nated manual template, including word swapping and movement.
Distinct Textual Semantics: Two sentences must have distinct textual semanties. Oth- erwise, the pairs are considered invalid.
Visually Distinguishable: Two sentences should be visually distinct, with a clear dif- ferentiation in the visual characteristics of the objects or scenes described. Subtle differences that require very close observation are not considered distinct visual dif- ferences.
Recognizable
Item-Specific Scenes:
Scenes in sentences should be identifiable, with key elements maintained for recognition. Otherwise, identification may be challenging. For in- stance, sentence describing library where "bookshelves outnumber books" might be unrecognizable, as we typically expect a library to contain many books.
hem-Speeiffe Characters: When sentence depicts a character through associations with specific items, these items or behaviors should remain consistent for easy identi- fication. Otherwise, the character may be hard to recognize. For instance, che are usually associated with "chef's 's attire, cooking utensils, and kitchens".
Attire-Basse Based Characters: When sentence presents characters identifiable by their at- tire, such as firef Fifighttess pol i ce officers, soldiers, doctors, and nurses, their clothing should remain consistent for clear recognition. Changes in attire could obscure their identities.
- Action-Based Characters: When sentence features characters defined by specific actions interactions, such as bartenders (mixing drinks), bus insssmen (ne- gotiating). journalis lists (intervicwing), divers (deep-sea diving), their typical activities should remain consistent. Altering distinctive features or placing characters in unusual scenarios may obscure their identities.
Automatic Annotation. We employ a machine-humant hybrid verification process filter out in- valid samples that violate any characteristic. We use LLMs to judge whether each sample violates any of the specific criteria, labeling them "yes" or "no" and providing confidence scores. The sam- ples whose confidence exceeds threshold of 0.8 are removed from the dataset. We initially col- lected 48K samples, each including 3 sentences. The automatic filtering helped eliminate over 42% of them, resulting in a final corpus of 27K samples.
Human Annotation. We use 15 annotators and 3 experienced experts to manually verify the sam- ples. All annotators have linguistic knowledge and are provided with detailed annotation guidelines. Each sample independently annotated by two annotators. Then, an experienced expert reviews the controversial annotations and makes the final decision. After annotation, randomly sampled 100 valid samples to assess annotation accuracy. Two experts evaluated that 99% of the samples were valid. Finally, we obtained 11,479 valid, non-duplicated samples.
Hard Samples Selection. To effectively evaluate T2I models, it is crucial to select challenging samples rather than simple ones. Initially, we generate images using SOTA models like DALL-ES 3,
24

---

Published as a conference paper at ICLR 2025
Category
Train
Test
Total
Relation
Absolute Locution
1,716
50
1,766
Relative Location
1,111
1,161
Action
216
48
264
Interaction
153
43
196
Direction
342
375
Spatio-temporal
234
50
284
Attribute Comparison
Vague amount
1,839
50
1,889
Size
2,168
3,118
Height
253
50
303
Weight
5
5
10
Attribute Value
Color
4,451
50
4,501
Appearance
1,972
Texture
542
50
592
Shupe
190
50
2401
Size
$16
$66
Material
227
50
277
Manner
194
243
Sentiment
114
Age
22
11
33
Temperature
14
Counting
614
50
664
Total
15,518
819
14,699
Totaldedduplication))
11,454
684
10,770
Table 7: Statistics of SemVar VarBenc.
and flagging those with alignment scores below 0.7. Then, we aggregate the votes from these models to determine the most representative candidates, and select those with the highest votes for further filtering. To ensure diversity, we categorize these samples based on permutation types, as shown in Fig. 5, with maximum limit of 50 samples per category. Finally, 684 samples were included in our benchmark.
C.3 DATA STATISTICS
Category. The samples VarBench are divided into 20 categories based on their permuta- tion types. Furthermore, these categories further classified into three aspects based triple types, as illustrated Tab. 24. These aspects are Relation, Atribute Comparison and Atrrbute Value. Specifically, Relation aspect includes six categories: Action, Interaction, Absolute Location, Relative Location, Spatial-Temporat Direction. Attribute Contrast includes four categories: Size, Height, Weight, Vague Amount. Attribute Value includes ten categories: Color, Counling, Texture, Material, Shape, Age, Sentiment, Temperature, Manner, and Appearance.
Scale and Split. VerVerhnnch comprises 11,454 valid samples of (Ta, (Ta,Tpe: Tp:). totaling 34,362 sentences. We divide it into a training set and test set. The training set contains 10,806 samples, while the test set consists of 648 challenging samples for effective evaluation, as shown in Tab. 7. All our evaluations are conducted on the test set.
Distribution. Since some permutations contain multiple words, they may fall into more than one category. In the training set, 51.06% of the permutations involve only one category, 35.12% involve two categories, 7.35% involve three categories, and 0.5% involve more than four categories. In the test set, 82.75% of permutations involve only one category, 14.77% involve two categories, and 2.49% involve three categories. As result, the total count of categorized samples exceeds the actual number of unique samples.
Sem VarBench vs. Other benchmarks. Compared with existing benchmarks, Sem VarBench fo- cuses on the understanding of semantic variations for text-to-image synthesis, which includes two types permutations: cermutation-varinnn and -ntuuuaaiiiooiivianceeee Other comparisons, such source, scale, annotation, and split, illustrated in Tab. 8.
25

---

Published as a conference paper at ICLR 2025
Benchmark
Concentration
Data Source
#Prompts
Annotation
Split
DrawBench (Sabaria et al., 2022)
General
Human
200
Human
Test
PariPromps (Yu al, 2022)
General
Human
1600
Human
Test
PaintSkills (Cho al., 2023a)
General
Template
73.3K
Train/Tsst
HRS-Bench (Bakr et al, 2023)
General
Templute & LLM
45.0K
Human
Test
SRai (Gokhale
2022)
Compositional
Dataset
25.3K
Test
ABC-6K (Feng et all,
Compositional
Dataset
6.4K
Test
CC-500 (Feng all, 2023s)
Compositional
Template
500
Test
TIFA v1.0 (Hu et 2023)
Compositional
Dataset
4.1K
Test
VPEval-skill (Cho al,, 2023b)
Compositional
Dataset
3.8K
Test
DSG-IK IK (Cho al., 2024)
Compositional
Dutaset
1.KK
Test
T21 Bel-ConpBench (Huang et al,, 2023)
Compositional
Template & I.I.M
6.0K
Train/Tes
Winoground (Thrush al., 2022)
Permutation Variance
Human
800
Human L.M Human
Test
Winoground-T21 (Zhu al., 2023)
Permutation Variance
Template L.LM
22K
Test
Permutation- Variance
Sem (orBench((urss)
Template & L.M
34K
LLM & Human
Train/Tsst
Permutation Invariance
Table 8: Comparison between Sem VarBench and other T2I synthesis benchmarks.
D DETAILS OF EXPERIMENT SETTING
D.1 T21 SYNTHESIS MODELS
We evaluate 13 mainstream T2I models as shown in Tab. 1. For each sentence, we generate one image, resulting in total of 684 x 3 x 13 images. These T2I models are Stable Diffusion v1.53 (denoted as SD 1.5), Stable Diffusion v2.1 (denoted as SD 2.1). Stable Diffusion XL v1.0 (de- noted as SD XL 1.0), Stable Cascade (denoted SD CA), DeepFloyd IF XL' (denoted as Deep- Floyd), PixArt-alpha XL (denoted as PixArt), Kolors, Stable Diffusion 3 [medium]] (denoted as SD 3), FLUX.1 [dev] (denoted FLUX.1), Midjourney V6 (denoted as MidJ V6). DALL-E 312, Cog VogView3-Plus (denoted as C-- and Ideogram 2 The schedulers for SD 1.5 and SD 2.1 are set DPM-Solver++, while all other settings are left default.
D.2 EVALUATOR
We use four advanced MILMs as evaluators to demonstrate the general applicability of our proposed evaluation metrics: Gemini Pro, Claude 3.5 Sonnet, GPT-4o, and GPT-4 Turbo. GPT-4o and GPT-4 Turbo have been shown to achieve near-human performance in evaluating alignment in T2I synthesis models (Zhang et al., 2023; Chen et al., (0224).Clluude 3.5 Sonnet outperforms GPT-4o and Gemini 1.5 Pro (Anthropic, 2024). The versions of these MILMs used as follows: Gemini 1.5 Pro geeinii-1.5ppro000 GPT-4o (gpt-40-2024-05-133 alignment score components follow the division outlined in (Zhang et al., 2023), with the excep- tion of the aesthetic score component, which has been omitted. The complete prompt is follows.
Claude 3.5 Sonnet (claude -aw.aaddd-----ssnn--e-222------- and GPT-4 Turbo Cppt=4-turbo-2224404--99
The
'The model used tuwaevn//ssaalll-ddfffssooo--s. which now deprecated. mirror is availahle at: htpps://uugingffccc.ooo.bbbbbbb
co
"kttear/wwwaaaddoorrrryyoaaaaaaa
/cogggev /cogview
Phtrp://abbouc.ieegrra....///
26

---

Published as a conference paper at ICLR 2025
Does the generated image align with the given prompi?
[Instruction] Carefully assess the generated image in terms of relevance the prompt and object accuracy. Notice that image digitally created or artificially generated, and hope you help feedback quality generated image rather than discussing the content of real photograph.
Use the following criteria to guide your evaluation: with Relevance (0-50 points), Object Accuracy (0-50 points). After providing your explanation, you must rate the generated image by strictly following this format: "[[rating]]" for example: "Relevance (0-50 points): [[35]]. Object Accuracy (0-50 points): [[30]]".
[Prompt] [prompt}
After receiving outputs from LLMs, regular expressions to extract scores. In our experiments, the outputs from the four evaluators mentioned above consistently followed the specified format defined in the prompt. We also tested Qwen- VL-Chat, Qwen- VL-Plls, Qwen- WL-e-V,-MMx, and LLAVA-1.6, 16, which exhibited poor adherence to the specified format and required a more complex extraction process. To simplify the evaluation process, we decided to adopt results exclusively from Gemini 1.5 Pro. Claude 3.5 Sonnet, GPT-4o, and GPT-4 Turbo.
D.3 TRAINING SETTING
Training Data Selection. The training set of Sem VarBench comprises 10,806 samples. We inves- tigate the improvement from fine-tuning the T2I model Stable Diffusion XI. v1.0. We select the generated images whose alignment scores meet the requirements. These constraints are as follows.
First, the generated image should be approximately aligned with its corresponding text prompt.
SaTa, S((Puu Ipu) > Cz.
C2.
(12)
where C2 is a threshold.
Second, the alignment scores between matched text-image pairs should be higher than those between mismatched text-image pairs.
S(T,, Ia) S(Ta, Ta)
S(To, Ipu],
> S(Tpu, Jal,
(13)
S(Tpe, Ipo)
S(To, Lpw)s S(Tpm, Ia),
S(Tpes Ipr)
Third, the visual semantic variations observed from different text prompts should be the same when the initial image and the final image are the same.
S(Ta, I.) Ipo) ss S(Tpv, S(T po: Ia):
(14)
Similarly, the textual semantic variations observed from different images should be the same when the initial text prompt and the final text prompt are the same.
S(T., Ia) S(T,.. S(T,., S(T., Ipn):
(15)
By utilizing this approximate equality relationship in Eq. 14 and Eq. 15, we constrain the alignment score using the following inequality:
(T. : I.) S(T., [p.)) (S(T Ipo) S(Tpu, ((T));aa)) la)) < Ca.
(16)
[SSTT. I.) S(Tp.: 1.)) (S(T Ipo) S(T., Ipo)]] < Ca.
In experiments, utilized Stable Diffusion XL v1.0 to generate image for each text prompt within the training set. To select the training data, we designated C2 = 0.8 and C. = 0.1. Uli- mately, selected 327 samples, resulting in 981 sentences.
Supervised Fine-Tuning (SFT). All text-image pairs (T,, I,) are incorporated into the training set. Each sample (Ta, Tpoy Tn:) results in three text-image pairs: (Ta, Ia), (Tpm: Iwp), and (Tpis Ipi),
27

---

Published as a conference paper at ICLR 2025
which leads to total of 981 diverse pairs. The selected set of samples is denoted as D,. The loss function for SFT remains unchanged (Kingma et al., 2021; Song et al., 2021), which is defined as
[)) (i..))DD [||e co(=,, t. w)1
(17)
where y, t, 2 are the representations of the image I, text prompt T; timestamp t, and the latent representation of the image at timestamp t, respectively. We conducted two separate fine-tuning processes using the diffusers library'5: We only fine-tuned the LoRA model either the UNet or the text encoder for 5,000 steps, with training batch size of 1. Our computational resources included an NVIDIA GeForce RTX 4090 with 25.2 GB of VRAM and 16-core AMD EPYC 9354 processor, with 60.1 GB of system memory available. We trained the LoRA model with a rank of on UNet or text encoders, and the training process took approximately 0.5 hours.
Direct Policy Optimization (DPO). We added text-image tuples of the form (Tú, 1,, 15) to the train- ing set, where the semantic content of T, does not match Tj. For each input T,, I; represents the chosen image and 1 the rejected one. Each sample (Ta. Tp. Ty:) results in four text-image tuples: (Ta; Ja, (po): (Tpe pe: Ia), (T pu: Ieu, Ipi), and (Tpi, Ipi, (, which leads to total of 1, 1,308 tuples.
The loss function for DPO remains unchanged (Wallace et al., 2024), which is defined as
(())= (E DD. -t (<"""""""""""""->""""""
logor
- -0(1e" co(2) t, v)ll - ||e" - En (z) t, v)11
(18)
(||e' - co( 4, w0l - ||e' - coe L, (.1..,...)))) 1)))))))
where x", x², y, t, 22", 2/, a are the representations of the chosen image 4t the rejected image Lj. text prompt Tü timestamp t, the latent representation of the chosen image at timestamp Ł, the latent representation of the rejected image at timestamp L, and the sigmoid function. We executed two separate tuninn processes using the DiffusionDPO 16 We only fine-tuned the LoRA model either on the UNet or the text encoder for 5,000 steps, with a training batch size of 1. Our computational resources included Tesla V100-SXM2 with 32 GB of VRAM and an 11-core Intel(R) Xeon(R) Platinum 8163 processor, with 88.0 GB of system memory available. We trained the LoRA model with rank of on the UNet text encoders, and the training process took approximately 4.5 hours.
E MORE EXPERIMENT RESULTS
E.
RESULTS ON CATEGORIES
15
10
SerVarEffect Score
SerVarEffect Score
0.0
-05
-055
-1.0
Aisulide Lacatton
Reaaive Location Aetion
Inter Direction
Spatis Tempora
Mague Arrount Height
Size
Weight
Color
Material Apprarand
Age
Tem per sture
Sentiment
Counting
Absuliste Locatton Relative pncation
Action
Spaile Tempural
Drevitin
Vagus Armunt
Size
Welght
Height
Color
Raterial
Tempin sture
Acee
Shape
Spottmmnn Manter Counting
Interacter
App
Figure 11: The distribution of Sem VarEffect scores across various categories for the Ideogram 2 and the DALL-E 3 model, as evaluated by GPT-4 Turbo. Left: Ideogram 2. Right: DALL-E 3.
15hhe diffusers library supports fine- tne-tuning of both the Unet and the Unet + text encoder. We made minor modifications to support tine-tuning only the text encoder. The url of scripts provided by diffusees is: https: //github. com/ wngggnngace///oo /aii susersstrres www.reeeaainn/iee /tt///an////eeeaa meaaii/examppeee / /tttttttttttttt/et//to///t///ttt Lattttig_ggeeeee We used the code provided by the diffusers library, which supports fine-tuning of the Unet. We made minor modifications to support tne-tuning only the text encoder. The url of scripts provided by diffusers is: https:/ //gtttttt/. thub. com/ com/hagginntieoo aagfae/ddeee di ffusers / rre/ma i n/ examples research research_projecttee frus on_dpo/
28

---

Published as a conference paper at ICLR 2025
Relation
Altribuic Comparison
Models
Absolute Location
Relative
Action
Interaction
Direction
Spatia Temporal
Size
Weight
Vague
Height
Open Source Models
Difrusion
-001
-1106
-0.11
Stable Diffusion v2.1
-0.01
-0.96
-0.08
0.02
-0.02
-0.00
0.03
-0.30
0.02
0.06
Stable Diffusion v1.0
-0.02
-0.08
-0.01
0.05
0.05
-0.05
0.07
0.16
0.03
0.09
Stable Cascade
0.02
0.01
-D113
-0.02
-0.09
-001
DeepFloyd IF XL
-0.01
-0.90
0.01
-0.01
0.04
0.01
0.03
0.05
-0.04
0.03
PixArt- alpha XL
0.00
-0.01
0.03
0.00
-0.04
-0.03
0.07
0.10
0.10
0.03
Kolors
-0.3
-0.07
0.02
0.05
-0.02
-0.06
-0.10
0.07
0.77
Stable Diflusion
-0.03
0.01
-0.02
0.05
-0.08
-0.04
0.07
-0.02
0.10
0.04
FIUX.1
-0.33
0.03
0.03
0.08
-0.04
-0.00
0.09
0.23
0.05
0.09
Models
Midjourncy V6
0.07
0.01
0.01
0.03
0.03
0.08
0.07
-0.12
0.07
0.02
DALL-E3
-0.00
0.12
0.11
0.08
0.13
0.11
0.08
-0.00
0.09
0.15
Co
-0.03
0.22
Idongram 2
0.01
0.04
0.13
0.29
-0.02
-0.02
0.12
0.04
0.17
0.17
Table 9: The results of Sem VarEffect scores A on aspects Relation and Attribute Comparison. The evaluator is GPT-4 Turbo.
Models
Attribute Value
AVG
Color
Material
Appearance
Age
Shape
Temperature
Texture
Sentiment
Mannet
Counting
Open Source Models
Stable Diflusion v1.5
0.09
0.02
0.04
-0.20
0..11
0.06
0.82
-0.06
-0.05
-0.07
40.01
Stable Diffusion
-0.00
Stablc Diffusion XL v1.0
0.13
0.09
-0.01
0.01
-0.01
0.05
-0.00
0.03
-0.00
0.00
0.02
Stable Cascade DeepFloyd XI.
0.14
0.05
0.10
-0.03
-0.02
-0.15
-0.03
-0.05
0.06
0.01
0.04
0.14
-0.19
0.84
-0,02
0.05
-0.05
0.06
0.04
PixAtt-alpha XL
0.11
0.09
0.80
0.15
0.01
0.13
-0.02
0.02
0.03
-0.00
0.02
Kolors Stable Diffusion
0.21
0.07
-0.01
0.01
-0.09
0.01
0.10
-0.01
-0.04
0.00
0.01
0.33
0.10
0.11
0.04
0.08
0.11
0.08
0.01
0.03
0.06
0.06
MUXX1
0.35
0.21
0.09
40.13
0.04
-0.06
0.07
0.10
API-bnsed Models
Midjoumey
-0.03
-0.21
0.11
-0.05
-0.02
DALL-E3 3
0.23
0.17
0.17
0.23
0.14
0.22
0.22
0.19
0.11
0.06
0.12
CogView3-Pee Plus
0.35
0.15
0.17
0.31
0.16
0.30
0.17
0.21
0.27
0.07
0.15
Ideogram
0.12
Table 10: The results of Sem VarEffect scores aspects Attibute Value. AVG represents the average effect score of all samples on aspect Relation, Attribute Comparison and Attrbute Value. The evaluator is GPT-4 Turbo.
Effects of Semantic Variations on Different Categories. The impact of semantic variations is not uniform across different semantic classes, shown Fig. 7, with exact scores listed in Tabs. and Tab. 10. For Relation, most models consistently show low scores, as indicated by the dark blue shading Fig. 7. This suggests that models handle samples involving Relations-sun Location, Relative Location, and Actions-
as Absolute limited accuracy. For Atribute Value, models such as Ideogram2 perform significantly better at capturing attributes such as Color, as shown by the prominent red shading in Fig. 7. These models demonstrate clear advantage in both generating and recognizing these attributes. In contrast, models such as DALL-E and CogV3-Plus display more balanced but average performance across most categories (shaded in light orange and light blue). For Attribute Comparison (e.g., Size, Weight, Height), most models score lower, indicating their weaker ability handle complex attribute comparisons.
Although most T2I models struggle with capturing semantic variations in many categories, some categories, such as Color and Age, demonstrate slightly better performance, as indicated by higher median values. Fig. 11 illustrates the distribution of Sem VarEffect scores across various categories for the Ideogram model, while Fig. 12 shows the scores for different T2I models the Color and Direction categories. Most categories have medians (marked by the orange line) close to zero, indicating that T2I models generally struggle to capture the semantic variations introduced by word order changes, particularly in the Direction category. However, some categories, such as Weight and Color, show slightly higher median values, indicating that semantic variation caused by word order changes may have minor positive effeet in these instances. Categories such Absolute Location and Counting show greater variability in model responses, while categories such as Sentiment and Texture show consistent effects with narrower distributions.
29

---

Published as a conference paper at ICLR 2025
1.00
0.75
SerVarEffert Scare
SemVarEffect Scare
0.50
0.5
0.25
0.00
-025
-05
000
1.5
2.1
5D
Despitedd
Piddet
Katand
503
Fuux11
And
DALL &
Copu PLUS
Ideogramz
SD
SD
XL
5D
Deapfleyd
Podert
Kolona
SD 3
Fuux.1
Mijl vi
DALLE 3
Cop
Idergrmm
50
50
Figure 12: The distribution Sem VarEffect scores across various T2I models within the Color and Direction categories, as evaluated by GPT-4 Turbo. Top: Color. Bottom: Direction.
E.2 RESULTS ON HUMAN EVALUATION
Human Evaluation. To validate the effectiveness of MLLMs used, conducted human evalu- ation with three raters on 80 samples (20 from each of the SOTA models: Midjourney v6, DALL-E 3, Cog View3-Pluss and Ideogram2) through stratified sampling (one for each category). Following the same scoring protocol as our automatic evaluation, each rater scores the semantic alignment of matched mismatched image-text pairs, and use their mean scores to calculate Sem VarEfffect. The results demonstrate the reliability of our MLLM-based evaluation approach. First, we observe consistent performance trends between human raters and the four MLLMs across all evaluated mod- els (Tab. I1). Second, our correlation analysis on Cog 3ogView3-Plus reveals moderate Pearson's 's P, Spearman's c. and Cohen's Kappa Keahen coefficients between machine and human scores (Tab. 12), suggesting that our selected MLLMs can serve validates our MLLM-based evaluation while confirming the current limitations of T2I models. The interfaces and criteria for human evaluation are shown in Fig, 13 and Tab. 13.
reliable proxy for human evaluation. This
Models
S(t)
7w((1
Two (1)
x(t)
Models
p(t)
o(*)
Reohen (f)
Midjourney V6
0.59
0.37
0.49
-0.12
GPT-4o
0.54
0.53
0.53
DALL-E
0.63
0.53
0.50
0.03
GPT-4y
0.50
0.51
0.54
CogViow3-Plus
0.52
0.33
0.19
Seude-3.5-Sommee
0.42
0.37
Ideogram
0.50
0.19
Gemini-Pro-15 Pro-115
0.27
Table 11: Human evaluation results of differ- ent T2I models in understanding semantic vari- ations.
Table 12:
Correlation coefficients between
GPT-4o, GPT-4v, Claude 3.5 Sonnet, Gemini Pro 1.5 and human evaluations of the Sem Var- Effect for Cog VogWe3w3-Plus.
F MORE ANALYSIS
F.1 TEXT ENCODER
Do different encoders themselves distinguish semantic variations caused by linguistic per- mutations in the text space? We explore the efficacy of diverse text encoders in discerning such nuances. Fig. 14 compares the text similarity between T, and Tpo across models utilizing different text encoder models. SD 1.5, SD 2.1, SD XL v1.0, and SC utilize CLIP series models as text en- coders, while DeepFloyd, PixArt, and DALL-E utilize T5 series models. The similarity metric is depicted cosine Tpu, with higher values indicating stronger ability of the text encoder to differentiate between the semantics the two sentences. This indicates that the choice of text encoder significantly influences the model's semantic discrimination capabilities.
30

---

Published as a conference paper at ICLR 2025
Group 0:
Pair 0:
Does the generated image align with the given text?
boorc Myy woy stpwint)
Text, The athlete with medal and
Input Text:
without medal applasds.
Tee,0: The athlol with medal colebrato and the atfiats without modallappludd.
Soorc tenenniccooouee pewnol
"Selectee" Image.
Output Image:
50
InagiRR
Submit Score
applouds and the sthiete withouta
celebrates.
Score
mage.
Submit
Text athinte thout medal appllaud and the athiata arrecal
2
Image
Figure 13: Interface for the human evaluation
Score Object Accuracy (0-50 points): The existence and attributes of objects.
50
Perfectly Accurate (All objects cxist, and their attributes are described correctly).
40
Highly Accurate (All objects exist, hut some minor details may differ slightly).
30
Partially Accurate (Most objects exist, but there are noticeable errors in appearance, color, or shape).
20
Mostly Inaccurate (Significant deviation from the description, with multiple attribute errors).
10
Severely Inaccurate (Most object details contradict the description).
0
Completely Inaccurate (Objects described are missing or entirely replaced by different objects).
Score
Relevance (0-50 points): Relationships, positions, and interactions between objects, Perfect Match (Object positions and interactions fully align with the description).
50
Highly Matched (Most object positions and relationships align, with minor mismatches that do not affect overall understanding).
40
Partially Matched (Key objects exist, but there are significant mismatches in position relationships, leading to a slightly different overall context).
20
Mostly Mismatched (Object relationships do not align with the description). Severely Misaligned (Most object relationships are chactic).
10
0
Completely Mismatched (All Object relationships are completely different from the description).
Table 13: Criteria for the human evaluation
action
2.1
color
3D
Interaction
005
amount
appearance
manner
DeepFloyd
Deeofloyd
PoWrrt
Plwort
absolute
weight
counting
temperature
location
relative location
height
texture
sentiment
spatlo temppoal
size
material
age
direction
shape
Figure 14: The semantic discrimination capabilities of different text encoders measured by 1 cosine p.
31

---

Published as a conference paper at ICLR 2025
Category
(Ta, Tpo)
Tw))
(To, Trandom)
Relation
Absolute Location
11.94
27.24
52.40
Relutive Location
12.26
28.62
46.98
Action
13.94
31.85
48.35
Interaction
13.56
29.58
44.26
Direction
12.03
27.18
50.21
Spatio-temporal
17.40
42.74
59.94
Attribute Comparison
Vigue amount
19.38
36.56
50.38
11.38
26.00
46.82
Height
13.04
23.00
31.22
Weight
10.00
26,20
22.20
Attribute Value
Color
11.80
30.86
46.90
Material
12.40
Appearance
13,86
44.14
59.14
Age
14.73
34.73
46.64
Shape
13.34
33.48
40.98
Temperature
11.00
27.50
38.50
Texture
11.74
31.20
54,22
Sentiment
11.96
33.15
48.65
Manner
13.37
33.71
$2.90
Counting
8.44
29.06
44.18
Average
54.30
Table 14: The average edit distance between sentences in different categories.
Why do permutations without semantic changes exhibit higher text similarity scores compared to those with semantic changes? This phenomenon is closely related methodology, where Tpi is generated by swapping two long phrases located on either side of a coordinating conjunction
dataset's 's construction
predicate, such the and in Fig. 4. We observed that permutations with semantic changes in our benchmark have significantly smaller edit distances from the anchor sentence compared to synonymous sentences, shown in Tab. 14. The average edit distances Tpi) and (Ta, T Trandom) are 13, 32, and 53. As our analysis does not rely on
between (Ta, Tpe), (Ta, the similarity scores of synonymous sentences, this does not affect our previous findings.
F.2 EVALUATION METRICS
F.2.1 ALIAGNMENT SCORE SEMVAREFFECT SCORE
Detailed Analysis on Text-image Alignment Scores vs. SemVar VarEffect Score Fig. 15 illustrates that although the distributions of the SerVarEffect score and the alignment score are similar, the Sem VarEffect score demonstrates higher degree of differentiation, especially when distinguishing between FLUX.1 1 and SD Based on the alignment score, it could be concluded that FLUX.1, SD 3, and SD XL 1.0 have comparable performance levels and may be grouped into the same cluster: However, based on the Sem VarEffect score, it becomes evident that FLUX. I and SD differ distinctly from SD XL 1.0. SD XL 1.0 responds more similarly to semantic variations caused by word order changes in a manner similar to SD 1.5, SD 2.1, and SD CA. Correspondingly, we observe that when using the T5-XXL series model the text encoder, the difference between DALL-E and other models, such as PixArt and DeepFloyd, becomes more pronounced when assessed by the Sem VarBffect score.
32

---

Published as a conference paper at ICLR 2025
0.25
CUIP Setes Text Encoder Hybri text Encoder
DALLES
CUI Series Encoder
CLINTS Hyond texe Encoder
15 Sor es Text Enooder
13 Serics Taxt Encoder
0.60
ChotGLM Text Encoder
SemVarEffect Score
ChoeGLM Text Enooder
Alignment Score
FLUK
50
0.75
SD
Deed bayd
Kolors
2.0
PSCAT DespFloyd
SD CA
XL
0.65
SD
9D 1.5g
50 CA
3.92
0.54
Cosine Similarity
Cosine Similarity
Figure 15: A comparison of alignment scores and the Sem VarEffect score under the same conditions of text similarity. The squares are results of permutations of permutation-Vee variance. The evaluator is GPT-4 Turbo.
It should be noted that Sem VarEffect is not intended to challenge alignment evaluation, but rather to serve as a complementary metric to alignment score S. This metric focuses only on the consistency of variations rather than the absolute quality of generations. Thus, model could still score well SemVar Faffecttt even consistently generates incorrect attributes (such as believing oranges are blue maintains consistent semantic variations. Sem VarEffect helps
and bananas are green), as long validate the reliability of high alignment scores S: high with low к indicates limited semantic understanding, while high S with high x suggests effective semantic understanding.
F.3 TRAINING DATA
F.3.1 DATA FILTER CHOICES
How do the filtering standards of training data using MMLM potentially affeet the evaluation results, particularly for categories with limited samples? We examine filtering standards across data-rich and data-limited categories study the effects of data scale and data quality. The filtering criteria are follows:
Alignment Score Thresholds: Values of 0.8 and 0.6 for C2 (defined in Eq. 12)
Existence of Strict Filtering Semantic Variations: Clear distinction between matched and unmatched text-image pairs through strict filtering criteria (defined Eq. 13-16).
High filtering. High filtering standards reduce the available data in categories with limited data, leading to decreased alignment scores and Sem VarEffect scores. Given the same initial data size, Absolute location has lower alignment scores (> 0.6) than Heigh//Direction (> 0.7) due to stricter filtering effects, as shown in Tab. 3. However, due to its large initial data pool (1.7k candi- dates), cissolutelocatiinn retains sufficient high-quality samples for effective fine-tuning under the "0.8+strict" setting, despite the low retention rate, as shown in Tabl. 15. This comparison between categories with different data scales demonstrates that initial data volume is crucial for achieving improvements.
Relaxing filtering. Results show that relaxing filtering criteria allow more training data leads to worse performance compared to the strictest standards. Removing "0.8+strict filtering" leads to decreased performance across all categories, even those with sufficient data like Color and Abso- lutetton as shown in Tab. 15. For categories with limited data, "0.8+strict filtering" minimizes the decrease in both alignment scores and semantic variation effects, as shown in Tab. 16. This demonstrates that even with limited data, lowering filtering standards is not viable solution.
F.3.2 COMMONSENSE BIASES
Is the model's insensitivity to word order variations caused by commonsense biases in training data? To investigate this question, we conducted controlled experiments with balanced training data containing both commonsense and anti-commonsens. examples, and evaluated the model's generalization to novel object pairs. For training. we created a balanced dataset consisting of 40 plausible images for "cat chasing mouse" (human-selected from DALL-E 3 generations) and 40 anti-commonsense images for "mouse chasing cat". The anti-commonsense images were manually created, where 31 images were generated via DALLL-S and professionally edited by designers, and
33

---

Published as a conference paper at ICLR 2025
Filter Constrains
S(t)
Tw(1)
Two(4)
x(1)
Filter Constrains
S(t)
Tw(1)
Two(4)
<(t)
zeroshot
0.73
0.33
0.25
0.08
zeroshot
0.64
0.29
0.34
-0.05
0.8 strict filtering
0.78(1)
0,44(1)
0.24())
0.20(1)
0.8 strict filtering
0.65(1)
0,42(1)
0.35(1)
0.07(1)
0.8 no filtering
0.70(J)
0.42(1)
0.29(1)
0.14(1)
0.8 no filtering
0.54(_)
0.25(3)
0.31())
-0.05(-)
0.6 + strict filtering
0.73(-)
0.44())
0.23())
0.22(1)
0.6 + strict filtering
0,56(J)
0.36(7)
0.38(1)
-0.02(1)
no filtering
0.72())
0.44())
0.27(1)
0.19(1)
no filtring
0.57(J)
0.31())
0.40(1)
-0.09(L)
Table 15: Fine-tuning Results on Color (left) and Absolute (right) with different data tering standard.
Filter Constrains
S(t)
Tw(1)
Twa(4)
x<1)
Filter Constrains
S(t)
Tw(1)
Twa(4)
<(t)
zeroshot
0.77
0.34
0.23
0.10
zeroshot
0.79
0.20
0.15
0.05
0.8 strict filtering
0.77(-)
0.27(J)
0.09(L)
0.07(J)
0.8 strict filtering
0.77(.)
0.25(1)
0.24(1)
0.02(L)
0.8 no filtering
0.74())
0.32(J)
0.28(1)
0.04())
0.8 no filtering
0.75(_)
0.24(7)
0.24(1)
0.00XL)
0.6 + strict filtering
0.72(
0.29(L)
0,38(1)
0.05(1)
0.6 + strict filtering
0.75(C)
0.19(L)
0.28(1)
.0.09(])
0.6 no filtering
0.73())
0.33()
0.29(1)
0.04())
0.8 no filtering
0.74())
0.20(-)
0.27(1)
-0.07())
Table 16: Fine-tuning Results on Height (left) and Direction (right) with different data filtering standard.
images were created using vector graphies compositing, as shown Figure 16. To evaluate the model's ss performance, we categorized generation errors into five types:
Missing Objects: generated images lacking one or more required objects.
No Interaction: all objects present but without any interaction.
Wrong Interaction: objects interacting but not performing the required chasing action.
Wrong Direction: objects running but not in chasing formation (e.g.. running in opposite directions).
Reversed Roles: correct chasing action but with reversed subject-object roles (e.g., mouse chasing cat when cat chasing mouse was required).
Our training performance results showed that while generation quality improved (the number of right cases increased from 0-2 to 10-14), Reversed Role errors also increased, suggesting that the model learned to generate chase scenes but struggled with directional semantics, as shown Tab. 17. When testing novel object pairs with the same "chasing" relationship (both plausible pairs like "hippo<><lephant" and "bull+-man") "), we observed consistently poor performance across all new pairs, with similar increasing trends in No Interaction errors and Reversed Roles, shown in Tab. 18 and Tab. 19. Notably, there was no significant difference between plausible and anti- commonsense scenarios. These findings suggest that the core challenge isn't commonsense bias, but rather a fundamental limitation in processing directional relationships. The model struggles equally with both plausible and onti-commoneense scenarios, indicating an inability establish proper subject-object relationships regardless of semantic plausibility.
F.3.3 PLAUSIBLE SCENARIOS
Can current T21 models effectively distinguish different semantic relations in plausible scenar- ios? We investigate the ability of advanced commercial T2I models handle semantic variations through two plausible scenarios: "A cat chasing dog" and "A dog chasing a cat". Both scenarios are
Class
Reasons
SD XL cat-mouse
FT SD XL (trained on mouser-ccat) mouse->cat
mouse-icn
cat->mouse
Missing Objects
12
4
No Interaction
3
7
7
Wrong
Wrong Interaction
3
Wrong Direction
7
0
Reversed Role
2
6
Right
Partial/Full Match
0
10
Table 17: Training performance of SD XL before and after fine- moeettnnng on the balanced mouse data.
34

---

Published as a conference paper at ICLR 2025
Figure 16: Examples of an anti-commonsense scenario "a mouse chasing a cat". Top: DALL- E generated images with professional designer editing in Photoshop. Bottom: Vector graphics compositions.
SD XI.
FT (trained on monser-scat) bull-sman
Class
Reasons
bull->man man-+bull
man-<bull
Missing Objects
0
No Interaction
12
14
15
Wrong
Wrong Interaction
Wrong Direction
12
Reversed Role
2
Right
Purtial/Full Match
2
Table 18: Testing performance on unseen bull+>man pairs after fine-tume fine-tuning SD XL on the balanced couse<>cat data.
possible in real life. Using DALL-E 3 and Cog Cog we generated 30 images for each prompt and evaluated them based on strict criteria: (a) both animals, (b) running, (c) in the same direction, (d) clear spatial relationships (chaser behind chased), and (e) proper chase interaction. Results show significant performance differences, as shown Tab. 20. For "A dog chasing cat", DALL-E achieved 70% accuracy (21/30). However, for "A cat chasing dog". performance dropped dramat- ically. DALL-E 3 achieved only 13.3% accuracy while Cog View3-Plus failed completely. Failed cases either involved No Interaction Reversed Role errors. Even though these scenarios more plausible than "a mouse chasing a cat", particularly "a cat chasing dog", current advanced T2I models still struggle with semantic role reversals.
F.3.4 DATA IMBALANCE
Do T21 models' struggles with semantic relationships stem from training data imbalance? We conducted experiments to test the performance of fine-tuned SDXL trained with balanced train- ing data. We used human-filtere balanced dataset of cat<>dog chasing interactions for training, with 80 images for "a dog chasing a cat" and 80 images for "a cat chasing a dog", selected from
Class
Reasons
hippo -elephant dephant-hhppe
SD XL
FT (trained on mouse->cat) hippo--eleplie elephant-
Missing Objects
No Interaction
11
10
16
14
Wrong
Wrong Interaction
2
Wrong Direction
0
Reversed Role
Right
Partial/Fuil Match
Table 19: Testing performance on unseen hippot>relehhnt pairs after fine-tuning SD XL on the balanced mouset>ccat data.
35

---

Published as a conference paper at ICLR 2025
DALL-E3
Cogview 3-Pllu Plus
Class
Reasons
cat -ddo
dog icat
cat
-dog
dog >eut
Miss Objects
0
0
No Interaction
17
12
22
Wrong
Wrong Interaction
0
Wrong Direction
Reversed Role
9
18
Right
Partial/Full Match
Table 20: Error analysis of advanced commercial text-to-image models (DALL-E 3 and Cogview3- Plus) in generating directional relationships.
SD XI.
FT SD XI. (trained cal+-sdog)
Reasons
mouse- -aat
cat-Hmouse
mouse -aat
cat->mouse
Missing Objects
14
21
No Interaction
Wrong
Wrong Interaction
Wrong Direction
Reversed Role
Right
Partial/Full Match
Table 21: Testing performance unseen mouse<>cat pairs after fine-tuning SD XL on the balanced cat+>dog data.
DALL-E generations. We used two unseen prompt pairs involving the same "chasing" relation- ship between common objects for testing. The experiment results are shown in Tab. 21 and Tab. 22. Despite balanced training, the model showed consistently poor generalization: accuracy remained low for both matiscommonsense scenarios (mousc++cca, 3-3/30) and plausible scenarios (bull++man, 4-5/30). Failure analysis revealed three main categories: (1) Missing Objects, where the model fails to generate all required objects; (2) Relationship Understanding Failures, where objects appear ei- ther without interaction or with incorrect interactions (e.g. No Interaction, Wrong Interaction, and Wrong Direction), indicating the model's inability to comprehend the "chasing" concept; and (3) Reversed Roles, where the model fails properly assign who chases whom. Even among generated images, correct relationships occurred less frequently than these failure cases, suggesting random performance rather than true understanding. Thus, the models' struggles persist even with perfectly balanced training data, suggesting the core issue lies in relationship understanding rather than data imbalance.
F.4 TRAINING MECHANISM
F.4.1 TOKEN-LEVER EMENT vs. SEMANTIC-LEVEL IMPROVEMENT
Does fine-tuning enhance token-level or semantic-level understanding? The semantic perfor- mance of fine-tuned SD XL shown Tab. 3. To distinguish between improvements in token-level semantic level, additionally conducted experiments at token level. We instructed GPT-4 to verify whether "words with specific meaning" (including nouns, verbs, adjectives, adverbs, and relationship-describing prepositions) from the input text reflected in the generated image. For example, "a dog chasing cat", GPT-4 would identify three content words ("dog", "chasing". "cat") and verify their presence independently. Our analysis of the Absolute_locatic and Height
Class
Reusons
SD XL bull man->bull
FT SD XL (trained on cat<>soog) man-bull
bull->man
Missing Objects
0
0
No Interaction
6
12
13
Wrong
Wrong Interaction
5
Wrong Direction
16
12
Reversed Role
2
2
Right
Partial/Full Match
2
Table 22: Testing performance on unseen bull->man pairs after tie-ttning SD XL on the balanced cat+>doog data.
36

---

Published as a conference paper at ICLR 2025
categories revealed improved token-level accuracy after fine-tuning, shown in Tab. 4. While fine-tuning improves token accuracy for the Height category, it actually leads to a degradation in semantic understanding. This pattern suggests that the model, while better at incorporating individ- ual tokens from the prompt after fine-tuning, fails to maintain or improve its understanding of the semantic relationships between these elements.
This phenomenon is further illustrated in Figure 17, where the fine-tuned model successfully in- cludes most prompted elements (e.g., "shirt", "mannequin", "dress", "hanger". "floral" and "polka dot") but fails to establish correct semantic relationships between them. For instance, while both clothing items and patterns appear in the generated images, their associations are incorrect, demon- strating enhanced token-level accuracy but persistent semantic relationship errors. These findings indicate that current fine-tuning approaches may prioritize token-level matching over semantic com- prehension, suggesting need for training strategies that better preserve and enhance semantic un- derstanding. More examples are illustrated in Figure 18.
shirt mannequin has a floral pattern and the the hanger polka dot pattern.
mannequin has a polka dot pattern and on hanger floral pattern.
hanger has polka dot pattern and the shirt mannequin floral pattern.
SD XL
FT SD XL
Figure 17: Qualitative comparison showing the disconnect between token presence and semantic un- derstanding after fine-tuning: while fine-tuning improves token presence (e.g.. "shirt", 'mmanneuin", "dress", "hanger". "floral" and "polka dot" all appear in the image), the model fails to capture cor- rect semantic relationships between these tokens. This illustrates enhanced token-level accuracy but persistent semantic relationship errors.
F.4.2 SHORTCUT LEARNING PHENOMENON IN DATA-LIMITED CATEGORIES DURING FINE-TUNING
We observed a "shortcut" phenomenon during fine-tuning models, where some improvement in the average alignment score is misleading. For data-limited categories like Direction, generated images per prompt and constructed multiple test samples using various image combinations. The results reveal inconsistent patterns, where average alignment scores S (averaged across T., Tpm and Tpx) increased in sets (Imageset 2,3,6,8) but decreased in 4 sets (Imageset 1, 1,4.5,7) of experiments, as shown in Tab. 23. For all improved results, they consistently exhibited substantial gains in alignment score of the original text Tī and decreases in alignment score of the permutations Tp and Tps- The magnitude of anchor improvements was so large that artificially inflated all other average alignment metrics, masking performance issues in other aspects, shown in the bold in Tab. 23. This finding raises concerns about evaluation methods in current literature, which often evaluate gen- eration using only the average of alignment scores. Improvements in alignment scores may be mis- leading. For instance, the fine-tuned model shows higher average scores(O.7) 780..000000 780/7.000777 70.00//.////// 799 on Imageset 2/3/6/8) compared to zeroshot(O. (774).... However, deeper analysis using Sem VarEf- fect x scores revealed consistent declines, primarily due to Troo not decreasing as expected, indicat- ing the model fails understand true semanties.
37

---

Published as a conference paper at ICLR 2025
S(()
S.,(f)
Spias())
(Su,a + Sprap 1/2(t)
(Su, Spiai//2(1)
7.(1)
(w)(4)
s(t)
0.768
0.800
0.774
0.788
Imageset 2
0.284L.
0.802(1)
0.758LJ
0.7931L)
0.780011
0.797171
0.244(1)
0.20ctt)
0.38(J)
FT SD
Imageeset 6
Imageset
0.290(")
0.787()
0.800()
0.806:13
0.768()
0.754())
0.794())
0.80X-)
0.784113
0.780X T3
0.79711)
0.80717)
0.264(1)
B.20X()
0.224(1
0.36311)
-0.53())
0.00111
Imageset 8
0,798(")
0,841(1)
0.7561))
0.797(6
0.79901)
031997)
0.230(1)
0.26XC1)
-0.441)
Table 23: Analysis of alignment metrics revealing the shortcut phenomenon fine-tuning on Di- rection category. While anchor scores (Sa,a) show consistent improvements (1) across different Imagesets, permutation scores (Spv.pu. Sps.ps) remain unchanged or decrease after fine-tuning SD XL (FT SD XL). The blue items deceptively indicates improvement in results achieved through shortcut learning methods, but in reality, there has been no actual enhancement. The bold shows that magnitude of anchor improvements was so large that it artificially inflated all other average alignment metrics, masking performance issues in other aspects.
G MORE CASE STUDIES
In this section, present examples that demonstrate understanding of semantic variations and examples that do not. These examples are all generated from the advanced commercial T2I model DALL-E 3. Examples that grasp semantic variations typically have high alignment scores (S::) and high effect scores (k), as illustrated in Fig. 19. Conversely, examples that lack this understanding often have high alignment scores (S²) but low effect scores (x), as depicted in Figures 20 and 21. The Sem VarEffect scores allow us to distinguish the models' abilities to accurately interpret and visually represent semantic variations. However, in practice, evaluation accuracy can be significantly affected by errors in generated images biases in evaluators' ratings. Severe errors can particularly distort the evaluation's 's accuracy, as evidenced Figures and 27. To enhance the accuracy of our evaluations, will utilize more precise evaluators in future work.
H LIMITATION
We would like highlight that the size of Sem VarBench is constrained by the necessity for manual verification due to the unsatisfactory accuracy of LLM's validation, which incurs high costs. Fur- thermore, the scale of evaluation is limited by the high costs of image generation and LLM-based evaluation, both in terms of time and money, thus restricting the extent of such evaluations.
38

---

Published as a conference paper at ICLR 2025
Aspect
Category
Example
cal stands. T A dog stands and cut sits.
Ta: dog
Action
Toi: A cat stands and a dog sits.
Ta: An old person kisses young person.
Relation
Interaction
Tpo: A young person kisses an old person.
T pe A young person is kissed by an old person.
T. The soft teddy bear is on the bed and the hard toy on the shelf.
Absolute Location
T Tp: The soft teddy bear is on the shelf and the hard toy car is on the bed.
7 pi. The hard toy car on the shelf and the soft teddy bear on the bed.
Ta: green apple sils slop red leal.
Relative Location
Tpu A red lcaf sits atop green apple.
p4: red leaf sits below a green apple.
T.: Sushi roll; first put the fish on the seaweed, and then put the rice top.
Spatial-Temporal
T Tpo Sushi roll; first put the rice on the seaweed, und then put the fish on top.
T pc. Sushi roll; first apply the fish the scaweed. and then place the rice top.
Ta boy jumps away from the fence and towards the river.
Direction
T pu: A boy jumps away from the river and towards the fence.
T pi: boy towards the river and jumps away from the fence.
To: The cake and the plate; the cake is too big for the plate.
Size
Tpu: The cake and the plate: the plate is too big for the cake.
Attribute Comparison
T Te: The plate and the cake; place is too small for cake.
Ta A dinosaur towering over a human.
Height
Tpu: A human towering over dinosaur.
Tpt: A human being towered over by a dinosaur.
T. The athlete with a heavy backpack is walking quite slowly and the one with light
bag is running faster.
Tpu The athlete with light backpack is walking quite slowly and the with heavy
Weight
bag is running faster.
Tp: The athlete with a light bag is running faster and one with a heavy backpack
walking quite slowly.
T. cake with more Trosting on the top than on the slides.
Vague Amount
T po: cake with more frosting the slides than the top.
T Tpc A cake with less frosting on the slides than on the top.
T. man in purple shirt is carrying brown suilcase.
Color
T po: man brown shict is carrying puppl suitcase.
Tpt: A brown suitcase is being carried by man in a purple shirt.
Ta: Four dogs deghouse and one dog harking outside.
Counting
T Tpo: One dogs doghouse and four dog barking outside.
Attribute
Tpe One dog barking outside and four dogs in a doghouse.
Values
T. Two lish; the one in the tank has stripes and the one in the bowl doesn's i.
Texture
T Two fish; the one the bowl has stripes and the one in the tank doesn't.
Tp:: Two fish; the one in the bowl has no stripes and the one in the tank does.
Ta: There's a satin teddy bear with a furry bow.
Material
T There's 's a furry teddy bear with satin bow.
T A satin teddy bear has furry bow.
Ta: The circular suitcase
an oblong lock.
Shape
T The oblong suitcase has an circular lock.
T An oblong lock on the circular suitcase.
1.: The person on the left old and the person the right young.
Age
Tpo: The person on the right is old and the person on the left is young.
Tpa The person on the right is young and the person on the left is old.
T. The happy child playing
to sad clown.
Sentiment
Tpu: The sad child is playing next to happy clown.
Tpi Next sad clown, a happy child playing.
T. lced coffee and steaming
Temperature
Tpu: Steaming coffee and iced tea.
T pp: Steaming tea and iced coffee.
T.: The building on the corner has modern design and the monument in the park has
a clussic design.
T pu: The building on the comeer has a classic design and the monument in the park has
Manner
modern design.
Tp The monumeat in the park has classic design and the building on the corner has
modern design.
T.: The boy with a blue shirt has long hair and the girl in the pink dress has short hair.
Appeurance
Tpu: The boy with blue shirt has short hair and the girl in the pink dress has long hair,
Tp:: The girl the pink dress has short hair and the boy with blue shirt has long hair.
Table 24: Permutation-base valid sentences (Ta: T Tp:) in diverse categories.
39

---

Published as a conference paper at ICLR 2025
mountain distance has snowy peak and the hill by green
mountain has green peak and the hill by snowy peak.
The by the green peak and the mountain in the distance snowy peak.
SD XL
SD XL
The person the front short while the person in the back tall.
The person in the back short while the in the front is tall.
The person in the back tall while person front
SD XL
FT SD XL
The the red and car in the driveway is blue.
The in the blue and the car in the driveway is red.
The the garage red, and the car the driveway is
SD XL
FT SD XL
Figure 18: More examples for qualitative comparison showing the disconnect between token pres- ence and semantic understanding after fine-tuning.
40

---

Published as a conference paper at ICLR 2025
Anchor Text
Permutation- -aarianc Bermutation-Variaree
SemVarEffect Score
photograph, more smiles than frowns.
frowns photograph.
smiles the photograph.
GPT-4 Turbo
50%
0.98
Matched
5(Fu.. 0.95
5p 0.97
0.98
Yer
9(sx. 0.45
0.03
Mismacched pairs
0.55
S(Y.. 0.98
s(T../a.) 0.95
0.90
In the pool, there are four floatie and one diving board.
In the pool, there are one floatie and four diving boards.
In the pool, there is one diving board and four flosties.
1.7.-la) 0.96
Matched
5(7p=1pr)
1.00 S(Tpu
Yor
S('.. 0.35
7ayo 0.04
Mismatched pairs
S(F.. = 0.65
S(Ta-1a1) 1100
The camels are toller then the horses.
The horses are taller than the camels.
The horses are shorter than the camels.
(0...a) 0.93
Matched
0.95
S(Tx.-p))
0.98
May
5(Fgu=))
0.55
0.08
Mismatched
S(Tn=(uw) = 0.59
pains
S.
S(Ta.15))
0.98
0.66
Copper pots pots copper plates.
plates copper
plates.
pots.
S,T,-1)) 0.95
Matched pairs
S(Tpu-lpa)
0.94
Sy 0.96
S(Tpu-po)
1.00
Yuy 0.60
S(Fp...uu) = 0.66
Mujo 0,05
Mismaacheed
S(Ua-1pp))
pairs
((fpe.la)
0.95
((T..f,)) 0.95
0.63
There's sleek, modern phone with an old, clunky computer.
There's 's an old, clunky phone with a sleek, modern computer.
There's s an old, clunky computer with sleek, modern phone.
50a...)
S(Tpu=lpu) m 0.75
3g
pairs
5(Fxc-pg)
Y 0.7U
S(Tsu.1.) - 0.65
Mismatched
S(Va-fp>
pairs
s(Y,u.)) 1.00
s(T..b,))
0.61
The happy dog wapging while the sleeping.
The happy dog sleeping while the cat wagging
While the cat slesping. the wagging
Matched
S(Tpn- )ax) 0.93
Sy 0.93
pairs
S(,,.-A.)
051
S(Tse 0.65
Mismatohed
S(F,.
0.75
pairs
S(Txo
0.67
S("..ar))
# 0.46
Figure 19: The cases which understand semantic variations.
41

---

Published as a conference paper at ICLR 2025
Anchor Text
Permutation- Cermutation-Variance
Permutation- Variiacce
SemVarEffect Score
the park, few benches many trees.
and many
At the park, many trees benches.
GPT-4
Mutchad pairs
SVFn.he) 0.60
5(%-1.) - 0.93
S, 0.03
((iu 0.5
Pay 0.28
1(%a,.1a) = 0.50
Yaye 0.01
s(T,.'po)
((Ta.))) 0.94
S(T..l..) 0.93
0.27
heavy the hook on table is not.
bag heavy and the one on the hook is not.
heavy and bag on the hook
5(%-1a) 0.89
Matched
0.85
Mp 0.89
(gan-9r) = 0.93
Fey
90',..a) 0.885
Yaju 0.10
Mismacched pairs
s(T.. = 0.55
5('go 0.90
s(T../a))
0.82
0.24
Baked potato; first and then put the sour cream top.
Baked potato; first the potato, and then put the butter on top.
Baked potato; first the potato, then top with sour cream.
potato,
1.,, - 0.94
Matched S(unceas) 0.95
Sp
S(Tpu)..) 0.95
S(',....) 0.87
0.06
Mismatccedd
S(Ta. = 0.75
pairs
9(F
0.89
The computer is on the desk and the phone on the nightstand.
The computer is on the nightstand and the phone the desk.
The phone on the nightstand and the computer the desk.
S(T,.la)
S(F 0.45
Sg
2
pairs
S(T (w)
Y 0.03
5(Tpu. -
Yuys
Mismatched pairs
((Ta-1a)) 0.65
S(Yec.le)
S(T..
0.20
happy family walking next to sad ghost.
sad family walking next to a happy ghost.
ghost, a
happy family walking.
S.T,..)) = 0.94
Matched
S(Tze-pe)) S(Txu-f)) 0.9
8888
Yu.
S(Y 0.65
0.06
Mismatcheed pairs
((T..1z)) = 0.63
S(Tec-le) s(T..fx)) 0.90
The the realistic ones on the floor are abstract.
The paintings wall floor are realistic.
The floor paintings on the wall are realistic.
Matched pairs
S(T..- (wx) 0.45 S(Tcc
Saaa-aa 0.65
S 0.70
1.00
0.30
S(Fpu..la) = 0.35 (9%,.fae)
Y 1.00
Mismatched
S(Fgc-1.) 0.30 5(%a.
-0.70
Figure 20: The cases which don't understand semantic variations.
42

---

Published as a conference paper at ICLR 2025
Anchor Text
Permutation- -Varianee
Srrnuutation-Vaiinnne baby crawis.
SemVarEffect Score
perent walks.
parent crawls.
parent
GPT-4 Turbo
50%
Matched pairs
5(Fp = 0.93
3 0.95
5(Tn bd 0.96
0.25
9(T. [.] 0.70
Payo
Mismatched
s(7..
((F.. 0.93
S(U..h:))
0.19
A full glass is next to empty plate.
An empty glass is next to full plate.
An empty plate is next to full glass.
1(7alla) 0.95
Matched
5(7m=1pr) 0.85
S(Tpu
0.98
May
S('.....)
Payo
Mismatched pairs
s(T,.
0.80
9(Fsa S(Yo.19)) 1.00
= 0.19
The skater wears denim vest over graphic t-shirt with round neck collar.
The skater wears a graphic vest over denim t-shirt with round neck collar.
denim vest worn by the skater over graphic t-shirt with round neck collar.
5(%,-.)) - 0.97
Matched S(Fnncla))
S(Tuu).)
0.89
Pay
S('.....) 0.98
0.13
Mismatched pairs
((T.. = 0.93
S(T..1p))
1.00
0.07
young on the elder teacher's shoulder.
young shoulder under elder teacher's 's hand.
on the young student's shoulder.
5.%- - 0.95
Matched S(Ua. 0.90 pairs
5, 0.93
s(FnuAa)) 0.93
S(F...1.) - 0.85
0.02
Mismatched pairs
s(Ta.
S(T.....) 0.95
0L03
mountain distancc has snowy peak and the hill by the river has peak.
by green peak and the mountain the distance
distance has green peak and the hill by the river has
peak.
5aT,
S(Ep.. = 0.79
3,
pairs
5(Tp bc) 0.95
P 0.09
5(T = 0.75
Pajjo
Mismatched pairs
S(Ta. 0.95
(. 0.95
0L03
robot serving tea to a group of children noxt to
parent serving tea to group of children next to
rabot next to a parent is serving tea group of children.
S.%,-))) 0.95
5(Tp=- 0.94
Sp
0.95
rf, 0.03
s(()....)) - 0.95
Payj
Mismatched
5(T2.
0.93
pairs
S(F,,,
0.95
0L03
Figure 21: More cases which don't 't understand semantic variations.
43

---

Published as a conference paper at ICLR 2025
Anchor Text
Inrnuutation-Vaiiinne
Permutation Variance
SemVarEffect Score
kids riding street and one kid skateboarcling.
riding the street and four kids skateboarding.
four kids
riding bikes
Turbo
skateboarding.
40, 0.80
Matched S(1
S(Tpo-pr)
0.93
Pe:
5(F 0.30
0.16
Mismatched
S(T,. = 0.55
pairs
S(T
s(fa.
0.93
The adult the bench reading stroller is sleeping.
sleeping and adult on the bench reading.
reading and the adult on the bench is sleeping.
1.)..)) 0.90
Matched
5(7p (po)
0.92
0.90 S(Tpu.aa)
Y 0.63
8(%p.. 0.50
0.11
S(Tin
0.65
pairs
5(f....)
0.92
s(T..fp:) 0.98
There's cup saucer.
cup
With ceramic plastic
plastic
I,,, 5(Fe--Faa)
Matched pairs
0.80
Sy 0.01
s(T,:
0.09
Fay 0.70
8(7ju.1,) 0.35
Yayo
Mismatched
S(T..1pp)
0.50
pairs
S(Tgx =
S(Ta.fp))
0.45
The waiter eyes menu.
the
The covering eyes menu.
The waiter eyes
menu.
SC7..l) =.990 0.90
Matched S(Up=lpu) 0.60
3 0.73
S(Tn-Aa)) = 0.70
Fey
S(F,..1a) 0.30
0.35
0.45
pairs
3(Y,.4,)
S(T..1..) 0.60
swing higher than other children on the seesew.
the lower than the other children the seesaw.
The other children seesaw on the swing.
S.T,-,)
S(Tpn-lar 0.25
Sj
s(F,r 0.92
FAy 0.60
S(",...,)
Yayu
Mismatched
5(Ta.
0.91
pairs
S(Fau..) 0.65
S(Y'..u))
0.39
The athlcte with medal celebrates the athlete without medal applauds.
The athiete with a medal applauds and athiete without celebrates.
The athlete without medal applauds and athlete celebrates.
Matcched pairs
S(Tp. = 0.65
Sy 0.68
S('n.
Y/, 0.90
S(Fre.Aa) = 0.95
Mismatched
s(r,-(w) 0.55
2
S(Fx. 1.) 0.96
0.33
Figure 22: Cases with minor emrors which understand semantic variations.
44

---

Published as a conference paper at ICLR 2025
Anchor Text
Permutation-Vaie -aariance
Permutation-Varinn
Variiacce
SemVarEffect Score
child wearing
child wearing a business suit with their fists the and wearing superhero cape.
A parent wearing business suit and child superhero cape
superhero cape with their fists the air and wearing suit.
GPT-4 Turbo
0.88
Matched
0.60
Sp 0.81
pairs
5(7p
0.95
Far 0.55
9(%p==1) 0.15
Paye 0.20
Mismatched
s(7,.
0.96
pais
0.95
The waiter wearing black vest over white shirt.
The waiter is wearing white vest over black shirt.
black vest being worn by the waiter over white shirt.
5(7p- 0.93
Si
5((p) 0.93
Fay 0.35
90,,..h,) =
Payu
Mismatchedd
5(7.-1o0)
0.93
S(Fpu.l)
1.00
0.28
baby's mother's chest.
the baby's s chest.
baby's foot.
1. (g) 0.75
Mattched
S(Tpr-pa)) 0.53
Su 0.61
pairs
S(Fpu/a:) 0.55
r4, 0.33
9(%,..1.) - 0.30
Kayo 0.15
Mismatched pairs
S(Ta.1gu)
((pn.l))
s(Co.Au)) 0.65
balloons chair, and in air, three belloons are floating.
and three balloons floating the air.
chair and two balloons floating in the eir.
Sa 0.65
Matched
8(Fxe-pe)) S(Teu-f))
0.67
u.65
Yve 0.05
S(i 0.65
Kayp 0.D0
Mismatched pairs
s(C.....)) 1.65
S(T.c.e)
s(T.. = 0.65
Chefs white uniforms golden frying
Chefs in golden uniforms frying pan their hands.
In white uniforms with golden frying their
S(Ta.le) 0.95 S(Fre-fue)
Matched pairs
5 0.77
S(Teu- (w) 0.81
0.17
S(Tp.:fa) 0.55
Koys 0.29
Mismatched pairs
0.98
s(T..(,) 0.83
younger child hugping the leg of older parent.
older parent is hugging the leg of younger child.
The leg of older parent being hugged by a younger child.
SCa,, S(Fra-loo) S(T,u-f,) 0.95
Matched
0.65
Yuy 0.35
S(Tpu.
Ywys
Mismatchad pairs
S(T..
0.65
S(T..-I.)
S(T..
0.95
-0155
Figure 23: Cases with minor emrors which don't understand semantic variations. Several alignment scores, which incorrect according GPT-4V, labeled in red.
45

---

Published as a conference paper at ICLR 2025
Anchor Text
Cermutation-Varianee Permutation Varriacce
SemVarEffect Score
wearing pink dress
wearing blue shorts boy wearing pink dress playing the sand.
wearing blue wearing pink dress with her hair in the air playing in the sand.
a boy wearing blue shorts
GPT-4 Turbo
playing in the sand.
((%,-1))
S(U 0.85
3p
pairs
s(Fachs))
Kay 0.00
s(Fre.A)) = 0.78
Payu
Mismarched s(T,,
s(Tgr (a) 0.92
((%.411)
-0.03
There's steel knife with wooden handle.
There's wooden knife with steel handle.
There's a knife with wooden handle, made of steel.
s(Fnn 0.60
S(T,))b)) 0.93
Yi, 0.02
s(7).:1.)
Paye
Mismatched pairs
5(Ta.
0.98
S(Fg..
0.9G
0.03
giraffe zebra.
giraffe.
giraffe.
S(Tar 0.98
Matched pairs
S(F
U.55
Sg 0.79
S(Tpo-fcc
0.85
Twr
S(Tpu.1a) 0.55
Kuys 0.15
S(T..fpe)
pairs
s(T,u s(T..(() 0.98
parent
group children next
group children next robot.
serving tes to group
parent.
children.
5(T.-Ta) = 0.95
Matched
S(Up.-Iae) 0.99
3p 085
s(Tpulac) 0.95
S(Tx.-1.] 0.95
Payp 0.00
Mismatched
S(Ta.
0.93
pairs
S(C..Aa))
0.95
0L03
Figure 24: Examples of acceptable outliers include negative Sem VarEffect (x) values that are close to zero. Outliers with Sem VarEffect score (I)) slightly below 0 are acceptable.
46

---

Published as a conference paper at ICLR 2025
Anchor Text
Permutation- Variance
Coomutation-Varianae
SemVarEffect Score
The person the hat is smiling and the person without frowning.
The person the is frowning and the person without hat smiling.
The person without hat frowning and the person smiling.
GPT-4 Turbo
Sg
-.40
pairs
S(Ta
0.30
Yey 0.65
5(75u (.)
Mismatched pairs
1.00
1.00
-0.40
people chopsticks except one who eats with fork.
Except for who chopsticks, people cat with a fork.
except for with chopsticks.
S(T..a) 0.55 8(T
Matched
U.54
5(7xa (w) 0.10
Yer 0.56
S(Tgo.la) 0.80
1.00
Mismatched pairs
((am = 0.93
S(Tec.l)
s(f.-fc)) 0.00
The wooden spoon is in the drawer and the metal spetula is the counter.
The metal spoon in the drawer and the wooden spatula the
The metal spatula on the counter and the wooden
Matched pairs
0.65
5g 0.66
X 0.38
S(Fou)e) = 0.75
Fwys
Mismetiched fpr
0.70
s(Fa..e)) = 0.98
s(T,.)..)
0.93
The hot coffee the mug the glass.
The cold the cold the hot mug.
glass.
Matched pairs
1.%. 0.98
0.25
5, 0.73
s(FuuAa)) 0.95
P4, 1:18
s(F....))) - 0.95
Yaye 0,07
Mismarchad sQT,.
S(Tuul,) 1.00
S(E...g.) 1.00
111
cup
melting while the loe cream in the cup frazen.
melting while the ice cream in the cone is frozen.
frazen while cream in the cone is melting.
SCa...) 0.45
Matched
5(pwl/j)) S(T,.-u)) 0.58
Fay 0.45
S(C,u 0.60
Tw/s DL83
Mismatched pairs
S(Taxfpx) - 0.65
S(T,,-,)
S(T..),)) 1.00
The pocketss left side of the jacket are big and the ones the right side
The pockets on the side of the jacket are small and the ones on the right side big.
The jacket has big packets the left side and small the right
0.70
pairs
S(T,u
0.98
PLy 0.15
5(T,a.1))
Yajo
Mismatched pairs
S(Tar
0.95
S(Y,,,.)
0.65
-0229
Figure 25: Examples of acceptable outliers include negative к values that are with a Sem Bffeffect score outside the range [0, being considered unacceptable. This discrepancy may be due incor- rect text-image alignment scores provided by evaluators or low quality images.
47

---

Published as a conference paper at ICLR 2025
Anchor Text
Permutation- Varriace
Permutation-
c Variance
SemVarEffect Score
shiny ring is next to watch.
dull ring next to shiny watch.
shiny
ring.
Matched
0.93
0.50
Su 0.77
pairs
S(Tuu
0.28
0.98
S(Tgu 0.85
Fayj 0.12
Mismatched
S.Tsc.a)) = 0.95
((T..)))
0.98
police officer in uniform is holding black flashlight.
A police officer holdinng white flashlight in black uniform.
uniform holding white Hlahhlight.
POLICE
SCT..a) 0.98 Matched 8( S(Txa-b))
8.86
0.98
Yer:
S(Tporla) 0.73
0.18
Mismatched pairs
S(T, = 0.35 S(T,r.1,)
f(F, 0.93
green apple apple green stem.
brown green apple.
stem.
S0aala) S(Fp- S(Tx.
Matched pairs
0.55
Sp 083
Yd 0.64
S(F,..1)) - n.55
Maye
Mismatiched
S(Ta.
S(Fa.la) 1.00 S(F...u))
0.53
The pizza on the tray round and the sandwich the plate
The pizza on tray is square and the sandwich the plate round.
The sandwich on the plate is square and the pizza on the tray round.
Matched pairs
S(Fan-Aa) 0.93
1.%- -
Sg 0.95
Ply 0.52
s(Fiu.A.) - 0.50
Kaye
Mismatched
S(T..fux)
S(Tau-l)) 0.93
9(F..-I.))
0L48
The happy child the worried parent the edge.
The worried child the happy the edge.
The worried parent the edge happy the pool.
Matched pains
0.92
S 0.92
P U.51
S(Tgu-1.) f.) - 0.50
Yayu 0.00
Mismatched
(. 0.98
b.)
0.43
bird with colarful feathers flying above a bird without feathers.
A bird without feathers is flying above bird with calortul feathers.
Above bird without feathers, bird with colorful feathers flying.
S(T. 0.05
Matched
5(7p.
S(Ta (x) 0.09
K
S(",..,,)
7ajo 0.09
Mismatched pairs
5(F.. = 0.55
S(Tga 0
S(T..1a)) 0.93
= 0.43
Figure 26: Errors only due incorrect scoring by GPT-4V, where images are essentially correct.
48

---

Published as a conference paper at ICLR 2025
Anchor Text
Permutation- -arriace
Permutation-Variane
SemVarEffect Score
happy couple restaurant and the grumpy waiter is at the table.
grumpy couple restaurant and the happy waiter is at the table.
restaurant, happy couple and the grumpy waiter are the table.
GPT-4 Turbo
s(Fnn
0.93
Sy
pairs
s(Tu
0.94
Fly 0.30
S(Fp::1)) -
Paye
Mismatched pairs
5(T..
0.93
0,97
- 0.32
giant squid attacking ship, and the squid is bigger than
A ship attacking a giant squid, and the ship bigger than squid.
squid. bigger than the ship, attacking ship.
5(Ga, 1.)
Matched pairs
S(Foo-fpe) = 0.93
X, 0.94
5(Tpr-21)
Kwy 0.19
S(Tpu.la) - 0.75
Pays 0.07
Mismatcheed
S(%a-fae)
s(F 0.93
0.12
swimmer the pool swimming towards the edge.
edge swimming towards the pool.
Towards edge, swimmer in the pool is swimming.
S(Too-lp)) = 0.95
Sj
pairs
5(7p-fp)) 0.80
X/, 0.30
S(Tpurtu) S(Ta-1ar)
Kays
Mismatiched
0.95
pairs
S(Tp..l.) 0.90
s(r..fe))
- 0.05
silver with gold handle.
gold spoon gold silver handle.
with a silver spoon.
Matched pairs
S(Tu=-fa))
1c). 0.5
0.50
My 0.60
s(Faa-pi) 0.60
You 1146
S(Tpm.4a) = 0.91
Vins 0.02
Mismatcceed pairs
S(Txalo) 0.97
s(Te. 1.00
0.04
The coffee cup is green.
The coffee the green cup black,
The in the green the the mug is black.
5Ca. (.) 1.00
Matched S(Tuefpr))
0.90
5(Tpofp)) 0.99
5(T (.) 0.85
Mismatched pairs
S(u.Alo)) 0.94
S(Tpx ((T..ig)
1.00
K= 0.05
The happy baby the and unhappy baby is stroller.
The unhappy baby the happy baby stroiler.
The unhappy baby the stroller the happy baby
S(T,, 1.) 1.00
Matched pairs
S(Fpu-fps)
U.99
Sp 0.93
S(Tprfg:) 0.80
Yw/
S(fpu.(a) 1.00
YAps 0.17
5(i.lee))
1.00
pairs
S(Txu-
s(T..(,)
0.98
Figure 27: Errors only due to incorrect scoring by GPT-4V, where images are essentially correct.
The errors heavily influence the Sem VarEffect scores.
49


Reviews:
[{"review_id": "M8WS4RWsip", "reviewer": "Reviewer_UuMu", "summary": "The paper presents a novel metric, SemVarEffect, and a corresponding benchmark, SemVarBench, designed to evaluate the causality between semantic variations in input text and output images for text-to-image (T2I) synthesis models. The authors identify a gap in current evaluation methods, which often rely on indirect metrics like text-image similarity, failing to adequately assess how models handle semantic changes due to word order. SemVarEffect measures the average causal effect of input semantic variations on output images, while SemVarBench provides a dataset with controlled linguistic permutations to test model performance. The experiments reveal that even state-of-the-art models struggle with semantic variations, indicating room for improvement in understanding and generating accurate visual representations from textual instructions.", "questions": "See weakness part.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "- The paper directly addresses a critical issue in the evaluation of T2I models, which is the accurate interpretation of human instructions and evaluate the finegrained instruction following ability of the models.", "weaknesses": "1. The paper suffers from significant presentation issues, making the proposed method difficult to comprehend. The overall structure of the methodology section requires refinement:\n   - Figure 2, which is crucial to understanding the proposed method, is challenging to interpret.\n   - In Section 2.2, the term \"localized change\" needs clarification.\n   - The definition of \"integrated visual semantic variations\" is also unclear.\n   - Line 278 uses the phrase \"evoke vivid mental images,\" but its meaning is ambiguous in the given context.\n   - Table 2 includes numerous annotations, but offers no explanation of what the different scores represent.\n\n2. In my opinion, the method appears overly complex. The core idea is straightforward: we anticipate that semantic alignment of generated images should be strong if the inputs are semantically identical, and less aligned if the inputs differ semantically. Thus, a simple metric like S(I_a, I_pu) - S(I_a, I_pi) could be employed. What advantages does the proposed causal relationship provide over this simple baseline? I would appreciate deeper insight and explanations on this matter.\n\n3. Regarding model-based evaluation, there is no discussion on whether the MLLM used to compute the alignment score is effective for this task, nor is there any human evaluation. This omission raises concerns about the reliability and trustworthiness of the scores presented.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fUqoszdAbU", "reviewer": "Reviewer_nmou", "summary": "This paper proposes a novel metric called SemVarEffect and a benchmark called SemVarBench to evaluate text-to-image (T2I) models' ability to capture semantic variations caused by word order changes in input prompts. The key contributions are:\n- A causal inference framework to measure how semantic variations in text inputs affect visual semantic variations in generated images.\n- A carefully curated dataset of text triplets (anchor, permutation with changed meaning, permutation with unchanged meaning) to test T2I models.\n- Comprehensive evaluation of  state-of-the-art T2I models. This also includes fine-grained Analysis of the role of text encoders and cross-modal alignment in handling semantic variations.", "questions": "How does the choice of filtering training data using MMLM potentially affect the evaluation results, particularly for categories with limited samples?\n\nWith the 4 metics you had mentioned, which one is the main metric that would make sense quickly for readers? It is a little hard to follow with different metrics presented at different parts, which I can imagine is a little confusing especially for readers who are looking for a single metric for benchmarking.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Strengths:\nNovel and important problem formulation: The paper addresses a critical gap in evaluating T2I models' language understanding capabilities.\n\nThe causal inference based framework provides a principled way to isolate the effect of semantic variations.\n\nSemVarBench is carefully curated to avoid easily predictable variations and focus on challenging semantic permutations, with comprehensive coverage of different type of variations. LLM generated results were checked.", "weaknesses": "The training dataset is generated using the same base SDXL model and filtered by MMLM, which could introduce self-distillation artifacts and bias, particularly affecting performance on categories with less data. \n\nThe paper suggests that fine-tuning improves alignment at the word level rather than semantic variants, but doesn't provide sufficient discussion or analysis to support this claim", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BTNfKEUboH", "reviewer": "Reviewer_kcHM", "summary": "The paper primarily focuses on the impact of word order variations in text prompt on text-to-image synthesis. It proposes a metric to evaluate a model's semantic sensitivity to changes in word order. The main idea is to calculate the difference in visual semantics of generated images between semantic-variant and semantic-invariant text rearrangements. Additionally, a dataset of approximately 11k samples has been collected as an evaluation benchmark, and popular models have been assessed and analyzed based on this benchmark.", "questions": "- In section 4.1, If text-image similarity is obtained using MLLMs, could biases inherent in the MLLM itself (insensitivity to changes in word order) affect the evaluation results?\n- Text-to-image synthesis models have difficulty accurately reflecting semantic changes caused by variations in word order, possibly due to certain commonsense biases inherent in the training data. This could be seen as a form of “built-in bias.” For example, as illustrated in Figure 1 of the paper, “A cat chasing a mouse” is a reasonable depiction, whereas the reordered phrase “A mouse chasing a cat” is less plausible, leading to poorer generation results. It might be helpful to add related experiments to validate this hypothesis.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The authors find that existing text-to-image synthesis models struggle to capture semantic shifts caused by word order changes and proposed an evaluation metric and benchmark. \n- The SemVarEffect metric is based on the difference between semantic-variant and semantic-invariant text rearrangements, similar to the differential denoising principle in circuits, which I find quite novel. \n- The paper is well-organized and easy to understand.", "weaknesses": "- The definition of visual semantics: The primary contribution of this work is the metric to measure the impact of word order in text prompts on text-to-image synthesis. The basic principle is to alter the text and observe the resulting visual semantic changes in the generated image. However, within this metric, the “visual semantics” of the generated image is defined through text-image similarity, which may not be entirely accurate. Visual semantics in an image typically refers to the content within the image itself, such as object categories, attributes, or scene information. In contrast, text-image similarity only measures the alignment of high-level features between the two modalities; it does not represent the actual visual semantics. Ideally, visual semantics should be defined independently of text. \n- The mechanism for ensuring that the generated data is semantic-variant or semantic-invariant during benchmark creation seems to be insufficiently specified, and why are some data extraction templates and rules created using LLMs while others are crafted manually? What challenges are involved in this?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BuEGW03R3S", "reviewer": "Reviewer_xM6n", "summary": "This paper focuses on the performance of Text to Image (T2I) models in handling semantic variations. It identifies that existing T2I models tend to overlook the relationships between words in the input text when faced with similar semantics or semantic substitutions, making it challenging to generate accurate images. At the same time, the paper introduces the SemVarEffect metric and SemVarBench to evaluate this issue.", "questions": "See Weakness", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "- This paper offers an interesting and novel perspective on the current issues facing T2I models, and provides an effective metric and dataset to measure this issue.\n- The paper offers detailed and fair experiments to compare various T2I models.\n- The analysis section is thorough and carefully examines the causes of the semantic understanding issues in T2I models, providing a reliable path for future research.", "weaknesses": "- I noticed that some of the samples provided in the paper, such as \"A mouse chasing a cat,\" are somewhat unrealistic captions. However, this category is not emphasized in SemVarBench, as seen with sentences like \"A cat chasing a dog\" and \"A dog chasing a cat.\" These sentences also exhibit semantic variations, but I believe that current T2I models can handle these changes. Therefore, I think further discussion on this issue is needed.\n- I believe it is necessary to further discuss the relationship between this phenomenon and the training data in the discussion. In the problematic captions where semantic variations occur, a significant portion of the original captions may be included in the pre-training data. Therefore, similar to the discussion in paper [1], I suggest designing experiments to investigate whether T2I models can perform reverse inference of semantic variations.\n\n[1] Berglund, L., Tong, M., Kaufmann, M., Balesni, M., Stickland, A. C., Korbak, T., \\& Evans, O. (2024). *The Reversal Curse: LLMs trained on \"A is B\" fail to learn \"B is A\"*. arXiv:2309.12288.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)