Evaluating Semantic Variation in Text-to-Image Synthesis: A Causal Perspective

## Abstract

Abstract Accurate interpretation and visualization of human instructions are crucial for text-to-image (T2I) synthesis.
However, current models struggle to capture semantic variations from word order changes, and existing evaluations, relying on indirect metrics like text-image similarity, fail to reliably assess these challenges. This often obscures poor performance on complex or uncommon linguistic patterns by the focus on frequent word combinations.
To address these deficiencies, we propose a novel metric called SemVarEffect and a benchmark named SemVarBench , designed to evaluate the causality between semantic variations in inputs and outputs in T2I synthesis.
Semantic variations are achieved through two types of linguistic permutations, while avoiding easily predictable literal variations.
Experiments reveal that the CogView-3-Plus and Ideogram 2 performed the best, achieving a score of 0.2/1. Semantic variations in object relations are less understood than attributes, scoring 0.07/1 compared to 0.17-0.19/1.
We found that cross-modal alignment in UNet or Transformers plays a crucial role in handling semantic variations, a factor previously overlooked by a focus on textual encoders.
Our work establishes an effective evaluation framework that advances the T2I synthesis community’s exploration of human instruction understanding. 1 1 1 Our benchmark and code are available at https://github.com/zhuxiangru/SemVarBench .

## 1 Introduction

Accurately interpreting and visually depicting human instructions is essential for text-to-image (T2I) synthesis Cao et al. ().
Despite advancements in alignment Lee et al. (); Wu et al. (); Kirstain et al. (), composition Liu et al. (); Wang et al. (); Li et al. (); Feng et al. (), and long instructions Yang et al. (); Gani et al. (), these models still treat text prompts as bags of words, failing to depict the semantic variations in human instructions Yu et al. (); Mo et al. ().
As shown in Fig. 1, existing T2I models generate images with identical semantics, even when the inputs differ semantically (e.g., “a mouse chasing a cat” vs. “a cat chasing a mouse”).
This indicates that existing T2I models struggle to accurately capture the semantic variations caused by word orders changes.

Figure: Figure 2: Measuring the causality between semantic variations in inputs and outputs of T2I models. Blue areas show alignment scores (by GPT-4) and resulting semantic variations when the semantics are changed. Pink areas show alignment scores and resulting semantic variations when the semantics are maintained. Purple areas on the far right show the SemVarEffect score, reflecting the causal contribution of input variations to output variations based on the two interventions. Gray areas show the average alignment score commonly used in previous T2I studies, focusing on similarity rather than causality. For detailed explanations of the symbols and calculations, see Section 2.
Refer to caption: x2.png

There is a lack of direct metric to evaluate a T2I model’s ability to understand semantic variations caused by word order changes.
Existing NLP research typically evaluates semantic variation indirectly through downstream tasks.
For example, in language generation Gordon et al. (), the input sequences with different word orders are used as the actions in a navigation game and the model is evaluated based on the game’s accuracy. Similarly, in visual-language understanding Thrush et al. (); Diwan et al. (); Yüksekgönül et al. (); Wang et al. (); Burapacheep et al. (), models are evaluated via cross-modal retrieval and image-text matching, focusing on text-image similarity.
In T2I synthesis, the text-image alignment score offers an indirect performance measure but may not fully capture a model’s sensitivity and robustness to word order.
For example, as shown in the upper-right of Fig. 2,
an average alignment score of 85, as evaluated by GPT-4, might seem satisfying, but it may conceal the model’s proficiency with common word combinations while masking its inadequacy with less frequent or more complex linguistic patterns.

We propose a novel metric, called SemVarEffect, to evaluate the causality of semantic variations between inputs and outputs of T2I models.
Our approach uses inputs’ semantics as the only intervention to evaluate the average causal effect (ACE) of this intervention on outputs’ semantic variations, that is, the contribution of inputs to outputs.
A significant ACE would indicate that the T2I model can effectively capture and reflect input semantic variations.
On the contrary, a small ACE, such as the 0.09 shown in Fig. 2,
exposes a considerable weakness in the T2I model’s ability to understand and respond to sentence semantics.

To facilitate the evaluation, we present a new benchmark, called SemVarBench.
To avoid overt literal differences, semantic variations are achieved through two types of linguistic permutations Gerner (): permutation-variance, where different word orders result in different meanings, and permutation-invariance, where the meaning remains unchanged regardless of word orders.
Utilizing pre-defined templates and rules as the guidance in the generation stage, followed by a large amount of annotation and hard sample selection in the validation stage, we constructed a benchmark comprising 11,454 samples, where 10,806 are in the training set and 648 are in the test set.
We experimented with a variety of T2I models using our proposed benchmark and metric. The results show that even SOTA models like CogView-3-Plus and Ideogram 2 struggle, achieving scores far from the ideal, which highlights the need for further advancements in handling semantic variations.

Our contributions are summarized as follows:
(1) We are the first to conduct a comprehensive investigation into the problem of semantic variations in T2I models.
(2) We propose SemVarEffect, a novel metric specifically designed to measure the causality between semantic variations in inputs and outputs of T2I synthesis.
(3) We propose SemVarBench, a high-quality, expert-annotated benchmark for evaluating semantic variations in T2I synthesis, avoiding predictable literal differences and focusing on two key linguistic permutations: permutation-variance and permutation-invariance. This benchmark sets a new standard for evaluating T2I models on semantic understanding.
(4) We conduct a comprehensive benchmarking of SOTA T2I models on SemVarBench, revealing significant limitations in their handling of semantic variations. Our findings suggest several areas for improvement, including text encoding and cross-modal semantic alignment techniques, and offer insights into the challenges posed by different types of semantic variations.

## 2 Semantic Variation Evaluation for Text-to-Image Synthesis

### 2.1 Preliminary

The T2I model $f$ generates images $I$ for each input sentence $T$, represented as
$I=f(T)$
.
$S(T,I)$
is the text-image alignment score, measuring text-image similarity.
$S(\cdot)$
represents the scoring method.

Linguistic Permutation.
Linguistic permutation refers to changes in word order. Given an anchor sentence $T_{a}$, $T_{pv}$ and $T_{pi}$ are two permutations of $T_{a}$.
$T_{pv}$ exemplifies permutation-variance, which shows a change in meaning, while $T_{pi}$ exemplifies permutation-invariance, where the meaning remains unchanged.
The expected $I_{pv}$ is a permutation of objects or relations from $I_{a}$, while $I_{pi}$ is semantically equivalent to $I_{a}$, preserving the same visual objects and relations after transformation.

### 2.2 Definition of Visual Semantic Variations

First, we define the visual semantic variations observed from a single sentence $T$. For each $I$ and its localized variation $I+\Delta I$ in the image space,
the visual semantic variation at $I$, denoted as $\mu_{I}(T,I)$,
is the difference in alignment scores between the two images for the same sentence:
$\mu_{I}(T,I)=S(T,I+\Delta I)-S(T,I)$
.
If the anchor image $I_{a}$ is transformed into a permutation image $I_{p}$ through a series of localized changes, the total visual semantic variation from $I_{a}$ to $I_{p}$ is the sum of variations across all localized changes:
$\sum_{I_{a}}^{I_{p*}}\mu_{I}(T,I)=S(T,I_{p*})-S(T,I_{a})$
.

Second, we integrate the visual semantic variations observed across multiple sentences.
For the sentence $T_{a}$, the visual semantic variations
$\sum_{I_{a}}^{I_{p*}}\mu(T_{a},I)$
demonstrate a shift from a matched to a mismatched image-text pair, indicating a negative change. For the sentence $T_{p*}$, the visual semantic variations
$\sum_{I_{a}}^{I_{p*}}\mu(T_{p*},I)$
demonstrate a shift from a mismatched to a matched image-text pair, indicating a positive change.
To measure the total magnitude of these variations regardless of direction, we use the absolute values.
Therefore, the integrated visual semantic variations $\gamma^{I}$ is defined as:

$$ $\centering\begin{aligned} \gamma^{I}=\textstyle\sum_{T\in\{T_{a},T_{p*}\}} \left|\sum_{I_{a}}^{I_{p*}}\mu(T,I)\right|=\left|S(T_{a},I_{p*})-S(T_{a},I_{a} )\right|+\left|S(T_{p*},I_{p*})-S(T_{p*},I_{a})\right|.\end{aligned}\@add@centering$ (1) $$

### 2.3 The Causality between textual and visual semantic variations

Fig. 3 illustrates the causal relationship between input and output semantic variations. $T$ is the text input, serving as the input variable, while $I$ is the generated image, acting as a mediator. $S$ is the text-image alignment score, influenced by both $T$ and $I$, and serves as an intermediate result variable. $\gamma^{I}$ denotes visual semantic variation and is the final comparison result variable. $f(\cdot)$ is an exogenous variable representing a T2I model that maps $T$ to $I$. $S(\cdot)$ is an exogenous variable representing a scoring function that maps $T$ and $I$ to $S$.
The dashed line between $S$ and $\gamma^{I}$ indicates their derived relationship: $\gamma^{I}$ is the difference between two $S$ values under different output conditions, representing the comparative result of the alignment scores when the image changes.

According to the causal inference theory, we define the average causal effect (ACE) of textual semantic variations on visual semantic variations as the SemVarEffect score.
As shown in Fig. 3,
the sentence $T$ serves as an independent variable that influences the generated image $I$. The visual semantics variations is jointly influenced by $T$, $I$ and $S(\cdot)$.
Let
$\text{do}(T\neq T_{a})$
and
$\text{do}(T=T_{a})$
represent two types of interventions.
$\text{do}(T\neq T_{a})$
represents an intervention where $T$ differs in meaning from the anchor sentence $T_{a}$. The visual semantic variation caused by this intervention is denoted as:
Figure 3: Causal relationship between the input and the output semantic variations.
$\begin{aligned} \gamma_{w/}^{I}&=E[\gamma^{I}\mid\text{do}(T\neq T_{a})]=E[
\gamma^{I}\mid T=T_{pv}]\\
&=\left|S(T_{a},I_{pv})-S(T_{a},I_{a})\right|+\left|S(T_{pv},I_{pv})-S(T_{pv},
I_{a})\right|.\end{aligned}$
(2)
$\text{do}(T=T_{a})$
represents an intervention where $T$ match the meaning of the anchor sentence $T_{a}$.
The visual semantic variation caused by this intervention is denoted as:
$\begin{aligned} \gamma_{w/o}^{I}&=E[\gamma^{I}\mid\text{do}(T=T_{a})]=E[\gamma
^{I}\mid T=T_{pi}],\\
&=\left|S(T_{a},I_{pi})-S(T_{a},I_{a})\right|+\left|S(T_{pi},I_{pi})-S(T_{pi},
I_{a})\right|.\end{aligned}$
(3)
By comparing the visual semantic variations under the two interventions—one changing the meaning and the other maintaining it—we determine the ACE of textual semantic variations on visual semantic variations:
$\begin{aligned} \kappa&=E[\gamma^{I}\mid\text{do}(T\neq T_{a})]-E[\gamma^{I}
\mid\text{do}(T=T_{a})]=\gamma_{w/}^{I}-\gamma_{w/o}^{I}\\
&=\left|S(T_{a},I_{pv})-S(T_{a},I_{a})\right|+\left|S(T_{pv},I_{pv})-S(T_{pv},
I_{a})\right|\\
&-\left|S(T_{a},I_{pi})-S(T_{a},I_{a})\right|-\left|S(T_{pi},I_{pi})-S(T_{pi},
I_{a})\right|,\end{aligned}$
(4)

The SemVarEffect score $\kappa$ quantifies the influence of input semantic variations on output semantic variations.
The alignment score consists of object and relation (triple) components, each contributing up to 0.5 to the total score.
Under ideal conditions, where $f(\cdot)$ accurately represents the text through images and $S(\cdot)$ faithfully measures text-image alignment, $\kappa$ ranges from 0 to 1.
$\kappa$ is maximized when $\gamma_{w/}^{I}$ reaches its upper bound of 1, which occurs in extreme cases where no relation between objects are identical, and $\gamma_{w/o}^{I}$ reaches its optimal value of 0.
More detailed analysis of the SemVarEffect score can be found in Appendix B.3– B.5.

## 3 Semantic Variation Dataset for Text-to-Image Synthesis

Figure: Figure 4: The data collection process of SemVarBench. Top: Templates. Bottom: Generated Sentences. The templates are extracted from the seed pair “a dog is using a wheelchair and the dog is next to a person”/“a person is using a wheelchair and the person is next to a dog”.
Refer to caption: x4.png

We collect semantic variation datasets for T2I synthesis to fill the gaps in current benchmarks and evaluation practices. The textual semantic variations are created through two typical linguistic permutations.
First, we elaborate on the characteristics of the data. Then, we introduce the pipeline for data collection, annotation and statistics.

### 3.1 Characteristics of Data

Each sample $(T_{a},T_{pv},T_{pi})$ consists of three sentences: an anchor sentence $T_{a}$ and two permutations $T_{pv}$ and $T_{pi}$. They should adhere to the following characteristics:

Literal Similarity: $T_{a}$, $T_{pv}$ and $T_{pi}$ are
literally similar, differing only in word order.

Distinct Semantics: $T_{a}$ and $T_{pv}$ have distinct semantics. $T_{a}$ and $T_{pi}$ share the same semantics.

Reasonability: $T_{a}$, $T_{pv}$ and $T_{pi}$ are semantically reasonable in either the real or fictional world.

Visualizability: $T_{a}$, $T_{pv}$ and $T_{pi}$ evoke vivid mental images.

Discrimination: The images evoked by $T_{a}$ and $T_{pv}$ present distinguishable differences. The images evoked by $T_{a}$ and $T_{pi}$ appear similar.

Recognizability: The image evoked by $T_{a}$, $T_{pv}$ and $T_{pi}$ maintain key elements necessary for recognizing typical scenes and characters.

### 3.2 Data Collection

We use LLMs (GPT-3.5) to generate anchor sentences and their permutations, guided by templates.
However, LLMs tend to produce patterns common in their training data, which leads to the neglect of less common combinations specified by templates and rules. To address this issue, we employ a different process for generating $T_{a}$, $T_{pv}$ and $T_{pi}$.

Template Acquisition.
We choose all 171 sentence pairs suitable for T2I synthesis from Winoground Thrush et al. (); Diwan et al. () as seed pairs.
These pairs are used to extract templates and rules for $T_{a}$ and $T_{pv}$, while those for $T_{pi}$ are extended manually.
To increase diversity, we change the word orders according to the part of speech, including number, adjective, adjective phrase, noun, noun with adjective, noun with clause, noun with verb, noun with prepositional phrase, verb, verb with adverb, adverb, prepositional and prepositional phrase.
In Fig. 4, the top left shows an example of templates for $T_{a}$ and $T_{pv}$ derived from extraction, while the top right shows the corresponding templates for $T_{a}$ and $T_{pi}$ derived from manual completion.

Template-guided Generation for $T_{a}$.
We use LLMs to generate anchor sentences by filling template slots based on prior knowledge and maximum likelihood estimation.
In Fig. 4, the bottom middle sentence $T_{a}$ is generated using the template for $T_{a}$ as a guide.

Rule-guided Permutation for $T_{pv}$.
$T_{pv}$ is generated by swapping or rearranging words in $T_{a}$ based on predefined rules, ensuring that $T_{pv}$ introduces semantic variation. This method avoids a random generation or a semantically equivalent passive structure to $T_{a}$, which a common pitfall in autonomous generation by LLMs.
By following these rules, $T_{pv}$ includes many rare combinations not commonly found in existing NLP corpora.
In Fig. 4, $T_{pv}$ is generated by swapping [Noun1] and [Noun3] in $T_{a}$ (shown in the top left).

Paraphrasing-guided Permutation for $T_{pi}$. $T_{pi}$ can be generated by following rules, such as exchanging phrases connected by coordinating conjunctions.
However, not all sentences contain coordinating conjunctions, so we also allow other synonymous transformations, including passive voice and slight rephrasing.
Both $T_{pi}$ examples in Fig. 4 are acceptable.

### 3.3 Data Annotation and Statistics

LLM and Human Annotation.
We establish 14 specific criteria to define what constitutes a “valid” input sample.
LLMs check each sample against these criteria, labeling them as “yes” or “no” with confidence scores. Samples labeled “no” with confidence scores above 0.8 are removed.
Then, 15 annotators and 3 experienced experts manually verify the remaining samples.
Each sample is independently reviewed by two annotators, with an expert resolving any disagreements.
After expert verification, we obtained 11,454 valid, non-duplicated samples.
To rigorously evaluate T2I models, 684 challenging samples were selected based on thresholds and voting for the test set.
More details on annotation and selection are provided in Appendix C.2.

Scale and Split.
SemVarBench comprises 11,454 samples of $(T_{a},T_{pv},T_{pi})$,
divided into a training set and a test set.
The training set contains 10,806 samples, while the test set consists of 648 samples.
All our evaluations are conducted on the test set.

Category.
In SemVarBench, samples are divided into 20 categories based on their types of semantic variation.
These categories are further classified into three aspects: Relation, Attribute Comparison, and Attribute Values.
Fig. 5 shows the distribution of the test set in SemVarBench.

Figure: Figure 5: Distribution of semantic variations by category in the semVarBench test set.

## 4 Experiments

### 4.1 Experimental Setup

**Table 1: Diffusion Models to be evaluated. #DIM represents the pooled dimension of text encoders’ outputs. #TEP, #IGP, #IDP, #ToP represent the parameters of text encoders, image generators, image decoders and whole models.**
| Model | Abbr. | Type | #DIM | Text<br>Encoder | #TEP | Image<br>Generator | #IGP | Image<br>Decoder | #IDP | #ToP |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Open-source Models |  |  |  |  |  |  |  |  |  |  |
| Stable Diffusion v1.5<br>Rombach et al. () | SD 1.5 | Diffusion | 768 | CLIP ViT-L | 123.06M | UNet | 859.52M | VAE | 83.65M | 1.07B |
| Stable Diffusion v2.1<br>Rombach et al. () | SD 2.1 | Diffusion | 1024 | OpenCLIP ViT-H | 340.39M | UNet | 865.91M | VAE | 83.65M | 1.29B |
| Stable Diffusion XL v1.0<br>Podell et al. () | SD XL 1.0 | Diffusion | 2048 | CLIP ViT-L &<br>OpenCLIP ViT-bigG | 123.06M<br>694.66M | UNet | 4.83B | VAE | 83.65M | 6.51B |
| Stable Cascade<br>Pernias et al. () | SD CA | Diffusion | 1280 | CLIP ViT-G | 694.66M | UNet | 5.15B | VQGAN | 18.41M | 6.86B |
| DeepFloyd IF XL<br>Saharia et al. () | DeepFloyd | Diffusion | 4096 | T5-XXL | 4.76B | UNet | 6.02B | VAE | 55.33B | 11.18B |
| PixArt-alpha XL<br>Chen et al. () | PixArt | Diffusion | 4096 | Flan-T5-XXL | 4.76B | Transformer | 611.35M | VAE | 83.65M | 5.46B |
| Kolors<br>Team () | Kolors | Diffusion | 4096 | ChatGLM3 | 6.24B | UNet | 2.58B | VAE | 83.65M | 8.91B |
| Stable Diffusion 3[medium]<br>Esser et al. () | SD 3 | Diffusion | 2048 | CLIP ViT-L &<br>OpenCLIP ViT-bigG &<br>T5-XXL | 117.92M<br>662.48M<br>4.76B | Transformer | 2.03B | VAE | 83.82M | 7.69B |
| FLUX.1[dev] | FLUX.1 | Diffusion | 768 | CLIP ViT-L &<br>T5-XXL | 123.06M<br>4.76B | Transformer | 11.90B | VAE | 83.82M | 16.87B |
| API-based Models |  |  |  |  |  |  |  |  |  |  |
| Midjourney V6 | MidJ V6 | Diffusion | – | – | – | – | – | – | – | – |
| DALL-E 3<br>Betker et al. () | DALL-E 3 | Diffusion | – | T5-XXL | 4.76B | UNet | – | VAE | – | – |
| CogView-3-Plus | CogV3-Plus | Diffusion | – | T5-XXL^1 | 4.76B^1 | Transformer | – | VAE^1 | – | – |
| Ideogram 2 | Ideogram 2 | Diffusion | – | – | – | – | – | – | – | – |

T2I Synthesis Models. We evaluate 13 mainstream T2I models as shown
in Tab.  1.
For each sentence, we generate one image, resulting in a total of 684 $\times$ 3 $\times$ 13 images.
Each input prompt is the sentence itself, without any negative prompts or additional details expanded by prompt generators.

Evaluators.
We use 4 advanced MLLMs as the evaluators to calculate text-image alignment scores: Gemini 1.5 Pro, Claude 3.5 Sonnet, GPT-4o and GPT-4 Turbo. The latter two have demonstrated near-human performance in evaluating the text-image alignment in T2I synthesis Zhang et al. (); Chen et al. ().
We format a sentence and an image in a prompt and feed it into the evaluator, asking it to assign two scores: object accuracy (0-50 points) and relation accuracy (0-50 points).
The sum of these two scores is treated as the total score, which is then normalized to $[0,1]$.

Metrics.
We use 4 metrics for evaluation: the alignment score $\bar{S_{ii}}$, the visual semantic variation scores
$\gamma_{w/}^{I}$
and
$\gamma_{w/o}^{I}$
under different interventions as defined in Eq. 2 and Eq. 3, and the SemVarEffect score $\kappa$ as defined in Eq. 4.
For each sample,
$\bar{S_{ii}}=\frac{1}{\left|K\right|}\sum_{i\in K}S(T_{i},I_{i})$
, where
$K=\{a,pv,pi\}$
.
A T2I model with high $\bar{S_{ii}}$,
$\gamma_{w/}^{I}$
and $\kappa$, and low
$\gamma_{w/o}^{I}$
indicates a strong understanding of semantic variations.
For simplicity, we refer to $\bar{S_{ii}}$ as $\bar{S}$,
$\gamma_{w/}^{I}$
as $\gamma_{w}$, and
$\gamma_{w/o}^{I}$
as $\gamma_{wo}$ in the following sections.

Evaluation Dataset.
We evaluate T2I models on the test set in a zero-shot manner.
To demonstrate the improvements from fine-tuning, we collected sentences and their generated images from the training set, selecting only those with high quality, high discrimination, and consistent variations as the training data.
Details about the selection of the training data are provided in Appendix D.3.

### 4.2 Results

**Table 2: Evaluation results of different T2I models in understanding semantic variations. Bold indicates the best performance, and underline indicates the second best. Blue indicates average SemVarEffect scores between 0.05 and 0.10, and green indicates scores above 0.10.**
| Models | Gemini 1.5 Pro | Claude 3.5 Sonnet | GPT-4o | GPT-4 Turbo |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| $\bar{S}(\uparrow)$ | $\gamma_{w}(\uparrow)$ | $\gamma_{wo}(\downarrow)$ | $\kappa(\uparrow)$ | $\bar{S}(\uparrow)$ | $\gamma_{w}(\uparrow)$ | $\gamma_{wo}(\downarrow)$ | $\kappa(\uparrow)$ | $\bar{S}(\uparrow)$ | $\gamma_{w}(\uparrow)$ | $\gamma_{wo}(\downarrow)$ | $\kappa(\uparrow)$ | $\bar{S}(\uparrow)$ | $\gamma_{w}(\uparrow)$ | $\gamma_{wo}(\downarrow)$ | $\kappa(\uparrow)$ |  |
| Open-source Models |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| SD 1.5 | 0.55 | 0.43 | 0.46 | -0.03 | 0.64 | 0.19 | 0.20 | -0.01 | 0.63 | 0.34 | 0.33 | 0.01 | 0.65 | 0.32 | 0.32 | 0.00 |
| SD 2.1 | 0.58 | 0.45 | 0.46 | -0.01 | 0.66 | 0.21 | 0.20 | 0.01 | 0.65 | 0.33 | 0.31 | 0.02 | 0.68 | 0.35 | 0.34 | 0.01 |
| SD XL 1.0 | 0.62 | 0.39 | 0.39 | -0.00 | 0.69 | 0.19 | 0.18 | 0.00 | 0.71 | 0.31 | 0.28 | 0.03 | 0.72 | 0.32 | 0.28 | 0.03 |
| SD CA | 0.59 | 0.42 | 0.41 | 0.01 | 0.69 | 0.19 | 0.18 | 0.01 | 0.67 | 0.31 | 0.31 | -0.00 | 0.69 | 0.32 | 0.31 | 0.01 |
| DeepFloyd | 0.64 | 0.44 | 0.44 | 0.00 | 0.71 | 0.20 | 0.19 | 0.01 | 0.69 | 0.33 | 0.30 | 0.03 | 0.74 | 0.33 | 0.28 | 0.05 |
| PixArt | 0.60 | 0.35 | 0.32 | 0.02 | 0.69 | 0.17 | 0.15 | 0.02 | 0.70 | 0.29 | 0.26 | 0.03 | 0.71 | 0.29 | 0.27 | 0.02 |
| Kolors | 0.60 | 0.41 | 0.42 | -0.01 | 0.69 | 0.22 | 0.22 | -0.01 | 0.69 | 0.31 | 0.30 | 0.01 | 0.69 | 0.33 | 0.30 | 0.02 |
| SD 3 | 0.67 | 0.45 | 0.40 | 0.05 | 0.76 | 0.23 | 0.19 | 0.04 | 0.75 | 0.36 | 0.29 | 0.07 | 0.76 | 0.33 | 0.28 | 0.05 |
| FLUX.1 | 0.72 | 0.43 | 0.35 | 0.08 | 0.75 | 0.23 | 0.17 | 0.06 | 0.72 | 0.42 | 0.33 | 0.10 | 0.75 | 0.40 | 0.30 | 0.10 |
| API-based Models |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| MidJ V6 | 0.68 | 0.46 | 0.39 | 0.07 | 0.73 | 0.24 | 0.21 | 0.03 | 0.72 | 0.40 | 0.33 | 0.07 | 0.73 | 0.38 | 0.32 | 0.06 |
| DALL-E 3 | 0.75 | 0.46 | 0.33 | 0.14 | 0.80 | 0.25 | 0.18 | 0.06 | 0.82 | 0.36 | 0.22 | 0.13 | 0.83 | 0.35 | 0.30 | 0.10 |
| CogV3-Plus | 0.79 | 0.52 | 0.35 | 0.17 | 0.80 | 0.28 | 0.18 | 0.10 | 0.81 | 0.49 | 0.28 | 0.20 | 0.82 | 0.43 | 0.26 | 0.17 |
| Ideogram 2 | 0.80 | 0.47 | 0.29 | 0.18 | 0.79 | 0.26 | 0.17 | 0.09 | 0.81 | 0.46 | 0.27 | 0.20 | 0.81 | 0.40 | 0.24 | 0.15 |

The results of the influence of inputs semantic variations on outputs semantic variations in T2I synthesis are shown in Tab. 2.
The scores for $\bar{S}$ range between 0.6 and 0.8.
Despite the alignment score $\bar{S}$ reaching up to 0.8, this does not imply a strong grasp of semantics.
The following three metrics provide a more comprehensive view of the model’s ability to handle semantic variations.

Visual Semantic Variation with Changed Textual Semantics.
As shown in Tab. 2, the values of $\gamma_{w}$ are all below 0.52 for all evaluators, significantly lower than the optimal value of 1.
This indicates that none of the T2I models perform at an acceptable level. These models are highly insensitive to semantic variations.
This finding aligns with the widely accepted notion that T2I models tend to treat input text as a collection of isolated words
, leading them to interpret sentences with minor changes in word order as having the same meaning.

Visual Semantic Variation with Unchanged Textual Semantics.
The values of $\gamma_{wo}$ in Tab. 2
are unexpectedly much higher than the optimal value of 0.
Only the models highlighted in blue and green demonstrate slightly better performance, with $\gamma_{wo}$ scores consistently lower than $\gamma_{w}$.
These T2I models illustrate potential semantic variations caused by word order through images, yet they still struggle to accurately differentiate between inputs with varying meanings and those with invariant meanings.
These models primarily understand language based on word order rather than the underlying semantics.

Influence of Textual Semantics on Visual Semantic Variations.
In Tab. 2, the $\kappa$ values for all evaluators are below 0.20, indicating considerable room for improvement in T2I models’ understanding of semantic variations.
Models with higher alignment scores are more sensitive to semantic variations caused by word orders. However, models highlighted in blue overreact to permutations maintaining the meanings, resulting in higher $\gamma_{wo}$ values and subsequently lower $\kappa$ values. These models excel at capturing common alignments but struggles to handle semantic variations.

### 4.3 Analysis

Figure: Figure 6: Illustration of the text embedding similarity between the anchor text and the permuted text. Squares represent permutation-variance results (with changed textual semantics), while triangles represent permutation-invariance results (with unchanged textual semantics). The evaluator is GPT-4o. (a) The alignment score between the anchor image $I_{a}$ and a permutation $T_{p*}$ decreases as the text similarity between $T_{a}$ and $T_{p*}$ increases. (b) The semantic variation score $\gamma$ increases as the text similarity between $T_{a}$ and $T_{p*}$ increases. The cosine similarity for DALL-E 3, an API-driven model, is deduced using T5-XXL, indicated by hollow shapes.
Refer to caption: x6.png

Is a superior text encoder the exclusive solution for T2I models to grasp semantic variations?
We explore the relationship between the text encoder’s ability to discriminate semantic variations and the ability of two metrics—alignment scores $\bar{S}$ and visual semantic variation scores $\gamma$—to do the same, as illustrated in Fig. 6. We use text similarity(^2^22 Sentences for changed textual semantics unexpectedly show higher text similarity than those for unchanged textual semantics, likely due to the edit distance between our sentences. For further analysis, see Appendix F.) to measure the text encoder’s discriminative capability for semantic variations.
T2I models like PixArt and Kolors, which utilize T5 and ChatGLM as text encoders, fail to transfer the results of distinguishing semantic variations to image generators, as shown by permutation-variance (indicated by squares).
However, T2I models like FLUX.1, which utilize weaker CLIP-T5 hybrid models as text encoders, achieve higher alignment scores and greater differentiation in visual semantic variation scores, despite showing minimal changes in text similarity.
These results indicate that a model’s ability to distinguish semantic variations is not only dependent on the text encoder, and that further efforts are needed in cross-modal alignment to effectively transfer these differences to the image generators.

Figure: Figure 7: The distribution of categories across different T2I models based on SemVarEffect scores $\kappa$. The evaluator is GPT-4 Turbo.

Figure: Figure 8: The distribution of SemVarEffect scores for the SOTA model Ideogram 2 across different aspects of the samples. The evaluator is GPT-4 Turbo.

Does the influence of input semantic variations on output semantic variations vary by category?
As shown in Fig. 7, the semantic variations in Color have significantly influenced the output of T2I models, with the SemVarEffect score consistently exceeding 0.4 in many models. In contrast, the SemVarEffect scores in other categories are mostly below 0.1.
This suggests that T2I models understand semantic variations well only in the case of Color.
We found that the SemVarEffect scores of Ideogram 2 in Relation, Attribute Comparison, and Attribute Value are 0.07, 0.13, and 0.19.
To compare the distribution of SemVarEffect scores across different aspects, we set 0.2 and 0.5 as thresholds.
As shown in Fig. 8, the proportion of high scores in Attribute Values is significantly higher than those in Relation and Attribute Comparison.
T2I models lack the capability to discriminate semantic variations,
particularly in aspects emphasizing relations and comparisons.
Fig. 9 shows failed examples in Relation and Comparison.
Although T2I models can generate correct images for common relations, they tend to rigidly adhere to these common relations even when semantic variations occur, leading to incorrect images.
More examples are provided in Appendix G.

Does fine-tuning improve T2I model performance on semantic variations?
We examine improvements from fine-tuning text encoders and image generators. We use samples
in the training set to generated images
and select text-images pairs with high alignment scores and high discriminability as training data,
details shown in Appendix D.3. As shown in Tab. 3, for categories with sufficient high-quality data, such as Color, supervised fine-tuning (SFT) enhanced the performance of the T2I model. However, in categories with insufficient high-quality data, such as Direction, SFT led to a decline in performance. Additionally, direct preference optimization (DPO) resulted in performance drops due to failures in permutation-invariance, as evidenced by the increased $r_{wo}$.

It is crucial to strike a balance between sensitivity and robustness to semantic changes, as this determines whether performance can be enhanced. However, fine-tuning tends to improve sensitivity at the expense of robustness.
While T2I models become more sensitive to permutations with different meanings, this discrimination is quickly disrupted by over-sensitivity to permutations with similar meanings, leading to a decline in the model’s overall ability to discern differences.

This phenomenon may be linked to word-level alignment in the cross-attention mechanism and a lack of semantic-level constraints. Fine-tuning only improves alignment at the word level, rather than enhancing the understanding of semantic variations.
The samples from permutation-variance are inherently hard negative samples, as they differ only in word order. This confuses the models and leads to performance declines, especially during DPO.
Fine-tuning T2I models to better understand semantic variations caused by word order remains a formidable challenge.

**Table 3: Fine-tuned SD XL Results: Performance varies based on the quantity of high-quality samples, which are determined by category and sample size. Left Table: Color and Absolute Location with 4.4k and 1.7k candidates for training. Right Table: Height and Direction with 0.2k and 0.3k candidates for training.**
| Category | Models | GPT-4o |  |  |  |
| --- | --- | --- | --- | --- | --- |
| $\bar{S}(\uparrow)$ | $\gamma_{w}(\uparrow)$ | $\gamma_{wo}(\downarrow)$ | $\kappa(\uparrow)$ |  |  |
| Color | SD XL | 0.73 | 0.33 | 0.25 | 0.08 |
| + sft-unet | 0.78($\uparrow$) | 0.38($\uparrow$) | 0.20($\downarrow$) | 0.18($\uparrow$) |  |
| + sft-text | 0.73($-$) | 0.40($\uparrow$) | 0.27($\uparrow$) | 0.13($\uparrow$) |  |
| + dpo-unet | 0.69($\downarrow$) | 0.43($\uparrow$) | 0.27($\uparrow$) | 0.17($\uparrow$) |  |
|  | + dpo-text | 0.68($\downarrow$) | 0.47($\uparrow$) | 0.29($\uparrow$) | 0.18($\uparrow$) |
| Absolute<br>Location | SD XL | 0.64 | 0.29 | 0.34 | -0.05 |
| + sft-unet | 0.65($\uparrow$) | 0.34($\uparrow$) | 0.32($\downarrow$) | 0.02($\uparrow$) |  |
| + sft-text | 0.64($-$) | 0.31($\uparrow$) | 0.36($\uparrow$) | -0.05($-$) |  |
| + dpo-unet | 0.60($\downarrow$) | 0.29($\uparrow$) | 0.31 ($\downarrow$) | -0.02($\uparrow$) |  |
|  | + dpo-text | 0.57($\downarrow$) | 0.33($\uparrow$) | 0.39 ($\uparrow$) | -0.07($\downarrow$) |

Figure: Figure 9: Failed examples of DALL-E 3 on Relation and Attribute Comparison.
Refer to caption: x10.png

## 5 Related Work

Evaluation of T2I synthesis.
Benchmarks of T2I synthesis primarily focus on general alignment Saharia et al. (); Yu et al. (); Cho et al. (), composition Park et al. (); Feng et al. (); Park et al. (); Hu et al. (); Cho et al. (); Li et al. (), bias and fairness Lee et al. (); Luo et al. (; ), common sense Fu et al. () and creativity Lee et al. (). In these evaluations, the quality of images is measured by detection-based or alignment-based metrics. Detection-based metrics evaluate the accuracy of object detection models applied to the generated images,
while alignment-based metrics evaluate how well the visual content matches the semantic meaning of the text.
Recent research on T2I synthesis has explored samples involving semantic variations caused by word orders, typically using them to evaluate reasoning abilities with alignment-based metrics Marcus et al. (); Lee et al. (); Li et al. ().
However, a significant gap in this research is the underexplored area of whether the generated images consistently represent subtle but important semantic variations within the input text.

Semantic Variation Evaluation in VLMs.
In VLMs, semantic variations caused by word order has been evaluated by benchmarks like Winoground Thrush et al. () and its expansion in specific domain Burapacheep et al. (). Winoground is designed to challenge models with visio-linguistic compositional reasoning.
It requires models to accurately match two images to their respective captions, where the two captions are different permutations of the same set of words, resulting in different meanings.
To enhance performance on Winoground, studies have focused on expanding training datasets with negative samples
and optimizing training strategies to handle the resulting semantic variations Yüksekgönül et al. (); Hsieh et al. (); Burapacheep et al. ().

The application of Winoground to T2I synthesis faces several limitations due to the variety and quantity of its permutations.
First, the dataset, with 400 sentence pairs, provides only 171 suitable for text-image composition analysis Diwan et al. (), where samples are classified into three categories: object, relation, and both.
This limited variety is insufficient for a comprehensive evaluation of T2I models.
Second, the suitability of certain samples for T2I model evaluation is problematic.
Winoground primarily focuses on semantic distinctiveness for cross-modal retrieval Yüksekgönül et al. (); Ma et al. (); Cascante-Bonilla et al. (). It overlooks the criteria essential for T2I synthesis, such as sentence completeness, clarity of expression, unambiguity, and specificity in referencing image elements. All of these factors have been carefully considered in the quality control of our benchmark annotations.

## 6 Conclusion

We comprehensively study the challenge of semantic variations in T2I synthesis, specifically focusing on causality between semantic variations of inputs and outputs. We propose a new metric, SemVarEffect, to quantify the influence of input semantic variations on model outputs, and a novel benchmark, SemVarBench, designed to examine T2I models’ understanding of semantic variations.
Our experiments reveal that SOTA T2I models, including CogView-3-Plus and Ideogram 2, struggle with semantic variations, with most scoring below 0.2 on our benchmark. This indicates that these models have yet to develop the capability to effectively handle such variations. Fine-tuning efforts also show limited success, improving sensitivity to certain variations but at the cost of robustness.
These findings highlight the importance of our metric and benchmark in addressing this challenge.
Future work should focus on enhancing cross-modal alignment to better manage subtle semantic changes and improve overall T2I model performance.

## Appendix A Four Types of Semantic Variations Results in T2I Synthesis

The results of semantic variations in T2I synthesis, both in text and images, can be divided into four types, as shown in Fig. 10.

- •
Image Changing Semantics with Text Changing Semantics: The semantic consistency between the input and output in the first quadrant suggests that the model tends to understand the different semantics introduced by linguistic permutations. In this case, the value of $\gamma_{w/}$ will tend to 1.
- •
Image Maintaining Semantics with Text Changing Semantics: The semantic inconsistency between the input and output in the fourth quadrant suggests that the model does not understand the different semantics introduced by linguistic permutations. In this case, the value of $\gamma_{w/}$ will tend to 0.
- •
Image Changing Semantics with Text Maintaining Semantics: The semantic consistency between the input and output in the third quadrant suggests that the model tends to understand the similar semantics introduced by linguistic permutations. In this case, the value of $\gamma_{w/o}$ will tend to 0.
- •
Image Maintaining Semantics with Text Maintaining Semantics: The semantic inconsistency between the input and output in the second quadrant suggests that the model does not understand the similar semantics introduced by linguistic permutation. In this case, the value of $\gamma_{w/o}$ will tend to 1.

Figure: Figure 10: Four types of semantic variation results in T2I synthesis. The images with a red border represent an incorrect output. The images with a green border represent a correct output.
Refer to caption: x15.png

## Appendix B Properties of SemVarEffect

### B.1 Preliminary

A T2I generation model $f$ consists of one or more text encoders, image generators, and an image decoder.
The T2I generation model $f$ generates images $I\in\mathcal{I}$ for each input textual prompt $T\in\mathcal{T}$. $\mathcal{T}$ represents the textual space, and $\mathcal{I}$ represents the visual space. $S(T,I)$ is the alignment score between $T$ and $I$.

Let $T_{a}$ be an anchor textual prompt. Let $T_{p*}$ represent a permutation of $T_{a}$, where $T_{pv}$ is a permutation with a different meaning than $T_{a}$, and $T_{pi}$ is a permutation with the same meaning as $T_{a}$. Let $I_{a}$, $I_{pv}$, and $I_{pi}$ be the resulting images generated by a T2I model from $T_{a}$, $T_{pv}$, and $T_{pi}$, respectively. We expect that $I_{p*}$ should be a rearrangement of the objects or relations found within $I_{a}$.

### B.2 Textual vs. Visual Semantic Variations

The measurement of semantic variations in the transition from $(T_{a},I_{a})$ to $(T_{p*},I_{p*})$ can be defined from two perspectives:
(1) textual semantic variations $r^{T}$: The semantic changes in the text, observed through differences between images $I_{a}$ and $I_{p*}$, and (2) visual semantic variations $r^{I}$: The semantic changes in the images, observed through differences between texts $T_{a}$ and $T_{p*}$.

Specifically, we define the textual semantic variations observed from a single image $I$.
The initial anchor sentence $T_{a}$ is transformed into a permutation $T_{p*}$ after a series of localized linguistic permutations.
For each change from $T$ to $T+\Delta T$ in the text space, the textual semantic variation at position $T$, denoted as $\mu_{T}(T,I)$, is defined as the difference in alignment scores between the modified text and the original text with the image:
$\mu_{T}(T,I)=S(T+\Delta T,I)-S(T,I)$.
The textual semantic variation from $T_{a}$ to $T_{p*}$ is defined as the sum of the semantic variations produced by each localized permutation: $\sum_{T_{a}}^{T_{p*}}\mu_{T}(T,I)=S(T_{p*},I)-S(T_{a},I)$. Therefore, the integrated textual semantic variation is defined as $\gamma^{T}=\textstyle\sum_{I\in\{I_{a},I_{p*}\}}\left|\sum_{T_{a}}^{T_{p*}}\mu
(T,I)\right|$.

Similarly, we define the visual semantic variation observed from a single textual prompt $T$. Let $\mu_{I}(T,I)$ be the visual semantic variation observed from a given text $T$ in the image space.
The visual semantic variation from $I_{a}$ to $I_{p*}$ is defined as the sum of the visual semantic variations produced by each localized modification: $\sum_{I_{a}}^{I_{p*}}\mu_{I}(T,I)=S(I_{p*},T)-S(I_{a},T)$. Therefore, the integrated visual semantic variation is defined as
$\gamma^{I}=\textstyle\sum_{T\in\{T_{a},T_{p*}\}}\left|\sum_{I_{a}}^{I_{p*}}\mu
(T,I)\right|$.

We have not evaluated the influence by measuring the synchronicity of semantic changes between images and text, which has been applied in VLM Wang et al. (). This is because semantic variations introduce a unique challenge in the evaluation of T2I synthesis: images are not independent; they are influenced by both the input textual prompt and the inherent characteristics of the model, complicating the independent assessment of semantic changes.

Therefore, we conduct the evaluation by measuring the influence of external intervention on semantic variations within the corresponding images of T2I synthesis to avoid directly imposing interventions on images.

### B.3 Properties of Alignment Scores S 𝑆 S italic_S

Definition of Text-Image Alignment Score.
To facilitate semantic analysis, we structured these permutations by objects and triples.
Changing the word order affects the arrangement of objects or relations and leads to changes in syntactic dependencies and semantics.
Let $T_{p*}$ represent any permutation of $T_{a}$. $T_{a}$ and $T_{p*}$ share the same set of objects set $V$ and set of relations $R$. The triple set $E$ in $T_{a}$ is a subset of $V\times R\times V$.
Some triples in $T_{p*}$ may differ from those in $T_{a}$, but they have the same number of triples. For example, the initial triple set of $T_{a}$ contains (apple, on, box), (girl, touch, apple) and (girl, NULL, box). After swapping box and apple, the triple set of $T_{p*}$ contains (box, on, apple), (girl, touch, box) and (girl, NULL, apple).

To calculate the fine-grained alignment scores for objects and triples, we define the alignment score $S$ between $T$ and $I$ as the sum of the object and triple alignment scores:

$$ $S(T,I)=\sum_{i=1}^{\left|V\right|}S_{obj_{i}}(T,I)+\sum_{j=1}^{\left|E\right|} S_{tri_{j}}(T,I),$ (5) $$

where $\left|V\right|$ is the number of objects mentioned in $T$ and $\left|E\right|$ is the number of triples mentioned in $T$.
The components of the alignment score are defined as piecewise functions:

$$ $\displaystyle S_{obj_{i}}(T,I)=\begin{cases}w_{v_{i}}&\text{if the }i\text{-th object matches,}\\ 0&\text{if the }i\text{-th object does not match,}\end{cases}$ (6) $\displaystyle S_{tri_{j}}(T,I)=\begin{cases}w_{e_{j}}&\text{if the }j\text{-th triple matches,}\\ 0&\text{if the }j\text{-th triple does not match,}\end{cases}$ $$

where $w_{v_{i}}$ and $w_{e_{j}}$ are the weighted matching score for the $i$-th object and $j$-th triple.
We obtain the alignment function $S$ that satisfies the constraints in Eq. 10. Consequently, the alignment score of a matched text-image pair is calculated as:

$$ $\displaystyle S(T_{p*},I_{p*})=S(T_{a},I_{a})=\sum_{v_{i}\in V_{MA}}w_{v_{i}}+ \sum_{e_{j}\in E_{MA}}w_{e_{j}},$ (7) $$

where $V_{MA}$ and $E_{MA}$ represent the exactly matched objects and triples between a text prompt and its generated image, with $\left|V_{MA}\right|=\left|V\right|$ and $\left|E_{MA}\right|=\left|E\right|$.
The alignment score for a mismatched text-image pair is calculated as:

$$ $\displaystyle S(T_{p*},I_{a})=S(T_{a},I_{p*})=\sum_{v_{i}\in V_{MI}}w_{v_{i}}+ \sum_{e_{j}\in E_{MI}}w_{e_{j}},$ (8) $$

where $V_{MI}$ and $E_{MI}$ represent the partially matched objects and triples between a text prompt and a mismatched image, with $\left|V_{MI}\right|=\left|V\right|$ and $0\leq\left|E_{MI}\right|\leq\left|E\right|$.

Range of $S$.
If $f$ accurately depicts the text through images and $S$ faithfully measures the semantic changes between text space and image space, any alignment score $S(T,I)$ is bounded by:

$$ $\sum_{v_{i}\in V}w_{v_{i}}\leq S(T,I)\leq\sum_{v_{i}\in V}w_{v_{i}}+\sum_{e_{j }\in E}w_{e_{j}}.$ (9) $$

In our implementation, we set the value of $S(T,I)$ as an integer between 0 and 100, where the object accuracy is between 0 and 50 and the triple accuracy is in between 0 and 50. Then we normalize it into a real number within the range $[0,1]$. Based on the assumption of $f$ mentioned above, $0.5\leq S(T,I)\leq 1$.

However, limitations in the capabilities of the model $f(\cdot)$ and the alignment function $S(\cdot)$, often prevent the alignment score values from achieving the property in Eq. 10. For example, if a model $f$ generated a low-quality image $I$, it may fail to accurately depict all target objects ($V$), leading to $\left|V_{MA}\right|<\left|V\right|$ and $\left|V_{MI}\right|<\left|V\right|$. This results in an object accuracy below 0.5 (as illustrated in the bottom case of Fig. 21) and inconsistent relation accuracy (see cases in Fig. 16 and Fig. 17). Furthermore, inaccuracies in the scoring approach $S(\cdot)$ may incorrectly evaluate the similarity between text prompts and generated images, causing unpredictable fluctuations in semantic variation measurements (as illustrated in Fig. 23).

Identity Relation for $S$.
In the ideal scenario where $f$ accurately transforms all semantic variations from text space to image space, the alignment scores would satisfy the constraints:

$$ $S(T_{a},I_{a})\equiv S(T_{p*},I_{p*})\text{ and }S(T_{p*},I_{a})\equiv S(T_{a} ,I_{p*}).$ (10) $$

Eq. 10 is also demonstrated under the assumption that the alignment function is an equivariant map in the continuous textual feature space $\mathcal{T}$ and visual feature space $\mathcal{I}$, as detailed in Wang et al. ().
This assumption ensures that the alignment scores vary consistently with the semantic changes from images or text. The property is crucial for determining the characteristics of the data in SemVarBench and for designing the alignment functions.

However, the low-quality image $I$, resulting from a poorly performing model $f$, and the inaccuracies in the approach to scoring $S$, often prevent the alignment scores from meeting the criteria of Eq. 10. These limitations make the direct comparison of textual and visual semantic variation scores unreliable in T2I synthesis. This approach was previously explored in Wang et al. ().

### B.4 Property of Visual Semantic Variations γ I superscript 𝛾 𝐼 \gamma^{I} italic_γ start_POSTSUPERSCRIPT italic_I end_POSTSUPERSCRIPT

We analyze the theoretical relationship between visual semantic variations and model performance.
According to Eqs. 1, 7 and 8, we derive the visual semantic variations as follows:

$$ $\gamma^{I}=2\left|\sum_{v_{i}\in\{V_{MA}-V_{MI}\}}w_{v_{i}}+\sum_{e_{j}\in\{E_ {MA}-E_{MI}\}}w_{e_{j}}\right|.$ (11) $$

For both permutation-variance and permutation-invariance, the value of $\sum_{v_{i}\in\{V_{MA}-V_{MI}\}}w_{v_{i}}$ remains constant, which we denote as $C_{1}$. As a result, the visual semantic variation can be simplified to $\gamma^{I}=2\left|C_{1}+\sum_{e_{j}\in\{E_{MA}-E_{MI}\}}w_{e_{j}}\right|$, primarily depending on the size of the set $E_{MA}-E_{MI}$.
However, the size of the set varies dramatically between the two settings:

- •
In permutation-variance settings $(T_{a},T_{pv})$, the optimal value for the set $E_{MA}-E_{MI}$ is its maximum set $E$, resulting in an observed positive correlation between visual semantic variation $\gamma_{w/}^{I}$ and model performance.
- •
In permutation-invariance settings $(T_{a},T_{pi})$, the optimal value for the set $E_{MA}-E_{MI}$ is its minimum set $\emptyset$, resulting in an observed negative correlation between visual semantic variation $\gamma_{w/o}^{I}$ and model performance.

Therefore, we conclude that the visual semantic variations in permutation-variance and permutation-invariance differ significantly.

- •
A higher $\gamma_{w/}$ value indicates that the model effectively captures and reflects the intended semantic transformation in the input text.
- •
A lower $\gamma_{w/o}$ value indicates that the model maintains semantic consistency in the images despite variations in the input text.

### B.5 Property of SemVarEffect Score κ 𝜅 \kappa italic_κ

The SemVarEffect score on visual semantic variations, $\kappa$, is defined as the difference between $\gamma_{w/}^{I}$ and $\gamma_{w/o}^{I}$. It quantifies the model’s ability to discriminate between significant and negligible semantic changes in the text.

- •
If $\kappa$ is large, it suggests that the model is sensitive to semantic changes, recognizing variations in meaning. However, this does not necessarily imply strong alignment. The model might detect changes in semantics but still struggle to fully capture all objects and relationships described in the text, indicating a gap between sensitivity and complete alignment.
- •
If $\kappa$ is small or close to zero, it suggests that the model either fails to reflect meaningful semantic changes or overreacts to minor text variations. Regardless of the overall alignment score, the model may generate similar images even in the presence of significant semantic differences in the input text.

## Appendix C Construction Details

### C.1 Data Collection

Table: Table 4: Examples of extracted templates and transformation rules between templates of $(T_{a},T_{pv})$.

Template Acquisition
We designate 171 compositional cases in Winoground Thrush et al. (), which are labeled as “no-tag” in subsequent research Diwan et al. (), as $\text{SEED}_{0}$ and $\text{SEED}_{1}$.
The template of $T_{pv}$, the permutation with semantic changes from, is extracted from each pair of seeds by human annotators.
Then, we define the rule of $T_{pi}$, which is the permutation without semantic changes, as the original template of $T_{pi}$.
An examples is illustrated as follows.

$T_{a}$: [Noun1] [Verb] [Noun2] and [Noun1] behind [Noun3]
$T_{pv}$: [Noun3] [Verb] [Noun2] and [Noun3] behind [Noun1]
$T_{pi}$: [Noun1] behind [Noun3] and [Noun1] [Verb] [Noun2]
$T_{a}\rightarrow T_{pv}$: [Noun1] $\leftrightarrow$ [Noun3]
$T_{a}\rightarrow T_{pi}$: [Noun1] [Verb] [Noun2]$\leftrightarrow$[Noun1] behind [Noun3]

If there is no coordinating conjunction such as *and* and *while* for the template of $T_{pi}$, the template can be set *NULL*. In this case, the permutation $T_{pi}$ will be generated depends on the LLM according to other solutions.

$T_{a}$: [Noun1] [Verb1 (vi)] [Verb (vt)] [Noun2] [Verb2 (vi)]
$T_{pv}$: [Noun1] [Verb2 (vi)] [Verb (vt)] [Noun2] [Verb1 (vi)]
$T_{pi}$: NULL
$T_{a}\rightarrow T_{pv}$: [Verb1 (vi)] $\leftrightarrow$ [Verb2 (vi)]
$T_{a}\rightarrow T_{pi}$: NULL

Template-guided Generation for $T_{a}$. The prompt for generating the $T_{a}$, which is guided by the templates and seed pairs, is:

Assuming you are a linguist, you have the ability to create a similar sentence following the structure of given sentences.
The given two sentences are {$\text{SEED}_{0}$} and {$\text{SEED}_{1}$}. The structure of them are both “{$\text{Template }T_{a}$}”.
Please create a similar “{$\text{Template }T_{a}$}” sentence as “TEXT0”, and diversify your sentence as much as possible by using different themes, scenes, objects, predicate, verbs, and modifiers.
Output a list containing {NUM} json objects that contain the following keys: TEXT0. Use double quotes instead of single quotes for key and value. Now, let’s start. The output json object list:

Rule-guided Permutation for $T_{pv}$. The prompt for generating $T_{pv}$ based on $T_{a}$ and its corresponding rule is:

Assuming you are a linguist, you have the ability to judge the structure of existing sentences and imitate more new sentences with similar structure but varied content.
Step 1: Input some sentences structured by {$\text{Template }T_{a}$} and {$\text{Template }T_{pv}$}. We call each sentence as “TEXT0”.
Step 2: For each “TEXT0”, perform the change which is “{$\text{RULE of }T_{a}\rightarrow T_{pv}$}” and keep the other words unchanged as “TEXT1”.
For example, TEXT0={TEXT0}. Only swap/move {$\text{RULE of }T_{a}\rightarrow T_{pv}$} and keep the other words unchanged to generate TEXT1={TEXT1}.
Output a list containing {NUM} json objects that contain the following keys: TEXT0, TEXT1. Use double quotes instead of single quotes for key and value. Now, let’s start. The input is: {TEXT0}. The output json object list:

Paraphrasing-guided Permutation for $T_{pi}$. The prompt for generating $T_{pi}$ based on $T_{a}$ and its corresponding rule is:

[Instruction]
Please generate a sentence that has a similar length and meaning in the following six ways:
1. Change the word order: For example, “a red and yellow dog” can be changed to “a yellow and red dog.” In some languages, adjusting the order of words in a sentence can create a new sentence form without changing the meaning. For instance, “I like you” can be adjusted to “You are the person I like”.
2. Passive voice: For example, “a kid is flying a yellow kitec̈an be changed to “a yellow kite is being flown by a kid.”
3. Change the description: For example, “a boy is playing with a girl” can be changed by paraphrasing and altering the sentence structure to “a boy is playing. He is near a girl.”
4. Use synonyms: Replace words in the sentence with their synonyms. For example, “happy” can be replaced with “joyful”.
5. Use infinitive or gerund forms: For example, “He likes to run” can be changed to “He enjoys running”.
6. Simplify or expand: You can either simplify the sentence structure or add additional information to create a new sentence. For example, “The quick, brown fox jumps over the lazy dog” can be simplified to “The fox jumps over the dog”, or expanded to “The fox, which is quick and brown, jumps over the lazy dog”.
Now, please generate a similar sentence for input prompt given at the end. Provide one sentence for each of the six methods. If a sentence cannot be generated using a particular method, please output “None”.
Add the results as a list of JSON objects, containing 6 JSON objects. Each object should include the keys: number, modification method, and sentence.
[Prompt]
“{TEXT0}”

### C.2 Data Annotation

Table: Table 5: Error Examples of LLM-generated permutation-based sentences ($T_{a}$, $T_{pv}$, $T_{pi}$) and the criteria they violate.

Criteria for Valid Samples.
The primary challenge in annotation lies in defining the criteria for what qualifies as “valid”.
For T2I synthesis models, we define “valid” input text based on 14 specific criteria. First, we illustrate these criteria through examples of $T_{a}$ and $T_{pv}$. Second, for $T_{pi}$, we require it to apply one of the six synonymous transformations defined in the prompt for generating $T_{pi}$. From a semantic perspective, $T_{pi}$ must be strictly consistent with $T_{a}$, ensuring the consistency and accuracy of the entire dataset. We list these criteria and examples in the following.

- •
Basic
–
Complete Expression: Both sentences should be complete and free from obvious linguistic errors.
–
Clear and Concrete Objects: Both sentences must be clear and unambiguous, contextually or inherently, and specifically describe tangible objects, steering clear of abstract concepts.
–
Meaningful Sentence: Both sentences must maintain logical coherence in their respective contexts.
The reasonable definition includes real-world plausibility or scenarios typically seen as implausible in virtual or imaginative settings (like children’s literature, animations, or science fiction), such as flying pigs or dinosaurs piloting planes.
For example, “a shorter person can reach a higher shelf while a taller one cannot” is not reasonable in any world.
- •
Visualizable
–
Visually Depicted Element: Both sentences must convey visual elements, including objects, scenes, actions, and attributes, ensuring that the text prompts are visually depictable and the image content is identifiable during evaluation.
–
Static Scene or Multiple Exposure Scene:
Both sentences should be visually representable through images alone, negating the need for video, audio, or other sensory inputs like touch and smell. Temporal aspects, procedures, and comparisons in test cases must be conveyable within a single image’s scope.
–
A Moderate Level of Details: Sentences should maintain a moderate level of detail with similar scales for objects and scenes. Excessive or mismatched scales can result in sentences that are challenging to depict.
For example, comparing the quantity of books and magazines “in a library” is less suitable than “on a table”.
–
Quantifiable Comparison: Comparisons in both sentences should be quantifiable, using measures like counts, areas, or volumes.
For example, “There are more students in the classroom than words on the blackboard” are difficult to compare quantitatively.
- •
Discriminative
–
Following Permutation Rules: Generated samples $T_{pv}$ must strictly follow the designated manual template, including word swapping and moving.
–
Distinct Textual Semantics: Two sentences must have distinct textual semantics. Otherwise, the pairs are considered invalid.
–
Visually Distinguishable: Two sentences should be visually distinct, with clear differentiation regarding the visual characteristics of the objects or scenes described.
Subtle differences requiring very close observation are not considered distinct visual differences.
- •
Recognizable
–
Item-Specific Scenes: Scenes in sentences should be identifiable, maintaining key elements for recognition. Otherwise, identification may be challenging. For instance, a sentence describing a library where “bookshelves outnumber books” might be unrecognizable, as we typically expect a library to contain many books.
–
Item-Specific Characters: When a sentence depicts a character through associations with specific items, these items or behaviors should remain consistent for easy identification. If not, the character may be hard to recognize. For instance, chefs are usually associated with “chef’s attire, cooking utensils, and kitchens”.
–
Attire-Based Characters: When a sentence presents characters identifiable by their attire, such as firefighters, police officers, soldiers, doctors, and nurses, their clothing should remain consistent for clear recognition. Changes in attire could obscure their identities.
–
Action-Based Characters: When a sentence features characters defined by specific actions or interactions, such as bartenders (mixing drinks), businessmen (negotiating), journalists (interviewing), divers (deep-sea diving), their typical activities should be consistent. Altering distinctive features or placing characters in unusual scenarios may obscure their identities.

Automatic Annotation.
We employ machine-human hybrid verification to filter out invalid samples that violate any characteristic.
We use LLMs to judge whether each sample violates any of the specific criteria, labeling them “yes” or “no” and providing confidence scores.
The samples whose confidence exceeds a threshold of 0.8 are removed from the dataset.
We initially collected 48K samples, each including 3 sentences.
The automatic filtering helped eliminate over 42% of them, resulting in a final corpus of 27K samples.

Human Annotation. We use 15 annotators and 3 experienced experts to manually verify the samples.
All annotators have linguistic knowledge and are provided with detailed annotation guidelines. Each sample is independently annotated by two annotators. Then an experienced expert reviews the controversial annotations and makes the final decision. After annotation, we randomly sampled 100 samples from valid samples to assess annotation accuracy. Two experts evaluated that 99% of the samples were valid. Finally, we got 11,479 valid, non-duplicated samples.

Hard Samples Selection.
To effectively evaluate T2I models,
it is crucial to select challenging samples rather than simple ones.
Initially, we generate images using SOTA models like DALL-E3, and flagging those with alignment scores below 0.7. Then we aggregate the votes from these models to determine the most representative candidates, and select those with the highest votes for further filtering.
To ensure diversity, we categorize these samples based on permutation types, as shown in Fig. 5, with a maximum limit of 50 samples per category. Finally, 684 samples were included in our benchmark.

### C.3 Data Statistics

Category.
The samples in SemVarBench are divided into 20 categories based on their permutation types. Furthermore, these categories are further classified into three aspects based on triple types, as illustrated in Tab. 11.
These aspects are *Relation*, *Attribute Comparison* and *Attribute Value*. Specifically, *Relation* aspect includes six categories: *Action, Interaction, Absolute Location, Relative Location, Spatial-Temporal, Direction*.
*Attribute Contrast* includes four categories: *Size, Height, Weight, Vague Amount*.
*Attribute Value* includes ten categories: *Color, Counting, Texture, Material, Shape, Age, Sentiment, Temperature, Manner*, and *Appearance*.

Scale and Split.
SemVarBench comprises 11,454 valid samples of $(T_{a},T_{pv},T_{pi})$, totaling 34,362 sentences. We divide it into a training set and a test set. The training set contains 10,806 samples, while the test set consists of 648 challenging samples for effective evaluation, as shown in Tab. 6. All our evaluations are conducted on the test set.

Distribution. Since some permutations contain multiple words, they may fall into more than one category.
In the training set, 51.06% of the permutation involves only one category, 35.12% for two categories, and 7.35% for three categories and 0.5% for more than four categories.
In the test set, 82.75% of permutations involve only one category, 14.77% involve two categories, and 2.49% involve three categories. As a result, the total count of categorized samples exceeds the actual number of unique samples.

**Table 6: Statistics of SemVarBench.**
| Category | Train | Test | Total |
| --- | --- | --- | --- |
| Relation |  |  |  |
| Absolute Location | 1,716 | 50 | 1,766 |
| Relative Location | 1,111 | 50 | 1,161 |
| Action | 216 | 48 | 264 |
| Interaction | 153 | 43 | 196 |
| Direction | 342 | 33 | 375 |
| Spatio-temporal | 234 | 50 | 284 |
| Attribute Comparison |  |  |  |
| Vague amount | 1,839 | 50 | 1,889 |
| Size | 2,168 | 50 | 3,118 |
| Height | 253 | 50 | 303 |
| Weight | 5 | 5 | 10 |
| Attribute Value |  |  |  |
| Color | 4,451 | 50 | 4,501 |
| Appearance | 1,972 | 50 | 2,022 |
| Texture | 542 | 50 | 592 |
| Shape | 190 | 50 | 240 |
| Size | 516 | 50 | 566 |
| Material | 227 | 50 | 277 |
| Manner | 194 | 49 | 243 |
| Sentiment | 88 | 26 | 114 |
| Age | 22 | 11 | 33 |
| Temperature | 14 | 4 | 18 |
| Counting | 614 | 50 | 664 |
| Total | 15,518 | 819 | 14,699 |
| Total(deduplication) | 11,454 | 684 | 10,770 |

SemVarBench vs. Other benchmarks.
Compared with existing benchmarks, SemVarBench focuses on the understanding of semantic variations for text-to-image synthesis, which includes two types of permutation: permutation-variance and permutation-invariance. Other comparisons, such as source, scale, annotation and split, are illustrated in Tab. 7.

**Table 7: Comparison between SemVarBench and other T2I synthesis benchmarks.**
| Benchmark | Concentration | Data Source | #Prompts | Annotation | Split |
| --- | --- | --- | --- | --- | --- |
| DrawBench Saharia et al. () | General | Human | 200 | Human | Test |
| PartiPromps Yu et al. () | General | Human | 1600 | Human | Test |
| PaintSkills Cho et al. () | General | Template | 73.3K | – | Train/Test |
| HRS-Bench Bakr et al. () | General | Template & LLM | 45.0K | Human | Test |
| $\text{SR}_{\text{2D}}$ Gokhale et al. () | Compositional | Dataset | 25.3K | – | Test |
| ABC-6K Feng et al. () | Compositional | Dataset | 6.4K | – | Test |
| CC-500 Feng et al. () | Compositional | Template | 500 | – | Test |
| TIFA v1.0 Hu et al. () | Compositional | Dataset | 4.1K | – | Test |
| VPEval-skill Cho et al. () | Compositional | Dataset | 3.8K | – | Test |
| DSG-1K Cho et al. () | Compositional | Dataset | 1.1K | – | Test |
| T2I-CompBench Huang et al. () | Compositional | Template & LLM | 6.0K | – | Train/Test |
| Winoground Thrush et al. () | Permutation-Variance | Human | 800 | Human | Test |
| SemVarBench(ours) | Permutation-Variance<br>Permutation-Invariance | Template & LLM | 34K | LLM & Human | Train/Test |

## Appendix D Details of Experiment Setting

### D.1 T2I Synthesis Models

We generate one image using the mainstream T2I diffusion models in Fig. 1: Stable Diffusion v1.5(^3^33The model used is ruwnayml/stable-diffusion-v1-5, which is now deprecated. A mirror is available at: https://huggingface.co/stable-diffusion-v1-5/stable-diffusion-v1-5) (denoted as SD 1.5), Stable Diffusion v2.1(^4^44https://huggingface.co/stabilityai/stable-diffusion-2-1) (denoted as SD 2.1), Stable Diffusion XL v1.0(^5^55https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0;https://huggingface.co/stabilityai/stable-diffusion-xl-refiner-1.0) (denoted as SD XL 1.0), Stable Cascade(^6^66https://huggingface.co/stabilityai/stable-cascade-prior;https://huggingface.co/stabilityai/stable-cascade) (denoted as SD CA), DeepFloyd IF XL(^7^77https://huggingface.co/DeepFloyd/IF-I-XL-v1.0;https://huggingface.co/DeepFloyd/IF-II-L-v1.0;https://huggingface.co/stabilityai/stable-diffusion-x4-upscaler) (denoted as DeepFloyd), PixArt-alpha XL(^8^88https://huggingface.co/PixArt-alpha/PixArt-XL-2-1024-MS)(denoted as PixArt), Kolors, Stable Diffusion 3 [medium](^9^99https://huggingface.co/stabilityai/stable-diffusion-3-medium)(denoted as SD 3), FLUX.1 [dev](^10^1010https://huggingface.co/black-forest-labs/FLUX.1-dev) (denoted as FLUX.1), Midjourney V6(^11^1111https://www.midjourney.com/home) (denoted as MidJ V6), DALL-E 3(^12^1212https://openai.com/index/dall-e-3/), CogView3-Plus(^13^1313https://www.bigmodel.cn/dev/api/image-model/cogview) (denoted as CogV3-Plus), Ideogram 2(^14^1414https://about.ideogram.ai/2.0). The schedulers for SD 1.5 and SD 2.1 are set to DPM-Solver++, while all other settings are left as default.

### D.2 Evaluator

We use four advanced MLLMs as the evaluators to demonstrate the general applicability of our proposed evaluation metrics: Gemini 1.5 Pro, Claude 3.5 Sonnet, GPT-4o and GPT-4 Turbo. GPT-4o and GPT-4 Turbo have been shown to achieve near-human performance in evaluating alignment in T2I synthesis modelsZhang et al. (); Chen et al. ().Claude 3.5 Sonnet outperforms GPT-4o and Gemini 1.5 Pro Anthropic ().
The versions of these MLLMs used are as follows: Gemini 1.5 Pro (gemini-1.5-pro-001), Claude 3.5 Sonnet (claude-3-5-sonnet-20240620), GPT-4o (gpt-4o-2024-05-13), and GPT-4 Turbo (gpt-4-turbo-2024-04-09).
The alignment score components follow the division outlined in Zhang et al. (), with the exception of the aesthetic score component, which has been omitted. The complete prompt is as follows.

Does the generated image align with the given prompt?
[Instruction]
Carefully assess the generated image in terms of relevance to the prompt and object accuracy. Notice that the image is digitally created or artificially generated, and I hope you help feedback on the quality of a generated image rather than discussing the content of a real photograph.
Use the following criteria to guide your evaluation: with Relevance (0-50 points), Object Accuracy (0-50 points). After providing your explanation, you must rate the generated image by strictly following this format: “[[rating]]”, for example: “Relevance (0-50 points): [[35]], Object Accuracy (0-50 points): [[30]]”.
[Prompt]
{prompt}

After receiving outputs from LLMs, we utilize regular expressions to extract scores. In our experiments, the outputs from four evaluators mentioned above consistently followed the specified format as defined in the prompt. We also tested Qwen-VL-Chat, Qwen-VL-Plus, Qwen-VL-Max, and LLAVA-1.6, which exhibited poor adherence to the specified format and required a more complex extraction process. To simplify the evaluation process, we decided to adopt results exclusively from Gemini 1.5 Pro, Claude 3.5 Sonnet, GPT-4o, and GPT-4-Turbo.

### D.3 Training Setting

Training Data Selection.
The training set of SemVarBench comprises 10,806 samples. We investigate the improvement from fine-tuning the T2I model Stable Diffusion XL v1.0. We select the generated images whose alignment scores meet the requirements. These constraints are as follows.

First, the generated image should be approximately aligned with its corresponding text prompt.

$$ $\displaystyle\begin{cases}S(T_{a},I_{a})&>C_{2},\\ S(T_{pv},I_{pv})&>C_{2},\\ \end{cases}$ (12) $$

where $C_{2}$ is a threshold.

Second, the alignment scores between matched text-image pairs should be higher than those between mismatched text-image pairs.

$$ $\displaystyle\begin{cases}S(T_{a},I_{a})&>S(T_{a},I_{pv}),\\ S(T_{a},I_{a})&>S(T_{pv},I_{a}),\\ S(T_{pv},I_{pv})&>S(T_{a},I_{pv}),\\ S(T_{pv},I_{pv})&>S(T_{pv},I_{a}),\\ \end{cases}$ (13) $$

Third, the visual semantic variations observed from different text prompts should be the same when the initial image and the final image are the same.

$$ $\displaystyle S(T_{a},I_{a})-S(T_{a},I_{pv})\approx S(T_{pv},I_{pv})-S(T_{pv}, I_{a}),$ (14) $$

Similarly, the textual semantic variations observed from different images should be the same when the initial text prompt and the final text prompt are the same.

$$ $\displaystyle S(T_{a},I_{a})-S(T_{pv},I_{a})\approx S(T_{pv},I_{pv})-S(T_{a},I _{pv}),$ (15) $$

Utilizing this approximate equality relationship in Eq. 14 and Eq. 15, we constrain the alignment score using the following inequality:

$$ $\begin{aligned} \begin{cases}\left|(S(T_{a},I_{a})-S(T_{a},I_{pv}))-(S(T_{pv}, I_{pv})-S(T_{pv},I_{a}))\right|<C_{3},\\ \left|(S(T_{a},I_{a})-S(T_{pv},I_{a}))-(S(T_{pv},I_{pv})-S(T_{a},I_{pv})) \right|<C_{3},\\ \end{cases}\end{aligned}$ (16) $$

In our experiments, we utilized Stable Diffusion XL v1.0 to generate an image for each text prompt within the training set. To select the training data, we designated $C_{2}=0.8$ and $C_{3}=0.1$. Ultimately, we selected 327 samples, resulting in 981 sentences.

Supervised Fine-Tuning (SFT). Each text-image pair $(T_{i},I_{i})$ is incorporated into the training set. For every sample $(T_{a},T_{pv},T_{pi})$, which results in three text-image pairs: $(T_{a},I_{a})$, $(T_{pv},I_{pv})$ and $(T_{pi},I_{pi})$, leading to a total of 981 diverse pairs. The selected set of samples is denoted as $D_{s}$. The loss function for SFT remains unchanged Kingma et al. (); Song et al. (), which is defined as

$$ $\mathcal{L}(\theta)=\mathbb{E}_{(x,y)\in\mathcal{D}_{s}}\left[\left\|\epsilon- \epsilon_{\theta}(z_{t},t,y)\right\|_{2}^{2}\right],$ (17) $$

where $x$, $y$, $t$, $z_{t}$ are the representations of the image $I_{i}$, text prompt $T_{i}$, timestamp $t$, and the latent representation of the image at timestamp $t$, respectively.
We conducted two separate fine-tuning processes using the diffusers library(^15^1515The diffusers library support fine-tuning Unet and Unet + text encoder. We made minor modifications to support fine-tuning only the text encoder. The url of scripts provided by diffusers is: https://github.com/huggingface/diffusers/tree/main/examples/text_to_image/): only fine-tuned the LoRA model on either the UNet or on the text encoder for 5000 steps, with a training batch size of 1.
Our computational resources included a NVIDIA GeForce RTX 4090 with 25.2 GB of VRAM and a 16-core AMD EPYC 9354 processor, with 60.1 GB of system memory available. We train the LoRA model with a rank of 4 on UNet or text encoders, and the training process takes approximately 0.5 hours.

Direct Policy Optimization (DPO). In our experiments, we added text-image tuples of the form $(T_{i},I_{i},I_{j})$ to the training set, where the semantic content of $T_{i}$ does not match $T_{j}$. For each input $T_{i}$, $I_{i}$ represents the chosen image and $I_{j}$ the rejected one. For every sample $(T_{a},T_{pv},T_{pi})$, this results in four text-image tuples: $(T_{a},I_{a},I_{pv})$, $(T_{pv},I_{pv},I_{a})$, $(T_{pv},I_{pv},I_{pi})$, and $(T_{pi},I_{pi},I_{pv})$, resulting in a total of 1,308 tuples. The loss function for DPO remains unchanged Wallace et al. (), which is defined as

$$ $\begin{aligned} \mathcal{L}(\theta)=-\mathbb{E}_{(x^{w},x^{l},y)\sim\mathcal{D }_{s},z_{t}^{w}\sim q(z_{t}^{w}\mid x^{w}),z_{t}^{l}\sim q(z_{t}^{l}\mid x^{l} )}\log\sigma(\\ -\beta(\left\|\epsilon^{w}-\epsilon_{\theta}(z_{t}^{w},t,y)\right\|_{2}^{2}- \left\|\epsilon^{w}-\epsilon_{ref}(z_{t}^{w},t,y)\right\|_{2}^{2}-\\ (\left\|\epsilon^{l}-\epsilon_{\theta}(z_{t}^{l},t,y)\right\|_{2}^{2}-\left\| \epsilon^{l}-\epsilon_{ref}(z_{t}^{l},t,y)\right\|_{2}^{2}))),\end{aligned}$ (18) $$

where $x^{w}$, $x^{l}$, $y$, $t$, $z_{t}^{w}$, $z_{t}^{l}$, $\sigma$ are the representations of the chosen image $I_{i}$, the rejected image $I_{j}$, text prompt $T_{i}$, timestamp $t$, the latent representation of the chosen image at timestamp $t$, the latent representation of the rejected image at timestamp $t$ and the sigmoid function, respectively.
We executed two separate fine-tuning processes using the DiffusionDPO(^16^1616We used the code provided by the diffusers library, which supports fine-tuning of Unet. We made minor modifications to support fine-tuning only the text encoder. The url of scripts provided by diffusers is: https://github.com/huggingface/diffusers/tree/main/examples/research_projects/diffusion_dpo/): only fine-tuned the LoRA model on either the UNet or on the text encoder for 5000 steps, with a training batch size of 1.
Our computational resources included an Tesla V100-SXM2 with 32GB of VRAM and a 11-core Intel(R) Xeon(R) Platinum 8163 processor, with 88.0 GB of system memory available.
We train the LoRA model with a rank of 4 on UNet or text encoders, and the training process takes approximately 4.5 hours.

## Appendix E More Experiment Results

Figure: Figure 11: The distribution of SemVarEffect scores across various categories for the Ideogram 2 model and DALL E-3 model, as evaluated by GPT-4 Turbo. Left: Ideogram 2. Right: DALL E-3.

Effects of Semantic Variations on Different Categories. The impact of semantic variations is not uniform across different semantic classes, as shown in Fig. 7, with exact scores listed in Tabs. 8 and Tab. 9. For *Relation*, most models most models consistently show low scores, as indicated by the dark blue shading in Fig. 7. This suggests that models handle samples involving *Relations*—such as *Absolute Location*, *Relative Location*, and *Actions*—with limited accuracy. For *Attribute Value*, models such as Ideogram2 perform significantly better at capturing attributes such as *Color*, as shown by the prominent red shading in Fig. 7. These models demonstrate a clear advantage in both generating and recognizing these attributes. In contrast, models such as DALL-E 3 and CogV3-Plus display a more balanced yet average performance across most categories (shaded in light orange and light blue). For *Attribute Comparison* (e.g., *Size*, *Weight*, *Height*), most models score lower, indicating their weaker ability to handle complex attribute comparisons.

Although most T2I models struggle with capturing semantic variations in many categories, some categories, such as *Color* and *Age*, demonstrate slightly better performance, as indicated by higher median values. Fig. 11 illustrates the distribution of SemVarEffect scores across various categories for the Ideogram 2 model, while Fig. 12 shows the scores for different T2I models in the *Color* and *Direction* categories. Most categories have medians (marked by the orange line) close to zero, indicating that T2I models generally struggle to capture the semantic variations introduced by word order changes, particularly in the *Direction* category. However, some categories, such as *Weight* and *Color*, show slightly higher median values, indicating that semantic variation caused by word order changes may have a minor positive effect in these instances. Categories such as *Absolute Location* and *Counting* show greater variability in model responses, while categories such as *Sentiment* and *Texture* show more consistent effects with narrower distributions.

**Table 8: The results of SemVarEffect $\kappa$ on aspects *Relation* and *Attribute Comparison*. The evaluator is GPT-4 Turbo.**
| Models | Relation | Attribute Comparison |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Absolute<br>Location | Relative<br>Location | Action | Interaction | Direction | Spatial<br>-Temporal | Size | Weight | Vague<br>Amount | Height |  |
| Open Source Models |  |  |  |  |  |  |  |  |  |  |
| Stable Diffusion v1.5 | -0.01 | 0.00 | -0.06 | -0.11 | 0.05 | -0.01 | 0.11 | -0.01 | -0.07 | 0.01 |
| Stable Diffusion v2.1 | -0.01 | -0.06 | -0.08 | 0.02 | -0.02 | -0.00 | 0.03 | -0.10 | 0.02 | 0.06 |
| Stable Diffusion XL v1.0 | -0.02 | -0.08 | -0.01 | 0.05 | 0.05 | -0.05 | 0.07 | 0.16 | 0.03 | 0.09 |
| Stable Cascade | 0.02 | -0.03 | 0.01 | -0.03 | 0.02 | 0.02 | -0.02 | -0.09 | 0.08 | -0.01 |
| DeepFloyd IF XL | -0.01 | -0.00 | 0.01 | -0.01 | 0.04 | 0.01 | 0.03 | 0.05 | -0.04 | 0.03 |
| PixArt-alpha XL | 0.00 | -0.01 | 0.03 | 0.00 | -0.04 | -0.03 | 0.07 | 0.10 | 0.10 | 0.03 |
| Kolors | -0.03 | 0.02 | -0.07 | 0.02 | 0.03 | -0.02 | -0.06 | -0.10 | 0.07 | 0.07 |
| Stable Diffusion 3 | -0.03 | 0.01 | -0.02 | 0.05 | -0.08 | -0.04 | 0.07 | -0.02 | 0.10 | 0.04 |
| FlUX.1 | -0.03 | 0.03 | 0.03 | 0.08 | -0.04 | -0.00 | 0.09 | 0.23 | 0.05 | 0.09 |
| API-based Models |  |  |  |  |  |  |  |  |  |  |
| Midjourney V6 | 0.07 | 0.01 | 0.04 | 0.03 | 0.03 | 0.08 | 0.07 | -0.12 | 0.07 | 0.02 |
| DALL-E 3 | -0.00 | 0.12 | 0.11 | 0.08 | 0.13 | 0.11 | 0.08 | -0.00 | 0.09 | 0.15 |
| CogView3-Plus | 0.08 | 0.08 | 0.23 | 0.07 | 0.03 | -0.01 | 0.23 | -0.03 | 0.23 | 0.22 |
| Ideogram 2 | 0.01 | 0.04 | 0.13 | 0.29 | -0.02 | -0.02 | 0.12 | 0.04 | 0.17 | 0.17 |

**Table 9: The results of SemVarEffect $\kappa$ on aspects *Attribute Value*. The evaluator is GPT-4 Turbo. AVG represents the average effect score of all samples on aspect *Relation*, *Attribute Comparison* and *Attribute Value*. The evaluator is GPT-4 Turbo.**
| Models | Attribute Value | AVG |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Color | Material | Appearance | Age | Shape | Temperature | Texture | Sentiment | Manner | Counting |  |  |
| Open Source Models |  |  |  |  |  |  |  |  |  |  |  |
| Stable Diffusion v1.5 | 0.09 | 0.02 | 0.04 | -0.20 | 0.01 | 0.06 | 0.02 | -0.06 | -0.05 | -0.07 | -0.01 |
| Stable Diffusion v2.1 | 0.12 | 0.10 | -0.03 | -0.09 | -0.00 | -0.06 | -0.03 | -0.10 | -0.01 | 0.02 | 0.00 |
| Stable Diffusion XL v1.0 | 0.13 | 0.09 | -0.01 | 0.01 | -0.01 | 0.05 | -0.00 | 0.03 | -0.00 | 0.00 | 0.02 |
| Stable Cascade | 0.14 | 0.05 | 0.10 | -0.03 | -0.02 | -0.15 | -0.03 | -0.05 | 0.06 | 0.01 | 0.04 |
| DeepFloyd IF XL | 0.19 | 0.14 | 0.06 | -0.19 | 0.04 | -0.02 | 0.05 | -0.05 | 0.06 | 0.01 | 0.04 |
| PixArt-alpha XL | 0.11 | 0.09 | 0.00 | 0.15 | 0.01 | 0.13 | -0.02 | 0.02 | 0.03 | -0.00 | 0.02 |
| Kolors | 0.21 | 0.07 | -0.01 | 0.01 | -0.09 | 0.01 | 0.10 | -0.01 | -0.04 | 0.00 | 0.01 |
| Stable Diffusion 3 | 0.33 | 0.10 | 0.11 | 0.04 | 0.08 | 0.11 | 0.08 | 0.01 | 0.03 | 0.06 | 0.06 |
| FlUX.1 | 0.35 | 0.21 | 0.21 | 0.08 | 0.09 | -0.13 | 0.04 | -0.06 | 0.07 | 0.10 | 0.09 |
| API-based Models |  |  |  |  |  |  |  |  |  |  |  |
| Midjourney V6 | 0.20 | 0.12 | 0.13 | 0.10 | -0.03 | -0.21 | 0.11 | -0.05 | 0.09 | -0.02 | 0.05 |
| DALL-E 3 | 0.22 | 0.17 | 0.17 | 0.23 | 0.14 | 0.22 | 0.22 | 0.19 | 0.11 | 0.06 | 0.12 |
| CogView3-Plus | 0.35 | 0.15 | 0.17 | 0.31 | 0.16 | 0.30 | 0.17 | 0.21 | 0.27 | 0.07 | 0.15 |
| Ideogram 2 | 0.37 | 0.15 | 0.24 | 0.42 | 0.20 | 0.08 | 0.12 | 0.16 | 0.13 | 0.07 | 0.13 |

Figure: Figure 12: The distribution of SemVarEffect scores across various T2I models within the Color and Direction categories, as evaluated by GPT-4 Turbo. Top: Color. Bottom: Direction.

## Appendix F More Analysis

Is there a significant difference among various text encoders in discerning the semantic nuances caused by linguistic permutations?
We explore the efficacy of diverse text encoders in discerning such nuances.
Fig. 13 compares the text similarity between $T_{a}$ and $T_{pv}$ across models utilizing different text encoder models.
SD 1.5, SD 2.1, SD XL v1.0, and SC utilize CLIP series models as text encoders, while DeepFloyd, PixArt, and DALL-E 3 utilize T5 series models.
The similarity metric is depicted as $1-\text{cosine}(T_{a},T_{pv})$, with higher values indicating a stronger ability of the text encoder to differentiate between the semantics of two sentences.
This indicates that the choice of text encoder significantly influences the model’s semantic discrimination capabilities.

Figure: Figure 13: The semantic discrimination capabilities of different text encoders measured by $1-\text{cosine}(T_{a},T_{pv})$.

Detailed Analysis on Alignment Scores vs. SemVarEffect Score
Fig. 14 illustrates that although the distributions of the SemVarEffect score and the alignment score are similar, the SemVarEffect score demonstrates a higher degree of differentiation, especially when it comes to distinguishing between FLUX.1 and SD 3. Based on the alignment score, it could be concluded that FLUX.1, SD 3, and SD XL 1.0 have comparable performance levels and they may be grouped into the same cluster. However, based on the SemVarEffect score, it becomes evident that FLUX.1 and SD 3 differ distinctly from SD XL 1.0. SD XL 1.0 responds more similarly to semantic variations caused by word order changes in a manner similar to SD 1.5, SD 2.1, and SD CA. Correspondingly, we observe that when using the T5-XXL series model as the text encoder, the difference between DALL-E 3 and other models, such as PixArt and DeepFloyd, becomes more pronounced when assessed by the SemVarEffect score.

Figure: Figure 14: A comparison of alignment scores and the SemVarEffect score under the same conditions of text similarity. The squares are results of permutations of permutation-variance. The evaluator is GPT-4 Turbo.

Why do permutations without semantic changes exhibit higher text similarity scores compared to those with semantic changes?
This phenomenon is closely related to our dataset’s construction methodology, where $T_{pi}$ is generated by swapping two long phrases located on either side of a coordinating conjunction or a predicate, such as the *and* in Fig. 4.
We observed that permutations with semantic changes in our benchmark have significantly smaller edit distances from the anchor sentence compared to synonymous sentences, as shown in Tab. 10. The average edit distances between $(T_{a},T_{pv})$, $(T_{a},T_{pi})$ and $(T_{a},T_{random})$ are 13, 32 and 53.
As our analysis does not rely on the similarity scores of synonymous sentences, this does not affect our previous findings.

**Table 10: The average edit distance between sentences in different categories.**
| Category | $(T_{a},T_{pv})$ | $(T_{a},T_{pi})$ | $(T_{a},T_{random})$ |
| --- | --- | --- | --- |
| Relation |  |  |  |
| Absolute Location | 11.94 | 27.24 | 52.40 |
| Relative Location | 12.26 | 28.62 | 46.98 |
| Action | 13.94 | 31.85 | 48.35 |
| Interaction | 13.56 | 29.58 | 44.26 |
| Direction | 12.03 | 27.18 | 50.21 |
| Spatio-temporal | 17.40 | 42.74 | 59.94 |
| Attribute Comparison |  |  |  |
| Vague amount | 19.38 | 36.56 | 50.38 |
| Size | 11.38 | 26.00 | 46.82 |
| Height | 13.04 | 23.00 | 31.22 |
| Weight | 10.00 | 26.20 | 22.20 |
| Attribute Value |  |  |  |
| Color | 11.80 | 30.86 | 46.90 |
| Material | 12.40 | 28.42 | 41.92 |
| Appearance | 13.86 | 44.14 | 59.14 |
| Age | 14.73 | 34.73 | 46.64 |
| Shape | 13.34 | 33.48 | 40.98 |
| Temperature | 11.00 | 27.50 | 38.50 |
| Texture | 11.74 | 31.20 | 54.22 |
| Sentiment | 11.96 | 33.15 | 48.65 |
| Manner | 13.37 | 33.71 | 52.90 |
| Counting | 8.44 | 29.06 | 44.18 |
| Average | 13.12 | 31.65 | 54.30 |

## Appendix G More Case Studies

In this section, we present examples that demonstrate an understanding of semantic variations and examples that do not. Examples that grasp semantic variations typically have high alignment scores ($\bar{S_{ii}}$) and high effect scores ($\kappa$), as illustrated in Fig. 15. Conversely, examples that lack this understanding often have high alignment scores ($\bar{S_{ii}}$) but low effect scores ($\kappa$), as depicted in Figures 16 and 17. The SemVarEffect scores allow us to distinguish models’ abilities to accurately interpret and visually represent semantic variations. However, in practice, evaluation accuracy can be significantly affected by errors in generated images or biases in evaluators’ ratings. Severe errors can particularly distort the evaluation’s accuracy, as evidenced in Figures 21 and 23. To enhance the accuracy of our evaluations, we will utilize more precise evaluators in future work.

## Appendix H Limitation

We would like to highlight that the size of SemVarBench is constrained by the necessity for manual verification due to the unsatisfactory accuracy of LLM’s validation, which incurs high costs. Furthermore, the scale of evaluation is limited by the high costs of image generation and LLM-based evaluation, both in terms of time and money, thus restricting the extent of such evaluations.

Table: Table 11: Permutation-based valid sentences $(T_{a},T_{pv},T_{pi})$ in diverse categories.

Figure: Figure 15: The cases which understand semantic variations.
Refer to caption: x24.png

Figure: Figure 16: The cases which don’t understand semantic variations.
Refer to caption: x27.png

Figure: Figure 17: More cases which don’t understand semantic variations.
Refer to caption: x31.png

Figure: Figure 18: Cases with minor errors which understand semantic variations.
Refer to caption: x35.png

Figure: Figure 19: Cases with minor errors which don’t understand semantic variations. Several alignment scores, which are incorrect according to GPT-4V, are labeled in red.
Refer to caption: x38.png

Figure: Figure 20: Examples of acceptable outliers include negative SemVarEffect ($\kappa$) values that are close to zero. Outliers with a SemVarEffect score ($\kappa$) slightly below 0 are acceptable.
Refer to caption: x41.png

Figure: Figure 21: Examples of acceptable outliers include negative $\kappa$ values that are with a SemVarEffect score outside the range [0,1], being considered unacceptable. This discrepancy may be due to incorrect text-image alignment scores provided by evaluators or low quality images.
Refer to caption: x45.png

Figure: Figure 22: Errors only due to incorrect scoring by GPT-4V, where images are essentially correct.
Refer to caption: x52.png

Figure: Figure 23: Errors only due to incorrect scoring by GPT-4V, where images are essentially correct. The errors heavily influence the SemVarEffect scores.
Refer to caption: x57.png

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 3
presentation: 1
questions: See weakness part.
soundness: 2
strengths: - The paper directly addresses a critical issue in the evaluation of T2I models, which is the accurate interpretation of human instructions and evaluate the finegrained instruction following ability of the models.
summary: The paper presents a novel metric, SemVarEffect, and a corresponding benchmark, SemVarBench, designed to evaluate the causality between semantic variations in input text and output images for text-to-image (T2I) synthesis models. The authors identify a gap in current evaluation methods, which often rely on indirect metrics like text-image similarity, failing to adequately assess how models handle semantic changes due to word order. SemVarEffect measures the average causal effect of input semantic variations on output images, while SemVarBench provides a dataset with controlled linguistic permutations to test model performance. The experiments reveal that even state-of-the-art models struggle with semantic variations, indicating room for improvement in understanding and generating accurate visual representations from textual instructions.
weaknesses: 1. The paper suffers from significant presentation issues, making the proposed method difficult to comprehend. The overall structure of the methodology section requires refinement:
   - Figure 2, which is crucial to understanding the proposed method, is challenging to interpret.
   - In Section 2.2, the term "localized change" needs clarification.
   - The definition of "integrated visual semantic variations" is also unclear.
   - Line 278 uses the phrase "evoke vivid mental images," but its meaning is ambiguous in the given context.
   - Table 2 includes numerous annotations, but offers no explanation of what the different scores represent.

2. In my opinion, the method appears overly complex. The core idea is straightforward: we anticipate that semantic alignment of generated images should be strong if the inputs are semantically identical, and less aligned if the inputs differ semantically. Thus, a simple metric like S(I_a, I_pu) - S(I_a, I_pi) could be employed. What advantages does the proposed causal relationship provide over this simple baseline? I would appreciate deeper insight and explanations on this matter.

3. Regarding model-based evaluation, there is no discussion on whether the MLLM used to compute the alignment score is effective for this task, nor is there any human evaluation. This omission raises concerns about the reliability and trustworthiness of the scores presented.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: How does the choice of filtering training data using MMLM potentially affect the evaluation results, particularly for categories with limited samples?

With the 4 metics you had mentioned, which one is the main metric that would make sense quickly for readers? It is a little hard to follow with different metrics presented at different parts, which I can imagine is a little confusing especially for readers who are looking for a single metric for benchmarking.
soundness: 3
strengths: Strengths:
Novel and important problem formulation: The paper addresses a critical gap in evaluating T2I models' language understanding capabilities.

The causal inference based framework provides a principled way to isolate the effect of semantic variations.

SemVarBench is carefully curated to avoid easily predictable variations and focus on challenging semantic permutations, with comprehensive coverage of different type of variations. LLM generated results were checked.
summary: This paper proposes a novel metric called SemVarEffect and a benchmark called SemVarBench to evaluate text-to-image (T2I) models' ability to capture semantic variations caused by word order changes in input prompts. The key contributions are:
- A causal inference framework to measure how semantic variations in text inputs affect visual semantic variations in generated images.
- A carefully curated dataset of text triplets (anchor, permutation with changed meaning, permutation with unchanged meaning) to test T2I models.
- Comprehensive evaluation of  state-of-the-art T2I models. This also includes fine-grained Analysis of the role of text encoders and cross-modal alignment in handling semantic variations.
weaknesses: The training dataset is generated using the same base SDXL model and filtered by MMLM, which could introduce self-distillation artifacts and bias, particularly affecting performance on categories with less data. 

The paper suggests that fine-tuning improves alignment at the word level rather than semantic variants, but doesn't provide sufficient discussion or analysis to support this claim

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: - In section 4.1, If text-image similarity is obtained using MLLMs, could biases inherent in the MLLM itself (insensitivity to changes in word order) affect the evaluation results?
- Text-to-image synthesis models have difficulty accurately reflecting semantic changes caused by variations in word order, possibly due to certain commonsense biases inherent in the training data. This could be seen as a form of “built-in bias.” For example, as illustrated in Figure 1 of the paper, “A cat chasing a mouse” is a reasonable depiction, whereas the reordered phrase “A mouse chasing a cat” is less plausible, leading to poorer generation results. It might be helpful to add related experiments to validate this hypothesis.
soundness: 2
strengths: - The authors find that existing text-to-image synthesis models struggle to capture semantic shifts caused by word order changes and proposed an evaluation metric and benchmark. 
- The SemVarEffect metric is based on the difference between semantic-variant and semantic-invariant text rearrangements, similar to the differential denoising principle in circuits, which I find quite novel. 
- The paper is well-organized and easy to understand.
summary: The paper primarily focuses on the impact of word order variations in text prompt on text-to-image synthesis. It proposes a metric to evaluate a model's semantic sensitivity to changes in word order. The main idea is to calculate the difference in visual semantics of generated images between semantic-variant and semantic-invariant text rearrangements. Additionally, a dataset of approximately 11k samples has been collected as an evaluation benchmark, and popular models have been assessed and analyzed based on this benchmark.
weaknesses: - The definition of visual semantics: The primary contribution of this work is the metric to measure the impact of word order in text prompts on text-to-image synthesis. The basic principle is to alter the text and observe the resulting visual semantic changes in the generated image. However, within this metric, the “visual semantics” of the generated image is defined through text-image similarity, which may not be entirely accurate. Visual semantics in an image typically refers to the content within the image itself, such as object categories, attributes, or scene information. In contrast, text-image similarity only measures the alignment of high-level features between the two modalities; it does not represent the actual visual semantics. Ideally, visual semantics should be defined independently of text. 
- The mechanism for ensuring that the generated data is semantic-variant or semantic-invariant during benchmark creation seems to be insufficiently specified, and why are some data extraction templates and rules created using LLMs while others are crafted manually? What challenges are involved in this?

---

confidence: 3
contribution: 4
initial_rating: 8
presentation: 3
questions: See Weakness
soundness: 4
strengths: - This paper offers an interesting and novel perspective on the current issues facing T2I models, and provides an effective metric and dataset to measure this issue.
- The paper offers detailed and fair experiments to compare various T2I models.
- The analysis section is thorough and carefully examines the causes of the semantic understanding issues in T2I models, providing a reliable path for future research.
summary: This paper focuses on the performance of Text to Image (T2I) models in handling semantic variations. It identifies that existing T2I models tend to overlook the relationships between words in the input text when faced with similar semantics or semantic substitutions, making it challenging to generate accurate images. At the same time, the paper introduces the SemVarEffect metric and SemVarBench to evaluate this issue.
weaknesses: - I noticed that some of the samples provided in the paper, such as "A mouse chasing a cat," are somewhat unrealistic captions. However, this category is not emphasized in SemVarBench, as seen with sentences like "A cat chasing a dog" and "A dog chasing a cat." These sentences also exhibit semantic variations, but I believe that current T2I models can handle these changes. Therefore, I think further discussion on this issue is needed.
- I believe it is necessary to further discuss the relationship between this phenomenon and the training data in the discussion. In the problematic captions where semantic variations occur, a significant portion of the original captions may be included in the pre-training data. Therefore, similar to the discussion in paper [1], I suggest designing experiments to investigate whether T2I models can perform reverse inference of semantic variations.

[1] Berglund, L., Tong, M., Kaufmann, M., Balesni, M., Stickland, A. C., Korbak, T., \& Evans, O. (2024). *The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A"*. arXiv:2309.12288.

Meta-review: Summary: This paper focused on evaluating the causality between textual semantic variations and visual semantic variations in text-to-image synthesis task. This paper pointed out that existing T2I models often struggle to capture semantic variations caused by different word orders, and further proposed a novel metric called PermuteEffect and a benchmark named PermuteBench for better T2I evaluations. The key findings are that cross-modal alignment in UNet or Transformers is important in semantic variations.

Strengths: (1) The idea of interpreting and evaluating Text-to-Image qualities via causal perspective is interesting and novel. (2) An metric and evaluation benchmark with 11.4k examples (10.8k training data and 0.6k test data). (3) A first systematic study of semantic variations in T2I synthesis is conducted.

Remaining concerns after discussion are (1) the lack of comparison with different benchmarks and metrics, (2) rationale behind the metric definition, and (3) the balance between the method’s complexity and its practical benefits.

This paper received diverse ratings 8, 6, 5, 5 as final ratings. The AC checked reviewers' comments, authors' responses, and the revised paper. For the remaining concerns (1), the text-image alignment score metric is compared; for concern (2), Sec. 2.3 has explained the rationale behind the metric definition via causal perspective; for (3), the authors replied that "this complexity stems from the fundamental nature of the problem rather than arbitrary design choices", which AC agreed as the method come from the causality nature. Therefore, the AC recommend the paper as accept and encourages the authors to further revise the paper according to reviewers' comments.

Decision: Accept (Poster)