Uncovering Limitations in Text-to-Image Generation: A Contrastive Approach with Structured Semantic Alignment
Authors: Qianyu Feng
Yulei Sui
Hongyu Zhang
Venue: EMNLP 2023 Findings (2023)
Keywords: Multi-modal generation
Semantic consistency
Structure information learning
TL;DR: This work presents a contrastive approach that explores the limitations of text-to-image models in generating fine-grained semantics, ultimately revealing the challenges in real-world applications.

Abstract: Despite significant advancements in text-to-image generation models, they still face challenges when it comes to producing highly detailed or complex images based on textual descriptions.  In order to explore these limitations, we propose a Structured Semantic Alignment (SSA) method for evaluating text-to-image generation models. SSA focuses on learning structured semantic embeddings across different modalities and aligning them in a joint space. The method employs the following steps to achieve its objective: (i) Generating mutated prompts by substituting words with semantically equivalent or nonequivalent alternatives while preserving the original syntax; (ii) Representing the sentence structure through parsing trees obtained via syntax parsing; (iii) Learning fine-grained structured embeddings that project semantic features from different modalities into a shared embedding space; (iv) Evaluating the semantic consistency between the structured text embeddings and the corresponding visual embeddings. Through experiments conducted on various benchmarks, we have demonstrated that SSA offers improved measurement of semantic consistency of text-to-image generation models. Additionally, it unveils a wide range of generation errors including under-generation, incorrect constituency, incorrect dependency, and semantic confusion. By uncovering these biases and limitations embedded within the models, our proposed method provides valuable insights into their shortcomings when applied to real-world scenarios.

Uncovering Limitations in Text-to-lmage Generation: A Contrastive Approach with Structured Semantic Alignment
Qianyu Feng
Yulei Sui
Hongyu Zhang" Chongqing University hyzhang@cqu. edu. cn
Matrix Verse
University of New South Wales y. suiqunsw. edu. au
University of Newcastle
qianyufeng71 8@gma com
Abstract
to the visual clues (segmentation maps, regional edges) adopted for single-modal image synthesis, language provides an alternative but more intu- itive description. The progress in text-to-image generation models has opened possibilities in various domains, including content creation, vir- tual worlds, and entertainment. The continued advancements in deep learning techniques, com- bined with larger and more diverse datasets, hold promise for even more realistic and context-aware image synthesis from text the future. The jour- ney began with early attempts using convolutional neural networks (CNNs) and recurrent neural net- works (RNNs). However, these models struggled to capture the complex relationship between text and images. With the introduction of generative adversarial networks (GANs), the field witnessed major breakthrough. Progress continued with models such as DALL-E (Ramesh et al., 2021) and CLIP (Radford et al., 2021) leveraged rein- forcement learning to generate images from textual prompts. DALL-E demonstrated the ability to gen- crate highly creative and unique images based on textual input, while CLIP enabled image generation conditioned text descriptions. Diffusion models have emerged as powerful approach for image generation, capable of generating high-quality and diverse images. These models leverage the princi- ples of diffusion processes to iteratively transform a noise vector into a realistic image. Diffusion models can generate diverse and high-resolution images with rich details. They also provide fine- grained level of control over the generated images by conditioning on specific inputs.
Despite significant advancement, cext-to-mmage generation models still face challenges when producing highly detailed complex images based on textual descriptions. In this work, we propose Structured Semantic Alignment (SSA) method for evaluating text-to-image gen- eration models. SSA focuses on learning struc- tured semantic embeddings across different modalities and aligning them in a joint space. The method employs the following steps to achieve its objective: (i) Generating mutated prompts by substituting words with semanti- cally equivalent or nonequivalent alternatives while preserving the original syntax; (ii) Repre- senting the sentence structure through parsing trees obtained via syntax parsing; (iii) Learning fine-grained structured embeddings that project semantic features from different modalities into a shared embedding space: (iv) Evaluating the semantic consistency between the structured text embeddings and the corresponding visual embeddings. Through experiments conducted on various benchmarks, we have demonstrated that SSA offers improved measurement of se- mantic consistency of text-to-image generation models. Additionally, unveils a wide range of generation errors including generatennr,,,nn incorrect constituency, incorrect dependency, and semantic confusion. By uncovering these biases and limitations embedded within the models, our proposed method provides valu- able insights into their shortcomings when ap- plied to real-world scenarios.
1 Introduction
Text-to-image generation models have made sig- nificant progress over the past few years, lever- aging advancements in deep learning and natural language processing techniques. These models aim to generate visually coherent and semantically rele- vant images from textual descriptions, pushing the boundaries of generative techniques. Compared
However, the effective transfer and fusion of heterogeneous information from different modali- ties in text-to-image generation tasks remain a sig- nificant challenge due substantial domain gaps. Moreover, there are often many-to-man mapping relationships, meaning that one image may corre- spond to multiple textual descriptions, and vice
"Conresponding author.

---

versa. As result, the generated images may not always align with the text prompts provided by users, requiring manual validation and selection of satisfactory results. Various aspects of the gener- ated images, such as visual quality, object accuracy, attributes, and contextual information, need to be assessed. Therefore, an automatic evaluation pro- cess and comprehensive and objective evaluation metrics are erucial for evaluating the effectiveness of text-to-image generation models. However, eval- uating these models poses challenges as the gen- eration process may not always be stable, and the perception of image quality is often subjective. Cur- rently, the primary factors considered in the eval- uation process are image quality and text-image similarity. Popular metrics include Inception Score (IS) (Salimans al., 2016) and Frechet Inception Distance (FID) (Heusel al., 2017) for assessing the image fidelity, and R-precision (Xu al., 2018) and CLIP score (Radford et al., 2021) for mea- suring the cross-modal alignment. However, the existing metrics lack insights into the assessment of fine-grasd greegra semantic concepts, such as object attributes, context details, and semantic relation- ships. These factors are critical in evaluating the performance of text-to-image generation models, especially with complicated input prompts.
the original sentence with a semantically equivalent or nonequivalent word with similar syntax. This variation enables the exploration of different inputs. (2) Structured embedding learning: The sentence is represented using syntax parse trees with con- stituency or dependency parsing. This approach breaks down the input sentence into constituent parts, facilitating the understanding of its structure and meaning.
(3) Cross-modal feature fusion: creates a scene graph by parsing the image. The scene graph is a hierarchical representation of ob- jects and their attributes, well as their spatial relationships. Nodes represent objects, while edges represent relationships between them.
SSA automatically
(4) Contrastive semantic alignment: Lastly. SSA integrates multi-modal contrastive learning into the feature processing fusion module, which encodes similar representations for positive pairs and differ- ent representations for negative pairs.
The implementation of the SSA methodology allows for a comprehensive evaluation of text-to- image generation models, specifically their ability to accurately transfer semantics from text prompts to the generated images. This approach serves as a valuable tool for diagnosing and enhancing the per- formance of such models, ensuring their reliability across diverse semantic contexts. Moreover, SSA enables the identification of cases where the gen- erated images significantly deviate from the given prompt, enabling the assessment of accuracy and robustness in text-to-image generation models. Ad- ditionally, we conduct a thorough evaluation of the current state-of-the-ar methods for text-to-image models, utilizing SSA to identify potential weak- nesses in these models.
In this context, we introduce a novel approach called Structured Semantic Alignment (SSA) to as- sess the accuracy and robustness of text-to-image generation models. SSA aims to evaluate the successful transfer of joint semantics from text prompts to the generated images. The method in- volves generating similar prompts by replacing one word in given sentence with a semantically simi- lar and syntactically equivalent word. Images then generated using these prompt pairs. The under- lying concept of SSA is that reliable ext-tto-image generation model should consistently produce co- herent and aligned results when given prompts with similar meanings. Conversely, the model should generate distinct outputs when presented with prompts containing different semantic con- cepts. Thus, when there is a mutation in the prompt, the generated image should still align with the in- put, indicating the model's robustness. This allows for the evaluation and diagnosis of the model's per- formance under varying conditions. SSA consists the following key procedures:
Challenges
Text-to-image generation models have made signif- icant progress in recent years, but they still have some limitations and under-performance. The gen- eration is heavily dependent on the quality and quantity of training data. If the training data is bi- ased or limited, the generated images will show the contents dis-aligned with text. This problem can also be attributed to the complexity of image gener- ation models, which rely on deep neural networks and complex optimization algorithms to generate images. These models often have a limited un- derstanding of the semantics the given context and may struggle to maintain semantic consistency
(I) Mutated prompt generation: We first generate source prompt mutations by replacing a word in

---

while generating images.
prompts. Text-to-image models usually fail generate images of complex scenes with multiple objects or tokens with multiple semantic meanings. This can lead to unrealistic or inaccurate results. Complex scenes may include natural scenes, ur- ban environments, or indoor scenes, among oth- ers. These scenes require the model to understand the spatial relationships and interactions between the objects and the environment, well other attributes such as textures, lighting, and colors. Largely relying on the deep learning of different modalitics, models can map the visual content with the wrong text embeddings.
Fine-grained semantic consistency. Condi- tional image gencration models are designed to generate images based on some given context or condition, such as a text description or style im- age. However, these models often face the problem of limited ability in semantic consistency. The models can struggle present accurate content aligned with the text description, especially when dealing with rare concepts or complex scenarios. Ideally, if a model is given a text description and tasked with generating an image based on this de- scription, it should generate images consistent with the given context. However, due to the model's s limited ability to maintain semantic consistency, it may generate image that deviates from the given context. Examples in Fig. show some failures in the generation of details. In the right two images, models generate "an orange car" given "an orange near a car".
3 Method
3.1 Overview
In this study, the proposed Structured Semantic Alignment (SSA) model consists of three essen- tial modules as illustrated Fig. 2. During the generation process, it is observed that the gener- ated images may contain certain details that are not explicitly described in the input prompt. To ensure consistency and evaluate the alignment of corresponding semanties across modalities, we first extract the semantic components from the input prompt using syntax parsing, and from the gener- ated image using a scene graph parser (Tang et al., 2020). This approach enables us to identify and match relevant semantic elements between the text and image representations. Following that, we em- ploy graph attention layers to integrate node em- bedding from its neighbors.
Figure 1: Images generated by Stable Diffusion given "an elephant in the air", "a wooden horse stands by river", and "an orange near car".
In the second module, referred modal Feature Fusion, we present how to integrate multi-modal semantics into a joint space. Further- more, attention layers dynamically assign weights to each modality, enabling the formation of a com- prehensive embedding representation. This mod- ule enhances the precision and efficiency of se- mantic alignment, leading more effective fu- sion of multi-modal knowledge. Finally, through contrastive learning, we continuously compare negative and positive sample sets to obtain high- confidence modality embeddings. The mutated prompts are generated by substituting words with semantically equivalent or nonequivalent alterna- tives while preserving the original syntax.
Cross-
Data biases and ambiguity. Generating high- quality images with fine- mee-grained details can be chal- lenging for text-to-image models. This can lead to generating images that unnatural or lack im- portant details. Fine-grained details can include subtle textures, patterns, colors, and shapes that are unique to the objects or scenes described in the text. Another challenge is balancing the generation of fine-grained details with overall image quality. Generating high-quality images with fine-grained details requires the model to accurately represent the entities and their context from the input de- scription, which can be difficult task, especially when dealing with abstract concepts or complex descriptions.
3.2 Structured Embeddings
Given the similarity in structures between aligned semantics in different modalities, the graph struc-
Difficulty in handling long and complex

---

Cross-modil Fine-grained
Contrastive Semantic Alignment
Fonj
Eyet
oronge
Graph
Mext
Simp
Text-to- Imnge Generation
orange cat
on
sofa
Arter
Graph
Myem
Stm
Hatt
Syntax Parsing
orange cat
Graph
(Foay Brot)
Arther
Share
Mobj
cot
Graph
Artivct
Gy
Score Graph
Craph
Artet
Pima
Pooling
Global Visual Embedding
Figure 2: The framework of Structured Semantic Alignment (SSA) consists of three modules: (1) Structured embeddings learning by first parsing the text syntax and the generated image into scene graph and applying graph attention mechanism; (2) Cross- modal Feature Fusion with cross-attention applied on the node embeddings; (3) Contrastive Semantic Alignment encodes similar representations for positive pairs and different representations for negative pairs.
ture information becomes valuable for multi-modal tasks. this paper, graph attention networks are employed to effectively model the structural infor- mation of both Gy derived from the syntax tree of input text and Gy from the scene graph of gen- erated images. This allows for the direct integra- tion of graph-based representations enhance the alignment process. Each node aggregates the hid- den states of its neighbors N, through self-loops, represented as:
intra-modality dynamics, which establish discrimi- native boundaries for each modality in the embed- ding space.
To facilitate this, a multi-modal contrastive learn- ing module is integrated into the feature process- ing fusion module, incorporating contrastive loss. Positive and negative samples are created for each modality, and contrastive learning performed minimize the loss. The paper employs loss for- mula that encodes similar representations for posi- tive pairs and different representations for negative pairs, as shown below:
of ol
Σ JEN,
(1)
L LE,E") 1 EYd
Yd (ei,e)) +
where hy is the hidden state of node j. denotes the ReLU non-linear operation, and where at is the normalized attention coefficient obtained from the ken attention calculation with splicing opera- tion ||. is the number of attention heads. aij also represents the importance node to node i, calculated through self-attention:
2N
(3)
(1- Y) max (ō (ei, 0)²
where represents label of pairs, denotes the cosine similarity, indicates the number of batch samples, denotes the margin hyper-parameter. e; E and e E E' represent the corresponding semantic in two graphs Gy and Gy. The final score of SSA is the sum of the global similarity Simg d(uuzt, Vimg) and the fine-grained seman- tic similarity Simf between k pairs of text embed- dings and image embeddings:
exp (a (a Wh + Whyl))
aij E exp (a (A [W @ Wh,)))
(2)
3.3 Fine-grained Semantic Alignment
k
The feature processing and fusion module encom- passes aspects of work. Firstly, contrastive learning employed enhance the representation of intra-modality in the feature processing fusion module. This process strengthens the feature rep- resentation within each modality and captures the
SSA 1 = d(Utatt Ving)
da (4)
T:
3.4 Contrastive Prompt Transformation
The underlying principle of the proposed SSA is that an ideal text-to-image generation model should

---

produce similar outputs when provided with simi- lar prompts, while generating distinct results when the semantics of the input prompt change. To ac- complish this, the procedure of prompt mutation generation as follows: (1) Given an input prompt, the sentence is parsed into a syntax tree. (2) A se- mantic token (such as noun, adjective, preposi- tion) is selected from the text to be replaced. (3) A random selection is made for semantic equivalent inquuivalent word, which modifies the prompt and creates mutated version. Generated images
similar word or phrase. Therefore, change in the is a reasonable indication that structural invari- ance has been violated and presumably there is a generation error.
Experiment
4.1 Experimental Settings
Models In this section, we conduct experiments to evaluate evaluation method on the popular and open-sourced text-to-image generation mod- els including: (1) GAN-based models pre-trained on MS COCO: StackGAN (Zhang et al., 2017) generates high-resolution images by stacking two GANs, and DF-GAN (Tao et al., 2020), which en- hances the text-image semantic consistency with a target-aware discriminator; (2) DALL-Es: We ex- periment with two open-sourced implementations of DALL-E by (Dayma et al., 2021), i..., Dall-E mini and Dall-E mega 2. (3) Diffusion-based mod- els: Composable Diffusion (Liu et al., 2022) which introduces compositional operators with multiple diffusion models and Stable Diffusion (Rombach et al., 2022) which is latent text-to-image dif- fusion model using frozen CLIP ViT-L/14 text encoder to condition the model on text prompts and trained subset of LAION-5B (Schuhmann et al.. 2022).
This is lovely dog.
speefred
with birthday
Input text
playing
leeing on big sofa
Undescribed contents
Matched attributes
child
Contrastive pairs
Figure 3: Contrastive learning with positive and neg- ative pairs aligns the corresponding semantic embed- dings.
To achieve this mutation, we alter a single token in the input sentence, resulting in an alternative sentence that maintains the same syntactic struc- ture. This approach generates a set of sentences that possess identical structures while being se- mantically similar. Subsequently, new images generated using the mutated prompts, and the cross- modal consistency between the synthesized images and the mutated prompts examined. An accu- rate multi-modal synthesis model should generate images with comparable semantic content when provided with similar prompts, while producing different results when given different prompts.
Evaluation Metrics We evaluate our method using the widely-used metrics in the experiments. IS (Sal- imans et al., 2016) is metric that measures the quality and diversity of generated images. It is based pre-trained inception model that is used classify images. Higher IS scores indicate that the generated images are both high quality and di- verse. FID (Heusel al., 2017) is metric that measures the distance between the distributions the generated images and real-world images. It is based on a pre-trained inception model that is used to extract feature vectors from images. Lower FID scores indicate that the generated images are closer real-world images in terms of their visual content and quality.
The motivation behind this heuristic is that the constituents of a sentence should stay the same be- tween two sentences where only a single token of the same part of speech differs. In a robust con- ditional image generation system, this should be reflected in the target images as well. Hence, we calculate the similarity between the syntax tree GT of the input sentence and the scene graph Gy of the generated image by accumulating the distances of nodes and edges. Again, the motivation that the structured semantic embedding will ideally re- main unchanged when the token is replaced with a
CLIP (Hessel et al., 2021) is a metric that mea- sures the similarity between generated images and real-war world images. It is based on a pre-trained
DaallE E mini: https://huggingffceccoo/ http:///n......ccoaaleeeiiiiiiae dalle-mini D mega: https: httpps///tugiiiaaee...llla-ee-ee dalle-mega
hhttps://colabb httc///la...eeerrhhhheaeeeeeeeaa www.goo.oo..ooooooo www.gittttiiie thub// energy-based-mode

---

Table I: Quality metrics evaluating text-to-image generation models on mutated prompts from MS COCO.
Model
ISt
FID4
Rt
CLIP
SSAT
StackGAN
10.1
27.2
46.3
18.7
42.7
DF-GAN
1 11.3
22.4
50.6
20.5
53.6
Dall-E mini
14.6
17.8
63.1
23.2
62.5
Dall-E mega
19.9
14.6
69.5
25.3
65.8
Stable Diffusion
24.5
9.4
70.8
36.2
72.1
Composable Diffusion
24.1
11.2
71.3
33.7
76.6
Table 2: Evaluation metrics for models on DrawBench.
4.2
Evaluation Results
Model
R
CLIP
SSA
Most text-to-image works evaluate with metrics for the quality and fidelity of the generated images, e.g., FID and IS, which barely reveal the semantic consistency. Differently, CLIP and R measure the similarity between the embeddings of text prompts and generated images, but they are limited in veri- fying fine-grained semantics. Comparatively, SSA better measures the consistency of semantic con- cepts and provides more insights into fine- -re-grained alignment. The metrics evaluated on MS COCO are reported Table 1. Furthermore, we compare the semantic metrics CLIP, R, and SSA on more challenging benchmark DrawBench, as detailed in Table 2. The findings demonstrate that stable dif- fusion excels in producing images with superior fidelity. In contrast, composable diffusion exhibits stronger capacity to capture fine-grainee seman- tic nuances, as evidenced by its higher and SSA scores. noteworthy that SSA proves to less susceptible to the quality generated images, and it provides deeper insights into the intricacies of fine fine-graing grained semantic consistency.
StackGAN
44.5
16.2
34.6
DF-GAN
53.4
18.4
49.0
Dall-E mini
60.7
23.2
52.8
Dall-E mega
62.0
25.3
55.9
Stable Diffusion
67.8
32,5
62.7
Composable Diff.
69.4
31.2
66.3
model that can recognize visual features and asso- ciate them with words, allowing it to judge whether image is semantically meaningful or not. Higher CLIP scores indicate that the generated images similar real-world images. R precision (Xu al., 2018) is score that learns the embeddings of real images that are similar the embeddings of their corresponding captions. To evaluate genera- tive model, we use one of the captions to generate an image from it. We then sample another 29 cap- tions randomly from the dataset and calculate the similarity between the embedding of the generated image and the embeddings of the 30 captions.
In Table 3, we assess Dall-E and diffusion-basee models using prompts with varying degrees of mu- tation sourced from MS COCO. Overall, diffusion- based models perform better than Dall-E models while Composable Diffusson generates more ac- curate fine- aineigraneed details than Stable Diffusion. Table 4 reveals that Stable Diffusion yields the most favorable performance when mutations af- fect the subject term, while Composable Diffusion demonstrates greater resilience to diverse muta- tions impacting the relation term. Furthermore, our evaluation indicates that Dall-E mega, which bene- fits from additional training data, exhibits enhanced robustness to mutated prompts. Additionally, the evaluation demonstrates that SSA is a valuable tool for gauging the fine-graing graiggr consistency mutated prompt generation.
Datasets We first evaluate method the eval- uation set of MS COCO (Lin al., 2014). We use 250 randomly chosen maage-caption pairs from the validation set. Specifically, we generate syntax- similar prompt mutations per sentence. In total, we have 1500 prompts for testing the text-to-image generation models. DrawBench (Ruiz et al., 2023) is a dataset for multi-dimensionn evaluation of text-to-imanl -il-tiimmage models, with text prompts designed probe different semantic properties of models. contains 11 categories of prompts, testing different capabilities of models such as compositionality, car- dinality, spatial relations, and the ability to handle complex text prompts or prompts with rare words.

---

Table 3: Evaluation with SSA on prompts with different mutations. Percentage calculated with mutated words the total words in the sentence.
Model
Dall-E mini
Dall-E mega
Stable Diffusion
Composable Diffusion
Original
62.5
65.8
72.1
76.6
Mutations (13%)
58.6
62.2
70.4
74.3
Mutations (29%)
51.3
54.0
65.8
68.9
Mutations (54%)
47.4
50.5
62.1
64.7
Table Evaluation of Dall-Es and diffusion models on original prompts and prompts with different types of mutations with SSA.
generation errors often take place when an object is combined with context. Models also fail to generate all the constituencies when the description too long or too complicated.
Model
Prompt M_sub
M_obj M_rel
Dall-E mini
62.5
58.2
52.6
47.8
prompt
generation
prompt
generation
Dall-E mega
65.8
65.3
59.9
54.6
Stable diffusion
72.1
67.2
64.1
59.4
Composable diff.
76.6
69.7
64.5
64.2
green cup
pizza in suitcase with take.
and a blue phone.
4.3 Discovered Typical Generation Errors
The proposed SSA capable of discovering gen- eration errors of diverse kinds. In our experiments with DALLE and Diffusion models, we mainly find four types of generation errors: under-generation, incorrect constituency, incorrect dependency, and semantic ambiguity. To provide glimpse of the diversity of the uncovered errors, this section gives some examples of the errors.
Figure 5: Examples of incorrect constituency errors.
4.3.2 Incorrect Constituency
token/phrase could also be incorrectly generated to another concept that seems semantically unre- lated. For example, in Fig. 5, the image on the left-top is generated with two phones by mistake. The left-bottom image has the right objects but in- correct color attributes. In the right example, the top image contains pizzas in suitcase. While in the bottom image, a pizza and tool in suit- case are generated. Many errors of this type ap- pear when models are accordance with the right structure of the scene graph but fail to generate the correct objects or context.
prompt
generation
prompt
generation
parking near traffic light on the street.
hair drier underneath
sheep.
Figure 4: Under Under-generatee generationn examples.
4.3.3 Incorrect Dependency
all the constituencies are correctly generated but the relationship between them is incorrect, it is an incorrect dependency error. In Fig. 6, all the objects are generated correctly in the images. However, in the left example, the bicycle under the boat rather than "on top of".. In the right example, the cat is inside the box in the top image and on the left of the box the other image. It is also observed that models struggle to generate the right relationship that not prevalent in daily life.
4.3.1 Under-Generation
If some words are mistakenly missing (i.e. do not appear in the generated image), it is under- generation error. Fig. 4 presents examples that contain under-generation errors. In the genera- tion, "traffic light" and "hair drier" are mistakenly missing, which leads to the target image differ- ent semantic content. our observation, under-

---

nificantly advances research on multi-modal im- age synthesis. By leveraging contrastive language- image pre-training (CLIP) (Radford et al., 2021) 1 or GAN inversion techniques (Xia et al., 2022), pre- trained GANs can be applied to text-driven image synthesis and editing tasks. Recently, the power of handling input from multiple modalities has led to the popularity of Transformer models (Vaswani et al., 2017). Significant progress has been made in multi-modal image synthesis. For example, DALL- (Ramesh et al., 2021, 2022) trained a large-scale auto-regressive Transformer on a large amount of image-text pairs to produce a high-fidelity gener- ative model through text prompts. Stable Diffu- sion (Rombach et al., 2022) adopted latent diffu- sion models to achieve favorable results across wide range of multi-modal image synthesis tasks. Although high-fidelity image synthesis is signifi- cantly advanced by those efforts, the fine-grainee semantic alignment between the text and the gener- ated images is seldom deeply investigated.
prompt
generation
prompt
generation
A bicycle on top of a boat.
A cat under
Figure 6: Incorrect dependency error examples.
prompt
generation
prompt
generation
pear cut into seven pieces arranged in a ring.
An orange
and a car.
Figure 7: Semantic confusion generation errors.
Evaluation of Iext-to-Image Models. The rapid advance in ext-too-mmae generation offers unprece- dented generation realism and editing possibilities, which have influenced and will continue to influ- ence our society. Establishing an accurate, reliable, and systematic evaluation framework is highly nec- essary to evaluate text-to-image generation models. However, evaluating the quality of the image gen- erative models has proven be challenging, demonstrated in (Theis al., 2015). In most of the multi-modal image synthesis methods (Ramesh et al., 2021, 2022; Rombach et al., 2022), image quality and text-image aligmment are the main fac- tors considered in the evaluation process. Com- monly used evaluation metrics are Inception Score (IS) (Salimans et al., 2016) and the Frechet Incep- tion Distance (FID) (Heusel al., 2017) for im- age fidelity, and CLIP score (Radford al., 2021) for text-image alignment. The metric R-score (Xu et al., 2018) relies on the Cosine similarity between image and text embeddings to determine whether a generated image is more similar to the ground truth texts than random samples from the dataset. However, R-score does not directly measure se- mantic consistency and can be highly biased by the dataset. SOA (Hinz al., 2020) attempts address this issue by utilizing pre-trained object detection model evaluate whether an object men- tioned the text exists in the generated image. However, SOA is not suitable for datasets where
4.3.4 Semantic Ambiguity
Semantic confusion refers state of uncertainty or ambiguity that arises when there is a lack of clarity or consistency the meaning of terms or concepts. can occur in various contexts, such as in language, communication, or knowledge rep- resentation, and can lead to misunderstandings or errors in reasoning. In image generation, semantic confusion can arise due to the multiple meanings of words or phrases, idioms or figurative language, or the lack of context. In Fig. 7, on the left, the model fails to understand the "ring" describes the shape of pear pieces instead of the "ring" shown in the im- age. On the right, "orange" should be generated as a fruit but the model confuses and generates a car in the color of "orange". These examples indicate that the current generation models need to be im- proved with respect to the semantic understanding of token/phrase in its context.
5 Related Work
Text-to-Image Generation. The goal of text-to- image generation is to create realistic images with textual descriptions. The prosperity of large-scale Generative Adversarial Networks (GANs) (Brock et al., 2018; Karras et al., 2019, 2020, 2021) sig-

---

only one object appears in the generated images. CLIP score (Hessel al., 2021), which is designed for image captioning and adopts cosine similarity CLIP embeddings, may not explicitly associate attributes with objects and overlooks the semantic variations. Because none of the existing measures are comprehensive enough, current works utilize many metrics. The performance assessment is even more challenging in the text-to-image generation task due to the multi-modal complexity of text and image. This motivates us develop new evalua- tion method to compare text-to-image alignment.
work, the word embeddings and scene graph con- struction from the pre-trained models may intro- duce some in-domain biases. The effectiveness of SSA relies on the expressiveness of the representa- tion of semantic components from each modality. If the text token or generated content is out of the domain of pre-trained models. it can limit the com- prehensiveness of the evaluation. Similarly, the interpretation and analysis of semantic contrastive comparison results can be subjective, potentially introducing biases in the evaluation process.
Limited Assessment of Image Quality. The SSA method primarily focuses evaluating the align- ment and consistency between generated images based on prompt mutations. While this provides insights into the model's robustness and ability to maintain coherence with variations in prompts, it doesn't tt directly assess the quality and fidelity of the generated images. Factors like visual realism, level of detail, and adherence to specific attributes or style may not be adequately captured by the se- mantic contrastive comparison alone. Therefore, the SSA method may not provide holistic evalua- tion of the overall image generation capabilities of text-to-image models.
6 Conclusion
The effectiveness and robustness of text-to-image generation models are of vital importance to en- sure their practicality in real-world upplications. In this paper, we introduce Structured Semantic Alignment (SSA) with contrastive learning for text- to-image generation models. Through our exper- iments, we have identified limitations of existing state-of-the-an art models in text-to-image generation. Our assessment highlights that these models may struggle with synthesizing semantic concepts in a given context. Designing methods for overcoming the limitations and enhancing the model's ubility to handle wider range of semantic variations would be intriguing area for future research. Improving the understanding and measuring the fine-grained semantic consistency across different modalities is a critical aspect to consider in advancing multi- modal generative techniques.
It is important to recognize these limitations and consider them when interpreting the results and conclusions derived from the SSA approach. Fu- ture research and refinement of SSA and similar methodologies will aim to address these limitations to ensure a more reliable and comprehensive evalu- ation text-to- to--iiaiige generation models.
7 Limitations
References
While the proposed method described above offers a comprehensive evaluation approach for text-to- image generation models, it also has its limitations. Here are some limitations of this method:
Andrew Brock, Jeff Donahue, and Karen Simonyan. 2018. Large scale gan training for high fi- delity natural image synthesis. arXiv preprint 18Xiv: 1106.
Limited Semantic Expressiveness. The proposed SSA method relies on prompt mutations where single word in the original sentence replaced with a semantically similar or nonequivalent word. While this variation enables the exploration of dif- ferent inputs, it may not capture the full range of semantic concepts and complexities present in nat- ural language. As a result, the evaluation might not fully reflect the model's 's ability understand and generate diverse visual interpretations based on complex textual descriptions.
Boris Dayma, Suraj Patil, Pedro Cuenca, Khalid Saiful- lah, Tanishq Abraham, Phuc Le Khac, Luke Melas, and Ritobrata Ghosh. 2021. Dall-e mini.
Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. 2021. Clipscore: A reference- free evaluation metric for image captioning. arXiv preprint arXiv:2104.08718
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner. Bernhard Nessler, and Sepp Hochreiter. 2017. Gans trained by two time-scale update rule converge to local nash equilibrium. Advances neural informa- tion processing systems, 30.
Dependency on In-domain Data Resources. While SSA provides a systematic evaluation frame-

---

Tobias Hinz, Stefan Heinrich, and Stefan Wermter. 2020. Semantic object accuracy for generative text- to-image synthesis. IEEE transactions on pattern analysis and machine intelligence.
Tim Salimans, lan Goodfellow, Wojciech Zaremba, Vicki Cheung. Alec Radford, and Xi Chen. 2016. Improved techniques for training gans. Advances in neural information processing systems, 29.
Tero Karras, Miika Aittala, Samuli Laine, Erik Härkö- nen, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. 2021. Alias-free generative adversarial networks. Advances in Neural Information Processing Systems, 34:852-863.
Christoph Schuhmann. Romain Beaumont, Cade W Gor- don, Ross Wightman, Theo Coombes, Aarush Katta, Clayton Mullis, Patrick Schramowski, Srivatsa R Kundurthy, Katherine Crowson, et al. 2022. Laion- 5b: An open large-scale dataset for training next generation image-text models.
Tero Karras, Samuli Laine, and Timo Aila, 2019. A style-based generator architecture for generative ad- versarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recogni- tion, pages 4401 44100.
Kaihua Tang, Yulei Niu, Jianqiang Huang, Jiaxin Shi, and Hanwang Zhang, 2020. Unbiased scene graph generation from biased training. In Conference on Computer Vision and Pattern Recognition.
Tero Karras, Samuli Laine, Miik Aittala, Janne Hell- sten, Jaakko Lehtinen, and Timo Aila. 2020. An- alyzing and improving the image quality of style- gan. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 81 10-8119.
Ming Tao, Hao Tang. Songsong Wu, Nicu Sebe. Xiao Yuan Jing, Fei Wu, and Bingkun Bao. 2020. Df-gan: Deep fusion generative adversarial net- works for text-to-image synthesis. arXiv preprint 1xXiv:0008.0886.
Lucas Theis, Aäron van den Oord, and Matthias Bethge, 2015. A note on the evaluation of generative models. arXiv preprint 1rXiv:1511.01844
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. In European confer- ence on computer vision, pages 140-755. Springer.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances neural information processing systems, 30.
Nan Liu, Shuang Li, Yilun Du Antonio Torralba, and Joshua Tenenbaum. 2022. Compositional visual generation with composable diffusion models. arXiv preprint (2Xx::220660711
Weihao Xia, Yulun Zhang. Yujiu Yang, Jing-Hao Xue, Bolei Zhou, and King-Hssan Yang. 2022. Gan in- version: A survey. IEEF Transactions Patlern Analysis and Machine Intelligence.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pages 8748-8763. PMLR.
Tao Xu, Pengchuan Zhang, Qiuyuan Huang. Han Zhang. Zhe Gan, Xiaolei Huang, and Xiaodong He. 2018. Attngan: Fine-grained text to image generation with attentional generative adversarial networks. In Pro- ceedings of the IEEF conference computer vision and pattern recognition, pages 1316-1324.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. 2022. Hierarchical text- conditional image generation with clip latents. arXiv preprint 2204:2200.
Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaogang Wang. Xiaolei Huang, and Dimitris N Metaxas. 2017. Stackgan: Text to photo-realistie image synthesis with stacked generative adversarial networks. In Proceedings of the IEEE international conference computer vision, pages 1915-5
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. 2021. Zero-shot text-to-image gen- eration. In International Conference on Machine Learning, pages
PMLR.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. 2022. High- resolution image synthesis with latent diffusion mod- cls. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10684-10695.
Nataniel Ruiz, Yuanzhen Li. Varun Jampani, Yacl Pritch. Michael Rubinstein, and Kfir Aberman. 2023. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In Proceed- ings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition, pages 22500-22510.

---

A Architecture
performs aggregation, providing multiple views or representations of the graph. Attribute feature embedding plays crucial role in our work as it leverages attributes to provide information about an object's 's details. Our approach involves two distinct stages. Firstly, all attributes within the graph are extracted and stored first. Subsequently, two sets of operations are performed.
A.1 Structure Embedding Extraction
Scene Graph is graph-based semantic represen- tation of image contents. They encode the objects in an image, their attributes, and the relationships between objects. Parsing an image into a scene graph involves several steps, including object de- tection, object recogmition, and relationship extrac- tion. A high-level overview of the process is as follows: (i) Object detection: The first step is to detect the objects in the image. Detection models be used to identify objects and their locations. (ii) Relationship recognition: We extract the rela- tionships between the objects in the image. This involves identifying how the objects are related to each other, such as whether one object is in front of or behind another, or whether two objects are next to each other. (iii) Scene graph construction: Once the objects and their relationships have been identified, scene graph can be constructed. It can be represented as directed graph, where the ob- jects are nodes and the relationships between them are edges.
Top results
Top results
CLP
CliP
55A
SSA
Score
Matric comparion Top
compariss
Top results
results
D.P
CUP
SSA
10
Mabric compurioon
Harriic compariton with Tap IC
Syntax Parsing aims to create structural repre- sentation of a sentence, typically in the form of a parse tree or dependency tree. We apply the de- pendency parsing which focuses on identifying the relationships between words in sentence by repre- senting them as directed dependencies. Each word in the sentence is considered node, and the rela- tionships between the words are represented as di- rected edges. The edges indicate the syntactic role each word and its dependence on other words. By analyzing the syntactic structure, models can identify the subject and object of sentence, extract key phrases, detect negation or sentiment-bearing words, and perform various linguistic analyses.
Figure 8: Different metrics evaluated on results ranked by CLIP score in the first row and results ranked by SSA in the second row.
In one set of operations, we treat object attributes similarly to object structure representation. This involves disregarding attribbte values and focus- ing on extracting each attribute associated with an object. When aligning objects from two differ- ent knowledge graphs, we observe that the objects to be aligned often possess similar attribute struc- tures. Exploiting this observation, we simulate the attribute structure to create comprehensive repre- sentation. In the other set of operations, we aim to represent the object attributes in suitable format. In this paper, set the embedding size for all layers 128 and used mini- method with a batch size of 512. For each experiment, the model is trained for 300 epochs and set the corresponding learning rates. In our approach, we utilize GloVE word embeddings with a hidden dimension of 300. Phrases are segmented into individual words based spaces.
A.2 Graph Attention Network
Due the similarity of the structures of aligned se- mantics across different modalities, graph structure information is utilized for multi-modal alignment tasks. The first step is to construct graph represen- tation of the data, where nodes represent entities or elements of interest, and edges capture the rela- tionships or connections between them. Each node in the graph is associated with an initial node em- bedding. To enhance the modeling capacity and capture different aspects of the graph, GAT often employs multiple attention heads. Each attention head independently computes attention weights and
words replaced with a special token, <UNK>.
To obtain object proposals, we employ an off- the-shelf Faster RCNN model as the object detector, The backbone of this detector is ResNet- 101, which has been pre- are-trained on Visual Genome, Regard-

---

"There is orange van parked on the street near building
Scene Graph
00
2-5
Parser
QN
Fuse- Attention Module
00
1-th L-there
S-van
Prediction Head
orange
park
00
00
#crango
6-parkeed
distill
Root node
9-street
12-bulding
near
street
building
2nd-layer nodes
BERT
Graph AttNet
Semantic Consistency Alignment
3rd-layer nodes
Figure 9: The pipeline of the evaluation of semantic consistency with SSA.
Original: A little bird on ree
Mutation: A little bird under tree
Mutation: A little bird on fire hydrant
Saub: 0.25
Suub 0.26
Syub: 0.21
Syub: 0.16
Saun: 0.05
Suua 0.10
Soby: 0.16
Subj: 0.23
Soby: 0.25
Soby: 0.19
Sony: 0.25 Syet
Saby: 0.21 Ste
Syat 0.22
Syat 0.15
Syat 0.09
Syet 0.12
Figure 10: Example mutations on the relation component and object component: Given "A little bird on a tree" the input prompt, change "on" to "under" and "tree" to "fire hydrant".
ing visual features, we use the 2048- dimensional feature vectors obtained from Bottom-up attention. These features are pre-trained using 1600 object labels and 400 attributes from the dataset. It is important to note that the visual features extracted from this process remain frozen during the training phase.
utilize word embeddings. This involves choosing words with high vector similarity and identical tags as replacements for the original token in the modi- fied sentences. However, since word embeddings lack contextual information, this approach often yields sentences that do not align with common language usage.
For instance, while the word "fork" may exhibit high vector similarity and share the same POS tag as the word "plate", the sentence "He uses the plate to eat" does not make sense, unlike the coherent sentence "He uses a fork to eat".
B Prompt Mutation
To assess structural invariance, necessary to compare two sentences with identical syntactic structures but differing in at least one token. Our approach focuses on modifying single token in an input sentence while adhering certain constraints, resulting in set of sentences that are both struc- turally identical and semantically similar. Specif- ically, we modify one token in the input sentence a time by replacing it with another token of the same part of speech. For instance, in the source sentence, we mask the token "lovely" and replace it with the top-k most similar tokens, generating k similar sentences. This process is applied to every candidate token in the sentence, limited to nouns and adjectives to maintain grammatical correctness.
Hence, the selection of replacement tokens must consider both vector similarity and contextual ap- propriateness to ensure that the generated sentences maintain linguistic coherence and align with natu- ral language usage.
There are also challenges selecting replace- ment tokens. One straightforward approach is to

---

Semantic concept count
Frequent words Subject
Frequent words Relation
Unique phrase
Man
With
Unique dass
Women
Newr
200
By
Sendwich
Under
1.0
2.0
2.5
50
100
150
100
200
300
400
500
Figure 1I: Statistics semantic concepts by phrase and by category in the prompts Draw Bench. The right two plots illustrate the most frequent words that appear "subject" and "relation" in the sentence syntax respectively.
CLIP
A cai lying in a bucket
A car besides blue cube
SSA
DALL-E
Comp. Diffusion
Stable Diffusion
DALL-E
Comp. Diffussonn
Stable Diffusion
Figure 12: Comparison of generation results ranked by CLIP and SSA given prompts: (i) "A cat lying in bucket" and (ii) "A car besides blue cube". The green and orange bounding boxes the prompts and images indicate the fine-grained alignment of semantic concepts.
Prompt: WWoan in red and blue jacket
Mutation: |gill
Mutation: man
Mutation: cow
Prompt: A man sits near dog
Mutation: puppy
Mutation: sheep
Mutation: traffic light
Prompt: A cat on a table
Mutation: above
Mutation: at
Mutation: under
Figure 13: Visualization of the examples generated by Stable Diffusion with different mutations of semantic concepts different prompts: row mutates subject the prompt, the second row shows mutation the object and the last mutates the relationship between the subject and the object in the sentences.


Reviews:
[{"review_id": "GocYEBmo5b", "reviewer": "Reviewer_kaoX", "summary": "", "questions": "Please see Reasons To Reject", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer kaoX for the positive feedback and insightful suggestions. We agree that further investigating the performance of the proposed method with longer prompts is an important aspect to strengthen the paper's contribution. We acknowledge the reviewer's suggestion to assess the effectiveness of both existing methods and SSA on longer descriptions.\n\n1. Dataset statistics \n\nFor the study, we selected a set of 250 sentences sourced from MS COCO. Each sentence underwent 5 distinct mutations. In total, the dataset comprised 1,500 sentences, including the original ones, with an average sentence length of 10 words. There are 345 sentences with lengths less than 6 words, 798 sentences ranging from 6 to 12 words, and 357 sentences with more than 12 words.\n\n2. Comparison of different prompt lengths\n\nIn the experiment, we found that 67% of the observed errors (326 out of 486) are generated from longer input prompts (beyond the average length of 10 words). This outcome reveals the fact that increased sentence length correlates with higher complexity. Additionally, it's important to note that the errors we uncovered exhibited lower SSA scores. This highlights a distinct pattern in the relationship between sentence length, difficulty, and error occurrence.\n\nTo further investigate this, we plan to conduct additional experiments using longer prompts from the MS COCO dataset, as well as prompts of varying lengths. We will evaluate both the existing methods and the SSA method on these prompts and provide a detailed analysis of the results. This will allow us to showcase the capabilities of SSA in handling complex and lengthy textual descriptions, which aligns with the core motivation of our work.\n\n3. Other baselines\n\nRegarding the suggestion to compare SSA with a wider range of compositional T2I baselines, we appreciate the feedback. In this work, we are evaluating the existing and most popular methods with open-sourced codes and model weights. We will certainly expand the scope of our comparison to include more compositional generation methods. This will provide a more comprehensive evaluation of the performance and robustness of our method across various generation models.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The proposed SSA framework is very interesting and I enjoyed reading this work. Experiments sufficiently demonstrate the utility of the proposed framework (although it could be further improved, please see reasons to reject). The notion of structured embeddings and the underlying principle of SSA presented in section 3.4 could be potentially useful in other tasks/domains. Overall the paper is solid with interesting and important contributions in T2I. ", "reasons_to_reject": "I do not find any major weaknesses in the proposed work. One aspect that could be further improved to make the contribution strong is that the paper could have shown how well the proposed evaluation method could handle long prompts. As the main contribution/utility of the paper is to provide a better way to evaluate complex or long prompts, it would be useful to show how well the existing methods and the proposed SSA method evaluate the generated images with long descriptions. It’s not very clear to me on the average length of prompts that were drawn from MS COCO. Please clarify. (I assume mutations do not change the length of the prompt). Another suggestion would be to add some comparisons of existing methods and SSA on prompts of varying lengths. That would be a really interesting analysis to perform. Finally, the paper compares only one compositional T2I baseline (i.e. composable diffusion). For evaluating SSA, it would have been better to also consider more compositional generation baselines such as below:\n\nFeng, Weixi, Xuehai He, Tsu-Jui Fu, Varun Jampani, Arjun Akula, Pradyumna Narayana, Sugato Basu, Xin Eric Wang, and William Yang Wang. \"Training-free structured diffusion guidance for compositional text-to-image synthesis.\" arXiv preprint arXiv:2212.05032 (2022).\n\nJiménez, Á. B. (2023). Mixture of diffusers for scene composition and high resolution image generation. arXiv preprint arXiv:2302.02412.\n\nKumari, N., Zhang, B., Zhang, R., Shechtman, E., & Zhu, J. Y. (2023). Multi-concept customization of text-to-image diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 1931-1941). ", "questions_to_authors": "Please see Reasons To Reject", "paper_topic_and_main_contributions": "The paper proposes a new method for evaluation of text-to-image generation models. The key idea of the paper is to learn structured semantic embeddings across different modalities and aligning them in a joint space. Experiments demonstrate that the proposed evaluation allows for comprehensive and improved measurement of semantic consistency of text-to-image generation models.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "Please see Reasons To Reject"}, "extra_text": {}}, {"review_id": "nb52FLzBo2", "reviewer": "Reviewer_L4hu", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the feedback from reviewer L4hu regarding the necessity of human evaluation in assessing the effectiveness of the proposed method. We understand the concern regarding proving the superiority of SSA over existing metrics. \n\n1. Advantages of the proposed method\n\nFirstly, our method not only provides a solution to evaluate the generation in a fine-grained approach but also investigates the potential misalignment in the text-image pair. Specifically, SSA helps to decompose the rich semantics in different modalities and match the structured embeddings together in a contrastive way. \n\nMoreover, our assessment involved a visual inspection of the generated images with SSA lower than 0.5. Out of the total 486 observed errors, 364 of them exhibited an SSA lower than the CLIP score. Notably, the average CLIP score was approximately half of the SSA value. Besides, in the appendix, we provided the comparison of metrics on the generated images ranked by CLIP and SSA. It is discovered that SSA has better consistency with R score, while CLIP score is more correlated with IS which measures the image quality not the semantic accuracy. \n\nTo address this concern, we will incorporate human-based evaluation into our experimental setup by conducting user studies to assess the quality and alignment. We will also perform a comprehensive analysis by comparing a range of human-centered evaluation methods, including \"ImageReward\", in our future work.\n    \n2. Accuracy of the scene graph \n\nRegarding the dependency on scene graph generation, we acknowledge the concern. While SSA does rely on information from the scene graph, we believe that its emphasis on structured semantic alignment across modalities offers insights beyond the scene graph quality alone. This is also the first attempt to adopt structured embeddings to help measure the alignment of rich semantics existing in both modalities. Furthermore, it is observed that SSA can assist in discovering different types of generation errors. Nevertheless, we will make sure to discuss this potential limitation in more detail and explore ways to mitigate its impact.\n\nOverall, we are grateful for the thoughtful suggestions, which will undoubtedly contribute to the robustness and significance of our proposed method's evaluation. We look forward to addressing these concerns in our revised paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The idea of using mutated prompts to construct sontrastive pair makes sence.", "reasons_to_reject": "Human evaluation is necessary to access whether the evaluation of the proposed SSA could better align with human judgement. Otherwise, how to prove the proposed SSA is better than existing metrics like CLIP-score?\n\nThe accuacy of SSA is hignly depend on the scene graph generation of image. Therefore, I doubt about the practical effectiveness since existing scene graph generation models are not very preceise in practical use.\n\nComparison with human-based evaluation suce as \"ImageReward\" is necessary.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper focuses on evaluating text-to-image generation models. Specifically, this paper propose a new evaluation metric named Structured Semantic Alignment (SSA) to evaluate the semantic consistency between text and generated images. SSA first projects text/image features into a shared embedding space by text parsing and scene graph generation respectively. Then SSA is learned by contrastive learning. The positive and negative samples are generated by substituting words with semantically equivalent or nonequivalent alternatives.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generation"}, "extra_text": {}}, {"review_id": "ERc78IhebS", "reviewer": "Reviewer_4Ntp", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the feedback provided by reviewer 4Ntp regarding the need for a more comprehensive comparison with other evaluation methods and a more in-depth analysis of the experimental results. Your suggestions align well with our commitment to presenting a thorough and well-rounded assessment of the proposed method.\n\nTo address these points, we will significantly expand the discussion section of our paper. By presenting a comprehensive analysis of SSA's performance alongside these established methods, we aim to showcase its unique strengths and limitations in a broader context.\n\nFurthermore, we will dedicate more space to discussing the experimental results. This will involve a detailed breakdown of the performance of SSA across different scenarios, including variations in prompt length, complexity, and other relevant factors. We will also explore cases where SSA successfully uncovers limitations not captured by other methods, thereby providing valuable insights into the text-to-image generation process.\n\nWe thank the reviewer for these valuable suggestions, and we are fully committed to enhancing the depth and breadth of our paper's discussion and analysis sections to better highlight the contributions and implications of our work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper presents a well-motivated and novel approach to evaluate text-to-image generation models. \n2. The proposed approach is supported by experimental results that demonstrate its effectiveness in measuring semantic consistency between textual descriptions and generated images. \n3. The paper is well-written and clearly presents the motivation, approach, and experimental results. ", "reasons_to_reject": "The paper could benefit from a more detailed discussion of how it compares to other evaluation methods and a more detailed analysis of the experimental results. ", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper proposes a new approach called Structured Semantic Alignment (SSA) to evaluate the accuracy and robustness of text-to-image generation models. The main contribution of the paper is the development of a new evaluation method that measures the semantic consistency between textual descriptions and generated images. The authors argue that existing metrics lack insights into the assessment of fine-grained semantic concepts, such as object attributes, context details, and semantic relationships. The experimental results demonstrate the effectiveness of the SSA approach in evaluating text-to-image generation models.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a method for evaluating text to image generation models. They propose a metric Structured Semantic Alignment (SSA) SSA to address the gap in  existing evaluation methods on the assessment of fine-grained semantic concepts, such as object attributes, context details, and semantic relationships. SSA projects text/image features into a shared embedding space by text parsing and scene graph generation respectively, and is trained using contrastive learning.

The reviewers agree with the importance of the problem and are mostly excited about the work. However the reviewers do voice some key concerns:

&nbsp;  (1) the lack of comprehensive comparisons with existing methods, and lack of human evaluations.

&nbsp;  (2) lack of detailed analysis of the experimental results e.g. on longer prompts, complexity etc.

&nbsp; (3) comparisons using different compositional generation approaches.

Nevertheless, the reviewers find the idea interesting and well motivated. The authors' responses also indicate that they are committed to addressing the above issues. It is important for the authors to address these valid and important issues to substantiate the main claims and arguments made in the paper and strengthen their paper.
Decision: Accept-Findings