Probing into the Fine-grained Manifestation in Multi-modal Image Synthesis
Authors: Qianyu Feng
Peike Li
Yulei Sui
Hongyu Zhang
Venue: Submitted to ICLR 2023 (2023)
Keywords: Multi-modal image synthesis
semantic consistency measurement
robustness testing

Abstract: The ever-growing development of multi-modal image synthesis brings unprecedented realism to generation tasks.  In practice, it is straightforward to judge the visual quality and reality of an image. However, it is labor-consuming to verify the correctness of semantic consistency in the auto-generation, which requires a comprehensive understanding and mapping of different modalities. The results of existing models are sorted and displayed largely relying on the global visual-text similarity. However, this coarse-grained approach does not capture the fine-grained semantic alignment between image regions and text spans. To address this issue, we first present a new method to evaluate the cross-modal consistency by inspecting the decomposed semantic concepts. We then introduce a new metric, called MIS-Score, which is designed to measure the fine-grained semantic alignment between a prompt and its generation quantitatively. Moreover, we have also developed an automated robustness testing technique with referential transforms to test and measure the robustness of multi-modal synthesis models. We have conducted comprehensive experiments to evaluate the performance of recent popular models for text-to-image generation. Our study demonstrates that the proposed metric MIS-Score represents better evaluation criteria than existing coarse-grained ones (e.g., CLIP) to understand the semantic consistency of the synthesized results. Our robustness testing method also proves the existence of biases embedded in the models, hence uncovering their limitations in real applications.

Under review as a conference paper at ICLR 2023
PROBING INTO THE FINE-GRAINED MANIFESTATION IN MULTI-MODAL IMAGE SYNTHESIS
Anonymous authors Paper under double-bling review
ABSTRACT
The ever-grow ever-growing development of multi-modal image synthesis brings unprece- dented realism to generation tasks. In practice, it is straightforward to judge the visual quality and reality of an image. However, it is labor-consuming to verify the correctness of semantic consistency in the auto-generation, which requires a comprehensive understanding and mapping of different modalities. The results of existing models are sorted and displayed largely relying the global visual- text similarity. However, this coarse-grained approach does not capture the fine- grained semantic alignment between image regions and text spans. To address this issue, we first present a new method to evaluate the cross-modal consistency by inspecting the decomposed semantic concepts. We then introduce ric, called MIS-Score, which is designed to measure the fine-grained semantic alignment between prompt and its generation quantitatively. Moreover, have also developed an automated robustness testing technique with referential trans- forms and measure the robustness of multi-modal synthesis models. We have conducted comprehensive experiments to evaluate the performance of re- cent popular models for text-to-image generation. Our study demonstrates that the proposed metric MIS-Score represents better evaluation criteria than existing coarse-grained ones (e.g., CLIP) to understand the semantic consistency the synthesized results. Our robustness testing method also proves the existence of biases embedded in the models, hence uncovering their limitations in real appli- cations.
met-
CLIP
car lyvnn in
bucckt
car besides a blue cube
MIS
Score
DALL-E
Comp. Diffusson
Stable Diffuusio
DALL-E
Comp. Diffusion
Stable Diffusion
Figure 1: The generated images of sfte-oo-thee-an ttt-thh-rrt art multi-modal image synthesis models (DALL-
(Cho et al., 2022), Composable Diffusion (Liu et al. 2022). Stable Diffusion (Rombach et al.) 2022 selected by CLIP score and MIS-Score for two prompts: (i) A cal lying in bucket. (ii) A car besides blue cube. The green and orange bounding boxes the prompts and the images indicate the fine-grained alignment of semantic concepts.

---

Under review as a conference paper at ICLR 2023
1 INTRODUCTION
Multi-mode image synthesis (Esser et al. 2021 Ramesh et al. 2021; Rombach et al. 2022) aims to generate images given input prompts such as natural language descriptions or set of keywords. This multi-modal synthesis has a variety of potential applications, e.g., computer-aided design, text- guided photo editing, etc. In recent few years, it has witnessed unprecedented development based on the advance of deep learning techniques like Generative Adversarial Networks (GANs) (Good-) fellow et al.] 2020; Zhang et al., 20177] Tao et al. (220)) and cross- cross-modal pre-training models, e.g., CLIP (Radford et al., 2021 Ramesh et al. (021). Compared to the visual clues (segmentation maps, regional edges, etc.) adopted for single-modal image synthesis, cross-modal guidance, e-g., lan- guage descriptions, provides an alternative but more intuitive and natural ways to express semantic concepts. The flexibility of text-to-image generation greatly lower the barrier for wide range of public users to unleash their creativity in image generation and editing.
However, due to the substantial domain gap, effective transfer and fusion of heterogeneous informa- tion from different modalities remains big challenge in the multi-modal synthesis tasks. Moreover, there usually exist many-to-many mapping relationships in the multi-modal image synthesis tasks, i.e., one image may correspond to multiple textual descriptions counterparts and vice versa. In practice, the synthesized images are not always consistent with the text prompts given by users. Ad- ditional effort is usually required to manually validate and select the satisfying synthesized results. Various aspects of the synthesized images need to be qualified, e.g. visual quality and the accuracy of objects, attributes, and contextual information. Thus, automatic evaluation process and com- prehensive and objective evaluation metric are of vital importance in assessing the effectiveness of multi-modal image synthesis models.
To achieve these goals, previous works usually report multiple metrics to cover different aspects. The primary factors considered in the evaluation process are image quality and text-image similarity. Popular metrics include Inception Score (IS) (Salimans et al.) (016)) and Frechet Inception Distance
(FID) (Heusel et al.] 2017) for assessing the image fidelity, and R- R-precision(RP) (Xu et al., 2018) and CLIP score (Radford et al. (021)) for measuring the cross-modal alignment. These metrics work well for the generation from simple prompts, e.g. description of a single object. However, for prompts with multiple objects and additional context information, simply adopting these metrics is insufficient and may lead inaccurate or inconsistent results. As shown in the first row of Fig, T the ranking of the synthesized images based on one of the most common-used metrics (i.e., CLIP score) not strongly correlated to the language descriptions. Additionally, the existing metrics lack insights in the assessment of fine-grained semantic concepts, such object existence, attribute accuracy, spatial location, etc. These factors are critical in evaluating the performance of multi- modal image synthesis methods, especially when input prompts are composed of multiple semantic concepts.
To address the above issues, introduce MIS-Score, a new metric for Multi- Image Synthesis to measure the cross-modal semantic consistency by inspecting and capturing the fine- grained mapping across the input language and output vision modalities. As shown in Fig. T given a prompt "A cat lying in a bucket", we first parse the semantic concepts in the language description as ("a cat", "a bucket"). We then perform visual grounding by locating each semantic concept with the visual components in the generated image to calculate the fine-grained text-image alignment. These semantic concepts can include but not limited to: subject, object, location, attribute, and re- lationship. The cross-modal consistency is measured with MIS-Score by aggregating the alignment score on each semantic concept. Based on the proposed metric, we develop an automatic testing technique, referential transform, to evaluate the robustness of models with different combinations of visual concepts. The key idea is that an accurate multi-modal synthesis model should produce con- sistent generations given prompts with similar meanings. Correspondingly, models should produce different results given prompts with different semantic concepts. And when there is any mutation in the prompt, the synthesis result is required to always be consistent with the input. In this way, the robustness of models can be evaluated and measured by MIS-Score
Our major contributions in this work are summarized as follows,
To the best of our knowledge, we, for the first time, propose a new approach for measuring the inneggraied semantic consistency for the multi-modal image synthesis tasks.
2

---

Under review as a conference paper at ICLR 2023
(a) Tmaaee Generation
(b) Fine tine-grained Semantic Consistency
car near bloe tube
Prompt
Fe
Image
BT
Semanntic concepts
car
C_obj
blue cube
Ey
Location
C_rel
Regression
Gr.
SMIS
A car near red bal!
(c) Referential Transform
Ey
Text Encoder
Semontic
Text-to-lmagg hest---mmggee Generation
G₁
Image Generator
cube
bue
Fine-grained
Score
Semantic
E
Image
red ball
Consistency
MIS-
red ball
Cross-modal Verification
(a) Text-to- Image Generation: Given an input prompt, the results are generated by first encoding the text embeddings and then being transformed into 3-dimension RGB image. (b) Fine- grained Semantic Consistency: It first parses the prompt into semantic concepts for inspection. Each term is first encoded into embeddings and then grounded in the image for the corresponding visual components. The cross-modal consistency is measured with MIS-Score SMIS by aggregating the verification of each semantic concept. (c) Referential Transform: We first execute mutation on selected semantic concept and then apply the image synthesis with the mutated prompt. The robustness of generation models can be measured by calculating the MIS-Score.
Figure 2:
A new metric, MIS-Score, is developed for evaluating the cross-modal alignment, which outperforms the existing coarse gaatseerriind ranking mechanism for the synthesized images.
A robustness testing technique is designed and implemented evaluate the robustness of text-to-ine text-to-image generation models.
METHOD
In this section, new method is proposed for automatically evaluating the performance of multi- modal image synthesis models. Different from metries comparing the global representation of each modality, we aim to measure the manifestation of fine-grained semantics in the generation. The cross-modal semantic consistency is assessed by inspecting the fine-grained terms from the input prompts. Accordingly, we design new metric MIS-Score, denoted as SMTS. which is the aggrega- tion of the cross- moddll consistency on all the semantic concepts. The framework of our method is illustrated in Fig. [ In the following, The detailed explanation of MIS-Score can found in Sec. and the robustness testing procedure with referential transform discussed in Sec.
first introduce the technique for text-to-image generation.
2.1 TRANSFERABLE VISUAL. MODELS
standard image model jointly trains image feature extractor and a decoder to predict the target modality. As typical example, CLIP proposed by (Radford et al. (021) jointly trains an image en- coder and text encoder to predict the correct pairings of a batch (image, text) training examples. During the testing time, the learned text encoder synthesizes a zero-shot linear classifier by embed- ding the names or descriptions of the target as labels. Inspired by this, recent eext-to-image mod- els (Ramesh et al. 2021 Liu et al.) 2022 Rombach et al. 2022) aim to train transformer /Vaswani eet al.) 2017) to model the text and image embeddings as single stream of data. They first train an image auto-encoder and then train auto-regressive transformer model the joint distribution

---

Under review as a conference paper at ICLR 2023
the text and image tokens. In (Ramesh et al. (021), the overall learning objective is to maximize the evidence lower bound (ELB) on the joint likelihood the model distribution over images z, captions y, and the tokens for the encoded images:
In (o. 2
E (In y,,))
(1)
BKI.. (yo(y, x)), pos (v.:)))
However, it is widely discussed that current models cannot effectively handle long prompts with various compositions of concepts (Saharia et al. [022] Cho et al. 2022; Liu et al.) (022). Models tend to concentrate on the salient component (e.g. the main object in the description), and overlook other contextual details. It also explains why models often fail in precise generation given prompts with a combination objects.
2.2 FINE-GRAINED CROSS CONSISTENCY
Fine-grained semantic concepts. A single image usually contains rich semantics, covering mul- tiple objects with attributes and/or the description of their context. It is costly annotate all these semantic concepts with different granularity in large-scale image-text datasets. To this end, we use sentence constituent parsing (Mitchell 1994) extract the semantic concepts in each prompt. As shown in Fig. 2 we choose to collect pool of concepts including subject, object, and relation, by using off-the-shelf language parser (Schuster et al., 2015). Given the concept pool, the semantic representations for regions are created in two steps: T) We extract short phrase for each concept by grouping the words in the children of each target word. For example, the object phrase in "a car near blue cube" will be parsed "a blue cube". 2) Each phrase is then encoded into semantic rep- resentations by using the pretrained language encoder Ey. Ey is a language encoder that converts natural language text semantic representation. Finally, all regional concepts represented by their semantic embeddings (ei) C' denotes the size of concept pool. While the semantic concepts are parsed from image descriptions, our method is not constrained by the particular texts that pair with images. More importantly, in light of the powerful language encoder which has seen easily expand our concept pool and scale it up by collecting
many words in natural language, more specific attributes, like color and gender, which is difficult to achieve by human annotations.
Visual component validation. Each semantic embedding e, Rox L encoded by Er is then located in the generated image to its corresponding visual features. Our goal is to measure the semantic concept in regional visual-semantic space, which covers rich object concepts that can be used for measuring the semantic alignment at scale. In the visual-semantic space, the visual regional embeddings eu RcxHx w extracted from image representations should be matched to text representation éz. Ey is a visual encoder that takes image I and region location and outputs a visual representation for this region.
Here, the semantic embeddings serve as the query and the visual embeddings e, act as the key and value. With the multi-head attention, the related visual features will be gathered for each semantic embedding. The gathered semantic features are organized as semantic map e, E Rex Hx w that spatially aligned with the visual feature map eg. Thereafter, we project both the feature maps e and e, to the same semantic space (via linear projection and 1.2 normalization) obtaining el and e' and compute their semantic correlation as the consistency score for each pair (e,, C.) as:
(1
c,e22) 202
F(er, c2) = exxp
exp
(2)
where and a are learnable parameters. The verification scores model the semantic relevance of each visual feature to the linguistic expression. Therefore, we can establish a more salient feature map for the referred semantic concept by modulating the visual features with the verification scores pixel-wisely. Last, the cross-modal consistency between the input prompt and the generated image is calculated by aggregating the score for each semantic concept re-weighted by wy:
SMIS =
Σ w. Fle 1=0
(3)
4

---

Under review as a conference paper at ICLR 2023
2.3 ROBUSTNESS TESTING WITH REFERENTIAL TRANSFORM
As mentioned in (Rombach et al. 2022 Cho et al. 2022). current models tend to encode biases in the training dataset, which largely limits the generality and robustness of the generation. Referential transparency (Søndcrgaard & Sestoft, 1990; Mitchell 2002) has been adopted a key property of programming languages. An expression is 'referentially transparent" if replacing term in that expression with another term that refers to the same value does not alter the semantic meaning. If the term is replaced by another term with different value, the semantic meaning of the expression should be altered.
Inspired by referential transparency, we propose referential transform, an automated semantic con- sisteney checking technique for testing the robustness of text-to-image generation models. It is a one-way evaluation method without referring back to source modality. The general idea that perfect multi-modal synthesis should produce alike generations given similar prompts, while producing different results when the semantic concept changes. For a referred semantic concept, referential transform is to change it another semantic concept. For example, in Fig. El given "A little bird on a tree", we change the relationship between "little bird" and "tree" from "on" to "under". It is discovered in our study that the existing models often struggle in generating precise images under certain combinations of semantic concepts.
Original: A little bird on tree
Mutation: A little bird under tree
Mulation: A little bird on a fire hydrant
Saua 0.25
Saua: 0.26
Saub 0.21
Saud: 0.16
Syas: 0.05
Syud: 0.10
Saby: 0.16
Soty: 0.23
Sayy: 0.25
Soty: 0.19
Soa 0.25
Sany: 0.21
Syet 0.22
Syet 0.15
Syet 0.09
Syet 0.12
Syet 0.03
Syet 0.24
Figure 3: Demonstration of robustness testing with referential transforms: Given "A litte bird on a tree" as the input prompt, we first parse the semantic concepts in it. Mutation for relation term: change "on" + "under" and Mutation for object term: "tree" "fire hydrant".
Our robustness testing procedure is follows: (i) Given input prompt for first parse the phrase of the semantic concept for inspection in the following step. (ii) We then select another semantic concept replace the original one in the language description, yielding a mutated prompt fm- (iii) we generate new images with bn and then check the cross-modal consistency using SMTS between the newly synthesized images and the mutated prompt. An accurate multi-modal synthesis model should produce similar MIS-Scores given similar prompts, while producing different results given different prompts.
EXPERIMENTS
3.1 IMPLEMENTATION
Multi-modal image synthesis. In this section, conduct experiments to evaluate the accuracy
and robustness of current models for text-to-image generation. The models we used include (1)
GAN-based models pretrained MSCOCO: StackGAN (Zhang et al.] 2017) which generates high-resolution images by stacking two GANs and DF-GAN (Tao et al. 2020) which is a new one-stage text-to-image backbone that enhances the text-image semantic consistency with target- aware discriminator; (2) DALL-Es: Since the pretrained checkpoints of original DALL-E (Ramesh (02I)) have not been released, we experiment with two open-sourced implementations of DALL-E by (Dayma et al. 2021), i.e., DALLE-mmnni and DAL..E-mega During training, a VQ- GAN (Esser al.. 202 model pretrained on ImageNet is used the image encoder. BART trans-
'DALLE-mini:
httph
FDm...--meaa

---

Under review as a conference paper at ICLR 2023
Table 1: Evaluation of Text-to-lmage Synthesis models with CLIP score and proposed MIS
Score by measuring MIS-subject, MIS-object, and MIS-relation for the corresponding semantic concepts in the input prompts. Results are shown as percentages. Method
CLIPT
MIS-subject MIS-object MIS-relation
MIS-Score
Reference images
29.7
28.3
27.5
28.0
83.8
StackGAN
15.4
14.2
9.1
14.6
37.9
DF-GAN
12.6
16.8
12.6
15.8
45.2
DALLE-mini
22.6
20.5
17.9
24.7
63.1
DALLE-mega
23.5
22.3
21.7
25.5
69.5
Composable Diffusion
27.9
24.3
24.8
22.2
71.3
Stable Diffusion
28.6
24.5
21.6
24.7
70.8
former (1 Lewis et al. 2019) is trained on 15M image-text pairs from Conceptual Captions (Sharmal
al. 2018| Changpinyo et al. (02T) and OpenAl subset of YFCC100M (Thomee al., (016)): (3)
cuftusion-basee models: Composable Diffusson (Liu ct al. 2022) which introduces compositional
operators with multiple diffusion models, and Stable Diffusion (Rombach et al.) 2022) which is a
latent text-to-ine text-to-image diffusion model using frozen CLIP VIT-L/14 text encoder to condition the
model on text prompts and trained a subset of LAION-5B (Schuhmann et al. 2022). We adopt the open-sourced code in colat for experiments.
Semantic concept parsing. While such models are highly flexible, they struggle to understand the composition of certain concepts, such confusing the attributes of different objects relations between objects. In our experiments, sentence parsing is first applied to parse or decompose the prompt sentence into different semantic concepts. In linguistics, parsing means to break down sentence into parts to understand the sentence in comprehensive manner. In our experiments, we selected the three most common terms in linguistic typology: subject, object, and the relation between the former two terms. The weight parameter is set as 1/3 to calculate the mean average value on terms.
Visual components validation. Our model is pretrained on RefCOCO (Yu et al. 2016) for visual grounding. The feature encoder composed of CNN, transformer encoder layers and BERT. We use ResNet-101 as the CNN backbone followed by 6 transformer encoder layers in the visual feature extraction branch. These layers are initialized with the corresponding weights DETR (Carion
et al. 2020 The textual embedding encoder is initialized with BERT (Devlin et al. 2018
DALLE -mini
Composable
Stable Diffusion
DALLE
DALLE
Composuble Diffusion
Stuble
-mega
Diffusion
-mini
-megn
Diffusion
kid plays with a laptop
The chicken rucket
Figure 4: Visualization of text-to-image generation by DALLE-mini, DALLE-mega, Composable Diffusion, and Stable Diffusion. Images with red dotted border represent inaccurate generations of specific semantic concepts.
3.2 EVALUATION OF MANIFESTATION: DEVILS IN THE DETAILS
Though MSCOCO (Lin et al., 2014) valuable benchmark, it is increasingly clear that it has limited spectrum of prompts that lack enough insight into differences between generation models.
https://colal research goog hew.arahh...coee...o..aa.toaaaea model1
Composi cinolltttiinalioo visual htw.://www..........oo...o..a.a.

---

Under review as a conference paper at ICLR 2023
Table 2: Evaluation text-to-generation results re-ranked with MIS Score.
Method
CLIP
MIS-sub.
MIS-obj.
MIS-rel.
MIS-Scoret
SOAT
DALLE-mini 17.2
20.5
18.9
24.7
64.1
58.2
DALLE-mega
20.6
23.2
22.3
25.5
71.0
63.6
Composable Diffusion
25.3
24.3
24.8
22.2
71.3
65.4
Stable Diffusion
27.8
24.5
21.6
22.7
69.8
64.1
Recent works propose to evaluate models on smaller but more diverse natural language descriptions to systematically evaluate visual generation capability and the underlying biases. Motivated simi- larly, we introduce GroundBench, comprehensive set of prompts that support the evaluation and comparison of text-to-ing to-image models. Unlike previous methods, GroundBench is sourced from Ref- COCO Yuu et al. 2016) with paired images and texts, and contains diverse prompts to test different capabilities of models such as the ability to faithfully render various objects, spatial relations and some unpopular interactions between objects. We also filter out prompts with length larger than 15 words and prompts with rare words. Hence, is easier to use and more flexible to apply dif- ferent experimental settings. GroundBench comprises 2368 prompts in total, satisfying the need for a diverse and comprehensive dataset. More details are available on our project webpage !
Quantitative results. The evaluation of the recent text-to-image models is reported with quantitative results in Table [ We compare our proposed
with CLIP score (Radford et al.) (021). which calculates the cosine similarity of the global features of the image and text. Unlike the CLIP score, MIS- measures the distances between the fine-grained inspection terms (i.e., "subject", "object", "relation") and the grounded regional visual features. It is worth noting that our work focuses measuring the cross-modal semantic consistency rather than the reality of image quality. Some visualization examples are shown in Fig. [ With MIS-Score, it is casy to obtain more insights into individual semantic concepts: (i) Stable Diffusion generates better results aligned with the subject term in the prompt; (ii) Composable Diffusion performs better on generating the object term than other models; (iii) The diffusion- based models
DALLE- mega in aligning the
generated visual components with the relationships.
Top results| CLIP
Top results MIS
Top results CLIP
Top K results MIS
CLIP
25
CLIP
CLIP
25
CLIP MIS
MIS
SOA
20
SOA
20
5-
5
min
mn
min
min
2
(a)
4
2
(b)
4
2 (c)
4
2
(d)
4
Figure 5: Different metrics evaluated results ranked by CLIP score in (a) and (c) and results
ranked by MIS-Score in (b) and (d). Comparison with SOA score in (a-b), with IS in (c-d).
Better semantic-consistent results with MIS-Score. Most text-to-image works report metrics of image quality, e.g., FID and IS, which are not informative enough. CLIP score measures the text- image similarity which has limitations in verifying fine-grained semantics. We show results gener- ated by DALLE-mega ranked by CLIP and MIS-Score in Fig. 6 CLIP focuses more on matching images with the "sandwich" while MIS- Score better balances the verification on another semantic concept of the "plate". In Table 2, we report the evaluation of results re-ranked by MIS-Score. SOA (Hinz et al. (020) measures the accuracy of objects across modalities. Compared to CLIP, MIS better measures the consistency of semantic concepts and provides more insights in the fine- grained alignment. Additionally, we compare metrics CLIP. MIS with SOA on the top K results ranked by CLIP and MIS. For results ranked by CLIP in Fig. 5(a), MIS also gives a similar ranking and more distinguishable scores. And in Fig, ((b), CLIP is more fluctuant on results ranked by MIS.
https:://accymmous. 4open s sancer/MMl Scw.e-

---

Under review as a conference paper at ICLR 2023
Table 3: Robustness testing on DALL-Es and Diffusion-based models.
Method
Original prompt
Mutate-sub.
Mutate-obj.
Mutate-rel.
DALLE-mini
63.1
58.2
52.6
47.8
DAgLE-mega
69.5
65.3
59.9
54.6
Composable Diffusion
71.3
66.7
64.5
64.2
Stable Diffusion
70.8
67.2
64.1
59.4
In accordance with SOA, MIS better reflects the alignment on semantic accuracy. Given results in Fig. sf--d), we compare CLIP, MIS with IS. is shown that the results ranked by MIS-Score are less entangled with image quality.
0.22
0.21
0.18
0.16
0.12
0.11
CLIP Ranking
MIS Ranking
0.67
0.63
0.58
0.51
0.39
0.27
Figure 6: Images generated given the prompt "A sandwich placed on a plate". The first row shows the ranking results by the CL.IP score. The second row shows the results re-ranked by MIS-Score.
3.3 ROBUSTNESS TESTING WITH REFERENTIAL TRANSFORM
Woman and blue jacket
Mutation:
Mutation: man
Mutation:
Prompt: man sits near dog
Mutation: puppy
Mutation: sheep
Mutation: traffic light
Prompt: cation table
Mutation: above
Mutation: at
Mutation: under
Figure 7: We show the generation by Stable Diffusion with different mutations of semantic concepts on different prompts: the first row mutates the subject term, the second row shows mutation on the object term, and the last row mutates the relation between the subject term and the object term.
Most multi-modal image synthesis models are trained or using pretrained weights on large-scale datasets. The biases in the data collection are rooted in-depth. The effect of task-specific training on biases is an interesting open question. With referential transform, we operate three types of mutations for testing and results are listed in Table 3 Overall, the results consistent with prior analysis (Cho et al. 2022. Rombach et al.) 2022) where the performances of both DALLE-basee based and diffusion-basse based models drop when generating images with mutated prompts. We also have
8

---

Under review as a conference paper at ICLR 2023
additional evaluation results on different types of mutations subject, object and relation terms. They are not identical based on which biases are discovered.
In Table 3 it is discovered that Stable Diffusion performs the best for mutations the subject term. Composable Diffusion is more robust to different mutations on the relation term. Besides, comparing the results of DALLE-mini and DALLE-mege it can be observed that DALLE-mega which is trained on more data is more robust to the referential transforms. More visualized examples are given in Fig. 7 The results indicate that the model tend struggle for generating long-tailed semantic concepts and rare combinations of different concepts.
4 RELATED WORK
Multi-modal Image Synthesis. The goal of multi-modal image synthesis is to create realistic im- ages with natural textures, given cross-modal guidance, e.g., using text descriptions as a flexible and natural way to express visual concepts. The prosperity of large-scale Generative Adversar- ial Networks (GANs) (Brock et al. 2018, Karras et al. 2019; 2020 (02T) significantly advances research on multi-modal image synthesis. By leveraging contrastive language-image pre-training (CLIP) (Radford et al.] 2021) or GAN inversion techniques (Xia et al.) 2022), pre-trained GANs become applicable to achieve text-driven image synthesis and editing tasks, Recently, the power of handling input from multiple modalities has led to the popularity of Transformer models [Vaswani et al. 2017). Significant progress has been made in multi-modal image synthesis. For example, DALL-E (Ramesh et al., 2021; 2022) trained a large-scale auto-regressive Transformer on a large amount of image-text pairs to produce high-fidelity generative model through text prompts. For another example, Stable Diffusion (Rombach et al. 2022) adopted latent diffusion models to achieve favorable results across wide range of multi- modal image synthesis tasks. Although high-fidelity image synthesis is significantly advanced by those efforts, the fine-grained semantic alignment be- tween the text and the generated images is seldom quantitatively investigated.
Evaluation and Limitation. The rapid advance in multi-modal image synthesis offers unprece- dented generation realism and editing possibilities, which have influenced and will continue influ- ence our society. The establishment of an accurate, reliable and systematic evaluation framework is highly necessary to evaluate multi-modal image synthesis models and direct future directions. How- ever, evaluating the quality of the image generative models has been proven to be challenging as demonstrated in (Theis et al. 2015). In most of the multi-modal image synthesis methods (Ramesh) ct al. 202 I 2022 Rombach et al. 2022), image quality and text-image alignment are the main fac- tors considered in the evaluation process. Commonly used evaluation metrics are Inception Score (IS) (Salimans et al., 2016) and the Frechet Inception Distance (FID) (Heusel et al.) (017)) for image fidelity, and CLIP score (Radford et al., 202I) for text-image alignment. Because none of the exist- ing measures are perfect, it is usual to report many metrics, each of which assesses certain aspect. The performance assessment is even more challenging in the text-to-image synthesis task due to the multi-modal complexity of text and image, which motivates us to develop a new evaluation metric to compare text-to-image alignment fairly and confidently.
5 CONCLUSION
Although a line of methods has been proposed to advance the multi-modal image synthesis tasks, there have been comparatively fewer works investigating metrics for better evaluation. Most ex- isting metrics concentrate on measuring the image quality but are not informative enough in the find-grained semantic space, where different modalities should be well-aligned. In this work, we propose new evaluation metric for better understanding and measuring the fine-grained semantic consistency across different modalities. We believe this is critical avenue for future research and advancement. It is demonstrated in our experiments that existing state-of-the the text-to-image mod- els exhibit limitations when generating images depicting certain textual descriptions. Besides, we also propose referential transform for testing the robustness of models. is discovered in our assess- ment that existing models underperform when synthesizing semantic concepts in the context. They may fail when the mutation goes against certain phrases or commonsense. Developing methods to mitigate the biases and improve the robustness will be an interesting future research direction.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. arXiv preprint a809. 1096, 2018.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European conference on computer vision, pp. 213-229. Springer, 2020.
Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 1558-3568, 2021.
Jaemin Cho, Abhay Zala, and Mohit Bansal. Dall-eval: Probing the reasoning skills and social biases of text-to-image generative transformers. arXiv preprint arXiv:2202.04053. 2022.
Boris Dayma, Suraj Patil, Pedro Cuenca, Khalid Saifullah, Tanishq Abraham, Phuc Le Khac, Luke Melas, and Ritobrata Ghosh. Dall-e mini, 2021. URL. https::/g/t thub. com/
Aborisdaypn doddalllemnn lle-mini]
Jacob Devlin, Ming- Wei Chang. Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.0480) 2018.
Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF conference computer vision and pattern recogni- fion, pp. 12873-12883 2021.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, 63(1 1):19-1114, 2020.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Tobias Hinz, Stefan Heinrich, and Stefan Wermter. Semantic object accuracy for generative text-to- image synthesis. IEEE transactions on pattern analysis and machine intelligence, 2020.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings the IEEE/CVF conference on computer vision and pattern recognition, pp. 4401-4410, 2019.
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyz- ing and improving the image quality of stylegan. In Proceedings of the IEFF/CVF conference on computer vision and pattern recognition, pp. 1110-8119 2020.
Tero Karras, Mika Aittala, Samuli Laine, Erik Härkönen, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Alias-free generative adversarial networks. Advances in Neural Information Process- ing Systems, 34:852-863, 2021.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. Bart: Denoising scquenee-to-sequencc pre- training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.13461. 2019.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Nan Liu, Shuang Li, Yilun Du, Antonio Torralba, and Joshua B Tenenbaum. Compositional visual generation with composable diffusion models. arXiv preprint arXiv:20060/7114, 2022.
Don C Mitchell. Sentence parsing. Handbook of psycholinguistic, pp. 375-409, 1994.
10

---

Under review as a conference paper at ICLR 2023
John C. Mitchell. Concepts in programming languages. Cambridge University Press, 2002.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pp. 8748-8763. PMLR, 2021.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image image generation. In International Conference on Machine Learning, pp. 8821-8831. PMLR, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.061125, 2022.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image symthesis with latent difffusion models. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pp. 10684- 10645, 2022.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kam- yar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, et al. Photorealistic text-to-image diffusion models with deep language understanding. arXiv preprint 114i: 2022.
Tim Salimans, Ian Goodfellow, Wojeiech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. Advances in neural information processing systems, 29, 2016.
Christoph Schuhmann, Romain Beaumont, Cade W Gordon, Ross Wightman, Theo Coombes, Aarush Katta, Clayton Mullis, Patrick Schramowski, Srivatsa R Kundurthy, Katherine Crowson, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. 2022.
Sebastian Schuster, Ranjay Krishna, Angel Chang, Li Fei-Fei, and Christopher D Manning. Gen- erating semantically precise scene graphs from textual descriptions for improved image retrieval. In Proceedings of the fourth workshop on vision and language, pp. 70-80, 2015.
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Anmual Meeting of the Association for Computational Linguistics (Volume Long Papers), pp. 2556-2565, 2018.
Harald Sondergaard and Peter Sestoft. Referential transparency, definiteness and unfoldability. Acta Informatica, 5055-5755-1 1990.
Ming Tao, Hao Tang, Songsong Wu, Nieu Sebe, Xiao-Yuar Jing, Fei Wu, and Bingkun Bao. Di- gan: Deep fusion generative adversarial networks for text-to-image synthesis. arXiv preprint arXiv:2008. 05865, 2020.
Lucas Theis, Aäron den Oord, and Matthias Bethge. A note the evaluation of generative models. arXiv preprint a /5:::/5/1.0444 2015.
Bart Thomee, David Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li. Yfcc100m: The new data in multimedia research. Communications of the ACM, 59(2):64-73,200 2016.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informa- tion processing systems, 30, 2017.
Weihao Xia, Yulun Zhang, Yujiu Yang, Jing-Hao Xue, Bolei Zhou, and Ming-Hsuan Yang. Gan inversion: A survey, IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
Tao Xu, Pengchuan Zhang, Qiuyuan Huang, Han Zhang, Zhe Gan, Xiaolei Huang, and Xiaodong He. Attngan: Fine-grained text to image generation with attentional generative adversarial net- works. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1316-1324, 2018.
11

---

Under review as a conference paper at ICLR 2023
Licheng Yu, Patrick Poirson, Shan Yang. Alexander C Berg. and Tamara L Berg. Modeling context in referring expressions. In European Conference on Computer Vision, pp. 69 85. Springer, 2016.
Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaogang Wang, Xiaolei Huang, and Dim- itris N Metaxas. Stackgan: Text to photo-realistic image synthesis with stacked generative ad- versarial networks. In Proceedings of the IEEE international conference on computer vision, pp. 5907-5915, 2017.
12


Reviews:
[{"review_id": "YPV_JSYwEt", "reviewer": "Reviewer_jHfC", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a new metric to evaluate the image quality for multi-modal image synthesis task,  which relies on a parser to find different semantic concepts and a detector to find the corresponding objects and attributes in the image. Besides, a robustness testing technique is proposed to evaluate the robustness of a generative model.", "summary_of_the_review": "Please see above weaknesses, and I am happy to change my rating based on authors' responses.", "strength_and_weaknesses": "Strength:\n\n1. The paper is well written and easy to follow.\n\n2. Authors further consider the fine-grained semantic consistency between the given text and synthetic image to completely evaluate the correlation between them.\n\nWeaknesses:\n1. Basically, the proposed metric mainly focuses on measuring the semantic consistency between the given text and the generated image, which is different from IS and FID. So some statements shown in introduction, e.g., \"These metrics work well for the generation from simple prompts, e.g., description of a single object. However, for prompts with multiple objects and additional context information, simply adopting these metrics is insufficient and may lead to inaccurate or inconsistent results. \" might not be completely accurate. Also, the R-precision and CLIP score can also be adopted to measure the correlation between multiple object by adopting a similar detector used in the propose method.\n\n2. Although the proposed MIS-Score further considers the fine-grained correlation between the text and image, it mainly depends on a parser to find semantic concepts and a detector, so its performance might be affected by both components. \n\n3. For equation 2, could authors give more details to explain why it can be used to measure the correlation?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "Xx3aOGDZdq", "reviewer": "Reviewer_nMLm", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The author proposes a new metric for evaluating the multi-mode image synthesis method, and has tested new metrics. However, method details are limited. In addition, the author's contribution point 1,3 is difficult to capture in the introduction. Besides, the papers investigated are old.", "summary_of_the_review": "The research topic is worth recommending, but the author's description of the method needs to be strengthened.", "strength_and_weaknesses": "+The author proposes a new metric for evaluating multi-mode image synthesis methods\n+The author used a large number of experiments to verify\n\n-Recently, there are many meaningful works for multi-mode image synthesis. I don't know why the author has not conducted more detailed research in the Introduction section. Instead, follow the work of 2020 and 2021.\n\nDalmaz O, Yurt M, Çukur T. ResViT: residual vision transformers for multimodal medical image synthesis[J]. IEEE Transactions on Medical Imaging, 2022, 41(10): 2598-2614.\nIsaac-Medina B K S, Bhowmik N, Willcocks C G, et al. Cross-Modal Image Synthesis Within Dual-Energy X-Ray Security Imagery[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022: 333-341.\nWu G, Chen X, Shi Z, et al. Convolutional neural network with coarse-to-fine resolution fusion and residual learning structures for cross-modality image synthesis[J]. Biomedical Signal Processing and Control, 2022, 71: 103199.\nLv Z, Li X, Niu Z, et al. Semantic-shape Adaptive Feature Modulation for Semantic Image Synthesis[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022: 11214-11223.\nTan Z, Chu Q, Chai M, et al. Semantic Probability Distribution Modeling for Diverse Semantic Image Synthesis[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.\n\n-In denying that Measure is an important metric, as Section 7 of [1], I hope that the author can further compare the advantages of the metric and have more proof of the metrics.\n\n[1] Zhou R, Jiang C, Xu Q. A survey on generative adversarial network-based text-to-image synthesis[J]. Neurocomputing, 2021, 451: 316-336.\n\n-In the introduce section, the author introduces MIS-Score, but it is difficult capture the first contribution point and the third contribution point by reading this section.\n\n-The introduction of the method is limited, and it is difficult to capture the details of the method. The author's description of the implementation framework is incomplete and it is difficult to understand the proposed framework.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "VvBTpTr1_S", "reviewer": "Reviewer_1VeC", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "1, Present a new method to evaluate the cross-modal consistency by inspecting the decomposed semantic concepts.\n2, Introduce a new metric, called MIS-Score, which is designed to measure the fine-grained semantic alignment between a prompt and its generation quantitatively.\n3, Developed an automated robustness testing technique with referential transforms to test and measure the robustness of multi-modal synthesis models.", "summary_of_the_review": "A new metric is designed for measuring the fine-grained semantic consistency for the multi-modal image synthesis tasks. The measurement for multi-modal image synthesis tasks are important.", "strength_and_weaknesses": "Strength:\n1, The first time to propose a new approach for measuring the fine-grained semantic consistency for the multi-modal image synthesis tasks.\n2, A robustness testing technique is designed and implemented to evaluate the robustness of text-to-image generation models.\n3, Comparisions between different methods are interesting.\nWeaknesses:\n1, How to calculate weight wi in eq3?\n2, Can you calculate your metric directly on the benchmark dataset to check the annotation quality of the caption?\n3, Why Stable Diffusion is not the best model here?  It seems that they use more data.\n4, Do you still need a pretrained CLIP to calculate your score?\n5, How can you locate the object region?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject