LLMScore: Unveiling the Power of Large Language Models in Text-to-Image Synthesis Evaluation
Authors: Yujie Lu
Xianjun Yang
Xiujun Li
Xin Eric Wang
William Yang Wang
Venue: NeurIPS 2023 poster (2023)
Keywords: Text-to-Image Evaluation
Visio-linguistic Compositionality
Large Language Models
TL;DR: We propose LLMScore, a new framework that can offer interpretable evaluation scores to measure how well the generated images are aligned with text prompts for text-to-image evaluation.

Abstract: Existing automatic evaluation on text-to-image synthesis can only provide an image-text matching score, without considering the object-level compositionality, which results in poor correlation with human judgments. In this work, we propose LLMScore, a new framework that offers evaluation scores with multi-granularity compositionality. LLMScore leverages the large language models (LLMs) to evaluate text-to-image models. Initially, it transforms the image into image-level and object-level visual descriptions. Then an evaluation instruction is fed into the LLMs to measure the alignment between the synthesized image and the text, ultimately generating a score accompanied by a rationale. 
Our substantial analysis reveals the highest correlation of LLMScore with human judgments on a wide range of datasets (Attribute Binding Contrast, Concept Conjunction, MSCOCO, DrawBench, PaintSkills). Notably, our LLMScore achieves Kendall's tau correlation with human evaluations that is 58.8% and 31.2% higher than the commonly-used text-image matching metrics CLIP and BLIP, respectively.

LLMScore: Unveiling the Power of Large Language Models in Text-to-lmage Synthesis Evaluation
Yujie Lu Xianjun Yang', Xiujun Li", Xin Eric Wang', William Yang Wang'
University of California, Santa Barbara
University of Washington University of California, Santa Cruz httpps///gghhh..../////eeeee//// com./YujiiLaul1/1LMScrrr
Abstract
Existing automatic evaluation on text-to-image synthesis can only provide an image-text matching score, without considering the object-level compositionality, which results in poor correlation with human judgments. this work, we propose LLMScore, new framework that offers evaluation scores with multi-granularity compositionality. I.LMScore leverages the large language models (LLMMs) evaluate text-to-image models. Initially, it transforms the image into image-level and object-level visual descriptions. Then an evaluation instruction is fed into the LLMs to measure the alignment between the synthesized image and the text, ultimately generating a score accompanied by a rationale. Our substantial analysis wide
reveals the highest correlation of LLMScore with human judgments range of datasets (Attribute Binding Contrast, Concept Conjunction, MSCOCO, LLMScore achieves Kendall's correlation
DrawBench, PaintSkills). Notably, with human evaluations that is 58.8% and 31.2% higher than the commonly-used text-image matching metrics CLIP and BIIP, respectively.
Introduction
In recent years, research in text-to-image synthesis has made significant progress [9, 11, 38, 43]. However, evaluation metries have lagged behind due to challenges such as accurately capturing composite text-image alignment (e.g. color, counting, location) [47], interpretably producing the score, and adaptively evaluating with various objectives.
Established evaluation metrics for text-to-image synthesis like CLIPScore [17] and BLIP [24]. while widely used and highly effective [20, 36], have encountered challenges when it comes to capturing object-level alignment between text and image [12, 26]. Figure illustrates
example from the Concept Conjunction dataset [12), given the text prompt "A red book and a yellow vase", the left image aligns with the text prompt, while the right image fails generate red book, and the correct color for the vase, also with an extra yellow flower. Human judges can make the correct and clear assessment (1.00 v.s. 0.45/0.55) of these two images on both overall and error counting objectives, while the existing metrics (CLIP, NegCLIP J. BLIP) predicts similar scores for both images, failing distinguish the correct image (on the left) from the wrong one (on the right). Furthermore, these metrics provide single, non non- -on-interpretable score, obscuring the underlying reasoning behind the alignment of the synthesized images with the given text prompts. Additionally, these model-basee metrics are static, unable to follow varied guidelines that prioritize different objectives of the sot-to-image evaluation. For example, the evaluation can range from accessing image-level semantics (Overall) to finer object-level details (Error Counting). These issues hinder the existing metries from aligning with human evaluations.
"Corresponding Author: yujielu10@ gmail..com
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Text Prompt: A red book and yellow vase.
Left
Right
LLMScore Overall Rationule
Human
L.00
image quite low due to the sisnificant discrevanciis botweeon the chiects described in the text
The overall quality
Ervor Counting
1.00
0.55
Bareline
CLIP
0.27
0.31
promps and those portrayed in the image
NegC1 IP
0.26
0
0.32
LLMSoore Error Counting Rationale
BLIP-ITM
1.00
The book from the proenpt not
BLIP-ITC
0.48
0.49
image. The vase is deseribed as red in the
image, while the prompt specified
LLMSoore Ovemull
1.00
0.50
vellow vase. Over bier-spciiiication of the
Ecror Counting
1,00
0.44
yyllow flowers in the image.
Figure 1: The two images generated using from the Concept Conjunction dataset. Baseline section shows the scores from the existing model- based evaluation metrics, Human section is the rating score from the human evaluation, LLMSSorr section is our proposed metric. The right column also shows the rationale generated by LLMScore.
based on the text prompt sampled
In this paper, we introduce LLMScore, novel framework to evaluate text-image alignment in text- to-image synthesis by unveiling the powerful reasoning abilities of large language models (LLMs). Our inspiration stems from the human approach of measuring text-image alignment, which involves checking the correctness of objects and attributes specified in the text prompt. With the incredible text reasoning and generation ability LLMs, LLMScore can imitate the human evaluation process to access compositionality at multi-granularity and produce alignment scores with rationales, delivering more insight into the model performance and reasons behind the scores.
To enhance the evaluation of composite eox-tto-imaee synthesis, our LLMSccre elicits grounding visio-linguise information from vision and language models and LLMs, thereby capturing multi- granularity compositionality in the text and image. Specifically, our approach leverages vision and language models transform the image into multi-granularity (image-level and object-level) visual descriptions, which allows us to capture the compositional aspects of multiple objects in the language format. Then concatenate these descriptions with text prompts and feed them into large language models (LLMs, for example, GPT-4 [32]) to reason the alignment between text prompts and images. While existing metries struggle capture compositionality. our LLMScore captures the object-level alignment between text and image, producing scores that are significantly correlated with human evaluation, complete with reasonable rationales (Figure 1). In addition, our LLMScore can adaptively follow various guidelines (overall or error counting) by simply customizing the evaluation instruction for LLMs. For example, we can access the overall objective by prompting the LLMs with the instruction 'Rate the overall alignment of text prompt and image." " or validate error counting objective with the instruction "How many compositional errors are in the image?". And we explicitly include guidance on error types of text-to-image models in the evaluation instruction to keep the I.I.M's 's decision deterministic. This flexibility empowers our framework as a versatile tool for wide range of text-to-image tasks and various evaluation guidelines.
We validate the effectiveness of LLMScore through extensive experiments, demonstrating its align- ment with human judgments without any demand for additional training. Our experimental setup contains state-of-the-arr -o-aate- models such as Stable Diffusion [38] and DALL-E [37]. evaluated over diverse datasets, including prompt datasets for general purpose (MSCOCO [28], Draw- Bench [42], PaintSkills [8]) and for compositional purposes (Abstract Concept Conjunction [12], Attribute Binding Contrast [12]). Our L..MScore achieve the highest human correlation across all datasets. On compositional datasets, we achieve 58.8% and 31 27% higher Kendall's over widely used metrics CLIP and BLIP respectively.
To sum up. we present LLMScore, the first attempt unveil the power of the large language models for text-to-image evaluation, in particular, our paper makes the following contributions:
We propose LLMScore, a new framework to evaluate the alignment between text prompts and synthesized images in text-to-image synthesis, offering scores that accurately capture multi-granularity compositionality (image-level and object-level).
Our L.MSScore produces accurate alignment scores with rationales following various evalu- ation instructions (overall and error counting).

---

red car and white sheep. Test-lwage Instuction-Followii Matching
Prompt
Image
Text Prompt
Evaluation
Visual Description
Overal
Counting
CLIPIBLIP
Score
Sentence Matoking
Instuction evaluation
erter counting
Text
LLMSccre (Overall)
LLM
LLMSecre (Errr Counting)
Prompt
Caption
Rationale: Generally mutches..
Rational:: The image refers a herd of shepp. instead of single white sheep..
METEOR
Score
Disrrpuncy yuantty shogp
Figure 2: Comparison of maxt-Image Matching, Sentence Matching, and our LLMM-bssed Instruction- Following Matching pipeline for text-to-image synthesis evaluation. Our LLMScore automatically provides accurate scores and reasonable rationales for text-to-imape synthesis based on text prompts, and visual descriptions following various evaluation instructions.
We validate the LLMScore on a wide range of datasets (both general purpose and composi- tional purpose). Our proposed LLMScore achieves the highest human correlation among the commonly used metrics (CLIP, BLIP).
Background
Existing Automatic Metrics for Text-to-Image Synthesis Though Inception Score (IS) [45] and the Frechet Inception Distance (FID) [18] are recognized metrics for image fidelity, they do not measure how well the synthesized images align with the text prompts. Existing metrics (21, 22, 34, 44] for measuring such alignment text-to-image synthesis generally fall into two categories: 1) text- image matching, and 2) sentence matching. For direct comparison between the text and image, the metrics typically rely on the pre-trained text-image matching models, among these, CLIP-based and BLIP-based are most common. Alternatively, the centence-matching pipeline transforms synthesized images into captions and then measures the sentence-level similarity with the text prompts. This can be achieved by transforming the synthesized images into free-form captions using the image captioning model BLIPv2 [25]. Then we can apply reference based image caption metrics such as text-based CLIPScore [17] and METEOR [3] to measure the alignment.
Large Language Models as Evaluation Metries 33] have achieved incredible performance in evaluating natural language generation tasks [14, 29, 55]. This success stems from the LLMs' powerful reasoning and instruction-followiin evaluation of diverse objectives simply by altering the prompt. In addition to language-only tasks, recent studies have demonstrated the effectiveness of eliciting vision and language reasoning abilities of LLMs by incorporating image descriptions [13, 30, 40, 49] fusing multimodal features [31, 56]. concurrent work TIFA [19] utilizes LLMs to generate questions for validating text-to-image faithfulness. To our best knowledge, we the first introduce object-centric descriptions of images into LLMs for evaluating the multi-granularity compositionality in text-to-image synthesis.
Very recently, large language models (LLMs) [32,
enabling the
LLMScore
Our proposed LLMScore aims to evaluate the alignment between the generated image and the text prompt while capturing multi-granie multi-granularity compositionality. As depicted in Figure 3, LLMScore has two main components: 1) LLMS As Multi-Granularity Visual Descriptor: transforming the image into multi-gn coltti-rrauularit object-centric descriptions (Section 3.1), and 2) LLMs As Text-to-Imann Image Evaluator: feeding the evaluation instructions into LLMs, generating the score and rationale (Section 3.2).
3.1 LLMs As Multi-Granularity Visual Descriptor
As illustrated in Figure 3, we first decompose the image into two-level visual descriptions: image- level global description; and region-level local descriptions. Then we employ the I.LMs to fuse them into a coherent, object-centric, multi nulti-granullriiy description of the image.

---

Instance Detection
Genarated Image
Descriptio:
Object 1: white sheep eeting grass: (394, 482, 624, 689) Object while car parked: (590. 272, 767, 485) Object sheep grezing
Extracted Regions
[38, 426, 647, 708)
Global Description:
Image Caption Model
Two sheep slanding on the road.
Region Region 2
Region
Description
Object-Centric Description:
Reglon-to-Tel Model
LLM
Description
Visual Descriptor
landscape
green hillaide,
sheep are peacefully standing on the road. The sky stretches the meating the at the horizon A white car can be seen parked on the backórop.
LLMScore
Evaluation
Visual Description
grass
Instructions
Text Prompt
Figure LLMScore pipeline for Text-to-Image evaluation. This image is generated by Stable- Diffusion-2 using the text prompt "A red car and white sheep.", which is sampled from the Concept Conjunction dataset. Il and 12 represent two different evaluation instruction settings: Overall and 2) Error Counting. Each produced LLMScore is accompanied by rationale.
Global Image Description The global image description is composed of image-level captions and image meta-information. Given an image, for example, in Figure 3, we first transform it into global description using the state-of-the-ar image captioning model BLIPv2, which encapsulates the primary context of the image in a single sentence. Then we concatenate it with the meta-ini veta-information of the image, such as the resolution (i.e. 512x512) which aids in the understanding of the location of each object, and interaction among objects.
Local Region Descriptions Global description can only offer high-level information for the image, detailed descriptions for each region are necessary to capture object-level local information. Here we utilize GRiT [50] to extract regions of interest and transform them into textual descriptions of the regions. GRİT is a model pre-trained with detection and dense caption objectives jointly on the Visual Genome dataset, which contains local fine-grained descriptions for objects in the image. Specifically, we format the description of each object as " "[Object] [Dense Caption] [Bounding box]". As shown in Figure 3, the descriptions of all the objects are concatenated together as our local region descriptions for each image.
Object-Centee Centric Visual Description Though local region descriptions capture dense information about objects in the image, they lose global context compared with global image descriptions, which may lack accurate interpretation of spatial and interaction relationships among objects. By incorporating both the global image description and the local region descriptions, obtain object-centric visual descriptions that capture multi-granuulaitty compositionality, such as the attributes of the objects and relationships among the objects. Specifically, we feed the local and global descriptions into LLMs (GPT-4 [32] by default) with the template [GLOBAL DESCRIPTION] [LOCAL DESCRIPTION] DESCRIPTION INSTRUCTION" We fill the slot [GLOBAL DESCRIPTION] with the global image description and the slot [LOCAL DESCRIPTIONT with the local region description. And the hand-crafted slot DESCRIPTION INSTRUCTION ("generate object-centric description" in Figure 3) is replaced with "Based on the above information of the image, generate the object-centric visual description regarding the numerical counting, shape, color, size, location, materials of the object and the spatial and interaction relationships among the objects."
able to
3.2 LLMs As Text-to-Image Evaluator
Large language models have demonstrated strong reasoning abilities (mathematical, coding etc.) on many complex tasks [6]. Here we employ the L.LMs to measure the alignments between the generated image and text, and utilize their reasoning ability to understand the compositional attributes of objects and the complex interactions among multiple objects. Given the above-generated visual descriptions (Section 3.1), the whole evaluation process contains three steps: 1) instruction-folloo followiig rating, 2) rule-enhanced rating, and 3) rationale generation.

---

Evallaation Objectives
Evaluation Instructions
Visual Description
12
Prompt
Interpretable Multi Multi-granularity Text-to-Image Evaluation Metric
Objective (Emor Counding)
Oijjetivve [Dverall)
Concelerede
b
What stepa
evakutte the overall
Text Prompt:
LLMSCore (Overall):
alignment belaeen the lext and the imace.
A red car while sheap.
Instruction
Rationale: general, the image not correctly depict the text prompt
Description:
Following Evaluation
LLMSScre (Error Counting): 0.77
LLMIUser
F Formulation
Instruction
green hillside, two sheep are pescafully standing
landscape
Rationale:
There are two discrepancles and the mentons image
belween lext
Instruction
on the road...
X LLM
caption. text
Instruction 1/12)
H1: Rate the overall quality of the image In torms matching the text prompt. Provide the number of composition text prompt.
white sheep, eaption describes a scone with white car and two shrep. The errors include the incorrect car color (white instead of red) and the presence of an additional sheep.
Instruction 1 (ff)
Rate the overal quality of the image in terms of matchhing the text prompt.
image compared the
Figure 4: Large Language Models As Text-to- text-to-Image Evaluator. By defining evaluation instructions for various evaluation objectives, large language models can follow the instructions to evaluate text-image alignment based on the aforementioned visual descriptions and text prompts.
EntrruttoonFoolowwiin Rating As shown in Figure 4, we design evaluation instructions to guide the L.LM in evaluating the image based on specific criteria, such as overall semantics, error counting, etc. The instructions are explicit and detailed, and can include specific guidance on error types (i.e. counting/hape/ccolo/siin) commonly seen in text-to-image models. This ensures the LLM's evaluation remains deterministic and is not influenced by any inherent biases in the pre-training data. The visual descriptions and the evaluation instructions are fed into the LLM to generate score. For example, for Overall quality evaluation, the hand-crafted slot [EVALUATION INSTRUCTION] (denoted as Instruction I1/12 in Figure 4) is replaced with "Rate the overall quality of the image in terms of matching the text prompt. Similarly, for Error Counting evaluation, the hand-crafted slot [EVALUATION INSTRUCTION] is replaced with "Provide the number of compositi tinalliona errors in the image compared to the text prompt." ". We define the error type as the object-level difference. For example, the counting/happc/cllrrsiiz difference in the image and text prompt should be counted as error. This approach can either speed up the process of human annotators or serve as an interpretable and consistent evaluation pipeline.
Rule-Enhane Enhaneed Rating While directly using L.LMs to produce evaluation scores has its merits, presents certain challenges. One notable issue for LLM is that it can only generate discrete outputs, making it challenging to produce decimal scores even when the text prompt explicitly specifies that. To mitigate this issue, where N is default set as 100) in integer and downscale to a smaller range of scale (divided by N by default) to enforce decimal. This approach provides more flexibility to capture small discrepancies compared to directly prompting with the decimal score.
restrict the LLMs to rate over wider range of scale (on a scale of I-N,
To further enhance the consistency of LLMs' ratings, we propose breaking down the evaluation process into deterministic atomic tasks and employing basic heuristic rules to imitate the human eval- uation process more accurately and consistently. In the first step, the I.MMs receive the concatenated information (text prompt, image description, and evaluation instruction) and produce predictions for pre-defined sequence of atomic tasks. The atomic tasks include obtaining: 1) the number of objects specified in the text prompt (X1), 2) the number of matched objects in the image (X2), 3) the number of specified attributes in the text prompt (YI), and 4) the number of matched attributes in the text prompt (Y2). The results for these atomic tasks are more deterministic compared with high-level tasks. In the second step, use basic heuristies rules inspired by human evaluation processes, which reason how well the objects specified in the text prompt are presented in the image descriptions and depicted in the image. Thus combine the results from atomic
how correctly the attributes tasks and derive the final evaluation score by (X2/X0 11/11)/2+ + (Y2/Y1)/2. This is inspired by recent work [7, 15] that separates the calculation in LLMS to achieve more reliable results. This approach allows us to overcome the limitations of decimal value generation and consistency in L.MSs. Thus we obtain evaluation results that are more accurate, interpretable, and closer to human judgment.
The score generation process involves the LLMs' understanding of various evaluation instructions, such as assessing the overall text-image alignment or precisely counting the number of errors in the image. Then the LLMs apply the understanding of the evaluation instruction
Generating Rationale
5

---

to the concatenated visual description and text prompt, generating score that reflects the alignment between the image and the text multi- granlliggt compositionality (image-level and object-level), Each score is accompanied by rationale. To be specific, we add the explanation instruction prompt for LLMss such as "Explain the overall rating X within one paragraph." for the overall objective and "Explain the error counting within one paragraph" for the error counting objective. As shown in Figure 4, our LLMScore generate reasonable rationales for both overall and error counting objectives. The rationales for the scores further provide insights into the LLM's decision-making process.
4 Experiments
4.1 Experiments Setup
Datasets For general purpose evaluation (GeneralBench), we sample 200 text prompts from each dataset, including MSCOCO [27] 2014 and 2017, DrawBench [42], and PaintSkills [8]. For compo- sitional purpose evaluation (CompBench), we sample 200 text prompts from Concept Conjunction (CC) and Attribute Binding Contrast (ABC) datasets that designed for evaluating compositional text-to-image synthesis [12]. In total, we gather 1200 text prompts for human correlation experiments.
Text-to-Image Models For each sampled text prompt, we generate two images using two widely used text-to-image models. Stable Diffusion [38] and DALL-B[37) which demonstrate extraordinary generation quality. To be specific, we use Stable Diffusion 2. 1-v from Hugging Face, and DALL-E using OpenAl API in April 2023. All the images are generated at resolution of 512x512. The total text-image pairs prompts used in human correlation experiments are 2400.
Baseline Metrics We consider these publicly available model-based evaluation metrics, which fall into the text-image matching pipeline (depicted Figure 2) evaluating text-to-image synthesis.
1. CLIP [17, 36] measures the cosine similarity of image and text prompt representations extracted from the CLIP feature extractors. This is widely used model-based metric to measure the text-image alignment in text-to-image synthesis.
2. NegCLIP [51] uses fine-tuned CLIP with improved compositionality understanding to measure the cosine similarity of the image and the text prompt.
3. BLIP-ITC [24, 25] uses a cosine similarity function over the extracted image and text features by BLIPv2, similarly as CLIP.
4. BLIP-ITM [24, 25] uses cross-attention fuse multimodal features extracted by BLIPv2 to compute fine-grained similarity.
We discuss the sentence-matching pipeline as our ablations in Section 4.3. Notice that we focus on evaluating how well the synthesized images are aligned with the text prompts. Thus the widely used Inception Score (IS) and the Frechet Inception Distance (FID) for evaluating image quality are not considered in our baselines, since they do not compare the image with the text prompts.
Implementation Details We extract the global image deseription using the pre-trained 2.7B BLIPv2 [24] model equipped with large language model OPT [54]. We extract local region description using the dense caption model GRiT [50] with ViT-Base [10] pre-trained on COCO 2017. We obtain the object-centric visual description using GPT-4 [32] default language models to combine the global and local information. The is by default generated by GPT-4 with The GPT-4 model is the default descriptor to combine global and local descriptions into object-centric visual descriptions and the default evaluator to produce the score with rationale. All experiments conducted with GPT models are using OpenAl API from April 2023 to May 2023 with temperature 0.7 using greedy decoding by default.
4.2 Human Ratings
For each generated text-image pair, we ask 2 human annotators to provide ratings over these synthe- sized images in terms of Overall quality and Error Counting:

---

Table Composition focuseed Prompt Bench. The correlation between automatic evaluation metrics and human rankings on rext-to-image synthesis. LLMScore significantly surpasses existing metrics terms of Kendall's r and Spearmanr's p with p < 0.001.
Concept Conjunction
Attribute Binding Contrast
Human
Metric
Stable Diffusion
DALLE
Stable Diffusion
DALLE
7(1)
(())
<(1)
(())
=(1)
p(t)
r(*)
(()
CLIP
0.1698
0.2459
-0.0049
-0.0058
0.0186
0.0320
0.0396
0.0548
NegCIIP
0.1724
0.2504
0.0682
0.0995
0.0151
0.0211
0.1145
0.1634
Overall
BLIP-ITM
0.4058
0.5618
0.3768
0.5266
0.1799
0.2559
0.1500
0.2134
BLIP-ITC
0.2378
0.3398
0.0991
0.1413
0.1982
0.2814
0.0252
0.0344
LLMScore
0.4871
0.6956
0.5167
0.7230
0.4005
0,5480
0.3955
0.5506
CLIP
0.2012
0.2864
-0.0782
-0.1107
0.0061
0.0071
0.0914
0.1286
NegCLIP
0.22455
0.3240
0.0353
0,0502
0.0339
0.0418
0.0796
0.1130
Error Counting
BLIP-ITM
0.3341
0.4561
0,1105
0.1668
0.0696
0.0968
0.1249
0.1783
BLIP-ITC
0.2210
0.3124
0.0775
-0.1071
0.0895
0.1315
0.0533
0.0786
LLMScore
0.3779
0.5443
0,2880
0.4428
0.1863
0,2821
0.2326
0.3351
Table 2: The correlation between automatic evaluation metrics and human rankings on text-to-image
synthesis. LLMScore significantly surpass existing metrics in terms of Kendall's r and Spearmanr's P with p < 0.001.
COC02014
COCO2017
DrawBench
PaintSkills
Human
Metric
r(t)
(()
r(t)
x()
r(1)
p(t)
r(t)
p(t)
CLIP
0.1971
0.2655
0.2227
0.2771
0.1530
0.2143
0.4715
0.5869
Ne
0.2164
0.2905
0.2793
0.3523
0.1463
0.1999
0.4911
0.6313
Overall
BLIP-ITM
0.3252
0.4255
0.0928
0.1155
0.1044
0.1455
0.4755
0.6214
BLIP-ITC
0.3465
0.4535
0.1703
0.2121
0.1569
0.2171
0.4743
0.5864
LLMScore
0.3629
0.4612
0.3357
0.4275
0.2230
0.3023
0,5600
0.6853
CLIP
0.1464
0.2142
0.1888
0,2677
0,1360
0,1910
0.3052
0,2891
NegCLIP
0.2116
0.3061
0.1795
0,2581
0.1179
0,1596
0.4563
0.4908
Error Counting
BLIP-ITM
0.2251
0.3289
0.1137
0.1635
0.0871
0.1189
0.4622
0.4997
BLIP-ITC
0.2636
0.3739
0.1849
0.2620
0.1506
0,2029
0.6178
0.6511
M.
0.2830
0.3992
0.2038
0.3027
0.2134
0,2865
0.6437
0.7325
Overall: gencral-purpose text-to-image evaluation, which applies to most existing metrics. Human annotators are required to rate the overall quality of the synthesized images in terms of matching the Text Prompt.
Error Counting: Human annotators are required to provide the number of compositional errors in the synthesized images compared to the text prompt. The error types include: 1) compositional errors: wrong attributes (color, spatial position, shape, size, material) of the objects and wrong relationship among objects, mentioned in the text prompt are not present in the image, and 3) over-specification errors: specified in the text prompt.
missing object errors: the objects
the image hallucinates irrelevant objects in the image that
The averaged inter-rater agreement is 0.62 under Krippendorff's alpha agreement measure. We provide clear guidance for human annotators, with details shown in Appendix C.
4.3 Human Correlation
In Table I and Table 2. we use Kendall's tau (T) and Spearman's 's rho (p) to measure the ranking correlation to both Overall and Error Counting human rating for compositional bench and general bench. All the model-based metric scores and human ratings are normalized to 0-1 for comparison.
Overall Results concluded in Table & 2 and Figure 5, 1). the most popular text-image alignment metric (CLIP) for text-to-image evaluations is less correlated with human ratings than
7

---

0.40
0.35
0.4
BUP-TM
0.175
0.33
OUP
0.150
0.5
027
0.22
0.25
025
0.100
01
0072
0.07
0.15
0.30
0.06
0.23
0.55
CompBeach Overall
-vveral
CompBench Error Counting
GeneraBench Overall
GeneraiBench Emor Counting
(a) Averaged Kendall's 7 Ranking Correlation with Human Ratings.
0.63
0.28
0.5
0.48
NegCUF
BUP-TTM
0.20
0.25
0.22
0.4
0.57
0.39
0.14
0.32
0.39
0.15
0.24
02
0.10
0.08
0.04
6.3
0:00
0:13
0.65
0.2
0.00
0.0
Comppencc Overall
CompBench- Counting
GeneralBench
GeneralBench Error Counting
(b) Averaged Spearmanr's Ranking Correlation Human Ratings.
Figure 5: The rank correlation is aggregated across the compositional prompt dataset (Concept Conjunction, Attribute Binding Contrast) on the left two columns (CompBench) and the general prompt dataset (MSCOCO, DrawBench, PaintSkills) on the right two columns (GeneralBench).
expected. NegCLIP utilizes hard negatives to improve the compositionality of CLIP; 2). BLIP-based metrics (BLIP-ITM and BLIP-ITC) surpass these CLIP- based metrics (CLIP and NegCLIP). We suppose BLIP-based metrics benefits from the better object-level vision-language presentations learned by grounding tasks, compared with image-level matching learned CLIP. 3). LLMScore is significantly better than the existing metrics with large margins.
Accurate Error Counting in the Image is Challenging As shown in Figure 5, all the metrics achieve better correlation with Overall human ratings than Error Counting human ratings given the fact that there various error types and capture each of them requires more accurate composi- tional visio-ling linguisic understanding and generation.
Object-centric Visual Descriptions Improve Compositional Understanding In Figure 5, we show that our LLMSScrre achieves larger correlation gain over the text prompts the compositional bench (e.g., counting, position, size, color, relations). This further confirms our superiority in capturing compositionality with the introduced object-centric visual descriptions.
Image Captions Visual Deseriptions We
consider two categories of variants for LLM- Score:
0.4
Counting
1. Text-Caption Matching: CapCLIP and Cap- METEOR use the CLIP and METEOR to measure the similarity between the cap- tions (which is the global image description
0.3
Kendall's
0.2
0.0
in Section 3.1) the synthesized images and text prompts.
CapCLIP CapMETEOR DescCLIP DescMETEORLLMMCOM
Figure 6: Comparison between Sentence Match- ing (CLIP. METEOR) and CnrrructiooFFollowwiii Matching (L..MScore). I.I.MScore achieves best averaged Kendall's correlation with human ratings over the GeneralBench (MSCOCO, Draw- Becnh, and PaintSkills).
2. Text-Description Matching: DescCLIP and DescMeteor, which use the CLIPScore [17] and METEOR [3] to calculate the similar- ity score between visual descriptions (in
Section 3.1) and text prompts.

---

Table Effects of Large Language Models. LLMScore (Overall) can obtain performance gain from GPT-4 compared with using GPT-3.5 as the evaluator. Using GPT-4 as default visual descriptors do not improve the evaluation performance when only using image caption metrics (DescCLII DescMeteor). All numbers are averaged on the GeneralBench (MSCOCO, DrawBecnh, and PaintSkills).
CIIP
DescMeteor
L.L.Score
Human
LLM
r(t)
(())
r(t)
A()
7(1)
A(t
GPT-3.5
0.1479
0.1956
0.0042
0.0073
0.2480
0.3285
Overall
GPT-4
0.1128
0.1485
0.0297
0.0374
0.2793
0.3649
GPT-3.5
0.0467
0.0670
-0.0397
-0.0835
0.2205
0.3013
Error Counting
GPT-4
0.0149
0.0228
1087
-0.1494
0.2131
0.2981
red clock gold sultcase.
L. MScre (Overall)
0.40
Oveeral Rationale
Humon Overall
0.33
The image capticn deseribes two red suitceses, a vinitgee clock on the wall, and blue cartain in the backgroumd, but the test prompt only mentions red clock and gold suitesse. aligument between text prompt image caption weak.
Error Counting
0.70
Bazeline
CLIP
0.30
L.I. MScore (Fimor Counting) 0.55
NegCLIP
0.36
Counting Rationale
BLIP-ITM
0.99
The composition errons include incerrett suincase calons. incorreet clock colar. addirional elements mentioned text prompt blue curtain).
BLIP-ITC
This kitchen has wooden cabinets and silver black appliances
LLoe (Overall)
Overall Rationale
Human Overall
1.00
coprion accumtelyy desorines image aliens the text prompt effering additional details about vañous elements and creating clear picture of
Error Counting
0.00
0.24
LLMScore (Erter Counting) 0.00
NegCLIP
Ervor Counting Rationalc
BLIP-ITM
0.40
caption accurately describes the colors, materials, and spatial positions of all relevant objects within the setting,
compositio
BLIP-ITC
0.42
ailhering the luxi procept's focus wooden cabincts and black appliances.
Figure Examples showing the LLMScore captures the object-level discrepancies (TOP) and similarities (BOTTOM) between the image and the text prompt. The two text prompts are sampled from the Concept Conjunction dataset.
The main drawback of such a sentence-matching pipeline is that the caption metrics favor the coverage or similar language structure instead of capturing compositional semantics. Figure 6 shows the average correlation (Kendall's r) on the general bench set, LLMScore significantly outperforms these two variants, indicating that, caption metrics is not good means measure the alignment between image and text. Despite the advance introduced by object-centric visual descriptions, the limitations associated with using caption metrics to measure semantic similarity hinder the performance gain of text-description matching metrics DescCLIP and DescMETEOR over text-caption matching metrics CapCLIP and CapMETEOR.
Ditfering Large Language Models In Table 3, we compare two variants of large language models: GPT-3.5 and GPT-4. 1). For the same group (for example, GPT-4 based) visual descriptions, LLMScore is better than DescCLIP and DescMeteor, indicating that LLMs reasoning is the key component to capture the semantic relations in the visual descriptions. 2). We observe that GPT-4 based LLMScore (Overall) has an average better correlation than GPT-3.5, indicating the performance benefits from the better reasoning ability in the larger-scale language model. However, GPT-4 based I.LMScore (Error Counting) only comparable with GPT-3.5, indicating that the counting still un-resolved in large-scale language models.

---

4.4 Muaiu-Graullarity Rationale for Text-to-Image Evaluation
In Figure 7, we show two examples, our LLMScore not only produces human-correlated text-image scores but also provides a rationale for each metric value. The rationale correctly explains both the differences and similarities between the descriptions and text prompts. The model performs well in explaining the emrors counted and the overall comparisons.
Conclusion
In this paper, we re-examine the existing model-based text-to- to-image metrics and propose LLMScore to evaluate text-to-image synthesis by unveiling the power of large language models. Our LLMScore can capture the multi-gran guaniliganulrrity compositionality between the synthesized images and the text prompt, producing accurate alignment scores with rationales. Our LLMScore demonstrates significantly better correlation with human scores on several datasets, paving the way more adaptable text-to-image evaluation, capable of following human instructions to evaluate the text-image alignment.
Broader Impact
The framework proposed in this paper first integrates GPT-4 for text-to-image evaluation and show- cases how to take advantage of the existing large-scale pre-trained models (GPT-4) for measuring the alignment between the generated images and text, we also propose a new metric, LLMScore which provides interpretable rating and well aligns with the human scores several datasets. This work sheds light on the value of large language models on the evaluation of text-to-image synthesis, we hope it can help the future text-to-image synthesis work on improving the groundedness and compo- sitionality, either as reward signal or evaluation metrie; our preliminary work on the interpretability of I.I M.Sorr,,,, may have the potential to be used for explanation, controllable generation, and image editing.
Limitations
One limitation of our work is that it relies on GPT, which is not free for the public, and may limit its fast plug-in capability, future work may consider replacing this component with a publicly available LLM model (e.g., LLaMA) or our in-house finetuned image captioning model. Another potential issue for this work is, since it incorporates the exsiting large language models, it may inherit its own biases that could propagate to the metric. The future work who considers adopting our LLMScore metric should be cautious on the specific domains make sure no harmful biases get propagated.
Acknowledgments
The research was sponsored by the National Science Foundation under Grant No. 2048122. The views and conclusions contained in this document those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the sponsor.
References
[1] Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. Spice: Semantic propositional image caption evaluation. In Computer Vision-ECCV 2016; 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part 14, pages 382-398. Springer, 2016. 14
[2] Omri Avrahami, Ohad Fried, and Dani Lischinski. arXiv:2206.02779 2022, 14
Blended latent diffusion.
arXiv preprint
[3] Satanjeev Banerjee and Alon Lavie. METEOR: correlation with human judgments. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pages 65-72, Ann Arbor, Michigan, June 2005. Association for Computational Linguistics. 3,8
An automatic metric MT evaluation with improved
10

---

[4] Tim Brooks, Aleksander Holynski, and Alexei A Efros. Instructpix2pix: Learning to follow image editing instructions. arXiv preprint arXiv:2211.09800 2022. 14
[5] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learnrrs. Advances in neural information processing systems, 33:1877-1901, 2020. 14
[6] Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Eee Kamnr, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks of artificial general intelligence: Early experiments with gpt-4, arXiv preprint arXiv:2303./2712, 2023.
[7] Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W. Cohen. Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks, 2022. 5
[8] Jaemin Cho, Abhay Zala, and Mohit Bansal. Dall-eval: Probing the reasoning skills and social biases of text-to-image generative models, 2022. 2, 6, 15
[9] Prafulla Dhariwal and Alex Nichol. Diffusion models beat gans on image synthesis. ArXiv, ahs/2105.05233,, 2021. 1
[10] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenhorn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlshy. An image is worth 16x16 words: Transformers for image recognition at scale, 2021. 6
[11] Patrick Esser, Robin Rombach, and Bjöm Ommer. Taming transformers for high-resolution image synthesis, 2020.
[12] Weixi Feng. Xuehai He. Tsu-Jui Fu, Varun Jampani, Arjun Akula, Pradyumna Narayana, Sugato Basu, Xin Eric Wang, and William Yang Wang. Training-fres free structured diffusion guidance for compositional text-to-image synthesis, 2023. 1,2, 6, 15
[13] Weixi Feng. Wanrong Zhu. Tsu-jui Fu, Varun Jampani, Arjun Akula, Xuehai He, Sugato Basu, Xin Eric Wang. and William Yang Wang. Layoutgpt: Compositional visual planning and generation with large language models. arXiv preprint arXiv:2305. 15393, 2023. 3
[14] Jinlan Fu, Kiong Ng. Zhenghao Jiang, and Pengfei Liu. Gptscore: Evaluate as you desire, 2023. 3
[15] Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamic Callan, and Graham Neubig. Pal: Program-aided language models, 2023. 5
[16] Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yacl Pritch, and Daniel Cohen- Or. Prompt-to- prompt image editing with cross attention control. arXiv preprint arXiv: 0/626, 2022. 14
[17] Jack Hessel, Holtzman, Maxwell Forbes. Ronan Le Bras, and Yejin Choi. CLIPScore: A reference-free free evaluation metric for image captioning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7514-7528, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguisties. 1, 3, 6, 8, 14
[18] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017. 3
[19] Yushi Hu, Benlin Liu, Jungo Kasai, Yizhong Wang, Mari Ostendorf, Ranjay Krishna, and Noah A Smith. Tifa: Accurate and interpretable text-to-image faithfulness evaluation with question answering. arXiv preprint arXiv: 1897, 2023. 3
[20] Gwanghyun Kim, Taesung Kwon, and Jong Chul Ye. Diffusionclip: Text-guided diffusion models for robust image manipulation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2426-2435, June 2022.
[21] Jin-Hwa Kim, Yunji Kim, Jiyoung Lec, Kang Min Yoo, and Sang-Woo Woo Lec. Mutual information divergence: A unified metric for multimodal generative models. arXiv preprint or arXiv:2005.13444 2022. 3
[22] Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved precision and recall metric for assessing generative models. Advances in Neural Information Processing Systems, 32, 2019. 3
[23] Kimin Lee, Hao Liu. Moonkyung Ryu, Olivia Watkins, Yuqing Du Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, and Shixiang Shane Gu. Aligning eexttto-image models using human feedback, 2023. 14
11

---

[24] Junnan Li. Dongxu Li, Silvio Savarese, and Steven Hoi. 22ip-2: Bootstrapping language- -mnge wre-training with frozen image encoders and large language models, 2023. 1, 6, 14
[25] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. International Conference on Machine Learning, pages 12888-122 12900. PMLR, 2022. 3, 6, 14
[26] Zhiheng Li, Martin Renqiang Min, Kai Li, and Chenliang Xu. Stylet2i: Toward compositional and high-fidelity text-to-image synthesis. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2022. 1
[27] Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft coco: Common objects context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13. pages 740-755. Springer, 2014. 6, 14
[28] Tsung- Yi Lin, Michael Maire, Serge J. Belongie, James Hays. Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft coco: Common objects in context. In European Conference on Computer Vision, 2014. 2
[29] Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang. Ruochen Xu, and Chenguang Zhu. G-eval: Nig evaluation using gpt-4 with better human alignment, 2023.
[30] Yujie Lu, Pan Lu, Zhiyu Chen, Wanrong Zhu, Xin Eric Wang, and William Yang Wang. Multimodal procedural planning via dual text-image prompting, 2023. 3
[31] Yujie Lu, Wanrong Zhu, Xin Wang, Miguel Eckstein, and William Yang Wang. Imagination-aaugmentee natural language understanding. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computationll Linguistics: Human Language Technologies (NAACL.) pages 4392- 4402, Seattle, United States, July 2022. Association for Computational Linguistics. 3
[32] OpenAI. Gpt-4 technical report, 2023, 2,3,4,6
[33] Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. arXiv preprint arXiv:203.002/55 2022.
[34] Dong Huk Park, Samaneh Azadi, Xihui Liu, Trevor Darrell, and Anna Rohrbach. Benehmark for compositional text-to-image synthesis. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1), 2021. 3
[35] André Susano Pinto, Alexander Kolesnikov, Yuge Shi, Lucas Beyer, and Xiaohua Zhai. Tuning computer vision models with task rewards. arXiv preprint 2rXiv:2332.002222 2023. 14
[36] Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transfcrable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763. PMLR,
2021. 1, 6, 14
[37] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. ArXiv, abs/2204.06125, 2022. 2, 6, 14
[38] Robin Rombach, A. Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10674-10685, 2021. 1, 6, 14
[39] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10684-10695, 2022. 14
[40] Daniel Rose. Vaishnavi Himakunthala, Andy Ouyang. Ryan He, Alex Mei, Yujie Lu, Michael Saxon, Chinmay Sonar, Diba Mirza, and William Yang Wang. Visual chain of thought: Bridging logical gaps with multimodal infillings, 2023. 3
[41] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driee -riven generation. arXiv preprint arXiv:2208. 12242, 2022. 14
12

---

[42] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton. Kamyar Ghasemipour, Raphael Gontijo Lopes. Burcu Karagol Ayan, Tim Salimans, al. Photorealistic text-to- image diffusion models with deep language understanding. Advances in Neural Information Processing Systems, 35:36479-36494, 2022. 2, 6, 14
[43] Chitwan Saharia. William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L. Denton, Seyed Kam- yar Seyed Ghasemipour, Burcu Karagol Ayan, Seyedeh Sara Mahdavi, Raphael Gontijo Lopes. Tim Salimans, Jonathan Ho, David J. Fleet, and Mohammad Norouzi. Photorealistic text-to-image diffusion models with deep language understanding. ArXiv, abs/2205. 1487, 2022. 1, 14
[44] Mehdi SM Sajjadi, Olivier Bachem, Mario Lucic, Olivier Bousquet, and Sylvain Gelly. Assessing generative models via precision and recall. Advances in neural information processing systems, 31, 2018.
[45] Tim Salimans, Ian Goodfellow, Wojeiech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. Advances in neural information processing systems, 29, 2016. 3
[46] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv nen.:00777..440. 06347, 2017. 14
[47] Tristan Thrush, Ryan Jiang, Max Bartolo, Amanpreet Singh, Adina Williams, Douwe Kiela, and Candace Ross. Winoground: Probing vision and language models for visio- linguisii compositionality. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5238 5248, 2022.
1, 14
[48] Ramakrishna Vedantam, Lawrence Zitmick, and Devi Parikh. Cider:
Consensus besseedu--bsed based image description evaluation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4566-4575, 2015. 14
[49] Zhenhailong Wang, Manling Li, Ruochen Xu, Luowei Zhou, Jie Iei, Xudong Lin, Shuohang Wang, Ziyi Yang, Chenguang Zhu, Derek Hoiem, Shih-Fu Chang, Mohit Bansal, and Heng Ji. Language models with image descriptors are strong few-shot video-language learners, 2022. 3
[50] Jialian Wu, Jianfeng Wang, Zhengyuan Yang, Zhe Gan, Zicheng Liu, Junsong Yuan, and Lijuan Wang. Grit: A generative region-to-text transformer for object understanding, 2022. 4, 6
[51] Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou. When and why vision-language models behave like bags-of-worde words, and what to do about it? arXiv e-prints, pages arXiv-2210, 2022. 1, 14
[52] Lvmin Zhang and Maneesh Agrawala, Adding conditional control to text-to-image diffusion models. arXiv preprint 2rx 2023. 14
[53] Shu Zhang, Xinyi Yang. Yihao Feng, Can Qin, Chia-Chih Chen, Ning Yu, Zeyuan Chen, Huan Wang. Silvio Savarese, Stefano Ermon, al. Hive: Harnessing human feedback for instructional visual editing. arXiv preprint arXiv:2303.09611
2023. 14
[54] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Oit, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. Opt: Open pre-trained transformer language models, 2022. 6
[55] Ming Zhong, Yang Liu, Da Yin, Yuning Mao, Yizhu Jiao, Pengfei Liu, Chenguang Zhu, Heng Ji, and Jiawei Han. Towards unified multi-dimensional evaluator for text generation. 2022. 3
[56] Wanrong Zhu, An Yan, Yujie Lu, Wenda Xu, Xin Eric Wang, Miguel Eckstein, and William Yang Wang. Visualize before you write: Imagination-guided open-ended text generation, 2023. 3
13

---

A Background
A.1 Text-conditioned Image Synthesis
The popular large-scale image generation models such Imagen [43], DALL-E2 [37] and Stable Diffusion [39] demonstrate extraordinary generation quality, but subtle difference in input prompt could lead to a dramatic change of semantic style, thus not directly suitable for image editing [2,41] Besides, the prompttto-prompl [16] achieves image editing with cross-attention control on the observation of interaction between the pixels to the text embedding. InstructPix2Pix [4] leverages the complementary abilities of pre-trained language model GPT-3 [5]. and pre-trained lext-to-image model Stable Diffusion [38] to generate large pairs of multi-modal training data and perform image editing following human instructions, ControlNet [52] learns task-specific conditions in an end-to-end way and achieves robust control effects. Recently, growing interests [35] in computer vision focus on aligning text-to to-image synthesis [23] or visual editing [53] by using human feedback. Typically, a reward model is expected to evaluate images by training on task rewards using proximal policy optimization (PPO [46]).
A.2 Image-level Evaluation Metric
CLIP-based CLIP [36] was pre-trained on large-scale image-caption pairs through contrastive learning, making it a highly versatile tool for natural language processing and computer vision appli- cations. CLIPScore [17] measures the cosine similarity value of image and caption representations extracted from the CLIP feature extractors. Formally, CLII PScore = max max(cos (cos(fr(e:), fc(e)) 0), where fr, is the image and caption feature extractor. CLIPScore is a reference-free metric and outperforms previous reference- based metrics like CIDEr [48] and SPICE [1]. Within the text-to- image domain, previous work also relies on the same approach to measure the alignment between the text prompt and the generated image. However, vision-ande -a---add-lapp and-language models exhibit deficiencies in compositional understanding and are insensitive to word orders. [51], they show BLIP [25] and CLIPP [36] only achieve random chance level understanding ability on attribution, relation, and order understanding. They furthermore propose NegCL.IP to improve the original CL.IP model by generating additional hard negative captions and optimizing the same contrastive objective. They show that obtaining specific and low-cost negative examples can result in significant enhancements in compositional tasks without losing existing ability.
BLIP-based BLIP [25] filters out noisy synthetic captions to effectively make use of the noisy web data through bootstrapping based novel multimodal mixture of Incoder-Decoded Beyond that, BLIPv2 [24] introduces Query Transformer that bootstraps image and text representation learning and then bootstraps large language model for image-to-text generations. BLIPv2 achieves state-of-the-art performance on a wide range of understanding-bast
and generation-basee vision-language tasks, including image-text retrieval, image captioning, and visual question answering. We suppose the grounding objective in BLIPv2's 's pre-training can bootstrap its performance in evaluating text-to- image synthesis. Specifically, we utilize BLIPv2 compute the image-text matching score using "ITM" head and "ITC" head. BLIP-ITC uses a simple cosine similarity function over the extracted image and text features. In contrast, BLIP-ITM uses cross-attention fuse multimodal features to capture fine-grained similarity.
A.3 Evaluation Datasets
The MSCOCO [27] has been widely used for object segmentation, although there is a dearth of varied prompts, indicating a lack of diversity. Winoground [47] designed for evaluating the ability of vision and language models to conduct visio- ciniio-ligusttic compositional reasoning. For pair of two distinct images, their captions are composed of identical sets of words, but in a different order. Many state-of-te ofr-the-rttttttt vision and language models only achieve random chance performance, making it good testbed for evaluation. DrawBench [42] tackles prompt diversity issues by collecting challenging descriptions for image generation. There are set of 11 prompt categories that test various capabilities of models, including the ability to accurately depict colors, numbers of objects, spatial relations, and text, as well as more complex prompts such as long textual descriptions, rare words, and misspelled
14

---

prompts. In [8], they evaluate the visual reasoning of text-to- text-to-image models and propose PaintSkills, diagnostic dataset and evaluation toolkit designed measure object recognition, counting, and spatial understanding. Recent studies in compositional text-to-image synthesis[12 [12] collect Concept Conjunction prompt dataset which focuses on two objects with different colors in the text prompt, and Attribute Binding prompt dataset that is sampled from COCO captions.
B More Results
In Table B, we show the full table that includes variants our LLMScore (CapCLIP, CapMETHOR, DescCLIP, DescMETEOR) General Bench.
Table The correlation between automatic evaluation metries and human rankings on text-to- -image synthesis. Our devised metrics LLMScore significantly surpass existing metrics in terms of Kendall's s and Spearmanr's P with p < 0.001.
COCO2014
COCO2017
Draw Bench
PaintSkills
Human
Metric
=(t)
(()
r(t)
(())
r(*)
p(t)
r(1)
(()
CLIP
0.1971
0.2655
0.2227
0.2771
0.1530
0.2143
0.4715
0.5869
NegCLIP
0.2164
0.29055
0.2793
0.3523
0.1463
0.1999
0.4911
0.6313
BLIP-ITM
0.3252
0.4255
0.0928
0.1155
0.1044
0.1455
0.4755
0.6214
BLIP-ITC
0.3465
0.4535
0.1703
0.2121
0.1569
0.2171
0.4743
0.5864
Overall
CapCLIP
0.0263
0.0335
-0.0274
-0.0315
0.0056
0.0072
0.3035
0.3751
CapMETROR
0.0710
0.0960
0.0512
0.0650
0.0951
0.1312
0.2315
0.3056
DeseLLIP
0.1377
0.1799
0.1130
0.1424
0.1136
0.1537
0.3825
0.4917
DescMETERR
0.1175
0.199
0.0567
0.0702
0.0028
0.0048
0.0678
0.0827
LLMScore
0.3629
0.4612
0.3357
0.4275
0.2230
0.3023
0.5600
0.6853
CLIP
0.1464
0:2142
0,1888
0,2677
0.1360
0.1910
0.3052
0.2891
NegCLIP
0.2116
0.3061
0.1795
0.2581
0.1179
0.1596
0.4563
0.4908
BLIP-ITM
0.2251
0.3289
0.1137
0.1635
0.0871
0.1189
0.4622
0.4997
BLIP-ITC
0.2636
0.3739
0.1849
0.2620
0.1506
0.2029
0.6178
0.6511
Error Counting
CapCLIP
0.0266
0.0362
-0.0068
-0.0085
0.0544
0.0704
0.4963
0.5332
CapMETEOR
0.0822
0.1197
0.0004
0.0013
0.0173
0.0192
0.3274
0.3636
DescCLIP
0,1433
0.2145
0.0338
0.0477
-0,0039
-0,0022
0,2978
0.3289
DescMETEOR
0.1398
0.2010
-0.0829
-0.1198
-0.1348
-0.1791
0.0881
0.0924
LLMScore
0.2792
0.4006
0.2138
0.3125
0.2125
0.2839
0.6444
0.7066
C Human Annotation
For each image-text pair, we ask 2 annotators to rate the overall and error counting. We will show the details of annotation interface in Section C.I.
C.1 Human Ratings Interface
In Figure 8, we show the interface for human ratings over the image quality from two objectives, overall and error counting. Human annotators required to rate the overall quality of the image on a scale of 1-10 and count the errors in the image on a scale of 0--9
15

---

Question images below by rattigg on
Prompt
[Overall QualttyAccorrling Tiaxt Prompt, veriy Composiiionll quality scale of 1D.
9
6
gold ovanall quality
Imagas matching the Prompt:
Poor.
2 Very Bed.
Bad: Low quaiity algnedd with the text prompt.
4 Not
5 Neutral (Leaning Negative)
O 6 Neutral [Leaning Positive)
7 Okky
8 Good: High quality, aligned with the toxt prompt.
9 Vary Good
O 10 Perfect.
Given the Text Prompt "A gold chair and a red clock.":
5
[Error Counting] chair and clock."
Prompt
numbar composition errors (scale: D-9) Generated Images compared Promet. One enror should be counted for each incomeet color, spatial position, shape, size, material, relationship among objects. If an abject category mentioned in the Text Prompt is missing the caption, count it as 4 erross The over-spectiiations the image caption should be counted as
O No camposition fourd.
composition
composition
composition
composilion
composition
6 composiioo
compositon
compoeition enor.
above composition emor.
Figure 8: Amazon Mechanical Turk Platform. Questions Layout for Human Raters for Overall and Compositional ratings of the generated image given the text prompt.
16

---

D Visual Descriptions
Our approach involves utilizing both global and local descriptions of image. Initially, a general caption is generated for the image. Then, dense caption model is employed to describe the objects in detail. This technique enables the extraction of both the overall context of the image and the specific attributes of individual objects, thereby providing comprehensive description of the image.
Detector
Caption
Segmentation Caption
Figure Comparison between, detector, dense caption, and segmentation results over the synthesized image. Alternatively, an object segmentation model can be applied to identify the objects in the image more accurately.
17

---

E
Example of GPT Text-to-Image Evaluation.
Stage
Example
Prompt
a light brown bathroom with a white sink and mirror.
Description
This image showcases a clean and modern bathroom with a white theme. The main focus is the white sink, which is placed The counter is brown and extends across the entire width of the image, covering the lower part of the white wall behind it. Above the sink, there is a large mirror on the wall that reflects the shower curtain as well as a silver light fixture with two light bulbs, which are switched on and provide ample light to the room. The silver faucet complements the white sink and is slightly towards the left side of the sink. A white soap dish is positioned on the right side of the counter, next an additional sink. A silver handle can be seen on drawer beneath the counter, near the sink. On the left side of the picture, a brown wooden door can be partially seen, while visible. The overall setup of the bathroom, with its white walls, bright lights, and well well-arranged counter, gives off spacious and organized vibe.
wooden bathroom counter.
the right side, a small part of white toilet is
Rating Rationale
Overall Semantics: 0.8/1.0
The alignment between the image caption and the text prompt is mostly good. The generated image caption describes white sink, wooden counter, mirror, and other elements like a soap dish, faucet, and lights that complement the room. This is not far from the text prompt of light brown bathroom with white sink and mirror.
Rating Rationale
Error Counting: 1 (number of errors)
The composition error mainly lies in the description of the color of the bathroom. The text prompt mentions a light brown bathroom, but the image caption empha- sizes a white theme, with only the wooden counter and door being brown.
Table 5: Examples of GPT Text-to-Image to-a- Evaluation: the prompt is used for image generation, while the description is generated by GPT-4
We showcase examples for rext-to-image eval- uation in Table 5 and Table on synthesized im-
ages in Figure 10. The Prompt section illustrates the initial input provided to the eext-to-image model, which in this instance is short descrip- tion of bathroom. The subsequent Description section is the multi-granularity object-centric description for the image. Here, the model de- scribes light-filled, modern bathroom with vari- ous elements like a white sink, mirror, soap dish, and wooden counter. Then there are two rating settings, the first is Overall (Overall Semantics), and the second is Error Counting. Both settings evaluate the eext-to-image synthesis with a score and rationale. Here, score of 0.8 out of 1.0 reflects a high level of correspondence between the synthesized image and the prompt, despite few discrepancies. And represents there are only one errors found in the synthesized image compare to the prompt.
Figure 10: Left: Synthesized by Stable-diffusion-2 with text prompt "A light brown bathroom with a white sink and mirror" sampled from Attribute Binding Contrast dataset. Right: Synthesized by Stable-diffusion-2 with text prompt "A brown bird and pink sheep." sampled from COCO.
18

---

Stage
Example
Prompt
a brown bird and a pink sheep.
Description
In this vivid image, there are two unique pink animals standing on large patch of green grass that covers the entire ground. On the left side, there pink sheep with blue tag attached to its ear. The sheep has distinct blue ears and a prominent head, showcasing contrast in colors against its pink body. It also has two sturdy black legs standing firmly on the grass. On the right side of the image, there is pink pig with black eyes. The pig eyes embedded and white stuffed animal-like face, giving it cute and charming look. The pink sheep and the pig casting shadows towards the bottom of the image, blending with the green grass beneath. Overall, the colorful and playful scene provides a fantastical and captivating atmosphere, with the pink sheep and pink pig standing as the central focus.
pink
Rating Rationale
Overall Semantics: 0.8/1.0
There is a clear focus on the pink sheep in the image caption, but the prompt's s mention a brown bird is entirely missing. The overall composition good and somewhat aligned, but it could be better.
Rating Rationale
Error Counting: 4 (number of errors)
The composition error comes from the missing brown bird (4 errors). The color, size, and material of the pink sheep, well the relationship between the two animals are well-des,ribed, making this error count relatively low.
Table 6: Examples of GPT Eext-to-Image Evaluation: the prompt is used for image generation, while the description is generated by GPT-4
19


Reviews:
[{"review_id": "xRktAoPYBR", "reviewer": "Reviewer_9B3k", "summary": "The authors propose an evaluation metric that focuses on the object-level compositionality and the explainability of the evaluation process.\nThey utilize off-the-shelf models including BLIPv2 (global captioner), GRiT (object detector & captioner) and GPT-4 (large language model).\nIn specific, they first get global caption using BLIPv2, and detect object boxes in an image along with local descriptions with GRiT, and fed the incorporation of these results with an evaluation instruction to GPT-4, which would then produce the score and the rationale according to the instruction. This way, the proposed metric produces assessments that are more correlated with the human judgements.", "questions": "- How many objects are selected from GRiT? Are there any rules?\n- Isn't there any score variations according to the change in an evaluation instruction?\n- How much the performance is degraded when the rule-enhanced rating is not used?", "limitations": "please refer to the Weaknesses section.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The proposed metric tackles an important factor in evaluating image generative models.\n- The authors reveal the potential for utilizing LLMs in evaluating image generative models.", "weaknesses": "- The metric would work worse for image domains that are too different from MS-COCO, which is used for training GRiT. (for example, might not work for artistic images…)\n- The evaluation process consists of many sub-processes, which make it prone to have error accumulation issue.\n- The metric utilizes GPT-4, which is not easily accessed by many researchers.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Weaknesses\n\n1. The experiments in our paper include a wide range of prompts and synthesized images from Stable Diffusion and DALLE models,which indicate the LLMScore’s effectiveness in evaluating text-to-image synthesis in a general domain beyond the MSCOCO domain. In addition, the local image descriptor module can be replaced with other expert models including artistic domain.\n2. We acknowledge that the complexity of the evaluation might introduce potential error accumulation. However, LLMScore is carefully designed with a hierarchical structure that allows for both image-level and object-level visual descriptions, thereby enhancing the interpretability and granularity of the evaluation. Moreover, our substantial empirical analysis, including extensive ablation studies, demonstrates the robustness of the proposed method against such error accumulations. The high correlation of LLMScore with human judgments across various datasets (58.8% and 31.2% higher than CLIP and BLIP, respectively) further validates the effectiveness and reliability of our approach. We believe these results illustrate that the potential error accumulation has been well managed, and the innovative use of large language models within LLMScore offers a significant advancement in the text-to-image synthesis evaluation domain.\n3. Thank you for highlighting the concern regarding the accessibility of GPT-4, which is an essential component in our LLMScore framework. We fully understand that the accessibility of GPT-4 may pose a challenge for some researchers. However, it's worth noting that LLMScore is designed to be modular and can work with other large language models that might be more readily available. The key insight of our method lies in leveraging the power of large language models for visio-linguistic compositionality evaluation, not specifically tied to GPT-4. In our revision, we provide implementation details with alternative large language models (e.g., Vicuna) and also demonstrate the core functionality. This ensures that our framework remains adaptable, extendable, and relevant to a broad research community. Therefore, we believe LLMScore is a valuable contribution to the field, regardless of specific accessibility issues related to GPT-4.\n\nQuestions\n\n1. The objects are kept if they are beyond the confidence threshold, which is calculated by multiplying the confidence of the object extractor and object descriptor. We keep the default hyperparameter setting of GRiT to extract these objects.\n2. Yes, the LLMs will be prompted with various evaluation instructions to provide various scores.\n3. For example, under the Concept Conjunction dataset from CompBench, w/o rule enhancement lead to 0.3660 Kendall’s tau and 0.5208 Spearmanr’s correlation with human for Stable Diffusion Model. Similar to the DALLE model, it degrades to 0.3716 and 0.5382 respectively. Though the correlation is still on par with the runner-up model. This indicates a non-trivial performance boost from our rule-enhancement. We will update this in Appendix B in the paper revision."}, {"review_id": "wwRcOYQSq4", "reviewer": "Reviewer_BWFw", "summary": "This paper proposes a new evaluation metric for text-to-image models. The method works as follows:\n- Prompt the Blip-2 VLM to generate an overall text description for the image.\n- Use GRiT to generate coordinates and language descriptions for each object in the scene\n- Use GPT4 to combine these text descriptions into an overall description of the image.\n- Prompt GPT4 to score the image, both by asking it for the overall alignment and by asking it to to count the number of errors in the presence/absence or attributes of objects in the scene.\nThey show this score correlates better with human judgment than common baselines.", "questions": "Consider mentioning similar concurrent works (e.g. TIFA https://tifa-benchmark.github.io/)\n\n\n\n- Does the image get penalized for including objects which were not specified in the prompt but make sense in context (e.g. if it asked for a plate with silverware and a cup next to it, does the model get penalized for putting them on a table? Or for including a dining room background which includes miscellaneous objects?)\n\nNit: \n- Partial phrase on line 218\n- Line 217 \"default language models\" -> \"the default language model\"\n- Table 1 caption: \"Spearmanr's\" -> \"Spearman's\"\n- Specify that in the \"error counting\" metric, lower is better (Fig 7)\n- Table 3: bold best per column.", "limitations": "The one mentioned limitation is that error counting is still challenging.\n\nNot mentioned -- if future works optimize against scores on this metric, models could exploit errors in these metrics. (This is true of all automated metrics on this task though.)", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The approach is significantly more correlated with human judgments than baselines while still being cheaper and faster than human evaluation.\n- Evaluation is a big problem in text-to-image generation, and better quantitative metrics are useful.\n- The paper is well-written and clear, and includes a variety of strong baselines.\n- They break down the accuracy of the different types of evaluation (overall, error counting) so it's easy to see weaknesses.", "weaknesses": "The paper has two stages (1) getting a visual description and (2) computing a score. They explore baselines with different variations of (2) \nbut as far as I can tell ablate different versions of Stage 1 (i.e. using just global image descriptors vs just local object descriptors vs both while using the LLMScore scoring method).\n\n(I think this would be a useful extension, but the paper is strong without it.)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Weaknesses\n\nThank the reviewer for this thoughtful suggestion. We provide ablations to show the advantage of using both descriptions over global caption only on CLIP and METEOR metric. This also indicates that using both can better represent the synthesized image and thus we apply this with LLMScore. As suggested, we will also add quantitative analysis for this ablation in Appendix B. \n\nQuestions\n\n1. Thank you for the question. This remains an open-ended question, as some research work may take this as hallucinations. In our work, we follow the standard that is provided to the human annotators, which require the image to be strictly aligned with the text prompt. However, regarding the background scene (e.g., dining room), this will not be penalized, as both the human and model will think the misc objects as a part of the scene.\n2. Thank you for pointing out these. We will fix these typos as suggested.\n\nLimitations\n\n1. Yes, the error counting is still challenging. A possible solution is to break down the evaluation protocol as atomic as possible, and combine together to bypass the limitation of current models.\n2. To mitigate this, our LLMScore supports incorporating as many submodules/evaluation instructions as possible, and thus easily to validate the robustness of the models (e.g., come out of an unseen and reasonable evaluation criteria). And the LLMScore is extensible to combine multiple off-the-shelf submodules to increase the diversity of the evaluator to avoid such short-cut attacks.\n\n"}, {"review_id": "yedR9Dx6il", "reviewer": "Reviewer_mWLn", "summary": "This paper proposes a new automatic evaluation framework named LLMScore with multi-granularity compositionality to evaluate the synthesized image quality of the mainstream text-to-image synthesis models. \nIts core approach is to tranform images into global and text descriptions through image captioning and region-to-text models, then follow by an LLM to as a visual descriptor and a text-to-image evaluator with rationable output. \nOn six image captioning benchmark datasets, the correlation between its LLMscore and human evaluation results far exceeds the existing commonly used clip-like text-image alignment indicators, reaching into a new SOTA level.\nExtensive experimental analyzes strongly confirm the usability of the method.", "questions": "What does the average rank correlation of the two items in Figure 5 refer to? Could the author provide a more clearly clarification and revise it in their next edition?", "limitations": "As the author said in the supplementray limitation part, relying on OpenAI's closed-source GPT model and the internal bias of the LLM model are obstacles faced by this proposed evaluation framework.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. I truly appreciate the novelty of this paper. It dexterously transform synthesized images into text descriptions, so that it can make full use of LLM's powerful text understanding and reasoning capabilities with a series of hand-crafted constructed Prompts (including sentence analysis, text-image alignment scores estimation, and image generation error statistics). In my standpoint, compared to the previous Clip-like's text-image matching score, this paper undoubtedly set a new T2I quality evaluation paradigm. Aslo, considering the vast influence brought by the AIGC community at present, I believe the impact this paper can bring to the community is significant.\n\n2. The method of this paper is solid, not only in the multi-level extraction of image description, instruction-based and heuristic-based scoring mechanism with rationable generation, but also the considerable performance gains compared with multiple strong baseline methods is also a strong proof. \nThe selected evaluated T2I models are currently the two most popular classes - stable diffusion and DALL·E.\nIn addition, this paper gives ablation experiments of different LLMscore variants and LLMs (GPT3.5 vs GPT4), further confirming the necessity of each component.\n\n3. It brings my great joy is that this work introduces its main ideas and methods, as well as experimental settings and results, etc. in an easy-to-understand manner, so that readers can easily understand.", "weaknesses": "Some of the following problems in my perspective may need to solve for the future deployment: \n\n1. The scoring criteria for \"Human Annotation\" in the appendix should be more specific, and it should be able to guide the annotators to rate the quality of image-text alignment step by step. For example, what is the difference between a score of \"4\" and \"5\", is it because of the color generation of the object or the amount is biased?\nTo be honest, if the results of human annotation are not fine-grained and accurate enough, it may seriously destroy people's trust in the usability of the system. \n\n2. The experiment and analysis can be more perfect, but considering that this article is a fundamental work, the time for urgent submission of ddl may not be enough, so this amount of completion is completely acceptable to me.\n\n3. There are several different models applied in the overall evaluation pipeline. If a large model can be unified in the future to complete all evaluation steps (such as GPT4), the evaluation scheme may be more practical.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Weaknesses\n\n1. In addition to the interface visualized in Figure 8, we do provide clear comparisons to show what is a score “X” example. We agree though it might be better in the future to devise a new standard human evaluation pipeline for this.\n2. Thank the reviewer for the suggestion. We will add more analysis of LLMScore such as what are more possible other evaluations that can be aligned with human annotators, and what are the limitations.\n3. We agree, though LLMScore will still maintain its strengths of fine-grained interpretability of the evaluation process compared with a large foundation model.\n\nQuestions\n\nAs Table 1 shows the individual ranking correlation on each data source from the Composition-focused Prompt Bench (CompBench), the Figure 5 shows the averaged rank correlation of multiple data sources. The same goes to the GeneralBench (Tabel 2 vs. Figure 5).\n\nLimitations\n\nWe now support the open-sourced LLM (e.g., Vicuna). We hope it will foster future research based on our work.\n\n"}, {"review_id": "JSCGCBihlK", "reviewer": "Reviewer_nuAT", "summary": "This paper presents a new method LLMScore to evaluate the alignment between a textual prompt and the generated image (especially in the context of the text-to-image diffusion models).  The key motivation behind this work is to address the inability of popular vision-language models like CLIP to deal with the fine-grained details in a complex prompt. \nAs a result, in this paper, vision-language models such as BLIP is used to caption the image as well as provide object-centric descriptions. These are concatenated to create a final image description using an LLM. Furthermore, the alignment between the text prompt and the image description is computed using an LLM. Experiments on a variety of text-to-image generation benchmarks indicate that the proposed method LLMScore significantly outperforms prior methods of directly measuring the similarity of image and text embeddings. ", "questions": "I have one question regarding the reproducibility of the method. Specifically, with GPT-4, the default temperature of 0.7 is used. In my understanding, this would mean that different runs of the LLM could give different results. Therefore, it might make sense to repeat the experiments multiple times so that the stability of the results are ensured. Of course, if I am missing something, a clarification would be very useful. \n", "limitations": "Yes, there is a limitations section in the supplementary material, which covers these issues satisfactorily. ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1) The problem tackled in this paper is an important one, especially due to the inadequacy of existing models. \n2) The method proposed in the paper is also quite logical (since using an LLM would be able to perform better fine-grained understanding, as opposed to matching coarse grained features)\n3) The experimental results achieved in the paper are quite satisfactory, and show that LLMScore is able to achieve much better results as compared to the standard measures for evaluating image-text alignment. ", "weaknesses": "One concern about this paper is regarding the following paper [a], which could be treated as contemporary work. While it is understandable that this paper is certainly not required as a comparison as it was released less than 2 months before the submission deadline, there are a lot of similarities between [a] and LLMScore. Both rely on a vision-language model to provide descriptions of the image, and use an LLM to reason over the correctness of the generated image. Of course, there are sufficiently many differences between these 2 works ([a] first generating questions using an LLM and then answering using a VQA model, versus LLMScore generating captions/descriptions first, and then using an LLM to evaluate the description), however, it might be a good idea to atleast acknowledge [a] as a contemporary work in the related work section. \n\nRegarding the work itself, an interesting analysis to delve into (given the fine-grained and interpretable nature of LLMScore), would be to break down the errors/inaccuracies into the errors caused due to either the perception models, or with the LLMs (either in constructing the global image description or with the evaluation). Such a breakdown would also enable future work to focus on addressing the weakest aspect of LLMScore to improve it further. Unfortunately, there is no analysis regarding the failure-cases of LLMScore which would have been extremely useful for any reader. \n\n\n[a]: Hu et al. TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answering, Arxiv 2023", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Weaknesses\n\n1. Thank you for the good pointer to this great work. And yes, our work differs from this work in multiple aspects, though they share the same spirit to make the evaluation of text-to-image synthesis more interpretable and fine-grained. We will surely add this contemporary work in the related work and discuss the differences as well.\n2. Exactly, the breakdown interpretability is one of the most appealing benefits from LLMScore. And we observe the failure-cases, mainly come out: 1) when the synthesized images are in bad quality and even human cannot recognize the objects in the images, the descriptions of the image may make hallucinations, 2) when the visual descriptions are all correct, the LLM can sometimes make a mistake in following the evaluation instruction. Potential solution would be further fine-tuning the visual descriptor in the image synthesis domain to make the descriptions more robust. And we can also instruction-tune the large language model to be more aligned with human evaluations in comparing the text prompt and visual descriptions.\n\nQuestions\n\nThank you for pointing this out. Yes, the different runs of the LLM could give different results. Thus the process of generating the visual descriptions and evaluations using LLMs is repeated twice altogether in our experiments to ensure the robustness of the performance gain over the baseline metrics."}, {"review_id": "59hVF8xikL", "reviewer": "Reviewer_ZDTU", "summary": "The authors proposed a framework to evaluate text-to-image synthesis with multi-granularity compositionality powered by LLMs. Two big steps are transforming an image into whole-level and object-level visual descriptions in a text to be used for a part of the prompt to LLMs, then LLMs measure the alignment with a detailed instruction prompt to give the LLMScore. They argue that the correlation between LLMScore and human judgments, along with qualitative evaluations with the generated rationale, confirm the superiority compared with commonly used metrics like CLIPScore.", "questions": "Figure 2-4 includes the brand logo of OpenAI. According to their brand guideline (https://openai.com/brand#things-to-avoid-when-using-our-logo), the authors should not alter their logo using custom colors. Besides, the authors used the OpenAI logo to represent LLMs; however, it is inappropriate to represent LLMs with the brand logo of a for-profit company in the manuscript.", "limitations": "L1. This work decisively relies on the performance of pretrained models. 1) image captioning model BLIPv2, 2) GRiT [41] for local region descriptions, and 3) GPT-4 for LLM evaluator. How do you assess the robustness toward potential errors of visual descriptors or LLM evaluators?\n\nL2. Social bias and fairness issues can also be inherited from pretrained models. \n\nL3. Without the commercial license to use Open API, reproducibility cannot be checked or used. Could you provide the results using the open-sourced LLM evaluators?", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "They proposed an evaluation framework exploiting the power of LLMs to evaluate the challenging text-to-image generations. Giving the rationale for evaluation justification would be a new feature compared with previous works. The experimental procedure is solid to evaluate the proposed metric, although the comparison is limited to commonly used methods, i.e., cosine similarity or cross-attention using CLIP and BLIP features. The manuscript is well-written, well-organized, and easy to follow. ", "weaknesses": "W1. The most concern is the comparison with recent work from NeurIPS 2022, \"Mutual Information Divergence: A Unified Metric for Multimodal Generative Models\" (MID) [17] (the citing number is from the submitted paper). MID similarly exploit pretrained features, i.e., CLIP, but possibly with BLIP, to calculate the text-and-image matching score; however, their method (MID) uses the statistical distribution of the features for finer evaluations than cosine similarity. The authors should compare this recent representative work to show the effectiveness and significance of the proposed method. Specifically, what would be the results of MID using BLIP features for fair comparison?\n\nW2. Does visual descriptor have significance? GPT-4 can accept images as inputs, which means instead of visual descriptions (Sec. 3.1), one may exploit the power of LLMs directly, giving a synthesized image as a prompt. Fine-grained examination of visual input would get good results rather than using cumbersome generations of visual descriptions utilizing other pretrained models. This ablation would support the argument for the core role of the proposed visual descriptor. Otherwise, this work would be reduced to a naive application of multimodal LLMs, e.g., GPT-4.\n\nW3. Robustness to error accumulations. LLM Evaluator inevitably relies on the accuracy of visual descriptions. Since there is no systemical analysis of the catastrophic degradation from an error in visual descriptions may hinder the reliability of the proposed method. \n\nI am eager to reassess my score based on the authors' faithful feedback in handling the issues raised above.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Weaknesses\n1. We thank the reviewer for pointing out this interesting work. We found out this MID metric require the real image features and fake image features with respect to the text prompt for text-to-image synthesis evaluation. This a different setting from recent text-to-image benchmark that is not paired real images, but only the text prompts (e.g., Draw Bench, ABC-6000, CC-500). And our setting is following this trend that our GeneralBench and CompBench are not all paired with reference images. We will discuss this difference in the related work. \n2. The benefit of the visual descriptor comes from two aspects: 1) it provides the breakdown interpretability of the evaluator and it’s easy to trace any evaluation error caused by the module, 2) it’s flexible to be replaced by new SOTA submodules. Besides, the GPT-4 image input API is not released to the public, we’re still the pioneer to pushing the text-to-image synthesis evaluation by utilizing existing released large language model capability.\n3. Thank the reviewer for this insightful suggestion. We ask annotators and manually sample from the intermediate output to check the quality of the visual descriptor and analyze how many portions of bad evaluations are caused by this. And we find out that most of the accuracy of visual descriptions regarding the synthesis evaluation satisfies the evaluation purpose. Though, it remains a bottleneck of the system and we keep it a fundamental solution and keep the modules able to be replaced with any new SOTA modules to furthermore mitigate the error propagation. We want to highlight that the benefit from this interpretable pipeline can also point out the future improvement directions in this line of research.  \n\nQuestions\n\nThank you for pointing this out. We will remove the logo of OpenAI and use a new logo to represent the LLMs in the paper.\n\nLimitations\n1. We agree that the social biases and fairness from the pre-trained models should be considered important. We’re glad to incorporate the mitigation strategies on top of these models to reduce these biases and at the same time still benefit from the rich knowledge from these models.\n2. Thank you for pointing it out. We’ve supported the open-sourced LLM Vicuna to foster future research.\n"}]

Author rebuttal:
General Response to Ethics Reviews

Response to Ethics Reviewer kdWQ

Thanks for pointing out the issues, we address the concerns point-wisely as follow:
1. We hire amazon mechanical turk workers for evaluation. The hourly wage paid to human annotators is estimated at $15.
2. First, we will remove the prompt that is automatically detected as offensive by OpenAI DALLE API. Furthermore, we manually look at more than 30% of the samples uploaded for human evaluation to ensure no offensive images are released.
3. We will remove the logo from the figures to avoid violating the brand guidelines.

Response to Ethics Reviewer kdWQ

The architecture design is to incorporate any large language model, and that we’ve already supported other LLMs outside of OpenAI ones (e.g., Vicuna). Thus, as the reviewer suggested, we will remove the OpenAI logo and leave it as the general LLMs module.

Meta-review: This paper proposes a new evaluation metric for image-text matching in text-to-image generation.   
Text-image matching evalution is a very important and practical topic as the t2i has become prevalent and many t2i applications have emerged. 

All reviewers appreciated for the idea, the metric quality, novelty, and writing quality, and thus gave acceptance score.

AC also agrees with the reviewers, and recommend accepting this paper. 

Additionally, introducing GPT APIs requires some expense for APIs, and GPT4-API is not cheap, in particular. 
AC thinks that it might be helpful for researchers to add the result on using opensource sLLM such as LLaMA2.

Please revise the paper considering the reviewers' feedback and the ethics reviewers' requests.
Decision: Accept (poster)