Title: LLMScore: Unveiling the Power of Large Language Models in Text-to-Image Synthesis Evaluation
Venue: NeurIPS 2023
Accepted: yes
Keywords: Text-to-Image Evaluation

Visio-linguistic Compositionality

Large Language Models
Abstract: Existing automatic evaluation on text-to-image synthesis can only provide an image-text matching score, without considering the object-level compositionality, which results in poor correlation with human judgments. In this work, we propose LLMScore, a new framework that offers evaluation scores with multi-granularity compositionality. LLMScore leverages the large language models (LLMs) to evaluate text-to-image models. Initially, it transforms the image into image-level and object-level visual descriptions. Then an evaluation instruction is fed into the LLMs to measure the alignment between the synthesized image and the text, ultimately generating a score accompanied by a rationale. 
Our substantial analysis reveals the highest correlation of LLMScore with human judgments on a wide range of datasets (Attribute Binding Contrast, Concept Conjunction, MSCOCO, DrawBench, PaintSkills). Notably, our LLMScore achieves Kendall's tau correlation with human evaluations that is 58.8% and 31.2% higher than the commonly-used text-image matching metrics CLIP and BLIP, respectively.

LLMScore: Unveiling the Power of Large Language Models in Text-to-Image Synthesis Evaluation

Abstract

Existing automatic evaluation on text-to-image synthesis can only provide an image-text matching score, without considering the object-level compositionality, which results in poor correlation with human judgments. In this work, we propose LLMScore, a new framework that offers evaluation scores with multi-granularity compositionality. LLMScore leverages the large language models (LLMs) to evaluate text-to-image models. Initially, it transforms the image into image-level and object-level visual descriptions. Then an evaluation instruction is fed into the LLMs to measure the alignment between the synthesized image and the text, ultimately generating a score accompanied by a rationale. Our substantial analysis reveals the highest correlation of LLMScore with human judgments on a wide range of datasets (Attribute Binding Contrast, Concept Conjunction, MSCOCO, DrawBench, PaintSkills). Notably, our LLMScore achieves Kendall's τ correlation with human evaluations that is 58 . 8% and 31 . 2% higher than the commonly-used text-image matching metrics CLIP and BLIP, respectively.

1 Introduction

In recent years, research in text-to-image synthesis has made significant progress [9, 11, 38, 43]. However, evaluation metrics have lagged behind due to challenges such as accurately capturing composite text-image alignment (e.g. color, counting, location) [47], interpretably producing the score, and adaptively evaluating with various objectives.

Established evaluation metrics for text-to-image synthesis like CLIPScore [17] and BLIP [24], while widely used and highly effective [20, 36], have encountered challenges when it comes to capturing object-level alignment between text and image [12, 26]. Figure 1 illustrates an example from the Concept Conjunction dataset [12], given the text prompt 'A red book and a yellow vase' , the left image aligns with the text prompt, while the right image fails to generate a red book, and the correct color for the vase, also with an extra yellow flower . Human judges can make the correct and clear assessment ( 1.00 v.s. 0.45/0.55 ) of these two images on both overall and error counting objectives, while the existing metrics (CLIP, NegCLIP [51], BLIP) predicts similar scores for both images, failing to distinguish the correct image (on the left) from the wrong one (on the right). Furthermore, these metrics provide a single, non-interpretable score, obscuring the underlying reasoning behind the alignment of the synthesized images with the given text prompts. Additionally, these model-based metrics are static, unable to follow varied guidelines that prioritize different objectives of the text-to-image evaluation. For example, the evaluation can range from accessing image-level semantics (Overall) to finer object-level details (Error Counting). These issues hinder the existing metrics from aligning with human evaluations.

Text Prompt:

A red book and a yellow vase.

CLIP

BLIP-ITM

1.00

Overall

BLIP-ITC

1.00

0.26

0.48

0.27

Baseline

Left

Right

0.32

0.49

0.31

1.00

0.55

0.45

Error Counting

1.00

1.00

0.50

Rationale

The overall quality of the image is quite

low due to the significant discrepancies

between the objects described in the text

prompt and those portrayed in the image.

LLMScore - Error Counting

Rationale

The red book from the prompt is not in the

image. The vase is described as red in the

image, while the text prompt specified a

yellow vase. Over-specification of the

yellow flowers in the image.

LLMScore

NegCLIP

0.44

0.99

Overall

Error Counting

Human

In this paper, we introduce LLMScore, a novel framework to evaluate text-image alignment in textto-image synthesis by unveiling the powerful reasoning abilities of large language models (LLMs). Our inspiration stems from the human approach of measuring text-image alignment, which involves checking the correctness of objects and attributes specified in the text prompt. With the incredible text reasoning and generation ability of LLMs, LLMScore can imitate the human evaluation process to access compositionality at multi-granularity and produce alignment scores with rationales, delivering more insight into the model performance and reasons behind the scores.

To enhance the evaluation of composite text-to-image synthesis, our LLMScore elicits grounding visio-linguistic information from vision and language models and LLMs, thereby capturing multigranularity compositionality in the text and image. Specifically, our approach leverages vision and language models to transform the image into multi-granularity (image-level and object-level) visual descriptions, which allows us to capture the compositional aspects of multiple objects in the language format. Then we concatenate these descriptions with text prompts and feed them into large language models (LLMs, for example, GPT-4 [32]) to reason the alignment between text prompts and images. While existing metrics struggle to capture compositionality, our LLMScore captures the object-level alignment between text and image, producing scores that are significantly correlated with human evaluation, complete with reasonable rationales (Figure 1). In addition, our LLMScore can adaptively follow various guidelines (overall or error counting) by simply customizing the evaluation instruction for LLMs. For example, we can access the overall objective by prompting the LLMs with the instruction 'Rate the overall alignment of text prompt and image.' or validate error counting objective with the instruction 'How many compositional errors are in the image?' . And we explicitly include guidance on error types of text-to-image models in the evaluation instruction to keep the LLM's decision deterministic. This flexibility empowers our framework as a versatile tool for a wide range of text-to-image tasks and various evaluation guidelines.

We validate the effectiveness of LLMScore through extensive experiments, demonstrating its alignment with human judgments without any demand for additional training. Our experimental setup contains state-of-the-art text-to-image models such as Stable Diffusion [38] and DALL · E [37], evaluated over diverse datasets, including prompt datasets for general purpose (MSCOCO [28], DrawBench [42], PaintSkills [8]) and for compositional purposes (Abstract Concept Conjunction [12], Attribute Binding Contrast [12]). Our LLMScore achieve the highest human correlation across all datasets. On compositional datasets, we achieve 58 . 8% and 31 . 27% higher Kendall's τ over widely used metrics CLIP and BLIP respectively.

To sum up, we present LLMScore, the first attempt to unveil the power of the large language models for text-to-image evaluation, in particular, our paper makes the following contributions:

We propose LLMScore, a new framework to evaluate the alignment between text prompts and synthesized images in text-to-image synthesis, offering scores that accurately capture multi-granularity compositionality (image-level and object-level).

Our LLMScore produces accurate alignment scores with rationales following various evaluation instructions (overall and error counting).

A red car and a white sheep.

Visual

Description

Image

Caption

Text

Prompt

Text

Prompt

Text

Prompt

Generated

Image

LLM

CLIP/BLIP

METEOR

Text-Image Matching

Sentence Matching

I2: Error Counting

I1: Overall

Instruction-Following Matching

Score

Score

Rationale:

Generally matches.

Discrepancy in the quantity of sheep

Rationale

: The image refers to a herd of

sheep, instead of a single white sheep.

LLMScore (Error Counting)

LLMScore (Overall)

Instruction for overall evaluation

Instruction for error counting evaluation

Evaluation Instructions

We validate the LLMScore on a wide range of datasets (both general purpose and compositional purpose). Our proposed LLMScore achieves the highest human correlation among the commonly used metrics (CLIP, BLIP).

2 Background

Existing Automatic Metrics for Text-to-Image Synthesis Though Inception Score (IS) [45] and the Frechet Inception Distance (FID) [18] are recognized metrics for image fidelity, they do not measure how well the synthesized images align with the text prompts. Existing metrics [21, 22, 34, 44] for measuring such alignment in text-to-image synthesis generally fall into two categories: 1) textimage matching, and 2) sentence matching. For direct comparison between the text and image, the metrics typically rely on the pre-trained text-image matching models, among these, CLIP-based and BLIP-based are most common. Alternatively, the sentence-matching pipeline transforms synthesized images into captions and then measures the sentence-level similarity with the text prompts. This can be achieved by transforming the synthesized images v into free-form captions using the image captioning model BLIPv2 [25]. Then we can apply reference-based image caption metrics such as text-based CLIPScore [17] and METEOR [3] to measure the alignment.

Large Language Models as Evaluation Metrics Very recently, large language models (LLMs) [32, 33] have achieved incredible performance in evaluating natural language generation tasks [14, 29, 55]. This success stems from the LLMs' powerful reasoning and instruction-following, enabling the evaluation of diverse objectives simply by altering the prompt. In addition to language-only tasks, recent studies have demonstrated the effectiveness of eliciting vision and language reasoning abilities of LLMs by incorporating image descriptions [13, 30, 40, 49] or fusing multimodal features [31, 56]. A concurrent work TIFA [19] utilizes LLMs to generate questions for validating text-to-image faithfulness. To our best knowledge, we are the first to introduce object-centric descriptions of images into LLMs for evaluating the multi-granularity compositionality in text-to-image synthesis.

3 LLMScore

Our proposed LLMScore aims to evaluate the alignment between the generated image and the text prompt while capturing multi-granularity compositionality. As depicted in Figure 3, LLMScore has two main components: 1) LLMs As Multi-Granularity Visual Descriptor: transforming the image into multi-granularity object-centric descriptions (Section 3.1), and 2) LLMs As Text-to-Image Evaluator: feeding the evaluation instructions into LLMs, generating the score and rationale (Section 3.2).

3.1 LLMs As Multi-Granularity Visual Descriptor

As illustrated in Figure 3, we first decompose the image into two-level visual descriptions: imagelevel global description; and region-level local descriptions. Then we employ the LLMs to fuse them into a coherent, object-centric, multi-granularity description of the image.

Visual Description

Generated

Image

Image Caption Model

Global Description

Visual Descriptor

LLMScore

LLM

I1

I2

Instance Detection

Extracted Regions

Local Description

Region-to-Text Model

Region 1   Region 2 … Region 6

Text Prompt

Evaluation

Instructions

…

Rationale

In the serene landscape of a dense green hillside, two

sheep are peacefully standing on the road. The sky

stretches across the scene, meeting the field and grass

at the horizon ... A white car can be seen parked on the

road in the backdrop.

LLM

Local Description:

Object 1: a white sheep eating grass: [384, 492, 624, 686]

Object 2: white car is parked: [590, 272, 767, 495]

Object 6:  two sheep grazing in a field: [38, 426, 647, 708]

Global  Description:

Two sheep are standing on the road.

Generate Object-Centric Description:

Global Image Description The global image description is composed of image-level captions and image meta-information. Given an image, for example, in Figure 3, we first transform it into a global description using the state-of-the-art image captioning model BLIPv2, which encapsulates the primary context of the image in a single sentence. Then we concatenate it with the meta-information of the image, such as the resolution (i.e. 512 × 512), which aids in the understanding of the location of each object, and interaction among objects.

Local Region Descriptions Global description can only offer high-level information for the image, detailed descriptions for each region are necessary to capture object-level local information. Here we utilize GRiT [50] to extract regions of interest and transform them into textual descriptions of the regions. GRiT is a model pre-trained with detection and dense caption objectives jointly on the Visual Genome dataset, which contains local fine-grained descriptions for objects in the image. Specifically, we format the description of each object as ' [Object]:[Dense Caption]:[Bounding box] '. As shown in Figure 3, the descriptions of all the objects are concatenated together as our local region descriptions for each image.

Object-Centric Visual Description Though local region descriptions capture dense information about objects in the image, they lose global context compared with global image descriptions, which may lack accurate interpretation of spatial and interaction relationships among objects. By incorporating both the global image description and the local region descriptions, we are able to obtain object-centric visual descriptions that capture multi-granularity compositionality, such as the attributes of the objects and relationships among the objects. Specifically, we feed the local and global descriptions into LLMs (GPT-4 [32] by default) with the template ' [GLOBAL DESCRIPTION] [LOCAL DESCRIPTION] DESCRIPTION INSTRUCTION '. We fill the slot [GLOBAL DESCRIPTION] with the global image description and the slot [LOCAL DESCRIPTION] with the local region description. And the hand-crafted slot DESCRIPTION INSTRUCTION ('generate object-centric description' in Figure 3) is replaced with 'Based on the above information of the image, generate the object-centric visual description regarding the numerical counting, shape, color, size, location, materials of the object and the spatial and interaction relationships among the objects.'

3.2 LLMs As Text-to-Image Evaluator

Large language models have demonstrated strong reasoning abilities (mathematical, coding etc.) on many complex tasks [6]. Here we employ the LLMs to measure the alignments between the generated image and text, and utilize their reasoning ability to understand the compositional attributes of objects and the complex interactions among multiple objects. Given the above-generated visual descriptions (Section 3.1), the whole evaluation process contains three steps: 1) instruction-following rating, 2) rule-enhanced rating, and 3) rationale generation.

Objective n …

What are the steps to

evaluate the overall

alignment between the text

and the image.

Image Description:

In the serene landscape of a dense green

hillside, two  sheep are peacefully standing

on the road

…

A red car and a white sheep.

Interpretable Multi-granularity

Text-to-Image Evaluation Metric

LLMScore (Overall):

0.7

Rationale:

In general, the image does not

correctly depict the text prompt.

LLMScore (Error Counting):

0.77

Rationale:

There are two discrepancies

between the text prompt and the image

caption. The text prompt mentions one red

car and one white sheep, while the image

caption describes a scene with a white car

and two sheep. The errors include the

incorrect car color (white instead of red) and

the presence of an additional sheep.

LLM

Visual Description

Text Prompt

Instruction

Following

Evaluation

I1

I2

Evaluation

Instructions

…

Concatenate

Instruction n …

Rate the overall quality of

the image in terms of

matching the text prompt.

Instruction 1 (I2)

Rate the overall quality of

the image in terms of

matching the text prompt.

Instruction 1 (I1)

Rate the overall quality of

the image in terms of

matching the text prompt.

Objective 1 (Error Counting)

What are the steps to

evaluate the overall

alignment between the text

and the image.

Objective 1 (Overall)

What are the steps to

evaluate the overall

alignment between the text

and the image.

Instruction

Formulation

LLM

/

User

Evaluation Objectives

Text Prompt:

I1: Rate the overall quality of the image

in terms of matching the text prompt.

I2: Provide the number of composition

errors in the image compared to the

text prompt.

Instruction-Following Rating As shown in Figure 4, we design evaluation instructions to guide the LLM in evaluating the image based on specific criteria, such as overall semantics, error counting, etc. The instructions are explicit and detailed, and can include specific guidance on error types (i.e. counting/shape/color/size) commonly seen in text-to-image models. This ensures the LLM's evaluation remains deterministic and is not influenced by any inherent biases in the pre-training data. The visual descriptions and the evaluation instructions are fed into the LLM to generate a score. For example, for Overall quality evaluation, the hand-crafted slot [EVALUATION INSTRUCTION] (denoted as Instruction I1/I2 in Figure 4) is replaced with ' Rate the overall quality of the image in terms of matching the text prompt. ' Similarly, for Error Counting evaluation, the hand-crafted slot [EVALUATION INSTRUCTION] is replaced with ' Provide the number of compositional errors in the image compared to the text prompt. '. We define the error type as the object-level difference. For example, the counting/shape/color/size difference in the image and text prompt should be counted as one error. This approach can either speed up the process of human annotators or serve as an interpretable and consistent evaluation pipeline.

Rule-Enhanced Rating While directly using LLMs to produce evaluation scores has its merits, it presents certain challenges. One notable issue for LLM is that it can only generate discrete outputs, making it challenging to produce decimal scores even when the text prompt explicitly specifies that. To mitigate this issue, we can restrict the LLMs to rate over a wider range of scale (on a scale of 1-N, where N is default set as 100) in integer and downscale to a smaller range of scale (divided by N by default) to enforce decimal. This approach provides more flexibility to capture small discrepancies compared to directly prompting with the decimal score.

To further enhance the consistency of LLMs' ratings, we propose breaking down the evaluation process into deterministic atomic tasks and employing basic heuristic rules to imitate the human evaluation process more accurately and consistently. In the first step, the LLMs receive the concatenated information (text prompt, image description, and evaluation instruction) and produce predictions for a pre-defined sequence of atomic tasks. The atomic tasks include obtaining: 1) the number of objects specified in the text prompt (X1), 2) the number of matched objects in the image (X2), 3) the number of specified attributes in the text prompt (Y1), and 4) the number of matched attributes in the text prompt (Y2). The results for these atomic tasks are more deterministic compared with high-level tasks. In the second step, we use basic heuristics rules inspired by human evaluation processes, which reason how well the objects specified in the text prompt are presented in the image descriptions and how correctly the attributes are depicted in the image. Thus we combine the results from atomic tasks and derive the final evaluation score by ( X 2 /X 1) / 2 + ( Y 2 /Y 1) / 2 . This is inspired by recent work [7, 15] that separates the calculation in LLMs to achieve more reliable results. This approach allows us to overcome the limitations of decimal value generation and consistency in LLMs. Thus we obtain evaluation results that are more accurate, interpretable, and closer to human judgment.

Generating Rationale The score generation process involves the LLMs' understanding of various evaluation instructions, such as assessing the overall text-image alignment or precisely counting the number of errors in the image. Then the LLMs apply the understanding of the evaluation instruction

to the concatenated visual description and text prompt, generating a score that reflects the alignment between the image and the text at multi-granularity compositionality (image-level and object-level). Each score is accompanied by a rationale. To be specific, we add the explanation instruction prompt for LLMs, such as 'Explain the overall rating X within one paragraph.' for the overall objective and 'Explain the error counting within one paragraph' for the error counting objective. As shown in Figure 4, our LLMScore generate reasonable rationales for both overall and error counting objectives. The rationales for the scores further provide insights into the LLM's decision-making process.

4 Experiments

4.1 Experiments Setup

Datasets For general purpose evaluation (GeneralBench), we sample 200 text prompts from each dataset, including MSCOCO [27] 2014 and 2017, DrawBench [42], and PaintSkills [8]. For compositional purpose evaluation (CompBench), we sample 200 text prompts from Concept Conjunction (CC) and Attribute Binding Contrast (ABC) datasets that are designed for evaluating compositional text-to-image synthesis [12]. In total, we gather 1200 text prompts for human correlation experiments.

Text-to-Image Models For each sampled text prompt, we generate two images using two widely used text-to-image models, Stable Diffusion [38] and DALL · E[37], which demonstrate extraordinary generation quality. To be specific, we use Stable Diffusion 2.1-v from Hugging Face, and DALL · E 2 using OpenAI API in April 2023. All the images are generated at a resolution of 512 × 512. The total text-image pairs prompts used in human correlation experiments are 2400 .

Baseline Metrics We consider these publicly available model-based evaluation metrics, which fall into the text-image matching pipeline (depicted in Figure 2) in evaluating text-to-image synthesis.

CLIP [17, 36] measures the cosine similarity of image and text prompt representations extracted from the CLIP feature extractors. This is a widely used model-based metric to measure the text-image alignment in text-to-image synthesis.

NegCLIP [51] uses a fine-tuned CLIP with improved compositionality understanding to measure the cosine similarity of the image and the text prompt.

BLIP-ITC [24, 25] uses a cosine similarity function over the extracted image and text features by BLIPv2, similarly as CLIP.

BLIP-ITM [24, 25] uses cross-attention to fuse multimodal features extracted by BLIPv2 to compute fine-grained similarity.

We discuss the sentence-matching pipeline as our ablations in Section 4.3. Notice that we focus on evaluating how well the synthesized images are aligned with the text prompts. Thus the widely used Inception Score (IS) and the Frechet Inception Distance (FID) for evaluating image quality are not considered in our baselines, since they do not compare the image with the text prompts.

Implementation Details We extract the global image description using the pre-trained 2.7B BLIPv2 [24] model equipped with large language model OPT [54]. We extract local region description using the dense caption model GRiT [50] with ViT-Base [10] pre-trained on COCO 2017. We obtain the object-centric visual description using GPT-4 [32] as default language models to combine the global and local information. The is by default generated by GPT-4 with The GPT-4 model is the default descriptor to combine global and local descriptions into object-centric visual descriptions and the default evaluator to produce the score with rationale. All experiments conducted with GPT models are using OpenAI API from April 2023 to May 2023 with temperature 0 . 7 using greedy decoding by default.

4.2 Human Ratings

For each generated text-image pair, we ask 2 human annotators to provide ratings over these synthesized images in terms of Overall quality and Error Counting :

Overall : a general-purpose text-to-image evaluation, which applies to most existing metrics. Human annotators are required to rate the overall quality of the synthesized images in terms of matching the Text Prompt.

Error Counting : Human annotators are required to provide the number of compositional errors in the synthesized images compared to the text prompt. The error types include: 1) compositional errors: wrong attributes (color, spatial position, shape, size, material) of the objects and wrong relationship among objects, 2) missing object errors: the objects mentioned in the text prompt are not present in the image, and 3) over-specification errors: the image hallucinates irrelevant objects in the image that are not specified in the text prompt.

The averaged inter-rater agreement is 0 . 62 under Krippendorff's alpha agreement measure. We provide clear guidance for human annotators, with details shown in Appendix C.

4.3 Human Correlation

In Table 1 and Table 2, we use Kendall's tau ( τ ) and Spearman's rho ( ρ ) to measure the ranking correlation to both Overall and Error Counting human rating for compositional bench and general bench. All the model-based metric scores and human ratings are normalized to 0 -1 for comparison.

Overall Results As concluded in Table 1 & 2 and Figure 5, 1). the most popular text-image alignment metric (CLIP) for text-to-image evaluations is less correlated with human ratings than

(a) Averaged Kendall's τ Ranking Correlation with Human Ratings.

0.4

0.3

LLMScore

CLIP

NegCLIP

BLIP-ITM

BLIP-ITC

0.28

0.45

0.175

0.150

0.125

0.16

0.19

0.35

0.40

0.25

0.30

0.29

0.31

0.3

0.33

0.4

0.35

0.30

0.25

0.27

0.26

0.35

0.38

0.100

0.20

0.20

0.2

0.2

0.14

0.075

0.06

0.06

0.07

0.15

0.15

0.1

0.09

0.050

0.10

0.10

0.06

0.025

0.05

0.05

0.0

0.000

0.00

0.00

CompBench-Overall

CompBench-Error Counting

GeneralBench-Overall

GeneralBench-ErrorCounting

0.6

CLIP

0.63

0.25

0.26

0.5

0.5

0.5

0.48

0.5

I

LLMScore

NegCLIP

BLIP-ITM

BLIP-ITC

0.20

0.22

0.4

0.37

0.4

0.39

0.42

0.4

0.34

0.32

0.42

0.4

0.39

0.15

0.3

0.3

0.24

0.3

0.2

0.2

0.10

0.08

0.09

0.1

0.2

0.2

0.1

0.08

0.13

0.05

0.1

0.1

0.0

0.00

0.0

0.0

CompBench-Overall

CompBench-Error Counting

GeneralBench-Overall

GeneralBench-ErrorCounting

expected. NegCLIP utilizes hard negatives to improve the compositionality of CLIP; 2). BLIP-based metrics (BLIP-ITM and BLIP-ITC) surpass these CLIP-based metrics (CLIP and NegCLIP). We suppose BLIP-based metrics benefits from the better object-level vision-language presentations learned by grounding tasks, compared with image-level matching learned in CLIP. 3). LLMScore is significantly better than the existing metrics with large margins.

Accurate Error Counting in the Image is Challenging As shown in Figure 5, all the metrics achieve better correlation with Overall human ratings than Error Counting human ratings given the fact that there are various error types and capture each of them requires more accurate compositional visio-linguistic understanding and generation.

Object-centric Visual Descriptions Improve Compositional Understanding In Figure 5, we show that our LLMScore achieves larger correlation gain over the text prompts on the compositional bench (e.g., counting, position, size, color, relations). This further confirms our superiority in capturing compositionality with the introduced object-centric visual descriptions.

Image Captions v.s. Visual Descriptions We consider two categories of variants for LLMScore:

Text-Caption Matching: CapCLIP and CapMETEOR use the CLIP and METEOR to measure the similarity between the captions (which is the global image description in Section 3.1) of the synthesized images and text prompts.

Text-Description Matching: DescCLIP and DescMeteor, which use the CLIPScore [17] and METEOR [3] to calculate the similarity score between visual descriptions (in Section 3.1) and text prompts.

CapCLIP

CapMETEOR DescCLIP DescMETEORLLMScore

0.0

0.1

0.2

0.3

0.4

Kendall's tau

Overall

Error Counting

A red clock and a gold suitcase.

Overall Rationale

The image caption describes two red suitcases, a vintage clock on the wall, and a

blue curtain in the background, but the text prompt only mentions a red clock and

a gold suitcase. The alignment between the text prompt and image caption is

weak.

CLIP

NegCLIP

BLIP-ITM

0.70

Overall

BLIP-ITC

Error Counting Rationale

The composition errors include incorrect suitcase colors, incorrect clock color,

and additional elements not mentioned in the text prompt (a second suitcase and

blue curtain).

0.33

Error Counting

0.36

0.45

0.30

0.99

Human

Baseline

LLMScore (Error Counting)        0.55

LLMScore (Overall)       0.40

This kitchen has wooden cabinets

and silver and black appliances

Overall Rationale

The caption accurately describes the image and aligns well with the text prompt,

offering additional details about various elements and creating a clear picture of

the room.

CLIP

NegCLIP

BLIP-ITM

0.00

Overall

BLIP-ITC

Error Counting Rationale

There are no composition errors in the caption as it accurately describes the

colors, materials, and spatial positions of all relevant objects within the setting,

adhering to the text prompt's focus on wooden cabinets and black appliances.

1.00

Error Counting

0.25

0.42

0.24

0.40

Human

Baseline

LLMScore (Error Counting)        0.00

LLMScore (Overall)       1.00

5

X

The main drawback of such a sentence-matching pipeline is that the caption metrics favor the coverage or similar language structure instead of capturing compositional semantics. Figure 6 shows the average correlation (Kendall's τ ) on the general bench set, LLMScore significantly outperforms these two variants, indicating that, caption metrics is not a good means to measure the alignment between image and text. Despite the advance introduced by object-centric visual descriptions, the limitations associated with using caption metrics to measure semantic similarity hinder the performance gain of text-description matching metrics DescCLIP and DescMETEOR over text-caption matching metrics CapCLIP and CapMETEOR.

Differing Large Language Models In Table 3, we compare two variants of large language models: GPT-3.5 and GPT-4. 1). For the same group (for example, GPT-4 based) visual descriptions, LLMScore is better than DescCLIP and DescMeteor, indicating that LLMs reasoning is the key component to capture the semantic relations in the visual descriptions. 2). We observe that GPT-4 based LLMScore (Overall) has an average better correlation than GPT-3.5, indicating the performance benefits from the better reasoning ability in the larger-scale language model. However, GPT-4 based LLMScore (Error Counting) is only comparable with GPT-3.5, indicating that the counting is still un-resolved in large-scale language models.

4.4 Multi-Granularity Rationale for Text-to-Image Evaluation

In Figure 7, we show two examples, our LLMScore not only produces human-correlated text-image scores but also provides a rationale for each metric value. The rationale correctly explains both the differences and similarities between the descriptions and text prompts. The model performs well in explaining the errors counted and the overall comparisons.

5 Conclusion

In this paper, we re-examine the existing model-based text-to-image metrics and propose LLMScore to evaluate text-to-image synthesis by unveiling the power of large language models. Our LLMScore can capture the multi-granularity compositionality between the synthesized images and the text prompt, producing accurate alignment scores with rationales. Our LLMScore demonstrates significantly better correlation with human scores on several datasets, paving the way for a more adaptable text-to-image evaluation, capable of following human instructions to evaluate the text-image alignment.

Broader Impact

The framework proposed in this paper first integrates GPT-4 for text-to-image evaluation and showcases how to take advantage of the existing large-scale pre-trained models (GPT-4) for measuring the alignment between the generated images and text, we also propose a new metric, LLMScore which provides interpretable rating and well aligns with the human scores on several datasets. This work sheds light on the value of large language models on the evaluation of text-to-image synthesis, we hope it can help the future text-to-image synthesis work on improving the groundedness and compositionality, either as a reward signal or evaluation metric; our preliminary work on the interpretability of LLMScore, may have the potential to be used for explanation, controllable generation, and image editing.

Limitations

One limitation of our work is that it relies on GPT, which is not free for the public, and may limit its fast plug-in capability, future work may consider replacing this component with a publicly available LLM model (e.g., LLaMA) or our in-house finetuned image captioning model. Another potential issue for this work is, since it incorporates the exsiting large language models, it may inherit its own biases that could propagate to the metric. The future work who considers adopting our LLMScore metric should be cautious on the specific domains to make sure no harmful biases get propagated.

Acknowledgments

The research was sponsored by the National Science Foundation under Grant No. 2048122. The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the sponsor.

References

Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. Spice: Semantic propositional image caption evaluation. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part V 14 , pages 382-398. Springer, 2016. 14

Omri Avrahami, Ohad Fried, and Dani Lischinski. Blended latent diffusion. arXiv preprint arXiv:2206.02779 , 2022. 14

Satanjeev Banerjee and Alon Lavie. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization , pages 65-72, Ann Arbor, Michigan, June 2005. Association for Computational Linguistics. 3, 8

Tim Brooks, Aleksander Holynski, and Alexei A Efros. Instructpix2pix: Learning to follow image editing instructions. arXiv preprint arXiv:2211.09800 , 2022. 14

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems , 33:1877-1901, 2020. 14

Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712 , 2023. 4

Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W. Cohen. Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks, 2022. 5

Jaemin Cho, Abhay Zala, and Mohit Bansal. Dall-eval: Probing the reasoning skills and social biases of text-to-image generative models, 2022. 2, 6, 15

Prafulla Dhariwal and Alex Nichol. Diffusion models beat gans on image synthesis. ArXiv , abs/2105.05233, 2021. 1

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale, 2021. 6

Patrick Esser, Robin Rombach, and Björn Ommer. Taming transformers for high-resolution image synthesis, 2020. 1

Weixi Feng, Xuehai He, Tsu-Jui Fu, Varun Jampani, Arjun Akula, Pradyumna Narayana, Sugato Basu, Xin Eric Wang, and William Yang Wang. Training-free structured diffusion guidance for compositional text-to-image synthesis, 2023. 1, 2, 6, 15

Weixi Feng, Wanrong Zhu, Tsu-jui Fu, Varun Jampani, Arjun Akula, Xuehai He, Sugato Basu, Xin Eric Wang, and William Yang Wang. Layoutgpt: Compositional visual planning and generation with large language models. arXiv preprint arXiv:2305.15393 , 2023. 3

Jinlan Fu, See-Kiong Ng, Zhengbao Jiang, and Pengfei Liu. Gptscore: Evaluate as you desire, 2023. 3

Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. Pal: Program-aided language models, 2023. 5

Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Prompt-toprompt image editing with cross attention control. arXiv preprint arXiv:2208.01626 , 2022. 14

Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. CLIPScore: A reference-free evaluation metric for image captioning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , pages 7514-7528, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. 1, 3, 6, 8, 14

Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems , 30, 2017. 3

Yushi Hu, Benlin Liu, Jungo Kasai, Yizhong Wang, Mari Ostendorf, Ranjay Krishna, and Noah A Smith. Tifa: Accurate and interpretable text-to-image faithfulness evaluation with question answering. arXiv preprint arXiv:2303.11897 , 2023. 3

Gwanghyun Kim, Taesung Kwon, and Jong Chul Ye. Diffusionclip: Text-guided diffusion models for robust image manipulation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 2426-2435, June 2022. 1

Jin-Hwa Kim, Yunji Kim, Jiyoung Lee, Kang Min Yoo, and Sang-Woo Lee. Mutual information divergence: A unified metric for multimodal generative models. arXiv preprint arXiv:2205.13445 , 2022. 3

Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved precision and recall metric for assessing generative models. Advances in Neural Information Processing Systems , 32, 2019. 3

Kimin Lee, Hao Liu, Moonkyung Ryu, Olivia Watkins, Yuqing Du, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, and Shixiang Shane Gu. Aligning text-to-image models using human feedback, 2023. 14

Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models, 2023. 1, 6, 14

Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In International Conference on Machine Learning , pages 12888-12900. PMLR, 2022. 3, 6, 14

Zhiheng Li, Martin Renqiang Min, Kai Li, and Chenliang Xu. Stylet2i: Toward compositional and high-fidelity text-to-image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2022. 1

Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13 , pages 740-755. Springer, 2014. 6, 14

Tsung-Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft coco: Common objects in context. In European Conference on Computer Vision , 2014. 2

Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. G-eval: Nlg evaluation using gpt-4 with better human alignment, 2023. 3

Yujie Lu, Pan Lu, Zhiyu Chen, Wanrong Zhu, Xin Eric Wang, and William Yang Wang. Multimodal procedural planning via dual text-image prompting, 2023. 3

Yujie Lu, Wanrong Zhu, Xin Wang, Miguel Eckstein, and William Yang Wang. Imagination-augmented natural language understanding. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL) , pages 43924402, Seattle, United States, July 2022. Association for Computational Linguistics. 3

OpenAI. Gpt-4 technical report, 2023. 2, 3, 4, 6

Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.02155 , 2022. 3

Dong Huk Park, Samaneh Azadi, Xihui Liu, Trevor Darrell, and Anna Rohrbach. Benchmark for compositional text-to-image synthesis. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1) , 2021. 3

André Susano Pinto, Alexander Kolesnikov, Yuge Shi, Lucas Beyer, and Xiaohua Zhai. Tuning computer vision models with task rewards. arXiv preprint arXiv:2302.08242 , 2023. 14

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning , pages 8748-8763. PMLR, 2021. 1, 6, 14

Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. ArXiv , abs/2204.06125, 2022. 2, 6, 14

Robin Rombach, A. Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 10674-10685, 2021. 1, 2, 6, 14

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10684-10695, 2022. 14

Daniel Rose, Vaishnavi Himakunthala, Andy Ouyang, Ryan He, Alex Mei, Yujie Lu, Michael Saxon, Chinmay Sonar, Diba Mirza, and William Yang Wang. Visual chain of thought: Bridging logical gaps with multimodal infillings, 2023. 3

Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. arXiv preprint arXiv:2208.12242 , 2022. 14

Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-toimage diffusion models with deep language understanding. Advances in Neural Information Processing Systems , 35:36479-36494, 2022. 2, 6, 14

Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L. Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, Seyedeh Sara Mahdavi, Raphael Gontijo Lopes, Tim Salimans, Jonathan Ho, David J. Fleet, and Mohammad Norouzi. Photorealistic text-to-image diffusion models with deep language understanding. ArXiv , abs/2205.11487, 2022. 1, 14

Mehdi SM Sajjadi, Olivier Bachem, Mario Lucic, Olivier Bousquet, and Sylvain Gelly. Assessing generative models via precision and recall. Advances in neural information processing systems , 31, 2018. 3

Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. Advances in neural information processing systems , 29, 2016. 3

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 , 2017. 14

Tristan Thrush, Ryan Jiang, Max Bartolo, Amanpreet Singh, Adina Williams, Douwe Kiela, and Candace Ross. Winoground: Probing vision and language models for visio-linguistic compositionality. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 5238-5248, 2022. 1, 14

Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. Cider: Consensus-based image description evaluation. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 4566-4575, 2015. 14

Zhenhailong Wang, Manling Li, Ruochen Xu, Luowei Zhou, Jie Lei, Xudong Lin, Shuohang Wang, Ziyi Yang, Chenguang Zhu, Derek Hoiem, Shih-Fu Chang, Mohit Bansal, and Heng Ji. Language models with image descriptors are strong few-shot video-language learners, 2022. 3

Jialian Wu, Jianfeng Wang, Zhengyuan Yang, Zhe Gan, Zicheng Liu, Junsong Yuan, and Lijuan Wang. Grit: A generative region-to-text transformer for object understanding, 2022. 4, 6

Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou. When and why vision-language models behave like bags-of-words, and what to do about it? arXiv e-prints , pages arXiv-2210, 2022. 1, 6, 14

Lvmin Zhang and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. arXiv preprint arXiv:2302.05543 , 2023. 14

Shu Zhang, Xinyi Yang, Yihao Feng, Can Qin, Chia-Chih Chen, Ning Yu, Zeyuan Chen, Huan Wang, Silvio Savarese, Stefano Ermon, et al. Hive: Harnessing human feedback for instructional visual editing. arXiv preprint arXiv:2303.09618 , 2023. 14

Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. Opt: Open pre-trained transformer language models, 2022. 6

Ming Zhong, Yang Liu, Da Yin, Yuning Mao, Yizhu Jiao, Pengfei Liu, Chenguang Zhu, Heng Ji, and Jiawei Han. Towards a unified multi-dimensional evaluator for text generation, 2022. 3

Wanrong Zhu, An Yan, Yujie Lu, Wenda Xu, Xin Eric Wang, Miguel Eckstein, and William Yang Wang. Visualize before you write: Imagination-guided open-ended text generation, 2023. 3

A Background

A.1 Text-conditioned Image Synthesis

The popular large-scale image generation models such as Imagen [43], DALL-E2 [37] and Stable Diffusion [39] demonstrate extraordinary generation quality, but a subtle difference in input prompt could lead to a dramatic change of semantic style, thus not directly suitable for image editing [2, 41]. Besides, the prompt-to-prompt [16] achieves image editing with cross-attention control on the observation of interaction between the pixels to the text embedding. InstructPix2Pix [4] leverages the complementary abilities of a pre-trained language model GPT-3 [5], and a pre-trained text-to-image model Stable Diffusion [38] to generate large pairs of multi-modal training data and perform image editing following human instructions. ControlNet [52] learns task-specific conditions in an end-to-end way and achieves robust control effects. Recently, growing interests [35] in computer vision focus on aligning text-to-image synthesis [23] or visual editing [53] by using human feedback. Typically, a reward model is expected to evaluate images by training on task rewards using proximal policy optimization (PPO [46]).

A.2 Image-level Evaluation Metric

CLIP-based CLIP [36] was pre-trained on large-scale image-caption pairs through contrastive learning, making it a highly versatile tool for natural language processing and computer vision applications. CLIPScore [17] measures the cosine similarity value of image and caption representations extracted from the CLIP feature extractors. Formally, CLIPScore = max ( cos ( f I ( v i ) , f C ( c i )) , 0) , where f I , f C is the image and caption feature extractor. CLIPScore is a reference-free metric and outperforms previous reference-based metrics like CIDEr [48] and SPICE [1]. Within the text-toimage domain, previous work also relies on the same approach to measure the alignment between the text prompt and the generated image. However, vision-and-language models exhibit deficiencies in compositional understanding and are insensitive to word orders. In [51], they show BLIP [25] and CLIP [36] only achieve random chance level understanding ability on attribution, relation, and order understanding. They furthermore propose NegCLIP to improve the original CLIP model by generating additional hard negative captions and optimizing the same contrastive objective. They show that obtaining specific and low-cost negative examples can result in significant enhancements in compositional tasks without losing existing ability.

BLIP-based BLIP [25] filters out noisy synthetic captions to effectively make use of the noisy web data through bootstrapping based on a novel multimodal mixture of Encoder-Decoder. Beyond that, BLIPv2 [24] introduces Query Transformer that bootstraps image and text representation learning and then bootstraps large language model for image-to-text generations. BLIPv2 achieves state-of-the-art performance on a wide range of understanding-based and generation-based vision-language tasks, including image-text retrieval, image captioning, and visual question answering. We suppose the grounding objective in BLIPv2's pre-training can bootstrap its performance in evaluating text-toimage synthesis. Specifically, we utilize BLIPv2 to compute the image-text matching score using 'ITM' head and 'ITC' head. BLIP-ITC uses a simple cosine similarity function over the extracted image and text features. In contrast, BLIP-ITM uses cross-attention to fuse multimodal features to capture fine-grained similarity.

A.3 Evaluation Datasets

The MSCOCO [27] has been widely used for object segmentation, although there is a dearth of varied prompts, indicating a lack of diversity. Winoground [47] is designed for evaluating the ability of vision and language models to conduct visio-linguistic compositional reasoning. For a pair of two distinct images, their captions are composed of identical sets of words, but in a different order. Many state-of-the-art vision and language models only achieve random chance performance, making it a good testbed for evaluation. DrawBench [42] tackles prompt diversity issues by collecting challenging descriptions for image generation. There are a set of 11 prompt categories that test various capabilities of models, including the ability to accurately depict colors, numbers of objects, spatial relations, and text, as well as more complex prompts such as long textual descriptions, rare words, and misspelled

prompts. In [8], they evaluate the visual reasoning of text-to-image models and propose PaintSkills, a diagnostic dataset and evaluation toolkit designed to measure object recognition, counting, and spatial understanding. Recent studies in compositional text-to-image synthesis[12] collect Concept Conjunction prompt dataset which focuses on two objects with different colors in the text prompt, and Attribute Binding prompt dataset that is sampled from COCO captions.

B More Results

In Table B, we show the full table that includes variants of our LLMScore (CapCLIP, CapMETEOR, DescCLIP, DescMETEOR) on General Bench.

C Human Annotation

For each image-text pair, we ask 2 annotators to rate the overall and error counting. We will show the details of annotation interface in Section C.1.

C.1 Human Ratings Interface

In Figure 8, we show the interface for human ratings over the image quality from two objectives, overall and error counting. Human annotators are required to rate the overall quality of the image on a scale of 1-10 and count the errors in the image on a scale of 0-9.

6

5

9

5

Question

GiventheTextPrompt"Agoldchairandaredclock.":

a scale of 10.

6

5

TextPrompt"A gold chair and a red clock.""

Rate the overallquality of theGenerated Imagesintermsof matching theTextPrompt:

GiventheTextPrompt"Agoldchairandaredclock.":

9

5

TextPrompt "gold chair and a redclock."

[ErrorCounting]Provide the numberof composition errorsY(scale:O-9) in theGenerated Imagescompared totheTextPrompt. Oneerrorshouldbecountedforeachincorrectcolor,spatialposition,shape,size,material,orrelationshipamongobjects.lfanobject categorymentioned intheTextPrompt ismissinginthecaption,countit as4errors.Theover-specificationsintheimagecaptionshould becounted asonlyoneerror:

D Visual Descriptions

Our approach involves utilizing both global and local descriptions of an image. Initially, a general caption is generated for the image. Then, a dense caption model is employed to describe the objects in detail. This technique enables the extraction of both the overall context of the image and the specific attributes of individual objects, thereby providing a comprehensive description of the image.

Detector

sheep 96%

a dark green hillside

E Example of GPT Text-to-Image Evaluation.

We showcase examples for text-to-image evaluation in Table 5 and Table 6 on synthesized images in Figure 10. The Prompt section illustrates the initial input provided to the text-to-image model, which in this instance is a short description of a bathroom. The subsequent Description section is the multi-granularity object-centric description for the image. Here, the model describes a light-filled, modern bathroom with various elements like a white sink, mirror, soap dish, and a wooden counter. Then there are two rating settings, the first is Overall (Overall Semantics), and the second is Error Counting . Both settings evaluate the text-to-image synthesis with a score and a rationale. Here, a score of 0.8 out of 1.0 reflects a high level of correspondence between the synthesized image and the prompt, despite a

few discrepancies. And 1 represents there are only one errors found in the synthesized image compare to the prompt.

Table 6: Examples of GPT Text-to-Image Evaluation: the prompt is used for image generation, while the description is generated by GPT-4

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors proposed a framework to evaluate text-to-image synthesis with multi-granularity compositionality powered by LLMs. Two big steps are transforming an image into whole-level and object-level visual descriptions in a text to be used for a part of the prompt to LLMs, then LLMs measure the alignment with a detailed instruction prompt to give the LLMScore. They argue that the correlation between LLMScore and human judgments, along with qualitative evaluations with the generated rationale, confirm the superiority compared with commonly used metrics like CLIPScore.\n\nSTRENGTHS:\nThey proposed an evaluation framework exploiting the power of LLMs to evaluate the challenging text-to-image generations. Giving the rationale for evaluation justification would be a new feature compared with previous works. The experimental procedure is solid to evaluate the proposed metric, although the comparison is limited to commonly used methods, i.e., cosine similarity or cross-attention using CLIP and BLIP features. The manuscript is well-written, well-organized, and easy to follow.\n\nWEAKNESSES:\nW1. The most concern is the comparison with recent work from NeurIPS 2022, \"Mutual Information Divergence: A Unified Metric for Multimodal Generative Models\" (MID) [17] (the citing number is from the submitted paper). MID similarly exploit pretrained features, i.e., CLIP, but possibly with BLIP, to calculate the text-and-image matching score; however, their method (MID) uses the statistical distribution of the features for finer evaluations than cosine similarity. The authors should compare this recent representative work to show the effectiveness and significance of the proposed method. Specifically, what would be the results of MID using BLIP features for fair comparison?\n\nW2. Does visual descriptor have significance? GPT-4 can accept images as inputs, which means instead of visual descriptions (Sec. 3.1), one may exploit the power of LLMs directly, giving a synthesized image as a prompt. Fine-grained examination of visual input would get good results rather than using cumbersome generations of visual descriptions utilizing other pretrained models. This ablation would support the argument for the core role of the proposed visual descriptor. Otherwise, this work would be reduced to a naive application of multimodal LLMs, e.g., GPT-4.\n\nW3. Robustness to error accumulations. LLM Evaluator inevitably relies on the accuracy of visual descriptions. Since there is no systemical analysis of the catastrophic degradation from an error in visual descriptions may hinder the reliability of the proposed method. \n\nI am eager to reassess my score based on the authors' faithful feedback in handling the issues raised above.\n\nQUESTIONS:\nFigure 2-4 includes the brand logo of OpenAI. According to their brand guideline (https://openai.com/brand#things-to-avoid-when-using-our-logo), the authors should not alter their logo using custom colors. Besides, the authors used the OpenAI logo to represent LLMs; however, it is inappropriate to represent LLMs with the brand logo of a for-profit company in the manuscript.\n\nLIMITATIONS:\nL1. This work decisively relies on the performance of pretrained models. 1) image captioning model BLIPv2, 2) GRiT [41] for local region descriptions, and 3) GPT-4 for LLM evaluator. How do you assess the robustness toward potential errors of visual descriptors or LLM evaluators?\n\nL2. Social bias and fairness issues can also be inherited from pretrained models. \n\nL3. Without the commercial license to use Open API, reproducibility cannot be checked or used. Could you provide the results using the open-sourced LLM evaluators?", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-16"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a new method LLMScore to evaluate the alignment between a textual prompt and the generated image (especially in the context of the text-to-image diffusion models).  The key motivation behind this work is to address the inability of popular vision-language models like CLIP to deal with the fine-grained details in a complex prompt. \nAs a result, in this paper, vision-language models such as BLIP is used to caption the image as well as provide object-centric descriptions. These are concatenated to create a final image description using an LLM. Furthermore, the alignment between the text prompt and the image description is computed using an LLM. Experiments on a variety of text-to-image generation benchmarks indicate that the proposed method LLMScore significantly outperforms prior methods of directly measuring the similarity of image and text embeddings.\n\nSTRENGTHS:\n1) The problem tackled in this paper is an important one, especially due to the inadequacy of existing models. \n2) The method proposed in the paper is also quite logical (since using an LLM would be able to perform better fine-grained understanding, as opposed to matching coarse grained features)\n3) The experimental results achieved in the paper are quite satisfactory, and show that LLMScore is able to achieve much better results as compared to the standard measures for evaluating image-text alignment.\n\nWEAKNESSES:\nOne concern about this paper is regarding the following paper [a], which could be treated as contemporary work. While it is understandable that this paper is certainly not required as a comparison as it was released less than 2 months before the submission deadline, there are a lot of similarities between [a] and LLMScore. Both rely on a vision-language model to provide descriptions of the image, and use an LLM to reason over the correctness of the generated image. Of course, there are sufficiently many differences between these 2 works ([a] first generating questions using an LLM and then answering using a VQA model, versus LLMScore generating captions/descriptions first, and then using an LLM to evaluate the description), however, it might be a good idea to atleast acknowledge [a] as a contemporary work in the related work section. \n\nRegarding the work itself, an interesting analysis to delve into (given the fine-grained and interpretable nature of LLMScore), would be to break down the errors/inaccuracies into the errors caused due to either the perception models, or with the LLMs (either in constructing the global image description or with the evaluation). Such a breakdown would also enable future work to focus on addressing the weakest aspect of LLMScore to improve it further. Unfortunately, there is no analysis regarding the failure-cases of LLMScore which would have been extremely useful for any reader. \n\n\n[a]: Hu et al. TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answering, Arxiv 2023\n\nQUESTIONS:\nI have one question regarding the reproducibility of the method. Specifically, with GPT-4, the default temperature of 0.7 is used. In my understanding, this would mean that different runs of the LLM could give different results. Therefore, it might make sense to repeat the experiments multiple times so that the stability of the results are ensured. Of course, if I am missing something, a clarification would be very useful.\n\nLIMITATIONS:\nYes, there is a limitations section in the supplementary material, which covers these issues satisfactorily.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new automatic evaluation framework named LLMScore with multi-granularity compositionality to evaluate the synthesized image quality of the mainstream text-to-image synthesis models. \nIts core approach is to tranform images into global and text descriptions through image captioning and region-to-text models, then follow by an LLM to as a visual descriptor and a text-to-image evaluator with rationable output. \nOn six image captioning benchmark datasets, the correlation between its LLMscore and human evaluation results far exceeds the existing commonly used clip-like text-image alignment indicators, reaching into a new SOTA level.\nExtensive experimental analyzes strongly confirm the usability of the method.\n\nSTRENGTHS:\n1. I truly appreciate the novelty of this paper. It dexterously transform synthesized images into text descriptions, so that it can make full use of LLM's powerful text understanding and reasoning capabilities with a series of hand-crafted constructed Prompts (including sentence analysis, text-image alignment scores estimation, and image generation error statistics). In my standpoint, compared to the previous Clip-like's text-image matching score, this paper undoubtedly set a new T2I quality evaluation paradigm. Aslo, considering the vast influence brought by the AIGC community at present, I believe the impact this paper can bring to the community is significant.\n\n2. The method of this paper is solid, not only in the multi-level extraction of image description, instruction-based and heuristic-based scoring mechanism with rationable generation, but also the considerable performance gains compared with multiple strong baseline methods is also a strong proof. \nThe selected evaluated T2I models are currently the two most popular classes - stable diffusion and DALL·E.\nIn addition, this paper gives ablation experiments of different LLMscore variants and LLMs (GPT3.5 vs GPT4), further confirming the necessity of each component.\n\n3. It brings my great joy is that this work introduces its main ideas and methods, as well as experimental settings and results, etc. in an easy-to-understand manner, so that readers can easily understand.\n\nWEAKNESSES:\nSome of the following problems in my perspective may need to solve for the future deployment: \n\n1. The scoring criteria for \"Human Annotation\" in the appendix should be more specific, and it should be able to guide the annotators to rate the quality of image-text alignment step by step. For example, what is the difference between a score of \"4\" and \"5\", is it because of the color generation of the object or the amount is biased?\nTo be honest, if the results of human annotation are not fine-grained and accurate enough, it may seriously destroy people's trust in the usability of the system. \n\n2. The experiment and analysis can be more perfect, but considering that this article is a fundamental work, the time for urgent submission of ddl may not be enough, so this amount of completion is completely acceptable to me.\n\n3. There are several different models applied in the overall evaluation pipeline. If a large model can be unified in the future to complete all evaluation steps (such as GPT4), the evaluation scheme may be more practical.\n\nQUESTIONS:\nWhat does the average rank correlation of the two items in Figure 5 refer to? Could the author provide a more clearly clarification and revise it in their next edition?\n\nLIMITATIONS:\nAs the author said in the supplementray limitation part, relying on OpenAI's closed-source GPT model and the internal bias of the LLM model are obstacles faced by this proposed evaluation framework.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new evaluation metric for text-to-image models. The method works as follows:\n- Prompt the Blip-2 VLM to generate an overall text description for the image.\n- Use GRiT to generate coordinates and language descriptions for each object in the scene\n- Use GPT4 to combine these text descriptions into an overall description of the image.\n- Prompt GPT4 to score the image, both by asking it for the overall alignment and by asking it to to count the number of errors in the presence/absence or attributes of objects in the scene.\nThey show this score correlates better with human judgment than common baselines.\n\nSTRENGTHS:\n- The approach is significantly more correlated with human judgments than baselines while still being cheaper and faster than human evaluation.\n- Evaluation is a big problem in text-to-image generation, and better quantitative metrics are useful.\n- The paper is well-written and clear, and includes a variety of strong baselines.\n- They break down the accuracy of the different types of evaluation (overall, error counting) so it's easy to see weaknesses.\n\nWEAKNESSES:\nThe paper has two stages (1) getting a visual description and (2) computing a score. They explore baselines with different variations of (2) \nbut as far as I can tell ablate different versions of Stage 1 (i.e. using just global image descriptors vs just local object descriptors vs both while using the LLMScore scoring method).\n\n(I think this would be a useful extension, but the paper is strong without it.)\n\nQUESTIONS:\nConsider mentioning similar concurrent works (e.g. TIFA https://tifa-benchmark.github.io/)\n\n\n\n- Does the image get penalized for including objects which were not specified in the prompt but make sense in context (e.g. if it asked for a plate with silverware and a cup next to it, does the model get penalized for putting them on a table? Or for including a dining room background which includes miscellaneous objects?)\n\nNit: \n- Partial phrase on line 218\n- Line 217 \"default language models\" -> \"the default language model\"\n- Table 1 caption: \"Spearmanr's\" -> \"Spearman's\"\n- Specify that in the \"error counting\" metric, lower is better (Fig 7)\n- Table 3: bold best per column.\n\nLIMITATIONS:\nThe one mentioned limitation is that error counting is still challenging.\n\nNot mentioned -- if future works optimize against scores on this metric, models could exploit errors in these metrics. (This is true of all automated metrics on this task though.)", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

comments: SUMMARY:
The authors propose an evaluation metric that focuses on the object-level compositionality and the explainability of the evaluation process.
They utilize off-the-shelf models including BLIPv2 (global captioner), GRiT (object detector & captioner) and GPT-4 (large language model).
In specific, they first get global caption using BLIPv2, and detect object boxes in an image along with local descriptions with GRiT, and fed the incorporation of these results with an evaluation instruction to GPT-4, which would then produce the score and the rationale according to the instruction. This way, the proposed metric produces assessments that are more correlated with the human judgements.

STRENGTHS:
- The proposed metric tackles an important factor in evaluating image generative models.
- The authors reveal the potential for utilizing LLMs in evaluating image generative models.

WEAKNESSES:
- The metric would work worse for image domains that are too different from MS-COCO, which is used for training GRiT. (for example, might not work for artistic images…)
- The evaluation process consists of many sub-processes, which make it prone to have error accumulation issue.
- The metric utilizes GPT-4, which is not easily accessed by many researchers.

QUESTIONS:
- How many objects are selected from GRiT? Are there any rules?
- Isn't there any score variations according to the change in an evaluation instruction?
- How much the performance is degraded when the rule-enhanced rating is not used?

LIMITATIONS:
please refer to the Weaknesses section.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07