SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yujie Lu<sup>1</sup>; Xianjun Yang<sup>1</sup>, Xiujun Li<sup>2</sup>, Xin Eric Wang<sup>3</sup>, William Yang Wang<sup>1</sup>
<sup>1</sup>University of California, Santa Barbara
$^{2}$ University of Washington  $^{3}$ University of California, Santa Cruz
https://github.com/YujieLu10/LLMScore
Existing automatic evaluation on text-to-image synthesis can only provide an image-text matching score, without considering the object-level compositionality, which results in poor correlation with human judgments. In this work, we propose LLMScore, a new framework that offers evaluation scores with multi-granularity compositionality. LLMScore leverages the large language models (LLMs) to evaluate text-to-image models. Initially, it transforms the image into image-level and object-level visual descriptions. Then an evaluation instruction is fed into the LLMs to measure the alignment between the synthesized image and the text, ultimately generating a score accompanied by a rationale. Our substantial analysis reveals the highest correlation of LLMScore with human judgments on a wide range of datasets (Attribute Binding Contrast, Concept Conjunction, MSCOCO, DrawBench, PaintSkills). Notably, our LLMScore achieves Kendall's  $\tau$  correlation with human evaluations that is  $58.8\%$  and  $31.2\%$  higher than the commonly-used text-image matching metrics CLIP and BLIP, respectively.
In recent years, research in text-to-image synthesis has made significant progress [9, 11, 38, 43]. However, evaluation metrics have lagged behind due to challenges such as accurately capturing composite text-image alignment (e.g. color, counting, location) [47], interpretably producing the score, and adaptively evaluating with various objectives.
Established evaluation metrics for text-to-image synthesis like CLIPScore [17] and BLIP [24], while widely used and highly effective [20, 36], have encountered challenges when it comes to capturing object-level alignment between text and image [12, 26]. Figure 1 illustrates an example from the Concept Conjunction dataset [12], given the text prompt "A red book and a yellow vase", the left image aligns with the text prompt, while the right image fails to generate a red book, and the correct color for the vase, also with an extra yellow flower. Human judges can make the correct and clear assessment (1.00 v.s. 0.45/0.55) of these two images on both overall and error counting objectives, while the existing metrics (CLIP, NegCLIP [51], BLIP) predicts similar scores for both images, failing to distinguish the correct image (on the left) from the wrong one (on the right). Furthermore, these metrics provide a single, non-interpretable score, obscuring the underlying reasoning behind the alignment of the synthesized images with the given text prompts. Additionally, these model-based metrics are static, unable to follow varied guidelines that prioritize different objectives of the text-to-image evaluation. For example, the evaluation can range from accessing image-level semantics (Overall) to finer object-level details (Error Counting). These issues hinder the existing metrics from aligning with human evaluations.
*Corresponding Author: yujielu10@gmail.com
Figure 1: The two images are generated using Stable-Diffusion-2 based on the text prompt sampled from the Concept Conjunction dataset. Baseline section shows the scores from the existing model-based evaluation metrics, Human section is the rating score from the human evaluation, LLMScore section is our proposed metric. The right column also shows the rationale generated by LLMScore.
Human	Overall
Error Counting
Baseline	CLIP
NegCLIP
BLIP-ITM
BLIP-ITC
LLMScore	Overall
Error Counting
1.00	0.45
1.00	0.55
0.27	0.31
0.26	0.32
0.99	1.00
0.48	0.49
1.00	0.50
1.00	0.44
LLMScore - Overall Rationale
The overall quality of the image is quite
low due to the significant discrepancies
between the objects described in the text
prompt and those portrayed in the image.
LLMScore - Error Counting Rationale
The red book from the prompt is not in the
image. The vase is described as red in the
image, while the text prompt specified a
yellow vase. Over-specification of the
yellow flowers in the image.
In this paper, we introduce LLMScore, a novel framework to evaluate text-image alignment in text-to-image synthesis by unveiling the powerful reasoning abilities of large language models (LLMs). Our inspiration stems from the human approach of measuring text-image alignment, which involves checking the correctness of objects and attributes specified in the text prompt. With the incredible text reasoning and generation ability of LLMs, LLMScore can imitate the human evaluation process to access compositionality at multi-granularity and produce alignment scores with rationales, delivering more insight into the model performance and reasons behind the scores.
To enhance the evaluation of composite text-to-image synthesis, our LLMScore elicits grounding visio-linguistic information from vision and language models and LLMs, thereby capturing multi-granularity compositionality in the text and image. Specifically, our approach leverages vision and language models to transform the image into multi-granularity (image-level and object-level) visual descriptions, which allows us to capture the compositional aspects of multiple objects in the language format. Then we concatenate these descriptions with text prompts and feed them into large language models (LLMs, for example, GPT-4 [32]) to reason the alignment between text prompts and images. While existing metrics struggle to capture compositionality, our LLMScore captures the object-level alignment between text and image, producing scores that are significantly correlated with human evaluation, complete with reasonable rationales (Figure 1). In addition, our LLMScore can adaptively follow various guidelines (overall or error counting) by simply customizing the evaluation instruction for LLMs. For example, we can access the overall objective by prompting the LLMs with the instruction "Rate the overall alignment of text prompt and image." or validate error counting objective with the instruction "How many compositional errors are in the image?". And we explicitly include guidance on error types of text-to-image models in the evaluation instruction to keep the LLM's decision deterministic. This flexibility empowers our framework as a versatile tool for a wide range of text-to-image tasks and various evaluation guidelines.
We validate the effectiveness of LLMScore through extensive experiments, demonstrating its alignment with human judgments without any demand for additional training. Our experimental setup contains state-of-the-art text-to-image models such as Stable Diffusion [38] and DALL-E [37], evaluated over diverse datasets, including prompt datasets for general purpose (MSCOCO [28], Draw-Bench [42], PaintSkills [8]) and for compositional purposes (Abstract Concept Conjunction [12], Attribute Binding Contrast [12]). Our LLMScore achieve the highest human correlation across all datasets. On compositional datasets, we achieve  $58.8\%$  and  $31.27\%$  higher Kendall's  $\tau$  over widely used metrics CLIP and BLIP respectively.
To sum up, we present LLMScore, the first attempt to unveil the power of the large language models for text-to-image evaluation, in particular, our paper makes the following contributions:
- We propose LLMScore, a new framework to evaluate the alignment between text prompts and synthesized images in text-to-image synthesis, offering scores that accurately capture multi-granularity compositionality (image-level and object-level).
- Our LLMScore produces accurate alignment scores with rationales following various evaluation instructions (overall and error counting).
Text Prompt: A red book and a yellow vase.
A red car and a white sheep.
Text-Image Matching
Figure 2: Comparison of Text-Image Matching, Sentence Matching, and our LLM-based Instruction-Following Matching pipeline for text-to-image synthesis evaluation. Our LLMScore automatically provides accurate scores and reasonable rationales for text-to-image synthesis based on text prompts, and visual descriptions following various evaluation instructions.
Instruction-Following Matching
- We validate the LLMScore on a wide range of datasets (both general purpose and compositional purpose). Our proposed LLMScore achieves the highest human correlation among the commonly used metrics (CLIP, BLIP).
Existing Automatic Metrics for Text-to-Image Synthesis Though Inception Score (IS) [45] and the Frechet Inception Distance (FID) [18] are recognized metrics for image fidelity, they do not measure how well the synthesized images align with the text prompts. Existing metrics [21, 22, 34, 44] for measuring such alignment in text-to-image synthesis generally fall into two categories: 1) text-image matching, and 2) sentence matching. For direct comparison between the text and image, the metrics typically rely on the pre-trained text-image matching models, among these, CLIP-based and BLIP-based are most common. Alternatively, the sentence-matching pipeline transforms synthesized images into captions and then measures the sentence-level similarity with the text prompts. This can be achieved by transforming the synthesized images  $v$  into free-form captions using the image captioning model BLIPv2 [25]. Then we can apply reference-based image caption metrics such as text-based CLIPScore [17] and METEOR [3] to measure the alignment.
Large Language Models as Evaluation Metrics Very recently, large language models (LLMs) [32, 33] have achieved incredible performance in evaluating natural language generation tasks [14, 29, 55]. This success stems from the LLMs' powerful reasoning and instruction-following, enabling the evaluation of diverse objectives simply by altering the prompt. In addition to language-only tasks, recent studies have demonstrated the effectiveness of eliciting vision and language reasoning abilities of LLMs by incorporating image descriptions [13, 30, 40, 49] or fusing multimodal features [31, 56]. A concurrent work TIFA [19] utilizes LLMs to generate questions for validating text-to-image faithfulness. To our best knowledge, we are the first to introduce object-centric descriptions of images into LLMs for evaluating the multi-granularity compositionality in text-to-image synthesis.
3 LLMScore
Our proposed LLMScore aims to evaluate the alignment between the generated image and the text prompt while capturing multi-granularity compositionality. As depicted in Figure 3, LLMScore has two main components: 1) LLMs As Multi-Granularity Visual Descriptor: transforming the image into multi-granularity object-centric descriptions (Section 3.1), and 2) LLMs As Text-to-Image Evaluator: feeding the evaluation instructions into LLMs, generating the score and rationale (Section 3.2).
3.1 LLMs As Multi-Granularity Visual Descriptor
As illustrated in Figure 3, we first decompose the image into two-level visual descriptions: image-level global description; and region-level local descriptions. Then we employ the LLMs to fuse them into a coherent, object-centric, multi-granularity description of the image.
Figure 3: LLMScore pipeline for Text-to-Image evaluation. This image is generated by StableDiffusion-2 using the text prompt "A red car and a white sheep," which is sampled from the Concept Conjunction dataset. I1 and I2 represent two different evaluation instruction settings: 1) Overall and 2) Error Counting. Each produced LLMScore is accompanied by a rationale.
Global Image Description The global image description is composed of image-level captions and image meta-information. Given an image, for example, in Figure 3, we first transform it into a global description using the state-of-the-art image captioning model BLIPv2, which encapsulates the primary context of the image in a single sentence. Then we concatenate it with the meta-information of the image, such as the resolution (i.e.  $512 \times 512$ ), which aids in the understanding of the location of each object, and interaction among objects.
Local Region Descriptions Global description can only offer high-level information for the image, detailed descriptions for each region are necessary to capture object-level local information. Here we utilize GRiT [50] to extract regions of interest and transform them into textual descriptions of the regions. GRiT is a model pre-trained with detection and dense caption objectives jointly on the Visual Genome dataset, which contains local fine-grained descriptions for objects in the image. Specifically, we format the description of each object as "[Object]: [Dense Caption]: [Bounding box)". As shown in Figure 3, the descriptions of all the objects are concatenated together as our local region descriptions for each image.
Object-Centric Visual Description Though local region descriptions capture dense information about objects in the image, they lose global context compared with global image descriptions, which may lack accurate interpretation of spatial and interaction relationships among objects. By incorporating both the global image description and the local region descriptions, we are able to obtain object-centric visual descriptions that capture multi-granularity compositionality, such as the attributes of the objects and relationships among the objects. Specifically, we feed the local and global descriptions into LLMs (GPT-4 [32] by default) with the template "[GLOBAL DESCRIPTION] [LOCAL DESCRIPTION] DESCRIPTION INSTRUCTION". We fill the slot [GLOBAL DESCRIPTION] with the global image description and the slot [LOCAL DESCRIPTION] with the local region description. And the hand-crafted slot DESCRIPTION INSTRUCTION ("generate object-centric description" in Figure 3) is replaced with "Based on the above information of the image, generate the object-centric visual description regarding the numerical counting, shape, color, size, location, materials of the object and the spatial and interaction relationships among the objects."
3.2 LLMs As Text-to-Image Evaluator
Large language models have demonstrated strong reasoning abilities (mathematical, coding etc.) on many complex tasks [6]. Here we employ the LLMs to measure the alignments between the generated image and text, and utilize their reasoning ability to understand the compositional attributes of objects and the complex interactions among multiple objects. Given the above-generated visual descriptions (Section 3.1), the whole evaluation process contains three steps: 1) instruction-following rating, 2) rule-enhanced rating, and 3) rationale generation.
Figure 4: Large Language Models As Text-to-Image Evaluator. By defining evaluation instructions for various evaluation objectives, large language models can follow the instructions to evaluate text-image alignment based on the aforementioned visual descriptions and text prompts.
Instruction-Following Rating As shown in Figure 4, we design evaluation instructions to guide the LLM in evaluating the image based on specific criteria, such as overall semantics, error counting, etc. The instructions are explicit and detailed, and can include specific guidance on error types (i.e. counting/shape/color/size) commonly seen in text-to-image models. This ensures the LLM's evaluation remains deterministic and is not influenced by any inherent biases in the pre-training data. The visual descriptions and the evaluation instructions are fed into the LLM to generate a score. For example, for Overall quality evaluation, the hand-crafted slot [EVALUATION INSTRUCTION] (denoted as Instruction I1/I2 in Figure 4) is replaced with "Rate the overall quality of the image in terms of matching the text prompt." Similarly, for Error Counting evaluation, the hand-crafted slot [EVALUATION INSTRUCTION] is replaced with "Provide the number of compositional errors in the image compared to the text prompt." We define the error type as the object-level difference. For example, the counting/shape/color/size difference in the image and text prompt should be counted as one error. This approach can either speed up the process of human annotators or serve as an interpretable and consistent evaluation pipeline.
Rule-Enhanced Rating While directly using LLMs to produce evaluation scores has its merits, it presents certain challenges. One notable issue for LLM is that it can only generate discrete outputs, making it challenging to produce decimal scores even when the text prompt explicitly specifies that. To mitigate this issue, we can restrict the LLMs to rate over a wider range of scale (on a scale of 1-N, where N is default set as 100) in integer and downscale to a smaller range of scale (divided by N by default) to enforce decimal. This approach provides more flexibility to capture small discrepancies compared to directly prompting with the decimal score.
To further enhance the consistency of LLMs' ratings, we propose breaking down the evaluation process into deterministic atomic tasks and employing basic heuristic rules to imitate the human evaluation process more accurately and consistently. In the first step, the LLMs receive the concatenated information (text prompt, image description, and evaluation instruction) and produce predictions for a pre-defined sequence of atomic tasks. The atomic tasks include obtaining: 1) the number of objects specified in the text prompt (X1), 2) the number of matched objects in the image (X2), 3) the number of specified attributes in the text prompt (Y1), and 4) the number of matched attributes in the text prompt (Y2). The results for these atomic tasks are more deterministic compared with high-level tasks. In the second step, we use basic heuristics rules inspired by human evaluation processes, which reason how well the objects specified in the text prompt are presented in the image descriptions and how correctly the attributes are depicted in the image. Thus we combine the results from atomic tasks and derive the final evaluation score by  $(X2 / X1) / 2 + (Y2 / Y1) / 2$ . This is inspired by recent work [7, 15] that separates the calculation in LLMs to achieve more reliable results. This approach allows us to overcome the limitations of decimal value generation and consistency in LLMs. Thus we obtain evaluation results that are more accurate, interpretable, and closer to human judgment.
Generating Rationale The score generation process involves the LLMs' understanding of various evaluation instructions, such as assessing the overall text-image alignment or precisely counting the number of errors in the image. Then the LLMs apply the understanding of the evaluation instruction
to the concatenated visual description and text prompt, generating a score that reflects the alignment between the image and the text at multi-granularity compositionality (image-level and object-level). Each score is accompanied by a rationale. To be specific, we add the explanation instruction prompt for LLMs, such as "Explain the overall rating X within one paragraph." for the overall objective and "Explain the error counting within one paragraph" for the error counting objective. As shown in Figure 4, our LLMScore generate reasonable rationales for both overall and error counting objectives. The rationales for the scores further provide insights into the LLM's decision-making process.
4.1 Experiments Setup
Datasets For general purpose evaluation (GeneralBench), we sample 200 text prompts from each dataset, including MSCOCO [27] 2014 and 2017, DrawBench [42], and PaintSkills [8]. For compositional purpose evaluation (CompBench), we sample 200 text prompts from Concept Conjunction (CC) and Attribute Binding Contrast (ABC) datasets that are designed for evaluating compositional text-to-image synthesis [12]. In total, we gather 1200 text prompts for human correlation experiments.
Text-to-Image Models For each sampled text prompt, we generate two images using two widely used text-to-image models, Stable Diffusion [38] and DALL-E[37], which demonstrate extraordinary generation quality. To be specific, we use Stable Diffusion 2.1-v from Hugging Face, and DALL-E 2 using OpenAI API in April 2023. All the images are generated at a resolution of  $512 \times 512$ . The total text-image pairs prompts used in human correlation experiments are 2400.
Baseline Metrics We consider these publicly available model-based evaluation metrics, which fall into the text-image matching pipeline (depicted in Figure 2) in evaluating text-to-image synthesis.
1. CLIP [17, 36] measures the cosine similarity of image and text prompt representations extracted from the CLIP feature extractors. This is a widely used model-based metric to measure the text-image alignment in text-to-image synthesis.
2. NegCLIP [51] uses a fine-tuned CLIP with improved compositionality understanding to measure the cosine similarity of the image and the text prompt.
3. BLIP-ITC [24, 25] uses a cosine similarity function over the extracted image and text features by BLIPv2, similarly as CLIP.
4. BLIP-ITM [24, 25] uses cross-attention to fuse multimodal features extracted by BLIPv2 to compute fine-grained similarity.
We discuss the sentence-matching pipeline as our ablations in Section 4.3. Notice that we focus on evaluating how well the synthesized images are aligned with the text prompts. Thus the widely used Inception Score (IS) and the Frechet Inception Distance (FID) for evaluating image quality are not considered in our baselines, since they do not compare the image with the text prompts.
Implementation Details We extract the global image description using the pre-trained 2.7B BLIPv2 [24] model equipped with large language model OPT [54]. We extract local region description using the dense caption model GRiT [50] with ViT-Base [10] pre-trained on COCO 2017. We obtain the object-centric visual description using GPT-4 [32] as default language models to combine the global and local information. The is by default generated by GPT-4 with The GPT-4 model is the default descriptor to combine global and local descriptions into object-centric visual descriptions and the default evaluator to produce the score with rationale. All experiments conducted with GPT models are using OpenAI API from April 2023 to May 2023 with temperature 0.7 using greedy decoding by default.
4.2 Human Ratings
For each generated text-image pair, we ask 2 human annotators to provide ratings over these synthesized images in terms of Overall quality and Error Counting:
Table 1: Composition-focused Prompt Bench. The correlation between automatic evaluation metrics and human rankings on text-to-image synthesis. LLMScore significantly surpasses existing metrics in terms of Kendall's  $\tau$  and Spearmanr's  $\rho$  with  $p < 0.001$ .
Human	Metric	Concept Conjunction	Attribute Binding Contrast
Stable Diffusion	DALLE	Stable Diffusion	DALLE
τ(↑)	ρ(↑)	τ(↑)	ρ(↑)	τ(↑)	ρ(↑)	τ(↑)	ρ(↑)
Overall	CLIP	0.1698	0.2459	-0.0049	-0.0058	0.0186	0.0320	0.0396	0.0548
NegCLIP	0.1724	0.2504	0.0682	0.0995	0.0151	0.0211	0.1145	0.1634
BLIP-ITM	0.4058	0.5618	0.3768	0.5266	0.1799	0.2559	0.1500	0.2134
BLIP-ITC	0.2378	0.3398	0.0991	0.1413	0.1982	0.2814	0.0252	0.0344
LLMScore	0.4871	0.6956	0.5167	0.7230	0.4005	0.5480	0.3955	0.5506
Error Counting	CLIP	0.2012	0.2864	-0.0782	-0.1107	0.0061	0.0071	0.0914	0.1286
NegCLIP	0.2245	0.3240	-0.0353	-0.0502	-0.0339	-0.0418	0.0796	0.1130
BLIP-ITM	0.3341	0.4561	0.1105	0.1668	0.0696	0.0968	0.1249	0.1783
BLIP-ITC	0.2210	0.3124	-0.0755	-0.1071	0.0895	0.1315	0.0533	0.0786
LLMScore	0.3779	0.5443	0.2880	0.4428	0.1863	0.2821	0.2326	0.3351