SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 2: The correlation between automatic evaluation metrics and human rankings on text-to-image synthesis. LLMScore significantly surpass existing metrics in terms of Kendall's  $\tau$  and Spearman's  $\rho$  with  $p < 0.001$ .
Human	Metric	COCO2014	COCO2017	DrawBench	PaintSkills
τ(↑)	ρ(↑)	τ(↑)	ρ(↑)	τ(↑)	ρ(↑)	τ(↑)	ρ(↑)
Overall	CLIP	0.1971	0.2655	0.2227	0.2771	0.1530	0.2143	0.4715	0.5869
NegCLIP	0.2164	0.2905	0.2793	0.3523	0.1463	0.1999	0.4911	0.6313
BLIP-ITM	0.3252	0.4255	0.0928	0.1155	0.1044	0.1455	0.4755	0.6214
BLIP-ITC	0.3465	0.4535	0.1703	0.2121	0.1569	0.2171	0.4743	0.5864
LLMScore	0.3629	0.4612	0.3357	0.4275	0.2230	0.3023	0.5600	0.6853
Error Counting	CLIP	0.1464	0.2142	0.1888	0.2677	0.1360	0.1910	0.3052	0.2891
NegCLIP	0.2116	0.3061	0.1795	0.2581	0.1179	0.1596	0.4563	0.4908
BLIP-ITM	0.2251	0.3289	0.1137	0.1635	0.0871	0.1189	0.4622	0.4997
BLIP-ITC	0.2636	0.3739	0.1849	0.2620	0.1506	0.2029	0.6178	0.6511
LLMScore	0.2830	0.3992	0.2038	0.3027	0.2134	0.2865	0.6437	0.7325
- Overall: a general-purpose text-to-image evaluation, which applies to most existing metrics. Human annotators are required to rate the overall quality of the synthesized images in terms of matching the Text Prompt.
- Error Counting: Human annotators are required to provide the number of compositional errors in the synthesized images compared to the text prompt. The error types include: 1) compositional errors: wrong attributes (color, spatial position, shape, size, material) of the objects and wrong relationship among objects, 2) missing object errors: the objects mentioned in the text prompt are not present in the image, and 3) over-specification errors: the image hallucinates irrelevant objects in the image that are not specified in the text prompt.
The averaged inter-rater agreement is 0.62 under Krippendorff's alpha agreement measure. We provide clear guidance for human annotators, with details shown in Appendix C.
4.3 Human Correlation
In Table 1 and Table 2, we use Kendall's tau  $(\tau)$  and Spearman's rho  $(\rho)$  to measure the ranking correlation to both Overall and Error Counting human rating for compositional bench and general bench. All the model-based metric scores and human ratings are normalized to 0-1 for comparison.
Overall Results As concluded in Table 1 & 2 and Figure 5, 1), the most popular text-image alignment metric (CLIP) for text-to-image evaluations is less correlated with human ratings than
(a) Averaged Kendall's  $\tau$  Ranking Correlation with Human Ratings.
(b) Averaged Spearman's  $\rho$  Ranking Correlation with Human Ratings.
Figure 5: The rank correlation is aggregated across the compositional prompt dataset (Concept Conjunction, Attribute Binding Contrast) on the left two columns (CompBench) and the general prompt dataset (MSCOCO, DrawBench, PaintSkills) on the right two columns (GeneralBench).
expected. NegCLIP utilizes hard negatives to improve the compositionality of CLIP; 2). BLIP-based metrics (BLIP-ITM and BLIP-ITC) surpass these CLIP-based metrics (CLIP and NegCLIP). We suppose BLIP-based metrics benefits from the better object-level vision-language presentations learned by grounding tasks, compared with image-level matching learned in CLIP. 3). LLMScore is significantly better than the existing metrics with large margins.
Accurate Error Counting in the Image is Challenging As shown in Figure 5, all the metrics achieve better correlation with Overall human ratings than Error Counting human ratings given the fact that there are various error types and capture each of them requires more accurate compositional visio-linguistic understanding and generation.
Object-centric Visual Descriptions Improve Compositional Understanding In Figure 5, we show that our LLMScore achieves larger correlation gain over the text prompts on the compositional bench (e.g., counting, position, size, color, relations). This further confirms our superiority in capturing compositionality with the introduced object-centric visual descriptions.
Image Captions v.s. Visual Descriptions We consider two categories of variants for LLM-Score:
1. Text-Caption Matching: CapCLIP and CapMETEOR use the CLIP and METEOR to measure the similarity between the captions (which is the global image description in Section 3.1) of the synthesized images and text prompts.
2. Text-Description Matching: DescCLIP and DescMeteor, which use the CLIPScore [17] and METEOR [3] to calculate the similarity score between visual descriptions (in Section 3.1) and text prompts.
Figure 6: Comparison between Sentence Matching (CLIP, METEOR) and Instruction-Following Matching (LLMScore). LLMScore achieves the best averaged Kendall's  $\tau$  correlation with human ratings over the GeneralBench (MSCOCO, Draw-Becnh, and PaintSkills).
Table 3: Effects of Large Language Models. LLMScore (Overall) can obtain performance gain from GPT-4 compared with using GPT-3.5 as the evaluator. Using GPT-4 as default visual descriptors do not improve the evaluation performance when only using image caption metrics (DescCLIP, DescMeteor). All numbers are averaged on the GeneralBench (MSCOCO, DrawBecnh, and PaintSkills).
Human	LLM	DescCLIP	DescMeteor	LLMScore
τ(↑)	ρ(↑)	τ(↑)	ρ(↑)	τ(↑)	ρ(↑)
Overall	GPT-3.5	0.1479	0.1956	0.0042	0.0073	0.2480	0.3285
GPT-4	0.1128	0.1485	0.0297	0.0374	0.2793	0.3649
Error Counting	GPT-3.5	0.0467	0.0670	-0.0597	-0.0835	0.2205	0.3013
GPT-4	0.0149	0.0228	-0.1087	-0.1494	0.2131	0.2981
A red clock and a gold suitcase.
LLMScore (Overall)
Human	Overall	0.33
Error	Counting	0.70
Baseline	CLIP	0.30 ×
NegCLIP	0.36 ×
BLIP-ITM	0.99 ×
BLIP-ITC	0.45 ×
Overall Rationale
The image caption describes two red suitcases. a vintage clock on the wall, and a blue curtain in the background, but the text prompt only mentions a red clock and a gold suitcase. The alignment between the text prompt and image caption is weak.
LLMScore (Error Counting)
Error Counting Rationale
The composition errors include incorrect suitcase colors, incorrect clock color, and additional elements not mentioned in the text prompt (a second suitcase and a blue curtain).
Human	Overall	1.00
Error	Counting	0.00
Baseline	CLIP	0.24 ✘
NegCLIP	0.25 ✘
BLIP-ITM	0.40 ✘
BLIP-ITC	0.42 ✘
Figure 7: Examples showing the LLMScore captures the object-level discrepancies (TOP) and similarities (BOTTOM) between the image and the text prompt. The two text prompts are sampled from the Concept Conjunction dataset.
The caption accurately describes the image and aligns well with the text prompt, offering additional details about various elements and creating a clear picture of the room.
There are no composition errors in the caption as it accurately describes the colors, materials, and spatial positions of all relevant objects within the setting, adhering to the text prompt's focus on wooden cabinets and black appliances.
The main drawback of such a sentence-matching pipeline is that the caption metrics favor the coverage or similar language structure instead
of capturing compositional semantics. Figure 6 shows the average correlation (Kendall's  $\tau$ ) on the general bench set, LLMScore significantly outperforms these two variants, indicating that, caption metrics is not a good means to measure the alignment between image and text. Despite the advance introduced by object-centric visual descriptions, the limitations associated with using caption metrics to measure semantic similarity hinder the performance gain of text-description matching metrics DescCLIP and DescMETEOR over text-caption matching metrics CapCLIP and CapMETEOR.
Differing Large Language Models In Table 3, we compare two variants of large language models: GPT-3.5 and GPT-4. 1). For the same group (for example, GPT-4 based) visual descriptions, LLMScore is better than DescCLIP and DescMeteor, indicating that LLMs reasoning is the key component to capture the semantic relations in the visual descriptions. 2). We observe that GPT-4 based LLMScore (Overall) has an average better correlation than GPT-3.5, indicating the performance benefits from the better reasoning ability in the larger-scale language model. However, GPT-4 based LLMScore (Error Counting) is only comparable with GPT-3.5, indicating that the counting is still un-resolved in large-scale language models.
4.4 Multi-Granularity Rationale for Text-to-Image Evaluation
In Figure 7, we show two examples, our LLMScore not only produces human-correlated text-image scores but also provides a rationale for each metric value. The rationale correctly explains both the differences and similarities between the descriptions and text prompts. The model performs well in explaining the errors counted and the overall comparisons.
In this paper, we re-examine the existing model-based text-to-image metrics and propose LLMScore to evaluate text-to-image synthesis by unveiling the power of large language models. Our LLMScore can capture the multi-granularity compositionality between the synthesized images and the text prompt, producing accurate alignment scores with rationales. Our LLMScore demonstrates significantly better correlation with human scores on several datasets, paving the way for a more adaptable text-to-image evaluation, capable of following human instructions to evaluate the text-image alignment.
The framework proposed in this paper first integrates GPT-4 for text-to-image evaluation and showcases how to take advantage of the existing large-scale pre-trained models (GPT-4) for measuring the alignment between the generated images and text, we also propose a new metric, LLMScore which provides interpretable rating and well aligns with the human scores on several datasets. This work sheds light on the value of large language models on the evaluation of text-to-image synthesis, we hope it can help the future text-to-image synthesis work on improving the groundedness and compositionality, either as a reward signal or evaluation metric; our preliminary work on the interpretability of LLMScore, may have the potential to be used for explanation, controllable generation, and image editing.
One limitation of our work is that it relies on GPT, which is not free for the public, and may limit its fast plug-in capability, future work may consider replacing this component with a publicly available LLM model (e.g., LLaMA) or our in-house finetuned image captioning model. Another potential issue for this work is, since it incorporates the existing large language models, it may inherit its own biases that could propagate to the metric. The future work who considers adopting our LLMScore metric should be cautious on the specific domains to make sure no harmful biases get propagated.
The research was sponsored by the National Science Foundation under Grant No. 2048122. The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the sponsor.
[1] Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. Spice: Semantic propositional image caption evaluation. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part V 14, pages 382-398. Springer, 2016. 14
[2] Omri Avrahami, Ohad Fried, and Dani Lischinski. Blended latent diffusion. arXiv preprint arXiv:2206.02779, 2022. 14
[3] Satanjeev Banerjee and Alon Lavie. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pages 65-72, Ann Arbor, Michigan, June 2005. Association for Computational Linguistics. 3, 8
A Background
A.1 Text-conditioned Image Synthesis
The popular large-scale image generation models such as Imagen [43], DALL-E2 [37] and Stable Diffusion [39] demonstrate extraordinary generation quality, but a subtle difference in input prompt could lead to a dramatic change of semantic style, thus not directly suitable for image editing [2, 41]. Besides, the prompt-to-prompt [16] achieves image editing with cross-attention control on the observation of interaction between the pixels to the text embedding. InstructPix2Pix [4] leverages the complementary abilities of a pre-trained language model GPT-3 [5], and a pre-trained text-to-image model Stable Diffusion [38] to generate large pairs of multi-modal training data and perform image editing following human instructions. ControlNet [52] learns task-specific conditions in an end-to-end way and achieves robust control effects. Recently, growing interests [35] in computer vision focus on aligning text-to-image synthesis [23] or visual editing [53] by using human feedback. Typically, a reward model is expected to evaluate images by training on task rewards using proximal policy optimization (PPO [46]).
A.2 Image-level Evaluation Metric
CLIP-based CLIP [36] was pre-trained on large-scale image-caption pairs through contrastive learning, making it a highly versatile tool for natural language processing and computer vision applications. CLIPScore [17] measures the cosine similarity value of image and caption representations extracted from the CLIP feature extractors. Formally, CLIPScore = max(cos(fI(vi), fC(ci)), 0), where  $f_{I}, f_{C}$  is the image and caption feature extractor. CLIPScore is a reference-free metric and outperforms previous reference-based metrics like CIDEr [48] and SPICE [1]. Within the text-to-image domain, previous work also relies on the same approach to measure the alignment between the text prompt and the generated image. However, vision-and-language models exhibit deficiencies in compositional understanding and are insensitive to word orders. In [51], they show BLIP [25] and CLIP [36] only achieve random chance level understanding ability on attribution, relation, and order understanding. They furthermore propose NegCLIP to improve the original CLIP model by generating additional hard negative captions and optimizing the same contrastive objective. They show that obtaining specific and low-cost negative examples can result in significant enhancements in compositional tasks without losing existing ability.
BLIP-based BLIP [25] filters out noisy synthetic captions to effectively make use of the noisy web data through bootstrapping based on a novel multimodal mixture of Encoder-Decoder. Beyond that, BLIPv2 [24] introduces Query Transformer that bootstraps image and text representation learning and then bootstraps large language model for image-to-text generations. BLIPv2 achieves state-of-the-art performance on a wide range of understanding-based and generation-based vision-language tasks, including image-text retrieval, image captioning, and visual question answering. We suppose the grounding objective in BLIPv2's pre-training can bootstrap its performance in evaluating text-to-image synthesis. Specifically, we utilize BLIPv2 to compute the image-text matching score using "ITM" head and "ITC" head. BLIP-ITC uses a simple cosine similarity function over the extracted image and text features. In contrast, BLIP-ITM uses cross-attention to fuse multimodal features to capture fine-grained similarity.
A.3 Evaluation Datasets
The MSCOCO [27] has been widely used for object segmentation, although there is a dearth of varied prompts, indicating a lack of diversity. Winoground [47] is designed for evaluating the ability of vision and language models to conduct visio-linguistic compositional reasoning. For a pair of two distinct images, their captions are composed of identical sets of words, but in a different order. Many state-of-the-art vision and language models only achieve random chance performance, making it a good testbed for evaluation. DrawBench [42] tackles prompt diversity issues by collecting challenging descriptions for image generation. There are a set of 11 prompt categories that test various capabilities of models, including the ability to accurately depict colors, numbers of objects, spatial relations, and text, as well as more complex prompts such as long textual descriptions, rare words, and misspelled
prompts. In [8], they evaluate the visual reasoning of text-to-image models and propose PaintSkills, a diagnostic dataset and evaluation toolkit designed to measure object recognition, counting, and spatial understanding. Recent studies in compositional text-to-image synthesis[12] collect Concept Conjunction prompt dataset which focuses on two objects with different colors in the text prompt, and Attribute Binding prompt dataset that is sampled from COCO captions.
In Table B, we show the full table that includes variants of our LLMScore (CapCLIP, CapMETEOR, DescCLIP, DescMETEOR) on General Bench.
Table 4: The correlation between automatic evaluation metrics and human rankings on text-to-image synthesis. Our devised metrics LLMScore significantly surpass existing metrics in terms of Kendall's  $\tau$  and Spearmanr's  $\rho$  with  $p < 0.001$ .
Human	Metric	COCO2014	COCO2017	DrawBench	PaintSkills
τ(↑)	ρ(↑)	τ(↑)	ρ(↑)	τ(↑)	ρ(↑)	τ(↑)	ρ(↑)
Overall	CLIP	0.1971	0.2655	0.2227	0.2771	0.1530	0.2143	0.4715	0.5869
NegCLIP	0.2164	0.2905	0.2793	0.3523	0.1463	0.1999	0.4911	0.6313
BLIP-ITM	0.3252	0.4255	0.0928	0.1155	0.1044	0.1455	0.4755	0.6214
BLIP-ITC	0.3465	0.4535	0.1703	0.2121	0.1569	0.2171	0.4743	0.5864
CapCLIP	0.0263	0.0335	-0.0274	-0.0315	0.0056	0.0072	0.3035	0.3751
CapMETEOR	0.0710	0.0960	0.0512	0.0650	0.0951	0.1312	0.2315	0.3056
DescCLIP	0.1377	0.1799	0.1130	0.1424	0.1136	0.1557	0.3825	0.4917
DescMETEOR	0.1175	0.1549	0.0567	0.0702	0.0028	0.0048	0.0678	0.0827
LLMScore	0.3629	0.4612	0.3357	0.4275	0.2230	0.3023	0.5600	0.6853
Error Counting	CLIP	0.1464	0.2142	0.1888	0.2677	0.1360	0.1910	0.3052	0.2891
NegCLIP	0.2116	0.3061	0.1795	0.2581	0.1179	0.1596	0.4563	0.4908
BLIP-ITM	0.2251	0.3289	0.1137	0.1635	0.0871	0.1189	0.4622	0.4997
BLIP-ITC	0.2636	0.3739	0.1849	0.2620	0.1506	0.2029	0.6178	0.6511
CapCLIP	0.0266	0.0362	-0.0068	-0.0085	0.0544	0.0704	0.4963	0.5332
CapMETEOR	0.0822	0.1197	0.0004	0.0013	0.0173	0.0192	0.3274	0.3636
DescCLIP	0.1433	0.2145	0.0338	0.0477	-0.0039	-0.0022	0.2978	0.3289
DescMETEOR	0.1398	0.2010	-0.0829	-0.1198	-0.1348	-0.1791	0.0881	0.0924
LLMScore	0.2792	0.4006	0.2138	0.3125	0.2125	0.2839	0.6444	0.7066
For each image-text pair, we ask 2 annotators to rate the overall and error counting. We will show the details of annotation interface in Section C.1.
C.1 Human Ratings Interface
In Figure 8, we show the interface for human ratings over the image quality from two objectives, overall and error counting. Human annotators are required to rate the overall quality of the image on a scale of 1-10 and count the errors in the image on a scale of 0-9.
Given the Text Prompt "A gold chair and a red clock".
[Overall Quality]According to the Text Prompt, verify the Overall and Compositional quality of the Generated Images below by rating on a scale of 10.
Text Prompt "A gold chair and a red clock."
Rate the overall quality of the Generated Images in terms of matching the Text Prompt:
1 - Poor.
2 - Very Bad.
3-Bad:Low quality, merely aligned with the text prompt.
4-Not okay.
5 - Neutral (Leaning Negative)
6 - Neutral (Leaning Positive)
O 7-Okay
8 - Good: High quality, aligned with the text prompt.
9 - Very Good
10 - Perfect.
Figure 8: Amazon Mechanical Turk Platform. Questions Layout for Human Raters for Overall and Compositional ratings of the generated image given the text prompt.
Text Prompt "gold chair and a red clock."
[Error Counting] Provide the number of composition errors Y (scale: 0-9) in the Generated Images compared to the Text Prompt. One error should be counted for each incorrect color, spatial position, shape, size, material, or relationship among objects. If an object category mentioned in the Text Prompt is missing in the caption, count it as 4 errors. The over-specifications in the image caption should be counted as only one error::
No composition error
1 composition error.
2 composition error.
3 composition error.
4 composition error.
5 composition error.
6 composition error.
7 composition error.
8 composition error.
9 or above 9 composition error.
D Visual Descriptions
Our approach involves utilizing both global and local descriptions of an image. Initially, a general caption is generated for the image. Then, a dense caption model is employed to describe the objects in detail. This technique enables the extraction of both the overall context of the image and the specific attributes of individual objects, thereby providing a comprehensive description of the image.
Dense Caption
Segmentation + Dense Caption
Figure 9: Comparison between, detector, dense caption, and segmentation results over the synthesized image. Alternatively, an object segmentation model can be applied to identify the objects in the image more accurately.
E Example of GPT Text-to-Image Evaluation.
Stage	Example
Prompt	a light brown bathroom with a white sink and mirror.
Description	This image showcases a clean and modern bathroom with a white theme. The main focus is the white sink, which is placed on a wooden bathroom counter. The counter is brown and extends across the entire width of the image, covering the lower part of the white wall behind it. Above the sink, there is a large mirror on the wall that reflects the shower curtain as well as a silver light fixture with two light bulbs, which are switched on and provide ample light to the room. The silver faucet complements the white sink and is slightly towards the left side of the sink. A white soap dish is positioned on the right side of the counter, next to an additional sink. A silver handle can be seen on a drawer beneath the counter, near the sink. On the left side of the picture, a brown wooden door can be partially seen, while on the right side, a small part of a white toilet is visible. The overall setup of the bathroom, with its white walls, bright lights, and well-arranged counter, gives off a spacious and organized vibe.
Rating Rationale	Overall Semantics: 0.8/1.0
The alignment between the image caption and the text prompt is mostly good. The generated image caption describes a white sink, wooden counter, mirror, and other elements like a soap dish, faucet, and lights that complement the room. This is not far from the text prompt of a light brown bathroom with a white sink and mirror.
Rating Rationale	Error Counting: 1 (number of errors)
The composition error mainly lies in the description of the color of the bathroom. The text prompt mentions a light brown bathroom, but the image caption empha-sizes a white theme, with only the wooden counter and door being brown.
Table 5: Examples of GPT Text-to-Image Evaluation: the prompt is used for image generation, while the description is generated by GPT-4
We showcase examples for text-to-image evaluation in Table 5 and Table 6 on synthesized images in Figure 10. The Prompt section illustrates the initial input provided to the text-to-image model, which in this instance is a short description of a bathroom. The subsequent Description section is the multi-granularity object-centric description for the image. Here, the model describes a light-filled, modern bathroom with various elements like a white sink, mirror, soap dish, and a wooden counter. Then there are two rating settings, the first is Overall (Overall Semantics), and the second is Error Counting. Both settings evaluate the text-to-image synthesis with a score and a rationale. Here, a score of 0.8 out of 1.0 reflects a high level of correspondence between the synthesized image and the prompt, despite a
Figure 10: Left: Synthesized by Stable-diffusion-2 with text prompt "A light brown bathroom with a white sink and mirror" sampled from Attribute Binding Contrast dataset. Right: Synthesized by Stable-diffusion-2 with text prompt "A brown bird and a pink sheep." sampled from COCO.
few discrepancies. And 1 represents there are only one errors found in the synthesized image compared to the prompt.