SlowGuess/NeurIPS_2023_OCR huggingface dataset:
GENEVAL: An Object-Focused Framework for Evaluating Text-to-Image Alignment
Dhruba Ghosh<sup>1</sup>
<sup>1</sup>University of Washington
Hannaneh Hajishirzi $^{1,2}$
2Allen Institute for AI
Ludwig Schmidt $^{1,2,3}$
3LAION
* denotes equal contribution
Recent breakthroughs in diffusion models, multimodal pretraining, and efficient finetuning have led to an explosion of text-to-image generative models. Given human evaluation is expensive and difficult to scale, automated methods are critical for evaluating the increasingly large number of new models. However, most current automated evaluation metrics like FID or CLIPScore only offer a holistic measure of image quality or image-text alignment, and are unsuited for fine-grained or instance-level analysis. In this paper, we introduce GENEVAL, an object-focused framework to evaluate compositional image properties such as object co-occurrence, position, count, and color. We show that current object detection models can be leveraged to evaluate text-to-image models on a variety of generation tasks with strong human agreement, and that other discriminative vision models can be linked to this pipeline to further verify properties like object color. We then evaluate several open-source text-to-image models and analyze their relative generative capabilities on our benchmark. We find that recent models demonstrate significant improvement on these tasks, though they are still lacking in complex capabilities such as spatial relations and attribute binding. Finally, we demonstrate how GENEVAL might be used to help discover existing failure modes, in order to inform development of the next generation of text-to-image models. Our code to run the GENEVAL framework is publicly available at https://github.com/djghosh13/geneval.
Text-to-image (T2I) models have exploded in popularity in recent years. After the introduction of DALL-E [33], breakthroughs in diffusion models quickly led to the development of more capable T2I models like DALL-E 2 [1] and Stable Diffusion [36]. Since then, T2I models have found varied use cases in creative matters like art and research applications like training data generation. Research into parameter-efficient finetuning methods has also led to a large number of new models and finetuned checkpoints of popular models. Currently, the Huggingface Hub<sup>1</sup> hosts over 4,000 T2I-related models and repositories.
The current gold standard for evaluating T2I models is typically human preference comparison between models, which is costly to scale up. Thus, the increase in number of T2I models makes manual evaluation inadequate. This raises the need for reliable, automated evaluation methods. However, traditional automated evaluation methods cannot analyze compositional capabilities and lack fine-grained reporting. Metrics such as the Frechet Inception Distance [16] solely evaluate image quality without taking the prompt into account. Other common metrics such as CLIPScore [15] rely
<https://huggingface.co/models
Figure 1: Visualization of GENEVAL. Modern object detection models can be used to automatically verify text-to-image generations. The detected bounding boxes and segmentation masks can be used to verify object presence, count, and position, and then passed to downstream discriminative vision models to verify fine-grained object properties such as color. The example image was generated by Stable Diffusion v2.1.
on alignment of image and text embeddings, which are not strongly correlated with human judgment on complex tasks, as our experiments show.
In light of this, we propose GENEVAL, an automated object-focused framework for evaluating T2I model capabilities on structured tasks (Figure 1). GENEVAL centers around the use of an object detection model, which verifies that the generated image contains the objects specified in the text prompt. The bounding box information and segmentation masks returned by the model are used to verify properties specified in the prompt, such as counting and relative positioning between objects. This metadata is then also passed to other vision models to evaluate other properties, in our case, object color classification. Overall, this results in an interpretable and modular framework which provides fine-grained information about T2I model capabilities.
We verify that our evaluation framework aligns with human judgment through a human evaluation study of 6,000 fine-grained annotations over 1,200 images. Overall, GENEVAL attains  $83\%$  agreement with annotators about the correctness of generated images, compared to the interannotator agreement of  $88\%$ . This agreement rate is boosted to  $91\%$  on images that annotators unanimously agree on, showing that our evaluation framework is highly reliable and aligns with human judgment. We also find that GENEVAL obtains significantly greater human agreement than the CLIPScore image-text alignment metric on complex tasks that involve more compositional reasoning.
We then evaluate several popular open-source T2I image models using our GENEVAL framework. Our experiments show that the new IF model [11], with a larger text encoder and pixel-space diffusion, outperforms prior Stable Diffusion models [36], with IF-XL correctly generating  $61\%$  of images over  $50\%$  from Stable Diffusion v2.1. The most recent Stable Diffusion XL [30], with various architecture and training changes, beats v2.1 on certain tasks like depicting multiple objects but fails to improve at counting. Increasing model size leads to better performance on certain tasks, but increased pretraining time does not necessarily improve performance.
Moreover, all the tested models perform poorly on complex tasks such as relative positioning and attribute binding—with the best model generating only  $15\%$  and  $35\%$  of images correctly, respectively—showing there is still much progress to be made in text-guided image generation. This supports prior claims that spatial reasoning and attribute binding are difficult for T2I models [9, 13, 14]. However, we demonstrate how the fine-grained nature of GENEVAL can inform development of new models by uncovering failure modes and patterns in current image generation.
In summary, our contributions are as follows: (1) We introduce GENEVAL, an automated framework for evaluating T2I models using existing discriminative vision models; (2) we show that current object detection models are strongly aligned with human judgment and can be used to evaluate a variety of compositional capabilities; and (3) we evaluate several popular open-source T2I models
Figure 2: Comparison between GENEVAL and CLIPScore. CLIPScore returns a scalar value indicating image-text alignment, whereas GENEVAL breaks the prompt down into correct and incorrect elements before producing a final binary score. Compared to CLIPScore, GENEVAL obtains higher agreement with human judgment on complex compositional tasks.
and find significant performance improvement in recent models, though there is still much room to improve on complex compositional tasks.
Text-to-image models. While T2I models have been around since 2016 [25, 35], the field saw a surge in popularity starting with the autoregressive DALL-E in 2021 [33], followed by diffusion models like DALL-E 2 [34], Midjourney [2], GLIDE [27], and Stable Diffusion [36]. Subsequent improvements scaled the text encoder and made other architectural changes, such as in Imagen [37], IF [11], and Stable Diffusion XL [30]. Meanwhile, new GAN and autoregressive models are still being developed [20, 45]. However, most popularly used models and finetuned checkpoints for T2I are still based off of the Stable Diffusion architecture.
Automated evaluation. Automated T2I evaluation techniques primarily measure either image quality or image-text alignment. The Inception Score (IS) [38] and Frechet Inception Distance (FID) [16] metrics measure image quality independent of text. FID score estimates the distance between the distribution of generated images and a reference distribution of real-world images, and serves as a heuristic for how "realistic" the generated images are.
Other metrics evaluate image-text alignment. Many of these are reference-based: BLEU [28] and CIDEr [40] can be used to evaluate generations by performing captioning and comparing to a set of reference captions [17, 18], while R-Precision [29, 43] measures text recall scores from a reference dataset of captions. CLIPScore [15], on the other hand, is reference-free, and produces an alignment measure based on the cosine similarity of the prompt and generated image CLIP [31] embeddings. This is shown to be more strongly correlated with human judgment than previous reference-based metrics that compare generated captions with a set of potential captions [15].
Human preference-based evaluation. Inspired by past work in NLP, several new methods evaluate generated images using models trained on direct human preference data [21, 41, 42]. This involves manual annotation of a large dataset of images — ranking groups of two or more images generated from the same prompt. A model is then trained on this dataset to predict a scalar score which correlates with human preference, offering a holistic evaluation of generated images. This serves to directly measure the endgoal of human preference, but combines multiple aspects of image generation, e.g., aesthetics, realism, and image-text alignment, into one number, whereas we aim to provide a breakdown of exact errors in each generated image.
Object detection-based evaluation. A couple prior works also propose the use of object detection to evaluate T2I generation capabilities. Dall-Eval [9] trains a task-specific object detector for measuring each of several compositional reasoning tasks. They show that finetuning T2I models
on images generated from a 3D simulator can increase model performance on these tasks to some degree. In contrast, we find that with modern object detection models, training task-specific detectors is not necessary to obtain strong human agreement. This allows us to improve GENEVAL as better state-of-the-art vision models become available without need for further finetuning or reliance on training sets of synthetic 3D-rendered images.
Meanwhile, VISOR [14] is a different metric which allows a thorough evaluation of relative position (spatial reasoning) capabilities of T2I models. They perform a comprehensive evaluation over all triplets of object pairs and spatial relations to provide in-depth analysis of models' spatial reasoning capabilities. In contrast, we aim to cover a greater diversity of tasks, and show that object detector outputs can also be passed to downstream models that predict individual object properties such as color, expanding the scope of evaluation.
VQA-based evaluation. A concurrent work, TIFA [19], demonstrates the usefulness of large language models (LLM) combined with visual question answering (VQA) models to perform fine-grained T2I evaluation. They use an LLM to generate atomic verification questions from the text prompts, and apply a VQA model to answer the question given the generated image. This is a flexible approach which can cover a diversity of prompt types, depending on the training distribution of the LLM and VQA models. Similar recent works use LLMs to evaluate generated images, either by passing in a visual description [24] or using a multimodal vision-LLM (VLLM) to directly answer questions about the image [6, 44]. In comparison, we find that GENEVAL outputs and failure modes are more easily interpretable, as the object detector and discriminative models produce detailed bounding box and confidence scores on a per-object basis. Furthermore, each component of our framework can be independently upgraded as better models are developed.
3 GENEVAL: Our object-focused evaluation framework
In order to produce a fine-grained verification of how well a generated image matches the description provided in the text prompt, we break the prompt down into the types of objects, their properties (such as color and number), and their relations to other objects (such as relative position).
Text-to-image tasks. We focus on 6 different tasks of varying difficulty requiring various compositional skills, enumerated along with their prompts in Table 1. Here, we briefly summarize each task.
- single object: rendering the specified type of object. This is the simplest task, and is trivial for the modern T2I models we test.
- two object: rendering two different objects. This can be challenging in and of itself, as our benchmark results show (Table 2), and also serves as a base for more complex tasks like position and attribute binding.
- counting: rendering a specific number of one type of object. T2I models may struggle with this task, even for a small number of objects.
- colors: rendering an object with a specific color. We show that this can be verified reliably with a zero-shot image classifier by masking out the background.
- position: rendering two objects with specified positions relative to each other. Spatial understanding has been found by prior works to be challenging for T2I models [14, 9]; our findings show this is still true.
- attribute binding: rendering two different objects with two different colors. Prior works qualitatively observe that binding attributes like color to their respective objects is difficult for T2I models, often resulting in swapping (when the colors of two objects are swapped) or leakage (when the color instead appears on background objects) [13].
Prompt generation. All our text prompts are generated from task-specific templates filled in with randomly sampled object names, colors, numbers, and relative positions. Object names are drawn from the 80 MS COCO [23] class names, with some classes renamed to remove ambiguity, e.g.,
Task	Description	# Prompts	Template
Single object	One object	80	“a photo of a/an [OBJECT]”
Two object	Two different objects	99	“a photo of a/an [OBJECT A] and a/an [OBJECT B]”
Counting	Specified number of an object	80	“a photo of [NUMBER] [OBJECT] s”
Colors	One object with a specified color	94	“a photo of a/an [COLOR] [OBJECT]”
Position	Two objects with specified relative position	100	“a photo of a/an [OBJECT A] [RELATIVE POSITION] a/an [OBJECT B]”
Attribute binding	Two objects with different specified colors	100	“a photo of a/an [COLOR A] [OBJECT A] and a/an [COLOR B] [OBJECT B]”
Table 1: List of GENEVAL tasks. "A/an" in the templates are decided based on whether the following words start with a vowel.
"mouse" to "computer mouse". This choice is driven by the fact that most state-of-the-art object detection models are trained on the MS COCO set of objects. Colors are taken from a list of 11 basic color terms from Berlin-Kay basic color theory [4]. For the counting task, the number is chosen to be either 2, 3, or 4. For the position task, the relative position is one of "above", "below", "to the left of", or "to the right of".
3.2 Evaluation framework
Object detection. For object detection and instance segmentation, we choose the best instance segmentation model available from the MMDetection toolbox [5], a Mask2Former [7] trained on MS COCO. For each image, we extract all detected objects above the default confidence threshold of 0.3. We find that when multiple objects of the same type are in an image (specifically, for the counting task), the detector tends to detect an excessive number of low confidence bounding boxes. To alleviate this, we raise the minimum confidence threshold to 0.9 for the counting task only. We also confirm that further pre-processing such as non-maximal suppression does not improve object detector performance in these cases. Specific details on hyperparameter decisions are provided in Appendix C.
For all tasks, we generate an intermediate score for object presence, marking whether the desired object types are present in the image. For the single object and two object task, this is also the final score, since the prompt only asks for the specified objects to exist. For the counting task, we further verify that the number of detected objects matches the number specified in the prompt. For the position task, we use the 2D coordinates of the detected bounding box centroids to compute relative position. We find that T2I models often generate overlapping objects; thus, we set a minimum distance along each axis (proportional to the bounding box dimensions) after which the objects will be classified as "left", "right", "above", or "below" one another.
Color classification. For the colors and attribute binding tasks, we use the CLIP ViT-L/14 model [31] to classify object color. For each object, using the information from the object detector, the image is cropped to the bounding box of the object. The segmentation mask is used to replace the background pixels with a solid gray background. This cropped and processed image is then passed to the CLIP model, which performs zero-shot classification between prompts of the form "a photo of a [COLOR] [OBJECT]" with all of the different candidate colors. We find that image cropping and background masking greatly improve the performance of the color classification model.
Scoring. For each image, the GENEVAL score is a binary classification of image correctness: whether all elements specified in the prompt were correctly rendered in the image. This score is averaged across all images generated for each task to obtain task-specific scores, and then averaged across the six tasks to produce an overall score for a given T2I model. If an image is incorrect, the framework also produces a description of how the image deviates from the expectation: whether required objects are missing, or how the computed count, position, or color of objects differs from the
Figure 3: Human study agreement results. GENEVAL obtains higher agreement with human annotators on the more complex tasks (counting, position, and attribute binding) than thresholded CLIPScore, even when the CLIPScore threshold is tuned separately for each task. The difference is especially significant for the position task.
prompt. This can be helpful in understanding failures of both the generative T2I models (Figure 6) and the discriminative models used for evaluation (Figure 4).
4 Measuring alignment with human judgment
We perform a human study to verify that the object detection-based GENEVAL aligns with human perception on machine-generated images. While the Mask2Former model attains high box AP and mask AP scores on the MS COCO validation set of real-world images [5], this needs to be confirmed for AI-generated images on our distribution of prompts. We compare against interannotator agreement as well as the CLIPScore [15] evaluation metric, as it is also a reference-free evaluation method for judging image-text alignment. Since GENEVAL returns a binary correctness classification and CLIPScore returns a scalar cosine similarity score, we compare against human annotations by thresholding CLIPScore. For a fair comparison, we choose the best CLIPScore threshold for each task separately.
The original CLIPScore metric from [15] uses the CLIP ViT-B/32 model from [31] to compute image and text embeddings. Since then, many improved CLIP models have been released from various sources. We test several of these models and opt to compare GENEVAL against CLIPScore with OpenCLIP ViT-H/14 [8], which shows the highest human agreement on our annotated samples. The comparison of our tested CLIP models is enumerated in Table 5.
Format. We conduct the study through Amazon Mechanical Turk, and gather 6,000 annotations on a total of 1,200 images, with 400 images collected from each of Stable Diffusion v2.1, IF-XL, and LAION-5B with CLIP retrieval. All models and the CLIP baseline are listed in Section 5.1. For each image, annotators are asked to list all objects they can recognize. This helps ground following responses, especially since AI-generated images may be difficult to parse. Then, for each type of object in the prompt for that image, they are asked to mark how many of that object, what primary color(s), and how realistic the objects are in the image. If there are two objects in the prompt, they are also asked about the relative position between the two objects, both horizontally and vertically. Finally, they give an overall score (on a scale of 1–4) for how well the image matches the text prompt. A screenshot of the annotation interface and further details are provided in Appendix D.
Analysis. The results of our human study are presented in Figure 3. Overall, we find that the object detector and color classifier strongly correlate with human perception of the images. Across all images, GENEVAL obtains  $83\%$  agreement with human annotators, where pairwise interannotator agreement is  $88\%$ . By comparison, CLIPScore obtains  $80\%$  overall agreement with human annotators. While a threshold-tuned CLIPScore has slightly higher agreement on the simpler single object and colors tasks, GENEVAL shows higher agreement on the other four, more complex tasks. This
Confusing background color
Stable Diffusion v2.1
Found red cake
Found purple chair
$\Rightarrow$  Correct
Figure 4: Failure modes of GENEVAL. (Left) Holes in the object which are incorporated into the segmentation mask can mislead downstream color classification. (Center) Images with overlapping objects of the same type are difficult for object detectors. (Right) Simpler artistic renderings are out-of-distribution for the detector, which reduces classification accuracy.
Merging objects
IF-XL
Found banana
Found 1 banana
$\Rightarrow$  Incorrect
Artistic renders
a photo of a blue cow
CLIP retrieval
Found no cows
$\Rightarrow$  Incorrect
difference is especially pronounced in the counting task, where GENEVAL shows a 22 point improvement in human agreement over CLIPScore.
We also perform a qualitative analysis of the successes and failure modes of GENEVAL. An example comparison between GENEVAL and CLIPScore is shown in Figure 2. When comparing images generated by two different models from the same prompt, CLIPScore may assign significantly different scores to the images with no simple way to explain the scores. In contrast, GENEVAL outputs a sequence of verifications that explain why an image was marked correct or incorrect.
This also makes it easier to debug cases where GENEVAL does not match human judgment (Figure 4). The color classifier can be confused by objects with holes, as the segmentation masks generated by our object detector may erroneously include these holes as part of the object. Other hard cases for the object detector include multiple overlapping objects of the same type, which may result in merged bounding boxes, and out-of-distribution images such as clip art. Despite these occasional failure modes, we find that GENEVAL aligns with human judgment significantly more than the CLIPScore baseline: on the 860 examples where human scores (across 5 annotators) are unanimous, GENEVAL obtains  $91\%$  overall agreement, while CLIPScore obtains  $87\%$  overall agreement.
5 Benchmarking progress in recent T2I models
Models. We evaluate a variety of open-source text-to-image models. This includes all versions of Stable Diffusion (SD) v1 and v2 [36], the recently released SD XL model, the IF pixel-space diffusion models from DeepFloyd [11], and the older model minDALL-E [22, 10], inspired by the original DALL-E model. In our primary results, we display only the latest or largest variant of each model, namely, IF-XL, SDv2.1, SDv1.5, and SD-XL 1.0. We also compare these models against a baseline of real images from LAION-5B [39], selected using CLIP ViT-L/14 image retrieval [3]. Further details are provided in Appendix C.
Image generation. We evaluate the models on a set of 553 prompts spanning all six tasks enumerated in Table 1. For each prompt, we generate 4 images, and the GENEVAL score is averaged over all generated images. This choice is motivated by current text-to-image APIs like DALL-E 2 [1] and Midjourney [2], which generate 4 images for a prompt to allow the user more choice. Other parameters, such as image resolution, sampling steps, and sampling method, are left at their default values for each model.