GenEval: An object-focused framework for evaluating text-to-image alignment
Authors: Dhruba Ghosh
Hannaneh Hajishirzi
Ludwig Schmidt
Venue: NeurIPS 2023 Datasets and Benchmarks Poster (2023)
Keywords: text-to-image
vision-and-language
object detection
compositional
evaluation
framework
TL;DR: We show that object detectors and other vision models can be used for fine-grained evaluation of text-to-image models on complex generation tasks.

Abstract: Recent breakthroughs in diffusion models, multimodal pretraining, and efficient finetuning have led to an explosion of text-to-image generative models. 
Given human evaluation is expensive and difficult to scale, automated methods are critical for evaluating the increasingly large number of new models. 
However, most current automated evaluation metrics like FID or CLIPScore only offer a distribution-level measure of image quality or image-text alignment, and are unsuited for fine-grained or instance-level analysis. 
In this paper, we introduce GenEval, an object-focused framework to evaluate compositional image properties such as object co-occurrence, position, count, and color. 
We show that current object detection models can be leveraged to evaluate text-to-image models on a variety of generation tasks with strong human agreement, and that other discriminative vision models can be linked to this pipeline to further verify properties like object color. 
We then evaluate several open-source text-to-image models and analyze their relative reasoning capabilities on our benchmark. 
We find that recent models demonstrate significant improvement on these tasks, though they are still lacking in complex capabilities such as spatial relations and attribute binding. 
Finally, we demonstrate how GenEval might be used to help discover existing failure modes, in order to inform development of the next generation of text-to-image models. 
Our code to run the GenEval framework will be made publicly available at https://github.com/djghosh13/geneval.

GENEVAL: An Object-Focused Framework for Evaluating Text-to-Image Alignment
Dhruba Ghosh |
Hannaneh Hajishirzi
Ludwig Schmidt 12,33
University of Washington
2 Allen Institute for Al
LAION
denotes equal contribution
Abstract
Recent breakthroughs in diffusion models, multimodal pretraining, and efficient finetuning have led explosion of text-to-image generative models. Given human evaluation is expensive and difficult to scale, automated methods are critical for evaluating the increasingly large number new models. However, most current automated evaluation metrics like FID or CLIPScore only offer a holistic measure of image quality or image-text alignment, and are unsuited for fine-grained or instance- level analysis. In this paper, we introduce GENEVAL, object-focused framework to evaluate compositional image properties such as object co-occurrence, position, count, and color. We show that current object detection models can be leveraged to evaluate text-to-image models on variety of generation tasks with strong human agreement, and that other discriminative vision models can be linked to this pipeline to further verify properties like object color. We then evaluate several open- source text-to-inet -mage models and analyze their relative generative capabilities our benchmark. We find that recent models demonstrate significant improvement these tasks, though they still lacking in complex capabilities such spatial relations and attribute binding. Finally, we demonstrate how GENEVAL might be used to help discover existing failure modes, in order to inform development of the next generation text-to-image models. Our code to run the GENEVAL framework is publicly available at httttt:t//giiiiei httpg//huu.ccm/jjgiggaae hotp./////o/iigohh//////aaaalll o
Introduction
Text-to-image (T2I) models have exploded in popularity in recent years. After the introduction of DALL-E (33, breakthroughs in diffusion models quickly led to the development of more capable T2I models like DALL-E 2 [1] and Stable Diffusion (36). Since then, T2I models have found varied use cases in creative matters like art and research applications like training data generation. Research into parameter-efficient finetuning methods has also led to large number of new models and finetuned checkpoints of popular models. Currently, the Huggingface Hut hosts over 4,000 T21-related models and repositories.
The current gold standard for evaluating T2I models is typically human preference comparison between models, which is costly to scale up. Thus. the increase in number of T2I models makes manual evaluation inadequate. This raises the need for reliable, automated evaluation methods. However, traditional automated evaluation methods cannot analyze compositional capabilities and lack fine- lne-grained reporting. Metrics such as the Frechet Inception Distance [16] solely evaluate image quality without taking the prompt into account. Other common metrics such CLIPScore [15 rely
37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks.

---

photo
photo
[purpee baackpack and a [white [umbrella
[purple bbackpakk and [white ] [umbrella
[purple and [white [umbrrlla
[backpack
umibeella
purple
umbrolaa
yallow
backpack
purple
Text-lo-image
Object detection
Color classification
Figure I: Visualization of GENEVAL. Modern object detection models can be used to automatically verify text-to-image generations. The detected bounding boxes and segmentation masks can be used to verify object presence, count, and position, and then passed to downstream discriminative vision models to verify fine-grained object properties such color. The example image was generated by Stable Diffusion v2.1.
on alignment of image and text embeddings, which are not strongly correlated with human judgment on complex tasks, as our experiments show.
In light of this, we propose GENEVAL, an automated object-focused framework for evaluating T2I model capabilities on structured tasks (Figure [.). GENEVAL centers around the of an object detection model, which verifies that the generated image contains the objects specified in the text prompt. The bounding box information and segmentation masks returned by the model used to verify properties specified in the prompt, such as counting and relative positioning between objects. This metadata is then also passed other vision models to evaluate other properties, in our case, object color classification. Overall, this results in an interpretable and modular framework which provides fine-graid fine-grained information about T2I model capabilities.
We verify that our evaluation framework aligns with human judgment through a human evaluation study of 6,000 fine- fine-grained annotations over 1,200 images. Overall, GENEVAL attains 83% agreement with annotators about the correctness of generated images, compared to the interannotator agreement of 88%. This agreement rate boosted to 91 % images that annotators unanimously agree on, showing that our evaluation framework is highly reliable and aligns with human judgment. We also find that GENEVAL obtains significantly greater human agreement than the CLIPPScom image-text alignment metric on complex tasks that involve more compositional reasoning.
We then evaluate several popular open-source T2I image models using our GENEVAL framework. Our experiments show that the new IF model 111, with a larger text encoder and pixel-space diffusion, outperforms prior Stable Diffusion models [36.. with IF-XL correctly generating 31% of images over 50% from Stable Diffusion v2.1. The most recent Stable Diffusion XL (30), with various architecture and training changes, beats v2.1 on certain tasks like depicting multiple objects but fails to improve counting. Increasing model size leads to better performance on certain tasks, but increased pretraining time does not necessarily improve performance.
Moreover, all the tested models perform poorly on complex tasks such as relative positioning and attribute binding-win the best model generating only 15% and 35% of images correctly, there still much progress to be made text-guided image generation. This supports prior claims that spatial reasoning and attribute binding are difficult for T2I models [3] [4]. However, we demonstrate how the fine- mn-ggrineed nature GENEVAL can inform development of new models by uncovering failure modes and patterns in current image generation.
In summary, our contributions are as follows: (1) We introduce GENEVAL, an automated framework for evaluating T21 models using existing discriminative vision models; (2) we show that current object detection models are strongly aligned with human judgment and can used to evaluate varicty of compositional capabilities; and (3) evaluate several popular open-source T2I models

---

photo gless
right dog
wine
Human gold label
CLIPScore
33.8
36.7
Bost dlog
GenEval (ours)
wine glado
arinng sale
Found hot Found wine Found wine glasa right of hot dog Conrect
Found wine
Expected wine glaso right hot dog, found above hos Incorreot
Figure Comparison between GENEVAL and CLIPScore. CLIPScore returns scalar value indicating image-text alignment, whereas GENEVAL breaks the prompt down into correct and incorrect elements before producing final binary score. Compared CLIPScore, GENEVAL obtains higher agreement with human judgment complex compositional tasks.
and find significant performance improvement recent models, though there still much room to improve complex compositional tasks.
2
Related work
Text-to-image models. While T2I models have been around since 2016 22 35]. the field saw surge popularity starting with the autoregressive DALL-E in 2021 [33. followed by diffusion models like DALL-E [34], Midjoureyy [2], GLIDE (27]. and Stable Diffusion [36.. Subsequent improvements scaled the text encoder and made other architectural changes, such as in Imagen (37] IF [1I, and Stable Diffusion XI. [30]. Meanwhile, new GAN and autoregressive models are still being developed (20 45) However, most popularly used models and finetuned checkpoints for T2I are still based off of the Stable Diffusion architecture.
Automated evaluation. Automated T2I evaluation techniques primarily measure either image quality image-text alignment. The Inception Score (IS) (38) and Frechet Inception Distance (FID) [16] metrics measure image quality independent of text. FID score estimates the distance between the distribution of generated images and reference distribution of real-world images, and serves heuristic for how "realistic" the generated images are.
Other metrics evaluate image-text alignment. Many of these reference-based. BLEU (88) and CIDEr [40] can be used to evaluate generations by performing captioning and comparing to a set of reference captions 18) while R- R-Precision [29 43 measures text recall scores from a reference
dataset captions. CLIPScore [5]. on the other hand, is reference-free, and produces an alignment
measure based on the cosine similarity of the prompt and generated image CLIP [1] embeddings.
This is shown to be more strongly correlated with human judgment than previous reference- based metrics that compare generated captions with set of potential captions [15].
Human preference-base evaluation. Inspired by past work in NLP, several new methods evaluate generated images using models trained direct human preference data 21 42). This involves manual annotation of large dataset of images ranking groups of two or more images generated from the same prompt. model is then trained on this dataset predict scalar score which correlates with human preference, offering a holistic evaluation of generated images. This serves to directly measure the endgoal human preference, but combines multiple aspects of image generation, e.g., aesthetics, realism, and image-text alignment, into one number, whereas we aim to provide breakdown of exact errors in each generated image.
Object detection-based evaluation. A couple prior works also propose the use of object detection to evaluate T2I generation capabilities. Dall-Eval [9] trains a task-specific object detector for measuring each of several compositional reasoning tasks. They show that finetuning T2I models
3

---

on images generated from a 3D simulator can increase model performance on these tasks to some degree. In contrast, we find that with modern object detection models, training mns--specific detectors is not necessary to obtain strong human agreement. This allows us to improve GENEVAL. as better state-of-the-art vision models become available without need for further finetuning or reliance on training sets of synthetic 3D-rendered images.
Meanwhile, VISOR [4] is different metric which allows a thorough evaluation of relative position (spatial reasoning) capabilities of T2I models. They perform a comprehensive evaluation over all triplets of object pairs and spatial relations to provide in-depth analysis of models' spatial reasoning capabilities. In contrast, we aim to cover a greater diversity of tasks, and show that object detector outputs can also be passed to downstream models that predict individual object properties such as color, expanding the scope of evaluation.
VQA-based evaluation. A concurrent work, TIFA [9], demonstrates the usefulness of large language models (LLM) combined with visual question answering (VQA) models to perform fine- grained T2I evaluation. They use an LLM to generate atomic verification questions from the text prompts, and apply a VQA model to answer the question given the generated image. This is flexible approach which can cover a diversity of prompt types, depending on the training distribution of the LLM and VQA models. Similar recent works LLMs to evaluate generated images, either by passing in a visual description (24) or using a multimodal vision-LLM (VLLM) to directly answer questions about the image 6 44. In comparison, we find that GENEVAL outputs and failure modes are more easily interpretable, as the object detector and discriminative models produce detailed bounding box and confidence scores on a per-object basis. Furthermore, each component of our framework can be independently upgraded as better models are developed.
3 GENEVAL: Our object-focused evaluation framework
3.1 Setup
In order produce fine-graing mee-grained verification of how well generated image matches the description provided in the text prompt, we break the prompt down into the rypes of objects, their properties (such color and number), and their relations to other objects (such as relative position).
Text-to-image tasks. We focus on 6 different tasks of varying difficulty requiring various compo- sitional skills, enumerated along with their prompts in Table Here, we briefly summarize each task.
single object: rendering the specified type of object. This is the simplest task, and is trivial for the modern T2I models we test.
two object: rendering two different objects. This can be challenging in and of itself, as our benchmark results show (Table and also serves as base for more complex tasks like position and attribute binding.
counting: rendering a specific number of one type of object. T2I models may struggle with this task, even for small number of objects.
rendering an object with specific color. We show that this can be verified reliably with a zero-shot image classifier by masking out the background.
colors:
position: rendering two objects with specified positions relative to each other. Spatial understanding has been found by prior works to be challenging for T2I models findings show this is still true.
our
attribute binding: rendering two different objects with two different colors. Prior works qualitatively observe that binding attributes like color to their respective objects is difficult for T2I models, often resulting in swapping (when the colors of two objects are swapped) or leakage (when the color instead appears on background objects) [3]
Prompt generation. All our text prompts are generated from task-specific templates filled in with randomly sampled object names, colors, numbers, and relative positions. Object names are drawn from the 80 MS COCO 23 class names, with some classes renamed to remove ambiguity, e.g..

---

Task
Description
# Prompts
Template
Single object
One object
80
"a photo of a/an [OBJECT]"
Two object
Two different objects
99
"a photo of a/an [OBJECT A] and a/an
[OBJECT B]"
Counting
Specified number of an
80
"a photo of [NUMBER] [OBJECT]s"
object
Colors
One object with specified color
94
photo of a/an [COLOR] [OBJECT]"
Position
Two objects with specified
100
"a photo of a/an [OBJECT A] [RELATIVE POSITION] a/an [OBJECT B] "a photo a/an [COLOR A] [OBJECT A]
relative position
Attribute hinding
Two objects with different
100
specified colors
and a/an [COLOR B] [OBJECT B]"
Table I:
List of GENEVAL tasks. "A/an" in the templates are decided based on whether the following words starts with a vowel.
"mouse" to "computer mouse". This choice is driven by the fact that most tfate-of-the-an of-thh-art object detection models are trained on the MS COCO set of objects. Colors are taken from a list of basic color terms from Berlin-Kay basic color theory [4) For the counting task, the number is chosen to be either 2, 3, or 4. For the positiosn task, the relative position is one of "above", "below", "to the left of", or "to the right of".
3.2 Evaluation framework
Object detection. For object detection and instance segmentation, we choose the best instance segmentation model available from the MMDetection toolbox (5), Mask2Former (7] trained on MS COCO. For each image, we extract all detected objects above the default confidence threshold of 0.3. We find that when multiple objects of the same type are in image (specifically, for the counting task), the detector tends to detect an excessive number of low confidence bounding boxes. To alleviate this, we raise the minimum confidence threshold 0.9 for the counting task only. We also confirm that further pre-processing such as non-maximal suppression does not improve object detector performance these cases. Specific details on hyperparameter decisions are provided in Appendix C
For all tasks, generate intermediate score for object presence, marking whether the desired object types are present in the image. For the single object and two object task, this is also the final score, since the prompt only asks for the specified objects exist. For the counting task, we further verify that the number of detected objects matches the number specified in the prompt. For the position task,
the 2D coordinates of the detected bounding box centroids to compute relative position. We find that T2I models often generate overlapping objects; thus, we set minimum distance along each axis (proportional the bounding box dimensions) after which the objects will be classified as "left", "right", "above", or "below" one another.
Color classification. For the colors and attribute binding tasks, we use the CLIP VIT-L/14 model [I]] to classify object color. For each object, using the information from the object detector, the image is cropped to the bounding box of the object. The segmentation mask is used to replace the background pixels with solid gray background. This cropped and processed image then passed to the CLIP model, which performs zero-shot classification between prompts of the form "a photo of [COLOR] [OBJECT]" with all of the different candidate colors. We find that image cropping and background masking greatly improve the performance of the color classification model.
Scoring. For each image, the GENEVAL score is a binary classification of image correctness: whether all elements specified in the prompt were correctly rendered in the image. This score is averaged across all images generated for each task obtain task-specific scores, and then averaged across the six tasks to produce overall score for given T2I model. If an image is incorrect, the framework also produces description of how the image deviates from the expectation: whether required objects missing, or how the computed count, position, color of objects differs from the
5

---

Annotator agreement
1.0
CLIPScore
GenEval
Interannetator
0.9
0.8
0.7
0.6
0.5
Overall
Single object
Two oBject
Counting
Colors
Position
Attribute binding
Figure 3: Human study agreement results. GENEVAL obtains higher agreement with human annota- tors the more complex tasks (counting, posi tion, and attribute binding) than thresholded CLIPScore, even when the CLIPScore threshold tuned separately for each task. The difference especially significant for the position task.
prompt. This can be helpful in understanding failures of both the generative T2I models (Figure and the discriminative models used for evaluation (Figure
Measuring alignment with human judgment
perform human study to verify that the object detection-based GENEVAL aligns with human perception on machine-generated images. While the Mask2Former model attains high box AP and mask AP scores on the MS COCO validation set of real-world images [5), this needs to be confirmed for Al-generated images on our distribution of prompts. We compare against interannotator agreement as well as the CLIPScore [I5] evaluation metric, as it is also a reference- evaluation method for judging image-text alignment. Since GENEVAL returns binary correctness classification and CLIPScore returns a scalar cosine similarity score, we compare against human annotations by thresholding CLIPScore. For a fair comparison, choose the best CLIPScore threshold for each task separately.
The original CLIPScore metric from 1 uses the CLIP ViT-B/32 model from 31 to compute image and text embeddings. Since then, many improved CLIP models have been released from various sources. We test several of these models and opt to compare GENEVAL against CLIPScore with OpenCLIP ViT-H/14 [8]. which shows the highest human agreement on our annotated samples. The comparison of our tested CLIP models is enumerated in Table
Format. We conduct the study through Amazon Mechanical Turk, and gather 6,000 annotations on total of 1,200 images, with 400 images collected from each Stable Diffusion v2.1, IF-XL. and LAION-5B with CLIP retrieval. All models and the CLIP baseline are listed in Section 5.1 For each image, annotators are asked to list all objects they can recognize. This helps ground following responses, especially since Al Al-generated images may be difficult to parse. Then, for each type of object in the prompt for that image, they are asked to mark how many of that object, what primary color(s), and how realistic the objects are in the image. If there are two objects in the prompt, they are also asked about relative position between the two objects, both horizontally and vertically. Finally, they give overall score (on scale of 1-4) for how well the image matches the text prompt. screenshot of the annotation interface and further details are provided in Appendix D
The results of our human study are presented in Figure Overall, we find that the object detector and color classifier strongly correlate with human perception of the images. Across all images, GENEVAL obtains 83% agreement with human annotators, where pairwise interannotator agreement is 88%. By comparison, CLIPScore obtains 80% overall agreement with human annotators. While a threshold-tuned CLIPScore has slightly higher agreement the simpler single object and colors tasks, GENEVAL shows higher agreement the other four, more complex tasks. This
Analysis.

---

Confusing background color
Merging objects
Artistic renders
a photo cake and purple chair
a photo of two bananas
a photo of a blue cow
Stable Diffusion v
dog
Baning SakAo
Found red cake Found purple chair
Found banana bananaa Incorrect
Found no cows Incorrect
Figure 4: Failure modes GENEVAL. (Left) Holes in the object which are incorporated into the segmentation mask can mislead downstream color classification. (Center) Images with overlapping objects of the same type are difficult for object detectors. (Right) Simpler artistic renderings are diito---iitiittionn for the detector, which reduces classification accuracy.
difference is especially pronounced in the counting task, where GENEVAL shows 22 point improvement human agreement over CLIPScore.
We also perform a qualitative analysis of the successes and failure modes of GENEVAL. An example comparison between GENEVAL and CLIPScore is shown in Figure 2 When comparing images generated by two different models from the same prompt, CLIPScore may assign significantly different scores to the images with no simple way to explain the scores. In contrast, VAL outputs sequence of verifications that explain why image was marked correct or incorrect.
This also makes it easier to debug cases where GENEVAL does not match human judgment (Figure The color classifier can be confused by objects with holes, as the segmentation masks generated by our object detector may erroneously include these holes as part of the object. Other hard cases for the object detector include multiple overlapping objects of the same type, which may result in merged bounding boxes, and modes, we find that GENEVAL aligns with human judgment significantly more than the CLIPScore baseline: on the 860 examples where human scores (across annotators) are unanimous, GENEVAL obtains 91% % overall agreement, while CLIPScore obtains 87% overall agreement.
images such as clip art. Despite these occasional failure
5
Benchmarking progress in recent T2I models
5.1 Experiments
Models. We evaluate variety of open-source text-to-image models. This includes all versions of Stable Diffusion (SD) and v2 (36], the recently released SD model, the IF pixel-space diffusion models from DeepFloyd [1I, and the older model minDALL-E [22 [T). inspired by the original DALL-E model. In our primary results, we display only the latest or largest variant of each model, namely, IF-XL, 1. SDv1.5. and SD-XL 1.0. We also compare these models against a baseline of real images from LAION-5B [39]. selected using CLIP ViT-L/14 image retrieval [3]. Further details are provided in Appendix
Image generation. We evaluate the models on set of 553 prompts spanning all six tasks enumerated in Table [] For each prompt, we generate 4 images, and the GENEVAL score is averaged over all generated images. This choice is motivated by current text-to-image APIs like DALL-E 2 and Midjourney [2). which generate 4 images for prompt to allow the user more choice. Other parameters, such as image resolution, sampling steps, and sampling method, are left at their default values for each model.
7

---

Single object
Two object
Attribute binding.
Counting
Position
Overall
CLiPSScore
CLIP retrieval
0.89
0.22
0.37
0.62
0.03
0.90
0.35
27.8
0.42
minDALL-E
0.73
0.02
0.23
27.3
SDv1.5
0.97
0.38
0.35
0.76
0.04
0.06
0.43
33.5
SDv2.1
SD-XL
0.98
0.74
0.39
0.85
0.15
0.23
0.55
36.7
IF-XL
0.97
0.74
0.66
0.13
0.35
36.5
0.72
Table GENEVAL scores over the main T2I models and baseline. The recent models, IF-XL and SD-XL, display a significant improvement over previous models on challenging tasks. However, tasks like counting, position, and attribute binding still show much room for improvement. Overall scores vary by 0.01-0.02 across random seeds. The relative ordering of the evaluated T2I models consistent with CLIPScore and human annotators, except with SD-XL, and IF-XI, which CLIPScore cannot distinguish between.
5.2 Results
Overall rankings. Table 2 reports the results of evaluating the T2I models and CLIP baseline on six GENEVAL tasks. Of these models, IF-XL has the best overall performance, followed by SD-XL. While the single object and colors tasks are relatively easy for all models (as expected from the complexity of the task), there is significant performance gap between models the other four tasks. In particular, two object performance has risen in the recent IF-XL and SD-XL models, while position tion and attribute binding tasks remain difficult overall. Unsurprisingly, the oldest model, minDALL-E, performs worst overall. Interestingly, the CLIP retrieval baseline surpasses minDALL-E and even SDv1.5 the counting task, but otherwise performs poorly. This is to be expected, as many object pairings and colorings, e.g. "a photo of a blue cow", would rarely appear in real-world dataset like LAION-5B.
Effects of model scale. Figure 5 compares the GENEVAL performance across the three different scales of the DeepFloyd IF models: IF-M, IF-L, and IF-XL. All three models utilize the same T5-XXL text encoder (32), with increasingly sized rext-to-image and image upscaling modules. We observe that on the two object, counting, and especially the attribute binding tasks, we see consistent improvement model size increases. However, do not such improvement the position task. This suggests that certain T2I capabilities may not be improved by vision model scale only; different training data better text encoder may be required.
Effects of increased pretraining. SDv1 is available in five checkpoints, corresponding to con- tinued training on increasing amounts of data from LAION-5B dataset. We evaluate all five model checkpoints and report the results in Figure Surprisingly, there is little increase in model
Performance across model scale
Stable Diffusion performance
1.0
IF-M
5Dv1
IF-L
5Dv2
0.8
IF-XL
Overall Score
0.5
Score
0.6
0.4
0.4
0.2
0.3
0.0
Single
Courking
0.2
11
1.2
1.3
1.4
1.5
2.0
2.1
object
binding
Stable Diffusion version
Figure 5: (Left) Change in model performance IF model scales. Overall, GENEVAL with model size, though this does not appear to be the case for the position task. (Right) Change in model performance over successive iterations of Stable Diffusion. Despite each subsequent version having been trained for more iterations, there is no consistent performance increase from v1.1 to v1.5. In contrast, v2. which uses different text encoder, significantly improves performance.
increases

---

IF-XL Position outcomes
Attribute binding outcomes
0.5
SDv2.1
0.4
IF XXL
Perquency
0.2
Proquency
0.3
0.1
0.0
0.0
Right
Above
Swrpppd
Direction
Colors
Figure 6: Failure modes of T21 models. (Left) IF-XL displays position bias where the first mentioned object is more likely to be on the left than the right of the second object. (Right) SDv2.1 is prone to swap the colors of two objects, failing correctly bind attributes to their respective objects.
performance with increased pretraining steps. While there appears to be a noticeable step up from v1.1 to v1.2, there is further increase in overall scores after that. However, SDv2 does show significant improvement over SDv1. This may be explained by the change in text encoder: SDv2 uses OpenCLiP ViT-H/14 (8), which is larger and trained different data compared to the CLIP ViT-L/14 BI used by SDv1. SD-XL also increases scores over SDv2, though there are numerous qualitative differences in architecture and training that may contribute this improvement.
Understanding T2I model failure cases. Our GENEVAL framework produces fine-gr tin-ggained output which facilitates quantitative analysis of when and how T2I models fail. We describe two interesting patterns in Figure 6 On the position task, it appears that when IF-XI. able to generate both objects, it is biased towards placing the first object the left of the second object, and biased against placing it to the right, even when the prompts are evenly distributed among all four directions. On the attribute binding task, find that Stable Diffusion v2.1 is significantly more susceptible to color swapping (applying one of the specified colors to the wrong object) as compared to IF-XL. These examples showcase how GENEVAL can be used analyze flaws and biases in T2I generations, suggesting avenues for future improvement in image generation. Qualitative examples of these findings are shown in Appendix
Limitations
GENEVAL is primarily limited by the performance of the object detector used and the availability of discriminative vision models. note that the best object detection algorithms are still trained or finetuned on the MS-COCO dataset, which has limited number of classes defined particular level of granularity. This means that while, for example, GENEVAL can verify the number of people present in generated image, it cannot verify the number of fingers each person's 's hands. Similarly, as noted in Figure object detectors trained primarily on photos do not generalize well to visually distinct art. These constraints may be removed in the future with the development of more powerful open-vocabulary object detectors trained on a wider distribution of images 26
7 Conclusion
We introduce GENEVAL, a new object-focused framework for automated evaluation of text-to-image models. GENEVAL evaluates T2I model capabilities across a suite of compositional reasoning tasks using object detection and color classification to verify fine- ane-grained object properties. We perform human study and find that GENEVAL scores align strongly with human judgment on an instance level, beating out prior approaches measuring overall image-text alignment. We then benchmark open-source T2I models and find that while complex compositional tasks like relative position and attribute binding are difficult for current T2I models, GENEVAL can help identify failure modes to facilitate future improvement. We hope to expand GENEVAL in future work to take advantage of the wide variety discriminative vision models that have been developed, produce an even broader highly interpretable evaluation framework for text-to-image generation.

---

Acknowledgments and Disclosure of Funding
We would like to thank Achal Dave, Vivek Ramanujan, Ellen Wu, Yushi Hu, Joongwon Kim, Taylor Sorensen, Jeffrey Li, Sebastin Santy, and many other members of UW and UW NLP for their discussion and constructive feedback. We would also like to thank Hyak computing cluster at the University of Washington for providing access to computational resources for running our experiments. This work was funded in part by the DARPA MCS program through NIWC Pacific (N66001-19-2-4033 NSF 2044604 Open Philanthropy, the Allen Institute for AI, and NSF grants DMS-2134012 and CCF-2019844 as a part of NSF Institute for Foundations Machine Learning (IFML).
References
[1] httpttpp:://ppeaa.coo////es
[2] URL
com/
[3] Romain Beaumont, Clip retrieval: Easily compute clip emheddings and build a clip retrieval system with them. httts:://ithhi: wwtp://ommm0///pppeeeeeeeeee
2022.
[4] Brent Berlin and Paul Kay. Basic color terms: Their universality and evolution. CLSI Publ., 2000.
[5] Kai Chen, Jiqqi Wang, Jiangmiao Pang. Yuhang Cao, Yu Xiong. Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu, Zheng Zhang, Dazhi Cheng, Chenchen Zhu, Tianheng Cheng, Qijie Zhao, Buyu Li, Xin La, Rui Zhu, Yue Wu, Jifeng Dai, Jingdong Wang, Jianping Shi, Wanli Ouyang, Chen Change Loy, and Dahua Lin. MMDetection: Open mmlab detection toolbox and benchmark. arXiv preprint arXiv: 10.Kvv1906.07155 2019.
[6] Yixiong Chen, Li Liu, and Chris Ding. X-iqe: explainable image quality evaluation for text-to-image generation with visual large language models, 2023.
[7] Bowen Cheng, Ishan Misra, Alexander G. Schwing, Alexander Kirillov, and Rohit Girdhar. Masked- attention mask transformer for universal image segmentation. 2022.
[8] Mehdi Cherti, Romain Beaumont, Ross Wightman. Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-image learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattenn Recognition, pp. 2818 2829, 2023.
[9] Jaemin Cho, Abhay Zala, and Mohit Bansal. Dall-eval: Probing the reasoning skills and social biases of text-to-image generative transformers. 2022.
[10] Boris Dayma, Suraj Patil, Pedro Cuenca, Khalid Saifullah, Tanishq Abraham, Phuc Le Khac, Luke Melas, and Ritobrata Ghosh. Dall-e mini, 7 2021. URL http:p///gggttbb.....///////////
[11] Deep-Floyd. Deep-floddíif. URL
[12] Yuxin Fang, Quan Sun, Xinggang Wang. Tiejun Huang, Xinlong Wang, and Yue Cao. Eva-02: A visual representation for neon genesis. arXiv preprint arXiv:2303. /131, 2023.
[13] Weixi Feng, Xuehai He, Tsu-Jui Fu, Varun Jampani. Arjun Reddy Akula. Pradyumna Narayana. Sugato Basu, Xin Eric Wang, and William Yang Wang. Training- free structured diffusion guidance for composi- tional text-to-image synthesis. In The Eleventh International Conference on Learning Representations, 2023. URL https:
net wwtt/formmiinn id- wwwpuumiidduPuuig PUIgUPPUUTTT22
[14] Tejas Gokhale, Hamid Palangi, Besmira Nushi, Vibhav Vineet, Eric Horvitz, Ece Kamar, Chitta Baral, and Yezhou Yang. Benchmarking spatial relationships in text-to- to-image generation, 2022.
[15] Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. CLIPScore: a reference-free evaluation metric for image captioning. In EMNLP, 2021.
[16] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS' 17, pp. 6629-66666 6640, Red Hook. NY, USA, 2017. Curran Associates Inc. ISBN 97815 1781500669644
10

---

[17] Tobias Hinz. Stefan Heinrich, and Stefan Wermier. Semantic object accuracy for generative text-to-image synthesis. IEEE Irunsactions on pattern amalysis and machine intelligence,
144 44(3):1552-1555, 2020.
[18] Seunghoon Hong. Dingdong Yang. Jongwook Choi, and Honglak Lee. Inferring semantic layout for hierarchical lext-to-image synthesis. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.
2018.
[19] Yushi Hu, Benlin Liu, Jungo Kasai, Yizhong Wang. Mari Ostendorf, Ranjay Krishna, and Noah A Smith. Tifa: Accurate and interpretable text-to-image faithfulness evaluation with question answering. arXiv preprint 2303. /897, 2023.
[20] Minguk Kang. Jun- Yan Zhu, Richard Zhang, Jaesik Park, Eli Shechtman, Sylvain Paris, and Taesung Park. Scaling up gans for text-to-imap synthesis. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[21] Yuval Kirstain, Adam Polyak, Uriel Singer. Shahbuland Matiana, Joe Penna, and Omer Levy. Pick-a-pic: An open datasct of user preferences for text-to-image generation, 2023,
[22] Kuprel. Kuprel/min-dalle Min(dall-e) is fast, minimal port of dall-e mini to pytorch. URL https: //github. wttpttuub.cco//uureee///nne /aalll
[23] Tsung-Ye Yi Lin, Michael Maire. Serge J. Belongie, Lubomir D. Bourdev, Ross B. Girshick, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll' a r. and C. Lawrence Zitnick. Microsoft COCO: common objects in context. CoRR, abs/1405500 1405.0311 021 2014, URL httthttpp//axxti http:///xxxvvo../////// abs/ 0312
[24] Yujie Lu, Xianjun Yang. Xiujun Li, Xin Eric Wang, and William Yang Wang. Llmscore: Unveiling the power of large language models in text-to-image synthesis evaluation, 2023.
[25] Elman Mansimov, Emilio Parisotto, Jimmy Ba, and Ruslan Salakhutdinov. Generating images from captions with attention. In ICLR. 2016.
[26] Matthias Minderer, Alexey Gritsenko, Austin Stone, Maxim Neumann, Dirk Weissenborn, Alexey Doso- vitskly, Aravindh Mahendran, Anurag Arnab, Mostafa Dehghani, Zhuoran Shen, et Simple open- vocabulary object detection with vision transformers. arXiv preprint arXiv:2205.00 0223000200.00630 2022.
[27] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, llya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2112 10741, 2021.
[28] Kishore Papineni, Salim Roukos. Todd Ward, and Wei-Jing Zhu. Bleu: A method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, ACL '02, pp. 311-318, USA, 2002. Association for Computational Linguistics. doi: 10.3115/ 0073083.1731333 URI hhttps://aaii wtt.//////.........../o. 51155/1770883.111 1073135
[29] Dong Huk Park, Samanch Azadi, Xihui Liu, Trevor Darrell, and Anna Rohrbach. Benchmark for compositional text-to-image synthesis. In Thirty-fifth Conference Neural Information Processing Systems Datasets and Benchmarks Track (Round 1), 2021. URL
id=bKBhQhPeKaF
[30] Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis, 2023.
[31] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763. PMLR, 2021.
[32] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research. 21(140): 2020. URL hattp://jmlll org/ papers/ v211 wrg/papers
20-074. htm. html
[33]
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shor text-to-image generation. Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 8821-8831. PMLR, 18-24 24 Jul 2021. URL htttss://rroceeee ramesh21a. html
mlr, press/ press/vv39/ v339/

---

[34] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint 22XX::2204406125,, 2022.
[35] Scott E. Reed, Zeynep Akata, Xinchen Yan, Lajanugen Logeswaran, Bernt Schiele, and Honglak Lee. Generative adversarial text image synthesis. CoRR, abs/1605.05396 2016. URL http:://rx// org/ abs/1605. 05396
[36] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models, 2021.
[37] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S. Sara Mahdavi, Rapha Gontijo Lopes. Tim Salimans, Jonathan Ho, David J Fleet, and Mohammad Norouzi. Photorealistic lext-to-image diffusion models with deep language undorstanding, 2022.
[38] Tim Salimans, lan Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. In Proceedings of the 30th International Conference on Neural Information Processing Systems, NIPS' 16, pp. 2234-2242, Red Hook, NY, USA, 2016. Curran Associates Inc. ISBN 9781150083819.
[39] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade W Gordon, Ross Wightman, Mchdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev. LAION-5h: An open large-scale dataset for training next generation image-text models. In Thirty-sixth Conference Neural Information Processing Systems Datasets and Benchmarks Track, 2022. URL https:/7
openreview attt:/forrmmmdd-mM//44aaaaaaaa
[40] Ramakrishna Vedantam, C. Lawrence Zitnick, and Devi Parikh, Cider: Consensus-based image description evaluation. CoRR, abs/1411.5722 2014. UtLp org/ http/bb//1e0 0ates/141..55 5726
[41] Xiaoshi Wu, Yiming Hao, Keqiang Sun, Yixiong Chen, Feng Zhu, Rui Zhao, and Hongsheng Li. Human preference score v2: A solid benchmark for evaluating human preferences of text-to-image synthesis, 2023.
[42] Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. Imagereward: Learning and evaluating human preferences for text-to-image generation, 2023.
[43] Tao Xu, Pengehuan Zhang, Qiuyuan Huang. Han Zhang, Zhe Gan, Xiaolci Huang, and Xiaodong He. Attngan: Fine-grainess text to image generation with attentional generative adversarial networks. CoRR, abs/1711.1044 2017. URL /ttp://t:///arzi www.aaawwww...vaaaaeee wtg/abs///eee abs//a/1...111/////
1711, 10485
[44] Michal Yarom, Yonatan Bitton, Soravit Changpinyo, Roce Aharoni, Jonathan Herzig, Oran Lang, Eran Ofek, and Idan Szpektor. What you see is what you read? improving text-image alignment evaluation. 2023.
[45] Jiahui Yu, Yuanzhong Xu, Jing Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, Ben Hutchinson, Wei Han, Zarana Parekh, Xin Li, Han Zhang, Jason Baldridge, and Yonghui Wu. Scaling autoregressive models for content-rich text- to-image generation. Transactions Machine Learning Research, 2022. ISSN 2835-8856. URL https://opearevieeu wwt/teeumt/enneeee AFDcYJKNND Featured Certification.
12

---

Single object
Two object
Attribute
Model
Overall
Counting
Colors
Position
binding
CLIP retrieval
0.35
0.89
0.22
0.37
0.62
0.03
0.00
minDALL-E
0.23
0.73
0.11
0.12
0.37
0.02
SD v1.1
0.41
0.98
0.31
0.33
0.77
0.02
0.05
SD v1.2
0,44
0.97
0.41
0.37
0.76
0.03
0.10
SD v1.3
0.43
0.97
0.38
0.35
0.77
0.03
0.05
SD v1.4
0.42
0.98
0.36
0.35
0.73
0.01
0.07
SD v1.5
0.43
0.38
0.35
0.76
0.04
0.06
SD v2.0
0.51
0.98
0.50
0,48
0.86
0.06
0.15
SD v2.1
0.50
0.98
0.51
0.44
0.85
0.07
0.17
SD-XL
0.55
0.98
0.74
0.39
0.85
0.15
0.23
IF-M
0.52
0.97
0.53
0.53
0.84
0.10
0.14
IF-L
0.54
0.98
0.60
0.57
0.80
0.15
0.17
IF-XL
0.61
0.97
0.74
0.66
0.81
0.13
0.35
Table 3: GENEVAL scores for all models evaluated, including different versions of each model. Overall scores have a standard deviation about 0.01 across random seeds.
A Further experiments
Table shows per-task GENEVAL scores for all the models evaluate. Here, cover additional ablations and analysis.
shows
A.1 Alignment with human judgment
Figure shows percent agreement with human judgment across tasks. The agreement rates vary across tasks partly due to differing scores across tasks, i.e., easier tasks will have higher baseline agreement. Thus, Figure 7 compares Cohen's kappa scores, which range from -1 (complete disagreement) (complete agreement) and take random agreement chances into account. single object kappa for GENEVAL is approximately 0, mainly because almost all images for that task are correct, which penalizes any disagreements greatly.
A.2 Evaluation parameters
Color classification. As explained in Section 3.2, our color classification involves two preprocessing steps: cropping the image the bounding box, and masking the background replace it with gray. Table shows the individual effects of these steps on the two color-related tasks. Both cropping
Annotator kappa score
0.8
CLIPScore
0.7
GenEval
Interannotator
0.6
0.5
Cohen's
0.4
0.3
0.2
0.1
0.0
Overall
Single
Counting
Colors
Position
Attribute binding
object
object
Figure 7: Human study Cohen's kappa results. CLIPscore shows strong correlation with human judgment the single object and object tasks, but beaten by GENEVAL for the other tasks. GENEVAL obtains a kappa score of 0 on the single object task because almost all annotated images were correct, penalizing disagreements more strongly.
13

---

Attribute
Method
Crop
Mask
Colors
binding
X
0.32
0,01
x
0.37
0.33
GENEVAL
v
0.43
0.47
v
0.45
0.49
CLIPScore
0.23
0.47
Interannotator
0.55
0.65
Table 4: Cohen's kappa agreement with human annotatios for different color classification methods. Bounding box cropping and background masking both increase alignment with human judgment, especially for the attribute binding task where the presence other objects may otherwise confuse the classifier. Combining cropping with masking provides small gain over just masking.
and masking are individually useful at removing distractions (i.e. other objects in the image), which manifests most often in the attribute binding task. Combining the two gives the best human agreement.
Counting task threshold. When images contain multiple instances of the same type of object, object detector tends to predict disproportionately more bounding boxes. remedy this, we increase the minimum confidence threshold for only the counting task (Figure (8) The optimal threshold is much higher, around 0.9, and increases Cohen's 's kappa agreement greatly. from 0.37 to 0.65. Using K-fold cross-validation (k 5). confirm that this is the optimal threshold across all splits, obtaining 0.823 + 0.013 agreement with human annotations on validation splits.
Position task minimum distance. As described in Appendix C.3 there are rare cases when two objects may be generated too close together to determine their relative position. In these cases, human annotators mark the objects as neutrally/not offset from one another. We find this has an insignificant effect on overall scores, but setting small minimum distance between objects before they are classified as to the leftrrigh//boov/bellww one another does improve alignment with human judgment (Figure {). Using K-fold cross-validation (k- 5), we confirm that our choice of threshold not overfit: we obtain 0.822 0.013 agreement with human annotations on validation splits.
Counting task kappa score
Position task kappa score
0.8
0.4
Cahen's kappa (x]
0.3
Cohen''
0.2
0.2
GenEval
GenEval
Interannotator
0.1
Interannotator
CLIPScore
CLIPScore
0.0
0.0
0.3
D.4
0.5
0.6
0.7
0.8
0.9
0.00
0.05
0.10
0.15
0.20
0.25
Confidence threshold
Minimum
Figure 8: Cohen's kappa agreement with human annotators for varying evaluation hyperparameters. (Left) counting task confidence threshold. The default threshold of 0.3 is too low when multiple instance of the same object are in the image, resulting in superfluous detected objects. (Right) position task minimum distance threshold. In cases when objects are merged or very close together, they should be classified as neutral/not offset in the given direction to match human judgment.
14

---

minDALL--E
SDv1.5
SDv2,1
IF-XL
CLIP retr.
Single object photo of carrot
Two object a photo of a fork and book
Counting photo of three cups
Colors photo of purple scissors
Position photo of above sink
Attribute binding photo of white dog and blue potted plant
Figure 9: Random examples of images generated by each model.
A.3 Qualitative examples
Figure D displays sample generated images for each model for example prompt from each task. In general, all of the T2I models struggle with objects that have more complicated structure; items like forks (second row) and scissors (fourth row) are difficult even for Stable Diffusion and IF. Certain object combinations are also surprisingly difficult for the models to generate at all (fifth row), while others are easier, perhaps due to co-occurrence in the training data. Note, however, that the T2I models often do succeed in generating unseen rarely seen object pairs: for example, the CLIP retrieval from LAION-SB in row does find image of dog and potted plant, whereas Stable Diffusion (primarily trained on LAION) succeeds in doing
15

---

photo dog right of a teddy bear
photo handbag and purple bed
X
X
X
Figure 10: T2I failure modes discovered through GENEVAL. (Left) IF-XI exhibits position bias where the first object more likely to be on the left of the second object. (Right) Stable Diffusion v2.1 has tendency to swap the specified colors in the attribute binding task, or sometimes to leak the specified color into the background instead, as in the top right example.
Figure shosswws some examples of T2I failure modes described in Section 5.2. The position bias in IF-XL and the color swapping tendency of Stable Diffusion can be discovered through quantitative analysis, and confirmed through qualitative observation of the generated images. These failure modes suggest avenues of research for future targeted improvement of T21 capabilities.
B Ethics statement
The primary purpose of this work is to facilitate the development better text-to-image generative models. While we hope that such models are used for benign uses such as generating art, drafting creative ideas, or producing synthetic training data for vision models, we aeknowledge that text-to- image technology deepfakes to cause individual or widespread harm. This is potential impact down the line for any research into developing text-to-image models, and careful consideration will be required to mitigate this harm.
be used for harm. This may manifest through spreading misinformation with
Another potential issue is amplification of biases present in the evaluation method. For example, the object detector classes implicitly place importance on certain types of objects, and the detector and the color classifier encode biases present in their respective training data. However, we hope that our evaluation framework will be expanded and improved with better models, so that mitigating bias upstream, e.g., by diversifying training data and expanding the number of object classes, will also lead to more fair evaluation.
C Technical details
C.1 Prompt generation
For each task, we generate 100 random prompts and then remove duplicates, sampling uniformly (with specific exceptions) from:
Objects: 80 MS-COCO object names
Numbers: "two", "three", or "four"
Positions: "above", "below", "left of", "right of"
Colors: "red", "orange", "yellow", "green", "blue", "purple", "pink", "brown", "black", "white"
16

---

Note that "gray" is excluded from the generation colors, because we find that gray objects tend be close to black or white, which makes those prompts unbiguous. This also enables us fill in the background of cropped images with gray when passed to the color classifier, since we can exclude gray as potential classification option. Furthermore, for the color-related tasks (colors and attr bute binding) we exclude the "person" class from the list of candidate objects.
The prompt text is generated from the templates listed in Table As stated, the choice between "a" and "an" is decided based on whether the following letter is a vowel or not. Plurals are not necessarily grammatical; an "s" is simply appended the object name. Metadata about each prompt, required for evaluation, is stored in JSON format. For example:
"tag": "colors",
"include":
("class": "bicycle", "count": 1, "color": "red")
},
"prompt": photo a red bicycle"
}
C.2 Image generation
Text-to-image models and baseline. We evaluate the following models using framework:
port of DALL-E Mini [10), inspired by the original DALL-E (33) minDALL-E is similarly autoregressive but replaces the discrete VAE with a VQGAN. The generated image resolution is 256x256.
minDALL-E (22),
Stable Diffusion v1 [36,, with five iterations from v1.1 to v1.5. SDv1 is a latent space diffusion model using the pretrained CLIP VIT-I/14 text encoder. The default image resolution is 512
Stable Diffusion v2 (3). with two iterations, v2.0 and v2.1. SDv2 shares similar architecture to SDv1, but notably, uses larger OpenCLIP ViT-H/14 text encoder trained on LAION-5B. is also trained on larger images, so the default image resolution 768x768.
IF [.. with three sizes: IF-M, IF-L, IF-XL IF is a three-stage pixel space diffusion model. We exclude the third stage upscaler in our evaluation because this time, they recommend using the Stable Diffusion 4x upscaler and the IF upscaler is not available on Huggingface. There are also only two second stage sizes, M and L. so IF-XL uses the L-size stage 2 model. The output resolution of the second stage is 256x256.
Stable Diffusion XI. [30,, which massively increases the UNet backbone size from Stable Diffusion v2 and incorporates two text encoders. There is a second refinement model, which we do not use as it does not affect the composition of the image.
As well as the following real image baseline:
CLIP retrieval 3 from LAION-SB. The LAION-5B dataset is indexed with CLIP ViT- L/14, and for each prompt, select the images with the highest image-text embedding alignment. These images are not fixed to any specific resolution or aspect ratio.
Generation. For each prompt, we generate 4 images (in the case of CLIP retrieval, we select the top matches). In typical usage of T21 models, there are wide variety of parameters that might be modified to improve generations, such as sampling method, number of diffuusinn steps, prompt prefixes/suffixes. and negative prompts. We do tune these hyperparameters due the compute cost of image generation, as well as the open-ended nature of text prompts. However, some preliminary tests, such as removing the "a photo of"" prefix or adding "universal" (general purpose) negative prompt, do not seem to affect GENEVAL scores positively. We expect that careful experimentation with individual models may lead small performance gains.
C,3 Evaluation
Object detection. We use the Former with Swin-S backbone trained for instance segmen- tation available from the MMDetection toolbox from OpenMMLab, released under an Apache 2.0
17

---

license. For all tasks except for counting, we take objects with confidence above the default threshold of 0.3. For counting, we find that higher threshold of 0.9 gives the highest human agree- ment results (Figure 8 particularly due to multiple bounding boxes being generated for overlapping objects of the same type. We also experiment with non-maximal suppression (NMS) within each class but find that this does not raise human agreement, For all but the count ing task, there is also no upper limit on the number of objects. For example, for the prompt "a photo of a book and a laptop". the image is counted as correct even if there are three books and two laptops.
Relative position. The relative position between objects for the position task is evaluated from the bounding box coordinates. Prior works g 14 use simple heuristic where the relative position determined by the difference between bounding box centroids: if object centered at (I. yA) and object B is centered at (I UB), then:
B ZA
B is right of A
2B < Σλ
=
B is left of A
VB > VA
B is below A
yB < VA
=
B is above A.
However, qualitatively, some images generate with both objects overlapping or merged. In these cases, human annotators do not perceive the objects as being offset from one another (whereas with this heuristic, that is only the case if the objects are aligned pixel-perfectly). Thus, we add another term denoting the minimum threshold before two objects are considered "visibly offset" in direction. If object A has dimensions WA X A and object B has dimensions wB x hB, then with a distance threshold of we have:
TB > ZA + c(wA + WB) = B is right of
In УB VA + chh + hB)
<(wA + wn)
B is left of A
= B is below
yB VA c(ha + hb)
= B is above A.
We go with this method for evaluating relative position, and find that = 0.1 optimizes human agreement.
Color classification. We use the CLIP VIT-L/14 model from OpenAl [31] released under an MIT license, as zeroshot color classifier. We also test this with the ViT-B/32 model, but find shows lower human agreement. For each candidate object, the image is cropped down to the bounding box of the object, and the segmentation mask is used to replace all background pixels with gray. We find that this shows higher agreement with human judgment than using the whole image or not replacing the background (Table 4). For the classification, we classify between the 10 candidate colors described above in with the following class templates:
"a photo of [COLOR] [OBJECT]" "
"a photo of [COLOR] ccolrred [OBJECT]"
"a photo of [COLOR] object"
where [COLOR] is replaced with the color name and [OBJECT] is replaced with the detected object's name. The normalized prompt embeddings for each color are averaged, and the prompt embedding with the highest cosine similarity to the cropped image determines the predicted color.
CLIPScore baseline. We compare our evaluation framework against CLIPScore [15]. This boils down to the cosine similarity between the prompt and image CLIP embeddings, clipped to be at least The original paper multiplies the score by 2.5 to span a wider natural range; since this is has no real impact comparing CLIPScore values, we simply scale the similarity to be between 0 (orthogonal) and 100 (perfectly aligned).
15 used the CLIP ViT-B/32 model from 31 available at the time; since newer improved CLIP models have been developed more recently, we test several different potential models and choose the model most aligned with human judgment as a baseline. We compare the ViT-B/32 model against the larger ViT-I/14 model
31. an even larger ViT-H/14 model from OpenCIIP ( and the EVA-
02-CLIP model from [2] Table displays the results of this comparison. Overall, the OpenCLIP
18

---

CLIP
Single
Two
Attribule binding
ImageNet--B 1k zeroshot
object
object
Counting
VIT-B/32
31
0.773
0.991
0.720
0.604
0.811
0.790
0.723
0.632
VIT-L/14
31
0.760
0.985
0.692
0.786
0.719
0.753
VIT-H/14
g
0.798
0.986
0.828
0.734
0,827
0.682
0.732
0,780
1/14
0.500
GENEVAL Interamootator
0.834
0.953
0.799
0.823
0.808
0.823
0.799
0.878
0,990
0.872
0.901
0.854
0.797
0.853
Table 5: Human study agreement with CLIPScore using various CLIP models. While the OpenCLIP ViT-H/14 obtains the best overall agreement with human judgment, it is significantly worse at evaluating position, suggesting qualitative difference in training data compared to [31] In addition while EVA-02-CLIP has the highest accuracy on ImageNet, this does not appear to transfer to tasks aside from single object recognition.
ViT-H/14 agrees most with human annotators however, none of the models are consistently better or worse for CLIPScore across our tasks.
D Human study details
The human study was conducted through Amazon Mechanical Turk. Annotators were asked to answer a series of questions about displayed image, as enumerated below and shown in Figure [1] There were very limited potential participant risks, if they were be exposed to an image that was disturbing not safe for work (NSFW). However, the images used were sampled from templated prompts which were not in themselves offensive. Furthermore, all images generated by our models or retrieved by CLIP were passed through NSFW filters which would black out any potentially unsafe images.
We annotated 400 images generated from 100 randomly chosen prompts for each of Stable Diffusion v2.1, IF-XL, and CLIP retrieval from LAION-5I with annotations for each image. Depending on the prompt (which affected the questions asked), each annotation took about 1-2 minutes. Each crowdworker was paid an estimated $15 per hour. total, across all 6,000 annotations and including Mechanical Turk fees, $3,600 were spent on our human study.
D.1
Full text
The task consists general instructions and a set of questions specific to each image. For each instance of the task, the " "[OBJECT]" in each question is replaced with the names of the objects from the prompt. If the prompt consists of two types of objects (e.g.. "a photo of a bird and a skateboard"), questions and 3 refer to each type respectively. If the prompt only has type of object (e.g.. "a photo of four handbags"), question 3 and question are skipped.
Instructions. Thanks for participating in this HIT! You will view an image, potentially generated by an Al image generator. Please answer all of the following questions to the best of your abilities. For each image you may asked about:
List of objects: What objects are you able to discern in the image?
Number of objects: How many objects of particular type visible, either fully or partially?
Color of objects:
What color(s) would you consider the object(s) to be?
Realism: How realistic or structurally correct are the objects?
Relative position: What the relative position in the image of one object to another?
Caption fit: How well does the image correspond to a provided caption?
few notes:
Take look at the examples if any of the questions unclear to you.
19

---

Complete question I before moving on to other questions. If you are unable to recognize particular object, guess what is use brief description.
Additional questions may appear based your previous answers. Make answer all questions shown.
When asked about color, select at least one color that best matches the true color of the object(s).
Do not consider color when deciding how realistic an object is; focus on the shape and form compared to real life.
When asked about position, object be both above/below and the left/right of another object. Mark position from your perspective looking at the 2D image, not from the camera perspective.
The image may be just black square, in which case, you should say there no objects.
Please answer with care: Some HITs will be checked by hand, and work may be rejected if there are too many errors.
Question 1. Briefly list all the objects visible in the image, separated by commas. If there are no objects the image, write "none".
Question 2a/3a.
How many [OBJECT] s are in the image?
Question 2b/3b.
What color of [OBJECT] s in the image? Choose all that apply.
Question 2c/3c. How realistic are the [OBJECT] s in the image? (1-3) Ignore the color of the object, and focus on any visible defects in the shape or structure.
Question 4a. Is the [OBJECT B] to the left or right of the [OBJECT A]? Provide your answer as viewer of the image, from the "camera's perspective".
Question 4b. Is the [OBJECT B] to above or below the [OBJECT A]? Provide your answer as viewer of the image, from the "camera's perspective".
20

---

Annotation task
Question Briefly list objects visible in image, separated by commas. If there no objects Image, write "none". teddy bear
Question 2a: How many teddy bears are in the image?
None
2
Question 2b: What color of teddy bears are in the image? Choose all that apply.
Red
Orange
Yellow
Green
Purple
Pink
Brown
Black
White
Gray
Question Zc: How realistic are the beads bears in the image?
(1-3) Ignore the color of the object, and focus on any visible defects in the shape or structure.
Realistic or al almost fully realistic
Obvicus mistakes deformations
Completely unrealistic
Question How well the image the caption below? (1-4) "Elements" refer individual objects attributes specifled by the text.
photo of white teddy bear
The image:
Fully captures the meaning the caption.
Mostly captures meaning, but misses one many elements in the caption.
Misses severol or most of the elements in the caption.
completely
caption.
Figure 11: Mechanical Turk annotation example.
21


Reviews:
[{"review_id": "Do8aJqQHFX", "reviewer": "Reviewer_ALyh", "summary": "", "questions": "", "limitations": "This method can only test the performance of the models on the 6 predefined prompt templates in table 1. This is a clear limitation compared to CLIPScore which can test performance on every possible prompt (however it is limited to the \"correctness\" of the returned score). I think this discussion is yet missing in the paper: How well do these 6 prompts cover the set of all possible prompts that users are likely to input? How could the template prompts be extended without making the method fail? \"In the wild\", prompts that users write often look like this \"{{Prompt}}, wildlife photography, photograph, high quality, wildlife, f 1.8, soft focus, 8k, national geographic, award - winning photograph by nick nichols\" (source: https://github.com/Dalabad/stable-diffusion-prompt-templates)\n\nIn line 32 the authors claim \"current automated evaluation methods cannot analyze compositional capabilities\" but in the related work chapter list \"Dall-Eval [7] trains a task-specific object detector for\nmeasuring each of several compositional reasoning tasks.\" (line 89) and \"VISOR\" which performs \"evaluation over all triplets of object pairs and spatial relations\" (line 97) which also seems like a compositional task. The statement on line 32 seems to contradict the discussed two related works.", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "- Method will improve when improved off-the-shelf object detectors and image-text similarity models are released.\n- Interpretability, since the method provides an explanation of the detected failures.\n- Solid coverage of related work\n- The human study shows the method is more suited than existing methods.\n- Good analysis of existing T2I models using the benchmark.\n\nIn general a helpful benchmark to test certain qualities of T2I models in detail.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "6LMuO0K88l", "reviewer": "Reviewer_Ygiv", "summary": "", "questions": "", "limitations": "Another drawback that exists is that the image obtained by the generative model may have some data distribution difference from the training data for target detection. And this is actually not considered in the article, and the current model of target detection may cause performance degradation due to the difference in data distribution.", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "GENEVAL provides fine-grained reporting and analysis by evaluating the performance of text-to-image generation models on compositional tasks at the object level. This allows for a detailed analysis of failure patterns and can help identify areas for improvement.\n", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "83cMSruF2h", "reviewer": "Reviewer_nUMc", "summary": "", "questions": "", "limitations": "Not applicable.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. The proposed method is more in line with the intuition of the evaluation, and the effect is better.\n2. The proposed evaluation indicators and methods are very practical.\n3. The author systematically proposes evaluation strategies and tasks, which are very helpful for evaluating the rationality of text-generated images.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "4SsFF53IXu", "reviewer": "Reviewer_Pjnj", "summary": "", "questions": "", "limitations": "- For constant evaluation of the framework, labelling is required each time to measure the degradation of the GenEval tool. This could happen for a factor of reasons such as: bad generated content, insufficient object detector, etc.", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "- A novel framework that is more accurate than CLIP based evaluation\n- GenEval framework breakdowns the prompt into smaller elements to compute the final score on the generated content, whereas CLIP based approach computes a scalar indicating the image-text alignment\n- Method is verified by comparing the results to those of annotators which labelled the generated images", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: While the paper received mixed rating initially, the common theme of lack of public code and dataset, benchmarking for additional related baselines have been addressed by the authors and received acknowledgement from the reviewers. Based on this, I am recommending an Accept
Decision: Accept (Poster)