SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Model	Single object	Two object	Counting	Colors	Position	Attribute binding	Overall	CLIPScore	Human
CLIP retrieval	0.89	0.22	0.37	0.62	0.03	0.00	0.35	27.8	0.42
minDALL-E	0.73	0.11	0.12	0.37	0.02	0.01	0.23	27.3	—
SDv1.5	0.97	0.38	0.35	0.76	0.04	0.06	0.43	33.5	—
SDv2.1	0.98	0.51	0.44	0.85	0.07	0.17	0.50	36.2	0.57
SD-XL	0.98	0.74	0.39	0.85	0.15	0.23	0.55	36.7	—
IF-XL	0.97	0.74	0.66	0.81	0.13	0.35	0.61	36.5	0.72
Table 2: GENEVAL scores over the main T2I models and baseline. The recent models, IF-XL and SD-XL, display a significant improvement over previous models on challenging tasks. However, tasks like counting, position, and attribute binding still show much room for improvement. Overall scores vary by 0.01–0.02 across random seeds. The relative ordering of the evaluated T2I models is consistent with CLIPScore and human annotators, except with SD-XL and IF-XL, which CLIPScore cannot distinguish between.
Overall rankings. Table 2 reports the results of evaluating the T2I models and CLIP baseline on the six GENEVAL tasks. Of these models, IF-XL has the best overall performance, followed by SD-XL. While the single object and colors tasks are relatively easy for all models (as expected from the complexity of the task), there is a significant performance gap between models on the other four tasks. In particular, two object performance has risen in the recent IF-XL and SD-XL models, while position and attribute binding tasks remain difficult overall. Unsurprisingly, the oldest model, minDALL-E, performs worst overall. Interestingly, the CLIP retrieval baseline surpasses minDALL-E and even SDv1.5 on the counting task, but otherwise performs poorly. This is to be expected, as many object pairings and colorings, e.g. "a photo of a blue cow", would rarely appear in a real-world dataset like LAION-5B.
Effects of model scale. Figure 5 compares the GENEVAL performance across the three different scales of the DeepFloyd IF models: IF-M, IF-L, and IF-XL. All three models utilize the same T5-XXL text encoder [32], with increasingly sized text-to-image and image upscaling modules. We observe that on the two object, counting, and especially the attribute binding tasks, we see consistent improvement as model size increases. However, we do not see such an improvement on the position task. This suggests that certain T2I capabilities may not be improved by vision model scale only; different training data or a better text encoder may be required.
Effects of increased pretraining. SDv1 is available in five checkpoints, corresponding to continued training on increasing amounts of data from the LAION-5B dataset. We evaluate all five model checkpoints and report the results in Figure 5. Surprisingly, there is little increase in model
Figure 5: (Left) Change in model performance IF model scales. Overall, GENEVAL score increases with model size, though this does not appear to be the case for the position task. (Right) Change in model performance over successive iterations of Stable Diffusion. Despite each subsequent version having been trained for more iterations, there is no consistent performance increase from v1.1 to v1.5. In contrast, v2, which uses a different text encoder, significantly improves performance.
Figure 6: Failure modes of T2I models. (Left) IF-XL displays a position bias where the first mentioned object is more likely to be on the left than the right of the second object. (Right) SDv2.1 is prone to swap the colors of two objects, failing to correctly bind attributes to their respective objects.
performance with increased pretraining steps. While there appears to be a noticeable step up from v1.1 to v1.2, there is no further increase in overall scores after that. However, SDv2 does show significant improvement over SDv1. This may be explained by the change in text encoder: SDv2 uses OpenCLIP ViT-H/14 [8], which is larger and trained on different data compared to the CLIP ViT-L/14 [31] used by SDv1. SD-XL also increases scores over SDv2, though there are numerous qualitative differences in architecture and training that may contribute to this improvement.
Understanding T2I model failure cases. Our GENEVAL framework produces fine-grained output which facilitates quantitative analysis of when and how T2I models fail. We describe two interesting patterns in Figure 6. On the position task, it appears that when IF-XL is able to generate both objects, it is biased towards placing the first object to the left of the second object, and biased against placing it to the right, even when the prompts are evenly distributed among all four directions. On the attribute binding task, we find that Stable Diffusion v2.1 is significantly more susceptible to color swapping (applying one of the specified colors to the wrong object) as compared to IF-XL. These examples showcase how GENEVAL can be used to analyze flaws and biases in T2I generations, suggesting avenues for future improvement in image generation. Qualitative examples of these findings are shown in Appendix A.
GENEVAL is primarily limited by the performance of the object detector used and the availability of discriminative vision models. We note that the best object detection algorithms are still trained or finetuned on the MS-COCO dataset, which has a limited number of classes defined at a particular level of granularity. This means that while, for example, GENEVAL can verify the number of people present in a generated image, it cannot verify the number of fingers on each person's hands. Similarly, as noted in Figure 4, object detectors trained primarily on photos do not generalize well to visually distinct art. These constraints may be removed in the future with the development of more powerful open-vocabulary object detectors trained on a wider distribution of images [26].
We introduce GENEVAL, a new object-focused framework for automated evaluation of text-to-image models. GENEVAL evaluates T2I model capabilities across a suite of compositional reasoning tasks using object detection and color classification to verify fine-grained object properties. We perform a human study and find that GENEVAL scores align strongly with human judgment on an instance level, beating out prior approaches measuring overall image-text alignment. We then benchmark open-source T2I models and find that while complex compositional tasks like relative position and attribute binding are difficult for current T2I models, GENEVAL can help identify failure modes to facilitate future improvement. We hope to expand GENEVAL in future work to take advantage of the wide variety of discriminative vision models that have been developed, to produce an even broader highly interpretable evaluation framework for text-to-image generation.
We would like to thank Achal Dave, Vivek Ramanujan, Ellen Wu, Yushi Hu, Joongwon Kim, Taylor Sorensen, Jeffrey Li, Sebastin Santy, and many other members of UW and UW NLP for their discussion and constructive feedback. We would also like to thank Hyak computing cluster at the University of Washington for providing access to computational resources for running our experiments. This work was funded in part by the DARPA MCS program through NIWC Pacific (N66001-19-2-4031), NSF IIS-2044660, Open Philanthropy, the Allen Institute for AI, and NSF grants DMS-2134012 and CCF-2019844 as a part of NSF Institute for Foundations of Machine Learning (IFML).
[1] URL https://openai.com/dall-e-2.
[2] URL https://www.midjourney.com/.
[3] Romain Beaumont. Clip retrieval: Easily compute clip embeddings and build a clip retrieval system with them. https://github.com/rom1504/clip-retrieval, 2022.
[4] Brent Berlin and Paul Kay. Basic color terms: Their universality and evolution. CLSI Publ., 2000.
[5] Kai Chen, Jiaqi Wang, Jiangmiao Pang, Yuhang Cao, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu, Zheng Zhang, Dazhi Cheng, Chenchen Zhu, Tianheng Cheng, Qijie Zhao, Buyu Li, Xin Lu, Rui Zhu, Yue Wu, Jifeng Dai, Jingdong Wang, Jianping Shi, Wanli Ouyang, Chen Change Loy, and Dahua Lin. MMDetection: Open mmlab detection toolbox and benchmark. arXiv preprint arXiv:1906.07155, 2019.
[6] Yixiong Chen, Li Liu, and Chris Ding. X-iqe: explainable image quality evaluation for text-to-image generation with visual large language models, 2023.
[7] Bowen Cheng, Ishan Misra, Alexander G. Schwing, Alexander Kirillov, and Rohit Girdhar. Masked-attention mask transformer for universal image segmentation. 2022.
[8] Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-image learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2818-2829, 2023.
[9] Jaemin Cho, Abhay Zala, and Mohit Bansal. Dall-eval: Probing the reasoning skills and social biases of text-to-image generative transformers. 2022.
[10] Boris Dayma, Suraj Patil, Pedro Cuenca, Khalid Saifullah, Tanishq Abraham, Phuc Le Khac, Luke Melas, and Ritobrata Ghosh. Dall-e mini, 7 2021. URL https://github.com/borisdayma/dalle-mini.
[11] Deep-Floyd. Deep-floyd/if. URL https://github.com/deep-floyd/IF.
[12] Yuxin Fang, Quan Sun, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. Eva-02: A visual representation for neon genesis. arXiv preprint arXiv:2303.11331, 2023.
[13] Weixi Feng, Xuehai He, Tsu-Jui Fu, Varun Jampani, Arjun Reddy Akula, Pradyumna Narayana, Sugato Basu, Xin Eric Wang, and William Yang Wang. Training-free structured diffusion guidance for compositional text-to-image synthesis. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=PUIqjT4rzq7.
[14] Tejas Gokhale, Hamid Palangi, Besmira Nushi, Vibhav Vineet, Eric Horvitz, Ece Kamar, Chitta Baral, and Yezhou Yang. Benchmarking spatial relationships in text-to-image generation, 2022.
[15] Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. CLIPScore: a reference-free evaluation metric for image captioning. In EMNLP, 2021.
[16] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS'17, pp. 6629-6640, Red Hook, NY, USA, 2017. Curran Associates Inc. ISBN 9781510860964.
Model	Overall	Single object	Two object	Counting	Colors	Position	Attribute binding
CLIP retrieval	0.35	0.89	0.22	0.37	0.62	0.03	0.00
minDALL-E	0.23	0.73	0.11	0.12	0.37	0.02	0.01
SD v1.1	0.41	0.98	0.31	0.33	0.77	0.02	0.05
SD v1.2	0.44	0.97	0.41	0.37	0.76	0.03	0.10
SD v1.3	0.43	0.97	0.38	0.35	0.77	0.03	0.05
SD v1.4	0.42	0.98	0.36	0.35	0.73	0.01	0.07
SD v1.5	0.43	0.97	0.38	0.35	0.76	0.04	0.06
SD v2.0	0.51	0.98	0.50	0.48	0.86	0.06	0.15
SD v2.1	0.50	0.98	0.51	0.44	0.85	0.07	0.17
SD-XL	0.55	0.98	0.74	0.39	0.85	0.15	0.23
IF-M	0.52	0.97	0.53	0.53	0.84	0.10	0.14
IF-L	0.54	0.98	0.60	0.57	0.80	0.15	0.17
IF-XL	0.61	0.97	0.74	0.66	0.81	0.13	0.35
Table 3: GENEVAL scores for all models evaluated, including different versions of each model. Overall scores have a standard deviation of about 0.01 across random seeds.
A Further experiments
Table 3 shows per-task GENEVAL scores for all the models we evaluate. Here, we cover additional ablations and analysis.
A.1 Alignment with human judgment
Figure 3 shows percent agreement with human judgment across tasks. The agreement rates vary across tasks partly due to differing scores across tasks, i.e., easier tasks will have higher baseline agreement. Thus, Figure 7 compares Cohen's kappa scores, which range from  $-1$  (complete disagreement) to 1 (complete agreement) and take random agreement chances into account. single object kappa for GENEVAL is approximately 0, mainly because almost all images for that task are correct, which penalizes any disagreements greatly.
A.2 Evaluation parameters
Color classification. As explained in Section 3.2, our color classification involves two preprocessing steps: cropping the image to the bounding box, and masking out the background to replace it with gray. Table 4 shows the individual effects of these steps on the two color-related tasks. Both cropping
Figure 7: Human study Cohen's kappa results. CLIPscore shows strong correlation with human judgment on the single object and two object tasks, but is beaten out by GENEVAL for the other tasks. GENEVAL obtains a kappa score of 0 on the single object task because almost all annotated images were correct, penalizing disagreements more strongly.
Method	Crop	Mask	Colors	Attribute binding
GENEVAL	X	X	0.32	0.01
✓	X	0.37	0.33
X	✓	0.43	0.47
✓	✓	0.45	0.49
CLIPScore	—	—	0.23	0.47
Interannotator	—	—	0.55	0.65
Table 4: Cohen's kappa agreement with human annotators for different color classification methods. Bounding box cropping and background masking both increase alignment with human judgment, especially for the attribute binding task where the presence of other objects may otherwise confuse the classifier. Combining cropping with masking provides a small gain over just masking.
and masking are individually useful at removing distractions (i.e. other objects in the image), which manifests most often in the attribute binding task. Combining the two gives the best human agreement.
Counting task threshold. When images contain multiple instances of the same type of object, the object detector tends to predict disproportionately more bounding boxes. To remedy this, we increase the minimum confidence threshold for only the counting task (Figure 8). The optimal threshold is much higher, around 0.9, and increases Cohen's kappa agreement greatly, from 0.37 to 0.65. Using K-fold cross-validation  $(k = 5)$ , we confirm that this is the optimal threshold across all splits, obtaining  $0.823 \pm 0.013$  agreement with human annotations on validation splits.
Position task minimum distance. As described in Appendix C.3, there are rare cases when two objects may be generated too close together to determine their relative position. In these cases, human annotators mark the objects as neutrally/not offset from one another. We find this has an insignificant effect on overall scores, but setting a small minimum distance between objects before they are classified as to the left/right/above/below one another does improve alignment with human judgment (Figure 8). Using K-fold cross-validation  $(k = 5)$ , we confirm that our choice of threshold is not overfit: we obtain  $0.822 \pm 0.013$  agreement with human annotations on validation splits.
Figure 8: Cohen's kappa agreement with human annotators for varying evaluation hyperparameters. (Left) counting task confidence threshold. The default threshold of 0.3 is too low when multiple instance of the same object are in the image, resulting in superfluous detected objects. (Right) position task minimum distance threshold. In rare cases when objects are merged or very close together, they should be classified as neutral/not offset in the given direction to match human judgment.
Figure 9: Random examples of images generated by each model.
A.3 Qualitative examples
Figure 9 displays sample generated images for each model for an example prompt from each task. In general, all of the T2I models struggle with objects that have more complicated structure; items like forks (second row) and scissors (fourth row) are difficult even for Stable Diffusion and IF. Certain object combinations are also surprisingly difficult for the models to generate at all (fifth row), while others are easier, perhaps due to co-occurrence in the training data. Note, however, that the T2I models often do succeed in generating unseen or rarely seen object pairs: for example, the CLIP retrieval from LAION-5B in row 6 does not find an image of a dog and a potted plant, whereas Stable Diffusion (primarily trained on LAION) succeeds in doing so.
a photo of a dog right of a teddy bear
Figure 10: T2I failure modes discovered through GENEVAL. (Left) IF-XL exhibits a position bias where the first object is more likely to be on the left of the second object. (Right) Stable Diffusion v2.1 has a tendency to swap the specified colors in the attribute binding task, or sometimes to leak the specified color into the background instead, as in the top right example.
Figure 10 shows some examples of T2I failure modes as described in Section 5.2. The position bias in IF-XL and the color swapping tendency of Stable Diffusion can be discovered through quantitative analysis, and confirmed through qualitative observation of the generated images. These failure modes suggest avenues of research for future targeted improvement of T2I capabilities.
a photo of a white handbag and a purple bed
The primary purpose of this work is to facilitate the development of better text-to-image generative models. While we hope that such models are used for benign uses such as generating art, drafting creative ideas, or producing synthetic training data for vision models, we acknowledge that text-to-image technology can be used for harm. This may manifest through spreading misinformation with deepfakes to cause individual or widespread harm. This is a potential impact down the line for any research into developing text-to-image models, and careful consideration will be required to mitigate this harm.
Another potential issue is amplification of biases present in the evaluation method. For example, the object detector classes implicitly place importance on certain types of objects, and the detector and the color classifier encode biases present in their respective training data. However, we hope that our evaluation framework will be expanded and improved with better models, so that mitigating bias upstream, e.g., by diversifying training data and expanding the number of object classes, will also lead to a more fair evaluation.
C.1 Prompt generation
For each task, we generate 100 random prompts and then remove duplicates, sampling uniformly (with specific exceptions) from:
- Objects: 80 MS-COCO object names
- Numbers: "two", "three", or "four"
- Positions: “above”, “below”, “left of”, “right of”
- Colors: "red", "orange", "yellow", "green", "blue", "purple", "pink", "brown", "black", "white"
Note that "gray" is excluded from the generation colors, because we find that gray objects tend to be close to black or white, which makes those prompts ambiguous. This also enables us to fill in the background of cropped images with gray when passed to the color classifier, since we can exclude gray as a potential classification option. Furthermore, for the color-related tasks (colors and attribute binding) we exclude the "person" class from the list of candidate objects.
The prompt text is generated from the templates listed in Table 1. As stated, the choice between "a" and "an" is decided based on whether the following letter is a vowel or not. Plurals are not necessarily grammatical; an "s" is simply appended to the object name. Metadata about each prompt, required for evaluation, is stored in JSON format. For example:
{ "tag": "colors", "include": [ {"class": "bicycle", "count": 1, "color": "red"} ], "prompt": "a photo of a red bicycle"   
}
C.2 Image generation
Text-to-image models and baseline. We evaluate the following models using our framework:
- minDALL-E [22], a port of DALL-E Mini [10], inspired by the original DALL-E [33]. minDALL-E is similarly autoregressive but replaces the discrete VAE with a VQGAN. The generated image resolution is 256x256.
- Stable Diffusion v1 [36], with five iterations from v1.1 to v1.5. SDv1 is a latent space diffusion model using the pretrained CLIP ViT-L/14 text encoder. The default image resolution is 512x512.
- Stable Diffusion v2 [36], with two iterations, v2.0 and v2.1. SDv2 shares a similar architecture to SDv1, but notably, uses a larger OpenCLIP ViT-H/14 text encoder trained on LAION-5B. It is also trained on larger images, so the default image resolution is 768x768.
- IF [11], with three sizes: IF-M, IF-L, IF-XL. IF is a three-stage pixel space diffusion model. We exclude the third stage upscaler in our evaluation because at this time, they recommend using the Stable Diffusion 4x upscaler and the IF upscaler is not available on Huggingface. There are also only two second stage sizes, M and L, so IF-XL uses the L-size stage 2 model. The output resolution of the second stage is  $256 \times 256$ .
- Stable Diffusion XL [30], which massively increases the UNet backbone size from Stable Diffusion v2 and incorporates two text encoders. There is a second refinement model, which we do not use as it does not affect the composition of the image.
As well as the following real image baseline:
- CLIP retrieval [3] from LAION-5B. The LAION-5B dataset is indexed with CLIP ViT-L/14, and for each prompt, we select the images with the highest image-text embedding alignment. These images are not fixed to any specific resolution or aspect ratio.
Generation. For each prompt, we generate 4 images (in the case of CLIP retrieval, we select the top 4 matches). In typical usage of T2I models, there are a wide variety of parameters that might be modified to improve generations, such as sampling method, number of diffusion steps, prompt prefixes/suffixes, and negative prompts. We do not tune these hyperparameters due to the compute cost of image generation, as well as the open-ended nature of text prompts. However, some preliminary tests, such as removing the "a photo of" prefix or adding a "universal" (general purpose) negative prompt, do not seem to affect GENEVAL scores positively. We expect that careful experimentation with individual models may lead to small performance gains.
C.3 Evaluation
Object detection. We use the Mask2Former with Swin-S backbone trained for instance segmentation available from the MMDetection toolbox from OpenMMLab, released under an Apache 2.0
license. For all tasks except for counting, we take objects with a confidence above the default threshold of 0.3. For counting, we find that a higher threshold of 0.9 gives the highest human agreement results (Figure 8), particularly due to multiple bounding boxes being generated for overlapping objects of the same type. We also experiment with non-maximal suppression (NMS) within each class but find that this does not raise human agreement. For all but the counting task, there is also no upper limit on the number of objects. For example, for the prompt "a photo of a book and a laptop", the image is counted as correct even if there are three books and two laptops.
Relative position. The relative position between objects for the position task is evaluated from the bounding box coordinates. Prior works [9, 14] use a simple heuristic where the relative position is determined by the difference between bounding box centroids: if object A is centered at  $(x_A, y_A)$  and object B is centered at  $(x_B, y_B)$ , then:
$$
x _ {B} > x _ {A} \Longrightarrow \mathrm {B} \text {i s r i g h t o f A}
$$
$$
x _ {B} <   x _ {A} \Rightarrow \mathrm {B} \text {i s l e f t o f A}
$$
$$
y _ {B} > y _ {A} \Longrightarrow \mathrm {B} \text {i s b e l o w A}
$$
$$
y _ {B} <   y _ {A} \Rightarrow \mathrm {B} \text {i s a b o v e A}.
$$
However, qualitatively, some images generate with both objects overlapping or merged. In these cases, human annotators do not perceive the objects as being offset from one another (whereas with this heuristic, that is only the case if the objects are aligned pixel-perfectly). Thus, we add another term denoting the minimum threshold before two objects are considered "visibly offset" in a direction. If object A has dimensions  $w_{A} \times h_{A}$  and object B has dimensions  $w_{B} \times h_{B}$ , then with a distance threshold of  $c$  we have:
$$
x _ {B} > x _ {A} + c \left(w _ {A} + w _ {B}\right) \Longrightarrow \mathrm {B} \text {i s r i g h t o f A}
$$
$$
x _ {B} <   x _ {A} - c \left(w _ {A} + w _ {B}\right) \Rightarrow \mathrm {B} \text {i s l e f t o f A}
$$
$$
y _ {B} > y _ {A} + c \left(h _ {A} + h _ {B}\right) \Rightarrow \mathrm {B} \text {i s b e l o w A}
$$
$$
y _ {B} > y _ {A} - c \left(h _ {A} + h _ {B}\right) \Rightarrow \mathrm {B} \text {i s a b o v e A}.
$$
We go with this method for evaluating relative position, and find that  $c = 0.1$  optimizes human agreement.