SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Tony Lee\*, Michihiro Yasunaga\*, Chenlin Meng\*1  
Yifan Mai\*, Joon Sung Park\*, Agrim Gupta\*, Yunzhi Zhang\*, Deepak Narayanan\*2  
Hannah Benita Teufel\*, Marco Bellagente\*, Minguk Kang\*, Taesung Park\*5  
Jure Leskovec\*, Jun-Yan Zhu\*6, Li Fei-Fei\*, Jiajun Wu\*7, Stefano Ermon\*1, Percy Liang\*1
$^{1}$ Stanford  ${}^{2}$ Microsoft  ${}^{3}$ Aleph Alpha  ${}^{4}$ POSTECH  ${}^{5}$ Adobe  ${}^{6}$ CMU \*Equal contribution
The stunning qualitative improvement of text-to-image models has led to their widespread attention and adoption. However, we lack a comprehensive quantitative understanding of their capabilities and risks. To fill this gap, we introduce a new benchmark, *Holistic Evaluation of Text-to-Image Models* (HEIM). Whereas previous evaluations focus mostly on image-text alignment and image quality, we identify 12 aspects, including text-image alignment, image quality, aesthetics, originality, reasoning, knowledge, bias, toxicity, fairness, robustness, multilinguality, and efficiency. We curate 62 scenarios encompassing these aspects and evaluate 26 state-of-the-art text-to-image models on this benchmark. Our results reveal that no single model excels in all aspects, with different models demonstrating different strengths. We release the generated images and human evaluation results for full transparency at https://crfm.stanford.edu/heim/latest and the code at https://github.com/stanford-crfm/helm, which is integrated with the HELM codebase [1].
In the last two years, there has been a proliferation of text-to-image models, such as DALL-E [2, 3] and Stable Diffusion [4], and many others [5, 6, 7, 8, 9, 10, 11, 12]. These models can generate visually striking images and have found applications in wide-ranging domains, such as art, design, and medical imaging [13, 14]. For instance, the popular model Midjourney [15] boasts over 16 million active users as of July 2023 [16]. Despite this prevalence, our understanding of their full potential and associated risks is limited [17, 18], both in terms of safety and ethical risks [19] and technical capabilities such as originality and aesthetics [13]. Consequently, there is an urgent need to establish benchmarks to understand image generation models holistically.
Existing benchmarks for text-to-image generation models [20, 21, 22] have limitations that hinder comprehensive model evaluation. Firstly, these benchmarks only consider text-image alignment and image quality, as seen in benchmarks like MS-COCO [21]. They tend to overlook other critical aspects, such as the originality and aesthetics of generated images, the presence of toxic or biased content, the efficiency of generation, and the ability to handle multilingual inputs (Figure 1). These aspects are vital for assessing the model's technological and societal impacts, including ethical concerns related to toxicity and bias, legal considerations such as copyright and trademark, and environmental implications like energy consumption [19]. Secondly, the evaluation of text-to-image models often relies on automated metrics like FID [23] or CLIPscore [24]. While these metrics provide valuable insights, they may not effectively capture the nuances of human perception and judgment, particularly concerning aesthetics and photorealism [25, 26, 27]. Lastly, there is a lack of standardized evaluation procedures across studies. Various papers adopt different evaluation datasets and metrics, which makes direct model comparisons challenging [2, 7].
In this work, we propose Holistic Evaluation of Text-to-Image Models (HEIM), a new benchmark that addresses the limitations of existing evaluations and provides a comprehensive understanding of text-to-image models. (1) HEIM evaluates text-to-image models across diverse aspects. We identify
Figure 1: Overview of our Holistic Evaluation of Text-to-Image Models (HEIM). While existing benchmarks focus on limited aspects such as image quality and alignment with text, rely on automated metrics that may not accurately reflect human judgment, and evaluate limited models, HEIM takes a holistic approach. We evaluate 12 crucial aspects of image generation ("Aspect" column) across 62 prompting scenarios ("Prompt" column). Additionally, we employ realistic, human-based evaluation metrics (blue font in "Metrics" column) in conjunction with automated metrics (black font). Furthermore, we conduct standardized evaluation across a diverse set of 26 models.
12 important aspects: text-image alignment, image quality (realism), aesthetics, originality, reasoning, knowledge, bias, toxicity, fairness, robustness, multilinguality, and efficiency (Figure 1), which are crucial to technological advancement and societal impact ( $\S 3$ ). To evaluate model performance across these aspects, we curate a diverse collection of 62 scenarios, which are datasets of prompts (Table 2), and 25 metrics, which are measurements used to assess the quality of generated images specific to each aspect (Table 3). (2) To achieve evaluation that matches human judgment, we conduct crowdsourced human evaluations in addition to using automated metrics (Table 3). Incorporating human evaluations captures criteria important to humans when assessing images, providing a comprehensive understanding of how these models meet human expectations. (3) Finally, we conduct standardized model comparisons. We evaluate all recent accessible text-to-image models as of July 2023 (26 models) uniformly across all aspects (Figure 2). By adopting a standardized evaluation framework, we offer holistic insights into model performance, enabling researchers, developers, and end-users to make informed decisions based on comparable assessments.
Our holistic evaluation has revealed several key findings:
1. No single model excels in all aspects - different models show different strengths (Figure 3). For example, DALL-E 2 excels in general alignment, Openjourney in aesthetics, and minDALL-E and Safe Stable Diffusion in bias and toxicity mitigation. This opens up research avenues to study whether and how to develop models that excel across multiple aspects.
2. Correlations between human-rated metrics and automated metrics are generally weak, particularly in photorealism and aesthetics. This highlights the significance of using human-rated metrics in evaluating image generation models.
3. Several aspects deserve greater attention. Most models perform poorly in reasoning, photorealism, and multilinguality. Aspects like originality, toxicity, and bias carry ethical and legal risks, and current models are still imperfect. Further research is necessary to address these aspects.
For full transparency and reproducibility, we release the evaluation pipeline and code at https://github.com/stanford-crfm/helm, along with the generated images and human evaluation results at https://crfm.stanford.edu/heim/latest. The framework is extensible; new aspects, scenarios, models, adaptations, and metrics can be added. We encourage the community to consider the different aspects when developing text-to-image models.
2 Core framework
We focus on evaluating text-to-image models, which take textual prompts as input and generate images. Inspired by HELM [1], we decompose the model evaluation into four key components: aspect, scenario, adaptation, and metric (Figure 4).
Figure 2: Standardized evaluation. Prior to HEIM (top panel), the evaluation of image generation models was not comprehensive: six of our 12 core aspects were not evaluated in existing models, and only  $11\%$  of the total evaluation space was studied (the percentage of  $\sqrt{}$  in the matrix of aspects  $\times$  models). After our evaluation (bottom panel), models are now evaluated under the same conditions in all aspects.
Aspect is a specific evaluative dimension that contributes to assessing the overall performance of image generation. Examples include image quality, originality, and bias. Evaluating multiple aspects allows us to capture diverse characteristics of generated images. We evaluate 12 aspects, listed in Table 1, through a combination of scenarios and metrics.
Figure 4: Evaluation components. Each evaluation run consists of an aspect (an evaluative dimension), a scenario (a specific use case), a model with an adaptation process (how the model is run), and one or more metrics (how good are the results).
Scenario represents a specific use case or a dataset of prompts for image generation. We consider various scenarios reflecting different domains and tasks, such as descriptions of common objects (MS-COCO) and logo design (Logos). The complete list of scenarios we use is provided in Table 2.
Adaptation is the specific procedure used to run a model. Examples include zero-shot prompting, few-shot prompting, prompt engineering, and finetuning. We focus on zero-shot prompting, applying the model to each prompt without additional tuning. We also explore prompt engineering techniques, such as Promptist [28], which use language models to refine the prompts before feeding into the model.
Metric quantifies how good the image generation results are. A metric can be human-rated (e.g., overall text-image alignment on a 1-5 scale) or automatically computed (e.g., CLIPScore). By using both human-rated and automated metrics, we capture both subjective and objective assessments of the generated images. The metrics we use are listed in Table 3.
In the subsequent sections of the paper, we delve into the details of aspects (§3), scenarios (§4), metrics (§5), and models (§6), followed by the discussion of experimental results and findings in §7.
3 Aspects
We evaluate diverse aspects of text-to-image models that are crucial for their deployment. Table 1 shows the 12 aspects and their definitions.
Figure 3: The current state of text-to-image generation models. The figure above shows examples of images generated by a select few text-to-image models for various prompts from different aspects (excluding efficiency). Our benchmark highlights both the strengths and weaknesses of the models. For example, DALL-E 2 has decent text-image alignment for both English and Chinese prompts but has clear gender and skin tone bias.
Table 1: Evaluation Aspects of Text-to-Image Models
Aspect	Definition
Alignment	Is the image semantically correct given the text (image-text alignment)
Quality	Do the generated images look like real images/photos
Aesthetics	Is the image aesthetically pleasing
Originality	Does the model generate creative images and prevent copyright infringement
Reasoning	Does the model understand objects, counts, and spatial relations (compositionality) [29]
Knowledge	Does the model have knowledge about the world or domains
Bias	Are the generated images biased in demographic representation (e.g., gender, skin tone) [1]
Toxicity	Does the model generate toxic or inappropriate images (e.g., violence, sexual, illegal content)
Fairness	Does the model exhibit performance disparities across social groups (e.g., gender, dialect) [1]
Robustness	Is the model robust to invariant input perturbations
Multilinguality	Does the model support non-English languages
Efficiency	How fast is inference for the model
For each aspect, we provide a rationale for its inclusion and discuss the scenarios and metrics that can be used to evaluate it (refer to Figure 1 for an illustration). Further details regarding all scenarios and metrics will be presented in §4 and §5.
Image-text alignment and image quality are commonly studied aspects in existing efforts to evaluate text-to-image models [23, 24, 35]. Since these are general aspects, any scenarios can be employed. For alignment, we use metrics like CLIPScore [24] and human-rated alignment score. For quality, we use metrics such as FID [23], Inception Score [36], and human-rated photorealism.
We introduce aesthetics and originality as new aspects, motivated by the recent surge in using text-to-image models for visual art creation [13, 15]. In particular, originality is crucial for addressing concerns of copyright infringement in generative AI [37]. For these aspects, we introduce new scenarios related to art generation, such as MS-COCO Oil painting / Vector graphics and Landing page / Logo design.
For aesthetics, we employ metrics like LAION aesthetics [38], fractal coefficient [39], human-rated subject clarity, and overall aesthetics. For originality, we employ metrics such as watermark detection [38] and human-rated originality scores.
Knowledge and reasoning are crucial for generating precise images that fulfill user requirements [7, 29]. For knowledge, we use scenarios involving specific entities, such as Historical Figures. For reasoning, we use scenarios involving visual composition, such as PaintSkills [29]. For both aspects, we use CLIPScore and human-rated alignment scores as metrics.
Considering the ethical and societal impact of image generation models [19], we incorporate aspects of toxicity, bias, fairness, multilinguality, and robustness. Our definitions, outlined in Table 1, align with [1]. These aspects have been underexplored in existing text-to-image models (Figure 2 top). However, they hold significant importance in real-world model deployment to regulate the generation of toxic and biased content (toxicity and bias) and ensure reliable performance across variations in inputs, such as different social groups (fairness), languages (multilinguality), and perturbations (robustness).
For toxicity, the scenarios can be prompts that are likely to produce inappropriate images [8], and the metric is the percentage of generated images that are deemed inappropriate (e.g., NSFW, nude, or blacked out). For bias, the scenarios can be prompts that may trigger stereotypical associations [33], and the metrics are the demographic biases in generated images, such as gender bias and skin tone bias. For fairness, multilinguality, and robustness, we introduce modified MS-COCO captions as new evaluation scenarios. Changes involve gender/dialect variations (fairness), translation into different
languages (multilinguality), or the introduction of typos (robustness). We then measure the change in model performance (e.g., CLIPScore) compared to the unmodified MS-COCO scenario.
Lastly, efficiency holds practical importance for usability and energy consumption of models [1]. Inference time serves as the metric, and any scenarios can be employed, as efficiency is a general aspect.
4 Scenarios
To evaluate the 12 aspects in image generation (§3), we curate diverse and practical prompting scenarios. Table 2 presents an overview of all the scenarios and their descriptions. Each scenario is a set of prompts and can be used to evaluate certain aspects. For instance, the "MS-COCO" scenario can be used to assess the alignment, quality and efficiency aspects, and the "Inappropriate Image Prompts (I2P)" scenario can be used to assess the toxicity aspect. Some scenarios may include sub-scenarios, indicating the sub-level categories or variations within them, such as "Hate" and "Violence" within I2P. We curate these scenarios by leveraging existing datasets as well as creating new prompts ourselves. In total, we have 62 scenarios including the sub-scenarios.
Notably, we create new scenarios (indicated with "New" in Table 2) for aspects that were previously underexplored and lacked dedicated datasets. These aspects include originality, aesthetics, bias, and fairness. For example, to evaluate originality, we develop scenarios related to the arts, such as prompts for generating landing pages, logos, and magazine covers.
5 Metrics
To evaluate the 12 aspects in image generation (§3), we curate a diverse and realistic set of metrics that can be informative for researchers, developers, and end-users. Table 3 presents an overview of all the metrics and their descriptions.
Our approach incorporates two novelties compared to existing metrics. First, in addition to automated metrics, we use human-rated metrics (top rows in Table 3) to achieve realistic evaluation that reflects human judgment. Specifically, we employ human-rated metrics for the overall image-text alignment
Table 4: Models evaluated in the HEIM effort.
Model	Creator	Type	# Parameters	Access	Reference
Stable Diffusion v1-4	Ludwig Maximilian Univer-sity of Munich CompVis	Diffusion	1B	Open	[4]
Stable Diffusion v1-5	Runway	Diffusion	1B	Open	[4]
Stable Diffusion v2 base	Stability AI	Diffusion	1B	Open	[4]
Stable Diffusion v2-1 base	Stability AI	Diffusion	1B	Open	[4]
Dreamlike Diffusion 1.0	Dreamlike.art	Diffusion	1B	Open	[43]
Dreamlike Photoreal 2.0	Dreamlike.art	Diffusion	1B	Open	[44]
Openjourney	PromptHero	Diffusion	1B	Open	[45]
Openjourney v4	PromptHero	Diffusion	1B	Open	[46]
Redshift Diffusion	nitrosackle	Diffusion	1B	Open	[47]
Vintedois (22h) Diffusion	22h	Diffusion	1B	Open	[48]
SafeStableDiffusion-Weak	TU Darmstadt	Diffusion	1B	Open	[8]
SafeStableDiffusion-Medium	TU Darmstadt	Diffusion	1B	Open	[8]
SafeStableDiffusion-Strong	TU Darmstadt	Diffusion	1B	Open	[8]
SafeStableDiffusion-Max	TU Darmstadt	Diffusion	1B	Open	[8]
Promptist + Stable Diffusion v1-4	Microsoft	Prompt engineering + Diffusion	1B	Open	[4, 28]
Lexica Search (Stable Diffusion v1-5)	Lexica	Diffusion + Retrieval	1B	Open	[49]
DALL-E 2	OpenAI	Diffusion	3.5B	Limited	[3]
DALL-E mini	craiyon	Autoregressive	0.4B	Open	[50]
DALL-E mega	craiyon	Autoregressive	2.6B	Open	[50]
minDALL-E	Kakao Brain Corp.	Autoregressive	1.3B	Open	[51]
CogView2	Tsinghua University	Autoregressive	6B	Open	[10]
MultiFusion	Aleph Alpha	Diffusion	13B	Limited	[52]
DeepFloyd-IF M v1.0	DeepFloyd	Diffusion	0.4B	Open	[53]
DeepFloyd-IF L v1.0	DeepFloyd	Diffusion	0.9B	Open	[53]
DeepFloyd-IF XL v1.0	DeepFloyd	Diffusion	4.3B	Open	[53]
GigaGAN	Adobe	GAN	1B	Limited	[12]
and photorealism, which are used for many evaluation aspects, including alignment, quality, knowledge, reasoning, fairness, robustness, and multilinguality. We also employ human-rated metrics for overall aesthetics and originality, for which capturing the nuances of human judgment is important. To conduct the human evaluation, we employ crowdsourcing following the methodology described in [35]. Concrete word descriptions are provided for each question and rating choice, and a minimum of 5 crowdsourced workers evaluate each image. We use at least 100 image samples for each aspect being evaluated. For a detailed description of the crowdsourcing procedure, please refer to Appendix G.
The second novelty involves introducing new metrics for aspects that have received limited attention in existing evaluation efforts, namely fairness, robustness, multilinguality, and efficiency, as discussed in §3. The new metrics aim to address the evaluation gaps in these aspects.
6 Models
We evaluate 26 recent text-to-image models, encompassing various types (e.g., diffusion, autoregressive), sizes (ranging from 0.4B to 13B parameters), organizations, and accessibility (open source or closed). Table 4 presents an overview of the models and their corresponding properties. In our evaluation, we employ the default inference configurations provided in the respective model's API, GitHub, or Hugging Face repositories.
7 Experiments and results
We evaluated 26 text-to-image models (§6) across the 12 aspects (§3), using 62 scenarios (§4) and 25 metrics (§5). All results are available at https://crfm.stanford.edu/heim/latest. We also provide the result summary in Table 5. Below, we describe the key findings. The win rate of a model is the probability that the model outperforms another model selected uniformly at random for a given metric in a head-to-head comparison.
1. Image-text alignment. DALLE-2 achieves the highest human-rated alignment score among all the models (https://crfm.stanford.edu/heim/v1.1.0/?group=heim Alignment scenarios). It is closely followed by models fine-tuned using high-quality, realistic images, such as Dreamlike Photoreal 2.0 and Vintedois Diffusion. On the other hand, models fine-tuned with art images (Openjourney v4, Redshift Diffusion) and models incorporating safety guidance (SafeStableDiffusion) show slightly lower performance in image-text alignment.
2. Photorealism. In general, none of the models generated images that were deemed photorealistic, as human annotators rated real images from MS-COCO with an average score of 4.48 out of 5 for
photorealism, while no model achieved a score higher than 3 (https://crfm.stanford.edu/heim/v1.1.0/?group=mscoco_base). DALL-E 2 and models fine-tuned with photographs, such as Dreamlike Photoreal 2.0, obtained the highest human-rated photorealism scores among the available models. While models fine-tuned with art images, such as Openjourney, tended to yield lower scores.
3. Aesthetics. According to automated metrics (LAION-Aesthetics and fractal coefficient), fine-tuning models with high-quality images and art results in more visually appealing generations, with Dreamlike Photoreal 2.0, Dreamlike Diffusion 1.0, and Openjourney achieving the highest win rates (https://crfm.stanford.edu/heim/v1.1.0/?group=heim_aesthetics(scenarios). Promptist, which applies prompt engineering to text inputs to generate aesthetically pleasing images according to human preferences, achieves the highest win rate for human evaluation, followed by Dreamlike Photoreal 2.0 and DALL-E 2.
4. Originality. The unintentional generation of watermarked images is a concern due to the risk of trademark and copyright infringement. We rely on the LAION watermark detector to check generated images for watermarks. Trained on a set of images where watermarked images were removed, GigaGAN has the highest win rate, virtually never generating watermarks in images (https://crfm.stanford.edu/heim/v1.1.0/?group=core scenarios). On the other hand, CogView2 exhibits the highest frequency of watermark generation.
Openjourney (86%) and Dreamlike Diffusion 1.0 (82%) achieve the highest win rates for human-rated originality (https://crfm.stanford.edu/heim/v1.1.0/?group=heim-originality(scenarios). Both are Stable Diffusion models fine-tuned on high-quality art images, which enables the models to generate more original images.