SlowGuess/NeurIPS_2023_OCR huggingface dataset:
To compute the FID, we randomly selected 30,000 text prompts from MS-COCO and generated a single image for each prompt using the text-to-image generation model that we are evaluating. Then, we resized the images to  $512 \times 512$  and used [70] to compute the FID between the set of real images associated with the prompts and the set of generated images.
Inception score (IS). Inception score (IS) [36] is a benchmark metric used for evaluating the quality of image-based generative models [4, 71]. We follow their settings and compute the IS scores for evaluating the image quality of the generative models. We use the implementation [72] to compute the metric.
LAION Aesthetics. We follow LAION-Aesthetics and use the open-sourced predictor<sup>1</sup> for evaluating the aesthetic score of a generated image.
Fractal coefficient. The fractal coefficient is a measure of the fractal patterns in an image, indicating the degree of detail and complexity present at different scales. It can be used to assess the aesthetic quality of images by quantifying their level of visual intricacy and richness [39]. We used the following code snippet to compute the metric: https://gist.github.com/viveksck/1110dfca01e4ec2c608515f0d5a5b1d1. We report the absolute difference of the fractal coefficient from 1.4, which is the most aesthetically pleasing to humans according to [39].
Watermark. The unintentional generation of watermarked images poses a risk of trademark and copyright infringement. We use the watermark detector from LAION [38] to determine whether an image contains watermarks.
LAION NSFW. We use the NSFW (not safe for work) classifier from LAION [38] to determine whether an image contains inappropriate content.
<https://laion.ai/blog/laion-aesthetics/>
NudeNet. In accordance with [8], we employ the pre-trained NudeNet [42] to determine whether an image contains nudity.
Blackout. Stable Diffusion [4] incorporates a module that post-processes images by blacking them out if they are considered inappropriate. This metric represents whether the image has been blacked out.
API rejection. The DALL-E 2 API [3] rejects requests when the provided prompt can lead to the generation of inappropriate content. This metric represents whether the image has been rejected. It pertains specifically to DALL-E 2.
Gender bias. This metric assesses the presence of binary gender bias within a set of images, following [33, 29]. Specifically, we obtain 8 output images from the model based on a given prompt. For each image, we employ CLIP to compare it against the prompts {"a photo of a male", "a photo of a female"} and select the gender with the higher probability. The proportion of female representation is then computed among the 8 images. Finally, we calculate the L1 norm distance between this proportion and 0.5, which serves as the measure of gender bias.
Skin tone bias. This metric assesses the presence of skin tone bias within a set of images, following [33, 29]. Specifically, we obtain 8 output images from the model based on a given prompt. For each image, we identify skin pixels by analyzing the RGBA and YCrCb color spaces. These skin pixels are then compared to a set of 10 MST (Monk Skin Tone) categories, and the closest category is selected. Using the 8 images, we compute the distribution across the 10 MST skin tone categories, resulting in a vector of length 10. Finally, we calculate the L1 norm distance between this vector and a uniform distribution vector (also length 10), with each value set to 0.1. This calculated error value serves as the measure of skin tone bias.
Fairness. This metric, inspired by [1], assesses changes in model performance (human-rated alignment score and CLIPScore) when the prompt is varied in terms of social groups. For instance, this involves modifying male terms to female terms or incorporating African American dialect into the prompt (see MS-COCO (gender) and MS-COCO (dialect) in  $\S D$ ). A fair model is expected to maintain consistent performance without experiencing a decline in its performance.
Robustness. This metric, inspired by [1], assesses changes in model performance (human-rated alignment score and CLIPScore) when the prompt is perturbed in a semantic-preserving manner, such as injecting typos (see MS-COCO (typos) in §D). A robust model is expected to maintain consistent performance without experiencing a decline in its performance.
Multiliguality. This metric assesses changes in model performance (human-rated alignment score and CLIPScore) when the prompt is translated into non-English languages, such as Spanish, Chinese, and Hindi. We use Google Translate for the translations (see MS-COCO (languages) in §D). A multilingual model is expected to maintain consistent performance without experiencing a decline in its performance.
Inference time. Using APIs introduces performance variability; for example, requests might experience queuing delay or interfere with each other. Consequently, we use two inference runtime metrics to separate out these concerns: raw runtime and a version with this performance variance factored out called the denoised runtime [73].
Object detection. We use the ViTDet [41] object detector with ViT-B [74] backbone and detector2 [75] library to automatically detect objects specified in the prompts. The object detection metrics are measured with three skills, similar to DALL-Eval [29]. First, we evaluate the object recognition skill by calculating the average accuracy over  $N$  test images, determining whether the object detector accurately identifies the target class from the generated images. Object counting skill is assessed similarly by calculating the average accuracy over  $N$  test images and evaluating whether the object detector correctly identifies all  $M$  objects of the target class from each generated image. Lastly, spatial relation understanding skill is evaluated based on whether the object detector correctly identifies both target object classes and the pairwise spatial relations between objects. The target class labels, object counts, and spatial relations come from the text prompts used to query the models being evaluated.
F Model details
Stable Diffusion {v1-4, v1-5, v2-base, v2-1}. Stable Diffusion (v1-4, v1-5, v2-base, v2-1) is a family of 1B-parameter text-to-image models based on latent diffusion [4] trained on LAION [38], a large-scale paired text-image dataset.
Specifically, Stable Diffusion v1-1 was trained 237k steps at resolution  $256 \times 256$  on laion2B-en and 194k steps at resolution  $512 \times 512$  on laion-high-resolution (170M examples from LAION-5B with resolution  $>= 1024 \times 1024$ ). Stable Diffusion v1-2 was initialized with v1-1 and trained 515k steps at resolution  $512 \times 512$  on laion-aesthetics v2  $5+$ . Stable Diffusion v1-4 is initialized with v1-2 and trained 225k steps at resolution  $512 \times 512$  on "laion-aesthetics v2  $5+$ " and  $10\%$  dropping of the text-conditioning to improve classifier-free guidance sampling. Similarly, Stable Diffusion v1-5 is initialized with v1-2 and trained 595k steps at resolution  $512 \times 512$  on "laion-aesthetics v2  $5+$ " and  $10\%$  dropping of the text-conditioning.
Stable Diffusion v2-base is trained from scratch 550k steps at resolution 256x256 on a subset of LAION-5B filtered for explicit pornographic material, using the LAION-NSFW classifier with punsafe  $= 0.1$  and an aesthetic score  $>= 4.5$ . Then it is further trained for 850k steps at resolution 512x512 on the same dataset on images with resolution  $>= 512x512$ . Stable Diffusion v2-1 is resumed from Stable diffusion v2-base and finetuned using a v-objective [76] on a filtered subset of the LAION dataset.
Lexica Search (Stable Diffusion v1-5). Lexica Search (Stable Diffusion v1-5) is an image search engine for searching images generated by Stable Diffusion v1-5 [4].
DALL-E 2. DALL-E 2 [3] is a 3.5B-parameter encoder-decoder-based latent diffusion model trained on large-scale paired text-image datasets. The model is available via the OpenAI API.
Dreamlike Diffusion 1.0. Dreamlike Diffusion 1.0 [43] is a Stable Diffusion v1-5 model fine-tuned on high-quality art images.
Dreamlike Photoreal 2.0. Dreamlike Photoreal 2.0 [44] is a photorealistic model fine-tuned from Stable Diffusion 1.5. While the original Stable Diffusion generates resolutions of  $512 \times 512$  by default, Dreamlike Photoreal 2.0 generates  $768 \times 768$  by default.
Openjourney {v1, v4}. Openjourney [45] is a Stable Diffusion model fine-tuned on Midjourney images. Openjourney v4 [46] was further fine-tuned using +124000 images, 12400 steps, 4 epochs +32 training hours. Openjourney v4 was previously referred to as Openjourney v2 in its Hugging Face repository.
Redshift Diffusion. Redshift Diffusion [47] is a Stable Diffusion model fine-tuned on high-resolution 3D artworks.
Vintedois (22h) Diffusion. Vintedois (22h) Diffusion [48] is a Stable Diffusion v1-5 model finetuned on a large number of high-quality images with simple prompts to generate beautiful images without a lot of prompt engineering.
SafeStableDiffusion-{Weak, Medium, Strong, Max}. Safe Stable Diffusion [8] is an enhanced version of the Stable Diffusion v1.5 model. It has an additional safety guidance mechanism that aims to suppress and remove inappropriate content (hate, harassment, violence, self-harm, sexual content, shocking images, and illegal activity) during image generation. The strength levels for inappropriate content removal are categorized as: {Weak, Medium, Strong, Max}.
Promptist + Stable Diffusion v1-4. Promptist [28] is a prompt engineering model, initialized by a 1.5 billion parameter GPT-2 model [77], specifically designed to refine user input into prompts that are favored by image generation models. To achieve this, Promptist was trained using a combination of hand-engineered prompts and a reward function that encourages the generation of aesthetically pleasing images while preserving the original intentions of the user. The optimization of Promptist was based on the Stable Diffusion v1-4 model.
DALL-E {mini, mega}. DALL-E {mini, mega} is a family of autoregressive Transformer-based text-to-image models created with the objective of replicating OpenAI DALL-E 1 [2]. The mini and mega variants have 0.4B and 2.6B parameters, respectively.
minDALL-E. minDALL-E [51], named after minGPT, is a 1.3B-parameter autoregressive transformer model for text-to-image generation. It was trained using 14 million image-text pairs.
CogView2. CogView2 [10] is a hierarchical autoregressive transformer (6B-9B-9B parameters) for text-to-image generation that supports both English and Chinese input text.
MultiFusion. MultiFusion (13B) [52] is a multimodal, multilingual diffusion model that extends the capabilities of Stable Diffusion v1.4 by integrating different pre-trained modules, which transfer capabilities to the downstream model. This combination results in novel decoder embeddings, which enable prompting of the image generation model with interleaved multimodal, multilingual inputs, despite being trained solely on monomodal data in a single language.
DeepFloyd-IF  $\{\mathbf{M},\mathbf{L},\mathbf{XL}\}$  v1.0. DeepFloyd-IF [53] is a pixel-based text-to-image triple-cascaded diffusion model with state-of-the-art photorealism and language understanding. Each cascaded diffusion module is designed to generate images of increasing resolution:  $64\times 64$ $256\times 256$  ,and  $1024\times 1024$  . All stages utilize a frozen T5 transformer to extract text embeddings, which are then fed into a UNet architecture enhanced with cross-attention and attention-pooling. The model is available in three different sizes: M, L, and XL. M has 0.4B parameters, L has 0.9B parameters, and XL has 4.3B parameters.
GigaGAN. GigaGAN [12] is a billion-parameter GAN model that quickly produces high-quality images. The model was trained on text and image pairs from LAION2B-en [38] and COYO-700M [78].
G Human evaluation procedure
G.1 Amazon Mechanical Turk
We used the Amazon Mechanical Turk (MTurk) platform to receive human feedback on the AI-generated images. Following [35], we applied the following filters for worker requirements when creating the MTurk project: 1) Maturity: Over 18 years old and agreed to work with potentially offensive content 2) Master: Good-performing and granted AMT Masters. We required five different annotators per sample. Figure 6 shows the design layout of the survey.
Based on an hourly wage of  $16 per hour, each annotator was paid$ 0.02 for answering a single multiple-choice question. The total amount spent for human annotations was $13,433.55.
G.2 Human Subjects Institutional Review Board (IRB)
We submitted a social and behavior (non-medical) human subjects IRB application with the research proposal for this work and details of human evaluation to the Research Compliance Office at Stanford University. The Research Compliance Office deemed that our IRB protocol did not meet the regulatory definition of human subjects research since we did not plan to draw conclusions about humans, nor were we evaluating any characteristics of the human raters. As such, the protocol has been withdrawn, and we were allowed to proceed without any additional IRB review.
Please answer the questions below about the following image and description.
Description: a furry red monster questioning life choices
Does not match at all
O Has significant discrepancies
O Has several minor discrepancies
O Has a few minor discrepancies
Matches exactly
Determine if the following image is Al-generated or real.
Does the image look like an Al-generated photo or a real photo?
$\bigcirc$  Al-generated photo
$\bigcirc$  Probably an Al-generated photo, but photorealistic.
Neutral
$\bigcirc$  Probably a real photo, but with irregular textures and shapes.
Real photo
Figure 6: Human annotation interface. Screenshots of the human annotation interface on Amazon Mechanical Turk. We opted for a simple layout where the general instruction is shown at the top, followed by the image, prompt (if necessary), and the questions below. Human raters were asked to answer multiple-choice questions about the alignment, photorealism, aesthetics, and originality of the displayed images, with the option to opt out of any task.
H More results
Table 5: Result summary for evaluating models (rows) across various aspects (columns). For each aspect, we show the win rate of each model. The full and latest results can be found at https://crfm.stanford.edu/heim/latest.
Models	Alignment	Quality	Aesthetics	Originality	Reasoning	Knowledge	Bias (gender)	Bias (skin)	Toxicity	Efficiency	Art styles
Stable Diffusion v1.4 (1B)	0.691	0.88	0.667	0.64	0.673	0.747	0.54	0.48	0	0.84	0.78
Stable Diffusion v1.5 (1B)	0.531	0.72	0.197	0.31	0.42	0.52	0.58	0.64	0.04	0.8	0.32
Stable Diffusion v2 base (1B)	0.514	0.84	0.197	0.17	0.613	0.387	0.66	0.64	0.88	0.88	0.26
Stable Diffusion v2.1 base (1B)	0.314	0.76	0.203	0.1	0.7	0.36	0.36	0.64	0.72	0.92	0.2
Dreamlike Diffusion v1.0 (1B)	0.6	0.52	0.68	0.82	0.62	0.707	0.06	0.12	0.56	0.44	0.7
Dreamlike Photoreal v2.0 (1B)	0.851	0.96	0.843	0.8	0.527	0.96	0.46	0	0.44	0.28	0.98
Openjourney v1 (1B)	0.617	0.08	0.837	0.86	0.327	0.693	0.26	0.52	0.28	0.64	0.88
Openjourney v2 (1B)	0.434	0.24	0.73	0.72	0.52	0.347	0.76	0.96	0.24	0.72	0.78
Redshift Diffusion (1B)	0.389	0.2	0.287	0.47	0.327	0.453	0.16	0.04	0.36	0.76	0.22
Vintedois (22h) Diffusion model v0.1 (1B)	0.549	0	0.157	0.27	0.62	0.44	0.44	0.24	0.16	0.68	0.32
Safe Stable Diffusion weak (1B)	0.577	0.92	0.227	0.11	0.46	0.56	0.44	0.56	0.08	0.48	0.5
Safe Stable Diffusion medium (1B)	0.497	0.8	0.27	0.18	0.56	0.307	0.46	0.44	0.2	0.6	0.2
Safe Stable Diffusion strong (1B)	0.617	0.6	0.837	0.73	0.62	0.707	0.88	0.54	0.32	0.52	0.78
Safe Stable Diffusion max (1B)	0.377	0.64	0.69	0.78	0.413	0.52	0.84	0.14	0.4	0.56	0.78
Promptist + Stable Diffusion v1.4 (1B)	0.589	0.04	0.883	0.73	0.527	0.72	0.32	0.42	0.12	0.24	0.76
Lexica Search with Stable Diffusion v1.5 (1B)	0.04	0.16	0.74	0.64	0.067	0.28	0.58	0.46	0.68	0.96	0.66
DALL-E 2 (3.5B)	0.971	1	0.843	0.67	0.993	0.947	0.1	0.76	0.8	0.36	0.32
DALL-E mini (0.4B)	0.44	0.28	0.787	0.73	0.487	0.52	0.96	0.7	1	0.2	0.88
DALL-E mega (2.6B)	0.589	0.36	0.537	0.73	0.527	0.493	0.66	0.5	0.92	0.16	0.66
minDALL-E (1.3B)	0.154	0.32	0.483	0.76	0.24	0.187	1	0.86	0.96	0.32	0.24
CogView2 (6B)	0.074	0.12	0.553	0.53	0.02	0	0.8	0.86	0.48	0.12	0.32
MultiFusion (13B)	0.48	0.68	0.567	0.5	0.287	0.173	0.38	0.66	0.76	0.4	0.7
DeepFloyd IF Medium (0.4B)	0.566	0.44	0.217	0.14	0.487	0.56	0.42	0.58	0.52	0.08	0.3
DeepFloyd IF Large (0.9B)	0.514	0.4	0.223	0.15	0.553	0.653	0.38	0.26	0.64	0.04	0.22
DeepFloyd IF X-Large (4.3B)	0.589	0.56	0.18	0.22	0.78	0.427	0.18	0.12	0.6	0	0.22
GigaGAN (1B)	0.434	0.48	0.167	0.24	0.633	0.333	0.32	0.86	0.84	1	0.02