Table 3: Metrics used for evaluating the 12 aspects of image generation models. We use realistic, human-rated metrics as well as automated and commonly-used existing metrics.
Metric	Main Aspect	Rated by	Description	Need reference images?	New or existing
Overall alignment	Alignment	Human	Human-rated score (1-5) for "How well does the image match the description?"	No	[6, 35]
Photorealism	Quality	Human	Human-rated score (1-5) for "Does the image look like a real photo or an AI-generated photo?"	No	[6, 35]
Subject clarity	Aesthetics	Human	Human-rated score (yes/no/else) for "Is it clear who the subject(s) of the image is?"	No	New
Overall aesthetics	Aesthetics	Human	Human-rated score (1-5) for "How aesthetically pleasing is the image?"	No	New
Overall originality	Originality	Human	Human-rated score (1-5) for "How original is the image, given it was created with the description?"	No	New
CLIPScore	Alignment	Automated	Text-image alignment measured by CLIP	No	[24]
CLIPScore with Multilingual CLIP	Alignment	Automated	Text-image alignment measured by CLIP	No	[24]
Fréchet Inception Distance (FID)	Quality	Automated	How similar generated images are to reference images, measured by Inception Net	Yes	[23]
Inception score (IS)	Quality	Automated	Quality of generated images, measured by Inception Net	No	[36]
LAION Aesthetics	Aesthetics	Automated	Aesthetic score measured by the LAION predictor	No	[38]
Fractal coefficient	Aesthetics	Automated	Aesthetic complexity measured by the fractal coefficient [39]	No	New
Object detection	Reasoning	Automated	Accuracy of visual composition measured by an object detector	No	[40, 41]
Watermark	Originality	Automated	Whether the image contains watermarks based on the LAION predictor	No	[38]
LAION NSFW	Toxicity	Automated	Whether the image is NSFW (not safe for work) based on the LAION predictor	No	[38]
NudeNet	Toxicity	Automated	Whether the image contains nudity based on NudeNet	No	[42]
Blackout	Toxicity	Automated	How often the model can generate inappropriate images, measured by Stable Diffusion's black out action. This metric is specific to Stable Diffusion models	No	[4]
API rejection	Toxicity	Automated	How often the model can generate inappropriate images, measured by DALLE-2 API's rejection action. This metric is specific to DALLE-2	No	[3]
Gender bias	Bias	Automated	Gender bias in a set of generated images, measured by detecting the gender of each image using CLIP	No	[33, 29]
Skin tone bias	Bias	Automated	Skin tone bias in a set of generated images, measured by detecting skin pixels in each image	No	[33, 29]
Fairness	Fairness	Automated	Performance change in CLIPScore or alignment when the prompt is varied in terms of social groups (e.g., gender/dialect changes)	No	New
Robustness	Robustness	Automated	Performance change in CLIPScore or alignment when the prompt is varied by semantic-preserving perturbations (e.g., typos)	No	New
Multilinguality	Multilinguality	Automated	Performance change in CLIPScore or alignment when the prompt is translated into non-English languages (e.g., Spanish, Chinese, Hindi)	No	New
Raw inference time	Efficiency	Automated	Wall-clock inference runtime	No	New
Denoised inference time	Efficiency	Automated	Wall-clock inference runtime with performance variation factored out	No	New