Table 2: Scenarios used for evaluating the 12 aspects of image generation models.
Scenario	Sub-Scenarios	Main Aspects	Description	New or existing
MS COCO (2014)	-	Quality, Alignment, Efficiency	A widely-used dataset of caption-image pairs about common objects. We use the 2014 validation set of MS COCO.	[21]
MS COCO (2014)	Oil painting / Watercolor / Pencil sketch / Animation / Vector graph-ics / Pixel art	Aesthetics, Alignment	Modified versions of MS COCO captions to which art style specifications (e.g., "oil painting") are added	New
MS COCO (2014)	Gender substitution / African Ameri-can dialect	Fairness	Modified versions of MS COCO captions to which gender substitu-tion or dialect is applied	New
MS COCO (2014)	Typos	Robustness	Modified version of MS COCO captions to which semantic-preserving perturbations (typos) are applied	New
MS COCO (2014)	Chinese / Hindi / Spanish	Multilinguality	Modified version of MS COCO captions, which are translated into non-English languages (Chinese, Hindi, Spanish)	New
CUB-200-2011	-	Alignment	A widely-used dataset of caption-image pairs about birds.	[22]
DrawBench	Colors / Text	Alignment	Prompts to generate colors, DALL-E images, or text letters	[6]
PartiPrompts (P2)	Artifacts / Food & Beverage / Vehi-cles / Arts / Indoor Scenes / Outdoor Scenes / Produce & Plants / People / Animals	Alignment	Prompts to generate various categories of objects (e.g., food, vehi-cles, animals)	[7]
Common Syntactic Processes	Negation / Binding principles / Pass-sives / Word order / Ellipsis / Ambi-guity / Coordination / Comparatives	Reasoning	Prompts that involve various categories of textual reasoning (e.g., negation, word order)	[30]
DrawBench	Counting / Descriptions / Gary Mar-cus et al. / DALL-E / Positional / Conflicting	Reasoning	Prompts that involve various categories of visual composition (e.g., counting, positioning, rare combination of objects)	[6]
PartiPrompts (P2)	Illustrations	Reasoning	Prompts to generate compositional illustrations (e.g., "a red box next to a blue box")	[7]
Relational Understanding	-	Reasoning	Compositional prompts about entities and relations motivated by cognitive, linguistic, and developmental literature	[31]
Detection (PaintSkills)	Object / Spatial / Count	Reasoning	Diagnostic prompts to test compositional visual reasoning (e.g., count, spatial relation)	[29]
Winoground	-	Reasoning	Prompts that involve visuo-linguistic reasoning (e.g., word order)	[32]
PartiPrompts (P2)	World Knowledge	Knowledge	Prompts about entities and places that exist in the world (e.g., "Sydney Opera House")	[7]
DrawBench	Reddit	Knowledge	Captions from Reddit, which typically contain specific entities (e.g., "Super Mario")	[6]
Historical Figures	-	Knowledge	People from TIME's "The 100 Most Significant Figures in History"	New
dailydall.e	-	Originality, Aesthetics	DALL-E 2 prompts from the artist Chad Nelson's Instagram	New
Landing Pages	-	Originality, Aesthetics	Prompts to design landing pages for mobile or web applications.	New
Logos	-	Originality, Aesthetics	Prompts to design logos for brands and companies	New
Magazine Covers	-	Originality, Aesthetics	Prompt to design magazine cover photos	New
Demographic Stereotypes	Descriptors / Occupations	Bias	Descriptors or occupations of people, which may exhibit stereotypi-cal associations with demographic groups	[33, 29]
Mental Disorders	-	Bias	Prompts about mental disorders. Motivated by [34], to evaluate stereotypical associations about mental disorders.	New
Inappropriate Image Prompts (I2P)	Hate / Harassment / Violence / Self-harm / Sexual content / Shocking im-age / Illegal activity	Toxicity	Collection of real user prompts that are likely to produce inappropri-ate images	[8]