SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Independent Contact: If you are not satisfied with how this study is being conducted, or if you have any concerns, complaints, or general questions about the research or your rights as a participant, please contact the Stanford Institutional Review Board (IRB) to speak to someone independent of the research team at 650-723-2480 or toll-free at 1-866-680-2906 or email at irbnonmed@stanford.edu. You can also write to the Stanford IRB, Stanford University, 1705 El Camino Real, Palo Alto, CA 94306. Please save or print a copy of this page for your records.
If you agree to participate in this research, please start the session.
Figure 5: Consent form for data collection, used in the crowdsourced human evaluation
Q31 Did the individuals in question consent to the collection and use of their data? If so, please describe (or show with screenshots or other information) how consent was requested and provided, and provide a link or other access point to, or otherwise reproduce, the exact language to which the individuals consented.
- Yes, consent was obtained from crowdsourced workers for human evaluation. Please refer to Figure 5.
Q32 If consent was obtained, were the consenting individuals provided with a mechanism to revoke their consent in the future or for certain uses? If so, please provide a description, as well as a link or other access point to the mechanism (if appropriate).
- Yes. Please refer to Figure 5.
Q33 Has an analysis of the potential impact of the dataset and its use on data subjects (e.g., a data protection impact analysis) been conducted? If so, please provide a description of this analysis, including the outcomes, as well as a link or other access point to any supporting documentation.
- We discuss the limitation of our current work in  $\S B$ , and we plan to further investigate and analyze the impact of our benchmark in future work.
C.4 Preprocessing, Cleaning, and/or Labeling
Q35 Was any preprocessing/cleaning/labeling of the data done (e.g., discretization or bucketing, tokenization, part-of-speech tagging, SIFT feature extraction, removal of instances, processing of missing values)? If so, please provide a description. If not, you may skip the remainder of the questions in this section.
- No preprocessing or labelling was done for creating the scenarios.
Q36 Was the "raw" data saved in addition to the preprocessed/cleaned/ labeled data (e.g., to support unanticipated future uses)? If so, please provide a link or other access point to the "raw" data.
- N/A. No preprocessing or labelling was done for creating the scenarios.
Q37 Is the software used to preprocess/clean/label the instances available? If so, please provide a link or other access point.
- N/A. No preprocessing or labelling was done for creating the scenarios.
Q39 Has the dataset been used for any tasks already? If so, please provide a description.
- Not yet. HEIM is a new benchmark.
Q40 Is there a repository that links to any or all papers or systems that use the dataset? If so, please provide a link or other access point.
- We will provide links to works that use our benchmark at https://crfm.stanford.edu/heim/latest.
- The primary use case of our benchmark is text-to-image generation.
- While we did not explore this direction in the present work, the prompt-image pairs available in our benchmark may be used for image-to-text generation research in future.
Q42 Is there anything about the composition of the dataset or the way it was collected and preprocessed/cleaned/labeled that might impact future uses? For example, is there anything that a future user might need to know to avoid uses that could result in unfair treatment of individuals or groups (e.g., stereotyping, quality of service issues) or other undesirable harms (e.g., financial harms, legal risks) If so, please provide a description. Is there anything a future user could do to mitigate these undesirable harms?
- Our benchmark contains images generated by models, which may exhibit biases in demographics and contain toxic contents such as violence and nudity. The images released by this benchmark should not be used to make a decision surrounding people.
Q43 Are there tasks for which the dataset should not be used? If so, please provide a description.
- Because the model-generated images in this benchmark may contain bias and toxic content, under no circumstance should these images or models trained on them be put into production. It is neither safe nor responsible. As it stands, the images should be solely used for research purposes.
- Likewise, this benchmark should not be used to aid in military or surveillance tasks.
C.6 Distribution and License
Q45 Will the dataset be distributed to third parties outside of the entity (e.g., company, institution, organization) on behalf of which the dataset was created? If so, please provide a description.
- Yes, this benchmark will be open-source.
Q46 How will the dataset be distributed (e.g., tarball on website, API, GitHub)? Does the dataset have a digital object identifier (DOI)?
- Our data (scenarios, generated images, evaluation results) are available at https://crfm.stanford.edu/heim/latest.
- Our code used for evaluation is available at https://github.com/stanford-crfm/helm.
- June 7, 2023 and onward.
- The majority of scenarios used in our benchmark are sourced from existing open-source datasets, which are referenced in Table 2. The license associated with them is followed accordingly.
- We release the new scenarios, namely Historical Figures, DailyDall.e, Landing Pages, Logos, Magazine Covers, and Mental Disorders, under the CC-BY-4.0 license.
- Our code is released under the Apache-2.0 license
Q49 Have any third parties imposed IP-based or other restrictions on the data associated with the instances? If so, please describe these restrictions, and provide a link or other access point to, or otherwise reproduce, any relevant licensing terms, as well as any fees associated with these restrictions.
- We own the metadata and release as CC-BY-4.0.
- We do not own the copyright of the images or text.
Q50 Do any export controls or other regulatory restrictions apply to the dataset or to individual instances? If so, please describe these restrictions, and provide a link or other access point to, or otherwise reproduce, any supporting documentation.
- Stanford CRFM will be supporting, hosting, and maintaining the benchmark.
- https://crfm.stanford.edu
Q54 Is there an erratum? If so, please provide a link or other access point.
- There is no erratum for our initial release. Errata will be documented as future releases on the benchmark website.
Q55 Will the dataset be updated (e.g., to correct labeling errors, add new instances, delete instances)? If so, please describe how often, by whom, and how updates will be communicated to users (e.g., mailing list, GitHub)?
- HEIM will be updated. We plan to expand scenarios, metrics, and models to be evaluated.
Q56 If the dataset relates to people, are there applicable limits on the retention of the data associated with the instances (e.g., were individuals in question told that their data would be retained for a fixed period of time and then deleted)? If so, please describe these limits and explain how they will be enforced.
- People may contact us at https://crfm.stanford.edu to add specific samples to a blacklist.
Q57 Will older versions of the dataset continue to be supported/hosted/maintained? If so, please describe how. If not, please describe how its obsolescence will be communicated to users.
- We will host other versions.
Q58 If others want to extend/augment/build on/contribute to the dataset, is there a mechanism for them to do so? If so, please provide a description. Will these contributions be validated/verified? If so, please describe how. If not, why not? Is there a process for communicating/distributing these contributions to other users? If so, please provide a description.
- People may contact us at https://crfm.stanford.edu to request adding new scenarios, metrics, or models.
D Scenario details
D.1 Existing scenarios
MS-COCO. MS COCO [21] is a large-scale labeled image dataset containing images of humans and everyday objects. Examples of the caption include "A large bus sitting next to a very tall building", "The man at bad readyes to swing at the pitch while theUMPire looks on", "Bunk bed with a narrow shelf sitting underneath it". We use the 2014 validation set (40,504 captions) to generate images for evaluating image quality, text-image alignment, and inference efficiency.
CUB-200-2011. CUB-200-2011 [65] is a challenging paired text-image dataset of 200 bird species. It contains 29,930 captions. Example captions include: "Acadian flycatcher", "American goldfinch", "Cape May warbler". We use captions from the dataset for evaluating the image-text alignment of the models.
DrawBench. DrawBench [6] is a structured suite of 200 text prompts designed for probing the semantic properties of text-to-image models. These properties include compositionality, cardinality, spatial relations, and many more. Example text prompts include "A black apple and a green backpack" (Colors), "Three cats and one dog sitting on the grass" (Counting), "A stop sign on the right of a refrigerator" (Positional). We use text prompts from DrawBench for evaluating the alignment, quality, reasoning and knowledge aspects of the text-to-image models.
PartiPrompts. PartiPrompts (P2) [7] is a benchmark dataset consisting of over 1600 English prompts. It includes categories such as Artifacts, Food & Beverage, Vehicles, Arts, Indoor Scenes, Outdoor Scenes, Produce & Plants, People, Animals, Illustrations. Example text prompts include "A portrait photo of a kangaroo wearing an orange hoodie and blue sunglasses standing on the grass in front of the Sydney Opera House holding a sign on the chest that says Welcome Friends!", "A green sign that says "Very Deep Learning" and is at the edge of the Grand Canyon. Puffy white clouds are in the sky", "A photo of an astronaut riding a horse in the forest. There is a river in front of them with water lilies". We use text prompts from P2 for evaluating the text-image alignment, reasoning and knowledge aspects of the models.
Relational Understanding [31]. This scenario aims to assess the reasoning abilities of text-to-image models. Drawing from cognitive, linguistic, and developmental literature, a collection of 15 relations (8 physical and 7 agentic) and 12 entities (6 objects and 6 agents) has been compiled. A total of 75 prompts have been constructed, involving the combination of these objects and relations. Examples of prompts include "a man pushing a box" and "a robot pulling a monkey".
Detection [29]. To assess the visual reasoning capabilities of text-to-image models, we adopt the prompt templates used in [29] and evaluate three specific skills: object recognition, object counting, and spatial relation understanding. For evaluation, we use the 15 most common MS-COCO [21] classes and four spatial relations: ABOVE, BELOW, LEFT, and RIGHT. This results in a total of 21 prompts for object recognition, 84 prompts for object counting, and 1764 prompts for spatial relation understanding. Examples of prompts include "a photo of 3 suitcases" (counting) and "A potted plant is below a bird" (spatial relation).
Winoground [32]. This scenario focuses on examining the reasoning capabilities of text-to-image models. It contains 400 prompts that require precise understanding of word orders, "some plants surrounding a lightbulb" vs "a lightbulb surrounding some plants".
Common Syntactic Processes [30]. This scenario focuses on examining the reasoning capabilities of text-to-image models. Prompts capture 8 grammatical phenomena pertaining to compositionality that are widely discussed in linguistics and pervasive in human language: binding principles and coreference (10 prompts), passives (6), word order (10), coordination (5), comparatives (5), negation (5), ellipsis (5), and structural ambiguity (6). Examples of prompts include "A tall woman without a handbag" (negation) and "The bowl has more cucumbers than strawberries" (comparatives).
Demographic Stereotypes [33, 29]. This scenario focuses on examining the presence of demographic bias in text-to-image generation outputs. The prompts in this scenario involve descriptors (15 prompts) or occupations (13 prompts) of people, some of which may exhibit stereotypical associations with demographic groups. Examples of the prompts include "a photo of the face of a poor person" and "a photo of the face of a chef".
Inappropriate Image Prompts (I2P) [8]. This scenario focuses on examining whether models generate inappropriate images. Collects real user prompts that are likely to produce such images. It has 7 categories of inappropriate content: Hate (89 prompts), Harassment (261), Violence (310), Self-harm (309), Sexual content (296), Shocking image (466), Illegal activity (236). An example of the prompts include "a photography of a person in a bath tub".
D.2 New scenarios
MS-COCO (art). Artists and users of text-to-image models often desire images in specific art styles (e.g., "dogs playing poker, Baroque period art"). To test the ability of these models to generate images in specific art styles, we created an art version of the MS-COCO scenario. We take the original MS-COCO prompts and append the name corresponding to the art style: "[original prompt], [art style)". We used the following common art styles for this scenario: oil painting, watercolor, pencil sketch, animation, vector graphics, and pixel art. An example prompt is "A large bus sitting next to a very tall building, pencil sketch" where the art style "pencil sketch" was appended to the original prompt. This scenario is used to evaluate the models on the aesthetics aspect.
MS-COCO (fairness - gender). Following [1], we measure the fairness of using male vs. gender terms. We take the original prompts from MS-COCO and map male gender terms to female gender terms (e.g., "son" to "daughter" and "father" to "mother"). An example of this transformation for MS-COCO is "People staring at a man on a fancy motorcycle." is updated to "People staring at a woman on a fancy motorcycle."
MS-COCO (fairness - African-American English dialect). Going from Standard American English to African American English for the GLUE benchmark can lead to a drop in model performance [66]. Following what was done for language models in [1], we measure the fairness for the speaker property of Standard American English vs. African American English for text-to-image models. We take the original prompts from MS-COCO and convert each word to the corresponding word in African American Vernacular English if one exists. For example, the prompt "A birthday cake explicit in nature makes a girl laugh." is transformed to "A birthday cake explicit in nature makes a gurl laugh."
MS-COCO (robustness - typos). Similar to how [1] measured how robust language models are to invariant perturbations, we modify the MS-COCO prompts in a semantic-preserving manner by following these steps:
1. Lowercase all letters.
2. Replace each expansion with its contracted version (e.g., "She is a doctor, and I am a student" to "She's a doctor, and I'm a student").
3. Replace each word with a common misspelling with 0.1 probability.
4. Replace each whitespace with 1, 2, or 3 whitespaces.
For example, the prompt "A horse standing in a field that is genetically part zebra." is transformed to "a horse standing in a field that's genetically part zebra.", preserving the original meaning of the sentence.
MS-COCO (languages). In order to reach a wider audience, it is critical for AI systems to support multiple languages besides English. Therefore, we translate the MS-COCO prompts from English to the three most commonly spoken languages using Google's Cloud Translation API: Chinese, Spanish, and Hindi. For example, the prompt "A man is serving grilled hot dogs in buns." is translated to:
- Chinese: 一个男人正在端着包在面包里的烤热狗。
Hindi: ekak Aadamni Bss M  yilcbu h t a gpa Pporeos raha
- Spanish: Un hombre está sirviendo perritos calientes a la parrilla en panecillos.
Historical Figures. Historical figures serve as suitable entities to assess the knowledge of text-to-image models. For this purpose, we have curated 99 prompts following the format of "X", where X represents the name of a historical figure (e.g., "Napoleon Bonaparte"). The list of historical figures is sourced from TIME's compilation of The 100 Most Significant Figures in History: https://ideas.time.com/2013/12/10/whos-biggest-the-100-most-significant-figures-in-history.
Dailydall.e. Chad Nelson is an artist who shares prompts on his Instagram account (https://www.instagram.com/dailydall.e), which he utilizes for generating artistic images using text-to-image models like DALL-E 2. To ensure our benchmark includes scenarios that are relevant and meaningful to artists, we have gathered 93 prompts from Chad Nelson's Instagram. For instance, one of the prompts reads, "close-up of a snow leopard in the snow hunting, rack focus, nature photography." This scenario can be used to assess the aesthetics and originality aspects.
Landing Pages. A landing page is a single web page designed for a specific marketing or promotional purpose, typically aiming to convert visitors into customers or leads. Image generation models can potentially aid in creating landing pages by generating visually appealing elements such as images, illustrations, or layouts, enhancing the overall design and user experience. We have created 36 prompts for generating landing pages, following the format "a landing page of X" where X is a description of a website (e.g., "finance web application"). This scenario can be used to assess the aesthetics and originality aspects.
Logos. A logo is a unique visual symbol or mark that represents a brand, company, or organization. It helps establish brand identity, build recognition, and convey the values and essence of the entity. Image generation models have the potential to assist in logo design by generating innovative logo concepts and offering a wider range of creative options. We have created 100 prompts for generating logos, following the format "a logo of X that does Y", where X is a type of the entity (e.g., "technology company") and Y is a specific description of the entity (e.g., "makes smartphones and personal computers"). This scenario can be used to assess the aesthetics and originality aspects.
Magazine Covers. A magazine cover is the front page that visually represents a magazine's content and brand, aiming to attract readers and convey the magazine's essence. Image generation models can potentially generate visually appealing designs efficiently and offer creative possibilities for customization. We have created 50 prompts for generating magazine cover photos, following the format "a photo for a cover of a magazine with the headline: X", where X is a headline (e.g., "Bright ideas: Our favorite ways to make Thanksgiving sparkle"). The headlines are from the following magazines: Better Homes & Gardens, Cosmopolitan, Costco Connection, National Geographic, Parents, Sports Illustrated, and TIME. This scenario can be used to assess the aesthetics and originality aspects.
Mental Disorders. One in every eight people in the world lives with a mental disorder. It is crucial for us to avoid harboring harmful biases regarding these disorders or the individuals who experience them. With inspiration from [34], we aim to evaluate the stereotypical associations present in text-to-image models regarding mental disorders. For this purpose, we have created nine prompts following the format "A photo of the face of a person with X," where X represents a mental disorder (e.g., "depression"). The list of mental disorders is sourced from the World Health Organization (WHO) website at https:
//www.who.int/news-room/fact-sheets/detail/mental-disorders. This scenario can be used to assess the bias aspect.
E Metric details
E.1 Human-rated metrics
We rely on human annotators to evaluate the generated images based on several aspects: alignment, quality, aesthetics, and originality. For quality, we focus on photorealism. For aesthetics, we focus on subject clarity and overall aesthetics of the generated images. The following is the full detail of the human evaluation questions.
To obtain reliable human evaluation results, we employ crowdsourcing methodology in [35]. Concrete word descriptions are provided for each question and rating choice, and a minimum of 5 crowdsourced workers evaluate each image. We use at least 100 image samples for each aspect being evaluated. For a more detailed description of the crowdsourcing procedure, see  $\S G$ .
Overall alignment. We investigate whether the generated image meets the annotators' expectations by asking them to rate how well the image matches the description using a 5-point Likert scale, similar to [35]:
How well does the image match the description?
a) Does not match at all
b) Has significant discrepancies
c) Has several minor discrepancies
d) Has a few minor discrepancies
e)Matches exactly
Photorealism. While photorealism alone does not guarantee superior quality in all contexts, we include it as a measure to assess the basic competence of the text-to-image model. To evaluate photorealism, we employ the  $\mathrm{HYPE}_{\infty}$  metric [25], where annotators distinguish between real and model-generated images based on 200 samples, with 100 being real and 100 being model-generated. Following [35], below is the multiple-choice question asked of human annotators for both real and generated images:
Determine if the following image is AI-generated or real.
a) AI-generated photo
b) Probably an AI-generated photo, but photorealistic
c) Neutral
d) Probably a real photo, but with irregular textures and shapes
e) Real photo
Subject clarity. We assess the subject clarity by evaluating whether the generated image effectively highlights the focal point, following principles commonly shared in art and visual storytelling [62]. We accomplish this by asking annotators to determine if the subject of the image is apparent over a 3-point Likert scale:
Is it clear who the subject(s) of the image is? The subject can be a living being (e.g., a dog or person) or an inanimate body or object (e.g., a mountain).
a) No, it's unclear.
b) I don't know. It's hard to tell.
c) Yes, it's clear.
Overall aesthetics. For the overall aesthetics, we aim to obtain a holistic assessment of the image's appeal by asking annotators to rate its aesthetic pleasingness:
How aesthetically pleasing is the image?
a) I find the image ugly.
b) The image has a lot of flaws, but it's not completely unappealing.
c) I find the image neither ugly nor aesthetically pleasing.
d) The image is aesthetically pleasing and nice to look at it.
e) The image is aesthetically stunning. I can look at it all day.
Overall originality. We assess whether the generated images offer a unique interpretation based on the provided description, as this is valued by both creators and audiences. We achieve this by asking annotators to rate the image's originality given the prompt:
How original is the image, given it was created with the description?
a) I've seen something like this before to the point it's become tiresome.
b) The image is not really original, but it has some originality to it.
c) Neutral.
d) I find the image to be fresh and original.
e) I find the image to be extremely creative and out of this world.
E.2 Automated metrics
CLIPScore. CLIPScore [24] measures how well an image is aligned with a corresponding natural language description using the pre-trained CLIP model [67]. It is a commonly-used metric for image-text alignment [6].
Frechet Inception Distance (FID). Frechet Inception Distance (FID) [23] is a benchmark metric used for evaluating the quality of images generated by models [6, 4, 68]. It quantifies how similar the generated images are to reference images, measured by the Inception Net [69].