SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Candice Schumann
cschumann@google.com
Gbolahan O. Olanubi
femio@google.com
Auriel Wright
aurielwright@google.com
Ellis Monk, Jr.
Harvard University*
emonk@fas.harvard.edu
Courtney Heldreth
cheldreth@google.com
Susanna Ricco
ricco@google.com
Understanding different human attributes and how they affect model behavior may become a standard need for all model creation and usage, from traditional computer vision tasks to the newest multimodal generative AI systems. In computer vision specifically, we have relied on datasets augmented with perceived attribute signals (e.g., gender presentation, skin tone, and age) and benchmarks enabled by these datasets. Typically labels for these tasks come from human annotators. However, annotating attribute signals, especially skin tone, is a difficult and subjective task. Perceived skin tone is affected by technical factors, like lighting conditions, and social factors that shape an annotator's lived experience.
This paper examines the subjectivity of skin tone annotation through a series of annotation experiments using the Monk Skin Tone (MST) scale [59], a small pool of professional photographers, and a much larger pool of trained crowdsourced annotators. Along with this study we release the Monk Skin Tone Examples (MST-E) dataset, containing 1515 images and 31 videos spread across the full MST scale. MST-E is designed to help train human annotators to annotate MST effectively. Our study shows that annotators can reliably annotate skin tone in a way that aligns with an expert in the MST scale, even under challenging environmental conditions. We also find evidence that annotators from different geographic regions rely on different mental models of MST categories resulting in annotations that systematically vary across regions. Given this, we advise practitioners to use a diverse set of annotators and a higher replication count for each image when annotating skin tone for fairness research.
Machine learning models are ubiquitous in today's society, influencing almost every aspect of our lives. Widespread adoption brings with it heightened scrutiny, with a focus on responsible model creation — from data curation to evaluations — evident in regulation and policy discussions [24, 40, 67]. As a result, understanding different human attributes and how they affect model behavior may become a standard need for all model creation and usage, from traditional computer vision tasks to the newest multimodal generative AI systems [41]. In computer vision specifically, we foresee skin tone becoming a significant part of this conversation. Indeed, the past few years have seen
*Ellis Monk performed this work while also a Visiting Researcher at Google.
Figure 1: Monk Skin Tone (MST) scale [59].
significant investment toward computer vision models that work well for everyone, enabled by improved standards for evaluation [8, 71], better data curation and analysis [22, 81], and algorithmic advances [4, 64, 76]. To leverage these advances, many practitioners rely on image datasets enriched with third-party fairness annotations, such as perceived age, gender presentation, and skin tone [77, 82]. In this paper we focus on the collection of fine-grained perceived skin tone annotations via third party annotators.
Our focus on skin tone is inspired by research in the social sciences and machine learning. Research shows that perceived skin tone affects how we view and treat other people. Specifically, it affects interpersonal interactions [44, 62, 65], job applications [12], interpersonal relationships [17], wages, educational attainment and occupation, and health outcomes among minority ethnic and racial groups in the United States [31, 52, 54, 55, 60, 61, 63, 74].
Machine learning algorithms often perform poorly on images of people with darker skin tones [38] reinforcing the importance of considering these subgroups during model selection. Skin tone measures have been a valuable tool allowing fairness researchers to address top-of-mind challenges [16]. Researchers can devise metrics to measure diversity and representation in dataset collection [70], image generation [5, 14], and image retrieval systems [10, 58]. Finer-grained skin tone scales, such as MST (see Figure 1), promise to further improve the accuracy and fairness of machine learning systems.
Annotating skin tone is an important but subjective task [34, 35, 48]. These subjective judgements can be affected by social factors and lived experience [18, 72]. For example, social factors such as political affiliation [9], or the inferred race/ethnicity of an individual [30] can affect the perception of skin tone. Noisy estimates can lead to incorrect causal attribution of model bias [42], and over-indexing on outlier annotations or using annotations that are unduly affected by transient environmental factors can overestimate measures of diversity and representation. We must address these challenges to ensure that downstream analyses using skin tone annotations are trustworthy.
This raises the question: What is the best method to systematically annotate skin tone to build datasets and benchmarks for fairness research? In this paper we focus on the following major questions:
a) Who can annotate skin tone and what are the implications for different types of annotators?
b) Does annotator location, a proxy for cultural context and lived experience, matter and have any effect on annotation quality? If so, how can this best be managed?
c) How can practitioners best navigate the complexity of skin tone annotation and are there any best practices to consider?
To address these questions, we release the MST-E dataset, a curated dataset of exemplar images with MST annotations, to help train new annotators and study the behavior of annotator pools. Based on the results, we provide suggestions for best practices for designing skin tone annotation tasks.
In this section we look into related work on subjective annotations, skin tone, and fairness in computer vision. For a more in-depth look at work in these areas see Appendix A.
Subjective Annotations Soliciting annotations for subjective concepts has been a subject of considerable research across machine learning, from natural language processing (NLP) and affective computing to classical computer vision tasks such as attribute recognition. Goyal et al. [32] found differences in annotations of toxicity in online conversations when using specialized annotator pools comprised of individuals identifying as African American and LGBTQ, respectively. Aroyo et al. [2]
annotate conversations for safety using annotator pools across two geographic regions. In both papers the authors significantly increased replications compared to standard practice, finding differences in annotator behavior between groups where a single annotator's judgements would not necessarily be stable across multiple views of the same content. Diaz et al. [20] argue for careful design of annotation tasks, accounting for the socio-cultural backgrounds of annotators and considering lived experiences as aspects of expertise. Given annotations for a subjective task from a diverse set of annotators, researchers are increasingly adopting model architectures that target individual annotators instead of or in addition to an overall consensus estimate [15, 18, 37]. Our work adds to the growing literature that considers the importance of better understanding and modeling the individual perspectives of annotators for subjective tasks, and extends the general recommendations for practitioners by focusing specifically on skin tone annotations for fairness in computer vision.
Skin Tone Research The Fitzpatrick Skin Type (FST) scale has become a gold standard of subjective skin tone measurement in dermatology [26]. The six-point scale, initially conceived as four-points, was developed as a dermatological tool to measure the response of different types of skin to phototherapy, exposure to ultraviolet light. Despite its wide use, there are some notable limitations of FST for skin tone annotation. In particular, the scale was not developed to assess skin tone; rather, skin, hair, eyes, and history of sunburn were used to assess likelihood of skin burning during phototherapy. In addition, to make the scale more inclusive of people with Asian, Indian, and African heritage, Type V and Type VI were later added to the four-point scale [27]. Even with the two darker additions, FST has been criticised for being skewed toward lighter skin tones [23, 29, 78] with researchers calling for finer grain skin tone measurements within dermatology [66] and for fairness in computer vision [3, 59]. That being said, fine-grained measurement is not without its own challenges and limitations.
Skin tone scales with too many shades, like the 36-point Felix von Luschan Skin Color chart [80], may be too granular and introduce additional cognitive load for annotators during an already challenging task, potentially reducing inter-rater agreement. In addition, it may be difficult for annotators to identify differences between skin tone shades at such fine-grained measurement, leading to lower annotator accuracy compared to an expert ground truth assessment.
We have chosen to have annotators use the 10-point MST scale (Figure 1). At 10 points, the MST scale was designed for a broader and more diverse range of skin tones compared to the FST without having too many options and overburdening annotators [59]. Although a finer-grained scale adds more complexity and difficulty to achieve consensus at the annotation stage (compared to, for example, annotations for a binarized skin tone annotation task), the MST allows for a more nuanced understanding of system behavior, better targeted disaggregated evaluations and audits, and more capable measures of representation. In addition, over the past year the MST has been adopted by computer vision fairness researchers to expand fairness annotations and evaluate models [14, 69]. We focus on exploring annotator behavior through the MST scale to better understand its potential as a more inclusive skin tone measurement tool in computer vision.
Annotations for ML Fairness in Computer Vision Perceived signals such as gender expression, skin tone, and age are commonly used in fairness evaluations and bias mitigation techniques in computer vision [8, 77, 82, 84]. There are four main methods currently used in collecting these annotations: third party crowdsourced annotations, annotations from captions, self reported annotations, and algorithmic or model generated annotations.
The most scalable solution for obtaining fairness annotations is through automated annotations with models or objective measures. However, most fairness annotation models require face or person detection or other preprocessing steps, and variable performance of upstream components across skin tone groups can lead to missing annotations for underrepresented groups during fairness evaluations [77]. In addition, these models must first be trained on annotated data. Alternatively, prior work has used measurements computed from pixel values of detected skin regions such as ITA [11] or matched skin tone pixels directly to an RGB scale [14]. In both cases these results can vary due to the influence of scene lighting or other imaging artifacts, making them less reliable as an annotation tool for in-the-wild datasets [25, 42, 43, 45, 48] (see Appendix G).
Image captions are often used to provide gender fairness annotations for images [57, 68, 86, 85]. However, relying on captions for skin tone fairness indicators is not a viable option because captions in existing datasets rarely include description of an image subject's skin tone.
Hazirbas et al. [38] released the Casual Conversations Dataset which includes both self-reported gender and age. Datasets with self-reported annotations are expensive to collect and tend to be very "clean" and not in-the-wild images or videos. There is also an important distinction between subjective attributes such as skin tone, non-subjective attributes such as gender identity and age, and the perception of these attributes. While self-reported values are extremely useful and important [38], the choice to use self-reported values vs third-party annotated values depends on the perspective practitioners want to understand: a) how models interact with the identities and non-subjective attributes of users or b) how models perform on different perceived attributes across a wide range of imagery (including generated imagery). We note that both Hazirbas et al. [38] and the expansion in Porgali et al. [69] choose the perspective of third-party annotations for skin tone.
Data collected with annotations from topical experts and crowdsourced annotators are common in computer vision. For skin tone, these annotations are often based on FST using dermatologists as annotators [8, 38, 46] due to their expertise with the FST as a dermatological assessment tool. However, topical expert annotation is not scalable to larger datasets, and has been reserved for relatively smaller annotation exercises. Crowdsourced annotations leveraging non-expert annotators has been a scalable alternative to expert annotation [38, 85]. Critically for this work, studies find that expert and crowdsourced annotators tend toward fairly similar annotations on the FST [33, 48, 50]. In this paper we take a third party annotation approach and use both topical expert and crowdsourced annotators using the MST scale.
3 The Monk Skin Tone (MST) Scale and the MST-E Dataset
The Monk Skin Tone (MST) scale defines 10 representative categories using exemplar color swatches, either as flat patches, or spheres which also incorporate shading into the stimuli. Figure 1 shows these spheres. An annotator's task is to translate from this "illustrative reference" [59] to assess the skin tone of people depicted in images captured under varying environmental conditions.
This paper introduces the open-access MST-E dataset consisting of images from 19 subjects spanning the 10 point scale (see Figure 2). The goal of this dataset is to enable and test consistent skin tone annotation across environment capture conditions. It provides a starting point to support practitioners to teach their annotators to annotate skin tone consistently on the fine-grained MST scale and a controlled set through which to study annotator behavior. See Appendix C for the MST-E data card.
MST-E models, and subsequent images, were sourced through TONL<sup>3</sup>. We selected 30 candidate subjects from an initial pool of hundreds of models, 19 of whom were eventually photographed. The 19 subjects span the full MST scale and include individuals of different ethnicities and gender identities to help human annotators decouple the concept of skin tone from race.
All images and videos of a subject were collected within a short timespan of 4 hours to eliminate skin tone variation across images due to seasonal or other temporal effects. Each subject was photographed in various poses, facial expressions (smiling, neutral, sad/frown face, and silly face), and lighting conditions. Some subjects provided a video of themselves talking to the camera, and some subjects provided images of themselves with a facial mask. See Figure 2 for example images of each of the 19 subjects. In total, the MST-E dataset contains 1515 images and 31 videos across all subjects.
The images in the dataset were reviewed and annotated using the MST scale by the authors as well as the MST creator, Dr. Ellis Monk, an expert in social perception and inequality. Images were first annotated by an author and sent to Dr. Monk for final golden annotations. Dr. Monk selected an image for each subject that best represents their skin tone (called the Golden Images set) and approved the corresponding MST annotation. We refer to these annotations as the oracle or intended annotations for the images. We curated a corresponding Challenging Images set which mirrored the Golden Images set with different lighting conditions. See Figure 4(b) for an example of "golden" versus "challenge" images and Appendix B for the full Golden Images and Challenging Images sets.
Consent and terms of use Each subject provided consent for their images and videos to be released. TONL has given permission for these images to be released and used for research, model evaluations,
$^{2}$ https://skintone.google/mste-dataset
<sup>3</sup>https://tonl.co/
Figure 2: The MST-E dataset contains 19 subjects that span the full MST scale. Each subject has images in a variety of lighting conditions and poses. Images by TONL. Copyright TONL.CO 2022 ALL RIGHTS RESERVED. Used with permission.
or annotator-training purposes only. The images cannot be used to train machine learning models. The images and annotations are licensed under CC BY.
4 The Annotation Experiment
We perform two different studies in this research. The first is a qualitative survey with a group of skin tone "experts", photographers from two different geographical regions. We describe our design choices concerning the qualifications of these experts in detail in Section 5. The "experts" annotated the Golden and Challenging MST-E sets (§3). The second study expands on the results of the small scale study in two ways: extending to a larger pool of trained crowdsourced annotators located in five different geographical regions, and to an ML-training-scale dataset (§6). Crowdsourced annotators were asked to annotate skin tone across the full MST-E dataset and a subset of the Open Images dataset [49] to explore whether findings from the controlled settings generalized to practical settings. Both of these studies were conducted under the guidance of our institution's ethics body.
In each study, we examine the degree to which annotators within the same region agree with each other, whether annotators from different regions agree, and how well the consensus annotations match the intended annotation across different conditions. In the following sections, when calculating the consensus MST annotation for an image we use the median MST annotation from the responses.
Typically, the only available indicator of annotation quality is a measure of inter-rater reliability. However, in this instance we have MST annotations provided by the creator of the MST scale. Thus, we have the unique opportunity to compare annotations from the annotators to his annotations.
Previous research has taken a similar approach comparing FST annotations between crowdsourced annotators and an expert dermatologist, and has found that crowdsourced annotations are typically within 1-point of expert dermatologist annotations [33]. This threshold of 1-point discrepancy has continued to be used when comparing skin tone annotations between crowdsourced annotators and experts [34]. Here, we consider alignment between the consensus from the photography experts or crowdsourced annotators and Dr. Monk's intent using two metrics: the percentage of median MST annotations that are within 1-point of discrepancy from the oracle annotations (higher is better), and the average median distance, the expected difference between the median annotation from a group of annotators and the oracle computed across all images (ideally less than 1).
5 Exploration with Experts
We used Gerson Lehrman Group (GLG) to recruit and compensate a panel of five experts in photography in India and five experts in photography in the United States to complete a skin tone annotation task. Experts in India were selected because we later recruit crowdsourced annotators from this market. The U.S. was selected because we sought a geographically distant comparison market in which GLG could recruit a similarly sized panel of experts. Similar skin tone annotation research has sought annotations from dermatologists [34], but we did not want experts who were practiced in reviewing images taken under highly controlled conditions for medical issues. Rather, we wanted experts who
were practiced in capturing and editing still images of people and examining skin in their photography. During an initial pilot we found that different types of experts may a) approach the task differently based on their discipline and b) this might result in differences in annotation. Specifically we found that photographers tended to approach the task as designed, assessing the skin tone in the image while correcting for extraneous image features. However, dermatologists interviewed in the pilot study were more likely to attempt to assess skin tone beneath clothing which was not depicted. This aligned with skin tone assessment in a medical scenario, but not with our goal of achieving skin tone annotation in images that aligns with social perception. For this purpose we recruited photographers who self-reported that people and portraits were the subjects of their work, had previous experience reviewing and editing skin tone in color correction software, and were most likely to approach the task as designed.
Experts were asked to annotate a total of 50 images, 19 from the MST-E Golden Images or Challenging Image sets (§3) and 31 aesthetically similar distractor images. Images were presented in a randomly assigned order and the participants were asked to view each image and select which skin tone on the MST best matched the person depicted in the image. The task was completed twice. Participants first annotated the Golden Images set and distractor images. Approximately one week later, participants annotated the Challenging Images set and a new set of distractor images. The experimental setup was designed to examine consistency of annotation across lighting conditions, with the time delay reducing the likelihood that participants would annotate the Challenging set based on their memory of their previous selection.
Experts were shown an image and asked to select the skin-tone that matches most closely on the MST scale, to select which part of the person's body they used in their assessment, and to rate on a 5-point scale the ease/difficulty of the annotation. An example of the task can be found in Appendix D.
Inter-Rater Reliability We examine if the skin tone annotations are consistent among experts within each market using intraclass correlation (ICC), a measure of inter-rater reliability. Each expert reviewed each image, so we use a two-way random, single score ICC where the subject and annotator are viewed as random effect [56]. We find that experts in the tested markets do consistently annotate skin tone on the MST. Figure 3(a) shows significant,  $p < .05$ , intraclass correlation in both markets and in both image quality conditions.
Regional Differences in Annotation We conducted a series of Mann Whitney U tests [53] on each individual image to determine if there were any significant differences in the annotation between the experts from the United States and those in India (see Appendix E for full comparisons). Differences were significant using  $p < 0.05$  for three of the images. Figure 4 shows the annotations for two of these in detail. The professional photographers in India tended to annotate the subject's skin as lighter (MST3 - MST5). Professional photographers in the U.S. tended to annotate the subject's skin as darker (MST5 - MST7). The differences observed provide suggestive evidence that geographic location affects
(a) Photography Experts ICC
(b) Crowdsource ICC
Figure 3: Inter-rater reliability (IRR) between annotators calculated via intraclass correlation (ICC). Figure 3(a) shows the ICC scores for expert photographers in the US and India on the Challenging set and the Golden set. Figure 3(b) shows the ICC scores for crowdsourced annotators in each region, as well as the global pool. These ICC scores are calculated per subject, as well as over the Challenging and Golden sets.
skin tone annotation, which we explore further and at scale using crowdsourced annotators.
One of the MST-E subjects was excluded from this study due to a programming error.
(a) Golden Image