SlowGuess/NeurIPS_2023_OCR huggingface dataset:
(b)Challenging Image
Figure 4: Photography experts in India (top plots) and the US (bottom plots) MST annotation counts for Subject 14. Figure 4(a) shows Subject 14's Golden Image chosen by the MST scale expert (\$3). Figure 4(b) shows Subject 14's matching Challenging Image. In large-scale experiments with trained annotators (\$6), we found consistently measurable differences in annotator behavior based on region.
Distance from Intended Annotations In both markets we found that a high percentage of the consensus annotations were with 1-point of discrepancy of the oracle annotations,  $88.9\%$  in India and  $83.4\%$  in the United States. In addition, in both markets the average median distance from the oracle annotation was not above the 1-point guardrail established in previous research [33, 34], 0.78 in India and 0.84 in the United States. These results suggest that photography experts in India and the U.S. annotate skin tone in a manner that aligns with the scale creator's intent.
6 Exploration of Crowdsourced Annotations
We collected annotations for the entire MST-E dataset and a subset of the Open Images dataset from annotators in India, the Philippines, Brazil, Hungary, and Ghana. We followed standard processes for data annotation for computer vision-related work, using the services of professional data annotators without specific domain expertise outside of the instructions they received for this task. Annotators are compensated for their work (see Appendix K for more details). Each task was replicated 5 times in each region resulting in 25 annotations per image.
During a task, annotators are shown a thumbnail of a subject and the full image with a box drawn around the location of the thumbnail. This allows annotators to understand the context and lighting of the image. Annotators were asked to select the best skin tone for the subject and report which part of the body they primarily used to make the selection (Face, Torso, Arms/Hands, Legs/Feet, or Other). Annotators could select that an image was difficult to annotate for skin tone, and skip the task if no skin was visible. See Appendix D for an example of the selection tool.
Since the MST-E dataset contains many images for each subject, the images were mixed into a much larger set of similar images to prevent face recall among annotators. In the results section we look at the annotations Per Subject - meaning annotations across all images of a single subject - as well as the annotations for the Golden Images and Challenging Images sets (\$3).
Inter-Rater Reliability Five annotators were randomly selected from a broader pool of annotators within each market to annotate each image. As the group of annotators was not consistent from image to image, we use a one-way random, single score ICC to measure inter-rater reliability, where the subject is a random effect for this analysis [56].
Consistent with our results from professional photographers, we find that non-expert annotators also consistently annotate skin tone on the MST, using a shared mental model of the MST categories. Figure 3(b) shows significant,  $p < 0.05$ , ICC values for each region and annotation task. For every region, the ICC is very high with Per Subject ICC ranging from 0.86 to 0.94. In addition, we find some evidence that annotators remain consistent in skin tone annotation even when image quality is diminished. The Golden Images ICC ranged from 0.9 to 0.96, and Challenging Images ICC ranged from 0.82 to 0.95 – a comparatively lower range but still significant and consistent.
Regional Differences in Annotation We found suggestive evidence for regional differences in annotation in Section 5, and know that geographical region affects annotations in other domains like opinion of the safety of chatbots [2]. Unlike the exploratory analysis of experts that used a subset of the MST-E dataset, in this setting we have enough statistical power to explicitly test the effect of the annotator's region on MST annotation. We performed a multivariate analysis of variance (MANOVA) [28] with annotator region as the independent variable and MST annotations for each subject as the dependent variable. We found that the subject annotations were a significantly high predictor of the region of the annotators (Pillai's trace  $= 1.97$ ,  $p < 0.05$ ). This result suggests that the shared mental models of skin tone may vary significantly across different geographic regions.
Distance from Intended Annotations Despite measurable regional variation, we find a high percentage of median MST annotations that are within 1 point of oracle annotations. Using the global pool, the consensus annotation agreed within 1 point for all but one subject (94.7%). In addition, we find the average median distance to oracle annotations to be less than 1 for every region and image condition. The distance was 0.78 for both well lit and poorly lit images. This suggests that crowdsourced annotators, similar to the expert photographers, arrive at consensus MST annotations that are similar to the scale creator's intent.
Replications The 25 annotations we acquired per image is larger than most practitioners collect. We use the Spearman-Brown prediction formula to determine inter-rater agreement over  $k$  replications, where  $k$  is the total number of replications over all regions (See Figure 5) [7, 79, 83]. We find that for datasets containing high quality images or annotations of a single individual based on a consensus across multiple images, one rater per region likely suffices (5 total annotators). For more challenging images, two annotators per region is beneficial. Therefore, if a practitioner's budget is limited, we would suggest filtering for images with good lighting conditions (or even images with okay lighting conditions). If annotations on images with low quality lighting is necessary, practitioners can then annotate with 10 annotators on just those low quality images.
Extension to In-the-Wild Annotations Open Images is a large-scale image dataset with a number of image and object annotations [49]. We randomly selected a subset of 41,471 images that contained at least one face. In the 41,471 images, 86,809 faces were present to annotate. The annotators used the same annotation tool as they did with the MST-E dataset (see Appendix D). Using this tool annotators could see the face thumbnail along with its location in the full image.
Unlike the MST-E dataset where images were taken in a controlled setting, Open Images is an "in-the-wild" dataset containing many more unique individuals and images of more varying quality. Even in these uncontrolled settings we expect that crowdsourced annotators will still consistently annotate skin tone on the Monk Skin Tone scale and that geographic region will have a significant effect on the annotations.
Figure 5: Spearman-Brown inter-rater reliability calculations over replications of annotators on the MST-E dataset per subject, over the Challenging Images, and over the Golden Images.
Consistent with our hypotheses, we find a high intraclass correlation across markets ranging from 0.875 to 0.91. In addition, the intraclass correlation from the Open Images dataset has no significant differences than crowdsourced annotators intraclass correlation in the MST-E dataset ( $\chi^2 = 0.26$ ,  $p = 0.99 > 0.05$ ). This suggests that crowdsourced annotators are similarly consistent in MST annotations in controlled datasets and in-the-wild data sets. We also find significant differences between annotation distributions between regions ( $V^2 = 0.0766$ ,  $p < 0.05$ ), consistent with the MST-E annotations. Overall, this shows that recommendations based on annotator behaviors observed in our controlled studies likely translate to real-world scenarios commonly faced by ML practitioners.
Figure 6: Comparison of the distribution of skin tone annotations in the MST-E dataset ("MSTE" in the plot) versus the "in-the-wild" Open Images dataset ("OI" in the plot). The Open Images dataset has a strong skew towards lighter skin tones.
7 Discussion and Suggestions for Practitioners
One of the first design decisions practitioners have to make when starting a skin tone annotation task is the composition of the annotator pool. Following from common practices in literature [8, 85], we analyse MST annotations from subject matter experts, professional photographers who focus on photographing people, and crowdsourced annotators who were provided training in MST skin tone annotations prior to the annotation task. We find that both the photographers and trained annotators were able to reliably annotate for skin tone suggesting that subject matter experts are not necessarily needed for MST skin tone annotation tasks. This allows for more scalable annotation tasks using trained annotators on large datasets.
Additionally we find that although trained annotators have high inter-rater reliability across all regions, there are regional differences between the MST annotations, suggesting that regional cultural contexts influence how people perceive skin tone. When combining the annotations across a variety of regions, we find that the median of the annotations are consistently close to Dr. Monk's annotations. This suggests that a geographically diverse annotator pool provides annotations that are in line with the intended use of the MST scale [59].
We find that annotators are able to learn to be invariant to environmental factors such as lighting quality, and find that they perform similarly on the in-the-wild subset of Open Images to the MST-E dataset, suggesting that performing a skin tone annotation task on in-the-wild images is possible.
The MST-E dataset is a valuable tool to help practitioners train their human annotators to annotate MST on images. But beyond that use case, the dataset can also be used to evaluate models. Particularly, the MST-E dataset can be used to evaluate fairness over the full MST scale as well as to evaluate performance of models over varying lighting conditions and poses. This is particularly useful since "in-the-wild" datasets like Open Images may have a large skew in skin tone distribution (See Figure 6).
Limitations and Ethical Considerations This paper discusses the variance introduced by different geographical regions and types of annotators. However, it does not consider other aspects of annotator identity and background, or other factors such as the language used in the task, different UI improvements, and if annotators are affected by other people in the same image. Future work should look at these issues.
This work suggests the use of 10 annotators per image if image quality is unknown. This is a very large jump from the status quo of 3 annotators per image, which could be a limiting factor for practitioners with lower budget. A potential alternative for very restricted budgets would be to use a large and geographically diverse pool of annotators but have only a subset of annotators provide explicit judgements for each image. One would then impute missing annotations based on a machine learned model personalized for each annotator. Similar approaches have been tested for other types of subjective annotations (in NLP [18] and in face similarity [1]) but not yet for skin tone. These methods should be investigated in future work.
The MST-E dataset has a small number of subjects (19), and while they do span the complete MST scale, they do not capture every variation in skin tone. Future work should extend this dataset to more subjects. In addition, the exploratory analyses of annotations from photography experts is more limited than the scale achieved with crowdsourced annotators. That being said, with the additional analysis of the Open Images dataset, we believe our results will translate to other domains. While we use thresholds for agreement (e.g., 1-point discrepancy) that are in line with prior work when describing the resulting annotations as reliable, future work should focus on usefulness of the annotations in fairness analysis or mitigations leading to more inclusive and robust computer vision systems.
The MST-E dataset release enables the annotation of an aspect of a person's appearance which has historically been the source of discrimination and continues to affect how people are treated in society. While there is the potential for malicious use of skin tone analysis, we believe the clear benefits for fairness research outweigh the potential risks. We chose not to collect or release other sensitive information about the subjects of the dataset or our annotators, beyond general geographic region, to protect their privacy. While we have consent from participants to release the MST-E dataset for research and annotator training purposes, there remains a risk that the images could be used maliciously in model training. In order to mitigate these risks we clearly state the terms of use and do not allow for anonymous access to the image dataset.
Suggestions for practitioners Extrapolating from our results, we make the following suggestions for practitioners who are pursuing a task of MST scale annotation.
- Diversity of annotators is important for subjective annotations. In our case cultural differences across geographic regions influenced skin tone annotations. Other subjective annotations may require different types of diversity (i.e., gender in heteronormative communities vs members of the LGBTQ+ community).
- Given the wide range of annotations we suggest having more than three annotators per subject (the current status quo) and instead have at least two annotators in at least five different geographical regions (10 ratings per image), particularly for challenging images.
- If you know the geographical region of your data or deployment scenario, it may be wise to use annotators from the same region. We recommend considering this design choice due to the difference in behaviors across geographic pools we observed. We hypothesize that co-located annotators may have a better cultural context for certain annotation tasks that would provide the most relevant data, although more careful study of this is needed.
- Train annotators with examples of individuals across the full MST spectrum in a variety of conditions. The MST-E dataset can be used to perform this training.
Fairness signals/attributes are important to determine how well models, and eventually products using these models, work for every person. Skin tone is particularly important for computer vision models and applications. However, annotating skin tone (even annotating one's own skin tone) is a subjective task. We make the following contributions:
a) We show that skin tone can be annotated reliably on the MST scale by both trained crowdsourced annotators and subject-matter experts such as professional photographers.
b) We show that annotators have an understanding of skin tone across conditions and, with training, can provide annotations invariant to lighting conditions.
c) We show that cultural contexts across geographical regions influence MST annotations. However, when collecting annotations from a diverse geographical pool, annotators are able to provide annotations that match the original intention behind the MST scale.
d) We provide actionable suggestions to practitioners collecting subjective annotations.
e) We release the MST-E dataset to help practitioners train annotators to be invariant to lighting conditions and support further research.
Although fine-grained skin tone annotation is a difficult task, it is possible and it opens up the door to more nuanced understanding of bias in model performance.
We wish to thank our colleagues across Google working on fairness and inclusion in computer vision for their contributions to this work, especially Marco Andreetto, Parker Barnes, Ken Burke, Benoit Corda, Tulsee Doshi, Rachel Hornung, David Madras, Shrikanth Narayanan, Utsav Prabhu, Sagar Savla, Alex Siegman, Komal Singh and Biao Wang. We also would like to thank Annie Jean-Baptiste, Florian Koenigsberger, Marc Repnyek, Maura O'Brien, and Dominique Mungin and the rest of the team who help supervise, fund, and coordinate our data collection.
[74] Igor Ryabov. Colorism and educational outcomes of asian americans: evidence from the national longitudinal study of adolescent health. Social Psychology of Education, 19:303-324, 2016.
[75] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade W Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022.
[76] Candice Schumann, Xuezhi Wang, Alex Beutel, Jilin Chen, Hai Qian, and Ed H Chi. Transfer of machine learning fairness across domains. arXiv preprint arXiv:1906.09688, 2019.
[77] Candice Schumann, Susanna Ricco, Utsav Prabhu, Vittorio Ferrari, and Caroline Pantofaru. A step toward more inclusive people annotations for fairness. In Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pages 916-925, 2021.
[78] Marilyn S Sommers, Jamison D Fargo, Yadira Regueira, Kathleen M Brown, Barbara L Beacham, Angela R Perfetti, Janine S Everett, and David J Margolis. Are the fitzpatrick skin phototypes valid for cancer risk assessment in a racially and ethnically diverse sample of women? Ethnicity & disease, 29(3):505, 2019.
[79] Charles Spearman. Correlation calculated from faulty data. British journal of psychology, 3(3):271, 1910.
[80] Felix von Luschan. Beiträge zur Völkerkunde der deutschen Schutzgebiete. D. Reimer, 1897.
[81] Angelina Wang, Alexander Liu, Ryan Zhang, Anat Kleiman, Leslie Kim, Dora Zhao, Iroha Shirai, Arvind Narayanan, and Olga Russakovsky. Revise: A tool for measuring and mitigating bias in visual datasets. International Journal of Computer Vision, 130(7):1790-1810, 2022.
[82] Zeyu Wang, Clint Qinami, Ioannis Christos Karakozis, Kyle Genova, Prem Nair, Kenji Hata, and Olga Russakovsky. Towards fairness in visual recognition: Effective strategies for bias mitigation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8919-8928, 2020.
[83] Ka Wong and Praveen Paritosh. k-rater reliability: The correct unit of reliability for aggregated human annotations. arXiv preprint arXiv:2203.12913, 2022.
[84] Yu Yang, Aayush Gupta, Jianwei Feng, Prateek Singhal, Vivek Yadav, Yue Wu, Pradeep Natarajan, Varsha Hedau, and Jungseock Joo. Enhancing fairness in face detection in computer vision systems by demographic bias mitigation. In Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pages 813-822, 2022.
[85] Dora Zhao, Angelina Wang, and Olga Russakovsky. Understanding and evaluating racial biases in image captioning. In International Conference on Computer Vision (ICCV), 2021.
[86] Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. Men also like shopping: Reducing gender bias amplification using corpus-level constraints. arXiv preprint arXiv:1707.09457, 2017.
A Expanded Related Work
A.1 Subjective Annotations
Soliciting annotations for subjective concepts is required for many modern machine learning tasks. As such, it has been a subject of considerable research across subfields, from NLP and affective computing to classical computer vision tasks such as attribute recognition. Our work adds to the growing literature that raises the importance of better understanding and modeling the individual perspectives of annotators for subjective tasks, especially as influenced by demographic or cultural factors, and we highlight a few examples here from across disciplines. Goyal et al. [32] study differences in annotations of toxicity in online conversations when using specialized annotator pools comprised of individuals identifying as African American and LGBTQ, respectively. Aroyo et al. [2] annotate conversations for safety using annotator pools across two geographic regions. They significantly increased replications compared to standard practice, finding differences in annotator behavior between groups where a single annotator's judgements would not necessarily be stable across multiple views of the same content. Diaz et al. [20] argue for careful design of annotation tasks, accounting for the socio-cultural backgrounds of annotators and considering lived experiences as aspects of expertise. Given annotations for a subjective task from a diverse set of annotators, researchers are increasingly adopting model architectures that target individual annotators instead of or in addition to an overall consensus estimate [18, 15, 37]. Our work supports these findings and extends the general recommendations for practitioners by focusing specifically on how they apply to annotations for fairness in computer vision.
A.2 Skin Tone Research
There are a number of different ways to describe skin tone that have been used in the social sciences, dermatology, and computer vision. Here, we focus on the most common definitions used in computer vision fairness research, including more options and more detail than discussed in the main paper.
The most common objective method used by computer vision scientists and AI researchers is the individual typology angle (ITA), which is a metric based on the  $L^{*}$  (lightness) and  $B^{*}$  (yellow/blue) components of the CIE  $L^{*}a^{*}b^{*}$  color space, defined as  $ITA = \arctan \left(\frac{L^{*} - 50}{b^{*}}\right)\left(\frac{180}{\pi}\right)$ . Skin color can then be classified using the ITA according to six categories, ranging from very light (category I) to dark (category VI) [11]. Measured ITA has been shown to be correlated with physical properties such as melanin content of skin [19]. The mathematical definition is conceptually appealing, with researchers describing it favorably as, for example, being "easily computed without subjective annotations." [43]. However, ITA values computed from images captured in uncontrolled environments can vary due to the influence of scene lighting or other imaging artifacts [73], making it less reliable as an annotation tool for in-the-wild datasets. Furthermore, in practice the continuous ITA values are often mapped to different categorical scales using boundaries between categories that reintroduce elements of subjectivity [11, 19].
The Fitzpatrick Skin Type (FST) scale has become a gold standard of subjective skin tone measurement in dermatology. The six-point scale, initially conceived as four-points, was developed as a dermatological tool to measure the response of different types of skin to phototherapy, exposure to ultraviolet light. For example Type I, the lightest skin type, is described as "Burns easily, never tans" and Type VI, the darkest skin type, is described as "Never burns, tans profusely" [27]. Outside of dermatology, computer vision scientists have also used the FST to label images for skin tone.
Most studies use either expert dermatologists or crowdsourced annotators to assign skin tone labels based on FST [46, 38]. Though lighting and image quality may pose a challenge in photo evaluation, experts and crowdsourced annotators tend toward fairly accurate FST annotations where judgements are expected to be within one FST value depending on image conditions [50, 48]. For example, an audit of the Fitzpatrick 17k dataset found their crowdsourced annotators had between  $71\%$  and  $85\%$  within one point accuracy to board-certified dermatologist ratings [33].
Despite its wide use, there are some notable limitations of FST for skin tone annotation. In particular, the scale was not developed to assess skin tone; rather, skin, hair, eyes, and history of sunburn were used to assess likelihood of skin burning during phototherapy. In addition, to make the scale more inclusive of people with Asian, Indian, and African heritage, Type V and Type VI were later
added to the four-point scale [27]. These additions to the scale implied a strong correlation among race/ethnicity, skin tone, and sun reactivity; however, self-reported race/ethnicity and skin tone have been found to be significant but weak predictors of FST [39]. That is, FST values may actually serve as a significant, but weak, correlate to one's actual skin tone.