Comparing Human and Machine Bias in Face Recognition
Authors: Samuel Dooley
George Zhihong Wei
Ryan Downing
Nathan Shankar
Bradon Michael Thymes
Gudrun Lilja Thorkelsdottir
Tiye A Kurtz-Miott
Rachel Mattson
Olufemi Obiwumi
Valeriia Cherepanova
Micah Goldblum
John P Dickerson
Tom Goldstein
Venue: Submitted to ICLR 2023 (2023)

Abstract: Much recent research has uncovered and discussed serious concerns of bias in facial analysis technologies, finding performance disparities between groups of people based on perceived gender, skin type, lighting condition, etc. These audits are immensely important and successful at measuring algorithmic bias but have two major challenges: the audits (1) use facial recognition datasets which lack quality metadata, like LFW and CelebA, and (2) do not compare their observed algorithmic bias to the biases of their human alternatives. In this paper, we release improvements to the LFW and CelebA datasets which will enable future researchers to obtain measurements of algorithmic bias that are not tainted by major flaws in the dataset (e.g. identical images appearing in both the gallery and test set). We also use these new data to develop a series of challenging facial identification and verification questions that we administered to various algorithms and a large, balanced sample of human reviewers. We find that both computer models and human survey participants perform significantly better at the verification task, generally obtain lower accuracy rates on dark-skinned or female subjects for both tasks, and obtain higher accuracy rates when their demographics match that of the question. Academic models exhibit comparable levels of gender bias to humans, but are significantly more biased against darker skin types than humans.

Under review as a conference paper at ICLR 2023
COMPARING HUMAN AND MACHINE BIAS IN FACE RECOGNITION
Anonymous authors Paper under double-blind review
ABSTRACT
Much recent research has uncovered and discussed serious concerns of bias in facial analysis technologies, finding performance disparities between groups of people based on perceived gender, skin type, lighting condition, etc. These audits are immensely important and successful at measuring algorithmic bias but have two major challenges: the audits (1) use facial recognition datasets which lack quality metadata, like LFW and CelebA, and (2) do not compare their observed algorithmic bias to the biases of their human alternatives. In this paper, release improvements to the LFW and CelebA datasets which will enable future researchers obtain measurements of algorithmic bias that are not tainted by major flaws in the dataset (e.g. identical images appearing in both the gallery and test set). We also use these new data to develop series of challenging facial identification and verification questions that we administered to various algorithms and large, balanced sample of human reviewers. We find that both computer models and human survey participants perform significantly better at the verification task, generally obtain lower accuracy rates on tasks, and obtain higher accuracy rates when their demographics match that of the question. Academic models exhibit comparable levels of gender bias to humans, but are significantly more biased against darker skin types than humans.
or female subjects for both
INTRODUCTION
Facial analysis systems have been the topic of intense research for decades, and instantiations of their deployment have been criticized recent years for their intrusive privacy concerns and
differential treatment of various demographic groups. Companies and governments have deployed
facial recognition systems (Derringer) 2019; Hartzog 2020: Weise & Singer 2020) which have
wide variety of applications from relatively mundane, e.g., improved search through personal
photos GGogle 202T). to rather controversial, e.g., target identification in warzones (Marson & Forrest 2021). A flashpoint issue for facial analysis systems is their potential for biased results by demographics (Garvie 2016: Lohr 20188) Buolamwini & Gebru 2018 Grother et al. [019 Dooley al. 2021), which make facial recognition systems controversial for socially important applications, such as use in law enforcement or the criminal justice system. To make things worse, many studies of machine bias in face recognition use datasets which themselves are imbalanced or riddled with errors,
resulting in inaccurate measurements of machine bias.
It is now widely accepted that computers perform as well as better than humans on variety of facial recognition tasks (Lu & Tang. 2015 Grother et al., 2019) in terms of accuracy, but what about bias? The algorithm's 's superior overall performance, as well as speed to inference, makes the use of facial recognition technologies widely appealing in many domain areas and comes at enhanced costs to those surveilled, monitored, or targeted by their use (Leiss) 2019 Kostka al.) (02T). Many previous studies which examine and critique these technologies through algorithmic audits do only up to the point of the algorithm's biases. They stop short of comparing these biases to that of their human alternatives. In this study, question how the bias of the algorithm compares to human bias in order to fill in one of the largest omissions in the facial recognition bias literature.
We investigate these questions by creating dataset through extensive hand curation which improves upon previous facial recognition bias auditing datasets. using images from two common facial recognition datasets (Huang al. 2008 Liu al. 2015) and fixing many of the imbalances and

---

Under review as a conference paper at ICLR 2023
erroneous labels. Common academic datasets contain many flaws that make them unacceptable for this purpose. For example, they contain many duplicate image pairs that differ only in their compression scheme cropping. As result, it is quite common for an image to appear in both the gallery and test set when evaluating image models, which distorts accuracy statistics when evaluating on either humans machines. Standard datasets also contain many incorrect labels and low quality images, the prevalence of which may be unequal across different demographic groups.
We also create survey instrument that we administer sample of non-expert human participants (n = 545) and ask machine models (both through academically trained models and commercial APIs) the same survey questions. In comparing the results of these two modalities, we conclude that, first, humans and academic models both perform better on questions with male subjects. Second, humans and academic models both perform better on questions with light-skinned subjects. Third, humans perform better on questions where the subject looks like they do. Fourth, commercial APIs are phenomenally accurate at facial recognition and we could not evaluate any major disparities in their performance across racial gender lines. Finally, overall we found that academic models exhibit comparable levels of gender bias to humans, but are significantly more biased against darker skin types than humans.
2 BACKGROUND AND PRIOR WORK
We provide brief overview of facial recognition and additional related work. We further detail similar comparative studies which contrast the performance of humans and machines. Much of the discussion of bias overlaps with the sub-field of machine learning that focuses on social and societal harms. We refer the reader to Chouldechova Roth (2018) and Barocas et al. (2019) for additional background of that broader ecosystem and discussion around bias in machine learning.
Facial Recognition In this overview, we focus review of the types of facial recognition technology rather than contrasting different implementations thereof. Within facial recognition, there are two large categories of tasks: verification and identification. Verification asks a I-to-1 question: is the person in the source image the same person as in the target image? Identification asks a I-to-many question: given the person in the source image, where does the person appear within a gallery composed of many target identities and their associated images, if at all? Modern facial recognition algorithms, such as He et al. (0166); Chen et al. (2018); : Wann et al. (2018) and Deng et al.. (2019), use deep neural networks to extract feature representations of faces and then compare those to match individuals. An overview of recent research on these topics can be found in Wang & Deng (2018). Other types of facial analysis technology include face detection, gender or age estimation, and facial expression recognition.
Bias in Facial Recognition Bias has been studied in facial recognition for the past decade. Early work, like that of Klare et al. (2012) and ToTole et al. (2012). focused on angle-demographie effects (specifically, race and gender), whereas the more recent work Buolammini & Gebru (2018) uncovers unequal performance from an intersectional perspective, specifically between gender and skin tone. The latter work has been and continues to be hugely impactful both within academia and at the industry level. For example, the 2019 update to NIST FRVT specifically focused on demographic mistreatment from commercial platforms (Grother et al. 2019.
While our work focuses on the identification and comparison of bias, existing work on remedying the ills of socially impactful technology and unfair systems can be split into three (or, arguably, four (Savani et al.] 20200) focus areas: pre-, in-, and post-processing. Pre-processing work largely focuses on dataset curation and preprocessing (c.g., Fellmaam et al., 2015 Ryu et al. 2018 Quadrianto
et al.
2019 Wang & Deng 2020). In-processing often constrains the ML training method or
optimization algorithm itself (e.g.. 7afar al. 2011aa] b. Agarwal et al. 2018 Donini al.
2018
Goel et al. 2018 Zafar et al. 2019; Diana et al., 2020 Lahoti et al. 2020: Martinez et al. 2020
Padala & Gujar 2020: Wang & Deng 2020), or focuses explicitly on so-called fair representation
learning (c.g.. Dwork et al. 2012 Zemel et al. 2013) Edwards & Storkey 2016 Beutel et al.
2017 Madras et al. 2018 Wang et 2019
Adeli Adel
al. 2021). Post-processing techniques adjust
decisioning at inference time to align with fairness definitions (e.g., Hardt et al. 2016 Wang et al.) 2020).
Human Performance Comparisons No work in the past to our knowledge has specifically focused on the question of comparing bias or disparity between humans and machines. Some prior work
2

---

Under review as a conference paper at ICLR 2023
Caminnas
(a) Incorrect Identities: these are la- heled as the same; but the left is Zoë Lister and the right is Zoe Lister-Jones.
(b) Incorrect Labelling: this in- dividual was labeled as not be-
(c) Black and White Images: some identities only have black and white photos.
ing pale skinned.
IVIDNI
Jt QU
Low-Quality Images
(e) Identical Attre/Background
Multiple Distinct Faces
Figure 1: Shortcomings present in existing facial identification datasets
has looked comparing overall performance or accuracy between the two groups. Tang & Wang (2004); Toooee et al. (2007): Phillips & O' toool (2014) compare human and computer- camputer-based face verification performance. [u & Tang (2015) was the first paper show machine accuracy outpacing human accuracy. Hu et al. 2)17);; Phillips et al. (2018); Robertson et al. (2016) compared face recognition performance of human specific sub-populations whereas White et al. (2015) looked at comparing overall performance of humans who use the oulputs of face recognition systems.
INTERRACE DATASET CURATION
We endeavor to answer two research questions: (RQ1) How and to what extent do humans exhibit bias their accuracy in facial recognition tasks? (RQ2) How does this compare to machine learning- based models? In order to answer these questions, created set of challenging identification and verification questions which we posed humans and machines from a novel dataset called InterRace for its application in intersectional facial recognition. The protocol around those experiments are described in Section 4
To create our dataset, we first ensured that we had accurately labeled and balanced metadata. This required us to hand-check all the labels in the dataset. After removing poor quality and redundant images, found that LFW lacked identities with dark skin tones, which is why further identities were drawn from CelebA. Though LFW does have an errata page, CelebA and other facial recognition datasets known have many missing incomplete metadata, and all CelebA images examined by an author of this paper before adding them to the dataset. Finally, after randomly generating survey questions, we hand checked that there were no questions for which the answer is apparent or unclear for reasons other than properties of the faces (see Figure In this section we detail our findings about the shortcomings in the metadata labels from LFW and CelebA and outline the steps we took to rectify and supplement these in the creation of the InterRace identities.
THE SHORTCOMINGS OF PREVIOUS DATASETS
In the process of trying to create reasonable set of identification and verification questions, we identified that the LFW and CelebA datasets generally suffer from range of problems that distort accuracy and bias metrics. We summarized these problems Figure
The first challenge we had to overcome is incorrect identities; this includes incorrect names, duplicated identities, as well as clearly incorrect matching between image and name. This problem is particularly harmful for facial recognition models which would be provided with galleries containing incorrect information about identities. In some cases, identities were split across multiple labels due spellings. We found that this happened almost exclusively with non-canomically western names. E.g., Mesut Ozil (labelled as "Mesut Zil"), , Jithan Ramesh (labelled as "Githan Ramesh"), Isha Koppikhar (labelled "Fesha Koppikhar"), etc. Examples of incorrect identity labels include
3

---

Under review as a conference paper at ICLR 2023
Neela Rasgotra, fictional character played by Parminder Singh and "All That Remains," band name with the pictured individual being Philip Labonte. In other cases, multiple distinct identities were merged into the same label. In CelebA, Jennifer Lopez was grouped with Jennifer Driver, and Zoë Lister and Zoe Lister-Jones were both listed under "Zoe Lister" (pictured in Figure Tap.
by
Additionally, these datasets exhibit metadata labelling problems that manifest in two ways: (1) clearly defined labels being incorrectly or non-uniformly applied, and (2) vague and sometimes harmful metadata. In the first category, CelebA has features such as gender and age which often are incorrect or mislabeled (i.e. a pale-skinned person being labelled as not having pale skin, Figure lbp. Further, many categories in CelebA are subjective and/or harmful. For example, there is a label for "Attractive." "Big Nose/Lips." or "Chubby."
We found that some identities have exclusively black and white images (Figure Icp. making trivial to identity two photos as being of the same label.
We filtered out low-quality images that could not be easily identified for reasons beyond properties of the face, such as poor light exposure, blurriness, facial obstruction, etc. We also removed "old-timey" photos that were easily associated with a specific time period, as this makes easy to match them with other similar photos.
We found that many questions could be answered without considering face features at all, and these were removed. For example if the subject is wearing identical attire and/or standing in front of an identical background
two images. Many identities contained multiple images from the same red carpet event or award reception (Figure Teb. It very often happens that the same image appears multiple times in the dataset, but with slightly different crops, compression, or contrast adjustments.
Finally, some images contained multiple faces. Some of these pictures clearly have one person in the foreground and are therefore not problematic, but in others this is not the case, creating ambiguity as to which person is the target individual. See Figure
The image types above create inaccuracies when evaluating face recognition systems and distort measurements of bias when these problems occur at rates that differ across groups. For this reason, many datasets designed for training face analysis systems are not appropriate for evaluating bias.
3.2 THE INTERRACE IDENTITIES
After a thorough review of the LFW and CelebA datasets, random generation of survey questions, and rigorous hand-checking of questions to remove irregularities, we obtained a battery of survey questions for evaluating both humans and machines. We also selected survey questions that were balanced across gender, age, and skin type. Since LFW is highly skewed towards lighter identities, we included CelebA images and identities well. We selected identities from LFW with at least two images of an individual, and then we hand labeled each identity for the following: their (1) birth date, (2) country of origin, (3) gender presentation, and (4) Fitzpatrick skin type. Labels 1-3 were assigned by an author of this paper, then that label was checked by at least two others, and modifications were made achieve agreement among the labelers. Skin type labels (4) were assigned by 8 raters, and the mode was used as the final label.
We note that part of this work does reify categories of gender and skin type that have broader social and political implications. Further, we undertook a task of labeling and categorizing individuals who we do not know and have not received consent from for this task. Every identity for which we created these labels is indeed a celebrity in the public space with Wikipedia entries. Gender labels were rendered from the celebrity's 's public comments on their own gender identity.
The Fitzpatrick scale (Fitzpatrick 1988) was used to help balance the survey to include subjects with diverse skin types. This scale widely used to classify skin complexions into 6 categories. While the Fitzpatrick scale is not perfect, it is the best systematic option currently for ensuring broad representation.
We looked up each celebrity's 's birth date online, mostly citing Wikipedia, and if we could not find it there, we continued to search on other websites. However, if we could still not find an individual's date of birth, we did not list it. To find an individual's 's country of origin, we again cited Wikipedia. If the individual came from a country that no longer existed (i.e. East and West Germany), we listed the current country. To label person's gender presentation, we took note of the person's preferred
4

---

Under review as a conference paper at ICLR 2023
pronouns online and in interviews. In the event that their pronouns were not available online, labeled their gender presentation. A major limitation of the CelebA and LFW datasets is that there were no individuals in our process who identified outside the gender binary as gender queer.
At the end of our data collection, we collected metadata on 2545 identities which comprised a total of 7447 images. The identities themselves are rather imbalanced, though we selected a subgroup from these identities to create a balanced survey, discussed in Section[4] [ There are 1744 lighter-skinned individuals (as defined by Fitzpatrick skin types I-III) and 801 darker-skinned individuals (skin types VV-, There are 1660 males and 885 females. This sample is an improvement over previous datasets has been extensively evaluated remove any errors in labeling and has a robust labeling for a wider array of skin types, unlike previous datasets which chose to label individuals as "pale." These data have a range of potential future use cases, such as being used for more evaluative facial recognition studies and commercial system audits.
4 EXPERIMENTS
With the high-quality metadata provided in the InterRace identities, we conduct two experi- ments that aim answer our main research questions regarding the performance disparities of humans and machines. In this section, outline how we selected the survey questions, administered the survey to human participants, and evaluated machine models. We describe the results in Section 5
For both experiments, we create two types of questions: identification and verification. Both tasks contain "source" image. In the identifica- tion task, 9 other images are presented a grid, with one being of the same identity the source and the others being of the same gender and skin type. For the verification task, second image is selected with equal probability of being the same identity the source image, some other of the same gender and skin type as the source. Examples of these two types of questions can be seen in Figure 2
Figure 2: Example questions from the InterRace question bank. (Top) An example of an identifica- tion question. (Bottom) An example of verifica- tion question. Notice that the demographics of all identities appearing in question are matched to ensure the questions are not trivial.
We generated a static question bank with 78 identification questions and 78 verification ques- tions for each of the combinations of gender of skin type. Of those demographics with more than 78 identities, the source identity for the 78 questions were randomly chosen without replacement. This provided a total of 936 questions for each task. Finally, a pass was done over all questions to remove any for which context around the face (e.g., background or clothes) could be used to identify a person (e.g., a verification question where both images feature the same sports jersey). This final set has 901 identification and 905 verification questions.
4.1 HUMAN EXPERIMENT
We conducted an institutional review board-approved survey through the crowdsource platform Cint. The survey had two parts, one for each type of question: identification and verification.
Each respondent was asked 36 identification questions and 72 verification questions, for target survey length of around 10 minutes. The questions for each user were randomly sampled from the total question bank such that even distribution of questions were asked for each demographic group. As such, each respondent was asked 3 identification questions and 6 verification questions for each intersectional demographic identity. When the user first entered the survey they were prompted with a consent form. After completing both tasks, respondents filled out a demographic self identification form which asked the participants their age range, gender, and skin type. When asking respondents
5

---

Under review as a conference paper at ICLR 2023
to evaluate their own Fitzpatrick skin type scale, provided brief description of the scale and respondents were also shown three examples of each skin type from our dataset. The entire text of the survey, including the demographic questions, can be seen in Appendix D
Within each task, an attention check question was presented after the first five questions and before the last five. For the identification task, the attention check questions used an identical image for the target and in the gallery. For verification, one question consisted of pairing a light skinned female with a dark skin male (obvious negative example), and the other contained two identical images (obvious positive). The images used in these questions do not appear elsewhere in the survey. If a user failed to answer an attention check question correctly, they were screened out and any of their responses were ignored in our analysis. Additionally, any user who passed the attention checks but took fewer than minutes to complete the survey was dropped from the final analysis. The first verification and identification questions seen by each user were removed, to account for the possibility that the user may have taken some time to adjust to the format of the questions.
Our survey sampled English-speaking participants who were 18 years or older and were US residents. Our fi- nal sample includes 545 participants. There are 146 self-identifier as dark- skinned (Fitzpatrick IV-VI) females, 128 light-skinned (Fitzpatrickk I-III) fe- males, 140 dark-skinned males, and 131 light-skinned males. Most respon- dents (375) came from the 20-39 and 40-59 age demographics.
Table I:
Demographic breakdown of human survey respon- dents used in final analysis.
Fitzpatrick
Age 0-19
Age 20-39
Age 40-59
Age
Total
60-79
80+
1-11
0
23
37
33
95
Male
III.IV
35
18
24
V-VI
43
33
17
26
Female
III-VV
4
33
26
27
90
V-VI
43
27
20
4.2 MACHINE EXPERIMENTS
Academic Models To measure disparities, we trained 6 face recognition models and evaluated them on InterRace questions. We trained ResNet- 18, ResNet-50 (He et al. 2016) and MobileFaceNet (Chen al.] 2018) neural networks with CosFace (Wang al.] 2018) and ArcFace (Deng al.) 2019) heads, which are designed to improve angular separation of the learned features. The models are trained using the focal loss Lin et al. (017). For the training data, we used 140 000 images of 7866 CelebA identities disjoint from identities selected for the InterRace dataset. The training data has equal number of female and male identities and images. At inference time, the models solve identification questions by finding the closest gallery image in the angular feature space. For verification questions, we threshold the cosine similarity between features extracted from images in the pair.
We trained neural networks for 100 epochs with a batch size of 512 using SGD optimizer with the initial learning rate of 0.1, momentum of 0.9 and weight decay of 5e-4. The learning rate was decreased by factor of 10 at epochs 35, 65 and 95. Prior to training, we aligned and re-scaled training images 112 112, during the training we randomly apply horizontal flip to images. For data pre-processing and training routines we adapt the code from the publicly available GitHub repository face. evoLVe. PyTorch.
Commercial Models We evaluated three commercial APIs: AWS Rekognition, Microsoft Azure, and Megvii Face++. We were able to evaluate face verification and identification on AWS and Azure, and only face verification on Face++. The AWS CompareFace function, which compares source and target image, was used for both identification and verification; the target image for identification was one image comprised of the nine gallery images stitched together. Azure has native identification and verification built into their Cognitive Services Face API. Face++ has similar set up to AWS, however they only compare the largest detected faces in the source and target images; thus we were only able to perform face verification.
4.3 ANALYSIS STRATEGY
We use two-tailed f-test with matched pairs (with given pair corresponding to a single respondent's or computer model's 's scores on the two sections) to compare the accuracy rates between tasks. We also two-tailed, unpaired I-tests to compare the overall accuracy of humans verification questions
6

---

Under review as a conference paper at ICLR 2023
Table 2:
Overall gender and skin type disparities exhibited by the human survey respondents, academic models, and commercial APIs.
Identification
Verilication
Lighter
Lighter
Darker
Fermaee
Male
Feraaee
Male
Fernale
Male
Femule
Male
Homan
67.2%
78.3%
55.5%
73.1%
78.7%
83.1%
73.4%
80.1%
Academic Modell
95.0%
96.2%
96.7%
Commercial Models
96.7%
98.7%
96.7%
97.6%%
97.6%%
98.9%
97.8%
99.9%
with the overall accuracy of computer models on verification questions, and the overall accuracy of humans on identification questions with the overall accuracy of computer models identification questions. The latter t-tests and all t-tests referred to in the rest of this section are conducted on the question-level: for instance, when comparing the verification accuracy of humans and machines, use all verification responses from all human test-takers as one sample, and all verification responses from all machines as the other.
We then analyze the disparity along gender and skin-type categories within our computer algorithms and human survey results. Users and question subjects binned by skin type. Since the Fitzpatrick is heavily skewed towards Western conceptions of skin tone, we use two categorizations: a binary categorization of "lighter" (I-III) and "darker" (IV-VI); and categorization by (I-II, (III-IV) and (V-VI). We use two-tailed unpaired t-tests to detect the presence of accuracy disparities based on the gender or Fitzpatrick type of the identities that formed the questions. We perform tests of this kind on data from the six individual computer models, and also on the aggregate data sets of all human question responses and computer algorithm responses.
We use logistic regression in our analysis to allow us to control for confounding variables. Results are reported as odds ratios, which compare the ratio of odds for baseline event with the odds for different event. We consider main model for human subjects which predicts whether an individual question taken by a respondent was answered correctly, with independent variables as the question target gender and skin-type, and test-taker age, gender, and skin-type. The logistic regressions we run on the computer model responses are similar, but do not include test-taker demographics. We do report separate results for different architectures.
RESULTS
Humans achieved higher accuracy on verification (78.9%) than identification (68.3%, significant with two-tailed matched-pair l-test with p < 0.001). For computer models as whole, this gap persists but is substantially narrowed- performance on verification is 94.6%, with 93.7% on identification and is longer statistically significant (p- 0.129).
The performance difference between machines and humans is highly significant (p < 0.001) on both tasks using unpaired t-tests which explore group-level changes between the two tasks. Furthermore, even when controlling for demographic effects in logistic model, humans have a much lower odds compared to computers of getting question right (OR 0.14 for verification, p 0.001, OR = 0.21 for identification, < 0.001).
Humans and Computers Perform Better Male Subjects For identification questions, do not observe statistically significant performance for MobileFacceeeAArdFiia (p = 0.09714 for MobildFaceNNtCosFaacce and p 0.05629 for ResNet50Cossacce, but we do observe statistically significant disparities in favor of males for each of the other three ResNet models (all < 0.025). In logistic regression, we observe an odds ratio for computer models on male identification subjects of 1.89 (p < 0.001). Similarly, humans have significantly (p < 0.001) better accuracy on identification questions with male subjects: 75.7% on male subjects versus 61.4% on female subjects. The same holds true for humans on verification questions: they attain an accuracy of 81.6% on male subjects, versus 76. 76.1% on female subjects (p < 0.001). Interestingly, all demographics of survey respondents (when grouped by gender and skin-type) perform substantially better males than females for each task. The results of the human-only logistic models confirm human biases towards male subjects in both verification (OR 1.39, p < 0.001) and identification (OR 1.97, p 0.001).
7

---

Under review as a conference paper at ICLR 2023
Academic models are found, through logistic regression, to exhibit statistically significant difference in performance between verification questions with male or female subjects (OR = 1.37, p = 0.01).
Humans and Computers Perform Worse on Darker-Skinned Subjects Humans collectively are proportionally 5.2% worse on dark-skinned subjects than light-skinned subjects for verification
questions (80.9% versus 76.7%, p < 0.001) when we aggregate the Fitzpatrick scale binary. On identification questions, this proportional difference grew to 11.7% in favor of light-skinned subjects (72.7% versus 64.2%, p < 0.001). This holds even when controlling for the demographies of the respondent: the odds ratio of dark-skinned compared to light-skinned question subjects for verification is 0.78 (p- 0.001) while for identification it is 0.67 (p- < 0.001). When we aggregate the Fitzpatrick scale as three groups. I-II. III- and V-VI. verification logistic regression finds statistically significant biases in favor of Fitzpatrick types I-II, over both III-V and V-VI questions compared (OR = 0.93, 0.023 for III, OR = 0.85, p < 0.001 for V-VI). For the identification task, even when controlling for respondent demographic, question subjects with Fitzpatrick values I-II have higher correct responses than that of values III-VI and V-VI (OR 0.92, p 0.04 for III OR = 0.70, p < 0.001 for V-VI).
On machines, we observe similar higher performance on lighter-skinned subjects. When we aggregate the Fitzpatrick scale as just "light" and "dark", we observe a statistically significant performance disparity of 3.7% in favor of light-skinned question subjects on the verification task (p < 0.001), and for identification, we observe a 5.2% disparity in favor of light-skinned question subjects (p < 0.001). When we aggregate the Fitzpatrick scale into three categories, I-II, IIIIV, and V-VI, we see a disparity for both tasks between the lightest (I-II) and darkest groups (V-VI) (p < 0.0041 and p 0.04 for both verification and identification). Academic model performance is revealed to be significantly different, even when controlling for gender, between the types I-I1 and V-VI (OR 0.36. p < 0.001 for identification; OR = 0.47, p < 0.001 for verification). However, I-Il and III-V do not show statistically significant differences for academically-traine trained models (OR = 0.94. p = 0.714 for identification; OR 0.92, p = 0.642 for verification).
Human Test- Takees Perform Better on Subjects of Similar Demographic We hypothesized that humans would more accurate on questions that contained subjects that looked like them. We find evidence to support this hypothesis in our data. On the verification task, humans perform significantly better on questions where the subjects match their gender identity (1.1%, p = 0.02), skin type (1.7%, p 0.002), and gender identity and skin type (1.3%, p 0.009). On the identification task, humans perform significantly better on questions where subjects match their skin type (2.3%, p = 0.011) and both their gender identity and skin type (3.3%, p 0.001).
Humans and Machines Exhibit Comparable Gender Disparity, but Machines have Greater Skin Type Disparity than Humans To test for whether the levels of disparity described above are comparable between humans and machines, we look at the confidence intervals for the odds ratios of comparable models. For both tasks, recall that observed a disparity on gender and skin type for humans and machines. For verification, we observe that the magnitude of the gender disparities are similar (OR 95% confidence intervals for humans are [1.33, 1.46) and for academic models are [1.07, 1.73).. For identification, we observe that the magnitude of the gender disparities also similar (OR 95% confidence intervals for humans are [1.84, 2.10] and for academic models are [1.50,2 2.39)). This allows us to conclude that when there is a gender disparity displayed by both humans and machines, the magnitudes and directions of that disparity are statistically similar.
On the other hand, the skin type disparity is more pronounced in academic models than in humans. Using the same analysis technique as above, we see that the OR 95% confidence intervals do not match for the darkest skin types in all cases (identification and verification as well as 2 and 3 skin type categories). Furthermore, the academically trained machines show larger disparity (smaller odds ratio) than humans do. In identification, we have confidence intervals for binary skin types as [0.62, 0.71] for humans and [0.32, 0.52] for academic models. For tertiary skin types, we have confidence intervals of [0.64, 0.75] for humans and [0.27, 0.48| for the darkest subjects. In verification, we have confidence intervals for binary skin types as [0.74, 0.82] for humans and [0.38, 0.63] for academic models. For tertiary skin types, we have confidence intervals of (0.80, 0.90) for humans and [0.35, 0.62] for academic models. Regression tables can be found in Appendix C
Commercial Facial Recognition Models Are Very Accurate The commercial models have very high accuracy. particularly AWS and Face++ which each scored above 97.3% accuracy on both verification and identification. As result, these systems do not have enough incorrect responses to
8

---

Under review as a conference paper at ICLR 2023
have any statistically significant conclusions. On the other hand, Azure achieves verification accuracy of 93.3% and identification accuracy of 82.9%. In this case, we see a bias towards question gender in favor of males (OR 1.76; p 0.041) which is comparable to the bias observed with humans and academic models.
DISCUSSION
The study described in this work is the first to compare disparities and bias between humans and machines. We see that the gender and skin type biases of humans also present in academic models. Interestingly the level of the disparities present in humans are comparable to that of the machines. These human disparities are present even when controlling for the demographies of the participant. We also find that humans perform better when the demographics of the question match their own. It might be easy to look back in hindsight on our results and say they are obvious. It may not be surprising that humans and machines have bias based on gender and skin type. However, we should not forget that this is the first study that directly compares the two with precisely the same questions which are pre-screened for difficulty. Our work is also the first to show a human preferential performance on subjects who look like them.
One key limitation of our human survey is that we analyze crowdsourced sample. While it is demographically diverse, does not represent sample of expert facial recognizers. Our results should not be extrapolated too far outside the sample of non-expert crowd workers located in the US. Additionally, the results we have for the computer models are limited to those which we included and do not represent how all models work or behave.
Our findings contribute meaningfully to the ongoing work of understanding the benefits and harms presented by facial recognition technology. Specifically, we that automated methods outperform non-expert humans across the board. When bias is detected in a machine, that bias is comparable to those exhibited by non-expert humans. In the future, further work should examine more targeted populations, such as the direct users of facial recognition technology, to understand how their native bias compares to the biases of machines or
teams.
6.1 ACTIONABLE INSIGHTS
The field has focused on high-level, aggregate statistics at the dataset level for evaluating model performance. We see that in most areas, including facial recognition, where leader-boards drive innovation and improvements. However, our work puts that into perspective by examining the performance of systems when we look at accurately labeled subgroups of those overall datasets. We also see that commercial models have significantly less bias than academic models or people, and academic models are replicating and have only slightly less bias than people. The first result has never been documented before and leads one to conclude that the commercial companies have expended extra effort and resources improve the accuracy and decrease the bias in facial identification and verification. This is likely related to incentives to minimize harm and maintain public images; but nevertheless, the different incentive has lead improvements which academic models don't see because of the improper supremacy of the dataset-level metrics. This paper has actionable contributions facial recognition for ML Practitioners and Dataset Curators:
(1) Balanced training does not lead to unbiased performance, (2) the CosFace head and ResNet backbones yield lower bias than others, and (3) the ML solutions are always at least as biased as humans. This last point is very important. When we replace a human task with computer task, obviously speed to do the task important, but we also want to be better humans in their biases. Our results, the first direct comparison of humans and machines in this domain, show that this has not been met yet. The models are always as biased, and in many cases, more biased than humans! Thus, provide stable benchmark ML practitioners can
ML Practitioners
improve upon.
Dataset Curators The labels and the means of collecting those labels is very important, and should almost always involve human review. We found that existing datasets' labels (often which used computer- computer-generated generated labeling methods) were widely insufficient and riddled with errors which have downstream implications analysis.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Ehsan Adeli, Qingyu Zhao, Adolf Piefferbaum, Edith V Sullivan, Li Fei-Fei, Juan Carlos Niebles, and Kilian M Pohl. Representation learning with statistical independence to mitigate bias. In Proceedings of the IEEE/CVF Winter Conference Applications of Computer Vision, pp. 25 13- 2523, 2021.
Alekh Agarwal, Alina Beygelzimer, Miroslav Dudik, John Langford, and Hanna Wallach. A reductions approach to fair classification. In Proceedings of the 35th International Conference on Machine Learning, volume 80, pp. 60 69, 2018. URL httpt:tttppccceee..ii.eee mlr p
v80/agarwal18a.11 hmmm
Solon Barocas, Moritz Hardt, and Arvind Narayanan. Fairness and Machine Learning. fairml- book.org, 2019. http://www.faimmlboooooo
Alex Beutel, Jilin Chen, Zhe Zhao, and Ed H Chi. Data decisions and theoretical implications when adversarially learning fair representations. arXiv preprint arXiv:1707.00077 2017.
Joy Buolamwini and Timnit Gebru. Gender shades: Intersectional accuracy disparities in commercial gender classification. In Proceedings of the 1st Conference on Fairness, Accountability and Transparency, volume 81, pp. 77-91, 2018. URL hhtt::://proeeee rggs ml press/ v81 buo amwi ni 8a. html
Sheng Chen, Yang Liu, Xiang Gao, and Zhen Han. Mobilefacenets: Efficient for accurate real-time face verification on mobile devices. In Chinese Conference on Biometric Recognition, pp. 428 438. Springer, 2018.
Alexandra Chouldechova and Aaron Roth. The frontiers of fairness in machine learning. arXiv preprint arXiv:1810.08810. 2018
Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou. Areface: Additive angular margin loss for deep face recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4690-4699, 2019.
William Derringer. A surveillance net blankets china's citics, giving Times, Dec. 17 2019. URL https:////// /ttkpp::/www.... wttp://wwwwpp.pp..........tt imes wwwp://ww.iiis.ccoooooo/////////
giving police vast powers. The New York
china-surve lance. html
Emily Diana, Wesley Gill, Michael Kearns, Krishnaram Kenthapadi, and Aaron Roth. Convergent algorithms for (relaxed) minimax fairness. arXiv preprint arXiv:20/ 1.03/08, 2020.
Michele Donini, Luca Oneto, Shai Ben-David, John Shawe-Taylor, and Massimiliano Pontil. Em- pirical risk minimization under fairness constraints. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS' 18, pp. 2796-2806 2018.
Samuel Dooley, Tom Goldstein, and John Dickerson. Robustness disparities in commercial face detection. arXiv preprint arXiv:2 108.::200../2500 /2508, 2021.
Cynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Richard Zemel. Fairness through awareness. In Proceedings of the 3rd Innovations in Theoretical Computer Science Conference, ITCS 122 pp. 214-226, New York, NY, USA, 2012. Association for Computing Machinery. ISBN 1781450311151, doi: 10. 011144/2200033.2200222 2090236.2090255
URL httpttss://ddi wtt.d/ooooo//1..111 1145/
Harrison Edwards and Amos J. Storkey. Censoring representations with an adversary. In 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico May 2-4, 2016, Conference Track Proceedings, 2016. URL http: /ttp::////xxv..../ www.aiiivrigii..eee.o abs 1511 05897
Michael Feldman, Sorelle A Friedler, John Moeller, Carlos Scheidegger, and Suresh Venkatasubra- manian. Certifying and removing disparate impact. In Knowledge Discovery and Data Mining, pp. 259-268, 2015.
10

---

Under review as a conference paper at ICLR 2023
Thomas B Fitzpatrick. The validity and practicality of sun-reactive skin types through vi. Archives of dermatology, 124(6):869-55) 871, 1988.
Clare Garvie. The perpetual line-up: Unregulated police face recognition in America. Georgetown Law, Center on Privacy & Technology. 2016.
Naman Goel, Mohammad Yaghini, and Boi Faltings. Non-discriminatory machine learning through convex fairness criteria. Proceedings the AAAI Conference on Artificial Intelligence, 32(1),
2018. URL Stthttts::
//ojs. aaal index. hwwp..pppppp.///////aaaaaa//aaaa htt.AAAArrriiceeeeveieaaee /1tel///eeeee//1e222222
Google. How google
pattern recognition to make sense of images. http://www....oo.oo/o/o//oooooea
2021.
Accessed:
2021-06-07.
Patrick Grother, Mei Ngan, and Kayee Hanaoka. Face Recognition Vendor Test (FVRT): Part 3, Demographic Effects. National Institute of Standards and Technology, 2019.
Moritz Hardt, Eric Price, Eric Price, and Nati Srebro. Equality of opportunity in super- vised learning. In Advances in Neural Information Processing Systems, volume 29, pp. 3315 3323, 2016. URL http:::/ 642682367c33 (eee777333335ffooo htt.a//200....eeaeeooeeeee/////. Pappea pdf
Woodrow Hartzog. The secretive company that might end privacy we know it. The New York Times, Jan. 18 2020. URL
.tml
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 770-778, 2016.
Ying Hu, Kelsey Jackson, Amy Yates, David White, P Jonathon Phillips, and Alice J O' Person recognition: Qualitative differences how forensic face examiners and untrained people rely on the face versus the body for identification. Visual Cognition, 25(4-6):492-506 2017.
Gary B Huang, Marwan Mattar, Tamara Berg, and Eric Searned-Millee Labeled faces in the wild: A database forstudying face recognition in unconstrained environments. In Workshop on faces in' Real-Life' Images; detection, alignment, and recognition, 2008
Brendan F Klare, Mark Burge, Joshua C Klontz, Richard W Vorder Bruegge, and Anil K Jain. Face recognition performance: Role of demographic information. IEEE Transactions on Information Forensics and Security, 766)1789 (16::1799-1801, 2012.
Genia Kostka, Léa Steinacker, and Miriam Meckel. Between security and convenience: Facial recognition technology in the eyes of citizens china, germany, the united kingdom, and the united states. Public Understanding of Science, pp. 0963662521 1001555, 2021.
Preethi Lahoti, Alex Beutel, Jilin Chen, Kang Lee, Flavien Prost, Nithum Thain, Xuezhi Wang, and Ed H. Chi. Faimess without demographics through adversarially reweighted learning, arXiv preprint arXiv:2006. 200:22006.13114, 2020.
Sarah Lewis. The racial bias built into photography. The New York Times, 25, 2019.
Tsung- Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, pp. 2980-2988, 2017.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes the wild. In Proceedings of International Conference Computer Vision (ICCV), December 2015.
Steve Lohr. Facial recognition accurate, if you're 're white guy. New York Times, 2018.
Chaochao Lu and Xiaoou Tang. Surpassing human-level face verification performance on Ifw with
gaussianface. In Twetty-ninth AAAI conference artificial intelligence, 2015.
11

---

Under review as a conference paper at ICLR 2023
David Madras, Elliot Creager, Toniann Pitassi, and Richard S. Zemel. Learning adversarially fair
and transferable representations. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmüssan, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pp. 3381-3390. PMLR, 2018. URL h
//proceedings.. ml press /vtp///aaaaaaaaaaaaaaamaa ht.ml
James Marson and Brett Forrest. Armed low-cost drones, made by turkey, reshape battlefields wtwpe///seeemme--wwwoootttseoeee low-cost-drones cost-drones-made-be made-by made-by-turkey
and geopolitics. https//wwww..jj.com//aiicccl... battlefildd--ad--eoopolll http:/ddddddddoooolllss---112211 Jun 2021. The Wall Street Journal.
Natalia Martinez, Martin Bertran, and Guillermo Sapiro. Minimax pareto fairness: A multi ob- jective perspective. In Proceedings of the 37th International Conference on Machine Learn- ing, volume 119, pp. 6755-6764 2020. URL http::/prrceeeiingggg.e wwwpiiinnnmm........mmmmme www.ppprsp./.11/////./.
mart ine h
Alice O'Toole, P Jonathon Phillips, Fang Jiang, Janet Ayyad, Nils Penard, and Herve Abdi. Face recognition algorithms surpass humans matching faces over changes in illumination. IEEE transactions on pattern analysis and machine intelligence, 29(9):6642 2007.
Alice J O'Toole, Jonathon Phillips, Xiaobo An, and Joseph Dunlop. Demographic effects on estimates of automatic face recognition performance. Image and Vision Computing, 30(3):669-170 2012.
Manisha Padala and Sujit Gujar. Fnne: Achieving fairness through neural networks. Proceedings of the Twetty-Nitth International Joint Conference Artificial Intelligence, IJCAI-20, pp. 2277- 2283, International Joint Conferences on Artificial Intelligence Organization, 2020. doi: 10. 2t163///caai2220/3353 htt.000011......../////////////. http://01..2299311..... 20200//313222222223333
Pew Research Center: In response climate change, citizens in advanced economies are willing to alter how they live and work. Technical report, Pew Research Center, Washington, D.C., Septem- ber 2021. URL htwps:://wwwww.//// www www.:eeewreeearcc.oooooooooo.oce org/g obal Raabal/sa 1/wp //pp////ooo wwwttottoontttaaaeeeeeeeee
sites/2/2021/09/PP 200___2221100000 09. 11t::114_ciaiaaa Climate FINAL. pdf
Jonathon Phillips and Alice J O'toole. Comparison of human and computer performance across face recognition experiments. Image and Vision Computing, 32(1):74-85, 2014.
P Jonathon Phillips, Amy N Yates, Ying Hu, Carina A Hahn, Eilidh Noyes, Kelsey Jackson, Jacque- line Cavazos, Géraldine Jeckeln, Rajeev Ranjan, Swami Sankaranarayanan, et al. Face recogni- tion accuracy of forensic examiners, superrecognizers, and face recognition algorithms. Proceed- ings of the National Academy of Sciences, 115(44))6111-6176 2018
Novi Quadrianto, Viktoriaa Sharmanska, and Oliver Thomas. Discovering fair representations in the data domain. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pp. 8227-8234 Computer Vision Foundation IEEE, 2019. doi: 10. 1O9/C//R..009.00082.0. CVPR 2019/htmL/Quadr intioaariianntooommm Dwa.ttooDDscoverringoommm Fair _epresentttiinnss.t... org.rppeinggapaaa.eeeerreetttttn
URL http: http:::/pppaccccssssstt http::/pppeecccsssseeee.ooooo www.:teecccccccottteeeeeeeeeeeee
Che_Data_____ www.DDneeinn_cr221111_______
html
David J Robertson, Filidh Noyes, Andrew J Dowsett, Rob Jenkins, and A Mike Burton. Face
recognition by metropolitan police super-recognisers. PloS one, ((2)xc0150036, 2016.
Hee Jung Ryu, Hartwig Adam, and Margaret Mitchell. Inclusivefacenet: Improving face attribute detection with race and gender diversity. arXiv preprint arXiv: 2018.
Yash Savani, Colin White, and Naveen Sundar Govindarajulu. Intra-processing methods for debiasing neural networks. In Proceedings of the Annual Conference on Neural Information Processing Systems (NeurIPS), 2020.
Xiaoou Tang and Xiaogang Wang. Face sketch recognition. IEEE Transactions on Circuits and Systems for video Technology, 14(1):50-57, 2004.
Hao Wang, Yitong Wang, Zheng Zhou, Xing Ji, Dihong Gong, Jingchao Zhou, Zhifeng Li, and Wei Liu. Cosface: Large margin cosine loss for deep face recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5265-5274, 2018
12

---

Under review as a conference paper at ICLR 2023
Mei Wang and Weihong Deng. Deep face recognition: A survey, arXiv preprint arXiv: 2018.
Mei Wang and Weihong Deng. Mitigating bias in face recognition using skewness- aware reinforce- ment learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 9322-9331, 2020.
Tianlu Wang, Jieyu Zhao, Mark Yatskar, Kai-Wei Chang, and Vicente Ordonez. Balanced datasets not enough: Estimating and mitigating gender bias in deep image representations. In Proceedings of the IEEE International Conference on Computer Vision, pp. 5310-5319, 2019.
Zeyu Wang, Klint Qinami, Ioannis Christos Karakozis, Kyle Genova, Prem Nair, Kenji Hata, and Olga Russakovsky. Towards fairness visual recognition: Effective strategies for bias mitigation, 2020.
Karen Weise and Natasha Singer. Amazon pauses police use of its facial recognition soft- ware. The New York Times, Jul 2020. URL wtwp.//...........a////////.//..
wwtp:///wwwww..iei......... Eechnology/ /ttp:///aacaaaaaaeeeeoooooooooaa
cwm com/2020/06/10/ html
David White, James Dunn, Alexandra C Schmid, and Richard Kemp. Error rates in users of automatic face recognition software. PloS one, 10(10)ee0139827, 2015.
Muhammad Bilal Zafar, Isabel Valera, Manuel Gomez Rodriguez, and Krishna Gummadi, Fairness beyond disparate treatment & disparate impact. Proceedings of the 26th International Conference on World Wide Web, Apr 2017a. doi: 0..1145//038912.30556666 URL URL BRLhttp::0 http: doi org/
10 10..114/3303 30055/3383330 912. 305 3052660
Muhammad Bilal Zafar, Isabel Valera, Manuel Comez-Rodriguees and Krishna P. Gummadi. Fairness constraints: Mechanisms for fair classification. Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, AISTATS 2017, 20-22 April 2017, Fort Lauderdale, FL, USA, volume 54 of Proceedings of Machine Learning Research, pp. 962-970. PMLR, 2017b. URL
http://proceeddnng mwr. /ttpa///////////aa1/..2..2
Muhammad Bilal Zafar, Isabel Valera, Manuel Gomez- and Krishna Gummadi. Fairness constraints: A flexible approach for fair classification. Journal of Machine Learning Research, 20 (75):1-42, 2019.
http://////..trrrrre////e/////// 2201111822222222001 262.httl
Rich Zemel, Yu Wu, Kevin Swersky, Toni Pitassi, and Cynthia Dwork. Learning fair representations. In International conference on machine learning, pp. 325-333. PMLR, 2013.
13


Reviews:
[{"review_id": "vC4-DiQ3dlu", "reviewer": "Reviewer_VUfd", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes an benchmark (dataset and procedure) to evaluate the bias of human performance in the face recognition problem. To offer more information, the paper reports a set of similar tests on machine performance. ", "summary_of_the_review": "The paper aims to a direction where contributions are welcomed, and there is an auditorium at ICLR. The main  problem is that the accuracy of the data was not well established and in this case is critical.", "strength_and_weaknesses": "Strengths:\n - I totally agree with the paper that there is need in the research to evaluate the bias and compare the human performance with machine performance for the face recognition problem. Contribution here are welcomed\n- release of curated database with reference methods evaluated should help the field.\n- the paper is well presented and clear\n\nWeaknesses:\n1. The main issue of such paper is the following: how it was established the ground truth? By human annotation?! One key aspect of the paper is that it reports a 67%-83% accuracy of human observers accuracy. On other hand the limited number of persons in the authors team provides the ground truth. It is quite hard to take this as an established ground truth especially since there no is strong indication that experts (i.e. authors here) behave much better than normal persons. It is further said several time in the paper, that \"one\" of the authors revised the data to make sure that no errors are further passed. Yet a person has up to 83% accuracy and quite often 70%. Based on the data provided, curation by machine learning is preferred. \nAt the end of the paper it is noted: \"The labels and the means of collecting those labels is very important, and should almost always involve human review. We found that existing datasets’ labels (often which used computer-generated labeling methods) were widely insufficient and riddled with errors which have downstream implications on analysis.\" I believe that this is true but is also applicable to the ground truth of this data, since there is no information that consensus of large number of experts was used.\n \nIn other words, in my view, the main limitation of the paper is that ground truth is not established accurately. A large number of observers should have been used and average where there is consensus should be placed as labels. \n\n2. The paper does not report the number of incorrect labels in LFW. From the data reported in the paper and in the supplementary material, it seems that the amount of errors was small and most of curation was about removing non-public persons. Unfortunately, in this way the contribution w.r.t the dataset is harder to establish.\n  ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "nh-hKskXvi", "reviewer": "Reviewer_NNHX", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a study that compares disparities and bias between humans and machines in performing face verification and recognition in challenging conditions. The authors use two datasets named LFW and CelebA for these studies. Human experiment consists of randomly sampled subjects answering verification/identification questions and machine experiments were run using existing academic and commercial models.    ", "summary_of_the_review": "Overall I think the study presented in paper lacks novelty and the presented results are expected.", "strength_and_weaknesses": "Strengths:\n- To the best of my knowledge, this is the first time such a study is done to compare human and machine bias on face verification/identification. \n- It looks like the authors have done a good job of curating the datasets by removing poor quality images and correcting the inconsistencies in its metadata. \n\nWeaknesses:\n- Other than curating the datasets, I do not see much scientific contribution of this work. All the algorithms used for conducting experiments are existing algorithms.\n- The results observed in the paper are expected results. I am not sure how these findings help in further improving the research in the face identification/verification area.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Ye2v9Fec8bE", "reviewer": "Reviewer_TNQq", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Bias in face recognition is a significant area of research that has/will have a meaningful impact on analyzing and managing disparities in downstream applications. The authors of this paper propose enhancements to the existing face datsets such as LFW and CelebA by improving the available metadata and also devise experiments to measure the bias exhibited by humans and models during identification and verification tasks.\n", "summary_of_the_review": "Overall, a well-written paper. The findings are interesting but may not be extrapolated as suggested by the authors.", "strength_and_weaknesses": "Strengths: \n1. Well-written and structured\n2. Experimentally strong\n3. Results \n\nWeaknesses: \n1. Missing technical novelty. \n2. No mention of how the existing model architectures were revised for the task on hand. \n3. It is not clear if the apparent bias in data (eg certain age groups were more populated than others) while training/validating and testing.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "6dehOB9O0IL", "reviewer": "Reviewer_WUnQ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors aim to answer two questions in this paper, \"How and to what extent do humans exhibit\nbias in their accuracy in facial recognition tasks?\" and \"How does this compare to machine learningbased models?\"  To address these questions the authors present improvements to the LFW and CelebA dataets as well as present an analysis showing that both humans and models result in lower accuracy on dark skinned and female subjects and humans obtain higher accuracy on the tasks when their demographics match the subjects in the data.  They also find that while academic models are comparable in gender bias compared to humans, they are significantly more biased against darker skin tones than humans.", "summary_of_the_review": "The paper is clear and the problem is a good one to address but it lacks some key details and presentation limitations that reduce its score.", "strength_and_weaknesses": "\nThe paper is clear and the work is well motivated.  It is certainly important to gauge model performance compared to humans.\n\nIt is great that the authors identify the weakness of these face datasets and present cleaned annotations and scrubbing of images that may not be desired in the set.\n\nFor some use cases, low quality images may be desired.  It may be preferable to include them but add a quality label to the annotations.  What are the specific thresholds used to decide to omit these images?\n\n\"old-timey\" should be better defined so as not to be subjectively determined.\n\nIf black and white and old-timey images were removed due to the being too easy to recognize as paired then this should be validated but no quantitative check is presented.\n\n\"We found that many questions could be answered without considering face features at all, and these\nwere removed. For example if the subject is wearing identical attire and/or standing in front of an\nidentical background in two images.\"  Again, this needs to be shown to be true quantitatively if the authors are referring to the models ability to match faces.  Many models will not use background features making the issue not as important as presented.\n\n\"While the Fitzpatrick scale is not perfect, it is the best systematic option currently for ensuring a\nbroad representation.\"  Not sure this is still true with the existence of the Monk scale which was specifically created for this task while Fitzpatrick was not, although Fitzpatrick is far more commonly used.\n\nWhat is the usefulness of the birth date if the date of image capture is not provided?\n\nIt would be helpful for future projects if more details on the process and workflow to manually check/verify the annotations in the data were provided.  Lessons learned and the final selected approach would be helpful for other researchers that are interested in a similar process.\n\nIt would be much better for the reader if results comparing the models and humans were presented in graphs, figures and tables to provide more detail than what is in Table 2.  This would also allow for high-level take-aways on the details to be better digestible.  A lot of specific statistical results are presented in the text but this should be reflected in figures as well.  Some details can be moved to an appendix if needed.\n\nIt would also be helpful to understand the per-human performance.  Some participants may be better at the tasks than others.  The human results would likely form a distribution of performance and this should be captured and compared to the per-model, and academic group as well as commercial group, model distribution.  The uncertainty from these distributions needs to be considered as we are comparing over multiple variables.\n\nOdds ratio is not commonly used by the face rec community so a better description may be helpful\n\nadditional metrics such as FMR and FNMR, or FAR and FRR, should be presented.  It would help to understand where the errors are occurring and if the disparities are a result of one type of error or the other.  This is also useful for comparing human to model performance.\n\nTypo at end of section 5: \" In this case, we see a bias towards question gender in favor of males\"", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject