A Deep Dive into Dataset Imbalance and Bias in Face Identification
Authors: Valeriia Cherepanova
Steven Reich
Samuel Dooley
Hossein Souri
John P Dickerson
Micah Goldblum
Tom Goldstein
Venue: Submitted to ICLR 2023 (2023)
Keywords: face recognition
dataset imbalance
bias
fairness

Abstract: As the deployment of automated face recognition (FR) systems proliferates, bias in these systems is not just an academic question, but a matter of public concern. Media portrayals often center imbalance as the main source of bias, i.e., that FR models perform worse on images of non-white people or women because these demographic groups are underrepresented in training data. Recent academic research paints a more nuanced picture of this relationship. However, previous studies of data imbalance in FR have focused exclusively on the face verification setting, while the face identification setting has been largely ignored, despite being deployed in sensitive applications such as law enforcement. This is an unfortunate omission, as 'imbalance' is a more complex matter in identification; imbalance may arise in not only the training data, but also the testing data, and furthermore may affect the proportion of identities belonging to each demographic group or the number of images belonging to each identity. In this work, we address this gap in the research by thoroughly exploring the effects of each kind of imbalance possible in face identification, and discuss other factors which may impact bias in this setting.

Under review as a conference paper at ICLR 2023
A DEEP DIVE INTO DATASET IMBALANCE AND BIAS IN FACE IDENTIFICATION
Anonymous authors Paper under double-blind review
ABSTRACT
As the deployment of automated face recognition (FR) systems proliferates, bias in these systems is not just an academic question, but a matter of public concern. Media portrayals often center imbalance as the main source of bias, i.e., that FR models perform worse on images of non-white people or women because these demographic groups are underrepresented in training data. Recent academic research paints more nuanced picture of this relationship. However, previous studies of data imbalance in FR have focused exclusively on the face verification setting, while the face identification setting has been largely ignored, despite being deployed in sensitive applications such as law enforcement. This an unfortunate omission, as 'imbalance" is more complex matter in identification; imbalance may arise in not only the training data, but also the testing data, and furthermore may affect the proportion of identities belonging each demographic group the number of images belonging to each identity. In this work, we address this gap in the research by thoroughly exploring the effects of each kind of imbalance possible in face identification, and discuss other factors which may impact bias in this setting.
INTRODUCTION
Automated face recognition is becoming increasingly prevalent in modern life, with applications ranging from improving user experience (such as automatic face-tagee face-tagging of photos) to security (e.g.. phone unlocking or crime suspect identification). While these advances are impressive achievements, decades of research have demonstrated disparate performance in FR systems depending on a subject's race (Phillips et al.] 2011 Cavazos al. 2020), gender presentation (Alvi al., 2018 Albiero et al., 2020). age (Klare et al. 2012). and other factors. This is especially concerning for FR systems deployed in sensitive applications like law enforcement; incorrectly tagging personal photo may be a mild inconvenience, but incorrectly identifying the subject of a surveillance image could have life-changing consequences. Accordingly, media and public scrutiny of bias in these systems has increased, in some cases resulting in policy changes.
One major source of model bias is dataset imbalance; disparities in rates of representation of different groups in the dataset. Modern FR systems employ neural networks trained on large datasets, so naturally much contemporary work focuses on what aspects of the training data may contribute to unequal performance across demographic groups. Some potential sources that have been studied include imbalance of the proportion of data belonging to each group (Wang & Deng 2020 Gwilliam et al., (022). low-quality poorly annotated images (Dooley et al.) 2021). )) and confounding variables entangled with group membership (Klare et al.) 2012 Kortylewski al. 2018 Albiero et al., 2020).
Dataset imbalance is much more complex and nuanced issue than it may seem at first blush. While a naive conception of 'dataset imbalance' is simply as disparity in the number of images per group, this disparity can manifest itself as either gap in the number of identities per group, or in the number of images per identity. Furthermore, dataset imbalance can be present in different ways in both the training and testing data, and these two source of imbalance can have radically different (and often opposite) effects on downstream model bias.
Past work has only considered the verification setting of FR, where testing consists of determining whether a pair of images belongs to the same identity. As such, 'imbalance' between demographic groups is not meaningful concept in the test data. Furthermore, the distinction between imbalance

---

Under review as a conference paper at ICLR 2023
Imbalance in the number of identities Example: more fernale identities than males
Imbalance in the number of images per identity Example: male identities have more images
1d
M
Id
Im Imbalance in composition the gallery
Probe Image
Example Gallery I: primarily male images photos match probe
Example Gallery 2:
primarily female images photos match probe
Recogaition
Model
Figure 1: Examples of imbalance in face identification. Top left: data containing more female identities than male identities. Top right: data containing the same number of male and female identities, but more images per male identity. Bottom: two possible test (gallery) sets showing how the effects of different kinds of imbalance may interact.
of identities belonging to certain demographic group versus that of images per identity in each demographic group has been carefully studied in either the testing or the training data. All of these facets of imbalance are present in the face identification setting, where testing involves matching probe image to gallery of many identities, each which contains multiple images. We illustrate this in Figure
In this work, we unravel the complex effects that dataset imbalance can have on model bias for face identification systems. We separately consider imbalance (both in terms of identities or images per identity) the train set and the test set. We also consider the realistic social use case in which a large dataset is collected from imbalanced population and then split at random, resulting in similar dataset imbalance in both the train and test set. We specifically focus on imbalance with respect to gender presentation, (when restricting to only male- and female-identified individuals) this allows the proportion data in each group to be tuned as single parameter, as well as the availability of an ethically obtained identification dataset with gender presentation metadata of sufficient size allow for subsampling without significantly degrading overall performance.
Our findings show that each type of imbalance has distinct effect model's performance each gender presentation. Furthermore, in the realistic scenario where the train and test set are similarly imbalanced, the train and test imbalance have the potential to interact in way that leads to systematic underestimation of the true bias of model during an audit. Thus any audit of model bias in face identification must carefully control for these effects.
The remainder of this paper is structured as follows: Section 2 discusses related work, and Section introduces the problem and experimental setup. Sections and 5 give experimental results related to imbalance in the training set and test set, respectively, and Section gives results for experiments where the imbalance the training set and test set are identical. In Section 7.1 we evaluate randomly initialized feature extractors test sets with various levels of imbalance to further isolate the effects
of this imbalance from the effects of training. In Section 7.2, we investigate the correlation between the performance of models trained with various levels of imbalance and human performance.
2 RELATED WORK
2.1 IMBALANCE IN VERIFICATION
Even before the advent neural network-basee face recognition systems, researchers have studied how the composition of training data affects verification performance. Phillips et al.) (2011) compared algorithms from the Face Recognition Vendor Test (Phillips et al. 2009) and found that those developed in East Asia performed better East Asian Faces, and those developed in Western
2

---

Under review as a conference paper at ICLR 2023
countries performed better Caucasian faces. Klare et al. (2012) expanded on these results by comparing performance across race, gender presentation, and age cohorts, observing that training exclusively on images of one demographic group improved performance that group and decreased performance on the others. They further conclude that training on data that is "well distributed across all demographies" helps prevent extreme bias.
Multiple verification datasets have been proposed in the interest of eliminating imbalance as a source of bias in face verification. The RUPT-BalancedFace dataset (Wang & Deng 2020) contains an approximately equal number of identities and images of four racial group Balanced Faces in the Wild (Robinson et al., 2020) goes a step further, balancing identities and images across eight categories of race-gentt geee-gender presentation combinations. Also of note is the BUPT-CBFar dataset (Zhang Deng. 2020), which class-balanced (each identity possesses the same number of images), rather than demographically balanced.
Some recent work in verification has questioned whether perfectly balanced training data is in fact an optimal setting for reducing bias. |Aliero et al.] (2020) studied sources of bias along gender presentation; among their findings, they observe that balancing the amount of male and female training images and identities in the training data reduces, but does not eliminate, the performance gap between gender presentations. Similarly, [wwiliiam et al. (202T) trained models on data with different racial makeups, finding that models which were trained with more images of African subjects had lower variance in performance on each race than those which were trained on balanced data.
2.2 BIAS IN IDENTIFICATION
Although the effect of imbalance on bias has only been explicitly studied in face verification, there is some research on identification which is relevant. The National Institutes of Standards and Technology performed large-scal, scale testing of commercial identification algorithms, finding that many (though not all) exhibit gender presentation racial bias (Grother et al., 2019). The evaluators speculate that the training data or procedures contribute to this bias, but could not study this hypothesis due to the proprietary nature of the models. Dooley et al. 22021) evaluated commercial and academic models on a variant of identification in which each probe image is compared to 9 gallery images of distinct identities, but belonging to the same skin type and gender presentation. They find that academic models (and some, but not all, commercial models) exhibit skin type and gender presentation bias despite a testing regime which makes imbalance effectively irrelevant.
2.3 IMBALANCE IN DEEP LEARNING
Outside the realm of facial recognition, there is much study about the impacts of class imbalance in
deep learning. In standard machine learning techniques, i.e., non-deep learning, there are many well-
studied and proven techniques for handling class imbalances like data-level techniques (Van Hulse
ct al. 2007| Chawla et al. 2002 2004), algorithm-leve. methods (Elkan Ling & Sheng 2008]
Krawezyk 2016), and hybrid approaches (Chawla et al. 2003 Sun al.) 2007 Liu et | 2008). In
deep learning, some take the approach of random over or under sampling (Hensman & Masko 2015]
Lee et al. 20166] Pouyanfar et 2018). Other methods adjust the learning procedure by changing the
loss function (Wang et al.] (016)) or learning cost-sensitive functions for imbalanced data (Khan et al.
2017). We refer the reader Buda al. (2018); Johnson & Khoshgoftaar (2019), for a thorough review of deep learning- based imbalance literature. Much of the slass-imbllance work has been on computer vision tasks. though generally has not examined specific analyses like we present in this
work like network initialization, face identification, or intersectional demographic imbalances.
2.4 OTHER SOURCES OF BIAS IN FACIAL RECOGNITION
Face recognition is a complex, sociotechnical system where biases can originate from the algo- rithms (Danks & London (017). preprocessing steps (Dooley et al. 2020). and human interpreta- tions (Chouldechova & Roth 2020). While we do not explicitly examine these sources, we refer the reader Mchrabi et al. (202 Suresh & Guttag (2019) for a broader overview of sources of bias in machine learning.
'This work also introduces BUPT-GlohalFace, which instead approximately matches the distribution across races that of the world population.
3

---

Under review as a conference paper at ICLR 2023
3 FACE IDENTIFICATION SETUP
Face recognition has two tasks: face verification and face identification. The first refers to verifying whether a person of interest (called the probe image) and a person in a reference photo are the same. This is the setting that might be applied, e.g., to phone unlocking or other identity confirmation. In contrast, face identification involves matching a probe image against a set of images (called the gallery) with known identities. This application is relevant to search tasks, such as identifying the subject of photo from a database of driver's license or mugshot photos.
In standard face recognition pipeline, an image is generally first pre-processed by a face detection system which may serve to locate and align target faces to provide more standardized images to the recognition model. State-of-the-art face recognition models exploit deep neural networks which are trained on large-scale face datasets for a classification task. At test time, the models work as feature extractors, so that the similarity between a probe image and reference photo (in verification) or gallery photos (in identification) is computed in the feature space. In verification, the similarity score is then compared with predefined threshold, while in identification k-nearest neighbors search is performed using the similarity scores with the gallery images.
We focus on the face identification task in our experiments and explore how different kinds of data balance affect the models performance across demographic groups (specifically, the disparity in performance on male and female targets). We also analyze how algorithmic bias correlates with human bias on InterRace, a manually curated dataset specifically designed for bias auditing, with challenging face recognition questions and provided annotations for gender presentation and skin color Dooley et al. (202T).
Our experiments use state-of-the-art face recognition models. We train MobileFaceNet (Chen et al. 2018), ResNet-50, and ResNet-152 (He et al. 2016) feature extractors each with CosFace and ArcFace head which improve the class separability of the features by adding angular margin during training (Deng et al. 2019 Wang et al. 2018). For training and evaluation we use the CelebA dataset (Liu et al. 2015), which provides annotations for gender presentation. As our main research questions focus the impact of class imbalance, we pay special attention to the balance of the gender presentation attribute in our training. The original dataset contains more female identities. As such, we create balanced training containing 140,000 images from 7,934 identities with equal number of identities and total number of images from each gender presentation. We also create a perfectly balanced test set containing 14,000 images from 812 identities. The identities the train and test sets are disjoint. We call these the default train and default test sets. All models are trained with balsnced sampling to ensure equal contribution of identities to the loss. We additionally include results for models trained without werrsaamling in Appendix A.5
Recall that our research question order to answer this question, we train models on a range of deliberately imbalanced subsamples of the default training set, and test models on range of deliberately imbalanced subsamples of the default test set, in order to explore the impact on the model's performance for each gender presentation.
investigate how class imbalances affect face identification. In
To evaluate the models, we compute rank- accuracy over the test set. Specifically, for each test image we treat the rest of the test set as gallery images and find if the closest gallery image in the feature space (as defined by cosine similarity) of model is an image of the same person.
When we make comparisons with human performance (Section 7.2). we use the InterRace dataset Dooley et al. (202I). Since the InterRace dataset is derived from both the CelebA and LFW (Huang al.] 2007) datasets, we additionally train models on the InterRace- -trinneeeetainnneee split of CelebA, containing images of identities not included in the InterRace dataset. Similar to other experiments, we train models with varying levels of either identity and image imbalance.
BALANCE IN THE TRAIN SET
4.1 BALANCING THE NUMBER OF IDENTITIES
Experiment Description. To explore the effect of train set balance in the number of identities on gender presentation bias, we construct train data splits with different ratios of female and male identities, while ensuring that the average number of images per identity is the across gender
4

---

Under review as a conference paper at ICLR 2023
MobileF NctNee CosFace
ResNet-152 CosFace
Faceleeaceeeet ArcFace
ResNet-152 ArcFace
Accuracy
0.00
Male
Famale
60
02
64
0.
0.3
10
8.3
0.2
04
0.6
2.8
1.8
00
0.2
04
05
0.6
1.0
0.3
0.2
0.4
86
0.
10
X-axis: proportion male identities (solid) or images (dashed) training set (for both blue and orange)
1.00
Accuracy
Male
660
Famale
879
86
X-axis: proportion of male (blue) female (red) identities (solid) or images (dashed) in the training
Figure 2: Train Set Imbalance. Results of experiments that change the train set gender presentation balance. Top row: male and female accuracy are plotted against the proportion of male data in the train set. Bottom row: for an alternate view, female accuracy is flipped horizontally, so that is plotted against the proportion of female data in the train set. All models are tested on the default balanced test set.
presentations. Therefore, in all splits we have the same total number of images and total number of identities, but the proportion of female and male identities varies. We consider splits with 0 10, 1 9, 2 8, .... 10: : 0 ratios, each having 70,000 total images from 3967 identities. We evaluate the models on the (perfectly balanced) default test set and report rank-1 face identification accuracy as described in Section 3 More details of train set splits can be found Table II
Results. We compute accuracy scores separately for male and female test images for models trained on each of the train splits and depict them in Figure 2 with solid lines. From the first row plots, observe that higher proportion of male identities the train set leads to increase in male accuracy and decrease in female accuracy, with the most significant drops occurring near the extreme 10: O imbalance. This indicates that it is very important to have at least few identities from the target demographic group in the train set; once the representation of the minority group reaches 10%, the marginal gain of additional identities becomes less. We also observe that for most models, the female accuracy drops slightly when the proportion of female identities exceeds 80% of the training data, which does not happen to the male group. Consult Table for the numerical results.
Regarding the model architectures, MobileFaceNet models trained with both CosFace and ArcFace heads outperform ResNet models on both female and male images and have smaller absolute accuracy gap. However, the еттоr ratio is similar across the models, see Table 2 Finally. the accuracy gap is closed for all models when the train set consists of about 10% male and 90% female identities.
In addition, in the second row of Figure 2 we compare how similar these trends are for females and males by plotting female accuracy against the proportion of female identities the train set. One can see that for MobileFaceNet models the accuracy on male and female images increases similarly when increasing the proportion of "target" identities up to 80%. However, for ResNet models adding more female identities in the train set results in smaller gains compared to the effect of adding more male identities on male accuracy.
4.2 BALANCING THE NUMBER OF IMAGES PER IDENTITY
In the previous subsection, fixed the average number of images per identity in each gender presentation and adjusted the number of identities. We now will do the reverse: fix the number of identities and vary the images per identity.
5

---

Under review as a conference paper at ICLR 2023
MobileFacele CosFace
ResNet- 152 CosFace
MobileFaceNet ArcFace
ResNet- 152 AroFace
Accuracy
0.9
Male
0.00
0.50
1.00
0.25
0.75
0.00
0.50
1.00
0.25
1.00
X-axis: proportion male identities (salid) images (dashed) the (for both orange)
1.0
Accuracy
0.9
Male
Famale
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
X-axis: proportion of male (blue) or female (red) identities (solid) or images (dashed) in the test set
Figure 3: Test Set Imbalance. Results experiments that change the test set gender presentation balance. Top row: male and female accuracy are plotted against the proportion of male data in the test set. Bottom row: for an alternate view, female accuracy is flipped horizontally, so that it is plotted against the proportion of female data in the test set. All models trained the default balanced train set. For each experiment, the test set was split with 5 random seeds, and the results are averaged across seeds.
Experiment Description. We change the average number of images per male and female identity, but fix the number identities each gender presentation. We consider ratios 8, .... 8 2, each having 70, 000 images from 7, 934 identities. We do not consider more extreme ratios, which would result in identities with fewer than 3 images.
The dashed lines in Figure 2 illustrate the accuracy of the models trained described data splits. From the first row plots we see that, similar to the previous experiment, increasing the number of male images in the train set leads to increased accuracy on male and decreased accuracy female images. Interestingly, we observe a decrease performance for both demographic groups when the images of that group constitute more than 60% of train data; this is most easily visible in the second row of 2 However, we find that this effect results from the widely used class- cisssbbllanced sampling training strategy, and models trained without the default oversampling are more robust to imbalance in the number of images per identity, see details in Section A.5 and Figure 8 The "fair point" where female accuracy closest to male accuracy occurs when around 20% of images are of males.
Results.
When comparing the effect of imbalance in the number of identities and the number of images per identity (solid and dashed lines respectively in Figure we see that ResNet models are more susceptible to image imbalance than to identity imbalance, which is also a phenomenon specific to common class-balanced sampling.
5 BALANCE IN THE TEST SET
5.1 BALANCING THE NUMBER OF IDENTITIES
Experiment Description. Analogous to the train set experiments, we split the test data (the gallery) with different ratios of female and male identities, while keeping the same average number of images per identity for both demographic groups. For each ratio, we split the test data with 5 random seeds and report average rank- 1 accuracy of the models trained on default train data. The results shown in the solid lines of Figure 3 as well as in Table
We observe that increasing the proportion of identities of target demographic group in the test set hurts the model's performance on that demographic group, and this trend is similar for male and female images. Intuitively, this is because face recognition models rarely match images one
Results.

---

Under review as a conference paper at ICLR 2023
MobileFacele CosFace
ResNet- 152 CosFace
MobileFaceNel ArcFace
ResNet- ArcFace
1.0
Accuracy
0.9
Male
Female
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
X-axis: proportion of male identities (solid) or images (dashed) in the test set (for both blue and orange)
1.0
Accuracy
Malo
0.8
Fomale
0.00
0.25
0.50
0.75
1.00
0.00
0.26
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.60
0.75
1.00
X-axis: proportion of male (blue) or female (red) identitis (solid) images (dashed) the test
Figure 4: Train & Test Set Imbalance. Results of experiments that adjust the gender presentation balance in both the train and test set. Top row: male and female accuracy are plotted against the proportion of male data used in both the train and test set. Bottom row: for alternate view, female accuracy is flipped horizontally, so that it is plotted against the proportion of female data in both the train and test set. For each experiment, the test set was split with random seeds, and the results averaged across seeds.
of a different demographic group; therefore by adding more identities of particular demographic group, we add more potential false matches for images from that demographic group, which leads to higher error rates. We also that ResNet models more sensitive to the number of identities in the gallery set than MobileFaceNet models.
5.2 BALANCING THE NUMBER OF IMAGES PER IDENTITY
Experiment Description. Now, investigate how increasing decreasing the number of images per identity affects the performance and bias of the models. Again, we split the test sets with different ratios of total number of images across gender presentations, but same number of identities, each with 5 random seeds. These results are recorded as dashed lines in Figure 3 as well as in Table 5
Results. Unlike the results with identity balance, increasing the average number of images per identity leads to performance gains, since this increases the probability of match with an image of the same person. Also, image balance affects the performance more significantly than identity balance, and these trends are similar across all the models and both gender presentations. Finally, we note that the "fair point" for image balance the test set occurs at about 30% male images; contrast this with identity balance, for which no fair point appears to exist.
6 A CAUTIONARY TALE: MATCHING THE BALANCE IN THE TRAIN AND GALLERY DATA
Using our findings from above, we conclude that common machine learning techniques to create train and test splits can lead to Simpson's 's paradoxes which lead to false belief that model is unbiased. It standard practice to make random train/test splits of dataset. If the original dataset imbalanced, as is commonly the case, the resulting splits will be imbalanced in similar ways. As we have seen above, the effects imbalance in the train and test splits may oppose one another, causing severe underestimation of model bias when measured using the test split. This occurs because the minority status of a group in the train split will bias the model towards low accuracy on that group, while the correspondingly small representation in the test split will cause increase in model accuracy, partially or entirely masking the true model bias. The results for these experiments are presented in Figure and Tables
7

---

Under review as a conference paper at ICLR 2023
Balancing the number of identities We create train and test sets with identical distributions of identities. Recalling the results from prior experiments, increasing the number of identities for the target group in the training stage improves accuracy that group, while adding more identities in the gallery degrades it. Interestingly, when we increase the proportion of male identities in both train and test sets, observe gains in both male and female accuracy, and that trend is especially strong for ResNet models.
Balancing the number of images per identity Having more images is beneficial in both train and test stages. Therefore, the effect of image balance is amplified when both train and test sets are imbalanced in a similar way. Similar to the train set experiments, having more than 70% female images in both train and test sets leads to slight drops in female accuracy on ResNet models, which again is a result of the default class-balanced oversampling strategy.
BIAS COMPARISONS
We ask two concluding questions: one about whether class imbalance captures all the inherent bias and the other about how the bias we see compares to human biases. First, we explore how data imbalances cause biases in random networks and find surprising conclusions. Then, we ask how class imbalances in machines compare to how humans exhibit bias on face identification tasks.
7.1 BIAS IN RANDOM FEATURE EXTRACTORS
Given a network with random initializations, we would expect that evaluation balanced test set would result in equal performance on males and females, and likewise that male performance set with a particular proportion of male identities would be the same as female performance when that proportion is reversed. However, this is not the case. We test randomly initialized feature ex- tractors on galleries with varying levels of image imbalance. Figure 55 summarizes the results of these experiments. We observe that both models have higher male performance when the test set perfectly balanced, and that performance on males higher when they make up 80% of the test set than female performance when they make up 80% of the test set. This provides strong evidence that there are sources of bias that lie outside what we explore here and which are potential confounders to thorough study of bias in face identification; further work on this is warranted.
MobileFaceNet
ResNet-152
Male
Accuracy
0.15
Female
0.10
0.05
0.25
0.50
0.75
0.25
0.50
0.75
Proportion of male images in the test set
Figure 5: Random Feature Extractors. The plot illustrates male (blue) and female (orange) accuracy of random feature extractors against the proportion of male images in the test set. The standard deviation is computed across 10 random initializations.
7.2 ARE MODELS BIASED LIKE HUMANS?
Numerous psychological and sociological studies have identified gender, racial, and other biases in human performance face recognition tasks. Dooleyy et al. (202T) studied whether humans and FR models exhibit similar biases. They evaluated human and machine performance on the curated InterRace test questions, and found models indeed tend to perform better on the same groups as, and with comparable gender presentation bias ratios to, humans. In this section, we use their human survey data to explore two related questions: how correlated are model and human performance as the question level, and how does this change with different levels of imbalance in training data?
To answer these questions, we define a metric which allows us to distinguish how well model performs on each InterRace identification question. Let
|| Uprobe Ufaise||2
La ratio
Uprobe Utrue 2|2 + || Uprobe Dfalse l22
8

---

Under review as a conference paper at ICLR 2023
MobileFaceNet CosFace
MobileFaceNet. ArcFace Arcaace
ResNett, 152 CosFace
ResNet, 152 ArcFace
0.500
0.45
Correlation
0.450
0.425
0.400
0.375
Hole
Female Overal
0.350
0.325
Image
1.0
1.0
Proportion of male identities (solid) d) or images (dashedd in the train set
Figure 6: Pearson correlation of L2 ratio vs. human accuracy for various models as proportion of male training data varies.
where Uprobes Utrues ffale are the feature representations of the probe image, the correct gallery image, and the nearest incorrect gallery image, respectively This value is when the probe and correct image's ş representations coincide, O when the probe and incorrect image's s representations coincide and 0.5 when the probe's ' representation is equidistant from those of the correct and incorrect image. Figure 7 depicts examples scatterplots comparing model confidence to human accuracy on each InterRace question.
Figure shows the correlation between L2 ratio and human performance for various models at each of the training imbalance settings that we have considered in carlier experiments. We that the correlation between these values over all questions tends to rise as the proportion of male training data increases. However, the correlation when separately considering male and female questions does not rise as monotonically, or as much, from left to right as the overall correlation does. This suggests that the correlation between human and machine performance is largely driven by the fact that models and humans both find identifying females more difficult than identifying males, and that this disparity exacerbated when the model question is trained on male- dominated data. On the other hand, the particular males and females that are easier or harder to identify appear to differ between models and humans, which suggests the reasons for bias humans and machines are different.
8 ACTIONABLE INSIGHTS
We note five actionable insights for machine learning engineers and other researchers from this work. First, overrepresenting the target demographic group can sometimes hurt that group. Sometimes having more balanced data is the key. Also, class- balanced sampling might hurt representation learning when the data is not balanced with respect to the number of images per identity. Second, gallery set balance is as important as train set balance, contrary to how face verification class imbalances work. Third, having the same distribution identities and average number of images per identity is not unbiased way to evaluate a model, since the effects of balance in train and test sets can be amplified (in case images) or cancel each other (in case of identities). Fourth, train not the only cause of bias in face identification evaluation since
and test class imbalances random models do not perform equally poorly on female and male images. Finally, even though both humans and machine find female images more difficult recognize, it seems that the reasons for bias are different in people and models. We know that this work sheds light on common mistakes in bias computations for many facial recognition tasks and hope that auditors and engineers will incorporate our insights into their methods.
We note that other measures of confidence in k-nearest neighhors setting, such those discussed in Dalitz 2009), are inappropriate for this application.

---

Under review as a conference paper at ICLR 2023
9 REPRODUCIBILITY STATEMENT
We include the code and instructions for reproducing the results of our experiments in the supplementary materials. In our experiments we use CelebA facial dataset, which is pub- licly available for research purposes only (Liu et al., (015) and can be downloaded at https://mmlab.ii.cm cuhk. htt.kkedd...//oojes/scceeeeabbe. to train the models in Appendix A.2
We also include details on hyperparameters used
REFERENCES
Vitor Albiero, Krishnapriya KS, Kushal Vangara, Kai Zhang, Michael C King, and Kevin W Bowyer. Analysis of gender inequality in face recognition accuracy. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision Workshops, pp. 81-89, 2020.
Mohsan Alvi, Andrew Zisserman, and Christoffer Nelläker. Turning blind eye: Explicit removal of biases and variation from deep neural network embeddings. In Proceedings of the European Conference Computer Vision (ECCV) Workshops, pp. 0-0, 2018.
Mateusz Buda, Atsuto Maki, and Macicj A Mazurowski. A systematic study of the class imbalance problem in convolutional neural networks. Neural Networks, 106:249-259, 2018
Jacqueline Cavazos, P Jonathon Phillips, Carlos D Castillo, and Alice J O'Toole. Accuracy comparison across face recognition algorithms: Where transactions on biometrics, behavior, and identity science, 311):001- 2020.
measuring race bias? IEEE
Nitesh V Chawla, Kevin W Bowyer, Lawrence O Hall, and W Philip Kegelmeyer. Smote: synthetic minority over-sampling technique. Journal of artificial intelligence research, 16:321-357, 2002.
Nitesh V Chawla, Aleksandar Lazarevic, Lawrence O Hall, and Kevin W Bowyer. Smoteboost: Improving prediction of the minority class in boosting. In European conference on principles of data mining and knowledge discovery, pp. 107-119. Springer, 2003.
Nitesh V Chawla, Nathalie Japkowicz, and Aleksander Kotcz. Special issue on learning from imbalanced data sets. ACM SIGKDD explorations newsletter, 6(1):1-6, 2004.
Sheng Chen, Yang Liu, Xiang Gao, and Zhen Han. Mobilefacenets: Efficient cnns for accurate real-time face verification on mobile devices. In Chinese Conference on Biometric Recognition, pp. 428-438. 438. Springer, 2018.
Alexandra Chouldechova and Aaron Roth. A snapshot of the frontiers of fairness in machine learning. Communications of the ACM, 63(5)882-89 09,882899-10 2020.
Christoph Dalitz. Reject options and confidence measures for knn classifters. Schriftenreihe des Fachbereichs Elektrotechnik und Informatik Hochschule Niederrhein, 8:16-38, 2009.
David Danks and Alex John London. Algorithmic bias in autonomous systems. In IJCAI, volume 17, pp. 4691-4697, 2017.
Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou. Arcface: Additive angular margin loss for deep face recognition. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 4690-4699, 2019.
Samuel Dooley, Tom Goldstein, and John P Dickerson. Robustness disparities in commercial face detection. arXiv preprint arXiv:2108. 12508, 2020.
Samuel Dooley, Ryan Downing, George Wei, Nathan Shankar, Bradon Thymes, Gudrun Thorkelsdot- tir, Tiye Kurtz-Micn Rachel Mattson, Olufemi Obiwumi, Valeriia Cherepanova, et al. Comparing human and machine bias in face recognition. arXiv preprint 1 2021.
Charles Elkan. The foundations of cost-sensitive learning. In International joint conference on artificial intelligence, volume 17, pp. 973-978. Lawrence Erlbaum Associates Ltd, 2001.
10

---

Under review as a conference paper at ICLR 2023
Patrick Grother, Mei L Ngan, Kayee K Hanaoka, al. Face recognition vendor test part 3: demographic effects. 2019.
Matthew Gwilliam, Srinidhi Hegde, Lade Tinubu, and Alex Hanson. Rethinking common assumptions to mitigate racial bias in face recognition datasets. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4123-4132, 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Paulina Hensman and David Masko. The impact of imbalanced training data for convolutional neural networks. Degree Project in Computer Science, KTH Royal Institute of TTTccholooyy, 2015.
Gary B. Huang, Manu Ramesh, Tamara Berg, and Erik Leamed-Miller. Labeled faces in the wild: A database for studying face recognition in unconstrained environments. Technical Report 07-49, University of Massachusetts, Amherst, October 2007.
Justin M Johnson and Taghi M Khoshgoftaar. Survey on deep learning with class imbalance. Journal of Big Data, 6(1):1-54, 2019.
Salman H Khan, Munawar Hayat, Mohammed Bennamoun, Ferdous A Sohel, and Roberto Togneri. Cost-sensitive learning of deep feature representations from imbalanced data. IEEE transactions on neural networks and learning systems, 29(8):3573-3587 2017.
Brendan F Klare, Mark J Burge, Joshua C Klontz, Richard W Vorder Bruegge, and Anil K. Jain. Face recognition performance: Role of demographic information. JEEE Transactions on Information Forensics and Security, 7(6): 180- 2012.
Adam Kortylewski, Bernhard Egger, Andreas Schneider, Thomas Gerig, Andreas Morel-Forster, and Thomas Vetter. Empirically analyzing the effect of dataset biases on deep face recognition systems. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition Workshops, pp. 2093-2102, 2018.
Bartosz Krawezyk. Learning from imbalanced data: open challenges and future directions. Progress in Artificial Intelligence, 5(4):221-232 2016.
Hansang Lee, Minseok Park, and Junmo Kim. Plankton classification on imbalanced large scale database via convolutional neural networks with transfer learning. In 2016 IEEE international conference on image processing (ICIP), pp. 3713-3717. IEEE, 2016.
Tsung-Yi Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, pp. 2980-2988, 2017.
Charles X Ling and Victor S Sheng. Cost-sensitive learning and the class imbalance problem. Encyclopedia of machine learning, 2011:231-2355 2008.
Xu- Ku-Ying Liu, Jianxin Wu, and Zhi-Hua Zhou. Exploratory undersampling for iaaaaac learning. IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics), 39(2): 539-5599 550, 2008.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. learning face attributes the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
Ninarch Mehrabi, Fred Morstatter, Nripsuta Saxena, Kristina Lerman, and Aram Galstyan. A survey on bias and fairness in machine learning. ACM Computing Surveys (CSUR), 54(6):1-35, 2021.
P Jonathon Phillips, W Todd Scruggs, Alice J O'Tooe,, Patrick J Flynn, Kevin W Bowyer, Cathy I. Schott, and Matthew Sharpe. Frvt 2006 and ice 2006 large-scale experimental results. IEEE transactions on pattern analysis and machine intelligence, 32(5):831-844 2009.
P Jonathon Phillips, Fang Jiang, Abhijit Narvekar, Julianne Ayyad, and Alice J O'Toole. An other- race effect for face recognition algorithms. ACM Transactions on Applied Perception (TAP), 8(2): 1-11, 201
11

---

Under review as a conference paper at ICLR 2023
Samira Pouyanfar, Yudong Tao, Anup Mohan, Haiman Tian, Ahmed Kaseb, Kent Gauen, Ryan Dailey, Sarah Aghajanzadeh, Kung-Hsiang Lu, Shu-Ching Chen, et al. Dynamic sampling in convolutional neural networks for imbalanced data classification. In 2018 IEEE conference multimedia information processing and retrieval (MIPR), pp. 112-117. IEEE, 2018.
Joseph P Robinson, Gennady Livitz, Yann Henon, Can Qin, Yun Fu, and Samson Timoner. Face recognition: too bias, or not too bias? Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 0-1, 2020.
Yanmin Sun, Mohamed S Kamel, Andrew KC Wong, and Yang Wang. Cost-sensitive boosting for classification of imbalanced data. Puttern recognition, 40(12):3358-3333 -3378, 2007.
Yi Sun, Yuheng Chen, Xiaogang Wang, and Xiaoou Tang. Deep learning face representation by joint ceaiificcationvveiiiaatto Advances in neural information processing systems, 27, 2014.
Harini Suresh and John V Guttag. A framework for understanding unintended consequences of machine learning. arXiv preprint 190.
2, 2019.
Jason Van Hulse, Taghi M Khoshgoftaar, and Amri Napolitano. Experimental perspectives on learning from imbalanced data. Proceedings of the 24th international conference on Machine learning, pp. 935-942, 2007.
Hao Wang, Yitong Wang, Zheng Zhou, Xing Ji, Dihong Gong, Jingchao Zhou, Zhifeng Li, and Wei Liu. Cosface: Large margin cosine loss for deep face recognition. In Proceedings of the IEFE conference on computer vision and pattern recognition, pp. 5265-5274, 2018.
Mei Wang and Weihong Deng. Mitigating bias in face recognition using skewness aware reinforce- ment learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9322-9331, 2020.
Shoujin Wang, Wei Liu, Jia Wu, Longbing Cao, Qinxue Meng, and Paul Kennedy. Training deep neural networks on imbalanced data sets. In 2016 international joint conference neural networks (IJCN), pp. 4368-4374, IEEE, 2016.
Yaobin Zhang and Weihong Deng. Class-balanced training for deep face recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 824-825, 2020.
12

---

Under review as a conference paper at ICLR 2023
A APPENDIX
A.1 BROADER IMPACT AND LIMITATIONS
In this work we explore the effects of various forms of data imbalance on bias in face identification, and we hope that practitioners will take our findings into account when performing bias auditing. However, it is important to understand that biases can originate from various sources besides data imbalance and therefore models should be carefully evaluated for other bias factors.
The availability of high quality datasets, which are suitable for identification task (as opposed to verification), have demographic metadata/ammotation for both train and test sets, and contain enough identities and images belonging to each demographic group to allow for subsampling, is extremely limited. For this reason focus solely gender bias and leverage CelebA dataset, which meets these criteria. Also, using a binary demographic attribute (such as gender, when restricting only to male- and female-presenting identities) allows the proportion of data in each group be conveniently tuned as single parameter, which in turn makes interpreting results more straightforward.
A.1.1 DETAILS ON CELEBA
CelebA is publicly available dataset, which is constructed from CelebFaces dataset Sun et al. 2014) and contains face images collected from the Internet.
A.2 TRAINING DETAILS
We pre-process CelebA images by aligning them using the provided facial landmarks and cropping to 1 11x111 12 size. All face recognition models are trained with Focal loss Lin et al. (2017) using SGD for 100 epochs with learning rate of 0.1, momentum of 0.9 and weight decay of 5e-4. The learning rate reduced by 10 times at epochs 35, 65 and 95. Horizontal flip data augmentation is used during training. For the model architectures, repository face. evoLVe PyTorch Ti machines and each experiment takes from 12 hours of compute time on one GPU,
implementation from publicly available github B We run our experiments on NVIDIA GeForce RTX 2080
A.3 EXPERIMENTAL DETAILS
Table 1: Details the number of identities, total number of images and average number of images per identity used in experiments with train and test data balance. We also report statistics for the default train and test sets. M denotes male, F denotes female.
Setting
M ids
F ids
Total ings
Total F imns
M ings/id
F imng/id
Total ids
Total inng
Train default
3967
3967
70k
70k
17.65
17.65
7934
140k
Train balance
3967
3967
70k
17.65
17.65
3967
Train ing balance
3967
3967
14k 36k
14k-56k
3.53 14,11
3.53 14.11
7934
70k
Test default
406
406
7k
17.24
17.24
812
14k
Test balance
0 406
0-406
0-7k
0-
17.24
17.24
406
Test ing balance
406
406
1.4k 5.6k
5.6k
3.45 13,80
3:45 - 13.80
812
7k
A.4 MODEL vs. HUMAN SCATTERPLOTS
Figure 7 shows two example scatterplots comparing model 1.2 ratio (our proxy for confidence defined
in section 7.2 against human accuracy on each question in the InterRace identification dataset. (Dooeyy et al. 2021).
A.5 RESULTS FOR MODELS TRAINED WITHOUT CLASS-RALANCER SAMPLING.
To explore the effect of class balanced sampling on the results of our experiments, we train additional models without any oversampling strategies. Figures g 10 show results of our experiments for MobileFaceNet and ResNet- 152 models trained without oversampling. We find that most trends are
13

---

Under review as a conference paper at ICLR 2023
Mare Female
Male
Femile
0.65
12
0.55
0.55
0.50
0.45
0.45
Hummn
Figure 7: Scatterplots of model L2 ratio vs. human accuracy on each question in the InterRace identification dataset. Both models are MobileFaceNets trained with CosFace loss, (Left) a model trained on exclusively female images. (Right) a model trained on exclusively male images.
similar to ones observed in the models trained with baas-bbllanced sampling, however models trained without oversampling are more robust to balance in the number of images per identity, see Figure & In particular, the effect balancing the number of images (dashed lines) similar to the effect of balancing the number of identities (solid lines) for all models, but Res trained with ArcFace head. This leads us to a conclusion that using class- balanced sampling strategy is not beneficial in scenarios of severe imbalance in number of images per identity in face recognition models.
MobileFaceNet CosFace
ResNet-152 CosFace
MobileFaceNi ArcFace
ResNet- ArcFace
Accuracy
0.85
0.6
0.8
0.4
0.6
X-axis: proportion of male identities (solid) or images (dashed) the training set (for both blue and orange)
1.00
0.95
Accuracy
Male
0.80
Female
0.75
0.0
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
1.0
0.0
0.2
0.4
0.6
0.8
1.0
X-axis: proportion of male (blue) or female (red) identities (solid) or images (dashed) in the training
Figure 8: Train Set Imbalance Results of experiments that change the train set gender presentation balance for MobileFaceNet and ResNet- 152 models trained without class-balanced sampling. Top row: male and female accuracy are plotted against the proportion of male data the train set. Bottom row: for an alternate view, female accuracy is flipped horizontally, so that it is plotted against the proportion of female data in the train set. All models are evaluated on the default balanced test set.
A.6 ADDITIONAL. PLOTS AND TABLES
Figure || shows the results of the train set imbalance experiment when evaluated on the InterRace test set. Figures 12 show results for ResNet-50 (with ResNet-152 152 results shown again for comparison). Tables 2 precisely detail the number of male and female identities and images used in each experiment, as well the accuracy on male and female targets and the female-to-male error ratio.
Figures
14
supplement those in sections
14

---

Under review as a conference paper at ICLR 2023
MobileFaceNett CosFace
ResNet-152 CosFace
MobileFaceNee ArcFace
ResNet-185 ArcFace
Accuracy
0.90
0.85
accuracy
accuracy
X-axis: proportion male identities (solid) or images (dashed) in test set (for both blue and orange)
1.00
0.95
Accuracy
0.90
Male
Famale accuracy
0.78
0.0
0.4
0.8
08
0.0
0.2
0.4
0.6
0.8
1.0
0.2
0.4
08
1.0
0.0
0.2
0.4
0.8
0.8
X-axis: proportion of male (blue) or female (red) identities (solid) or images (dashed) in the test set
Figure 9: Test Set Imbalance. Results of experiments that change the test set gender presentation balance for MobileFaceNett and ResNet- 152 models trained without class-balanced sampling. Top row: male and female accuracy are plotted against the proportion of male data in the test set. Bottom for alternate view, female accuracy flipped horizontally, proportion of female data in the test set. All models are trained on the default balanced train set. For each experiment, the was split with 5 random seeds, and the results are averaged across seeds.
that is plotted against the
MobileFaceNeet CosFace
ResNet-152 CosFace
MobileFaceNeet ArcFace
ResNet-152 ArcFace
Acouracy
0.65
Male accuracy
0.80
Female accuracy
1.00
X-axis: proportion of male identities (solid) or images (dashed) in the test set (for both blue and orange)
Accuracy
0.90
0.85
accuracy
accuracy
X-axis: proportion of male (blue) or female (red) identities (solid) or images (dashed) in the test set
Figure 10: Train & Test Set Imbalance. Results of experiments that adjust the gender presentation balance both the train and class-balanced sampling. Top row: male and female accuracy are plotted against the proportion of male data used in both train and test set. Bottom row: for an alternate view, female accuracy is flipped horizontally, so that it is plotted against the proportion of female data in both the train and test set. For each experiment, the test set was split with random seeds, and the results are averaged across seeds.
for MobileFaceNet and ResNet- 152 models trained without
15

---

Under review as a conference paper at ICLR 2023
Mobilef
ResMet
ResNet
0.85
Haie
3.80
Female
0.75
Image
0.6
0.8
1.0
0.4
0.6
0.8
0.4
0.6
0.8
0.4
0.6
0.8
Images (nashed)
Figure 11: Train Set Imbalance. Results of experiments testing models trained with different gender presentation balance the InterRace dataset. These plots are analogous the first of Figures and 12
ResNet-152 CosFace
ResNet-50 CosFace
ResNet-152 ArcFace
ResNet-50 ArcF
1.00
0.95
Accuracy
Male
0.80
Fomati
0.75
0.0
0.2
0.4
0.8
0.6
1,00,0
0.2
0.4
0.5
0.5
1.00.0
0.2
0.4
0.5
0.8
1.00.0
0.2
0.4
0.8
0.5
1.0
X-axis: proportion of male identities (solid) or images (dashed) in the training set (for both blue and orange)
Accuracy
0.88
0.80
Female
X-axis: proportion of male (blue) female (red) identities (solid) images (dashed) in the training set
Train Set Imbalance. Results of experiments that change the train set gender presentation balance for ResNet-155 152 and ResNer-50 models. Top row: male and female accuracy are plotted against the proportion of male data in the train set. Bottom row: for an alternate view, female accuracy flipped horizontally, so that it is plotted against the proportion of female data in the train set. All models evaluated on the default balanced test set. Cf. Figure ?
Figure 12:
16

---

Under review as a conference paper at ICLR 2023
ResNet-152 CosFace
ResNet-50 CosFact ace
ResNet-152 ArcFace
ResNet-50 ArcFace
Accuracy Accuracy
0.90
0.85
Famale
X-axis: proportion male identities (solid) or images (dashed) in test set (for both blue and orange)
0.88
0.90
Male
Famalo
08
0.2
0.8
1,00,0
0.2
0.6
1.0000
0.2
0.6
0.8
1.00.0
0.2
0.4
0.8
0.8
X-axis: proportion of male (blue) or female (red) identities (solid) or images (dashed) in the test set
Figure 13: Test Set Imbalance. Results of experiments that change the test set gender presentation male and female accuracy plotted against the proportion of male data the test set. Bottom row: for an alternate view, female accuracy flipped horizontally, so that it is plotted against the proportion of female data in the set. All models are trained on the default balanced train set. For each experiment, the test set was split with random seeds, and the results are averaged across seeds. Cf. Figure
balance for ResNet-152 152 and ResNer-50 models. Top
ResNet-152 CosFace
ResNet-50 CosFace
ResNet-152 ArcFace
ResNet-50 ArcFace
Accuracy
0.90
0.88
X-axis: proportion of male identities (solid) or images (dashed) the test set (for both blue and orange)
1.00
0.98
Accuracy
Male
Fermae
0.78
0.0
0.2
0.4
0.8
0.8
1,00,0
0.2
0.4
0.8
0.5
1.000
0.2
0.4
0.5
0.8
1.00.0
0.2
0.4
0.8
0.8
1.0
X-axis:
proportion male (blue) or female (red) identities (solid) or images (dashed) in test set
Figure 14: Train & Test Set Imbalance. Results of experiments that adjust the gender presentation balance in both the train and test set for ResNet- 152 and ResNet-50 models. Top row: male and female accuracy are plotted against the proportion of male data used in both the train and test set. Bottom row: for alternate view, female accuracy is flipped horizontally, that it is plotted against the proportion of female data both the train and test set. For each experiment, the test set was split with 5 random seeds, and the results are averaged across seeds. Cf. Figure 4
17

---

Under review as a conference paper at ICLR 2023
Table 2:
Train Set Id Imbalance. The female and male accuracy computed over the default balanced test set for models trained on data with various ratios of number of male and female identities. See details of the experiment in Section 4.1
Model
Ids Ratio
ids
M imgs
imgs
M
Facc
Error Ratio
0:
3967
70k
0.938
0.76
3570
2 8
793
3174
14k
56k
0.946
0.941
L09
1190
2777
21k
49k
0.952
0.942
1.21
2380
42k
MFN CosFacc
5 5
1984
1984
35k
35k
0.961
0.940
1.54
2580
1587
42%
28k
0.964
0.936
1.78
8 8:2
3174
793
56k
14k
0.964
0.928
2.00
9
3570
397
63k
7k
0,968
0.924
2.37
U: 10
()
3967
70K
0311
0.937
0.71
397
3570
63k
0.977
0.940
0.95
793
3174
56k
0.948
0.939
1.17
3 7
1190
2777
21K
49%
0.952
0.939
1.27
1587
2380
28k
42k
0.953
1.26
MFN ArcPace
6 2380
1587
42k
28k
0.96
0.937
1.00
7
2777
1190
49k
21k
0.963
0.934
1.78
9 3570
397
63%
7k
0.966
0.914
2.53
10 0
3967
0
70K
0
0.966
0.886
3.35
70k
1 9
397
3570
7K
63k
0,902
0.894
L.8
2
793
3174
56k.
0.918
0.896
1.27
4:6
1387
2380
28k
42k
0.931
0.892
1.57
ResNet-152 152 CosFace
5 5:5
1984
1984
35L
35k
0,936
0.897
1.11
7:3
2777
L190
49%
21k
0.949
0.889
2.18
8
3174
793
56k
14k
0.951
0.886
2.33
0.872
10 D
3967
70L
0
0.952
0.872
3.71
D: 10
0)
3967
70K
0.803
0.368
0.67
2
793
3174
56k
0.885
0.866
1.17
3:7
1190
2777
21k
49k
0.897
0.859
1.37
ResNet-152 ArcFace
5 5:5
1984
1984
35k
35k
0.913
0.863
1.57
6
2380
1587
42k
28k
0.920
0,850
1.88
8 :
3174
793
14k
0.922
0.832
2.47
9:
3570
397
63k
7k
0.931
0,814
2.70
0: 10
O
3967
70
0.828
0.873
0.74
1 9
397
3570
63
0
0.876
1.04
3
1190
2777
49
0.100
0.879
1.34
4
1587
2380
28
42
0.917
0.881
1.43
ResNet-50
1984
6 14:44
2380
1567
28
0.944
0.878
1.85
7:3
2777
1190
49
21
0.931
0.868
1.91
8
3174
793
14
0.868
2.13
1:9
3570
397
0.944
0.853
2.63
0: 10
3967
70
0
0.940
0.807
3.22
1 1:9
397
3570
0.836
0.852
0.90
2 8
793
3174
56
0.871
0.854
1.13
4:
1587
2380
0.893
0.845
1.45
ResNet-50 ArcFace
51
1984
1984
35
35
0.897
0.845
1.50
6
2380
1587
28
0913
0.843
1.80
2777
1190
0.917
2,00
8 8:2
3174
793
56
14
0.924
0.823
2.33
3570
397
7
0.966
0.797
2.74
0.734
18

---

Under review as a conference paper at ICLR 2023
Table 3: Train Set Img Imbalance. The female and male accuracy computed over the default balanced test set for models trained data with various ratios of number of images per male and female identity. See details of the experiment in Section
Model
Ing Ratio
M ids
F ids
M imgs
# F imgs
M Acc
F Acc
Error Ratio
2
3967
3967
14k
56k
0.932
0.927
1.07
3 7
3967
3967
21k
49k
0.949
0.931
1.35
3967
28k
42k
0.955
0.931
1.53
MFN CosFace
5 5
3967
3967
35k
35k
0.956
0.930
1.59
3967
42k
0.959
0.929
7
3967
3967
49K
21k
0.957
0.918
1.91
8 2
3967
3967
56k
14k
0.957
0.892
2
3967
0.937
L.13
3 7
3967
3967
21k
49k
0.953
0.939
1.30
MFN ArcFace
5
3967
3967
35K
35k
0.962
0.939
1.61
6
3967
3967
42k
28k
0.963
0.937
1.70
7
3967
3967
49%
21k
0.961
0.929
1.82
8
3967
3967
56k
14k
0.960
0.914
2.15
3967
3
3967
3967
21k
49k
0.908
0.886
1.24
4
3967
3967
28k
42K
0.923
0.890
ResNet-152 CosPace
5
3967
3967
35k
33k
0.935
0.888
1.72
6
3967
3967
42k
28k
0.934
0.862
2.09
8 2
3967
3967
56k
14k
0.928
0.753
3.43
2 8
3967
3967
14k
50k
0.839
0,851
3:7 7
3967
3967
21k
49k
0.899
0.873
1.26
4
3967
3967
28k
42k
0.916
0.881
1.42
ResNet-152
6
3967
3967
42k
28k
0.928
0.856
2.00
7:
3967
21k
0.925
8
3967
3967
56k
14k
0.922
0.748
3.23
2
3967
3967
14k
56k
0.829
0.845
0.91
3967
4 6
3967
3967
28k
42k
0.909
0.870
1.43
ResNet-50 CosFace
5:
3967
3967
35k
35k
0.917
0.864
6:
3967
3967
42k
28k
0.920
0.844
1.95
7
3967
3967
49k
21k
0.922
0.817
8
3967
3967
56k
14k
0,914
0.722
3,23
2
3967
3967
14k
56k
0.808
0.823
0.92
3
3967
3967
21k
49k
0.875
0.845
1.24
4 6
3967
3967
28k
42k
0.900
0.853
1.47
ResNet ArcFuce
3967
3967
35k
35k
0.916
0.853
1.75
6:
3967
3967
42k
28k
0.915
0.837
1.92
7
3967
3967
49k
21k
0.917
0.798
2.43
19

---

Under review as a conference paper at ICLR 2023
Table 4: Test Set Id Imbalance. The female and male accuracy for models trained on default train set computed on test set with various ratios number of male and female identities. See details of experiment in Section 5.1
Model
Ids Ratio
M ids
# FF ids
# M imgs
# Finngs
M Acc
F Acc
Ertor Ratio
0: 10
0
406
0
7000
0.961
1
41
365
700
6300
0.983
0.961
2:25
3
122
284
2100
4900
0.981
0,960
2,09
4
162
244
2800
4200
0.981
0.962
2.00
MPN
6
244
162
4200
2800
0.980
0.963
1.83
7
284
122
4900
2100
0.979
0,964
1.77
1:9
365
41
6300
700
0.978
0.962
1.72
406
7000
0
0.978
7000
0,959
1 9
41
365
700
6300
0.980
0.959
2.07
2
81
325
1400
5600
0,980
0.960
1.98
4
162
2800
4200
0.98
0.960
2.05
MFN ArdFace
5 5
203
203
3500
3500
0.979
0,961
1.89
1.81
7 :
284
4900
2100
0.979
0.962
1.84
8
325
81
5600
1400
0.979
0.968
1,50
406
7000
0.978
U
U
U
7000
0.344
2 8
81
325
1400
5600
0.979
0.945
2.58
3
122
284
2100
4900
0.977
0.946
2.37
ResNet- 152 CosPace
5
203
203
3500
3500
0.974
0.947
2.01
6
244
162
4200
2800
0.974
0,949
1.99
7
284
4900
2100
0.974
0.952
1.87
8 3::22
325
5600
1400)
0.973
0.957
1.59
1 ::
365
41
6300
700
0.971
0.958
1.47
406
7000
0:
7000
0.920
1
41
365
700
6300
0.974
0.920
3,09
3
122
2100
4900
0.968
0.922
2.42
4
162
244
2800
4200
0.966
0.928
2.12
ResNet-152 ArcFace
203
6
244
4200
2800
0.962
0.933
1.76
7 3
284
122
4900
21000
0.96
0.936
1.65
325
1400
1 9
365
6300
700
0.959
0.950
1,20
0 10
406
7000
0
0.958
1
41
365
700
6300
0.973
0.931
2.54
2 8
81
725
1400
5600
0.972
0.933
2.35
3
4900
0.933
2.15
4
162
244
2800
4200
0.967
0.937
1,89
ResNet-50 CosFace
5
203
203
3500
3500
0.965
0.936
1.83
6
244
4200
2800
0.965
0.939
1.74
7 :
284
4900
2100
0.964
0.942
1.61
8
325
81
$600
1400)
0.964
0.950
1.38
365
6300
700
0.949
1.31
406
7000
0.961
0 T0
()
406
01
7000
0.504
2.66
2
81
325
1400
5600
0.960
0.908
2.33
3
122
284
21000
4900
0.957
0.907
2.18
2800
4200
1.99
ResNet-50 50 ArcFace
203
3500
6
244
162
4200
2800
0.950
0.920
1.60
7
284
122
4900
2100
0.950
0.924
1.52
1400
1:9
365
41
6300
700
0.946
0.940
1.11
0:
406
7000
0.946
20

---

Under review as a conference paper at ICLR 2023
Table 5: Test Set Img Imbalance. The female and male accuracy for models trained on default train computed on set with various ratios of number of images per male and female identities. See details of the experiment in Section 5.2
Model
Img Ratio
M ids
# ids
M imgs
# F imgs
M Acc
F Acc
Error Ratio
2
406
406
1400
5600
0.941
0.957
0.72
3 7
406
406
2100
4900
0.959
0.956
1.06
406
2800
4200
0.962
0.952
1.27
MFN CosFace
5 5
406
406
3500
3500
0.967
0.946
1.64
406
2800
0.970
0.940
7
406
406
4900
2100
0.973
0.925
2.75
8
406
406
5600
1400
0.975
0.894
4,23
2
406
5000
0.939
0.72
3 7
406
406
2100
4900
0.956
0.954
1.03
MFN ArcFace
5
406
406
3500
3500
0.966
0.947
1.54
6
406
4200
2800
0,969
0.941
1.91
7
406
406
4900
2100
0.972
0.928
2.58
8
406
406
5600
1400
0.974
0.901
3.87
3
406
406
2100
4900
0.946
0.934
1.21
4
406
2800
4200
0.952
0.927
ResNet-152 CosPace
5
406
406
3500
3500
0.958
0.921
1.89
6
406
406
4200
2800
0.962
0.912
2.32
8 2
406
406
$600
1400
0.967
0.855
4.37
2: 8
406
406
1400
5600
0.838
0,912
0.79
3: 7
406
406
2100
4900
0.916
0.909
1.09
4:
406
406
2800
4200
0.930
0.901
1.42
ResNet-152 ArcFace
6:
406
406
4200
2800
0.946
0.878
2.27
7:
406
0.950
0.853
8 2
406
406
$600
1400
0.954
0.798
4.38
2
406
406
1400
5600
0.905
0.974
0.80
4 6
406
406
2800
4200
0.940
0.914
1.43
ResNet-50 CosFace
5:
406
3500
0.947
0.904
6:
406
406
4200
2800
0.952
0.894
2.23
7
406
4900
2100
0.956
0.872
2.87
8
406
406
$600
1400
0,958
0.827
2
406
406
1400
5600
0.870
0.893
0.82
3
406
2100
4900
0.904
0.888
4:66
406
406
2800
4200
0.916
0.879
1.45
ResNet ArcFuce
5
406
3500
3500
0.925
0.866
1.79
6
406
406
4200
2800
0.933
0.853
2.20
7
406
406
4900
2100
0.936
0.826
2.74
0.766
21

---

Under review as a conference paper at ICLR 2023
Table Train & Test Set Id Imbalance. The female male accuracy models trained and tested on data with the same ratios of male and female identities. See details of experiment in Section
Model
Ratio
Acc
Error
1: 9
0.963
0.943
1.54
8 0,966
0.947
1,56
6
0.967
0.945
1.63
MPN CosFace
5 6
0.965
0.943
1.63
7: 3
0,968
0,946
1.66
8: 2
0,969
0.946
1.72
0.971
0 10
0,972
0 IO
0.345
0,962
0.962
0.947
1.42
0.962
0.943
1.52
MPN AreFace
5
0,964
0,944
1.54
4
0.968
0.944
1.72
0,969
0.947
1: 9
0.968
0.949
1.63
10
0,969
0 10
0.501
: 9
0.943
0.906
1.65
0,947
0.902
1.86
6
0.946
0.907
1.70
ResNet-152 CosPace
0.946
0,952
0.916
1.73
7: 3
0.955
0.919
1.79
2
0.956
0.925
1.69
9
0,954
0.931
1.49
0 10
0.956
9
0.925
0.874
1.67
2 8
0.924
0,878
1.61
0.926
0.877
1.67
0.925
0.877
ResNet-152 ArcPace
5 5
0.928
0.882
1.64
0,930
0,890
1.58
3
0.938
1.75
2
0.937
U 0,900
1.59
9 0,936
0.906
1,46
0.942
0 TO
0.890
9
0.933
0,886
1.69
8
0.930
0,890
1.57
3. 7
0.934
0.892
1.63
0.934
0.895
1.60
ResNet-50 CosFace
0.936
0.898
1.58
G 4
0.942
0.901
1,70
5
0,940
0.900
1.66
1 9
0.948
0.921
1.52
0 10
0.947
TO
0.862
1 9
0.911
0.862
1.56
0.915
0.870
1.53
4 6
0.916
0.865
1.61
ResNet-50 ArcFace
0.917
0.865
1.63
3
0.926
0.879
1.64
2
0.930
0,888
1.61
0 10
0.933
22

---

Under review as a conference paper at ICLR 2023
Table 7: Train & Test Set Img Imbalance. The female and male accuracy for models trained and tested on data with the same ratios of number of images per male and female identity. See details of experiment in Section 6
Model
Ing Ratio
M Acc
F FAcee
Error Ratio
8
0.821
0.923
0.43
3 7
0.901
0.922
0.78
G
0.928
0.919
1.12
MFN CosFace
5 5
0.942
0.906
1.62
4
0.952
2.24
7: 3
0.951
0.848
3.12
8: 2
0.954
0.740
5.70
0.46
3 7
0.916
0.933
0.79
MFN ArcFace
5 5
0.951
0.919
1.64
6 4
0.955
0.907
2.09
7: 3
0.957
0.871
3.01
8 2
0.958
0.779
5.31
3 7
0,832
0.873
0.76
4 G
0.879
0.866
1.11
ResNet-152 CosFace
5 5
0.912
0.848
1.74
G 1
0,916
0.798
2.41
8: 2
0,922
0.483
6.66
3: 2: 8
0.638
0.840
0.44
0.817
0.859
0.77
4 6
0,870
0.855
1.12
ResNet-152 ArcFace
6
0,911
0.792
2.34
7
0,917
3.51
8: 2
0,915
0.488
6.02
2 8
0,611
0.839
0.41
6
0,854
0.844
1.07
ResNet-50 CosPace
5
0,889
1.63
6 4
0.899
0.766
2.31
7:3
0.911
0.688
3.51
8: 2
0,908
0.449
6.00
2: 8
0.579
0.817
0.43
0.786
0.837
0.76
6
0,844
0.824
1.13
ResNei- 50 ArcFace
5
0.887
0.817
1.62
0.895
0.763
2.27
7: 3
0.905
0.666
3.50
23


Reviews:
[{"review_id": "bfS7mi60hI", "reviewer": "Reviewer_vAAf", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper aims to explore the problem of data set imbalance in face identification systems. They specifically focus on imbalance with respect to gender presentation. Aiming at the imbalances(both in terms of identities or images per identity) in the train set and the test set, the author conducted several experiments on the Celeb A dataset and got some conclusions.", "summary_of_the_review": "The authors have carried out various experiments, but the conclusion is relatively simple. The authors reveal many common errors in bias calculation in face recognition tasks, but do not provide effective strategies to avoid these problems", "strength_and_weaknesses": "Strength:\n1. The problem of data imbalance is a good direction and deserves further study.\n2. The authors have carried out experiments on the problem of data imbalance in different situations and draw conclusions according to the experimental results.\n\nWeaknesses:\n1. The author carried out experiments (session 4 and session 5) respectively under the condition that the training set and the test set are not balanced, but the conclusion obtained is relatively shallow. At the same time, in the case of imbalanced training set images, the abnormal occurrence is caused by resampling, that is, the bias is partly caused by human factors and the result is not rigorous enough.\n2. The authors conclude in session 7.1 that females are generally more difficult to identify than males, but the authors only conducted experiments on the Celeb A dataset.  There are differences in the difficulty of different data sets.  If the experiment can be carried out in more datasets, the conclusion will be more convincing.\n3. In session 7.2, the author argues that there are reasons for bias in humans and machines are different. But the author does not analyze what reason is and why they are different. I think this conclusion is meaningless.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "KoSdK3QG1Nv", "reviewer": "Reviewer_qe6A", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper can be categorized into an \"analysis type\" paper, which doesn't propose anything new; instead, it demonstrates that standard image classification modeles, such as RestNet, MobileNet etc., can lead to biased predictions on face recognition tasks and datasets.\n", "summary_of_the_review": "The paper can be categorized into an \"analysis type\" paper, which doesn't propose anything new; instead, it demonstrates that standard image classification modeles, such as RestNet, MobileNet etc., can lead to biased predictions on face recognition tasks and datasets.\n\nThe paper is poorly written. It's not even clear after reading Section 3 (Face Identification Setup) that what the task actually is. Is the model trained to predict a particular class (ot of say p ones), i.e., $\\theta: \\vec{x} \\mapsto \\mathbb{Z}_p$ or is the task that of metric learning, i.e., $\\theta: \\vec{x} \\times \\vec{z} \\mapsto \\mathbb{R}$?\n\nIn addition to gender, there're plenty of other attributes in the CelebA dataset that can be considered for stereotypical biases (checking for parity in the posteriors), e.g., \"attractiveness\" is one such feature, and there could be easily be biases discovered by a combination of more than one attribute, e.g., \"females wearing glasses are less attractive\" etc.\n\nThe paper reports the biases but doesn't propose a method to alleviate them, which in my opinion, severly limits its contributions in terms of novelty.\n\nThe paper lacks any kind of analysis. What's presented as actionable insights, such as \"overrepresenting the target demographic group can sometimes hurt that group...\", an academic paper should provide insights into why is that the case? What's the difference between the performance from different models? Is MobileNet more biased than ResNet? If so, why, if not then why not?", "strength_and_weaknesses": "Strengths:\n\n1. Fair number of experiments done on a standard dataset.\n2. Multiple models investigated.\n3. Extensive analysis conducted\n\nWeaknesses:\n1. Limited novelty in the sense that this is purely an analysis paper, and that too this type of analysis has been done before if not on the same dataset.\n2. Paper not well-written, e.g., even the task isn't clearly described (see the summary).\n3. No insights is provided into what exactly is the (likely) reason for the observations.\n4. No attempt to alleviate the bias is made.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "CE-IPlyjr1", "reviewer": "Reviewer_AVTo", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper unravelled the complex effects that dataset imbalance can have on the model bias for face identification systems. \n\nThis paper separately considered imbalance in terms of identities and images per identity in both the train set and the test set.\n\nThis paper thoroughly explored the effects of each kind of imbalance possible in face identification, and discuss factors which may impact bias.", "summary_of_the_review": "Bias in face recognition is a timely topic. This paper employed data sampling techniques on different dimensions, such as training data, test data, identity and image. Extensive results supported some conclusions on the bias problem of face recognition. However, the technical contribution of this paper is limited and some observations are not in-depth.", "strength_and_weaknesses": "Strength:\n\nBias in face recognition is a timely topic.\n\nThis paper provided comprehensive statistics regarding bias in the training set and test set.\n\nWeakness:\n\nIn Section 7.1, 'We observe that both models have higher male performance when the test set is perfectly balanced'. This observation is not explored in depth. Is it because female images usually contain make-up?\n\nIn Fig 6, there is no black line in the figure. The legends (identity, image)  are confusing. It is better to show male identity, male image and so on.\n\nThere is no reference to MobileFaceNet, CosFace and ArcFace.\n\nThe technical contribution of this paper is limited as only sampling strategy is used with some simple statistics on the results.\n\nThere are some useful conclusions in Section 8, but it is not very insightful for training unbiased face recognition models", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject