Title: A Deep Dive into Dataset Imbalance and Bias in Face Identification
Authors: Valeriia Cherepanova
Steven Reich
Samuel Dooley
Hossein Souri
John P Dickerson
Micah Goldblum
Tom Goldstein
Venue: ICLR 2023
Keywords: face recognition
dataset imbalance
bias
fairness

Abstract: As the deployment of automated face recognition (FR) systems proliferates, bias in these systems is not just an academic question, but a matter of public concern. Media portrayals often center imbalance as the main source of bias, i.e., that FR models perform worse on images of non-white people or women because these demographic groups are underrepresented in training data. Recent academic research paints a more nuanced picture of this relationship. However, previous studies of data imbalance in FR have focused exclusively on the face verification setting, while the face identification setting has been largely ignored, despite being deployed in sensitive applications such as law enforcement. This is an unfortunate omission, as 'imbalance' is a more complex matter in identification; imbalance may arise in not only the training data, but also the testing data, and furthermore may affect the proportion of identities belonging to each demographic group or the number of images belonging to each identity. In this work, we address this gap in the research by thoroughly exploring the effects of each kind of imbalance possible in face identification, and discuss other factors which may impact bias in this setting.

TL;DR: In this work, we explore the effects of different kinds of data imbalance on bias in face identification problem. 

Review discussion:
Strength:
1. The problem of data imbalance is a good direction and deserves further study.
2. The authors have carried out experiments on the problem of data imbalance in different situations and draw conclusions according to the experimental results.
Weaknesses:
1. The author carried out experiments (session 4 and session 5) respectively under the condition that the training set and the test set are not balanced, but the conclusion obtained is relatively shallow.
At the same time, in the case of imbalanced training set images, the abnormal occurrence is caused by resampling, that is, the bias is partly caused by human factors and the result is not rigorous enough.
2. The authors conclude in session 7.1 that females are generally more difficult to identify than males, but the authors only conducted experiments on the Celeb A dataset.  There are differences in the difficulty of different data sets.  If the experiment can be carried out in more datasets, the conclusion will be more convincing.
3. In session 7.2, the author argues that there are reasons for bias in humans and machines are different.
But the author does not analyze what reason is and why they are different. I think this conclusion is meaningless.
Clarity: This paper is clear and easy to understand
Quality: This paper is of average quality and has some loopholes
Novelty: The direction of this paper is innovative
Reproducibility: This paper is reproducible
NO.
Thank you for your thoughtful review and feedback, we address each of your comments below:
1. In our work, we study how different kinds of imbalance, such as imbalance with respect to the number of images and identities, in the train and in the gallery set affect bias in face identification systems.
While train data imbalance is a relatively explored topic, gallery set imbalance has not been considered prior to our work to the best of our knowledge. Our findings show that each type of imbalance has a distinct effect on a model’s performance on each gender presentation. One of our observations is that overrepresenting the target demographic group in the train data can hurt that demographic group when an oversampling strategy is used to “balance” the skewed distribution of data. We additionally conduct experiments without oversampling strategy and show the results in Figure 8 in Appendix.
2. In Section 7.1 we test randomly initialized feature extractors on galleries with varying levels of image imbalance. We find that random models have higher male performance even when the gallery set is perfectly balanced, which suggests that there are sources of bias in face identification systems which lie outside of the data imbalance aspect.
However we do not conclude that female images are generally more difficult to recognize than male images.
3. In Section 7.2 we compare human and model bias on InterRace dataset. We find that although both humans and models find identifying females more difficult than identifying males, particular images that are easier or harder to identify appear to differ between models and humans. We believe that this is an interesting observation showing that gender bias in face identification systems and humans come from different sources.
Strengths:
1. Fair number of experiments done on a standard dataset.
2. Multiple models investigated.
3. Extensive analysis conducted
Weaknesses:
1. Limited novelty in the sense that this is purely an analysis paper, and that too this type of analysis has been done before if not on the same dataset.
2. Paper not well-written, e.g., even the task isn't clearly described (see the summary).
3. No insights is provided into what exactly is the (likely) reason for the observations.
4. No attempt to alleviate the bias is made.
The novelty is limited (see the summary).
Clarity is not good in terms of precisely defining the task and the protected attributes.
NO.
Thank you for your thoughtful review and feedback, we address your comments below:
1. To the best of our knowledge, our paper is the first work exploring how different kinds of data imbalance, including imbalance with respect to the number of images and identities, in the train and gallery data affects bias of face identification systems. We would appreciate it if you could point us to previous works doing similar analysis. We would be happy to compare contributions and cite these related works.
2.
Regarding the task, we conduct experiments in a face identification setup, where the goal is to match a probe image against a set of images (called the gallery) with known identities. Face identification models are trained as image classifiers, where classes correspond to identities. At inference time, they identify the closest gallery image to a probe image using distance measures in the feature space.
3.
Regarding additional attributes and stereotypical biases, although CelebA indeed contains multiple attribute annotations, such as “attractiveness”, which could be used for exploring stereotypical biases, our work focuses on different aspects of model bias, such as the effect of different kinds of data imbalance on bias of face identification systems.
4. *No insights is provided into what exactly is the (likely) reason for the observations.*
Although we do not elaborate on the reasons for certain trends in the concluding section, we discuss them in detail in the corresponding Sections 4,5,6,7.
In particular, we discuss how different kinds of data imbalance in the train or gallery sets affect bias in face identification models and provide evidence through experiments. In the “Actionable Insights” section we summarize our findings for practitioners.
Strength:
Bias in face recognition is a timely topic.
This paper provided comprehensive statistics regarding bias in the training set and test set.
Weakness:
In Section 7.1, 'We observe that both models have higher male performance when the test set is perfectly balanced'. This observation is not explored in depth. Is it because female images usually contain make-up?
In Fig 6, there is no black line in the figure. The legends (identity, image)  are confusing. It is better to show male identity, male image and so on.
There is no reference to MobileFaceNet, CosFace and ArcFace.
The technical contribution of this paper is limited as only sampling strategy is used with some simple statistics on the results.
There are some useful conclusions in Section 8, but it is not very insightful for training unbiased face recognition models
The writing is clear and straightforward.
But the technical contribution of this paper is limited. This paper is easy to reproduce the statistics reported in the paper.
NO.
No ethics concerns.
Thank you for your thoughtful review and feedback, we address your comments below:
1. It is a very interesting question why randomly initialized models have higher male performance when the test set is perfectly balanced. One reason could indeed be a larger variance in the input (image) space for female images, for example because of make-up or more variability in the hairstyle.
However, it is not entirely clear how to measure the variance in the input space. We did experiments with measuring the variance in the feature space of a neural network and found that female images have higher within-class variance, than male images.
However, we note that this could be a result of higher variance in the input space OR algorithmic bias of a neural network itself.
2. *In Fig 6, there is no black line in the figure.*
We include the black solid and dashed lines in the legend of Fig 6 to distinguish between the balance with respect to the number of identities (solid lines) and balance with respect to the number of images (dashed line).
3. *There is no reference to MobileFaceNet, CosFace and ArcFace.*
Thank you for pointing this out, we have now updated our draft to include the references.
4.
Regarding our contribution, our paper provides insights into how different kinds of data imbalance affect bias of face identification systems and how train and gallery set imbalances may interact with each other and lead to underestimation of true bias. We hope that our findings will help practitioners avoid common mistakes in bias computations for many facial recognition tasks. We additionally analyze whether data imbalance captures all the inherent bias and how the bias we see in face identification models compares to human biases.

Meta-review:
role: PC
data: metareview:_summary,_strengths_and_weaknesses: The paper studies face identification and demonstrates how various types of data imbalance can lead to model bias. AC agrees with the reviewers that albeit it’s an important problem, limited technical contribution (how to resolve existing model failures) and a narrow application domain (the paper studies face identification) are two critical issues that place the contributions below acceptance bar. We hope the reviews are useful to improve the manuscript. 
justification_for_why_not_higher_score: N/A
justification_for_why_not_lower_score: N/A

Decision: Reject