Year: 2021

Abstract:
Despite being widely used, face recognition models suffer from bias: the probability of a false positive (incorrect face match) strongly depends on sensitive attributes such as the ethnicity of the face. As a result, these models can disproportionately and negatively impact minority groups, particularly when used by law enforcement. The majority of bias reduction methods have several drawbacks: they use an end-toend retraining approach, may not be feasible due to privacy issues, and often reduce accuracy. An alternative approach is post-processing methods that build fairer decision classifiers using the features of pre-trained models, thus avoiding the cost of retraining. However, they still have drawbacks: they reduce accuracy (AGENDA, PASS, FTC), or require retuning for different false positive rates (FSN). In this work, we introduce the Fairness Calibration (FairCal) method, a post-training approach that simultaneously: (i) increases model accuracy (improving the stateof-the-art), (ii) produces fairly-calibrated probabilities, (iii) significantly reduces the gap in the false positive rates, (iv) does not require knowledge of the sensitive attribute, and (v) does not require retraining, training an additional model, or retuning. We apply it to the task of Face Verification, and obtain state-of-the-art results with all the above advantages.

heading: 1 INTRODUCTION
text: Face recognition (FR) systems are being increasingly deployed worldwide in a variety of contexts, from policing and border control to providing security for everyday consumer electronics. According to Garvie et al. (2016), face images of around half of all American adults are searchable in police databases. FR systems have achieved impressive results in maximizing overall accuracy (Jain et al., 2016). However, they have also been shown to exhibit significant bias against certain demographic subgroups (Buolamwini & Gebru, 2018; Orcutt, 2016; Alvi et al., 2018), defined by a sensitive attribute such as ethnicity, gender, or age. Many FR systems have much higher false positive rates (FPRs) for non-white faces than white faces (Grother et al., 2019). Therefore, when FR is employed by law enforcement, non-white individuals may be more likely to be falsely detained (Allyn, 2020). Thus, it is of utmost importance to devise an easily-implementable solution to mitigate FR bias.
Most efforts to mitigate FR bias have been directed towards learning less biased representations (Liang et al., 2019; Wang et al., 2019b; Kortylewski et al., 2019; Yin et al., 2019; Gong et al., 2020; Wang & Deng, 2020; Huang et al., 2020; Gong et al., 2021). These approaches have enjoyed varying degrees of success: though the bias is reduced, is reduced (Gong et al., 2020). They often require retraining the models from scratch, which is computationally expensive. Moreover, they often require the sensitive attribute of the face (such as ethnicity) (Gong et al., 2020; Dhar et al., 2020; Terhörst et al., 2020a; Dhar et al., 2021; Robinson et al., 2020), which may not be feasible to obtain. Hence, an approach that improves fairness in existing models without reducing accuracy, or knowing the sensitive attribute, or require retraining, is missing.
In this work, we focus on the face verification problem, a major subproblem in FR: determine whether two face images depict the same person. Current state-of-the-art (SOTA) classifiers are based on deep neural networks that embed images into a low-dimensional space (Taigman et al., 2014). The “Baseline” method is one that deems a pair of images is a match if the cosine similarity between their embeddings exceeds a certain threshold. We illustrate and compare the bias of current models in Figure 1, and compare their characteristics in Table 1.
The fairness of face verification classifiers is typically assessed by three quantities, explained below: (i) fairness calibration across subgroups, (ii) equal false positive rates (FPRs) across subgroups i.e. predictive equality, (iii) equal false negative rates (FNRs) across subgroups, i.e. equal opportunity.
A classifier is said to be globally calibrated if its predicted probability ppred of face matches in a dataset is equal to the fraction ptrue of true matches in the dataset (Guo et al., 2017). This requires the classifier to additionally output an estimate of the true probability of a match, i.e. confidence. However, we suggest it is even more crucial for models to be fairly calibrated, i.e., for calibration to hold conditionally on each subgroup (see Definition 2). Otherwise, if the same predicted probability is known to carry different meanings for different demographic groups, users including law enforcement officers and judges may be motivated to take sensitive attributes into account when making critical decisions about arrests or sentencing (Pleiss et al., 2017). Moreover, previous works such as Canetti et al. (2019) and Pleiss et al. (2017) presume the use of a fairly-calibrated model for their success.
While global calibration can be achieved with off-the-shelf post-hoc calibration methods (Platt, 1999; Zadrozny & Elkan, 2001; Niculescu-Mizil & Caruana, 2005; Guo et al., 2017), they do not achieve fairness-calibration. This is illustrated in Figure 2. However, achieving all three fairness definitions mentioned above: (i) fairness-calibration, (ii) equal FPRs, (iii) equal FNRs, is impossible in practice as several authors have shown (Hardt et al., 2016; Chouldechova, 2017; Kleinberg et al., 2017; Pleiss et al., 2017). We elaborate further on this in section 3, the key takeaway is that we may aim to satisfy at most two of the three fairness notions. In the particular context of policing, equal FPRs i.e. predictive equality is considered more important than equal FNRs, as false positive errors (false arrests) risk causing significant harm, especially to members of subgroups already at disproportionate risk for police scrutiny or violence. As a result, our goals are to achieve fairness-calibration and predictive equality across subgroups, while maintaining accuracy, without retraining.
We further assume that we do not have access to the sensitive attributes, as this may not be feasible in practice due to (a) privacy concerns, (b) challenges in defining the sensitive attribute (e.g. ethnicity cannot be neatly divided into discrete categories), and (c) laborious and expensive to collect.
In this paper, we succeed in achieving all of the objectives mentioned above. The main contribution is the introduction of a post-training Fairness Calibration (FairCal) method:
1. Accuracy: FairCal achieves state-of-the-art accuracy (Table 2) for face verification on two large datasets.
2. Fairness-calibration: Our face verification classifier outputs state-of-the-art fairnesscalibrated probabilities without knowledge of the sensitive attribute (Table 3).
3. Predictive equality: Our method reduces the gap in FPRs across sensitive attributes (Table 4). For example, in Figure 2, at a Global FPR of 5% using the baseline method Black people are 15X more likely to false match than white people. Our method reduces this to 1.2X (while SOTA for post-hoc methods is 1.7X).
4. No sensitive attribute required: Our approach does not require the sensitive attribute, neither at training nor at test time. In fact, it outperforms models that use this knowledge.
5. No retraining: Our method has no need for retraining or tuning the feature representation model, or training an additional model, since it performs statistical calibration post-hoc.

heading: 2 RELATED WORK
text: Fairness, or Bias mitigation: Work on bias mitigation for deep FR models can be divided into two main camps: (i) methods that learn less biased representations, and (ii) post-processing approaches that attempt to remove bias from a pre-trained feature representation model by building fairer decision systems (Srinivas et al., 2019; Dhar et al., 2020; 2021; Terhörst et al., 2020a;b; Robinson et al., 2020). Several approaches have been pursued in (i), including domain adaptation (Wang et al., 2019a), margin-based losses (Khan et al., 2019; Huang et al., 2020), data augmentation (Kortylewski et al., 2019), feature augmentation (Yin et al., 2019; Wang et al., 2019b), reinforcement learning (Wang & Deng, 2020), adversarial learning (Liu et al., 2018; Gong et al., 2020), and attention mechanisms (Gong et al., 2021). While these methods mitigate bias, this is often achieved at the expense of recognition accuracy and/or at high computational cost. The work we present in this paper fits into (ii), and thus we focus on reviewing those approaches. For an in-depth literature review of bias mitigation for FR models, see Drozdowski et al. (2020). We compare all these methods in Table 1.
Srinivas et al. (2019) proposed an ensemble approach, exploring different strategies for fusing the scores of multiple models. This work does not directly measure bias reduction, and presents only the results of applying the method to a protected subgroup. Terhörst et al. (2020b) show that while this method is effective in increasing overall accuracy, it fails to reliably mitigate bias.
Dhar et al. (2020) proposed the Adversarial Gender De-biasing algorithm (AGENDA) to train a shallow network that removes the gender information of the embeddings of a pre-trained network. Dhar et al. (2021) extended this work with PASS to deal with any sensitive attribute, and proposed a novel discriminator training strategy. This leads to reduced gender bias in face verification, but at the cost of accuracy. Also, AGENDA and PASS require the sensitive attributes during training.
Terhörst et al. (2020a) proposed the Fair Template Comparison (FTC) method, which replaces the computation of the cosine similarity score by an additional shallow neural network trained using cross-entropy loss, with a fairness penalization term and an l2 penalty term to prevent overfitting. While this method does indeed reduce a model’s bias, it comes at the expense of an overall decrease in accuracy. Moreover, it requires training the shallow neural network and tuning the loss weights. Most importantly, it requires the sensitive attributes during training.
Robinson et al. (2020) proposed GST which uses a group-specific threshold. However, the calibration sets in GST are defined by the sensitive attributes themselves. Terhörst et al. (2020b) proposed the Fair Score Normalization (FSN) method, which is essentially GST with unsupervised clusters. FSN normalizes the scores by requiring the model’s FPRs across unsupervised clusters to be the same at a predefined global FPR. These methods are not designed to be fairly-calibrated. Moreover, if a different global FPR is desired, the method needs to be recomputed. In contrast, instead of fixing one FPR threshold, our method converts the cosine similarity scores into calibrated probabilities, which can then be used for any choice of fair FPRs. In addition, our approach can be extended to group fairness for K-classification problems, which is not possible with GST or FSN.
Calibration: Calibration is closely related to uncertainty estimation for deep networks (Guo et al., 2017). Several post-hoc calibration methods have been proposed such as Platt’s scaling or temperature scaling (Platt, 1999; Guo et al., 2017), histogram binning (Zadrozny & Elkan, 2001), isotonic regression (Niculescu-Mizil & Caruana, 2005), spline calibration (Gupta et al., 2021), and beta calibration (Kull et al., 2017), among others. All of these methods involve computing a calibration function. As such, any calibration method can be readily applied, leading to models that are calibrated but not fairly-calibrated. An algorithm to achieve the latter for a binary classifier has been proposed in Hebert-Johnson et al. (2018), but the work remains theoretical and no practical implementation is known. Fairly calibrated models are the missing ingredient in the works of Canetti et al. (2019) and Pleiss et al. (2017), since they work with the presumption that a fairly calibrated model exists.

heading: 3 FACE VERIFICATION AND FAIRNESS
text: In order to discuss bias mitigation strategies and their effectiveness, one must first agree on what constitutes a fair algorithm. We start by rigorously defining the face verification problem as a binary classification problem. Then we present the notion of a probabilistic classifier, which outputs calibrated probabilities that the classification is correct. Finally, since several different definitions of fairness have been proposed (see Verma & Rubin (2018); Garg et al. (2020) for a comparative analysis), we review the ones pertinent to our work.

heading: 3.1 BINARY CLASSIFIERS AND SCORE OUTPUTS
text: Let f denote a trained neural network that encodes an image x into an embedding f(x) = z ∈ Z = Rd. Pairs of images of faces are drawn from a global pair distribution (x1,x2) ∼ P . Given such a pair, let Y (x1,x2) = 1 if the identities of the two images are the same and Y = 0 otherwise. The face verification problem consists of learning a binary classifier for Y .
The baseline classifier for the face verification problem is based on the cosine similarity between the feature embeddings of the two images, s(x1,x2) = f(x1) T f(x2)
‖f(x1)‖‖f(x2)‖ . The cosine similarity score is
used to define a binary classifier Ŷ : X ×X → {0, 1} by thresholding with a choice of sthr ∈ [−1, 1], which is determined by a target FPR. Ŷ (x1,x2) = 1 if s(x1,x2) ≥ sthr; 0 otherwise

heading: 3.2 CALIBRATION
text: A calibrated probabilistic model provides a meaningful output: its confidence reflects the probability of a genuine match, in the sense of the following definition.
Definition 1. The probabilistic model Ĉ : X × X → [0, 1] is said to be calibrated if the true probability of a match is equal to the model’s confidence output c: Px1,x2∼P(Y = 1 | Ĉ = c) = c.
Any score-based classifier can be converted to a calibrated probabilistic model using standard post-hoc calibration methods (Zadrozny & Elkan, 2001; Niculescu-Mizil & Caruana, 2005; Platt, 1999; Kull et al., 2017) (see Appendix G for a brief description of ones used in this work). This probabilistic model can be converted to a binary classifier by thresholding the output probability.
Applied to face verification, a calibrated probabilistic model outputs the likelihood/confidence that a pair of images (x1,x2) are a match. Calibration is achieved by using a calibration set Scal to learn a calibration map µ from the cosine similarity scores to probabilities. Since calibration defines a monotonically increasing calibration map µ, a binary classifier with a score threshold sthr can be obtained by thresholding the probabilistic classifier at µ(sthr).

heading: 3.3 FAIRNESS
text: In the context of face verification, fairness implies that the classifier has similar performance on different population subgroups, i.e. being calibrated conditional on subgroup membership. Let g(x) ∈ G = {gi} denote the sensitive/protected attribute of x, such as ethnicity, gender, or age. Each sensitive attribute gi ∈ G induces a population subgroup, whose distribution on the intra-subgroup pairs we denote by Gi. Definition 2. The probabilistic model Ĉ is fairly-calibrated1 for subgroups g1 and g2 if the classifier is calibrated when conditioned on each subgroup.
Px1,x2∼G1(Y = 1 | Ĉ = c) = Px1,x2∼G2(Y = 1 | Ĉ = c) = c.
Intuitively, a model is considered biased if its accuracy alters when tested on different subgroups. However, in the case of face recognition applications, predictive equality (defined below), meaning equal FPRs across subgroups, is often of primary importance.
Definition 3. A binary classifier Ŷ exhibits predictive equality for subgroups g1 and g2 if the classifier has equal FPRs for each subgroup,
P(x1,x2)∼G1(Ŷ = 1 | Y = 0) = P(x1,x2)∼G2(Ŷ = 1 | Y = 0).
In certain applications of FR (such as office security, where high FNRs would cause disruption), equal opportunity (defined below), meaning equal FNRs across subgroups, is also important.
Definition 4. A binary classifier Ŷ exhibits equal opportunity for subgroups g1 and g2 if the classifier has equal FNRs for each subgroup,
P(x1,x2)∼G1(Ŷ = 0 | Y = 1) = P(x1,x2)∼G2(Ŷ = 0 | Y = 1).
Prior works (Hardt et al., 2016; Chouldechova, 2017; Kleinberg et al., 2017; Pleiss et al., 2017) have shown that it is impossible in practice to satisfy all three fairness properties at the same time: (i) fairness calibration (definition 2), (ii) predictive equality (definition 3), (iii) equal opportunity (definition 4). At most, two of these three desirable properties can be achieved simultaneously in practice. In the context of our application, predictive equality is more critical than equal opportunity. Hence we choose to omit equal opportunity as our goal.

heading: 4 FAIRNESS CALIBRATION (FAIRCAL)
text: In this section we describe our FairCal method, which constitutes the main contribution of this work. Simply calibrating a model based on cosine similarity scores will not improve fairness: if the baseline score-based classifier is biased, the resulting probabilistic classifier remains equally biased. This is illustrated in Figure 2, where for both Baseline and Baseline Calibration, any choice of global threshold will lead to different FPRs across sensitive subgroups; consequently, these models will fail to achieve predictive equality.
Our proposed solution is to introduce conditional calibration methods, which involve partitioning the pairs into sets, and calibrating each set. Given an image pair, we first identify its set membership, and then apply the corresponding calibration map. In our FairCal method, the sets are formed by clustering the images’ feature vectors in an unsupervised fashion. In our Oracle method, the sets are defined by the sensitive attributes themselves. Both methods are designed to achieve fairness-calibration.
1This notion is also referred to as well-calibration (Verma & Rubin, 2018) and calibration within subgroups (Kleinberg et al., 2017; Berk et al., 2021)

heading: 4.1 FAIRNESS CALIBRATION (FAIRCAL, OUR METHOD)
text: (i) Apply the K-means algorithm to the image features, Zcal, partitioning the embedding space Z into K clusters Z1, . . . ,Zk. These form the K calibration sets:
Scalk = {s(x1,x2) : f(x1) ∈ Zk or f(x2) ∈ Zk} , k = 1, . . . ,K (1)
(ii) For each calibration set Scalk , use a post-hoc calibration method to compute the calibration map µk that maps scores s(x1,x2) to cluster-conditional probabilities µk(s(x1,x2)). We use beta calibration (Kull et al., 2017), the recommended method when dealing with bounded scores.
(iii) For an image pair (x1, x2), find the clusters they belong to. If both images fall into the same cluster k, define the pair’s calibrated score as the cluster’s calibrated score:
c(x1,x2) = µk(s(x1,x2)) (2)
Else, if the images are in different clusters, k1 and k2, respectively, define the pair’s calibrated score as the weighted average of the calibrated scores in each cluster:
c(x1,x2) = θ µk1(s(x1,x2)) + (1− θ) µk2(s(x1,x2)), (3)
where the weight θ is the relative population fraction of the two clusters, θ = ∣∣Scalk1 ∣∣/(∣∣Scalk1 ∣∣+ ∣∣Scalk2 ∣∣) (4)
Since FairCal uses unsupervised clusters, it does not require knowledge of the sensitive attributes. Moreover, FairCal is a post-training statistical method, hence it does not require any retraining.

heading: 4.2 COMPARISON WITH FSN (TERHÖRST ET AL., 2020B)
text: Building the unsupervised clusters allows us to not rely on knowing the sensitive attributes. In contrast, the FSN method normalizes the scores by shifting them such that thresholding at a predefined global FPR leads to that same FPR on each calibration set. This is limiting in two crucial ways: 1) FSN only applies to the binary classification problem (e.g. face verification); 2) the normalizing shift in FSN depends on the global FPR chosen a priori. Our FairCal method has neither of these limitations. By converting the scores to calibrated probabilities, we can extended it to the multi-class setting, and we do not need to choose a global FPR a priori.
A simple analogy to explain the difference is : consider the problem of fairly assessing two classrooms of students with different distribution of grades. Fair calibration means the same grade should mean the same for both classrooms. Equal fail rate implies the percentage of students who fail is the same in both classrooms. Global (unfair) methods would choose to pass or fail students from both classrooms using the same threshold. Two possible fair approaches are: (A) Estimate a different threshold for each classroom based on a fixed fail rate for both. (B) Calibrate the grades so the distributions of the two classrooms match, i.e. the same grade means the same in both classrooms, and then choose a fair threshold. Method B automatically ensures equal fail rate for both classes. Method A is what FSN does, Method B is what our FairCal method achieves.

heading: 4.3 ORACLE CALIBRATION (SUPERVISED FAIRCAL, ALSO OURS)
text: We include a second calibration method, Oracle, which proceeds like the FairCal defined above, but creates the calibration sets based on the sensitive attribute instead in an unsupervised fashion. If the images belong to different subgroups i.e. g(x1) 6= g(x2), then the classifier correctly outputs zero. This method is not feasible in practice, since the sensitive attribute may not be available, or because using the sensitive attribute may be not permitted for reasons of discrimination or privacy. However, the Oracle method represents an ideal baseline for fairness-calibration.

heading: 5 EXPERIMENTAL DETAILS
text: Models: We used three distinct pretrained models: two Inception Resnet models (Szegedy et al., 2017) obtained from Esler (2021) (MIT License), one trained on the VGGFace2 dataset (Cao et al., 2018) and another on the CASIA-Webface dataset (Yi et al., 2014), and an ArcFace model obtained
from Sharma (2021) (Apache 2.0 license) and trained on the refined version of MS-Celeb-1M (Guo et al., 2016). We will refer to the models as Facenet (VGGFace2), Facenet (Webface), and ArcFace, respectively. As is standard for face recognition models, we pre-processed the images by cropping them using a Multi-Task Convolution Neural Network (MTCNN) algorithm (Zhang et al., 2016). If the algorithm failed to identify a face, the pair was removed from the analysis.
Datasets: We present experiments on two different datasets: Racial Faces in the Wild (RFW) (Wang et al., 2019a) and Balanced Faces in the Wild (BFW) (Robinson et al., 2020), both of which are available under licenses for non-commercial research purposes only. Both datasets already include predefined pairs separated into five folds. The results we present are the product of leave-one-out cross-validation. The RFW dataset contains a 1:1 ratio of genuine/imposter pairs and 23,541 pairs in total (after applying the MTCNN). The dataset’s images are labeled by ethnicity (African, Asian, Caucasian, or Indian), with all pairs consisting of same-ethnicity images. The BFW dataset, which possesses a 1:3 ratio of genuine/imposter pairs, is comprised of 890,347 pairs (after applying the MTCNN). Its images are labeled by ethnicity (African, Asian, Caucasian, or Indian) and gender (Female or Male), and it includes mixed-gender and mixed-ethnicity pairs. The RFW and BFW datasets are made up of images taken from MS-Celeb-1M (Guo et al., 2016) and VGGFace (Cao et al., 2018), respectively. Thus, to expose the models to new images, only the two FaceNet models can be evaluated on the RFW dataset, while only the FaceNet (Webface) and ArcFace models can be evaluated on the BFW dataset.
Methods: For both the FSN and FairCal method we used K = 100 clusters for the K-means algorithm, as recommended by Terhörst et al. (2020b). For FairCal, we employed the recently proposed beta calibration method (Kull et al., 2017). Our FairCal method is robust to the choice of number of clusters and post-hoc calibration method (see Appendix H for more details). We discuss the parameters used in training AGENDA (Dhar et al., 2020), PASS (Dhar et al., 2021) and FTC (Terhörst et al., 2020a) in Appendix B, in Appendix C and Appendix D, respectively.
Notice that the prior relevant methods (Baseline, AGENDA, PASS, FTC, GST, and FSN) output scores that, even when rescaled to [0,1], do not result in calibrated probabilities and hence, by design, cannot be fairly-calibrated. Therefore, in order to fully demonstrate that our method is superior to those approaches, when measuring fairness-calibration we apply beta calibration to their final score outputs as well, which is the same post-hoc calibration method used in our FairCal method.

heading: 6 RESULTS
text: In this section we report the performance of our models with respect to the three metrics: (i) accuracy, (ii) fairness calibration, and (iii) predictive equality. Our results show that among post hoc calibration methods,
1. FairCal is best at global accuracy, see Table 2 2. FairCal is best on fairness calibration, see Table 3. 3. FairCal is best on predictive equality, i.e., equal FPRs, see Table 4. 4. FairCal does not require the sensitive attribute, and outperforms methods that use this
knowledge, including a variant of FairCal that uses the sensitive attribute (Oracle). 5. FairCal does not require retraining of the classifier.
We discuss these results in further detail below. We provide additional detailed discussion and results, including on equal opportunity and the standard deviations that result from the 5-fold cross-validation study, in the Appendix. Overall, our method that satisfies both fairness definitions without decreasing baseline model accuracy. In contrast, while the FTC method obtains slightly better predictive equality results than our method in one situation, this is achieved only at the expense of a significant decrease in accuracy.
It is important to emphasize that accuracy and predictive equality metrics are determined at a specified global FPR. This entails determining for each prior method a threshold that achieves the desired global FPR. However, this itself promotes our method’s advantages: (a) previous methods such as FSN rely on a predefined FPR, while ours does not; (b) while other methods need recomputation for different thresholds, our method simultaneously removes the bias at multiple global FPRs.
6.1 GLOBAL ACCURACY
The receiver operating characteristic (ROC) curve plots the true positive rate (TPR = 1−FNR) against the FPR, obtained by thresholding the model at different values. The area under the ROC curve (AUROC) is thus a holistic metric that summarizes the accuracy of the classifiers (Davis & Goadrich, 2006). A higher AUROC is better, an uninformative classifier has an AUROC of 50%.
Our FairCal method achieves SOTA results, as shown in Table 2. We report the AUROC for the different pre-trained models and datasets, as well as the TPRs at 0.1% and 1% global FPR thresholds. FairCal achieves the best values of AUROC in all cases, the highest TPR in seven of the eight cases, and surpasses GST and our Oracle method (which use subgroup information) on the RFW dataset.
This overall accuracy improvement of our FairCal method can be explained as follows: the feature vectors contain more information than the score alone, so they can be used to identify pairs where the probability of an error is higher or lower, allowing per-cluster calibration to give better accuracy.
In our FairCal method, the subgroups are formed based on the features of the model, and not on manual human-assigned attributes. This allows for potentially more complex and effective subgroups. We visualized the cluster images, and found they indeed have semantic meaning, see Appendix A.
6.2 FAIRNESS CALIBRATION
The prior relevant methods (Baseline, AGENDA, PASS, FTC, and FSN) do not output probabilities, hence by design they cannot be fairly-calibrated. However, in order to fully demonstrate that our method is superior to those approaches, we apply beta calibration (the same post-hoc calibration method used in our FairCal method) to their score outputs.
Calibration error is a measure of the deviation in a model’s output probabilities of matches from the actual results. When considering subgroups, this is measured by two quantities: (i) the calibration error on each subgroup (lower is better, since it means fewer errors), which can be measured using the Brier score (BS) (DeGroot & Fienberg, 1983), Expected Calibration Error (ECE) (Guo et al., 2017), or Kolmogorov-Smirnov (KS) (Gupta et al., 2021) (discussed in Appendix E). We present results for KS, which has been established to be the best measure (Gupta et al., 2021). (ii) The deviation in these quantities across subgroups (lower is better, since it is more fair), measured as: Average Absolute Deviation (AAD), Maximum Absolute Deviation (MAD), and Standard Deviation (STD).
Our FairCal method achieves the best results, as shown in Table 3: the best mean in three of the four cases (smaller errors) and the best deviation in nine of the nine cases (more fair across subgroups). The improvement is most significant on the less accurate models. We discuss these results in more detail in the Appendix.
6.3 PREDICTIVE EQUALITY
As predictive equality is achieved through equal FPRs between different subgroups, we can measure bias by quantifying the deviation of these FPRs. We report three measures of deviation (AAD, MAD, and STD) at two different global FPRs: 0.1% and at 1.0%.
Our FairCal method achieves the best results, as shown in Table 4: the best or tied measure of predictive equality in 18 of the 24 cases. In the cases where FairCal is not best, i.e., when the ArcFace model evaluated on the BFW dataset, FTC and GST provide the best results, but the differences between AGENDA’s, PASS’s, FTC’s, GST’s, FSN’s, and FairCal’s deviation measures are within a fraction of 1%. Moreover, when applied to ArcFace, the FTC and GST methods reduce the model’s accuracy.

heading: 7 CONCLUSION
text: We introduce FairCal, a post-training calibration method that (i) achieves SOTA accuracy, (ii) is fairly-calibrated, (iii) achieves predictive equality (equal FPRs) on challenging face verification datasets, (iv) without the knowledge of any sensitive attribute. It can be readily applied to existing FR systems, as it is a statistical method that (v) does not require any retraining. It can aid humanusers in using FR systems to make better informed decisions based on interpretable, unbiased results.
Societal Impact: Bolstered by dramatically improving accuracy, Facial Recognition (FR) systems have exploded in popularity in recent years and have been applied to innumerable settings under various use cases. However, the severe biases within these systems limit their applicability and open the door to widespread social harm. In this work, we address this issue by proposing a novel postprocessing method that significantly reduces the false positive rates across demographic subgroups without requiring knowledge of any sensitive attributes.
Limitations: Only under very strict conditions (e.g. a perfect classifier) can all three fairness notions (fairness calibration, predictive equality, equal opportunity) be satisfied. While we have shown that our FairCal method can achieve two out of the three definitions—which is the best we can hope for in practice—it requires the use of a calibration dataset. Deploying our method on a substantially different dataset would most likely require further calibration. Calibrating models for never-seen data is currently an open problem.
ETHICS STATEMENT
Our paper is primarily focused on addressing the lack of fairness as well as potential unethical uses of current face recognition systems, specifically in the context of face verification. At many places in the paper, we have referenced several prior works that mention and analyze the current and potential risks posed by such systems. We identified the shortcomings in the previous methods, and methodically established a scientific way of identifying and measuring fairness in the current systems. Our experiments involve large public datasets of faces. Our work explicitly addresses problems with face recognition systems that could misuse the sensitive attributes in these datasets, such as ethnicity, race, gender, etc. The motivation of our method is in trying to mitigate bias and improve fairness in the systems using these datasets.
REPRODUCIBILITY STATEMENT
In order to ensure the reproducibility of our work, we described our proposed methods, FairCal and Oracle, in detail in section 4. The details of our experiments, including the datasets and pre-processing steps, can be found in section 5. Additional details on how we implemented AGENDA and FTC, which require training additional models, are provided in Appendix B and Appendix D, respectively. The embeddings from the pretrained models were obtained on a machine with one GeForce GTX 1080 Ti GPU. All methods were implemented in Python, and the code is provided in the supplemental material.

heading: A EXAMPLES OF THE UNSUPERVISED CLUSTERS
text: In order to not rely on the sensitive attribute like the Oracle method, our FairCal method uses unsupervised clusters computed with the K-means algorithm based on the feature embeddings of the images. We found them to have semantic meaning. Some examples are included in Figure 3.

heading: B AGENDA METHOD
text: The Adversarial Gender De-biasing algorithm (AGENDA) learns a shallow network that removes the gender information of the embeddings from a pre-trained network producing new embeddings. The algorithm entails training a generator model M , a classifier C and a discriminator E. As proposed, the algorithm only removes the gender information using an adversarial loss that encourages the discriminator to produce equal probabilities for both male and female gender. Since the RFW dataset
contains the ethnicity attribute (as opposed to gender) and the BFW dataset contains both gender and ethnicity, we modify the loss to encourage the discriminator to produce equal probabilities amongst all possible sensitive attributes.
The shallow networks used for M , C and E are the same as the ones specified in Dhar et al. (2020), with the exception that E has as many outputs as possible values of sensitive attributes (4 for RFW and 8 for BFW).
AGENDA requires the use of a validation set to determine if the discriminator should continue to be updated or not. Hence the embeddings used for training into a 80/20 split, with the 20 split used for the validation.
In Stage 1, the generator M and classifier C are trained for 50 epochs. Then the bulk of training consists of 100 episodes: (i) Stage 2 is repeated every 10 episodes and consists in training the discriminator E for 25 epochs; (ii) in Stage 3 both M and C are trained with the adversarial loss for 5 epochs with λ = 10; (iii) in Stage 4 the discriminator is updated for 5 epochs, unless its accuracy on the validations set is higher than 90%. All training is done with a batch size of 400 and an ADAM optimizer with a learning rate of 10−3. For more details, we refer to the code provided in the supplemental material.

heading: C PASS METHOD
text: Similar to AGENDA, PASS (Dhar et al., 2021) learns a shallow network to removes the sensitive information of the embeddings from a pre-trained network producing new embeddings. The algorithm entails training a generator model M , a classifier C and ensemble of discriminator E.
The shallow networks used for M , C and E are the same as the ones specified in Dhar et al. (2021). Like AGENDA, it requires the use of a validation set. The same partition as in AGENDA was used.
In Stage 1, the generator M and classifier C are trained for 100 epochs. Then the bulk of training consists of 100 episodes: (i) Stage 2 is repeated every 10 episodes and consists in training the discriminator E for 100 epochs; (ii) in Stage 3 both M and C are trained with the adversarial loss for 5 epochs with λ = 10; (iii) in Stage 4 we use the One-At-a-time strategy to update one of the discriminators in the ensemble E. It is updated for 5 epochs, unless its accuracy on the validations set is higher than 95%. All training is done with a batch size of 400 and an ADAM optimizer with a learning rate of 10−3.
The availability of sensitive attributes determined the choice of number of discriminators used. For the RFW dataset, since only the race is know, we use an ensemble with K = 2 discriminators. For the BFW, since both race and gender are known, we use an ensemble of K = 2 discriminators for the race sensitive attribute and K = 3 discriminators for the gender sensitive attribute. The latter in fact constitutes the MultiPASS variant of PASS. We choose to name it here PASS for simplicity. For more details, we refer to the code provided in the supplemental material.

heading: D FAIR TEMPLATE COMPARISON (FTC) METHOD
text: The Fair Template Comparison (FTC) method Terhörst et al. (2020a) learns a shallow network with the goal of outputting fairer decisions. We implemented the FTC method as follow. In order to keep the ratios between the dimensions of layers the same as in the original paper Terhörst et al. (2020a), we used a 512-dimensional input layer, followed by two 2048-dimensional intermediate layers. The final layer is a fully connected linear layer with 2-dimensional output with a softmax activation. All intermediate layers are followed by a ReLU activation function and dropout (with p = 0.3). The network was trained with a batchsize of b = 200 over 50 epochs, using an Adam optimizer with a learning rate of 10−3 and weight decay of 10−4. Two losses, one based on subgroup fairness and the other on both subgroup and individual fairness, were proposed in Terhörst et al. (2020a). Based on the paper’s recommendations, we used the individual fairness loss with a trade-off parameter of λ = 0.5.

heading: E MEASURING CALIBRATION ERROR
text: There are different metrics available to measure if a probabilistic classifier is calibrated or fairlycalibrated. Calibration error is the error between the true and estimated confidences and is typically measured by the Expected Calibration Error (ECE) Guo et al. (2017):
Despite being the most popular calibration error metric, the ECE has several weaknesses, chief among which is its dependence on the binning scheme Nixon et al. (2019). Recently, Gupta et al. (2021) introduced a simple, bin-free calibration measure. For calibrated scores P (Y = 1|C = c) = c we have, by Bayes’ rule:
P (Y = 1, C = c) = cP (C = c).
Inspired by the Kolmogorov-Smirnov (KS↓ ) statistic test, Gupta et al. (2021) proposed to measure the calibration error by comparing the cumulative distributions of P (Y = 1, C = c) and cP (C = c), which empirically correspond to computing the sequences
hi = hi−1 + 1yi=1/N and h̃i = h̃i−1 + ci/N
with h0 = h̃0 = 0, and N is the total number of samples. Then the KS calibration error metric is given by
KS = max i ∣∣∣hi − h̃i∣∣∣ . Another measure is the Brier score (BS↓) (DeGroot & Fienberg, 1983), which estimates the mean squared error between the correctness of prediction and the confidence score:
BS(D) = 1 |D| ∑ (ci,yi)∈D (1ŷi=yi − ci) 2 (5)
For all the above metrics (ECE, KS, BS), lower is better.

heading: F FAIRNESS CALIBRATION AND EQUAL OPPORTUNITY (EQUAL FNR)
text: F.1 FAIRNESS CALIBRATION
Since the calibration map produced by beta calibration is monotone, the ordering of the images provided by the scores is the same as the ordering provided by the probabilities; therefore, the accuracy of the methods wheh thresholding remains unchanged. The calibration error (CE) measured with an adaptation of the Kolmogorov-Smirnov (KS) test (described in the Appendix) is computed for each subgroup of interest. Notice that for the BFW dataset we consider the eight subgroups that result from the intersection of the ethnicity and gender subgroups.
We first observe that all methods are equally globally calibrated (i.e., the calibration error is low) after the post-hoc calibration method is applied, except for the FTC on the RFW dataset (see the Global column in Table 5 and Table 6).
By inspecting Table 5 and Table 6, we notice that, after calibration, the Baseline method results in models that are not fairly-calibrated, though perhaps not in the way one would expect. Typically, bias is directed against minority groups, but in this case, it is the Caucasian subgroups that have the higher CEs. This is a consequence of the models’ above average accuracy on this subgroup, which is underestimated and therefore not captured by the calibration procedure. It is important to point out that this is not a failure of the calibration procedure, since the global CE (i.e., the CE measured on all pairs) is low, as discussed above.
F.2 EQUAL OPPORTUNITY
While equal opportunity (equal FNRs between subgroups) is not prioritized for the FR systems when used by law enforcement, it may be prioritized in different contexts such as office building security. Empirically, our method also mitigates the equal opportunity bias at low global FNRs, as can be seen in Table 7.

heading: G STANDARD POST-HOC CALIBRATION METHODS
text: For completeness, we provide a brief description of the post-hoc calibration methods used in this work. Beta calibration Kull et al. (2017) was used to obtain our main results, but we show below that choosing another method (histogram binning Zadrozny & Elkan (2001), isotonic regression (Zadrozny & Elkan, 2001; Niculescu-Mizil & Caruana, 2005)) does not impact the performance of our FairCal method.
Throughout this section, we denote by Pcal the pairs of images in the calibration set and Scal their respective cosine similarity scores such that
Scal = {s(x̂1, x̂2) : (x̂1, x̂2) ∈ Pcal}.
G.1 HISTOGRAM BINNING
In histogram binning Zadrozny & Elkan (2001), we partition Scal intom binsBi, where i = 1, . . . ,m. Then, given a pair of images (x1,x2) with score s(x1,x2) ∈ Bi, we define
c(x1,x2) = 1 |Bi| ∑
s(x̂1,x̂2)∈Bi (x1,x2)∈Pcal
1I(x̂1)=I(x̂2) (6)
In other words, we simply count the number of scores in each bin that correspond to genuine pairs of images, i.e., images that belong to the same person. By construction, a confidence score c (Equation 6) satisfies the binned version of the standard calibration (Definition 1). As for the bins, they can be chosen so as to have equal mass or to be equally spaced, or else by maximizing mutual information, as recently proposed in Patel et al. (2021). In this work, we created bins with equal mass.
Despite being an extremely computationally efficient method and providing good calibration, histogram binning is not guaranteed to preserve the monotonicity between scores and confidences, which is typically a desired property. Monotonicity ensures that the accuracy of the classifier is the same when thresholding either the scores or the calibrated confidences.
G.2 ISOTONIC REGRESSION
Isotonic Regression (Zadrozny & Elkan, 2001; Niculescu-Mizil & Caruana, 2005) learns a monotonic function µ : R→ R by solving
arg min µ
1 |Pcal| ∑
(x1,x2)∈Pcal
( µ(s(x1,x2))− 1I(x̂1)=I(x̂2) )2 The confidence score is then given by c(x1,x2) = µ(s(x1,x2)).
G.3 BETA CALIBRATION
Beta calibration Patel et al. (2021) is a parametric calibration method, which learns a calibration map µ : R→ R of the form
cθ(s) = µ(s; θ1, θ2, θ3) = 1 1 + 1/ ( eθ3 s θ1
(1−s)θ2 ) where the parameters θ = (θ1, θ2, θ3) ∈ R3 are chosen by minimizing the log-loss function
LL(c, y) = y(− log(c)) + (1− y)(− log(1− c))
where c = µ(s(x1,x2)). By restricting, a and b to be positive, the calibration map is monotone.

heading: H ROBUSTNESS OF FAIRCAL RESULTS TO PARAMETERS
text: In this section we show that the results presented in the main paper still hold if we vary model hyperparameters, such as the number K of clusters used in FairCal, and the calibration method.
Choice of post-hoc calibration: The implementation of the FairCal method requires choosing a post-hoc calibration method and the number of clusters K in the K-means algorithm. Our method is robust to the choice of both with respect to fairness-calibration (the metric of interest when it comes to calibration) and its bias as depicted in Figure 4, Figure 5, Figure 6 and Figure 7. We compare with binning and isotonic regression. For the former, we chose 10 and 25 bins for the RFW and BFW datasets, respectively, given the different number of pairs in each dataset.
Comparison to FSN (Terhörst et al., 2020b): The improved performance of FairCal over FSN is consistent across different choices of K. Fixing the choice of the post-hoc calibration method as beta calibration as in the results in the paper, we compare the two, together with Baseline and Oracle for additional baselines. Results are displayed in Figure 8, Figure 9, Figure 10, Figure 11, Figure 12, Figure 13 and Figure 14

heading: I EXTENDING FAIRCAL TO MULTI-CLASS CLASSIFICATION
text: Consider a probabilistic classifier p : X → ∆k that outputs that outputs class probabilities for k classes 1, . . . , k. Here X denotes the input space and ∆k = {(q1, . . . , qk) : ∑k i=1 qk = 1}. Suppose p is computed with a deep neural network where f denotes its feature encoder and g the final fully connected layer. Then we write p = softmax(g(f(x))). Post-hoc calibration methods (like Dirichlet calibration Kull et al. (2019), the generalization of Beta calibration) compute a calibration map µ : ∆k → ∆k using a calibration set X cal ⊆ X such that µ ◦ p is calibrated. FairCal can be extended to multi-class classification as follows:
(i) Compute Zcal = {f(x) : x ∈ X cal}. (ii) Apply the K-means algorithm to the image features, Zcal, partitioning the embedding space Z
into K clusters Z1, . . . ,Zk. These form the K calibration sets:
X calk = {x : f(x) ∈ Zk} , k = 1, . . . ,K (7)
(iii) For each calibration set X calk , use a post-hoc calibration method to compute the calibration map µk.
(iv) For a new input x find the clusters it belongs to and output µk(p(x)).
Notice that in the multi-class classification setting, we are no longer dealing with pairs and therefore each input (now a single image) belongs to one and only one cluster, thus simplifying the final setup. We leave for future work its implementation and study through a thorough empirical evaluation.

heading: J RESULTS PRESENTED WITH STANDARD DEVIATIONS
text: Recall that the results presented in the main text were computed by taking the mean of a 5-fold leave-one-out cross-validation. Below, we report the corresponding standard deviations of the five folds. The standard deviations for the results on accuracy reported in Table 2 can be found in Table 8, Table 9, Table 10. For fairness-calibration in Table 3, they can be found in Table 11, Table 12,Table 13, Table 14. Finally, for predictive equality and equal opportunity in Table 4 and Table 7, they can be found in Table 15, Table 16, Table 17 and Table 18, Table 19, Table 20.

heading: K ADDITIONAL FIGURES
text: In Figure 1, we provided a qualitative analysis of the improved fairness/reduction in bias in terms of predictive equality on the RFW dataset with the FaceNet (Webface) model. We replicate this figure for the FaceNet (VGGFace2) model on the RFW dataset in ?? and on the BFW dataset for the FaceNet (Webface) model in Figure 16 and ArcFace model in Figure 17.

Review summaries:
The paper proposes an approach, called FairCal, for fairly calibrating face verification models. The method does not require access to sensitive attributes for calibrating, instead the paper uses k-means algorithm to find clusters of data and applies beta-calibrating algorithm on each cluster. The authors show that after calibrating with FairCal the model still achieves good accuracy and outputs more fairness-calibrated probabilities. Also, the method reduces the FPR gap across sensitive attributes. 

This paper introduces a Fairness Calibration method for the task of face verification problem. The authors claimed that the proposed method can produce fairly-calibrated probabilities, reduce the gap in the false positive rates, does not require knowledge of the sensitive attribute, and does not require retraining, training an additional model, or retuning. Experimental results demonstrate that the proposed method can achieve state-of-the-art performance over the RFW and BFW databases.

The authors propose a simple cluster-conditional post-hoc strategy to calibrate the similarity scores obtained using a pre-trained network. In essence, this work proposes to use a cluster-specific (or group-specific in case of Oracle) calibration function (\mu), which results in improving the scores pertaining to verification of a x1-x2 pair (where both x1 and x2 belong to the same demographic group) mapping function. This is similar to choosing group-specific threshold demonstrated in the BFW paper (Robinson et al, CVPRW 2020). Unlike existing methods such as AGENDA, FTC etc. FairCal does not require the protected attribute labels during training and testing.